Puppet - Bug #16315 agents stop by execution-expired after logrotate having reloading them

Size: px

Start display at page:

Download "Puppet - Bug #16315 agents stop by execution-expired after logrotate having reloading them"

Theodora Hodge
6 years ago
Views:

1 Puppet - Bug #16315 agents stop by execution-expired after logrotate having reloading them 09/10/ :01 am - Andreas Unterkircher Status: Needs More Information Start date: 09/09/2012 Priority: Normal Due date: Assignee: Andreas Unterkircher % Done: 0% Category: Estimated time: 0.00 hour Target version: Affected Puppet version: Branch: Keywords: Description Having about 350 agents active in our network, now for some weeks all of them are suffering the same problem after being reloaded by logrotate on Sunday morning. Agents and master are running version on Debian Squeeze (ruby 1.8.7).Our agent config is [main] logdir=/var/log/puppet vardir=/var/lib/puppet rundir=/var/run/puppet [agent] server=puppet.example.com storeconfigs=false listen=false report=false splay=true runinterval = 3600 syslogfacility = local3 masterport=8140 As so many of our agents were affected I first thought it is related to our master.i have reconfigured it from Webrick (which was running very well at that time) to Passenger.But the issue remained the same. What happens is that on Sunday 6:25 logrotate gets activated by cron.logrotate then rotates /var/log/puppet/http.log according to /etc/logrotate.d/puppet. /var/log/puppet/*log { missingok create 0644 puppet puppet compress rotate 4 postrotate [ -e /etc/init.d/puppetmaster ] && /etc/init.d/puppetmaster reload >/dev/null 2>&1 true [ -e /etc/init.d/puppet ] && /etc/init.d/puppet reload > /dev/null 2>&1 true endscript } 05/01/2016 1/6

2 After being reloaded, all agents are successfully executing one (and only one) more run. After that all of them start to log execution-expired messages and stop working. In syslog it looks like this. >>>> after being reloaded at 6:25, this is the first run Sep 9 06:38:47 server puppet-agent[3316]: (/Stage[main]/Nagios-common/Modulefile[/var/run/nagios.tick]/File[/var/run/nagios.tick]/content) content changed '{md5}611d0e76b249daa2ba57118ccd63bc4b' to '{md5}0e11de68be51c68e1d3d2f947479b756' Sep 9 06:39:00 server puppet-agent[3316]: Finished catalog run in seconds >>>> starting from here, only execution-expired's Sep 9 07:27:18 server puppet-agent[3316]: Could not retrieve catalog from remote server: execution expired Sep 9 07:27:19 server puppet-agent[3316]: Using cached catalog Sep 9 08:27:18 server puppet-agent[3316]: Could not retrieve catalog from remote server: execution expired Sep 9 08:27:19 server puppet-agent[3316]: Using cached catalog Sep 9 08:27:19 server puppet-agent[3316]: Could not retrieve catalog; skipping run Sep 9 09:27:22 server puppet-agent[3316]: Could not retrieve catalog from remote server: execution expired Sep 9 09:27:23 server puppet-agent[3316]: Using cached catalog Sep 9 09:27:23 server puppet-agent[3316]: Could not retrieve catalog; skipping run Sep 9 10:27:28 server puppet-agent[3316]: Could not retrieve catalog from remote server: execution expired Sep 9 10:27:28 server puppet-agent[3316]: Using cached catalog The same picture on all of our other nodes.i can trigger a successful manual run on those machine at any time.so I don t think that it is caused by the master. I have tried already changing the reload in logrotate to restart.but the issue remains the same. Even keeping the agent running in foreground and having debug & trace enabled doesn t give me a hint.it just starts logging those execution-expired messages and is even not attempting to connect to the master (verified by tcpdump). It s even hard to reproduce.when I m sending a manual SIGHUP to a running agent it doesn t trigger this behaviour.it seems that only if the agent was running for some time (days) and then being reloaded/restart by logrotate triggers our problem. Does anyone have a hint how I can further try to track down this issue? History #1-09/10/ :02 am - Andreas Unterkircher Forgot to say to recover from it, I have to kill all agents sudo killall puppet and start them again sudo /etc/init.d/puppet start #2-09/10/ :49 am - eric sorenson - Status changed from Unreviewed to Needs More Information 05/01/2016 2/6

3 - Assignee set to Andreas Unterkircher Hallo Andreas I have not heard of this happening before. Can you post this report to puppet-users and reference the bug in case other users have seen this? One thing I don t understand, if it is your agents, why do they have a /var/log/puppet/http.log logrotate, and a reload of puppetmaster afterwards, configured on them at all? It almost seems like the name resolution is failing after the reload, like it is trying to connect to a server name that times out. When you remove this logrotate entry on the agents, they run normally through the night? #3-09/10/ :18 pm - Andreas Unterkircher Hallo Andreas I have not heard of this happening before. Can you post this report to puppet-users and reference the bug in case other users have seen this? Good tip! I will try. One thing I don t understand, if it is your agents, why do they have a /var/log/puppet/http.log logrotate, and a reload of puppetmaster afterwards, configured on them at all? Only if puppetmaster is installed what is an extra package on Debian (probably related #12591). It almost seems like the name resolution is failing after the reload, like it is trying to connect to a server name that times out. I will strace now one of them. Maybe it gives a hint what the agents are doing when they stumble. When you remove this logrotate entry on the agents, they run normally through the night? Exactly. Then they keep on running. #4-09/11/ :51 am - Andreas Unterkircher That happens on tracing the process Hanging around waiting for its splaylimit I guess stat64("/etc/puppet/puppet.conf", {st_mode=s_ifreg 0644, st_size=312,...}) = select(1, [], [], [], {15, 0}) = 0 (Timeout) gettimeofday({ , }, NULL) = gettimeofday({ , }, NULL) = gettimeofday({ , }, NULL) = stat64("/etc/puppet/puppet.conf", {st_mode=s_ifreg 0644, st_size=312,...}) = select(1, [], [], [], {15, 0}) = 0 (Timeout) gettimeofday({ , }, NULL) = 0 05/01/2016 3/6

4 22613 gettimeofday({ , }, NULL) = gettimeofday({ , }, NULL) = stat64("/etc/puppet/puppet.conf", {st_mode=s_ifreg 0644, st_size=312,...}) = 0 once it is time to wake up, I see ruby libraries get loaded, facter runs, but the interesting part is once it starts connecting to the master recvfrom(4, "r\236\201\200\0\1\0\1\0\0\0\0\6puppet\tmm-karton\3co"..., 1024, 0, {sa_family=af_inet, sin_port=htons (53), sin_addr=inet_addr(" ")}, [16]) = close(4) = socket(pf_inet, SOCK_STREAM, IPPROTO_TCP) = fcntl64(4, F_GETFL) = 0x2 (flags O_RDWR) fcntl64(4, F_SETFL, O_RDWR O_NONBLOCK) = connect(4, {sa_family=af_inet, sin_port=htons(8140), sin_addr=inet_addr(" ")}, 16) = -1 EINPROGRESS (Op eration now in progress) clock_gettime(clock_monotonic, { , }) = select(6, [], [4], [4], {119, } 1747 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) 1747 clock_gettime(clock_realtime, { , }) = futex(0xb77b4370, FUTEX_WAKE_PRIVATE, 1) = clock_gettime(clock_realtime, { , }) = futex(0xb77b4344, FUTEX_WAIT_PRIVATE, , {0, }) = -1 ETIMEDOUT (Connection timed out) 1747 clock_gettime(clock_realtime, { , }) = futex(0xb77b4344, FUTEX_WAIT_PRIVATE, , {0, } <... select resumed> ) = 1 (out [4], left {119, }) connect(4, {sa_family=af_inet, sin_port=htons(8140), sin_addr=inet_addr(" ")}, 16) = fcntl64(4, F_SETFL, O_RDWR) = 0 there is now a successfully opened TCP connection to our master at :8140.Then the agents parent process seems to create another thread (2016) read(4, "ibz\3\264@\2253\360\16\355\236\25,\3340+\336+\27\353dy\10\211z\n\340\374\376\"<"..., 48) = write(4, "\27\3\1\1\0\33\34s\347\326\\\v.\231\334\321}\310\305=rU\"\7\6\t\266Bq\224g\275"..., 261) = write(4, "\27\3\1\v\200\325\303\237\357\367E\357~\370\273uZ\\\317a\353w\5\351k\257;\\\25\263\36\320"..., 2949) = rt_sigaction(sigvtalrm, {0xb770f1b0, [], 0}, {0xb770f1b0, [], 0}, 8) = clone(child_stack=0xb6a98494, flags=clone_vm CLONE_FS CLONE_FILES CLONE_SIGHAND CLONE_THREAD CLONE_SYSVSEM CLONE_SETTLS CLONE_PARENT_SETTID CL T_SETTID CLONE_CHILD_CLEARTID, parent_tidptr=0xb6a98bd8, {entry_number:6, base_addr:0xb6a98b70, limit: , seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}, child_tidptr=0xb6a98bd8) = set_robust_list(0xb6a98be0, 0xc futex(0xb77b43a4, FUTEX_WAIT_PRIVATE, 2411, NULL and that one is immediately telling us connection-timed-out 2016 clock_gettime(clock_realtime, { , }) = clock_gettime(clock_realtime, { , }) = futex(0xb77b4344, FUTEX_WAIT_PRIVATE, , {0, }) = -1 ETIMEDOUT (Connection timed out) 2016 clock_gettime(clock_realtime, { , }) = 0 05/01/2016 4/6

5 2016 clock_gettime(clock_realtime, { , }) = futex(0xb77b4344, FUTEX_WAIT_PRIVATE, , {0, }) = -1 ETIMEDOUT (Connection timed out) 2016 clock_gettime(clock_realtime, { , }) = clock_gettime(clock_realtime, { , }) = futex(0xb77b4344, FUTEX_WAIT_PRIVATE, , {0, }) = -1 ETIMEDOUT (Connection timed out) 2016 clock_gettime(clock_realtime, { , }) = 0... till some when has enough, kills the thread and logs the execution-expired message futex(0xb77b4344, FUTEX_WAIT_PRIVATE, , {0, }) = -1 ETIMEDOUT (Connection timed out) 2016 clock_gettime(clock_realtime, { , }) = clock_gettime(clock_realtime, { , }) = futex(0xb77b4344, FUTEX_WAIT_PRIVATE, , {0, }) = -1 ETIMEDOUT (Connection timed out) 2016 clock_gettime(clock_realtime, { , }) = clock_gettime(clock_realtime, { , }) = futex(0xb77b4344, FUTEX_WAIT_PRIVATE, , {0, } <... select resumed> ) = 0 (Timeout) clock_gettime(clock_monotonic, { , }) = select(5, [4], [], [], {0, 0}) = 0 (Timeout) time(null) = select(5, [4], [], [], {0, 0}) = 0 (Timeout) rt_sigprocmask(sig_setmask, [], NULL, 8) = rt_sigprocmask(sig_setmask, [], NULL, 8) = futex(0xb77b4344, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xb77b4340, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1} 2016 <... futex resumed> ) = 0 But at the same time I can get a manual run by puppetd --test --masterport server I m rather clueless what is going here #5-09/27/ :54 am - Martijn Grendelman Not sure if my problem is exactly the same, but I am having similar issues. Every once in a while, all of my seventy-something agents start producing execution expired messages, that do not seem related to master load or something like that. In the past, the problem magically went away after some time, or so it seemed. I could never find the cause (concentrating on the master), and restarting the Puppet master never had any real effect. After finding this bug report yesterday, when everything was acting up again, I stopped and restarted all the agents, and the problem went away 05/01/2016 5/6

6 Master and all agents are on Ubuntu Lucid. I am not sure if I had this problem with earlier versions. #6-09/27/ :52 pm - Andreas Unterkircher What I find strange that even if I adapt the logrotate script that runs on Sunday morning to postrotate /etc/init.d/puppet stop && sleep 3 && killall puppet && sleep 3 && /etc/init.d/puppet start endscript the situation remains exactly the same. agents start, are doing exactly one more run and after it, only execution-expired. I will now try to remove the logrotate script (/etc/logrotate.d/puppet) and see what is going to happen over the weekend. #7-10/02/ :17 am - Andreas Unterkircher Almost as expected without logrotate restating all agents, none of them got into this execution-expired state. And it looks like the Debian puppet agent packages (>= Debian Squeeze) have dropped /etc/logrotate.d/puppet already and are no longer shipping. Only puppetmasters Debian package contains a logrotate script. Mine must be a residue from an older package version that has not got purged during upgrades. So either that all has something to do with the way the agents get restarted by logrotate.or it has something to do with the fact that puppetmaster (apache & passanger) gets also reloaded at the same time by logrotate to rotate Apache s logfiles while the agents are restarting too. 05/01/2016 6/6

On a Ubuntu system which was updated today to facter (diretly via apt.puppetlabs.com) I also see exactly the same error.

On a Ubuntu system which was updated today to facter (diretly via apt.puppetlabs.com) I also see exactly the same error. Facter - Bug #17383 Could not retrieve local facts: undefined method `kernel_fact_value' for Facter::Util::Processor:Module 11/01/2012 11:31 pm - Anand Kumria Status: Re-opened Start date: Priority: Normal