Created attachment 167991 [details] update rc.d scripts to SIGTERM only master PID We were losing many hundred zabbix agents out of many thousands when we would restart zabbix_agentd using the rc.d script in ports currently. I believe the root cause is that rc.subr by default checks against the command name and was sending SIGTERM to the parent and all the child processes (we run w/~32 children because if you run out with concurrent polls, the check will fail). The processes have a sighandler and may terminate uncleanly if the other process are all SIGTERMing. ipcs may show shared memory and/or a semaphore even though there are no processes running, and the agent cannot be [re]started until those are manually cleared with ipcrm. The SIGTERM to all processes also seemed to cause restart to race occasionally, in that some of the child processes were still around while the new one was starting even if there was no IPC leak.. may be hard to trigger in practice unless you have long running userparams(?). This patch also sets a default PATH and allows the user to override it in rc.conf. This is often necessary for userparams on the agent, or externals on the proxy and server. I've only tested the agent rc.d changes, but assume the proxy and server face these same issues. I'm not very familiar with rc.subr so suggestions for improvement welcome. Interesting poll of other platforms I can easily check with respect to pidfile: Ubuntu 14.04 SysV init gets it right, Ubuntu 14.04 upstart job gets it wrong, CentOS 5 SysV init gets it wrong, and CentOS 7 systemd service gets it right.
I've confirmed with two global restarts I no longer lose any agents.
I'll take this, pending maintainer feedback
In the near future I'll rewiev patch.
Approve, thanks!
Ready for commit? Needs to happen to zabbix3 now too.
A commit references this bug: Author: lme Date: Sun Apr 24 11:48:28 UTC 2016 New revision: 413935 URL: https://svnweb.freebsd.org/changeset/ports/413935 Log: - Fixes in {agent,proxy,server} rc start scripts: - SysV IPC leak - Restart race - Allow to specify a custom PATH in rc.conf - Bump PORTREVISION PR: 207896 Submitted by: kbowling Approved by: maintainer MFH: 2016Q2 Sponsored by: Essen Linuxhotel Hackathon 2016 Changes: head/net-mgmt/zabbix2-server/Makefile head/net-mgmt/zabbix2-server/files/zabbix_agentd.in head/net-mgmt/zabbix2-server/files/zabbix_proxy.in head/net-mgmt/zabbix2-server/files/zabbix_server.in head/net-mgmt/zabbix22-server/Makefile head/net-mgmt/zabbix22-server/files/zabbix_agentd.in head/net-mgmt/zabbix22-server/files/zabbix_proxy.in head/net-mgmt/zabbix22-server/files/zabbix_server.in head/net-mgmt/zabbix24-server/Makefile head/net-mgmt/zabbix24-server/files/zabbix_agentd.in head/net-mgmt/zabbix24-server/files/zabbix_proxy.in head/net-mgmt/zabbix24-server/files/zabbix_server.in head/net-mgmt/zabbix3-server/Makefile head/net-mgmt/zabbix3-server/files/zabbix_agentd.in head/net-mgmt/zabbix3-server/files/zabbix_proxy.in head/net-mgmt/zabbix3-server/files/zabbix_server.in
Committed together with the changes to the zabbix3 ports.
A commit references this bug: Author: lme Date: Sun Apr 24 11:59:39 UTC 2016 New revision: 413936 URL: https://svnweb.freebsd.org/changeset/ports/413936 Log: MFH: r413935 - Fixes in {agent,proxy,server} rc start scripts: - SysV IPC leak - Restart race - Allow to specify a custom PATH in rc.conf - Bump PORTREVISION PR: 207896 Submitted by: kbowling Approved by: maintainer Sponsored by: Essen Linuxhotel Hackathon 2016 Approved by: portmgr (bapt) Changes: _U branches/2016Q2/ branches/2016Q2/net-mgmt/zabbix2-server/Makefile branches/2016Q2/net-mgmt/zabbix2-server/files/zabbix_agentd.in branches/2016Q2/net-mgmt/zabbix2-server/files/zabbix_proxy.in branches/2016Q2/net-mgmt/zabbix2-server/files/zabbix_server.in branches/2016Q2/net-mgmt/zabbix22-server/Makefile branches/2016Q2/net-mgmt/zabbix22-server/files/zabbix_agentd.in branches/2016Q2/net-mgmt/zabbix22-server/files/zabbix_proxy.in branches/2016Q2/net-mgmt/zabbix22-server/files/zabbix_server.in branches/2016Q2/net-mgmt/zabbix24-server/Makefile branches/2016Q2/net-mgmt/zabbix24-server/files/zabbix_agentd.in branches/2016Q2/net-mgmt/zabbix24-server/files/zabbix_proxy.in branches/2016Q2/net-mgmt/zabbix24-server/files/zabbix_server.in branches/2016Q2/net-mgmt/zabbix3-server/Makefile branches/2016Q2/net-mgmt/zabbix3-server/files/zabbix_agentd.in branches/2016Q2/net-mgmt/zabbix3-server/files/zabbix_proxy.in branches/2016Q2/net-mgmt/zabbix3-server/files/zabbix_server.in