Bug 207896 - net-mgmt/zabbix2-server zabbix{2,22,24}-{agent,proxy,server) rc.d fixes for SysV IPC leak, restart race, custom PATH
Summary: net-mgmt/zabbix2-server zabbix{2,22,24}-{agent,proxy,server) rc.d fixes for S...
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Lars Engels
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-03-11 08:41 UTC by Kevin Bowling
Modified: 2016-04-24 12:00 UTC (History)
3 users (show)

See Also:
bugzilla: maintainer-feedback? (pg)


Attachments
update rc.d scripts to SIGTERM only master PID (14.85 KB, patch)
2016-03-11 08:41 UTC, Kevin Bowling
pg: maintainer-approval+
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Kevin Bowling freebsd_committer 2016-03-11 08:41:52 UTC
Created attachment 167991 [details]
update rc.d scripts to SIGTERM only master PID

We were losing many hundred zabbix agents out of many thousands when we would restart zabbix_agentd using the rc.d script in ports currently.

I believe the root cause is that rc.subr by default checks against the command name and was sending SIGTERM to the parent and all the child processes (we run w/~32 children because if you run out with concurrent polls, the check will fail).  The processes have a sighandler and may terminate uncleanly if the other process are all SIGTERMing.  ipcs may show shared memory and/or a semaphore even though there are no processes running, and the agent cannot be [re]started until those are manually cleared with ipcrm.  The SIGTERM to all processes also seemed to cause restart to race occasionally, in that some of the child processes were still around while the new one was starting even if there was no IPC leak.. may be hard to trigger in practice unless you have long running userparams(?).

This patch also sets a default PATH and allows the user to override it in rc.conf.  This is often necessary for userparams on the agent, or externals on the proxy and server.

I've only tested the agent rc.d changes, but assume the proxy and server face these same issues.  I'm not very familiar with rc.subr so suggestions for improvement welcome.

Interesting poll of other platforms I can easily check with respect to pidfile:  Ubuntu 14.04 SysV init gets it right, Ubuntu 14.04 upstart job gets it wrong, CentOS 5 SysV init gets it wrong, and CentOS 7 systemd service gets it right.
Comment 1 Kevin Bowling freebsd_committer 2016-03-12 04:04:49 UTC
I've confirmed with two global restarts I no longer lose any agents.
Comment 2 Mark Felder freebsd_committer 2016-03-15 13:22:00 UTC
I'll take this, pending maintainer feedback
Comment 3 pg 2016-03-15 13:55:58 UTC
In the near future I'll rewiev patch.
Comment 4 pg 2016-03-24 12:03:37 UTC
Approve, thanks!
Comment 5 Kevin Bowling freebsd_committer 2016-04-16 04:02:04 UTC
Ready for commit?  Needs to happen to zabbix3 now too.
Comment 6 commit-hook freebsd_committer 2016-04-24 11:49:02 UTC
A commit references this bug:

Author: lme
Date: Sun Apr 24 11:48:28 UTC 2016
New revision: 413935
URL: https://svnweb.freebsd.org/changeset/ports/413935

Log:
  - Fixes in {agent,proxy,server} rc start scripts:
    - SysV IPC leak
    - Restart race
    - Allow to specify a custom PATH in rc.conf
  - Bump PORTREVISION

  PR:		207896
  Submitted by:	kbowling
  Approved by:	maintainer
  MFH:		2016Q2
  Sponsored by:	Essen Linuxhotel Hackathon 2016

Changes:
  head/net-mgmt/zabbix2-server/Makefile
  head/net-mgmt/zabbix2-server/files/zabbix_agentd.in
  head/net-mgmt/zabbix2-server/files/zabbix_proxy.in
  head/net-mgmt/zabbix2-server/files/zabbix_server.in
  head/net-mgmt/zabbix22-server/Makefile
  head/net-mgmt/zabbix22-server/files/zabbix_agentd.in
  head/net-mgmt/zabbix22-server/files/zabbix_proxy.in
  head/net-mgmt/zabbix22-server/files/zabbix_server.in
  head/net-mgmt/zabbix24-server/Makefile
  head/net-mgmt/zabbix24-server/files/zabbix_agentd.in
  head/net-mgmt/zabbix24-server/files/zabbix_proxy.in
  head/net-mgmt/zabbix24-server/files/zabbix_server.in
  head/net-mgmt/zabbix3-server/Makefile
  head/net-mgmt/zabbix3-server/files/zabbix_agentd.in
  head/net-mgmt/zabbix3-server/files/zabbix_proxy.in
  head/net-mgmt/zabbix3-server/files/zabbix_server.in
Comment 7 Lars Engels freebsd_committer 2016-04-24 11:51:02 UTC
Committed together with the changes to the zabbix3 ports.
Comment 8 commit-hook freebsd_committer 2016-04-24 12:00:04 UTC
A commit references this bug:

Author: lme
Date: Sun Apr 24 11:59:39 UTC 2016
New revision: 413936
URL: https://svnweb.freebsd.org/changeset/ports/413936

Log:
  MFH: r413935

  - Fixes in {agent,proxy,server} rc start scripts:
    - SysV IPC leak
    - Restart race
    - Allow to specify a custom PATH in rc.conf
  - Bump PORTREVISION

  PR:		207896
  Submitted by:	kbowling
  Approved by:	maintainer
  Sponsored by:	Essen Linuxhotel Hackathon 2016

  Approved by:	portmgr (bapt)

Changes:
_U  branches/2016Q2/
  branches/2016Q2/net-mgmt/zabbix2-server/Makefile
  branches/2016Q2/net-mgmt/zabbix2-server/files/zabbix_agentd.in
  branches/2016Q2/net-mgmt/zabbix2-server/files/zabbix_proxy.in
  branches/2016Q2/net-mgmt/zabbix2-server/files/zabbix_server.in
  branches/2016Q2/net-mgmt/zabbix22-server/Makefile
  branches/2016Q2/net-mgmt/zabbix22-server/files/zabbix_agentd.in
  branches/2016Q2/net-mgmt/zabbix22-server/files/zabbix_proxy.in
  branches/2016Q2/net-mgmt/zabbix22-server/files/zabbix_server.in
  branches/2016Q2/net-mgmt/zabbix24-server/Makefile
  branches/2016Q2/net-mgmt/zabbix24-server/files/zabbix_agentd.in
  branches/2016Q2/net-mgmt/zabbix24-server/files/zabbix_proxy.in
  branches/2016Q2/net-mgmt/zabbix24-server/files/zabbix_server.in
  branches/2016Q2/net-mgmt/zabbix3-server/Makefile
  branches/2016Q2/net-mgmt/zabbix3-server/files/zabbix_agentd.in
  branches/2016Q2/net-mgmt/zabbix3-server/files/zabbix_proxy.in
  branches/2016Q2/net-mgmt/zabbix3-server/files/zabbix_server.in