Bug 217055

Summary:	Consolidate random sleeps in periodic scripts
Product:	Base System	Reporter:	Alan Somers <asomers>
Component:	conf	Assignee:	Alan Somers <asomers>
Status:	Closed FIXED
Severity:	Affects Many People	CC:	brd
Priority:	---	Flags:	asomers: mfc-stable11+
Version:	CURRENT
Hardware:	Any
OS:	Any

Description Alan Somers freebsd_committer

2017-02-13 05:28:03 UTC

On my system, there are three separate periodic scripts that try to sleep for a random amount of time to reduce congestion on a download server somewhere: 300.statistics (from sysutils/bsdstat), 410.pkg-audit (from ports-mgmt/pkg), and 480.leapfile-ntpd (from the base).  There may be more throughout the ports tree.  A single sleep would be sufficient, but if all three scripts decide to sleep, the total sleep time can get quite long.

I propose we add a 000.anticongestion periodic script, enabled by default for daily, weekly, and monthly.  It will sleep for 0 - 60 minutes, unless periodic is being run interactively.  Then we can remove the sleep in 480.leapfile-ntpd, and conditionalize the ports sleeps on __FreeBSD_version.

Comment 1 Alan Somers freebsd_committer

2017-02-13 05:28:23 UTC

I'll take it.

Comment 2 Brad Davis freebsd_committer

2017-02-13 16:27:33 UTC

I think this is a good idea.

Especially with a variable so you can override the randomness and gain some control of when your scripts run.  Like a variable to set the delay, i.e.:

anticongestion_sleep_override=900

Comment 3 commit-hook freebsd_committer

2017-04-01 04:44:45 UTC

A commit references this bug:

Author: asomers
Date: Sat Apr  1 04:42:35 UTC 2017
New revision: 316342
URL: https://svnweb.freebsd.org/changeset/base/316342

Log:
  Consolidate random sleeps in periodic scripts

  Multiple periodic scripts sleep for a random amount of time in order to
  mitigate the thundering herd problem. This is bad, because the sum of
  multiple uniformly distributed random variables approaches a normal
  distribution, so the problem isn't mitigated as effectively as it would be
  with a single sleep.

  This change creates a single configurable anticongestion sleep. periodic
  will only sleep if at least one script requires it, and it will never sleep
  more than once per invocation. It also won't sleep if periodic was run
  interactively, fixing an unrelated longstanding bug.

  PR:		217055
  PR:		210188
  Reviewed by:	cy
  MFC after:	3 weeks
  Differential Revision:	https://reviews.freebsd.org/D10211

Changes:
  head/etc/defaults/periodic.conf
  head/etc/periodic/daily/480.leapfile-ntpd
  head/share/man/man5/periodic.conf.5
  head/usr.sbin/periodic/periodic.sh

Comment 4 commit-hook freebsd_committer

2017-04-24 15:29:44 UTC

A commit references this bug:

Author: asomers
Date: Mon Apr 24 15:29:18 UTC 2017
New revision: 317373
URL: https://svnweb.freebsd.org/changeset/base/317373

Log:
  MFC r316342, r316358

  r316342:
  Consolidate random sleeps in periodic scripts

  Multiple periodic scripts sleep for a random amount of time in order to
  mitigate the thundering herd problem. This is bad, because the sum of
  multiple uniformly distributed random variables approaches a normal
  distribution, so the problem isn't mitigated as effectively as it would be
  with a single sleep.

  This change creates a single configurable anticongestion sleep. periodic
  will only sleep if at least one script requires it, and it will never sleep
  more than once per invocation. It also won't sleep if periodic was run
  interactively, fixing an unrelated longstanding bug.

  PR:		217055
  PR:		210188
  Reviewed by:	cy
  MFC after:	3 weeks
  Differential Revision:	https://reviews.freebsd.org/D10211

  r316358:
  Fix man page typo from r316342

  Reported by:	rgrimes
  MFC after:	20 days
  X-MFC-With:	316342

Changes:
_U  stable/11/
  stable/11/etc/defaults/periodic.conf
  stable/11/etc/periodic/daily/480.leapfile-ntpd
  stable/11/share/man/man5/periodic.conf.5
  stable/11/usr.sbin/periodic/periodic.sh