Bug 210188 - periodic daily sleeps even when invoked from a terminal
Summary: periodic daily sleeps even when invoked from a terminal
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 10.3-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: Alan Somers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-06-10 10:07 UTC by Martin S. Weber
Modified: 2017-04-24 16:15 UTC (History)
1 user (show)

See Also:
asomers: mfc-stable11+


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Martin S. Weber 2016-06-10 10:07:22 UTC
periodic daily, when invoked via terminal, can be witnessed sleeping (culprit being fetching port vulnerabilities, see below)

# sh -x /usr/sbin/periodic daily                                                                                                 
+ [ 1 -lt 1 ]
+ [ -r /etc/defaults/periodic.conf ]
+ . /etc/defaults/periodic.conf
+ periodic_conf_files='/etc/periodic.conf /etc/periodic.conf.local'
+ local_periodic=/usr/local/etc/periodic
+ daily_output=root
+ daily_show_success=YES
+ daily_show_info=YES
+ daily_show_badconfig=NO
+ daily_clean_disks_enable=NO
+ daily_clean_disks_files='[#,]* .#* a.out *.core *.CKP .emacs_[0-9]*'
+ daily_clean_disks_days=3
+ daily_clean_disks_verbose=YES
+ daily_clean_tmps_enable=NO
+ daily_clean_tmps_dirs=/tmp
+ daily_clean_tmps_days=3
+ daily_clean_tmps_ignore='.X*-lock .X11-unix .ICE-unix .font-unix .XIM-unix'
+ daily_clean_tmps_ignore='.X*-lock .X11-unix .ICE-unix .font-unix .XIM-unix quota.user quota.group .snap'
+ daily_clean_tmps_ignore='.X*-lock .X11-unix .ICE-unix .font-unix .XIM-unix quota.user quota.group .snap .sujournal'
+ daily_clean_tmps_verbose=YES
+ daily_clean_preserve_enable=YES
+ daily_clean_preserve_days=7
+ daily_clean_preserve_verbose=YES
+ daily_clean_msgs_enable=YES
+ daily_clean_msgs_days=''
+ daily_clean_rwho_enable=YES
+ daily_clean_rwho_days=7
+ daily_clean_rwho_verbose=YES
+ daily_clean_hoststat_enable=YES
+ daily_backup_passwd_enable=YES
+ daily_backup_aliases_enable=YES
+ daily_backup_pkgdb_enable=YES
+ daily_backup_pkgdb_dir=/var/backups
+ daily_calendar_enable=NO
+ daily_accounting_enable=YES
+ daily_accounting_compress=NO
+ daily_accounting_flags=-q
+ daily_accounting_save=3
+ daily_news_expire_enable=YES
+ daily_status_disks_enable=YES
+ daily_status_disks_df_flags='-l -h'
+ daily_status_graid_enable=NO
+ daily_status_zfs_enable=NO
+ daily_status_zfs_zpool_list_enable=YES
+ daily_status_gmirror_enable=NO
+ daily_status_graid3_enable=NO
+ daily_status_gstripe_enable=NO
+ daily_status_gconcat_enable=NO
+ daily_status_network_enable=YES
+ daily_status_network_usedns=YES
+ daily_status_network_netstat_flags=-d
+ daily_status_rwho_enable=YES
+ daily_status_mailq_enable=YES
+ daily_status_mailq_shorten=NO
+ daily_status_include_submit_mailq=YES
+ daily_status_security_enable=YES
+ daily_status_security_inline=NO
+ daily_status_security_output=root
+ daily_status_mail_rejects_enable=YES
+ daily_status_mail_rejects_logs=3
+ daily_status_mail_rejects_shorten=NO
+ daily_ntpd_leapfile_enable=NO
+ daily_ntpd_avoid_congestion=YES
+ daily_status_ntpd_enable=NO
+ daily_queuerun_enable=YES
+ daily_submit_queuerun=YES
+ daily_status_world_kernel=YES
+ daily_scrub_zfs_enable=NO
+ daily_scrub_zfs_pools=''
+ daily_scrub_zfs_default_threshold=35
+ daily_local=/etc/daily.local
+ weekly_output=root
+ weekly_show_success=YES
+ weekly_show_info=YES
+ weekly_show_badconfig=NO
+ weekly_locate_enable=YES
+ weekly_whatis_enable=YES
+ weekly_catman_enable=NO
+ weekly_noid_enable=NO
+ weekly_noid_dirs=/
+ weekly_status_security_enable=YES
+ weekly_status_security_inline=NO
+ weekly_status_security_output=root
+ weekly_local=/etc/weekly.local
+ monthly_output=root
+ monthly_show_success=YES
+ monthly_show_info=YES
+ monthly_show_badconfig=NO
+ monthly_accounting_enable=YES
+ monthly_status_security_enable=YES
+ monthly_status_security_inline=NO
+ monthly_status_security_output=root
+ monthly_local=/etc/monthly.local
+ security_status_logdir=/var/log
+ security_status_diff_flags='-b -u'
+ security_status_chksetuid_enable=YES
+ security_status_chksetuid_period=daily
+ security_status_neggrpperm_enable=YES
+ security_status_neggrpperm_period=daily
+ security_status_chkmounts_enable=YES
+ security_status_chkmounts_period=daily
+ security_status_noamd=NO
+ security_status_chkuid0_enable=YES
+ security_status_chkuid0_period=daily
+ security_status_passwdless_enable=YES
+ security_status_passwdless_period=daily
+ security_status_logincheck_enable=YES
+ security_status_logincheck_period=daily
+ security_status_chkportsum_enable=NO
+ security_status_chkportsum_period=daily
+ security_status_ipfwdenied_enable=YES
+ security_status_ipfwdenied_period=daily
+ security_status_ipfdenied_enable=YES
+ security_status_ipfdenied_period=daily
+ security_status_pfdenied_enable=YES
+ security_status_pfdenied_period=daily
+ security_status_ipfwlimit_enable=YES
+ security_status_ipfwlimit_period=daily
+ security_status_ipf6denied_enable=YES
+ security_status_ipf6denied_period=daily
+ security_status_kernelmsg_enable=YES
+ security_status_kernelmsg_period=daily
+ security_status_loginfail_enable=YES
+ security_status_loginfail_period=daily
+ security_status_tcpwrap_enable=YES
+ security_status_tcpwrap_period=daily
+ [ -z '' ]
+ source_periodic_confs_defined=yes
+ source_periodic_confs
+ local i sourced_files
+ sourced_files=:/etc/periodic.conf:
+ [ -r /etc/periodic.conf ]
+ sourced_files=:/etc/periodic.conf::/etc/periodic.conf.local:
+ [ -r /etc/periodic.conf.local ]
+ . /etc/periodic.conf.local
+ daily_backup_aliases_enable=NO
+ daily_clean_hoststat_enable=NO
+ daily_status_mail_rejects_enable=NO
+ daily_status_include_submit_mailq=NO
+ daily_submit_queuerun=NO
+ daily_output=/var/log/daily.log
+ security_output=/var/log/security
+ weekly_output=/var/log/weekly.log
+ monthly_output=/var/log/monthly.log
+ hostname
+ host=hephaistos.local
+ export host
+ [ daily != LOCKED ]
+ ret=0
+ lockfile=/var/run/periodic.daily.lock
+ lockf -t 0 /var/run/periodic.daily.lock /bin/sh /usr/sbin/periodic LOCKED daily
load: 0.31  cmd: find 1730 [running] 1.97r 0.13u 1.76s 18% 9952k
load: 0.34  cmd: find 1730 [biord] 9.94r 0.33u 3.38s 17% 12760k
load: 0.31  cmd: find 1730 [biord] 14.85r 0.36u 3.61s 12% 14020k
load: 0.31  cmd: find 1730 [running] 17.80r 0.38u 3.85s 11% 14024k
load: 0.31  cmd: find 1730 [biord] 18.51r 0.38u 3.90s 10% 14024k
load: 0.28  cmd: sleep 1853 [nanslp] 33.72r 0.00u 0.00s 0% 1904k
load: 0.28  cmd: sleep 1853 [nanslp] 35.25r 0.00u 0.00s 0% 1912k
load: 0.22  cmd: sleep 1853 [nanslp] 46.57r 0.00u 0.00s 0% 1912k
load: 0.22  cmd: sleep 1853 [nanslp] 46.95r 0.00u 0.00s 0% 1912k
load: 0.19  cmd: sleep 1853 [nanslp] 149.61r 0.00u 0.00s 0% 1912k


(the output at the end naturally comes from me hitting ^T)

periodic line 118, invoking $file < /dev/null does probably not play too well with [ -t 0 ] tests to determine whether the script is being invoked from cron.

One would assume that it's one of the /etc/periodic scripts that's sleeping. Interestingly, # grep -lri sleep .
./daily/480.leapfile-ntpd

and given my daily_ntpd_leapfile_enable=NO, that shouldn't be doing anything.

So let's look at /usr/local/etc/periodic :   # grep -lri sleep .                                                                    
./security/410.pkg-audit

and indeed there we can see (L59+):
                # Random delay so the mirrors do not get slammed when run by periodic(8)
                if [ ! -t 0 ]; then
                        sleep `jot -r 1 0 3600`
                fi

As we've seen above, this script gets invoked with < /dev/null > tempfile, so [ -t 0 ] will always be false. But when being invoked from the terminal, one shouldn't sleep randomly. It doesn't make sense. The invocation time itself is random already, we're not coming from a cron ran periodic incantation.
Comment 1 Alan Somers freebsd_committer freebsd_triage 2017-02-13 05:16:28 UTC
Looks like this behavior was added back in 2000 by revision 65843, probably unintentionally.
Comment 2 commit-hook freebsd_committer freebsd_triage 2017-04-01 04:44:47 UTC
A commit references this bug:

Author: asomers
Date: Sat Apr  1 04:42:35 UTC 2017
New revision: 316342
URL: https://svnweb.freebsd.org/changeset/base/316342

Log:
  Consolidate random sleeps in periodic scripts

  Multiple periodic scripts sleep for a random amount of time in order to
  mitigate the thundering herd problem. This is bad, because the sum of
  multiple uniformly distributed random variables approaches a normal
  distribution, so the problem isn't mitigated as effectively as it would be
  with a single sleep.

  This change creates a single configurable anticongestion sleep. periodic
  will only sleep if at least one script requires it, and it will never sleep
  more than once per invocation. It also won't sleep if periodic was run
  interactively, fixing an unrelated longstanding bug.

  PR:		217055
  PR:		210188
  Reviewed by:	cy
  MFC after:	3 weeks
  Differential Revision:	https://reviews.freebsd.org/D10211

Changes:
  head/etc/defaults/periodic.conf
  head/etc/periodic/daily/480.leapfile-ntpd
  head/share/man/man5/periodic.conf.5
  head/usr.sbin/periodic/periodic.sh
Comment 3 commit-hook freebsd_committer freebsd_triage 2017-04-24 15:29:41 UTC
A commit references this bug:

Author: asomers
Date: Mon Apr 24 15:29:18 UTC 2017
New revision: 317373
URL: https://svnweb.freebsd.org/changeset/base/317373

Log:
  MFC r316342, r316358

  r316342:
  Consolidate random sleeps in periodic scripts

  Multiple periodic scripts sleep for a random amount of time in order to
  mitigate the thundering herd problem. This is bad, because the sum of
  multiple uniformly distributed random variables approaches a normal
  distribution, so the problem isn't mitigated as effectively as it would be
  with a single sleep.

  This change creates a single configurable anticongestion sleep. periodic
  will only sleep if at least one script requires it, and it will never sleep
  more than once per invocation. It also won't sleep if periodic was run
  interactively, fixing an unrelated longstanding bug.

  PR:		217055
  PR:		210188
  Reviewed by:	cy
  MFC after:	3 weeks
  Differential Revision:	https://reviews.freebsd.org/D10211

  r316358:
  Fix man page typo from r316342

  Reported by:	rgrimes
  MFC after:	20 days
  X-MFC-With:	316342

Changes:
_U  stable/11/
  stable/11/etc/defaults/periodic.conf
  stable/11/etc/periodic/daily/480.leapfile-ntpd
  stable/11/share/man/man5/periodic.conf.5
  stable/11/usr.sbin/periodic/periodic.sh