Since base r366857 (MFC to 12-STABLE: base r367049) dhclient gets killed uncleanly upon interface desctruction. Arguably this is just cosmetic, however this is new behaviour and should probably be tidied up. Stopping devd. Waiting for PIDS: 797. lo0: link state changed to DOWN Dec 2 15:30:29 dtk59 dhclient[785]: receive_packet failed on bge0.31: Device not configured Dec 2 15:30:29 dtk59 dhclient[785]: ioctl(SIOCGIFFLAGS) on bge0.31: Operation not permitted Dec 2 15:30:29 dtk59 dhclient[785]: Interface bge0.31 no longer appears valid. Dec 2 15:30:29 dtk59 dhclient[785]: No live interfaces to poll on - exiting. Dec 2 15:30:29 dtk59 dhclient[785]: exiting. Dec 2 15:30:29 dtk59 dhclient[785]: connection closed Dec 2 15:30:29 dtk59 dhclient[785]: exiting. /etc/rc.shutdown: WARNING: bge0.21 does not exist. Skipped. /etc/rc.shutdown: WARNING: bge0.31 does not exist. Skipped. /etc/rc.shutdown: WARNING: bge0.32 does not exist. Skipped. Stopping Network: lo0 bge0 em0 em1.
Created attachment 220173 [details] Try this I don't see these messages here. However my dhclient listens on lagg0 (bge0 failover to wlan0). Try this patch to see if it provides relief.
(In reply to Cy Schubert from comment #1) That didn't help, same messages still. Even worse, in addition there now is: Writing entropy file: . Writing early boot entropy file: . /etc/rc.shutdown: ERROR: /etc/rc.shutdown: no interface specified Terminated . In my case bge0 doesn't have an IP on its own, but it has 3 subinterfaces on 3 different VLANs, one of which (bge0.31) has dhclient listening and the other 2 are static.
That explains why I don't see the problem. Can you post your rc.conf, please.
(In reply to Cy Schubert from comment #3) The network related part: kld_list="if_bge if_em snd_hda ichsmb smb crypto" ifconfig_bge0="up" ifconfig_bge0_21="inet 172.16.126.9 netmask 255.255.255.0" ifconfig_bge0_31="DHCP fib 1" ifconfig_bge0_32="inet 192.168.0.7 netmask 255.255.255.0" vlans_bge0="21 31 32" defaultrouter="172.16.126.1" route_default_fib1="default 192.168.0.1 -fib 1" static_routes="default_fib1" Yes I have a tendency keeping things terse. IMHO it's related to base r366857 (MFC to 12-STABLE: base r367049). While the rationale behind reaping cloned interfaces during shutdown is certainly valid, it's happening too early apparently. It's done at a stage with listening processes still active, which might get confused if their interface is destroyed. dhclient is just one example. Likely there are others. Maybe they won't bark as verbose as dhclient, but then their unclean shutdown has the potential of triggering undesired side effects.
Created attachment 220201 [details] Revert and create a new script. This will run netif stop at the end of shutdown.
Created attachment 220203 [details] Tested and works. This should fix the problem.
Hmmm... doesn't change a bit for me: Dec 3 08:11:29 dtk59 kernel: lo0: link state changed to DOWN Dec 3 08:11:29 dtk59 dhclient[772]: receive_packet failed on bge0.31: Device not configured Dec 3 08:11:29 dtk59 dhclient[772]: ioctl(SIOCGIFFLAGS) on bge0.31: Operation not permitted Dec 3 08:11:29 dtk59 dhclient[772]: Interface bge0.31 no longer appears valid. Dec 3 08:11:29 dtk59 dhclient[772]: No live interfaces to poll on - exiting. Dec 3 08:11:29 dtk59 dhclient[772]: exiting. Dec 3 08:11:29 dtk59 dhclient[772]: connection closed Dec 3 08:11:29 dtk59 dhclient[772]: exiting. Dec 3 08:11:30 dtk59 root[1764]: /etc/rc.shutdown: WARNING: bge0.21 does not exist. Skipped. Dec 3 08:11:30 dtk59 root[1766]: /etc/rc.shutdown: WARNING: bge0.31 does not exist. Skipped. Dec 3 08:11:30 dtk59 root[1768]: /etc/rc.shutdown: WARNING: bge0.32 does not exist. Skipped. Dec 3 08:11:30 dtk59 syslogd: exiting on signal 15 I rebooted twice and double checked netif and netifshutdown but doesn't seem to work. rcorder -pkshutdown /etc/rc.d/* /etc/rc.d/swap /etc/rc.d/mountcritlocal /etc/rc.d/gssd /etc/rc.d/zfs /etc/rc.d/mixer /etc/rc.d/ugidfw /etc/rc.d/random /etc/rc.d/addswap /etc/rc.d/ipfs /etc/rc.d/rtsold /etc/rc.d/devd /etc/rc.d/zfsd /etc/rc.d/local_unbound /etc/rc.d/netifdown /etc/rc.d/kdc /etc/rc.d/nfsuserd /etc/rc.d/kfd /etc/rc.d/ipropd_slave /etc/rc.d/hostapd /etc/rc.d/nfscbd /etc/rc.d/kpasswdd /etc/rc.d/kadmind /etc/rc.d/ipropd_master /etc/rc.d/hastd /etc/rc.d/localpkg /etc/rc.d/watchdogd /etc/rc.d/bsnmpd /etc/rc.d/auditd /etc/rc.d/auditdistd /etc/rc.d/rpcbind /etc/rc.d/nfsclient /etc/rc.d/ypserv /etc/rc.d/ypbind /etc/rc.d/ypupdated /etc/rc.d/ypxfrd /etc/rc.d/ypldap /etc/rc.d/ypset /etc/rc.d/amd /etc/rc.d/yppasswdd /etc/rc.d/keyserv /etc/rc.d/automount /etc/rc.d/mountd /etc/rc.d/nfsd /etc/rc.d/statd /etc/rc.d/lockd /etc/rc.d/timed /etc/rc.d/rwho /etc/rc.d/rtadvd /etc/rc.d/hcsecd /etc/rc.d/utx /etc/rc.d/powerd /etc/rc.d/ntpd /etc/rc.d/nscd /etc/rc.d/lpd /etc/rc.d/sdpd /etc/rc.d/local /etc/rc.d/ubthidhci /etc/rc.d/ftp-proxy /etc/rc.d/moused /etc/rc.d/bthidd /etc/rc.d/swaplate /etc/rc.d/ftpd /etc/rc.d/jail /etc/rc.d/cron /etc/rc.d/sshd /etc/rc.d/inetd
Is this perhaps related to 12-STABLE only? My machine is on base r368270.
I ran the same rcorder without the patch applied. The rcorder impact of the patch is: /etc/rc.d/mixer /etc/rc.d/ugidfw /etc/rc.d/random /etc/rc.d/addswap /etc/rc.d/ipfs -/etc/rc.d/netif /etc/rc.d/rtsold /etc/rc.d/devd /etc/rc.d/zfsd -/etc/rc.d/local_unbound +/etc/rc.d/local_unbound /etc/rc.d/netifdown /etc/rc.d/kdc /etc/rc.d/nfsuserd /etc/rc.d/kfd /etc/rc.d/ipropd_slave /etc/rc.d/hostapd /etc/rc.d/nfscbd /etc/rc.d/kpasswdd /etc/rc.d/kadmind /etc/rc.d/ipropd_master /etc/rc.d/hastd /etc/rc.d/localpkg /etc/rc.d/watchdogd /etc/rc.d/bsnmpd /etc/rc.d/auditd So it looks like the netif stuff is just moved a few steps earlier. It doesn't look to me like any of the rc scripts within the window changed above would bring down dhclient orderly, or am I mistaken?
Created attachment 220219 [details] Be a little more aggressive. First, rc.shutdown doesn't use the -p flag and there is no way to specify that in rc.conf. I'm not sure why you are in your tests. Anyhow, it doesn't matter, people can still change rc.shutdown themselves. This patch takes this into account.
Created attachment 220220 [details] That was wrong. The original patch was correct. I didn't catch your mistake. To test run the below. Remember, rc.shutdown reverses the order it receives from rc.shutdown. And, rc.shutdown doesn't use the -p flag (unless you've customized your rc and rc.shutdown). slippy$ rcorder -kshutdown /etc/rc.d/* /usr/local/etc/rc.d/* | tail -r /usr/local/etc/rc.d/fetchmail /usr/local/etc/rc.d/smartd /usr/local/etc/rc.d/apache24 /usr/local/etc/rc.d/cbsd-statsd-bhyve /usr/local/etc/rc.d/cbsd-statsd-hoster /usr/local/etc/rc.d/cbsd-statsd-jail /usr/local/etc/rc.d/ezjail /usr/local/etc/rc.d/postgresql /usr/local/etc/rc.d/dma_flushq /usr/local/etc/rc.d/postfix /usr/local/etc/rc.d/zabbix_proxy /usr/local/etc/rc.d/zabbix_server /usr/local/etc/rc.d/cbsdd /usr/local/etc/rc.d/nagios /usr/local/etc/rc.d/squid /usr/local/etc/rc.d/nginx /usr/local/etc/rc.d/nrpe3 /usr/local/etc/rc.d/nsd /usr/local/etc/rc.d/ntimed /usr/local/etc/rc.d/vnstat /usr/local/etc/rc.d/volmand /usr/local/etc/rc.d/virtlogd /usr/local/etc/rc.d/cbsdrsyncd /usr/local/etc/rc.d/phd /usr/local/etc/rc.d/php-fpm /usr/local/etc/rc.d/postfwd /usr/local/etc/rc.d/spamass-milter /etc/rc.d/sshd /usr/local/etc/rc.d/cf-execd /usr/local/etc/rc.d/qjail.bootime /usr/local/etc/rc.d/socat /usr/local/etc/rc.d/rplayd /usr/local/etc/rc.d/cf-monitord /usr/local/etc/rc.d/rsyncd /usr/local/etc/rc.d/saned /usr/local/etc/rc.d/cf-serverd /usr/local/etc/rc.d/vboxwebsrv /usr/local/etc/rc.d/dovecot /usr/local/etc/rc.d/sa-spamd /usr/local/etc/rc.d/darkstat /usr/local/etc/rc.d/vboxwatchdog /usr/local/etc/rc.d/vboxheadless /etc/rc.d/ftpd /usr/local/etc/rc.d/firebird /usr/local/etc/rc.d/fossil /usr/local/etc/rc.d/swapmon /usr/local/etc/rc.d/mysql-server /usr/local/etc/rc.d/xdm /etc/rc.d/inetd /etc/rc.d/jail /usr/local/etc/rc.d/atop /usr/local/etc/rc.d/jail.jailconf.bootime /usr/local/etc/rc.d/jail.rcconf.bootime /usr/local/etc/rc.d/svscan /usr/local/etc/rc.d/jailrc /usr/local/etc/rc.d/jenkins /usr/local/etc/rc.d/sshguard /usr/local/etc/rc.d/kpropd /usr/local/etc/rc.d/libvirtd /usr/local/etc/rc.d/mailman /usr/local/etc/rc.d/mbmon /usr/local/etc/rc.d/bastille /usr/local/etc/rc.d/vpnc /usr/local/etc/rc.d/memcached /etc/rc.d/cron /usr/local/etc/rc.d/cups_browsed /usr/local/etc/rc.d/cupsd /usr/local/etc/rc.d/samba_server /usr/local/etc/rc.d/mrtg_daemon /etc/rc.d/swaplate /etc/rc.d/bthidd /usr/local/etc/rc.d/webcamd /etc/rc.d/ubthidhci /usr/local/etc/rc.d/diskcheckd /usr/local/etc/rc.d/fail2ban /usr/local/etc/rc.d/git_daemon /etc/rc.d/moused /usr/local/etc/rc.d/httpry /usr/local/etc/rc.d/innd /usr/local/etc/rc.d/isc-dhcrelay /usr/local/etc/rc.d/isc-dhcrelay6 /usr/local/etc/rc.d/mdnsd /usr/local/etc/rc.d/mdnsresponderposix /usr/local/etc/rc.d/openssh /usr/local/etc/rc.d/openvpn /usr/local/etc/rc.d/oss /usr/local/etc/rc.d/pfstatd /usr/local/etc/rc.d/poudriered /usr/local/etc/rc.d/rrdcached /usr/local/etc/rc.d/slpd /etc/rc.d/ftp-proxy /usr/local/etc/rc.d/imap4d /usr/local/etc/rc.d/comsatd /usr/local/etc/rc.d/swapd /usr/local/etc/rc.d/swapexd /usr/local/etc/rc.d/swatchdog /usr/local/etc/rc.d/uhidd /usr/local/etc/rc.d/watchd /usr/local/etc/rc.d/zabbix_agentd /usr/local/etc/rc.d/timed /usr/local/etc/rc.d/svnserve /usr/local/etc/rc.d/snort /usr/local/etc/rc.d/rinetd /usr/local/etc/rc.d/proftpd /usr/local/etc/rc.d/monitorix /usr/local/etc/rc.d/isc-dhcpd6 /usr/local/etc/rc.d/isc-dhcpd /usr/local/etc/rc.d/ipv6mon /usr/local/etc/rc.d/gkrellmd /usr/local/etc/rc.d/doinkd /usr/local/etc/rc.d/dhcp6s /usr/local/etc/rc.d/dhcp6relay /etc/rc.d/utx /etc/rc.d/rwho /etc/rc.d/rtadvd /etc/rc.d/sdpd /etc/rc.d/powerd /etc/rc.d/ntpd /etc/rc.d/nscd /etc/rc.d/lpd /etc/rc.d/local /etc/rc.d/hcsecd /usr/local/etc/rc.d/conserver /usr/local/etc/rc.d/saslauthd /usr/local/etc/rc.d/pop3d /usr/local/etc/rc.d/milter-opendkim /etc/rc.d/lockd /etc/rc.d/statd /etc/rc.d/nfsd /etc/rc.d/mountd /etc/rc.d/automount /etc/rc.d/keyserv /usr/local/etc/rc.d/automounter /etc/rc.d/yppasswdd /usr/local/etc/rc.d/amd /etc/rc.d/ypset /etc/rc.d/ypbind /etc/rc.d/ypldap /etc/rc.d/ypxfrd /etc/rc.d/ypupdated /usr/local/etc/rc.d/obspamlogd /etc/rc.d/ypserv /usr/local/etc/rc.d/radiusd /usr/local/etc/rc.d/stunnel /usr/local/etc/rc.d/monit /usr/local/etc/rc.d/stund /usr/local/etc/rc.d/obspamd /usr/local/etc/rc.d/stubby /usr/local/etc/rc.d/ipacctd /etc/rc.d/hastd /usr/local/etc/rc.d/named /usr/local/etc/rc.d/tcsd /usr/local/etc/rc.d/tpmd /usr/local/etc/rc.d/dnsmasq /usr/local/etc/rc.d/vm /etc/rc.d/nfsclient /etc/rc.d/rpcbind /etc/rc.d/auditdistd /usr/local/etc/rc.d/ipa /etc/rc.d/bsnmpd /usr/local/etc/rc.d/relayd /etc/rc.d/watchdogd /etc/rc.d/auditd /etc/rc.d/localpkg /usr/local/etc/rc.d/nut_upsmon /etc/rc.d/hostapd /etc/rc.d/nfscbd /etc/rc.d/kpasswdd /etc/rc.d/kadmind /usr/local/etc/rc.d/kiconv /usr/local/etc/rc.d/nut_upslog /etc/rc.d/ipropd_slave /etc/rc.d/ipropd_master /usr/local/etc/rc.d/tproxy /usr/local/etc/rc.d/tftpd /etc/rc.d/nfsuserd /usr/local/etc/rc.d/netdumpd /usr/local/etc/rc.d/kea /usr/local/etc/rc.d/ftpsesame /usr/local/etc/rc.d/distccd /etc/rc.d/kdc /etc/rc.d/kfd /usr/local/etc/rc.d/sndiod /usr/local/etc/rc.d/nut /etc/rc.d/netifdown /usr/local/etc/rc.d/unbound /etc/rc.d/local_unbound /etc/rc.d/netwait /etc/rc.d/zfsd /etc/rc.d/rtsold /etc/rc.d/devd /usr/local/etc/rc.d/slapd /usr/local/etc/rc.d/mpd5 /usr/local/etc/rc.d/dhcp6c /etc/rc.d/ipfs /etc/rc.d/addswap /etc/rc.d/mixer /etc/rc.d/ugidfw /etc/rc.d/random /usr/local/etc/rc.d/jaildaemon /etc/rc.d/gssd /etc/rc.d/mountcritlocal /etc/rc.d/swap slippy$ Anyhow, I've tested it here and it works. This fixes the problem jhb@ had. Your dhclient will still complain because vlans are not shut down gracefully when your parent interfaces are. This is a different problem that might requre an additional loop.
(In reply to Cy Schubert from comment #10) Sorry for confusing with the -p flag. It was just cosmetic sugar meant to avoid ambiguity in the sequence. The net observation is simply that /etc/rc.d/rtsold, /etc/rc.d/devd, /etc/rc.d/zfsd, and /etc/rc.d/local_unbound are the scripts that are run between /etc/rc.d/netif (previously) and /etc/rc.d/netifdown (after patch). The only exception is that /etc/rc.d/local_unbound may run before /etc/rc.d/netifdown *or not* as these two have the same "rank". The expected outcome of the patch would be that one of these four scripts or /etc/rc.d/netifdown would stop dhclient. But apparently none does. Also I think that none of the four scripts would ever touch anything related to a network interface, but maybe I'm mistaken. So I think your conclusion is correct: client interfaces should be shut gracefully before the parent but it seems they aren't - in contrast to the commit text of base r366857 (MFC to 12-STABLE: base r367049).
Created attachment 220222 [details] Only shut down cloned interfaces There were two other issues that jhb@ and emaste@ had identified with r366857. My thinking is now to only shut down cloned interfaces as John had initially suggested.
(In reply to Cy Schubert from comment #13) That fixes it! Silent shutdown as before base r366857. Please MFC as well.
A commit references this bug: Author: cy Date: Fri Dec 4 19:31:16 UTC 2020 New revision: 368345 URL: https://svnweb.freebsd.org/changeset/base/368345 Log: Revert r366857. r366857 created a number of problems, tearing down interfaces too early in shutdown. This resulted in: - hung ssh sessions when shutting down or rebooting remotely using shutdown (I've used exec shutdown, for years, as apposed to simply shutdown). - NFS mounted filesystems "disappear" prior to unmount. - dhclient attached to a VLAN on an interface who's parent interface has already shut down prints errors. The path forward is to teach lagg(4) and vlan(4) about WOL. PR: 251531, 251540 PR: 158734, 109980 are broken again Reported by: jhb, emaste, jtl, Helge Oldach<freebsd_oldach.net> Martin Birgmeier <d8zNeCFG_aon.at> MFC after: Immediately Discussion at: https://reviews.freebsd.org/D27459 Changes: head/libexec/rc/rc.d/netif
A commit references this bug: Author: cy Date: Fri Dec 4 19:35:43 UTC 2020 New revision: 368346 URL: https://svnweb.freebsd.org/changeset/base/368346 Log: Revert r366857. r366857 created a number of problems, tearing down interfaces too early in shutdown. This resulted in: - hung ssh sessions when shutting down or rebooting remotely using shutdown (I've used exec shutdown, for years, as apposed to simply shutdown). - NFS mounted filesystems "disappear" prior to unmount. - dhclient attached to a VLAN on an interface who's parent interface has already shut down prints errors. The path forward is to teach lagg(4) and vlan(4) about WOL. PR: 251531, 251540 PR: 158734, 109980 are broken again Reported by: jhb, emaste, jtl, Helge Oldach<freebsd_oldach.net> Martin Birgmeier <d8zNeCFG_aon.at> Discussion at: https://reviews.freebsd.org/D27459 Changes: _U stable/12/ stable/12/libexec/rc/rc.d/netif
Presumably this PR is resolved by the revert?
Correct. The revert resolves this issue. The PRs the reverted commit fixed are now broken again. Anyone enabling WOL on a lagg(4) member device will discover that WOL will no longer work. I am working on a kernel patch to add this functionality to the kernel.