I get the following error message when rebooting a CURRENT system (up to date at the moment) with varying refcount: Khelp module "ertt" can't unload until its refcount drops from 14 to 0. I am using the default TCP stack and congestion control and the GENERIC configuration.
What is the output of `kldstat`?
kldstat on the running system produced this: Id Refs Address Size Name 1 65 0xffffffff80200000 1d4f878 kernel 2 1 0xffffffff81f51000 3815d0 vmm.ko 3 1 0xffffffff822d3000 36c0 coretemp.ko 4 1 0xffffffff822d7000 39d0 nmdm.ko 5 1 0xffffffff83800000 461ab0 zfs.ko 6 1 0xffffffff83610000 799d ip_mroute.ko 7 1 0xffffffff83618000 3370 acpi_wmi.ko 8 1 0xffffffff8361c000 4248 ichsmb.ko 9 1 0xffffffff83621000 2178 smbus.ko 10 1 0xffffffff83624000 5ee0 ig4.ko 11 1 0xffffffff8362a000 8770 if_bridge.ko 12 1 0xffffffff83633000 60d8 bridgestp.ko 13 1 0xffffffff8363a000 3360 uhid.ko 14 1 0xffffffff8363e000 33a0 usbhid.ko 15 1 0xffffffff83642000 32a8 hidbus.ko 16 1 0xffffffff83646000 3340 wmt.ko 17 1 0xffffffff8364a000 2a50 mac_ntpd.ko I then did a shutdown -r now, and got this on the console: Khelp module "ertt" can't unload until its refcount drops from 9 to 0. Sounds like ertt isn't a separate module here.
(In reply to Mike Karels from comment #2) OK. I don't see the message on my local test system. Are you using multiple jails or vnets? Is `sockstat -CSPtcp` reporting that all TCP end points use the `freebsd` stack and the `cubic` congestion control module?
No jails or vnets in use. All connections using freebsd and cubic. NFS client is in use (v4 and v3).
(In reply to Mike Karels from comment #4) Is NFS using TCP or UDP?
NFS is using TCP only. btw, IPv6 is configured as well as v4, but I don't see any connections using it, just listening sockets.
(In reply to Mike Karels from comment #6) Are you using the NFS mount for booting the OS or are you just mounting some data partitions? Do you see the message for some time or is this new?
The NFS mounts are for /home, /usr/src and /usr/obj; not for booting. This message has been printed at shutdown time for some time; I just got around to writing a bug. (And I am often not watching the shutdown/reboot console messages.)
I saw the same error on an up-to-date CURRENT today, running under bhyve with no NFS or ZFS and no TCP tuning. Syncing disks, vnodes remaining... 3 0 0 done All buffers synced. Uptime: 2h29m5s Khelp module "ertt" can't unload until its refcount drops from 2 to 0. The operating system has halted.
This bug still exists in -CURRENT as of today, and I saw it in one of the last two 14.0 RCs.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=415c1c748d5492e41328fedf96b6bf3c9be94595 commit 415c1c748d5492e41328fedf96b6bf3c9be94595 Author: Mike Karels <karels@FreeBSD.org> AuthorDate: 2023-11-17 18:51:18 +0000 Commit: Mike Karels <karels@FreeBSD.org> CommitDate: 2023-11-17 18:51:18 +0000 khelp: suppress useless warning message on shutdown If a module (e.g. the ertt hhook for TCP) can't clean up at shutdown, there is nothing to be done about it. In the ertt case, cleanup is just shutting down a UMA zone, which doesn't need to be done. Suppress EBUSY warnings on shutdown. PR: 271677 Reviewed by: tuexen, imp Differential Revision: https://reviews.freebsd.org/D42650 sys/kern/kern_khelp.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3569e21f07147c8dd4bee82e5e7efe69c98edeb1 commit 3569e21f07147c8dd4bee82e5e7efe69c98edeb1 Author: Mike Karels <karels@FreeBSD.org> AuthorDate: 2023-11-17 18:51:18 +0000 Commit: Mike Karels <karels@FreeBSD.org> CommitDate: 2023-11-24 18:11:16 +0000 khelp: suppress useless warning message on shutdown If a module (e.g. the ertt hhook for TCP) can't clean up at shutdown, there is nothing to be done about it. In the ertt case, cleanup is just shutting down a UMA zone, which doesn't need to be done. Suppress EBUSY warnings on shutdown. PR: 271677 Reviewed by: tuexen, imp Differential Revision: https://reviews.freebsd.org/D42650 (cherry picked from commit 415c1c748d5492e41328fedf96b6bf3c9be94595) sys/kern/kern_khelp.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Fixed on main and stable/14.
I am running tests with automated installations in VirtualBox (https://github.com/simula/nornet-vmimage-builder-scripts). The issue appears with both, 14.0-RELEASE and 14.0-STABLE (20240229, this is the latest ISO at the moment). Used ISOs: * https://download.freebsd.org/releases/ISO-IMAGES/14.0/FreeBSD-14.0-RELEASE-amd64-disc1.iso * https://download.freebsd.org/snapshots/amd64/amd64/ISO-IMAGES/14.0/FreeBSD-14.0-STABLE-amd64-20240229-d19769300126-266911-disc1.iso The only difference is that the STABLE version just hangs instead of shutting down, while the RELEASE version shows the message "Khelp module "ertt" can't unload until its refcount drops ..." before hanging. I am going to attach screenshots of both variants.
Created attachment 248918 [details] Screenshot of the issue with 14.0-RELEASE
Created attachment 248919 [details] Screenshot of the issue with 14.0-STABLE (20240229)
Could it be that when using Virtual Box the VM just does not shutdown? So this might be unrelated to the ertt issue?
No, this is a different bug (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277450). I have to set sysctl hw.efi.poweroff=0, so that the machine actually should power off as intended. The ertt module unload issue appears always, after a freshly installed machine (including successful "freebsd-update fetch install", "pkg upgrade", and setting hw.efi.poweroff=0 in /etc/sysctl.conf) has been rebooted, then configured via SSH, and finally instructed to shut down. However, the ertt issue module issue seems to not occur later when manually stopping and restarting this machine. On the other hand, the hw.efi.poweroff issue (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277450) can always be reproduced on this machine.
(In reply to Thomas Dreibholz from comment #18) I'm confused: Assuming you set hw.efi.poweroff=0 in /boot/loader.conf, not /etc/sysctl.conf, are you observing any problem other than an error message showing up?
No, it is a sysctl: $ sysctl hw.efi.poweroff hw.efi.poweroff: 0 The default is 1. See also: https://forums.freebsd.org/threads/efi-virtualbox-computer-non-stop-after-successful-shutdown-of-freebsd.84856/ .
(In reply to Thomas Dreibholz from comment #20) I think it is a loader tunable. At least it is listed when you use `sysctl -T`. My question: Do you experience unexpected behavior when you use hw.efi.poweroff=0? If yes, which?
No, with setting hw.efi.poweroff=0, an EFI machine usually shuts down as expected. The exception is the ertt module unload failure. With the default of hw.efi.poweroff=1, an EFI machine never shuts down. I can provide a VirtualBox OVA file, if this is useful.
(In reply to Thomas Dreibholz from comment #22) So I think everything you are complaining about is covered in bug 277450. So why reporting here?
No, the machine does not power off in case of the ertt module unload failure, even with hw.efi.poweroff=0. The patch in STABLE seems to just suppress the ertt error message, but it reproducibly fails to shut off the machine at the same place where it also fails for RELEASE.
I strongly suspect that the ertt issue is unrelated to the failure to power down. The message is printed (in the release) after all related processing is done, so the system must be getting through the ertt code.