Basically: kldstat | grep -c dtrace # returns "0", no kernel modules loaded yet kldload dtrace # ok kldload dtraceall # ok kldstat | grep -c dtrace # returns "2" - two matching kernel modules loaded - all ok. And here's the issue: kldunload dtrace # ok - shouldn't be ok. dtrace module is still loaded at this point but command returns 0 - success kldstat | grep -c dtrace # returns "2" again. kldunload dtrace # causes (expected) fail: kldunload: can't unload file: Device busy
It's more stuff like that after further digging: kldunload opensolaris # returns 0 (nothing happens) kldunload opensolaris # fails (zfs requirement so no go) kldunload kernel # returns 0 (nothing happens) kldunload kernel # returns: kldunload: can't unload file: Device bus
This is by design, though the behaviour is admittedly not intuitive. When you kldload dtrace, dtrace.ko gets a reference; loading dtraceall.ko adds a second reference to dtrace.ko. The first attempt to unload dtrace releases the initial reference, but dtraceall.ko's reference ensures that dtrace.ko remains loaded. The second attempt to unload fails because dtraceall.ko still depends on dtrace.ko. Unloading dtraceall.ko should cause dtrace.ko to be unloaded automatically in that case.
Perhaps we could make a note of this in the kldunload man page.