observed while shutting down. host: FreeBSD 15.0-CURRENT #3 lf/main-n269068-2cff93ced1d: Wed Oct 23 02:48:20 BST 2024 srcmastr@hemlock.eden.le-fay.org:/data/build/obj/freebsd/data/build/src/freebsd/lf/main/amd64.amd64/sys/GENERIC root@hemlock:~ # shutdown -r now Shutdown NOW! shutdown: [pid 74216] root@hemlStopping jails:. Stopping bird. Waiting for PIDS: 26888Oct 23 06:44:39 hemlock bird[26888]: Shutdown completed . Stopping cron. Waiting for PIDS: 19791. Stopping inetd. Waiting for PIDS: 28413. Stopping smartctl_exporter. Waiting for PIDS: 40315. Stopping rsyncd. Waiting for PIDS: 35838. Stopping node_exporter. Waiting for PIDS: 34016. Stopping nginx. Waiting for PIDS: 30716. Stopping nfsd. Waiting for PIDS: 19020 19719. Stopping smbd. Waiting for PIDS: 16334. Stopping mountd. Waiting for PIDS: 7148. Stopping nscd. Waiting for PIDS: 89764. /etc/rc.shutdown: WARNING: $buildbot_enable is not set properly - see rc.conf(5). Stopping openssh. Waiting for PIDS: 2821, 2821. eval: utx: not found Stopping powerd. Waiting for PIDS: 99443. Stopping ntpd. Waiting for PIDS: 98249. Stopping unbound. Waiting for PIDS: 42838. Beginning shutdown process * parallel shutdown of non-ordered guests: system.grp system.grp system.pwd system.pwd * stopping lab3 * waiting 2 second(s) * stopping lab2 * waiting 2 second(s) * stopping lab1 * waiting 2 second(s) * stopping ivy * waiting 2 second(s) * stopping lily * waiting 2 second(s) * stopping iris * waiting 2 second(s) * stopping media Waiting for PIDS: 29703 97832 1242 4323 90 second watchdOct 23 06:46:10 hemlock init[1]: /etc/rc.shutdown terminated abnormally, going to single user mode Oct 23 06:46:10 hemlock syslogd: exiting on signal 15 vmnet8: link state changed to DOWN vmnet7: link state changed to DOWN Waiting (max 60 seconds) for system process `vnlru' to stop... done Waiting (max 60 seconds) for system process `syncer' to stop... Syncing disks, vnodes remaining... 0 0 0 0 0 0 0 0 0 0 done All buffers synced. Uptime: 19m16s panic: VERIFY(avl_is_empty(&sk->sk_dsl_keys)) failed cpuid = 0 time = 1729662389 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0104810ac0 vpanic() at vpanic+0x13f/frame 0xfffffe0104810bf0 spl_panic() at spl_panic+0x3a/frame 0xfffffe0104810c50 spa_keystore_fini() at spa_keystore_fini+0xe4/frame 0xfffffe0104810c90 spa_deactivate() at spa_deactivate+0x388/frame 0xfffffe0104810ce0 spa_evict_all() at spa_evict_all+0xc8/frame 0xfffffe0104810d00 spa_fini() at spa_fini+0x18/frame 0xfffffe0104810d40 zfs_kmod_fini() at zfs_kmod_fini+0x7e/frame 0xfffffe0104810d60 zfs_shutdown() at zfs_shutdown+0x2b/frame 0xfffffe0104810d70 kern_reboot() at kern_reboot+0x4f3/frame 0xfffffe0104810db0 sys_reboot() at sys_reboot+0x396/frame 0xfffffe0104810e00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe0104810f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0104810f30 --- syscall (55, FreeBSD ELF64, reboot), rip = 0x2884ba, rsp = 0x820811528, rbp = 0x820811610 --- KDB: enter: panic [ thread pid 1 tid 100002 ] Stopped at kdb_enter+0x33: movq $0,0x1054b02(%rip)
Did you you zpool upgrade your pools or simply upgrade without zpool upgrade? (Not saying one is correct and the other not, just gathering information.)
the pool hasn't been upgraded yet, "longname" and "large_microzap" are not enabled.
Has it occurred just the one time? Presumably you have some encrypted datasets?
when i encountered the problem, it was 100% reproducible on reboot (i rebooted several times for an unrelated issue). however, after shutting down the affected just now, the problem didn't occur. yes, i do have some encrypted datasets. i will do some more testing tonight and see if i can get the issue to trigger again.