Bug 282281 - zfs: panic: VERIFY(avl_is_empty(&sk->sk_dsl_keys)) failed
Summary: zfs: panic: VERIFY(avl_is_empty(&sk->sk_dsl_keys)) failed
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 15.0-CURRENT
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-fs (Nobody)
URL:
Keywords: crash
Depends on:
Blocks:
 
Reported: 2024-10-23 06:06 UTC by Lexi Winter
Modified: 2024-11-02 11:58 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Lexi Winter freebsd_triage 2024-10-23 06:06:56 UTC
observed while shutting down.

host: FreeBSD 15.0-CURRENT #3 lf/main-n269068-2cff93ced1d: Wed Oct 23 02:48:20 BST 2024     srcmastr@hemlock.eden.le-fay.org:/data/build/obj/freebsd/data/build/src/freebsd/lf/main/amd64.amd64/sys/GENERIC

root@hemlock:~ # shutdown -r now
Shutdown NOW!
shutdown: [pid 74216]
root@hemlStopping jails:.
Stopping bird.
Waiting for PIDS: 26888Oct 23 06:44:39 hemlock bird[26888]: Shutdown completed
.
Stopping cron.
Waiting for PIDS: 19791.
Stopping inetd.
Waiting for PIDS: 28413.
Stopping smartctl_exporter.
Waiting for PIDS: 40315.
Stopping rsyncd.
Waiting for PIDS: 35838.
Stopping node_exporter.
Waiting for PIDS: 34016.
Stopping nginx.
Waiting for PIDS: 30716.
Stopping nfsd.
Waiting for PIDS: 19020 19719.
Stopping smbd.
Waiting for PIDS: 16334.
Stopping mountd.
Waiting for PIDS: 7148.
Stopping nscd.
Waiting for PIDS: 89764.
/etc/rc.shutdown: WARNING: $buildbot_enable is not set properly - see rc.conf(5).
Stopping openssh.
Waiting for PIDS: 2821, 2821.
eval: utx: not found
Stopping powerd.
Waiting for PIDS: 99443.
Stopping ntpd.
Waiting for PIDS: 98249.
Stopping unbound.
Waiting for PIDS: 42838.
Beginning shutdown process
  * parallel shutdown of non-ordered guests: system.grp system.grp system.pwd system.pwd
  * stopping lab3
  * waiting 2 second(s)
  * stopping lab2
  * waiting 2 second(s)
  * stopping lab1
  * waiting 2 second(s)
  * stopping ivy
  * waiting 2 second(s)
  * stopping lily
  * waiting 2 second(s)
  * stopping iris
  * waiting 2 second(s)
  * stopping media

Waiting for PIDS: 29703 97832 1242 4323
90 second watchdOct 23 06:46:10 hemlock init[1]: /etc/rc.shutdown terminated abnormally, going to single user mode
Oct 23 06:46:10 hemlock syslogd: exiting on signal 15
vmnet8: link state changed to DOWN
vmnet7: link state changed to DOWN
Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining... 0 0 0 0 0 0 0 0 0 0 done
All buffers synced.
Uptime: 19m16s
panic: VERIFY(avl_is_empty(&sk->sk_dsl_keys)) failed

cpuid = 0
time = 1729662389
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0104810ac0
vpanic() at vpanic+0x13f/frame 0xfffffe0104810bf0
spl_panic() at spl_panic+0x3a/frame 0xfffffe0104810c50
spa_keystore_fini() at spa_keystore_fini+0xe4/frame 0xfffffe0104810c90
spa_deactivate() at spa_deactivate+0x388/frame 0xfffffe0104810ce0
spa_evict_all() at spa_evict_all+0xc8/frame 0xfffffe0104810d00
spa_fini() at spa_fini+0x18/frame 0xfffffe0104810d40
zfs_kmod_fini() at zfs_kmod_fini+0x7e/frame 0xfffffe0104810d60
zfs_shutdown() at zfs_shutdown+0x2b/frame 0xfffffe0104810d70
kern_reboot() at kern_reboot+0x4f3/frame 0xfffffe0104810db0
sys_reboot() at sys_reboot+0x396/frame 0xfffffe0104810e00
amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe0104810f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0104810f30
--- syscall (55, FreeBSD ELF64, reboot), rip = 0x2884ba, rsp = 0x820811528, rbp = 0x820811610 ---
KDB: enter: panic
[ thread pid 1 tid 100002 ]
Stopped at      kdb_enter+0x33: movq    $0,0x1054b02(%rip)
Comment 1 Cy Schubert freebsd_committer freebsd_triage 2024-10-23 15:02:20 UTC
Did you you zpool upgrade your pools or simply upgrade without zpool upgrade? (Not saying one  is correct and the other not, just gathering information.)
Comment 2 Lexi Winter freebsd_triage 2024-10-25 05:22:08 UTC
the pool hasn't been upgraded yet, "longname" and "large_microzap" are not enabled.
Comment 3 Mark Johnston freebsd_committer freebsd_triage 2024-11-01 14:17:15 UTC
Has it occurred just the one time?  Presumably you have some encrypted datasets?
Comment 4 Lexi Winter freebsd_triage 2024-11-02 11:58:01 UTC
when i encountered the problem, it was 100% reproducible on reboot (i rebooted several times for an unrelated issue).

however, after shutting down the affected just now, the problem didn't occur.

yes, i do have some encrypted datasets.

i will do some more testing tonight and see if i can get the issue to trigger again.