@jpaetzel: so loading vmblock.ko now results in a panic: https://paste.debian.net/plain/1105651 last working kernel was r352764, now it's at r353358. Anything changed recently in locking primitives, or modules support? panic: lock 0xffffffff82533890 is not initialized cpuid = 3 time = 1570691595 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe004c2ac490 vpanic() at vpanic+0x17e/frame 0xfffffe004c2ac4f0 panic() at panic+0x43/frame 0xfffffe004c2ac550 lock_destroy() at lock_destroy+0x33/frame 0xfffffe004c2ac570 VMBlockUninit() at VMBlockUninit+0x10/frame 0xfffffe004c2ac580 vfs_modevent() at vfs_modevent+0x41c/frame 0xfffffe004c2ac5b0 module_register_init() at module_register_init+0xbd/frame 0xfffffe004c2ac5e0 linker_load_module() at linker_load_module+0xbf9/frame 0xfffffe004c2ac900 kern_kldload() at kern_kldload+0xef/frame 0xfffffe004c2ac950 sys_kldload() at sys_kldload+0x5b/frame 0xfffffe004c2ac980 amd64_syscall() at amd64_syscall+0x2b5/frame 0xfffffe004c2acab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe004c2acab0 --- syscall (304, FreeBSD ELF64, sys_kldload), rip = 0x8002d6d6a, rsp = 0x7fffffffe2c8, rbp = 0x7fffffffe840 --- KDB: enter: panic [ thread pid 1431 tid 100157 ] Stopped at kdb_enter+0x37: movq $0,0x1099bf6(%rip)
I don't think this is a ports bug (base - kernel ?)
We shall see. I default to "out of tree driver needs to chase kernel changes" until proven otherwise. Either way I'm on it, whatever the actual problem is.
I think this is a result of r353150 which added a new entry to vfsops, leading to a mismatch between vmblock and the kernel. vfs_init is calling vmblock's uninit. (kgdb) bt #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=0) at /usr/src/sys/kern/kern_shutdown.c:392 #2 0xffffffff804948fa in db_dump (dummy=<optimized out>, dummy2=<unavailable>, dummy3=<unavailable>, dummy4=<unavailable>) at /usr/src/sys/ddb/db_command.c:575 #3 0xffffffff804946bc in db_command (last_cmdp=<optimized out>, cmd_table=<optimized out>, dopager=1) at /usr/src/sys/ddb/db_command.c:482 #4 0xffffffff8049442d in db_command_loop () at /usr/src/sys/ddb/db_command.c:535 #5 0xffffffff80497658 in db_trap (type=<optimized out>, code=<optimized out>) at /usr/src/sys/ddb/db_main.c:252 #6 0xffffffff80c09d07 in kdb_trap (type=3, code=0, tf=<optimized out>) at /usr/src/sys/kern/subr_kdb.c:692 #7 0xffffffff8102cad7 in trap (frame=0xfffffe00ac4ec3c0) at /usr/src/sys/amd64/amd64/trap.c:585 #8 <signal handler called> #9 kdb_enter (why=0xffffffff811da3a3 "panic", msg=<optimized out>) at /usr/src/sys/kern/subr_kdb.c:479 #10 0xffffffff80bbe8ba in vpanic (fmt=<optimized out>, ap=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:897 #11 0xffffffff80bbe653 in panic (fmt=0xffffffff81c85938 <cnputs_mtx> "\257\363\031\201\377\377\377\377") at /usr/src/sys/kern/kern_shutdown.c:835 #12 0xffffffff80c0aa33 in lock_destroy (lock=0xffffffff825218e0 <VMBlockPathnameZone>) at /usr/src/sys/kern/subr_lock.c:104 #13 0xffffffff82520d60 in VMBlockUninit () from /usr/local/lib/vmware-tools/modules/drivers/vmblock.ko #14 0xfffffe00ac4ec5b0 in ?? () #15 0xffffffff80c888fc in vfs_register (vfc=0xffffffff825219d8 <vmblock_vfsconf+32>) at /usr/src/sys/kern/vfs_init.c:497 #16 vfs_modevent (mod=<optimized out>, type=<optimized out>, data=0xffffffff825219d8 <vmblock_vfsconf+32>) at /usr/src/sys/kern/vfs_init.c:582 Backtrace stopped: frame did not save the PC (kgdb) f 15 #15 0xffffffff80c888fc in vfs_register (vfc=0xffffffff825219d8 <vmblock_vfsconf+32>) at /usr/src/sys/kern/vfs_init.c:497 497 vfc->vfc_vfsops_sd->vfs_init(vfc); (kgdb) p *vfc->vfc_vfsops $5 = {vfs_mount = 0xffffffff8251f960 <VMBlockVFSMount>, vfs_cmount = 0x0, vfs_unmount = 0xffffffff8251fc10 <VMBlockVFSUnmount>, vfs_root = 0xffffffff8251fd60 <VMBlockVFSRoot>, vfs_cachedroot = 0x0, vfs_quotactl = 0xffffffff8251fdc0 <VMBlockVFSStatFS+16>, vfs_statfs = 0xffffffff8251fea0 <VMBlockVFSVGet+16>, vfs_sync = 0xffffffff8251feb0 <VMBlockVFSVGet+32>, vfs_vget = 0xffffffff80c85b50 <vfs_stdvget>, vfs_fhtovp = 0xffffffff80c85b60 <vfs_stdfhtovp>, vfs_checkexp = 0xffffffff82520cd0 <VMBlockInit+32>, vfs_init = 0xffffffff82520d50 <VMBlockUninit+32>, vfs_uninit = 0xffffffff80c85b80 <vfs_stduninit>, vfs_extattrctl = 0xffffffff80c85b90 <vfs_stdextattrctl>, vfs_sysctl = 0xffffffff80c85bd0 <vfs_stdsysctl>, vfs_susp_clean = 0x0, vfs_reclaim_lowervp = 0x0, vfs_unlink_lowervp = 0x0, vfs_purge = 0x0, vfs_spare = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}
This looks like a binary incompatibility issue where the kernel structure has changed due to the addition of vfs_cachedroot in r353150 (per Vishal's note). https://github.com/freebsd/freebsd/blob/master/sys/sys/mount.h#L708 @jpaetzel, were you using a vmblock.ko binary? Likely recompiling open-vm-tools should fix this issue. Do you know how to indicate in ports how to bump to indicate this breaking change?
A commit references this bug: Author: jpaetzel Date: Mon Nov 18 17:36:37 UTC 2019 New revision: 517880 URL: https://svnweb.freebsd.org/changeset/ports/517880 Log: Bump PORTREVISION to rebuild pkg in HEAD For users tracking stable/12, stable/11, or supported releases this will result in an upgrade of the port with no functional change. For users tracking HEAD before r353150 the new binary pkg will likely cause your system to panic. PR: 241180 Sponsored by: Panzura Fix suggested by: mp Changes: head/emulators/open-vm-tools/Makefile