Bug 241180

Summary: emulators/open-vm-tools: kldload vmblock panics
Product: Ports & Packages Reporter: Josh Paetzel <jpaetzel>
Component: Individual Port(s)Assignee: Josh Paetzel <jpaetzel>
Status: Closed FIXED    
Severity: Affects Only Me CC: dim, mp, vbhakta, w.schwarzenfeld
Priority: ---    
Version: Latest   
Hardware: Any   
OS: Any   

Description Josh Paetzel freebsd_committer freebsd_triage 2019-10-10 13:20:46 UTC
@jpaetzel: so loading vmblock.ko now results in a panic: https://paste.debian.net/plain/1105651

last working kernel was r352764, now it's at r353358. Anything changed recently in locking primitives, or modules support?

panic: lock 0xffffffff82533890 is not initialized
cpuid = 3
time = 1570691595
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe004c2ac490
vpanic() at vpanic+0x17e/frame 0xfffffe004c2ac4f0
panic() at panic+0x43/frame 0xfffffe004c2ac550
lock_destroy() at lock_destroy+0x33/frame 0xfffffe004c2ac570
VMBlockUninit() at VMBlockUninit+0x10/frame 0xfffffe004c2ac580
vfs_modevent() at vfs_modevent+0x41c/frame 0xfffffe004c2ac5b0
module_register_init() at module_register_init+0xbd/frame 0xfffffe004c2ac5e0
linker_load_module() at linker_load_module+0xbf9/frame 0xfffffe004c2ac900
kern_kldload() at kern_kldload+0xef/frame 0xfffffe004c2ac950
sys_kldload() at sys_kldload+0x5b/frame 0xfffffe004c2ac980
amd64_syscall() at amd64_syscall+0x2b5/frame 0xfffffe004c2acab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe004c2acab0
--- syscall (304, FreeBSD ELF64, sys_kldload), rip = 0x8002d6d6a, rsp = 0x7fffffffe2c8, rbp = 0x7fffffffe840 ---
KDB: enter: panic
[ thread pid 1431 tid 100157 ]
Stopped at      kdb_enter+0x37: movq    $0,0x1099bf6(%rip)
Comment 1 Walter Schwarzenfeld freebsd_triage 2019-10-10 13:32:35 UTC
I don't think this is a ports bug (base - kernel ?)
Comment 2 Josh Paetzel freebsd_committer freebsd_triage 2019-10-10 14:10:10 UTC
We shall see.  I default to "out of tree driver needs to chase kernel changes" until proven otherwise.  Either way I'm on it, whatever the actual problem is.
Comment 3 Vishal Bhakta 2019-10-11 02:01:35 UTC
I think this is a result of r353150 which added a new entry to vfsops, leading to a mismatch between vmblock and the kernel.

vfs_init is calling vmblock's uninit.

(kgdb) bt
#0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1  doadump (textdump=0) at /usr/src/sys/kern/kern_shutdown.c:392
#2  0xffffffff804948fa in db_dump (dummy=<optimized out>, dummy2=<unavailable>, dummy3=<unavailable>, dummy4=<unavailable>)
    at /usr/src/sys/ddb/db_command.c:575
#3  0xffffffff804946bc in db_command (last_cmdp=<optimized out>, cmd_table=<optimized out>, dopager=1) at /usr/src/sys/ddb/db_command.c:482
#4  0xffffffff8049442d in db_command_loop () at /usr/src/sys/ddb/db_command.c:535
#5  0xffffffff80497658 in db_trap (type=<optimized out>, code=<optimized out>) at /usr/src/sys/ddb/db_main.c:252
#6  0xffffffff80c09d07 in kdb_trap (type=3, code=0, tf=<optimized out>) at /usr/src/sys/kern/subr_kdb.c:692
#7  0xffffffff8102cad7 in trap (frame=0xfffffe00ac4ec3c0) at /usr/src/sys/amd64/amd64/trap.c:585
#8  <signal handler called>
#9  kdb_enter (why=0xffffffff811da3a3 "panic", msg=<optimized out>) at /usr/src/sys/kern/subr_kdb.c:479
#10 0xffffffff80bbe8ba in vpanic (fmt=<optimized out>, ap=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:897
#11 0xffffffff80bbe653 in panic (fmt=0xffffffff81c85938 <cnputs_mtx> "\257\363\031\201\377\377\377\377")
    at /usr/src/sys/kern/kern_shutdown.c:835
#12 0xffffffff80c0aa33 in lock_destroy (lock=0xffffffff825218e0 <VMBlockPathnameZone>) at /usr/src/sys/kern/subr_lock.c:104
#13 0xffffffff82520d60 in VMBlockUninit () from /usr/local/lib/vmware-tools/modules/drivers/vmblock.ko
#14 0xfffffe00ac4ec5b0 in ?? ()
#15 0xffffffff80c888fc in vfs_register (vfc=0xffffffff825219d8 <vmblock_vfsconf+32>) at /usr/src/sys/kern/vfs_init.c:497
#16 vfs_modevent (mod=<optimized out>, type=<optimized out>, data=0xffffffff825219d8 <vmblock_vfsconf+32>) at /usr/src/sys/kern/vfs_init.c:582
Backtrace stopped: frame did not save the PC

(kgdb) f 15
#15 0xffffffff80c888fc in vfs_register (vfc=0xffffffff825219d8 <vmblock_vfsconf+32>) at /usr/src/sys/kern/vfs_init.c:497
497                     vfc->vfc_vfsops_sd->vfs_init(vfc);

(kgdb) p *vfc->vfc_vfsops
$5 = {vfs_mount = 0xffffffff8251f960 <VMBlockVFSMount>, vfs_cmount = 0x0, vfs_unmount = 0xffffffff8251fc10 <VMBlockVFSUnmount>,
  vfs_root = 0xffffffff8251fd60 <VMBlockVFSRoot>, vfs_cachedroot = 0x0, vfs_quotactl = 0xffffffff8251fdc0 <VMBlockVFSStatFS+16>,
  vfs_statfs = 0xffffffff8251fea0 <VMBlockVFSVGet+16>, vfs_sync = 0xffffffff8251feb0 <VMBlockVFSVGet+32>,
  vfs_vget = 0xffffffff80c85b50 <vfs_stdvget>, vfs_fhtovp = 0xffffffff80c85b60 <vfs_stdfhtovp>,
  vfs_checkexp = 0xffffffff82520cd0 <VMBlockInit+32>, vfs_init = 0xffffffff82520d50 <VMBlockUninit+32>,
  vfs_uninit = 0xffffffff80c85b80 <vfs_stduninit>, vfs_extattrctl = 0xffffffff80c85b90 <vfs_stdextattrctl>,
  vfs_sysctl = 0xffffffff80c85bd0 <vfs_stdsysctl>, vfs_susp_clean = 0x0, vfs_reclaim_lowervp = 0x0, vfs_unlink_lowervp = 0x0,
  vfs_purge = 0x0, vfs_spare = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}
Comment 4 Mark Peek freebsd_committer freebsd_triage 2019-11-12 15:27:39 UTC
This looks like a binary incompatibility issue where the kernel structure has changed due to the addition of vfs_cachedroot in r353150 (per Vishal's note).
https://github.com/freebsd/freebsd/blob/master/sys/sys/mount.h#L708

@jpaetzel, were you using a vmblock.ko binary? Likely recompiling open-vm-tools should fix this issue. Do you know how to indicate in ports how to bump to indicate this breaking change?
Comment 5 commit-hook freebsd_committer freebsd_triage 2019-11-18 17:36:39 UTC
A commit references this bug:

Author: jpaetzel
Date: Mon Nov 18 17:36:37 UTC 2019
New revision: 517880
URL: https://svnweb.freebsd.org/changeset/ports/517880

Log:
  Bump PORTREVISION to rebuild pkg in HEAD

  For users tracking stable/12, stable/11, or supported releases
  this will result in an upgrade of the port
  with no functional change.

  For users tracking HEAD before r353150 the new binary pkg
  will likely cause your system to panic.

  PR:	241180
  Sponsored by:	Panzura
  Fix suggested by:	mp

Changes:
  head/emulators/open-vm-tools/Makefile