# kgdb kernel.debug /var/crash/vmcore.0
(kgdb) list *0xffffffff821f542b
0xffffffff821f542b is in vmx_disable (cpufunc.h:423).
418     }
419
420     static __inline void
421     load_cr4(u_long data)
422     {
423             __asm __volatile("movq %0,%%cr4" : : "r" (data));
424     }
425
426     static __inline u_long
427     rcr4(void)
Current language:  auto; currently minimal
(kgdb) backtrace
#0  doadump (textdump=<value optimized out>) at pcpu.h:219
#1  0xffffffff8095a5a2 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:451
#2  0xffffffff8095a985 in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:758
#3  0xffffffff8095a813 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:687
#4  0xffffffff80d98b1b in trap_fatal (frame=<value optimized out>, eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:851
#5  0xffffffff80d9879c in trap (frame=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:203
#6  0xffffffff80d7e832 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:236
#7  0xffffffff821f542b in vmx_disable (arg=0x0) at /usr/src/sys/modules/vmm/../../amd64/vmm/intel/vmx.c:481
#8  0xffffffff809a372c in smp_rendezvous_action () at /usr/src/sys/kern/subr_smp.c:439
#9  0xffffffff80d7fc49 in Xrendezvous () at apic_vector.S:295
#10 0xffffffff80d7be56 in acpi_cpu_c1 () at /usr/src/sys/amd64/acpica/acpi_machdep.c:95
#11 0xffffffff8037de1a in acpi_cpu_idle (sbt=<value optimized out>) at /usr/src/sys/dev/acpica/acpi_cpu.c:1038
#12 0xffffffff80d8238f in cpu_idle_acpi (sbt=83757098) at /usr/src/sys/amd64/amd64/machdep.c:682
#13 0xffffffff80d82430 in cpu_idle (busy=0) at /usr/src/sys/amd64/amd64/machdep.c:828
#14 0xffffffff809872e5 in sched_idletd (dummy=<value optimized out>) at /usr/src/sys/kern/sched_ule.c:2662
#15 0xffffffff809243aa in fork_exit (callout=0xffffffff80987110 <sched_idletd>, arg=0x0, frame=0xfffffe082cdb6ac0) at /usr/src/sys/kern/kern_fork.c:1018
#16 0xffffffff80d7ed6e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:611
#17 0x0000000000000000 in ?? ()
(kgdb)

I neglected to mention this machine is using VirtualBox; it hosts several VMs (here all VMs are VirtualBox's VMs). 

I did following tests:

0) fresh boot without any VirtualBox kernel modules
1) fresh boot with VirtualBox modules loaded but no VM running
2) same as 1) but with at least one VM running

During kldunload kernel crashed only when VM was running. Any other time unload was successful. Also if I started the VM, suspended it and then removed the vmm everything was ok. 

Crash occurs in the same function (step) as mentioned above. I was able to reproduce it every time when VM was running.

Tested on different HW - same behavior.

Comment 4 Jason Unovitch 2015-10-31 15:21:25 UTC

Forums cross reference:  https://forums.FreeBSD.org/threads/kernel-panic-when-trying-to-unload-vmm.53697/

Comment 5 John Baldwin 2015-12-05 14:03:20 UTC

In general I think VM monitors assume that no other monitors are running.  I believe OS X has a kernel-level API for monitors to use to try to mitigate this, but FreeBSD does not.  For example, I fixed bhyve VMs to work across suspend and resume (of a host laptop), but that fix is specific to bhyve and does not support other VM monitors.

In general VM monitors like bhyve assume that they "own" all of the VT-x (or SVM) state.  They assume they are not sharing it.  Just loading vmm.ko will _set_ various CPU control registers (MSRs) related to VT-x (VT-x includes a host of optional features that can be enabled selectively) which might confuse some other VMM that had set these controls to different values.  (For bhyve see the sys/amd64/vmm/vmx/vmx.c vmx_init() routine run by vmm_init() in sys/amd64/vmm/vmm.c on module load.)

In summary, it is not safe to even load multiple VMMs at the same time, much less run VMs from different VMMs concurrently.  If FreeBSD does grow an API to support VM monitors the first iteration of it will probably fail attempts to load more than one VMM for this reason.

(In reply to John Baldwin from comment #5)
Yeah, it does make sense that only one monitor should be running. I went trough it went I was trying to figure out where the problem was, or at least what it was causing. 

Not that I understand what it does exactly, but I got the idea at least. 

For others who may experience the same issue I created a thread on forums and here I created a PR, maybe it catches somebody's attention.

FreeBSD 10.2-RELEASE-p14 #13 r296979

I have had the same issue running kldunload vmm crashed my server and I'm running virtualbox as well.

Comment 8 John Baldwin 2016-04-04 17:19:12 UTC

To be clear, yes, this is not a supported configuration (having both vmm.ko and virtualbox's module loaded at the same time, much less unloading one while the other is running).  I don't expect there to be a "fix" for this anytime soon, but if there is one it will consist of making vmm.ko fail to load if virtual box is loaded and vice versa.

What about loading virtualboxes into bhyve kernels? I haven't tried it but would that be acceptable?

From my point of view it was important to know why it crashed. Deeper investigation showed where and why. That was important to me. 

Kernel level API like on OS X would be appreciated feature .. but I get that priority is somewhere else.

Comment 11 John Baldwin 2016-04-04 22:53:27 UTC

(In reply to nospam from comment #9)
Do you mean kldloading virtual box in an instance of FreeBSD running in a bhyve VM?  I think that will just not work as bhyve doesn't let the guest see any of the VT-x capabilities of the CPU since it doesn't support nested VT-x.  The reverse might work if VirtualBox supports nested VT-x (I haven't tried).

No I've tried the reverse because when I wanted to play with bhyve, I installed virtualbox to do so, but that didn't work at all.