Virtualbox is causing a kernel panic during VM start. I'm using a recent CURRENT: $ uname -a FreeBSD capeta 12.0-CURRENT FreeBSD 12.0-CURRENT #10 r337378M: Mon Aug 6 11:06:26 -03 2018 danilo@capeta:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG amd64 I'm not using the binary module, I built it from ports. The system panics immediately and doesn't drop me to the debugger. The message is: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x28 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80c7e806 stack pointer = 0x28:0xfffffe009c0b84c0 frame pointer = 0x28:0xfffffe009c0b84f0 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 3505 (VBoxNetAdpCtl) I've built the kernel module with and without VIMAGE to test, same problem.
Created attachment 196017 [details] SMAP support (POC) Try attached patch. But... it does not support !SMAP case
(In reply to Vladimir Kondratyev from comment #1) My bad. In SMAP case fault code should be "supervisor read data, protection violation" not "page not present" and current process should be a VirtualBox. False alarm. Sorry.
Actually, I got this message as well: vboxnet0: Ethernet address: 0a:00:27:00:00:00 vboxdrv: XXXXXXXXXXXXXXXX VMMR0.r0 vboxdrv: XXXXXXXXXXXXXXXX VBoxDDR0.r0 vboxnet0: promiscuous mode enabled Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 02 fault virtual address = 0x804209ab8 fault code = supervisor read data, protection violation instruction pointer = 0x20:0xffffffff835cc164 stack pointer = 0x28:0xfffffe00b33031f0 frame pointer = 0x28:0xfffffe00b33031f0 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 21985 (VBoxHeadless)
Created attachment 196026 [details] Vbox SMAP support Now it should support both SMAP and !SMAP cases
(In reply to Danilo Egea Gondolfo from comment #3) If the panic is caused by SMAP you should see VBoxDrvFreeBSDIOCtl() somewhere in backtrace.
All the ioctl handlers should be patched? As I'm seeing a panic caused by VBoxNetAdpCtl, maybe this one should be patched as well. (VBoxNetAdpFreeBSDCtrlioctl) virtualbox-ose-kmod/work/VirtualBox-5.2.16/src/VBox/HostDrivers/VBoxNetAdp/freebsd/VBoxNetAdp-freebsd.c Does it make sense?
I've started vbox inside a qemu VM, it doesn't booted the guest OS but the system didn't crash. vbox initiated the network (created the interface) at least. On my system it crashes immediately after start. Could this be an evidence that both the panics I'm seeing are issues with SMAP?
Created attachment 196033 [details] panic
Ok, now I got the stack trace (I needed to boot without the intel drm drivers). Probably nothing related with SMAP...
(In reply to Danilo Egea Gondolfo from comment #9) You can do several useful things: 1. Describe your setup. 2. Bisect kernel commits to find the culprit. 3. Find the code line which caused the panic. Install gdb from ports, than run it with kernel path as parameter # gdb /boot/kernel/kernel And than type (gdb) info line *(void *)(if_alloc+0xC6)
FreeBSD BSD_12 12.0-ALPHA1 FreeBSD 12.0-ALPHA1 #74 r337592: Fri Aug 10 18:11:16 EDT 2018 root@BSD_12:/usr/obj/usr/src/amd64.amd64/sys/MYKERNEL amd64 1200077 Unfortunately, with the patch applied I am still getting crash/reboot ---<<BOOT>>--- Copyright (c) 1992-2018 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-ALPHA1 #74 r337592: Fri Aug 10 18:11:16 EDT 2018 root@BSD_12:/usr/obj/usr/src/amd64.amd64/sys/MYKERNEL amd64 FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1) VT(vga): resolution 640x480 CPU: AMD Ryzen 5 1600X Six-Core Processor (3600.09-MHz K8-class CPU) Origin="AuthenticAMD" Id=0x800f11 Family=0x17 Model=0x1 Stepping=1 Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> Features2=0x7ed8320b<SSE3,PCLMULQDQ,MON,SSSE3,FMA,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND> AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM> AMD Features2=0x35c233ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,SKINIT,WDT,TCE,Topology,PCXC,PNXC,DBE,PL2I,MWAITX> Structured Extended Features=0x209c01a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA> XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES> AMD Extended Feature Extensions ID EBX=0x7<CLZERO,IRPerf,XSaveErPtr> SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=32768 TSC: P-state invariant, performance statistics
(In reply to andy from comment #11) > Unfortunately, with the patch applied I am still getting crash/reboot Note: Although emulators/virtualbox-ose is patched, emulators/virtualbox-ose-kmod is to be rebuild. To check if your panic is SMAP-relaed you can disable it. Just revert the patch, rebuild emulators/virtualbox-ose-kmod, add "hw.cpu_stdext_disable=0x00100000" to the /boot/loader.conf (or set it at the loader prompt) and reboot.
I can confirm that with SMAP disabled (hw.cpu_stdext_disable=0x00100000 /boot/loader.conf) unpatched Virtualbox works fine.
Yeah, I just tested it here. With SMAP disabled it works, no panics.
Hello, I've just added myself to Cc: list cause I have the same issue and it's reproducible. Also, I'd like to confirm that with SMAP disabled, i.e.(hw.cpu_stdext_disable="0x00100000" in /boot/loader.conf) fixes a kernel panic with VBoxSDL.
CC Konstantin on SMAP issues If base changes are required, please create a seperate blocking issue
Why vbox changes are needed at all ? In other words, why this code accesses userspace directly ? This is wrong, it should use copyin(9)/copyout(9) KPI.
(In reply to Konstantin Belousov from comment #17) Should the system still panic / be panic'able via this use of code?
(In reply to Kubilay Kocak from comment #18) Although the patch fixed VBox for me, it appeared to be not 100% correct. VBox tries to utilize both methods of SMAP control (CR4 and EFLAGS) depending on preprocessor flags and it looks like EFLAGS is default way now. I will rework the patch to use EFLAGS and will post it here.
(In reply to Vladimir Kondratyev from comment #19) Before you further tweaks the patch, can you please explain two things to me: 1. Why vbox needs to access userspace 2. If the item 1 is legitimate, why doesn't vbox use copyin(9) KPI.
(In reply to Konstantin Belousov from comment #20) > 1. Why vbox needs to access userspace > 2. If the item 1 is legitimate, why doesn't vbox use copyin(9) KPI. I don't know. I can only guess it is to reduce amount of system-dependent code. Really, vbox kmod is very strange thing. After a few hours of digging I got an impression that it is not a kernel part of VBox but is a loader that loads into kernel memory and initializes RING0 parts of emulator on each? VM start.
(In reply to Konstantin Belousov from comment #20) > 1. Why vbox needs to access userspace > 2. If the item 1 is legitimate, why doesn't vbox use copyin(9) KPI. <CITE> For historic reasons, VirutalBox kernel code needs to access user-level code because the complete virtual memory of the guest is mapped on the host side and on 32-bit hosts we have the 1G limitation in the kernel part of the address space (3G for userland, 1G for kernel, kernel starts at 0xC0000000). And we don't want to limit guests on 32-bit hosts to 1GB RAM. <\CITE> https://www.virtualbox.org/ticket/13820
For infor, loader does not set hw.cpu_stdext_disable=0x00100000 on ALPHA2. It sets this fine on ALPHA1. I had to compile/install loader code from ALPHA1 in order to disable SMAP, so I can continue using Virtualbox.
Created attachment 196484 [details] Vbox SMAP support (EFLAGS) I have been running attached port of Darwin's SMAP support for about 1 week w/o any issues. I am not sure if it supports some edge cases like VboxNetFlt netgraph nodes.
(In reply to Vladimir Kondratyev from comment #24) This fails to build for me. I can email you the logs if you like.
Created attachment 196560 [details] Vbox SMAP support (EFLAGS) Some unrelated changes leaked into previous diff. (Hopefully) Fixed.
(In reply to Vladimir Kondratyev from comment #26) Yep, builds fine now.
(In reply to Steve Wills from comment #27) And I can confirm the updated patch fixes the issue for me. Previous versions did not.
I also had panics with VirtualBox and this patch fixed it for me.
Not sure if this is related, but sometimes my Arch Linux VM hangs and in dmesg there are lines vboxdrv: XXXXXXXXXXXXXXXX VMMR0.r0 vboxdrv: XXXXXXXXXXXXXXXX VBoxDDR0.r0
(In reply to Gleb Popov from comment #30) > Not sure if this is related, but sometimes my Arch Linux VM hangs and in dmesg > there are lines > vboxdrv: XXXXXXXXXXXXXXXX VMMR0.r0 > vboxdrv: XXXXXXXXXXXXXXXX VBoxDDR0.r0 These messages are normal. Your VM is hanging for another reason. (Make sure to use host I/O Cache)
I'll take it.
A commit references this bug: Author: jkim Date: Sat Oct 20 04:42:56 UTC 2018 New revision: 482464 URL: https://svnweb.freebsd.org/changeset/ports/482464 Log: Support SMAP for amd64. This should stop kernel panics on SMAP supported CPUs after r336876. PR: 230460 MFH: 2018Q4 Changes: head/emulators/virtualbox-ose/files/patch-src-VBox-HostDrivers-VBoxNetFlt-freebsd-VBoxNetFlt-freebsd.c head/emulators/virtualbox-ose/files/patch-src-VBox-HostDrivers-VBoxNetFlt-freebsd-files_vboxnetflt head/emulators/virtualbox-ose/files/patch-src_VBox_HostDrivers_Support_SUPDrv.cpp head/emulators/virtualbox-ose/files/patch-src_VBox_HostDrivers_Support_freebsd_SUPDrv-freebsd.c head/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_alloc-r0drv-freebsd.c head/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_assert-r0drv-freebsd.c head/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_memobj-r0drv-freebsd.c head/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_memuserkernel-r0drv-freebsd.c head/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_mp-r0drv-freebsd.c head/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_semevent-r0drv-freebsd.c head/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_semeventmulti-r0drv-freebsd.c head/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_semfastmutex-r0drv-freebsd.c head/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_semmutex-r0drv-freebsd.c head/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_spinlock-r0drv-freebsd.c head/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_the-freebsd-kernel.h head/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_thread-r0drv-freebsd.c head/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_thread2-r0drv-freebsd.c head/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_timer-r0drv-freebsd.c head/emulators/virtualbox-ose-additions/Makefile head/emulators/virtualbox-ose-kmod/Makefile
Committed with some changes. Thanks!
A commit references this bug: Author: jkim Date: Tue Oct 23 19:41:52 UTC 2018 New revision: 482871 URL: https://svnweb.freebsd.org/changeset/ports/482871 Log: MFH: r482464 Support SMAP for amd64. This should stop kernel panics on SMAP supported CPUs after r336876. PR: 230460 Approved by: ports-secteam (miwi) Changes: _U branches/2018Q4/ branches/2018Q4/emulators/virtualbox-ose/files/patch-src-VBox-HostDrivers-VBoxNetFlt-freebsd-VBoxNetFlt-freebsd.c branches/2018Q4/emulators/virtualbox-ose/files/patch-src-VBox-HostDrivers-VBoxNetFlt-freebsd-files_vboxnetflt branches/2018Q4/emulators/virtualbox-ose/files/patch-src_VBox_HostDrivers_Support_SUPDrv.cpp branches/2018Q4/emulators/virtualbox-ose/files/patch-src_VBox_HostDrivers_Support_freebsd_SUPDrv-freebsd.c branches/2018Q4/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_alloc-r0drv-freebsd.c branches/2018Q4/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_assert-r0drv-freebsd.c branches/2018Q4/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_memobj-r0drv-freebsd.c branches/2018Q4/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_memuserkernel-r0drv-freebsd.c branches/2018Q4/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_mp-r0drv-freebsd.c branches/2018Q4/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_semevent-r0drv-freebsd.c branches/2018Q4/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_semeventmulti-r0drv-freebsd.c branches/2018Q4/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_semfastmutex-r0drv-freebsd.c branches/2018Q4/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_semmutex-r0drv-freebsd.c branches/2018Q4/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_spinlock-r0drv-freebsd.c branches/2018Q4/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_the-freebsd-kernel.h branches/2018Q4/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_thread-r0drv-freebsd.c branches/2018Q4/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_thread2-r0drv-freebsd.c branches/2018Q4/emulators/virtualbox-ose/files/patch-src_VBox_Runtime_r0drv_freebsd_timer-r0drv-freebsd.c branches/2018Q4/emulators/virtualbox-ose-additions/Makefile branches/2018Q4/emulators/virtualbox-ose-kmod/Makefile
I'm still seeing kernel panics despite the patch r482464 with a recent CURRENT & Virtualbox (built from source): # uname -a FreeBSD Server 12.0-BETA4 FreeBSD 12.0-BETA4 #1 r328635M: Mon Nov 12 10:58:05 CET 2018 root@Server:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 # pkg info -f virtualbox-ose\* | egrep "^Name|^Installed" Name : virtualbox-ose Installed on : Thu Nov 15 11:21:39 2018 UTC Name : virtualbox-ose-kmod Installed on : Thu Nov 15 10:50:59 2018 UTC r482464 is installed # ll patch-src-VBox-HostDrivers-VBoxNetFlt-freebsd-files_vboxnetflt -rw-r--r-- 1 root wheel 638B Oct 20 04:42 patch-src-VBox-HostDrivers-VBoxNetFlt-freebsd-files_vboxnetflt The panic cause seems to match this bug: Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 02 fault virtual address = 0x28 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80c94496 stack pointer = 0x28:0xfffffe005a3766c0 frame pointer = 0x28:0xfffffe005a3766f0 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 1222 (VBoxNetAdpCtl) trap number = 12 panic: page fault cpuid = 1 time = 1542282364 KDB: stack backtrace: #0 0xffffffff80be6fa7 at kdb_backtrace+0x67 #1 0xffffffff80b9ab93 at vpanic+0x1a3 #2 0xffffffff80b9a9e3 at panic+0x43 #3 0xffffffff8107388f at trap_fatal+0x35f #4 0xffffffff810738e9 at trap_pfault+0x49 #5 0xffffffff81072f0e at trap+0x29e #6 0xffffffff8104e335 at calltrap+0x8 #7 0xffffffff82cdc02a at vboxNetAdpOsCreate+0x1a #8 0xffffffff82cdc64c at vboxNetAdpCreate+0xcc #9 0xffffffff82cdc526 at VBoxNetAdpFreeBSDCtrlioctl+0x86 #10 0xffffffff80a5167d at devfs_ioctl+0xad #11 0xffffffff811f828e at VOP_IOCTL_APV+0x7e #12 0xffffffff80c79514 at vn_ioctl+0x1a4 #13 0xffffffff80a51c5f at devfs_ioctl_f+0x1f #14 0xffffffff80c0456d at kern_ioctl+0x26d #15 0xffffffff80c0428e at sys_ioctl+0x15e #16 0xffffffff81074369 at amd64_syscall+0x369 #17 0xffffffff8104ec1d at fast_syscall_common+0x101 Any ideas?
Exactly same crash on 12.0-RELEASE (on Intel E6550 CPU) But the bug status is marked as "Closed FIXED"
The crash occurs when I try to start VM on: FreeBSD 12.0-RELEASE r341666 GENERIC amd64 CPU: Intel E5-2623 v4 CPU microcode: updated from 0xb00002a to 0xb00002e virtualbox-ose-nox11-5.2.22_2 (installed from PORTS) The same system worked fine with FreeBSD 11.2-RELEASE and virtualbox-ose-nox11-5.2.12_1
I guess this might be related to bug #220003. The VM starts with NAT adapter and immediately crashes the host with bridged adapter. setting hw.cpu_stdext_disable="0x00100000" in /boot/loader.conf has no effect (I'm not sure if loader pays attention to it).
I found a solution - compiling emulators/virtualbox-ose-kmod with VIMAGE support on solved the problem. The rebuild of all ports inherited the old value - VIMAGE off. I see it was turned on by default on 24 Oct 2017. I guess if this option is mandatory (at least for the bridged networking) there should be some scary warning during ports build
I can confirm that turning on VIMAGE support in virtualbox-ose-kmod "solves" the problem.
I am using the i386 12.0-RELEASE with a GENERIC kernel. I have the same problem. Virtualbox causes kernel panic immediately when a virtual machine starts. I have installed virtualbox-ose-kmod 5.2.22_1 from port and turned on VIMAGE support. However, those workarounds cannot stop the kernel panic.
Enabling VIMAGE did the trick for me too, thanks.
Regarding comment 42, the core dump file is attached here. Is this SMAP related? FreeBSD mars.myhome.westell.com 12.0-RELEASE-p1 FreeBSD 12.0-RELEASE-p1 r342385 GENERIC i386 panic: page fault GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"...BFD: /boot/kernel/kernel: invalid relocation type 42 BFD: /boot/kernel/kernel: invalid relocation type 42 BFD: /boot/kernel/kernel: invalid relocation type 42 Unread portion of the kernel message buffer: <6>pid 2913 (VirtualBox) is attempting to use unsafe AIO requests - not logging anymore Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 01 fault virtual address = 0x34ccf560 fault code = supervisor read, page not present instruction pointer = 0x20:0x1f8b6529 stack pointer = 0x28:0x145cd7ac frame pointer = 0x28:0x145cd7c4 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 2913 (VirtualBox) trap number = 12 panic: page fault cpuid = 1 time = 1548853347 KDB: stack backtrace: #0 0x110854f at kdb_backtrace+0x4f #1 0x10bb517 at vpanic+0x147 #2 0x10bb3cb at panic+0x1b #3 0x1691955 at trap_fatal+0x395 #4 0x1691993 at trap_pfault+0x33 #5 0x1690fdf at trap+0x3cf #6 0xffc0315d at PTDpde+0x4165 #7 0x1f887d87 at _DYNAMIC+0x2279d87 #8 0x1f8b0620 at _DYNAMIC+0x22a2620 #9 0x1f8aecbb at _DYNAMIC+0x22a0cbb #10 0x1f986391 at _DYNAMIC+0x2378391 #11 0x23e8754 at supdrvIOCtlInnerUnrestricted+0x8c4 #12 0x23e7c40 at supdrvIOCtl+0xb0 #13 0x23f70a4 at VBoxDrvFreeBSDIOCtl+0xe4 #14 0xf72641 at devfs_ioctl+0xb1 #15 0x16c546d at VOP_IOCTL_APV+0x6d #16 0x1198894 at vn_ioctl+0x164 #17 0xf72bcd at devfs_ioctl_f+0x3d Uptime: 24m24s Physical memory: 3024 MB
Thanks. Enabling VIMAGE fixed this for me. Surely it deserves a mention in ports/UPDATING and pkg-message.
Actually, now that I think about it a bit more... Why does it have to cause a panic? Surely it should just return an error and refuse to start the VM.