bhyve guest: FreeBSD 15.0-CURRENT #0 lf/main-n269061-b4479cf2b3b9: Wed Sep 18 20:43:02 BST 2024 srcmastr@daphne.eden.le-fay.org:/src/obj/src/freebsd/lf/main/amd64.amd64/sys/LFV6 bhyve host: FreeBSD 15.0-CURRENT #43 lf/main-n269060-ffd494371643: Tue Sep 17 05:59:59 BST 2024 srcmastr@daphne.eden.le-fay.org:/src/obj/src/freebsd/lf/main/amd64.amd64/sys/LF 'kldloat virtio_pci' causes a kernel panic when vtnet is loaded. stack trace (i have a vmcore if needed): root@lily:~ # kldload virtio_pci virtio_pci0: <VirtIO PCI (legacy) Network adapter> port 0x2000-0x203f mem 0xc0000000-0xc0001fff irq 18 at device 5.0 on pci0 root@lily:~ # vtnet0: <VirtIO Networking Adapter> on virtio_pci0 Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 01 fault virtual address = 0x28 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff8051deb8 stack pointer = 0x28:0xfffffe004b379810 frame pointer = 0x28:0xfffffe004b379850 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 701 (devctl) rdi: fffff80001021300 rsi: fffff80001021300 rdx: 0000000000000001 rcx: 0000000000000001 r8: 0000000000000010 r9: fffffe0001e0a040 rax: 0000000000000000 rbx: fffff80001f02000 rbp: fffffe004b379850 r10: fffff800011b9800 r11: 9e99ff93939e0000 r12: fffff8000186a800 r13: fffff800018c0b30 r14: ffffffff8070e492 r15: 0000000000000000 trap number = 12 panic: page fault cpuid = 1 time = 1729195870 KDB: stack backtrace: #0 0xffffffff8045ed4d at kdb_backtrace+0x5d #1 0xffffffff80414bdf at vpanic+0x13f #2 0xffffffff80414a93 at panic+0x43 #3 0xffffffff8069d39b at trap_fatal+0x40b #4 0xffffffff8069d3e6 at trap_pfault+0x46 #5 0xffffffff806747d8 at calltrap+0x8 #6 0xffffffff8051cd8e at if_attach_internal+0x4e #7 0xffffffff80527f6c at ether_ifattach+0x2c #8 0xffffffff8181d6d1 at vtnet_setup_interface+0x391 #9 0xffffffff8181cb53 at vtnet_attach+0x1a63 #10 0xffffffff8044e96c at device_attach+0x3ac #11 0xffffffff81812497 at vtpci_legacy_probe_and_attach_child+0x77 #12 0xffffffff8044c029 at devclass_driver_added+0x29 #13 0xffffffff8045453b at device_do_deferred_actions+0x3b #14 0xffffffff80453e3f at devctl2_ioctl+0x20f #15 0xffffffff803811ab at devfs_ioctl+0xcb #16 0xffffffff804fb30e at vn_ioctl+0xce #17 0xffffffff8038180e at devfs_ioctl_f+0x1e Uptime: 14s Dumping 113 out of 1002 MB:..15%..29%..43%..57%..71%..85%..99% Dump complete Automatic reboot in 15 seconds - press a key on the console to abort
forgot to mention, this works fine if virtio_pci.ko and if_vtnet.ko are loaded prior to boot in loader.conf. it's only kldload that triggers the problem.
What is in your kernel config?
Created attachment 254318 [details] kernel config
Created attachment 254319 [details] kernel config
kernel config attached, note it's two files since "LFV6" includes "LF". "LFV6" is the running kernel on this system.
This is due to a VNET bug. When a network driver attaches, it needs to have the current VNET set, e.g., because if_attach_internal()->if_addgroup() accesses the VNET-global variable V_ifg_head. Normally this happens in device_probe_and_attach(), which is how drivers get attached if they're compiled into the kernel, but we're taking a different path here. I posted a patch here which fixes the problem for me: https://reviews.freebsd.org/D47174
thanks, i'm in the process of doing a src+ports build now but once that's done i'll test this patch and see if it fixes the panic for me.
i can't build world with the patch from D47174 on top of fa573868f187956b384722a90392866769f4965a. --- subr_bus.o --- /data/build/src/freebsd/lf/main/sys/kern/subr_bus.c:2577:26: error: incomplete definition of type 'struct prison' 2577 | KASSERT(IS_DEFAULT_VNET(TD_TO_VNET(curthread)), | ~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~ 2578 | ("device_attach: curthread is not in default vnet")); | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /data/build/src/freebsd/lf/main/sys/net/vnet.h:248:25: note: expanded from macro 'TD_TO_VNET' 248 | #define TD_TO_VNET(td) CRED_TO_VNET((td)->td_ucred) | ^ /data/build/src/freebsd/lf/main/sys/net/vnet.h:247:41: note: expanded from macro 'CRED_TO_VNET' 247 | #define CRED_TO_VNET(cr) (cr)->cr_prison->pr_vnet | ^ /data/build/src/freebsd/lf/main/sys/net/vnet.h:245:32: note: expanded from macro 'IS_DEFAULT_VNET' 245 | #define IS_DEFAULT_VNET(arg) ((arg) == vnet0) | ^ /data/build/src/freebsd/lf/main/sys/sys/kassert.h:144:24: note: expanded from macro 'KASSERT' 144 | if (__predict_false(!(exp))) \ | ~~~~~~~~~~~~~~~~~~^~~~~ /data/build/src/freebsd/lf/main/sys/sys/cdefs.h:315:51: note: expanded from macro '__predict_false' 315 | #define __predict_false(exp) __builtin_expect((exp), 0) | ^~~ /data/build/src/freebsd/lf/main/sys/sys/cpuset.h:152:8: note: forward declaration of 'struct prison' 152 | struct prison; | ^ /data/build/src/freebsd/lf/main/sys/kern/subr_bus.c:2579:20: error: incomplete definition of type 'struct prison' 2579 | CURVNET_SET_QUIET(TD_TO_VNET(curthread)); | ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~ /data/build/src/freebsd/lf/main/sys/net/vnet.h:248:25: note: expanded from macro 'TD_TO_VNET' 248 | #define TD_TO_VNET(td) CRED_TO_VNET((td)->td_ucred) | ^ /data/build/src/freebsd/lf/main/sys/net/vnet.h:247:41: note: expanded from macro 'CRED_TO_VNET' 247 | #define CRED_TO_VNET(cr) (cr)->cr_prison->pr_vnet | ^ /data/build/src/freebsd/lf/main/sys/net/vnet.h:221:15: note: expanded from macro 'CURVNET_SET_QUIET' 221 | VNET_ASSERT((arg) != NULL && (arg)->vnet_magic_n == VNET_MAGIC_N, \ | ~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 222 | ("CURVNET_SET at %s:%d %s() curvnet=%p vnet=%p", \ | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 223 | __FILE__, __LINE__, __func__, curvnet, (arg))); \ | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /data/build/src/freebsd/lf/main/sys/net/vnet.h:184:8: note: expanded from macro 'VNET_ASSERT' 184 | if (!(exp)) \ | ^~~ /data/build/src/freebsd/lf/main/sys/sys/cpuset.h:152:8: note: forward declaration of 'struct prison' 152 | struct prison; | ^ /data/build/src/freebsd/lf/main/sys/kern/subr_bus.c:2579:20: error: incomplete definition of type 'struct prison' 2579 | CURVNET_SET_QUIET(TD_TO_VNET(curthread)); | ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~ /data/build/src/freebsd/lf/main/sys/net/vnet.h:248:25: note: expanded from macro 'TD_TO_VNET' 248 | #define TD_TO_VNET(td) CRED_TO_VNET((td)->td_ucred) | ^ /data/build/src/freebsd/lf/main/sys/net/vnet.h:247:41: note: expanded from macro 'CRED_TO_VNET' 247 | #define CRED_TO_VNET(cr) (cr)->cr_prison->pr_vnet | ^ /data/build/src/freebsd/lf/main/sys/net/vnet.h:221:32: note: expanded from macro 'CURVNET_SET_QUIET' 221 | VNET_ASSERT((arg) != NULL && (arg)->vnet_magic_n == VNET_MAGIC_N, \ | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 222 | ("CURVNET_SET at %s:%d %s() curvnet=%p vnet=%p", \ | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 223 | __FILE__, __LINE__, __func__, curvnet, (arg))); \ | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /data/build/src/freebsd/lf/main/sys/net/vnet.h:184:8: note: expanded from macro 'VNET_ASSERT' 184 | if (!(exp)) \ | ^~~ /data/build/src/freebsd/lf/main/sys/sys/cpuset.h:152:8: note: forward declaration of 'struct prison' 152 | struct prison; | ^ /data/build/src/freebsd/lf/main/sys/kern/subr_bus.c:2579:20: error: incomplete definition of type 'struct prison' 2579 | CURVNET_SET_QUIET(TD_TO_VNET(curthread)); | ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~ /data/build/src/freebsd/lf/main/sys/net/vnet.h:248:25: note: expanded from macro 'TD_TO_VNET' 248 | #define TD_TO_VNET(td) CRED_TO_VNET((td)->td_ucred) | ^ /data/build/src/freebsd/lf/main/sys/net/vnet.h:247:41: note: expanded from macro 'CRED_TO_VNET' 247 | #define CRED_TO_VNET(cr) (cr)->cr_prison->pr_vnet | ^ /data/build/src/freebsd/lf/main/sys/net/vnet.h:223:46: note: expanded from macro 'CURVNET_SET_QUIET' 221 | VNET_ASSERT((arg) != NULL && (arg)->vnet_magic_n == VNET_MAGIC_N, \ | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 222 | ("CURVNET_SET at %s:%d %s() curvnet=%p vnet=%p", \ | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 223 | __FILE__, __LINE__, __func__, curvnet, (arg))); \ | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~ /data/build/src/freebsd/lf/main/sys/net/vnet.h:185:9: note: expanded from macro 'VNET_ASSERT' 185 | panic msg; \ | ^~~ /data/build/src/freebsd/lf/main/sys/sys/cpuset.h:152:8: note: forward declaration of 'struct prison' 152 | struct prison; | ^ /data/build/src/freebsd/lf/main/sys/kern/subr_bus.c:2579:20: error: incomplete definition of type 'struct prison' 2579 | CURVNET_SET_QUIET(TD_TO_VNET(curthread)); | ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~ /data/build/src/freebsd/lf/main/sys/net/vnet.h:248:25: note: expanded from macro 'TD_TO_VNET' 248 | #define TD_TO_VNET(td) CRED_TO_VNET((td)->td_ucred) | ^ /data/build/src/freebsd/lf/main/sys/net/vnet.h:247:41: note: expanded from macro 'CRED_TO_VNET' 247 | #define CRED_TO_VNET(cr) (cr)->cr_prison->pr_vnet | ^ /data/build/src/freebsd/lf/main/sys/net/vnet.h:225:12: note: expanded from macro 'CURVNET_SET_QUIET' 225 | curvnet = arg; | ^~~ /data/build/src/freebsd/lf/main/sys/sys/cpuset.h:152:8: note: forward declaration of 'struct prison' 152 | struct prison; | ^ 5 errors generated. *** [subr_bus.o] Error code 1 make[2]: stopped making "all" in /data/build/obj/freebsd/data/build/src/freebsd/lf/main/amd64.amd64/sys/GENERIC
(In reply to Lexi Winter from comment #8) Oops. Fixed by the latest revision.
*** Bug 275381 has been marked as a duplicate of this bug. ***
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=f4e35c044c8988b7452cefbdcc417f5fd723e021 commit f4e35c044c8988b7452cefbdcc417f5fd723e021 Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2024-10-19 13:03:56 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2024-10-19 13:03:56 +0000 bus: Set the current VNET in device_attach() Some drivers, in particular anything which creates an ifnet during attach, need to have the current VNET set, as if_attach_internal() and its callees access VNET-global variables. device_probe_and_attach() handles this, but this is not the only way to arrive in DEVICE_ATTACH. In particular, bus drivers may invoke device_attach() directly, as does devctl2's DEV_ENABLE ioctl handler. So, set the current VNET in device_attach() instead. I believe it is always safe to use vnet0, as devctl2 ioctls are not permitted within a jail. PR: 282168 Reviewed by: zlei, kevans, bz, imp, glebius MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D47174 sys/kern/subr_bus.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)