This is initially observed while checking sysctl knobs (loader tunables) in modules. I managed to narrow it down. Tested with current/15 and 14.0-BETA5 on amd64 machine. Steps to repeat: ``` # kldload sdt # kldload virtio.ko ``` The kernel backtrace: ``` dumped core - see /var/crash/vmcore.3 Sun Oct 8 10:25:33 CST 2023 FreeBSD 14.0-BETA5 FreeBSD 14.0-BETA5 #1 releng/14.0-n265192-d67558ef3149: Sat Oct 7 18:03:25 CST 2023 zlei@:/usr/obj/home/zlei/freebsd-src-releng-14.0/amd64.amd64/sys/GENERIC amd64 panic: page fault GNU gdb (GDB) 13.2 [GDB v13.2 for FreeBSD] Copyright (C) 2023 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd14.0". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /boot/kernel/kernel... Reading symbols from /usr/lib/debug//boot/kernel.beta5/kernel.debug... interface virtio.1 already present in the KLD 'kernel'! Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 02 fault virtual address = 0x0 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff8321b134 stack pointer = 0x28:0xfffffe00b04e7920 frame pointer = 0x28:0xfffffe00b04e7950 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 97755 (kldload) rdi: fffff80178d6a6a0 rsi: fffffe00b04e7928 rdx: 0000000000000000 rcx: ffffffff88ff5470 r8: 0000000000000004 r9: 00000000000000f3 rax: 0000000000000000 rbx: fffffe00b04e7974 rbp: fffffe00b04e7950 r10: 0000000000000002 r11: ffffffffffffff90 r12: fffff80178bfd780 r13: fffffe00b04e7974 r14: fffff80178bfd7c0 r15: ffffffff88ff5468 trap number = 12 panic: page fault cpuid = 1 time = 1696731847 KDB: stack backtrace: #0 0xffffffff80b9017d at kdb_backtrace+0x5d #1 0xffffffff80b43282 at vpanic+0x132 #2 0xffffffff80b43143 at panic+0x43 #3 0xffffffff8100b81c at trap_fatal+0x40c #4 0xffffffff8100b86f at trap_pfault+0x4f #5 0xffffffff80fe2848 at calltrap+0x8 #6 0xffffffff80b0f61c at linker_file_unload+0xcc #7 0xffffffff810be9c8 at link_elf_load_file+0x198 #8 0xffffffff80b0ee93 at linker_load_module+0x9e3 #9 0xffffffff80b10b9a at kern_kldload+0x16a #10 0xffffffff80b10cbc at sys_kldload+0x5c #11 0xffffffff8100c0d9 at amd64_syscall+0x109 #12 0xffffffff80fe315b at fast_syscall_common+0xf8 Uptime: 16m43s Dumping 727 out of 8100 MB:..3%..11%..22%..31%..42%..51%..62%..71%..82%..91% __curthread () at /home/zlei/freebsd-src-releng-14.0/sys/amd64/include/pcpu_aux.h:57 57 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu, (kgdb) #0 __curthread () at /home/zlei/freebsd-src-releng-14.0/sys/amd64/include/pcpu_aux.h:57 #1 doadump (textdump=<optimized out>) at /home/zlei/freebsd-src-releng-14.0/sys/kern/kern_shutdown.c:405 #2 0xffffffff80b42e17 in kern_reboot (howto=260) at /home/zlei/freebsd-src-releng-14.0/sys/kern/kern_shutdown.c:526 #3 0xffffffff80b432ef in vpanic (fmt=0xffffffff81135a78 "%s", ap=ap@entry=0xfffffe00b04e7770) at /home/zlei/freebsd-src-releng-14.0/sys/kern/kern_shutdown.c:970 #4 0xffffffff80b43143 in panic (fmt=<unavailable>) at /home/zlei/freebsd-src-releng-14.0/sys/kern/kern_shutdown.c:894 #5 0xffffffff8100b81c in trap_fatal (frame=0xfffffe00b04e7860, eva=0) at /home/zlei/freebsd-src-releng-14.0/sys/amd64/amd64/trap.c:952 #6 0xffffffff8100b86f in trap_pfault (frame=0xfffffe00b04e7860, usermode=false, signo=<optimized out>, ucode=<optimized out>) at /home/zlei/freebsd-src-releng-14.0/sys/amd64/amd64/trap.c:760 #7 <signal handler called> #8 0xffffffff8321b134 in sdt_kld_unload_try () from /boot/kernel/sdt.ko #9 0xffffffff80b0f61c in linker_file_unload (file=0xfffffe00b04e7974, file@entry=0xfffff8015497b480, flags=flags@entry=1) at /home/zlei/freebsd-src-releng-14.0/sys/kern/kern_linker.c:673 #10 0xffffffff810be9c8 in link_elf_load_file (cls=<optimized out>, filename=<optimized out>, result=<optimized out>) at /home/zlei/freebsd-src-releng-14.0/sys/kern/link_elf_obj.c:1241 #11 0xffffffff80b0ee93 in LINKER_LOAD_FILE ( cls=0xffffffff817550a8 <link_elf_class>, result=0xfffffe00b04e7c00, filename=<optimized out>) at ./linker_if.h:214 #12 linker_load_file (filename=<optimized out>, result=<optimized out>) at /home/zlei/freebsd-src-releng-14.0/sys/kern/kern_linker.c:459 #13 linker_load_module (kldname=kldname@entry=0xfffff80154965800 "virtio.ko", modname=modname@entry=0x0, parent=parent@entry=0x0, verinfo=verinfo@entry=0x0, lfpp=lfpp@entry=0xfffffe00b04e7d90) at /home/zlei/freebsd-src-releng-14.0/sys/kern/kern_linker.c:2203 #14 0xffffffff80b10b9a in kern_kldload (td=td@entry=0xfffffe00b0d3a3a0, file=file@entry=0xfffff80154965800 "virtio.ko", fileid=fileid@entry=0xfffffe00b04e7de4) at /home/zlei/freebsd-src-releng-14.0/sys/kern/kern_linker.c:1162 #15 0xffffffff80b10cbc in sys_kldload (td=0xfffffe00b0d3a3a0, uap=<optimized out>) at /home/zlei/freebsd-src-releng-14.0/sys/kern/kern_linker.c:1185 #16 0xffffffff8100c0d9 in syscallenter (td=0xfffffe00b0d3a3a0) at /home/zlei/freebsd-src-releng-14.0/sys/amd64/amd64/../../kern/subr_syscall.c:187 #17 amd64_syscall (td=0xfffffe00b0d3a3a0, traced=0) at /home/zlei/freebsd-src-releng-14.0/sys/amd64/amd64/trap.c:1197 #18 <signal handler called> #19 0x000013e93dad93fa in ?? () Backtrace stopped: Cannot access memory at address 0x13e93c3e81c8 (kgdb) ``` Note: `kldload virtio.ko` can trigger the panic but `kldload virtio` will not.
This looks like a problem with DTrace, there is some logic in sdt.ko's kldunload handler to unregister providers, and something's going wrong there. virtqueue.c defines two rather odd SDT probes, so we're trying to unregister them.
(In reply to Mark Johnston from comment #1) > #8 0xffffffff8321b134 in sdt_kld_unload_try () from /boot/kernel/sdt.ko There is no line number of the source code for sdt.ko. I tried addr2line but without luck. So how should I debug this ? And it would be great if we can get line number of source code. Modules are efi-object and relocatable I guess we need extra effort to get the right line number.
** UPDATE ** With 14.0-RC3 I am able to get the line number of sdt.ko. Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 04 fault virtual address = 0x0 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff82e3a134 stack pointer = 0x28:0xfffffe00b0540920 frame pointer = 0x28:0xfffffe00b0540950 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 29038 (kldload) rdi: fffff8002ce880e0 rsi: fffffe00b0540928 rdx: 0000000000000000 rcx: ffffffff82e9d470 r8: 0000000000000004 r9: 00000000000000f3 rax: 0000000000000000 rbx: fffffe00b0540974 rbp: fffffe00b0540950 r10: 0000000000000002 r11: ffffffffffffff90 r12: fffff8000a0d8000 r13: fffffe00b0540974 r14: fffff8000a1e5c40 r15: ffffffff82e9d468 trap number = 12 panic: page fault cpuid = 2 time = 1698979988 KDB: stack backtrace: #0 0xffffffff80b9002d at kdb_backtrace+0x5d #1 0xffffffff80b43132 at vpanic+0x132 #2 0xffffffff80b42ff3 at panic+0x43 #3 0xffffffff8100c85c at trap_fatal+0x40c #4 0xffffffff8100c8af at trap_pfault+0x4f #5 0xffffffff80fe3818 at calltrap+0x8 #6 0xffffffff80b0f73c at linker_file_unload+0xcc #7 0xffffffff810bfa78 at link_elf_load_file+0x198 #8 0xffffffff80b0efb3 at linker_load_module+0x9e3 #9 0xffffffff80b10cba at kern_kldload+0x16a #10 0xffffffff80b10ddc at sys_kldload+0x5c #11 0xffffffff8100d119 at amd64_syscall+0x109 #12 0xffffffff80fe412b at fast_syscall_common+0xf8 Uptime: 1m14s Dumping 452 out of 8100 MB:..4%..11%..22%..32%..43%..54%..61%..71%..82%..92% (kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57 #1 doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:405 #2 0xffffffff80b42cc7 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:526 #3 0xffffffff80b4319f in vpanic (fmt=0xffffffff81136b3b "%s", ap=ap@entry=0xfffffe00b0540770) at /usr/src/sys/kern/kern_shutdown.c:970 #4 0xffffffff80b42ff3 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:894 #5 0xffffffff8100c85c in trap_fatal (frame=0xfffffe00b0540860, eva=0) at /usr/src/sys/amd64/amd64/trap.c:952 #6 0xffffffff8100c8af in trap_pfault (frame=0xfffffe00b0540860, usermode=false, signo=<optimized out>, ucode=<optimized out>) at /usr/src/sys/amd64/amd64/trap.c:760 #7 <signal handler called> #8 0xffffffff82e3a134 in sdt_kld_unload_try (arg=<optimized out>, lf=<optimized out>, error=0xfffffe00b0540974) at /usr/src/sys/cddl/dev/sdt/sdt.c:343 #9 0xffffffff80b0f73c in linker_file_unload ( file=file@entry=0xfffff8003d292180, flags=flags@entry=1) at /usr/src/sys/kern/kern_linker.c:673 #10 0xffffffff810bfa78 in link_elf_load_file (cls=<optimized out>, filename=<optimized out>, result=<optimized out>) at /usr/src/sys/kern/link_elf_obj.c:1241 #11 0xffffffff80b0efb3 in LINKER_LOAD_FILE ( cls=0xffffffff817560a8 <link_elf_class>, result=0xfffffe00b0540c00, filename=<optimized out>) at ./linker_if.h:214 #12 linker_load_file (filename=<optimized out>, result=<optimized out>) at /usr/src/sys/kern/kern_linker.c:459 #13 linker_load_module (kldname=kldname@entry=0xfffff8000a0c3800 "virtio.ko", modname=modname@entry=0x0, parent=parent@entry=0x0, verinfo=verinfo@entry=0x0, lfpp=lfpp@entry=0xfffffe00b0540d90) at /usr/src/sys/kern/kern_linker.c:2203 #14 0xffffffff80b10cba in kern_kldload (td=td@entry=0xfffffe00b1522720, file=file@entry=0xfffff8000a0c3800 "virtio.ko", fileid=fileid@entry=0xfffffe00b0540de4) at /usr/src/sys/kern/kern_linker.c:1162 #15 0xffffffff80b10ddc in sys_kldload (td=0xfffffe00b1522720, uap=<optimized out>) at /usr/src/sys/kern/kern_linker.c:1185 #16 0xffffffff8100d119 in syscallenter (td=0xfffffe00b1522720) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:187 #17 amd64_syscall (td=0xfffffe00b1522720, traced=0) at /usr/src/sys/amd64/amd64/trap.c:1197 #18 <signal handler called> #19 0x00002ab829148b3a in ?? () Backtrace stopped: Cannot access memory at address 0x2ab8285939b8 (kgdb)
(In reply to Mark Johnston from comment #1) > This looks like a problem with DTrace, there is some logic in sdt.ko's kldunload > handler to unregister providers, and something's going wrong there. virtqueue.c > defines two rather odd SDT probes, so we're trying to unregister them. That happens when a module file with SDT probes is failed to be loaded by kernel linker. While the kernel try to invoke kld_unload_try event handlers for it and apparently it should not, as the file is in the progress of loading, i.e. it is partially linked. As for > Note: `kldload virtio.ko` can trigger the panic but `kldload virtio` will not. kernel linkers will check module named `virtio` firstly and refuse to load so no bad things happen. But for `virtio.ko` the kernel will load the file and get all modules in it before it knows `virtio` is present. Proposed fix https://reviews.freebsd.org/D44594.