I got this rctl-related panic, with no rtcl rule loaded at the time. Userland and kernel at r287568. Fatal trap 12: page fault while in kernel mode cpuid = 3; apic id = 03 fault virtual address = 0xa8 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff8095963c stack pointer = 0x28:0xfffffe01af1c17c0 frame pointer = 0x28:0xfffffe01af1c1860 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 826 (beam.smp) trap number = 12 panic: page fault cpuid = 3 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe01af1c12a0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe01af1c1350 vpanic() at vpanic+0x126/frame 0xfffffe01af1c1390 panic() at panic+0x43/frame 0xfffffe01af1c13f0 trap_fatal() at trap_fatal+0x36b/frame 0xfffffe01af1c1450 trap_pfault() at trap_pfault+0x2ed/frame 0xfffffe01af1c14f0 trap() at trap+0x47a/frame 0xfffffe01af1c1700 calltrap() at calltrap+0x8/frame 0xfffffe01af1c1700 --- trap 0xc, rip = 0xffffffff8095963c, rsp = 0xfffffe01af1c17c0, rbp = 0xfffffe01af1c1860 --- rctl_enforce() at rctl_enforce+0x3c/frame 0xfffffe01af1c1860 racct_proc_fork_done() at racct_proc_fork_done+0xda/frame 0xfffffe01af1c1890 fork1() at fork1+0x1f57/frame 0xfffffe01af1c1970 sys_vfork() at sys_vfork+0x1f/frame 0xfffffe01af1c1990 amd64_syscall() at amd64_syscall+0x357/frame 0xfffffe01af1c1ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe01af1c1ab0 --- syscall (66, FreeBSD ELF64, sys_vfork), rip = 0x8017f3a2d, rsp = 0x7fffdf32c6c0, rbp = 0x7fffdf32c800 --- Uptime: 4h0m2s Dumping 1068 out of 6115 MB:..2%..11%..21%..32%..41%..51%..62%..71%..81%..92% Reading symbols from /boot/kernel/nullfs.ko.symbols...done. Loaded symbols for /boot/kernel/nullfs.ko.symbols Reading symbols from /boot/kernel/unionfs.ko.symbols...done. Loaded symbols for /boot/kernel/unionfs.ko.symbols Reading symbols from /boot/kernel/zfs.ko.symbols...done. Loaded symbols for /boot/kernel/zfs.ko.symbols Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. Loaded symbols for /boot/kernel/opensolaris.ko.symbols Reading symbols from /boot/kernel/geom_eli.ko.symbols...done. Loaded symbols for /boot/kernel/geom_eli.ko.symbols Reading symbols from /boot/kernel/geom_mirror.ko.symbols...done. Loaded symbols for /boot/kernel/geom_mirror.ko.symbols Reading symbols from /boot/kernel/geom_uzip.ko.symbols...done. Loaded symbols for /boot/kernel/geom_uzip.ko.symbols Reading symbols from /boot/kernel/if_tap.ko.symbols...done. Loaded symbols for /boot/kernel/if_tap.ko.symbols Reading symbols from /boot/kernel/pf.ko.symbols...done. Loaded symbols for /boot/kernel/pf.ko.symbols Reading symbols from /boot/kernel/bridgestp.ko.symbols...done. Loaded symbols for /boot/kernel/bridgestp.ko.symbols Reading symbols from /boot/kernel/if_bridge.ko.symbols...done. Loaded symbols for /boot/kernel/if_bridge.ko.symbols Reading symbols from /boot/kernel/ipmi.ko.symbols...done. Loaded symbols for /boot/kernel/ipmi.ko.symbols Reading symbols from /boot/kernel/smbus.ko.symbols...done. Loaded symbols for /boot/kernel/smbus.ko.symbols Reading symbols from /boot/kernel/nmdm.ko.symbols...done. Loaded symbols for /boot/kernel/nmdm.ko.symbols Reading symbols from /boot/kernel/fdescfs.ko.symbols...done. Loaded symbols for /boot/kernel/fdescfs.ko.symbols Reading symbols from /boot/modules/vboxguest.ko...done. Loaded symbols for /boot/modules/vboxguest.ko Reading symbols from /boot/kernel/linux.ko.symbols...done. Loaded symbols for /boot/kernel/linux.ko.symbols Reading symbols from /boot/kernel/blank_saver.ko.symbols...done. Loaded symbols for /boot/kernel/blank_saver.ko.symbols #0 doadump (textdump=1) at pcpu.h:219 219 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump (textdump=1) at pcpu.h:219 #1 0xffffffff80962487 in kern_reboot (howto=260) at /usr/storage/OS/src/10/src/sys/kern/kern_shutdown.c:451 #2 0xffffffff80962885 in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/storage/OS/src/10/src/sys/kern/kern_shutdown.c:758 #3 0xffffffff80962713 in panic (fmt=0x0) at /usr/storage/OS/src/10/src/sys/kern/kern_shutdown.c:687 #4 0xffffffff80d973fb in trap_fatal (frame=<value optimized out>, eva=<value optimized out>) at /usr/storage/OS/src/10/src/sys/amd64/amd64/trap.c:851 #5 0xffffffff80d976fd in trap_pfault (frame=0xfffffe01af1c1710, usermode=<value optimized out>) at /usr/storage/OS/src/10/src/sys/amd64/amd64/trap.c:674 #6 0xffffffff80d96d9a in trap (frame=0xfffffe01af1c1710) at /usr/storage/OS/src/10/src/sys/amd64/amd64/trap.c:440 #7 0xffffffff80d7c892 in calltrap () at /usr/storage/OS/src/10/src/sys/amd64/amd64/exception.S:236 #8 0xffffffff8095963c in rctl_enforce (p=0xfffff80059e6d4e8, resource=6, amount=0) at /usr/storage/OS/src/10/src/sys/kern/kern_rctl.c:350 #9 0xffffffff809581da in racct_proc_fork_done (child=0xfffff80059e6d4e8) at /usr/storage/OS/src/10/src/sys/kern/kern_racct.c:969 #10 0xffffffff8092ae07 in fork1 (td=<value optimized out>, flags=<value optimized out>, pages=Cannot access memory at address 0x4 ) at /usr/storage/OS/src/10/src/sys/kern/kern_fork.c:952 #11 0xffffffff8092ae9f in sys_vfork (td=0xfffff8008947b940, uap=<value optimized out>) at /usr/storage/OS/src/10/src/sys/kern/kern_fork.c:152 #12 0xffffffff80d97d17 in amd64_syscall (td=0xfffff8008947b940, traced=0) at subr_syscall.c:134 #13 0xffffffff80d7cb7b in Xfast_syscall () at /usr/storage/OS/src/10/src/sys/amd64/amd64/exception.S:396 #14 0x00000008017f3a2d in ?? () Previous frame inner to this frame (corrupt stack?) Current language: auto; currently minimal (kgdb)
This looks like a use-after-free I mentioned some time ago. do_fork makes newproc runnable and fork1 does not pin it in any way, thus by the time do_fork returns the process could have already exited. Interestingly do_fork itself has this problem. Here faulting address 0xa8 matches what would be linked list access in a struct racct if read pointer was null. Pointer in question is nullified on process exit and initialized on fork. I'll ponder a reasonable fix.
(In reply to Mateusz Guzik from comment #1) Quick data point: I never saw this again so far (following -STABLE).
Got something related today: FreeBSD olgeni 10.3-PRERELEASE FreeBSD 10.3-PRERELEASE #4 r296610: Thu Mar 10 11:09:56 CET 2016 root@olgeni:/usr/obj/usr/src/sys/KERNEL amd64 #0 doadump (textdump=1) at pcpu.h:219 #1 0xffffffff8096f557 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:486 #2 0xffffffff8096f955 in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:889 #3 0xffffffff8096f7e3 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:818 #4 0xffffffff80da757b in trap_fatal (frame=<value optimized out>, eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:858 #5 0xffffffff80da787d in trap_pfault (frame=0xfffffe046a799860, usermode=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:681 #6 0xffffffff80da6efa in trap (frame=0xfffffe046a799860) at /usr/src/sys/amd64/amd64/trap.c:447 #7 0xffffffff80d8c8e2 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:236 #8 0xffffffff8096637c in rctl_enforce (p=0xfffff80412f379e0, resource=6, amount=0) at /usr/src/sys/kern/kern_rctl.c:350 #9 0xffffffff80964f1a in racct_proc_fork_done (child=0xfffff80412f379e0) at /usr/src/sys/kern/kern_racct.c:967 #10 0xffffffff8093785b in fork1 (td=<value optimized out>, flags=<value optimized out>, pages=Cannot access memory at address 0x4 ) at /usr/src/sys/kern/kern_fork.c:964 #11 0xffffffff809378ff in sys_vfork (td=0xfffff80101576960, uap=<value optimized out>) at /usr/src/sys/kern/kern_fork.c:153 #12 0xffffffff80da7f4f in amd64_syscall (td=0xfffff80101576960, traced=0) at subr_syscall.c:141 #13 0xffffffff80d8cbcb in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396 #14 0x00000008017fa65d in ?? () Previous frame inner to this frame (corrupt stack?) Current language: auto; currently minimal
Had a look with kgdb - it stops in "LIST_FOREACH(link, &p->p_racct->r_rule_links, rrl_next)": #1 0xffffffff8096f557 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:486 486 doadump(TRUE); #2 0xffffffff8096f955 in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:889 889 kern_reboot(bootopt); #3 0xffffffff8096f7e3 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:818 818 vpanic(fmt, ap); #4 0xffffffff80da757b in trap_fatal (frame=<value optimized out>, eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:858 858 panic("%s", trap_msg[type]); #5 0xffffffff80da787d in trap_pfault (frame=0xfffffe046a799860, usermode=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:681 681 trap_fatal(frame, eva); #6 0xffffffff80da6efa in trap (frame=0xfffffe046a799860) at /usr/src/sys/amd64/amd64/trap.c:447 447 (void) trap_pfault(frame, FALSE); #7 0xffffffff80d8c8e2 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:236 236 call trap Current language: auto; currently asm #8 0xffffffff8096637c in rctl_enforce (p=0xfffff80412f379e0, resource=6, amount=0) at /usr/src/sys/kern/kern_rctl.c:350 350 LIST_FOREACH(link, &p->p_racct->r_rule_links, rrl_next) { Current language: auto; currently minimal #9 0xffffffff80964f1a in racct_proc_fork_done (child=0xfffff80412f379e0) at /usr/src/sys/kern/kern_racct.c:967 967 rctl_enforce(child, RACCT_NPROC, 0); #10 0xffffffff8093785b in fork1 (td=<value optimized out>, flags=<value optimized out>, pages=Cannot access memory at address 0x4 ) at /usr/src/sys/kern/kern_fork.c:964 964 racct_proc_fork_done(newproc); #11 0xffffffff809378ff in sys_vfork (td=0xfffff80101576960, uap=<value optimized out>) at /usr/src/sys/kern/kern_fork.c:153 153 error = fork1(td, flags, 0, &p2, NULL, 0); #12 0xffffffff80da7f4f in amd64_syscall (td=0xfffff80101576960, traced=0) at subr_syscall.c:141 141 error = (sa->callp->sy_call)(td, sa->args); #13 0xffffffff80d8cbcb in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396 396 call amd64_syscall Current language: auto; currently asm #14 0x00000008017fa65d in ?? ()
batch change: For bugs that match the following - Status Is In progress AND - Untouched since 2018-01-01. AND - Affects Base System OR Documentation DO: Reset to open status. Note: I did a quick pass but if you are getting this email it might be worthwhile to double check to see if this bug ought to be closed.
I never saw this again since 2016 on both 10 and 11, so I guess this can be closed unless something else comes up.