I recently updated to FreeBSD 14 while keeping the same configuration otherwise. I just experienced a kernel panic which appears to have been caused by ipfw. From kgdb: Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 02 fault virtual address = 0x0 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff81008300 stack pointer = 0x28:0xfffffe00aece5380 frame pointer = 0x28:0xfffffe00aece5380 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 91770 (python3.10) rdi: fffff80095e17998 rsi: 0000000000000000 rdx: 000000000000003c rcx: 000000000000003c r8: fffff80095e17998 r9: ffffffff80c67b10 rax: fffff80095e17998 rbx: fffff80095e17900 rbp: fffffe00aece5380 r10: 0000000000000000 r11: fffffe00aece5610 r12: 000000000000003c r13: 0000000000000068 r14: 0000000000000006 r15: fffff801e118cc00 trap number = 12 panic: page fault cpuid = 1 time = 1711859735 KDB: stack backtrace: #0 0xffffffff80b9009d at kdb_backtrace+0x5d #1 0xffffffff80b431a2 at vpanic+0x132 #2 0xffffffff80b43063 at panic+0x43 #3 0xffffffff8100c85c at trap_fatal+0x40c #4 0xffffffff8100c8af at trap_pfault+0x4f #5 0xffffffff80fe3ac8 at calltrap+0x8 #6 0xffffffff80bda858 at m_pullup+0x1c8 #7 0xffffffff82e39516 at ipfw_chk+0x11f6 #8 0xffffffff82e3e3f4 at ipfw_check_frame_mbuf+0xd4 #9 0xffffffff80c871d8 at pfil_mbuf_out+0x38 #10 0xffffffff80c68362 at ether_output_frame+0xe2 #11 0xffffffff80c68198 at ether_output+0x688 #12 0xffffffff80d02156 at ip_output+0x1266 #13 0xffffffff80d1b4ef at tcp_default_output+0x1eef #14 0xffffffff80d2c70e at tcp_usr_ready+0x18e #15 0xffffffff80b413e5 at sendfile_iodone+0x115 #16 0xffffffff80b40703 at vn_sendfile+0x1163 #17 0xffffffff80b41779 at sendfile+0x119 It appears this was caused by ipfw, and it looks like possibly it would have been applying some layer-2 rules, which may be somewhat less commonly exercised. In case it helps, here are my ethernet ipfw rules: 00010 deny ip from any to any MAC any 54:e0:19:92:69:ba in recv em0 00010 deny ip from any to any MAC any 9c:76:13:14:64:08 in recv em0 00020 allow ip from any to any MAC any any via em0 (These rules block-list a few devices on my home network which occasionally advertise themselves via ARP as the network's router.) I will try disabling layer-2 for now, as I have employed a different workaround for the devices on my LAN that I needed to block, and see if that prevents a recurrence. I'll also keep my vmcore around in case there are any commands someone wishes me to run against it.
uname from this host: 14.0-RELEASE-p6 FreeBSD 14.0-RELEASE-p6 #0: Tue Mar 26 20:26:20 UTC 2024 root@amd64-builder.daemonology.net:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
I can confirm that disabling layer-2 filtering in ipfw seems to eliminate this crash. The system in question has been up for 46 days now, when previously it was crashing after just a few days.