A router running multiple wireguard tunnels running 14.2-RELEASE-p1 with kernel FreeBSD caesium 14.2-RELEASE-p1 FreeBSD 14.2-RELEASE-p1 GENERIC amd64 keep panic with dmesg written below. We have observed that the rate of panic is much lower after a Mikrotik peer was shutdown. Fatal trap 12: page fault while in kernel mode cpuid = 6; apic id = 0c fault virtual address = 0x10 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80cfc37f stack pointer = 0x28:0xfffffe010c480c20 frame pointer = 0x28:0xfffffe010c480cb0 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 0 (wg_tqg_6) rdi: 0000000000000000 rsi: fffffe010c4808c8 rdx: ffffffff82a16090 rcx: 00000000ffffffff r8: ffffffff82a16090 r9: 000000000000012c rax: 0000000000000000 rbx: fffff800057f6d70 rbp: fffffe010c480cb0 r10: 0000000000000006 r11: fffffe001ebd6468 r12: fffff800057f6d00 r13: fffff8007dd72000 r14: fffff800057f6d00 r15: 0000000008080808 trap number = 12 panic: page fault cpuid = 6 time = 1738843725 KDB: stack backtrace: #0 0xffffffff80b8b88d at kdb_backtrace+0x5d #1 0xffffffff80b3dc11 at vpanic+0x131 #2 0xffffffff80b3dad3 at panic+0x43 #3 0xffffffff81025a0b at trap_fatal+0x40b #4 0xffffffff81025a56 at trap_pfault+0x46 #5 0xffffffff80ffc388 at calltrap+0x8 #6 0xffffffff80cfeb4b at ip_input+0x32b #7 0xffffffff80c8387e at netisr_dispatch_src+0x9e #8 0xffffffff83169683 at wg_deliver_in+0x363 #9 0xffffffff80b8a24e at gtaskqueue_run_locked+0x14e #10 0xffffffff80b89ed2 at gtaskqueue_thread_loop+0xc2 #11 0xffffffff80af760f at fork_exit+0x7f #12 0xffffffff80ffd3ee at fork_trampoline+0xe Uptime: 2m24s Dumping 866 out of 16343 MB:..2%..12%..21%..32%..41%..52%..61%..71%..82%..91% Dump complete
A dump of /var/crash can be found at https://webresources.dreamry.org/dump284606.tar.gz
% kgdb /boot/kernel/kernel vmcore.0 ... Reading symbols from /boot/kernel/zfs.ko... Reading symbols from /usr/lib/debug//boot/kernel/zfs.ko.debug... Reading symbols from /boot/kernel/intpm.ko... Reading symbols from /usr/lib/debug//boot/kernel/intpm.ko.debug... Reading symbols from /boot/kernel/smbus.ko... Reading symbols from /usr/lib/debug//boot/kernel/smbus.ko.debug... Reading symbols from /boot/kernel/vmci.ko... Reading symbols from /usr/lib/debug//boot/kernel/vmci.ko.debug... Reading symbols from /boot/kernel/if_epair.ko... Reading symbols from /usr/lib/debug//boot/kernel/if_epair.ko.debug... Reading symbols from /boot/kernel/ipfw.ko... Reading symbols from /usr/lib/debug//boot/kernel/ipfw.ko.debug... Reading symbols from /boot/kernel/ipfw_nat.ko... Reading symbols from /usr/lib/debug//boot/kernel/ipfw_nat.ko.debug... Reading symbols from /boot/kernel/libalias.ko... Reading symbols from /usr/lib/debug//boot/kernel/libalias.ko.debug... Reading symbols from /boot/kernel/ipfw_pmod.ko... Reading symbols from /usr/lib/debug//boot/kernel/ipfw_pmod.ko.debug... Reading symbols from /boot/kernel/if_wg.ko... Reading symbols from /usr/lib/debug//boot/kernel/if_wg.ko.debug... ... (kgdb) frame 8 #8 0xffffffff80cfc37f in ip_tryforward (m=0x0) at /usr/src/sys/netinet/ip_fastfwd.c:319 319 ip = mtod(m, struct ip *); /* m may have changed by pfil hook */ (kgdb) list 309 if (!PFIL_HOOKED_IN(V_inet_pfil_head)) 310 goto passin; 311 312 if (pfil_mbuf_in(V_inet_pfil_head, &m, m->m_pkthdr.rcvif, 313 NULL) != PFIL_PASS) 314 goto drop; 315 316 M_ASSERTVALID(m); 317 M_ASSERTPKTHDR(m); 318 319 ip = mtod(m, struct ip *); /* m may have changed by pfil hook */ 320 dest.s_addr = ip->ip_dst.s_addr; 321 322 /* 323 * Destination address changed? 324 */ 325 if (odest.s_addr != dest.s_addr) { 326 /* 327 * Is it now for a local address on this host? 328 */ (kgdb) p m $14 = (struct mbuf *) 0x0 (kgdb) p ip $15 = (struct ip *) 0xfffff800057f6d70 (kgdb) I noticed that ipfw(4) and related modules were loaded. From the debug info of core dump, pfil hook returns PFIL_PASS but set m to NULL ( the original mbuf be freed ), it appears ipfw(4) is to be blamed.
Since this panic happens frequently, please let me know if there's anything I can do to collect more info about this.
(In reply to Wencey Wang from comment #3) I do not have Mikrotik devices so maybe this issue will be hard to repeat. Can you share the ipfw config that related to wg(4) interfaces ? Also CC Gleb, Andrey V and Kristof who have contributed to ipfw recently.
(In reply to Zhenlei Huang from comment #4) ipfw config is shown below. I am willing to provide any needed information, and such panic can be reproduced easily in my environment. The only issue is that I wasn't familiar with debugging the kernel. Please give me instructions to collect any needed information. root@caesium ~ # cat /etc/ipfw.rules #!/bin/sh - ipfw -q -f flush ipfw nat 1 config if vmx0 redirect_port udp 10.0.40.108:25565 19133 redirect_port tcp 10.0.40.108:25565 19133 ipfw nat 2 config if vmx3 ipfw add 00001 setfib tablearg ip from any to any sockarg ipfw add 00040 tcp-setmss 1280 ip from any to any ipfw add 00050 nat 1 ip4 from 10.0.0.0/14 to any out via vmx0 ipfw add 00051 nat 1 ip4 from any to any in via vmx0 ipfw add 00060 nat 2 ip4 from 10.0.0.0/14 to any out via vmx3 ipfw add 00061 nat 2 ip4 from any to any in via vmx3 ipfw add 10000 allow all from any to any
After testing over these days, we found the actual cause is not Mikrotik, but some packets sent by an Xbox plugged into the Mikrotik. This is confirmed by applying ip rule to exclude any packet from that Xbox. Is there any way for me to maybe also save the content of m while core dumping? I would like to address this issue since this kernel panic can be used to DoS.
(In reply to Wencey Wang from comment #6) The ipfw(4) and net stack have lots of asserting. That is most useful for developing, but I think that will be useful for your case. May you please enable the debug option INVARIANTS and test the kernel and ipfw ? Ref: https://docs.freebsd.org/en/books/developers-handbook/kerneldebug/#kerneldebug-options
Hello, teams. After more digging, I have replicated this issue stability on a BuyVM VPS, without using Mikrotik devices, but only from packets generated from another FreeBSD server, and the stack trace now starts from epair. I have also built the 14.3-RELEASE-p5 kernel using GENERIC-DEBUG, which panics in ipfw_check_packet() and then enters kdb. Since the environment is now separate, I am willing to provide VNC connections for you to debug, if that helps.
Created attachment 265006 [details] stacktrace on 14.3-RELEASE-p5 GENERIC-DEBUG Please let me know if anything further I can help with, since the networking environment that panics the kernel is really time-consuming to build.
Seen exactly the same panic with net/amnezia-kmod instead of if_wg, it is based on if_wg, svery possible that root case is the same. In may case - caused by following ipfw rule: ipfw add tcp-setmss 1380 tcp from any to any out xmit wg0 tcpflags syn I have a kernel cash dump: without debug it is very similar to this one: Fatal trap 12: page fault while in kernel mode ... --- trap 0xc, rip = 0xffffffff80cd9ac4, rsp = 0xfffffe0159ac9c20, rbp = 0xfffffe0159ac9cb0 --- ip_tryforward() at ip_tryforward+0x274/frame 0xfffffe0159ac9cb0 ip_input() at ip_input+0x321/frame 0xfffffe0159ac9d10 netisr_dispatch_src() at netisr_dispatch_src+0x9f/frame 0xfffffe0159ac9d60 wg_deliver_in() at wg_deliver_in+0x3ad/frame 0xfffffe0159ac9e40 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x14e/frame 0xfffffe0159ac9ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xc2/frame 0xfffffe0159ac9ef0 ... (kgdb) fr 8 #8 0xffffffff80cd9ac4 in ip_tryforward (m=0x0) at /usr/src/sys/netinet/ip_fastfwd.c:416 416 ip = mtod(m, struct ip *); (kgdb) p m $1 = (struct mbuf *) 0x0 (kgdb) fr 9 #9 0xffffffff80cdc251 in ip_input (m=0xfffff8020f13a500) at /usr/src/sys/netinet/ip_input.c:587 587 if ((m = ip_tryforward(m)) == NULL) (kgdb) p m $2 = (struct mbuf *) 0xfffff8020f13a500 (kgdb) so, that means that pfil_mbuf_fwd() returned with m == NULL and PFIL_PASS here: https://github.com/freebsd/freebsd-src/blob/main/sys/netinet/ip_fastfwd.c#L407 which is not expected then I have reproduced it with debug kernel - it died earlier, in firewall: Unread portion of the kernel message buffer: panic: ipfw_check_packet: m0 is NULL cpuid = 9 time = 1761568018 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0164266950 vpanic() at vpanic+0x161/frame 0xfffffe0164266a80 panic() at panic+0x43/frame 0xfffffe0164266ae0 ipfw_check_packet() at ipfw_check_packet+0x6ba/frame 0xfffffe0164266bd0 pfil_mbuf_out() at pfil_mbuf_out+0x58/frame 0xfffffe0164266c00 ip_tryforward() at ip_tryforward+0x2a5/frame 0xfffffe0164266ca0 ip_input() at ip_input+0x3af/frame 0xfffffe0164266d00 netisr_dispatch_src() at netisr_dispatch_src+0xb4/frame 0xfffffe0164266d60 wg_deliver_in() at wg_deliver_in+0x3ad/frame 0xfffffe0164266e40 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x18e/frame 0xfffffe0164266ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xd3/frame 0xfffffe0164266ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0164266f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0164266f30 --- trap 0xdeadc0de, rip = 0xdeadc0dedeadc0de, rsp = 0xdeadc0dedeadc0de, rbp = 0xdeadc0dedeadc0de --- (kgdb) bt #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57 #1 doadump (textdump=textdump@entry=1) at /usr/src/sys/kern/kern_shutdown.c:405 #2 0xffffffff80b0d420 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:523 #3 0xffffffff80b0d939 in vpanic (fmt=0xffffffff835056aa "%s: m0 is NULL", ap=ap@entry=0xfffffe0164266ac0) at /usr/src/sys/kern/kern_shutdown.c:967 #4 0xffffffff80b0d6c3 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:891 #5 0xffffffff834e666a in ipfwlog_clone_create (ifc=<optimized out>, unit=0, params=<optimized out>) at /usr/src/sys/netpfil/ipfw/ip_fw_bpf.c:134 #6 0xffffffff80c71d18 in pfil_mbuf_common (pch=<optimized out>, m=0x2, m@entry=0xfffffe0164266b98, ifp=0xfffffe0164266c38, flags=131072, inp=0xfffff80004f2e000, inp@entry=0x0) at /usr/src/sys/net/pfil.c:212 #7 pfil_mbuf_out (head=<optimized out>, m=0x2, m@entry=0xfffffe0164266c38, ifp=0xfffffe0164266c38, inp=0xfffff80004f2e000, inp@entry=0x0) at /usr/src/sys/net/pfil.c:233 #8 0xffffffff80cf1e75 in ip_tryforward (m=0x0) at /usr/src/sys/netinet/ip_fastfwd.c:409 #9 0xffffffff80cf4dff in ip_input (m=0xfffff8020c324700) at /usr/src/sys/netinet/ip_input.c:587 #10 0xffffffff80c6dea4 in netisr_dispatch_src (proto=1, source=0, m=0xfffff8020c324700) at /usr/src/sys/net/netisr.c:1152 #11 0xffffffff8352663d in wg_deliver_in () from /boot/modules/if_amn.ko #12 0xffffffff80b5c7ce in gtaskqueue_run_locked (queue=0x2, queue@entry=0xfffff8021de61900) at /usr/src/sys/kern/subr_gtaskqueue.c:369 #13 0xffffffff80b5c503 in gtaskqueue_thread_loop (arg=arg@entry=0xfffffe016409f0e0) at /usr/src/sys/kern/subr_gtaskqueue.c:545 #14 0xffffffff80ac0022 in fork_exit (callout=0xffffffff80b5c430 <gtaskqueue_thread_loop>, arg=0xfffffe016409f0e0, frame=0xfffffe0164266f40) at /usr/src/sys/kern/kern_fork.c:1153 #15 <signal handler called> #16 0xdeadc0dedeadc0de in ?? () Backtrace stopped: Cannot access memory at address 0xdeadc0dedeadc0de (kgdb) bt #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57 #1 doadump (textdump=textdump@entry=1) at /usr/src/sys/kern/kern_shutdown.c:405 #2 0xffffffff80b0d420 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:523 #3 0xffffffff80b0d939 in vpanic (fmt=0xffffffff835056aa "%s: m0 is NULL", ap=ap@entry=0xfffffe0164266ac0) at /usr/src/sys/kern/kern_shutdown.c:967 #4 0xffffffff80b0d6c3 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:891 #5 0xffffffff834e666a in ipfwlog_clone_create (ifc=<optimized out>, unit=0, params=<optimized out>) at /usr/src/sys/netpfil/ipfw/ip_fw_bpf.c:134 #6 0xffffffff80c71d18 in pfil_mbuf_common (pch=<optimized out>, m=0x2, m@entry=0xfffffe0164266b98, ifp=0xfffffe0164266c38, flags=131072, inp=0xfffff80004f2e000, inp@entry=0x0) at /usr/src/sys/net/pfil.c:212 #7 pfil_mbuf_out (head=<optimized out>, m=0x2, m@entry=0xfffffe0164266c38, ifp=0xfffffe0164266c38, inp=0xfffff80004f2e000, inp@entry=0x0) at /usr/src/sys/net/pfil.c:233 #8 0xffffffff80cf1e75 in ip_tryforward (m=0x0) at /usr/src/sys/netinet/ip_fastfwd.c:409 #9 0xffffffff80cf4dff in ip_input (m=0xfffff8020c324700) at /usr/src/sys/netinet/ip_input.c:587 #10 0xffffffff80c6dea4 in netisr_dispatch_src (proto=1, source=0, m=0xfffff8020c324700) at /usr/src/sys/net/netisr.c:1152 #11 0xffffffff8352663d in wg_deliver_in () from /boot/modules/if_amn.ko #12 0xffffffff80b5c7ce in gtaskqueue_run_locked (queue=0x2, queue@entry=0xfffff8021de61900) at /usr/src/sys/kern/subr_gtaskqueue.c:369 #13 0xffffffff80b5c503 in gtaskqueue_thread_loop (arg=arg@entry=0xfffffe016409f0e0) at /usr/src/sys/kern/subr_gtaskqueue.c:545 #14 0xffffffff80ac0022 in fork_exit (callout=0xffffffff80b5c430 <gtaskqueue_thread_loop>, arg=0xfffffe016409f0e0, frame=0xfffffe0164266f40) at /usr/src/sys/kern/kern_fork.c:1153 #15 <signal handler called> #16 0xdeadc0dedeadc0de in ?? () Backtrace stopped: Cannot access memory at address 0xdeadc0dedeadc0de (kgdb) fr 9 #9 0xffffffff80cf4dff in ip_input (m=0xfffff8020c324700) at /usr/src/sys/netinet/ip_input.c:587 587 if ((m = ip_tryforward(m)) == NULL) (kgdb) p m $1 = (struct mbuf *) 0xfffff8020c324700 (kgdb) p *m $2 = {{m_next = 0xdeadc0dedeadc0de, m_slist = {sle_next = 0xdeadc0dedeadc0de}, m_stailq = {stqe_next = 0xdeadc0dedeadc0de}}, {m_nextpkt = 0xdeadc0dedeadc0de, m_slistpkt = {sle_next = 0xdeadc0dedeadc0de}, m_stailqpkt = {stqe_next = 0xdeadc0dedeadc0de}}, m_data = 0xdeadc0dedeadc0de <error: Cannot access memory at address 0xdeadc0dedeadc0de>, m_len = -559038242, m_type = 222, m_flags = 14593472, {{{m_pkthdr = {{ snd_tag = 0xdeadc0dedeadc0de, rcvif = 0xdeadc0dedeadc0de, {rcvidx = 49374, rcvgen = 57005}}, {leaf_rcvif = 0xdeadc0dedeadc0de, {leaf_rcvidx = 49374, leaf_rcvgen = 57005}}, tags = {slh_first = 0xdeadc0dedeadc0de}, len = -559038242, flowid = 3735929054, csum_flags = 3735929054, fibnum = 49374, numa_domain = 173 '\255', rsstype = 222 '\336', {rcv_tstmp = 16045693110842147038, {l2hlen = 222 '\336', l3hlen = 192 '\300', l4hlen = 173 '\255', l5hlen = 222 '\336', inner_l2hlen = 222 '\336', inner_l3hlen = 192 '\300', inner_l4hlen = 173 '\255', inner_l5hlen = 222 '\336'}}, PH_per = {eight = "\336\300\255\336\336\300\255", <incomplete sequence \336>, sixteen = {49374, 57005, 49374, 57005}, thirtytwo = {3735929054, 3735929054}, sixtyfour = {16045693110842147038}, unintptr = {16045693110842147038}, ptr = 0xdeadc0dedeadc0de}, {PH_loc = {eight = "\336\300\255\336\336\300\255", <incomplete sequence \336>, sixteen = {49374, 57005, 49374, 57005}, thirtytwo = {3735929054, 3735929054}, sixtyfour = {16045693110842147038}, unintptr = {16045693110842147038}, ptr = 0xdeadc0dedeadc0de}, memlen = 3735929054}}, {m_epg_npgs = 222 '\336', m_epg_nrdy = 192 '\300', m_epg_hdrlen = 173 '\255', m_epg_trllen = 222 '\336', m_epg_1st_off = 49374, m_epg_last_len = 57005, m_epg_flags = 222 '\336', m_epg_record_type = 192 '\300', __spare = "\255", <incomplete sequence \336>, m_epg_enc_cnt = -559038242, m_epg_tls = 0xdeadc0dedeadc0de, m_epg_so = 0xdeadc0dedeadc0de, m_epg_seqno = 16045693110842147038, m_epg_stailq = {stqe_next = 0xdeadc0dedeadc0de}}}, {m_ext = {{ext_count = 3735929054, ext_cnt = 0xdeadc0dedeadc0de}, ext_size = 3735929054, ext_type = 222, ext_flags = 14593472, {{ext_buf = 0xdeadc0dedeadc0de <error: Cannot access memory at address 0xdeadc0dedeadc0de>, ext_arg2 = 0xdeadc0dedeadc0de}, {extpg_pa = {16045693110842147038, 16045693110842147038, 16045693110842147038, 16045693110842147038, 16045693110842147038}, extpg_trail = "\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255", <incomplete sequence \336>, extpg_hdr = "\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255"}}, ext_free = 0xdeadc0dedeadc0de, ext_arg1 = 0xdeadc0dedeadc0de}, m_pktdat = 0xfffff8020c324760 "\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255", <incomplete sequence \336>...}}, m_dat = 0xfffff8020c324720 "\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255", <incomplete sequence \336>...}} (kgdb) fr 8 #8 0xffffffff80cf1e75 in ip_tryforward (m=0x0) at /usr/src/sys/netinet/ip_fastfwd.c:409 warning: Source file is more recent than executable. 409 if (pfil_mbuf_out(V_inet_pfil_head, &m, nh->nh_ifp, (kgdb) p m $3 = (struct mbuf *) 0x0 So, sounds like if firewall updates packet something get broken. Additional considiration, on other system I have pf firewall updating MSS and it does not breaks. But I can't say for sure, as far as I have no exact reproduction, it just fires few times a day with some wg clients.
(In reply to vova from comment #10) for the non-debug kernel - mbuf contents on (kgdb) fr 9 #9 0xffffffff80cdc251 in ip_input (m=0xfffff8020f13a500) at /usr/src/sys/netinet/ip_input.c:587 587 if ((m = ip_tryforward(m)) == NULL) (kgdb) p *m $8 = {{m_next = 0xfffff802220fbc00, m_slist = {sle_next = 0xfffff802220fbc00}, m_stailq = {stqe_next = 0xfffff802220fbc00}}, {m_nextpkt = 0x0, m_slistpkt = {sle_next = 0x0}, m_stailqpkt = {stqe_next = 0x0}}, m_data = 0xfffff8020f13a570 "E", m_len = 44, m_type = 1, m_flags = 2, {{{m_pkthdr = {{snd_tag = 0xfffff8019c449000, rcvif = 0xfffff8019c449000, {rcvidx = 36864, rcvgen = 40004}}, {leaf_rcvif = 0x0, {leaf_rcvidx = 0, leaf_rcvgen = 0}}, tags = {slh_first = 0x0}, len = 589, flowid = 1323187103, csum_flags = 0, fibnum = 0, numa_domain = 255 '\377', rsstype = 0 '\000', {rcv_tstmp = 0, {l2hlen = 0 '\000', l3hlen = 0 '\000', l4hlen = 0 '\000', l5hlen = 0 '\000', inner_l2hlen = 0 '\000', inner_l3hlen = 0 '\000', inner_l4hlen = 0 '\000', inner_l5hlen = 0 '\000'}}, PH_per = {eight = "\000\000\000\000\000\000\000", sixteen = {0, 0, 0, 0}, thirtytwo = {0, 0}, sixtyfour = {0}, unintptr = {0}, ptr = 0x0}, {PH_loc = {eight = "\000\000\000\000\000\000\000", sixteen = {0, 0, 0, 0}, thirtytwo = {0, 0}, sixtyfour = {0}, unintptr = {0}, ptr = 0x0}, memlen = 0}}, {m_epg_npgs = 0 '\000', m_epg_nrdy = 144 '\220', m_epg_hdrlen = 68 'D', m_epg_trllen = 156 '\234', m_epg_1st_off = 63489, m_epg_last_len = 65535, m_epg_flags = 0 '\000', m_epg_record_type = 0 '\000', __spare = "\000", m_epg_enc_cnt = 0, m_epg_tls = 0x0, m_epg_so = 0x4ede3b9f0000024d, m_epg_seqno = 71776119061217280, m_epg_stailq = {stqe_next = 0x0}}}, {m_ext = {{ext_count = 4189736551, ext_cnt = 0x159a9294f9ba4e67}, ext_size = 6162, ext_type = 0, ext_flags = 0, {{ext_buf = 0x4000004d020045 <error: Cannot access memory at address 0x4000004d020045>, ext_arg2 = 0x70416ac86e7063f}, { extpg_pa = {18014399801458757, 505553988425811519, 13475177729205796113, 3215111323, 84116934427344}, extpg_trail = "\002\004\004\330\001\003\003\n\004\002\b\n,\300\016K\202\304q\023\250\325\001\000\000\001\000\000\000\000\000\000\006zabbix\004fbsd\002r\000\037\312$\364\033\230\267\205\037x\216\b\000E\000\005<", extpg_hdr = "0\337\000\000@\021\223ڬ\026\002\am\374\224\336\3229\005l\005(\266"}}, ext_free = 0x0, ext_arg1 = 0x0}, m_pktdat = 0xfffff8020f13a560 "gN\272\371\224\222\232\025\022\030"}}, m_dat = 0xfffff8020f13a520 ""}} (kgdb) p *m->m_next $9 = {{m_next = 0x0, m_slist = {sle_next = 0x0}, m_stailq = {stqe_next = 0x0}}, {m_nextpkt = 0x0, m_slistpkt = {sle_next = 0x0}, m_stailqpkt = {stqe_next = 0x0}}, m_data = 0xfffff80036e0d858 "\001\003\003\006\001\001\b\n\276\323\312i", m_len = 545, m_type = 1, m_flags = 1, {{{m_pkthdr = {{snd_tag = 0xfffff80004f5e800, rcvif = 0xfffff80004f5e800, {rcvidx = 59392, rcvgen = 1269}}, {leaf_rcvif = 0x0, {leaf_rcvidx = 0, leaf_rcvgen = 0}}, tags = {slh_first = 0x0}, len = 624, flowid = 1323187103, csum_flags = 251658240, fibnum = 0, numa_domain = 255 '\377', rsstype = 63 '?', {rcv_tstmp = 0, {l2hlen = 0 '\000', l3hlen = 0 '\000', l4hlen = 0 '\000', l5hlen = 0 '\000', inner_l2hlen = 0 '\000', inner_l3hlen = 0 '\000', inner_l4hlen = 0 '\000', inner_l5hlen = 0 '\000'}}, PH_per = {eight = "\000\000\000\000\377\377\000", sixteen = { 0, 0, 65535, 0}, thirtytwo = {0, 65535}, sixtyfour = {281470681743360}, unintptr = {281470681743360}, ptr = 0xffff00000000}, {PH_loc = {eight = "\000\000\000\000\000\000\000", sixteen = {0, 0, 0, 0}, thirtytwo = {0, 0}, sixtyfour = {0}, unintptr = {0}, ptr = 0x0}, memlen = 0}}, {m_epg_npgs = 0 '\000', m_epg_nrdy = 232 '\350', m_epg_hdrlen = 245 '\365', m_epg_trllen = 4 '\004', m_epg_1st_off = 63488, m_epg_last_len = 65535, m_epg_flags = 0 '\000', m_epg_record_type = 0 '\000', __spare = "\000", m_epg_enc_cnt = 0, m_epg_tls = 0x0, m_epg_so = 0x4ede3b9f00000270, m_epg_seqno = 4611404543702335488, m_epg_stailq = {stqe_next = 0x0}}}, {m_ext = {{ext_count = 1, ext_cnt = 0x1}, ext_size = 2048, ext_type = 6, ext_flags = 1, {{ext_buf = 0xfffff80036e0d800 "E\030\002\214-\276", ext_arg2 = 0x0}, {extpg_pa = { 18446735278537234432, 0, 0, 0, 0}, extpg_trail = '\000' <repeats 63 times>, extpg_hdr = '\000' <repeats 22 times>}}, ext_free = 0x0, ext_arg1 = 0x0}, m_pktdat = 0xfffff802220fbc60 "\001"}}, m_dat = 0xfffff802220fbc20 ""}}
(In reply to vova from comment #10) I was misled by the environment previously. Not only from WireGuard, this panic can also be triggered by a packet sent into epair. As shown in my screenshot, it's now from ether_input(), I assume that with a correctly crafted packet from normal Ethernet can also be effective, which exposes a great security problem.
(In reply to Wencey Wang from comment #12) can you confirm, if removing tcp-setmss ipfw rule fixes the issue or not? (in my case it fixes, but I do not do any nat with firewall)
Looking at my panic above: looks like fr 5 was in stack was broken backtrace shows: #5 0xffffffff834e666a in ipfwlog_clone_create (ifc=<optimized out>, unit=0, params=<optimized out>) at /usr/src/sys/netpfil/ipfw/ip_fw_bpf.c:134 in fact, it should be ipfw_check_packet and according to panic - it was ipfw_check_packet() at https://github.com/freebsd/freebsd-src/blob/087fc6ae0a8c/sys/netpfil/ipfw/ip_fw_pfil.c#L153 but in dump - this frame was overriten with bad data somehow
My completely uneducated guess, just looking at some of the related code, is that this[0] case in tcpmod_setmss is wrong. m_pullup failed so the original mbuf was freed, but we return a simple IP_FW_DENY that doesn't get interpreted as a terminal state up in the O_EXTERNAL_ACTION handler. Rule processing continues. [0] https://cgit.freebsd.org/src/tree/sys/netpfil/ipfw/pmod/tcpmod.c#n76
Created attachment 265018 [details] Proposed diff(1) Seeing if this helps would be a useful exercise
(In reply to Kyle Evans from comment #16) The patch looks good to me.
(In reply to Kyle Evans from comment #16) will try the patch, thanks technically, patch means - if, we have fragmented TCP syn packet, with no enough data received -> we will just drop the packet. should we have re-assemple functionality here? or better to offload it to firewall configuration with `reass` rule before?
(In reply to vova from comment #13) I have removed the tcp-setmss, no panic so far. I'll test the patch today.
(In reply to Kyle Evans from comment #16) This patch works in my case.
(In reply to Kyle Evans from comment #16) no panics so far also, will check more, as far as I had not that stable reproduction in my environment
(In reply to vova from comment #18) TCP sync and sync-ack shouldn't be large enough to be fragmented? And a normal TCP packet should already have the DF bit set.
(In reply to Wencey Wang from comment #22) yes ... unless it is TCP Fast Open, not sure if anybody uses it now
(In reply to vova from comment #23) dns/unbound has fastopen settable as an option
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=c0382512bfce872102d213b9bc2550de0bc30b67 commit c0382512bfce872102d213b9bc2550de0bc30b67 Author: Kyle Evans <kevans@FreeBSD.org> AuthorDate: 2025-11-01 17:34:11 +0000 Commit: Kyle Evans <kevans@FreeBSD.org> CommitDate: 2025-11-01 17:34:11 +0000 ipfw: pmod: avoid further rule processing after tcp-mod failures m_pullup() here will have freed the mbuf chain, but we pass back an IP_FW_DENY without any signal that the outer loop should finish. Thus, rule processing continues without an mbuf and there's a chance that we conclude that the packet may pass (but there's no mbuf remaining) depending on the rules that follow it. PR: 284606 Reviewed by: ae MFC after: 1 week sys/netpfil/ipfw/pmod/tcpmod.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-)
Tentatively not planning to MFC this to stable/13 or request an EN, but I'm admittedly not familiar enough with ipfw to gauge how disruptive/severe this is. I'm open to being convinced that it's worth an EN, since the patch is relatively trivial.
(In reply to Kyle Evans from comment #26) I think this can be easily used to perform DoS attacks with a fairly wide attack surface, since the only requirement is sending an IP packet to hosts with such a configuration. I discovered this issue in our production environment, as it's causing severe service outages. A security advisory should be effective at informing users. But also, this hasn't been reported before; maybe not that many people use such a feature?
(In reply to commit-hook from comment #25) by the way - I've found a way to reproduce the issue - fastopen is the way: on server VM (vtnet0 172.22.7.15/24): setup script (run in bash): ifconfig wg create name wg0 192.168.1.1/24 up wg set wg0 private-key <(echo "6G0V1B5MxxN8vBh3STwTmXIe/lelKL9mH6MgSoEAW3U=") \ listen-port 55555 peer obKySGvy/PhXVXTd8qmlVE8AIBvvMw9xHm778OG2QE0= \ allowed-ips 192.168.1.2/32 ifconfig lo1 create 192.168.1.11/32 up sysctl net.inet.ip.forwarding=1 sysctl net.inet.tcp.fastopen.server_enable=1 kldload ipfw kldload ipfw_pmod ipfw add tcp-setmss 1000 tcp from any to any via wg0 tcpflags syn ipfw add pass ip from any to any fetch https://people.freebsd.org/~pkelsey/tfo-tools/tfo-srv.c cc -o tfo-srv tfo-srv.c ./tfo-srv ------ on client VM (vtnet0 172.22.7.14/24): setup script (run in bash): ifconfig wg create name wg0 192.168.1.2/24 up wg set wg0 private-key <(echo "gCWPuTVuD2+YBkz3OnQycmq78KpnjVbyBOiEBpBEQXE=") \ peer oN8c0McyLVdBm/+u6jJAEUU71pOo4hHrnmDjlS+IaSs= \ allowed-ips 192.168.1.9/24 endpoint 172.22.7.15:55555 ping -c1 192.168.1.1 ifconfig lo1 create 172.22.1.12/32 up fetch https://people.freebsd.org/~pkelsey/tfo-tools/tfo-client.c cc -o tfo-client tfo-client.c --- trigger panic --- ./tfo-client -r 1000 -s 1000 192.168.1.11 22222 # that run ok ./tfo-client -r 1000 -s 1000 192.168.1.11 22222 # that panics on server ----- VMs should be able to reach each other, patch clearly prevents panic Kyle, thanks a lot
(In reply to Wencey Wang from comment #27) Oh, that's a fair point- this is in the rx path. It will be done-
On Sun Nov 2 00:48:12 2025 UTC, vova@fbsd.ru wrote: > (In reply to commit-hook from comment #25) > by the way - I've found a way to reproduce the issue - fastopen is the way: Making an ATF test case for tests/netpfil/ipfw would be great!
A commit in branch stable/15 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=21d55ae111aada3c5426632253ad8df9103d3423 commit 21d55ae111aada3c5426632253ad8df9103d3423 Author: Kyle Evans <kevans@FreeBSD.org> AuthorDate: 2025-11-01 17:34:11 +0000 Commit: Kyle Evans <kevans@FreeBSD.org> CommitDate: 2025-11-04 00:51:57 +0000 ipfw: pmod: avoid further rule processing after tcp-mod failures m_pullup() here will have freed the mbuf chain, but we pass back an IP_FW_DENY without any signal that the outer loop should finish. Thus, rule processing continues without an mbuf and there's a chance that we conclude that the packet may pass (but there's no mbuf remaining) depending on the rules that follow it. PR: 284606 Reviewed by: ae (cherry picked from commit c0382512bfce872102d213b9bc2550de0bc30b67) sys/netpfil/ipfw/pmod/tcpmod.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-)
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=deb684f9d1d6a3681e451d3af31f768c567f7dbe commit deb684f9d1d6a3681e451d3af31f768c567f7dbe Author: Kyle Evans <kevans@FreeBSD.org> AuthorDate: 2025-11-01 17:34:11 +0000 Commit: Kyle Evans <kevans@FreeBSD.org> CommitDate: 2025-11-04 00:52:54 +0000 ipfw: pmod: avoid further rule processing after tcp-mod failures m_pullup() here will have freed the mbuf chain, but we pass back an IP_FW_DENY without any signal that the outer loop should finish. Thus, rule processing continues without an mbuf and there's a chance that we conclude that the packet may pass (but there's no mbuf remaining) depending on the rules that follow it. PR: 284606 Reviewed by: ae (cherry picked from commit c0382512bfce872102d213b9bc2550de0bc30b67) sys/netpfil/ipfw/pmod/tcpmod.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=94360584542abc49a1d947844cf4d8ca1369d4df commit 94360584542abc49a1d947844cf4d8ca1369d4df Author: Kyle Evans <kevans@FreeBSD.org> AuthorDate: 2025-11-01 17:34:11 +0000 Commit: Kyle Evans <kevans@FreeBSD.org> CommitDate: 2025-11-04 00:52:12 +0000 ipfw: pmod: avoid further rule processing after tcp-mod failures m_pullup() here will have freed the mbuf chain, but we pass back an IP_FW_DENY without any signal that the outer loop should finish. Thus, rule processing continues without an mbuf and there's a chance that we conclude that the packet may pass (but there's no mbuf remaining) depending on the rules that follow it. PR: 284606 Reviewed by: ae (cherry picked from commit c0382512bfce872102d213b9bc2550de0bc30b67) sys/netpfil/ipfw/pmod/tcpmod.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-)
Created attachment 265160 [details] Proposed SA template Proposed SA (this could use some extra technical proofreading, for sure) -- over to secteam@
(In reply to Kyle Evans from comment #34) small proposal on SA: 1. "Module: ipfw" in fact it requires ipfw and ipfw_pmod (or custom kernel with built in ipfw_pmod) probably better to mention this, as all systems that has generic kernel and no ipfw_pmod loaded - unaffected 2. "No workaround is available" I would propose as w/a either - to remove tcp-setmss rule or - replace ipfw's "tcp-setmss" with pf's "scrub(max-mss ...)"
(In reply to vova from comment #35) I don't think anyone interprets that 'Module' to mean a specific kmod? My understanding is that it's only ever intended to be the general area affected -- the detail you're hoping for is communicated futher in the message where we describe which systems are not affected. I don't object to re-working the workaround section a little bit, but I hadn't really recommended pf's scrubbing because it's typically not straightforward to just kind of port your entire ruleset over to pf. I also hadn't really considered recommending removing tcp-setmss since it's noted that the directive processing is what causes the problem in the first place (so it seems obvious), but I don't feel that strongly about it.
(In reply to Kyle Evans from comment #36) up to you, just my 2c
A commit in branch releng/15.0 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=508f9b68379f2aae57444f67f2a0e971c338d305 commit 508f9b68379f2aae57444f67f2a0e971c338d305 Author: Kyle Evans <kevans@FreeBSD.org> AuthorDate: 2025-11-01 17:34:11 +0000 Commit: Colin Percival <cperciva@FreeBSD.org> CommitDate: 2025-11-05 19:36:38 +0000 ipfw: pmod: avoid further rule processing after tcp-mod failures m_pullup() here will have freed the mbuf chain, but we pass back an IP_FW_DENY without any signal that the outer loop should finish. Thus, rule processing continues without an mbuf and there's a chance that we conclude that the packet may pass (but there's no mbuf remaining) depending on the rules that follow it. Approved by: re (cperciva) PR: 284606 Reviewed by: ae (cherry picked from commit c0382512bfce872102d213b9bc2550de0bc30b67) (cherry picked from commit 21d55ae111aada3c5426632253ad8df9103d3423) sys/netpfil/ipfw/pmod/tcpmod.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-)
A commit in branch releng/14.3 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=c0cb68169bebc4210f9eccf643aeb73bf6d34654 commit c0cb68169bebc4210f9eccf643aeb73bf6d34654 Author: Kyle Evans <kevans@FreeBSD.org> AuthorDate: 2025-11-01 17:34:11 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2025-12-16 13:57:34 +0000 ipfw: pmod: avoid further rule processing after tcp-mod failures m_pullup() here will have freed the mbuf chain, but we pass back an IP_FW_DENY without any signal that the outer loop should finish. Thus, rule processing continues without an mbuf and there's a chance that we conclude that the packet may pass (but there's no mbuf remaining) depending on the rules that follow it. PR: 284606 Reviewed by: ae Approved by: so Security: FreeBSD-SA-25:11.ipfw Security: CVE-2025-14769 (cherry picked from commit c0382512bfce872102d213b9bc2550de0bc30b67) (cherry picked from commit deb684f9d1d6a3681e451d3af31f768c567f7dbe) sys/netpfil/ipfw/pmod/tcpmod.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-)
A commit in branch releng/13.5 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=60026b06366f0178e2473441aaf9e601061c07a4 commit 60026b06366f0178e2473441aaf9e601061c07a4 Author: Kyle Evans <kevans@FreeBSD.org> AuthorDate: 2025-11-01 17:34:11 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2025-12-16 01:55:38 +0000 ipfw: pmod: avoid further rule processing after tcp-mod failures m_pullup() here will have freed the mbuf chain, but we pass back an IP_FW_DENY without any signal that the outer loop should finish. Thus, rule processing continues without an mbuf and there's a chance that we conclude that the packet may pass (but there's no mbuf remaining) depending on the rules that follow it. PR: 284606 Reviewed by: ae Approved by: so Security: FreeBSD-SA-25:11.ipfw Security: CVE-2025-14769 (cherry picked from commit c0382512bfce872102d213b9bc2550de0bc30b67) (cherry picked from commit 94360584542abc49a1d947844cf4d8ca1369d4df) sys/netpfil/ipfw/pmod/tcpmod.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-)