My 11.0-CURRENT @ 285923 is panicing on boot due to a 0xdeadc0de dereference in pf_frag_tree_RB_FIND(). I had a 'scrub in all' and the panic is avoided by commenting it out. IPv6 is compiled in, but not configured aside from ::1 and a fe80:: link local address that isn't otherwise used. #8 0xffffffff807d71c3 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:235 #9 0xffffffff807228d4 in pf_frag_tree_RB_FIND (head=<value optimized out>, elm=0xfffffe066501e588) at /usr/src/sys/netpfil/pf/pf_norm.c:217 #10 0xffffffff80724d55 in pf_find_fragment (key=0xfffffe066501e588, tree=0xffffffff80d52a40) at /usr/src/sys/netpfil/pf/pf_norm.c:331 #11 0xffffffff80723a86 in pf_normalize_ip (m0=0xfffffe066501e7d8, dir=1, kif=0xfffff8000a106800, reason=0xfffffe066501e72e, pd=0xfffffe066501e698) at /usr/src/sys/netpfil/pf/pf_norm.c:1268 #12 0xffffffff807099f4 in pf_test (dir=1, ifp=<value optimized out>, m0=0xfffffe066501e7d8, inp=0x0) at /usr/src/sys/netpfil/pf/pf.c:5750 #13 0xffffffff8071ac9d in pf_check_in (arg=<value optimized out>, m=0xfffffe066501e7d8, ifp=0xfffffe066501e588, dir=<value optimized out>, inp=0x149) at /usr/src/sys/netpfil/pf/pf_ioctl.c:3555 #14 0xffffffff8068ac83 in pfil_run_hooks (ph=0xffffffff80e4f3e8, mp=0xfffffe066501e860, ifp=0xfffff8000850c800, dir=1, inp=0x0) at /usr/src/sys/net/pfil.c:83 #15 0xffffffff806b4439 in ip_input (m=0xffffffff063091d8) at /usr/src/sys/netinet/ip_input.c:523 #16 0xffffffff80689b06 in netisr_dispatch_src (proto=<value optimized out>, source=<value optimized out>, m=0xfffff80016a9bb00) at /usr/src/sys/net/netisr.c:972 It appears to be inside pf_frag_compare() as inlined into the RB lookup. The actual crash is a dereference of %rbx register, which had the value 0xdeadc0dedeadc0de. INVARIANTS was enabled so that means a use-after-free. Notable config fragments: options KDB #Enable the kernel debugger options DDB #Enable the kernel debugger options KDB_TRACE options INVARIANTS #Enable calls of extra sanity checking options INVARIANT_SUPPORT #Extra sanity checks of internal structures, required by INVARIANTS options ALT_BREAK_TO_DEBUGGER device pf device pflog options ALTQ options ALTQ_CBQ # Class Bases Queueing options ALTQ_RED # Random Early Drop options ALTQ_RIO # RED In/Out options ALTQ_HFSC # Hierarchical Packet Scheduler options ALTQ_CDNR # Traffic conditioner options ALTQ_PRIQ # Priority Queueing options NETGRAPH # netgraph(4) system options NETGRAPH_IFACE options NETGRAPH_KSOCKET options NETGRAPH_SOCKET The netgraph/altq stuff is a leftover of experiments years ago. It isn't in use, although it is present. General structure of pf.conf: ext_if="bge1" ext_ip4="XX.XX.XX.XX" set loginterface $ext_if set block-policy return set skip on lo # commented out to avoid panic #scrub in all nat on $ext_if from 127.0.1.0/24 to any -> $ext_ip4 block return in log on $ext_if all pass in on $ext_if inet proto tcp from any to $ext_ip4 port ssh pass in on $ext_if inet proto tcp from any to $ext_ip4 port imap pass in on $ext_if inet proto tcp from any to $ext_ip4 port imaps pass in on $ext_if inet proto icmp all icmp-type echoreq pass out on $ext_if inet all There are a number of jails on lo1 with their own 127.0.1.x address. It consistently died with the same crash, 100% repeatable. dmesg of the vmcore shows it died during jail startup. The host's unbound had started. It is configured for dnssec validation so it likely that large udp packets were in play.
Thanks for the report. This is very likely the same issue as bug #201879. A corruption of the fragment cache (a freed element not being removed from the list/tree). I've not yet been able to reproduce this myself, and I can't figure out how it might happen from simple code reading. Knowing that it's related to the v4 path is useful though.
(In reply to Kristof Provost from comment #1) Indeed. I hadn't made any headway on a cause in bug 201879. I had two variations on the panic depending on whether openntpd was the first service to talk outbound or whether it was a client on the LAN side browsing to the web. In both cases I have WAN side scrub rules and will see a panic.
I've managed to reproduce the panic, and I have a good suspicion of the cause (and why it took me so long to reproduce!). pf has two different modes for handling fragments, and two different lists to keep the fragments in. I think we get confused between the two at some point and end up iterating the wrong list. I've got a pending patch to remove the second mode, but hadn't gotten around to cleaning it up and committing it yet. With that second mode removed the panic is gone. Of course I'd been testing with that patch, so I never saw the panic. It's a bit of a big change for 10.2 though, so I'll see about getting a fix rather than the removal of the second mode.
(In reply to Kristof Provost from comment #3) I'll attempt to get you quick feedback on the end result when you post up a patch to test.
Created attachment 159347 [details] Fix It turns out I forgot to set the pf_fragment.fr_flags field to zero in pf_fillup_fragment(). As a result we would sometimes think the fragment was tracked in V_pf_cachequeue rather than V_pf_fragqueue. That ultimately resulted in the panic. Jason, I'd appreciate if if you could confirm this fixed the panic for you too before I commit this.
Created attachment 159349 [details] Fix v2 There was another issue which prevented the reassembly from working correctly. The panic was gone, but there was still a use-after-free.
(In reply to Kristof Provost from comment #6) Kristof, No more boot time panic for me. :) This was pretty clear cut and reproducible before. I rebooted a few times (4 to be precise) just for good measure and had no issues. For reference, here's my `uname -a`. The "fix v2" is the only only change from a fresh CURRENT. FreeBSD xju-rtr 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r285989M: Tue Jul 28 23:49:02 UTC 2015 root@xts-bsd:/usr/obj/nanobsd.soekris/i386.i386/usr/src/head/sys/GENERIC i386 Thanks!
A commit references this bug: Author: kp Date: Wed Jul 29 06:35:37 UTC 2015 New revision: 285999 URL: https://svnweb.freebsd.org/changeset/base/285999 Log: pf: Always initialise pf_fragment.fr_flags When we allocate the struct pf_fragment in pf_fillup_fragment() we forgot to initialise the fr_flags field. As a result we sometimes mistakenly thought the fragment to not be a buffered fragment. This resulted in panics because we'd end up freeing the pf_fragment but not removing it from V_pf_fragqueue (believing it to be part of V_pf_cachequeue). The next time we iterated V_pf_fragqueue we'd use a freed object and panic. While here also fix a pf_fragment use after free in pf_normalize_ip(). pf_reassemble() frees the pf_fragment, so we can't use it any more. PR: 201879, 201932 MFC after: 5 days Changes: head/sys/netpfil/pf/pf_norm.c