I have an ipf rule that performs routing based on source address (for VPN purposes, using a pipsec tunnel) : block out log quick on tun0 to tun1:10.3.0.1 from VPN.IP.ADDR.ESS/32 to any group 11 (group 11 is the outbound group. When an outbound packet has the VPN tunnel interface address as its source, route it back through the VPN tunnel (tun1) instead of the default route (tun0)). This rule used to work as expected with ipfilter 3.4.16 under FreeBSD 4.3-STABLE. With 4.4-PRERELEASE (ipfilter 3.4.20), it freezes the machine. On one of my attempts, I obtained a kernel crash dump. One possible hypothesis is that ipfilter has corrupted an mbuf while moving the packet from one interface to another: Script started on Fri Aug 10 00:00:56 2001 $ gdb -k /usr/obj/usr/src/sys/MELUSINE/kernel.debug vmcore.5 GNU gdb 4.18 Copyright 1998 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-unknown-freebsd"... IdlePTD 4108288 initial pcb at 344dc0 panicstr: page fault panic messages: --- Fatal trap 12: page fault while in kernel mode fault virtual address = 0x6c2f6c71 fault code = supervisor read, page not present instruction pointer = 0x8:0xc018b2af stack pointer = 0x10:0xc8f4ad88 frame pointer = 0x10:0xc8f4ad98 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 518 (pipsecd) interrupt mask = net trap number = 12 panic: page fault syncing disks... 19 done Uptime: 26m53s dumping to dev #ad/0x20009, offset 270360 dump ata0: resetting devices .. done 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 [CTRL-C to abort] 42 [CTRL-C to abort] 41 [CTRL-C to abort] 40 [CTRL-C to abort] 39 [CTRL-C to abort] 38 [CTRL-C to abort] 37 [CTRL-C to abort] 36 [CTRL-C to abort] 35 34 [CTRL-C to abort] 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 --- #0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:472 472 if (dumping++) { (kgdb) bt #0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:472 #1 0xc016fbfd in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:312 #2 0xc016ffe5 in panic (fmt=0xc02f928c "%s") at /usr/src/sys/kern/kern_shutdown.c:580 #3 0xc02ae91b in trap_fatal (frame=0xc8f4ad48, eva=1815047281) at /usr/src/sys/i386/i386/trap.c:951 #4 0xc02ae5d5 in trap_pfault (frame=0xc8f4ad48, usermode=0, eva=1815047281) at /usr/src/sys/i386/i386/trap.c:844 #5 0xc02ae17f in trap (frame={tf_fs = 134479888, tf_es = -1065877488, tf_ds = -923533296, tf_edi = 6685184, tf_esi = 1815047265, tf_ebp = -923488872, tf_isp = -923488908, tf_ebx = -1065820160, tf_edx = 6685184, tf_ecx = -923488560, tf_eax = 6685184, tf_trapno = 12, tf_err = 0, tf_eip = -1072123217, tf_cs = 8, tf_eflags = 66054, tf_esp = -1065820160, tf_ss = -1065820160}) at /usr/src/sys/i386/i386/trap.c:443 #6 0xc018b2af in m_freem (m=0x6c2f6c61) at /usr/src/sys/kern/uipc_mbuf.c:618 #7 0xc018b2cd in m_freem (m=0xc0759700) at /usr/src/sys/kern/uipc_mbuf.c:618 #8 0xc01b5c0a in tunread (dev=0xc0cc1e80, uio=0xc8f4aed0, flag=8323072) at /usr/src/sys/net/if_tun.c:584 #9 0xc01a7e27 in spec_read (ap=0xc8f4ae5c) at /usr/src/sys/miscfs/specfs/spec_vnops.c:253 #10 0xc0242888 in ufsspec_read (ap=0xc8f4ae5c) at /usr/src/sys/ufs/ufs/ufs_vnops.c:1834 ---Type <return> to continue, or q <return> to quit--- #11 0xc0242e7d in ufs_vnoperatespec (ap=0xc8f4ae5c) at /usr/src/sys/ufs/ufs/ufs_vnops.c:2391 #12 0xc01a3daf in vn_read (fp=0xc0d67f80, uio=0xc8f4aed0, cred=0xc0731880, flags=0, p=0xc8edba40) at vnode_if.h:334 #13 0xc017e149 in dofileread (p=0xc8edba40, fp=0xc0d67f80, fd=9, buf=0x804e660, nbyte=4096, offset=-1, flags=0) at /usr/src/sys/sys/file.h:146 #14 0xc017e00a in read (p=0xc8edba40, uap=0xc8f4af80) at /usr/src/sys/kern/sys_generic.c:117 #15 0xc02aebba in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47, tf_edi = 134538848, tf_esi = 0, tf_ebp = -1077936800, tf_isp = -923488300, tf_ebx = 9, tf_edx = 134538496, tf_ecx = 134538532, tf_eax = 3, tf_trapno = 7, tf_err = 2, tf_eip = 672790868, tf_cs = 31, tf_eflags = 518, tf_esp = -1077937148, tf_ss = 47}) at /usr/src/sys/i386/i386/trap.c:1150 #16 0xc029fdd5 in Xint0x80_syscall () #17 0x80490ef in ?? () (kgdb) fr 8 #8 0xc01b5c0a in tunread (dev=0xc0cc1e80, uio=0xc8f4aed0, flag=8323072) at /usr/src/sys/net/if_tun.c:584 584 m_freem(m0); (kgdb) list 579 m0 = m; 580 } 581 582 if (m0) { 583 TUNDEBUG("%s%d: Dropping mbuf\n", ifp->if_name, ifp->if_unit); 584 m_freem(m0); 585 } 586 return error; 587 } 588 (kgdb) print ifp $1 = (struct ifnet *) 0xc0d5c308 (kgdb) print *ifp $2 = {if_softc = 0xc0d5c300, if_name = 0xc02d5d20 "tun", if_link = { tqe_next = 0x0, tqe_prev = 0xc0cfef10}, if_addrhead = { tqh_first = 0xc0d5c200, tqh_last = 0xc0d69c60}, if_pcount = 0, if_bpf = 0x0, if_index = 5, if_unit = 1, if_timer = 0, if_flags = -32687, if_ipending = 0, if_linkmib = 0x0, if_linkmiblen = 0, if_data = { ifi_type = 23 '\027', ifi_physical = 0 '\000', ifi_addrlen = 0 '\000', ifi_hdrlen = 0 '\000', ifi_recvquota = 0 '\000', ifi_xmitquota = 0 '\000', ifi_mtu = 1500, ifi_metric = 0, ifi_baudrate = 0, ifi_ipackets = 999, ifi_ierrors = 0, ifi_opackets = 1127, ifi_oerrors = 0, ifi_collisions = 0, ifi_ibytes = 139268, ifi_obytes = 1176565, ifi_imcasts = 0, ifi_omcasts = 0, ifi_iqdrops = 0, ifi_noproto = 0, ifi_hwassist = 0, ifi_unused = 0, ifi_lastchange = {tv_sec = 997392138, tv_usec = 502288}}, if_multiaddrs = {lh_first = 0xc0d54c20}, if_amcount = 0, if_output = 0xc01b53f8 <tunoutput>, if_start = 0, if_done = 0, if_ioctl = 0xc01b52a0 <tunifioctl>, if_watchdog = 0, if_poll_recv = 0, if_poll_xmit = 0, if_poll_intren = 0, if_poll_slowinput = 0, if_init = 0, if_resolvemulti = 0, if_snd = {ifq_head = 0x0, ifq_tail = 0x0, ifq_len = 0, ifq_maxlen = 50, ifq_drops = 0}, if_poll_slowq = 0x0, if_prefixhead = { tqh_first = 0x0, tqh_last = 0xc0d5c3d8}} (kgdb) fr 6 #6 0xc018b2af in m_freem (m=0x6c2f6c61) at /usr/src/sys/kern/uipc_mbuf.c:618 618 MFREE(m, n); Note: This was also reported as FreeBSD PR kern/ Fix: None known. How-To-Repeat: Create a rule set with a 'to' rule diverting packets from one tun interface to another tun interface [it is unknown whether this problem occurs with non-tun interfaces]. Trigger the rule by sending out a matching packet.
Responsible Changed From-To: freebsd-bugs->darrenr Darren is the ipf maintainer
Darren, I have just re-tested the problem described in FreeBSD PR kern/29583 ('to' redirection based on source IP address causes crash). On a 4.4-STABLE cvsupped today, with your 3.4.22 IPFilter release, I still obtained an instantaneous reboot as soon as a packet matching the rule went out of the machine. Thomas. -- Thomas.Quinot@Cuivre.FR.EU.ORG
In a previous email, I wrote: > I have just re-tested the problem described in FreeBSD PR kern/29583 > ('to' redirection based on source IP address causes crash). On a > 4.4-STABLE cvsupped today, with your 3.4.22 IPFilter release, I still > obtained an instantaneous reboot as soon as a packet matching the > rule went out of the machine. Tried 3.4.23 on 4.5-RC. There is definitely an improvement: the system does not crash anymore -- it hangs instead, stuck in a tight loop in the kernel. Fortunately, I was able to drop into DDB and make it panic, so we now have a clean crash dump, and an identification of the point where it is hanging. See attached typescript... Please let me know if there is anything I can do to help tracking this problem down. Script started on Tue Jan 15 22:59:18 2002 # gdb -k /usr/obj/usr/src/sys/MELUSINE/kernel.debug vmcore.0 GNU gdb 4.18 Copyright 1998 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-unknown-freebsd"... IdlePTD 4194304 initial pcb at 3555e0 panicstr: from debugger panic messages: --- panic: from debugger syncing disks... 16 done Uptime: 5m45s dumping to dev #ad/0x20009, offset 262168 dump ata0: resetting devices .. done [...] --- #0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:473 473 if (dumping++) { (kgdb) bt #0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:473 #1 0xc0174a81 in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:313 #2 0xc0174e71 in panic (fmt=0xc02d9ae4 "from debugger") at /usr/src/sys/kern/kern_shutdown.c:581 #3 0xc014aa31 in db_panic (addr=-1070927120, have_addr=0, count=-1, modif=0xd9c5bb78 "") at /usr/src/sys/ddb/db_command.c:435 #4 0xc014a9d0 in db_command (last_cmdp=0xc0315c24, cmd_table=0xc0315a64, aux_cmd_tablep=0xc034e058) at /usr/src/sys/ddb/db_command.c:333 #5 0xc014aa96 in db_command_loop () at /usr/src/sys/ddb/db_command.c:457 #6 0xc014cbc3 in db_trap (type=3, code=0) at /usr/src/sys/ddb/db_trap.c:71 #7 0xc02af09a in kdb_trap (type=3, code=0, regs=0xd9c5bc98) at /usr/src/sys/i386/i386/db_interface.c:158 #8 0xc02bbd94 in trap (frame={tf_fs = -641400816, tf_es = 16, tf_ds = 16, tf_edi = -1070175424, tf_esi = 0, tf_ebp = -641352480, tf_isp = -641352508, tf_ebx = 134, tf_edx = 4325376, tf_ecx = 32, tf_eax = 38, tf_trapno = 3, tf_err = 0, tf_eip = -1070927120, tf_cs = 8, tf_eflags = 582, tf_esp = -1070567809, tf_ss = -1070581751}) at /usr/src/sys/i386/i386/trap.c:574 #9 0xc02af2f0 in Debugger (msg=0xc0303809 "manual escape to debugger") at machine/cpufunc.h:67 #10 0xc02aabb6 in scgetc (sc=0xc036c540, flags=2) at /usr/src/sys/dev/syscons/syscons.c:3148 #11 0xc02a73c5 in sckbdevent (thiskbd=0xc0365240, event=0, arg=0xc036c540) at /usr/src/sys/dev/syscons/syscons.c:616 ---Type <return> to continue, or q <return> to quit--- #12 0xc029ef9e in atkbd_intr (kbd=0xc0365240, arg=0x0) at /usr/src/sys/dev/kbd/atkbd.c:462 #13 0xc02c9140 in atkbd_isa_intr (arg=0xc0365240) at /usr/src/sys/isa/atkbd_isa.c:140 #14 0xc0172e8e in add_interrupt_randomness (vsc=0xc036bdcc) at /usr/src/sys/kern/kern_random.c:245 #15 0xc01952b4 in sbappendaddr (sb=0xd7ccdf0c, asa=0xc031e020, m0=0xc1826500, control=0xc1826500) at /usr/src/sys/kern/uipc_socket2.c:657 #16 0xc019837e in uipc_send (so=0xd7ccd740, flags=0, m=0xc1826500, nam=0x0, control=0x0, p=0xd6ddac20) at /usr/src/sys/kern/uipc_usrreq.c:300 #17 0xc0192ce7 in sosend (so=0xd7ccd740, addr=0x0, uio=0xd9c5bec4, top=0xc1826500, control=0x0, flags=0, p=0xd6ddac20) at /usr/src/sys/kern/uipc_socket.c:611 #18 0xc019662d in sendit (p=0xd6ddac20, s=3, mp=0xd9c5bf04, flags=0) at /usr/src/sys/kern/uipc_syscalls.c:583 #19 0xc0196732 in sendto (p=0xd6ddac20, uap=0xd9c5bf80) at /usr/src/sys/kern/uipc_syscalls.c:636 #20 0xc02bc6ba in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47, tf_edi = -1077943328, tf_esi = 672154340, tf_ebp = -1077943412, tf_isp = -641351724, tf_ebx = 672333636, tf_edx = 97, tf_ecx = -1077942260, tf_eax = 133, tf_trapno = 22, tf_err = 2, tf_eip = 672036456, tf_cs = 31, tf_eflags = 647, tf_esp = -1077943472, tf_ss = 47}) at /usr/src/sys/i386/i386/trap.c:1157 #21 0xc02afe55 in Xint0x80_syscall () ---Type <return> to continue, or q <return> to quit--- #22 0x281046d7 in ?? () #23 0x28104351 in ?? () #24 0x80685da in ?? () #25 0x80665cb in ?? () #26 0x80668c9 in ?? () #27 0x8065eb9 in ?? () #28 0x8065986 in ?? () #29 0x806117f in ?? () #30 0x80515c9 in ?? () #31 0x805cf4f in ?? () #32 0x805cc28 in ?? () #33 0x8049a57 in ?? () (kgdb) fr 15 #15 0xc01952b4 in sbappendaddr (sb=0xd7ccdf0c, asa=0xc031e020, m0=0xc1826500, control=0xc1826500) at /usr/src/sys/kern/uipc_socket2.c:657 657 sballoc(sb, n); (kgdb) list 652 n->m_next = m0; /* concatenate data to control */ 653 else 654 control = m0; 655 m->m_next = control; 656 for (n = m; n; n = n->m_next) 657 sballoc(sb, n); 658 n = sb->sb_mb; 659 if (n) { 660 while (n->m_nextpkt) 661 n = n->m_nextpkt; (kgdb) print n $1 = (struct mbuf *) 0xc1826500 (kgdb) print n->m_hdr.mh_next $3 = (struct mbuf *) 0xc1826500 (kgdb) print *sb $5 = {sb_cc = 1983817059, sb_hiwat = 4096, sb_mbcnt = 1676293120, sb_mbmax = 32768, sb_lowat = 1, sb_mb = 0xc1853f00, sb_sel = {si_pid = 0, si_note = {slh_first = 0x0}, si_flags = 0}, sb_flags = 0, sb_timeo = 0} (kgdb) quit -- Thomas.Quinot@Cuivre.FR.EU.ORG
Guido van Rooij provided the following explanation for the cause of the crashes: ----- Forwarded message from Guido van Rooij <guido@gvr.org> ----- Date: Wed, 16 Jan 2002 16:18:14 +0100 From: Guido van Rooij <guido@gvr.org> Cc: ipfilter@coombs.anu.edu.au Subject: Re: Routing based on source address problem (with request to Darren) On Wed, Jan 16, 2002 at 03:40:35PM +0100, Thomas Quinot wrote: > All the details are in PR kern/29583, with the exact ipf rules involved > and a few debug traces: > http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/29583 > The most important part of PR is the following rule: block out log quick on tun0 to tun1:10.3.0.1 from VPN.IP.ADDR.ESS/32 to any group 11 Darren: I think IPfilter should forbid blocking of a packet using the 'to' directive. Explanation: "to" will ensure that the packet gets fastrouted. This means the mbuf will be queued at the specific interface queue to be sent out. If you block the packet, it will be freed (not sure if that is in ipfilter or in ip_input or whatever). This means we have a race: the freeing versus the use in the tun1 interface queue. I guess that is the problem here. [...] ----- End forwarded message ----- A work-around was to rewrite the aforementioned rule as: pass out log quick on tun0 to tun1:10.3.0.1 from VPN.IP.ADDR.ESS/32 to any group 11 which, contrary to what is written in the IPF-HOWTO, won't panic the machine. Thomas. -- Thomas.Quinot@Cuivre.FR.EU.ORG
State Changed From-To: open->feedback please test this with -stabel or -current as of today. many changes have been made which more than likely stop these panics (I hope!)
Le 2002-04-27, darrenr@FreeBSD.org écrivait : > please test this with -stabel or -current as of today. many changes > have been made which more than likely stop these panics (I hope!) Confirmed. IPF 3.4.27, MFC'd in FreeBSD 4-STABLE, does not exhibit this crash anymore. Thanks! -- Thomas.Quinot@Cuivre.FR.EU.ORG
State Changed From-To: feedback->closed fixed in both -stable and -current