Bug 29583

Summary: 4.4-PREREL/ipf 3.4.20: 'to' rule with tun causes crash
Product: Base System Reporter: Thomas Quinot <thomas>
Component: kernAssignee: Darern Reed <darrenr>
Status: Closed FIXED    
Severity: Affects Only Me    
Priority: Normal    
Version: 4.4-PRERELEASE   
Hardware: Any   
OS: Any   

Description Thomas Quinot 2001-08-09 23:40:01 UTC
I have an ipf rule that performs routing based on source address
(for VPN purposes, using a pipsec tunnel) :

block out log quick on tun0 to tun1:10.3.0.1
  from VPN.IP.ADDR.ESS/32 to any group 11

(group 11 is the outbound group. When an outbound packet has the
VPN tunnel interface address as its source, route it back through
the VPN tunnel (tun1) instead of the default route (tun0)).

This rule used to work as expected with ipfilter 3.4.16 under
FreeBSD 4.3-STABLE. With 4.4-PRERELEASE (ipfilter 3.4.20), it
freezes the machine. On one of my attempts, I obtained a
kernel crash dump. One possible hypothesis is that ipfilter
has corrupted an mbuf while moving the packet from one
interface to another:

Script started on Fri Aug 10 00:00:56 2001
$ gdb -k /usr/obj/usr/src/sys/MELUSINE/kernel.debug vmcore.5
GNU gdb 4.18
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-unknown-freebsd"...

IdlePTD 4108288
initial pcb at 344dc0
panicstr: page fault
panic messages:
---
Fatal trap 12: page fault while in kernel mode
fault virtual address	= 0x6c2f6c71
fault code		= supervisor read, page not present
instruction pointer	= 0x8:0xc018b2af
stack pointer	        = 0x10:0xc8f4ad88
frame pointer	        = 0x10:0xc8f4ad98
code segment		= base rx0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, def32 1, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 518 (pipsecd)
interrupt mask		= net 
trap number		= 12
panic: page fault

syncing disks... 19 
done
Uptime: 26m53s

dumping to dev #ad/0x20009, offset 270360
dump ata0: resetting devices .. done
127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 [CTRL-C to abort] 42 [CTRL-C to abort] 41 [CTRL-C to abort] 40 [CTRL-C to abort] 39 [CTRL-C to abort] 38 [CTRL-C to abort] 37 [CTRL-C to abort] 36 [CTRL-C to abort] 35 34 [CTRL-C to abort] 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 
---
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:472
472		if (dumping++) {
(kgdb) bt
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:472
#1  0xc016fbfd in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:312
#2  0xc016ffe5 in panic (fmt=0xc02f928c "%s")
    at /usr/src/sys/kern/kern_shutdown.c:580
#3  0xc02ae91b in trap_fatal (frame=0xc8f4ad48, eva=1815047281)
    at /usr/src/sys/i386/i386/trap.c:951
#4  0xc02ae5d5 in trap_pfault (frame=0xc8f4ad48, usermode=0, eva=1815047281)
    at /usr/src/sys/i386/i386/trap.c:844
#5  0xc02ae17f in trap (frame={tf_fs = 134479888, tf_es = -1065877488, 
      tf_ds = -923533296, tf_edi = 6685184, tf_esi = 1815047265, 
      tf_ebp = -923488872, tf_isp = -923488908, tf_ebx = -1065820160, 
      tf_edx = 6685184, tf_ecx = -923488560, tf_eax = 6685184, tf_trapno = 12, 
      tf_err = 0, tf_eip = -1072123217, tf_cs = 8, tf_eflags = 66054, 
      tf_esp = -1065820160, tf_ss = -1065820160})
    at /usr/src/sys/i386/i386/trap.c:443
#6  0xc018b2af in m_freem (m=0x6c2f6c61) at /usr/src/sys/kern/uipc_mbuf.c:618
#7  0xc018b2cd in m_freem (m=0xc0759700) at /usr/src/sys/kern/uipc_mbuf.c:618
#8  0xc01b5c0a in tunread (dev=0xc0cc1e80, uio=0xc8f4aed0, flag=8323072)
    at /usr/src/sys/net/if_tun.c:584
#9  0xc01a7e27 in spec_read (ap=0xc8f4ae5c)
    at /usr/src/sys/miscfs/specfs/spec_vnops.c:253
#10 0xc0242888 in ufsspec_read (ap=0xc8f4ae5c)
    at /usr/src/sys/ufs/ufs/ufs_vnops.c:1834
---Type <return> to continue, or q <return> to quit--- 
#11 0xc0242e7d in ufs_vnoperatespec (ap=0xc8f4ae5c)
    at /usr/src/sys/ufs/ufs/ufs_vnops.c:2391
#12 0xc01a3daf in vn_read (fp=0xc0d67f80, uio=0xc8f4aed0, cred=0xc0731880, 
    flags=0, p=0xc8edba40) at vnode_if.h:334
#13 0xc017e149 in dofileread (p=0xc8edba40, fp=0xc0d67f80, fd=9, 
    buf=0x804e660, nbyte=4096, offset=-1, flags=0)
    at /usr/src/sys/sys/file.h:146
#14 0xc017e00a in read (p=0xc8edba40, uap=0xc8f4af80)
    at /usr/src/sys/kern/sys_generic.c:117
#15 0xc02aebba in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47, 
      tf_edi = 134538848, tf_esi = 0, tf_ebp = -1077936800, 
      tf_isp = -923488300, tf_ebx = 9, tf_edx = 134538496, tf_ecx = 134538532, 
      tf_eax = 3, tf_trapno = 7, tf_err = 2, tf_eip = 672790868, tf_cs = 31, 
      tf_eflags = 518, tf_esp = -1077937148, tf_ss = 47})
    at /usr/src/sys/i386/i386/trap.c:1150
#16 0xc029fdd5 in Xint0x80_syscall ()
#17 0x80490ef in ?? ()
(kgdb) fr 8
#8  0xc01b5c0a in tunread (dev=0xc0cc1e80, uio=0xc8f4aed0, flag=8323072)
    at /usr/src/sys/net/if_tun.c:584
584			m_freem(m0);
(kgdb) list
579			m0 = m;
580		}
581	
582		if (m0) {
583			TUNDEBUG("%s%d: Dropping mbuf\n", ifp->if_name, ifp->if_unit);
584			m_freem(m0);
585		}
586		return error;
587	}
588	
(kgdb) print ifp
$1 = (struct ifnet *) 0xc0d5c308
(kgdb) print *ifp
$2 = {if_softc = 0xc0d5c300, if_name = 0xc02d5d20 "tun", if_link = {
    tqe_next = 0x0, tqe_prev = 0xc0cfef10}, if_addrhead = {
    tqh_first = 0xc0d5c200, tqh_last = 0xc0d69c60}, if_pcount = 0, 
  if_bpf = 0x0, if_index = 5, if_unit = 1, if_timer = 0, if_flags = -32687, 
  if_ipending = 0, if_linkmib = 0x0, if_linkmiblen = 0, if_data = {
    ifi_type = 23 '\027', ifi_physical = 0 '\000', ifi_addrlen = 0 '\000', 
    ifi_hdrlen = 0 '\000', ifi_recvquota = 0 '\000', ifi_xmitquota = 0 '\000', 
    ifi_mtu = 1500, ifi_metric = 0, ifi_baudrate = 0, ifi_ipackets = 999, 
    ifi_ierrors = 0, ifi_opackets = 1127, ifi_oerrors = 0, ifi_collisions = 0, 
    ifi_ibytes = 139268, ifi_obytes = 1176565, ifi_imcasts = 0, 
    ifi_omcasts = 0, ifi_iqdrops = 0, ifi_noproto = 0, ifi_hwassist = 0, 
    ifi_unused = 0, ifi_lastchange = {tv_sec = 997392138, tv_usec = 502288}}, 
  if_multiaddrs = {lh_first = 0xc0d54c20}, if_amcount = 0, 
  if_output = 0xc01b53f8 <tunoutput>, if_start = 0, if_done = 0, 
  if_ioctl = 0xc01b52a0 <tunifioctl>, if_watchdog = 0, if_poll_recv = 0, 
  if_poll_xmit = 0, if_poll_intren = 0, if_poll_slowinput = 0, if_init = 0, 
  if_resolvemulti = 0, if_snd = {ifq_head = 0x0, ifq_tail = 0x0, ifq_len = 0, 
    ifq_maxlen = 50, ifq_drops = 0}, if_poll_slowq = 0x0, if_prefixhead = {
    tqh_first = 0x0, tqh_last = 0xc0d5c3d8}}
(kgdb) fr 6
#6  0xc018b2af in m_freem (m=0x6c2f6c61) at /usr/src/sys/kern/uipc_mbuf.c:618
618			MFREE(m, n);

Note: This was also reported as FreeBSD PR kern/

Fix: 

None known.
How-To-Repeat: 
        Create a rule set with a 'to' rule diverting packets from one tun
        interface to another tun interface [it is unknown whether this
        problem occurs with non-tun interfaces].

        Trigger the rule by sending out a matching packet.
Comment 1 Kris Kennaway freebsd_committer freebsd_triage 2001-08-10 01:07:05 UTC
Responsible Changed
From-To: freebsd-bugs->darrenr

Darren is the ipf maintainer
Comment 2 Thomas Quinot 2001-12-07 00:15:36 UTC
Darren,

I have just re-tested the problem described in FreeBSD PR kern/29583
('to' redirection based on source IP address causes crash). On a
4.4-STABLE cvsupped today, with your 3.4.22 IPFilter release, I still
obtained an instantaneous reboot as soon as a packet matching the
rule went out of the machine.

Thomas.

-- 
    Thomas.Quinot@Cuivre.FR.EU.ORG
Comment 3 Thomas Quinot 2002-01-15 22:11:41 UTC
In a previous email, I wrote:

> I have just re-tested the problem described in FreeBSD PR kern/29583
> ('to' redirection based on source IP address causes crash). On a
> 4.4-STABLE cvsupped today, with your 3.4.22 IPFilter release, I still
> obtained an instantaneous reboot as soon as a packet matching the
> rule went out of the machine.

Tried 3.4.23 on 4.5-RC. There is definitely an improvement: the system
does not crash anymore -- it hangs instead, stuck in a tight loop in
the kernel. Fortunately, I was able to drop into DDB and make it panic,
so we now have a clean crash dump, and an identification of the point
where it is hanging. See attached typescript...

Please let me know if there is anything I can do to help tracking this
problem down.

Script started on Tue Jan 15 22:59:18 2002
# gdb -k /usr/obj/usr/src/sys/MELUSINE/kernel.debug vmcore.0
GNU gdb 4.18
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-unknown-freebsd"...
IdlePTD 4194304
initial pcb at 3555e0
panicstr: from debugger
panic messages:
---
panic: from debugger

syncing disks... 16 
done
Uptime: 5m45s

dumping to dev #ad/0x20009, offset 262168
dump ata0: resetting devices .. done
[...]
---
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:473
473		if (dumping++) {
(kgdb) bt
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:473
#1  0xc0174a81 in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:313
#2  0xc0174e71 in panic (fmt=0xc02d9ae4 "from debugger")
    at /usr/src/sys/kern/kern_shutdown.c:581
#3  0xc014aa31 in db_panic (addr=-1070927120, have_addr=0, count=-1, 
    modif=0xd9c5bb78 "") at /usr/src/sys/ddb/db_command.c:435
#4  0xc014a9d0 in db_command (last_cmdp=0xc0315c24, cmd_table=0xc0315a64, 
    aux_cmd_tablep=0xc034e058) at /usr/src/sys/ddb/db_command.c:333
#5  0xc014aa96 in db_command_loop () at /usr/src/sys/ddb/db_command.c:457
#6  0xc014cbc3 in db_trap (type=3, code=0) at /usr/src/sys/ddb/db_trap.c:71
#7  0xc02af09a in kdb_trap (type=3, code=0, regs=0xd9c5bc98)
    at /usr/src/sys/i386/i386/db_interface.c:158

#8  0xc02bbd94 in trap (frame={tf_fs = -641400816, tf_es = 16, tf_ds = 16, 
      tf_edi = -1070175424, tf_esi = 0, tf_ebp = -641352480, 
      tf_isp = -641352508, tf_ebx = 134, tf_edx = 4325376, tf_ecx = 32, 
      tf_eax = 38, tf_trapno = 3, tf_err = 0, tf_eip = -1070927120, tf_cs = 8, 
      tf_eflags = 582, tf_esp = -1070567809, tf_ss = -1070581751})
    at /usr/src/sys/i386/i386/trap.c:574
#9  0xc02af2f0 in Debugger (msg=0xc0303809 "manual escape to debugger")
    at machine/cpufunc.h:67
#10 0xc02aabb6 in scgetc (sc=0xc036c540, flags=2)
    at /usr/src/sys/dev/syscons/syscons.c:3148
#11 0xc02a73c5 in sckbdevent (thiskbd=0xc0365240, event=0, arg=0xc036c540)
    at /usr/src/sys/dev/syscons/syscons.c:616
---Type <return> to continue, or q <return> to quit---
#12 0xc029ef9e in atkbd_intr (kbd=0xc0365240, arg=0x0)
    at /usr/src/sys/dev/kbd/atkbd.c:462
#13 0xc02c9140 in atkbd_isa_intr (arg=0xc0365240)
    at /usr/src/sys/isa/atkbd_isa.c:140
#14 0xc0172e8e in add_interrupt_randomness (vsc=0xc036bdcc)
    at /usr/src/sys/kern/kern_random.c:245
#15 0xc01952b4 in sbappendaddr (sb=0xd7ccdf0c, asa=0xc031e020, m0=0xc1826500, 
    control=0xc1826500) at /usr/src/sys/kern/uipc_socket2.c:657
#16 0xc019837e in uipc_send (so=0xd7ccd740, flags=0, m=0xc1826500, nam=0x0, 
    control=0x0, p=0xd6ddac20) at /usr/src/sys/kern/uipc_usrreq.c:300
#17 0xc0192ce7 in sosend (so=0xd7ccd740, addr=0x0, uio=0xd9c5bec4, 
    top=0xc1826500, control=0x0, flags=0, p=0xd6ddac20)
    at /usr/src/sys/kern/uipc_socket.c:611
#18 0xc019662d in sendit (p=0xd6ddac20, s=3, mp=0xd9c5bf04, flags=0)
    at /usr/src/sys/kern/uipc_syscalls.c:583
#19 0xc0196732 in sendto (p=0xd6ddac20, uap=0xd9c5bf80)
    at /usr/src/sys/kern/uipc_syscalls.c:636
#20 0xc02bc6ba in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47, 
      tf_edi = -1077943328, tf_esi = 672154340, tf_ebp = -1077943412, 
      tf_isp = -641351724, tf_ebx = 672333636, tf_edx = 97, 
      tf_ecx = -1077942260, tf_eax = 133, tf_trapno = 22, tf_err = 2, 
      tf_eip = 672036456, tf_cs = 31, tf_eflags = 647, tf_esp = -1077943472, 
      tf_ss = 47}) at /usr/src/sys/i386/i386/trap.c:1157
#21 0xc02afe55 in Xint0x80_syscall ()
---Type <return> to continue, or q <return> to quit---
#22 0x281046d7 in ?? ()
#23 0x28104351 in ?? ()
#24 0x80685da in ?? ()
#25 0x80665cb in ?? ()
#26 0x80668c9 in ?? ()
#27 0x8065eb9 in ?? ()
#28 0x8065986 in ?? ()
#29 0x806117f in ?? ()
#30 0x80515c9 in ?? ()
#31 0x805cf4f in ?? ()
#32 0x805cc28 in ?? ()
#33 0x8049a57 in ?? ()
(kgdb) fr 15
#15 0xc01952b4 in sbappendaddr (sb=0xd7ccdf0c, asa=0xc031e020, m0=0xc1826500, 
    control=0xc1826500) at /usr/src/sys/kern/uipc_socket2.c:657
657			sballoc(sb, n);
(kgdb) list
652			n->m_next = m0;		/* concatenate data to control */
653		else
654			control = m0;
655		m->m_next = control;
656		for (n = m; n; n = n->m_next)
657			sballoc(sb, n);
658		n = sb->sb_mb;
659		if (n) {
660			while (n->m_nextpkt)
661				n = n->m_nextpkt;
(kgdb) print n
$1 = (struct mbuf *) 0xc1826500
(kgdb) print n->m_hdr.mh_next
$3 = (struct mbuf *) 0xc1826500
(kgdb) print *sb
$5 = {sb_cc = 1983817059, sb_hiwat = 4096, sb_mbcnt = 1676293120, 
  sb_mbmax = 32768, sb_lowat = 1, sb_mb = 0xc1853f00, sb_sel = {si_pid = 0, 
    si_note = {slh_first = 0x0}, si_flags = 0}, sb_flags = 0, sb_timeo = 0}
(kgdb) quit

-- 
    Thomas.Quinot@Cuivre.FR.EU.ORG
Comment 4 Thomas Quinot 2002-01-22 22:43:51 UTC
Guido van Rooij provided the following explanation for the cause
of the crashes:

----- Forwarded message from Guido van Rooij <guido@gvr.org> -----

Date: Wed, 16 Jan 2002 16:18:14 +0100
From: Guido van Rooij <guido@gvr.org>
Cc: ipfilter@coombs.anu.edu.au
Subject: Re: Routing based on source address problem (with request to Darren)

On Wed, Jan 16, 2002 at 03:40:35PM +0100, Thomas Quinot wrote:
> All the details are in PR kern/29583, with the exact ipf rules involved
> and a few debug traces:
>   http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/29583
> 

The most important part of PR is the following rule:

block out log quick on tun0 to tun1:10.3.0.1
  from VPN.IP.ADDR.ESS/32 to any group 11

Darren: I think IPfilter should forbid blocking of a packet using
the 'to' directive.

Explanation:

"to" will ensure that the packet gets fastrouted. This means the mbuf will
be queued at the specific interface queue to be sent out.
If you block the packet, it will be freed (not sure if that is
in ipfilter or in ip_input or whatever).
This means we have a race: the freeing versus the use in the tun1
interface queue.

I guess that is the problem here.

[...]

----- End forwarded message -----

A work-around was to rewrite the aforementioned rule as:
  pass out log quick on tun0 to tun1:10.3.0.1
    from VPN.IP.ADDR.ESS/32 to any group 11
which, contrary to what is written in the IPF-HOWTO,
won't panic the machine.

Thomas.

-- 
    Thomas.Quinot@Cuivre.FR.EU.ORG
Comment 5 Darern Reed freebsd_committer freebsd_triage 2002-04-27 21:05:30 UTC
State Changed
From-To: open->feedback

please test this with -stabel or -current as of today. many changes 
have been made which more than likely stop these panics (I hope!)
Comment 6 Thomas Quinot 2002-05-02 23:00:20 UTC
Le 2002-04-27, darrenr@FreeBSD.org écrivait :

> please test this with -stabel or -current as of today. many changes
> have been made which more than likely stop these panics (I hope!)

Confirmed. IPF 3.4.27, MFC'd in FreeBSD 4-STABLE, does not exhibit
this crash anymore. Thanks!

-- 
    Thomas.Quinot@Cuivre.FR.EU.ORG
Comment 7 Darern Reed freebsd_committer freebsd_triage 2002-05-02 23:38:27 UTC
State Changed
From-To: feedback->closed

fixed in both -stable and -current