Bug 50216

Summary: kernel panic on 5.0-current when use ipfw2 with dynamic rules
Product: Base System Reporter: Liu Kang <lazykang>
Component: kernAssignee: Andre Oppermann <andre>
Status: Closed FIXED    
Severity: Affects Only Me    
Priority: Normal    
Version: 5.0-CURRENT   
Hardware: Any   
OS: Any   

Description Liu Kang 2003-03-23 18:10:14 UTC
  I tried to use ipfw2 with dynamic rules by commands shown below:
ipfw add allow tcp from any to any established
ipfw add allow tcp from 192.168.0.0/16 to server_ip some_ports limit 
src-addr 20 setup
ipfw add allow udp from 192.168.0.0/16 to server_ip some_ports
ipfw add allow tcp from some_ip to server_ip some_ports limit src-addr 80 
setup
... and so on
The kernel will panic immediately while network connection is active. If I 
use static rules instead of those dynamic rules or disable network 
connection by use "ifconfig bge0 down", nothing happens.
I've add "options DDB" and some other debug options into my kernel configure 
file, I get the following message when kernel panic:
----Start of message---
Memory modified after free 0xc9471f0 (124)
panic: Most recently used by IpFw/IpAcct
cpuid=3;lapic.id=0300000
Stack bactrace:
backtrace(c0349879,3000000,C035a79a,e231fabe,1) at backtrace+0x17
panic(c035a79a,c03524d7,7c,c082ab64,c082ab40) at panic+0x10a
mtrash_ctor(c9471f00,80,0,54d,3) at mtrash_ctor+0x5d
uma_zalloc_arg(c082ab40,0,101,c034bc1,1be) at uma_zalloc_arg+0x17f
malloc(48,c037ef20,101,2700001b,e231fc70) at malloc+0xdc
add_dyn_rule(e231fc90,27,c9472180,c9472180,0)at add_dyn_rule+0x7b
install_state(c8fdd70,c8fdd764,e231fc70,e231fbf4,c01f1954)at 
install_state+0x1fd
ipfw_chk(e231fc70,c01cc3ad,c03a96a0,1,c0348bc1)at ipfw_chk+0x9a1
ip_input(c3b52000,0,c0351940,e9,c8b6a240)at ip_input+0x2c3
swi_net(0,0,c0347569,217,c3b18000)at swi_net+0x112
ithread_loop(c3b17080,e231fd48,c0347300,363,0)at ithread_loop+0x182
fork_exit(c01c3190,c3b17080,e231fd48)at fork_exit+0xc4
fork_trampoline()at fork_trampoline+0x1a
--trap 0x1,eip=0,esp=0xe231fd7c,ebp=0--
Debygger("panic")
Stopped at Debugger+0xff:xchgl %ebx,in_Debugger,0
----End of Message---
I copy the message above from screen,I'm not sure whether I've typed it 
exactly as displayed on the screen.I hope it is helpful.

Fix: 

Sorry, I can not give and patch now.
The only way I found to get rid of this problem is run ipfw2 with static 
rules instead of dynamic rules.

_________________________________________________________________
The new MSN 8: smart spam protection and 2 months FREE*  
http://join.msn.com/?page=features/junkmail
How-To-Repeat: use dynamic rules with ipfw2.
(I do not have a machine with only a single CPU to run freebsd-current, the 
problem might be related to SMP or HTT).
Comment 1 Liu Kang 2003-03-24 02:39:02 UTC
There is the same problem in stable-branch. I've tested ipfw2 with dynamic 
rules on the latest 4.8-RC,while ipfw(I mean ipfw-1) works on well with 
dynamic rule.

_________________________________________________________________
Add photos to your messages with MSN 8. Get 2 months FREE*. 
http://join.msn.com/?page=features/featuredemail
Comment 2 Coleman Kane freebsd_committer freebsd_triage 2003-03-24 17:52:04 UTC
I have experienced this problem as well, on three 5.0-R boxes (one an
alpha). It seems to manifest itself when using limit rules in the
firewall. I can successfully use keep-state/setup rules on all of
these machines. I have not had time to look into the source due to
work/school constraints. I figured I would put in my experience to
help out luigi or whomever else decides to hash this out.

--
coleman kane
Comment 3 Coleman Kane freebsd_committer freebsd_triage 2003-03-24 18:58:46 UTC
More info:
Using the following rule:
allow tcp from any to me port 25 limit src-addr 10

The system will panic as stated, though having multiple clients hammer
the smtp port will cause this panic in about 5 minutes or so. Only
connecting repeatedly from one client doesn't seem to have this effect.

On Mon, Mar 24, 2003 at 12:52:04PM -0500, Coleman Kane wrote:
> I have experienced this problem as well, on three 5.0-R boxes (one an
> alpha). It seems to manifest itself when using limit rules in the
> firewall. I can successfully use keep-state/setup rules on all of
> these machines. I have not had time to look into the source due to
> work/school constraints. I figured I would put in my experience to
> help out luigi or whomever else decides to hash this out.
> 
> --
> coleman kane
Comment 4 Maxim Konovalov 2003-03-24 19:10:24 UTC
On 11:00-0800, Mar 24, 2003, Coleman Kane wrote:

> The following reply was made to PR kern/50216; it has been noted by GNATS.
>
> From: Coleman Kane <cokane@FreeBSD.org>
> To: freebsd-gnats-submit@FreeBSD.org, lazyking@hotmail.com
> Cc:
> Subject: Re: kern/50216: kernel panic on 5.0-current when use ipfw2 with dynamic rules
> Date: Mon, 24 Mar 2003 13:58:46 -0500
>
>  More info:
>  Using the following rule:
>  allow tcp from any to me port 25 limit src-addr 10

Could you please show the whole ipfw ruleset?  Is your box UP or MP?
Can you obtain a kernel coredump?
Comment 5 Johan Karlsson freebsd_committer freebsd_triage 2003-05-06 20:35:46 UTC
Responsible Changed
From-To: freebsd-bugs->ipfw

Over to maintainer group.
Comment 6 Kang Liu 2003-09-30 17:47:17 UTC
I reproduced it on the latest 5.1current.

Here is the backtrace:
#####
GNU gdb 5.2.1 (FreeBSD)
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-undermydesk-freebsd"...
panic: Most recently used by cred

panic messages:
---
panic: Most recently used by cred

Stack backtrace:

syncing disks, buffers remaining... 628 628 628 628 628 628 628 628 628 628 628 628 628 628 628 628 628 628 628 628 
giving up on 520 buffers
Uptime: 16m0s
Dumping 255 MB
 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240
---
Reading symbols from /usr/obj/usr/src/sys/IPFW/modules/usr/src/sys/modules/acpi/acpi.ko.debug...done.
Loaded symbols for /usr/obj/usr/src/sys/IPFW/modules/usr/src/sys/modules/acpi/acpi.ko.debug
Reading symbols from /boot/kernel/daemon_saver.ko...done.
Loaded symbols for /boot/kernel/daemon_saver.ko
#0  doadump () at /usr/src/sys/kern/kern_shutdown.c:240
240		dumping++;
(kgdb) bt
#0  doadump () at /usr/src/sys/kern/kern_shutdown.c:240
#1  0xc01a0221 in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:372
#2  0xc01a05b7 in panic () at /usr/src/sys/kern/kern_shutdown.c:550
#3  0xc02817f7 in mtrash_ctor (mem=0xc29f8a80, size=0, arg=0x0)
    at /usr/src/sys/vm/uma_dbg.c:137
#4  0xc028002e in uma_zalloc_arg (zone=0xc083ab40, udata=0x0, flags=257)
    at /usr/src/sys/vm/uma_core.c:1413
#5  0xc0194a23 in malloc (size=3229854528, type=0xc03020c0, flags=257)
    at /usr/src/sys/vm/uma.h:234
#6  0xc021e03f in add_dyn_rule (id=0xcd7bfc90, dyn_type=39 '\'', 
    rule=0xc2815e00) at /usr/src/sys/netinet/ip_fw2.c:976
#7  0xc021e43e in install_state (rule=0xc28f3a80, cmd=0xc28f3ac0, 
    args=0xcd7bfc70) at /usr/src/sys/netinet/ip_fw2.c:1140
#8  0xc021f4dc in ipfw_chk (args=0xcd7bfc70)
    at /usr/src/sys/netinet/ip_fw2.c:1942
#9  0xc0221dd7 in ip_input (m=0xc0ed9800)
    at /usr/src/sys/netinet/ip_input.c:489
#10 0xc0211a82 in swi_net (dummy=0x0) at /usr/src/sys/net/netisr.c:236
#11 0xc018c762 in ithread_loop (arg=0xc0ebec80)
    at /usr/src/sys/kern/kern_intr.c:534
#12 0xc018b76f in fork_exit (callout=0xc018c5e0 <ithread_loop>, arg=0x0, 
    frame=0x0) at /usr/src/sys/kern/kern_fork.c:796
(kgdb) 

#####
Here is my full ipfw rule set script:
# cat ./ipfwpanic.sh 
dumpon -v /dev/ad0s1b
/sbin/ipfw add allow tcp from any to any established
/sbin/ipfw add allow ip from a.b.c.0 to a.b.c.d
/sbin/ipfw add allow tcp from any to a.b.c.d 80 limit src-addr 20 setup
/sbin/ipfw add allow ip from a.b.c.d to any

And I added "IPFIREWALL_DEFAULT_TO_ACCEPT" into kernel configure file.

#####
Here is my test script. I installed an apache on that machine, and use ab to connect 80 port.
cat panicstart.sh 
#!/bin/sh
number=0
while (test $number -lt 10000)
do
echo "$number"
ab -c 100 http://a.b.c.d/
number=`expr $number + 1`
done

#####
This problem can be reproduced on both MP and UP machine.
I've tested it on a dell poweredge2650(with 2 P4xeon, HTT enabled/disabled) and a DIY PC(1 PIII CPU).
The backtrace I post above is produced on PC(1CPU).
Comment 7 Sam Leffler 2003-09-30 18:28:19 UTC
On Tuesday 30 September 2003 09:47 am, Kang Liu wrote:
> I reproduced it on the latest 5.1current.

Thanks.  I'll look at it.

	Sam
Comment 8 Kang Liu 2003-10-29 15:24:12 UTC
it seems that the stable branch does not affected by this problem after
Mckusick's commit. :-) 
(src/sys/netinet/ip_fw2.c Revision 1.6.2.18)
testing 5.1current...
Comment 9 Andre Oppermann freebsd_committer freebsd_triage 2003-12-22 20:02:58 UTC
This bug has been fixed in -CURRENT netinet/ip_fw2.c rev. 1.40 too.

-- 
Andre
Comment 10 Andre Oppermann freebsd_committer freebsd_triage 2003-12-22 20:08:41 UTC
State Changed
From-To: open->closed

Revision 1.40 of netinet/ip_fw2.c fixes this for -CURRENT too. Closing case. 


Comment 11 Andre Oppermann freebsd_committer freebsd_triage 2003-12-22 20:08:41 UTC
Responsible Changed
From-To: ipfw->andre

Revision 1.40 of netinet/ip_fw2.c fixes this for -CURRENT too. Closing case.