166937 – [panic] Random and frequent kernel crash, reason unknown

Bug 166937 - [panic] Random and frequent kernel crash, reason unknown

Summary: [panic] Random and frequent kernel crash, reason unknown

Status:	Closed FIXED

Alias:	None

Product:	Base System
Classification:	Unclassified
Component:	kern (show other bugs)
Version:	8.2-STABLE
Hardware:	Any Any

Importance:	Normal Affects Only Me
Assignee:	freebsd-bugs (Nobody)

URL:
Keywords:

Depends on:
Blocks:

Reported:	2012-04-14 07:20 UTC by Dennis Yusupoff
Modified:	2019-09-23 17:00 UTC (History)
CC List:	1 user (show)

See Also:	238017

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.

Description Dennis Yusupoff 2012-04-14 07:20:07 UTC

Our server start frequently and randomly crashing and we don't know why. :(

That server do billing (netflow, apache, mysql) and gateway (nat, policy-based routing with 2 WANs, ipfw, dummynet) for relatively small network (summary traffic up to 100 Mbit/sec)
Part of crash.txt.* info:

-------------------------------------------------------------------------
Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer	= 0x20:0xffffffff8032b15f
stack pointer	        = 0x28:0xffffff80ed216a50
frame pointer	        = 0x28:0xffffff80ed216aa0
code segment		= base rx0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 0 (dummynet)
trap number		= 9
panic: general protection fault
cpuid = 0
Uptime: 8d1h40m32s
Dumping 667 out of 3318 MB:..3%..12%..22%..32%..41%..51%..63%..72%..82%..92%

[...skipped....]

#0  doadump () at pcpu.h:224
224	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) #0  doadump () at pcpu.h:224
#1  0xffffffff802629f0 in boot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:441
#2  0xffffffff80262e73 in panic (fmt=0x0)
    at /usr/src/sys/kern/kern_shutdown.c:614
#3  0xffffffff803dd08d in trap_fatal (frame=0xffffffff805d2840, eva=Variable "eva" is not available.
)
    at /usr/src/sys/amd64/amd64/trap.c:825
#4  0xffffffff803dd54a in trap (frame=0xffffff80ed2169a0)
    at /usr/src/sys/amd64/amd64/trap.c:621
#5  0xffffffff803c4f14 in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:228
#6  0xffffffff8032b15f in dn_ht_scan_bucket (ht=0xffffff00069a3300, bucket=Variable "bucket" is not available.
)
    at /usr/src/sys/netinet/ipfw/dn_heap.c:537
#7  0xffffffff8032e4e3 in drain_scheduler_sch_cb (_s=Variable "_s" is not available.
)
    at /usr/src/sys/netinet/ipfw/ip_dummynet.c:1985
#8  0xffffffff8032b16a in dn_ht_scan_bucket (ht=0xffffff0001d1f400, bucket=Variable "bucket" is not available.
)
    at /usr/src/sys/netinet/ipfw/dn_heap.c:537
#9  0xffffffff8032e230 in dn_drain_scheduler ()
    at /usr/src/sys/netinet/ipfw/ip_dummynet.c:2001
#10 0xffffffff80331a25 in dummynet_task (context=Variable "context" is not available.
)
    at /usr/src/sys/netinet/ipfw/ip_dn_io.c:608
#11 0xffffffff8029f325 in taskqueue_run_locked (queue=0xffffff0001d1aa80)
    at /usr/src/sys/kern/subr_taskqueue.c:250
#12 0xffffffff8029f4be in taskqueue_thread_loop (arg=Variable "arg" is not available.
)
    at /usr/src/sys/kern/subr_taskqueue.c:387
#13 0xffffffff80239cdf in fork_exit (
    callout=0xffffffff8029f470 <taskqueue_thread_loop>, 
    arg=0xffffffff805fc080, frame=0xffffff80ed216c50)
    at /usr/src/sys/kern/kern_fork.c:876
#14 0xffffffff803c545e in fork_trampoline ()
    at /usr/src/sys/amd64/amd64/exception.S:602
#15 0x0000000000000000 in ?? ()
#16 0x0000000000000000 in ?? ()
#17 0x0000000000000000 in ?? ()
#18 0x0000000000000000 in ?? ()
#19 0x0000000000000000 in ?? ()
#20 0x0000000000000000 in ?? ()
#21 0x0000000000000000 in ?? ()
#22 0x0000000000000000 in ?? ()
#23 0x0000000000000000 in ?? ()
#24 0x0000000000000000 in ?? ()
#25 0x0000000000000000 in ?? ()
#26 0x0000000000000000 in ?? ()
#27 0x0000000000000000 in ?? ()
#28 0x0000000000000000 in ?? ()
#29 0x0000000000000000 in ?? ()
#30 0x0000000000000000 in ?? ()
#31 0x0000000000000000 in ?? ()
#32 0x0000000000000000 in ?? ()
#33 0x0000000000000000 in ?? ()
#34 0x0000000000000000 in ?? ()
#35 0x0000000000000000 in ?? ()
#36 0x0000000000000000 in ?? ()
#37 0x0000000000000000 in ?? ()
#38 0x0000000000000000 in ?? ()
#39 0xffffffff805f7038 in sleepq_chains ()
#40 0xffffff0001c87430 in ?? ()
#41 0x0000000000000000 in ?? ()
#42 0xffffff0001c87000 in ?? ()
#43 0xffffff80ed216b00 in ?? ()
#44 0xffffff80ed216aa8 in ?? ()
#45 0xffffff0001ff88c0 in ?? ()
#46 0xffffffff802879f2 in sched_switch (td=0xffffffff8029f470, 
    newtd=0xffffffff805fc080, flags=Variable "flags" is not available.
) at /usr/src/sys/kern/sched_ule.c:1861
Previous frame inner to this frame (corrupt stack?)
(kgdb) 
-------------------------------------------------------------------------



Shortly after change was made in net.isr.direct 1 -> 0:
-------------------------------------------------------------------------
Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer	= 0x20:0xffffffff8081b13e
stack pointer	        = 0x28:0xffffff80000484c0
frame pointer	        = 0x28:0xffffff8000048520
code segment		= base rx0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 12 (swi1: netisr 0)
trap number		= 9
panic: general protection fault
cpuid = 0
Uptime: 1d1h50m1s

..
#0  doadump () at pcpu.h:224
224	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) #0  doadump () at pcpu.h:224
#1  0xffffffff802629f0 in boot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:441
#2  0xffffffff80262e73 in panic (fmt=0x0)
    at /usr/src/sys/kern/kern_shutdown.c:614
#3  0xffffffff803dd08d in trap_fatal (frame=0xffffff0001855000, eva=Variable "eva" is not available.
)
    at /usr/src/sys/amd64/amd64/trap.c:825
#4  0xffffffff803dd54a in trap (frame=0xffffff8000048410)
    at /usr/src/sys/amd64/amd64/trap.c:621
#5  0xffffffff803c4f14 in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:228
#6  0xffffffff8081b13e in _FindLinkIn (la=0xffffff800096e000, dst_addr=
      {s_addr = 406912350}, alias_addr={s_addr = 4224137940}, dst_port=8645, 
    alias_port=57626, link_type=6, replace_partial_links=1)
    at /usr/src/sys/modules/libalias/libalias/../../../netinet/libalias/alias_db.c:1215
#7  0xffffffff8081b314 in FindLinkIn (la=0xffffff800096e000, dst_addr=Variable "dst_addr" is not available.
)
    at /usr/src/sys/modules/libalias/libalias/../../../netinet/libalias/alias_db.c:1304
#8  0xffffffff8081b56b in FindUdpTcpIn (la=0xffffff800096e000, dst_addr=Variable "dst_addr" is not available.
)
    at /usr/src/sys/modules/libalias/libalias/../../../netinet/libalias/alias_db.c:1522
#9  0xffffffff80818886 in LibAliasInLocked (la=0xffffff800096e000, 
    ptr=0xffffff005e3af810 "E ", maxpacketsize=2032)
    at /usr/src/sys/modules/libalias/libalias/../../../netinet/libalias/alias.c:924
#10 0xffffffff8081934d in LibAliasIn (la=0xffffff800096e000, 
    ptr=0xffffff005e3af810 "E ", maxpacketsize=2032)
    at /usr/src/sys/modules/libalias/libalias/../../../netinet/libalias/alias.c:1302
#11 0xffffffff808303b1 in ng_nat_rcvdata (hook=0xffffff0001dcde00, 
    item=0xffffff0049feed00)
    at /usr/src/sys/modules/netgraph/nat/../../../netgraph/ng_nat.c:712
#12 0xffffffff80825bb0 in ng_apply_item (node=0xffffff004c107300, 
    item=0xffffff0049feed00, rw=1)
    at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2327
#13 0xffffffff80824c7e in ng_snd_item (item=Variable "item" is not available.
)
    at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2244
#14 0xffffffff80339d07 in ipfw_check_hook (arg=Variable "arg" is not available.
)
    at /usr/src/sys/netinet/ipfw/ip_fw_pfil.c:225
#15 0xffffffff8031192c in pfil_run_hooks (ph=Variable "ph" is not available.
) at /usr/src/sys/net/pfil.c:82
#16 0xffffffff8033f07e in ip_input (m=0xffffff003f63f300)
    at /usr/src/sys/netinet/ip_input.c:532
#17 0xffffffff803112d9 in swi_net (arg=Variable "arg" is not available.
) at /usr/src/sys/net/netisr.c:653
#18 0xffffffff8023cab4 in intr_event_execute_handlers (p=Variable "p" is not available.
)
    at /usr/src/sys/kern/kern_intr.c:1216
#19 0xffffffff8023e145 in ithread_loop (arg=0xffffff00018537a0)
    at /usr/src/sys/kern/kern_intr.c:1229
#20 0xffffffff80239cdf in fork_exit (
    callout=0xffffffff8023e0b0 <ithread_loop>, arg=0xffffff00018537a0, 
    frame=0xffffff8000048c50) at /usr/src/sys/kern/kern_fork.c:876
#21 0xffffffff803c545e in fork_trampoline ()
    at /usr/src/sys/amd64/amd64/exception.S:602
#22 0x0000000000000000 in ?? ()
#23 0x0000000000000000 in ?? ()
#24 0x0000000000000001 in ?? ()
#25 0x0000000000000000 in ?? ()
#26 0x0000000000000000 in ?? ()
#27 0x0000000000000000 in ?? ()
#28 0x0000000000000000 in ?? ()
#29 0x0000000000000000 in ?? ()
#30 0x0000000000000000 in ?? ()
#31 0x0000000000000000 in ?? ()
#32 0x0000000000000000 in ?? ()
#33 0x0000000000000000 in ?? ()
#34 0x0000000000000000 in ?? ()
#35 0x0000000000000000 in ?? ()
#36 0x0000000000000000 in ?? ()
#37 0x0000000000000000 in ?? ()
#38 0x0000000000000000 in ?? ()
#39 0x0000000000000000 in ?? ()
#40 0x0000000000000000 in ?? ()
#41 0x0000000000000000 in ?? ()
#42 0x0000000000000000 in ?? ()
#43 0x0000000000000000 in ?? ()
#44 0x0000000000000000 in ?? ()
#45 0x0000000000000000 in ?? ()
#46 0xffffffff805dce00 in affinity ()
#47 0x0000000000000000 in ?? ()
#48 0x0000000000000000 in ?? ()
#49 0xffffff00018598c0 in ?? ()
#50 0xffffff8000047c30 in ?? ()
#51 0xffffff8000047bd8 in ?? ()
#52 0xffffff0001c88460 in ?? ()
#53 0xffffffff802879f2 in sched_switch (td=0xffffffff8023e0b0, 
    newtd=0xffffff00018537a0, flags=Variable "flags" is not available.
) at /usr/src/sys/kern/sched_ule.c:1861
Previous frame inner to this frame (corrupt stack?)
-------------------------------------------------------------------------


Once more:
-------------------------------------------------------------------------
panic: page fault

[...skipped (there was no unread portion of the kernel message buffer)...]

#0  doadump () at pcpu.h:224
224	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) #0  doadump () at pcpu.h:224
#1  0xffffffff802629f0 in boot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:441
#2  0xffffffff80262e73 in panic (fmt=0x0)
    at /usr/src/sys/kern/kern_shutdown.c:614
#3  0xffffffff803dd08d in trap_fatal (frame=0xffffffff805d2840, eva=Variable "eva" is not available.
)
    at /usr/src/sys/amd64/amd64/trap.c:825
#4  0xffffffff803dd3c1 in trap_pfault (frame=0xffffff80ed2363d0, usermode=0)
    at /usr/src/sys/amd64/amd64/trap.c:741
#5  0xffffffff803dd79f in trap (frame=0xffffff80ed2363d0)
    at /usr/src/sys/amd64/amd64/trap.c:478
#6  0xffffffff803c4f14 in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:228
#7  0xffffffff80819cf1 in DeleteLink (lnk=0xffffff009cffec80)
    at /usr/src/sys/modules/libalias/libalias/../../../netinet/libalias/alias_db.c:859
#8  0xffffffff80819f01 in HouseKeeping (la=0xffffff8000980000)
    at /usr/src/sys/modules/libalias/libalias/../../../netinet/libalias/alias_db.c:849
#9  0xffffffff808175ab in LibAliasOutLocked (la=0xffffff8000980000, ptr=dwarf2_read_address: Corrupted DWARF expression.
)
    at /usr/src/sys/modules/libalias/libalias/../../../netinet/libalias/alias.c:1451
#10 0xffffffff808185e4 in LibAliasOut (la=0xffffff8000980000, 
    ptr=0xffffff009936a010 "E", maxpacketsize=2032)
    at /usr/src/sys/modules/libalias/libalias/../../../netinet/libalias/alias.c:1418
#11 0xffffffff80830348 in ng_nat_rcvdata (hook=0xffffff0001dcc580, 
    item=0xffffff004e38eb00)
    at /usr/src/sys/modules/netgraph/nat/../../../netgraph/ng_nat.c:722
#12 0xffffffff80825bb0 in ng_apply_item (node=0xffffff0032ad6000, 
    item=0xffffff004e38eb00, rw=1)
    at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2327
#13 0xffffffff80824c7e in ng_snd_item (item=Variable "item" is not available.
)
    at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2244
#14 0xffffffff80339d07 in ipfw_check_hook (arg=Variable "arg" is not available.
)
    at /usr/src/sys/netinet/ipfw/ip_fw_pfil.c:225
#15 0xffffffff8031192c in pfil_run_hooks (ph=Variable "ph" is not available.
) at /usr/src/sys/net/pfil.c:82
#16 0xffffffff80341410 in ip_output (m=0xffffff00324c8100, opt=Variable "opt" is not available.
)
    at /usr/src/sys/netinet/ip_output.c:511
#17 0xffffffff8033178f in dummynet_send (m=0xffffff00324c8100)
    at /usr/src/sys/netinet/ipfw/ip_dn_io.c:652
#18 0xffffffff80331996 in dummynet_task (context=Variable "context" is not available.
)
    at /usr/src/sys/netinet/ipfw/ip_dn_io.c:615
#19 0xffffffff8029f325 in taskqueue_run_locked (queue=0xffffff0001d18a80)
    at /usr/src/sys/kern/subr_taskqueue.c:250
#20 0xffffffff8029f4be in taskqueue_thread_loop (arg=Variable "arg" is not available.
)
    at /usr/src/sys/kern/subr_taskqueue.c:387
#21 0xffffffff80239cdf in fork_exit (
    callout=0xffffffff8029f470 <taskqueue_thread_loop>, 
    arg=0xffffffff805fc080, frame=0xffffff80ed236c50)
    at /usr/src/sys/kern/kern_fork.c:876
#22 0xffffffff803c545e in fork_trampoline ()
    at /usr/src/sys/amd64/amd64/exception.S:602
#23 0x0000000000000000 in ?? ()
#24 0x0000000000000000 in ?? ()
#25 0x0000000000000000 in ?? ()
#26 0x0000000000000000 in ?? ()
#27 0x0000000000000000 in ?? ()
#28 0x0000000000000000 in ?? ()
#29 0x0000000000000000 in ?? ()
#30 0x0000000000000000 in ?? ()
#31 0x0000000000000000 in ?? ()
#32 0x0000000000000000 in ?? ()
#33 0x0000000000000000 in ?? ()
#34 0x0000000000000000 in ?? ()
#35 0x0000000000000000 in ?? ()
#36 0x0000000000000000 in ?? ()
#37 0x0000000000000000 in ?? ()
#38 0x0000000000000000 in ?? ()
#39 0x0000000000000000 in ?? ()
#40 0x0000000000000000 in ?? ()
#41 0x0000000000000000 in ?? ()
#42 0x0000000000000000 in ?? ()
#43 0x0000000000000000 in ?? ()
#44 0x0000000000000000 in ?? ()
#45 0x0000000000000000 in ?? ()
#46 0x0000000000000000 in ?? ()
#47 0xffffffff805f6b38 in sleepq_chains ()
#48 0xffffff0001c87430 in ?? ()
#49 0x0000000000000000 in ?? ()
#50 0xffffff0001c87000 in ?? ()
#51 0xffffff80ed236b00 in ?? ()
#52 0xffffff80ed236aa8 in ?? ()
#53 0xffffff0001858000 in ?? ()
#54 0xffffffff802879f2 in sched_switch (td=0xffffffff8029f470, 
    newtd=0xffffffff805fc080, flags=Variable "flags" is not available.
) at /usr/src/sys/kern/sched_ule.c:1861
Previous frame inner to this frame (corrupt stack?)
(kgdb) 

-------------------------------------------------------------------------


And more:
-------------------------------------------------------------------------
Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer	= 0x20:0xffffffff8032e431
stack pointer	        = 0x28:0xffffff80ed236a90
frame pointer	        = 0x28:0xffffff80ed236ac0
code segment		= base rx0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 0 (dummynet)
trap number		= 9
panic: general protection fault
cpuid = 0
Uptime: 2d4h4m21s

[...skipped...]

#0  doadump () at pcpu.h:224
224	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) #0  doadump () at pcpu.h:224
#1  0xffffffff802629f0 in boot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:441
#2  0xffffffff80262e73 in panic (fmt=0x0)
    at /usr/src/sys/kern/kern_shutdown.c:614
#3  0xffffffff803dd08d in trap_fatal (frame=0xffffffff805d2840, eva=Variable "eva" is not available.
)
    at /usr/src/sys/amd64/amd64/trap.c:825
#4  0xffffffff803dd54a in trap (frame=0xffffff80ed2369e0)
    at /usr/src/sys/amd64/amd64/trap.c:621
#5  0xffffffff803c4f14 in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:228
#6  0xffffffff8032e431 in drain_scheduler_cb (_si=0x1100006b8bc26b, arg=0x0)
    at /usr/src/sys/netinet/ipfw/ip_dummynet.c:1958
#7  0xffffffff8032e4a5 in drain_scheduler_sch_cb (_s=Variable "_s" is not available.
)
    at /usr/src/sys/netinet/ipfw/ip_dummynet.c:1990
#8  0xffffffff8032b16a in dn_ht_scan_bucket (ht=0xffffff0001d1d400, bucket=Variable "bucket" is not available.
)
    at /usr/src/sys/netinet/ipfw/dn_heap.c:537
#9  0xffffffff8032e230 in dn_drain_scheduler ()
    at /usr/src/sys/netinet/ipfw/ip_dummynet.c:2001
#10 0xffffffff80331a25 in dummynet_task (context=Variable "context" is not available.
)
    at /usr/src/sys/netinet/ipfw/ip_dn_io.c:608
#11 0xffffffff8029f325 in taskqueue_run_locked (queue=0xffffff0001d18a80)
    at /usr/src/sys/kern/subr_taskqueue.c:250
#12 0xffffffff8029f4be in taskqueue_thread_loop (arg=Variable "arg" is not available.
)
    at /usr/src/sys/kern/subr_taskqueue.c:387
#13 0xffffffff80239cdf in fork_exit (
    callout=0xffffffff8029f470 <taskqueue_thread_loop>, 
    arg=0xffffffff805fc080, frame=0xffffff80ed236c50)
    at /usr/src/sys/kern/kern_fork.c:876
#14 0xffffffff803c545e in fork_trampoline ()
    at /usr/src/sys/amd64/amd64/exception.S:602
#15 0x0000000000000000 in ?? ()
#16 0x0000000000000000 in ?? ()
#17 0x0000000000000000 in ?? ()
#18 0x0000000000000000 in ?? ()
#19 0x0000000000000000 in ?? ()
#20 0x0000000000000000 in ?? ()
#21 0x0000000000000000 in ?? ()
#22 0x0000000000000000 in ?? ()
#23 0x0000000000000000 in ?? ()
#24 0x0000000000000000 in ?? ()
#25 0x0000000000000000 in ?? ()
#26 0x0000000000000000 in ?? ()
#27 0x0000000000000000 in ?? ()
#28 0x0000000000000000 in ?? ()
#29 0x0000000000000000 in ?? ()
#30 0x0000000000000000 in ?? ()
#31 0x0000000000000000 in ?? ()
#32 0x0000000000000000 in ?? ()
#33 0x0000000000000000 in ?? ()
#34 0x0000000000000000 in ?? ()
#35 0x0000000000000000 in ?? ()
#36 0x0000000000000000 in ?? ()
#37 0x0000000000000000 in ?? ()
#38 0x0000000000000000 in ?? ()
#39 0xffffffff805f6b38 in sleepq_chains ()
#40 0xffffff0001c87430 in ?? ()
#41 0x0000000000000000 in ?? ()
#42 0xffffff0001c87000 in ?? ()
#43 0xffffff80ed236b00 in ?? ()
#44 0xffffff80ed236aa8 in ?? ()
#45 0xffffff00018598c0 in ?? ()
#46 0xffffffff802879f2 in sched_switch (td=0xffffffff8029f470, 
    newtd=0xffffffff805fc080, flags=Variable "flags" is not available.
) at /usr/src/sys/kern/sched_ule.c:1861
Previous frame inner to this frame (corrupt stack?)
(kgdb) 
-------------------------------------------------------------------------




kernel config:

options	CONFIG_AUTOGENERATED
ident	ROUTER_HOME_NETS
machine	amd64
cpu	HAMMER
makeoptions	DEBUG=-g
options	HWPMC_HOOKS
options	ENABLE_ALART
options	KDB_UNATTENDED
options	HZ=1000
options	DUMMYNET
options	IPDIVERT
options	IPFIREWALL_FORWARD
options	IPFIREWALL_DEFAULT_TO_ACCEPT
options	IPFIREWALL_VERBOSE_LIMIT=400
options	IPFIREWALL_VERBOSE
options	IPFIREWALL
options	CONSPEED=115200
options	ATA_STATIC_ID
options	ADAPTIVE_LOCKMGRS
options	ACCEPT_FILTER_HTTP
options	ACCEPT_FILTER_DATA
options	INCLUDE_CONFIG_FILE
options	AUDIT
options	ZERO_COPY_SOCKETS
options	SC_HISTORY_SIZE=8192
options	SC_KERNEL_CONS_ATTR=(FG_YELLOW|BG_BLACK)
options	SC_NORM_ATTR=(FG_GREEN|BG_BLACK)
options	SC_DISABLE_REBOOT
options	ROUTETABLES=4
options	FLOWTABLE
options	KBD_INSTALL_CDEV
options	_KPOSIX_PRIORITY_SCHEDULING
options	SYSVSEM
options	SYSVMSG
options	SYSVSHM
options	STACK
options	KTRACE
options	GEOM_PART_GPT
options	GEOM_LABEL
options	PSEUDOFS
options	PROCFS
options	CD9660
options	MSDOSFS
options	MD_ROOT
options	UFS_GJOURNAL
options	UFS_DIRHASH
options	SOFTUPDATES
options	FFS
options	INET
options	IPI_PREEMPTION
options	PREEMPTION
options	SCHED_ULE
options	SMP
options	GEOM_PART_MBR
options	GEOM_PART_EBR_COMPAT
options	GEOM_PART_EBR
options	GEOM_PART_BSD
device	isa
device	mem
device	io
device	uart_ns8250
device	pci
device	acpi
device	ata
device	atadisk
device	atapicd
device	atkbdc
device	atkbd
device	psm
device	vga
device	sc
device	re
device	miibus
device	loop
device	random
device	ether
device	pty
device	md
device	bpf
device	intpm
device	iicbus
device	smbus
device	iicsmb
device	hwpmc


/boot/loader.conf:

autoboot_delay="1"
if_em_load="YES"
geom_mirror_load="YES"          # RAID1 disk driver (see gmirror(8))
vfs.root.mountfrom="ufs:/dev/mirror/root"
cc_chd_load="YES"
coretemp_load="YES"
net.graph.maxdata=16384
net.isr.numthreads=2
net.isr.maxthreads=2
net.isr.bindthreads=1
hw.em.rxd=4096
hw.em.txd=4096


/etc/sysctl.conf:

dev.em.0.rx_abs_int_delay=4000
dev.em.0.rx_int_delay=200
dev.em.0.rx_processing_limit=4096
dev.em.0.tx_abs_int_delay=4000
dev.em.0.tx_int_delay=200
kern.ipc.maxsockbuf=83886080
kern.ipc.nmbclusters=262144
net.graph.maxdgram=8388608
net.graph.recvspace=8388608
net.inet.icmp.drop_redirect=1
net.inet.icmp.icmplim=2048
net.inet.icmp.log_redirect=1
net.inet.ip.dummynet.expire=0
net.inet.ip.dummynet.hash_size=512
net.inet.ip.dummynet.io_fast=1
net.inet.ip.dummynet.pipe_byte_limit=2097152
net.inet.ip.dummynet.pipe_slot_limit=1000
net.inet.ip.fw.dyn_max=32768
net.inet.ip.fw.one_pass=0
net.inet.ip.fw.verbose=1
net.inet.ip.fw.verbose_limit=100
net.inet.ip.redirect=0
net.inet.tcp.blackhole=1
net.inet.udp.blackhole=1
net.isr.direct=0
net.isr.direct_force=0
net.link.ether.ipfw=0


ipfw list:

00100 allow ip from any to any via lo0
00200 allow ip from 10.1.0.0/24 to 10.1.0.0/24
00400 setfib 1 ip from any to any via vlan500 // WAN-2
00500 setfib 1 ip from table(2) to any // WAN-2
00600 setfib 1 ip from x.x.199.248/29 to any // WAN-2
00700 setfib 1 ip from any to x.x.199.248/29 // WAN-2
00800 fwd x.x.199.249 ip from x.x.199.250 to any xmit vlan720 // My traffic to WAN-2
00900 setfib 1 ip from x.x.27.0/24 to any // WAN-2's real ips
01000 count log logamount 29999 tcp from any to any dst-port 25 setup out recv vlan100 // Dumb antispam rule
01200 allow tcp from any to me dst-port 22 setup keep-state // SSH
01400 allow tcp from table(11) to me dst-port 20,21,22,3306 setup keep-state // FTP and HTTP for trusted clients
01600 allow tcp from any to me dst-port 443 setup keep-state // HTTPS for any clients (ACLs in Apache, for payments)
02000 fwd 127.0.0.1,3128 tcp from any to 212.193.229.77,92.241.171.120 dst-port 80 recv vlan100 // TProxy for real gismeteo.ru
03000 skipto 6000 ip from table(1) to any // Skip deny rules for allowed ips
03100 unreach filter-prohib ip from any to not me in recv vlan100 // Verbose deny for internal lan
04000 netgraph 4000 ip from any to me recv vlan720 // Inside NAT
04100 netgraph 4100 ip from any to x.x.199.250 recv vlan500 // Inside NAT
06000 pipe tablearg ip from any to table(4) xmit vlan100
06000 pipe tablearg ip from table(5) to any xmit vlan720
06000 pipe tablearg ip from table(5) to any xmit vlan500
06500 ngtee 1 ip from table(1) to any // Netflow Accounting from users
06600 ngtee 2 ip from any to table(1) in recv vlan720 // Netflow Accounting to users from WAN-1
06600 ngtee 2 ip from any to table(1) in recv vlan500 // Netflow Accounting to users from WAN-2
07000 netgraph 7000 ip from 10.1.0.0/16 to any xmit vlan720 // Outside NAT
07100 netgraph 7100 ip from 10.1.0.0/16 to any xmit vlan500 // Outside NAT
08000 allow ip from me to any xmit vlan720 // Allow all from me
08100 allow ip from x.x.199.248/29,x.x.27.0/24 to any xmit vlan500
08150 allow tcp from me to any setup keep-state // Tcp from me
08160 allow udp from any to me keep-state // Open my udp
08170 allow udp from me to any keep-state // Open my udp
08200 allow ip from any to any established
08300 allow ip from table(1) to any
08400 allow ip from any to table(1)
08500 allow icmp from any to any icmptypes 0,3,4,8,11
65530 deny log logamount 100 ip from any to any
65535 allow ip from any to any

How-To-Repeat: Don't know

Comment 1 Gleb Smirnoff freebsd_committer

2012-04-15 12:48:08 UTC

On Sat, Apr 14, 2012 at 06:18:13AM +0000, Dennis wrote:
D> FreeBSD utm.leskolovo.ru 8.2-STABLE FreeBSD 8.2-STABLE #4: Sun Mar 25 10:26:56 MSK 2012     root@utm.leskolovo.ru:/usr/obj/usr/src/sys/ROUTER_HOME_NETS  amd64
D> >Description:
D> Our server start frequently and randomly crashing and we don't know why. :(
D> 
D> That server do billing (netflow, apache, mysql) and gateway (nat, policy-based routing with 2 WANs, ipfw, dummynet) for relatively small network (summary traffic up to 100 Mbit/sec)

Can you try to live w/o dummynet? Would the server get stable if you disable
dummynet?

-- 
Totus tuus, Glebius.

Comment 2 Gleb Smirnoff freebsd_committer

2012-04-15 12:49:15 UTC

On Sat, Apr 14, 2012 at 06:18:13AM +0000, Dennis wrote:
D> FreeBSD utm.leskolovo.ru 8.2-STABLE FreeBSD 8.2-STABLE #4: Sun Mar 25 10:26:56 MSK 2012     root@utm.leskolovo.ru:/usr/obj/usr/src/sys/ROUTER_HOME_NETS  amd64
D> >Description:
D> Our server start frequently and randomly crashing and we don't know why. :(
D> 
D> That server do billing (netflow, apache, mysql) and gateway (nat, policy-based routing with 2 WANs, ipfw, dummynet) for relatively small network (summary traffic up to 100 Mbit/sec)

One more question: does the server create and destroy interfaces at run time?

-- 
Totus tuus, Glebius.

Comment 3 Dennis Yusupoff 2012-04-16 09:49:13 UTC

15.04.2012 15:48, Gleb Smirnoff ÎÁÐÉÓÁÌ:
> Can you try to live w/o dummynet? Would the server get stable if you disable
> dummynet?

I can't, dummynet is important part. However, I removed that server from
production and it's on my desk now, so I could do any tests as you wish.
Instead of this one I setted up two servers based on FreeBSD 9.0-RELEASE
-one for billing-related, one for gateway-related tasks.

> One more question: does the server create and destroy interfaces at run time?

No. The only one dynamic interface was mpd5 L2TP tunnel from other
server, but we tried also with disabled mpd5.
By the way, colleague said what *probably* (he doesn't sure) kernel
panic begun after mpd5 has been installed, even then it was disabled.


P. S. Part of cron's dynamically generated rules:

Shaping:
---
# ID 1310, Bandwidth 1024 Kb/s , IP 10.1.45.24
/sbin/ipfw pipe 1310 config bw 1075Kb/s mask dst-ip 0xFFFFFFFF type FIFO
buckets 16 burst 134
/sbin/ipfw pipe 11311 config bw 537Kb/s mask src-ip 0xFFFFFFFF type FIFO
buckets 16 burst 134
/sbin/ipfw table 4 add 10.1.45.24 1310
/sbin/ipfw table 5 add 10.1.45.24 11311

# ID 1314, Bandwidth 1024 Kb/s , IP 10.1.0.1 10.1.26.34
/sbin/ipfw pipe 1314 config bw 1075Kb/s mask dst-ip 0xFFFFFFFF type FIFO
buckets 16 burst 134
/sbin/ipfw pipe 11315 config bw 537Kb/s mask src-ip 0xFFFFFFFF type FIFO
buckets 16 burst 134
/sbin/ipfw table 4 add 10.1.0.1 1314
/sbin/ipfw table 5 add 10.1.0.1 11315
/sbin/ipfw table 4 add 10.1.26.34 1314
/sbin/ipfw table 5 add 10.1.26.34 11315
---

Routing:
---
/sbin/ipfw -q table 1 add 10.1.15.13 1 && echo -n "."
/sbin/ipfw -q table 1 add 10.1.74.43 1 && echo -n "."
/sbin/ipfw -q table 1 delete 10.1.76.17 2>/dev/null && echo -n "-"
/sbin/ipfw -q table 1 delete 10.1.22.29 2>/dev/null && echo -n "-"
---

Comment 4 Dennis Yusupoff 2012-04-19 11:56:07 UTC

Hello!

Likely we found the cause of problem - memtest found few faults in the
memory (non-ECC), while other (stress) tests found nothing. However, we
haven't tests networking part except of create/modify ipfw customer's
pipes in loop.

---
With best regards,
Dennis Yusupoff,

Comment 5 Gavin Atkinson freebsd_committer

2012-04-29 13:56:42 UTC

State Changed
From-To: open->closed

Close, submitter discovered memory faults.