After giving Varnish some load, FreeBSD kernel crashes: login: Sleeping thread (tid 100038, pid 31) owns a non-sleepable lock sched_switch() at sched_switch+0x184 mi_switch() at mi_switch+0x189 sleepq_wait() at sleepq_wait+0x3b _sx_slock_hard() at _sx_slock_hard+0x19d fr_check() at fr_check+0x2b7 pfil_run_hooks() at pfil_run_hooks+0x9c ip_output() at ip_output+0x339 tcp_output() at tcp_output+0x982 tcp_do_segment() at tcp_do_segment+0x9f8 tcp_input() at tcp_input+0x759 ip_input() at ip_input+0xa8 ether_demux() at ether_demux+0x1b4 ether_input() at ether_input+0x1bb bce_intr() at bce_intr+0x24f ithread_loop() at ithread_loop+0x180 fork_exit() at fork_exit+0x11f fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffae3f9d30, rbp = 0 --- panic: sleeping thread cpuid = 6 KDB: enter: panic [thread pid 1170 tid 100523 ] Stopped at kdb_enter+0x31: leave db> bt Tracing pid 1170 tid 100523 td 0xffffff00229869c0 kdb_enter() at kdb_enter+0x31 panic() at panic+0x173 propagate_priority() at propagate_priority+0x1ec turnstile_wait() at turnstile_wait+0x1be _mtx_lock_sleep() at _mtx_lock_sleep+0x9e in_getsockaddr() at in_getsockaddr+0xb3 kern_getsockname() at kern_getsockname+0x71 getsockname() at getsockname+0x63 syscall() at syscall+0x254 Xfast_syscall() at Xfast_syscall+0xab --- syscall (32, FreeBSD ELF64, getsockname), rip = 0x800c640ec, rsp = 0x7fffdb2d85e8, rbp = 0x7fffdb2d86c0 --- db> I have two quad-core processors like this: CPU: Intel(R) Xeon(R) CPU X5355 @ 2.66GHz (2666.78-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6f7 Stepping = 7 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x4e3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA> AMD Features=0x20000800<SYSCALL,LM> AMD Features2=0x1<LAHF> Cores per package: 4 Fix: N/A How-To-Repeat: 1) Install FreeBSD RELENG_7. Mine is as of october 15. 2) Install Varnish/trunk (up to date to commit 2096), from http://varnish.projects.linpro.no/. 3) Start Varnish. Preferrably on a SMP system with several data files for storage. I use an 8-core system with 8 GB RAM and 3 data files on separate RAID volumes. 4) Give Varnish load.
I should note that I was running IP Filter on this system. Removing IP Filter, the problem goes away. This is related to PR 117182? -- Anders.
Responsible Changed From-To: freebsd-bugs->darrenr Over to maintainer.
darrenr 2007-10-30 15:23:27 UTC FreeBSD src repository Modified files: sys/contrib/ipfilter/netinet fil.c ip_auth.c ip_compat.h ip_fil_freebsd.c ip_log.c ip_nat.c ip_state.c Log: Apply a few changes from ipfilter-current: * Do not hold any locks over calls to copyin/copyout. * Clean up some #ifdefs * fix a possible mbuf leak when NAT fails on policy routed packets PR: 117216 Revision Changes Path 1.54 +4 -4 src/sys/contrib/ipfilter/netinet/fil.c 1.46 +1 -1 src/sys/contrib/ipfilter/netinet/ip_auth.c 1.35 +1 -1 src/sys/contrib/ipfilter/netinet/ip_compat.h 1.8 +7 -6 src/sys/contrib/ipfilter/netinet/ip_fil_freebsd.c 1.35 +6 -5 src/sys/contrib/ipfilter/netinet/ip_log.c 1.44 +44 -26 src/sys/contrib/ipfilter/netinet/ip_nat.c 1.41 +6 -1 src/sys/contrib/ipfilter/netinet/ip_state.c _______________________________________________ cvs-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/cvs-all To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
darrenr 2007-10-31 05:00:38 UTC FreeBSD src repository Modified files: (Branch: RELENG_7) contrib/ipfilter HISTORY Makefile ip_fil.c md5.h radix.c radix_ipf.h contrib/ipfilter/BSD Makefile kupgrade contrib/ipfilter/iplang Makefile contrib/ipfilter/ipsend iptests.c sock.c contrib/ipfilter/l4check Makefile l4check.c contrib/ipfilter/lib Makefile alist_new.c ipft_tx.c printnat.c printpacket.c printpool_live.c printstate.c contrib/ipfilter/man ippool.5 contrib/ipfilter/test Makefile dotest nattest test.format contrib/ipfilter/test/expected f11 i21 in1 in6 contrib/ipfilter/test/input f11 l1 contrib/ipfilter/test/regress i21 i3 in1 in6 contrib/ipfilter/tools ipf_y.y ipfstat.c ipmon.c ipnat.c ipnat_y.y lexer.c sys/contrib/ipfilter/netinet fil.c ip_auth.c ip_compat.h ip_fil.h ip_fil_freebsd.c ip_frag.c ip_htable.c ip_log.c ip_lookup.c ip_lookup.h ip_nat.c ip_nat.h ip_pool.c ip_pool.h ip_proxy.c ip_rpcb_pxy.c ip_scan.c ip_state.c ip_state.h ip_sync.c ipl.h mlfk_ipl.c Log: MFC the following: Apply a few changes from ipfilter-current: * Do not hold any locks over calls to copyin/copyout. * Clean up some #ifdefs * fix a possible mbuf leak when NAT fails on policy routed packets PR: 117216 Approved by: re Revision Changes Path 1.1.1.12.2.1 +10 -2 src/contrib/ipfilter/BSD/Makefile 1.1.1.7.2.1 +9 -5 src/contrib/ipfilter/BSD/kupgrade 1.1.1.27.2.1 +99 -1 src/contrib/ipfilter/HISTORY 1.7.2.1 +9 -12 src/contrib/ipfilter/Makefile 1.5.2.1 +7 -4 src/contrib/ipfilter/ip_fil.c 1.1.1.4.10.1 +5 -4 src/contrib/ipfilter/iplang/Makefile 1.13.2.1 +7 -3 src/contrib/ipfilter/ipsend/iptests.c 1.18.2.1 +7 -3 src/contrib/ipfilter/ipsend/sock.c 1.1.1.1.24.1 +1 -1 src/contrib/ipfilter/l4check/Makefile 1.2.10.1 +31 -14 src/contrib/ipfilter/l4check/l4check.c 1.1.1.4.2.1 +1 -7 src/contrib/ipfilter/lib/Makefile 1.1.1.1.2.1 +5 -3 src/contrib/ipfilter/lib/alist_new.c 1.6.2.1 +24 -13 src/contrib/ipfilter/lib/ipft_tx.c 1.4.2.1 +10 -5 src/contrib/ipfilter/lib/printnat.c 1.4.2.1 +3 -3 src/contrib/ipfilter/lib/printpacket.c 1.1.1.1.2.1 +9 -4 src/contrib/ipfilter/lib/printpool_live.c 1.5.2.1 +3 -3 src/contrib/ipfilter/lib/printstate.c 1.2.10.1 +2 -2 src/contrib/ipfilter/man/ippool.5 1.2.10.1 +2 -2 src/contrib/ipfilter/md5.h 1.4.2.1 +7 -1 src/contrib/ipfilter/radix.c 1.4.2.1 +3 -3 src/contrib/ipfilter/radix_ipf.h 1.1.1.16.2.1 +14 -10 src/contrib/ipfilter/test/Makefile 1.1.1.4.2.1 +7 -1 src/contrib/ipfilter/test/dotest 1.1.1.2.24.1 +124 -0 src/contrib/ipfilter/test/expected/f11 1.1.1.1.2.1 +6 -0 src/contrib/ipfilter/test/expected/i21 1.1.1.5.2.1 +1 -0 src/contrib/ipfilter/test/expected/in1 1.1.1.2.2.1 +1 -0 src/contrib/ipfilter/test/expected/in6 1.1.1.3.10.1 +11 -11 src/contrib/ipfilter/test/input/f11 1.1.1.2.10.1 +8 -8 src/contrib/ipfilter/test/input/l1 1.1.1.2.10.1 +8 -1 src/contrib/ipfilter/test/nattest 1.1.1.1.2.1 +1 -0 src/contrib/ipfilter/test/regress/i21 1.1.1.3.10.1 +4 -2 src/contrib/ipfilter/test/regress/i3 1.1.1.4.2.1 +1 -0 src/contrib/ipfilter/test/regress/in1 1.1.1.2.2.1 +1 -0 src/contrib/ipfilter/test/regress/in6 1.1.1.4.2.1 +4 -1 src/contrib/ipfilter/test/test.format 1.6.2.1 +25 -1 src/contrib/ipfilter/tools/ipf_y.y 1.6.2.1 +4 -4 src/contrib/ipfilter/tools/ipfstat.c 1.7.2.1 +33 -4 src/contrib/ipfilter/tools/ipmon.c 1.5.2.1 +63 -4 src/contrib/ipfilter/tools/ipnat.c 1.5.2.1 +2 -1 src/contrib/ipfilter/tools/ipnat_y.y 1.4.2.1 +40 -17 src/contrib/ipfilter/tools/lexer.c 1.52.2.1 +164 -125 src/sys/contrib/ipfilter/netinet/fil.c 1.44.2.1 +19 -19 src/sys/contrib/ipfilter/netinet/ip_auth.c 1.33.2.1 +127 -57 src/sys/contrib/ipfilter/netinet/ip_compat.h 1.35.2.1 +32 -21 src/sys/contrib/ipfilter/netinet/ip_fil.h 1.6.2.1 +136 -149 src/sys/contrib/ipfilter/netinet/ip_fil_freebsd.c 1.32.2.1 +9 -9 src/sys/contrib/ipfilter/netinet/ip_frag.c 1.4.2.1 +40 -52 src/sys/contrib/ipfilter/netinet/ip_htable.c 1.33.2.1 +22 -16 src/sys/contrib/ipfilter/netinet/ip_log.c 1.1.1.3.2.1 +43 -7 src/sys/contrib/ipfilter/netinet/ip_lookup.c 1.1.1.3.2.1 +1 -2 src/sys/contrib/ipfilter/netinet/ip_lookup.h 1.42.2.1 +189 -69 src/sys/contrib/ipfilter/netinet/ip_nat.c 1.26.2.1 +7 -3 src/sys/contrib/ipfilter/netinet/ip_nat.h 1.1.1.3.2.1 +36 -49 src/sys/contrib/ipfilter/netinet/ip_pool.c 1.1.1.3.2.1 +2 -2 src/sys/contrib/ipfilter/netinet/ip_pool.h 1.29.2.1 +7 -5 src/sys/contrib/ipfilter/netinet/ip_proxy.c 1.1.1.3.2.1 +1 -1 src/sys/contrib/ipfilter/netinet/ip_rpcb_pxy.c 1.1.1.4.2.1 +4 -2 src/sys/contrib/ipfilter/netinet/ip_scan.c 1.39.2.1 +109 -65 src/sys/contrib/ipfilter/netinet/ip_state.c 1.19.2.1 +5 -7 src/sys/contrib/ipfilter/netinet/ip_state.h 1.5.2.1 +6 -6 src/sys/contrib/ipfilter/netinet/ip_sync.c 1.26.2.1 +5 -5 src/sys/contrib/ipfilter/netinet/ipl.h 1.19.2.1 +11 -2 src/sys/contrib/ipfilter/netinet/mlfk_ipl.c _______________________________________________ cvs-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/cvs-all To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
State Changed From-To: open->feedback To submitter: did this commit fix your problem?
Hi, While I unfortunately have not been able to try this with 7-current yet, I do see a crash that happens rather often in 6.3-PRERELEASE (up to date to 30 december) which has the same version of IP Filter (4.1.28): Fatal trap 12: page fault while in kernel mode fault virtual address = 0xc fault code = supervisor read, page not present instruction pointer = 0x20:0xc05100e7 stack pointer = 0x28:0xc7775b28 frame pointer = 0x28:0xc7775b4c code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 11 (swi1: net) trap number = 12 panic: page fault KDB: stack backtrace: kdb_backtrace(100,c110f780,28,c7775ae8,c,...) at kdb_backtrace+0x29 panic(c06397a8,c06565e6,0,fffff,c110ea9b,...) at panic+0xa8 trap_fatal(c7775ae8,c,c110f780,0,c,...) at trap_fatal+0x2a6 trap_pfault(c7775ae8,0,c) at trap_pfault+0x1f3 trap(8,28,180028,c11e8d54,588,...) at trap+0x325 calltrap() at calltrap+0x5 --- trap 0xc, eip = 0xc05100e7, esp = 0xc7775b28, ebp = 0xc7775b4c --- m_copym(0,5dc,5c8,1,14,...) at m_copym+0x2f ip_fragment(c134f80e,c7775c04,5dc,0,1,...) at ip_fragmestray irq7 nt+0x214 ip_output(c130d800,0,c7775bd0,1,0,0) at ip_output+0x85e ip_forward(c130d800,0) at ip_forward+0x280 ip_input(c130d800) at ip_input+0x59f netisr_processqueue(c0698118) at netisr_processqueue+0x9f swi_net(0) at swi_net+0xf2 ithread_execute_handlers(c110ea78,c1101500) at ithread_execute_handlers+0x121 ithread_loop(c10f8770,c7775d38) at ithread_loop+0x54 fork_exit(c04c3344,c10f8770,c7775d38) at fork_exit+0x70 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xc7775d6c, ebp = 0 --- Uptime: 1h12m56s Cannot dump. No dump device defined. Automatic reboot in 15 seconds - press a key on the console to abort Rebooting... PC Engines WRAP.1C/1D/1E v1.08 640 KB Base Memory 153603174448128645128089697280113664130048 KB Extended Memory This is on my home firewall: - Even with just pass in all/pass out all rules. - Nat rules: map ath0 192.168.78.0/24 -> 0/32 proxy port ftp ftp/tcp map ath0 192.168.78.0/24 -> 0/32 proxy port 500 ipsec/udp map ath0 192.168.78.0/24 -> 0/32 portmap tcp/udp 40000:60000 map ath0 192.168.78.0/24 -> 0/32 - Typically happens when I rsync large datasets through it... This might be a different bug than this PR originally was about. I'll try to get that checked soonish. On Mon, Nov 05, 2007 at 06:29:49AM +0000, linimon@FreeBSD.org wrote: > Synopsis: [ipfilter] FreeBSD 7-PRERELEASE crashes upon load when running Varnish trunk > > State-Changed-From-To: open->feedback > State-Changed-By: linimon > State-Changed-When: Mon Nov 5 06:29:05 UTC 2007 > State-Changed-Why: > To submitter: did this commit fix your problem? > > http://www.freebsd.org/cgi/query-pr.cgi?pr=117216 -- Anders.
Hi, On Sun, Dec 30, 2007 at 08:33:06PM +0100, Anders Nordby wrote: > panic: page fault > KDB: stack backtrace: > kdb_backtrace(100,c110f780,28,c7775ae8,c,...) at kdb_backtrace+0x29 > panic(c06397a8,c06565e6,0,fffff,c110ea9b,...) at panic+0xa8 > trap_fatal(c7775ae8,c,c110f780,0,c,...) at trap_fatal+0x2a6 > trap_pfault(c7775ae8,0,c) at trap_pfault+0x1f3 > trap(8,28,180028,c11e8d54,588,...) at trap+0x325 > calltrap() at calltrap+0x5 > --- trap 0xc, eip = 0xc05100e7, esp = 0xc7775b28, ebp = 0xc7775b4c --- > m_copym(0,5dc,5c8,1,14,...) at m_copym+0x2f > ip_fragment(c134f80e,c7775c04,5dc,0,1,...) at ip_fragmestray irq7 > nt+0x214 > ip_output(c130d800,0,c7775bd0,1,0,0) at ip_output+0x85e > ip_forward(c130d800,0) at ip_forward+0x280 > ip_input(c130d800) at ip_input+0x59f > netisr_processqueue(c0698118) at netisr_processqueue+0x9f > swi_net(0) at swi_net+0xf2 > ithread_execute_handlers(c110ea78,c1101500) at > ithread_execute_handlers+0x121 > ithread_loop(c10f8770,c7775d38) at ithread_loop+0x54 > fork_exit(c04c3344,c10f8770,c7775d38) at fork_exit+0x70 > fork_trampoline() at fork_trampoline+0x8 > --- trap 0x1, eip = 0, esp = 0xc7775d6c, ebp = 0 --- > Uptime: 1h12m56s > Cannot dump. No dump device defined. > Automatic reboot in 15 seconds - press a key on the console to abort > Rebooting... > PC Engines WRAP.1C/1D/1E v1.08 > 640 KB Base Memory > 153603174448128645128089697280113664130048 KB Extended Memory I'm sorry, but this also happens with PF. The problem seems to be with sis interfaces and polling. After turning off polling on my sis interface, I don't get these panics anymore. As said, I'll get back to the original problem for this PR. Bye, -- Anders.
State Changed From-To: feedback->closed This bug was raised against ipfilter and some potential fixes offered. The submitter now believes it is an sis driver problem, so i'd like to close this and encourage the original submitter to file a new bug.