Bug 116172
| Summary: | [tun] [nd6] [panic] Network / ipv6 recursive mutex panic | ||
|---|---|---|---|
| Product: | Base System | Reporter: | peter |
| Component: | kern | Assignee: | freebsd-bugs (Nobody) <bugs> |
| Status: | Open --- | ||
| Severity: | Affects Only Me | Keywords: | crash |
| Priority: | Normal | ||
| Version: | 7.0-CURRENT | ||
| Hardware: | Any | ||
| OS: | Any | ||
|
Description
peter
2007-09-07 07:50:01 UTC
Responsible Changed From-To: freebsd-bugs->freebsd-net Over to maintainer. I've narrowed down the panic trigger. I have two userland processes doing a tun0 tunnel. The purpose is to do IPv6 encapsulated in UDP. The local end has an open ssh connection to the remote end. If I reboot at that instant, both the ssh and the tunnel driver receive a sigterm at the same time. The death of the tun0 driver causes the tun0 interface to be torn down and the routes cleaned up. At the same time, the death of the ssh process attempts to send a tcp6 FIN to be sent. This causes the panic described above. A simple workaround is to do a 'ssh -4' to the remote end rather than over the tunnel. This avoids the simultanious tun0 route teardown and the ssh teardown. It only seems to be a problem with ssh to the precise remote endpoint. ssh over the tunnel to other machines does not cause the panic when the machine is rebooted. Sample tun0 driver to trigger the panic: http://people.freebsd.org/~peter/qd_tun.c - quick & dirty tunnel :-) Ifconfig some ipv6 addresses to each end. ssh to the other end, reboot(8) locally, wait for the kaboom! -Peter Hi,
I've been getting this (what I believe to be) same panic on my FreeBSD
6.2-p9 machines.
The key difference is that I am NOT using a tunnel. This is *native* ipv6.
Not enabling IPV6 at boot solves the problem for me, but is not a real
solution.
(kgdb) bt
#0 doadump () at pcpu.h:165
#1 0xc056b85e in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2 0xc056bb68 in panic (fmt=0xc07501cf "%s") at
/usr/src/sys/kern/kern_shutdown.c:565
#3 0xc072406d in trap_fatal (frame=0xe70c9838, eva=0) at
/usr/src/sys/i386/i386/trap.c:837
#4 0xc0723785 in trap (frame=
{tf_fs = -1067122680, tf_es = -418643928, tf_ds = 40, tf_edi =
-995259392, tf_esi = -947956992, tf_ebp = -418604932, tf_isp =
-418604956, tf_ebx = -928072192, tf_edx = -947956992, tf_ecx = 4, tf_eax
= 4, tf_trapno = 12, tf_err = 0, tf_eip = -1067904549, tf_cs = 32,
tf_eflags = 65539, tf_esp = -947956992, tf_ss = -418604896}) at
/usr/src/sys/i386/i386/trap.c:270
#5 0xc0710b0a in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#6 0xc05911db in turnstile_setowner (ts=0xc8aebe00, owner=0x4) at
/usr/src/sys/kern/subr_turnstile.c:432
#7 0xc0591507 in turnstile_wait (lock=0xc4e93ed0, owner=0x4) at
/usr/src/sys/kern/subr_turnstile.c:591
#8 0xc0560167 in _mtx_lock_sleep (m=0xc4e93ed0, tid=3347010304, opts=0,
file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:579
#9 0xc0650912 in nd6_output (ifp=0xc4ad8c00, origifp=0x4,
m0=0xc923c600, dst=0xc4d0c9dc, rt0=0xc4e9f000)
at /usr/src/sys/netinet6/nd6.c:2004
#10 0xc0649c58 in ip6_output (m0=0xe70c9a78, opt=0x0, ro=0xe70c9a78,
flags=0, im6o=0x0, ifpp=0x0, inp=0xc970a0b4)
at /usr/src/sys/netinet6/ip6_output.c:994
#11 0xc06267e4 in tcp_output (tp=0xc92d13a0) at
/usr/src/sys/netinet/tcp_output.c:1059
#12 0xc062ed52 in tcp_usr_send (so=0xc93c36f4, flags=0, m=0xc950b700,
nam=0x0, control=0x0, td=0xc77f5300)
at /usr/src/sys/netinet/tcp_usrreq.c:698
#13 0xc05ad988 in sosend (so=0xc93c36f4, addr=0x0, uio=0xe70c9cb0,
top=0xc950b700, control=0x0, flags=0, td=0xc77f5300)
at /usr/src/sys/kern/uipc_socket.c:836
#14 0xc05998c8 in soo_write (fp=0x4, uio=0xe70c9cb0,
active_cred=0xc8a77180, flags=0, td=0xc77f5300) at
/usr/src/sys/kern/sys_socket.c:118
#15 0xc0592ff0 in dofilewrite (td=0xc77f5300, fd=4, fp=0xc90d6288,
auio=0xe70c9cb0, offset=Unhandled dwarf expression opcode 0x93
) at file.h:252
#16 0xc0592e27 in kern_writev (td=0xc77f5300, fd=21, auio=0x4) at
/usr/src/sys/kern/sys_generic.c:402
#17 0xc0592cf9 in write (td=0x4, uap=0x4) at
/usr/src/sys/kern/sys_generic.c:326
#18 0xc0724423 in syscall (frame=
{tf_fs = -1078001605, tf_es = -1078001605, tf_ds = -1078001605,
tf_edi = 137218624, tf_esi = 137218624, tf_ebp = 154, tf_isp =
-418603676, tf_ebx = 270840404, tf_edx = 137166848, tf_ecx = 272127008,
tf_eax = 4, tf_trapno = 22, tf_err = 2, tf_eip = 272063395, tf_cs = 51,
tf_eflags = 530, tf_esp = -1077942612, tf_ss = 59}) at
/usr/src/sys/i386/i386/trap.c:983
#19 0xc0710b5f in Xint0x80_syscall () at
/usr/src/sys/i386/i386/exception.s:200
#20 0x00000033 in ?? ()
I actually did not use rc.conf to setup my ipv6 either, here is my
script for activating the v6 on boot:
ifconfig em0 inet6 2610:88:1::3:2 prefixlen 112
ifconfig em0 inet6 2610:88:1::3:3 prefixlen 112
route add -inet6 default 2610:88:1::3:0
What's odd is that these panics only began when my datacenter had a fire
in their 'router cage', and replaced the router. I'm not sure how this
is related, if at all, but the panics definitely only began on that
date, on two separate machines.
Bryan Drewery
Administrator
Xzibition Data Communications
Have you tried to make the gateway: 2610:88:1::3:1 ? -Gregg -----Original Message----- From: Bryan Drewery [mailto:bryan@xzibition.com] Sent: Tuesday, February 12, 2008 10:38 AM To: bug-followup@FreeBSD.org; peter@wemm.org Subject: Re: kern/116172: [tun] [panic] Network / ipv6 recursive mutex pani= c Hi, I've been getting this (what I believe to be) same panic on my FreeBSD 6.2-p9 machines. The key difference is that I am NOT using a tunnel. This is *native* ipv6. Not enabling IPV6 at boot solves the problem for me, but is not a real solution. (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc056b85e in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:4= 09 #2 0xc056bb68 in panic (fmt=3D0xc07501cf "%s") at /usr/src/sys/kern/kern_shutdown.c:565 #3 0xc072406d in trap_fatal (frame=3D0xe70c9838, eva=3D0) at /usr/src/sys/i386/i386/trap.c:837 #4 0xc0723785 in trap (frame=3D {tf_fs =3D -1067122680, tf_es =3D -418643928, tf_ds =3D 40, tf_edi = =3D -995259392, tf_esi =3D -947956992, tf_ebp =3D -418604932, tf_isp =3D -418604956, tf_ebx =3D -928072192, tf_edx =3D -947956992, tf_ecx =3D 4, tf_= eax =3D 4, tf_trapno =3D 12, tf_err =3D 0, tf_eip =3D -1067904549, tf_cs =3D 32= , tf_eflags =3D 65539, tf_esp =3D -947956992, tf_ss =3D -418604896}) at /usr/src/sys/i386/i386/trap.c:270 #5 0xc0710b0a in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #6 0xc05911db in turnstile_setowner (ts=3D0xc8aebe00, owner=3D0x4) at /usr/src/sys/kern/subr_turnstile.c:432 #7 0xc0591507 in turnstile_wait (lock=3D0xc4e93ed0, owner=3D0x4) at /usr/src/sys/kern/subr_turnstile.c:591 #8 0xc0560167 in _mtx_lock_sleep (m=3D0xc4e93ed0, tid=3D3347010304, opts= =3D0, file=3D0x0, line=3D0) at /usr/src/sys/kern/kern_mutex.c:579 #9 0xc0650912 in nd6_output (ifp=3D0xc4ad8c00, origifp=3D0x4, m0=3D0xc923c600, dst=3D0xc4d0c9dc, rt0=3D0xc4e9f000) at /usr/src/sys/netinet6/nd6.c:2004 #10 0xc0649c58 in ip6_output (m0=3D0xe70c9a78, opt=3D0x0, ro=3D0xe70c9a78, flags=3D0, im6o=3D0x0, ifpp=3D0x0, inp=3D0xc970a0b4) at /usr/src/sys/netinet6/ip6_output.c:994 #11 0xc06267e4 in tcp_output (tp=3D0xc92d13a0) at /usr/src/sys/netinet/tcp_output.c:1059 #12 0xc062ed52 in tcp_usr_send (so=3D0xc93c36f4, flags=3D0, m=3D0xc950b700, nam=3D0x0, control=3D0x0, td=3D0xc77f5300) at /usr/src/sys/netinet/tcp_usrreq.c:698 #13 0xc05ad988 in sosend (so=3D0xc93c36f4, addr=3D0x0, uio=3D0xe70c9cb0, top=3D0xc950b700, control=3D0x0, flags=3D0, td=3D0xc77f5300) at /usr/src/sys/kern/uipc_socket.c:836 #14 0xc05998c8 in soo_write (fp=3D0x4, uio=3D0xe70c9cb0, active_cred=3D0xc8a77180, flags=3D0, td=3D0xc77f5300) at /usr/src/sys/kern/sys_socket.c:118 #15 0xc0592ff0 in dofilewrite (td=3D0xc77f5300, fd=3D4, fp=3D0xc90d6288, auio=3D0xe70c9cb0, offset=3DUnhandled dwarf expression opcode 0x93 ) at file.h:252 #16 0xc0592e27 in kern_writev (td=3D0xc77f5300, fd=3D21, auio=3D0x4) at /usr/src/sys/kern/sys_generic.c:402 #17 0xc0592cf9 in write (td=3D0x4, uap=3D0x4) at /usr/src/sys/kern/sys_generic.c:326 #18 0xc0724423 in syscall (frame=3D {tf_fs =3D -1078001605, tf_es =3D -1078001605, tf_ds =3D -1078001605, tf_edi =3D 137218624, tf_esi =3D 137218624, tf_ebp =3D 154, tf_isp =3D -418603676, tf_ebx =3D 270840404, tf_edx =3D 137166848, tf_ecx =3D 27212700= 8, tf_eax =3D 4, tf_trapno =3D 22, tf_err =3D 2, tf_eip =3D 272063395, tf_cs = =3D 51, tf_eflags =3D 530, tf_esp =3D -1077942612, tf_ss =3D 59}) at /usr/src/sys/i386/i386/trap.c:983 #19 0xc0710b5f in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200 #20 0x00000033 in ?? () I actually did not use rc.conf to setup my ipv6 either, here is my script for activating the v6 on boot: ifconfig em0 inet6 2610:88:1::3:2 prefixlen 112 ifconfig em0 inet6 2610:88:1::3:3 prefixlen 112 route add -inet6 default 2610:88:1::3:0 What's odd is that these panics only began when my datacenter had a fire in their 'router cage', and replaced the router. I'm not sure how this is related, if at all, but the panics definitely only began on that date, on two separate machines. Bryan Drewery Administrator Xzibition Data Communications Hi, Some changes were made to sys/net/route.c and sys/netinet6/nd6.c in October to fix some routing related panics that were occurring on 7-CURRENT and 6.3. I know this fixed an issue where the system would panic when the gateway disappeared, but I think it may also have fixed the recursive mutex panic too. Are you still seeing this panic with more recent sources (e.g 7.0-RC2)? Cheers, Bruce State Changed From-To: open->feedback Bruce asked for feedback Just to note that kern/113457 (closed as a dupe of this one) shows a second way of recreating the problem. State Changed From-To: feedback->suspended unfortunately we haven't received feedback and the state of this issue is still unclear if it has been solved or not by updating to a recent version. suspend for now until either we're seeing feedback or this ticket can be closed within a reasonable timeframe. Responsible Changed From-To: freebsd-net->vwe track State Changed From-To: suspended->open Reopen. Problem not solved, happened again yesterday. Responsible Changed From-To: vwe->bz Reopen. Problem not solved, happened again yesterday. Responsible Changed From-To: bz->gnn I shall not use bugzilla (at least until we will have a CLI). For bugs matching the following criteria: Status: In Progress Changed: (is less than) 2014-06-01 Reset to default assignee and clear in-progress tags. Mail being skipped Keyword:
crash
– in lieu of summary line prefix:
[panic]
* bulk change for the keyword
* summary lines may be edited manually (not in bulk).
Keyword descriptions and search interface:
<https://bugs.freebsd.org/bugzilla/describekeywords.cgi>
|
