Bug 122283

Summary: [ip6] [panic] Panic in ip_output related to IPv6 routes
Product: Base System Reporter: Nick Sayer <nsayer>
Component: kernAssignee: freebsd-bugs (Nobody) <bugs>
Status: Closed FIXED    
Severity: Affects Only Me    
Priority: Normal    
Version: 7.0-RELEASE   
Hardware: Any   
OS: Any   

Description Nick Sayer 2008-03-31 07:50:01 UTC
About every 7-10 days, the machine panics. Almost always the
panic is in line 169 of ip_output.c, although a recent example
had the panic in line 235. In every case, *ro was an IPv6 route,
and ro->ro_rt->rt_flags was a crap address.

I haven't really tried to tackle this one by myself yet. I'm
instead hoping that someone else has already done the heavy
lifting.

Fix: 

Unknown.
How-To-Repeat: 	
Just wait a week.
Comment 1 Volker Werth freebsd_committer freebsd_triage 2008-04-08 23:04:17 UTC
State Changed
From-To: open->feedback


submitter: Please don't be afraid to show us the kernel dump 
(panic message + backtrace) 
also an `ifconfig', `netstat -rn' and your kernel modifications may help
Comment 2 Nick Sayer 2008-04-08 23:30:49 UTC
Latest example:

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address	= 0x34
fault code		= supervisor read, page not present
instruction pointer	= 0x20:0xc06f25b4
stack pointer	        = 0x28:0xe688f798
frame pointer	        = 0x28:0xe688f804
code segment		= base rx0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, def32 1, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 1174 (imapd)
trap number		= 12
panic: page fault
cpuid = 0
Uptime: 6h12m31s
Physical memory: 1015 MB
Dumping 154 MB: 139 123 107 91 75 59 43 27 11


(kgdb) bt
#0  doadump () at pcpu.h:195
#1  0xc062e2a7 in boot (howto=260) at /usr/src/sys/kern/ 
kern_shutdown.c:409
#2  0xc062e569 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3  0xc084ce2c in trap_fatal (frame=0xe688f758, eva=52)
     at /usr/src/sys/i386/i386/trap.c:899
#4  0xc084d0b0 in trap_pfault (frame=0xe688f758, usermode=0, eva=52)
     at /usr/src/sys/i386/i386/trap.c:812
#5  0xc084da5c in trap (frame=0xe688f758) at /usr/src/sys/i386/i386/ 
trap.c:490
#6  0xc0833d3b in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7  0xc06f25b4 in ip_output (m=0xc402f400, opt=0x0, ro=0xc3ee3344,  
flags=0,
     imo=0x0, inp=0x0) at /usr/src/sys/netinet/ip_output.c:235
#8  0xc06ca751 in stf_output (ifp=0xc3ee8400, m=0xc402f400,  
dst=0xe688fa0c,
     rt=0xc3fd6d20) at /usr/src/sys/net/if_stf.c:533
#9  0xc077772d in nd6_output (ifp=0xc3ee8400, origifp=0xc3ee8400,
     m0=0xc3e8b200, dst=0xe688fa0c, rt0=0xc3fd6d20)
     at /usr/src/sys/netinet6/nd6.c:2123
#10 0xc07749f2 in ip6_output (m0=0xc3e8b200, opt=0x0, ro=0xe688fa08,  
flags=0,
     im6o=0x0, ifpp=0x0, inp=0xc461f654)
     at /usr/src/sys/netinet6/ip6_output.c:927
#11 0xc0750c21 in tcp_output (tp=0xc4a253a0)
     at /usr/src/sys/netinet/tcp_output.c:1114
#12 0xc075af7a in tcp_usr_send (so=0xc48924a4, flags=Variable "flags"  
is not available.
)
     at /usr/src/sys/netinet/tcp_usrreq.c:843
---Type <return> to continue, or q <return> to quit---
#13 0xc0681785 in sosend_generic (so=0xc48924a4, addr=0x0,  
uio=0xe688fc60,
     top=0xc402f300, control=0x0, flags=0, td=0xc4a0c210)
     at /usr/src/sys/kern/uipc_socket.c:1240
#14 0xc067d74f in sosend (so=0xc48924a4, addr=0x0, uio=0xe688fc60,  
top=0x0,
     control=0x0, flags=0, td=0xc4a0c210)
     at /usr/src/sys/kern/uipc_socket.c:1286
#15 0xc0667d4b in soo_write (fp=0xc43ba288, uio=0xe688fc60,
     active_cred=0xc4640400, flags=0, td=0xc4a0c210)
     at /usr/src/sys/kern/sys_socket.c:103
#16 0xc06613f7 in dofilewrite (td=0xc4a0c210, fd=1, fp=0xc43ba288,
     auio=0xe688fc60, offset=-1, flags=0) at file.h:254
#17 0xc06616d8 in kern_writev (td=0xc4a0c210, fd=1, auio=0xe688fc60)
     at /usr/src/sys/kern/sys_generic.c:401
#18 0xc066174f in write (td=0xc4a0c210, uap=0xe688fcfc)
     at /usr/src/sys/kern/sys_generic.c:317
#19 0xc084d405 in syscall (frame=0xe688fd38)
     at /usr/src/sys/i386/i386/trap.c:1035
#20 0xc0833da0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/ 
exception.s:196
#21 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)

ifconfig -au
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu  
1500
	options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
	ether 00:17:31:e9:bc:66
	inet6 fe80::217:31ff:fee9:bc66%re0 prefixlen 64 scopeid 0x1
	inet 71.141.64.1 netmask 0xfffffff0 broadcast 71.141.64.15
	inet6 2002:478d:4001:0:217:31ff:fee9:bc66 prefixlen 64
	inet6 2002:478d:4001:: prefixlen 64 anycast
	media: Ethernet autoselect (1000baseTX <full-duplex>)
	status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
	inet 127.0.0.1 netmask 0xff000000
stf0: flags=1<UP> metric 0 mtu 1280
	inet6 2002:478d:4001::1 prefixlen 16


Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif  
Expire
default            71.141.64.14       UGS         0  4858740    re0
71.141.64.0/28     link#1             UC          0        0    re0
71.141.64.1        00:17:31:e9:bc:66  UHLW        1      558    lo0
71.141.64.2        00:1b:63:f4:52:c8  UHLW        2   181265    re0     
920
71.141.64.14       00:02:3b:02:a7:51  UHLW        2        0    re0    
1198
127.0.0.1          127.0.0.1          UH          0  1736581    lo0

Internet6:
Destination                       Gateway                        
Flags      Netif Expire
::/96                             ::1                            
UGRS        lo0 =>
default                           2002:c058:6301::               
UGS        stf0
::1                               ::1                            
UHL         lo0
::ffff:0.0.0.0/96                 ::1                            
UGRS        lo0
2002::/24                         ::1                            
UGRS        lo0 =>
2002::/16                         2002:478d:4001::1              
U          stf0
2002:478d:4001::                  00:17:31:e9:bc:66              
UHL         lo0 =>
2002:478d:4001::/64               link#1                         
UC          re0
2002:478d:4001::1                 link#3                         
UHL         lo0
2002:478d:4001:0:217:31ff:fee9:bc66 00:17:31:e9:bc:66              
UHL         lo0
2002:7f00::/24                    ::1                            
UGRS        lo0
2002:e000::/20                    ::1                            
UGRS        lo0
2002:ff00::/24                    ::1                            
UGRS        lo0
fe80::/10                         ::1                            
UGRS        lo0
fe80::%re0/64                     link#1                         
UC          re0
fe80::217:31ff:fee9:bc66%re0      00:17:31:e9:bc:66              
UHL         lo0
fe80::%lo0/64                     fe80::1%lo0                    
U           lo0
fe80::1%lo0                       link#2                         
UHL         lo0
ff01:1::/32                       link#1                         
UC          re0
ff01:2::/32                       ::1                            
UC          lo0
ff02::/16                         ::1                            
UGRS        lo0
ff02::%re0/32                     link#1                         
UC          re0
ff02::%lo0/32                     ::1                            
UC          lo0
Comment 3 Pekka Savola 2008-05-13 12:04:31 UTC
FYI,

I got hit by this after upgrading a dual-CPU system (6to4 relay) from 6.3 to 
7.0.  At the same time I enabled SMP.  I'm going to try to disable SMP but I 
don't know if it helps.

The backtrace seems somewhat similar, unfortunately I have less information 
than you do:

Tracing pid 12 tid 100003 td 0xc4d08880
ip_output(c5434700,0,c5001804,0,0,...) at ip_output+0x15c
stf_output(c4fd3c00,c5430a00,c08166c4,c51ced20,0,...) at stf_output+0x431
nd6_output(c4fd3c00,c4fd3c00,c5430a00,c08166c4,c51ced20,...) at 
nd6_output+0x70d
ip6_forward(c5430a00,0,10,1,0,...) at ip6_forward+0x88d
ip6_input(c5430a00,c058019e,1,93472,c4d08880,...) at ip6_input+0xd7e
netisr_processqueue(1,e0633985,202,1000000,c4d08880,...) at 
netisr_processqueue+0xcd
swi_net(0,0,c0788ef0,46b,0,...) at swi_net+0xbe
ithread_loop(c4cc58f0,e529dd38,0,0,0,...) at ithread_loop+0x1ab
fork_exit(c056d8e0,c4cc58f0,e529dd38) at fork_exit+0x99
fork_trampoline() at fork_trampoline+0x8

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
Comment 4 Nick Sayer 2008-06-06 18:10:56 UTC
That stack trace looks like one of the alternate stack traces I have  
observed. With the debugging symbols, it turns out to be in line 518  
of if_stf.c, which says

RTFREE(sc->sc_ro.ro_rt);

which, once again, points back to something being pooched in the route  
table.

The full stack trace is

#0  doadump () at pcpu.h:195
#1  0xc062e277 in boot (howto=260) at /usr/src/sys/kern/ 
kern_shutdown.c:409
#2  0xc062e539 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3  0xc084cdfc in trap_fatal (frame=0xe683e7cc, eva=76)
     at /usr/src/sys/i386/i386/trap.c:899
#4  0xc084d080 in trap_pfault (frame=0xe683e7cc, usermode=0, eva=76)
     at /usr/src/sys/i386/i386/trap.c:812
#5  0xc084da2c in trap (frame=0xe683e7cc) at /usr/src/sys/i386/i386/ 
trap.c:490
#6  0xc0833d0b in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7  0xc06ca63b in stf_output (ifp=0xc3ef8800, m=0xc75bcc00,  
dst=0xe683ea0c,
     rt=0xc3fbaca8) at /usr/src/sys/net/if_stf.c:518
#8  0xc07776fd in nd6_output (ifp=0xc3ef8800, origifp=0xc3ef8800,
     m0=0xc6386700, dst=0xe683ea0c, rt0=0xc3fbaca8)
     at /usr/src/sys/netinet6/nd6.c:2123
#9  0xc07749c2 in ip6_output (m0=0xc6386700, opt=0x0, ro=0xe683ea08,  
flags=0,
     im6o=0x0, ifpp=0x0, inp=0xc49ddb40)
     at /usr/src/sys/netinet6/ip6_output.c:927
#10 0xc0750bf1 in tcp_output (tp=0xc5fb2570)
     at /usr/src/sys/netinet/tcp_output.c:1114
#11 0xc075af4a in tcp_usr_send (so=0xc4ae5948, flags=Variable "flags"  
is not available.
)
     at /usr/src/sys/netinet/tcp_usrreq.c:843
#12 0xc0681755 in sosend_generic (so=0xc4ae5948, addr=0x0,  
uio=0xe683ec60,
     top=0xc3fc4300, control=0x0, flags=0, td=0xc4679630)
     at /usr/src/sys/kern/uipc_socket.c:1240
#13 0xc067d71f in sosend (so=0xc4ae5948, addr=0x0, uio=0xe683ec60,  
top=0x0,
     control=0x0, flags=0, td=0xc4679630)
     at /usr/src/sys/kern/uipc_socket.c:1286
#14 0xc0667d1b in soo_write (fp=0xc4add708, uio=0xe683ec60,
     active_cred=0xc4601d00, flags=0, td=0xc4679630)
     at /usr/src/sys/kern/sys_socket.c:103
#15 0xc06613c7 in dofilewrite (td=0xc4679630, fd=1, fp=0xc4add708,
     auio=0xe683ec60, offset=-1, flags=0) at file.h:254
#16 0xc06616a8 in kern_writev (td=0xc4679630, fd=1, auio=0xe683ec60)
     at /usr/src/sys/kern/sys_generic.c:401
#17 0xc066171f in write (td=0xc4679630, uap=0xe683ecfc)
     at /usr/src/sys/kern/sys_generic.c:317
#18 0xc084d3d5 in syscall (frame=0xe683ed38)
     at /usr/src/sys/i386/i386/trap.c:1035
#19 0xc0833d70 in Xint0x80_syscall () at /usr/src/sys/i386/i386/ 
exception.s:196
#20 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)

print *sc says

$2 = {sc_ifp = 0xc3ef8800, __sc_ro46 = {__sc_ro4 = {ro_rt = 0xc3fba9d8,
       ro_dst = {sa_len = 16 '\020', sa_family = 2 '\002',
         sa_data = "\000\000G\215@\002\000\000\000\000\000\000\000"}},
     __sc_ro6 = {ro_rt = 0xc3fba9d8, ro_dst = {sin6_len = 16 '\020',
         sin6_family = 2 '\002', sin6_port = 0, sin6_flowinfo =  
37784903,
         sin6_addr = {__u6_addr = {__u6_addr8 = '\0' <repeats 15 times>,
             __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0,  
0, 0,
               0}}}, sin6_scope_id = 0}}}, encap_cookie = 0xc3ef7c00}

but print sc->sc_ro.ro_rt says that there is no member sc_ro in sc.

I'll hang on to this core dump in the hopes that someone will want to  
examine it.
Comment 5 Nick Sayer 2008-06-28 23:53:15 UTC
It appears that this is definitely SMP related. Setting  
kern.smp.disabled=1 appears to keep this from happening.
Comment 6 Nick Sayer 2008-07-07 19:39:48 UTC
Well, I have kern.smp.disabled=1, but I had the machine panic over the  
weekend despite that. The stack trace is, more or less, the same as  
all the rest of them related to this problem.

Perhaps it's not SMP related, then?
Comment 7 Pekka Savola 2008-08-21 09:02:42 UTC
FYI,

I've just updated to a newer version of 7.0-STABLE (about Mon Aug 18 
22:56:38 EEST 2008), and when I tried re-enabling SMP, I think I hit 
the same, or very similar thing (the line is slightly different) 
again:

(kgdb) up 7
#7  0xc065450f in ip_output (m=0xc551a200, opt=0x0, ro=0xc5037344, flags=0, imo=0x0, inp=0x0) at /usr/src/sys/netinet/ip_output.c:259
259                     mtu = ro->ro_rt->rt_rmx.rmx_mtu;

(kgdb) print *m
$6 = {m_hdr = {mh_next = 0xc5514300, mh_nextpkt = 0x0, mh_data = 
0xc551a2ec "E", mh_len = 20, mh_flags = 2, mh_type = 1, pad = "\000"},
   M_dat = {MH = {MH_pkthdr = {rcvif = 0xc4e4f800, header = 0x0, len = 
80, csum_flags = 0, csum_data = 0, tso_segsz = 0, ether_vtag = 0,
         tags = {slh_first = 0x0}}, MH_dat = {MH_ext = {ext_buf = 
0x1c000000 <Address 0x1c000000 out of bounds>, ext_free = 0x60,
           ext_args = 0x7f062000, ext_size = 288, ref_cnt = 0x509e3741, 
ext_type = -1808119544},
  [[ removed MH_databuf and M_databuf here ]]

Is the '<address 0x1c000000 out of bounds>' relevant here?  If not, 
I'm not seeing anything very relevant here, except perhaps locking 
problems.

(kgdb) print *ro
$1 = {ro_rt = 0xc51ed000, ro_dst = {sa_len = 16 '\020', sa_family = 2 '\002', sa_data = "\000\000ÉO]á\000\000\000\000\000\000\000"}}

(kgdb) print *ro->ro_rt
$3 = {rt_nodes = {{rn_mklist = 0xc4e5abf0, rn_parent = 0xc4fc1434, 
rn_bit = -1, rn_bmask = 0 '\0', rn_flags = 4 '\004', rn_u = {
         rn_leaf = {rn_Key = 0xc4f9f960 "\020\002", rn_Mask = 
0xc4e57800 "", rn_Dupedkey = 0x0}, rn_node = {rn_Off = -990250656,
           rn_L = 0xc4e57800, rn_R = 0x0}}}, {rn_mklist = 0x0, 
rn_parent = 0x0, rn_bit = 0, rn_bmask = 0 '\0', rn_flags = 0 '\0', 
rn_u = {
         rn_leaf = {rn_Key = 0x0, rn_Mask = 0x0, rn_Dupedkey = 0x0}, 
rn_node = {rn_Off = 0, rn_L = 0x0, rn_R = 0x0}}}},
   rt_gateway = 0xc4f9f970, rt_flags = 2051, rt_ifp = 0xc4dd3400, 
rt_ifa = 0xc506ce00, rt_rmx = {rmx_mtu = 1500, rmx_expire = 0,
     rmx_pksent = 346345}, rt_refcnt = 1, rt_genmask = 0x0, rt_llinfo = 
0x0, rt_gwroute = 0xc51dfe88, rt_parent = 0x0, rt_fibnum = 0,
   rt_mtx = {lock_object = {lo_name = 0xc0788254 "rtentry", lo_type = 
0xc0788254 "rtentry", lo_flags = 21168128, lo_witness_data = {
         lod_list = {stqe_next = 0x0}, lod_witness = 0x0}}, mtx_lock = 
4, mtx_recurse = 0}}

Therein is "rt_rmx = {rmx_mtu = 1500, rmx_expire = 0, rmx_pksent = 
346345}".

Also:

When I disabled SMP and recompiled, I haven't hit this again.  On the 
other hand, I've hit various other memory corruption problems on a 
less frequent basis.

==================

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x40
fault code              = supervisor read, page not present
instruction pointer     = 0x20:0xc065450f
stack pointer           = 0x28:0xe530c9c0
frame pointer           = 0x28:0xe530ca30
code segment            = base rx0, limit 0xfffff, type 0x1b
                          = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 14 (swi1: net)
trap number             = 12
panic: page fault
cpuid = 0
Uptime: 4m24s
Physical memory: 2039 MB
Dumping 67 MB: 52 36 20 4

#0  doadump () at pcpu.h:195
195             __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) bt
#0  doadump () at pcpu.h:195
#1  0xc058bc37 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
#2  0xc058bef9 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:572
#3  0xc073a48c in trap_fatal (frame=0xe530c980, eva=64) at /usr/src/sys/i386/i386/trap.c:899
#4  0xc073a710 in trap_pfault (frame=0xe530c980, usermode=0, eva=64) at /usr/src/sys/i386/i386/trap.c:812
#5  0xc073b08c in trap (frame=0xe530c980) at /usr/src/sys/i386/i386/trap.c:490
#6  0xc0720b1b in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7  0xc065450f in ip_output (m=0xc551a200, opt=0x0, ro=0xc5037344, flags=0, imo=0x0, inp=0x0) at /usr/src/sys/netinet/ip_output.c:259
#8  0xc0628e26 in stf_output (ifp=0xc5070c00, m=0xc551a200, dst=0xc08078e4, rt=0xc51de364) at /usr/src/sys/net/if_stf.c:537
#9  0xc068708d in nd6_output (ifp=0xc5070c00, origifp=0xc5070c00, m0=0xc5514300, dst=0xc08078e4, rt0=0xc51de364)
      at /usr/src/sys/netinet6/nd6.c:2123
#10 0xc067c0bd in ip6_forward (m=0xc5514300, srcrt=0) at /usr/src/sys/netinet6/ip6_forward.c:605
#11 0xc067e0ee in ip6_input (m=0xc5514300) at /usr/src/sys/netinet6/ip6_input.c:717
#12 0xc062b87d in netisr_processqueue (ni=0xc0800d64) at /usr/src/sys/net/netisr.c:143
#13 0xc062bb0e in swi_net (dummy=0x0) at /usr/src/sys/net/netisr.c:250
#14 0xc056c31b in ithread_loop (arg=0xc4cc58d0) at /usr/src/sys/kern/kern_intr.c:1088
#15 0xc0568eb9 in fork_exit (callout=0xc056c160 <ithread_loop>, arg=0xc4cc58d0, frame=0xe530cd38) at /usr/src/sys/kern/kern_fork.c:781
#16 0xc0720b90 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205
Comment 8 Pekka Savola 2008-08-21 09:11:28 UTC
FYI,

Here's another, slightly different, crash also with SMP, which occurs 
in the same place as Nick's first crash:

(kgdb) up 7
#7  0xc065427c in ip_output (m=0xc51ef800, opt=0x0, ro=0xc50fec84, 
flags=0, imo=0x0, inp=0x0) at /usr/src/sys/netinet/ip_output.c:171
171                     RTFREE(ro->ro_rt);

(kgdb) list
166              * cache with IPv6.
167              */
168             if (ro->ro_rt && ((ro->ro_rt->rt_flags & RTF_UP) == 0 ||
169                               dst->sin_family != AF_INET ||
170                               dst->sin_addr.s_addr != ip->ip_dst.s_addr)) {
171                     RTFREE(ro->ro_rt);
172                     ro->ro_rt = (struct rtentry *)NULL;
173             }
174     #ifdef IPFIREWALL_FORWARD
175             if (ro->ro_rt == NULL && fwd_tag == NULL) {


(kgdb) print *ro
$1 = {ro_rt = 0x0, ro_dst = {sa_len = 16 '\020', sa_family = 2 '\002', 
sa_data = "\000\000SYB\224\000\000\000\000\000\000\000"}}

so ro->ro_rt is zero, and RTFREE is doing locking here which gives a 
hint why SMP might be a factor here.

This is a rather busy box also running Teredo relay (5-10kpps).  I get 
hit by this crash in minutes or hours if SMP is enabled.

=========================

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x4c
fault code              = supervisor read, page not present
instruction pointer     = 0x20:0xc065427c
stack pointer           = 0x28:0xe7781788
frame pointer           = 0x28:0xe77817f8
code segment            = base rx0, limit 0xfffff, type 0x1b
                          = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 929 (miredo)
trap number             = 12
panic: page fault
cpuid = 0
Uptime: 16m9s
Physical memory: 2039 MB
Dumping 176 MB: 161 145 129 113 97 81 65 49 33 17 1

#0  doadump () at pcpu.h:195
195             __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) bt
#0  doadump () at pcpu.h:195
#1  0xc058bc37 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
#2  0xc058bef9 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:572
#3  0xc073a48c in trap_fatal (frame=0xe7781748, eva=76)
      at /usr/src/sys/i386/i386/trap.c:899
#4  0xc073a710 in trap_pfault (frame=0xe7781748, usermode=0, eva=76)
      at /usr/src/sys/i386/i386/trap.c:812
#5  0xc073b08c in trap (frame=0xe7781748) at /usr/src/sys/i386/i386/trap.c:490
#6  0xc0720b1b in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7  0xc065427c in ip_output (m=0xc51ef800, opt=0x0, ro=0xc50fec84, flags=0,
      imo=0x0, inp=0x0) at /usr/src/sys/netinet/ip_output.c:171
#8  0xc0628e26 in stf_output (ifp=0xc4fd6c00, m=0xc51ef800, dst=0xe7781a00,
      rt=0xc51da8b8) at /usr/src/sys/net/if_stf.c:537
#9  0xc068708d in nd6_output (ifp=0xc4fd6c00, origifp=0xc4fd6c00,
      m0=0xc51ef800, dst=0xe7781a00, rt0=0xc51da8b8)
      at /usr/src/sys/netinet6/nd6.c:2123
#10 0xc0684342 in ip6_output (m0=0xc51ef800, opt=0x0, ro=0xe77819fc, flags=0,
      im6o=0x0, ifpp=0xe7781a80, inp=0xc52cb924)
      at /usr/src/sys/netinet6/ip6_output.c:944
#11 0xc068f4cb in rip6_output (m=0xc51ef800)
      at /usr/src/sys/netinet6/raw_ip6.c:448
#12 0xc068fad8 in rip6_send (so=0xc52d51a0, flags=0, m=0xc51ef800,
      nam=0xc5007960, control=0x0, td=0xc52ec000)
---Type <return> to continue, or q <return> to quit---
      at /usr/src/sys/netinet6/raw_ip6.c:790
#13 0xc05e30a5 in sosend_generic (so=0xc52d51a0, addr=0xc5007960,
      uio=0xe7781be8, top=0xc51ef800, control=0x0, flags=0, td=0xc52ec000)
      at /usr/src/sys/kern/uipc_socket.c:1246
#14 0xc05debbf in sosend (so=0xc52d51a0, addr=0xc5007960, uio=0xe7781be8,
      top=0x0, control=0x0, flags=0, td=0xc52ec000)
      at /usr/src/sys/kern/uipc_socket.c:1292
#15 0xc05e5856 in kern_sendit (td=0xc52ec000, s=6, mp=0xe7781c64, flags=0,
      control=0x0, segflg=UIO_USERSPACE) at /usr/src/sys/kern/uipc_syscalls.c:805
#16 0xc05e81b2 in sendit (td=0xc52ec000, s=6, mp=0xe7781c64, flags=0)
      at /usr/src/sys/kern/uipc_syscalls.c:742
#17 0xc05e83ef in sendto (td=0xc52ec000, uap=0xe7781cfc)
      at /usr/src/sys/kern/uipc_syscalls.c:857
#18 0xc073aa49 in syscall (frame=0xe7781d38)
      at /usr/src/sys/i386/i386/trap.c:1035
#19 0xc0720b80 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:196
#20 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)





-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
Comment 9 dwmalone freebsd_committer freebsd_triage 2008-09-25 13:35:12 UTC
State Changed
From-To: feedback->patched

I've committed some changes to -current that I think should help 
with this. I'll try to MFC them before 7.1. 

David.
Comment 10 dfilter service freebsd_committer freebsd_triage 2008-09-25 13:35:18 UTC
dwmalone    2008-09-25 12:35:01 UTC

  FreeBSD src repository

  Modified files:
    sys/net              if_stf.c 
  Log:
  SVN rev 183351 on 2008-09-25 12:35:01Z by dwmalone
  
  Some people's 6to4 routers seem to have been blowing up because of
  the unlocked route caching in if_stf. Add a mutex that protects
  access to cached route. This seemed to fix problems for Pekka Savola.
  
  Nick Sayer had similar problems, and in his case completly disabling
  the route cache seemed to help. Add a sysctl net.link.stf.route_cache
  that can be used to turn off route caching in if_stf.
  
  PR:             122283
  MFC after:      2 weeks
  Tested by:      Pekka Savola, Nick Sayer.
  
  Revision  Changes    Path
  1.64      +30 -6     src/sys/net/if_stf.c
_______________________________________________
cvs-all@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/cvs-all
To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
Comment 11 Volker Werth freebsd_committer freebsd_triage 2008-10-27 23:14:02 UTC
State Changed
From-To: patched->closed

patched and MFC'd to RELENG_7: rev 183893