Bug 121555

# kgdb /usr/obj/usr/src/sys/proliant-dl385/kernel.debug /var/crash/vmcore.1
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd".

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address   = 0x258
fault code              = supervisor read data, page not present
instruction pointer     = 0x8:0xffffffff802d44de
stack pointer           = 0x10:0xffffffffabd3c910
frame pointer           = 0x10:0xffffffffabd3c930
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 12 (swi1: net)
Physical memory: 2039 MB
Dumping 404 MB: 389 373 357 341 325 309 293 277 261 245 229 213 197 181 165 149 133 117 101 85 69 53 37 21 5

#0  doadump () at pcpu.h:194
194     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) where
#0  doadump () at pcpu.h:194
#1  0xffffffff801b646c in db_fncall (dummy1=Variable "dummy1" is not available.
) at /usr/src/sys/ddb/db_command.c:486
#2  0xffffffff801b6984 in db_command_loop () at /usr/src/sys/ddb/db_command.c:401
#3  0xffffffff801b865f in db_trap (type=Variable "type" is not available.
) at /usr/src/sys/ddb/db_main.c:222
#4  0xffffffff80309b40 in kdb_trap (type=12, code=0, tf=0xffffffffabd3c860)
    at /usr/src/sys/kern/subr_kdb.c:502
#5  0xffffffff8047bb91 in trap_fatal (frame=0xffffffffabd3c860, eva=600)
    at /usr/src/sys/amd64/amd64/trap.c:719
#6  0xffffffff8047bf64 in trap_pfault (frame=0xffffffffabd3c860, usermode=0)
    at /usr/src/sys/amd64/amd64/trap.c:641
#7  0xffffffff8047c8cb in trap (frame=0xffffffffabd3c860) at /usr/src/sys/amd64/amd64/trap.c:410
#8  0xffffffff8046050e in calltrap () at /usr/src/sys/amd64/amd64/exception.S:169
#9  0xffffffff802d44de in _mtx_lock_sleep (m=0xffffff00177eacf0, tid=18446742974215541584, opts=Variable "opts" is not available.
)
    at /usr/src/sys/kern/kern_mutex.c:335
#10 0xffffffff80388176 in rt_check (lrt=0xffffffffabd3c998, lrt0=0xffffffffabd3c988,
    dst=0xffffff00171c2980) at /usr/src/sys/net/route.c:1296
#11 0xffffffff8039dc94 in arpresolve (ifp=0xffffff000160d000, rt0=0xffffff00177eac30,
    m=0xffffff000c3a7500, dst=0xffffff00171c2980, desten=0xffffffffabd3ca10 "ю╗1\001")
    at /usr/src/sys/netinet/if_ether.c:375
#12 0xffffffff80377870 in ether_output (ifp=0xffffff000160d000, m=0xffffff000c3a7500,
    dst=0xffffff00171c2980, rt0=Variable "rt0" is not available.
) at /usr/src/sys/net/if_ethersubr.c:175
#13 0xffffffff8039eb50 in arpintr (m=0xffffff000c087e00) at /usr/src/sys/netinet/if_ether.c:777
#14 0xffffffff80381f8c in netisr_processqueue (ni=0xffffffff806bff10)
    at /usr/src/sys/net/netisr.c:143
#15 0xffffffff803822dc in swi_net (dummy=Variable "dummy" is not available.
) at /usr/src/sys/net/netisr.c:256
#16 0xffffffff802c3b77 in ithread_loop (arg=0xffffff00010364a0)
    at /usr/src/sys/kern/kern_intr.c:1036
#17 0xffffffff802c0779 in fork_exit (callout=0xffffffff802c3a00 <ithread_loop>,
    arg=0xffffff00010364a0, frame=0xffffffffabd3cc80) at /usr/src/sys/kern/kern_fork.c:781
#18 0xffffffff804608de in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:415
#19 0x0000000000000000 in ?? ()
#20 0x0000000000000000 in ?? ()
#21 0x0000000000000001 in ?? ()
#22 0x0000000000000000 in ?? ()
#23 0x0000000000000000 in ?? ()
#24 0x0000000000000000 in ?? ()
#25 0x0000000000000000 in ?? ()
---Type <return> to continue, or q <return> to quit---
#26 0x0000000000000000 in ?? ()
#27 0x0000000000000000 in ?? ()
#28 0x0000000000000000 in ?? ()
#29 0x0000000000000000 in ?? ()
#30 0x0000000000000000 in ?? ()
#31 0x0000000000000000 in ?? ()
#32 0x0000000000000000 in ?? ()
#33 0x0000000000000000 in ?? ()
#34 0x0000000000000000 in ?? ()
#35 0x0000000000000000 in ?? ()
#36 0x0000000000000000 in ?? ()
#37 0x0000000000000000 in ?? ()
#38 0x0000000000000000 in ?? ()
#39 0x0000000000000000 in ?? ()
#40 0x0000000000000000 in ?? ()
#41 0x0000000000000000 in ?? ()
#42 0x0000000000000000 in ?? ()
#43 0x00000000008ba000 in ?? ()
#44 0xffffff00010cd658 in ?? ()
#45 0xffffffff806b8cc0 in tdq_groups ()
#46 0xffffff00010cd350 in ?? ()
#47 0x000000000000000b in ?? ()
#48 0xffffffffabd3c1b0 in ?? ()
#49 0xffffffffabd3c148 in ?? ()
#50 0x0000000000000001 in ?? ()
#51 0xffffffff802ff04f in sched_switch (td=0x0, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1898
Previous frame inner to this frame (corrupt stack?)

How-To-Repeat: heavy load 7.0-RELEASE network subsystem