Description Anders Nordby 2010-02-11 23:50:02 UTC

On the console:

Fatal trap 12: page fault while in kernel mode^M
cpuid = 6; apic id = 06^M
fault virtual address   = 0x110^M
fault code              = supervisor write data, page not present^M
instruction pointer     = 0x8:0xffffffff805a6286^M
stack pointer           = 0x10:0xffffffffb82e22b0^M
frame pointer           = 0x10:0xffffffffb82e23d0^M
code segment            = base rx0, limit 0xfffff, type 0x1b^M
                        = DPL 0, pres 1, long 1, def32 0, gran 1^M
processor eflags        = interrupt enabled, resume, IOPL = 0^M
current process         = 48810 (rsync)^M
trap number             = 12^M
panic: page fault^M
cpuid = 6^M
Uptime: 1d18h5m6s^M
Physical memory: 16370 MB^M
Dumping 1078 MB: 1063 1047 1031 1015 999 983 967 951 935 919 903 887 871 855 839
 823 807 791 775 759 743 727 711 695 679 663 647 631 615 599 583 567 551 535 519
 503 487 471 455 439 423 407 391 375 359 343 327 311 295 279 263 247 231 215 199
 183 167 151 135 119 103 87 71 55 39 23 7^M
Dump complete^M
Automatic reboot in 15 seconds - press a key on the console to abort^M
Rebooting...^M
cpu_reset: Stopping other CPUs^M

From kgdb:

root@vm3:/usr/obj/usr/src/sys/VM3# kgdb kernel.debug /var/crash/vmcore.1
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd".

Unread portion of the kernel message buffer:
<6>nfs send error 57 for server unixfile:/data/diintern


Fatal trap 12: page fault while in kernel mode
cpuid = 6; apic id = 06
fault virtual address	= 0x110
fault code		= supervisor write data, page not present
instruction pointer	= 0x8:0xffffffff805a6286
stack pointer	        = 0x10:0xffffffffb82e22b0
frame pointer	        = 0x10:0xffffffffb82e23d0
code segment		= base rx0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 48810 (rsync)
trap number		= 12
panic: page fault
cpuid = 6
Uptime: 1d18h5m6s
Physical memory: 16370 MB
Dumping 1078 MB: 1063 1047 1031 1015 999 983 967 951 935 919 903 887 871 855 839 823 807 791 775 759 743 727 711 695 679 663 647 631 615 599 583 567 551 535 519 503 487 471 455 439 423 407 391 375 359 343 327 311 295 279 263 247 231 215 199 183 167 151 135 119 103 87 71 55 39 23 7

#0  doadump () at pcpu.h:194
194		__asm __volatile("movq %%gs:0,%0" : "=r" (td));
(kgdb) list *0xffffffff805a6286
0xffffffff805a6286 is in tcp_ctloutput (atomic.h:142).
137	static __inline int
138	atomic_cmpset_long(volatile u_long *dst, u_long exp, u_long src)
139	{
140		u_char res;
141	
142		__asm __volatile(
143		"	" MPLOCKED "		"
144		"	cmpxchgq %2,%1 ;	"
145		"       sete	%0 ;		"
146		"1:				"
(kgdb) backtrace
#0  doadump () at pcpu.h:194
#1  0xffffffff804b5718 in boot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:409
#2  0xffffffff804b5b77 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3  0xffffffff80705dcf in trap_fatal (frame=0xc, eva=Variable "eva" is not available.
)
    at /usr/src/sys/amd64/amd64/trap.c:724
#4  0xffffffff807061b4 in trap_pfault (frame=0xffffffffb82e2200, usermode=0)
    at /usr/src/sys/amd64/amd64/trap.c:641
#5  0xffffffff80706b6a in trap (frame=0xffffffffb82e2200)
    at /usr/src/sys/amd64/amd64/trap.c:410
#6  0xffffffff806ebe5e in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:169
#7  0xffffffff805a6286 in tcp_ctloutput (so=0xffffff039f7b2828, 
    sopt=0xffffffffb82e2450) at atomic.h:142
#8  0xffffffff8050cec1 in sosetopt (so=0xffffff039f7b2828, 
    sopt=0xffffffffb82e2450) at /usr/src/sys/kern/uipc_socket.c:2144
#9  0xffffffff805c760c in nfs_connect (nmp=0xffffff028d075d50, 
    rep=0xffffff03d30f1a00) at /usr/src/sys/nfsclient/nfs_socket.c:416
#10 0xffffffff805c784f in nfs_reconnect (rep=0xffffff03d30f1a00)
    at /usr/src/sys/nfsclient/nfs_socket.c:524
#11 0xffffffff805c967b in nfs_request (vp=0xffffff0244f64ba0, mrest=Variable "mrest" is not available.
)
    at /usr/src/sys/nfsclient/nfs_socket.c:736
#12 0xffffffff805d4aad in nfs_lookup (ap=Variable "ap" is not available.
)
    at /usr/src/sys/nfsclient/nfs_vnops.c:897
#13 0xffffffff807619a0 in VOP_LOOKUP_APV (vop=0xffffffff809d50e0, 
    a=0xffffffffb82e2810) at vnode_if.c:99
#14 0xffffffff80529b31 in lookup (ndp=0xffffffffb82e2940) at vnode_if.h:57
#15 0xffffffff8052a9cc in namei (ndp=0xffffffffb82e2940)
    at /usr/src/sys/kern/vfs_lookup.c:219
#16 0xffffffff8053761e in kern_lstat (td=0xffffff0144b8d6a0, path=Variable "path" is not available.
)
    at /usr/src/sys/kern/vfs_syscalls.c:2161
#17 0xffffffff8053779a in lstat (td=Variable "td" is not available.
) at /usr/src/sys/kern/vfs_syscalls.c:2144
#18 0xffffffff80706434 in syscall (frame=0xffffffffb82e2c70)
    at /usr/src/sys/amd64/amd64/trap.c:852
#19 0xffffffff806ec06b in Xfast_syscall ()
    at /usr/src/sys/amd64/amd64/exception.S:290
#20 0x000000080075733c in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) quit

Fix: 

N/A
How-To-Repeat: Have a NFS server with lots of packet loss (40% or so). Do lots of rsync and find searches on a NFS mount from it.