Bug 203691 - [panic] Fatal trap 12: page fault while in kernel mode (in function vfs_cache_lookup)
Summary: [panic] Fatal trap 12: page fault while in kernel mode (in function vfs_cache...
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 10.2-RELEASE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-bugs mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-10-11 05:21 UTC by neovortex
Modified: 2017-06-22 09:02 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description neovortex 2015-10-11 05:21:57 UTC
A system that runs ZFS, a number of jails (some of which use nullfs across ZFS filesystems) and one bhyve VM is crashing every so often (crashes are usually occur with between a week and a couple of months).

Running 10.2-RELEASE-p4 on amd64 with a GENERIC kernel installed from freebsd-update, although crashes have occurred on earlier 10.x versions as well although I can't recall exact version.

Hardware has been tested (memtest and cpu stress tests) without issues. Disks are attached using a mix of AHCI and a LSI Logic SAS2008 card (mps driver).


Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 06
fault virtual address   = 0x10000002f
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff809d6e14
stack pointer           = 0x28:0xfffffe0654b534e0
frame pointer           = 0x28:0xfffffe0654b53560
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 8041 (find)
trap number             = 12
panic: page fault
cpuid = 3
KDB: stack backtrace:
#0 0xffffffff80984e30 at kdb_backtrace+0x60
#1 0xffffffff809489e6 at vpanic+0x126
#2 0xffffffff809488b3 at panic+0x43
#3 0xffffffff80d4aadb at trap_fatal+0x36b
#4 0xffffffff80d4addd at trap_pfault+0x2ed
#5 0xffffffff80d4a47a at trap+0x47a
#6 0xffffffff80d307f2 at calltrap+0x8
#7 0xffffffff809d83dc at vfs_cache_lookup+0xac
#8 0xffffffff80e71621 at VOP_LOOKUP_APV+0xa1
#9 0xffffffff809e0951 at lookup+0x5a1
#10 0xffffffff809e00b4 at namei+0x4d4
#11 0xffffffff809f471e at kern_statat_vnhook+0xae
#12 0xffffffff809f462c at sys_fstatat+0x2c
#13 0xffffffff80d4b3f7 at amd64_syscall+0x357
#14 0xffffffff80d30adb at Xfast_syscall+0xfb


(kgdb) #0  doadump (textdump=<value optimized out>) at pcpu.h:219
#1  0xffffffff80948642 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:451
#2  0xffffffff80948a25 in vpanic (fmt=<value optimized out>,
    ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:758
#3  0xffffffff809488b3 in panic (fmt=0x0)
    at /usr/src/sys/kern/kern_shutdown.c:687
#4  0xffffffff80d4aadb in trap_fatal (frame=<value optimized out>,
    eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:851
#5  0xffffffff80d4addd in trap_pfault (frame=0xfffffe0654b53430,
    usermode=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:674
#6  0xffffffff80d4a47a in trap (frame=0xfffffe0654b53430)
    at /usr/src/sys/amd64/amd64/trap.c:440
#7  0xffffffff80d307f2 in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:236
#8  0xffffffff809d6e14 in cache_lookup (dvp=0xfffff8059a1fbce8,
    vpp=0xfffffe0654b53858, cnp=0xfffffe0654b53880, tsp=0x0, ticksp=0x0)
    at /usr/src/sys/kern/vfs_cache.c:548
#9  0xffffffff809d83dc in vfs_cache_lookup (ap=<value optimized out>)
    at /usr/src/sys/kern/vfs_cache.c:1039
#10 0xffffffff80e71621 in VOP_LOOKUP_APV (vop=<value optimized out>,
    a=<value optimized out>) at vnode_if.c:129
#11 0xffffffff809e0951 in lookup (ndp=0xfffffe0654b537f8) at vnode_if.h:54
#12 0xffffffff809e00b4 in namei (ndp=0xfffffe0654b537f8)
    at /usr/src/sys/kern/vfs_lookup.c:302
#13 0xffffffff809f471e in kern_statat_vnhook (td=0xfffff805d4e01000,
    flag=<value optimized out>, fd=-100,
    path=0x801169a88 <Address 0x801169a88 out of bounds>,
    pathseg=UIO_USERSPACE, sbp=0xfffffe0654b53910, hook=0xfffffe0000dcb000)
    at /usr/src/sys/kern/vfs_syscalls.c:2298
#14 0xffffffff809f462c in sys_fstatat (td=0x7e891088, uap=0xfffffe0654b53a40)
    at /usr/src/sys/kern/vfs_syscalls.c:2278
#15 0xffffffff80d4b3f7 in amd64_syscall (td=0xfffff805d4e01000, traced=0)
    at subr_syscall.c:134
#16 0xffffffff80d30adb in Xfast_syscall ()
    at /usr/src/sys/amd64/amd64/exception.S:396
#17 0x00000008008b5f0a in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language:  auto; currently minimal
(kgdb)


I have crash dumps enabled now, although only I only have the core from the most recent crash as mentioned above. Previous crashes have had the same backtrace.
Comment 1 neovortex 2015-11-03 12:35:29 UTC
Yet another crash with an identical backtrace (which makes me suspect not hardware related which would be more random?):

Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 06
fault virtual address   = 0x10000002f
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff809d6e14
stack pointer           = 0x28:0xfffffe0654b534e0
frame pointer           = 0x28:0xfffffe0654b53560
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 8041 (find)
trap number             = 12
panic: page fault
cpuid = 3
KDB: stack backtrace:
#0 0xffffffff80984e30 at kdb_backtrace+0x60
#1 0xffffffff809489e6 at vpanic+0x126
#2 0xffffffff809488b3 at panic+0x43
#3 0xffffffff80d4aadb at trap_fatal+0x36b
#4 0xffffffff80d4addd at trap_pfault+0x2ed
#5 0xffffffff80d4a47a at trap+0x47a
#6 0xffffffff80d307f2 at calltrap+0x8
#7 0xffffffff809d83dc at vfs_cache_lookup+0xac
#8 0xffffffff80e71621 at VOP_LOOKUP_APV+0xa1
#9 0xffffffff809e0951 at lookup+0x5a1
#10 0xffffffff809e00b4 at namei+0x4d4
#11 0xffffffff809f471e at kern_statat_vnhook+0xae
#12 0xffffffff809f462c at sys_fstatat+0x2c
#13 0xffffffff80d4b3f7 at amd64_syscall+0x357
#14 0xffffffff80d30adb at Xfast_syscall+0xfb
Uptime: 8d23h59m1s
(da0:mps0:0:0:0): Synchronize cache failed
(da1:mps0:0:1:0): Synchronize cache failed
(da2:mps0:0:2:0): Synchronize cache failed
(da3:mps0:0:8:0): Synchronize cache failed
Dumping 6357 out of 24233 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
Comment 2 neovortex 2017-06-22 09:02:45 UTC
No longer relevant, affected system decommissioned.