Bug 203100

Summary: kernel: Fatal trap 12: page fault while in kernel mode (__vfs_statfs)
Product: Base System Reporter: Daniel Neades <djn>
Component: kernAssignee: freebsd-bugs (Nobody) <bugs>
Status: New ---    
Severity: Affects Only Me CC: djn
Priority: ---    
Version: 10.2-RELEASE   
Hardware: amd64   
OS: Any   

Description Daniel Neades 2015-09-14 13:27:17 UTC 
I have seen this same trap twice over the last few days using the 10.2-RELEASE GENERIC kernel, though I cannot reproduce it at will. On both occasions, the trap occurred while running an 'iocage -f destroy' on a jail – iocage is a jail management utility (see https://github.com/iocage/iocage) implemented using /bin/sh, and I don’t believe is doing anything too esoteric.

First time:

Sep 12 09:09:57 bhs-0 kernel: Fatal trap 12: page fault while in kernel mode
Sep 12 09:09:57 bhs-0 kernel: cpuid = 5; apic id = 05
Sep 12 09:09:57 bhs-0 kernel: fault virtual address     = 0x38
Sep 12 09:09:58 bhs-0 kernel: fault code                = supervisor read data, page not present
Sep 12 09:09:58 bhs-0 kernel: instruction pointer       = 0x20:0xffffffff809e58fd
Sep 12 09:09:58 bhs-0 kernel: stack pointer             = 0x28:0xfffffe2021dc64c0
Sep 12 09:09:58 bhs-0 kernel: frame pointer             = 0x28:0xfffffe2021dc64f0
Sep 12 09:09:58 bhs-0 kernel: code segment              = base rx0, limit 0xfffff, type 0x1b
Sep 12 09:09:58 bhs-0 kernel: = DPL 0, pres 1, long 1, def32 0, gran 1
Sep 12 09:09:58 bhs-0 kernel: processor eflags  =

Second time:

Sep 14 12:57:19 bhs-0 kernel: Fatal trap 12: page fault while in kernel mode
Sep 14 12:57:19 bhs-0 kernel: cpuid = 12; apic id = 0c
Sep 14 12:57:19 bhs-0 kernel: fault virtual address     = 0x38
Sep 14 12:57:20 bhs-0 kernel: fault code                = supervisor read data, page not present
Sep 14 12:57:20 bhs-0 kernel: instruction pointer       = 0x20:0xffffffff809e58fd
Sep 14 12:57:20 bhs-0 kernel: stack pointer             = 0x28:0xfffffe2021db24c0
Sep 14 12:57:20 bhs-0 kernel: frame pointer             = 0x28:0xfffffe2021db24f0
Sep 14 12:57:20 bhs-0 kernel: code segment              = base rx0, limit 0xfffff, type 0x1b
Sep 14 12:57:20 bhs-0 kernel: = DPL 0, pres 1, long 1, def32 0, gran 1

nm suggests the problem may be in __vfs_statfs:

# nm -n /boot/kernel/kernel.symbols |grep ffffffff809e58f
ffffffff809e58f0 T __vfs_statfs

I did not have a dumpdev configured, so unfortunately I do not have a crash dump. I have now configured dumpdev, and will endeavour to capture a crash dump if this problem happens again (note, however, that I only have 4GB of swap space and 128 GB of RAM, so I’m not sure whether a crash dump will be successful).

Here’s the relevant chunk of dmesg output describing my hardware:

Copyright (c) 1992-2015 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 12 15:26:37 UTC 2015
    root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
CPU: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz (2600.05-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306e4  Family=0x6  Model=0x3e  Stepping=4
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7fbee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  Structured Extended Features=0x281<FSGSBASE,SMEP,ERMS>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics
real memory  = 137447342080 (131080 MB)
avail memory = 133482815488 (127299 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: < >
FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
FreeBSD/SMP: 1 package(s) x 8 core(s) x 2 SMT threads
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
 cpu4 (AP): APIC ID:  4
 cpu5 (AP): APIC ID:  5
 cpu6 (AP): APIC ID:  6
 cpu7 (AP): APIC ID:  7
 cpu8 (AP): APIC ID:  8
 cpu9 (AP): APIC ID:  9
 cpu10 (AP): APIC ID: 10
 cpu11 (AP): APIC ID: 11
 cpu12 (AP): APIC ID: 12
 cpu13 (AP): APIC ID: 13
 cpu14 (AP): APIC ID: 14
 cpu15 (AP): APIC ID: 15
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
random: <Software, Yarrow> initialized
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-XTS> on motherboard
acpi0: <SUPERM SMCI--MB> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
cpu8: <ACPI CPU> on acpi0
cpu9: <ACPI CPU> on acpi0
cpu10: <ACPI CPU> on acpi0
cpu11: <ACPI CPU> on acpi0
cpu12: <ACPI CPU> on acpi0
cpu13: <ACPI CPU> on acpi0
cpu14: <ACPI CPU> on acpi0
cpu15: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz quality 340
Event timer "HPET3" frequency 14318180 Hz quality 340
Event timer "HPET4" frequency 14318180 Hz quality 340
Event timer "HPET5" frequency 14318180 Hz quality 340
Event timer "HPET6" frequency 14318180 Hz quality 340
Event timer "HPET7" frequency 14318180 Hz quality 340
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 27 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.8.3> port 0xe020-0xe03f mem 0xfa200000-0xfa3fffff,0xfa404000-0xfa407fff irq 16 at device 0.0 on pci1
ix0: Using MSIX interrupts with 9 vectors
ix0: Ethernet address: 0c:c4:7a:1d:a1:5a
ix0: PCI Express Bus: Speed 5.0GT/s Width x8
ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.8.3> port 0xe000-0xe01f mem 0xfa000000-0xfa1fffff,0xfa400000-0xfa403fff irq 17 at device 0.1 on pci1
ix1: Using MSIX interrupts with 9 vectors
ix1: Ethernet address: 0c:c4:7a:1d:a1:5b
ix1: PCI Express Bus: Speed 5.0GT/s Width x8
pcib2: <ACPI PCI-PCI bridge> irq 33 at device 2.0 on pci0
pci3: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 41 at device 3.0 on pci0
pci4: <ACPI PCI bus> on pcib3
mps0: <Avago Technologies (LSI) SAS2308> port 0xd000-0xd0ff mem 0xfbe40000-0xfbe4ffff,0xfbe00000-0xfbe3ffff irq 40 at device 0.0 on pci4
mps0: Firmware: 19.00.00.00, Driver: 20.00.00.00-fbsd
mps0: IOCCapabilities: 4185c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR>
pcib4: <ACPI PCI-PCI bridge> irq 41 at device 3.2 on pci0
pci5: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 17.0 on pci0
pci6: <ACPI PCI bus> on pcib5
pci0: <simple comms> at device 22.0 (no driver attached)
pci0: <simple comms> at device 22.1 (no driver attached)
ehci0: <Intel Patsburg USB 2.0 controller> mem 0xfbf23000-0xfbf233ff irq 16 at device 26.0 on pci0
usbus0: EHCI version 1.0
usbus0 on ehci0
ehci1: <Intel Patsburg USB 2.0 controller> mem 0xfbf22000-0xfbf223ff irq 23 at device 29.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci1
pcib6: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci7: <ACPI PCI bus> on pcib6
vgapci0: <VGA-compatible display> mem 0xf9000000-0xf9ffffff,0xfb800000-0xfb803fff,0xfb000000-0xfb7fffff irq 16 at device 4.0 on pci7
vgapci0: Boot video device
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel Patsburg AHCI SATA controller> port 0xf050-0xf057,0xf040-0xf043,0xf030-0xf037,0xf020-0xf023,0xf000-0xf01f mem 0xfbf21000-0xfbf217ff irq 18 at device 31.2 on pci0
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ahciem0: <AHCI enclosure management bridge> on ahci0
pcib7: <ACPI Host-PCI bridge> on acpi0
pci255: <ACPI PCI bus> on pcib7
acpi_button0: <Power Button> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart2: <16550 or compatible> port 0x3e8-0x3ef irq 10 on acpi0
uart2: console (115200,n,8,1)
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xce000-0xcefff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: cannot reserve I/O port range
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
est2: <Enhanced SpeedStep Frequency Control> on cpu2
est3: <Enhanced SpeedStep Frequency Control> on cpu3
est4: <Enhanced SpeedStep Frequency Control> on cpu4
est5: <Enhanced SpeedStep Frequency Control> on cpu5
est6: <Enhanced SpeedStep Frequency Control> on cpu6
est7: <Enhanced SpeedStep Frequency Control> on cpu7
est8: <Enhanced SpeedStep Frequency Control> on cpu8
est9: <Enhanced SpeedStep Frequency Control> on cpu9
est10: <Enhanced SpeedStep Frequency Control> on cpu10
est11: <Enhanced SpeedStep Frequency Control> on cpu11
est12: <Enhanced SpeedStep Frequency Control> on cpu12
est13: <Enhanced SpeedStep Frequency Control> on cpu13
est14: <Enhanced SpeedStep Frequency Control> on cpu14
est15: <Enhanced SpeedStep Frequency Control> on cpu15
random: unblocking device.
usbus0: 480Mbps High Speed USB v2.0
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
usbus1: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
ugen0.2: <vendor 0x8087> at usbus0
uhub2: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0
ugen1.2: <vendor 0x8087> at usbus1
uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus1
mps0: SAS Address for SATA device = db235358b6fcdd88
mps0: SAS Address from SATA device = db235358b6fcdd88
mps0: SAS Address for SATA device = db235358b6fdb684
mps0: SAS Address from SATA device = db235358b6fdb684
mps0: SAS Address for SATA device = db235358b6fdb572
mps0: SAS Address from SATA device = db235358b6fdb572
mps0: SAS Address for SATA device = db235358b6fdd687
mps0: SAS Address from SATA device = db235358b6fdd687
uhub2: 6 ports with 6 removable, self powered
uhub3: 8 ports with 8 removable, self powered
ugen0.3: <Winbond Electronics Corp> at usbus0
ukbd0: <Winbond Electronics Corp Hermon USB hidmouse Device, class 0/0, rev 1.10/0.01, addr 3> on usbus0
kbd2 at ukbd0
ugen1.3: <Kingston> at usbus1
umass0: <Kingston DataTraveler 3.0, class 0/0, rev 2.10/1.00, addr 3> on usbus1
umass0:  SCSI over Bulk-Only; quirks = 0x8100
umass0:8:0:-1: Attached to scbus8
ses0 at ahciem0 bus 0 scbus7 target 0 lun 0
ses0: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device
ses0: SEMB SES Device
da0 at mps0 bus 0 scbus0 target 2 lun 0
da1 at mps0 bus 0 scbus0 target 3 lun 0
da2 at mps0 bus 0 scbus0 target 4 lun 0
da0: <ATA INTEL SSDSC2BB80 0380> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number BTWH507100ZH800CGN  
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da1: <ATA INTEL SSDSC2BB80 0380> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number BTWH5071013D800CGN  
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 763097MB (1562824368 512 byte sectors: 255H 63S/T 97281C)
da2: <ATA INTEL SSDSC2BB80 0380> Fixed Direct Access SPC-4 SCSI device
da0: 763097MB (1562824368 512 byte sectors: 255H 63S/T 97281C)
da3 at mps0 bus 0 scbus0 target 5 lun 0
da3: <ATA INTEL SSDSC2BB80 0380> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number BTWH507101SG800CGN  
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 763097MB (1562824368 512 byte sectors: 255H 63S/T 97281C)
da2: Serial Number BTWH50710122800CGN  
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 763097MB (1562824368 512 byte sectors: 255H 63S/T 97281C)
da4 at umass-sim0 bus 0 scbus8 target 0 lun 0
da4: <Kingston DataTraveler 3.0 PMAP> Removable Direct Access SPC-4 SCSI device
da4: Serial Number 50E549C692DBEF51594F1BC1
da4: 40.000MB/s transfers
da4: 15004MB (30728832 512 byte sectors: 255H 63S/T 1912C)
da4: quirks=0x2<NO_6_BYTE>
SMP: AP CPU #1 Launched!
SMP: AP CPU #9 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #15 Launched!
SMP: AP CPU #13 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #10 Launched!
SMP: AP CPU #12 Launched!
SMP: AP CPU #8 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #11 Launched!
SMP: AP CPU #14 Launched!
SMP: AP CPU #4 Launched!
Timecounter "TSC-low" frequency 1300025024 Hz quality 1000
Enter passphrase for da0p4: GEOM_MIRROR: Force device swap start due to timeout.
GEOM_ELI: Device da0p4.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_MIRROR: Device mirror/swap launched (1/2).
GEOM_ELI: Device da1p4.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da2p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da3p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
Trying to mount root from zfs:zroot/ROOT/default []...
GEOM_ELI: Device mirror/swap.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:     Crypto: hardware
ums0: <Winbond Electronics Corp Hermon USB hidmouse Device, class 0/0, rev 1.10/0.01, addr 3> on usbus0
ums0: 3 buttons and [Z] coordinates ID=0
pf: started
ix0: link state changed to UP