I have seen this same trap twice over the last few days using the 10.2-RELEASE GENERIC kernel, though I cannot reproduce it at will. On both occasions, the trap occurred while running an 'iocage -f destroy' on a jail – iocage is a jail management utility (see https://github.com/iocage/iocage) implemented using /bin/sh, and I don’t believe is doing anything too esoteric. First time: Sep 12 09:09:57 bhs-0 kernel: Fatal trap 12: page fault while in kernel mode Sep 12 09:09:57 bhs-0 kernel: cpuid = 5; apic id = 05 Sep 12 09:09:57 bhs-0 kernel: fault virtual address = 0x38 Sep 12 09:09:58 bhs-0 kernel: fault code = supervisor read data, page not present Sep 12 09:09:58 bhs-0 kernel: instruction pointer = 0x20:0xffffffff809e58fd Sep 12 09:09:58 bhs-0 kernel: stack pointer = 0x28:0xfffffe2021dc64c0 Sep 12 09:09:58 bhs-0 kernel: frame pointer = 0x28:0xfffffe2021dc64f0 Sep 12 09:09:58 bhs-0 kernel: code segment = base rx0, limit 0xfffff, type 0x1b Sep 12 09:09:58 bhs-0 kernel: = DPL 0, pres 1, long 1, def32 0, gran 1 Sep 12 09:09:58 bhs-0 kernel: processor eflags = Second time: Sep 14 12:57:19 bhs-0 kernel: Fatal trap 12: page fault while in kernel mode Sep 14 12:57:19 bhs-0 kernel: cpuid = 12; apic id = 0c Sep 14 12:57:19 bhs-0 kernel: fault virtual address = 0x38 Sep 14 12:57:20 bhs-0 kernel: fault code = supervisor read data, page not present Sep 14 12:57:20 bhs-0 kernel: instruction pointer = 0x20:0xffffffff809e58fd Sep 14 12:57:20 bhs-0 kernel: stack pointer = 0x28:0xfffffe2021db24c0 Sep 14 12:57:20 bhs-0 kernel: frame pointer = 0x28:0xfffffe2021db24f0 Sep 14 12:57:20 bhs-0 kernel: code segment = base rx0, limit 0xfffff, type 0x1b Sep 14 12:57:20 bhs-0 kernel: = DPL 0, pres 1, long 1, def32 0, gran 1 nm suggests the problem may be in __vfs_statfs: # nm -n /boot/kernel/kernel.symbols |grep ffffffff809e58f ffffffff809e58f0 T __vfs_statfs I did not have a dumpdev configured, so unfortunately I do not have a crash dump. I have now configured dumpdev, and will endeavour to capture a crash dump if this problem happens again (note, however, that I only have 4GB of swap space and 128 GB of RAM, so I’m not sure whether a crash dump will be successful). Here’s the relevant chunk of dmesg output describing my hardware: Copyright (c) 1992-2015 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 12 15:26:37 UTC 2015 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512 CPU: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz (2600.05-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306e4 Family=0x6 Model=0x3e Stepping=4 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x7fbee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> AMD Features2=0x1<LAHF> Structured Extended Features=0x281<FSGSBASE,SMEP,ERMS> XSAVE Features=0x1<XSAVEOPT> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics real memory = 137447342080 (131080 MB) avail memory = 133482815488 (127299 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: < > FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs FreeBSD/SMP: 1 package(s) x 8 core(s) x 2 SMT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 cpu8 (AP): APIC ID: 8 cpu9 (AP): APIC ID: 9 cpu10 (AP): APIC ID: 10 cpu11 (AP): APIC ID: 11 cpu12 (AP): APIC ID: 12 cpu13 (AP): APIC ID: 13 cpu14 (AP): APIC ID: 14 cpu15 (AP): APIC ID: 15 ioapic0 <Version 2.0> irqs 0-23 on motherboard ioapic1 <Version 2.0> irqs 24-47 on motherboard random: <Software, Yarrow> initialized kbd1 at kbdmux0 cryptosoft0: <software crypto> on motherboard aesni0: <AES-CBC,AES-XTS> on motherboard acpi0: <SUPERM SMCI--MB> on motherboard acpi0: Power Button (fixed) cpu0: <ACPI CPU> on acpi0 cpu1: <ACPI CPU> on acpi0 cpu2: <ACPI CPU> on acpi0 cpu3: <ACPI CPU> on acpi0 cpu4: <ACPI CPU> on acpi0 cpu5: <ACPI CPU> on acpi0 cpu6: <ACPI CPU> on acpi0 cpu7: <ACPI CPU> on acpi0 cpu8: <ACPI CPU> on acpi0 cpu9: <ACPI CPU> on acpi0 cpu10: <ACPI CPU> on acpi0 cpu11: <ACPI CPU> on acpi0 cpu12: <ACPI CPU> on acpi0 cpu13: <ACPI CPU> on acpi0 cpu14: <ACPI CPU> on acpi0 cpu15: <ACPI CPU> on acpi0 attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 340 Event timer "HPET2" frequency 14318180 Hz quality 340 Event timer "HPET3" frequency 14318180 Hz quality 340 Event timer "HPET4" frequency 14318180 Hz quality 340 Event timer "HPET5" frequency 14318180 Hz quality 340 Event timer "HPET6" frequency 14318180 Hz quality 340 Event timer "HPET7" frequency 14318180 Hz quality 340 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pcib1: <ACPI PCI-PCI bridge> irq 27 at device 1.0 on pci0 pci1: <ACPI PCI bus> on pcib1 ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.8.3> port 0xe020-0xe03f mem 0xfa200000-0xfa3fffff,0xfa404000-0xfa407fff irq 16 at device 0.0 on pci1 ix0: Using MSIX interrupts with 9 vectors ix0: Ethernet address: 0c:c4:7a:1d:a1:5a ix0: PCI Express Bus: Speed 5.0GT/s Width x8 ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.8.3> port 0xe000-0xe01f mem 0xfa000000-0xfa1fffff,0xfa400000-0xfa403fff irq 17 at device 0.1 on pci1 ix1: Using MSIX interrupts with 9 vectors ix1: Ethernet address: 0c:c4:7a:1d:a1:5b ix1: PCI Express Bus: Speed 5.0GT/s Width x8 pcib2: <ACPI PCI-PCI bridge> irq 33 at device 2.0 on pci0 pci3: <ACPI PCI bus> on pcib2 pcib3: <ACPI PCI-PCI bridge> irq 41 at device 3.0 on pci0 pci4: <ACPI PCI bus> on pcib3 mps0: <Avago Technologies (LSI) SAS2308> port 0xd000-0xd0ff mem 0xfbe40000-0xfbe4ffff,0xfbe00000-0xfbe3ffff irq 40 at device 0.0 on pci4 mps0: Firmware: 19.00.00.00, Driver: 20.00.00.00-fbsd mps0: IOCCapabilities: 4185c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR> pcib4: <ACPI PCI-PCI bridge> irq 41 at device 3.2 on pci0 pci5: <ACPI PCI bus> on pcib4 pcib5: <ACPI PCI-PCI bridge> irq 16 at device 17.0 on pci0 pci6: <ACPI PCI bus> on pcib5 pci0: <simple comms> at device 22.0 (no driver attached) pci0: <simple comms> at device 22.1 (no driver attached) ehci0: <Intel Patsburg USB 2.0 controller> mem 0xfbf23000-0xfbf233ff irq 16 at device 26.0 on pci0 usbus0: EHCI version 1.0 usbus0 on ehci0 ehci1: <Intel Patsburg USB 2.0 controller> mem 0xfbf22000-0xfbf223ff irq 23 at device 29.0 on pci0 usbus1: EHCI version 1.0 usbus1 on ehci1 pcib6: <ACPI PCI-PCI bridge> at device 30.0 on pci0 pci7: <ACPI PCI bus> on pcib6 vgapci0: <VGA-compatible display> mem 0xf9000000-0xf9ffffff,0xfb800000-0xfb803fff,0xfb000000-0xfb7fffff irq 16 at device 4.0 on pci7 vgapci0: Boot video device isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 ahci0: <Intel Patsburg AHCI SATA controller> port 0xf050-0xf057,0xf040-0xf043,0xf030-0xf037,0xf020-0xf023,0xf000-0xf01f mem 0xfbf21000-0xfbf217ff irq 18 at device 31.2 on pci0 ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported ahcich0: <AHCI channel> at channel 0 on ahci0 ahcich1: <AHCI channel> at channel 1 on ahci0 ahcich2: <AHCI channel> at channel 2 on ahci0 ahcich3: <AHCI channel> at channel 3 on ahci0 ahcich4: <AHCI channel> at channel 4 on ahci0 ahcich5: <AHCI channel> at channel 5 on ahci0 ahciem0: <AHCI enclosure management bridge> on ahci0 pcib7: <ACPI Host-PCI bridge> on acpi0 pci255: <ACPI PCI bus> on pcib7 acpi_button0: <Power Button> on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart2: <16550 or compatible> port 0x3e8-0x3ef irq 10 on acpi0 uart2: console (115200,n,8,1) orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xce000-0xcefff on isa0 sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ppc0: cannot reserve I/O port range est0: <Enhanced SpeedStep Frequency Control> on cpu0 est1: <Enhanced SpeedStep Frequency Control> on cpu1 est2: <Enhanced SpeedStep Frequency Control> on cpu2 est3: <Enhanced SpeedStep Frequency Control> on cpu3 est4: <Enhanced SpeedStep Frequency Control> on cpu4 est5: <Enhanced SpeedStep Frequency Control> on cpu5 est6: <Enhanced SpeedStep Frequency Control> on cpu6 est7: <Enhanced SpeedStep Frequency Control> on cpu7 est8: <Enhanced SpeedStep Frequency Control> on cpu8 est9: <Enhanced SpeedStep Frequency Control> on cpu9 est10: <Enhanced SpeedStep Frequency Control> on cpu10 est11: <Enhanced SpeedStep Frequency Control> on cpu11 est12: <Enhanced SpeedStep Frequency Control> on cpu12 est13: <Enhanced SpeedStep Frequency Control> on cpu13 est14: <Enhanced SpeedStep Frequency Control> on cpu14 est15: <Enhanced SpeedStep Frequency Control> on cpu15 random: unblocking device. usbus0: 480Mbps High Speed USB v2.0 ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 1.000 msec usbus1: 480Mbps High Speed USB v2.0 ugen0.1: <Intel> at usbus0 uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0 ugen1.1: <Intel> at usbus1 uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1 uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered ugen0.2: <vendor 0x8087> at usbus0 uhub2: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0 ugen1.2: <vendor 0x8087> at usbus1 uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus1 mps0: SAS Address for SATA device = db235358b6fcdd88 mps0: SAS Address from SATA device = db235358b6fcdd88 mps0: SAS Address for SATA device = db235358b6fdb684 mps0: SAS Address from SATA device = db235358b6fdb684 mps0: SAS Address for SATA device = db235358b6fdb572 mps0: SAS Address from SATA device = db235358b6fdb572 mps0: SAS Address for SATA device = db235358b6fdd687 mps0: SAS Address from SATA device = db235358b6fdd687 uhub2: 6 ports with 6 removable, self powered uhub3: 8 ports with 8 removable, self powered ugen0.3: <Winbond Electronics Corp> at usbus0 ukbd0: <Winbond Electronics Corp Hermon USB hidmouse Device, class 0/0, rev 1.10/0.01, addr 3> on usbus0 kbd2 at ukbd0 ugen1.3: <Kingston> at usbus1 umass0: <Kingston DataTraveler 3.0, class 0/0, rev 2.10/1.00, addr 3> on usbus1 umass0: SCSI over Bulk-Only; quirks = 0x8100 umass0:8:0:-1: Attached to scbus8 ses0 at ahciem0 bus 0 scbus7 target 0 lun 0 ses0: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device ses0: SEMB SES Device da0 at mps0 bus 0 scbus0 target 2 lun 0 da1 at mps0 bus 0 scbus0 target 3 lun 0 da2 at mps0 bus 0 scbus0 target 4 lun 0 da0: <ATA INTEL SSDSC2BB80 0380> Fixed Direct Access SPC-4 SCSI device da0: Serial Number BTWH507100ZH800CGN da0: 600.000MB/s transfers da0: Command Queueing enabled da1: <ATA INTEL SSDSC2BB80 0380> Fixed Direct Access SPC-4 SCSI device da1: Serial Number BTWH5071013D800CGN da1: 600.000MB/s transfers da1: Command Queueing enabled da1: 763097MB (1562824368 512 byte sectors: 255H 63S/T 97281C) da2: <ATA INTEL SSDSC2BB80 0380> Fixed Direct Access SPC-4 SCSI device da0: 763097MB (1562824368 512 byte sectors: 255H 63S/T 97281C) da3 at mps0 bus 0 scbus0 target 5 lun 0 da3: <ATA INTEL SSDSC2BB80 0380> Fixed Direct Access SPC-4 SCSI device da3: Serial Number BTWH507101SG800CGN da3: 600.000MB/s transfers da3: Command Queueing enabled da3: 763097MB (1562824368 512 byte sectors: 255H 63S/T 97281C) da2: Serial Number BTWH50710122800CGN da2: 600.000MB/s transfers da2: Command Queueing enabled da2: 763097MB (1562824368 512 byte sectors: 255H 63S/T 97281C) da4 at umass-sim0 bus 0 scbus8 target 0 lun 0 da4: <Kingston DataTraveler 3.0 PMAP> Removable Direct Access SPC-4 SCSI device da4: Serial Number 50E549C692DBEF51594F1BC1 da4: 40.000MB/s transfers da4: 15004MB (30728832 512 byte sectors: 255H 63S/T 1912C) da4: quirks=0x2<NO_6_BYTE> SMP: AP CPU #1 Launched! SMP: AP CPU #9 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #15 Launched! SMP: AP CPU #13 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #10 Launched! SMP: AP CPU #12 Launched! SMP: AP CPU #8 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #11 Launched! SMP: AP CPU #14 Launched! SMP: AP CPU #4 Launched! Timecounter "TSC-low" frequency 1300025024 Hz quality 1000 Enter passphrase for da0p4: GEOM_MIRROR: Force device swap start due to timeout. GEOM_ELI: Device da0p4.eli created. GEOM_ELI: Encryption: AES-XTS 256 GEOM_ELI: Crypto: hardware GEOM_MIRROR: Device mirror/swap launched (1/2). GEOM_ELI: Device da1p4.eli created. GEOM_ELI: Encryption: AES-XTS 256 GEOM_ELI: Crypto: hardware GEOM_ELI: Device da2p1.eli created. GEOM_ELI: Encryption: AES-XTS 256 GEOM_ELI: Crypto: hardware GEOM_ELI: Device da3p1.eli created. GEOM_ELI: Encryption: AES-XTS 256 GEOM_ELI: Crypto: hardware Trying to mount root from zfs:zroot/ROOT/default []... GEOM_ELI: Device mirror/swap.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: hardware ums0: <Winbond Electronics Corp Hermon USB hidmouse Device, class 0/0, rev 1.10/0.01, addr 3> on usbus0 ums0: 3 buttons and [Z] coordinates ID=0 pf: started ix0: link state changed to UP