received kernel panic and coredump on newly built system (old hardware, ran FBSD 8-STABLE for years). vm_fault: fault on nofault entry, addr: fffffe001ff0a000 I was attempting to open a web page (in firefox) from a link in thunderbird. Firefox was already running at the time, with several windows on different virtual desktops (using mate) showing multiple web pages per window. There was also a linux VM running in virtualbox (with ~2G RAM assigned to it). The new packages have been recently installed using 'pkg' based on 11-RELEASE. I have seen what I believe to be crashes that are related to video before, when running 8-STABLE (with ports from over 2 years ago). This is one of the reasons I re-built the system from scratch on a new hard drive, entirely on ZFS. uname output: FreeBSD hack.SFT.local 11.0-RELEASE-p1 FreeBSD 11.0-RELEASE-p1 #0 r306420: Thu Sep 29 01:43:23 UTC 2016 root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 these are the loaded kernel modules: Id Refs Address Size Name 1 39 0xffffffff80200000 1fa7c38 kernel 2 1 0xffffffff821a9000 30aec0 zfs.ko 3 2 0xffffffff824b4000 adc0 opensolaris.ko 4 1 0xffffffff82621000 4cd3 cuse4bsd.ko 5 3 0xffffffff82626000 4feed vboxdrv.ko 6 2 0xffffffff82676000 2a1f vboxnetflt.ko 7 2 0xffffffff82679000 befc netgraph.ko 8 1 0xffffffff82685000 423b ng_ether.ko 9 1 0xffffffff8268a000 3f86 vboxnetadp.ko 10 2 0xffffffff8268e000 3de40 linux.ko 11 2 0xffffffff826cc000 7b08 linux_common.ko 12 1 0xffffffff826d4000 389f4 linux64.ko 13 1 0xffffffff8270d000 2c0a coretemp.ko 14 1 0xffffffff82710000 964559 nvidia.ko This is the dmesg output (in its entirety) related to the crash/dump. I included everything since it has some information in it that might be helpful. Copyright (c) 1992-2016 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.0-RELEASE-p1 #0 r306420: Thu Sep 29 01:43:23 UTC 2016 root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0) VT(vga): resolution 640x480 CPU: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (2400.13-MHz K8-class CPU) Origin="GenuineIntel" Id=0x6fb Family=0x6 Model=0xf Stepping=11 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0xe3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM> AMD Features=0x20100800<SYSCALL,NX,LM> AMD Features2=0x1<LAHF> VT-x: HLT,PAUSE TSC: P-state invariant, performance statistics real memory = 8589934592 (8192 MB) avail memory = 8254439424 (7872 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: <GBT GBTUACPI> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) random: unblocking device. ioapic0: Changing APIC ID to 2 ioapic0 <Version 2.0> irqs 0-23 on motherboard random: entropy device external interface kbd1 at kbdmux0 netmap: loaded module module_register_init: MOD_LOAD (vesa, 0xffffffff8101c950, 0) error 19 vtvga0: <VT VGA driver> on motherboard cryptosoft0: <software crypto> on motherboard acpi0: <GBT GBTUACPI> on motherboard acpi0: Power Button (fixed) cpu0: <ACPI CPU> on acpi0 cpu1: <ACPI CPU> on acpi0 cpu2: <ACPI CPU> on acpi0 cpu3: <ACPI CPU> on acpi0 attimer0: <AT timer> port 0x40-0x43 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 450 Event timer "HPET1" frequency 14318180 Hz quality 440 Event timer "HPET2" frequency 14318180 Hz quality 440 Event timer "HPET3" frequency 14318180 Hz quality 440 atrtc0: <AT realtime clock> port 0x70-0x73 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 acpi_button0: <Power Button> on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pcib1: <PCI-PCI bridge> irq 16 at device 6.0 on pci0 pci1: <PCI bus> on pcib1 vgapci0: <VGA-compatible display> port 0xa000-0xa07f mem 0xe6000000-0xe6ffffff,0xd0000000-0xdfffffff,0xe4000000-0xe5ffffff irq 16 at device 0.0 on pci1 vgapci0: Boot video device uhci0: <Intel 82801JI (ICH10) USB controller USB-D> port 0xe000-0xe01f irq 16 at device 26.0 on pci0 uhci0: LegSup = 0x2f00 usbus0 on uhci0 uhci1: <Intel 82801JI (ICH10) USB controller USB-E> port 0xe100-0xe11f irq 21 at device 26.1 on pci0 uhci1: LegSup = 0x2f00 usbus1 on uhci1 uhci2: <Intel 82801JI (ICH10) USB controller USB-F> port 0xe200-0xe21f irq 18 at device 26.2 on pci0 uhci2: LegSup = 0x2f00 usbus2 on uhci2 ehci0: <Intel 82801JI (ICH10) USB 2.0 controller USB-B> mem 0xeb305000-0xeb3053ff irq 18 at device 26.7 on pci0 usbus3: EHCI version 1.0 usbus3 on ehci0 hdac0: <Intel 82801JI HDA Controller> mem 0xeb300000-0xeb303fff irq 22 at device 27.0 on pci0 pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0 pcib2: [GIANT-LOCKED] pcib3: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0 pcib3: [GIANT-LOCKED] pci2: <ACPI PCI bus> on pcib3 atapci0: <JMicron JMB368 UDMA133 controller> port 0xb000-0xb007,0xb100-0xb103,0xb200-0xb207,0xb300-0xb303,0xb400-0xb40f irq 19 at device 0.0 on pci2 ata2: <ATA channel> at channel 0 on atapci0 pcib4: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0 pcib4: [GIANT-LOCKED] pci3: <ACPI PCI bus> on pcib4 re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xc000-0xc0ff mem 0xeb010000-0xeb010fff,0xeb000000-0xeb00ffff irq 16 at device 0.0 on pci3 re0: Using 1 MSI-X message re0: Chip rev. 0x3c000000 re0: MAC rev. 0x00400000 miibus0: <MII bus> on re0 rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus0 rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re0: Using defaults for TSO: 65518/35/2048 re0: Ethernet address: 00:1f:d0:2d:d5:ba re0: netmap queues/slots: TX 1/256, RX 1/256 pcib5: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0 pcib5: [GIANT-LOCKED] pci4: <ACPI PCI bus> on pcib5 re1: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xd000-0xd0ff mem 0xeb110000-0xeb110fff,0xeb100000-0xeb10ffff irq 17 at device 0.0 on pci4 re1: Using 1 MSI-X message re1: Chip rev. 0x3c000000 re1: MAC rev. 0x00400000 miibus1: <MII bus> on re1 rgephy1: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus1 rgephy1: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re1: Using defaults for TSO: 65518/35/2048 re1: Ethernet address: 00:1f:d0:2d:d5:d9 re1: netmap queues/slots: TX 1/256, RX 1/256 uhci3: <Intel 82801JI (ICH10) USB controller USB-A> port 0xe300-0xe31f irq 23 at device 29.0 on pci0 uhci3: LegSup = 0x2f00 usbus4 on uhci3 uhci4: <Intel 82801JI (ICH10) USB controller USB-B> port 0xe400-0xe41f irq 19 at device 29.1 on pci0 uhci4: LegSup = 0x2f00 usbus5 on uhci4 uhci5: <Intel 82801JI (ICH10) USB controller USB-C> port 0xe500-0xe51f irq 18 at device 29.2 on pci0 uhci5: LegSup = 0x2f00 usbus6 on uhci5 ehci1: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> mem 0xeb304000-0xeb3043ff irq 23 at device 29.7 on pci0 usbus7: EHCI version 1.0 usbus7 on ehci1 pcib6: <ACPI PCI-PCI bridge> at device 30.0 on pci0 pci5: <ACPI PCI bus> on pcib6 pci5: <serial bus, FireWire> at device 7.0 (no driver attached) isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 atapci1: <Intel ICH10 SATA300 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf000-0xf00f,0xf100-0xf10f at device 31.2 on pci0 ata0: <ATA channel> at channel 0 on atapci1 ata1: <ATA channel> at channel 1 on atapci1 atapci2: <Intel ICH10 SATA300 controller> port 0xe700-0xe707,0xe800-0xe803,0xe900-0xe907,0xea00-0xea03,0xeb00-0xeb0f,0xec00-0xec0f irq 19 at device 31.5 on pci0 ata3: <ATA channel> at channel 0 on atapci2 ata4: <ATA channel> at channel 1 on atapci2 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77b irq 7 drq 3 on acpi0 ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode ppc0: FIFO with 16/16/16 bytes threshold ppbus0: <Parallel port bus> on ppc0 lpt0: <Printer> on ppbus0 lpt0: Interrupt-driven port ppi0: <Parallel I/O> on ppbus0 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] psm0: <PS/2 Mouse> irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: model IntelliMouse, device ID 3 acpi_perf0: <ACPI CPU Frequency Control> on cpu0 est1: <Enhanced SpeedStep Frequency Control> on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 9250925060c0625 device_attach: est1 attach returned 6 est2: <Enhanced SpeedStep Frequency Control> on cpu2 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 9250925060c0625 device_attach: est2 attach returned 6 est3: <Enhanced SpeedStep Frequency Control> on cpu3 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 9250925060c0625 device_attach: est3 attach returned 6 usbus0: 12Mbps Full Speed USB v1.0 ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 1.000 msec nvme cam probe device init hdacc0: <Realtek ALC889A HDA CODEC> at cad 2 on hdac0 hdaa0: <Realtek ALC889A Audio Function Group> at nid 1 on hdacc0 pcm0: <Realtek ALC889A (Rear Analog 7.1/2.0)> at nid 20,22,21,23 and 24,26 on hdaa0 pcm1: <Realtek ALC889A (Front Analog)> at nid 27 and 25 on hdaa0 pcm2: <Realtek ALC889A (Rear Digital)> at nid 30 and 31 on hdaa0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 480Mbps High Speed USB v2.0 usbus4: 12Mbps Full Speed USB v1.0 ugen1.1: <Intel> at usbus1 uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1 ugen0.1: <Intel> at usbus0 uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 ugen2.1: <Intel> at usbus2 uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2 ugen3.1: <Intel> at usbus3 uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3 ugen4.1: <Intel> at usbus4 uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4 usbus5: 12Mbps Full Speed USB v1.0 usbus6: 12Mbps Full Speed USB v1.0 usbus7: 480Mbps High Speed USB v2.0 ugen5.1: <Intel> at usbus5 uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5 ugen6.1: <Intel> at usbus6 uhub6: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6 ugen7.1: <Intel> at usbus7 uhub7: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus7 uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered uhub6: 2 ports with 2 removable, self powered ada0 at ata0 bus 0 scbus1 target 0 lun 0 ada0: <WDC WD2003FYPS-27Y2B0 04.05G11> ATA8-ACS SATA 2.x device ada0: Serial Number WD-WCAVY7180271 ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes) ada0: 1907728MB (3907027055 512 byte sectors) ada1 at ata0 bus 0 scbus1 target 1 lun 0 ada1: <MAXTOR STM3500630AS 3.AAE> ATA-7 SATA 1.x device ada1: Serial Number 9QG7YY78 ada1: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) ada1: 476940MB (976773168 512 byte sectors) ada2 at ata3 bus 0 scbus3 target 0 lun 0 ada2: <Hitachi HDS721010CLA332 JP4OA39C> ATA8-ACS SATA 2.x device ada2: Serial Number JP2921HQ23DVBA ada2: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes) ada2: 953869MB (1953525168 512 byte sectors) cd0 at ata1 bus 0 scbus2 target 0 lun 0 SMP: AP CPU #3 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #1 Launched! cd0: <ATAPI DVD D DH16D2S EP53> Removable CD-ROM SCSI device cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed Timecounter "TSC-low" frequency 1200066277 Hz quality 1000 Trying to mount root from zfs:zroot/ROOT/default []... Root mount waiting for: usbus7 usbus3 Root mount waiting for: usbus7 usbus3 uhub7: 6 ports with 6 removable, self powered uhub3: 6 ports with 6 removable, self powered Cuse4BSD v0.1.36 @ /dev/cuse re0: link state changed to DOWN re0: link state changed to UP re1: link state changed to DOWN vboxdrv: fAsync=0 offMin=0x52b offMax=0x164a coretemp0: <CPU On-Die Thermal Sensors> on cpu0 coretemp1: <CPU On-Die Thermal Sensors> on cpu1 est1: <Enhanced SpeedStep Frequency Control> on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 9250925060c0625 device_attach: est1 attach returned 6 coretemp2: <CPU On-Die Thermal Sensors> on cpu2 est2: <Enhanced SpeedStep Frequency Control> on cpu2 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 9250925060c0625 device_attach: est2 attach returned 6 coretemp3: <CPU On-Die Thermal Sensors> on cpu3 est3: <Enhanced SpeedStep Frequency Control> on cpu3 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 9250925060c0625 device_attach: est3 attach returned 6 WARNING: /32-bit was not properly dismounted WARNING: /32-bit/tmp was not properly dismounted WARNING: /32-bit/var was not properly dismounted WARNING: /32-bit/usr was not properly dismounted WARNING: /32-bit/d-drive was not properly dismounted nvidia0: <GeForce 9600 GT> on vgapci0 vgapci0: child nvidia0 requested pci_enable_io vgapci0: child nvidia0 requested pci_enable_io WARNING: /old/var was not properly dismounted WARNING: /old/tmp was not properly dismounted WARNING: /old/usr was not properly dismounted WARNING: /old/e-drive was not properly dismounted vboxnet0: Ethernet address: 0a:00:27:00:00:00 vboxdrv: ffffffff83075020 VMMR0.r0 vboxdrv: ffffffff8317a020 VBoxDDR0.r0 pid 10579 (mate-system-monitor), uid 103: exited on signal 5 re0: promiscuous mode enabled re0: promiscuous mode disabled panic: vm_fault: fault on nofault entry, addr: fffffe001ff0a000 cpuid = 0 KDB: stack backtrace: #0 0xffffffff80b24077 at kdb_backtrace+0x67 #1 0xffffffff80ad93e2 at vpanic+0x182 #2 0xffffffff80ad9253 at panic+0x43 #3 0xffffffff80e12601 at vm_fault_hold+0x2721 #4 0xffffffff80e0fe98 at vm_fault+0x78 #5 0xffffffff80fa0e39 at trap_pfault+0xf9 #6 0xffffffff80fa04cc at trap+0x26c #7 0xffffffff80f84141 at calltrap+0x8 Uptime: 22h46m49s Dumping 3688 out of 8155 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% info file on crash dump: Dump header from device: /dev/ada0p2 Architecture: amd64 Architecture Version: 2 Dump Length: 3867783168 Blocksize: 512 Dumptime: Thu May 18 03:42:48 2017 Hostname: hack.SFT.local Magic: FreeBSD Kernel Dump Version String: FreeBSD 11.0-RELEASE-p1 #0 r306420: Thu Sep 29 01:43:23 UTC 2016 root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC Panic String: vm_fault: fault on nofault entry, addr: fffffe001ff0a000 Dump Parity: 3670130591 Bounds: 1 Dump Status: good VIDEO INFORMATION The nvidia kernel driver is x11/nvidia-driver-340 (necessary since I have an older graphics card). in Xorg.0.log (the one running now) the NVidia display is reported as follows: [ 505.906] (II) NVIDIA(0): NVIDIA GPU GeForce 9600 GT (G94) at PCI:1:0:0 (GPU-0) [ 505.906] (--) NVIDIA(0): Memory: 524288 kBytes [ 505.906] (--) NVIDIA(0): VideoBIOS: 62.94.0d.00.71 [ 505.906] (II) NVIDIA(0): Detected PCI Express Link width: 16X The video adapter is pretty old. It dates back from around 2008. The CPU is the same age. The motherboard is a year newer and the RAM was replaced a few years ago. RELATED OBSERVATIONS I have observed crashes in 8-STABLE before, usually associated with doing things like switching out of full screen video mode. However the crashes back then were somewhat random. Another possibly related problem is that swap space would occasionally be allocated (and not free'd up), with no indication as to which application had done it. Usually it would be about 5% of the available (8G) swap space. In 8-STABLE, when this happend, the system could go weeks without crashing. However, I don't recall whether swap space was never allocated when the system crashed. In short this problem could have been present for years. It could also be hardware related. It could have something to do with a kernel module that incorrectly allocates pages (or corrupts the page table), sometimes resulting in swap space being allocated for no reason. I have observed the swap space phenomenon ONCE with this new 11-RELEASE kernel. I plan on updating kernel/world with the latest 11-STABLE once I have finished putting everything back where it was (more or less) under 8. If it happens again I will annotate it in this bug report. (I wasn't able to get any really useful information out of kgdb; however, I will keep the core dump file handy in case you need it)
It appears that one other factor may be involved. As I recall, the problem is NOT showing up when the icecast server isn't running, while being fed an audio stream by 'ices'. Often I was leaving this running even when I wasn't listening to the stream on a streaming radio that's on my LAN. However, lately, I've been shutting off the audio feed by killing 'ices' (icecast remains running). The installed packages are: icecast2-2.4.3,1 ices-2.0.2_3,1 In the past I have seen some odd performance-related problems with transferring large amounts of data over 'localhost', with applications like iperf (stalls, lost packets and 'out of buffer space' and similar problems, basically). I have not attempted to reproduce this in FBSD-11 however, but if the stack hasn't changed much this COULD be related. When ices feeds icecast with content, I'm pretty sure it uses localhost to do it. If not, it's probably using a socket or a pipe. Whichever case this is, it's running for DAYS or even WEEKS sometimes, possibly with leakage, or internal corruption, or something else. A typical trigger for the reported problem is a video-related event, such as switching from full screen video back to whatever video mode the GUI is using. In a few cases it has been caused by attempting to open up a new firefox window or a new tab within firefox (i.e. following a link from something else, such as an e-mail). It's possible that it's "the trigger" PLUS the socket/pipe/whatever that ices+icecast is using. It's also possible that it's just creating a rare condition within memory or the kernel configuration that surfaces a bug. But if it _IS_ network stack related, particularly with the 'localhost' adaptor, then this information might be helpful in isolating a potentially big problem.