Running the latest 8-STABLE. I've re-cvs'd and rebuilt the kernel & world a few times the last 2 days and the problem won't go away. I recompiled with INVARIANTS, WITNESS, KDB, etc to get a backtrace and any other information that might be required to resolve this problem. Screenshots of the panic and backtrace: http://bayimg.com/oAMIiAaCP http://bayimg.com/oAmIJAAcP http://bayimg.com/oaMIKaacp dmesg output: Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.1-PRERELEASE #3: Tue May 25 14:32:02 NZST 2010 martinm@kinetic.pulse.local:/usr/obj/usr/src/sys/DIAMOND amd64 WARNING: WITNESS option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU E5320 @ 1.86GHz (1866.43-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6f8 Family = 6 Model = f Stepping = 8 Features=0xfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS> Features2=0x2201<SSE3,SSSE3,CX16> AMD Features=0x20100800<SYSCALL,NX,LM> AMD Features2=0x1<LAHF> TSC: P-state invariant real memory = 1073741824 (1024 MB) avail memory = 1016025088 (968 MB) ACPI APIC Table: <PTLTD APIC > MADT: Forcing active-low polarity and level trigger for SCI ioapic0 <Version 1.1> irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: <PTLTD RSDT> on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 cpu0: <ACPI CPU> on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0 pci1: <ACPI PCI bus> on pcib1 isab0: <PCI-ISA bridge> at device 7.0 on pci0 isa0: <ISA bus> on isab0 atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1050-0x105f at device 7.1 on pci0 ata0: <ATA channel 0> on atapci0 ata0: [ITHREAD] ata1: <ATA channel 1> on atapci0 ata1: [ITHREAD] pci0: <bridge> at device 7.3 (no driver attached) vgapci0: <VGA-compatible display> port 0x1060-0x106f mem 0xf8000000-0xfbffffff,0xf4000000-0xf47fffff at device 15.0 on pci0 mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x1080-0x10ff mem 0xf4810000-0xf4810fff irq 17 at device 16.0 on pci0 mpt0: [ITHREAD] mpt0: MPI Version=1.2.0.0 em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.1> port 0x1400-0x143f mem 0xf4820000-0xf483ffff,0xf4800000-0xf480ffff irq 18 at device 17.0 on pci0 em0: Memory Access and/or Bus Master bits were not set! em0: [FILTER] em0: Ethernet address: 00:50:56:a5:20:03 acpi_acad0: <AC Adapter> on acpi0 atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: <PS/2 Mouse> irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model IntelliMouse, device ID 3 ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppc0: [ITHREAD] ppbus0: <Parallel port bus> on ppc0 plip0: <PLIP network interface> on ppbus0 plip0: [ITHREAD] lpt0: <Printer> on ppbus0 lpt0: [ITHREAD] lpt0: Interrupt-driven port ppi0: <Parallel I/O> on ppbus0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: [FILTER] fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xca000-0xcafff,0xdc000-0xdffff,0xe0000-0xe3fff on isa0 sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 acpi_throttle0: <ACPI CPU Throttling> on cpu0 ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. ZFS filesystem version 3 ZFS storage pool version 14 Timecounter "TSC" frequency 1866434311 Hz quality 800 Timecounters tick every 10.000 msec acd0: CDROM <VMware Virtual IDE CDROM Drive/00000001> at ata0-master UDMA33 da0 at mpt0 bus 0 scbus0 target 0 lun 0 da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da0: Command Queueing enabled da0: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C) da1 at mpt0 bus 0 scbus0 target 1 lun 0 da1: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device da1: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da1: Command Queueing enabled da1: 143360MB (293601280 512 byte sectors: 255H 63S/T 18275C) WARNING: WITNESS option enabled, expect reduced performance. Trying to mount root from ufs:/dev/da0s1a WARNING: / was not properly dismounted VMware memory control driver initialized Fatal double fault rip = 0xffffffff8037ff6b rsp = 0xffffff808d05ef40 rbp = 0xffffff808d05f070 cpuid = 0; apic id = 00 panic: double fault cpuid = 0 KDB: enter: panic Uptime: 1m39s Cannot dump. Device not defined or unavailable. Automatic reboot in 15 seconds - press a key on the console to abort Rebooting... Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.1-PRERELEASE #3: Tue May 25 14:32:02 NZST 2010 martinm@kinetic.pulse.local:/usr/obj/usr/src/sys/DIAMOND amd64 WARNING: WITNESS option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU E5320 @ 1.86GHz (1866.46-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6f8 Family = 6 Model = f Stepping = 8 Features=0xfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS> Features2=0x2201<SSE3,SSSE3,CX16> AMD Features=0x20100800<SYSCALL,NX,LM> AMD Features2=0x1<LAHF> TSC: P-state invariant real memory = 1073741824 (1024 MB) avail memory = 1016025088 (968 MB) ACPI APIC Table: <PTLTD APIC > MADT: Forcing active-low polarity and level trigger for SCI ioapic0 <Version 1.1> irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: <PTLTD RSDT> on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 cpu0: <ACPI CPU> on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0 pci1: <ACPI PCI bus> on pcib1 isab0: <PCI-ISA bridge> at device 7.0 on pci0 isa0: <ISA bus> on isab0 atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1050-0x105f at device 7.1 on pci0 ata0: <ATA channel 0> on atapci0 ata0: [ITHREAD] ata1: <ATA channel 1> on atapci0 ata1: [ITHREAD] pci0: <bridge> at device 7.3 (no driver attached) vgapci0: <VGA-compatible display> port 0x1060-0x106f mem 0xf8000000-0xfbffffff,0xf4000000-0xf47fffff at device 15.0 on pci0 mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x1080-0x10ff mem 0xf4810000-0xf4810fff irq 17 at device 16.0 on pci0 mpt0: [ITHREAD] mpt0: MPI Version=1.2.0.0 em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.1> port 0x1400-0x143f mem 0xf4820000-0xf483ffff,0xf4800000-0xf480ffff irq 18 at device 17.0 on pci0 em0: Memory Access and/or Bus Master bits were not set! em0: [FILTER] em0: Ethernet address: 00:50:56:a5:20:03 acpi_acad0: <AC Adapter> on acpi0 atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: <PS/2 Mouse> irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model IntelliMouse, device ID 3 ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppc0: [ITHREAD] ppbus0: <Parallel port bus> on ppc0 plip0: <PLIP network interface> on ppbus0 plip0: [ITHREAD] lpt0: <Printer> on ppbus0 lpt0: [ITHREAD] lpt0: Interrupt-driven port ppi0: <Parallel I/O> on ppbus0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: [FILTER] fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xca000-0xcafff,0xdc000-0xdffff,0xe0000-0xe3fff on isa0 sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 acpi_throttle0: <ACPI CPU Throttling> on cpu0 ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. ZFS filesystem version 3 ZFS storage pool version 14 Timecounter "TSC" frequency 1866457425 Hz quality 800 Timecounters tick every 10.000 msec acd0: CDROM <VMware Virtual IDE CDROM Drive/00000001> at ata0-master UDMA33 da0 at mpt0 bus 0 scbus0 target 0 lun 0 da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da0: Command Queueing enabled da0: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C) da1 at mpt0 bus 0 scbus0 target 1 lun 0 da1: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device da1: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da1: Command Queueing enabled da1: 143360MB (293601280 512 byte sectors: 255H 63S/T 18275C) WARNING: WITNESS option enabled, expect reduced performance. Trying to mount root from ufs:/dev/da0s1a WARNING: / was not properly dismounted /: mount pending error: blocks 444 files 29 VMware memory control driver initialized Fatal double fault rip = 0xffffffff8037ff6b rsp = 0xffffff808d05ef40 rbp = 0xffffff808d05f070 cpuid = 0; apic id = 00 panic: double fault cpuid = 0 KDB: enter: panic Uptime: 1m39s Cannot dump. Device not defined or unavailable. Automatic reboot in 15 seconds - press a key on the console to abort --> Press a key on the console to reboot, --> or switch off the system now. Rebooting... Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.1-PRERELEASE #3: Tue May 25 14:32:02 NZST 2010 martinm@kinetic.pulse.local:/usr/obj/usr/src/sys/DIAMOND amd64 WARNING: WITNESS option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU E5320 @ 1.86GHz (1866.44-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6f8 Family = 6 Model = f Stepping = 8 Features=0xfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS> Features2=0x2201<SSE3,SSSE3,CX16> AMD Features=0x20100800<SYSCALL,NX,LM> AMD Features2=0x1<LAHF> TSC: P-state invariant real memory = 1073741824 (1024 MB) avail memory = 1016025088 (968 MB) ACPI APIC Table: <PTLTD APIC > MADT: Forcing active-low polarity and level trigger for SCI ioapic0 <Version 1.1> irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: <PTLTD RSDT> on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 cpu0: <ACPI CPU> on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0 pci1: <ACPI PCI bus> on pcib1 isab0: <PCI-ISA bridge> at device 7.0 on pci0 isa0: <ISA bus> on isab0 atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1050-0x105f at device 7.1 on pci0 ata0: <ATA channel 0> on atapci0 ata0: [ITHREAD] ata1: <ATA channel 1> on atapci0 ata1: [ITHREAD] pci0: <bridge> at device 7.3 (no driver attached) vgapci0: <VGA-compatible display> port 0x1060-0x106f mem 0xf8000000-0xfbffffff,0xf4000000-0xf47fffff at device 15.0 on pci0 mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x1080-0x10ff mem 0xf4810000-0xf4810fff irq 17 at device 16.0 on pci0 mpt0: [ITHREAD] mpt0: MPI Version=1.2.0.0 em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.1> port 0x1400-0x143f mem 0xf4820000-0xf483ffff,0xf4800000-0xf480ffff irq 18 at device 17.0 on pci0 em0: Memory Access and/or Bus Master bits were not set! em0: [FILTER] em0: Ethernet address: 00:50:56:a5:20:03 acpi_acad0: <AC Adapter> on acpi0 atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: <PS/2 Mouse> irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model IntelliMouse, device ID 3 ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppc0: [ITHREAD] ppbus0: <Parallel port bus> on ppc0 plip0: <PLIP network interface> on ppbus0 plip0: [ITHREAD] lpt0: <Printer> on ppbus0 lpt0: [ITHREAD] lpt0: Interrupt-driven port ppi0: <Parallel I/O> on ppbus0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: [FILTER] fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xca000-0xcafff,0xdc000-0xdffff,0xe0000-0xe3fff on isa0 sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 acpi_throttle0: <ACPI CPU Throttling> on cpu0 ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. ZFS filesystem version 3 ZFS storage pool version 14 Timecounter "TSC" frequency 1866436348 Hz quality 800 Timecounters tick every 10.000 msec acd0: CDROM <VMware Virtual IDE CDROM Drive/00000001> at ata0-master UDMA33 da0 at mpt0 bus 0 scbus0 target 0 lun 0 da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da0: Command Queueing enabled da0: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C) da1 at mpt0 bus 0 scbus0 target 1 lun 0 da1: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device da1: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da1: Command Queueing enabled da1: 143360MB (293601280 512 byte sectors: 255H 63S/T 18275C) WARNING: WITNESS option enabled, expect reduced performance. Trying to mount root from ufs:/dev/da0s1a WARNING: / was not properly dismounted /: mount pending error: blocks 388 files 20 VMware memory control driver initialized Fatal double fault rip = 0xffffffff8037ff6b rsp = 0xffffff808d05ef40 rbp = 0xffffff808d05f070 cpuid = 0; apic id = 00 panic: double fault cpuid = 0 KDB: enter: panic Uptime: 1m39s Cannot dump. Device not defined or unavailable. Automatic reboot in 15 seconds - press a key on the console to abort --> Press a key on the console to reboot, --> or switch off the system now. Rebooting... Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.1-PRERELEASE #3: Tue May 25 14:32:02 NZST 2010 martinm@kinetic.pulse.local:/usr/obj/usr/src/sys/DIAMOND amd64 WARNING: WITNESS option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU E5320 @ 1.86GHz (1866.46-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6f8 Family = 6 Model = f Stepping = 8 Features=0xfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS> Features2=0x2201<SSE3,SSSE3,CX16> AMD Features=0x20100800<SYSCALL,NX,LM> AMD Features2=0x1<LAHF> TSC: P-state invariant real memory = 1073741824 (1024 MB) avail memory = 1016025088 (968 MB) ACPI APIC Table: <PTLTD APIC > MADT: Forcing active-low polarity and level trigger for SCI ioapic0 <Version 1.1> irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: <PTLTD RSDT> on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 cpu0: <ACPI CPU> on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0 pci1: <ACPI PCI bus> on pcib1 isab0: <PCI-ISA bridge> at device 7.0 on pci0 isa0: <ISA bus> on isab0 atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1050-0x105f at device 7.1 on pci0 ata0: <ATA channel 0> on atapci0 ata0: [ITHREAD] ata1: <ATA channel 1> on atapci0 ata1: [ITHREAD] pci0: <bridge> at device 7.3 (no driver attached) vgapci0: <VGA-compatible display> port 0x1060-0x106f mem 0xf8000000-0xfbffffff,0xf4000000-0xf47fffff at device 15.0 on pci0 mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x1080-0x10ff mem 0xf4810000-0xf4810fff irq 17 at device 16.0 on pci0 mpt0: [ITHREAD] mpt0: MPI Version=1.2.0.0 em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.1> port 0x1400-0x143f mem 0xf4820000-0xf483ffff,0xf4800000-0xf480ffff irq 18 at device 17.0 on pci0 em0: Memory Access and/or Bus Master bits were not set! em0: [FILTER] em0: Ethernet address: 00:50:56:a5:20:03 acpi_acad0: <AC Adapter> on acpi0 atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: <PS/2 Mouse> irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model IntelliMouse, device ID 3 ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppc0: [ITHREAD] ppbus0: <Parallel port bus> on ppc0 plip0: <PLIP network interface> on ppbus0 plip0: [ITHREAD] lpt0: <Printer> on ppbus0 lpt0: [ITHREAD] lpt0: Interrupt-driven port ppi0: <Parallel I/O> on ppbus0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: [FILTER] fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xca000-0xcafff,0xdc000-0xdffff,0xe0000-0xe3fff on isa0 sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 acpi_throttle0: <ACPI CPU Throttling> on cpu0 ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. ZFS filesystem version 3 ZFS storage pool version 14 Timecounter "TSC" frequency 1866456340 Hz quality 800 Timecounters tick every 10.000 msec acd0: CDROM <VMware Virtual IDE CDROM Drive/00000001> at ata0-master UDMA33 da0 at mpt0 bus 0 scbus0 target 0 lun 0 da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da0: Command Queueing enabled da0: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C) da1 at mpt0 bus 0 scbus0 target 1 lun 0 da1: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device da1: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da1: Command Queueing enabled da1: 143360MB (293601280 512 byte sectors: 255H 63S/T 18275C) WARNING: WITNESS option enabled, expect reduced performance. Trying to mount root from ufs:/dev/da0s1a WARNING: / was not properly dismounted /: mount pending error: blocks 396 files 20 KINETIC kernel config: # # GENERIC -- Generic kernel configuration file for FreeBSD/amd64 # # For more information on this file, please read the config(5) manual page, # and/or the handbook section on Kernel Configuration Files: # # http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html # # The handbook is also available locally in /usr/share/doc/handbook # if you've installed the doc distribution, otherwise always see the # FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the # latest information. # # An exhaustive list of options and more detailed explanations of the # device lines is also present in the ../../conf/NOTES and NOTES files. # If you are in doubt as to the purpose or necessity of a line, check first # in NOTES. # # $FreeBSD: src/sys/amd64/conf/GENERIC,v 1.531.2.13 2010/05/02 06:24:17 imp Exp $ cpu HAMMER ident KINETIC # To statically compile in device wiring instead of /boot/device.hints #hints "GENERIC.hints" # Default places to look for devices. # Use the following to compile in values accessible to the kernel # through getenv() (or kenv(1) in userland). The format of the file # is 'variable=value', see kenv(1) # # env "GENERIC.env" #makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols options SCHED_ULE # ULE scheduler options PREEMPTION # Enable kernel thread preemption options INET # InterNETworking #options INET6 # IPv6 communications protocols options SCTP # Stream Control Transmission Protocol options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_ACL # Support for access control lists options UFS_DIRHASH # Improve performance on big directories options UFS_GJOURNAL # Enable gjournal-based UFS journaling options MD_ROOT # MD is a potential root device options NFSCLIENT # Network Filesystem Client options NFSSERVER # Network Filesystem Server options NFSLOCKD # Network Lock Manager options NFS_ROOT # NFS usable as /, requires NFSCLIENT options MSDOSFS # MSDOS Filesystem options CD9660 # ISO 9660 Filesystem options PROCFS # Process filesystem (requires PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options GEOM_PART_GPT # GUID Partition Tables. options GEOM_LABEL # Provides labelization options COMPAT_43TTY # BSD 4.3 TTY compat (sgtty) options COMPAT_FREEBSD32 # Compatible with i386 binaries options COMPAT_FREEBSD4 # Compatible with FreeBSD4 options COMPAT_FREEBSD5 # Compatible with FreeBSD5 options COMPAT_FREEBSD6 # Compatible with FreeBSD6 options COMPAT_FREEBSD7 # Compatible with FreeBSD7 options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI options KTRACE # ktrace(1) support options STACK # stack(9) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options P1003_1B_SEMAPHORES # POSIX-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options PRINTF_BUFR_SIZE=128 # Prevent printf output being interspersed. options KBD_INSTALL_CDEV # install a CDEV entry in /dev options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) options AUDIT # Security event auditing options MAC # TrustedBSD MAC Framework options FLOWTABLE # per-cpu routing cache #options KDTRACE_FRAME # Ensure frames are compiled in #options KDTRACE_HOOKS # Kernel DTrace hooks options INCLUDE_CONFIG_FILE # Include this file in kernel # Make an SMP-capable kernel by default options SMP # Symmetric MultiProcessor Kernel # CPU frequency control device cpufreq # Bus support. device acpi device pci # Floppy drives device fdc # ATA and ATAPI devices device ata device atadisk # ATA disk drives device ataraid # ATA RAID drives device atapicd # ATAPI CDROM drives device atapifd # ATAPI floppy drives device atapist # ATAPI tape drives options ATA_STATIC_ID # Static device numbering # SCSI Controllers #device ahc # AHA2940 and onboard AIC7xxx devices options AHC_REG_PRETTY_PRINT # Print register bitfields in debug # output. Adds ~128k to driver. #device ahd # AHA39320/29320 and onboard AIC79xx devices options AHD_REG_PRETTY_PRINT # Print register bitfields in debug # output. Adds ~215k to driver. #device amd # AMD 53C974 (Tekram DC-390(T)) #device hptiop # Highpoint RocketRaid 3xxx series #device isp # Qlogic family #device ispfw # Firmware for QLogic HBAs- normally a module #device mpt # LSI-Logic MPT-Fusion #device ncr # NCR/Symbios Logic #device sym # NCR/Symbios Logic (newer chipsets + those of `ncr') #device trm # Tekram DC395U/UW/F DC315U adapters #device adv # Advansys SCSI adapters #device adw # Advansys wide SCSI adapters #device aic # Adaptec 15[012]x SCSI adapters, AIC-6[23]60. #device bt # Buslogic/Mylex MultiMaster SCSI adapters # SCSI peripherals device scbus # SCSI bus (required for SCSI) device ch # SCSI media changers device da # Direct Access (disks) device sa # Sequential Access (tape etc) device cd # CD device pass # Passthrough device (direct SCSI access) device ses # SCSI Environmental Services (and SAF-TE) # RAID controllers interfaced to the SCSI subsystem #device amr # AMI MegaRAID #device arcmsr # Areca SATA II RAID #XXX it is not 64-bit clean, -scottl #device asr # DPT SmartRAID V, VI and Adaptec SCSI RAID #device ciss # Compaq Smart RAID 5* #device dpt # DPT Smartcache III, IV - See NOTES for options #device hptmv # Highpoint RocketRAID 182x #device hptrr # Highpoint RocketRAID 17xx, 22xx, 23xx, 25xx #device iir # Intel Integrated RAID #device ips # IBM (Adaptec) ServeRAID #device mly # Mylex AcceleRAID/eXtremeRAID #device twa # 3ware 9000 series PATA/SATA RAID # RAID controllers #device aac # Adaptec FSA RAID #device aacp # SCSI passthrough for aac (requires CAM) #device ida # Compaq Smart RAID #device mfi # LSI MegaRAID SAS #device mlx # Mylex DAC960 family #XXX pointer/int warnings #device pst # Promise Supertrak SX6000 #device twe # 3ware ATA RAID # atkbdc0 controls both the keyboard and the PS/2 mouse device atkbdc # AT keyboard controller device atkbd # AT keyboard device psm # PS/2 mouse device kbdmux # keyboard multiplexer device vga # VGA video card driver device splash # Splash screen and screen saver support # syscons is the default console driver, resembling an SCO console device sc device agp # support several AGP chipsets # PCCARD (PCMCIA) support # PCMCIA and cardbus bridge support #device cbb # cardbus (yenta) bridge #device pccard # PC Card (16-bit) bus #device cardbus # CardBus (32-bit) bus # Serial (COM) ports device uart # Generic UART driver # Parallel port device ppc device ppbus # Parallel port bus (required) device lpt # Printer device plip # TCP/IP over parallel device ppi # Parallel port interface device #device vpo # Requires scbus and da # If you've got a "dumb" serial or parallel PCI card that is # supported by the puc(4) glue driver, uncomment the following # line to enable it (connects to sio, uart and/or ppc drivers): #device puc # PCI Ethernet NICs. #device de # DEC/Intel DC21x4x (``Tulip'') device em # Intel PRO/1000 Gigabit Ethernet Family #device igb # Intel PRO/1000 PCIE Server Gigabit Family #device ixgbe # Intel PRO/10GbE PCIE Ethernet Family #device le # AMD Am7900 LANCE and Am79C9xx PCnet #device ti # Alteon Networks Tigon I/II gigabit Ethernet #device txp # 3Com 3cR990 (``Typhoon'') #device vx # 3Com 3c590, 3c595 (``Vortex'') # PCI Ethernet NICs that use the common MII bus controller code. # NOTE: Be sure to keep the 'device miibus' line in order to use these NICs! device miibus # MII bus support #device ae # Attansic/Atheros L2 FastEthernet #device age # Attansic/Atheros L1 Gigabit Ethernet #device alc # Atheros AR8131/AR8132 Ethernet #device ale # Atheros AR8121/AR8113/AR8114 Ethernet #device bce # Broadcom BCM5706/BCM5708 Gigabit Ethernet #device bfe # Broadcom BCM440x 10/100 Ethernet #device bge # Broadcom BCM570xx Gigabit Ethernet #device dc # DEC/Intel 21143 and various workalikes #device et # Agere ET1310 10/100/Gigabit Ethernet #device fxp # Intel EtherExpress PRO/100B (82557, 82558) #device jme # JMicron JMC250 Gigabit/JMC260 Fast Ethernet #device lge # Level 1 LXT1001 gigabit Ethernet #device msk # Marvell/SysKonnect Yukon II Gigabit Ethernet device nfe # nVidia nForce MCP on-board Ethernet #device nge # NatSemi DP83820 gigabit Ethernet #device nve # nVidia nForce MCP on-board Ethernet Networking #device pcn # AMD Am79C97x PCI 10/100 (precedence over 'le') #device re # RealTek 8139C+/8169/8169S/8110S #device rl # RealTek 8129/8139 #device sf # Adaptec AIC-6915 (``Starfire'') #device sge # Silicon Integrated Systems SiS190/191 #device sis # Silicon Integrated Systems SiS 900/SiS 7016 #device sk # SysKonnect SK-984x & SK-982x gigabit Ethernet #device ste # Sundance ST201 (D-Link DFE-550TX) #device stge # Sundance/Tamarack TC9021 gigabit Ethernet #device tl # Texas Instruments ThunderLAN #device tx # SMC EtherPower II (83c170 ``EPIC'') #device vge # VIA VT612x gigabit Ethernet #device vr # VIA Rhine, Rhine II #device wb # Winbond W89C840F #device xl # 3Com 3c90x (``Boomerang'', ``Cyclone'') # ISA Ethernet NICs. pccard NICs included. #device cs # Crystal Semiconductor CS89x0 NIC # 'device ed' requires 'device miibus' #device ed # NE[12]000, SMC Ultra, 3c503, DS8390 cards #device ex # Intel EtherExpress Pro/10 and Pro/10+ #device ep # Etherlink III based cards #device fe # Fujitsu MB8696x based cards #device sn # SMC's 9000 series of Ethernet chips #device xe # Xircom pccard Ethernet # Wireless NIC cards #device wlan # 802.11 support options IEEE80211_DEBUG # enable debug msgs options IEEE80211_AMPDU_AGE # age frames in AMPDU reorder q's options IEEE80211_SUPPORT_MESH # enable 802.11s draft support #device wlan_wep # 802.11 WEP support #device wlan_ccmp # 802.11 CCMP support #device wlan_tkip # 802.11 TKIP support #device wlan_amrr # AMRR transmit rate control algorithm #device an # Aironet 4500/4800 802.11 wireless NICs. #device ath # Atheros pci/cardbus NIC's #device ath_hal # pci/cardbus chip support options AH_SUPPORT_AR5416 # enable AR5416 tx/rx descriptors #device ath_rate_sample # SampleRate tx rate control for ath #device ral # Ralink Technology RT2500 wireless NICs. #device wi # WaveLAN/Intersil/Symbol 802.11 wireless NICs. # Pseudo devices. device loop # Network loopback device random # Entropy device device ether # Ethernet support device vlan # 802.1Q VLAN support device tun # Packet tunnel. device pty # BSD-style compatibility pseudo ttys device md # Memory "disks" #device gif # IPv6 and IPv4 tunneling #device faith # IPv6-to-IPv4 relaying (translation) device firmware # firmware assist module # The `bpf' device enables the Berkeley Packet Filter. # Be aware of the administrative consequences of enabling this! # Note that 'bpf' is required for DHCP. device bpf # Berkeley packet filter # USB support options USB_DEBUG # enable debug msgs device uhci # UHCI PCI->USB interface device ohci # OHCI PCI->USB interface device ehci # EHCI PCI->USB interface (USB 2.0) device usb # USB Bus (required) #device udbp # USB Double Bulk Pipe devices device uhid # "Human Interface Devices" device ukbd # Keyboard device ulpt # Printer device umass # Disks/Mass storage - Requires scbus and da device ums # Mouse device urio # Diamond Rio 500 MP3 player # USB Serial devices #device uark # Technologies ARK3116 based serial adapters #device ubsa # Belkin F5U103 and compatible serial adapters #device uftdi # For FTDI usb serial adapters #device uipaq # Some WinCE based devices #device uplcom # Prolific PL-2303 serial adapters #device uslcom # SI Labs CP2101/CP2102 serial adapters #device uvisor # Visor and Palm devices #device uvscom # USB serial support for DDI pocket's PHS # USB Ethernet, requires miibus #device aue # ADMtek USB Ethernet #device axe # ASIX Electronics USB Ethernet #device cdce # Generic USB over Ethernet #device cue # CATC USB Ethernet #device kue # Kawasaki LSI USB Ethernet #device rue # RealTek RTL8150 USB Ethernet #device udav # Davicom DM9601E USB # USB Wireless #device rum # Ralink Technology RT2501USB wireless NICs #device uath # Atheros AR5523 wireless NICs #device ural # Ralink Technology RT2500USB wireless NICs #device zyd # ZyDAS zb1211/zb1211b wireless NICs # FireWire support #device firewire # FireWire bus code #device sbp # SCSI over FireWire (Requires scbus and da) #device fwe # Ethernet over FireWire (non-standard!) #device fwip # IP over FireWire (RFC 2734,3146) #device dcons # Dumb console driver #device dcons_crom # Configuration ROM for dcons -- end Fix: No, but I would like one :) How-To-Repeat: The problem happens to me constantly, sometimes within minutes or seconds of booting. Would not know how you could possibly reproduce my environment.
Seems to be a ZFS issue... it happens on zpool import, or just accessing the pool, though a zpool scrub passes just fine. And opensolaris has NO trouble with the same pool. Wish I knew how to trouble shoot the issue further, though. Martin.
Responsible Changed From-To: freebsd-bugs->freebsd-fs possibly a ZFS issue.
Doublefault is often a stack overflow problem, not sure if this is the case here. Can you try to recompile kernel with larger stack size and test again (bump KSTACK_PAGES)? -- Andriy Gapon
I have a similiar problem with 8.2-RC on two SuperMicro 6026T-3RF servers with 24GB RAM, but it happens randomly on heavy ZFS usage. Is the server a SuperMicro? Dmesg looks very like a SuperMicro server with LSI SAS controller.
Sorry, didn't see this was vmware :)