Bug 146941

Summary: [zfs] [panic] Kernel Double Fault - Happens constantly
Product: Base System Reporter: Martin Minkus <martin.minkus>
Component: kernAssignee: freebsd-fs (Nobody) <fs>
Status: Closed Feedback Timeout    
Severity: Affects Only Me CC: smh
Priority: Normal    
Version: Unspecified   
Hardware: Any   
OS: Any   

Description Martin Minkus 2010-05-25 06:20:01 UTC 
Running the latest 8-STABLE. I've re-cvs'd and rebuilt the kernel & world
a few times the last 2 days and the problem won't go away.

I recompiled with INVARIANTS, WITNESS, KDB, etc to get a backtrace and any
other information that might be required to resolve this problem.

Screenshots of the panic and backtrace:

http://bayimg.com/oAMIiAaCP
http://bayimg.com/oAmIJAAcP
http://bayimg.com/oaMIKaacp


dmesg output:

Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.1-PRERELEASE #3: Tue May 25 14:32:02 NZST 2010
    martinm@kinetic.pulse.local:/usr/obj/usr/src/sys/DIAMOND amd64
WARNING: WITNESS option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU           E5320  @ 1.86GHz (1866.43-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x6f8  Family = 6  Model = f  Stepping = 8
  Features=0xfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS>
  Features2=0x2201<SSE3,SSSE3,CX16>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant
real memory  = 1073741824 (1024 MB)
avail memory = 1016025088 (968 MB)
ACPI APIC Table: <PTLTD  	 APIC  >
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <PTLTD   RSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1050-0x105f at device 7.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
pci0: <bridge> at device 7.3 (no driver attached)
vgapci0: <VGA-compatible display> port 0x1060-0x106f mem 0xf8000000-0xfbffffff,0xf4000000-0xf47fffff at device 15.0 on pci0
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x1080-0x10ff mem 0xf4810000-0xf4810fff irq 17 at device 16.0 on pci0
mpt0: [ITHREAD]
mpt0: MPI Version=1.2.0.0
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.1> port 0x1400-0x143f mem 0xf4820000-0xf483ffff,0xf4800000-0xf480ffff irq 18 at device 17.0 on pci0
em0: Memory Access and/or Bus Master bits were not set!
em0: [FILTER]
em0: Ethernet address: 00:50:56:a5:20:03
acpi_acad0: <AC Adapter> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model IntelliMouse, device ID 3
ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppc0: [ITHREAD]
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
plip0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart1: [FILTER]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xca000-0xcafff,0xdc000-0xdffff,0xe0000-0xe3fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
            to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version 3
ZFS storage pool version 14
Timecounter "TSC" frequency 1866434311 Hz quality 800
Timecounters tick every 10.000 msec
acd0: CDROM <VMware Virtual IDE CDROM Drive/00000001> at ata0-master UDMA33 
da0 at mpt0 bus 0 scbus0 target 0 lun 0
da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device 
da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da0: Command Queueing enabled
da0: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C)
da1 at mpt0 bus 0 scbus0 target 1 lun 0
da1: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device 
da1: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da1: Command Queueing enabled
da1: 143360MB (293601280 512 byte sectors: 255H 63S/T 18275C)
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/da0s1a
WARNING: / was not properly dismounted
VMware memory control driver initialized

Fatal double fault
rip = 0xffffffff8037ff6b
rsp = 0xffffff808d05ef40
rbp = 0xffffff808d05f070
cpuid = 0; apic id = 00
panic: double fault
cpuid = 0
KDB: enter: panic
Uptime: 1m39s
Cannot dump. Device not defined or unavailable.
Automatic reboot in 15 seconds - press a key on the console to abort
Rebooting...
Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.1-PRERELEASE #3: Tue May 25 14:32:02 NZST 2010
    martinm@kinetic.pulse.local:/usr/obj/usr/src/sys/DIAMOND amd64
WARNING: WITNESS option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU           E5320  @ 1.86GHz (1866.46-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x6f8  Family = 6  Model = f  Stepping = 8
  Features=0xfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS>
  Features2=0x2201<SSE3,SSSE3,CX16>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant
real memory  = 1073741824 (1024 MB)
avail memory = 1016025088 (968 MB)
ACPI APIC Table: <PTLTD  	 APIC  >
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <PTLTD   RSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1050-0x105f at device 7.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
pci0: <bridge> at device 7.3 (no driver attached)
vgapci0: <VGA-compatible display> port 0x1060-0x106f mem 0xf8000000-0xfbffffff,0xf4000000-0xf47fffff at device 15.0 on pci0
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x1080-0x10ff mem 0xf4810000-0xf4810fff irq 17 at device 16.0 on pci0
mpt0: [ITHREAD]
mpt0: MPI Version=1.2.0.0
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.1> port 0x1400-0x143f mem 0xf4820000-0xf483ffff,0xf4800000-0xf480ffff irq 18 at device 17.0 on pci0
em0: Memory Access and/or Bus Master bits were not set!
em0: [FILTER]
em0: Ethernet address: 00:50:56:a5:20:03
acpi_acad0: <AC Adapter> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model IntelliMouse, device ID 3
ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppc0: [ITHREAD]
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
plip0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart1: [FILTER]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xca000-0xcafff,0xdc000-0xdffff,0xe0000-0xe3fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
            to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version 3
ZFS storage pool version 14
Timecounter "TSC" frequency 1866457425 Hz quality 800
Timecounters tick every 10.000 msec
acd0: CDROM <VMware Virtual IDE CDROM Drive/00000001> at ata0-master UDMA33 
da0 at mpt0 bus 0 scbus0 target 0 lun 0
da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device 
da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da0: Command Queueing enabled
da0: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C)
da1 at mpt0 bus 0 scbus0 target 1 lun 0
da1: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device 
da1: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da1: Command Queueing enabled
da1: 143360MB (293601280 512 byte sectors: 255H 63S/T 18275C)
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/da0s1a
WARNING: / was not properly dismounted
/: mount pending error: blocks 444 files 29
VMware memory control driver initialized

Fatal double fault
rip = 0xffffffff8037ff6b
rsp = 0xffffff808d05ef40
rbp = 0xffffff808d05f070
cpuid = 0; apic id = 00
panic: double fault
cpuid = 0
KDB: enter: panic
Uptime: 1m39s
Cannot dump. Device not defined or unavailable.
Automatic reboot in 15 seconds - press a key on the console to abort
--> Press a key on the console to reboot,
--> or switch off the system now.
Rebooting...
Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.1-PRERELEASE #3: Tue May 25 14:32:02 NZST 2010
    martinm@kinetic.pulse.local:/usr/obj/usr/src/sys/DIAMOND amd64
WARNING: WITNESS option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU           E5320  @ 1.86GHz (1866.44-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x6f8  Family = 6  Model = f  Stepping = 8
  Features=0xfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS>
  Features2=0x2201<SSE3,SSSE3,CX16>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant
real memory  = 1073741824 (1024 MB)
avail memory = 1016025088 (968 MB)
ACPI APIC Table: <PTLTD  	 APIC  >
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <PTLTD   RSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1050-0x105f at device 7.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
pci0: <bridge> at device 7.3 (no driver attached)
vgapci0: <VGA-compatible display> port 0x1060-0x106f mem 0xf8000000-0xfbffffff,0xf4000000-0xf47fffff at device 15.0 on pci0
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x1080-0x10ff mem 0xf4810000-0xf4810fff irq 17 at device 16.0 on pci0
mpt0: [ITHREAD]
mpt0: MPI Version=1.2.0.0
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.1> port 0x1400-0x143f mem 0xf4820000-0xf483ffff,0xf4800000-0xf480ffff irq 18 at device 17.0 on pci0
em0: Memory Access and/or Bus Master bits were not set!
em0: [FILTER]
em0: Ethernet address: 00:50:56:a5:20:03
acpi_acad0: <AC Adapter> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model IntelliMouse, device ID 3
ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppc0: [ITHREAD]
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
plip0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart1: [FILTER]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xca000-0xcafff,0xdc000-0xdffff,0xe0000-0xe3fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
            to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version 3
ZFS storage pool version 14
Timecounter "TSC" frequency 1866436348 Hz quality 800
Timecounters tick every 10.000 msec
acd0: CDROM <VMware Virtual IDE CDROM Drive/00000001> at ata0-master UDMA33 
da0 at mpt0 bus 0 scbus0 target 0 lun 0
da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device 
da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da0: Command Queueing enabled
da0: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C)
da1 at mpt0 bus 0 scbus0 target 1 lun 0
da1: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device 
da1: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da1: Command Queueing enabled
da1: 143360MB (293601280 512 byte sectors: 255H 63S/T 18275C)
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/da0s1a
WARNING: / was not properly dismounted
/: mount pending error: blocks 388 files 20
VMware memory control driver initialized

Fatal double fault
rip = 0xffffffff8037ff6b
rsp = 0xffffff808d05ef40
rbp = 0xffffff808d05f070
cpuid = 0; apic id = 00
panic: double fault
cpuid = 0
KDB: enter: panic
Uptime: 1m39s
Cannot dump. Device not defined or unavailable.
Automatic reboot in 15 seconds - press a key on the console to abort
--> Press a key on the console to reboot,
--> or switch off the system now.
Rebooting...
Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.1-PRERELEASE #3: Tue May 25 14:32:02 NZST 2010
    martinm@kinetic.pulse.local:/usr/obj/usr/src/sys/DIAMOND amd64
WARNING: WITNESS option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU           E5320  @ 1.86GHz (1866.46-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x6f8  Family = 6  Model = f  Stepping = 8
  Features=0xfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS>
  Features2=0x2201<SSE3,SSSE3,CX16>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant
real memory  = 1073741824 (1024 MB)
avail memory = 1016025088 (968 MB)
ACPI APIC Table: <PTLTD  	 APIC  >
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <PTLTD   RSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1050-0x105f at device 7.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
pci0: <bridge> at device 7.3 (no driver attached)
vgapci0: <VGA-compatible display> port 0x1060-0x106f mem 0xf8000000-0xfbffffff,0xf4000000-0xf47fffff at device 15.0 on pci0
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x1080-0x10ff mem 0xf4810000-0xf4810fff irq 17 at device 16.0 on pci0
mpt0: [ITHREAD]
mpt0: MPI Version=1.2.0.0
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.1> port 0x1400-0x143f mem 0xf4820000-0xf483ffff,0xf4800000-0xf480ffff irq 18 at device 17.0 on pci0
em0: Memory Access and/or Bus Master bits were not set!
em0: [FILTER]
em0: Ethernet address: 00:50:56:a5:20:03
acpi_acad0: <AC Adapter> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model IntelliMouse, device ID 3
ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppc0: [ITHREAD]
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
plip0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart1: [FILTER]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xca000-0xcafff,0xdc000-0xdffff,0xe0000-0xe3fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
            to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version 3
ZFS storage pool version 14
Timecounter "TSC" frequency 1866456340 Hz quality 800
Timecounters tick every 10.000 msec
acd0: CDROM <VMware Virtual IDE CDROM Drive/00000001> at ata0-master UDMA33 
da0 at mpt0 bus 0 scbus0 target 0 lun 0
da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device 
da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da0: Command Queueing enabled
da0: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C)
da1 at mpt0 bus 0 scbus0 target 1 lun 0
da1: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device 
da1: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da1: Command Queueing enabled
da1: 143360MB (293601280 512 byte sectors: 255H 63S/T 18275C)
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/da0s1a
WARNING: / was not properly dismounted
/: mount pending error: blocks 396 files 20



KINETIC kernel config:

#
# GENERIC -- Generic kernel configuration file for FreeBSD/amd64
#
# For more information on this file, please read the config(5) manual page,
# and/or the handbook section on Kernel Configuration Files:
#
#    http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html
#
# The handbook is also available locally in /usr/share/doc/handbook
# if you've installed the doc distribution, otherwise always see the
# FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the
# latest information.
#
# An exhaustive list of options and more detailed explanations of the
# device lines is also present in the ../../conf/NOTES and NOTES files.
# If you are in doubt as to the purpose or necessity of a line, check first
# in NOTES.
#
# $FreeBSD: src/sys/amd64/conf/GENERIC,v 1.531.2.13 2010/05/02 06:24:17 imp Exp $

cpu		HAMMER
ident		KINETIC

# To statically compile in device wiring instead of /boot/device.hints
#hints		"GENERIC.hints"		# Default places to look for devices.

# Use the following to compile in values accessible to the kernel
# through getenv() (or kenv(1) in userland). The format of the file
# is 'variable=value', see kenv(1)
#
# env		"GENERIC.env"

#makeoptions	DEBUG=-g		# Build kernel with gdb(1) debug symbols

options 	SCHED_ULE		# ULE scheduler
options 	PREEMPTION		# Enable kernel thread preemption
options 	INET			# InterNETworking
#options 	INET6			# IPv6 communications protocols
options 	SCTP			# Stream Control Transmission Protocol
options 	FFS			# Berkeley Fast Filesystem
options 	SOFTUPDATES		# Enable FFS soft updates support
options 	UFS_ACL			# Support for access control lists
options 	UFS_DIRHASH		# Improve performance on big directories
options 	UFS_GJOURNAL		# Enable gjournal-based UFS journaling
options 	MD_ROOT			# MD is a potential root device
options 	NFSCLIENT		# Network Filesystem Client
options 	NFSSERVER		# Network Filesystem Server
options 	NFSLOCKD		# Network Lock Manager
options 	NFS_ROOT		# NFS usable as /, requires NFSCLIENT
options 	MSDOSFS			# MSDOS Filesystem
options 	CD9660			# ISO 9660 Filesystem
options 	PROCFS			# Process filesystem (requires PSEUDOFS)
options 	PSEUDOFS		# Pseudo-filesystem framework
options 	GEOM_PART_GPT		# GUID Partition Tables.
options 	GEOM_LABEL		# Provides labelization
options 	COMPAT_43TTY		# BSD 4.3 TTY compat (sgtty)
options 	COMPAT_FREEBSD32	# Compatible with i386 binaries
options 	COMPAT_FREEBSD4		# Compatible with FreeBSD4
options 	COMPAT_FREEBSD5		# Compatible with FreeBSD5
options 	COMPAT_FREEBSD6		# Compatible with FreeBSD6
options 	COMPAT_FREEBSD7		# Compatible with FreeBSD7
options 	SCSI_DELAY=5000		# Delay (in ms) before probing SCSI
options 	KTRACE			# ktrace(1) support
options 	STACK			# stack(9) support
options 	SYSVSHM			# SYSV-style shared memory
options 	SYSVMSG			# SYSV-style message queues
options 	SYSVSEM			# SYSV-style semaphores
options 	P1003_1B_SEMAPHORES	# POSIX-style semaphores
options 	_KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options 	PRINTF_BUFR_SIZE=128	# Prevent printf output being interspersed.
options 	KBD_INSTALL_CDEV	# install a CDEV entry in /dev
options 	HWPMC_HOOKS		# Necessary kernel hooks for hwpmc(4)
options 	AUDIT			# Security event auditing
options 	MAC			# TrustedBSD MAC Framework
options		FLOWTABLE		# per-cpu routing cache
#options 	KDTRACE_FRAME		# Ensure frames are compiled in
#options 	KDTRACE_HOOKS		# Kernel DTrace hooks
options 	INCLUDE_CONFIG_FILE     # Include this file in kernel

# Make an SMP-capable kernel by default
options 	SMP			# Symmetric MultiProcessor Kernel

# CPU frequency control
device		cpufreq

# Bus support.
device		acpi
device		pci

# Floppy drives
device		fdc

# ATA and ATAPI devices
device		ata
device		atadisk		# ATA disk drives
device		ataraid		# ATA RAID drives
device		atapicd		# ATAPI CDROM drives
device		atapifd		# ATAPI floppy drives
device		atapist		# ATAPI tape drives
options 	ATA_STATIC_ID	# Static device numbering

# SCSI Controllers
#device		ahc		# AHA2940 and onboard AIC7xxx devices
options 	AHC_REG_PRETTY_PRINT	# Print register bitfields in debug
					# output.  Adds ~128k to driver.
#device		ahd		# AHA39320/29320 and onboard AIC79xx devices
options 	AHD_REG_PRETTY_PRINT	# Print register bitfields in debug
					# output.  Adds ~215k to driver.
#device		amd		# AMD 53C974 (Tekram DC-390(T))
#device		hptiop		# Highpoint RocketRaid 3xxx series
#device		isp		# Qlogic family
#device		ispfw		# Firmware for QLogic HBAs- normally a module
#device		mpt		# LSI-Logic MPT-Fusion
#device		ncr		# NCR/Symbios Logic
#device		sym		# NCR/Symbios Logic (newer chipsets + those of `ncr')
#device		trm		# Tekram DC395U/UW/F DC315U adapters

#device		adv		# Advansys SCSI adapters
#device		adw		# Advansys wide SCSI adapters
#device		aic		# Adaptec 15[012]x SCSI adapters, AIC-6[23]60.
#device		bt		# Buslogic/Mylex MultiMaster SCSI adapters

# SCSI peripherals
device		scbus		# SCSI bus (required for SCSI)
device		ch		# SCSI media changers
device		da		# Direct Access (disks)
device		sa		# Sequential Access (tape etc)
device		cd		# CD
device		pass		# Passthrough device (direct SCSI access)
device		ses		# SCSI Environmental Services (and SAF-TE)

# RAID controllers interfaced to the SCSI subsystem
#device		amr		# AMI MegaRAID
#device		arcmsr		# Areca SATA II RAID
#XXX it is not 64-bit clean, -scottl
#device		asr		# DPT SmartRAID V, VI and Adaptec SCSI RAID
#device		ciss		# Compaq Smart RAID 5*
#device		dpt		# DPT Smartcache III, IV - See NOTES for options
#device		hptmv		# Highpoint RocketRAID 182x
#device		hptrr		# Highpoint RocketRAID 17xx, 22xx, 23xx, 25xx
#device		iir		# Intel Integrated RAID
#device		ips		# IBM (Adaptec) ServeRAID
#device		mly		# Mylex AcceleRAID/eXtremeRAID
#device		twa		# 3ware 9000 series PATA/SATA RAID

# RAID controllers
#device		aac		# Adaptec FSA RAID
#device		aacp		# SCSI passthrough for aac (requires CAM)
#device		ida		# Compaq Smart RAID
#device		mfi		# LSI MegaRAID SAS
#device		mlx		# Mylex DAC960 family
#XXX pointer/int warnings
#device		pst		# Promise Supertrak SX6000
#device		twe		# 3ware ATA RAID

# atkbdc0 controls both the keyboard and the PS/2 mouse
device		atkbdc		# AT keyboard controller
device		atkbd		# AT keyboard
device		psm		# PS/2 mouse

device		kbdmux		# keyboard multiplexer

device		vga		# VGA video card driver

device		splash		# Splash screen and screen saver support

# syscons is the default console driver, resembling an SCO console
device		sc

device		agp		# support several AGP chipsets

# PCCARD (PCMCIA) support
# PCMCIA and cardbus bridge support
#device		cbb		# cardbus (yenta) bridge
#device		pccard		# PC Card (16-bit) bus
#device		cardbus		# CardBus (32-bit) bus

# Serial (COM) ports
device		uart		# Generic UART driver

# Parallel port
device		ppc
device		ppbus		# Parallel port bus (required)
device		lpt		# Printer
device		plip		# TCP/IP over parallel
device		ppi		# Parallel port interface device
#device		vpo		# Requires scbus and da

# If you've got a "dumb" serial or parallel PCI card that is
# supported by the puc(4) glue driver, uncomment the following
# line to enable it (connects to sio, uart and/or ppc drivers):
#device		puc

# PCI Ethernet NICs.
#device		de		# DEC/Intel DC21x4x (``Tulip'')
device		em		# Intel PRO/1000 Gigabit Ethernet Family
#device		igb		# Intel PRO/1000 PCIE Server Gigabit Family
#device		ixgbe		# Intel PRO/10GbE PCIE Ethernet Family
#device		le		# AMD Am7900 LANCE and Am79C9xx PCnet
#device		ti		# Alteon Networks Tigon I/II gigabit Ethernet
#device		txp		# 3Com 3cR990 (``Typhoon'')
#device		vx		# 3Com 3c590, 3c595 (``Vortex'')

# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
device		miibus		# MII bus support
#device		ae		# Attansic/Atheros L2 FastEthernet
#device		age		# Attansic/Atheros L1 Gigabit Ethernet
#device		alc		# Atheros AR8131/AR8132 Ethernet
#device		ale		# Atheros AR8121/AR8113/AR8114 Ethernet
#device		bce		# Broadcom BCM5706/BCM5708 Gigabit Ethernet
#device		bfe		# Broadcom BCM440x 10/100 Ethernet
#device		bge		# Broadcom BCM570xx Gigabit Ethernet
#device		dc		# DEC/Intel 21143 and various workalikes
#device		et		# Agere ET1310 10/100/Gigabit Ethernet
#device		fxp		# Intel EtherExpress PRO/100B (82557, 82558)
#device		jme		# JMicron JMC250 Gigabit/JMC260 Fast Ethernet
#device		lge		# Level 1 LXT1001 gigabit Ethernet
#device		msk		# Marvell/SysKonnect Yukon II Gigabit Ethernet
device		nfe		# nVidia nForce MCP on-board Ethernet
#device		nge		# NatSemi DP83820 gigabit Ethernet
#device		nve		# nVidia nForce MCP on-board Ethernet Networking
#device		pcn		# AMD Am79C97x PCI 10/100 (precedence over 'le')
#device		re		# RealTek 8139C+/8169/8169S/8110S
#device		rl		# RealTek 8129/8139
#device		sf		# Adaptec AIC-6915 (``Starfire'')
#device		sge		# Silicon Integrated Systems SiS190/191
#device		sis		# Silicon Integrated Systems SiS 900/SiS 7016
#device		sk		# SysKonnect SK-984x & SK-982x gigabit Ethernet
#device		ste		# Sundance ST201 (D-Link DFE-550TX)
#device		stge		# Sundance/Tamarack TC9021 gigabit Ethernet
#device		tl		# Texas Instruments ThunderLAN
#device		tx		# SMC EtherPower II (83c170 ``EPIC'')
#device		vge		# VIA VT612x gigabit Ethernet
#device		vr		# VIA Rhine, Rhine II
#device		wb		# Winbond W89C840F
#device		xl		# 3Com 3c90x (``Boomerang'', ``Cyclone'')

# ISA Ethernet NICs.  pccard NICs included.
#device		cs		# Crystal Semiconductor CS89x0 NIC
# 'device ed' requires 'device miibus'
#device		ed		# NE[12]000, SMC Ultra, 3c503, DS8390 cards
#device		ex		# Intel EtherExpress Pro/10 and Pro/10+
#device		ep		# Etherlink III based cards
#device		fe		# Fujitsu MB8696x based cards
#device		sn		# SMC's 9000 series of Ethernet chips
#device		xe		# Xircom pccard Ethernet

# Wireless NIC cards
#device		wlan		# 802.11 support
options 	IEEE80211_DEBUG	# enable debug msgs
options 	IEEE80211_AMPDU_AGE # age frames in AMPDU reorder q's
options 	IEEE80211_SUPPORT_MESH	# enable 802.11s draft support
#device		wlan_wep	# 802.11 WEP support
#device		wlan_ccmp	# 802.11 CCMP support
#device		wlan_tkip	# 802.11 TKIP support
#device		wlan_amrr	# AMRR transmit rate control algorithm
#device		an		# Aironet 4500/4800 802.11 wireless NICs.
#device		ath		# Atheros pci/cardbus NIC's
#device		ath_hal		# pci/cardbus chip support
options 	AH_SUPPORT_AR5416	# enable AR5416 tx/rx descriptors
#device		ath_rate_sample	# SampleRate tx rate control for ath
#device		ral		# Ralink Technology RT2500 wireless NICs.
#device		wi		# WaveLAN/Intersil/Symbol 802.11 wireless NICs.

# Pseudo devices.
device		loop		# Network loopback
device		random		# Entropy device
device		ether		# Ethernet support
device		vlan		# 802.1Q VLAN support
device		tun		# Packet tunnel.
device		pty		# BSD-style compatibility pseudo ttys
device		md		# Memory "disks"
#device		gif		# IPv6 and IPv4 tunneling
#device		faith		# IPv6-to-IPv4 relaying (translation)
device		firmware	# firmware assist module

# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device		bpf		# Berkeley packet filter

# USB support
options 	USB_DEBUG	# enable debug msgs
device		uhci		# UHCI PCI->USB interface
device		ohci		# OHCI PCI->USB interface
device		ehci		# EHCI PCI->USB interface (USB 2.0)
device		usb		# USB Bus (required)
#device		udbp		# USB Double Bulk Pipe devices
device		uhid		# "Human Interface Devices"
device		ukbd		# Keyboard
device		ulpt		# Printer
device		umass		# Disks/Mass storage - Requires scbus and da
device		ums		# Mouse
device		urio		# Diamond Rio 500 MP3 player
# USB Serial devices
#device		uark		# Technologies ARK3116 based serial adapters
#device		ubsa		# Belkin F5U103 and compatible serial adapters
#device		uftdi		# For FTDI usb serial adapters
#device		uipaq		# Some WinCE based devices
#device		uplcom		# Prolific PL-2303 serial adapters
#device		uslcom		# SI Labs CP2101/CP2102 serial adapters
#device		uvisor		# Visor and Palm devices
#device		uvscom		# USB serial support for DDI pocket's PHS
# USB Ethernet, requires miibus
#device		aue		# ADMtek USB Ethernet
#device		axe		# ASIX Electronics USB Ethernet
#device		cdce		# Generic USB over Ethernet
#device		cue		# CATC USB Ethernet
#device		kue		# Kawasaki LSI USB Ethernet
#device		rue		# RealTek RTL8150 USB Ethernet
#device		udav		# Davicom DM9601E USB
# USB Wireless
#device		rum		# Ralink Technology RT2501USB wireless NICs
#device		uath		# Atheros AR5523 wireless NICs
#device		ural		# Ralink Technology RT2500USB wireless NICs
#device		zyd		# ZyDAS zb1211/zb1211b wireless NICs

# FireWire support
#device		firewire	# FireWire bus code
#device		sbp		# SCSI over FireWire (Requires scbus and da)
#device		fwe		# Ethernet over FireWire (non-standard!)
#device		fwip		# IP over FireWire (RFC 2734,3146)
#device		dcons		# Dumb console driver
#device		dcons_crom	# Configuration ROM for dcons

-- end

Fix: 

No, but I would like one :)
How-To-Repeat: The problem happens to me constantly, sometimes within minutes or seconds of booting. Would not know how you could possibly reproduce my environment.
Comment 1 Martin Minkus 2010-05-26 02:45:03 UTC 
Seems to be a ZFS issue... it happens on zpool import, or just accessing
the pool, though a zpool scrub passes just fine.

And opensolaris has NO trouble with the same pool.

Wish I knew how to trouble shoot the issue further, though.

Martin.
Comment 2 Mark Linimon freebsd_committer freebsd_triage 2010-08-04 04:26:07 UTC 
Responsible Changed
From-To: freebsd-bugs->freebsd-fs

possibly a ZFS issue.
Comment 3 Andriy Gapon 2010-08-04 12:48:26 UTC 
Doublefault is often a stack overflow problem, not sure if this is the case here.
Can you try to recompile kernel with larger stack size and test again (bump
KSTACK_PAGES)?

-- 
Andriy Gapon
Comment 4 Martin Matuska freebsd_committer freebsd_triage 2011-01-24 23:58:02 UTC 
I have a similiar problem with 8.2-RC on two SuperMicro 6026T-3RF
servers with 24GB RAM, but it happens randomly on heavy ZFS usage.
Is the server a SuperMicro? Dmesg looks very like a SuperMicro server
with LSI SAS controller.
Comment 5 Martin Matuska freebsd_committer freebsd_triage 2011-01-25 00:06:41 UTC 
Sorry, didn't see this was vmware :)