Bug 207667 - Reproducable kernel panics whilst scrubbing a zvol pool...
Summary: Reproducable kernel panics whilst scrubbing a zvol pool...
Status: Closed Unable to Reproduce
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 9.3-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-fs mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-03-03 02:19 UTC by Michelle Sullivan
Modified: 2017-03-14 19:17 UTC (History)
5 users (show)

See Also:


Attachments
possible patch for stable/9 zfs (3.16 KB, patch)
2016-04-06 21:10 UTC, Andriy Gapon
no flags Details | Diff
console screenshot from the link (209.73 KB, image/jpeg)
2017-03-12 10:27 UTC, Andriy Gapon
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Michelle Sullivan 2016-03-03 02:19:47 UTC
[michelle@colossus /usr/home/michelle]$ df -k
Filesystem          1024-blocks        Used      Avail Capacity  Mounted on
/dev/mirror/roots1a   112728536    55676220   48034036    54%    /
devfs                         1           1          0   100%    /dev
VirtualDisks           73256856          31   73256825     0%    /VirtualDisks
sorbs               35884081755 31821431244 4062650511    89%    /sorbs
[michelle@colossus /usr/home/michelle]$ zpool status -v
  pool: VirtualDisks
 state: ONLINE
status: One or more devices are configured to use a non-native block size.
    Expect reduced performance.
action: Replace affected devices with devices that support the
    configured block size, or migrate data to a properly configured
    pool.
  scan: none requested
config:

    NAME                       STATE     READ WRITE CKSUM
    VirtualDisks               ONLINE       0     0     0
      zvol/sorbs/VirtualDisks  ONLINE       0     0     0  block size: 512B configured, 8192B native

errors: No known data errors

  pool: sorbs
 state: ONLINE
  scan: scrub repaired 0 in 60h14m with 0 errors on Mon Apr 13 17:05:29 2015
config:

    NAME        STATE     READ WRITE CKSUM
    sorbs       ONLINE       0     0     0
      raidz2-0  ONLINE       0     0     0
        mfid0   ONLINE       0     0     0
        mfid1   ONLINE       0     0     0
        mfid2   ONLINE       0     0     0
        mfid3   ONLINE       0     0     0
        mfid4   ONLINE       0     0     0
        mfid5   ONLINE       0     0     0
        mfid6   ONLINE       0     0     0
        mfid7   ONLINE       0     0     0
        mfid15  ONLINE       0     0     0
        mfid8   ONLINE       0     0     0
        mfid9   ONLINE       0     0     0
        mfid10  ONLINE       0     0     0
        mfid11  ONLINE       0     0     0
        mfid12  ONLINE       0     0     0
        mfid13  ONLINE       0     0     0
    spares
      mfid14    AVAIL  

errors: No known data errors
[michelle@colossus /usr/home/michelle]$ sudo su -
Password:
root@colossus:~ # zpool list
NAME           SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
VirtualDisks  1016G   593G   423G    58%  1.00x  ONLINE  -
sorbs         40.8T  35.7T  5.03T    87%  1.00x  ONLINE  -
root@colossus:~ # zpool list -v
NAME                        SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
VirtualDisks               1016G   593G   423G    58%  1.00x  ONLINE  -
  zvol/sorbs/VirtualDisks  1016G   593G   423G         -
sorbs                      40.8T  35.7T  5.03T    87%  1.00x  ONLINE  -
  raidz2                   40.8T  35.7T  5.03T         -
    mfid0                      -      -      -         -
    mfid1                      -      -      -         -
    mfid2                      -      -      -         -
    mfid3                      -      -      -         -
    mfid4                      -      -      -         -
    mfid5                      -      -      -         -
    mfid6                      -      -      -         -
    mfid7                      -      -      -         -
    mfid15                     -      -      -         -
    mfid8                      -      -      -         -
    mfid9                      -      -      -         -
    mfid10                     -      -      -         -
    mfid11                     -      -      -         -
    mfid12                     -      -      -         -
    mfid13                     -      -      -         -




No idea how to fix it without reloading everything... but one thing is for sure...

root@colossus:~ # zpool scrub VirtualDisks

Will cause instant and completely reproducible kernel panics..

..and for some reason I'm not getting cores...

Here's the dmesg, /var/crash and /etc/rc.conf.local

[michelle@colossus /usr/home/michelle]$ dmesg
Copyright (c) 1992-2014 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.3-RELEASE-p33 #0: Wed Jan 13 17:55:39 UTC 2016
    root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
gcc version 4.2.1 20070831 patched [FreeBSD]
CPU: Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz (3300.10-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x306a9  Family = 0x6  Model = 0x3a  Stepping = 9
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7fbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  Standard Extended Features=0x281<GSFSBASE,SMEP,ENHMOVSB>
  TSC: P-state invariant, performance statistics
real memory  = 17188257792 (16392 MB)
avail memory = 16491753472 (15727 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <SUPERM SMCI--MB>
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 SMT threads
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
 cpu4 (AP): APIC ID:  4
 cpu5 (AP): APIC ID:  5
 cpu6 (AP): APIC ID:  6
 cpu7 (AP): APIC ID:  7
ACPI Warning: FADT (revision 5) is longer than ACPI 2.0 version, truncating length 268 to 244 (20110527/tbfadt-320)
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <SUPERM SMCI--MB> on motherboard
acpi0: Power Button (fixed)
acpi0: reservation of 67, 1 (4) failed
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 1.1 on pci0
pci2: <ACPI PCI bus> on pcib2
mfi0: <LSI MegaSAS Gen2> port 0xe000-0xe0ff mem 0xdfa60000-0xdfa63fff,0xdfa00000-0xdfa3ffff irq 17 at device 0.0 on pci2
mfi0: Using MSI
mfi0: Megaraid SAS driver Ver 4.23 
mfi0: FW MaxCmds = 1008, limiting to 128
em0: <Intel(R) PRO/1000 Network Connection 7.3.8> port 0xf020-0xf03f mem 0xdfb00000-0xdfb1ffff,0xdfb23000-0xdfb23fff irq 20 at device 25.0 on pci0
em0: Using an MSI interrupt
em0: Ethernet address: 0c:c4:7a:0b:8a:83
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xdfb22000-0xdfb223ff irq 16 at device 26.0 on pci0
usbus0: EHCI version 1.0
usbus0 on ehci0
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
pci4: <ACPI PCI bus> on pcib4
em1: <Intel(R) PRO/1000 Network Connection 7.3.8> port 0xd000-0xd01f mem 0xdf900000-0xdf91ffff,0xdf920000-0xdf923fff irq 16 at device 0.0 on pci4
em1: Using MSIX interrupts with 3 vectors
em1: Ethernet address: 0c:c4:7a:0b:8a:82
ehci1: <EHCI (generic) USB 2.0 controller> mem 0xdfb21000-0xdfb213ff irq 23 at device 29.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci1
pcib5: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci5: <ACPI PCI bus> on pcib5
vgapci0: <VGA-compatible display> mem 0xde000000-0xdeffffff,0xdf800000-0xdf803fff,0xdf000000-0xdf7fffff irq 23 at device 3.0 on pci5
vgapci0: Boot video device
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel Cougar Point AHCI SATA controller> port 0xf070-0xf077,0xf060-0xf063,0xf050-0xf057,0xf040-0xf043,0xf000-0xf01f mem 0xdfb20000-0xdfb207ff irq 19 at device 31.2 on pci0
ahci0: AHCI v1.30 with 6 3Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
acpi_button0: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
acpi_tz1: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model Generic PS/2 mouse, device ID 0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart2: <16550 or compatible> port 0x3e8-0x3ef irq 10 on acpi0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xce7ff,0xce800-0xcf7ff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: cannot reserve I/O port range
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
coretemp4: <CPU On-Die Thermal Sensors> on cpu4
est4: <Enhanced SpeedStep Frequency Control> on cpu4
p4tcc4: <CPU Frequency Thermal Control> on cpu4
coretemp5: <CPU On-Die Thermal Sensors> on cpu5
est5: <Enhanced SpeedStep Frequency Control> on cpu5
p4tcc5: <CPU Frequency Thermal Control> on cpu5
coretemp6: <CPU On-Die Thermal Sensors> on cpu6
est6: <Enhanced SpeedStep Frequency Control> on cpu6
p4tcc6: <CPU Frequency Thermal Control> on cpu6
coretemp7: <CPU On-Die Thermal Sensors> on cpu7
est7: <Enhanced SpeedStep Frequency Control> on cpu7
p4tcc7: <CPU Frequency Thermal Control> on cpu7
Timecounters tick every 1.000 msec
vboxdrv: fAsync=0 offMin=0x558 offMax=0x91c
mfi0: 7628 (510289460s/0x0020/info) - Shutdown command received from host
mfi0: 7629 (boot + 4s/0x0020/info) - Firmware initialization started (PCI ID 0079/1000/9276/1000)
mfi0: 7630 (boot + 4s/0x0020/info) - Firmware version 2.130.403-3066
mfi0: 7631 (boot + 4s/0x0020/info) - Firmware initialization started (PCI ID 0079/1000/9276/1000)
mfi0: 7632 (boot + 4s/0x0020/info) - Firmware version 2.130.403-3066
mfi0: 7633 (boot + 9s/0x0008/info) - Battery Present
mfi0: 7634 (boot + 10s/0x0020/info) - Package version 12.15.0-0189
mfi0: 7635 (boot + 10s/0x0020/info) - Board Revision 20D
mfi0: 7636 (boot + 31s/0x0004/info) - Enclosure PD f5(c None/p1) communication restored
mfi0: 7637 (boot + 31s/0x0002/info) - Inserted: Encl PD f5
mfi0: 7638 (boot + 31s/0x0002/info) - Inserted: PD f5(c None/p1) Info: enclPd=f5, scsiType=d, portMap=00, sasAddr=500062b2009b18bd,0000000000000000
mfi0: 7639 (boot + 31s/0x0002/info) - Inserted: PD 10(e0xf5/s4)
mfi0: 7640 (boot + 31s/0x0002/info) - Inserted: PD 10(e0xf5/s4) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b188c,0000000000000000
mfi0: 7641 (boot + 31s/0x0002/info) - Inserted: PD 11(e0xf5/s5)
mfi0: 7642 (boot + 31s/0x0002/info) - Inserted: PD 11(e0xf5/s5) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b188d,0000000000000000
mfi0: 7643 (boot + 31s/0x0002/info) - Inserted: PD 12(e0xf5/s6)
mfi0: 7644 (boot + 31s/0x0002/info) - Inserted: PD 12(e0xf5/s6) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b188e,0000000000000000
mfi0: 7645 (boot + 31s/0x0002/info) - Inserted: PD 13(e0xf5/s7)
mfi0: 7646 (boot + 31s/0x0002/info) - Inserted: PD 13(e0xf5/s7) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b188f,0000000000000000
mfi0: 7647 (boot + 31s/0x0002/info) - Inserted: PD 14(e0xf5/s0)
mfi0: 7648 (boot + 31s/0x0002/info) - Inserted: PD 14(e0xf5/s0) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b1890,0000000000000000
mfi0: 7649 (boot + 31s/0x0002/info) - Inserted: PD 15(e0xf5/s1)
mfi0: 7650 (boot + 31s/0x0002/info) - Inserted: PD 15(e0xf5/s1) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b1891,0000000000000000
mfi0: 7651 (boot + 31s/0x0002/info) - Inserted: PD 16(e0xf5/s2)
mfi0: 7652 (boot + 31s/0x0002/info) - Inserted: PD 16(e0xf5/s2) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b1892,0000000000000000
mfi0: 7653 (boot + 31s/0x0002/info) - Inserted: PD 17(e0xf5/s3)
mfi0: 7654 (boot + 31s/0x0002/info) - Inserted: PD 17(e0xf5/s3) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b1893,0000000000000000
mfi0: 7655 (boot + 31s/0x0002/info) - Inserted: PD 18(e0xf5/s12)
mfi0: 7656 (boot + 31s/0x0002/info) - Inserted: PD 18(e0xf5/s12) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b1898,0000000000000000
mfi0: 7657 (boot + 31s/0x0002/info) - Inserted: PD 19(e0xf5/s13)
mfi0: 7658 (boot + 31s/0x0002/info) - Inserted: PD 19(e0xf5/s13) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b1899,0000000000000000
mfi0: 7659 (boot + 31s/0x0002/info) - Inserted: PD 1a(e0xf5/s14)
mfi0: 7660 (boot + 31s/0x0002/info) - Inserted: PD 1a(e0xf5/s14) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b189a,0000000000000000
mfi0: 7661 (boot + 31s/0x0002/info) - Inserted: PD 1b(e0xf5/s15)
mfi0: 7662 (boot + 31s/0x0002/info) - Inserted: PD 1b(e0xf5/s15) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b189b,0000000000000000
mfi0: 7663 (boot + 31s/0x0002/info) - Inserted: PD 1c(e0xf5/s8)
mfi0: 7664 (boot + 31s/0x0002/info) - Inserted: PD 1c(e0xf5/s8) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b18a0,0000000000000000
mfi0: 7665 (boot + 31s/0x0002/info) - Inserted: PD 1d(e0xf5/s9)
mfi0: 7666 (boot + 31s/0x0002/info) - Inserted: PD 1d(e0xf5/s9) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b18a1,0000000000000000
mfi0: 7667 (boot + 31s/0x0002/info) - Inserted: PD 1e(e0xf5/s10)
mfi0: 7668 (boot + 31s/0x0002/info) - Inserted: PD 1e(e0xf5/s10) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b18a2,0000000000000000
mfi0: 7669 (boot + 31s/0x0002/info) - Inserted: PD 1f(e0xf5/s11)
mfi0: 7670 (boot + 31s/0x0002/info) - Inserted: PD 1f(e0xf5/s11) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b18a3,0000000000000000
mfi0: 7671 (boot + 32s/0x0002/info) - Unexpected sense: PD 17(e0xf5/s3) Path 500062b2009b1893, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7672 (boot + 32s/0x0002/info) - Unexpected sense: PD 16(e0xf5/s2) Path 500062b2009b1892, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7673 (boot + 32s/0x0002/info) - Unexpected sense: PD 15(e0xf5/s1) Path 500062b2009b1891, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7674 (boot + 32s/0x0002/info) - Unexpected sense: PD 14(e0xf5/s0) Path 500062b2009b1890, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7675 (boot + 32s/0x0002/info) - Unexpected sense: PD 13(e0xf5/s7) Path 500062b2009b188f, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7676 (boot + 32s/0x0002/info) - Unexpected sense: PD 12(e0xf5/s6) Path 500062b2009b188e, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7677 (boot + 32s/0x0002/info) - Unexpected sense: PD 11(e0xf5/s5) Path 500062b2009b188d, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7678 (boot + 32s/0x0002/info) - Unexpected sense: PD 10(e0xf5/s4) Path 500062b2009b188c, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7679 (boot + 32s/0x0002/info) - Unexpected sense: PD 1f(e0xf5/s11) Path 500062b2009b18a3, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7680 (boot + 32s/0x0002/info) - Unexpected sense: PD 1e(e0xf5/s10) Path 500062b2009b18a2, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7681 (boot + 32s/0x0002/info) - Unexpected sense: PD 1d(e0xf5/s9) Path 500062b2009b18a1, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7682 (boot + 32s/0x0002/info) - Unexpected sense: PD 1b(e0xf5/s15) Path 500062b2009b189b, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7683 (boot + 32s/0x0002/info) - Unexpected sense: PD 1a(e0xf5/s14) Path 500062b2009b189a, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7684 (boot + 32s/0x0002/info) - Unexpected sense: PD 19(e0xf5/s13) Path 500062b2009b1899, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7685 (boot + 32s/0x0002/info) - Unexpected sense: PD 18(e0xf5/s12) Path 500062b2009b1898, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7686 (510289519s/0x0020/info) - Time established as 03/03/16  3:05:19; (42 seconds since power on)
mfi0: 7687 (510289541s/0x0008/info) - Battery started charging
mfi0: 7688 (510289541s/0x0008/info) - Battery temperature is normal
mfi0: 7689 (510289561s/0x0020/info) - Host driver is loaded and operational
mfi0: 7690 (510289601s/0x0008/info) - Battery charge complete
mfi0: 7691 (boot + 4s/0x0020/info) - Firmware initialization started (PCI ID 0079/1000/9276/1000)
mfi0: 7692 (boot + 4s/0x0020/info) - Firmware version 2.130.403-3066
mfi0: 7693 (boot + 4s/0x0020/info) - Firmware initialization started (PCI ID 0079/1000/9276/1000)
mfi0: 7694 (boot + 4s/0x0020/info) - Firmware version 2.130.403-3066
mfi0: 7695 (boot + 9s/0x0008/info) - Battery Present
mfi0: 7696 (boot + 10s/0x0020/info) - Package version 12.15.0-0189
mfi0: 7697 (boot + 10s/0x0020/info) - Board Revision 20D
mfi0: 7698 (boot + 31s/0x0004/info) - Enclosure PD f5(c None/p1) communication restored
mfi0: 7699 (boot + 31s/0x0002/info) - Inserted: Encl PD f5
usbus0: 480Mbps High Speed USB v2.0
usbus1: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
mfid0 on mfi0
mfid0: 2861056MB (5859442688 sectors) RAID volume (no label) is optimal
mfid1 on mfi0
mfid1: 2861056MB (5859442688 sectors) RAID volume (no label) is optimal
mfid2 on mfi0
mfid2: 2861056MB (5859442688 sectors) RAID volume (no label) is optimal
mfid3 on mfi0
mfid3: 2861056MB (5859442688 sectors) RAID volume (no label) is optimal
mfid4 on mfi0
mfid4: 2861056MB (5859442688 sectors) RAID volume (no label) is optimal
mfid5 on mfi0
mfid5: 2861056MB (5859442688 sectors) RAID volume (no label) is optimal
mfid6 on mfi0
mfid6: 2861056MB (5859442688 sectors) RAID volume (no label) is optimal
mfid7 on mfi0
mfid7: 2861056MB (5859442688 sectors) RAID volume (no label) is optimal
mfid8 on mfi0
mfid8: 2861056MB (5859442688 sectors) RAID volume (no label) is optimal
mfid9 on mfi0
mfid9: 2861056MB (5859442688 sectors) RAID volume (no label) is optimal
mfid10 on mfi0
mfid10: 2861056MB (5859442688 sectors) RAID volume (no label) is optimal
mfid11 on mfi0
mfid11: 2861056MB (5859442688 sectors) RAID volume (no label) is optimal
mfid12 on mfi0
mfid12: 2861056MB (5859442688 sectors) RAID volume (no label) is optimal
mfid13 on mfi0
mfid13: 2861056MB (5859442688 sectors) RAID volume (no label) is optimal
mfid14 on mfi0
mfid14: 2861056MB (5859442688 sectors) RAID volume (no label) is optimal
mfid15 on mfi0
mfid15: 2861056MB (5859442688 sectors) RAID volume (no label) is optimal
mfi0: 7700 (boot + 31s/0x0002/info) - Inserted: PD f5(c None/p1) Info: enclPd=f5, scsiType=d, portMap=00, sasAddr=500062b2009b18bd,0000000000000000
mfi0: 7701 (boot + 31s/0x0002/info) - Inserted: PD 10(e0xf5/s4)
mfi0: 7702 (boot + 31s/0x0002/info) - Inserted: PD 10(e0xf5/s4) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b188c,0000000000000000
mfi0: 7703 (boot + 31s/0x0002/info) - Inserted: PD 11(e0xf5/s5)
mfi0: 7704 (boot + 31s/0x0002/info) - Inserted: PD 11(e0xf5/s5) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b188d,0000000000000000
mfi0: 7705 (boot + 31s/0x0002/info) - Inserted: PD 12(e0xf5/s6)
mfi0: 7706 (boot + 31s/0x0002/info) - Inserted: PD 12(e0xf5/s6) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b188e,0000000000000000
mfi0: 7707 (boot + 31s/0x0002/info) - Inserted: PD 13(e0xf5/s7)
mfi0: 7708 (boot + 31s/0x0002/info) - Inserted: PD 13(e0xf5/s7) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b188f,0000000000000000
mfi0: 7709 (boot + 31s/0x0002/info) - Inserted: PD 14(e0xf5/s0)
mfi0: 7710 (boot + 31s/0x0002/info) - Inserted: PD 14(e0xf5/s0) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b1890,0000000000000000
mfi0: 7711 (boot + 31s/0x0002/info) - Inserted: PD 15(e0xf5/s1)
mfi0: 7712 (boot + 31s/0x0002/info) - Inserted: PD 15(e0xf5/s1) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b1891,0000000000000000
mfi0: 7713 (boot + 31s/0x0002/info) - Inserted: PD 16(e0xf5/s2)
mfi0: 7714 (boot + 31s/0x0002/info) - Inserted: PD 16(e0xf5/s2) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b1892,0000000000000000
mfi0: 7715 (boot + 31s/0x0002/info) - Inserted: PD 17(e0xf5/s3)
mfi0: 7716 (boot + 31s/0x0002/info) - Inserted: PD 17(e0xf5/s3) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b1893,0000000000000000
mfi0: 7717 (boot + 31s/0x0002/info) - Inserted: PD 18(e0xf5/s12)
mfi0: 7718 (boot + 31s/0x0002/info) - Inserted: PD 18(e0xf5/s12) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b1898,0000000000000000
mfi0: 7719 (boot + 31s/0x0002/info) - Inserted: PD 19(e0xf5/s13)
mfi0: 7720 (boot + 31s/0x0002/info) - Inserted: PD 19(e0xf5/s13) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b1899,0000000000000000
mfi0: 7721 (boot + 31s/0x0002/info) - Inserted: PD 1a(e0xf5/s14)
mfi0: 7722 (boot + 31s/0x0002/info) - Inserted: PD 1a(e0xf5/s14) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b189a,0000000000000000
mfi0: 7723 (boot + 31s/0x0002/info) - Inserted: PD 1b(e0xf5/s15)
mfi0: 7724 (boot + 31s/0x0002/info) - Inserted: PD 1b(e0xf5/s15) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b189b,0000000000000000
mfi0: 7725 (boot + 31s/0x0002/info) - Inserted: PD 1c(e0xf5/s8)
mfi0: 7726 (boot + 31s/0x0002/info) - Inserted: PD 1c(e0xf5/s8) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b18a0,0000000000000000
mfi0: 7727 (boot + 31s/0x0002/info) - Inserted: PD 1d(e0xf5/s9)
mfi0: 7728 (boot + 31s/0x0002/info) - Inserted: PD 1d(e0xf5/s9) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b18a1,0000000000000000
mfi0: 7729 (boot + 31s/0x0002/info) - Inserted: PD 1e(e0xf5/s10)
mfi0: 7730 (boot + 31s/0x0002/info) - Inserted: PD 1e(e0xf5/s10) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b18a2,0000000000000000
mfi0: 7731 (boot + 31s/0x0002/info) - Inserted: PD 1f(e0xf5/s11)
mfi0: 7732 (boot + 31s/0x0002/info) - Inserted: PD 1f(e0xf5/s11) Info: enclPd=f5, scsiType=0, portMap=00, sasAddr=500062b2009b18a3,0000000000000000
mfi0: 7733 (boot + 32s/0x0002/info) - Unexpected sense: PD 17(e0xf5/s3) Path 500062b2009b1893, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7734 (boot + 32s/0x0002/info) - Unexpected sense: PD 16(e0xf5/s2) Path 500062b2009b1892, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7735 (boot + 32s/0x0002/info) - Unexpected sense: PD 15(e0xf5/s1) Path 500062b2009b1891, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7736 (boot + 32s/0x0002/info) - Unexpected sense: PD 14(e0xf5/s0) Path 500062b2009b1890, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7737 (boot + 32s/0x0002/info) - Unexpected sense: PD 13(e0xf5/s7) Path 500062b2009b188f, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7738 (boot + 32s/0x0002/info) - Unexpected sense: PD 12(e0xf5/s6) Path 500062b2009b188e, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7739 (boot + 32s/0x0002/info) - Unexpected sense: PD 11(e0xf5/s5) Path 500062b2009b188d, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7740 (boot + 32s/0x0002/info) - Unexpected sense: PD 10(e0xf5/s4) Path 500062b2009b188c, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7741 (boot + 32s/0x0002/info) - Unexpected sense: PD 1f(e0xf5/s11) Path 500062b2009b18a3, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7742 (boot + 32s/0x0002/info) - Unexpected sense: PD 1e(e0xf5/s10) Path 500062b2009b18a2, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7743 (boot + 32s/0x0002/info) - Unexpected sense: PD 1d(e0xf5/s9) Path 500062b2009b18a1, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7744 (boot + 32s/0x0002/info) - Unexpected sense: PD 1b(e0xf5/s15) Path 500062b2009b189b, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7745 (boot + 32s/0x0002/info) - Unexpected sense: PD 1a(e0xf5/s14) Path 500062b2009b189a, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7746 (boot + 32s/0x0002/info) - Unexpected sense: PD 19(e0xf5/s13) Path 500062b2009b1899, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7747 (boot + 32s/0x0002/info) - Unexpected sense: PD 18(e0xf5/s12) Path 500062b2009b1898, CDB: 1a 08 c8 00 ff 00, Sense: 5/24/00
mfi0: 7748 (510290015s/0x0020/info) - Time established as 03/03/16  3:13:35; (42 seconds since power on)
mfi0: 7749 (510290037s/0x0008/info) - Battery started charging
mfi0: 7750 (510290037s/0x0008/info) - Battery temperature is normal
mfi0: 7751 (510290056s/0x0020/info) - Host driver is loaded and operational
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <SAMSUNG MZ7WD120HAFV-00003 DXM87W3Q> ATA-9 SATA 3.x device
ada0: Serial Number S16KNYAF400781
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 114473MB (234441648 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad4
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <SAMSUNG MZ7WD120HAFV-00003 DXM87W3Q> ATA-9 SATA 3.x device
ada1: Serial Number S16KNYAF400785
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 114473MB (234441648 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad6
SMP: AP CPU #1 Launched!
cd0 at ahcich2 bus 0 scbus2 target 0 lun 0
cd0: <TEAC DV-28S-A 9.2A> Removable CD-ROM SCSI-0 device 
cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present - tray open
SMP: AP CPU #6 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #7 Launched!
Timecounter "TSC-low" frequency 1650048222 Hz quality 1000
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
ugen0.2: <vendor 0x8087> at usbus0
uhub2: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0
ugen1.2: <vendor 0x8087> at usbus1
uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus1
uhub2: 6 ports with 6 removable, self powered
uhub3: 6 ports with 6 removable, self powered
ugen1.3: <vendor 0x0409> at usbus1
uhub4: <vendor 0x0409 product 0x55aa, class 9/0, rev 1.10/2.00, addr 3> on usbus1
ugen0.3: <Winbond Electronics Corp> at usbus0
ums0: <Winbond Electronics Corp Hermon USB hidmouse Device, class 0/0, rev 1.10/0.01, addr 3> on usbus0
ums0: 3 buttons and [Z] coordinates ID=0
ukbd0: <Winbond Electronics Corp Hermon USB hidmouse Device, class 0/0, rev 1.10/0.01, addr 3> on usbus0
kbd2 at ukbd0
GEOM_MIRROR: Device mirror/root launched (1/2).
GEOM_MIRROR: Device root: rebuilding provider ada0.
uhub4: 3 ports with 2 removable, bus powered
Root mount waiting for: usbus1
ugen1.4: <LOGITECH> at usbus1
ukbd1: <LOGITECH USB Keyboard, class 0/0, rev 1.10/3.00, addr 4> on usbus1
kbd3 at ukbd1
uhid0: <LOGITECH USB Keyboard, class 0/0, rev 1.10/3.00, addr 4> on usbus1
Root mount waiting for: usbus1
ugen1.5: <Cypress Sem> at usbus1
ums1: <EndPoint1 Int> on usbus1
ums1: 5 buttons and [XYZ] coordinates ID=0
Trying to mount root from ufs:/dev/mirror/roots1a [rw]...
WARNING: / was not properly dismounted
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
mfi0: 7752 (510290097s/0x0008/info) - Battery charge complete
[michelle@colossus /usr/home/michelle]$ sudo ls -l /var/crash/
total 8
-rw-r--r--  1 root  wheel  5 Sep 27  2013 minfree
[michelle@colossus /usr/home/michelle]$ more /etc/rc.conf.local 
dumpdev="AUTO"
dumpdir="/var/crash"
crashinfo_enable="YES"

static_routes="sorbs"
route_sorbs="-net 113.52.8.0/24 10.10.0.254"

zfs_enable="YES"

vboxwebsrv_enable="YES"
vboxguest_enable="YES"
vboxservice_enable="YES"
vboxnet_enable="YES"


Feel free to ask for more info, I'll give what I can, as this is very and instantly reproducible...
Comment 1 Andriy Gapon freebsd_committer 2016-04-04 11:41:45 UTC
If you are not able to get a crash dump, then, at least, please get a screenshot of the panic screen.
Comment 2 Michelle Sullivan 2016-04-05 21:00:53 UTC
I hate to be beligerent, but exactly how would one do that... take a phone into the DC and take a photo of the console?

What would be nice would be to know how to get it to dump core... I see no reason why it shouldn't...  

in /etc/rc.conf:

# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"

Then in /etc/rc.conf.local:

dumpdev="AUTO"
dumpdir="/var/crash"
crashinfo_enable="YES"


And just to make sure...

# ls -l /var/crash/
total 4
-rw-r--r--  1 root  wheel  5 Sep 27  2013 minfree
[root@colossus ~]# more /var/crash/minfree 
2048
[root@colossus ~]# df -h
Filesystem             Size    Used   Avail Capacity  Mounted on
/dev/mirror/roots1a    107G     50G     48G    51%    /
devfs                  1.0k    1.0k      0B   100%    /dev
VirtualDisks            69G     31k     69G     0%    /VirtualDisks
sorbs                   33T     29T    3.8T    89%    /sorbs
[root@colossus ~]# more /etc/fstab 
# Device                        Mountpoint      FStype  Options Dump    Pass#
/dev/mirror/roots1a             /               ufs     rw      1       1
/dev/da0s1                      none            swap    sw      1       0

Even did this:

[root@colossus ~]# dumpon -v /dev/da0s1 
kernel dumps on /dev/da0s1 

(da0s1 = 28G(ish) USB drive - 16G RAM in the machine)

...

Oh well photograph the console seems to be the answer ... if it helps:

http://flashback.sorbs.net/packages/zpool-scrub-panic-console.jpg
Comment 3 Palle Girgensohn freebsd_committer 2016-04-05 21:43:18 UTC
You need enough swap to fit the kernel's core dump, or it will be lost at reboot and never be moved to /var/crash by dumpon (this happens after reboot)

Palle
Comment 4 Michelle Sullivan 2016-04-05 22:08:22 UTC
Understood...


From 'top':

Mem: 91M Active, 233M Inact, 2060M Wired, 281M Buf, 13G Free
ARC: 1536M Total, 307M MFU, 1209M MRU, 10M Anon, 5994K Header, 4495K Other
Swap: 29G Total, 29G Free

or if you prefer from dmesg:

real memory  = 17188257792 (16392 MB)
avail memory = 16491753472 (15727 MB)

da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
da0: <Kingston DataTraveler 3.0 1.00> Removable Direct Access SCSI-6 device 
da0: Serial Number 60A44C3FACC9FE118997019C
da0: 40.000MB/s transfers
da0: 29510MB (60437492 512 byte sectors: 255H 63S/T 3762C)
da0: quirks=0x2<NO_6_BYTE>

And just for completeness...

[root@colossus ~]# fdisk /dev/da0
******* Working on device /dev/da0 *******
parameters extracted from in-core disklabel are:
cylinders=3762 heads=255 sectors/track=63 (16065 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=3762 heads=255 sectors/track=63 (16065 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
    start 63, size 60436467 (29509 Meg), flag 80 (active)
	beg: cyl 0/ head 1/ sector 1;
	end: cyl 689/ head 254/ sector 63
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>
[root@colossus ~]# bsdlabel /dev/da0s1 
# /dev/da0s1:
8 partitions:
#          size     offset    fstype   [fsize bsize bps/cpg]
  a:   60436451         16      swap                    
  c:   60436467          0    unused        0     0     # "raw" part, don't edit


I'd suggest there is enough space... yet it doesn't produce dumps - infact when it panics it shows the uptime then locks solid (won't respond to the 3 finger salute) ... reset button or power button for +3 seconds is the only way to get a response...
Comment 5 Palle Girgensohn freebsd_committer 2016-04-05 23:09:34 UTC
That sucks, then maybe you're out of luck getting a core dump. :-(
Comment 6 Benjamin Kaduk freebsd_committer 2016-04-06 03:08:26 UTC
I don't think coredump to swap-on-USB is expected to work.   I am not familiar with the details, but am given to understand that the architecture of the USB support makes it basically not work in panic context.
Comment 7 Michelle Sullivan 2016-04-06 05:21:15 UTC
That would make sense, but can't swap to zfs either (options there are file - which won't work (documented) or zvol which is not documented but trying it just gave an inappropriate ioctl when trying to add the dump dev) possibly the other thing I could do it take the hotspare device out of the pool and turn it into a sump dev... but really trying not to screw with the existing config....


Did you take a look at the photo of the console?  Does it seem interesting?  It is 100% reproducable and the panic is instant.
Comment 8 Benjamin Kaduk freebsd_committer 2016-04-06 12:43:26 UTC
(In reply to Michelle Sullivan from comment #7)

> That would make sense, but can't swap to zfs either (options there are file - which won't work (documented) or zvol which is not documented but trying it just gave an inappropriate ioctl when trying to add the dump dev) possibly the other thing I could do it take the hotspare device out of the pool and turn it into a sump dev... but really trying not to screw with the existing config....

Of those, I would only expect taking the hotspare out of the pool and using the raw device for swap to provide a usable crash dump, but am not at the point of recommending that you do so.

> Did you take a look at the photo of the console?  Does it seem interesting?  It is 100% reproducable and the panic is instant.

I did look at it, and it seems interesting, but unfortunately outside my area of expertise.  Hopefully Andriy will take a look.
Comment 9 Andriy Gapon freebsd_committer 2016-04-06 17:10:10 UTC
(In reply to Michelle Sullivan)

Replying to multiple comments at once.

Yes, taking a picture of a monitor is a fine last measure.  A serial console, real or emulated via, e.g., IPMI or AMT, is better, of course.

The panic looks like a result of a well-known (among developers) and very hard to fix problem with interaction between GEOM and ZVOLs that arises from using ZFS pools on top of ZFS volumes.  I can not offer you any fix although I have got some work-in-progress on this issue.

I can suggest a work-around: create a FreeBSD VM using the volumes comprising the pool as the VM's disks and run the scrub within the VM.

Besides zpool scrub commands like zpool reopen, zpool split, zpool online [-e] can also trigger the same bug for pools on top of ZFS volumes.
Comment 10 Andriy Gapon freebsd_committer 2016-04-06 21:10:32 UTC
Created attachment 169058 [details]
possible patch for stable/9 zfs

In fact, this patch might help with the problem at hand (while not solving the larger problem, but that would be a different story).
Comment 11 Andriy Gapon freebsd_committer 2017-03-12 10:27:03 UTC
Created attachment 180741 [details]
console screenshot from the link

For posterity.

And quick gocr-ed text version of the stack trace:
KDB: stack backtrace:
 #% 0xfffErffE8%9257c6 at kdb_b4cktrace_0x66
 #1 %xfffEffE_808eb38e at panic+0xlce
 #2 0xEffEfEfE80c_2aa0 at trap_fatal_0x290
  #3 %xfffEfEEf80cd2e01 at trap_pfault+0x21l
  #_ 0xfEfFEfFE80cd3403 at trap+0x363
  45 _xfEfTEEff80cbc5e3 at calltrap_0x8
  #b _xffEfffFE81ad304c at vdev_geom_io_start+0x4c
  #7 0xfFEEEfff81aa45d7 at zio_vdev_io_start+0x1r7
   #8 0xfffEfffE81aa407e at zio_execute+_xbe
   _9 0xfffffffE81a85cb_ at vdev_probe+0x1__
   _10 0xftEErfff81a86129 at vdev_open'0x379
   #11 0xfffffEff81a86b81 at vdev_open_children+_x_1
   412 _xffffffE181a8fb95 at vdev_root_open'_x65
   #13 _xfEffffEE81a85e95 at vdev_open'0xe5
    _1Q 0xffE1fEff81a862be at vdev_reopen'_x2e
    _15 0xfff_fffE81a6293c at dsl_scan'0x2c
    g16 %xrf_frf__81ab7b61 at zfs_ioc_pool_scan'@xq1
    #11 0xrEfrrfr_81abaa6d at zfsdev_ioctl'0x58d
     Uptime; 1m59s
Comment 12 Andriy Gapon freebsd_committer 2017-03-12 10:37:20 UTC
Can't reproduce with FreeBSD head (configured with vfs.zfs.vol.recursive=1).
If the crash can be reproduced on any supported FreeBSD version, please re-open.
Comment 13 Michelle Sullivan 2017-03-14 19:17:16 UTC
It does, not going to bother with this anymore though now.  Got better things to do with my life than chase down bugs a year after notification on an OS that I am migrating away from because of these very 'obsolete version' closure of bugs.  I will however tell you it's how its setup that causes the issue, don't remember how I did it because I'm unable to get it setup in the same way again.