Bug 239566 - [zfs] panic: solaris assert: avl_find(&scn->scn_queue, sds, &where) .... dsl_scan.c line 1029
Summary: [zfs] panic: solaris assert: avl_find(&scn->scn_queue, sds, &where) .... dsl_...
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.0-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-fs mailing list
URL:
Keywords: panic
Depends on:
Blocks:
 
Reported: 2019-07-31 22:10 UTC by pascal.guitierrez
Modified: 2019-08-14 16:38 UTC (History)
1 user (show)

See Also:


Attachments
Kernel panic screenshot (110.68 KB, image/png)
2019-07-31 22:10 UTC, pascal.guitierrez
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description pascal.guitierrez 2019-07-31 22:10:05 UTC
Created attachment 206183 [details]
Kernel panic screenshot

Panic hit whilst having a number of (around 50) zfs recv's happening (this box is purely a zfs recv backup machine) and a scrub is running.

Machine is 12.0-RELEASEp8, has been running fine, however this is the first time we've had so many recvs whilst running a scrub.

System has panicked 3x in the last few hours.

Panic output is attached as a screenshot.

dmesg is below:

---<<BOOT>>---
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-RELEASE-p8 GENERIC amd64
FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1)
VT(vga): resolution 640x480
CPU: Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz (2400.05-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206c2  Family=0x6  Model=0x2c  Stepping=2
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x9ee3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  Structured Extended Features3=0x9c000000<IBPB,STIBP,L1DFL,SSBD>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 206158430208 (196608 MB)
avail memory = 200875819008 (191570 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <070918 APIC2257>
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s)
random: unblocking device.
Firmware Warning (ACPI): 32/64X length mismatch in FADT/Gpe0Block: 128/64 (20181003/tbfadt-748)
ioapic0: Changing APIC ID to 1
ioapic1: Changing APIC ID to 3
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
Launching APs: 3 2 7 5 4 1 6
Timecounter "TSC-low" frequency 1200024188 Hz quality 1000
random: entropy device external interface
kbd1 at kbdmux0
netmap: loaded module
[ath_hal] loaded
module_register_init: MOD_LOAD (vesa, 0xffffffff810f9a00, 0) error 19
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
aesni0: No AES or SHA support.
acpi0: <NEC > on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz quality 340
Event timer "HPET3" frequency 14318180 Hz quality 340
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
igb0: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0xef80-0xef9f mem 0xfafe0000-0xfaffffff,0xfafc0000-0xfafdffff,0xfaf9c000-0xfaf9ffff irq 28 at device 0.0 on pci1
igb0: attach_pre capping queues at 8
igb0: using 1024 tx descriptors and 1024 rx descriptors
igb0: msix_init qsets capped at 8
igb0: pxm cpus: 8 queue msgs: 9 admincnt: 1
igb0: using 8 rx queues 8 tx queues 
igb0: Using MSIX interrupts with 9 vectors
igb0: allocated for 8 tx_queues
igb0: allocated for 8 rx_queues
igb0: Ethernet address: 00:25:90:2c:58:7a
igb0: netmap queues/slots: TX 8/1024, RX 8/1024
igb1: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0xef40-0xef5f mem 0xfaf60000-0xfaf7ffff,0xfaf40000-0xfaf5ffff,0xfaf1c000-0xfaf1ffff irq 40 at device 0.1 on pci1
igb1: attach_pre capping queues at 8
igb1: using 1024 tx descriptors and 1024 rx descriptors
igb1: msix_init qsets capped at 8
igb1: pxm cpus: 8 queue msgs: 9 admincnt: 1
igb1: using 8 rx queues 8 tx queues 
igb1: Using MSIX interrupts with 9 vectors
igb1: allocated for 8 tx_queues
igb1: allocated for 8 rx_queues
igb1: Ethernet address: 00:25:90:2c:58:7b
igb1: netmap queues/slots: TX 8/1024, RX 8/1024
pcib2: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci2: <ACPI PCI bus> on pcib2
mps0: <Avago Technologies (LSI) SAS2008> port 0xd000-0xd0ff mem 0xfae3c000-0xfae3ffff,0xfae40000-0xfae7ffff irq 24 at device 0.0 on pci2
mps0: Firmware: 20.00.04.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
pcib3: <ACPI PCI-PCI bridge> at device 5.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 7.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pci0: <base peripheral, interrupt controller> at device 20.0 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.1 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.2 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.3 (no driver attached)
uhci0: <Intel 82801JI (ICH10) USB controller USB-D> port 0xce80-0xce9f irq 16 at device 26.0 on pci0
uhci0: LegSup = 0x2f00
usbus0 on uhci0
usbus0: 12Mbps Full Speed USB v1.0
uhci1: <Intel 82801JI (ICH10) USB controller USB-E> port 0xce20-0xce3f irq 21 at device 26.1 on pci0
uhci1: LegSup = 0x2f00
usbus1 on uhci1
usbus1: 12Mbps Full Speed USB v1.0
uhci2: <Intel 82801JI (ICH10) USB controller USB-F> port 0xcdc0-0xcddf irq 19 at device 26.2 on pci0
uhci2: LegSup = 0x2f00
usbus2 on uhci2
usbus2: 12Mbps Full Speed USB v1.0
ehci0: <Intel 82801JI (ICH10) USB 2.0 controller USB-B> mem 0xfadf6000-0xfadf63ff irq 18 at device 26.7 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci0
usbus3: 480Mbps High Speed USB v2.0
uhci3: <Intel 82801JI (ICH10) USB controller USB-A> port 0xcf00-0xcf1f irq 23 at device 29.0 on pci0
uhci3: LegSup = 0x2f00
usbus4 on uhci3
usbus4: 12Mbps Full Speed USB v1.0
uhci4: <Intel 82801JI (ICH10) USB controller USB-B> port 0xcec0-0xcedf irq 19 at device 29.1 on pci0
uhci4: LegSup = 0x2f00
usbus5 on uhci4
usbus5: 12Mbps Full Speed USB v1.0
uhci5: <Intel 82801JI (ICH10) USB controller USB-C> port 0xcea0-0xcebf irq 18 at device 29.2 on pci0
uhci5: LegSup = 0x2f00
usbus6 on uhci5
usbus6: 12Mbps Full Speed USB v1.0
ehci1: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> mem 0xfadf8000-0xfadf83ff irq 23 at device 29.7 on pci0
usbus7: EHCI version 1.0
usbus7 on ehci1
usbus7: 480Mbps High Speed USB v2.0
pcib5: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci5: <ACPI PCI bus> on pcib5
vgapci0: <VGA-compatible display> mem 0xf8000000-0xf8ffffff,0xf9ffc000-0xf9ffffff,0xfa000000-0xfa7fffff irq 16 at device 3.0 on pci5
vgapci0: Boot video device
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH10 SATA300 controller> port 0xcff0-0xcff7,0xcf8c-0xcf8f,0xcfe0-0xcfe7,0xcf88-0xcf8b,0xcfa0-0xcfaf,0xcf90-0xcf9f irq 19 at device 31.2 on pci0
ata2: <ATA channel> at channel 0 on atapci0
ata3: <ATA channel> at channel 1 on atapci0
atapci1: <Intel ICH10 SATA300 controller> port 0xcf80-0xcf87,0xcf7c-0xcf7f,0xcf28-0xcf2f,0xcf24-0xcf27,0xcf60-0xcf6f,0xcf30-0xcf3f irq 19 at device 31.5 on pci0
ata4: <ATA channel> at channel 0 on atapci1
ata5: <ATA channel> at channel 1 on atapci1
acpi_button0: <Power Button> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart2: <16550 or compatible> port 0x3e8-0x3ef irq 5 on acpi0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc87ff,0xc8800-0xc97ff pnpid ORM0000 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbdc0: non-PNP ISA device will be removed from GENERIC in FreeBSD 12.
est0: <Enhanced SpeedStep Frequency Control> on cpu0
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
ugen7.1: <Intel EHCI root HUB> at usbus7
ugen6.1: <Intel UHCI root HUB> at usbus6
ugen2.1: <Intel UHCI root HUB> at usbus2
ugen0.1: <Intel UHCI root HUB> at usbus0
ugen1.1: <Intel UHCI root HUB> at usbus1
ugen3.1: <Intel EHCI root HUB> at usbus3
ugen4.1: <Intel UHCI root HUB> at usbus4
ugen5.1: <Intel UHCI root HUB> at usbus5
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus7
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
uhub5: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
uhub6: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
uhub7: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
uhub2: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub7: 2 ports with 2 removable, self powered
uhub6: 2 ports with 2 removable, self powered
uhub0: 6 ports with 6 removable, self powered
uhub5: 6 ports with 6 removable, self powered
ugen2.2: <American Megatrends Inc. Virtual Keyboard and Mouse> at usbus2
ukbd0 on uhub1
ugen7.2: <Generic USB Storage> at usbus7
umass0 on uhub0
umass0: <Generic USB Storage, class 0/0, rev 2.00/2.72, addr 2> on usbus7
umass0:  SCSI over Bulk-Only; quirks = 0xc100
umass0:5:0: Attached to scbus5
ukbd0: <Keyboard Interface> on usbus2
kbd2 at ukbd0
ugen7.3: <Generic USB Storage> at usbus7
umass1 on uhub0
umass1: <Generic USB Storage, class 0/0, rev 2.00/2.72, addr 3> on usbus7
umass1:  SCSI over Bulk-Only; quirks = 0xc100
umass1:6:1: Attached to scbus6
ses0 at mps0 bus 0 scbus0 target 21 lun 0
ses0: <LSI CORP SAS2X28 0717> Fixed Enclosure Services SPC-3 SCSI device
ses0: Serial Number 
ses0: 600.000MB/s transfers
ses0: Command Queueing enabled
ses0: SCSI-3 ENC Device
ses1 at mps0 bus 0 scbus0 target 41 lun 0
ses1: <LSI CORP SAS2X36 0717> Fixed Enclosure Services SPC-3 SCSI device
ses1: Serial Number 
ses1: 600.000MB/s transfers
ses1: Command Queueing enabled
ses1: SCSI-3 ENC Device
da1 at mps0 bus 0 scbus0 target 9 lun 0
da1: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number 7SGA0SVC
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 7630885MB (1953506646 4096 byte sectors)
da5 at mps0 bus 0 scbus0 target 13 lun 0
da5: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device
da5: Serial Number 7SGDWJYC
da5: 600.000MB/s transfers
da5: Command Queueing enabled
da5: 7630885MB (1953506646 4096 byte sectors)
da8 at mps0 bus 0 scbus0 target 16 lun 0
da8: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device
da8: Serial Number 7SGDP3MC
da8: 600.000MB/s transfers
da8: Command Queueing enabled
da8: 7630885MB (1953506646 4096 byte sectors)
da11 at mps0 bus 0 scbus0 target 147 lun 0
da11: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device
da11: Serial Number 7SGDR52C
da11: 600.000MB/s transfers
da11: Command Queueing enabled
da11: 7630885MB (1953506646 4096 byte sectors)
da2 at mps0 bus 0 scbus0 target 10 lun 0
da2: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number 7SGBVZZC
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 7630885MB (1953506646 4096 byte sectors)
da13 at mps0 bus 0 scbus0 target 149 lun 0
da13: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device
da13: Serial Number 7SGJTBBC
da13: 600.000MB/s transfers
da13: Command Queueing enabled
da13: 7630885MB (1953506646 4096 byte sectors)
da18 at mps0 bus 0 scbus0 target 154 lun 0
da18: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device
da18: Serial Number 7SGBBYBC
da18: 600.000MB/s transfers
da18: Command Queueing enabled
da18: 7630885MB (1953506646 4096 byte sectors)
da19 at mps0 bus 0 scbus0 target 155 lun 0
da19: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device
da19: Serial Number 7SGDHK9C
da19: 600.000MB/s transfers
da19: Command Queueing enabled
da19: 7630885MB (1953506646 4096 byte sectors)
da12 at mps0 bus 0 scbus0 target 148 lun 0
da12: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device
da12: Serial Number 7SGJRZ9C
da12: 600.000MB/s transfers
da12: Command Queueing enabled
da12: 7630885MB (1953506646 4096 byte sectors)
da23 at mps0 bus 0 scbus0 target 159 lun 0
da23: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device
da23: Serial Number 7SGJ46ZC
da23: 600.000MB/s transfers
da23: Command Queueing enabled
da23: 7630885MB (1953506646 4096 byte sectors)
da21 at mps0 bus 0 scbus0 target 157 lun 0
da21: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device
da21: Serial Number 7SGDMJXC
da21: 600.000MB/s transfers
da21: Command Queueing enabled
da21: 7630885MB (1953506646 4096 byte sectors)
da14 at mps0 bus 0 scbus0 target 150 lun 0
da14: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device
da14: Serial Number 7SH4YJ5L
da14: 600.000MB/s transfers
da14: Command Queueing enabled
da14: 7630885MB (1953506646 4096 byte sectors)
da7 at mps0 bus 0 scbus0 target 15 lun 0
da7: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device
da7: Serial Number 7SH4YHKL
da7: 600.000MB/s transfers
da7: Command Queueing enabled
da7: 7630885MB (1953506646 4096 byte sectors)
da10 at mps0 bus 0 scbus0 target 18 lun 0
da10: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device
da10: Serial Number 7SGDEZAC
da10: 600.000MB/s transfers
da10: Command Queueing enabled
da10: 7630885MB (1953506646 4096 byte sectors)
da17 at mps0 bus 0 scbus0 target 153 lun 0
da17: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device
da17: Serial Number 7SH4YJHL
da17: 600.000MB/s transfers
da17: Command Queueing enabled
da17: 7630885MB (1953506646 4096 byte sectors)
da16 at mps0 bus 0 scbus0 target 152 lun 0
da16: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device
da16: Serial Number 7SGJ7VDC
da16: 600.000MB/s transfers
da16: Command Queueing enabled
da16: 7630885MB (1953506646 4096 byte sectors)
da9 at mps0 bus 0 scbus0 target 17 lun 0
da9: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device
da9: Serial Number 7SGE899C
da9: 600.000MB/s transfers
da9: Command Queueing enabled
da9: 7630885MB (1953506646 4096 byte sectors)
da6 at mps0 bus 0 scbus0 target 14 lun 0
da6: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device
da6: Serial Number 7SGDLLEC
da6: 600.000MB/s transfers
da6: Command Queueing enabled
da6: 7630885MB (1953506646 4096 byte sectors)
da3 at mps0 bus 0 scbus0 target 11 lun 0
da3: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number 7SGBL59C
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 7630885MB (1953506646 4096 byte sectors)
da20 at mps0 bus 0 scbus0 target 156 lun 0
da20: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device
da20: Serial Number 7SGJG60C
da20: 600.000MB/s transfers
da20: Command Queueing enabled
da20: 7630885MB (1953506646 4096 byte sectors)
da15 at mps0 bus 0 scbus0 target 151 lun 0
da15: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device
da15: Serial Number 7SGJEM1C
da15: 600.000MB/s transfers
da15: Command Queueing enabled
da15: 7630885MB (1953506646 4096 byte sectors)
da4 at mps0 bus 0 scbus0 target 12 lun 0
da4: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device
da4: Serial Number 7SGUV03C
da4: 600.000MB/s transfers
da4: Command Queueing enabled
da4: 7630885MB (1953506646 4096 byte sectors)
da22 at mps0 bus 0 scbus0 target 158 lun 0
da22: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device
da22: Serial Number 7SH4YN5L
da22: 600.000MB/s transfers
da22: Command Queueing enabled
da22: 7630885MB (1953506646 4096 byte sectors)
da24 at umass-sim0 bus 0 scbus5 target 0 lun 0
da24: <Generic STORAGE DEVICE 0272> Removable Direct Access SCSI device
da24: Serial Number 000000000272
da24: 40.000MB/s transfers
da24: 3968MB (8126464 512 byte sectors)
da24: quirks=0x3<NO_SYNC_CACHE,NO_6_BYTE>
da0 at mps0 bus 0 scbus0 target 8 lun 0
da0: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number 7SGK036C
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 7630885MB (1953506646 4096 byte sectors)
da25 at umass-sim1 bus 1 scbus6 target 0 lun 0
da25: <Generic STORAGE DEVICE 0272> Removable Direct Access SCSI device
da25: Serial Number 000000000272
da25: 40.000MB/s transfers
da25: 3968MB (8126464 512 byte sectors)
da25: quirks=0x3<NO_SYNC_CACHE,NO_6_BYTE>
ses1: da23,pass25: Element descriptor: 'Slot 01'
ses1: da23,pass25: SAS Device Slot Element: 1 Phys at Slot 0
ses1:  phy 0: SAS device type 1 id 0
ses1:  phy 0: protocols: Initiator( None ) Target( SSP )
ses1:  phy 0: parent 500304800107877f addr 5000cca2521d5731
ses1: da22,pass24: Element descriptor: 'Slot 02'
ses1: da22,pass24: SAS Device Slot Element: 1 Phys at Slot 1
ses1:  phy 0: SAS device type 1 id 0
ses1:  phy 0: protocols: Initiator( None ) Target( SSP )
ses1:  phy 0: parent 500304800107877f addr 5000cca25241612d
ses1: da21,pass23: Element descriptor: 'Slot 03'
ses1: da21,pass23: SAS Device Slot Element: 1 Phys at Slot 2
ses1:  phy 0: SAS device type 1 id 0
ses1:  phy 0: protocols: Initiator( None ) Target( SSP )
ses1:  phy 0: parent 500304800107877f addr 5000cca25216f74d
ses1: da20,pass22: Element descriptor: 'Slot 04'
ses1: da20,pass22: SAS Device Slot Element: 1 Phys at Slot 3
ses1:  phy 0: SAS device type 1 id 0
ses1:  phy 0: protocols: Initiator( None ) Target( SSP )
ses1:  phy 0: parent 500304800107877f addr 5000cca2521dece1
ses1: da19,pass21: Element descriptor: 'Slot 05'
ses1: da19,pass21: SAS Device Slot Element: 1 Phys at Slot 4
ses1:  phy 0: SAS device type 1 id 0
ses1:  phy 0: protocols: Initiator( None ) Target( SSP )
ses1:  phy 0: parent 500304800107877f addr 5000cca25216bb6d
ses1: da18,pass20: Element descriptor: 'Slot 06'
ses1: da18,pass20: SAS Device Slot Element: 1 Phys at Slot 5
ses1:  phy 0: SAS device type 1 id 0
ses1:  phy 0: protocols: Initiator( None ) Target( SSP )
ses1:  phy 0: parent 500304800107877f addr 5000cca25214b3b9
ses1: da17,pass19: Element descriptor: 'Slot 07'
ses1: da17,pass19: SAS Device Slot Element: 1 Phys at Slot 6
ses1:  phy 0: SAS device type 1 id 0
ses1:  phy 0: protocols: Initiator( None ) Target( SSP )
ses1:  phy 0: parent 500304800107877f addr 5000cca252415f65
ses1: da16,pass18: Element descriptor: 'Slot 08'
ses1: da16,pass18: SAS Device Slot Element: 1 Phys at Slot 7
ses1:  phy 0: SAS device type 1 id 0
ses1:  phy 0: protocols: Initiator( None ) Target( SSP )
ses1:  phy 0: parent 500304800107877f addr 5000cca2521d8da5
ses1: da15,pass17: Element descriptor: 'Slot 09'
ses1: da15,pass17: SAS Device Slot Element: 1 Phys at Slot 8
ses1:  phy 0: SAS device type 1 id 0
ses1:  phy 0: protocols: Initiator( None ) Target( SSP )
ses1:  phy 0: parent 500304800107877f addr 5000cca2521de42d
ses1: da14,pass16: Element descriptor: 'Slot 10'
ses1: da14,pass16: SAS Device Slot Element: 1 Phys at Slot 9
ses1:  phy 0: SAS device type 1 id 0
ses1:  phy 0: protocols: Initiator( None ) Target( SSP )
ses1:  phy 0: parent 500304800107877f addr 5000cca252415f3d
ses1: da13,pass15: Element descriptor: 'Slot 11'
ses1: da13,pass15: SAS Device Slot Element: 1 Phys at Slot 10
ses1:  phy 0: SAS device type 1 id 0
ses1:  phy 0: protocols: Initiator( None ) Target( SSP )
ses1:  phy 0: parent 500304800107877f addr 5000cca2521e85a1
ses1: da5,pass5: Element descriptor: 'Slot 12'
ses1: da5,pass5: SAS Device Slot Element: 1 Phys at Slot 11
ses1:  phy 0: SAS device type 1 id 0
ses1:  phy 0: protocols: Initiator( None ) Target( SSP )
ses1:  phy 0: parent 500304800107877f addr 5000cca252176f71
ses1: da12,pass14: Element descriptor: 'Slot 13'
ses1: da12,pass14: SAS Device Slot Element: 1 Phys at Slot 12
ses1:  phy 0: SAS device type 1 id 0
ses1:  phy 0: protocols: Initiator( None ) Target( SSP )
ses1:  phy 0: parent 500304800107877f addr 5000cca2521e70c5
ses1: da11,pass13: Element descriptor: 'Slot 14'
ses1: da11,pass13: SAS Device Slot Element: 1 Phys at Slot 13
ses1:  phy 0: SAS device type 1 id 0
ses1:  phy 0: protocols: Initiator( None ) Target( SSP )
ses1:  phy 0: parent 500304800107877f addr 5000cca252171e9d
ses1: da10,pass10: Element descriptor: 'Slot 15'
ses1: da10,pass10: SAS Device Slot Element: 1 Phys at Slot 14
ses1:  phy 0: SAS device type 1 id 0
ses1:  phy 0: protocols: Initiator( None ) Target( SSP )
ses1:  phy 0: parent 500304800107877f addr 5000cca25216a3b5
ses1: da6,pass6: Element descriptor: 'Slot 16'
ses1: da6,pass6: SAS Device Slot Element: 1 Phys at Slot 15
ses1:  phy 0: SAS device type 1 id 0
ses1:  phy 0: protocols: Initiator( None ) Target( SSP )
ses1:  phy 0: parent 500304800107877f addr 5000cca25216e905
ses1: da3,pass3: Element descriptor: 'Slot 17'
ses1: da3,pass3: SAS Device Slot Element: 1 Phys at Slot 16
ses1:  phy 0: SAS device type 1 id 0
ses1:  phy 0: protocols: Initiator( None ) Target( SSP )
ses1:  phy 0: parent 500304800107877f addr 5000cca25215112d
ses1: da2,pass2: Element descriptor: 'Slot 18'
ses1: da2,pass2: SAS Device Slot Element: 1 Phys at Slot 17
ses1:  phy 0: SAS device type 1 id 0
ses1:  phy 0: protocols: Initiator( None ) Target( SSP )
ses1:  phy 0: parent 500304800107877f addr 5000cca2521595bd
ses1: da9,pass9: Element descriptor: 'Slot 19'
ses1: da9,pass9: SAS Device Slot Element: 1 Phys at Slot 18
ses1:  phy 0: SAS device type 1 id 0
ses1:  phy 0: protocols: Initiator( None ) Target( SSP )
ses1:  phy 0: parent 500304800107877f addr 5000cca252181fed
ses1: da8,pass8: Element descriptor: 'Slot 20'
ses1: da8,pass8: SAS Device Slot Element: 1 Phys at Slot 19
ses1:  phy 0: SAS device type 1 id 0
ses1:  phy 0: protocols: Initiator( None ) Target( SSP )
ses1:  phy 0: parent 500304800107877f addr 5000cca252170ee5
ses1: da7,pass7: Element descriptor: 'Slot 21'
ses1: da7,pass7: SAS Device Slot Element: 1 Phys at Slot 20
ses1:  phy 0: SAS device type 1 id 0
ses1:  phy 0: protocols: Initiator( None ) Target( SSP )
ses1:  phy 0: parent 500304800107877f addr 5000cca252415ef1
ses1: da4,pass4: Element descriptor: 'Slot 22'
ses1: da4,pass4: SAS Device Slot Element: 1 Phys at Slot 21
ses1:  phy 0: SAS device type 1 id 0
ses1:  phy 0: protocols: Initiator( None ) Target( SSP )
ses1:  phy 0: parent 500304800107877f addr 5000cca2522efb91
ses1: da1,pass1: Element descriptor: 'Slot 23'
ses1: da1,pass1: SAS Device Slot Element: 1 Phys at Slot 22
ses1:  phy 0: SAS device type 1 id 0
ses1:  phy 0: protocols: Initiator( None ) Target( SSP )
ses1:  phy 0: parent 500304800107877f addr 5000cca252123a65
ses1: da0,pass0: Element descriptor: 'Slot 24'
ses1: da0,pass0: SAS Device Slot Element: 1 Phys at Slot 23
ses1:  phy 0: SAS device type 1 id 0
ses1:  phy 0: protocols: Initiator( None ) Target( SSP )
ses1:  phy 0: parent 500304800107877f addr 5000cca2521eeac9
Trying to mount root from zfs:zroot []...
lo0: link state changed to UP
igb0: link state changed to UP
ioat0: <TBG IOAT Ch0> mem 0xfadf0000-0xfadf3fff irq 43 at device 22.0 on pci0
ioat0: Capabilities: 77<Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break>
ioat1: <TBG IOAT Ch1> mem 0xfadec000-0xfadeffff irq 44 at device 22.1 on pci0
ioat1: Capabilities: 77<Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break>
ioat2: <TBG IOAT Ch2> mem 0xfade8000-0xfadebfff irq 45 at device 22.2 on pci0
ioat2: Capabilities: 77<Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break>
ioat3: <TBG IOAT Ch3> mem 0xfade4000-0xfade7fff irq 46 at device 22.3 on pci0
ioat3: Capabilities: 77<Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break>
ioat4: <TBG IOAT Ch4> mem 0xfade0000-0xfade3fff irq 43 at device 22.4 on pci0
ioat4: Capabilities: 77<Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break>
ioat5: <TBG IOAT Ch5> mem 0xfaddc000-0xfaddffff irq 44 at device 22.5 on pci0
ioat5: Capabilities: 77<Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break>
ioat6: <TBG IOAT Ch6> mem 0xfadd8000-0xfaddbfff irq 45 at device 22.6 on pci0
ioat6: Capabilities: 77<Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break>
ioat7: <TBG IOAT Ch7> mem 0xfadd4000-0xfadd7fff irq 46 at device 22.7 on pci0
ioat7: Capabilities: 77<Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break>
ums0 on uhub1
ums0: <Mouse Interface> on usbus2
Security policy loaded: MAC/ntpd (mac_ntpd)
Comment 1 Andriy Gapon freebsd_committer 2019-08-14 14:54:51 UTC
Some debugging information for the crash:
Unread portion of the kernel message buffer:
panic: solaris assert: avl_find(&scn->scn_queue, sds, &where) == ((void *)0) (0xfffff80f108a1ac0 == 0x0), file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c, line: 1029
cpuid = 6
time = 1565487588
KDB: stack backtrace:
#0 0xffffffff80be7967 at kdb_backtrace+0x67
#1 0xffffffff80b9b543 at vpanic+0x1a3
#2 0xffffffff80b9b393 at panic+0x43
#3 0xffffffff829fe22c at assfail3+0x2c
#4 0xffffffff826cafbb at dsl_scan_ds_clone_swapped+0x1db
#5 0xffffffff826ba4e5 at dsl_dataset_clone_swap_sync_impl+0x8c5
#6 0xffffffff826a0de0 at dmu_recv_end_sync+0x310
#7 0xffffffff826c369e at dsl_sync_task_sync+0xae
#8 0xffffffff826c204b at dsl_pool_sync+0x3cb
#9 0xffffffff826eb635 at spa_sync+0xab5
#10 0xffffffff826f8f98 at txg_sync_thread+0x208
#11 0xffffffff80b5bf13 at fork_exit+0x83
#12 0xffffffff810506ae at fork_trampoline+0xe
Uptime: 14h3m53s
netdump: overwriting mbuf zone pointers
netdump in progress. searching for server...
netdumping to 10.10.5.2 (ac:1f:6b:48:3a:ac)
Dumping 21259 out of 196556 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

__curthread () at ./machine/pcpu.h:234
234             struct thread *td;
(kgdb) bt
#0  __curthread () at ./machine/pcpu.h:234
#1  doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:366
#2  0xffffffff80b9b12b in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:446
#3  0xffffffff80b9b5a3 in vpanic (fmt=<optimized out>, ap=0xfffffe0171b705c0) at /usr/src/sys/kern/kern_shutdown.c:872
#4  0xffffffff80b9b393 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:799
#5  0xffffffff829fe22c in assfail3 (a=<unavailable>, lv=<unavailable>, op=<unavailable>, rv=<unavailable>, f=<unavailable>, l=<optimized out>) at /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:91
#6  0xffffffff826cafbb in scan_ds_queue_insert (scn=<optimized out>, dsobj=85951, txg=1) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c:1029
#7  dsl_scan_ds_clone_swapped (ds1=<optimized out>, ds2=<optimized out>, tx=0xfffff8013b108b00) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c:2036
#8  0xffffffff826ba4e5 in dsl_dataset_clone_swap_sync_impl (clone=0xfffff8186d34f800, origin_head=0xfffff80bf0b90800, tx=<optimized out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c:3663
#9  0xffffffff826a0de0 in dmu_recv_end_sync (arg=0xfffffe0171b93510, tx=0xfffff8013b108b00) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c:3302
#10 0xffffffff826c369e in dsl_sync_task_sync (dst=0xfffffe0171b932e8, tx=0xfffff8013b108b00) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_synctask.c:234
#11 0xffffffff826c204b in dsl_pool_sync (dp=<optimized out>, txg=<optimized out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c:819
#12 0xffffffff826eb635 in spa_sync (spa=<optimized out>, txg=<optimized out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:8088
#13 0xffffffff826f8f98 in txg_sync_thread (arg=0xfffff80106632800) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:543
#14 0xffffffff80b5bf13 in fork_exit (callout=0xffffffff826f8d90 <txg_sync_thread>, arg=0xfffff80106632800, frame=0xfffffe0171b70c00) at /usr/src/sys/kern/kern_fork.c:1057
#15 <signal handler called>
(kgdb) fr 8
#8  0xffffffff826ba4e5 in dsl_dataset_clone_swap_sync_impl (clone=0xfffff8186d34f800, origin_head=0xfffff80bf0b90800, tx=<optimized out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c:3663
(kgdb) p clone->ds_object
$1 = 85951
(kgdb) p origin_head->ds_object
$2 = 85918
Comment 2 Andriy Gapon freebsd_committer 2019-08-14 14:57:31 UTC
The root cause is that both 85951 and 85918 were already in scn_queue and the code in dsl_scan_ds_clone_swapped fails to properly handle that case.
It has other logic problems as well.