Created attachment 206183 [details] Kernel panic screenshot Panic hit whilst having a number of (around 50) zfs recv's happening (this box is purely a zfs recv backup machine) and a scrub is running. Machine is 12.0-RELEASEp8, has been running fine, however this is the first time we've had so many recvs whilst running a scrub. System has panicked 3x in the last few hours. Panic output is attached as a screenshot. dmesg is below: ---<<BOOT>>--- Copyright (c) 1992-2018 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-RELEASE-p8 GENERIC amd64 FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1) VT(vga): resolution 640x480 CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz (2400.05-MHz K8-class CPU) Origin="GenuineIntel" Id=0x206c2 Family=0x6 Model=0x2c Stepping=2 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x9ee3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> AMD Features2=0x1<LAHF> Structured Extended Features3=0x9c000000<IBPB,STIBP,L1DFL,SSBD> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 206158430208 (196608 MB) avail memory = 200875819008 (191570 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <070918 APIC2257> FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 2 package(s) x 4 core(s) random: unblocking device. Firmware Warning (ACPI): 32/64X length mismatch in FADT/Gpe0Block: 128/64 (20181003/tbfadt-748) ioapic0: Changing APIC ID to 1 ioapic1: Changing APIC ID to 3 ioapic0 <Version 2.0> irqs 0-23 on motherboard ioapic1 <Version 2.0> irqs 24-47 on motherboard Launching APs: 3 2 7 5 4 1 6 Timecounter "TSC-low" frequency 1200024188 Hz quality 1000 random: entropy device external interface kbd1 at kbdmux0 netmap: loaded module [ath_hal] loaded module_register_init: MOD_LOAD (vesa, 0xffffffff810f9a00, 0) error 19 nexus0 vtvga0: <VT VGA driver> on motherboard cryptosoft0: <software crypto> on motherboard aesni0: No AES or SHA support. acpi0: <NEC > on motherboard acpi0: Power Button (fixed) cpu0: <ACPI CPU> on acpi0 attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 340 Event timer "HPET2" frequency 14318180 Hz quality 340 Event timer "HPET3" frequency 14318180 Hz quality 340 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0 pci1: <ACPI PCI bus> on pcib1 igb0: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0xef80-0xef9f mem 0xfafe0000-0xfaffffff,0xfafc0000-0xfafdffff,0xfaf9c000-0xfaf9ffff irq 28 at device 0.0 on pci1 igb0: attach_pre capping queues at 8 igb0: using 1024 tx descriptors and 1024 rx descriptors igb0: msix_init qsets capped at 8 igb0: pxm cpus: 8 queue msgs: 9 admincnt: 1 igb0: using 8 rx queues 8 tx queues igb0: Using MSIX interrupts with 9 vectors igb0: allocated for 8 tx_queues igb0: allocated for 8 rx_queues igb0: Ethernet address: 00:25:90:2c:58:7a igb0: netmap queues/slots: TX 8/1024, RX 8/1024 igb1: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0xef40-0xef5f mem 0xfaf60000-0xfaf7ffff,0xfaf40000-0xfaf5ffff,0xfaf1c000-0xfaf1ffff irq 40 at device 0.1 on pci1 igb1: attach_pre capping queues at 8 igb1: using 1024 tx descriptors and 1024 rx descriptors igb1: msix_init qsets capped at 8 igb1: pxm cpus: 8 queue msgs: 9 admincnt: 1 igb1: using 8 rx queues 8 tx queues igb1: Using MSIX interrupts with 9 vectors igb1: allocated for 8 tx_queues igb1: allocated for 8 rx_queues igb1: Ethernet address: 00:25:90:2c:58:7b igb1: netmap queues/slots: TX 8/1024, RX 8/1024 pcib2: <ACPI PCI-PCI bridge> at device 3.0 on pci0 pci2: <ACPI PCI bus> on pcib2 mps0: <Avago Technologies (LSI) SAS2008> port 0xd000-0xd0ff mem 0xfae3c000-0xfae3ffff,0xfae40000-0xfae7ffff irq 24 at device 0.0 on pci2 mps0: Firmware: 20.00.04.00, Driver: 21.02.00.00-fbsd mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc> pcib3: <ACPI PCI-PCI bridge> at device 5.0 on pci0 pci3: <ACPI PCI bus> on pcib3 pcib4: <ACPI PCI-PCI bridge> at device 7.0 on pci0 pci4: <ACPI PCI bus> on pcib4 pci0: <base peripheral, interrupt controller> at device 20.0 (no driver attached) pci0: <base peripheral, interrupt controller> at device 20.1 (no driver attached) pci0: <base peripheral, interrupt controller> at device 20.2 (no driver attached) pci0: <base peripheral, interrupt controller> at device 20.3 (no driver attached) uhci0: <Intel 82801JI (ICH10) USB controller USB-D> port 0xce80-0xce9f irq 16 at device 26.0 on pci0 uhci0: LegSup = 0x2f00 usbus0 on uhci0 usbus0: 12Mbps Full Speed USB v1.0 uhci1: <Intel 82801JI (ICH10) USB controller USB-E> port 0xce20-0xce3f irq 21 at device 26.1 on pci0 uhci1: LegSup = 0x2f00 usbus1 on uhci1 usbus1: 12Mbps Full Speed USB v1.0 uhci2: <Intel 82801JI (ICH10) USB controller USB-F> port 0xcdc0-0xcddf irq 19 at device 26.2 on pci0 uhci2: LegSup = 0x2f00 usbus2 on uhci2 usbus2: 12Mbps Full Speed USB v1.0 ehci0: <Intel 82801JI (ICH10) USB 2.0 controller USB-B> mem 0xfadf6000-0xfadf63ff irq 18 at device 26.7 on pci0 usbus3: EHCI version 1.0 usbus3 on ehci0 usbus3: 480Mbps High Speed USB v2.0 uhci3: <Intel 82801JI (ICH10) USB controller USB-A> port 0xcf00-0xcf1f irq 23 at device 29.0 on pci0 uhci3: LegSup = 0x2f00 usbus4 on uhci3 usbus4: 12Mbps Full Speed USB v1.0 uhci4: <Intel 82801JI (ICH10) USB controller USB-B> port 0xcec0-0xcedf irq 19 at device 29.1 on pci0 uhci4: LegSup = 0x2f00 usbus5 on uhci4 usbus5: 12Mbps Full Speed USB v1.0 uhci5: <Intel 82801JI (ICH10) USB controller USB-C> port 0xcea0-0xcebf irq 18 at device 29.2 on pci0 uhci5: LegSup = 0x2f00 usbus6 on uhci5 usbus6: 12Mbps Full Speed USB v1.0 ehci1: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> mem 0xfadf8000-0xfadf83ff irq 23 at device 29.7 on pci0 usbus7: EHCI version 1.0 usbus7 on ehci1 usbus7: 480Mbps High Speed USB v2.0 pcib5: <ACPI PCI-PCI bridge> at device 30.0 on pci0 pci5: <ACPI PCI bus> on pcib5 vgapci0: <VGA-compatible display> mem 0xf8000000-0xf8ffffff,0xf9ffc000-0xf9ffffff,0xfa000000-0xfa7fffff irq 16 at device 3.0 on pci5 vgapci0: Boot video device isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 atapci0: <Intel ICH10 SATA300 controller> port 0xcff0-0xcff7,0xcf8c-0xcf8f,0xcfe0-0xcfe7,0xcf88-0xcf8b,0xcfa0-0xcfaf,0xcf90-0xcf9f irq 19 at device 31.2 on pci0 ata2: <ATA channel> at channel 0 on atapci0 ata3: <ATA channel> at channel 1 on atapci0 atapci1: <Intel ICH10 SATA300 controller> port 0xcf80-0xcf87,0xcf7c-0xcf7f,0xcf28-0xcf2f,0xcf24-0xcf27,0xcf60-0xcf6f,0xcf30-0xcf3f irq 19 at device 31.5 on pci0 ata4: <ATA channel> at channel 0 on atapci1 ata5: <ATA channel> at channel 1 on atapci1 acpi_button0: <Power Button> on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart2: <16550 or compatible> port 0x3e8-0x3ef irq 5 on acpi0 orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc87ff,0xc8800-0xc97ff pnpid ORM0000 on isa0 atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbdc0: non-PNP ISA device will be removed from GENERIC in FreeBSD 12. est0: <Enhanced SpeedStep Frequency Control> on cpu0 ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 1.000 msec ugen7.1: <Intel EHCI root HUB> at usbus7 ugen6.1: <Intel UHCI root HUB> at usbus6 ugen2.1: <Intel UHCI root HUB> at usbus2 ugen0.1: <Intel UHCI root HUB> at usbus0 ugen1.1: <Intel UHCI root HUB> at usbus1 ugen3.1: <Intel EHCI root HUB> at usbus3 ugen4.1: <Intel UHCI root HUB> at usbus4 ugen5.1: <Intel UHCI root HUB> at usbus5 uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus7 uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2 uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1 uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5 uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4 uhub5: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3 uhub6: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 uhub7: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6 uhub2: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub7: 2 ports with 2 removable, self powered uhub6: 2 ports with 2 removable, self powered uhub0: 6 ports with 6 removable, self powered uhub5: 6 ports with 6 removable, self powered ugen2.2: <American Megatrends Inc. Virtual Keyboard and Mouse> at usbus2 ukbd0 on uhub1 ugen7.2: <Generic USB Storage> at usbus7 umass0 on uhub0 umass0: <Generic USB Storage, class 0/0, rev 2.00/2.72, addr 2> on usbus7 umass0: SCSI over Bulk-Only; quirks = 0xc100 umass0:5:0: Attached to scbus5 ukbd0: <Keyboard Interface> on usbus2 kbd2 at ukbd0 ugen7.3: <Generic USB Storage> at usbus7 umass1 on uhub0 umass1: <Generic USB Storage, class 0/0, rev 2.00/2.72, addr 3> on usbus7 umass1: SCSI over Bulk-Only; quirks = 0xc100 umass1:6:1: Attached to scbus6 ses0 at mps0 bus 0 scbus0 target 21 lun 0 ses0: <LSI CORP SAS2X28 0717> Fixed Enclosure Services SPC-3 SCSI device ses0: Serial Number ses0: 600.000MB/s transfers ses0: Command Queueing enabled ses0: SCSI-3 ENC Device ses1 at mps0 bus 0 scbus0 target 41 lun 0 ses1: <LSI CORP SAS2X36 0717> Fixed Enclosure Services SPC-3 SCSI device ses1: Serial Number ses1: 600.000MB/s transfers ses1: Command Queueing enabled ses1: SCSI-3 ENC Device da1 at mps0 bus 0 scbus0 target 9 lun 0 da1: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device da1: Serial Number 7SGA0SVC da1: 600.000MB/s transfers da1: Command Queueing enabled da1: 7630885MB (1953506646 4096 byte sectors) da5 at mps0 bus 0 scbus0 target 13 lun 0 da5: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device da5: Serial Number 7SGDWJYC da5: 600.000MB/s transfers da5: Command Queueing enabled da5: 7630885MB (1953506646 4096 byte sectors) da8 at mps0 bus 0 scbus0 target 16 lun 0 da8: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device da8: Serial Number 7SGDP3MC da8: 600.000MB/s transfers da8: Command Queueing enabled da8: 7630885MB (1953506646 4096 byte sectors) da11 at mps0 bus 0 scbus0 target 147 lun 0 da11: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device da11: Serial Number 7SGDR52C da11: 600.000MB/s transfers da11: Command Queueing enabled da11: 7630885MB (1953506646 4096 byte sectors) da2 at mps0 bus 0 scbus0 target 10 lun 0 da2: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device da2: Serial Number 7SGBVZZC da2: 600.000MB/s transfers da2: Command Queueing enabled da2: 7630885MB (1953506646 4096 byte sectors) da13 at mps0 bus 0 scbus0 target 149 lun 0 da13: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device da13: Serial Number 7SGJTBBC da13: 600.000MB/s transfers da13: Command Queueing enabled da13: 7630885MB (1953506646 4096 byte sectors) da18 at mps0 bus 0 scbus0 target 154 lun 0 da18: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device da18: Serial Number 7SGBBYBC da18: 600.000MB/s transfers da18: Command Queueing enabled da18: 7630885MB (1953506646 4096 byte sectors) da19 at mps0 bus 0 scbus0 target 155 lun 0 da19: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device da19: Serial Number 7SGDHK9C da19: 600.000MB/s transfers da19: Command Queueing enabled da19: 7630885MB (1953506646 4096 byte sectors) da12 at mps0 bus 0 scbus0 target 148 lun 0 da12: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device da12: Serial Number 7SGJRZ9C da12: 600.000MB/s transfers da12: Command Queueing enabled da12: 7630885MB (1953506646 4096 byte sectors) da23 at mps0 bus 0 scbus0 target 159 lun 0 da23: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device da23: Serial Number 7SGJ46ZC da23: 600.000MB/s transfers da23: Command Queueing enabled da23: 7630885MB (1953506646 4096 byte sectors) da21 at mps0 bus 0 scbus0 target 157 lun 0 da21: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device da21: Serial Number 7SGDMJXC da21: 600.000MB/s transfers da21: Command Queueing enabled da21: 7630885MB (1953506646 4096 byte sectors) da14 at mps0 bus 0 scbus0 target 150 lun 0 da14: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device da14: Serial Number 7SH4YJ5L da14: 600.000MB/s transfers da14: Command Queueing enabled da14: 7630885MB (1953506646 4096 byte sectors) da7 at mps0 bus 0 scbus0 target 15 lun 0 da7: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device da7: Serial Number 7SH4YHKL da7: 600.000MB/s transfers da7: Command Queueing enabled da7: 7630885MB (1953506646 4096 byte sectors) da10 at mps0 bus 0 scbus0 target 18 lun 0 da10: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device da10: Serial Number 7SGDEZAC da10: 600.000MB/s transfers da10: Command Queueing enabled da10: 7630885MB (1953506646 4096 byte sectors) da17 at mps0 bus 0 scbus0 target 153 lun 0 da17: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device da17: Serial Number 7SH4YJHL da17: 600.000MB/s transfers da17: Command Queueing enabled da17: 7630885MB (1953506646 4096 byte sectors) da16 at mps0 bus 0 scbus0 target 152 lun 0 da16: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device da16: Serial Number 7SGJ7VDC da16: 600.000MB/s transfers da16: Command Queueing enabled da16: 7630885MB (1953506646 4096 byte sectors) da9 at mps0 bus 0 scbus0 target 17 lun 0 da9: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device da9: Serial Number 7SGE899C da9: 600.000MB/s transfers da9: Command Queueing enabled da9: 7630885MB (1953506646 4096 byte sectors) da6 at mps0 bus 0 scbus0 target 14 lun 0 da6: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device da6: Serial Number 7SGDLLEC da6: 600.000MB/s transfers da6: Command Queueing enabled da6: 7630885MB (1953506646 4096 byte sectors) da3 at mps0 bus 0 scbus0 target 11 lun 0 da3: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device da3: Serial Number 7SGBL59C da3: 600.000MB/s transfers da3: Command Queueing enabled da3: 7630885MB (1953506646 4096 byte sectors) da20 at mps0 bus 0 scbus0 target 156 lun 0 da20: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device da20: Serial Number 7SGJG60C da20: 600.000MB/s transfers da20: Command Queueing enabled da20: 7630885MB (1953506646 4096 byte sectors) da15 at mps0 bus 0 scbus0 target 151 lun 0 da15: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device da15: Serial Number 7SGJEM1C da15: 600.000MB/s transfers da15: Command Queueing enabled da15: 7630885MB (1953506646 4096 byte sectors) da4 at mps0 bus 0 scbus0 target 12 lun 0 da4: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device da4: Serial Number 7SGUV03C da4: 600.000MB/s transfers da4: Command Queueing enabled da4: 7630885MB (1953506646 4096 byte sectors) da22 at mps0 bus 0 scbus0 target 158 lun 0 da22: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device da22: Serial Number 7SH4YN5L da22: 600.000MB/s transfers da22: Command Queueing enabled da22: 7630885MB (1953506646 4096 byte sectors) da24 at umass-sim0 bus 0 scbus5 target 0 lun 0 da24: <Generic STORAGE DEVICE 0272> Removable Direct Access SCSI device da24: Serial Number 000000000272 da24: 40.000MB/s transfers da24: 3968MB (8126464 512 byte sectors) da24: quirks=0x3<NO_SYNC_CACHE,NO_6_BYTE> da0 at mps0 bus 0 scbus0 target 8 lun 0 da0: <HGST HUH721008AL4200 A384> Fixed Direct Access SPC-4 SCSI device da0: Serial Number 7SGK036C da0: 600.000MB/s transfers da0: Command Queueing enabled da0: 7630885MB (1953506646 4096 byte sectors) da25 at umass-sim1 bus 1 scbus6 target 0 lun 0 da25: <Generic STORAGE DEVICE 0272> Removable Direct Access SCSI device da25: Serial Number 000000000272 da25: 40.000MB/s transfers da25: 3968MB (8126464 512 byte sectors) da25: quirks=0x3<NO_SYNC_CACHE,NO_6_BYTE> ses1: da23,pass25: Element descriptor: 'Slot 01' ses1: da23,pass25: SAS Device Slot Element: 1 Phys at Slot 0 ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 500304800107877f addr 5000cca2521d5731 ses1: da22,pass24: Element descriptor: 'Slot 02' ses1: da22,pass24: SAS Device Slot Element: 1 Phys at Slot 1 ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 500304800107877f addr 5000cca25241612d ses1: da21,pass23: Element descriptor: 'Slot 03' ses1: da21,pass23: SAS Device Slot Element: 1 Phys at Slot 2 ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 500304800107877f addr 5000cca25216f74d ses1: da20,pass22: Element descriptor: 'Slot 04' ses1: da20,pass22: SAS Device Slot Element: 1 Phys at Slot 3 ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 500304800107877f addr 5000cca2521dece1 ses1: da19,pass21: Element descriptor: 'Slot 05' ses1: da19,pass21: SAS Device Slot Element: 1 Phys at Slot 4 ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 500304800107877f addr 5000cca25216bb6d ses1: da18,pass20: Element descriptor: 'Slot 06' ses1: da18,pass20: SAS Device Slot Element: 1 Phys at Slot 5 ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 500304800107877f addr 5000cca25214b3b9 ses1: da17,pass19: Element descriptor: 'Slot 07' ses1: da17,pass19: SAS Device Slot Element: 1 Phys at Slot 6 ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 500304800107877f addr 5000cca252415f65 ses1: da16,pass18: Element descriptor: 'Slot 08' ses1: da16,pass18: SAS Device Slot Element: 1 Phys at Slot 7 ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 500304800107877f addr 5000cca2521d8da5 ses1: da15,pass17: Element descriptor: 'Slot 09' ses1: da15,pass17: SAS Device Slot Element: 1 Phys at Slot 8 ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 500304800107877f addr 5000cca2521de42d ses1: da14,pass16: Element descriptor: 'Slot 10' ses1: da14,pass16: SAS Device Slot Element: 1 Phys at Slot 9 ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 500304800107877f addr 5000cca252415f3d ses1: da13,pass15: Element descriptor: 'Slot 11' ses1: da13,pass15: SAS Device Slot Element: 1 Phys at Slot 10 ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 500304800107877f addr 5000cca2521e85a1 ses1: da5,pass5: Element descriptor: 'Slot 12' ses1: da5,pass5: SAS Device Slot Element: 1 Phys at Slot 11 ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 500304800107877f addr 5000cca252176f71 ses1: da12,pass14: Element descriptor: 'Slot 13' ses1: da12,pass14: SAS Device Slot Element: 1 Phys at Slot 12 ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 500304800107877f addr 5000cca2521e70c5 ses1: da11,pass13: Element descriptor: 'Slot 14' ses1: da11,pass13: SAS Device Slot Element: 1 Phys at Slot 13 ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 500304800107877f addr 5000cca252171e9d ses1: da10,pass10: Element descriptor: 'Slot 15' ses1: da10,pass10: SAS Device Slot Element: 1 Phys at Slot 14 ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 500304800107877f addr 5000cca25216a3b5 ses1: da6,pass6: Element descriptor: 'Slot 16' ses1: da6,pass6: SAS Device Slot Element: 1 Phys at Slot 15 ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 500304800107877f addr 5000cca25216e905 ses1: da3,pass3: Element descriptor: 'Slot 17' ses1: da3,pass3: SAS Device Slot Element: 1 Phys at Slot 16 ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 500304800107877f addr 5000cca25215112d ses1: da2,pass2: Element descriptor: 'Slot 18' ses1: da2,pass2: SAS Device Slot Element: 1 Phys at Slot 17 ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 500304800107877f addr 5000cca2521595bd ses1: da9,pass9: Element descriptor: 'Slot 19' ses1: da9,pass9: SAS Device Slot Element: 1 Phys at Slot 18 ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 500304800107877f addr 5000cca252181fed ses1: da8,pass8: Element descriptor: 'Slot 20' ses1: da8,pass8: SAS Device Slot Element: 1 Phys at Slot 19 ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 500304800107877f addr 5000cca252170ee5 ses1: da7,pass7: Element descriptor: 'Slot 21' ses1: da7,pass7: SAS Device Slot Element: 1 Phys at Slot 20 ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 500304800107877f addr 5000cca252415ef1 ses1: da4,pass4: Element descriptor: 'Slot 22' ses1: da4,pass4: SAS Device Slot Element: 1 Phys at Slot 21 ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 500304800107877f addr 5000cca2522efb91 ses1: da1,pass1: Element descriptor: 'Slot 23' ses1: da1,pass1: SAS Device Slot Element: 1 Phys at Slot 22 ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 500304800107877f addr 5000cca252123a65 ses1: da0,pass0: Element descriptor: 'Slot 24' ses1: da0,pass0: SAS Device Slot Element: 1 Phys at Slot 23 ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 500304800107877f addr 5000cca2521eeac9 Trying to mount root from zfs:zroot []... lo0: link state changed to UP igb0: link state changed to UP ioat0: <TBG IOAT Ch0> mem 0xfadf0000-0xfadf3fff irq 43 at device 22.0 on pci0 ioat0: Capabilities: 77<Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break> ioat1: <TBG IOAT Ch1> mem 0xfadec000-0xfadeffff irq 44 at device 22.1 on pci0 ioat1: Capabilities: 77<Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break> ioat2: <TBG IOAT Ch2> mem 0xfade8000-0xfadebfff irq 45 at device 22.2 on pci0 ioat2: Capabilities: 77<Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break> ioat3: <TBG IOAT Ch3> mem 0xfade4000-0xfade7fff irq 46 at device 22.3 on pci0 ioat3: Capabilities: 77<Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break> ioat4: <TBG IOAT Ch4> mem 0xfade0000-0xfade3fff irq 43 at device 22.4 on pci0 ioat4: Capabilities: 77<Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break> ioat5: <TBG IOAT Ch5> mem 0xfaddc000-0xfaddffff irq 44 at device 22.5 on pci0 ioat5: Capabilities: 77<Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break> ioat6: <TBG IOAT Ch6> mem 0xfadd8000-0xfaddbfff irq 45 at device 22.6 on pci0 ioat6: Capabilities: 77<Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break> ioat7: <TBG IOAT Ch7> mem 0xfadd4000-0xfadd7fff irq 46 at device 22.7 on pci0 ioat7: Capabilities: 77<Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break> ums0 on uhub1 ums0: <Mouse Interface> on usbus2 Security policy loaded: MAC/ntpd (mac_ntpd)
Some debugging information for the crash: Unread portion of the kernel message buffer: panic: solaris assert: avl_find(&scn->scn_queue, sds, &where) == ((void *)0) (0xfffff80f108a1ac0 == 0x0), file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c, line: 1029 cpuid = 6 time = 1565487588 KDB: stack backtrace: #0 0xffffffff80be7967 at kdb_backtrace+0x67 #1 0xffffffff80b9b543 at vpanic+0x1a3 #2 0xffffffff80b9b393 at panic+0x43 #3 0xffffffff829fe22c at assfail3+0x2c #4 0xffffffff826cafbb at dsl_scan_ds_clone_swapped+0x1db #5 0xffffffff826ba4e5 at dsl_dataset_clone_swap_sync_impl+0x8c5 #6 0xffffffff826a0de0 at dmu_recv_end_sync+0x310 #7 0xffffffff826c369e at dsl_sync_task_sync+0xae #8 0xffffffff826c204b at dsl_pool_sync+0x3cb #9 0xffffffff826eb635 at spa_sync+0xab5 #10 0xffffffff826f8f98 at txg_sync_thread+0x208 #11 0xffffffff80b5bf13 at fork_exit+0x83 #12 0xffffffff810506ae at fork_trampoline+0xe Uptime: 14h3m53s netdump: overwriting mbuf zone pointers netdump in progress. searching for server... netdumping to 10.10.5.2 (ac:1f:6b:48:3a:ac) Dumping 21259 out of 196556 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% __curthread () at ./machine/pcpu.h:234 234 struct thread *td; (kgdb) bt #0 __curthread () at ./machine/pcpu.h:234 #1 doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:366 #2 0xffffffff80b9b12b in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:446 #3 0xffffffff80b9b5a3 in vpanic (fmt=<optimized out>, ap=0xfffffe0171b705c0) at /usr/src/sys/kern/kern_shutdown.c:872 #4 0xffffffff80b9b393 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:799 #5 0xffffffff829fe22c in assfail3 (a=<unavailable>, lv=<unavailable>, op=<unavailable>, rv=<unavailable>, f=<unavailable>, l=<optimized out>) at /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:91 #6 0xffffffff826cafbb in scan_ds_queue_insert (scn=<optimized out>, dsobj=85951, txg=1) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c:1029 #7 dsl_scan_ds_clone_swapped (ds1=<optimized out>, ds2=<optimized out>, tx=0xfffff8013b108b00) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c:2036 #8 0xffffffff826ba4e5 in dsl_dataset_clone_swap_sync_impl (clone=0xfffff8186d34f800, origin_head=0xfffff80bf0b90800, tx=<optimized out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c:3663 #9 0xffffffff826a0de0 in dmu_recv_end_sync (arg=0xfffffe0171b93510, tx=0xfffff8013b108b00) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c:3302 #10 0xffffffff826c369e in dsl_sync_task_sync (dst=0xfffffe0171b932e8, tx=0xfffff8013b108b00) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_synctask.c:234 #11 0xffffffff826c204b in dsl_pool_sync (dp=<optimized out>, txg=<optimized out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c:819 #12 0xffffffff826eb635 in spa_sync (spa=<optimized out>, txg=<optimized out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:8088 #13 0xffffffff826f8f98 in txg_sync_thread (arg=0xfffff80106632800) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:543 #14 0xffffffff80b5bf13 in fork_exit (callout=0xffffffff826f8d90 <txg_sync_thread>, arg=0xfffff80106632800, frame=0xfffffe0171b70c00) at /usr/src/sys/kern/kern_fork.c:1057 #15 <signal handler called> (kgdb) fr 8 #8 0xffffffff826ba4e5 in dsl_dataset_clone_swap_sync_impl (clone=0xfffff8186d34f800, origin_head=0xfffff80bf0b90800, tx=<optimized out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c:3663 (kgdb) p clone->ds_object $1 = 85951 (kgdb) p origin_head->ds_object $2 = 85918
The root cause is that both 85951 and 85918 were already in scn_queue and the code in dsl_scan_ds_clone_swapped fails to properly handle that case. It has other logic problems as well.
A commit references this bug: Author: avg Date: Thu Sep 19 09:43:56 UTC 2019 New revision: 352506 URL: https://svnweb.freebsd.org/changeset/base/352506 Log: fix dsl_scan_ds_clone_swapped logic It was incorrect with respect to swapping dataset IDs both in the on-disk ZAP object and the in-memory queue. In both cases, if only ds1 was already present, then it would be first replaced with ds2 and then ds2 would be replaced back with ds1. Also, both cases did not properly handle a situation where both ds1 and ds2 are already queued. A duplicate insertion would be attempted and its failure would result in a panic. This change has also been submitted to ZoL as zfsonlinux/zfs@dd262c9 PR: 239566 Reported by: pascal.guitierrez@gmail.com MFC after: 4 days Sponsored by: CyberSecure Changes: head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c
A commit references this bug: Author: avg Date: Wed Sep 25 20:01:00 UTC 2019 New revision: 352723 URL: https://svnweb.freebsd.org/changeset/base/352723 Log: MFC r352506: fix dsl_scan_ds_clone_swapped logic PR: 239566 Changes: _U stable/12/ stable/12/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c
A commit references this bug: Author: avg Date: Wed Sep 25 20:01:50 UTC 2019 New revision: 352724 URL: https://svnweb.freebsd.org/changeset/base/352724 Log: MFC r352506: fix dsl_scan_ds_clone_swapped logic PR: 239566 Changes: _U stable/11/ stable/11/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c
A commit references this bug: Author: avg Date: Mon Sep 30 15:08:46 UTC 2019 New revision: 352905 URL: https://svnweb.freebsd.org/changeset/base/352905 Log: Merge from stable/12 r352723: fix dsl_scan_ds_clone_swapped logic PR: 239566 Approved by: re (gjb) Changes: _U releng/12.1/ releng/12.1/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c