Jan  9 13:51:32 zfsraid kernel: FreeBSD 11.2-STABLE #0 r342400: Tue Dec 25 20:39:35 PST 2018
Jan  9 13:51:32 zfsraid kernel:     unixwiz@cdbuilder:/rw/usr/obj.1.11R1-amd64-zfs/rw/zfsraid/admin/FreeBSD/sources/11-STABLE-r342400/sys/FBCD64.zfs amd64
Jan  9 13:51:32 zfsraid kernel: FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1)
Jan  9 13:51:32 zfsraid kernel: VT(vga): resolution 640x480
Jan  9 13:51:32 zfsraid kernel: CPU: Intel(R) Xeon(R) CPU E5-2407 0 @ 2.20GHz (2194.75-MHz K8-class CPU)
Jan  9 13:51:32 zfsraid kernel:   Origin="GenuineIntel"  Id=0x206d7  Family=0x6  Model=0x2d  Stepping=7
Jan  9 13:51:32 zfsraid kernel:   Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Jan  9 13:51:32 zfsraid kernel:   Features2=0x1fbee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX>
Jan  9 13:51:32 zfsraid kernel:   AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
Jan  9 13:51:32 zfsraid kernel:   AMD Features2=0x1<LAHF>
Jan  9 13:51:32 zfsraid kernel:   Structured Extended Features3=0x9c000000<IBPB,STIBP,L1DFL,SSBD>
Jan  9 13:51:32 zfsraid kernel:   XSAVE Features=0x1<XSAVEOPT>
Jan  9 13:51:32 zfsraid kernel:   VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID
Jan  9 13:51:32 zfsraid kernel:   TSC: P-state invariant, performance statistics
Jan  9 13:51:32 zfsraid kernel: real memory  = 25769803776 (24576 MB)
Jan  9 13:51:32 zfsraid kernel: avail memory = 23862988800 (22757 MB)
...
Jan  9 13:51:32 zfsraid kernel: ahci0: <Intel Patsburg AHCI SATA controller> port 0x2070-0x2077,0x2060-0x2063,0x2050-0x2057,0x2040-0x2043,0x2020-0x203f mem 0xd1300000-0xd13007ff irq 21 at device 31.2 on pci0
Jan  9 13:51:32 zfsraid kernel: ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
Jan  9 13:51:32 zfsraid kernel: ahcich0: <AHCI channel> at channel 0 on ahci0
Jan  9 13:51:32 zfsraid kernel: ahcich1: <AHCI channel> at channel 1 on ahci0
Jan  9 13:51:32 zfsraid kernel: ahcich2: <AHCI channel> at channel 2 on ahci0
Jan  9 13:51:32 zfsraid kernel: ahcich3: <AHCI channel> at channel 3 on ahci0
Jan  9 13:51:32 zfsraid kernel: ahcich4: <AHCI channel> at channel 4 on ahci0
Jan  9 13:51:32 zfsraid kernel: ahcich5: <AHCI channel> at channel 5 on ahci0
Jan  9 13:51:32 zfsraid kernel: ahciem0: <AHCI enclosure management bridge> on ahci0
...
Jan  9 13:51:32 zfsraid kernel: ses0 at ahciem0 bus 0 scbus6 target 0 lun 0
Jan  9 13:51:32 zfsraid kernel: ses0: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device
Jan  9 13:51:32 zfsraid kernel: ses0: SEMB SES Device
Jan  9 13:51:32 zfsraid kernel: ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
Jan  9 13:51:32 zfsraid kernel: ada0: <WDC WD2005FBYZ-01YCBB3 RR09> ACS-3 ATA SATA 3.x device
Jan  9 13:51:32 zfsraid kernel: ada0: Serial Number WD-WMC6N0M966KP
Jan  9 13:51:32 zfsraid kernel: ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
Jan  9 13:51:32 zfsraid kernel: ada0: Command Queueing enabled
Jan  9 13:51:32 zfsraid kernel: ada0: 1907729MB (3907029168 512 byte sectors)
Jan  9 13:51:32 zfsraid kernel: ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
Jan  9 13:51:32 zfsraid kernel: ada1: <WDC WD200MFYYZ-01D45B0 01.01K01> ATA8-ACS SATA 3.x device
Jan  9 13:51:32 zfsraid kernel: ada1: Serial Number WD-WCC130309711
Jan  9 13:51:32 zfsraid kernel: ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
Jan  9 13:51:32 zfsraid kernel: ada1: Command Queueing enabled
Jan  9 13:51:32 zfsraid kernel: ada1: 1907729MB (3907029168 512 byte sectors)
Jan  9 13:51:32 zfsraid kernel: ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
Jan  9 13:51:32 zfsraid kernel: ada2: <WDC WD2004FBYZ-01YCBB1 RR03> ACS-3 ATA SATA 3.x device
Jan  9 13:51:32 zfsraid kernel: ada2: Serial Number WD-WMC6N0D265EP
Jan  9 13:51:32 zfsraid kernel: ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
Jan  9 13:51:32 zfsraid kernel: ada2: Command Queueing enabled
Jan  9 13:51:32 zfsraid kernel: ada2: 1907729MB (3907029168 512 byte sectors)
Jan  9 13:51:32 zfsraid kernel: ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
Jan  9 13:51:32 zfsraid kernel: ada3: <WDC WD2004FBYZ-01YCBB1 RR03> ACS-3 ATA SATA 3.x device
Jan  9 13:51:32 zfsraid kernel: ada3: Serial Number WD-WMC6N0D6ETDA
Jan  9 13:51:32 zfsraid kernel: ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
Jan  9 13:51:32 zfsraid kernel: ada3: Command Queueing enabled
Jan  9 13:51:32 zfsraid kernel: ada3: 1907729MB (3907029168 512 byte sectors)
...
Jan 14 21:56:48 zfsraid kernel: (ada0:ahcich0:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 80 28 60 4b 40 82 00 00 00 00 00
Jan 14 21:56:48 zfsraid kernel: (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
Jan 14 21:56:48 zfsraid kernel: (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF )
Jan 14 21:56:48 zfsraid kernel: (ada0:ahcich0:0:0:0): RES: 41 10 28 60 4b 00 82 00 00 00 00
Jan 14 21:56:48 zfsraid kernel: (ada0:ahcich0:0:0:0): Retrying command
Jan 14 21:57:04 zfsraid kernel: (ada0:ahcich0:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 90 b8 bd 5e 40 30 00 00 00 00 00
Jan 14 21:57:04 zfsraid kernel: (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
Jan 14 21:57:04 zfsraid kernel: (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF )
Jan 14 21:57:04 zfsraid kernel: (ada0:ahcich0:0:0:0): RES: 41 10 b8 bd 5e 00 30 00 00 00 00
Jan 14 21:57:04 zfsraid kernel: (ada0:ahcich0:0:0:0): Retrying command
...tons more of the same, then...
Jan 15 03:02:56 zfsraid kernel: ahcich0: Timeout on slot 4 port 0
Jan 15 03:02:56 zfsraid kernel: ahcich0: is 00000000 cs 00000000 ss f000003f rs f000003f tfd 40 serr 00000000 cmd 0000c517
Jan 15 03:02:56 zfsraid kernel: (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 08 30 11 bf 40 d3 00 00 00 00 00
Jan 15 03:02:56 zfsraid kernel: (ada0:ahcich0:0:0:0): CAM status: Command timeout
Jan 15 03:02:56 zfsraid kernel: (ada0:ahcich0:0:0:0): Retrying command
Jan 15 03:03:50 zfsraid kernel: ahcich0: Timeout on slot 4 port 0
Jan 15 03:03:50 zfsraid kernel: ahcich0: is 00000000 cs 00000000 ss 000001d8 rs 000001d8 tfd 40 serr 00000000 cmd 0000c817
Jan 15 03:03:50 zfsraid kernel: (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 08 50 1a 27 40 e3 00 00 00 00 00
Jan 15 03:03:50 zfsraid kernel: (ada0:ahcich0:0:0:0): CAM status: Command timeout
Jan 15 03:03:50 zfsraid kernel: (ada0:ahcich0:0:0:0): Retrying command
Jan 15 03:05:18 zfsraid kernel: (ada0:ahcich0:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 18 b0 ee a4 40 35 00 00 00 00 00
Jan 15 03:05:18 zfsraid kernel: (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
Jan 15 03:05:18 zfsraid kernel: (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF )
Jan 15 03:05:18 zfsraid kernel: (ada0:ahcich0:0:0:0): RES: 41 10 b0 ee a4 00 35 00 00 00 00
Jan 15 03:05:18 zfsraid kernel: (ada0:ahcich0:0:0:0): Retrying command
Jan 15 03:05:31 zfsraid kernel: (ada0:ahcich0:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 60 30 24 a5 40 35 00 00 00 00 00
Jan 15 03:05:31 zfsraid kernel: (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
Jan 15 03:05:31 zfsraid kernel: (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF )
Jan 15 03:05:31 zfsraid kernel: (ada0:ahcich0:0:0:0): RES: 41 10 30 24 a5 00 35 00 00 00 00
Jan 15 03:05:31 zfsraid kernel: (ada0:ahcich0:0:0:0): Retrying command
Jan 15 03:08:04 zfsraid kernel: (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 20 60 b8 c8 40 04 00 00 00 00 00
Jan 15 03:08:04 zfsraid kernel: (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
Jan 15 03:08:04 zfsraid kernel: (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jan 15 03:08:04 zfsraid kernel: (ada0:ahcich0:0:0:0): RES: 41 40 60 b8 c8 00 04 00 00 00 00
Jan 15 03:08:04 zfsraid kernel: (ada0:ahcich0:0:0:0): Retrying command
Jan 15 03:08:45 zfsraid kernel: ahcich0: Timeout on slot 20 port 0
Jan 15 03:08:45 zfsraid kernel: ahcich0: is 00000000 cs 00000000 ss 00d00000 rs 00d00000 tfd 40 serr 00000000 cmd 0000d717
Jan 15 03:08:45 zfsraid kernel: ahcich0: is 00000000 cs 00000000 ss 00d00000 rs 00d00000 tfd 40 serr 00000000 cmd 0000d717
Jan 15 03:08:45 zfsraid kernel: (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 08 a8 c5 4c 40 6e 00 00 00 00 00
Jan 15 03:08:45 zfsraid kernel: (ada0:ahcich0:0:0:0): CAM status: Command timeout
Jan 15 03:08:45 zfsraid kernel: (ada0:ahcich0:0:0:0): Retrying command
Jan 15 03:09:08 zfsraid kernel: (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 08 10 05 ee 40 5d 00 00 00 00 00
Jan 15 03:09:08 zfsraid kernel: (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
Jan 15 03:09:08 zfsraid kernel: (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jan 15 03:09:08 zfsraid kernel: (ada0:ahcich0:0:0:0): RES: 41 40 10 05 ee 00 5d 00 00 00 00
Jan 15 03:09:08 zfsraid kernel: (ada0:ahcich0:0:0:0): Retrying command
Jan 15 03:10:06 zfsraid kernel: (ada0:ahcich0:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 60 b4 a5 40 35 00 00 00 00 00
Jan 15 03:10:06 zfsraid kernel: (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
Jan 15 03:10:06 zfsraid kernel: (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF )
Jan 15 03:10:06 zfsraid kernel: (ada0:ahcich0:0:0:0): RES: 41 10 60 b4 a5 00 35 00 00 00 00
Jan 15 03:10:06 zfsraid kernel: (ada0:ahcich0:0:0:0): Retrying command
Jan 15 03:11:11 zfsraid kernel: ahcich0: Timeout on slot 18 port 0
Jan 15 03:11:11 zfsraid kernel: ahcich0: is 00000000 cs 00000000 ss 003ff800 rs 003ff800 tfd 40 serr 00000000 cmd 0000d517
Jan 15 03:11:11 zfsraid kernel: (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 08 58 55 b2 40 de 00 00 00 00 00
Jan 15 03:11:11 zfsraid kernel: (ada0:ahcich0:0:0:0): CAM status: Command timeout
Jan 15 03:11:11 zfsraid kernel: (ada0:ahcich0:0:0:0): Retrying command
Jan 15 03:12:44 zfsraid kernel: (ada0:ahcich0:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 18 00 77 ae 40 36 00 00 00 00 00
Jan 15 03:12:44 zfsraid kernel: (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
Jan 15 03:12:44 zfsraid kernel: (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF )
Jan 15 03:12:44 zfsraid kernel: (ada0:ahcich0:0:0:0): RES: 41 10 00 77 ae 00 36 00 00 00 00
Jan 15 03:12:44 zfsraid kernel: (ada0:ahcich0:0:0:0): Retrying command
Jan 15 03:13:52 zfsraid kernel: (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 50 c8 49 3c 40 d8 00 00 00 00 00
Jan 15 03:13:52 zfsraid kernel: (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
Jan 15 03:13:52 zfsraid kernel: (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jan 15 03:13:52 zfsraid kernel: (ada0:ahcich0:0:0:0): RES: 41 40 c8 49 3c 00 d8 00 00 00 00
Jan 15 03:13:52 zfsraid kernel: (ada0:ahcich0:0:0:0): Retrying command
Jan 15 03:14:30 zfsraid kernel: (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 08 e0 56 ed 40 31 00 00 00 00 00
Jan 15 03:14:30 zfsraid kernel: (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
Jan 15 03:14:30 zfsraid kernel: (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jan 15 03:14:30 zfsraid kernel: (ada0:ahcich0:0:0:0): RES: 41 40 e0 56 ed 00 31 00 00 00 00
Jan 15 03:14:30 zfsraid kernel: (ada0:ahcich0:0:0:0): Retrying command
...etc...