| Summary: | Performance regression in em(4) driver with bridge/tap and used with bhyve | ||
|---|---|---|---|
| Product: | Base System | Reporter: | Jason Tubnor <jason> |
| Component: | kern | Assignee: | freebsd-net (Nobody) <net> |
| Status: | Open --- | ||
| Severity: | Affects Some People | CC: | allanjude, bwidawsk, d8zNeCFG, net, sbruno, virtualization |
| Priority: | --- | Keywords: | IntelNetworking, bhyve, performance, regression |
| Version: | 12.0-RELEASE | ||
| Hardware: | amd64 | ||
| OS: | Any | ||
This might be related to bug #235031 and bug #234520. -- Martin |
After performing an in-place upgrade of a bhyve hypervisor host from 11.2-p7 to 12.0-RELEASE, network performance tanked on the em0 physical interface on the host that is used in a bridge with bhyve taps for a virtualised firewall. Internet traffic went from 50Mb/s to 3.5Mb/s. When dumping interface traffic in the OpenBSD guest the following appeared while performing pkg upgrade on the hypervisor host after initial base upgrade: truncated-ip - 886 bytes missing!149.20.1.201.80 truncated-ip6 - 906 bytes missing!2001:4f8:1:11::50:1.80 To replicate, create a clean, bare metal host. Setup vlans to use em0 as a parent and then bridge the vlans to insert taps into and create a bhyve guest and use it as a firewall. Further testing showed that no filtering was needed on the guest, the guest simply using the default route, out the require vlan on the em0 interface showed the issue, so it isn't some sort of firewall issue. Turning LRO on and off on the hosts em0 interface changed the behavior. LRO off, and things return to normal, however, load increases on the CPU due to the workload now returning to the kernel. Turn LRO back on and the issue comes back. Workaround: ifconfig em0 -lro up # pciconf -lcvb hostb0@pci0:0:0:0: class=0x060000 card=0x7a721462 chip=0x591f8086 rev=0x05 hdr=0x00 vendor = 'Intel Corporation' device = 'Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers' class = bridge subclass = HOST-PCI cap 09[e0] = vendor (length 16) Intel cap 0 version 1 vgapci0@pci0:0:2:0: class=0x030000 card=0x7a721462 chip=0x59128086 rev=0x04 hdr=0x00 vendor = 'Intel Corporation' device = 'HD Graphics 630' class = display subclass = VGA bar [10] = type Memory, range 64, base rxde000000, size 16777216, enabled bar [18] = type Prefetchable Memory, range 64, base rxc0000000, size 268435456, enabled bar [20] = type I/O Port, range 32, base rxf000, size 64, enabled cap 09[40] = vendor (length 12) Intel cap 0 version 1 cap 10[70] = PCI-Express 2 root endpoint max data 128(128) FLR cap 05[ac] = MSI supports 1 message cap 01[d0] = powerspec 2 supports D0 D3 current D0 ecap 001b[100] = unknown 1 ecap 000f[200] = ATS 1 ecap 0013[300] = unknown 1 none0@pci0:0:8:0: class=0x088000 card=0x7a721462 chip=0x19118086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = 'Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th Gen Core Processor Gaussian Mixture Model' class = base peripheral bar [10] = type Memory, range 64, base rxdf22f000, size 4096, enabled cap 05[90] = MSI supports 1 message cap 01[dc] = powerspec 2 supports D0 D3 current D0 cap 13[f0] = PCI Advanced Features: FLR TP xhci0@pci0:0:20:0: class=0x0c0330 card=0x7a721462 chip=0xa2af8086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '200 Series/Z370 Chipset Family USB 3.0 xHCI Controller' class = serial bus subclass = USB bar [10] = type Memory, range 64, base rxdf210000, size 65536, enabled cap 01[70] = powerspec 2 supports D0 D3 current D0 cap 05[80] = MSI supports 8 messages, 64 bit enabled with 1 message none1@pci0:0:20:2: class=0x118000 card=0x7a721462 chip=0xa2b18086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '200 Series PCH Thermal Subsystem' class = dasp bar [10] = type Memory, range 64, base rxdf22e000, size 4096, enabled cap 01[50] = powerspec 3 supports D0 D3 current D0 cap 05[80] = MSI supports 1 message none2@pci0:0:22:0: class=0x078000 card=0x7a721462 chip=0xa2ba8086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '200 Series PCH CSME HECI' class = simple comms bar [10] = type Memory, range 64, base rxdf22d000, size 4096, enabled cap 01[50] = powerspec 3 supports D0 D3 current D0 cap 05[8c] = MSI supports 1 message, 64 bit ahci0@pci0:0:23:0: class=0x010601 card=0x7a721462 chip=0xa2828086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '200 Series PCH SATA controller [AHCI mode]' class = mass storage subclass = SATA bar [10] = type Memory, range 32, base rxdf228000, size 8192, enabled bar [14] = type Memory, range 32, base rxdf22c000, size 256, enabled bar [18] = type I/O Port, range 32, base rxf090, size 8, enabled bar [1c] = type I/O Port, range 32, base rxf080, size 4, enabled bar [20] = type I/O Port, range 32, base rxf060, size 32, enabled bar [24] = type Memory, range 32, base rxdf22b000, size 2048, enabled cap 05[80] = MSI supports 1 message enabled with 1 message cap 01[70] = powerspec 3 supports D0 D3 current D0 cap 12[a8] = SATA Index-Data Pair pcib1@pci0:0:27:0: class=0x060400 card=0x00000000 chip=0xa2eb8086 rev=0xf0 hdr=0x01 vendor = 'Intel Corporation' device = '200 Series PCH PCI Express Root Port' class = bridge subclass = PCI-PCI cap 10[40] = PCI-Express 2 root port max data 256(256) ARI disabled link x4(x4) speed 2.5(8.0) slot 24 power limit 250 mW cap 05[80] = MSI supports 1 message cap 0d[90] = PCI Bridge card=0x00000000 cap 01[a0] = powerspec 3 supports D0 D3 current D0 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 000d[140] = ACS 1 ecap 0019[220] = PCIe Sec 1 lane errors 0 pcib2@pci0:0:28:0: class=0x060400 card=0x7a721462 chip=0xa2948086 rev=0xf0 hdr=0x01 vendor = 'Intel Corporation' device = '200 Series PCH PCI Express Root Port' class = bridge subclass = PCI-PCI cap 10[40] = PCI-Express 2 root port max data 256(256) ARI disabled link x2(x2) speed 8.0(8.0) slot 8 power limit 250 mW cap 05[80] = MSI supports 1 message cap 0d[90] = PCI Bridge card=0x7a721462 cap 01[a0] = powerspec 3 supports D0 D3 current D0 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected ecap 000d[140] = ACS 1 ecap 0019[220] = PCIe Sec 1 lane errors 0 pcib3@pci0:0:28:6: class=0x060400 card=0x7a721462 chip=0xa2968086 rev=0xf0 hdr=0x01 vendor = 'Intel Corporation' device = '200 Series PCH PCI Express Root Port' class = bridge subclass = PCI-PCI cap 10[40] = PCI-Express 2 root port max data 128(256) ARI disabled link x1(x1) speed 2.5(8.0) slot 10 power limit 100 mW cap 05[80] = MSI supports 1 message cap 0d[90] = PCI Bridge card=0x7a721462 cap 01[a0] = powerspec 3 supports D0 D3 current D0 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 000d[140] = ACS 1 ecap 0019[220] = PCIe Sec 1 lane errors 0 isab0@pci0:0:31:0: class=0x060100 card=0x7a721462 chip=0xa2c48086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '200 Series PCH LPC Controller (H270)' class = bridge subclass = PCI-ISA none3@pci0:0:31:2: class=0x058000 card=0x7a721462 chip=0xa2a18086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '200 Series/Z370 Chipset Family Power Management Controller' class = memory bar [10] = type Memory, range 32, base rxdf224000, size 16384, enabled hdac0@pci0:0:31:3: class=0x040300 card=0xfa721462 chip=0xa2f08086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '200 Series PCH HD Audio' class = multimedia subclass = HDA bar [10] = type Memory, range 64, base rxdf220000, size 16384, enabled bar [20] = type Memory, range 64, base rxdf200000, size 65536, enabled cap 01[50] = powerspec 3 supports D0 D3 current D0 cap 05[60] = MSI supports 1 message, 64 bit enabled with 1 message none4@pci0:0:31:4: class=0x0c0500 card=0x7a721462 chip=0xa2a38086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '200 Series/Z370 Chipset Family SMBus Controller' class = serial bus subclass = SMBus bar [10] = type Memory, range 64, base rxdf22a000, size 256, enabled bar [20] = type I/O Port, range 32, base rxf040, size 32, enabled em0@pci0:1:0:0: class=0x020000 card=0x115e8086 chip=0x105e8086 rev=0x06 hdr=0x00 vendor = 'Intel Corporation' device = '82571EB/82571GB Gigabit Ethernet Controller D0/D1 (copper applications)' class = network subclass = ethernet bar [10] = type Memory, range 32, base rxdf1a0000, size 131072, enabled bar [14] = type Memory, range 32, base rxdf180000, size 131072, enabled bar [18] = type I/O Port, range 32, base rxe020, size 32, enabled cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message cap 10[e0] = PCI-Express 1 endpoint max data 256(256) RO NS link x4(x4) speed 2.5(2.5) ASPM disabled(L0s) ecap 0001[100] = AER 1 0 fatal 1 non-fatal 0 corrected ecap 0003[140] = Serial 1 6805caffffaabbcc em1@pci0:1:0:1: class=0x020000 card=0x115e8086 chip=0x105e8086 rev=0x06 hdr=0x00 vendor = 'Intel Corporation' device = '82571EB/82571GB Gigabit Ethernet Controller D0/D1 (copper applications)' class = network subclass = ethernet bar [10] = type Memory, range 32, base rxdf140000, size 131072, enabled bar [14] = type Memory, range 32, base rxdf120000, size 131072, enabled bar [18] = type I/O Port, range 32, base rxe000, size 32, enabled cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message cap 10[e0] = PCI-Express 1 endpoint max data 256(256) RO NS link x4(x4) speed 2.5(2.5) ASPM disabled(L0s) ecap 0001[100] = AER 1 0 fatal 1 non-fatal 0 corrected ecap 0003[140] = Serial 1 6805caffffaabbcc xhci1@pci0:2:0:0: class=0x0c0330 card=0x7a721462 chip=0x21421b21 rev=0x00 hdr=0x00 vendor = 'ASMedia Technology Inc.' class = serial bus subclass = USB bar [10] = type Memory, range 64, base rxdf000000, size 32768, enabled cap 05[50] = MSI supports 8 messages, 64 bit enabled with 1 message cap 11[68] = MSI-X supports 8 messages Table in map 0x10[0x2000], PBA in map 0x10[0x2080] cap 01[78] = powerspec 3 supports D0 D3 current D0 cap 10[80] = PCI-Express 2 legacy endpoint max data 256(512) RO NS link x2(x2) speed 8.0(8.0) ASPM disabled(L0s/L1) ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected ecap 0019[200] = PCIe Sec 1 lane errors 0 ecap 0018[300] = LTR 1 pcib4@pci0:3:0:0: class=0x060400 card=0x00000000 chip=0x10801b21 rev=0x03 hdr=0x01 vendor = 'ASMedia Technology Inc.' device = 'ASM1083/1085 PCIe to PCI Bridge' class = bridge subclass = PCI-PCI cap 05[50] = MSI supports 1 message, 64 bit cap 01[78] = powerspec 3 supports D0 D1 D2 D3 current D0 cap 10[80] = PCI-Express 1 PCI bridge max data 128(128) RO NS link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1) cap 0d[c0] = PCI Bridge card=0x00000000 ecap 0002[100] = VC 1 max VC0 # dmesg ---<<BOOT>>--- Copyright (c) 1992-2018 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-RELEASE r341666 GENERIC amd64 FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1) VT(efifb): resolution 800x600 CPU: Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz (3600.14-MHz K8-class CPU) Origin="GenuineIntel" Id=0x906e9 Family=0x6 Model=0x9e Stepping=9 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> AMD Features2=0x121<LAHF,ABM,Prefetch> Structured Extended Features=0x29c6fbf<FSGSBASE,TSCADJ,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE> XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 34359738368 (32768 MB) avail memory = 33228730368 (31689 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <ALASKA A M I > FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 hardware threads random: unblocking device. ioapic0 <Version 2.0> irqs 0-119 on motherboard Launching APs: 1 5 2 6 3 4 7 Timecounter "TSC-low" frequency 1800069504 Hz quality 1000 random: entropy device external interface kbd1 at kbdmux0 netmap: loaded module [ath_hal] loaded module_register_init: MOD_LOAD (vesa, 0xffffffff810f9770, 0) error 19 random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" nexus0 efirtc0: <EFI Realtime Clock> on motherboard efirtc0: registered as a time-of-day clock, resolution 1.000000s cryptosoft0: <software crypto> on motherboard acpi0: <ALASKA A M I > on motherboard acpi0: Power Button (fixed) cpu0: <ACPI CPU> on acpi0 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 24000000 Hz quality 950 Event timer "HPET" frequency 24000000 Hz quality 550 atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0 atrtc0: Warning: Couldn't map I/O. atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0xde000000-0xdeffffff,0xc0000000-0xcfffffff irq 16 at device 2.0 on pci0 vgapci0: Boot video device xhci0: <Intel Union Point USB 3.0 controller> mem 0xdf210000-0xdf21ffff irq 16 at device 20.0 on pci0 xhci0: 32 bytes context size, 64-bit DMA usbus0 on xhci0 usbus0: 5.0Gbps Super Speed USB v3.0 pci0: <simple comms> at device 22.0 (no driver attached) ahci0: <Intel Union Point AHCI SATA controller> port 0xf090-0xf097,0xf080-0xf083,0xf060-0xf07f mem 0xdf228000-0xdf229fff,0xdf22c000-0xdf22c0ff,0xdf22b000-0xdf22b7ff irq 16 at device 23.0 on pci0 ahci0: AHCI v1.31 with 6 6Gbps ports, Port Multiplier not supported ahcich0: <AHCI channel> at channel 0 on ahci0 ahcich1: <AHCI channel> at channel 1 on ahci0 ahcich2: <AHCI channel> at channel 2 on ahci0 ahcich3: <AHCI channel> at channel 3 on ahci0 ahcich4: <AHCI channel> at channel 4 on ahci0 ahcich5: <AHCI channel> at channel 5 on ahci0 ahciem0: <AHCI enclosure management bridge> on ahci0 pcib1: <ACPI PCI-PCI bridge> irq 16 at device 27.0 on pci0 pci1: <ACPI PCI bus> on pcib1 em0: <Intel(R) PRO/1000 Network Connection> port 0xe020-0xe03f mem 0xdf1a0000-0xdf1bffff,0xdf180000-0xdf19ffff irq 16 at device 0.0 on pci1 em0: attach_pre capping queues at 1 em0: using 1024 tx descriptors and 1024 rx descriptors em0: msix_init qsets capped at 1 em0: Unable to map MSIX table em0: Using an MSI interrupt em0: allocated for 1 tx_queues em0: allocated for 1 rx_queues em0: Ethernet address: 68:05:ca:aa:bb:cc em0: netmap queues/slots: TX 1/1024, RX 1/1024 em1: <Intel(R) PRO/1000 Network Connection> port 0xe000-0xe01f mem 0xdf140000-0xdf15ffff,0xdf120000-0xdf13ffff irq 17 at device 0.1 on pci1 em1: attach_pre capping queues at 1 em1: using 1024 tx descriptors and 1024 rx descriptors em1: msix_init qsets capped at 1 em1: Unable to map MSIX table em1: Using an MSI interrupt em1: allocated for 1 tx_queues em1: allocated for 1 rx_queues em1: Ethernet address: 68:05:ca:bb:cc:dd em1: netmap queues/slots: TX 1/1024, RX 1/1024 pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0 pci2: <ACPI PCI bus> on pcib2 xhci1: <XHCI (generic) USB 3.0 controller> mem 0xdf000000-0xdf007fff irq 16 at device 0.0 on pci2 xhci1: 32 bytes context size, 64-bit DMA xhci1: Unable to map MSI-X table usbus1 on xhci1 usbus1: 5.0Gbps Super Speed USB v3.0 pcib3: <ACPI PCI-PCI bridge> irq 18 at device 28.6 on pci0 pci3: <ACPI PCI bus> on pcib3 pcib4: <ACPI PCI-PCI bridge> irq 18 at device 0.0 on pci3 pci4: <ACPI PCI bus> on pcib4 isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 pci0: <memory> at device 31.2 (no driver attached) hdac0: <Intel Kabylake-H HDA Controller> mem 0xdf220000-0xdf223fff,0xdf200000-0xdf20ffff irq 16 at device 31.3 on pci0 acpi_button0: <Sleep Button> on acpi0 acpi_button1: <Power Button> on acpi0 acpi_tz0: <Thermal Zone> on acpi0 acpi_tz1: <Thermal Zone> on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 orm0: <ISA Option ROM> at iomem 0xc0000-0xcffff pnpid ORM0000 on isa0 atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbdc0: non-PNP ISA device will be removed from GENERIC in FreeBSD 12. uart1: <16550 or compatible> at port 0x2f8 irq 3 on isa0 uart1: non-PNP ISA device will be removed from GENERIC in FreeBSD 12. est0: <Enhanced SpeedStep Frequency Control> on cpu0 ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 1.000 msec hdacc0: <Realtek ALC887 HDA CODEC> at cad 0 on hdac0 hdaa0: <Realtek ALC887 Audio Function Group> at nid 1 on hdacc0 pcm0: <Realtek ALC887 (Rear Analog)> at nid 20 and 24,26 on hdaa0 pcm1: <Realtek ALC887 (Front Analog)> at nid 27 and 25 on hdaa0 hdacc1: <Intel Kabylake HDA CODEC> at cad 2 on hdac0 hdaa1: <Intel Kabylake Audio Function Group> at nid 1 on hdacc1 pcm2: <Intel Kabylake (HDMI/DP 8ch)> at nid 3 on hdaa1 ugen0.1: <0x8086 XHCI root HUB> at usbus0 ugen1.1: <0x1b21 XHCI root HUB> at usbus1 uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 uhub1: <0x1b21 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1 ses0 at ahciem0 bus 0 scbus6 target 0 lun 0 ses0: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device ses0: SEMB SES Device ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: <Samsung SSD 850 EVO 250GB EMT01B6Q> ACS-2 ATA SATA 3.x device ada0: Serial Number S21MNSAFC26263P ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes) ada0: Command Queueing enabled ada0: 238475MB (488397168 512 byte sectors) ada0: quirks=0x3<4K,NCQ_TRIM_BROKEN> ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 ada1: <Samsung SSD 850 EVO 250GB EMT01B6Q> ACS-2 ATA SATA 3.x device ada1: Serial Number S21MNSAFC26262X ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes) ada1: Command Queueing enabled ada1: 238475MB (488397168 512 byte sectors) ada1: quirks=0x3<4K,NCQ_TRIM_BROKEN> ada2 at ahcich2 bus 0 scbus2 target 0 lun 0 ada2: <TOSHIBA MD04ACA400 FP2A> ATA8-ACS SATA 3.x device ada2: Serial Number 473PK36VFSAA ada2: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 8192bytes) ada2: Command Queueing enabled ada2: 3815447MB (7814037168 512 byte sectors) ada3 at ahcich3 bus 0 scbus3 target 0 lun 0 ada3: <TOSHIBA MD04ACA400 FP2A> ATA8-ACS SATA 3.x device ada3: Serial Number 473TK4EEFSAA ada3: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 8192bytes) ada3: Command Queueing enabled ada3: 3815447MB (7814037168 512 byte sectors) ada4 at ahcich4 bus 0 scbus4 target 0 lun 0 ada4: <TOSHIBA MD04ACA400 FP2A> ATA8-ACS SATA 3.x device ada4: Serial Number 473UK2WOFSAA ada4: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 8192bytes) ada4: Command Queueing enabled ada4: 3815447MB (7814037168 512 byte sectors) Trying to mount root from zfs:tb1/ROOT/default []... GEOM_MIRROR: Device mirror/swap launched (2/2). uhub1: 4 ports with 4 removable, self powered Root mount waiting for: usbus0 uhub0: 24 ports with 24 removable, self powered GEOM_ELI: Device mirror/swap.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: software lo0: link state changed to UP nd6_dad_timer: cancel DAD on vlan2 because of ND6_IFF_IFDISABLED. nd6_dad_timer: cancel DAD on vlan4 because of ND6_IFF_IFDISABLED. nd6_dad_timer: cancel DAD on vlan3 because of ND6_IFF_IFDISABLED. em0: link state changed to UP vlan1: link state changed to UP vlan2: link state changed to UP vlan3: link state changed to UP vlan4: link state changed to UP