Bug 234550 - Performance regression in em(4) driver with bridge/tap and used with bhyve
Summary: Performance regression in em(4) driver with bridge/tap and used with bhyve
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.0-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-net mailing list
URL:
Keywords: IntelNetworking, bhyve, performance, regression
Depends on:
Blocks:
 
Reported: 2019-01-01 22:58 UTC by Jason Tubnor
Modified: 2019-01-20 08:26 UTC (History)
6 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jason Tubnor 2019-01-01 22:58:41 UTC
After performing an in-place upgrade of a bhyve hypervisor host from 11.2-p7 to 12.0-RELEASE, network performance tanked on the em0 physical interface on the host that is used in a bridge with bhyve taps for a virtualised firewall.

Internet traffic went from 50Mb/s to 3.5Mb/s.  When dumping interface traffic in the OpenBSD guest the following appeared while performing pkg upgrade on the hypervisor host after initial base upgrade:

truncated-ip - 886 bytes missing!149.20.1.201.80
truncated-ip6 - 906 bytes missing!2001:4f8:1:11::50:1.80

To replicate, create a clean, bare metal host.  Setup vlans to use em0 as a parent and then bridge the vlans to insert taps into and create a bhyve guest and use it as a firewall.  Further testing showed that no filtering was needed on the guest, the guest simply using the default route, out the require vlan on the em0 interface showed the issue, so it isn't some sort of firewall issue.  Turning LRO on and off on the hosts em0 interface changed the behavior.  LRO off, and things return to normal, however, load increases on the CPU due to the workload now returning to the kernel.  Turn LRO back on and the issue comes back.

Workaround:

ifconfig em0 -lro up

# pciconf -lcvb
hostb0@pci0:0:0:0:      class=0x060000 card=0x7a721462 chip=0x591f8086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers'
    class      = bridge
    subclass   = HOST-PCI
    cap 09[e0] = vendor (length 16) Intel cap 0 version 1
vgapci0@pci0:0:2:0:     class=0x030000 card=0x7a721462 chip=0x59128086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'HD Graphics 630'
    class      = display
    subclass   = VGA
    bar   [10] = type Memory, range 64, base rxde000000, size 16777216, enabled
    bar   [18] = type Prefetchable Memory, range 64, base rxc0000000, size 268435456, enabled
    bar   [20] = type I/O Port, range 32, base rxf000, size 64, enabled
    cap 09[40] = vendor (length 12) Intel cap 0 version 1
    cap 10[70] = PCI-Express 2 root endpoint max data 128(128) FLR
    cap 05[ac] = MSI supports 1 message
    cap 01[d0] = powerspec 2  supports D0 D3  current D0
    ecap 001b[100] = unknown 1
    ecap 000f[200] = ATS 1
    ecap 0013[300] = unknown 1
none0@pci0:0:8:0:       class=0x088000 card=0x7a721462 chip=0x19118086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th Gen Core Processor Gaussian Mixture Model'
    class      = base peripheral
    bar   [10] = type Memory, range 64, base rxdf22f000, size 4096, enabled
    cap 05[90] = MSI supports 1 message
    cap 01[dc] = powerspec 2  supports D0 D3  current D0
    cap 13[f0] = PCI Advanced Features: FLR TP
xhci0@pci0:0:20:0:      class=0x0c0330 card=0x7a721462 chip=0xa2af8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '200 Series/Z370 Chipset Family USB 3.0 xHCI Controller'
    class      = serial bus
    subclass   = USB
    bar   [10] = type Memory, range 64, base rxdf210000, size 65536, enabled
    cap 01[70] = powerspec 2  supports D0 D3  current D0
    cap 05[80] = MSI supports 8 messages, 64 bit enabled with 1 message
none1@pci0:0:20:2:      class=0x118000 card=0x7a721462 chip=0xa2b18086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '200 Series PCH Thermal Subsystem'
    class      = dasp
    bar   [10] = type Memory, range 64, base rxdf22e000, size 4096, enabled
    cap 01[50] = powerspec 3  supports D0 D3  current D0
    cap 05[80] = MSI supports 1 message
none2@pci0:0:22:0:      class=0x078000 card=0x7a721462 chip=0xa2ba8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '200 Series PCH CSME HECI'
    class      = simple comms
    bar   [10] = type Memory, range 64, base rxdf22d000, size 4096, enabled
    cap 01[50] = powerspec 3  supports D0 D3  current D0
    cap 05[8c] = MSI supports 1 message, 64 bit
ahci0@pci0:0:23:0:      class=0x010601 card=0x7a721462 chip=0xa2828086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '200 Series PCH SATA controller [AHCI mode]'
    class      = mass storage
    subclass   = SATA
    bar   [10] = type Memory, range 32, base rxdf228000, size 8192, enabled
    bar   [14] = type Memory, range 32, base rxdf22c000, size 256, enabled
    bar   [18] = type I/O Port, range 32, base rxf090, size 8, enabled
    bar   [1c] = type I/O Port, range 32, base rxf080, size 4, enabled
    bar   [20] = type I/O Port, range 32, base rxf060, size 32, enabled
    bar   [24] = type Memory, range 32, base rxdf22b000, size 2048, enabled
    cap 05[80] = MSI supports 1 message enabled with 1 message
    cap 01[70] = powerspec 3  supports D0 D3  current D0
    cap 12[a8] = SATA Index-Data Pair
pcib1@pci0:0:27:0:      class=0x060400 card=0x00000000 chip=0xa2eb8086 rev=0xf0 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = '200 Series PCH PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[40] = PCI-Express 2 root port max data 256(256) ARI disabled
                 link x4(x4) speed 2.5(8.0)
                 slot 24 power limit 250 mW
    cap 05[80] = MSI supports 1 message
    cap 0d[90] = PCI Bridge card=0x00000000
    cap 01[a0] = powerspec 3  supports D0 D3  current D0
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
    ecap 000d[140] = ACS 1
    ecap 0019[220] = PCIe Sec 1 lane errors 0
pcib2@pci0:0:28:0:      class=0x060400 card=0x7a721462 chip=0xa2948086 rev=0xf0 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = '200 Series PCH PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[40] = PCI-Express 2 root port max data 256(256) ARI disabled
                 link x2(x2) speed 8.0(8.0)
                 slot 8 power limit 250 mW
    cap 05[80] = MSI supports 1 message
    cap 0d[90] = PCI Bridge card=0x7a721462
    cap 01[a0] = powerspec 3  supports D0 D3  current D0
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 000d[140] = ACS 1
    ecap 0019[220] = PCIe Sec 1 lane errors 0
pcib3@pci0:0:28:6:      class=0x060400 card=0x7a721462 chip=0xa2968086 rev=0xf0 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = '200 Series PCH PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[40] = PCI-Express 2 root port max data 128(256) ARI disabled
                 link x1(x1) speed 2.5(8.0)
                 slot 10 power limit 100 mW
    cap 05[80] = MSI supports 1 message
    cap 0d[90] = PCI Bridge card=0x7a721462
    cap 01[a0] = powerspec 3  supports D0 D3  current D0
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
    ecap 000d[140] = ACS 1
    ecap 0019[220] = PCIe Sec 1 lane errors 0
isab0@pci0:0:31:0:      class=0x060100 card=0x7a721462 chip=0xa2c48086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '200 Series PCH LPC Controller (H270)'
    class      = bridge
    subclass   = PCI-ISA
none3@pci0:0:31:2:      class=0x058000 card=0x7a721462 chip=0xa2a18086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '200 Series/Z370 Chipset Family Power Management Controller'
    class      = memory
    bar   [10] = type Memory, range 32, base rxdf224000, size 16384, enabled
hdac0@pci0:0:31:3:      class=0x040300 card=0xfa721462 chip=0xa2f08086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '200 Series PCH HD Audio'
    class      = multimedia
    subclass   = HDA
    bar   [10] = type Memory, range 64, base rxdf220000, size 16384, enabled
    bar   [20] = type Memory, range 64, base rxdf200000, size 65536, enabled
    cap 01[50] = powerspec 3  supports D0 D3  current D0
    cap 05[60] = MSI supports 1 message, 64 bit enabled with 1 message
none4@pci0:0:31:4:      class=0x0c0500 card=0x7a721462 chip=0xa2a38086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '200 Series/Z370 Chipset Family SMBus Controller'
    class      = serial bus
    subclass   = SMBus
    bar   [10] = type Memory, range 64, base rxdf22a000, size 256, enabled
    bar   [20] = type I/O Port, range 32, base rxf040, size 32, enabled
em0@pci0:1:0:0: class=0x020000 card=0x115e8086 chip=0x105e8086 rev=0x06 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82571EB/82571GB Gigabit Ethernet Controller D0/D1 (copper applications)'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 32, base rxdf1a0000, size 131072, enabled
    bar   [14] = type Memory, range 32, base rxdf180000, size 131072, enabled
    bar   [18] = type I/O Port, range 32, base rxe020, size 32, enabled
    cap 01[c8] = powerspec 2  supports D0 D3  current D0
    cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
    cap 10[e0] = PCI-Express 1 endpoint max data 256(256) RO NS
                 link x4(x4) speed 2.5(2.5) ASPM disabled(L0s)
    ecap 0001[100] = AER 1 0 fatal 1 non-fatal 0 corrected
    ecap 0003[140] = Serial 1 6805caffffaabbcc
em1@pci0:1:0:1: class=0x020000 card=0x115e8086 chip=0x105e8086 rev=0x06 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82571EB/82571GB Gigabit Ethernet Controller D0/D1 (copper applications)'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 32, base rxdf140000, size 131072, enabled
    bar   [14] = type Memory, range 32, base rxdf120000, size 131072, enabled
    bar   [18] = type I/O Port, range 32, base rxe000, size 32, enabled
    cap 01[c8] = powerspec 2  supports D0 D3  current D0
    cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
    cap 10[e0] = PCI-Express 1 endpoint max data 256(256) RO NS
                 link x4(x4) speed 2.5(2.5) ASPM disabled(L0s)
    ecap 0001[100] = AER 1 0 fatal 1 non-fatal 0 corrected
    ecap 0003[140] = Serial 1 6805caffffaabbcc
xhci1@pci0:2:0:0:       class=0x0c0330 card=0x7a721462 chip=0x21421b21 rev=0x00 hdr=0x00
    vendor     = 'ASMedia Technology Inc.'
    class      = serial bus
    subclass   = USB
    bar   [10] = type Memory, range 64, base rxdf000000, size 32768, enabled
    cap 05[50] = MSI supports 8 messages, 64 bit enabled with 1 message
    cap 11[68] = MSI-X supports 8 messages
                 Table in map 0x10[0x2000], PBA in map 0x10[0x2080]
    cap 01[78] = powerspec 3  supports D0 D3  current D0
    cap 10[80] = PCI-Express 2 legacy endpoint max data 256(512) RO NS
                 link x2(x2) speed 8.0(8.0) ASPM disabled(L0s/L1)
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0019[200] = PCIe Sec 1 lane errors 0
    ecap 0018[300] = LTR 1
pcib4@pci0:3:0:0:       class=0x060400 card=0x00000000 chip=0x10801b21 rev=0x03 hdr=0x01
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1083/1085 PCIe to PCI Bridge'
    class      = bridge
    subclass   = PCI-PCI
    cap 05[50] = MSI supports 1 message, 64 bit
    cap 01[78] = powerspec 3  supports D0 D1 D2 D3  current D0
    cap 10[80] = PCI-Express 1 PCI bridge max data 128(128) RO NS
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
    cap 0d[c0] = PCI Bridge card=0x00000000
    ecap 0002[100] = VC 1 max VC0

# dmesg
---<<BOOT>>---
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-RELEASE r341666 GENERIC amd64
FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1)
VT(efifb): resolution 800x600
CPU: Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz (3600.14-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x906e9  Family=0x6  Model=0x9e  Stepping=9
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x29c6fbf<FSGSBASE,TSCADJ,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 34359738368 (32768 MB)
avail memory = 33228730368 (31689 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 hardware threads
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-119 on motherboard
Launching APs: 1 5 2 6 3 4 7
Timecounter "TSC-low" frequency 1800069504 Hz quality 1000
random: entropy device external interface
kbd1 at kbdmux0
netmap: loaded module
[ath_hal] loaded
module_register_init: MOD_LOAD (vesa, 0xffffffff810f9770, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
nexus0
efirtc0: <EFI Realtime Clock> on motherboard
efirtc0: registered as a time-of-day clock, resolution 1.000000s
cryptosoft0: <software crypto> on motherboard
acpi0: <ALASKA A M I > on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 24000000 Hz quality 950
Event timer "HPET" frequency 24000000 Hz quality 550
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0xde000000-0xdeffffff,0xc0000000-0xcfffffff irq 16 at device 2.0 on pci0
vgapci0: Boot video device
xhci0: <Intel Union Point USB 3.0 controller> mem 0xdf210000-0xdf21ffff irq 16 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <simple comms> at device 22.0 (no driver attached)
ahci0: <Intel Union Point AHCI SATA controller> port 0xf090-0xf097,0xf080-0xf083,0xf060-0xf07f mem 0xdf228000-0xdf229fff,0xdf22c000-0xdf22c0ff,0xdf22b000-0xdf22b7ff irq 16 at device 23.0 on pci0
ahci0: AHCI v1.31 with 6 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ahciem0: <AHCI enclosure management bridge> on ahci0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 27.0 on pci0
pci1: <ACPI PCI bus> on pcib1
em0: <Intel(R) PRO/1000 Network Connection> port 0xe020-0xe03f mem 0xdf1a0000-0xdf1bffff,0xdf180000-0xdf19ffff irq 16 at device 0.0 on pci1
em0: attach_pre capping queues at 1
em0: using 1024 tx descriptors and 1024 rx descriptors
em0: msix_init qsets capped at 1
em0: Unable to map MSIX table
em0: Using an MSI interrupt
em0: allocated for 1 tx_queues
em0: allocated for 1 rx_queues
em0: Ethernet address: 68:05:ca:aa:bb:cc
em0: netmap queues/slots: TX 1/1024, RX 1/1024
em1: <Intel(R) PRO/1000 Network Connection> port 0xe000-0xe01f mem 0xdf140000-0xdf15ffff,0xdf120000-0xdf13ffff irq 17 at device 0.1 on pci1
em1: attach_pre capping queues at 1
em1: using 1024 tx descriptors and 1024 rx descriptors
em1: msix_init qsets capped at 1
em1: Unable to map MSIX table
em1: Using an MSI interrupt
em1: allocated for 1 tx_queues
em1: allocated for 1 rx_queues
em1: Ethernet address: 68:05:ca:bb:cc:dd
em1: netmap queues/slots: TX 1/1024, RX 1/1024
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib2
xhci1: <XHCI (generic) USB 3.0 controller> mem 0xdf000000-0xdf007fff irq 16 at device 0.0 on pci2
xhci1: 32 bytes context size, 64-bit DMA
xhci1: Unable to map MSI-X table
usbus1 on xhci1
usbus1: 5.0Gbps Super Speed USB v3.0
pcib3: <ACPI PCI-PCI bridge> irq 18 at device 28.6 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> irq 18 at device 0.0 on pci3
pci4: <ACPI PCI bus> on pcib4
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
pci0: <memory> at device 31.2 (no driver attached)
hdac0: <Intel Kabylake-H HDA Controller> mem 0xdf220000-0xdf223fff,0xdf200000-0xdf20ffff irq 16 at device 31.3 on pci0
acpi_button0: <Sleep Button> on acpi0
acpi_button1: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
acpi_tz1: <Thermal Zone> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
orm0: <ISA Option ROM> at iomem 0xc0000-0xcffff pnpid ORM0000 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbdc0: non-PNP ISA device will be removed from GENERIC in FreeBSD 12.
uart1: <16550 or compatible> at port 0x2f8 irq 3 on isa0
uart1: non-PNP ISA device will be removed from GENERIC in FreeBSD 12.
est0: <Enhanced SpeedStep Frequency Control> on cpu0
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
hdacc0: <Realtek ALC887 HDA CODEC> at cad 0 on hdac0
hdaa0: <Realtek ALC887 Audio Function Group> at nid 1 on hdacc0
pcm0: <Realtek ALC887 (Rear Analog)> at nid 20 and 24,26 on hdaa0
pcm1: <Realtek ALC887 (Front Analog)> at nid 27 and 25 on hdaa0
hdacc1: <Intel Kabylake HDA CODEC> at cad 2 on hdac0
hdaa1: <Intel Kabylake Audio Function Group> at nid 1 on hdacc1
pcm2: <Intel Kabylake (HDMI/DP 8ch)> at nid 3 on hdaa1
ugen0.1: <0x8086 XHCI root HUB> at usbus0
ugen1.1: <0x1b21 XHCI root HUB> at usbus1
uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
uhub1: <0x1b21 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1
ses0 at ahciem0 bus 0 scbus6 target 0 lun 0
ses0: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device
ses0: SEMB SES Device
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <Samsung SSD 850 EVO 250GB EMT01B6Q> ACS-2 ATA SATA 3.x device
ada0: Serial Number S21MNSAFC26263P
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada0: Command Queueing enabled
ada0: 238475MB (488397168 512 byte sectors)
ada0: quirks=0x3<4K,NCQ_TRIM_BROKEN>
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <Samsung SSD 850 EVO 250GB EMT01B6Q> ACS-2 ATA SATA 3.x device
ada1: Serial Number S21MNSAFC26262X
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada1: Command Queueing enabled
ada1: 238475MB (488397168 512 byte sectors)
ada1: quirks=0x3<4K,NCQ_TRIM_BROKEN>
ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
ada2: <TOSHIBA MD04ACA400 FP2A> ATA8-ACS SATA 3.x device
ada2: Serial Number 473PK36VFSAA
ada2: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 3815447MB (7814037168 512 byte sectors)
ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
ada3: <TOSHIBA MD04ACA400 FP2A> ATA8-ACS SATA 3.x device
ada3: Serial Number 473TK4EEFSAA
ada3: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 3815447MB (7814037168 512 byte sectors)
ada4 at ahcich4 bus 0 scbus4 target 0 lun 0
ada4: <TOSHIBA MD04ACA400 FP2A> ATA8-ACS SATA 3.x device
ada4: Serial Number 473UK2WOFSAA
ada4: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 3815447MB (7814037168 512 byte sectors)
Trying to mount root from zfs:tb1/ROOT/default []...
GEOM_MIRROR: Device mirror/swap launched (2/2).
uhub1: 4 ports with 4 removable, self powered
Root mount waiting for: usbus0
uhub0: 24 ports with 24 removable, self powered
GEOM_ELI: Device mirror/swap.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:     Crypto: software
lo0: link state changed to UP
nd6_dad_timer: cancel DAD on vlan2 because of ND6_IFF_IFDISABLED.
nd6_dad_timer: cancel DAD on vlan4 because of ND6_IFF_IFDISABLED.
nd6_dad_timer: cancel DAD on vlan3 because of ND6_IFF_IFDISABLED.
em0: link state changed to UP
vlan1: link state changed to UP
vlan2: link state changed to UP
vlan3: link state changed to UP
vlan4: link state changed to UP
Comment 1 Martin Birgmeier 2019-01-20 08:26:56 UTC
This might be related to bug #235031 and bug #234520.

-- Martin