Bug 242891 - mlx5_core0 ERR
Summary: mlx5_core0 ERR
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.1-STABLE
Hardware: amd64 Any
: --- Affects Many People
Assignee: freebsd-net (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-12-26 10:19 UTC by Michael
Modified: 2020-06-15 03:44 UTC (History)
3 users (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Michael 2019-12-26 10:19:40 UTC 
Mellanox ConnectX-4 Lx
HPE Ethernet 10/25Gb 2-port 640FLR-SFP28 Adapter
  Firmware Version 14.26.1040, Driver Version 2.30.21713.0

mlx5en: Mellanox Ethernet driver 3.5.2 (September 2019)

Errors are:
 mlx5_core0: ERR: mlx5_init_once:918:(pid 0): Unable to find vendor specific capabilities
 mlx5_core0: WARN: mlx5_fwdump_prep:76:(pid 0): Failed to find vendor-specific capability, error 2
 hn0: mce0 SIOCGRSSKEY failed: 22

While in test mode, the work of the network card is not suspicious, but maybe there are some pitfalls?
Is it possible to put this network card in a production server under high load with such error messages?

Hyper-V 2019 server; VM - FreeBSD 13.0-CURRENT

root@frw02:~# pciconf -lv | grep mlx -C 10
mlx5_core0@pci1:0:2:0:  class=0x020000 rev=0x80 hdr=0x00 vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d3
    vendor     = 'Mellanox Technologies'
    device     = 'MT27710 Family [ConnectX-4 Lx Virtual Function]'
    class      = network
    subclass   = ethernet

root@frw02:~# sysctl -a | grep mlx
mlx5en: Mellanox Ethernet driver 3.5.2 (September 2019)
mlx5_core0: <mlx5_core> at device 2.0 on pci0
vmbus0: allocated type 3 (0xfe0100000-0xfe01fffff) for rid 10 of mlx5_core0
mlx5_core0: Lazy allocation of 0x100000 bytes rid 0x10 type 3 at 0xfe0100000
mlx5_core0: ERR: mlx5_init_once:918:(pid 0): Unable to find vendor specific capabilities
mlx5_core0: attempting to allocate 8 MSI-X vectors (8 supported)
mlx5_core0: using IRQs 24-31 for MSI-X
mlx5_core0: WARN: mlx5_fwdump_prep:76:(pid 0): Failed to find vendor-specific capability, error 2
<6>mce0: ERR: mlx5e_ioctl:3320:(pid 0): tso6 disabled due to -txcsum6.
<6>mce0: ERR: mlx5e_ioctl:3346:(pid 0): enable txcsum6 first.
device	mlx
device	mlxfw
device	mlx4
device	mlx4en
device	mlx5
device	mlx5en
irq24: mlx5_core0:63 @cpu0(domain0): 0
irq25: mlx5_core0:65 @cpu0(domain0): 6626
irq26: mlx5_core0:67 @cpu0(domain0): 0
irq27: mlx5_core0:69 @cpu0(domain0): 20
irq28: mlx5_core0:71 @cpu0(domain0): 267
irq29: mlx5_core0:73 @cpu0(domain0): 14
irq30: mlx5_core0:75 @cpu0(domain0): 8
irq31: mlx5_core0:77 @cpu0(domain0): 8
hw.mlx5.calibr.normal: 30
hw.mlx5.calibr.fast: 1
hw.mlx5.calibr.duration: 20
hw.mlx5.relaxed_ordering_write: 0
hw.mlx5.auto_fw_update: 0
hw.mlx5.fast_unload_enabled: 1
hw.mlx5.prof_sel: 2
hw.mlx5.debug_mask: 0
hw.mlx5.sw_reset_timeout: 1200
hw.mlx5.fw_reset_enable: 1
dev.mce.0.conf.device_name: mlx5_core0
dev.mlx5_core.0.hw.board_id: HP_2690110034
dev.mlx5_core.0.hw.fw_version: 14.26.1040
dev.mlx5_core.0.pme_stats.module_unplug: 0
dev.mlx5_core.0.pme_stats.module_plug: 0
dev.mlx5_core.0.pme_stats.errors.cable_shorted: 0
dev.mlx5_core.0.pme_stats.errors.high_temp: 0
dev.mlx5_core.0.pme_stats.errors.unknown_id: 0
dev.mlx5_core.0.pme_stats.errors.enforce_part_number: 0
dev.mlx5_core.0.pme_stats.errors.no_eeprom: 0
dev.mlx5_core.0.pme_stats.errors.bus_stuck: 0
dev.mlx5_core.0.pme_stats.errors.long_range: 0
dev.mlx5_core.0.pme_stats.errors.power_budget: 0
dev.mlx5_core.0.power_value: 0
dev.mlx5_core.0.power_status: 0
dev.mlx5_core.0.msix_eqvec: 0
dev.mlx5_core.0.%parent: pci0
dev.mlx5_core.0.%pnpinfo: vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d3 class=0x020000
dev.mlx5_core.0.%location: slot=2 function=0 dbsf=pci1:0:2:0
dev.mlx5_core.0.%driver: mlx5_core
dev.mlx5_core.0.%desc: Mellanox Core driver 3.5.2 (September 2019)
dev.mlx5_core.%parent: 
compat.linuxkpi.mlx4_inline_thold: 104
compat.linuxkpi.mlx4_pfcrx: 0
compat.linuxkpi.mlx4_pfctx: 0
compat.linuxkpi.mlx4_udp_rss: 1
compat.linuxkpi.mlx4_log_mtts_per_seg: 3
compat.linuxkpi.mlx4_use_prio: 0
compat.linuxkpi.mlx4_log_num_vlan: 0
compat.linuxkpi.mlx4_log_num_mac: 7
compat.linuxkpi.mlx4_enable_4k_uar: 0
compat.linuxkpi.mlx4_enable_64b_cqe_eqe: 1
compat.linuxkpi.mlx4_log_num_mgm_entry_size: 10
compat.linuxkpi.mlx4_msi_x: 1
compat.linuxkpi.mlx4_enable_qos: 0
compat.linuxkpi.mlx4_internal_err_reset: 1


boot_verbose="YES"
---> /var/log/messages :

 kernel: ---<<BOOT>>---
 kernel: Table 'FACP' at 0x3e7f7000
 kernel: Table 'APIC' at 0x3e7f8000
 kernel: APIC: Found table at 0x3e7f8000
 kernel: APIC: Using the MADT enumerator.
 kernel: Copyright (c) 1992-2019 The FreeBSD Project.
 kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
 kernel: 	The Regents of the University of California. All rights reserved.
 kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
 kernel: FreeBSD 13.0-CURRENT #0 b3a86c4242f-c264893(master): Sat Dec  7 15:59:14 EET 2019
 kernel:     root@frw02.hv.lan:/usr/obj/usr/src/amd64.amd64/sys/FNET amd64
 kernel: FreeBSD clang version 9.0.0 (tags/RELEASE_900/final 372316) (based on LLVM 9.0.0)
 kernel: Table 'FACP' at 0x3e7f7000
 kernel: Table 'APIC' at 0x3e7f8000
 kernel: Table 'OEM0' at 0x3e7f6000
 kernel: Table 'WAET' at 0x3e7f5000
 kernel: Table 'SRAT' at 0x3e7f4000
 kernel: SRAT: Found table at 0x3e7f4000
 kernel: SRAT: Found CPU APIC ID 0 domain 0: enabled
 kernel: SRAT: Found CPU APIC ID 1 domain 0: enabled
 kernel: SRAT: Found CPU APIC ID 2 domain 0: enabled
 kernel: SRAT: Found CPU APIC ID 3 domain 0: enabled
 kernel: SRAT: Found CPU APIC ID 4 domain 1: enabled
 kernel: SRAT: Found CPU APIC ID 5 domain 1: enabled
 kernel: SRAT: Found CPU APIC ID 6 domain 1: enabled
 kernel: SRAT: Found CPU APIC ID 7 domain 1: enabled
 kernel: SRAT: Found memory domain 0 addr 0x0 len 0x3e800000: enabled
 kernel: SRAT: Found memory domain 1 addr 0x3e800000 len 0x3e800000: enabled
 kernel: SRAT: Found memory domain 0 addr 0x7d000000 len 0x3d800000: enabled
 kernel: SRAT: Ignoring memory at addr 0x7d000000
 kernel: SRAT: Found memory domain 0 addr 0x100000000 len 0x770000000: enabled
 kernel: SRAT: Ignoring memory at addr 0x100000000
 kernel: SRAT: Found memory domain 0 addr 0x1000000000 len 0x7800000000: enabled
 kernel: SRAT: Ignoring memory at addr 0x1000000000
 kernel: SRAT: Found memory domain 0 addr 0x10000000000 len 0x8000000000: enabled
 kernel: SRAT: Ignoring memory at addr 0x10000000000
 kernel: SRAT: Found memory domain 0 addr 0x20000000000 len 0x10000000000: enabled
 kernel: SRAT: Ignoring memory at addr 0x20000000000
 kernel: SRAT: Found memory domain 0 addr 0x40000000000 len 0x20000000000: enabled
 kernel: SRAT: Ignoring memory at addr 0x40000000000
 kernel: SRAT: Found memory domain 0 addr 0x80000000000 len 0x40000000000: enabled
 kernel: SRAT: Ignoring memory at addr 0x80000000000
 kernel: SRAT: Found memory domain 1 addr 0xba800000 len 0x3d800000: enabled
 kernel: SRAT: Ignoring memory at addr 0xba800000
 kernel: SRAT: Found memory domain 1 addr 0x870000000 len 0x770000000: enabled
 kernel: SRAT: Ignoring memory at addr 0x870000000
 kernel: SRAT: Found memory domain 1 addr 0x8800000000 len 0x7800000000: enabled
 kernel: SRAT: Ignoring memory at addr 0x8800000000
 kernel: SRAT: Found memory domain 1 addr 0x18000000000 len 0x8000000000: enabled
 kernel: SRAT: Ignoring memory at addr 0x18000000000
 kernel: SRAT: Found memory domain 1 addr 0x30000000000 len 0x10000000000: enabled
 kernel: SRAT: Ignoring memory at addr 0x30000000000
 kernel: SRAT: Found memory domain 1 addr 0x60000000000 len 0x20000000000: enabled
 kernel: SRAT: Ignoring memory at addr 0x60000000000
 kernel: SRAT: Found memory domain 1 addr 0xc0000000000 len 0x40000000000: enabled
 kernel: SRAT: Ignoring memory at addr 0xc0000000000
 kernel: Table 'FACP' at 0x3e7f7000
 kernel: Table 'APIC' at 0x3e7f8000
 kernel: Table 'OEM0' at 0x3e7f6000
 kernel: Table 'WAET' at 0x3e7f5000
 kernel: Table 'SRAT' at 0x3e7f4000
 kernel: Table 'FPDT' at 0x3e7f3000
 kernel: Table 'BGRT' at 0x3d601000
 kernel: ACPI: No SLIT table found
 kernel: PPIM 0: PA=0xf8000000, VA=0xffffffff83910000, size=0x300000, mode=0x1
 kernel: PPIM 1: PA=0x7cfe7000, VA=0xffffffff83c10000, size=0x1000, mode=0x6
 kernel: pmap: large map 8 PML4 slots (4096 GB)
 kernel: VT(efifb): resolution 1024x768
 kernel: Hyper-V Version: 10.0.17763 [SP0]
 kernel:   Features=0x2e7f<VPRUNTIME,TMREFCNT,SYNIC,SYNTM,APIC,HYPERCALL,VPINDEX,REFTSC,IDLE,TMFREQ>
 kernel:   PM Features=0x0 [C2]
 kernel:   Features3=0xbed7b2<DEBUG,XMMHC,IDLE,NUMA,TMFREQ,SYNCMC,CRASH,NPIEP>
 kernel:   Recommends: 00062c2c 00000fff
 kernel:   Limits: Vcpu:240 Lcpu:512 Int:4824
 kernel:   HW Features: 0000000f, AMD: 00000000
 kernel: Timecounter "Hyper-V" frequency 10000000 Hz quality 2000
 kernel: Preloaded elf kernel "/boot/kernel/kernel" at 0xffffffff836fb000.
 kernel: Preloaded elf obj module "/boot/kernel/zfs.ko" at 0xffffffff837046f8.
 kernel: Preloaded elf obj module "/boot/kernel/opensolaris.ko" at 0xffffffff83704ee0.
 kernel: Preloaded boot_entropy_cache "/boot/entropy" at 0xffffffff83705550.
 kernel: Table 'FACP' at 0x3e7f7000
 kernel: FACP: Found table at 0x3e7f7000
 kernel: Calibrating TSC clock ... TSC clock: 2294477338 Hz
 kernel: CPU: Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz (2294.48-MHz K8-class CPU)
 kernel:   Origin="GenuineIntel"  Id=0x206d7  Family=0x6  Model=0x2d  Stepping=7
 kernel:   Features=0x1f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT>
 kernel:   Features2=0x9e9a2203<SSE3,PCLMULQDQ,SSSE3,CX16,PCID,SSE4.1,SSE4.2,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,HV>
 kernel:   AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
 kernel:   AMD Features2=0x1<LAHF>
 kernel:   Structured Extended Features3=0xbc000400<MD_CLEAR,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>
 kernel:   XSAVE Features=0x1<XSAVEOPT>
 kernel: Data TLB0: 2-MByte or 4 MByte pages, 4-way set associative, 32 entries
 kernel: Data TLB: 4 KB pages, 4-way set associative, 64 entries
 kernel: Instruction TLB: 2M/4M pages, fully associative, 8 entries
 kernel: Instruction TLB: 4KByte pages, 4-way set associative, 64 entries
 kernel: 64-Byte prefetching
 kernel: Shared 2nd-Level TLB: 4 KByte pages, 4-way associative, 512 entries
 kernel: L2 cache: 256 kbytes, 8-way associative, 64 bytes/line
 kernel: Hypervisor: Origin = "Microsoft Hv"
 kernel: real memory  = 2097152000 (2000 MB)
 kernel: Physical memory chunk(s):
 kernel: 0x0000000000001000 - 0x000000000009efff, 647168 bytes (158 pages)
 kernel: 0x0000000000100000 - 0x00000000001fffff, 1048576 bytes (256 pages)
 kernel: 0x0000000003901000 - 0x000000003b9fffff, 940568576 bytes (229631 pages)
 kernel: 0x000000003d400000 - 0x000000003d5f5fff, 2056192 bytes (502 pages)
 kernel: 0x000000003d602000 - 0x000000003d6e9fff, 950272 bytes (232 pages)
 kernel: 0x000000003d71b000 - 0x000000003e79afff, 17301504 bytes (4224 pages)
 kernel: 0x000000003e802000 - 0x000000007b3d7fff, 1019043840 bytes (248790 pages)
 kernel: 0x000000007ce00000 - 0x000000007cfcdfff, 1892352 bytes (462 pages)
 kernel: avail memory = 1973084160 (1881 MB)
 kernel: MADT: Found CPU APIC ID 0 ACPI ID 1: enabled
 kernel: SMP: Added CPU 0 (AP)
 kernel: MADT: Found CPU APIC ID 1 ACPI ID 2: enabled
 kernel: SMP: Added CPU 1 (AP)
 kernel: MADT: Found CPU APIC ID 2 ACPI ID 3: enabled
 kernel: SMP: Added CPU 2 (AP)
 kernel: MADT: Found CPU APIC ID 3 ACPI ID 4: enabled
 kernel: SMP: Added CPU 3 (AP)
 kernel: MADT: Found CPU APIC ID 4 ACPI ID 5: enabled
 kernel: SMP: Added CPU 4 (AP)
 kernel: MADT: Found CPU APIC ID 5 ACPI ID 6: enabled
 kernel: SMP: Added CPU 5 (AP)
 kernel: MADT: Found CPU APIC ID 6 ACPI ID 7: enabled
 kernel: SMP: Added CPU 6 (AP)
 kernel: MADT: Found CPU APIC ID 7 ACPI ID 8: enabled
 kernel: SMP: Added CPU 7 (AP)
 kernel: MADT: Found CPU APIC ID 8 ACPI ID 9: disabled
 kernel: MADT: Found CPU APIC ID 9 ACPI ID 10: disabled
. . . ---//--- . . .
 kernel: MADT: Found CPU APIC ID 238 ACPI ID 239: disabled
 kernel: MADT: Found CPU APIC ID 239 ACPI ID 240: disabled
 kernel: Event timer "LAPIC" quality 100
 kernel: LAPIC: ipi_wait() us multiplier 1 (r 588318946 tsc 2294477338)
 kernel: ACPI APIC Table: <VRTUAL MICROSFT>
 kernel: Package ID shift: 2
 kernel: L3 cache ID shift: 2
 kernel: L2 cache ID shift: 1
 kernel: L1 cache ID shift: 1
 kernel: Core ID shift: 1
 kernel: INTR: Adding local APIC 2 as a target
 kernel: INTR: Adding local APIC 4 as a target
 kernel: INTR: Adding local APIC 6 as a target
 kernel: FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
 kernel: FreeBSD/SMP: 2 package(s) x 2 core(s) x 2 hardware threads
 kernel: Package HW ID = 0
 kernel: 	Core HW ID = 0
 kernel: 		CPU0 (BSP): APIC ID: 0
 kernel: 		CPU1 (AP/HT): APIC ID: 1
 kernel: 	Core HW ID = 1
 kernel: 		CPU2 (AP): APIC ID: 2
 kernel: 		CPU3 (AP/HT): APIC ID: 3
 kernel: Package HW ID = 1
 kernel: 	Core HW ID = 2
 kernel: 		CPU4 (AP): APIC ID: 4
 kernel: 		CPU5 (AP/HT): APIC ID: 5
 kernel: 	Core HW ID = 3
 kernel: 		CPU6 (AP): APIC ID: 6
 kernel: 		CPU7 (AP/HT): APIC ID: 7
 kernel: APIC: CPU 0 has ACPI ID 1
 kernel: APIC: CPU 1 has ACPI ID 2
 kernel: APIC: CPU 2 has ACPI ID 3
 kernel: APIC: CPU 3 has ACPI ID 4
 kernel: APIC: CPU 4 has ACPI ID 5
 kernel: APIC: CPU 5 has ACPI ID 6
 kernel: APIC: CPU 6 has ACPI ID 7
 kernel: APIC: CPU 7 has ACPI ID 8
 kernel: Pentium Pro MTRR support enabled
 kernel: x86bios:  IVT 0x000000-0x0004ff at 0xfffff80000000000
 kernel: x86bios: SSEG 0x001000-0x001fff at 0xfffffe0003b90000
 kernel: x86bios:  ROM 0x0a0000-0x0fefff at 0xfffff800000a0000
 kernel: SRAT: CPU 0 has memory domain 0
 kernel: SRAT: CPU 1 has memory domain 0
 kernel: SRAT: CPU 2 has memory domain 0
 kernel: SRAT: CPU 3 has memory domain 0
 kernel: SRAT: CPU 4 has memory domain 1
 kernel: SRAT: CPU 5 has memory domain 1
 kernel: SRAT: CPU 6 has memory domain 1
 kernel: SRAT: CPU 7 has memory domain 1
 kernel: random: read 4096 bytes from preloaded cache
 kernel: random: unblocking device.
 kernel: VIMAGE (virtualized network stack) enabled
 kernel: ULE: setup cpu 0
 kernel: ULE: setup cpu 1
 kernel: ULE: setup cpu 2
 kernel: ULE: setup cpu 3
 kernel: ULE: setup cpu 4
 kernel: ULE: setup cpu 5
 kernel: ULE: setup cpu 6
 kernel: ULE: setup cpu 7
 kernel: ACPI: RSDP 0x000000003E7FA014 000024 (v02 VRTUAL)
 kernel: ACPI: XSDT 0x000000003E7F90E8 00005C (v01 VRTUAL MICROSFT 00000001 MSFT 00000001)
 kernel: ACPI: FACP 0x000000003E7F7000 000114 (v06 VRTUAL MICROSFT 00000001 MSFT 00000001)
 kernel: ACPI: DSDT 0x000000003D6EA000 0078C1 (v02 MSFTVM DSDT01   00000001 MSFT 05000000)
 kernel: ACPI: FACS 0x000000003E7FE000 000040
 kernel: ACPI: APIC 0x000000003E7F8000 0007D2 (v02 VRTUAL MICROSFT 00000001 MSFT 00000001)
 kernel: ACPI: OEM0 0x000000003E7F6000 000064 (v01 VRTUAL MICROSFT 00000001 MSFT 00000001)
 kernel: ACPI: WAET 0x000000003E7F5000 000028 (v01 VRTUAL MICROSFT 00000001 MSFT 00000001)
 kernel: ACPI: SRAT 0x000000003E7F4000 000330 (v02 VRTUAL MICROSFT 00000001 MSFT 00000001)
 kernel: ACPI: FPDT 0x000000003E7F3000 000034 (v01 VRTUAL MICROSFT 00000001 MSFT 00000001)
 kernel: ACPI: BGRT 0x000000003D601000 000038 (v01 VRTUAL MICROSFT 00000001 MSFT 00000001)
 kernel: MADT: Found IO APIC ID 8, Interrupt 0 at 0xfec00000
 kernel: ioapic0: ver 0x11 maxredir 0x17
 kernel: ioapic0: Routing external 8259A's -> intpin 0
 kernel: lapic: Routing NMI -> LINT1
 kernel: lapic: LINT1 trigger: edge
 kernel: lapic: LINT1 polarity: high
 kernel: MADT: Interrupt override: source 0, irq 2
 kernel: ioapic0: Routing IRQ 0 -> intpin 2
 kernel: MADT: Interrupt override: source 9, irq 9
 kernel: ioapic0: intpin 9 trigger: level
 kernel: ioapic0 <Version 1.1> irqs 0-23
 kernel: lapic: Divisor 2, Frequency 100001390 Hz
 kernel: cpu0 BSP:
 kernel:      ID: 0x00000000   VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff
 kernel:   lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
 kernel:   timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
 kernel: SMP: AP CPU #1 Launched!
 kernel: cpu1 AP:
 kernel:      ID: 0x01000000   VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff
 kernel:   lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
 kernel:   timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
 kernel: SMP: AP CPU #7 Launched!
 kernel: cpu7 AP:
 kernel:      ID: 0x07000000   VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff
 kernel:   lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
 kernel:   timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
 kernel: SMP: AP CPU #6 Launched!
 kernel: cpu6 AP:
 kernel:      ID: 0x06000000   VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff
 kernel:   lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
 kernel:   timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
 kernel: SMP: AP CPU #3 Launched!
 kernel: cpu3 AP:
 kernel:      ID: 0x03000000   VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff
 kernel:   lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
 kernel:   timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
 kernel: SMP: AP CPU #4 Launched!
 kernel: cpu4 AP:
 kernel:      ID: 0x04000000   VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff
 kernel:   lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
 kernel:   timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
 kernel: SMP: AP CPU #5 Launched!
 kernel: cpu5 AP:
 kernel:      ID: 0x05000000   VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff
 kernel:   lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
 kernel:   timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
 kernel: SMP: AP CPU #2 Launched!
 kernel: cpu2 AP:
 kernel:      ID: 0x02000000   VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff
 kernel:   lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
 kernel:   timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
 kernel: TSC timecounter disables C2 and C3.
 kernel: TSC timecounter discards lower 1 bit(s)
 kernel: Timecounter "TSC-low" frequency 1147238669 Hz quality -100
 kernel: TCP_ratelimit: Is now initialized
 kernel: hyperv: Hypercall created
 kernel: Timecounter "Hyper-V-TSC" frequency 10000000 Hz quality 3000
 kernel: random: entropy device external interface
 kernel: wlan: <802.11 Link Layer>
 kernel: snd_unit_init() u=0x00ff8000 [512] d=0x00007c00 [32] c=0x000003ff [1024]
 kernel: feeder_register: snd_unit=-1 snd_maxautovchans=16 latency=2 feeder_rate_min=1 feeder_rate_max=2016000 feeder_rate_round=25
 kernel: nfslock: pseudo-device
 kernel: WARNING: Device "kbd" is Giant locked and may be deleted before FreeBSD 13.0.
 kernel: kbd0 at kbdmux0
 kernel: crypto: <crypto core>
 kernel: mem: <memory>
 kernel: 000.000046 [4336] netmap_init               netmap: loaded module
 kernel: null: <full device, null device, zero device>
 kernel: cpuctl: access to MSR registers/cpuid info.
 kernel: io: <I/O>
 kernel: [ath_hal] loaded
 kernel: tcp_log: tcp_log device
 kernel: mlx5en: Mellanox Ethernet driver 3.5.2 (September 2019)
 kernel: nexus0
 kernel: efirtc0: <EFI Realtime Clock>
 kernel: efirtc0: registered as a time-of-day clock, resolution 1.000000s
 kernel: cryptosoft0: <software crypto>
 kernel: crypto: assign cryptosoft0 driver id 0, flags 0x6000000
 kernel: crypto: cryptosoft0 registers alg 1 flags 0 maxoplen 0
 kernel: crypto: cryptosoft0 registers alg 2 flags 0 maxoplen 0
. . . ---//--- . . .
 kernel: crypto: cryptosoft0 registers alg 39 flags 0 maxoplen 0
 kernel: crypto: cryptosoft0 registers alg 38 flags 0 maxoplen 0
 kernel: aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS>
 kernel: crypto: assign aesni0 driver id 1, flags 0x5000000
 kernel: crypto: aesni0 registers alg 11 flags 0 maxoplen 0
 kernel: crypto: aesni0 registers alg 23 flags 0 maxoplen 0
 kernel: crypto: aesni0 registers alg 25 flags 0 maxoplen 0
 kernel: crypto: aesni0 registers alg 26 flags 0 maxoplen 0
 kernel: crypto: aesni0 registers alg 27 flags 0 maxoplen 0
 kernel: crypto: aesni0 registers alg 28 flags 0 maxoplen 0
 kernel: crypto: aesni0 registers alg 22 flags 0 maxoplen 0
 kernel: crypto: aesni0 registers alg 40 flags 0 maxoplen 0
 kernel: crypto: aesni0 registers alg 39 flags 0 maxoplen 0
 kernel: acpi0: <VRTUAL MICROSFT>
 kernel: ACPI: 1 ACPI AML tables successfully acquired and loaded
 kernel: ioapic0: routing intpin 9 (ISA IRQ 9) to lapic 0 vector 48
 kernel: acpi0: wakeup code va 0xfffffe0003a2a000 pa 0x9e000
 kernel: Table 'FACP' at 0x3e7f7000
 kernel: FACP: Found table at 0x3e7f7000
 kernel: atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
 kernel: atrtc0: registered as a time-of-day clock, resolution 1.000000s
 kernel: ioapic0: routing intpin 8 (ISA IRQ 8) to lapic 0 vector 49
 kernel: Event timer "RTC" frequency 32768 Hz quality 0
 kernel: ACPI timer: 1/2 1/2 1/49 1/2 1/48 1/2 1/13 1/2 1/2 1/2 -> 10
 kernel: Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
 kernel: acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
 kernel: Table 'FACP' at 0x3e7f7000
 kernel: FACP: Found table at 0x3e7f7000
 kernel: cpu0: Processor \P001 (ACPI ID 1) -> APIC ID 0
 kernel: cpu0: <ACPI CPU> on acpi0
 kernel: cpu0: switching to generic Cx mode
 kernel: Table 'FACP' at 0x3e7f7000
 kernel: FACP: Found table at 0x3e7f7000
 kernel: cpu1: Processor \P002 (ACPI ID 2) -> APIC ID 1
 kernel: cpu1: <ACPI CPU> on acpi0
 kernel: Table 'FACP' at 0x3e7f7000
 kernel: FACP: Found table at 0x3e7f7000
 kernel: cpu2: Processor \P003 (ACPI ID 3) -> APIC ID 2
 kernel: cpu2: <ACPI CPU> on acpi0
 kernel: Table 'FACP' at 0x3e7f7000
 kernel: FACP: Found table at 0x3e7f7000
 kernel: cpu3: Processor \P004 (ACPI ID 4) -> APIC ID 3
 kernel: cpu3: <ACPI CPU> on acpi0
 kernel: Table 'FACP' at 0x3e7f7000
 kernel: FACP: Found table at 0x3e7f7000
 kernel: cpu4: Processor \P005 (ACPI ID 5) -> APIC ID 4
 kernel: cpu4: <ACPI CPU> on acpi0
 kernel: Table 'FACP' at 0x3e7f7000
 kernel: FACP: Found table at 0x3e7f7000
 kernel: cpu5: Processor \P006 (ACPI ID 6) -> APIC ID 5
 kernel: cpu5: <ACPI CPU> on acpi0
 kernel: Table 'FACP' at 0x3e7f7000
 kernel: FACP: Found table at 0x3e7f7000
 kernel: cpu6: Processor \P007 (ACPI ID 7) -> APIC ID 6
 kernel: cpu6: <ACPI CPU> on acpi0
 kernel: Table 'FACP' at 0x3e7f7000
 kernel: FACP: Found table at 0x3e7f7000
 kernel: cpu7: Processor \P008 (ACPI ID 8) -> APIC ID 7
 kernel: cpu7: <ACPI CPU> on acpi0
 kernel: Table 'FACP' at 0x3e7f7000
 kernel: FACP: Found table at 0x3e7f7000
 kernel: acpi_syscontainer0: <System Container> on acpi0
 kernel: vmbus0: <Hyper-V Vmbus> on acpi_syscontainer0
 kernel: Table 'FACP' at 0x3e7f7000
 kernel: FACP: Found table at 0x3e7f7000
 kernel: Table 'FACP' at 0x3e7f7000
 kernel: FACP: Found table at 0x3e7f7000
 kernel: Table 'FACP' at 0x3e7f7000
 kernel: FACP: Found table at 0x3e7f7000
 kernel: vmbus_res0: <Hyper-V Vmbus Resource> irq 5 on acpi0
 kernel: ACPI: Enabled 1 GPEs in block 00 to 0F
 kernel: Device configuration finished.
 kernel: procfs registered
 kernel: ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
 kernel:             to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
 kernel: ZFS filesystem version: 5
 kernel: ZFS storage pool version: features support (5000)
 kernel: Timecounters tick every 1.000 msec
 kernel: crypto: <crypto device>
 kernel: WARNING: Device "crypto" is Giant locked and may be deleted before FreeBSD 13.0.
 kernel: vlan: initialized, using hash tables with chaining
 kernel: lo0: bpf attached
 kernel: enc0: bpf attached
 kernel: IPsec: Initialized Security Association Processing.
 kernel: tcp_init: net.inet.tcp.tcbhashsize auto tuned to 16384
 kernel: ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to deny, logging disabled
 kernel: DUMMYNET 0xfffff8003f0292c0 with IPv6 initialized (100409)
 kernel: load_dn_sched dn_sched FQ_PIE loaded
 kernel: pflog0: bpf attached
 kernel: load_dn_sched dn_sched PRIO loaded
 kernel: load_dn_sched dn_sched QFQ loaded
 kernel: pfsync0: bpf attached
 kernel: load_dn_sched dn_sched RR loaded
 kernel: load_dn_sched dn_sched WF2Q+ loaded
 kernel: load_dn_sched dn_sched FIFO loaded
 kernel: load_dn_sched dn_sched FQ_CODEL loaded
 kernel: load_dn_aqm dn_aqm CODEL loaded
 kernel: load_dn_aqm dn_aqm PIE loaded
 kernel: vmbus0: intrhook
 kernel: vmbus_res0: walking _CRS, pass=0
 kernel: acpi0: walking _CRS, pass=0
 kernel: acpi0: _CRS: not found, pass=0
 kernel: acpi_syscontainer0: walking _CRS, pass=0
 kernel: vmbus0: decoding 3 range 0xfe0000000-0xfffffffff
 kernel: vmbus_res0: walking _CRS, pass=1
 kernel: acpi0: walking _CRS, pass=1
 kernel: acpi0: _CRS: not found, pass=1
 kernel: acpi_syscontainer0: walking _CRS, pass=1
 kernel: vmbus0: decoding 3 range 0xf8000000-0xfed3ffff
 kernel: vmbus0: vmbus IDT vector 251
 kernel: vmbus0: smp_started = 1
 kernel: vmbus0: version 4.0
 kernel: hvet0: <Hyper-V event timer> on vmbus0
 kernel: Event timer "Hyper-V" frequency 10000000 Hz quality 1000
 kernel: vmbus0: chan2 subidx0 offer
 kernel: vmbus0: chan2 assigned to cpu0 [vcpu0]
 kernel: vmbus0: chan3 subidx0 offer
 kernel: vmbus0: chan3 assigned to cpu0 [vcpu0]
 kernel: vmbus0: chan4 subidx0 offer
 kernel: vmbus0: chan4 assigned to cpu0 [vcpu0]
 kernel: vmbus0: chan5 subidx0 offer
 kernel: hvkbd0: <Hyper-V KBD>vmbus0: chan5 assigned to cpu0 [vcpu0]
 kernel: vmbus0:  on vmbus0
 kernel: chan7 subidx0 offer
 kernel: hvkbd0: vmbus0: chan7 assigned to cpu0 [vcpu0]
 kernel: vmbus0: chan4 update cpu0 flag_cnt to 1
 kernel: chan8 subidx0 offer
 kernel: vmbus0: chan8 assigned to cpu0 [vcpu0]
 kernel: vmbus0: chan9 subidx0 offer
 kernel: vmbus0: chan9 assigned to cpu0 [vcpu0]
 kernel: vmbus0: chan10 subidx0 offer
 kernel: vmbus0: chan10 assigned to cpu0 [vcpu0]
 kernel: vmbus0: chan11 subidx0 offer
 kernel: vmbus0: chan11 assigned to cpu0 [vcpu0]
 kernel: vmbus0: chan14 subidx0 offer
 kernel: vmbus0: chan14 assigned to cpu0 [vcpu0]
 kernel: vmbus0: chan1 subidx0 offer
 kernel: vmbus0: chan1 assigned to cpu0 [vcpu0]
 kernel: vmbus0: chan6 subidx0 offer
 kernel: vmbus0: chan6 assigned to cpu0 [vcpu0]
 kernel: vmbus0: chan12 subidx0 offer
 kernel: vmbus0: chan12 assigned to cpu0 [vcpu0]
 kernel: vmbus0: chan13 subidx0 offer
 kernel: vmbus0: chan13 assigned to cpu0 [vcpu0]
 kernel: hvkbd0: gpadl_conn(chan4) succeeded
 kernel: hvkbd0: chan4 opened
 kernel: kbd: new array size 4
 kernel: kbd1: hvkbd0, generic (0), config:0x0, flags:0x3d0000
 kernel: hvheartbeat0: <Hyper-V Heartbeat> on vmbus0
 kernel: hvheartbeat0: gpadl_conn(chan7) succeeded
 kernel: hvheartbeat0: sel framework version: 3.0
 kernel: hvheartbeat0: hvheartbeat0: supp framework version: 1.0
 kernel: chan7 opened
 kernel: hvkvp0: <Hyper-V KVP>hvheartbeat0: supp framework version: 3.0
 kernel:  on vmbus0
 kernel: hvheartbeat0: sel message version: 3.0
 kernel: hvheartbeat0: supp message version: 3.0
 kernel: hvheartbeat0: supp message version: 3.1
 kernel: hvkvp0: gpadl_conn(chan8) succeeded
 kernel: hvkvp0: chan8 opened
 kernel: hvshutdown0: <Hyper-V Shutdown> on vmbus0
 kernel: hvshutdown0: gpadl_conn(chan9) succeeded
 kernel: hvshutdown0: chan9 opened
 kernel: hvshutdown0: sel framework version: 3.0
 kernel: hvtimesync0: <Hyper-V Timesync>hvshutdown0: supp framework version: 1.0
 kernel:  on vmbus0
 kernel: hvshutdown0: supp framework version: 3.0
 kernel: hvshutdown0: sel message version: 3.0
 kernel: hvshutdown0: supp message version: 1.0
 kernel: hvshutdown0: supp message version: 3.0
 kernel: hvshutdown0: supp message version: 3.1
 kernel: hvshutdown0: supp message version: 3.2
 kernel: hvtimesync0: gpadl_conn(chan10) succeeded
 kernel: hvtimesync0: chan10 opened
 kernel: hvtimesync0: sel framework version: 3.0
 kernel: hvvss0: <Hyper-V VSS>hvtimesync0: supp framework version: 1.0
 kernel:  on vmbus0
 kernel: hvtimesync0: supp framework version: 3.0
 kernel: hvtimesync0: sel message version: 4.0
 kernel: hvtimesync0: supp message version: 1.0
 kernel: hvtimesync0: supp message version: 3.0
 kernel: hvtimesync0: supp message version: 4.0
 kernel: hvtimesync0: RTT
 kernel: hvvss0: gpadl_conn(chan11) succeeded
 kernel: hvtimesync0: apply sync request, hv: 1577353266503230600, vm: 1901363599
 kernel: hvvss0: chan11 opened
 kernel: storvsc0: Enlightened SCSI device detected
 syslogd: last message repeated 1 times
 kernel: storvsc0: <Hyper-V SCSI> on vmbus0
 kernel: storvsc ringbuffer size: 262144, max_io: 512
 kernel: storvsc0: chan14 assigned to cpu0 [vcpu0]
 kernel: storvsc0: gpadl_conn(chan14) succeeded
 kernel: storvsc0: chan14 opened
 kernel: storvsc0: max chans 2, multi-chan capable
 kernel: storvsc0: chan15 subidx1 offer
 kernel: storvsc0: chan15 assigned to cpu0 [vcpu0]
 kernel: storvsc0: chan15 assigned to cpu1 [vcpu1]
 kernel: storvsc0: chan15 update cpu1 flag_cnt to 1
 kernel: storvsc0: gpadl_conn(chan15) succeeded
 kernel: storvsc0: chan15 opened
 kernel: Storvsc create multi-channel success!
 kernel: (probe0:storvsc0:0:0:0): Down reving Protocol Version from 4 to 2?
 kernel: hn0: <Hyper-V Network Interface>(probe1:storvsc0:0:1:0): Down reving Protocol Version from 4 to 2?
 kernel:  on vmbus0
 kernel: (probe0:storvsc0:0:0:0): storvsc inquiry (36) [0 0 5 2 1f ... ]
 kernel: (probe1:storvsc0:0:1:0): storvsc inquiry (36) [7f 0 5 2 1f ... ]
 kernel: (probe1:storvsc0:0:1:1): Down reving Protocol Version from 4 to 2?
 kernel: (probe0:storvsc0:0:0:0): storvsc scsi_status = 2, srb_status = 6
 kernel: hn0: LRO: entry count 128
 kernel: (probe0:storvsc0:0:0:0): storvsc inquiry (5) [0 0 0 8 0 ... ]
 kernel: (probe1:storvsc0:0:1:1): invalid LUN 1 for op: INQUIRY
 kernel: hn0: link RX ring 0 to chan13
 kernel: hn0: link TX ring 0 to chan13
 kernel: (probe0:storvsc0:0:0:0): storvsc inquiry (6) [0 83 0 30 1 ... ]
 kernel: hn0: chan13 assigned to cpu0 [vcpu0]
 kernel: hn0: gpadl_conn(chan13) succeeded
 kernel: (probe0:storvsc0:0:0:1): storvsc inquiry (36) [5 80 0 2 1f ... ]
 kernel: (probe0:storvsc0:0:0:1): set version from 0 to 5
 kernel: (probe0:storvsc0:0:0:1): storvsc inquiry (5) [5 0 0 2 0 ... ]
 kernel: hn0: (probe0:storvsc0:0:0:1): set version from 0 to 5
 kernel: chan13 opened
 kernel: hn0: NVS version 0x50000, NDIS version 6.30
 kernel: (probe0:storvsc0:0:0:1): storvsc inquiry (6) [5 83 0 1c 1 ... ]
 kernel: hn0: nvs ndis conf done
 kernel: (probe0:storvsc0:0:0:1): set version from 0 to 5
 kernel: hn0: gpadl_conn(chan13) succeeded
 syslogd: last message repeated 1 times
 kernel: hn0: chimney sending buffer 6144/2560
 kernel: hn0: RNDIS ver 1.0, aggpkt size 4026531839, aggpkt cnt 8, aggpkt align 8
 kernel: hn0: hwcaps rev 3
 kernel: hn0: hwcaps csum: ip4 tx 0x155/0x2 rx 0x155/0x2, ip6 tx 0x55/0x2 rx 0x55/0x2
 kernel: hn0: hwcaps lsov2: ip4 maxsz 62780 minsg 2 encap 0x2, ip6 maxsz 62780 minsg 2 encap 0x2 opts 0x5
 kernel: hn0: NDIS TSO szmax 62780 sgmin 2
 kernel: hn0: offload csum: ip4 4, tcp4 4, udp4 4, tcp6 4, udp6 4
 kernel: hn0: offload lsov2: ip4 2, ip6 2
 kernel: hn0: offload config done
 kernel: hn0: 64 RX rings
 kernel: hn0: RSS indirect table size 128
 kernel: hn0: RSS caps 0x301
 kernel: hn0: RX rings offered 64, requested 8
 kernel: hn0: chan16 subidx1 offer
 kernel: hn0: chan16 assigned to cpu0 [vcpu0]
 kernel: hn0: chan17 subidx2 offer
 kernel: hn0: chan17 assigned to cpu0 [vcpu0]
 kernel: hn0: chan18 subidx3 offer
 kernel: hn0: chan18 assigned to cpu0 [vcpu0]
 kernel: hn0: chan19 subidx4 offer
 kernel: hn0: chan19 assigned to cpu0 [vcpu0]
 kernel: hn0: chan20 subidx5 offer
 kernel: hn0: chan20 assigned to cpu0 [vcpu0]
 kernel: hn0: chan21 subidx6 offer
 kernel: hn0: chan21 assigned to cpu0 [vcpu0]
 kernel: hn0: chan22 subidx7 offer
 kernel: hn0: chan22 assigned to cpu0 [vcpu0]
 kernel: vmbus0: chan23 subidx0 offer
 kernel: vmbus0: chan23 assigned to cpu0 [vcpu0]
 kernel: hn0: got notify, nvs type 128
 kernel: hn0: 8 TX ring, 8 RX ring
 kernel: hn0: link RX ring 1 to chan16
 kernel: hn0: link TX ring 1 to chan16
 kernel: hn0: chan16 assigned to cpu1 [vcpu1]
 kernel: hn0: gpadl_conn(chan16) succeeded
 kernel: hn0: chan16 opened
 kernel: hn0: link RX ring 2 to chan17
 kernel: hn0: link TX ring 2 to chan17
 kernel: hn0: chan17 assigned to cpu2 [vcpu2]
 kernel: hn0: chan17 update cpu2 flag_cnt to 1
 kernel: hn0: gpadl_conn(chan17) succeeded
 kernel: hn0: chan17 opened
 kernel: hn0: link RX ring 3 to chan18
 kernel: hn0: link TX ring 3 to chan18
 kernel: hn0: chan18 assigned to cpu3 [vcpu3]
 kernel: hn0: chan18 update cpu3 flag_cnt to 1
 kernel: hn0: gpadl_conn(chan18) succeeded
 kernel: hn0: chan18 opened
 kernel: hn0: link RX ring 4 to chan19
 kernel: hn0: link TX ring 4 to chan19
 kernel: hn0: chan19 assigned to cpu4 [vcpu4]
 kernel: hn0: chan19 update cpu4 flag_cnt to 1
 kernel: hn0: gpadl_conn(chan19) succeeded
 kernel: hn0: chan19 opened
 kernel: hn0: link RX ring 5 to chan20
 kernel: hn0: link TX ring 5 to chan20
 kernel: hn0: chan20 assigned to cpu5 [vcpu5]
 kernel: hn0: chan20 update cpu5 flag_cnt to 1
 kernel: hn0: gpadl_conn(chan20) succeeded
 kernel: hn0: chan20 opened
 kernel: hn0: link RX ring 6 to chan21
 kernel: hn0: link TX ring 6 to chan21
 kernel: hn0: chan21 assigned to cpu6 [vcpu6]
 kernel: hn0: chan21 update cpu6 flag_cnt to 1
 kernel: hn0: gpadl_conn(chan21) succeeded
 kernel: hn0: chan21 opened
 kernel: hn0: link RX ring 7 to chan22
 kernel: hn0: link TX ring 7 to chan22
 kernel: hn0: chan22 assigned to cpu7 [vcpu7]
 kernel: hn0: chan22 update cpu7 flag_cnt to 1
 kernel: hn0: gpadl_conn(chan22) succeeded
 kernel: hn0: chan22 opened
 kernel: hn0: 7 sub-channels attached
 kernel: hn0: setup default RSS key
 kernel: hn0: setup default RSS indirect table
 kernel: hn0: RSS indirect table size 128, hash 0x00005701
 kernel: hn0: RSS config done
 kernel: hn0: TX agg size 6144, pkts 8, align 8
 kernel: hn0: set RX filter 0x00000000 done
 kernel: hn0: RNDIS mtu 1500
 kernel: hn0: support HASHVAL pktinfo
 kernel: hn0: TSO size max 62762
 kernel: hn0: bpf attached
 kernel: hn0: Ethernet address: 00:15:5d:00:f0:8a
 kernel: hn0: TSO segcnt 31 segsz 4096
 kernel: hn0: link state changed to UP
 kernel: pcib0: <Hyper-V PCI Express Pass Through> on vmbus0
 kernel: vmbus0: allocated type 3 (0xfe0000000-0xfe0001fff) for rid 0 of pcib0
 kernel: vmbus0: device scan, probe and attach done
 kernel: GEOM: new disk da0
 kernel: pass0 at storvsc0 bus 0 scbus0 target 0 lun 0
 kernel: pass0: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-3 SCSI device
 kernel: pass0: 300.000MB/s transfers
 kernel: pass0: Command Queueing enabled
 kernel: pass1 at storvsc0 bus 0 scbus0 target 0 lun 1
 kernel: pass1: <Msft Virtual DVD-ROM 1.0> Removable CD-ROM SPC-3 SCSI device
 kernel: pass1: 300.000MB/s transfers
 kernel: (cd0:storvsc0:0:0:1): storvsc scsi_status = 2, srb_status = 4
 kernel: cd0 at storvsc0 bus 0 scbus0 target 0 lun 1
 kernel: cd0: <Msft Virtual DVD-ROM 1.0> Removable CD-ROM SPC-3 SCSI device
 kernel: cd0: 300.000MB/s transfers
 kernel: cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed
 kernel: da0 at storvsc0 bus 0 scbus0 target 0 lun 0
 kernel: da0: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-3 SCSI device
 kernel: da0: 300.000MB/s transfers
 kernel: da0: Command Queueing enabled
 kernel: da0: 81920MB (167772160 512 byte sectors)
 kernel: (da0:storvsc0:0:0:0): storvsc inquiry (6) [0 b2 0 4 1 ... ]
 kernel: (da0:storvsc0:0:0:0): storvsc inquiry (5) [0 b0 0 3c 0 ... ]
 kernel: (da0:storvsc0:0:0:0): storvsc inquiry (5) [0 b1 0 3c 0 ... ]
 kernel: da0: Delete methods: <UNMAP(*),ZERO>
 kernel: Trying to mount root from zfs:zroot/ROOT/default []...
 kernel: pcib0: gpadl_conn(chan23) succeeded
 kernel: pcib0: chan23 opened
 kernel: pci0: <PCI bus> on pcib0
 kernel: pci0: domain=1, physical bus=0
 kernel: GEOM: new disk cd0
 kernel: found->	vendor=0x15b3, dev=0x1016, revid=0x80
 kernel: 	domain=1, bus=0, slot=2, func=0
 kernel: 	class=02-00-00, hdrtype=0x00, mfdev=0
 kernel: 	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
 kernel: 	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
 kernel: 	MSI-X supports 8 messages in map 0x10
 kernel: 	map[10]: type Prefetchable Memory, range 64, base r, size 20, memory disabled
 kernel: mlx5_core0: <mlx5_core> at device 2.0 on pci0
 kernel: vmbus0: allocated type 3 (0xfe0100000-0xfe01fffff) for rid 10 of mlx5_core0
 kernel: mlx5_core0: Lazy allocation of 0x100000 bytes rid 0x10 type 3 at 0xfe0100000
 kernel: (cd0:storvsc0:0:0:1): storvsc scsi_status = 2, srb_status = 4
 syslogd: last message repeated 4 times
 kernel: mlx5_core0: ERR: mlx5_init_once:918:(pid 0): Unable to find vendor specific capabilities
 kernel: mlx5_core0: attempting to allocate 8 MSI-X vectors (8 supported)
 kernel: msi: routing MSI-X IRQ 24 to local APIC 0 vector 50
 kernel: msi: routing MSI-X IRQ 25 to local APIC 2 vector 48
 kernel: msi: routing MSI-X IRQ 26 to local APIC 0 vector 51
 kernel: msi: routing MSI-X IRQ 27 to local APIC 2 vector 49
 kernel: msi: routing MSI-X IRQ 28 to local APIC 0 vector 52
 kernel: msi: routing MSI-X IRQ 29 to local APIC 2 vector 50
 kernel: msi: routing MSI-X IRQ 30 to local APIC 0 vector 53
 kernel: msi: routing MSI-X IRQ 31 to local APIC 2 vector 51
 kernel: mlx5_core0: using IRQs 24-31 for MSI-X
 kernel: mce0: bpf attached
 kernel: mce0: Ethernet address: 00:15:5d:00:f0:8a
 kernel: mce0: link state changed to DOWN
 kernel: hn0: link state changed to DOWN
 kernel: mlx5_core0: WARN: mlx5_fwdump_prep:76:(pid 0): Failed to find vendor-specific capability, error 2
 kernel: efirtc0: providing initial system time
 kernel: start_init: trying /sbin/init
 kernel: mce0: ERR: mlx5e_ioctl:3320:(pid 0): tso6 disabled due to -txcsum6.
 kernel: mce0: ERR: mlx5e_ioctl:3346:(pid 0): enable txcsum6 first.
 kernel: lo0: link state changed to UP
 kernel: hn0: set RX filter 0x00000009 done
 kernel: hn0: try bringing up mce0
 kernel: mce0: link state changed to UP
 kernel: hn0: link state changed to UP
 kernel: hn0: mce0 SIOCGRSSKEY failed: 22
 kernel: hn0: set RX filter 0x0000000d done
 kernel: hvtimesync0: apply sample request, hv: 1577353271520633700, vm: 1577353272415602299
 kernel: hvkvp0: sel framework version: 3.0
 kernel: hvkvp0: supp framework version: 1.0
 kernel: hvkvp0: supp framework version: 3.0
 kernel: hvkvp0: sel message version: 4.0
 kernel: hvkvp0: supp message version: 3.0
 kernel: hvkvp0: supp message version: 4.0
 kernel: hvkvp0: supp message version: 5.0
 kernel: hvvss0: sel framework version: 3.0
 kernel: hvvss0: supp framework version: 1.0
 kernel: hvvss0: supp framework version: 3.0
 kernel: hvvss0: sel message version: 5.0
 kernel: hvvss0: supp message version: 5.0
 kernel: hvvss0: supp message version: 6.0
 kernel: hvvss0: supp message version: 7.0
Comment 1 Hans Petter Selasky freebsd_committer freebsd_triage 2019-12-27 00:22:35 UTC 
This error should go away if you upgrade the firmware I believe.

--HPS
Comment 2 Michael 2019-12-28 07:56:32 UTC 
(In reply to Hans Petter Selasky from comment #1)

https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_2b1b10070f40416cbc3f997ff0#tab4
Version: 1.0.0.11(20 Dec 2019)
FW version 14.26.1040

This is the newest firmware I have already installed.
Comment 3 Konstantin Belousov freebsd_committer freebsd_triage 2019-12-28 15:19:52 UTC 
(In reply to Hans Petter Selasky from comment #1)
I suspect the error would not go away.  Am I right that the device is in pass-through mode on Hyper-V, and FreeBSD runs inside a VM ?

If yes, then most likely hypervisor (rightfully) filters out unknown capabilities.
Comment 4 Michael 2019-12-29 19:33:26 UTC 
(In reply to Konstantin Belousov from comment #3)
Yes, FreeBSD is truly virtualized in Hyper-V. Without this, I would go broke on servers and electricity.
Well, maybe the specifics of the vendor are not scary that they are filtered out, but I would like to hear the opinion "does this pose a problem?" or I can ignore this, especially this message
  hn0: mce0 SIOCGRSSKEY failed: 22
Comment 5 Michael 2019-12-29 20:26:05 UTC 
By the way, there are no errors in virtualized Windows systems.
Later I will look at virtualized Linux for such errors.
Comment 6 commit-hook freebsd_committer freebsd_triage 2019-12-30 09:23:55 UTC 
A commit references this bug:

Author: hselasky
Date: Mon Dec 30 09:22:53 UTC 2019
New revision: 356201
URL: https://svnweb.freebsd.org/changeset/base/356201

Log:
  Fix spelling.

  PR:		242891
  MFC after:	1 week
  Sponsored by:	Mellanox Technologies

Changes:
  head/sys/dev/hyperv/netvsc/if_hn.c
Comment 7 Michael 2019-12-30 13:31:08 UTC 
Maybe this can clarify the problem. Messages of mlx5 in the Windows system log:

HPE Ethernet 10/25Gb 2-port 640FLR-SFP28 Adapter #2 has got:
    vendor_id      	15b3
    device_id      	1016
    subvendor_id   	1590
    subsystem_id   	00d3
    HW revision    	80
    FW version     	14.26.1040
    port type      	ETH

HPE Ethernet 10/25Gb 2-port 640FLR-SFP28 Adapter #2: Zero Touch RoCE: Some of the required capabilities are not supported by FW (Required 7 Supported 5).

HPE Ethernet 10/25Gb 2-port 640FLR-SFP28 Adapter #2: Zero Touch RoCE is supported. Current configuration is:
    Slow restart 0
    TX window    0
    Adp Retrans  0.

HPE Ethernet 10/25Gb 2-port 640FLR-SFP28 Adapter #2: Zero Touch RoCE: New configuration is set:
    Slow restart 1
    TX window    0
    Adp Retrans  1.

HPE Ethernet 10/25Gb 2-port 640FLR-SFP28 Adapter #2 device detects that the link is up, and has initiated a normal operation.

HPE Ethernet 10/25Gb 2-port 640FLR-SFP28 Adapter #2: is currently running:
     GUID:                 	0015:5dff:fd00:f100
     MAC:                  	00-15-5d-00-f1-00

HPE Ethernet 10/25Gb 2-port 640FLR-SFP28 Adapter #2: configuration updated: value ZeroTouchRoceCfgMask changed from 5 to 7.

HPE Ethernet 10/25Gb 2-port 640FLR-SFP28 Adapter #2: is currently running:
     Driver Version:       	2.30.21713.0
     Firmware Version:     	14.26.1040
     PSID number:          	HP_2690110034

Eventid 46; Hyper-V-Netvsc; Message:
NDK function message not received from VF adapter "\DEVICE\{CC634541-26A7-4DF0-B2D1-A4025E8351FE}".

In Hyper-V log message are:
Eventid 102; Microsoft-Windows-Hyper-V-VmSwitch; Message:
Networking driver in VM1 is loaded and the protocol version is negotiated to the most recent version (Virtual machine ID 9B4A1702-666D-40BE-8E83-C280326BF300).
Comment 8 commit-hook freebsd_committer freebsd_triage 2020-01-06 09:51:18 UTC 
A commit references this bug:

Author: hselasky
Date: Mon Jan  6 09:50:44 UTC 2020
New revision: 356410
URL: https://svnweb.freebsd.org/changeset/base/356410

Log:
  MFC r356201:
  Fix spelling.

  PR:		242891
  Sponsored by:	Mellanox Technologies

Changes:
_U  stable/12/
  stable/12/sys/dev/hyperv/netvsc/if_hn.c
Comment 9 commit-hook freebsd_committer freebsd_triage 2020-01-06 09:52:20 UTC 
A commit references this bug:

Author: hselasky
Date: Mon Jan  6 09:51:22 UTC 2020
New revision: 356411
URL: https://svnweb.freebsd.org/changeset/base/356411

Log:
  MFC r356201:
  Fix spelling.

  PR:		242891
  Sponsored by:	Mellanox Technologies

Changes:
_U  stable/11/
  stable/11/sys/dev/hyperv/netvsc/if_hn.c
Comment 10 commit-hook freebsd_committer freebsd_triage 2020-01-06 09:52:22 UTC 
A commit references this bug:

Author: hselasky
Date: Mon Jan  6 09:51:55 UTC 2020
New revision: 356412
URL: https://svnweb.freebsd.org/changeset/base/356412

Log:
  MFC r356201:
  Fix spelling.

  PR:		242891
  Sponsored by:	Mellanox Technologies

Changes:
_U  stable/10/
  stable/10/sys/dev/hyperv/netvsc/if_hn.c
Comment 11 Michael 2020-01-06 14:59:15 UTC 
I did not find a more informative output of messages from different Linux. Everyone has a system log as some sort of chaotic nonsense. CentOS v8 garbage collection of messages when adding a network interface with SR-IOV enabled:

/var/log/messages :

 nm-dispatcher[40509]: req:1 'dhcp4-change' [eth0]: new request (3 scripts)
 nm-dispatcher[40509]: req:1 'dhcp4-change' [eth0]: start running ordered scripts...
 systemd-udevd[40520]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
 NetworkManager[996]: <info>  [1578320647.9579] manager: (eth1): new Ethernet device (/org/freedesktop/NetworkManager/Devices/5)
 NetworkManager[996]: <info>  [1578320647.9795] device (eth1): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
 kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
 kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
 NetworkManager[996]: <info>  [1578320647.9954] device (eth1): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'managed')
 kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
 NetworkManager[996]: <info>  [1578320648.1653] device (eth1): carrier: link connected
 kernel: hv_netvsc cc91b9de-cf8f-4729-8e30-bc4cf0e3f604 eth1: VF slot 2 added
 kernel: hv_vmbus: registering driver hv_pci
 kernel: hv_pci c2f22477-5994-443d-ae4b-7012efabe867: PCI VMBus probing: Using version 0x10002
 kernel: hv_pci c2f22477-5994-443d-ae4b-7012efabe867: PCI host bridge to bus ae4b:00
 kernel: pci_bus ae4b:00: root bus resource [mem 0xfe0000000-0xfe00fffff window]
 kernel: pci ae4b:00:02.0: enabling Extended Tags
 kernel: pci ae4b:00:02.0: 0.000 Gb/s available PCIe bandwidth, limited by Unknown speed x0 link at ae4b:00:02.0 (capable of 63.008 Gb/s with 8 GT/s x8 link)
 kernel: pci ae4b:00:02.0: BAR 0: assigned [mem 0xfe0000000-0xfe00fffff 64bit pref]
 systemd[1]: Starting Initialize the iWARP/InfiniBand/RDMA stack in the kernel...
 kernel: mlx5_core ae4b:00:02.0: firmware version: 14.26.1040
 kernel: mlx5_core ae4b:00:02.0: handle_hca_cap:537:(pid 12693): log_max_qp value in current profile is 18, changing it to HCA capability limit (12)
 kernel: mlx5_core ae4b:00:02.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(64) RxCqeCmprss(0)
 kernel: hv_netvsc cc91b9de-cf8f-4729-8e30-bc4cf0e3f604 eth1: VF registering: eth2
 kernel: mlx5_core ae4b:00:02.0 eth2: joined to eth1
 NetworkManager[996]: <info>  [1578320649.0723] manager: (eth2): new Ethernet device (/org/freedesktop/NetworkManager/Devices/6)
 NetworkManager[996]: <info>  [1578320649.0726] device (eth2): enslaved to non-master-type device eth1; ignoring
 kernel: mlx5_ib: Mellanox Connect-IB Infiniband driver v5.0-0
 systemd-udevd[40538]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
 kernel: mlx5_core ae4b:00:02.0 enP44619s2: renamed from eth2
 NetworkManager[996]: <info>  [1578320649.1427] device (eth2): interface index 6 renamed iface from 'eth2' to 'enP44619s2'
 NetworkManager[996]: <info>  [1578320649.1431] device (enP44619s2): enslaved to non-master-type device eth1; ignoring
 NetworkManager[996]: <info>  [1578320649.1543] device (enP44619s2): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
 systemd[1]: Created slice system-rdma\x2dload\x2dmodules.slice.
 systemd[1]: Starting Load RDMA modules from /etc/rdma/modules/rdma.conf...
 systemd[1]: Starting Load RDMA modules from /etc/rdma/modules/infiniband.conf...
 systemd[1]: Starting Load RDMA modules from /etc/rdma/modules/roce.conf...
 systemd[1]: Starting RDMA Node Description Daemon...
 systemd[1]: Started RDMA Node Description Daemon.
 systemd[1]: Started Load RDMA modules from /etc/rdma/modules/roce.conf.
 kernel: Loading iSCSI transport class v2.0-870.
 kernel: mlx5_core ae4b:00:02.0 enP44619s2: Link up
 kernel: hv_netvsc cc91b9de-cf8f-4729-8e30-bc4cf0e3f604 eth1: Data path switched to VF: enP44619s2
 NetworkManager[996]: <info>  [1578320649.2483] device (enP44619s2): carrier: link connected
 NetworkManager[996]: <info>  [1578320649.2562] device (enP44619s2): enslaved to non-master-type device eth1; ignoring
 NetworkManager[996]: <info>  [1578320649.2564] device (enP44619s2): enslaved to non-master-type device eth1; ignoring
 NetworkManager[996]: <info>  [1578320649.2565] device (enP44619s2): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'managed')
 systemd-modules-load[40640]: Inserted module 'ib_ipoib'
 systemd-modules-load[40640]: Inserted module 'ib_umad'
 NetworkManager[996]: <info>  [1578320649.3233] device (enP44619s2): enslaved to non-master-type device eth1; ignoring
 systemd[1]: Started Load RDMA modules from /etc/rdma/modules/infiniband.conf.
 kernel: iscsi: registered transport (iser)
 systemd-modules-load[40639]: Inserted module 'ib_iser'
 systemd-modules-load[40639]: Inserted module 'rdma_ucm'
 kernel: RPC: Registered rdma transport module.
 kernel: RPC: Registered rdma backchannel transport module.
 systemd-modules-load[40639]: Inserted module 'rpcrdma'
 systemd[1]: Started Load RDMA modules from /etc/rdma/modules/rdma.conf.
 systemd[1]: Reached target RDMA Hardware.
 kernel: Rounding down aligned max_sectors from 4294967295 to 4294967288
 kernel: db_root: cannot open: /etc/target
 kernel: ib_srpt MAD registration failed for mlx5_0-1.
 kernel: ib_srpt srpt_add_one(mlx5_0) failed.
 systemd[1]: Started Initialize the iWARP/InfiniBand/RDMA stack in the kernel.
 ModemManager[876]: <info>  Couldn't check support for device '/sys/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0004:00/VMBUS:00/c2f22477-5994-443d-ae4b-7012efabe867/pciae4b:00/ae4b:00:02.0': not supported by any plugin
Comment 12 Li-Wen Hsu freebsd_committer freebsd_triage 2020-06-13 10:10:32 UTC 
I also see this on 12.1-R, while 12.0-R works.
Comment 13 Hans Petter Selasky freebsd_committer freebsd_triage 2020-06-13 10:29:44 UTC 
Can you test -stable?
Comment 14 Li-Wen Hsu freebsd_committer freebsd_triage 2020-06-15 03:42:37 UTC 
I've tested 12-STABLE (r362180) which doesn't work, while -CURRENT (r362186) works.

They have similar dmesg:

FreeBSD 12.1-STABLE r362180 GENERIC

root@antest:~ # dmesg -a | grep mlx
mlx5en: Mellanox Ethernet driver 3.5.2 (September 2019)
mlx5_core0: <mlx5_core> at device 2.0 on pci1
mlx5_core0: ERR: mlx5_init_once:937:(pid 0): Unable to find vendor specific capabilities
mlx5_core0: WARN: mlx5_fwdump_prep:76:(pid 0): Failed to find vendor-specific capability, error 2
mce0: ERR: mlx5e_ioctl:3303:(pid 0): tso6 disabled due to -txcsum6.
mce0: ERR: mlx5e_ioctl:3323:(pid 0): enable txcsum6 first.
root@antest:~ # dmesg -a | grep mce0
mce0: Ethernet address: 00:0d:3a:8a:b6:bd
mce0: link state changed to DOWN
mce0: ERR: mlx5e_ioctl:3303:(pid 0): tso6 disabled due to -txcsum6.
mce0: ERR: mlx5e_ioctl:3323:(pid 0): enable txcsum6 first.
mce0: link state changed to UP
hn0: mce0 SIOCGIFRSSKEY failed: 22
Starting Network: lo0 hn0 mce0.
mce0: flags=8a43<UP,BROADCAST,RUNNING,ALLMULTI,SIMPLEX,MULTICAST> metric 0 mtu 1500

FreeBSD 13.0-CURRENT r362186 GENERIC

lwhsu@antest:~ > dmesg -a | grep mlx
mlx5en: Mellanox Ethernet driver 3.5.2 (September 2019)
mlx5_core0: <mlx5_core> at device 2.0 on pci1
mlx5_core0: ERR: mlx5_init_once:938:(pid 0): Unable to find vendor specific capabilities
mlx5_core0: WARN: mlx5_fwdump_prep:76:(pid 0): Failed to find vendor-specific capability, error 2
mce0: ERR: mlx5e_ioctl:3341:(pid 0): tso6 disabled due to -txcsum6.
mce0: ERR: mlx5e_ioctl:3367:(pid 0): enable txcsum6 first.
lwhsu@antest:~ > dmesg -a | grep mce0
mce0: Ethernet address: 00:0d:3a:8a:b6:bd
mce0: link state changed to DOWN
mce0: ERR: mlx5e_ioctl:3341:(pid 0): tso6 disabled due to -txcsum6.
mce0: ERR: mlx5e_ioctl:3367:(pid 0): enable txcsum6 first.
mce0: link state changed to UP
hn0: mce0 SIOCGIFRSSKEY failed: 22
Starting Network: lo0 hn0 mce0.
mce0: flags=8a43<UP,BROADCAST,RUNNING,ALLMULTI,SIMPLEX,MULTICAST> metric 0 mtu 1500

I'm testing these on Azure with manually enabling accelerated networking:

https://microsoft.github.io/AzureTipsAndTricks/blog/tip226.html