Attachment 244147 Details for Bug 266325 – Bhyve AMD PCI-pass issues.

Bhyve AMD PCI-pass issues.

bhyve-amd-pci_pass.txt (text/plain), 14.35 KB, created by Santiago Martinez on 2023-08-16 11:22:19 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Santiago Martinez

Created: 2023-08-16 11:22:19 UTC

Size: 14.35 KB

patch

obsolete

>SUMMARY:
>
>    *Device get bricked when using PCI-passthru on AMD systems.
>    *This can be reproduced all the time.
>    *Sometimes it affects NVME and machine reboots after NVME commands times out.
>    *It used to work without issues (same hardware and same config), initially i thought it was related to AMD Microcode Upgrade, but seems it started with 13.2
>    *At the moment box is running 13 Stable but it happens also with 13.2P1 and CURRENT.
>
>
>DMIDECODE:
>    Found SMBIOS entry point in EFI, reading table from /dev/mem.
>    SMBIOS 3.2 present.
>
>    Handle 0x0029, DMI type 4, 48 bytes
>    Processor Information
>            Socket Designation: CPU
>            Type: Central Processor
>            Family: Zen
>            Manufacturer: Advanced Micro Devices, Inc.
>            ID: 10 0F 83 00 FF FB 8B 17
>            Signature: Family 23, Model 49, Stepping 0
>            Flags:
>                    FPU (Floating-point unit on-chip)
>                    VME (Virtual mode extension)
>                    DE (Debugging extension)
>                    PSE (Page size extension)
>                    TSC (Time stamp counter)
>                    MSR (Model specific registers)
>                    PAE (Physical address extension)
>                    MCE (Machine check exception)
>                    CX8 (CMPXCHG8 instruction supported)
>                    APIC (On-chip APIC hardware supported)
>                    SEP (Fast system call)
>                    MTRR (Memory type range registers)
>                    PGE (Page global enable)
>                    MCA (Machine check architecture)
>                    CMOV (Conditional move instruction supported)
>                    PAT (Page attribute table)
>                    PSE-36 (36-bit page size extension)
>                    CLFSH (CLFLUSH instruction supported)
>                    MMX (MMX technology supported)
>                    FXSR (FXSAVE and FXSTOR instructions supported)
>                    SSE (Streaming SIMD extensions)
>                    SSE2 (Streaming SIMD extensions 2)
>                    HTT (Multi-threading)
>            Version: AMD EPYC 7552 48-Core Processor
>            Voltage: 1.1 V
>            External Clock: 100 MHz
>            Max Speed: 3300 MHz
>            Current Speed: 2200 MHz
>            Status: Populated, Enabled
>            Upgrade: Socket SP3
>            L1 Cache Handle: 0x0026
>            L2 Cache Handle: 0x0027
>            L3 Cache Handle: 0x0028
>            Serial Number: Unknown
>            Asset Tag: Unknown
>            Part Number: Unknown
>            Core Count: 48
>            Core Enabled: 48
>            Thread Count: 96
>            Characteristics:
>                    64-bit capable
>                    Multi-Core
>                    Hardware Thread
>                    Execute Protection
>                    Enhanced Virtualization
>                    Power/Performance Control
>
>
>
>
>
>FreeBSD VERSION:
>    freebsd-version -uk
>    13.2-STABLE
>    13.2-RELEASE-p1
>
>
>LOADED MODULES:
>
>    Id Refs Address                Size Name
>    1  103 0xffffffff80200000  14578a8 kernel
>    2    1 0xffffffff81658000     7258 amdtemp.ko
>    3    2 0xffffffff81660000     3e48 amdsmn.ko
>    4    1 0xffffffff81664000     3d58 coretemp.ko
>    5    1 0xffffffff81668000    41c90 if_bnxt.ko
>    6    3 0xffffffff816aa000    28210 iflib.ko
>    7    1 0xffffffff816d3000     be98 cuse.ko
>    8    1 0xffffffff816df000     5860 amdgpio.ko
>    9    2 0xffffffff816e5000     f040 gpiobus.ko
>    10    1 0xffffffff816f5000     4c80 cpuctl.ko
>    11    1 0xffffffff816fa000     6180 amdpm.ko
>    12    4 0xffffffff81701000     4508 smbus.ko
>    13    1 0xffffffff81706000     5290 amdsbwd.ko
>    14    1 0xffffffff8170c000    14b08 ipmi.ko
>    15    1 0xffffffff81721000    120c0 ioat.ko
>    16    1 0xffffffff81734000   5d5db8 zfs.ko
>    17    1 0xffffffff81d0a000     b388 cryptodev.ko
>    18    1 0xffffffff82210000     3530 fdescfs.ko
>    19    1 0xffffffff82214000     3218 intpm.ko
>    20    1 0xffffffff82218000     76c0 if_bridge.ko
>    21    1 0xffffffff82220000     60d8 bridgestp.ko
>    22    1 0xffffffff82227000     3340 uhid.ko
>    23    1 0xffffffff8222b000     4350 ums.ko
>    24    1 0xffffffff82230000     3380 usbhid.ko
>    25    1 0xffffffff82234000     32b0 hidbus.ko
>    26    1 0xffffffff82400000   339438 vmm.ko
>    27    1 0xffffffff82238000     21cc nmdm.ko
>    28    1 0xffffffff8223b000     2a08 mac_ntpd.ko
>    29    1 0xffffffff8223e000    2f214 if_ixl.ko
>
>
>SYSCTL:
>    hw.machine: amd64
>    hw.model: AMD EPYC 7552 48-Core Processor
>    hw.ncpu: 96
>    hw.byteorder: 1234
>    hw.physmem: 274704781312
>    hw.usermem: 272530210816
>    hw.pagesize: 4096
>    hw.floatingpoint: 1
>    hw.machine_arch: amd64
>    hw.realmem: 276204355584
>
>    hw.vmm.amdvi.domain_id: 0
>    hw.vmm.amdvi.disable_io_fault: 0
>    hw.vmm.amdvi.ptp_level: 4
>    hw.vmm.amdvi.host_ptp: 1
>    hw.vmm.amdvi.enable: 1
>    hw.vmm.amdvi.count: 4
>    hw.vmm.npt.pmap_flags: 508
>    hw.vmm.svm.num_asids: 32768
>    hw.vmm.svm.disable_npf_assist: 0
>    hw.vmm.svm.features: 1293567
>    hw.vmm.svm.vmcb_clean: 1023
>    hw.vmm.vmx.l1d_flush_sw: 0
>    hw.vmm.vmx.l1d_flush: 0
>    hw.vmm.vmx.vpid_alloc_failed: 0
>    hw.vmm.vmx.posted_interrupt_vector: -1
>    hw.vmm.vmx.cap.posted_interrupts: 0
>    hw.vmm.vmx.cap.virtual_interrupt_delivery: 0
>    hw.vmm.vmx.cap.tpr_shadowing: 0
>    hw.vmm.vmx.cap.invpcid: 0
>    hw.vmm.vmx.cap.monitor_trap: 0
>    hw.vmm.vmx.cap.unrestricted_guest: 0
>    hw.vmm.vmx.cap.rdtscp: 0
>    hw.vmm.vmx.cap.rdpid: 0
>    hw.vmm.vmx.cap.wbinvd_exit: 0
>    hw.vmm.vmx.cap.pause_exit: 0
>    hw.vmm.vmx.cap.halt_exit: 0
>    hw.vmm.vmx.initialized: 0
>    hw.vmm.vmx.cr4_zeros_mask: 0
>    hw.vmm.vmx.cr4_ones_mask: 0
>    hw.vmm.vmx.cr0_zeros_mask: 0
>    hw.vmm.vmx.cr0_ones_mask: 0
>    hw.vmm.vmx.no_flush_rsb: 0
>    hw.vmm.ept.pmap_flags: 0
>    hw.vmm.vrtc.flag_broken_time: 1
>    hw.vmm.ppt.devices: 0
>    hw.vmm.iommu.enable: 1
>    hw.vmm.iommu.initialized: 0
>    hw.vmm.bhyve_xcpuids: 1108
>    hw.vmm.topology.cpuid_leaf_b: 1
>    hw.vmm.create: beavis
>    hw.vmm.destroy: beavis
>    hw.vmm.maxcpu: 96
>    hw.vmm.trap_wbinvd: 0
>    hw.vmm.trace_guest_exceptions: 0
>    hw.vmm.ipinum: 252
>    hw.vmm.halt_detection: 1
>
>
>PCI INFO (relevant?):
>    amdiommu0@pci0:192:0:2: class=0x080600 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1481 subvendor=0x1022 subdevice=0x1481
>    bnxt0@pci0:199:0:0:     class=0x020000 rev=0x01 hdr=0x00 vendor=0x14e4 device=0x16d8 subvendor=0x15d9 subdevice=0x16d8
>    bnxt1@pci0:199:0:1:     class=0x020000 rev=0x01 hdr=0x00 vendor=0x14e4 device=0x16d8 subvendor=0x15d9 subdevice=0x16d8
>    amdiommu1@pci0:128:0:2: class=0x080600 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1481 subvendor=0x1022 subdevice=0x1481
>    amdiommu2@pci0:64:0:2:  class=0x080600 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1481 subvendor=0x1022 subdevice=0x1481
>    ppt0@pci0:65:0:0:       class=0x020000 rev=0x02 hdr=0x00 vendor=0x8086 device=0x1583 subvendor=0x15d9 subdevice=0x084a
>    ppt1@pci0:65:0:1:       class=0x020000 rev=0x02 hdr=0x00 vendor=0x8086 device=0x1583 subvendor=0x15d9 subdevice=0x0000
>    amdiommu3@pci0:0:0:2:   class=0x080600 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1481 subvendor=0x1022 subdevice=0x1481
>    mpr0@pci0:1:0:0:        class=0x010700 rev=0x02 hdr=0x00 vendor=0x1000 device=0x0097 subvendor=0x15d9 subdevice=0x0808
>
>
>DEVICE PREP:
>    pciconf -l | egrep "ixl|ppt"
>    ixl0@pci0:65:0:0:       class=0x020000 rev=0x02 hdr=0x00 vendor=0x8086 device=0x1583 subvendor=0x15d9 subdevice=0x084a
>    ixl1@pci0:65:0:1:       class=0x020000 rev=0x02 hdr=0x00 vendor=0x8086 device=0x1583 subvendor=0x15d9 subdevice=0x0000
>
>    devctl detach pci0:65:0:0
>    devctl set driver pci0:65:0:0 ppt
>
>    pciconf -l | egrep "ixl|ppt"
>    ppt0@pci0:65:0:0:       class=0x020000 rev=0x02 hdr=0x00 vendor=0x8086 device=0x1583 subvendor=0x15d9 subdevice=0x084a
>    ixl1@pci0:65:0:1:       class=0x020000 rev=0x02 hdr=0x00 vendor=0x8086 device=0x1583 subvendor=0x15d9 subdevice=0x0000
>
>SYSCTL:
>    hw.vmm.ppt.devices: 1
>
>
>VM config:
>    loader="uefi"
>    cpu=32
>    graphics=1
>    memory=16G
>    network0_type="virtio-net"
>    network0_switch="BR_DCN_VF"
>    disk0_type="virtio-blk"
>    disk0_name="disk0.img"
>    passthru0="65/0/0"
>    uuid="9df51226-a5ef-11ed-bbef-ac1f6be79f0e"
>    network0_mac="58:9c:fc:0a:e7:95"
>
>VM LOG:
>    Aug 16 13:46:54: initialising
>    Aug 16 13:46:54:  [loader: uefi]
>    Aug 16 13:46:54:  [cpu: 32]
>    Aug 16 13:46:54:  [memory: 16G]
>    Aug 16 13:46:54:  [hostbridge: standard]
>    Aug 16 13:46:54:  [com ports: com1]
>    Aug 16 13:46:54:  [uuid: 9df51226-a5ef-11ed-bbef-ac1f6be79f0e]
>    Aug 16 13:46:54:  [debug mode: no]
>    Aug 16 13:46:54:  [primary disk: disk0.img]
>    Aug 16 13:46:54:  [primary disk dev: file]
>    Aug 16 13:46:54: initialising network device tap3
>    Aug 16 13:46:54: adding tap3 -> BR_DCN_VF (BR_DCN_VF addm)
>    Aug 16 13:46:54: bring up tap3 -> BR_DCN_VF (BR_DCN_VF addm)
>    Aug 16 13:46:54: dynamically allocated port 5903 for vnc connections
>    Aug 16 13:46:54: booting
>    Aug 16 13:46:54:  [bhyve options: -c 32 -m 16G -Hwl bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -U 9df51226-a5ef-11ed-bbef-ac1f6be79f0e -u -S]
>    Aug 16 13:46:54:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,virtio-blk,/pool01/vm/FREEBSD01/disk0.img -s 5:0,virtio-net,tap3,mac=58:9c:fc:0a:e7:95 -s 6:0,passthru,65/0/0 -s 7:0,fbuf,tcp=0.0.0.0:5903]
>    Aug 16 13:46:54:  [bhyve console: -l com1,/dev/nmdm-FREEBSD01.1A]
>
>
>ERRORS (sometimes nvme also stops reponding and machines reboots, this time only affected bxnt cards)
>    Aug 16 13:46:57 hyper02 kernel[139]: ivhd0: EVT INTR 0 Status:0x1a EVT Head:0x0 Tail:0x10]
>    Aug 16 13:46:57 hyper02 kernel[139]:   [CMD Total 0x20] Tail:0x200, Head:0x200.
>    Aug 16 13:46:57 hyper02 kernel[139]: ivhd0:     [Event0: Head:0x0 Tail:0x30]
>    Aug 16 13:46:57 hyper02 kernel[139]:    [IO_PAGE_FAULT EVT: devId:0xc701 DomId:0x0 Addr:0x10da33000 0xa0<RW,RZ>]
>    Aug 16 13:46:57 hyper02 kernel[139]: ivhd0:     [Event1: Head:0x10 Tail:0x30]
>    Aug 16 13:46:57 hyper02 kernel[139]:    [IO_PAGE_FAULT EVT: devId:0xc701 DomId:0x0 Addr:0x10da36000 0xa0<RW,RZ>]
>    Aug 16 13:46:57 hyper02 kernel[139]: ivhd0:     [Event2: Head:0x20 Tail:0x30]
>    Aug 16 13:46:57 hyper02 kernel[139]:    [IO_PAGE_FAULT EVT: devId:0xc701 DomId:0x0 Addr:0x10d9b2000 0xa0<RW,RZ>]
>    Aug 16 13:46:58 hyper02 kernel[140]: bnxt1: Timeout sending HWRM_PORT_QSTATS: (timeout: 500) seq: 53
>    Aug 16 13:46:58 hyper02 kernel[140]: ivhd0: EVT INTR 1 Status:0x1a EVT Head:0x30 Tail:0x40]
>    Aug 16 13:46:58 hyper02 kernel[140]:   [CMD Total 0x20] Tail:0x200, Head:0x200.
>    Aug 16 13:46:58 hyper02 kernel[140]: ivhd0:     [Event0: Head:0x30 Tail:0x40]
>    Aug 16 13:46:58 hyper02 kernel[140]:    [IO_PAGE_FAULT EVT: devId:0xc701 DomId:0x0 Addr:0x10d9b2000 0xa0<RW,RZ>]
>    Aug 16 13:46:58 hyper02 kernel[140]: bnxt1: Timeout sending HWRM_PORT_PHY_QCFG: (timeout: 500) seq: 54
>    Aug 16 13:46:58 hyper02 kernel[140]: ivhd0: EVT INTR 2 Status:0x1a EVT Head:0x40 Tail:0x50]
>    Aug 16 13:46:58 hyper02 kernel[140]:   [CMD Total 0x20] Tail:0x200, Head:0x200.
>    Aug 16 13:46:58 hyper02 kernel[140]: ivhd0:     [Event0: Head:0x40 Tail:0x70]
>    Aug 16 13:46:58 hyper02 kernel[140]:    [IO_PAGE_FAULT EVT: devId:0xc701 DomId:0x0 Addr:0x10da33000 0xa0<RW,RZ>]
>    Aug 16 13:46:58 hyper02 kernel[140]: ivhd0:     [Event1: Head:0x50 Tail:0x70]
>    Aug 16 13:46:58 hyper02 kernel[140]:    [IO_PAGE_FAULT EVT: devId:0xc701 DomId:0x0 Addr:0x10da36000 0xa0<RW,RZ>]
>    Aug 16 13:46:58 hyper02 kernel[140]: ivhd0:     [Event2: Head:0x60 Tail:0x70]
>    Aug 16 13:46:58 hyper02 kernel[140]:    [IO_PAGE_FAULT EVT: devId:0xc701 DomId:0x0 Addr:0x10d9b2000 0xa0<RW,RZ>]
>    Aug 16 13:46:59 hyper02 kernel[141]: bnxt1: Timeout sending HWRM_PORT_QSTATS: (timeout: 500) seq: 55
>    Aug 16 13:46:59 hyper02 kernel[141]: ivhd0: EVT INTR 3 Status:0x1a EVT Head:0x70 Tail:0x80]
>    Aug 16 13:46:59 hyper02 kernel[141]:   [CMD Total 0x20] Tail:0x200, Head:0x200.
>    Aug 16 13:46:59 hyper02 kernel[141]: ivhd0:     [Event0: Head:0x70 Tail:0x80]
>    Aug 16 13:46:59 hyper02 kernel[141]:    [IO_PAGE_FAULT EVT: devId:0xc701 DomId:0x0 Addr:0x10d9b2000 0xa0<RW,RZ>]
>    Aug 16 13:47:10 hyper02 kernel[141]: bnxt1: Timeout sending HWRM_PORT_PHY_QCFG: (timeout: 500) seq: 56
>    Aug 16 13:47:10 hyper02 kernel[152]: ivhd0: EVT INTR 4 Status:0x1a EVT Head:0x80 Tail:0x90]
>    Aug 16 13:47:10 hyper02 kernel[152]:   [CMD Total 0x20] Tail:0x200, Head:0x200.
>    Aug 16 13:47:10 hyper02 kernel[152]: ivhd0:     [Event0: Head:0x80 Tail:0x90]
>    Aug 16 13:47:10 hyper02 kernel[152]:    [IO_PAGE_FAULT EVT: devId:0xc700 DomId:0x0 Addr:0x10d00a500 0x80<RZ>]
>    Aug 16 13:47:56 hyper02 login[51880]: ROOT LOGIN (root) ON ttyv0
>    Aug 16 13:47:56 hyper02 kernel[152]: bnxt0: Timeout sending HWRM_PORT_QSTATS: (timeout: 500) seq: 1018
>    Aug 16 13:47:56 hyper02 kernel[153]: bnxt0: Timeout sending HWRM_PORT_QSTATS: (timeout: 500) seq: 1019
>    Aug 16 13:47:56 hyper02 kernel[154]: bnxt0: Timeout sending HWRM_PORT_QSTATS: (timeout: 500) seq: 1020
>    Aug 16 13:47:56 hyper02 kernel[155]: bnxt0: Timeout sending HWRM_PORT_QSTATS: (timeout: 500) seq: 1021
>    Aug 16 13:47:56 hyper02 kernel[156]: bnxt0: Timeout sending HWRM_PORT_QSTATS: (timeout: 500) seq: 1022
>    Aug 16 13:47:56 hyper02 kernel[157]: bnxt0: Timeout sending HWRM_PORT_QSTATS: (timeout: 500) seq: 1023
>    Aug 16 13:47:56 hyper02 kernel[158]: bnxt0: Timeout sending HWRM_PORT_QSTATS: (timeout: 500) seq: 1024
>    Aug 16 13:47:56 hyper02 kernel[159]: bnxt0: Timeout sending HWRM_PORT_QSTATS: (timeout: 500) seq: 1025
>    Aug 16 13:47:56 hyper02 kernel[160]: bnxt0: Timeout sending HWRM_PORT_QSTATS: (timeout: 500) seq: 1026
>    Aug 16 13:47:56 hyper02 kernel[160]: bnxt0: Timeout sending HWRM_PORT_PHY_QCFG: (timeout: 500) seq: 1027
>    Aug 16 13:47:56 hyper02 kernel[161]: bnxt0: Timeout sending HWRM_PORT_QSTATS: (timeout: 500) seq: 1028
>    Aug 16 13:47:56 hyper02 kernel[161]: bnxt0: Timeout sending HWRM_PORT_PHY_QCFG: (timeout: 500) seq: 1029
>    Aug 16 13:47:56 hyper02 kernel[162]: bnxt0: Timeout sending HWRM_PORT_QSTATS: (timeout: 500) seq: 1030
>    Aug 16 13:47:56 hyper02 kernel[162]: bnxt0: Timeout sending HWRM_PORT_PHY_QCFG: (timeout: 500) seq: 1031
>    Aug 16 13:47:56 hyper02 kernel[163]: bnxt0: Timeout sending HWRM_PORT_QSTATS: (timeout: 500) seq: 1032
>    Aug 16 13:47:56 hyper02 kernel[163]: bnxt0: Timeout sending HWRM_PORT_QSTATS: (timeout: 500) seq: 1033
>    Aug 16 13:47:56 hyper02 kernel[164]: bnxt0: Timeout sending HWRM_PORT_QSTATS: (timeout: 500) seq: 1034
>    Aug 16 13:47:56 hyper02 kernel[165]: bnxt0: Timeout sending HWRM_PORT_QSTATS: (timeout: 500) seq: 1035
>

Actions: View

Attachments on bug 266325: 244147