Bug 266325 - vm ivhd iommu bhyve nic passthru broken
Summary: vm ivhd iommu bhyve nic passthru broken
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: bhyve (show other bugs)
Version: 14.0-RELEASE
Hardware: amd64 Any
: --- Affects Many People
Assignee: freebsd-virtualization (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-09-09 18:46 UTC by attila.kover
Modified: 2023-12-07 21:06 UTC (History)
5 users (show)

See Also:

Attachments
Bhyve AMD PCI-pass issues. (14.35 KB, text/plain)
2023-08-16 11:22 UTC, Santiago Martinez 		no flags Details
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description attila.kover 2022-09-09 18:46:18 UTC 
Symptom:
Starting any vm with pci passthru result thousands of error messages on the host in a couple of seconds. If I don't issue a reboot command very soon then the host completely freezes, only hard reboot helps.

 Hardware:
CPU: AMD FX(tm)-6300 Six-Core Processor
Motherboard: M5A99X EVO R2.0
NICs: Intel 82580 (HP NC365T)
      Intel I350-T4 (CISCO 74-10521-01)
      Broadcom BCM5720 (HP 332T)

 OS:
Host OS: FreeBSD 13.1-RELEASE-p2 (same since 13.1-RELEASE)
Guest OS: FreeBSD 13.1-RELEASE-p2

 cat /boot/loader.conf|grep ^ppt
pptdevs="2/0/3 9/0/3 7/0/1"
pptdevs2="2/0/1 2/0/2 9/0/2"
pptdevs3="9/0/1 7/0/0"


 one of the vm configs:
loader="bhyveload"
cpu=1
memory=1G
network0_type="virtio-net"
network0_switch="gyumi"
passthru0="9/0/1"
passthru1="2/0/1"
disk0_type="virtio-blk"
disk0_name="szerver-node5.img"
uuid="eef22ae1-6b38-11ec-91a4-ac162da2c818"
network0_mac="xx:xx:xx:xx:xx:xx"


 Error messages:
Sep  9 16:43:19 fx kernel: ivhd0: EVT INTR 0 Status:0x1a EVT Head:0x0 Tail:0x10]
Sep  9 16:43:19 fx kernel:   [CMD Total 0x23] Tail:0x230, Head:0x230.
Sep  9 16:43:19 fx kernel: ivhd0:       [Event0: Head:0x0 Tail:0x10]
Sep  9 16:43:19 fx kernel:      [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
Sep  9 16:43:19 fx kernel: ivhd0: EVT INTR 1 Status:0x1a EVT Head:0x10 Tail:0x20]
Sep  9 16:43:19 fx kernel:   [CMD Total 0x23] Tail:0x230, Head:0x230.
Sep  9 16:43:19 fx kernel: ivhd0:       [Event0: Head:0x10 Tail:0x20]
Sep  9 16:43:19 fx kernel:      [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
Sep  9 16:43:19 fx kernel: ivhd0: EVT INTR 2 Status:0x1a EVT Head:0x20 Tail:0x30]
Sep  9 16:43:19 fx kernel:   [CMD Total 0x23] Tail:0x230, Head:0x230.
Sep  9 16:43:19 fx kernel: ivhd0:       [Event0: Head:0x20 Tail:0x30]
Sep  9 16:43:19 fx kernel:      [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
Sep  9 16:43:19 fx kernel: ivhd0: EVT INTR 3 Status:0x1a EVT Head:0x30 Tail:0x40]
Sep  9 16:43:19 fx kernel:   [CMD Total 0x23] Tail:0x230, Head:0x230.
Sep  9 16:43:19 fx kernel: ivhd0:       [Event0: Head:0x30 Tail:0x40]
Sep  9 16:43:19 fx kernel:      [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
Sep  9 16:43:19 fx kernel: ivhd0: EVT INTR 4 Status:0x1a EVT Head:0x40 Tail:0x50]
Sep  9 16:43:19 fx kernel:   [CMD Total 0x23] Tail:0x230, Head:0x230.
Sep  9 16:43:19 fx kernel: ivhd0:       [Event0: Head:0x40 Tail:0x50]
Sep  9 16:43:19 fx kernel:      [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
Sep  9 16:43:19 fx kernel: ivhd0: EVT INTR 5 Status:0x1a EVT Head:0x50 Tail:0x60]
Sep  9 16:43:19 fx kernel:   [CMD Total 0x23] Tail:0x230, Head:0x230.
Sep  9 16:43:19 fx kernel: ivhd0:       [Event0: Head:0x50 Tail:0x60]
Sep  9 16:43:19 fx kernel:      [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
Sep  9 16:43:19 fx kernel: ivhd0: EVT INTR 6 Status:0x1a EVT Head:0x60 Tail:0x70]
Sep  9 16:43:19 fx kernel:   [CMD Total 0x25] Tail:0x250, Head:0x250.
Sep  9 16:43:19 fx kernel: ivhd0:       [Event0: Head:0x60 Tail:0x70]
Sep  9 16:43:19 fx kernel:      [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
Sep  9 16:43:19 fx kernel: ivhd0: EVT INTR 7 Status:0x1a EVT Head:0x70 Tail:0x80]
Sep  9 16:43:19 fx kernel:   [CMD Total 0x25] Tail:0x250, Head:0x250.
Sep  9 16:43:19 fx kernel: ivhd0:       [Event0: Head:0x70 Tail:0x80]
Sep  9 16:43:19 fx kernel:      [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
Sep  9 16:43:19 fx kernel: ivhd0: EVT INTR 8 Status:0x1a EVT Head:0x80 Tail:0x90]
Sep  9 16:43:19 fx kernel:   [CMD Total 0x25] Tail:0x250, Head:0x250.
Sep  9 16:43:19 fx kernel: ivhd0:       [Event0: Head:0x80 Tail:0x90]
Sep  9 16:43:19 fx kernel:      [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
Sep  9 16:43:19 fx kernel: ivhd0: EVT INTR 9 Status:0x1a EVT Head:0x90 Tail:0xa0]
Sep  9 16:43:19 fx kernel:   [CMD Total 0x25] Tail:0x250, Head:0x250.
Sep  9 16:43:19 fx kernel: ivhd0:       [Event0: Head:0x90 Tail:0xa0]
Sep  9 16:43:19 fx kernel:      [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
Sep  9 16:43:19 fx kernel: ivhd0: EVT INTR 10 Status:0x1a EVT Head:0xa0 Tail:0xb0]
Sep  9 16:43:19 fx kernel:   [CMD Total 0x25] Tail:0x250, Head:0x250.
Sep  9 16:43:19 fx kernel: ivhd0:       [Event0: Head:0xa0 Tail:0xb0]
Sep  9 16:43:19 fx kernel:      [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
Sep  9 16:43:19 fx kernel: ivhd0: EVT INTR 11 Status:0x1a EVT Head:0xb0 Tail:0xc0]
Sep  9 16:43:19 fx kernel:   [CMD Total 0x25] Tail:0x250, Head:0x250.
Sep  9 16:43:19 fx kernel: ivhd0:       [Event0: Head:0xb0 Tail:0xc0]
Sep  9 16:43:19 fx kernel:      [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]


 Workaround: freebsd-update rollback (...to an EOLd system - apologies for calling this a workaround)
Comment 1 attila.kover 2023-08-05 12:33:14 UTC 
The problem is still there in 13.2-RELEASE and 14.0-CURRENT.
Comment 2 attila.kover 2023-08-07 21:06:17 UTC 
A bit more information from a recent test under 14.0-CURRENT (different hardware):


vm-bhyve.log:

Aug 03 02:25:16: initialising
Aug 03 02:25:16:  [loader: bhyveload]
Aug 03 02:25:16:  [cpu: 2]
Aug 03 02:25:16:  [memory: 4G]
Aug 03 02:25:16:  [hostbridge: standard]
Aug 03 02:25:16:  [com ports: com1]
Aug 03 02:25:16:  [uuid: 2333ba09-dfe1-11ed-9ed6-9c5c8e54ad77]
Aug 03 02:25:16:  [debug mode: no]
Aug 03 02:25:16:  [primary disk: ptproba.img]
Aug 03 02:25:16:  [primary disk dev: file]
Aug 03 02:25:16: booting
Aug 03 02:25:16: bhyveload -c /dev/nmdm-ptproba.1A -S -m 4G -e
autoboot_delay=3 -d /root/vm-bhyve/ptproba/ptproba.img ptproba
Aug 03 02:25:22:  [bhyve options: -c 2 -m 4G -AHP -U
2333ba09-dfe1-11ed-9ed6-9c5c8e54ad77 -u -S]
Aug 03 02:25:22:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s
4:0,virtio-blk,/root/vm-bhyve/ptproba/ptproba.img -s 5:0,passthru,8/0/0]
Aug 03 02:25:22:  [bhyve console: -l com1,/dev/nmdm-ptproba.1A]
Aug 03 02:25:22: starting bhyve (run 1)


The first few dozen messages are like the followings (couple of hundred
lines):

ivhd0: EVT INTR 0 Status:0x1a EVT Head:0x0 Tail:0x10]
  [CMD Total 0x24] Tail:0x240, Head:0x240.
ivhd0:  [Event0: Head:0x0 Tail:0x10]
        [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
ivhd0: EVT INTR 1 Status:0x1a EVT Head:0x10 Tail:0x20]
  [CMD Total 0x26] Tail:0x260, Head:0x260.
ivhd0:  [Event0: Head:0x10 Tail:0x20]
        [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
ivhd0: EVT INTR 2 Status:0x1a EVT Head:0x20 Tail:0x30]
  [CMD Total 0x26] Tail:0x260, Head:0x260.
ivhd0:  [Event0: Head:0x20 Tail:0x30]
        [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]



...then the constant flood (about 35 lines per second in /var/log/messages):

ivhd0:  [Event63: Head:0x3e0 Tail:0x8d0]
        [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
ivhd0:  [Event64: Head:0x3f0 Tail:0x8d0]
        [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
ivhd0:  [Event65: Head:0x400 Tail:0x8d0]
        [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
Comment 3 Santiago Martinez 2023-08-16 11:22:19 UTC 
Created attachment 244147 [details]
Bhyve AMD PCI-pass issues.

Output of a SuperMicro box with AMD having the same issue.
Comment 4 Santiago Martinez 2023-08-17 16:03:02 UTC 
Have done some testing today and I can confirm that in my case, PCI-PASS works well with 13.1 ( I have rolled back to releng/13.1). Booted the VM with two pci-pass nic and it works.
Comment 5 Santiago Martinez 2023-08-17 22:56:10 UTC 
Update.. the server got bricked after 7 hours of running 2xVM with PCI-PASS.
Comment 6 attila.kover 2023-10-14 17:51:54 UTC 
This problem still persist in 14.0-RC1
Comment 7 attila.kover 2023-11-07 13:17:30 UTC 
This problem is still present in 14.0-RC4.