| Summary: | vm ivhd iommu bhyve nic passthru broken | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Base System | Reporter: | attila.kover | ||||
| Component: | bhyve | Assignee: | freebsd-virtualization (Nobody) <virtualization> | ||||
| Status: | Open --- | ||||||
| Severity: | Affects Many People | CC: | attila.kover, freebsd, michael.osipov, raul.munoz, sm | ||||
| Priority: | --- | ||||||
| Version: | 14.0-RELEASE | ||||||
| Hardware: | amd64 | ||||||
| OS: | Any | ||||||
| Attachments: |
|
||||||
The problem is still there in 13.2-RELEASE and 14.0-CURRENT.
A bit more information from a recent test under 14.0-CURRENT (different hardware):
vm-bhyve.log:
Aug 03 02:25:16: initialising
Aug 03 02:25:16: [loader: bhyveload]
Aug 03 02:25:16: [cpu: 2]
Aug 03 02:25:16: [memory: 4G]
Aug 03 02:25:16: [hostbridge: standard]
Aug 03 02:25:16: [com ports: com1]
Aug 03 02:25:16: [uuid: 2333ba09-dfe1-11ed-9ed6-9c5c8e54ad77]
Aug 03 02:25:16: [debug mode: no]
Aug 03 02:25:16: [primary disk: ptproba.img]
Aug 03 02:25:16: [primary disk dev: file]
Aug 03 02:25:16: booting
Aug 03 02:25:16: bhyveload -c /dev/nmdm-ptproba.1A -S -m 4G -e
autoboot_delay=3 -d /root/vm-bhyve/ptproba/ptproba.img ptproba
Aug 03 02:25:22: [bhyve options: -c 2 -m 4G -AHP -U
2333ba09-dfe1-11ed-9ed6-9c5c8e54ad77 -u -S]
Aug 03 02:25:22: [bhyve devices: -s 0,hostbridge -s 31,lpc -s
4:0,virtio-blk,/root/vm-bhyve/ptproba/ptproba.img -s 5:0,passthru,8/0/0]
Aug 03 02:25:22: [bhyve console: -l com1,/dev/nmdm-ptproba.1A]
Aug 03 02:25:22: starting bhyve (run 1)
The first few dozen messages are like the followings (couple of hundred
lines):
ivhd0: EVT INTR 0 Status:0x1a EVT Head:0x0 Tail:0x10]
[CMD Total 0x24] Tail:0x240, Head:0x240.
ivhd0: [Event0: Head:0x0 Tail:0x10]
[INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
ivhd0: EVT INTR 1 Status:0x1a EVT Head:0x10 Tail:0x20]
[CMD Total 0x26] Tail:0x260, Head:0x260.
ivhd0: [Event0: Head:0x10 Tail:0x20]
[INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
ivhd0: EVT INTR 2 Status:0x1a EVT Head:0x20 Tail:0x30]
[CMD Total 0x26] Tail:0x260, Head:0x260.
ivhd0: [Event0: Head:0x20 Tail:0x30]
[INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
...then the constant flood (about 35 lines per second in /var/log/messages):
ivhd0: [Event63: Head:0x3e0 Tail:0x8d0]
[INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
ivhd0: [Event64: Head:0x3f0 Tail:0x8d0]
[INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
ivhd0: [Event65: Head:0x400 Tail:0x8d0]
[INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
Created attachment 244147 [details]
Bhyve AMD PCI-pass issues.
Output of a SuperMicro box with AMD having the same issue.
Have done some testing today and I can confirm that in my case, PCI-PASS works well with 13.1 ( I have rolled back to releng/13.1). Booted the VM with two pci-pass nic and it works. Update.. the server got bricked after 7 hours of running 2xVM with PCI-PASS. This problem still persist in 14.0-RC1 This problem is still present in 14.0-RC4. |
Symptom: Starting any vm with pci passthru result thousands of error messages on the host in a couple of seconds. If I don't issue a reboot command very soon then the host completely freezes, only hard reboot helps. Hardware: CPU: AMD FX(tm)-6300 Six-Core Processor Motherboard: M5A99X EVO R2.0 NICs: Intel 82580 (HP NC365T) Intel I350-T4 (CISCO 74-10521-01) Broadcom BCM5720 (HP 332T) OS: Host OS: FreeBSD 13.1-RELEASE-p2 (same since 13.1-RELEASE) Guest OS: FreeBSD 13.1-RELEASE-p2 cat /boot/loader.conf|grep ^ppt pptdevs="2/0/3 9/0/3 7/0/1" pptdevs2="2/0/1 2/0/2 9/0/2" pptdevs3="9/0/1 7/0/0" one of the vm configs: loader="bhyveload" cpu=1 memory=1G network0_type="virtio-net" network0_switch="gyumi" passthru0="9/0/1" passthru1="2/0/1" disk0_type="virtio-blk" disk0_name="szerver-node5.img" uuid="eef22ae1-6b38-11ec-91a4-ac162da2c818" network0_mac="xx:xx:xx:xx:xx:xx" Error messages: Sep 9 16:43:19 fx kernel: ivhd0: EVT INTR 0 Status:0x1a EVT Head:0x0 Tail:0x10] Sep 9 16:43:19 fx kernel: [CMD Total 0x23] Tail:0x230, Head:0x230. Sep 9 16:43:19 fx kernel: ivhd0: [Event0: Head:0x0 Tail:0x10] Sep 9 16:43:19 fx kernel: [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0] Sep 9 16:43:19 fx kernel: ivhd0: EVT INTR 1 Status:0x1a EVT Head:0x10 Tail:0x20] Sep 9 16:43:19 fx kernel: [CMD Total 0x23] Tail:0x230, Head:0x230. Sep 9 16:43:19 fx kernel: ivhd0: [Event0: Head:0x10 Tail:0x20] Sep 9 16:43:19 fx kernel: [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0] Sep 9 16:43:19 fx kernel: ivhd0: EVT INTR 2 Status:0x1a EVT Head:0x20 Tail:0x30] Sep 9 16:43:19 fx kernel: [CMD Total 0x23] Tail:0x230, Head:0x230. Sep 9 16:43:19 fx kernel: ivhd0: [Event0: Head:0x20 Tail:0x30] Sep 9 16:43:19 fx kernel: [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0] Sep 9 16:43:19 fx kernel: ivhd0: EVT INTR 3 Status:0x1a EVT Head:0x30 Tail:0x40] Sep 9 16:43:19 fx kernel: [CMD Total 0x23] Tail:0x230, Head:0x230. Sep 9 16:43:19 fx kernel: ivhd0: [Event0: Head:0x30 Tail:0x40] Sep 9 16:43:19 fx kernel: [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0] Sep 9 16:43:19 fx kernel: ivhd0: EVT INTR 4 Status:0x1a EVT Head:0x40 Tail:0x50] Sep 9 16:43:19 fx kernel: [CMD Total 0x23] Tail:0x230, Head:0x230. Sep 9 16:43:19 fx kernel: ivhd0: [Event0: Head:0x40 Tail:0x50] Sep 9 16:43:19 fx kernel: [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0] Sep 9 16:43:19 fx kernel: ivhd0: EVT INTR 5 Status:0x1a EVT Head:0x50 Tail:0x60] Sep 9 16:43:19 fx kernel: [CMD Total 0x23] Tail:0x230, Head:0x230. Sep 9 16:43:19 fx kernel: ivhd0: [Event0: Head:0x50 Tail:0x60] Sep 9 16:43:19 fx kernel: [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0] Sep 9 16:43:19 fx kernel: ivhd0: EVT INTR 6 Status:0x1a EVT Head:0x60 Tail:0x70] Sep 9 16:43:19 fx kernel: [CMD Total 0x25] Tail:0x250, Head:0x250. Sep 9 16:43:19 fx kernel: ivhd0: [Event0: Head:0x60 Tail:0x70] Sep 9 16:43:19 fx kernel: [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0] Sep 9 16:43:19 fx kernel: ivhd0: EVT INTR 7 Status:0x1a EVT Head:0x70 Tail:0x80] Sep 9 16:43:19 fx kernel: [CMD Total 0x25] Tail:0x250, Head:0x250. Sep 9 16:43:19 fx kernel: ivhd0: [Event0: Head:0x70 Tail:0x80] Sep 9 16:43:19 fx kernel: [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0] Sep 9 16:43:19 fx kernel: ivhd0: EVT INTR 8 Status:0x1a EVT Head:0x80 Tail:0x90] Sep 9 16:43:19 fx kernel: [CMD Total 0x25] Tail:0x250, Head:0x250. Sep 9 16:43:19 fx kernel: ivhd0: [Event0: Head:0x80 Tail:0x90] Sep 9 16:43:19 fx kernel: [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0] Sep 9 16:43:19 fx kernel: ivhd0: EVT INTR 9 Status:0x1a EVT Head:0x90 Tail:0xa0] Sep 9 16:43:19 fx kernel: [CMD Total 0x25] Tail:0x250, Head:0x250. Sep 9 16:43:19 fx kernel: ivhd0: [Event0: Head:0x90 Tail:0xa0] Sep 9 16:43:19 fx kernel: [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0] Sep 9 16:43:19 fx kernel: ivhd0: EVT INTR 10 Status:0x1a EVT Head:0xa0 Tail:0xb0] Sep 9 16:43:19 fx kernel: [CMD Total 0x25] Tail:0x250, Head:0x250. Sep 9 16:43:19 fx kernel: ivhd0: [Event0: Head:0xa0 Tail:0xb0] Sep 9 16:43:19 fx kernel: [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0] Sep 9 16:43:19 fx kernel: ivhd0: EVT INTR 11 Status:0x1a EVT Head:0xb0 Tail:0xc0] Sep 9 16:43:19 fx kernel: [CMD Total 0x25] Tail:0x250, Head:0x250. Sep 9 16:43:19 fx kernel: ivhd0: [Event0: Head:0xb0 Tail:0xc0] Sep 9 16:43:19 fx kernel: [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0] Workaround: freebsd-update rollback (...to an EOLd system - apologies for calling this a workaround)