I run a couple of FreeBSD based VMs in Proxmox. Upgrading to Proxmox 6 which includes QEMU 4 broke the PCIe passthrough I had setup in both VMs. On further investigation someone on the proxmox forums suggested that the problem is a device ID conflict. https://forum.proxmox.com/threads/vm-w-pcie-passthrough-not-working-after-upgrading-to-6-0.56021/post-274339 Apparently the QEMU vendor and device ids are being used for the ptnetmap-memdev device. https://svnweb.freebsd.org/base/release/12.0.0/sys/net/netmap_virt.h?view=markup#l44 https://devicehunt.com/view/type/pci/vendor/1B36/device/000C Any chance this can be fixed. I am currently having to run my VMs with an older version of the Q35 machine in QEMU to make them work and I don't see any way I can upgrade to later revisions until this is fixed. I suspect there are quite a few people affected by this.
(In reply to kevo from comment #0) HI Kevo & Vincenzo, I did the following tests: QEMU Q35 V4.0 / OVMF with VirtIO SCSI & NIC == e1000 + vmxnet + netmap patched FreeBSD: SCSI - VirtIO SCS disk not found NIC - em0 & vmx0 found QEMU Q35 V4.0 / OVMF with SATA & NIC == e1000 + vmxnet + netmap patched FreeBSD: SATA - disk found NIC - em0 & vmx0 found QEMU Q35 V4.0 OVMF with SATA & NIC == e1000e + vmxnet + netmap patched FreeBSD: SATA - disk found NIC - em - not found, vmx0 found So this appears to confirm that there is general issues with Q35 V4.0 PCIe devices (VirtIO PCIe and e100e), as identified by Tommy T here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241774#c77 Could this be confirmed by doing QEMU Q35 PCIe passthrough using netmap patched (which addresses PCI ID conflicts) FreeBSD VM ? I can run this test if requried as I have test machine with Intel PCIe NIC installed and should be able to try passthrough to this. Cheers, John Hartley.
(In reply to kevo from comment #0) You are right. Sorry for that. This has been fixed in r356805.
From others' and my investigations in BR 236922 & 241774, I doubt netmap is the cause. If the configuration is imported from pre QEMU 4.0 environment, everything works as expected. If the VM is created in the new QEMU 4.0 environment, all of the peripherals connected to the PCIe stops working in FreeBSD because of the new: -device pcie-pci-bridge,id=pci.2,bus=pci.1,addr=0x0 instead of the older: -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 Initially I thought it was this configuration: -machine pc-q35-4.0,accel=kvm,usb=off,vmport=off,dump-guest-core=off But when changed to: -machine pc-q35-2.11,accel=kvm,usb=off,vmport=off,dump-guest-core=off has no effect. I thought it was limitations of IRQ # from dmesg's verbose: Failed to allocate interrupt for PCI-e events But further research [1] shows that it's not since I can not find that entry in the specified code of 12.1 r356769. I did find an error report of similar issue [2] but no official documentation into the specifics of PCIE-PCI bridge. There are plenty of official documentation on PCI-PCI bridge including the recent addition of NEW_PCIB [3]. There are 2 others in BR 236922 & 241774 have also experienced similar issue with QEMU 4.0 [1] https://lists.freebsd.org/pipermail/freebsd-current/2018-April/069203.html [2] https://lists.freebsd.org/pipermail/freebsd-drivers/2007-December/000584.html [3] https://wiki.freebsd.org/NEW_PCIB =================================================================================== Here are the devices of my testing VM with different 12.1 kernels: **** Original kernel from ISO media --- Operational VirtIO PCIe - Console no PCIe - Memory Balloon no PCIe - Network no PCIe - SCSI no PCIe - Serial no NIC (non VirtIO NIC) PCI - e1000 Intel 82545EM (em) no PCIe - e1000e Intel 82574 (em ?) no PCI - rtl8139 no Mass Storage (non VirtIO) PCI - SATA (AHCI) yes PCI - SCSI (LSI/Symbios sym) no **** Custom r356769 kernel w/ applied patches [4] from BR 236922 with netmap VirtIO PCIe - Console no PCIe - Memory Balloon no PCIe - Network no PCIe - SCSI no PCIe - Serial no NIC (non VirtIO NIC) PCI - e1000 Intel 82545EM (em) yes PCIe - e1000e Intel 82574 (em ?) no PCI - rtl8139 yes Mass Storage (non VirtIO) PCI - SATA (AHCI) yes PCI - SCSI (LSI/Symbios sym) yes **** Custom r356769 kernel w/ applied patches [4] from BR 236922 without netmap VirtIO PCIe - Console no PCIe - Memory Balloon no PCIe - Network no PCIe - SCSI no PCIe - Serial no NIC (non VirtIO NIC) PCI - e1000 Intel 82545EM (em) yes PCIe - e1000e Intel 82574 (em ?) no PCI - rtl8139 yes Mass Storage (non VirtIO) PCI - SATA (AHCI) yes PCI - SCSI (LSI/Symbios sym) yes If the hard drive is attached to an imported configuration from pre QEMU 4.x, all of the VirtIO works utilizing: -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 instead of the newer: -device pcie-pci-bridge,id=pci.2,bus=pci.1,addr=0x0 ----------------------------------------------------------------------------------- [4] BR 236922 - VirtIO support for PCIe in Q35 https://bugs.freebsd.org/bugzilla/attachment.cgi?id=210737 BR 236922 & 241774 - Disable VirtIO + netmap interop since netmap doesn't fully supports PCIe ATM. https://bugs.freebsd.org/bugzilla/attachment.cgi?id=210783
Created attachment 211053 [details] Data collection during tests Custom kernel QEMU 4.0 VM config dmesg verbose pciconf sysctl of hw.pci
(In reply to Tommy P from comment #3) Hi Tommy, I have also done testing on this: Q35 / v4 / OVMF / SATA / e1000 / e1000e / vmxnet3 / PCI Passthrough to Intel X550 10GbE / FreeBSD 12.1 with disable "dev netmap" (sys/amd64/conf/GENERIC): Results: SATA - OK e1000 - ok comes up as em0 e1000e - fails does not get detected (PCIe connected virtual device) vmxnet3 - ok comes up as vmx0 X550 10GbE - fails does not get detected (PCIe connected physical device) dmest error (as per yours): <<DMESG>> ... pcib2: <PCI-PCI bridge> mem 0xc8b87000-0xc8b87fff irq 22 at device 2.1 on pci0 pcib2: Failed to allocate interrupt for PCI-e events pcib3: <PCI-PCI bridge> mem 0xc8b86000-0xc8b86fff irq 22 at device 2.2 on pci0 pcib3: Failed to allocate interrupt for PCI-e events pcib4: <PCI-PCI bridge> mem 0xc8b85000-0xc8b85fff irq 22 at device 2.3 on pci0 pcib4: Failed to allocate interrupt for PCI-e events pcib5: <PCI-PCI bridge> mem 0xc8b84000-0xc8b84fff irq 22 at device 2.4 on pci0 pcib5: Failed to allocate interrupt for PCI-e events ... <<END DMESG>> ifconfig <<IFCONFIG>> # ifconfig -a lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=e403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 52:53:01:17:15:aa inet XX.XXX.XXX.53 netmask 0xffffff80 broadcast 203.XXX.XXX.127 media: Ethernet autoselect status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> em0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=81209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER> ether 52:54:00:a4:13:df media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> <<END IFCONFIG>> I then retested with except with QEMU Q35 V3.1: SATA - OK e1000 - ok comes up as em0 e1000e - ok comes up as em1 vmxnet3 - ok comes up as vmx0 X550 10GbE - ok comes up as ix0 ifconfig <<IFCONFIG>> # ifconfig -a vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=e403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 52:53:01:17:15:aa inet XXX.XXX.XXX.53 netmask 0xffffff80 broadcast XXX.XXX.XXX.127 media: Ethernet autoselect status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> em0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=81209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER> ether 52:54:00:a4:13:df media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> ix0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether b4:96:91:21:4a:ce media: Ethernet autoselect status: no carrier nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> em1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=81249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER> ether 52:54:00:f8:3b:94 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> <<END IFCONFIG>> Here is PCI passthrough XML snippet: <<LIBVIRT XML>> virsh dumpxml test-freebsd-12.1 <domain type='kvm' id='5'> <name>test-freebsd-12.1</name> <uuid>a50005d7-7425-435f-82e9-e76f18784693</uuid> <metadata> <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0"> <libosinfo:os id="http://freebsd.org/freebsd/12.0"/> </libosinfo:libosinfo> </metadata> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>2</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-3.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader> <nvram>/home/XXX/DIR/OVMF_VARS.fd</nvram> </os> <features> <acpi/> <apic/> <vmport state='off'/> </features> <cpu mode='custom' match='exact' check='full'> <model fallback='forbid'>Broadwell-IBRS</model> <vendor>Intel</vendor> <feature policy='require' name='vme'/> <feature policy='require' name='ss'/> <feature policy='require' name='vmx'/> <feature policy='require' name='f16c'/> <feature policy='require' name='rdrand'/> <feature policy='require' name='hypervisor'/> <feature policy='require' name='arat'/> <feature policy='require' name='tsc_adjust'/> <feature policy='require' name='umip'/> <feature policy='require' name='md-clear'/> <feature policy='require' name='stibp'/> <feature policy='require' name='arch-capabilities'/> <feature policy='require' name='ssbd'/> <feature policy='require' name='xsaveopt'/> <feature policy='require' name='pdpe1gb'/> <feature policy='require' name='abm'/> <feature policy='disable' name='skip-l1dfl-vmentry'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/home/XXX/DIR/test-hd1-01.qcow2'/> <backingStore/> <target dev='sda' bus='sata'/> <boot order='1'/> <alias name='sata0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/XXX/DIR/FreeBSD-12.1-RELEASE-amd64-dvd1.iso'/> <backingStore/> <target dev='sdb' bus='sata'/> <readonly/> <boot order='2'/> <alias name='sata0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> ... ... <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-to-pci-bridge'> <model name='pcie-pci-bridge'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0x11'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x12'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0x13'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0x14'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:18:15:aa'/> <source bridge='br20'/> <target dev='vnet0'/> <model type='vmxnet3'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </interface> <interface type='bridge'> <mac address='52:54:00:a4:13:df'/> <source bridge='br20'/> <target dev='vnet1'/> <model type='e1000'/> <alias name='net1'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/> </interface> <interface type='bridge'> <mac address='52:54:00:f8:3b:94'/> <source bridge='br20'/> <target dev='vnet2'/> <model type='e1000e'/> <alias name='net2'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </interface> ... ... <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/> </source> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </hostdev> ... ... </domain> <<END LIBVIRT XML>> So PCI Passthrough is affected by Q35 V4.0 issues and netmap is not root cause. Can we get FreeBSD PCI guru to look into QEMU V4.0 issue ? Cheers, John Hartley.
(In reply to Tommy P from comment #3) Yes, it's a combination of two or more issues. One of these issues is the netmap one, for which I provided a fix for head, stable/12 and stable/11.
Since this bug report is about the netmap PCI conflict, I think it would make sense to close it, since a fix has been provided for stable/11 (r356805).
(In reply to Vincenzo Maffione from comment #7) Hi Vincenzo, agree, not sure how to close this one but I have opened a new and more specific bug report on PCIe device section and QEMU Q35 V4.x: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=243640 Cheers, John Hartley.
(In reply to Vincenzo Maffione from comment #2) Hi Vincenzo, Not sure if you saw this blog post by Gerd Kraxel : https://www.kraxel.org/blog/2020/01/qemu-pci-ids/ It outline process to get QEMU pci ids , to avoid conflicts that we experienced with netmap . Process seems to be pretty straight forward and would ensure netplan gets long term ids that will get managed as part of QEMU code base. Cheers, John Hartley
I've seen it, thanks. However I plan to allocate the ids once (and only if) bhyve gets support for the netmap virtual devices.
(In reply to Vincenzo Maffione from comment #10) Hi Vincenzo, I have added Peter Grehan onto this bug based on your comment: "I've seen it, thanks. However I plan to allocate the ids once (and only if) bhyve gets support for the netmap virtual devices." Peter might be able to comment on possibility of adding netmap support into bhyve. Cheers, John Hartley
Hi John, Bhyve already supports netmap(4), since you can attach a bhyve VM to a vale(4) switch rather than attaching it to an if_bridge(4) through an if_tuntap(4) interface. No PCI IDs are necessary to support this. However, netmap also supports a "passthough" mode, that allows you to make a netmap port (of the host OS) visible within a VM. This particular feature requires support within libvmm API, and also requires the allocation of an id.