BUG: FreeBSD VM's on Ubuntu KVM / QEMU / OVMF host have SCSI Disk and Network Issues. Network Interface: em / qemu e1000 / Intel Pro 1G no longer works with 11.3 (it did work with 11.2) or 12.0 TESTING: I have tested various FreeBSD VM configuration and have found issues with SCSI and Network Device support when using: Ubuntu 19.04 / 19.10 KVM / QEMU Host with: Q35 Machine and OVMF EFI Firmware. Here is summary of testing: FreeBSD 11.2 (on KVM / QEMU Q35 with OVMF) SCSI disk - none work SATA disk - works Networking - only e1000 works FreeBSD 11.2 (on KVM / QEMU Q35 with BIOS) SCSI disk - works with lsisas1068 controller type Network - only e1000 works FreeBSD 11.3 (on KVM / QEMU Q35 with OVMF) SCSI disk - none work SATA disk - works Networking - nothing works NOTE: I also tested with custom kernel build (ie remove all bit minimal drivers but leaving MPT/MPS & VirtIO SCSI and intel em and VirtIO Networking, still does not work FreeBSD 12.0 (on KVM / QEMU with OVMF) SCSI disk - none works SATA disk - works Networking - nothing works So in summary, in move from FreeBSD 11.2 -> 11.3 / 12.X - networking is broken with KVM / QEMU / OVMF - SCSI disk support is broken for all KVM / QEMU / OVMF So while there is work around of SATA and BIOS, for new VMs, I have existing VMs which need to be upgraded and this cannot be done, as network or disk subsystem fails. I can do additional testing / validation if required.
Created attachment 208964 [details] dmesg with 3 disks SATA, SCSI/lsisas1068, VirtIO + 2 NICs e1000, VirtIO This attachment has the dmesg results after booting with: QEMU / KVM - Q35 with OVMF 3 Disks Attached: SATA - Found and ok SCSI with lsisas1068 controller - now found VirtIO - not found and 2 NICS e1000 VirtIO I may be trying to many things at once here but thought it might be simpler to try various disk at once.
Created attachment 208965 [details] /var/log/syslog with 3 Disks SATA, SCSI/lsisas1068, VirtIO + 2 NICs e1000, VirtIO This syslog from: QEMU / KVM - Q35 with OVMF FreeBSD 11.3 (regular Kernel) 3 Disks SATA - ok SCSI/lsisas1068 - not found VirtIO - not found 2 NICS e1000 - not found VirtIO - not found
Created attachment 208966 [details] pciconf with 3 Disks SATA, SCSI/lsisas1608, VirtIO + 2 NICs e1000, VirtIO Here is result from: pciconf -lcve FreeBSD 11.3 3 Disks SATA - ok SCSI/lsisas1068 - not found VirtIO - not found 2 NICs e1000 - not found VirtIO - not found
To single view of various variations of disk/nic, I created this VM with 3 disks and 2 NICS and collected the dmesg, syslog and pciconf results. Here is QEMU VM Desriptor for: Q35 with OVMF - FreeBSD 11.3 VM <domain type='kvm'> <name>newt-dev-freebsd-11.3</name> <uuid>d66dcbb4-d28e-4362-b3c5-486c85b8c4f3</uuid> <metadata> <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0"> <libosinfo:os id="http://freebsd.org/freebsd/11.2"/> </libosinfo:libosinfo> </metadata> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>2</vcpu> <os> <type arch='x86_64' machine='pc-q35-4.0'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader> <nvram>/home/XXX/OVMF_VARS.fd</nvram> </os> <features> <acpi/> <apic/> <vmport state='off'/> </features> <cpu mode='host-model' check='partial'> <model fallback='allow'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/home/newt-hd1-01.qcow2'/> <target dev='sdb' bus='sata'/> <boot order='1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/FreeBSD-11.3-RELEASE-amd64-dvd1.iso'/> <target dev='sdc' bus='sata'/> <readonly/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/home/newt-hd2-01.qcow2'/> <target dev='sda' bus='scsi'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/home/newt-hd3-01.qcow2'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/> </controller> <controller type='scsi' index='0' model='lsisas1068'> <address type='pci' domain='0x0000' bus='0x03' slot='0x02' function='0x0'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x11'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-to-pci-bridge'> <model name='pcie-pci-bridge'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x12'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0x13'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0x14'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0x15'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0x16'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:b6:23:1a'/> <source bridge='br01'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </interface> <interface type='bridge'> <mac address='52:54:00:98:20:03'/> <source bridge='br01'/> <model type='e1000'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='keyboard' bus='usb'> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' autoport='yes'> <listen type='address'/> <image compression='off'/> </graphics> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video> <redirdev bus='usb' type='spicevmc'> <address type='usb' bus='0' port='2'/> </redirdev> <redirdev bus='usb' type='spicevmc'> <address type='usb' bus='0' port='3'/> </redirdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </memballoon> </devices> </domain>
I'm also experiencing this issue. Discovered after upgrading 11.2 to 11.3 and e1000 network interface was no longer detected. Rolling back to 11.2 restored NIC. Reproduced the issue by downloading 11.3, 12, 12.1 ISOs and creating new basic VMs. None of the installers are able to detect emulated NIC. 11.2 installer works fine. Something clearly changed in 11.3+. Currently stuck on eol 11.2 :( Would love to at least see a workaround. host OS is unRAID 6.7.2(Slackware?) Q35-3.1 OVMF SATA e1000
Matts and others, I have a number of BSD related VMs running on KVM / QEMU / libvirt: MacOS (up to Catalina) with Clover & OVMF UEFI with NIC == e1000-82545em - runs well pfsense (2.4.3) Router with Q35 / OVMF with NIC == e1000 - runs well I have also tried: rtl8130 (Realtek) with 12.0 / OVMF - not working Given that MacOS works with Clover + OVMF, will test with Clover and see if this is able to aid getting NI?C going. MacOS / Darwin is BSD based so this is not so far fetched as it might initially seem. I resorted to using VMWare for BSD testing at the moment as this works with 11.3 with VMWare VMX specifies: e1000 NIC and lsisas SCSI disk (which is equivalent to KVM/QEMU lsisas1068 SCSI). In past I have been able to take VMWare vmdk images and run: > qemu-img convert -f vmdk -O qcow2 input.vmdk output.qcow2 And then by using e1000 & lsisas1068 they are bootable within KVM / QEMU. I believe that the NIC issues is more significant than the Disk one, as if SATA breaks then it will get instant focus, while NIC's don't get picked up until latter in testing. Given that BSD has strong source code culture the solution to my mind is to creates a ports package that be build and installed from within the host as kernel loadable module. So pending getting fix in based could get solution going independent of this. It might also be worth looking at VMTools for FreeBSD as I believe this is available as source code. I will report back once I have done further testing.
John, Have you made any progress? I'll add that my motherboard has 2 NICs and I passed through the I219-V (em driver), and have the same issue as the emulated e1000 NIC. 11.2 works fine, NIC not detected on 11.3+
Hi MattS, no progress sorry, as I hit catch-22... Do get the VMWare VMTools Port built, I needed access to network, which I do not have ... :-( A bit of Murphy's law hit as well, as at the same time as finding the 11.3 bug I also hit Ubuntu 19.10 kernel bug, which I also reported and have been working to get this resolved. Given current dead-lock with networking (and X-mas / new year break having passed), I might try: 1. Create 11.3 build with VMWare Tools installed via VMWare and convert this to KVM/QEMU VM via qemu-img utility and see if this works. 2. See if I can get full source code build going .... What we really need is person with serious FreeBSD Kernel / Driver contributor to weigh in on this bug... I report back on strategy (1) once later today, as this is relatively simple to try. Cheers, John
Hi Matts, did some additional testing in hope of find easy resolution. 1. Tested FreeBSD 11.3 on VMWare with vmxnet3 network interface This works with build in vmx driver that is already part of pre-build drivers so there is no need to explicitly install Open VMWare Tools to get vmxnet3 driver working. 2. Change QEMU / KVM network device type to: vmxnet3 I was not aware that VMWare vmxnet3 was an available option for QEMU, but found this when searching VMWare tools and QEMU / KVM. Result: still no network interface found... As counter point test I run 11.2 Live CD with vmxnet3 on QEMU / KVM and network interface was found. So there is definitely vmxnet3 support within QEMU / KVM / LibVirt on Ubuntu 19.10. Again problem appears to be 11.3 . So it looks like all potential quick work-around options are now closed off and only way to fix this is via kernel / driver changes. Cheers, John.
Thanks for the update. It's my understanding that Proxmox(host)is not experiencing this issue. I don't use Proxmox, but I guess it doesn't use libvirt. So maybe a clue there. Any idea on how to actually get some attention on this bug? I really hate being stuck on an EoL OS.
Hi John, I decided to post here instead of my bug report (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236922) since it adds OVMF testing. I've just created a OVMF + Q35 VM for 12.0 fresh install. Intel NIC driver works fine for me with the original default install kernel and custom updated kernel. Note the 'Hypervisor default' SCSI (sym0) loaded below: root@fbsd12-ovmf:~ # uname -a FreeBSD fbsd12-ovmf 12.0-RELEASE FreeBSD 12.0-RELEASE r341666 GENERIC amd64 root@fbsd12-ovmf:~ # dmesg | egrep -i 'scsi|mass stor|sym|sata' sym0: <895a> port 0xb000-0xb0ff mem 0x98c02000-0x98c023ff,0x98c00000-0x98c01fff irq 22 at device 0.0 on pci2 sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking pci3: <mass storage, SCSI> at device 0.0 (no driver attached) ahci0: <Intel ICH9 AHCI SATA controller> port 0xc240-0xc25f mem 0x99002000-0x99002fff irq 16 at device 31.2 on pci0 ada0: <QEMU HARDDISK 2.5+> ATA-7 SATA device ada0: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) cd0: <QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes) root@fbsd12-ovmf:~ # uname -a FreeBSD fbsd12-ovmf 12.0-RELEASE-p12 FreeBSD 12.0-RELEASE-p12 r356437 GENERIC amd64 Please note that all VirtIO connected devices still fail to load as per my bug report. Unfortunately, UEFI boot process is a bit more complicated and, thus, I can't simply switch the HDD bus from SATA to SCSI :(. When I tried a fresh install of 12.0 to the 'Hypervisor default' SCSI, the install is unable to detect the attached SCSI HDD... Exiting the install to shell to debug, sym driver is already loaded in the kernel but doesn't show up dmesg. Thus, unable to troubleshoot further with installing to or booting from the SCSI HDD. :( If I boot from SATA, all non-VirtIO works as expected: root@fbsd12-ovmf:~ # uname -a FreeBSD fbsd12-ovmf 12.0-RELEASE-p12 FreeBSD 12.0-RELEASE-p12 r356437 GENERIC amd64 root@fbsd12-ovmf:~ # dmesg | egrep -i 'scsi|mass stor|sym|sata|em0' sym0: <895a> port 0xb000-0xb0ff mem 0x98c02000-0x98c023ff,0x98c00000-0x98c01fff irq 22 at device 0.0 on pci2 sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking pci3: <mass storage, SCSI> at device 0.0 (no driver attached) em0: <Intel(R) PRO/1000 Network Connection> port 0x6000-0x603f mem 0x98000000-0x9801ffff irq 21 at device 1.0 on pci8 em0: attach_pre capping queues at 1 em0: using 1024 tx descriptors and 1024 rx descriptors em0: allocated for 1 tx_queues em0: allocated for 1 rx_queues em0: Ethernet address: 52:54:00:f2:70:9c ahci0: <Intel ICH9 AHCI SATA controller> port 0xc240-0xc25f mem 0x99002000-0x99002fff irq 16 at device 31.2 on pci0 da0 at sym0 bus 0 scbus0 target 2 lun 0 da0: <QEMU QEMU HARDDISK 2.5+> Fixed Direct Access SPC-3 SCSI device ada0: <QEMU HARDDISK 2.5+> ATA-7 SATA device ada0: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) cd0: <QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes) em0: link state changed to UP Perhaps something broken during the upgrade from 11.2? Is your boot/OS disk SATA or SCSI? Just a FIY, not all SCSI/RAID hardware HBA supports UEFI boot, IIRC. Thus, I don't think the emulated 'Hypervisor default' SCSI does either :( Here's my pciconf output when boot from SATA: none1@pci0:1:0:0: class=0x020000 card=0x11001af4 chip=0x10411af4 rev=0x01 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'Virtio network device' class = network subclass = ethernet bar [14] = type Memory, range 32, base rx98e00000, size 4096, enabled bar [20] = type Prefetchable Memory, range 64, base rx800000000, size 16384, enabled cap 11[dc] = MSI-X supports 3 messages Table in map 0x14[0x0], PBA in map 0x14[0x800] cap 09[c8] = vendor (length 20) cap 09[b4] = vendor (length 20) cap 09[a4] = vendor (length 16) cap 09[94] = vendor (length 16) cap 09[84] = vendor (length 16) cap 01[7c] = powerspec 3 supports D0 D3 current D0 cap 10[40] = PCI-Express 2 endpoint max data 128(128) link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) sym0@pci0:2:0:0: class=0x010000 card=0x10000000 chip=0x00121000 rev=0x00 hdr=0x00 vendor = 'LSI Logic / Symbios Logic' device = '53c895a' class = mass storage subclass = SCSI bar [10] = type I/O Port, range 32, base rxb000, size 256, enabled bar [14] = type Memory, range 32, base rx98c02000, size 1024, enabled bar [18] = type Memory, range 32, base rx98c00000, size 8192, enabled none2@pci0:3:0:0: class=0x010000 card=0x11001af4 chip=0x10481af4 rev=0x01 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'Virtio SCSI' class = mass storage subclass = SCSI bar [14] = type Memory, range 32, base rx98a00000, size 4096, enabled bar [20] = type Prefetchable Memory, range 64, base rx800100000, size 16384, enabled cap 11[dc] = MSI-X supports 4 messages Table in map 0x14[0x0], PBA in map 0x14[0x800] cap 09[c8] = vendor (length 20) cap 09[b4] = vendor (length 20) cap 09[a4] = vendor (length 16) cap 09[94] = vendor (length 16) cap 09[84] = vendor (length 16) cap 01[7c] = powerspec 3 supports D0 D3 current D0 cap 10[40] = PCI-Express 2 endpoint max data 128(128) link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) BTW, my host OS is an updated Ubuntu 18.04.3 LTS. I don't know if that has any bearing vs 19.x
Hi Tommy T, I am a bit puzzled, you our posting... Your pciconf (below): <<START>> none1@pci0:1:0:0: class=0x020000 card=0x11001af4 chip=0x10411af4 rev=0x01 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'Virtio network device' class = network subclass = ethernet bar [14] = type Memory, range 32, base rx98e00000, size 4096, enabled bar [20] = type Prefetchable Memory, range 64, base rx800000000, size 16384, enabled cap 11[dc] = MSI-X supports 3 messages Table in map 0x14[0x0], PBA in map 0x14[0x800] cap 09[c8] = vendor (length 20) cap 09[b4] = vendor (length 20) cap 09[a4] = vendor (length 16) cap 09[94] = vendor (length 16) cap 09[84] = vendor (length 16) cap 01[7c] = powerspec 3 supports D0 D3 current D0 cap 10[40] = PCI-Express 2 endpoint max data 128(128) link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) sym0@pci0:2:0:0: class=0x010000 card=0x10000000 chip=0x00121000 rev=0x00 hdr=0x00 vendor = 'LSI Logic / Symbios Logic' device = '53c895a' class = mass storage subclass = SCSI bar [10] = type I/O Port, range 32, base rxb000, size 256, enabled bar [14] = type Memory, range 32, base rx98c02000, size 1024, enabled bar [18] = type Memory, range 32, base rx98c00000, size 8192, enabled none2@pci0:3:0:0: class=0x010000 card=0x11001af4 chip=0x10481af4 rev=0x01 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'Virtio SCSI' class = mass storage subclass = SCSI bar [14] = type Memory, range 32, base rx98a00000, size 4096, enabled bar [20] = type Prefetchable Memory, range 64, base rx800100000, size 16384, enabled cap 11[dc] = MSI-X supports 4 messages Table in map 0x14[0x0], PBA in map 0x14[0x800] cap 09[c8] = vendor (length 20) cap 09[b4] = vendor (length 20) cap 09[a4] = vendor (length 16) cap 09[94] = vendor (length 16) cap 09[84] = vendor (length 16) cap 01[7c] = powerspec 3 supports D0 D3 current D0 cap 10[40] = PCI-Express 2 endpoint max data 128(128) link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) <<END>> This shows VirtIO SCSI & VirtIO Network... but you state that you cannot get VirtIO going (which is same problem I have). Is this pciconf report from i440FX QEMU/KVM machine rather than a Q35 one ? I have not managed to flip my test VM from Q35 to i440FX as per discussion in your bug thread yet as I got into large regression build. I am now part way through building new kernel that is 11.3 with mutant mostly 11.2 networking sub-system. I say mostly 11.2 networking, as I believe that the VirtIO network version ID bug you identified in combination of possible introduction of defect with changes in the netmap 11.3 code. This is why your fix was not enough and why all the network interfaces are broken not just VirtIO. My mutant 11.3 Kernel + 11.2 Network build got much bigger than expected as the 11.3 netmap code results in ripple through impact down into all of the driver code and also up into the network abstraction layers :-( So when I wound back to 11.2 netmap I have to also wind back on these impacts. I am almost there I believe this this "regressed 11.3" custom kernel. If I am right about netmap problem (I hope I am ...), then I can home on the specific issue with this code section. I hope to test soon, after many many builds... Thanks again for your tips and insights. Cheers, John
(In reply to John Hartley from comment #12) Hi John, If you look again at the pciconf output: none1@pci0:1:0:0 Implies that no driver were loaded for that pci device at slot 0:1:0:0 since FreeBSD's VirtIO drivers currently does not support PCI-E bus. The rest of the pciconf are for debugging driver. For the other output such as: sym0@pci0:2:0:0 Implies that FreeBSD's LSI/Symbios SCSI (sym) were loaded successfully. Thus, any HDD attached to that controller would work. Unfortunately, I don't think this SCSI controller supports UEFI boot. Here are the details of pciconf, from my understanding, of how FreeBSD's VirtIO drivers work: none1@pci0:1:0:0: class=0x020000 card=0x11001af4 chip=0x10411af4 rev=0x01 hdr=0x00 *) rev=0x01 implies PCI-E bus while rev=0x00 implies PCI bus *) last 4 digits of card/chip implies the vendor ID: 0x1af4 (per /usr/src/sys/dev/virtio/pci/virtio_pci.h) *) if device is on PCI bus, the first 4 digits of 'card' imply device type (per /usr/src/sys/dev/virtio/virtio_ids.h) *) if device is on PCI-E bus, the first 4 digits of 'chip' - 0x1040 should match device type from /usr/src/sys/dev/virtio/virtio_ids.h. Thus, 0x1041 - 0x1040 = 0x1 which is VirtIO NIC per pciconf's verbose output: #define VIRTIO_ID_NETWORK 1 And 0x1048 - 0x1040 = 0x8 which is VirtIO SCSI: #define VIRTIO_ID_SCSI 8 The current VirtIO drivers does not have the necessary codes to detect the device ID connected to the PCI-E bus nor have the correct memory address to attach the driver for that same bus. As for non VirtIO not working, my fresh install of 12.0 using OVMF works OK for both sym SCSI and em NIC. I don't have 11.2 OVMF to test upgrade. But all of my non OVMF VMs work OK too except where Q35 + VirtIO are concerned as per my bug report.
Hi Tommy T, thanks again for into. Your freebsd 12.0 pciconf is definitely different to mine. Is this due to your debug kernel reporting differently to standard GENERIC ? I am now just doing Live CD boots to see what works and what doesn't as it avoids having to go through lots of install steps that just result in failed install... ;-) BTW: here is my Ubuntu and QEMU version info: > virsh -c qemu:///system version --daemon Compiled against library: libvirt 5.4.0 Using library: libvirt 5.4.0 Using API: QEMU 5.4.0 Running hypervisor: QEMU 4.0.0 Running against daemon: 5.4.0 uname -a Linux blue 5.3.0-26-generic #28-Ubuntu SMP Wed Dec 18 05:37:46 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux $ cat /etc/*-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=19.10 DISTRIB_CODENAME=eoan DISTRIB_DESCRIPTION="Ubuntu 19.10" NAME="Ubuntu" VERSION="19.10 (Eoan Ermine)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 19.10" VERSION_ID="19.10" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=eoan UBUNTU_CODENAME=eoan Still working on my GENERIC 11.3 (with 11.2) build. I am almost there now. Cheers, John,
(In reply to John Hartley from comment #14) Hi John, I used pciconf -lBbcevV. As for the kernel, I only commented out unneeded stuff in GENERIC like SCSI/RAID adapters, NICs, WLAN, xen & vmware, netmap, debug, etc. root@d-build-fbsd12:/usr/obj # du -hsc /boot/kernel* 96M /boot/kernel 96M /boot/kernel.old 132M /boot/kernel.ori.12.0 93M /boot/kernel.virtio 418M total My host is Ubuntu 18.04.3 LTS so the KVM/QEMU version will be older than yours. HTH, Tommy
(In reply to John Hartley from comment #14) Hi John, Good news! After some debugging, I was able to get VirtIO to work with Q35 with OVMF for 12.0-RELEASE-p12 r356438M and without OVMF for 12.0-RELEASE-p12 r356535M! Please see my bug report (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236922) for the attachment and caution regarding netmap. Here's the pciconf output for the VirtIO: vtpcim0@pci0:1:0:0: class=0x020000 card=0x11001af4 chip=0x10411af4 rev=0x01 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'Virtio network device' class = network subclass = ethernet bar [14] = type Memory, range 32, base rx98e00000, size 4096, enabled bar [20] = type Prefetchable Memory, range 64, base rx800000000, size 16384, enabled cap 11[dc] = MSI-X supports 3 messages, enabled Table in map 0x14[0x0], PBA in map 0x14[0x800] cap 09[c8] = vendor (length 20) cap 09[b4] = vendor (length 20) cap 09[a4] = vendor (length 16) cap 09[94] = vendor (length 16) cap 09[84] = vendor (length 16) cap 01[7c] = powerspec 3 supports D0 D3 current D0 cap 10[40] = PCI-Express 2 endpoint max data 128(128) link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) sym0@pci0:2:0:0: class=0x010000 card=0x10000000 chip=0x00121000 rev=0x00 hdr=0x00 vendor = 'LSI Logic / Symbios Logic' device = '53c895a' class = mass storage subclass = SCSI bar [10] = type I/O Port, range 32, base rxb000, size 256, enabled bar [14] = type Memory, range 32, base rx98c02000, size 1024, enabled bar [18] = type Memory, range 32, base rx98c00000, size 8192, enabled vtpcim1@pci0:3:0:0: class=0x010000 card=0x11001af4 chip=0x10481af4 rev=0x01 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'Virtio SCSI' class = mass storage subclass = SCSI bar [14] = type Memory, range 32, base rx98a00000, size 4096, enabled bar [20] = type Prefetchable Memory, range 64, base rx800100000, size 16384, enabled cap 11[dc] = MSI-X supports 4 messages, enabled Table in map 0x14[0x0], PBA in map 0x14[0x800] cap 09[c8] = vendor (length 20) cap 09[b4] = vendor (length 20) cap 09[a4] = vendor (length 16) cap 09[94] = vendor (length 16) cap 09[84] = vendor (length 16) cap 01[7c] = powerspec 3 supports D0 D3 current D0 cap 10[40] = PCI-Express 2 endpoint max data 128(128) link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) vtpcim2@pci0:4:0:0: class=0x078000 card=0x11001af4 chip=0x10431af4 rev=0x01 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'Virtio console' class = simple comms bar [14] = type Memory, range 32, base rx98800000, size 4096, enabled bar [20] = type Prefetchable Memory, range 64, base rx800200000, size 16384, enabled cap 11[dc] = MSI-X supports 2 messages Table in map 0x14[0x0], PBA in map 0x14[0x800] cap 09[c8] = vendor (length 20) cap 09[b4] = vendor (length 20) cap 09[a4] = vendor (length 16) cap 09[94] = vendor (length 16) cap 09[84] = vendor (length 16) cap 01[7c] = powerspec 3 supports D0 D3 current D0 cap 10[40] = PCI-Express 2 endpoint max data 128(128) link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) Note that previously it was none#@pci now it's vtpcim#@pci. It also seems to support booting from both SATA and VirtIO Serial: root@fbsd12-ovmf:~ # ls /dev/{da,ada,vtbd}* /dev/vtbd0 /dev/vtbd0p1 /dev/vtbd0p2 root@fbsd12-ovmf:~ # ifconfig vtnet0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6> ether ... media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=81209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER> ether ... inet ... netmask 0xffffff00 broadcast 10.100.0.255 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> This was a fresh install of 12.0-RELEASE onto SATA. After replacing and patching /usr/src in addition to kernel rebuild, it now boots as VirtIO Serial!
This is great work! Can this patch be applied to 11.3, or easily created for 11.3?
(In reply to Tommy P from comment #16) Hi Tommy P, Hooray great to see you have got VirtIO sorted !! I have now got my mutant 11.3 with 11.2 network kernel going and ... It works !! uname -a FreeBSD newt.in.graphica.com.au 11.3-RELEASE FreeBSD 11.3-RELEASE #26: Thu Jan 9 18:38:41 AEDT 2020 root@newt.in.graphica.com.au:/usr/obj/usr/src/sys/GENERIC2 amd64 Only tested e1000 at the moment: % ifconfig -a lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff000000 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> groups: lo em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC> ether 52:54:00:4e:50:91 hwaddr 52:54:00:4e:50:91 inet 192.168.73.131 netmask 0xffffff00 broadcast 192.168.73.255 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active My approach was a bit sledge hammer as I have ended up with the following modules all being regressed back to 11.2: % find . -name '*.bak*' -print ./sbin/ipfw.bak ./sys/modules/mlx5en.bak ./sys/modules/netmap/Makefile.bak ./sys/modules/cxgbe.bak ./sys/modules/cxgbe.bak/if_cxgbe/Makefile.bak ./sys/modules/ipfw_nat64.bak ./sys/modules/mlx5.bak ./sys/modules/urtwn.bak ./sys/amd64/vmm.bak ./sys/amd64/pci.bak ./sys/dev/virtio/network.bak ./sys/dev/vmware/vmxnet3.bak ./sys/dev/pci/pcireg.h.bak.11.2 ./sys/dev/cxgbe/firmware.bak.11.3 ./sys/dev/usb/wlan.bak ./sys/dev/pci.bak ./sys/dev/netmap.bak ./sys/dev/e1000.bak ./sys/dev/ixgbe.bak ./sys/dev/re.bak ./sys/dev/iwi.bak ./sys/dev/malo.bak ./sys/dev/mwl.bak ./sys/dev/ral.bak ./sys/dev/ixl.bak ./sys/dev/bwi.bak ./sys/dev/cxgbe.bak ./sys/dev/mlx5.bak ./sys/dev/oce.bak ./sys/dev/rtwn.bak ./sys/dev/urtwn.bak ./sys/conf/files.bak ./sys/net.bak ./sys/netinet.bak ./sys/net80211.bak ./sys/netinet6.bak ./sys/netpfil.bak The need to regress so much was due ripple up / down impacts of /dev/netmap . As you can see this is pretty much then entire network subsystem... :-( I am only able to post this as now network is up I can ssh into box to get result out. First though I am going to test all the other network devices... Now done: % ifconfig -a lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff000000 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> groups: lo em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC> ether 52:54:00:4e:50:91 hwaddr 52:54:00:4e:50:91 inet 192.168.73.131 netmask 0xffffff00 broadcast 192.168.73.255 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active vmx0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6> ether 52:54:00:0a:cc:0d hwaddr 52:54:00:0a:cc:0d nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect status: active re0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE> ether 52:54:00:b2:20:53 hwaddr 52:54:00:b2:20:53 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect (100baseTX <full-duplex,flowcontrol,rxpause,txpause>) status: active So I have: e1000 - intel - em0 vmxnet3 - VMWare vmxnet3 Virtual NIC driver (VirtIO alternate) - vmx0 rtl8139 - RealTec - re0 This test also include VirtIO, but as per your testing this does not work and I need to include your fix. So in summary: Issue is not with QEMU / KVM , but with FreeBSD code from 11.2 -> 11.3 which 12.x has inherited There are two bugs: 1. confirmed - VirtIO - the one Tommy T has helped resolve 2. Speculatively - /dev/netmap bug or else where Bug (2) is that is breaking on the other Q35 (non VirtIO) interfaces (e1000, vmxnet3, rtl8139). Now that I have a working base I am going for first move forward the PCI code and validate that is ok and then look at the netmap code, which I believe is what is causing the issue with all the other network devices. Once I get to specific commit/s that introduced the bug I will provide update. BTW - I am really surprise that such a large and impactful change on network sub-system was part of minor release cycle and nothing was mentioned in the release notes. Cheers and thanks again Tommy. John Hartley.
Tommy P & Matt S, wound forward PCI code to 11.3 and network still working ok: % find . -name '*.bak*' -print ./sbin/ipfw.bak ./sys/modules/mlx5en.bak ./sys/modules/netmap/Makefile.bak ./sys/modules/cxgbe.bak ./sys/modules/cxgbe.bak/if_cxgbe/Makefile.bak ./sys/modules/ipfw_nat64.bak ./sys/modules/mlx5.bak ./sys/modules/urtwn.bak ./sys/amd64/pci.bak.11.2 ./sys/amd64/vmm.bak.11.2 ./sys/dev/virtio/network.bak ./sys/dev/vmware/vmxnet3.bak ./sys/dev/cxgbe/firmware.bak.11.3 ./sys/dev/usb/wlan.bak ./sys/dev/netmap.bak ./sys/dev/e1000.bak ./sys/dev/ixgbe.bak ./sys/dev/re.bak ./sys/dev/iwi.bak ./sys/dev/malo.bak ./sys/dev/mwl.bak ./sys/dev/ral.bak ./sys/dev/ixl.bak ./sys/dev/bwi.bak ./sys/dev/cxgbe.bak ./sys/dev/mlx5.bak ./sys/dev/oce.bak ./sys/dev/rtwn.bak ./sys/dev/urtwn.bak ./sys/dev/pci.bak.11.2 ./sys/dev/pci.bak.11.2/pcireg.h.bak.11.2 ./sys/conf/files.bak ./sys/net.bak ./sys/netinet.bak ./sys/net80211.bak ./sys/netinet6.bak ./sys/netpfil.bak All (non VirtIO network ok): % ifconfig -a lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff000000 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> groups: lo em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC> ether 52:54:00:4e:50:91 hwaddr 52:54:00:4e:50:91 inet 192.168.73.131 netmask 0xffffff00 broadcast 192.168.73.255 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active vmx0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6> ether 52:54:00:0a:cc:0d hwaddr 52:54:00:0a:cc:0d nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect status: active re0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE> ether 52:54:00:b2:20:53 hwaddr 52:54:00:b2:20:53 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect (100baseTX <full-duplex,flowcontrol,rxpause,txpause>) So this confirm that source of issue is in the /dev/netmap code. I will try to wind this forward on a commit by commit basis to see where it breaks. Cheers, John Hartley.
Hi Tommy T, some helpful advise here please, did diff of netmap code from 11.2 -> 11.3: % diff netmap netmap.bak | wc 15891 67956 464277 Files 19 -> 25: % ls netmap.bak | wc 25 25 371 % ls netmap | wc 19 19 288 and grep of FBSDIDs: diff netmap netmap.bak | grep FBSDID < __FBSDID("$FreeBSD: releng/11.2/sys/dev/netmap/netmap_generic.c 331722 2018-03-29 02:50:57Z eadler $"); > __FBSDID("$FreeBSD: releng/11.3/sys/dev/netmap/netmap_generic.c 344047 2019-02-12 09:26:05Z vmaffione $"); < __FBSDID("$FreeBSD: releng/11.2/sys/dev/netmap/netmap_mem2.c 331722 2018-03-29 02:50:57Z eadler $"); > __FBSDID("$FreeBSD: releng/11.3/sys/dev/netmap/netmap_mem2.c 344047 2019-02-12 09:26:05Z vmaffione $"); < __FBSDID("$FreeBSD: releng/11.2/sys/dev/netmap/netmap_vale.c 331722 2018-03-29 02:50:57Z eadler $"); > __FBSDID("$FreeBSD: releng/11.3/sys/dev/netmap/netmap_vale.c 344047 2019-02-12 09:26:05Z vmaffione $"); Given the amount of change it would seem prudent to get committer to look at this: vmaffione What are your thought and how to we get additional eyes on this ? Thanks. Cheers, John Hartley.
Can you please elaborate about the bug that you think there is in netmap? The large changes are due to refactoring and new features, but it is nothing that should affect driver operation when the NIC is not open in netmap mode.
(In reply to Vincenzo Maffione from comment #21) Hi Vincenzo, thank you very much of for responding. My testing has been to try to isolate where the problems lies. The issues appears when you go from 11.2 -> 11.3 . As the problem relates to device detection / initialisation, my initial thought was that problem might be in PCI/PCIe code. To test this I tried to create a custom 11.3 kernel with 11.2 PCI subsystem. The impact of regression to 11.2 PCI was that I also had to regress netmap, network and driver code back to 11.2 as well. The resulting 11.3 (with 11.2 pci / networking) kernel no longer had the issue (ie e1000, vmxnet3 & rtl8139 all working ok) So I then rolled forward the pci code back to 11.3 . Again testing worked with each of e1000, vmxnet3 & rtl8139 working. So this seemed to have issue as being in netmap or drivers ... In looking at record of code updates, the netmap related changes (refactoring etc) and adding support for netmap in e1000 & vmxnet3 drivers all related to MFC r339639 (which I believe was big upstream netmap alignment and refactoring). So right now I have not confirmed that the bug is in netmap, but it might be... On a related front this VirtIO bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236922 required changes to both VirtIO driver and netmap. Right now I am a bit stuck in finding the defect, as netmap and driver code is more complex and requires deeper understand than I have. My objective was just try to pin point where the problem was. Having looked at this I now have a much greater appreciation of what the dev/netmap code does and why it impacts such a large portion of network code. On my working 11.3 / 11.2 custom kernel here is dmesg result from em0: em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x9000-0x903f mem 0xc8600000-0xc861ffff irq 23 at device 1.0 on pci2 em0: Ethernet address: 52:54:00:4e:50:91 em0: netmap queues/slots: TX 1/256, RX 1/256 So it appears that netmap is enabled by default. The only configuration I have done is set DHCP on: # cat /etc/rc.conf clear_tmp_enable="YES" ... ifconfig_em0="DHCP" local_unbound_enable="YES" sshd_enable="YES" # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable dumpdev="AUTO" Right now I have not managed to get problem isolated so specific code item... Your advise on helping to get to the bottom of problem would be appreciated. Cheers, John Hartley
@John, Are you encountering issues while upgrading 11.2 to 11.3 with Q35 + OVMF? I don't have any issues: root@fbsd11-ovmf:~ # dmesg|grep em0 em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x6000-0x603f mem 0x98000000-0x9801ffff irq 21 at device 1.0 on pci8 em0: Ethernet address: 52:54:00:4e:d4:8f em0: netmap queues/slots: TX 1/256, RX 1/256 em0: link state changed to UP root@fbsd11-ovmf:~ # uname -a FreeBSD fbsd11-ovmf 11.2-RELEASE-p15 FreeBSD 11.2-RELEASE-p15 #0 r356556: Thu Jan 9 19:50:57 PST 2020 --------------------------------------------------- root@fbsd11-ovmf:~ # dmesg | egrep 'sym0|mass stor|scsi' sym0: <895a> port 0xb000-0xb0ff mem 0x98c02000-0x98c023ff,0x98c00000-0x98c01fff irq 22 at device 0.0 on pci2 sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking pci3: <mass storage, SCSI> at device 0.0 (no driver attached) root@fbsd11-ovmf:~ # dmesg|grep em0 em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x6000-0x603f mem 0x98000000-0x9801ffff irq 21 at device 1.0 on pci2 em0: Ethernet address: 52:54:00:4e:d4:8f em0: netmap queues/slots: TX 1/256, RX 1/256 em0: link state changed to UP root@fbsd11-ovmf:~ # uname -a FreeBSD fbsd11-ovmf 11.3-RELEASE-p5 FreeBSD 11.3-RELEASE-p5 #0: Tue Nov 12 08:59:04 UTC 2019 VirtIO still doesn't work for 11.x :( The pci3 is the VirtIO SCSI (note the 'no driver attached'). @Matt, (from comment #17) From what I've seen so far, there are significant changes in the networking from 11.x to 12.x which the VirtIO, supporting PCI-E, is based upon. I doubt it's going to be easy to 'back port' to 11.x. Regards, Tommy
(In reply to John Hartley from comment #22) MFC r339639 does not add netmap support to e1000, vtnet & co. Such support exists since FreeBSD 10. Moreover, netmap code is platform independent, so it does not make sense that a bug in netmap would show up in Q35 and not in the other platform. It is more reasonable that the issue is related to the PCI code. If I get it correctly, your issue is that you don't detect devices (e.g. you don't see em0 or vtnet0 in ifconfig). If this happens, it means that the device_probe routine of the proper driver is not called, or maybe it is called but it does not recognize the device for some reason. For example, you have em_probe() or lem_probe() for e1000 devices. You may add a print statement there to see whether it is called or not. Note that device probe is a functionality related to the PCI/PCIe bus, irrespective of netmap. Have you tried to remove "dev netmap" from the kernel config (for a clean 11.3)? This will confirm that the problem is still there irrespective of netmap being included or not.
(In reply to Vincenzo Maffione from comment #24) Hi Vincenzo, as suggested, I have compiled kernel with netmap removed: # diff GENERIC GENERIC2 22c22 < ident GENERIC --- > ident GENERIC2 360c360 < device netmap # netmap(4) support --- > #device netmap # netmap(4) support # uname -a FreeBSD newt.in.graphica.com.au 11.3-RELEASE FreeBSD 11.3-RELEASE #31: Sat Jan 11 08:25:05 AEDT 2020 root@newt.in.graphica.com.au:/usr/obj/usr/src/sys/GENERIC2 amd64 I had previously tried to disable netmap from e1000 only and still get result of no network devices being found. Now with netmap removed from kernel the e1000 device is found: # ifconfig -a lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff000000 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> groups: lo em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC> ether 52:54:00:4e:50:91 hwaddr 52:54:00:4e:50:91 inet 192.168.73.131 netmask 0xffffff00 broadcast 192.168.73.255 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active Here is PCI Conf results: # pciconf -lvce hostb0@pci0:0:0:0: class=0x060000 card=0x11001af4 chip=0x29c08086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82G33/G31/P35/P31 Express DRAM Controller' class = bridge subclass = HOST-PCI vgapci0@pci0:0:1:0: class=0x030000 card=0x11001af4 chip=0x01001b36 rev=0x04 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'QXL paravirtual graphic card' class = display subclass = VGA pcib1@pci0:0:2:0: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x1(x32) speed 2.5(undef) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(engaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 000d[148] = ACS 1 pcib2@pci0:0:2:1: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x1(x32) speed 2.5(undef) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 000d[148] = ACS 1 pcib3@pci0:0:2:2: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x1(x32) speed 2.5(undef) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 000d[148] = ACS 1 pcib4@pci0:0:2:3: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x32(x32) speed undef(undef) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(empty) surprise Attn Button PC(off) EI(disengaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 000d[148] = ACS 1 uhci0@pci0:0:29:0: class=0x0c0300 card=0x11001af4 chip=0x29348086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB UHCI Controller' class = serial bus subclass = USB uhci1@pci0:0:29:1: class=0x0c0300 card=0x11001af4 chip=0x29358086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB UHCI Controller' class = serial bus subclass = USB uhci2@pci0:0:29:2: class=0x0c0300 card=0x11001af4 chip=0x29368086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB UHCI Controller' class = serial bus subclass = USB ehci0@pci0:0:29:7: class=0x0c0320 card=0x11001af4 chip=0x293a8086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB2 EHCI Controller' class = serial bus subclass = USB isab0@pci0:0:31:0: class=0x060100 card=0x11001af4 chip=0x29188086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82801IB (ICH9) LPC Interface Controller' class = bridge subclass = PCI-ISA ahci0@pci0:0:31:2: class=0x010601 card=0x11001af4 chip=0x29228086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]' class = mass storage subclass = SATA cap 05[80] = MSI supports 1 message, 64 bit enabled with 1 message cap 12[a8] = SATA Index-Data Pair none0@pci0:0:31:3: class=0x0c0500 card=0x11001af4 chip=0x29308086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) SMBus Controller' class = serial bus subclass = SMBus pcib5@pci0:1:0:0: class=0x060400 card=0x00000000 chip=0x000e1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' class = bridge subclass = PCI-PCI cap 05[8c] = MSI supports 1 message, 64 bit, vector masks cap 01[84] = powerspec 3 supports D0 D3 current D0 cap 10[48] = PCI-Express 2 PCI bridge max data 128(128) ARI disabled link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) cap 0c[40] = unknown ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected em0@pci0:2:1:0: class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82540EM Gigabit Ethernet Controller' class = network subclass = ethernet So there is some subtle interaction happing across PCI discovery and netmap code. I will now add other adaptors into test and post additional results shortly. Thanks for the advise it really helped with diagnosis. Cheers, John Hartley.
(In reply to Vincenzo Maffione from comment #24) Hi Vincenzo, here are the results for e1000 (em), vmxnet3 (vmx) virtio (vtnet) and rtl813 (re): # ifconfig -a lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff000000 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> groups: lo em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC> ether 52:54:00:4e:50:91 hwaddr 52:54:00:4e:50:91 inet 192.168.73.131 netmask 0xffffff00 broadcast 192.168.73.255 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active vmx0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6> ether 52:54:00:fb:be:74 hwaddr 52:54:00:fb:be:74 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect status: active re0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE> ether 52:54:00:b9:0c:12 hwaddr 52:54:00:b9:0c:12 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect (100baseTX <full-duplex,flowcontrol,rxpause,txpause>) status: active And PCI Conf: # pciconf -lcve hostb0@pci0:0:0:0: class=0x060000 card=0x11001af4 chip=0x29c08086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82G33/G31/P35/P31 Express DRAM Controller' class = bridge subclass = HOST-PCI vgapci0@pci0:0:1:0: class=0x030000 card=0x11001af4 chip=0x01001b36 rev=0x04 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'QXL paravirtual graphic card' class = display subclass = VGA pcib1@pci0:0:2:0: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x1(x32) speed 2.5(undef) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(engaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 000d[148] = ACS 1 pcib2@pci0:0:2:1: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x1(x32) speed 2.5(undef) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 000d[148] = ACS 1 pcib3@pci0:0:2:2: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x1(x32) speed 2.5(undef) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 000d[148] = ACS 1 pcib4@pci0:0:2:3: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x1(x32) speed 2.5(undef) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 000d[148] = ACS 1 uhci0@pci0:0:29:0: class=0x0c0300 card=0x11001af4 chip=0x29348086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB UHCI Controller' class = serial bus subclass = USB uhci1@pci0:0:29:1: class=0x0c0300 card=0x11001af4 chip=0x29358086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB UHCI Controller' class = serial bus subclass = USB uhci2@pci0:0:29:2: class=0x0c0300 card=0x11001af4 chip=0x29368086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB UHCI Controller' class = serial bus subclass = USB ehci0@pci0:0:29:7: class=0x0c0320 card=0x11001af4 chip=0x293a8086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB2 EHCI Controller' class = serial bus subclass = USB isab0@pci0:0:31:0: class=0x060100 card=0x11001af4 chip=0x29188086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82801IB (ICH9) LPC Interface Controller' class = bridge subclass = PCI-ISA ahci0@pci0:0:31:2: class=0x010601 card=0x11001af4 chip=0x29228086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]' class = mass storage subclass = SATA cap 05[80] = MSI supports 1 message, 64 bit enabled with 1 message cap 12[a8] = SATA Index-Data Pair none0@pci0:0:31:3: class=0x0c0500 card=0x11001af4 chip=0x29308086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) SMBus Controller' class = serial bus subclass = SMBus pcib5@pci0:1:0:0: class=0x060400 card=0x00000000 chip=0x000e1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' class = bridge subclass = PCI-PCI cap 05[8c] = MSI supports 1 message, 64 bit, vector masks cap 01[84] = powerspec 3 supports D0 D3 current D0 cap 10[48] = PCI-Express 2 PCI bridge max data 128(128) ARI disabled link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) cap 0c[40] = unknown ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected em0@pci0:2:1:0: class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82540EM Gigabit Ethernet Controller' class = network subclass = ethernet vmx0@pci0:2:2:0: class=0x020000 card=0x07b015ad chip=0x07b015ad rev=0x01 hdr=0x00 vendor = 'VMware' device = 'VMXNET3 Ethernet Controller' class = network subclass = ethernet cap 11[9c] = MSI-X supports 25 messages, enabled Table in map 0x18[0x0], PBA in map 0x18[0x1000] cap 05[84] = MSI supports 1 message, 64 bit re0@pci0:2:3:0: class=0x020000 card=0x11001af4 chip=0x813910ec rev=0x20 hdr=0x00 vendor = 'Realtek Semiconductor Co., Ltd.' device = 'RTL-8100/8101L/8139 PCI Fast Ethernet Adapter' class = network subclass = ethernet All devices discovered with exception of virtio. I will put in fix that Tommy T found for virtio and retest. I am glad that we appear to be making progress in diagnosing issue. Cheers, John Hartley.
(In reply to John Hartley from comment #25) Just to double check: you are using 11.3-release, and not your "frankestein kernel" (11.3 with backported code from 11.2)? Assuming a positive answer, you are saying that with GENERIC you don't see em0 showing up, and if you remove "dev netmap" (GENERIC2) then em0 shows up? That would be very weird... Can you point me at the revision that you used for the build?
(In reply to Vincenzo Maffione from comment #27) Hi Vincenzo, the 11.3 kernel I compiled was standard (ie not my 11.3 / 11.2 Frankenstein) only change to GENERIC being "device netmap" disablement. Result was all Q35 network devices: e1000 (em), vmxnet3 (vmx) and rtl8139 (re) are working. Now I have isolated problem I have "retired" Frankenstein ;-) I have also now done equivalent kernel build with 12.0 and get the same result: # uname -a FreeBSD agent.in.graphica.com.au 12.0-RELEASE FreeBSD 12.0-RELEASE GENERIC2 amd64 # diff /usr/src/sys/amd64/conf/GENERIC2 /usr/src/sys/amd64/conf/GENERIC 22c22 < ident GENERIC2 --- > ident GENERIC 362c362 < #device netmap # netmap(4) support --- > device netmap # netmap(4) support # ifconfig -a vmx0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6> ether 52:54:00:07:92:50 media: Ethernet autoselect status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=81209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER> ether 52:54:00:cb:db:07 inet 192.168.73.102 netmask 0xffffff00 broadcast 192.168.73.255 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> re0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE> ether 52:54:00:64:37:eb media: Ethernet autoselect (100baseTX <full-duplex,flowcontrol,rxpause,txpause>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> # pciconf -l hostb0@pci0:0:0:0: class=0x060000 card=0x11001af4 chip=0x29c08086 rev=0x00 hdr=0x00 vgapci0@pci0:0:1:0: class=0x030000 card=0x11001af4 chip=0x01001b36 rev=0x04 hdr=0x00 pcib1@pci0:0:2:0: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 pcib3@pci0:0:2:1: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 pcib4@pci0:0:2:2: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 pcib5@pci0:0:2:3: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 pcib6@pci0:0:2:4: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 uhci0@pci0:0:29:0: class=0x0c0300 card=0x11001af4 chip=0x29348086 rev=0x03 hdr=0x00 uhci1@pci0:0:29:1: class=0x0c0300 card=0x11001af4 chip=0x29358086 rev=0x03 hdr=0x00 uhci2@pci0:0:29:2: class=0x0c0300 card=0x11001af4 chip=0x29368086 rev=0x03 hdr=0x00 ehci0@pci0:0:29:7: class=0x0c0320 card=0x11001af4 chip=0x293a8086 rev=0x03 hdr=0x00 isab0@pci0:0:31:0: class=0x060100 card=0x11001af4 chip=0x29188086 rev=0x02 hdr=0x00 ahci0@pci0:0:31:2: class=0x010601 card=0x11001af4 chip=0x29228086 rev=0x02 hdr=0x00 none0@pci0:0:31:3: class=0x0c0500 card=0x11001af4 chip=0x29308086 rev=0x02 hdr=0x00 pcib2@pci0:1:0:0: class=0x060400 card=0x00000000 chip=0x000e1b36 rev=0x00 hdr=0x01 vmx0@pci0:2:1:0: class=0x020000 card=0x07b015ad chip=0x07b015ad rev=0x01 hdr=0x00 em0@pci0:2:2:0: class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00 re0@pci0:2:3:0: class=0x020000 card=0x11001af4 chip=0x813910ec rev=0x20 hdr=0x00 none1@pci0:5:0:0: class=0x00ff00 card=0x11001af4 chip=0x10451af4 rev=0x01 hdr=0x00 So problem in 12.X is consistent with that in 11.3 . Let me know if there is any other testing or information you would like to do / provide. Thanks. Cheers, John Hartley.
John & Matt, What is/are the CPU/s for your host OS? Did you verify that the relevant virtual technology features are enabled on the host: IOMMU for AMD and VT/D (?) for Intel. IIRC, some older versions of Linux (CentOS, Ubuntu, and Fedora) requires additional manual entry in the grub's boot loader. The modern Linux _shouldn't_ but doesn't hurt to confirm: dmesg|grep -i ommu [ 0.000000] AGP: Please enable the IOMMU option in the BIOS setup [ 3.434650] iommu: Adding device 0000:00:00.0 to group 0 <snip/> [ 3.536829] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 [ 3.536834] AMD-Vi: Found IOMMU at 0000:40:00.2 cap 0x40 grep -e '^#' -v /etc/default/grub GRUB_DEFAULT=0 GRUB_TIMEOUT=10 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` GRUB_CMDLINE_LINUX_DEFAULT="" GRUB_CMDLINE_LINUX="" GRUB_DISABLE_SUBMENU=y How are you creating/managing your VMs on the host? I use virt-manager (to reduce typos, misconfiguration, or unsupported configuation errors) on Ubuntu to manage. Did you customize any of the configurations for the VM in the shell script (if used to maintain the VMs) or XML backend? Here's a screen shot for the VM's config: https://imgur.com/a/1WbzvhQ Below is the command line startup for that VM without any manual customization in the XML configuration file: ps -ef|grep 'fbsd11_ovmf' root 8914 20540 0 15:19 pts/7 00:00:00 grep --color=auto fbsd11_ovmf libvirt+ 25561 1 40 Jan09 ? 17:21:30 qemu-system-x86_64 -enable-kvm -name guest=z_fbsd12_ovmf,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-35-z_fbsd12_ovmf/master-key.aes -machine pc-q35-2.11,accel=kvm,usb=off,vmport=off,dump-guest-core=off -cpu Opteron_G5,vme=on,ht=on,monitor=on,osxsave=on,bmi1=on,mmxext=on,fxsr_opt=on,cmp_legacy=on,extapic=on,cr8legacy=on,osvw=on,ibs=on,skinit=on,wdt=on,tce=on,nodeid_msr=on,topoext=on,perfctr_core=on,perfctr_nb=on,invtsc=on,ibpb=on -drive file=/usr/share/OVMF/OVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/z_fbsd12_ovmf_VARS.fd,if=pflash,format=raw,unit=1 <...snip...> -no-user-config -nodefaults <...snip...> -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global ICH9-LPC.disable_s3=1 -global ICH9-LPC.disable_s4=1 -boot menu=on,strict=on -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 (<<< this should be em0 NIC) -device pcie-root-port,port=0x10,chassis=3,id=pci.3,bus=pcie.0,multifunction=on,addr=0x2 (<<< this should be the VirtIO NIC) -device pcie-root-port,port=0x11,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x1 (<<< sym0 LSI/Symbios SCSI) -device pcie-root-port,port=0x12,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x2 (<<< VirtIO SCSI) -device pcie-root-port,port=0x13,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x3 (<<< VirtIO Serial) -device pcie-root-port,port=0x14,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x4 (<<< VirtIO Memory Balloon) -device pcie-root-port,port=0x15,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x5 (<<< 'Controller PCIe 0' seen in the screen shot ?) -device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x1d.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x1d -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x1d.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x1d.0x2 -device lsi,id=scsi0,bus=pci.4,addr=0x0 (<<< sym0 LSI/Symbios SCSI) -device virtio-scsi-pci,id=scsi1,bus=pci.5,addr=0x0 -device virtio-serial-pci,id=virtio-serial0,bus=pci.6,addr=0x0 <...snip...> -netdev tap,fd=39,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:4e:d4:8f,bus=pci.2,addr=0x1 -netdev tap,fd=41,id=hostnet1,vhost=on,vhostfd=42 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:a9:08:fb,bus=pci.3,addr=0x0 <...snip...> -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 <...snip...> -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x1 <...snip...> -device virtio-balloon-pci,id=balloon0,bus=pci.7,addr=0x0 -msg timestamp=on --------------------------------------------------------------- 11.3-RELEASE-p5 r356613 boot log messages for em0 NIC: root@fbsd11-ovmf:~ # egrep 'em0' /var/log/messages Jan 10 00:29:26 fbsd11-ovmf kernel: em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x6000-0x603f mem 0x98000000-0x9801ffff irq 21 at device 1.0 on pci2 Jan 10 00:29:26 fbsd11-ovmf kernel: em0: Ethernet address: 52:54:00:4e:d4:8f Jan 10 00:29:26 fbsd11-ovmf kernel: em0: netmap queues/slots: TX 1/256, RX 1/256 Jan 10 00:29:26 fbsd11-ovmf kernel: em0: link state changed to UP Jan 10 01:08:26 fbsd11-ovmf kernel: em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x6000-0x603f mem 0x98000000-0x9801ffff irq 21 at device 1.0 on pci2 Jan 10 01:08:26 fbsd11-ovmf kernel: em0: Ethernet address: 52:54:00:4e:d4:8f Jan 10 01:08:26 fbsd11-ovmf kernel: em0: netmap queues/slots: TX 1/256, RX 1/256 Jan 10 01:08:26 fbsd11-ovmf kernel: em0: link state changed to UP Jan 10 01:26:48 fbsd11-ovmf kernel: em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x6000-0x603f mem 0x98000000-0x9801ffff irq 21 at device 1.0 on pci2 Jan 10 01:26:48 fbsd11-ovmf kernel: em0: Ethernet address: 52:54:00:4e:d4:8f Jan 10 01:26:48 fbsd11-ovmf kernel: em0: netmap queues/slots: TX 1/256, RX 1/256 Jan 10 01:26:48 fbsd11-ovmf kernel: em0: link state changed to UP Jan 11 16:00:03 fbsd11-ovmf kernel: em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x6000-0x603f mem 0x98000000-0x9801ffff irq 21 at device 1.0 on pci2 Jan 11 16:00:03 fbsd11-ovmf kernel: em0: Ethernet address: 52:54:00:4e:d4:8f Jan 11 16:00:03 fbsd11-ovmf kernel: em0: netmap queues/slots: TX 1/256, RX 1/256 Jan 11 16:00:03 fbsd11-ovmf kernel: em0: link state changed to UP --------------------------------------------------------------- Snippets of my custom kernel since 11.2-RELEASE-p15 and upgraded to 11.3-RELEASE-p5 r356613: root@fbsd11-ovmf:~ # grep -e '^#' -v /usr/src/sys/amd64/conf/Custom|grep -e '^$' -v cpu HAMMER ident GENERIC options SCHED_ULE # ULE scheduler options PREEMPTION # Enable kernel thread preemption <snip> device acpi options ACPI_DMAR device pci options PCI_HP # PCI-Express native HotPlug options PCI_IOV # PCI SR-IOV support device ahci # AHCI-compatible SATA controllers device ata # Legacy ATA/SATA controllers <snip> device em # Intel PRO/1000 Gigabit Ethernet Family device miibus # MII bus support device re # RealTek 8139C+/8169/8169S/8110S <snip> device uhci # UHCI PCI->USB interface device ohci # OHCI PCI->USB interface device ehci # EHCI PCI->USB interface (USB 2.0) device xhci # XHCI PCI->USB interface (USB 3.0) device usb # USB Bus (required) device ukbd # Keyboard device umass # Disks/Mass storage - Requires scbus and da device virtio # Generic VirtIO bus (required) device virtio_pci # VirtIO PCI device device vtnet # VirtIO Ethernet device device virtio_blk # VirtIO Block device device virtio_scsi # VirtIO SCSI device device virtio_balloon # VirtIO Memory Balloon device device netmap # netmap(4) support device crypto # Required by IPSEC --------------------------------------------------------------- dmesg of 11.2 & upgraded to 11.3 CPU detection which matches my Opteron 6370p: FreeBSD 11.2-RELEASE-p15 #0 r356556: Thu Jan 9 19:50:57 PST 2020 root@fbsd11-ovmf:/usr/obj/usr/src11.2/sys/Custom amd64 FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0) VT(efifb): resolution 800x600 CPU: AMD Opteron 63xx class CPU (2003.51-MHz K8-class CPU) Origin="AuthenticAMD" Id=0x600f20 Family=0x15 Model=0x2 Stepping=0 Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT> Features2=0xbeb83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,HV> AMD Features=0x26500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,LM> AMD Features2=0x210bf7<LAHF,CMP,SVM,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,XOP,FMA4,TBM> Structured Extended Features=0x8<BMI1> AMD Extended Feature Extensions ID EBX=0x1000 SVM: NAsids=16 TSC: P-state invariant Hypervisor: Origin = "KVMKVMKVM" FreeBSD 11.3-RELEASE-p5 #0 r356613: Sat Jan 11 15:56:34 PST 2020 root@fbsd11-ovmf:/usr/obj/usr/src11.3/sys/Custom amd64 FreeBSD clang version 8.0.0 (tags/RELEASE_800/final 356365) (based on LLVM 8.0.0) VT(efifb): resolution 800x600 CPU: AMD Opteron 63xx class CPU (2004.09-MHz K8-class CPU) Origin="AuthenticAMD" Id=0x600f20 Family=0x15 Model=0x2 Stepping=0 Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT> Features2=0xbeb83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,HV> AMD Features=0x26500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,LM> AMD Features2=0x210bf7<LAHF,CMP,SVM,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,XOP,FMA4,TBM> Structured Extended Features=0x8<BMI1> AMD Extended Feature Extensions ID EBX=0x1000 SVM: NAsids=16 TSC: P-state invariant Hypervisor: Origin = "KVMKVMKVM" cat /proc/cpuinfo (on host Ubuntu 18.0.4.3 LTS) processor : 0 vendor_id : AuthenticAMD cpu family : 21 model : 2 model name : AMD Opteron(tm) Processor 6370P stepping : 0 <snip> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall bmi1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold bugs : fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass <snip> --------------------------------------------------------------- This VM was initially created to ensure that my proposed VirtIO patches would also work in Q35 + OMVF for FreeBSD 12 (hence it's name: z_fbsd12_ovmf). I've since used this configuration to install FreeBSD 11.2 to test viability of backporting my proposed patches. The order of the devices seen in the command line startup matches the order of customizations I've made/added in the GUI virt-manager during the creation of the VM. Did you have a chane to test my proposed VirtIO patches for FreeBSD 12.x within your environment? The VirtIO devices should work along with SATA storage devices and em NICs. I'm unsure about the VirtIO + netmap. Regards, Tommy
(In reply to Tommy P from comment #29) Hi Tommy P, my machine is Lenovo X3650 M5 Type 8871 with Intel E5-2697A v4 CPUs. I think this corresponds to "Broadwell Architecture". So it is VT-D HW Virtualisation, with GRUB setup up for PCI Passthrough etc: cat /etc/default/grub # If you change this file, run 'update-grub' afterwards to update # /boot/grub/grub.cfg. # For full documentation of the options in this file, see: # info -f grub -n 'Simple configuration' GRUB_DEFAULT=0 ... GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=1 intel_iommu=on ipv6.disable=1" GRUB_CMDLINE_LINUX="ipv6.disable=1" QEMU / KVM is going fine including PCI pass through (which requires grub GRUB_CMDLINE_LINUX_DEFAULT config above). I have also enabled nested hypervisor support, but have not used it so far. Running combination of: FreeBSD, Ubuntu, Windows and MacOS VMs, without problems (bar this one). Sorry for delayed response I have been working on seeing if I can get backport of your VirtIO fix on 11.3 . Have done all the coding and now testing, unsuccessfully so far :-( . My VirtIO backport included: Merging - virtio.c/.h, balloon & mmio code Adding - endian support code Replacing virtio/pci code with your sub-tree. Not sure why it is not working at this point. Backport of virtio/network code is likely to be harder as this has adopted CK (concurrency kit) queues and I have not managed to get clean compile. I will post on backport success / failure in our bug thread. Cheers, John Hartley.
(In reply to John Hartley from comment #28) So this very same issue (em0 not detected with "dev netmap", but detected without it) happens also with 12.x? Can you please attach the configuration (xml) that you are using with virsh or virt-manager? (e.g., using "virsh dumpxml ${VMNAME}")
(In reply to John Hartley from comment #30) Hi John, Backporting to 11.x is not easy. There new significant changes from what I've seen thus far in 12.x such as: sys/netinet sys/netinet/netdump Coincidentally, while I do not any issues with netmap and non VirtIO devices within my environment, netmap does break VirtIO detection and loading for Q35, OVMF or not. Regards, Tommy
(In reply to Vincenzo Maffione from comment #31) Hi Vincenzo, find following virsh dump of 11.3 test VM. My 12.0 test VM is essentially the same. Both find network interfaces with disabled (via kernel config sys/amd64/conf/GENERIC2). You will notice that i have 3 NICs on VM (for em0, vmx0 & re0 in FreeBSD). I have removed VirtoIO NIC as we know that there is seperate issue with this that Tommy T and Brian V has been dealing with. NOTE: I am managing QEMU / KVM using "Virtual Machine Manager" UI. # cat newt-qemu-02.xml <domain type='kvm' id='56'> <name>newt-dev-freebsd-11.3</name> <uuid>01d93f61-8527-42f0-b79d-fda37cf87f71</uuid> <metadata> <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0"> <libosinfo:os id="http://freebsd.org/freebsd/11.2"/> </libosinfo:libosinfo> </metadata> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>2</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-4.0'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader> <nvram>/home/XXX/DDD/newt.dev.freebsd/OVMF_VARS.fd</nvram> </os> <features> <acpi/> <apic/> <vmport state='off'/> </features> <cpu mode='custom' match='exact' check='full'> <model fallback='forbid'>Broadwell-IBRS</model> <vendor>Intel</vendor> <feature policy='require' name='vme'/> <feature policy='require' name='ss'/> <feature policy='require' name='vmx'/> <feature policy='require' name='f16c'/> <feature policy='require' name='rdrand'/> <feature policy='require' name='hypervisor'/> <feature policy='require' name='arat'/> <feature policy='require' name='tsc_adjust'/> <feature policy='require' name='umip'/> <feature policy='require' name='md-clear'/> <feature policy='require' name='stibp'/> <feature policy='require' name='arch-capabilities'/> <feature policy='require' name='ssbd'/> <feature policy='require' name='xsaveopt'/> <feature policy='require' name='pdpe1gb'/> <feature policy='require' name='abm'/> <feature policy='disable' name='skip-l1dfl-vmentry'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='cdrom'> <driver name='qemu'/> <target dev='sdb' bus='sata'/> <readonly/> <boot order='1'/> <alias name='sata0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/home/XXX/DDD/newt.dev.freebsd/newt-hd1-01.qcow2'/> <backingStore/> <target dev='sdc' bus='sata'/> <boot order='2'/> <alias name='sata0-0-2'/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/home/XXX/DDD/test.freebsd/test-hd1-01.qcow2'/> <backingStore/> <target dev='sdd' bus='sata'/> <alias name='sata0-0-3'/> <address type='drive' controller='0' bus='0' target='0' unit='3'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/> </controller> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-to-pci-bridge'> <model name='pcie-pci-bridge'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0x11'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x12'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0x13'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0x14'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:4e:50:91'/> <source bridge='br90'/> <target dev='vnet31'/> <model type='e1000'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </interface> <interface type='bridge'> <mac address='52:54:00:fb:be:74'/> <source bridge='br90'/> <target dev='vnet32'/> <model type='vmxnet3'/> <alias name='net1'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/> </interface> <interface type='bridge'> <mac address='52:54:00:b9:0c:12'/> <source bridge='br90'/> <target dev='vnet33'/> <model type='rtl8139'/> <alias name='net2'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/28'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/28'> <source path='/dev/pts/28'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='keyboard' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'> <alias name='input1'/> </input> <input type='keyboard' bus='ps2'> <alias name='input2'/> </input> <graphics type='spice' port='5932' autoport='yes' listen='127.0.0.1'> <listen type='address' address='127.0.0.1'/> <image compression='off'/> </graphics> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video> <redirdev bus='usb' type='spicevmc'> <alias name='redir0'/> <address type='usb' bus='0' port='2'/> </redirdev> <redirdev bus='usb' type='spicevmc'> <alias name='redir1'/> <address type='usb' bus='0' port='3'/> </redirdev> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='apparmor' relabel='yes'> <label>libvirt-01d93f61-8527-42f0-b79d-fda37cf87f71</label> <imagelabel>libvirt-01d93f61-8527-42f0-b79d-fda37cf87f71</imagelabel> </seclabel> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+64055:+106</label> <imagelabel>+64055:+106</imagelabel> </seclabel> </domain> Tommy T and other are now also able to reproduced the with/without netmap and NIC detection issue. Thanks again for assistance. Cheers, John Hartley.
(In reply to Tommy P from comment #32) Hi Tommy T, thanks for tip on backport. I have come across the netdump issue with my backport and #defined any code for this out to avoid compile issues. Currently I have compiled and bootable 11.3 with the virtio backport, but virtio is not working yet. I am just testing with disk at the moment as to your point virtio network is other issues which Vincenzo (from upstream netmap team) is helping to look at. I believe with Vincenzo (netmap), Brian V (virtio) and yourself we have the right people to get to root cause of netmap issue. I put in weekend with backport but doubt I will able to do much concentrated work with it during week due to work commitments. Good thing is we are now getting multiple people now reproducing the Q35 netmap issue. I still believe that this is some subtle PCI / netmap interaction, as the detection has already failed by early kernel load stage of boot. Cheers, John Hartley.
(In reply to John Hartley from comment #34) I found the issue, and you are basically right. It's a PCI ID conflict between the QEMU PCI express root port device and the netmap passthrough device. I must assume the PCIE root port is present in Q35 and not in i440fx, which explains why the issue only affects Q35. I will disable the netmap passthrough devices until the proper support is committed to QEMU. Sorry for the inconvenience. I'll commit the fix tomorrow.
Created attachment 210670 [details] Patch to fix q35 issue This patch will fix the issue by disabling ptnetmap devices (in conflict with newer QEMU virtual PCI devices).
(In reply to Vincenzo Maffione from comment #35) HI Vincenzo, great !! Yes you are right as well about PCIe root devices. The reason is that Q35 is native PCIe VM and so have PCIe root devices while i440FX is PCI/ISA (not PCE-e) machine. I will apply patch and advise on results. Cheers, John Hartley.
Created attachment 210673 [details] unraid create vm gui
On my Host OS unRAID 6.8.0, i440fx with seabios or OVFM FreeBSD just hangs on boot. One of the unRAID developers (jonp) spent a little time on it, and was also unsuccessful. It's possible that it's an unRAID specific issue. Not sure at this point. https://forums.unraid.net/topic/84692-freebsd-113-no-network-interface-after-upgrade/page/2/?tab=comments#comment-809306 I was able to get virtio NIC working on 12.1 with TommyP's patch. (Thank you!) *************************************** What is/are the CPU/s for your host OS? cat /proc/cpuinfo (unRAID 6.8 running Linux 4.19.88 kernel) processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 158 model name : Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz stepping : 10 microcode : 0xca cpu MHz : 4299.634 cache size : 12288 KB physical id : 0 siblings : 12 core id : 0 cpu cores : 6 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 22 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit bogomips : 6384.00 clflush size : 64 cache_alignment : 64 address sizes : 39 bits physical, 48 bits virtual power management: *************************************** Libvirt version: 5.8.0 QEMU version: 4.1.1 *************************************** Did you verify that the relevant virtual technology features are enabled on the host: IOMMU for AMD and VT/D (?) for Intel? I'm not sure how to show this on unRAID, but VT/D is enabled and PCIe ACS Override is enabled for better IMMO grouping. I have a variety of VMs and Dockers working. Ubuntu, Windows 10, FreeBSD 11.2, MacOS, Kali Linux. Multiple GPU's, NIC, audio, and USB cards passed through to VMs. **************************************** How are you creating/managing your VMs on the host? VM configuration is done via GUI (see attached) with an option to edit XML if needed. By default, the template network adapter type is "virtio", I then have to switch to XML view and change to e1000. XML below is just a default configuration with NIC changed to e1000 (FreeBSD 11.3, Q35-4.1 <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' id='21'> <name>cylon20</name> <uuid>522c2643-079b-7464-7120-70fdc2f7ec57</uuid> <description>FreeBSD 11.3</description> <metadata> <vmtemplate xmlns="unraid" name="FreeBSD" icon="freebsd.png" os="freebsd"/> </metadata> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='1'/> <vcpupin vcpu='1' cpuset='7'/> <vcpupin vcpu='2' cpuset='5'/> <vcpupin vcpu='3' cpuset='11'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-4.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/522c2643-079b-7464-7120-70fdc2f7ec57_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='2' threads='2'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/FreeBSD-11.3-RELEASE-amd64-dvd1.iso'/> <backingStore/> <target dev='hda' bus='sata'/> <readonly/> <boot order='2'/> <alias name='sata0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/user/domains/cylon20/vdisk1.img'/> <backingStore/> <target dev='hdc' bus='sata'/> <boot order='1'/> <alias name='sata0-0-2'/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> </disk> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x11'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0x12'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x13'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:bc:6a:0e'/> <source bridge='br0'/> <target dev='vnet1'/> <model type='e1000'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/1'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/1'> <source path='/dev/pts/1'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-21-cylon20/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'> <alias name='input1'/> </input> <input type='keyboard' bus='ps2'> <alias name='input2'/> </input> <graphics type='vnc' port='5901' autoport='yes' websocket='5701' listen='0.0.0.0' keymap='en-us'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+100</label> <imagelabel>+0:+100</imagelabel> </seclabel> </domain> +++++++++++++++++++++++++++++++++++++++ -uuid 522c2643-079b-7464-7120-70fdc2f7ec57 \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=37,server,nowait \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=utc,driftfix=slew \ -global kvm-pit.lost_tick_policy=delay \ -no-hpet \ -no-shutdown \ -boot strict=on \ -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \ -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \ -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \ -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \ -device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x7.0x7 \ -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x7 \ -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x7.0x1 \ -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x7.0x2 \ -device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \ -drive file=/mnt/user/isos/FreeBSD-11.3-RELEASE-amd64-dvd1.iso,format=raw,if=none,id=drive-sata0-0-0,readonly=on \ -device ide-cd,bus=ide.0,drive=drive-sata0-0-0,id=sata0-0-0,bootindex=2 \ -drive file=/mnt/user/domains/cylon20/vdisk1.img,format=raw,if=none,id=drive-sata0-0-2,cache=writeback \ -device ide-hd,bus=ide.2,drive=drive-sata0-0-2,id=sata0-0-2,bootindex=1,write-cache=on \ -netdev tap,fd=41,id=hostnet0 \ -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:bc:6a:0e,bus=pci.1,addr=0x0 \ -chardev pty,id=charserial0 \ -device isa-serial,chardev=charserial0,id=serial0 \ -chardev socket,id=charchannel0,fd=42,server,nowait \ -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \ -device usb-tablet,id=input0,bus=usb.0,port=1 \ -vnc 0.0.0.0:1,websocket=5701 \ -k en-us \ -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x1 \ -device virtio-balloon-pci,id=balloon0,bus=pci.3,addr=0x0 \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=o +++++++++++++++++++++++++++++++++++++++ pciconf does not show any intel NIC device detected. (same with passed through Intel NIC). When I compile 11.3 kernel without NETMAP, I see NIC interface detected as "none2@pci0:1:0:0", but still does not work. (see attached) See DMESG output screenshots. "em0: Unable to allocate bus resource: interrupt" *************************************** Okay, I fail at adding attachments...
Created attachment 210674 [details] 11.3 dmesg output with generic kernel
Created attachment 210675 [details] 11.3 dmesg output with custom kernel netmap removed
(In reply to Vincenzo Maffione from comment #36) This is great news. I didn't notice the progress on this while I was working on my previous replies. Will this patch be committed to 11.3+?
(In reply to Vincenzo Maffione from comment #35) Hi Vincenzo, I came to the same conclusion just now too: ptnetmap-memdev0: <ptnetmap-memdev PCI adapter> mem 0xfd412000-0xfd412fff irq 22 at device 2.0 on pci0 000.000080 [ 871] ptn_memdev_attach ptn_memdev_driver attach ptnetmap-memdev0: cannot map I/O space device_attach: ptnetmap-memdev0 attach returned 6 If I'm correct, it looks like netmap is trying to attach PCIe device on a PCI bus. What I find puzzling is that netmap supports igb (sys/dev/netmap/if_igb_netmap.h) which is PCIe (I don't have an igb to verify) but somehow fail to attach other PCIe devices: https://www.freebsd.org/cgi/man.cgi?query=igb&apropos=0&sektion=4&manpath=FreeBSD+12.1-RELEASE+and+Ports&arch=default&format=html Thanks, Tommy
(In reply to Vincenzo Maffione from comment #36) I got a compile error, the line below was missing the trailing slash "+#endif /* WITH_PTNETMAP *"
(In reply to MattS from comment #44) Hi MattS & Vincenzo, I have build new 12.0 based kernel with variation of patches provides (as I believe these where against 12.1). Good results ! Here is info: # diff GENERIC GENERIC2 22c22 < ident GENERIC --- > ident GENERIC2 So no longer disabled netmap # uname -av FreeBSD agent.in.graphica.com.au 12.0-RELEASE FreeBSD 12.0-RELEASE GENERIC2 amd64 So looks like uname -a not longer provides compile date info.... # ifconfig -a vmx0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6> ether 52:54:00:07:92:50 media: Ethernet autoselect status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=81209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER> ether 52:54:00:cb:db:07 inet 192.168.73.102 netmask 0xffffff00 broadcast 192.168.73.255 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> re0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE> ether 52:54:00:64:37:eb media: Ethernet autoselect (100baseTX <full-duplex,flowcontrol,rxpause,txpause>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> All network devices bar virtio are ok. # pciconf -lcve hostb0@pci0:0:0:0: class=0x060000 card=0x11001af4 chip=0x29c08086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82G33/G31/P35/P31 Express DRAM Controller' class = bridge subclass = HOST-PCI vgapci0@pci0:0:1:0: class=0x030000 card=0x11001af4 chip=0x01001b36 rev=0x04 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'QXL paravirtual graphic card' class = display subclass = VGA pcib1@pci0:0:2:0: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 000d[148] = ACS 1 pcib3@pci0:0:2:1: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(empty) surprise Attn Button PC(off) EI(disengaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 000d[148] = ACS 1 pcib4@pci0:0:2:2: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(empty) surprise Attn Button PC(off) EI(disengaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 000d[148] = ACS 1 pcib5@pci0:0:2:3: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 000d[148] = ACS 1 pcib6@pci0:0:2:4: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(empty) surprise Attn Button PC(off) EI(disengaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 000d[148] = ACS 1 uhci0@pci0:0:29:0: class=0x0c0300 card=0x11001af4 chip=0x29348086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB UHCI Controller' class = serial bus subclass = USB uhci1@pci0:0:29:1: class=0x0c0300 card=0x11001af4 chip=0x29358086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB UHCI Controller' class = serial bus subclass = USB uhci2@pci0:0:29:2: class=0x0c0300 card=0x11001af4 chip=0x29368086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB UHCI Controller' class = serial bus subclass = USB ehci0@pci0:0:29:7: class=0x0c0320 card=0x11001af4 chip=0x293a8086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB2 EHCI Controller' class = serial bus subclass = USB isab0@pci0:0:31:0: class=0x060100 card=0x11001af4 chip=0x29188086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82801IB (ICH9) LPC Interface Controller' class = bridge subclass = PCI-ISA ahci0@pci0:0:31:2: class=0x010601 card=0x11001af4 chip=0x29228086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]' class = mass storage subclass = SATA cap 05[80] = MSI supports 1 message, 64 bit enabled with 1 message cap 12[a8] = SATA Index-Data Pair none0@pci0:0:31:3: class=0x0c0500 card=0x11001af4 chip=0x29308086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) SMBus Controller' class = serial bus subclass = SMBus pcib2@pci0:1:0:0: class=0x060400 card=0x00000000 chip=0x000e1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' class = bridge subclass = PCI-PCI cap 05[8c] = MSI supports 1 message, 64 bit, vector masks cap 01[84] = powerspec 3 supports D0 D3 current D0 cap 10[48] = PCI-Express 2 PCI bridge max data 128(128) ARI disabled link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) cap 0c[40] = unknown ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected vmx0@pci0:2:1:0: class=0x020000 card=0x07b015ad chip=0x07b015ad rev=0x01 hdr=0x00 vendor = 'VMware' device = 'VMXNET3 Ethernet Controller' class = network subclass = ethernet cap 11[9c] = MSI-X supports 25 messages, enabled Table in map 0x18[0x0], PBA in map 0x18[0x1000] cap 05[84] = MSI supports 1 message, 64 bit em0@pci0:2:2:0: class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82540EM Gigabit Ethernet Controller' class = network subclass = ethernet re0@pci0:2:3:0: class=0x020000 card=0x11001af4 chip=0x813910ec rev=0x20 hdr=0x00 vendor = 'Realtek Semiconductor Co., Ltd.' device = 'RTL-8100/8101L/8139 PCI Fast Ethernet Adapter' class = network subclass = ethernet none1@pci0:5:0:0: class=0x00ff00 card=0x11001af4 chip=0x10451af4 rev=0x01 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'Virtio memory balloon' class = old cap 09[c8] = vendor (length 20) cap 09[b4] = vendor (length 20) cap 09[a4] = vendor (length 16) cap 09[94] = vendor (length 16) cap 09[84] = vendor (length 16) cap 01[7c] = powerspec 3 supports D0 D3 current D0 cap 10[40] = PCI-Express 2 endpoint max data 128(128) link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) In additional to network devices getting virtio balloon reported. Based on this my strategy for 11.2 is likely to be; 1. Wait for 12.X update with netmap patch with either fixes issue or disables netmap (as per Vincenzo's current update). 2. Upgrade from 11.2 -> 12.X The reason is that it is now some time since 11.3 and working on backport seems to be significant effort with little value. Best to just jump to 12.x fix and avoid going to intermediate releases. I tried to backport the 12.X VirtoIO port and this is messy and even Vincenzo's netmap update needs to be reworks for 11.3 based fix. If Vincenzo provides commit that flow through as update then I will take copy of production 11.2 VM and validate doing upgrade to 12.X on this. This should provide path for others caught on 11.2 . Cheers, John Hartley.
So I'm unclear on how long this takes to make it to RELEASE. Are you saying we have to wait for version 12.2 in October 2020? Or will it be applied to 12.1 as a security patch? My kernel build failed when I tried to apply Vincenzo's patch along with TommyP's patch. I ran out of time, so I'll be trying again later. I'd did get a working virtio NIC with TommyP's patch, so ultimately that's all I need.
Can someone provide a 1-patch so we can apply without wondering "do we apply only patch1 + patch3, or patch1+2+3, or what ever.."
(In reply to Dobri Dobrev from comment #47) Hi Dobri, by way of testing I am doing: 1. Build 12.0-RELEASE with "dev netmap" commented out of GENERIC kernel to get network access on any of em, vmx or re . You have to keep the kernel with GENERIC id as otherwise update in step 2 will fail. If you are on 12.0 then I assume you must have SATA disk as Virto and SCSI will not work. 2. Use "freebsd-update" (freebsd-update -r 9.1-RELEASE upgrade) to do upgrade from 12.0-RELEASE to 12.1-RELEASE. This will require two reboots via to "free-update" steps (both /usr/sbin/freebsd-update install). 3. Build 12.1-RELEASE with "dev netmap" commented out of GENERIC kernel and then do final "freebsd-update" (/usr/sbin/freebsd-update install). 4. Do your Ports or Package updates using postmaster or other tools in usual way. You should now have working 12.1 and can apply VirtIO patch (again via kernel rebuild) if you want to change from SATA to VirtIO based disk. At this point you should be able to start to use "freebsd-update" to and other utilities to get binary security updates. In fullness of time bugs / patches for fix of VirtIO and netmap should flow through as either binary updates via "freebsd-update" or be part of new release (12.2). Right now I am testing process and have got to step 4 (it is slow as each kernel rebuild takes some time). Once I have done upgrade test from 12.0 -> 12.1 I will then try some approach but starting from 11.2 and go to 12.1. Once on 12.1 I can test VirtIO fix without need to worry about trying to backport it 11.x . Both VirtIO and netmap are "performance optimisations". So the extent of need is driven by throughput requirements of your server. A while back I did upgrade of quite a large number of FreeBSD VMs (from 8.x, 9.x & 10.x to 11.x) and at the time I was forced to change disk to SATA as SCSI (and VirtIO) was broken already at that stage. The final breaking of networking was what prompted me to raise this bug report, as I can live with SATA but I can't not have networking. So I believe you can get out of the "bind" without needing any of patches in the first instance. Cheers, John Hartley.
(In reply to John Hartley from comment #48) I'm doing directly on 12.1-release, placing the 1st patch (comment 24 I think it was) on am usb stick, patch the src, rebuild kernel - virtio starts working. Problem is - there are several patches here, now I don't know what to use / not use which is why I asked if a single patch can be made.
(In reply to Dobri Dobrev from comment #49) Hi Dobri, Assuming you are on Q35 with OVMF (as this is what I have tested against). If you are happy to use SATA then you don't need any patches. Just recompile your kernel with /usr/src/sys/amd64/conf/GENERIC but with the "device netmap" commented out in GENERIC. 362c362 < #device netmap # netmap(4) support --- > device netmap # netmap(4) support This will give you a machine with SATA HD and working networking. If you also want to be able to use VirtIO then you will also need to apply the VirtIO patch. However if you do this then you must make sure you also recompile with netmap disabled if you want networking. The result with be a machine that has VirtIO HD and any of em, vm or re networking. Right now I do not believe you can have VirtIO networking only HD. If you do decide to build with VirtIO support then you will need to change your /etc/fstab mounts to reflect the different disk naming: SATA - adaNpX VirtIO SCSI - daNpX VirtIO block - vtbdNpX Where N is disk id and X is partition number. So what you do reflects the choices you make, until this is all fully fixed as part of a release. I hope this is clear... Cheers, John Hartley.
(In reply to MattS from comment #42) I will commit it to head and MFC to stable/12 and stable/11, so that it will be picked up for the next 11.x release. Would that be good enough?
(In reply to MattS from comment #44) Sorry about the typo, I created it in a hurry yesterday night. I did test the correct patch on my Q35 Freebsd 12.x VM, and it worked.
(In reply to MattS from comment #46) I'll try to push for an hotfix. Let's see how it goes.
(In reply to Vincenzo Maffione from comment #51) Can you post the 11.x patch so I can manually install it? I think John enlightened me on why I was having problems with i440fx testing, partition scheme incompatible when switching to seabios. Brief test shows installer ISO does detect NIC with i440fx/seabios. Not sure if I can convert from GPT to MBR. I have a few options at this point, so I'm happy. Thank you again for your attention on this issue.
(In reply to Vincenzo Maffione from comment #53) Pardon my ignorance, does a hotfix mean it will be pulled in with "freebsd-update fetch"?
A commit references this bug: Author: vmaffione Date: Mon Jan 13 21:47:24 UTC 2020 New revision: 356704 URL: https://svnweb.freebsd.org/changeset/base/356704 Log: netmap: disable passthrough with no hypervisor support The netmap passthrough subsystem requires proper support in the hypervisor. In particular, two PCI device ids (from the Red Hat PCI vendor id 0x1b36) need to be assigned to the two netmap virtual devices. We then disable these devices until the ids have not been assigned, in order to avoid conflicts with other virtual devices emulated by upstream QEMU. PR: 241774 MFC after: 3 days Changes: head/sys/dev/netmap/if_ptnet.c head/sys/dev/netmap/netmap_kern.h head/sys/net/netmap_virt.h
(In reply to Vincenzo Maffione from comment #53) Hi Vincenzo, it would be good to get hot fix. My understanding of your fix proposal is that due to conflicts with PCEe IDs that you are "disabling netmap passthrough support". Does this mean that we loose the performance advantages provide netmap completely ? For final fix (ie perfect future we dream of ;-) ) , will get Q35 PCIe hypervisor with full advantages of netmap based buffer management ? I have since read the netmap paper: https://github.com/luigirizzo/netmap#references and it would be a significant loss to have netmap disabled any future releases. Cheers, John Hartley.
Created attachment 210719 [details] Netmap patch for 11.x
(In reply to MattS from comment #55) I've never used it, but yes, I think so. According to the manual and handbook, that's what we should expect, provided I manage to get the fix in the releng branches.
(In reply to John Hartley from comment #57) The patch is not disabling netmap. You can and should keep using it. The patch is only disabling a specific netmap feature that QEMU (and bhyve) does not support yet. IOW it is disabling a feature that you cannot use right now (unless you use a non-standard patched QEMU). In practice, this means that we are not losing functionality.
(In reply to Vincenzo Maffione from comment #60) Great to hear Vincenzo and many thanks for acting so quickly on this. Ciao from Australia ;-) John Hartley
(In reply to Dobri Dobrev from comment #49) See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236922#c41
(In reply to Vincenzo Maffione from comment #58) There may still be a problem on 11.x. I successfully upgraded a copy of my production vm to 11.3 (QEMU 3.1, OVMF, e1000, sata). Everything seems great. I then switched to QEMU 4.0, 4.01, 4.1 and e1000 is no longer working. Switch back to 3.1 and e1000 is working again. pciconf none2@pci0:1:0:0: class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82540EM Gigabit Ethernet Controller' class = network subclass = ethernet dmesg em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x9000-0x903f mem 0x98600000-0x9861ffff irq 22 at device 0.0 on pci1 em0: Unable to allocate bus resource: interrupt device_attach: em0 attach returned 6
A commit references this bug: Author: vmaffione Date: Thu Jan 16 20:55:15 UTC 2020 New revision: 356804 URL: https://svnweb.freebsd.org/changeset/base/356804 Log: MFC r356704 netmap: disable passthrough with no hypervisor support The netmap passthrough subsystem requires proper support in the hypervisor. In particular, two PCI device ids (from the Red Hat PCI vendor id 0x1b36) need to be assigned to the two netmap virtual devices. We then disable these devices until the ids have not been assigned, in order to avoid conflicts with other virtual devices emulated by upstream QEMU. PR: 241774 Changes: _U stable/12/ stable/12/sys/dev/netmap/if_ptnet.c stable/12/sys/dev/netmap/netmap_kern.h stable/12/sys/net/netmap_virt.h
A commit references this bug: Author: vmaffione Date: Thu Jan 16 20:57:30 UTC 2020 New revision: 356805 URL: https://svnweb.freebsd.org/changeset/base/356805 Log: MFC r356704 netmap: disable passthrough with no hypervisor support The netmap passthrough subsystem requires proper support in the hypervisor. In particular, two PCI device ids (from the Red Hat PCI vendor id 0x1b36) need to be assigned to the two netmap virtual devices. We then disable these devices until the ids have not been assigned, in order to avoid conflicts with other virtual devices emulated by upstream QEMU. PR: 241774 Changes: _U stable/11/ stable/11/sys/dev/netmap/netmap_kern.h stable/11/sys/net/netmap_virt.h
(In reply to MattS from comment #63) This may be an unrelated problem... can you attach the guest dmesg of the problematic configuration? (e.g., QEMU 4.x). When you say "it does not work", do you mean that em0 is not showing up?
(In reply to Vincenzo Maffione from comment #66) That is correct, does not show up under ifconfig. pciconf shows device as none2@pci0:1:0:0 vs em0@pci0:4:0:0
(In reply to MattS from comment #67) My last comment was cut off for some reason. I'll create clean 11.3 vm with your netmap patch, verify issue, and post full dmesg/pciconf
(In reply to Vincenzo Maffione from comment #66) Installed clean 11.3 RELEASE as QEMU 4.1. No NIC shows up under ifconfig, Change to QEMU 3.1 and em0 shows up. Let me know if you need anything else. ################################################# 11.3 +netmap patch, QEMU 4.1, OVMF, SATA, E1000 +++++++++++++++++++++++++++++++++++++++++++++++++ ifconfig lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff000000 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> groups: lo +++++++++++++++++++++++++++++++++++++++++++++++++ pciconf -lv hostb0@pci0:0:0:0: class=0x060000 card=0x11001af4 chip=0x29c08086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82G33/G31/P35/P31 Express DRAM Controller' class = bridge subclass = HOST-PCI vgapci0@pci0:0:1:0: class=0x030000 card=0x11001af4 chip=0x01001b36 rev=0x04 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'QXL paravirtual graphic card' class = display subclass = VGA pcib1@pci0:0:2:0: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI pcib2@pci0:0:2:1: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI pcib3@pci0:0:2:2: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI pcib4@pci0:0:2:3: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI uhci0@pci0:0:7:0: class=0x0c0300 card=0x11001af4 chip=0x29348086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB UHCI Controller' class = serial bus subclass = USB uhci1@pci0:0:7:1: class=0x0c0300 card=0x11001af4 chip=0x29358086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB UHCI Controller' class = serial bus subclass = USB uhci2@pci0:0:7:2: class=0x0c0300 card=0x11001af4 chip=0x29368086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB UHCI Controller' class = serial bus subclass = USB none0@pci0:0:7:7: class=0x0c0320 card=0x11001af4 chip=0x293a8086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB2 EHCI Controller' class = serial bus subclass = USB isab0@pci0:0:31:0: class=0x060100 card=0x11001af4 chip=0x29188086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82801IB (ICH9) LPC Interface Controller' class = bridge subclass = PCI-ISA ahci0@pci0:0:31:2: class=0x010601 card=0x11001af4 chip=0x29228086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]' class = mass storage subclass = SATA none1@pci0:0:31:3: class=0x0c0500 card=0x11001af4 chip=0x29308086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) SMBus Controller' class = serial bus subclass = SMBus none2@pci0:1:0:0: class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82540EM Gigabit Ethernet Controller' class = network subclass = ethernet +++++++++++++++++++++++++++++++++++++++++++++++++ dmesg Copyright (c) 1992-2019 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.3-RELEASE #1: Thu Jan 16 16:19:11 PST 2020 root@cylon20:/usr/obj/usr/src/sys/MYKERNEL amd64 FreeBSD clang version 8.0.0 (tags/RELEASE_800/final 356365) (based on LLVM 8.0.0) VT(efifb): resolution 800x600 CPU: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz (3192.06-MHz K8-class CPU) Origin="GenuineIntel" Id=0x906ea Family=0x6 Model=0x9e Stepping=10 Features=0x1f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT> Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> AMD Features2=0x121<LAHF,ABM,Prefetch> Structured Extended Features=0x1c4fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP> Structured Extended Features2=0x4<UMIP> Structured Extended Features3=0x8c000400<MD_CLEAR,IBPB,STIBP,SSBD> XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES> real memory = 4294967296 (4096 MB) avail memory = 4063502336 (3875 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <BOCHS BXPCAPIC> WARNING: L1 data cache covers less APIC IDs than a core 0 < 1 FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 hardware threads ioapic0 <Version 1.1> irqs 0-23 on motherboard SMP: AP CPU #1 Launched! random: entropy device external interface kbd1 at kbdmux0 000.000030 [4213] netmap_init netmap: loaded module module_register_init: MOD_LOAD (vesa, 0xffffffff81004eb0, 0) error 19 random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" nexus0 cryptosoft0: <software crypto> on motherboard acpi0: <BOCHS BXPCFACP> on motherboard acpi0: Power Button (fixed) cpu0: <ACPI CPU> on acpi0 cpu1: <ACPI CPU> on acpi0 atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x608-0x60b on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 vgapci0: <VGA-compatible display> port 0xa0c0-0xa0df mem 0x94000000-0x97ffffff,0x90000000-0x93ffffff,0x98800000-0x98801fff irq 21 at device 1.0 on pci0 vgapci0: Boot video device pcib1: <ACPI PCI-PCI bridge> mem 0x98807000-0x98807fff irq 22 at device 2.0 on pci0 pcib1: [GIANT-LOCKED] pcib2: <PCI-PCI bridge> mem 0x98806000-0x98806fff irq 22 at device 2.1 on pci0 pcib2: Failed to allocate interrupt for PCI-e events pcib3: <PCI-PCI bridge> mem 0x98805000-0x98805fff irq 22 at device 2.2 on pci0 pcib3: Failed to allocate interrupt for PCI-e events pcib4: <PCI-PCI bridge> mem 0x98804000-0x98804fff irq 22 at device 2.3 on pci0 pcib4: Failed to allocate interrupt for PCI-e events uhci0: <Intel 82801I (ICH9) USB controller> port 0xa0a0-0xa0bf irq 23 at device 7.0 on pci0 usbus0 on uhci0 uhci1: <Intel 82801I (ICH9) USB controller> port 0xa080-0xa09f irq 20 at device 7.1 on pci0 usbus1 on uhci1 uhci2: <Intel 82801I (ICH9) USB controller> port 0xa060-0xa07f irq 21 at device 7.2 on pci0 usbus2 on uhci2 ehci0: <Intel 82801I (ICH9) USB 2.0 controller> mem 0x98803000-0x98803fff irq 22 at device 7.7 on pci0 ehci0: Could not allocate irq device_attach: ehci0 attach returned 6 isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 ahci0: <Intel ICH9 AHCI SATA controller> port 0xa040-0xa05f mem 0x98802000-0x98802fff irq 16 at device 31.2 on pci0 ahci0: AHCI v1.00 with 6 1.5Gbps ports, Port Multiplier not supported ahcich0: <AHCI channel> at channel 0 on ahci0 ahcich1: <AHCI channel> at channel 1 on ahci0 ahcich2: <AHCI channel> at channel 2 on ahci0 ahcich3: <AHCI channel> at channel 3 on ahci0 ahcich4: <AHCI channel> at channel 4 on ahci0 ahcich5: <AHCI channel> at channel 5 on ahci0 acpi_syscontainer0: <System Container> on acpi0 acpi_syscontainer1: <System Container> port 0xcd8-0xce3 on acpi0 acpi_syscontainer2: <System Container> port 0x620-0x62f on acpi0 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] psm0: <PS/2 Mouse> irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: model IntelliMouse Explorer, device ID 4 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (115200,n,8,1) attimer0: <AT timer> at port 0x40 on isa0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 ppc0: cannot reserve I/O port range Timecounters tick every 10.000 msec usbus1: 12Mbps Full Speed USB v1.0 usbus0: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 pci1: <ACPI PCI bus> on pcib1 em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x9000-0x903f mem 0x98600000-0x9861ffff irq 22 at device 0.0 on pci1 em0: Unable to allocate bus resource: interrupt device_attach: em0 attach returned 6 ugen2.1: <Intel UHCI root HUB> at usbus2 ugen1.1: <Intel UHCI root HUB> at usbus1 uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2 ugen0.1: <Intel UHCI root HUB> at usbus0 uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1 uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 ada0 at ahcich2 bus 0 scbus2 target 0 lun 0 ada0: <QEMU HARDDISK 2.5+> ATA-7 SATA device ada0: Serial Number QM00005 ada0: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) ada0: Command Queueing enabled ada0: 25600MB (52428800 512 byte sectors) cd0 at ahcich0 bus 0 scbus0 target 0 lun 0 Trying to mount root from ufs:/dev/ada0p2 [rw]... cd0: <QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device cd0: Serial Number QM00001 cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes) cd0: 3490MB (1786895 2048 byte sectors) random: unblocking device. uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered lo0: link state changed to UP ugen0.2: <QEMU QEMU USB Tablet> at usbus0 uhid0 on uhub2 uhid0: <QEMU QEMU USB Tablet, class 0/0, rev 2.00/0.00, addr 2> on usbus0 ################################################# 11.3 +netmap patch, QEMU 3.1, OVMF, SATA, E1000 +++++++++++++++++++++++++++++++++++++++++++++++++ ifconfig em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC> ether 52:54:00:17:32:d0 hwaddr 52:54:00:17:32:d0 inet 192.168.0.20 netmask 0xffffff00 broadcast 192.168.0.255 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> groups: lo +++++++++++++++++++++++++++++++++++++++++++++++++ pciconf -lv hostb0@pci0:0:0:0: class=0x060000 card=0x11001af4 chip=0x29c08086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82G33/G31/P35/P31 Express DRAM Controller' class = bridge subclass = HOST-PCI vgapci0@pci0:0:1:0: class=0x030000 card=0x11001af4 chip=0x01001b36 rev=0x04 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'QXL paravirtual graphic card' class = display subclass = VGA pcib1@pci0:0:2:0: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI pcib2@pci0:0:2:1: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI pcib3@pci0:0:2:2: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI pcib4@pci0:0:2:3: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI uhci0@pci0:0:7:0: class=0x0c0300 card=0x11001af4 chip=0x29348086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB UHCI Controller' class = serial bus subclass = USB uhci1@pci0:0:7:1: class=0x0c0300 card=0x11001af4 chip=0x29358086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB UHCI Controller' class = serial bus subclass = USB uhci2@pci0:0:7:2: class=0x0c0300 card=0x11001af4 chip=0x29368086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB UHCI Controller' class = serial bus subclass = USB ehci0@pci0:0:7:7: class=0x0c0320 card=0x11001af4 chip=0x293a8086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB2 EHCI Controller' class = serial bus subclass = USB isab0@pci0:0:31:0: class=0x060100 card=0x11001af4 chip=0x29188086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82801IB (ICH9) LPC Interface Controller' class = bridge subclass = PCI-ISA ahci0@pci0:0:31:2: class=0x010601 card=0x11001af4 chip=0x29228086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]' class = mass storage subclass = SATA none0@pci0:0:31:3: class=0x0c0500 card=0x11001af4 chip=0x29308086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) SMBus Controller' class = serial bus subclass = SMBus em0@pci0:1:0:0: class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82540EM Gigabit Ethernet Controller' class = network subclass = ethernet none1@pci0:2:0:0: class=0x078000 card=0x11001af4 chip=0x10431af4 rev=0x01 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'Virtio console' class = simple comms none2@pci0:3:0:0: class=0x00ff00 card=0x11001af4 chip=0x10451af4 rev=0x01 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'Virtio memory balloon' class = old +++++++++++++++++++++++++++++++++++++++++++++++++ pciconf -lv Copyright (c) 1992-2019 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.3-RELEASE #1: Thu Jan 16 16:19:11 PST 2020 root@cylon20:/usr/obj/usr/src/sys/MYKERNEL amd64 FreeBSD clang version 8.0.0 (tags/RELEASE_800/final 356365) (based on LLVM 8.0.0) VT(efifb): resolution 800x600 CPU: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz (3192.06-MHz K8-class CPU) Origin="GenuineIntel" Id=0x906ea Family=0x6 Model=0x9e Stepping=10 Features=0x1f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT> Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> AMD Features2=0x121<LAHF,ABM,Prefetch> Structured Extended Features=0x1c4fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP> Structured Extended Features2=0x4<UMIP> Structured Extended Features3=0x8c000400<MD_CLEAR,IBPB,STIBP,SSBD> XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES> real memory = 4294967296 (4096 MB) avail memory = 4063502336 (3875 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <BOCHS BXPCAPIC> WARNING: L1 data cache covers less APIC IDs than a core 0 < 1 FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 hardware threads ioapic0 <Version 1.1> irqs 0-23 on motherboard SMP: AP CPU #1 Launched! random: entropy device external interface kbd1 at kbdmux0 000.000030 [4213] netmap_init netmap: loaded module module_register_init: MOD_LOAD (vesa, 0xffffffff81004eb0, 0) error 19 random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" nexus0 cryptosoft0: <software crypto> on motherboard acpi0: <BOCHS BXPCFACP> on motherboard acpi0: Power Button (fixed) cpu0: <ACPI CPU> on acpi0 cpu1: <ACPI CPU> on acpi0 atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x608-0x60b on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 vgapci0: <VGA-compatible display> port 0xa0c0-0xa0df mem 0x94000000-0x97ffffff,0x90000000-0x93ffffff,0x98800000-0x98801fff irq 21 at device 1.0 on pci0 vgapci0: Boot video device pcib1: <ACPI PCI-PCI bridge> mem 0x98807000-0x98807fff irq 22 at device 2.0 on pci0 pci1: <ACPI PCI bus> on pcib1 em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x9000-0x903f mem 0x98600000-0x9861ffff irq 22 at device 0.0 on pci1 em0: Ethernet address: 52:54:00:17:32:d0 em0: netmap queues/slots: TX 1/256, RX 1/256 pcib2: <PCI-PCI bridge> mem 0x98806000-0x98806fff irq 22 at device 2.1 on pci0 pci2: <PCI bus> on pcib2 pci2: <simple comms> at device 0.0 (no driver attached) pcib3: <PCI-PCI bridge> mem 0x98805000-0x98805fff irq 22 at device 2.2 on pci0 pci3: <PCI bus> on pcib3 pci3: <old> at device 0.0 (no driver attached) pcib4: <PCI-PCI bridge> mem 0x98804000-0x98804fff irq 22 at device 2.3 on pci0 pci4: <PCI bus> on pcib4 uhci0: <Intel 82801I (ICH9) USB controller> port 0xa0a0-0xa0bf irq 23 at device 7.0 on pci0 usbus0 on uhci0 usbus0: 12Mbps Full Speed USB v1.0 uhci1: <Intel 82801I (ICH9) USB controller> port 0xa080-0xa09f irq 20 at device 7.1 on pci0 usbus1 on uhci1 usbus1: 12Mbps Full Speed USB v1.0 uhci2: <Intel 82801I (ICH9) USB controller> port 0xa060-0xa07f irq 21 at device 7.2 on pci0 usbus2 on uhci2 usbus2: 12Mbps Full Speed USB v1.0 ehci0: <Intel 82801I (ICH9) USB 2.0 controller> mem 0x98803000-0x98803fff irq 22 at device 7.7 on pci0 usbus3: EHCI version 1.0 usbus3 on ehci0 usbus3: 480Mbps High Speed USB v2.0 isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 ahci0: <Intel ICH9 AHCI SATA controller> port 0xa040-0xa05f mem 0x98802000-0x98802fff irq 16 at device 31.2 on pci0 ahci0: AHCI v1.00 with 6 1.5Gbps ports, Port Multiplier not supported ahcich0: <AHCI channel> at channel 0 on ahci0 ahcich1: <AHCI channel> at channel 1 on ahci0 ahcich2: <AHCI channel> at channel 2 on ahci0 ahcich3: <AHCI channel> at channel 3 on ahci0 ahcich4: <AHCI channel> at channel 4 on ahci0 ahcich5: <AHCI channel> at channel 5 on ahci0 acpi_syscontainer0: <System Container> on acpi0 acpi_syscontainer1: <System Container> port 0xcd8-0xce3 on acpi0 acpi_syscontainer2: <System Container> port 0x620-0x62f on acpi0 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] psm0: <PS/2 Mouse> irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: model IntelliMouse Explorer, device ID 4 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (115200,n,8,1) attimer0: <AT timer> at port 0x40 on isa0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 ppc0: cannot reserve I/O port range Timecounters tick every 10.000 msec ugen3.1: <Intel EHCI root HUB> at usbus3 ugen2.1: <Intel UHCI root HUB> at usbus2 uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3 uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2 ugen0.1: <Intel UHCI root HUB> at usbus0 ugen1.1: <Intel UHCI root HUB> at usbus1 uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1 ada0 at ahcich2 bus 0 scbus2 target 0 lun 0 ada0: <QEMU HARDDISK 2.5+> ATA-7 SATA device ada0: Serial Number QM00005 ada0: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) ada0: Command Queueing enabled ada0: 25600MB (52428800 512 byte sectors) cd0 at ahcich0 bus 0 scbus0 target 0 lun 0 cd0: <QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device cd0: Serial Number QM00001 cd0: 150.000MB/s transfers (SATA 1.x, Trying to mount root from ufs:/dev/ada0p2 [rw]... UDMA5, ATAPI 12bytes, PIO 8192bytes) cd0: 3490MB (1786895 2048 byte sectors) random: unblocking device. uhub1: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered lo0: link state changed to UP uhub0: 6 ports with 6 removable, self powered em0: link state changed to UP ugen3.2: <QEMU QEMU USB Tablet> at usbus3 uhid0 on uhub0 uhid0: <QEMU QEMU USB Tablet, class 0/0, rev 2.00/0.00, addr 2> on usbus3 em0: link state changed to DOWN em0: link state changed to UP #################################################
(In reply to MattS from comment #69) I see. This looks like a different issue. The PCI-e code is failing to allocate the interrupt resources needed by ehci0 and em0, so the drivers cannot attach. It does not look like netmap is the cause here... but it may be worth trying to see what happens without "dev netmap"
(In reply to Vincenzo Maffione from comment #70) You were correct, removed netmap, no change. I checked 11.2 and it also seems to have the issue. Should a new bug report be created? I'm not sure it's actually necessary at this point since we have an upgrade path available now.
(In reply to MattS from comment #71) Do you mean that upgrading to 12.x fixes this latter problem? In that case I guess it's up to you to decide if you wish to create another issue. For sure this one become too crowded, discouraging any developer to take a look..
(In reply to Vincenzo Maffione from comment #72) I just tested clean 12.1 with your netmap patch from comment #36. I am seeing similar results to 11.X. QEMU 3.1, OVMF, SATA, e1000 works. Switching to QEMU 4.x causes panic on boot. I do see the em0 detected before panic. Re: my comment on upgrade path, I was just referring to options to upgrade from 11.2.
John and Matt, I think I found the root cause for NICs not working in QEMU 4.x. According to the QEMU's change log: ------------------------------------------------------------------------- PCI/PCIe Generic PCIe root port link speed and width enhancements: Starting with the Q35 QEMU 4.0 machine type, generic pcie-root-port will default to the maximum PCIe link speed (16GT/s) and width (x32) provided by the PCIe 4.0 specification. Experimental options x-speed= and x-width= are provided for custom tuning, but it is expected that the default over-provisioning of bandwidth is optimal for the vast majority of use cases. Previous machine versions and ioh3420 root ports will continue to default to 2.5GT/x1 links. ------------------------------------------------------------------------- and your VM's config having machine type 'pc-q35-4.0', I _doubt_ that FreeBSD RELENG/STABLE branches supports PCI-e gen 4 specs. What happens when you edit the XML and change the machine type to 'pc-q35-2.11'? You'll may need to recycle the libvirtd, remove and then re-add the devices and controllers. Regards, Tommy
John & Matt, QEMU 4.0 features got me interested and I've decided to dig deeper into your report em NIC not working. Since I don't have a spare bare metal to setup QEMU 4.0 environment, I did nested virtualization. Main host is Ubuntu 18.04.3 on bare metal Nested host is Ubuntu 19.10 (QEMU 4.0) as main host's VM Nested guests is in QEMU 4.0 I first created a new VM within the QEMU 4.0 and the em didn't work as you've reported. I then imported a config from my main host (QEMU 2.11) started it within the nested. It works as expected. When I compare the differences, here is what I found that would possibly cause the NIC to failed: Imported config from main: -name guest=z_fbsd12_ovmf,debug-threads=on -machine pc-q35-2.11,accel=kvm,usb=off,vmport=off,dump-guest-core=off -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:4e:d4:8f,bus=pci.2,addr=0x1 Created new config in nested: -name guest=freebsd12.0,debug-threads=on -machine pc-q35-2.11,accel=kvm,usb=off,vmport=off,dump-guest-core=off -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 -device pcie-pci-bridge,id=pci.4,bus=pci.1,addr=0x0 -device e1000e,netdev=hostnet0,id=net0,mac=52:54:00:87:83:4a,bus=pci.3,addr=0x0 As you can see from the command line startup, the imported config have the em0 on the PCI bus while the created new config is on the PCIe. Thus, my previously assessment (comment #74) was partially accurate. I'm sure that the bug is that the current em NIC driver does not support PCIe. If you manage to apply my proposed VirtIO patches, you should have working VirtIO (including the VirtIO NIC). I suggest that you file a separate bug report regarding the em NIC driver not supporting PCIe. Regards, Tommy
(In reply to Tommy P from comment #75) TommyP, Thanks for looking into this a bit. Since qemu 3.1 works, was there some reason you used qemu 2.11 in your test? Below is from my qemu 3.1 test vm -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:d0:8c:7a,bus=pci.1,addr=0x0 \
(In reply to MattS from comment #76) Hi Matt, I used 2.11 because I have working 2.11 (Ubuntu 18.04.3) and not 3.1. It'll be easier for me to compare and investigate why pretty much every PCIe devices and controllers break in QEMU 4.x. It's also easier for me to verify that nested virtualization should not have any effect, other than the obvious performance degradation, as confirmed when I imported the config from main host (18.04.3) running on bare metal into virtualized host (19.10). The all devices and controllers in the nested guest of the imported config are functional as if not nested. Would you mind attaching a working 3.1 XML? I'd like to narrow down the VM configurations which breaks FreeBSD. --------------------------------------------------- @Everyone, Since my last posting, I've been digging deeper into the issue of PCIe and QEMU 4.x. I think that both of my assessment (comment #74 and comment #75) were correct. Since someone else already reported QEMU 4.0 issue even though I think his/her initial assumption of netmap interference is incorrect: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241581 I'll post my findings there. In short, I'm pretty sure that the root cause is FreeBSD's ACPI is unable to retrieve the interrupt routing table for all the PCIe devices and controllers. Thus, those devices and controllers are unable to get the IRQ ID (Failed to allocate interrupt for PCI-e events) and would not work. Regards, Tommy
(In reply to Tommy P from comment #77) Hi Tommy P, here are some more rest results for you: QEMU with OVMF, Boot from patched VirtIO SCSI with patched netmap Networking. Results: Q35 V2.12 - Boots and both em & vmx network devices found: cat qemu-xml-v211-01.xml <domain type='kvm' id='127'> <name>test-freebsd-12.0</name> <uuid>XXX</uuid> <metadata> <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0"> <libosinfo:os id="http://freebsd.org/freebsd/12.0"/> </libosinfo:libosinfo> </metadata> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>2</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-2.12'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader> <nvram>/home/XXX/DIR/test.freebsd/OVMF_VARS.fd</nvram> </os> <features> <acpi/> <apic/> <vmport state='off'/> </features> <cpu mode='custom' match='exact' check='full'> <model fallback='forbid'>Broadwell-IBRS</model> <vendor>Intel</vendor> <feature policy='require' name='vme'/> <feature policy='require' name='ss'/> <feature policy='require' name='vmx'/> <feature policy='require' name='f16c'/> <feature policy='require' name='rdrand'/> <feature policy='require' name='hypervisor'/> <feature policy='require' name='arat'/> <feature policy='require' name='tsc_adjust'/> <feature policy='require' name='umip'/> <feature policy='require' name='md-clear'/> <feature policy='require' name='stibp'/> <feature policy='require' name='arch-capabilities'/> <feature policy='require' name='ssbd'/> <feature policy='require' name='xsaveopt'/> <feature policy='require' name='pdpe1gb'/> <feature policy='require' name='abm'/> <feature policy='disable' name='skip-l1dfl-vmentry'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='cdrom'> <driver name='qemu'/> <target dev='sda' bus='sata'/> <readonly/> <boot order='1'/> <alias name='sata0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/home/XXX/DIR/test.freebsd/test-hd1-01.qcow2'/> <backingStore/> <target dev='sdb' bus='scsi'/> <boot order='2'/> <alias name='scsi0-0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/> </controller> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x11'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-to-pci-bridge'> <model name='pcie-pci-bridge'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x12'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0x13'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0x14'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0x15'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:cb:db:07'/> <source bridge='br90'/> <target dev='vnet34'/> <model type='e1000'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x02' function='0x0'/> </interface> <interface type='bridge'> <mac address='52:54:00:a5:4c:f3'/> <source bridge='br90'/> <target dev='vnet35'/> <model type='vmxnet3'/> <alias name='net1'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/31'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/31'> <source path='/dev/pts/31'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <input type='keyboard' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'> <alias name='input1'/> </input> <input type='keyboard' bus='ps2'> <alias name='input2'/> </input> <graphics type='spice' port='5934' autoport='yes' listen='127.0.0.1'> <listen type='address' address='127.0.0.1'/> <image compression='off'/> </graphics> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video> <redirdev bus='usb' type='spicevmc'> <alias name='redir0'/> <address type='usb' bus='0' port='2'/> </redirdev> <redirdev bus='usb' type='spicevmc'> <alias name='redir1'/> <address type='usb' bus='0' port='3'/> </redirdev> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='apparmor' relabel='yes'> <label>libvirt-4b1c030f-dfe8-4bd1-9288-be1ced293543</label> <imagelabel>libvirt-4b1c030f-dfe8-4bd1-9288-be1ced293543</imagelabel> </seclabel> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+64055:+106</label> <imagelabel>+64055:+106</imagelabel> </seclabel> </domain> Q35 V3.5 - Boots and em & vmx network devices found cat qemu-xml-v35-01.xml <domain type='kvm' id='125'> <name>test-freebsd-12.0</name> <uuid>XXXX</uuid> <metadata> <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0"> <libosinfo:os id="http://freebsd.org/freebsd/12.0"/> </libosinfo:libosinfo> </metadata> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>2</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-3.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader> <nvram>/home/XXX/DIR/test.freebsd/OVMF_VARS.fd</nvram> </os> <features> <acpi/> <apic/> <vmport state='off'/> </features> <cpu mode='custom' match='exact' check='full'> <model fallback='forbid'>Broadwell-IBRS</model> <vendor>Intel</vendor> <feature policy='require' name='vme'/> <feature policy='require' name='ss'/> <feature policy='require' name='vmx'/> <feature policy='require' name='f16c'/> <feature policy='require' name='rdrand'/> <feature policy='require' name='hypervisor'/> <feature policy='require' name='arat'/> <feature policy='require' name='tsc_adjust'/> <feature policy='require' name='umip'/> <feature policy='require' name='md-clear'/> <feature policy='require' name='stibp'/> <feature policy='require' name='arch-capabilities'/> <feature policy='require' name='ssbd'/> <feature policy='require' name='xsaveopt'/> <feature policy='require' name='pdpe1gb'/> <feature policy='require' name='abm'/> <feature policy='disable' name='skip-l1dfl-vmentry'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='cdrom'> <driver name='qemu'/> <target dev='sda' bus='sata'/> <readonly/> <boot order='1'/> <alias name='sata0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/home/XXX/DIR/test.freebsd/test-hd1-01.qcow2'/> <backingStore/> <target dev='sdb' bus='scsi'/> <boot order='2'/> <alias name='scsi0-0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/> </controller> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x11'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-to-pci-bridge'> <model name='pcie-pci-bridge'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x12'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0x13'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0x14'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0x15'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:cb:db:07'/> <source bridge='br90'/> <target dev='vnet34'/> <model type='e1000'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x02' function='0x0'/> </interface> <interface type='bridge'> <mac address='52:54:00:a5:4c:f3'/> <source bridge='br90'/> <target dev='vnet35'/> <model type='vmxnet3'/> <alias name='net1'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/31'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/31'> <source path='/dev/pts/31'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <input type='keyboard' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'> <alias name='input1'/> </input> <input type='keyboard' bus='ps2'> <alias name='input2'/> </input> <graphics type='spice' port='5934' autoport='yes' listen='127.0.0.1'> <listen type='address' address='127.0.0.1'/> <image compression='off'/> </graphics> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video> <redirdev bus='usb' type='spicevmc'> <alias name='redir0'/> <address type='usb' bus='0' port='2'/> </redirdev> <redirdev bus='usb' type='spicevmc'> <alias name='redir1'/> <address type='usb' bus='0' port='3'/> </redirdev> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='apparmor' relabel='yes'> <label>libvirt-4b1c030f-dfe8-4bd1-9288-be1ced293543</label> <imagelabel>libvirt-4b1c030f-dfe8-4bd1-9288-be1ced293543</imagelabel> </seclabel> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+64055:+106</label> <imagelabel>+64055:+106</imagelabel> </seclabel> </domain> Q35 V4.0 - Boots to mountroot prompt but then unable to find disk (ie looks like UEFI starting is working but then FreeBSD disk driver fails), both emo & vmx network devices look to have been found cat qemu-xml-v40-01.xml <domain type='kvm' id='126'> <name>test-freebsd-12.0</name> <uuid>4b1c030f-dfe8-4bd1-9288-be1ced293543</uuid> <metadata> <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0"> <libosinfo:os id="http://freebsd.org/freebsd/12.0"/> </libosinfo:libosinfo> </metadata> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>2</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-4.0'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader> <nvram>/home/XXX/DIR/test.freebsd/OVMF_VARS.fd</nvram> </os> <features> <acpi/> <apic/> <vmport state='off'/> </features> <cpu mode='custom' match='exact' check='full'> <model fallback='forbid'>Broadwell-IBRS</model> <vendor>Intel</vendor> <feature policy='require' name='vme'/> <feature policy='require' name='ss'/> <feature policy='require' name='vmx'/> <feature policy='require' name='f16c'/> <feature policy='require' name='rdrand'/> <feature policy='require' name='hypervisor'/> <feature policy='require' name='arat'/> <feature policy='require' name='tsc_adjust'/> <feature policy='require' name='umip'/> <feature policy='require' name='md-clear'/> <feature policy='require' name='stibp'/> <feature policy='require' name='arch-capabilities'/> <feature policy='require' name='ssbd'/> <feature policy='require' name='xsaveopt'/> <feature policy='require' name='pdpe1gb'/> <feature policy='require' name='abm'/> <feature policy='disable' name='skip-l1dfl-vmentry'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='cdrom'> <driver name='qemu'/> <target dev='sda' bus='sata'/> <readonly/> <boot order='1'/> <alias name='sata0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/home/XXX/DIR/test.freebsd/test-hd1-01.qcow2'/> <backingStore/> <target dev='sdb' bus='scsi'/> <boot order='2'/> <alias name='scsi0-0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/> </controller> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x11'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-to-pci-bridge'> <model name='pcie-pci-bridge'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x12'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0x13'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0x14'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0x15'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:cb:db:07'/> <source bridge='br90'/> <target dev='vnet34'/> <model type='e1000'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x02' function='0x0'/> </interface> <interface type='bridge'> <mac address='52:54:00:a5:4c:f3'/> <source bridge='br90'/> <target dev='vnet35'/> <model type='vmxnet3'/> <alias name='net1'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/31'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/31'> <source path='/dev/pts/31'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <input type='keyboard' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'> <alias name='input1'/> </input> <input type='keyboard' bus='ps2'> <alias name='input2'/> </input> <graphics type='spice' port='5934' autoport='yes' listen='127.0.0.1'> <listen type='address' address='127.0.0.1'/> <image compression='off'/> </graphics> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video> <redirdev bus='usb' type='spicevmc'> <alias name='redir0'/> <address type='usb' bus='0' port='2'/> </redirdev> <redirdev bus='usb' type='spicevmc'> <alias name='redir1'/> <address type='usb' bus='0' port='3'/> </redirdev> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='apparmor' relabel='yes'> <label>libvirt-4b1c030f-dfe8-4bd1-9288-be1ced293543</label> <imagelabel>libvirt-4b1c030f-dfe8-4bd1-9288-be1ced293543</imagelabel> </seclabel> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+64055:+106</label> <imagelabel>+64055:+106</imagelabel> </seclabel> </domain> NOTE: I have used Q35 V2.12 as this this that I am using for a number of MacOS machines. Thanks for continuing to looking these problems. Cheers, John Hartley.
(In reply to Tommy P from comment #77) Here is my 3.1 and 4.2 xml ################################################# QEMU 3.1 <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm'> <name>cylon21</name> <uuid>c5cd0551-fe92-5842-d520-f089d2593f61</uuid> <description>FreeBSD 12.1</description> <metadata> <vmtemplate xmlns="unraid" name="FreeBSD" icon="freebsd.png" os="freebsd"/> </metadata> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>2</vcpu> <cputune> <vcpupin vcpu='0' cpuset='1'/> <vcpupin vcpu='1' cpuset='7'/> </cputune> <os> <type arch='x86_64' machine='pc-q35-3.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/c5cd0551-fe92-5842-d520-f089d2593f61_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='1' threads='2'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/FreeBSD-12.1-RELEASE-amd64-dvd1.iso'/> <target dev='hda' bus='sata'/> <readonly/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/user/domains/cylon21/vdisk1.img'/> <target dev='hdc' bus='sata'/> <boot order='1'/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> </disk> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x11'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0x12'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x13'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:d0:8c:7a'/> <source bridge='br0'/> <model type='e1000'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </memballoon> </devices> </domain> ################################################# QEMU 4.2 <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm'> <name>cylon21</name> <uuid>c5cd0551-fe92-5842-d520-f089d2593f61</uuid> <description>FreeBSD 12.1</description> <metadata> <vmtemplate xmlns="unraid" name="FreeBSD" icon="freebsd.png" os="freebsd"/> </metadata> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>2</vcpu> <cputune> <vcpupin vcpu='0' cpuset='1'/> <vcpupin vcpu='1' cpuset='7'/> </cputune> <os> <type arch='x86_64' machine='pc-q35-4.2'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/c5cd0551-fe92-5842-d520-f089d2593f61_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='1' threads='2'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/FreeBSD-12.1-RELEASE-amd64-dvd1.iso'/> <target dev='hda' bus='sata'/> <readonly/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/user/domains/cylon21/vdisk1.img'/> <target dev='hdc' bus='sata'/> <boot order='1'/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x11'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0x12'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x13'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:d0:8c:7a'/> <source bridge='br0'/> <model type='e1000'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </memballoon> </devices> </domain> #################################################
(In reply to MattS from comment #79) Thanks, Matt. Were configurations imported of an older config or did you used the virt-manager to create? Do you use shell scripts to maintain the VMs or a GUI? I'm asking because they're missing: pcie-to-pci-bridge or pci-bridge
(In reply to Tommy P from comment #80) My host OS is unRAID 6.8.1. Currently on Libvirt 5.10.0. VMs are created and managed via GUI(see comment 38), with the option of switching to XML view and making manual changes. My understanding is they provide a pretty vanilla implementation. If there is a template error, I will certainly report it. At the very least the required XML needs to be documented.
(In reply to Tommy P from comment #75) Hi Tommy P, I did the following tests: QEMU Q35 V4.0 / OVMF with VirtIO SCSI & NIC == e1000 + vmxnet + netmap patched FreeBSD: SCSI - VirtIO SCS disk not found NIC - em0 & vmx0 found QEMU Q35 V4.0 / OVMF with SATA & NIC == e1000 + vmxnet + netmap patched FreeBSD: SATA - disk found NIC - em0 & vmx0 found QEMU Q35 V4.0 OVMF with SATA & NIC == e1000e + vmxnet + netmap patched FreeBSD: SATA - disk found NIC - em - not found, vmx0 found So this appears to confirm that there is general issues with Q35 V4.0 PCIe devices (VirtIO PCIe and e100e). I believe this is new bug and should be track in new thread, as it does not relate to the PCI ID conflict that Vincenzo identified, as I already have the fix and still get issue. I will also add this testing info to the other bug report you have found: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241581 Cheers, John Hartley.
Hi Vincenzo, I have retested with 11.4 and this bug is not fixed in 11.4: Tested with Q35 v3.1 with OVMF: VM with: e1000e, e1000, vmxnet3 & rtl8139 all network devices correctly detected: << Start of Test >> #ifconfig -a em0: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC> ether 52:54:00:3d:41:fa hwaddr 52:54:00:3d:41:fa nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active em1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC> ether 52:54:00:dc:37:ab hwaddr 52:54:00:dc:37:ab nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6> ether 52:54:00:a1:30:02 hwaddr 52:54:00:a1:30:02 inet 192.168.73.136 netmask 0xffffff00 broadcast 192.168.73.255 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect status: active re0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE> ether 52:54:00:7e:47:f6 hwaddr 52:54:00:7e:47:f6 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect (100baseTX <full-duplex,flowcontrol,rxpause,txpause>) status: active lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 inet 127.0.0.1 netmask 0xff000000 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> groups: lo << End of Test >> Thank you. Cheers, John Hartley/
(In reply to John Hartley from comment #83) Apologies should be "now fixed in 11.4" So bug can be closed.
(In reply to John Hartley from comment #84) My upgrade was successful as well. Thanks again to everyone!
Please correct me if I'm not reporting at the right bug. I noticed that since the original issue started, it branched off to different bugs. I've just tried to install a new 11.4 machine on a KVM host that we're managing with oVirt. My experience is that I can now get 11.4 to work, but not out of the box: - disk only works with SATA, not with VirtIO or VirtIO-SCSI - NICs are only detected when "custom emulated machine" is set to "pc-q35-rhel8.0.0" or lower. - Even then, NICs only work with e1000 or rtl8139, not with VirtIO
(In reply to Jeroen from comment #86) Hi Jeroen, the virtio bug fix has not flowed into 11.4 See: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236922 Until patch flows through you only have SATA. There is another bug with Q35 4.x, see: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=243640 If you set Q35 version to 3.1 or less you should get: e1000, e1000e, vmxnet3 & rtl8139 working. Cheers, John.
^Triage: to submitter: now that 11.4 is long out of support, can this be closed as Overcome By Events?
(In reply to Mark Linimon from comment #88) Hi Mark, yes agree, I have re-tested with: Ubuntu 22.04 with Libvirt/QEMU/Hypervisor: 8.0.0 / 8.0.0 / 6.2.0 Running VM Q35 Ver 6.2 and all of: Disk: SATA Virtio SCSI/Virtio Network: virtio e1000e vmxnet3 All working ok. So closing "Overcome by Events" as fixed in current FreeBSD 13 version. Thank you. John Hartley
(In reply to John Hartley from comment #89) Clarification, retested with FreeBSD 13.2 Now fixed with current FreeBSD release.