Bug 241774 - FreeBSD 11.3 & 12.0 has broken SCSI & Networking on KVM/QEMU Q35 with OVMF
Summary: FreeBSD 11.3 & 12.0 has broken SCSI & Networking on KVM/QEMU Q35 with OVMF
Status: In Progress
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 11.3-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: Vincenzo Maffione
URL:
Keywords: regression
Depends on:
Blocks:
 
Reported: 2019-11-07 06:00 UTC by John Hartley
Modified: 2020-07-15 14:52 UTC (History)
6 users (show)

See Also:


Attachments
dmesg with 3 disks SATA, SCSI/lsisas1068, VirtIO + 2 NICs e1000, VirtIO (7.61 KB, text/plain)
2019-11-08 08:49 UTC, John Hartley
no flags Details
/var/log/syslog with 3 Disks SATA, SCSI/lsisas1068, VirtIO + 2 NICs e1000, VirtIO (12.22 KB, text/plain)
2019-11-08 08:54 UTC, John Hartley
no flags Details
pciconf with 3 Disks SATA, SCSI/lsisas1608, VirtIO + 2 NICs e1000, VirtIO (6.78 KB, text/plain)
2019-11-08 09:04 UTC, John Hartley
no flags Details
Patch to fix q35 issue (1.64 KB, patch)
2020-01-12 22:30 UTC, Vincenzo Maffione
no flags Details | Diff
unraid create vm gui (58.38 KB, image/png)
2020-01-12 23:37 UTC, MattS
no flags Details
11.3 dmesg output with generic kernel (51.38 KB, image/png)
2020-01-12 23:43 UTC, MattS
no flags Details
11.3 dmesg output with custom kernel netmap removed (57.68 KB, image/png)
2020-01-12 23:44 UTC, MattS
no flags Details
Netmap patch for 11.x (1.19 KB, patch)
2020-01-13 22:08 UTC, Vincenzo Maffione
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description John Hartley 2019-11-07 06:00:27 UTC
BUG:

FreeBSD VM's on Ubuntu KVM / QEMU / OVMF host have SCSI Disk and Network Issues.

Network Interface: em / qemu e1000 / Intel Pro 1G no longer works with 11.3 (it did work with 11.2) or 12.0

TESTING:

I have tested various FreeBSD VM configuration and have found issues with SCSI and Network Device support when using: Ubuntu 19.04 / 19.10 KVM / QEMU Host with: Q35 Machine and OVMF EFI Firmware.

Here is summary of testing:

FreeBSD 11.2 (on KVM / QEMU Q35 with OVMF)
SCSI disk - none work
SATA disk - works
Networking - only e1000 works

FreeBSD 11.2 (on KVM / QEMU Q35 with BIOS)
SCSI disk - works with lsisas1068 controller type
Network - only e1000 works

FreeBSD 11.3 (on KVM / QEMU Q35 with OVMF)
SCSI disk - none work
SATA disk - works
Networking - nothing works
NOTE: I also tested with custom kernel build (ie remove all bit minimal drivers but leaving MPT/MPS & VirtIO SCSI and intel em and VirtIO Networking, still does not work

FreeBSD 12.0 (on KVM / QEMU with OVMF)
SCSI disk - none works
SATA disk - works
Networking - nothing works

So in summary, in move from FreeBSD 11.2 -> 11.3 / 12.X
- networking is broken with KVM / QEMU / OVMF
- SCSI disk support is broken for all KVM / QEMU / OVMF

So while there is work around of SATA and BIOS, for new VMs, I have existing VMs which need to be upgraded and this cannot be done, as network or disk subsystem fails.

I can do additional testing / validation if required.
Comment 1 John Hartley 2019-11-08 08:49:42 UTC
Created attachment 208964 [details]
dmesg with 3 disks SATA, SCSI/lsisas1068, VirtIO + 2 NICs e1000, VirtIO

This attachment has the dmesg results after booting with:

QEMU / KVM - Q35 with OVMF
3 Disks Attached:
  SATA - Found and ok
  SCSI with lsisas1068 controller - now found
  VirtIO - not found

and 2 NICS
  e1000
  VirtIO

I may be trying to many things at once here but thought it might be simpler to try various disk at once.
Comment 2 John Hartley 2019-11-08 08:54:13 UTC
Created attachment 208965 [details]
/var/log/syslog with 3 Disks SATA, SCSI/lsisas1068, VirtIO + 2 NICs e1000, VirtIO


This syslog from: QEMU / KVM - Q35 with OVMF

FreeBSD 11.3 (regular Kernel)

3 Disks
  SATA - ok
  SCSI/lsisas1068 - not found
  VirtIO - not found

2 NICS
  e1000 - not found
  VirtIO - not found
Comment 3 John Hartley 2019-11-08 09:04:08 UTC
Created attachment 208966 [details]
pciconf with 3 Disks SATA, SCSI/lsisas1608, VirtIO + 2 NICs e1000, VirtIO


Here is result from:

pciconf -lcve

FreeBSD 11.3

3 Disks
  SATA - ok
  SCSI/lsisas1068 - not found
  VirtIO - not found

2 NICs
  e1000 - not found
  VirtIO - not found
Comment 4 John Hartley 2019-11-08 09:17:40 UTC
To single view of various variations of disk/nic, I created this VM with 3 disks and 2 NICS and collected the dmesg, syslog and pciconf results.

Here is QEMU VM Desriptor for:

Q35 with OVMF - FreeBSD 11.3 VM

<domain type='kvm'>
  <name>newt-dev-freebsd-11.3</name>
  <uuid>d66dcbb4-d28e-4362-b3c5-486c85b8c4f3</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://freebsd.org/freebsd/11.2"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-4.0'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
    <nvram>/home/XXX/OVMF_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state='off'/>
  </features>
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/newt-hd1-01.qcow2'/>
      <target dev='sdb' bus='sata'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/FreeBSD-11.3-RELEASE-amd64-dvd1.iso'/>
      <target dev='sdc' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/newt-hd2-01.qcow2'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/newt-hd3-01.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
    </controller>
    <controller type='scsi' index='0' model='lsisas1068'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x02' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0x16'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:b6:23:1a'/>
      <source bridge='br01'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:98:20:03'/>
      <source bridge='br01'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='keyboard' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'>
      <listen type='address'/>
      <image compression='off'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
</domain>
Comment 5 MattS 2019-11-21 22:51:13 UTC
I'm also experiencing this issue. Discovered after upgrading 11.2 to 11.3 and e1000 network interface was no longer detected. Rolling back to 11.2 restored NIC. Reproduced the issue by downloading 11.3, 12, 12.1 ISOs and creating new basic VMs. None of the installers are able to detect emulated NIC. 11.2 installer works fine. Something clearly changed in 11.3+.

Currently stuck on eol 11.2 :(

Would love to at least see a workaround.

host OS is unRAID 6.7.2(Slackware?) 
Q35-3.1
OVMF
SATA
e1000
Comment 6 John Hartley 2019-11-22 09:47:49 UTC
Matts and others,

I have a number of BSD related VMs running on KVM / QEMU / libvirt:

MacOS (up to Catalina) with Clover & OVMF UEFI with NIC == e1000-82545em - runs well
pfsense (2.4.3) Router with Q35 / OVMF with NIC == e1000 - runs well

I have also tried: rtl8130 (Realtek) with 12.0 / OVMF - not working

Given that MacOS works with Clover + OVMF, will test with Clover and see if this is able to aid getting NI?C going. MacOS / Darwin is BSD based so this is not so far fetched as it might initially seem.

I resorted to using VMWare for BSD testing at the moment as this works with 11.3 with VMWare VMX specifies: e1000 NIC and lsisas SCSI disk (which is equivalent to KVM/QEMU lsisas1068 SCSI).

In past I have been able to take VMWare vmdk images and run:
  > qemu-img convert -f vmdk -O qcow2 input.vmdk output.qcow2

And then by using e1000 & lsisas1068 they are bootable within KVM / QEMU.

I believe that the NIC issues is more significant than the Disk one, as if SATA breaks then it will get instant focus, while NIC's don't get picked up until latter in testing.

Given that BSD has strong source code culture the solution to my mind is to creates a ports package that be build and installed from within the host as kernel loadable module. So pending getting fix in based could get solution going independent of this.

It might also be worth looking at VMTools for FreeBSD as I believe this is available as source code.

I will report back once I have done further testing.
Comment 7 MattS 2020-01-04 02:29:47 UTC
John, Have you made any progress?

I'll add that my motherboard has 2 NICs and I passed through the I219-V (em driver), and have the same issue as the emulated e1000 NIC. 11.2 works fine, NIC not detected on 11.3+
Comment 8 John Hartley 2020-01-06 00:49:43 UTC
Hi MattS,

no progress sorry, as I hit catch-22...

Do get the VMWare VMTools Port built, I needed access to network, which I do not have ... :-(

A bit of Murphy's law hit as well, as at the same time as finding the 11.3 bug I also hit Ubuntu 19.10 kernel bug, which I also reported and have been working to get this resolved.

Given current dead-lock with networking (and X-mas / new year break having passed), I might try:

1. Create 11.3 build with VMWare Tools installed via VMWare and convert this to KVM/QEMU VM via qemu-img utility and see if this works.

2. See if I can get full source code build going  ....


What we really need is person with serious FreeBSD Kernel / Driver contributor to weigh in on this bug...

I report back on strategy (1) once later today, as this is relatively simple to try.

Cheers,


John
Comment 9 John Hartley 2020-01-06 05:38:44 UTC
Hi Matts,

did some additional testing in hope of find easy resolution.

1. Tested FreeBSD 11.3 on VMWare with vmxnet3 network interface

This works with build in vmx driver that is already part of pre-build drivers so there is no need to explicitly install Open VMWare Tools to get vmxnet3 driver working.


2. Change QEMU / KVM network device type to: vmxnet3

I was not aware that VMWare vmxnet3 was an available option for QEMU, but found this when searching VMWare tools and QEMU / KVM.

Result: still no network interface found...

As counter point test I run 11.2 Live CD with vmxnet3 on QEMU / KVM and network interface was found.

So there is definitely vmxnet3 support within QEMU / KVM / LibVirt on Ubuntu 19.10.

Again problem appears to be 11.3 .

So it looks like all potential quick work-around options are now closed off and only way to fix this is via kernel / driver changes.

Cheers,


John.
Comment 10 MattS 2020-01-06 21:56:15 UTC
Thanks for the update. It's my understanding that Proxmox(host)is not experiencing this issue. I don't use Proxmox, but I guess it doesn't use libvirt. So maybe a clue there.

Any idea on how to actually get some attention on this bug? I really hate being stuck on an EoL OS.
Comment 11 Tommy P 2020-01-07 16:51:12 UTC
Hi John,

I decided to post here instead of my bug report (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236922) since it adds OVMF testing.

I've just created a OVMF + Q35 VM for 12.0 fresh install.  Intel NIC driver works fine for me with the original default install kernel and custom updated kernel.  Note the 'Hypervisor default' SCSI (sym0) loaded below:

root@fbsd12-ovmf:~ # uname -a
FreeBSD fbsd12-ovmf 12.0-RELEASE FreeBSD 12.0-RELEASE r341666 GENERIC  amd64

root@fbsd12-ovmf:~ # dmesg | egrep -i 'scsi|mass stor|sym|sata'
sym0: <895a> port 0xb000-0xb0ff mem 0x98c02000-0x98c023ff,0x98c00000-0x98c01fff irq 22 at device 0.0 on pci2
sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking
pci3: <mass storage, SCSI> at device 0.0 (no driver attached)
ahci0: <Intel ICH9 AHCI SATA controller> port 0xc240-0xc25f mem 0x99002000-0x99002fff irq 16 at device 31.2 on pci0
ada0: <QEMU HARDDISK 2.5+> ATA-7 SATA device
ada0: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes)
cd0: <QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device
cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes)

root@fbsd12-ovmf:~ # uname -a
FreeBSD fbsd12-ovmf 12.0-RELEASE-p12 FreeBSD 12.0-RELEASE-p12 r356437 GENERIC  amd64


Please note that all VirtIO connected devices still fail to load as per my bug report.  Unfortunately, UEFI boot process is a bit more complicated and, thus, I can't simply switch the HDD bus from SATA to SCSI :(.  When I tried a fresh install of 12.0 to the 'Hypervisor default' SCSI, the install is unable to detect the attached SCSI HDD...  Exiting the install to shell to debug, sym driver is already loaded in the kernel but doesn't show up dmesg.  Thus, unable to troubleshoot further with installing to or booting from the SCSI HDD. :(  If I boot from SATA, all non-VirtIO works as expected:

root@fbsd12-ovmf:~ # uname -a
FreeBSD fbsd12-ovmf 12.0-RELEASE-p12 FreeBSD 12.0-RELEASE-p12 r356437 GENERIC  amd64

root@fbsd12-ovmf:~ # dmesg | egrep -i 'scsi|mass stor|sym|sata|em0'
sym0: <895a> port 0xb000-0xb0ff mem 0x98c02000-0x98c023ff,0x98c00000-0x98c01fff irq 22 at device 0.0 on pci2
sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking
pci3: <mass storage, SCSI> at device 0.0 (no driver attached)
em0: <Intel(R) PRO/1000 Network Connection> port 0x6000-0x603f mem 0x98000000-0x9801ffff irq 21 at device 1.0 on pci8
em0: attach_pre capping queues at 1
em0: using 1024 tx descriptors and 1024 rx descriptors
em0: allocated for 1 tx_queues
em0: allocated for 1 rx_queues
em0: Ethernet address: 52:54:00:f2:70:9c
ahci0: <Intel ICH9 AHCI SATA controller> port 0xc240-0xc25f mem 0x99002000-0x99002fff irq 16 at device 31.2 on pci0
da0 at sym0 bus 0 scbus0 target 2 lun 0
da0: <QEMU QEMU HARDDISK 2.5+> Fixed Direct Access SPC-3 SCSI device
ada0: <QEMU HARDDISK 2.5+> ATA-7 SATA device
ada0: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes)
cd0: <QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device
cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes)
em0: link state changed to UP

Perhaps something broken during the upgrade from 11.2?  Is your boot/OS disk SATA or SCSI?  Just a FIY, not all SCSI/RAID hardware HBA supports UEFI boot, IIRC.  Thus, I don't think the emulated 'Hypervisor default' SCSI does either :(  Here's my pciconf output when boot from SATA:

none1@pci0:1:0:0:       class=0x020000 card=0x11001af4 chip=0x10411af4 rev=0x01 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'Virtio network device'
    class      = network
    subclass   = ethernet
    bar   [14] = type Memory, range 32, base rx98e00000, size 4096, enabled
    bar   [20] = type Prefetchable Memory, range 64, base rx800000000, size 16384, enabled
    cap 11[dc] = MSI-X supports 3 messages
                 Table in map 0x14[0x0], PBA in map 0x14[0x800]
    cap 09[c8] = vendor (length 20)
    cap 09[b4] = vendor (length 20)
    cap 09[a4] = vendor (length 16)
    cap 09[94] = vendor (length 16)
    cap 09[84] = vendor (length 16)
    cap 01[7c] = powerspec 3  supports D0 D3  current D0
    cap 10[40] = PCI-Express 2 endpoint max data 128(128)
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
sym0@pci0:2:0:0:        class=0x010000 card=0x10000000 chip=0x00121000 rev=0x00 hdr=0x00
    vendor     = 'LSI Logic / Symbios Logic'
    device     = '53c895a'
    class      = mass storage
    subclass   = SCSI
    bar   [10] = type I/O Port, range 32, base rxb000, size 256, enabled
    bar   [14] = type Memory, range 32, base rx98c02000, size 1024, enabled
    bar   [18] = type Memory, range 32, base rx98c00000, size 8192, enabled
none2@pci0:3:0:0:       class=0x010000 card=0x11001af4 chip=0x10481af4 rev=0x01 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'Virtio SCSI'
    class      = mass storage
    subclass   = SCSI
    bar   [14] = type Memory, range 32, base rx98a00000, size 4096, enabled
    bar   [20] = type Prefetchable Memory, range 64, base rx800100000, size 16384, enabled
    cap 11[dc] = MSI-X supports 4 messages
                 Table in map 0x14[0x0], PBA in map 0x14[0x800]
    cap 09[c8] = vendor (length 20)
    cap 09[b4] = vendor (length 20)
    cap 09[a4] = vendor (length 16)
    cap 09[94] = vendor (length 16)
    cap 09[84] = vendor (length 16)
    cap 01[7c] = powerspec 3  supports D0 D3  current D0
    cap 10[40] = PCI-Express 2 endpoint max data 128(128)
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)

BTW, my host OS is an updated Ubuntu 18.04.3 LTS.  I don't know if that has any bearing vs 19.x
Comment 12 John Hartley 2020-01-08 01:34:31 UTC
Hi Tommy T,

I am a bit puzzled, you our posting...

Your pciconf (below):


<<START>>

none1@pci0:1:0:0:       class=0x020000 card=0x11001af4 chip=0x10411af4 rev=0x01 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'Virtio network device'
    class      = network
    subclass   = ethernet
    bar   [14] = type Memory, range 32, base rx98e00000, size 4096, enabled
    bar   [20] = type Prefetchable Memory, range 64, base rx800000000, size 16384, enabled
    cap 11[dc] = MSI-X supports 3 messages
                 Table in map 0x14[0x0], PBA in map 0x14[0x800]
    cap 09[c8] = vendor (length 20)
    cap 09[b4] = vendor (length 20)
    cap 09[a4] = vendor (length 16)
    cap 09[94] = vendor (length 16)
    cap 09[84] = vendor (length 16)
    cap 01[7c] = powerspec 3  supports D0 D3  current D0
    cap 10[40] = PCI-Express 2 endpoint max data 128(128)
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
sym0@pci0:2:0:0:        class=0x010000 card=0x10000000 chip=0x00121000 rev=0x00 hdr=0x00
    vendor     = 'LSI Logic / Symbios Logic'
    device     = '53c895a'
    class      = mass storage
    subclass   = SCSI
    bar   [10] = type I/O Port, range 32, base rxb000, size 256, enabled
    bar   [14] = type Memory, range 32, base rx98c02000, size 1024, enabled
    bar   [18] = type Memory, range 32, base rx98c00000, size 8192, enabled
none2@pci0:3:0:0:       class=0x010000 card=0x11001af4 chip=0x10481af4 rev=0x01 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'Virtio SCSI'
    class      = mass storage
    subclass   = SCSI
    bar   [14] = type Memory, range 32, base rx98a00000, size 4096, enabled
    bar   [20] = type Prefetchable Memory, range 64, base rx800100000, size 16384, enabled
    cap 11[dc] = MSI-X supports 4 messages
                 Table in map 0x14[0x0], PBA in map 0x14[0x800]
    cap 09[c8] = vendor (length 20)
    cap 09[b4] = vendor (length 20)
    cap 09[a4] = vendor (length 16)
    cap 09[94] = vendor (length 16)
    cap 09[84] = vendor (length 16)
    cap 01[7c] = powerspec 3  supports D0 D3  current D0
    cap 10[40] = PCI-Express 2 endpoint max data 128(128)
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)

<<END>>

This shows VirtIO SCSI & VirtIO Network... but you state that you cannot get VirtIO going (which is same problem I have).

Is this pciconf report from i440FX QEMU/KVM machine rather than a Q35 one ?

I have not managed to flip my test VM from Q35 to i440FX as per discussion in your bug thread yet as I got into large regression build.

I am now part way through building new kernel that is 11.3 with mutant mostly 11.2 networking sub-system.

I say mostly 11.2 networking, as I believe that the VirtIO network version ID bug you identified in combination of possible introduction of defect with changes in the netmap 11.3 code. This is why your fix was not enough and why all the network interfaces are broken not just VirtIO.

My mutant 11.3 Kernel + 11.2 Network build got much bigger than expected as the 11.3 netmap code results in ripple through impact down into all of the driver code and also up into the network abstraction layers :-(

So when I wound back to 11.2 netmap I have to also wind back on these impacts.

I am almost there I believe this this "regressed 11.3" custom kernel.

If I am right about netmap problem (I hope I am ...), then I can home on the specific issue with this code section.

I hope to test soon, after many many builds...

Thanks again for your tips and insights.

Cheers,


John
Comment 13 Tommy P 2020-01-08 04:56:47 UTC
(In reply to John Hartley from comment #12)

Hi John,

If you look again at the pciconf output:

none1@pci0:1:0:0

Implies that no driver were loaded for that pci device at slot 0:1:0:0 since FreeBSD's VirtIO drivers currently does not support PCI-E bus.  The rest of the pciconf are for debugging driver.  For the other output such as:

sym0@pci0:2:0:0

Implies that FreeBSD's LSI/Symbios SCSI (sym) were loaded successfully.  Thus, any HDD attached to that controller would work.  Unfortunately, I don't think this SCSI controller supports UEFI boot.

Here are the details of pciconf, from my understanding, of how FreeBSD's VirtIO drivers work:

none1@pci0:1:0:0:       class=0x020000 card=0x11001af4 chip=0x10411af4 rev=0x01 hdr=0x00

*) rev=0x01 implies PCI-E bus while rev=0x00 implies PCI bus
*) last 4 digits of card/chip implies the vendor ID: 0x1af4 (per /usr/src/sys/dev/virtio/pci/virtio_pci.h)
*) if device is on PCI bus, the first 4 digits of 'card' imply device type (per /usr/src/sys/dev/virtio/virtio_ids.h)
*) if device is on PCI-E bus, the first 4 digits of 'chip' - 0x1040 should match device type from /usr/src/sys/dev/virtio/virtio_ids.h.

Thus, 0x1041 - 0x1040 = 0x1 which is VirtIO NIC per pciconf's verbose output:
   #define VIRTIO_ID_NETWORK       1

And 0x1048 - 0x1040 = 0x8 which is VirtIO SCSI:
   #define VIRTIO_ID_SCSI          8

The current VirtIO drivers does not have the necessary codes to detect the device ID connected to the PCI-E bus nor have the correct memory address to attach the driver for that same bus.

As for non VirtIO not working, my fresh install of 12.0 using OVMF works OK for both sym SCSI and em NIC.  I don't have 11.2 OVMF to test upgrade.  But all of my non OVMF VMs work OK too except where Q35 + VirtIO are concerned as per my bug report.
Comment 14 John Hartley 2020-01-08 21:03:38 UTC
Hi Tommy T,

thanks again for into.

Your freebsd 12.0 pciconf is definitely different to mine.

Is this due to your debug kernel reporting differently to standard GENERIC ?

I am now just doing Live CD boots to see what works and what doesn't as it avoids having to go through lots of install steps that just result in failed install... ;-)

BTW: here is my Ubuntu and QEMU version info:

> virsh -c qemu:///system version --daemon
Compiled against library: libvirt 5.4.0
Using library: libvirt 5.4.0
Using API: QEMU 5.4.0
Running hypervisor: QEMU 4.0.0
Running against daemon: 5.4.0

uname -a
Linux blue 5.3.0-26-generic #28-Ubuntu SMP Wed Dec 18 05:37:46 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/*-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=19.10
DISTRIB_CODENAME=eoan
DISTRIB_DESCRIPTION="Ubuntu 19.10"
NAME="Ubuntu"
VERSION="19.10 (Eoan Ermine)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 19.10"
VERSION_ID="19.10"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=eoan
UBUNTU_CODENAME=eoan

Still working on my GENERIC 11.3 (with 11.2) build. I am almost there now.

Cheers,


John,
Comment 15 Tommy P 2020-01-09 01:28:20 UTC
(In reply to John Hartley from comment #14)

Hi John,

I used pciconf -lBbcevV.  As for the kernel, I only commented out unneeded stuff in GENERIC like SCSI/RAID adapters, NICs, WLAN, xen & vmware, netmap, debug, etc.

root@d-build-fbsd12:/usr/obj # du -hsc /boot/kernel*
 96M    /boot/kernel
 96M    /boot/kernel.old
132M    /boot/kernel.ori.12.0
 93M    /boot/kernel.virtio
418M    total

My host is Ubuntu 18.04.3 LTS so the KVM/QEMU version will be older than yours.

HTH,
Tommy
Comment 16 Tommy P 2020-01-09 03:18:56 UTC
(In reply to John Hartley from comment #14)

Hi John,

Good news!  After some debugging, I was able to get VirtIO to work with Q35 with OVMF for 12.0-RELEASE-p12 r356438M and without OVMF for 12.0-RELEASE-p12 r356535M!
Please see my bug report (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236922) for the attachment and caution regarding netmap.  Here's the pciconf output for the VirtIO:

vtpcim0@pci0:1:0:0:     class=0x020000 card=0x11001af4 chip=0x10411af4 rev=0x01 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'Virtio network device'
    class      = network
    subclass   = ethernet
    bar   [14] = type Memory, range 32, base rx98e00000, size 4096, enabled
    bar   [20] = type Prefetchable Memory, range 64, base rx800000000, size 16384, enabled
    cap 11[dc] = MSI-X supports 3 messages, enabled
                 Table in map 0x14[0x0], PBA in map 0x14[0x800]
    cap 09[c8] = vendor (length 20)
    cap 09[b4] = vendor (length 20)
    cap 09[a4] = vendor (length 16)
    cap 09[94] = vendor (length 16)
    cap 09[84] = vendor (length 16)
    cap 01[7c] = powerspec 3  supports D0 D3  current D0
    cap 10[40] = PCI-Express 2 endpoint max data 128(128)
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
sym0@pci0:2:0:0:        class=0x010000 card=0x10000000 chip=0x00121000 rev=0x00 hdr=0x00
    vendor     = 'LSI Logic / Symbios Logic'
    device     = '53c895a'
    class      = mass storage
    subclass   = SCSI
    bar   [10] = type I/O Port, range 32, base rxb000, size 256, enabled
    bar   [14] = type Memory, range 32, base rx98c02000, size 1024, enabled
    bar   [18] = type Memory, range 32, base rx98c00000, size 8192, enabled
vtpcim1@pci0:3:0:0:     class=0x010000 card=0x11001af4 chip=0x10481af4 rev=0x01 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'Virtio SCSI'
    class      = mass storage
    subclass   = SCSI
    bar   [14] = type Memory, range 32, base rx98a00000, size 4096, enabled
    bar   [20] = type Prefetchable Memory, range 64, base rx800100000, size 16384, enabled
    cap 11[dc] = MSI-X supports 4 messages, enabled
                 Table in map 0x14[0x0], PBA in map 0x14[0x800]
    cap 09[c8] = vendor (length 20)
    cap 09[b4] = vendor (length 20)
    cap 09[a4] = vendor (length 16)
    cap 09[94] = vendor (length 16)
    cap 09[84] = vendor (length 16)
    cap 01[7c] = powerspec 3  supports D0 D3  current D0
    cap 10[40] = PCI-Express 2 endpoint max data 128(128)
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
vtpcim2@pci0:4:0:0:     class=0x078000 card=0x11001af4 chip=0x10431af4 rev=0x01 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'Virtio console'
    class      = simple comms
    bar   [14] = type Memory, range 32, base rx98800000, size 4096, enabled
    bar   [20] = type Prefetchable Memory, range 64, base rx800200000, size 16384, enabled
    cap 11[dc] = MSI-X supports 2 messages
                 Table in map 0x14[0x0], PBA in map 0x14[0x800]
    cap 09[c8] = vendor (length 20)
    cap 09[b4] = vendor (length 20)
    cap 09[a4] = vendor (length 16)
    cap 09[94] = vendor (length 16)
    cap 09[84] = vendor (length 16)
    cap 01[7c] = powerspec 3  supports D0 D3  current D0
    cap 10[40] = PCI-Express 2 endpoint max data 128(128)
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)


Note that previously it was none#@pci now it's vtpcim#@pci.  It also seems to support booting from both SATA and VirtIO Serial:

root@fbsd12-ovmf:~ # ls /dev/{da,ada,vtbd}*
/dev/vtbd0      /dev/vtbd0p1    /dev/vtbd0p2

root@fbsd12-ovmf:~ # ifconfig
vtnet0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6>
        ether ...
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=81209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
        ether ...
        inet ... netmask 0xffffff00 broadcast 10.100.0.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

This was a fresh install of 12.0-RELEASE onto SATA.  After replacing and patching /usr/src in addition to kernel rebuild, it now boots as VirtIO Serial!
Comment 17 MattS 2020-01-09 03:58:42 UTC
This is great work! Can this patch be applied to 11.3, or easily created for 11.3?
Comment 18 John Hartley 2020-01-09 08:44:25 UTC
(In reply to Tommy P from comment #16)

Hi Tommy P,

Hooray great to see you have got VirtIO sorted !!

I have now got my mutant 11.3 with 11.2 network kernel going and ...

It works !!

uname -a
FreeBSD newt.in.graphica.com.au 11.3-RELEASE FreeBSD 11.3-RELEASE #26: Thu Jan  9 18:38:41 AEDT 2020     root@newt.in.graphica.com.au:/usr/obj/usr/src/sys/GENERIC2  amd64

Only tested e1000 at the moment:

% ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
	inet 127.0.0.1 netmask 0xff000000
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
	ether 52:54:00:4e:50:91
	hwaddr 52:54:00:4e:50:91
	inet 192.168.73.131 netmask 0xffffff00 broadcast 192.168.73.255
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active

My approach was a bit sledge hammer as I have ended up with the following modules all being regressed back to 11.2:


 % find . -name '*.bak*' -print
./sbin/ipfw.bak
./sys/modules/mlx5en.bak
./sys/modules/netmap/Makefile.bak
./sys/modules/cxgbe.bak
./sys/modules/cxgbe.bak/if_cxgbe/Makefile.bak
./sys/modules/ipfw_nat64.bak
./sys/modules/mlx5.bak
./sys/modules/urtwn.bak
./sys/amd64/vmm.bak
./sys/amd64/pci.bak
./sys/dev/virtio/network.bak
./sys/dev/vmware/vmxnet3.bak
./sys/dev/pci/pcireg.h.bak.11.2
./sys/dev/cxgbe/firmware.bak.11.3
./sys/dev/usb/wlan.bak
./sys/dev/pci.bak
./sys/dev/netmap.bak
./sys/dev/e1000.bak
./sys/dev/ixgbe.bak
./sys/dev/re.bak
./sys/dev/iwi.bak
./sys/dev/malo.bak
./sys/dev/mwl.bak
./sys/dev/ral.bak
./sys/dev/ixl.bak
./sys/dev/bwi.bak
./sys/dev/cxgbe.bak
./sys/dev/mlx5.bak
./sys/dev/oce.bak
./sys/dev/rtwn.bak
./sys/dev/urtwn.bak
./sys/conf/files.bak
./sys/net.bak
./sys/netinet.bak
./sys/net80211.bak
./sys/netinet6.bak
./sys/netpfil.bak

The need to regress so much was due ripple up / down impacts of /dev/netmap .
As you can see this is pretty much then entire network subsystem... :-(

I am only able to post this as now network is up I can ssh into box to get result out.

First though I am going to test all the other network devices...

Now done:

% ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
	inet 127.0.0.1 netmask 0xff000000
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
	ether 52:54:00:4e:50:91
	hwaddr 52:54:00:4e:50:91
	inet 192.168.73.131 netmask 0xffffff00 broadcast 192.168.73.255
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
vmx0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 52:54:00:0a:cc:0d
	hwaddr 52:54:00:0a:cc:0d
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
re0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
	ether 52:54:00:b2:20:53
	hwaddr 52:54:00:b2:20:53
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect (100baseTX <full-duplex,flowcontrol,rxpause,txpause>)
	status: active


So I have:

e1000 - intel - em0
vmxnet3 - VMWare vmxnet3 Virtual NIC driver (VirtIO alternate) - vmx0
rtl8139 - RealTec - re0

This test also include VirtIO, but as per your testing this does not work and I need to include your fix.

So in summary:

Issue is not with QEMU / KVM , but with FreeBSD code from 11.2 -> 11.3 which 12.x has inherited

There are two bugs:

1. confirmed - VirtIO - the one Tommy T has helped resolve
2. Speculatively - /dev/netmap bug or else where

Bug (2) is that is breaking on the other Q35 (non VirtIO) interfaces (e1000, vmxnet3, rtl8139).

Now that I have a working base I am going for first move forward the PCI code and validate that is ok and then look at the netmap code, which I believe is what is causing the issue with all the other network devices.

Once I get to specific commit/s that introduced the bug I will provide update.

BTW - I am really surprise that such a large and impactful change on network sub-system was part of minor release cycle and nothing was mentioned in the release notes.

Cheers and thanks again Tommy.

John Hartley.
Comment 19 John Hartley 2020-01-09 09:30:02 UTC
Tommy P & Matt S,

wound forward PCI code to 11.3 and network still working ok:

% find . -name '*.bak*' -print
./sbin/ipfw.bak
./sys/modules/mlx5en.bak
./sys/modules/netmap/Makefile.bak
./sys/modules/cxgbe.bak
./sys/modules/cxgbe.bak/if_cxgbe/Makefile.bak
./sys/modules/ipfw_nat64.bak
./sys/modules/mlx5.bak
./sys/modules/urtwn.bak
./sys/amd64/pci.bak.11.2
./sys/amd64/vmm.bak.11.2
./sys/dev/virtio/network.bak
./sys/dev/vmware/vmxnet3.bak
./sys/dev/cxgbe/firmware.bak.11.3
./sys/dev/usb/wlan.bak
./sys/dev/netmap.bak
./sys/dev/e1000.bak
./sys/dev/ixgbe.bak
./sys/dev/re.bak
./sys/dev/iwi.bak
./sys/dev/malo.bak
./sys/dev/mwl.bak
./sys/dev/ral.bak
./sys/dev/ixl.bak
./sys/dev/bwi.bak
./sys/dev/cxgbe.bak
./sys/dev/mlx5.bak
./sys/dev/oce.bak
./sys/dev/rtwn.bak
./sys/dev/urtwn.bak
./sys/dev/pci.bak.11.2
./sys/dev/pci.bak.11.2/pcireg.h.bak.11.2
./sys/conf/files.bak
./sys/net.bak
./sys/netinet.bak
./sys/net80211.bak
./sys/netinet6.bak
./sys/netpfil.bak

All (non VirtIO network ok):

% ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
	inet 127.0.0.1 netmask 0xff000000
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
	ether 52:54:00:4e:50:91
	hwaddr 52:54:00:4e:50:91
	inet 192.168.73.131 netmask 0xffffff00 broadcast 192.168.73.255
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
vmx0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 52:54:00:0a:cc:0d
	hwaddr 52:54:00:0a:cc:0d
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
re0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
	ether 52:54:00:b2:20:53
	hwaddr 52:54:00:b2:20:53
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect (100baseTX <full-duplex,flowcontrol,rxpause,txpause>)


So this confirm that source of issue is in the /dev/netmap code.

I will try to wind this forward on a commit by commit basis to see where it breaks.

Cheers,


John Hartley.
Comment 20 John Hartley 2020-01-09 09:49:05 UTC
Hi Tommy T,

some helpful advise here please, did diff of netmap code from 11.2 -> 11.3:

% diff netmap netmap.bak | wc
   15891   67956  464277

Files 19 -> 25:

% ls netmap.bak | wc
      25      25     371
% ls netmap | wc
      19      19     288

and grep of FBSDIDs:

diff netmap netmap.bak | grep FBSDID
< __FBSDID("$FreeBSD: releng/11.2/sys/dev/netmap/netmap_generic.c 331722 2018-03-29 02:50:57Z eadler $");
> __FBSDID("$FreeBSD: releng/11.3/sys/dev/netmap/netmap_generic.c 344047 2019-02-12 09:26:05Z vmaffione $");
< __FBSDID("$FreeBSD: releng/11.2/sys/dev/netmap/netmap_mem2.c 331722 2018-03-29 02:50:57Z eadler $");
> __FBSDID("$FreeBSD: releng/11.3/sys/dev/netmap/netmap_mem2.c 344047 2019-02-12 09:26:05Z vmaffione $");
< __FBSDID("$FreeBSD: releng/11.2/sys/dev/netmap/netmap_vale.c 331722 2018-03-29 02:50:57Z eadler $");
> __FBSDID("$FreeBSD: releng/11.3/sys/dev/netmap/netmap_vale.c 344047 2019-02-12 09:26:05Z vmaffione $");


Given the amount of change it would seem prudent to get committer to look at this: vmaffione

What are your thought and how to we get additional eyes on this ?

Thanks.

Cheers,


John Hartley.
Comment 21 Vincenzo Maffione freebsd_committer 2020-01-09 22:59:29 UTC
Can you please elaborate about the bug that you think there is in netmap?
The large changes are due to refactoring and new features, but it is nothing that should affect driver operation when the NIC is not open in netmap mode.
Comment 22 John Hartley 2020-01-10 06:25:23 UTC
(In reply to Vincenzo Maffione from comment #21)

Hi Vincenzo,

thank you very much of for responding.

My testing has been to try to isolate where the problems lies.

The issues appears when you go from 11.2 -> 11.3 .

As the problem relates to device detection / initialisation, my initial thought was that problem might be in PCI/PCIe code.

To test this I tried to create a custom 11.3 kernel with 11.2 PCI subsystem.

The impact of regression to 11.2 PCI was that I also had to regress netmap, network and driver code back to 11.2 as well.

The resulting 11.3 (with 11.2 pci / networking) kernel no longer had the issue (ie e1000, vmxnet3 & rtl8139 all working ok)

So I then rolled forward the pci code back to 11.3 .

Again testing worked with each of e1000, vmxnet3 & rtl8139 working.

So this seemed to have issue as being in netmap or drivers ...

In looking at record of code updates, the netmap related changes (refactoring etc) and adding support for netmap in e1000 & vmxnet3 drivers all related to MFC r339639 (which I believe was big upstream netmap alignment and refactoring).

So right now I have not confirmed that the bug is in netmap, but it might be...

On a related front this VirtIO bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236922 required changes to both VirtIO driver and netmap.

Right now I am a bit stuck in finding the defect, as netmap and driver code is more complex and requires deeper understand than I have. My objective was just try to pin point where the problem was.

Having looked at this I now have a much greater appreciation of what the dev/netmap code does and why it impacts such a large portion of network code.

On my working 11.3 / 11.2 custom kernel here is dmesg result from em0:

em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x9000-0x903f mem 0xc8600000-0xc861ffff irq 23 at device 1.0 on pci2
em0: Ethernet address: 52:54:00:4e:50:91
em0: netmap queues/slots: TX 1/256, RX 1/256


So it appears that netmap is enabled by default.
The only configuration I have done is set DHCP on:

# cat /etc/rc.conf
clear_tmp_enable="YES"
...
ifconfig_em0="DHCP"
local_unbound_enable="YES"
sshd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"



Right now I have not managed to get problem isolated so specific code item...

Your advise on helping to get to the bottom of problem would be appreciated.


Cheers,

John Hartley
Comment 23 Tommy P 2020-01-10 07:41:22 UTC
@John,

Are you encountering issues while upgrading 11.2 to 11.3 with Q35 + OVMF?  I don't have any issues:

root@fbsd11-ovmf:~ # dmesg|grep em0
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x6000-0x603f mem 0x98000000-0x9801ffff irq 21 at device 1.0 on pci8
em0: Ethernet address: 52:54:00:4e:d4:8f
em0: netmap queues/slots: TX 1/256, RX 1/256
em0: link state changed to UP

root@fbsd11-ovmf:~ # uname -a
FreeBSD fbsd11-ovmf 11.2-RELEASE-p15 FreeBSD 11.2-RELEASE-p15 #0 r356556: Thu Jan  9 19:50:57 PST 2020
---------------------------------------------------
root@fbsd11-ovmf:~ # dmesg | egrep 'sym0|mass stor|scsi'
sym0: <895a> port 0xb000-0xb0ff mem 0x98c02000-0x98c023ff,0x98c00000-0x98c01fff irq 22 at device 0.0 on pci2
sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking
pci3: <mass storage, SCSI> at device 0.0 (no driver attached)

root@fbsd11-ovmf:~ # dmesg|grep em0
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x6000-0x603f mem 0x98000000-0x9801ffff irq 21 at device 1.0 on pci2
em0: Ethernet address: 52:54:00:4e:d4:8f
em0: netmap queues/slots: TX 1/256, RX 1/256
em0: link state changed to UP

root@fbsd11-ovmf:~ # uname -a
FreeBSD fbsd11-ovmf 11.3-RELEASE-p5 FreeBSD 11.3-RELEASE-p5 #0: Tue Nov 12 08:59:04 UTC 2019


VirtIO still doesn't work for 11.x :(  The pci3 is the VirtIO SCSI (note the 'no driver attached').


@Matt, (from comment #17)

From what I've seen so far, there are significant changes in the networking from 11.x to 12.x which the VirtIO, supporting PCI-E, is based upon.  I doubt it's going to be easy to 'back port' to 11.x.

Regards,
Tommy
Comment 24 Vincenzo Maffione freebsd_committer 2020-01-10 19:07:07 UTC
(In reply to John Hartley from comment #22)
MFC r339639 does not add netmap support to e1000, vtnet & co. Such support exists since FreeBSD 10. Moreover, netmap code is platform independent, so it does not make sense that a bug in netmap would show up in Q35 and not in the other platform.

It is more reasonable that the issue is related to the PCI code.
If I get it correctly, your issue is that you don't detect devices (e.g. you don't see em0 or vtnet0 in ifconfig). If this happens, it means that the device_probe routine of the proper driver is not called, or maybe it is called but it does not recognize the device for some reason. For example, you have em_probe() or lem_probe() for e1000 devices. You may add a print statement there to see whether it is called or not.
Note that device probe is a functionality related to the PCI/PCIe bus, irrespective of netmap.

Have you tried to remove "dev netmap" from the kernel config (for a clean 11.3)? This will confirm that the problem is still there irrespective of netmap being included or not.
Comment 25 John Hartley 2020-01-10 23:28:24 UTC
(In reply to Vincenzo Maffione from comment #24)

Hi Vincenzo,

as suggested, I have compiled kernel with netmap removed:


# diff GENERIC GENERIC2
22c22
< ident		GENERIC
---
> ident		GENERIC2
360c360
< device		netmap			# netmap(4) support
---
> #device		netmap			# netmap(4) support


# uname -a
FreeBSD newt.in.graphica.com.au 11.3-RELEASE FreeBSD 11.3-RELEASE #31: Sat Jan 11 08:25:05 AEDT 2020     root@newt.in.graphica.com.au:/usr/obj/usr/src/sys/GENERIC2  amd64

I had previously tried to disable netmap from e1000 only and still get result of no network devices being found.

Now with netmap removed from kernel the e1000 device is found:

# ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
	inet 127.0.0.1 netmask 0xff000000
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
	ether 52:54:00:4e:50:91
	hwaddr 52:54:00:4e:50:91
	inet 192.168.73.131 netmask 0xffffff00 broadcast 192.168.73.255
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active

Here is PCI Conf results:

# pciconf -lvce
hostb0@pci0:0:0:0:	class=0x060000 card=0x11001af4 chip=0x29c08086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82G33/G31/P35/P31 Express DRAM Controller'
    class      = bridge
    subclass   = HOST-PCI
vgapci0@pci0:0:1:0:	class=0x030000 card=0x11001af4 chip=0x01001b36 rev=0x04 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'QXL paravirtual graphic card'
    class      = display
    subclass   = VGA
pcib1@pci0:0:2:0:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x1(x32) speed 2.5(undef) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(engaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 000d[148] = ACS 1
pcib2@pci0:0:2:1:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x1(x32) speed 2.5(undef) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 000d[148] = ACS 1
pcib3@pci0:0:2:2:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x1(x32) speed 2.5(undef) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 000d[148] = ACS 1
pcib4@pci0:0:2:3:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x32(x32) speed undef(undef) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(empty) surprise Attn Button PC(off) EI(disengaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 000d[148] = ACS 1
uhci0@pci0:0:29:0:	class=0x0c0300 card=0x11001af4 chip=0x29348086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
uhci1@pci0:0:29:1:	class=0x0c0300 card=0x11001af4 chip=0x29358086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
uhci2@pci0:0:29:2:	class=0x0c0300 card=0x11001af4 chip=0x29368086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
ehci0@pci0:0:29:7:	class=0x0c0320 card=0x11001af4 chip=0x293a8086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB2 EHCI Controller'
    class      = serial bus
    subclass   = USB
isab0@pci0:0:31:0:	class=0x060100 card=0x11001af4 chip=0x29188086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801IB (ICH9) LPC Interface Controller'
    class      = bridge
    subclass   = PCI-ISA
ahci0@pci0:0:31:2:	class=0x010601 card=0x11001af4 chip=0x29228086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]'
    class      = mass storage
    subclass   = SATA
    cap 05[80] = MSI supports 1 message, 64 bit enabled with 1 message
    cap 12[a8] = SATA Index-Data Pair
none0@pci0:0:31:3:	class=0x0c0500 card=0x11001af4 chip=0x29308086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) SMBus Controller'
    class      = serial bus
    subclass   = SMBus
pcib5@pci0:1:0:0:	class=0x060400 card=0x00000000 chip=0x000e1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    class      = bridge
    subclass   = PCI-PCI
    cap 05[8c] = MSI supports 1 message, 64 bit, vector masks 
    cap 01[84] = powerspec 3  supports D0 D3  current D0
    cap 10[48] = PCI-Express 2 PCI bridge max data 128(128) ARI disabled
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
    cap 0c[40] = unknown
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
em0@pci0:2:1:0:	class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82540EM Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet

So there is some subtle interaction happing across PCI discovery and netmap code.

I will now add other adaptors into test and post additional results shortly.

Thanks for the advise it really helped with diagnosis.

Cheers,


John Hartley.
Comment 26 John Hartley 2020-01-10 23:42:29 UTC
(In reply to Vincenzo Maffione from comment #24)

Hi Vincenzo,


here are the results for e1000 (em), vmxnet3 (vmx) virtio (vtnet) and rtl813 (re):

# ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
	inet 127.0.0.1 netmask 0xff000000
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
	ether 52:54:00:4e:50:91
	hwaddr 52:54:00:4e:50:91
	inet 192.168.73.131 netmask 0xffffff00 broadcast 192.168.73.255
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
vmx0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 52:54:00:fb:be:74
	hwaddr 52:54:00:fb:be:74
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
re0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
	ether 52:54:00:b9:0c:12
	hwaddr 52:54:00:b9:0c:12
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect (100baseTX <full-duplex,flowcontrol,rxpause,txpause>)
	status: active

And PCI Conf:

# pciconf -lcve
hostb0@pci0:0:0:0:	class=0x060000 card=0x11001af4 chip=0x29c08086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82G33/G31/P35/P31 Express DRAM Controller'
    class      = bridge
    subclass   = HOST-PCI
vgapci0@pci0:0:1:0:	class=0x030000 card=0x11001af4 chip=0x01001b36 rev=0x04 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'QXL paravirtual graphic card'
    class      = display
    subclass   = VGA
pcib1@pci0:0:2:0:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x1(x32) speed 2.5(undef) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(engaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 000d[148] = ACS 1
pcib2@pci0:0:2:1:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x1(x32) speed 2.5(undef) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 000d[148] = ACS 1
pcib3@pci0:0:2:2:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x1(x32) speed 2.5(undef) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 000d[148] = ACS 1
pcib4@pci0:0:2:3:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x1(x32) speed 2.5(undef) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 000d[148] = ACS 1
uhci0@pci0:0:29:0:	class=0x0c0300 card=0x11001af4 chip=0x29348086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
uhci1@pci0:0:29:1:	class=0x0c0300 card=0x11001af4 chip=0x29358086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
uhci2@pci0:0:29:2:	class=0x0c0300 card=0x11001af4 chip=0x29368086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
ehci0@pci0:0:29:7:	class=0x0c0320 card=0x11001af4 chip=0x293a8086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB2 EHCI Controller'
    class      = serial bus
    subclass   = USB
isab0@pci0:0:31:0:	class=0x060100 card=0x11001af4 chip=0x29188086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801IB (ICH9) LPC Interface Controller'
    class      = bridge
    subclass   = PCI-ISA
ahci0@pci0:0:31:2:	class=0x010601 card=0x11001af4 chip=0x29228086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]'
    class      = mass storage
    subclass   = SATA
    cap 05[80] = MSI supports 1 message, 64 bit enabled with 1 message
    cap 12[a8] = SATA Index-Data Pair
none0@pci0:0:31:3:	class=0x0c0500 card=0x11001af4 chip=0x29308086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) SMBus Controller'
    class      = serial bus
    subclass   = SMBus
pcib5@pci0:1:0:0:	class=0x060400 card=0x00000000 chip=0x000e1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    class      = bridge
    subclass   = PCI-PCI
    cap 05[8c] = MSI supports 1 message, 64 bit, vector masks 
    cap 01[84] = powerspec 3  supports D0 D3  current D0
    cap 10[48] = PCI-Express 2 PCI bridge max data 128(128) ARI disabled
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
    cap 0c[40] = unknown
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
em0@pci0:2:1:0:	class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82540EM Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet
vmx0@pci0:2:2:0:	class=0x020000 card=0x07b015ad chip=0x07b015ad rev=0x01 hdr=0x00
    vendor     = 'VMware'
    device     = 'VMXNET3 Ethernet Controller'
    class      = network
    subclass   = ethernet
    cap 11[9c] = MSI-X supports 25 messages, enabled
                 Table in map 0x18[0x0], PBA in map 0x18[0x1000]
    cap 05[84] = MSI supports 1 message, 64 bit 
re0@pci0:2:3:0:	class=0x020000 card=0x11001af4 chip=0x813910ec rev=0x20 hdr=0x00
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTL-8100/8101L/8139 PCI Fast Ethernet Adapter'
    class      = network
    subclass   = ethernet


All devices discovered with exception of virtio.
I will put in fix that Tommy T found for virtio and retest.

I am glad that we appear to be making progress in diagnosing issue.

Cheers,


John Hartley.
Comment 27 Vincenzo Maffione freebsd_committer 2020-01-11 12:32:25 UTC
(In reply to John Hartley from comment #25)
Just to double check: you are using 11.3-release, and not your "frankestein kernel" (11.3 with backported code from 11.2)?

Assuming a positive answer, you are saying that with GENERIC you don't see em0 showing up, and if you remove "dev netmap" (GENERIC2) then em0 shows up?
That would be very weird...
Can you point me at the revision that you used for the build?
Comment 28 John Hartley 2020-01-11 23:26:27 UTC
(In reply to Vincenzo Maffione from comment #27)

Hi Vincenzo,

the 11.3 kernel I compiled was standard (ie not my 11.3 / 11.2 Frankenstein) only change to GENERIC being "device netmap" disablement.

Result was all Q35 network devices: e1000 (em), vmxnet3 (vmx) and rtl8139 (re) are working.

Now I have isolated problem I have "retired" Frankenstein ;-)

I have also now done equivalent kernel build with 12.0 and get the same result:

# uname -a
FreeBSD agent.in.graphica.com.au 12.0-RELEASE FreeBSD 12.0-RELEASE GENERIC2  amd64

# diff /usr/src/sys/amd64/conf/GENERIC2 /usr/src/sys/amd64/conf/GENERIC
22c22
< ident		GENERIC2
---
> ident		GENERIC
362c362
< #device		netmap			# netmap(4) support
---
> device		netmap			# netmap(4) support


# ifconfig -a
vmx0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 52:54:00:07:92:50
	media: Ethernet autoselect
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=81209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
	ether 52:54:00:cb:db:07
	inet 192.168.73.102 netmask 0xffffff00 broadcast 192.168.73.255 
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
re0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
	ether 52:54:00:64:37:eb
	media: Ethernet autoselect (100baseTX <full-duplex,flowcontrol,rxpause,txpause>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 
	inet 127.0.0.1 netmask 0xff000000 
	groups: lo 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

# pciconf -l
hostb0@pci0:0:0:0:	class=0x060000 card=0x11001af4 chip=0x29c08086 rev=0x00 hdr=0x00
vgapci0@pci0:0:1:0:	class=0x030000 card=0x11001af4 chip=0x01001b36 rev=0x04 hdr=0x00
pcib1@pci0:0:2:0:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
pcib3@pci0:0:2:1:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
pcib4@pci0:0:2:2:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
pcib5@pci0:0:2:3:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
pcib6@pci0:0:2:4:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
uhci0@pci0:0:29:0:	class=0x0c0300 card=0x11001af4 chip=0x29348086 rev=0x03 hdr=0x00
uhci1@pci0:0:29:1:	class=0x0c0300 card=0x11001af4 chip=0x29358086 rev=0x03 hdr=0x00
uhci2@pci0:0:29:2:	class=0x0c0300 card=0x11001af4 chip=0x29368086 rev=0x03 hdr=0x00
ehci0@pci0:0:29:7:	class=0x0c0320 card=0x11001af4 chip=0x293a8086 rev=0x03 hdr=0x00
isab0@pci0:0:31:0:	class=0x060100 card=0x11001af4 chip=0x29188086 rev=0x02 hdr=0x00
ahci0@pci0:0:31:2:	class=0x010601 card=0x11001af4 chip=0x29228086 rev=0x02 hdr=0x00
none0@pci0:0:31:3:	class=0x0c0500 card=0x11001af4 chip=0x29308086 rev=0x02 hdr=0x00
pcib2@pci0:1:0:0:	class=0x060400 card=0x00000000 chip=0x000e1b36 rev=0x00 hdr=0x01
vmx0@pci0:2:1:0:	class=0x020000 card=0x07b015ad chip=0x07b015ad rev=0x01 hdr=0x00
em0@pci0:2:2:0:	class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00
re0@pci0:2:3:0:	class=0x020000 card=0x11001af4 chip=0x813910ec rev=0x20 hdr=0x00
none1@pci0:5:0:0:	class=0x00ff00 card=0x11001af4 chip=0x10451af4 rev=0x01 hdr=0x00

So problem in 12.X is consistent with that in 11.3 .

Let me know if there is any other testing or information you would like to do / provide.

Thanks.

Cheers,

John Hartley.
Comment 29 Tommy P 2020-01-12 02:22:38 UTC
John & Matt,

What is/are the CPU/s for your host OS?  Did you verify that the relevant virtual technology features are enabled on the host: IOMMU for AMD and VT/D (?) for Intel.  IIRC, some older versions of Linux (CentOS, Ubuntu, and Fedora) requires additional manual entry in the grub's boot loader.  The modern Linux _shouldn't_ but doesn't hurt to confirm:

dmesg|grep -i ommu
[    0.000000] AGP: Please enable the IOMMU option in the BIOS setup
[    3.434650] iommu: Adding device 0000:00:00.0 to group 0
<snip/>
[    3.536829] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    3.536834] AMD-Vi: Found IOMMU at 0000:40:00.2 cap 0x40


grep -e '^#' -v /etc/default/grub
GRUB_DEFAULT=0
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX=""
GRUB_DISABLE_SUBMENU=y


How are you creating/managing your VMs on the host?  I use virt-manager (to reduce typos, misconfiguration, or unsupported configuation errors) on Ubuntu to manage.  Did you customize any of the configurations for the VM in the shell script (if used to maintain the VMs) or XML backend?

Here's a screen shot for the VM's config: https://imgur.com/a/1WbzvhQ
Below is the command line startup for that VM without any manual customization in the XML configuration file:

ps -ef|grep 'fbsd11_ovmf'
root      8914 20540  0 15:19 pts/7    00:00:00 grep --color=auto fbsd11_ovmf
libvirt+ 25561     1 40 Jan09 ?        17:21:30 qemu-system-x86_64 -enable-kvm -name guest=z_fbsd12_ovmf,debug-threads=on -S
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-35-z_fbsd12_ovmf/master-key.aes
-machine pc-q35-2.11,accel=kvm,usb=off,vmport=off,dump-guest-core=off
-cpu Opteron_G5,vme=on,ht=on,monitor=on,osxsave=on,bmi1=on,mmxext=on,fxsr_opt=on,cmp_legacy=on,extapic=on,cr8legacy=on,osvw=on,ibs=on,skinit=on,wdt=on,tce=on,nodeid_msr=on,topoext=on,perfctr_core=on,perfctr_nb=on,invtsc=on,ibpb=on
-drive file=/usr/share/OVMF/OVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on
-drive file=/var/lib/libvirt/qemu/nvram/z_fbsd12_ovmf_VARS.fd,if=pflash,format=raw,unit=1
<...snip...>
-no-user-config
-nodefaults
<...snip...>
-rtc base=utc,driftfix=slew
-global kvm-pit.lost_tick_policy=delay
-no-hpet
-no-shutdown
-global ICH9-LPC.disable_s3=1
-global ICH9-LPC.disable_s4=1
-boot menu=on,strict=on
-device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e
-device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0  (<<< this should be em0 NIC)
-device pcie-root-port,port=0x10,chassis=3,id=pci.3,bus=pcie.0,multifunction=on,addr=0x2  (<<< this should be the VirtIO NIC)
-device pcie-root-port,port=0x11,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x1  (<<< sym0 LSI/Symbios SCSI)
-device pcie-root-port,port=0x12,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x2  (<<< VirtIO SCSI)
-device pcie-root-port,port=0x13,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x3  (<<< VirtIO Serial)
-device pcie-root-port,port=0x14,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x4  (<<< VirtIO Memory Balloon)
-device pcie-root-port,port=0x15,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x5  (<<< 'Controller PCIe 0' seen in the screen shot ?)
-device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x1d.0x7
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x1d
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x1d.0x1
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x1d.0x2
-device lsi,id=scsi0,bus=pci.4,addr=0x0        (<<< sym0 LSI/Symbios SCSI)
-device virtio-scsi-pci,id=scsi1,bus=pci.5,addr=0x0
-device virtio-serial-pci,id=virtio-serial0,bus=pci.6,addr=0x0
<...snip...>
-netdev tap,fd=39,id=hostnet0
-device e1000,netdev=hostnet0,id=net0,mac=52:54:00:4e:d4:8f,bus=pci.2,addr=0x1
-netdev tap,fd=41,id=hostnet1,vhost=on,vhostfd=42
-device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:a9:08:fb,bus=pci.3,addr=0x0
<...snip...>
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
<...snip...>
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x1
<...snip...>
-device virtio-balloon-pci,id=balloon0,bus=pci.7,addr=0x0
-msg timestamp=on
---------------------------------------------------------------
11.3-RELEASE-p5 r356613 boot log messages for em0 NIC:

root@fbsd11-ovmf:~ # egrep 'em0' /var/log/messages
Jan 10 00:29:26 fbsd11-ovmf kernel: em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x6000-0x603f mem 0x98000000-0x9801ffff irq 21 at device 1.0 on pci2
Jan 10 00:29:26 fbsd11-ovmf kernel: em0: Ethernet address: 52:54:00:4e:d4:8f
Jan 10 00:29:26 fbsd11-ovmf kernel: em0: netmap queues/slots: TX 1/256, RX 1/256
Jan 10 00:29:26 fbsd11-ovmf kernel: em0: link state changed to UP
Jan 10 01:08:26 fbsd11-ovmf kernel: em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x6000-0x603f mem 0x98000000-0x9801ffff irq 21 at device 1.0 on pci2
Jan 10 01:08:26 fbsd11-ovmf kernel: em0: Ethernet address: 52:54:00:4e:d4:8f
Jan 10 01:08:26 fbsd11-ovmf kernel: em0: netmap queues/slots: TX 1/256, RX 1/256
Jan 10 01:08:26 fbsd11-ovmf kernel: em0: link state changed to UP
Jan 10 01:26:48 fbsd11-ovmf kernel: em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x6000-0x603f mem 0x98000000-0x9801ffff irq 21 at device 1.0 on pci2
Jan 10 01:26:48 fbsd11-ovmf kernel: em0: Ethernet address: 52:54:00:4e:d4:8f
Jan 10 01:26:48 fbsd11-ovmf kernel: em0: netmap queues/slots: TX 1/256, RX 1/256
Jan 10 01:26:48 fbsd11-ovmf kernel: em0: link state changed to UP
Jan 11 16:00:03 fbsd11-ovmf kernel: em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x6000-0x603f mem 0x98000000-0x9801ffff irq 21 at device 1.0 on pci2
Jan 11 16:00:03 fbsd11-ovmf kernel: em0: Ethernet address: 52:54:00:4e:d4:8f
Jan 11 16:00:03 fbsd11-ovmf kernel: em0: netmap queues/slots: TX 1/256, RX 1/256
Jan 11 16:00:03 fbsd11-ovmf kernel: em0: link state changed to UP
---------------------------------------------------------------
Snippets of my custom kernel since 11.2-RELEASE-p15 and upgraded to 11.3-RELEASE-p5 r356613:

root@fbsd11-ovmf:~ # grep -e '^#' -v /usr/src/sys/amd64/conf/Custom|grep -e '^$' -v
cpu             HAMMER
ident           GENERIC
options         SCHED_ULE               # ULE scheduler
options         PREEMPTION              # Enable kernel thread preemption
<snip>
device          acpi
options         ACPI_DMAR
device          pci
options         PCI_HP                  # PCI-Express native HotPlug
options         PCI_IOV                 # PCI SR-IOV support
device          ahci                    # AHCI-compatible SATA controllers
device          ata                     # Legacy ATA/SATA controllers
<snip>
device          em                      # Intel PRO/1000 Gigabit Ethernet Family
device          miibus                  # MII bus support
device          re                      # RealTek 8139C+/8169/8169S/8110S
<snip>
device          uhci                    # UHCI PCI->USB interface
device          ohci                    # OHCI PCI->USB interface
device          ehci                    # EHCI PCI->USB interface (USB 2.0)
device          xhci                    # XHCI PCI->USB interface (USB 3.0)
device          usb                     # USB Bus (required)
device          ukbd                    # Keyboard
device          umass                   # Disks/Mass storage - Requires scbus and da
device          virtio                  # Generic VirtIO bus (required)
device          virtio_pci              # VirtIO PCI device
device          vtnet                   # VirtIO Ethernet device
device          virtio_blk              # VirtIO Block device
device          virtio_scsi             # VirtIO SCSI device
device          virtio_balloon          # VirtIO Memory Balloon device
device          netmap                  # netmap(4) support
device          crypto                  # Required by IPSEC
---------------------------------------------------------------
dmesg of 11.2 & upgraded to 11.3 CPU detection which matches my Opteron 6370p:

FreeBSD 11.2-RELEASE-p15 #0 r356556: Thu Jan  9 19:50:57 PST 2020
    root@fbsd11-ovmf:/usr/obj/usr/src11.2/sys/Custom amd64
FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
VT(efifb): resolution 800x600
CPU: AMD Opteron 63xx class CPU (2003.51-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x600f20  Family=0x15  Model=0x2  Stepping=0
  Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0xbeb83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,HV>
  AMD Features=0x26500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,LM>
  AMD Features2=0x210bf7<LAHF,CMP,SVM,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,XOP,FMA4,TBM>
  Structured Extended Features=0x8<BMI1>
  AMD Extended Feature Extensions ID EBX=0x1000
  SVM: NAsids=16
  TSC: P-state invariant
Hypervisor: Origin = "KVMKVMKVM"

FreeBSD 11.3-RELEASE-p5 #0 r356613: Sat Jan 11 15:56:34 PST 2020
    root@fbsd11-ovmf:/usr/obj/usr/src11.3/sys/Custom amd64
FreeBSD clang version 8.0.0 (tags/RELEASE_800/final 356365) (based on LLVM 8.0.0)
VT(efifb): resolution 800x600
CPU: AMD Opteron 63xx class CPU (2004.09-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x600f20  Family=0x15  Model=0x2  Stepping=0
  Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0xbeb83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,HV>
  AMD Features=0x26500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,LM>
  AMD Features2=0x210bf7<LAHF,CMP,SVM,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,XOP,FMA4,TBM>
  Structured Extended Features=0x8<BMI1>
  AMD Extended Feature Extensions ID EBX=0x1000
  SVM: NAsids=16
  TSC: P-state invariant
Hypervisor: Origin = "KVMKVMKVM"

cat /proc/cpuinfo (on host Ubuntu 18.0.4.3 LTS)
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 21
model           : 2
model name      : AMD Opteron(tm) Processor 6370P
stepping        : 0
<snip>
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall bmi1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
bugs            : fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
<snip>
---------------------------------------------------------------

This VM was initially created to ensure that my proposed VirtIO patches would also work in Q35 + OMVF for FreeBSD 12 (hence it's name: z_fbsd12_ovmf).  I've since used this configuration to install FreeBSD 11.2 to test viability of backporting my proposed patches.  The order of the devices seen in the command line startup matches the order of customizations I've made/added in the GUI virt-manager during the creation of the VM.  Did you have a chane to test my proposed VirtIO patches for FreeBSD 12.x within your environment?  The VirtIO devices should work along with SATA storage devices and em NICs.  I'm unsure about the VirtIO + netmap.

Regards,
Tommy
Comment 30 John Hartley 2020-01-12 08:19:53 UTC
(In reply to Tommy P from comment #29)

Hi Tommy P,

my machine is Lenovo X3650 M5 Type 8871 with Intel E5-2697A v4 CPUs.

I think this corresponds to "Broadwell Architecture".

So it is VT-D HW Virtualisation, with GRUB setup up for PCI Passthrough etc:

cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
...


GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=1 intel_iommu=on ipv6.disable=1"
GRUB_CMDLINE_LINUX="ipv6.disable=1"


QEMU / KVM is going fine including PCI pass through (which requires grub GRUB_CMDLINE_LINUX_DEFAULT config above).
I have also enabled nested hypervisor support, but have not used it so far.

Running combination of: FreeBSD, Ubuntu, Windows and MacOS VMs, without problems (bar this one).

Sorry for delayed response I have been working on seeing if I can get backport of your VirtIO fix on 11.3 . Have done all the coding and now testing, unsuccessfully so far :-( .

My VirtIO backport included:

Merging - virtio.c/.h, balloon & mmio code
Adding - endian support code
Replacing virtio/pci code with your sub-tree.

Not sure why it is not working at this point.

Backport of virtio/network code is likely to be harder as this has adopted CK (concurrency kit) queues and I have not managed to get clean compile.

I will post on backport success / failure in our bug thread.

Cheers,


John Hartley.
Comment 31 Vincenzo Maffione freebsd_committer 2020-01-12 08:52:31 UTC
(In reply to John Hartley from comment #28)

So this very same issue (em0 not detected with "dev netmap", but detected without it) happens also with 12.x?
Can you please attach the configuration (xml) that you are using with virsh or virt-manager? (e.g., using "virsh dumpxml ${VMNAME}")
Comment 32 Tommy P 2020-01-12 20:45:04 UTC
(In reply to John Hartley from comment #30)

Hi John,

Backporting to 11.x is not easy.  There new significant changes from what I've seen thus far in 12.x such as:

sys/netinet 
sys/netinet/netdump

Coincidentally, while I do not any issues with netmap and non VirtIO devices within my environment, netmap does break VirtIO detection and loading for Q35, OVMF or not.

Regards,
Tommy
Comment 33 John Hartley 2020-01-12 21:56:13 UTC
(In reply to Vincenzo Maffione from comment #31)

Hi Vincenzo,

find following virsh dump of 11.3 test VM.
My 12.0 test VM is essentially the same.
Both find network interfaces with disabled (via kernel config sys/amd64/conf/GENERIC2).

You will notice that i have 3 NICs on VM (for em0, vmx0 & re0 in FreeBSD).
I have removed VirtoIO NIC as we know that there is seperate issue with this that Tommy T and Brian V has been dealing with.

NOTE: I am managing QEMU / KVM using "Virtual Machine Manager" UI.

# cat newt-qemu-02.xml
<domain type='kvm' id='56'>
  <name>newt-dev-freebsd-11.3</name>
  <uuid>01d93f61-8527-42f0-b79d-fda37cf87f71</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://freebsd.org/freebsd/11.2"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-4.0'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
    <nvram>/home/XXX/DDD/newt.dev.freebsd/OVMF_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state='off'/>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Broadwell-IBRS</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='f16c'/>
    <feature policy='require' name='rdrand'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='arat'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='umip'/>
    <feature policy='require' name='md-clear'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='arch-capabilities'/>
    <feature policy='require' name='ssbd'/>
    <feature policy='require' name='xsaveopt'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='abm'/>
    <feature policy='disable' name='skip-l1dfl-vmentry'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu'/>
      <target dev='sdb' bus='sata'/>
      <readonly/>
      <boot order='1'/>
      <alias name='sata0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/XXX/DDD/newt.dev.freebsd/newt-hd1-01.qcow2'/>
      <backingStore/>
      <target dev='sdc' bus='sata'/>
      <boot order='2'/>
      <alias name='sata0-0-2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/XXX/DDD/test.freebsd/test-hd1-01.qcow2'/>
      <backingStore/>
      <target dev='sdd' bus='sata'/>
      <alias name='sata0-0-3'/>
      <address type='drive' controller='0' bus='0' target='0' unit='3'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x11'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x12'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x13'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x14'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:4e:50:91'/>
      <source bridge='br90'/>
      <target dev='vnet31'/>
      <model type='e1000'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:fb:be:74'/>
      <source bridge='br90'/>
      <target dev='vnet32'/>
      <model type='vmxnet3'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:b9:0c:12'/>
      <source bridge='br90'/>
      <target dev='vnet33'/>
      <model type='rtl8139'/>
      <alias name='net2'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/28'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/28'>
      <source path='/dev/pts/28'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='keyboard' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='spice' port='5932' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
      <image compression='off'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir0'/>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir1'/>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='apparmor' relabel='yes'>
    <label>libvirt-01d93f61-8527-42f0-b79d-fda37cf87f71</label>
    <imagelabel>libvirt-01d93f61-8527-42f0-b79d-fda37cf87f71</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+64055:+106</label>
    <imagelabel>+64055:+106</imagelabel>
  </seclabel>
</domain>

Tommy T and other are now also able to reproduced the with/without netmap and NIC detection issue.

Thanks again for assistance.

Cheers,


John Hartley.
Comment 34 John Hartley 2020-01-12 22:08:45 UTC
(In reply to Tommy P from comment #32)

Hi Tommy T,

thanks for tip on backport.

I have come across the netdump issue with my backport and #defined any code for this out to avoid compile issues.

Currently I have compiled and bootable 11.3 with the virtio backport, but virtio is not working yet.

I am just testing with disk at the moment as to your point virtio network is other issues which Vincenzo (from upstream netmap team) is helping to look at.

I believe with Vincenzo (netmap), Brian V (virtio) and yourself we have the right people to get to root cause of netmap issue.

I put in weekend with backport but doubt I will able to do much concentrated work with it during week due to work commitments.

Good thing is we are now getting multiple people now reproducing the Q35 netmap issue.

I still believe that this is some subtle PCI / netmap interaction, as the detection has already failed by early kernel load stage of boot.

Cheers,


John Hartley.
Comment 35 Vincenzo Maffione freebsd_committer 2020-01-12 22:28:53 UTC
(In reply to John Hartley from comment #34)

I found the issue, and you are basically right.
It's a PCI ID conflict between the QEMU PCI express root port device and the netmap passthrough device. I must assume the PCIE root port is present in Q35 and not in i440fx, which explains why the issue only affects Q35.

I will disable the netmap passthrough devices until the proper support is committed to QEMU.
Sorry for the inconvenience. I'll commit the fix tomorrow.
Comment 36 Vincenzo Maffione freebsd_committer 2020-01-12 22:30:53 UTC
Created attachment 210670 [details]
Patch to fix q35 issue

This patch will fix the issue by disabling ptnetmap devices (in conflict with newer QEMU virtual PCI devices).
Comment 37 John Hartley 2020-01-12 23:05:17 UTC
(In reply to Vincenzo Maffione from comment #35)

HI Vincenzo,

great !!

Yes you are right as well about PCIe root devices.

The reason is that Q35 is native PCIe VM and so have PCIe root devices while i440FX is PCI/ISA (not PCE-e) machine.

I will apply patch and advise on results.

Cheers,

John Hartley.
Comment 38 MattS 2020-01-12 23:37:45 UTC
Created attachment 210673 [details]
unraid create vm gui
Comment 39 MattS 2020-01-12 23:40:06 UTC
On my Host OS unRAID 6.8.0, i440fx with seabios or OVFM FreeBSD just hangs on boot. One of the unRAID developers (jonp) spent a little time on it, and was also unsuccessful. It's possible that it's an unRAID specific issue. Not sure at this point.

https://forums.unraid.net/topic/84692-freebsd-113-no-network-interface-after-upgrade/page/2/?tab=comments#comment-809306

I was able to get virtio NIC working on 12.1 with TommyP's patch. (Thank you!)

***************************************
What is/are the CPU/s for your host OS?

cat /proc/cpuinfo (unRAID 6.8 running Linux 4.19.88 kernel)
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 158
model name      : Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
stepping        : 10
microcode       : 0xca
cpu MHz         : 4299.634
cache size      : 12288 KB
physical id     : 0
siblings        : 12
core id         : 0
cpu cores       : 6
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit
bogomips        : 6384.00
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:
***************************************
Libvirt version:
5.8.0
QEMU version:
4.1.1
***************************************
Did you verify that the relevant virtual technology features are enabled on the host: IOMMU for AMD and VT/D (?) for Intel?

I'm not sure how to show this on unRAID, but VT/D is enabled and PCIe ACS Override is enabled for better IMMO grouping. I have a variety of VMs and Dockers working. Ubuntu, Windows 10, FreeBSD 11.2, MacOS, Kali Linux. Multiple GPU's, NIC, audio, and USB cards passed through to VMs. 
****************************************
How are you creating/managing your VMs on the host?

VM configuration is done via GUI (see attached) with an option to edit XML if needed. By default, the template network adapter type is "virtio", I then have to switch to XML view and change to e1000.

XML below is just a default configuration with NIC changed to e1000 (FreeBSD 11.3, Q35-4.1

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' id='21'>
  <name>cylon20</name>
  <uuid>522c2643-079b-7464-7120-70fdc2f7ec57</uuid>
  <description>FreeBSD 11.3</description>
  <metadata>
    <vmtemplate xmlns="unraid" name="FreeBSD" icon="freebsd.png" os="freebsd"/>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='1'/>
    <vcpupin vcpu='1' cpuset='7'/>
    <vcpupin vcpu='2' cpuset='5'/>
    <vcpupin vcpu='3' cpuset='11'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-4.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/522c2643-079b-7464-7120-70fdc2f7ec57_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='2' threads='2'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/FreeBSD-11.3-RELEASE-amd64-dvd1.iso'/>
      <backingStore/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/cylon20/vdisk1.img'/>
      <backingStore/>
      <target dev='hdc' bus='sata'/>
      <boot order='1'/>
      <alias name='sata0-0-2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:bc:6a:0e'/>
      <source bridge='br0'/>
      <target dev='vnet1'/>
      <model type='e1000'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-21-cylon20/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='vnc' port='5901' autoport='yes' websocket='5701' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
</domain>
+++++++++++++++++++++++++++++++++++++++
-uuid 522c2643-079b-7464-7120-70fdc2f7ec57 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=37,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x7.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x7 \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x7.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x7.0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \
-drive file=/mnt/user/isos/FreeBSD-11.3-RELEASE-amd64-dvd1.iso,format=raw,if=none,id=drive-sata0-0-0,readonly=on \
-device ide-cd,bus=ide.0,drive=drive-sata0-0-0,id=sata0-0-0,bootindex=2 \
-drive file=/mnt/user/domains/cylon20/vdisk1.img,format=raw,if=none,id=drive-sata0-0-2,cache=writeback \
-device ide-hd,bus=ide.2,drive=drive-sata0-0-2,id=sata0-0-2,bootindex=1,write-cache=on \
-netdev tap,fd=41,id=hostnet0 \
-device e1000,netdev=hostnet0,id=net0,mac=52:54:00:bc:6a:0e,bus=pci.1,addr=0x0 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=42,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-vnc 0.0.0.0:1,websocket=5701 \
-k en-us \
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x1 \
-device virtio-balloon-pci,id=balloon0,bus=pci.3,addr=0x0 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=o
+++++++++++++++++++++++++++++++++++++++

pciconf does not show any intel NIC device detected. (same with passed through Intel NIC). 

When I compile 11.3 kernel without NETMAP, I see NIC interface detected as "none2@pci0:1:0:0", but still does  not work. (see attached)

See DMESG output screenshots. "em0: Unable to allocate bus resource: interrupt"

***************************************

Okay, I fail at adding attachments...
Comment 40 MattS 2020-01-12 23:43:33 UTC
Created attachment 210674 [details]
11.3 dmesg output with generic kernel
Comment 41 MattS 2020-01-12 23:44:29 UTC
Created attachment 210675 [details]
11.3 dmesg output with custom kernel netmap removed
Comment 42 MattS 2020-01-13 00:07:56 UTC
(In reply to Vincenzo Maffione from comment #36)

This is great news. I didn't notice the progress on this while I was working on my previous replies.

Will this patch be committed to 11.3+?
Comment 43 Tommy P 2020-01-13 01:48:44 UTC
(In reply to Vincenzo Maffione from comment #35)

Hi Vincenzo,

I came to the same conclusion just now too:

ptnetmap-memdev0: <ptnetmap-memdev PCI adapter> mem 0xfd412000-0xfd412fff irq 22 at device 2.0 on pci0
000.000080 [ 871] ptn_memdev_attach         ptn_memdev_driver attach
ptnetmap-memdev0: cannot map I/O space
device_attach: ptnetmap-memdev0 attach returned 6

If I'm correct, it looks like netmap is trying to attach PCIe device on a PCI bus.  What I find puzzling is that netmap supports igb (sys/dev/netmap/if_igb_netmap.h) which is PCIe (I don't have an igb to verify) but somehow fail to attach other PCIe devices:

https://www.freebsd.org/cgi/man.cgi?query=igb&apropos=0&sektion=4&manpath=FreeBSD+12.1-RELEASE+and+Ports&arch=default&format=html

Thanks,
Tommy
Comment 44 MattS 2020-01-13 03:57:17 UTC
(In reply to Vincenzo Maffione from comment #36)

I got a compile error, the line below was missing the trailing slash

"+#endif /* WITH_PTNETMAP *"
Comment 45 John Hartley 2020-01-13 05:07:30 UTC
(In reply to MattS from comment #44)

Hi MattS & Vincenzo,

I have build new 12.0 based kernel with variation of patches provides (as I believe these where against 12.1).

Good results !

Here is info:

# diff GENERIC GENERIC2
22c22
< ident		GENERIC
---
> ident		GENERIC2

So no longer disabled netmap

# uname -av
FreeBSD agent.in.graphica.com.au 12.0-RELEASE FreeBSD 12.0-RELEASE GENERIC2  amd64

So looks like uname -a not longer provides compile date info....

# ifconfig -a
vmx0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 52:54:00:07:92:50
	media: Ethernet autoselect
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=81209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
	ether 52:54:00:cb:db:07
	inet 192.168.73.102 netmask 0xffffff00 broadcast 192.168.73.255 
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
re0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
	ether 52:54:00:64:37:eb
	media: Ethernet autoselect (100baseTX <full-duplex,flowcontrol,rxpause,txpause>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 
	inet 127.0.0.1 netmask 0xff000000 
	groups: lo 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

All network devices bar virtio are ok.

# pciconf -lcve
hostb0@pci0:0:0:0:	class=0x060000 card=0x11001af4 chip=0x29c08086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82G33/G31/P35/P31 Express DRAM Controller'
    class      = bridge
    subclass   = HOST-PCI
vgapci0@pci0:0:1:0:	class=0x030000 card=0x11001af4 chip=0x01001b36 rev=0x04 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'QXL paravirtual graphic card'
    class      = display
    subclass   = VGA
pcib1@pci0:0:2:0:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 000d[148] = ACS 1
pcib3@pci0:0:2:1:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(empty) surprise Attn Button PC(off) EI(disengaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 000d[148] = ACS 1
pcib4@pci0:0:2:2:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(empty) surprise Attn Button PC(off) EI(disengaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 000d[148] = ACS 1
pcib5@pci0:0:2:3:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 000d[148] = ACS 1
pcib6@pci0:0:2:4:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(empty) surprise Attn Button PC(off) EI(disengaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 000d[148] = ACS 1
uhci0@pci0:0:29:0:	class=0x0c0300 card=0x11001af4 chip=0x29348086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
uhci1@pci0:0:29:1:	class=0x0c0300 card=0x11001af4 chip=0x29358086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
uhci2@pci0:0:29:2:	class=0x0c0300 card=0x11001af4 chip=0x29368086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
ehci0@pci0:0:29:7:	class=0x0c0320 card=0x11001af4 chip=0x293a8086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB2 EHCI Controller'
    class      = serial bus
    subclass   = USB
isab0@pci0:0:31:0:	class=0x060100 card=0x11001af4 chip=0x29188086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801IB (ICH9) LPC Interface Controller'
    class      = bridge
    subclass   = PCI-ISA
ahci0@pci0:0:31:2:	class=0x010601 card=0x11001af4 chip=0x29228086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]'
    class      = mass storage
    subclass   = SATA
    cap 05[80] = MSI supports 1 message, 64 bit enabled with 1 message
    cap 12[a8] = SATA Index-Data Pair
none0@pci0:0:31:3:	class=0x0c0500 card=0x11001af4 chip=0x29308086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) SMBus Controller'
    class      = serial bus
    subclass   = SMBus
pcib2@pci0:1:0:0:	class=0x060400 card=0x00000000 chip=0x000e1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    class      = bridge
    subclass   = PCI-PCI
    cap 05[8c] = MSI supports 1 message, 64 bit, vector masks 
    cap 01[84] = powerspec 3  supports D0 D3  current D0
    cap 10[48] = PCI-Express 2 PCI bridge max data 128(128) ARI disabled
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
    cap 0c[40] = unknown
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
vmx0@pci0:2:1:0:	class=0x020000 card=0x07b015ad chip=0x07b015ad rev=0x01 hdr=0x00
    vendor     = 'VMware'
    device     = 'VMXNET3 Ethernet Controller'
    class      = network
    subclass   = ethernet
    cap 11[9c] = MSI-X supports 25 messages, enabled
                 Table in map 0x18[0x0], PBA in map 0x18[0x1000]
    cap 05[84] = MSI supports 1 message, 64 bit 
em0@pci0:2:2:0:	class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82540EM Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet
re0@pci0:2:3:0:	class=0x020000 card=0x11001af4 chip=0x813910ec rev=0x20 hdr=0x00
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTL-8100/8101L/8139 PCI Fast Ethernet Adapter'
    class      = network
    subclass   = ethernet
none1@pci0:5:0:0:	class=0x00ff00 card=0x11001af4 chip=0x10451af4 rev=0x01 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'Virtio memory balloon'
    class      = old
    cap 09[c8] = vendor (length 20)
    cap 09[b4] = vendor (length 20)
    cap 09[a4] = vendor (length 16)
    cap 09[94] = vendor (length 16)
    cap 09[84] = vendor (length 16)
    cap 01[7c] = powerspec 3  supports D0 D3  current D0
    cap 10[40] = PCI-Express 2 endpoint max data 128(128)
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)

In additional to network devices getting virtio balloon reported.

Based on this my strategy for 11.2 is likely to be;

1. Wait for 12.X update with netmap patch with either fixes issue or disables netmap (as per Vincenzo's current update).

2. Upgrade from 11.2 -> 12.X

The reason is that it is now some time since 11.3 and working on backport seems to be significant effort with little value. Best to just jump to 12.x fix and avoid going to intermediate releases.

I tried to backport the 12.X VirtoIO port and this is messy and even Vincenzo's netmap update needs to be reworks for 11.3 based fix.

If Vincenzo provides commit that flow through as update then I will take copy of production 11.2 VM and validate doing upgrade to 12.X on this.
This should provide path for others caught on 11.2 .


Cheers,


John Hartley.
Comment 46 MattS 2020-01-13 05:49:58 UTC
So I'm unclear on how long this takes to make it to RELEASE. Are you saying we have to wait for version 12.2 in October 2020? Or will it be applied to 12.1 as a security patch?

My kernel build failed when I tried to apply Vincenzo's patch along with TommyP's patch. I ran out of time, so I'll be trying again later. I'd did get a working virtio NIC with TommyP's patch, so ultimately that's all I need.
Comment 47 Dobri Dobrev 2020-01-13 09:35:18 UTC
Can someone provide a 1-patch so we can apply without wondering "do we apply only patch1 + patch3, or patch1+2+3, or what ever.."
Comment 48 John Hartley 2020-01-13 17:27:45 UTC
(In reply to Dobri Dobrev from comment #47)

Hi Dobri,

by way of testing I am doing:

1. Build 12.0-RELEASE with "dev netmap" commented out of GENERIC kernel to get network access on any of em, vmx or re . You have to keep the kernel with GENERIC id as otherwise update in step 2 will fail. If you are on 12.0 then I assume you must have SATA disk as Virto and SCSI will not work.

2. Use "freebsd-update" (freebsd-update -r 9.1-RELEASE upgrade) to do upgrade from 12.0-RELEASE to 12.1-RELEASE. This will require two reboots via to "free-update" steps (both /usr/sbin/freebsd-update install).

3. Build 12.1-RELEASE with "dev netmap" commented out of GENERIC kernel and then do final "freebsd-update" (/usr/sbin/freebsd-update install).

4. Do your Ports or Package updates using postmaster or other tools in usual way.

You should now have working 12.1 and can apply VirtIO patch (again via kernel rebuild) if you want to change from SATA to VirtIO based disk.

At this point you should be able to start to use "freebsd-update" to and other utilities to get binary security updates.

In fullness of time bugs / patches for fix of VirtIO and netmap should flow through as either binary updates via "freebsd-update" or be part of new release (12.2).

Right now I am testing process and have got to step 4 (it is slow as each kernel rebuild takes some time).

Once I have done upgrade test from 12.0 -> 12.1 I will then try some approach but starting from 11.2 and go to 12.1.

Once on 12.1 I can test VirtIO fix without need to worry about trying to backport it 11.x .

Both VirtIO and netmap are "performance optimisations". So the extent of need is driven by throughput requirements of your server. A while back I did upgrade of quite a large number of FreeBSD VMs (from 8.x, 9.x & 10.x to 11.x) and at the time I was forced to change disk to SATA as SCSI (and VirtIO) was broken already at that stage.

The final breaking of networking was what prompted me to raise this bug report, as I can live with SATA but I can't not have networking.

So I believe you can get out of the "bind" without needing any of patches in the first instance.

Cheers,

John Hartley.
Comment 49 Dobri Dobrev 2020-01-13 18:56:19 UTC
(In reply to John Hartley from comment #48)

I'm doing directly on 12.1-release, placing the 1st patch (comment 24 I think it was) on am usb stick, patch the src, rebuild kernel - virtio starts working.

Problem is - there are several patches here, now I don't know what to use / not use which is why I asked if a single patch can be made.
Comment 50 John Hartley 2020-01-13 19:27:47 UTC
(In reply to Dobri Dobrev from comment #49)

Hi Dobri,

Assuming you are on Q35 with OVMF (as this is what I have tested against).

If you are happy to use SATA then you don't need any patches.

Just recompile your kernel with /usr/src/sys/amd64/conf/GENERIC but with the "device netmap" commented out in GENERIC.

362c362
< #device		netmap			# netmap(4) support
---
> device		netmap			# netmap(4) support

This will give you a machine with SATA HD and working networking.

If you also want to be able to use VirtIO then you will also need to apply the VirtIO patch.

However if you do this then you must make sure you also recompile with netmap disabled if you want networking.

The result with be a machine that has VirtIO HD and any of em, vm or re networking.

Right now I do not believe you can have VirtIO networking only HD.

If you do decide to build with VirtIO support then you will need to change your /etc/fstab mounts to reflect the different disk naming:

SATA - adaNpX
VirtIO SCSI - daNpX
VirtIO block - vtbdNpX

Where N is disk id and X is partition number.

So what you do reflects the choices you make, until this is all fully fixed as part of a release.

I hope this is clear...

Cheers,


John Hartley.
Comment 51 Vincenzo Maffione freebsd_committer 2020-01-13 20:51:32 UTC
(In reply to MattS from comment #42)
I will commit it to head and MFC to stable/12 and stable/11, so that it will be picked up for the next 11.x release. Would that be good enough?
Comment 52 Vincenzo Maffione freebsd_committer 2020-01-13 20:53:11 UTC
(In reply to MattS from comment #44)
Sorry about the typo, I created it in a hurry yesterday night.
I did test the correct patch on my Q35 Freebsd 12.x VM, and it worked.
Comment 53 Vincenzo Maffione freebsd_committer 2020-01-13 21:14:21 UTC
(In reply to MattS from comment #46)
I'll try to push for an hotfix. Let's see how it goes.
Comment 54 MattS 2020-01-13 21:34:21 UTC
(In reply to Vincenzo Maffione from comment #51)

Can you post the 11.x patch so I can manually install it?

I think John enlightened me on why I was having problems with i440fx testing, partition scheme incompatible when switching to seabios. Brief test shows installer ISO does detect NIC with i440fx/seabios. Not sure if I can convert from GPT to MBR.

I have a few options at this point, so I'm happy. Thank you again for your attention on this issue.
Comment 55 MattS 2020-01-13 21:37:25 UTC
(In reply to Vincenzo Maffione from comment #53)

Pardon my ignorance, does a hotfix mean it will be pulled in with "freebsd-update fetch"?
Comment 56 commit-hook freebsd_committer 2020-01-13 21:48:23 UTC
A commit references this bug:

Author: vmaffione
Date: Mon Jan 13 21:47:24 UTC 2020
New revision: 356704
URL: https://svnweb.freebsd.org/changeset/base/356704

Log:
  netmap: disable passthrough with no hypervisor support

  The netmap passthrough subsystem requires proper support in the
  hypervisor. In particular, two PCI device ids (from the Red Hat
  PCI vendor id 0x1b36) need to be assigned to the two netmap
  virtual devices. We then disable these devices until the ids have
  not been assigned, in order to avoid conflicts with other
  virtual devices emulated by upstream QEMU.

  PR:	241774
  MFC after:	3 days

Changes:
  head/sys/dev/netmap/if_ptnet.c
  head/sys/dev/netmap/netmap_kern.h
  head/sys/net/netmap_virt.h
Comment 57 John Hartley 2020-01-13 22:04:15 UTC
(In reply to Vincenzo Maffione from comment #53)

Hi Vincenzo,

it would be good to get hot fix.

My understanding of your fix proposal is that due to conflicts with PCEe IDs that you are "disabling netmap passthrough support".

Does this mean that we loose the performance advantages provide netmap completely ?

For final fix (ie perfect future we dream of ;-) ) , will get Q35 PCIe hypervisor with full advantages of netmap based buffer management ?

I have since read the netmap paper: https://github.com/luigirizzo/netmap#references and it would be a significant loss to have netmap disabled any future releases.

Cheers,


John Hartley.
Comment 58 Vincenzo Maffione freebsd_committer 2020-01-13 22:08:46 UTC
Created attachment 210719 [details]
Netmap patch for 11.x
Comment 59 Vincenzo Maffione freebsd_committer 2020-01-13 22:12:19 UTC
(In reply to MattS from comment #55)
I've never used it, but yes, I think so. According to the manual and handbook, that's what we should expect, provided I manage to get the fix in the releng branches.
Comment 60 Vincenzo Maffione freebsd_committer 2020-01-13 22:17:48 UTC
(In reply to John Hartley from comment #57)
The patch is not disabling netmap.
You can and should keep using it.

The patch is only disabling a specific netmap feature that QEMU (and bhyve) does not support yet. IOW it is disabling a feature that you cannot use right now (unless you use a non-standard patched QEMU). In practice, this means that we are not losing functionality.
Comment 61 John Hartley 2020-01-13 22:20:30 UTC
(In reply to Vincenzo Maffione from comment #60)

Great to hear Vincenzo and many thanks for acting so quickly on this.

Ciao from Australia ;-)

John Hartley
Comment 62 Tommy P 2020-01-14 17:00:03 UTC
(In reply to Dobri Dobrev from comment #49)
See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236922#c41
Comment 63 MattS 2020-01-16 01:08:09 UTC
(In reply to Vincenzo Maffione from comment #58)

There may still be a problem on 11.x.

I successfully upgraded a copy of my production vm to 11.3 (QEMU 3.1, OVMF, e1000, sata). Everything seems great. I then switched to QEMU 4.0, 4.01, 4.1 and e1000 is no longer working. Switch back to 3.1 and e1000 is working again.

pciconf
none2@pci0:1:0:0:       class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82540EM Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet

dmesg
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x9000-0x903f mem 0x98600000-0x9861ffff irq 22 at device 0.0 on pci1
em0: Unable to allocate bus resource: interrupt
device_attach: em0 attach returned 6
Comment 64 commit-hook freebsd_committer 2020-01-16 20:55:57 UTC
A commit references this bug:

Author: vmaffione
Date: Thu Jan 16 20:55:15 UTC 2020
New revision: 356804
URL: https://svnweb.freebsd.org/changeset/base/356804

Log:
  MFC r356704

  netmap: disable passthrough with no hypervisor support

  The netmap passthrough subsystem requires proper support in the
  hypervisor. In particular, two PCI device ids (from the Red Hat
  PCI vendor id 0x1b36) need to be assigned to the two netmap
  virtual devices. We then disable these devices until the ids have
  not been assigned, in order to avoid conflicts with other
  virtual devices emulated by upstream QEMU.

  PR:     241774

Changes:
_U  stable/12/
  stable/12/sys/dev/netmap/if_ptnet.c
  stable/12/sys/dev/netmap/netmap_kern.h
  stable/12/sys/net/netmap_virt.h
Comment 65 commit-hook freebsd_committer 2020-01-16 20:57:59 UTC
A commit references this bug:

Author: vmaffione
Date: Thu Jan 16 20:57:30 UTC 2020
New revision: 356805
URL: https://svnweb.freebsd.org/changeset/base/356805

Log:
  MFC r356704

  netmap: disable passthrough with no hypervisor support

  The netmap passthrough subsystem requires proper support in the
  hypervisor. In particular, two PCI device ids (from the Red Hat
  PCI vendor id 0x1b36) need to be assigned to the two netmap
  virtual devices. We then disable these devices until the ids have
  not been assigned, in order to avoid conflicts with other
  virtual devices emulated by upstream QEMU.

  PR:     241774

Changes:
_U  stable/11/
  stable/11/sys/dev/netmap/netmap_kern.h
  stable/11/sys/net/netmap_virt.h
Comment 66 Vincenzo Maffione freebsd_committer 2020-01-16 21:04:10 UTC
(In reply to MattS from comment #63)
This may be an unrelated problem... can you attach the guest dmesg of the problematic configuration? (e.g., QEMU 4.x).
When you say "it does not work", do you mean that em0 is not showing up?
Comment 67 MattS 2020-01-16 21:31:52 UTC
(In reply to Vincenzo Maffione from comment #66)

That is correct, does not show up under ifconfig.

pciconf shows device as none2@pci0:1:0:0 vs em0@pci0:4:0:0
Comment 68 MattS 2020-01-16 21:34:37 UTC
(In reply to MattS from comment #67)


My last comment was cut off for some reason.

I'll create clean 11.3 vm with your netmap patch, verify issue, and post full dmesg/pciconf
Comment 69 MattS 2020-01-17 02:14:30 UTC
(In reply to Vincenzo Maffione from comment #66)

Installed clean 11.3 RELEASE as QEMU 4.1. No NIC shows up under ifconfig, 
Change to QEMU 3.1 and em0 shows up. Let me know if you need anything else.

#################################################
11.3 +netmap patch, QEMU 4.1, OVMF, SATA, E1000
+++++++++++++++++++++++++++++++++++++++++++++++++
ifconfig

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
	inet 127.0.0.1 netmask 0xff000000
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo
+++++++++++++++++++++++++++++++++++++++++++++++++
pciconf -lv

hostb0@pci0:0:0:0:	class=0x060000 card=0x11001af4 chip=0x29c08086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82G33/G31/P35/P31 Express DRAM Controller'
    class      = bridge
    subclass   = HOST-PCI
vgapci0@pci0:0:1:0:	class=0x030000 card=0x11001af4 chip=0x01001b36 rev=0x04 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'QXL paravirtual graphic card'
    class      = display
    subclass   = VGA
pcib1@pci0:0:2:0:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
pcib2@pci0:0:2:1:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
pcib3@pci0:0:2:2:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
pcib4@pci0:0:2:3:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
uhci0@pci0:0:7:0:	class=0x0c0300 card=0x11001af4 chip=0x29348086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
uhci1@pci0:0:7:1:	class=0x0c0300 card=0x11001af4 chip=0x29358086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
uhci2@pci0:0:7:2:	class=0x0c0300 card=0x11001af4 chip=0x29368086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
none0@pci0:0:7:7:	class=0x0c0320 card=0x11001af4 chip=0x293a8086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB2 EHCI Controller'
    class      = serial bus
    subclass   = USB
isab0@pci0:0:31:0:	class=0x060100 card=0x11001af4 chip=0x29188086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801IB (ICH9) LPC Interface Controller'
    class      = bridge
    subclass   = PCI-ISA
ahci0@pci0:0:31:2:	class=0x010601 card=0x11001af4 chip=0x29228086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]'
    class      = mass storage
    subclass   = SATA
none1@pci0:0:31:3:	class=0x0c0500 card=0x11001af4 chip=0x29308086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) SMBus Controller'
    class      = serial bus
    subclass   = SMBus
none2@pci0:1:0:0:	class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82540EM Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet

+++++++++++++++++++++++++++++++++++++++++++++++++
dmesg

Copyright (c) 1992-2019 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.3-RELEASE #1: Thu Jan 16 16:19:11 PST 2020
    root@cylon20:/usr/obj/usr/src/sys/MYKERNEL amd64
FreeBSD clang version 8.0.0 (tags/RELEASE_800/final 356365) (based on LLVM 8.0.0)
VT(efifb): resolution 800x600
CPU: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz (3192.06-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x906ea  Family=0x6  Model=0x9e  Stepping=10
  Features=0x1f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT>
  Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x1c4fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP>
  Structured Extended Features2=0x4<UMIP>
  Structured Extended Features3=0x8c000400<MD_CLEAR,IBPB,STIBP,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
real memory  = 4294967296 (4096 MB)
avail memory = 4063502336 (3875 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <BOCHS  BXPCAPIC>
WARNING: L1 data cache covers less APIC IDs than a core
0 < 1
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 hardware threads
ioapic0 <Version 1.1> irqs 0-23 on motherboard
SMP: AP CPU #1 Launched!
random: entropy device external interface
kbd1 at kbdmux0
000.000030 [4213] netmap_init               netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff81004eb0, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
nexus0
cryptosoft0: <software crypto> on motherboard
acpi0: <BOCHS BXPCFACP> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x608-0x60b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xa0c0-0xa0df mem 0x94000000-0x97ffffff,0x90000000-0x93ffffff,0x98800000-0x98801fff irq 21 at device 1.0 on pci0
vgapci0: Boot video device
pcib1: <ACPI PCI-PCI bridge> mem 0x98807000-0x98807fff irq 22 at device 2.0 on pci0
pcib1: [GIANT-LOCKED]
pcib2: <PCI-PCI bridge> mem 0x98806000-0x98806fff irq 22 at device 2.1 on pci0
pcib2: Failed to allocate interrupt for PCI-e events
pcib3: <PCI-PCI bridge> mem 0x98805000-0x98805fff irq 22 at device 2.2 on pci0
pcib3: Failed to allocate interrupt for PCI-e events
pcib4: <PCI-PCI bridge> mem 0x98804000-0x98804fff irq 22 at device 2.3 on pci0
pcib4: Failed to allocate interrupt for PCI-e events
uhci0: <Intel 82801I (ICH9) USB controller> port 0xa0a0-0xa0bf irq 23 at device 7.0 on pci0
usbus0 on uhci0
uhci1: <Intel 82801I (ICH9) USB controller> port 0xa080-0xa09f irq 20 at device 7.1 on pci0
usbus1 on uhci1
uhci2: <Intel 82801I (ICH9) USB controller> port 0xa060-0xa07f irq 21 at device 7.2 on pci0
usbus2 on uhci2
ehci0: <Intel 82801I (ICH9) USB 2.0 controller> mem 0x98803000-0x98803fff irq 22 at device 7.7 on pci0
ehci0: Could not allocate irq
device_attach: ehci0 attach returned 6
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel ICH9 AHCI SATA controller> port 0xa040-0xa05f mem 0x98802000-0x98802fff irq 16 at device 31.2 on pci0
ahci0: AHCI v1.00 with 6 1.5Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
acpi_syscontainer0: <System Container> on acpi0
acpi_syscontainer1: <System Container> port 0xcd8-0xce3 on acpi0
acpi_syscontainer2: <System Container> port 0x620-0x62f on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (115200,n,8,1)
attimer0: <AT timer> at port 0x40 on isa0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
ppc0: cannot reserve I/O port range
Timecounters tick every 10.000 msec
usbus1: 12Mbps Full Speed USB v1.0
usbus0: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
pci1: <ACPI PCI bus> on pcib1
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x9000-0x903f mem 0x98600000-0x9861ffff irq 22 at device 0.0 on pci1
em0: Unable to allocate bus resource: interrupt
device_attach: em0 attach returned 6
ugen2.1: <Intel UHCI root HUB> at usbus2
ugen1.1: <Intel UHCI root HUB> at usbus1
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen0.1: <Intel UHCI root HUB> at usbus0
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ada0 at ahcich2 bus 0 scbus2 target 0 lun 0
ada0: <QEMU HARDDISK 2.5+> ATA-7 SATA device
ada0: Serial Number QM00005
ada0: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 25600MB (52428800 512 byte sectors)
cd0 at ahcich0 bus 0 scbus0 target 0 lun 0
Trying to mount root from ufs:/dev/ada0p2 [rw]...
cd0: <QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device
cd0: Serial Number QM00001
cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes)
cd0: 3490MB (1786895 2048 byte sectors)
random: unblocking device.
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
lo0: link state changed to UP
ugen0.2: <QEMU QEMU USB Tablet> at usbus0
uhid0 on uhub2
uhid0: <QEMU QEMU USB Tablet, class 0/0, rev 2.00/0.00, addr 2> on usbus0

#################################################
11.3 +netmap patch, QEMU 3.1, OVMF, SATA, E1000
+++++++++++++++++++++++++++++++++++++++++++++++++
ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
	ether 52:54:00:17:32:d0
	hwaddr 52:54:00:17:32:d0
	inet 192.168.0.20 netmask 0xffffff00 broadcast 192.168.0.255
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
	inet 127.0.0.1 netmask 0xff000000
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo
+++++++++++++++++++++++++++++++++++++++++++++++++
pciconf -lv

hostb0@pci0:0:0:0:	class=0x060000 card=0x11001af4 chip=0x29c08086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82G33/G31/P35/P31 Express DRAM Controller'
    class      = bridge
    subclass   = HOST-PCI
vgapci0@pci0:0:1:0:	class=0x030000 card=0x11001af4 chip=0x01001b36 rev=0x04 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'QXL paravirtual graphic card'
    class      = display
    subclass   = VGA
pcib1@pci0:0:2:0:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
pcib2@pci0:0:2:1:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
pcib3@pci0:0:2:2:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
pcib4@pci0:0:2:3:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
uhci0@pci0:0:7:0:	class=0x0c0300 card=0x11001af4 chip=0x29348086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
uhci1@pci0:0:7:1:	class=0x0c0300 card=0x11001af4 chip=0x29358086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
uhci2@pci0:0:7:2:	class=0x0c0300 card=0x11001af4 chip=0x29368086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
ehci0@pci0:0:7:7:	class=0x0c0320 card=0x11001af4 chip=0x293a8086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB2 EHCI Controller'
    class      = serial bus
    subclass   = USB
isab0@pci0:0:31:0:	class=0x060100 card=0x11001af4 chip=0x29188086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801IB (ICH9) LPC Interface Controller'
    class      = bridge
    subclass   = PCI-ISA
ahci0@pci0:0:31:2:	class=0x010601 card=0x11001af4 chip=0x29228086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]'
    class      = mass storage
    subclass   = SATA
none0@pci0:0:31:3:	class=0x0c0500 card=0x11001af4 chip=0x29308086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) SMBus Controller'
    class      = serial bus
    subclass   = SMBus
em0@pci0:1:0:0:	class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82540EM Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet
none1@pci0:2:0:0:	class=0x078000 card=0x11001af4 chip=0x10431af4 rev=0x01 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'Virtio console'
    class      = simple comms
none2@pci0:3:0:0:	class=0x00ff00 card=0x11001af4 chip=0x10451af4 rev=0x01 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'Virtio memory balloon'
    class      = old
+++++++++++++++++++++++++++++++++++++++++++++++++
pciconf -lv

Copyright (c) 1992-2019 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.3-RELEASE #1: Thu Jan 16 16:19:11 PST 2020
    root@cylon20:/usr/obj/usr/src/sys/MYKERNEL amd64
FreeBSD clang version 8.0.0 (tags/RELEASE_800/final 356365) (based on LLVM 8.0.0)
VT(efifb): resolution 800x600
CPU: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz (3192.06-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x906ea  Family=0x6  Model=0x9e  Stepping=10
  Features=0x1f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT>
  Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x1c4fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP>
  Structured Extended Features2=0x4<UMIP>
  Structured Extended Features3=0x8c000400<MD_CLEAR,IBPB,STIBP,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
real memory  = 4294967296 (4096 MB)
avail memory = 4063502336 (3875 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <BOCHS  BXPCAPIC>
WARNING: L1 data cache covers less APIC IDs than a core
0 < 1
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 hardware threads
ioapic0 <Version 1.1> irqs 0-23 on motherboard
SMP: AP CPU #1 Launched!
random: entropy device external interface
kbd1 at kbdmux0
000.000030 [4213] netmap_init               netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff81004eb0, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
nexus0
cryptosoft0: <software crypto> on motherboard
acpi0: <BOCHS BXPCFACP> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x608-0x60b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xa0c0-0xa0df mem 0x94000000-0x97ffffff,0x90000000-0x93ffffff,0x98800000-0x98801fff irq 21 at device 1.0 on pci0
vgapci0: Boot video device
pcib1: <ACPI PCI-PCI bridge> mem 0x98807000-0x98807fff irq 22 at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x9000-0x903f mem 0x98600000-0x9861ffff irq 22 at device 0.0 on pci1
em0: Ethernet address: 52:54:00:17:32:d0
em0: netmap queues/slots: TX 1/256, RX 1/256
pcib2: <PCI-PCI bridge> mem 0x98806000-0x98806fff irq 22 at device 2.1 on pci0
pci2: <PCI bus> on pcib2
pci2: <simple comms> at device 0.0 (no driver attached)
pcib3: <PCI-PCI bridge> mem 0x98805000-0x98805fff irq 22 at device 2.2 on pci0
pci3: <PCI bus> on pcib3
pci3: <old> at device 0.0 (no driver attached)
pcib4: <PCI-PCI bridge> mem 0x98804000-0x98804fff irq 22 at device 2.3 on pci0
pci4: <PCI bus> on pcib4
uhci0: <Intel 82801I (ICH9) USB controller> port 0xa0a0-0xa0bf irq 23 at device 7.0 on pci0
usbus0 on uhci0
usbus0: 12Mbps Full Speed USB v1.0
uhci1: <Intel 82801I (ICH9) USB controller> port 0xa080-0xa09f irq 20 at device 7.1 on pci0
usbus1 on uhci1
usbus1: 12Mbps Full Speed USB v1.0
uhci2: <Intel 82801I (ICH9) USB controller> port 0xa060-0xa07f irq 21 at device 7.2 on pci0
usbus2 on uhci2
usbus2: 12Mbps Full Speed USB v1.0
ehci0: <Intel 82801I (ICH9) USB 2.0 controller> mem 0x98803000-0x98803fff irq 22 at device 7.7 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci0
usbus3: 480Mbps High Speed USB v2.0
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel ICH9 AHCI SATA controller> port 0xa040-0xa05f mem 0x98802000-0x98802fff irq 16 at device 31.2 on pci0
ahci0: AHCI v1.00 with 6 1.5Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
acpi_syscontainer0: <System Container> on acpi0
acpi_syscontainer1: <System Container> port 0xcd8-0xce3 on acpi0
acpi_syscontainer2: <System Container> port 0x620-0x62f on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (115200,n,8,1)
attimer0: <AT timer> at port 0x40 on isa0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
ppc0: cannot reserve I/O port range
Timecounters tick every 10.000 msec
ugen3.1: <Intel EHCI root HUB> at usbus3
ugen2.1: <Intel UHCI root HUB> at usbus2
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen0.1: <Intel UHCI root HUB> at usbus0
ugen1.1: <Intel UHCI root HUB> at usbus1
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ada0 at ahcich2 bus 0 scbus2 target 0 lun 0
ada0: <QEMU HARDDISK 2.5+> ATA-7 SATA device
ada0: Serial Number QM00005
ada0: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 25600MB (52428800 512 byte sectors)
cd0 at ahcich0 bus 0 scbus0 target 0 lun 0
cd0: <QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device
cd0: Serial Number QM00001
cd0: 150.000MB/s transfers (SATA 1.x, Trying to mount root from ufs:/dev/ada0p2 [rw]...
UDMA5, ATAPI 12bytes, PIO 8192bytes)
cd0: 3490MB (1786895 2048 byte sectors)
random: unblocking device.
uhub1: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
lo0: link state changed to UP
uhub0: 6 ports with 6 removable, self powered
em0: link state changed to UP
ugen3.2: <QEMU QEMU USB Tablet> at usbus3
uhid0 on uhub0
uhid0: <QEMU QEMU USB Tablet, class 0/0, rev 2.00/0.00, addr 2> on usbus3
em0: link state changed to DOWN
em0: link state changed to UP
#################################################
Comment 70 Vincenzo Maffione freebsd_committer 2020-01-17 19:03:25 UTC
(In reply to MattS from comment #69)
I see. This looks like a different issue. The PCI-e code is failing to allocate the interrupt resources needed by ehci0 and em0, so the drivers cannot attach.
It does not look like netmap is the cause here... but it may be worth trying to see what happens without "dev netmap"
Comment 71 MattS 2020-01-17 21:13:34 UTC
(In reply to Vincenzo Maffione from comment #70)

You were correct, removed netmap, no change. I checked 11.2 and it also seems to have the issue.

Should a new bug report be created? I'm not sure it's actually necessary at this point since we have an upgrade path available now.
Comment 72 Vincenzo Maffione freebsd_committer 2020-01-19 08:12:20 UTC
(In reply to MattS from comment #71)
Do you mean that upgrading to 12.x fixes this latter problem?
In that case I guess it's up to you to decide if you wish to create another issue.
For sure this one become too crowded, discouraging any developer to take a look..
Comment 73 MattS 2020-01-19 18:43:21 UTC
(In reply to Vincenzo Maffione from comment #72)

I just tested clean 12.1 with your netmap patch from comment #36. I am seeing similar results to 11.X. 

QEMU 3.1, OVMF, SATA, e1000 works. 

Switching to QEMU 4.x causes panic on boot. I do see the em0 detected before panic.

Re: my comment on upgrade path, I was just referring to options to upgrade from 11.2.
Comment 74 Tommy P 2020-01-20 15:09:27 UTC
John and Matt,

I think I found the root cause for NICs not working in QEMU 4.x.  According to the QEMU's change log:
-------------------------------------------------------------------------
PCI/PCIe

    Generic PCIe root port link speed and width enhancements: Starting with the Q35 QEMU 4.0 machine type, generic pcie-root-port will default to the maximum PCIe link speed (16GT/s) and width (x32) provided by the PCIe 4.0 specification. Experimental options x-speed= and x-width= are provided for custom tuning, but it is expected that the default over-provisioning of bandwidth is optimal for the vast majority of use cases. Previous machine versions and ioh3420 root ports will continue to default to 2.5GT/x1 links.
-------------------------------------------------------------------------
and your VM's config having machine type 'pc-q35-4.0', I _doubt_ that FreeBSD RELENG/STABLE branches supports PCI-e gen 4 specs.  What happens when you edit the XML and change the machine type to 'pc-q35-2.11'?  You'll may need to recycle the libvirtd, remove and then re-add the devices and controllers.

Regards,
Tommy
Comment 75 Tommy P 2020-01-20 21:46:37 UTC
John & Matt,

QEMU 4.0 features got me interested and I've decided to dig deeper into your report em NIC not working.  Since I don't have a spare bare metal to setup QEMU 4.0 environment, I did nested virtualization.

Main host is Ubuntu 18.04.3 on bare metal
Nested host is Ubuntu 19.10 (QEMU 4.0) as main host's VM
Nested guests is in QEMU 4.0

I first created a new VM within the QEMU 4.0 and the em didn't work as you've reported.  I then imported a config from my main host (QEMU 2.11) started it within the nested.  It works as expected.  When I compare the differences, here is what I found that would possibly cause the NIC to failed:

Imported config from main:
-name guest=z_fbsd12_ovmf,debug-threads=on
-machine pc-q35-2.11,accel=kvm,usb=off,vmport=off,dump-guest-core=off
-device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e
-device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0
-device e1000,netdev=hostnet0,id=net0,mac=52:54:00:4e:d4:8f,bus=pci.2,addr=0x1

Created new config in nested:
-name guest=freebsd12.0,debug-threads=on
-machine pc-q35-2.11,accel=kvm,usb=off,vmport=off,dump-guest-core=off
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2
-device pcie-pci-bridge,id=pci.4,bus=pci.1,addr=0x0
-device e1000e,netdev=hostnet0,id=net0,mac=52:54:00:87:83:4a,bus=pci.3,addr=0x0

As you can see from the command line startup, the imported config have the em0 on the PCI bus while the created new config is on the PCIe.  Thus, my previously assessment (comment #74) was partially accurate.  I'm sure that the bug is that the current em NIC driver does not support PCIe.  If you manage to apply my proposed VirtIO patches, you should have working VirtIO (including the VirtIO NIC).  I suggest that you file a separate bug report regarding the em NIC driver not supporting PCIe.

Regards,
Tommy
Comment 76 MattS 2020-01-21 05:28:58 UTC
(In reply to Tommy P from comment #75)

TommyP, 

Thanks for looking into this a bit. Since qemu 3.1 works, was there some reason you used qemu 2.11 in your test? 

Below is from my qemu 3.1 test vm
-device e1000,netdev=hostnet0,id=net0,mac=52:54:00:d0:8c:7a,bus=pci.1,addr=0x0 \
Comment 77 Tommy P 2020-01-21 07:04:25 UTC
(In reply to MattS from comment #76)
Hi Matt,

I used 2.11 because I have working 2.11 (Ubuntu 18.04.3) and not 3.1.  It'll be easier for me to compare and investigate why pretty much every PCIe devices and controllers break in QEMU 4.x.  It's also easier for me to verify that nested virtualization should not have any effect, other than the obvious performance degradation, as confirmed when I imported the config from main host (18.04.3) running on bare metal into virtualized host (19.10).  The all devices and controllers in the nested guest of the imported config are functional as if not nested.

Would you mind attaching a working 3.1 XML?  I'd like to narrow down the VM configurations which breaks FreeBSD.

---------------------------------------------------

@Everyone,

Since my last posting, I've been digging deeper into the issue of PCIe and QEMU 4.x.  I think that both of my assessment (comment #74 and comment #75) were correct.  Since someone else already reported QEMU 4.0 issue even though I think his/her initial assumption of netmap interference is incorrect:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241581

I'll post my findings there.  In short, I'm pretty sure that the root cause is FreeBSD's ACPI is unable to retrieve the interrupt routing table for all the PCIe devices and controllers.  Thus, those devices and controllers are unable to get the IRQ ID (Failed to allocate interrupt for PCI-e events) and would not work.

Regards,
Tommy
Comment 78 John Hartley 2020-01-21 10:27:47 UTC
(In reply to Tommy P from comment #77)

Hi Tommy P,

here are some more rest results for you:

QEMU with OVMF, Boot from patched VirtIO SCSI with patched netmap Networking.

Results:

Q35 V2.12 - Boots and both em & vmx network devices found:

cat qemu-xml-v211-01.xml 
<domain type='kvm' id='127'>
  <name>test-freebsd-12.0</name>
  <uuid>XXX</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://freebsd.org/freebsd/12.0"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-2.12'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
    <nvram>/home/XXX/DIR/test.freebsd/OVMF_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state='off'/>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Broadwell-IBRS</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='f16c'/>
    <feature policy='require' name='rdrand'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='arat'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='umip'/>
    <feature policy='require' name='md-clear'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='arch-capabilities'/>
    <feature policy='require' name='ssbd'/>
    <feature policy='require' name='xsaveopt'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='abm'/>
    <feature policy='disable' name='skip-l1dfl-vmentry'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu'/>
      <target dev='sda' bus='sata'/>
      <readonly/>
      <boot order='1'/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/XXX/DIR/test.freebsd/test-hd1-01.qcow2'/>
      <backingStore/>
      <target dev='sdb' bus='scsi'/>
      <boot order='2'/>
      <alias name='scsi0-0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x12'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x13'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x14'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x15'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:cb:db:07'/>
      <source bridge='br90'/>
      <target dev='vnet34'/>
      <model type='e1000'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x02' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:a5:4c:f3'/>
      <source bridge='br90'/>
      <target dev='vnet35'/>
      <model type='vmxnet3'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/31'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/31'>
      <source path='/dev/pts/31'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='keyboard' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='spice' port='5934' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
      <image compression='off'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir0'/>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir1'/>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='apparmor' relabel='yes'>
    <label>libvirt-4b1c030f-dfe8-4bd1-9288-be1ced293543</label>
    <imagelabel>libvirt-4b1c030f-dfe8-4bd1-9288-be1ced293543</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+64055:+106</label>
    <imagelabel>+64055:+106</imagelabel>
  </seclabel>
</domain>

Q35 V3.5 - Boots and em & vmx network devices found

cat qemu-xml-v35-01.xml 
<domain type='kvm' id='125'>
  <name>test-freebsd-12.0</name>
  <uuid>XXXX</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://freebsd.org/freebsd/12.0"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-3.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
    <nvram>/home/XXX/DIR/test.freebsd/OVMF_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state='off'/>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Broadwell-IBRS</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='f16c'/>
    <feature policy='require' name='rdrand'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='arat'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='umip'/>
    <feature policy='require' name='md-clear'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='arch-capabilities'/>
    <feature policy='require' name='ssbd'/>
    <feature policy='require' name='xsaveopt'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='abm'/>
    <feature policy='disable' name='skip-l1dfl-vmentry'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu'/>
      <target dev='sda' bus='sata'/>
      <readonly/>
      <boot order='1'/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/XXX/DIR/test.freebsd/test-hd1-01.qcow2'/>
      <backingStore/>
      <target dev='sdb' bus='scsi'/>
      <boot order='2'/>
      <alias name='scsi0-0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x12'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x13'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x14'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x15'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:cb:db:07'/>
      <source bridge='br90'/>
      <target dev='vnet34'/>
      <model type='e1000'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x02' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:a5:4c:f3'/>
      <source bridge='br90'/>
      <target dev='vnet35'/>
      <model type='vmxnet3'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/31'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/31'>
      <source path='/dev/pts/31'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='keyboard' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='spice' port='5934' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
      <image compression='off'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir0'/>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir1'/>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='apparmor' relabel='yes'>
    <label>libvirt-4b1c030f-dfe8-4bd1-9288-be1ced293543</label>
    <imagelabel>libvirt-4b1c030f-dfe8-4bd1-9288-be1ced293543</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+64055:+106</label>
    <imagelabel>+64055:+106</imagelabel>
  </seclabel>
</domain>

Q35 V4.0 - Boots to mountroot prompt but then unable to find disk (ie looks like UEFI starting is working but then FreeBSD disk driver fails), both emo & vmx network devices look to have been found

cat qemu-xml-v40-01.xml 
<domain type='kvm' id='126'>
  <name>test-freebsd-12.0</name>
  <uuid>4b1c030f-dfe8-4bd1-9288-be1ced293543</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://freebsd.org/freebsd/12.0"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-4.0'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
    <nvram>/home/XXX/DIR/test.freebsd/OVMF_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state='off'/>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Broadwell-IBRS</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='f16c'/>
    <feature policy='require' name='rdrand'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='arat'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='umip'/>
    <feature policy='require' name='md-clear'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='arch-capabilities'/>
    <feature policy='require' name='ssbd'/>
    <feature policy='require' name='xsaveopt'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='abm'/>
    <feature policy='disable' name='skip-l1dfl-vmentry'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu'/>
      <target dev='sda' bus='sata'/>
      <readonly/>
      <boot order='1'/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/XXX/DIR/test.freebsd/test-hd1-01.qcow2'/>
      <backingStore/>
      <target dev='sdb' bus='scsi'/>
      <boot order='2'/>
      <alias name='scsi0-0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x12'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x13'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x14'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x15'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:cb:db:07'/>
      <source bridge='br90'/>
      <target dev='vnet34'/>
      <model type='e1000'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x02' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:a5:4c:f3'/>
      <source bridge='br90'/>
      <target dev='vnet35'/>
      <model type='vmxnet3'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/31'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/31'>
      <source path='/dev/pts/31'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='keyboard' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='spice' port='5934' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
      <image compression='off'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir0'/>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir1'/>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='apparmor' relabel='yes'>
    <label>libvirt-4b1c030f-dfe8-4bd1-9288-be1ced293543</label>
    <imagelabel>libvirt-4b1c030f-dfe8-4bd1-9288-be1ced293543</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+64055:+106</label>
    <imagelabel>+64055:+106</imagelabel>
  </seclabel>
</domain>

NOTE: I have used Q35 V2.12 as this this that I am using for a number of MacOS machines.

Thanks for continuing to looking these problems.

Cheers,


John Hartley.
Comment 79 MattS 2020-01-21 17:57:06 UTC
(In reply to Tommy P from comment #77)

Here is my 3.1 and 4.2 xml

#################################################
QEMU 3.1

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>cylon21</name>
  <uuid>c5cd0551-fe92-5842-d520-f089d2593f61</uuid>
  <description>FreeBSD 12.1</description>
  <metadata>
    <vmtemplate xmlns="unraid" name="FreeBSD" icon="freebsd.png" os="freebsd"/>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>2</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='1'/>
    <vcpupin vcpu='1' cpuset='7'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-3.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/c5cd0551-fe92-5842-d520-f089d2593f61_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='1' threads='2'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/FreeBSD-12.1-RELEASE-amd64-dvd1.iso'/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/cylon21/vdisk1.img'/>
      <target dev='hdc' bus='sata'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:d0:8c:7a'/>
      <source bridge='br0'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
</domain>
#################################################
QEMU 4.2

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>cylon21</name>
  <uuid>c5cd0551-fe92-5842-d520-f089d2593f61</uuid>
  <description>FreeBSD 12.1</description>
  <metadata>
    <vmtemplate xmlns="unraid" name="FreeBSD" icon="freebsd.png" os="freebsd"/>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>2</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='1'/>
    <vcpupin vcpu='1' cpuset='7'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/c5cd0551-fe92-5842-d520-f089d2593f61_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='1' threads='2'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/FreeBSD-12.1-RELEASE-amd64-dvd1.iso'/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/cylon21/vdisk1.img'/>
      <target dev='hdc' bus='sata'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:d0:8c:7a'/>
      <source bridge='br0'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
</domain>

#################################################
Comment 80 Tommy P 2020-01-21 23:32:55 UTC
(In reply to MattS from comment #79)
Thanks, Matt.  Were configurations imported of an older config or did you used the virt-manager to create?  Do you use shell scripts to maintain the VMs or a GUI?  I'm asking because they're missing:

pcie-to-pci-bridge or pci-bridge
Comment 81 MattS 2020-01-22 00:09:10 UTC
(In reply to Tommy P from comment #80)

My host OS is unRAID 6.8.1. Currently on Libvirt 5.10.0. VMs are created and managed via GUI(see comment 38), with the option of switching to XML view and making manual changes. My understanding is they provide a pretty vanilla implementation. If there is a template error, I will certainly report it. At the very least the required XML needs to be documented.
Comment 82 John Hartley 2020-01-22 02:17:28 UTC
(In reply to Tommy P from comment #75)

Hi Tommy P,

I did the following tests:

QEMU Q35 V4.0 / OVMF with VirtIO SCSI & NIC == e1000 + vmxnet + netmap patched FreeBSD:

SCSI - VirtIO SCS disk not found
NIC - em0 & vmx0 found

QEMU Q35 V4.0 / OVMF with SATA & NIC == e1000 + vmxnet + netmap patched FreeBSD:

SATA - disk found
NIC - em0 & vmx0 found

QEMU Q35 V4.0 OVMF with SATA & NIC == e1000e + vmxnet + netmap patched FreeBSD:

SATA - disk found
NIC - em - not found, vmx0 found

So this appears to confirm that there is general issues with Q35 V4.0 PCIe devices (VirtIO PCIe and e100e).

I believe this is new bug and should be track in new thread, as it does not relate to the PCI ID conflict that Vincenzo identified, as I already have the fix and still get issue.

I will also add this testing info to the other bug report you have found: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241581


Cheers,

John Hartley.
Comment 83 John Hartley 2020-07-01 13:01:50 UTC
Hi Vincenzo,

I have retested with 11.4 and this bug is not fixed in 11.4:

Tested with Q35 v3.1 with OVMF:

VM with: e1000e, e1000, vmxnet3 & rtl8139 all network devices correctly detected:

<< Start of Test >>
#ifconfig -a
em0: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
	ether 52:54:00:3d:41:fa
	hwaddr 52:54:00:3d:41:fa
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
	ether 52:54:00:dc:37:ab
	hwaddr 52:54:00:dc:37:ab
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 52:54:00:a1:30:02
	hwaddr 52:54:00:a1:30:02
	inet 192.168.73.136 netmask 0xffffff00 broadcast 192.168.73.255
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
re0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
	ether 52:54:00:7e:47:f6
	hwaddr 52:54:00:7e:47:f6
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect (100baseTX <full-duplex,flowcontrol,rxpause,txpause>)
	status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
	inet 127.0.0.1 netmask 0xff000000
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo

<< End of Test >>


Thank you.

Cheers,


John Hartley/
Comment 84 John Hartley 2020-07-01 13:03:58 UTC
(In reply to John Hartley from comment #83)

Apologies should be "now fixed in 11.4"

So bug can be closed.
Comment 85 MattS 2020-07-05 02:12:29 UTC
(In reply to John Hartley from comment #84)

My upgrade was successful as well. Thanks again to everyone!
Comment 86 Jeroen 2020-07-15 13:43:51 UTC
Please correct me if I'm not reporting at the right bug. I noticed that since the original issue started, it branched off to different bugs.

I've just tried to install a new 11.4 machine on a KVM host that we're managing with oVirt.

My experience is that I can now get 11.4 to work, but not out of the box:

- disk only works with SATA, not with VirtIO or VirtIO-SCSI
- NICs are only detected when "custom emulated machine" is set to "pc-q35-rhel8.0.0" or lower.
- Even then, NICs only work with e1000 or rtl8139, not with VirtIO
Comment 87 John Hartley 2020-07-15 14:52:14 UTC
(In reply to Jeroen from comment #86)
Hi Jeroen,

the virtio bug fix has not flowed into 11.4

See: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236922

Until patch flows through you only have SATA.

There is another bug with Q35 4.x, see: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=243640

If you set Q35 version to 3.1 or less you should get: e1000, e1000e, vmxnet3 & rtl8139 working.

Cheers,

John.