Bug 236922 - Virtio fails as QEMU-KVM guest with Q35 chipset on Ubuntu 18.04.2 LTS
Summary: Virtio fails as QEMU-KVM guest with Q35 chipset on Ubuntu 18.04.2 LTS
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.0-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-virtualization (Nobody)
URL:
Keywords: needs-qa
Depends on:
Blocks:
 
Reported: 2019-04-01 01:53 UTC by Tommy P
Modified: 2020-10-28 12:31 UTC (History)
12 users (show)

See Also:


Attachments
various data collection for bug report (168.66 KB, application/x-zip-compressed)
2019-04-01 01:53 UTC, Tommy P
no flags Details
/var/log/messages file as requested with verbose logging during boot (18.05 KB, application/x-zip-compressed)
2019-04-01 04:59 UTC, Tommy P
no flags Details
dmesg output (20.93 KB, application/x-xz)
2019-08-25 15:18 UTC, htwfbsd
htwfbsd: maintainer-approval?
Details
VirtIO /usr/src files to support Q35 (75.91 KB, application/x-xz)
2020-01-09 02:52 UTC, Tommy P
no flags Details
VirtIO + netmap to support Q35 (3.26 KB, patch)
2020-01-09 05:20 UTC, Tommy P
no flags Details | Diff
Errors (895.95 KB, image/jpeg)
2020-01-12 16:10 UTC, Dobri Dobrev
no flags Details
VirtIO + netmap to support Q35 (11.93 KB, patch)
2020-01-13 17:29 UTC, Tommy P
no flags Details | Diff
Recent update in stable/12 for VirtIO (1.81 KB, patch)
2020-01-13 23:26 UTC, Tommy P
no flags Details | Diff
VirtIO to support Q35 backported for FreeBSD 11.x (79.25 KB, application/x-xz)
2020-01-14 06:31 UTC, Tommy P
no flags Details
VirtIO to support Q35 backported for FreeBSD 11.x (79.82 KB, application/x-xz)
2020-01-14 15:25 UTC, Tommy P
no flags Details
VirtIO to support Q35 for FreeBSD 12.x (80.54 KB, application/x-xz)
2020-01-14 16:57 UTC, Tommy P
no flags Details
12.x Disable Virtio + netmap interop (4.28 KB, patch)
2020-01-16 00:51 UTC, Tommy P
no flags Details | Diff
Consolidated Differences across VirtIO / netmap (97.31 KB, patch)
2020-01-17 07:42 UTC, John Hartley
no flags Details | Diff
VirtIO to support Q35 for FreeBSD 13-CURRENT (78.34 KB, application/x-xz)
2020-02-01 18:24 UTC, Tommy P
no flags Details
Proper FreeBSD 12.x patch for VirtIO (297.66 KB, patch)
2020-03-20 20:36 UTC, Tommy P
no flags Details | Diff
Proper FreeBSD 11.x patch for VirtIO (311.21 KB, patch)
2020-03-20 20:37 UTC, Tommy P
no flags Details | Diff
Proper FreeBSD 12.x patch for VirtIO (297.61 KB, patch)
2020-03-20 20:45 UTC, Tommy P
no flags Details | Diff
Proper FreeBSD 11.x patch for VirtIO (311.21 KB, patch)
2020-03-20 20:45 UTC, Tommy P
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Tommy P 2019-04-01 01:53:16 UTC
Created attachment 203279 [details]
various data collection for bug report

Environment:

Host OS:  Ubuntu 18.04.2 LTS (updated)
Libvirt:  (libvirt-bin) 4.0.0-1ubuntu8.8
QEMU:     1:2.11+dfsg-1ubuntu7.12
QEMU-KVM: 1:2.11+dfsg-1ubuntu7.12

Guest OS: FreeBSD 12.0-RELEASE

If the VM configuration uses i440FX chipset, everything works as expected.  When configured to use Q35 chipset, virtio failed to load properly and any device configured to use virtio would not be detected.

Attachments within zip of data gathered from the same VM:

QEMU-KVM_freebsd_virtio_1.png   - screen shot of the VM configuration as seen in virt-manager
z_fbsd_q35.xml                  - Libvirt/QEMU XML configuration for the same VM seen in the screen shot
pciconf_12.0_r341666_ori.txt    - output of 'pciconf -lvce' of the original kernel r341666 used during install
pciconf_12.0_r345757_custom.txt - output of 'pciconf -lvce' of the kernel r345757 built from updated src
sysctl.hw_r345757_custom.txt    - output of 'sysctl hw' of the kernel r345757 built from updated src
Custom12                        - kernel configuration (copy of GENERIC with unneeded drivers and configs removed)
make.conf                       - make.conf used in building kernel r345757
src.conf                        - src.conf used in building kernel r345757.  Buildworld was not done.  Only buildkernel and installkernel.

In the screen shot QEMU-KVM_freebsd_virtio_1.png, the "Controller SCSI 1" is added as SCSI controller having "Hypervisor default" model.  FreeBSD detected and loaded the 'sym' SCSI driver. While the "Controller Virtio SCSI 0" is added as SCSI controller having "VirtIO SCSI" model.  The custom kernel show the VirtIO PCI controller in pciconf but any virtio HDD attached does not show up in /dev.


How to repeat:

1) Install Ubuntu 18.04.2 desktop to support virtualization using libvirt, QEMU-KVM, and virt-manager.
2) Use virt-manager to create/import a VM
2a) Select x86_64 architecture
2b) Make sure to check "Customize configuration before install"
3a) In the popup customization window, change the chipset from i440FX to Q35 in the overview.
3b) Select the HDD and change the "Disk bus" in "Advanced options" to VirtIO
3c) Select the NIC and change the "Device model" to virtio
4) Boot from imported HDD or from ISO media to install

Utilizing the Q35 chipset, FreeBSD would neither boot properly nor install when both HDD and NIC are virtio.  If the HDD is SATA type and NIC is other virtio, everything works.
Comment 1 Tommy P 2019-04-01 01:56:54 UTC
PS: I forgot to mention that I did a diff of src/sys/dev/virtio of the original src during install (presumably r341666) against the updated src (r345757) via svnlite checkout of 12.0.  diff reported no differences.
Comment 2 Kubilay Kocak freebsd_committer freebsd_triage 2019-04-01 03:28:25 UTC
Can you try later versions of qemu (in particular 4.x.x-RC* series) to see if the issue is reproducible there.

I am going to hazard a guess that the best first place to diagnose/isolate this issue is upstream with QEMU

A complete verbose boot log might also prove handy.

See Also: https://forums.unraid.net/topic/70123-pc-q35-211-issues/ - with report (and xml) of a working 2.11 configuration on unraid/pfSense (FreeBSD)
Comment 3 Tommy P 2019-04-01 04:59:01 UTC
Created attachment 203281 [details]
/var/log/messages file as requested with verbose logging during boot
Comment 4 Tommy P 2019-04-01 05:05:25 UTC
(In reply to Kubilay Kocak from comment #2)

Thank you for the feedback.  I'm hesitant to try QEMU 4.0 since it's not officially supported on Ubuntu (even the current 18.10):

https://packages.ubuntu.com/search?keywords=qemu

Aside from that, it still doesn't explain why the original kernel (r341666) from the install ISO did not detect the VirtIO PCI controllers while the custom built updated from src (r345757) did detect the VirtIO PCI controllers when the diff comparison of src/sys/dev/virtio doesn't show any changes.  If anything, I'd expect the custom built kernel not to detect any VirtIO PCI controllers like the original kernel of the install.  Since the custom kernel did in fact the detect the VirtIO PCI controllers, it lead me to believe that the drivers are not working correctly.  To confirm that the virtio drivers are loaded, I got this:

root@fbsd12:~ # kldload virtio
kldload: can't load virtio: module already loaded or in kernel

yet:

root@fbsd12:~ # kldstat
Id Refs Address                Size Name
 1   10 0xffffffff80200000  1099a70 kernel
 2    1 0xffffffff8129b000   372508 zfs.ko
 3    2 0xffffffff8160e000     a2e8 opensolaris.ko

when 'pciconf -lvce' shows the VirtIO PCI controllers.

I've attached the /var/log/messages file with verbose logging on boot as requested.  As side note, I was able to install just about every guest OS (Fedora, Ubuntu, and Windows 2008 R2 to Windows 2016) successfully utilizing the Q35 chipset and VirtIO HDD+NIC
Comment 5 Tommy P 2019-04-01 06:54:44 UTC
I think I may found the bug...  Please correct me if I'm wrong since it's been a very long time since I've dealt with C/C++ :(

File src/sys/dev/virtio/pci/virtio_pci.h  has this:

/* VirtIO PCI vendor/device ID. */
#define VIRTIO_PCI_VENDORID     0x1AF4
#define VIRTIO_PCI_DEVICEID_MIN 0x1000
#define VIRTIO_PCI_DEVICEID_MAX 0x103F

When the detected device ID is as follow for 12.0-RELEASE when booted with Q35 chipset:

VirtIO SCSI:            0x1048
VirtIO console:         0x1043
VirtIO memory balloon:  0x1045
VirtIO network device:  0x1041
VirtIO block device:    0x1042

All those IDs are beyond the defined range.  Thus, the controllers are detected since they match the vendor ID but the drivers are not loaded.  As verification, I've modified the source code to:

#define VIRTIO_PCI_DEVICEID_MAX 0x104F

and rebuilt the kernel. The drivers still didn't load.  Here's the pciconf for the SCSI when boot with i440FX chipset:

virtio_pci2@pci0:0:7:0: class=0x010000 card=0x00081af4 chip=0x10041af4 rev=0x00 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'Virtio SCSI'
    class      = mass storage
    subclass   = SCSI
    cap 11[98] = MSI-X supports 4 messages, enabled
                 Table in map 0x14[0x0], PBA in map 0x14[0x800]
    cap 09[84] = vendor (length 20)
    cap 09[70] = vendor (length 20)
    cap 09[60] = vendor (length 16)
    cap 09[50] = vendor (length 16)
    cap 09[40] = vendor (length 16)

and when boot with Q35 chipset:

none1@pci0:1:0:0:       class=0x010000 card=0x11001af4 chip=0x10481af4 rev=0x01 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'Virtio SCSI'
    class      = mass storage
    subclass   = SCSI
    cap 11[dc] = MSI-X supports 4 messages
                 Table in map 0x14[0x0], PBA in map 0x14[0x800]
    cap 09[c8] = vendor (length 20)
    cap 09[b4] = vendor (length 20)
    cap 09[a4] = vendor (length 16)
    cap 09[94] = vendor (length 16)
    cap 09[84] = vendor (length 16)
    cap 01[7c] = powerspec 3  supports D0 D3  current D0
    cap 10[40] = PCI-Express 2 endpoint max data 128(128)
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)

Aside from the device ID changes, looks like there's a revision (?) change from 0x00 to 0x01.  Both match the Windows' SCSI INF driver definition:

%RHELScsi.DeviceDesc% = rhelscsi_inst, PCI\VEN_1AF4&DEV_1004&SUBSYS_00081AF4&REV_00
%RHELScsi.DeviceDesc% = rhelscsi_inst, PCI\VEN_1AF4&DEV_1048&SUBSYS_11001AF4&REV_01

from the virtio driver 0.1.126.
Comment 6 Tommy P 2019-04-01 08:02:23 UTC
Upon further investigation, I did a quick and dirty hack of the source code:

pci/virtio_pci.c:247:/* if (pci_get_revid(dev) != VIRTIO_PCI_ABI_VERSION) */
pci/virtio_pci.c:248:   if (pci_get_revid(dev) != VIRTIO_PCI_ABI_VERSION_MIN &&
pci/virtio_pci.c:249:       pci_get_revid(dev) != VIRTIO_PCI_ABI_VERSION_MAX)
pci/virtio_pci.h:47:/* #define VIRTIO_PCI_ABI_VERSION   0 */
pci/virtio_pci.h:48:#define VIRTIO_PCI_ABI_VERSION_MIN  0
pci/virtio_pci.h:49:#define VIRTIO_PCI_ABI_VERSION_MAX  1

and rebuilt the kernel.  Upon reboot with the modified source codes of virtio_pci.*

the virtio driver still didn't load but I got this in dmesg:

virtio_pci0: <VirtIO PCI Unknown adapter> mem 0xfce00000-0xfce00fff,0xfea00000-0xfea03fff irq 22 at device 0.0 on pci1
virtio_pci0: cannot map I/O space
device_attach: virtio_pci0 attach returned 6
pcib2: <PCI-PCI bridge> mem 0xfd013000-0xfd013fff irq 22 at device 2.1 on pci0
pci2: <PCI bus> on pcib2
virtio_pci0: <VirtIO PCI Unknown adapter> mem 0xfcc00000-0xfcc00fff,0xfe800000-0xfe803fff irq 22 at device 0.0 on pci2
virtio_pci0: cannot map I/O space
device_attach: virtio_pci0 attach returned 6
pcib3: <PCI-PCI bridge> mem 0xfd014000-0xfd014fff irq 22 at device 2.2 on pci0
pci3: <PCI bus> on pcib3
virtio_pci0: <VirtIO PCI Unknown adapter> mem 0xfe600000-0xfe603fff irq 22 at device 0.0 on pci3
virtio_pci0: cannot map I/O space
device_attach: virtio_pci0 attach returned 6
pcib4: <PCI-PCI bridge> mem 0xfd015000-0xfd015fff irq 22 at device 2.3 on pci0
pci4: <PCI bus> on pcib4
virtio_pci0: <VirtIO PCI Unknown adapter> mem 0xfc880000-0xfc880fff,0xfe400000-0xfe403fff irq 22 at device 0.0 on pci4
virtio_pci0: cannot map I/O space
device_attach: virtio_pci0 attach returned 6
pcib5: <PCI-PCI bridge> mem 0xfd016000-0xfd016fff irq 22 at device 2.4 on pci0
pci5: <PCI bus> on pcib5
virtio_pci0: <VirtIO PCI Unknown adapter> mem 0xfc600000-0xfc600fff,0xfe200000-0xfe203fff irq 22 at device 0.0 on pci5
virtio_pci0: cannot map I/O space
device_attach: virtio_pci0 attach returned 6

Looks like a lot more needed than my quick dirty hack and way beyond my knowledge of FreeBSD kernel :(.
Comment 7 Tommy P 2019-04-01 08:32:59 UTC
After reviewing the Linux code for the virtio, I'm more confident that my hunch is correct regarding the FreeBSD virtio driver.



	if (pci_dev->device < 0x1040) {
		/* Transitional devices: use the PCI subsystem device id as
		 * virtio device id, same as legacy driver always did.
		 */
		vp_dev->vdev.id.device = pci_dev->subsystem_device;
	} else {
		/* Modern devices: simply use PCI device id, but start from 0x1040. */
		vp_dev->vdev.id.device = pci_dev->device - 0x1040;
}


from: https://github.com/torvalds/linux/blob/v5.0/drivers/virtio/virtio_pci_modern.c
Comment 8 Bryan Venteicher freebsd_committer 2019-04-04 00:03:52 UTC
The device IDs you are seeing are for the VirtIO V1 spec. FreeBSD only supports the pre-V1 (aka legacy) VirtIO spec. I suppose there was a QEMU change to use V1 with Q35 chipset.

I have some rather dated work that adds V1 support but I hadn't had the time to commit: https://github.com/bryanv/freebsd/tree/virtio/
Comment 9 Tommy P 2019-04-04 03:02:21 UTC
(In reply to Bryan Venteicher from comment #8)

Hi Bryan,

Thank you for the feedback.  I've been digging further to learn more about the FreeBSD's internal while improving my C/C++ skills.  I've found that there are 2 sets of device IDs:  PCI (which FreeBSD currently support) and the PCIe (which I think your work would support).  I'll give them a try a report back if I've found any issue.

While doing a lot of trial and error, I've found something puzzling.  If I build my custom kernel w/o debug, traces, and netmap (in addition to unnecessary drivers), pciconf shows correctly for the PCIe root:

pcib1@pci0:0:2:0:       class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected

However, when added back in the debug, traces, and netmap (still w/o the extra drivers) similar to the GENERIC kernel, that same controller becomes:

none0@pci0:0:2:0:       class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    bar   [10] = type Memory, range 32, base rxfd212000, size 4096, enabled
    bus range  = 1-1
    window[1c] = type I/O Port, range 16, addr 0xe000-0xdfff, disabled
    window[20] = type Memory, range 32, addr 0xfd000000-0xfd1fffff, enabled
    window[24] = type Prefetchable Memory, range 64, addr 0xfea00000-0xfebfffff, enabled
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected

as seen from pciconf.  Note how it's 'pcib1' w/o debug+traces and 'none0' with debug+traces.  I don't know if the bug is with the VirtIO driver or the debug/trace.  Perhaps its by design?  This from src r345840.
Comment 10 Tommy P 2019-04-04 03:31:42 UTC
(In reply to Bryan Venteicher from comment #8)

Hi Bryan,

I just downloaded the from git and tried to compile but receive this error:

--- virtio_pci.o ---
/usr/src12.0/sys/dev/virtio/pci/virtio_pci.c:54:10: fatal error: 'virtio_pci_if.h' file not found
#include "virtio_pci_if.h"

I've check git again and the file is missing.

Thank you for your time.
Comment 11 Bryan Venteicher freebsd_committer 2019-04-04 15:00:07 UTC
virtio_pci_if.h is supposed to be generated as a part of the kernel build
Comment 12 Tommy P 2019-04-06 19:42:31 UTC
(In reply to Bryan Venteicher from comment #11)
Hi Brian,

I've just built the kernel using the entire src tree from Git.  The buildkernel was successful.  Upon reboot, I think the Virtio PCI are detected properly (since the boot message scrolled by too fast) along with SATA and ZFS.  However, the SATA drive having ZFS which FreeBSD is installed does not boot properly since it's unable to find the pool label:

Solaris: NOTICE: Cannot find the pool label for 'fbsd12'
Solaris: NOTICE: Cannot find the pool label for 'fbsd12'
Solaris: NOTICE: Cannot find the pool label for 'fbsd12'
...

even though the kernel detected ada0 device.  Since it won't boot properly, I can't troubleshoot further.  I had to revert back to the original install kernel or the old kernel to boot.  

I've also tried just using the Virtio drivers built from your src with the official kernel but the VirtIO drivers didn't work as before.
Comment 13 Bryan Venteicher freebsd_committer 2019-04-08 22:35:43 UTC
The kernel sources in that repo are rather old so perhaps your SATA issues is related to that.

If you tried the modules built from that repo but with newer FreeBSD then that's probably not going to work. If you copy over sys/dev/virtio and sys/modules/virtio onto a newer FreeBSD tree, you'll probably have some compile errors to sort though but it shouldn't be anything major. Sorry, I have not had the time to rebase that repo yet though.
Comment 14 htwfbsd 2019-08-25 15:16:03 UTC
Sorry to wake this thread but I've also run into similar issues which I thought was a disk adapter handling problem in freeBSD+Q35 because I ran into freeNAS not handling all of the disks, then say it didn't pick up the other disk adapters
Until I tried again with freeBSD 11.3 and 12.0 with more adapters and it didnt pick up the network adapter.
Realised when I read this, it is more of adapter namespace exhaustion.
I'll upload some dmesg from previous tries. Unfortunately I couldnt save the recent dmesg output.
Comment 15 htwfbsd 2019-08-25 15:18:18 UTC
Created attachment 206894 [details]
dmesg output
Comment 16 John Hartley 2019-11-17 05:26:49 UTC
The driver issues that we are seeing with KVM are impacting more than just VirtIO.

Once you go to FreeBSD 11.3 or 12.X you cannot get any networking at all with KVM / QEMU Q35 + OVMF.

I posted bug report here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241774

I believe that these are all related.

We need to do a KVM / QEMU / FreeBSD driver compatibility matrix, as range of options keeps shrinking as FreeBSD releases roll forward.

Regards,

John.
Comment 17 MattS 2020-01-03 00:56:09 UTC
Is there any way to work around this? I'm stuck on 11.2 due to this issue.
Comment 18 Tommy P 2020-01-07 00:17:56 UTC
(In reply to John Hartley from comment #16)

Thanks John for the feedback.  Are you using the default GENERIC kernel?  I didn't have any issues with my custom (trimmed of the GENERIC by removing unused drivers) kernel in conjunction with i440FX chipset. All of the VirtIO (block devices, NIC, memory balloon, etc) work OK.  All of the SCSI controllers (Hypervisor default and VirtIO SCSI models) in addition VirtIO Serial work OK.  It's when I created a VM configuration with Q35 chipset instead of i440FX that none of the VirtIO devices (including the VirtIO Serial & SCSI) are working as per my previous troubleshooting (comment #5 - #7).  Thus, I don't think it's relating the OVMF firmware configuration within KVM-QEMU nor the FreeBSD version.  I'm pretty sure it's how the VirtIO devices are detected and loaded as per FreeBSD's source code /usr/src/sys/dev/virtio/virtio_ids.h.  If you compare the pciconf on a working i440FX VM system vs. a non working Q35 VM, you'll notice the matching 'card' ID to virtio_ids.h on the working i440FX VM while the non-working Q35 VM doesn't have any.  For example:

none2@pci0:2:0:0:       class=0x010000 card=0x11001af4 chip=0x10481af4 rev=0x01 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'Virtio SCSI'
    class      = mass storage
    subclass   = SCSI

Here are the images of the my config:

https://imgur.com/a/rftzjO3

Devices that are loaded and not loaded (no driver attached)

root@d-fbsd:~ # ls /dev/*da*
/dev/ada0       /dev/ada1       /dev/ada2p1     /dev/ada4       /dev/ada5p1     /dev/da1        /dev/da2p1      /dev/da4        /dev/da5p1      /dev/da6p1
/dev/ada0p1     /dev/ada1p1     /dev/ada3       /dev/ada4p1     /dev/da0        /dev/da1p1      /dev/da3        /dev/da4p1      /dev/da5p2      /dev/da6p2
/dev/ada0p2     /dev/ada2       /dev/ada3p1     /dev/ada5       /dev/da0p1      /dev/da2        /dev/da3p1      /dev/da5        /dev/da6
root@d-fbsd:~ # dmesg | egrep -i 'scsi|mass stor|sym'
pci2: <mass storage, SCSI> at device 0.0 (no driver attached)
sym0: <895a> port 0xe000-0xe0ff mem 0xfca02000-0xfca023ff,0xfca00000-0xfca01fff irq 22 at device 0.0 on pci4
sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking
sym1: <895a> port 0xd000-0xd0ff mem 0xfc802000-0xfc8023ff,0xfc800000-0xfc801fff irq 22 at device 0.0 on pci5
sym1: No NVRAM, ID 7, Fast-40, LVD, parity checking
da0 at sym0 bus 0 scbus0 target 0 lun 0
da0: <QEMU QEMU HARDDISK 2.5+> Fixed Direct Access SPC-3 SCSI device
pass13: da1 at sym0 bus 0 scbus0 target 1 lun 0
<QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device
da1: <QEMU QEMU HARDDISK 2.5+> Fixed Direct Access SPC-3 SCSI device
da2 at sym0 bus 0 scbus0 target 2 lun 0
pass13: 150.000MB/s transfers
da2: <QEMU QEMU HARDDISK 2.5+> Fixed Direct Access SPC-3 SCSI device
da3 at sym0 bus 0 scbus0 target 3 lun 0
da3: <QEMU QEMU HARDDISK 2.5+> Fixed Direct Access SPC-3 SCSI device
da4 at sym0 bus 0 scbus0 target 4 lun 0
da4: <QEMU QEMU HARDDISK 2.5+> Fixed Direct Access SPC-3 SCSI device
da5 at sym1 bus 0 scbus1 target 5 lun 0
da5: <QEMU QEMU HARDDISK 2.5+> Fixed Direct Access SPC-3 SCSI device
da6 at sym1 bus 0 scbus1 target 6 lun 0
da6: <QEMU QEMU HARDDISK 2.5+> Fixed Direct Access SPC-3 SCSI device
root@d-fbsd:~ # uname -a
FreeBSD d-fbsd 11.2-RELEASE-p15 FreeBSD 11.2-RELEASE-p15 #0 r356353: Sat Jan  4 16:10:17 PST 2020     root@d-fbsd:/usr/obj/usr/src11.2/sys/Custom  amd64


Regarding the matrix, from my troubleshooting, everything should work as long as you use the i440FX.  If you decide to use the Q35, do not use anything relating to VirtIO (SCSI and Serial storages, NIC, etc).  BTW, if you use shell scripts to maintain your VMs, simply changing the configuration of i440FX to Q35 only would still work since the existing VirtIO devices are attached to the PCI-ISA bus instead of PCIe-PCI (please see https://forums.freebsd.org/threads/freebsd-12-release-guest-in-qemu-kvm.70207/ and https://wiki.qemu.org/Features/Q35).
Comment 19 Tommy P 2020-01-07 00:53:46 UTC
More specific VM's configuration:

PCI-E bus: -device virtio-serial-pci,id=virtio-serial0,bus=pci.1,addr=0x0

PCI bus: -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 

Note the bus=pci.1 vs bus=pci.0.  If I'm correct, the pci.1 (PCI_ABI_VERSION) configure that device is connected to the PCI-E bus instead PCI (pci.0), which utilize different ID as per /usr/src/sys/dev/virtio/pci/virtio_pci.h:

/* VirtIO PCI vendor/device ID. */
#define VIRTIO_PCI_VENDORID     0x1AF4
#define VIRTIO_PCI_DEVICEID_MIN 0x1000
#define VIRTIO_PCI_DEVICEID_MAX 0x103F

/* VirtIO ABI version, this must match exactly. */
#define VIRTIO_PCI_ABI_VERSION  0
Comment 20 John Hartley 2020-01-07 04:07:27 UTC
Hi Tommy P,

thanks very much for response.

Yes I am using GENERIC kernel, I did try a custom kernel removing all the unused devices but that also failed.

Yes agree that issue relates to Q35.

Since original posting I have done testing with Q35 with BIOS & OVMF Firmware and find that with 11.3 and 12.x that VirtIO based device all fail.

Unfortunately with 11.3 & 12.x all available NIC device options (not just VirtIO) also fail (e1000, e1000-82545em, rtl8130 & vmxnet3).

So issue extends beyond just VirtIO based devices.

It seems that between 11.2 and 11.3 there where some changes make to PCI sub-system the result of which is that the PCIe (??) enumeration if failing to detect that NICs at all.

Yesterday I tried to build a 11.3 kernel with 11.2 dev/pci code, but ended up pulling in more code than I wanted, so likely pulled in the associated buggy commits to get kernel complied.

Your tip on swapping to i440FX is worth looking at I assumed that as I have built OVMF VMs that thee would only work with Q35 as OVMF option is tied to this.

I will do more building and testing later today.

For my next build I will only keep dev/pci/pcireg.h file from 11.3 and wind back to 11.2 any (non PCI code)flow through changes that this breaks as part of this test.

I am focused on pci code as Q35 is PCIe native VM and the dev/pci source tree has a hold host of commit/reverts so hoping to at least be able to further isolate problem here.

If I can get generic NICs enumerating then could fold in fix you have identified with VirtIO code and PCI identifiers...


Thanks very much for feedback.

Cheers,

John Hartley.
Comment 21 John Hartley 2020-01-07 04:10:01 UTC
(In reply to Kubilay Kocak from comment #2)

Hi Kubilay,

I have tried with Q35 v4 and v2.11 configured VMs.

Always getting issue going from 11.2 to 11.3 or 12.x

Cheers,

John Hartley
Comment 22 Tommy P 2020-01-09 01:07:03 UTC
(In reply to Bryan Venteicher from comment #13)

Hi Bryan,

I tried the method you've suggested to use the modules/virtio and dev/virtio from your repo on 12.0-RELEASE-p12 r356482 and encountered the following issues:

1) /usr/src/sys/dev/virtio/random/virtio_random.c:241:6: error: too many arguments to function call, expected 3, have 4
            RANDOM_PURE_VIRTIO);
            ^~~~~~~~~~~~~~~~~~
I changed it from:
    random_harvest_queue(&value, sizeof(value), sizeof(value) * NBBY / 2,
          RANDOM_PURE_VIRTIO);
to:
    random_harvest_queue(&value, sizeof(value), RANDOM_PURE_VIRTIO);

2) /usr/src/sys/dev/virtio/pci/virtio_pci.c:54:10: fatal error: 'virtio_pci_if.h' file not found

I used sys/tools/makeobjops.awk to generate the missing virtio_pci_if.h from virtio_pci_if.m

3) A lot of "ld: error: undefined symbol"
--- kernel ---
linking kernel
ld: error: undefined symbol: virtio_pci_disable_vq_desc
>>> referenced by virtio_pci.c
>>>               virtio_pci.o:(vtpci_release_child_resources)

They appear to all from virtio_pci_if.h 's "extern struct kobjop_desc "

---------------------------------------------------------
I also compared the compile object files between your repo and the src r356482.  I think there are some missing (virtio_pci_if, virtio_pci_legacy, virtio_pci_modern):

root@d-build-fbsd12:/usr/obj # ll usr/src_virtio/amd64.amd64/sys/Custom/virt*
-rw-r--r--  1 root  wheel  11312 Jan  8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio.o
-rw-r--r--  1 root  wheel  12480 Jan  8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio_balloon.o
-rw-r--r--  1 root  wheel  24608 Jan  8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio_blk.o
-rw-r--r--  1 root  wheel   2232 Jan  8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio_bus_if.c
-rw-r--r--  1 root  wheel   7531 Jan  8 06:09 usr/src_virtio/amd64.amd64/sys/Custom/virtio_bus_if.h
-rw-r--r--  1 root  wheel   2720 Jan  8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio_bus_if.o
-rw-r--r--  1 root  wheel    797 Jan  8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio_if.c
-rw-r--r--  1 root  wheel   1226 Jan  8 06:09 usr/src_virtio/amd64.amd64/sys/Custom/virtio_if.h
-rw-r--r--  1 root  wheel   1272 Jan  8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio_if.o
-rw-r--r--  1 root  wheel  17632 Jan  8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio_pci.o
-rw-r--r--  1 root  wheel   1283 Jan  8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio_pci_if.c
-rw-r--r--  1 root  wheel   4088 Jan  8 06:09 usr/src_virtio/amd64.amd64/sys/Custom/virtio_pci_if.h
-rw-r--r--  1 root  wheel   1792 Jan  8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio_pci_if.o
-rw-r--r--  1 root  wheel  15312 Jan  8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio_pci_legacy.o
-rw-r--r--  1 root  wheel  26664 Jan  8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio_pci_modern.o
-rw-r--r--  1 root  wheel  41248 Jan  8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio_scsi.o
-rw-r--r--  1 root  wheel   8904 Jan  8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtqueue.o
root@d-build-fbsd12:/usr/obj # ll usr/src12.0/amd64.amd64/sys/Custom/virt*
-rw-r--r--  1 root  wheel  11312 Jan  8 14:34 usr/src12.0/amd64.amd64/sys/Custom/virtio.o
-rw-r--r--  1 root  wheel  12480 Jan  8 14:34 usr/src12.0/amd64.amd64/sys/Custom/virtio_balloon.o
-rw-r--r--  1 root  wheel  24608 Jan  8 14:34 usr/src12.0/amd64.amd64/sys/Custom/virtio_blk.o
-rw-r--r--  1 root  wheel   2229 Jan  8 14:34 usr/src12.0/amd64.amd64/sys/Custom/virtio_bus_if.c
-rw-r--r--  1 root  wheel   7528 Jan  8 14:33 usr/src12.0/amd64.amd64/sys/Custom/virtio_bus_if.h
-rw-r--r--  1 root  wheel   2720 Jan  8 14:34 usr/src12.0/amd64.amd64/sys/Custom/virtio_bus_if.o
-rw-r--r--  1 root  wheel    794 Jan  8 14:34 usr/src12.0/amd64.amd64/sys/Custom/virtio_if.c
-rw-r--r--  1 root  wheel   1223 Jan  8 14:33 usr/src12.0/amd64.amd64/sys/Custom/virtio_if.h
-rw-r--r--  1 root  wheel   1272 Jan  8 14:34 usr/src12.0/amd64.amd64/sys/Custom/virtio_if.o
-rw-r--r--  1 root  wheel  17632 Jan  8 14:34 usr/src12.0/amd64.amd64/sys/Custom/virtio_pci.o
-rw-r--r--  1 root  wheel  41248 Jan  8 14:34 usr/src12.0/amd64.amd64/sys/Custom/virtio_scsi.o
-rw-r--r--  1 root  wheel   8904 Jan  8 14:34 usr/src12.0/amd64.amd64/sys/Custom/virtqueue.o

The module's virtio/pci/Makefile does include those source files but they didn't appear to be compiled:

pci/Makefile:25:
pci/Makefile:26:.PATH: ${SRCTOP}/sys/dev/virtio/pci
pci/Makefile:27:
pci/Makefile:28:KMOD=   virtio_pci
pci/Makefile:29:SRCS=   virtio_pci.c virtio_pci_legacy.c virtio_pci_modern.c
pci/Makefile:30:SRCS+=  virtio_pci_if.c virtio_pci_if.h
pci/Makefile:31:SRCS+=  virtio_bus_if.h virtio_if.h
pci/Makefile:32:SRCS+=  bus_if.h device_if.h pci_if.h
pci/Makefile:33:
pci/Makefile:34:.include <bsd.kmod.mk>



Do you have any suggestion to resolve #3 in order to proceed?  Thanks.
Comment 23 Tommy P 2020-01-09 02:44:19 UTC
Bryan,

After some more debugging, I input some missing entries in sys/conf/files.  The kernel was able to compile successfully with the VirtIO from your repo.  After reboot, all VirtIO devices (SCSI, NIC, memory balloon, etc) are working for 12.0-RELEASE-p12 r356482M!

Thanks again!

As for those who want the VirtIO working in your Q35 environment, I'll attach the the needed stuff including my fix of the VirtIO's random.c so you don't need to download from Bryan's github repo (comment #8).  I don't recommend running in production as it's not officially committed.  Also note that this version of the VirtIO, supporting Q35, will break netmap which you'll need to disable in the kernel configuration prior the build process:

/usr/src/sys/dev/netmap/if_vtnet_netmap.h:224:34: error: no member named 'vtntx_shrhdr' in 'struct vtnet_txq'
                        err = sglist_append(sg, &txq->vtntx_shrhdr, sc->vtnet_hdr_size);
Comment 24 Tommy P 2020-01-09 02:52:23 UTC
Created attachment 210551 [details]
VirtIO /usr/src files to support Q35

How to use this attachment:

1) cd /usr/src
2) tar -xJvf <path/to>/virtio.xz
3) patch < sys_conf_files.patch
4) make kernel
5) reboot

As previously mentioned, this will break kernel's netmap which you'll need to disable/comment in the configuration prior to the build process.  Enjoy :)
Comment 25 MattS 2020-01-09 04:17:02 UTC
TommyP, 

Thank you for your work on this! Can this patch be applied to 11.3, or can it be easily created?
Comment 26 Bryan Venteicher freebsd_committer 2020-01-09 04:21:04 UTC
Tommy,

I'm in the process of rebasing the virtio branch to HEAD since it is ~!8 months old by now and hope to commit that work to HEAD in the next 1-2 months. You can try out the virtio-rebase branch from the same repo but I believe what is pushed there has only been compiled tested. Please contact me directly if you experience any issues in the meanwhile.
Comment 27 Tommy P 2020-01-09 05:20:58 UTC
Created attachment 210556 [details]
VirtIO + netmap to support Q35

Apply this patch in /usr/src (cd /usr/src && patch < sys_dev_virtio_netmap.patch).  I currently don't have netmap configured within my environment to fully test VirtIO + netmap but a full compile of the GENERIC was successfull.
Comment 28 Dobri Dobrev 2020-01-12 14:08:30 UTC
Tried both patches on 12.1-release - no virtio devices show up.
Comment 29 Dobri Dobrev 2020-01-12 16:10:41 UTC
Created attachment 210660 [details]
Errors

Please ignore my previous comment, it worked.

I also compiled with the 2nd patch + netmap enabled - netmap is visible in /dev/, however there seem to be some errors during boot, see the attachment.
Comment 30 Tommy P 2020-01-12 20:33:12 UTC
(In reply to Dobri Dobrev from comment #29)

Hi Dobri,

Thanks for the feedback.  Upon further investigation after submitting the netmap patch, it seems netmap breaks the VirtIO during probing of the devices in Q35, even when trying Bryan's sys/dev/netmap/if_vtnetmap.h.  Everything works fine, including netmap, for i440FX.  I'll see what I can do to get netmap + VirtIO for Q35.  If it's beyond my understanding of the FreeBSD's networking, I'll have to refer to the FreeBSD gurus :).

Regards,
Tommy
Comment 31 Tommy P 2020-01-12 20:51:24 UTC
Jan 12 02:11:23 fbsd12 kernel: vtpcil2: <VirtIO PCI (legacy) SCSI adapter> port 0xc180-0xc1bf mem 0xfc13b000-0xfc13bfff,0xfebf4000-0xfebf7fff irq 10 at device 9.0 on pci0
Jan 12 02:11:23 fbsd12 kernel: vtscsi0: <VirtIO SCSI Adapter> on vtpcil2
Jan 12 02:11:23 fbsd12 kernel: vtpcil3: <VirtIO PCI (legacy) Block adapter> port 0xc1c0-0xc1ff mem 0xfc13c000-0xfc13cfff,0xfebf8000-0xfebfbfff irq 10 at device 10.0 on pci0
Jan 12 02:11:23 fbsd12 kernel: vtblk0: <VirtIO Block Adapter> on vtpcil3
Jan 12 02:11:23 fbsd12 kernel: vtpcil4: <VirtIO PCI (legacy) Network adapter> port 0xc260-0xc27f mem 0xfc13d000-0xfc13dfff,0xfebfc000-0xfebfffff irq 11 at device 11.0 on pci0
Jan 12 02:11:23 fbsd12 kernel: vtnet0: <VirtIO Network Adapter> on vtpcil4
Jan 12 02:11:23 fbsd12 kernel: vtnet0: Ethernet address: 52:54:00:88:a5:32
Jan 12 02:11:23 fbsd12 kernel: 000.000854 [ 502] vtnet_netmap_attach       max rings 1
Jan 12 02:11:23 fbsd12 kernel: vtnet0: netmap queues/slots: TX 1/1024, RX 1/1024
Jan 12 02:11:23 fbsd12 kernel: 000.000869 [ 507] vtnet_netmap_attach       virtio attached txq=1, txd=1024 rxq=1, rxd=1024
Jan 12 02:11:23 fbsd12 kernel: ada0: <QEMU HARDDISK 2.5+> ATA-7 SATA device
Jan 12 02:11:23 fbsd12 kernel: ada0: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes)
Jan 12 02:11:23 fbsd12 kernel: cd0: <QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device
Jan 12 02:11:23 fbsd12 kernel: cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes)
Jan 12 02:11:23 fbsd12 kernel: em0: link state changed to UP
Jan 12 02:55:08 fbsd12 kernel: em0: <Intel(R) PRO/1000 Network Connection> port 0xc100-0xc13f mem 0xfc100000-0xfc11ffff irq 11 at device 3.0 on pci0
Jan 12 02:55:08 fbsd12 kernel: em0: attach_pre capping queues at 1
Jan 12 02:55:08 fbsd12 kernel: em0: using 1024 tx descriptors and 1024 rx descriptors
Jan 12 02:55:08 fbsd12 kernel: em0: allocated for 1 tx_queues
Jan 12 02:55:08 fbsd12 kernel: em0: allocated for 1 rx_queues
Jan 12 02:55:08 fbsd12 kernel: em0: Ethernet address: 52:54:00:ea:98:ec
Jan 12 02:55:08 fbsd12 kernel: em0: netmap queues/slots: TX 1/1024, RX 1/1024
Jan 12 02:55:08 fbsd12 kernel: vtpcil0: <VirtIO PCI (legacy) Console adapter> port 0xc140-0xc17f mem 0xfc138000-0xfc138fff,0xfebec000-0xfebeffff irq 10 at device 5.0 on pci0
Jan 12 02:55:08 fbsd12 kernel: vtpcil1: <VirtIO PCI (legacy) Balloon adapter> port 0xc220-0xc23f mem 0xfebf0000-0xfebf3fff irq 10 at device 6.0 on pci0
Jan 12 02:55:08 fbsd12 kernel: vtballoon0: <VirtIO Balloon Adapter> on vtpcil1

As you can see VirtIO + netmap works fine in legacy i440fx.  Attaching this HDD to the Q35 VM breaks the VirtIO detection.  If netmap is diabled in the kernel, VirtIO works fine for Q35.
Comment 32 Vincenzo Maffione freebsd_committer 2020-01-12 21:04:24 UTC
(In reply to Tommy P from comment #31)
So I should be able to reproduce this by creating a FreeBSD 12.x VM on virt-manager with Q35?
Comment 33 John Hartley 2020-01-12 23:24:58 UTC
(In reply to Tommy P from comment #30)

Hi Tommy T & Dobri,

Vincenzo has found root cause and indicated that he will commit fix.

See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241774#c35

So looks like we are making progress on both VirtIO & netmap front.

Cheers,

John Hartley.
Comment 34 Tommy P 2020-01-13 17:29:25 UTC
Created attachment 210713 [details]
VirtIO + netmap to support Q35

This netmap patch is a merge between official (r356535M) and Bryan's VirtIO work (not the virtio-rebase since it's based on 13-CURRENT).  This patch works in legacy i440fx but fails in Q35 due to bug in netmap supporting PCI-e.
Comment 35 Tommy P 2020-01-13 17:36:04 UTC
(In reply to Vincenzo Maffione from comment #32)

Hi Vincenzo,

Yes, I created the patch (comment #34) to produce the output seen in comment #31.  My intended goal was the proposed patch should be applied to the legacy i440fx VM and verify that all configurations and applications work as before.  Then simply removing the HDD and attaching it to a non OVMF Q35 VM for quick and easy migration.

I left a few commented code as I'm still not very familiar with FreeBSD's networking, more so for the netmap (ie whether to merge vtnet_netmap_free_bufs and vtnet_free_used).

Thanks,
Tommy
Comment 36 John Hartley 2020-01-13 19:00:04 UTC
(In reply to Tommy P from comment #24)

Hi Tommy,

now I have networking workaround (disable netmap), I finally did update of my 12.0 VM to 12.1 and applied the VirtIO patch.

I successfully getting Virto devices in my PCI-e config:

# pciconf -lcve 
hostb0@pci0:0:0:0:	class=0x060000 card=0x11001af4 chip=0x29c08086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82G33/G31/P35/P31 Express DRAM Controller'
    class      = bridge
    subclass   = HOST-PCI
vgapci0@pci0:0:1:0:	class=0x030000 card=0x11001af4 chip=0x01001b36 rev=0x04 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'QXL paravirtual graphic card'
    class      = display
    subclass   = VGA
pcib1@pci0:0:2:0:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 000d[148] = ACS 1
pcib3@pci0:0:2:1:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 000d[148] = ACS 1
pcib4@pci0:0:2:2:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 000d[148] = ACS 1
pcib5@pci0:0:2:3:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 000d[148] = ACS 1
pcib6@pci0:0:2:4:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 000d[148] = ACS 1
uhci0@pci0:0:29:0:	class=0x0c0300 card=0x11001af4 chip=0x29348086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
uhci1@pci0:0:29:1:	class=0x0c0300 card=0x11001af4 chip=0x29358086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
uhci2@pci0:0:29:2:	class=0x0c0300 card=0x11001af4 chip=0x29368086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
ehci0@pci0:0:29:7:	class=0x0c0320 card=0x11001af4 chip=0x293a8086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB2 EHCI Controller'
    class      = serial bus
    subclass   = USB
isab0@pci0:0:31:0:	class=0x060100 card=0x11001af4 chip=0x29188086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801IB (ICH9) LPC Interface Controller'
    class      = bridge
    subclass   = PCI-ISA
ahci0@pci0:0:31:2:	class=0x010601 card=0x11001af4 chip=0x29228086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]'
    class      = mass storage
    subclass   = SATA
    cap 05[80] = MSI supports 1 message, 64 bit enabled with 1 message
    cap 12[a8] = SATA Index-Data Pair
none0@pci0:0:31:3:	class=0x0c0500 card=0x11001af4 chip=0x29308086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) SMBus Controller'
    class      = serial bus
    subclass   = SMBus
pcib2@pci0:1:0:0:	class=0x060400 card=0x00000000 chip=0x000e1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    class      = bridge
    subclass   = PCI-PCI
    cap 05[8c] = MSI supports 1 message, 64 bit, vector masks 
    cap 01[84] = powerspec 3  supports D0 D3  current D0
    cap 10[48] = PCI-Express 2 PCI bridge max data 128(128) ARI disabled
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
    cap 0c[40] = unknown
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
vmx0@pci0:2:1:0:	class=0x020000 card=0x07b015ad chip=0x07b015ad rev=0x01 hdr=0x00
    vendor     = 'VMware'
    device     = 'VMXNET3 Ethernet Controller'
    class      = network
    subclass   = ethernet
    cap 11[9c] = MSI-X supports 25 messages, enabled
                 Table in map 0x18[0x0], PBA in map 0x18[0x1000]
    cap 05[84] = MSI supports 1 message, 64 bit 
em0@pci0:2:2:0:	class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82540EM Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet
re0@pci0:2:3:0:	class=0x020000 card=0x11001af4 chip=0x813910ec rev=0x20 hdr=0x00
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTL-8100/8101L/8139 PCI Fast Ethernet Adapter'
    class      = network
    subclass   = ethernet
vtpcim0@pci0:3:0:0:	class=0x010000 card=0x11001af4 chip=0x10421af4 rev=0x01 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'Virtio block device'
    class      = mass storage
    subclass   = SCSI
    cap 11[dc] = MSI-X supports 2 messages, enabled
                 Table in map 0x14[0x0], PBA in map 0x14[0x800]
    cap 09[c8] = vendor (length 20)
    cap 09[b4] = vendor (length 20)
    cap 09[a4] = vendor (length 16)
    cap 09[94] = vendor (length 16)
    cap 09[84] = vendor (length 16)
    cap 01[7c] = powerspec 3  supports D0 D3  current D0
    cap 10[40] = PCI-Express 2 endpoint max data 128(128)
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
vtpcim1@pci0:4:0:0:	class=0x010000 card=0x11001af4 chip=0x10481af4 rev=0x01 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'Virtio SCSI'
    class      = mass storage
    subclass   = SCSI
    cap 11[dc] = MSI-X supports 4 messages, enabled
                 Table in map 0x14[0x0], PBA in map 0x14[0x800]
    cap 09[c8] = vendor (length 20)
    cap 09[b4] = vendor (length 20)
    cap 09[a4] = vendor (length 16)
    cap 09[94] = vendor (length 16)
    cap 09[84] = vendor (length 16)
    cap 01[7c] = powerspec 3  supports D0 D3  current D0
    cap 10[40] = PCI-Express 2 endpoint max data 128(128)
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
vtpcim2@pci0:5:0:0:	class=0x00ff00 card=0x11001af4 chip=0x10451af4 rev=0x01 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'Virtio memory balloon'
    class      = old
    cap 09[c8] = vendor (length 20)
    cap 09[b4] = vendor (length 20)
    cap 09[a4] = vendor (length 16)
    cap 09[94] = vendor (length 16)
    cap 09[84] = vendor (length 16)
    cap 01[7c] = powerspec 3  supports D0 D3  current D0
    cap 10[40] = PCI-Express 2 endpoint max data 128(128)
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
vtpcim3@pci0:6:0:0:	class=0x010000 card=0x11001af4 chip=0x10421af4 rev=0x01 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'Virtio block device'
    class      = mass storage
    subclass   = SCSI
    cap 11[dc] = MSI-X supports 2 messages, enabled
                 Table in map 0x14[0x0], PBA in map 0x14[0x800]
    cap 09[c8] = vendor (length 20)
    cap 09[b4] = vendor (length 20)
    cap 09[a4] = vendor (length 16)
    cap 09[94] = vendor (length 16)
    cap 09[84] = vendor (length 16)
    cap 01[7c] = powerspec 3  supports D0 D3  current D0
    cap 10[40] = PCI-Express 2 endpoint max data 128(128)
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)

I was able to easily flip my Q35 SATA based 12.1 to VirtIO SCSI by editing /etc/fstab and change SATA ada0p2, atap3 ... to SCSI da0p2, da01p3 ...

Simillary if I wanted to flip to direct VirtIO (rather than via VirtIO SCSI) you need to edit your /etc/fstab mounts to use vtbd0p2, vtbd0p3 etc

So happy to report that VirtIO patch worked for me with 12.1 rebuilt kernel.

NOTE: the partition numbering here is due to using OVMF which results in FAT32 EFI partition always being on p1 hence root is on p2 in my case.

Thanks again for hard work.

Cheers,


John Hartley.
Comment 37 Tommy P 2020-01-13 20:43:43 UTC
(In reply to John Hartley from comment #36)


Hi John,

Thanks for the feedback.  I use GPT ID for mounting in fstab.  Thus, regardless of device type (SATA, SCSI, Virtio SCSI, or VirtIO Serial), it always boot without modifying the fstab provided that the storage controllers work as expected.  To me, it's more efficient for testing.

Regards,
Tommy
Comment 38 Tommy P 2020-01-13 23:26:24 UTC
Created attachment 210723 [details]
Recent update in stable/12 for VirtIO

r347233:

Remove non-functional SCTP checksum offload support for virtio.

Checksum offloading for SCTP is not currently specified for virtio.
If the hypervisor announces checksum offloading support, it means TCP
and UDP checksum offload. If an SCTP packet is sent and the host announced
checksum offload support, the hypervisor inserts the IP checksum (16-bit)
at the correct offset, but this is not the right checksum, which is a CRC32c.
This results in all outgoing packets having the wrong checksum and therefore
breaking SCTP based communications.

This patch removes SCTP checksum offloading support from the virtio
network interface.

Thanks to Felix Weinrank for making me aware of the issue.

Reviewed by:		bryanv@
MFC after:		1 week
https://svnweb.freebsd.org/base?view=revision&revision=347233
-------------------------------------------------------------
r349285:

VirtIO SCSI:  validate seg_max on attach

Until r349278, bhyve presented a seg_max to the guest that was too large.
Detect this case and clamp it to the virtqueue size.  Otherwise, we would
fail the "too many segments to enqueue" assertion in virtqueue_enqueue().

I hit this by running a guest with a MAXPHYS of 256 KB.

Reviewed by:	bryanv cem
MFC after:	1 week
Sponsored by:	Dell EMC Isilon
https://svnweb.freebsd.org/base?view=revision&revision=349285
Comment 39 Tommy P 2020-01-14 06:31:47 UTC
Created attachment 210728 [details]
VirtIO to support Q35 backported for FreeBSD 11.x

--------------------------------------------------

Thank you Bryan for your help with getting past (CK_S)TAILQ_FOREACH compile errors!  I greatly appreciate it!

--------------------------------------------------

root@fbsd11-ovmf:~ # egrep -i 'sata|scsi|mass stor|virtio|sym0|em0|vtnet' /var/log/messages | grep 'Jan 13'
Jan 13 22:10:55 fbsd11-ovmf kernel: vtpcim0: <VirtIO PCI (modern) Network adapter> mem 0x98e00000-0x98e00fff,0x800000000-0x800003fff irq 22 at device 0.0 on pci1
Jan 13 22:10:55 fbsd11-ovmf kernel: vtnet0: <VirtIO Network Adapter> on vtpcim0
Jan 13 22:10:55 fbsd11-ovmf kernel: vtnet0: Ethernet address: 52:54:00:a9:08:fb
Jan 13 22:10:55 fbsd11-ovmf kernel: sym0: <895a> port 0xb000-0xb0ff mem 0x98c02000-0x98c023ff,0x98c00000-0x98c01fff irq 22 at device 0.0 on pci2
Jan 13 22:10:55 fbsd11-ovmf kernel: sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking
Jan 13 22:10:55 fbsd11-ovmf kernel: vtpcim1: <VirtIO PCI (modern) SCSI adapter> mem 0x98a00000-0x98a00fff,0x800100000-0x800103fff irq 22 at device 0.0 on pci3
Jan 13 22:10:55 fbsd11-ovmf kernel: vtscsi0: <VirtIO SCSI Adapter> on vtpcim1
Jan 13 22:10:55 fbsd11-ovmf kernel: vtpcim2: <VirtIO PCI (modern) Console adapter> mem 0x98800000-0x98800fff,0x800200000-0x800203fff irq 22 at device 0.0 on pci4
Jan 13 22:10:55 fbsd11-ovmf kernel: vtpcim3: <VirtIO PCI (modern) Balloon adapter> mem 0x800300000-0x800303fff irq 22 at device 0.0 on pci5
Jan 13 22:10:55 fbsd11-ovmf kernel: vtballoon0: <VirtIO Balloon Adapter> on vtpcim3
Jan 13 22:10:55 fbsd11-ovmf kernel: vtpcim4: <VirtIO PCI (modern) Block adapter> mem 0x98400000-0x98400fff,0x800400000-0x800403fff irq 22 at device 0.0 on pci6
Jan 13 22:10:55 fbsd11-ovmf kernel: vtblk0: <VirtIO Block Adapter> on vtpcim4
Jan 13 22:10:55 fbsd11-ovmf kernel: em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x6000-0x603f mem 0x98000000-0x9801ffff irq 21 at device 1.0 on pci8
Jan 13 22:10:55 fbsd11-ovmf kernel: em0: Ethernet address: 52:54:00:4e:d4:8f
Jan 13 22:10:55 fbsd11-ovmf kernel: ahci0: <Intel ICH9 AHCI SATA controller> port 0xc240-0xc25f mem 0x99002000-0x99002fff irq 16 at device 31.2 on pci0
Jan 13 22:10:55 fbsd11-ovmf kernel: ada0: <QEMU HARDDISK 2.5+> ATA-7 SATA device
Jan 13 22:10:55 fbsd11-ovmf kernel: ada0: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes)
Jan 13 22:10:55 fbsd11-ovmf kernel: pass0: <QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device
Jan 13 22:10:55 fbsd11-ovmf kernel: pass0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes)
Jan 13 22:10:55 fbsd11-ovmf kernel: em0: link state changed to UP

root@fbsd11-ovmf:~ # ls /dev/{ada,da,vtbd}*
/dev/ada0       /dev/ada0p1     /dev/ada0p2     /dev/vtbd0      /dev/vtbd0p1    /dev/vtbd0p2

root@fbsd11-ovmf:~ # ifconfig
vtnet0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6>
        ether 52:54:00:a9:08:fb
        hwaddr 52:54:00:a9:08:fb
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 52:54:00:4e:d4:8f
        hwaddr 52:54:00:4e:d4:8f
        inet ... netmask 0xffffff00 broadcast ...255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo

root@fbsd11-ovmf:~ # uname -a
FreeBSD fbsd11-ovmf 11.3-RELEASE-p5 FreeBSD 11.3-RELEASE-p5 #0 r356701M: Mon Jan 13 22:08:55 PST 2020
--------------------------------------------------
The above is from OVMF + Q35 VM.  Should also work w/o OVMF.  This all inclusive (netmap + updates in stable/12).

1) cd /usr/src
2) tar -xJvf <path/to/>virtio_11.xz
3) patch < sys_conf_files_v11.2.patch
4) make kernel
5) reboot

I'll update the patches as soon as netmap is fixed for PCI-e bus.  Please let me know if you have any issues.
Comment 40 Tommy P 2020-01-14 15:25:40 UTC
Created attachment 210734 [details]
VirtIO to support Q35 backported for FreeBSD 11.x

Forgot to include Makefiles when making the tar package.
Comment 41 Tommy P 2020-01-14 16:57:13 UTC
Created attachment 210737 [details]
VirtIO to support Q35 for FreeBSD 12.x
Comment 42 John Hartley 2020-01-15 08:53:00 UTC
(In reply to Tommy P from comment #41)

Hi Tommy P & Brian V & Vicenzo,

the VirtIO network update conflicts with Vincenzo's netmap fix.

In my earlier test of VirtIO patch test I had just disable netmap via GENERIC conf file and not applied netmap fix.

I have now tested against 12.1 where I first applied the netmap fix.


Here is compile error:

cc -target x86_64-unknown-freebsd12.1 --sysroot=/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin -c -O2 -pipe -fno-strict-aliasing  -g -nostdinc  -I. -I/usr/src/sys -I/usr/src/sys/contrib/ck/include -I/usr/src/sys/contrib/libfdt -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h  -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -MD  -MF.depend.if_vtnet.o -MTif_vtnet.o -fdebug-prefix-map=./machine=/usr/src/sys/amd64/include -fdebug-prefix-map=./x86=/usr/src/sys/x86/include -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float  -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -gdwarf-2 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-address-of-packed-member  -mno-aes -mno-avx  -std=iso9899:1999 -Werror  /usr/src/sys/dev/virtio/network/if_vtnet.c
In file included from /usr/src/sys/dev/virtio/network/if_vtnet.c:351:
/usr/src/sys/dev/netmap/if_vtnet_netmap.h:94:2: error: implicit declaration of function 'D' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        D("freed %d mbufs, %d netmap bufs on %d queues",
        ^
/usr/src/sys/dev/netmap/if_vtnet_netmap.h:94:2: error: this function declaration is not a prototype [-Werror,-Wstrict-prototypes]
/usr/src/sys/dev/netmap/if_vtnet_netmap.h:297:3: error: implicit declaration of function 'ND' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
                ND(3,"sent %d packets, hwcur %d", n, nm_i);
                ^
/usr/src/sys/dev/netmap/if_vtnet_netmap.h:297:3: error: this function declaration is not a prototype [-Werror,-Wstrict-prototypes]
/usr/src/sys/dev/netmap/if_vtnet_netmap.h:306:7: error: implicit declaration of function 'ND' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
                    ND(5, "pure notify ? head %d tail %d nused %d %d",
                    ^
/usr/src/sys/dev/netmap/if_vtnet_netmap.h:340:3: error: implicit declaration of function 'ND' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
                ND(3, "disable intr, hwcur %d", nm_i);
                ^
/usr/src/sys/dev/netmap/if_vtnet_netmap.h:343:3: error: implicit declaration of function 'ND' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
                ND(3, "enable intr, hwcur %d", nm_i);
                ^
/usr/src/sys/dev/netmap/if_vtnet_netmap.h:297:37: error: variable 'n' is uninitialized when used here [-Werror,-Wuninitialized]
                ND(3,"sent %d packets, hwcur %d", n, nm_i);
                                                  ^
/usr/src/sys/dev/netmap/if_vtnet_netmap.h:245:9: note: initialize the variable 'n' to silence this warning
        u_int n;
               ^
                = 0
/usr/src/sys/dev/netmap/if_vtnet_netmap.h:486:6: error: implicit declaration of function 'RD' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
                                        RD(1, "Truncated virtio-net-header, "
                                        ^
/usr/src/sys/dev/netmap/if_vtnet_netmap.h:486:6: error: this function declaration is not a prototype [-Werror,-Wstrict-prototypes]
/usr/src/sys/dev/netmap/if_vtnet_netmap.h:498:2: error: implicit declaration of function 'ND' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        ND("[B] h %d c %d hwcur %d hwtail %d",
        ^
/usr/src/sys/dev/netmap/if_vtnet_netmap.h:566:4: error: implicit declaration of function 'D' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
                        D("strange, null netmap ring %d", r);
                        ^
/usr/src/sys/dev/netmap/if_vtnet_netmap.h:662:2: error: implicit declaration of function 'D' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        D("max rings %d", sc->vtnet_max_vq_pairs);
        ^
/usr/src/sys/dev/virtio/network/if_vtnet.c:3411:3: error: implicit declaration of function 'D' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
                D("try to attach again");
                ^
14 errors generated.
*** Error code 1

As sys/dev/netmap/if_vtnet_netmap.h was one of replaced files that was provided with fix, I put back the original 12.1 version (this was not changed or update by Vincenzo's netmap fix).

The result was failure in compiling sys/dev/virtio/network/if_vtnet.c

Here is compile result:

cc -target x86_64-unknown-freebsd12.1 --sysroot=/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin -c -O2 -pipe -fno-strict-aliasing  -g -nostdinc  -I. -I/usr/src/sys -I/usr/src/sys/contrib/ck/include -I/usr/src/sys/contrib/libfdt -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h  -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -MD  -MF.depend.if_vtnet.o -MTif_vtnet.o -fdebug-prefix-map=./machine=/usr/src/sys/amd64/include -fdebug-prefix-map=./x86=/usr/src/sys/x86/include -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float  -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -gdwarf-2 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-address-of-packed-member  -mno-aes -mno-avx  -std=iso9899:1999 -Werror  /usr/src/sys/dev/virtio/network/if_vtnet.c
/usr/src/sys/dev/virtio/network/if_vtnet.c:3219:6: error: implicit declaration of function 'vtnet_netmap_init_rx_buffers' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]
        if (vtnet_netmap_init_rx_buffers(sc))
            ^
/usr/src/sys/dev/virtio/network/if_vtnet.c:3219:6: error: this function declaration is not a prototype [-Werror,-Wstrict-prototypes]
/usr/src/sys/dev/virtio/network/if_vtnet.c:3411:3: error: implicit declaration of function 'D' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]
                D("try to attach again");
                ^
/usr/src/sys/dev/virtio/network/if_vtnet.c:3411:3: error: this function declaration is not a prototype [-Werror,-Wstrict-prototypes]
4 errors generated.
*** Error code 1

Stop.
make[2]: stopped in /usr/obj/usr/src/amd64.amd64/sys/GENERIC
*** Error code 1

Given that both Brian (virtio committer) and Vincenzo (netmap committer) have both been involved in helping to resolve VirtIO and netmap related issues, it would be great if we could get VirtIO and netmap fixes that worked together.

Thanks,


John Hartley.
Comment 43 Tommy P 2020-01-16 00:51:37 UTC
Created attachment 210783 [details]
12.x Disable Virtio + netmap interop

Jan 15 16:47:29 d-build-fbsd12 kernel: vtpcim0: <VirtIO PCI (modern) Network adapter> mem 0xfd480000-0xfd480fff,0xfea00000-0xfea03fff irq 22 at device 0.0 on pci1
Jan 15 16:47:29 d-build-fbsd12 kernel: vtnet0: <VirtIO Network Adapter> on vtpcim0
Jan 15 16:47:29 d-build-fbsd12 kernel: sym0: <895a> port 0xd000-0xd0ff mem 0xfd202000-0xfd2023ff,0xfd200000-0xfd201fff irq 22 at device 0.0 on pci2
Jan 15 16:47:29 d-build-fbsd12 kernel: sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking
Jan 15 16:47:29 d-build-fbsd12 kernel: vtpcim1: <VirtIO PCI (modern) SCSI adapter> mem 0xfd000000-0xfd000fff,0xfe600000-0xfe603fff irq 22 at device 0.0 on pci3
Jan 15 16:47:29 d-build-fbsd12 kernel: vtscsi0: <VirtIO SCSI Adapter> on vtpcim1
Jan 15 16:47:29 d-build-fbsd12 kernel: vtpcim2: <VirtIO PCI (modern) Console adapter> mem 0xfce00000-0xfce00fff,0xfe400000-0xfe403fff irq 22 at device 0.0 on pci4
Jan 15 16:47:29 d-build-fbsd12 kernel: vtpcim3: <VirtIO PCI (modern) Balloon adapter> mem 0xfe200000-0xfe203fff irq 22 at device 0.0 on pci5
Jan 15 16:47:29 d-build-fbsd12 kernel: vtballoon0: <VirtIO Balloon Adapter> on vtpcim3
Jan 15 16:47:29 d-build-fbsd12 kernel: em0: <Intel(R) PRO/1000 Network Connection> port 0xc000-0xc03f mem 0xfc180000-0xfc19ffff irq 21 at device 1.0 on pci11
Jan 15 16:47:29 d-build-fbsd12 kernel: em0: Using 1024 TX descriptors and 1024 RX descriptors
Jan 15 16:47:29 d-build-fbsd12 kernel: em0: Ethernet address: 52:54:00:7c:b9:29
Jan 15 16:47:29 d-build-fbsd12 kernel: em0: netmap queues/slots: TX 1/1024, RX 1/1024
Jan 15 16:47:29 d-build-fbsd12 kernel: ahci0: <Intel ICH9 AHCI SATA controller> port 0xe0c0-0xe0df mem 0xfd61c000-0xfd61cfff irq 16 at device 31.2 on pci0
Jan 15 16:47:29 d-build-fbsd12 kernel: ada0: <QEMU HARDDISK 2.5+> ATA-7 SATA device
Jan 15 16:47:29 d-build-fbsd12 kernel: ada0: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes)
Jan 15 16:47:29 d-build-fbsd12 kernel: ada1: <QEMU HARDDISK 2.5+> ATA-7 SATA device
Jan 15 16:47:29 d-build-fbsd12 kernel: ada1: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes)
Jan 15 16:47:29 d-build-fbsd12 kernel: ada2: <QEMU HARDDISK 2.5+> ATA-7 SATA device
Jan 15 16:47:29 d-build-fbsd12 kernel: ada2: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes)
Jan 15 16:47:29 d-build-fbsd12 kernel: ada3: <QEMU HARDDISK 2.5+> ATA-7 SATA device
Jan 15 16:47:29 d-build-fbsd12 kernel: ada3: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes)
Jan 15 16:47:29 d-build-fbsd12 kernel: em0: link state changed to UP

uname -a
FreeBSD d-build-fbsd12 12.1-RELEASE-p1 FreeBSD 12.1-RELEASE-p1 #0 r356482M:
Comment 44 John Hartley 2020-01-16 11:18:08 UTC
(In reply to Tommy P from comment #43)

Hi Tommy P,

thanks again for continuing to work on this.

I have applied a slight variation to what you posted, which did:

#if defined(DEV_NETMAP) && defined(WITH_PTNETMAP)

Conditional compilation rather than remove the netmap code.

See diffs here:

# diff sys/dev/virtio/network ~/virtio-12/sys/dev/virtio/network
diff sys/dev/virtio/network/if_vtnet.c /home/jbh/virtio-12/sys/dev/virtio/network/if_vtnet.c
350c350
< #if defined(DEV_NETMAP) && defined(WITH_PTNETMAP)
---
> #ifdef DEV_NETMAP
367c367
< #if defined(DEV_NETMAP) && defined(WITH_PTNETMAP)
---
> #ifdef DEV_NETMAP
475c475
< #if defined(DEV_NETMAP) && defined(WITH_PTNETMAP)
---
> #ifdef DEV_NETMAP
507c507
< #if defined(DEV_NETMAP) && defined(WITH_PTNETMAP)
---
> #ifdef DEV_NETMAP
2031c2031
< #if defined(DEV_NETMAP) && defined(WITH_PTNETMAP)
---
> #ifdef DEV_NETMAP
2766c2766
< #if defined(DEV_NETMAP) && defined(WITH_PTNETMAP)
---
> #ifdef DEV_NETMAP
3070c3070
< #if defined(DEV_NETMAP) && defined(WITH_PTNETMAP)
---
> #ifdef DEV_NETMAP
3218c3218
< #if defined(DEV_NETMAP) && defined(WITH_PTNETMAP)
---
> #ifdef DEV_NETMAP
3409c3409
< #if defined(DEV_NETMAP) && defined(WITH_PTNETMAP)
---
> #ifdef DEV_NETMAP
diff sys/dev/virtio/network/if_vtnetvar.h /home/jbh/virtio-12/sys/dev/virtio/network/if_vtnetvar.h
85c85
< #if defined(DEV_NETMAP) & defined(WITH_PTNETMAP)
---
> #ifdef DEV_NETMAP
124c124
< #if defined(DEV_NETMAP) && defined(WITH_PTNETMAP)

This is just a suggestion as it allows for easy "re-enablement" of netmap in the event that Vincenzo re-adds "#define WITH_PTNETMAP" into sys/dev/netmap/netmap_kern.h netmap parameter file.

In my testing I got clean compile across your, Brian's and Vincenzo's updates, but did not get any VirtIO devices showing up :-( .  This is likely due to either:
a. me having incorrectly apply update or other issue, due to all my add/removing  and patching code
b. some other QEMU / Q35 version issues (which MattS has reported on other defect thread).

Cheers,


John Hartley.
Comment 45 Vincenzo Maffione freebsd_committer 2020-01-16 20:49:19 UTC
(In reply to John Hartley from comment #44)
If you are adding checks on WITH_PTNETMAP, that's wrong.
PTNETMAP has no relationships with if_vtnet.
Look, the netmap fix will be committed to stable/12 (and stable/11) in short, so you can cherry-pick from there.
Comment 46 Vincenzo Maffione freebsd_committer 2020-01-16 20:51:01 UTC
(In reply to John Hartley from comment #42)
I don't understand what you are trying to do, but the D, ND and RD macros have been renamed in the following way:

D --> nm_prinf (or nm_prerr)
ND --> nm_prdis
RD --> nm_prlim
Comment 47 John Hartley 2020-01-17 07:40:09 UTC
(In reply to Vincenzo Maffione from comment #46)

Hi Vincenzo,

I was attempting to get machine with: Q35, OVMF, 12.1, VirtIO, all networking (em, vmx, re, vtnet).

Thank for tip on ND, D & RD debug macros, that did get me past compile error but did not get me to working VirtIO.

I wound out the defined(DEV_NETMAP) && defined(WITH_PTNETMAP) hack which was my amateur attempt to try to get this to work as Tommy's version removed all ifdef DEV_NETMAP surrounded compilation which seem a bit drastic to me.

My final 12.1 virtio with netmap fix merge was:

1. Use patched netmap
2. Use patched virtio (with exception of network)
3. Use base virtio network
4. Update code to avoid netmap debug macro compile errors: D, RD & ND as per your guidance

Against clean install the differences is over 3000 lines....
So huge effort on VirtIO by Tommy P and Brian V.

I am doing to now wait potentially for the updates to flow through.

Cheers,

John Hartley.
Comment 48 John Hartley 2020-01-17 07:42:20 UTC
Created attachment 210808 [details]
Consolidated Differences across VirtIO / netmap

Here is set of consolidated differences across VirtIO and netmap.
This is not working set.
Networking is working but not VirtIO.
Comment 49 John Hartley 2020-01-18 08:59:28 UTC
(In reply to John Hartley from comment #47)

Hi VirtIO'ers,

I have not got: QEMU Q35 with OVMF, VirtIO SCSI (boot device), netmap + em & vmx.

I had to sacrifice vtnet (VirtIO networking) as it inclusion seems to break the rest of VirtIO (if you have netmap).

Here is are key differences:

1. Remove VirtIO networking from kernel

diff sys/amd64/conf/GENERIC sys/amd64/conf/GENERIC.bak.12-1-REL 
349c349
< # device		vtnet			# VirtIO Ethernet device
---
> device		vtnet			# VirtIO Ethernet device

 
2. Change netmap if_net.c NETMAP patch so it essentially stops VirtIO code generation, by moving it up to stop #include <dev/virtio/network/virtio_net.h>

# diff sys/dev/netmap/if_ptnet.c sys/dev/netmap/if_ptnet.c.bak.12-1-NETMAP 
88,89d87
< 
< #ifdef WITH_PTNETMAP
91a90
> #ifdef WITH_PTNETMAP

here is snippet of result:

<<snip>>
...
#include <dev/netmap/netmap_mem2.h>

#ifdef WITH_PTNETMAP
#include <dev/virtio/network/virtio_net.h>

#ifndef INET
#error "INET not defined, cannot support offloadings"
#endif
...
<<end snip>>

NOTE: you could likely get same result by just remove compile from sys/conf/files

3. Remove VirtIO network compile from sys/conf/files , but keep other VirtIO patches

# diff sys/conf/files sys/conf/files.bak.12-1-REL
3481,3483d3480
< dev/virtio/pci/virtio_pci_if.m          optional        virtio_pci
< dev/virtio/pci/virtio_pci_legacy.c      optional        virtio_pci
< dev/virtio/pci/virtio_pci_modern.c      optional        virtio_pci
3485a3483
> dev/virtio/network/if_vtnet.c		optional	vtnet

<<snip>>
...
dev/virtio/virtio.c                     optional        virtio
dev/virtio/virtqueue.c                  optional        virtio
dev/virtio/virtio_bus_if.m              optional        virtio
dev/virtio/virtio_if.m                  optional        virtio
dev/virtio/pci/virtio_pci.c             optional        virtio_pci
dev/virtio/pci/virtio_pci_if.m          optional        virtio_pci
dev/virtio/pci/virtio_pci_legacy.c      optional        virtio_pci
dev/virtio/pci/virtio_pci_modern.c      optional        virtio_pci
dev/virtio/mmio/virtio_mmio.c           optional        virtio_mmio fdt
dev/virtio/mmio/virtio_mmio_if.m        optional        virtio_mmio fdt
dev/virtio/block/virtio_blk.c           optional        virtio_blk
dev/virtio/balloon/virtio_balloon.c     optional        virtio_balloon
dev/virtio/scsi/virtio_scsi.c           optional        virtio_scsi
dev/virtio/random/virtio_random.c       optional        virtio_random
dev/virtio/console/virtio_console.c     optional        virtio_console
...
<<end snip>>

4. Replace VirtIO 12.1-RELEASE with updated VirtIO fixes from  but stop VirtIO networking

diff -q sys/dev/virtio sys/dev/virtio.bak.12-1-REL
Common subdirectories: sys/dev/virtio/balloon and sys/dev/virtio.bak.12-1-REL/balloon
Common subdirectories: sys/dev/virtio/block and sys/dev/virtio.bak.12-1-REL/block
Common subdirectories: sys/dev/virtio/console and sys/dev/virtio.bak.12-1-REL/console
Common subdirectories: sys/dev/virtio/mmio and sys/dev/virtio.bak.12-1-REL/mmio
Only in sys/dev/virtio.bak.12-1-REL: network
Only in sys/dev/virtio: network.stop
Common subdirectories: sys/dev/virtio/pci and sys/dev/virtio.bak.12-1-REL/pci
Common subdirectories: sys/dev/virtio/random and sys/dev/virtio.bak.12-1-REL/random
Common subdirectories: sys/dev/virtio/scsi and sys/dev/virtio.bak.12-1-REL/scsi
Files sys/dev/virtio/virtio.c and sys/dev/virtio.bak.12-1-REL/virtio.c differ
Files sys/dev/virtio/virtio.h and sys/dev/virtio.bak.12-1-REL/virtio.h differ
Files sys/dev/virtio/virtio_bus_if.m and sys/dev/virtio.bak.12-1-REL/virtio_bus_if.m differ
Files sys/dev/virtio/virtio_config.h and sys/dev/virtio.bak.12-1-REL/virtio_config.h differ
Only in sys/dev/virtio: virtio_endian.h
Files sys/dev/virtio/virtio_ids.h and sys/dev/virtio.bak.12-1-REL/virtio_ids.h differ
Files sys/dev/virtio/virtio_if.m and sys/dev/virtio.bak.12-1-REL/virtio_if.m differ
Files sys/dev/virtio/virtio_ring.h and sys/dev/virtio.bak.12-1-REL/virtio_ring.h differ
Files sys/dev/virtio/virtqueue.c and sys/dev/virtio.bak.12-1-REL/virtqueue.c differ
Only in sys/dev/virtio: virtqueue.c.ori
Files sys/dev/virtio/virtqueue.h and sys/dev/virtio.bak.12-1-REL/virtqueue.h differ

5. Replace 12.1-RELEASE VirtIO modules config files with VirtIO batch version, but also remove network from Makefile

# diff -q sys/modules/virtio sys/modules/virtio.bak.12-1-REL/
Files sys/modules/virtio/Makefile and sys/modules/virtio.bak.12-1-REL/Makefile differ
Common subdirectories: sys/modules/virtio/balloon and sys/modules/virtio.bak.12-1-REL/balloon
Common subdirectories: sys/modules/virtio/block and sys/modules/virtio.bak.12-1-REL/block
Common subdirectories: sys/modules/virtio/console and sys/modules/virtio.bak.12-1-REL/console
Only in sys/modules/virtio.bak.12-1-REL: network
Only in sys/modules/virtio: network.stop
Common subdirectories: sys/modules/virtio/pci and sys/modules/virtio.bak.12-1-REL/pci
Common subdirectories: sys/modules/virtio/random and sys/modules/virtio.bak.12-1-REL/random
Common subdirectories: sys/modules/virtio/scsi and sys/modules/virtio.bak.12-1-REL/scsi
Common subdirectories: sys/modules/virtio/virtio and sys/modules/virtio.bak.12-1-REL/virtio

# diff sys/modules/virtio/Makefile sys/modules/virtio.bak.12-1-REL/Makefile
2c2
< # $FreeBSD$
---
> # $FreeBSD: releng/12.1/sys/modules/virtio/Makefile 273515 2014-10-23 04:47:32Z bryanv $
26c26
< SUBDIR=	virtio pci block balloon scsi random console
---
> SUBDIR=	virtio pci network block balloon scsi random console

6. Use the other 12.1 netmap patches as is:

# diff sys/dev/netmap/netmap_kern.h sys/dev/netmap/netmap_kern.h.bak.12-1-REL 
79c79
< /* #define WITH_PTNETMAP */	/* ptnetmap guest support */
---
> #define WITH_PTNETMAP	/* ptnetmap guest support */


# diff sys/net/netmap_virt.h sys/net/netmap_virt.h.bak.12-1-REL
47,48c47,48
< #define PTNETMAP_PCI_DEVICE_ID          0xcccc  /* memory device */
< #define PTNETMAP_PCI_NETIF_ID           0xcccd  /* ptnet network interface */
---
> #define PTNETMAP_PCI_DEVICE_ID          0x000c  /* memory device */
> #define PTNETMAP_PCI_NETIF_ID           0x000d  /* ptnet network interface */


see (2) above for: sys/dev/netmap/if_ptnet.c

Are test results:

# ifconfig -a
vmx0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=e403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 52:54:00:a5:4c:f3
	media: Ethernet autoselect
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=81209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
	ether 52:54:00:cb:db:07
	inet 192.168.73.102 netmask 0xffffff00 broadcast 192.168.73.255
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

# pciconf -lcve
hostb0@pci0:0:0:0:	class=0x060000 card=0x11001af4 chip=0x29c08086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82G33/G31/P35/P31 Express DRAM Controller'
    class      = bridge
    subclass   = HOST-PCI
vgapci0@pci0:0:1:0:	class=0x030000 card=0x11001af4 chip=0x01001b36 rev=0x04 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'QXL paravirtual graphic card'
    class      = display
    subclass   = VGA
pcib1@pci0:0:2:0:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 000d[148] = ACS 1
pcib3@pci0:0:2:1:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 000d[148] = ACS 1
pcib4@pci0:0:2:2:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(empty) surprise Attn Button PC(off) EI(disengaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 000d[148] = ACS 1
pcib5@pci0:0:2:3:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 000d[148] = ACS 1
pcib6@pci0:0:2:4:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(empty) surprise Attn Button PC(off) EI(disengaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 000d[148] = ACS 1
pcib7@pci0:0:2:5:	class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
    cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
                 slot 0 power limit 0 mW HotPlug(empty) surprise Attn Button PC(off) EI(disengaged)
    cap 11[48] = MSI-X supports 1 message
                 Table in map 0x10[0x0], PBA in map 0x10[0x800]
    cap 0d[40] = PCI Bridge card=0x00001b36
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 000d[148] = ACS 1
uhci0@pci0:0:29:0:	class=0x0c0300 card=0x11001af4 chip=0x29348086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
uhci1@pci0:0:29:1:	class=0x0c0300 card=0x11001af4 chip=0x29358086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
uhci2@pci0:0:29:2:	class=0x0c0300 card=0x11001af4 chip=0x29368086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB UHCI Controller'
    class      = serial bus
    subclass   = USB
ehci0@pci0:0:29:7:	class=0x0c0320 card=0x11001af4 chip=0x293a8086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) USB2 EHCI Controller'
    class      = serial bus
    subclass   = USB
isab0@pci0:0:31:0:	class=0x060100 card=0x11001af4 chip=0x29188086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801IB (ICH9) LPC Interface Controller'
    class      = bridge
    subclass   = PCI-ISA
ahci0@pci0:0:31:2:	class=0x010601 card=0x11001af4 chip=0x29228086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]'
    class      = mass storage
    subclass   = SATA
    cap 05[80] = MSI supports 1 message, 64 bit enabled with 1 message
    cap 12[a8] = SATA Index-Data Pair
none0@pci0:0:31:3:	class=0x0c0500 card=0x11001af4 chip=0x29308086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) SMBus Controller'
    class      = serial bus
    subclass   = SMBus
pcib2@pci0:1:0:0:	class=0x060400 card=0x00000000 chip=0x000e1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    class      = bridge
    subclass   = PCI-PCI
    cap 05[8c] = MSI supports 1 message, 64 bit, vector masks 
    cap 01[84] = powerspec 3  supports D0 D3  current D0
    cap 10[48] = PCI-Express 2 PCI bridge max data 128(128) ARI disabled
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
    cap 0c[40] = unknown
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
vmx0@pci0:2:1:0:	class=0x020000 card=0x07b015ad chip=0x07b015ad rev=0x01 hdr=0x00
    vendor     = 'VMware'
    device     = 'VMXNET3 Ethernet Controller'
    class      = network
    subclass   = ethernet
    cap 11[9c] = MSI-X supports 25 messages, enabled
                 Table in map 0x18[0x0], PBA in map 0x18[0x1000]
    cap 05[84] = MSI supports 1 message, 64 bit 
em0@pci0:2:2:0:	class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82540EM Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet
vtpcim0@pci0:3:0:0:	class=0x010000 card=0x11001af4 chip=0x10481af4 rev=0x01 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'Virtio SCSI'
    class      = mass storage
    subclass   = SCSI
    cap 11[dc] = MSI-X supports 4 messages, enabled
                 Table in map 0x14[0x0], PBA in map 0x14[0x800]
    cap 09[c8] = vendor (length 20)
    cap 09[b4] = vendor (length 20)
    cap 09[a4] = vendor (length 16)
    cap 09[94] = vendor (length 16)
    cap 09[84] = vendor (length 16)
    cap 01[7c] = powerspec 3  supports D0 D3  current D0
    cap 10[40] = PCI-Express 2 endpoint max data 128(128)
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)
vtpcim1@pci0:5:0:0:	class=0x00ff00 card=0x11001af4 chip=0x10451af4 rev=0x01 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'Virtio memory balloon'
    class      = old
    cap 09[c8] = vendor (length 20)
    cap 09[b4] = vendor (length 20)
    cap 09[a4] = vendor (length 16)
    cap 09[94] = vendor (length 16)
    cap 09[84] = vendor (length 16)
    cap 01[7c] = powerspec 3  supports D0 D3  current D0
    cap 10[40] = PCI-Express 2 endpoint max data 128(128)
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)


ls /dev/da*
/dev/da0	/dev/da0p1	/dev/da0p2	/dev/da0p3

As I am using Q35 VirtIO SCSI rather then VirtIO Block.

I am still a bit puzzled why enabling VirtIO networking with netmap results in losing VirtIO Disk support but again it looks like there is some subtle interaction across VirtIO and netmap that needs to be looked at closer.

My testing is complete for the time being, as I have got degree of flexibility required:

Disk - SATA, SCSI (with VirtIO) and VirtIO
Networking - em, vmx and re

This is huge improvement from 11.3 & 12.x current situation.

Thanks to all for helping.

Cheers.

John Hartley
Comment 50 John Hartley 2020-01-19 08:58:37 UTC
(In reply to Tommy P from comment #43)

Hi Tomny P,

Teting: Q35, OVMF, VirtIO Storage, em / vmx / re with netmap networking

I believe I have not got repeatable way to get VirtIO (minus networking) and other general networking going on 12.1 .

I outlined process here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236922#c49

Which in summary:

1. Use new VirtIO update provide by your patch
2. Disable VirtIO network from dev/virtio and modules/virtio
3. Apply netmap updates, but extend scope of condition compilation in sys/dev/netmap/if_ptnet.c to as per this snippet:

<<snip>>
...
#include <dev/netmap/netmap_mem2.h>

#ifdef WITH_PTNETMAP
#include <dev/virtio/network/virtio_net.h>

#ifndef INET
#error "INET not defined, cannot support offloadings"
#endif
...
<<end snip>>

4. Remove vtnet driver from sys/amd64/conf/GENERIC & virtio network from sys/conf/files

5. Ensure you have right QEMU Q35 machine version.

There is a caveat. This build works on with Q35 v 3.1 not 4.0.

From: virsh dumpxml

<<snip>>
...
<type arch='x86_64' machine='pc-q35-3.1'>hvm</type>
...
<<end snip>>

So VirtIO is seeing some difference in behaviour. So that for:
- Q35-3.1 you get VirtIO Storage
- Q35-4.0 you do not get VirtIO Storage

Matts has also been seeing variation in netmap networking cross QEMU 3.1 and QEMU 4.1, so this machine variation across release is likely related:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241774#c69

Likely as per MattS case this is new bug and explains why people are getting different testing results.

Cheers,

John Hartley.
Comment 51 Tommy P 2020-01-20 06:03:57 UTC
(In reply to John Hartley from comment #47)

Hi John,

Before making the patch for disabling VirtIO + netmap interop, I thought of possible scenarios:

1) netmap w/o VirtIO
2) netmap + VirtIO interop
3) VirtIO w/o netmap

For 1 & 3, disabling in the kernel, IMO, is the best approach over all.  Thus, no need for any change in the code.  As for 2, netmap currently doesn't fully supports PCI-e.  Thus, the approach I took is what I think is best (kernel compile time, kernel size, least time spent of making a patch that works which was less than 5 minutes, easier future code adaptation to Vincenzo's changes of netmap).  This would allow both netmap and VirtIO to work in conjunction except netmap + VirtIO NIC.

As for QEMU 4.0+ versus older versions (comment #50), I think it may come down to compatibility matrix between FreeBSD vs QEMU versions.  I previously had problems booting older versions of FreeBSD when I specify CPU model Opterion_G4 or Opteron_G5 within QEMU/KVM configuration.  Now, I can run both 11 & 12 with Opteron_G5.  If you think as a developer/programmer and consider the SDLC, would you want to make something that's constantly changing or something that's stable?  For those reasons, I stopped using non LTS for Ubuntu a couple of years ago because the downstream needs time in accordance with SDLC to update for the changes.

Regards,
Tommy
Comment 52 John Hartley 2020-01-20 10:10:01 UTC
(In reply to Tommy P from comment #51)

Hi Tommy,

I have now done production update of one of my stranded 11.2 machines.

I used strategy:

0) Do freebsd-update -r 12.1-RELEASE upgrade / install followed by rebuild of kernel with "dev netmap" disabled in GENERIC.

The combination of slow update download (due to going from 11.2 -> 12.1) need to rebuild kernel and using postmaster to re-build ports meant the entire process took around 6 hours. This was for machine with: bind, mysql & apache. Which is about as complicated as my FreeBSD VMs get.

My other FreeBSD machines are dedicated to running single application: bind, apache/php or postgresql. The only one which is likely to problematic is the postgresql machine, as I don't want database to be unvailable for a long time waiting kernel to rebuild, so will handle this last.

Agree it is nice to have LTS stability, but I ended up where I am am due to needed some new features / fixes in QEMU / KVM hosting and related utilities. So looking forward to Ubuntu 20.04 which should be stable and meet my needs.

For FreeBSD Releases & QEMU / KVM Testing:

Having some repeatable test process in place to validate VirtIO against: 1 release back, current and 1 release forward to ensure that things keep working correctly.

I believe this and the networking QEMU / KVM thread has been very helpful in finding source of issues, fixes and work arounds.

Thanks you.


Cheers,


John Hartley.
Comment 53 Tommy P 2020-02-01 18:24:20 UTC
Created attachment 211258 [details]
VirtIO to support Q35 for FreeBSD 13-CURRENT

Based upon Bryan's virtio-rebase work.  Repackaged with patch for sys/conf/files.
Comment 54 Tommy P 2020-03-20 20:36:44 UTC
Created attachment 212556 [details]
Proper FreeBSD 12.x patch for VirtIO
Comment 55 Tommy P 2020-03-20 20:37:32 UTC
Created attachment 212558 [details]
Proper FreeBSD 11.x patch for VirtIO
Comment 56 Tommy P 2020-03-20 20:45:04 UTC
Created attachment 212559 [details]
Proper FreeBSD 12.x patch for VirtIO
Comment 57 Tommy P 2020-03-20 20:45:30 UTC
Created attachment 212560 [details]
Proper FreeBSD 11.x patch for VirtIO
Comment 58 John Hartley 2020-07-01 13:21:50 UTC
Hi Tommy P,

I have been doing some testing lately with 11.4 and noticed that fix for virtio does not appear to have flowed into 11.4 release.

Do you think this fix will come be part of 12.2 ?

Thanks,


John Hartley.
Comment 59 Eugene M. Zheganin 2020-09-22 11:13:47 UTC
I'm getting this on 12.2-BETA1, but not only on Q35 chipset, but in general. Furthermore, it's enough to only have just one virtio device - either network adapter or disk drive. With both FreeBSD freezes around some minutes of uptime, with one device it's able to live a couple of days.
Comment 60 Jeroen 2020-10-28 12:31:27 UTC
(In reply to John Hartley from comment #58)

This bug did not get referenced in the release notes for 12.2, https://www.freebsd.org/releases/12.2R/relnotes.html. Did it slip in under the radar?

If not, is there anything we can do to move this along?