Created attachment 203279 [details] various data collection for bug report Environment: Host OS: Ubuntu 18.04.2 LTS (updated) Libvirt: (libvirt-bin) 4.0.0-1ubuntu8.8 QEMU: 1:2.11+dfsg-1ubuntu7.12 QEMU-KVM: 1:2.11+dfsg-1ubuntu7.12 Guest OS: FreeBSD 12.0-RELEASE If the VM configuration uses i440FX chipset, everything works as expected. When configured to use Q35 chipset, virtio failed to load properly and any device configured to use virtio would not be detected. Attachments within zip of data gathered from the same VM: QEMU-KVM_freebsd_virtio_1.png - screen shot of the VM configuration as seen in virt-manager z_fbsd_q35.xml - Libvirt/QEMU XML configuration for the same VM seen in the screen shot pciconf_12.0_r341666_ori.txt - output of 'pciconf -lvce' of the original kernel r341666 used during install pciconf_12.0_r345757_custom.txt - output of 'pciconf -lvce' of the kernel r345757 built from updated src sysctl.hw_r345757_custom.txt - output of 'sysctl hw' of the kernel r345757 built from updated src Custom12 - kernel configuration (copy of GENERIC with unneeded drivers and configs removed) make.conf - make.conf used in building kernel r345757 src.conf - src.conf used in building kernel r345757. Buildworld was not done. Only buildkernel and installkernel. In the screen shot QEMU-KVM_freebsd_virtio_1.png, the "Controller SCSI 1" is added as SCSI controller having "Hypervisor default" model. FreeBSD detected and loaded the 'sym' SCSI driver. While the "Controller Virtio SCSI 0" is added as SCSI controller having "VirtIO SCSI" model. The custom kernel show the VirtIO PCI controller in pciconf but any virtio HDD attached does not show up in /dev. How to repeat: 1) Install Ubuntu 18.04.2 desktop to support virtualization using libvirt, QEMU-KVM, and virt-manager. 2) Use virt-manager to create/import a VM 2a) Select x86_64 architecture 2b) Make sure to check "Customize configuration before install" 3a) In the popup customization window, change the chipset from i440FX to Q35 in the overview. 3b) Select the HDD and change the "Disk bus" in "Advanced options" to VirtIO 3c) Select the NIC and change the "Device model" to virtio 4) Boot from imported HDD or from ISO media to install Utilizing the Q35 chipset, FreeBSD would neither boot properly nor install when both HDD and NIC are virtio. If the HDD is SATA type and NIC is other virtio, everything works.
PS: I forgot to mention that I did a diff of src/sys/dev/virtio of the original src during install (presumably r341666) against the updated src (r345757) via svnlite checkout of 12.0. diff reported no differences.
Can you try later versions of qemu (in particular 4.x.x-RC* series) to see if the issue is reproducible there. I am going to hazard a guess that the best first place to diagnose/isolate this issue is upstream with QEMU A complete verbose boot log might also prove handy. See Also: https://forums.unraid.net/topic/70123-pc-q35-211-issues/ - with report (and xml) of a working 2.11 configuration on unraid/pfSense (FreeBSD)
Created attachment 203281 [details] /var/log/messages file as requested with verbose logging during boot
(In reply to Kubilay Kocak from comment #2) Thank you for the feedback. I'm hesitant to try QEMU 4.0 since it's not officially supported on Ubuntu (even the current 18.10): https://packages.ubuntu.com/search?keywords=qemu Aside from that, it still doesn't explain why the original kernel (r341666) from the install ISO did not detect the VirtIO PCI controllers while the custom built updated from src (r345757) did detect the VirtIO PCI controllers when the diff comparison of src/sys/dev/virtio doesn't show any changes. If anything, I'd expect the custom built kernel not to detect any VirtIO PCI controllers like the original kernel of the install. Since the custom kernel did in fact the detect the VirtIO PCI controllers, it lead me to believe that the drivers are not working correctly. To confirm that the virtio drivers are loaded, I got this: root@fbsd12:~ # kldload virtio kldload: can't load virtio: module already loaded or in kernel yet: root@fbsd12:~ # kldstat Id Refs Address Size Name 1 10 0xffffffff80200000 1099a70 kernel 2 1 0xffffffff8129b000 372508 zfs.ko 3 2 0xffffffff8160e000 a2e8 opensolaris.ko when 'pciconf -lvce' shows the VirtIO PCI controllers. I've attached the /var/log/messages file with verbose logging on boot as requested. As side note, I was able to install just about every guest OS (Fedora, Ubuntu, and Windows 2008 R2 to Windows 2016) successfully utilizing the Q35 chipset and VirtIO HDD+NIC
I think I may found the bug... Please correct me if I'm wrong since it's been a very long time since I've dealt with C/C++ :( File src/sys/dev/virtio/pci/virtio_pci.h has this: /* VirtIO PCI vendor/device ID. */ #define VIRTIO_PCI_VENDORID 0x1AF4 #define VIRTIO_PCI_DEVICEID_MIN 0x1000 #define VIRTIO_PCI_DEVICEID_MAX 0x103F When the detected device ID is as follow for 12.0-RELEASE when booted with Q35 chipset: VirtIO SCSI: 0x1048 VirtIO console: 0x1043 VirtIO memory balloon: 0x1045 VirtIO network device: 0x1041 VirtIO block device: 0x1042 All those IDs are beyond the defined range. Thus, the controllers are detected since they match the vendor ID but the drivers are not loaded. As verification, I've modified the source code to: #define VIRTIO_PCI_DEVICEID_MAX 0x104F and rebuilt the kernel. The drivers still didn't load. Here's the pciconf for the SCSI when boot with i440FX chipset: virtio_pci2@pci0:0:7:0: class=0x010000 card=0x00081af4 chip=0x10041af4 rev=0x00 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'Virtio SCSI' class = mass storage subclass = SCSI cap 11[98] = MSI-X supports 4 messages, enabled Table in map 0x14[0x0], PBA in map 0x14[0x800] cap 09[84] = vendor (length 20) cap 09[70] = vendor (length 20) cap 09[60] = vendor (length 16) cap 09[50] = vendor (length 16) cap 09[40] = vendor (length 16) and when boot with Q35 chipset: none1@pci0:1:0:0: class=0x010000 card=0x11001af4 chip=0x10481af4 rev=0x01 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'Virtio SCSI' class = mass storage subclass = SCSI cap 11[dc] = MSI-X supports 4 messages Table in map 0x14[0x0], PBA in map 0x14[0x800] cap 09[c8] = vendor (length 20) cap 09[b4] = vendor (length 20) cap 09[a4] = vendor (length 16) cap 09[94] = vendor (length 16) cap 09[84] = vendor (length 16) cap 01[7c] = powerspec 3 supports D0 D3 current D0 cap 10[40] = PCI-Express 2 endpoint max data 128(128) link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) Aside from the device ID changes, looks like there's a revision (?) change from 0x00 to 0x01. Both match the Windows' SCSI INF driver definition: %RHELScsi.DeviceDesc% = rhelscsi_inst, PCI\VEN_1AF4&DEV_1004&SUBSYS_00081AF4&REV_00 %RHELScsi.DeviceDesc% = rhelscsi_inst, PCI\VEN_1AF4&DEV_1048&SUBSYS_11001AF4&REV_01 from the virtio driver 0.1.126.
Upon further investigation, I did a quick and dirty hack of the source code: pci/virtio_pci.c:247:/* if (pci_get_revid(dev) != VIRTIO_PCI_ABI_VERSION) */ pci/virtio_pci.c:248: if (pci_get_revid(dev) != VIRTIO_PCI_ABI_VERSION_MIN && pci/virtio_pci.c:249: pci_get_revid(dev) != VIRTIO_PCI_ABI_VERSION_MAX) pci/virtio_pci.h:47:/* #define VIRTIO_PCI_ABI_VERSION 0 */ pci/virtio_pci.h:48:#define VIRTIO_PCI_ABI_VERSION_MIN 0 pci/virtio_pci.h:49:#define VIRTIO_PCI_ABI_VERSION_MAX 1 and rebuilt the kernel. Upon reboot with the modified source codes of virtio_pci.* the virtio driver still didn't load but I got this in dmesg: virtio_pci0: <VirtIO PCI Unknown adapter> mem 0xfce00000-0xfce00fff,0xfea00000-0xfea03fff irq 22 at device 0.0 on pci1 virtio_pci0: cannot map I/O space device_attach: virtio_pci0 attach returned 6 pcib2: <PCI-PCI bridge> mem 0xfd013000-0xfd013fff irq 22 at device 2.1 on pci0 pci2: <PCI bus> on pcib2 virtio_pci0: <VirtIO PCI Unknown adapter> mem 0xfcc00000-0xfcc00fff,0xfe800000-0xfe803fff irq 22 at device 0.0 on pci2 virtio_pci0: cannot map I/O space device_attach: virtio_pci0 attach returned 6 pcib3: <PCI-PCI bridge> mem 0xfd014000-0xfd014fff irq 22 at device 2.2 on pci0 pci3: <PCI bus> on pcib3 virtio_pci0: <VirtIO PCI Unknown adapter> mem 0xfe600000-0xfe603fff irq 22 at device 0.0 on pci3 virtio_pci0: cannot map I/O space device_attach: virtio_pci0 attach returned 6 pcib4: <PCI-PCI bridge> mem 0xfd015000-0xfd015fff irq 22 at device 2.3 on pci0 pci4: <PCI bus> on pcib4 virtio_pci0: <VirtIO PCI Unknown adapter> mem 0xfc880000-0xfc880fff,0xfe400000-0xfe403fff irq 22 at device 0.0 on pci4 virtio_pci0: cannot map I/O space device_attach: virtio_pci0 attach returned 6 pcib5: <PCI-PCI bridge> mem 0xfd016000-0xfd016fff irq 22 at device 2.4 on pci0 pci5: <PCI bus> on pcib5 virtio_pci0: <VirtIO PCI Unknown adapter> mem 0xfc600000-0xfc600fff,0xfe200000-0xfe203fff irq 22 at device 0.0 on pci5 virtio_pci0: cannot map I/O space device_attach: virtio_pci0 attach returned 6 Looks like a lot more needed than my quick dirty hack and way beyond my knowledge of FreeBSD kernel :(.
After reviewing the Linux code for the virtio, I'm more confident that my hunch is correct regarding the FreeBSD virtio driver. if (pci_dev->device < 0x1040) { /* Transitional devices: use the PCI subsystem device id as * virtio device id, same as legacy driver always did. */ vp_dev->vdev.id.device = pci_dev->subsystem_device; } else { /* Modern devices: simply use PCI device id, but start from 0x1040. */ vp_dev->vdev.id.device = pci_dev->device - 0x1040; } from: https://github.com/torvalds/linux/blob/v5.0/drivers/virtio/virtio_pci_modern.c
The device IDs you are seeing are for the VirtIO V1 spec. FreeBSD only supports the pre-V1 (aka legacy) VirtIO spec. I suppose there was a QEMU change to use V1 with Q35 chipset. I have some rather dated work that adds V1 support but I hadn't had the time to commit: https://github.com/bryanv/freebsd/tree/virtio/
(In reply to Bryan Venteicher from comment #8) Hi Bryan, Thank you for the feedback. I've been digging further to learn more about the FreeBSD's internal while improving my C/C++ skills. I've found that there are 2 sets of device IDs: PCI (which FreeBSD currently support) and the PCIe (which I think your work would support). I'll give them a try a report back if I've found any issue. While doing a lot of trial and error, I've found something puzzling. If I build my custom kernel w/o debug, traces, and netmap (in addition to unnecessary drivers), pciconf shows correctly for the PCIe root: pcib1@pci0:0:2:0: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected However, when added back in the debug, traces, and netmap (still w/o the extra drivers) similar to the GENERIC kernel, that same controller becomes: none0@pci0:0:2:0: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI bar [10] = type Memory, range 32, base rxfd212000, size 4096, enabled bus range = 1-1 window[1c] = type I/O Port, range 16, addr 0xe000-0xdfff, disabled window[20] = type Memory, range 32, addr 0xfd000000-0xfd1fffff, enabled window[24] = type Prefetchable Memory, range 64, addr 0xfea00000-0xfebfffff, enabled cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected as seen from pciconf. Note how it's 'pcib1' w/o debug+traces and 'none0' with debug+traces. I don't know if the bug is with the VirtIO driver or the debug/trace. Perhaps its by design? This from src r345840.
(In reply to Bryan Venteicher from comment #8) Hi Bryan, I just downloaded the from git and tried to compile but receive this error: --- virtio_pci.o --- /usr/src12.0/sys/dev/virtio/pci/virtio_pci.c:54:10: fatal error: 'virtio_pci_if.h' file not found #include "virtio_pci_if.h" I've check git again and the file is missing. Thank you for your time.
virtio_pci_if.h is supposed to be generated as a part of the kernel build
(In reply to Bryan Venteicher from comment #11) Hi Brian, I've just built the kernel using the entire src tree from Git. The buildkernel was successful. Upon reboot, I think the Virtio PCI are detected properly (since the boot message scrolled by too fast) along with SATA and ZFS. However, the SATA drive having ZFS which FreeBSD is installed does not boot properly since it's unable to find the pool label: Solaris: NOTICE: Cannot find the pool label for 'fbsd12' Solaris: NOTICE: Cannot find the pool label for 'fbsd12' Solaris: NOTICE: Cannot find the pool label for 'fbsd12' ... even though the kernel detected ada0 device. Since it won't boot properly, I can't troubleshoot further. I had to revert back to the original install kernel or the old kernel to boot. I've also tried just using the Virtio drivers built from your src with the official kernel but the VirtIO drivers didn't work as before.
The kernel sources in that repo are rather old so perhaps your SATA issues is related to that. If you tried the modules built from that repo but with newer FreeBSD then that's probably not going to work. If you copy over sys/dev/virtio and sys/modules/virtio onto a newer FreeBSD tree, you'll probably have some compile errors to sort though but it shouldn't be anything major. Sorry, I have not had the time to rebase that repo yet though.
Sorry to wake this thread but I've also run into similar issues which I thought was a disk adapter handling problem in freeBSD+Q35 because I ran into freeNAS not handling all of the disks, then say it didn't pick up the other disk adapters Until I tried again with freeBSD 11.3 and 12.0 with more adapters and it didnt pick up the network adapter. Realised when I read this, it is more of adapter namespace exhaustion. I'll upload some dmesg from previous tries. Unfortunately I couldnt save the recent dmesg output.
Created attachment 206894 [details] dmesg output
The driver issues that we are seeing with KVM are impacting more than just VirtIO. Once you go to FreeBSD 11.3 or 12.X you cannot get any networking at all with KVM / QEMU Q35 + OVMF. I posted bug report here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241774 I believe that these are all related. We need to do a KVM / QEMU / FreeBSD driver compatibility matrix, as range of options keeps shrinking as FreeBSD releases roll forward. Regards, John.
Is there any way to work around this? I'm stuck on 11.2 due to this issue.
(In reply to John Hartley from comment #16) Thanks John for the feedback. Are you using the default GENERIC kernel? I didn't have any issues with my custom (trimmed of the GENERIC by removing unused drivers) kernel in conjunction with i440FX chipset. All of the VirtIO (block devices, NIC, memory balloon, etc) work OK. All of the SCSI controllers (Hypervisor default and VirtIO SCSI models) in addition VirtIO Serial work OK. It's when I created a VM configuration with Q35 chipset instead of i440FX that none of the VirtIO devices (including the VirtIO Serial & SCSI) are working as per my previous troubleshooting (comment #5 - #7). Thus, I don't think it's relating the OVMF firmware configuration within KVM-QEMU nor the FreeBSD version. I'm pretty sure it's how the VirtIO devices are detected and loaded as per FreeBSD's source code /usr/src/sys/dev/virtio/virtio_ids.h. If you compare the pciconf on a working i440FX VM system vs. a non working Q35 VM, you'll notice the matching 'card' ID to virtio_ids.h on the working i440FX VM while the non-working Q35 VM doesn't have any. For example: none2@pci0:2:0:0: class=0x010000 card=0x11001af4 chip=0x10481af4 rev=0x01 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'Virtio SCSI' class = mass storage subclass = SCSI Here are the images of the my config: https://imgur.com/a/rftzjO3 Devices that are loaded and not loaded (no driver attached) root@d-fbsd:~ # ls /dev/*da* /dev/ada0 /dev/ada1 /dev/ada2p1 /dev/ada4 /dev/ada5p1 /dev/da1 /dev/da2p1 /dev/da4 /dev/da5p1 /dev/da6p1 /dev/ada0p1 /dev/ada1p1 /dev/ada3 /dev/ada4p1 /dev/da0 /dev/da1p1 /dev/da3 /dev/da4p1 /dev/da5p2 /dev/da6p2 /dev/ada0p2 /dev/ada2 /dev/ada3p1 /dev/ada5 /dev/da0p1 /dev/da2 /dev/da3p1 /dev/da5 /dev/da6 root@d-fbsd:~ # dmesg | egrep -i 'scsi|mass stor|sym' pci2: <mass storage, SCSI> at device 0.0 (no driver attached) sym0: <895a> port 0xe000-0xe0ff mem 0xfca02000-0xfca023ff,0xfca00000-0xfca01fff irq 22 at device 0.0 on pci4 sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking sym1: <895a> port 0xd000-0xd0ff mem 0xfc802000-0xfc8023ff,0xfc800000-0xfc801fff irq 22 at device 0.0 on pci5 sym1: No NVRAM, ID 7, Fast-40, LVD, parity checking da0 at sym0 bus 0 scbus0 target 0 lun 0 da0: <QEMU QEMU HARDDISK 2.5+> Fixed Direct Access SPC-3 SCSI device pass13: da1 at sym0 bus 0 scbus0 target 1 lun 0 <QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device da1: <QEMU QEMU HARDDISK 2.5+> Fixed Direct Access SPC-3 SCSI device da2 at sym0 bus 0 scbus0 target 2 lun 0 pass13: 150.000MB/s transfers da2: <QEMU QEMU HARDDISK 2.5+> Fixed Direct Access SPC-3 SCSI device da3 at sym0 bus 0 scbus0 target 3 lun 0 da3: <QEMU QEMU HARDDISK 2.5+> Fixed Direct Access SPC-3 SCSI device da4 at sym0 bus 0 scbus0 target 4 lun 0 da4: <QEMU QEMU HARDDISK 2.5+> Fixed Direct Access SPC-3 SCSI device da5 at sym1 bus 0 scbus1 target 5 lun 0 da5: <QEMU QEMU HARDDISK 2.5+> Fixed Direct Access SPC-3 SCSI device da6 at sym1 bus 0 scbus1 target 6 lun 0 da6: <QEMU QEMU HARDDISK 2.5+> Fixed Direct Access SPC-3 SCSI device root@d-fbsd:~ # uname -a FreeBSD d-fbsd 11.2-RELEASE-p15 FreeBSD 11.2-RELEASE-p15 #0 r356353: Sat Jan 4 16:10:17 PST 2020 root@d-fbsd:/usr/obj/usr/src11.2/sys/Custom amd64 Regarding the matrix, from my troubleshooting, everything should work as long as you use the i440FX. If you decide to use the Q35, do not use anything relating to VirtIO (SCSI and Serial storages, NIC, etc). BTW, if you use shell scripts to maintain your VMs, simply changing the configuration of i440FX to Q35 only would still work since the existing VirtIO devices are attached to the PCI-ISA bus instead of PCIe-PCI (please see https://forums.freebsd.org/threads/freebsd-12-release-guest-in-qemu-kvm.70207/ and https://wiki.qemu.org/Features/Q35).
More specific VM's configuration: PCI-E bus: -device virtio-serial-pci,id=virtio-serial0,bus=pci.1,addr=0x0 PCI bus: -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 Note the bus=pci.1 vs bus=pci.0. If I'm correct, the pci.1 (PCI_ABI_VERSION) configure that device is connected to the PCI-E bus instead PCI (pci.0), which utilize different ID as per /usr/src/sys/dev/virtio/pci/virtio_pci.h: /* VirtIO PCI vendor/device ID. */ #define VIRTIO_PCI_VENDORID 0x1AF4 #define VIRTIO_PCI_DEVICEID_MIN 0x1000 #define VIRTIO_PCI_DEVICEID_MAX 0x103F /* VirtIO ABI version, this must match exactly. */ #define VIRTIO_PCI_ABI_VERSION 0
Hi Tommy P, thanks very much for response. Yes I am using GENERIC kernel, I did try a custom kernel removing all the unused devices but that also failed. Yes agree that issue relates to Q35. Since original posting I have done testing with Q35 with BIOS & OVMF Firmware and find that with 11.3 and 12.x that VirtIO based device all fail. Unfortunately with 11.3 & 12.x all available NIC device options (not just VirtIO) also fail (e1000, e1000-82545em, rtl8130 & vmxnet3). So issue extends beyond just VirtIO based devices. It seems that between 11.2 and 11.3 there where some changes make to PCI sub-system the result of which is that the PCIe (??) enumeration if failing to detect that NICs at all. Yesterday I tried to build a 11.3 kernel with 11.2 dev/pci code, but ended up pulling in more code than I wanted, so likely pulled in the associated buggy commits to get kernel complied. Your tip on swapping to i440FX is worth looking at I assumed that as I have built OVMF VMs that thee would only work with Q35 as OVMF option is tied to this. I will do more building and testing later today. For my next build I will only keep dev/pci/pcireg.h file from 11.3 and wind back to 11.2 any (non PCI code)flow through changes that this breaks as part of this test. I am focused on pci code as Q35 is PCIe native VM and the dev/pci source tree has a hold host of commit/reverts so hoping to at least be able to further isolate problem here. If I can get generic NICs enumerating then could fold in fix you have identified with VirtIO code and PCI identifiers... Thanks very much for feedback. Cheers, John Hartley.
(In reply to Kubilay Kocak from comment #2) Hi Kubilay, I have tried with Q35 v4 and v2.11 configured VMs. Always getting issue going from 11.2 to 11.3 or 12.x Cheers, John Hartley
(In reply to Bryan Venteicher from comment #13) Hi Bryan, I tried the method you've suggested to use the modules/virtio and dev/virtio from your repo on 12.0-RELEASE-p12 r356482 and encountered the following issues: 1) /usr/src/sys/dev/virtio/random/virtio_random.c:241:6: error: too many arguments to function call, expected 3, have 4 RANDOM_PURE_VIRTIO); ^~~~~~~~~~~~~~~~~~ I changed it from: random_harvest_queue(&value, sizeof(value), sizeof(value) * NBBY / 2, RANDOM_PURE_VIRTIO); to: random_harvest_queue(&value, sizeof(value), RANDOM_PURE_VIRTIO); 2) /usr/src/sys/dev/virtio/pci/virtio_pci.c:54:10: fatal error: 'virtio_pci_if.h' file not found I used sys/tools/makeobjops.awk to generate the missing virtio_pci_if.h from virtio_pci_if.m 3) A lot of "ld: error: undefined symbol" --- kernel --- linking kernel ld: error: undefined symbol: virtio_pci_disable_vq_desc >>> referenced by virtio_pci.c >>> virtio_pci.o:(vtpci_release_child_resources) They appear to all from virtio_pci_if.h 's "extern struct kobjop_desc " --------------------------------------------------------- I also compared the compile object files between your repo and the src r356482. I think there are some missing (virtio_pci_if, virtio_pci_legacy, virtio_pci_modern): root@d-build-fbsd12:/usr/obj # ll usr/src_virtio/amd64.amd64/sys/Custom/virt* -rw-r--r-- 1 root wheel 11312 Jan 8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio.o -rw-r--r-- 1 root wheel 12480 Jan 8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio_balloon.o -rw-r--r-- 1 root wheel 24608 Jan 8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio_blk.o -rw-r--r-- 1 root wheel 2232 Jan 8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio_bus_if.c -rw-r--r-- 1 root wheel 7531 Jan 8 06:09 usr/src_virtio/amd64.amd64/sys/Custom/virtio_bus_if.h -rw-r--r-- 1 root wheel 2720 Jan 8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio_bus_if.o -rw-r--r-- 1 root wheel 797 Jan 8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio_if.c -rw-r--r-- 1 root wheel 1226 Jan 8 06:09 usr/src_virtio/amd64.amd64/sys/Custom/virtio_if.h -rw-r--r-- 1 root wheel 1272 Jan 8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio_if.o -rw-r--r-- 1 root wheel 17632 Jan 8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio_pci.o -rw-r--r-- 1 root wheel 1283 Jan 8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio_pci_if.c -rw-r--r-- 1 root wheel 4088 Jan 8 06:09 usr/src_virtio/amd64.amd64/sys/Custom/virtio_pci_if.h -rw-r--r-- 1 root wheel 1792 Jan 8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio_pci_if.o -rw-r--r-- 1 root wheel 15312 Jan 8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio_pci_legacy.o -rw-r--r-- 1 root wheel 26664 Jan 8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio_pci_modern.o -rw-r--r-- 1 root wheel 41248 Jan 8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtio_scsi.o -rw-r--r-- 1 root wheel 8904 Jan 8 06:11 usr/src_virtio/amd64.amd64/sys/Custom/virtqueue.o root@d-build-fbsd12:/usr/obj # ll usr/src12.0/amd64.amd64/sys/Custom/virt* -rw-r--r-- 1 root wheel 11312 Jan 8 14:34 usr/src12.0/amd64.amd64/sys/Custom/virtio.o -rw-r--r-- 1 root wheel 12480 Jan 8 14:34 usr/src12.0/amd64.amd64/sys/Custom/virtio_balloon.o -rw-r--r-- 1 root wheel 24608 Jan 8 14:34 usr/src12.0/amd64.amd64/sys/Custom/virtio_blk.o -rw-r--r-- 1 root wheel 2229 Jan 8 14:34 usr/src12.0/amd64.amd64/sys/Custom/virtio_bus_if.c -rw-r--r-- 1 root wheel 7528 Jan 8 14:33 usr/src12.0/amd64.amd64/sys/Custom/virtio_bus_if.h -rw-r--r-- 1 root wheel 2720 Jan 8 14:34 usr/src12.0/amd64.amd64/sys/Custom/virtio_bus_if.o -rw-r--r-- 1 root wheel 794 Jan 8 14:34 usr/src12.0/amd64.amd64/sys/Custom/virtio_if.c -rw-r--r-- 1 root wheel 1223 Jan 8 14:33 usr/src12.0/amd64.amd64/sys/Custom/virtio_if.h -rw-r--r-- 1 root wheel 1272 Jan 8 14:34 usr/src12.0/amd64.amd64/sys/Custom/virtio_if.o -rw-r--r-- 1 root wheel 17632 Jan 8 14:34 usr/src12.0/amd64.amd64/sys/Custom/virtio_pci.o -rw-r--r-- 1 root wheel 41248 Jan 8 14:34 usr/src12.0/amd64.amd64/sys/Custom/virtio_scsi.o -rw-r--r-- 1 root wheel 8904 Jan 8 14:34 usr/src12.0/amd64.amd64/sys/Custom/virtqueue.o The module's virtio/pci/Makefile does include those source files but they didn't appear to be compiled: pci/Makefile:25: pci/Makefile:26:.PATH: ${SRCTOP}/sys/dev/virtio/pci pci/Makefile:27: pci/Makefile:28:KMOD= virtio_pci pci/Makefile:29:SRCS= virtio_pci.c virtio_pci_legacy.c virtio_pci_modern.c pci/Makefile:30:SRCS+= virtio_pci_if.c virtio_pci_if.h pci/Makefile:31:SRCS+= virtio_bus_if.h virtio_if.h pci/Makefile:32:SRCS+= bus_if.h device_if.h pci_if.h pci/Makefile:33: pci/Makefile:34:.include <bsd.kmod.mk> Do you have any suggestion to resolve #3 in order to proceed? Thanks.
Bryan, After some more debugging, I input some missing entries in sys/conf/files. The kernel was able to compile successfully with the VirtIO from your repo. After reboot, all VirtIO devices (SCSI, NIC, memory balloon, etc) are working for 12.0-RELEASE-p12 r356482M! Thanks again! As for those who want the VirtIO working in your Q35 environment, I'll attach the the needed stuff including my fix of the VirtIO's random.c so you don't need to download from Bryan's github repo (comment #8). I don't recommend running in production as it's not officially committed. Also note that this version of the VirtIO, supporting Q35, will break netmap which you'll need to disable in the kernel configuration prior the build process: /usr/src/sys/dev/netmap/if_vtnet_netmap.h:224:34: error: no member named 'vtntx_shrhdr' in 'struct vtnet_txq' err = sglist_append(sg, &txq->vtntx_shrhdr, sc->vtnet_hdr_size);
Created attachment 210551 [details] VirtIO /usr/src files to support Q35 How to use this attachment: 1) cd /usr/src 2) tar -xJvf <path/to>/virtio.xz 3) patch < sys_conf_files.patch 4) make kernel 5) reboot As previously mentioned, this will break kernel's netmap which you'll need to disable/comment in the configuration prior to the build process. Enjoy :)
TommyP, Thank you for your work on this! Can this patch be applied to 11.3, or can it be easily created?
Tommy, I'm in the process of rebasing the virtio branch to HEAD since it is ~!8 months old by now and hope to commit that work to HEAD in the next 1-2 months. You can try out the virtio-rebase branch from the same repo but I believe what is pushed there has only been compiled tested. Please contact me directly if you experience any issues in the meanwhile.
Created attachment 210556 [details] VirtIO + netmap to support Q35 Apply this patch in /usr/src (cd /usr/src && patch < sys_dev_virtio_netmap.patch). I currently don't have netmap configured within my environment to fully test VirtIO + netmap but a full compile of the GENERIC was successfull.
Tried both patches on 12.1-release - no virtio devices show up.
Created attachment 210660 [details] Errors Please ignore my previous comment, it worked. I also compiled with the 2nd patch + netmap enabled - netmap is visible in /dev/, however there seem to be some errors during boot, see the attachment.
(In reply to Dobri Dobrev from comment #29) Hi Dobri, Thanks for the feedback. Upon further investigation after submitting the netmap patch, it seems netmap breaks the VirtIO during probing of the devices in Q35, even when trying Bryan's sys/dev/netmap/if_vtnetmap.h. Everything works fine, including netmap, for i440FX. I'll see what I can do to get netmap + VirtIO for Q35. If it's beyond my understanding of the FreeBSD's networking, I'll have to refer to the FreeBSD gurus :). Regards, Tommy
Jan 12 02:11:23 fbsd12 kernel: vtpcil2: <VirtIO PCI (legacy) SCSI adapter> port 0xc180-0xc1bf mem 0xfc13b000-0xfc13bfff,0xfebf4000-0xfebf7fff irq 10 at device 9.0 on pci0 Jan 12 02:11:23 fbsd12 kernel: vtscsi0: <VirtIO SCSI Adapter> on vtpcil2 Jan 12 02:11:23 fbsd12 kernel: vtpcil3: <VirtIO PCI (legacy) Block adapter> port 0xc1c0-0xc1ff mem 0xfc13c000-0xfc13cfff,0xfebf8000-0xfebfbfff irq 10 at device 10.0 on pci0 Jan 12 02:11:23 fbsd12 kernel: vtblk0: <VirtIO Block Adapter> on vtpcil3 Jan 12 02:11:23 fbsd12 kernel: vtpcil4: <VirtIO PCI (legacy) Network adapter> port 0xc260-0xc27f mem 0xfc13d000-0xfc13dfff,0xfebfc000-0xfebfffff irq 11 at device 11.0 on pci0 Jan 12 02:11:23 fbsd12 kernel: vtnet0: <VirtIO Network Adapter> on vtpcil4 Jan 12 02:11:23 fbsd12 kernel: vtnet0: Ethernet address: 52:54:00:88:a5:32 Jan 12 02:11:23 fbsd12 kernel: 000.000854 [ 502] vtnet_netmap_attach max rings 1 Jan 12 02:11:23 fbsd12 kernel: vtnet0: netmap queues/slots: TX 1/1024, RX 1/1024 Jan 12 02:11:23 fbsd12 kernel: 000.000869 [ 507] vtnet_netmap_attach virtio attached txq=1, txd=1024 rxq=1, rxd=1024 Jan 12 02:11:23 fbsd12 kernel: ada0: <QEMU HARDDISK 2.5+> ATA-7 SATA device Jan 12 02:11:23 fbsd12 kernel: ada0: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) Jan 12 02:11:23 fbsd12 kernel: cd0: <QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device Jan 12 02:11:23 fbsd12 kernel: cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes) Jan 12 02:11:23 fbsd12 kernel: em0: link state changed to UP Jan 12 02:55:08 fbsd12 kernel: em0: <Intel(R) PRO/1000 Network Connection> port 0xc100-0xc13f mem 0xfc100000-0xfc11ffff irq 11 at device 3.0 on pci0 Jan 12 02:55:08 fbsd12 kernel: em0: attach_pre capping queues at 1 Jan 12 02:55:08 fbsd12 kernel: em0: using 1024 tx descriptors and 1024 rx descriptors Jan 12 02:55:08 fbsd12 kernel: em0: allocated for 1 tx_queues Jan 12 02:55:08 fbsd12 kernel: em0: allocated for 1 rx_queues Jan 12 02:55:08 fbsd12 kernel: em0: Ethernet address: 52:54:00:ea:98:ec Jan 12 02:55:08 fbsd12 kernel: em0: netmap queues/slots: TX 1/1024, RX 1/1024 Jan 12 02:55:08 fbsd12 kernel: vtpcil0: <VirtIO PCI (legacy) Console adapter> port 0xc140-0xc17f mem 0xfc138000-0xfc138fff,0xfebec000-0xfebeffff irq 10 at device 5.0 on pci0 Jan 12 02:55:08 fbsd12 kernel: vtpcil1: <VirtIO PCI (legacy) Balloon adapter> port 0xc220-0xc23f mem 0xfebf0000-0xfebf3fff irq 10 at device 6.0 on pci0 Jan 12 02:55:08 fbsd12 kernel: vtballoon0: <VirtIO Balloon Adapter> on vtpcil1 As you can see VirtIO + netmap works fine in legacy i440fx. Attaching this HDD to the Q35 VM breaks the VirtIO detection. If netmap is diabled in the kernel, VirtIO works fine for Q35.
(In reply to Tommy P from comment #31) So I should be able to reproduce this by creating a FreeBSD 12.x VM on virt-manager with Q35?
(In reply to Tommy P from comment #30) Hi Tommy T & Dobri, Vincenzo has found root cause and indicated that he will commit fix. See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241774#c35 So looks like we are making progress on both VirtIO & netmap front. Cheers, John Hartley.
Created attachment 210713 [details] VirtIO + netmap to support Q35 This netmap patch is a merge between official (r356535M) and Bryan's VirtIO work (not the virtio-rebase since it's based on 13-CURRENT). This patch works in legacy i440fx but fails in Q35 due to bug in netmap supporting PCI-e.
(In reply to Vincenzo Maffione from comment #32) Hi Vincenzo, Yes, I created the patch (comment #34) to produce the output seen in comment #31. My intended goal was the proposed patch should be applied to the legacy i440fx VM and verify that all configurations and applications work as before. Then simply removing the HDD and attaching it to a non OVMF Q35 VM for quick and easy migration. I left a few commented code as I'm still not very familiar with FreeBSD's networking, more so for the netmap (ie whether to merge vtnet_netmap_free_bufs and vtnet_free_used). Thanks, Tommy
(In reply to Tommy P from comment #24) Hi Tommy, now I have networking workaround (disable netmap), I finally did update of my 12.0 VM to 12.1 and applied the VirtIO patch. I successfully getting Virto devices in my PCI-e config: # pciconf -lcve hostb0@pci0:0:0:0: class=0x060000 card=0x11001af4 chip=0x29c08086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82G33/G31/P35/P31 Express DRAM Controller' class = bridge subclass = HOST-PCI vgapci0@pci0:0:1:0: class=0x030000 card=0x11001af4 chip=0x01001b36 rev=0x04 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'QXL paravirtual graphic card' class = display subclass = VGA pcib1@pci0:0:2:0: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 000d[148] = ACS 1 pcib3@pci0:0:2:1: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 000d[148] = ACS 1 pcib4@pci0:0:2:2: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 000d[148] = ACS 1 pcib5@pci0:0:2:3: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 000d[148] = ACS 1 pcib6@pci0:0:2:4: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 000d[148] = ACS 1 uhci0@pci0:0:29:0: class=0x0c0300 card=0x11001af4 chip=0x29348086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB UHCI Controller' class = serial bus subclass = USB uhci1@pci0:0:29:1: class=0x0c0300 card=0x11001af4 chip=0x29358086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB UHCI Controller' class = serial bus subclass = USB uhci2@pci0:0:29:2: class=0x0c0300 card=0x11001af4 chip=0x29368086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB UHCI Controller' class = serial bus subclass = USB ehci0@pci0:0:29:7: class=0x0c0320 card=0x11001af4 chip=0x293a8086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB2 EHCI Controller' class = serial bus subclass = USB isab0@pci0:0:31:0: class=0x060100 card=0x11001af4 chip=0x29188086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82801IB (ICH9) LPC Interface Controller' class = bridge subclass = PCI-ISA ahci0@pci0:0:31:2: class=0x010601 card=0x11001af4 chip=0x29228086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]' class = mass storage subclass = SATA cap 05[80] = MSI supports 1 message, 64 bit enabled with 1 message cap 12[a8] = SATA Index-Data Pair none0@pci0:0:31:3: class=0x0c0500 card=0x11001af4 chip=0x29308086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) SMBus Controller' class = serial bus subclass = SMBus pcib2@pci0:1:0:0: class=0x060400 card=0x00000000 chip=0x000e1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' class = bridge subclass = PCI-PCI cap 05[8c] = MSI supports 1 message, 64 bit, vector masks cap 01[84] = powerspec 3 supports D0 D3 current D0 cap 10[48] = PCI-Express 2 PCI bridge max data 128(128) ARI disabled link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) cap 0c[40] = unknown ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected vmx0@pci0:2:1:0: class=0x020000 card=0x07b015ad chip=0x07b015ad rev=0x01 hdr=0x00 vendor = 'VMware' device = 'VMXNET3 Ethernet Controller' class = network subclass = ethernet cap 11[9c] = MSI-X supports 25 messages, enabled Table in map 0x18[0x0], PBA in map 0x18[0x1000] cap 05[84] = MSI supports 1 message, 64 bit em0@pci0:2:2:0: class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82540EM Gigabit Ethernet Controller' class = network subclass = ethernet re0@pci0:2:3:0: class=0x020000 card=0x11001af4 chip=0x813910ec rev=0x20 hdr=0x00 vendor = 'Realtek Semiconductor Co., Ltd.' device = 'RTL-8100/8101L/8139 PCI Fast Ethernet Adapter' class = network subclass = ethernet vtpcim0@pci0:3:0:0: class=0x010000 card=0x11001af4 chip=0x10421af4 rev=0x01 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'Virtio block device' class = mass storage subclass = SCSI cap 11[dc] = MSI-X supports 2 messages, enabled Table in map 0x14[0x0], PBA in map 0x14[0x800] cap 09[c8] = vendor (length 20) cap 09[b4] = vendor (length 20) cap 09[a4] = vendor (length 16) cap 09[94] = vendor (length 16) cap 09[84] = vendor (length 16) cap 01[7c] = powerspec 3 supports D0 D3 current D0 cap 10[40] = PCI-Express 2 endpoint max data 128(128) link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) vtpcim1@pci0:4:0:0: class=0x010000 card=0x11001af4 chip=0x10481af4 rev=0x01 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'Virtio SCSI' class = mass storage subclass = SCSI cap 11[dc] = MSI-X supports 4 messages, enabled Table in map 0x14[0x0], PBA in map 0x14[0x800] cap 09[c8] = vendor (length 20) cap 09[b4] = vendor (length 20) cap 09[a4] = vendor (length 16) cap 09[94] = vendor (length 16) cap 09[84] = vendor (length 16) cap 01[7c] = powerspec 3 supports D0 D3 current D0 cap 10[40] = PCI-Express 2 endpoint max data 128(128) link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) vtpcim2@pci0:5:0:0: class=0x00ff00 card=0x11001af4 chip=0x10451af4 rev=0x01 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'Virtio memory balloon' class = old cap 09[c8] = vendor (length 20) cap 09[b4] = vendor (length 20) cap 09[a4] = vendor (length 16) cap 09[94] = vendor (length 16) cap 09[84] = vendor (length 16) cap 01[7c] = powerspec 3 supports D0 D3 current D0 cap 10[40] = PCI-Express 2 endpoint max data 128(128) link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) vtpcim3@pci0:6:0:0: class=0x010000 card=0x11001af4 chip=0x10421af4 rev=0x01 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'Virtio block device' class = mass storage subclass = SCSI cap 11[dc] = MSI-X supports 2 messages, enabled Table in map 0x14[0x0], PBA in map 0x14[0x800] cap 09[c8] = vendor (length 20) cap 09[b4] = vendor (length 20) cap 09[a4] = vendor (length 16) cap 09[94] = vendor (length 16) cap 09[84] = vendor (length 16) cap 01[7c] = powerspec 3 supports D0 D3 current D0 cap 10[40] = PCI-Express 2 endpoint max data 128(128) link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) I was able to easily flip my Q35 SATA based 12.1 to VirtIO SCSI by editing /etc/fstab and change SATA ada0p2, atap3 ... to SCSI da0p2, da01p3 ... Simillary if I wanted to flip to direct VirtIO (rather than via VirtIO SCSI) you need to edit your /etc/fstab mounts to use vtbd0p2, vtbd0p3 etc So happy to report that VirtIO patch worked for me with 12.1 rebuilt kernel. NOTE: the partition numbering here is due to using OVMF which results in FAT32 EFI partition always being on p1 hence root is on p2 in my case. Thanks again for hard work. Cheers, John Hartley.
(In reply to John Hartley from comment #36) Hi John, Thanks for the feedback. I use GPT ID for mounting in fstab. Thus, regardless of device type (SATA, SCSI, Virtio SCSI, or VirtIO Serial), it always boot without modifying the fstab provided that the storage controllers work as expected. To me, it's more efficient for testing. Regards, Tommy
Created attachment 210723 [details] Recent update in stable/12 for VirtIO r347233: Remove non-functional SCTP checksum offload support for virtio. Checksum offloading for SCTP is not currently specified for virtio. If the hypervisor announces checksum offloading support, it means TCP and UDP checksum offload. If an SCTP packet is sent and the host announced checksum offload support, the hypervisor inserts the IP checksum (16-bit) at the correct offset, but this is not the right checksum, which is a CRC32c. This results in all outgoing packets having the wrong checksum and therefore breaking SCTP based communications. This patch removes SCTP checksum offloading support from the virtio network interface. Thanks to Felix Weinrank for making me aware of the issue. Reviewed by: bryanv@ MFC after: 1 week https://svnweb.freebsd.org/base?view=revision&revision=347233 ------------------------------------------------------------- r349285: VirtIO SCSI: validate seg_max on attach Until r349278, bhyve presented a seg_max to the guest that was too large. Detect this case and clamp it to the virtqueue size. Otherwise, we would fail the "too many segments to enqueue" assertion in virtqueue_enqueue(). I hit this by running a guest with a MAXPHYS of 256 KB. Reviewed by: bryanv cem MFC after: 1 week Sponsored by: Dell EMC Isilon https://svnweb.freebsd.org/base?view=revision&revision=349285
Created attachment 210728 [details] VirtIO to support Q35 backported for FreeBSD 11.x -------------------------------------------------- Thank you Bryan for your help with getting past (CK_S)TAILQ_FOREACH compile errors! I greatly appreciate it! -------------------------------------------------- root@fbsd11-ovmf:~ # egrep -i 'sata|scsi|mass stor|virtio|sym0|em0|vtnet' /var/log/messages | grep 'Jan 13' Jan 13 22:10:55 fbsd11-ovmf kernel: vtpcim0: <VirtIO PCI (modern) Network adapter> mem 0x98e00000-0x98e00fff,0x800000000-0x800003fff irq 22 at device 0.0 on pci1 Jan 13 22:10:55 fbsd11-ovmf kernel: vtnet0: <VirtIO Network Adapter> on vtpcim0 Jan 13 22:10:55 fbsd11-ovmf kernel: vtnet0: Ethernet address: 52:54:00:a9:08:fb Jan 13 22:10:55 fbsd11-ovmf kernel: sym0: <895a> port 0xb000-0xb0ff mem 0x98c02000-0x98c023ff,0x98c00000-0x98c01fff irq 22 at device 0.0 on pci2 Jan 13 22:10:55 fbsd11-ovmf kernel: sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking Jan 13 22:10:55 fbsd11-ovmf kernel: vtpcim1: <VirtIO PCI (modern) SCSI adapter> mem 0x98a00000-0x98a00fff,0x800100000-0x800103fff irq 22 at device 0.0 on pci3 Jan 13 22:10:55 fbsd11-ovmf kernel: vtscsi0: <VirtIO SCSI Adapter> on vtpcim1 Jan 13 22:10:55 fbsd11-ovmf kernel: vtpcim2: <VirtIO PCI (modern) Console adapter> mem 0x98800000-0x98800fff,0x800200000-0x800203fff irq 22 at device 0.0 on pci4 Jan 13 22:10:55 fbsd11-ovmf kernel: vtpcim3: <VirtIO PCI (modern) Balloon adapter> mem 0x800300000-0x800303fff irq 22 at device 0.0 on pci5 Jan 13 22:10:55 fbsd11-ovmf kernel: vtballoon0: <VirtIO Balloon Adapter> on vtpcim3 Jan 13 22:10:55 fbsd11-ovmf kernel: vtpcim4: <VirtIO PCI (modern) Block adapter> mem 0x98400000-0x98400fff,0x800400000-0x800403fff irq 22 at device 0.0 on pci6 Jan 13 22:10:55 fbsd11-ovmf kernel: vtblk0: <VirtIO Block Adapter> on vtpcim4 Jan 13 22:10:55 fbsd11-ovmf kernel: em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x6000-0x603f mem 0x98000000-0x9801ffff irq 21 at device 1.0 on pci8 Jan 13 22:10:55 fbsd11-ovmf kernel: em0: Ethernet address: 52:54:00:4e:d4:8f Jan 13 22:10:55 fbsd11-ovmf kernel: ahci0: <Intel ICH9 AHCI SATA controller> port 0xc240-0xc25f mem 0x99002000-0x99002fff irq 16 at device 31.2 on pci0 Jan 13 22:10:55 fbsd11-ovmf kernel: ada0: <QEMU HARDDISK 2.5+> ATA-7 SATA device Jan 13 22:10:55 fbsd11-ovmf kernel: ada0: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) Jan 13 22:10:55 fbsd11-ovmf kernel: pass0: <QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device Jan 13 22:10:55 fbsd11-ovmf kernel: pass0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes) Jan 13 22:10:55 fbsd11-ovmf kernel: em0: link state changed to UP root@fbsd11-ovmf:~ # ls /dev/{ada,da,vtbd}* /dev/ada0 /dev/ada0p1 /dev/ada0p2 /dev/vtbd0 /dev/vtbd0p1 /dev/vtbd0p2 root@fbsd11-ovmf:~ # ifconfig vtnet0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6> ether 52:54:00:a9:08:fb hwaddr 52:54:00:a9:08:fb nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC> ether 52:54:00:4e:d4:8f hwaddr 52:54:00:4e:d4:8f inet ... netmask 0xffffff00 broadcast ...255 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> groups: lo root@fbsd11-ovmf:~ # uname -a FreeBSD fbsd11-ovmf 11.3-RELEASE-p5 FreeBSD 11.3-RELEASE-p5 #0 r356701M: Mon Jan 13 22:08:55 PST 2020 -------------------------------------------------- The above is from OVMF + Q35 VM. Should also work w/o OVMF. This all inclusive (netmap + updates in stable/12). 1) cd /usr/src 2) tar -xJvf <path/to/>virtio_11.xz 3) patch < sys_conf_files_v11.2.patch 4) make kernel 5) reboot I'll update the patches as soon as netmap is fixed for PCI-e bus. Please let me know if you have any issues.
Created attachment 210734 [details] VirtIO to support Q35 backported for FreeBSD 11.x Forgot to include Makefiles when making the tar package.
Created attachment 210737 [details] VirtIO to support Q35 for FreeBSD 12.x
(In reply to Tommy P from comment #41) Hi Tommy P & Brian V & Vicenzo, the VirtIO network update conflicts with Vincenzo's netmap fix. In my earlier test of VirtIO patch test I had just disable netmap via GENERIC conf file and not applied netmap fix. I have now tested against 12.1 where I first applied the netmap fix. Here is compile error: cc -target x86_64-unknown-freebsd12.1 --sysroot=/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin -c -O2 -pipe -fno-strict-aliasing -g -nostdinc -I. -I/usr/src/sys -I/usr/src/sys/contrib/ck/include -I/usr/src/sys/contrib/libfdt -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -MD -MF.depend.if_vtnet.o -MTif_vtnet.o -fdebug-prefix-map=./machine=/usr/src/sys/amd64/include -fdebug-prefix-map=./x86=/usr/src/sys/x86/include -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -gdwarf-2 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-address-of-packed-member -mno-aes -mno-avx -std=iso9899:1999 -Werror /usr/src/sys/dev/virtio/network/if_vtnet.c In file included from /usr/src/sys/dev/virtio/network/if_vtnet.c:351: /usr/src/sys/dev/netmap/if_vtnet_netmap.h:94:2: error: implicit declaration of function 'D' is invalid in C99 [-Werror,-Wimplicit-function-declaration] D("freed %d mbufs, %d netmap bufs on %d queues", ^ /usr/src/sys/dev/netmap/if_vtnet_netmap.h:94:2: error: this function declaration is not a prototype [-Werror,-Wstrict-prototypes] /usr/src/sys/dev/netmap/if_vtnet_netmap.h:297:3: error: implicit declaration of function 'ND' is invalid in C99 [-Werror,-Wimplicit-function-declaration] ND(3,"sent %d packets, hwcur %d", n, nm_i); ^ /usr/src/sys/dev/netmap/if_vtnet_netmap.h:297:3: error: this function declaration is not a prototype [-Werror,-Wstrict-prototypes] /usr/src/sys/dev/netmap/if_vtnet_netmap.h:306:7: error: implicit declaration of function 'ND' is invalid in C99 [-Werror,-Wimplicit-function-declaration] ND(5, "pure notify ? head %d tail %d nused %d %d", ^ /usr/src/sys/dev/netmap/if_vtnet_netmap.h:340:3: error: implicit declaration of function 'ND' is invalid in C99 [-Werror,-Wimplicit-function-declaration] ND(3, "disable intr, hwcur %d", nm_i); ^ /usr/src/sys/dev/netmap/if_vtnet_netmap.h:343:3: error: implicit declaration of function 'ND' is invalid in C99 [-Werror,-Wimplicit-function-declaration] ND(3, "enable intr, hwcur %d", nm_i); ^ /usr/src/sys/dev/netmap/if_vtnet_netmap.h:297:37: error: variable 'n' is uninitialized when used here [-Werror,-Wuninitialized] ND(3,"sent %d packets, hwcur %d", n, nm_i); ^ /usr/src/sys/dev/netmap/if_vtnet_netmap.h:245:9: note: initialize the variable 'n' to silence this warning u_int n; ^ = 0 /usr/src/sys/dev/netmap/if_vtnet_netmap.h:486:6: error: implicit declaration of function 'RD' is invalid in C99 [-Werror,-Wimplicit-function-declaration] RD(1, "Truncated virtio-net-header, " ^ /usr/src/sys/dev/netmap/if_vtnet_netmap.h:486:6: error: this function declaration is not a prototype [-Werror,-Wstrict-prototypes] /usr/src/sys/dev/netmap/if_vtnet_netmap.h:498:2: error: implicit declaration of function 'ND' is invalid in C99 [-Werror,-Wimplicit-function-declaration] ND("[B] h %d c %d hwcur %d hwtail %d", ^ /usr/src/sys/dev/netmap/if_vtnet_netmap.h:566:4: error: implicit declaration of function 'D' is invalid in C99 [-Werror,-Wimplicit-function-declaration] D("strange, null netmap ring %d", r); ^ /usr/src/sys/dev/netmap/if_vtnet_netmap.h:662:2: error: implicit declaration of function 'D' is invalid in C99 [-Werror,-Wimplicit-function-declaration] D("max rings %d", sc->vtnet_max_vq_pairs); ^ /usr/src/sys/dev/virtio/network/if_vtnet.c:3411:3: error: implicit declaration of function 'D' is invalid in C99 [-Werror,-Wimplicit-function-declaration] D("try to attach again"); ^ 14 errors generated. *** Error code 1 As sys/dev/netmap/if_vtnet_netmap.h was one of replaced files that was provided with fix, I put back the original 12.1 version (this was not changed or update by Vincenzo's netmap fix). The result was failure in compiling sys/dev/virtio/network/if_vtnet.c Here is compile result: cc -target x86_64-unknown-freebsd12.1 --sysroot=/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin -c -O2 -pipe -fno-strict-aliasing -g -nostdinc -I. -I/usr/src/sys -I/usr/src/sys/contrib/ck/include -I/usr/src/sys/contrib/libfdt -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -MD -MF.depend.if_vtnet.o -MTif_vtnet.o -fdebug-prefix-map=./machine=/usr/src/sys/amd64/include -fdebug-prefix-map=./x86=/usr/src/sys/x86/include -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -gdwarf-2 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-address-of-packed-member -mno-aes -mno-avx -std=iso9899:1999 -Werror /usr/src/sys/dev/virtio/network/if_vtnet.c /usr/src/sys/dev/virtio/network/if_vtnet.c:3219:6: error: implicit declaration of function 'vtnet_netmap_init_rx_buffers' is invalid in C99 [-Werror,-Wimplicit-function-declaration] if (vtnet_netmap_init_rx_buffers(sc)) ^ /usr/src/sys/dev/virtio/network/if_vtnet.c:3219:6: error: this function declaration is not a prototype [-Werror,-Wstrict-prototypes] /usr/src/sys/dev/virtio/network/if_vtnet.c:3411:3: error: implicit declaration of function 'D' is invalid in C99 [-Werror,-Wimplicit-function-declaration] D("try to attach again"); ^ /usr/src/sys/dev/virtio/network/if_vtnet.c:3411:3: error: this function declaration is not a prototype [-Werror,-Wstrict-prototypes] 4 errors generated. *** Error code 1 Stop. make[2]: stopped in /usr/obj/usr/src/amd64.amd64/sys/GENERIC *** Error code 1 Given that both Brian (virtio committer) and Vincenzo (netmap committer) have both been involved in helping to resolve VirtIO and netmap related issues, it would be great if we could get VirtIO and netmap fixes that worked together. Thanks, John Hartley.
Created attachment 210783 [details] 12.x Disable Virtio + netmap interop Jan 15 16:47:29 d-build-fbsd12 kernel: vtpcim0: <VirtIO PCI (modern) Network adapter> mem 0xfd480000-0xfd480fff,0xfea00000-0xfea03fff irq 22 at device 0.0 on pci1 Jan 15 16:47:29 d-build-fbsd12 kernel: vtnet0: <VirtIO Network Adapter> on vtpcim0 Jan 15 16:47:29 d-build-fbsd12 kernel: sym0: <895a> port 0xd000-0xd0ff mem 0xfd202000-0xfd2023ff,0xfd200000-0xfd201fff irq 22 at device 0.0 on pci2 Jan 15 16:47:29 d-build-fbsd12 kernel: sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking Jan 15 16:47:29 d-build-fbsd12 kernel: vtpcim1: <VirtIO PCI (modern) SCSI adapter> mem 0xfd000000-0xfd000fff,0xfe600000-0xfe603fff irq 22 at device 0.0 on pci3 Jan 15 16:47:29 d-build-fbsd12 kernel: vtscsi0: <VirtIO SCSI Adapter> on vtpcim1 Jan 15 16:47:29 d-build-fbsd12 kernel: vtpcim2: <VirtIO PCI (modern) Console adapter> mem 0xfce00000-0xfce00fff,0xfe400000-0xfe403fff irq 22 at device 0.0 on pci4 Jan 15 16:47:29 d-build-fbsd12 kernel: vtpcim3: <VirtIO PCI (modern) Balloon adapter> mem 0xfe200000-0xfe203fff irq 22 at device 0.0 on pci5 Jan 15 16:47:29 d-build-fbsd12 kernel: vtballoon0: <VirtIO Balloon Adapter> on vtpcim3 Jan 15 16:47:29 d-build-fbsd12 kernel: em0: <Intel(R) PRO/1000 Network Connection> port 0xc000-0xc03f mem 0xfc180000-0xfc19ffff irq 21 at device 1.0 on pci11 Jan 15 16:47:29 d-build-fbsd12 kernel: em0: Using 1024 TX descriptors and 1024 RX descriptors Jan 15 16:47:29 d-build-fbsd12 kernel: em0: Ethernet address: 52:54:00:7c:b9:29 Jan 15 16:47:29 d-build-fbsd12 kernel: em0: netmap queues/slots: TX 1/1024, RX 1/1024 Jan 15 16:47:29 d-build-fbsd12 kernel: ahci0: <Intel ICH9 AHCI SATA controller> port 0xe0c0-0xe0df mem 0xfd61c000-0xfd61cfff irq 16 at device 31.2 on pci0 Jan 15 16:47:29 d-build-fbsd12 kernel: ada0: <QEMU HARDDISK 2.5+> ATA-7 SATA device Jan 15 16:47:29 d-build-fbsd12 kernel: ada0: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) Jan 15 16:47:29 d-build-fbsd12 kernel: ada1: <QEMU HARDDISK 2.5+> ATA-7 SATA device Jan 15 16:47:29 d-build-fbsd12 kernel: ada1: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) Jan 15 16:47:29 d-build-fbsd12 kernel: ada2: <QEMU HARDDISK 2.5+> ATA-7 SATA device Jan 15 16:47:29 d-build-fbsd12 kernel: ada2: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) Jan 15 16:47:29 d-build-fbsd12 kernel: ada3: <QEMU HARDDISK 2.5+> ATA-7 SATA device Jan 15 16:47:29 d-build-fbsd12 kernel: ada3: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) Jan 15 16:47:29 d-build-fbsd12 kernel: em0: link state changed to UP uname -a FreeBSD d-build-fbsd12 12.1-RELEASE-p1 FreeBSD 12.1-RELEASE-p1 #0 r356482M:
(In reply to Tommy P from comment #43) Hi Tommy P, thanks again for continuing to work on this. I have applied a slight variation to what you posted, which did: #if defined(DEV_NETMAP) && defined(WITH_PTNETMAP) Conditional compilation rather than remove the netmap code. See diffs here: # diff sys/dev/virtio/network ~/virtio-12/sys/dev/virtio/network diff sys/dev/virtio/network/if_vtnet.c /home/jbh/virtio-12/sys/dev/virtio/network/if_vtnet.c 350c350 < #if defined(DEV_NETMAP) && defined(WITH_PTNETMAP) --- > #ifdef DEV_NETMAP 367c367 < #if defined(DEV_NETMAP) && defined(WITH_PTNETMAP) --- > #ifdef DEV_NETMAP 475c475 < #if defined(DEV_NETMAP) && defined(WITH_PTNETMAP) --- > #ifdef DEV_NETMAP 507c507 < #if defined(DEV_NETMAP) && defined(WITH_PTNETMAP) --- > #ifdef DEV_NETMAP 2031c2031 < #if defined(DEV_NETMAP) && defined(WITH_PTNETMAP) --- > #ifdef DEV_NETMAP 2766c2766 < #if defined(DEV_NETMAP) && defined(WITH_PTNETMAP) --- > #ifdef DEV_NETMAP 3070c3070 < #if defined(DEV_NETMAP) && defined(WITH_PTNETMAP) --- > #ifdef DEV_NETMAP 3218c3218 < #if defined(DEV_NETMAP) && defined(WITH_PTNETMAP) --- > #ifdef DEV_NETMAP 3409c3409 < #if defined(DEV_NETMAP) && defined(WITH_PTNETMAP) --- > #ifdef DEV_NETMAP diff sys/dev/virtio/network/if_vtnetvar.h /home/jbh/virtio-12/sys/dev/virtio/network/if_vtnetvar.h 85c85 < #if defined(DEV_NETMAP) & defined(WITH_PTNETMAP) --- > #ifdef DEV_NETMAP 124c124 < #if defined(DEV_NETMAP) && defined(WITH_PTNETMAP) This is just a suggestion as it allows for easy "re-enablement" of netmap in the event that Vincenzo re-adds "#define WITH_PTNETMAP" into sys/dev/netmap/netmap_kern.h netmap parameter file. In my testing I got clean compile across your, Brian's and Vincenzo's updates, but did not get any VirtIO devices showing up :-( . This is likely due to either: a. me having incorrectly apply update or other issue, due to all my add/removing and patching code b. some other QEMU / Q35 version issues (which MattS has reported on other defect thread). Cheers, John Hartley.
(In reply to John Hartley from comment #44) If you are adding checks on WITH_PTNETMAP, that's wrong. PTNETMAP has no relationships with if_vtnet. Look, the netmap fix will be committed to stable/12 (and stable/11) in short, so you can cherry-pick from there.
(In reply to John Hartley from comment #42) I don't understand what you are trying to do, but the D, ND and RD macros have been renamed in the following way: D --> nm_prinf (or nm_prerr) ND --> nm_prdis RD --> nm_prlim
(In reply to Vincenzo Maffione from comment #46) Hi Vincenzo, I was attempting to get machine with: Q35, OVMF, 12.1, VirtIO, all networking (em, vmx, re, vtnet). Thank for tip on ND, D & RD debug macros, that did get me past compile error but did not get me to working VirtIO. I wound out the defined(DEV_NETMAP) && defined(WITH_PTNETMAP) hack which was my amateur attempt to try to get this to work as Tommy's version removed all ifdef DEV_NETMAP surrounded compilation which seem a bit drastic to me. My final 12.1 virtio with netmap fix merge was: 1. Use patched netmap 2. Use patched virtio (with exception of network) 3. Use base virtio network 4. Update code to avoid netmap debug macro compile errors: D, RD & ND as per your guidance Against clean install the differences is over 3000 lines.... So huge effort on VirtIO by Tommy P and Brian V. I am doing to now wait potentially for the updates to flow through. Cheers, John Hartley.
Created attachment 210808 [details] Consolidated Differences across VirtIO / netmap Here is set of consolidated differences across VirtIO and netmap. This is not working set. Networking is working but not VirtIO.
(In reply to John Hartley from comment #47) Hi VirtIO'ers, I have not got: QEMU Q35 with OVMF, VirtIO SCSI (boot device), netmap + em & vmx. I had to sacrifice vtnet (VirtIO networking) as it inclusion seems to break the rest of VirtIO (if you have netmap). Here is are key differences: 1. Remove VirtIO networking from kernel diff sys/amd64/conf/GENERIC sys/amd64/conf/GENERIC.bak.12-1-REL 349c349 < # device vtnet # VirtIO Ethernet device --- > device vtnet # VirtIO Ethernet device 2. Change netmap if_net.c NETMAP patch so it essentially stops VirtIO code generation, by moving it up to stop #include <dev/virtio/network/virtio_net.h> # diff sys/dev/netmap/if_ptnet.c sys/dev/netmap/if_ptnet.c.bak.12-1-NETMAP 88,89d87 < < #ifdef WITH_PTNETMAP 91a90 > #ifdef WITH_PTNETMAP here is snippet of result: <<snip>> ... #include <dev/netmap/netmap_mem2.h> #ifdef WITH_PTNETMAP #include <dev/virtio/network/virtio_net.h> #ifndef INET #error "INET not defined, cannot support offloadings" #endif ... <<end snip>> NOTE: you could likely get same result by just remove compile from sys/conf/files 3. Remove VirtIO network compile from sys/conf/files , but keep other VirtIO patches # diff sys/conf/files sys/conf/files.bak.12-1-REL 3481,3483d3480 < dev/virtio/pci/virtio_pci_if.m optional virtio_pci < dev/virtio/pci/virtio_pci_legacy.c optional virtio_pci < dev/virtio/pci/virtio_pci_modern.c optional virtio_pci 3485a3483 > dev/virtio/network/if_vtnet.c optional vtnet <<snip>> ... dev/virtio/virtio.c optional virtio dev/virtio/virtqueue.c optional virtio dev/virtio/virtio_bus_if.m optional virtio dev/virtio/virtio_if.m optional virtio dev/virtio/pci/virtio_pci.c optional virtio_pci dev/virtio/pci/virtio_pci_if.m optional virtio_pci dev/virtio/pci/virtio_pci_legacy.c optional virtio_pci dev/virtio/pci/virtio_pci_modern.c optional virtio_pci dev/virtio/mmio/virtio_mmio.c optional virtio_mmio fdt dev/virtio/mmio/virtio_mmio_if.m optional virtio_mmio fdt dev/virtio/block/virtio_blk.c optional virtio_blk dev/virtio/balloon/virtio_balloon.c optional virtio_balloon dev/virtio/scsi/virtio_scsi.c optional virtio_scsi dev/virtio/random/virtio_random.c optional virtio_random dev/virtio/console/virtio_console.c optional virtio_console ... <<end snip>> 4. Replace VirtIO 12.1-RELEASE with updated VirtIO fixes from but stop VirtIO networking diff -q sys/dev/virtio sys/dev/virtio.bak.12-1-REL Common subdirectories: sys/dev/virtio/balloon and sys/dev/virtio.bak.12-1-REL/balloon Common subdirectories: sys/dev/virtio/block and sys/dev/virtio.bak.12-1-REL/block Common subdirectories: sys/dev/virtio/console and sys/dev/virtio.bak.12-1-REL/console Common subdirectories: sys/dev/virtio/mmio and sys/dev/virtio.bak.12-1-REL/mmio Only in sys/dev/virtio.bak.12-1-REL: network Only in sys/dev/virtio: network.stop Common subdirectories: sys/dev/virtio/pci and sys/dev/virtio.bak.12-1-REL/pci Common subdirectories: sys/dev/virtio/random and sys/dev/virtio.bak.12-1-REL/random Common subdirectories: sys/dev/virtio/scsi and sys/dev/virtio.bak.12-1-REL/scsi Files sys/dev/virtio/virtio.c and sys/dev/virtio.bak.12-1-REL/virtio.c differ Files sys/dev/virtio/virtio.h and sys/dev/virtio.bak.12-1-REL/virtio.h differ Files sys/dev/virtio/virtio_bus_if.m and sys/dev/virtio.bak.12-1-REL/virtio_bus_if.m differ Files sys/dev/virtio/virtio_config.h and sys/dev/virtio.bak.12-1-REL/virtio_config.h differ Only in sys/dev/virtio: virtio_endian.h Files sys/dev/virtio/virtio_ids.h and sys/dev/virtio.bak.12-1-REL/virtio_ids.h differ Files sys/dev/virtio/virtio_if.m and sys/dev/virtio.bak.12-1-REL/virtio_if.m differ Files sys/dev/virtio/virtio_ring.h and sys/dev/virtio.bak.12-1-REL/virtio_ring.h differ Files sys/dev/virtio/virtqueue.c and sys/dev/virtio.bak.12-1-REL/virtqueue.c differ Only in sys/dev/virtio: virtqueue.c.ori Files sys/dev/virtio/virtqueue.h and sys/dev/virtio.bak.12-1-REL/virtqueue.h differ 5. Replace 12.1-RELEASE VirtIO modules config files with VirtIO batch version, but also remove network from Makefile # diff -q sys/modules/virtio sys/modules/virtio.bak.12-1-REL/ Files sys/modules/virtio/Makefile and sys/modules/virtio.bak.12-1-REL/Makefile differ Common subdirectories: sys/modules/virtio/balloon and sys/modules/virtio.bak.12-1-REL/balloon Common subdirectories: sys/modules/virtio/block and sys/modules/virtio.bak.12-1-REL/block Common subdirectories: sys/modules/virtio/console and sys/modules/virtio.bak.12-1-REL/console Only in sys/modules/virtio.bak.12-1-REL: network Only in sys/modules/virtio: network.stop Common subdirectories: sys/modules/virtio/pci and sys/modules/virtio.bak.12-1-REL/pci Common subdirectories: sys/modules/virtio/random and sys/modules/virtio.bak.12-1-REL/random Common subdirectories: sys/modules/virtio/scsi and sys/modules/virtio.bak.12-1-REL/scsi Common subdirectories: sys/modules/virtio/virtio and sys/modules/virtio.bak.12-1-REL/virtio # diff sys/modules/virtio/Makefile sys/modules/virtio.bak.12-1-REL/Makefile 2c2 < # $FreeBSD$ --- > # $FreeBSD: releng/12.1/sys/modules/virtio/Makefile 273515 2014-10-23 04:47:32Z bryanv $ 26c26 < SUBDIR= virtio pci block balloon scsi random console --- > SUBDIR= virtio pci network block balloon scsi random console 6. Use the other 12.1 netmap patches as is: # diff sys/dev/netmap/netmap_kern.h sys/dev/netmap/netmap_kern.h.bak.12-1-REL 79c79 < /* #define WITH_PTNETMAP */ /* ptnetmap guest support */ --- > #define WITH_PTNETMAP /* ptnetmap guest support */ # diff sys/net/netmap_virt.h sys/net/netmap_virt.h.bak.12-1-REL 47,48c47,48 < #define PTNETMAP_PCI_DEVICE_ID 0xcccc /* memory device */ < #define PTNETMAP_PCI_NETIF_ID 0xcccd /* ptnet network interface */ --- > #define PTNETMAP_PCI_DEVICE_ID 0x000c /* memory device */ > #define PTNETMAP_PCI_NETIF_ID 0x000d /* ptnet network interface */ see (2) above for: sys/dev/netmap/if_ptnet.c Are test results: # ifconfig -a vmx0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=e403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 52:54:00:a5:4c:f3 media: Ethernet autoselect status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=81209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER> ether 52:54:00:cb:db:07 inet 192.168.73.102 netmask 0xffffff00 broadcast 192.168.73.255 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> # pciconf -lcve hostb0@pci0:0:0:0: class=0x060000 card=0x11001af4 chip=0x29c08086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82G33/G31/P35/P31 Express DRAM Controller' class = bridge subclass = HOST-PCI vgapci0@pci0:0:1:0: class=0x030000 card=0x11001af4 chip=0x01001b36 rev=0x04 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'QXL paravirtual graphic card' class = display subclass = VGA pcib1@pci0:0:2:0: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 000d[148] = ACS 1 pcib3@pci0:0:2:1: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 000d[148] = ACS 1 pcib4@pci0:0:2:2: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(empty) surprise Attn Button PC(off) EI(disengaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 000d[148] = ACS 1 pcib5@pci0:0:2:3: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(present) surprise Attn Button PC(on) EI(disengaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 000d[148] = ACS 1 pcib6@pci0:0:2:4: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(empty) surprise Attn Button PC(off) EI(disengaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 000d[148] = ACS 1 pcib7@pci0:0:2:5: class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' device = 'QEMU PCIe Root port' class = bridge subclass = PCI-PCI cap 10[54] = PCI-Express 2 root port max data 128(128) ARI disabled link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) slot 0 power limit 0 mW HotPlug(empty) surprise Attn Button PC(off) EI(disengaged) cap 11[48] = MSI-X supports 1 message Table in map 0x10[0x0], PBA in map 0x10[0x800] cap 0d[40] = PCI Bridge card=0x00001b36 ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 000d[148] = ACS 1 uhci0@pci0:0:29:0: class=0x0c0300 card=0x11001af4 chip=0x29348086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB UHCI Controller' class = serial bus subclass = USB uhci1@pci0:0:29:1: class=0x0c0300 card=0x11001af4 chip=0x29358086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB UHCI Controller' class = serial bus subclass = USB uhci2@pci0:0:29:2: class=0x0c0300 card=0x11001af4 chip=0x29368086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB UHCI Controller' class = serial bus subclass = USB ehci0@pci0:0:29:7: class=0x0c0320 card=0x11001af4 chip=0x293a8086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) USB2 EHCI Controller' class = serial bus subclass = USB isab0@pci0:0:31:0: class=0x060100 card=0x11001af4 chip=0x29188086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82801IB (ICH9) LPC Interface Controller' class = bridge subclass = PCI-ISA ahci0@pci0:0:31:2: class=0x010601 card=0x11001af4 chip=0x29228086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]' class = mass storage subclass = SATA cap 05[80] = MSI supports 1 message, 64 bit enabled with 1 message cap 12[a8] = SATA Index-Data Pair none0@pci0:0:31:3: class=0x0c0500 card=0x11001af4 chip=0x29308086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82801I (ICH9 Family) SMBus Controller' class = serial bus subclass = SMBus pcib2@pci0:1:0:0: class=0x060400 card=0x00000000 chip=0x000e1b36 rev=0x00 hdr=0x01 vendor = 'Red Hat, Inc.' class = bridge subclass = PCI-PCI cap 05[8c] = MSI supports 1 message, 64 bit, vector masks cap 01[84] = powerspec 3 supports D0 D3 current D0 cap 10[48] = PCI-Express 2 PCI bridge max data 128(128) ARI disabled link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) cap 0c[40] = unknown ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected vmx0@pci0:2:1:0: class=0x020000 card=0x07b015ad chip=0x07b015ad rev=0x01 hdr=0x00 vendor = 'VMware' device = 'VMXNET3 Ethernet Controller' class = network subclass = ethernet cap 11[9c] = MSI-X supports 25 messages, enabled Table in map 0x18[0x0], PBA in map 0x18[0x1000] cap 05[84] = MSI supports 1 message, 64 bit em0@pci0:2:2:0: class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82540EM Gigabit Ethernet Controller' class = network subclass = ethernet vtpcim0@pci0:3:0:0: class=0x010000 card=0x11001af4 chip=0x10481af4 rev=0x01 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'Virtio SCSI' class = mass storage subclass = SCSI cap 11[dc] = MSI-X supports 4 messages, enabled Table in map 0x14[0x0], PBA in map 0x14[0x800] cap 09[c8] = vendor (length 20) cap 09[b4] = vendor (length 20) cap 09[a4] = vendor (length 16) cap 09[94] = vendor (length 16) cap 09[84] = vendor (length 16) cap 01[7c] = powerspec 3 supports D0 D3 current D0 cap 10[40] = PCI-Express 2 endpoint max data 128(128) link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) vtpcim1@pci0:5:0:0: class=0x00ff00 card=0x11001af4 chip=0x10451af4 rev=0x01 hdr=0x00 vendor = 'Red Hat, Inc.' device = 'Virtio memory balloon' class = old cap 09[c8] = vendor (length 20) cap 09[b4] = vendor (length 20) cap 09[a4] = vendor (length 16) cap 09[94] = vendor (length 16) cap 09[84] = vendor (length 16) cap 01[7c] = powerspec 3 supports D0 D3 current D0 cap 10[40] = PCI-Express 2 endpoint max data 128(128) link x1(x1) speed 2.5(2.5) ASPM disabled(L0s) ls /dev/da* /dev/da0 /dev/da0p1 /dev/da0p2 /dev/da0p3 As I am using Q35 VirtIO SCSI rather then VirtIO Block. I am still a bit puzzled why enabling VirtIO networking with netmap results in losing VirtIO Disk support but again it looks like there is some subtle interaction across VirtIO and netmap that needs to be looked at closer. My testing is complete for the time being, as I have got degree of flexibility required: Disk - SATA, SCSI (with VirtIO) and VirtIO Networking - em, vmx and re This is huge improvement from 11.3 & 12.x current situation. Thanks to all for helping. Cheers. John Hartley
(In reply to Tommy P from comment #43) Hi Tomny P, Teting: Q35, OVMF, VirtIO Storage, em / vmx / re with netmap networking I believe I have not got repeatable way to get VirtIO (minus networking) and other general networking going on 12.1 . I outlined process here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236922#c49 Which in summary: 1. Use new VirtIO update provide by your patch 2. Disable VirtIO network from dev/virtio and modules/virtio 3. Apply netmap updates, but extend scope of condition compilation in sys/dev/netmap/if_ptnet.c to as per this snippet: <<snip>> ... #include <dev/netmap/netmap_mem2.h> #ifdef WITH_PTNETMAP #include <dev/virtio/network/virtio_net.h> #ifndef INET #error "INET not defined, cannot support offloadings" #endif ... <<end snip>> 4. Remove vtnet driver from sys/amd64/conf/GENERIC & virtio network from sys/conf/files 5. Ensure you have right QEMU Q35 machine version. There is a caveat. This build works on with Q35 v 3.1 not 4.0. From: virsh dumpxml <<snip>> ... <type arch='x86_64' machine='pc-q35-3.1'>hvm</type> ... <<end snip>> So VirtIO is seeing some difference in behaviour. So that for: - Q35-3.1 you get VirtIO Storage - Q35-4.0 you do not get VirtIO Storage Matts has also been seeing variation in netmap networking cross QEMU 3.1 and QEMU 4.1, so this machine variation across release is likely related: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241774#c69 Likely as per MattS case this is new bug and explains why people are getting different testing results. Cheers, John Hartley.
(In reply to John Hartley from comment #47) Hi John, Before making the patch for disabling VirtIO + netmap interop, I thought of possible scenarios: 1) netmap w/o VirtIO 2) netmap + VirtIO interop 3) VirtIO w/o netmap For 1 & 3, disabling in the kernel, IMO, is the best approach over all. Thus, no need for any change in the code. As for 2, netmap currently doesn't fully supports PCI-e. Thus, the approach I took is what I think is best (kernel compile time, kernel size, least time spent of making a patch that works which was less than 5 minutes, easier future code adaptation to Vincenzo's changes of netmap). This would allow both netmap and VirtIO to work in conjunction except netmap + VirtIO NIC. As for QEMU 4.0+ versus older versions (comment #50), I think it may come down to compatibility matrix between FreeBSD vs QEMU versions. I previously had problems booting older versions of FreeBSD when I specify CPU model Opterion_G4 or Opteron_G5 within QEMU/KVM configuration. Now, I can run both 11 & 12 with Opteron_G5. If you think as a developer/programmer and consider the SDLC, would you want to make something that's constantly changing or something that's stable? For those reasons, I stopped using non LTS for Ubuntu a couple of years ago because the downstream needs time in accordance with SDLC to update for the changes. Regards, Tommy
(In reply to Tommy P from comment #51) Hi Tommy, I have now done production update of one of my stranded 11.2 machines. I used strategy: 0) Do freebsd-update -r 12.1-RELEASE upgrade / install followed by rebuild of kernel with "dev netmap" disabled in GENERIC. The combination of slow update download (due to going from 11.2 -> 12.1) need to rebuild kernel and using postmaster to re-build ports meant the entire process took around 6 hours. This was for machine with: bind, mysql & apache. Which is about as complicated as my FreeBSD VMs get. My other FreeBSD machines are dedicated to running single application: bind, apache/php or postgresql. The only one which is likely to problematic is the postgresql machine, as I don't want database to be unvailable for a long time waiting kernel to rebuild, so will handle this last. Agree it is nice to have LTS stability, but I ended up where I am am due to needed some new features / fixes in QEMU / KVM hosting and related utilities. So looking forward to Ubuntu 20.04 which should be stable and meet my needs. For FreeBSD Releases & QEMU / KVM Testing: Having some repeatable test process in place to validate VirtIO against: 1 release back, current and 1 release forward to ensure that things keep working correctly. I believe this and the networking QEMU / KVM thread has been very helpful in finding source of issues, fixes and work arounds. Thanks you. Cheers, John Hartley.
Created attachment 211258 [details] VirtIO to support Q35 for FreeBSD 13-CURRENT Based upon Bryan's virtio-rebase work. Repackaged with patch for sys/conf/files.
Created attachment 212556 [details] Proper FreeBSD 12.x patch for VirtIO
Created attachment 212558 [details] Proper FreeBSD 11.x patch for VirtIO
Created attachment 212559 [details] Proper FreeBSD 12.x patch for VirtIO
Created attachment 212560 [details] Proper FreeBSD 11.x patch for VirtIO
Hi Tommy P, I have been doing some testing lately with 11.4 and noticed that fix for virtio does not appear to have flowed into 11.4 release. Do you think this fix will come be part of 12.2 ? Thanks, John Hartley.
I'm getting this on 12.2-BETA1, but not only on Q35 chipset, but in general. Furthermore, it's enough to only have just one virtio device - either network adapter or disk drive. With both FreeBSD freezes around some minutes of uptime, with one device it's able to live a couple of days.
(In reply to John Hartley from comment #58) This bug did not get referenced in the release notes for 12.2, https://www.freebsd.org/releases/12.2R/relnotes.html. Did it slip in under the radar? If not, is there anything we can do to move this along?
A workaround to use the virtio devices on the QEMU Q35 chipset seems to be to connect the virtio devices to a PCI bus (and not under PCIe). This causes QEMU to attach the older devices, for which FreeBSD has drivers.
I would also like to know how to get this fixed, permanently.
(In reply to Ryan Libby from comment #61) Suggestions for how to do that would be appreciated. I can't find any obvious means in ovirt to do this. As a workaround I've been using 11.4 so far, but I'd like to be able to upgrade before support runs out.
(In reply to Jeroen from comment #63) To be honest I don't know much about ovirt. What I was able to do with libvirt driving qemu was either explicitly set the virtio device type as "virtio-transitional" and plug them into PCIe slots, or plug them into legacy PCI slots. I did this by editing libvirt xml files. If ovirt is also using libvirt to drive qemu, perhaps something similar will work. Can you first try seeing if there is a way to edit the vm devices to change the model types from virtio to virtio-transitional?
I recently committed V1 support to current and that will be included in 13. I want to wait for that to settle (and see what the release schedule for 12 is) before doing an MFC.
i just want to take a second to thank you for your hard work. i recently upgraded my pkgbase build server: https://alpha.pkgbase.live/ to include your changes, and then scaled them up on Hetzner to the slightly more expensive variant that have double the power, but use newer qemu version, and previously didn't work. now it works perfectly, it's blazing fast, and tonight will be updated to stable to be faster still!
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/doc/commit/?id=b0a8663bb4efc2c4ada9150835fa8e370e7fb562 commit b0a8663bb4efc2c4ada9150835fa8e370e7fb562 Author: Mina GaliÄ <me@igalic.co> AuthorDate: 2021-03-05 22:31:20 +0000 Commit: Daniel Ebdrup Jensen <debdrup@FreeBSD.org> CommitDate: 2021-03-05 22:31:42 +0000 add a release note about VirtIO v1 fixes This explains that we can now run on Q35 chipsets, and that the PR below has been fixed. PR: 236922 Reviewed by: gjb Differential Revision: https://reviews.freebsd.org/D29003 website/content/en/releases/13.0R/relnotes.adoc | 3 +++ 1 file changed, 3 insertions(+)
MARKED AS SPAM
(In reply to Anna Jane from comment #68) Did you mean to put those extra links to a website in your bug report? As well, your Bugzilla name doesn't match the name you use to sign your report with. Is there something wrong with your clipboard?
Or your browser actually. Is some malware inserting/replacing stuff when you submit a form?
(In reply to Eric Joyner from comment #70) It looks like the comment you are referring to was created by a bot our of existing comments...
I recently tried to install 13.0-RELEASE and 13.0-STABLE (20210701) on a kvm host, and it hangs the same way described above with q35, but works with i440-fx I don't have access to the host (I'm a VM customer) I can get a prompt from the q35 instance by setting in the loader via the boot menu: set hw.ata.ata_dma=0 set hw.ata.atapi_dma=0 However, things are weird (The "Install" menu button doesn't do anything, but "Shell" does) And "reboot" completely hangs after saying all disks synced. I've attached kenv output. Anything else that would help? Cheers, Jamie
Created attachment 226337 [details] kenv output from both instances
I just ran into this incompatibility with the newer QEMU-KVM machine_type (pc-q35-4.0 in my case) and no disk to be found on boot with latest pfSense release 2.5.2 (https://docs.netgate.com/pfsense/en/latest/releases/2-5-2.html). when switching to the older pc-i440fx-3.1 type the same machine boots up just fine. This is unfortunate as usually one has no influence on the virtualized hardware when i.e. using a hosted service / Cloud provider to run the VMs.
Comment on attachment 226337 [details] kenv output from both instances Please ignore my post. My issue was related to the virtio_random module which had also been removed by my host along with the pc-type change. Details of that are here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253175 Sorry for the noise. (In reply to Christian Rohmann from comment #74) Christian, may your issue be the same as shown in the bug I just mentioned?
(In reply to Bryan Venteicher from comment #65) I hit the same problem with virtio-scsi using VirtualBox in 12.2 but it seems fixed in 13.0. Will this fix be moved to 12-STABLE for the next 12.3?
(In reply to Ryan Libby from comment #64) Just wanted to confirm that changing all model strings from "virtio" (or "virtio-scsi") to "virtio-transitional" is a sufficient workaround for booting FreeBSD 12.3 on a Q35 VM on libvirt+QEMU. Big thanks to Bryan V for getting the new work in to 13.0. Looking forward to either an MFC or the release of 13.1 :)
^Triage: assign to committer that resolved back in 2021. Now in all supported branches.