Dear freebsd dev community, currently, it is not possible to pass-thru a NVIDIA GPU using bhyve. One is presented with the following error in the dmesg of e.g. linux: [ 77.208984] NVRM: Can't find an IRQ for your NVIDIA card! [ 77.212697] NVRM: Please check your BIOS settings. [ 77.212699] NVRM: [Plug & Play OS] should be set to NO [ 77.212700] NVRM: [Assign IRQ to VGA] should be set to YES [ 77.212702] nvidia: probe of 0000:00:07.0 failed with error -1 This issue was discussed in the forums and a solution found with a set of patches to bhyve: # cd /usr/ # rm -rf /usr/src # git clone https://github.com/beckhoff/freebsd-src /usr/src # cd /usr/src # git checkout -f origin/phab/corvink/14.2/nvidia-wip # cd /usr/src/usr.sbin/bhyve # make && make install I can confirm this works for my (AMD Ryzen 9 7900X / NVIDIA GA104 [GeForce RTX 3060]). But also others in the forums reported that it works. This is a kind request to maybe checkout the changes in the PR and check if they can be included in a next release of bhyve. Thanks a lot for this amazing system. Links: * Forums discussion: https://forums.freebsd.org/threads/current-state-of-bhyve-nvidia-passthrough.88244/page-5) * Related Bugs: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277627
It would be nice if the changes can be obtained as a patch to original src tree. Without it, no discussion is possible, as it's almost impossible to know what's changed throughout the whole bunch of src tree for this. I'm on discussion you pointed and once tried to let github to generate diffs between the branch you mentioned and seemingly corresponding branch, but it didn't finish (no results are shown) within several hours. (As I myself don't use GPU passthrough, I don't want to clone the repo just to obtain / generate diffs locally. So I gave up. (Not sure the original author did pull request or not.) And even with the repo, fatal problems are seen in the thread. At least different guest requires different fingerprint of the host VM (here, bhyve), some require "KVMKVMKVM" and some others require something other, forcing the user reporting it to rebuild FreeBSD modifying the patch to switch. At least, the fingerprint SHALL be configurable per-instance (conf files, sysctl,...). Not shure how trivial the fix would be. In my humble opinion, passing through any hardware is quite fragile and not a recommended way. I believe making UEFI firmware (or its successor) the VM host and forcing hardware accesses to be runtime services is the way to go, like mainframes are doing for CPUs, memories and so on (PPAR - Physical PARtitioning, LPAR - Logical PARtitioning, and any kind of virtualizations). As the idea of virtual machines itself came from mainframes.
I've started investigating which bits of my patches are required for NVIDIA GPUs. I've noticed that it's sufficient (for Linux guests, Windows untested) to set a valid INTPIN value in the PCI config space. Unfortunately, I'm unfamiliar with PCI legacy interrupts and bhyve doesn't support them for passthrough devices. So, I don't know which side effects can occur due to this change. I've created a review: https://reviews.freebsd.org/D51892
Thanks a lot Corvin, will check your changes against 14.3 today.
I applied the patches against 14.3 and can confirm it works! I only had to do a minor change to the Makefile. Thanks again Corvin! --- 8< --- diff --git a/usr.sbin/bhyve/Makefile b/usr.sbin/bhyve/Makefile index f4dae2f09904..1ec4623375a1 100644 --- a/usr.sbin/bhyve/Makefile +++ b/usr.sbin/bhyve/Makefile @@ -41,6 +41,7 @@ SRCS= \ pci_hostbridge.c \ pci_nvme.c \ pci_passthru.c \ + pci_passthru_quirks.c \ pci_virtio_9p.c \ pci_virtio_block.c \ pci_virtio_console.c \ diff --git a/usr.sbin/bhyve/pci_passthru_quirks.c b/usr.sbin/bhyve/pci_passthru_quirks.c new file mode 100644 index 000000000000..805e469303ba --- /dev/null +++ b/usr.sbin/bhyve/pci_passthru_quirks.c @@ -0,0 +1,49 @@ +/*- + * SPDX-License-Identifier: BSD-2-Clause + * + * Copyright (c) 2025 Beckhoff Automation GmbH & Co. KG + * Author: Corvin Köhne <c.koehne@beckhoff.com> + */ + +#include <dev/pci/pcireg.h> + +#include <errno.h> + +#include "pci_passthru.h" + +#define PCI_VENDOR_NVIDIA 0x10DE + +static int +nvidia_gpu_probe(struct pci_devinst *const pi) +{ + struct passthru_softc *sc; + uint16_t vendor; + uint8_t class; + + sc = pi->pi_arg; + + vendor = pci_host_read_config(passthru_get_sel(sc), PCIR_VENDOR, 0x02); + if (vendor != PCI_VENDOR_NVIDIA) + return (ENXIO); + + class = pci_host_read_config(passthru_get_sel(sc), PCIR_CLASS, 0x01); + if (class != PCIC_DISPLAY) + return (ENXIO); + + return (0); +}
+ +static int +nvidia_gpu_init(struct pci_devinst *const pi, nvlist_t *const nvl __unused) +{ + pci_set_cfgdata8(pi, PCIR_INTPIN, 1); + + return (0); +} + +static struct passthru_dev nvidia_gpu = { + .probe = nvidia_gpu_probe, + .init = nvidia_gpu_init, +}; +PASSTHRU_DEV_SET(nvidia_gpu); +
Also tested against 14.3p2, and it fixes the issue for my NVIDIA P400 passthrough.
The fix was committed to CURRENT, stable/15 and stable/14 [1], so all future FreeBSD will include it. Btw. note that QEMU has some additional quirks for NVIDIA GPUs in [2]. So, if some additional issues occur, we could try to port those to bhyve. For now, I'm closing this bug, feel free to open a new bug for additional issues. [1] https://reviews.freebsd.org/D51892 [2] https://elixir.bootlin.com/qemu/v10.1.0/source/hw/vfio/pci-quirks.c#L482-L484
When the error mentioned in this bug occurs ? I ask this because I'm not seeing it when I pass my Geforce RTX 2080 ti to a Linux or Windows VM...
(In reply to Corvin Köhne from comment #7) No it doesn't, I tried 14.3 p5, still not there, only on stable.
(In reply to dan from comment #9) Passthrough completely broken on freebsd 15 when I tried over weekend to pass through any device.
(In reply to dan from comment #9) It won't be merged for any existing FreeBSD version. 14.3 only receive security fixes, so not such kind of fixes. (In reply to dan from comment #10) Please provide some more details. Additionally, if passthrough is completely broken, it would be a good idea to open a new bug report as it's unrelated to Nvidia GPU passthrough.