Bug 265522 - base system/bhyve : gvt_d_setup_opregion: Unable to get OpRegion base and length
Summary: base system/bhyve : gvt_d_setup_opregion: Unable to get OpRegion base and length
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: bhyve (show other bugs)
Version: 13.1-RELEASE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-virtualization (Nobody)
URL:
Keywords: bhyve
Depends on:
Blocks:
 
Reported: 2022-07-30 17:05 UTC by mario felicioni
Modified: 2022-08-02 17:02 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description mario felicioni 2022-07-30 17:05:34 UTC
Hello.

I'm running FreeBSD 13.1-RELEASE and I'm trying to passthru my CoffeeLake-S GT2 [UHD Graphics 630] on a Linux VM but it failed. This graphic card :


ppt4@pci0:0:2:0:        class=0x038000 rev=0x02 hdr=0x00 vendor=0x8086 device=0x3e98 subvendor=0x1458 subdevice=0xd000
vendor     = 'Intel Corporation'
device     = 'CoffeeLake-S GT2 [UHD Graphics 630]'
class      = display


below there are the bhyve parameters that I've used :


bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H -A \
-s 0,hostbridge \
-s 1,nvme,/dev/$vmdisk0,bootindex=1 \
-s 2,ahci-hd,/dev/$vmdisk2 \
-s 3,ahci-hd,/dev/$vmdisk4 \
-s 4,passthru,0/2/0 \
-s 7,virtio-net,tap4 \
-s 10,hda,play=/dev/dsp,rec=/dev/dsp \
-s 29,fbuf,tcp=0.0.0.0:5904,w=1500,h=950 \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm4 < /dev/null & sleep 2 && vncviewer 0:4


this is the error I get :


bhyve: Warning: Unable to reuse host address of Graphics Stolen Memory. 
GPU passthrough might not work properly.bhyve: gvt_d_setup_opregion: Unable to get OpRegion base and length ;
bhyve: gvt_d_init: Unable to setup OpRegion ; device emulation initialization
error: Operation not supported by device


A bhyve developer replied to me in this way :


I thought in general we couldn't pass the GPU if it was being used by the host? (Thus you would need to have two GPUs, one to allow the host to display its output and one to passthrough. Unless the host is completely headless).


At this point I think that the intel GPU is actually in use,but I don't know what could keep it in use. Infact :


I've removed the xf86-video-intel driver from the system with :


pkg remove xf86-video-intel


2) I haven't any i915kms or legacy intel module inside the memory :


# kldstat | grep i915 : nothing
# kldstat | grep kms : nothing
# kldstat | grep intel : nothing


3) this is my xorg.conf (where is there is no trace of intel driver)


Section "ServerLayout"
Identifier     "X.org Configured"
Screen      0  "Screen0" LeftOf "Screen1"
Screen      1  "Screen1"
InputDevice    "Mouse0" "CorePointer"
InputDevice    "Keyboard0" "CoreKeyboard"
EndSection

Section "Files"
ModulePath   "/usr/local/lib/xorg/modules"
FontPath     "/usr/local/share/fonts/misc/"
FontPath     "/usr/local/share/fonts/TTF/"
FontPath     "/usr/local/share/fonts/OTF/"
FontPath "/usr/local/share/fonts/Type1/"
FontPath "/usr/local/share/fonts/100dpi/"
FontPath "/usr/local/share/fonts/75dpi/"
FontPath "catalogue:/usr/local/etc/X11/fontpath.d"
EndSection

Section "Module"
Load  "glx"
EndSection

Section "InputDevice"
Identifier  "Keyboard0"
Driver      "kbd"
EndSection

Section "InputDevice"
Identifier  "Mouse0"
Driver      "mouse"
Option        "Protocol" "auto"
Option        "Device" "/dev/sysmouse"
Option        "ZAxisMapping" "4 5 6 7"
EndSection

Section "Monitor"
Identifier   "Monitor0"
VendorName   "Monitor Vendor"
ModelName    "Monitor Model"
EndSection

Section "Monitor"
Identifier   "Monitor1"
VendorName   "Monitor Vendor"
ModelName    "Monitor Model"
EndSection

Section "Device"
Identifier  "Card0"
Driver      "nvidia"
BusID       "PCI:1:0:0"
EndSection

Section "Screen"
Identifier "Screen0"
Device     "Card0"
Monitor    "Monitor0"
SubSection "Display"
Viewport   0 0
Depth     1
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     4
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     8
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     15
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     16
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     24
EndSubSection
EndSection

Section "Screen"
Identifier "Screen1"
Device     "Card0"
Monitor    "Monitor1"
SubSection "Display"
Viewport   0 0
Depth     1
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     4
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     8
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     15
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     16
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     24
EndSubSection
EndSection


I've just booted a linux vm passing thru the RTX 2080 ti using the same parameters used before and it worked. I've detached the HDMI cable from the 1060 and I've attached it to the 2080 ti and I've got what I wanted from the beginning : to choose between virtualize an OS with bhyve or more easily,only using the second monitor without bhyve. Now I want also pass thru the intel GPU.

What video card do I think provides the image I'm looking at on my monitor ? The following scheme is valid to answer this question :


1) monitor AOC + monitor Samsung -> 1 HDMI cable (for the samsung) + 1 DP cable (for the AOC) -> geforce 1060 - without a bhyve-Linux VM

2) monitor AOC + monitor Samsung -> 1 DP cable attached to the AOC and 1 hdmi cable attached to the RTX 2080 ti - if I want to use a bhyve / Linux vm passing the RTX 2080 ti GPU

3) monitor AOC + monitor Samsung -> 1 DP cable attached to the AOC and 1 HDMI cable attached to the Intel GPU - if I want to use a bhyve / Linux VM passing the Intel GPU


The Intel GPU is always kept free as well as the RTX 2080 ti. Even in the BIOS,since I've chosen the 1060 as default GPU,but the Intel GPU is enabled.
Comment 1 mario felicioni 2022-07-31 20:18:26 UTC
I have 3 GPUs in my PC. Actually the BIOS uses the geforce 1060 to start it. So,two GPUs are out and kept free. The Geforce RTX 2080 ti and the Intel CoffeeLake-S GT2 [UHD Graphics 630]. I'm able to pass through the RTX 2080 ti to a linux vm,but I'm not able to do the same with the UHD Graphics 630. As I have explained in the post,there is nothing that keeps it busy. I'm not using it. Both my monitors are connected to the 1060 using the twinview mode. One is using the DisplayPort,the other one is using the HDMI port. When I want to pass the RTX 2080 inside a Linux VM and I need to use my 2 monitor,I detach the HDMI cable from the 1060 and I attach it to the RTX 2080 ti and it's done. I can use it inside my bhyve VM. If I don't want to use bhyve,I connect the HDMI cable to the 1060 and in this case I can use two screens at the same time without running a VM. Cool. I can do this only with the RTX 2080 ti,but not with the Intel GPU. If I'm obliged to use it even if I don't use it,I can't imagine how and in which kind of situation I can pass it through.
Comment 2 mario felicioni 2022-08-01 17:21:13 UTC
Finally I've found the solution for this problem. Or,at least I'm on the right path. I've reinstalled FreeBSD 13.1-RELEASE without applying the Corvin patches and my GVT-d GPU has been recognised by Windows 11 :

https://forums.freebsd.org/attachments/screenshot_2022-08-01_18-50-49-png.14523/

So,the problem is inside the Corvin's repository. Unfortunately he doesn't reply to my clarification messages about what to do (it has been the first operation I did,but he does not reply from days,so I've looked for some help elsewhere). Anyway I can't use the bhyve code that has been upstreamed on the FreeBSD servers,because it lacks of the nvidia patches.
Comment 3 Graham Perrin freebsd_committer freebsd_triage 2022-08-02 17:02:00 UTC
Cross-reference: 

<https://old.reddit.com/r/freebsd/comments/wbykh9/-/>