Hello to everyone. I want to show you what it seems to be a bug in bhyve. Please evaluate the situation below : /usr/sbin/./bhyve-win -S -c sockets=4,cores=2,threads=1 -m 8G -w -H \ -S -c sockets=4,cores=2,threads=1 -m 8G -w -H \ -s 0,hostbridge \ -s 1,ahci-hd,/mnt/$vmdisk0'p2'/bhyve/img/Windows/Windows11.img,bootindex=1 \ -s 2,ahci-hd,/dev/$vmdisk5 \ -s 8:0,passthru,2/0/0 \ -s 8:1,passthru,2/0/1 \ -s 8:2,passthru,2/0/2 \ -s 8:3,passthru,2/0/3 \ -s 13,virtio-net,tap18 \ -s 29,fbuf,tcp=0.0.0.0:5918,w=1600,h=950,wait \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \ vm0:18 < /dev/null & sleep 2 && vncviewer 0:18 as you can see I have attached all the slots of my GPU. Infact my Nvidia GPU has 4 slots : 02:00.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] (rev a1) 02:00.1 Audio device: NVIDIA Corporation TU102 High Definition Audio Controller (rev a1) 02:00.2 USB controller: NVIDIA Corporation TU102 USB 3.1 Host Controller (rev a1) 02:00.3 Serial bus controller: NVIDIA Corporation TU102 USB Type-C UCSI Controller (rev a1) As you can see my GPU has one USB controller and one USB port mini integrated with the controller. Well,I attached a pen drive on the port of the USB controller and then I tried to boot the Windows 11 vm. What's happened ? that,despite I have declared that I want boot the Windows11 vm at slot 1 by adding the parameter bootindex=1,instead,it booted from the USB stick attached to the mini USB port of the USB controller (yes because on that USB stick I have installed Windows 7,so it can boot). This seems to be a bug,because it does not do what I want. And it is easy to reproduce. Just put bootindex=1 to every OS that you want to boot and then attach a bootable USB stick to the usb port of the GPU (it should be recent,it should be a GPU with the USB controller integrated,but the latest high quality GPU models have it),and just run the bhyve VM and you will see that it will not boot the OS you want.
From https://man.freebsd.org/cgi/man.cgi?bhyve: bootindex=index Add the device to the bootorder at index. A fwcfg file is used to specify the bootorder. The guest firmware may ignore or doesn't sup- port this fwcfg file. In that case, this fea- ture doesn't work as expected. EDK2 for bhyve currently doesn't support the bootorder fwcfg file. Therefore, it's ignored and uses it's default bootorder. I'm going to send a patch to upstream EDK2.
https://github.com/tianocore/edk2/pull/5774 When it's merged upstream, we can backport it to our EDK2 port.
https://reviews.freebsd.org/D45768
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=54eaea1ccd17ac679d065531243253b8bc682851 commit 54eaea1ccd17ac679d065531243253b8bc682851 Author: Corvin Köhne <corvink@FreeBSD.org> AuthorDate: 2024-06-28 06:54:01 +0000 Commit: Corvin Köhne <corvink@FreeBSD.org> CommitDate: 2024-07-01 13:01:49 +0000 sysutils/edk2: make bhyve's bootorder feature work Bhyve supports a bootindex property for bootable devices. When adding it to at least one device, bhyve creates a bootorder and passes it over the Qemu FwCfg interface to the guest. The guest firmware has to pick up the FwCfg file and has to apply the bootorder. Otherwise, the bootindex property won't have any effect. Recently, a commit was merged by EDKII to support that feature and we can simply cherry-pick it. PR: 279720 Reviewed by: manu Approved by: manu Sponsored by: Beckhoff Automation GmbH & Co. KG Differential Revision: https://reviews.freebsd.org/D45768 sysutils/edk2/Makefile | 3 ++- sysutils/edk2/distinfo | 4 +++- 2 files changed, 5 insertions(+), 2 deletions(-)
Can you explain to me a little bit how to apply that patch ? Can you confirm that the approach that I'm going to take is the correct one ? I've downloaded this package : https://cgit.freebsd.org/ports/snapshot/ports-54eaea1ccd17ac679d065531243253b8bc682851.tar.gz and I've extracted the port "edk2" directory from the package and I will exchange it with what I already have,not patched. And then,I will recompile it doing make / make install.
It's already merged. So, you can simply install it from ports (or wait until it's build and available on the official package server): https://docs.freebsd.org/en/books/handbook/ports/#ports-using In short: git clone https://github.com/freebsd/freebsd-ports /usr/ports cd /usr/ports/sysutils/edk2 FLAVOR=bhyve make reinstall
ok. I'm very curious to see if this patch will break or not the correct passthru of my Geforce RTX 2080 ti in a Windows VM. I still don't know why it's working and I'm also scared that one of the next patches that I will apply,will break it.
Sorry,your patch does not work for me : ..... make[2]: Leaving directory '/usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/X64/MdeModulePkg/Universal/Variable/RuntimeDxe/VariableRuntimeDxe' Fd File Name:BHYVE (/usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/BHYVE.fd) Generate Region at Offset 0x0 Region Size = 0x40000 Region Name = DATA Generate Region at Offset 0x40000 Region Size = 0x1000 Region Name = None Generate Region at Offset 0x41000 Region Size = 0x1000 Region Name = DATA Generate Region at Offset 0x42000 Region Size = 0x42000 Region Name = None Generate Region at Offset 0x84000 Region Size = 0x348000 Region Name = FV Generating FVMAIN_COMPACT FV Generating PEIFV FV #### Generating DXEFV FV ######### Generate Region at Offset 0x3CC000 Region Size = 0x34000 Region Name = FV Generating SECFV FV # Fd File Name:BHYVE_VARS (/usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/BHYVE_VARS.fd) Generate Region at Offset 0x0 Region Size = 0x40000 Region Name = DATA Generate Region at Offset 0x40000 Region Size = 0x1000 Region Name = None Generate Region at Offset 0x41000 Region Size = 0x1000 Region Name = DATA Generate Region at Offset 0x42000 Region Size = 0x42000 Region Name = None Fd File Name:BHYVE_CODE (/usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/BHYVE_CODE.fd) Generate Region at Offset 0x0 Region Size = 0x348000 Region Name = FV Generate Region at Offset 0x348000 Region Size = 0x34000 Region Name = FV Fd File Name:MEMFD (/usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/MEMFD.fd) Generate Region at Offset 0x0 Region Size = 0x6000 Region Name = None Generate Region at Offset 0x6000 Region Size = 0x1000 Region Name = None Generate Region at Offset 0x7000 Region Size = 0x1000 Region Name = None Padding region starting from offset 0x8000, with size 0x8000 Generate Region at Offset 0x8000 Region Size = 0x8000 Region Name = None Generate Region at Offset 0x10000 Region Size = 0x10000 Region Name = None Generate Region at Offset 0x20000 Region Size = 0xE0000 Region Name = FV Generate Region at Offset 0x100000 Region Size = 0xB00000 Region Name = FV GUID cross reference file can be found at /usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/Guid.xref FV Space Information SECFV [5%Full] 212992 (0x34000) total, 11808 (0x2e20) used, 201184 (0x311e0) free PEIFV [11%Full] 917504 (0xe0000) total, 106104 (0x19e78) used, 811400 (0xc6188) free DXEFV [45%Full] 11534336 (0xb00000) total, 5210672 (0x4f8230) used, 6323664 (0x607dd0) free FVMAIN_COMPACT [41%Full] 3440640 (0x348000) total, 1437760 (0x15f040) used, 2002880 (0x1e8fc0) free - Done - Build end time: 17:34:22, Jul.01 2024 Build total time: 00:01:29 ===> Staging for edk2-bhyve-g202308_5 ===> Generating temporary packing list /bin/mkdir -p /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/edk2-bhyve/ install -m 0644 /usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/BHYVE.fd /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/edk2-bhyve/BHYVE_UEFI.fd install -m 0644 /usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/BHYVE_CODE.fd /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/edk2-bhyve/BHYVE_UEFI_CODE.fd install -m 0644 /usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/BHYVE_VARS.fd /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/edk2-bhyve/BHYVE_UEFI_VARS.fd install -m 0644 /usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/BHYVE_CODE.fd /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/edk2-bhyve/BHYVE_UEFI.fd /bin/mkdir -p /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/uefi-firmware/ install -l rs /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/edk2-bhyve/BHYVE_UEFI.fd /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/uefi-firmware/BHYVE_UEFI.fd install -l rs /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/edk2-bhyve/BHYVE_UEFI_CODE.fd /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd install -l rs /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/edk2-bhyve/BHYVE_UEFI_VARS.fd /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/uefi-firmware/BHYVE_UEFI_VARS.fd ====> Compressing man pages (compress-man) ===> Installing for edk2-bhyve-g202308_5 ===> Checking if edk2-bhyve is already installed ===> Registering installation for edk2-bhyve-g202308_5 Installing edk2-bhyve-g202308_5... [root@marietto /usr/ports/sysutils/edk2]==> pkg remove edk2-bhyve Updating database digests format: 100% Checking integrity... done (0 conflicting) Deinstallation has been requested for the following 2 packages (of 0 packages in the universe): Installed packages to be REMOVED: bhyve-firmware: 1.0_2 edk2-bhyve: g202308_5 Number of packages to be removed: 2 The operation will free 7 MiB. Proceed with deinstalling packages? [y/N]: y [1/2] Deinstalling bhyve-firmware-1.0_2... [1/2] Deleting files for bhyve-firmware-1.0_2: 100% [2/2] Deinstalling edk2-bhyve-g202308_5... [2/2] Deleting files for edk2-bhyve-g202308_5: 100% [root@marietto /usr/ports/sysutils/edk2]==> FLAVOR=bhyve make reinstall ===> Deinstalling for edk2-bhyve ===> edk2-bhyve not installed, skipping ===> Installing for edk2-bhyve-g202308_5 ===> Checking if edk2-bhyve is already installed ===> Registering installation for edk2-bhyve-g202308_5 Installing edk2-bhyve-g202308_5... I launch the vm using these params : /usr/sbin/./bhyve-win -S -c sockets=4,cores=2,threads=1 -m 8G -w -H \ -S -c sockets=4,cores=2,threads=1 -m 8G -w -H \ -s 0,hostbridge \ -s 1,ahci-hd,/mnt/$vmdisk0'p2'/bhyve/img/Windows/11/Windows11.img,bootindex=1 \ -s 2,ahci-hd,/dev/$vmdisk5 \ -s 8:0,passthru,2/0/0 \ -s 8:1,passthru,2/0/1 \ -s 8:2,passthru,2/0/2 \ -s 8:3,passthru,2/0/3 \ -s 13,virtio-net,tap12 \ -s 29,fbuf,tcp=0.0.0.0:5912,w=1600,h=950,wait \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \ vm0:12 < /dev/null & sleep 2 && vncviewer 0:12 this is what happens : vm exit[0] reason VMX rip 0x00000000000f0015 inst_length 3 status 0 exit_reason 2 (Triple fault) qualification 0x0000000000000000 inst_type 0 inst_error 0
Maybe try a simpler bhyve call without passthrough devices.
^Triage: assign to committer.