Hello to everyone. I want to show you what it seems to be a bug in bhyve. Please evaluate the situation below : /usr/sbin/./bhyve-win -S -c sockets=4,cores=2,threads=1 -m 8G -w -H \ -S -c sockets=4,cores=2,threads=1 -m 8G -w -H \ -s 0,hostbridge \ -s 1,ahci-hd,/mnt/$vmdisk0'p2'/bhyve/img/Windows/Windows11.img,bootindex=1 \ -s 2,ahci-hd,/dev/$vmdisk5 \ -s 8:0,passthru,2/0/0 \ -s 8:1,passthru,2/0/1 \ -s 8:2,passthru,2/0/2 \ -s 8:3,passthru,2/0/3 \ -s 13,virtio-net,tap18 \ -s 29,fbuf,tcp=0.0.0.0:5918,w=1600,h=950,wait \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \ vm0:18 < /dev/null & sleep 2 && vncviewer 0:18 as you can see I have attached all the slots of my GPU. Infact my Nvidia GPU has 4 slots : 02:00.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] (rev a1) 02:00.1 Audio device: NVIDIA Corporation TU102 High Definition Audio Controller (rev a1) 02:00.2 USB controller: NVIDIA Corporation TU102 USB 3.1 Host Controller (rev a1) 02:00.3 Serial bus controller: NVIDIA Corporation TU102 USB Type-C UCSI Controller (rev a1) As you can see my GPU has one USB controller and one USB port mini integrated with the controller. Well,I attached a pen drive on the port of the USB controller and then I tried to boot the Windows 11 vm. What's happened ? that,despite I have declared that I want boot the Windows11 vm at slot 1 by adding the parameter bootindex=1,instead,it booted from the USB stick attached to the mini USB port of the USB controller (yes because on that USB stick I have installed Windows 7,so it can boot). This seems to be a bug,because it does not do what I want. And it is easy to reproduce. Just put bootindex=1 to every OS that you want to boot and then attach a bootable USB stick to the usb port of the GPU (it should be recent,it should be a GPU with the USB controller integrated,but the latest high quality GPU models have it),and just run the bhyve VM and you will see that it will not boot the OS you want.
From https://man.freebsd.org/cgi/man.cgi?bhyve: bootindex=index Add the device to the bootorder at index. A fwcfg file is used to specify the bootorder. The guest firmware may ignore or doesn't sup- port this fwcfg file. In that case, this fea- ture doesn't work as expected. EDK2 for bhyve currently doesn't support the bootorder fwcfg file. Therefore, it's ignored and uses it's default bootorder. I'm going to send a patch to upstream EDK2.
https://github.com/tianocore/edk2/pull/5774 When it's merged upstream, we can backport it to our EDK2 port.
https://reviews.freebsd.org/D45768
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=54eaea1ccd17ac679d065531243253b8bc682851 commit 54eaea1ccd17ac679d065531243253b8bc682851 Author: Corvin Köhne <corvink@FreeBSD.org> AuthorDate: 2024-06-28 06:54:01 +0000 Commit: Corvin Köhne <corvink@FreeBSD.org> CommitDate: 2024-07-01 13:01:49 +0000 sysutils/edk2: make bhyve's bootorder feature work Bhyve supports a bootindex property for bootable devices. When adding it to at least one device, bhyve creates a bootorder and passes it over the Qemu FwCfg interface to the guest. The guest firmware has to pick up the FwCfg file and has to apply the bootorder. Otherwise, the bootindex property won't have any effect. Recently, a commit was merged by EDKII to support that feature and we can simply cherry-pick it. PR: 279720 Reviewed by: manu Approved by: manu Sponsored by: Beckhoff Automation GmbH & Co. KG Differential Revision: https://reviews.freebsd.org/D45768 sysutils/edk2/Makefile | 3 ++- sysutils/edk2/distinfo | 4 +++- 2 files changed, 5 insertions(+), 2 deletions(-)
Can you explain to me a little bit how to apply that patch ? Can you confirm that the approach that I'm going to take is the correct one ? I've downloaded this package : https://cgit.freebsd.org/ports/snapshot/ports-54eaea1ccd17ac679d065531243253b8bc682851.tar.gz and I've extracted the port "edk2" directory from the package and I will exchange it with what I already have,not patched. And then,I will recompile it doing make / make install.
It's already merged. So, you can simply install it from ports (or wait until it's build and available on the official package server): https://docs.freebsd.org/en/books/handbook/ports/#ports-using In short: git clone https://github.com/freebsd/freebsd-ports /usr/ports cd /usr/ports/sysutils/edk2 FLAVOR=bhyve make reinstall
ok. I'm very curious to see if this patch will break or not the correct passthru of my Geforce RTX 2080 ti in a Windows VM. I still don't know why it's working and I'm also scared that one of the next patches that I will apply,will break it.
Sorry,your patch does not work for me : ..... make[2]: Leaving directory '/usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/X64/MdeModulePkg/Universal/Variable/RuntimeDxe/VariableRuntimeDxe' Fd File Name:BHYVE (/usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/BHYVE.fd) Generate Region at Offset 0x0 Region Size = 0x40000 Region Name = DATA Generate Region at Offset 0x40000 Region Size = 0x1000 Region Name = None Generate Region at Offset 0x41000 Region Size = 0x1000 Region Name = DATA Generate Region at Offset 0x42000 Region Size = 0x42000 Region Name = None Generate Region at Offset 0x84000 Region Size = 0x348000 Region Name = FV Generating FVMAIN_COMPACT FV Generating PEIFV FV #### Generating DXEFV FV ######### Generate Region at Offset 0x3CC000 Region Size = 0x34000 Region Name = FV Generating SECFV FV # Fd File Name:BHYVE_VARS (/usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/BHYVE_VARS.fd) Generate Region at Offset 0x0 Region Size = 0x40000 Region Name = DATA Generate Region at Offset 0x40000 Region Size = 0x1000 Region Name = None Generate Region at Offset 0x41000 Region Size = 0x1000 Region Name = DATA Generate Region at Offset 0x42000 Region Size = 0x42000 Region Name = None Fd File Name:BHYVE_CODE (/usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/BHYVE_CODE.fd) Generate Region at Offset 0x0 Region Size = 0x348000 Region Name = FV Generate Region at Offset 0x348000 Region Size = 0x34000 Region Name = FV Fd File Name:MEMFD (/usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/MEMFD.fd) Generate Region at Offset 0x0 Region Size = 0x6000 Region Name = None Generate Region at Offset 0x6000 Region Size = 0x1000 Region Name = None Generate Region at Offset 0x7000 Region Size = 0x1000 Region Name = None Padding region starting from offset 0x8000, with size 0x8000 Generate Region at Offset 0x8000 Region Size = 0x8000 Region Name = None Generate Region at Offset 0x10000 Region Size = 0x10000 Region Name = None Generate Region at Offset 0x20000 Region Size = 0xE0000 Region Name = FV Generate Region at Offset 0x100000 Region Size = 0xB00000 Region Name = FV GUID cross reference file can be found at /usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/Guid.xref FV Space Information SECFV [5%Full] 212992 (0x34000) total, 11808 (0x2e20) used, 201184 (0x311e0) free PEIFV [11%Full] 917504 (0xe0000) total, 106104 (0x19e78) used, 811400 (0xc6188) free DXEFV [45%Full] 11534336 (0xb00000) total, 5210672 (0x4f8230) used, 6323664 (0x607dd0) free FVMAIN_COMPACT [41%Full] 3440640 (0x348000) total, 1437760 (0x15f040) used, 2002880 (0x1e8fc0) free - Done - Build end time: 17:34:22, Jul.01 2024 Build total time: 00:01:29 ===> Staging for edk2-bhyve-g202308_5 ===> Generating temporary packing list /bin/mkdir -p /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/edk2-bhyve/ install -m 0644 /usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/BHYVE.fd /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/edk2-bhyve/BHYVE_UEFI.fd install -m 0644 /usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/BHYVE_CODE.fd /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/edk2-bhyve/BHYVE_UEFI_CODE.fd install -m 0644 /usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/BHYVE_VARS.fd /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/edk2-bhyve/BHYVE_UEFI_VARS.fd install -m 0644 /usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/BHYVE_CODE.fd /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/edk2-bhyve/BHYVE_UEFI.fd /bin/mkdir -p /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/uefi-firmware/ install -l rs /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/edk2-bhyve/BHYVE_UEFI.fd /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/uefi-firmware/BHYVE_UEFI.fd install -l rs /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/edk2-bhyve/BHYVE_UEFI_CODE.fd /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd install -l rs /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/edk2-bhyve/BHYVE_UEFI_VARS.fd /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/uefi-firmware/BHYVE_UEFI_VARS.fd ====> Compressing man pages (compress-man) ===> Installing for edk2-bhyve-g202308_5 ===> Checking if edk2-bhyve is already installed ===> Registering installation for edk2-bhyve-g202308_5 Installing edk2-bhyve-g202308_5... [root@marietto /usr/ports/sysutils/edk2]==> pkg remove edk2-bhyve Updating database digests format: 100% Checking integrity... done (0 conflicting) Deinstallation has been requested for the following 2 packages (of 0 packages in the universe): Installed packages to be REMOVED: bhyve-firmware: 1.0_2 edk2-bhyve: g202308_5 Number of packages to be removed: 2 The operation will free 7 MiB. Proceed with deinstalling packages? [y/N]: y [1/2] Deinstalling bhyve-firmware-1.0_2... [1/2] Deleting files for bhyve-firmware-1.0_2: 100% [2/2] Deinstalling edk2-bhyve-g202308_5... [2/2] Deleting files for edk2-bhyve-g202308_5: 100% [root@marietto /usr/ports/sysutils/edk2]==> FLAVOR=bhyve make reinstall ===> Deinstalling for edk2-bhyve ===> edk2-bhyve not installed, skipping ===> Installing for edk2-bhyve-g202308_5 ===> Checking if edk2-bhyve is already installed ===> Registering installation for edk2-bhyve-g202308_5 Installing edk2-bhyve-g202308_5... I launch the vm using these params : /usr/sbin/./bhyve-win -S -c sockets=4,cores=2,threads=1 -m 8G -w -H \ -S -c sockets=4,cores=2,threads=1 -m 8G -w -H \ -s 0,hostbridge \ -s 1,ahci-hd,/mnt/$vmdisk0'p2'/bhyve/img/Windows/11/Windows11.img,bootindex=1 \ -s 2,ahci-hd,/dev/$vmdisk5 \ -s 8:0,passthru,2/0/0 \ -s 8:1,passthru,2/0/1 \ -s 8:2,passthru,2/0/2 \ -s 8:3,passthru,2/0/3 \ -s 13,virtio-net,tap12 \ -s 29,fbuf,tcp=0.0.0.0:5912,w=1600,h=950,wait \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \ vm0:12 < /dev/null & sleep 2 && vncviewer 0:12 this is what happens : vm exit[0] reason VMX rip 0x00000000000f0015 inst_length 3 status 0 exit_reason 2 (Triple fault) qualification 0x0000000000000000 inst_type 0 inst_error 0
Maybe try a simpler bhyve call without passthrough devices.
^Triage: assign to committer.
Does this really work now? I have installed: edk2-bhyve-g202308_5, vm-bhyve-1.6.2_1 Patched vm-bhyve locally to add bootindex=1 to ahci-cd: Aug. 05 11:23:09: initialising Aug. 05 11:23:09: [loader: uefi] Aug. 05 11:23:09: [cpu: 8] Aug. 05 11:23:09: [memory: 12G] Aug. 05 11:23:09: [hostbridge: standard] Aug. 05 11:23:09: [com ports: com1] Aug. 05 11:23:09: [uuid: 291fed2d-71d4-11f0-8490-b499bac04fbe] Aug. 05 11:23:09: [debug mode: yes] Aug. 05 11:23:09: [primary disk: disk0.img] Aug. 05 11:23:09: [primary disk dev: file] Aug. 05 11:23:09: initialising network device tap1 Aug. 05 11:23:09: adding tap1 -> bce1bridge (VL-G-DE-SP02-4181 addm) Aug. 05 11:23:09: bring up tap1 -> bce1bridge (VL-G-DE-SP02-4181 addm) Aug. 05 11:23:09: dynamically allocated port 5901 for vnc connections Aug. 05 11:23:09: booting Aug. 05 11:23:09: [bhyve options: -c 8 -m 12G -AHPw -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/usr/local/bhyve/kinexon/uefi-vars.fd -K de -U 291fed2d-71d4-11f0-8490-b499bac04fbe -u] Aug. 05 11:23:09: [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,nvme,/usr/local/bhyve/kinexon/disk0.img -s 5:0,virtio-net,tap1,mac=58:9c:fc:09:5e:14 -s 6:0,fbuf,tcp=127.0.0.1:5901,w=1280,h=1024,wait -s 7:0,xhci,tablet] Aug. 05 11:23:09: [bhyve console: -l com1,/dev/nmdm-kinexon.1A] Aug. 05 11:23:09: [bhyve iso device: -s 3:0,ahci-cd,/usr/local/bhyve/.iso/void-live-x86_64-20250202-base.iso,ro,bootindex=1] Aug. 05 11:23:09: starting bhyve (run 1) It does not boot from the ISO, reset from UEFI shell I see the following when entering UEFI config: Boot Manager Menu UEFI bhyve-NVMe NVME-4-0 UEFI PXEv4 (MAC:589CFC095E14) UEFI PXEv6 (MAC:589CFC095E14) EFI Internal Shell UEFI BHYVE SATA DVD ROM BHYVE-F134-08D4-1073 From the shell: Option: 00. Variable: Boot0001 Desc - UEFI bhyve-NVMe NVME-4-0 DevPath - PciRoot(0x0)/Pci(0x4,0x0)/NVMe(0x1,01-00-D6-E9-20-FC-9C-58) Optional- Y 00000000: 4E AC 08 81 11 9F 59 4D-85 0E E2 1A 52 2C 59 B2 *N.....YM....R,Y.* Option: 01. Variable: Boot0002 Desc - UEFI PXEv4 (MAC:589CFC095E14) DevPath - PciRoot(0x0)/Pci(0x5,0x0)/MAC(589CFC095E14,0x1)/IPv4(0.0.0.0) Optional- Y 00000000: 4E AC 08 81 11 9F 59 4D-85 0E E2 1A 52 2C 59 B2 *N.....YM....R,Y.* Option: 02. Variable: Boot0003 Desc - UEFI PXEv6 (MAC:589CFC095E14) DevPath - PciRoot(0x0)/Pci(0x5,0x0)/MAC(589CFC095E14,0x1)/IPv6(0000:0000:0000:0000:0000:0000:0000:0000) Optional- Y 00000000: 4E AC 08 81 11 9F 59 4D-85 0E E2 1A 52 2C 59 B2 *N.....YM....R,Y.* Option: 03. Variable: Boot0004 Desc - EFI Internal Shell DevPath - Fv(7CB8BDC9-F8EB-4F34-AAEA-3EE4AF6516A1)/FvFile(7C04A583-9E3E-4F1C-AD65-E05268D0B4D1) Optional- N Option: 04. Variable: Boot0000 Desc - UiApp DevPath - Fv(7CB8BDC9-F8EB-4F34-AAEA-3EE4AF6516A1)/FvFile(462CAA21-7614-4503-836E-8AB6F4662331) Optional- N Option: 05. Variable: Boot0005 Desc - UEFI BHYVE SATA DVD ROM BHYVE-F134-08D4-1073 DevPath - PciRoot(0x0)/Pci(0x3,0x0)/Sata(0x0,0xFFFF,0x0) Optional- Y 00000000: 4E AC 08 81 11 9F 59 4D-85 0E E2 1A 52 2C 59 B2 *N.....YM....R,Y.* What did I miss here?
(In reply to Michael Osipov from comment #11) Which version of bhyve are you using?
(In reply to Corvin Köhne from comment #12) I am on FreeBSD deblndw013x.innomotics.net 13.5-STABLE FreeBSD 13.5-STABLE 0106e5d72 GENERIC amd64
The bootorder should be supported by 13.5. You have to set lpc.fwcfg to qemu: -s 31,lpc,fwcfg=qemu or -o lpc.fwcfg=qemu
(In reply to Corvin Köhne from comment #14) Let me double check...
(In reply to Corvin Köhne from comment #14) Success, it does indeed work: > Boot Manager Menu > > UEFI BHYVE SATA DVD ROM BHYVE-F134-08D4-1073 > UEFI bhyve-NVMe NVME-4-0 > UEFI PXEv4 (MAC:589CFC095E14) > UEFI PXEv6 (MAC:589CFC095E14) > EFI Internal Shell Void Linux was booted from ISO. Both configs work, even the VNC screen is larger than before. Insane, thank you! I need to raise a ticket with vm-bhyve. Side question: Is there any drawback to set fwcfg to qemu permanently?
(In reply to Michael Osipov from comment #16) > Side question: Is there any drawback to set fwcfg to qemu permanently? The default fwcfg=bhyve is mostly a nop. It was only used to pass the number of vcpus to OVMF. This information is also passed by fwcfg=qemu to the guest and picked up by OVMF since edk2-stable202205.
(In reply to Corvin Köhne from comment #17) Wouldn't it make sense to change the default value from bhyve to qemu on 15 then?
I have created to PRs for vm-bhyve: * https://github.com/freebsd/vm-bhyve/pull/22 * https://github.com/freebsd/vm-bhyve/pull/21
I think we can close this since I have verified it to work. I expect that 14/15 do work as well. ckoehne@ an objections?
(In reply to Michael Osipov from comment #20) Yes, the patches are included in 14 and 15 too. So, it should work there too.
This feature doesn't work for me still. E.g. I boot a VM with: sudo bhyve -c 2 -m 4096 -A -I -u -H -P \ -s 0:0,hostbridge \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/var/lib/libvirt/bhyve/nvram/ubuntu2504_VARS.fd,fwcfg=qemu \ -s 2:0,xhci,tablet -s 1:0,lpc,fwcfg=qemu \ -s 3:0,ahci,hd:/data/img/ubuntu2504.img,bootindex=2,cd:/data/isos/ubuntu-25.04-live-server-amd64.iso,bootindex=1 \ -s 4:0,virtio-net,tap4,mnac=52:54:00:47:f4:dc -s 5:0,fbuf,tcp=127.0.0.1:5920 \ -l com1,stdio \ ubuntu2504 That works. Then for the next boot I want to boot from the disk image, I do: sudo bhyve -c 2 -m 4096 -A -I -u -H -P \ -s 0:0,hostbridge \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/var/lib/libvirt/bhyve/nvram/ubuntu2504_VARS.fd,fwcfg=qemu \ -s 2:0,xhci,tablet -s 1:0,lpc,fwcfg=qemu \ -s 3:0,ahci,hd:/data/img/ubuntu2504.img,bootindex=2,cd:/data/isos/ubuntu-25.04-live-server-amd64.iso,bootindex=3 \ -s 4:0,virtio-net,tap4,mnac=52:54:00:47:f4:dc -s 5:0,fbuf,tcp=127.0.0.1:5920 \ -l com1,stdio \ ubuntu2504 It doesn't work, it still boots from CD. Additionally, sections about fwcfg in bhyve(8) seem confusing to me. It's not obvious if fwcfg should be a part of `-l bootrom` configuration or a part of `-s 31,lpc` configuration (or both?).
(In reply to Roman Bogorodskiy from comment #22) > -s 3:0,ahci,hd:/data/img/ubuntu2504.img,bootindex=2,cd:/data/isos/ubuntu-25.04-live-server-amd64.iso,bootindex=1 \ The hypervisor has to pass the bootpath to the guest firmware. Currently, bhyve uses a very simple bootpath, just the PCI address. This works for many use cases but not for yours where multiple boot devices are attached to a single PCI device.
(In reply to Corvin Köhne from comment #23) Hm, that's very strange and I don't quite get what's going on there. What I tried: > -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/var/lib/libvirt/bhyve/nvram/ubuntu2504_VARS.fd > ... > -s 3:0,ahci-hd,/data/img/ubuntu2504.img,bootindex=2 -s 4:0,ahci-cd,/data/isos/ubuntu-25.04-live-server-amd64.iso,bootindex=1 This didn't work, system was booting from HD, despite CD having "bootindex=1". Then I tried the same with VARS.fd removed: > -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd > ... > -s 3:0,ahci-hd,/data/img/ubuntu2504.img,bootindex=2 -s 4:0,ahci-cd,/data/isos/ubuntu-25.04-live-server-amd64.iso,bootindex=1 Yay, this works! VM boots from CD (bootindex=1). Now I swap the indexes and expect it to boot from HD. It still boots from CD. I paste a complete command for once to avoid any confusing: bhyve -c 2 -m 4096 -A -I -u -H -P \ -s 0:0,hostbridge \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ -s 2:0,xhci,tablet -s 1:0,lpc,fwcfg=qemu \ -s 3:0,ahci-hd,/data/img/ubuntu2504.img,bootindex=1 -s 4:0,ahci-cd,/data/isos/ubuntu-25.04-live-server-amd64.iso,bootindex=2 \ -s 5:0,fbuf,tcp=127.0.0.1:5920 \ -l com1,stdio \ ubuntu2504 In console I see: fbuf frame buffer base: 0x280638200000 [sz 33554432] 0 0 BdsDxe: loading Boot0001 "UEFI BHYVE SATA DVD ROM BHYVE-D1CE-3E63-5ED8" from PciRoot(0x0)/Pci(0x4,0x0)/Sata(0x0,0xFFFF,0x0) BdsDxe: starting Boot0001 "UEFI BHYVE SATA DVD ROM BHYVE-D1CE-3E63-5ED8" from PciRoot(0x0)/Pci(0x4,0x0)/Sata(0x0,0xFFFF,0x0) error: can't find command `grub_platform'. EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path It looks like it didn't even try to boot from HD. PS I was about to submit, but double-checked the man page, and it says: Add the device to the bootorder at index. A fwcfg file is used to specify the bootorder. It's not quite clear, what does "fwcfg file" mean here, does it mean "lpc,fwcfg=qemu", or the VARS.fd file, or something else?
(In reply to Roman Bogorodskiy from comment #24) > Hm, that's very strange and I don't quite get what's going on there. What I tried: The VARS file saves UEFI variables. There's a BootOrder variable which determines the boot order. So, by removing the VARS file, you're falling back to the default BootOrder which might be different. > -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ You're missing the fwcfg=qemu option here, use: > -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=qemu \ > Add the device to the bootorder at index. A fwcfg file is used to specify the bootorder. > > It's not quite clear, what does "fwcfg file" mean here, does it mean "lpc,fwcfg=qemu", or the VARS.fd file, or something else? This sentence should give you a hint, that a fwcfg file is required, so you need the "fwcfg=qemu" option (at "-l bootrom,<rom>,fwcfg=qemu" not at "-s xxx,lpc"). FwCfg is an interface invented by QEMU to pass data from the hypervisor to the guest. Data is provided in so called files, e.g. the file "opt/bhyve/hw.ncpu" contains the vCPU count, "etc/table-loader" contains ACPI tables and "bootorder" contains the BootOrder. Most Linux distributions expose those files by sysfs in /sys/firmware/fw_cfg [1]. You can check it in your VM to make sure the bootorder was exposed by bhyve. To get more information about it, it might be worth checking the QEMU and Linux docs about it. Btw. you can even add your own fwcfg files to bhyve, if you like to: > -f name,[string|file]=data > Add a fw_cfg file name to the fw_cfg interface. If a > string is specified, the fw_cfg file contains the string as > data. If a file is specified, bhyve reads the file and > adds the file content as fw_cfg data. This is useful to pass custom data to the guest, e.g. Fedora Core OS uses it to provision the guest [2]. [1] https://elixir.bootlin.com/linux/v6.16.7/source/drivers/firmware/Kconfig#L122 [2] https://docs.fedoraproject.org/en-US/fedora-coreos/provisioning-qemu/#_setting_up_a_new_vm
(In reply to Corvin Köhne from comment #25) That doesn't work still. Command I run: sudo bhyve -c 2 -m 4096 -A -I -u -H -P \ -s 0:0,hostbridge \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/var/lib/libvirt/bhyve/nvram/ubuntu2504_VARS.fd,fwcfg=qemu \ -s 2:0,xhci,tablet -s 1:0,lpc \ -s 3:0,ahci-hd,/data/img/ubuntu2504.img,bootindex=2 -s 4:0,ahci-cd,/data/isos/ubuntu-25.04-live-server-amd64.iso,bootindex=1 \ -s 5:0,fbuf,tcp=127.0.0.1:5920 \ -l com1,stdio -s 6:0,virtio-net,tap4,mac=52:54:00:47:f4:dc \ ubuntu2504 I expect it to boot from ISO, it boots from HD instead. In the guest, I do not see /sys/firmware/fw_cfg. However, I see qemu_fw_cfg. root@ubuntu2504:/sys/firmware# ls -1 acpi dmi efi memmap qemu_fw_cfg root@ubuntu2504:/sys/firmware# cat qemu_fw_cfg/ by_key/ by_name/ rev root@ubuntu2504:/sys/firmware# cat qemu_fw_cfg/by_name/bootorder/raw /pci@i0cf8/pci@4,0 /pci@i0cf8/pci@3,0 root@ubuntu2504:/sys/firmware# This order here seems to be valid.
(In reply to Roman Bogorodskiy from comment #26) What's your version of edk2-bhyve?
(In reply to Corvin Köhne from comment #27) It's g202508 from PR #285554. $ pkg which /usr/local/share/uefi-firmware/BHYVE_UEFI.fd /usr/local/share/uefi-firmware/BHYVE_UEFI.fd was installed by package edk2-bhyve-g202508 $
(In reply to Roman Bogorodskiy from comment #28) Just out of curiosity, try to remove the VARS file from your bhyve command.
(In reply to Corvin Köhne from comment #29) Trying two commands: 1. bhyve -c 2 -m 4096 -A -I -u -H -P \ -s 0:0,hostbridge \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=qemu \ -s 2:0,xhci,tablet -s 1:0,lpc \ -s 3:0,ahci-hd,/data/img/ubuntu2504.img,bootindex=2 -s 4:0,ahci-cd,/data/isos/ubuntu-25.04-live-server-amd64.iso,bootindex=1 \ -s 5:0,fbuf,tcp=127.0.0.1:5920 \ -l com1,stdio -s 6:0,virtio-net,tap4,mac=52:54:00:47:f4:dc \ ubuntu2504 Boots from CD, works as expected. No changes in qemu_fw_cfg/by_name/bootorder/raw. 2. bhyve -c 2 -m 4096 -A -I -u -H -P \ -s 0:0,hostbridge \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=qemu \ -s 2:0,xhci,tablet -s 1:0,lpc \ -s 3:0,ahci-hd,/data/img/ubuntu2504.img,bootindex=1 -s 4:0,ahci-cd,/data/isos/ubuntu-25.04-live-server-amd64.iso,bootindex=2 \ -s 5:0,fbuf,tcp=127.0.0.1:5920 \ -l com1,stdio -s 6:0,virtio-net,tap4,mac=52:54:00:47:f4:dc \ ubuntu2504 Expected to boot from HD, boots from CD instead. qemu_fw_cfg is changed: root@ubuntu-server:/sys/firmware/qemu_fw_cfg/by_name# cat bootorder/raw /pci@i0cf8/pci@3,0 /pci@i0cf8/pci@4,0 root@ubuntu-server:/sys/firmware/qemu_fw_cfg/by_name# Looks like qemu_fw_cfg is always correct, e.g. corresponds to what's requested via cmdline.
(In reply to Roman Bogorodskiy from comment #30) Could you try to build a debug version of the firmware and share the log? Therefore, build the bhyve flavor of sysutils/edk2 on your own [1][2] and apply following patch: diff --git a/sysutils/edk2/Makefile b/sysutils/edk2/Makefile index 5b1947c88b2d..e0636acf7512 100644 --- a/sysutils/edk2/Makefile +++ b/sysutils/edk2/Makefile @@ -116,7 +116,8 @@ ONLY_FOR_ARCHS_REASON= Bhyve only runs on x64 PLAT= bhyve PLAT_ARCH= X64 PLAT_ARGS+= -D SECURE_BOOT_ENABLE=TRUE -D TPM2_ENABLE=TRUE -PLAT_TARGET= RELEASE +PLAT_ARGS+= -D DEBUG_ON_SERIAL_PORT=TRUE +PLAT_TARGET= DEBUG PLATFILE= OvmfPkg/Bhyve/BhyveX64.dsc PLAT_RESULT= BhyveX64/${PLAT_TARGET}_GCC5/FV/BHYVE.fd PLAT_RESULT_CODE= BhyveX64/${PLAT_TARGET}_GCC5/FV/BHYVE_CODE.fd [1] https://docs.freebsd.org/en/books/handbook/ports/#ports-using-installation-methods [2] https://docs.freebsd.org/en/books/handbook/ports/#_installing_ports Use FLAVOR=bhyve make install instead of just make install.
Created attachment 263879 [details] edk2 debug log for ubuntu but with bootindex not working (In reply to Corvin Köhne from comment #31) I assume we need to get back to a version of the command which uses the VARS file. So I'm going back to this command: bhyve -c 2 -m 4096 -A -I -u -H -P \ -s 0:0,hostbridge \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/var/lib/libvirt/bhyve/nvram/ubuntu2504_VARS.fd,fwcfg=qemu \ -s 2:0,xhci,tablet -s 1:0,lpc \ -s 3:0,ahci-hd,/data/img/ubuntu2504.img,bootindex=2 -s 4:0,ahci-cd,/data/isos/ubuntu-25.04-live-server-amd64.iso,bootindex=1 \ -s 5:0,fbuf,tcp=127.0.0.1:5920 \ -l com1,stdio -s 6:0,virtio-net,tap4,mac=52:54:00:47:f4:dc \ ubuntu2504 2>&1 | tee ~/ubuntu_edk2_bootindex.log Which is expected to boot from CD, but boots from HD instead. Attaching the log.
I have asked to use codex (gpt-5.6-sol) to localize the issue, and, interestingly, it came up with a one line fix: --- a/usr.sbin/bhyve/pci_emul.c +++ b/usr.sbin/bhyve/pci_emul.c @@ -1516,7 +1516,7 @@ init_bootorder(void) } fclose(fp); - return (qemu_fwcfg_add_file("bootorder", bootorder_len, bootorder)); + return (qemu_fwcfg_add_file("bootorder", bootorder_len + 1, bootorder)); } Its explanation is the original length trims '\0', and EDK2 rejects a configuration which is not properly terminated. To check this claim, sysutils/edk2 was rebuilt with the following patch (and debugging enabled): --- OvmfPkg/Library/QemuBootOrderLib/QemuBootOrderLib.c.orig 2025-08-22 00:00:00 UTC +++ OvmfPkg/Library/QemuBootOrderLib/QemuBootOrderLib.c @@ -1599,7 +1599,14 @@ ConnectDevicesFromQemu ( QemuFwCfgSelectItem (FwCfgItem); QemuFwCfgReadBytes (FwCfgSize, FwCfg); if (FwCfg[FwCfgSize - 1] != '\0') { + DEBUG (( + DEBUG_ERROR, + "%a: bootorder is not NUL-terminated, size=%Lu, last=0x%02x\n", + __func__, + (UINT64)FwCfgSize, + (UINT8)FwCfg[FwCfgSize - 1] + )); Status = RETURN_INVALID_PARAMETER; goto FreeFwCfg; } @@ -1745,7 +1752,14 @@ StoreQemuBootOrder ( QemuFwCfgSelectItem (FwCfgItem); QemuFwCfgReadBytes (FwCfgSize, FwCfg); if (FwCfg[FwCfgSize - 1] != '\0') { + DEBUG (( + DEBUG_ERROR, + "%a: bootorder is not NUL-terminated, size=%Lu, last=0x%02x\n", + __func__, + (UINT64)FwCfgSize, + (UINT8)FwCfg[FwCfgSize - 1] + )); Status = RETURN_INVALID_PARAMETER; goto FreeFwCfg; } @@ -2206,7 +2220,14 @@ SetBootOrderFromQemu ( QemuFwCfgSelectItem (FwCfgItem); QemuFwCfgReadBytes (FwCfgSize, FwCfg); if (FwCfg[FwCfgSize - 1] != '\0') { + DEBUG (( + DEBUG_ERROR, + "%a: bootorder is not NUL-terminated, size=%Lu, last=0x%02x\n", + __func__, + (UINT64)FwCfgSize, + (UINT8)FwCfg[FwCfgSize - 1] + )); Status = RETURN_INVALID_PARAMETER; goto ErrorFreeFwCfg; } Use it to run a test VM: $ script -q /tmp/bootindex-279720.log sudo bhyve \ -c 1 -m 1G -A -H -P \ -s 0,hostbridge \ -s 1,ahci-hd,/data/img/freebsd.img,ro,bootindex=1 \ -s 2,ahci-cd,/data/isos/FreeBSD-14.2-RELEASE-amd64-disc1.iso,bootindex=2 \ -s 31,lpc \ -l bootrom,/usr/home/novel/ports/sysutils/edk2/work-bhyve/edk2-edk2-stable202508/Build/BhyveX64/DEBUG_GCC/FV/BHYVE_CODE.fd,/tmp/bootindex-279720-vars.fd,fwcfg=qemu \ -l com1,stdio \ bootindex-279720 $ grep "not NUL" /tmp/bootindex-279720.log ConnectDevicesFromQemu: bootorder is not NUL-terminated, size=38, last=0x0A SetBootOrderFromQemu: bootorder is not NUL-terminated, size=38, last=0x0A $ So we indeed end up rejected by these checks. After applying the suggested bhyve fix these messages are gone. Now repeating the tests I did in comment 30: 1. bhyve -c 2 -m 4096 -A -I -u -H -P \ -s 0:0,hostbridge \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=qemu \ -s 2:0,xhci,tablet -s 1:0,lpc \ -s 3:0,ahci-hd,/data/img/ubuntu2504.img,bootindex=2 -s 4:0,ahci-cd,/data/isos/ubuntu-25.04-live-server-amd64.iso,bootindex=1 \ -s 5:0,fbuf,tcp=127.0.0.1:5920 \ -l com1,stdio -s 6:0,virtio-net,tap4,mac=52:54:00:47:f4:dc \ ubuntu2504 Same result as earlier run: boots from CD, works as expected. 2. bhyve -c 2 -m 4096 -A -I -u -H -P \ -s 0:0,hostbridge \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=qemu \ -s 2:0,xhci,tablet -s 1:0,lpc \ -s 3:0,ahci-hd,/data/img/ubuntu2504.img,bootindex=1 -s 4:0,ahci-cd,/data/isos/ubuntu-25.04-live-server-amd64.iso,bootindex=2 \ -s 5:0,fbuf,tcp=127.0.0.1:5920 \ -l com1,stdio -s 6:0,virtio-net,tap4,mac=52:54:00:47:f4:dc \ ubuntu2504 This is now different: this time, it boots from HD, as expected. So the fix appears to be valid, at least in this scenario.
(In reply to Roman Bogorodskiy from comment #33) Great finding! Are there maybe other spots which might have the same failing result?
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=b27a221ef79d6d77d6173897f12118e1126453d8 commit b27a221ef79d6d77d6173897f12118e1126453d8 Author: Roman Bogorodskiy <novel@FreeBSD.org> AuthorDate: 2026-09-20 14:41:55 +0000 Commit: Roman Bogorodskiy <novel@FreeBSD.org> CommitDate: 2026-09-21 16:54:45 +0000 bhyve: fix boot device ordering EDK2's QemuBootOrderLib inspects the bootorder file provided via fw_cfg and requires it to be NUL-terminated. Otherwise, it rejects the supplied bootorder and falls back to its default boot order. Currently, bhyve registers bootorder with qemu_fwcfg_add_file() using bootorder_len returned by open_memstream(), which excludes the trailing NUL byte. Fix that by passing bootorder_len + 1 to qemu_fwcfg_add_file() so the fw_cfg payload is properly NUL-terminated. PR: 279720 Reviewed by: markj Found with: codex (gpt-5.6-sol) MFC after: 1 week Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D59859 usr.sbin/bhyve/pci_emul.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(In reply to Michael Osipov from comment #34) Right now I don't recall any similar issues.
A commit in branch stable/15 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=1a6e0c38d023f975d6015570b9fc34f09175dc03 commit 1a6e0c38d023f975d6015570b9fc34f09175dc03 Author: Roman Bogorodskiy <novel@FreeBSD.org> AuthorDate: 2026-09-20 14:41:55 +0000 Commit: Roman Bogorodskiy <novel@FreeBSD.org> CommitDate: 2026-09-28 15:59:17 +0000 bhyve: fix boot device ordering EDK2's QemuBootOrderLib inspects the bootorder file provided via fw_cfg and requires it to be NUL-terminated. Otherwise, it rejects the supplied bootorder and falls back to its default boot order. Currently, bhyve registers bootorder with qemu_fwcfg_add_file() using bootorder_len returned by open_memstream(), which excludes the trailing NUL byte. Fix that by passing bootorder_len + 1 to qemu_fwcfg_add_file() so the fw_cfg payload is properly NUL-terminated. PR: 279720 Reviewed by: markj Found with: codex (gpt-5.6-sol) MFC after: 1 week Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D59859 (cherry picked from commit b27a221ef79d6d77d6173897f12118e1126453d8) usr.sbin/bhyve/pci_emul.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=c901f931f6eb102cc0347fddb71bdb409374bdf9 commit c901f931f6eb102cc0347fddb71bdb409374bdf9 Author: Roman Bogorodskiy <novel@FreeBSD.org> AuthorDate: 2026-09-20 14:41:55 +0000 Commit: Roman Bogorodskiy <novel@FreeBSD.org> CommitDate: 2026-09-28 18:39:05 +0000 bhyve: fix boot device ordering EDK2's QemuBootOrderLib inspects the bootorder file provided via fw_cfg and requires it to be NUL-terminated. Otherwise, it rejects the supplied bootorder and falls back to its default boot order. Currently, bhyve registers bootorder with qemu_fwcfg_add_file() using bootorder_len returned by open_memstream(), which excludes the trailing NUL byte. Fix that by passing bootorder_len + 1 to qemu_fwcfg_add_file() so the fw_cfg payload is properly NUL-terminated. PR: 279720 Reviewed by: markj Found with: codex (gpt-5.6-sol) MFC after: 1 week Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D59859 (cherry picked from commit b27a221ef79d6d77d6173897f12118e1126453d8) usr.sbin/bhyve/pci_emul.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)