Bug 279720 - virtualization/bhyve: it does not boot from the slot where the parameter bootindex=1 is declared.
Summary: virtualization/bhyve: it does not boot from the slot where the parameter boot...
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: bhyve (show other bugs)
Version: Unspecified
Hardware: Any Any
: --- Affects Only Me
Assignee: Corvin Köhne
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-06-13 19:50 UTC by mario felicioni
Modified: 2026-09-28 18:51 UTC (History)
4 users (show)

See Also:
novel: mfc-stable15+
novel: mfc-stable14+


Attachments
edk2 debug log for ubuntu but with bootindex not working (93.55 KB, text/plain)
2025-09-17 15:03 UTC, Roman Bogorodskiy
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description mario felicioni 2024-06-13 19:50:58 UTC
Hello to everyone.

I want to show you what it seems to be a bug in bhyve. Please evaluate the situation below :

/usr/sbin/./bhyve-win -S -c sockets=4,cores=2,threads=1 -m 8G -w -H \
-S -c sockets=4,cores=2,threads=1 -m 8G -w -H \
-s 0,hostbridge \
-s 1,ahci-hd,/mnt/$vmdisk0'p2'/bhyve/img/Windows/Windows11.img,bootindex=1 \
-s 2,ahci-hd,/dev/$vmdisk5 \
-s 8:0,passthru,2/0/0 \
-s 8:1,passthru,2/0/1 \
-s 8:2,passthru,2/0/2 \
-s 8:3,passthru,2/0/3 \
-s 13,virtio-net,tap18 \
-s 29,fbuf,tcp=0.0.0.0:5918,w=1600,h=950,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
vm0:18 < /dev/null & sleep 2 && vncviewer 0:18

as you can see I have attached all the slots of my GPU. Infact my Nvidia GPU has 4 slots :

02:00.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] (rev a1)
02:00.1 Audio device: NVIDIA Corporation TU102 High Definition Audio Controller (rev a1)
02:00.2 USB controller: NVIDIA Corporation TU102 USB 3.1 Host Controller (rev a1)
02:00.3 Serial bus controller: NVIDIA Corporation TU102 USB Type-C UCSI Controller (rev a1)

As you can see my GPU has one USB controller and one USB port mini integrated with the controller. Well,I attached a pen drive on the port of the USB controller and then I tried to boot the Windows 11 vm. What's happened ? that,despite I have declared that I want boot the Windows11 vm at slot 1 by adding the parameter bootindex=1,instead,it booted from the USB stick attached to the mini USB port of the USB controller (yes because on that USB stick I have installed Windows 7,so it can boot). This seems to be a bug,because it does not do what I want. And it is easy to reproduce. Just put bootindex=1 to every OS that you want to boot and then attach a bootable USB stick to the usb port of the GPU (it should be recent,it should be a GPU with the USB controller integrated,but the latest high quality GPU models have it),and just run the bhyve VM and you will see that it will not boot the OS you want.
Comment 1 Corvin Köhne freebsd_committer freebsd_triage 2024-06-14 06:39:14 UTC
From https://man.freebsd.org/cgi/man.cgi?bhyve:

		   bootindex=index
			       Add  the	 device	 to the	bootorder at index.  A
			       fwcfg file is used to  specify  the  bootorder.
			       The  guest  firmware may	ignore or doesn't sup-
			       port this fwcfg file.  In that case, this  fea-
			       ture doesn't work as expected.

EDK2 for bhyve currently doesn't support the bootorder fwcfg file. Therefore, it's ignored and uses it's default bootorder.

I'm going to send a patch to upstream EDK2.
Comment 2 Corvin Köhne freebsd_committer freebsd_triage 2024-06-14 06:51:05 UTC
https://github.com/tianocore/edk2/pull/5774

When it's merged upstream, we can backport it to our EDK2 port.
Comment 3 Corvin Köhne freebsd_committer freebsd_triage 2024-06-28 09:21:21 UTC
https://reviews.freebsd.org/D45768
Comment 4 commit-hook freebsd_committer freebsd_triage 2024-07-01 13:03:53 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=54eaea1ccd17ac679d065531243253b8bc682851

commit 54eaea1ccd17ac679d065531243253b8bc682851
Author:     Corvin Köhne <corvink@FreeBSD.org>
AuthorDate: 2024-06-28 06:54:01 +0000
Commit:     Corvin Köhne <corvink@FreeBSD.org>
CommitDate: 2024-07-01 13:01:49 +0000

    sysutils/edk2: make bhyve's bootorder feature work

    Bhyve supports a bootindex property for bootable devices. When adding it
    to at least one device, bhyve creates a bootorder and passes it over the
    Qemu FwCfg interface to the guest. The guest firmware has to pick up the
    FwCfg file and has to apply the bootorder. Otherwise, the bootindex
    property won't have any effect. Recently, a commit was merged by EDKII
    to support that feature and we can simply cherry-pick it.

    PR:                     279720
    Reviewed by:            manu
    Approved by:            manu
    Sponsored by:           Beckhoff Automation GmbH & Co. KG
    Differential Revision:  https://reviews.freebsd.org/D45768

 sysutils/edk2/Makefile | 3 ++-
 sysutils/edk2/distinfo | 4 +++-
 2 files changed, 5 insertions(+), 2 deletions(-)
Comment 5 mario felicioni 2024-07-01 14:20:45 UTC
Can you explain to me a little bit how to apply that patch ? Can you confirm that the approach that I'm going to take is the correct one ?

I've downloaded this package :

https://cgit.freebsd.org/ports/snapshot/ports-54eaea1ccd17ac679d065531243253b8bc682851.tar.gz

and I've extracted the port "edk2" directory from the package and I will exchange it with what I already have,not patched. And then,I will recompile it doing make / make install.
Comment 6 Corvin Köhne freebsd_committer freebsd_triage 2024-07-01 14:38:11 UTC
It's already merged. So, you can simply install it from ports (or wait until it's build and available on the official package server):

https://docs.freebsd.org/en/books/handbook/ports/#ports-using

In short:

git clone https://github.com/freebsd/freebsd-ports /usr/ports
cd /usr/ports/sysutils/edk2
FLAVOR=bhyve make reinstall
Comment 7 mario felicioni 2024-07-01 15:07:41 UTC
ok. I'm very curious to see if this patch will break or not the correct passthru of my Geforce RTX 2080 ti in a Windows VM. I still don't know why it's working and I'm also scared that one of the next patches that I will apply,will break it.
Comment 8 mario felicioni 2024-07-01 16:03:21 UTC
Sorry,your patch does not work for me :

.....
make[2]: Leaving directory '/usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/X64/MdeModulePkg/Universal/Variable/RuntimeDxe/VariableRuntimeDxe'

Fd File Name:BHYVE (/usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/BHYVE.fd)

Generate Region at Offset 0x0
   Region Size = 0x40000
   Region Name = DATA

Generate Region at Offset 0x40000
   Region Size = 0x1000
   Region Name = None

Generate Region at Offset 0x41000
   Region Size = 0x1000
   Region Name = DATA

Generate Region at Offset 0x42000
   Region Size = 0x42000
   Region Name = None

Generate Region at Offset 0x84000
   Region Size = 0x348000
   Region Name = FV

Generating FVMAIN_COMPACT FV

Generating PEIFV FV
####
Generating DXEFV FV
#########
Generate Region at Offset 0x3CC000
   Region Size = 0x34000
   Region Name = FV

Generating SECFV FV
#
Fd File Name:BHYVE_VARS (/usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/BHYVE_VARS.fd)

Generate Region at Offset 0x0
   Region Size = 0x40000
   Region Name = DATA

Generate Region at Offset 0x40000
   Region Size = 0x1000
   Region Name = None

Generate Region at Offset 0x41000
   Region Size = 0x1000
   Region Name = DATA

Generate Region at Offset 0x42000
   Region Size = 0x42000
   Region Name = None

Fd File Name:BHYVE_CODE (/usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/BHYVE_CODE.fd)

Generate Region at Offset 0x0
   Region Size = 0x348000
   Region Name = FV

Generate Region at Offset 0x348000
   Region Size = 0x34000
   Region Name = FV

Fd File Name:MEMFD (/usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/MEMFD.fd)

Generate Region at Offset 0x0
   Region Size = 0x6000
   Region Name = None

Generate Region at Offset 0x6000
   Region Size = 0x1000
   Region Name = None

Generate Region at Offset 0x7000
   Region Size = 0x1000
   Region Name = None
Padding region starting from offset 0x8000, with size 0x8000

Generate Region at Offset 0x8000
   Region Size = 0x8000
   Region Name = None

Generate Region at Offset 0x10000
   Region Size = 0x10000
   Region Name = None

Generate Region at Offset 0x20000
   Region Size = 0xE0000
   Region Name = FV

Generate Region at Offset 0x100000
   Region Size = 0xB00000
   Region Name = FV

GUID cross reference file can be found at /usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/Guid.xref

FV Space Information
SECFV [5%Full] 212992 (0x34000) total, 11808 (0x2e20) used, 201184 (0x311e0) free
PEIFV [11%Full] 917504 (0xe0000) total, 106104 (0x19e78) used, 811400 (0xc6188) free
DXEFV [45%Full] 11534336 (0xb00000) total, 5210672 (0x4f8230) used, 6323664 (0x607dd0) free
FVMAIN_COMPACT [41%Full] 3440640 (0x348000) total, 1437760 (0x15f040) used, 2002880 (0x1e8fc0) free

- Done -
Build end time: 17:34:22, Jul.01 2024
Build total time: 00:01:29

===>  Staging for edk2-bhyve-g202308_5
===>   Generating temporary packing list
/bin/mkdir -p /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/edk2-bhyve/
install  -m 0644 /usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/BHYVE.fd /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/edk2-bhyve/BHYVE_UEFI.fd
install  -m 0644 /usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/BHYVE_CODE.fd /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/edk2-bhyve/BHYVE_UEFI_CODE.fd
install  -m 0644 /usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/BHYVE_VARS.fd /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/edk2-bhyve/BHYVE_UEFI_VARS.fd
install  -m 0644 /usr/ports/sysutils/edk2/work-bhyve/Build/BhyveX64/RELEASE_GCC5/FV/BHYVE_CODE.fd /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/edk2-bhyve/BHYVE_UEFI.fd
/bin/mkdir -p /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/uefi-firmware/
install -l rs /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/edk2-bhyve/BHYVE_UEFI.fd /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/uefi-firmware/BHYVE_UEFI.fd
install -l rs /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/edk2-bhyve/BHYVE_UEFI_CODE.fd /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd
install -l rs /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/edk2-bhyve/BHYVE_UEFI_VARS.fd /usr/ports/sysutils/edk2/work-bhyve/stage//usr/local/share/uefi-firmware/BHYVE_UEFI_VARS.fd
====> Compressing man pages (compress-man)
===>  Installing for edk2-bhyve-g202308_5
===>  Checking if edk2-bhyve is already installed
===>   Registering installation for edk2-bhyve-g202308_5
Installing edk2-bhyve-g202308_5...

[root@marietto /usr/ports/sysutils/edk2]==> pkg remove edk2-bhyve      

Updating database digests format: 100%
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 2 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
	bhyve-firmware: 1.0_2
	edk2-bhyve: g202308_5

Number of packages to be removed: 2

The operation will free 7 MiB.

Proceed with deinstalling packages? [y/N]: y
[1/2] Deinstalling bhyve-firmware-1.0_2...
[1/2] Deleting files for bhyve-firmware-1.0_2: 100%
[2/2] Deinstalling edk2-bhyve-g202308_5...
[2/2] Deleting files for edk2-bhyve-g202308_5: 100%

[root@marietto /usr/ports/sysutils/edk2]==> FLAVOR=bhyve make reinstall

===>  Deinstalling for edk2-bhyve
===>   edk2-bhyve not installed, skipping
===>  Installing for edk2-bhyve-g202308_5
===>  Checking if edk2-bhyve is already installed
===>   Registering installation for edk2-bhyve-g202308_5
Installing edk2-bhyve-g202308_5...

I launch the vm using these params :

/usr/sbin/./bhyve-win -S -c sockets=4,cores=2,threads=1 -m 8G -w -H \
-S -c sockets=4,cores=2,threads=1 -m 8G -w -H \
-s 0,hostbridge \
-s 1,ahci-hd,/mnt/$vmdisk0'p2'/bhyve/img/Windows/11/Windows11.img,bootindex=1 \
-s 2,ahci-hd,/dev/$vmdisk5 \
-s 8:0,passthru,2/0/0 \
-s 8:1,passthru,2/0/1 \
-s 8:2,passthru,2/0/2 \
-s 8:3,passthru,2/0/3 \
-s 13,virtio-net,tap12 \
-s 29,fbuf,tcp=0.0.0.0:5912,w=1600,h=950,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
vm0:12 < /dev/null & sleep 2 && vncviewer 0:12

this is what happens :

vm exit[0]
	reason		VMX
	rip		0x00000000000f0015
	inst_length	3
	status		0
	exit_reason	2 (Triple fault)
	qualification	0x0000000000000000
	inst_type		0
	inst_error		0
Comment 9 Corvin Köhne freebsd_committer freebsd_triage 2024-07-02 07:21:42 UTC
Maybe try a simpler bhyve call without passthrough devices.
Comment 10 Mark Linimon freebsd_committer freebsd_triage 2024-09-30 05:28:09 UTC
^Triage: assign to committer.
Comment 11 Michael Osipov freebsd_committer freebsd_triage 2025-08-05 09:29:47 UTC
Does this really work now?

I have installed: edk2-bhyve-g202308_5, vm-bhyve-1.6.2_1

Patched vm-bhyve locally to add bootindex=1 to ahci-cd:
Aug. 05 11:23:09: initialising
Aug. 05 11:23:09:  [loader: uefi]
Aug. 05 11:23:09:  [cpu: 8]
Aug. 05 11:23:09:  [memory: 12G]
Aug. 05 11:23:09:  [hostbridge: standard]
Aug. 05 11:23:09:  [com ports: com1]
Aug. 05 11:23:09:  [uuid: 291fed2d-71d4-11f0-8490-b499bac04fbe]
Aug. 05 11:23:09:  [debug mode: yes]
Aug. 05 11:23:09:  [primary disk: disk0.img]
Aug. 05 11:23:09:  [primary disk dev: file]
Aug. 05 11:23:09: initialising network device tap1
Aug. 05 11:23:09: adding tap1 -> bce1bridge (VL-G-DE-SP02-4181 addm)
Aug. 05 11:23:09: bring up tap1 -> bce1bridge (VL-G-DE-SP02-4181 addm)
Aug. 05 11:23:09: dynamically allocated port 5901 for vnc connections
Aug. 05 11:23:09: booting
Aug. 05 11:23:09:  [bhyve options: -c 8 -m 12G -AHPw -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/usr/local/bhyve/kinexon/uefi-vars.fd -K de -U 291fed2d-71d4-11f0-8490-b499bac04fbe -u]
Aug. 05 11:23:09:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,nvme,/usr/local/bhyve/kinexon/disk0.img -s 5:0,virtio-net,tap1,mac=58:9c:fc:09:5e:14 -s 6:0,fbuf,tcp=127.0.0.1:5901,w=1280,h=1024,wait -s 7:0,xhci,tablet]
Aug. 05 11:23:09:  [bhyve console: -l com1,/dev/nmdm-kinexon.1A]
Aug. 05 11:23:09:  [bhyve iso device: -s 3:0,ahci-cd,/usr/local/bhyve/.iso/void-live-x86_64-20250202-base.iso,ro,bootindex=1]
Aug. 05 11:23:09: starting bhyve (run 1)

It does not boot from the ISO, reset from UEFI shell I see the following when entering UEFI config:
   Boot Manager Menu

   UEFI bhyve-NVMe NVME-4-0
   UEFI PXEv4 (MAC:589CFC095E14)
   UEFI PXEv6 (MAC:589CFC095E14)
   EFI Internal Shell
   UEFI BHYVE SATA DVD ROM BHYVE-F134-08D4-1073


From the shell:
Option: 00. Variable: Boot0001
  Desc    - UEFI bhyve-NVMe NVME-4-0
  DevPath - PciRoot(0x0)/Pci(0x4,0x0)/NVMe(0x1,01-00-D6-E9-20-FC-9C-58)
  Optional- Y
  00000000: 4E AC 08 81 11 9F 59 4D-85 0E E2 1A 52 2C 59 B2  *N.....YM....R,Y.*
Option: 01. Variable: Boot0002
  Desc    - UEFI PXEv4 (MAC:589CFC095E14)
  DevPath - PciRoot(0x0)/Pci(0x5,0x0)/MAC(589CFC095E14,0x1)/IPv4(0.0.0.0)
  Optional- Y
  00000000: 4E AC 08 81 11 9F 59 4D-85 0E E2 1A 52 2C 59 B2  *N.....YM....R,Y.*
Option: 02. Variable: Boot0003
  Desc    - UEFI PXEv6 (MAC:589CFC095E14)
  DevPath - PciRoot(0x0)/Pci(0x5,0x0)/MAC(589CFC095E14,0x1)/IPv6(0000:0000:0000:0000:0000:0000:0000:0000)
  Optional- Y
  00000000: 4E AC 08 81 11 9F 59 4D-85 0E E2 1A 52 2C 59 B2  *N.....YM....R,Y.*
Option: 03. Variable: Boot0004
  Desc    - EFI Internal Shell
  DevPath - Fv(7CB8BDC9-F8EB-4F34-AAEA-3EE4AF6516A1)/FvFile(7C04A583-9E3E-4F1C-AD65-E05268D0B4D1)
  Optional- N
Option: 04. Variable: Boot0000
  Desc    - UiApp
  DevPath - Fv(7CB8BDC9-F8EB-4F34-AAEA-3EE4AF6516A1)/FvFile(462CAA21-7614-4503-836E-8AB6F4662331)
  Optional- N
Option: 05. Variable: Boot0005
  Desc    - UEFI BHYVE SATA DVD ROM BHYVE-F134-08D4-1073
  DevPath - PciRoot(0x0)/Pci(0x3,0x0)/Sata(0x0,0xFFFF,0x0)
  Optional- Y
  00000000: 4E AC 08 81 11 9F 59 4D-85 0E E2 1A 52 2C 59 B2  *N.....YM....R,Y.*

What did I miss here?
Comment 12 Corvin Köhne freebsd_committer freebsd_triage 2025-08-05 14:35:33 UTC
(In reply to Michael Osipov from comment #11)
Which version of bhyve are you using?
Comment 13 Michael Osipov freebsd_committer freebsd_triage 2025-08-05 17:47:32 UTC
(In reply to Corvin Köhne from comment #12)

I am on
FreeBSD deblndw013x.innomotics.net 13.5-STABLE FreeBSD 13.5-STABLE 0106e5d72 GENERIC amd64
Comment 14 Corvin Köhne freebsd_committer freebsd_triage 2025-08-06 06:27:06 UTC
The bootorder should be supported by 13.5. You have to set lpc.fwcfg to qemu:

-s 31,lpc,fwcfg=qemu

or

-o lpc.fwcfg=qemu
Comment 15 Michael Osipov freebsd_committer freebsd_triage 2025-08-06 07:40:39 UTC
(In reply to Corvin Köhne from comment #14)

Let me double check...
Comment 16 Michael Osipov freebsd_committer freebsd_triage 2025-08-06 07:56:54 UTC
(In reply to Corvin Köhne from comment #14)

Success, it does indeed work:
>    Boot Manager Menu
> 
>    UEFI BHYVE SATA DVD ROM BHYVE-F134-08D4-1073
>    UEFI bhyve-NVMe NVME-4-0
>    UEFI PXEv4 (MAC:589CFC095E14)
>    UEFI PXEv6 (MAC:589CFC095E14)
>    EFI Internal Shell

Void Linux was booted from ISO. Both configs work, even the VNC screen is larger than before. Insane, thank you!

I need to raise a ticket with vm-bhyve.

Side question: Is there any drawback to set fwcfg to qemu permanently?
Comment 17 Corvin Köhne freebsd_committer freebsd_triage 2025-08-06 09:25:58 UTC
(In reply to Michael Osipov from comment #16)

> Side question: Is there any drawback to set fwcfg to qemu permanently?

The default fwcfg=bhyve is mostly a nop. It was only used to pass the number of vcpus to OVMF. This information is also passed by fwcfg=qemu to the guest and picked up by OVMF since edk2-stable202205.
Comment 18 Michael Osipov freebsd_committer freebsd_triage 2025-08-06 10:30:46 UTC
(In reply to Corvin Köhne from comment #17)

Wouldn't it make sense to change the default value from bhyve to qemu on 15 then?
Comment 19 Michael Osipov freebsd_committer freebsd_triage 2025-08-06 12:03:56 UTC
I have created to PRs for vm-bhyve:
* https://github.com/freebsd/vm-bhyve/pull/22
* https://github.com/freebsd/vm-bhyve/pull/21
Comment 20 Michael Osipov freebsd_committer freebsd_triage 2025-08-06 17:02:44 UTC
I think we can close this since I have verified it to work. I expect that 14/15 do work as well. ckoehne@ an objections?
Comment 21 Corvin Köhne freebsd_committer freebsd_triage 2025-08-07 08:17:08 UTC
(In reply to Michael Osipov from comment #20)
Yes, the patches are included in 14 and 15 too. So, it should work there too.
Comment 22 Roman Bogorodskiy freebsd_committer freebsd_triage 2025-09-12 19:32:47 UTC
This feature doesn't work for me still.

E.g. I boot a VM with:

sudo bhyve -c 2 -m 4096 -A -I -u -H -P \
-s 0:0,hostbridge \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/var/lib/libvirt/bhyve/nvram/ubuntu2504_VARS.fd,fwcfg=qemu \
-s 2:0,xhci,tablet -s 1:0,lpc,fwcfg=qemu \
-s 3:0,ahci,hd:/data/img/ubuntu2504.img,bootindex=2,cd:/data/isos/ubuntu-25.04-live-server-amd64.iso,bootindex=1 \
-s 4:0,virtio-net,tap4,mnac=52:54:00:47:f4:dc -s 5:0,fbuf,tcp=127.0.0.1:5920 \
-l com1,stdio \
ubuntu2504

That works. Then for the next boot I want to boot from the disk image, I do:

sudo bhyve -c 2 -m 4096 -A -I -u -H -P \
-s 0:0,hostbridge \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/var/lib/libvirt/bhyve/nvram/ubuntu2504_VARS.fd,fwcfg=qemu \
-s 2:0,xhci,tablet -s 1:0,lpc,fwcfg=qemu \
-s 3:0,ahci,hd:/data/img/ubuntu2504.img,bootindex=2,cd:/data/isos/ubuntu-25.04-live-server-amd64.iso,bootindex=3 \
-s 4:0,virtio-net,tap4,mnac=52:54:00:47:f4:dc -s 5:0,fbuf,tcp=127.0.0.1:5920 \
-l com1,stdio \
ubuntu2504

It doesn't work, it still boots from CD.

Additionally, sections about fwcfg in bhyve(8) seem confusing to me. It's not obvious if fwcfg should be a part of `-l bootrom` configuration or a part of `-s 31,lpc` configuration (or both?).
Comment 23 Corvin Köhne freebsd_committer freebsd_triage 2025-09-15 11:42:26 UTC
(In reply to Roman Bogorodskiy from comment #22)

> -s 3:0,ahci,hd:/data/img/ubuntu2504.img,bootindex=2,cd:/data/isos/ubuntu-25.04-live-server-amd64.iso,bootindex=1 \

The hypervisor has to pass the bootpath to the guest firmware. Currently, bhyve uses a very simple bootpath, just the PCI address. This works for many use cases but not for yours where multiple boot devices are attached to a single PCI device.
Comment 24 Roman Bogorodskiy freebsd_committer freebsd_triage 2025-09-15 18:00:38 UTC
(In reply to Corvin Köhne from comment #23)

Hm, that's very strange and I don't quite get what's going on there. What I tried:

> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/var/lib/libvirt/bhyve/nvram/ubuntu2504_VARS.fd
> ...
> -s 3:0,ahci-hd,/data/img/ubuntu2504.img,bootindex=2 -s 4:0,ahci-cd,/data/isos/ubuntu-25.04-live-server-amd64.iso,bootindex=1

This didn't work, system was booting from HD, despite CD having "bootindex=1".

Then I tried the same with VARS.fd removed:

> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd
> ...
> -s 3:0,ahci-hd,/data/img/ubuntu2504.img,bootindex=2 -s 4:0,ahci-cd,/data/isos/ubuntu-25.04-live-server-amd64.iso,bootindex=1

Yay, this works! VM boots from CD (bootindex=1).

Now I swap the indexes and expect it to boot from HD. It still boots from CD. I paste a complete command for once to avoid any confusing:

bhyve -c 2 -m 4096 -A -I -u -H -P \                                                                                                                                                                                                        
-s 0:0,hostbridge \                                                                                                                                                                                                                                                     
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \                                                                                                                                                                                                               
-s 2:0,xhci,tablet -s 1:0,lpc,fwcfg=qemu \                                                                                                                                                                                                                              
-s 3:0,ahci-hd,/data/img/ubuntu2504.img,bootindex=1 -s 4:0,ahci-cd,/data/isos/ubuntu-25.04-live-server-amd64.iso,bootindex=2 \                                                                                                                                          
-s 5:0,fbuf,tcp=127.0.0.1:5920 \                                                                                                                                                                                                                                        
-l com1,stdio \                                                                                                                                                                                                                                                         
ubuntu2504 

In console I see:

fbuf frame buffer base: 0x280638200000 [sz 33554432]                                                                                                                                                                                                                    
0                                                                                                                                                                                                                                                                       
0                                                                                                                                                                                                                                                                       
BdsDxe: loading Boot0001 "UEFI BHYVE SATA DVD ROM BHYVE-D1CE-3E63-5ED8" from PciRoot(0x0)/Pci(0x4,0x0)/Sata(0x0,0xFFFF,0x0)                                                                                                                                             
BdsDxe: starting Boot0001 "UEFI BHYVE SATA DVD ROM BHYVE-D1CE-3E63-5ED8" from PciRoot(0x0)/Pci(0x4,0x0)/Sata(0x0,0xFFFF,0x0)                                                                                                                                            
error: can't find command `grub_platform'.                                                                                                                                                                                                                              
EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path

It looks like it didn't even try to boot from HD.

PS I was about to submit, but double-checked the man page, and it says:

Add the device to the bootorder at index.  A fwcfg file is
                 used to specify the bootorder.

It's not quite clear, what does "fwcfg file" mean here, does it mean "lpc,fwcfg=qemu", or the VARS.fd file, or something else?
Comment 25 Corvin Köhne freebsd_committer freebsd_triage 2025-09-16 06:17:13 UTC
(In reply to Roman Bogorodskiy from comment #24)

> Hm, that's very strange and I don't quite get what's going on there. What I tried:

The VARS file saves UEFI variables. There's a BootOrder variable which determines the boot order. So, by removing the VARS file, you're falling back to the default BootOrder which might be different.

> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \

You're missing the fwcfg=qemu option here, use:

> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=qemu \

> Add the device to the bootorder at index. A fwcfg file is used to specify the bootorder.
> 
> It's not quite clear, what does "fwcfg file" mean here, does it mean "lpc,fwcfg=qemu", or the VARS.fd file, or something else?

This sentence should give you a hint, that a fwcfg file is required, so you need the "fwcfg=qemu" option (at "-l bootrom,<rom>,fwcfg=qemu" not at "-s xxx,lpc").

FwCfg is an interface invented by QEMU to pass data from the hypervisor to the guest. Data is provided in so called files, e.g. the file "opt/bhyve/hw.ncpu" contains the vCPU count, "etc/table-loader" contains ACPI tables and "bootorder" contains the BootOrder. Most Linux distributions expose those files by sysfs in /sys/firmware/fw_cfg [1]. You can check it in your VM to make sure the bootorder was exposed by bhyve.

To get more information about it, it might be worth checking the QEMU and Linux docs about it.

Btw. you can even add your own fwcfg files to bhyve, if you like to:

>       -f name,[string|file]=data
>		   Add a fw_cfg	file name  to  the  fw_cfg  interface.	 If  a
>		   string is specified,	the fw_cfg file	contains the string as
>		   data.   If  a  file	is specified, bhyve reads the file and
>		   adds	the file content as fw_cfg data.

This is useful to pass custom data to the guest, e.g. Fedora Core OS uses it to provision the guest [2].

[1] https://elixir.bootlin.com/linux/v6.16.7/source/drivers/firmware/Kconfig#L122
[2] https://docs.fedoraproject.org/en-US/fedora-coreos/provisioning-qemu/#_setting_up_a_new_vm
Comment 26 Roman Bogorodskiy freebsd_committer freebsd_triage 2025-09-16 09:22:10 UTC
(In reply to Corvin Köhne from comment #25)

That doesn't work still.

Command I run:

sudo bhyve -c 2 -m 4096 -A -I -u -H -P \
-s 0:0,hostbridge \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/var/lib/libvirt/bhyve/nvram/ubuntu2504_VARS.fd,fwcfg=qemu \
-s 2:0,xhci,tablet -s 1:0,lpc \
-s 3:0,ahci-hd,/data/img/ubuntu2504.img,bootindex=2 -s 4:0,ahci-cd,/data/isos/ubuntu-25.04-live-server-amd64.iso,bootindex=1 \
-s 5:0,fbuf,tcp=127.0.0.1:5920 \
-l com1,stdio -s 6:0,virtio-net,tap4,mac=52:54:00:47:f4:dc \
ubuntu2504

I expect it to boot from ISO, it boots from HD instead.

In the guest, I do not see /sys/firmware/fw_cfg. However, I see qemu_fw_cfg.

root@ubuntu2504:/sys/firmware# ls -1
acpi
dmi
efi
memmap
qemu_fw_cfg
root@ubuntu2504:/sys/firmware# cat qemu_fw_cfg/
by_key/  by_name/ rev      
root@ubuntu2504:/sys/firmware# cat qemu_fw_cfg/by_name/bootorder/raw 
/pci@i0cf8/pci@4,0
/pci@i0cf8/pci@3,0
root@ubuntu2504:/sys/firmware# 

This order here seems to be valid.
Comment 27 Corvin Köhne freebsd_committer freebsd_triage 2025-09-16 11:40:53 UTC
(In reply to Roman Bogorodskiy from comment #26)
What's your version of edk2-bhyve?
Comment 28 Roman Bogorodskiy freebsd_committer freebsd_triage 2025-09-16 12:22:17 UTC
(In reply to Corvin Köhne from comment #27)

It's g202508 from PR #285554.

$ pkg which /usr/local/share/uefi-firmware/BHYVE_UEFI.fd
/usr/local/share/uefi-firmware/BHYVE_UEFI.fd was installed by package edk2-bhyve-g202508
$
Comment 29 Corvin Köhne freebsd_committer freebsd_triage 2025-09-16 14:40:13 UTC
(In reply to Roman Bogorodskiy from comment #28)
Just out of curiosity, try to remove the VARS file from your bhyve command.
Comment 30 Roman Bogorodskiy freebsd_committer freebsd_triage 2025-09-16 15:03:48 UTC
(In reply to Corvin Köhne from comment #29)

Trying two commands:

1. bhyve -c 2 -m 4096 -A -I -u -H -P \
-s 0:0,hostbridge \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=qemu \
-s 2:0,xhci,tablet -s 1:0,lpc \
-s 3:0,ahci-hd,/data/img/ubuntu2504.img,bootindex=2 -s 4:0,ahci-cd,/data/isos/ubuntu-25.04-live-server-amd64.iso,bootindex=1 \
-s 5:0,fbuf,tcp=127.0.0.1:5920 \
-l com1,stdio -s 6:0,virtio-net,tap4,mac=52:54:00:47:f4:dc \
ubuntu2504

Boots from CD, works as expected.

No changes in qemu_fw_cfg/by_name/bootorder/raw.

2. bhyve -c 2 -m 4096 -A -I -u -H -P \                                                                                                                                                                                                        
-s 0:0,hostbridge \                                                                                                                                                                                                                                                     
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=qemu \                                                                                                                                                                                                    
-s 2:0,xhci,tablet -s 1:0,lpc \                                                                                                                                                                                                                                         
-s 3:0,ahci-hd,/data/img/ubuntu2504.img,bootindex=1 -s 4:0,ahci-cd,/data/isos/ubuntu-25.04-live-server-amd64.iso,bootindex=2 \                                                                                                                                          
-s 5:0,fbuf,tcp=127.0.0.1:5920 \                                                                                                                                                                                                                                        
-l com1,stdio -s 6:0,virtio-net,tap4,mac=52:54:00:47:f4:dc \                                                                                                                                                                                                            
ubuntu2504                                                   

Expected to boot from HD, boots from CD instead.

qemu_fw_cfg is changed:

root@ubuntu-server:/sys/firmware/qemu_fw_cfg/by_name# cat bootorder/raw 
/pci@i0cf8/pci@3,0
/pci@i0cf8/pci@4,0
root@ubuntu-server:/sys/firmware/qemu_fw_cfg/by_name#

Looks like qemu_fw_cfg is always correct, e.g. corresponds to what's requested via cmdline.
Comment 31 Corvin Köhne freebsd_committer freebsd_triage 2025-09-17 06:32:17 UTC
(In reply to Roman Bogorodskiy from comment #30)
Could you try to build a debug version of the firmware and share the log? Therefore, build the bhyve flavor of sysutils/edk2 on your own [1][2] and apply following patch:

diff --git a/sysutils/edk2/Makefile b/sysutils/edk2/Makefile
index 5b1947c88b2d..e0636acf7512 100644
--- a/sysutils/edk2/Makefile
+++ b/sysutils/edk2/Makefile
@@ -116,7 +116,8 @@ ONLY_FOR_ARCHS_REASON=      Bhyve only runs on x64
 PLAT=          bhyve
 PLAT_ARCH=     X64
 PLAT_ARGS+=    -D SECURE_BOOT_ENABLE=TRUE -D TPM2_ENABLE=TRUE
-PLAT_TARGET=   RELEASE
+PLAT_ARGS+=    -D DEBUG_ON_SERIAL_PORT=TRUE
+PLAT_TARGET=   DEBUG
 PLATFILE=      OvmfPkg/Bhyve/BhyveX64.dsc
 PLAT_RESULT=   BhyveX64/${PLAT_TARGET}_GCC5/FV/BHYVE.fd
 PLAT_RESULT_CODE=      BhyveX64/${PLAT_TARGET}_GCC5/FV/BHYVE_CODE.fd

[1] https://docs.freebsd.org/en/books/handbook/ports/#ports-using-installation-methods
[2] https://docs.freebsd.org/en/books/handbook/ports/#_installing_ports

Use FLAVOR=bhyve make install instead of just make install.
Comment 32 Roman Bogorodskiy freebsd_committer freebsd_triage 2025-09-17 15:03:03 UTC
Created attachment 263879 [details]
edk2 debug log for ubuntu but with bootindex not working

(In reply to Corvin Köhne from comment #31)

I assume we need to get back to a version of the command which uses the VARS file.

So I'm going back to this command:

bhyve -c 2 -m 4096 -A -I -u -H -P \
-s 0:0,hostbridge \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/var/lib/libvirt/bhyve/nvram/ubuntu2504_VARS.fd,fwcfg=qemu \
-s 2:0,xhci,tablet -s 1:0,lpc \
-s 3:0,ahci-hd,/data/img/ubuntu2504.img,bootindex=2 -s 4:0,ahci-cd,/data/isos/ubuntu-25.04-live-server-amd64.iso,bootindex=1 \
-s 5:0,fbuf,tcp=127.0.0.1:5920 \
-l com1,stdio -s 6:0,virtio-net,tap4,mac=52:54:00:47:f4:dc \
ubuntu2504 2>&1 | tee ~/ubuntu_edk2_bootindex.log

Which is expected to boot from CD, but boots from HD instead.

Attaching the log.
Comment 33 Roman Bogorodskiy freebsd_committer freebsd_triage 2026-09-20 14:39:46 UTC
I have asked to use codex (gpt-5.6-sol) to localize the issue, and, interestingly, it came up with a one line fix:

--- a/usr.sbin/bhyve/pci_emul.c
+++ b/usr.sbin/bhyve/pci_emul.c
@@ -1516,7 +1516,7 @@ init_bootorder(void)
        }
        fclose(fp);

-       return (qemu_fwcfg_add_file("bootorder", bootorder_len, bootorder));
+       return (qemu_fwcfg_add_file("bootorder", bootorder_len + 1, bootorder));
 }

Its explanation is the original length trims '\0', and EDK2 rejects a configuration which is not properly terminated. To check this claim, sysutils/edk2 was rebuilt with the following patch (and debugging enabled):

--- OvmfPkg/Library/QemuBootOrderLib/QemuBootOrderLib.c.orig    2025-08-22 00:00:00 UTC                                                                                                                                                                                                                                
+++ OvmfPkg/Library/QemuBootOrderLib/QemuBootOrderLib.c                                                                                                                                                                                                                                                                
@@ -1599,7 +1599,14 @@ ConnectDevicesFromQemu (                                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                                       
   QemuFwCfgSelectItem (FwCfgItem);                                                                                                                                                                                                                                                                                    
   QemuFwCfgReadBytes (FwCfgSize, FwCfg);                                                                                                                                                                                                                                                                              
   if (FwCfg[FwCfgSize - 1] != '\0') {                                                                                                                                                                                                                                                                                 
+    DEBUG ((                                                                                                                                                                                                                                                                                                          
+      DEBUG_ERROR,                                                                                                                                                                                                                                                                                                    
+      "%a: bootorder is not NUL-terminated, size=%Lu, last=0x%02x\n",                                                                                                                                                                                                                                                 
+      __func__,                                                                                                                                                                                                                                                                                                       
+      (UINT64)FwCfgSize,                                                                                                                                                                                                                                                                                              
+      (UINT8)FwCfg[FwCfgSize - 1]                                                                                                                                                                                                                                                                                     
+      ));                                                                                                                                                                                                                                                                                                             
     Status = RETURN_INVALID_PARAMETER;                                                                                                                                                                                                                                                                                
     goto FreeFwCfg;                                                                                                                                                                                                                                                                                                   
   }                                                                                                                                                                                                                                                                                                                   
@@ -1745,7 +1752,14 @@ StoreQemuBootOrder (                                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                                                                       
   QemuFwCfgSelectItem (FwCfgItem);                                                                                                                                                                                                                                                                                    
   QemuFwCfgReadBytes (FwCfgSize, FwCfg);                                                                                                                                                                                                                                                                              
   if (FwCfg[FwCfgSize - 1] != '\0') {                                                                                                                                                                                                                                                                                 
+    DEBUG ((                                                                                                                                                                                                                                                                                                          
+      DEBUG_ERROR,                                                                                                                                                                                                                                                                                                    
+      "%a: bootorder is not NUL-terminated, size=%Lu, last=0x%02x\n",                                                                                                                                                                                                                                                 
+      __func__,                                                                                                                                                                                                                                                                                                       
+      (UINT64)FwCfgSize,                                                                                                                                                                                                                                                                                              
+      (UINT8)FwCfg[FwCfgSize - 1]                                                                                                                                                                                                                                                                                     
+      ));                                                                                                                                                                                                                                                                                                             
     Status = RETURN_INVALID_PARAMETER;                                                                                                                                                                                                                                                                                
     goto FreeFwCfg;                                                                                                                                                                                                                                                                                                   
   }                                                                                                                                                                                                                                                                                                                   
@@ -2206,7 +2220,14 @@ SetBootOrderFromQemu (                                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                                                                       
   QemuFwCfgSelectItem (FwCfgItem);                                                                                                                                                                                                                                                                                    
   QemuFwCfgReadBytes (FwCfgSize, FwCfg);                                                                                                                                                                                                                                                                              
   if (FwCfg[FwCfgSize - 1] != '\0') {                                                                                                                                                                                                                                                                                 
+    DEBUG ((                                                                                                                                                                                                                                                                                                          
+      DEBUG_ERROR,                                                                                                                                                                                                                                                                                                    
+      "%a: bootorder is not NUL-terminated, size=%Lu, last=0x%02x\n",                                                                                                                                                                                                                                                 
+      __func__,                                                                                                                                                                                                                                                                                                       
+      (UINT64)FwCfgSize,                                                                                                                                                                                                                                                                                              
+      (UINT8)FwCfg[FwCfgSize - 1]                                                                                                                                                                                                                                                                                     
+      ));                                                                                                                                                                                                                                                                                                             
     Status = RETURN_INVALID_PARAMETER;                                                                                                                                                                                                                                                                                
     goto ErrorFreeFwCfg;                                                                                                                                                                                                                                                                                              
   } 

Use it to run a test VM:

$ script -q /tmp/bootindex-279720.log sudo bhyve \
    -c 1 -m 1G -A -H -P \
    -s 0,hostbridge \
    -s 1,ahci-hd,/data/img/freebsd.img,ro,bootindex=1 \
    -s 2,ahci-cd,/data/isos/FreeBSD-14.2-RELEASE-amd64-disc1.iso,bootindex=2 \
    -s 31,lpc \
    -l bootrom,/usr/home/novel/ports/sysutils/edk2/work-bhyve/edk2-edk2-stable202508/Build/BhyveX64/DEBUG_GCC/FV/BHYVE_CODE.fd,/tmp/bootindex-279720-vars.fd,fwcfg=qemu \
    -l com1,stdio \
    bootindex-279720


$ grep "not NUL" /tmp/bootindex-279720.log
ConnectDevicesFromQemu: bootorder is not NUL-terminated, size=38, last=0x0A
SetBootOrderFromQemu: bootorder is not NUL-terminated, size=38, last=0x0A
$

So we indeed end up rejected by these checks.

After applying the suggested bhyve fix these messages are gone.

Now repeating the tests I did in comment 30:

1. bhyve -c 2 -m 4096 -A -I -u -H -P \
-s 0:0,hostbridge \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=qemu \
-s 2:0,xhci,tablet -s 1:0,lpc \
-s 3:0,ahci-hd,/data/img/ubuntu2504.img,bootindex=2 -s 4:0,ahci-cd,/data/isos/ubuntu-25.04-live-server-amd64.iso,bootindex=1 \
-s 5:0,fbuf,tcp=127.0.0.1:5920 \
-l com1,stdio -s 6:0,virtio-net,tap4,mac=52:54:00:47:f4:dc \
ubuntu2504

Same result as earlier run: boots from CD, works as expected.

2. bhyve -c 2 -m 4096 -A -I -u -H -P \                                                                                                                                                                                                        
-s 0:0,hostbridge \                                                                                                                                                                                                                                                     
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=qemu \                                                                                                                                                                                                    
-s 2:0,xhci,tablet -s 1:0,lpc \                                                                                                                                                                                                                                         
-s 3:0,ahci-hd,/data/img/ubuntu2504.img,bootindex=1 -s 4:0,ahci-cd,/data/isos/ubuntu-25.04-live-server-amd64.iso,bootindex=2 \                                                                                                                                          
-s 5:0,fbuf,tcp=127.0.0.1:5920 \                                                                                                                                                                                                                                        
-l com1,stdio -s 6:0,virtio-net,tap4,mac=52:54:00:47:f4:dc \                                                                                                                                                                                                            
ubuntu2504

This is now different: this time, it boots from HD, as expected. So the fix appears to be valid, at least in this scenario.
Comment 34 Michael Osipov freebsd_committer freebsd_triage 2026-09-20 16:18:09 UTC
(In reply to Roman Bogorodskiy from comment #33)

Great finding! Are there maybe other spots which might have the same failing result?
Comment 35 commit-hook freebsd_committer freebsd_triage 2026-09-21 16:57:14 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=b27a221ef79d6d77d6173897f12118e1126453d8

commit b27a221ef79d6d77d6173897f12118e1126453d8
Author:     Roman Bogorodskiy <novel@FreeBSD.org>
AuthorDate: 2026-09-20 14:41:55 +0000
Commit:     Roman Bogorodskiy <novel@FreeBSD.org>
CommitDate: 2026-09-21 16:54:45 +0000

    bhyve: fix boot device ordering

    EDK2's QemuBootOrderLib inspects the bootorder file provided
    via fw_cfg and requires it to be NUL-terminated. Otherwise,
    it rejects the supplied bootorder and falls back to its
    default boot order.

    Currently, bhyve registers bootorder with qemu_fwcfg_add_file()
    using bootorder_len returned by open_memstream(), which excludes
    the trailing NUL byte.

    Fix that by passing bootorder_len + 1 to qemu_fwcfg_add_file() so
    the fw_cfg payload is properly NUL-terminated.

    PR:             279720
    Reviewed by:    markj
    Found with:     codex (gpt-5.6-sol)
    MFC after:      1 week
    Sponsored by:   The FreeBSD Foundation
    Differential Revision:  https://reviews.freebsd.org/D59859

 usr.sbin/bhyve/pci_emul.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 36 Roman Bogorodskiy freebsd_committer freebsd_triage 2026-09-21 17:24:11 UTC
(In reply to Michael Osipov from comment #34)

Right now I don't recall any similar issues.
Comment 37 commit-hook freebsd_committer freebsd_triage 2026-09-28 16:01:55 UTC
A commit in branch stable/15 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=1a6e0c38d023f975d6015570b9fc34f09175dc03

commit 1a6e0c38d023f975d6015570b9fc34f09175dc03
Author:     Roman Bogorodskiy <novel@FreeBSD.org>
AuthorDate: 2026-09-20 14:41:55 +0000
Commit:     Roman Bogorodskiy <novel@FreeBSD.org>
CommitDate: 2026-09-28 15:59:17 +0000

    bhyve: fix boot device ordering

    EDK2's QemuBootOrderLib inspects the bootorder file provided
    via fw_cfg and requires it to be NUL-terminated. Otherwise,
    it rejects the supplied bootorder and falls back to its
    default boot order.

    Currently, bhyve registers bootorder with qemu_fwcfg_add_file()
    using bootorder_len returned by open_memstream(), which excludes
    the trailing NUL byte.

    Fix that by passing bootorder_len + 1 to qemu_fwcfg_add_file() so
    the fw_cfg payload is properly NUL-terminated.

    PR:             279720
    Reviewed by:    markj
    Found with:     codex (gpt-5.6-sol)
    MFC after:      1 week
    Sponsored by:   The FreeBSD Foundation
    Differential Revision:  https://reviews.freebsd.org/D59859

    (cherry picked from commit b27a221ef79d6d77d6173897f12118e1126453d8)

 usr.sbin/bhyve/pci_emul.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 38 commit-hook freebsd_committer freebsd_triage 2026-09-28 18:44:25 UTC
A commit in branch stable/14 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=c901f931f6eb102cc0347fddb71bdb409374bdf9

commit c901f931f6eb102cc0347fddb71bdb409374bdf9
Author:     Roman Bogorodskiy <novel@FreeBSD.org>
AuthorDate: 2026-09-20 14:41:55 +0000
Commit:     Roman Bogorodskiy <novel@FreeBSD.org>
CommitDate: 2026-09-28 18:39:05 +0000

    bhyve: fix boot device ordering

    EDK2's QemuBootOrderLib inspects the bootorder file provided
    via fw_cfg and requires it to be NUL-terminated. Otherwise,
    it rejects the supplied bootorder and falls back to its
    default boot order.

    Currently, bhyve registers bootorder with qemu_fwcfg_add_file()
    using bootorder_len returned by open_memstream(), which excludes
    the trailing NUL byte.

    Fix that by passing bootorder_len + 1 to qemu_fwcfg_add_file() so
    the fw_cfg payload is properly NUL-terminated.

    PR:             279720
    Reviewed by:    markj
    Found with:     codex (gpt-5.6-sol)
    MFC after:      1 week
    Sponsored by:   The FreeBSD Foundation
    Differential Revision:  https://reviews.freebsd.org/D59859

    (cherry picked from commit b27a221ef79d6d77d6173897f12118e1126453d8)

 usr.sbin/bhyve/pci_emul.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)