In version FreeBSD12.2 to FreeBSD 14.2 (include) bhyve host don't boot guest FreeBSD any version on BSD partition. # uname -a FreeBSD HOST 14.2-RELEASE FreeBSD 14.2-RELEASE c8918d6c7 HOST amd64 FreeBSD guest no boot if partitioning BSD-label. This has been going on since version FreeBSD 12.2 and to this moment, see report https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254813 Reproduce: Fresh install FreeBSD 14.2, install vm-bhyve. Install into bhyve new FreeBSD 14.2 (in sample, it doesn't really depend on whether it's line 13, 14 or 15) guest. =============================================================================== # cat testboot.conf uuid="c3b3fb80-bab9-11ef-a22f-ac1f6bf2c45c" loader="bhyveload" cpu=8 memory=16G #disk0_type="ahci-hd" #disk0_type="virtio-blk" disk0_type="nvme" disk0_name="system.img" --------------------------------- # vm start testboot Starting FreeBSD14.2 * found guest in /VM/test * booting... =============================================================================== After installing system in partition "BSD label": # vm console test ... nvme0: temperature threshold not supported nvme0: SET_FEATURES (09) sqid:0 cid:9 nsid:0 cdw10:00000008 cdw11:00000000 nvme0: FEATURE NOT CHANGEABLE (01/0e) crd:0 m:0 dnr:0 p:1 sqid:0 cid:9 cdw0:0 nda0 at nvme0 bus 0 scbus1 target 0 lun 1 nda0: <bhyve-NVMe 1.0 NVME-4-0> nda0: Serial Number NVME-4-0 nda0: nvme version 1.4 nda0: 40960MB (83886080 512 byte sectors) cd0 at ahcich0 bus 0 scbus0 target 0 lun 0 cd0: <BHYVE BHYVE DVD-ROM 001> Removable CD-ROM SCSI device cd0: Serial Number BHYVE-0FEB-8A5B-3E90 cd0: 600.000MB/s transfers (SATA 3.x, UDMA6, ATAPI 12bytes, PIO 8192bytes) cd0: 1249MB (639668 2048 byte sectors) Loader variables: Manual root filesystem specification: <fstype>:<device> [options] Mount <device> using filesystem <fstype> and with the specified (optional) option list. eg. ufs:/dev/da0s1a zfs:zroot/ROOT/default cd9660:/dev/cd0 ro (which is equivalent to: mount -t cd9660 -o ro /dev/cd0 /) ? List valid disk boot devices . Yield 1 second (for background tasks) <empty line> Abort manual input mountroot> ? List of GEOM managed disk devices: iso9660/14_2_RELEASE_AMD64_CD cd0 diskid/DISK-NVME-4-0a ufsid/675e8fff44f37a2ea diskid/DISK-NVME-4-0 ufsid/675e8fff44f37a2e nda0a nda0 --------------------------------- If I press ENTER result: panic: mountroot: unable to (re-)mount root. cpuid = 0 time = 9 KDB: stack backtrace: #0 0xffffffff80b8b89d at kdb_backtrace+0x5d #1 0xffffffff80b3dc01 at vpanic+0x131 #2 0xffffffff80b3dac3 at panic+0x43 #3 0xffffffff80c2152c at vfs_mountroot+0x1cac #4 0xffffffff80ac6f83 at start_init+0x23 #5 0xffffffff80af760f at fork_exit+0x7f #6 0xffffffff80ffd3fe at fork_trampoline+0xe Uptime: 9s Automatic reboot in 15 seconds - press a key on the console to abort --------------------------------- If I write ufs:/dev/nda0a after prompt system normal loading. --------------------------------- Guest system: # uname -a FreeBSD test 14.2-RELEASE FreeBSD 14.2-RELEASE releng/14.2-n269506-c8918d6c7412 GENERIC amd64 ===============================================================================
I was able to reproduce this issue on a FreeBSD 14.2 host with a 14.2 guest using bhyveload. The earlier issue commented about it regressing in 12.2 so looking at the commits before that time had this one: https://github.com/freebsd/freebsd-src/commit/14243f8de746e756ad3466aa6920a80e39157d9c https://reviews.freebsd.org/D19262 which was MFC'd into 12.2: https://github.com/freebsd/freebsd-src/commit/7c7ef3e8f9f2309430d171f93e3cb85f463e87e6 bhyveload will load the host loader into the VM which is why it would regress when upgrading to 12.2 (and beyond). I applied this patch to revert a functional change and was able to boot diff --git a/stand/userboot/userboot/main.c b/stand/userboot/userboot/main.c index cf1d9b12d69f..8b8cc30ea91a 100644 --- a/stand/userboot/userboot/main.c +++ b/stand/userboot/userboot/main.c @@ -248,7 +248,7 @@ extract_currdev(void) dev.dd.d_dev = &userboot_disk; dev.dd.d_unit = 0; dev.d_slice = D_SLICEWILD; - dev.d_partition = D_PARTWILD; + dev.d_partition = 0; /* * If we cannot auto-detect the partition type then * access the disk as a raw device. This is not the correct long term patch but does indicate where the regression occurred. I've cc'd a couple of other people that might know more about what should be done here.
It's a shame that the discussion ended in nothing at the time. I believe that two different behaviors should not be described by one value. In fact, I recently encountered some strange behavior of the disk subsystem. Background. There are two M.2 slots on the motherboard, one of them was NVMe with a system (GPT+ZFS). It was installed “full automatic”. Needed to clear NVMe 4Tb and partitioned to create another FreeBSD. The 4Tb previously had GPT partitioning and three partitions: efi, system, home. History. After installing on Mb they were seen as nvd0p1, nvd0p2, nvd0p3 And the working system was on nvd1. I had no way to repartition this 4Tb disk. Until I _physically_ swapped these NVMe disks to Mb. Booted by the label from the correct disk, and when mounting it. partition nvd0p3 was attached (there was nothing there, only zeros), and the normal nvd1p3 was mounted on top of it. And I couldn't do anything with nvd0 because it was busy. It turned out that Mb thought that the first slot was the slot farther away from the processor. And when there was no disk in number 1, the second one was considered first. These are all peculiarities of Mb. However, it was the fact that at boot up, there was a mess of confusion with how the partitions were mounted was because the different modes modes are described by one value. P.S. IMHO. Finding and mounting partitions should be clearly deterministic. If it is written that it is monitored by label -- mount by label. Or by partition type/name in the sense of physical device numbering. And it should be possible to specify how to perceive different partitions.