Description Dave Cottlehuber 2020-10-15 22:14:04 UTC

FreeBSD 12.x amd64 (efi) loaders seems to behave differently than 11.x
under xhyve OSX based hypervisor, where 11.x used to work.

Using xhyve[1], 12.x amd64 all fails *after successful install* on reboot at:
 
/boot/kernel/kernel text=0x16bdcc4 data=0x140 data=0x75fe80 ZFS: i/o error - all block copies unavailable

elf64_loadimage: read failed
can't load file '/boot/kernel/kernel': input/output error

Checking the resulting disk image by mounting on FreeBSD via mdconfig & zpool scrub etc seems to be fine. Remounting it from the install CD is also fine.

I have used xhyve on OSX to distribute FreeBSD VMs to developers for a few years now, and it's worked just fine on 10.x and 11.x for a simple reliable boot, including 11.4R.

I re-tested these with 12.0,12.1 and now 12.2 RC2 and they all fail with the same error, but only after install completed successfully, which I don't yet understand.

Finally, updating kernel from a working 11.4R -> 12.1R breaks with the same error on first reboot. Suggesting the newer loader itself is the issue.

This seems a bit of an odd situation.

What might be different between the install phase, and resulting loader detection after install?

The error appears to be unaffected by block size, and is present since 12.0 at least.

Rolling back xhyve patches to the original ~2017-2018 version also shows no variation, so this appears to be something different in FreeBSD 12.x and not
the hypervisor itself.

[1]: https://github.com/machyve/xhyve "bhyve for OSX"
[2]: https://github.com/machyve/xhyve/issues/111 

The same image, whether produced from release ISOs on FreeBSD or on xhyve, fails when launched from xhyve.

how to reproduce (requires a mac with homebrew at least for the final step)

- create a 5G sparse disk image
- start FreeBSD and do install iso dance
- move the dis image after install to OSX
- test via following invocation of xhyve

#!/bin/sh
UUID="-U deaddead-dead-dead-dead-deaddeaddead"

CD="/downloads/images/FreeBSD-12.1-RELEASE-amd64-disc1.iso"
IMG="xhyve-12.1R.img"
USERBOOT="$(brew --prefix xhyve)/share/xhyve/test/userboot.so"
BOOTVOLUME=$CD

KERNELENV=""

MEM="-m 8G"
SMP="-c 2"
PCI_DEV="-s 0:0,hostbridge -s 31,lpc"
NET="-s 2:0,virtio-net"
IMG_CD="-s 3:0,ahci-cd,$CD"
IMG_HDD="-s 4:0,virtio-blk,$IMG"
LPC_DEV="-l com1,stdio"
ACPI="-A"

sudo xhyve $ACPI $MEM $SMP \
    $PCI_DEV $LPC_DEV \
    $NET \
    $IMG_CD $IMG_HDD \
    $UUID -f fbsd,$USERBOOT,$BOOTVOLUME,"$KERNELENV"