Created attachment 239740 [details] console log I have upgraded to the most recent stable/13 to give some feedback as the branch is entering the code slush phase. Both host and guest got upgraded to stable/13-n254170-df8c42f5ad12 but it was done in two steps. After the upgrade, the host booted fine and was able to run non-upgraded guest in bhyve(8). The guest was upgraded then and unfortunately, it can't boot (see the boot log attached), but old FreeBSD 13.1-STABLE #28 stable/13-n253260-a4e0b5e28fdc: Sat Dec 10 still boots fine.
I need to describe this setup better. It's a VM run in sysutils/vm-bhyve having root on ZFS pool which is nested in the host's zvol volume. The VM was set up almost 2 years ago, upgraded frequently together with the bhyve host to follow stable/13 and so far run flawlessly. I have done some more tests today and upgraded the guest to PRERELEASE in the meantime. The guest is up and running now FreeBSD 13.2-PRERELEASE stable/13test-n254194-a1362d625340, booted fine but it's still fragile, ie. any attempts of rebooting the guest are unlikely to succeed. I found that importing and exporting the guest's boot zpool nested on the freebsd-zfs GPT partition of the zvol volume done from the host, has made it bootable again. Under normal conditions vfs.zfs.vol.recursive is set to 0 and nested zpool is not accessed from the host. Anyway, it still looks like either ZFS nesting or bhyve(8) issue.
Both, the host and the guest are up and running now FreeBSD 13.2-PRERELEASE #33 stable/13-n254194-a1362d625340. Surprisingly, the guest seems to reboot fine now. Also, nothing suspicious was revealed during the verbose boot but these lines, are probably harmless. GEOM: new disk vtbd0 GEOM_PART: partition 1 on (vtbd0, GPT) is not aligned on 8192 bytes GEOM_PART: partition 2 on (vtbd0, GPT) is not aligned on 8192 bytes GEOM_PART: partition 3 on (vtbd0, GPT) is not aligned on 8192 bytes I am closing the PR as not reproducible.