Hello, I just had some time to track this down. The loader has been wrong on CURRENT with my setup for a couple days, particularly since this was merged: https://reviews.freebsd.org/D25324 (If I revert the mentioned diff, HEAD's loader works just fine) Judging by the messages I see when booting, the boot loader tries the disk partitions one by one as expected, finds the one with the OS and tries the active Boot Environment. And then it bails out in this bit of code from vdev_clear_pad2(vdev_t *vdev): + if (vdev_write(vdev, vdev->v_read_priv, off, be, VDEV_PAD_SIZE)) { + printf("failed to clear be area of primary vdev: %d\n", + errno); + } The errno given for the right boot environment is 45, which appears to be EOPNOTSUPP. This is the message I'm seeing (modulo typos, transcript): Setting currdev to zfs:zroot/ROOT/13-current: failed to clear be area of primary vdev: 45 zfs nextboot: zfs:zroot/ROOT/12-release: ERROR: cannot open /boot/lua/loader.lua: no such file or directory. The setup is pretty simple, full disk geli-encrypted system created from the installer with zfs on root with Boot Environments. I'd try to fix this but ATM I don't feel too comfortable touching things that appear to write to the disk :-D; can I help provide more information to fix this? Thank you,
Adding tsoome to CC since they may still have this fresh in memory.
Forgot to mention that 12-release is a Boot Environment that hasn't existed in months.
(In reply to Evilham from comment #2) Hi! The problem is that you have had set the "zfs:zroot/ROOT/12-release:" with zfsbootcfg, and now the loader is able to read it out (is it UEFI boot?). But since you have GELI encryption set, we can not write (GELI writes are not yet implemented). So the fix in your case would be to use: zfsbootcfg "" This will wipe out the BE name from pad2 area and you will get normal boot again.
(In reply to Toomas Soome from comment #3) Wow, thank you; that indeed solved my issue. Since you asked: I am using UEFI boot and as a curiosity: I don't recall ever using zfsbootcfg! Maybe I did a long time ago and forgot or maybe it was the installer, will check on a different machine when I get the chance. Should I leave this bug open as a mark for "GELI writes are not yet implemented"?
(In reply to Evilham from comment #4) Til lthis patch the UEFI loader was not reading the config created by zfsbootcfg, with BIOS version you would have seen it. Yes, at this time we still do miss GELI writes, it is in the queue. Yep, lets leave it open.
A commit references this bug: Author: tsoome Date: Sat Jul 11 06:51:43 UTC 2020 New revision: 363090 URL: https://svnweb.freebsd.org/changeset/base/363090 Log: loader: implement GELI writes Bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247482 This patch is based on initial work from allanjude. PR: 247482 Obtained from: https://reviews.freebsd.org/D10236 Differential Revision: https://reviews.freebsd.org/D25605 Changes: head/stand/i386/gptboot/gptboot.c head/stand/libsa/geli/geliboot.c head/stand/libsa/geli/geliboot.h head/stand/libsa/geli/geliboot_crypto.c head/stand/libsa/geli/geliboot_internal.h head/stand/libsa/geli/gelidev.c
A commit in branch stable/12 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=1f152c027551e887d150ea837afc06cec3fe5fcf commit 1f152c027551e887d150ea837afc06cec3fe5fcf Author: Toomas Soome <tsoome@FreeBSD.org> AuthorDate: 2020-07-11 06:51:42 +0000 Commit: Kyle Evans <kevans@FreeBSD.org> CommitDate: 2021-10-08 08:11:27 +0000 loader: implement GELI writes Bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247482 This patch is based on initial work from allanjude. PR: 247482 (cherry picked from commit de776da3239ee3edc5f77bd0e48d0bb15262024b) stand/i386/gptboot/gptboot.c | 2 +- stand/libsa/geli/geliboot.c | 8 ++-- stand/libsa/geli/geliboot.h | 10 ++++- stand/libsa/geli/geliboot_crypto.c | 33 +++++++++------- stand/libsa/geli/geliboot_internal.h | 4 +- stand/libsa/geli/gelidev.c | 75 ++++++++++++++++++++++++------------ 6 files changed, 86 insertions(+), 46 deletions(-)