Description Eugene Grosbein 2021-03-30 00:06:43 UTC

Hi!

I've found that adding loading of several kernel modules to /boot/loader.conf can result in failure to mount ZFS root file system at boot time and going to 
mountroot> prompt.

I've faced to problem after successfully installing and running FreeBSD 12.2/amd64 as VM guest at some hoster providing cheap 1GB RAM/20GB HDD plan.
I've reproduced the problem using my own VirtualBox.

How to repeat:

Use any hypervisor to create new VM for FreeBSD 12.2-RELEASE/amd64 with 1024MB RAM and 20GB HDD (thin provisioning is OK, network is not required nor used).
Attach official FreeBSD-12.2-RELEASE-amd64-disc1.iso image to virtual CD and boot it. Do not use bsdinstall installation procedures but enter Shell directly to perform installation this way (maybe replace ada0 with da0 or vtbd0 depending on hypervisor):

# make MBR with two slices, first one for swap (2GB), the rest for ZFS boot pool
gpart destroy -F ada0
gpart create -s MBR ada0
gpart add -b 504 -t freebsd -s 2g ada0
glabel label swap ada0s1
gpart add -t freebsd ada0

# install bootloaders according to zfsboot(8) manual page
gpart bootcode -b /boot/boot0 ada0
gpart set -a active -i 2 ada0
boot0cfg -s 2 ada0
dd if=/dev/zero of=/dev/ada0s2 count=2
dd if=/boot/zfsboot of=/dev/ada0s2 count=1
dd if=/boot/zfsboot of=/dev/ada0s2 iseek=1 oseek=1024

# create and populate ZFS boot pool named "os"
zpool create -f -O reservation=100m -O compression=lz4 -O atime=off -m / -R /mnt os /dev/ada0s2
cd /mnt
tar xf /usr/freebsd-dist/kernel.txz
tar xf /usr/freebsd-dist/base.txz

cat > boot/loader.conf <<EOF
kern.vty=sc
beastie_disable="YES"
autoboot_delay="6"
hw.memtest.tests=0
zfs_load="YES"
vfs.root.mountfrom="zfs:os"
dumpdev="/dev/label/swap"
EOF

cat > etc/rc.conf <<EOF
hostname="test.localdomain"
zfs_enable="YES"
EOF

cat > etc/fstab <<EOF
# Device        Mountpoint      FStype  Options         Dump    Pass#
/dev/label/swap none            swap    sw              0       0
EOF

cd /
umount -f /mnt
zpool export os
halt -p

Now the system is ready to boot off emulated HDD. Reconfigure the VM to boot off HDD instead of CD and boot it. It should boot to multiuser just fine.

Now add 4 lines to /boot/loader.conf:

aesni_load="YES"
mac_portacl_load="YES"
dummynet_load="YES"
libalias_load="YES"

Reboot the VM and see the failure:

Mounting from zfs:os failed with error 5; retrying for more 3 seconds
Mounting from zfs:os failed with error 5.
...
mountroot>

To recover, one needs to escape to loader prompt at boot time and use "unload" command then manually load kernel, opensolaris.ko and zfs.ko then proceed with booting.