Bug 254649

Summary: 12.2-RELEASE: possible memory overrun in the loader(8)
Product: Base System Reporter: Eugene Grosbein <eugen>
Component: kernAssignee: freebsd-bugs (Nobody) <bugs>
Status: New ---    
Severity: Affects Some People CC: imp, kevans, tsoome
Priority: ---    
Version: 12.2-RELEASE   
Hardware: Any   
OS: Any   

Description Eugene Grosbein freebsd_committer 2021-03-30 00:06:43 UTC

I've found that adding loading of several kernel modules to /boot/loader.conf can result in failure to mount ZFS root file system at boot time and going to 
mountroot> prompt.

I've faced to problem after successfully installing and running FreeBSD 12.2/amd64 as VM guest at some hoster providing cheap 1GB RAM/20GB HDD plan.
I've reproduced the problem using my own VirtualBox.

How to repeat:

Use any hypervisor to create new VM for FreeBSD 12.2-RELEASE/amd64 with 1024MB RAM and 20GB HDD (thin provisioning is OK, network is not required nor used).
Attach official FreeBSD-12.2-RELEASE-amd64-disc1.iso image to virtual CD and boot it. Do not use bsdinstall installation procedures but enter Shell directly to perform installation this way (maybe replace ada0 with da0 or vtbd0 depending on hypervisor):

# make MBR with two slices, first one for swap (2GB), the rest for ZFS boot pool
gpart destroy -F ada0
gpart create -s MBR ada0
gpart add -b 504 -t freebsd -s 2g ada0
glabel label swap ada0s1
gpart add -t freebsd ada0

# install bootloaders according to zfsboot(8) manual page
gpart bootcode -b /boot/boot0 ada0
gpart set -a active -i 2 ada0
boot0cfg -s 2 ada0
dd if=/dev/zero of=/dev/ada0s2 count=2
dd if=/boot/zfsboot of=/dev/ada0s2 count=1
dd if=/boot/zfsboot of=/dev/ada0s2 iseek=1 oseek=1024

# create and populate ZFS boot pool named "os"
zpool create -f -O reservation=100m -O compression=lz4 -O atime=off -m / -R /mnt os /dev/ada0s2
cd /mnt
tar xf /usr/freebsd-dist/kernel.txz
tar xf /usr/freebsd-dist/base.txz

cat > boot/loader.conf <<EOF

cat > etc/rc.conf <<EOF

cat > etc/fstab <<EOF
# Device        Mountpoint      FStype  Options         Dump    Pass#
/dev/label/swap none            swap    sw              0       0

cd /
umount -f /mnt
zpool export os
halt -p

Now the system is ready to boot off emulated HDD. Reconfigure the VM to boot off HDD instead of CD and boot it. It should boot to multiuser just fine.

Now add 4 lines to /boot/loader.conf:


Reboot the VM and see the failure:

Mounting from zfs:os failed with error 5; retrying for more 3 seconds
Mounting from zfs:os failed with error 5.

To recover, one needs to escape to loader prompt at boot time and use "unload" command then manually load kernel, opensolaris.ko and zfs.ko then proceed with booting.
Comment 1 Eugene Grosbein freebsd_committer 2021-03-30 00:09:36 UTC
Forgot to note, FreeBSD 11.4/amd64 is not affected.
Comment 2 Eugene Grosbein freebsd_committer 2021-03-30 03:57:34 UTC
Here is OVA-exported virtual machine in last state (cannot mount root), 643M:
Comment 3 Toomas Soome freebsd_committer 2021-03-30 07:08:18 UTC
(In reply to Eugene Grosbein from comment #0)

I assume, this VM is using BIOS, not UEFI? What happens, if you give VM more memory?
Comment 4 Eugene Grosbein freebsd_committer 2021-03-30 13:35:51 UTC
(In reply to Toomas Soome from comment #3)

Yes, the hoster uses SeaBIOS and I'm sure VirtualBox uses some kind of BIOS, too.

Increase RAM for VM upto 8GB does not solve the problem.