Looks like root_hold_wait should be added to /etc/rc.d/zfs script after r355010.
If root is on NVMe and e.g. /var/db is on ZFS on top of ATA then root gets mounted and /etc/rc.d/zfs get started before ata bus scan completed. System boots with ZFS filesystems unmounted. There is root_hold_wait calls in /etc/rc.d/mountcritlocal, but zfs filesystems are not mentioned in fstab.
Created attachment 220145 [details]
Add root_hold_wait to zpool, zfs, zfsbe, dumpon
Attached patch solves the problem for me
Would it be possible to add the output from the commands 'mount' and 'zfs list' to ensure I'm reproducing this correctly?
Created attachment 221677 [details]
Created attachment 221678 [details]
zfs list output
(In reply to Chuck Tuffli from comment #2)
Here they are. Without patch all zfs systems are not mounted
I was able to reproduce this using bhyve by delaying the response to identify for 10 seconds. When I discussed this with allanjude@, he believed adding root_hold_wait to /etc/rc.d/zpool should be the only change necessary.
This setup without root_hold_wait results in ZFS not mounting the file system on the ATA drive. Adding root_hold_wait to /etc/rc.d/zpool (i.e. a subset of the OP's patch) allows ZFS to find its file system on the slower ATA drive:
nvd0: 20480MB (41943040 512 byte sectors)
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
Setting hostuuid: ebf68f0a-7861-11eb-9340-589cfc0c0ccc.
Setting hostid: 0xea7b6da7.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Waiting 30s for the root mount holders: CAM.........ada0 at ata2 bus 0 scbus0 target 0 lun 0
ada0: <BHYVE ATA IDE DISK 1.0> ATA-6 device
ada0: Serial Number 123456
ada0: 16.700MB/s transfers (WDMA2, PIO 65536bytes)
ada0: 51200MB (104857600 512 byte sectors)
Starting file system checks:
(In reply to Chuck Tuffli from comment #6)
Thanks for looking at this.
I've dropped changes for everything but zpool and dumpon scripts and zfs filesystems got mounted on startup. So zfs and zfsbe changes are not really needed.
Without root_hold_wait in dumpon script core dumps were lost because swap device (on the HDD also) was not ready early enough.
OK, I'll include the change to dumpon and post a review later tonight. Thank you for tracking this down and helping make FreeBSD better!
Created attachment 222983 [details]
Add root_hold_wait to zpool and dumpon
(In reply to Chuck Tuffli from comment #8)
My current version of patch (waiting was moved just before importing pools not to slow startup down if there is nothing to import)
A commit in branch main references this bug:
Author: Chuck Tuffli <chuck@FreeBSD.org>
AuthorDate: 2021-03-05 16:13:23 +0000
Commit: Chuck Tuffli <chuck@FreeBSD.org>
CommitDate: 2021-04-05 16:25:04 +0000
wait for device mounts in zpool and dumpon
If the root file system is composed from multiple devices, wait for
devices to be ready before running zpool and dumpon rc scripts.
An example of this is if the bulk of the root file system exists on a
fast device (e.g. NVMe) but the /var directory comes from a ZFS dataset
on a slower device (e.g. SATA). In this case, it is possible that the
zpool import may run before the slower device has finished being probed,
leaving the system in an intermediate state.
Fix is to add root_hold_wait to the zpool and dumpon (which has a
similar issue) rc scripts.
Reported by: firstname.lastname@example.org
Reviewed by: allanjude
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D29101
libexec/rc/rc.d/dumpon | 2 ++
libexec/rc/rc.d/zpool | 9 ++++++++-
2 files changed, 10 insertions(+), 1 deletion(-)