Looks like root_hold_wait should be added to /etc/rc.d/zfs script after r355010. If root is on NVMe and e.g. /var/db is on ZFS on top of ATA then root gets mounted and /etc/rc.d/zfs get started before ata bus scan completed. System boots with ZFS filesystems unmounted. There is root_hold_wait calls in /etc/rc.d/mountcritlocal, but zfs filesystems are not mentioned in fstab.
Created attachment 220145 [details] Add root_hold_wait to zpool, zfs, zfsbe, dumpon Attached patch solves the problem for me
Would it be possible to add the output from the commands 'mount' and 'zfs list' to ensure I'm reproducing this correctly?
Created attachment 221677 [details] mount output
Created attachment 221678 [details] zfs list output
(In reply to Chuck Tuffli from comment #2) Here they are. Without patch all zfs systems are not mounted
I was able to reproduce this using bhyve by delaying the response to identify for 10 seconds. When I discussed this with allanjude@, he believed adding root_hold_wait to /etc/rc.d/zpool should be the only change necessary. This setup without root_hold_wait results in ZFS not mounting the file system on the ATA drive. Adding root_hold_wait to /etc/rc.d/zpool (i.e. a subset of the OP's patch) allows ZFS to find its file system on the slower ATA drive: nvd0: 20480MB (41943040 512 byte sectors) WARNING: WITNESS option enabled, expect reduced performance. Trying to mount root from ufs:/dev/gpt/rootfs [rw]... Setting hostuuid: ebf68f0a-7861-11eb-9340-589cfc0c0ccc. Setting hostid: 0xea7b6da7. ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Waiting 30s for the root mount holders: CAM.........ada0 at ata2 bus 0 scbus0 target 0 lun 0 ada0: <BHYVE ATA IDE DISK 1.0> ATA-6 device ada0: Serial Number 123456 ada0: 16.700MB/s transfers (WDMA2, PIO 65536bytes) ada0: 51200MB (104857600 512 byte sectors) Starting file system checks:
(In reply to Chuck Tuffli from comment #6) Thanks for looking at this. I've dropped changes for everything but zpool and dumpon scripts and zfs filesystems got mounted on startup. So zfs and zfsbe changes are not really needed. Without root_hold_wait in dumpon script core dumps were lost because swap device (on the HDD also) was not ready early enough.
OK, I'll include the change to dumpon and post a review later tonight. Thank you for tracking this down and helping make FreeBSD better!
Created attachment 222983 [details] Add root_hold_wait to zpool and dumpon (In reply to Chuck Tuffli from comment #8) My current version of patch (waiting was moved just before importing pools not to slow startup down if there is nothing to import)
Review https://reviews.freebsd.org/D29101
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=f30f11f878fe2aa535cd286810d31c92793a3d95 commit f30f11f878fe2aa535cd286810d31c92793a3d95 Author: Chuck Tuffli <chuck@FreeBSD.org> AuthorDate: 2021-03-05 16:13:23 +0000 Commit: Chuck Tuffli <chuck@FreeBSD.org> CommitDate: 2021-04-05 16:25:04 +0000 wait for device mounts in zpool and dumpon If the root file system is composed from multiple devices, wait for devices to be ready before running zpool and dumpon rc scripts. An example of this is if the bulk of the root file system exists on a fast device (e.g. NVMe) but the /var directory comes from a ZFS dataset on a slower device (e.g. SATA). In this case, it is possible that the zpool import may run before the slower device has finished being probed, leaving the system in an intermediate state. Fix is to add root_hold_wait to the zpool and dumpon (which has a similar issue) rc scripts. PR: 242189 Reported by: osidorkin@gmail.com Reviewed by: allanjude MFC after: 1 month Differential Revision: https://reviews.freebsd.org/D29101 libexec/rc/rc.d/dumpon | 2 ++ libexec/rc/rc.d/zpool | 9 ++++++++- 2 files changed, 10 insertions(+), 1 deletion(-)