I have the following nullfs mounts: /var/jail/builder/src /usr/src nullfs ro,late 0 0 /var/jail/builder/obj /usr/obj nullfs ro,late 0 0 /var/jail/builder/src and /var/jail/builder/obj are ZFS datasets. Exporting them like this: /usr/src -maproot=root -network=192.168.0.0/16 -ro /usr/obj -maproot=root -network=192.168.0.0/16 -ro worked fine on any 11.x and 12.x version. It stopped working after upgrade to 13.0-RC2, giving these errors: mountd[13382]: can't change attributes for /usr/src: netcred already exists for given addr/mask mountd[13382]: bad exports list line '/usr/src -maproot' Workaround for now is to directly export the datasets instead.
With this PR sitting for quite a while now, I'd like to kindly ask: Is it expected behavior that you can't export a dataset via a nullfs mount, or is there a chance that this will be "fixed" to work like it did in previous releases?
The problem is your specification of "late" on the nullfs mount lines. If you look in /etc/rc.d, you'll find "statd" requires "nfsd", which requires "mountd". However, it specifies BEFORE DAEMON. "mountlate" requires DAEMON. --> As such, mountd is executed before mountlate, which is what mounts the "late" file systems. To fix this, you could: - Drop "late" from the /etc/fstab lines, if you don't need to mount them "late". or - If you don't run rpc.statd and rpc.lockd, you could remove DAEMON from "statd" and "lockd" and then add "mountlate" to the REQUIRES line in "mountd". I don't know why this worked for FreeBSD-12, but I suspect that some DAEMON related dependency was added to the scripts (ie. they were broken). - Although rpc.statd and rpc.lockd are NFSv3 related protocols I avoid using, they need to be running before nfsd (for server side locking) and NFS mounts are done for client side locking, if you are using them. I can't think of a way to fix this so that mountd can start after "mountlate" and still work correctly, for those who are using rpc.statd, rpc.lockd.
Here's the snippet from "man fstab" for the "late" option. If the option "late" is specified, the file system will be automatically mounted at a stage of system startup after remote mount points are mounted. For more detail about this option, see the mount(8) manual page. Do you really need to do the nullfs mounts after remote mounts?
(In reply to Rick Macklem from comment #3) > Do you really need to do the nullfs mounts after remote mounts? No, for the nullfs mounts, that's not relevant (just copied the options from the hosts that mount /usr/src and /usr/obj via NFS, where it is relevant). Still I have doubts about that explanation, because I tried a lot to get it working again, including restarting mountd several times, and it always produces the error messages shown above ("can't change attributes"), with the nullfs mounts mounted... Of course, will try to reproduce on the next server reboot without the "late" option anyways.
I just tried basically what you are doing with a nullfs mount on top of UFS file systems and it worked. I don't do ZFS. It changed a lot for 13, so maybe some change in ZFS causes it. Please update when you've tried without "late".
Finally remembered to test that, and well, turns out I was wrong: The 'late' option *is* required. /var/jail/builder/(src|obj) are ZFS datasets and without 'late', the system tries to mount the nullfs before ZFS datasets are mounted. Still I have doubts that's really the issue, because, as mentioned above, restarting of NFS-related services doesn't solve the issue either.
For me, the mountd does not see a nullfs as a separate filesystem and tries to set the attributes for the upper filesystem. The simple workaround that worked for me is to create an empty ZFS dataset mounted at the same point as nullfs, but before nullfs, eg.: zfs create -o mountpoint=/usr/src zpool/nfs-usr-src mount_nullfs /var/jail/builder/src /usr/src After that, the NFS exports work fine for me. Tested on 13-RELEASE-p4.
Created attachment 229500 [details] make mountd start after mountlate I think this patch to the /etc/rc.d scripts (applied when in the root directory) will fix the problem. --> The nullfs mount(s) need the "late" option. --> Without this patch, mountd starts before mountlate happens, so the exports get applied to the underlying file system and not the nullfs mount. This patch forces mountd to be started after mountlate. Unfortunately, I am not sure if it safe to start lockd before nfsd. The only obvious reason is to make sure the nfscommon.ko must be loaded before lockd starts, so I have added this to the lockd script. Hopefully the reporter can test this patch.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=f72926eab00ccd956298e44831b519daa704a868 commit f72926eab00ccd956298e44831b519daa704a868 Author: Rick Macklem <rmacklem@FreeBSD.org> AuthorDate: 2022-01-23 22:17:40 +0000 Commit: Rick Macklem <rmacklem@FreeBSD.org> CommitDate: 2022-01-23 22:17:40 +0000 mountd: Delay starting mountd until after mountlate PR#254282 reports a problem where nullfs mounts cannot be exported via mountd for FreeBSD 13.0. The problem seems to be that, to do the nullfs mounts in /etc/fstab, they require the "late" mount option, so that the underlying filesystem is mounted (ZFS for the PR). Adding "mountlate" to the REQUIRE list in /etc/rc.d/mountd fixes the problem, but that results in a dependency cycle because /etc/rc.d/lockd specifies: REQUIRE: nfsd BEFORE: DAEMON --> which forces mountd to preceed DAEMON. This patch removes "nfsd" from REQUIRE for lockd and statd, then adds mountlate to REQUIRE for mountd, to fix this problem. Having lockd REQUIRE nfsd was done in the NetBSD code when it was pulled into FreeBSD and there does not seem to be a need for this. In case this causes problems, a long MFC has been specified. PR: 254282 Differential Revision: https://reviews.freebsd.org/D33256 MFC after: 3 months libexec/rc/rc.d/lockd | 2 +- libexec/rc/rc.d/mountd | 2 +- libexec/rc/rc.d/statd | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
Patch has been committed and will be MFC'd.
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=92ecb030d2547a95956b3a6e5240e895ad4eed4e commit 92ecb030d2547a95956b3a6e5240e895ad4eed4e Author: Rick Macklem <rmacklem@FreeBSD.org> AuthorDate: 2022-01-23 22:17:40 +0000 Commit: Rick Macklem <rmacklem@FreeBSD.org> CommitDate: 2022-04-23 22:21:45 +0000 mountd: Delay starting mountd until after mountlate PR#254282 reports a problem where nullfs mounts cannot be exported via mountd for FreeBSD 13.0. The problem seems to be that, to do the nullfs mounts in /etc/fstab, they require the "late" mount option, so that the underlying filesystem is mounted (ZFS for the PR). Adding "mountlate" to the REQUIRE list in /etc/rc.d/mountd fixes the problem, but that results in a dependency cycle because /etc/rc.d/lockd specifies: REQUIRE: nfsd BEFORE: DAEMON --> which forces mountd to preceed DAEMON. This patch removes "nfsd" from REQUIRE for lockd and statd, then adds mountlate to REQUIRE for mountd, to fix this problem. Having lockd REQUIRE nfsd was done in the NetBSD code when it was pulled into FreeBSD and there does not seem to be a need for this. In case this causes problems, a long MFC has been specified. PR: 254282 (cherry picked from commit f72926eab00ccd956298e44831b519daa704a868) libexec/rc/rc.d/lockd | 2 +- libexec/rc/rc.d/mountd | 2 +- libexec/rc/rc.d/statd | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
Patch has been committed and MFC'd.