Bug 254282

Comment 1 Felix Palmen 2021-05-13 01:02:35 UTC

With this PR sitting for quite a while now, I'd like to kindly ask:

Is it expected behavior that you can't export a dataset via a nullfs mount, or is there a chance that this will be "fixed" to work like it did in previous releases?

Comment 2 Rick Macklem 2021-06-22 01:58:20 UTC

The problem is your specification of "late" on
the nullfs mount lines.

If you look in /etc/rc.d, you'll find "statd"
requires "nfsd", which requires "mountd".
However, it specifies BEFORE DAEMON.
"mountlate" requires DAEMON.
--> As such, mountd is executed before mountlate,
    which is what mounts the "late" file systems.

To fix this, you could:
- Drop "late" from the /etc/fstab lines, if you
  don't need to mount them "late".
or
- If you don't run rpc.statd and rpc.lockd, you
  could remove DAEMON from "statd" and "lockd"
  and then add "mountlate" to the REQUIRES line
  in "mountd".

I don't know why this worked for FreeBSD-12, but
I suspect that some DAEMON related dependency was
added to the scripts (ie. they were broken).
- Although rpc.statd and rpc.lockd are NFSv3 related
  protocols I avoid using, they need to be running
  before nfsd (for server side locking) and NFS
  mounts are done for client side locking, if you
  are using them.

I can't think of a way to fix this so that mountd
can start after "mountlate" and still work correctly,
for those who are using rpc.statd, rpc.lockd.

Comment 3 Rick Macklem 2021-06-22 02:43:46 UTC

Here's the snippet from "man fstab" for the "late" option.
     If the option "late" is specified, the file system will be automatically
     mounted at a stage of system startup after remote mount points are
     mounted.  For more detail about this option, see the mount(8) manual
     page.

Do you really need to do the nullfs mounts after remote mounts?

Comment 4 Felix Palmen 2021-06-22 06:44:53 UTC

(In reply to Rick Macklem from comment #3)

> Do you really need to do the nullfs mounts after remote mounts?

No, for the nullfs mounts, that's not relevant (just copied the options from the hosts that mount /usr/src and /usr/obj via NFS, where it is relevant).

Still I have doubts about that explanation, because I tried a lot to get it working again, including restarting mountd several times, and it always produces the error messages shown above ("can't change attributes"), with the nullfs mounts mounted...

Of course, will try to reproduce on the next server reboot without the "late" option anyways.

Comment 5 Rick Macklem 2021-06-22 15:06:27 UTC

I just tried basically what you are doing with a
nullfs mount on top of UFS file systems and it worked.

I don't do ZFS. It changed a lot for 13, so maybe
some change in ZFS causes it.

Please update when you've tried without "late".

Comment 6 Felix Palmen 2021-09-03 04:40:49 UTC

Finally remembered to test that, and well, turns out I was wrong:

The 'late' option *is* required.

/var/jail/builder/(src|obj) are ZFS datasets and without 'late', the system tries to mount the nullfs before ZFS datasets are mounted.

Still I have doubts that's really the issue, because, as mentioned above, restarting of NFS-related services doesn't solve the issue either.

For me, the mountd does not see a nullfs as a separate filesystem and tries to set the attributes for the upper filesystem. 

The simple workaround that worked for me is to create an empty ZFS dataset mounted at the same point as nullfs, but before nullfs, eg.:

zfs create -o mountpoint=/usr/src zpool/nfs-usr-src 
mount_nullfs /var/jail/builder/src /usr/src

After that, the NFS exports work fine for me. Tested on 13-RELEASE-p4.

Comment 8 Rick Macklem 2021-11-15 00:13:21 UTC

Created attachment 229500 [details]
make mountd start after mountlate

I think this patch to the /etc/rc.d scripts (applied when
in the root directory) will fix the problem.
--> The nullfs mount(s) need the "late" option.
--> Without this patch, mountd starts before mountlate happens,
    so the exports get applied to the underlying file system and
    not the nullfs mount.

This patch forces mountd to be started after mountlate.

Unfortunately, I am not sure if it safe to start lockd before
nfsd.  The only obvious reason is to make sure the nfscommon.ko
must be loaded before lockd starts, so I have added this to the
lockd script.

Hopefully the reporter can test this patch.

Comment 9 commit-hook 2022-01-23 22:19:33 UTC

A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=f72926eab00ccd956298e44831b519daa704a868

commit f72926eab00ccd956298e44831b519daa704a868
Author:     Rick Macklem <rmacklem@FreeBSD.org>
AuthorDate: 2022-01-23 22:17:40 +0000
Commit:     Rick Macklem <rmacklem@FreeBSD.org>
CommitDate: 2022-01-23 22:17:40 +0000

    mountd: Delay starting mountd until after mountlate

    PR#254282 reports a problem where nullfs mounts cannot be
    exported via mountd for FreeBSD 13.0.

    The problem seems to be that, to do the nullfs mounts in
    /etc/fstab, they require the "late" mount option, so that the
    underlying filesystem is mounted (ZFS for the PR).

    Adding "mountlate" to the REQUIRE list in /etc/rc.d/mountd
    fixes the problem, but that results in a dependency cycle
    because /etc/rc.d/lockd specifies:

    REQUIRE: nfsd
    BEFORE: DAEMON
    --> which forces mountd to preceed DAEMON.

    This patch removes "nfsd" from REQUIRE for lockd and statd,
    then adds mountlate to REQUIRE for mountd, to fix this
    problem.  Having lockd REQUIRE nfsd was done in the NetBSD
    code when it was pulled into FreeBSD and there does not
    seem to be a need for this.

    In case this causes problems, a long MFC has been specified.

    PR:     254282
    Differential Revision:  https://reviews.freebsd.org/D33256
    MFC after:      3 months

 libexec/rc/rc.d/lockd  | 2 +-
 libexec/rc/rc.d/mountd | 2 +-
 libexec/rc/rc.d/statd  | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

Comment 10 Rick Macklem 2022-01-23 22:24:58 UTC

Patch has been committed and will be MFC'd.

Comment 11 commit-hook 2022-04-23 22:26:24 UTC

A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=92ecb030d2547a95956b3a6e5240e895ad4eed4e

commit 92ecb030d2547a95956b3a6e5240e895ad4eed4e
Author:     Rick Macklem <rmacklem@FreeBSD.org>
AuthorDate: 2022-01-23 22:17:40 +0000
Commit:     Rick Macklem <rmacklem@FreeBSD.org>
CommitDate: 2022-04-23 22:21:45 +0000

    mountd: Delay starting mountd until after mountlate

    PR#254282 reports a problem where nullfs mounts cannot be
    exported via mountd for FreeBSD 13.0.

    The problem seems to be that, to do the nullfs mounts in
    /etc/fstab, they require the "late" mount option, so that the
    underlying filesystem is mounted (ZFS for the PR).

    Adding "mountlate" to the REQUIRE list in /etc/rc.d/mountd
    fixes the problem, but that results in a dependency cycle
    because /etc/rc.d/lockd specifies:

    REQUIRE: nfsd
    BEFORE: DAEMON
    --> which forces mountd to preceed DAEMON.

    This patch removes "nfsd" from REQUIRE for lockd and statd,
    then adds mountlate to REQUIRE for mountd, to fix this
    problem.  Having lockd REQUIRE nfsd was done in the NetBSD
    code when it was pulled into FreeBSD and there does not
    seem to be a need for this.

    In case this causes problems, a long MFC has been specified.

    PR:     254282
    (cherry picked from commit f72926eab00ccd956298e44831b519daa704a868)

 libexec/rc/rc.d/lockd  | 2 +-
 libexec/rc/rc.d/mountd | 2 +-
 libexec/rc/rc.d/statd  | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

Comment 12 Rick Macklem 2022-04-23 22:28:07 UTC

Patch has been committed and MFC'd.