Comment 1 Alan Somers 2021-07-14 23:10:44 UTC

Firstly, does that symlink cross a mountpoint?  It may be that the target exists as seen from the NFS client, but not on the server.

Assuming that isn't the problem, dtrace is your friend.  Try something like this:

sudo dtrace -i 'sdt:zfs::set-error /arg0 == 2/ {stack();}' -c "readlink /foo/bar/baz"

If the error is coming from ZFS, you should find out where.  But if it isn't coming from ZFS, then run this instead:

sudo dtrace -i 'fbt:kernel::return /arg1 == 2 && pid == $target/ {trace(".");}' -c "readlink /foo/bar/baz"

That command will have plenty of false positives, so don't print the stack traces just yet.  If you spy a function that looks likely, then suitably modify your D script and prints its stack trace.

Comment 2 Garrett Wollman 2021-07-15 00:03:46 UTC

(In reply to Alan Somers from comment #1)
I get two hits on sdt:zfs::set-error, which I think are actually the same:

dtrace: description 'sdt:zfs::set-error ' matched 1 probe
dtrace: pid 87135 exited with status 1
CPU     ID                    FUNCTION:NAME
 14  47740                   none:set-error 
              zfs.ko`sa_attr_op+0x1b5
              zfs.ko`sa_lookup_uio+0x69
              zfs.ko`zfs_freebsd_readlink+0x7c
              kernel`0xffffffff80b1bf2b
              kernel`vn_rdwr+0x173
              kernel`vn_rdwr+0x26
              kernel`AES_GCM_decrypt+0xa87
              kernel`ssdtosyssd+0x8e

 14  47740                   none:set-error 
              zfs.ko`sa_lookup_uio+0x69
              zfs.ko`zfs_freebsd_readlink+0x7c
              kernel`0xffffffff80b1bf2b
              kernel`vn_rdwr+0x173
              kernel`vn_rdwr+0x26
              kernel`AES_GCM_decrypt+0xa87
              kernel`ssdtosyssd+0x8e

Looks like the only place this can happen is:

                switch (data_op) {
                case SA_LOOKUP:
                        if (bulk[i].sa_addr == NULL)
                                return (SET_ERROR(ENOENT));

which I guess means that zfs_freebsd_readlink() tried to read the ZPL_SYMLINK attribute and it wasn't found -- but that still doesn't explain how it is that readlink() succeeds over NFS and not locally.  (A symbolic link is just uninterpreted text as far as readlink() is concerned, it shouldn't matter what its value is!)

This indicates a second bug: that [ENOENT] should get transformed into a more appropriate error on exit to indicate an internal filesystem inconsistency.

Comment 3 Alan Somers 2021-07-15 00:34:53 UTC

I would suggest using dtrace to trace both the local and NFS paths.  They might be looking up the link differently.  For example, in zfs_readlink you can see there are two paths, based on zp->z_is_sa.

That raises another question, what is the value of the xattr property for your filesystem?  And are you using the builtin ZFS in FreeBSD 12.2, or the openzfs kmod?

Comment 4 Garrett Wollman 2021-07-15 01:05:31 UTC

(In reply to Alan Somers from comment #3)
Stock 12.2, xattr is off.