Testing environment: * server: FreeBSD 13.1-RELEASE running KDC inside jail (MIT kerberos security/krb5 from ports) and base NFSv4 server in the jailhost (nfs.my.domain) * client: Alpine Linux edge with MIT kerberos (client.local) I have setup Kerberos host principals correctly in each of the server and client keytabs, and acquired a user principal ticket that corresponds to the same user on both the client and server. TESTS: **Test #1**: /etc/exports in the server: V4: /nfshome -sec=krb5p /nfshome -sec=krb5p When mounting in the client: # mount nfs.my.domain:/ /mnt mount.nfs: access denied by server while mounting nfs.my.domain:/ mount: mounting nfs.my.domain:/ on /mnt failed: Permission denied I can see in this test that on the server, gssd logs the correct principal->uid mapping and the request is clearly coming through. **Test #2** /etc/exports in the server: V4: /nfshome -sec=krb5p:krb5i /nfshome -sec=krb5p:krb5i When mounting in the client: # mount nfs.my.domain:/ /mnt # The mount completes successfully and I am able to read/write files to the NFS share. When I look at the mount information: $ mount [...] nfs.my.domain:/ on /mnt type nfs4 (rw,nosuid,nodev,noexec,relatime,vers=4.2,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5p,clientaddr=192.168.0.11,local_lock=none,addr=192.168.0.201) It indicates a successful mount of -sec=krb5p **Test #3** /etc/exports in the server: V4: /nfshome -sec=krb5i /nfshome -sec=krb5i When mounting in the client: # mount nfs.my.domain:/ /mnt # The mount completes successfully and I am able to read/write files to the NFS share. When I look at the mount information: $ mount [...] nfs.my.domain:/ on /mnt type nfs4 (rw,nosuid,nodev,noexec,relatime,vers=4.2,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5i,clientaddr=192.168.0.11,local_lock=none,addr=192.168.0.201) It indicates a successful mount of -sec=krb5i CONCLUSION: I expect that when the server exports are listed as in Test #1, I should successfully be able to mount as -sec=krb5p. Why am I getting "permission denied" in this case, but not in Test #2?
Linux clients love to use krb5i for state maintenance operations no matter what you specify. If you capture packets when the mount is done and then look at them in wireshark, I'm pretty sure you'll find it using "integrity" (in the RPC credential) for RPCs that do stuff like EXCHANGEID, CREATESESSION, RECLAIM_COMPLETE. The Linux folk consider this a feature, for NFSv4.1/4.2 mounts. (A mount with "minorversion=0" would probably work, but you don't want to use 4.0 when 4.1/4.2 is supported.) It just so happens I reported this to linux-nfs@vger.kernel.org and, if you look at the reply in the email archive for it, you'll see they consider it a feature. (For my case it was sec=krb5, but I think you'll find it is the same.)
Btw, a recent commit (that is in 13.2) modifies the server so that it will allow the use of krb5i for state maintenance operations for NFSv4.1/4.2 when SP4_NONE is used. This should allow the Linux NFS client to do the mount, without "krb5i" being in the V4: line in /etc/exports. I will note that adding "krb5i" to the -sec option only in the V4: line should make the Linux mounts work to versions prior to 13.2 and for 13.2 and later, it is not required and your failing test case would work. As such, I'll close this bug report.
Thanks for the info and detailed explanation! I'll wait for 13.2-RELEASE to update my /etc/exports then.
Revisiting this bug as I'm running into it again with a 14.0-RELEASE-p4 installation. > I will note that adding "krb5i" to the -sec option only in the V4: line should make the Linux mounts work to versions prior to 13.2 and for 13.2 and later, it is not required and your failing test case would work. This is not working in 14.0-RELEASE (at least, with my relatively vanilla configuration). Instead, the behaviour is different now. In test environment #1, the linux client is successfully mounting the NFSv4.2 share, AND is successfully reading and deleting files. When it comes to *writing* files, it creates the file on the server-side, but fails to close() and write() bytes successfully to it. This results in the creation of a server-side 0-byte file with nothing in it (although, with the correct permissions and user mapping). Test environment #2 and #3 work just as they did before just fine. Rick, if you have the time, could you link me to the following things: * the linux-nfs mailing list thread regarding the Linux kernel community deeming it to be a "feature" * The FreeBSD commit that you mentioned is in 13.2 which will solve the issue on the FreeBSD server side Thanks! I'd like to help clean this up if possible since it's a bit misleading to explicitly mention krb5i as an allowed mounting sec level, especially when the intention of my /etc/exports is to ensure fully encrypted data transfer.
Well, the Linux client always mixes krb5i with krb5p, even when sec=krb5p is specified, from what I've seen. If they feel this is a feature, then there is little that can be done. If you capture a packet trace of a failing case (starting before the client mount is done), I can look at it in wireshark and check to see that everything conforms to the RFCs. However, it sounds like you will have to specify krb5i for both the V4: line and all exported file systems. Btw, NFS over TLS (or RPC with TLS as some Linux folk call it) allows for everything to be encrypted on the wire and then you can avoid use of krb5p.
You can use: # tcpdump -s 0 -w out.pcap host <nfs-client-hostname> to capture the packets (you do not need wireshark) and then I can look at out.pcap.
I have just upgraded from 13.2 to 14.1 and I am experiencing what seems like the same issue as Siva has re-raised in comment #4. On NFS client machines: $ cp ~/a-file /run/media/system/nas/<any share>/ cp: failed to close './a-file': Operation not permitted $ ls -l ./a-file -rw-r--r-- 1 me me 0 Jul 29 20:51 a-file Like Siva mentioned, strace shows specifically that close and write are failing: $ strace cp ~/a-file /run/media/system/nas/<any share>/ [...lines omitted...] close(4) = -1 EPERM (Operation not permitted) [...lines omitted...] write(2, "failed to close './a-file'", 29failed to close './a-file') = 29 write(2, ": Operation not permitted", 25: Operation not permitted) = 25 write(2, "\n", 1 ) = 1 close(3) = 0 munmap(0x788dad3be000, 270336) = 0 lseek(0, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek) close(0) = 0 close(1) = 0 close(2) = 0 exit_group(1) = ? +++ exited with 1 +++ Similarly: $ echo "foo" > /run/media/system/nas/<any share>/test $ cat /run/media/system/nas/<any share>/test $ $ ls -l /run/media/system/nas/<any share>/test -rw-r--r-- 1 me me 0 Jul 29 21:03 /run/media/system/nas/<any share>/test Some operations, for example saving a file from Firefox, fail silently. Existing files can be read without issue. Clients are Arch Linux and Fedora. I have one client that hadn't been updated for a while which worked before the 14.1 upgrade and also doesn't work now, so that probably rules out any recent changes in Linux. No changes in NFS settings before or after the upgrade, below is what I've been using for years: $ cat /etc/exports V4: /data/share -sec=krb5i:krb5p $ zfs get sharenfs data sharenfs off default data/share sharenfs -sec=krb5p local data/share/photos sharenfs -sec=krb5p local [...and other datasets, same sharenfs settings...] And on the clients: $ grep nas /etc/fstab nas.my-domain.com:/ /run/media/system/nas nfs4 defaults,user,noatime,noexec,noauto,nofail,x-systemd.automount,x-systemd.device-timeout=100ms,x-systemd.mount-timeout=100ms,x-systemd.requires=network-online.target,sec=krb5p,vers=4.2,hard,retrans=5,_netdev 0 0 Finally, like Siva I can confirm that setting sharenfs="-sec=krb5p:krb5i" on the zfs datasets restores the ability to write to the shares. I note that there were a number of NFS changes between 13.2 and 14.1, however nothing that obviously suggests this change in behaviour was expected from my reading of the release notes. It seems more like some form of regression to me.
If you want to track this down further, you'll need to capture packets. # tcpdump -s 0 -w out.pcap host <client-host> run on the server and started before the client does a mount should do the trick. You need to capture packets from before the mount to until the failure occurs. Then you or I need to look at out.pcap in wireshark. - Near the beginning, there will be a EXCHANGEID. In it I suspect you will find SP4_MACH_CRED, which is followed by two bitmaps that specify operations that must and may be done with machine credentials. (Linux always chooses to do RPCs that use machine credentials with krb5i, even when you specify -sec=krb5p on the mount, afaik.) This list of musts and mays (as bitmaps of operation #s) is the first critical bit (no pun intended;-) of info. Then you look at where the failure occurs. Go into the reply and see which operation fails. --> Then you look to see if that operation is in either the must or may lists. (I can look at the "out.pcap" if you email it to me as an attachment.)