Okay seems that there's a problem with NFSv3 implementation on Freebsd 10.1. I have gone through and compiled and configured a GENERIC build with the new NFSD kernel option in place. If I mount a NFSv3 filesystem from a Solaris 10 server to the Freebsd 10.0 client which is running dovecot 2.2.13. Now don't jump to say well this is a dovecot issue not a base kern issue. As if I mount the filesystem: root@NFSCLIENT:<NFSv3 Mount>/mail # ls -lh A1 ---------- 1 bob wheel 0B Aug 29 13:55 A1 Now if I mount the exact same filesystem this time with the -2 option in the fstab: NFSSERVER:<NFS Filesystem> <MOUNT POINT> nfs rw,soft,-2 0 0 Re-mount the filesystem and using the dovecot to create the same file on the same filesystem I get: root@NFSCLIENT:<NFSv2 Mount>/mail # ls -lh A1 -rw-r--r-- 1 bob wheel 0B Aug 29 12:53 A1 So it seems that with NFSv3 on Freebsd10.1 it doesn't carry the perms over. Yet if I change to NFSv2 the problem goes away. Yet of course I'll end up with issues with files greater than 2gb which is an issue as a few users have 2gb mbox files... I actually this this issue with dovecot writing to the NFSv3 filesystem is somehow linked to the same problem illustrated in BugID: 186293. Please help.
Just to follow up. I have also tested setting the nfs to oldnfs within the fstab and mounting this without any success. With respects to SA: http://www.freebsd.org/security/advisories/FreeBSD-SA-13:05.nfsserver.asc I recompiled the kernel removing the options NFSD and NFSCL and putting in place NFSSERVER and NFSCLIENT which was the old NFSv3 implementation prior to Freebsd 9.x. Yet even with this change in place it still seems that NFSv3 mounted filesystems seem not to carry the correct perms. The reason I'm raising this is that I have a Freebsd 8.x system mounting the same file system right now using NFSv3 without any issues. Just migrating to 10.1 and of course NFS issues...
This sounds like a reported case of a problem that was pretty clearly a Solaris server bug. From the wireshark trace I have, the client: - does an exclusive open, which succeeds - does a setattr with mode 0644, which succeeds --> Then the file's attributes are returned by the subsequent write with mode 0, even though it replied NFS_OK to the setattr for mode 0644 When I tried to contact someone in the NFS Solaris engineering group, I got a "we don't talk to anyone without a maintenance contract". The person who reported this via email tried making a bug report to Solaris, but ended up converting his server to FreeBSD to fix the problem. His test case for the above mentioned wireshark trace was extracting a small tarball with one small file in it. You could try the same test and see if the you get the same result. If unrolling the tarball doesn't cause the problem, you could email me a packet trace for the failure, but please try and make it as small as possible. (Or look at it in wireshark and look for an Exclusive CREATE, followed by a SETATTR that sets the mode, both returning NFS_OK.) I guess you are stuck with NFSv2 (which does file creation differently) or bugging Oracle/Solaris about the bug. Also, if the simple "tar xpf <file>" fails for FreeBSD 10 but succeeds for FreeBSD8.3 against the Solaris server, I could look at a packet trace from the FreeBSD8.3 case and see how it differs. (It still seems clear it is a Solaris server bug, but maybe a client change could be created to work around it.) To get a packet trace, you can do the following on the FreeBSD client: # tcpdump -s 0 -w <file>.pcap host <server-hostname> Good luck with it, rick ps: I'll email the packet trace I have to you, so you can look at it in wireshark.
I'll take this one.
Hi Rick, The I responded to your email with the traces. Note that I have not migrated to a FreeBSD NFS server just yet as I have quite a few clients right now and I haven't been able to truly test the stability of migrating. This bug with Solaris has though pushed me to revert to the oldnfs client code and modify some of the kernel code. From what I take from the traces is that Freebsd set's the perms and the touch time within the same NFS request. The Solaris server commits the time change but ignores the perms modifications. Thus the changes that I did was to let the server set the time on the file rather than the client. All my hosts are time sync'ed so the effect is minimized.
Ok, this is useful information. I can probably add a mount option that forces times to be set to server times (and document in the man page that it is required for some Solaris server). Good work figuring this out. (I suspected that it was something in the Setattr that confused the server or everyone would be complaining.) I'll leave this open until a mount option has been added.
After discussion on freebsd-fs@, instead of adding the mount option, the pre-FreeBSD9.2 behaviour of using xx_TOSERVER for setting times for the Setattr RPC used on exclusive create unconditionally has been done in head via r276347. This patch will be MFC'd to stable/9 and stable/10 in April 2015. Until then, the problem can be avoided by applying the patch and building a custom kernel or using an NFSv2, since NFSv2 doesn't exhibit the problem. The r276347 patch is: --- fs/nfsclient/nfs_clport.c.sav 2014-12-25 12:54:25.000000000 -0500 +++ fs/nfsclient/nfs_clport.c 2014-12-25 12:55:49.000000000 -0500 @@ -1096,9 +1096,16 @@ nfscl_checksattr(struct vattr *vap, stru * us to do a SETATTR RPC. FreeBSD servers store the verifier * in atime, but we can't really assume that all servers will * so we ensure that our SETATTR sets both atime and mtime. + * Set the VA_UTIMES_NULL flag for this case, so that + * the server's time will be used. This is needed to + * work around a bug in some Solaris servers, where + * setting the time TOCLIENT causes the Setattr RPC + * to return NFS_OK, but not set va_mode. */ - if (vap->va_mtime.tv_sec == VNOVAL) + if (vap->va_mtime.tv_sec == VNOVAL) { vfs_timestamp(&vap->va_mtime); + vap->va_vaflags |= VA_UTIMES_NULL; + } if (vap->va_atime.tv_sec == VNOVAL) vap->va_atime = vap->va_mtime; return (1);
Since commit r276347 works around this problem and it has been MFC'd, I'll close this. We never found out if Solaris has fixed their bug.