Bug 168947

I set this to serious/medium since it can cause hangs, and not critical since it is only one server and I can work around it, and I'm not sure if it still hangs with the latest FreeBSD I'm running.

-------------------------------------
The messed up .zfs snapshot directory 
(non-critical, but likely related): 
-------------------------------------

FreeBSD local (FreeBSD nfs client looks the same):
root@bcnas1bak:/tank/.zfs/snapshot# ls -ld daily*
drwxr-xr-x  29 root  wheel  29 Apr 11 13:47 daily-2012-04-13T00:00:00
drwxr-xr-x  29 root  wheel  29 Apr 11 13:47 daily-2012-04-14T00:00:01
drwxr-xr-x  29 root  wheel  29 Apr 11 13:47 daily-2012-04-15T00:00:00
drwxr-xr-x  29 root  wheel  29 Apr 11 13:47 daily-2012-04-16T00:00:00
drwxr-xr-x  29 root  wheel  29 Apr 11 13:47 daily-2012-04-17T00:00:00
drwxr-xr-x  29 root  wheel  29 Apr 11 13:47 daily-2012-04-18T00:00:00
..

Linux nfs client:
# mount -t nfs -o nolock bcnas1bak:/tank/bcnasvm1 tank
# cd tank/.zfs/snapshot
# ls -ld daily*

(this line looks fine)
drwxr-xr-x 29 root  root         29 Apr 11 13:47 daily-2012-04-13T00:00:00

(this line is a file, not a directory, and has wrong uid and gid)
-r-xr-xr-x  1 peter vboxusers 50384 Jan 18  2010 daily-2012-06-10T00:18:00

(this line is a link, not a directory)
lrwxrwxrwx  1 root  root         21 May 17  2011 daily-2012-05-20T00:18:00 -> package-x-generic.png

etc.


-------------------------------------
The hang (last seen 8.2-STABLE 
(critical?)
-------------------------------------

But unfortunately, I have failed every attempt to create a 2nd server that will hang. Only one production server hangs this way. The replicated backup server with all the same snapshots (shown above) does not hang.

The hang is likely not related to using up all memory mounting the snapshots. I can also hang it just as easily mounting a small number of them. After running "ls -l" (after a fresh reboot due to previous PR I submitted) on the Linux nfs client, arc is only using 667.29 MiB, and the system has 44 GB free.

When it hangs, only the one dataset is affected. All nfs services are frozen. "ls" in the dataset directories hangs. If I remember correctly, "zpool" commands work, and "zfs list" will hang, but "zfs list tank/someunrelateddataset" will not.

Also note I have not hung my production server this way yet with 8.3. The last hang was with 8.2-STABLE Sept2011.

-------------------------------------
more info...
-------------------------------------

Sometimes I would see messages like this:
Oct 31 14:55:39 bcnas1 mountd[47733]: can't delete exports for /tank/.zfs/snapshot/daily-2011-10-06T09:27:52: Invalid argument

But lately I don't, possibly because I hid the .zfs directory with "zfs set snapdir=hidden tank". Or possibly because of a newer FreeBSD.

How-To-Repeat: To repeat the messed up looking snapshot directory problem, just take a snapshot every day (or maybe just make lots all at once), (and then maybe restart nfs or reboot,) and then connect with a Linux client, and use ls -l in the .zfs/snapshot directory.