This is probably related to the ZFS feature of throttling down the transaction size when writing data to very near-full (refquota) filesystems but anyway. System: Servers with many ZFS filesystems (HOME directories) for many users, shared via NFSv4 and SMB (Samba). We are seeing DoS issues for NFS users on the same machine where some _other_ user have filled their HOME directories so they are at, or very near (MB's), their refquota limits at the time when we are taking our hourly recursive snapshots of all filesystems on those servers. ("zfs snapshot -r DATA/homes" basically) The NFS users (they have their HOME directories mounted from those servers) at those times are seeing freezes for over a minute where the NFS server basically seems to be unresponsive. We are also seeing NFS mount requests beeing denied (or rather timed out) at the same times. We haven't seen the same type of bug reports from our SMB users, but their usage patterns are a bit different so this might just be that they haven't noticed the same issue. Possible workarounds (not tested yet): 1. Don't use "zfs snapshot -r" but instead manually loop over all filesystems and skip taking snapshots of the ones that are nearly full. (Modify the source for the "zfs" command to have an option to skip nearly-full filesystems when doing a recursive snap) 2. Temporarily increase the refquota of problematic filesystems before running "zfs snapshot -r". (Problem: We might not be able to lower the quota afterwards). 3. Modify the zfs snapshot stuff in the kernel to ignore the quotas (since it's run by the root users I think if might be reasonable, but possibly not so easy to implement). In general this slowing down of everything ZFS-related when a filesystem is nearly full is starting to _really_ become a pain in the... 3.
Created attachment 212018 [details] Patch to "zfs" command to add a "-m minfree" option for "snapshot"
Uploaded a patch that implements a "-m minfree" option for "zfs snapshot". If specified then any filesystems with less free space available will be skipped. Useful if you have thousands of filesystems and are doing recursive snapshots: zfs snapshot -r -m 10G DATA/users
Created attachment 212053 [details] New patch to add -v, -n & -m<minfree> options to "zfs snapshot" Updated version of the patch that adds -n, -v & -m<minfree> flags to "zfs snapshot". Also updates the man page for zfs.
I'm closing this. I have a workaround (a modified "zfs" command that can intelligently skip near-full datasets). And it's better to see if ZoL handles this better in the future. - Peter