Bug 233161

Summary:	bsd find finds only 32756 directories with '-type d' switch
Product:	Base System	Reporter:	emz
Component:	bin	Assignee:	freebsd-bugs (Nobody) <bugs>
Status:	New ---
Severity:	Affects Only Me	CC:	emaste, sigsys
Priority:	---
Version:	11.2-RELEASE
Hardware:	Any
OS:	Any

Description emz 2018-11-12 13:41:15 UTC

I have a file storage with up to 5 milions objects on 1st level (this is running on zfs):

# ls -ld /usr/local/public/profiles
drwxr-xr-x+ 32767 root  wheel  5097384 Nov 12 16:24 /usr/local/public/profiles

'find /usr/local/public/profiles -depth 1' is able to find these objects, and they seem all to be directories (I've checked randomly, plus my colleagues that wrote a software that works with this storage told me there should be no files there).

but find /usr/local/public/profiles -type d -depth 1 is able to find only 32765 directories. I suspect this could be the MAXDIRS constant from ufs.h.

So I made two output files, one produced with 'find /usr/local/public/profiles -depth 1' and one produced with 'find /usr/local/public/profiles -depth 1 -type d', first file contains 5 millions of objects and second contains only 32756 objects, and pretty much all of the missing objects from the bigger file are directories (though I didn't check all the 5 millions for obvious reasons).

Comment 1 sigsys 2018-11-12 14:23:14 UTC

This seems to be caused by an optimization in lib/libc/gen/fts.c that assumes that there can be no more subdirectories in a given directory than indicated by st_nlink (since each subdirectory increases the link count of the parent with its ".." entry).  This doesn't work when st_nlink gets capped by ZFS.  AFAIK, UFS does not allow more subdirectories than st_nlink can count so the problem does not happen there.

Removing "zfs" from ufslike_filesystems[] in fts.c should avoid the bug.  A better fix would be to check if st_nlink seems saturated (with pathconf(2)/_PC_LINK_MAX I guess) and disable the optimization only then.

nlink_t is 64 bits on 12.X so the problem probably doesn't exist there.