Bug 233161 - bsd find finds only 32756 directories with '-type d' switch
Summary: bsd find finds only 32756 directories with '-type d' switch
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 11.2-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-bugs mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-11-12 13:41 UTC by emz
Modified: 2018-11-12 14:42 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description emz 2018-11-12 13:41:15 UTC
I have a file storage with up to 5 milions objects on 1st level (this is running on zfs):

# ls -ld /usr/local/public/profiles
drwxr-xr-x+ 32767 root  wheel  5097384 Nov 12 16:24 /usr/local/public/profiles

'find /usr/local/public/profiles -depth 1' is able to find these objects, and they seem all to be directories (I've checked randomly, plus my colleagues that wrote a software that works with this storage told me there should be no files there).

but find /usr/local/public/profiles -type d -depth 1 is able to find only 32765 directories. I suspect this could be the MAXDIRS constant from ufs.h.

So I made two output files, one produced with 'find /usr/local/public/profiles -depth 1' and one produced with 'find /usr/local/public/profiles -depth 1 -type d', first file contains 5 millions of objects and second contains only 32756 objects, and pretty much all of the missing objects from the bigger file are directories (though I didn't check all the 5 millions for obvious reasons).
Comment 1 sigsys 2018-11-12 14:23:14 UTC
This seems to be caused by an optimization in lib/libc/gen/fts.c that assumes that there can be no more subdirectories in a given directory than indicated by st_nlink (since each subdirectory increases the link count of the parent with its ".." entry).  This doesn't work when st_nlink gets capped by ZFS.  AFAIK, UFS does not allow more subdirectories than st_nlink can count so the problem does not happen there.

Removing "zfs" from ufslike_filesystems[] in fts.c should avoid the bug.  A better fix would be to check if st_nlink seems saturated (with pathconf(2)/_PC_LINK_MAX I guess) and disable the optimization only then.

nlink_t is 64 bits on 12.X so the problem probably doesn't exist there.