Bug 233436 - rm -rf fails when the number of subfolders of a folder in a tree is too high
Summary: rm -rf fails when the number of subfolders of a folder in a tree is too high
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 11.2-STABLE
Hardware: Any Any
: --- Affects Some People
Assignee: freebsd-bugs mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-11-23 14:02 UTC by Lars Liedtke
Modified: 2018-12-04 23:02 UTC (History)
4 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Lars Liedtke 2018-11-23 14:02:24 UTC
Hi,

I stumbled over something, which I think should not happen:

A customer asked me to delete some releases of theirs, which their deployment tool did not clean up properly. 

so I tried to rm -rf <Release>, which gave me the following result.

rm -rf <release>
rm: <release>/foo/bar/5a4b9dcdc0385d3e9d0827907f0e5f10792a8e7f: Operation not permitted
rm: <release>/foo/bar/f98dba83270209e862dbe2295bd078702bb9fa21: Operation not permitted
rm: <release>/foo/bar/48c4b74bceb996f66faec0e21aa97fb8d0607516: Operation not permitted
rm: <release>/foo/bar/415a52f0bd4ccbea7aadc1a15193b06da3366c37: Operation not permitted
rm: <release>/foo/bar/e475e6ed6fa28e141979cae142ef3a2faf6d9726: Operation not permitted
rm: <release>/foo/bar/dd7c1138264e4e8f3cebcbc16b00fac10fa629b5: Operation not permitted
rm: <release>/foo/bar/76789f907156f96e34f442785dfa7915c8c785ed: Operation not permitted
rm: <release>/foo/bar/7ecf83bcb737a84d8de66e7365b91ce27f778ea4: Operation not permitted
rm: <release>/foo/bar/b601d1dd9812e1b57e50d46edd2a891969b353f3: Operation not permitted
rm: <release>/foo/bar/0dd0c3382236ba9f2d6b3828dc305c2a01f45e87: Operation not permitted
rm: <release>/foo/bar: Directory not empty
rm: <release>/foo: Directory not empty
rm: <release>: Directory not empty

So I went into <release>/foo/bar and did an ls, which showed me a big number of subdirectories like this:

ls -la <Release>/foo/bar

drwxrwxrwx      2 proserver  proserver      3 Jun  7 10:44 ffc6821a8591392ebe03779270c8ec6e843a2b96
drwxrwxrwx      2 proserver  proserver      3 Jun  7 10:44 ffc927f8cc14ce68abdefcbea8f01542c892aea6
drwxrwxrwx      2 proserver  proserver      3 Jun  7 10:42 ffcaabb22cd01c67eb90dc8ee90f488427e987c3
drwxrwxrwx      2 proserver  proserver      3 Jun  7 10:43 ffcafbcd303e3ed81789be0930ba3b8999827443
drwxrwxrwx      2 proserver  proserver      3 Jun  7 10:42 ffcc53a0a31a38f18065861c42d0de0636140067
drwxrwxrwx      2 proserver  proserver      3 Jun  7 10:44 ffccb4ae42018024a90d0cdb7e635f714daef3d6
drwxrwxrwx      2 proserver  proserver      3 Jun  7 10:44 ffcd2c48e4171f25b96587da22a262f11c3123b8
drwxrwxrwx      2 proserver  proserver      3 Jun  7 10:43 ffcd73a168063217dab0ebf35a9ba16b5843c9f1
drwxrwxrwx      2 proserver  proserver      3 Jun  7 10:44 ffd0a1faba88fef9e2dec753ecfcbf2391f44f66
drwxrwxrwx      2 proserver  proserver      3 Jun  7 10:43 ffd28f6ecac48500499e4803aef1a0b1f904129b

counting the number of subdirectories gave me this:

ls -la | wc -l
51550

in other release-directories this could be even 80000+
trying to delete them in <release>/foo/bar:

rm -rf *
su: /bin/rm: Argument list too long

which was expected, but not with rm -rf

I was able to delete them with find in the end but I thought rm -rf would do the trick. Am I wrong here or is it a bug?
Comment 1 pierogmorski 2018-11-29 17:34:39 UTC
Hi,

Just a quick thought, is the immutable flag (schg) set for these files?

You can check via: `ls -lo /path/to/file` and clear it via:
`chflags noschg /path/to/file`.  Then try `rm` again.
Comment 2 pprocacci 2018-12-03 15:17:49 UTC
As the previous poster commented, "Operation not permitted" implies (most likely) an immutable bit set on the files you are attempting to remove.  Though, this doesn't explain why find was successful in removing such files.

Argument list too long *is* expect with `rm -rf *` if you are using a glob in a very large directory. The glob gets expanded to all files within the current directory and sure enough, exceeds the number of arguments available to pass to the utility.

As a side note; an alternative to find that I myself would have probably used:
ls <release>/foo/bar/ | xargs rm -rf
Comment 3 Jilles Tjoelker freebsd_committer 2018-12-04 23:02:36 UTC
This is probably another instance of bug #197336, except that instead of find(1), rm(1) is affected.

The bug is in the common fts(3) code, which assumes that there are no more subdirectories than the st_nlink field reports (minus two for . and ..). This assumption is incorrect with ZFS, since it supports more subdirectories than fit in the st_nlink field's data type (nlink_t). As a result, fts(3) reports to rm(1) that the subdirectories are not directories, and rm(1) tries to unlink(2) them, which fails with [EPERM].

In 12.0, nlink_t has been enlarged so it can report the correct number of subdirectories with ZFS, so this bug should no longer occur on that version. The fts(3) code in 11.x could be fixed in various ways as discussed in bug #197336.