I've got a 11.3p6/amd64 UFS-based system, where Samba 4.10 is running in a jail as an AD member. Yesterday I tried enabling vfs_fruit and streams_xattr: apparently everything worked fine for a while and extended attributes were created on files/directory a Mac client accessed. After a while the client lost connection to the server: I checked and saw a smbd process taking 100% CPU; such a process was unkillable to the point it even prevented rebooting. Reset button had to be pressed. I pinpointed this to reading the extended attributes that were attached to some file in a particular directory. I.e. Something like: find {dir} -exec lsextattr user "{}" ";" produces a process: lsextattr user {file} taking 100% cpu and not willing to die in any way. Again, only the reset button could get me out of this situation. "fsck -y" found no sign of troubles on that filesystem. I was able to use rsync to make a copy of the whole share without extended attributes, so the server is working again. However, for now, I'm keeping the "bad" copy around for testing. System is 11.3p6/amd64 with custom kernel, i.e.: _ UFS_ACL, UFS_GJOURNAL, QUOTA, MD_ROOT, NFS*, COMPAT*, AUDIT, CAPABILIT*, MAC, RACCT*, RCTL were removed, along with unused drivers; _ GEOM_MIRROR, aesni and NULLFS were added. # tunefs -p /dev/mirror/gm0f tunefs: POSIX.1e ACLs: (-a) disabled tunefs: NFSv4 ACLs: (-N) disabled tunefs: MAC multilabel: (-l) disabled tunefs: soft updates: (-n) enabled tunefs: soft update journaling: (-j) disabled tunefs: gjournal: (-J) disabled tunefs: trim: (-t) disabled tunefs: maximum blocks per file in a cylinder group: (-e) 4096 tunefs: average file size: (-f) 16384 tunefs: average number of files in a directory: (-s) 64 tunefs: minimum percentage of free space: (-m) 8% tunefs: space to hold for metadata blocks: (-k) 6408 tunefs: optimization preference: (-o) time Note soft update journaling is now temporarily disabled, but was enabled when the problem first arose. One intersting thing would be to exfiltrate this "bad" directory and copy it to a test server, as I cannot perform many things on this remote production system (especially considering I cannot press reset remotely). However just reading the data hangs it, so I have no idea how to do this.
A commit references this bug: Author: cem Date: Fri Oct 30 19:00:43 UTC 2020 New revision: 367181 URL: https://svnweb.freebsd.org/changeset/base/367181 Log: UFS2: Fix DoS due to corrupted extattrfile Prior versions of FreeBSD (11.x) may have produced a corrupt extattr file. (Specifically, r312416 accidentally fixed this defect by removing a strcpy.) CURRENT FreeBSD supports disk images from those prior versions of FreeBSD. Validate the internal structure as soon as we read it in from disk, to prevent these extattr files from causing invariants violations and DoS. Attempting to access the extattr portion of these files results in EINTEGRITY. At this time, the only way to repair files damaged in this way is to copy the contents to another file and move it over the original. PR: 244089 Reported by: Andrea Venturoli <ml AT netfence.it> Reviewed by: kib Discussed with: mckusick (earlier draft) Security: no Differential Revision: https://reviews.freebsd.org/D27010 Changes: head/sys/ufs/ffs/ffs_vnops.c head/sys/ufs/ufs/extattr.h
cem: any reason not to MFC this?
None that I know of.
Created attachment 221281 [details] Patch against 11.4 While at it, this patch (againg 11.4) avoids the problem arising in the first place (at least in my case).
A commit in branch stable/12 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=4696baaaa95ce9a75a7b1989474382c85981f805 commit 4696baaaa95ce9a75a7b1989474382c85981f805 Author: Conrad Meyer <cem@FreeBSD.org> AuthorDate: 2020-10-30 19:00:42 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2021-01-05 13:56:23 +0000 UFS2: Fix DoS due to corrupted extattrfile Prior versions of FreeBSD (11.x) may have produced a corrupt extattr file. (Specifically, r312416 accidentally fixed this defect by removing a strcpy.) CURRENT FreeBSD supports disk images from those prior versions of FreeBSD. Validate the internal structure as soon as we read it in from disk, to prevent these extattr files from causing invariants violations and DoS. Attempting to access the extattr portion of these files results in EINTEGRITY. At this time, the only way to repair files damaged in this way is to copy the contents to another file and move it over the original. PR: 244089 Reported by: Andrea Venturoli <ml AT netfence.it> Reviewed by: kib Discussed with: mckusick (earlier draft) (cherry picked from commit e6790841f749b3cc1ac7d338181d9358aae04d0b) sys/ufs/ffs/ffs_vnops.c | 31 ++++++++++++++++++++----------- sys/ufs/ufs/extattr.h | 2 +- 2 files changed, 21 insertions(+), 12 deletions(-)
It looks like the bug was present when this functionality was introduced, 0176455bc801fbdd4c265de7415843eb7b3ecf6c I think we should at least apply your strcpy->memcpy change, although perhaps we want to apply the full 675c187cc41a91dad988fc43434ea63d4370c2de.
A commit in branch stable/11 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=eebccaae36722f62bc8f05e6c71b867d69faca5f commit eebccaae36722f62bc8f05e6c71b867d69faca5f Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2021-01-18 16:58:38 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2021-01-18 18:49:56 +0000 ffs: avoid creating corrupt extattrfile This is part of r312416 / e6790841f749, suggested by ml@netfence.it, and at least means we will stop creating corrupt extattr that is not handled by some later versions. PR: 244089 sys/ufs/ffs/ffs_vnops.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
I believe the current status is: 11.4 will create a fs with the issue, minimal fix is in stable/11 (eebccaae3672) 12.1 and 12.2 contain r312416 and so will not create a fs with the issue None of the releases have r367181 and all may misbehave when presented with a fs with this issue. I'm not sure how straightforward merging r367181 to stable/11 will be. My suggestion is that we include eebccaae3672 in an 11.4 errata update to avoid creating new fs with this issue but leave cem's r367181 for 12.3 and 13.0.
A commit in branch releng/11.4 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=612e060c036f9328e6c19a3525d33d1d33006e04 commit 612e060c036f9328e6c19a3525d33d1d33006e04 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2021-01-18 16:58:38 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2021-01-29 00:06:55 +0000 ffs: avoid creating corrupt extattrfile This is part of r312416 / e6790841f749, suggested by ml@netfence.it, and will stop the kernel from creating corrupt extattr. PR: 244089 (cherry picked from commit eebccaae36722f62bc8f05e6c71b867d69faca5f) Approved by: so sys/ufs/ffs/ffs_vnops.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Thank you for your report and investigation! The source of the corruption is now fixed in all supported branches. As it stands the robustness improvements will appear in 12.3 and 13.0.
(In reply to Ed Maste from comment #10) Big +1, thanks Andrea for the report and investigation.