Bug 244089 - lsextattr on a specific UFS file locks system
Summary: lsextattr on a specific UFS file locks system
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 11.3-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-fs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-02-13 08:22 UTC by ml
Modified: 2021-01-29 21:58 UTC (History)
4 users (show)

See Also:


Attachments
Patch against 11.4 (418 bytes, text/plain)
2021-01-05 07:28 UTC, ml
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description ml 2020-02-13 08:22:24 UTC
I've got a 11.3p6/amd64 UFS-based system, where Samba 4.10 is running in a jail as an AD member.

Yesterday I tried enabling vfs_fruit and streams_xattr: apparently everything worked fine for a while and extended attributes were created on files/directory a Mac client accessed.
After a while the client lost connection to the server: I checked and saw a smbd process taking 100% CPU; such a process was unkillable to the point it even prevented rebooting. Reset button had to be pressed.

I pinpointed this to reading the extended attributes that were attached to some file in a particular directory.
I.e.
Something like:
 find {dir} -exec lsextattr user "{}" ";"
produces a process:
 lsextattr user {file}
taking 100% cpu and not willing to die in any way.

Again, only the reset button could get me out of this situation.
"fsck -y" found no sign of troubles on that filesystem.



I was able to use rsync to make a copy of the whole share without extended attributes, so the server is working again.
However, for now, I'm keeping the "bad" copy around for testing.



System is 11.3p6/amd64 with custom kernel, i.e.:
_ UFS_ACL, UFS_GJOURNAL, QUOTA, MD_ROOT, NFS*, COMPAT*, AUDIT, CAPABILIT*, MAC, RACCT*, RCTL were removed, along with unused drivers;
_ GEOM_MIRROR, aesni and NULLFS were added.

# tunefs -p /dev/mirror/gm0f 
tunefs: POSIX.1e ACLs: (-a)                                disabled
tunefs: NFSv4 ACLs: (-N)                                   disabled
tunefs: MAC multilabel: (-l)                               disabled
tunefs: soft updates: (-n)                                 enabled
tunefs: soft update journaling: (-j)                       disabled
tunefs: gjournal: (-J)                                     disabled
tunefs: trim: (-t)                                         disabled
tunefs: maximum blocks per file in a cylinder group: (-e)  4096
tunefs: average file size: (-f)                            16384
tunefs: average number of files in a directory: (-s)       64
tunefs: minimum percentage of free space: (-m)             8%
tunefs: space to hold for metadata blocks: (-k)            6408
tunefs: optimization preference: (-o)                      time

Note soft update journaling is now temporarily disabled, but was enabled when the problem first arose.




One intersting thing would be to exfiltrate this "bad" directory and copy it to a test server, as I cannot perform many things on this remote production system (especially considering I cannot press reset remotely).
However just reading the data hangs it, so I have no idea how to do this.
Comment 1 commit-hook freebsd_committer 2020-10-30 19:01:36 UTC
A commit references this bug:

Author: cem
Date: Fri Oct 30 19:00:43 UTC 2020
New revision: 367181
URL: https://svnweb.freebsd.org/changeset/base/367181

Log:
  UFS2: Fix DoS due to corrupted extattrfile

  Prior versions of FreeBSD (11.x) may have produced a corrupt extattr file.
  (Specifically, r312416 accidentally fixed this defect by removing a strcpy.)
  CURRENT FreeBSD supports disk images from those prior versions of FreeBSD.
  Validate the internal structure as soon as we read it in from disk, to
  prevent these extattr files from causing invariants violations and DoS.

  Attempting to access the extattr portion of these files results in
  EINTEGRITY.  At this time, the only way to repair files damaged in this way
  is to copy the contents to another file and move it over the original.

  PR:		244089
  Reported by:	Andrea Venturoli <ml AT netfence.it>
  Reviewed by:	kib
  Discussed with:	mckusick (earlier draft)
  Security:	no
  Differential Revision:	https://reviews.freebsd.org/D27010

Changes:
  head/sys/ufs/ffs/ffs_vnops.c
  head/sys/ufs/ufs/extattr.h
Comment 2 Ed Maste freebsd_committer 2021-01-04 23:50:26 UTC
cem: any reason not to MFC this?
Comment 3 Conrad Meyer freebsd_committer 2021-01-05 00:57:25 UTC
None that I know of.
Comment 4 ml 2021-01-05 07:28:46 UTC
Created attachment 221281 [details]
Patch against 11.4

While at it, this patch (againg 11.4) avoids the problem arising in the first place (at least in my case).
Comment 5 commit-hook freebsd_committer 2021-01-05 13:57:19 UTC
A commit in branch stable/12 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=4696baaaa95ce9a75a7b1989474382c85981f805

commit 4696baaaa95ce9a75a7b1989474382c85981f805
Author:     Conrad Meyer <cem@FreeBSD.org>
AuthorDate: 2020-10-30 19:00:42 +0000
Commit:     Ed Maste <emaste@FreeBSD.org>
CommitDate: 2021-01-05 13:56:23 +0000

    UFS2: Fix DoS due to corrupted extattrfile

    Prior versions of FreeBSD (11.x) may have produced a corrupt extattr file.
    (Specifically, r312416 accidentally fixed this defect by removing a strcpy.)
    CURRENT FreeBSD supports disk images from those prior versions of FreeBSD.
    Validate the internal structure as soon as we read it in from disk, to
    prevent these extattr files from causing invariants violations and DoS.

    Attempting to access the extattr portion of these files results in
    EINTEGRITY.  At this time, the only way to repair files damaged in this way
    is to copy the contents to another file and move it over the original.

    PR:             244089
    Reported by:    Andrea Venturoli <ml AT netfence.it>
    Reviewed by:    kib
    Discussed with: mckusick (earlier draft)

    (cherry picked from commit e6790841f749b3cc1ac7d338181d9358aae04d0b)

 sys/ufs/ffs/ffs_vnops.c | 31 ++++++++++++++++++++-----------
 sys/ufs/ufs/extattr.h   |  2 +-
 2 files changed, 21 insertions(+), 12 deletions(-)
Comment 6 Ed Maste freebsd_committer 2021-01-18 16:59:10 UTC
It looks like the bug was present when this functionality was introduced, 0176455bc801fbdd4c265de7415843eb7b3ecf6c

I think we should at least apply your strcpy->memcpy change, although perhaps we want to apply the full 675c187cc41a91dad988fc43434ea63d4370c2de.
Comment 7 commit-hook freebsd_committer 2021-01-18 18:53:55 UTC
A commit in branch stable/11 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=eebccaae36722f62bc8f05e6c71b867d69faca5f

commit eebccaae36722f62bc8f05e6c71b867d69faca5f
Author:     Ed Maste <emaste@FreeBSD.org>
AuthorDate: 2021-01-18 16:58:38 +0000
Commit:     Ed Maste <emaste@FreeBSD.org>
CommitDate: 2021-01-18 18:49:56 +0000

    ffs: avoid creating corrupt extattrfile

    This is part of r312416 / e6790841f749, suggested by ml@netfence.it,
    and at least means we will stop creating corrupt extattr that is not
    handled by some later versions.

    PR:             244089

 sys/ufs/ffs/ffs_vnops.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 8 Ed Maste freebsd_committer 2021-01-18 19:48:48 UTC
I believe the current status is:
11.4 will create a fs with the issue, minimal fix is in stable/11 (eebccaae3672)
12.1 and 12.2 contain r312416 and so will not create a fs with the issue
None of the releases have r367181 and all may misbehave when presented with a fs with this issue.

I'm not sure how straightforward merging r367181 to stable/11 will be.

My suggestion is that we include eebccaae3672 in an 11.4 errata update to avoid creating new fs with this issue but leave cem's r367181 for 12.3 and 13.0.
Comment 9 commit-hook freebsd_committer 2021-01-29 00:19:55 UTC
A commit in branch releng/11.4 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=612e060c036f9328e6c19a3525d33d1d33006e04

commit 612e060c036f9328e6c19a3525d33d1d33006e04
Author:     Ed Maste <emaste@FreeBSD.org>
AuthorDate: 2021-01-18 16:58:38 +0000
Commit:     Ed Maste <emaste@FreeBSD.org>
CommitDate: 2021-01-29 00:06:55 +0000

    ffs: avoid creating corrupt extattrfile

    This is part of r312416 / e6790841f749, suggested by ml@netfence.it,
    and will stop the kernel from creating corrupt extattr.

    PR:             244089
    (cherry picked from commit eebccaae36722f62bc8f05e6c71b867d69faca5f)

    Approved by:    so

 sys/ufs/ffs/ffs_vnops.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 10 Ed Maste freebsd_committer 2021-01-29 20:12:17 UTC
Thank you for your report and investigation! The source of the corruption is now fixed in all supported branches. As it stands the robustness improvements will appear in 12.3 and 13.0.
Comment 11 Conrad Meyer freebsd_committer 2021-01-29 21:58:37 UTC
(In reply to Ed Maste from comment #10)
Big +1, thanks Andrea for the report and investigation.