Created attachment 221685 [details] Demonstrate fstyp's failure to detect UFS on a suitably evolved volume The code fstyp uses to detect FAT and EXFAT look for magic strings in a block that UFS doesn't use, and fstyp checks for FAT formats before UFS. Together, these have consequence that fstyp will misidentify the file system type on a volume that has at one time been formatted with either of those formats and is then reformatted with newfs without any intervening operation that overwrite the first block on the volume. Reproduction attached. Next, while I haven't reproduced, it looks as if the same issue will apply to a volume that was once NTFS formatted (because the NTFS magic string is in the same location, and the check for NTFS precedes the one for UFS). Finally, because the default autofs special_media map uses fstyp to guess the file system format on a volume, this combined behavior of newfs & fstyp suffices to render such a file system unmountable by that map (though the volume is mountable manually). I ran into this issue on a recently purchased USB storage device after changing the MBR slice type (but not the slice's size) and creating a UFS file system on that slice using newfs, only to find that the automounter couldn't mount the volume. Anyway, it's easy enough for a user to fix a given volume by overwriting some bytes where FAT/EXFAT/NTFS stores its magic string, so perhaps this is just a documentation issue, perhaps for the fstyp man page. (It's not clear to me that any of newfs, fstyp, or the special_media map is doing anything wrong. But they happen not to combine to achieve the desired result in this specific scenario, so something seems to want improvement.)
In 13 (r355996), fstyp can locate exFAT labels and probably avoid a false positive match with UFS most of the time. (Specifically, we look at the first 11 sectors now and verify them against a checksum in sector 12.) (Unfortunately, some (all?) of the 'goto done's in fstyp_exfat() should be changed to 'goto fail'. Until that is done, 13's fstyp is no better than 12's as far as just looking for the magic "EXFAT" string.) newfs probably avoids writing the first sector for historical reasons (don't want to clobber a boot sector) which probably means there is an inherent collision between reimaged classic FAT / UFS volumes. Maybe fstyp could be smarter here (some tie-breaking scheme for multiple positive results) if it is straightforward to be "really sure" a volume is UFS and not FAT.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=ddf61156132b610915325769cbb93ea11be0d433 commit ddf61156132b610915325769cbb93ea11be0d433 Author: Conrad Meyer <cem@FreeBSD.org> AuthorDate: 2021-01-17 19:55:06 +0000 Commit: Conrad Meyer <cem@FreeBSD.org> CommitDate: 2021-01-17 19:55:06 +0000 fstyp(8): fix exfat detection In the presence of high-level errors (spec violations, bad boot blocks checksum), report non-detection instead of detection. PR: 252787 (related, but does not fully address) usr.sbin/fstyp/exfat.c | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-)
By inspection, it appears that the improvements in 13 do not address this issue: I tested with a checkout that includes commit ddf61156, using my attached reproduction to create a suitably goofy disk image, and it still reports exfat after newfs. The UFS superblock starts at 8192 bytes, i.e., 16 sectors, from the start of a volume, right? If that's correct, then the exfat code's examination of the first 12 sectors won't avoid this false positive. Anyhow, if someone wishes to consider this a defect in fstyp, one potential minimally disruptive approach would be to reorder the checks in fstyp.c so that the more common file system formats are checked for last (right now things are checked for in alphabetical order). For the purpose of heuristically guessing the format of a removable medium, I'd say that EXFAT or FAT is most likely (since commodity media is often pre-formatted that way), and so "falling through" to check for those after ruling out UFS, ZFS, etc., will arrive at the correct answer while avoiding this particular false positive. That said, I'd be satisfied considering the possibility of a false positive from fstyp just a documentation issue. It's easy enough to overwrite the first sector of a disk with dd, for instance, so long as one knows that that's the cause of the false positive.
(In reply to Richard M Kreuter from comment #3) I think this approach sounds reasonable - UFS newfs on top of a preformatted FAT/exFAT USB stick seems a likely case.
A commit in branch stable/12 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=a072e2133ed36f1edf899068e6b026b129c919d8 commit a072e2133ed36f1edf899068e6b026b129c919d8 Author: Conrad Meyer <cem@FreeBSD.org> AuthorDate: 2021-01-17 19:55:06 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2021-02-02 21:12:31 +0000 fstyp(8): fix exfat detection In the presence of high-level errors (spec violations, bad boot blocks checksum), report non-detection instead of detection. PR: 252787 (related, but does not fully address) (cherry picked from commit ddf61156132b610915325769cbb93ea11be0d433) usr.sbin/fstyp/exfat.c | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-)
I just ran into this problem on 13. fstyp was reporting that the partition was "ex2fs", even though I had newly created the partition and run newfs on it seconds before.
^Triage: is this still a problem on modern versions of FreeBSD?
I believe there have been no changes since cem's improvement (which did not fully resolve the issue).
Created attachment 255444 [details] Tweak the reproducer script slightly. The initial version of this script depends on having mkextfatfs installed (but explains how to reproduce using newfs_msdos instead). This changes the script to use newfs_msdos, so that it'll reproduce on a vanilla base system; it also adds 'set -e', as nothing in this script ought to exit non-zero.
I don't currently have anything newer than 14.1-RELEASE-p6 to try it on, but the reproducer script still shows the bug. (I've just tweaked it slightly so as not to depend on anything outside of the base system.)
proposed fix: https://reviews.freebsd.org/D47855
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=6830340cfad6a69715d03f5062f76b65e988e6e9 commit 6830340cfad6a69715d03f5062f76b65e988e6e9 Author: Robert Wing <rew@FreeBSD.org> AuthorDate: 2024-12-04 07:19:23 +0000 Commit: Robert Wing <rew@FreeBSD.org> CommitDate: 2024-12-04 08:45:09 +0000 fstyp: search for file system headers with the largest offset first fstyp can misidentify a UFS file system as MS-DOS if the device was repurposed from MS-DOS to UFS via newfs. This happens for the following reasons: - the header for MS-DOS begins at offset 0 - the superblock for UFS begins at offset 64k, 8k, 0k, or 256k - newfs does not clear the area in front of UFS's superblock, leaving the MS-DOS header intact. - fstyp searches for file system headers alphabetically To avoid this misidentification, have fstyp search for file system headers with the largest offset first instead of alphabetically. The implemented fix was suggested by reporter, Richard M. Kreuter. PR: 252787 Reviewed by: imp, emaste Differential Revision: https://reviews.freebsd.org/D47855 usr.sbin/fstyp/fstyp.c | 42 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 36 insertions(+), 6 deletions(-)