Bug 238663 - [UFS] Corrupted files since migration to FreeBSD11
Summary: [UFS] Corrupted files since migration to FreeBSD11
Status: Closed Unable to Reproduce
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 11.2-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-fs (Nobody)
Keywords: regression
Depends on:
Reported: 2019-06-17 12:20 UTC by Masse Nicolas
Modified: 2021-02-26 23:06 UTC (History)
5 users (show)

See Also:

script use to compute the md5 of the files stored in the archive. (318 bytes, application/x-shellscript)
2019-06-20 14:26 UTC, Masse Nicolas
no flags Details
script used to trigger the issue (1.48 KB, application/x-shellscript)
2019-06-20 14:28 UTC, Masse Nicolas
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Masse Nicolas 2019-06-17 12:20:24 UTC
Since we migrated from FreeBSD-10 to FreeBSD11, we see that some files get corrupted.
We were able to reproduce the issue by running a script with the following scenario:

- mount the filesystem read-write with the -async flag
- decompress some archives
- unmount the filesystem
- remount the filesystem readonly

Once we do that, we generally see one or two files being corrupted.

Note that this only happens when using some recents SSD.
Also note that unmounting the filesystem is necessary to see the problem. When accessing the files before the umount command, they all seems ok.

AFAICS, this looks like as if one page was loaded in memory, but wasn't written physically on the disk.

Here is a shortned version of the script I am using:

mount -o async $PARTITION /mnt/disk
tar --no-same-owner -C /mnt/disk/new_dir -zxUf archive1.tgz
7zr x -y -bd -o/mnt/disk ./archive2.7z
7zr x -y -bd -o/mnt/disk ./archive3.7z
mv /mnt/disk/dir /mnt/disk/old_dir
mv /mnt/disk/new_dir /mnt/disk/dir
rm -rf /mnt/disk/old_dir
umount $MAJ_PARTITION /mnt/disk/
mount -o ro $MAJ_PARTITION /mnt/disk/
check_integrity # here we check the md5 of the decompressed files

The filesystem use the following tunings:
SNI40A16C0818A9>tunefs -p /dev/ufs/disk
tunefs: POSIX.1e ACLs: (-a)                                disabled
tunefs: NFSv4 ACLs: (-N)                                   disabled
tunefs: MAC multilabel: (-l)                               disabled
tunefs: soft updates: (-n)                                 disabled
tunefs: soft update journaling: (-j)                       disabled
tunefs: gjournal: (-J)                                     disabled
tunefs: trim: (-t)                                         enabled
tunefs: maximum blocks per file in a cylinder group: (-e)  4096
tunefs: average file size: (-f)                            16384
tunefs: average number of files in a directory: (-s)       64
tunefs: minimum percentage of free space: (-m)             8%
tunefs: space to hold for metadata blocks: (-k)            2616
tunefs: optimization preference: (-o)                      time
tunefs: volume label: (-L)                                 main
Comment 1 Kirk McKusick freebsd_committer 2019-06-18 17:48:16 UTC
Would it be possible for you to send me your tar file so that I can try to reproduce the problem? If so, please do so to me directly at mckusick@freebsd.org rather than posting it here.
Comment 2 Masse Nicolas 2019-06-20 14:26:55 UTC
Created attachment 205236 [details]
script use to compute the md5 of the files stored in the archive.
Comment 3 Masse Nicolas 2019-06-20 14:27:13 UTC
In fact I can't give you the file I ran my tests with, but fortunately I was able to reproduce the issue with the archive llvm50-5.0.2_1.txz from pkg (took that one because it was on my computer).
I also update my scripts in order to have something more generic, so that I can attach them to this issue.

Here is how I did reproduce:
- grab the file https://pkg.freebsd.org/FreeBSD:10:amd64/latest/All/llvm50-5.0.2_1.txz (other archives should work too, but this is the one I used for my tests)
- untar the archive and generate the md5 file using the script compute_md5.sh
- copy the tar/md5 files on a specific device*
- run the test in loop using the command "./test_disk.sh llvm50-5.0.2_1.txz /dev/<my_partition> md5_llvm5.txt"

* only devices with a recent SSD did show the problem

The number of retry to have the problem may vary from 3 up to more than 20 attempts (my last run got one corrupted file after 33 attempts).
Comment 4 Masse Nicolas 2019-06-20 14:28:34 UTC
Created attachment 205237 [details]
script used to trigger the issue
Comment 5 Kirk McKusick freebsd_committer 2019-06-20 17:29:40 UTC
Thanks. I will see if I am able to reproduce the problem.
Comment 6 Masse Nicolas 2019-07-04 13:31:24 UTC
Small update because some people here are reporting the fact that they got the issue while not using the -async flag. Still, the issue happens a lot less often in this case.
Comment 7 Masse Nicolas 2019-08-01 07:10:36 UTC

were you able to reproduce the issue?
Right you i'm trying to investigate this problem further. If necessary, i can do some tests and take some logs using ktr/dtrace.
Comment 8 Kirk McKusick freebsd_committer 2019-08-01 07:22:38 UTC
(In reply to Masse Nicolas from comment #7)
I was not able to reproduce the problem and had set it aside for now, but will try to get back to it in the next week.
Comment 9 Masse Nicolas 2019-08-14 08:56:51 UTC
Small update here to add some of my recent observations:
- I was able to reproduce the issue by branching a disk on my workstation directly which is running FreeBSD-12 and use a different cpu/chipset.
- The issue seems trim-related since I wasn't able to reproduce it anymore after de-activating the trim using tunefs.
- I also had the issue using a new empty disk. Still, so far we were able to reproduce it only using a specific model. (you can contact me by private mail if you want more information. We can also send you one of the affected drive if you think this is necessary).
Comment 10 Kirk McKusick freebsd_committer 2019-08-14 17:49:25 UTC
(In reply to Masse Nicolas from comment #9)
I was going to suggest that you try disabling TRIM on your disk to see if that helped solve the problem. Some folks that I am working with at Netflix have found some rather severe bugs in TRIM support on some of the flash drives that they have been using.

I have bought an array of cheap flash drives on Amazon and tried them out using your suggested tests. I have not been able to precisely reproduce your problems, but I have gotten problems that are caused by what appears to be faulty TRIM implementations (i.e., the problem goes away when I disable TRIM).

Is the drive on which you are experiencing the problem a flash drive, and if so does the problem go away when you disable TRIM? If so, then I would chalk up the problem to faulty TRIM microcode.

I added TRIM consolidation in -r338517 which helps reduce the number of TRIM
commands sent to the drive by aggregating them into bigger pieces. But the bigger pieces can hit other TRIM bugs that do not deal well with big TRIM requests. So you can try disabling vfs.ffs.dotrimcons to see if that makes a difference.
Comment 11 Warner Losh freebsd_committer 2019-08-14 19:39:22 UTC
I'd be quite keen to know what the 'bad drives' are. As far as I know, we have a complete database of the relatively few drives that don't implement trim correctly, but I'm always eager to increase it when a new one is found. You can email me privately if you don't want to name the vendor without proper validation they don't get it right in  public.
Comment 12 Masse Nicolas 2019-08-16 10:03:36 UTC
(In reply to Kirk McKusick from comment #10)
Tried with vfs.ffs.dotrimcons set to 0. The problem is still present.
Comment 13 Kirk McKusick freebsd_committer 2021-02-26 23:06:44 UTC
I made another pass over a newer collection of flash drives and have not been able to reproduce these errors. In any event, it seems this is happening due to software glitches in the SSD firmware rather than a bug in FreeBSD code. Feel free to reopen this report if you have reason to believe that you have new evidence of a problem in FreeBSD.