Since we migrated from FreeBSD-10 to FreeBSD11, we see that some files get corrupted. We were able to reproduce the issue by running a script with the following scenario: - mount the filesystem read-write with the -async flag - decompress some archives - unmount the filesystem - remount the filesystem readonly Once we do that, we generally see one or two files being corrupted. Note that this only happens when using some recents SSD. Also note that unmounting the filesystem is necessary to see the problem. When accessing the files before the umount command, they all seems ok. AFAICS, this looks like as if one page was loaded in memory, but wasn't written physically on the disk. Here is a shortned version of the script I am using: #!/bin/sh mount -o async $PARTITION /mnt/disk tar --no-same-owner -C /mnt/disk/new_dir -zxUf archive1.tgz 7zr x -y -bd -o/mnt/disk ./archive2.7z 7zr x -y -bd -o/mnt/disk ./archive3.7z mv /mnt/disk/dir /mnt/disk/old_dir mv /mnt/disk/new_dir /mnt/disk/dir rm -rf /mnt/disk/old_dir umount $MAJ_PARTITION /mnt/disk/ mount -o ro $MAJ_PARTITION /mnt/disk/ check_integrity # here we check the md5 of the decompressed files The filesystem use the following tunings: SNI40A16C0818A9>tunefs -p /dev/ufs/disk tunefs: POSIX.1e ACLs: (-a) disabled tunefs: NFSv4 ACLs: (-N) disabled tunefs: MAC multilabel: (-l) disabled tunefs: soft updates: (-n) disabled tunefs: soft update journaling: (-j) disabled tunefs: gjournal: (-J) disabled tunefs: trim: (-t) enabled tunefs: maximum blocks per file in a cylinder group: (-e) 4096 tunefs: average file size: (-f) 16384 tunefs: average number of files in a directory: (-s) 64 tunefs: minimum percentage of free space: (-m) 8% tunefs: space to hold for metadata blocks: (-k) 2616 tunefs: optimization preference: (-o) time tunefs: volume label: (-L) main
Would it be possible for you to send me your tar file so that I can try to reproduce the problem? If so, please do so to me directly at mckusick@freebsd.org rather than posting it here.
Created attachment 205236 [details] script use to compute the md5 of the files stored in the archive.
In fact I can't give you the file I ran my tests with, but fortunately I was able to reproduce the issue with the archive llvm50-5.0.2_1.txz from pkg (took that one because it was on my computer). I also update my scripts in order to have something more generic, so that I can attach them to this issue. Here is how I did reproduce: - grab the file https://pkg.freebsd.org/FreeBSD:10:amd64/latest/All/llvm50-5.0.2_1.txz (other archives should work too, but this is the one I used for my tests) - untar the archive and generate the md5 file using the script compute_md5.sh - copy the tar/md5 files on a specific device* - run the test in loop using the command "./test_disk.sh llvm50-5.0.2_1.txz /dev/<my_partition> md5_llvm5.txt" * only devices with a recent SSD did show the problem The number of retry to have the problem may vary from 3 up to more than 20 attempts (my last run got one corrupted file after 33 attempts).
Created attachment 205237 [details] script used to trigger the issue
Thanks. I will see if I am able to reproduce the problem.
Small update because some people here are reporting the fact that they got the issue while not using the -async flag. Still, the issue happens a lot less often in this case.
Hi, were you able to reproduce the issue? Right you i'm trying to investigate this problem further. If necessary, i can do some tests and take some logs using ktr/dtrace.
(In reply to Masse Nicolas from comment #7) I was not able to reproduce the problem and had set it aside for now, but will try to get back to it in the next week.
Small update here to add some of my recent observations: - I was able to reproduce the issue by branching a disk on my workstation directly which is running FreeBSD-12 and use a different cpu/chipset. - The issue seems trim-related since I wasn't able to reproduce it anymore after de-activating the trim using tunefs. - I also had the issue using a new empty disk. Still, so far we were able to reproduce it only using a specific model. (you can contact me by private mail if you want more information. We can also send you one of the affected drive if you think this is necessary).
(In reply to Masse Nicolas from comment #9) I was going to suggest that you try disabling TRIM on your disk to see if that helped solve the problem. Some folks that I am working with at Netflix have found some rather severe bugs in TRIM support on some of the flash drives that they have been using. I have bought an array of cheap flash drives on Amazon and tried them out using your suggested tests. I have not been able to precisely reproduce your problems, but I have gotten problems that are caused by what appears to be faulty TRIM implementations (i.e., the problem goes away when I disable TRIM). Is the drive on which you are experiencing the problem a flash drive, and if so does the problem go away when you disable TRIM? If so, then I would chalk up the problem to faulty TRIM microcode. I added TRIM consolidation in -r338517 which helps reduce the number of TRIM commands sent to the drive by aggregating them into bigger pieces. But the bigger pieces can hit other TRIM bugs that do not deal well with big TRIM requests. So you can try disabling vfs.ffs.dotrimcons to see if that makes a difference.
I'd be quite keen to know what the 'bad drives' are. As far as I know, we have a complete database of the relatively few drives that don't implement trim correctly, but I'm always eager to increase it when a new one is found. You can email me privately if you don't want to name the vendor without proper validation they don't get it right in public.
(In reply to Kirk McKusick from comment #10) Tried with vfs.ffs.dotrimcons set to 0. The problem is still present.