Bug 217560 - FAT32 - Time stamp of file is one hour off
Summary: FAT32 - Time stamp of file is one hour off
Status: In Progress
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 11.0-RELEASE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-fs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-03-05 11:14 UTC by Erik Nordstrøm
Modified: 2018-06-07 03:32 UTC (History)
5 users (show)

See Also:


Attachments
Use vfs_timestamp() instead of getnanotime() in msdosfs (2.14 KB, patch)
2018-04-29 03:37 UTC, Damjan Jovanovic
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Erik Nordstrøm 2017-03-05 11:14:08 UTC
I live in time zone GMT + 1.

While trying something for another issue (#217440), I created a file on a FAT32 formatted USB stick using my Sony PlayStation 4.

The PlayStation 4 is showing the correct local time and I have verified that it has its time zone configured correctly as "UTC+01:00 Oslo".

My laptop is showing the correct local time and I have the TZ env variable set correctly as "Europe/Oslo".

I took a screenshot in the game Alien: Isolation on the PS4 and transferred it to my USB stick.

I mounted the USB stick with

# mount_msdosfs -L en_US.UTF-8 /dev/da1s1 /mnt/

as per the discussion in issue #217440

but when I list the directory contents, the time stamp of the files is off by one hour into the future compared to actual time when the file was created as shown below. Note also that the file name itself contains the correct time when the file was created. I transferred the file to my memory stick within minutes of having taken the screenshot.

I don't have a windows system to create files from at the moment so I can't verify whether this is a problem with FreeBSD handling of FAT32 or if the incorrect timestamp was caused by the PS4.

I have attached an image of a minimal FAT32 partition which contains the directory structure and the screenshot which was written to the USB stick by the PS4.

The USB stick was prepared with

# umount /dev/da1s1
$ gpart delete -i 1 /dev/da1
$ gpart destroy /dev/da1
# dd if=/dev/zero of=/dev/da1 bs=16m
$ gpart create -s mbr /dev/da1
$ gpart add -t fat32 -s 4g -i 1 /dev/da1
# newfs_msdos -F32 /dev/da1s1
# sync

$ gpart show da1
=>      63  30277569  da1  MBR  (14G)
        63      8161       - free -  (4.0M)
      8224   8388608    1  fat32  (4.0G)
   8396832  21880800       - free -  (10G)

Unplugged the USB stick, put it in the PS4, took a screenshot and transferred it to the USB stick. Put it back in the computer and made a minimal image of the FAT32 partition;

# umount /dev/da1s1
$ dd if=/dev/da1s1 of=usbstick.img conv=sparse bs=16m count=1
$ hd usbstick.img > tmp.img
$ xxd -r tmp.img > usbstick.img
$ rm tmp.img
$ ls -hal usbstick.img
-rw-r--r--  1 erikn  erikn   1.5M Mar  5 12:10 usbstick.img

To mount the image;

# mdconfig -a -t vnode -f usbstick.img -u 0
# mount_msdosfs -L en_US.UTF-8 /dev/md0 /mnt/

Here we have the file with the mismatching timestamp.

$ ls -al /mnt/PS4/SHARE/Screenshots/Alien_\ Isolation™/
total 768
drwxrwxrwx  1 root  wheel   32768 Mar  5 13:07 ./
drwxrwxrwx  1 root  wheel   32768 Mar  5 13:07 ../
-rwxrwxrwx  1 root  wheel  305981 Mar  5 13:07 Alien_ Isolation™_20170305120653.jpg*

Unmount image;

# umount /dev/md0
# mdconfig -d -u 0
Comment 1 Erik Nordstrøm 2017-03-05 11:21:03 UTC
Image was about half of a megabyte too big to be attached.

You can download a copy of the ~1.5MiB file from https://www.nordstroem.no/blob/7e/bc/32e9d7c-1534776.img
Comment 2 David Bright freebsd_committer 2018-02-20 20:31:27 UTC
If I'm not mistaken, the FAT file system stores timestamps in local time. When that FAT file system is mounted in a Unix-like system (e.g., FreeBSD), the timestamp is interpreted as if it is UTC; that system cannot know what the timezone was on the system that wrote the FAT file system. So, since you are +1:00 from UTC, you see the timestamp one hour in the future.
Comment 3 David Bright freebsd_committer 2018-02-20 20:33:41 UTC
(In reply to David Bright from comment #2)

The discussion above presumes that your PlayStation 4 actually records the time as an MS-DOS/Windows system does. I don't know if that is actually the case.
Comment 4 Damjan Jovanovic 2018-04-29 01:37:05 UTC
There is definitely some timestamp bug here. In my UTC+2 timezone:

$ date  
Sun Apr 29 03:23:58 SAST 2018
$ touch bsd-201804290323.txt
$ stat -x bsd-201804290323.txt 
  File: "bsd-201804290323.txt"
  Size: 0            FileType: Regular File
  Mode: (0755/-rwxr-xr-x)         Uid: ( 1002/   user)  Gid: (    0/   wheel)
Device: 0,139   Inode: 59620    Links: 1
Access: Sun Apr 29 02:00:00 2018
Modify: Sun Apr 29 05:23:58 2018
Change: Sun Apr 29 05:23:58 2018

Windows shows the correct modified/changed times of 03:23. Why is FreeBSD adding my time zone offset (2 hours) to the times when stat reads them, even though they are already in local time?
Comment 5 Damjan Jovanovic 2018-04-29 03:37:54 UTC
Created attachment 192889 [details]
Use vfs_timestamp() instead of getnanotime() in msdosfs

I've noticed msdosfs uses getnanotime() to generate timestamps, unlike ufs, ext2fs, devfs, nandfs, nfsclient, nfsserver, and tmpfs, which all use the better vfs_timestamp() function instead (which allows filesystem timestamp granularity to be configured using the vfs.timestamp_precision sysctl). Only autofs and msdosfs use getnanotime().

Patching msdosfs to use vfs_timestamp() instead, seems to also fix this bug here, with msdosfs timestamps having the timezone offset added to them a second time when read.

Please find my patch attached.
Comment 6 Konstantin Belousov freebsd_committer 2018-04-29 12:55:20 UTC
(In reply to Damjan Jovanovic from comment #5)
Change to use vfs_timestamp() should be fine by itself, but I cannot see how could it solve 'off by a hour' issue.
Comment 7 Damjan Jovanovic 2018-04-29 13:00:06 UTC
(In reply to Konstantin Belousov from comment #6)

By default vfs.timestamp_precision is TSP_USEC, which returns microtime() instead of getnanotime(). The timezone for microtime() seems to be more correct than for getnanotime().
Comment 8 Konstantin Belousov freebsd_committer 2018-04-29 13:23:59 UTC
(In reply to Damjan Jovanovic from comment #7)
This does not make sense, there is no 'timezone', and esp. there is no 'more correct' timezone.

The difference between e.g. getmicrotime() and microtime() is that get return current clock hands without quering hardware to get the most precise possible value for the timecounter.  So the typical precision of of getXXX() is 1/hz, while precision of XXX() is around the precision of hardware.  But both variants return UTC time as known by kernel.
Comment 9 Damjan Jovanovic 2018-04-29 14:07:32 UTC
(In reply to Konstantin Belousov from comment #8)

Thank you, I'll try that test again with a clean kernel rebuild.

MICROTIME(9) does not document the timezone for the time functions.
Comment 10 Damjan Jovanovic 2018-04-29 14:30:16 UTC
(In reply to Damjan Jovanovic from comment #9)

Something must have been wrong with my setup. With a clean kernel rebuild, I can no longer reproduce this bug, even without my patch. Sorry.

That patch may still be worth committing regardless ;).
Comment 11 Konstantin Belousov freebsd_committer 2018-04-29 16:07:45 UTC
(In reply to Damjan Jovanovic from comment #10)
Did you tweaked machdep.adjkerntz or machdep.wall_cmos_clock between tests ?
Comment 12 Damjan Jovanovic 2018-04-29 18:14:47 UTC
(In reply to Konstantin Belousov from comment #11)

During my early tests, I did change CMOS clock to local time in the BIOS, and run tzsetup selecting that the CMOS clock is in local time, but I still got the same bad times after that.

Most probably, between experimenting with several patches to the msdosfs code that passed utc=1 to fattime2timespec() and/or timespec2fattime(), and sometimes forgetting to run "make installkernel" after "make buildkernel", I ended up with a badly patched msdosfs binary, that kept giving bad times until I did a clean rebuild.
Comment 13 David Bright freebsd_committer 2018-06-07 03:01:20 UTC
Commits base r333311 (-current) and  base r334742 (MFC to stable/11) applied the suggested patch (or similar). Can this bug now be closed? Added pfg@ (who committed the change) to CC List to get his input on this question, too.
Comment 14 Pedro F. Giffuni freebsd_committer 2018-06-07 03:29:28 UTC
(In reply to David Bright from comment #13)
The patch was committed but it is unrelated the bug, and that's why the PR was not mentioned in the commit.

I think the issue might have been related to an update of the timezone database though. Can one of the affected users confirm the bug is gone?
Comment 15 Pedro F. Giffuni freebsd_committer 2018-06-07 03:32:23 UTC
Comment on attachment 192889 [details]
Use vfs_timestamp() instead of getnanotime() in msdosfs

Obsolete the patch as it was committed in r333311 and MFCd to 11/stable.