Bug 199874

Summary: sysutils/fusefs-exfat: add support for CREATE operations
Product: Ports & Packages Reporter: danny
Component: Individual Port(s)Assignee: Jan Beich <jbeich>
Status: Closed FIXED    
Severity: Affects Some People CC: bright.boat3517, cs, jbeich, samm
Priority: --- Keywords: needs-qa, patch
Version: Latest   
Hardware: Any   
OS: Any   
Bug Depends on: 210823    
Bug Blocks:    
Attachments:
Description Flags
patch-fuse-main.c none

Description danny 2015-05-03 07:35:21 UTC
Created attachment 156261 [details]
patch-fuse-main.c

It looks like fusefs-exfat does not support fuse's CREATE file operator.  Because of this, creating new files does not work on exfat mounted filesystems.

Most other file operations work just fine, such as creating directories or writing to existing files.  Just not anything that (I am assuming) calls creat(2) or O_CREAT under the hood.  Which seems like most everything.

The attached patch adds CREATE support to fusefs-exfat, which makes new file operations work again in FreeBSD.  


Here are examples of create file operations failing (including snippets from fusefs debug to show what is breaking):

  $ touch testfile
  touch: testfile: Function not implemented

    unique: 4, opcode: CREATE (35), nodeid: 1, insize: 53, pid: 5678
       unique: 4, error: -78 (Function not implemented), outsize: 16

  $ echo "test" > testfile
  mksh: can't create testfile: Invalid argument

    unique: 26, opcode: CREATE (35), nodeid: 1, insize: 57, pid: 10351
       unique: 26, error: -78 (Function not implemented), outsize: 16


Here are the same file operations working after the patch:

  $ touch testfile

    unique: 66, opcode: CREATE (35), nodeid: 1, insize: 57, pid: 10880
    create flags: 0x202 /testfile 0100644 umask=0000
       create[34385160192] flags: 0x202 /testfile
    getattr /testfile
       NODEID: 3
       unique: 66, success, outsize: 152
    unique: 67, opcode: GETATTR (3), nodeid: 3, insize: 40, pid: 10880
    getattr /testfile
       unique: 67, success, outsize: 112
    unique: 68, opcode: OPEN (14), nodeid: 3, insize: 48, pid: 10880
    open flags: 0x1 /testfile
       open[34385160192] flags: 0x1 /testfile
       unique: 68, success, outsize: 32

  $ echo "test" > testfile

    unique: 134, opcode: CREATE (35), nodeid: 1, insize: 57, pid: 1687
    create flags: 0x202 /testfile 0100644 umask=0000
       create[34385160192] flags: 0x202 /testfile
    getattr /testfile
       NODEID: 5
       unique: 134, success, outsize: 152
    unique: 135, opcode: GETATTR (3), nodeid: 5, insize: 40, pid: 1687
    getattr /testfile
       unique: 135, success, outsize: 112
    unique: 136, opcode: OPEN (14), nodeid: 5, insize: 48, pid: 1687
    open flags: 0x1 /testfile
       open[34385160192] flags: 0x1 /testfile
       unique: 136, success, outsize: 32
    unique: 137, opcode: GETATTR (3), nodeid: 5, insize: 40, pid: 1687
    getattr /testfile
       unique: 137, success, outsize: 112
    unique: 138, opcode: WRITE (16), nodeid: 5, insize: 69, pid: 1687
    write[34385160192] 5 bytes to 0 flags: 0x0
       write[34385160192] 5 bytes to 0
       unique: 138, success, outsize: 24


Additional information can be see about the fusefs CREATE operator here:

http://fuse.sourceforge.net/doxygen/structfuse__operations.html#a97243e0f9268a96236bc3b6f2bacee17
Comment 1 Oleksii Samorukov freebsd_committer freebsd_triage 2015-05-03 07:42:19 UTC
Hi, which version of FreeBSD you are using? It was working for me before, so probably some recent changes in fuse/bsd support?
Comment 2 danny 2015-05-03 07:49:23 UTC
Apologies, can't believe I left that out!

$ uname -r -s -m
FreeBSD 10.1-RELEASE-p6 amd64

$ pkg info | grep fuse
fusefs-exfat-1.0.1    Full-featured exFAT FS implementation as a FUSE module
fusefs-libs-2.9.3_4   FUSE allows filesystem implementation in userspace
Comment 3 Oleksii Samorukov freebsd_committer freebsd_triage 2015-05-03 12:41:46 UTC
It seems to be a bug of the FreeBSD implementation, see http://fuse.sourceforge.net/doxygen/structfuse__operations.html#a97243e0f9268a96236bc3b6f2bacee17 . 

"If this method is not implemented or under Linux kernel versions earlier than 2.6.15, the mknod() and open() methods will be called instead."

I would recommend to report bug to the upstream because it should affect not only exfat, but every pre-2.5 filesystem.
Comment 4 Oleksii Samorukov freebsd_committer freebsd_triage 2015-05-03 12:54:58 UTC
Anyway, lets keep this PR open, I will try to check myself. It seems to be an issue with fuse-libs.
Comment 5 danny 2015-05-03 18:48:13 UTC
Thanks, and agreed!

From looking at the changelogs, it looks like the feature you are referring to was introduced in fusefs back in 2005.  So it probably isn't a recent upstream change causing this, but instead something with the kernel fuse module introduced in FreeBSD 10.x.

I stumbled on a resolved PR (bug #192563) for sysutils/fusefs-encfs that deals with the same issue, and has a very similar patch to the one I submitted (that would have saved me some time!).

There is another PR (bug #187485) against sysutils/fusefs-wdfs with a similar patch, but that port was dropped so it was never applied.

So we know that this applies to all fusefs modules running under 10.x, and that other modules have hacked in a create -> mknod+open wrapper to get things working (just to confirm that it is a sane workaround).

Agreed about figuring out where the real issue is coming from, I will do more research as I can't find the source of the problem in the kernel source at a glance.  I should probably file a bug against the fuse kernel module.
Comment 6 Oleksii Samorukov freebsd_committer freebsd_triage 2015-05-04 16:52:31 UTC
I can reproduce this issue on 9.1 install with recent ports. As far as i could see - it is kernel driver issue. In Linux it is handled by 
http://lxr.free-electrons.com/source/fs/fuse/dir.c#L499

but in FreeBSD there is no such code, it will just fail.
Comment 7 Oleksii Samorukov freebsd_committer freebsd_triage 2015-05-04 17:08:19 UTC
Also i found that kmod-fuse was backported from 10.0 source to 9.x port, that explains why it stops working on 9.1 as well. So this needs to be addressed in the kernel module, not in fs.
Comment 8 Oleksii Samorukov freebsd_committer freebsd_triage 2015-05-04 17:29:50 UTC
I created PR for the fuse kernel module, see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=199934
Comment 9 danny 2015-05-04 18:14:14 UTC
Thank you for digging deeper and filing a new PR against the kernel module, I will keep an eye on the other PR as well.
Comment 10 danny 2015-09-18 18:17:56 UTC
This bug still exists on 10.2-RELEASE.

Below are steps to repro the bug without needed a physical exfat device available.

I still get "invalid argument" when trying to create a new file via fuse-exfat.  The patch I submitted still fixes the problem fusefs-exfat, but the fix should probably be implemented lower down in the kernel mod as per PR #199934.

Example of problem on 10.2-RELEASE:

  $ uname -r
  10.2-RELEASE
  $ mkdir testdir
  $ touch testfile
  touch: testfile: Invalid argument

Bug repro steps:

  # Create empty file
  dd if=/dev/zero of=fuse-exfat-test bs=1k count=5k

  # Attach as memory disk, grab the device number as $MD
  MD=`mdconfig -f fuse-exfat-test`

  # Format as exfat
  mkexfatfs /dev/$MD

  # Mount the new filesystem via fuse-exfat
  mount.exfat-fuse /dev/$MD /mnt

  # Attempt to create a new directory and file on device
  cd /mnt
  mkdir testdir
  touch testfile
Comment 11 Adonis 2016-06-10 18:24:57 UTC
Still seeing this issue on 10.3. Essentially prevents me from creating folders and writing files to my ExFat drive. Is there no fix in the works?
Comment 12 Jan Beich freebsd_committer freebsd_triage 2016-07-04 22:39:58 UTC
Fixed upstream since v1.2.0. I plan to close this bug once bug 210823 lands.

https://github.com/relan/exfat/commit/5b42458
Comment 13 danny 2016-07-05 01:38:18 UTC
(In reply to Jan Beich from comment #12)
Thanks for keeping an eye on this, I appreciate it.  Looking forward to upstream fix being applied in the future!
Comment 14 commit-hook freebsd_committer freebsd_triage 2016-07-18 18:30:07 UTC
A commit references this bug:

Author: jbeich
Date: Mon Jul 18 18:30:00 UTC 2016
New revision: 418728
URL: https://svnweb.freebsd.org/changeset/ports/418728

Log:
  sysutils/*exfat*: update to 1.2.4

  - Chase project move to GitHub
  - Chase patent licensing URL [1]
  - Restore GPLv2 after r328060 [2]
  - Fix build on DragonFly
  - Drop unused iconv dependency

  PR:		199874, 210823
  PR:		210162 [1]
  Suggested by:	decke, tijl [2] (via bug 193629)
  Approved by:	maintainer timeout (2 weeks)
  MFH:		2016Q3

Changes:
  head/sysutils/exfat-utils/Makefile
  head/sysutils/exfat-utils/distinfo
  head/sysutils/exfat-utils/files/patch-SConstruct
  head/sysutils/exfat-utils/files/patch-libexfat_platform.h
  head/sysutils/exfat-utils/files/patch-ublio
  head/sysutils/exfat-utils/pkg-descr
  head/sysutils/fusefs-exfat/Makefile
  head/sysutils/fusefs-exfat/distinfo
  head/sysutils/fusefs-exfat/files/patch-SConstruct
  head/sysutils/fusefs-exfat/files/patch-libexfat_platform.h
  head/sysutils/fusefs-exfat/files/patch-ublio
  head/sysutils/fusefs-exfat/pkg-descr
Comment 15 commit-hook freebsd_committer freebsd_triage 2016-07-18 20:26:36 UTC
A commit references this bug:

Author: jbeich
Date: Mon Jul 18 20:26:09 UTC 2016
New revision: 418745
URL: https://svnweb.freebsd.org/changeset/ports/418745

Log:
  MFH: r418728

  sysutils/*exfat*: update to 1.2.4

  - Chase project move to GitHub
  - Chase patent licensing URL [1]
  - Restore GPLv2 after r328060 [2]
  - Fix build on DragonFly
  - Drop unused iconv dependency

  PR:		199874, 210823
  PR:		210162 [1]
  Suggested by:	decke, tijl [2] (via bug 193629)
  Approved by:	maintainer timeout (2 weeks)
  Approved by:	ports-secteam (feld)

Changes:
_U  branches/2016Q3/
  branches/2016Q3/sysutils/exfat-utils/Makefile
  branches/2016Q3/sysutils/exfat-utils/distinfo
  branches/2016Q3/sysutils/exfat-utils/files/patch-SConstruct
  branches/2016Q3/sysutils/exfat-utils/files/patch-libexfat_platform.h
  branches/2016Q3/sysutils/exfat-utils/files/patch-ublio
  branches/2016Q3/sysutils/exfat-utils/pkg-descr
  branches/2016Q3/sysutils/fusefs-exfat/Makefile
  branches/2016Q3/sysutils/fusefs-exfat/distinfo
  branches/2016Q3/sysutils/fusefs-exfat/files/patch-SConstruct
  branches/2016Q3/sysutils/fusefs-exfat/files/patch-libexfat_platform.h
  branches/2016Q3/sysutils/fusefs-exfat/files/patch-ublio
  branches/2016Q3/sysutils/fusefs-exfat/pkg-descr