Bug 206238 - FUSE can't enter to DIRECT_IO mode during file create.
Summary: FUSE can't enter to DIRECT_IO mode during file create.
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 10.2-STABLE
Hardware: amd64 Any
: --- Affects Some People
Assignee: Rick Macklem
Depends on:
Reported: 2016-01-14 14:59 UTC by MooseFS FreeBSD Team
Modified: 2016-05-29 12:20 UTC (History)
4 users (show)

See Also:

patch fuse so it forces DIRECT_IO for WRONLY opens (1.76 KB, patch)
2016-01-16 23:22 UTC, rmacklem
no flags Details | Diff
patch fuse so that a create with DIRECT_IO opens with DIRECT_IO (439 bytes, patch)
2016-01-16 23:24 UTC, rmacklem
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description MooseFS FreeBSD Team 2016-01-14 14:59:02 UTC
When we want to use direct IO flag it's mostly working ok.
But in very specific situation still uses cache omitting DIRECT FLAG.

This situation appears only when we are creating file.
We suspect that this problem is connected with function:

fuse_vnode_open(vp, 0, td)

in /usr/src/sys/fs/fuse/fuse_fnops.c file.
The second parameter of this function is always set to 0 value and in some cases removing DIRECT flag set by other functions.
Comment 1 rmacklem 2016-01-15 04:05:01 UTC
Are you referring to cases where the MooseFS fuse file system
sets FOPEN_DIRECT_IO in the reply to a fuse open?
Are you referring to processes on FreeBSD that open with O_DIRECT?

Btw, if you look at PR#194293, I have a patch there that forces
DIRECT_IO for the case of WRONLY opens, so that a small buffered
write doesn't get into trouble when the buffer cache code tries
to read the entire block in.
Comment 2 rmacklem 2016-01-16 23:22:02 UTC
Created attachment 165690 [details]
patch fuse so it forces DIRECT_IO for WRONLY opens
Comment 3 rmacklem 2016-01-16 23:24:50 UTC
Created attachment 165691 [details]
patch fuse so that a create with DIRECT_IO opens with DIRECT_IO

Please try applying both of these patches in order.
The second one fixes your problem, I think?
(Assuming it was a create with FOPEN_DIRECT_IO in
 its reply.)
The first is from PR#194293 and forces DIRECT_IO when
a file is WRONLY opened.

Please let me know if this fixes your problem.
Comment 4 MooseFS FreeBSD Team 2016-01-19 07:49:53 UTC
Thank you for your patches.
We applied them and all seems to work fine now.

We have a question.
When it will be added to official kernel?

Best regards
MooseFS Support Team
Comment 5 rmacklem 2016-01-20 04:50:17 UTC
A guess w.r.t. getting these patches in a release would
be 6mon->1year. Now, the patches will probably make it into
what FreeBSD calls head/current in April and in stable/10
in May. (This assumes that head isn't frozen for the FreeBSD11
release in April.)
(It's too late for the 10.3 release and I'm not sure if/when
 a 10.4 release is planned.)
Comment 6 Rick Macklem freebsd_committer 2016-02-05 01:09:20 UTC
The reporter comfirmed that the patches fixed the problem
for them. I am planning on committing these patches in April 2016.
Comment 7 MooseFS FreeBSD Team 2016-04-28 16:14:59 UTC

just to remember you about this :)

Peter from MooseFS Team
Comment 8 commit-hook freebsd_committer 2016-05-14 20:04:11 UTC
A commit references this bug:

Author: rmacklem
Date: Sat May 14 20:03:22 UTC 2016
New revision: 299753
URL: https://svnweb.freebsd.org/changeset/base/299753

  Fix fuse to use DIRECT_IO when required.

  When a file is opened write-only and a partial block was written,
  buffered I/O would try and read the whole block in. This would
  result in a hung thread, since there was no open (fuse filehandle)
  that allowed reading. This patch avoids the problem by forcing
  DIRECT_IO for this case.
  It also sets DIRECT_IO when the file system specifies the FN_DIRECTIO
  flag in its reply to the open.

  Tested by:	nishida@asusa.net, freebsd@moosefs.com
  PR:		194293, 206238
  MFC after:	2 weeks

Comment 9 Rick Macklem freebsd_committer 2016-05-28 22:48:35 UTC
The patches that fix this have been committed and MFC'd to stable/10.
Comment 10 MooseFS FreeBSD Team 2016-05-29 12:20:16 UTC
Thanks! :)

Peter from MooseFS Team