When we want to use direct IO flag it's mostly working ok. But in very specific situation still uses cache omitting DIRECT FLAG. This situation appears only when we are creating file. We suspect that this problem is connected with function: fuse_vnode_open(vp, 0, td) in /usr/src/sys/fs/fuse/fuse_fnops.c file. The second parameter of this function is always set to 0 value and in some cases removing DIRECT flag set by other functions.
Are you referring to cases where the MooseFS fuse file system sets FOPEN_DIRECT_IO in the reply to a fuse open? or Are you referring to processes on FreeBSD that open with O_DIRECT? Btw, if you look at PR#194293, I have a patch there that forces DIRECT_IO for the case of WRONLY opens, so that a small buffered write doesn't get into trouble when the buffer cache code tries to read the entire block in.
Created attachment 165690 [details] patch fuse so it forces DIRECT_IO for WRONLY opens
Created attachment 165691 [details] patch fuse so that a create with DIRECT_IO opens with DIRECT_IO Please try applying both of these patches in order. The second one fixes your problem, I think? (Assuming it was a create with FOPEN_DIRECT_IO in its reply.) The first is from PR#194293 and forces DIRECT_IO when a file is WRONLY opened. Please let me know if this fixes your problem.
Thank you for your patches. We applied them and all seems to work fine now. We have a question. When it will be added to official kernel? Best regards MooseFS Support Team
A guess w.r.t. getting these patches in a release would be 6mon->1year. Now, the patches will probably make it into what FreeBSD calls head/current in April and in stable/10 in May. (This assumes that head isn't frozen for the FreeBSD11 release in April.) (It's too late for the 10.3 release and I'm not sure if/when a 10.4 release is planned.)
The reporter comfirmed that the patches fixed the problem for them. I am planning on committing these patches in April 2016.
Hi, just to remember you about this :) Peter from MooseFS Team
A commit references this bug: Author: rmacklem Date: Sat May 14 20:03:22 UTC 2016 New revision: 299753 URL: https://svnweb.freebsd.org/changeset/base/299753 Log: Fix fuse to use DIRECT_IO when required. When a file is opened write-only and a partial block was written, buffered I/O would try and read the whole block in. This would result in a hung thread, since there was no open (fuse filehandle) that allowed reading. This patch avoids the problem by forcing DIRECT_IO for this case. It also sets DIRECT_IO when the file system specifies the FN_DIRECTIO flag in its reply to the open. Tested by: nishida@asusa.net, freebsd@moosefs.com PR: 194293, 206238 MFC after: 2 weeks Changes: head/sys/fs/fuse/fuse_file.c head/sys/fs/fuse/fuse_vnops.c
The patches that fix this have been committed and MFC'd to stable/10.
Thanks! :) Peter from MooseFS Team