FreeBSD 12.1 introduced a bug that was not present in previous versions. When using a FUSE-based file system, user tries to append data to a file that is not cached. File is opened WRITE ONLY, but the first call after open is read, not write. This is not an allowed behaviour and some file systems will return EACCESS error. If the file is cached, then there is no read, only write. This bug is still present in 13.0 CURRENT (20200130).
Can you share a specific ordering of IPCs, including exact open flags used and FUSE procedures seen on the server side? For cached IO (bio) without O_DIRECT, we probably attempt to read the last page/sector of a file we are appending to. Any local filesystem would do something similar (read a block from the backing volume when a write-only append handle is open for an uncached, cacheable file), if the file is not a perfect multiple of the page size. I think we mostly try to open separate read handles for this, but it's we missed a case. Any specific details you can share would be helpful. Thanks.
This is not a bug; it's the documented behavior of the FUSE protocol. See https://github.com/libfuse/libfuse/blob/562223325e6dd2fde2f4e0077dac7e1c44e3dd18/include/fuse.h#L403 . The purpose of the open flags is access control. FUSE file systems should always be prepared to handle reads. If the file system can't, then it should not allow itself to be mounted with the writeback cache. File system daemons using protocol 7.23 or later can control whether the writeback cache is enabled. For older daemons, the writeback cache is controlled by the vfs.fusefs.data_cache_mode sysctl. Before I close this bug, can you please confirm your current cache mode? To do that, report the value of the vfs.fusefs.data_cache_mode sysctl, report the name and version of the FUSE file system you're using, and (if it uses libfuse), show the daemon's debug output, if it provides any.
Yes, the affected file system used write back cache. This is quite unexpected behaviour, nevertheless, since it's documented, it's not a bug, it's a feature ;) Thank you for pointing it out.