|Summary:||[fusefs] returns cached truncated data|
|Product:||Base System||Reporter:||Alan Somers <asomers>|
|Component:||kern||Assignee:||Alan Somers <asomers>|
|Status:||In Progress ---|
|Severity:||Affects Many People||CC:||cem, emaste, fsu, sigsys|
Description Alan Somers 2018-12-04 17:50:45 UTC
fuse(4) caches data in the kernel, even when mounted with "-o direct_io". After a truncation, it may return cached data from past the truncation point. This data is invalid and should've been dropped during the truncation. I've reproduced this behavior with libfuse's "passthrough" example program using both CURRENT @r340987 and 12.0-BETA4. Steps to reproduce: $ cd /path/to/freebsd/sources/tools/regression/fsx $ make $ sudo pkg install pkgconf fusefs-libs3 $ git clone firstname.lastname@example.org:libfuse/libfuse.git $ cd libfuse/example $ cc -Wall -I/usr/local/include/fuse3 -L/usr/local/lib -lfuse3 -lpthread passthrough.c -o passthrough $ sudo kldload fuse $ mkdir /tmp/mnt $ sudo ./passthrough -f -sd -o allow_other /tmp/mnt Then, run fsx in a separate session: $ cd /tmp/mnt/tmp $ /usr/obj/whatever/amd64.amd64/tools/regression/fsx/fsx -WR -P /tmp -S10 fsx.bin fsx will terminate with this output. It's reproducible since we specified the seed: mapped writes DISABLED Seed set to 10 skipping zero size read skipping zero size read skipping zero size read skipping zero size read skipping zero size read skipping zero size read skipping zero size read truncating to largest ever: 0x10016 READ BAD DATA: offset = 0xe1e6, size = 0xe229 OFFSET GOOD BAD RANGE 0x1b8df 0x0000 0xb708 0x b2b operation# (mod 256) for the bad data may be 8 LOG DUMP (11 total operations): 1(1 mod 256): SKIPPED (no operation) 2(2 mod 256): SKIPPED (no operation) 3(3 mod 256): SKIPPED (no operation) 4(4 mod 256): SKIPPED (no operation) 5(5 mod 256): SKIPPED (no operation) 6(6 mod 256): SKIPPED (no operation) 7(7 mod 256): SKIPPED (no operation) 8(8 mod 256): WRITE 0x1b8df thru 0x21ac6 (0x61e8 bytes) HOLE ***WWWW 9(9 mod 256): TRUNCATE DOWN from 0x21ac7 to 0x10016 ******WWWW 10(10 mod 256): WRITE 0x3e90a thru 0x3ffff (0x16f6 bytes) HOLE ***WWWW 11(11 mod 256): READ 0xe1e6 thru 0x1c40e (0xe229 bytes) ***RRRR*** Correct content saved for comparison (maybe hexdump "fsx.bin" vs "fsx.bin.fsxgood") By comparing FSX's output to passthrough's, we can deduce where fuse.ko serviced a read from its cache: fsx's log passthrough's log --------- ----------------- write 0x1b8df 0x61e8 write 112863 18209 write 131072 6855 trunc 137927 trunc 0x10016 trunc 65558 write 0x3d90a 0x16f6 read 196608 65536 write 256266 5878 trunc 262144 read 0xe1e6 0x1c40e read 0 65536 (services 57830 - 65536) <read from cache> (services 65536 - 173556) The miscompare happened at address 0x1b8df, which was in the part of the read serviced from the cache, and it's the first part of that cached read that ever contained non-zero data in the lifetime of this test.
Comment 2 Conrad Meyer 2018-12-04 18:24:37 UTC
Everything about our fuse driver is kinda crappy. The attribute caching isn't correct, the data caching isn't correct, yada yada. I'm not sure if we're supposed to be caching data at all or if it's just a (broken) performance optimization.
Comment 3 Alan Somers 2019-04-02 19:35:45 UTC
This bug is still reproducible on 13, though it now requires "sysctl vfs.fusefs.data_cache_mode=2". Presumably that's due to cem's change r344186.
Comment 4 commit-hook 2019-04-03 02:30:15 UTC
A commit references this bug: Author: asomers Date: Wed Apr 3 02:29:56 UTC 2019 New revision: 345823 URL: https://svnweb.freebsd.org/changeset/base/345823 Log: fusefs: during ftruncate, discard cached data past truncation point During truncate, fusefs was discarding entire cached blocks, but it wasn't zeroing out the unused portion of a final partial block. This resulted in reads returning stale data. PR: 233783 Reported by: fsx Sponsored by: The FreeBSD Foundation Changes: projects/fuse2/sys/fs/fuse/fuse_node.c projects/fuse2/tests/sys/fs/fusefs/setattr.cc
Comment 5 Alan Somers 2019-04-19 22:32:50 UTC
This is complete on the fuse2 branch.
Comment 6 commit-hook 2019-04-23 22:26:33 UTC
A commit references this bug: Author: asomers Date: Tue Apr 23 22:25:50 UTC 2019 New revision: 346607 URL: https://svnweb.freebsd.org/changeset/base/346607 Log: fusefs: use vfs_bio_clrbuf in fuse_vnode_setsize Reuse fuse_vnode_setsize instead of reinventing the wheel. This is what ext2_ind_truncate does. PR: 233783 Sponsored by: The FreeBSD Foundation Changes: projects/fuse2/sys/fs/fuse/fuse_node.c
Comment 7 commit-hook 2019-08-07 00:39:39 UTC
A commit references this bug: Author: asomers Date: Wed Aug 7 00:38:28 UTC 2019 New revision: 350665 URL: https://svnweb.freebsd.org/changeset/base/350665 Log: fusefs: merge from projects/fuse2 This commit imports the new fusefs driver. It raises the protocol level from 7.8 to 7.23, fixes many bugs, adds a test suite for the driver, and adds many new features. New features include: * Optional kernel-side permissions checks (-o default_permissions) * Implement VOP_MKNOD, VOP_BMAP, and VOP_ADVLOCK * Allow interrupting FUSE operations * Support named pipes and unix-domain sockets in fusefs file systems * Forward UTIME_NOW during utimensat(2) to the daemon * kqueue support for /dev/fuse * Allow updating mounts with "mount -u" * Allow exporting fusefs file systems over NFS * Server-initiated invalidation of the name cache or data cache * Respect RLIMIT_FSIZE * Try to support servers as old as protocol 7.4 Performance enhancements include: * Implement FUSE's FOPEN_KEEP_CACHE and FUSE_ASYNC_READ flags * Cache file attributes * Cache lookup entries, both positive and negative * Server-selectable cache modes: writethrough, writeback, or uncached * Write clustering * Readahead * Use counter(9) for statistical reporting PR: 199934 216391 233783 234581 235773 235774 235775 PR: 236226 236231 236236 236291 236329 236381 236405 PR: 236327 236466 236472 236473 236474 236530 236557 PR: 236560 236844 237052 237181 237588 238565 Reviewed by: bcr (man pages) Reviewed by: cem, ngie, rpokala, glebius, kib, bde, emaste (post-commit review on project branch) MFC after: 3 weeks Relnotes: yes Sponsored by: The FreeBSD Foundation Pull Request: https://reviews.freebsd.org/D21110 Changes: _U head/ head/MAINTAINERS head/UPDATING head/etc/mtree/BSD.tests.dist head/sbin/mount_fusefs/mount_fusefs.8 head/sbin/mount_fusefs/mount_fusefs.c head/share/man/man5/fusefs.5 head/sys/fs/fuse/fuse.h head/sys/fs/fuse/fuse_device.c head/sys/fs/fuse/fuse_file.c head/sys/fs/fuse/fuse_file.h head/sys/fs/fuse/fuse_internal.c head/sys/fs/fuse/fuse_internal.h head/sys/fs/fuse/fuse_io.c head/sys/fs/fuse/fuse_io.h head/sys/fs/fuse/fuse_ipc.c head/sys/fs/fuse/fuse_ipc.h head/sys/fs/fuse/fuse_kernel.h head/sys/fs/fuse/fuse_main.c head/sys/fs/fuse/fuse_node.c head/sys/fs/fuse/fuse_node.h head/sys/fs/fuse/fuse_param.h head/sys/fs/fuse/fuse_vfsops.c head/sys/fs/fuse/fuse_vnops.c head/sys/sys/param.h head/tests/sys/fs/Makefile head/tests/sys/fs/fusefs/