SUMMARY Running on FreeBSD 10.3-RELEASE-p6/i386 with fuse compiled into kernel and with fusefs-ntfs 2016.2.22 installed, there is a fuse_vnode leak (though it seems it may be more of a complete failure to reclaim vnodes) resulting in quick resource exhaustion. REPRODUCTION This is easily reproduced with the following: ntfs-3g /dev/xxx /mnt/yyyy cd /mnt/yyyy find . -exec touch {} \; In another virtual terminal: vmstat | head -n1; vmstat -m | sed 1d | sort -hk 3,3 ACTUAL RESULTS fuse_vnode will continuously balloon, and will not be reclaimed until the filesystem is unmounted. (likewise, fuse_msgbuff also balloons but unlike fuse_vnode, it is never reclaimed. Separate PR?) EXPECTED RESULTS fuse_vnode entries should be reclaimed ADDITIONAL INFORMATION Here's a snapshot of the fuse-related vmstat entries after this process: fuse_vnode 36020 9005K - 502349 256 fuse_msgbuf 58141 14895K - 311095 256,512,1024,2048,4096,8192
Is this still relevant?
The fact that fuse_msgbuf is high indicates that the FUSE server isn't responding to some commands. However, you shouldn't read anything into the fuse_vnode memory. That doesn't get freed until VOP_RECLAIM, which won't happen unless another file system needs those vnodes. So I think this is a bug in the port, not the kernel. However, I will try to reproduce it if you give me some better instructions. Being unfamiliar with fuse-ntfs3g, I need more detailed reproduction instructions. Include which packages to install, the command for formatting a new disk, and how to fill the filesystem.
I can look into creating a reproducer, but vmstat doesn't report fuse_vnode and fuse_msgbuf anymore. Where else may I look for these values?
(In reply to Gleb Popov from comment #3) It should. Are you sure the fusefs kld is loaded? > vmstat -m | grep fuse fuse_vnode 3 2K 12 384 fuse_msgbuf 3873 2074K 4147 256,384,16384,32768,65536 fuse_filefilehandle 4 1K 159 64
(In reply to Alan Somers from comment #4) The "vmstat -m | grep fuse" command shows nothing for me. # kldload fusefs kldload: can't load fusefs: module already loaded or in kernel I'm running 14-CURRENT 443e6eccbd8f, Fri Oct 21
(In reply to Gleb Popov from comment #5) Ok, I can duplicate that too now. It looks like those fields won't show up until after you mount your first fuse file system.
Ok, the bug report seems to still be relevant. Preparation steps: 1. pkg install fusefs-ntfs. 2. Insert an USB flash disk. A memory device should probably be fine too. 3. mkntfs -f /dev/da0 4. ntfs-3g /dev/da0 /mnt 5. cp -r /usr/include /mnt 6. umount /mnt The copying took quite a time for me, but maybe it is just the stick. Reproduction steps: 1. kldunload fusefs 2. kldload fusefs 3. ntfs-3g /dev/da0 /mnt 4. vmstat | head -n1 ; vmstat -m | sed 1d | sort -hk 3,3 | grep fuse procs memory page disks faults cpu fuse_vnode 1 1K 1 384 fuse_msgbuf 265 67K 265 256,384 5. find /mnt -exec touch {} \; 6. Wait the find process to finish. 7. vmstat | head -n1 ; vmstat -m | sed 1d | sort -hk 3,3 | grep fuse procs memory page disks faults cpu fuse_filefilehandle 1 1K 327 64 fuse_msgbuf 881 269K 1309 256,384,512,1024,2048,4096 fuse_vnode 2763 1037K 2763 384 8. umount /mnt Umounting takes considerable amount of time. 9. vmstat | head -n1 ; vmstat -m | sed 1d | sort -hk 3,3 | grep fuse procs memory page disks faults cpu fuse_vnode 0 0K 2763 384 fuse_filefilehandle 0 0K 327 64 fuse_msgbuf 3256 863K 3685 256,384,512,1024,2048,4096 Hope this helps.
Alan, did you have a chance to try out my reproduction steps?
(In reply to Gleb Popov from comment #8) No, I haven't yet. I might be able to do it this weekend. But it sounds like all of the vnodes get released at unmount, right? I think that's expected behavior.
(In reply to Alan Somers from comment #9) Something feels of still, as the unmount operation takes insane amount of time. It is probably busy garbage collecting some stuff. Also, the fuse_msgbuf value doesn't go down even after unmounting, although I'm not sure if it is bad or not.
Ok, here are the results of my investigations: 1) It's not a problem that the vnode number keeps rising. The kernel doesn't reclaim vnodes as soon as all their file descriptors get closed. Rather, the vnodes remain allocated just in case the same file system decides it needs to use some more. The kernel won't reclaim vnodes until unmount or until it starts to run low on vnodes, possibly due to activity on some other file system. 2) It's mostly not a problem that the fuse_msgbuf value keeps rising (but see caveat below). Similarly to the vnode situation, uma does not free items as soon as they are no longer needed. But it will free them in response to memory pressure. See uma(9). 3) There are some actual resource leaks, which can be observed by the value of "sysctl vfs.fusefs.stats.ticket_count". Those are probably responsible for some of the fuse_msgbuf memory consumption, but probably a minority. I'll work on fixing these. The first one I found is in an error path, triggerable by a malicious server. 4) You probably aren't the only person to notice slow unmounts. That's probably why the FUSE protocol describes a BATCH_FORGET operation. It isn't implemented in FreeBSD, but probably could be. I don't know if fuse-ntfs is able to take advantage.
Alan, did you make some progress on this? You said you've found some minor problems in your previous comment.
Yes, I fixed the ticket leaks in https://github.com/freebsd/freebsd-src/commit/1bdf879b97b686a8f36fdba050b68f9e7493f363 . As for BATCH_FORGET, I haven't done anything about that one. But grepping its codebase, I see that fusefs-ntfs doesn't support it either, so that won't help you.