Bug 211674 - sysutils/fusefs-ntfs: fuse_vnode leak/reclamation failure
Summary: sysutils/fusefs-ntfs: fuse_vnode leak/reclamation failure
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: freebsd-ports-bugs (Nobody)
URL:
Keywords: needs-qa
Depends on:
Blocks:
 
Reported: 2016-08-08 17:43 UTC by Mahmoud Al-Qudsi
Modified: 2023-05-18 20:12 UTC (History)
5 users (show)

See Also:
bugzilla: maintainer-feedback? (freebsd)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Mahmoud Al-Qudsi 2016-08-08 17:43:55 UTC
SUMMARY

Running on FreeBSD 10.3-RELEASE-p6/i386 with fuse compiled into kernel and with fusefs-ntfs 2016.2.22 installed, there is a fuse_vnode leak (though it seems it may be more of a complete failure to reclaim vnodes) resulting in quick resource exhaustion.

REPRODUCTION

This is easily reproduced with the following:

ntfs-3g /dev/xxx /mnt/yyyy
cd /mnt/yyyy
find . -exec touch {} \;

In another virtual terminal:

vmstat | head -n1; vmstat -m | sed 1d | sort -hk 3,3

ACTUAL RESULTS

fuse_vnode will continuously balloon, and will not be reclaimed until the filesystem is unmounted.

(likewise, fuse_msgbuff also balloons but unlike fuse_vnode, it is never reclaimed. Separate PR?)

EXPECTED RESULTS

fuse_vnode entries should be reclaimed

ADDITIONAL INFORMATION

Here's a snapshot of the fuse-related vmstat entries after this process:

fuse_vnode 36020 9005K - 502349 256
fuse_msgbuf 58141 14895K - 311095 256,512,1024,2048,4096,8192
Comment 1 Walter Schwarzenfeld freebsd_triage 2018-01-17 10:38:09 UTC
Is this still relevant?
Comment 2 Alan Somers freebsd_committer freebsd_triage 2019-04-03 15:04:01 UTC
The fact that fuse_msgbuf is high indicates that the FUSE server isn't responding to some commands.  However, you shouldn't read anything into the fuse_vnode memory.  That doesn't get freed until VOP_RECLAIM, which won't happen unless another file system needs those vnodes.

So I think this is a bug in the port, not the kernel.  However, I will try to reproduce it if you give me some better instructions.  Being unfamiliar with fuse-ntfs3g, I need more detailed reproduction instructions.  Include which packages to install, the command for formatting a new disk, and how to fill the filesystem.
Comment 3 Gleb Popov freebsd_committer freebsd_triage 2023-01-26 17:14:01 UTC
I can look into creating a reproducer, but vmstat doesn't report fuse_vnode and fuse_msgbuf anymore. Where else may I look for these values?
Comment 4 Alan Somers freebsd_committer freebsd_triage 2023-01-26 17:32:34 UTC
(In reply to Gleb Popov from comment #3)
It should.  Are you sure the fusefs kld is loaded?

> vmstat -m | grep fuse
   fuse_vnode     3     2K       12  384
  fuse_msgbuf  3873  2074K     4147  256,384,16384,32768,65536
fuse_filefilehandle     4     1K      159  64
Comment 5 Gleb Popov freebsd_committer freebsd_triage 2023-01-26 17:36:29 UTC
(In reply to Alan Somers from comment #4)
The "vmstat -m | grep fuse" command shows nothing for me.

# kldload fusefs
kldload: can't load fusefs: module already loaded or in kernel

I'm running 14-CURRENT 443e6eccbd8f, Fri Oct 21
Comment 6 Alan Somers freebsd_committer freebsd_triage 2023-01-26 17:40:22 UTC
(In reply to Gleb Popov from comment #5)
Ok, I can duplicate that too now.  It looks like those fields won't show up until after you mount your first fuse file system.
Comment 7 Gleb Popov freebsd_committer freebsd_triage 2023-01-26 18:36:43 UTC
Ok, the bug report seems to still be relevant.

Preparation steps:

1. pkg install fusefs-ntfs.
2. Insert an USB flash disk. A memory device should probably be fine too.
3. mkntfs -f /dev/da0
4. ntfs-3g /dev/da0 /mnt
5. cp -r /usr/include /mnt
6. umount /mnt

The copying took quite a time for me, but maybe it is just the stick.

Reproduction steps:

1. kldunload fusefs
2. kldload fusefs
3. ntfs-3g /dev/da0 /mnt
4. vmstat | head -n1 ; vmstat -m | sed 1d | sort -hk 3,3 | grep fuse

procs     memory       page                      disks     faults       cpu
   fuse_vnode     1     1K        1  384
  fuse_msgbuf   265    67K      265  256,384

5. find /mnt -exec touch {} \;
6. Wait the find process to finish.
7. vmstat | head -n1 ; vmstat -m | sed 1d | sort -hk 3,3 | grep fuse

procs     memory       page                      disks     faults       cpu
fuse_filefilehandle     1     1K      327  64
  fuse_msgbuf   881   269K     1309  256,384,512,1024,2048,4096
   fuse_vnode  2763  1037K     2763  384

8. umount /mnt

Umounting takes considerable amount of time.

9. vmstat | head -n1 ; vmstat -m | sed 1d | sort -hk 3,3 | grep fuse

procs     memory       page                      disks     faults       cpu
   fuse_vnode     0     0K     2763  384
fuse_filefilehandle     0     0K      327  64
  fuse_msgbuf  3256   863K     3685  256,384,512,1024,2048,4096


Hope this helps.
Comment 8 Gleb Popov freebsd_committer freebsd_triage 2023-02-10 19:27:11 UTC
Alan, did you have a chance to try out my reproduction steps?
Comment 9 Alan Somers freebsd_committer freebsd_triage 2023-02-10 20:11:22 UTC
(In reply to Gleb Popov from comment #8)
No, I haven't yet.  I might be able to do it this weekend.  But it sounds like all of the vnodes get released at unmount, right?  I think that's expected behavior.
Comment 10 Gleb Popov freebsd_committer freebsd_triage 2023-02-11 07:47:18 UTC
(In reply to Alan Somers from comment #9)
Something feels of still, as the unmount operation takes insane amount of time. It is probably busy garbage collecting some stuff. Also, the fuse_msgbuf value doesn't go down even after unmounting, although I'm not sure if it is bad or not.
Comment 11 Alan Somers freebsd_committer freebsd_triage 2023-02-11 23:42:50 UTC
Ok, here are the results of my investigations:
1) It's not a problem that the vnode number keeps rising.  The kernel doesn't reclaim vnodes as soon as all their file descriptors get closed.  Rather, the vnodes remain allocated just in case the same file system decides it needs to use some more.  The kernel won't reclaim vnodes until unmount or until it starts to run low on vnodes, possibly due to activity on some other file system.

2) It's mostly not a problem that the fuse_msgbuf value keeps rising (but see caveat below).  Similarly to the vnode situation, uma does not free items as soon as they are no longer needed.  But it will free them in response to memory pressure.  See uma(9).

3) There are some actual resource leaks, which can be observed by the value of "sysctl vfs.fusefs.stats.ticket_count".  Those are probably responsible for some of the fuse_msgbuf memory consumption, but probably a minority.  I'll work on fixing these.  The first one I found is in an error path, triggerable by a malicious server.

4) You probably aren't the only person to notice slow unmounts.  That's probably why the FUSE protocol describes a BATCH_FORGET operation.  It isn't implemented in FreeBSD, but probably could be.  I don't know if fuse-ntfs is able to take advantage.
Comment 12 Gleb Popov freebsd_committer freebsd_triage 2023-05-18 06:09:38 UTC
Alan, did you make some progress on this? You said you've found some minor problems in your previous comment.
Comment 13 Alan Somers freebsd_committer freebsd_triage 2023-05-18 20:12:06 UTC
Yes, I fixed the ticket leaks in https://github.com/freebsd/freebsd-src/commit/1bdf879b97b686a8f36fdba050b68f9e7493f363 .  As for BATCH_FORGET, I haven't done anything about that one.  But grepping its codebase, I see that fusefs-ntfs doesn't support it either, so that won't help you.