Bug 230726 - sysutils/kf5-baloo: OS performance suffers – sometimes, as if frozen – when reaching the virtual memory vnode limit
Summary: sysutils/kf5-baloo: OS performance suffers – sometimes, as if frozen – when r...
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: Po-Chuan Hsieh
URL: https://www.freshports.org/sysutils/k...
Keywords: needs-qa, performance
Depends on: 256269
Blocks:
  Show dependency treegraph
 
Reported: 2018-08-18 04:42 UTC by Ting-Wei Lan
Modified: 2022-10-23 02:51 UTC (History)
11 users (show)

See Also:
tcberner: maintainer-feedback+


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Ting-Wei Lan 2018-08-18 04:42:08 UTC
After upgrading from x11/kde4 to x11/kde5, I found there is a process called 'baloo_file' which seems to use a lot of system resources. If I forget to kill it within minutes after starting a GNOME session, the entire system will freeze. Yes, installing KDE can affect GNOME, as /usr/local/etc/xdg/autostart/baloo_file.desktop has X-GNOME-Autostart-enabled=true. I can change it to false to avoid the problem in GNOME, but it sounds like a workaround.

When baloo_file freezes the system, I can still move the mouse but no desktop applications are responding. SSH server becomes almost non-responsive and it takes several minutes just to get a login shell. Typing any simple commands such as 'killall' takes minutes to complete, and even logging in from a local VT is hard. When you type your user name, it takes several minutes to show the password prompt. If you press Ctrl-T at that time, you will see the following message.

load: 0.47  cmd: login 1141 [vlruwk] 204838.88r 0.00u 0.00s 0% 240

Most of the time it is stuck at [vlruwk] or [ufs] while the system load is low. I am not sure whether it is related to the number of files under my home directory. My home directory has about 2500000 files because I store hundreds of git repositories, mostly GNOME source code, and builds of GNOME and FreeBSD in it.
Comment 1 Tobias C. Berner freebsd_committer 2018-12-26 11:07:12 UTC
You can disable baloo via "balooctl disable". Or you can configure it to not index certain directories.

It would be interesting to know how many files baloo_files has open, and how many are free / available by the system.
Comment 2 Edward Tomasz Napierala freebsd_committer 2019-01-17 11:39:59 UTC
I suspect this might be related to the fact that SDDM doesn't apply resource limits.  There's a patch for this: https://reviews.freebsd.org/D17297
Comment 3 Martin Birgmeier 2019-01-20 10:27:38 UTC
Could this be related to bug #234830?
Comment 4 Tobias C. Berner freebsd_committer 2019-01-20 10:29:22 UTC
(In reply to Martin Birgmeier from comment #3)
I don't think so. The file in this one just seems to hang libtiff.
Comment 5 Edward Tomasz Napierala freebsd_committer 2019-02-20 17:49:34 UTC
I've just realized that even with SDDM patched to apply resource limits, there is no default limit for "openfiles".

Would it be possible to modify libinotify to limit itself to eg half of kern.maxvnodes?
Comment 6 Tobias C. Berner freebsd_committer 2020-01-19 07:53:24 UTC
(In reply to Edward Tomasz Napierala from comment #5)
Probably yes -- over to the maintainer of libinotify.
Comment 7 Graham Perrin freebsd_committer 2021-07-17 09:28:41 UTC
From bug 257227 comment 1 (emphasis more on baloo_file_extractor than on baloo_file): 

> … I should, ideally, positively identify the cause of 
> hogging in my case, but this will be a bug for another day.

(In reply to Tobias C. Berner from comment #1)

> … It would be interesting to know how many files baloo_files 
> has open, and how many are free / available by the system.

Reading the above _loosely_ alongside bug 234830 (closed, fixed) …

If it's true that some types of file are more likely to trigger high CPU usage, when handled by baloo_file or baloo_file_extractor, will it help to identify those types?
Comment 8 Graham Perrin freebsd_committer 2021-07-17 09:45:17 UTC
<https://invent.kde.org/frameworks/baloo/-/tree/master#user-content-file-indexing-plugins>: 

> Baloo relies on KFileMetaData to extract content from the files. 
> KFileMetadata ships with a number of plugins which can be enabled 
> or disabled. We recommend shipping all KFileMetaData plugins. Specially 
> ffmpeg by default. Without the indexers, Baloo cannot function to its 
> full potential.

% pkg info --list devel/kf5-kfilemetadata | grep plugins
        /usr/local/lib/qt5/plugins/kf5/kfilemetadata/kfilemetadata_epubextractor.so
        /usr/local/lib/qt5/plugins/kf5/kfilemetadata/kfilemetadata_exiv2extractor.so
        /usr/local/lib/qt5/plugins/kf5/kfilemetadata/kfilemetadata_ffmpegextractor.so
        /usr/local/lib/qt5/plugins/kf5/kfilemetadata/kfilemetadata_odfextractor.so
        /usr/local/lib/qt5/plugins/kf5/kfilemetadata/kfilemetadata_office2007extractor.so
        /usr/local/lib/qt5/plugins/kf5/kfilemetadata/kfilemetadata_officeextractor.so
        /usr/local/lib/qt5/plugins/kf5/kfilemetadata/kfilemetadata_plaintextextractor.so
        /usr/local/lib/qt5/plugins/kf5/kfilemetadata/kfilemetadata_poextractor.so
        /usr/local/lib/qt5/plugins/kf5/kfilemetadata/kfilemetadata_popplerextractor.so
        /usr/local/lib/qt5/plugins/kf5/kfilemetadata/kfilemetadata_postscriptdscextractor.so
        /usr/local/lib/qt5/plugins/kf5/kfilemetadata/kfilemetadata_taglibextractor.so
        /usr/local/lib/qt5/plugins/kf5/kfilemetadata/kfilemetadata_xmlextractor.so
        /usr/local/lib/qt5/plugins/kf5/kfilemetadata/writers/kfilemetadata_taglibwriter.so
% uname -KrU
14.0-CURRENT 1400025 1400025
% pkg info -x metadata
kf5-kfilemetadata-5.83.0
py38-importlib-metadata-3.3.0_1
%
Comment 9 Karel Gardas 2021-11-03 19:52:40 UTC
(In reply to Tobias C. Berner from comment #1)
balooctl disable -- does not seems to work on FreeBSD 13-p4 amd64. At least I issued that command, output claims baloo is disabled, but then when startx launches kde5 baloo is again back in business on freezing my machine.
Comment 10 Karel Gardas 2021-11-03 19:55:13 UTC
(In reply to Graham Perrin from comment #7)
I've just started to get baloo releted freezez recently. This is on freebsd 13-p4 on amd64. I keep system patched and does not remember any change other than
(1) populating home directory with several qemu soutre trees
(2) putting 2 qemu raw disk images into the home directory.
I guess (2) here is probably the culprit. Both images do have 22GB each. Both are in raw format with .raw extension.
Comment 11 Karel Gardas 2021-11-03 20:05:23 UTC
(In reply to Karel Gardas from comment #10)
Looks like I've been mistaken. I've removed baloo core and whole src directory content from home and viola kde5 started to behave well. I've even make sure balooctl is enabled and restarted and again no problem.
So what seems to do problem in my case was either:
- 256GB of baloo core file
or
- src content which was configured and *compiled* trees of:
  (i) llvm-12.0.1 whole project tree
  (ii) qemu 5.1.0 tree
  (iii) qemu 5.2.0 tree
  (iv) qemu 6.1.0 tree

The file system in my case is ufs with default config (journaled and soft-updates).
Comment 12 Karel Gardas 2021-11-05 10:22:18 UTC
(In reply to Karel Gardas from comment #11)
Found some time, made 7 subdirs in src and unpacked into each llvm-project-12.0.1.src.tar.xz and baloo is freezing my box again.
Although box is freezing and kde is unresponsive, before whole unpacking I've started few shells with `iostat 1` and top and I can see that box is >90% idle and there is nearly no I/O traffic. So where freezing is happening? Any advice how to debug that? Any d-trace hints? Thanks!
Comment 13 Karel Gardas 2021-11-05 12:42:24 UTC
Also in my setup with 7 llvm 12 project trees. It looks like baloo is not the main cuprit in freeze as it finishes its business, or claims to so:

$ balooctl status
Baloo File Indexer is not running
Total files indexed: 228,151
Files waiting for content indexing: 0
Files failed to index: 3
Current size of index is 1.24 GiB

and yet, machine is basically neraly unusable on command-line (ssh) and completely unusable in KDE.
Comment 14 Mateusz Guzik freebsd_committer 2021-12-29 19:21:40 UTC
The system is not freezing, it runs into the vnode limit and there obtaining new vnodes is rate limited to 1 per second, which is arguably rather buggy and should be fixed.

In the meantime you can bump sysctl kern.maxvnodes