Sometimes it is necessary to share the same directory between jails and in this case there is a issue with free up space after deleting a files. For example, the file created in jail A and then deleted in jail B will not free up space as long as jail A is running. # cat /etc/fstab.test1 /storage/data /usr/jail/test1/storage/data nullfs rw,noatime 0 0 # cat /etc/fstab.test2 /storage/data /usr/jail/test2/storage/data nullfs rw,noatime 0 0 # cat /etc/jail.conf path = "/usr/jail/${name}"; exec.clean; exec.start = "sh /etc/rc"; exec.stop = "sh /etc/rc.shutdown jail"; host.hostname = "${name}"; mount.fstab = "/etc/fstab.${name}"; mount.devfs; test1 { } test2 { } # df -ht ufs Filesystem Size Used Avail Capacity Mounted on /dev/da0p2 8.9G 3.2G 5.0G 39% / # service jail start Starting jails: test1 test2. # df -ht ufs Filesystem Size Used Avail Capacity Mounted on /dev/da0p2 8.9G 3.2G 5.0G 39% / # jexec test1 dd if=/dev/random of=/storage/data/random.dd bs=1M count=1024 1024+0 records in 1024+0 records out 1073741824 bytes transferred in 3.723429 secs (288374490 bytes/sec) # df -ht ufs Filesystem Size Used Avail Capacity Mounted on /dev/da0p2 8.9G 4.2G 4.0G 52% / # jexec test2 rm /storage/data/random.dd # df -ht ufs Filesystem Size Used Avail Capacity Mounted on /dev/da0p2 8.9G 4.2G 4.0G 52% / # service jail stop test2 Stopping jails: test2. # df -ht ufs Filesystem Size Used Avail Capacity Mounted on /dev/da0p2 8.9G 4.2G 4.0G 52% / # service jail stop test1 Stopping jails: test1. # df -ht ufs Filesystem Size Used Avail Capacity Mounted on /dev/da0p2 8.9G 3.2G 5.0G 39% /
Let's loop in fs as well, since this is at the intersection of jails and filesystems.
Add "nocache" option to both nullfs mounts.
(In reply to Konstantin Belousov from comment #2) It is not documented and mount -t nullfs does not indicate that the option is used but it work. One of the jails is working as file server and there is a lot of read operations. Should I worry about hard disk head contention?
this seems to be related https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=245688
and this https://marc.info/?l=freebsd-fs&m=150581490210158&w=2 Konstantin Belousov, according to the newsletter, this problem has been going on since at least 2017?
*** Bug 270546 has been marked as a duplicate of this bug. ***
FreeBSD 13.2: The same thing happens when I use a program to optimise images (the images size shrinks but ZFS shows more usage than before). I have to restart jails to get the correct usage.
^Triage: is this still a problem on supported versions of FreeBSD?
Yes, on 14.1-RELEASE at least.
Hi, I can confirm this is still happening on a recent 14.2-STABLE. Regards,
The jails are not needed. It looks purely nullfs. # zfs create zroot/data/pr-test # cd /data/pr-test # mkdir storage test1 test2 # mount_nullfs -o rw,noatime ./storage ./test1 # mount_nullfs -o rw,noatime ./storage ./test2 # df -h | grep "pr-test" zroot/data/pr-test 327G 112K 327G 0% /data/pr-test /data/pr-test/storage 327G 112K 327G 0% /data/pr-test/test1 /data/pr-test/storage 327G 112K 327G 0% /data/pr-test/test2 # dd if=/dev/random of=./test1/random.dd bs=1M count=1024 # df -h | grep "pr-test" zroot/data/pr-test 327G 1.0G 326G 0% /data/pr-test /data/pr-test/storage 327G 1.0G 326G 0% /data/pr-test/test1 /data/pr-test/storage 327G 1.0G 326G 0% /data/pr-test/test2 # rm ./test2/random.dd # df -h | grep "pr-test" zroot/data/pr-test 327G 1.0G 326G 0% /data/pr-test /data/pr-test/storage 327G 1.0G 326G 0% /data/pr-test/test1 /data/pr-test/storage 327G 1.0G 326G 0% /data/pr-test/test2 Unmount of test1 releases the blocks. NB: mount_nullfs option nocache does not help for me.
Try https://reviews.freebsd.org/D48825 (I did not checked it)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=1a0cb938f7b461edc40316cb44a80438df674174 commit 1a0cb938f7b461edc40316cb44a80438df674174 Author: Konstantin Belousov <kib@FreeBSD.org> AuthorDate: 2025-02-04 00:23:05 +0000 Commit: Konstantin Belousov <kib@FreeBSD.org> CommitDate: 2025-02-04 22:19:50 +0000 unlink, rmdir: call notify_upper from VOP pre method instead of syscall Suppose that there are two or more nullfs mounts over some fs, and suppose the we unlink a file on one of the nullfs mount. This way notify_upper get called from the lower vnode as well, allowing the other nullfs mounts to note that and drop their caches for the unlinked vnode. PR: 254210 Reviewed by: olce Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D48825 sys/kern/vfs_subr.c | 2 ++ sys/kern/vfs_syscalls.c | 2 -- 2 files changed, 2 insertions(+), 2 deletions(-)
With the commit of comment #13 installed I can't reproduce the test case of comment #11. # uname -a FreeBSD rpi4 15.0-CURRENT FreeBSD 15.0-CURRENT #20 main-123911af5e63-dirty: Sat Feb 8 13:51:07 CET 2025 ronald@rpi4:/data/ronald/freebsd/obj/data/ronald/freebsd/src/main/arm64.aarch64/sys/GENERIC-NODEBUG arm64 👍
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=cba91bf59c79eadc3b60b83fe2185116496f08fb commit cba91bf59c79eadc3b60b83fe2185116496f08fb Author: Konstantin Belousov <kib@FreeBSD.org> AuthorDate: 2025-02-04 00:23:05 +0000 Commit: Konstantin Belousov <kib@FreeBSD.org> CommitDate: 2025-02-11 01:11:49 +0000 unlink, rmdir: call notify_upper from VOP pre method instead of syscall PR: 254210 (cherry picked from commit 1a0cb938f7b461edc40316cb44a80438df674174) sys/kern/vfs_subr.c | 2 ++ sys/kern/vfs_syscalls.c | 2 -- 2 files changed, 2 insertions(+), 2 deletions(-)
Any chance to get this into the next errata patches?
Hi, Unfortunately, we are still facing this issue in the FreeBSD infra. Two jails are involved, gitmir and cgit, plus ZFS. We are using 14-STABLE with this specific update. The gitmir dataset usually consumes only about 5GB. $ uname -a FreeBSD xxx.yyy.freebsd.org 14.2-STABLE FreeBSD 14.2-STABLE #0 stable/14-n270391-cba91bf59c79: Tue Feb 11 01:59:07 UTC 2025 root@build-14.freebsd.org:/usr/obj/usr/src/sys/CLUSTER14 amd64 $ cat /etc/fstab.cgit /j/jails/gitmir/home/git/ /j/jails/cgit/home/git/ nullfs ro 0 0 zroot/j/jails/gitmir 440G 296G 440G /j/jails/gitmir # /etc/rc.d/jail restart cgit zroot/j/jails/gitmir 5.51G 731G 5.51G /j/jails/gitmir
Hi, I'm reopening this and trying to find a way to reproduce it, but it's very weird. Sometimes it happens, and sometimes it doesn't. We have this on our git/cgit mirrors: % cat /etc/fstab.cgit /j/jails/gitmir/home/git/ /j/jails/cgit/home/git/ nullfs ro,nocache 0 0 I've tried both with nocache and without it. So, the second jail (cgit) only reads the files and starts several fcgiwrap processes. Stopping all the processes in the jail doesn't solve the problem; I need to restart the second jail to free up space. The primary jail is constantly writing and changing things. It's basically running a git pull/fetch repeatedly (10-second interval) for the src/doc/ports repositories.
ah, and the servers are currently running cba91bf59c79