Sometimes it is necessary to share the same directory between jails and in this case there is a issue with free up space after deleting a files. For example, the file created in jail A and then deleted in jail B will not free up space as long as jail A is running. # cat /etc/fstab.test1 /storage/data /usr/jail/test1/storage/data nullfs rw,noatime 0 0 # cat /etc/fstab.test2 /storage/data /usr/jail/test2/storage/data nullfs rw,noatime 0 0 # cat /etc/jail.conf path = "/usr/jail/${name}"; exec.clean; exec.start = "sh /etc/rc"; exec.stop = "sh /etc/rc.shutdown jail"; host.hostname = "${name}"; mount.fstab = "/etc/fstab.${name}"; mount.devfs; test1 { } test2 { } # df -ht ufs Filesystem Size Used Avail Capacity Mounted on /dev/da0p2 8.9G 3.2G 5.0G 39% / # service jail start Starting jails: test1 test2. # df -ht ufs Filesystem Size Used Avail Capacity Mounted on /dev/da0p2 8.9G 3.2G 5.0G 39% / # jexec test1 dd if=/dev/random of=/storage/data/random.dd bs=1M count=1024 1024+0 records in 1024+0 records out 1073741824 bytes transferred in 3.723429 secs (288374490 bytes/sec) # df -ht ufs Filesystem Size Used Avail Capacity Mounted on /dev/da0p2 8.9G 4.2G 4.0G 52% / # jexec test2 rm /storage/data/random.dd # df -ht ufs Filesystem Size Used Avail Capacity Mounted on /dev/da0p2 8.9G 4.2G 4.0G 52% / # service jail stop test2 Stopping jails: test2. # df -ht ufs Filesystem Size Used Avail Capacity Mounted on /dev/da0p2 8.9G 4.2G 4.0G 52% / # service jail stop test1 Stopping jails: test1. # df -ht ufs Filesystem Size Used Avail Capacity Mounted on /dev/da0p2 8.9G 3.2G 5.0G 39% /
Let's loop in fs as well, since this is at the intersection of jails and filesystems.
Add "nocache" option to both nullfs mounts.
(In reply to Konstantin Belousov from comment #2) It is not documented and mount -t nullfs does not indicate that the option is used but it work. One of the jails is working as file server and there is a lot of read operations. Should I worry about hard disk head contention?
this seems to be related https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=245688
and this https://marc.info/?l=freebsd-fs&m=150581490210158&w=2 Konstantin Belousov, according to the newsletter, this problem has been going on since at least 2017?
*** Bug 270546 has been marked as a duplicate of this bug. ***
FreeBSD 13.2: The same thing happens when I use a program to optimise images (the images size shrinks but ZFS shows more usage than before). I have to restart jails to get the correct usage.
^Triage: is this still a problem on supported versions of FreeBSD?
Yes, on 14.1-RELEASE at least.
Hi, I can confirm this is still happening on a recent 14.2-STABLE. Regards,
The jails are not needed. It looks purely nullfs. # zfs create zroot/data/pr-test # cd /data/pr-test # mkdir storage test1 test2 # mount_nullfs -o rw,noatime ./storage ./test1 # mount_nullfs -o rw,noatime ./storage ./test2 # df -h | grep "pr-test" zroot/data/pr-test 327G 112K 327G 0% /data/pr-test /data/pr-test/storage 327G 112K 327G 0% /data/pr-test/test1 /data/pr-test/storage 327G 112K 327G 0% /data/pr-test/test2 # dd if=/dev/random of=./test1/random.dd bs=1M count=1024 # df -h | grep "pr-test" zroot/data/pr-test 327G 1.0G 326G 0% /data/pr-test /data/pr-test/storage 327G 1.0G 326G 0% /data/pr-test/test1 /data/pr-test/storage 327G 1.0G 326G 0% /data/pr-test/test2 # rm ./test2/random.dd # df -h | grep "pr-test" zroot/data/pr-test 327G 1.0G 326G 0% /data/pr-test /data/pr-test/storage 327G 1.0G 326G 0% /data/pr-test/test1 /data/pr-test/storage 327G 1.0G 326G 0% /data/pr-test/test2 Unmount of test1 releases the blocks. NB: mount_nullfs option nocache does not help for me.
Try https://reviews.freebsd.org/D48825 (I did not checked it)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=1a0cb938f7b461edc40316cb44a80438df674174 commit 1a0cb938f7b461edc40316cb44a80438df674174 Author: Konstantin Belousov <kib@FreeBSD.org> AuthorDate: 2025-02-04 00:23:05 +0000 Commit: Konstantin Belousov <kib@FreeBSD.org> CommitDate: 2025-02-04 22:19:50 +0000 unlink, rmdir: call notify_upper from VOP pre method instead of syscall Suppose that there are two or more nullfs mounts over some fs, and suppose the we unlink a file on one of the nullfs mount. This way notify_upper get called from the lower vnode as well, allowing the other nullfs mounts to note that and drop their caches for the unlinked vnode. PR: 254210 Reviewed by: olce Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D48825 sys/kern/vfs_subr.c | 2 ++ sys/kern/vfs_syscalls.c | 2 -- 2 files changed, 2 insertions(+), 2 deletions(-)