When setting the property "jailed=on" for a dataset and this dataset is mounted in a jail: any snapshot will not be accessible: How to reproduce: On host create a dataset # zfs create zdisk/test # zfs set mountpoint=/mnt/famnas # zfs set jailed=on zdisk/test Make sure that /mnt is writable in the jail... go into jail: # zfs list NAME USED AVAIL REFER MOUNTPOINT zdisk 5.08T 8.97T 88K /zdisk zdisk/test 5.08T 8.97T 136K /mnt/test # zfs snapshot zdisk/test@now # ls -li /mnt/myjail/.zfs/snapshot ls: now: Operation not permitted total 0 On older FreeBSD versions this worked.
I can confirm that ZFS snapshots are accessible in jails on my systems based on 12.3-STABLE.
I can confirm numerous systems from 13.0-STABLE #0 stable/13-n245258-5b8039d5ed3e and later that do exhibit this bug. Further, it is not necessary to set "jailed=on" .... none of my systems use that. My experience suggests that it is sufficient simply to have a jail rooted at the mountpoint of a ZFS filesystem. Prior to 13.0-RELEASE the jail had read-only access to the snapshots of its own filesystem, for purposes of restoring single files vi "cp" or "rsync" or simply "diff". @Urs, can you please test to confirm that you still observe the PR behavior even when "jailed=off", and update with your finding? Thank you!
We are also encountering this bug. But the workarounds suggesting jailed=off do not work when using iocage - when we set jailed=off then start the jail with iocage start <jail_name> - iocage sets the jailed property to on. As an effect we are completely unable tu access the snapshots contents from within or without the jails (except if we shutdown the jail, set jailed=off, change the mount point and then access the content of the snapshot - which is too cumbersome to be usable in practice). This is a regression as it worked under 12.X Tested with FreeBSD 13.0-RELEASE-p7 and py38-iocage-1.2_9 HELP!!! We really need to access the snapshots from within jails !
The following openzfs commit fixes this, https://github.com/openzfs/zfs/commit/4a1195ca5041cbff2a6b025a31937fef84876c52 - which doesn't appear to be in stable/13. I'll cc @allanjude and @mm; maybe they know whether it will be in 13.1?
Thank you, @rew. Please update this PR when that commit goes in to 13-STABLE. I track that branch and can verify whether it resolves the issues my systems are experiencing. Thank you!
Don't mean to be pushy. But is this fix going to be in 13.1 RELEASE? or in which version will it be? Because this basically broke our complete stack of FreeBSD+zfs+jails !
I think stable/13 is supposed to pick up fixes off of the openzfs 2.1.3 branch. The aforementioned commit isn't present there, so I suppose we should merge it upstream. I think we could simply cherry-pick it straight to stable/13 and releng/13.1 in the meantime. Allan, Martin, is there any reason not to do this?
Sorry to insist. This breaks our all FreeBSD 13 + Jails + ZFS + access to snapshots from inside the jails. To the point of being unable to deploy new jails and having to revert to 12.3 entire production servers hosting dozens of jails. The cost is already in the several hundreds of hours for us. There is no mention of this bug fix in the release notes for 13.1-RELEASE ! Is there a way to 'add' this fix to a working 13.1-RELEASE installation? Is it going to be in 13.1? 13.2? Thanks for your help in seeing a bit more clearly into this.
I have the following comment from Mateusz Guzik <mjg@FreeBSD.org> oh this patch: On 29. 3. 2022 17:53, Mateusz Guzik wrote: > whoa, stop right there. > > The commit at hand introduces a security problem by modifying proc > credentials. Another thread in the same process can race to snatch > these creds permanently for itself. > > The patch is definitely wrong. I don't want to open a security hole, we need a new fix for this.
Given that this is a regression, has anyone bisected the commit history to find which commit created this bug, and if so, could they post a link to it? Thank you!
(In reply to Jim Long from comment #11) This is the commit you're looking for: https://github.com/openzfs/zfs/commit/eff621071ffd33764b833ea16894bee47d6ffb34 The author of that commit suggested that it might be better to revert eff62107 instead of taking the work around that was committed in 4a1195ca.
(In reply to Robert Wing from comment #12) Thanks for sharing this. It seems that this bug will not be fixed in 13.1. Will it be fixed in 13.2? I hope that such regressions get more attention in the future. We may be able to sponsor this fix - how? In meantime we need to stick with 12.3, unfortunately.
I would like to second the request to resolve this issue prior to 13.2-RELEASE. My workplace uses numerous jailed Samba instances where we'd like to allow users to access their own backups via the /.zfs/snapshot/ directory in the data share.
Proposed fix https://github.com/openzfs/zfs/pull/13758
Any update on whether this fix will be included in 13.2-RELEASE? I'm running a PRERELEASE built yesterday, stable/13-n254268-821f1352112e and there's no sign of improvement. Thank you!
(In reply to Jim Long from comment #16) Looks like https://github.com/openzfs/zfs/pull/13758 was not merged into zfs-2.1-release because of an easy to resolve conflict. I will cherry-pick it manually i to stable/13 and open an issue at OpenZFS.
(In reply to Martin Matuska from comment #17) Thank you Martin
Yes, thank you, Martin. Please update that thread when this is committed and I'll be happy to do another pull, build and test.
Sorry, THIS thread!
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3d7a882488a4c1c69283614ae179df7c1d2f3a7e commit 3d7a882488a4c1c69283614ae179df7c1d2f3a7e Author: Martin Matuska <mm@FreeBSD.org> AuthorDate: 2023-02-04 18:39:12 +0000 Commit: Martin Matuska <mm@FreeBSD.org> CommitDate: 2023-02-04 19:50:33 +0000 zfs: fix mounting snapshots in .zfs/snapshot as a regular user This is a cherry-pick from vendor/openzfs/master resulting in a direct commit to stable/13 The cherry-pick to branch zfs-2.1-release has been proposed upstream. Vendor pull requeset merges: #13758 Allow mounting snapshots in .zfs/snapshot as a regular user #14218 Bugfix: Avoid a null pointer dereference in zfs_mount() on FreeBSD PR: 260160 (cherry picked from commit 595d3ac2ed61331124feda2cf5787c3dd4c7ae09) (cherry picked from commit d27a00283faf4ec2b997ff2376dee4a080e1ca7b) .../openzfs/module/os/freebsd/spl/spl_vfs.c | 10 +- .../openzfs/module/os/freebsd/zfs/zfs_vfsops.c | 279 ++++++++++++++++++++- 2 files changed, 276 insertions(+), 13 deletions(-)
Well, no luck yet, but I think the first obstacle I have is: # cat /etc/jail.conf path = "/jail/$name"; test { host.hostname = "test.example.com"; ip4.addr = "vlan3|192.168.1.50/32"; allow.raw_sockets = 1; exec.clean; exec.system_user = "root"; exec.jail_user = "root"; exec.start += "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown"; exec.consolelog = "/var/log/jail_test.log"; mount.devfs; # When this is supported, that likely means the jail vs. snapshot bug has # been fixed: zfs.mount_snapshot=1; allow.set_hostname = 0; allow.sysvipc = 0; } But the jail is not seeing the mount_snapshot attribute: # service jail restart; jexec test sysctl -a | grep zfs\.mount Stopping jails: test. Starting jails: test. security.jail.param.zfs.mount_snapshot: 0 Without that attribute set, then of course: # jexec test ls -l /.zfs/snapshot/prior-to-installkernel ls: /.zfs/snapshot/prior-to-installkernel: Operation not permitted What am I overlooking?
(In reply to Jim Long from comment #22) It appears that it doesn't work if that parameter is set when creating a jail. If you set the parameter on an existing jail (e.g. with "jail -m test" or with "jail -m jid=somenumber zfs.mount_snapshot=1) then you can mount the snapshots, but inside the jail the sysctl still shows a zero. Maybe Allan Jude has more info on this.
Thanks to everyone who has contributed to the significant progress on this issue. It definitely feels like progress, but IMO it would be infinitely better if the "zfs.mount_snapshot" attribute was honored at startup. I'll spend some time thinking about how to write a service script that can either piggy-back on /etc/rc.d/jail or a patch to /etc/rc.d/jail that will incorporate the "jail -m" in the start-up process. But "The Right Thing" would certainly seem to be to have /etc/rc.d/jail honor the settings placed in /etc/jail.conf. Interestingly, my testing suggests that once a "jail -m" operation has been done, the mount_snapshot setting somehow survives any subsequent destruction and re-creation of the jail. Here's a dialogue commenced shortly after a fresh boot: # w 12:26PM up 54 secs, 3 users, load averages: 0.89, 0.24, 0.09 USER TTY FROM LOGIN@ IDLE WHAT jim v0 - 12:26PM - /bin/sh /usr/local/bin/startx jim pts/0 :0 12:26PM - - jim pts/1 :0 12:26PM - w # jls JID IP Address Hostname Path 1 192.168.1.50 test.example.edu /jail/test 2 192.168.1.50 test2.example.edu /jail/test2 # jexec test ls -ld /.zfs/snapshot/xfer ls: /.zfs/snapshot/xfer: Operation not permitted # jexec test2 ls -ld /.zfs/snapshot/xfer ls: /.zfs/snapshot/xfer: Operation not permitted # jail -m test test: updated # jail -m test2 test2: updated # jexec test ls -ld /.zfs/snapshot/xfer drwxr-xr-x 33 root wheel 38 Feb 6 10:22 /.zfs/snapshot/xfer # jexec test2 ls -ld /.zfs/snapshot/xfer drwxr-xr-x 33 root wheel 38 Feb 6 10:22 /.zfs/snapshot/xfer The odd thing to me is that once "jail -m" has been performed, the change persists even beyond the destruction and re-creation of either jail. Here I stop jail 1 and jail 2, then start jail 3 and jail 4. Without any "jail -m" command, jails 3 and 4 are able to see snapshots: # jls JID IP Address Hostname Path 1 192.168.1.50 test.example.edu /jail/test 2 192.168.1.50 test2.example.edu /jail/test2 # service jail stop Stopping jails: test test2. # service jail start Starting jails: test test2. # jls JID IP Address Hostname Path 3 192.168.1.50 test.example.edu /jail/test 4 192.168.1.50 test2.example.edu /jail/test2 # jexec test ls -ld /.zfs/snapshot/xfer drwxr-xr-x 33 root wheel 38 Feb 6 10:22 /.zfs/snapshot/xfer # jexec test2 ls -ld /.zfs/snapshot/xfer drwxr-xr-x 33 root wheel 38 Feb 6 10:22 /.zfs/snapshot/xfer Thanks again to Allan, Martin, Ryan, and everyone else for their time and expertise.
For those affected by this regression who are looking for a workaround that eliminates (or rather, automates) the need to run 'jail -m ...' on each jail that requires access to its snapshots: This simple change to the globals section of /etc/jail.conf uses the "exec.poststart" hook to have /etc/rc.d/jail spawn a small subshell process after each jail is started (either at boot time or by "service jail [re]start"). From man 8 jail: exec.poststart Command(s) to run in the system environment after a jail is created, and after any exec.start commands have completed. This is very much a temporary hack until a proper fix is merged into the codebase, but the method appears to be both effective and minimally intrusive: /etc/jail.conf: # Global: # .. whatever global stuff you already have zfs.mount_snapshot = 1; exec.poststart = "( /bin/sleep 30; /usr/sbin/jail -m $name ) &"; # ... and whatever else you like ... jail1 { # ... your normal per-jail parameters here } jail2 { # ... your normal per-jail parameters here } If you prefer not to do this globally, the "poststart" directive can be deleted from the global context and inserted into each individual jail where it is needed. For my purposes the global context is more convenient, because my primary use case is several jailed Samba instances where I want to allow end-user access to ZFS snapshots of the data share, so that users can access backups without any sysadmin intervention. A secondary use case is to simply allow root-users of jailed FreeBSD instances to access snapshot backups of their own system files. In both cases, however, it is also desired that the root user of the jail NOT be able to destroy snapshots. Hence the reason for not simply setting "jailed=on" and delegating the entire ZFS filesystem to the jail. Effectively this makes the backups readable by the jail, but immutable by even the jail root. HTH, Jim
With some debugging I discovered this did not happen if you set the 'persist' flag on the jail, or if you created the jail manually. This lead to the discovery that the rc.d/jail script setting the 'nopersist' flag was incorrectly removing the per-jail ZFS settings because it was misinterpreted as 'there are no settings, inherit instead' I've opened an upstream pull request: https://github.com/openzfs/zfs/pull/14509 but will also work on getting this landed for 13.2
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=8b04c1cbfc1cb71a1ce53b3a7855f1d45866fcfb commit 8b04c1cbfc1cb71a1ce53b3a7855f1d45866fcfb Author: Allan Jude <allanjude@FreeBSD.org> AuthorDate: 2023-02-18 01:44:34 +0000 Commit: Allan Jude <allanjude@FreeBSD.org> CommitDate: 2023-02-21 22:42:28 +0000 Fix per-jail zfs.mount_snapshot setting When jail.conf set the nopersist flag during startup, it was incorrectly destroying the per-jail ZFS settings. PR: 260160 Reported by: imp (previous version), mm (upstream), freqlabs (upstream) MFC after: immediately Sponsored by: Modirum MDPay Sponsored by: Klara, Inc. Differential Revision: https://reviews.freebsd.org/D38662 sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vfsops.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=426ed00525409d084e97dc44397722aff2cc0bb3 commit 426ed00525409d084e97dc44397722aff2cc0bb3 Author: Allan Jude <allanjude@FreeBSD.org> AuthorDate: 2023-02-18 01:44:34 +0000 Commit: Allan Jude <allanjude@FreeBSD.org> CommitDate: 2023-02-21 22:47:21 +0000 Fix per-jail zfs.mount_snapshot setting When jail.conf set the nopersist flag during startup, it was incorrectly destroying the per-jail ZFS settings. PR: 260160 Reported by: imp (previous version), mm (upstream), freqlabs (upstream) MFC after: immediately Sponsored by: Modirum MDPay Sponsored by: Klara, Inc. Differential Revision: https://reviews.freebsd.org/D38662 (cherry picked from commit 8b04c1cbfc1cb71a1ce53b3a7855f1d45866fcfb) sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vfsops.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
A commit in branch releng/13.2 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=b772fe0d662650d8a517bc07f3c433486c24b347 commit b772fe0d662650d8a517bc07f3c433486c24b347 Author: Allan Jude <allanjude@FreeBSD.org> AuthorDate: 2023-02-18 01:44:34 +0000 Commit: Allan Jude <allanjude@FreeBSD.org> CommitDate: 2023-02-22 13:48:20 +0000 Fix per-jail zfs.mount_snapshot setting When jail.conf set the nopersist flag during startup, it was incorrectly destroying the per-jail ZFS settings. PR: 260160 Reviewed by: imp (previous version), mm (upstream), freqlabs (upstream) Approved by: re (cperciva) Sponsored by: Modirum MDPay Sponsored by: Klara, Inc. Differential Revision: https://reviews.freebsd.org/D38662 (cherry picked from commit 8b04c1cbfc1cb71a1ce53b3a7855f1d45866fcfb) (cherry picked from commit 426ed00525409d084e97dc44397722aff2cc0bb3) sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vfsops.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
I'm happy to say that using a build as of: stable/13-n254696-18936d3526f3 my systems are showing this issue as resolved. The exec.poststart kludge above is no longer necessary. @Urs, @c.tineo, @fk, what do you report? Again, my sincerest thanks to Allan, Martin, Ryan, and everyone else who contributed.
(In reply to Jim Long from comment #30) Using kernel13.2 I can manage dataset from jail, mount/snapshot it, but I can't umount snapshots nor destroy snapshots. I can manage snapshots of that dataset without problems from the physical host. I have the following jail parameters: allow.mount=1 allow.mount.zfs=1 enforce_statfs=1 exec.poststart = 'zfs jail PostgreSQL00 zroot/data_PostgreSQL00'; The dataset was created with the following commands: zfs create -o mountpoint=/var/db/postgres/data15 -o canmount=noauto zroot/data_PostgreSQL00 And applied jail-friendly parameter: zfs set jailed=on zroot/data_PostgreSQL00 If I try to umount it I get: umount /var/db/postgres/data15/.zfs/snapshot/backup umount: unmount of /var/db/postgres/data15/.zfs/snapshot/backup failed: Operation not permitted If I try to destroy it I get: zfs destroy zroot/data_PostgreSQL00@backup cannot destroy snapshot zroot/data_PostgreSQL00@backup: dataset is busy Am I doing anything wrong? Maybe its a bug?
I have tested this issue on 14.0. The zfs snapshots work fine within jails, great! Time to update everything to 14.0! Many thanks to everyone who has helped to resolve this regression. It is very much appreciated. I close this issue, just before the 2023 ends! Happy New Year to everyone! -Urs