Bug 274263
| Summary: | Access to zfs snapshots within a jail return EPERM after a while of operation | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Base System | Reporter: | Markus Wild <freebsd-bugs> | ||||
| Component: | kern | Assignee: | Jamie Gritton <jamie> | ||||
| Status: | Closed FIXED | ||||||
| Severity: | Affects Only Me | CC: | andrew.hotlab, jamie, ml, tom, zlei | ||||
| Priority: | --- | ||||||
| Version: | 13.2-RELEASE | ||||||
| Hardware: | Any | ||||||
| OS: | Any | ||||||
| Attachments: |
|
||||||
Update: i just encountered the same bug on a different host system, that does not yet use custom web-specific mounts in the jail. So I guess we can rule out nullfs or zfs submounts from within the jail as the culprit. This system is running 13.2-RELEASE. I just hit this problem after upgrading from 12.4 to 13.2... what the heck! I waited as long as I could before letting OpenZFS bring my pools in its hands, but now the time has come... sigh! :'( To me it looks like the snapshot is unreadable within the jail until it has been accessed from the host. Anyone has happened to further investigate this issue since October? (In reply to Andrew from comment #2) I've now installed a user crontab entry like the following on all affected hosts (host, not jail): # weird bug leads to EPERM for access to /.zfs/snapshot/* within jails after a while, and gets cleared # with a hostside access 0 * * * * ls /services/jails/*/.zfs/snapshot/* >/dev/null this is obviously a workaround and not a fix.... Cheers, Markus I just noticed that the same issue is present in releng/14.0... it looks like pretty annoying to us, since we use jails everywhere! Someone can tell me how can I help to analyze the problem? Since it appeared after upgrading to the 13 branch, I guess it might be related to OpenZFS inclusion in FreeBSD. Maybe this bug should be assigned to fs@freebsd.org ? I'm very sorry to notice that the same bugs affect the just relesed FreeBSD 14.1-RELEASE! :( Any suggestion at this point? I just managed to find the solution while I was asking for help in this Matrix room: #lobby:matrix-dev.freebsd.org @dch pointed me to looking at jails parameters, and I discovered "zfs.mount_snapshot", which was introduced by this upstream commit: https://github.com/openzfs/zfs/pull/13758/files, and ported to FreeBSD by this one: https://cgit.freebsd.org/src/commit/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vfsops.c?id=dbd5678dc You can find more info about this in Bug 260160, which I think this one can be set the duplicate of. Personally, I think that "zfs.mount_snapshot" deserves to be referenced in jail(8), or maybe it should have been set to "1" by default, in order to preserve the behavior of jails in pre-OpenZFS FreeBSD versions. Such a PR would never have been created in both of cases. Yes, that went completely under the radar, probably because it never touched any of the jail source files. I agree that it should be documented in jail.8, and I'll add it there. I also agree it should have been default on (or perhaps even not an option), but that's probably better left for ZFS people to decide. A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=9cbf1de7e34a6fced041388fad5d9180cb7705fe commit 9cbf1de7e34a6fced041388fad5d9180cb7705fe Author: Jamie Gritton <jamie@FreeBSD.org> AuthorDate: 2024-06-24 20:00:49 +0000 Commit: Jamie Gritton <jamie@FreeBSD.org> CommitDate: 2024-06-24 20:00:49 +0000 Mark the zfs.mount_snapshot jail parameter as boolean. PR: 274263 Differential Revision: https://reviews.freebsd.org/D45647 sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vfsops.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=63875db56bd545896e019c922378f96ad5adc892 commit 63875db56bd545896e019c922378f96ad5adc892 Author: Jamie Gritton <jamie@FreeBSD.org> AuthorDate: 2024-06-24 20:03:45 +0000 Commit: Jamie Gritton <jamie@FreeBSD.org> CommitDate: 2024-06-24 20:03:45 +0000 Document the zfs.mount_snapshot parameter in jail(8). PR: 274263 Differential Revision: https://reviews.freebsd.org/D45647 usr.sbin/jail/jail.8 | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=c8f147943d2e0926793af3442dd26a8e5aebcda7 commit c8f147943d2e0926793af3442dd26a8e5aebcda7 Author: Jamie Gritton <jamie@FreeBSD.org> AuthorDate: 2024-06-24 20:00:49 +0000 Commit: Jamie Gritton <jamie@FreeBSD.org> CommitDate: 2024-06-28 19:44:56 +0000 MFC zfs/jail: Mark the zfs.mount_snapshot jail parameter as boolean. PR: 274263 Differential Revision: https://reviews.freebsd.org/D45647 (cherry picked from commit 9cbf1de7e34a6fced041388fad5d9180cb7705fe) sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vfsops.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=bb7b7b0eb7cf64109f05136d921a357f58839ba1 commit bb7b7b0eb7cf64109f05136d921a357f58839ba1 Author: Jamie Gritton <jamie@FreeBSD.org> AuthorDate: 2024-06-24 20:03:45 +0000 Commit: Jamie Gritton <jamie@FreeBSD.org> CommitDate: 2024-06-28 19:49:28 +0000 MFC zfs/jail: Document the zfs.mount_snapshot parameter in jail(8). PR: 274263 Differential Revision: https://reviews.freebsd.org/D45647 (cherry picked from commit 63875db56bd545896e019c922378f96ad5adc892) usr.sbin/jail/jail.8 | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=172fbf12fc22395b55d9dba6b81e7088c7ec43bd commit 172fbf12fc22395b55d9dba6b81e7088c7ec43bd Author: Jamie Gritton <jamie@FreeBSD.org> AuthorDate: 2024-06-24 20:00:49 +0000 Commit: Jamie Gritton <jamie@FreeBSD.org> CommitDate: 2024-06-28 19:50:44 +0000 MFC zfs/jail: Mark the zfs.mount_snapshot jail parameter as boolean. PR: 274263 Differential Revision: https://reviews.freebsd.org/D45647 (cherry picked from commit 9cbf1de7e34a6fced041388fad5d9180cb7705fe) sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vfsops.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=0e391d97fbe4d0369b001f1e645f6984c8c1bca1 commit 0e391d97fbe4d0369b001f1e645f6984c8c1bca1 Author: Jamie Gritton <jamie@FreeBSD.org> AuthorDate: 2024-06-24 20:03:45 +0000 Commit: Jamie Gritton <jamie@FreeBSD.org> CommitDate: 2024-06-28 19:51:58 +0000 MFC zfs/jail: Document the zfs.mount_snapshot parameter in jail(8). PR: 274263 Differential Revision: https://reviews.freebsd.org/D45647 (cherry picked from commit 63875db56bd545896e019c922378f96ad5adc892) usr.sbin/jail/jail.8 | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) Sorry to disturb again, but I've just upgraded my fist server from releng/13.3 to 13.4 and all jails which I configured with the parameter "zfs.mount_snapshot" failed to start! If I start them without it and then I try to set it on a running jail, I receive this error: # jail -m jid=12 zfs.mount_snapshot=1 jail: jail_set: Invalid argument I guessed that this parameter was no longer needed in this new release, hoping the classical behavior was restored, but all snapshot are no longer accessible from the jail! :( Created attachment 253669 [details]
Hack to set the zfs.mount_snapshot parameter
It looks like I implemented half a solution, which was worse than no solution at all. I change the description of the parameter to boolean, without realizing that continuing to expect an integer made the kernel reject jail(8)'s attempts to set it. I must have tested my fix on a mismatched kernel and userspace, or something.
All is not lost, as there's a hack available to work around this. jail_set(2) can still send an integer, even though jail(8) and libjail(3) will try not to. I've attached a small hack program than can do the job. You'll need to run it after the jail is created, e.g. with the following in jail.conf:
exec.created += "/path/to/setzfsmount $name";
Unfortunately, this workaround will remain necessary until the kernel
is fixed, probably on the next release.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=ae1a0648b05acf798816e7b83b3c10856de5c8e5 commit ae1a0648b05acf798816e7b83b3c10856de5c8e5 Author: Jamie Gritton <jamie@FreeBSD.org> AuthorDate: 2024-09-19 19:49:01 +0000 Commit: Jamie Gritton <jamie@FreeBSD.org> CommitDate: 2024-09-19 19:49:01 +0000 jail: unbreak the zfs.mount_snapshot option The change of its description from integer to boolean didn't actually change it to a boolean, but only made it impossible to set as either a boolean or an integer. Rather than make it work as a boolean parameter should, just revert to the old (working) integer parameter, and change the documentation to match. PR: 274263 Reported by: andrew.hotlab at hotmail sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vfsops.c | 2 +- usr.sbin/jail/jail.8 | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) Thank you very much Jamie! As a side note, I think that (boolean) parameter should be set "true" by default in order to maintain the same behavior of pre-OpenZFS FreeBSD versions. I recently started working with Illumos, and also inside their "zones" (as it has always been in FreeBSD jails) ZFS snapshots are mounted seamlessly on-demand, without the need to set any specific parameter. As a sysadmin, I cannot see any security problem in allowing jailed users to see previous versions of the file system they belong to (and obviously POSIX permissions and ACLs are effective on read-only mounted snapshots too). A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=80c5d78d90e65042cf7364b79f5afd3a0509c92e commit 80c5d78d90e65042cf7364b79f5afd3a0509c92e Author: Jamie Gritton <jamie@FreeBSD.org> AuthorDate: 2024-09-19 19:49:01 +0000 Commit: Jamie Gritton <jamie@FreeBSD.org> CommitDate: 2024-09-24 17:14:22 +0000 MFC jail: unbreak the zfs.mount_snapshot option The change of its description from integer to boolean didn't actually change it to a boolean, but only made it impossible to set as either a boolean or an integer. Rather than make it work as a boolean parameter should, just revert to the old (working) integer parameter, and change the documentation to match. PR: 274263 Reported by: andrew.hotlab at hotmail (cherry picked from commit ae1a0648b05acf798816e7b83b3c10856de5c8e5) sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vfsops.c | 2 +- usr.sbin/jail/jail.8 | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=f616b13bb35a5ffcc904d5282f09d93fe9d3a305 commit f616b13bb35a5ffcc904d5282f09d93fe9d3a305 Author: Jamie Gritton <jamie@FreeBSD.org> AuthorDate: 2024-09-19 19:49:01 +0000 Commit: Jamie Gritton <jamie@FreeBSD.org> CommitDate: 2024-09-24 17:18:41 +0000 MFC jail: unbreak the zfs.mount_snapshot option The change of its description from integer to boolean didn't actually change it to a boolean, but only made it impossible to set as either a boolean or an integer. Rather than make it work as a boolean parameter should, just revert to the old (working) integer parameter, and change the documentation to match. PR: 274263 Reported by: andrew.hotlab at hotmail (cherry picked from commit ae1a0648b05acf798816e7b83b3c10856de5c8e5) sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vfsops.c | 2 +- usr.sbin/jail/jail.8 | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) ^Triage: committed and MFCed. |
Overview: - host-system with jails, that run hosting environments (webs etc) - jails run in their own zfs datasets, but filesystems are managed by the host, not by the jails - daily zfs snapshots provide jail content of the previous days - customers are taught to, and use these snapshots for easy access to previous content - some time after a reboot (I'm not aware of the specific trigger), access to /.zfs/snapshot/{SPECIFICSNAP} from within the jail fails with EPERM, no matter whether the user is unprivileged or root. Listing of the snapshot directory itself (/.zfs/snapshot) works fine. - I found that if I list that snapshot directory from the host system (even as unprivileged user), access from within the jail works fine again (for a while) Specifics (possibly too verbose, sorry): - current host system: 13.1-RELEASE-p6 - sample zfs layout: tank/services/jails/reseller123 tank/services/jails/reseller123/chrimg/someweb.com tank/services/jails/reseller123/chrimg/someweb.com/tmp - corresponding zfs filesystem mountpoints: /services/jails/reseller123 /services/jails/reseller123/services/webs/someweb.com/system /services/jails/reseller123/services/webs/someweb.com/system/tmp - we clone specific php-version based template filesystems to a system directory within a web root to implement different php environments. these clones are all read-only - there's an additional devfs and a nullfs mount per web: devfs on /services/jails/reseller123/services/webs/someweb.com/dev /services/addons nullfs read-only to /services/jails/reseller123/services/webs/someweb.com/system/services/addons the nullfs mount grants access to common typo3 version templates the customer uses within webs - customer accesses /.zfs/snapshot via a symlink we provide in /services/webs/backup, but the observed behavior happens via direct access as well as via symlink, so I don't think this is relevant In the failure case, I get this: $ ls /services/webs/backup/2023-10-04-00:00:04 ls: /services/webs/backup/2023-10-04-00:00:04: Operation not permitted or with ktrace: ... 34393 ls CALL fstatat(AT_FDCWD,0x8006df098,0x7fffffffe2b0,0) 34393 ls NAMI "/services/webs/backup/2023-10-04-00:00:04" 34393 ls RET fstatat -1 errno 1 Operation not permitted 34393 ls CALL fstatat(AT_FDCWD,0x8006df098,0x7fffffffe2b0,0x200<AT_SYMLINK_NOFOLLOW>) 34393 ls NAMI "/services/webs/backup/2023-10-04-00:00:04" 34393 ls RET fstatat -1 errno 1 Operation not permitted Now, if I do the same ls on the host system (also as unprivileged user), it works fine. And as soon as I've done that, the jail access works again. Could there be permission issues with nullfs in jails, such as in this setup? Unfortunately, I've so far not found a way to forcibly cause the failure.