Bug 274263 - Access to zfs snapshots within a jail return EPERM after a while of operation
Summary: Access to zfs snapshots within a jail return EPERM after a while of operation
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 13.2-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-jail (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-10-04 13:33 UTC by Markus Wild
Modified: 2024-03-25 13:32 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Markus Wild 2023-10-04 13:33:33 UTC
Overview:
- host-system with jails, that run hosting environments (webs etc)
- jails run in their own zfs datasets, but filesystems are managed by the host, not by the jails
- daily zfs snapshots provide jail content of the previous days
- customers are taught to, and use these snapshots for easy access to previous content
- some time after a reboot (I'm not aware of the specific trigger), access to /.zfs/snapshot/{SPECIFICSNAP}
  from within the jail fails with EPERM, no matter whether the user is unprivileged or root. Listing of
  the snapshot directory itself (/.zfs/snapshot) works fine.
- I found that if I list that snapshot directory from the host system (even as unprivileged user), access
  from within the jail works fine again (for a while)

Specifics (possibly too verbose, sorry):
- current host system: 13.1-RELEASE-p6
- sample zfs layout:
  tank/services/jails/reseller123
  tank/services/jails/reseller123/chrimg/someweb.com
  tank/services/jails/reseller123/chrimg/someweb.com/tmp
- corresponding zfs filesystem mountpoints:
  /services/jails/reseller123
  /services/jails/reseller123/services/webs/someweb.com/system
  /services/jails/reseller123/services/webs/someweb.com/system/tmp
- we clone specific php-version based template filesystems to a system directory within a web root to implement different php environments.
  these clones are all read-only
- there's an additional devfs and a nullfs mount per web:
  devfs on /services/jails/reseller123/services/webs/someweb.com/dev
  /services/addons nullfs read-only to /services/jails/reseller123/services/webs/someweb.com/system/services/addons
  the nullfs mount grants access to common typo3 version templates the customer uses within webs
- customer accesses /.zfs/snapshot via a symlink we provide in /services/webs/backup, but the observed behavior
  happens via direct access as well as via symlink, so I don't think this is relevant

In the failure case, I get this:
$ ls /services/webs/backup/2023-10-04-00:00:04
ls: /services/webs/backup/2023-10-04-00:00:04: Operation not permitted

or with ktrace:
...
 34393 ls       CALL  fstatat(AT_FDCWD,0x8006df098,0x7fffffffe2b0,0)
 34393 ls       NAMI  "/services/webs/backup/2023-10-04-00:00:04"
 34393 ls       RET   fstatat -1 errno 1 Operation not permitted
 34393 ls       CALL  fstatat(AT_FDCWD,0x8006df098,0x7fffffffe2b0,0x200<AT_SYMLINK_NOFOLLOW>)
 34393 ls       NAMI  "/services/webs/backup/2023-10-04-00:00:04"
 34393 ls       RET   fstatat -1 errno 1 Operation not permitted


Now, if I do the same ls on the host system (also as unprivileged user), it works fine. And as soon as I've
done that, the jail access works again.

Could there be permission issues with nullfs in jails, such as in this setup? 
Unfortunately, I've so far not found a way to forcibly cause the failure.
Comment 1 Markus Wild 2023-10-10 08:35:00 UTC
Update: 
i just encountered the same bug on a different host system,
that does not yet use custom web-specific mounts in the jail.
So I guess we can rule out nullfs or zfs submounts from within 
the jail as the culprit.

This system is running 13.2-RELEASE.
Comment 2 Andrew 2023-12-28 17:50:12 UTC
I just hit this problem after upgrading from 12.4 to 13.2... what the heck!
I waited as long as I could before letting OpenZFS bring my pools in its hands, but now the time has come... sigh! :'(

To me it looks like the snapshot is unreadable within the jail until it has been accessed from the host.
Anyone has happened to further investigate this issue since October?
Comment 3 Markus Wild 2023-12-29 08:07:53 UTC
(In reply to Andrew from comment #2)
I've now installed a user crontab entry like the following on all
affected hosts (host, not jail):

# weird bug leads to EPERM for access to /.zfs/snapshot/* within jails after a while, and gets cleared
# with a hostside access
0       *       *       *       *       ls /services/jails/*/.zfs/snapshot/* >/dev/null

this is obviously a workaround and not a fix.... 

Cheers,
Markus
Comment 4 Andrew 2024-03-23 11:46:19 UTC
I just noticed that the same issue is present in releng/14.0... it looks like pretty annoying to us, since we use jails everywhere!

Someone can tell me how can I help to analyze the problem? Since it appeared after upgrading to the 13 branch, I guess it might be related to OpenZFS inclusion in FreeBSD.

Maybe this bug should be assigned to fs@freebsd.org ?