Bug 221512

Summary:	Contents of ZFS datasets invisible after being mounted inside a jail with nullfs (Even after a reboot!)
Product:	Base System	Reporter:	Troels Just <tsuroerusu>
Component:	kern	Assignee:	freebsd-bugs (Nobody) <bugs>
Status:	Closed Not A Bug
Severity:	Affects Only Me
Priority:	---
Version:	11.1-RELEASE
Hardware:	amd64
OS:	Any

Description Troels Just 2017-08-14 08:49:58 UTC

I am running a FreeBSD 11.1 system with ZFS and jails and I mount a file system on my storage pool (/storage/cloud) into the jail (/jails/cloud/storage) via nullfs, and that works fine for what the jail does. However I just noticed that outside the jail, I can only see the mount points of the datasets/file system children, but not the contents.

For example if I do: ls -la /storage/cloud/* (As root) then the children of "cloud" just appear empty, but the data (files and folders) are present inside the jail at the nullfs-mounted location. Even if I stop the jail, they still are not present outside the jail.

I just tried disabling jails in /etc/rc.conf and rebooting the system and EVEN THEN the files do not reappear outside the jail in their original location, yet zfs list reports the space being used. But if I then (without rebooting) enable the jail, the files show up in the nullfs-mounted location inside the jail, but still remain invisible outside the jail!

As an experiment, I tried creating /mnt/test and /mnt/test2 and creates a folder and a few files in test, and then nullfs-mounted it onto test2, and I could then see the files in both locations.

The fact that my data stays hidden after a reboot is rather worrisome.

Comment 1 Troels Just 2017-08-14 12:06:20 UTC

With a little bit of help, I managed to figure out what the problem was. It turned out that I had run, head first, into the wall of nullfs not operating across file systems. Thus inside the jail, the data was being saved to a regular directory on my "storage/cloud"-dataset (Mountpoint: /storage/cloud), whereas outside the jail the child file system "storage/cloud/bc", having been left empty, was mounted over the folder "bc" on "storage/cloud", where the files actually resided, thus "hiding" them from view outside the jail. A simple "zfs unmount storage/cloud/bc" revealed them.

Anybody reading this in the future after a Google search: If you are using ZFS and want to put a file system and all of its children into a jail then you cannot use nullfs. Instead you need to either change the actual mount point using "zfs set mountpoint=/jails/jail1/storage storage/path/to/filesystem" or to attach the entire dataset to the jail, and allow it to be managed from within the jail using "zfs jail".

I would like to propose that the man page for mount_nullfs contain a short sentence about nullfs not crossing file system boundaries. From what I can tell by, admittedly, giving it a quick look, I didn't see anything about that, so it's possible that I simply missed it if it actually is in there.

I apologize for any inconvenience caused by this report.