Bug 186360

Summary: [jail] jail using nullfs and unionfs doesn't mount devfs
Product: Base System Reporter: scottro11
Component: kernAssignee: freebsd-jail (Nobody) <jail>
Status: In Progress ---    
Severity: Affects Only Me CC: agh, gerald.gutierrez, jo, me, rwestlun, vkl
Priority: Normal    
Version: 10.0-RELEASE   
Hardware: Any   
OS: Any   

Description scottro11 2014-02-02 03:00:00 UTC 
	When creating a jail to share a nullfs jail, as described in detail at http://home.roadrunner.com/~computertaijutsu/nullfsjail.html, based
on the article at https://www.freebsd.org/doc/handbook/jails-application.html, devfs does not get mounted on the application jail, despite an
entry for it in /etc/jail.conf.  When using a standard jail (or jails), devfs does mount as expected.   
However, when using the nullfs method to share jails, after the application jail is started, its /dev only shows two directories, log and null.
I realize that this is a fairly uncommon use case, but it is something that worked without problem in FreeBSD-9.2, (although in that case, entries 
were made in /etc/rc.conf rather than /etc/jail.conf


 
How-To-Repeat:
	Install a jail as described in the link above, that shares a filesystem with a nullfs jail.   
Have the entry for the application jail to mount devfs and start the jail.

An example jail.conf entry

www_jail {
	path = /jails/_www_jail;
	mount.devfs;
	devfs_ruleset = 4;
	host.hostname = www_jail;
	ip4.addr = 192.168.1.199;
	exec.start = "/bin/sh /etc/rc";
	exec.stop = "/bin/sh /etc/rc.shutdown";
}

Fix: 

The only workarounds that I have found are to either manually mount devfs after starting the jail 
or to create an /etc/rc.local with a line to mount devfs on each application jail.
Comment 1 Mark Linimon freebsd_committer freebsd_triage 2014-04-20 04:17:50 UTC 
Responsible Changed
From-To: freebsd-bugs->freebsd-jail

Over to maintainer(s).
Comment 2 Scott Robbins 2014-05-26 20:14:02 UTC 
Just an additional note--the roadrunner page listed will be defunct after
May 31, 2014.  However, the page is available at
http://www.srobb.net/nullfsjail.html
-- 
Scott Robbins
PGP keyID EB3467D6
( 1B48 077D 66F6 9DB0 FDC2 A409 FA54 EB34 67D6 )
gpg --keyserver pgp.mit.edu --recv-keys EB3467D6
Comment 3 joeb1 2014-05-26 23:56:31 UTC 
jail(8) became available in 9.1-RELEASE and was very buggy.
Some things got fixed in 9.2 but not the mount devfs function.
Even in 10.0-RELEASE the mount devfs function was still broken.

It finally got fixed in 10.0-RELEASE-p1

The legacy rc.conf rc.d jail method is deprecated in 10.0 and scheduled for
removal in 11.0.

Suggest you test your jails using jail(8) method on 10.0-RELEASE-p1 to
verify your pr problem is still in effect.

The sysutils/qjail utility uses the same jail config as your now building by
hand.
Comment 4 Scott Robbins 2014-05-28 02:05:18 UTC 
On Mon, May 26, 2014 at 06:56:31PM -0400, joeb1 wrote:
> jail(8) became available in 9.1-RELEASE and was very buggy.
> Some things got fixed in 9.2 but not the mount devfs function.
> Even in 10.0-RELEASE the mount devfs function was still broken.
> 
> It finally got fixed in 10.0-RELEASE-p1

I am still having the issue with 10.0-RELEASE-p1


-- 
Scott Robbins
PGP keyID EB3467D6
( 1B48 077D 66F6 9DB0 FDC2 A409 FA54 EB34 67D6 )
gpg --keyserver pgp.mit.edu --recv-keys EB3467D6
Comment 5 jo 2014-10-05 01:18:46 UTC 
I have the same problem on:

FreeBSD fred 9.3-STABLE FreeBSD 9.3-STABLE #0 r272448: Fri Oct  3 19:02:13 BST 2014     root@fred:/usr/obj/usr/src/sys/FRED  amd64

By the looks of it, something deletes the stuff in the to-be-jailed /dev after devfs is mounted there. Unmounting and remount it makes it work properly. Also, mounting devfs from a prestart script (and preventing jail start up by returning exit code 1), makes devfs behave well.
So I'd suspect when jail(8) goes about starting up everything it messes up the device nodes.
Comment 6 vkl 2015-01-07 14:09:36 UTC 
I have the same problem on:

FreeBSD wsh.sc.local 10.1-RELEASE-p3 FreeBSD 10.1-RELEASE-p3 #0 r276500M: 
Fri Jan  2 20:41:41 MSK 2015     vk@wsh.sc.local:/usr/local/obj/usr/local/src/sys/SCWS_101_64  amd64

That problem have a place when I try to start jail based on unionfs root filesystem, but when jail start on nullfs based root filesystem all works as expected.
Comment 7 jo 2015-01-07 14:14:58 UTC 
I found a work-around for 9-stable.
In your prestart exec script, simply run an `ls /jail/whatever/root/dev` to populate the device nodes.

The ls gets executed before the jail is up, so devfs does the right thing.
Looks like devfs has a bug or permission problem when populating its mountpoint from within a jail.
Comment 8 vkl 2015-01-07 16:30:04 UTC 
(In reply to johannes from comment #7)

Thank you indeed Johannes !

You work-around works fine ! 
I was very amazed, because devfs in branch 10 mounts to jail from /sbin/jail, against /etc/rc.d/jail in 9, and more changed initialization procedure to start jail in 10. Especially since, I have not any problem whith start jails on unionfs based rootdir in 9.1-9.3 versions FreeBSD.
Comment 9 scottro11 2015-05-04 18:34:32 UTC 
I've also begun using that exec.prestart line and have added it to my page.  Thank you, it seems more elegant than an rc.local script to mount devfs in each jail.
Comment 10 Gerald Gutierrez 2018-12-31 19:26:38 UTC 
This is still a problem on FreeBSD xxx 12.0-RELEASE FreeBSD 12.0-RELEASE r341666 GENERIC amd64.