Bug 181794

Comment 1 Mark Linimon 2014-05-04 06:29:38 UTC

Responsible Changed
From-To: freebsd-bugs->freebsd-jail

Over to maintainer(s).

Created attachment 188486 [details]
A patch for the jexec 8 man page warning about fib issues

After spending a lot of hours trying to troubleshoot using jails
with fibs, the revelation that I was not using the jail's fib when
executing a jexec utility command in a jail was heartbreaking.

This is anti-intuitive and must be warned about.  I submit the
following patch to the jexec.8 manual page for consideration.

It looks something like this:

...
BUGS
     If the jail is not identified by jid there is a possible race in between
     the lookup of the jail and executing the command inside the jail.  Giving
     a jid has a similar race as another process can stop the jail and start
     another one after the user looked up the jid.

     If a forwarding information base (FIB, routing table) has been set within
     a jail using the exec.fib parameter, the jexec utility does not honour
     it, and command will be executed using the default fib (usually fib 0).
     This can confuse users of jails when trying to troubleshoot jail packet
     flows.  To properly execute a command in a fib-configured jail using the
     jexec utility, it must be prefixed with the setfib(1) utility as follows.

           setfib -F 1 jexec testjail netstat -rn
     or
           setfib 1 jexec testjail netstat -rn

     where testjail has been assigned fib 1 in jail.conf(5) as follows:

           testjail {
                   path = /tmp/jail/testjail;
                   mount.devfs;
                   host.hostname = testhostname;
                   ip4.addr = 192.0.2.100;
                   exec.fib = 1;
                   interface = ed0;
                   exec.start = "/bin/sh /etc/rc";
                   exec.stop = "/bin/sh /etc/rc.shutdown";
           }

FreeBSD 11.1-RELEASE-p1         April 24, 2016         FreeBSD 11.1-RELEASE-p1

Comment 4 Eitan Adler 2018-05-23 10:27:58 UTC

batch change of PRs untouched in 2018 marked "in progress" back to open.

I propose we rather make jail(8) store the default FIB in the prison struct (alongside where the jails VNET and start-up parameters are stored), and have jexec(8) use this information to execute commands using the expected FIB.

This will likely break the following command:
# setfib 3 jexec some_jail_with_default_fib_2 ping 10.0.3.1
as jexec would now blindly set the FIB to 2 and not the expected 3, as there doesn't seem to be any way to tell if it was called using inherited FIB or explicitly set FIB.

Therefore, my proposed fix will break current behavior, but will make jexec(8) behave as intuitively expected. 
Furthermore, the new behavior would be in line with my understanding of how calling cpuset(1) before executing a command in a jail which has a configured cpu mask would work: It'd pretty much ignore the preceding cpuset(1) call. (Please do correct me if I'm wrong.)

I've already begun working on a patch implementing the above. Will submit a review once it's in a workable state, and update this bug.

Comment 6 Jamie Gritton 2018-06-19 15:17:05 UTC

I we store the FIB in the prison struct, then rather than having jexec(8) look there and set the FIB, it should just be done in the kernel, in jail_attach(2).  That's been the solution that's been floating around on my back burner (for too long, I know).

That would still break attempts to enter a jail explicitly under a different FIB, but I don't see a way around that.  But then, I also don't see a reason to support that.

(In reply to Jamie Gritton from comment #6)
That does indeed sound like a more correct implementation.

Furthermore, it should still be possible to override the default FIB by running setfib inside the jail, as such:
# jexec some_jail_with_default_fib_2 setfib 3 ping 10.0.3.1

At least I didn't get any immediate errors when doing this, and "setfib 1 jexec devsamba setfib 0 ping 8.8.8.8" made ping run in FIB 0 on my system.

I've definitely been "caught" by the "jexec problem" myself and would welcome this change.

Perhaps an enhancement request as it requires some thought as to how to do it, but that a jail can't be made "safe" to FIB changes

    # jexec some_jail_with_default_fib_2 setfib 3 ping 10.0.3.1

is somewhat concerning. Being able to configure a jail so that one could "lock down" the FIB and its contents would be a welcome enhancement, one day.

(In reply to Jeff Kletsky from comment #8)
Yep, that sounds like a useful security enhancement, but should probably be discussed in a separate PR (or directly on mailing list?) to not clutter this PR too much. :) (PS: You can apply such restrictions using IPFW, but it's more involved than a jail.conf knob)

Agreed, "locking down the FIB" is out of scope for this ticket. 

I brought it up here as there may be more than one implementation approach for this specific issue and one might be more amenable to such a future enhancement than another.