Bug 181794 - jexec(8) runs commands in Jails without taking into account of the Jail's FIB
Summary: jexec(8) runs commands in Jails without taking into account of the Jail's FIB
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 9.2-BETA2
Hardware: Any Any
: Normal Affects Only Me
Assignee: freebsd-jail (Nobody)
Keywords: patch
Depends on:
Reported: 2013-09-04 07:10 UTC by karl
Modified: 2018-06-20 12:56 UTC (History)
8 users (show)

See Also:

A patch for the jexec 8 man page warning about fib issues (1.18 KB, patch)
2017-12-03 09:21 UTC, Mark G.
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description karl 2013-09-04 07:10:00 UTC
When using jails, if you specify a FIB to be used in the jail (so it can have it's own copy of the routing table) - running commands with 'jexec' ignores this FIB - and launches the command specified using the system default FIB (i.e. FIB 0).

This makes troubleshooting FIB issues very tricky unless you're aware of this issue (think lots of lost hours! :)


Either jexec documentation needs a warning that it will launch the process with the default FIB - not the one for the jail, and that you should use 'setfib jail-fib jexec 1 tcsh'.

Or, have jexec actually honour the FIB set in /etc/jail.conf - so that 'jexec 1 tcsh' will set the correct FIB before launching the tcsh in jail #1.
How-To-Repeat: Setup the system to support FIBs (multiple routing tables) - and configure a jail to use a specific FIB, i.e. using '/etc/jail.conf' - e.g.

jail {
  exec.fib = 1;

When you launch the jail with 'jail -c jail' it will be created, and it will use the FIB specified.

If you then attach to the jail, e.g. using 'jexec 1 tcsh' - that process will be launched with the default FIB (i.e. FIB 0) - and not the one that the jail is using.
Comment 1 Mark Linimon freebsd_committer freebsd_triage 2014-05-04 06:29:38 UTC
Responsible Changed
From-To: freebsd-bugs->freebsd-jail

Over to maintainer(s).
Comment 2 Mark G. 2017-12-03 09:21:38 UTC
Created attachment 188486 [details]
A patch for the jexec 8 man page warning about fib issues

After spending a lot of hours trying to troubleshoot using jails
with fibs, the revelation that I was not using the jail's fib when
executing a jexec utility command in a jail was heartbreaking.

This is anti-intuitive and must be warned about.  I submit the
following patch to the jexec.8 manual page for consideration.
Comment 3 Mark G. 2017-12-03 09:23:17 UTC
It looks something like this:

     If the jail is not identified by jid there is a possible race in between
     the lookup of the jail and executing the command inside the jail.  Giving
     a jid has a similar race as another process can stop the jail and start
     another one after the user looked up the jid.

     If a forwarding information base (FIB, routing table) has been set within
     a jail using the exec.fib parameter, the jexec utility does not honour
     it, and command will be executed using the default fib (usually fib 0).
     This can confuse users of jails when trying to troubleshoot jail packet
     flows.  To properly execute a command in a fib-configured jail using the
     jexec utility, it must be prefixed with the setfib(1) utility as follows.

           setfib -F 1 jexec testjail netstat -rn
           setfib 1 jexec testjail netstat -rn

     where testjail has been assigned fib 1 in jail.conf(5) as follows:

           testjail {
                   path = /tmp/jail/testjail;
                   host.hostname = testhostname;
                   ip4.addr =;
                   exec.fib = 1;
                   interface = ed0;
                   exec.start = "/bin/sh /etc/rc";
                   exec.stop = "/bin/sh /etc/rc.shutdown";

FreeBSD 11.1-RELEASE-p1         April 24, 2016         FreeBSD 11.1-RELEASE-p1
Comment 4 Eitan Adler freebsd_committer freebsd_triage 2018-05-23 10:27:58 UTC
batch change of PRs untouched in 2018 marked "in progress" back to open.
Comment 5 Marie Helene Kvello-Aune 2018-06-19 09:23:23 UTC
I propose we rather make jail(8) store the default FIB in the prison struct (alongside where the jails VNET and start-up parameters are stored), and have jexec(8) use this information to execute commands using the expected FIB.

This will likely break the following command:
# setfib 3 jexec some_jail_with_default_fib_2 ping
as jexec would now blindly set the FIB to 2 and not the expected 3, as there doesn't seem to be any way to tell if it was called using inherited FIB or explicitly set FIB.

Therefore, my proposed fix will break current behavior, but will make jexec(8) behave as intuitively expected. 
Furthermore, the new behavior would be in line with my understanding of how calling cpuset(1) before executing a command in a jail which has a configured cpu mask would work: It'd pretty much ignore the preceding cpuset(1) call. (Please do correct me if I'm wrong.)

I've already begun working on a patch implementing the above. Will submit a review once it's in a workable state, and update this bug.
Comment 6 Jamie Gritton freebsd_committer 2018-06-19 15:17:05 UTC
I we store the FIB in the prison struct, then rather than having jexec(8) look there and set the FIB, it should just be done in the kernel, in jail_attach(2).  That's been the solution that's been floating around on my back burner (for too long, I know).

That would still break attempts to enter a jail explicitly under a different FIB, but I don't see a way around that.  But then, I also don't see a reason to support that.
Comment 7 Marie Helene Kvello-Aune 2018-06-19 16:33:57 UTC
(In reply to Jamie Gritton from comment #6)
That does indeed sound like a more correct implementation.

Furthermore, it should still be possible to override the default FIB by running setfib inside the jail, as such:
# jexec some_jail_with_default_fib_2 setfib 3 ping

At least I didn't get any immediate errors when doing this, and "setfib 1 jexec devsamba setfib 0 ping" made ping run in FIB 0 on my system.
Comment 8 Jeff Kletsky 2018-06-20 02:16:16 UTC
I've definitely been "caught" by the "jexec problem" myself and would welcome this change.

Perhaps an enhancement request as it requires some thought as to how to do it, but that a jail can't be made "safe" to FIB changes

    # jexec some_jail_with_default_fib_2 setfib 3 ping

is somewhat concerning. Being able to configure a jail so that one could "lock down" the FIB and its contents would be a welcome enhancement, one day.
Comment 9 Marie Helene Kvello-Aune 2018-06-20 10:53:06 UTC
(In reply to Jeff Kletsky from comment #8)
Yep, that sounds like a useful security enhancement, but should probably be discussed in a separate PR (or directly on mailing list?) to not clutter this PR too much. :) (PS: You can apply such restrictions using IPFW, but it's more involved than a jail.conf knob)
Comment 10 Jeff Kletsky 2018-06-20 11:49:05 UTC
Agreed, "locking down the FIB" is out of scope for this ticket. 

I brought it up here as there may be more than one implementation approach for this specific issue and one might be more amenable to such a future enhancement than another.