When using jails, if you specify a FIB to be used in the jail (so it can have it's own copy of the routing table) - running commands with 'jexec' ignores this FIB - and launches the command specified using the system default FIB (i.e. FIB 0). This makes troubleshooting FIB issues very tricky unless you're aware of this issue (think lots of lost hours! :) Fix: Either jexec documentation needs a warning that it will launch the process with the default FIB - not the one for the jail, and that you should use 'setfib jail-fib jexec 1 tcsh'. Or, have jexec actually honour the FIB set in /etc/jail.conf - so that 'jexec 1 tcsh' will set the correct FIB before launching the tcsh in jail #1. How-To-Repeat: Setup the system to support FIBs (multiple routing tables) - and configure a jail to use a specific FIB, i.e. using '/etc/jail.conf' - e.g. jail { [blah] exec.fib = 1; } When you launch the jail with 'jail -c jail' it will be created, and it will use the FIB specified. If you then attach to the jail, e.g. using 'jexec 1 tcsh' - that process will be launched with the default FIB (i.e. FIB 0) - and not the one that the jail is using.
Responsible Changed From-To: freebsd-bugs->freebsd-jail Over to maintainer(s).
Created attachment 188486 [details] A patch for the jexec 8 man page warning about fib issues After spending a lot of hours trying to troubleshoot using jails with fibs, the revelation that I was not using the jail's fib when executing a jexec utility command in a jail was heartbreaking. This is anti-intuitive and must be warned about. I submit the following patch to the jexec.8 manual page for consideration.
It looks something like this: ... BUGS If the jail is not identified by jid there is a possible race in between the lookup of the jail and executing the command inside the jail. Giving a jid has a similar race as another process can stop the jail and start another one after the user looked up the jid. If a forwarding information base (FIB, routing table) has been set within a jail using the exec.fib parameter, the jexec utility does not honour it, and command will be executed using the default fib (usually fib 0). This can confuse users of jails when trying to troubleshoot jail packet flows. To properly execute a command in a fib-configured jail using the jexec utility, it must be prefixed with the setfib(1) utility as follows. setfib -F 1 jexec testjail netstat -rn or setfib 1 jexec testjail netstat -rn where testjail has been assigned fib 1 in jail.conf(5) as follows: testjail { path = /tmp/jail/testjail; mount.devfs; host.hostname = testhostname; ip4.addr = 192.0.2.100; exec.fib = 1; interface = ed0; exec.start = "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown"; } FreeBSD 11.1-RELEASE-p1 April 24, 2016 FreeBSD 11.1-RELEASE-p1
batch change of PRs untouched in 2018 marked "in progress" back to open.
I propose we rather make jail(8) store the default FIB in the prison struct (alongside where the jails VNET and start-up parameters are stored), and have jexec(8) use this information to execute commands using the expected FIB. This will likely break the following command: # setfib 3 jexec some_jail_with_default_fib_2 ping 10.0.3.1 as jexec would now blindly set the FIB to 2 and not the expected 3, as there doesn't seem to be any way to tell if it was called using inherited FIB or explicitly set FIB. Therefore, my proposed fix will break current behavior, but will make jexec(8) behave as intuitively expected. Furthermore, the new behavior would be in line with my understanding of how calling cpuset(1) before executing a command in a jail which has a configured cpu mask would work: It'd pretty much ignore the preceding cpuset(1) call. (Please do correct me if I'm wrong.) I've already begun working on a patch implementing the above. Will submit a review once it's in a workable state, and update this bug.
I we store the FIB in the prison struct, then rather than having jexec(8) look there and set the FIB, it should just be done in the kernel, in jail_attach(2). That's been the solution that's been floating around on my back burner (for too long, I know). That would still break attempts to enter a jail explicitly under a different FIB, but I don't see a way around that. But then, I also don't see a reason to support that.
(In reply to Jamie Gritton from comment #6) That does indeed sound like a more correct implementation. Furthermore, it should still be possible to override the default FIB by running setfib inside the jail, as such: # jexec some_jail_with_default_fib_2 setfib 3 ping 10.0.3.1 At least I didn't get any immediate errors when doing this, and "setfib 1 jexec devsamba setfib 0 ping 8.8.8.8" made ping run in FIB 0 on my system.
I've definitely been "caught" by the "jexec problem" myself and would welcome this change. Perhaps an enhancement request as it requires some thought as to how to do it, but that a jail can't be made "safe" to FIB changes # jexec some_jail_with_default_fib_2 setfib 3 ping 10.0.3.1 is somewhat concerning. Being able to configure a jail so that one could "lock down" the FIB and its contents would be a welcome enhancement, one day.
(In reply to Jeff Kletsky from comment #8) Yep, that sounds like a useful security enhancement, but should probably be discussed in a separate PR (or directly on mailing list?) to not clutter this PR too much. :) (PS: You can apply such restrictions using IPFW, but it's more involved than a jail.conf knob)
Agreed, "locking down the FIB" is out of scope for this ticket. I brought it up here as there may be more than one implementation approach for this specific issue and one might be more amenable to such a future enhancement than another.