If you want a specific primary IP in a jail, add that IP on the host before any other IP used by that jail. Allan Jude and I spoke about this on IRC and he asked that I create a PR for it.
I am totally confused by this report. It does not make sense to me. You can have a primary jail IP and only add it to the host after you started the jail and it will then do the right thing as described in the man page. What is the real problem?
From IRC, posted with consent of bz@: [Oct 08 11:37] <@battlez> dvl: the real question is, which order does jls -av or similar list the addresses when you run this from the base system? [Oct 08 11:37] <@battlez> ifconfig is "random order" and desn't matter [Oct 08 11:38] <@dvl> battlez: that is interesting. jls -av lists: 199.xxx 10.xxx 10.xxx 199.xxx [Oct 08 11:38] <@battlez> dvl: and the theory that the jail does depend on the order in "ifconfig" on the host is entirely worng as well unfortunately; it might depend on what route -get <destination> thinks though [Oct 08 11:39] <@battlez> dvl: so 4 IP addresses for that jail and the 199.xxx is the first one? [Oct 08 11:39] <@dvl> battlez: Then why does changing the order on the host get me a working route? [Oct 08 11:39] <@dvl> battlez: yes. [Oct 08 11:39] <@dvl> battlez: and in jail.conf I have 199.xxx, 199.xxx, 10.xxx, 10.xxx [Oct 08 11:40] <@battlez> dvl: does this machine have a connected 10.x.x.x network on ne interface and 199.x.x.x on another interface or are they both on the same interface? [Oct 08 11:40] <@dvl> battlez: both on same nic [Oct 08 11:41] <@dvl> FYI, This particular 10.70.x.x network is local to this machine, and is for the jails only. Kind of a back channel way for me to admin them. [Oct 08 11:41] <@dvl> I suppose I could clone a nic and add such addresses to it. [Oct 08 11:41] <@battlez> can you show me the utput line of `netstat -f inet -rn | grep default`? [Oct 08 11:42] <@dvl> default 199.x.x.x UGS em1 [Oct 08 11:43] <@dvl> battlez: and in my notes, 199.x.x.x is definintely my gateway. [Oct 08 11:44] <@battlez> dvl: what does sysctl security.jail.param.ip4.saddrsel think? [Oct 08 11:45] <@dvl> battlez: security.jail.param.ip4.saddrsel: 0 [Oct 08 11:46] <@battlez> dvl: and I guess you do not set ip4.saddrsel in jail.conf for any jails either? [Oct 08 11:48] <@dvl> battlez: correct. Also, this system, FWIW, is runnging 4x jails via iocage, and 1 via jail.conf [Oct 08 11:48] <@dvl> battlez: The host is 10.2 and the jail we are discussing is 8.2 (please, don't ask... :) [Oct 08 11:49] <@battlez> dvl: only the 10.2 kernel matters. [Oct 08 11:49] <@battlez> dvl: from all you tell me, there is a bug somewhere and someone broke the source address selection :-( [Oct 08 11:50] <@battlez> dvl: with the current setting you will not use the primary jail IP and it should do proper source address selection which means it should only pick a 10.x IP if you connect to that specific subnet or you have a route to the destination via a 10.x gateway; for everythign else your default route should find the connected subnet address of that gateway which is 199.xxxx assuming all 199.xxx is n the proper same subnet as your default gateway is [Oct 08 11:52] <@battlez> dvl: if you want the primary jail IP as a fallback flip that switch either using the jail command or jail.conf and rather than source address selection it'll bypass and force the first jail address for all unbound outgoing connections, which also means that cnnections to 10.x will be sourced from 199.x unless explicitly specified otherwise [Oct 08 11:53] <@battlez> dvl: does that make some sense to you? The behavour you seem to experience is not the behaviour I once implemented :-)
Another thing that made me worried after thinking more about this is, whether IPv4 source address selection on a simple plain (base) system still does the right thing, as it goes through the same code path.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/doc/commit/?id=612b7cc1721224c494c5b2600188e1508bb5611b commit 612b7cc1721224c494c5b2600188e1508bb5611b Author: Sergio Carlavilla Delgado <carlavilla@FreeBSD.org> AuthorDate: 2023-09-20 08:15:40 +0000 Commit: Sergio Carlavilla Delgado <carlavilla@FreeBSD.org> CommitDate: 2023-09-20 08:15:40 +0000 Handbook - Jails: Rewrite the chapter Changes: - Add jail types info: Thick, Thin, VNET and Linux Jails - Add info about the necessary configuration on the host system - Add info about types of networking in the jails - Explain the jail config file - Add info about how to configure Thick, Thin, VNET and Linux Jails - Explain how to upgrade the different types of jails - Add info about the resources limits in a Jail - Add info about different jail managers and containers Information obtained from: - https://man.freebsd.org/cgi/man.cgi?query=jail&sektion=8 - https://wiki.freebsd.org/Jails - https://man.freebsd.org/cgi/man.cgi?rctl(8) - https://vermaden.wordpress.com/2023/06/28/freebsd-jails-containers/ - https://clinta.github.io/freebsd-jails-the-hard-way/ - https://jacob.ludriks.com/2017/06/07/FreeBSD-Thin-Jails/ - Jails and VNET a guide - Derik J. Ramirez - https://wb-hk.blogspot.com/2016/04/freebsd-jails-4-thin-jails-using-nullfs.html - https://blog.uidrafter.com/freebsd-jails-network-setup - https://forums.freebsd.org/threads/setting-up-a-debian-linux-jail-on-freebsd.68434/ - https://github.com/msimerson/Mail-Toaster-6/wiki/Linux-Jails - https://etherealwake.com/2021/08/freebsd-jail-networking/ - https://wiki.freebsd.org/TomMarcoen/JailNetworking - https://weblog.antranigv.am/posts/2020/06/vnet-jail-howto/ PR: 203641, 200905, 209157, 248150, 263330, 264317, 226282, 239861, 166358, 178221, 265410 Reviewed by: karels@, fernape@, jrm@ Differential Revision: https://reviews.freebsd.org/D41450 .../content/en/books/handbook/jails/_index.adoc | 1492 ++++++++++---------- 1 file changed, 715 insertions(+), 777 deletions(-)
I'm gonna mark this as overcome by events since the chapter has been rewritten in: https://cgit.freebsd.org/doc/commit/?id=612b7cc1