Bug 203641 - order of IPs in jails depends on order on host
Summary: order of IPs in jails depends on order on host
Status: Closed Overcome By Events
Alias: None
Product: Documentation
Classification: Unclassified
Component: Books & Articles (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Many People
Assignee: Sergio Carlavilla Delgado
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-10-08 15:31 UTC by Dan Langille
Modified: 2023-09-20 09:00 UTC (History)
5 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Dan Langille freebsd_committer freebsd_triage 2015-10-08 15:31:20 UTC
If you want a specific primary IP in a jail, add that IP on the host before any other IP used by that jail.


Allan Jude and I spoke about this on IRC and he asked that I create a PR for it.
Comment 1 Bjoern A. Zeeb freebsd_committer freebsd_triage 2015-10-08 15:35:27 UTC
I am totally confused by this report.  It does not make sense to me.

You can have a primary jail IP and only add it to the host after you started the jail and it will then do the right thing as described in the man page.

What is the real problem?
Comment 2 Dan Langille freebsd_committer freebsd_triage 2015-10-08 16:27:26 UTC
From IRC, posted with consent of bz@:

[Oct 08 11:37]  <@battlez>	dvl: the real question is, which order does jls -av or similar list the addresses when you run this from the base system?
[Oct 08 11:37]  <@battlez>	ifconfig is "random order" and desn't matter
[Oct 08 11:38]  <@dvl>	battlez: that is interesting.  jls -av lists: 199.xxx 10.xxx 10.xxx 199.xxx
[Oct 08 11:38]  <@battlez>	dvl: and the theory that the jail does depend on the order in "ifconfig" on the host is entirely worng as well unfortunately;  it might depend on what route -get <destination> thinks though
[Oct 08 11:39]  <@battlez>	dvl: so 4 IP addresses for that jail and the 199.xxx is the first one?
[Oct 08 11:39]  <@dvl>	battlez: Then why does changing the order on the host get me a working route?
[Oct 08 11:39]  <@dvl>	battlez: yes.
[Oct 08 11:39]  <@dvl>	battlez: and in jail.conf I have 199.xxx, 199.xxx, 10.xxx, 10.xxx
[Oct 08 11:40]  <@battlez>	dvl: does this machine have a connected 10.x.x.x network on ne interface and 199.x.x.x on another interface or are they both on the same interface?
[Oct 08 11:40]  <@dvl>	battlez: both on same nic
[Oct 08 11:41]  <@dvl>	FYI, This particular 10.70.x.x network is local to this machine, and is for the jails only.  Kind of a back channel way  for me to admin them.  
[Oct 08 11:41]  <@dvl>	I suppose I could clone a nic and add such addresses to it.
[Oct 08 11:41]  <@battlez>	can you show me the utput line of `netstat -f inet -rn  | grep default`?
[Oct 08 11:42]  <@dvl>	default            199.x.x.x    UGS         em1
[Oct 08 11:43]  <@dvl>	battlez: and in my notes, 199.x.x.x is definintely my gateway.
[Oct 08 11:44]  <@battlez>	dvl: what does sysctl security.jail.param.ip4.saddrsel think?
[Oct 08 11:45]  <@dvl>	battlez: security.jail.param.ip4.saddrsel: 0
[Oct 08 11:46]  <@battlez>	dvl: and I guess you do not set ip4.saddrsel in jail.conf for any jails either?
[Oct 08 11:48]  <@dvl>	battlez: correct.  Also, this system, FWIW, is runnging 4x jails via iocage, and 1 via jail.conf
[Oct 08 11:48]  <@dvl>	battlez: The host is 10.2 and the jail we are discussing is 8.2 (please, don't ask... :)
[Oct 08 11:49]  <@battlez>	dvl:  only the 10.2 kernel matters.
[Oct 08 11:49]  <@battlez>	dvl: from all you tell me, there is a bug somewhere and someone broke the source address selection :-(
[Oct 08 11:50]  <@battlez>	dvl: with the current setting you will not use the primary jail IP and it should do proper source address selection which means it should only pick a 10.x IP if you connect to that specific subnet or you have a route to the destination via a 10.x gateway;  for everythign else your default route should find the connected subnet address of that gateway which is 199.xxxx assuming all 199.xxx is n the proper same subnet as your default gateway is
[Oct 08 11:52]  <@battlez>	dvl: if you want the primary jail IP as a fallback flip that switch either using the jail command or jail.conf and rather than source address selection it'll bypass and force the first jail address for all unbound outgoing connections, which also means that cnnections to 10.x will be sourced from 199.x unless explicitly specified otherwise
[Oct 08 11:53]  <@battlez>	dvl: does that make some sense to you?   The behavour you seem to experience is not the behaviour I once implemented :-)
Comment 3 Bjoern A. Zeeb freebsd_committer freebsd_triage 2015-10-08 17:43:07 UTC
Another thing that made me worried after thinking more about this is, whether IPv4 source address selection on a simple plain (base) system still does the right thing, as it goes through the same code path.
Comment 4 commit-hook freebsd_committer freebsd_triage 2023-09-20 08:34:04 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/doc/commit/?id=612b7cc1721224c494c5b2600188e1508bb5611b

commit 612b7cc1721224c494c5b2600188e1508bb5611b
Author:     Sergio Carlavilla Delgado <carlavilla@FreeBSD.org>
AuthorDate: 2023-09-20 08:15:40 +0000
Commit:     Sergio Carlavilla Delgado <carlavilla@FreeBSD.org>
CommitDate: 2023-09-20 08:15:40 +0000

    Handbook - Jails: Rewrite the chapter

    Changes:
    - Add jail types info: Thick, Thin, VNET and Linux Jails
    - Add info about the necessary configuration on the host system
    - Add info about types of networking in the jails
    - Explain the jail config file
    - Add info about how to configure Thick, Thin, VNET and Linux Jails
    - Explain how to upgrade the different types of jails
    - Add info about the resources limits in a Jail
    - Add info about different jail managers and containers

    Information obtained from:
    - https://man.freebsd.org/cgi/man.cgi?query=jail&sektion=8
    - https://wiki.freebsd.org/Jails
    - https://man.freebsd.org/cgi/man.cgi?rctl(8)
    - https://vermaden.wordpress.com/2023/06/28/freebsd-jails-containers/
    - https://clinta.github.io/freebsd-jails-the-hard-way/
    - https://jacob.ludriks.com/2017/06/07/FreeBSD-Thin-Jails/
    - Jails and VNET a guide - Derik J. Ramirez
    - https://wb-hk.blogspot.com/2016/04/freebsd-jails-4-thin-jails-using-nullfs.html
    - https://blog.uidrafter.com/freebsd-jails-network-setup
    - https://forums.freebsd.org/threads/setting-up-a-debian-linux-jail-on-freebsd.68434/
    - https://github.com/msimerson/Mail-Toaster-6/wiki/Linux-Jails
    - https://etherealwake.com/2021/08/freebsd-jail-networking/
    - https://wiki.freebsd.org/TomMarcoen/JailNetworking
    - https://weblog.antranigv.am/posts/2020/06/vnet-jail-howto/

    PR:                     203641, 200905, 209157, 248150, 263330, 264317,
                            226282, 239861, 166358, 178221, 265410
    Reviewed by:            karels@, fernape@, jrm@
    Differential Revision:  https://reviews.freebsd.org/D41450

 .../content/en/books/handbook/jails/_index.adoc    | 1492 ++++++++++----------
 1 file changed, 715 insertions(+), 777 deletions(-)
Comment 5 Sergio Carlavilla Delgado freebsd_committer freebsd_triage 2023-09-20 09:00:55 UTC
I'm gonna mark this as overcome by events since the chapter has been rewritten
in: https://cgit.freebsd.org/doc/commit/?id=612b7cc1