Bug 168678

Summary: [jail] raw sockets incorrectly choose source address when jail has multiple subnets
Product: Base System Reporter: Mark Felder <feld>
Component: kernAssignee: Bjoern A. Zeeb <bz>
Status: Open ---    
Severity: Affects Only Me CC: bz, dch, info, ports, thy295, wblock
Priority: Normal    
Version: Unspecified   
Hardware: Any   
OS: Any   

Description Mark Felder freebsd_committer freebsd_triage 2012-06-04 19:50:17 UTC
bz@ is aware of this bug but I figured it's best if I make a proper PR before I forget about the issue.

Scenario: 

FreeBSD server with multiple subnets/VLANs across multiple interfaces. You want to map several of these into a single jail. This works correctly; the jail starts fine. Outgoing TCP/UDP traffic works, but anything that requires a raw socket (notably ICMP) will fail to any subnet that falls outside of the first IP address's range.

Fix: 

Bug bz@ until he fixes it :-)

He knows what's wrong, and will take care of it after some important ipv6 work is completed.
How-To-Repeat: Create jail which has following IPs:

192.168.1.10
172.16.5.10
10.1.4.10

You can ping anything on the 192.168.1.0/24 subnet just fine, but you can't on the 172. or 10. subnets. Sniffing shows all source traffic to the 172. and 10. subnets incorrectly using 192.168.1.10 as the source address.
Comment 1 Gavin Atkinson freebsd_committer freebsd_triage 2012-06-18 15:16:53 UTC
Responsible Changed
From-To: freebsd-bugs->freebsd-jail

Fix assignment per submitter request
Comment 2 Bjoern A. Zeeb freebsd_committer freebsd_triage 2012-06-19 13:33:35 UTC
Responsible Changed
From-To: freebsd-jail->bz

bz feels the bugs.  The ping patch has been living in my browser 
window for weeks now; I should get the kernel fix done as well. 

http://people.freebsd.org/~bz/20120407-01-ping-source-addr.diff 

The problem is described here: 
http://svnweb.freebsd.org/base/head/sys/netinet/raw_ip.c?annotate=229265#l461
Comment 3 Bjoern A. Zeeb freebsd_committer freebsd_triage 2014-05-18 05:59:04 UTC
Responsible Changed
From-To: bz->gnn

I shall not use bugzilla (at least until we will have a CLI).
Comment 4 Warren Block freebsd_committer freebsd_triage 2014-08-06 01:41:08 UTC
This also affects jails on machines with more than one network interface and gateway_enable="YES".  If the jail is given a unique loopback interface, pings fail with "Can't assign requested address".

ezjail-admin create examplejail "lo1|127.0.1.2,fxp0|192.168.1.101"
sed -i'' -e 's/parameters=""/parameters="allow.raw_sockets=1"/' /usr/local/etc/ezjail/examplejail
ezjail-admin start examplejail
ezjail-admin console examplejail
# ping 192.168.1.101 (jail address)
PING 192.168.1.101 (192.168.1.101): 56 data bytes
64 bytes from 192.168.1.101: icmp_seq=0 ttl=64 time=0.122 ms
^C
--- 192.168.1.101 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
# ping 8.8.178.110 # (freebsd.org)
PING 8.8.178.110 (8.8.178.110): 56 data bytes
ping: sendto: Can't assign requested address
ping: sendto: Can't assign requested address
^C
2 packets transmitted, 0 packets received, 100.0% packet loss

If the jail uses the host's loopback, it works.  Change /usr/local/etc/ezjail/examplejail from
export jail_examplejail_ip="lo1|127.0.1.2,fxp0|192.168.1.101"
export jail_examplejail_ip="fxp0|192.168.1.101"

Or set gateway_enable="NO".
Comment 5 Warren Block freebsd_committer freebsd_triage 2014-08-06 01:48:41 UTC
My tests were on 10-STABLE r269604 amd64.
Comment 6 Bjoern A. Zeeb freebsd_committer freebsd_triage 2016-01-07 21:42:04 UTC
Yes I should really fix this (I am sure I did) but it might have never made it into the kernel.
Comment 7 Sascha Biberhofer 2016-05-14 13:06:20 UTC
This is still an issue on 10.3-RELEASE and doesn't seem to be limited to raw sockets.

More specifically, if I try to assign both a global IPv4 as well as an address in the RFC 1918 range (both on the same interface), then the first assigned address gets chosen regardless of the target network. 

In particular, the jail will either only use the local address for outgoing connections (if the local address was assigned first) or will fail to connect to addresses in the local network (if the global address was assigned first), e.g. for resolving DNS queries via a local DNS resolver or delivering emails to an internal host. This kind of breaks most setups that rely on internal services. :/
Comment 8 Eitan Adler freebsd_committer freebsd_triage 2018-05-28 19:42:41 UTC
batch change:

For bugs that match the following
-  Status Is In progress 
AND
- Untouched since 2018-01-01.
AND
- Affects Base System OR Documentation

DO:

Reset to open status.


Note:
I did a quick pass but if you are getting this email it might be worthwhile to double check to see if this bug ought to be closed.
Comment 9 Mark Felder freebsd_committer freebsd_triage 2019-11-13 19:07:54 UTC
This is one of my favorite jail bugs and I still think about it a few times a year.

Big thanks for help diagnosing it that night back in 2012. I think this rarely hits anyone, but would love someone to confirm it's finally fixed.

Maybe a christmas present this year? :-)