uname -a; FreeBSD lrrr.guld.sen 12.0-RELEASE FreeBSD 12.0-RELEASE r341666 GENERIC amd64 uname -U; 1200086 uname -K; 1200086 lagg0; laggproto lacp lagghash l2,l3,l4 laggport: em0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> groups: lagg fib0 contains Internet: Destination Gateway Flags Netif Expire default 192.168.23.1 UGS lagg0 192.168.23.0/24 link#4 U lagg0 192.168.23.124 link#4 UHS lo0 fib1 contains Internet: Destination Gateway Flags Netif Expire default 192.168.22.1 UGS lagg0.200 192.168.22.0/24 link#17 U lagg0.200 192.168.22.13 tt:uu:vv:xx:yy:zz UHS lagg0.200 When attempting communication in fib 1 with remote hosts within the same routing domain , i.e. 192.168.22.1 it fails and the following logentry is written: Dec 13 21:00:03 lrrr kernel: arpresolve: can't allocate llinfo for 192.168.22.1 on lagg0.200 When adding the following entry in fib0 the communication works. setfib 0 route add -net 192.168.22.0/24 -iface lagg0.200 In addition, when adding a static ARP entry the communication works. setfib 1 arp -s 192.168.22.1 ss:uu:vv:xx:yy:zz It seems to me that ARP should work in either fib regardless of fib0 having the network in question. This issue seems to have existed before according to https://lists.freebsd.org/pipermail/freebsd-net/2012-May/032340.html
I have, since I filed the PR, abandoned the idea of having certain jails use an alternate routing table (FIB) and instead moved to using VIMAGE/VNET. The PR should, however, be reproducible on any system running the filed setup.
(In reply to Patrik Hildingsson from comment #1) Could you please also post here the value of "sysctl net.add_addr_allfibs" setting?
(In reply to Marek Zarychta from comment #2) /boot/loader.conf is set to net.add_addr_allfibs=0
What command did you use when you attempted to communicate from fib 1? If you didn't somehow change the process's fib to 1 (like with setfib(1)), then the command would use fib 0.
(In reply to Alan Somers from comment #4) I associated a jail to fib 1, running a java app inside of the jail. Several other programs, such as telnet, were executed inside of the jail. It is my understanding that once you are inside the jail, i.e. running a shell inside the jail fib 1 would be used regardless of running setfib fibno. prior to all commands. Executing traceroute inside the jail showed that traceroute used the correct default route in fib 1.
The surest way to tell if you set the fib correctly is to do "ps -ax -O fib -O jid".
(In reply to Patrik Hildingsson from comment #3) If net.add_addr_allfibs is set to 0, then IMHO everything works as intended. You have only to imprison lagg0.200 interface with the command "ifconfig lagg0.200 fib 1" and set appropriate routes for fib 1. You can, of course, add some routes with -iface option to reduce the impact of net.add_addr_allfibs=0 setting. I can't confirm any issues with IPv4/6 routing and arp/ndp resolution running fib based jails on 12.0-STABLE with net.add_addr_allfibs=0 set.
(In reply to Marek Zarychta from comment #7) Imprisoning the interface to a certain fib is probably what I was lacking in the first place. I will test it on a test system later this month. Please go ahead and close this PR meanwhile. Thank you all for engaging yourselves in the matter.