Bug 255264
| Summary: | Unable to add default route for fib | ||
|---|---|---|---|
| Product: | Base System | Reporter: | Neal Nelson <ports> |
| Component: | bin | Assignee: | freebsd-net (Nobody) <net> |
| Status: | Closed Works As Intended | ||
| Severity: | Affects Only Me | CC: | dan, lwhsu, melifaro, pat, pinkfluid, ports, zarychtam |
| Priority: | --- | Flags: | koobs:
maintainer-feedback?
(ports) |
| Version: | 13.0-RELEASE | ||
| Hardware: | Any | ||
| OS: | Any | ||
|
Description
Neal Nelson
2021-04-20 14:24:44 UTC
The default behaviour has changed. Now in dmesg we get: "WARNING: Adding ifaddrs to all fibs has been turned off by default. Consider tuning net.add_addr_allfibs if needed" Please add globally the sysctl net.add_addr_allfibs=1 to revert to the old behaviour. You can also solve this by assigning fib 1 to the interface which is supposed to carry this route or by installing the network route for this interface in fib 1. Thanks for the information. I saw the message upon boot, but it looks to be a large sweeping setting which made me nervous. Also I wasn't sure if it referred to this situation precisely as frankly I don't really understand what it's saying. After setting this sysctl, all seems to be working as it was before. Thanks for the very rapid response (unlike on the forum). (In reply to Neal Nelson from comment #2) Hi Neal, you mentioned that the message wasn't helpful. Do you by any chance have any suggestions/comments on how can the message be improved to provide more context? Thank you! ^Triage: Re-open pending potential messaging improvement, request feedback from reporter The entry in /usr/src/UPDATING that has been pointed out to me explains the situation a little better than the current message. I'm sure the current message makes sense to those that know the ins and outs, but not those of us not quite so well versed in the intricacies of network routing: Default value of net.add_addr_allfibs has been changed to 0. If you have multi-fib configuration and rely on existence of all interface routes in every fib, you need to set the above sysctl to 1. It's a bit wordy, but more helpful. If it's too wordy, a reference to the entry in UPDATING could be made. As a further suggestion; it might also be useful to add the entries from UPDATING to the release notes. I found the entry there stating the change, but it was of no use to me whatsoever as it didn't state the consequences of the change. I realise that the release notes are rather long already, but I'm sure that some method could be devised to auto include UPDATING entries. I'm running 13.1-RC3 and it looks like I'm experiencing exactly the same problem, except that the "net.add_addr_allfibs=1" did not fix my issue. My problem seems to be somehow related to the interface carrier. What I'm experiencing is that adding a default route to FIB 1 fails until carrier is detected on the interface (configuring the route via rc.conf failed, while adding it manually over SSH always succeeded). What I'm trying to do is to add an alternate default route that would go through 192.168.1.1 on re0. The default gateway for FIB 0 is on a VLAN interface (the XX.XXX.192.0 network below, hidden because it's a public IP), which has no issues. I did confirm this behavior by using the following rc.local file: #!/bin/sh set -x while ! setfib 1 netstat -rn | grep -q default do setfib 1 netstat -rn4 ifconfig re0 | grep -E 're0|inet|media:' setfib 1 route add default 192.168.1.1 -fib 1 sleep 0.5 done It basically tries to set the default route in a loop until it succeeds while showing the re0 interface status and routes in FIB 1. This is an abridged output of the script above, but it shows that the `setfib 1 route add default 192.168.1.1` command fails until carrier is detected on the interface. Apr 14 15:21:14 marin kernel: Internet: Apr 14 15:21:14 marin kernel: Destination Gateway Flags Netif Expire Apr 14 15:21:14 marin kernel: XX.XXX.192.0/18 link#4 U vlan2 Apr 14 15:21:14 marin kernel: 127.0.0.1 link#2 UH lo0 Apr 14 15:21:14 marin kernel: 127.1.0.1 link#3 UH lo1 Apr 14 15:21:14 marin kernel: 127.1.0.2 link#3 UH lo1 Apr 14 15:21:14 marin kernel: 192.168.1.0/24 link#1 U re0 Apr 14 15:21:14 marin kernel: 192.168.10.0/24 link#5 U wg0 Apr 14 15:21:14 marin kernel: + ifconfig re0 Apr 14 15:21:14 marin kernel: + grep -E 're0|inet|media:' Apr 14 15:21:14 marin kernel: re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 Apr 14 15:21:14 marin kernel: inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255 Apr 14 15:21:14 marin kernel: media: Ethernet autoselect (none) Apr 14 15:21:14 marin kernel: + setfib 1 route add default 192.168.1.1 -fib 1 Apr 14 15:21:14 marin kernel: route: writing to routing socket: Network is unreachable Apr 14 15:21:14 marin kernel: add net default: gateway 192.168.1.1 fib 1: Network is unreachable Apr 14 15:21:14 marin kernel: + sleep 0.5 Apr 14 15:21:14 marin kernel: + setfib 1 netstat -rn Apr 14 15:21:14 marin kernel: + grep -q default Apr 14 15:21:14 marin kernel: + setfib 1 netstat -rn4 Apr 14 15:21:14 marin kernel: Routing tables (fib: 1) Apr 14 15:21:14 marin kernel: Apr 14 15:21:14 marin kernel: Internet: Apr 14 15:21:14 marin kernel: Destination Gateway Flags Netif Expire Apr 14 15:21:14 marin kernel: XX.XXX.192.0/18 link#4 U vlan2 Apr 14 15:21:14 marin kernel: 127.0.0.1 link#2 UH lo0 Apr 14 15:21:14 marin kernel: 127.1.0.1 link#3 UH lo1 Apr 14 15:21:14 marin kernel: 127.1.0.2 link#3 UH lo1 Apr 14 15:21:14 marin kernel: 192.168.1.0/24 link#1 U re0 Apr 14 15:21:14 marin kernel: 192.168.10.0/24 link#5 U wg0 Apr 14 15:21:14 marin kernel: + ifconfig re0 Apr 14 15:21:14 marin kernel: + grep -E 're0|inet|media:' Apr 14 15:21:14 marin kernel: re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 Apr 14 15:21:14 marin kernel: inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255 Apr 14 15:21:14 marin kernel: media: Ethernet autoselect (none) Apr 14 15:21:14 marin kernel: + setfib 1 route add default 192.168.1.1 -fib 1 Apr 14 15:21:14 marin kernel: route: writing to routing socket: Network is unreachable Apr 14 15:21:14 marin kernel: add net default: gateway 192.168.1.1 fib 1: Network is unreachable Apr 14 15:21:14 marin kernel: + sleep 0.5 Apr 14 15:21:15 marin kernel: + setfib 1 netstat -rn Apr 14 15:21:15 marin kernel: + grep -q default Apr 14 15:21:15 marin kernel: + setfib 1 netstat -rn4 Apr 14 15:21:15 marin kernel: Routing tables (fib: 1) Apr 14 15:21:15 marin kernel: Apr 14 15:21:15 marin kernel: Internet: Apr 14 15:21:15 marin kernel: Destination Gateway Flags Netif Expire Apr 14 15:21:15 marin kernel: XX.XXX.192.0/18 link#4 U vlan2 Apr 14 15:21:15 marin kernel: 127.0.0.1 link#2 UH lo0 Apr 14 15:21:15 marin kernel: 127.1.0.1 link#3 UH lo1 Apr 14 15:21:15 marin kernel: 127.1.0.2 link#3 UH lo1 Apr 14 15:21:15 marin kernel: 192.168.1.0/24 link#1 U re0 Apr 14 15:21:15 marin kernel: 192.168.10.0/24 link#5 U wg0 Apr 14 15:21:15 marin kernel: + ifconfig re0 Apr 14 15:21:15 marin kernel: + grep -E 're0|inet|media:' Apr 14 15:21:15 marin kernel: re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 Apr 14 15:21:15 marin kernel: inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255 Apr 14 15:21:15 marin kernel: media: Ethernet autoselect (none) Apr 14 15:21:15 marin kernel: + setfib 1 route add default 192.168.1.1 -fib 1 Apr 14 15:21:15 marin kernel: route: writing to routing socket: Network is unreachable Apr 14 15:21:15 marin kernel: add net default: gateway 192.168.1.1 fib 1: Network is unreachable Apr 14 15:21:15 marin kernel: + sleep 0.5 Apr 14 15:21:15 marin kernel: + setfib 1 netstat -rn Apr 14 15:21:15 marin kernel: + grep -q default Apr 14 15:21:15 marin kernel: + setfib 1 netstat -rn4 Apr 14 15:21:15 marin kernel: Routing tables (fib: 1) Apr 14 15:21:15 marin kernel: Apr 14 15:21:15 marin kernel: Internet: Apr 14 15:21:15 marin kernel: Destination Gateway Flags Netif Expire Apr 14 15:21:15 marin kernel: XX.XXX.192.0/18 link#4 U vlan2 Apr 14 15:21:15 marin kernel: 127.0.0.1 link#2 UH lo0 Apr 14 15:21:15 marin kernel: 127.1.0.1 link#3 UH lo1 Apr 14 15:21:15 marin kernel: 127.1.0.2 link#3 UH lo1 Apr 14 15:21:15 marin kernel: 192.168.1.0/24 link#1 U re0 Apr 14 15:21:15 marin kernel: 192.168.10.0/24 link#5 U wg0 Apr 14 15:21:15 marin kernel: + ifconfig re0 Apr 14 15:21:15 marin kernel: + grep -E 're0|inet|media:' Apr 14 15:21:15 marin kernel: re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 Apr 14 15:21:15 marin kernel: inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255 Apr 14 15:21:15 marin kernel: media: Ethernet autoselect (none) Apr 14 15:21:15 marin kernel: + setfib 1 route add default 192.168.1.1 -fib 1 Apr 14 15:21:15 marin kernel: route: writing to routing socket: Network is unreachable Apr 14 15:21:15 marin kernel: add net default: gateway 192.168.1.1 fib 1: Network is unreachable Apr 14 15:21:15 marin kernel: + sleep 0.5 Apr 14 15:21:16 marin kernel: + setfib 1 netstat -rn Apr 14 15:21:16 marin kernel: + grep -q default Apr 14 15:21:16 marin kernel: + setfib 1 netstat -rn4 Apr 14 15:21:16 marin kernel: Routing tables (fib: 1) Apr 14 15:21:16 marin kernel: Apr 14 15:21:16 marin kernel: Internet: Apr 14 15:21:16 marin kernel: Destination Gateway Flags Netif Expire Apr 14 15:21:16 marin kernel: XX.XXX.192.0/18 link#4 U vlan2 Apr 14 15:21:16 marin kernel: 127.0.0.1 link#2 UH lo0 Apr 14 15:21:16 marin kernel: 127.1.0.1 link#3 UH lo1 Apr 14 15:21:16 marin kernel: 127.1.0.2 link#3 UH lo1 Apr 14 15:21:16 marin kernel: 192.168.1.0/24 link#1 U re0 Apr 14 15:21:16 marin kernel: 192.168.10.0/24 link#5 U wg0 Apr 14 15:21:16 marin kernel: + ifconfig re0 Apr 14 15:21:16 marin kernel: + grep -E 're0|inet|media:' Apr 14 15:21:16 marin kernel: re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 Apr 14 15:21:16 marin kernel: inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255 Apr 14 15:21:16 marin kernel: media: Ethernet autoselect (1000baseT <full-duplex>) Apr 14 15:21:16 marin kernel: + setfib 1 route add default 192.168.1.1 -fib 1 Apr 14 15:21:16 marin kernel: add net default: gateway 192.168.1.1 fib 1 Apr 14 15:21:16 marin kernel: + sleep 0.5 Apr 14 15:21:16 marin kernel: + setfib 1 netstat -rn Apr 14 15:21:16 marin kernel: + grep -q default Apr 14 15:21:16 marin kernel: + echo . Apr 14 15:21:16 marin kernel: . Apr 14 15:21:16 marin kernel: + _return=0 Apr 14 15:21:16 marin kernel: + [ 0 -ne 0 ] Apr 14 15:21:16 marin kernel: + return 0 Apr 14 15:21:16 marin kernel: + _run_rc_postcmd Apr 14 15:21:16 marin kernel: + [ -n '' ] Apr 14 15:21:16 marin kernel: + return 0 Apr 14 15:21:16 marin syslogd: last message repeated 1 times I have set the following sysctls: - net.add_addr_allfibs=1 - net.fibs=2 What I have tried so far without any luck: - disabling pf - moving net.fibs=2 from /etc/sysctl.conf to /boot/loader.conf - using netwait (it's started after routing) The script above certainly works but I would like to understand what's the core issue behind this and at this point I'm out of ideas. Any hint or help would be really appreciated. (In reply to Mitja Horvat from comment #6) To succeed early (before the link goes up) you need to assign re0 to the correct fib: ifconfig_re0="inet 192.168.1.2 netmask 255.255.255.0 fib 1" I don't know if is it a bug or feature change of the new routing stack, but I am biased toward the latter. (In reply to Marek Zarychta from comment #7) Thank you very much for the prompt response. Indeed, setting the fib of the re0 interface 1 makes the route work. However this causes all the traffic routed through re0 to use fib 1 as default, which is not really what I want. The use case I'm trying to solve is to have a mix of processes/jail running on fib 1 using setfib, everything else should go through the interface normally using fib 0. Even if it's a feature, I find it very strange that a valid static route would fail if there's no carrier. I even tried unplugging the ethernet cable and the script above basically loops until I connect the cable and carrier is detected. I think I'll play with pf to get what I need (or just reset re0's fib to 0 in rc.local), but I still find this to be a very peculiar behavior. (In reply to Mitja Horvat from comment #8) fwiw I’ve found fib to be tedious compared to vnet when it comes to configuring jails. I don’t know if vnet is available in your setup though. I find that vnet provides straightforward networking and routing, as opposed to the multiple-routes tables approach using fib. (In reply to Mitja Horvat from comment #8) If changing the default fib of re0 is inconvenient, then please try this: "route add default 192.168.1.1 -ifa 192.168.1.2 -fib 1" You can eventually add static route to rc.conf static_routes="default_fib1 ..." route_default_fib1="default 192.168.1.1 -ifa 192.168.1.2 -fib 1" (In reply to Marek Zarychta from comment #10) Marek, that worked splendidly. I could even leave net.add_addr_allfibs=0. rc.conf: static_routes="fib1_lan fib1_default" route_fib1_lan="-fib 1 -net 192.168.1.0/24 -interface re0" route_fib1_default="-fib 1 default -ifa 192.168.1.2 192.168.1.1" The console log show there was no issues when adding the route: Apr 14 21:39:56 marin kernel: /sbin/route add -fib 1 default -ifa 192.168.1.2 192.168.1.1 Apr 14 21:39:56 marin kernel: add net default: gateway 192.168.1.1 fib 1 And the final result is this: root@marin:/usr/home/mitja # netstat -rn4 -F1 Routing tables (fib: 1) Internet: Destination Gateway Flags Netif Expire default 192.168.1.1 UGS re0 127.0.0.1 link#2 UHS lo0 192.168.1.0/24 link#1 US re0 Perfect, thank you! (In reply to Pat Maddox from comment #9) Pat, thanks for the suggestion. If I didn't find a satisfactory workaround (which it seems I did), vnets were definitely first on the list. I assume most people have switched from openvpn to wireguard by now, you'll have to set up fib directly in wg0.conf whenever you do. As far as vtnet's go, I saw there is a problem with vnet's in same sense as the ongoing ALTQ for PF where you don't get full bandwidth with parallelization across all CPU's with it, although ALTQ should get in FreeBSD 14, I'm not sure if James Skon will get around to vlans or not unless he's shown interest. https://www.youtube.com/watch?v=OD7Jop6aGGk&t=11321s |
