When a route is set, the interface corresponding to the destination hop is looked up among all existing interfaces, not only those that are marked up, causing the wrong interface to be selected in some cases. Fix: Fix not determined yet. This problem can be worked around by changing the addresses on the down interface so it won't clash with those of the up one: # ifconfig gif0 192.168.99.1 192.168.99.2 # ifconfig gif0 down # route delete 192.168.1.0 192.168.0.1 # route add 192.168.1.0 192.168.0.1 # netstat -rn | grep 192.168 192.168.0.1 192.168.0.253 UH 1 0 gif1 192.168.1 192.168.0.1 UGS 0 0 gif1 How-To-Repeat: This issue can be demonstrated using gif interfaces: # ifconfig gif0 create # ifconfig gif0 192.168.0.254 192.168.0.1 # ifconfig gif0 down # ifconfig gif1 create # ifconfig gif1 192.168.0.253 192.168.0.1 Note at this point that 192.168.0.1 is reachable through interface gif1. # route add 192.168.1.0 192.168.0.1 # netstat -rn | grep 192.168 192.168.0.1 192.168.0.253 UH 1 0 gif1 192.168.1 192.168.0.1 UGS 0 0 gif0 Note then that, even though 192.168.0.1 is marked as reachable through gif1, for the purpose of routing to 192.168.1.0 the selected interface is gif0 (which is down).
Just tell it the interface you really want: route add -net 192.168.1 -iface gif1 Cheers, -- Ruslan Ermilov ru@FreeBSD.org FreeBSD committer
On Wed, Sep 08, 2004 at 12:09:39PM +0200, Thomas Quinot wrote: > * Ruslan Ermilov, 2004-09-08 : > > > This is *not* how I spelled it. Try the above as shown. > > I stand corrected. The command as proposed does work. Unfortunately this > applies only to point-to-point interfaces, not in the case where a > specific next-hop address is required. > But your test case only applies to P2P interfaces, when both ends of two P2P interfaces share the same IP address. I'd call this a misconfiguration. I do not see how this could apply to broadcast interfaces. Cheers, -- Ruslan Ermilov ru@FreeBSD.org FreeBSD committer
On Wed, Sep 08, 2004 at 02:26:29PM +0200, Thomas Quinot wrote: > * Ruslan Ermilov, 2004-09-08 : > > > But your test case only applies to P2P interfaces, when both ends > > of two P2P interfaces share the same IP address. > > No, this is not the case. Only the remote ends of the two interfaces > share the same IP address. I described that test case because using gif > interfaces makes it easier to reproduce, but I initially observed this > symptom with one bge and one Intel wireless (ndis) interface. > That's what I meant (two remote ends of both interfaces). > > I do not see how this could apply to broadcast interfaces. > > As follows. Consider a machine with two Ethernet interfaces I1 and I2. > > ifconfig I1 192.168.0.1 netmask 255.255.255.0 > ifconfig I1 down > ifconfig I2 192.168.0.2 netmask 255.255.255.0 > route add default 192.168.0.254 > > The default route will be set to 192.168.0.254 on I1, not on I2 as > expected. > It's still a misconfiguration -- if you do not bring the I1 interface down, the "interface" route for I2 will not be installed, and the ifconfig(8) command will be aborted. Anyway... I reproduced this with two ng_eiface(4) nodes. There's still a useful option in the route(8) utility -- you can supply an explicit reference to the interface, like this: : route add default 192.168.0.254 -ifp I1 This is also vaguely documented in the route(8) manpage, FWIW: : In a change or add command where the destination and gateway are not suf- : ficient to specify the route ..., the -ifp or -ifa modifiers may be used : to determine the interface or interface address. Cheers, -- Ruslan Ermilov ru@FreeBSD.org FreeBSD committer
On Wed, Sep 08, 2004 at 03:06:48PM +0200, Thomas Quinot wrote: > * Ruslan Ermilov, 2004-09-08 : > > > > ifconfig I1 192.168.0.1 netmask 255.255.255.0 > > > ifconfig I1 down > > > ifconfig I2 192.168.0.2 netmask 255.255.255.0 > > > route add default 192.168.0.254 > > > > > > The default route will be set to 192.168.0.254 on I1, not on I2 as > > > expected. > > > > > It's still a misconfiguration -- if you do not bring the I1 interface > > down, the "interface" route for I2 will not be installed, and the > > ifconfig(8) command will be aborted. Anyway... > > I am not sure I get what you mean. The point of this PR is that I *do* > bring I1 down, that the kernel has an unambiguous indication of how to > reach the specified gateway ('ping 192.168.0.254' works in the example > described above), and in spite of that the wrong interface is selected > when creating a route. > This is a misconfiguration because you cannot expect two broadcast interfaces configured with the same IP network to work. The fact that it works at all, in a scenario you shown, is probably a bug. > > I reproduced this with two ng_eiface(4) nodes. There's still a useful > > option in the route(8) utility -- you can supply an explicit reference > > to the interface, like this: > > : route add default 192.168.0.254 -ifp I1 > > Right, I was not aware of that option. > > > This is also vaguely documented in the route(8) manpage, FWIW: > > *Vaguely* is the right term I think. :-) > > Having browsed through the code, I think the interface lookup for route > additions should use a modified version of ifa_ifwithaddr ignoring > non-up interfaces. > If we change ifa_ifwith*() to ignore !IFF_UP interfaces, I suspect this will break too many things. I think the correct change here would be to set the interface of the route to the interface of its gateway (if route is through the gateway (RTF_GATEWAY)). This should be easy to fix, but I don't have a time for this now. Cheers, -- Ruslan Ermilov ru@FreeBSD.org FreeBSD committer
Responsible Changed From-To: freebsd-bugs->bms I'll take this
Hi, I've been looking at this PR. A few points, based on code inspection and reproducing the test case. *ifa_ifwithroute() already checks for RTF_GATEWAY, and unconditionally calls ifa_ifwithdstaddr() to obtain the ifa pointer. * The rtmsg passed in from userland is well-formed (RTM_ADD). * ifa_ifwithroute() in this case is called from rt_getifa() during RTM_ADD. * ifa_ifwithdstaddr() is called from many other parts of the stack, so being conservative about the change seems sensible. * This isn't really a multipath problem so much as a next-hop resolution problem. Perhaps if we change ifa_ifwithdstaddr() to ifa_ifwithdstaddrflags() and pass IFF_UP as a necessary flag for the interface match that would be an acceptable fix, and change other consumers to pass 0 or use a macro to emulate the old function. We have to call ifa_ifwithdstaddr() to look up the correct ifaddr for the rt->rt_ifa anyway; the rt->rt_ifp can in turn be looked up from that. Just my 2c, BMS
For a related patch, see also: http://lists.freebsd.org/pipermail/freebsd-net/2006-February/009884.html (Provided by glebius)
State Changed From-To: open->analyzed We can reproduce the problem, we've analyzed it, there are possible fixes, we need to decide what to do.
This looks like an architectural problem stemming from how next-hop resolution is performed in FreeBSD's forwarding plane. Fear not -- XORP has similar problems; recursively resolving next-hops when they change is not a trivial issue. To my mind this needs to be fixed with a rewrite. Looking at the code, we'd have to do another tree walk on PRC_IFDOWN. We already do this, but we ignore static routes. The reason why the default route doesn't get its ifa/ifp updated when the interface it refers to goes down, is because it's static.
This sounds like a job for a 'floating static' forwarding entry. A 'floating static' entry is one whose next-hop may be re-resolved although the entry itself is static and should be treated as such by the forwarding code. A 'floating' flag would allow the PRC_IFDOWN path to tell the difference, and update the ifp in those cases where the next-hop would change, without clobbering the state of the next-hop for FIB entries marked as static. See IOS or JunOS documentation for more information on 'floating static routes'.
Responsible Changed From-To: bms->none I'm afraid I have no free time for the foreseeable future which I can commit directly to FreeBSD work, due to a demanding new role. I am happy to review the occasional patch, but regrettably cannot be directly involved in engineering work beyond this. Thank you for your understanding.
Responsible Changed From-To: none->freebsd-bugs Canonicalize assignment.
Responsible Changed From-To: freebsd-bugs->freebsd-net On second thought, it probably belongs to -net.
batch change: For bugs that match the following - Status Is In progress AND - Untouched since 2018-01-01. AND - Affects Base System OR Documentation DO: Reset to open status. Note: I did a quick pass but if you are getting this email it might be worthwhile to double check to see if this bug ought to be closed.