Bug 236833

Summary: RADIX_MPATH is incompatible with multiple unnumbered P2P interfaces
Product: Base System Reporter: Grzes <grzesiek>
Component: kernAssignee: freebsd-net (Nobody) <net>
Status: Open ---    
Severity: Affects Only Me CC: eugen, kmacy, melifaro, zlei
Priority: ---    
Version: 11.2-RELEASE   
Hardware: amd64   
OS: Any   
Attachments:
Description Flags
Kernel config
none
Loader.conf
none
sysctl.conf none

Description Grzes 2019-03-27 18:53:37 UTC 
When the RADIX_MPATH option is added in the kernel, an error occurs.

a_maintain_loopback_route: switch failed for interface ngx: 3

Configuration 11.2-RELEASE-p9. Demon MPD5 as pppoe server with two or more connected pppoe sessions, when disconnecting the interface ng0 when in the routing table on the interface ng0 is loopback does not change the entry on interface ng1 and there is an error "fa_maintain_loopback_route: switch failed for interface ng1: 3"

netstat -rn
192.168.249.1 link#9 UHS lo0
192.1.1.2 link#9 UH ng0
192.1.1.3 link#10 UH ng1

after disconnecting ng0
netstat -rn
192.168.249.1 link#9 UHS lo0
192.1.1.2 link#10 UH ng1

when the disconnection of both sessions appears an error "
ifa_maintain_loopback_route: deletion failed for interface ng1: 3 "and you can not delete the entry in the routing table.

netstat -rn
192.168.249.1 link#9 UHS lo0
Comment 1 Grzes 2019-03-27 18:58:50 UTC 
correction 
after disconnecting ng0
netstat -rn
192.168.249.1 link#9 UHS lo0
192.1.1.3 link#10 UH ng1
Comment 2 Grzes 2019-03-27 18:59:41 UTC 
(In reply to grzesiek from comment #0)
Comment 3 Eugene Grosbein freebsd_committer freebsd_triage 2019-03-29 11:39:20 UTC 
Please supply more information: add full output of ifconfig command for ngX interfaces when both PPPoE sessions connected and describe what do you use RADIX_MPATH for?
Comment 4 Grzes 2019-03-29 17:42:29 UTC 
RADIX_MPATH I use to route to two default gateways to the edge router. I have two links leased from different providers and balances outgoing and incoming traffic, in case of problems with one whole traffic goes the other thanks to ospfd with frr5.0 which supports multipath routing. I get on bandwidth and I have a backup provider failure. Router is FreeBSD 11.2 with RADIX_MPATH, ospfd with frr5. Each provider on a different network infrastructure.

ifconfig
ng0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1492
        inet 192.168.249.1 --> 192.1.1.2 netmask 0xffffffff
        nd6 options=9<PERFORMNUD,IFDISABLED>
        groups: tun
ng1: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1480
        inet 192.168.249.1 --> 192.1.1.3 netmask 0xffffffff
        nd6 options=1<PERFORMNUD>
        groups: tun

netstat -rn
192.168.249.1      link#9             UHS         lo0
192.1.1.3          link#10            UH          ng1
x.x.x.x/28         link#5             U           em0
x.x.x.x            link#5             UHS         lo0
192.1.1.2          link#9             UH          ng0



After disconnecting all pppoe sessions and reconnecting with the error "ifa_maintain_loopback_route: deletion failed for interface ng1: 3" the routing table has a doubled entry

netstat -rn

192.168.249.1      link#9             UHS         lo0
192.168.249.1      link#9             UHS         lo0
192.1.1.3          link#10            UH          ng1
192.1.1.2          link#9             UH          ng0

ifconfig
ng0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1492
        inet 192.168.249.1 --> 192.1.1.2 netmask 0xffffffff
        nd6 options=1<PERFORMNUD>
        groups: tun
ng1: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1492
        inet 192.168.249.1 --> 192.1.1.3 netmask 0xffffffff
        nd6 options=1<PERFORMNUD>
        groups: tun

Log from the production machine:
Mar 29 17:51:22 xxxxx kernel: ifa_maintain_loopback_route: switch failed for interface ng239: 3
Mar 29 17:58:55 xxxxx kernel: ifa_maintain_loopback_route: switch failed for interface ng270: 3
Mar 29 18:05:14 xxxxx kernel: ifa_maintain_loopback_route: switch failed for interface ng236: 3
Mar 29 18:06:45 xxxxx kernel: ifa_maintain_loopback_route: switch failed for interface ng236: 3
Mar 29 18:07:38 xxxxx kernel: ifa_maintain_loopback_route: switch failed for interface ng220: 3
Mar 29 18:07:51 xxxxx kernel: ifa_maintain_loopback_route: switch failed for interface ng5: 3
Mar 29 18:29:26 xxxxx kernel: ifa_maintain_loopback_route: switch failed for interface ng279: 3
Mar 29 18:32:25 xxxxx kernel: ifa_maintain_loopback_route: switch failed for interface ng1: 3
Comment 5 Grzes 2019-03-29 18:22:37 UTC 
You also can not delete an entry from the routing table

route delete  192.168.249.1

route: route has not been found
delete host 192.168.249.1 fib 0: not in table
Comment 6 Eugene Grosbein freebsd_committer freebsd_triage 2019-03-29 20:07:40 UTC 
Do you have options FLOWTABLE in your kernel configuration? Attach the configuration, if possible.
Comment 7 Eugene Grosbein freebsd_committer freebsd_triage 2019-03-29 20:09:05 UTC 
Also, do you use multiple FIBs? Do you have any non-default system tuning in /boot/loader.conf or /etc/sysctl.conf? If so, attach them too.
Comment 8 Grzes 2019-03-29 21:45:15 UTC 
Created attachment 203246 [details]
Kernel config
Comment 9 Grzes 2019-03-29 21:45:52 UTC 
Created attachment 203247 [details]
Loader.conf
Comment 10 Grzes 2019-03-29 21:46:24 UTC 
Created attachment 203248 [details]
sysctl.conf
Comment 11 Grzes 2019-03-29 21:51:54 UTC 
Not use multiple FIBS.
Not have options FLOWTABLE .
If i remove the RADIX_MPATH options from the kernel, the error does not occur.
Comment 12 Eugene Grosbein freebsd_committer freebsd_triage 2019-03-29 22:35:34 UTC 
It seems you currently cannot use RADIX_MPATH and such PPPoE server in single machine as RADIX_MPATH code does not handle this configuration allowing to install multiple unneeded identical loopback routes and failing to remove them.

Until this is fixed, I'd suggest you move PPPoE server to distinct FreeBSD machine.

CC'ng kmacy and melifaro who may have some thoughts on how this should be fixed.