Bug 236833 - RADIX_MPATH is incompatible with multiple unnumbered P2P interfaces
Summary: RADIX_MPATH is incompatible with multiple unnumbered P2P interfaces
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 11.2-RELEASE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-net mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-03-27 18:53 UTC by Grzes
Modified: 2019-04-30 08:26 UTC (History)
4 users (show)

See Also:


Attachments
Kernel config (14.58 KB, text/plain)
2019-03-29 21:45 UTC, Grzes
no flags Details
Loader.conf (441 bytes, text/plain)
2019-03-29 21:45 UTC, Grzes
no flags Details
sysctl.conf (769 bytes, text/plain)
2019-03-29 21:46 UTC, Grzes
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Grzes 2019-03-27 18:53:37 UTC
When the RADIX_MPATH option is added in the kernel, an error occurs.

a_maintain_loopback_route: switch failed for interface ngx: 3

Configuration 11.2-RELEASE-p9. Demon MPD5 as pppoe server with two or more connected pppoe sessions, when disconnecting the interface ng0 when in the routing table on the interface ng0 is loopback does not change the entry on interface ng1 and there is an error "fa_maintain_loopback_route: switch failed for interface ng1: 3"

netstat -rn
192.168.249.1 link#9 UHS lo0
192.1.1.2 link#9 UH ng0
192.1.1.3 link#10 UH ng1

after disconnecting ng0
netstat -rn
192.168.249.1 link#9 UHS lo0
192.1.1.2 link#10 UH ng1

when the disconnection of both sessions appears an error "
ifa_maintain_loopback_route: deletion failed for interface ng1: 3 "and you can not delete the entry in the routing table.

netstat -rn
192.168.249.1 link#9 UHS lo0
Comment 1 Grzes 2019-03-27 18:58:50 UTC
correction 
after disconnecting ng0
netstat -rn
192.168.249.1 link#9 UHS lo0
192.1.1.3 link#10 UH ng1
Comment 2 Grzes 2019-03-27 18:59:41 UTC
(In reply to grzesiek from comment #0)
Comment 3 Eugene Grosbein freebsd_committer 2019-03-29 11:39:20 UTC
Please supply more information: add full output of ifconfig command for ngX interfaces when both PPPoE sessions connected and describe what do you use RADIX_MPATH for?
Comment 4 Grzes 2019-03-29 17:42:29 UTC
RADIX_MPATH I use to route to two default gateways to the edge router. I have two links leased from different providers and balances outgoing and incoming traffic, in case of problems with one whole traffic goes the other thanks to ospfd with frr5.0 which supports multipath routing. I get on bandwidth and I have a backup provider failure. Router is FreeBSD 11.2 with RADIX_MPATH, ospfd with frr5. Each provider on a different network infrastructure.

ifconfig
ng0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1492
        inet 192.168.249.1 --> 192.1.1.2 netmask 0xffffffff
        nd6 options=9<PERFORMNUD,IFDISABLED>
        groups: tun
ng1: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1480
        inet 192.168.249.1 --> 192.1.1.3 netmask 0xffffffff
        nd6 options=1<PERFORMNUD>
        groups: tun

netstat -rn
192.168.249.1      link#9             UHS         lo0
192.1.1.3          link#10            UH          ng1
x.x.x.x/28         link#5             U           em0
x.x.x.x            link#5             UHS         lo0
192.1.1.2          link#9             UH          ng0



After disconnecting all pppoe sessions and reconnecting with the error "ifa_maintain_loopback_route: deletion failed for interface ng1: 3" the routing table has a doubled entry

netstat -rn

192.168.249.1      link#9             UHS         lo0
192.168.249.1      link#9             UHS         lo0
192.1.1.3          link#10            UH          ng1
192.1.1.2          link#9             UH          ng0

ifconfig
ng0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1492
        inet 192.168.249.1 --> 192.1.1.2 netmask 0xffffffff
        nd6 options=1<PERFORMNUD>
        groups: tun
ng1: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1492
        inet 192.168.249.1 --> 192.1.1.3 netmask 0xffffffff
        nd6 options=1<PERFORMNUD>
        groups: tun

Log from the production machine:
Mar 29 17:51:22 xxxxx kernel: ifa_maintain_loopback_route: switch failed for interface ng239: 3
Mar 29 17:58:55 xxxxx kernel: ifa_maintain_loopback_route: switch failed for interface ng270: 3
Mar 29 18:05:14 xxxxx kernel: ifa_maintain_loopback_route: switch failed for interface ng236: 3
Mar 29 18:06:45 xxxxx kernel: ifa_maintain_loopback_route: switch failed for interface ng236: 3
Mar 29 18:07:38 xxxxx kernel: ifa_maintain_loopback_route: switch failed for interface ng220: 3
Mar 29 18:07:51 xxxxx kernel: ifa_maintain_loopback_route: switch failed for interface ng5: 3
Mar 29 18:29:26 xxxxx kernel: ifa_maintain_loopback_route: switch failed for interface ng279: 3
Mar 29 18:32:25 xxxxx kernel: ifa_maintain_loopback_route: switch failed for interface ng1: 3
Comment 5 Grzes 2019-03-29 18:22:37 UTC
You also can not delete an entry from the routing table

route delete  192.168.249.1

route: route has not been found
delete host 192.168.249.1 fib 0: not in table
Comment 6 Eugene Grosbein freebsd_committer 2019-03-29 20:07:40 UTC
Do you have options FLOWTABLE in your kernel configuration? Attach the configuration, if possible.
Comment 7 Eugene Grosbein freebsd_committer 2019-03-29 20:09:05 UTC
Also, do you use multiple FIBs? Do you have any non-default system tuning in /boot/loader.conf or /etc/sysctl.conf? If so, attach them too.
Comment 8 Grzes 2019-03-29 21:45:15 UTC
Created attachment 203246 [details]
Kernel config
Comment 9 Grzes 2019-03-29 21:45:52 UTC
Created attachment 203247 [details]
Loader.conf
Comment 10 Grzes 2019-03-29 21:46:24 UTC
Created attachment 203248 [details]
sysctl.conf
Comment 11 Grzes 2019-03-29 21:51:54 UTC
Not use multiple FIBS.
Not have options FLOWTABLE .
If i remove the RADIX_MPATH options from the kernel, the error does not occur.
Comment 12 Eugene Grosbein freebsd_committer 2019-03-29 22:35:34 UTC
It seems you currently cannot use RADIX_MPATH and such PPPoE server in single machine as RADIX_MPATH code does not handle this configuration allowing to install multiple unneeded identical loopback routes and failing to remove them.

Until this is fixed, I'd suggest you move PPPoE server to distinct FreeBSD machine.

CC'ng kmacy and melifaro who may have some thoughts on how this should be fixed.