Bug 256393 - Issue with recreation of ppp/tun interfaces
Summary: Issue with recreation of ppp/tun interfaces
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 13.0-STABLE
Hardware: Any Any
: --- Affects Some People
Assignee: Alexander V. Chernikov
URL:
Keywords: needs-qa, regression
Depends on:
Blocks:
 
Reported: 2021-06-03 02:46 UTC by Nathan Whitehorn
Modified: 2021-06-07 14:38 UTC (History)
7 users (show)

See Also:
koobs: mfc-stable13?
koobs: mfc-stable12-
koobs: mfc-stable11-


Attachments
Patch to routed to avoid creating pointless loopback routes (1.39 KB, patch)
2021-06-03 18:37 UTC, Nathan Whitehorn
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Nathan Whitehorn freebsd_committer 2021-06-03 02:46:28 UTC
I am running a machine acting as a ppp server using userland ppp(8) and running 13-STABLE (this issue does not occur on 13.0-RELEASE and earlier). When a client establishes a connection, everything is fine and a connection is established without issue. When that connection ends and the client attempts to re-establish the connection, however, IPCP fails and just continues in a loop, never setting the IP address on the local tun(4) interface. If I try to set it by hand, I get this:

root@conference:/home/nwhitehorn # ifconfig tun0
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet6 fe80::f816:3eff:fe26:d92a%tun0 prefixlen 64 scopeid 0x3
        groups: tun
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 1359
root@conference:/home/nwhitehorn # ifconfig tun0 192.168.42.1 192.168.42.2
ifconfig: ioctl (SIOCAIFADDR): File exists

This seems like a regression in tun(4), but I haven't had a chance to really debug it.
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2021-06-03 04:42:20 UTC
@Nathan 

 - What stable/13 revision? (uname -a) Is there a known working previous/last revision?
 - Any relevent error logs (console or messages) with respect to 'IPCP fails' etc?
 - Any chance of a bisection of sys/dev/tun ?
 - Any additional info with sysctl debug.if_tun_debug=1 ?
 - Any additional logging with ppp.conf set log all
 - Can you include the relevent ppp.conf (sanitized) as an attachment
Comment 2 Nathan Whitehorn freebsd_committer 2021-06-03 17:45:07 UTC
(In reply to Kubilay Kocak from comment #1)

Here are a whole bunch of logs:
FreeBSD xxxxxxxxxxxxxxx 13.0-STABLE FreeBSD 13.0-STABLE #1 stable/13-n245818-ae23d302479: Wed Jun  2 10:27:55 EDT 2021     nwhitehorn@terel.tachypleus.net:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64

From the ppp log (key portion is "Error: ipcp_InterfaceUp: unable to set ip address"):
Jun  3 17:10:07 conference syslogd: last message repeated 1 times
Jun  3 17:10:07 conference ppp[8508]: tun0: IPCP: deflink: RecvConfigAck(2) state = Ack-Sent
Jun  3 17:10:07 conference ppp[8508]: tun0: IPCP:  IPADDR[6] 192.168.42.1
Jun  3 17:10:07 conference ppp[8508]: tun0: IPCP:  COMPPROTO[6] 16 VJ slots with slot compression
Jun  3 17:10:07 conference ppp[8508]: tun0: IPCP: deflink: State change Ack-Sent --> Opened
Jun  3 17:10:07 conference ppp[8508]: tun0: IPCP: deflink: LayerUp.
Jun  3 17:10:07 conference ppp[8508]: tun0: IPCP: myaddr 192.168.42.1 hisaddr = 192.168.42.2
Jun  3 17:10:07 conference ppp[8508]: tun0: Warning: iface add: ioctl(SIOCAIFADDR, 192.168.42.1 -> 192.168.42.2): File exists
Jun  3 17:10:07 conference ppp[8508]: tun0: Error: ipcp_InterfaceUp: unable to set ip address
Jun  3 17:10:07 conference ppp[8508]: tun0: IPCP: deflink: LayerDown: 192.168.42.1
Jun  3 17:10:07 conference ppp[8508]: tun0: IPCP: deflink: SendTerminateReq(3) state = Opened
Jun  3 17:10:07 conference ppp[8508]: tun0: IPCP: deflink: State change Opened --> Closing
Jun  3 17:10:07 conference ppp[8508]: tun0: LCP: deflink: SendIdent(4) state = Opened
Jun  3 17:10:07 conference ppp[8508]: tun0: LCP:  MAGICNUM c6d8cf0b
Jun  3 17:10:07 conference ppp[8508]: tun0: LCP:  TEXT user-ppp 3.4.2
Jun  3 17:10:08 conference ppp[8508]: tun0: Warning: ipv4_Input: IPCP not open - packet dropped
Jun  3 17:10:10 conference syslogd: last message repeated 3 times
Jun  3 17:10:10 conference ppp[8508]: tun0: IPCP: deflink: SendTerminateReq(3) state = Closing
Jun  3 17:10:11 conference ppp[8508]: tun0: Warning: ipv4_Input: IPCP not open - packet dropped
Jun  3 17:10:13 conference syslogd: last message repeated 5 times
Jun  3 17:10:14 conference ppp[8508]: tun0: IPCP: deflink: SendTerminateReq(3) state = Closing
Jun  3 17:10:14 conference ppp[8508]: tun0: Warning: ipv4_Input: IPCP not open - packet dropped


Jun  3 17:43:15 conference syslogd: last message repeated 1 times
Jun  3 17:43:15 conference ppp[8811]: tun0: IPCP: deflink: RecvConfigAck(2) state = Ack-Sent
Jun  3 17:43:15 conference ppp[8811]: tun0: IPCP:  IPADDR[6] 192.168.42.1
Jun  3 17:43:15 conference ppp[8811]: tun0: IPCP:  COMPPROTO[6] 16 VJ slots with slot compression
Jun  3 17:43:15 conference ppp[8811]: tun0: IPCP: deflink: State change Ack-Sent --> Opened
Jun  3 17:43:15 conference ppp[8811]: tun0: IPCP: deflink: LayerUp.
Jun  3 17:43:15 conference ppp[8811]: tun0: IPCP: myaddr 192.168.42.1 hisaddr = 192.168.42.2
Jun  3 17:43:15 conference ppp[8811]: tun0: Warning: iface add: ioctl(SIOCAIFADDR, 192.168.42.1 -> 192.168.42.2): File exists
Jun  3 17:43:15 conference ppp[8811]: tun0: Error: ipcp_InterfaceUp: unable to set ip address
Jun  3 17:43:15 conference ppp[8811]: tun0: IPCP: deflink: LayerDown: 192.168.42.1
Jun  3 17:43:15 conference ppp[8811]: tun0: IPCP: deflink: SendTerminateReq(3) state = Opened
Jun  3 17:43:15 conference ppp[8811]: tun0: IPCP: deflink: State change Opened --> Closing
Jun  3 17:43:15 conference ppp[8811]: tun0: LCP: deflink: SendIdent(4) state = Opened
Jun  3 17:43:15 conference ppp[8811]: tun0: LCP:  MAGICNUM d57f3e26
Jun  3 17:43:15 conference ppp[8811]: tun0: LCP:  TEXT user-ppp 3.4.2
Jun  3 17:43:15 conference ppp[8811]: tun0: Warning: ipv4_Input: IPCP not open - packet dropped
Jun  3 17:43:18 conference syslogd: last message repeated 3 times

Nothing in dmesg with tun_debug set except this:
tun0: tunpoll
tun0: tunpoll waiting

I can try to bisect, but it might be a little while as this is an important production machine.
Comment 3 Nathan Whitehorn freebsd_committer 2021-06-03 18:01:19 UTC
I at least found the proximate problem:

root@conference:/home/nwhitehorn # netstat -rn -f inet
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            128.135.X.X    UGS      vtnet0
127.0.0.1          link#2             UH          lo0
128.135.X.X/25 link#1             U        vtnet0
128.135.X.X    link#1             UHS         lo0
169.254.169.254    128.135.X.X    UGHS     vtnet0
192.168.42.1       127.0.0.1          UH          lo0  <- this is the IP that the tunnel will use once established, and it didn't go away when the tunnel interface did

If I delete the route to the tunnel endpoint on the loopback interface that appeared:

root@conference:/home/nwhitehorn # route delete 192.168.42.1
delete host 192.168.42.1

Then the connection re-establishes itself without issue.

The route in question seems to have been added by routed(8), replacing the link-level default route, though I am not sure why it is not going away when routed(8) exits and the interface disappears or why routed(8) is re-writing it:


root@conference:/home/nwhitehorn # netstat -rn -f inet
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            128.135.X.X    UGS      vtnet0
127.0.0.1          link#2             UH          lo0
128.135.X.X/25 link#1             U        vtnet0
128.135.X.X    link#1             UHS         lo0
169.254.X.X    128.135.X.X    UGHS     vtnet0
192.168.42.0/24    192.168.42.2       UGS        tun0
192.168.42.1       link#3             UHS         lo0 <- Original loopback route
192.168.42.2       link#3             UH         tun0
root@conference:/home/nwhitehorn # service routed start
Starting routed.
root@conference:/home/nwhitehorn # netstat -rn -f inet
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            128.135.X.X    UGS      vtnet0
10.7.114.0/24      192.168.42.2       UG         tun0
127.0.0.1          link#2             UH          lo0
128.135.X.X/25 link#1             U        vtnet0
128.135.X.X    link#1             UHS         lo0
157.132.6.8/29     192.168.42.2       UG         tun0
169.254.X.X    128.135.158.134    UGHS     vtnet0
192.168.1.0/24     192.168.42.2       UG         tun0
192.168.2.0/24     192.168.42.2       UG         tun0
192.168.3.0/24     192.168.42.2       UG         tun0
192.168.5.0/24     192.168.42.2       UG         tun0
192.168.42.0/24    192.168.42.2       UGS        tun0
192.168.42.1       127.0.0.1          UH          lo0 <- Rewritten by routed(8)
192.168.42.2       link#3             UH         tun0
192.168.176.0/24   192.168.42.2       UG         tun0

On the 12.2 system on the other end of this link, the same loopback host route is present, but it doesn't seem to prevent re-IP'ing the link:

$ netstat -rn -f inet
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
192.168.42.1       link#5             UHS        tun0
192.168.42.2       127.0.0.1          UH          lo0

My suspicion is that the actual problem is that some of the routing-table verification code that went in recently is preventing adding an IP address with an address to which there is already a route on another interface.
Comment 4 Eugene Grosbein freebsd_committer 2021-06-03 18:27:46 UTC
(In reply to Nathan Whitehorn from comment #3)

> My suspicion is that the actual problem is that some of the routing-table verification code that went in recently is preventing adding an IP address with an address to which there is already a route on another interface.

If such "verification" was really added, it is wrong and must be corrected. Only PINNED routes should remain intact, other routes should be replaceable by daemons like ppp or net/mpd5 or alike even if they were already added by some routing daemons.

Adding some more eyes to the CC of this PR.
Comment 5 Nathan Whitehorn freebsd_committer 2021-06-03 18:36:26 UTC
One workaround to this is to just delete the logic in routed that adds this loopback route. The code is original from SGI in 1996 and I think never did anything useful on FreeBSD -- it certainly doesn't seem to now -- but didn't cause harm until whatever kernel change is preventing setting an IP on this interface.

That said, while I'm happy to commit the attached patch to routed, the underlying behavior in the kernel, whatever it is, that is making SIOCAIFADDR fail seems bogus to me.
Comment 6 Nathan Whitehorn freebsd_committer 2021-06-03 18:37:10 UTC
Created attachment 225533 [details]
Patch to routed to avoid creating pointless loopback routes
Comment 7 Nathan Whitehorn freebsd_committer 2021-06-03 18:40:22 UTC
I should also mention that I am no longer confident that this issue wasn't in 13.0-RELEASE. routed(8) is what triggers it here and routed(8) does not work at all in 13.0-RELEASE because of changes to the routing code in the kernel, so my test on the release was not valid.
Comment 8 Eugene Grosbein freebsd_committer 2021-06-03 18:53:41 UTC
> This seems like a regression in tun(4), but I haven't had a chance to really debug it.

> Nothing in dmesg with tun_debug set except this:
> tun0: tunpoll
> tun0: tunpoll waiting

Please configure your syslog daemon to write kern.debug messages to some file, reproduce the problem and post that logs.
Comment 9 Eugene Grosbein freebsd_committer 2021-06-03 19:01:48 UTC
(In reply to Eugene Grosbein from comment #8)

> Please configure your syslog daemon to write kern.debug messages

Though, it is quite possible you will not get anything at kern.debug level because writing debugging logs for EEXIST case explicitly skipped in the ifa_maintain_loopback_route()
Comment 10 Nathan Whitehorn freebsd_committer 2021-06-03 19:07:21 UTC
I unfortunately also can't do any more debugging on this now that I have a workaround; the system is mission-critical and I can't justify taking it back down for the time being.
Comment 11 Alexander V. Chernikov freebsd_committer 2021-06-03 21:41:15 UTC
Okay, so there are two things mentioned here:
1) refusal to set an interface address when p2p route is already present in the system
2) routed(8) adding loopback routes and causing (1)


For 1)
Could you check that net.inet.ip.no_same_prefix is set to 0?
In my tests, `ifconfig tun0 192.168.42.1 192.168.42.2` always work on 13-S, regardless of 42.1 and 42.2 routes present in the rtable or not.

For 2)
As far as I understand the reason is the fix of host route dualism ( https://reviews.freebsd.org/D28668 ). Effectively "host/32" and "host w/empty netmask" routes were different routes in the routing table.
Routed always tried to manipulate /32 route, while kernel operated host routes w/o netmask. As a result, routed would install /32, which would not affect forwarding as route w/o netmask is the most specific.
After the diff, "host/32" routes gets automatically converted to the "host w/empty netmask" resulting in routed touching kernel routes.

That said, I'm all up with removing the loopback manipulation piece from routed(8).
Comment 12 Nathan Whitehorn freebsd_committer 2021-06-03 22:33:11 UTC
net.inet.ip.no_same_prefix is indeed zero. If I set a static route by hand, it seems to work OK here too -- the issue only seems to happen with dynamic routes. I'm not sure why.
Comment 13 Eugene Grosbein freebsd_committer 2021-06-04 01:29:23 UTC
(In reply to Alexander V. Chernikov from comment #11)

> That said, I'm all up with removing the loopback manipulation piece from routed(8).

Does it mean that system behaviour will keep changed comparing to stable/12 and earlier branches? I mean, if one assigns /30 network with such mask to a P2P interface (tun, ng etc.) then packets to local IP will not be delivered over loopback but travel over P2P interface corresponding to /30 route?
Comment 14 Alexander V. Chernikov freebsd_committer 2021-06-04 08:34:16 UTC
(In reply to Eugene Grosbein from comment #13)
> I mean, if one assigns /30 network with such mask to a P2P interface (tun, ng etc.) then packets to local IP will not be delivered over loopback but travel over P2P interface corresponding to /30 route?
Kernel behaviour is the same. It will work as expected unless you run routed(8).

Potentially we can counter-action routed(8) actions by adding PINNED to the loopback/destination routes, however the desired state is that routed(8) does not touch delete loopback routes installed by the kernel.
Comment 15 Rodney W. Grimes freebsd_committer 2021-06-04 10:32:46 UTC
(In reply to Alexander V. Chernikov from comment #14)
Let me again assert, it is WRONG and causing troubles for 25 years of router code to have FreeBSD's kernel attempt to maintain these loopback routes.  That is the domain of a routing daemon, and here again we have yet another we have to fix the routing daemon cause we did something stupid in the FreeBSD kernel.

PLEASE rip out the stupid attempt for the kernel to implement a routing policy and restore this to the domain of routing code which has a far better chance of knowing what goes on.

PLEASE STOP ripping out code from routing daemons that is actually the exact correct way to maintain loop back routes.
Comment 16 Nathan Whitehorn freebsd_committer 2021-06-04 15:00:08 UTC
I'm not sure what the routing daemon is doing currently is sensible, but it at least does seem like nothing in the routing table should prevent setting the IP of an interface.
Comment 17 Eugene Grosbein freebsd_committer 2021-06-04 17:40:31 UTC
(In reply to Alexander V. Chernikov from comment #14)

I'm talking not about kernel behaviour only but abouth the whole complex of generally used scenarios. Considering also the comment of rgrimes@, let's think about following cases:

1) Some routing daemon installs to FIB some /32 route learned dynamically. It may have its reasons and it should not fail unless there is already such PINNED route in the FIB. Later some PPP daemon tries to assign that address to its interface as address of local or remote side and it should not fail with EEXIST but override non-PINNED route. It should fail with EEXIST if PINNED route exists already.

2) Same in case of a routing daemon doing same things but route(8) instead of another daemon trying to create a route or ifconfig(8) trying to assign same address, they both should fail only due to existing PINNED route. They should not fail otherwise and silently override possibly pre-existing non-PINNED route including one installed by still running routing daemon.
Comment 18 Alexander V. Chernikov freebsd_committer 2021-06-04 20:33:56 UTC
(In reply to Nathan Whitehorn from comment #12)

Interesting.

Are you absolutely sure it's stock 13-S?
Any custom sysctl's / multiple fibs ?

The logs above show commit https://cgit.freebsd.org/src/commit/sys/net/route/route_ifaddrs.c?h=stable/13&id=ae23d302479 , but does not explicitly specify the hostname (in that line).

Currently loopback routes in 13-S are PINNED: https://cgit.freebsd.org/src/tree/sys/net/route/route_ifaddrs.c?h=stable/13&id=ae23d302479#n163 , so it shouldn't be possible for routed(8) to delete such routes.

PINNED logic also should take precedence over any other non-PINNED routes, allowing to replace these on route addition during interface address assignment.

Could you by any chance show the following:

1) netstat -4rnW, netstat -4onW
2) route -n monitor
3) if possible, run `dtrace -i 'fbt:kernel:*:return /arg1==17/ {stack()}'`

4) Run commmand-to-fail: `ifconfig tun0 192.168.42.1 192.168.42.2`

5) netstat -4rnW, netstat -4onW
Comment 19 Alexander V. Chernikov freebsd_committer 2021-06-04 20:52:16 UTC
(In reply to Rodney W. Grimes from comment #15)
Do I understand correctly that you're suggesting that loopback routes should be installed by the routing daemons instead of kernel?
If yes, I'm not sure how one would handle non-router cases (e.g. a server with a single interface). I'm also not sure how can this work with modern routing software. IIRC frr does not care about any route which is not RTF_GATEWAY. It is certainly possible to configure such routes in bird, but it has to be done on per-prefix basis.
Could you share a bit more details on what is the proposed alternative?
Comment 20 Alexander V. Chernikov freebsd_committer 2021-06-04 21:08:55 UTC
(In reply to Eugene Grosbein from comment #17)
> 1) Some routing daemon installs to FIB some /32 route learned dynamically. It may have its reasons and it should not fail unless there is already such PINNED route in the FIB. Later some PPP daemon tries to assign that address to its interface as address of local or remote side and it should not fail with EEXIST but override non-PINNED route. It should fail with EEXIST if PINNED route exists already.

"and it should not fail unless there is already such PINNED route in the FIB" - currently we have 2-level priority system (PINNED and non-PINNED, https://cgit.freebsd.org/src/tree/sys/net/route/route_ctl.c?h=stable/13&id=ae23d302479#n467 ). Routes within a single priority are treated equally. If something tries to insert a route, which already exists, the following options are possible:
1) different priorities for the current/inserted route -> fail if lower, replace if higher
2) same priorities, one of nexthop is not multipath-capable (interface or temporary route) -> EEXIST
3) same priorities, both nexthops are multipath-capable -> extend/form multipath group

Other than the clarification above, the described behaviour is the current behavior.


> 2) Same in case of a routing daemon doing same things but route(8) instead of another daemon trying to create a route or ifconfig(8) trying to assign same address, they both should fail only due to existing PINNED route. They should not fail otherwise and silently override possibly pre-existing non-PINNED route including one installed by still running routing daemon.

IIRC route(8) does not use RTF_PINNED during addition, so route installation may fail even w/o PINNED route. I'm not sure if that's something that we should change.
Also, currently, override triggers a couple of rtsock notifications to allow routing daemons to track the changes, so it's not exactly "silent".

The rest describes the current system behaviour.
Comment 21 Rodney W. Grimes freebsd_committer 2021-06-04 21:31:54 UTC
(In reply to Alexander V. Chernikov from comment #19
> Do I understand correctly that you're suggesting that loopback routes should be installed by the routing daemons instead of kernel?

Suggest? No, my words are stronger than that.  The KERNEL should NOT implement
ANY routing policies.  A loopback route IS a routing policy.

Further loopback routes are a micro-optimazation that was originally done to short circuit the MTU of 1500 on ethernet, and much short in the days of IMP's
and slip lines to use the larger MTU of the loopback interfaces.  A BSD system can run perfectly fine with NONE of these loopback routes, they are nothing more than an optimization.


> If yes, I'm not sure how one would handle non-router cases (e.g. a server with a single interface).

Well this use to be handled by a simple static route, but someone couldnt handle the fact that the route goes away if you down the interface and thought that the kernel should maintain this route for them.  This is arguable a lack of skill or understanding that if you take an interface down ALL routes are gona go away, and you need to re install them.  

> I'm also not sure how can this work with modern routing software. IIRC frr does not care about any route which is not RTF_GATEWAY. It is certainly possible to configure such routes in bird, but it has to be done on per-prefix basis.

I'll discuss this with the FRR folks, but I do believe that software already knows how to maintain loopback routes.  Usually on a "router" you do NOT want these routes in place, as this hides interface errors for locally sent packets to a local address.


> Could you share a bit more details on what is the proposed alternative?  Well I think part of why we are here right now is that routed is trying to maintain these routes and it is conflicting/having issue with what the kernel is doing.  I also know of older routing code that maintained these without issue.  And finally these routes are a micro optimazation that are simply not needed in most cases.
Comment 22 Eugene Grosbein freebsd_committer 2021-06-05 08:29:02 UTC
(In reply to Alexander V. Chernikov from comment #20)

>IIRC route(8) does not use RTF_PINNED during addition, so route installation may fail even w/o PINNED route. I'm not sure if that's something that we should change. Also, currently, override triggers a couple of rtsock notifications to allow routing daemons to track the changes, so it's not exactly "silent".
>The rest describes the current system behaviour.

Thank you very much for clarificaion.

If there is daemon-installed route (learned dynamically), there should be a way to overrided it with manually installed route. I'm sure "route delete/route add" is racy and won't work because of daemon re-installing its route.

Will "route change" do the job reliably in such case?
Comment 23 Eugene Grosbein freebsd_committer 2021-06-05 08:35:25 UTC
(In reply to Rodney W. Grimes from comment #21)

> Suggest? No, my words are stronger than that.  The KERNEL should NOT implement
ANY routing policies.  A loopback route IS a routing policy.

I'm not sure I understand what "routing policies" mean in general. Nevertheless, I'd like we have some knob (sysctl?) to optionally preserve current behaviour when kernel assigns loopback routes, so packet filters still deal with local traffic as going through loopback interface, BPF "sees" it on lo0 too etc.

Too many things will break if we just remove loopback routes unconditionally.
Comment 24 Rodney W. Grimes freebsd_committer 2021-06-05 17:00:18 UTC
(In reply to Eugene Grosbein from comment #23)
> Too many things will break if we just remove loopback routes unconditionally.

I agree, that would be necessary.  Though as is you should already be filtering this on both interfaces as it is possible to down lo0 and have your filter stop working, and the traffic contine to flow, as I have stated you do NOT need these routes at all, they are a micro-optimization.
Comment 25 Eugene Grosbein freebsd_committer 2021-06-05 22:38:18 UTC
(In reply to Rodney W. Grimes from comment #24)

> I agree, that would be necessary.  Though as is you should already be filtering this on both interfaces

Almost never had a reason to filter traffic flowing through loopback with exceptionally rare case.

> they are a micro-optimization

It heavily depends on local condition and there may be lots of local traffic.

Also, I do not understand how a router can deal without delivering traffic locally for its local IPs in many cases like running squid proxy or web/smtp server or even routinely running ping to check if distinct uplink is still alive.
Comment 26 Rodney W. Grimes freebsd_committer 2021-06-05 23:11:06 UTC
(In reply to Eugene Grosbein from comment #25)
> Also, I do not understand how a router can deal without delivering traffic locally for its local IPs in many cases like running squid proxy or web/smtp server or even routinely running ping to check if distinct uplink is still alive.

You seem to not understand that traffic works just FINE without the loopback route, packets sent locally to the local ip address of an interface are delivered without any issue, the different is that the MTU used for the packets well be the MTU of the ethernet/ppp/slip/whatever interface rather than the MTU of the loopback device.

Further in your specific case of a ping check... that is near worthless to a lo0 route as it can't tell when the carrier has dropped on an interface, and infact this is one of the reasons I absolutely HATE this loopback route by the kernel code and have ripped it out of my systems.

I think somehow you think that IP does not work correctly without the loopback routes, that is a not the case.  I repeat, all the interface loopback routes are not needed, the system functions fine without them. all be it with a 10x packet count due to MTU delta for lo0 vs an ethernet.
Comment 27 Eugene Grosbein freebsd_committer 2021-06-06 06:29:51 UTC
(In reply to Rodney W. Grimes from comment #26)

> Further in your specific case of a ping check... that is near worthless to a lo0 route as it can't tell when the carrier has dropped on an interface

No, no, I was thinking about incoming ICMP echo replies delivery. Some external resource is checked by its IP address, of course. Not even remote IP of PPP interface if we need to check global connectivity - we ping some global resource with source IP of distinct link.
Comment 28 Rodney W. Grimes freebsd_committer 2021-06-06 14:36:21 UTC
(In reply to Eugene Grosbein from comment #27)
> No, no, I was thinking about incoming ICMP echo replies delivery. Some external resource is checked by its IP address, of course. Not even remote IP of PPP interface if we need to check global connectivity - we ping some global resource with source IP of distinct link.


How would that in any way involve any loopback route????  You seem to be confused on exactly when these loopback routes come into play.  An incoming ICMP echo reply is going to be picked off as "to this host" at ip_input processing and the route table wont even be looked at.
Comment 29 Kubilay Kocak freebsd_committer freebsd_triage 2021-06-07 06:11:03 UTC
Hey folks, just a heads up that the temperature of this issue is coming across higher than it probably should be.

Let's please focus on the substance of the issue "out there" with everyone on the same side looking at it together to figure it out.

Alternatively, if there's *key* design decision(s) to be made with respect to 'resolving' the issue, or a ton of questions that need to be thrown back and forth, either do so succinctly (as questions and answers) here, which is fine, or hash it out as a broader discussion on the mailing lists until an outcome is reached and then reference the list discussion and outcome here.

Thanks!
Comment 30 Eugene Grosbein freebsd_committer 2021-06-07 06:41:19 UTC
(In reply to Kubilay Kocak from comment #29)

I'm just fine with current "temperature" :-) Аnd I see nothing wrong with discussion in comments here.
Comment 31 Kubilay Kocak freebsd_committer freebsd_triage 2021-06-07 07:01:11 UTC
(In reply to Eugene Grosbein from comment #30)

I know you're both tough, might look different to the community from the outside though :)

Just a reminder to be mindful is all. 

Carry on!
Comment 32 Nathan Whitehorn freebsd_committer 2021-06-07 14:38:39 UTC
(In reply to Alexander V. Chernikov from comment #18)

It is stock 13-STABLE, with no custom sysctls or fibs. I'll try to set a maintenance window to test the things you asked for some time in the next few days.