Bug 256681 - [route] Incorrect loopback route for aliases IP addresses
Summary: [route] Incorrect loopback route for aliases IP addresses
Status: In Progress
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 13.0-STABLE
Hardware: Any Any
: --- Affects Some People
Assignee: Alexander V. Chernikov
URL: https://reviews.freebsd.org/D30811
Keywords:
Depends on:
Blocks:
 
Reported: 2021-06-18 02:52 UTC by Zhenlei Huang
Modified: 2021-06-21 08:48 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Zhenlei Huang 2021-06-18 02:52:58 UTC
Observed this regression on stable/13 and current/14.

Steps to repeat:

# ifconfig tap0 create inet 192.0.2.1/24
# ifconfig tap0 inet 192.0.2.2/32 alias

To verify the route table:

# netstat -4rnW
---------------------------------------
Destination        Gateway            Flags   Nhop#    Mtu      Netif Expire
...
192.0.2.0/24       link#4             U           5   1500       tap0
192.0.2.1          link#4             UHS         6  16384        lo0
192.0.2.2          link#4             UH          7   1500       tap0
...
---------------------------------------

See the loopback route for alias IP address 192.0.2.2 is incorrect.

To verify the impact:
# ping 192.0.2.2
PING 192.0.2.2 (192.0.2.2): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host


Expected route table:
192.0.2.2          link#4             UHS         7  16384       lo0

The Mtu should be 16384 and Netif should be lo0. The Flags should contains `S`, ie. should be same with 192.0.2.1

Expected behavior:
ping alias IP address with success
Comment 1 Zhenlei Huang 2021-06-18 02:57:05 UTC
It seems https://reviews.freebsd.org/D28246 introduce this side effect.
Comment 2 Zhenlei Huang 2021-06-18 08:04:11 UTC
Proposed patch: https://reviews.freebsd.org/D30811
Comment 3 Rodney W. Grimes freebsd_committer 2021-06-18 15:46:44 UTC
I strongly disagree that this is the "expected" route for a /32
placed on an "interface"

Expected route table:
192.0.2.2          link#4             UHS         7  16384       lo0

The bug to find is why can we no longer access 192.0.2.2 via the routing table and the tap0 interface address, this is a local interface address and should be reachable no matter WHAT the routing table looks like.
Comment 4 Zhenlei Huang 2021-06-19 18:02:01 UTC
(In reply to Rodney W. Grimes from comment #3)
Comparing to stable/12 and stable/11, I call the route is "expected".

Spent hours to repeat this on previous releases.

Prior to FreeBSD 8.0, upon adding IP address to interface, only the on-link prefix route will be created. As for /32 aliases, the prefix route is /32. For output routine, ip_output() will consult rtalloc() to "generate" the loopback route for local IP address and use it.

From FreeBSD 8.0 to FreeBSD 12.2, the on-link prefix route will be created, as well as the loopback route for local IP address. See [1]. 

From FreeBSD 13.0 and onward, due to significant rework of routing subsystem, route for the /32 aliases is treated specially. See [2].

I reviewed the design of loopback route. If I understand correctly, the loopback route short cut the out path of packets those are destined for local, it directly forward it to local and bypass the physical interface. See [3].

So in some cases, if the loopback route is disabled and the hardware/logical interface can not forward those packets destined for local, 'No route to host' should be generated.

PS, there is a 'net.link.ether.inet.useloopback' sysctl tunable, it is default on, and  controls whether installing a loopback route or not. This tunable existed at least since 4.11 and was removed from stable/11. See [4].


[1]: https://cgit.freebsd.org/src/commit/sys/netinet/in.c?h=stable/8&id=ebc90701ac6c1f814c5bd6f3e19f0113ebe06156
[2]: https://reviews.freebsd.org/D28668
[3]: https://cgit.freebsd.org/src/tree/sys/netinet/if_ether.c?h=stable/4#n109
[4]: https://cgit.freebsd.org/src/commit/sys/netinet/if_ether.c?h=stable/11&id=b1b9dcae46803ae79255a9994584cb03d2a77048
Comment 5 Zhenlei Huang 2021-06-21 08:36:50 UTC
> So in some cases, if the loopback route is disabled and the hardware/logical interface can not forward those packets destined for local, 'No route to host' should be generated.

Since release/8.1, a new feature 'IFCAP_LINKSTATE' was introduced, see [5] and [6]. If an interface have 'IFCAP_LINKSTATE' capability, then the "link" state should be checked before passing those packets to it.

I verified this feature on stable/12 and stable/13. The steps:
1. ifconfig vxlan0 create vxlanid 100 vxlanlocal 10.x.x.x vxlanremote 10.y.y.y
2. ifconfig vxlan0 inet 192.0.2.1/24
3. ifconfig vxlan0 inet 192.0.2.2/32 alias
4. netstat -rnWf inet | grep 192.0.2
5. ping -c4 192.0.2.1
6. ping -c4 192.0.2.2
7. route delete 192.0.2.1
8. route delete 192.0.2.2
9. repeat step 4
10. ifconfig vxlan0 down && 

If create vxlan0 without vxlanid vxlanlocal and vxlanremote, then the link state is not ready. If we delete the loopback route to 192.0.2.1 and 192.0.2.2, ping will response with 'No route to host'.


[5]: https://cgit.freebsd.org/src/commit/sys/netinet/ip_output.c?h=stable/8&id=c951da56b4f19a637c7fdf734fc500560a9555de
[6]: https://cgit.freebsd.org/src/commit/sys?h=stable/8&id=94190b3925795b145fbd1fbc39df0841ef52f5d5
Comment 6 Zhenlei Huang 2021-06-21 08:48:08 UTC
(In reply to Rodney W. Grimes from comment #3)
> The bug to find is why can we no longer access 192.0.2.2 via the routing table and
> the tap0 interface address, this is a local interface address and should be 
> reachable no matter WHAT the routing table looks like.

I can conclude this is not bug that the local interface address is not reachable for tap0 interface. It behaves correctly as it should be. It is a FEATURE.

@CC Alexander V. Chernikov