Bug 263202

Summary: "route already in table" - even though it's not, and can't be deleted
Product: Base System Reporter: Pat Maddox <pat>
Component: kernAssignee: freebsd-net (Nobody) <net>
Status: Closed FIXED    
Severity: Affects Some People CC: grahamperrin, zlei
Priority: ---    
Version: 13.0-RELEASE   
Hardware: Any   
OS: Any   

Description Pat Maddox 2022-04-10 17:55:21 UTC
FreeBSD 13 claims to not be able to add a route because it's already in the table. It doesn't appear to be, and attempting to delete it claims that it's not:

    # route add 100.120.128.187 -interface lo0
    add host 100.120.128.187: gateway lo0 fib 0: route already in table
    
    # route delete 100.120.128.187 -interface lo0
    route: route has not been found
    delete host 100.120.128.187: gateway lo0 fib 0: not in table

The route that does exist for that host goes over a different interface (and it's showing the hostname rather than IP):

    # netstat -r4
    ...
    gcp-freebsd-12-3.p link#3             UHS    tailscal

I can delete the route for that host, and then add a lo0 route:

    # route delete 100.120.128.187
    delete host 100.120.128.187
    
    # route add 100.120.128.187 -interface lo0
    add host 100.120.128.187: gateway lo0
Comment 1 Pat Maddox 2022-04-10 17:56:18 UTC
Note: This problem appears to be new in 13.0. It works fine in 12.3.
Comment 2 Pat Maddox 2022-04-10 21:01:12 UTC
It also works in 13.1-RC2. I have confirmed this both with upgrading 12.3 (works) -> 13.0 (doesn't work) -> 13.1-RC2 (works), as well as a fresh install of 13.1-RC2.
Comment 3 Zhenlei Huang freebsd_committer freebsd_triage 2022-04-11 02:21:09 UTC
(In reply to Pat Maddox from comment #2)
Hi, can you please also try latest stable/13 and current ?
Comment 4 Pat Maddox 2022-04-11 05:53:49 UTC
(In reply to Zhenlei Huang from comment #3)

My understanding of release branches is that those would both be 13.1, and thus have the fix.

Are you asking me to see if this behavior has broken again, after 13.1-RC2?
Comment 5 Zhenlei Huang freebsd_committer freebsd_triage 2022-04-11 06:22:32 UTC
(In reply to Pat Maddox from comment #4)
No, there is no changes of routing subsystem after 13.1-RC2.

Probably stable/13 and current do not have this issue, just want to be sure.
Comment 6 Pat Maddox 2022-04-11 20:53:16 UTC
(In reply to Zhenlei Huang from comment #5)

Okay. I think I can't use freebsd-update for that, right? I read https://docs.freebsd.org/en/books/handbook/cutting-edge/#current-stable and it looks like the only way to do STABLE or CURRENT is to build from source. Is that correct?
Comment 7 Pat Maddox 2022-04-11 21:11:39 UTC
(In reply to Pat Maddox from comment #6)

That is of course answered in the opening section of freebsd-update(8):

    the FreeBSD Security Team only builds	updates	for
     releases shipped in binary	form by	the FreeBSD Release Engineering	Team,
     e.g., FreeBSD 11.2-RELEASE	and FreeBSD 12.0-RELEASE, but not FreeBSD
     11.2-STABLE or FreeBSD 13.0-CURRENT

I will give it a shot and report back!
Comment 8 Pat Maddox 2022-04-12 15:55:02 UTC
(In reply to Pat Maddox from comment #7)

I have tested this on stable/13 (62c0ba91c78fc) and main (94cb21cda66b3) and both are fine.

The problem appears to be limited to the 13.0 line.
Comment 9 Zhenlei Huang freebsd_committer freebsd_triage 2022-04-13 10:06:12 UTC
(In reply to Pat Maddox from comment #8)
Thanks for your report.

It is weird that I managed to reproduce on 13.0 and also 13.1-RC2.

This is what I had tried:
# ifconfig tap0 create inet 100.120.128.187/24
# netstat -4rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
...
100.120.128.0/24   link#3             U          tap0
100.120.128.187    link#3             UHS         lo0
127.0.0.1          link#2             UH          lo0
...

# route add 100.120.128.187 -interface lo0
add host 100.120.128.187: gateway lo0 fib 0: route already in table

# route delete 100.120.128.187 -interface lo0
route: route has not been found
delete host 100.120.128.187: gateway lo0 fib 0: not in table

# route delete 100.120.128.187
delete host 100.120.128.187

# route add 100.120.128.187 -interface lo0
add host 100.120.128.187: gateway lo0

# netstat -4rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
...
100.120.128.0/24   link#3             U          tap0
100.120.128.187    link#2             UHS         lo0
127.0.0.1          link#2             UH          lo0
...
Comment 10 Zhenlei Huang freebsd_committer freebsd_triage 2022-04-14 11:35:51 UTC
Tested on latest stable/13 and current, both behave the same with 13.0.
Comment 11 Pat Maddox 2022-04-14 15:27:04 UTC
It is weird indeed. One difference I see is that I am using a tun device, whereas you’re using tap. I don’t see why that would lead to different behavior with the routing table though.

What git commit in stable did you use? I can try again with that one and see if something has changed (seems unlikely but you never know).
Comment 12 Pat Maddox 2022-04-14 15:32:57 UTC
(In reply to Zhenlei Huang from comment #9)

Actually it looks like you do in fact have that lo0 route in the table already (the second line). So I would expect your first add command to fail. What’s strange is that you can’t delete it, even though netstat shows it.

My problem is a bit different - I can’t add that route at all.

Then at that point we have a similar problem, where we can’t delete the route because it supposedly doesn’t exist. Although in your case, it sure looks like it exists!
Comment 13 Zhenlei Huang freebsd_committer freebsd_triage 2022-04-15 04:54:54 UTC
(In reply to Pat Maddox from comment #11)
stable/13: 01f095a355606e9b, main: bf13db086b84c29c
Comment 14 Zhenlei Huang freebsd_committer freebsd_triage 2022-04-15 05:00:34 UTC
(In reply to Pat Maddox from comment #12)
For tun(4) devices, a dest_address is also required. Could you please share more details how is your tun device set up?
Comment 15 Pat Maddox 2022-04-16 17:28:19 UTC
(In reply to Zhenlei Huang from comment #14)

Mine is set up using tailscale.
Comment 16 Zhenlei Huang freebsd_committer freebsd_triage 2022-04-17 08:26:09 UTC
(In reply to Pat Maddox from comment #15)
Suppose tailscale set up the tun device as tun0, the following will be sufficient:

 1. # ifconfig tun0
 2. # netstat -4rn

Note please do NOT remove any route entries related to tun0 before report back the output of the above command.

You an omit or replace any sensitive info such as your real IP address.
Comment 17 Pat Maddox 2022-05-05 08:48:08 UTC
(In reply to Zhenlei Huang from comment #16)

=== 12.3-RELEASE-p5

$ ifconfig tailscale0
tailscale0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 1280
	options=80000<LINKSTATE>
	inet 100.123.67.255 netmask 0xffffffff broadcast 100.123.67.255
	inet6 fd7a:115c:a1e0:efe3::647b:43ff prefixlen 48
	groups: tun
	nd6 options=101<PERFORMNUD,NO_DAD>
	Opened by PID 21991

$ ping -c 1 100.123.67.255
PING 100.123.67.255 (100.123.67.255): 56 data bytes
64 bytes from 100.123.67.255: icmp_seq=0 ttl=64 time=0.041 ms

--- 100.123.67.255 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss

$ netstat -4rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            137.184.80.1       UGS      vtnet0
10.48.0.0/16       link#1             U        vtnet0
10.48.0.5          link#1             UHS         lo0
10.124.0.0/20      link#2             U        vtnet1
10.124.0.2         link#2             UHS         lo0
100.82.35.64/32    tailscale0         US     tailscal
100.100.100.100/32 tailscale0         US     tailscal
100.116.30.19/32   tailscale0         US     tailscal
100.116.80.150/32  tailscale0         US     tailscal
100.123.67.255     link#4             UHS         lo0
100.123.67.255/32  link#4             U      tailscal
127.0.0.1          link#3             UH          lo0



=== 13.0-RELEASE-p11

$ ifconfig tailscale0
tailscale0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 1280
	options=80000<LINKSTATE>
	inet 100.116.80.150 netmask 0xffffffff broadcast 100.116.80.150
	inet6 fd7a:115c:a1e0:efe3::6474:5096 prefixlen 48
	groups: tun
	nd6 options=101<PERFORMNUD,NO_DAD>
	Opened by PID 1159

$ ping -c 1 100.116.80.150
PING 100.116.80.150 (100.116.80.150): 56 data bytes

--- 100.116.80.150 ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss

$ netstat -4rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            137.184.176.1      UGS      vtnet0
10.48.0.0/16       link#1             U        vtnet0
10.48.0.6          link#1             UHS         lo0
10.124.0.0/20      link#2             U        vtnet1
10.124.0.3         link#2             UHS         lo0
100.82.35.64       link#4             UHS    tailscal
100.100.100.100    link#4             UHS    tailscal
100.116.30.19      link#4             UHS    tailscal
100.116.80.150     link#4             UH     tailscal
100.123.67.255     link#4             UHS    tailscal
127.0.0.1          link#3             UH          lo0
Comment 18 Zhenlei Huang freebsd_committer freebsd_triage 2022-05-07 05:15:19 UTC
So the dev tailscale0 is a tun and is not a POINTOPOINT interface. I'll try that config.

See TUN(4) and https://github.com/WireGuard/wireguard-go/blob/ef5c587f782d944005971c7ba5da88405f9b000b/tun/tun_freebsd.go#L202-L205
Comment 19 Zhenlei Huang freebsd_committer freebsd_triage 2022-05-12 01:39:20 UTC
(In reply to Zhenlei Huang from comment #18)
Managed to repeat the problem on 13.0. stable/13 and 13.1-RC5 looks good.
Comment 20 Pat Maddox 2023-03-27 11:21:13 UTC
Closing since 13.0 is no longer supported. Fixed somewhere in 13.1.