Example route -4 add -host 169.254.1.1 127.0.0.1 route -4 add -net 169.254.1.2/32 127.0.0.1 netstat -nrW4 Internet: Destination Gateway Flags Use Mtu Netif Expire default 192.168.232.254 UGS 7328 1500 lan1 169.254.1.1 127.0.0.1 UGHS 256 16384 lo0 169.254.1.2/32 127.0.0.1 UGS 7008 16384 lo0 ipfw table 8 list 2.2.2.2/32 169.254.1.1 2.2.2.3/32 169.254.1.2 ipfw show 7000 7010 07000 26151 1297044 fwd tablearg ip from any to table(8) 07010 384 32256 fwd 169.254.1.1 ip from any to 2.2.1.1 07010 512 43008 fwd 169.254.1.2 ip from any to 2.2.1.2 On FreeBSD 11.1, 11.2 all the ping & traceroute to 2.2.2.2, 2.2.2.3, 2.2.1,1, 2.2.1,2 are identical. ping 2.2.2.3 PING 2.2.2.3 (2.2.2.3): 56 data bytes 36 bytes from localhost (127.0.0.1): Time to live exceeded Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 0054 21c2 0 0000 01 01 0000 192.168.232.246 2.2.2.3 traceroute -m 4 -I 2.2.2.2 traceroute to 2.2.2.2 (2.2.2.2), 4 hops max, 48 byte packets 1 localhost (127.0.0.1) 0.075 ms 0.046 ms 0.067 ms 2 localhost (127.0.0.1) 0.041 ms 0.043 ms 0.037 ms 3 localhost (127.0.0.1) 0.038 ms 0.044 ms 0.038 ms 4 localhost (127.0.0.1) 0.040 ms 0.048 ms 0.040 ms But on FreeBSD 11.3 traffic to 2.2.2.2 sent via default, ignoring UGHS route. traceroute -m 4 -In 2.2.2.2 traceroute to 2.2.2.2 (2.2.2.2), 4 hops max, 48 byte packets 1 192.168.232.254 1.521 ms 0.386 ms 0.469 ms 2 192.168.232.249 0.459 ms 0.559 ms 0.579 ms
This bug appears on 12.2 also. Maybe example above is too complicated, below one is simpler. route -4 add default your_gw route -4 add -host 169.254.1.1 127.0.0.1 route -4 add -net 169.254.1.2/32 127.0.0.1 ipfw table 8 create type addr valtype ipv4 ipfw table 8 add 8.8.8.8 169.254.1.1 ipfw table 8 add 8.8.4.4 169.254.1.2 ipfw add 8 fwd tablearg ip from any to 'table(8)' > ping 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: icmp_seq=0 ttl=106 time=17.320 ms > ping 8.8.4.4 PING 8.8.4.4 (8.8.4.4): 56 data bytes 92 bytes from 127.0.0.1: Time to live exceeded Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 0054 424a 0 0000 01 01 0000 192.168.111.111 8.8.4.4 Routing must be same for both destinations, why different?
May I ask you to provide a tcpdump of the ICMP packet from the lan1 and lo respectively? For local destinations, the documented "ipfw fwd" behavior is different (packet unmodified).
Created attachment 224785 [details] tcpdump lan1
Created attachment 224786 [details] tcpdump lo0
The analysis is similar to PR 255705. mbuf_tag PACKET_TAG_IPFORWARD is not honored for host routes (but for network routes even /32) starting with 11.3 So you can use the network routes as a workaround. Hence the problem is in the routing stack. I'll reclassify the bug report.
I'm not sure that ipfw is quite innocent. As mentioned in the 1st message there is confusion with tablearg and direct IP. In this example 8.8.8.8 is routed via default gw, while 8.8.4.4 via 192.168.68.1 ipfw list 8 fwd tablearg ip from any to table(8) 9 fwd 169.254.1.1 ip from any to 8.8.4.4 ipfw table 8 list 8.8.8.8/32 169.254.1.1 Routing tables Destination Gateway Flags Use Mtu Netif Expire default 192.168.68.64 UGS 16 1500 lan1 169.254.1.1 192.168.68.1 UGHS 6 1500 lan1 192.168.68.0/24 link#1 U 76669362 1500 lan1 192.168.68.111 link#1 UHS 7 16384 lo0
(In reply to bugs.freebsd.org from comment #6) This is different from the discussed one. Can you please state exactly which tests have which result at which OS release?
Yes. This issue was first time appeared on 11.3-RELEASE and observed on 11.4-12.2. Only the combination of IPv4, tablearg and host route is out of order, any other works properly, also IPv6, tablearg and host route is ok. I suspect that the bug is present on 13.0 also, but now hidden due to new bug https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255705
(In reply to bugs.freebsd.org from comment #8) The ipfw code does not do any lookup of the route table, it's only adding a tag to the mbuf containing the packet. So if there is any difference between handling ha host route and a network route the difference might be located in the routeing code. I'll try to investigate.