In addition to https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=245277 on 13-0.RELEASE routing table is fully ignored by ipfw fwd. Example: route -4 add default 192.168.68.64 (is a gateway which rejects all) route -4 add -host 169.254.1.1 192.168.68.1 (is a real gateway) Routing tables Internet: Destination Gateway Flags Use Mtu Netif Expire default 192.168.68.64 UGS 26 1500 lan1 127.0.0.1 link#3 UH 2589737 16384 lo0 169.254.1.1 192.168.68.1 UGHS 18 1500 lan1 192.168.68.0/24 link#1 U 8126264 1500 lan1 192.168.68.125 link#1 UHS 0 16384 lo0 ipfw table 8 add 8.8.8.8 169.254.1.1 ipfw add 8 fwd tablearg ip from any to table(8) ipfw add 9 fwd 169.254.1.1 ip from any to 8.8.4.4 ipfw add 10 fwd 192.168.68.1 ip from any to 1.1.1.1 on 10.3-RELEASE (and up to 11.2-RELEASE) PING 5.5.5.5 (5.5.5.5): 56 data bytes 36 bytes from 192.168.68.64: Destination Host Unreachable Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 0054 5ac8 0 0000 3f 01 11b2 192.168.68.125 5.5.5.5 but 64 bytes from 8.8.8.8: icmp_seq=0 ttl=106 time=17.876 ms 64 bytes from 8.8.4.4: icmp_seq=0 ttl=109 time=16.055 ms 64 bytes from 1.1.1.1: icmp_seq=0 ttl=54 time=23.887 ms on 13.0-RELEASE it forwards all the destinations via default gw PING 1.1.1.1 (1.1.1.1): 56 data bytes 36 bytes from 192.168.68.64: Destination Host Unreachable Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 0054 b346 0 0000 3f 01 c3d5 192.168.68.125 1.1.1.1
May I ask you to provide a tcpdump on the lan1 interface for the relevant packets? Thank you.
Created attachment 224787 [details] tcpdump lan1
On the previous versions (11.3 to 12.2) only packets to 8.8.8.8 has wrong routing is the same conditions.
First let me quote the man page of ipfw(8). I verfied, that this description matches the behavior of the source code in branch "main". ----8/---- If ipaddr is a local address, then matching packets will be forwarded to port (or the port number in the packet if one is not specified in the rule) on the local machine. If ipaddr is not a local address, then the port number (if specified) is ignored, and the packet will be forwarded to the remote address, using the route as found in the local routing table for that IP. The fwd action does not change the contents of the packet at all. In particular, the destination address remains unmodified, so packets forwarded to another system will usually be rejected by that system unless there is a matching rule on that system to capture them. For packets forwarded locally, the local address of the socket will be set to the original destination address of the packet. This makes the netstat(1) entry look rather weird but is intended for use with transparent proxy servers. ----8/---- Now let's have a look at your rules. PING 5.5.5.5 does not match the ipfw rule set, does not match a specific route, and is forwarded to the default gateway. This gateway rejects the the packet, which is documented in your example. PING 8.8.8.8 does match your ipfw table(8) rule, hence the packet is send to the next hop 169.254.1.1. 169.254.1.1 is in your routing table and will use the next hop 192.168.68.1, which is your Internet gateway. So the unmodified packet is send to your Internet gateway, travel all the way through the net, will be reach the Google server, will get a response. This response is documented in your example. PING 8.8.4.4 does match your ipfw rule #9, hence the packet is send to the next hop 169.254.1.1. 169.254.1.1 is in your routing table and will use the next hop 192.168.68.1, which is your Internet gateway. So the unmodified packet is send to your Internet gateway, travel all the way through the net, will be reach the Google server, will get a response. This response is documented in your example. PING 1.1.1.1 does match your ipfw rule #10, hence the packet is send to the next hop 192.168.68.1, which is your Internet gateway. So the unmodified packet is send to your Internet gateway, travel all the way through the net, will be reach the Google server, will get a response. This response is documented in your example. So far for anything up to 11.2. Your traceroute shows that on 13.0 all packets are send to the default gateway regardless of the specified next_hop attribute (mbuf_tag = PACKET_TAG_IPFORWARD), as set by the ipfw processing. Hence this problem is a routing problem and not an ipfw problem. I'll reclassify the bug report.
The problem does persist in recent stable/13. I did apply patches from bug 256828 and got my site back to working condition with these. Patches seem to be applicable to 13.1-RC1 also.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=7d98cc096b995ca3bfd85277ed009b0f872c3e1b commit 7d98cc096b995ca3bfd85277ed009b0f872c3e1b Author: Andrey V. Elsukov <ae@FreeBSD.org> AuthorDate: 2022-04-01 13:49:25 +0000 Commit: Andrey V. Elsukov <ae@FreeBSD.org> CommitDate: 2022-04-11 11:16:43 +0000 Fix ipfw fwd that doesn't work in some cases For IPv4 use dst pointer as destination address in fib4_lookup(). It keeps destination address from IPv4 header and can be changed when PACKET_TAG_IPFORWARD tag was set by packet filter. For IPv6 override destination address with address from dst_sa.sin6_addr, that was set from PACKET_TAG_IPFORWARD tag. Reviewed by: eugen MFC after: 1 week PR: 256828, 261697, 255705 Differential Revision: https://reviews.freebsd.org/D34732 sys/netinet/ip_output.c | 2 +- sys/netinet6/ip6_output.c | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=17c9c2049004038ed6f2dc23a64cb9f74411ec52 commit 17c9c2049004038ed6f2dc23a64cb9f74411ec52 Author: Andrey V. Elsukov <ae@FreeBSD.org> AuthorDate: 2022-04-01 13:49:25 +0000 Commit: Andrey V. Elsukov <ae@FreeBSD.org> CommitDate: 2022-04-18 08:58:45 +0000 Fix ipfw fwd that doesn't work in some cases For IPv4 use dst pointer as destination address in fib4_lookup(). It keeps destination address from IPv4 header and can be changed when PACKET_TAG_IPFORWARD tag was set by packet filter. For IPv6 override destination address with address from dst_sa.sin6_addr, that was set from PACKET_TAG_IPFORWARD tag. Reviewed by: eugen PR: 256828, 261697, 255705 Differential Revision: https://reviews.freebsd.org/D34732 (cherry picked from commit 7d98cc096b995ca3bfd85277ed009b0f872c3e1b) sys/netinet/ip_output.c | 2 +- sys/netinet6/ip6_output.c | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-)