First box is 12.2 root@freebsd:~ # uname -a FreeBSD freebsd 12.2-RELEASE-p7 FreeBSD 12.2-RELEASE-p7 GENERIC amd64 root@freebsd:~ # ipfw show 01000 164 13146 fwd 146.185.211.254 ip4 from 146.185.210.33 to any out 65534 682 61472 allow ip from any to any 65535 0 0 deny ip from any to any root@freebsd:~ # ifconfig vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> ether fa:16:3e:6b:d5:ee inet 89.208.84.44 netmask 0xfffffc00 broadcast 89.208.87.255 inet6 fe80::f816:3eff:fe6b:d5ee%vtnet0 prefixlen 64 scopeid 0x1 media: Ethernet 10Gbase-T <full-duplex> status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> vtnet1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> ether fa:16:3e:80:95:dc inet 146.185.210.33 netmask 0xfffffc00 broadcast 146.185.211.255 inet6 fe80::f816:3eff:fe80:95dc%vtnet1 prefixlen 64 scopeid 0x2 media: Ethernet 10Gbase-T <full-duplex> status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> root@freebsd:~ # netstat -4rn Routing tables Internet: Destination Gateway Flags Netif Expire default 89.208.87.254 UGS vtnet0 89.208.84.0/22 link#1 U vtnet0 89.208.84.44 link#1 UHS lo0 127.0.0.1 link#3 UH lo0 146.185.208.0/22 link#2 U vtnet1 146.185.210.33 link#2 UHS lo0 Ping from outside to second IP works as expected with ipfw fwd rule. woody@unknown ~ % ping 146.185.210.33 PING 146.185.210.33 (146.185.210.33): 56 data bytes 64 bytes from 146.185.210.33: icmp_seq=0 ttl=51 time=28.814 ms 64 bytes from 146.185.210.33: icmp_seq=1 ttl=51 time=33.822 ms 64 bytes from 146.185.210.33: icmp_seq=2 ttl=51 time=38.074 ms 64 bytes from 146.185.210.33: icmp_seq=3 ttl=51 time=42.863 ms 64 bytes from 146.185.210.33: icmp_seq=4 ttl=51 time=38.847 ms
Second box was upgraded from 12.1 to 13.0 root@freebsd:~ # uname -a FreeBSD freebsd 13.0-RELEASE-p1 FreeBSD 13.0-RELEASE-p1 #0: Wed May 26 22:12:31 UTC 2021 root@amd64-builder.daemonology.net:/usr/obj/usr/src/i386.i386/sys/GENERIC i386 root@freebsd:~ # ifconfig vtnet0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6> ether fa:16:3e:41:3f:66 inet 185.241.193.112 netmask 0xfffffc00 broadcast 185.241.195.255 inet6 fe80::f816:3eff:fe41:3f66%vtnet0 prefixlen 64 scopeid 0x1 media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL> vtnet1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6> ether fa:16:3e:83:5e:a0 inet 185.86.145.31 netmask 0xfffffc00 broadcast 185.86.147.255 inet6 fe80::f816:3eff:fe83:5ea0%vtnet1 prefixlen 64 scopeid 0x2 media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL> root@freebsd:~ # ipfw show 01000 984 82430 fwd 185.241.195.254 ip4 from 185.241.193.112 to any out 65534 8980 6911385 allow ip from any to any 65535 0 0 deny ip from any to any root@freebsd:~ # netstat -4rn Routing tables Internet: Destination Gateway Flags Netif Expire default 185.86.147.254 UGS vtnet1 127.0.0.1 link#3 UH lo0 185.86.144.0/22 link#2 U vtnet1 185.86.145.31 link#2 UHS lo0 185.241.192.0/22 link#1 U vtnet0 185.241.193.112 link#1 UHS lo0 root@freebsd:~ # cat /etc/rc.conf hostname="freebsd" ifconfig_DEFAULT="DHCP inet6 accept_rtadv" growfs_enable="YES" defaultrouter="185.86.147.254" ifconfig_vtnet1="inet 185.86.145.31/22" ifconfig_vtnet0="inet 185.241.193.112/22" sshd_enable="YES" gateway_enable="YES" firewall_enable="YES" firewall_script="/usr/local/etc/ipfw.sh" External ping to second IP stopped working after upgrade! woody@unknown ~ % ping 185.241.193.112 PING 185.241.193.112 (185.241.193.112): 56 data bytes Request timeout for icmp_seq 0 Request timeout for icmp_seq 1 Request timeout for icmp_seq 2 Request timeout for icmp_seq 3 root@freebsd:~ # tcpdump -en -i vtnet0 icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vtnet0, link-type EN10MB (Ethernet), capture size 262144 bytes 12:35:35.797653 02:37:b3:65:6a:42 > fa:16:3e:41:3f:66, ethertype IPv4 (0x0800), length 98: 176.59.17.29 > 185.241.193.112: ICMP echo request, id 64342, seq 20, length 64 12:35:36.804656 02:37:b3:65:6a:42 > fa:16:3e:41:3f:66, ethertype IPv4 (0x0800), length 98: 176.59.17.29 > 185.241.193.112: ICMP echo request, id 64342, seq 21, length 64 12:35:37.815712 02:37:b3:65:6a:42 > fa:16:3e:41:3f:66, ethertype IPv4 (0x0800), length 98: 176.59.17.29 > 185.241.193.112: ICMP echo request, id 64342, seq 22, length 64 12:35:38.804542 02:37:b3:65:6a:42 > fa:16:3e:41:3f:66, ethertype IPv4 (0x0800), length 98: 176.59.17.29 > 185.241.193.112: ICMP echo request, id 64342, seq 23, length 64 12:35:39.807677 02:37:b3:65:6a:42 > fa:16:3e:41:3f:66, ethertype IPv4 (0x0800), length 98: 176.59.17.29 > 185.241.193.112: ICMP echo request, id 64342, seq 24, length 64 12:35:40.807667 02:37:b3:65:6a:42 > fa:16:3e:41:3f:66, ethertype IPv4 (0x0800), length 98: 176.59.17.29 > 185.241.193.112: ICMP echo request, id 64342, seq 25, length 64 root@freebsd:~ # tcpdump -en -i vtnet1 icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vtnet1, link-type EN10MB (Ethernet), capture size 262144 bytes 12:36:12.915754 fa:16:3e:83:5e:a0 > 02:37:b3:65:6a:42, ethertype IPv4 (0x0800), length 98: 185.241.193.112 > 176.59.17.29: ICMP echo reply, id 64342, seq 57, length 64 12:36:13.922502 fa:16:3e:83:5e:a0 > 02:37:b3:65:6a:42, ethertype IPv4 (0x0800), length 98: 185.241.193.112 > 176.59.17.29: ICMP echo reply, id 64342, seq 58, length 64 12:36:14.907498 fa:16:3e:83:5e:a0 > 02:37:b3:65:6a:42, ethertype IPv4 (0x0800), length 98: 185.241.193.112 > 176.59.17.29: ICMP echo reply, id 64342, seq 59, length 64 12:36:15.924737 fa:16:3e:83:5e:a0 > 02:37:b3:65:6a:42, ethertype IPv4 (0x0800), length 98: 185.241.193.112 > 176.59.17.29: ICMP echo reply, id 64342, seq 60, length 64 12:36:16.924447 fa:16:3e:83:5e:a0 > 02:37:b3:65:6a:42, ethertype IPv4 (0x0800), length 98: 185.241.193.112 > 176.59.17.29: ICMP echo reply, id 64342, seq 61, length 64 ICMP echo replies goes back via defaultrouter interfaces. ipfw keeps increasing rule count
Might be a problem with the new routing framework.
Same problem. root@localhost:~ # ifconfig em0 em0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=481009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER,NOMAP> ether 00:0c:29:3f:8e:40 inet 192.168.1.3 netmask 0xffffff00 broadcast 192.168.1.255 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> root@localhost:~ # netstat -rn Routing tables Internet: Destination Gateway Flags Netif Expire default 192.168.1.254 UGS em0 127.0.0.1 link#2 UH lo0 192.168.1.0/24 link#1 U em0 192.168.1.3 link#1 UHS lo0 root@localhost:~ # ipfw list 00100 allow ip from any to any via lo0 00200 deny ip from any to 127.0.0.0/8 00300 deny ip from 127.0.0.0/8 to any 00400 deny ip from any to ::1 00500 deny ip from ::1 to any 00600 allow ipv6-icmp from :: to ff02::/16 00700 allow ipv6-icmp from fe80::/10 to fe80::/10 00800 allow ipv6-icmp from fe80::/10 to ff02::/16 00900 allow ipv6-icmp from any to any icmp6types 1 01000 allow ipv6-icmp from any to any icmp6types 2,135,136 01100 fwd 192.168.1.253 ip4 from me 80 to any 01200 allow ip from any to any via em0 01300 deny log ip from any to any 65535 deny ip from any to any Traffic from 80 port still going via default route.
Created attachment 228631 [details] ip_output patch checking ipfw fwd was used
Hello, I have same issue in 13.0-STABLE It's look like 'ip_output' function try to find route/interface based on destination IP address (even if ipfw fwd was used). I wrote a (ugly) patch that worked in my case (ipv4 only)
Created attachment 228669 [details] ip6_output patch checking ipfw fwd was used Hello, I had similar issue on IPv6 Here is my dirty patch. Best regards.
(In reply to Sylvain Galliano from comment #5) Thank you, works for me. 13.0-STABLE FreeBSD 13.0-STABLE #2 stable/13-n248611-f15f29c975f-dirty
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=7d98cc096b995ca3bfd85277ed009b0f872c3e1b commit 7d98cc096b995ca3bfd85277ed009b0f872c3e1b Author: Andrey V. Elsukov <ae@FreeBSD.org> AuthorDate: 2022-04-01 13:49:25 +0000 Commit: Andrey V. Elsukov <ae@FreeBSD.org> CommitDate: 2022-04-11 11:16:43 +0000 Fix ipfw fwd that doesn't work in some cases For IPv4 use dst pointer as destination address in fib4_lookup(). It keeps destination address from IPv4 header and can be changed when PACKET_TAG_IPFORWARD tag was set by packet filter. For IPv6 override destination address with address from dst_sa.sin6_addr, that was set from PACKET_TAG_IPFORWARD tag. Reviewed by: eugen MFC after: 1 week PR: 256828, 261697, 255705 Differential Revision: https://reviews.freebsd.org/D34732 sys/netinet/ip_output.c | 2 +- sys/netinet6/ip6_output.c | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=17c9c2049004038ed6f2dc23a64cb9f74411ec52 commit 17c9c2049004038ed6f2dc23a64cb9f74411ec52 Author: Andrey V. Elsukov <ae@FreeBSD.org> AuthorDate: 2022-04-01 13:49:25 +0000 Commit: Andrey V. Elsukov <ae@FreeBSD.org> CommitDate: 2022-04-18 08:58:45 +0000 Fix ipfw fwd that doesn't work in some cases For IPv4 use dst pointer as destination address in fib4_lookup(). It keeps destination address from IPv4 header and can be changed when PACKET_TAG_IPFORWARD tag was set by packet filter. For IPv6 override destination address with address from dst_sa.sin6_addr, that was set from PACKET_TAG_IPFORWARD tag. Reviewed by: eugen PR: 256828, 261697, 255705 Differential Revision: https://reviews.freebsd.org/D34732 (cherry picked from commit 7d98cc096b995ca3bfd85277ed009b0f872c3e1b) sys/netinet/ip_output.c | 2 +- sys/netinet6/ip6_output.c | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-)
Fixed before 13.1-RELEASE
*** Bug 261697 has been marked as a duplicate of this bug. ***
uname -a FreeBSD xxxx.ru 13.1-RELEASE-p2 FreeBSD 13.1-RELEASE-p2 GENERIC amd64 "ipfw fwd RemoteGWip ...." is broken in FreeBSD 13.1-p2 - packet flow via default route
(In reply to dol from comment #13) Please open new PR describing your setup in details. Perhaps, the patch was incomplete and does not cover your case; or it was broken again later.
(In reply to Eugene Grosbein from comment #11) In fact, the fix did not hit 13.1-RELEASE but 13.2
*** Bug 266439 has been marked as a duplicate of this bug. ***
*** Bug 270177 has been marked as a duplicate of this bug. ***