Hi, I have been testing my ipfw/nat64 configuration on 13.0-RELEASE however this doesn't now work (the configuration is identical to the working configuration on 12.2-RELEASE). I have included the configuration details below - essentially the intent is to run a bunch of IPv6 only VNET jails with NAT64 on the host (this works fine on 12.2-RELEASE). The tcpdump output below shows that when I try an IPMPv6 ping to a NAT64 address (64:ff9b::1.1.1.1) I can see the outbound NAT64 conversion and the IPv4 ICMP response however on 13.0-RELEASE I see a strange ICMP redirect which doesn't happen with 12.2-RELEASE and it looks like the packets are rejected by the nat64lsn instance as 'discarded due to unsupported protocol' >> 16:34:03.718757 IP 127.0.0.1 > 192.168.1.55: ICMP redirect 1.1.1.1 to host 0.0.0.0, length 44 Any ideas? Regards, Paul ======== ifconfig -a ======== vtnet0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE> ether 58:9c:fc:08:4f:d0 inet 192.168.1.55 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::5a9c:fcff:fe08:4fd0%vtnet0 prefixlen 64 scopeid 0x1 inet6 2001:470:1d41:1::55 prefixlen 64 media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 58:9c:fc:10:ff:96 inet6 fe80::5a9c:fcff:fe10:ff96%bridge0 prefixlen 64 scopeid 0x3 inet6 2001:470:1d41:55::1 prefixlen 64 inet6 fe80::1%bridge0 prefixlen 64 scopeid 0x3 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 groups: bridge nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536 groups: ipfw ======== /etc/ipfw.rules ======== IPV4_LOCAL="192.168.1.55/32" IPV6_LOCAL="2001:470:1d41:1::55/128" NAT64_NETWORK="2001:470:1d41:55::/64" : ${LOG:=} # Flush ipfw -q flush ipfw -q nat64lsn NAT64 destroy # Create nat64 instance ipfw nat64lsn NAT64 create log prefix4 ${IPV4_LOCAL} prefix6 64:ff9b::/96 # Allow established connections ipfw add check-state # Allow icmp6 neighbour advertisment ipfw add allow ${LOG} icmp6 from any to any icmp6types 135,136 # Allow incoming icmp echo-requests (need keep-state to allow icmp from nat64) ipfw add allow ${LOG} icmp from any to ${IPV4_LOCAL} icmptypes 8 keep-state # Allow incoming SSH/DNS (IPv4) ipfw add allow ${LOG} ip4 from any to ${IPV4_LOCAL} 22 ipfw add allow ${LOG} ip4 from any to ${IPV4_LOCAL} 53 # Enable NAT64 ipfw add nat64lsn NAT64 ${LOG} ip6 from ::1 to 64:ff9b::/96 in ipfw add nat64lsn NAT64 ${LOG} ip6 from ${IPV6_LOCAL} to 64:ff9b::/96 in ipfw add nat64lsn NAT64 ${LOG} ip6 from ${NAT64_NETWORK} to 64:ff9b::/96 in ipfw add nat64lsn NAT64 ${LOG} ip4 from any to ${IPV4_LOCAL} in # Allow outgoing IPv4 (keep-state to skip nat64) ipfw add allow ${LOG} ip4 from ${IPV4_LOCAL} to any keep-state # Allow all ipfw add allow ${LOG} all from any to any # Set NAT64 route route -6 add 64:ff9b::/96 fe80::1%lo0 # Enable direct output sysctl net.inet.ip.fw.nat64_direct_output=1 ======== ipfw show ======== # ipfw show 00100 0 0 check-state :default 00200 82 5576 allow log ipv6-icmp from any to any icmp6types 135,136 00300 0 0 allow log icmp from any to 192.168.1.55 icmptypes 8 keep-state :default 00400 0 0 allow log ip4 from any to 192.168.1.55 22 00500 0 0 allow log ip4 from any to 192.168.1.55 53 00600 0 0 nat64lsn NAT64 log ip6 from ::1 to 64:ff9b::/96 in 00700 2 112 nat64lsn NAT64 log ip6 from 2001:470:1d41:1::55 to 64:ff9b::/96 in 00800 0 0 nat64lsn NAT64 log ip6 from 2001:470:1d41:55::/64 to 64:ff9b::/96 in 00900 2 128 nat64lsn NAT64 log ip4 from any to 192.168.1.55 in 01000 6 216 allow log ip4 from 192.168.1.55 to any keep-state :default 01100 939 127470 allow log ip from any to any ======== ping6 -c 1 64:ff9b::1.1.1.1 ======== # ping6 -c 1 64:ff9b::1.1.1.1 PING6(56=40+8+8 bytes) 2001:470:1d41:1::55 --> 64:ff9b::101:101 --- 64:ff9b::1.1.1.1 ping6 statistics --- 1 packets transmitted, 0 packets received, 100.0% packet loss ======== tcpdump -nqi ipfw0 icmp or icmp6 ======== # tcpdump -nqi ipfw0 icmp or icmp6 16:34:03.718627 IP6 2001:470:1d41:1::55 > 64:ff9b::101:101: ICMP6, echo request, seq 0, length 16 16:34:03.718654 IP6 2001:470:1d41:1::55 > 64:ff9b::101:101: ICMP6, echo request, seq 0, length 16 16:34:03.718681 IP 192.168.1.55 > 1.1.1.1: ICMP echo request, id 1024, seq 0, length 16 16:34:03.718684 IP 192.168.1.55 > 1.1.1.1: ICMP echo request, id 1024, seq 0, length 16 16:34:03.718757 IP 127.0.0.1 > 192.168.1.55: ICMP redirect 1.1.1.1 to host 0.0.0.0, length 44 16:34:03.718762 IP 127.0.0.1 > 192.168.1.55: ICMP redirect 1.1.1.1 to host 0.0.0.0, length 44 16:34:03.738308 IP 1.1.1.1 > 192.168.1.55: ICMP echo reply, id 1024, seq 0, length 16 ======== ipfw nat64lsn NAT64 stats ======== # ipfw nat64lsn NAT64 stats nat64lsn NAT64 2 packets translated from IPv6 to IPv4 0 packets translated from IPv4 to IPv6 0 IPv6 fragments created 0 IPv4 fragments received 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no IPv4 route 0 output packets discarded due to no IPv6 route 2 packets discarded due to unsupported protocol 0 packets discarded due to memory allocation problems 0 packets discarded due to some errors 0 packets not matched with IPv4 prefix 1 mbufs queued for post processing 1 times the job queue was processed 1 job requests queued 0 job requests queue limit reached 0 job requests failed due to memory allocation problems 1 hosts allocated 1 hosts requested 0 host requests failed 0 portgroups requested 1 portgroups allocated 0 portgroups deleted 0 portgroup requests failed 0 portgroups allocated for TCP 0 portgroups allocated for UDP 1 portgroups allocated for ICMP 2 states created 2 states deleted
Just in case, did you try to enable allow_private option for NAT64 instance?
I didn't think that allow_private applied in this case (destination address isn't in private address range) but I just tried adding this and it didn't make any difference.
(In reply to PaulC from comment #0) According your dump I think what was happened are: > 16:34:03.718627 IP6 2001:470:1d41:1::55 > 64:ff9b::101:101: ICMP6, echo request, seq 0, length 16 This entry generated by rule 1100 for outbound ICMPv6 echo request. > 16:34:03.718654 IP6 2001:470:1d41:1::55 > 64:ff9b::101:101: ICMP6, echo request, seq 0, length 16 Then packet was routed to lo0 due to static route, and this entry was generated by rule 700 as inbound packet. It was translated by NAT64 and directly put to outbound interface. > 16:34:03.718681 IP 192.168.1.55 > 1.1.1.1: ICMP echo request, id 1024, seq 0, length 16 I'm not sure how this entry was created. Probably you have some strange IPv4 routing and somehow you have extra firewall check, and this was created by rule 1000 as inbound packet. Thus dynamic state was created. This also means that the packet was handled by ip_input and then passed to ip_tryforward. And this is "inbound" pass from ip_tryforward. > 16:34:03.718684 IP 192.168.1.55 > 1.1.1.1: ICMP echo request, id 1024, seq 0, length 16 This entry was created by the "outbound" firewall pass from ip_tryforward. And translated packet finally gone to the destination. > 16:34:03.718757 IP 127.0.0.1 > 192.168.1.55: ICMP redirect 1.1.1.1 to host 0.0.0.0, length 44 Since net.inet.ip.redirect is enabled by default, and mbuf's rcvif matches interfice given by route, you have ICMP redirect. And this entry was created by rule 1100 by firewall pass from icmp_error->ip_output. > 16:34:03.718762 IP 127.0.0.1 > 192.168.1.55: ICMP redirect 1.1.1.1 to host 0.0.0.0, length 44 This entry probably was created by rule 900 and since ICMP redirect was not translated, it counted as unsupported protocol in NAT64. > 16:34:03.738308 IP 1.1.1.1 > 192.168.1.55: ICMP echo reply, id 1024, seq 0, length 16 This entry was created by rule 100 as reverse match from the dynamic state created before, so it did not hit NAT64 to be translated back into IPv6. If you are 100% sure that exactly this configuration worked before 13.0, I add melifaro@ to CC list, he did major routing rewriting in 13.0. Also, please provide what you have in `netstat -rn`. Also show `sysctl net.inet.ip | egrep "forward|redir"`. Also, you can create ipfwlog0 interface and see what will report NAT64 instance.
Thanks for having a look at this. Additional information below. # sysctl net.inet.ip | egrep "forward|redir" net.inet.ip.forwarding: 1 net.inet.ip.redirect: 1 # netstat -rn Routing tables Internet: Destination Gateway Flags Netif Expire default 192.168.1.1 UGS vtnet0 127.0.0.1 link#2 UH lo0 192.168.1.0/24 link#1 U vtnet0 192.168.1.55 link#1 UHS lo0 Internet6: Destination Gateway Flags Netif Expire ::/96 ::1 UGRS lo0 default fe80::1%vtnet0 UGS vtnet0 ::1 link#2 UHS lo0 ::ffff:0.0.0.0/96 ::1 UGRS lo0 64:ff9b::/96 ::1 UGS lo0 2001:470:1d41:1::/64 link#1 U vtnet0 2001:470:1d41:1::55 link#1 UHS lo0 2001:470:1d41:55::/64 link#3 U bridge0 2001:470:1d41:55::1 link#3 UHS lo0 fe80::/10 ::1 UGRS lo0 fe80::%vtnet0/64 link#1 U vtnet0 fe80::5a9c:fcff:fe08:4fd0%vtnet0 link#1 UHS lo0 fe80::%lo0/64 link#2 U lo0 fe80::1%lo0 link#2 UHS lo0 fe80::%bridge0/64 link#3 U bridge0 fe80::1%bridge0 link#3 UHS lo0 fe80::5a9c:fcff:fe10:ff96%bridge0 link#3 UHS lo0 ff02::/16 ::1 UGRS lo0 # tcpdump -nqi ipfw0 icmp or 'icmp6[icmp6type]=icmp6-echo' 16:39:35.351581 IP6 2001:470:1d41:1::55 > 64:ff9b::101:101: ICMP6, echo request, seq 0, length 16 16:39:35.351603 IP6 2001:470:1d41:1::55 > 64:ff9b::101:101: ICMP6, echo request, seq 0, length 16 16:39:35.351629 IP 192.168.1.55 > 1.1.1.1: ICMP echo request, id 1025, seq 0, length 16 16:39:35.351632 IP 192.168.1.55 > 1.1.1.1: ICMP echo request, id 1025, seq 0, length 16 16:39:35.351692 IP 127.0.0.1 > 192.168.1.55: ICMP redirect 1.1.1.1 to host 0.0.0.0, length 44 16:39:35.351696 IP 127.0.0.1 > 192.168.1.55: ICMP redirect 1.1.1.1 to host 0.0.0.0, length 44 16:39:35.369844 IP 1.1.1.1 > 192.168.1.55: ICMP echo reply, id 1025, seq 0, length 16 # tcpdump -nqei ipfwlog0 16:39:35.351611 rule 3232235831..67174657/0(match) [uid 0]: nat in on NAT64LSN: 2001:470:1d41:1::55 > 64:ff9b::101:101: ICMP6, echo request, seq 0, length 16 16:39:35.351619 rule 3232235831..67174657/0(match) [uid 0]: nat out on NAT64LSN: 192.168.1.55 > 1.1.1.1: ICMP echo request, id 1025, seq 0, length 16 The same configuration definitely worked on 12.2-RELEASE (systems are configured from the same build script). The difference seems to be the way the the translated ICMP request is handled. In 13.0 it seems to go back through the ipfw rules (generating the ICMP redirect and the IPFW state that prevents the packet getting back to the NAT64 rule) whereas on 12.2 this doesn't seem to happen. From an equivalent a 12.2-RELEASE system: # uname -a FreeBSD v6jail.pchak.net 12.2-RELEASE-p6 FreeBSD 12.2-RELEASE-p6 GENERIC amd64 # ping6 -c1 64:ff9b::1.1.1.1 PING6(56=40+8+8 bytes) 2001:470:1d41:1::50 --> 64:ff9b::101:101 16 bytes from 64:ff9b::101:101, icmp_seq=0 hlim=57 time=30.070 ms --- 64:ff9b::1.1.1.1 ping6 statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 30.070/30.070/30.070/0.000 ms # tcpdump -nqi ipfw0 icmp or 'icmp6[icmp6type]=icmp6-echo' 16:45:03.534468 IP6 2001:470:1d41:1::50 > 64:ff9b::101:101: ICMP6, echo request, seq 0, length 16 16:45:03.534483 IP6 2001:470:1d41:1::50 > 64:ff9b::101:101: ICMP6, echo request, seq 0, length 16 16:45:03.564421 IP 1.1.1.1 > 192.168.1.50: ICMP echo reply, id 1024, seq 0, length 16 # tcpdump -nqei ipfwlog0 16:45:03.537560 rule 3232235826..67109121/0(match) [uid 0]: nat in on NAT64LSN: 2001:470:1d41:1::50 > 64:ff9b::101:101: ICMP6, echo request, seq 0, length 16 16:45:03.537563 rule 3232235826..67109121/0(match) [uid 0]: nat out on NAT64LSN: 192.168.1.50 > 1.1.1.1: ICMP echo request, id 1024, seq 0, length 16 16:45:03.564423 rule 3232235826..67109121/0(match) [uid 0]: nat in on NAT64LSN: 1.1.1.1 > 192.168.1.50: ICMP echo reply, id 1024, seq 0, length 16 16:45:03.564425 rule 3232235826..67109121/0(match) [uid 0]: nat out on NAT64LSN: 64:ff9b::101:101 > 2001:470:1d41:1::50: ICMP6, echo reply, seq 0, length 16 # ipfw show 00100 0 0 check-state :default 00200 0 0 allow log ipv6-icmp from any to any icmp6types 135,136 00300 0 0 allow log icmp from any to 192.168.1.50 icmptypes 8 keep-state :default 00400 0 0 allow log ip4 from any to 192.168.1.50 22 00500 0 0 allow log ip4 from any to 192.168.1.50 53 00600 0 0 nat64lsn NAT64 log ip6 from ::1 to 64:ff9b::/96 in 00700 0 0 nat64lsn NAT64 log ip6 from 2001:470:1d41:1::50 to 64:ff9b::/96 in 00800 0 0 nat64lsn NAT64 log ip6 from 2001:470:1d41:50::/64 to 64:ff9b::/96 in 00900 0 0 nat64lsn NAT64 log ip4 from any to 192.168.1.50 in 01000 0 0 allow log ip4 from 192.168.1.50 to any keep-state :default 01100 19 1936 allow log ip from any to any 65535 0 0 allow ip from any to any # ifconfig -a vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE> ether 58:9c:fc:01:71:9d inet 192.168.1.50 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::5a9c:fcff:fe01:719d%vtnet0 prefixlen 64 scopeid 0x1 inet6 2001:470:1d41:1::50 prefixlen 64 media: Ethernet 10Gbase-T <full-duplex> status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 02:dd:a6:3d:7b:00 inet6 fe80::dd:a6ff:fe3d:7b00%bridge0 prefixlen 64 scopeid 0x3 inet6 2001:470:1d41:50::1 prefixlen 64 inet6 fe80::1%bridge0 prefixlen 64 scopeid 0x3 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 groups: bridge nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536 groups: ipfw ipfwlog0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536 groups: ipfwlog # netstat -rn Routing tables Internet: Destination Gateway Flags Netif Expire default 192.168.1.1 UGS vtnet0 127.0.0.1 link#2 UH lo0 192.168.1.0/24 link#1 U vtnet0 192.168.1.50 link#1 UHS lo0 Internet6: Destination Gateway Flags Netif Expire ::/96 ::1 UGRS lo0 default fe80::1%vtnet0 UGS vtnet0 ::1 link#2 UH lo0 ::ffff:0.0.0.0/96 ::1 UGRS lo0 64:ff9b::/96 fe80::1%lo0 UGS lo0 2001:470:1d41:1::/64 link#1 U vtnet0 2001:470:1d41:1::50 link#1 UHS lo0 2001:470:1d41:50::/64 link#3 U bridge0 2001:470:1d41:50::1 link#3 UHS lo0 fe80::/10 ::1 UGRS lo0 fe80::%vtnet0/64 link#1 U vtnet0 fe80::5a9c:fcff:fe01:719d%vtnet0 link#1 UHS lo0 fe80::%lo0/64 link#2 U lo0 fe80::1%lo0 link#2 UHS lo0 fe80::%bridge0/64 link#3 U bridge0 fe80::1%bridge0 link#3 UHS lo0 fe80::dd:a6ff:fe3d:7b00%bridge0 link#3 UHS lo0 ff02::/16 ::1 UGRS lo0
I've done some more testing with a cut-down ruleset and it looks like the problem is caused by the 'net.inet.ip.fw.nat64_direct_output=1' sysctl. I had set this to 1 to make sure the NAT64 traffic wasn't re-evaluated by ipfw so that I could still use the host IPv4 address locally (using a keep-state outgoing rule to avoid all the traffic being grabbed by the nat64 rule) however with a bit of fiddling with the ruleset I can do this using tags with 'net.inet.ip.fw.nat64_direct_output=0'. It does look like there is still a regression (this worked on 12.2) but could be a pretty unusual case.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=da3a09d8941dc29f20447e263b3a6d60370c6203 commit da3a09d8941dc29f20447e263b3a6d60370c6203 Author: Andrey V. Elsukov <ae@FreeBSD.org> AuthorDate: 2021-08-26 10:48:23 +0000 Commit: Andrey V. Elsukov <ae@FreeBSD.org> CommitDate: 2021-08-26 10:48:23 +0000 ipfw_nat64: fix direct output mode In nat64_find_route[46] handle NHF_GATEWAY flag and use destination address from next hop to do link layer address lookup. PR: 255928 Reviewed by: melifaro Obtained from: Yandex LLC MFC after: 1 week Sponsored by: Yandex LLC Differential Revision: https://reviews.freebsd.org/D31680 sys/netpfil/ipfw/nat64/nat64_translate.c | 32 ++++++++++++++------------------ 1 file changed, 14 insertions(+), 18 deletions(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=26302099fb9176a33936af002fa6de6864cea6b2 commit 26302099fb9176a33936af002fa6de6864cea6b2 Author: Andrey V. Elsukov <ae@FreeBSD.org> AuthorDate: 2021-08-26 10:48:23 +0000 Commit: Andrey V. Elsukov <ae@FreeBSD.org> CommitDate: 2021-09-03 06:35:43 +0000 ipfw_nat64: fix direct output mode In nat64_find_route[46] handle NHF_GATEWAY flag and use destination address from next hop to do link layer address lookup. PR: 255928 Reviewed by: melifaro Obtained from: Yandex LLC Sponsored by: Yandex LLC Differential Revision: https://reviews.freebsd.org/D31680 (cherry picked from commit da3a09d8941dc29f20447e263b3a6d60370c6203) sys/netpfil/ipfw/nat64/nat64_translate.c | 32 ++++++++++++++------------------ 1 file changed, 14 insertions(+), 18 deletions(-)
Fixed in main and stable/13. Thanks!