Created attachment 233913 [details]
nat64lsn and dns64 test bed - 4 VMs on FreeBSD host

See the attached figure (nat64_issue.png) for address layout.

The FreeBSD host (my laptop), v6only VM, firewall VM, and the DNShost VM are all 13.0-RELEASE-p11.  The external1 host (IPV4 only host) is 13.0-RELEASE-p8.

This nat64lsn (stateful NAT64) example follows the BSD Router Project address layout and ruleset.  However, these are stock FreeBSD 13.0 VMs (qemu-system-x86_64 version 6.2.0), not BSDRP images.


TCP session request from IPv6 only host to IPv4 only host *almost* works.  The initial SYN packet is NAT64ed correctly and reaches the destination IPv4 host who sends a SYN/ACK back.  The ipfw instance on the router moves the SYN/ACK packet back through the ruleset and writes it out the proper interface (em1).

But - the packet sent back to the IPV6 host on that interface has a malformed destination MAC address. ipfw_nat64 duplicated the em1 interface MAC address (the source) in the destination field:

Wireshark trace shows SYN/ACK reply packet has duplicated source and destination MAC addresses:


Ethernet II, Src: 02:49:50:46:57:42 (02:49:50:46:57:42), Dst: 02:49:50:46:57:42 (02:49:50:46:57:42)
    Destination: 02:49:50:46:57:42 (02:49:50:46:57:42)
        Address: 02:49:50:46:57:42 (02:49:50:46:57:42)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Source: 02:49:50:46:57:42 (02:49:50:46:57:42)
        Address: 02:49:50:46:57:42 (02:49:50:46:57:42)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: IPv6 (0x86dd)


IPV6 host config:
oot@v6only:~ # ifconfig em0
em0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=481209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
        ether 02:49:de:ad:be:ef
        inet6 2001:db8:12::1 prefixlen 64
        inet6 fe80::49:deff:fead:beef%em0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>


ipfw router config:
root@firewall:~ # ifconfig -a
em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=481209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
        ether 02:49:50:46:57:41
        inet 2.2.2.2 netmask 0xffffff00 broadcast 2.2.2.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
em1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=481209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
        ether 02:49:50:46:57:42
        inet6 2001:db8:12::2 prefixlen 64
        inet6 fe80::49:50ff:fe46:5742%em1 prefixlen 64 scopeid 0x2
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
root@firewall:~ # 

root@firewall:~ # ndp -a
Neighbor                             Linklayer Address  Netif Expire    S Flags
v6only.example.com                   02:49:de:ad:be:ef    em1 23h8m28s  S 
2001:db8:12::2                       02:49:50:46:57:42    em1 permanent R 
fe80::49:50ff:fe46:5742%em1          02:49:50:46:57:42    em1 permanent R 
dnshost.example.com                  02:49:53:53:54:54    em1 22h49m2s  S 
root@firewall:~ # 

ipfw setup:
root@firewall:~ # ipfw nat64lsn NAT64 show config
nat64lsn NAT64 prefix4 2.2.1.0/24 prefix6 64:ff9b::/96 log
root@firewall:~ # 
root@firewall:~ # ipfw show
00100 12  816 allow log ipv6-icmp from any to any icmp6types 135,136
00200  7  512 nat64lsn NAT64 log ip from 2001:db8:12::/64 to 64:ff9b::/96 in
00300 16  912 nat64lsn NAT64 log ip from any to 2.2.1.0/24 in
00400 58 5920 allow log ip from any to any
00500  0    0 allow log ip6 from any to any
65535  0    0 deny ip from any to any


startup script:

root@firewall:~ # cat bsdrp.sh 
#!/bin/sh
set -x

kldunload ipfw_nat64
kldunload ipfw

sleep 1

kldload ipfw
kldload ipfw_nat64

# Logging: 0 interfaces, 1 syslog
sysctl net.inet.ip.fw.verbose=1 

# Debug nat64
sysctl net.inet.ip.fw.nat64_debug=1

fwcmd="/sbin/ipfw"
${fwcmd} -f flush
${fwcmd} nat64lsn NAT64 create log  prefix4 2.2.1.0/24
${fwcmd} add allow log icmp6 from any to any icmp6types 135,136
${fwcmd} add nat64lsn NAT64 log ip from 2001:db8:12::/64 to 64:ff9b::/96 in
${fwcmd} add nat64lsn NAT64 log ip from any to 2.2.1.0/24 in
${fwcmd} add allow log ip from any to any
${fwcmd} add allow log ip6 from any to any

# Direct output: 1 enable, 0 disable (packet goes back into ruleset)
sysctl net.inet.ip.fw.nat64_direct_output=1


Note that I've been running two dozen or more different ipfw tests using this same testbed and I have not encountered a similar issue with MAC addresses.

I will double check all this when 13.1 lands.