Bug 263974 - ipfw_nat64lsn reply destination mac address error
Summary: ipfw_nat64lsn reply destination mac address error
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 13.0-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-ipfw (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-05-14 15:38 UTC by Jim B.
Modified: 2022-05-16 11:04 UTC (History)
2 users (show)

See Also:


Attachments
nat64lsn and dns64 test bed - 4 VMs on FreeBSD host (62.22 KB, image/png)
2022-05-14 15:38 UTC, Jim B.
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jim B. 2022-05-14 15:38:17 UTC
Created attachment 233913 [details]
nat64lsn and dns64 test bed - 4 VMs on FreeBSD host

See the attached figure (nat64_issue.png) for address layout.

The FreeBSD host (my laptop), v6only VM, firewall VM, and the DNShost VM are all 13.0-RELEASE-p11.  The external1 host (IPV4 only host) is 13.0-RELEASE-p8.

This nat64lsn (stateful NAT64) example follows the BSD Router Project address layout and ruleset.  However, these are stock FreeBSD 13.0 VMs (qemu-system-x86_64 version 6.2.0), not BSDRP images.


TCP session request from IPv6 only host to IPv4 only host *almost* works.  The initial SYN packet is NAT64ed correctly and reaches the destination IPv4 host who sends a SYN/ACK back.  The ipfw instance on the router moves the SYN/ACK packet back through the ruleset and writes it out the proper interface (em1).

But - the packet sent back to the IPV6 host on that interface has a malformed destination MAC address. ipfw_nat64 duplicated the em1 interface MAC address (the source) in the destination field:

Wireshark trace shows SYN/ACK reply packet has duplicated source and destination MAC addresses:


Ethernet II, Src: 02:49:50:46:57:42 (02:49:50:46:57:42), Dst: 02:49:50:46:57:42 (02:49:50:46:57:42)
    Destination: 02:49:50:46:57:42 (02:49:50:46:57:42)
        Address: 02:49:50:46:57:42 (02:49:50:46:57:42)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Source: 02:49:50:46:57:42 (02:49:50:46:57:42)
        Address: 02:49:50:46:57:42 (02:49:50:46:57:42)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: IPv6 (0x86dd)


IPV6 host config:
oot@v6only:~ # ifconfig em0
em0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=481209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
        ether 02:49:de:ad:be:ef
        inet6 2001:db8:12::1 prefixlen 64
        inet6 fe80::49:deff:fead:beef%em0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>


ipfw router config:
root@firewall:~ # ifconfig -a
em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=481209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
        ether 02:49:50:46:57:41
        inet 2.2.2.2 netmask 0xffffff00 broadcast 2.2.2.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
em1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=481209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
        ether 02:49:50:46:57:42
        inet6 2001:db8:12::2 prefixlen 64
        inet6 fe80::49:50ff:fe46:5742%em1 prefixlen 64 scopeid 0x2
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
root@firewall:~ # 

root@firewall:~ # ndp -a
Neighbor                             Linklayer Address  Netif Expire    S Flags
v6only.example.com                   02:49:de:ad:be:ef    em1 23h8m28s  S 
2001:db8:12::2                       02:49:50:46:57:42    em1 permanent R 
fe80::49:50ff:fe46:5742%em1          02:49:50:46:57:42    em1 permanent R 
dnshost.example.com                  02:49:53:53:54:54    em1 22h49m2s  S 
root@firewall:~ # 

ipfw setup:
root@firewall:~ # ipfw nat64lsn NAT64 show config
nat64lsn NAT64 prefix4 2.2.1.0/24 prefix6 64:ff9b::/96 log
root@firewall:~ # 
root@firewall:~ # ipfw show
00100 12  816 allow log ipv6-icmp from any to any icmp6types 135,136
00200  7  512 nat64lsn NAT64 log ip from 2001:db8:12::/64 to 64:ff9b::/96 in
00300 16  912 nat64lsn NAT64 log ip from any to 2.2.1.0/24 in
00400 58 5920 allow log ip from any to any
00500  0    0 allow log ip6 from any to any
65535  0    0 deny ip from any to any


startup script:

root@firewall:~ # cat bsdrp.sh 
#!/bin/sh
set -x

kldunload ipfw_nat64
kldunload ipfw

sleep 1

kldload ipfw
kldload ipfw_nat64

# Logging: 0 interfaces, 1 syslog
sysctl net.inet.ip.fw.verbose=1 

# Debug nat64
sysctl net.inet.ip.fw.nat64_debug=1

fwcmd="/sbin/ipfw"
${fwcmd} -f flush
${fwcmd} nat64lsn NAT64 create log  prefix4 2.2.1.0/24
${fwcmd} add allow log icmp6 from any to any icmp6types 135,136
${fwcmd} add nat64lsn NAT64 log ip from 2001:db8:12::/64 to 64:ff9b::/96 in
${fwcmd} add nat64lsn NAT64 log ip from any to 2.2.1.0/24 in
${fwcmd} add allow log ip from any to any
${fwcmd} add allow log ip6 from any to any

# Direct output: 1 enable, 0 disable (packet goes back into ruleset)
sysctl net.inet.ip.fw.nat64_direct_output=1


Note that I've been running two dozen or more different ipfw tests using this same testbed and I have not encountered a similar issue with MAC addresses.

I will double check all this when 13.1 lands.
Comment 1 Marek Zarychta 2022-05-14 15:47:58 UTC
Setting net.inet.ip.fw.nat64_direct_output=0 helped in my case.
Comment 2 Jim B. 2022-05-15 23:09:55 UTC
I replaced the FreeBSD 13.0-p11 firewall with an instance of the BSDP Router Project running on Qemu.

BSDRP Platform:
FreeBSD router.bsdrp.net 14.0-CURRENT FreeBSD 14.0-CURRENT BSDRP-AMD64  amd64

The architecture and ruleset were unchanged.

The nat64lsn instance works correctly and the communication is successful.

sysctl net.inet.ip.fw.nat64_direct_output=0  is required for the ruleset shown.

It seems that the problem is narrowed to ipfw_nat64 13.0 code.

I will check again with FreeBSD 13.1 when it drops.

Jim B.
Comment 3 Andrey V. Elsukov freebsd_committer 2022-05-16 11:01:25 UTC
This was probably fixed with https://reviews.freebsd.org/D31680