Bug 263974 - ipfw_nat64lsn reply destination mac address error
Summary: ipfw_nat64lsn reply destination mac address error
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 13.0-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-ipfw (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-05-14 15:38 UTC by Jim B.
Modified: 2022-05-17 03:19 UTC (History)
2 users (show)

See Also:


Attachments
nat64lsn and dns64 test bed - 4 VMs on FreeBSD host (62.22 KB, image/png)
2022-05-14 15:38 UTC, Jim B.
no flags Details
nat64lsn on my addressing scheme not working (56.51 KB, image/png)
2022-05-16 16:47 UTC, Jim B.
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jim B. 2022-05-14 15:38:17 UTC
Created attachment 233913 [details]
nat64lsn and dns64 test bed - 4 VMs on FreeBSD host

See the attached figure (nat64_issue.png) for address layout.

The FreeBSD host (my laptop), v6only VM, firewall VM, and the DNShost VM are all 13.0-RELEASE-p11.  The external1 host (IPV4 only host) is 13.0-RELEASE-p8.

This nat64lsn (stateful NAT64) example follows the BSD Router Project address layout and ruleset.  However, these are stock FreeBSD 13.0 VMs (qemu-system-x86_64 version 6.2.0), not BSDRP images.


TCP session request from IPv6 only host to IPv4 only host *almost* works.  The initial SYN packet is NAT64ed correctly and reaches the destination IPv4 host who sends a SYN/ACK back.  The ipfw instance on the router moves the SYN/ACK packet back through the ruleset and writes it out the proper interface (em1).

But - the packet sent back to the IPV6 host on that interface has a malformed destination MAC address. ipfw_nat64 duplicated the em1 interface MAC address (the source) in the destination field:

Wireshark trace shows SYN/ACK reply packet has duplicated source and destination MAC addresses:


Ethernet II, Src: 02:49:50:46:57:42 (02:49:50:46:57:42), Dst: 02:49:50:46:57:42 (02:49:50:46:57:42)
    Destination: 02:49:50:46:57:42 (02:49:50:46:57:42)
        Address: 02:49:50:46:57:42 (02:49:50:46:57:42)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Source: 02:49:50:46:57:42 (02:49:50:46:57:42)
        Address: 02:49:50:46:57:42 (02:49:50:46:57:42)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: IPv6 (0x86dd)


IPV6 host config:
oot@v6only:~ # ifconfig em0
em0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=481209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
        ether 02:49:de:ad:be:ef
        inet6 2001:db8:12::1 prefixlen 64
        inet6 fe80::49:deff:fead:beef%em0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>


ipfw router config:
root@firewall:~ # ifconfig -a
em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=481209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
        ether 02:49:50:46:57:41
        inet 2.2.2.2 netmask 0xffffff00 broadcast 2.2.2.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
em1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=481209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
        ether 02:49:50:46:57:42
        inet6 2001:db8:12::2 prefixlen 64
        inet6 fe80::49:50ff:fe46:5742%em1 prefixlen 64 scopeid 0x2
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
root@firewall:~ # 

root@firewall:~ # ndp -a
Neighbor                             Linklayer Address  Netif Expire    S Flags
v6only.example.com                   02:49:de:ad:be:ef    em1 23h8m28s  S 
2001:db8:12::2                       02:49:50:46:57:42    em1 permanent R 
fe80::49:50ff:fe46:5742%em1          02:49:50:46:57:42    em1 permanent R 
dnshost.example.com                  02:49:53:53:54:54    em1 22h49m2s  S 
root@firewall:~ # 

ipfw setup:
root@firewall:~ # ipfw nat64lsn NAT64 show config
nat64lsn NAT64 prefix4 2.2.1.0/24 prefix6 64:ff9b::/96 log
root@firewall:~ # 
root@firewall:~ # ipfw show
00100 12  816 allow log ipv6-icmp from any to any icmp6types 135,136
00200  7  512 nat64lsn NAT64 log ip from 2001:db8:12::/64 to 64:ff9b::/96 in
00300 16  912 nat64lsn NAT64 log ip from any to 2.2.1.0/24 in
00400 58 5920 allow log ip from any to any
00500  0    0 allow log ip6 from any to any
65535  0    0 deny ip from any to any


startup script:

root@firewall:~ # cat bsdrp.sh 
#!/bin/sh
set -x

kldunload ipfw_nat64
kldunload ipfw

sleep 1

kldload ipfw
kldload ipfw_nat64

# Logging: 0 interfaces, 1 syslog
sysctl net.inet.ip.fw.verbose=1 

# Debug nat64
sysctl net.inet.ip.fw.nat64_debug=1

fwcmd="/sbin/ipfw"
${fwcmd} -f flush
${fwcmd} nat64lsn NAT64 create log  prefix4 2.2.1.0/24
${fwcmd} add allow log icmp6 from any to any icmp6types 135,136
${fwcmd} add nat64lsn NAT64 log ip from 2001:db8:12::/64 to 64:ff9b::/96 in
${fwcmd} add nat64lsn NAT64 log ip from any to 2.2.1.0/24 in
${fwcmd} add allow log ip from any to any
${fwcmd} add allow log ip6 from any to any

# Direct output: 1 enable, 0 disable (packet goes back into ruleset)
sysctl net.inet.ip.fw.nat64_direct_output=1


Note that I've been running two dozen or more different ipfw tests using this same testbed and I have not encountered a similar issue with MAC addresses.

I will double check all this when 13.1 lands.
Comment 1 Marek Zarychta 2022-05-14 15:47:58 UTC
Setting net.inet.ip.fw.nat64_direct_output=0 helped in my case.
Comment 2 Jim B. 2022-05-15 23:09:55 UTC
I replaced the FreeBSD 13.0-p11 firewall with an instance of the BSDP Router Project running on Qemu.

BSDRP Platform:
FreeBSD router.bsdrp.net 14.0-CURRENT FreeBSD 14.0-CURRENT BSDRP-AMD64  amd64

The architecture and ruleset were unchanged.

The nat64lsn instance works correctly and the communication is successful.

sysctl net.inet.ip.fw.nat64_direct_output=0  is required for the ruleset shown.

It seems that the problem is narrowed to ipfw_nat64 13.0 code.

I will check again with FreeBSD 13.1 when it drops.

Jim B.
Comment 3 Andrey V. Elsukov freebsd_committer freebsd_triage 2022-05-16 11:01:25 UTC
This was probably fixed with https://reviews.freebsd.org/D31680
Comment 4 Jim B. 2022-05-16 16:47:31 UTC
Created attachment 233966 [details]
nat64lsn on my addressing scheme not working

In my comment #2 I explained that I substituted the BSD Router Project instance and got the example working with the addressing scheme layout shown here:
https://bsdrp.net/documentation/examples/nat64

But when I reconfigured my example to the layout shown in this attachment
(nat64lsn_myaddr_notworking.png)  the nat64lsn instance *stopped* working.

I'm going to spend some time going over both configurations with a fine tooth comb to ensure I didn't miss anything, but for now, I'll say I still have a problem.
Comment 5 Jim B. 2022-05-16 23:26:26 UTC
Ok, Comment #4 is solved by use of the "allow_private" keyword on the nat64lsn create statement.  While the ipfw(8) man page mentions that IPv4 RFC 1918 addresses are not processed, it does not mention RFC 5737 addresses (192.0.2.0/24, 198.51.100.0/24, and 203.0.113.0/24 - the one I was using.  Adding the "allow_private" keyword fixed that for the use of the BSD Router Project instance described in Comment #4.


However, going back and using my original 13.0-RELEASE-p11 VM instance, the addition of the "allow_private" keyword does not fix the duplicated source MAC address in the destination field in the reply.  

Now that 13.1 has dropped, I will spin up a VM and test with that.
Comment 6 Jim B. 2022-05-17 03:19:06 UTC
Ok, as promised in Comment #5, I spun up a new VM with 13.1-RELEASE and retested this issue.

I now report that nat64lsn (stateful NAT64) *does work* under the following conditions:

 * The addressing scheme is as described in the attachment "nat64lsn on my addressing scheme not working"  now *does work*.  

 * The ipfw rules to implement are:

ipfw nat64lsn foo create prefix4 203.0.112.0/24 allow_private
ipfw add allow log ipv6-icmp from any to any icmp6types 135,136
ipfw add nat64lsn foo log ip from 2001:db8:12::/64 to 64:ff9b::/96 in
ipfw add nat64lsn foo log ip from any to 203.0.112.0/24 in
ipfw add allow log ip from any to any


 * The direct_output sysctl had to be set to 1 (not zero):

sysctl net.inet.ip.fw.nat64_direct_output=1


 * I also set the nat64_debug sysctl and the firewall verbose sysctl:

sysctl net.inet.ip.fw.nat64_debug=1
sysctl net.inet.ip.fw.verbose=1

See /var/log/security for output.

-----

With these conditions the following tests were successful:

[root@v6only ~]# ping6 -c 3 64:ff9b::203.0.113.10
PING6(56=40+8+8 bytes) 2001:db8:12::30 --> 64:ff9b::cb00:710a
16 bytes from 64:ff9b::cb00:710a, icmp_seq=0 hlim=63 time=8.401 ms
16 bytes from 64:ff9b::cb00:710a, icmp_seq=1 hlim=63 time=3.429 ms
16 bytes from 64:ff9b::cb00:710a, icmp_seq=2 hlim=63 time=3.398 ms

--- 64:ff9b::203.0.113.10 ping6 statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 3.398/5.076/8.401/2.351 ms

And using lynx to grab the nginx home page was successful:

lynx external1.example.com
-------                                                                                Welcome to nginx!
                                         Welcome to nginx!

   If you see this page, the nginx web server is successfully installed and working. Further
   configuration is required.

   This is Machine 1 - 203.0.113.10

   For online documentation and support please refer to nginx.org.
   Commercial support is available at nginx.com.

   Thank you for using nginx.

Are you sure you want to quit? (y) 
  Arrow keys: Up and Down to move.  Right to follow a link; Left to go back.
-------

I am closing the ticket with the caveat that nat64lsn under 13.0 may still need fixing (identical source and destination MAC addresses in the reply).

Closed : Works as Intended  <--- but only in 13.1

Jim B.