Through troubleshooting and subsequent source code verification (sys/netinet/if_ether.c) it appears that FreeBSD does not perform proxy ARP for ARP requests whose target host reside in any FIB other than FIB 0. Code at line 1073 indicates this: /* XXX MRT use table 0 for arp checks */ if (fib4_lookup_nh_basic(0, isaddr, 0, 0, &nh4) != 0) goto drop; This seems to be an incomplete implementation of the FIB functionality. FreeBSD should 1) lookup the FIB of the interface on which the ARP request arrived 2) perform a route lookup of the target in correct FIB 3) perform sanity checks 4) proxy-reply to the ARP request accordingly Without this, operations like dropping some cloned interfaces into a non-zero FIB are futile. Thanks.
^Triage: Looks #1083 came in via base r292015 request feedback from committer
Thank you for the report Scott Can you provide a network configuration (rc.conf , netstat -rn, ifconfig -a output as attachments, sanitized if necessary) that reproduces the issue?
Thank you for the report, Scott. Let me setup the scenario and take a look.
I think the code at line 1051 is actually the culprit: /* XXX MRT use table 0 for arp reply */ if (fib4_lookup_nh_basic(0, itaddr, 0, 0, &nh4) != 0) goto drop; I believe this is where the target of the ARP request is looked up (line 1073 also validates the source - which would also require fixing as it only checks FIB 0). TESTING SETUP: Host_1: IP 10.1.1.60, constantly pinging 10.1.4.99 via router (.1) Host_2: IP 10.1.4.20/24 on vmx0, 10.1.4.99/32 on lo1, default via 10.1.4.1 Host_2: # sysctl net.add_addr_allfibs net.inet.ip.forwarding net.link.ether.inet.proxyall net.fibs net.add_addr_allfibs: 0 net.inet.ip.forwarding: 1 net.link.ether.inet.proxyall: 1 net.fibs: 2 TESTS (run on Host_2): --- Test FIB 0 and proxy ARP: # ifconfig vmx0 vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=e403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 00:50:56:8a:93:4e inet 10.1.4.20 netmask 0xffffff00 broadcast 10.1.4.255 media: Ethernet autoselect status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> # netstat -rn4 Routing tables Internet: Destination Gateway Flags Netif Expire default 10.1.4.1 UGS vmx0 10.1.4.0/24 link#1 U vmx0 10.1.4.20 link#1 UHS lo0 127.0.0.1 link#2 UH lo0 # ifconfig lo1 create inet 10.1.4.99/32 (ping 10.1.4.99 succeeds as expected) --- Test FIB 0 and NO proxy ARP: # sysctl net.link.ether.inet.proxyall=0 net.link.ether.inet.proxyall: 1 -> 0 (ARP entry on router for 10.1.4.99 deleted) (ping 10.1.4.99 fails as expected) --- Test FIB 0 and proxy ARP restored: # sysctl net.link.ether.inet.proxyall=1 net.link.ether.inet.proxyall: 0 -> 1 (ping 10.1.4.99 succeeds as expected) All test results as expected. --- Test FIB 1 and proxy ARP: # ifconfig vmx0 inet delete # ifconfig vmx0 inet 10.1.4.20/24 fib 1 # ifconfig lo1 destroy # ifconfig lo1 create inet 10.1.4.99/32 fib 1 # setfib 1 route add default 10.1.4.1 add net default: gateway 10.1.4.1 fib 1 # netstat -rn4 Routing tables Internet: Destination Gateway Flags Netif Expire 127.0.0.1 link#2 UH lo0 # setfib 1 netstat -rn4 Routing tables (fib: 1) Internet: Destination Gateway Flags Netif Expire default 10.1.4.1 UGS vmx0 10.1.4.0/24 link#1 U vmx0 10.1.4.20 link#1 UHS lo0 10.1.4.99 link#4 UH lo1 127.0.0.1 lo0 UHS lo0 (ARP entry on router for 10.1.4.99 deleted) (ping 10.1.4.99 fails NOT as expected) # tcpdump -i vmx0 arp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vmx0, link-type EN10MB (Ethernet), capture size 262144 bytes 12:04:28.438590 ARP, Request who-has 10.1.4.99 tell 10.1.4.1, length 46
Created attachment 212895 [details] ARP FIB patch The attatched patch worked for me. Some caveats: 1) I'm not a good progranner 2) The patch is noisy and should have the logging removed 3) I have no idea what negative interactions it may have (as per point 1) Scott
Yep, looking good. I've moved the patch to https://reviews.freebsd.org/D24244 . Will commit tomorrow.
A commit references this bug: Author: melifaro Date: Thu Apr 2 20:06:37 UTC 2020 New revision: 359580 URL: https://svnweb.freebsd.org/changeset/base/359580 Log: Use interface fib for proxyarp checks. Before the change, proxyarp checks for src and dst addresses were performed using default fib, breaking multi-fib scenario. PR: 245181 Submitted by: Scott Aitken (original version) MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D24244 Changes: head/sys/netinet/if_ether.c
^Triage: committed back in 2020. The mfc-stable* flags' values are now OBE.