Created attachment 272580 [details]
Proposed patch fixing the issue

IPv6 does not work over IPoIB between FreeBSD and Linux. On an InfiniBand
fabric with IPoIB interfaces up on both sides, ping6 from a Linux host
(AlmaLinux 10, kernel 6.x) to a FreeBSD host gets 100% packet loss and the
Linux neighbor entry for the FreeBSD address stays FAILED. IPv4 ARP on the
same interfaces works fine. Since IPv6 neighbor resolution never completes,
everything above it is dead too: rdma_cm cannot resolve IPv6 addresses to
GIDs, so NFS-over-RDMA-over-InfiniBand is unusable with IPv6.

The problem is in how FreeBSD builds and parses the ND link-layer address
option on IPoIB. IPoIB uses 20-byte hardware addresses, and RFC 4391
(section 9.3) says the option must be: Type (1 byte), Length (1 byte) = 3,
then two zero octets, then the 20-byte address ("two octets of zero MUST be
prepended to fill the total option length of 24 octets"). Linux does this
(net/ipv6/ndisc.c, ndisc_addr_option_pad() returns 2 for
ARPHRD_INFINIBAND).

FreeBSD assumes the address always starts right after the 2-byte option
header. That is correct for Ethernet (RFC 4861, 4.6.1), wrong for IPoIB.
The assumption is baked into both sides of the code:

- parse: nd6_ns_input(), nd6_na_input() (sys/netinet6/nd6_nbr.c),
  nd6_rs_input(), nd6_ra_opt_src_lladdr() (sys/netinet6/nd6_rtr.c),
  icmp6_redirect_input() (sys/netinet6/icmp6.c): all
  `lladdr = (char *)(opt + 1);`
- build: nd6_ns_output_fib(), nd6_na_output_fib() (nd6_nbr.c),
  icmp6_redirect_output() (icmp6.c): all `bcopy(mac, opt + 1, if_addrlen)`,
  which leaves the pad at the end of the option instead of the start.

Nothing catches this. The option-length sanity check
`((if_addrlen + 2 + 7) & ~7) != lladdrlen` computes 24 for
if_addrlen = 20 under either layout, so the bad options pass silently.

On the wire (tcpdump -XX, FreeBSD 16.0-CURRENT server, Linux client) the
failure looks like this:

1. The Linux NS carries the client's address at option offset 4, after the
   reserved field. FreeBSD reads offset 2 and caches a 2-byte-shifted
   garbage lladdr; the GID embedded in it does not exist on the fabric.
2. FreeBSD then sends its solicited NA (and its own unicast NS probes) to
   that nonexistent IPoIB destination. They never arrive: the client counts
   0 inbound packets, its neighbor entry stays FAILED, and FreeBSD's ndp
   never learns the client either.
3. FreeBSD's own NA/NS options carry its address at offset 2 with the pad
   trailing, so even delivered options would be mis-parsed by a conformant
   receiver. Both directions are broken.

FreeBSD-to-FreeBSD IPv6-over-IPoIB appears to work only because both ends
share the same non-conformant layout. The bug shows against any conformant
stack.

Instructions on how to reproduce:

    freebsd# ifconfig ib0 inet6 fc00:10:10::1/64 alias
    linux# ip -6 addr add fc00:10:10::2/64 dev ib0
    linux# ping6 -c3 fc00:10:10::1        # 100% loss
    linux# ip -6 neigh show dev ib0       # fc00:10:10::1 FAILED
    freebsd# tcpdump -ni ib0 -e -vv -XX icmp6
    # inbound NS option: 01 03 00 00 <20-byte addr>   (RFC 4391 layout)
    # outbound NA option: 02 03 <20-byte addr> 00 00  (address at wrong offset)

Proposed patch:

Patch attached (against main). It adds nd6_lladdr_opt_pad(ifp), the analog
of Linux's ndisc_addr_option_pad(): returns 2 for IFT_INFINIBAND /
IFT_INFINIBANDLAG, 0 otherwise. The five parse sites read the address at
(opt + 1) + pad, and the three build sites size the option as
hdr + pad + addrlen, zero it, and copy the address at (opt + 1) + pad.
While here: icmp6_redirect_output() previously did not zero the option
area, which leaked 2 uninitialized mbuf bytes in the trailing pad on
20-byte-addrlen links; the patch fixes that too. No functional change for
pad == 0 link types (Ethernet etc.).