Bug 265089

Summary: Connection errors with IPv6 source address validation
Product: Base System Reporter: Frank Behrens <frank>
Component: kernAssignee: Alexander V. Chernikov <melifaro>
Status: Closed FIXED    
Severity: Affects Only Me CC: alster
Priority: ---    
Version: CURRENT   
Hardware: amd64   
OS: Any   

Description Frank Behrens 2022-07-08 12:10:06 UTC 
With a current kernel I get connection errors (UDP and TCP) on some destination address using IPv6. After a longer bisect session I determined commit 
https://cgit.freebsd.org/src/commit/sys/netinet6?id=1817be481b8703ae86730b151a6f49cc3022930f as possible reason.

On my server the address 2a01:170:1023::1:1 is assigned to a bridge and a bind(named) listens on that address. With IPv6 source address validation switched on, the is no local UDP connection possible and the TCP connection needs several seconds for connection setup due to repeated packets.


# sysctl net.inet6.ip6.source_address_validation=1
net.inet6.ip6.source_address_validation: 1 -> 1

# dig +tcp www.freebsd.org @2a01:170:1023::1:1
...
# tcpdump -nv -l -s0 -ttt -i lo0 port 53 and host 2a01:170:1023::1:1
tcpdump: listening on lo0, link-type NULL (BSD loopback), capture size 262144 bytes
 00:00:00.000000 IP6 (flowlabel 0x0954e, hlim 64, next-header TCP (6) payload length: 40) 2a01:170:1023::1:1.45851 > 2a01:170:1023::1:1.53: Flags [S], cksum 0x775a (incorrect -> 0x1ea5), seq 361203126, win 65535, options [mss 1220,nop,wscale 6,sackOK,TS val 1282807836 ecr 0], length 0
 00:00:01.007286 IP6 (flowlabel 0x0954e, hlim 64, next-header TCP (6) payload length: 40) 2a01:170:1023::1:1.45851 > 2a01:170:1023::1:1.53: Flags [S], cksum 0x775a (incorrect -> 0x1ab5), seq 361203126, win 65535, options [mss 1220,nop,wscale 6,sackOK,TS val 1282808844 ecr 0], length 0
 00:00:02.198040 IP6 (flowlabel 0x0954e, hlim 64, next-header TCP (6) payload length: 40) 2a01:170:1023::1:1.45851 > 2a01:170:1023::1:1.53: Flags [S], cksum 0x775a (incorrect -> 0x121f), seq 361203126, win 65535, options [mss 1220,nop,wscale 6,sackOK,TS val 1282811042 ecr 0], length 0
 00:00:00.000036 IP6 (flowlabel 0xf1abb, hlim 64, next-header TCP (6) payload length: 40) 2a01:170:1023::1:1.53 > 2a01:170:1023::1:1.45851: Flags [S.], cksum 0x775a (incorrect -> 0x07de), seq 3974044347, ack 361203127, win 65535, options [mss 1220,nop,wscale 6,sackOK,TS val 339537496 ecr 1282811042], length 0
 00:00:00.000023 IP6 (flowlabel 0x0954e, hlim 64, next-header TCP (6) payload length: 32) 2a01:170:1023::1:1.45851 > 2a01:170:1023::1:1.53: Flags [.], cksum 0x7752 (incorrect -> 0x31b3), ack 1, win 1030, options [nop,nop,TS val 1282811042 ecr 339537496], length 0
 00:00:00.000212 IP6 (flowlabel 0x0954e, hlim 64, next-header TCP (6) payload length: 90) 2a01:170:1023::1:1.45851 > 2a01:170:1023::1:1.53: Flags [P.], cksum 0x778c (incorrect -> 0x83c0), seq 1:59, ack 1, win 1030, options [nop,nop,TS val 1282811042 ecr 339537496], length 58 19397+ [1au] A? www.freebsd.org. (56)
 00:00:00.040982 IP6 (flowlabel 0xf1abb, hlim 64, next-header TCP (6) payload length: 32) 2a01:170:1023::1:1.53 > 2a01:170:1023::1:1.45851: Flags [.], cksum 0x7752 (incorrect -> 0x3150), ack 59, win 1030, options [nop,nop,TS val 339537537 ecr 1282811042], length 0
 00:00:00.011714 IP6 (flowlabel 0xf1abb, hlim 64, next-header TCP (6) payload length: 156) 2a01:170:1023::1:1.53 > 2a01:170:1023::1:1.45851: Flags [P.], cksum 0x77ce (incorrect -> 0xd425), seq 1:125, ack 59, win 1030, options [nop,nop,TS val 339537546 ecr 1282811042], length 124 19397$ 2/0/1 www.freebsd.org. CNAME wfe2.nyi.freebsd.org., wfe2.nyi.freebsd.org. A 96.47.72.95 (122)
 00:00:00.180334 IP6 (flowlabel 0x0954e, hlim 64, next-header TCP (6) payload length: 90) 2a01:170:1023::1:1.45851 > 2a01:170:1023::1:1.53: Flags [P.], cksum 0x778c (incorrect -> 0x82d7), seq 1:59, ack 1, win 1030, options [nop,nop,TS val 1282811275 ecr 339537496], length 58 19397+ [1au] A? www.freebsd.org. (56)
 00:00:00.048581 IP6 (flowlabel 0xf1abb, hlim 64, next-header TCP (6) payload length: 168) 2a01:170:1023::1:1.53 > 2a01:170:1023::1:1.45851: Flags [P.], cksum 0x77da (incorrect -> 0x6586), seq 1:125, ack 59, win 1030, options [nop,nop,TS val 339537778 ecr 1282811275,nop,nop,sack 1 {1:59}], length 124 19397$ 2/0/1 www.freebsd.org. CNAME wfe2.nyi.freebsd.org., wfe2.nyi.freebsd.org. A 96.47.72.95 (122)
 00:00:00.001475 IP6 (flowlabel 0x0954e, hlim 64, next-header TCP (6) payload length: 32) 2a01:170:1023::1:1.45851 > 2a01:170:1023::1:1.53: Flags [F.], cksum 0x7752 (incorrect -> 0x2ec8), seq 59, ack 125, win 1030, options [nop,nop,TS val 1282811324 ecr 339537778], length 0
 00:00:00.000026 IP6 (flowlabel 0xf1abb, hlim 64, next-header TCP (6) payload length: 32) 2a01:170:1023::1:1.53 > 2a01:170:1023::1:1.45851: Flags [.], cksum 0x7752 (incorrect -> 0x2ec8), ack 60, win 1030, options [nop,nop,TS val 339537778 ecr 1282811324], length 0
 00:00:00.000064 IP6 (flowlabel 0xf1abb, hlim 64, next-header TCP (6) payload length: 32) 2a01:170:1023::1:1.53 > 2a01:170:1023::1:1.45851: Flags [F.], cksum 0x7752 (incorrect -> 0x2ec7), seq 125, ack 60, win 1030, options [nop,nop,TS val 339537778 ecr 1282811324], length 0
 00:00:00.000018 IP6 (flowlabel 0x0954e, hlim 64, next-header TCP (6) payload length: 32) 2a01:170:1023::1:1.45851 > 2a01:170:1023::1:1.53: Flags [.], cksum 0x7752 (incorrect -> 0x2ec8), ack 126, win 1029, options [nop,nop,TS val 1282811324 ecr 339537778], length 0

# route -vn6 get 2a01:170:1023::1:1
RTA_DST: inet6 2a01:170:1023::1:1; RTA_IFP: link ; RTM_GET: Report Metrics: len 240, pid: 0, seq 1, errno 0, flags:<UP,GATEWAY,HOST,STATIC>
locks:  inits:
sockaddrs: <DST,IFP>
 2a01:170:1023::1:1 link#0
   route to: 2a01:170:1023::1:1
destination: 2a01:170:1023::1:1
        fib: 0
  interface: lo0
      flags: <UP,HOST,DONE,STATIC,PINNED>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0     16384         1         0

locks:  inits:
sockaddrs: <DST,GATEWAY,IFP,IFA>
 2a01:170:1023::1:1 link#4 lo0 ::1
 
# ifconfig bridge0
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ...
        inet6 2a01:170:1023::1:1 prefixlen 64

# ifconfig lo0
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
 

Without IPv6 sources address validation everything works as expected:

# sysctl net.inet6.ip6.source_address_validation=0
net.inet6.ip6.source_address_validation: 1 -> 0

# dig +tcp www.freebsd.org @2a01:170:1023::1:1
...
# tcpdump -nv -l -s0 -ttt -i lo0 port 53 and host 2a01:170:1023::1:1
tcpdump: listening on lo0, link-type NULL (BSD loopback), capture size 262144 bytes
 00:00:00.000000 IP6 (flowlabel 0xec843, hlim 64, next-header TCP (6) payload length: 40) 2a01:170:1023::1:1.22949 > 2a01:170:1023::1:1.53: Flags [S], cksum 0x775a (incorrect -> 0x1b12), seq 180291145, win 65535, options [mss 1220,nop,wscale 6,sackOK,TS val 414004004 ecr 0], length 0
 00:00:00.000040 IP6 (flowlabel 0xa7579, hlim 64, next-header TCP (6) payload length: 40) 2a01:170:1023::1:1.53 > 2a01:170:1023::1:1.22949: Flags [S.], cksum 0x775a (incorrect -> 0x7d2e), seq 774534521, ack 180291146, win 65535, options [mss 1220,nop,wscale 6,sackOK,TS val 4249746656 ecr 414004004], length 0
 00:00:00.000021 IP6 (flowlabel 0xec843, hlim 64, next-header TCP (6) payload length: 32) 2a01:170:1023::1:1.22949 > 2a01:170:1023::1:1.53: Flags [.], cksum 0x7752 (incorrect -> 0xa703), ack 1, win 1030, options [nop,nop,TS val 414004004 ecr 4249746656], length 0
 00:00:00.000116 IP6 (flowlabel 0xec843, hlim 64, next-header TCP (6) payload length: 90) 2a01:170:1023::1:1.22949 > 2a01:170:1023::1:1.53: Flags [P.], cksum 0x778c (incorrect -> 0x2098), seq 1:59, ack 1, win 1030, options [nop,nop,TS val 414004004 ecr 4249746656], length 58 40814+ [1au] A? www.freebsd.org. (56)
 00:00:00.036556 IP6 (flowlabel 0xa7579, hlim 64, next-header TCP (6) payload length: 32) 2a01:170:1023::1:1.53 > 2a01:170:1023::1:1.22949: Flags [.], cksum 0x7752 (incorrect -> 0xa69f), ack 59, win 1030, options [nop,nop,TS val 4249746698 ecr 414004004], length 0
 00:00:00.007160 IP6 (flowlabel 0xa7579, hlim 64, next-header TCP (6) payload length: 156) 2a01:170:1023::1:1.53 > 2a01:170:1023::1:1.22949: Flags [P.], cksum 0x77ce (incorrect -> 0x3ec0), seq 1:125, ack 59, win 1030, options [nop,nop,TS val 4249746704 ecr 414004004], length 124 40814$ 2/0/1 www.freebsd.org. CNAME wfe2.nyi.freebsd.org., wfe2.nyi.freebsd.org. A 96.47.72.95 (122)
 00:00:00.001365 IP6 (flowlabel 0xec843, hlim 64, next-header TCP (6) payload length: 32) 2a01:170:1023::1:1.22949 > 2a01:170:1023::1:1.53: Flags [F.], cksum 0x7752 (incorrect -> 0xa5ec), seq 59, ack 125, win 1030, options [nop,nop,TS val 414004052 ecr 4249746704], length 0
 00:00:00.000024 IP6 (flowlabel 0xa7579, hlim 64, next-header TCP (6) payload length: 32) 2a01:170:1023::1:1.53 > 2a01:170:1023::1:1.22949: Flags [.], cksum 0x7752 (incorrect -> 0xa5ec), ack 60, win 1030, options [nop,nop,TS val 4249746704 ecr 414004052], length 0
 00:00:00.000060 IP6 (flowlabel 0xa7579, hlim 64, next-header TCP (6) payload length: 32) 2a01:170:1023::1:1.53 > 2a01:170:1023::1:1.22949: Flags [F.], cksum 0x7752 (incorrect -> 0xa5eb), seq 125, ack 60, win 1030, options [nop,nop,TS val 4249746704 ecr 414004052], length 0
 00:00:00.000017 IP6 (flowlabel 0xec843, hlim 64, next-header TCP (6) payload length: 32) 2a01:170:1023::1:1.22949 > 2a01:170:1023::1:1.53: Flags [.], cksum 0x7752 (incorrect -> 0xa5ec), ack 126, win 1029, options [nop,nop,TS val 414004052 ecr 4249746704], length 0
Comment 1 commit-hook freebsd_committer freebsd_triage 2022-07-10 12:48:45 UTC 
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=50fa27e795eaae97dae87ac4532799f7aea87e9f

commit 50fa27e795eaae97dae87ac4532799f7aea87e9f
Author:     Alexander V. Chernikov <melifaro@FreeBSD.org>
AuthorDate: 2022-07-10 12:27:23 +0000
Commit:     Alexander V. Chernikov <melifaro@FreeBSD.org>
CommitDate: 2022-07-10 12:47:47 +0000

    netinet6: fix interface handling for loopback traffic

    Currently, processing of IPv6 local traffic is partially broken:
     link-local connection fails and global unicast connect() takes
     3 seconds to complete.
    This happens due to the combination of multiple factors.
    IPv6 code passes original interface "origifp" when passing
    traffic via loopack to retain the scope that is mandatory for the
    correct hadling of link-local traffic. First problem is that the logic
    of passing source interface is not working correcly for TCP connections,
    resulting in passing "origifp" on the first 2 connection attempts and
    lo0 on the subsequent ones. Second problem is that source address
    validation logic skips its checks iff the source interface is loopback,
    which doesn't cover "origifp" case.
    More detailed description is available at https://reviews.freebsd.org/D35732

    Fix the first problem by untangling&simplifying ifp/origifp logic.
    Fix the second problem by switching source address validation check to
    using M_LOOP mbuf flag instead of interface type.

    PR:             265089
    Reviewed by:    ae, bz(previous version)
    Differential Revision:  https://reviews.freebsd.org/D35732
    MFC after:      2 weeks

 sys/netinet6/ip6_input.c              |  2 +-
 sys/netinet6/ip6_output.c             | 61 ++++++++++++++++++-----------------
 tests/sys/netinet6/test_ip6_output.py |  4 +--
 3 files changed, 35 insertions(+), 32 deletions(-)
Comment 2 Frank Behrens 2022-07-11 11:55:37 UTC 
Hello Alexander, thanks for your fast reponse!
I confirm, that your patch fixed the problem on my system. From my side the bug may be closed.
Kind regards, Frank