Created attachment 272594 [details]
a reproducer of the bug in C

RFC 5961 section 3.2 requires that a RST whose sequence number exactly equals
RCV.NXT resets the connection, and that an in-window RST whose sequence number is
NOT RCV.NXT draws a challenge ACK instead. FreeBSD's check implements the
exact-match test against last_ack_sent rather than rcv_nxt.

When the receiver has received data it has not yet ACKed (delayed/stretch ACK, so
rcv_nxt > last_ack_sent), an incoming RST carrying SEG.SEQ == rcv_nxt -- which is
exactly what a correct peer sends (SND.NXT) -- matches neither last_ack_sent nor
the insecure-RST override, so it is treated as in-window-but-not-exact and
answered with a challenge ACK instead of a reset. The connection stays
ESTABLISHED (half-open):

  - its EVFILT_READ registration never fires,
  - getpeername() still succeeds,
  - getsockopt(SO_ERROR) returns 0,
  - recv(..., MSG_PEEK) returns EAGAIN (errno 35).

The peer that sent the RST is correct -- the RST goes out with SEG.SEQ ==
SND.NXT == the receiver's rcv_nxt, exactly RFC-conformant. It is the receiver
that declines to reset. The originating side has already fully closed (its tcpcb
is gone after the abortive close()), so nothing retransmits the RST; the receiver
simply stays ESTABLISHED until the application gives up and closes the socket
itself. In the capture below that is a FIN ~3 s later -- long after an
application timeout has already observed the hang.


ROOT CAUSE

sys/netinet/tcp_input.c, RFC 5961 section 3.2 handling (FreeBSD 15.1-RELEASE, the
"if (thflags & TH_RST)" block at line 2148, check at line 2160):

    if ((SEQ_GEQ(th->th_seq, tp->last_ack_sent) &&
         SEQ_LT(th->th_seq, tp->last_ack_sent + tp->rcv_wnd)) ||
        (tp->rcv_wnd == 0 && tp->last_ack_sent == th->th_seq)) {
        ...
        if (V_tcp_insecure_rst ||
            tp->last_ack_sent == th->th_seq) {   // <-- should also accept rcv_nxt
            /* reset the connection */
        } else {
            tcp_send_challenge_ack(tp, th, m);   // <-- taken for a valid RST at rcv_nxt
        }
    }
    goto drop;

The inner exact-match test is last_ack_sent == th_seq. The code comment above it
even records the deliberate-but-incomplete choice: "Note: to take into account
delayed ACKs, we should test against last_ack_sent instead of rcv_nxt." It tests
last_ack_sent ONLY, and so misses the RFC 5961-mandated RCV.NXT match. With the
default net.inet.tcp.insecure_rst=0, the connection resets iff last_ack_sent ==
th_seq; when rcv_nxt > last_ack_sent, a RST at rcv_nxt is challenge-ACKed instead
of resetting.


GROUND-TRUTH PROOF (DTrace, at the moment of the drop)

dtrace/rst_drop.d hooks fbt::tcp_do_segment:entry and prints the receiver's TCP
control block for any RST that fails to reset an ESTABLISHED connection
(last_ack_sent != th_seq). One hang, captured natively on FreeBSD
15.1-RELEASE/amd64:

  HANG-RST th_seq=1888935822 rcv_nxt=1888935822 last_ack_sent=1888919490 rcv_wnd=16640
    | seq-rcv_nxt=0  seq-last_ack_sent=16332  cur_edge(las+wnd)-seq=+308  fix_edge(rcvnxt+wnd)-seq=+16640

  - seq - rcv_nxt = 0: the RST sits exactly at rcv_nxt -- the final data was
    already absorbed; only the ACK was delayed. Per RFC 5961 this MUST reset.
  - seq - last_ack_sent = 16332: last_ack_sent lags rcv_nxt by one full segment
    (delayed ACK). Because the code tests last_ack_sent, not rcv_nxt, the reset
    is declined.
  - cur_edge(las+wnd)-seq = +308: the RST is inside the current window check
    (positive margin) -- so this is NOT a window/right-edge drop. The outer
    clause passes; the inner exact-match is what fails.


ON-THE-WIRE PROOF (same connection, same run)

lo0, absolute sequence numbers (tcpdump -S). 52374 is the closing peer, 22467 is
the peer left ESTABLISHED:

  52374->22467  . seq 1888919490:1888935822, length 16332   # closer's final full segment
  52374->22467  R. seq 1888935822                            # the RST (seq == rcv_nxt)
  22467->52374  . ack 1888935822, win 175                    # CHALLENGE ACK -- not a reset
     ... 3.07 s later ...
  22467->52374  F. seq 1850944913, ack 1888935822            # app gives up and closes (FIN)
  52374->22467  R  seq 1888935822                            # closer's host RSTs the orphan FIN

The RST's seq (1888935822) matches the DTrace th_seq exactly. The receiver's
reply is a challenge ACK (ack 1888935822), not a reset; the connection stays up
for ~3 s until the application closes it.


SUGGESTED FIX

Two changes to the RFC 5961 RST handling are required; a patched kernel with only
the first still hangs (see VALIDATION below).

1. Accept rcv_nxt as an exact match in the reset test -- the RFC 5961-mandated
   point -- in addition to the existing last_ack_sent leniency:

     			if (V_tcp_insecure_rst ||
     -			    tp->last_ack_sent == th->th_seq) {
     +			    tp->last_ack_sent == th->th_seq ||
     +			    tp->rcv_nxt == th->th_seq) {
     				TCPSTAT_INC(tcps_drops);
     				/* Drop the connection. */

   This is strictly more RFC 5961-conformant (a RST at RCV.NXT MUST reset) and
   does not weaken the anti-spoofing intent: an off-path attacker must still guess
   rcv_nxt (or last_ack_sent) exactly; in-window/non-exact RSTs still draw a
   challenge ACK.

2. Anchor the outer window clause's right edge at rcv_nxt. When the delayed-ACK
   gap (rcv_nxt - last_ack_sent) exceeds rcv_wnd, the current right edge
   (last_ack_sent + rcv_wnd) falls below a RST at rcv_nxt, so the outer clause
   silently drops it before the inner test is ever reached. rcv_nxt + rcv_wnd is
   the true right edge of the receive window:

     		if ((SEQ_GEQ(th->th_seq, tp->last_ack_sent) &&
     -		    SEQ_LT(th->th_seq, tp->last_ack_sent + tp->rcv_wnd)) ||
     +		    SEQ_LT(th->th_seq, tp->rcv_nxt + tp->rcv_wnd)) ||
     		    (tp->rcv_wnd == 0 && tp->last_ack_sent == th->th_seq)) {


PRIOR ART: the sibling BSDs already anchor on rcv_nxt

Anchoring RST validation on rcv_nxt is not novel -- both other kqueue BSDs
adopted it years ago, each landing the change with a commit message that names
the delayed-ACK RST failure directly:

  - OpenBSD -- markus, 2005-12-01 and 2006-12-11. Accepts a RST whose seq matches
    last_ack_sent, rcv_nxt, or rcv_nxt + 1. "allow RST if the th_seq matches
    rcv_nxt in case the RST follows the data immediately. otherwise we would
    ignore RST for delayed acks."
    sys/netinet/tcp_input.c rev 1.194:
    https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/netinet/tcp_input.c.diff?r1=1.193&r2=1.194

  - NetBSD -- christos, 2009-06-20 (patch from Joanne M Mikkelson). Validates a
    RST against rcv_nxt exactly. "Don't check against the last ack received, but
    the expected sequence number. This makes RST handling independent of delayed
    ACK." (Following draft-ietf-tcpm-tcpsecure-11, which became RFC 5961.)
    sys/netinet/tcp_input.c rev 1.296:
    https://cvsweb.netbsd.org/bsdweb.cgi/src/sys/netinet/tcp_input.c.diff?r1=1.295&r2=1.296

Running the same c/repro.c under QEMU/KVM confirms the anchor is what matters --
the two rcv_nxt-anchored BSDs do not reproduce the hang, while the two that
anchor on last_ack_sent do (180 s run each):

  kernel          RST validation anchor                      repro run       hangs
  --------------  -----------------------------------------  --------------  -----
  OpenBSD 7.4     last_ack_sent / rcv_nxt / rcv_nxt + 1      1,512,615 conns  0
  NetBSD 9.3      rcv_nxt (exact)                              915,874 conns  0
  FreeBSD 15.1    last_ack_sent only                         (hangs, below)  --
  DragonFly 6.4.2 window on last_ack_sent, no challenge ACK   450,584 conns  0 so far
                  (DragonFly shares only the rare residual path)

FreeBSD's own in-tree comment has acknowledged the delayed-ACK gap since 2014
without acting on it. The change requested here brings FreeBSD in line with
deployed, reviewed prior art rather than introducing new behaviour.


VALIDATION (patched kernel)

Built into FreeBSD 15.1-RELEASE-p1 GENERIC kernels under QEMU/KVM and run against
c/repro.c. To remove any ambiguity about which kernel produced which result, the
two patched kernels carry distinct idents (verified live with uname -i):
RSTINNER (change 1 only) and RSTBOTH (changes 1 + 2).

  stock GENERIC        hangs -- e.g. 15 / 1,919,209 (~1 in 128k); DTrace
                       confirmed every hang is the challenge-ACK case (change 1's
                       target)
  RSTINNER (change 1)  primary hang gone, but the residual persists -- first hang
                       seen at 287,392 / 19,946,282 connections in separate runs
  RSTBOTH (1 + 2)      0 / 31,651,215

The residual is the outer-clause drop (change 2's target), captured on the wire
on RSTINNER (tcpdump -S; 45605 closes, 50033 is left hung):

  S->C  ack 3142883995, win 65 (=16640 B)          # last ACK: last_ack_sent = 3142883995
  C->S  . seq 3142883995:3142900327, length 16332  # final data segment (delayed-ACKed)
  C->S  R. seq 3142900327                           # RST at rcv_nxt (= snd_max)
  S->C  . ack 3142900327, win 320                   # server acks the data; RST silently dropped

last_ack_sent (3142883995) lags rcv_nxt (3142900327) by a full 16332-byte segment
(delayed ACK), and rcv_wnd has shrunk to ~308, so the RST's seq sits far beyond
last_ack_sent + rcv_wnd = 3142884303 -> the outer clause drops it before the
inner test. Change 2 (rcv_nxt + rcv_wnd = 3142900635 > seq) accepts it, then
change 1 resets. RSTBOTH eliminates it (0 across 31.65M connections, where
RSTINNER hit it within ~20M).

Caveat -- the residual is a timing-sensitive Heisenbug. It needs a tight race
(the RST landing after the receiver absorbs the final segment but before its
delayed ACK, with the window already below the gap), so its rate is extremely
bursty (a first hang anywhere from ~290k to >18M connections) and it is perturbed
by fbt DTrace on tcp_do_segment (which adds hot-path latency); the wire capture
above uses tcpdump's out-of-band BPF tap, which does not disturb it. The primary
hang is not timing-fragile (DTrace matched it exactly, 17/17).


HOW TO REPRODUCE

Self-contained C reproducer, no dependencies beyond libc:
https://github.com/graingert/asyncio-macos-loopback-rst-hang  (c/repro.c)

  cc -O2 -o repro c/repro.c
  ./repro 600

Each iteration, using only sockets + kqueue/kevent:

  1. Establish a loopback client + accepted server socket (both non-blocking).
  2. The server never reads, so the client's writes fill the server's receive
     buffer and the client's send buffer.
  3. Register the client fd (EVFILT_READ + EVFILT_WRITE); fill until send()
     returns EAGAIN.
  4. EV_DELETE the client's filters, run one kevent poll, then abortively close
     it (SO_LINGER {1,0} + close()), which emits a RST at SND.NXT.
  5. Register the server fd (EVFILT_READ); wait up to 1s for the disconnect.
  6. If the disconnect never arrives, the RST was challenge-ACKed instead of
     resetting. The program prints the half-open port pair and exits non-zero.

The kevent poll in step 4 lets the server absorb a final data segment without
ACKing it before the RST arrives (creating the rcv_nxt > last_ack_sent gap),
which raises the hit rate. asyncio does exactly this via call_soon, which is how
the bug was first found. The DTrace probe (dtrace/rst_drop.d) can be run
alongside to print the receiver's TCB at the drop.


EXPECTED RESULT

The abortive close() resets the peer -- a recv() returning ECONNRESET, surfaced
as a readable EVFILT_READ. This is what happens on the overwhelming majority of
iterations (and on Linux/epoll across many millions of iterations with zero
failures).


ACTUAL RESULT

Occasionally the peer never observes the disconnect within the timeout. Its
socket stays ESTABLISHED -- the reproducer confirms via getpeername() (succeeds),
SO_ERROR == 0, and recv(MSG_PEEK) == EAGAIN:

  UNDELIVERED: 20208<->63743 SO_ERROR=0 MSG_PEEK=errno 35

Observed rate on FreeBSD 15.1-RELEASE (amd64): 312 / 10,336,808 connections
(~1 in 33,000) in a single 600-second run. On macOS (Darwin, the other kqueue
platform, same RFC 5961 lineage) the rate is far higher -- up to ~28-46% on
macOS 26.


ENVIRONMENT

  - FreeBSD 15.1-RELEASE, GENERIC, amd64
    (FreeBSD 15.1-RELEASE releng/15.1-n283562-96841ea08dcf), on a GitHub-hosted
    vmactions/freebsd-vm guest.
  - Interface: loopback (lo0), IPv4 127.0.0.1.
  - net.inet.tcp.insecure_rst = 0 (default).
  - Not reproducible on Linux (epoll) across many millions of iterations.


RELATED REPORTS (same reproducer, other platforms)

  - Apple Feedback Assistant FB23590387 (macOS -- much higher rate, same
    challenge-ACK-instead-of-reset signature on the wire).
  - CPython python/cpython#153117 (surfaces as a silent asyncio hang, because
    asyncio defers close() via call_soon, which lets the server absorb an
    un-ACKed final segment before the RST).