Bug 243590 - TCP ECN not adhering extremely strictly to RFC3168 can cause massive TCP perf issues
Summary: TCP ECN not adhering extremely strictly to RFC3168 can cause massive TCP perf...
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 11.2-STABLE
Hardware: Any Any
: --- Affects Some People
Assignee: Richard Scheffenegger
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-01-25 16:19 UTC by Richard Scheffenegger
Modified: 2020-06-24 16:18 UTC (History)
3 users (show)

See Also:
rscheff: maintainer-feedback-
rscheff: mfc-stable12?
rscheff: mfc-stable11+
rscheff: mfc-stable10-
rscheff: exp-run-


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Richard Scheffenegger freebsd_committer 2020-01-25 16:19:02 UTC
The TCP/IP ECN signaling loop looks like this

         -CE->   (latches ECE)
         <----  ACK, ECE
ACK, CWR ---->   (unlatches ECE)

However, RFC3168 has a SHOULD for sending the CWR with a *new* data packet.

New data packets usually are also marked as ECN-capable transport (ECT), while
control packets, pure ACKs and retransmissions are sent without IP ECT codepoint.

Some overly restrictive clients (Linux) explicitly only accept and process
CWR, if they are received on (new) data segments, but ignore them completely when set in other packets. While such an overly restrictive behavior is also problematic, BSD *should* adhere to the guidance of RFC3168 and sent out
CWR flags only when (new) data packets are sent out (sec 6.1.2) rather than simply the very next segment (sec. 6.1):

RFC3168, section 6.1:
  
      * The sender sets the CWR flag in the TCP header of the next
        packet sent to the receiver to acknowledge its receipt of and
        reaction to the ECN-Echo flag.

RFC3168, section 6.1.2:

   When an ECN-Capable TCP sender reduces its congestion window for any
   reason (because of a retransmit timeout, a Fast Retransmit, or in
   response to an ECN Notification), the TCP sender sets the CWR flag in
   the TCP header of the first new data packet sent after the window
   reduction. 


The interaction can lead to a race to the bottom, where cwnd is continually
decreased and kept at a very small value. This results in extremely poor
performance for ECN-enabled TCP sessions.
Comment 1 commit-hook freebsd_committer 2020-05-24 17:51:28 UTC
A commit references this bug:

Author: rscheff
Date: Sun May 24 17:51:14 UTC 2020
New revision: 361436
URL: https://svnweb.freebsd.org/changeset/base/361436

Log:
  MFC r361347: With RFC3168 ECN, CWR SHOULD only be sent with new data

  Overly conservative data receivers may ignore the CWR flag on other
  packets, and keep ECE latched. This can result in continous reduction
  of the congestion window, and very poor performance when ECN is
  enabled.

  This does NOT contain the merge of the change to RACK since at this
  time that code does not exist in stable/11, and there is no plan to
  merge RACK to stable/11.

  PR:		243590
  Reviewed by:	rgrimes (mentor), rrs
  Approved by:	rgrimes (mentor), tuexen (mentor)
  MFC after:	3 days
  Sponsored by:	NetApp, Inc.
  Differential Revision:	https://reviews.freebsd.org/D23364

Changes:
_U  stable/11/
  stable/11/sys/netinet/tcp_input.c
  stable/11/sys/netinet/tcp_output.c
Comment 2 Richard Scheffenegger freebsd_committer 2020-05-24 18:02:48 UTC
rS361347 fixes this issue in HEAD, scheduled for MFC to stable/12 soonish, after rS355273 has been MFC'd
Comment 3 commit-hook freebsd_committer 2020-05-27 22:35:22 UTC
A commit references this bug:

Author: rscheff
Date: Wed May 27 22:34:47 UTC 2020
New revision: 361565
URL: https://svnweb.freebsd.org/changeset/base/361565

Log:
  MFS r361436: MFC r361347: With RFC3168 ECN, CWR SHOULD only be sent with new data.

  Overly conservative data receivers may ignore the CWR flag on other
  packets, and keep ECE latched. This can result in continuous reduction
  of the congestion window, and very poor performance when ECN is
  enabled.

  This does NOT contain the merge of the change to RACK since at this
  time that code does not exist in stable/11, and there is no plan to
  merge RACK to stable/11.

  PR:		243590
  Reviewed by:	rgrimes (mentor), rrs
  Approved by:	re(gjb)
  Sponsored by:	NetApp, Inc.
  Differential Revision:	https://reviews.freebsd.org/D23364

Changes:
_U  releng/11.4/
  releng/11.4/sys/netinet/tcp_input.c
  releng/11.4/sys/netinet/tcp_output.c
Comment 4 commit-hook freebsd_committer 2020-06-24 16:18:17 UTC
A commit references this bug:

Author: rscheff
Date: Wed Jun 24 16:17:59 UTC 2020
New revision: 362586
URL: https://svnweb.freebsd.org/changeset/base/362586

Log:
  MFC r361347: With RFC3168 ECN, CWR SHOULD only be sent with new data

  Overly conservative data receivers may ignore the CWR flag on other
  packets, and keep ECE latched. This can result in continous reduction
  of the congestion window, and very poor performance when ECN is
  enabled.

  PR:		243590
  Reviewed by:	rgrimes (mentor), rrs
  Approved by:	rgrimes (mentor, blanket)
  MFC after:	3 weeks
  Sponsored by:	NetApp, Inc.
  Differential Revision:	https://reviews.freebsd.org/D23364

Changes:
_U  stable/12/
  stable/12/sys/netinet/tcp_input.c
  stable/12/sys/netinet/tcp_output.c
  stable/12/sys/netinet/tcp_stacks/rack.c