Bug 211003

Summary: net.inet.tcp.delayed_ack=0 delaying local tcp requests
Product: Base System Reporter: Konstantin Morshnev <moko>
Component: kernAssignee: Michael Tuexen <tuexen>
Status: Closed FIXED    
Severity: Affects Many People CC: admin, hiren, rkoberman, tuexen
Priority: --- Keywords: regression
Version: 10.2-STABLE   
Hardware: Any   
OS: Any   
Attachments:
Description Flags
packetdrill script reproducing the issue none

Description Konstantin Morshnev 2016-07-11 12:37:21 UTC
We have many FreeBSD installations and some of them historically have the following settings in /etc/sysctl.conf:

net.inet.tcp.delayed_ack=0
net.inet.tcp.recvspace=8192
net.inet.tcp.sendspace=150000

After upgrading to 10.2 we've run into an issue - local http requests were delaying after every 8kb of data for 5 seconds.

lynx --mime_header "http://<server ip>/any file around 200kb"
first we get 4Kb (incuding headers)
5 seconds delay (exact value, not an approximate)
next 8Kb
5 seconds delay
next 8Kb
... and so on.

Everything is fine if the file is requested remotely, only local requests have such delays. Nginx or apache - both have the delays. We also where able to reproduce the delays in virtualized environment. There are no issues with this sysctl settings in FreeBSD 8.X and FreeBSD 9.X.

Setting net.inet.tcp.sendspace=4096 "hides" the issue. Increasing net.inet.tcp.recvspace to more then 32664 also "hides" the issue. Restoring net.inet.tcp.delayed_ack to default 1 also "hides" the issue but there are still some performance issues if net.inet.tcp.recvspace is low:

>sysctl net.inet.tcp.recvspace=8192
>time lynx --mime_header "local file around 200 kb" > /dev/null
0.009u 0.000s 0:01.41 0.0%	0+0k 0+0io 0pf+0w

>sysctl net.inet.tcp.recvspace=32768
>time lynx --mime_header "local file around 200 kb" > /dev/null
0.009u 0.000s 0:00.01 0.0%	0+0k 0+0io 0pf+0w
Comment 1 Hiren Panchasara freebsd_committer 2016-12-23 07:36:00 UTC
Exact 5s delay is very interesting. It usually points to persist timer but not sure how to relate that here. Looks like lynx is not draining the recv space fast enough? I am not quite sure but if possible, can you try 1) capturing the transfer 2) something other than lynx on recv side 3) -HEAD or 11?
Comment 2 rkoberman 2016-12-24 02:05:08 UTC
You might look at using siftr(4) to collect statistics on the transfer. It is often overlooked, but is a part of the base system and captures detailed information on TCP activity. This can be very useful for analyzing TCP performance issues. The man page is quite detailed.
Comment 3 Konstantin Morshnev 2017-02-02 12:57:55 UTC
Note: we've also tested FreeBSD 11, and can confirm the same behavior. By default all is fine, but just after you change

net.inet.tcp.delayed_ack=0
net.inet.tcp.recvspace=8192
net.inet.tcp.sendspace=150000

local http requests are delaying after every 8kb of data for 5 seconds.

> 2) something other than lynx on recv side

Initially we have found this issue between apache and nginx, and we've also used telnet, so it's for sure not related to lynx.

> You might look at using siftr(4) to collect statistics on the transfer.

Done.

enable_time_secs=1486035625     enable_time_usecs=746854        siftrver=1.2.4  hz=1000 tcp_rtt_scale=32        sysname=FreeBSD sysver=1003000  ipmode=4

o,0x00000000,1486035625.768531,195.218.200.149,10047,195.218.200.149,80,1073725440,1073725440,0,0,0,0,0,2,536,0,1,672,3000,65535,0,8192,0,0,0,0,0
i,0x00000000,1486035625.768622,195.218.200.149,10047,195.218.200.149,80,1073725440,1073725440,0,0,0,0,0,2,536,0,1,672,3000,65535,0,8192,0,1,0,0,0
o,0x00000000,1486035625.768661,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,993,230,81660,0,8192,0,0,0,0,0
o,0x00000000,1486035625.768979,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,992,230,81660,241,8192,0,0,0,0,0
i,0x00000000,1486035625.768994,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,996,230,81660,0,8192,0,0,0,0,0
o,0x00000000,1486035625.769020,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,997,230,81660,0,8192,241,0,0,0,0
i,0x00000000,1486035625.769028,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,992,230,81660,241,8192,0,241,0,0,0
o,0x00000000,1486035625.769903,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,996,230,81660,33009,8192,0,0,0,0,0
i,0x00000000,1486035625.769926,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,7936,8192,6,6,4,16332,7,1,992,230,81660,0,8192,0,0,0,0,0
o,0x00000000,1486035625.769960,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,525281,230,81660,0,8192,8192,0,0,0,0
i,0x00000000,1486035625.769967,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,996,230,81660,33009,8192,0,8192,0,0,0
o,0x00000000,1486035630.769156,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,0,8192,6,6,4,16332,7,1,8389604,230,81660,81660,8192,0,0,0,0,0
i,0x00000000,1486035630.769184,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,525280,230,81660,0,8192,0,0,0,0,0
o,0x00000000,1486035630.769219,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,993,230,81660,0,8192,1,0,0,0,0
i,0x00000000,1486035630.769225,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,0,8192,6,6,4,16332,7,1,996,230,81660,81660,8192,0,1,0,0,0
o,0x00000000,1486035630.769251,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8128,8192,6,6,4,16332,7,1,996,230,81660,81659,8192,0,0,0,0,0
i,0x00000000,1486035630.769256,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,992,230,81660,0,8192,0,0,0,0,0
o,0x00000000,1486035630.769264,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,525281,230,81660,0,8192,8128,0,0,0,0
i,0x00000000,1486035630.769266,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8128,8192,6,6,4,16332,7,1,996,230,81660,81659,8192,0,8128,0,0,0
o,0x00000000,1486035635.769158,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,0,8192,6,6,4,16332,7,1,8389604,230,81660,81660,8192,0,0,0,0,0
i,0x00000000,1486035635.769193,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,525280,230,81660,0,8192,0,0,0,0,0
o,0x00000000,1486035635.769243,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,993,230,81660,0,8192,1,0,0,0,0
i,0x00000000,1486035635.769252,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,0,8192,6,6,4,16332,7,1,996,230,81660,81660,8192,0,1,0,0,0
o,0x00000000,1486035635.769280,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8128,8192,6,6,4,16332,7,1,996,230,81660,81659,8192,0,0,0,0,0
i,0x00000000,1486035635.769286,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,992,230,81660,0,8192,0,0,0,0,0
o,0x00000000,1486035635.769294,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,525281,230,81660,0,8192,8066,0,0,0,0
i,0x00000000,1486035635.769296,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8128,8192,6,6,4,16332,7,1,996,230,81660,81659,8192,0,8128,0,0,0
o,0x00000000,1486035640.769155,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,0,8192,6,6,4,16332,7,1,8389604,230,81660,81660,8192,0,0,0,0,0
i,0x00000000,1486035640.769185,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,525280,230,81660,0,8192,0,0,0,0,0
o,0x00000000,1486035640.769235,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,993,230,81660,0,8192,0,0,0,0,0
i,0x00000000,1486035640.769244,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,0,8192,6,6,4,16332,7,1,996,230,81660,81660,8192,0,1,0,0,0
o,0x00000000,1486035640.769271,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8128,8192,6,6,4,16332,7,1,996,230,81660,81659,8192,0,0,0,0,0
i,0x00000000,1486035640.769276,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,992,230,81660,0,8192,0,0,0,0,0
o,0x00000000,1486035640.769284,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,525281,230,81660,0,8192,8128,0,0,0,0
i,0x00000000,1486035640.769286,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8128,8192,6,6,4,16332,7,1,996,230,81660,81659,8192,0,8128,0,0,0
o,0x00000000,1486035645.769158,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,0,8192,6,6,4,16332,7,1,8389604,230,81660,81660,8192,0,0,0,0,0
i,0x00000000,1486035645.769191,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,525280,230,81660,0,8192,0,0,0,0,0
o,0x00000000,1486035645.769245,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,993,230,81660,0,8192,1,0,0,0,0
i,0x00000000,1486035645.769255,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,0,8192,6,6,4,16332,7,1,996,230,81660,81660,8192,0,1,0,0,0
o,0x00000000,1486035645.769284,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8128,8192,6,6,4,16332,7,1,996,230,81660,81659,8192,0,0,0,0,0
i,0x00000000,1486035645.769290,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,992,230,81660,0,8192,0,0,0,0,0
o,0x00000000,1486035645.769298,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,525281,230,81660,0,8192,7940,0,0,0,0
i,0x00000000,1486035645.769300,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8128,8192,6,6,4,16332,7,1,996,230,81660,81659,8192,0,8128,0,0,0
o,0x00000000,1486035650.769158,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,0,8192,6,6,4,16332,7,1,8389604,230,81660,81660,8192,0,0,0,0,0
i,0x00000000,1486035650.769194,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,525280,230,81660,0,8192,0,0,0,0,0
o,0x00000000,1486035650.769250,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,993,230,81660,0,8192,0,0,0,0,0
i,0x00000000,1486035650.769259,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,0,8192,6,6,4,16332,7,1,996,230,81660,81660,8192,0,1,0,0,0
o,0x00000000,1486035650.769288,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8128,8192,6,6,4,16332,7,1,996,230,81660,81659,8192,0,0,0,0,0
i,0x00000000,1486035650.769294,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,992,230,81660,0,8192,0,0,0,0,0
o,0x00000000,1486035650.769302,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,525281,230,81660,0,8192,8128,0,0,0,0
i,0x00000000,1486035650.769303,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8128,8192,6,6,4,16332,7,1,996,230,81660,81659,8192,0,8128,0,0,0
o,0x00000000,1486035655.769166,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,0,8192,6,6,4,16332,7,1,8389604,230,81660,81660,8192,0,0,0,0,0
...
Comment 4 Michael Tuexen freebsd_committer 2017-02-07 08:13:22 UTC
Created attachment 179696 [details]
packetdrill script reproducing the issue
Comment 5 Michael Tuexen freebsd_committer 2017-02-07 08:14:29 UTC
The problem is that FreeBSD only sends a window update if at least 2 MSS are freed. This is never true in this example. Will provide a patch.
Comment 6 Michael Tuexen freebsd_committer 2017-02-07 08:16:41 UTC
(In reply to Michael Tuexen from comment #5)
See https://svnweb.freebsd.org/base/head/sys/netinet/tcp_output.c?view=markup#l694
Comment 7 Michael Tuexen freebsd_committer 2017-02-07 13:34:15 UTC
Patch for review available at https://reviews.freebsd.org/D9475.
Comment 8 commit-hook freebsd_committer 2017-02-23 18:15:10 UTC
A commit references this bug:

Author: tuexen
Date: Thu Feb 23 18:14:36 UTC 2017
New revision: 314155
URL: https://svnweb.freebsd.org/changeset/base/314155

Log:
  TCP window updates are only sent if the window can be increased by at
  least 2 * MSS. However, if the receive buffer size is small, this might
  be impossible. Add back a criterion to send a TCP window update if
  the window can be increased by at least half of the receive buffer size.
  This condition was removed in r242252. This patch simply brings it back.
  PR:			211003
  Reviewed by:		gnn
  MFC after:		1 week
  Sponsored by:		Netflix, Inc.
  Differential Revision:	https://reviews.freebsd.org/D9475

Changes:
  head/sys/netinet/tcp_output.c
Comment 9 commit-hook freebsd_committer 2017-06-01 08:20:41 UTC
A commit references this bug:

Author: tuexen
Date: Thu Jun  1 08:19:45 UTC 2017
New revision: 319397
URL: https://svnweb.freebsd.org/changeset/base/319397

Log:
  MFC r314155:

  TCP window updates are only sent if the window can be increased by at
  least 2 * MSS. However, if the receive buffer size is small, this might
  be impossible. Add back a criterion to send a TCP window update if
  the window can be increased by at least half of the receive buffer size.
  This condition was removed in r242252. This patch simply brings it back.

  PR:			211003
  Reviewed by:		gnn
  Differential Revision:	https://reviews.freebsd.org/D9475

Changes:
_U  stable/11/
  stable/11/sys/netinet/tcp_output.c