We have many FreeBSD installations and some of them historically have the following settings in /etc/sysctl.conf: net.inet.tcp.delayed_ack=0 net.inet.tcp.recvspace=8192 net.inet.tcp.sendspace=150000 After upgrading to 10.2 we've run into an issue - local http requests were delaying after every 8kb of data for 5 seconds. lynx --mime_header "http://<server ip>/any file around 200kb" first we get 4Kb (incuding headers) 5 seconds delay (exact value, not an approximate) next 8Kb 5 seconds delay next 8Kb ... and so on. Everything is fine if the file is requested remotely, only local requests have such delays. Nginx or apache - both have the delays. We also where able to reproduce the delays in virtualized environment. There are no issues with this sysctl settings in FreeBSD 8.X and FreeBSD 9.X. Setting net.inet.tcp.sendspace=4096 "hides" the issue. Increasing net.inet.tcp.recvspace to more then 32664 also "hides" the issue. Restoring net.inet.tcp.delayed_ack to default 1 also "hides" the issue but there are still some performance issues if net.inet.tcp.recvspace is low: >sysctl net.inet.tcp.recvspace=8192 >time lynx --mime_header "local file around 200 kb" > /dev/null 0.009u 0.000s 0:01.41 0.0% 0+0k 0+0io 0pf+0w >sysctl net.inet.tcp.recvspace=32768 >time lynx --mime_header "local file around 200 kb" > /dev/null 0.009u 0.000s 0:00.01 0.0% 0+0k 0+0io 0pf+0w
Exact 5s delay is very interesting. It usually points to persist timer but not sure how to relate that here. Looks like lynx is not draining the recv space fast enough? I am not quite sure but if possible, can you try 1) capturing the transfer 2) something other than lynx on recv side 3) -HEAD or 11?
You might look at using siftr(4) to collect statistics on the transfer. It is often overlooked, but is a part of the base system and captures detailed information on TCP activity. This can be very useful for analyzing TCP performance issues. The man page is quite detailed.
Note: we've also tested FreeBSD 11, and can confirm the same behavior. By default all is fine, but just after you change net.inet.tcp.delayed_ack=0 net.inet.tcp.recvspace=8192 net.inet.tcp.sendspace=150000 local http requests are delaying after every 8kb of data for 5 seconds. > 2) something other than lynx on recv side Initially we have found this issue between apache and nginx, and we've also used telnet, so it's for sure not related to lynx. > You might look at using siftr(4) to collect statistics on the transfer. Done. enable_time_secs=1486035625 enable_time_usecs=746854 siftrver=1.2.4 hz=1000 tcp_rtt_scale=32 sysname=FreeBSD sysver=1003000 ipmode=4 o,0x00000000,1486035625.768531,195.218.200.149,10047,195.218.200.149,80,1073725440,1073725440,0,0,0,0,0,2,536,0,1,672,3000,65535,0,8192,0,0,0,0,0 i,0x00000000,1486035625.768622,195.218.200.149,10047,195.218.200.149,80,1073725440,1073725440,0,0,0,0,0,2,536,0,1,672,3000,65535,0,8192,0,1,0,0,0 o,0x00000000,1486035625.768661,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,993,230,81660,0,8192,0,0,0,0,0 o,0x00000000,1486035625.768979,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,992,230,81660,241,8192,0,0,0,0,0 i,0x00000000,1486035625.768994,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,996,230,81660,0,8192,0,0,0,0,0 o,0x00000000,1486035625.769020,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,997,230,81660,0,8192,241,0,0,0,0 i,0x00000000,1486035625.769028,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,992,230,81660,241,8192,0,241,0,0,0 o,0x00000000,1486035625.769903,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,996,230,81660,33009,8192,0,0,0,0,0 i,0x00000000,1486035625.769926,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,7936,8192,6,6,4,16332,7,1,992,230,81660,0,8192,0,0,0,0,0 o,0x00000000,1486035625.769960,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,525281,230,81660,0,8192,8192,0,0,0,0 i,0x00000000,1486035625.769967,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,996,230,81660,33009,8192,0,8192,0,0,0 o,0x00000000,1486035630.769156,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,0,8192,6,6,4,16332,7,1,8389604,230,81660,81660,8192,0,0,0,0,0 i,0x00000000,1486035630.769184,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,525280,230,81660,0,8192,0,0,0,0,0 o,0x00000000,1486035630.769219,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,993,230,81660,0,8192,1,0,0,0,0 i,0x00000000,1486035630.769225,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,0,8192,6,6,4,16332,7,1,996,230,81660,81660,8192,0,1,0,0,0 o,0x00000000,1486035630.769251,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8128,8192,6,6,4,16332,7,1,996,230,81660,81659,8192,0,0,0,0,0 i,0x00000000,1486035630.769256,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,992,230,81660,0,8192,0,0,0,0,0 o,0x00000000,1486035630.769264,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,525281,230,81660,0,8192,8128,0,0,0,0 i,0x00000000,1486035630.769266,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8128,8192,6,6,4,16332,7,1,996,230,81660,81659,8192,0,8128,0,0,0 o,0x00000000,1486035635.769158,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,0,8192,6,6,4,16332,7,1,8389604,230,81660,81660,8192,0,0,0,0,0 i,0x00000000,1486035635.769193,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,525280,230,81660,0,8192,0,0,0,0,0 o,0x00000000,1486035635.769243,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,993,230,81660,0,8192,1,0,0,0,0 i,0x00000000,1486035635.769252,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,0,8192,6,6,4,16332,7,1,996,230,81660,81660,8192,0,1,0,0,0 o,0x00000000,1486035635.769280,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8128,8192,6,6,4,16332,7,1,996,230,81660,81659,8192,0,0,0,0,0 i,0x00000000,1486035635.769286,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,992,230,81660,0,8192,0,0,0,0,0 o,0x00000000,1486035635.769294,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,525281,230,81660,0,8192,8066,0,0,0,0 i,0x00000000,1486035635.769296,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8128,8192,6,6,4,16332,7,1,996,230,81660,81659,8192,0,8128,0,0,0 o,0x00000000,1486035640.769155,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,0,8192,6,6,4,16332,7,1,8389604,230,81660,81660,8192,0,0,0,0,0 i,0x00000000,1486035640.769185,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,525280,230,81660,0,8192,0,0,0,0,0 o,0x00000000,1486035640.769235,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,993,230,81660,0,8192,0,0,0,0,0 i,0x00000000,1486035640.769244,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,0,8192,6,6,4,16332,7,1,996,230,81660,81660,8192,0,1,0,0,0 o,0x00000000,1486035640.769271,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8128,8192,6,6,4,16332,7,1,996,230,81660,81659,8192,0,0,0,0,0 i,0x00000000,1486035640.769276,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,992,230,81660,0,8192,0,0,0,0,0 o,0x00000000,1486035640.769284,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,525281,230,81660,0,8192,8128,0,0,0,0 i,0x00000000,1486035640.769286,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8128,8192,6,6,4,16332,7,1,996,230,81660,81659,8192,0,8128,0,0,0 o,0x00000000,1486035645.769158,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,0,8192,6,6,4,16332,7,1,8389604,230,81660,81660,8192,0,0,0,0,0 i,0x00000000,1486035645.769191,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,525280,230,81660,0,8192,0,0,0,0,0 o,0x00000000,1486035645.769245,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,993,230,81660,0,8192,1,0,0,0,0 i,0x00000000,1486035645.769255,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,0,8192,6,6,4,16332,7,1,996,230,81660,81660,8192,0,1,0,0,0 o,0x00000000,1486035645.769284,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8128,8192,6,6,4,16332,7,1,996,230,81660,81659,8192,0,0,0,0,0 i,0x00000000,1486035645.769290,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,992,230,81660,0,8192,0,0,0,0,0 o,0x00000000,1486035645.769298,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,525281,230,81660,0,8192,7940,0,0,0,0 i,0x00000000,1486035645.769300,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8128,8192,6,6,4,16332,7,1,996,230,81660,81659,8192,0,8128,0,0,0 o,0x00000000,1486035650.769158,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,0,8192,6,6,4,16332,7,1,8389604,230,81660,81660,8192,0,0,0,0,0 i,0x00000000,1486035650.769194,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,525280,230,81660,0,8192,0,0,0,0,0 o,0x00000000,1486035650.769250,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,993,230,81660,0,8192,0,0,0,0,0 i,0x00000000,1486035650.769259,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,0,8192,6,6,4,16332,7,1,996,230,81660,81660,8192,0,1,0,0,0 o,0x00000000,1486035650.769288,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8128,8192,6,6,4,16332,7,1,996,230,81660,81659,8192,0,0,0,0,0 i,0x00000000,1486035650.769294,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,992,230,81660,0,8192,0,0,0,0,0 o,0x00000000,1486035650.769302,195.218.200.149,10047,195.218.200.149,80,1073725440,32664,0,8192,8192,6,6,4,16332,7,1,525281,230,81660,0,8192,8128,0,0,0,0 i,0x00000000,1486035650.769303,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,8128,8192,6,6,4,16332,7,1,996,230,81660,81659,8192,0,8128,0,0,0 o,0x00000000,1486035655.769166,195.218.200.149,80,195.218.200.149,10047,1073725440,32664,0,0,8192,6,6,4,16332,7,1,8389604,230,81660,81660,8192,0,0,0,0,0 ...
Created attachment 179696 [details] packetdrill script reproducing the issue
The problem is that FreeBSD only sends a window update if at least 2 MSS are freed. This is never true in this example. Will provide a patch.
(In reply to Michael Tuexen from comment #5) See https://svnweb.freebsd.org/base/head/sys/netinet/tcp_output.c?view=markup#l694
Patch for review available at https://reviews.freebsd.org/D9475.
A commit references this bug: Author: tuexen Date: Thu Feb 23 18:14:36 UTC 2017 New revision: 314155 URL: https://svnweb.freebsd.org/changeset/base/314155 Log: TCP window updates are only sent if the window can be increased by at least 2 * MSS. However, if the receive buffer size is small, this might be impossible. Add back a criterion to send a TCP window update if the window can be increased by at least half of the receive buffer size. This condition was removed in r242252. This patch simply brings it back. PR: 211003 Reviewed by: gnn MFC after: 1 week Sponsored by: Netflix, Inc. Differential Revision: https://reviews.freebsd.org/D9475 Changes: head/sys/netinet/tcp_output.c
A commit references this bug: Author: tuexen Date: Thu Jun 1 08:19:45 UTC 2017 New revision: 319397 URL: https://svnweb.freebsd.org/changeset/base/319397 Log: MFC r314155: TCP window updates are only sent if the window can be increased by at least 2 * MSS. However, if the receive buffer size is small, this might be impossible. Add back a criterion to send a TCP window update if the window can be increased by at least half of the receive buffer size. This condition was removed in r242252. This patch simply brings it back. PR: 211003 Reviewed by: gnn Differential Revision: https://reviews.freebsd.org/D9475 Changes: _U stable/11/ stable/11/sys/netinet/tcp_output.c