Bug 15095

The FreeBSD implementation does not scale the advertised window
immediately when it discovers that window scaling is being used. The result
is that irrespective of advertised window, in the first round-trip after
connection establishment, a FreeBSD TCP sender cannot send more data than
the unscaled value of advertised window.

One can argue that this is not important given that TCP does slow-start in
the first round-trip. Well, people are looking at rate-based pacing where
you don't have to do slow-start. Also the above is important in LANs where
FreeBSD doesn't use slow-start. In my case, I'm emulating a WAN to see
the benefits of rate-based pacing and so is extremely important.

Fix: The following patch applied to /sys/netinet/tcp_input.c fixes the problem:
How-To-Repeat: Try running a TCP transfer over an emulated (or an actual) WAN. Make
sure that slow-start is turned off (it is if FreeBSD thinks its operating 
on a LAN - i.e the other host has the same subnet address). Also ensure
that RFC1323 is being used (i.e. Window scaling is in effect) and the
flow control window is big enough on the TCP receiver. Use tcpdump to
check the number of packets sent in the first round-trip - these will
correspond to the unscaled value of the advertised window.

A simpler method to repeat the problem is to instrument the kernel
to print out the value of the variable tp->snd_wnd when TCP starts
sending data. You'll see that it corresponds to the unscaled value of
the flow control (or advertised) window.