On certain connections, for example through some wireless devices, FreeBSD ssh/scp will copy data very slowly. Typically I see copy rates at 1/3 normal speed. This is happening because the IPQoS option for both ssh/scp (ssh_config) and sshd (sshd_config) default the IPQoS setting to 'af21'. Changing this setting to instead default to 'lowdelay' will produce normal data transmission speeds out-of-the-box for new FreeBSD installs thus making many users happier not having to trouble shoot and track down this weird random problem when it occurs. The low speed transmission issue only occurs on some connection paths, mostly I have seen it happen when copying data through a wireless bridge or access point. In hind sight, I see that on Linux, Openssh DOES default IPQoS = lowdelay, and perhaps that is not by accident?
The IPQoS setting for the non-interactive sessions also needs to be changed from 'cs1' to 'throughput'.
+des@ and emaste@. Some of my notes: 1) the IPQoS setting of 'af21' was from upstream: https://github.com/openssh/openssh-portable/commit/5ee8448ad7c306f05a9f56769f95336a8269f379 . This happened to FreeBSD in d46065df2d60b (OpenSSH 7.8p1 import). 2) Debian reverted it in https://salsa.debian.org/ssh-team/openssh/-/commit/03e56e6aedd9e0a12c8b4adb0ddffd67a479bac2 citing compatibility issues with VMWare, as discussed in https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/1822370 and in #20 it was fixed in a future version of VMWare (as of 2019-04-03), as well as iptables the revert is supposed to be "temporary", but it stayed as of today. Related Debian bugs: - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=926229 [VMware] - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=923879 [iptables]
I don't think this issue was ever resolved. I wonder if we should change the default values for ip_qos_interactive and ip_qos_bulk to IPTOS_LOWDELAY and IPTOS_THROUGHPUT, respectively. What do you thin, Ed?
For the record, this causes an extremely hard to debug issue when using rsync over ssh when transferring files from freenas or truenas. It took me nearly two years to hunt down this particular issue in the bug tracker. The symptoms were extremely slow transfers running at well under 800kB/s on connections that were otherwise pushing 15MB/s. It took digging through wireshark records to finally notice that SSH packets coming from the freebsd system were absurdly small. Hopefully this note can prove useful for anyone else who has the misfortune to encounter this behavior until the default is updated.
One additional note that might help with replicating this issue. I am able to produce the behavior 100% of the time because I my rsync transfers over ssh are between hosts on opposite sides of the US. The ping time between hosts is about 70ms. I'm not sure that high latency itself is the root cause but maybe that information could help someone else track down why the current default sometimes falls off a cliff. The high latency also resulted in a misdiagnosis of the issue being related to tcp window size.