Bug 235333 - SSH data transmission speed unnecessarily slow due to IPQoS default setting
Summary: SSH data transmission speed unnecessarily slow due to IPQoS default setting
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: conf (show other bugs)
Version: 12.0-RELEASE
Hardware: amd64 Any
: --- Affects Many People
Assignee: Dag-Erling Smørgrav
Depends on:
Reported: 2019-01-30 21:35 UTC by tony@accesslab.com
Modified: 2024-02-10 05:41 UTC (History)
3 users (show)

See Also:


Note You need to log in before you can comment on or make changes to this bug.
Description tony@accesslab.com 2019-01-30 21:35:44 UTC
On certain connections, for example through some wireless devices, FreeBSD ssh/scp will copy data very slowly. Typically I see copy rates at 1/3 normal speed. This is happening because the IPQoS option for both ssh/scp (ssh_config) and sshd (sshd_config) default the IPQoS setting to 'af21'. Changing this setting to instead default to 'lowdelay' will produce normal data transmission speeds out-of-the-box for new FreeBSD installs thus making many users happier not having to trouble shoot and track down this weird random problem when it occurs. The low speed transmission issue only occurs on some connection paths, mostly I have seen it happen when copying data through a wireless bridge or access point. In hind sight, I see that on Linux, Openssh DOES default IPQoS = lowdelay, and perhaps that is not by accident?
Comment 1 tony@accesslab.com 2019-01-30 21:53:49 UTC
The IPQoS setting for the non-interactive sessions also needs to be changed from 'cs1' to 'throughput'.
Comment 2 Xin LI freebsd_committer freebsd_triage 2023-01-19 00:22:03 UTC
+des@ and emaste@.

Some of my notes:

1) the IPQoS setting of 'af21' was from upstream: https://github.com/openssh/openssh-portable/commit/5ee8448ad7c306f05a9f56769f95336a8269f379 .  This happened to FreeBSD in d46065df2d60b (OpenSSH 7.8p1 import).

2) Debian reverted it in https://salsa.debian.org/ssh-team/openssh/-/commit/03e56e6aedd9e0a12c8b4adb0ddffd67a479bac2 citing compatibility issues with VMWare, as discussed in https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/1822370 and in #20 it was fixed in a future version of VMWare (as of 2019-04-03), as well as iptables the revert is supposed to be "temporary", but it stayed as of today.

Related Debian bugs:

 - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=926229 [VMware]
 - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=923879 [iptables]
Comment 3 Dag-Erling Smørgrav freebsd_committer freebsd_triage 2023-09-19 11:02:10 UTC
I don't think this issue was ever resolved.  I wonder if we should change the default values for ip_qos_interactive and ip_qos_bulk to IPTOS_LOWDELAY and IPTOS_THROUGHPUT, respectively.  What do you thin, Ed?
Comment 4 Tom Gillespie 2024-02-10 05:21:20 UTC
For the record, this causes an extremely hard to debug issue when using rsync over ssh when transferring files from freenas or truenas. It took me nearly two years to hunt down this particular issue in the bug tracker. The symptoms were extremely slow transfers running at well under 800kB/s on connections that were otherwise pushing 15MB/s. It took digging through wireshark records to finally notice that SSH packets coming from the freebsd system were absurdly small. Hopefully this note can prove useful for anyone else who has the misfortune to encounter this behavior until the default is updated.
Comment 5 Tom Gillespie 2024-02-10 05:41:16 UTC
One additional note that might help with replicating this issue.

I am able to produce the behavior 100% of the time because I my rsync transfers over ssh are between hosts on opposite sides of the US. The ping time between hosts is about 70ms. I'm not sure that high latency itself is the root cause but maybe that information could help someone else track down why the current default sometimes falls off a cliff. The high latency also resulted in a misdiagnosis of the issue being related to tcp window size.