Summary: | SSH data transmission speed unnecessarily slow due to IPQoS default setting | ||
---|---|---|---|
Product: | Base System | Reporter: | tony <tony> |
Component: | conf | Assignee: | Dag-Erling Smørgrav <des> |
Status: | Open --- | ||
Severity: | Affects Many People | CC: | delphij, emaste, tgbugs |
Priority: | --- | ||
Version: | 12.0-RELEASE | ||
Hardware: | amd64 | ||
OS: | Any |
Description
tony@accesslab.com
2019-01-30 21:35:44 UTC
The IPQoS setting for the non-interactive sessions also needs to be changed from 'cs1' to 'throughput'. +des@ and emaste@. Some of my notes: 1) the IPQoS setting of 'af21' was from upstream: https://github.com/openssh/openssh-portable/commit/5ee8448ad7c306f05a9f56769f95336a8269f379 . This happened to FreeBSD in d46065df2d60b (OpenSSH 7.8p1 import). 2) Debian reverted it in https://salsa.debian.org/ssh-team/openssh/-/commit/03e56e6aedd9e0a12c8b4adb0ddffd67a479bac2 citing compatibility issues with VMWare, as discussed in https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/1822370 and in #20 it was fixed in a future version of VMWare (as of 2019-04-03), as well as iptables the revert is supposed to be "temporary", but it stayed as of today. Related Debian bugs: - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=926229 [VMware] - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=923879 [iptables] I don't think this issue was ever resolved. I wonder if we should change the default values for ip_qos_interactive and ip_qos_bulk to IPTOS_LOWDELAY and IPTOS_THROUGHPUT, respectively. What do you thin, Ed? For the record, this causes an extremely hard to debug issue when using rsync over ssh when transferring files from freenas or truenas. It took me nearly two years to hunt down this particular issue in the bug tracker. The symptoms were extremely slow transfers running at well under 800kB/s on connections that were otherwise pushing 15MB/s. It took digging through wireshark records to finally notice that SSH packets coming from the freebsd system were absurdly small. Hopefully this note can prove useful for anyone else who has the misfortune to encounter this behavior until the default is updated. One additional note that might help with replicating this issue. I am able to produce the behavior 100% of the time because I my rsync transfers over ssh are between hosts on opposite sides of the US. The ping time between hosts is about 70ms. I'm not sure that high latency itself is the root cause but maybe that information could help someone else track down why the current default sometimes falls off a cliff. The high latency also resulted in a misdiagnosis of the issue being related to tcp window size. |