It appears that UPLOADING to sshd (eg. scp localfile remote.tld:remotefile) will happen at very limited bandwidth.
I haven't been able to isolate which circumstances those are precisely, but, between two servers when the command is reversed (eg. scp remote.tld:remotefile ./localfile) the transfer happens at full available speed.
To make this clear, for a particular set of servers, for example:
on server A:
scp localfile B:remotefile -> 40Mbps (upload to sshd)
scp B:remotefile localfile -> 120Mbps (download from sshd)
scp localfile B:remotefile -> 200Mbps (upload to sshd, Debian on B)
scp B:remotefile localfile -> 120Mbps (download from sshd, Debian on B)
on server B:
scp localfile A:remotefile -> 30Mbps (upload to sshd)
scp A:remotefile localfile -> 120Mbps (download from sshd)
scp localfile A:remotefile -> 40Mbps (upload to sshd, Debian on B)
scp B:remotefile localfile -> 200Mbps (download from sshd, Debian on B)
NOTE: When upload is _TO_ FreeBSD sshd, the bandwidth is 30-40Mbps.
Which means the differences between upload and download are not due to network congestion or particular datacenter setup, whether a server is dedicated or VPS, UFS2 filesystem or ZFS, 1GB of RAM or 16GB. Though all were 64bit. Tested between Leaseweb Amsterdam, DigitalOcean Amsterdam, DigitalOcean Frankfurt, Hetzner Germany (Falkenstein) and Rackspace in London.
This also affects rsync over ssh, so it's not just scp. The only thing that changes are absolute bandwidth numbers for a given set of servers.
Testing other protocols (http, https, netcat, ...) shows full network speed available between testing servers.
Different IPQoS or Cipher settings do not improve anything.
(In reply to Vlad K. from comment #0)
Errata, that last scp should be:
scp A:remotefile localfile ...
It appears that using OpenSSH-portable on the same servers mostly fixes the issue.
Why is it "mostly" and not for all servers, I'm still investigating, could be that one particular pair of servers I'm still seeing 3Mpbs (on a 100MBps link, 8 hops and 11msec away from each other) are affected by something in the provider's network configuration.
I'm closing my own report, I can't replicate the issue, in some networks it works as expected in some it doesn't and I can't find the common denominator. I can easily file another report when I'm certain of the cause.