Bug 206716

Summary: HPN-SSH: getsockopt(SO_SNDBUF) returns high watermark, not setsockopt(SO_SNDBUF)
Product: Base System Reporter: Chris Stankevitz <chris-freebsd-bugs>
Component: binAssignee: freebsd-bugs (Nobody) <bugs>
Status: Closed Works As Intended    
Severity: Affects Only Me CC: amd64, des, timon
Priority: ---    
Version: 10.1-RELEASE   
Hardware: amd64   
OS: Any   

Description Chris Stankevitz 2016-01-28 19:22:22 UTC 
getsockopt(SO_SNDBUF) does not return the value set by getsockopt(SO_SNDBUF).

HPN-SSH would like to determine the value set by setsockopt(SO_SNDBUF) so that it can fill it up.  This helps to use more bandwidth on a high BDP link.

HPN-SSH  tries to determine this value by periodically calling getsockopt(SO_SNDBUF) at [1]. However, on FreeBSD, getsockopt(SO_SNDBUF) does not return the value set by setsockopt(SO_SNDBUF).  Rather it returns the "high watermark".

The net result is that on high BDP links, HPN-SSH does nothing to increase bytes-in-flight.

[1] /usr/src/crypto/openssh/roaming_common.c line 59 in get_snd_buf_size: getsockopt(fd, SOL_SOCKET, SO_SNDBUF, &optval, &optvallen)
Comment 1 Chris Stankevitz 2016-01-29 00:42:59 UTC 
Typo on the first line of comment #0: get->set

getsockopt(SO_SNDBUF) does not return the value set by setsockopt(SO_SNDBUF).
Comment 2 Mark Linimon freebsd_committer freebsd_triage 2016-01-29 05:04:17 UTC 
des@, is this of interest?
Comment 3 Dag-Erling Smørgrav freebsd_committer freebsd_triage 2016-01-29 08:29:42 UTC 
No, this is a kernel issue.
Comment 4 timon 2016-03-02 18:43:09 UTC 
Maybe I miss something, but looks like there is no bug in FreeBSD.
Yes, on FreeBSD getsockopt(..., SO_SNDBUF, ...) returns "high watermark", but
setsockopt(..., SO_SNDBUF, ...) attempts to resize the buffer and, on success, sets "high watermark" to the new buffer size. So getsockopt() returns the value set by getsockopt().

I have written a simple test and run it on the 10.1-RELEASE and 10.2-RELEASE:
-----8<-----
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <err.h>
#include <stdio.h>

int main () {
        int s, size;
        socklen_t sizelen;

        s = socket(AF_INET6, SOCK_RAW, IPPROTO_IP);

        sizelen = sizeof size;
        getsockopt(s, SOL_SOCKET, SO_SNDBUF, &size, &sizelen);
        printf("Current socket send buffer size: %i\n", size);
        size /= 2;
        setsockopt(s, SOL_SOCKET, SO_SNDBUF, &size, sizelen);
        getsockopt(s, SOL_SOCKET, SO_SNDBUF, &size, &sizelen);
        printf("New socket send buffer size: %i\n", size);
}
-----8<-----
$ cc test.so_sndbuf.c
$ sudo ./a.out 
Current socket send buffer size: 9216
New socket send buffer size: 4608
Comment 5 Chris Stankevitz 2016-03-28 21:49:04 UTC 
(In reply to timon from comment #4)

Thank you.  I modified the example to increase the buffer size (vs. decrease) and I see the same thing.  It must be that the conclusions reached in this discussion are wrong:

== begin quote of [1] ==

So, I looked at what getsockopt SO_RCVBUF returns, and it returns: 
                case SO_RCVBUF: 
                        optval = so->so_rcv.sb_hiwat; 

Which is NOT S-BMAX, so it looks like OpenSSH only ever gets 66052 bytes 
for the buffer... 

If it's decided to base it's waiting for ack on SO_RCVBUF, then this 
is probably where the issue is... 

== end quote of [1] ===

So it sounds like this bug should be closed/worksforme.

Chris

[1] http://freebsd.1045724.n5.nabble.com/ssh-over-WAN-TCP-window-too-small-tp6035112p6035301.html