When sending data over SCTP and using e.g., sctp_sendv(), SO_SNDLOWAT is not accounted for. For example, let's say that I have a socket send buffer of 32768 bytes and I have set SO_SNDLOWAT to 10000 bytes. If the send buffer is filled up to say 25000 bytes, then I shouldn't be able to send anything until there are SO_SNDLOWAT (1000) amount of bytes (+ 1?) bytes free in the socket send buffer. From (https://www.freebsd.org/cgi/man.cgi?query=setsockopt&sektion=2): SO_SNDLOWAT is an option to set the minimum count for output operations. Most output operations process all of the data supplied by the call, delivering data to the protocol for transmission and blocking as necessary for flow control. Nonblocking output operations will process as much data as permitted subject to flow control without blocking, but will process no data if flow control does not allow the smaller of the low water mark value or the entire request to be processed. This has been tested on FreeBSD 12.2-RELEASE and FreeBSD 13.0-RELEASE as well as on Linux (lksctp). The FreeBSD releases have the same behavior while it works in Linux. I have only tested this for non-blocking sockets so I don't know the behavior of blocking sockets. I guess it should block until the same criterion has happened. At least for non-blocking sockets, I believe that this can be fixed by adding a line in the sctp_lower_sosend() function in (sys/netinet/sctp_output.c). More specifically, these lines can be changed: if ((SCTP_SB_LIMIT_SND(so) < (amount + inqueue_bytes + stcb->asoc.sb_send_resv)) || (stcb->asoc.chunks_on_out_queue >= SCTP_BASE_SYSCTL(sctp_max_chunks_on_queue))) { To include this (or something similar): || (so->so_snd.sb_hiwat - so->so_snd.sb_acc) <= (u_int)so->so_snd.sb_lowat) Sorry for not uploading a patch, but I am unsure if my fix is of use or not.
Created attachment 229789 [details] Test files for reproducing the bug I added some tests if it helps. They can be compiled using: g++ -o client client.cpp utils.cpp g++ -o server server.cpp utils.cpp I have tested it in VMs with the SCTP module enabled. I have also added alias addresses to the loopback interface so that I could run both the server and client over the loopback interface. These alias addresess were added: 127.0.0.2 127.0.0.3 127.0.0.4 127.0.0.5 127.0.0.6 127.0.0.7 ::2 ::3 ::4 ::5 ::6 ::7 The test can be started using: ./server --poll --ipv4 --sctp --test-lower-water-mark ./client --poll --ipv4 --sctp --test-lower-water-mark IPv6 as well as TCP can be tested as well to compare the behavior.
I think that I have misinterpreted the SO_SNDLOWAT functionality and provided a faulty/misleading example. To quote SO_SNDLOWAT again from the man: "Nonblocking output operations will process as much data as permitted subject to flow control without blocking, but will process no data if flow control does not allow the smaller of the low water mark value or the entire request to be processed." Also, I don't know why I made the comment about blocking, since blocking is blocking, it will just block until all data has been buffered. I probably had too much in my head since I submitted many bugs at once :) Anyway, I get it as one of the uses for SO_SNDLOWAT is to be able to guarantee buffering of entire "messages". For example, let's say that I have a SCTP socket which is message based. I know that the largest message I want to be able to send is 8192 bytes so I set SO_SNDLOWAT to this value. Now, let's say that I have some congestion and my send buffer only has 2048 bytes free. Then I shouldn't be able to send any 8192 bytes messages, not even parts of them. Not until the send buffer has at least 8192 bytes free. But I guess that I should be able to send a message of 512 bytes still? Since then the full message can be buffered.
I have come to my senses that this isn't a bug. I cannot find any man for sctp_sendv but for the other sctp_send* functions it says: "If the message is too long to pass atomically through the underlying protocol, errno is set to EMSGSIZE, -1 is returned, and the message is not transmitted." So because of that, it will never buffer any partial messages. I am closing this, sorry for any inconvenience.