I'm currently running FreeNas 11 RC1. I've edited the tunables for this item kern.ipc.soacceptqueue to allow for 2048 connections, but it doesn't propagate to the jails on my system. Below is the error I received saying the listen queue is full. May 19 18:43:24 maverick kernel: sonewconn: pcb 0xfffff8035bef6740: Listen queue overflow: 193 already in queue awaiting acceptance (138 occurrences) May 19 19:02:16 maverick kernel: sonewconn: pcb 0xfffff8035bef6740: Listen queue overflow: 193 already in queue awaiting acceptance (200 occurrences) May 19 19:03:17 maverick kernel: sonewconn: pcb 0xfffff8035bef6740: Listen queue overflow: 193 already in queue awaiting acceptance (209 occurrences) May 19 19:04:17 maverick kernel: sonewconn: pcb 0xfffff8035bef6740: Listen queue overflow: 193 already in queue awaiting acceptance (199 occurrences) May 19 19:05:17 maverick kernel: sonewconn: pcb 0xfffff8035bef6740: Listen queue overflow: 193 already in queue awaiting acceptance (202 occurrences) Here are the Netstat outputs from my main FreeNas system: tcp4 0/0/2048 127.0.0.1.8542 tcp4 0/0/2048 127.0.0.1.8600 tcp4 0/0/2048 127.0.0.1.8500 tcp4 0/0/2048 127.0.0.1.8400 And the output from Netstat for my jail: tcp4 0/0/128 192.168.0.20.12348 tcp6 0/0/128 *.51413 tcp4 0/0/128 *.51413 tcp4 0/0/128 *.9091 If I run sysctl kern.ipc.soacceptqueue in the jail it shows the following: # sysctl kern.ipc.soacceptqueue kern.ipc.soacceptqueue: 2048
At least on vanilla FreeBSD kern.ipc.soacceptqueue merely specifies an upper limit, it does not prevent the application from requesting a smaller one. Many applications use a hardcoded value like 128 without checking if a higher value would work. For details see the "listen" and "getsockopt" man pages.
I've checked multiple jails on my system, which include CouchPotato, Plex and Transmission and all have the same max of 128, yet when running sysctl kern.ipc.soacceptqueue it says that is 2048. I find it highly unlikely that 3 separate applications are applying limits to the jails.
kern.ipc.soacceptqueue is SYSCTL_PROC defined in sys/kern/uipc_socket.c without CTLFLAG_VNET, so it is not VIMAGE/VNET-aware currently.
That makes more sense. Is there a way around this or a way to increase the connection queue in a jail that has VIMAGE enabled?
(In reply to john.leo from comment #4) You could just patch sys/sys/socket.h and increase value on a line "#define SOMAXCONN 128" (this is used to specify initial value for a sysctl only). Or, if you are curious enough, you can add CTLFLAG_VNET flag to declaration of soacceptqueue in sys/kern/uipc_socket.c and see if it will work or crash :-) That is, until SomeOne (TM) prepares complete solution.
Created attachment 183099 [details] make per-VNET soacceptqueue/somaxconn and numopensockets At attempt to make sysctl soacceptqueue (somaxconn) and numopensockets per-VNET/VIMAGE instead of global.
(In reply to john.leo from comment #4) You may also try attached patch. Beware, as it is compile-only tested.
Thanks, would this be for the main OS itself or for use in the jail? Also I'm having trouble finding that file within the OS, I couldn't find it at the path you provided.
(In reply to john.leo from comment #8) The patch is for sys/kern/uipc_socket.c - that is, for kernel. Jails do not have own kernels, there is only one kernel in running system. You need to rebuild and reinstall kernel after patch and reboot the system to apply changes.
That is, for main OS and /usr/src/sys/kern/uipc_socket.c
Thanks, for some reason it is showing that /usr/src is an empty directory on my system. I think I'd also like to set up a dev system to test this out rather than run it in production.
Making these variables per-VNET is not necessarily a good idea; it means a VNET-jail consumer could possibly DoS the system without the administrator having a chance to prevent this easily by exceeding resources. Need to be very careful. I'd hope if this should go into HEAD that there'll be a way to "cap" the values or reject excessive requests by some metric at least.
(In reply to Bjoern A. Zeeb from comment #12) These variables are global currently but this does not mean the limits they impose are "global" in any way: static u_int somaxconn is just default for per-socket backlog limit so->so_qlimit (struct socket *so) and this change makes it possible to assign different defaults per-jail. Yes, increase of such limit allows jailed root to get more space in the queue of not accepted yet sockets but theres is already plenty ways to consume such resources (f.e. by creating listening socket and making tons of local connections). Perhaps, this sysctl should be made read-only for jailed root, if possible. V_numopensockets is purely informational.
Any news about this? I need to increase kern.ipc.soacceptqueue in several VNET jails on several working systems. Loading home-made patched kernel seems to be heavy task for future updates. What about propagation of the parameter set on a host system to all VNET jails?
(In reply to Peter TKATCHENKO from comment #14) Nobody ever reported testing results. Can you test the patch?
(In reply to Eugene Grosbein from comment #15) Do you think it could be applied to 13.x or 14.x or I need to deploy 11.x?
Try with any supported version.
The patch cannot be applied 'as is' on 14.2 kernel. I modified it a bit, applied and rebuilt the GENERIC kernel. Unfortunately, the kernel panics on boot: Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 02 fault virtual address = 0x28 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80be2e44 stack pointer = 0x28:0xfffffe00cdbb7d00 frame pointer = 0x28:0xfffffe00cdbb7d50 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 155 (ip6addrctl) rdi: fffff80006d57c00 rsi: ffffffff818d56d8 rdx: ffffffff811da680 rcx: 0000000000000000 r8: 0000000000320000 r9: 0000000000000000 rax: 0000000000000000 rbx: fffff800134df400 rbp: fffffe00cdbb7d50 r10: 0000000000000000 r11: 0000000000000400 r12: fffff80006f2b000 r13: 0000000000000000 r14: fffff800030cd4c0 r15: fffff800134df510 trap number = 12 panic: page fault cpuid = 2 time = 1735911923 KDB: stack backtrace: #0 0xffffffff80b8c89d at kdb_backtrace+0x5d #1 0xffffffff80b3ec01 at vpanic+0x131 #2 0xffffffff80b3eac3 at panic+0x43 #3 0xffffffff81026a0b at trap_fatal+0x40b #4 0xffffffff81026a56 at trap_pfault+0x46 #5 0xffffffff80ffd408 at calltrap+0x8 #6 0xffffffff80be2a5a at socreate+0xba #7 0xffffffff80bed1b0 at kern_socket+0xb0 #8 0xffffffff810272c5 at amd64_syscall+0x115 #9 0xffffffff80ffdd1b at fast_syscall_common+0xf8
https://reviews.freebsd.org/D48314 This makes kern.ipc.soacceptqueue per-vnet.
https://reviews.freebsd.org/D48315
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=59498e099cc055da7afca8266087b7668be6d7cb commit 59498e099cc055da7afca8266087b7668be6d7cb Author: Gleb Smirnoff <glebius@FreeBSD.org> AuthorDate: 2025-01-13 18:08:58 +0000 Commit: Gleb Smirnoff <glebius@FreeBSD.org> CommitDate: 2025-01-13 18:11:46 +0000 sockets: virtualize kern.ipc.numopensockets To avoid breaking POLA on the host machine it reports the same value as before. In a VNET jail it now reports number of sockets in this jail. PR: 219655 Differential Revision: https://reviews.freebsd.org/D48315 sys/kern/uipc_socket.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=4155be454c46bc1ab725aca5c6969b064b74be38 commit 4155be454c46bc1ab725aca5c6969b064b74be38 Author: Gleb Smirnoff <glebius@FreeBSD.org> AuthorDate: 2025-01-13 18:08:51 +0000 Commit: Gleb Smirnoff <glebius@FreeBSD.org> CommitDate: 2025-01-13 18:11:39 +0000 sockets: virtualize kern.ipc.soacceptqueue PR: 219655 Differential Revision: https://reviews.freebsd.org/D48314 sys/kern/uipc_socket.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-)
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=797bb39ccd5f57c3206665949d3cba35328cfdeb commit 797bb39ccd5f57c3206665949d3cba35328cfdeb Author: Gleb Smirnoff <glebius@FreeBSD.org> AuthorDate: 2025-01-13 18:08:58 +0000 Commit: Eugene Grosbein <eugen@FreeBSD.org> CommitDate: 2025-01-17 19:34:16 +0000 sockets: virtualize kern.ipc.numopensockets To avoid breaking POLA on the host machine it reports the same value as before. In a VNET jail it now reports number of sockets in this jail. PR: 219655 Differential Revision: https://reviews.freebsd.org/D48315 (cherry picked from commit 59498e099cc055da7afca8266087b7668be6d7cb) sys/kern/uipc_socket.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=08384f613cd808e31fa3ae3dd86fa078b49d3f4b commit 08384f613cd808e31fa3ae3dd86fa078b49d3f4b Author: Gleb Smirnoff <glebius@FreeBSD.org> AuthorDate: 2025-01-13 18:08:58 +0000 Commit: Eugene Grosbein <eugen@FreeBSD.org> CommitDate: 2025-01-17 19:41:51 +0000 sockets: virtualize kern.ipc.numopensockets To avoid breaking POLA on the host machine it reports the same value as before. In a VNET jail it now reports number of sockets in this jail. PR: 219655 Differential Revision: https://reviews.freebsd.org/D48315 (cherry picked from commit 59498e099cc055da7afca8266087b7668be6d7cb) sys/kern/uipc_socket.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-)
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=c348cdf132572ab821f7c75a97453c1d238239b9 commit c348cdf132572ab821f7c75a97453c1d238239b9 Author: Gleb Smirnoff <glebius@FreeBSD.org> AuthorDate: 2025-01-13 18:08:51 +0000 Commit: Eugene Grosbein <eugen@FreeBSD.org> CommitDate: 2025-01-17 19:44:15 +0000 sockets: virtualize kern.ipc.soacceptqueue PR: 219655 Differential Revision: https://reviews.freebsd.org/D48314 (cherry picked from commit 4155be454c46bc1ab725aca5c6969b064b74be38) sys/kern/uipc_socket.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=45d1cff2bfc679aee4d7f8ae8cc827bc163fbd08 commit 45d1cff2bfc679aee4d7f8ae8cc827bc163fbd08 Author: Gleb Smirnoff <glebius@FreeBSD.org> AuthorDate: 2025-01-13 18:08:51 +0000 Commit: Eugene Grosbein <eugen@FreeBSD.org> CommitDate: 2025-01-17 19:44:36 +0000 sockets: virtualize kern.ipc.soacceptqueue PR: 219655 Differential Revision: https://reviews.freebsd.org/D48314 (cherry picked from commit 4155be454c46bc1ab725aca5c6969b064b74be38) sys/kern/uipc_socket.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-)