|Summary:||[request] Wakeup only one thread while kqueue events are available|
|Product:||Base System||Reporter:||Andy Chen <andy.yx.chen>|
|Component:||kern||Assignee:||freebsd-threads mailing list <threads>|
|Severity:||Affects Many People||CC:||andy.yx.chen, dab, jilles, kib|
Description Andy Chen 2016-05-16 19:01:20 UTC
There is no reason to wakeup all threads while there are events available, as that would make same event to be consumed by multiple threads at the same time and result in errors. a fix: https://github.com/freebsd/freebsd/pull/70
Comment 1 Konstantin Belousov 2016-05-22 09:14:36 UTC
(In reply to Andy Chen from comment #0) Which errors are you talking about ? In which situation 'same event may be consumed by multiple threads' ? Can you provide a test program illustrating the issue ?
Comment 2 Andy Chen 2016-05-22 15:03:42 UTC
I have an repro program at http://andytech.net/a_292.html, in case of a system with four cpu cores, when a connection comes in, more than one thread will be waken up, and only one thread could get the connection while the rest will stuck on accept call for ever.
Comment 3 Jilles Tjoelker 2016-05-22 15:30:49 UTC
With the test program's current design, it should enable non-blocking mode on the socket and ignore [EAGAIN] and [ECONNABORTED] errors from accept(). Apart from that, I appreciate the thundering herd issue with the program. However, I'm a bit afraid of applications getting stuck for a while when the awakened thread exits without waking another thread. Also, randomly distributing socket events across threads seems inefficient; it should be more efficient to tie each socket to one thread and have one kqueue per thread.
Comment 4 Konstantin Belousov 2016-05-22 18:45:25 UTC
(In reply to Jilles Tjoelker from comment #3) Such program must not use single kqueue for several threads, if the desire is to avoid the thundering herd issue. More important is that socket events are level triggered, i.e. readiness of the listen socket must be reported until the incoming connection is consumed. So the program behaviour is as intended. Change of wakeup(9) to wakeup_one(9) hides userspace bug, you just do not get all _required_ notifications since less threads are runnable.
Comment 5 Andy Chen 2016-05-22 23:18:00 UTC
(In reply to Konstantin Belousov from comment #4) Though this has been closed, I still have two cents about this (so to make it as active again), 1. the good stuff for kqueue is it could be used to served almost all sorts of events, like file access, timer, socket etc. that leads to a very common design that having a kqueue event based asynchronous event processing framework, like libuv, as well as nginx event module. For FreeBSD, in order to implement this, most likely that people would do is having a kqueue to hold all handles and one thread (because of this issue) to poll events from event queue, then, since most of the servers would not just have one CPU core, in order to leverage all CPU resources, the framework may have a thread pool (size == # of CPUs), then the polling thread needs to do event dispatching to thread pool, which works fine, however, a little bit odd, comparing to how Windows IOCP does and linux's EPOLL, in which all threads in thread pool keep reading from IOCP (or EPOLL, http://lxr.free-electrons.com/source/include/linux/wait.h#L230), and process any events that are available (no job re-dispatching required) 2. echoing to the point of "thread exits before event has been processed", even you wake up all threads, all of them may also die before reach kqueue, which is not a good reason to keep this behavior. moreover, we can make a kernel to be robust, say, impossible to be crashed by usermode's fault, but it sounds impossible to me that the kernel can take care of usermode's errors, all the kernel needs to guarantee is ensure the system is still good even there are tens of thousands of errors in usermode. 3. moreover, having a thread to handle a socket is good, however, for many cases, server applications would serve hundreds of connections in one second, which doesn't means we need to create hundreds of threads for an application, and even worse that might causes unbalanced CPU load, such as two cores, with three requests, in async mode, it's very likely that core #1 takes req 1 and 3, while core #2 takes req 2, while req 1 is very heavy, while req #2 is lightweight, then request #3 may get stuck (because of core #1 is busy on req 1, though req 2 has already finished by core #2) I believe many async IO applications/frameworks are having a very different way for kqueue because of this (my point #1) such as asio c++ library http://think-async.com/ Async IO is extreme important for today's server applications, which also have a very strong requirement not to create too many threads (#of threads == #of cpu cores), having this fix would help many developers to write asyc io applications/frameworks in the same way as windows and linux, which I think is valuable