Bug 258310

Summary: kevent() does not see signal with zero timeout
Product: Base System Reporter: Alex Richardson <arichardson>
Component: kernAssignee: Konstantin Belousov <kib>
Status: Closed FIXED    
Severity: Affects Only Me CC: jan.kokemueller, kib
Priority: ---    
Version: 13.0-RELEASE   
Hardware: Any   
OS: Any   

Description Alex Richardson freebsd_committer freebsd_triage 2021-09-06 10:50:30 UTC
While adding patches to natively support FreeBSD in wayland, I noticed that epoll-shim (used to emulate epoll with kqueue) didn't report signals sent to self via kill() unless I added a non-zero timeout (even in a single-threaded test program). In https://github.com/jiixyj/epoll-shim/pull/32 
Jan Kokemüller provided a reduced test case using only kqueue and kevent:

```
#ifdef NDEBUG
#undef NDEBUG
#endif

#define _GNU_SOURCE

#include <sys/types.h>

#include <sys/event.h>

#include <assert.h>
#include <errno.h>
#include <signal.h>
#include <stdlib.h>

#include <poll.h>
#include <unistd.h>

int
main(void)
{
	int rv;

	sigset_t set;
	rv = sigemptyset(&set);
	assert(rv == 0);
	rv = sigaddset(&set, SIGUSR1);
	assert(rv == 0);

	rv = sigprocmask(SIG_BLOCK, &set, NULL);
	assert(rv == 0);

	int skq = kqueue();
	assert(skq >= 0);

	struct kevent kev;
	EV_SET(&kev, SIGUSR1, EVFILT_SIGNAL, EV_ADD, 0, 0, 0);
	rv = kevent(skq, &kev, 1, NULL, 0, NULL);
	assert(rv == 0);

	int kq = kqueue();
	assert(kq >= 0);

	EV_SET(&kev, skq, EVFILT_READ, EV_ADD | EV_CLEAR, 0, 0, 0);
	rv = kevent(kq, &kev, 1, NULL, 0, NULL);
	assert(rv == 0);

	for (;;) {
		rv = kill(getpid(), SIGUSR1);
		assert(rv == 0);

		/* Turn this into `#if 1` to avoid the race. */
#if 0
		rv = kevent(kq, NULL, 0, &kev, 1, NULL);
#else
		rv = kevent(kq, NULL, 0, &kev, 1, &(struct timespec) { 0, 0 });
#endif
		assert(rv == 1);
		rv = kevent(kq, NULL, 0, &kev, 1, &(struct timespec) { 0, 0 });
		assert(rv == 0);

		rv = kevent(skq, NULL, 0, &kev, 1, &(struct timespec) { 0, 0 });
		assert(rv == 1);
		rv = kevent(skq, NULL, 0, &kev, 1, &(struct timespec) { 0, 0 });
		assert(rv == 0);

		siginfo_t siginfo;

		rv = sigtimedwait(&set, &siginfo, &(struct timespec) { 0, 0 });
		assert(rv == SIGUSR1);

		rv = sigtimedwait(&set, &siginfo, &(struct timespec) { 0, 0 });
		assert(rv < 0);
		assert(errno == EAGAIN);
	}
}
```

Is this behaviour expected or a bug?
Comment 1 Konstantin Belousov freebsd_committer freebsd_triage 2021-09-06 11:20:31 UTC
So the problem is actually not that kevent does not get signal notification.  It
is that kqueue attached to kqueue with the signal event suddenly reports no
events when you specify zero timeout, i.e. non-blocking peek.  Am I right?

This is because kqueue subsystem has to fight with the recursion and deadlock.
If kqueue has another kqueue attached to it, then notification of the attached
kqueue is delegated to a task.  See knote_enqueue()->kqueue_wakeup()->
kqueue_schedtask().  Until task run, kq->kq_count == 0 and this together
with timespec being 0 (AKA unset) causes kqueue_scan() to immediately return 0.

I am not sure if this is worth complications to make the wakeup synchronous.
Comment 2 Alex Richardson freebsd_committer freebsd_triage 2021-09-06 11:27:08 UTC
(In reply to Konstantin Belousov from comment #1)
Yes, it's only the non-blocking peek that appears to be affected.
Comment 3 Konstantin Belousov freebsd_committer freebsd_triage 2021-09-06 11:48:30 UTC
The crucial part is not the peek, but recursion.
Anyway it is relatively simple to fix it seems
https://reviews.freebsd.org/D31858
Comment 4 commit-hook freebsd_committer freebsd_triage 2021-09-06 23:44:43 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=98168a6e6c12dab8f608f6b5f5b0b175d2b87ef0

commit 98168a6e6c12dab8f608f6b5f5b0b175d2b87ef0
Author:     Konstantin Belousov <kib@FreeBSD.org>
AuthorDate: 2021-09-06 11:43:06 +0000
Commit:     Konstantin Belousov <kib@FreeBSD.org>
CommitDate: 2021-09-06 23:43:34 +0000

    kqueue: drain kqueue taskqueue if syscall tickled it

    Otherwise return from the syscall and next syscall, which could be
    kevent(2) on the kqueue that should be notified, races with the kqueue
    taskqueue thread, and potentially misses the wakeup.  This is reliably
    visible when kevent(2) only peeks into events using zeroed timeout.

    PR:     258310
    Reported by:    arichardson, Jan Kokemüller <jan.kokemueller@gmail.com>
    Reviewed by:    arichardson, markj
    Sponsored by:   The FreeBSD Foundation
    MFC after:      1 week
    Differential revision:  https://reviews.freebsd.org/D31858

 sys/kern/kern_event.c | 11 +++++++++++
 sys/kern/subr_trap.c  |  7 ++++++-
 sys/sys/event.h       |  1 +
 sys/sys/proc.h        |  2 +-
 4 files changed, 19 insertions(+), 2 deletions(-)
Comment 5 Mark Linimon freebsd_committer freebsd_triage 2021-09-07 21:20:14 UTC
^Triage: assign to committer.
Comment 6 commit-hook freebsd_committer freebsd_triage 2021-09-12 12:18:57 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=c43e14590772d8043048d7c9acf333764d21a1cf

commit c43e14590772d8043048d7c9acf333764d21a1cf
Author:     Konstantin Belousov <kib@FreeBSD.org>
AuthorDate: 2021-09-06 11:43:06 +0000
Commit:     Konstantin Belousov <kib@FreeBSD.org>
CommitDate: 2021-09-12 12:17:27 +0000

    kqueue: drain kqueue taskqueue if syscall tickled it

    PR:     258310

    (cherry picked from commit 98168a6e6c12dab8f608f6b5f5b0b175d2b87ef0)

 sys/kern/kern_event.c | 11 +++++++++++
 sys/kern/subr_trap.c  |  7 ++++++-
 sys/sys/event.h       |  1 +
 sys/sys/proc.h        |  2 +-
 4 files changed, 19 insertions(+), 2 deletions(-)