Bug 229741

Summary: kevent is not properly adding EVFILT_READ and EVFILT_WRITE for unix sockets when used in one call
Product: Base System Reporter: Ivan Radovanovic <radovanovic>
Component: kernAssignee: freebsd-bugs (Nobody) <bugs>
Status: Closed Works As Intended    
Severity: Affects Many People CC: kevans, markj, pi, rakuco
Priority: ---    
Version: 10.1-RELEASE   
Hardware: Any   
OS: Any   
Attachments:
Description Flags
simple test none

Description Ivan Radovanovic 2018-07-12 18:51:35 UTC
Created attachment 195091 [details]
simple test

When adding two events to watch (EVFILT_READ and EVFILT_WRITE) in single call kevent is not handling them properly (at least for unix sockets). If both are added in one call which one is reported depends on their order; however if they are added using 2 calls everything works as expected

I.e.

struct kevent kev[2];
...
kevent(kqueue, kev, 2, NULL, 0, NULL);

and

struct kevent kev[2];
...
kevent(kqueue, kev, 1, NULL, 0, NULL);
kevent(kqueue, kev + 1, 1, NULL, 0, NULL); 

do not give identical results.

Simple test case is attached - expected result would be for test program to output at least one line saying:

BothReadWrite received EVFILT_WRITE
Comment 1 Ivan Radovanovic 2018-07-12 18:53:34 UTC
I don't know how to add other versions but it is the same behavior on 11.0 and 11.1
Comment 2 Kyle Evans freebsd_committer freebsd_triage 2020-04-22 01:06:30 UTC
The root of the problem here is that EV_RECEIPT on the first event will halt processing because nevents == 0; this is common amongst {Net,Free,Open}BSD, at least.

EV_RECEIPT should only be set on the last event you're passing in to get the semantics you're wanting; if an error happens on any event before that one you'll get the same EV_ERROR return (if feasible) and will not drain the queue. If no error happens, you'll hit that one and still not drain the queue because EV_RECEIPT is set.

Here's the diff I applied to your example:

<<<EOF
--- kq.c.orig	2020-04-21 20:02:08.123750000 -0500
+++ kq.c	2020-04-21 20:02:24.577357000 -0500
@@ -108,7 +108,7 @@
 
 		kq = kqueue();
 
-		EV_SET(kev, client, EVFILT_READ, EV_ADD | EV_CLEAR | EV_RECEIPT, 0, 0, 0);
+		EV_SET(kev, client, EVFILT_READ, EV_ADD | EV_CLEAR, 0, 0, 0);
 		EV_SET(kev + 1, client, EVFILT_WRITE, EV_ADD | EV_CLEAR | EV_RECEIPT, 0, 0, 0);
 		kevent(kq, kev, 2, NULL, 0, NULL);
 
@@ -131,7 +131,7 @@
 
 		kq = kqueue();
 
-		EV_SET(kev, client, EVFILT_WRITE, EV_ADD | EV_CLEAR | EV_RECEIPT, 0, 0, 0);
+		EV_SET(kev, client, EVFILT_WRITE, EV_ADD | EV_CLEAR, 0, 0, 0);
 		EV_SET(kev + 1, client, EVFILT_READ, EV_ADD | EV_CLEAR | EV_RECEIPT, 0, 0, 0);
 		kevent(kq, kev, 2, NULL, 0, NULL);
 
EOF

I'm tentatively closing this as "works as intended," but please do feel free to re-open if you believe a change is necessary here after this explanation.
Comment 3 Kyle Evans freebsd_committer freebsd_triage 2020-04-22 01:08:00 UTC
(Alternatively, you can provide enough events to hold EV_ERROR for each change)
Comment 4 commit-hook freebsd_committer freebsd_triage 2020-04-22 03:46:41 UTC
A commit references this bug:

Author: kevans
Date: Wed Apr 22 03:45:52 UTC 2020
New revision: 360182
URL: https://svnweb.freebsd.org/changeset/base/360182

Log:
  kqueue(2): add a note about EV_RECEIPT

  In the below-referenced PR, a case is attached of a simple reproducer that
  exhibits suboptimal behavior: EVFILT_READ and EVFILT_WRITE being set in the
  same kevent(2) call will only honor the first one. This is, in-fact, how
  it's supposed to work.

  A read of the manpage leads me to believe we could be more clear about this;
  right now there's a logical leap to make in the relevant statement: "When
  passed as input, it forces EV_ERROR to always be returned." -- the logical
  leap being that this indicates the caller should have allocated space for
  the change to be returned with EV_ERROR indicated in the events, or
  subsequent filters will get dropped on the floor.

  Another possible workaround that accomplishes similar effect without needing
  space for all events is just setting EV_RECEIPT on the final change being
  passed in; if any errored before it, the kqueue would not be drained. If we
  made it to the final change with EV_RECEIPT set, then we would return that
  one with EV_ERROR and still not drain the kqueue. This would seem to not be
  all that advisable.

  PR:		229741
  MFC after:	1 week

Changes:
  head/lib/libc/sys/kqueue.2
Comment 5 Kyle Evans freebsd_committer freebsd_triage 2020-04-22 03:51:20 UTC
I decided that my diff was bad and this should really just be passing in enough events to receive the EV_ERROR events. I've committed an update to the manpage to indicate that this will happen; it's technically the exact same behavior you get from non-EV_RECEIPT driven errors, but it's worth noting explicitly IMO.
Comment 6 commit-hook freebsd_committer freebsd_triage 2020-04-27 22:43:54 UTC
A commit references this bug:

Author: kevans
Date: Mon Apr 27 22:43:25 UTC 2020
New revision: 360406
URL: https://svnweb.freebsd.org/changeset/base/360406

Log:
  MFC r360182-r360183: kqueue(2): add note about EV_RECEIPT

  r360182:
  kqueue(2): add a note about EV_RECEIPT

  In the below-referenced PR, a case is attached of a simple reproducer that
  exhibits suboptimal behavior: EVFILT_READ and EVFILT_WRITE being set in the
  same kevent(2) call will only honor the first one. This is, in-fact, how
  it's supposed to work.

  A read of the manpage leads me to believe we could be more clear about this;
  right now there's a logical leap to make in the relevant statement: "When
  passed as input, it forces EV_ERROR to always be returned." -- the logical
  leap being that this indicates the caller should have allocated space for
  the change to be returned with EV_ERROR indicated in the events, or
  subsequent filters will get dropped on the floor.

  Another possible workaround that accomplishes similar effect without needing
  space for all events is just setting EV_RECEIPT on the final change being
  passed in; if any errored before it, the kqueue would not be drained. If we
  made it to the final change with EV_RECEIPT set, then we would return that
  one with EV_ERROR and still not drain the kqueue. This would seem to not be
  all that advisable.

  r360183:
  kqueue(2): de-vandalize the random sentence in the middle

  A last minute change appears to have inadvertently vandalized unrelated
  parts of the manpage with the date. =-(

  PR:		229741

Changes:
_U  stable/11/
  stable/11/lib/libc/sys/kqueue.2
_U  stable/12/
  stable/12/lib/libc/sys/kqueue.2