Bug 207261

Summary: netmap: Doesn't do TX sync with kqueue
Product: Base System Reporter: slw
Component: kernAssignee: freebsd-net (Nobody) <net>
Status: Open ---    
Severity: Affects Some People CC: adrian, dab, net, slw, vmaffione
Priority: --- Keywords: needs-patch, needs-qa
Version: 10.2-STABLEFlags: koobs: maintainer-feedback? (vmaffione)
koobs: mfc-stable12?
koobs: mfc-stable11?
koobs: mfc-stable10?
Hardware: amd64   
OS: Any   
Attachments:
Description Flags
Example netmap program using kqueue to read or write from a netmap port. none

Description slw 2016-02-16 21:25:52 UTC
NETMAP with kevent don't do TX sync w/o next hack:

1. first register event EVFILT_WRITE with EV_DISABLE
2. second, after update tx ring, add event EVFILT_WRITE with EV_DISPATCH
Comment 1 Vincenzo Maffione freebsd_committer 2019-01-10 22:23:51 UTC
Can you:
  - specify which FreeBSD version you are using
  - specify which netmap port are you opening (ixgbe? ixl? em? vale?, ...)
  - provide a code snippet that shows the problem
?
Comment 2 slw 2019-01-25 13:20:49 UTC
(In reply to Vincenzo Maffione from comment #1)

10.2-STABLE, netmap port ixgbe.

I am don't retest after discovery in 2016.

Code snippet need to re-implement. About example:

EV_SET(kp, fd, EVFILT_WRITE, EV_ADD | EV_ENABLE, 0, 0, &event_info[kidx]);
...
while(1) {
...
  int nevent = kevent();
...
}

In this example kevent() call don't sync TX ring pointers.
Comment 3 Vincenzo Maffione freebsd_committer 2019-01-25 14:58:55 UTC
Created attachment 201394 [details]
Example netmap program using kqueue to read or write from a netmap port.

As far as I can see, now kqueue/kevent works correctly, and syncs the tail of the TX ring.
See for instance the program in attachment. I tried that on VALE and pipe ports.
Comment 4 Vincenzo Maffione freebsd_committer 2019-01-25 15:00:09 UTC
Marking in progress as it looks like this bug is not valid anymore.
Comment 5 Vincenzo Maffione freebsd_committer 2019-01-28 13:47:09 UTC
It looks like I was wrong, the TXSYNC is not being called (I tried with VALE ports and netmap pipe ports).
I'll have a look asap.
Comment 6 slw 2019-12-20 16:37:32 UTC
(In reply to Vincenzo Maffione from comment #5)

Do you have any progress on this?

My guess:

1. on kevent() no changes for netmap FD (associated w/ TX ring)
2. no events (nor RX and nor TX) in kernel for this FD (associated w/ TX ring)

 => no netmap_knread() call for this FD and no chance for txsync in netmap_poll().
Comment 7 Kubilay Kocak freebsd_committer freebsd_triage 2019-12-24 00:09:01 UTC
@Vincenzo Since no stable/10 and no 10.x-RELEASE's versions are supported any longer, what does the anticipated resolution look like with regard to merging, if and when we have a patch

Ideally (and if you don't mind), merges to all stable branches (including 10) would be nice

If stable/10 wont be included in the merge set, please set the mfc-stable10 flag to

^Triage: 

- Reset status (no In Progress without an Assignee)
- Needs patch and qa (review) to progress
Comment 8 slw 2019-12-24 13:44:51 UTC
(In reply to Kubilay Kocak from comment #7)

Latest test I am do on stable/11.
I mean stable/12 and current have same behavior and this is kqueue related.

I.e. txsync do on poll event, poll event cused by kqueue change list processing or kqueue waited events processing. No input packets case don't fire poll events and don't cause txsync.

Additional notice: multiple received packets cause massive fire of taskqueue enque fire and very high overhead (system loaded x3-x4 more comapred to poll())
Comment 9 Vincenzo Maffione freebsd_committer 2020-01-26 20:52:51 UTC
Sorry for being unresponsive on this.

The root of the problem is that when netmap was designed, kqueue was not taken in consideration. Support for kqueue was added afterwards. Because of that, there are some locking issues that make it hard for netmap to work well with kqueue.

The massive taskqueue usage that you observe are due to recent changes (r343579 and r344253) that fixed a possible deadlock (e.g. with bhyve guests).

It's unfortunate that netmap works so bad with kqueue. I would really like to improve the situation, as soon as I have some time to spend on this. It would also really help to have some opinion from people that know the kqueue internals, as I currently have very little knowledge.