$uname -a FreeBSD freebsd 13.1-RELEASE FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC amd64 $sysctl hw.model hw.machine hw.ncpu hw.model: AMD Ryzen Embedded V1202B with Radeon Vega Gfx hw.machine: amd64 hw.ncpu: 2 $pciconf -lv | grep -A1 -B3 network igb0@pci0:1:0:0: class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x1533 subvendor=0x8086 subdevice=0x1533 vendor = 'Intel Corporation' device = 'I210 Gigabit Network Connection' class = network subclass = ethernet $sysctl net.inet.tcp.functions_available net.inet.tcp.functions_available: Stack D Alias PCB count freebsd * freebsd 6 In netmap native mode after bombarding the machine with packets: $netstat -m 6145/2240/8385 mbufs in use (current/cache/total) 2048/2270/4318/438928 mbuf clusters in use (current/cache/total/max) 0/508 mbuf+clusters out of packet secondary zone in use (current/cache) 0/508/508/219464 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/65026 9k jumbo clusters in use (current/cache/total/max) 0/0/0/36577 16k jumbo clusters in use (current/cache/total/max) 5632K/7132K/12764K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 sendfile syscalls 0 sendfile syscalls completed without I/O request 0 requests for I/O initiated by sendfile 0 pages read by sendfile as part of a request 0 pages were valid at time of a sendfile request 0 pages were valid and substituted to bogus page 0 pages were requested for read ahead by applications 0 pages were read ahead by sendfile 0 times sendfile encountered an already busy page 0 requests for sfbufs denied 0 requests for sfbufs delayed In generic mode: $sysctl dev.netmap.admode=2 ***bombarding the machine with frames*** $netstat -m 442160/1330/443490 mbufs in use (current/cache/total) 438080/848/438928/438928 mbuf clusters in use (current/cache/total/max) 0/508 mbuf+clusters out of packet secondary zone in use (current/cache) 0/508/508/219464 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/65026 9k jumbo clusters in use (current/cache/total/max) 0/0/0/36577 16k jumbo clusters in use (current/cache/total/max) 986700K/4060K/990760K bytes allocated to network (current/cache/total) 0/264033/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 sendfile syscalls 0 sendfile syscalls completed without I/O request 0 requests for I/O initiated by sendfile 0 pages read by sendfile as part of a request 0 pages were valid at time of a sendfile request 0 pages were valid and substituted to bogus page 0 pages were requested for read ahead by applications 0 pages were read ahead by sendfile 0 times sendfile encountered an already busy page 0 requests for sfbufs denied 0 requests for sfbufs delayed $dmesg [zone: mbuf_cluster] kern.ipc.nmbclusters limit reached The number of mbuf clusters in use doesn't decrease even if I remove the network traffic or close the netmap port. I am testing withouth sending any frames from the machine. I have reproduced the behavior on 13.2-STABLE and 14.0-CURRENT. I am running exactly the same testprogram on HW with Intel Atom processor and the same NIC(I210) without a problem.
Which netmap application are you using? Do you see any console messages from netmap on the host?
(In reply to Mark Johnston from comment #1) For testing purposes, I use a simple test application written in C which uses poll and the function below. void consume_packet(struct nm_desc* nmd) { unsigned int ri; unsigned int while_iteration; char* buf; int i; for (ri = nmd->first_rx_ring; ri <= nmd->last_rx_ring; ri++) { struct netmap_ring* rxring; rxring = NETMAP_RXRING(nmd->nifp, ri); while (!nm_ring_empty(rxring)) { i = rxring->cur; buf = NETMAP_BUF(rxring, rxring->slot[i].buf_idx); printEthernetHeader(buf); rxring->cur = rxring->head = nm_ring_next(rxring, i); } } } I cannot see any console messages or log messages from netmap on the host.
(In reply to koverskeid from comment #2) If you'd be willing to share the full application, I can try to debug. I've been working on some related bug fixes for generic mode.
Created attachment 241302 [details] simple netmap receiver program A simple netmap receiver program which can be used to reproduce the bug
(In reply to Mark Johnston from comment #3) Sure, that would be great! Added the test application, which causes the behavior described above, as an attachment.
(In reply to koverskeid from comment #5) Thanks, I can reproduce the leak.
This patch should fix it: https://reviews.freebsd.org/D39426 Out of curiosity, why are you testing generic mode when iflib implements native netmap mode?
(In reply to Mark Johnston from comment #7) I am running an application using netmap on a Beckhoff industrial PC with TC/BSD installed, which combines the TwinCAT runtime with FreeBSD. When the TwinCAT I/O system is running, it examines all incoming frames and filters out the ones with real-time relevance. All other frames are passed on to FreeBSD. Netmap in native mode does not work when the TwinCAT I/O system is running. I can only guess that this is because they try to access the same ring buffers. Anyways, my only option so far is to use generic mode. From the summary of the patch, I didn't get why netmap in generic mode doesn't leak on different HW (with Intel Atom CPU) though.
(In reply to koverskeid from comment #8) > From the summary of the patch, I didn't get why netmap in generic mode doesn't leak on different HW (with Intel Atom CPU) though. It's not obvious to me either. Maybe there is some difference between the systems such that iflib is batching calls to if_input() on one but not the other. iflib_rxeof() calls if_input() in several places, at a glance it looks like the behaviour might be different depending on whether LRO is enabled.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=5f6d37787f1e6aaf9b18392e8cff65ed4e094f2c commit 5f6d37787f1e6aaf9b18392e8cff65ed4e094f2c Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2023-04-05 20:52:41 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2023-04-05 21:07:48 +0000 netmap: Handle packet batches in generic mode ifnets are allowed to pass batches of multiple packets to if_input, linked by the m_nextpkt pointer. iflib_rxeof() sometimes does this, for example. Netmap's generic mode did not handle this and would only deliver the first packet in the batch, leaking the rest. PR: 270636 Reviewed by: vmaffione MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D39426 sys/dev/netmap/netmap_freebsd.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3d55de76e30d34a411630d986153ae803915e95f commit 3d55de76e30d34a411630d986153ae803915e95f Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2023-04-05 20:52:41 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2023-04-12 14:18:48 +0000 netmap: Handle packet batches in generic mode ifnets are allowed to pass batches of multiple packets to if_input, linked by the m_nextpkt pointer. iflib_rxeof() sometimes does this, for example. Netmap's generic mode did not handle this and would only deliver the first packet in the batch, leaking the rest. PR: 270636 Reviewed by: vmaffione MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D39426 (cherry picked from commit 5f6d37787f1e6aaf9b18392e8cff65ed4e094f2c) sys/dev/netmap/netmap_freebsd.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
Thanks for the report and for the reproducer.