The kernel seems to generate FreeBSD kernel code (specifically the netmap module) issues with NIC...in my case Intel dual gigabit PCle 82575 where I would get a series of: 196.115874  netmap_grab_packets bad pkt at 445 1en 2164
or igb1 watchdog timeout. It seems that the kernel and the NIC driver conflict.
This is still occurring in FreeBSD 11.2-RELEASE-p3. Please see the following URL on the FreeBSD support forum: https://forums.freebsd.org/threads/intel-ethernet-adapter-with-intel-82580-controller.67988/. Can someone please take a look. Thank you.
(In reply to wings1446 from comment #1)
Are you using jumbo frames?
(In reply to Eric Joyner from comment #2)
I'm not using jumbo frames at the moment.
Adding vmaffione@FreeBSD.org, to cc: list as the netmap expert.
Just to give you an idea of what I'm seeing just in case it helps:
Oct 29 16:23:00 kernel 580.594684  netmap_grab_packets bad pkt at 259 len 2378
Oct 29 16:23:00 kernel 580.079591  netmap_grab_packets bad pkt at 194 len 2378
Oct 29 16:22:59 kernel 579.560714  netmap_grab_packets bad pkt at 178 len 2378
Oct 29 16:22:59 kernel 579.192987  netmap_grab_packets bad pkt at 166 len 2378
Oct 29 16:22:59 kernel 579.009581  netmap_grab_packets bad pkt at 158 len 2358
Oct 29 16:22:58 kernel 578.492871  netmap_grab_packets bad pkt at 145 len 2358
Oct 29 16:22:58 kernel 578.078858  netmap_grab_packets bad pkt at 136 len 2358
Oct 29 16:22:58 kernel 577.876158  netmap_grab_packets bad pkt at 126 len 2358
Oct 29 16:22:57 kernel 577.776789  netmap_grab_packets bad pkt at 123 len 2271
Oct 29 16:22:57 kernel 577.326387  netmap_grab_packets bad pkt at 95 len 2271
(In reply to wings1446 from comment #5)
Can you detail precisely your setup and test case? I assume you are attaching with pkt-gen but I'd be curious about any other information.
A partial copy from my FreeBSD support forum post:
I'm currently using pfSense 2.4.4 with FreeBSD 11.2-RELEASE-p3. I'm using an Intel E1G44HT I340-T4 4 port PCIe Ethernet Server Adapter that's supposed to have an Intel 82580 controller (from Intel's documentation: https://ark.intel.com/products/49186/Intel-Ethernet-Server-Adapter-I340-T4). The following URL shows that that the igb(4) driver is supposed to be compatible with the ethernet adapter I listed with the 82580 controller: https://www.freebsd.org/cgi/man.cgi?igb(4) . I'm using Suricata with pfSense for an IDPS in Inline IPS mode. I'm receiving netmap_grab_packets messages which I'm told means that only certain ethernet drivers work with netmap and has nothing to do with Suricata.
Here is a post from pfSense outlining the same that's pretty detaled:
I hope this information helps.
Today I had to disable Suricata due to the netmap issue I submitted. My internet was going offline way too much.
Hi, I'm sorry, I saw this only now.
Long story short: this looks like your netmap application behaving incorrectly, or is misconfigured, and netmap duly complaining about that.
Netmap uses packet buffers of fixed size, which is 2048 by default (sysctl dev.netmap.buf_size, see netmap(4)).
That piece of code (netmap_grab_packets) forwards packets from the netmap "host TX ring" (associated to the interface igb1) and injects those into the FreeBSD kernel stack, so that kernel stack "thinks" that those packets are coming from the igb1 link.
Who puts packets in the igb1 "host TX ring"? Well, that's an userspace netmap application (maybe Suricata in your deployment? It's not clear from your description. You should check in your code who is opening netmap ports with nm_open(...) or ioctl(.., NIOCREGIF, ..).
Unfortunately, some of the packets written to the host TX ring look larger than the netmap buffer size, which means that Suricata (or whoever) is setting the len field (in struct netmap_slot) to something larger than 2048. This is clearly wrong, and those messages mean that netmap is complaining about that.
So the real question is: why is your application is doing that?
If you don't find an answer for that of course you can increase dev.netmap.buf_size, e.g. double it to 4096 ...
The real solution to your problem may be to simply disable TSO (TCP Segmentation Offloading) and LRO (Large receive offloading) in the network interfaces of your system (through ifconfig), because TSO and LRO are likely the reason why you see packets larger than the MTU (which I assume is 1500, since you said you are not using jumbo frames).
Note that TSO and LRO should always be disabled when playing with netmap (read the very end of netmap(4)).