Today I tried using the hw.cxgbe.fl_pktshift sysctl to see if it would improve the overall packet rate. I was using hw.cxgbe.fl_pktshift=2 to produce 4 byte alignment with my Ethernet/IP data. After looking at the data, I realized the Ethernet frames were not being recognized. I am using the cc0 interface in netmap mode.
I captured packets from the interface and viewed them using tcpdump. It looks like the pktshift is not being accounted for by the driver as the first two byte are the EtherType from the Ethernet header.
17:42:12.555105 1a:84:9a:83:ca:44 > 00:00:8b:cb:f9:5d, ethertype Unknown (0x268f), length 811:
0x0000: 0800 4500 031d 0001 0000 4011 07bd 8fdb
I reviewed the driver code and found this segment in eth_rx():
m0->m_pkthdr.len -= sc->params.sge.fl_pktshift;
m0->m_len -= sc->params.sge.fl_pktshift;
m0->m_data += sc->params.sge.fl_pktshift;
So the driver is transparently removing the shift from the length and the data pointer. However, in service_nm_rxq() I see only:
ring->slot[fl_cidx].len = G_RSPD_LEN(lq) -
The length is being adjusted. However does the data pointer need adjusting as well? Is this sysctl not supported while in netmap mode?
I'm not aware of any way to specify to netmap that the received frame starts at
some offset from the start of the buffer.
Vincenzo, is it possible to do this? Otherwise we should just document that
fl_pktshift = 0 is the only valid setting when in netmap mode.