when using the lb utility for packet flows the rings appear to go out of bounds. System is using a 82599 based Intel adapter (ixgbe). tried with both bro 2.6.3 and bro 2.6.2. Netmap plugin tried from zkg and built from source. This problem does not exist in 11.2 p9. This may be related to driver changes ~april 2019 for ixgbe. When this issue happens lb stops reporting any incrementing of packet counters. Command issues to start lb: lb -i ix2 -p 20 -o 30 042.269723 [1721] nm_rxsync_prologue ix2 RX13: fail 'head < kring->nr_hwcur || head > kring->nr_hwtail' h 1442 c 1442 t 1443 rh 1442 rc 1442 rt 1443 hc 1443 ht 1443 042.269754 [1766] netmap_ring_reinit called for ix2 RX13 042.282035 [1721] nm_rxsync_prologue ix2 RX13: fail 'head < kring->nr_hwcur || head > kring->nr_hwtail' h 1826 c 1826 t 1827 rh 1826 rc 1826 rt 1827 hc 1827 ht 1827 042.282065 [1766] netmap_ring_reinit called for ix2 RX13 042.286129 [1721] nm_rxsync_prologue ix2 RX8: fail 'head < kring->nr_hwcur || head > kring->nr_hwtail' h 2801 c 2801 t 2804 rh 2801 rc 2801 rt 2804 hc 2804 ht 2804 042.286155 [1766] netmap_ring_reinit called for ix2 RX8 042.301701 [1721] nm_rxsync_prologue ix2 RX5: fail 'head < kring->nr_hwcur || head > kring->nr_hwtail' h 3990 c 3990 t 3988 rh 3990 rc 3990 rt 3988 hc 3987 ht 3988 042.301729 [1766] netmap_ring_reinit called for ix2 RX5 042.304510 [1721] nm_rxsync_prologue ix2 RX11: fail 'head < kring->nr_hwcur || head > kring->nr_hwtail' h 302 c 302 t 304 rh 302 rc 302 rt 304 hc 304 ht 304 042.304533 [1766] netmap_ring_reinit called for ix2 RX11 042.304558 [1766] netmap_ring_reinit called for ix2 RX15 042.307984 [1766] netmap_ring_reinit called for ix2 RX6 042.308521 [1766] netmap_ring_reinit called for ix2 RX7 042.313627 [1766] netmap_ring_reinit called for ix2 RX6 042.314349 [1766] netmap_ring_reinit called for ix2 RX1 042.317253 [1723] nm_rxsync_prologue ix2 RX5: fail 'cur < head || cur > kring->nr_hwtail' h 273 c 272 t 273 rh 273 rc 272 rt 273 hc 272 ht 273 042.319884 [1726] nm_rxsync_prologue ix2 RX4: fail 'head < kring->nr_hwcur && head > kring->nr_hwtail' h 4094 c 4094 t 2 rh 4094 rc 4094 rt 2 hc 4095 ht 2 042.321352 [1723] nm_rxsync_prologue ix2 RX10: fail 'cur < head || cur > kring->nr_hwtail' h 432 c 431 t 432 rh 432 rc 431 rt 432 hc 431 ht 432 042.321993 [1723] nm_rxsync_prologue ix2 RX1: fail 'cur < head || cur > kring->nr_hwtail' h 3751 c 3750 t 3751 rh 3751 rc 3750 rt 3751 hc 3750 ht 3751 042.360195 [1723] nm_rxsync_prologue ix2 RX2: fail 'cur < head || cur > kring->nr_hwtail' h 2678 c 2677 t 2678 rh 2678 rc 2677 rt 2678 hc 2677 ht 2678 042.380046 [1723] nm_rxsync_prologue ix2 RX6: fail 'cur < head || cur > kring->nr_hwtail' h 3787 c 3786 t 3787 rh 3787 rc 3786 rt 3787 hc 3786 ht 3787
Output of lb. Packets increment fine until bro-netmap attaches.
{"ts":1566432283.608048,"interface":"netmap:ix2","output_ring":null,"packets_received":72445437,"packets_forwarded":81900,"packets_dropped":72363537,"non_ip_packets":4198,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"free_buffer_slots":0} {"ts":1566432304.211968,"interface":"netmap:ix2{0/xT@1","output_ring":0,"packets_forwarded":4095,"packets_dropped":2769476,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432304.211968,"interface":"netmap:ix2{1/xT@1","output_ring":1,"packets_forwarded":4095,"packets_dropped":5054369,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432304.211968,"interface":"netmap:ix2{2/xT@1","output_ring":2,"packets_forwarded":4095,"packets_dropped":2752940,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432304.211968,"interface":"netmap:ix2{3/xT@1","output_ring":3,"packets_forwarded":4095,"packets_dropped":3039503,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432304.211968,"interface":"netmap:ix2{4/xT@1","output_ring":4,"packets_forwarded":4095,"packets_dropped":9581988,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432304.211968,"interface":"netmap:ix2{5/xT@1","output_ring":5,"packets_forwarded":4095,"packets_dropped":3085226,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432304.211968,"interface":"netmap:ix2{6/xT@1","output_ring":6,"packets_forwarded":4095,"packets_dropped":3838018,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432304.211968,"interface":"netmap:ix2{7/xT@1","output_ring":7,"packets_forwarded":4095,"packets_dropped":2593070,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432304.211968,"interface":"netmap:ix2{8/xT@1","output_ring":8,"packets_forwarded":4095,"packets_dropped":2395212,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432304.211968,"interface":"netmap:ix2{9/xT@1","output_ring":9,"packets_forwarded":4095,"packets_dropped":2625455,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432304.211968,"interface":"netmap:ix2{10/xT@1","output_ring":10,"packets_forwarded":4095,"packets_dropped":3050539,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432304.211968,"interface":"netmap:ix2{11/xT@1","output_ring":11,"packets_forwarded":4095,"packets_dropped":3292134,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432304.211968,"interface":"netmap:ix2{12/xT@1","output_ring":12,"packets_forwarded":4095,"packets_dropped":3922730,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432304.211968,"interface":"netmap:ix2{13/xT@1","output_ring":13,"packets_forwarded":4095,"packets_dropped":3056607,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432304.211968,"interface":"netmap:ix2{14/xT@1","output_ring":14,"packets_forwarded":4095,"packets_dropped":2903872,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432304.211968,"interface":"netmap:ix2{15/xT@1","output_ring":15,"packets_forwarded":4095,"packets_dropped":3084561,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432304.211968,"interface":"netmap:ix2{16/xT@1","output_ring":16,"packets_forwarded":4095,"packets_dropped":3407359,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432304.211968,"interface":"netmap:ix2{17/xT@1","output_ring":17,"packets_forwarded":4095,"packets_dropped":3478362,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432304.211968,"interface":"netmap:ix2{18/xT@1","output_ring":18,"packets_forwarded":4095,"packets_dropped":2682530,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432304.211968,"interface":"netmap:ix2{19/xT@1","output_ring":19,"packets_forwarded":4095,"packets_dropped":5749586,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432304.211968,"interface":"netmap:ix2","output_ring":null,"packets_received":72445437,"packets_forwarded":81900,"packets_dropped":72363537,"non_ip_packets":4198,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"free_buffer_slots":0} {"ts":1566432324.728495,"interface":"netmap:ix2{0/xT@1","output_ring":0,"packets_forwarded":4095,"packets_dropped":2769476,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432324.728495,"interface":"netmap:ix2{1/xT@1","output_ring":1,"packets_forwarded":4095,"packets_dropped":5054369,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432324.728495,"interface":"netmap:ix2{2/xT@1","output_ring":2,"packets_forwarded":4095,"packets_dropped":2752940,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432324.728495,"interface":"netmap:ix2{3/xT@1","output_ring":3,"packets_forwarded":4095,"packets_dropped":3039503,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432324.728495,"interface":"netmap:ix2{4/xT@1","output_ring":4,"packets_forwarded":4095,"packets_dropped":9581988,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432324.728495,"interface":"netmap:ix2{5/xT@1","output_ring":5,"packets_forwarded":4095,"packets_dropped":3085226,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432324.728495,"interface":"netmap:ix2{6/xT@1","output_ring":6,"packets_forwarded":4095,"packets_dropped":3838018,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432324.728495,"interface":"netmap:ix2{7/xT@1","output_ring":7,"packets_forwarded":4095,"packets_dropped":2593070,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432324.728495,"interface":"netmap:ix2{8/xT@1","output_ring":8,"packets_forwarded":4095,"packets_dropped":2395212,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432324.728495,"interface":"netmap:ix2{9/xT@1","output_ring":9,"packets_forwarded":4095,"packets_dropped":2625455,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432324.728495,"interface":"netmap:ix2{10/xT@1","output_ring":10,"packets_forwarded":4095,"packets_dropped":3050539,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432324.728495,"interface":"netmap:ix2{11/xT@1","output_ring":11,"packets_forwarded":4095,"packets_dropped":3292134,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432324.728495,"interface":"netmap:ix2{12/xT@1","output_ring":12,"packets_forwarded":4095,"packets_dropped":3922730,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432324.728495,"interface":"netmap:ix2{13/xT@1","output_ring":13,"packets_forwarded":4095,"packets_dropped":3056607,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432324.728495,"interface":"netmap:ix2{14/xT@1","output_ring":14,"packets_forwarded":4095,"packets_dropped":2903872,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432324.728495,"interface":"netmap:ix2{15/xT@1","output_ring":15,"packets_forwarded":4095,"packets_dropped":3084561,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432324.728495,"interface":"netmap:ix2{16/xT@1","output_ring":16,"packets_forwarded":4095,"packets_dropped":3407359,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432324.728495,"interface":"netmap:ix2{17/xT@1","output_ring":17,"packets_forwarded":4095,"packets_dropped":3478362,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432324.728495,"interface":"netmap:ix2{18/xT@1","output_ring":18,"packets_forwarded":4095,"packets_dropped":2682530,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432324.728495,"interface":"netmap:ix2{19/xT@1","output_ring":19,"packets_forwarded":4095,"packets_dropped":5749586,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432324.728495,"interface":"netmap:ix2","output_ring":null,"packets_received":72445437,"packets_forwarded":81900,"packets_dropped":72363537,"non_ip_packets":4198,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"free_buffer_slots":0} {"ts":1566432345.246671,"interface":"netmap:ix2{0/xT@1","output_ring":0,"packets_forwarded":4095,"packets_dropped":2769476,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432345.246671,"interface":"netmap:ix2{1/xT@1","output_ring":1,"packets_forwarded":4095,"packets_dropped":5054369,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432345.246671,"interface":"netmap:ix2{2/xT@1","output_ring":2,"packets_forwarded":4095,"packets_dropped":2752940,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432345.246671,"interface":"netmap:ix2{3/xT@1","output_ring":3,"packets_forwarded":4095,"packets_dropped":3039503,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432345.246671,"interface":"netmap:ix2{4/xT@1","output_ring":4,"packets_forwarded":4095,"packets_dropped":9581988,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432345.246671,"interface":"netmap:ix2{5/xT@1","output_ring":5,"packets_forwarded":4095,"packets_dropped":3085226,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432345.246671,"interface":"netmap:ix2{6/xT@1","output_ring":6,"packets_forwarded":4095,"packets_dropped":3838018,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432345.246671,"interface":"netmap:ix2{7/xT@1","output_ring":7,"packets_forwarded":4095,"packets_dropped":2593070,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432345.246671,"interface":"netmap:ix2{8/xT@1","output_ring":8,"packets_forwarded":4095,"packets_dropped":2395212,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432345.246671,"interface":"netmap:ix2{9/xT@1","output_ring":9,"packets_forwarded":4095,"packets_dropped":2625455,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432345.246671,"interface":"netmap:ix2{10/xT@1","output_ring":10,"packets_forwarded":4095,"packets_dropped":3050539,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432345.246671,"interface":"netmap:ix2{11/xT@1","output_ring":11,"packets_forwarded":4095,"packets_dropped":3292134,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432345.246671,"interface":"netmap:ix2{12/xT@1","output_ring":12,"packets_forwarded":4095,"packets_dropped":3922730,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432345.246671,"interface":"netmap:ix2{13/xT@1","output_ring":13,"packets_forwarded":4095,"packets_dropped":3056607,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432345.246671,"interface":"netmap:ix2{14/xT@1","output_ring":14,"packets_forwarded":4095,"packets_dropped":2903872,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432345.246671,"interface":"netmap:ix2{15/xT@1","output_ring":15,"packets_forwarded":4095,"packets_dropped":3084561,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432345.246671,"interface":"netmap:ix2{16/xT@1","output_ring":16,"packets_forwarded":4095,"packets_dropped":3407359,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432345.246671,"interface":"netmap:ix2{17/xT@1","output_ring":17,"packets_forwarded":4095,"packets_dropped":3478362,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432345.246671,"interface":"netmap:ix2{18/xT@1","output_ring":18,"packets_forwarded":4095,"packets_dropped":2682530,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432345.246671,"interface":"netmap:ix2{19/xT@1","output_ring":19,"packets_forwarded":4095,"packets_dropped":5749586,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432345.246671,"interface":"netmap:ix2","output_ring":null,"packets_received":72445437,"packets_forwarded":81900,"packets_dropped":72363537,"non_ip_packets":4198,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"free_buffer_slots":0} {"ts":1566432365.699971,"interface":"netmap:ix2{0/xT@1","output_ring":0,"packets_forwarded":4095,"packets_dropped":2769476,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432365.699971,"interface":"netmap:ix2{1/xT@1","output_ring":1,"packets_forwarded":4095,"packets_dropped":5054369,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432365.699971,"interface":"netmap:ix2{2/xT@1","output_ring":2,"packets_forwarded":4095,"packets_dropped":2752940,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432365.699971,"interface":"netmap:ix2{3/xT@1","output_ring":3,"packets_forwarded":4095,"packets_dropped":3039503,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432365.699971,"interface":"netmap:ix2{4/xT@1","output_ring":4,"packets_forwarded":4095,"packets_dropped":9581988,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432365.699971,"interface":"netmap:ix2{5/xT@1","output_ring":5,"packets_forwarded":4095,"packets_dropped":3085226,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432365.699971,"interface":"netmap:ix2{6/xT@1","output_ring":6,"packets_forwarded":4095,"packets_dropped":3838018,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432365.699971,"interface":"netmap:ix2{7/xT@1","output_ring":7,"packets_forwarded":4095,"packets_dropped":2593070,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432365.699971,"interface":"netmap:ix2{8/xT@1","output_ring":8,"packets_forwarded":4095,"packets_dropped":2395212,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432365.699971,"interface":"netmap:ix2{9/xT@1","output_ring":9,"packets_forwarded":4095,"packets_dropped":2625455,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432365.699971,"interface":"netmap:ix2{10/xT@1","output_ring":10,"packets_forwarded":4095,"packets_dropped":3050539,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432365.699971,"interface":"netmap:ix2{11/xT@1","output_ring":11,"packets_forwarded":4095,"packets_dropped":3292134,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432365.699971,"interface":"netmap:ix2{12/xT@1","output_ring":12,"packets_forwarded":4095,"packets_dropped":3922730,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432365.699971,"interface":"netmap:ix2{13/xT@1","output_ring":13,"packets_forwarded":4095,"packets_dropped":3056607,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432365.699971,"interface":"netmap:ix2{14/xT@1","output_ring":14,"packets_forwarded":4095,"packets_dropped":2903872,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432365.699971,"interface":"netmap:ix2{15/xT@1","output_ring":15,"packets_forwarded":4095,"packets_dropped":3084561,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432365.699971,"interface":"netmap:ix2{16/xT@1","output_ring":16,"packets_forwarded":4095,"packets_dropped":3407359,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432365.699971,"interface":"netmap:ix2{17/xT@1","output_ring":17,"packets_forwarded":4095,"packets_dropped":3478362,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432365.699971,"interface":"netmap:ix2{18/xT@1","output_ring":18,"packets_forwarded":4095,"packets_dropped":2682530,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432365.699971,"interface":"netmap:ix2{19/xT@1","output_ring":19,"packets_forwarded":4095,"packets_dropped":5749586,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} {"ts":1566432365.699971,"interface":"netmap:ix2","output_ring":null,"packets_received":72445437,"packets_forwarded":81900,"packets_dropped":72363537,"non_ip_packets":4198,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"free_buffer_slots":0}
This looks like related to this issue https://github.com/luigirizzo/netmap/issues/600
If 11.2-p9 works, it would help to see what changed in ixgbe and ixl between 11.2-p9 and 11.3. Are you able to generate a diff or provide pointers to the exact two source trees you are using?
This is the binary build of 11.2p9 (working) and 11.3p2 (non working) with netmap and ixgbe built in to kernel. Nic is 82599 (intel x520). Using Bro/Zeek 2.6.3 with bro-netmap plugin. Issue seems to really trigger once it starts pulling packets from the buffers which seems to trigger the out of bounds pointer. Nic output in dmesg: ix2: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k> port 0x8020-0x803f mem 0xca080000-0xca0fffff,0xca104000-0xca107fff irq 64 at device 0.0 numa-domain 1 on pci21 ix2: Using MSI-X interrupts with 17 vectors ix2: Ethernet address: ac:1f:6b:2d:f2:3c ix2: PCI Express Bus: Speed 5.0GT/s Width x8 ix2: netmap queues/slots: TX 16/4096, RX 16/4096 ix2: permanently promiscuous mode enabled Please let me know anything else I can include to help.
Some differences of note: in IXGBE driver: diff sys/dev/ixgbe/ixgbe_netmap.c 11.2/sys/dev/ixgbe/ixgbe_netmap.c 33c33 < /*$FreeBSD: releng/11.3/sys/dev/ixgbe/ixgbe_netmap.c 344047 2019-02-12 09:26:05Z vmaffione $*/ --- > /*$FreeBSD: releng/11.2/sys/dev/ixgbe/ixgbe_netmap.c 332481 2018-04-13 17:45:54Z shurd $*/ 61c61 < * $FreeBSD: releng/11.3/sys/dev/ixgbe/ixgbe_netmap.c 344047 2019-02-12 09:26:05Z vmaffione $ --- > * $FreeBSD: releng/11.2/sys/dev/ixgbe/ixgbe_netmap.c 332481 2018-04-13 17:45:54Z shurd $ 121c121 < nm_prinf("%s read HLREG 0x%x rxc 0x%x", --- > D("%s read HLREG 0x%x rxc 0x%x", 136c136 < nm_prinf("%s write HLREG 0x%x rxc 0x%x", --- > D("%s write HLREG 0x%x rxc 0x%x", 343,344c343,344 < if (unlikely(nic_i >= kring->nkr_num_slots)) { < nm_prinf("TDH wrap %d", nic_i); --- > if (nic_i >= kring->nkr_num_slots) { /* XXX can it happen ? */ > D("TDH wrap %d", nic_i); 411a412 > uint16_t slot_flags = kring->nkr_slot_flags; 423c424 < ring->slot[nm_i].flags = 0; --- > ring->slot[nm_i].flags = slot_flags;