240023 – netmap lb pointer out of bounds on ixgbe

Bug 240023 - netmap lb pointer out of bounds on ixgbe

Summary: netmap lb pointer out of bounds on ixgbe

Status:	New

Alias:	None

Product:	Base System
Classification:	Unclassified
Component:	kern (show other bugs)
Version:	11.3-RELEASE
Hardware:	amd64 Any

Importance:	--- Affects Some People
Assignee:	freebsd-net (Nobody)

URL:
Keywords:	IntelNetworking

Depends on:
Blocks:

Reported:	2019-08-21 18:30 UTC by mike
Modified:	2019-09-09 18:45 UTC (History)
CC List:	3 users (show)

See Also:	https://github.com/luigirizzo/netmap/issues/600

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.

Description mike 2019-08-21 18:30:31 UTC

when using the lb utility for packet flows the rings appear to go out of bounds. System is using a 82599 based Intel adapter (ixgbe). tried with both bro 2.6.3 and bro 2.6.2. Netmap plugin tried from zkg and built from source.  This problem does not exist in 11.2 p9. This may be related to driver changes ~april 2019 for ixgbe. When this issue happens lb stops reporting any incrementing of packet counters. 

Command issues to start lb: lb -i ix2 -p 20 -o 30



042.269723 [1721] nm_rxsync_prologue        ix2 RX13: fail 'head < kring->nr_hwcur || head > kring->nr_hwtail' h 1442 c 1442 t 1443 rh 1442 rc 1442 rt 1443 hc 1443 ht 1443
042.269754 [1766] netmap_ring_reinit        called for ix2 RX13
042.282035 [1721] nm_rxsync_prologue        ix2 RX13: fail 'head < kring->nr_hwcur || head > kring->nr_hwtail' h 1826 c 1826 t 1827 rh 1826 rc 1826 rt 1827 hc 1827 ht 1827
042.282065 [1766] netmap_ring_reinit        called for ix2 RX13
042.286129 [1721] nm_rxsync_prologue        ix2 RX8: fail 'head < kring->nr_hwcur || head > kring->nr_hwtail' h 2801 c 2801 t 2804 rh 2801 rc 2801 rt 2804 hc 2804 ht 2804
042.286155 [1766] netmap_ring_reinit        called for ix2 RX8
042.301701 [1721] nm_rxsync_prologue        ix2 RX5: fail 'head < kring->nr_hwcur || head > kring->nr_hwtail' h 3990 c 3990 t 3988 rh 3990 rc 3990 rt 3988 hc 3987 ht 3988
042.301729 [1766] netmap_ring_reinit        called for ix2 RX5
042.304510 [1721] nm_rxsync_prologue        ix2 RX11: fail 'head < kring->nr_hwcur || head > kring->nr_hwtail' h 302 c 302 t 304 rh 302 rc 302 rt 304 hc 304 ht 304
042.304533 [1766] netmap_ring_reinit        called for ix2 RX11
042.304558 [1766] netmap_ring_reinit        called for ix2 RX15
042.307984 [1766] netmap_ring_reinit        called for ix2 RX6
042.308521 [1766] netmap_ring_reinit        called for ix2 RX7
042.313627 [1766] netmap_ring_reinit        called for ix2 RX6
042.314349 [1766] netmap_ring_reinit        called for ix2 RX1
042.317253 [1723] nm_rxsync_prologue        ix2 RX5: fail 'cur < head || cur > kring->nr_hwtail' h 273 c 272 t 273 rh 273 rc 272 rt 273 hc 272 ht 273
042.319884 [1726] nm_rxsync_prologue        ix2 RX4: fail 'head < kring->nr_hwcur && head > kring->nr_hwtail' h 4094 c 4094 t 2 rh 4094 rc 4094 rt 2 hc 4095 ht 2
042.321352 [1723] nm_rxsync_prologue        ix2 RX10: fail 'cur < head || cur > kring->nr_hwtail' h 432 c 431 t 432 rh 432 rc 431 rt 432 hc 431 ht 432
042.321993 [1723] nm_rxsync_prologue        ix2 RX1: fail 'cur < head || cur > kring->nr_hwtail' h 3751 c 3750 t 3751 rh 3751 rc 3750 rt 3751 hc 3750 ht 3751
042.360195 [1723] nm_rxsync_prologue        ix2 RX2: fail 'cur < head || cur > kring->nr_hwtail' h 2678 c 2677 t 2678 rh 2678 rc 2677 rt 2678 hc 2677 ht 2678
042.380046 [1723] nm_rxsync_prologue        ix2 RX6: fail 'cur < head || cur > kring->nr_hwtail' h 3787 c 3786 t 3787 rh 3787 rc 3786 rt 3787 hc 3786 ht 3787

Comment 1 mike 2019-08-22 00:07:11 UTC

Output of lb. Packets increment fine until bro-netmap attaches.

Comment 2 mike 2019-08-22 00:07:32 UTC

{"ts":1566432283.608048,"interface":"netmap:ix2","output_ring":null,"packets_received":72445437,"packets_forwarded":81900,"packets_dropped":72363537,"non_ip_packets":4198,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"free_buffer_slots":0}
{"ts":1566432304.211968,"interface":"netmap:ix2{0/xT@1","output_ring":0,"packets_forwarded":4095,"packets_dropped":2769476,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432304.211968,"interface":"netmap:ix2{1/xT@1","output_ring":1,"packets_forwarded":4095,"packets_dropped":5054369,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432304.211968,"interface":"netmap:ix2{2/xT@1","output_ring":2,"packets_forwarded":4095,"packets_dropped":2752940,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432304.211968,"interface":"netmap:ix2{3/xT@1","output_ring":3,"packets_forwarded":4095,"packets_dropped":3039503,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432304.211968,"interface":"netmap:ix2{4/xT@1","output_ring":4,"packets_forwarded":4095,"packets_dropped":9581988,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432304.211968,"interface":"netmap:ix2{5/xT@1","output_ring":5,"packets_forwarded":4095,"packets_dropped":3085226,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432304.211968,"interface":"netmap:ix2{6/xT@1","output_ring":6,"packets_forwarded":4095,"packets_dropped":3838018,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432304.211968,"interface":"netmap:ix2{7/xT@1","output_ring":7,"packets_forwarded":4095,"packets_dropped":2593070,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432304.211968,"interface":"netmap:ix2{8/xT@1","output_ring":8,"packets_forwarded":4095,"packets_dropped":2395212,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432304.211968,"interface":"netmap:ix2{9/xT@1","output_ring":9,"packets_forwarded":4095,"packets_dropped":2625455,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432304.211968,"interface":"netmap:ix2{10/xT@1","output_ring":10,"packets_forwarded":4095,"packets_dropped":3050539,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432304.211968,"interface":"netmap:ix2{11/xT@1","output_ring":11,"packets_forwarded":4095,"packets_dropped":3292134,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432304.211968,"interface":"netmap:ix2{12/xT@1","output_ring":12,"packets_forwarded":4095,"packets_dropped":3922730,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432304.211968,"interface":"netmap:ix2{13/xT@1","output_ring":13,"packets_forwarded":4095,"packets_dropped":3056607,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432304.211968,"interface":"netmap:ix2{14/xT@1","output_ring":14,"packets_forwarded":4095,"packets_dropped":2903872,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432304.211968,"interface":"netmap:ix2{15/xT@1","output_ring":15,"packets_forwarded":4095,"packets_dropped":3084561,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432304.211968,"interface":"netmap:ix2{16/xT@1","output_ring":16,"packets_forwarded":4095,"packets_dropped":3407359,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432304.211968,"interface":"netmap:ix2{17/xT@1","output_ring":17,"packets_forwarded":4095,"packets_dropped":3478362,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432304.211968,"interface":"netmap:ix2{18/xT@1","output_ring":18,"packets_forwarded":4095,"packets_dropped":2682530,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432304.211968,"interface":"netmap:ix2{19/xT@1","output_ring":19,"packets_forwarded":4095,"packets_dropped":5749586,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432304.211968,"interface":"netmap:ix2","output_ring":null,"packets_received":72445437,"packets_forwarded":81900,"packets_dropped":72363537,"non_ip_packets":4198,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"free_buffer_slots":0}
{"ts":1566432324.728495,"interface":"netmap:ix2{0/xT@1","output_ring":0,"packets_forwarded":4095,"packets_dropped":2769476,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432324.728495,"interface":"netmap:ix2{1/xT@1","output_ring":1,"packets_forwarded":4095,"packets_dropped":5054369,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432324.728495,"interface":"netmap:ix2{2/xT@1","output_ring":2,"packets_forwarded":4095,"packets_dropped":2752940,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432324.728495,"interface":"netmap:ix2{3/xT@1","output_ring":3,"packets_forwarded":4095,"packets_dropped":3039503,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432324.728495,"interface":"netmap:ix2{4/xT@1","output_ring":4,"packets_forwarded":4095,"packets_dropped":9581988,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432324.728495,"interface":"netmap:ix2{5/xT@1","output_ring":5,"packets_forwarded":4095,"packets_dropped":3085226,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432324.728495,"interface":"netmap:ix2{6/xT@1","output_ring":6,"packets_forwarded":4095,"packets_dropped":3838018,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432324.728495,"interface":"netmap:ix2{7/xT@1","output_ring":7,"packets_forwarded":4095,"packets_dropped":2593070,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432324.728495,"interface":"netmap:ix2{8/xT@1","output_ring":8,"packets_forwarded":4095,"packets_dropped":2395212,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432324.728495,"interface":"netmap:ix2{9/xT@1","output_ring":9,"packets_forwarded":4095,"packets_dropped":2625455,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432324.728495,"interface":"netmap:ix2{10/xT@1","output_ring":10,"packets_forwarded":4095,"packets_dropped":3050539,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432324.728495,"interface":"netmap:ix2{11/xT@1","output_ring":11,"packets_forwarded":4095,"packets_dropped":3292134,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432324.728495,"interface":"netmap:ix2{12/xT@1","output_ring":12,"packets_forwarded":4095,"packets_dropped":3922730,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432324.728495,"interface":"netmap:ix2{13/xT@1","output_ring":13,"packets_forwarded":4095,"packets_dropped":3056607,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432324.728495,"interface":"netmap:ix2{14/xT@1","output_ring":14,"packets_forwarded":4095,"packets_dropped":2903872,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432324.728495,"interface":"netmap:ix2{15/xT@1","output_ring":15,"packets_forwarded":4095,"packets_dropped":3084561,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432324.728495,"interface":"netmap:ix2{16/xT@1","output_ring":16,"packets_forwarded":4095,"packets_dropped":3407359,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432324.728495,"interface":"netmap:ix2{17/xT@1","output_ring":17,"packets_forwarded":4095,"packets_dropped":3478362,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432324.728495,"interface":"netmap:ix2{18/xT@1","output_ring":18,"packets_forwarded":4095,"packets_dropped":2682530,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432324.728495,"interface":"netmap:ix2{19/xT@1","output_ring":19,"packets_forwarded":4095,"packets_dropped":5749586,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432324.728495,"interface":"netmap:ix2","output_ring":null,"packets_received":72445437,"packets_forwarded":81900,"packets_dropped":72363537,"non_ip_packets":4198,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"free_buffer_slots":0}
{"ts":1566432345.246671,"interface":"netmap:ix2{0/xT@1","output_ring":0,"packets_forwarded":4095,"packets_dropped":2769476,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432345.246671,"interface":"netmap:ix2{1/xT@1","output_ring":1,"packets_forwarded":4095,"packets_dropped":5054369,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432345.246671,"interface":"netmap:ix2{2/xT@1","output_ring":2,"packets_forwarded":4095,"packets_dropped":2752940,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432345.246671,"interface":"netmap:ix2{3/xT@1","output_ring":3,"packets_forwarded":4095,"packets_dropped":3039503,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432345.246671,"interface":"netmap:ix2{4/xT@1","output_ring":4,"packets_forwarded":4095,"packets_dropped":9581988,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432345.246671,"interface":"netmap:ix2{5/xT@1","output_ring":5,"packets_forwarded":4095,"packets_dropped":3085226,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432345.246671,"interface":"netmap:ix2{6/xT@1","output_ring":6,"packets_forwarded":4095,"packets_dropped":3838018,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432345.246671,"interface":"netmap:ix2{7/xT@1","output_ring":7,"packets_forwarded":4095,"packets_dropped":2593070,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432345.246671,"interface":"netmap:ix2{8/xT@1","output_ring":8,"packets_forwarded":4095,"packets_dropped":2395212,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432345.246671,"interface":"netmap:ix2{9/xT@1","output_ring":9,"packets_forwarded":4095,"packets_dropped":2625455,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432345.246671,"interface":"netmap:ix2{10/xT@1","output_ring":10,"packets_forwarded":4095,"packets_dropped":3050539,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432345.246671,"interface":"netmap:ix2{11/xT@1","output_ring":11,"packets_forwarded":4095,"packets_dropped":3292134,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432345.246671,"interface":"netmap:ix2{12/xT@1","output_ring":12,"packets_forwarded":4095,"packets_dropped":3922730,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432345.246671,"interface":"netmap:ix2{13/xT@1","output_ring":13,"packets_forwarded":4095,"packets_dropped":3056607,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432345.246671,"interface":"netmap:ix2{14/xT@1","output_ring":14,"packets_forwarded":4095,"packets_dropped":2903872,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432345.246671,"interface":"netmap:ix2{15/xT@1","output_ring":15,"packets_forwarded":4095,"packets_dropped":3084561,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432345.246671,"interface":"netmap:ix2{16/xT@1","output_ring":16,"packets_forwarded":4095,"packets_dropped":3407359,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432345.246671,"interface":"netmap:ix2{17/xT@1","output_ring":17,"packets_forwarded":4095,"packets_dropped":3478362,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432345.246671,"interface":"netmap:ix2{18/xT@1","output_ring":18,"packets_forwarded":4095,"packets_dropped":2682530,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432345.246671,"interface":"netmap:ix2{19/xT@1","output_ring":19,"packets_forwarded":4095,"packets_dropped":5749586,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432345.246671,"interface":"netmap:ix2","output_ring":null,"packets_received":72445437,"packets_forwarded":81900,"packets_dropped":72363537,"non_ip_packets":4198,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"free_buffer_slots":0}
{"ts":1566432365.699971,"interface":"netmap:ix2{0/xT@1","output_ring":0,"packets_forwarded":4095,"packets_dropped":2769476,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432365.699971,"interface":"netmap:ix2{1/xT@1","output_ring":1,"packets_forwarded":4095,"packets_dropped":5054369,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432365.699971,"interface":"netmap:ix2{2/xT@1","output_ring":2,"packets_forwarded":4095,"packets_dropped":2752940,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432365.699971,"interface":"netmap:ix2{3/xT@1","output_ring":3,"packets_forwarded":4095,"packets_dropped":3039503,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432365.699971,"interface":"netmap:ix2{4/xT@1","output_ring":4,"packets_forwarded":4095,"packets_dropped":9581988,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432365.699971,"interface":"netmap:ix2{5/xT@1","output_ring":5,"packets_forwarded":4095,"packets_dropped":3085226,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432365.699971,"interface":"netmap:ix2{6/xT@1","output_ring":6,"packets_forwarded":4095,"packets_dropped":3838018,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432365.699971,"interface":"netmap:ix2{7/xT@1","output_ring":7,"packets_forwarded":4095,"packets_dropped":2593070,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432365.699971,"interface":"netmap:ix2{8/xT@1","output_ring":8,"packets_forwarded":4095,"packets_dropped":2395212,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432365.699971,"interface":"netmap:ix2{9/xT@1","output_ring":9,"packets_forwarded":4095,"packets_dropped":2625455,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432365.699971,"interface":"netmap:ix2{10/xT@1","output_ring":10,"packets_forwarded":4095,"packets_dropped":3050539,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432365.699971,"interface":"netmap:ix2{11/xT@1","output_ring":11,"packets_forwarded":4095,"packets_dropped":3292134,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432365.699971,"interface":"netmap:ix2{12/xT@1","output_ring":12,"packets_forwarded":4095,"packets_dropped":3922730,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432365.699971,"interface":"netmap:ix2{13/xT@1","output_ring":13,"packets_forwarded":4095,"packets_dropped":3056607,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432365.699971,"interface":"netmap:ix2{14/xT@1","output_ring":14,"packets_forwarded":4095,"packets_dropped":2903872,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432365.699971,"interface":"netmap:ix2{15/xT@1","output_ring":15,"packets_forwarded":4095,"packets_dropped":3084561,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432365.699971,"interface":"netmap:ix2{16/xT@1","output_ring":16,"packets_forwarded":4095,"packets_dropped":3407359,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432365.699971,"interface":"netmap:ix2{17/xT@1","output_ring":17,"packets_forwarded":4095,"packets_dropped":3478362,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432365.699971,"interface":"netmap:ix2{18/xT@1","output_ring":18,"packets_forwarded":4095,"packets_dropped":2682530,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432365.699971,"interface":"netmap:ix2{19/xT@1","output_ring":19,"packets_forwarded":4095,"packets_dropped":5749586,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0}
{"ts":1566432365.699971,"interface":"netmap:ix2","output_ring":null,"packets_received":72445437,"packets_forwarded":81900,"packets_dropped":72363537,"non_ip_packets":4198,"data_forward_rate_Mbps":0.0000,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":0.0000,"packet_drop_rate_kpps":0.0000,"free_buffer_slots":0}

Comment 3 Vincenzo Maffione freebsd_committer

2019-08-25 07:22:12 UTC

This looks like related to this issue
https://github.com/luigirizzo/netmap/issues/600

Comment 4 Vincenzo Maffione freebsd_committer

2019-08-25 07:39:04 UTC

If 11.2-p9 works, it would help to see what changed in ixgbe and ixl between 11.2-p9 and 11.3. Are you able to generate a diff or provide pointers to the exact two source trees you are using?

Comment 5 mike 2019-08-26 16:48:09 UTC

This is the binary build of 11.2p9 (working) and 11.3p2 (non working) with netmap and ixgbe built in to kernel. Nic is 82599 (intel x520). Using Bro/Zeek 2.6.3 with bro-netmap plugin. Issue seems to really trigger once it starts pulling packets from the buffers which seems to trigger the out of bounds pointer. 

Nic output in dmesg:

ix2: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k> port 0x8020-0x803f mem 0xca080000-0xca0fffff,0xca104000-0xca107fff irq 64 at device 0.0 numa-domain 1 on pci21
ix2: Using MSI-X interrupts with 17 vectors
ix2: Ethernet address: ac:1f:6b:2d:f2:3c
ix2: PCI Express Bus: Speed 5.0GT/s Width x8
ix2: netmap queues/slots: TX 16/4096, RX 16/4096
ix2: permanently promiscuous mode enabled


Please let me know anything else I can include to help.

Comment 6 mike 2019-09-09 17:23:25 UTC


Some differences of note:
in IXGBE driver:
diff sys/dev/ixgbe/ixgbe_netmap.c 11.2/sys/dev/ixgbe/ixgbe_netmap.c
33c33
< /*$FreeBSD: releng/11.3/sys/dev/ixgbe/ixgbe_netmap.c 344047 2019-02-12 09:26:05Z vmaffione $*/
---
> /*$FreeBSD: releng/11.2/sys/dev/ixgbe/ixgbe_netmap.c 332481 2018-04-13 17:45:54Z shurd $*/
61c61
<  * $FreeBSD: releng/11.3/sys/dev/ixgbe/ixgbe_netmap.c 344047 2019-02-12 09:26:05Z vmaffione $
---
>  * $FreeBSD: releng/11.2/sys/dev/ixgbe/ixgbe_netmap.c 332481 2018-04-13 17:45:54Z shurd $
121c121
< 		nm_prinf("%s read  HLREG 0x%x rxc 0x%x",
---
> 		D("%s read  HLREG 0x%x rxc 0x%x",
136c136
< 		nm_prinf("%s write HLREG 0x%x rxc 0x%x",
---
> 		D("%s write HLREG 0x%x rxc 0x%x",
343,344c343,344
< 		if (unlikely(nic_i >= kring->nkr_num_slots)) {
< 			nm_prinf("TDH wrap %d", nic_i);
---
> 		if (nic_i >= kring->nkr_num_slots) { /* XXX can it happen ? */
> 			D("TDH wrap %d", nic_i);
411a412
> 		uint16_t slot_flags = kring->nkr_slot_flags;
423c424
< 			ring->slot[nm_i].flags = 0;
---
> 			ring->slot[nm_i].flags = slot_flags;