Bug 230465 - ixl: not working in netmap mode
Summary: ixl: not working in netmap mode
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 11.2-STABLE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-bugs mailing list
URL:
Keywords: IntelNetworking
Depends on:
Blocks:
 
Reported: 2018-08-08 17:57 UTC by Charles Goncalves
Modified: 2019-01-11 12:27 UTC (History)
6 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Charles Goncalves 2018-08-08 17:57:27 UTC
Hello!

I have a ixl NIC (chip=0x15728086 'Ethernet Controller X710 for 10GbE SFP+) and I'm trying to work with netmap.

When I was compiling kernel with ixl support I have errors with missing reference of: "'ixl_rx_miss', 'ixl_rx_miss_bufs and 'ixl_crcstrip'" so then I modify ixl_txrx.c and added this references like this:

#ifdef DEV_NETMAP                                         
#include <dev/netmap/if_ixl_netmap.h>                     
int ixl_rx_miss = 0, ixl_rx_miss_bufs = 0, ixl_crcstrip = 1;
#endif /* DEV_NETMAP */  

When I did this my kernel was compiled with sucess and now I see ixl interfaces in "ifconfig" command.

Then now I'm trying netmap on then, but seems not working. In my application on top of netmap I see "dmesg" like this:

Aug  8 14:46:56 rt1 kernel: 415.918289 [1637] nm_txsync_prologue        ixl0 TX3: fail 'kring->nr_hwcur >= n || kring->rhead >= n || kring->rtail >= n || kring->nr_hwtail >= n' h 2 c 2 t 1387166207 rh 1 rc 1 rt 1387166207 hc 1 ht 1387166207
Aug  8 14:46:56 rt1 kernel: 415.973692 [1758] netmap_ring_reinit        called for ixl0 TX3
Aug  8 14:46:56 rt1 kernel: 415.990602 [1783] netmap_ring_reinit        total 1 errors
Aug  8 14:46:56 rt1 kernel: 416.006198 [1787] netmap_ring_reinit        ixl0 TX3 reinit, cur 2 -> 1 tail 1387166207 -> 1387166207
Aug  8 14:46:56 rt1 kernel: 416.032990 [1637] nm_txsync_prologue        ixl0 TX3: fail 'kring->nr_hwcur >= n || kring->rhead >= n || kring->rtail >= n || kring->nr_hwtail >= n' h 2 c 2 t 1387166207 rh 1 rc 1 rt 1387166207 hc 1 ht 1387166207
Aug  8 14:46:56 rt1 kernel: 416.088614 [1758] netmap_ring_reinit        called for ixl0 TX3
Aug  8 14:46:56 rt1 kernel: 416.105520 [1783] netmap_ring_reinit        total 1 errors
Aug  8 14:46:56 rt1 kernel: 416.121113 [1787] netmap_ring_reinit        ixl0 TX3 reinit, cur 2 -> 1 tail 1387166207 -> 1387166207
Aug  8 14:46:57 rt1 kernel: 417.089185 [1637] nm_txsync_prologue        ixl0 TX3: fail 'kring->nr_hwcur >= n || kring->rhead >= n || kring->rtail >= n || kring->nr_hwtail >= n' h 2 c 2 t 1387166207 rh 1 rc 1 rt 1387166207 hc 1 ht 1387166207
Aug  8 14:46:57 rt1 kernel: 417.144605 [1758] netmap_ring_reinit        called for ixl0 TX3
Aug  8 14:46:57 rt1 kernel: 417.161510 [1783] netmap_ring_reinit        total 1 errors
Aug  8 14:46:57 rt1 kernel: 417.177110 [1787] netmap_ring_reinit        ixl0 TX3 reinit, cur 2 -> 1 tail 1387166207 -> 1387166207
Aug  8 14:46:58 rt1 kernel: 418.138193 [1637] nm_txsync_prologue        ixl0 TX3: fail 'kring->nr_hwcur >= n || kring->rhead >= n || kring->rtail >= n || kring->nr_hwtail >= n' h 2 c 2 t 1387166207 rh 1 rc 1 rt 1387166207 hc 1 ht 1387166207
Aug  8 14:46:58 rt1 kernel: 418.193599 [1758] netmap_ring_reinit        called for ixl0 TX3
Aug  8 14:46:58 rt1 kernel: 418.210507 [1783] netmap_ring_reinit        total 1 errors
Aug  8 14:46:58 rt1 kernel: 418.226096 [1787] netmap_ring_reinit        ixl0 TX3 reinit, cur 2 -> 1 tail 1387166207 -> 1387166207



Using pkt-get from netmap github I'm able to receive packets but not able to transmit then like this:

comand: pkt-gen -i ixl0 -f tx

637.872347 main [2593] interface is ixl0
637.872394 main [2727] running on 1 cpus (have 8)
637.872601 extract_ip_range [468] range is 10.0.0.1:1234 to 10.0.0.1:1234
637.872618 extract_ip_range [468] range is 10.1.0.1:1234 to 10.1.0.1:1234
638.046374 main [2822] mapped 294020KB at 0x801600000
Sending on netmap:ixl0: 8 queues, 1 threads and 1 cpus.
10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> ff:ff:ff:ff:ff:ff)
638.046466 main [2919] Sending 512 packets every  0.000000000 s
638.046507 start_threads [2274] Wait 2 secs for phy reset
640.145075 start_threads [2276] Ready...
640.145254 sender_body [1464] start, fd 3 main_fd 3
640.863306 sender_body [1538] poll error on 3 ring 0-7
641.198102 main_thread [2364] 7.780 Kpps (8.191 Kpkts 3.932 Mbps in 1052845 usec) 511.94 avg_batch 0 min_space
641.372908 main_thread [2391] ouch, thread 0 exited with error
Sent 8191 packets 491460 bytes 16 events 60 bytes each in -1533750640.15 seconds.
Speed: -0.000 pps Bandwidth: -0.003 bps (raw -0.004 bps). Average batch: 511.94 pkts



Then "dmesg" show:
Aug  8 14:51:48 rt1 kernel: 708.870527 [1637] nm_txsync_prologue        ixl0 TX1: fail 'kring->nr_hwcur >= n || kring->rhead >= n || kring->rtail >= n || kring->nr_hwtail >= n' h 0 c 0 t -278400513 rh 512 rc 512 rt -278400513 hc 512 ht -278400513
Aug  8 14:51:48 rt1 kernel: 708.927494 [1758] netmap_ring_reinit        called for ixl0 TX1
Aug  8 14:51:49 rt1 kernel: 708.944399 [1783] netmap_ring_reinit        total 1 errors
Aug  8 14:51:49 rt1 kernel: 708.959993 [1787] netmap_ring_reinit        ixl0 TX1 reinit, cur 0 -> 512 tail -278400513 -> -278400513
Aug  8 14:51:49 rt1 kernel: 708.987295 [1637] nm_txsync_prologue        ixl0 TX2: fail 'kring->nr_hwcur >= n || kring->rhead >= n || kring->rtail >= n || kring->nr_hwtail >= n' h 0 c 0 t -440139265 rh 512 rc 512 rt -440139265 hc 512 ht -440139265
Aug  8 14:51:49 rt1 kernel: 709.044489 [1758] netmap_ring_reinit        called for ixl0 TX2
Aug  8 14:51:49 rt1 kernel: 709.061399 [1783] netmap_ring_reinit        total 1 errors
Aug  8 14:51:49 rt1 kernel: 709.076993 [1787] netmap_ring_reinit        ixl0 TX2 reinit, cur 0 -> 512 tail -440139265 -> -440139265
Aug  8 14:51:49 rt1 kernel: 709.104291 [1637] nm_txsync_prologue        ixl0 TX3: fail 'kring->nr_hwcur >= n || kring->rhead >= n || kring->rtail >= n || 
Aug  8 14:51:49 rt1 kernel: kring->nr_hwtail >= n' h 0 c 0 t 1455426047 rh 512 rc 512 rt 1455426047 hc 512 ht 1455426047
Aug  8 14:51:49 rt1 kernel: 709.161491 [1758] netmap_ring_reinit        called for ixl0 TX3
Aug  8 14:51:49 rt1 kernel: 709.178394 [1783] netmap_ring_reinit        total 1 errors
Aug  8 14:51:49 rt1 kernel: 709.193987 [1787] netmap_ring_reinit        ixl0 TX3 reinit, cur 0 -> 512 tail 1455426047 -> 1455426047
Aug  8 14:51:49 rt1 kernel: 709.221304 [1637] nm_txsync_prologue        ixl0 TX4: fail 'kring->nr_hwcur >= n || kring->rhead >= n || kring->rtail >= n || kring->nr_hwtail >= n' h 0 c 0 t 1457261055 rh 512 rc 512 rt 1457261055 hc 512 ht 1457261055
Aug  8 14:51:49 rt1 kernel: 709.278488 [1758] netmap_ring_reinit        called for ixl0 TX4
Aug  8 14:51:49 rt1 kernel: 709.295391 [1783] netmap_ring_reinit        total 1 errors
Aug  8 14:51:49 rt1 kernel: 709.310998 [1787] netmap_ring_reinit        ixl0 TX4 reinit, cur 0 -> 512 tail 1457261055 -> 1457261055
Aug  8 14:51:49 rt1 kernel: 709.338286 [1637] nm_txsync_prologue        ixl0 TX5: fail 'kring->nr_hwcur >= n || kring->rhead >= n || kring->rtail >= n || kring->nr_hwtail >= n' h 0 c 0 t 1744312831 rh 512 rc 512 rt 1744312831 hc 512 ht 1744312831
Aug  8 14:51:49 rt1 kernel: 709.395485 [1758] netmap_ring_reinit        called for ixl0 TX5
Aug  8 14:51:49 rt1 kernel: 709.412388 [1783] netmap_ring_reinit        total 1 errors
Aug  8 14:51:49 rt1 kernel: 709.427981 [1787] netmap_ring_reinit        ixl0 TX5 reinit, cur 0 -> 512 tail 1744312831 -> 1744312831
Aug  8 14:51:49 rt1 kernel: 709.455284 [1637] nm_txsync_prologue        ixl0 TX6: fail 'kring->nr_hwcur >= n || kring->rhead >= n || kring->rtail >= n || kring->nr_hwtail >= n' h 0 c 0 t -1586220545 rh 512 rc 512 rt -1586220545 hc 512 ht -1586220545
Aug  8 14:51:49 rt1 kernel: 709.513263 [1758] netmap_ring_reinit        called for ixl0 TX6
Aug  8 14:51:49 rt1 kernel: 709.530166 [1783] netmap_ring_reinit        total 1 errors
Aug  8 14:51:49 rt1 kernel: 709.545759 [1787] netmap_ring_reinit        ixl0 TX6 reinit, cur 0 -> 512 tail -1586220545 -> -1586220545
Aug  8 14:51:49 rt1 kernel: 709.573581 [1637] nm_txsync_prologue        ixl0 TX7: fail 'kring->nr_hwcur >= n || kring->rhead >= n || kring->rtail >= n || kring->nr_hwtail >= n' h 0 c 0 t -1429192193 rh 512 rc 512 rt -1429192193 hc 512 ht -1429192193
Aug  8 14:51:49 rt1 kernel: 709.631560 [1758] netmap_ring_reinit        called for ixl0 TX7
Aug  8 14:51:49 rt1 kernel: 709.648463 [1783] netmap_ring_reinit        total 1 errors
Aug  8 14:51:49 rt1 kernel: 709.664056 [1787] netmap_ring_reinit        ixl0 TX7 reinit, cur 0 -> 512 tail -1429192193 -> -1429192193



Am I missing something?

Thanks!
Comment 1 Michael 2018-08-08 19:30:22 UTC
X710 has a couple of problem with netmap. 
Also experienced it with all available firmware versions for the NIC.

I just downgraded to X520 or Chelsio.

For me the NIC just freezes, carrier active and seeing incoming packets via tcpdump but nothing else ...
Comment 2 Charles Goncalves 2018-08-14 18:07:47 UTC
If I run pkt-gen with rate option then pkt-gen sends packets.

Like this: pkg-gen -f tx -R 150000

If I change value greater than 150k i.e: 160000 I got same error:

/pkt-gen -i ixl0 -f tx -R 160000 
812.653021 main [2593] interface is ixl0
812.653078 main [2727] running on 1 cpus (have 8)
812.653279 extract_ip_range [468] range is 10.0.0.1:1234 to 10.0.0.1:1234
812.653296 extract_ip_range [468] range is 10.1.0.1:1234 to 10.1.0.1:1234
812.826944 main [2822] mapped 294020KB at 0x801600000
Sending on netmap:ixl0: 8 queues, 1 threads and 1 cpus.
10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> ff:ff:ff:ff:ff:ff)
812.827043 main [2919] Sending 512 packets every  0.003200000 s
812.827085 start_threads [2274] Wait 2 secs for phy reset
814.836422 start_threads [2276] Ready...
814.836644 sender_body [1464] start, fd 3 main_fd 3
815.838007 main_thread [2364] 0.000 pps (0.000 pkts 0.000 bps in 1001364 usec) 0.00 avg_batch 0 min_space
816.134343 sender_body [1538] poll error on 3 ring 0-7
816.886366 main_thread [2364] 1.954 Kpps (2.048 Kpkts 983.040 Kbps in 1048359 usec) 341.33 avg_batch 99999 min_space
817.061069 main_thread [2391] ouch, thread 0 exited with error
Sent 2048 packets 122880 bytes 6 events 60 bytes each in -1534269816.00 seconds.
Speed: -0.000 pps Bandwidth: -0.001 bps (raw -0.001 bps). Average batch: 341.33 pkts


And kernel log:
Aug 14 15:03:36 rt1 kernel: 816.016016 [1637] nm_txsync_prologue        ixl0 TX1: fail 'kring->nr_hwcur >= n || kring->rhead >= n || kring->rtail >= n || kring->nr_hwtail >= n' h 275 c 275 t -278400513 rh 274 rc 274 rt -278400513 hc 274 ht -278400513
Aug 14 15:03:36 rt1 kernel: 816.074020 [1758] netmap_ring_reinit        called for ixl0 TX1
Aug 14 15:03:36 rt1 kernel: 816.090925 [1783] netmap_ring_reinit        total 1 errors
Aug 14 15:03:36 rt1 kernel: 816.106519 [1787] netmap_ring_reinit        ixl0 TX1 reinit, cur 275 -> 274 tail -278400513 -> -278400513



These errors seem to be related to synchronization of HEAD/CUR/TAIL in netmap ring
Comment 3 Charles Goncalves 2018-08-22 21:43:06 UTC
I've noted that in POLLIN this: 
poll(pfd, rxrings, -1);
Doesn't work, need it to be something like: 
poll(pfd, rxrings, 1); 
But with this I have a latency problem.

And I also have noted that I only can TX from TX-RING-0 if I use TX-RING more than 0, doesn't work. In my test scenario my ixl NIC has 8 rings.

Can anyone understand the reason?

Thanks!
Comment 4 Vincenzo Maffione freebsd_committer 2019-01-10 22:29:36 UTC
The compilation problems have been fixed.
Which FreeBSD version are you using? We need to understand if your ixl driver is backed by iflib or not.
Comment 5 Charles Goncalves 2019-01-11 12:27:41 UTC
(In reply to Vincenzo Maffione from comment #4)
Hello Vincenzo, thank you for your answer!

I was running tests in FreeBSD 11.2 STABLE.

For coming months I'll upgrade to 12.0 STABLE.