Bug 187718 - [igb] UDP bad performance and out-of-order packet with iperf3 and igb(4) drivers
Summary: [igb] UDP bad performance and out-of-order packet with iperf3 and igb(4) drivers
Status: Closed Overcome By Events
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: unspecified
Hardware: Any Any
: Normal Affects Only Me
Assignee: freebsd-net mailing list
URL:
Keywords: IntelNetworking
Depends on:
Blocks:
 
Reported: 2014-03-19 00:50 UTC by Olivier Cochard-Labbe
Modified: 2018-05-31 06:59 UTC (History)
4 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Olivier Cochard-Labbe 2014-03-19 00:50:00 UTC
Testing UDP throughput with iperf and iperf3 give totally different value:

With iperf:

[root@R1]~# iperf -u -c 99.99.99.100 -w65536 -l1460 -t30 -i4 -b1000m -P1                                              
------------------------------------------------------------
Client connecting to 99.99.99.100, UDP port 5001
Sending 1460 byte datagrams
UDP buffer size: 64.0 KByte
------------------------------------------------------------
[  3] local 99.99.99.99 port 64273 connected with 99.99.99.100 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 4.0 sec   373 MBytes   783 Mbits/sec
[  3]  4.0- 8.0 sec   374 MBytes   784 Mbits/sec
[  3]  8.0-12.0 sec   372 MBytes   780 Mbits/sec
[  3] 12.0-16.0 sec   373 MBytes   783 Mbits/sec
[  3] 16.0-20.0 sec   373 MBytes   783 Mbits/sec
[  3] 20.0-24.0 sec   373 MBytes   782 Mbits/sec
[  3] 24.0-28.0 sec   373 MBytes   783 Mbits/sec
[  3]  0.0-30.0 sec  2.73 GBytes   783 Mbits/sec
[  3] Sent 2010212 datagrams
[  3] Server Report:
[  3]  0.0-30.0 sec  2.73 GBytes   781 Mbits/sec   0.010 ms 4953/2010211 (0.25%)
[  3]  0.0-30.0 sec  954425 datagrams received out-of-order


with iperf3.

[root@R1]~# iperf3 -u -c 99.99.99.100 -w65536 -l1460 -t30 -i4 -b1000m -P1                                             
Connecting to host 99.99.99.100, port 5201
[  4] local 99.99.99.99 port 43694 connected to 99.99.99.100 port 5201
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-4.00   sec  48.7 MBytes   102 Mbits/sec  1295380  
[  4]   4.00-8.00   sec  28.2 MBytes  59.2 Mbits/sec  1334970  
[  4]   8.00-12.00  sec  31.6 MBytes  66.4 Mbits/sec  1334995  
[  4]  12.00-16.00  sec  32.0 MBytes  67.1 Mbits/sec  1330605  
[  4]  16.00-20.00  sec  47.3 MBytes  99.2 Mbits/sec  1335097  
[  4]  20.00-24.00  sec  30.6 MBytes  64.1 Mbits/sec  1335121  
[  4]  24.00-28.00  sec  35.3 MBytes  74.0 Mbits/sec  1331542  
[  4]  28.00-30.00  sec  19.3 MBytes  81.0 Mbits/sec  663836  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-30.00  sec   273 MBytes  76.4 Mbits/sec  7.755 ms  9381619/9954141 (94%)  
[  4] Sent 9954141 datagrams

How-To-Repeat: Load on the generator during the iperf3 test is:
[root@R1]~# top -nCHSIzs1
last pid: 14453;  load averages:  0.37,  0.19,  0.09  up 4+17:17:33    01:26:07
160 processes: 6 running, 104 sleeping, 50 waiting

Mem: 4192K Active, 26M Inact, 751M Wired, 27M Buf, 15G Free
Swap: 


  PID USERNAME PRI NICE   SIZE    RES STATE   C   TIME     CPU COMMAND
14452 root     100    0 14300K  1880K CPU1    1   0:26  96.39% iperf3
   11 root     -92    -     0K   800K WAIT    1   0:07   0.88% intr{irq279: igb2:que}
   11 root     -92    -     0K   800K WAIT    2   0:13   0.78% intr{irq280: igb2:que}
   11 root     -92    -     0K   800K WAIT    3   0:05   0.68% intr{irq281: igb2:que}
   11 root     -92    -     0K   800K WAIT    0   0:11   0.49% intr{irq278: igb2:que}


and an extract of the iperf receiver during this test:

iperf3: OUT OF ORDER - incoming packet = 9972872 and received packet = 9983051 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9950852 and received packet = 9983052 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9947731 and received packet = 9983052 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9972873 and received packet = 9983052 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9950853 and received packet = 9983053 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9947732 and received packet = 9983053 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9972874 and received packet = 9983053 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9950854 and received packet = 9983054 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9947733 and received packet = 9983054 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9972875 and received packet = 9983054 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9950855 and received packet = 9983055 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9947734 and received packet = 9983055 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9972876 and received packet = 9983055 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9950856 and received packet = 9983056 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9947735 and received packet = 9983056 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9972877 and received packet = 9983056 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9950857 and received packet = 9983057 AND SP = 5
[  5]  30.00-30.22  sec  25.6 MBytes   957 Mbits/sec  55.317 ms  41962/46554 (90%)  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-30.22  sec   318 MBytes  88.3 Mbits/sec  55.317 ms  9425274/9983057 (94%)  
[SUM]  0.0-30.2 sec  9425274 datagrams received out-of-order
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Comment 1 Mark Linimon freebsd_committer freebsd_triage 2014-04-16 02:17:25 UTC
Responsible Changed
From-To: freebsd-bugs->freebsd-net

Over to maintainer(s).
Comment 2 Eitan Adler freebsd_committer freebsd_triage 2018-05-28 19:43:26 UTC
batch change:

For bugs that match the following
-  Status Is In progress 
AND
- Untouched since 2018-01-01.
AND
- Affects Base System OR Documentation

DO:

Reset to open status.


Note:
I did a quick pass but if you are getting this email it might be worthwhile to double check to see if this bug ought to be closed.
Comment 3 Sean Bruno freebsd_committer 2018-05-29 15:29:00 UTC
Oliver:

Is this still true on -current?
Comment 4 Rodney W. Grimes freebsd_committer 2018-05-29 16:04:12 UTC
I suspect that the iperf3 data here is pretty meaningless as the single thread has saturated a core (data shows 96% cpu usage) and high packet losses, probably caused by that saturation.

To get meaningful data from iperf3 you must insure you do not saturate the single thread core it is running on.  I also suggest locking it to a core with affinity.
Comment 5 Olivier Cochard freebsd_committer 2018-05-31 06:59:20 UTC
I didn't meet this problem on -stable anymore:

[root@r630]~# uname -a
FreeBSD r630 11.2-BETA3 FreeBSD 11.2-BETA3  r334330M  amd64

[root@r630]~# iperf -u -c 127.0.0.1 -w65536 -l1460 -t30 -i4 -b1000m -P1
------------------------------------------------------------
Client connecting to 127.0.0.1, UDP port 5001
Sending 1460 byte datagrams, IPG target: 11.68 us (kalman adjust)
UDP buffer size: 64.0 KByte
------------------------------------------------------------
[  3] local 127.0.0.1 port 16041 connected with 127.0.0.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 4.0 sec   477 MBytes  1.00 Gbits/sec
[  3]  4.0- 8.0 sec   477 MBytes  1.00 Gbits/sec
[  3]  8.0-12.0 sec   477 MBytes  1.00 Gbits/sec
[  3] 12.0-16.0 sec   477 MBytes  1000 Mbits/sec
[  3] 16.0-20.0 sec   477 MBytes  1.00 Gbits/sec
[  3] 20.0-24.0 sec   477 MBytes  1.00 Gbits/sec
[  3] 24.0-28.0 sec   477 MBytes  1.00 Gbits/sec
[  3]  0.0-30.0 sec  3.49 GBytes  1000 Mbits/sec
[  3] Sent 2568514 datagrams
[  3] Server Report:
[  3]  0.0-30.0 sec  3.49 GBytes  1.00 Gbits/sec   0.000 ms    5/2568514 (0.00019%)
[root@r630]~# iperf3 -u -c 127.0.0.1 -w65536 -l1460 -t30 -i4 -b1000m -P1
Connecting to host 127.0.0.1, port 5201
[  5] local 127.0.0.1 port 53178 connected to 127.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-4.00   sec   477 MBytes  1000 Mbits/sec  342407
[  5]   4.00-8.00   sec   477 MBytes  1.00 Gbits/sec  342466
[  5]   8.00-12.00  sec   477 MBytes  1.00 Gbits/sec  342466
[  5]  12.00-16.00  sec   477 MBytes  1.00 Gbits/sec  342466
[  5]  16.00-20.00  sec   477 MBytes  1000 Mbits/sec  342464
[  5]  20.00-24.00  sec   477 MBytes  1.00 Gbits/sec  342466
[  5]  24.00-28.00  sec   477 MBytes  1.00 Gbits/sec  342466
[  5]  28.00-30.00  sec   238 MBytes  1.00 Gbits/sec  171233
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-30.00  sec  3.49 GBytes  1000 Mbits/sec  0.000 ms  0/2568434 (0%)  sender
[  5]   0.00-30.10  sec  3.49 GBytes   997 Mbits/sec  0.001 ms  14/2568434 (0.00055%)  receiver

iperf Done.