Bug 236067

Summary:	[em] Intel 82574L can not turn off flow control
Product:	Base System	Reporter:	Lev A. Serebryakov <lev>
Component:	kern	Assignee:	freebsd-net (Nobody) <net>
Status:	New ---
Severity:	Affects Only Me	CC:	adamkrisusa, erj, kbowling, shurd
Priority:	---	Keywords:	IntelNetworking
Version:	CURRENT
Hardware:	amd64
OS:	Any

Description Lev A. Serebryakov freebsd_committer

2019-02-26 22:21:23 UTC

I have two systems with 82574L NICs.

One, name it G (for "generator"), is powerful enough to saturate almost 1G links with small packets (more than 1100Kpps).

Other, name it DUT is underpowered and can not process such packet rate.

Both G and dut have "set dev.em.0.fc=0".

G try to generate traffic to DUT's MAC with netmap & pkt-gen. If G and DUT are connected (no matter how: with direct patchcord or via dedicated switch) G could generate only as much packets as DUT could process! If I remove DUT from dedicated switch, G generates almost-line-rate PPS. If I replace DUT with other system with I210 NIC, G generates same high PPS even if system with I210 can not process it.

Looks like 82574L can not turn off flow control.

Once again:

82574L -> Switch -> 82574L - pkt-gen generates PPS limited by receiver.
82574L -> 82574L - pkt-gen generates PPS limited by receiver.
82574L -> Switch -> "blackhole" (no other devices attached to switch) - pkt-gen generates line-rate PPS.
82574L -> I210 - pkt-gen generates line-rate PPS.
82574L -> Switch -> I210 - pkt-gen generates line-rate PPS.

Comment 1 Eric Joyner freebsd_committer

2019-02-27 22:12:51 UTC

Are you using the in-kernel driver in CURRENT on both sides? Do they both have 82574L?

Comment 2 Eric Joyner freebsd_committer

2019-02-27 22:15:00 UTC

(In reply to Eric Joyner from comment #1)

Sorry, I re-read your initial comment; it looks like this problem only occurs when both sides are 82574L. That's interesting.

Comment 3 Lev A. Serebryakov freebsd_committer

2019-02-28 11:18:12 UTC

(In reply to Eric Joyner from comment #2)
Both sides are r344277, drivers are compiled in kernel (not loaded as modules).

And I didn't tried "i210 -> 82574L" yet, I could try this combination around Monday.

Comment 4 Lev A. Serebryakov freebsd_committer

2019-03-02 12:42:07 UTC

(In reply to Eric Joyner from comment #2)
I've added i210 (in form of I210-T1 addon card) to my traffic generator system, and it is limited by receiver (82574L) too!

Both ends have FC turned off (with `dev.igb.0.fc=0` and `dev.em.0.fc=0` respective). I've tried with and without switch between. So, two more datapoints:

i210 -> Switch -> 82574L - pkt-gen generates PPS limited by receiver.
i210 -> 82574L - pkt-gen generates PPS limited by receiver.
i210 -> Switch -> "blackhole" (no other devices attached to switch) - pkt-gen generates line-rate PPS.

Comment 5 Lev A. Serebryakov freebsd_committer

2019-03-02 13:07:43 UTC

Inserting manageable switch (HP1800-8G) instead of simple one (HP1400-8G) with explicitly disabled flow control on all ports helps, BTW.
But still, i210->i210 doesn't require manageable switch to drop packets on the floor :-)

Comment 6 Kevin Bowling freebsd_committer

2021-09-30 01:53:30 UTC

(In reply to Lev A. Serebryakov from comment #5)
Are you able to test with more recent versions?  I am interested in this report although I don't have any 82574s that I know of.

Comment 7 Lev A. Serebryakov freebsd_committer

2021-09-30 17:13:42 UTC

(In reply to Kevin Bowling from comment #6)
Not now, this old equipment is turned off now and I'm not sure when I'll be able to setup it again...

Comment 8 Kevin Bowling freebsd_committer

2023-08-07 16:24:16 UTC

(In reply to Lev A. Serebryakov from comment #7)
I have some systems with 82574L now.  Over switched Ethernet I don't see any issues.  I will try a direct connect soon.

Would help to see sysctl dev.em.0.reg_dump of your machines.

Comment 9 Lev A. Serebryakov freebsd_committer

2023-08-17 09:11:30 UTC

(In reply to Kevin Bowling from comment #8)
I'm mboing to new country now and all my "lab" hardware is packed & stored somewhere between in shipping container. I'm not sure, when I will be able to unpack and install it back :-(

Comment 10 adamusa 2023-09-20 03:30:10 UTC

MARKED AS SPAM