Bug 202875 - ixv driver in 11.0-CURRENT doesn't pass traffic using KVM hypervisor
Summary: ixv driver in 11.0-CURRENT doesn't pass traffic using KVM hypervisor
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Many People
Assignee: Sean Bruno
URL: https://reviews.freebsd.org/D4186
Keywords: IntelNetworking, needs-qa, patch
Depends on:
Blocks:
 
Reported: 2015-09-03 16:35 UTC by Jeff Pieper
Modified: 2017-08-27 07:39 UTC (History)
5 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jeff Pieper 2015-09-03 16:35:47 UTC
ixv-1.4.0 (11.0-CURRENT snapshot 20150804) will not pass traffic using a KVM hypervisor on a Linux host. Tested using RHEL 7.1 x86_64 with ixgbe-4.1.2 as the host OS/PF driver. ixv-1.2.5 (10.2-RELEASE amd64) works as expected.

There were a few other commits at the same time, but this is the likely cause.

Reproduce steps
1. Load PF driver on the host and bring interface up.
2. Create 2 VF's on a single port using sysfs.
3. Attach one VF to a FreeBSD 11.0 guest and one VF to a FreeBSD 10.2 guest.
4. Boot both guests and load ixv (if not static)on both guests.
5. Assign an IP address to ixv0 and bring the interface up on both guests.
6. Attempt to pass traffic to a link partner. Both broadcast and unicast ICMP  traffic fails on the 11.0 guest, while the 10.2 guest passes traffic.

ixv-1.4.0:

ixv0: <Intel(R) PRO/10GbE Virtual Function Network Driver, Version - 1.4.0> mem 0xfc054000-0xfc057fff,0xfc058000-0xfc05bfff at device 9.0 on pci0
ixv0: Using MSIX interrupts with 2 vectors
ixv0: MBX API 2 negotiation: 0
ixv0: Ethernet address: 72:05:dc:1c:14:2a
ixv0: netmap queues/slots: TX 1/1024, RX 1/1024
ixv0: link state changed to UP

ixv0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=405bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWTSO>
	ether 72:05:dc:1c:14:2a
	inet 190.0.7.20 netmask 0xffff0000 broadcast 190.0.255.255 
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active

PING 190.0.255.255 (190.0.255.255): 56 data bytes

--- 190.0.255.255 ping statistics ---
26 packets transmitted, 0 packets received, 100.0% packet loss

ixv-1.2.5:

ixv0: <Intel(R) PRO/10GbE Virtual Function Network Driver, Version - 1.2.5> mem 0xfc054000-0xfc057fff,0xfc058000-0xfc05bfff at device 9.0 on pci0
ixv0: Using MSIX interrupts with 2 vectors
ixv0: Ethernet address: 26:a1:8c:3e:3c:fe
ixv0: link state changed to UP

ixv0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=405bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWTSO>
	ether 26:a1:8c:3e:3c:fe
	inet 190.0.7.21 netmask 0xffff0000 broadcast 190.0.255.255 
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active

PING 190.0.255.255 (190.0.255.255): 56 data bytes
64 bytes from 190.0.8.13: icmp_seq=0 ttl=64 time=0.382 ms
64 bytes from 190.0.8.10: icmp_seq=0 ttl=64 time=0.397 ms (DUP!)
64 bytes from 190.0.8.7: icmp_seq=0 ttl=64 time=0.410 ms (DUP!)
64 bytes from 190.0.8.14: icmp_seq=0 ttl=64 time=0.423 ms (DUP!)
64 bytes from 190.0.8.9: icmp_seq=0 ttl=64 time=0.439 ms (DUP!)

<Snip>

--- 190.0.255.255 ping statistics ---
14 packets transmitted, 14 packets received, +196 duplicates, 0.0% packet loss
round-trip min/avg/max/stddev = 0.200/0.750/3.637/0.906 ms

I believe this commit to be the culprit:

https://svnweb.freebsd.org/base/head/sys/dev/ixgbe/ixgbe_vf.c?revision=285590&view=markup

There were a few other commits at the same time, but this is the likely cause.
Comment 1 Mark Linimon freebsd_committer freebsd_triage 2015-09-05 02:02:25 UTC
Reassign and add possibly interested party to Cc: .
Comment 2 Patrick Kelsey freebsd_committer 2015-09-05 16:48:02 UTC
(In reply to Jeff Pieper from comment #0)

What are the results when you only attach an 11.0-CURRENT guest in step 3 of your reproduction steps?
Comment 3 Patrick Kelsey freebsd_committer 2015-09-05 17:45:36 UTC
(In reply to Jeff Pieper from comment #0)

Some additional thoughts:

1. In the reported configuration, the Linux ixgbe driver in the host is running the PF, and the FreeBSD ixgbe driver(s)s in the guest(s) are running the respective VFs.  The VF drivers communicate with the PF driver via the mailbox, so disagreements among the PF and VF driver implementations as to how the mailbox messaging works could in general be an issue.

2. Why do you feel r285590 is the likely culprit?  Have you built a kernel prior to that revision and shown it to be working?  r285590 does change the interpretation of a mailbox message and the generation of the response to that message in the FreeBSD PF driver, but the VF driver only uses the result for a printf() - that is, disagreement over that message between PF and VF won't bother the VF. You mention other changes being made "around that time", but since you are comparing the behavior of 10.2-RELEASE to 11.0-CURRENT, the set of changes to look at is all the differences between the code in 10.2-RELEASE and 11.0-CURRENT, and the code in 10.2-RELEASE is not the code that was in 11.0-CURRENT around the time 10.2-RELEASE was made, it's older.  Looking at the diffs between the two, there are substantial ixv-related changes between 10.2-RELEASE and 11.0-CURRENT that pre-date r285590.

3. Have you checked whether any packets make it to the wire from the 11.0-CURRENT guest?  That is, is this only an rx problem, or is it a tx and rx problem?
Comment 4 Jeff Pieper 2015-09-06 15:07:51 UTC
(In reply to Patrick Kelsey from comment #2)

Exactly the same.
Comment 5 Jeff Pieper 2015-09-06 17:47:10 UTC
(In reply to Patrick Kelsey from comment #3)

Our internal out-of-tree driver has all commits included in HEAD with the exceptions being r285590 and r285592, and It works as expected. During discussion with Eric Joyner, r285590 was determined to be the most likely cause.

This appears to be an rx issue, as tcpdump on SUT shows only ARP requests being sent,  but on a link partner tcpdump shows ARP requests being received and replies being sent.
Comment 6 Patrick Kelsey freebsd_committer 2015-09-06 19:22:18 UTC
(In reply to Jeff Pieper from comment #5)
> (In reply to Patrick Kelsey from comment #3)
> 
> Our internal out-of-tree driver has all commits included in HEAD with the
> exceptions being r285590 and r285592, and It works as expected. During
> discussion with Eric Joyner, r285590 was determined to be the most likely
> cause.

If you are down to two revisions in question, that's only one patch-and-test step away from knowing with certainty which is the culprit...

If r285590 is indeed the culprit, then the change I made to the receive tail pointer initialization needs to be looked at.  On the hardware I was using at least, initializing the tail pointer before enabling the queue was not sufficient because enabling the queue would clear the tail pointer (the driver was working only by coincidence as it would wind up falling into the rx ring refresh logic and wind up rewriting the tail pointer).  Moving the initialization to after the queue was enabled resolved this issue.  Perhaps there are other constraints/requirements on tail pointer initialization - for example, maybe on some hardware, enabling the queue with both the head and tail pointers set to zero wedges the queue, or maybe another wmb() is required between the tail pointer initialization where it is now and the register writes that follow.

The hardware I'm using:

ix0@pci0:6:0:0: class=0x020000 card=0x000c8086 chip=0x10fb8086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82599EB 10-Gigabit SFI/SFP+ Network Connection'

On the hardware you are seeing this issue on, are you able to test ixv under a FreeBSD 11.0-CURRENT host? By that, I mean run FreeBSD as the host on the system and create and test VFs within the host by using iovctl.
Comment 7 Jarrod Petz 2015-09-09 06:09:47 UTC
This bug reminded me that I have the same issue. But instead of KVM on Xen+Qemu(AWS EC2).

So as not to pollute this Bug I filled it separately as Bug 202983. But thought I would post an FYI here incase it is related or incase you think patches/changes here might help me and would like them tested.
Comment 8 Jeff Pieper 2015-09-09 16:39:15 UTC
(In reply to Patrick Kelsey from comment #6)

After performing some additional testing, this is what I have discovered:

- Reverting r285590 and r285592 do not fix the issue.
- I can reproduce the issue using our OOT driver on CURRENT.
- in ALL failing scenarios, I see the following in dmesg on the host after init:

ixgbe 0000:08:00.0 eth0: VF max_frame 1522 out of range

So I think it's safe to say that it isn't the driver causing this, but something else that is causing oversized frames.

In summary:

- 10.2 works using in-kernel and OOT drivers.
- CURRENT fails using in-kernel and OOT drivers.
Comment 9 Jeff Pieper 2015-10-06 14:29:30 UTC
We believe to have this isolated and are testing a fix now. We should have an update within a day or two.
Comment 10 Jeff Pieper 2015-11-19 16:45:12 UTC
This should be fixed in the series of patches currently under review in Phabricator: https://reviews.freebsd.org/D4186.
Comment 11 Jarrod Petz 2016-01-07 22:45:18 UTC
Looks like the diff has been committed. So this Bugzilla can probably be resolved now?
https://reviews.freebsd.org/D4186
https://reviews.freebsd.org/rS292674

I am resolving the separate Bug I raised for Xen. As the above diff along with this one below resolved my issue.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=202983
https://reviews.freebsd.org/D4788
https://reviews.freebsd.org/rS293338
Comment 12 Eric Joyner freebsd_committer 2017-08-24 23:52:53 UTC
It sounds like this can be closed, then.
Comment 13 Kubilay Kocak freebsd_committer freebsd_triage 2017-08-27 07:39:45 UTC
Assign to committer that resolved (in base r292674)