Bug 202875
| Summary: | ixv driver in 11.0-CURRENT doesn't pass traffic using KVM hypervisor | ||
|---|---|---|---|
| Product: | Base System | Reporter: | Jeff Pieper <jeffrey.e.pieper> |
| Component: | kern | Assignee: | Sean Bruno <sbruno> |
| Status: | Closed FIXED | ||
| Severity: | Affects Many People | CC: | erj, jlpetz, meyer.sydney, pkelsey, sbruno |
| Priority: | --- | Keywords: | IntelNetworking, needs-qa, patch |
| Version: | CURRENT | ||
| Hardware: | Any | ||
| OS: | Any | ||
| URL: | https://reviews.freebsd.org/D4186 | ||
|
Description
Jeff Pieper
2015-09-03 16:35:47 UTC
Reassign and add possibly interested party to Cc: . (In reply to Jeff Pieper from comment #0) What are the results when you only attach an 11.0-CURRENT guest in step 3 of your reproduction steps? (In reply to Jeff Pieper from comment #0) Some additional thoughts: 1. In the reported configuration, the Linux ixgbe driver in the host is running the PF, and the FreeBSD ixgbe driver(s)s in the guest(s) are running the respective VFs. The VF drivers communicate with the PF driver via the mailbox, so disagreements among the PF and VF driver implementations as to how the mailbox messaging works could in general be an issue. 2. Why do you feel r285590 is the likely culprit? Have you built a kernel prior to that revision and shown it to be working? r285590 does change the interpretation of a mailbox message and the generation of the response to that message in the FreeBSD PF driver, but the VF driver only uses the result for a printf() - that is, disagreement over that message between PF and VF won't bother the VF. You mention other changes being made "around that time", but since you are comparing the behavior of 10.2-RELEASE to 11.0-CURRENT, the set of changes to look at is all the differences between the code in 10.2-RELEASE and 11.0-CURRENT, and the code in 10.2-RELEASE is not the code that was in 11.0-CURRENT around the time 10.2-RELEASE was made, it's older. Looking at the diffs between the two, there are substantial ixv-related changes between 10.2-RELEASE and 11.0-CURRENT that pre-date r285590. 3. Have you checked whether any packets make it to the wire from the 11.0-CURRENT guest? That is, is this only an rx problem, or is it a tx and rx problem? (In reply to Patrick Kelsey from comment #2) Exactly the same. (In reply to Patrick Kelsey from comment #3) Our internal out-of-tree driver has all commits included in HEAD with the exceptions being r285590 and r285592, and It works as expected. During discussion with Eric Joyner, r285590 was determined to be the most likely cause. This appears to be an rx issue, as tcpdump on SUT shows only ARP requests being sent, but on a link partner tcpdump shows ARP requests being received and replies being sent. (In reply to Jeff Pieper from comment #5) > (In reply to Patrick Kelsey from comment #3) > > Our internal out-of-tree driver has all commits included in HEAD with the > exceptions being r285590 and r285592, and It works as expected. During > discussion with Eric Joyner, r285590 was determined to be the most likely > cause. If you are down to two revisions in question, that's only one patch-and-test step away from knowing with certainty which is the culprit... If r285590 is indeed the culprit, then the change I made to the receive tail pointer initialization needs to be looked at. On the hardware I was using at least, initializing the tail pointer before enabling the queue was not sufficient because enabling the queue would clear the tail pointer (the driver was working only by coincidence as it would wind up falling into the rx ring refresh logic and wind up rewriting the tail pointer). Moving the initialization to after the queue was enabled resolved this issue. Perhaps there are other constraints/requirements on tail pointer initialization - for example, maybe on some hardware, enabling the queue with both the head and tail pointers set to zero wedges the queue, or maybe another wmb() is required between the tail pointer initialization where it is now and the register writes that follow. The hardware I'm using: ix0@pci0:6:0:0: class=0x020000 card=0x000c8086 chip=0x10fb8086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = '82599EB 10-Gigabit SFI/SFP+ Network Connection' On the hardware you are seeing this issue on, are you able to test ixv under a FreeBSD 11.0-CURRENT host? By that, I mean run FreeBSD as the host on the system and create and test VFs within the host by using iovctl. This bug reminded me that I have the same issue. But instead of KVM on Xen+Qemu(AWS EC2). So as not to pollute this Bug I filled it separately as Bug 202983. But thought I would post an FYI here incase it is related or incase you think patches/changes here might help me and would like them tested. (In reply to Patrick Kelsey from comment #6) After performing some additional testing, this is what I have discovered: - Reverting r285590 and r285592 do not fix the issue. - I can reproduce the issue using our OOT driver on CURRENT. - in ALL failing scenarios, I see the following in dmesg on the host after init: ixgbe 0000:08:00.0 eth0: VF max_frame 1522 out of range So I think it's safe to say that it isn't the driver causing this, but something else that is causing oversized frames. In summary: - 10.2 works using in-kernel and OOT drivers. - CURRENT fails using in-kernel and OOT drivers. We believe to have this isolated and are testing a fix now. We should have an update within a day or two. This should be fixed in the series of patches currently under review in Phabricator: https://reviews.freebsd.org/D4186. Looks like the diff has been committed. So this Bugzilla can probably be resolved now? https://reviews.freebsd.org/D4186 https://reviews.freebsd.org/rS292674 I am resolving the separate Bug I raised for Xen. As the above diff along with this one below resolved my issue. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=202983 https://reviews.freebsd.org/D4788 https://reviews.freebsd.org/rS293338 It sounds like this can be closed, then. Assign to committer that resolved (in base r292674) |
