Bug 230996 - em/igb: Intel i210/i350: ifconfig: enabling "vlanhwtag" renders VLAN on i210/i350 NICs unusable
Summary: em/igb: Intel i210/i350: ifconfig: enabling "vlanhwtag" renders VLAN on i210/...
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.0-RELEASE
Hardware: Any Any
: Normal Affects Many People
Assignee: freebsd-net (Nobody)
URL:
Keywords: IntelNetworking, needs-qa, performance, regression
Depends on:
Blocks:
 
Reported: 2018-08-29 14:56 UTC by O. Hartmann
Modified: 2020-05-22 09:40 UTC (History)
12 users (show)

See Also:
koobs: maintainer-feedback? (freebsd)
koobs: mfc-stable12?


Attachments
dmesg of -CURRENT host with same issue. (10.04 KB, text/plain)
2020-01-15 05:50 UTC, Jason Tubnor
no flags Details
pciconf -lv of host running -CURRENT (5.40 KB, text/plain)
2020-01-15 21:45 UTC, Jason Tubnor
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description O. Hartmann 2018-08-29 14:56:42 UTC
Running a small howm-brewn router with only vanilla and standard FreeBSD on-board utilities on a PCEngines APU2C4, the built-in i210 Intel NICs stop working when they have VLAN tags assigned and enabling "vlanhwtag" on the base device (i.e., igb1 is the main device, igb1.10, igb1.100 and igb1.1000 are the VLAN devices).

This phenomenon/bug seems also present on Intel i350 NIcs (I use a server with a dual port PCIe NIC of that kind).
Comment 1 Lev A. Serebryakov freebsd_committer 2018-09-29 15:30:42 UTC
I have very similar problem: igb/I210, FreeBSD 12-ALPHA8 (r339009).
When I enable "vlanhwtag" on server, clients on this VLAN receive UDP with broken checksums. 
For example, client can not obtain address via DHCP from server with enabled "vlanhwtag", because DHCP client doesn't seen answers, because they are dropped by kernel due to invalid checksum: tcpdump sees DHCPOFFER on cleint's interface but dhclient doesn't receive anything.
Comment 2 Sean Bruno freebsd_committer 2018-09-29 18:03:47 UTC
I'm unclear if this is related to PR231416 or not.  There needs to be a bit more clarification in what the vlanhwtag needs to do or if setting this flag somehow breaks what the udp stack is expecting.
Comment 3 Lev A. Serebryakov freebsd_committer 2018-09-29 20:49:50 UTC
(In reply to Sean Bruno from comment #2)
My case is PR231416 for sure. Looks like TCP works with tis capability enabled, for example. I'm not sure about non-bpf originated UDP, though.
Comment 4 Jason Tubnor 2020-01-09 03:20:32 UTC
This issue still exists in 12.1-p1

While vlanhwtagging doesn't need to be disabled if using the host in a vanilla config, once and igb interface is placed into a bridge as a vlan with a tap device  for a bhyve guest, ingress of data to the guest comes to a halt under iperf3 testing. egress from the guest is not impacted and runs at full speed.

When vlanhwtagging is disabled, forcing the host to perform the function, data ingress or egress of the guest runs at full speed.

I'd like to enable vlanhwtagging as the CPU load put on the host is huge when transferring a single 1Gb/s stream of traffic from guests.
Comment 5 Jason Tubnor 2020-01-13 01:47:05 UTC
With vlanhwtag enabled:

$ iperf3 -c 10.1.1.1    
Connecting to host 10.1.1.1, port 5201
[  5] local 10.1.1.91 port 44086 connected to 10.1.1.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec  35.4 KBytes   287 Kbits/sec                  
[  5]   1.01-2.01   sec  2.83 KBytes  23.2 Kbits/sec                  
[  5]   2.01-3.02   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   3.02-4.02   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   4.02-5.01   sec  2.83 KBytes  23.4 Kbits/sec                  
[  5]   5.01-6.01   sec  5.66 KBytes  46.3 Kbits/sec                  
[  5]   6.01-7.01   sec  5.66 KBytes  46.3 Kbits/sec                  
[  5]   7.01-8.01   sec  4.24 KBytes  34.8 Kbits/sec                  
[  5]   8.01-9.01   sec  5.66 KBytes  46.3 Kbits/sec                  
[  5]   9.01-10.01  sec  5.66 KBytes  46.3 Kbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  67.9 KBytes  55.6 Kbits/sec                  sender
[  5]   0.00-10.20  sec  48.1 KBytes  38.6 Kbits/sec                  receiver

iperf Done.

$ iperf3 -c 10.1.1.1 -R
Connecting to host 10.1.1.1, port 5201
Reverse mode, remote host 10.1.1.1 is sending
[  5] local 10.1.1.91 port 11269 connected to 10.1.1.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  64.5 MBytes   541 Mbits/sec                  
[  5]   1.00-2.00   sec   105 MBytes   883 Mbits/sec                  
[  5]   2.00-3.00   sec   111 MBytes   932 Mbits/sec                  
[  5]   3.00-4.00   sec   111 MBytes   929 Mbits/sec                  
[  5]   4.00-5.00   sec   112 MBytes   938 Mbits/sec                  
[  5]   5.00-6.00   sec   112 MBytes   938 Mbits/sec                  
[  5]   6.00-7.00   sec   111 MBytes   934 Mbits/sec                  
[  5]   7.00-8.00   sec   112 MBytes   937 Mbits/sec                  
[  5]   8.00-9.00   sec   112 MBytes   936 Mbits/sec                  
[  5]   9.00-10.00  sec   111 MBytes   935 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.19  sec  1.04 GBytes   874 Mbits/sec                  sender
[  5]   0.00-10.00  sec  1.04 GBytes   890 Mbits/sec                  receiver

iperf Done.


With hwvlantag disabled (-hwvlantag):

$ iperf3 -c 10.1.1.1    
Connecting to host 10.1.1.1, port 5201
[  5] local 10.1.1.91 port 46471 connected to 10.1.1.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   105 MBytes   877 Mbits/sec                  
[  5]   1.00-2.00   sec   112 MBytes   936 Mbits/sec                  
[  5]   2.00-3.00   sec   112 MBytes   936 Mbits/sec                  
[  5]   3.00-4.00   sec   111 MBytes   933 Mbits/sec                  
[  5]   4.00-5.01   sec   112 MBytes   931 Mbits/sec                  
[  5]   5.01-6.00   sec   110 MBytes   933 Mbits/sec                  
[  5]   6.00-7.00   sec   112 MBytes   936 Mbits/sec                  
[  5]   7.00-8.00   sec   112 MBytes   935 Mbits/sec                  
[  5]   8.00-9.00   sec   111 MBytes   935 Mbits/sec                  
[  5]   9.00-10.00  sec   111 MBytes   935 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.08 GBytes   929 Mbits/sec                  sender
[  5]   0.00-10.20  sec  1.08 GBytes   911 Mbits/sec                  receiver

iperf Done.

$ iperf3 -c 10.1.1.1 -R 
Connecting to host 10.1.1.1, port 5201
Reverse mode, remote host 10.1.1.1 is sending
[  5] local 10.1.1.91 port 47836 connected to 10.1.1.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  76.1 MBytes   638 Mbits/sec                  
[  5]   1.00-2.00   sec   109 MBytes   912 Mbits/sec                  
[  5]   2.00-3.00   sec   112 MBytes   936 Mbits/sec                  
[  5]   3.00-4.00   sec   111 MBytes   933 Mbits/sec                  
[  5]   4.00-5.00   sec   112 MBytes   939 Mbits/sec                  
[  5]   5.00-6.00   sec   112 MBytes   939 Mbits/sec                  
[  5]   6.00-7.00   sec   110 MBytes   921 Mbits/sec                  
[  5]   7.00-8.00   sec   112 MBytes   939 Mbits/sec                  
[  5]   8.00-9.00   sec   112 MBytes   939 Mbits/sec                  
[  5]   9.00-10.00  sec   112 MBytes   938 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.19  sec  1.05 GBytes   887 Mbits/sec                  sender
[  5]   0.00-10.00  sec  1.05 GBytes   903 Mbits/sec                  receiver

iperf Done.

----

This is preventing an uplift of our bhyve hypervisor fleet from 11.3 to the 12 branch.  Thanks.
Comment 6 Kubilay Kocak freebsd_committer freebsd_triage 2020-01-15 03:03:25 UTC
@All For anyone affected, please provide (in a single attachment), so that the isolation/failure mode matrix is clearer:

- Exact FreeBSD version: uname -a output
- /var/run/dmesg.boot output
- pciconf -lv output
- *minimum* network (and jail if appropriate) configuration that reproduces the results, sanitized where necessary. 
- Additional details:

  - rx/tx affected, only rx, only tx?
  - only when bridge or tap?


@Jason If per our twitter conversation you could confirm/reproduce on latest HEAD, that would be great.

^Triage:

 - Track earliest affected FreeBSD version
 - Request feedback from maintainer
Comment 7 Jason Tubnor 2020-01-15 05:48:56 UTC
FreeBSD myhostname 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r356528: Thu Jan  9 04:56:46 UTC 2020     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64

See attachment for full dmesg

igb2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
        ether ac:1f:6b:71:aa:dd
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

vlan1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4200401<RXCSUM,LRO,RXCSUM_IPV6,NOMAP>
        ether ac:1f:6b:71:aa:dd
        inet6 fe80::ae1f:6bff:fe71:aadd%vlan1 prefixlen 64 tentative scopeid 0x6
        inet 10.1.1.10 netmask 0xffffff00 broadcast 10.1.1.255
        groups: vlan
        vlan: 1 vlanpcp: 0 parent interface: igb2
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

vm-vlan1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 8a:3c:0f:b9:aa:bb
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 11 priority 128 path cost 2000000
        member: vlan1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 20000
        groups: bridge vm-switch viid-05c3b@
        nd6 options=1<PERFORMNUD>

tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vmnet-guest-0-vlan1
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:ff:aa
        inet6 fe80::5a9c:fcff:fe10:ffaa%tap0 prefixlen 64 tentative scopeid 0xb
        groups: tap vm-port
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 10095


- Additional details:

  - only rx
  - when vlan interface is bridged to a tap and performing test between bhyve guest and physical computer on the same subnet external from the host
Comment 8 Jason Tubnor 2020-01-15 05:50:20 UTC
Created attachment 210748 [details]
dmesg of -CURRENT host with same issue.
Comment 9 Jason Tubnor 2020-01-15 21:45:52 UTC
Created attachment 210777 [details]
pciconf -lv of host running -CURRENT
Comment 10 Jason Tubnor 2020-03-30 04:04:06 UTC
Something has been committed in -HEAD which has fixed this issue.  Can others update their trees and confirm?  I'm not sure which changes over the last fortnight fixed this issue.  Will we see it back ported to 12?

Here is an in/out test from an OpenBSD-current guest on a 13-HEAD host:

$ iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.1.2.1, port 21866
[  5] local 10.1.1.1 port 5201 connected to 10.1.2.1 port 17536
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  57.0 MBytes   478 Mbits/sec
[  5]   1.00-2.00   sec   103 MBytes   861 Mbits/sec
[  5]   2.00-3.00   sec   106 MBytes   886 Mbits/sec
[  5]   3.00-4.00   sec   111 MBytes   932 Mbits/sec
[  5]   4.00-5.00   sec   112 MBytes   936 Mbits/sec
[  5]   5.00-6.00   sec   110 MBytes   920 Mbits/sec
[  5]   6.00-7.00   sec   112 MBytes   936 Mbits/sec
[  5]   7.00-8.00   sec   112 MBytes   938 Mbits/sec
[  5]   8.00-9.00   sec   111 MBytes   930 Mbits/sec
[  5]   9.00-10.00  sec   112 MBytes   938 Mbits/sec
[  5]  10.00-10.16  sec  18.1 MBytes   925 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.16  sec  1.04 GBytes   876 Mbits/sec                  sender
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.1.2.1, port 16277
[  5] local 10.1.1.1 port 5201 connected to 10.1.2.1 port 30495
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  80.7 MBytes   677 Mbits/sec
[  5]   1.00-2.00   sec   111 MBytes   931 Mbits/sec
[  5]   2.00-3.00   sec   111 MBytes   934 Mbits/sec
[  5]   3.00-4.00   sec   111 MBytes   934 Mbits/sec
[  5]   4.00-5.00   sec   111 MBytes   934 Mbits/sec
[  5]   5.00-6.00   sec   102 MBytes   859 Mbits/sec
[  5]   6.00-7.00   sec   111 MBytes   933 Mbits/sec
[  5]   7.00-8.00   sec   111 MBytes   934 Mbits/sec
[  5]   8.00-9.00   sec   111 MBytes   931 Mbits/sec
[  5]   9.00-10.00  sec   111 MBytes   934 Mbits/sec
[  5]  10.00-10.17  sec  18.7 MBytes   934 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.17  sec  1.07 GBytes   901 Mbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Comment 11 Kubilay Kocak freebsd_committer freebsd_triage 2020-03-30 04:06:38 UTC
(In reply to Jason Tubnor from comment #10)

What is the current revision of the system ("apparently fixed revision") and what was the previous revision?

Any chance you could bisect?
Comment 12 Eric Joyner freebsd_committer 2020-05-11 17:27:30 UTC
*bump*

Jason, could you answer Kubilay's questions? We could figure out what fixed this and merge it back to stable/12.
Comment 13 Jason Tubnor 2020-05-11 21:57:01 UTC
Sorry, Kubilay's update notification didn't hit my inbox for some reason.

There appears that some work was done in sys/net/iflib.* back in mid to late March that touch areas where this bug existed. I can't tell which change it was as I wasn't testing frequently enough but it was committed sometime mid March.  This was confirmed by emaste@ that this would have been the fix to the issue.

See here:
https://svnweb.freebsd.org/base/head/sys/net/

Uneducated guess that this was the change:
https://svnweb.freebsd.org/base?view=revision&revision=358998

I've check 12-stable, none of the fixes for the past few months in iflib have been back ported, so this need to be addressed if any of them are to make it into 12.2.