Running a small howm-brewn router with only vanilla and standard FreeBSD on-board utilities on a PCEngines APU2C4, the built-in i210 Intel NICs stop working when they have VLAN tags assigned and enabling "vlanhwtag" on the base device (i.e., igb1 is the main device, igb1.10, igb1.100 and igb1.1000 are the VLAN devices). This phenomenon/bug seems also present on Intel i350 NIcs (I use a server with a dual port PCIe NIC of that kind).
I have very similar problem: igb/I210, FreeBSD 12-ALPHA8 (r339009). When I enable "vlanhwtag" on server, clients on this VLAN receive UDP with broken checksums. For example, client can not obtain address via DHCP from server with enabled "vlanhwtag", because DHCP client doesn't seen answers, because they are dropped by kernel due to invalid checksum: tcpdump sees DHCPOFFER on cleint's interface but dhclient doesn't receive anything.
I'm unclear if this is related to PR231416 or not. There needs to be a bit more clarification in what the vlanhwtag needs to do or if setting this flag somehow breaks what the udp stack is expecting.
(In reply to Sean Bruno from comment #2) My case is PR231416 for sure. Looks like TCP works with tis capability enabled, for example. I'm not sure about non-bpf originated UDP, though.
This issue still exists in 12.1-p1 While vlanhwtagging doesn't need to be disabled if using the host in a vanilla config, once and igb interface is placed into a bridge as a vlan with a tap device for a bhyve guest, ingress of data to the guest comes to a halt under iperf3 testing. egress from the guest is not impacted and runs at full speed. When vlanhwtagging is disabled, forcing the host to perform the function, data ingress or egress of the guest runs at full speed. I'd like to enable vlanhwtagging as the CPU load put on the host is huge when transferring a single 1Gb/s stream of traffic from guests.
With vlanhwtag enabled: $ iperf3 -c 10.1.1.1 Connecting to host 10.1.1.1, port 5201 [ 5] local 10.1.1.91 port 44086 connected to 10.1.1.1 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.01 sec 35.4 KBytes 287 Kbits/sec [ 5] 1.01-2.01 sec 2.83 KBytes 23.2 Kbits/sec [ 5] 2.01-3.02 sec 0.00 Bytes 0.00 bits/sec [ 5] 3.02-4.02 sec 0.00 Bytes 0.00 bits/sec [ 5] 4.02-5.01 sec 2.83 KBytes 23.4 Kbits/sec [ 5] 5.01-6.01 sec 5.66 KBytes 46.3 Kbits/sec [ 5] 6.01-7.01 sec 5.66 KBytes 46.3 Kbits/sec [ 5] 7.01-8.01 sec 4.24 KBytes 34.8 Kbits/sec [ 5] 8.01-9.01 sec 5.66 KBytes 46.3 Kbits/sec [ 5] 9.01-10.01 sec 5.66 KBytes 46.3 Kbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.01 sec 67.9 KBytes 55.6 Kbits/sec sender [ 5] 0.00-10.20 sec 48.1 KBytes 38.6 Kbits/sec receiver iperf Done. $ iperf3 -c 10.1.1.1 -R Connecting to host 10.1.1.1, port 5201 Reverse mode, remote host 10.1.1.1 is sending [ 5] local 10.1.1.91 port 11269 connected to 10.1.1.1 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 64.5 MBytes 541 Mbits/sec [ 5] 1.00-2.00 sec 105 MBytes 883 Mbits/sec [ 5] 2.00-3.00 sec 111 MBytes 932 Mbits/sec [ 5] 3.00-4.00 sec 111 MBytes 929 Mbits/sec [ 5] 4.00-5.00 sec 112 MBytes 938 Mbits/sec [ 5] 5.00-6.00 sec 112 MBytes 938 Mbits/sec [ 5] 6.00-7.00 sec 111 MBytes 934 Mbits/sec [ 5] 7.00-8.00 sec 112 MBytes 937 Mbits/sec [ 5] 8.00-9.00 sec 112 MBytes 936 Mbits/sec [ 5] 9.00-10.00 sec 111 MBytes 935 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.19 sec 1.04 GBytes 874 Mbits/sec sender [ 5] 0.00-10.00 sec 1.04 GBytes 890 Mbits/sec receiver iperf Done. With hwvlantag disabled (-hwvlantag): $ iperf3 -c 10.1.1.1 Connecting to host 10.1.1.1, port 5201 [ 5] local 10.1.1.91 port 46471 connected to 10.1.1.1 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 105 MBytes 877 Mbits/sec [ 5] 1.00-2.00 sec 112 MBytes 936 Mbits/sec [ 5] 2.00-3.00 sec 112 MBytes 936 Mbits/sec [ 5] 3.00-4.00 sec 111 MBytes 933 Mbits/sec [ 5] 4.00-5.01 sec 112 MBytes 931 Mbits/sec [ 5] 5.01-6.00 sec 110 MBytes 933 Mbits/sec [ 5] 6.00-7.00 sec 112 MBytes 936 Mbits/sec [ 5] 7.00-8.00 sec 112 MBytes 935 Mbits/sec [ 5] 8.00-9.00 sec 111 MBytes 935 Mbits/sec [ 5] 9.00-10.00 sec 111 MBytes 935 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 1.08 GBytes 929 Mbits/sec sender [ 5] 0.00-10.20 sec 1.08 GBytes 911 Mbits/sec receiver iperf Done. $ iperf3 -c 10.1.1.1 -R Connecting to host 10.1.1.1, port 5201 Reverse mode, remote host 10.1.1.1 is sending [ 5] local 10.1.1.91 port 47836 connected to 10.1.1.1 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 76.1 MBytes 638 Mbits/sec [ 5] 1.00-2.00 sec 109 MBytes 912 Mbits/sec [ 5] 2.00-3.00 sec 112 MBytes 936 Mbits/sec [ 5] 3.00-4.00 sec 111 MBytes 933 Mbits/sec [ 5] 4.00-5.00 sec 112 MBytes 939 Mbits/sec [ 5] 5.00-6.00 sec 112 MBytes 939 Mbits/sec [ 5] 6.00-7.00 sec 110 MBytes 921 Mbits/sec [ 5] 7.00-8.00 sec 112 MBytes 939 Mbits/sec [ 5] 8.00-9.00 sec 112 MBytes 939 Mbits/sec [ 5] 9.00-10.00 sec 112 MBytes 938 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.19 sec 1.05 GBytes 887 Mbits/sec sender [ 5] 0.00-10.00 sec 1.05 GBytes 903 Mbits/sec receiver iperf Done. ---- This is preventing an uplift of our bhyve hypervisor fleet from 11.3 to the 12 branch. Thanks.
@All For anyone affected, please provide (in a single attachment), so that the isolation/failure mode matrix is clearer: - Exact FreeBSD version: uname -a output - /var/run/dmesg.boot output - pciconf -lv output - *minimum* network (and jail if appropriate) configuration that reproduces the results, sanitized where necessary. - Additional details: - rx/tx affected, only rx, only tx? - only when bridge or tap? @Jason If per our twitter conversation you could confirm/reproduce on latest HEAD, that would be great. ^Triage: - Track earliest affected FreeBSD version - Request feedback from maintainer
FreeBSD myhostname 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r356528: Thu Jan 9 04:56:46 UTC 2020 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 See attachment for full dmesg igb2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP> ether ac:1f:6b:71:aa:dd media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> vlan1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=4200401<RXCSUM,LRO,RXCSUM_IPV6,NOMAP> ether ac:1f:6b:71:aa:dd inet6 fe80::ae1f:6bff:fe71:aadd%vlan1 prefixlen 64 tentative scopeid 0x6 inet 10.1.1.10 netmask 0xffffff00 broadcast 10.1.1.255 groups: vlan vlan: 1 vlanpcp: 0 parent interface: igb2 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> vm-vlan1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 8a:3c:0f:b9:aa:bb id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 11 priority 128 path cost 2000000 member: vlan1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 6 priority 128 path cost 20000 groups: bridge vm-switch viid-05c3b@ nd6 options=1<PERFORMNUD> tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: vmnet-guest-0-vlan1 options=80000<LINKSTATE> ether 58:9c:fc:10:ff:aa inet6 fe80::5a9c:fcff:fe10:ffaa%tap0 prefixlen 64 tentative scopeid 0xb groups: tap vm-port media: Ethernet autoselect status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> Opened by PID 10095 - Additional details: - only rx - when vlan interface is bridged to a tap and performing test between bhyve guest and physical computer on the same subnet external from the host
Created attachment 210748 [details] dmesg of -CURRENT host with same issue.
Created attachment 210777 [details] pciconf -lv of host running -CURRENT
Something has been committed in -HEAD which has fixed this issue. Can others update their trees and confirm? I'm not sure which changes over the last fortnight fixed this issue. Will we see it back ported to 12? Here is an in/out test from an OpenBSD-current guest on a 13-HEAD host: $ iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 10.1.2.1, port 21866 [ 5] local 10.1.1.1 port 5201 connected to 10.1.2.1 port 17536 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 57.0 MBytes 478 Mbits/sec [ 5] 1.00-2.00 sec 103 MBytes 861 Mbits/sec [ 5] 2.00-3.00 sec 106 MBytes 886 Mbits/sec [ 5] 3.00-4.00 sec 111 MBytes 932 Mbits/sec [ 5] 4.00-5.00 sec 112 MBytes 936 Mbits/sec [ 5] 5.00-6.00 sec 110 MBytes 920 Mbits/sec [ 5] 6.00-7.00 sec 112 MBytes 936 Mbits/sec [ 5] 7.00-8.00 sec 112 MBytes 938 Mbits/sec [ 5] 8.00-9.00 sec 111 MBytes 930 Mbits/sec [ 5] 9.00-10.00 sec 112 MBytes 938 Mbits/sec [ 5] 10.00-10.16 sec 18.1 MBytes 925 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.16 sec 1.04 GBytes 876 Mbits/sec sender ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 10.1.2.1, port 16277 [ 5] local 10.1.1.1 port 5201 connected to 10.1.2.1 port 30495 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 80.7 MBytes 677 Mbits/sec [ 5] 1.00-2.00 sec 111 MBytes 931 Mbits/sec [ 5] 2.00-3.00 sec 111 MBytes 934 Mbits/sec [ 5] 3.00-4.00 sec 111 MBytes 934 Mbits/sec [ 5] 4.00-5.00 sec 111 MBytes 934 Mbits/sec [ 5] 5.00-6.00 sec 102 MBytes 859 Mbits/sec [ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec [ 5] 7.00-8.00 sec 111 MBytes 934 Mbits/sec [ 5] 8.00-9.00 sec 111 MBytes 931 Mbits/sec [ 5] 9.00-10.00 sec 111 MBytes 934 Mbits/sec [ 5] 10.00-10.17 sec 18.7 MBytes 934 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.17 sec 1.07 GBytes 901 Mbits/sec receiver ----------------------------------------------------------- Server listening on 5201 -----------------------------------------------------------
(In reply to Jason Tubnor from comment #10) What is the current revision of the system ("apparently fixed revision") and what was the previous revision? Any chance you could bisect?
*bump* Jason, could you answer Kubilay's questions? We could figure out what fixed this and merge it back to stable/12.
Sorry, Kubilay's update notification didn't hit my inbox for some reason. There appears that some work was done in sys/net/iflib.* back in mid to late March that touch areas where this bug existed. I can't tell which change it was as I wasn't testing frequently enough but it was committed sometime mid March. This was confirmed by emaste@ that this would have been the fix to the issue. See here: https://svnweb.freebsd.org/base/head/sys/net/ Uneducated guess that this was the change: https://svnweb.freebsd.org/base?view=revision&revision=358998 I've check 12-stable, none of the fixes for the past few months in iflib have been back ported, so this need to be addressed if any of them are to make it into 12.2.
While one area was fixed regarding this bug, during edge case testing on FreeBSD 12.2-BETA1, this issue remains where IPSec/VPN traffic that is sent through this host (which cannot be fragmented) is fragmented, thus tanking the send stream. Below are snips of traffic with and without vlanhwtag enabled and what the bhyve guest sees coming from the FreeBSD 12.2-BETA1 host when vlanhwtag is enabled (not included is the guest when disabled as there is no invalid traffic.
Created attachment 218196 [details] Disabled vlanhwtag traffic
Created attachment 218197 [details] Enabled vlanhwtag traffic
Created attachment 218198 [details] Guest Enabled vlanhwtag traffic
Not sure if related, but an em driver, Intel 1000 Pro, I have been having some major stability issues with hard freezes, unresponsive even on the console. Tried changing RAM and CPU, and randomly came across this bug report. I've been trying to get VLAN + bhyve VM via bridge interface bridging em0 with the VM tap interface. Ever since I deleted the VLAN interface/bridge/etc it's been running smooth, no problems for the last 3 days. As part of my VLAN/bridge attempt, VM was using DHCP to get a IP, and the DHCP reply from the router was not being received on the VLAN interface/bridge, but on the non-VLAN side (thread on https://forums.freebsd.org/threads/vlans-with-bhyve-guests-not-getting-dhcp.77647/ I started around this). This is current 12.2-RELEASE. Usually nothing on the console or in dmesg, debug.log, messages after power cycling. However, there was once a few weeks ago (before I found this bug report) where I did find a panic trace on the console, took a photo. It's a spin lock held too long panic. Here's my pciconf -lv of the device em0@pci0:4:0:0: class=0x020000 card=0x10938086 chip=0x107d8086 rev=0x06 hdr=0x00 vendor = 'Intel Corporation' device = '82572EI Gigabit Ethernet Controller (Copper)' class = network subclass = ethernet
Created attachment 220541 [details] Photo of backtrace mentioned
We're seeing the same, or at least very similar, issue on 13.0-RELEASE. In our case, we only have the problem if the port is in gigabit mode. If we limit the switch to 100, we don't see the issue. Our configuration is passing a vlan over a bridge interface: ifconfig_igb0="DHCP" ifconfig_igb1="inet 172.28.1.1 netmask 255.255.255.0" dhcpd_ifaces="igb1" vlans_igb0="1501" vlans_igb1="1501" ifconfig_igb0_1501="up" ifconfig_igb1_1501="up" cloned_interfaces="bridge0" ifconfig_bridge0="addm igb0.1501 addm igb1.1501 up" Adding -vlanhwtag to igb0 and igb1 resolves it. pciconf -lv: igb0@pci0:2:0:0: class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x1533 subvendor=0x15d9 subdevice=0x1533 vendor = 'Intel Corporation' device = 'I210 Gigabit Network Connection' class = network subclass = ethernet
Can all of you test this https://reviews.freebsd.org/D30002 - no guarantees but there were some obvious problems to fix. Aaron, your issue I am not sure about. It looks completely unrelated to networking. Can you capture a crash dump?
I tested my case with this patch, and it doesn't seem to make an improvement. I'm still seeing extremely slow throughput across the interface when vlanhwtag is enabled.
(In reply to Kevin Bowling from comment #21) So after much poking around, turns out there's something in FreeBSD bridges and VLANs, where if you have a bridge on the untagged interface, it causes problems with tagged traffic making it to bridges on the tagged interfaces. Once I switched to using bridges only on VLAN tagged interfaces, everything miraculously started working. Extremely annoying.
Created attachment 224497 [details] vlan patch How about this patch? 1. change initializing order. setting up vlan inside the function em_initialize_receive_unit() 2. correct calculation of receive buffer size 3. correct writing method of vfta(vlan filter table array)
I've incorporated Kaho Toshikazu's fixes in https://reviews.freebsd.org/D30002 if anyone can retest. It does sound like there is special interaction with if_bridge I need to chase down, but I would appreciate any feedback on the above patch as it should be ready to go barring any further review feedback.
(In reply to Kevin Bowling from comment #25) I don't test review D30002, but I think it contains some inappropriate parts relating in "1. Call down to the hw support routine on vlan register and unregister events". I think that "call hw support" would not be solve the problem but make other trouble. I think vlan register/unregister parts in current codes are not broken. I think that using E1000_WRITE_REG_ARRAY directly is a obvious bug which was introduced at iflib conversion. The driver stores a device specific function for writing vfta into the e1000_write_vfta variable, and should use this variable as a function of writing vfta. At first, correcting only this area is a good course, I think.
I tested D30002 this morning, and in my case it didn't improve anything but didn't seem to break anything obvious either. Still poor throughput across the VLAN with hwtag enabled.
Created attachment 224588 [details] iflib-vlan patch It contains a correction of iflib side with debug code, and it does not conflicts other patch.
(In reply to Kaho Toshikazu from comment #28) Can you explain how this helps? It looks to me that the current iflib_vlan_register compares two softcs as expected.
(In reply to Kevin Bowling from comment #29) EVENTHANDLER_REGISTER seems to require (struct ifnet *) which typedef if_t at if_var.h. When a vlan interface is created/destroyed, the vlan_config/unconfig handler is called, but I can not observe any execution of iflib_vlan_register or iflib_vlan_unregister at using current code. The conversion form ctx to ifp makes execution of handlers. Then these handlers are called when a event is happened regardless for other devices or for itself. The compare is required to pick up a event related to a device driver. The handler calls with if_t because of being registered if_t instead of if_ctx_t by the above modification.
Following up with a "me too" on FreeBSD 13.0-RELEASE, with the following configuration: igb0/igb1/igb2/igb3 in a lagg0 on top of which there are multiple vlan interfaces that are created. On FreeBSD 13, even disabling all the hardware assists I was unable to get traffic to flow, on FreeBSD 12.2, no issues at all and everything functions as designed. Some information: igb0@pci0:1:0:0: class=0x020000 card=0x00008086 chip=0x15398086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = 'I211 Gigabit Network Connection' class = network subclass = ethernet igb0: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0xe000-0xe01f mem 0xdf500000-0xdf51ffff,0xdf520000-0xdf523fff irq 16 at device 0.0 on pci1 igb0: Using 1024 TX descriptors and 1024 RX descriptors igb0: Using 2 RX queues 2 TX queues igb0: Using MSI-X interrupts with 3 vectors igb0: Ethernet address: 40:62:31:08:92:76 igb0: netmap queues/slots: TX 2/1024, RX 2/1024 root@Breached:/usr/home/xistence # ifconfig igb0 igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 40:62:31:08:92:76 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> root@Breached:/usr/home/xistence # ifconfig lagg0 lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 40:62:31:08:92:76 inet6 fe80::4262:31ff:fe08:9276%lagg0 prefixlen 64 scopeid 0x8 inet6 2604:5500:c22a:7f00:4262:31ff:fe08:9276 prefixlen 64 inet 172.16.109.1 netmask 0xffffff00 broadcast 172.16.109.255 laggproto lacp lagghash l2,l3,l4 laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> laggport: igb3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> groups: lagg media: Ethernet autoselect status: active nd6 options=61<PERFORMNUD,AUTO_LINKLOCAL,NO_RADR> root@Breached:/usr/home/xistence # ifconfig vlan10 vlan10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6> ether 40:62:31:08:92:76 inet 192.168.10.1 netmask 0xffffff00 broadcast 192.168.10.255 inet6 fe80::4262:31ff:fe08:9276%vlan10 prefixlen 64 scopeid 0x9 inet6 2604:5500:c22a:7f01:4262:31ff:fe08:9276 prefixlen 64 groups: vlan vlan: 10 vlanpcp: 0 parent interface: lagg0 media: Ethernet autoselect status: active nd6 options=61<PERFORMNUD,AUTO_LINKLOCAL,NO_RADR> This is unfortunately my primary router to the internet, so I am unable to experiment with patches. Things I did note: - traffic on lagg0 did function (untagged) - tcpdump on lagg0 did not show any 802.1q frames - tcpdump on igb0 DID show 802.1q frames, but only rx, no tx - rx worked on the vlan interfaces (saw packets coming in) - tx did NOT work, no data was transmitted If this is the wrong bug, please let me know and I can file a new one.
Bert, Through further investigation on how interfaces behave in a VLAN/non-VLAN stage where bridges are concerned, I have found the following interface adjustment in rc.conf remediated the issue (however, the driver should automagically do this for the user): ifconfig_igb0="-txcsum -txcsum6 -lro -tso up" Adjust for your specific interface. Can you try that and report back?
(In reply to Bert JW Regeer from comment #31) You seem to have missed the VLAN Hardware TSO (VLAN_HWTSO) as well. I have not experienced this bug, however my interfaces are configured in a similar way as Jason suggests: ifconfig_igb0="-lro -tso -vlanhwtso up"
I only needed to apply -vlanhwtso in early versions of 12. We have moved our appliance direct from 11.4 to 13 because of these weird issues in 12 that made it too complex from a configuration point of view and chasing option ghosts. While 13 still has bridged VLAN issues, we have been able to mitigate against them without it being too complex (options used are in my previous post).
I'm sorry for the extra noise all, I made a mistake. On my system FreeBSD 13.0 works perfectly now. What I missed and what Allan Jude mentioned on twitter was that I was using an out of sync kernel <-> user land due to rebooting after freebsd-install told me to do so, but without then continuing with the upgrade and rebooting again. root@Breached:/usr/home/xistence # uname -a FreeBSD Breached.home.arpa 13.0-RELEASE-p1 FreeBSD 13.0-RELEASE-p1 #0: Wed May 26 22:15:09 UTC 2021 root@amd64-builder.daemonology.net:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 root@Breached:/usr/home/xistence # ifconfig igb0 igb0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP> ether 40:62:31:08:92:76 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> root@Breached:/usr/home/xistence # ifconfig lagg0 lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP> ether 40:62:31:08:92:76 inet6 fe80::4262:31ff:fe08:9276%lagg0 prefixlen 64 scopeid 0x8 inet6 2604:5500:c22a:7f00:4262:31ff:fe08:9276 prefixlen 64 inet 172.16.109.1 netmask 0xffffff00 broadcast 172.16.109.255 laggproto lacp lagghash l2,l3,l4 laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> laggport: igb3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> groups: lagg media: Ethernet autoselect status: active nd6 options=61<PERFORMNUD,AUTO_LINKLOCAL,NO_RADR> root@Breached:/usr/home/xistence # ifconfig vlan10 vlan10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=4600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP> ether 40:62:31:08:92:76 inet 192.168.10.1 netmask 0xffffff00 broadcast 192.168.10.255 inet6 fe80::4262:31ff:fe08:9276%vlan10 prefixlen 64 scopeid 0x9 inet6 2604:5500:c22a:7f01:4262:31ff:fe08:9276 prefixlen 64 groups: vlan vlan: 10 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0 media: Ethernet autoselect status: active nd6 options=61<PERFORMNUD,AUTO_LINKLOCAL,NO_RADR Is fully functional and operational. I am also seeing 1 Gbps being routed when doing tests between different vlan's. Once again apologies for the extra noise!
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=2796f7cab10785ef40efbba97ef67ab319c96e9c commit 2796f7cab10785ef40efbba97ef67ab319c96e9c Author: Kevin Bowling <kbowling@FreeBSD.org> AuthorDate: 2021-09-15 14:47:19 +0000 Commit: Kevin Bowling <kbowling@FreeBSD.org> CommitDate: 2021-09-15 15:03:01 +0000 e1000: Fix up HW vlan ops * Don't reset the entire adapter for vlan changes, fix up the problems * Add some functions for vlan filter (vfta) manipulation * Don't muck with the vfta if we aren't doing HW vlan filtering * Disable interrupts when manipulating vfta on lem(4)-class NICs * On the I350 there is a specification update (2.4.20) in which the suggested workaround is to write to the vfta 10 times (if at first you don't succeed, try, try again). Our shared code has the goods, use it * Increase a VF's frame receive size in the case of vlans From the referenced PR, this reduced vlan configuration from minutes to seconds with hundreds or thousands of vlans and prevents wedging the adapter with needless adapter reinitialization for each vlan ID. PR: 230996 Reviewed by: markj Tested by: Ozkan KIRIK <ozkan.kirik@gmail.com> MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D30002 sys/dev/e1000/if_em.c | 166 ++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 126 insertions(+), 40 deletions(-)
The above commit I intended to mention 230996 and then this PR as partial. It is not obvious what is or isn't going on here because there is enough postulating and recanting of issues to confuse me. (In reply to Jason Tubnor from comment #34) Can you clarify which and why these flags are needed? I work bottom up and can't see the need, if you can help me see the bug I can take a shot at fixing it.
(In reply to Kevin Bowling from comment #37) Typical these features should be automatically stripped from an interfaces feature set when the interface has been added to a bridge (which happens correctly): ifconfig_igb0="-txcsum -txcsum6 -lro -tso up" However, if the interface is being used as a parent for a VLAN interface, when the VLAN is added to a bridge, the driver subset does not remove these features from the parent interface. Causing various issues to data flows including data not flowing at all. The above is added in /etc/rc.conf to work around this fault in the VLAN/igb drivers.
(In reply to Jason Tubnor from comment #38) The reason I am confused by this is I think I run a similar configuration on Chelsio and do not need to drop the flags: https://gist.github.com/kev009/b8171b1988d38477b116d0de3c7ec315 Therefore I am wondering if we are getting something wrong on the VLAN offloads. If you can test with main or stable/{12,13} with the above patch I'd be curious too. To help me orient on this, can you provide a toy rc.conf or sequence of ifconfig commands to build the "wrong" topology that triggers the bug?
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=9c6432dc4bb936f0842d766d8b3b19dfcde15da2 commit 9c6432dc4bb936f0842d766d8b3b19dfcde15da2 Author: Kevin Bowling <kbowling@FreeBSD.org> AuthorDate: 2021-09-15 14:47:19 +0000 Commit: Kevin Bowling <kbowling@FreeBSD.org> CommitDate: 2021-09-28 16:31:52 +0000 e1000: Fix up HW vlan ops 2796f7ca: * Don't reset the entire adapter for vlan changes, fix up the problems * Add some functions for vlan filter (vfta) manipulation * Don't muck with the vfta if we aren't doing HW vlan filtering * Disable interrupts when manipulating vfta on lem(4)-class NICs * On the I350 there is a specification update (2.4.20) in which the suggested workaround is to write to the vfta 10 times (if at first you don't succeed, try, try again). Our shared code has the goods, use it * Increase a VF's frame receive size in the case of vlans From the referenced PR, this reduced vlan configuration from minutes to seconds with hundreds or thousands of vlans and prevents wedging the adapter with needless adapter reinitialization for each vlan ID. PR: 230996 Reviewed by: markj Tested by: Ozkan KIRIK <ozkan.kirik@gmail.com> Differential Revision: https://reviews.freebsd.org/D30002 22b20b45: e1000: Fix variable typo Forgot to git add this in last commit Reported by: jenkins Fixes: 2796f7cab107 (cherry picked from commit 2796f7cab10785ef40efbba97ef67ab319c96e9c) (cherry picked from commit 22b20b45c9118bf6ef313c074cdb107a1eaca78e) sys/dev/e1000/if_em.c | 166 ++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 126 insertions(+), 40 deletions(-)
A commit in branch stable/12 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=d239e11a57c9b4ebc5e4289a8520511bb7b40b2d commit d239e11a57c9b4ebc5e4289a8520511bb7b40b2d Author: Kevin Bowling <kbowling@FreeBSD.org> AuthorDate: 2021-09-15 14:47:19 +0000 Commit: Kevin Bowling <kbowling@FreeBSD.org> CommitDate: 2021-09-28 16:34:18 +0000 e1000: Fix up HW vlan ops 2796f7ca: * Don't reset the entire adapter for vlan changes, fix up the problems * Add some functions for vlan filter (vfta) manipulation * Don't muck with the vfta if we aren't doing HW vlan filtering * Disable interrupts when manipulating vfta on lem(4)-class NICs * On the I350 there is a specification update (2.4.20) in which the suggested workaround is to write to the vfta 10 times (if at first you don't succeed, try, try again). Our shared code has the goods, use it * Increase a VF's frame receive size in the case of vlans From the referenced PR, this reduced vlan configuration from minutes to seconds with hundreds or thousands of vlans and prevents wedging the adapter with needless adapter reinitialization for each vlan ID. PR: 230996 Reviewed by: markj Tested by: Ozkan KIRIK <ozkan.kirik@gmail.com> Differential Revision: https://reviews.freebsd.org/D30002 22b20b45: e1000: Fix variable typo Forgot to git add this in last commit Reported by: jenkins Fixes: 2796f7cab107 (cherry picked from commit 2796f7cab10785ef40efbba97ef67ab319c96e9c) (cherry picked from commit 22b20b45c9118bf6ef313c074cdb107a1eaca78e) sys/dev/e1000/if_em.c | 166 ++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 126 insertions(+), 40 deletions(-)
I have added a comment to the review. I received an undeclared identifier: /usr/src/sys/dev/e1000/if_em.c:3329:28: error: use of undeclared identifier 'pszie'; did you mean 'psize'? e1000_rlpml_set_vf(hw, pszie); ^~~~~ psize /usr/src/sys/dev/e1000/if_em.c:3320:7: note: 'psize' declared here u32 psize, srrctl = 0; ^ 1 error generated. *** Error code 1 Stop. --- Changed the above to psize and all built and runs ok. Looking at the commit, this was fixed between revision and commit. Testing on stacked VLAN igb interfaces works as intended on 13-STABLE-20210923 without removing any interface options (was working fine before the patch), so no regression with existing systems. Also tested where VLANs were added to bridges and bhyve guest taps added to the bridge and no performance regressions were identified on the guests or hosts. Not tested with 12-stable.
I spoke with Jason Tubnor in a private email and I am not aware of any current issues with VLANs on i210/i350 on stable/12+. Are there any concerns from the reporters or cc list in closing this issue? If you have some other specific issue can you open a new PR?
(In reply to Kevin Bowling from comment #43) root@gate:~ # ifconfig igb0 igb0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: WAN-uplnk options=4e120bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP> The router I opened the PR upon is now running 13-STABLE and the above ifconfig result line indicates, that the same hardware is now operating as expected. This seems also true for 14-CURRENT. I can not speak for any flavour of 12, since I don't use this version any more. I'm free to close the PR. Thank you very much for solving the problem and making FreeBSD better! Kind regards, oh
(In reply to O. Hartmann from comment #44) I'm not certain I solved the bug in question here; I can confirm that iflib and e1000 are roughly in line between all of: stable/12, stable/13, and main as of now. Hopefully this results in correct function for everyone, otherwise please feel free to open new PRs with your issues!
Maybe bug 240818 should be closed too.