On 12-STALBE and -head (I didn't try with older version), creating a vlan generate interface DOWN + UP. Problem reproduced with I354 and I211. How to reproduce the problem: # ifconfig igb3.1000 create vlan 1000 vlandev igb3 igb3: link state changed to DOWN igb3.1000: link state changed to DOWN igb3: link state changed to UP igb3.1000: link state changed to UP So let's disable vlanhwfilter (this will trigger another down/up): # ifconfig igb3 -vlanhwfilter igb3: link state changed to DOWN igb3.1000: link state changed to DOWN igb3: link state changed to UP igb3.1000: link state changed to UP # ifconfig igb3.1100 create vlan 1100 vlandev igb3 => No more down/up with vlanhwfilter disabled
Currently, iflib based drivers issue an init after registering or unregistering vlans with IFCAP_VLAN_HWFILTER enabled ( https://svnweb.freebsd.org/base/stable/12/sys/net/iflib.c?revision=351276&view=markup#l4265 ). In case of e1000 driver, init is needed as vlan hw support setup is part of driver's initialization routine.
Here is a user use case: FreeBSD as a router serving multiple customer: one customer per vlan. Each time we add a new customer, we add a VLAN… and the link re-init, so all the customers are impacted. It's a big limitation for this use case.
It was reported by several people a year ago: https://lists.freebsd.org/pipermail/freebsd-net/2018-November/052184.html I also think that this behavior makes Intel cards with iflib drivers unusable for setups many vlans.
Wouldn't it be sufficient to init only on the first vlan ? For all further VLANs, the hardware would already be available ? I agree with the statement that that DOWN/UP events for every VLAN would make that type of interface useless for VLANs.
Need help testing this patch: https://reviews.freebsd.org/D24659
D24659 helps me on em, igb and ix
A commit references this bug: Author: erj Date: Mon May 11 17:42:05 UTC 2020 New revision: 360902 URL: https://svnweb.freebsd.org/changeset/base/360902 Log: em/ix/ixv/ixl/iavf: Implement ifdi_needs_restart iflib method Pursuant to r360398, implement driver-specific versions of the ifdi_needs_restart iflib device method. Some (if not most?) Intel network cards don't need reinitializing when a VLAN is added or removed from the device hardware, so these implement ifdi_needs_restart in a way that tell iflib not to bring the interface up or down when a VLAN is added or removed, regardless of whether the VLAN_HWFILTER interface capability flag is set or not. This could potentially solve several PRs relating to link flaps that occur when VLANs are added/removed to devices. Signed-off-by: Eric Joyner <erj@freebsd.org> PR: 240818, 241785 Reviewed by: gallatin@, olivier@ MFC after: 3 days MFC with: r360398 Sponsored by: Intel Corporation Differential Revision: https://reviews.freebsd.org/D24659 Changes: head/sys/dev/e1000/if_em.c head/sys/dev/ixgbe/if_ix.c head/sys/dev/ixgbe/if_ixv.c head/sys/dev/ixl/if_iavf.c head/sys/dev/ixl/if_ixl.c
A commit references this bug: Author: erj Date: Thu May 14 19:56:56 UTC 2020 New revision: 361053 URL: https://svnweb.freebsd.org/changeset/base/361053 Log: MFC r360398 and r360902 These commits introduce a new iflib device-dependent method and implements that method in the Intel ethernet network drivers; this method tells iflib if the network interface needs to be restarted when certain events happen. This fixes several issues that occur when VLANs are registered or unregistered with the network interface. PR: 240818, 241785 Sponsored by: Intel Corporation Changes: _U stable/12/ stable/12/sys/dev/e1000/if_em.c stable/12/sys/dev/ixgbe/if_ix.c stable/12/sys/dev/ixgbe/if_ixv.c stable/12/sys/dev/ixl/if_iavf.c stable/12/sys/dev/ixl/if_ixl.c stable/12/sys/net/ifdi_if.m stable/12/sys/net/iflib.c stable/12/sys/net/iflib.h
^Triage: - Assign to committer resolving (via base r360902) - Pending close of bug 240818
Beware I217 users. The r361053 has broken a bit startup process for em0 and its VLAN children in 12-STABLE. After bootup, the NIC is up and capable of sniffing traffic, the same for children VLAN subinterfaces. The only cons are that neither of them is able to send traffic. Bringing the NIC up and down solves the issue, unplugging and plugging media again doesn't change anything. em0@pci0:0:25:0: class=0x020000 card=0x1905103c chip=0x153a8086 rev=0x05 hdr=0x00 vendor = 'Intel Corporation' device = 'Ethernet Connection I217-LM' class = network subclass = ethernet cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message cap 13[e0] = PCI Advanced Features: FLR TP em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=81209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER> capabilities=953d9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP> ether 2c:44:fd:37:be:14 media: Ethernet autoselect (1000baseT <full-duplex>) status: active supported media: media autoselect media 1000baseT media 1000baseT mediaopt full-duplex media 100baseTX mediaopt full-duplex media 100baseTX media 10baseT/UTP mediaopt full-duplex media 10baseT/UTP nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> vlan7: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=3<RXCSUM,TXCSUM> capabilities=503<RXCSUM,TXCSUM,TSO4,LRO> ether 2c:44:fd:37:be:14 inet 10.0.x.y netmask 0xfffff800 broadcast 10.0.y.z groups: vlan vlan: 798 vlanpcp: 0 parent interface: em0 media: Ethernet autoselect (1000baseT <full-duplex>) status: active supported media: media autoselect # sysctl dev.em.0 dev.em.0.wake: 1 dev.em.0.interrupts.rx_overrun: 0 dev.em.0.interrupts.rx_desc_min_thresh: 0 dev.em.0.interrupts.tx_queue_min_thresh: 0 dev.em.0.interrupts.tx_queue_empty: 0 dev.em.0.interrupts.tx_abs_timer: 0 dev.em.0.interrupts.tx_pkt_timer: 0 dev.em.0.interrupts.rx_abs_timer: 0 dev.em.0.interrupts.rx_pkt_timer: 0 dev.em.0.interrupts.asserts: 916405 dev.em.0.mac_stats.tso_ctx_fail: 0 dev.em.0.mac_stats.tso_txd: 0 dev.em.0.mac_stats.tx_frames_1024_1522: 0 dev.em.0.mac_stats.tx_frames_512_1023: 0 dev.em.0.mac_stats.tx_frames_256_511: 0 dev.em.0.mac_stats.tx_frames_128_255: 0 dev.em.0.mac_stats.tx_frames_65_127: 0 dev.em.0.mac_stats.tx_frames_64: 0 dev.em.0.mac_stats.mcast_pkts_txd: 293 dev.em.0.mac_stats.bcast_pkts_txd: 38 dev.em.0.mac_stats.good_pkts_txd: 17046 dev.em.0.mac_stats.total_pkts_txd: 17046 dev.em.0.mac_stats.good_octets_txd: 5112315 dev.em.0.mac_stats.good_octets_recvd: 119853581 dev.em.0.mac_stats.rx_frames_1024_1522: 0 dev.em.0.mac_stats.rx_frames_512_1023: 0 dev.em.0.mac_stats.rx_frames_256_511: 0 dev.em.0.mac_stats.rx_frames_128_255: 0 dev.em.0.mac_stats.rx_frames_65_127: 0 dev.em.0.mac_stats.rx_frames_64: 0 dev.em.0.mac_stats.mcast_pkts_recvd: 323244 dev.em.0.mac_stats.bcast_pkts_recvd: 402143 dev.em.0.mac_stats.good_pkts_recvd: 995757 dev.em.0.mac_stats.total_pkts_recvd: 995757 dev.em.0.mac_stats.xoff_txd: 0 dev.em.0.mac_stats.xoff_recvd: 0 dev.em.0.mac_stats.xon_txd: 0 dev.em.0.mac_stats.xon_recvd: 0 dev.em.0.mac_stats.coll_ext_errs: 0 dev.em.0.mac_stats.alignment_errs: 0 dev.em.0.mac_stats.crc_errs: 0 dev.em.0.mac_stats.recv_errs: 0 dev.em.0.mac_stats.recv_jabber: 0 dev.em.0.mac_stats.recv_oversize: 0 dev.em.0.mac_stats.recv_fragmented: 0 dev.em.0.mac_stats.recv_undersize: 0 dev.em.0.mac_stats.recv_no_buff: 0 dev.em.0.mac_stats.missed_packets: 0 dev.em.0.mac_stats.defer_count: 0 dev.em.0.mac_stats.sequence_errors: 0 dev.em.0.mac_stats.symbol_errors: 0 dev.em.0.mac_stats.collision_count: 0 dev.em.0.mac_stats.late_coll: 0 dev.em.0.mac_stats.multiple_coll: 0 dev.em.0.mac_stats.single_coll: 0 dev.em.0.mac_stats.excess_coll: 0 dev.em.0.queue_rx_0.rx_irq: 0 dev.em.0.queue_rx_0.rxd_tail: 122 dev.em.0.queue_rx_0.rxd_head: 124 dev.em.0.queue_tx_0.tx_irq: 0 dev.em.0.queue_tx_0.txd_tail: 61 dev.em.0.queue_tx_0.txd_head: 63 dev.em.0.fc_low_water: 20552 dev.em.0.fc_high_water: 23584 dev.em.0.rx_control: 100892734 dev.em.0.device_control: 1075315264 dev.em.0.watchdog_timeouts: 0 dev.em.0.rx_overruns: 0 dev.em.0.link_irq: 0 dev.em.0.dropped: 0 dev.em.0.eee_control: 1 dev.em.0.itr: 488 dev.em.0.tx_abs_int_delay: 66 dev.em.0.rx_abs_int_delay: 66 dev.em.0.tx_int_delay: 66 dev.em.0.rx_int_delay: 0 dev.em.0.rs_dump: 0 dev.em.0.reg_dump: General Registers CTRL 40180240 STATUS 00080083 CTRL_EXIT 015a1006 Interrupt Registers ICR 00000000 RX Registers RCTL 0603803e RDLEN 00010000 RDH 0000007c RDT 0000007a RXDCTL 00010000 RDBAL 035c2000 RDBAH 00000001 TX Registers TCTL 3103f0fa TDBAL 0359a000 TDBAH 00000001 TDLEN 00010000 TDH 00000050 TDT 00000050 TXDCTL 0341011f TDFH 00000c46 TDFT 00000c46 TDFHS 00000c46 TDFPC 00000000 dev.em.0.fc: 3 dev.em.0.debug: -1 dev.em.0.nvm: -1 dev.em.0.iflib.rxq0.rxq_fl0.credits: 4095 dev.em.0.iflib.rxq0.rxq_fl0.cidx: 124 dev.em.0.iflib.rxq0.rxq_fl0.pidx: 123 dev.em.0.iflib.txq0.r_abdications: 0 dev.em.0.iflib.txq0.r_restarts: 0 dev.em.0.iflib.txq0.r_stalls: 0 dev.em.0.iflib.txq0.r_starts: 17094 dev.em.0.iflib.txq0.r_drops: 0 dev.em.0.iflib.txq0.r_enqueues: 17102 dev.em.0.iflib.txq0.ring_state: pidx_head: 0739 pidx_tail: 0739 cidx: 0739 state: IDLE dev.em.0.iflib.txq0.txq_cleaned: 24629 dev.em.0.iflib.txq0.txq_processed: 24671 dev.em.0.iflib.txq0.txq_in_use: 42 dev.em.0.iflib.txq0.txq_cidx_processed: 97 dev.em.0.iflib.txq0.txq_cidx: 59 dev.em.0.iflib.txq0.txq_pidx: 101 dev.em.0.iflib.txq0.no_tx_dma_setup: 0 dev.em.0.iflib.txq0.txd_encap_efbig: 0 dev.em.0.iflib.txq0.tx_map_failed: 0 dev.em.0.iflib.txq0.no_desc_avail: 0 dev.em.0.iflib.txq0.mbuf_defrag_failed: 0 dev.em.0.iflib.txq0.m_pullups: 5 dev.em.0.iflib.txq0.mbuf_defrag: 0 dev.em.0.iflib.override_nrxds: 4096 dev.em.0.iflib.override_ntxds: 4096 dev.em.0.iflib.separate_txrx: 0 dev.em.0.iflib.core_offset: 0 dev.em.0.iflib.tx_abdicate: 0 dev.em.0.iflib.rx_budget: 0 dev.em.0.iflib.disable_msix: 1 dev.em.0.iflib.override_qs_enable: 0 dev.em.0.iflib.override_nrxqs: 0 dev.em.0.iflib.override_ntxqs: 0 dev.em.0.iflib.driver_version: 7.6.1-k dev.em.0.%parent: pci0 dev.em.0.%pnpinfo: vendor=0x8086 device=0x153a subvendor=0x103c subdevice=0x1905 class=0x020000 dev.em.0.%location: slot=25 function=0 dbsf=pci0:0:25:0 handle=\_SB_.PCI0.GLAN dev.em.0.%driver: em dev.em.0.%desc: Intel(R) PRO/1000 Network Connection
I should have written "Bringing the NIC down and up solves the issue"; to clarify: I have not tested whether the NIC is able to transmit untagged frames.
Is anyone else seeing the problem that Marek is seeing? I don't have any 1G devices, so I can't try to reproduce this.
(In reply to Eric Joyner from comment #12) This could be I217-LM (em) specific or different flaw which came out after upgrade. Other 1G NICs like I219-LM (em) and I350 (igb) are initialising fine on 12-STABLE after this MFC.
Thanks to Mr Marek Zarychta here who has pointed me to this report, i can share that i have similar problems, related to r360902 I have a intel 82574L based network card who act as a link to my ISP, who is gaving me dhcp based authentication through a vlan after r360902 , the connectivity to my ISP can't be restored after a reboot or a power-on until, as Mr Marek Zarychta pointed i "Bring the NIC down and up " by the ifconfig(8) command and NOT by unplugging /replugging the ethernet cable back. I'll open a proprer bug report if needed to; meanwhile, i'll test the patch provided, in case of some interesting results Regards.
(In reply to martin.mato from comment #14) Well, i compiled successfully a new kernel, and it is working perfectly because the working generated if_em.c file has the same content than the revision pre-r360902, right?
(In reply to martin.mato from comment #14) Same situation here. After replacing isp router, no success obviously, the box ended relocated in a hurry on a data center that, just by chance, doesn't need vlan tagging to operate. I sadly didn't try down/up, but unplug/plug cable didn't help. em0@pci0:0:25:0: class=0x020000 card=0x20448086 chip=0x15038086 rev=0x04 hdr=0x00 vendor = 'Intel Corporation' device = '82579V Gigabit Network Connection' class = network subclass = ethernet cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message cap 13[e0] = PCI Advanced Features: FLR TP
(In reply to Raúl from comment #16) No, unfortunately: Unplugging/repluging the cable doesn't work, like i said. The only way is to manually reset by ifconfig(8) down and up the parent interface em0 , not the vlan interface(s). In the @current list ; Ian Freislich provided a patch who worked for him i tested succesfully as well on my box here it is Index: sys/dev/e1000/if_em.c =================================================================== --- sys/dev/e1000/if_em.c (revision 361538) +++ sys/dev/e1000/if_em.c (working copy) @@ -4054,7 +4054,7 @@ { switch (event) { case IFLIB_RESTART_VLAN_CONFIG: - return (false); + return (true); default: return (true); }
(In reply to martin.mato from comment #17) I'll look how to test it on site before returning it back home. Given covid-19 circumstances logistics are ... sort of worrisome. Thanks for the info.
I'll change em(4) so that it always does the interface down/up on VLAN config/unconfig, but leave igb(4) as-is. It seems like em(4) needs more investigation on what needs to be done to prevent a link flap.
A commit references this bug: Author: erj Date: Thu Jun 11 15:59:49 UTC 2020 New revision: 362063 URL: https://svnweb.freebsd.org/changeset/base/362063 Log: em(4): Always reinit interface when adding/removing VLAN This partially reverts r361053 since there have been reports by users that this breaks some functionality for em(4) devices; it seems at first glance that some sort of interface restart is required for those cards. This isn't a proper fix; this unbreaks those users until a proper fix is found for their issues. PR: 240818 Reported by: Marek Zarychta <zarychtam@plan-b.pwste.edu.pl> MFC after: 3 days Changes: head/sys/dev/e1000/if_em.c
(In reply to commit-hook from comment #20) Compiled a kernel and tested immediately upon receipt. So far, it's working very well, at least for me.
A commit references this bug: Author: erj Date: Thu Jul 30 18:31:25 UTC 2020 New revision: 363711 URL: https://svnweb.freebsd.org/changeset/base/363711 Log: MFC r362063: em(4): Always reinit interface when adding/removing VLAN PR: 240818 Sponsored by: Intel Corporation Changes: _U stable/12/ stable/12/sys/dev/e1000/if_em.c
(In reply to commit-hook from comment #22) Why is it closed as "FIXED"? Re-init interface on each VLAN setting change is only work-around, as old pre-iflib driver was able to re-confgure VLANs without interface up/down cycle!
See comment 23
I have confirmed this behaviour on RELENG_12 (r367411) e.g ifconfig vlan6 create 10.6.6.6/28 vlandev igb1 vlan 6 ifconfig vlan7 create 10.7.7.7/28 vlandev igb1 vlan 7 # dmesg | tail -19 vlan5: promiscuous mode enabled igb1: promiscuous mode disabled vlan5: promiscuous mode disabled igb1: promiscuous mode enabled igb1: promiscuous mode disabled igb1: link state changed to DOWN vlan5: link state changed to DOWN vlan6: link state changed to DOWN igb1: link state changed to UP vlan5: link state changed to UP vlan6: link state changed to UP igb1: link state changed to DOWN vlan5: link state changed to DOWN vlan6: link state changed to DOWN vlan7: link state changed to DOWN igb1: link state changed to UP vlan5: link state changed to UP vlan6: link state changed to UP vlan7: link state changed to UP ie. with each vlan created, igb1 and all the VLANS transition with a full carrier down/up (confirmed from the switch they are connected to) Confirmed on 3 different boards Supermicro X11SSH-F igb1@pci0:3:0:0: class=0x020000 card=0x153315d9 chip=0x15338086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = 'I210 Gigabit Network Connection' class = network subclass = ethernet bar [10] = type Memory, range 32, base rxdf200000, size 524288, enabled bar [18] = type I/O Port, range 32, base rxd000, size 32, enabled bar [1c] = type Memory, range 32, base rxdf280000, size 16384, enabled cap 01[40] = powerspec 3 supports D0 D3 current D0 cap 05[50] = MSI supports 1 message, 64 bit, vector masks cap 11[70] = MSI-X supports 5 messages, enabled Table in map 0x1c[0x0], PBA in map 0x1c[0x2000] cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR NS max read 512 link x1(x1) speed 2.5(2.5) ASPM L1(L0s/L1) ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected ecap 0003[140] = Serial 1 ac1f6bffff77ce57 ecap 0017[1a0] = TPH Requester 1 ASRockRack X470D4U # pciconf -lcvb igb1 igb1@pci0:36:0:0: class=0x020000 card=0x15331849 chip=0x15338086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = 'I210 Gigabit Network Connection' class = network subclass = ethernet bar [10] = type Memory, range 32, base rxf7300000, size 524288, enabled bar [18] = type I/O Port, range 32, base rxd000, size 32, enabled bar [1c] = type Memory, range 32, base rxf7380000, size 16384, enabled cap 01[40] = powerspec 3 supports D0 D3 current D0 cap 05[50] = MSI supports 1 message, 64 bit, vector masks cap 11[70] = MSI-X supports 5 messages, enabled Table in map 0x1c[0x0], PBA in map 0x1c[0x2000] cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS max read 512 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1) ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected ecap 0003[140] = Serial 1 7085c2fffffde3e8 ecap 0017[1a0] = TPH Requester 1 Supermicro SYS-5019C-M # pciconf -lvcb igb1 igb1@pci0:4:0:0: class=0x020000 card=0x153315d9 chip=0x15338086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = 'I210 Gigabit Network Connection' class = network subclass = ethernet bar [10] = type Memory, range 32, base rx93200000, size 524288, enabled bar [18] = type I/O Port, range 32, base rx4000, size 32, enabled bar [1c] = type Memory, range 32, base rx93280000, size 16384, enabled cap 01[40] = powerspec 3 supports D0 D3 current D0 cap 05[50] = MSI supports 1 message, 64 bit, vector masks cap 11[70] = MSI-X supports 5 messages, enabled Table in map 0x1c[0x0], PBA in map 0x1c[0x2000] cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS max read 512 link x1(x1) speed 2.5(2.5) ASPM L1(L0s/L1) ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected ecap 0003[140] = Serial 1 3cecefffff6d99ab ecap 0017[1a0] = TPH Requester 1
Probably fixed in https://cgit.freebsd.org/src/commit/?id=9c6432dc4bb936f0842d766d8b3b19dfcde15da2 See bug 230996.
^Triage: the 12 branch is now out of support. This was probably fixed in 2021.