Bug 240818
| Summary: | igb(4) vlanhwfilter feature generate link UP/DOWN for each new vlan created | ||
|---|---|---|---|
| Product: | Base System | Reporter: | Olivier Cochard <olivier> |
| Component: | kern | Assignee: | Eric Joyner <erj> |
| Status: | Closed FIXED | ||
| Severity: | Affects Many People | CC: | ae, afedorov, emaste, kirill, lev, martin.mato, mike, net, oleg, pi, piotr.pietruszewski, raul.munoz, zarychtam |
| Priority: | --- | Flags: | koobs:
mfc-stable12+
|
| Version: | CURRENT | ||
| Hardware: | Any | ||
| OS: | Any | ||
| See Also: | https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241785 | ||
|
Description
Olivier Cochard
2019-09-25 14:58:24 UTC
Currently, iflib based drivers issue an init after registering or unregistering vlans with IFCAP_VLAN_HWFILTER enabled ( https://svnweb.freebsd.org/base/stable/12/sys/net/iflib.c?revision=351276&view=markup#l4265 ). In case of e1000 driver, init is needed as vlan hw support setup is part of driver's initialization routine. Here is a user use case: FreeBSD as a router serving multiple customer: one customer per vlan. Each time we add a new customer, we add a VLAN… and the link re-init, so all the customers are impacted. It's a big limitation for this use case. It was reported by several people a year ago: https://lists.freebsd.org/pipermail/freebsd-net/2018-November/052184.html I also think that this behavior makes Intel cards with iflib drivers unusable for setups many vlans. Wouldn't it be sufficient to init only on the first vlan ? For all further VLANs, the hardware would already be available ? I agree with the statement that that DOWN/UP events for every VLAN would make that type of interface useless for VLANs. Need help testing this patch: https://reviews.freebsd.org/D24659 D24659 helps me on em, igb and ix A commit references this bug: Author: erj Date: Mon May 11 17:42:05 UTC 2020 New revision: 360902 URL: https://svnweb.freebsd.org/changeset/base/360902 Log: em/ix/ixv/ixl/iavf: Implement ifdi_needs_restart iflib method Pursuant to r360398, implement driver-specific versions of the ifdi_needs_restart iflib device method. Some (if not most?) Intel network cards don't need reinitializing when a VLAN is added or removed from the device hardware, so these implement ifdi_needs_restart in a way that tell iflib not to bring the interface up or down when a VLAN is added or removed, regardless of whether the VLAN_HWFILTER interface capability flag is set or not. This could potentially solve several PRs relating to link flaps that occur when VLANs are added/removed to devices. Signed-off-by: Eric Joyner <erj@freebsd.org> PR: 240818, 241785 Reviewed by: gallatin@, olivier@ MFC after: 3 days MFC with: r360398 Sponsored by: Intel Corporation Differential Revision: https://reviews.freebsd.org/D24659 Changes: head/sys/dev/e1000/if_em.c head/sys/dev/ixgbe/if_ix.c head/sys/dev/ixgbe/if_ixv.c head/sys/dev/ixl/if_iavf.c head/sys/dev/ixl/if_ixl.c A commit references this bug: Author: erj Date: Thu May 14 19:56:56 UTC 2020 New revision: 361053 URL: https://svnweb.freebsd.org/changeset/base/361053 Log: MFC r360398 and r360902 These commits introduce a new iflib device-dependent method and implements that method in the Intel ethernet network drivers; this method tells iflib if the network interface needs to be restarted when certain events happen. This fixes several issues that occur when VLANs are registered or unregistered with the network interface. PR: 240818, 241785 Sponsored by: Intel Corporation Changes: _U stable/12/ stable/12/sys/dev/e1000/if_em.c stable/12/sys/dev/ixgbe/if_ix.c stable/12/sys/dev/ixgbe/if_ixv.c stable/12/sys/dev/ixl/if_iavf.c stable/12/sys/dev/ixl/if_ixl.c stable/12/sys/net/ifdi_if.m stable/12/sys/net/iflib.c stable/12/sys/net/iflib.h ^Triage: - Assign to committer resolving (via base r360902) - Pending close of bug 240818 Beware I217 users. The r361053 has broken a bit startup process for em0 and its VLAN children in 12-STABLE. After bootup, the NIC is up and capable of sniffing traffic, the same for children VLAN subinterfaces. The only cons are that neither of them is able to send traffic. Bringing the NIC up and down solves the issue, unplugging and plugging media again doesn't change anything.
em0@pci0:0:25:0: class=0x020000 card=0x1905103c chip=0x153a8086 rev=0x05 hdr=0x00
vendor = 'Intel Corporation'
device = 'Ethernet Connection I217-LM'
class = network
subclass = ethernet
cap 01[c8] = powerspec 2 supports D0 D3 current D0
cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
cap 13[e0] = PCI Advanced Features: FLR TP
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=81209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
capabilities=953d9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP>
ether 2c:44:fd:37:be:14
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
supported media:
media autoselect
media 1000baseT
media 1000baseT mediaopt full-duplex
media 100baseTX mediaopt full-duplex
media 100baseTX
media 10baseT/UTP mediaopt full-duplex
media 10baseT/UTP
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan7: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=3<RXCSUM,TXCSUM>
capabilities=503<RXCSUM,TXCSUM,TSO4,LRO>
ether 2c:44:fd:37:be:14
inet 10.0.x.y netmask 0xfffff800 broadcast 10.0.y.z
groups: vlan
vlan: 798 vlanpcp: 0 parent interface: em0
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
supported media:
media autoselect
# sysctl dev.em.0
dev.em.0.wake: 1
dev.em.0.interrupts.rx_overrun: 0
dev.em.0.interrupts.rx_desc_min_thresh: 0
dev.em.0.interrupts.tx_queue_min_thresh: 0
dev.em.0.interrupts.tx_queue_empty: 0
dev.em.0.interrupts.tx_abs_timer: 0
dev.em.0.interrupts.tx_pkt_timer: 0
dev.em.0.interrupts.rx_abs_timer: 0
dev.em.0.interrupts.rx_pkt_timer: 0
dev.em.0.interrupts.asserts: 916405
dev.em.0.mac_stats.tso_ctx_fail: 0
dev.em.0.mac_stats.tso_txd: 0
dev.em.0.mac_stats.tx_frames_1024_1522: 0
dev.em.0.mac_stats.tx_frames_512_1023: 0
dev.em.0.mac_stats.tx_frames_256_511: 0
dev.em.0.mac_stats.tx_frames_128_255: 0
dev.em.0.mac_stats.tx_frames_65_127: 0
dev.em.0.mac_stats.tx_frames_64: 0
dev.em.0.mac_stats.mcast_pkts_txd: 293
dev.em.0.mac_stats.bcast_pkts_txd: 38
dev.em.0.mac_stats.good_pkts_txd: 17046
dev.em.0.mac_stats.total_pkts_txd: 17046
dev.em.0.mac_stats.good_octets_txd: 5112315
dev.em.0.mac_stats.good_octets_recvd: 119853581
dev.em.0.mac_stats.rx_frames_1024_1522: 0
dev.em.0.mac_stats.rx_frames_512_1023: 0
dev.em.0.mac_stats.rx_frames_256_511: 0
dev.em.0.mac_stats.rx_frames_128_255: 0
dev.em.0.mac_stats.rx_frames_65_127: 0
dev.em.0.mac_stats.rx_frames_64: 0
dev.em.0.mac_stats.mcast_pkts_recvd: 323244
dev.em.0.mac_stats.bcast_pkts_recvd: 402143
dev.em.0.mac_stats.good_pkts_recvd: 995757
dev.em.0.mac_stats.total_pkts_recvd: 995757
dev.em.0.mac_stats.xoff_txd: 0
dev.em.0.mac_stats.xoff_recvd: 0
dev.em.0.mac_stats.xon_txd: 0
dev.em.0.mac_stats.xon_recvd: 0
dev.em.0.mac_stats.coll_ext_errs: 0
dev.em.0.mac_stats.alignment_errs: 0
dev.em.0.mac_stats.crc_errs: 0
dev.em.0.mac_stats.recv_errs: 0
dev.em.0.mac_stats.recv_jabber: 0
dev.em.0.mac_stats.recv_oversize: 0
dev.em.0.mac_stats.recv_fragmented: 0
dev.em.0.mac_stats.recv_undersize: 0
dev.em.0.mac_stats.recv_no_buff: 0
dev.em.0.mac_stats.missed_packets: 0
dev.em.0.mac_stats.defer_count: 0
dev.em.0.mac_stats.sequence_errors: 0
dev.em.0.mac_stats.symbol_errors: 0
dev.em.0.mac_stats.collision_count: 0
dev.em.0.mac_stats.late_coll: 0
dev.em.0.mac_stats.multiple_coll: 0
dev.em.0.mac_stats.single_coll: 0
dev.em.0.mac_stats.excess_coll: 0
dev.em.0.queue_rx_0.rx_irq: 0
dev.em.0.queue_rx_0.rxd_tail: 122
dev.em.0.queue_rx_0.rxd_head: 124
dev.em.0.queue_tx_0.tx_irq: 0
dev.em.0.queue_tx_0.txd_tail: 61
dev.em.0.queue_tx_0.txd_head: 63
dev.em.0.fc_low_water: 20552
dev.em.0.fc_high_water: 23584
dev.em.0.rx_control: 100892734
dev.em.0.device_control: 1075315264
dev.em.0.watchdog_timeouts: 0
dev.em.0.rx_overruns: 0
dev.em.0.link_irq: 0
dev.em.0.dropped: 0
dev.em.0.eee_control: 1
dev.em.0.itr: 488
dev.em.0.tx_abs_int_delay: 66
dev.em.0.rx_abs_int_delay: 66
dev.em.0.tx_int_delay: 66
dev.em.0.rx_int_delay: 0
dev.em.0.rs_dump: 0
dev.em.0.reg_dump: General Registers
CTRL 40180240
STATUS 00080083
CTRL_EXIT 015a1006
Interrupt Registers
ICR 00000000
RX Registers
RCTL 0603803e
RDLEN 00010000
RDH 0000007c
RDT 0000007a
RXDCTL 00010000
RDBAL 035c2000
RDBAH 00000001
TX Registers
TCTL 3103f0fa
TDBAL 0359a000
TDBAH 00000001
TDLEN 00010000
TDH 00000050
TDT 00000050
TXDCTL 0341011f
TDFH 00000c46
TDFT 00000c46
TDFHS 00000c46
TDFPC 00000000
dev.em.0.fc: 3
dev.em.0.debug: -1
dev.em.0.nvm: -1
dev.em.0.iflib.rxq0.rxq_fl0.credits: 4095
dev.em.0.iflib.rxq0.rxq_fl0.cidx: 124
dev.em.0.iflib.rxq0.rxq_fl0.pidx: 123
dev.em.0.iflib.txq0.r_abdications: 0
dev.em.0.iflib.txq0.r_restarts: 0
dev.em.0.iflib.txq0.r_stalls: 0
dev.em.0.iflib.txq0.r_starts: 17094
dev.em.0.iflib.txq0.r_drops: 0
dev.em.0.iflib.txq0.r_enqueues: 17102
dev.em.0.iflib.txq0.ring_state: pidx_head: 0739 pidx_tail: 0739 cidx: 0739 state: IDLE
dev.em.0.iflib.txq0.txq_cleaned: 24629
dev.em.0.iflib.txq0.txq_processed: 24671
dev.em.0.iflib.txq0.txq_in_use: 42
dev.em.0.iflib.txq0.txq_cidx_processed: 97
dev.em.0.iflib.txq0.txq_cidx: 59
dev.em.0.iflib.txq0.txq_pidx: 101
dev.em.0.iflib.txq0.no_tx_dma_setup: 0
dev.em.0.iflib.txq0.txd_encap_efbig: 0
dev.em.0.iflib.txq0.tx_map_failed: 0
dev.em.0.iflib.txq0.no_desc_avail: 0
dev.em.0.iflib.txq0.mbuf_defrag_failed: 0
dev.em.0.iflib.txq0.m_pullups: 5
dev.em.0.iflib.txq0.mbuf_defrag: 0
dev.em.0.iflib.override_nrxds: 4096
dev.em.0.iflib.override_ntxds: 4096
dev.em.0.iflib.separate_txrx: 0
dev.em.0.iflib.core_offset: 0
dev.em.0.iflib.tx_abdicate: 0
dev.em.0.iflib.rx_budget: 0
dev.em.0.iflib.disable_msix: 1
dev.em.0.iflib.override_qs_enable: 0
dev.em.0.iflib.override_nrxqs: 0
dev.em.0.iflib.override_ntxqs: 0
dev.em.0.iflib.driver_version: 7.6.1-k
dev.em.0.%parent: pci0
dev.em.0.%pnpinfo: vendor=0x8086 device=0x153a subvendor=0x103c subdevice=0x1905 class=0x020000
dev.em.0.%location: slot=25 function=0 dbsf=pci0:0:25:0 handle=\_SB_.PCI0.GLAN
dev.em.0.%driver: em
dev.em.0.%desc: Intel(R) PRO/1000 Network Connection
I should have written "Bringing the NIC down and up solves the issue"; to clarify: I have not tested whether the NIC is able to transmit untagged frames. Is anyone else seeing the problem that Marek is seeing? I don't have any 1G devices, so I can't try to reproduce this. (In reply to Eric Joyner from comment #12) This could be I217-LM (em) specific or different flaw which came out after upgrade. Other 1G NICs like I219-LM (em) and I350 (igb) are initialising fine on 12-STABLE after this MFC. Thanks to Mr Marek Zarychta here who has pointed me to this report, i can share that i have similar problems, related to r360902 I have a intel 82574L based network card who act as a link to my ISP, who is gaving me dhcp based authentication through a vlan after r360902 , the connectivity to my ISP can't be restored after a reboot or a power-on until, as Mr Marek Zarychta pointed i "Bring the NIC down and up " by the ifconfig(8) command and NOT by unplugging /replugging the ethernet cable back. I'll open a proprer bug report if needed to; meanwhile, i'll test the patch provided, in case of some interesting results Regards. (In reply to martin.mato from comment #14) Well, i compiled successfully a new kernel, and it is working perfectly because the working generated if_em.c file has the same content than the revision pre-r360902, right? (In reply to martin.mato from comment #14) Same situation here. After replacing isp router, no success obviously, the box ended relocated in a hurry on a data center that, just by chance, doesn't need vlan tagging to operate. I sadly didn't try down/up, but unplug/plug cable didn't help. em0@pci0:0:25:0: class=0x020000 card=0x20448086 chip=0x15038086 rev=0x04 hdr=0x00 vendor = 'Intel Corporation' device = '82579V Gigabit Network Connection' class = network subclass = ethernet cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message cap 13[e0] = PCI Advanced Features: FLR TP (In reply to Raúl from comment #16) No, unfortunately: Unplugging/repluging the cable doesn't work, like i said. The only way is to manually reset by ifconfig(8) down and up the parent interface em0 , not the vlan interface(s). In the @current list ; Ian Freislich provided a patch who worked for him i tested succesfully as well on my box here it is Index: sys/dev/e1000/if_em.c =================================================================== --- sys/dev/e1000/if_em.c (revision 361538) +++ sys/dev/e1000/if_em.c (working copy) @@ -4054,7 +4054,7 @@ { switch (event) { case IFLIB_RESTART_VLAN_CONFIG: - return (false); + return (true); default: return (true); } (In reply to martin.mato from comment #17) I'll look how to test it on site before returning it back home. Given covid-19 circumstances logistics are ... sort of worrisome. Thanks for the info. I'll change em(4) so that it always does the interface down/up on VLAN config/unconfig, but leave igb(4) as-is. It seems like em(4) needs more investigation on what needs to be done to prevent a link flap. A commit references this bug: Author: erj Date: Thu Jun 11 15:59:49 UTC 2020 New revision: 362063 URL: https://svnweb.freebsd.org/changeset/base/362063 Log: em(4): Always reinit interface when adding/removing VLAN This partially reverts r361053 since there have been reports by users that this breaks some functionality for em(4) devices; it seems at first glance that some sort of interface restart is required for those cards. This isn't a proper fix; this unbreaks those users until a proper fix is found for their issues. PR: 240818 Reported by: Marek Zarychta <zarychtam@plan-b.pwste.edu.pl> MFC after: 3 days Changes: head/sys/dev/e1000/if_em.c (In reply to commit-hook from comment #20) Compiled a kernel and tested immediately upon receipt. So far, it's working very well, at least for me. A commit references this bug: Author: erj Date: Thu Jul 30 18:31:25 UTC 2020 New revision: 363711 URL: https://svnweb.freebsd.org/changeset/base/363711 Log: MFC r362063: em(4): Always reinit interface when adding/removing VLAN PR: 240818 Sponsored by: Intel Corporation Changes: _U stable/12/ stable/12/sys/dev/e1000/if_em.c (In reply to commit-hook from comment #22) Why is it closed as "FIXED"? Re-init interface on each VLAN setting change is only work-around, as old pre-iflib driver was able to re-confgure VLANs without interface up/down cycle! See comment 23 I have confirmed this behaviour on RELENG_12 (r367411)
e.g
ifconfig vlan6 create 10.6.6.6/28 vlandev igb1 vlan 6
ifconfig vlan7 create 10.7.7.7/28 vlandev igb1 vlan 7
# dmesg | tail -19
vlan5: promiscuous mode enabled
igb1: promiscuous mode disabled
vlan5: promiscuous mode disabled
igb1: promiscuous mode enabled
igb1: promiscuous mode disabled
igb1: link state changed to DOWN
vlan5: link state changed to DOWN
vlan6: link state changed to DOWN
igb1: link state changed to UP
vlan5: link state changed to UP
vlan6: link state changed to UP
igb1: link state changed to DOWN
vlan5: link state changed to DOWN
vlan6: link state changed to DOWN
vlan7: link state changed to DOWN
igb1: link state changed to UP
vlan5: link state changed to UP
vlan6: link state changed to UP
vlan7: link state changed to UP
ie. with each vlan created, igb1 and all the VLANS transition with a full carrier down/up (confirmed from the switch they are connected to)
Confirmed on 3 different boards
Supermicro X11SSH-F
igb1@pci0:3:0:0: class=0x020000 card=0x153315d9 chip=0x15338086 rev=0x03 hdr=0x00
vendor = 'Intel Corporation'
device = 'I210 Gigabit Network Connection'
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base rxdf200000, size 524288, enabled
bar [18] = type I/O Port, range 32, base rxd000, size 32, enabled
bar [1c] = type Memory, range 32, base rxdf280000, size 16384, enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 11[70] = MSI-X supports 5 messages, enabled
Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR NS
max read 512
link x1(x1) speed 2.5(2.5) ASPM L1(L0s/L1)
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0003[140] = Serial 1 ac1f6bffff77ce57
ecap 0017[1a0] = TPH Requester 1
ASRockRack X470D4U
# pciconf -lcvb igb1
igb1@pci0:36:0:0: class=0x020000 card=0x15331849 chip=0x15338086 rev=0x03 hdr=0x00
vendor = 'Intel Corporation'
device = 'I210 Gigabit Network Connection'
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base rxf7300000, size 524288, enabled
bar [18] = type I/O Port, range 32, base rxd000, size 32, enabled
bar [1c] = type Memory, range 32, base rxf7380000, size 16384, enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 11[70] = MSI-X supports 5 messages, enabled
Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
max read 512
link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0003[140] = Serial 1 7085c2fffffde3e8
ecap 0017[1a0] = TPH Requester 1
Supermicro SYS-5019C-M
# pciconf -lvcb igb1
igb1@pci0:4:0:0: class=0x020000 card=0x153315d9 chip=0x15338086 rev=0x03 hdr=0x00
vendor = 'Intel Corporation'
device = 'I210 Gigabit Network Connection'
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base rx93200000, size 524288, enabled
bar [18] = type I/O Port, range 32, base rx4000, size 32, enabled
bar [1c] = type Memory, range 32, base rx93280000, size 16384, enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 11[70] = MSI-X supports 5 messages, enabled
Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
max read 512
link x1(x1) speed 2.5(2.5) ASPM L1(L0s/L1)
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0003[140] = Serial 1 3cecefffff6d99ab
ecap 0017[1a0] = TPH Requester 1
Probably fixed in https://cgit.freebsd.org/src/commit/?id=9c6432dc4bb936f0842d766d8b3b19dfcde15da2 See bug 230996. ^Triage: the 12 branch is now out of support. This was probably fixed in 2021. |