Bug 236102 - When create or destroy vlan, the physical interface is flapping
Summary: When create or destroy vlan, the physical interface is flapping
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.0-STABLE
Hardware: amd64 Any
: --- Affects Many People
Assignee: freebsd-net mailing list
URL:
Keywords: IntelNetworking, regression
Depends on:
Blocks:
 
Reported: 2019-02-28 10:09 UTC by Alexey
Modified: 2019-03-01 11:42 UTC (History)
5 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Alexey 2019-02-28 10:09:48 UTC
I add some configuretion on rc.conf

[root@nas1 /home]# cat /etc/rc.conf |egrep "vlan111|ix0"
cloned_interfaces="vlan111 vlan780"
#ifconfig_ix0="up -tso4 -tso6 -lro -vlanhwtso description -=INTERFACE-TO-HERMES=-"
ifconfig_ix0="up -tso4 -tso6 -lro -vlanhwtso description -=INTERFACE-TO-HERMES=-"
ifconfig_vlan111="inet 192.168.0.8 netmask 255.255.255.224 vlan 111 vlandev ix0 description -=NAS-WORLD=-"
[root@nas1 /home]#

then do some commands:

ifconfig vlan111 create

Feb 28 12:05:34 nas1 kernel: ix0: link state changed to DOWN
Feb 28 12:05:34 nas1 kernel: vlan111: link state changed to DOWN
Feb 28 12:05:35 nas1 ntpd[1049]: Listen normally on 8 vlan111 192.168.0.8:123
Feb 28 12:05:36 nas1 kernel: ix0: link state changed to UP
Feb 28 12:05:36 nas1 kernel: vlan111: link state changed to UP

after this physical interface is flapping
when I doo:

ifconfig vlan111 destroy
physical interface also is flapping

Feb 28 12:05:57 nas1 kernel: ix0: link state changed to DOWN
Feb 28 12:05:58 nas1 ntpd[1049]: Deleting interface #8 vlan111, 192.168.0.8#123, interface stats: received=0, sent=0, dropped=0, active_time=23 secs
Feb 28 12:06:00 nas1 kernel: ix0: link state changed to UP
Comment 1 Alexey 2019-02-28 15:47:00 UTC
FreeBSD nas1 12.0-STABLE FreeBSD 12.0-STABLE r344575 NAS  amd64
Comment 2 Eugene Grosbein freebsd_committer 2019-02-28 15:55:34 UTC
Does it help if you do "ifconfig ix0 -vlanhwtag -vlanhwfilter" before creating vlan?

Some NICs chips re-establish link when host reprogramms their in-chip vlan mask. If you disable hardware-assisted vlan filtering to perform it at OS driver level, you shoud stop reprogramming chip while creating/destroying vlans.
Comment 3 Alexey 2019-02-28 16:01:49 UTC
Yes it helped.
Comment 4 Alexey 2019-02-28 16:06:17 UTC
Just a month ago, this did not happen.
Comment 5 Eugene Grosbein freebsd_committer 2019-02-28 16:37:47 UTC
(In reply to Alexey from comment #4)

What revision do you run as per "uname -a" ?
Comment 6 Eugene Grosbein freebsd_committer 2019-02-28 16:38:54 UTC
Sorry, missed that you already mentioned r344575. Can you add revision you used "month ago" that had not this problem?
Comment 7 Alexey 2019-03-01 08:48:42 UTC
I have retest on:
FreeBSD nas1 12.0-STABLE FreeBSD 12.0-STABLE r343868 NAS  amd64
and this behavior is also present on this revision.

This is a other server, with the same hardware configuration. There is no such problem on it.

FreeBSD nas2 12.0-STABLE FreeBSD 12.0-STABLE r341831 NAS  amd64

ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: -=INTERFACE-TO-CORE2-SWITCH=-
        options=e000bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:e0:ed:2e:14:e1
        media: Ethernet autoselect (10Gbase-Twinax <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Comment 8 Eugene Grosbein freebsd_committer 2019-03-01 09:55:02 UTC
It seems, the code of ixgbe(4) driver got not significant changes in stable/12 between revisions 341831 and 343868 but iflib did. CC'ing marius@ who might have a glue.

Alexey, please show part of "pciconf -lvvv" output for your ix0 device.
Comment 9 Eugene Grosbein freebsd_committer 2019-03-01 10:08:41 UTC
if_vlan(4) driver has changed in stable/12 between revisions 341831 and 343868, so CC'ing markj@ who changed it to give him a chance to look at this.
Comment 10 Alexey 2019-03-01 10:17:37 UTC
ix0@pci0:2:0:0: class=0x020000 card=0x17d3103c chip=0x10fb8086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82599ES 10-Gigabit SFI/SFP+ Network Connection'
    class      = network
    subclass   = ethernet
ix1@pci0:2:0:1: class=0x020000 card=0x17d3103c chip=0x10fb8086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82599ES 10-Gigabit SFI/SFP+ Network Connection'
    class      = network
    subclass   = ethernet
Comment 12 Alexey 2019-03-01 11:42:51 UTC
I rolled back on the current server to revision r341831 and it did not help, then I remembered that on the second server I used the drivers installed from the ports (/usr/ports/net/intel-ix-kmod/).
Now when using these drivers - there are no problems and the physical interface does not flap.
FreeBSD nas1 12.0-STABLE FreeBSD 12.0-STABLE r341831 NAS  amd64

Mar  1 13:38:29 nas1 ntpd[1127]: Listen normally on 8 vlan111 192.168.0.8:123
Mar  1 13:38:32 nas1 ntpd[1127]: Deleting interface #8 vlan111, 192.168.0.8#123, interface stats: received=0, sent=0, dropped=0, active_time=3 secs

[root@nas1 /home/pautina]#  cat /etc/rc.conf |egrep "vlan111|ix1"
cloned_interfaces="vlan111 vlan780"
ifconfig_ix1="up -tso4 -tso6 -lro -vlanhwtso description -=INTERFACE-TO-APHRODITE="
ifconfig_vlan111="inet 192.168.0.8 netmask 255.255.255.224 vlan 111 vlandev ix1 description -=NAS-WORLD=-"
[root@nas1 /home/pautina]#

[root@nas1 /home/pautina]# ifconfig vlan111
vlan111: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: -=NAS-WORLD=-
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        ether e4:11:5b:9b:72:b4
        inet 192.168.0.8 netmask 0xffffffe0 broadcast 192.168.0.31
        groups: vlan
        vlan: 111 vlanpcp: 0 parent interface: ix1
        media: Ethernet autoselect (10Gbase-Twinax <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
[root@nas1 /home/pautina]#

[root@nas1 /home/pautina]# ifconfig ix1
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: -=INTERFACE-TO-APHRODITE=
        options=e000bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        ether e4:11:5b:9b:72:b4
        media: Ethernet autoselect (10Gbase-Twinax <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
[root@nas1 /home/pautina]#

[root@nas1 /home/pautina]# grep if_ix_updated /boot/loader.conf
if_ix_updated_load="YES"
[root@nas1 /home/pautina]#

[root@nas1 /home/pautina]# kldstat
Id Refs Address                Size Name
 1    9 0xffffffff80200000  12f04e0 kernel
 2    1 0xffffffff814f1000    562f0 if_ix_updated.ko
 3    1 0xffffffff81621000      acf mac_ntpd.ko
[root@nas1 /home/pautina]#

[root@nas1 /home/pautina]# dmesg |grep ix1
ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.3.6> port 0xe000-0xe01f mem 0xdf980000-0xdf9fffff,0xdfb00000-0xdfb03fff irq 17 at device 0.1 on pci2
ix1: Using MSI-X interrupts with 5 vectors
ix1: Ethernet address: e4:11:5b:9b:72:b4
ix1: PCI Express Bus: Speed 5.0GT/s Width x8
ix1: link state changed to UP
[root@nas1 /home/pautina]#

ix0@pci0:2:0:0: class=0x020000 card=0x17d3103c chip=0x10fb8086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82599ES 10-Gigabit SFI/SFP+ Network Connection'
    class      = network
    subclass   = ethernet
ix1@pci0:2:0:1: class=0x020000 card=0x17d3103c chip=0x10fb8086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82599ES 10-Gigabit SFI/SFP+ Network Connection'
    class      = network
    subclass   = ethernet