Bug 229432

Summary: 11.2 network em driver - connection link flaps with dhclient and option 26 (interface mtu)
Product: Base System Reporter: Dave Deaven <deaven>
Component: kernAssignee: Eugene Grosbein <eugen>
Status: Closed FIXED    
Severity: Affects Many People CC: cem, eugen, franco, freebsd, mmpestorich, net, sbruno, steven_nikkel, wout
Priority: --- Keywords: IntelNetworking, regression
Version: 11.2-RELEASEFlags: koobs: mfc-stable11+
Hardware: Any   
OS: Any   
Attachments:
Description Flags
dmesg output for (working) 11.1 system
none
proposed fix for dhclient none

Description Dave Deaven 2018-06-30 17:05:23 UTC
Created attachment 194795 [details]
dmesg output for (working) 11.1 system

Seemingly uneventful upgrade from 11.1-RELEASE-p11 to 11.2-RELEASE-p0, resulted in constant link state changes running the new kernel:

Jun 30 11:00:06 hellcat kernel: em1: link state changed to DOWN
Jun 30 11:00:06 hellcat dhclient: New IP Address (em1): 24.211.23.77
Jun 30 11:00:06 hellcat dhclient: New Subnet Mask (em1): 255.255.240.0
Jun 30 11:00:06 hellcat dhclient: New Broadcast Address (em1): 255.255.255.255
Jun 30 11:00:06 hellcat dhclient: New Routers (em1): 24.211.16.1
Jun 30 11:00:10 hellcat kernel: em1: link state changed to UP
Jun 30 11:00:10 hellcat kernel: em1: link state changed to DOWN
Jun 30 11:00:10 hellcat dhclient: New IP Address (em1): 24.211.23.77
Jun 30 11:00:10 hellcat dhclient: New Subnet Mask (em1): 255.255.240.0
Jun 30 11:00:10 hellcat dhclient: New Broadcast Address (em1): 255.255.255.255
Jun 30 11:00:10 hellcat dhclient: New Routers (em1): 24.211.16.1
Jun 30 11:00:14 hellcat kernel: em1: link state changed to UP
Jun 30 11:00:14 hellcat kernel: em1: link state changed to DOWN

The connection is the WAN side of a FreeBSD router, connected to a Motorola SURFboard SB6141 DOCSIS 3.0 Cable Modem. During this activity, ISP connection up/down links are stable, only ethernet link is changing. It stays up for few hundred milliseconds, enough to route some traffic but obviously frequent dropouts.

New ethernet cable, various warm and cold reboots of modem and FreeBSD box, turning off pf service all made no difference.  Also confirmed running correct 11.2 config using "freebsd-update IDS".

Rolling back to 11.1, immediately works again.  This system has historically experienced link changes (which I assumed were associated with ISP issues) about once per 3 months, link goes down and comes back up within a minute. 

Here's the em part of kernel message during 11.1 boot:
em1: <Intel(R) PRO/1000 Network Connection 7.6.1-k> port 0x1000-0x101f mem 0x80020000-0x8003ffff,0x80000000-0x8001ffff,0x80040000-0x80043fff irq 17 at device 0.0 on pci2
em1: Using MSIX interrupts with 3 vectors
em1: Ethernet address: 00:22:4d:ac:a6:f0

I've attached the full dmesg output as well.
Comment 1 Dave Deaven 2018-06-30 17:15:23 UTC
The output of ifconfig for working 11.1:

# ifconfig em1
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
	ether 00:22:4d:ac:a6:f0
	hwaddr 00:22:4d:ac:a6:f0
	inet6 fe80::222:4dff:feac:a6f0%em1 prefixlen 64 scopeid 0x2 
	inet 24.211.23.77 netmask 0xfffff000 broadcast 255.255.255.255 
	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
Comment 2 Chris Ploski 2018-07-01 04:56:01 UTC
I'm seeing the same behavior as you - em0 goes up, gets IP over DHCP, then ends up in an endless loop of going down for a few seconds then coming back up. I've also got a similar setup where the WAN side is connected to a Motorola/Arris Surfboard 6183 cable modem.

In any case it appears the issue is with dhclient on FreeBSD 11.2 when using em interfaces. If you kill dhclient immediately after it gets the IP address then the link flapping stops. The workaround for me was to copy the dhclient binary from an 11.1 install.
Comment 3 Eugene Grosbein freebsd_committer 2018-07-01 05:38:16 UTC
This is known problem with recent changes in our dhclient and its combination with some DHCP servers and em(4) NICs that drop link when driver setups MTU on the interface. And dhclient now may setup MTU on the interface if DHCP server offers DHCP option 26 (it did not before).

As workaround, you can try to add to /etc/dhclient.conf:

interface "em0" {
  supersede interface-mtu 0;
}

This works for me as 0 is special value meaning that dhclient should not even try to change interface MTU, as it did before.
Comment 4 Eugene Grosbein freebsd_committer 2018-07-01 06:44:19 UTC
Created attachment 194808 [details]
proposed fix for dhclient

Here is proposed patch that makes dhclient verify if new MTU differs from current one and skip unneeded MTU change. This check eliminates infinite loop of MTU change / link flap / lease verification / MTU change / link flap etc.
Comment 5 wout 2018-07-10 10:28:46 UTC
I am seeing the same behavior on a PC Engines board on its igb0 interface since I upgraded to 11.2-RELEASE.

The above workaround in /etc/dhclient.conf fixed the issue.
Comment 6 Eugene Grosbein freebsd_committer 2018-07-10 14:51:51 UTC
I will commit the fix soon unless an objection is raised.
Comment 7 commit-hook freebsd_committer 2018-07-11 09:41:58 UTC
A commit references this bug:

Author: eugen
Date: Wed Jul 11 09:41:50 UTC 2018
New revision: 336195
URL: https://svnweb.freebsd.org/changeset/base/336195

Log:
  Make dhclient(8) verify if new MTU (option 26) differs from current one and skip unneeded MTU change.
  This check eliminates infinite loop of MTU change / link flap / lease verification / MTU change / link flap etc.
  in case of some NIC drivers like em(4) or igb(4).

  N.B.: obsolete u_int16_t is used in consistency with the rest of the file.

  PR:		229432
  Approved by:	mav (mentor)
  MFC after:	1 week

Changes:
  head/sbin/dhclient/dispatch.c
Comment 8 commit-hook freebsd_committer 2018-07-18 01:57:48 UTC
A commit references this bug:

Author: eugen
Date: Wed Jul 18 01:57:13 UTC 2018
New revision: 336440
URL: https://svnweb.freebsd.org/changeset/base/336440

Log:
  MFC r336195: unbreak dhclient(8) option 26 processing

    Make dhclient(8) verify if new MTU (option 26) differs from current one
    and skip unneeded MTU change. This check eliminates infinite loop
    of MTU change / link flap / lease verification / MTU change / link flap etc.
    in case of some NIC drivers like em(4) or igb(4).

  PR:		229432
  Approved by:	mav (mentor)

Changes:
_U  stable/11/
  stable/11/sbin/dhclient/dispatch.c
Comment 9 Kubilay Kocak freebsd_committer freebsd_triage 2018-10-20 07:18:21 UTC
*** Bug 231166 has been marked as a duplicate of this bug. ***