Bug 200221 - em0 watchdog timeout under load
Summary: em0 watchdog timeout under load
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 10.1-RELEASE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: Sean Bruno
URL: https://reviews.freebsd.org/D3192
Keywords: IntelNetworking, needs-qa, patch
Depends on:
Blocks:
 
Reported: 2015-05-15 10:46 UTC by anthony
Modified: 2017-04-14 11:08 UTC (History)
10 users (show)

See Also:
koobs: mfc-stable10+
koobs: mfc-stable9?


Attachments
dmesg.boot and pciconf -lv outputs for machine1 (Core 2 Quad) (54.47 KB, text/plain)
2015-05-15 10:46 UTC, anthony
no flags Details
dmesg.boot and pciconf -lv outputs for machine2 (2x Xeon) (20.64 KB, text/plain)
2015-05-15 10:47 UTC, anthony
no flags Details
svn di svn://svn0.eu.freebsd.org/base/stable/10/sys/dev/e1000 sys/dev/e1000 (9.64 KB, text/plain)
2015-09-02 19:14 UTC, anthony
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description anthony 2015-05-15 10:46:58 UTC
Created attachment 156796 [details]
dmesg.boot and pciconf -lv outputs for machine1 (Core 2 Quad)

I have two machines which are often experiencing watchdog timeouts on their ethernet interfaces when under load (machine 1 doing a nightly backup over NFS, machine 2 transferring large files periodically, also over NFS). Both use the em0 driver, with the machine doing nightly backups having a timeout 4-5 nights a week. It will typically reset multiple times over a period of 30 minutes while the backup runs.

Machine 1 is a consumer motherboard with an Intel Core 2 Quad Q6600 CPU.
Machine 2 is a server motherboard with 2 x Intel Xeon E5335 CPU's. It has two NICs, but only em0 is connected.


Full dmesg and pciconf outputs attached.

===dmesg for machine 1===

grep em0 /var/run/dmesg.boot 
em0: <Intel(R) PRO/1000 Network Connection 7.4.2> port 0x10c0-0x10df mem 0xe0100000-0xe011ffff,0xe0124000-0xe0124fff irq 20 at device 25.0 on pci0
em0: attempting to allocate 1 MSI vectors (1 supported)
em0: using IRQ 256 for MSI
em0: Using an MSI interrupt
em0: bpf attached
em0: Ethernet address: 00:1c:c0:08:c7:99
em0: Link is up 1000 Mbps Full Duplex

===pciconf for machine 1====

em0@pci0:0:25:0:        class=0x020000 card=0x00018086 chip=0x104a8086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82566DM Gigabit Network Connection'
    class      = network
    subclass   = ethernet

===Snip of messages for machine 1====
<snip> - large number of 'newnfs server X not responding'
May 15 07:51:27 urybsod kernel: em0: Watchdog timeout -- resetting
May 15 07:51:27 urybsod kernel: em0: Queue(0) tdh = 574, hw tdt = 538
May 15 07:51:27 urybsod kernel: em0: TX(0) desc avail = 31,Next TX to Clean = 569
May 15 07:51:27 urybsod kernel: em0: Link is Down
May 15 07:51:27 urybsod kernel: em0: link state changed to DOWN
May 15 07:51:30 urybsod kernel: em0: Link is up 1000 Mbps Full Duplex
May 15 07:51:30 urybsod kernel: em0: link state changed to UP
May 15 07:51:30 urybsod devd: Executing '/etc/rc.d/dhclient quietstart em0'
May 15 07:51:30 urybsod sshd[65879]: fatal: Read from socket failed: Connection reset by peer [preauth]
May 15 07:51:38 urybsod kernel: newnfs server X is alive again
May 15 07:51:39 urybsod last message repeated 19 times

 uname -a
FreeBSD urybsod 10.1-RELEASE-p9 FreeBSD 10.1-RELEASE-p9 #0: Tue Apr  7 01:09:46 UTC 2015     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

===dmesg for machine 2===

em0: <Intel(R) PRO/1000 Network Connection 7.4.2> port 0x4020-0x403f mem 0xb8820000-0xb883ffff,0xb8400000-0xb87fffff irq 18 at device 0.0 on pci5
em0: Using an MSI interrupt
em0: Ethernet address: 00:04:23:dd:37:cc
em1: <Intel(R) PRO/1000 Network Connection 7.4.2> port 0x4000-0x401f mem 0xb8800000-0xb881ffff,0xb8000000-0xb83fffff irq 19 at device 0.1 on pci5
em1: Using an MSI interrupt
em1: Ethernet address: 00:04:23:dd:37:cd


===pciconf -lv for machine 2===

em0@pci0:5:0:0: class=0x020000 card=0x346c8086 chip=0x10968086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '80003ES2LAN Gigabit Ethernet Controller (Copper)'
    class      = network
    subclass   = ethernet
em1@pci0:5:0:1: class=0x020000 card=0x346c8086 chip=0x10968086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '80003ES2LAN Gigabit Ethernet Controller (Copper)'
    class      = network
    subclass   = ethernet


===messages for machine 2===

<snip> - large number of 'newnfs server Y not responding'
May  4 02:45:40 urystv kernel: newnfs server Y: not responding
May  4 02:48:43 urystv kernel: em0: Watchdog timeout -- resetting
May  4 02:48:43 urystv kernel: em0: Queue(0) tdh = 294, hw tdt = 898
May  4 02:48:43 urystv kernel: em0: TX(0) desc avail = 411,Next TX to Clean = 285
May  4 02:48:43 urystv kernel: em0: link state changed to DOWN
May  4 02:48:46 urystv kernel: em0: link state changed to UP
May  4 02:48:46 urystv devd: Executing '/etc/rc.d/dhclient quietstart em0'
May  4 02:49:08 urystv kernel: newnfs server Y: is alive again
May  4 02:49:09 urystv last message repeated 19 times

uname -a
FreeBSD urystv 10.1-RELEASE-p9 FreeBSD 10.1-RELEASE-p9 #0: Tue Apr  7 01:09:46 UTC 2015     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
Comment 1 anthony 2015-05-15 10:47:36 UTC
Created attachment 156797 [details]
dmesg.boot and pciconf -lv outputs for machine2 (2x Xeon)
Comment 2 anthony 2015-05-15 10:55:06 UTC
(In reply to anthony from comment #0)
I should have added - neither machine experienced these problems until they were upgraded to 10.1-RELEASE.
Comment 3 Alexandr Krivulya 2015-05-15 11:35:02 UTC
Can you try to disable tso4 on em interfaces?
Comment 4 anthony 2015-05-17 10:49:37 UTC
Disabled tso4 on em0, not had any problems so far over two backup runs. I'll keep monitoring it, as it doesn't happen every time.
Comment 5 anthony 2015-05-23 12:48:23 UTC
It's now been a week, disabling tso4 does seem to have resolved this. What could be the reason for this now being required? As mentioned, I don't recall this happening before it was upgraded to 10.1-RELEASE (previously 10.0-RELEASE).
Comment 6 Alexandr Krivulya 2015-05-26 06:52:24 UTC
I think it's some bug due to em(4) upgrade in 10.1

https://svnweb.freebsd.org/base?view=revision&revision=269196
Comment 7 anthony 2015-06-03 12:40:43 UTC
I've now built a 10.1-RELEASE-p10 kernel with sys/dev/e1000 rolled back to before 269196. tso4 is enabled, I'll test this out over a few days and see what happens.
Comment 8 anthony 2015-06-19 08:40:04 UTC
Now had two weeks under the 10.1-RELEASE-p10 kernel with e1000 rolled back to 269196 and I've not seen any timeouts. I'll try moving e1000 forwards to try and find the problem commit that is the cause.
Comment 9 Alexandr Krivulya 2015-06-19 12:11:58 UTC
The are some updates of em(4) driver in CURRENT, which are already MFC'd to 10-STABLE. It has some fixes, which may be related to your issue. So you can try it.

https://svnweb.freebsd.org/base/head/sys/dev/e1000/if_em.c?revision=283959&view=markup&sortby=date
Comment 10 anthony 2015-07-15 12:21:50 UTC
Now using sys/dev/e1000@283959 from 10-STABLE (rest of the kernel 10.1-RELEASE as before), I'm seeing watchdog timeouts again, with tso4 enabled. Again, this only seems to happen at a point of high network load (nightly backup over NFS). I'll try disabling tso4 on this revision but failing that, will go back to e1000@269196.

There is slightly more detail in messages now, though: 

Jul  8 07:31:58 urybsod kernel: em0: Watchdog timeout Queue[0]-- resetting
Jul  8 07:31:58 urybsod kernel: Interface is RUNNING and ACTIVE
Jul  8 07:31:58 urybsod kernel: em0: TX Queue 0 ------
Jul  8 07:31:58 urybsod kernel: em0: hw tdh = 909, hw tdt = 1015
Jul  8 07:31:58 urybsod kernel: em0: Tx Queue Status = -2147483648
Jul  8 07:31:58 urybsod kernel: em0: TX descriptors avail = 914
Jul  8 07:31:58 urybsod kernel: em0: Tx Descriptors avail failure = 114
Jul  8 07:31:58 urybsod kernel: em0: RX Queue 0 ------
Jul  8 07:31:58 urybsod kernel: em0: hw rdh = 959, hw rdt = 958
Jul  8 07:31:58 urybsod kernel: em0: RX discarded packets = 0
Jul  8 07:31:58 urybsod kernel: em0: RX Next to Check = 959
Jul  8 07:31:58 urybsod kernel: em0: RX Next to Refresh = 958
Jul  8 07:31:58 urybsod kernel: em0: Link is Down
Jul  8 07:31:58 urybsod kernel: em0: link state changed to DOWN
Jul  8 07:32:01 urybsod kernel: em0: Link is up 1000 Mbps Full Duplex
Jul  8 07:32:01 urybsod kernel: em0: link state changed to UP
Jul  8 07:32:01 urybsod devd: Executing '/etc/rc.d/dhclient quietstart em0'
Comment 11 Sean Bruno freebsd_committer 2015-07-15 16:19:23 UTC
Can you try the following patch to if_em.c and see if it helps?  I'm trying to diagnose this here.

Index: dev/e1000/if_em.c
===================================================================
--- dev/e1000/if_em.c	(revision 285481)
+++ dev/e1000/if_em.c	(working copy)
@@ -3034,6 +3034,9 @@
 	if_setflags(ifp, IFF_BROADCAST | IFF_SIMPLEX | IFF_MULTICAST);
 	if_setioctlfn(ifp, em_ioctl);
 	if_setgetcounterfn(ifp, em_get_counter);
+	ifp->if_hw_tsomax = 65518; /* 32*MCLBYTES - max_mac_hdr_len*/
+	ifp->if_hw_tsomaxsegcount = EM_MAX_SCATTER;
+	ifp->if_hw_tsomaxsegsize = 65536;
 #ifdef EM_MULTIQUEUE
 	/* Multiqueue stack interface */
 	if_settransmitfn(ifp, em_mq_start);
Comment 12 anthony 2015-07-21 22:54:49 UTC
Had to make a couple of changes to get that patch into a 10.1-RELEASE kernel:

-- Brought sys/dev/e1000 up to 10-STABLE (as previous)
-- Merged r274043 and r274704 from 10-STABLE

Unfortunately, it hasn't resolved the issue - see below. TSO4 is enabled.

Jul 21 07:36:56 urybsod kernel: em0: Watchdog timeout Queue[0]-- resetting
Jul 21 07:36:56 urybsod kernel: Interface is RUNNING and ACTIVE
Jul 21 07:36:56 urybsod kernel: em0: TX Queue 0 ------
Jul 21 07:36:56 urybsod kernel: em0: hw tdh = 271, hw tdt = 814
Jul 21 07:36:56 urybsod kernel: em0: Tx Queue Status = -2147483648
Jul 21 07:36:56 urybsod kernel: em0: TX descriptors avail = 477
Jul 21 07:36:56 urybsod kernel: em0: Tx Descriptors avail failure = 0
Jul 21 07:36:56 urybsod kernel: em0: RX Queue 0 ------
Jul 21 07:36:56 urybsod kernel: em0: hw rdh = 9, hw rdt = 8
Jul 21 07:36:56 urybsod kernel: em0: RX discarded packets = 0
Jul 21 07:36:56 urybsod kernel: em0: RX Next to Check = 9
Jul 21 07:36:56 urybsod kernel: em0: RX Next to Refresh = 8
Jul 21 07:36:56 urybsod kernel: em0: Link is Down
Jul 21 07:36:56 urybsod kernel: em0: link state changed to DOWN
Jul 21 07:37:00 urybsod kernel: em0: Link is up 1000 Mbps Full Duplex
Jul 21 07:37:00 urybsod kernel: em0: link state changed to UP
Jul 21 07:37:00 urybsod devd: Executing '/etc/rc.d/dhclient quietstart em0'
Comment 13 Sean Bruno freebsd_committer 2015-08-03 16:44:05 UTC
(In reply to anthony from comment #12)
Ok, I think I've got this bug cornered.  Can you test the patch at https://reviews.freebsd.org/D3192 to see if it fixes your testcase?
Comment 14 commit-hook freebsd_committer 2015-08-16 19:44:24 UTC
A commit references this bug:

Author: sbruno
Date: Sun Aug 16 19:43:45 UTC 2015
New revision: 286831
URL: https://svnweb.freebsd.org/changeset/base/286831

Log:
  Increase EM_MAX_SCATTER to 64 such that the size of em_xmit()::segs[EM_MAX_SCATTER]
  doesn't get overrun by things like NFS that can and do shove more than 32 segs when
  being used with em(4) and TSO4.

  Update tso handling code in em_xmit() with update from jhb@ in email thread:
  https://lists.freebsd.org/pipermail/freebsd-net/2014-July/039306.html

  set ifp->if_hw_tsomax, ifp->if_hw_tsomaxsegcount & ifp->if_hw_tsomaxsegsize
  to appropriate values.

  Define a TSO workaround "magic" number  of 4 that is used to avoid an
  alignment issue in hardware.

  Change a couple of integer values that were used as booleans to actual
  bool types.

  Ensure that em_enable_intr() enables the appropriate mask of interrupts
  and not just a hardcoded define of values.

  PR:		200221 199174 195078
  Differential Revision:	https://reviews.freebsd.org/D3192
  Reviewed by:	erj jhb hiren
  MFC after:	2 weeks
  Sponsored by:	Limelight Networks

Changes:
  head/sys/dev/e1000/if_em.c
  head/sys/dev/e1000/if_em.h
Comment 15 anthony 2015-08-30 00:05:03 UTC
Apologies for the delay in updating - I had to wait for a period where I could reboot the server into a new kernel.

I've patched CURRENT@286381 (and also CURRENT@285879) onto the changes from 10.1-RELEASE as described in Comment 12. After about a week in use, I've seen no watchdog timeout issues, so this patch seems to have solved the issue - thanks! I'll now try the patch onto the other server which uses the e1000 driver - although that one occurs rarely.
Comment 16 Sean Bruno freebsd_committer 2015-09-02 16:33:58 UTC
You'll need this one as well to avoid a panic:

https://svnweb.freebsd.org/base?view=revision&revision=287330
Comment 17 anthony 2015-09-02 19:14:11 UTC
Created attachment 160662 [details]
svn di svn://svn0.eu.freebsd.org/base/stable/10/sys/dev/e1000 sys/dev/e1000
Comment 18 anthony 2015-09-02 19:16:40 UTC
Unfortunately, the server experienced a watchdog timeout this morning :

Sep  2 07:39:49 urybsod kernel: em0: Watchdog timeout Queue[0]-- resetting
Sep  2 07:39:49 urybsod kernel: Interface is RUNNING and ACTIVE
Sep  2 07:39:49 urybsod kernel: em0: TX Queue 0 ------
Sep  2 07:39:49 urybsod kernel: em0: hw tdh = 247, hw tdt = 495
Sep  2 07:39:49 urybsod kernel: em0: Tx Queue Status = -2147483648
Sep  2 07:39:49 urybsod kernel: em0: TX descriptors avail = 771
Sep  2 07:39:49 urybsod kernel: em0: Tx Descriptors avail failure = 237
Sep  2 07:39:49 urybsod kernel: em0: RX Queue 0 ------
Sep  2 07:39:49 urybsod kernel: em0: hw rdh = 344, hw rdt = 343
Sep  2 07:39:49 urybsod kernel: em0: RX discarded packets = 0
Sep  2 07:39:49 urybsod kernel: em0: RX Next to Check = 344
Sep  2 07:39:49 urybsod kernel: em0: RX Next to Refresh = 343
Sep  2 07:39:49 urybsod kernel: em0: Link is Down
Sep  2 07:39:49 urybsod kernel: em0: link state changed to DOWN
Sep  2 07:39:53 urybsod kernel: em0: Link is up 1000 Mbps Full Duplex
Sep  2 07:39:53 urybsod kernel: em0: link state changed to UP
Sep  2 07:39:53 urybsod devd: Executing '/etc/rc.d/dhclient quietstart em0'

So these patches appear to be improving stability, as this is the only occurrence in 13 days of uptime, compared to nearly every day previously.

I should reiterate what state this kernel is in :
sys/dev/e1000 is at 10-STABLE@HEAD with CURRENT@286381, CURRENT@285879 patched. This required some manual patching. Diff of e1000 to 10-STABLE attached.
Merged r274043 and r274704 from 10-STABLE.

I've not yet encountered a panic, but I'll apply 287330 to be on the safe side.
Comment 19 Franco Fichtner 2015-09-30 08:39:50 UTC
It would be great to have these MFC'd to 10-STABLE, thanks. :)
Comment 20 Julien Cigar 2015-11-09 09:35:12 UTC
Hello,

I'm running 10.2-RELEASE on a HP Proliant Microserver with the following network interface:

em0@pci0:2:0:0: class=0x020000 card=0xa01f8086 chip=0x10d38086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82574L Gigabit Network Connection'
    class      = network
    subclass   = ethernet

The machine run an NFS server and unless I mount the shared folder with a rsize/wsize of 32768 I'm experiencing watchdog timeouts after a few minutes under load (for ex streaming a movie):

Nov  7 23:25:26 mordor kernel: em0: Watchdog timeout Queue[0]-- resetting
Nov  7 23:25:26 mordor kernel: Interface is RUNNING and ACTIVE
Nov  7 23:25:26 mordor kernel: em0: TX Queue 0 ------
Nov  7 23:25:26 mordor kernel: em0: hw tdh = 508, hw tdt = 402
Nov  7 23:25:26 mordor kernel: em0: Tx Queue Status = -2147483648
Nov  7 23:25:26 mordor kernel: em0: TX descriptors avail = 93
Nov  7 23:25:26 mordor kernel: em0: Tx Descriptors avail failure = 0
Nov  7 23:25:26 mordor kernel: em0: RX Queue 0 ------
Nov  7 23:25:26 mordor kernel: em0: hw rdh = 206, hw rdt = 205
Nov  7 23:25:26 mordor kernel: em0: RX discarded packets = 0
Nov  7 23:25:26 mordor kernel: em0: RX Next to Check = 206
Nov  7 23:25:26 mordor kernel: em0: RX Next to Refresh = 205
Nov  7 23:25:26 mordor kernel: em0: link state changed to DOWN
Nov  7 23:25:30 mordor kernel: em0: link state changed to UP

... and I have to do an "hard" reboot
Comment 21 Julien Cigar 2015-11-09 09:38:49 UTC
Note that this is with TSO enabled:

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
        ether 68:05:ca:xx:xx:xx
        inet 192.168.0.254 netmask 0xffffff00 broadcast 192.168.0.255 
        inet 192.168.0.21 netmask 0xffffffff broadcast 192.168.0.21 
        inet 192.168.0.20 netmask 0xffffffff broadcast 192.168.0.20 
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active

I'll try to disable TSO and see if it fixes the problem ..
Comment 22 Kubilay Kocak freebsd_committer freebsd_triage 2015-11-10 12:18:30 UTC
I believe this still needs MFC? (commit log specified two weeks)

Assign to committer who originally resolved in HEAD
Comment 23 Andre 2015-12-07 21:24:25 UTC
I'm seeing the same issue on the very same hardware as Julien on current releng/10.2.

Currently using this /etc/rc.conf workaround:
ifconfig_em0="DHCP -tso -vlanhwtso"
Comment 24 commit-hook freebsd_committer 2016-01-27 22:32:13 UTC
A commit references this bug:

Author: marius
Date: Wed Jan 27 22:31:10 UTC 2016
New revision: 294958
URL: https://svnweb.freebsd.org/changeset/base/294958

Log:
  Sync the e1000 drivers with what's in head as of r294327, modulo parts
  that don't apply to stable/10 (driver API, if_inc_counter(), RSS changes
  etc.) and modulo r287465 (which reportedly breaks igb(4)), i. e. assorted
  fixes and improvements only:

  o MFC r267385 (partial):
    - Don't compare bus_dma map pointers for static DMA allocations against
      NULL to determine if bus_dmamap_unload() or bus_dmamem_free() should be
      called. Instead, check the associated bus and virtual addresses.
    - Don't clear static DMA maps to NULL.
  o MFC r284933:
    Delete the refernce to VLAN handling being disabled by default. This is
    no longer the case. [1]
  o MFC r285639:
    Add an adapter CORE lock in the DDB hook em_dump_queue to avoid WITNESS
    panic in em_init_locked() while debugging.
  o MFC r285879:
    - Remove unused txd_saved.
    - Intialize txd_upper, txd_lower and txd_used at declaration.
  o MFC r286162:
    Free mbufs when busdma loading fails.
  o MFC r286829:
    Add capability to disable CRC stripping as it breaks IPMI/BMC capabilities
    on certain adatpers. [2]
  o MFC r286831: [3]
    - Increase EM_MAX_SCATTER to 64 such that the size of em_xmit()::
      segs[EM_MAX_SCATTER] doesn't get overrun by things like NFS that can
      and do shove more than 32 segs when being used with em(4) and TSO4.
    - Update tso handling code in em_xmit() with update from jhb@
    - Set if_hw_tsomax, if_hw_tsomaxsegcount and if_hw_tsomaxsegsize to
      appropriate values.
    - Define a TSO workaround "magic" number of 4 that is used to avoid an
      alignment issue in hardware.
    - Change a couple of integer values that were used as booleans to actual
      bool types.
    - Ensure that em_enable_intr() enables the appropriate mask of interrupts
      and not just a hardcoded define of values.
  o MFC r286832:
    e1000/if_lem.c bump to 1.1.0
  o MFC r286833:
    Bump all copywrite dates to 2015.
  o MFC r287112:
    Style/whitespace cleanup in shared/common code.
  o MFC r293331:
    - Switch em(4) to the extended RX descriptor format.
    - Split rxbuffer and txbuffer apart to support the new RX descriptor
      format structures. Move rxbuffer manipulation to em_setup_rxdesc() to
      unify the new behavior changes.
    - Add a RSSKEYLEN macro for help in generating the RSSKEY data structures
      in the card.
    - Change em_receive_checksum() to process the new rxdescriptor format
      status bit.
  o MFC r293332:
    Disable the reuse of checksum offload context descriptors in the case
    of multiple queues in em(4). Document errata in the code.
  o MFC r293854:
    Given that em(4), lem(4) and igb(4) hardware doesn't require the
    alignment guarantees provided by m_defrag(9), use m_collapse(9)
    instead for performance reasons.
    While at it, sanitize the statistics softc members, i. e. retire
    unused ones and add SYSCTL nodes missing for actually used ones.

  PR:	118693 [1], 161277 [2], 195078 [3], 199174 [3], 200221 [3]

Changes:
_U  stable/10/
  stable/10/share/man/man4/em.4
  stable/10/sys/dev/e1000/e1000_80003es2lan.c
  stable/10/sys/dev/e1000/e1000_80003es2lan.h
  stable/10/sys/dev/e1000/e1000_82540.c
  stable/10/sys/dev/e1000/e1000_82541.c
  stable/10/sys/dev/e1000/e1000_82541.h
  stable/10/sys/dev/e1000/e1000_82542.c
  stable/10/sys/dev/e1000/e1000_82543.c
  stable/10/sys/dev/e1000/e1000_82543.h
  stable/10/sys/dev/e1000/e1000_82571.c
  stable/10/sys/dev/e1000/e1000_82571.h
  stable/10/sys/dev/e1000/e1000_82575.c
  stable/10/sys/dev/e1000/e1000_82575.h
  stable/10/sys/dev/e1000/e1000_api.c
  stable/10/sys/dev/e1000/e1000_api.h
  stable/10/sys/dev/e1000/e1000_defines.h
  stable/10/sys/dev/e1000/e1000_hw.h
  stable/10/sys/dev/e1000/e1000_i210.c
  stable/10/sys/dev/e1000/e1000_i210.h
  stable/10/sys/dev/e1000/e1000_ich8lan.c
  stable/10/sys/dev/e1000/e1000_ich8lan.h
  stable/10/sys/dev/e1000/e1000_mac.c
  stable/10/sys/dev/e1000/e1000_mac.h
  stable/10/sys/dev/e1000/e1000_manage.c
  stable/10/sys/dev/e1000/e1000_manage.h
  stable/10/sys/dev/e1000/e1000_mbx.c
  stable/10/sys/dev/e1000/e1000_mbx.h
  stable/10/sys/dev/e1000/e1000_nvm.c
  stable/10/sys/dev/e1000/e1000_nvm.h
  stable/10/sys/dev/e1000/e1000_osdep.c
  stable/10/sys/dev/e1000/e1000_osdep.h
  stable/10/sys/dev/e1000/e1000_phy.c
  stable/10/sys/dev/e1000/e1000_phy.h
  stable/10/sys/dev/e1000/e1000_regs.h
  stable/10/sys/dev/e1000/e1000_vf.c
  stable/10/sys/dev/e1000/e1000_vf.h
  stable/10/sys/dev/e1000/if_em.c
  stable/10/sys/dev/e1000/if_em.h
  stable/10/sys/dev/e1000/if_igb.c
  stable/10/sys/dev/e1000/if_igb.h
  stable/10/sys/dev/e1000/if_lem.c
  stable/10/sys/dev/e1000/if_lem.h
  stable/10/sys/dev/ixgb/if_ixgb.c
  stable/10/sys/dev/netmap/if_em_netmap.h
Comment 25 Kubilay Kocak freebsd_committer freebsd_triage 2017-04-14 09:43:04 UTC
Merged to stable/10, record appropriately.