Bug 258153 - em0: Hardware Initialization Failed - after wake up from sleep
Summary: em0: Hardware Initialization Failed - after wake up from sleep
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 13.0-STABLE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: Kevin Bowling
URL:
Keywords: IntelNetworking
Depends on:
Blocks:
 
Reported: 2021-08-29 21:21 UTC by iron.udjin
Modified: 2021-09-30 00:25 UTC (History)
2 users (show)

See Also:
kbowling: mfc-stable13+
kbowling: mfc-stable12+
kbowling: mfc-stable11-


Attachments
disable_ulp (827 bytes, patch)
2021-09-06 05:50 UTC, Kevin Bowling
no flags Details | Diff
e1000 k0s patch (3.76 KB, patch)
2021-09-22 01:37 UTC, Kevin Bowling
no flags Details | Diff
e1000 dpdk ich8lan sync (14.30 KB, patch)
2021-09-22 01:37 UTC, Kevin Bowling
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description iron.udjin 2021-08-29 21:21:07 UTC
After commit: 524ea0dea9b07183c1f36ec894d9494fcc0ffe5a system freeze when trying to wakeup after sleep.

dmesg:

kernel: em0: Hardware Initialization Failed
syslogd: last message repeated 1 times
kernel: em0: link state changed to DOWN
Comment 1 Kevin Bowling freebsd_committer freebsd_triage 2021-09-03 14:19:50 UTC
Thanks for the report, I'll have a patch ready for you to test this weekend
Comment 2 Kevin Bowling freebsd_committer freebsd_triage 2021-09-06 05:50:40 UTC
Created attachment 227701 [details]
disable_ulp

Can you try this?  You may need to hard cycle the laptop to get the NIC and ME into a known good state.
Comment 3 iron.udjin 2021-09-07 00:12:46 UTC
(In reply to Kevin Bowling from comment #2)

Hello Kevin,

I just tested your patch. Something strange is happening after the system being woke up. I can ping any host but cannot connect to any external host by TCP or UDP:

$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=118 time=16.000 ms

$ host -t a google.com 8.8.8.8
;; connection timed out; no servers could be reached

Even if I run: "ifconfig em0 down" and then "/etc/netstart" - I'm getting the same result. But if I unplug network wire from the network card and then plug it in - TCP and UDP are starting to work.

P.S: I have no firewall enabled or anything else which could possibly block connections.
Comment 4 Kevin Bowling freebsd_committer freebsd_triage 2021-09-07 04:31:35 UTC
(In reply to iron.udjin from comment #3)
Thanks, can you clarify (1) as true or false, and (2a) (2b) as true or false: 
(1) With the patch, it now wakes up but there is no connectivity
(2) Reverting 524ea0dea9b07183c1f36ec894d9494fcc0ffe5a fixes both (a) wakeup and (b) network after wakeup
Comment 5 iron.udjin 2021-09-07 18:19:20 UTC
(In reply to Kevin Bowling from comment #4)

1. true
2a. true
2b. true
Comment 6 Kevin Bowling freebsd_committer freebsd_triage 2021-09-17 16:28:54 UTC
(In reply to iron.udjin from comment #5)
Thanks!  Can you try this series of commits?  https://github.com/freebsd/freebsd-src/pull/538/commits

It leaves out the ULP change on suspend.  If that still isn't working for you try reverting the commit "e1000: Sync ich8lan with DPDK" in this series and I can inspect more closely these changes.
Comment 7 commit-hook freebsd_committer freebsd_triage 2021-09-17 21:05:47 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=a4378873e9ce1b35b55378c21f8eae69e58c2525

commit a4378873e9ce1b35b55378c21f8eae69e58c2525
Author:     Kevin Bowling <kbowling@FreeBSD.org>
AuthorDate: 2021-09-08 22:43:13 +0000
Commit:     Kevin Bowling <kbowling@FreeBSD.org>
CommitDate: 2021-09-17 21:03:08 +0000

    e1000: Revert Update intel shared code

    This reverts commit fc7682b17f3738573099b8b03f5628dcc8148adb.

    This will be done incrementally to help with bisecting an issue in
    later I21x devices (ich8lan).

    PR:             258153
    Approved by:    imp
    MFC after:      1 day

 sys/dev/e1000/e1000_80003es2lan.c |  36 ++---
 sys/dev/e1000/e1000_80003es2lan.h |  38 ++---
 sys/dev/e1000/e1000_82540.c       |  40 ++---
 sys/dev/e1000/e1000_82541.c       |  40 ++---
 sys/dev/e1000/e1000_82541.h       |  40 ++---
 sys/dev/e1000/e1000_82542.c       |  40 ++---
 sys/dev/e1000/e1000_82543.c       |  40 ++---
 sys/dev/e1000/e1000_82543.h       |  40 ++---
 sys/dev/e1000/e1000_82571.c       |  42 ++---
 sys/dev/e1000/e1000_82571.h       |  40 ++---
 sys/dev/e1000/e1000_82575.c       |  40 ++---
 sys/dev/e1000/e1000_82575.h       |  40 ++---
 sys/dev/e1000/e1000_api.c         |  56 +++----
 sys/dev/e1000/e1000_api.h         |  41 +++--
 sys/dev/e1000/e1000_defines.h     |  73 +++------
 sys/dev/e1000/e1000_hw.h          |  42 +++--
 sys/dev/e1000/e1000_i210.c        | 140 +++--------------
 sys/dev/e1000/e1000_i210.h        |  42 +++--
 sys/dev/e1000/e1000_ich8lan.c     | 316 ++++++++------------------------------
 sys/dev/e1000/e1000_ich8lan.h     |  47 +++---
 sys/dev/e1000/e1000_mac.c         |  40 ++---
 sys/dev/e1000/e1000_mac.h         |  40 ++---
 sys/dev/e1000/e1000_manage.c      |  40 ++---
 sys/dev/e1000/e1000_manage.h      |  40 ++---
 sys/dev/e1000/e1000_mbx.c         |  40 ++---
 sys/dev/e1000/e1000_mbx.h         |  40 ++---
 sys/dev/e1000/e1000_nvm.c         | 199 +++---------------------
 sys/dev/e1000/e1000_nvm.h         |  42 ++---
 sys/dev/e1000/e1000_osdep.c       |  40 ++---
 sys/dev/e1000/e1000_osdep.h       |  40 ++---
 sys/dev/e1000/e1000_phy.c         |  62 ++++----
 sys/dev/e1000/e1000_phy.h         |  49 +++---
 sys/dev/e1000/e1000_regs.h        |  55 +++----
 sys/dev/e1000/e1000_vf.c          |  43 +++---
 sys/dev/e1000/e1000_vf.h          |  40 ++---
 sys/dev/e1000/if_em.c             |  10 +-
 36 files changed, 755 insertions(+), 1298 deletions(-)
Comment 8 Kevin Bowling freebsd_committer freebsd_triage 2021-09-17 21:58:28 UTC
(In reply to Kevin Bowling from comment #6)
I reverted the problematic changes in main, brought forward the ones I think are safe.

I split the risky changes for I219 into https://github.com/freebsd/freebsd-src/pull/539 if you can test.  We may need to comb through this or get some help from intel.
Comment 9 iron.udjin 2021-09-17 22:43:55 UTC
(In reply to commit-hook from comment #7)

After this patch https://cgit.freebsd.org/src/diff/?id=a4378873e9ce1b35b55378c21f8eae69e58c2525 everything works fine.
Comment 10 commit-hook freebsd_committer freebsd_triage 2021-09-18 00:39:17 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=5f4ba94eb591bea8f41555a0738725a11ee29802

commit 5f4ba94eb591bea8f41555a0738725a11ee29802
Author:     Kevin Bowling <kbowling@FreeBSD.org>
AuthorDate: 2021-09-08 22:43:13 +0000
Commit:     Kevin Bowling <kbowling@FreeBSD.org>
CommitDate: 2021-09-18 00:38:35 +0000

    e1000: Revert Update intel shared code

    This reverts commit fc7682b17f3738573099b8b03f5628dcc8148adb.

    This will be done incrementally to help with bisecting an issue in
    later I21x devices (ich8lan).

    PR:             258153
    Approved by:    imp
    MFC after:      1 day

    (cherry picked from commit a4378873e9ce1b35b55378c21f8eae69e58c2525)

 sys/dev/e1000/e1000_80003es2lan.c |  36 ++---
 sys/dev/e1000/e1000_80003es2lan.h |  38 ++---
 sys/dev/e1000/e1000_82540.c       |  40 ++---
 sys/dev/e1000/e1000_82541.c       |  40 ++---
 sys/dev/e1000/e1000_82541.h       |  40 ++---
 sys/dev/e1000/e1000_82542.c       |  40 ++---
 sys/dev/e1000/e1000_82543.c       |  40 ++---
 sys/dev/e1000/e1000_82543.h       |  40 ++---
 sys/dev/e1000/e1000_82571.c       |  42 ++---
 sys/dev/e1000/e1000_82571.h       |  40 ++---
 sys/dev/e1000/e1000_82575.c       |  40 ++---
 sys/dev/e1000/e1000_82575.h       |  40 ++---
 sys/dev/e1000/e1000_api.c         |  56 +++----
 sys/dev/e1000/e1000_api.h         |  41 +++--
 sys/dev/e1000/e1000_defines.h     |  73 +++------
 sys/dev/e1000/e1000_hw.h          |  42 +++--
 sys/dev/e1000/e1000_i210.c        | 140 +++--------------
 sys/dev/e1000/e1000_i210.h        |  42 +++--
 sys/dev/e1000/e1000_ich8lan.c     | 316 ++++++++------------------------------
 sys/dev/e1000/e1000_ich8lan.h     |  47 +++---
 sys/dev/e1000/e1000_mac.c         |  40 ++---
 sys/dev/e1000/e1000_mac.h         |  40 ++---
 sys/dev/e1000/e1000_manage.c      |  40 ++---
 sys/dev/e1000/e1000_manage.h      |  40 ++---
 sys/dev/e1000/e1000_mbx.c         |  40 ++---
 sys/dev/e1000/e1000_mbx.h         |  40 ++---
 sys/dev/e1000/e1000_nvm.c         | 199 +++---------------------
 sys/dev/e1000/e1000_nvm.h         |  42 ++---
 sys/dev/e1000/e1000_osdep.c       |  40 ++---
 sys/dev/e1000/e1000_osdep.h       |  40 ++---
 sys/dev/e1000/e1000_phy.c         |  62 ++++----
 sys/dev/e1000/e1000_phy.h         |  49 +++---
 sys/dev/e1000/e1000_regs.h        |  55 +++----
 sys/dev/e1000/e1000_vf.c          |  43 +++---
 sys/dev/e1000/e1000_vf.h          |  40 ++---
 sys/dev/e1000/if_em.c             |  10 +-
 36 files changed, 755 insertions(+), 1298 deletions(-)
Comment 11 commit-hook freebsd_committer freebsd_triage 2021-09-18 00:40:19 UTC
A commit in branch stable/12 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=304bf88ad91e08fc61787b0de5ec43676df9c398

commit 304bf88ad91e08fc61787b0de5ec43676df9c398
Author:     Kevin Bowling <kbowling@FreeBSD.org>
AuthorDate: 2021-09-08 22:43:13 +0000
Commit:     Kevin Bowling <kbowling@FreeBSD.org>
CommitDate: 2021-09-18 00:39:24 +0000

    e1000: Revert Update intel shared code

    This reverts commit fc7682b17f3738573099b8b03f5628dcc8148adb.

    This will be done incrementally to help with bisecting an issue in
    later I21x devices (ich8lan).

    PR:             258153
    Approved by:    imp
    MFC after:      1 day

    (cherry picked from commit a4378873e9ce1b35b55378c21f8eae69e58c2525)

 sys/dev/e1000/e1000_80003es2lan.c |  36 ++---
 sys/dev/e1000/e1000_80003es2lan.h |  38 ++---
 sys/dev/e1000/e1000_82540.c       |  40 ++---
 sys/dev/e1000/e1000_82541.c       |  40 ++---
 sys/dev/e1000/e1000_82541.h       |  40 ++---
 sys/dev/e1000/e1000_82542.c       |  40 ++---
 sys/dev/e1000/e1000_82543.c       |  40 ++---
 sys/dev/e1000/e1000_82543.h       |  40 ++---
 sys/dev/e1000/e1000_82571.c       |  42 ++---
 sys/dev/e1000/e1000_82571.h       |  40 ++---
 sys/dev/e1000/e1000_82575.c       |  40 ++---
 sys/dev/e1000/e1000_82575.h       |  40 ++---
 sys/dev/e1000/e1000_api.c         |  56 +++----
 sys/dev/e1000/e1000_api.h         |  41 +++--
 sys/dev/e1000/e1000_defines.h     |  73 +++------
 sys/dev/e1000/e1000_hw.h          |  42 +++--
 sys/dev/e1000/e1000_i210.c        | 140 +++--------------
 sys/dev/e1000/e1000_i210.h        |  42 +++--
 sys/dev/e1000/e1000_ich8lan.c     | 316 ++++++++------------------------------
 sys/dev/e1000/e1000_ich8lan.h     |  47 +++---
 sys/dev/e1000/e1000_mac.c         |  40 ++---
 sys/dev/e1000/e1000_mac.h         |  40 ++---
 sys/dev/e1000/e1000_manage.c      |  40 ++---
 sys/dev/e1000/e1000_manage.h      |  40 ++---
 sys/dev/e1000/e1000_mbx.c         |  40 ++---
 sys/dev/e1000/e1000_mbx.h         |  40 ++---
 sys/dev/e1000/e1000_nvm.c         | 199 +++---------------------
 sys/dev/e1000/e1000_nvm.h         |  42 ++---
 sys/dev/e1000/e1000_osdep.c       |  40 ++---
 sys/dev/e1000/e1000_osdep.h       |  40 ++---
 sys/dev/e1000/e1000_phy.c         |  62 ++++----
 sys/dev/e1000/e1000_phy.h         |  49 +++---
 sys/dev/e1000/e1000_regs.h        |  55 +++----
 sys/dev/e1000/e1000_vf.c          |  43 +++---
 sys/dev/e1000/e1000_vf.h          |  40 ++---
 sys/dev/e1000/if_em.c             |  10 +-
 36 files changed, 755 insertions(+), 1298 deletions(-)
Comment 12 Kevin Bowling freebsd_committer freebsd_triage 2021-09-18 00:41:32 UTC
(In reply to iron.udjin from comment #9)
Thanks for the confirmation!  I have MFCed it to stable/{12,13}.

Are you able to test https://github.com/freebsd/freebsd-src/pull/539?
Comment 13 iron.udjin 2021-09-18 06:11:34 UTC
(In reply to Kevin Bowling from comment #12)

How can I get patch of those two commits?
Comment 14 Kevin Bowling freebsd_committer freebsd_triage 2021-09-22 01:37:14 UTC
Created attachment 228109 [details]
e1000 k0s patch
Comment 15 Kevin Bowling freebsd_committer freebsd_triage 2021-09-22 01:37:40 UTC
Created attachment 228110 [details]
e1000 dpdk ich8lan sync
Comment 16 Kevin Bowling freebsd_committer freebsd_triage 2021-09-22 01:38:26 UTC
(In reply to iron.udjin from comment #13)
I've uploaded two patches.  I am unsure if they will cleanly apply to stable/XX until I MFC the other changes.
Comment 17 iron.udjin 2021-09-22 11:28:54 UTC
(In reply to Kevin Bowling from comment #16)

I couldn't apply e1000 dpdk ich8lan sync patch.

After e1000 k0s patch everything seems fine. After wakeup network works fine.
Comment 18 commit-hook freebsd_committer freebsd_triage 2021-09-23 19:42:32 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=d5ad2f2a67df54ac40148cca21e726bc61a48982

commit d5ad2f2a67df54ac40148cca21e726bc61a48982
Author:     Wenzhuo Lu <wenzhuo.lu@intel.com>
AuthorDate: 2015-10-16 02:51:09 +0000
Commit:     Kevin Bowling <kbowling@FreeBSD.org>
CommitDate: 2021-09-23 19:41:37 +0000

    e1000: fix K1 configuration

    This patch is for the following updates to the K1 configurations:
    Tx idle period for entering K1 should be 128 ns.
    Minimum Tx idle period in K1 should be 256 ns.

    Signed-off-by: Wenzhuo Lu <wenzhuo.lu@intel.com>

    PR:             258153
    Reviewed by:    erj
    Tested by:      iron.udjin@gmail.com
    Approved by:    imp
    Obtained from:  DPDK (6f934fa24dfd437c90ead96bc7598ee77a117ede)
    MFC after:      1 week

 sys/dev/e1000/e1000_ich8lan.c | 45 ++++++++++++++++++++++++++++++++++++++++++-
 sys/dev/e1000/e1000_ich8lan.h |  1 +
 sys/dev/e1000/e1000_phy.h     |  7 +++++++
 3 files changed, 52 insertions(+), 1 deletion(-)
Comment 19 Kevin Bowling freebsd_committer freebsd_triage 2021-09-28 18:14:37 UTC
(In reply to iron.udjin from comment #17)
The patch should apply now if you update your stable/13.

I've noticed a couple regressions so I am not sure how to proceed; I will probably leave these changes out unless I can get some more information from Intel:  https://github.com/freebsd/freebsd-src/pull/539#issuecomment-928200563
Comment 20 iron.udjin 2021-09-28 18:17:16 UTC
(In reply to Kevin Bowling from comment #19)

I've updated system to latest -STABLE and everything works fine. I think we can close this issue.
Comment 21 commit-hook freebsd_committer freebsd_triage 2021-09-30 00:23:17 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=4cb203d8f3a10e927729e4daafee9d90045de0bc

commit 4cb203d8f3a10e927729e4daafee9d90045de0bc
Author:     Wenzhuo Lu <wenzhuo.lu@intel.com>
AuthorDate: 2015-10-16 02:51:09 +0000
Commit:     Kevin Bowling <kbowling@FreeBSD.org>
CommitDate: 2021-09-30 00:22:59 +0000

    e1000: fix K1 configuration

    This patch is for the following updates to the K1 configurations:
    Tx idle period for entering K1 should be 128 ns.
    Minimum Tx idle period in K1 should be 256 ns.

    Signed-off-by: Wenzhuo Lu <wenzhuo.lu@intel.com>

    PR:             258153
    Reviewed by:    erj
    Tested by:      iron.udjin@gmail.com
    Approved by:    imp
    Obtained from:  DPDK (6f934fa24dfd437c90ead96bc7598ee77a117ede)
    MFC after:      1 week

    (cherry picked from commit d5ad2f2a67df54ac40148cca21e726bc61a48982)

 sys/dev/e1000/e1000_ich8lan.c | 45 ++++++++++++++++++++++++++++++++++++++++++-
 sys/dev/e1000/e1000_ich8lan.h |  1 +
 sys/dev/e1000/e1000_phy.h     |  7 +++++++
 3 files changed, 52 insertions(+), 1 deletion(-)
Comment 22 commit-hook freebsd_committer freebsd_triage 2021-09-30 00:25:18 UTC
A commit in branch stable/12 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=b4c189e2fe07d67f8b2d92e5d29aaada48d6d882

commit b4c189e2fe07d67f8b2d92e5d29aaada48d6d882
Author:     Wenzhuo Lu <wenzhuo.lu@intel.com>
AuthorDate: 2015-10-16 02:51:09 +0000
Commit:     Kevin Bowling <kbowling@FreeBSD.org>
CommitDate: 2021-09-30 00:23:56 +0000

    e1000: fix K1 configuration

    This patch is for the following updates to the K1 configurations:
    Tx idle period for entering K1 should be 128 ns.
    Minimum Tx idle period in K1 should be 256 ns.

    Signed-off-by: Wenzhuo Lu <wenzhuo.lu@intel.com>

    PR:             258153
    Reviewed by:    erj
    Tested by:      iron.udjin@gmail.com
    Approved by:    imp
    Obtained from:  DPDK (6f934fa24dfd437c90ead96bc7598ee77a117ede)
    MFC after:      1 week

    (cherry picked from commit d5ad2f2a67df54ac40148cca21e726bc61a48982)

 sys/dev/e1000/e1000_ich8lan.c | 45 ++++++++++++++++++++++++++++++++++++++++++-
 sys/dev/e1000/e1000_ich8lan.h |  1 +
 sys/dev/e1000/e1000_phy.h     |  7 +++++++
 3 files changed, 52 insertions(+), 1 deletion(-)