Bug 235246 - if_em init fails to allocate IRQ in 12.0-RELEASE
Summary: if_em init fails to allocate IRQ in 12.0-RELEASE
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.0-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: Marius Strobl
URL:
Keywords: IntelNetworking, regression
Depends on:
Blocks:
 
Reported: 2019-01-27 14:43 UTC by pr
Modified: 2019-02-15 14:01 UTC (History)
5 users (show)

See Also:
koobs: mfc-stable12+


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description pr 2019-01-27 14:43:56 UTC
Up to 11.2 this hardware was working OK:
none2@pci0:11:0:0:      class=0x020000 card=0x84571043 chip=0x150c8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82583V Gigabit Network Connection'
    class      = network
    subclass   = ethernet

Since 12.0-RELEASE init is inable to allocate IRQ for rid:
em0: <Intel(R) PRO/1000 Network Connection> port 0xec00-0xec1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 17 at device 0.0 on pci11
em0: attach_pre capping queues at 1
em0: using 1024 tx descriptors and 1024 rx descriptors
em0: msix_init qsets capped at 1
em0: pxm cpus: 6 queue msgs: 0 admincnt: 1
em0: using 0 rx queues 0 tx queues 
em0: Using MSIX interrupts with 1 vectors
em0: allocated for 0 tx_queues
em0: allocated for 0 rx_queues
em0: failed to allocate IRQ for rid 0, name irq0.
em0: iflib_legacy_setup failed 12
device_attach: em0 attach returned 12

Latest GENERIC Kernel on amd64:
# uname -a
FreeBSD machine 12.0-RELEASE-p2 FreeBSD 12.0-RELEASE-p2 GENERIC  amd64


dmesg for 11.2 was:
em0: <Intel(R) PRO/1000 Network Connection 7.6.1-k> port 0xec00-0xec1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 17 at device 0.0 on pci11
em0: Using an MSI interrupt
em0: Ethernet address: bc:ae:c5:33:6d:98
em0: netmap queues/slots: TX 1/2048, RX 1/2048
Comment 1 pr 2019-01-30 10:02:31 UTC
Ok, here I have an update: I found a working and a not working hardware, so maybe it helps.

WORKING:
em0@pci0:1:0:0: class=0x020000 card=0x10d315d9 chip=0x10d38086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82574L Gigabit Network Connection'
    class      = network
    subclass   = ethernet

em0: <Intel(R) PRO/1000 Network Connection> port 0xd800-0xd81f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 48 at device 0.0 on pci1
em0: attach_pre capping queues at 2
em0: using 1024 tx descriptors and 1024 rx descriptors
em0: msix_init qsets capped at 2
em0: pxm cpus: 4 queue msgs: 4 admincnt: 1
em0: using 2 rx queues 2 tx queues 
em0: Using MSIX interrupts with 3 vectors
em0: allocated for 2 tx_queues
em0: allocated for 2 rx_queues
em0: Ethernet address: 00:25:90:20:e7:aa
em0: netmap queues/slots: TX 2/1024, RX 2/1024
em0: link state changed to UP

NOT WORKING:
none2@pci0:11:0:0:      class=0x020000 card=0x84571043 chip=0x150c8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82583V Gigabit Network Connection'
    class      = network
    subclass   = ethernet

em0: <Intel(R) PRO/1000 Network Connection> port 0xec00-0xec1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 17 at device 0.0 on pci11
em0: attach_pre capping queues at 1
em0: using 1024 tx descriptors and 1024 rx descriptors
em0: msix_init qsets capped at 1
em0: pxm cpus: 6 queue msgs: 0 admincnt: 1
em0: using 0 rx queues 0 tx queues 
em0: Using MSIX interrupts with 1 vectors
em0: allocated for 0 tx_queues
em0: allocated for 0 rx_queues
em0: failed to allocate IRQ for rid 0, name irq0.
em0: iflib_legacy_setup failed 12
device_attach: em0 attach returned 12
Comment 2 Arun Pereira 2019-01-31 00:30:24 UTC
We had the same issue. The 82583V will work if you turn off msix in loader.conf
hw.pci.enable_msix=0

I understand that this is not a fix for the issue but it is a workaround that seems to help.
Comment 3 Eric Joyner freebsd_committer 2019-01-31 23:41:30 UTC
This commit should fix this issue -- iflib now doesn't try using MSI-X on devices that don't support it. 82574 is the only em(4) device that does support MSI-X, so that's why you don't see errors with it.

https://svnweb.freebsd.org/base?view=revision&revision=343578
Comment 4 ncrogers 2019-02-05 02:37:35 UTC
Are there plans to MFC the fix to 12-STABLE?
Comment 5 pr 2019-02-07 15:34:29 UTC
Patch is for head, I only have 12s around, so I applied the patch (no too cleanly, required some manual edit) but it did not work:
em0: <Intel(R) PRO/1000 Network Connection> port 0xec00-0xec1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 17 at device 0.0 on pci11
em0: Using 1024 tx descriptors and 1024 rx descriptors
em0: Using 0 rx queues 0 tx queues
em0: Using MSI-X interrupts with 1 vectors
em0: failed to allocate IRQ for rid 0, name irq0.
em0: iflib_legacy_setup failed 12
device_attach: em0 attach returned 12

Disabling MSI-X in loader.conf (with a stock kernel) did the trick:
em0: <Intel(R) PRO/1000 Network Connection> port 0xec00-0xec1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 17 at device 0.0 on pci11
em0: attach_pre capping queues at 1
em0: using 1024 tx descriptors and 1024 rx descriptors
em0: msix_init qsets capped at 1
em0: System has MSIX disabled 
em0: Using an MSI interrupt
em0: allocated for 1 tx_queues
em0: allocated for 1 rx_queues
em0: Ethernet address: bc:ae:c5:33:6d:98
em0: netmap queues/slots: TX 1/1024, RX 1/1024

Which is different but similar to how it was under 11.2:
em0: <Intel(R) PRO/1000 Network Connection 7.6.1-k> port 0xec00-0xec1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 17 at device 0.0 on pci11
em0: Using an MSI interrupt
em0: Ethernet address: bc:ae:c5:33:6d:98
em0: netmap queues/slots: TX 1/1024, RX 1/1024

Before we close we shall test the patch under head, I cannot at the moment as the Intels I have are all in production. Anyone?
Comment 6 Arun Pereira 2019-02-07 15:56:11 UTC
It seems that the patch referenced by https://svnweb.freebsd.org/base?view=revision&revision=343578 is already in stable/12 https://github.com/freebsd/freebsd/commit/b1c306fca3679b61586013269f45b42ed6bb294b

Well, at least partially. I'll test it out today and see if it's fixed in stable/12 without having to disable msix.
Comment 7 commit-hook freebsd_committer 2019-02-09 11:59:48 UTC
A commit references this bug:

Author: marius
Date: Sat Feb  9 11:58:41 UTC 2019
New revision: 343934
URL: https://svnweb.freebsd.org/changeset/base/343934

Log:
  - Remove the redundant device disabled hint handling; ever since
    r241119 that's performed globally by device_attach(9).
  - As for the EM-class of devices, em(4) supports multiple queues
    and MSI-X respectively only with 82574 devices. However, since
    the conversion to iflib(4), em(4) relies on the interrupt type
    fallback mechanism, i. e. MSI-X -> MSI -> INTx, of iflib(4) to
    figure out the interrupt type to use for the EM-class (as well
    as the IGB-class) of MACs. Moreover, despite the datasheet for
    82583V not mentioning any support of MSI-X, there actually are
    82583V devices out there that report a varying number of MSI-X
    messages as supported. The interrupt type fallback of iflib(4)
    is causing two failure modes depending on the actual number of
    MSI-X messages supported for such instances of 82583V:
    1) With only one MSI-X message supported, none is left for the
       RX/TX queues as that one message gets assigned to the admin
       interrupt. Worse, later on - which will be addressed with a
       separate fix - iflib(4) interprets that one messages as MSI
       or INTx to be set up, but fails to actually do so as it has
       previously called pci_alloc_msix(9). [1, 2]
    2) With more message supported, their distribution is okay but
       then em_if_msix_intr_assign() doesn't work for 82583V, with
       the interface being left in a non-working state, too. [3]
    Thus, let em_if_attach_pre() indicate to iflib(4) to try MSI-X
    with 82574 only, and at most MSI for the remainder of EM-class
    devices.
    While at it, remove "try_second_bar" as it's polarity inverted
    and not actually needed.
  - Remove code from em_if_timer() that effectively is a NOP since
    the conversion to iflib(4) ("trigger" is no longer read).
    While at it, let the comment for em_if_timer() reflect reality
    after said conversion.
  - Implement an ifdi_watchdog_reset method which only updates the
    em(4) "watchdog_events" counter but doesn't perform any reset,
    so that the em(4) "watchdog_timeouts" SYSCTL (iflib(4) doesn't
    provide a counterpart) reflects reality and these timeouts add
    to IFCOUNTER_OERRORS again after the iflib(4) conversion.
  - Remove the "mbuf_defrag_fail" and "tx_dma_fail" SYSCTLS; since
    the iflib(4) conversion, associated counters are disconnected,
    but iflib(4) provides "mbuf_defrag_failed" and "tx_map_failed"
    respectively as equivalents.
  - Move the description preceding lem_smartspeed() to the correct
    spot before em_reset() and bring back appropriate comments for
    {igb,em}_initialize_rss_mapping() and lem_smartspeed() lost in
    the iflib(4) conversion.
  - Adapt some other function descriptions and INIT_DEBUGOUT() use
    to match reality after the iflib(4) conversion.
  - Put the debugging message of em_enable_vectors_82574() (missed
    in r343578) under bootverbose, too.

  PR:		219428 [1], 235246 [2], 235147 [3]
  Reviewed by:	erj (previous version)
  Differential Revision:	https://reviews.freebsd.org/D19108

Changes:
  head/sys/dev/e1000/if_em.c
  head/sys/dev/e1000/if_em.h
Comment 8 commit-hook freebsd_committer 2019-02-13 14:39:51 UTC
A commit references this bug:

Author: marius
Date: Wed Feb 13 14:39:17 UTC 2019
New revision: 344098
URL: https://svnweb.freebsd.org/changeset/base/344098

Log:
  MFC: r343934

  - Remove the redundant device disabled hint handling; ever since
    r241119 that's performed globally by device_attach(9).
  - As for the EM-class of devices, em(4) supports multiple queues
    and MSI-X respectively only with 82574 devices. However, since
    the conversion to iflib(4), em(4) relies on the interrupt type
    fallback mechanism, i. e. MSI-X -> MSI -> INTx, of iflib(4) to
    figure out the interrupt type to use for the EM-class (as well
    as the IGB-class) of MACs. Moreover, despite the datasheet for
    82583V not mentioning any support of MSI-X, there actually are
    82583V devices out there that report a varying number of MSI-X
    messages as supported. The interrupt type fallback of iflib(4)
    is causing two failure modes depending on the actual number of
    MSI-X messages supported for such instances of 82583V:
    1) With only one MSI-X message supported, none is left for the
       RX/TX queues as that one message gets assigned to the admin
       interrupt. Worse, later on - which will be addressed with a
       separate fix - iflib(4) interprets that one messages as MSI
       or INTx to be set up, but fails to actually do so as it has
       previously called pci_alloc_msix(9). [1, 2]
    2) With more message supported, their distribution is okay but
       then em_if_msix_intr_assign() doesn't work for 82583V, with
       the interface being left in a non-working state, too. [3]
    Thus, let em_if_attach_pre() indicate to iflib(4) to try MSI-X
    with 82574 only, and at most MSI for the remainder of EM-class
    devices.
    While at it, remove "try_second_bar" as it's polarity inverted
    and not actually needed.
  - Remove code from em_if_timer() that effectively is a NOP since
    the conversion to iflib(4) ("trigger" is no longer read).
    While at it, let the comment for em_if_timer() reflect reality
    after said conversion.
  - Implement an ifdi_watchdog_reset method which only updates the
    em(4) "watchdog_events" counter but doesn't perform any reset,
    so that the em(4) "watchdog_timeouts" SYSCTL (iflib(4) doesn't
    provide a counterpart) reflects reality and these timeouts add
    to IFCOUNTER_OERRORS again after the iflib(4) conversion.
  - Remove the "mbuf_defrag_fail" and "tx_dma_fail" SYSCTLS; since
    the iflib(4) conversion, associated counters are disconnected,
    but iflib(4) provides "mbuf_defrag_failed" and "tx_map_failed"
    respectively as equivalents.
  - Move the description preceding lem_smartspeed() to the correct
    spot before em_reset() and bring back appropriate comments for
    {igb,em}_initialize_rss_mapping() and lem_smartspeed() lost in
    the iflib(4) conversion.
  - Adapt some other function descriptions and INIT_DEBUGOUT() use
    to match reality after the iflib(4) conversion.
  - Put the debugging message of em_enable_vectors_82574() (missed
    in r343578) under bootverbose, too.

  PR:		219428 [1], 235246 [2], 235147 [3]
  Reviewed by:	erj (previous version)
  Differential Revision:	https://reviews.freebsd.org/D19108

Changes:
_U  stable/12/
  stable/12/sys/dev/e1000/if_em.c
  stable/12/sys/dev/e1000/if_em.h
Comment 9 Marius Strobl freebsd_committer 2019-02-13 14:42:36 UTC
Close; thanks for the report!
Comment 10 pr 2019-02-13 22:03:56 UTC
(In reply to Marius Strobl from comment #9)
Thank you so much Marius.

Both hardware version are now working:
82583V:
em0: <Intel(R) PRO/1000 Network Connection> port 0xec00-0xec1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 17 at device 0.0 on pci11
em0: Using 1024 tx descriptors and 1024 rx descriptors
em0: Using an MSI interrupt
em0: Ethernet address: bc:ae:c5:33:6d:98
em0: netmap queues/slots: TX 1/1024, RX 1/1024

82574L:
em0: <Intel(R) PRO/1000 Network Connection> port 0xd800-0xd81f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 48 at device 0.0 on pci1
em0: Using 1024 tx descriptors and 1024 rx descriptors
em0: Using 2 rx queues 2 tx queues
em0: Using MSI-X interrupts with 3 vectors
em0: Ethernet address: 00:25:90:20:e7:aa
em0: netmap queues/slots: TX 2/1024, RX 2/1024