Up to 11.2 this hardware was working OK: none2@pci0:11:0:0: class=0x020000 card=0x84571043 chip=0x150c8086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82583V Gigabit Network Connection' class = network subclass = ethernet Since 12.0-RELEASE init is inable to allocate IRQ for rid: em0: <Intel(R) PRO/1000 Network Connection> port 0xec00-0xec1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 17 at device 0.0 on pci11 em0: attach_pre capping queues at 1 em0: using 1024 tx descriptors and 1024 rx descriptors em0: msix_init qsets capped at 1 em0: pxm cpus: 6 queue msgs: 0 admincnt: 1 em0: using 0 rx queues 0 tx queues em0: Using MSIX interrupts with 1 vectors em0: allocated for 0 tx_queues em0: allocated for 0 rx_queues em0: failed to allocate IRQ for rid 0, name irq0. em0: iflib_legacy_setup failed 12 device_attach: em0 attach returned 12 Latest GENERIC Kernel on amd64: # uname -a FreeBSD machine 12.0-RELEASE-p2 FreeBSD 12.0-RELEASE-p2 GENERIC amd64 dmesg for 11.2 was: em0: <Intel(R) PRO/1000 Network Connection 7.6.1-k> port 0xec00-0xec1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 17 at device 0.0 on pci11 em0: Using an MSI interrupt em0: Ethernet address: bc:ae:c5:33:6d:98 em0: netmap queues/slots: TX 1/2048, RX 1/2048
Ok, here I have an update: I found a working and a not working hardware, so maybe it helps. WORKING: em0@pci0:1:0:0: class=0x020000 card=0x10d315d9 chip=0x10d38086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82574L Gigabit Network Connection' class = network subclass = ethernet em0: <Intel(R) PRO/1000 Network Connection> port 0xd800-0xd81f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 48 at device 0.0 on pci1 em0: attach_pre capping queues at 2 em0: using 1024 tx descriptors and 1024 rx descriptors em0: msix_init qsets capped at 2 em0: pxm cpus: 4 queue msgs: 4 admincnt: 1 em0: using 2 rx queues 2 tx queues em0: Using MSIX interrupts with 3 vectors em0: allocated for 2 tx_queues em0: allocated for 2 rx_queues em0: Ethernet address: 00:25:90:20:e7:aa em0: netmap queues/slots: TX 2/1024, RX 2/1024 em0: link state changed to UP NOT WORKING: none2@pci0:11:0:0: class=0x020000 card=0x84571043 chip=0x150c8086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82583V Gigabit Network Connection' class = network subclass = ethernet em0: <Intel(R) PRO/1000 Network Connection> port 0xec00-0xec1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 17 at device 0.0 on pci11 em0: attach_pre capping queues at 1 em0: using 1024 tx descriptors and 1024 rx descriptors em0: msix_init qsets capped at 1 em0: pxm cpus: 6 queue msgs: 0 admincnt: 1 em0: using 0 rx queues 0 tx queues em0: Using MSIX interrupts with 1 vectors em0: allocated for 0 tx_queues em0: allocated for 0 rx_queues em0: failed to allocate IRQ for rid 0, name irq0. em0: iflib_legacy_setup failed 12 device_attach: em0 attach returned 12
We had the same issue. The 82583V will work if you turn off msix in loader.conf hw.pci.enable_msix=0 I understand that this is not a fix for the issue but it is a workaround that seems to help.
This commit should fix this issue -- iflib now doesn't try using MSI-X on devices that don't support it. 82574 is the only em(4) device that does support MSI-X, so that's why you don't see errors with it. https://svnweb.freebsd.org/base?view=revision&revision=343578
Are there plans to MFC the fix to 12-STABLE?
Patch is for head, I only have 12s around, so I applied the patch (no too cleanly, required some manual edit) but it did not work: em0: <Intel(R) PRO/1000 Network Connection> port 0xec00-0xec1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 17 at device 0.0 on pci11 em0: Using 1024 tx descriptors and 1024 rx descriptors em0: Using 0 rx queues 0 tx queues em0: Using MSI-X interrupts with 1 vectors em0: failed to allocate IRQ for rid 0, name irq0. em0: iflib_legacy_setup failed 12 device_attach: em0 attach returned 12 Disabling MSI-X in loader.conf (with a stock kernel) did the trick: em0: <Intel(R) PRO/1000 Network Connection> port 0xec00-0xec1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 17 at device 0.0 on pci11 em0: attach_pre capping queues at 1 em0: using 1024 tx descriptors and 1024 rx descriptors em0: msix_init qsets capped at 1 em0: System has MSIX disabled em0: Using an MSI interrupt em0: allocated for 1 tx_queues em0: allocated for 1 rx_queues em0: Ethernet address: bc:ae:c5:33:6d:98 em0: netmap queues/slots: TX 1/1024, RX 1/1024 Which is different but similar to how it was under 11.2: em0: <Intel(R) PRO/1000 Network Connection 7.6.1-k> port 0xec00-0xec1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 17 at device 0.0 on pci11 em0: Using an MSI interrupt em0: Ethernet address: bc:ae:c5:33:6d:98 em0: netmap queues/slots: TX 1/1024, RX 1/1024 Before we close we shall test the patch under head, I cannot at the moment as the Intels I have are all in production. Anyone?
It seems that the patch referenced by https://svnweb.freebsd.org/base?view=revision&revision=343578 is already in stable/12 https://github.com/freebsd/freebsd/commit/b1c306fca3679b61586013269f45b42ed6bb294b Well, at least partially. I'll test it out today and see if it's fixed in stable/12 without having to disable msix.
A commit references this bug: Author: marius Date: Sat Feb 9 11:58:41 UTC 2019 New revision: 343934 URL: https://svnweb.freebsd.org/changeset/base/343934 Log: - Remove the redundant device disabled hint handling; ever since r241119 that's performed globally by device_attach(9). - As for the EM-class of devices, em(4) supports multiple queues and MSI-X respectively only with 82574 devices. However, since the conversion to iflib(4), em(4) relies on the interrupt type fallback mechanism, i. e. MSI-X -> MSI -> INTx, of iflib(4) to figure out the interrupt type to use for the EM-class (as well as the IGB-class) of MACs. Moreover, despite the datasheet for 82583V not mentioning any support of MSI-X, there actually are 82583V devices out there that report a varying number of MSI-X messages as supported. The interrupt type fallback of iflib(4) is causing two failure modes depending on the actual number of MSI-X messages supported for such instances of 82583V: 1) With only one MSI-X message supported, none is left for the RX/TX queues as that one message gets assigned to the admin interrupt. Worse, later on - which will be addressed with a separate fix - iflib(4) interprets that one messages as MSI or INTx to be set up, but fails to actually do so as it has previously called pci_alloc_msix(9). [1, 2] 2) With more message supported, their distribution is okay but then em_if_msix_intr_assign() doesn't work for 82583V, with the interface being left in a non-working state, too. [3] Thus, let em_if_attach_pre() indicate to iflib(4) to try MSI-X with 82574 only, and at most MSI for the remainder of EM-class devices. While at it, remove "try_second_bar" as it's polarity inverted and not actually needed. - Remove code from em_if_timer() that effectively is a NOP since the conversion to iflib(4) ("trigger" is no longer read). While at it, let the comment for em_if_timer() reflect reality after said conversion. - Implement an ifdi_watchdog_reset method which only updates the em(4) "watchdog_events" counter but doesn't perform any reset, so that the em(4) "watchdog_timeouts" SYSCTL (iflib(4) doesn't provide a counterpart) reflects reality and these timeouts add to IFCOUNTER_OERRORS again after the iflib(4) conversion. - Remove the "mbuf_defrag_fail" and "tx_dma_fail" SYSCTLS; since the iflib(4) conversion, associated counters are disconnected, but iflib(4) provides "mbuf_defrag_failed" and "tx_map_failed" respectively as equivalents. - Move the description preceding lem_smartspeed() to the correct spot before em_reset() and bring back appropriate comments for {igb,em}_initialize_rss_mapping() and lem_smartspeed() lost in the iflib(4) conversion. - Adapt some other function descriptions and INIT_DEBUGOUT() use to match reality after the iflib(4) conversion. - Put the debugging message of em_enable_vectors_82574() (missed in r343578) under bootverbose, too. PR: 219428 [1], 235246 [2], 235147 [3] Reviewed by: erj (previous version) Differential Revision: https://reviews.freebsd.org/D19108 Changes: head/sys/dev/e1000/if_em.c head/sys/dev/e1000/if_em.h
A commit references this bug: Author: marius Date: Wed Feb 13 14:39:17 UTC 2019 New revision: 344098 URL: https://svnweb.freebsd.org/changeset/base/344098 Log: MFC: r343934 - Remove the redundant device disabled hint handling; ever since r241119 that's performed globally by device_attach(9). - As for the EM-class of devices, em(4) supports multiple queues and MSI-X respectively only with 82574 devices. However, since the conversion to iflib(4), em(4) relies on the interrupt type fallback mechanism, i. e. MSI-X -> MSI -> INTx, of iflib(4) to figure out the interrupt type to use for the EM-class (as well as the IGB-class) of MACs. Moreover, despite the datasheet for 82583V not mentioning any support of MSI-X, there actually are 82583V devices out there that report a varying number of MSI-X messages as supported. The interrupt type fallback of iflib(4) is causing two failure modes depending on the actual number of MSI-X messages supported for such instances of 82583V: 1) With only one MSI-X message supported, none is left for the RX/TX queues as that one message gets assigned to the admin interrupt. Worse, later on - which will be addressed with a separate fix - iflib(4) interprets that one messages as MSI or INTx to be set up, but fails to actually do so as it has previously called pci_alloc_msix(9). [1, 2] 2) With more message supported, their distribution is okay but then em_if_msix_intr_assign() doesn't work for 82583V, with the interface being left in a non-working state, too. [3] Thus, let em_if_attach_pre() indicate to iflib(4) to try MSI-X with 82574 only, and at most MSI for the remainder of EM-class devices. While at it, remove "try_second_bar" as it's polarity inverted and not actually needed. - Remove code from em_if_timer() that effectively is a NOP since the conversion to iflib(4) ("trigger" is no longer read). While at it, let the comment for em_if_timer() reflect reality after said conversion. - Implement an ifdi_watchdog_reset method which only updates the em(4) "watchdog_events" counter but doesn't perform any reset, so that the em(4) "watchdog_timeouts" SYSCTL (iflib(4) doesn't provide a counterpart) reflects reality and these timeouts add to IFCOUNTER_OERRORS again after the iflib(4) conversion. - Remove the "mbuf_defrag_fail" and "tx_dma_fail" SYSCTLS; since the iflib(4) conversion, associated counters are disconnected, but iflib(4) provides "mbuf_defrag_failed" and "tx_map_failed" respectively as equivalents. - Move the description preceding lem_smartspeed() to the correct spot before em_reset() and bring back appropriate comments for {igb,em}_initialize_rss_mapping() and lem_smartspeed() lost in the iflib(4) conversion. - Adapt some other function descriptions and INIT_DEBUGOUT() use to match reality after the iflib(4) conversion. - Put the debugging message of em_enable_vectors_82574() (missed in r343578) under bootverbose, too. PR: 219428 [1], 235246 [2], 235147 [3] Reviewed by: erj (previous version) Differential Revision: https://reviews.freebsd.org/D19108 Changes: _U stable/12/ stable/12/sys/dev/e1000/if_em.c stable/12/sys/dev/e1000/if_em.h
Close; thanks for the report!
(In reply to Marius Strobl from comment #9) Thank you so much Marius. Both hardware version are now working: 82583V: em0: <Intel(R) PRO/1000 Network Connection> port 0xec00-0xec1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 17 at device 0.0 on pci11 em0: Using 1024 tx descriptors and 1024 rx descriptors em0: Using an MSI interrupt em0: Ethernet address: bc:ae:c5:33:6d:98 em0: netmap queues/slots: TX 1/1024, RX 1/1024 82574L: em0: <Intel(R) PRO/1000 Network Connection> port 0xd800-0xd81f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 48 at device 0.0 on pci1 em0: Using 1024 tx descriptors and 1024 rx descriptors em0: Using 2 rx queues 2 tx queues em0: Using MSI-X interrupts with 3 vectors em0: Ethernet address: 00:25:90:20:e7:aa em0: netmap queues/slots: TX 2/1024, RX 2/1024