Bug 245981 - BCM57414 not initializing
Summary: BCM57414 not initializing
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.1-RELEASE
Hardware: amd64 Any
: --- Affects Many People
Assignee: freebsd-net (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-04-27 20:38 UTC by Ian Smith
Modified: 2020-06-15 17:03 UTC (History)
1 user (show)

See Also:


Attachments
dmesg for ticket (29.05 KB, text/plain)
2020-06-12 22:02 UTC, Ian Smith
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Ian Smith 2020-04-27 20:38:44 UTC
Hello,

I am using the following system:

540-BCKU : Broadcom 57414 Dual-Port 25GbE SFP28 LOM Mezz Card
405-AAMS : PERC H740P RAID Controller, 8G B NV Cache, Mini card
400-BDSS : 240GB SSD SATA Mixed Use 6Gbps 512e 2.5in Hot-Plug S4610 Dri ve
400-AVEZ : 2.4TB 10K RPM SAS 12Gbps 512e 2.5in Hot-plug Hard Drive
384-BCJD : PowerEdge R7515 Motherboard, with 2 x 1Gb Onboard LOM (BCM57 20)
379-BDSS : SAS/SATA Backplane
370-AEVR : 3200MT/s RDIMMs
370-AEVQ : 16GB RDIMM, 3200MT/s, Dual Ran k
338-BSWO : AMD 7302P 3GHz,16C/32T,128M,15 5W,3200
321-BERX : 2.5" Chassis with up to 24 Hot Plug Hard Drives
210-ASVQ : PowerEdge R7515 Server

The network adapter at the top is non-functional in the 12.1 release.

I am able to check for the Broadcom 57414 drivers via:
>>kldstat | grep if_bnxt

It returns the memory space loaded and then also shows it is functional.

If I try to see the device or bring it online it does not show nor work:
>>ifconfig does not show bnxt0 or bnxt1
>>ifconfig bnxt0 up does not enable or show the device

If I use

lspci | grep NetX it does show the PCI device as being loaded and lists details within pciconf if I map it to the device ID of pci02:00:0

But if I force unload the if_bnxt and then reload:
>>kldunload if_bnxt
>>kldload /boot/kernel/if_bnxt.ko

It states it fails due to " bus_dma_tag_create failed return: 22"

Am I doing something wrong or?
Comment 1 Ian Smith 2020-04-27 20:47:01 UTC
Sometimes when force unloading and reloading I do have a kernel panic that happens.


https://i.imgur.com/COwfFZY.png
Comment 2 Ryan Moeller freebsd_committer 2020-06-12 21:27:09 UTC
Can you post dmesg and the pciconf?
Comment 3 Ian Smith 2020-06-12 22:02:38 UTC
Created attachment 215503 [details]
dmesg for ticket
Comment 4 Ian Smith 2020-06-12 22:08:02 UTC
truenas# pciconf -l -BbcevV bge0
bge0@pci0:193:0:0:      class=0x020000 card=0x08fd1028 chip=0x165f14e4 rev=0x00 hdr=0x00
    vendor     = 'Broadcom Inc. and subsidiaries'
    device     = 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe'
    class      = network
    subclass   = ethernet
    bar   [10] = type Prefetchable Memory, range 64, base rxc7030000, size 65536, enabled
    bar   [18] = type Prefetchable Memory, range 64, base rxc7040000, size 65536, enabled
    bar   [20] = type Prefetchable Memory, range 64, base rxc7050000, size 65536, enabled
    cap 01[48] = powerspec 3  supports D0 D3  current D0
    cap 03[50] = VPD
    cap 05[58] = MSI supports 8 messages, 64 bit enabled with 1 message
    cap 11[a0] = MSI-X supports 17 messages
                 Table in map 0x20[0x0], PBA in map 0x20[0x1000]
    cap 10[ac] = PCI-Express 2 endpoint max data 128(256) FLR
                 link x1(x2) speed 5.0(5.0) ClockPM disabled
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0003[13c] = Serial 1 00003448ede7ee1a
    ecap 0004[150] = Power Budgeting 1
    ecap 0002[160] = VC 1 max VC0
     Corrected = Advisory Non-Fatal Error
    VPD ident  = 'Broadcom NetXtreme Gigabit Ethernet'
    VPD ro PN  = 'BCM95720'
    VPD ro MN  = '1028'
    VPD ro V0  = 'FFV21.60.2'
    VPD ro V1  = 'DSV1028VPDR.VER1.0'
    VPD ro V2  = 'NPY2'
    VPD ro V3  = 'PMT1'
    VPD ro V4  = 'NMVBroadcom Corp'
    VPD ro V5  = 'DTINIC'
    VPD ro V6  = 'DCM1001008d452101008d45'
truenas# clear
truenas# pciconf -l -BbcevV bge1
bge1@pci0:193:0:1:      class=0x020000 card=0x08fd1028 chip=0x165f14e4 rev=0x00 hdr=0x00
    vendor     = 'Broadcom Inc. and subsidiaries'
    device     = 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe'
    class      = network
    subclass   = ethernet
    bar   [10] = type Prefetchable Memory, range 64, base rxc7000000, size 65536, enabled
    bar   [18] = type Prefetchable Memory, range 64, base rxc7010000, size 65536, enabled
    bar   [20] = type Prefetchable Memory, range 64, base rxc7020000, size 65536, enabled
    cap 01[48] = powerspec 3  supports D0 D3  current D0
    cap 03[50] = VPD
    cap 05[58] = MSI supports 8 messages, 64 bit enabled with 1 message
    cap 11[a0] = MSI-X supports 17 messages
                 Table in map 0x20[0x0], PBA in map 0x20[0x1000]
    cap 10[ac] = PCI-Express 2 endpoint max data 128(256) FLR
                 link x1(x2) speed 5.0(5.0) ClockPM disabled
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0003[13c] = Serial 1 00003448ede7ee1b
    ecap 0004[150] = Power Budgeting 1
    ecap 0002[160] = VC 1 max VC0
     Corrected = Advisory Non-Fatal Error
    VPD ident  = 'Broadcom NetXtreme Gigabit Ethernet'
    VPD ro PN  = 'BCM95720'
    VPD ro MN  = '1028'
    VPD ro V0  = 'FFV21.60.2'
    VPD ro V1  = 'DSV1028VPDR.VER1.0'
    VPD ro V2  = 'NPY2'
    VPD ro V3  = 'PMT1'
    VPD ro V4  = 'NMVBroadcom Corp'
    VPD ro V5  = 'DTINIC'
    VPD ro V6  = 'DCM1001008d452101008d45'
truenas# pciconf -l -BbcevV bge0
bge0@pci0:193:0:0:      class=0x020000 card=0x08fd1028 chip=0x165f14e4 rev=0x00 hdr=0x00
    vendor     = 'Broadcom Inc. and subsidiaries'
    device     = 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe'
    class      = network
    subclass   = ethernet
    bar   [10] = type Prefetchable Memory, range 64, base rxc7030000, size 65536, enabled
    bar   [18] = type Prefetchable Memory, range 64, base rxc7040000, size 65536, enabled
    bar   [20] = type Prefetchable Memory, range 64, base rxc7050000, size 65536, enabled
    cap 01[48] = powerspec 3  supports D0 D3  current D0
    cap 03[50] = VPD
    cap 05[58] = MSI supports 8 messages, 64 bit enabled with 1 message
    cap 11[a0] = MSI-X supports 17 messages
                 Table in map 0x20[0x0], PBA in map 0x20[0x1000]
    cap 10[ac] = PCI-Express 2 endpoint max data 128(256) FLR
                 link x1(x2) speed 5.0(5.0) ClockPM disabled
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0003[13c] = Serial 1 00003448ede7ee1a
    ecap 0004[150] = Power Budgeting 1
    ecap 0002[160] = VC 1 max VC0
     Corrected = Advisory Non-Fatal Error
    VPD ident  = 'Broadcom NetXtreme Gigabit Ethernet'
    VPD ro PN  = 'BCM95720'
    VPD ro MN  = '1028'
    VPD ro V0  = 'FFV21.60.2'
    VPD ro V1  = 'DSV1028VPDR.VER1.0'
    VPD ro V2  = 'NPY2'
    VPD ro V3  = 'PMT1'
    VPD ro V4  = 'NMVBroadcom Corp'
    VPD ro V5  = 'DTINIC'
    VPD ro V6  = 'DCM1001008d452101008d45'
Comment 5 Ryan Moeller freebsd_committer 2020-06-12 22:12:08 UTC
pciconf should have been for bnxt not bge.
Comment 6 Ian Smith 2020-06-12 22:42:49 UTC
Not sure how to obtain that, does not like pciconf bnxt0 or 1
Comment 7 Ryan Moeller freebsd_committer 2020-06-12 22:55:16 UTC
If you do pciconf -l -BbcevV it should be somewhere in the long list. If you can't get it though that's probably okay, there were some pretty good clues in the dmesg log already.
Comment 8 Ryan Moeller freebsd_committer 2020-06-13 02:48:05 UTC
bnxt0: <Broadcom BCM57414 NetXtreme-E 10Gb/25Gb Ethernet> mem 0xe1510000-0xe151ffff,0xe1400000-0xe14fffff,0xe1522000-0xe1523fff irq 54 at device 0.0 numa-domain 7 on pci1
bnxt0: Using 256 TX descriptors and 256 RX descriptors
bnxt0: Using 0 RX queues 0 TX queues
bnxt0: Using MSI-X interrupts with 1 vectors
bnxt0: iflib_dma_alloc_align: bus_dma_tag_create failed: 22
bnxt0: Unable to allocate device TX queue
bnxt0: Unable to allocate queue memory
device_attach: bnxt0 attach returned 22

For some reason either the firmware is not reporting the number of queues correctly, or something else is causing the number of queues to be forced to 0. Unfortunately I don't have access to hardware for debugging this myself.
Comment 9 Ian Smith 2020-06-15 17:03:50 UTC
Testing within Ubuntu 18.04 LTS is mapping the queues correctly, although it is loading the driver modules within the kernel for FreeBSD it does not allow for mapping it?

Is there any additional steps we can take to identify or solve this allocation of queues?