Bug 256264 - Devices behind PEX 8664 PCIe Switch not detected since 11.0-RELEASE
Summary: Devices behind PEX 8664 PCIe Switch not detected since 11.0-RELEASE
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 13.0-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-bugs (Nobody)
URL:
Keywords: regression
Depends on:
Blocks:
 
Reported: 2021-05-30 13:48 UTC by Christopher Forgeron
Modified: 2021-09-30 13:44 UTC (History)
4 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Christopher Forgeron 2021-05-30 13:48:29 UTC
Hello,

 SUMMARY:

 I have not been able to upgrade past FreeBSD 10.3 as since versions 11.0 -> 13.0 devices behind my fileserver's PCIe Switch are not detected. The switch itself is detected, which is a PEX 8664 PCIe Switch, link below. 

 --

 I'm using a Dell M610x blade server, which has a PEX/Broadcom 8664 Switch in it to provide 2 PCIe slots, which I use to run SAS controllers to a drive array.

 This arrangement has worked perfectly since the early days of 7/8.0, and I've pretty much ran each version of FreeBSD since then, until 11.0 came out.

 This is when I could no longer see/detect devices plugged into the PCIe slots. Things were busy, FreeBSD 10.2 worked well, so it was just left this way for years.

 Now I do need to upgrade software-wise to 13.0, so I've invested some time tracking down what is happening, with the hopes that a knowledgeable person can fix this up. 

 - I have confirmed with 10.3-RELEASE and a 11.0-RELEASE mini-iso boot that the problem starts in 11.0, and I can confirm it's still an issue in 12.0 and 13.0. I also boot into a 10.2-RELEASE because I happen to have that handy.  Everything works/detects fine in 10.3 or lower. 

 - These are GENERIC kernels, fresh from the iso/img I boot from, nothing custom. 

 - I currently have a 10.2 (or 10.3) boot and a 13.0 boot ready to swap back and forth to try and isolate / probe info. I'll attach a few lspci dumps after this message. 

 - The card in question is a LSI 2008 Chipset, mps0 - But it's nothing do do with that card/brand/config. I have also installed an Intel X540 Network card in the PCIe slots and find the same issue - There's nothing behind the PEC 8664 in FreeBSD 13.0, and no problems in 10.2. I have also confirmed that 13.0 boots up and detects the mps0 without issue when I use a different system without the PEX 8664

 - At some point in my research I came across a similar issue where the developer mentioned that for some reason they were not scanning behind the switch. It was a different chipset and issue, and  I don't have that thread handy at the moment, I will post if I come across it again. 

Ref:
https://www.broadcom.com/products/pcie-switches-bridges/pcie-switches/pex8664

I'm rusty, but fairly knowledgeable and useful if you need me to compile something, install patches, etc. I could give SSH access, but sending me a patch to test may be quicker. 

lspci dumps in the next message.

Thanks.
Comment 1 Christopher Forgeron 2021-05-30 13:58:05 UTC
Some lspci commands I used to try and narrow this down:


1) This is the 10.2 System. You can see both the Intel 82599ES and the LSI 2008 cards. Ignore the MegaRAID, that's a built in PERC card that's always detected (different bus):

lspci -vvnnt
-[0000:00]-+-00.0  Intel Corporation 5500 I/O Hub to ESI Port [8086:3403]
           +-01.0-[01]--+-00.0  Broadcom Corporation NetXtreme II BCM5709S Gigabit Ethernet [14e4:163a]
           |            \-00.1  Broadcom Corporation NetXtreme II BCM5709S Gigabit Ethernet [14e4:163a]
           +-07.0-[03-09]----00.0-[04-09]--+-00.0-[05]--
           |                               +-01.0-[06]--
           |                               +-02.0-[07]--
           |                               +-04.0-[08]--+-00.0  Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb]
           |                               |            \-00.1  Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb]
           |                               \-10.0-[09]----00.0  LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072]
           +-14.0  Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers [8086:342e]
           +-14.1  Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers [8086:3422]
           +-14.2  Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers [8086:3423]
           +-16.0  Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3430]
           +-16.1  Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3431]
           +-16.2  Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3432]
           +-16.3  Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3433]
           +-16.4  Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3429]
           +-16.5  Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:342a]
           +-16.6  Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:342b]
           +-16.7  Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:342c]
           +-1a.0  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 [8086:2937]
           +-1a.1  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 [8086:2938]
           +-1a.7  Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 [8086:293c]
           +-1c.0-[02]----00.0  LSI Logic / Symbios Logic MegaRAID SAS 1078 [1000:0060]
           +-1d.0  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 [8086:2934]
           +-1d.1  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 [8086:2935]
           +-1d.2  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 [8086:2936]
           +-1d.7  Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 [8086:293a]
           +-1e.0-[0a]----03.0  Matrox Electronics Systems Ltd. MGA G200eW WPCM450 [102b:0532]
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller [8086:2918]
           \-1f.5  Intel Corporation 82801I (ICH9 Family) 2 port SATA Controller [IDE mode] [8086:2926]

2) This is the same command in 13.0, same hardware:

lspci -vvnnt
-[0000:00]-+-00.0  Intel Corporation 5500 I/O Hub to ESI Port [8086:3403]
           +-01.0-[01]--+-00.0  Broadcom Inc. and subsidiaries NetXtreme II BCM5709S Gigabit Ethernet [14e4:163a]
           |            \-00.1  Broadcom Inc. and subsidiaries NetXtreme II BCM5709S Gigabit Ethernet [14e4:163a]
           +-07.0-[03-09]----00.0-[04-09]--+-00.0-[05]--
           |                               +-01.0-[06]--
           |                               +-02.0-[07]--
           |                               +-04.0-[08]--
           |                               \-10.0-[09]--
           +-14.0  Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers [8086:342e]
           +-14.1  Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers [8086:3422]
           +-14.2  Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers [8086:3423]
           +-16.0  Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3430]
           +-16.1  Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3431]
           +-16.2  Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3432]
           +-16.3  Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3433]
           +-16.4  Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3429]
           +-16.5  Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:342a]
           +-16.6  Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:342b]
           +-16.7  Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:342c]
           +-1a.0  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 [8086:2937]
           +-1a.1  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 [8086:2938]
           +-1a.7  Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 [8086:293c]
           +-1c.0-[02]----00.0  Broadcom / LSI MegaRAID SAS 1078 [1000:0060]
           +-1d.0  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 [8086:2934]
           +-1d.1  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 [8086:2935]
           +-1d.2  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 [8086:2936]
           +-1d.7  Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 [8086:293a]
           +-1e.0-[0a]----03.0  Matrox Electronics Systems Ltd. MGA G200eW WPCM450 [102b:0532]
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller [8086:2918]
           \-1f.5  Intel Corporation 82801I (ICH9 Family) 2 port SATA Controller [IDE mode] [8086:2926]



3) on 13.0 a quick summary:

lspci -M
00:00.0 Host bridge: Intel Corporation 5500 I/O Hub to ESI Port (rev 13)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13)
## 00:01.0 is a bridge from 00 to 01-01
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)
## 00:07.0 is a bridge from 00 to 03-09
00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 13)
00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13)
00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13)
00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.1 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.2 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.3 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.4 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.5 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.6 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.7 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:1a.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 02)
00:1a.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 02)
00:1a.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 02)
00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 (rev 02)
## 00:1c.0 is a bridge from 00 to 02-02
00:1d.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 02)
00:1d.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 02)
00:1d.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 02)
00:1d.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
## 00:1e.0 is a bridge from 00 to 0a-0a
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
00:1f.5 IDE interface: Intel Corporation 82801I (ICH9 Family) 2 port SATA Controller [IDE mode] (rev 02)
01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709S Gigabit Ethernet (rev 20)
01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709S Gigabit Ethernet (rev 20)
02:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS 1078 (rev 04)
03:00.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa)
## 03:00.0 is a bridge from 03 to 04-09
04:00.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa)
## 04:00.0 is a bridge from 04 to 05-05
04:01.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa)
## 04:01.0 is a bridge from 04 to 06-06
04:02.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa)
## 04:02.0 is a bridge from 04 to 07-07
04:04.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa)
## 04:04.0 is a bridge from 04 to 08-08
04:10.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa)
## 04:10.0 is a bridge from 04 to 09-09
0a:03.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)

Summary of buses:

00: Primary host bus
        1e.0 Bridge to 0a-0a
        1c.0 Bridge to 02-02
        07.0 Bridge to 03-09
        01.0 Bridge to 01-01
01: Entered via 00:01.0
02: Entered via 00:1c.0
03: Entered via 00:07.0
        00.0 Bridge to 04-09
04: Entered via 03:00.0
        10.0 Bridge to 09-09
        04.0 Bridge to 08-08
        02.0 Bridge to 07-07
        01.0 Bridge to 06-06
        00.0 Bridge to 05-05
0a: Entered via 00:1e.0

4) Now a dump of the switch I think is in front of my devices in question:

lspci -vv -s :04:10.0
04:10.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 255
        Bus: primary=04, secondary=09, subordinate=09, sec-latency=0
        I/O behind bridge: 0000e000-0000efff [size=4K]
        Memory behind bridge: df000000-df1fffff [size=2M]
        Prefetchable memory behind bridge: [disabled]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity+ SERR+ NoISA+ VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
                Address: 00000000fee22000  Data: 0030
                Masking: 00000000  Pending: 00000000
        Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00
                DevCap: MaxPayload 2048 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 256 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #16, Speed 5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <2us, L1 <2us
                        ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp-
                LnkCtl: ASPM Disabled; Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s (ok), Width x8 (downgraded)
                        TrErr- Train- SlotClk- DLActive+ BWMgmt- ABWMgmt-
                SltCap: AttnBtn+ PwrCtrl+ MRL+ AttnInd+ PwrInd+ HotPlug+ Surprise-
                        Slot #240, PowerLimit 25.000W; Interlock- NoCompl-
                SltCtl: Enable: AttnBtn+ PwrFlt+ MRL+ PresDet+ CmdCplt+ HPIrq+ LinkChg+
                        Control: AttnInd Off, PwrInd Off, Power+ Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
                        Changed: MRL- PresDet- LinkState-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- ARIFwd+
                         AtomicOpsCap: Routing-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled, ARIFwd-
                         AtomicOpsCtl: EgressBlck-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch
        Capabilities: [100 v1] Device Serial Number aa-86-00-10-b5-df-0e-00
        Capabilities: [fb4 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol+
                UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ AdvNonFatalErr+
                AERCap: First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [138 v1] Power Budgeting <?>
        Capabilities: [148 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=8
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=03 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff
                        Status: NegoPending- InProgress-
                        Port Arbitration Table <?>
        Capabilities: [e00 v1] Multicast
                McastCap: MaxGroups 64, ECRCRegen+
                McastCtl: NumGroups 1, Enable-
                McastBAR: IndexPos 0, BaseAddr 0000000000000000
                McastReceiveVec:      0000000000000000
                McastBlockAllVec:     0000000000000000
                McastBlockUntransVec: 0000000000000000
                McastOverlayBAR: OverlaySize 0 (disabled), BaseAddr 0000000000000000
        Capabilities: [f24 v1] Access Control Services
                ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>

5) Back to 10.2 to dump all the bridges:

root@pool31:~ # lspci -vv | grep bridge
00:00.0 Host bridge: Intel Corporation 5500 I/O Hub to ESI Port (rev 13)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13) (prog-if 00 [Normal decode])
        Memory behind bridge: da000000-ddffffff
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13) (prog-if 00 [Normal decode])
        I/O behind bridge: 0000e000-0000efff
        Memory behind bridge: df000000-df2fffff
00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 (rev 02) (prog-if 00 [Normal decode])
        I/O behind bridge: 0000f000-0000ffff
        Memory behind bridge: df300000-df3fffff
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) (prog-if 01 [Subtractive decode])
        Memory behind bridge: de000000-deffffff
        Prefetchable memory behind bridge: 00000000d9800000-00000000d9ffffff
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
03:00.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode])
        I/O behind bridge: 0000e000-0000efff
        Memory behind bridge: df000000-df1fffff
04:00.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode])
04:01.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode])
04:02.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode])
04:04.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode])
04:10.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode])
        I/O behind bridge: 0000e000-0000efff
        Memory behind bridge: df000000-df1fffff  



After this I think I'm just generating noise, as I'm not sure what info is needed. I await instruction. 

Thank you for reading this far.
Comment 2 Christopher Forgeron 2021-06-19 14:25:31 UTC
Hello, this is still an issue, although I don't know a polite way to escalate this to the attention of the proper people.
Comment 3 Mark Johnston freebsd_committer 2021-06-19 14:32:18 UTC
I suspect a verbose dmesg would be useful.  "boot -v" from the loader prompt will cause the kernel to generate lots of debug output when enumerating devices.  One dmesg each from a working kernel and a non-working kernel would be ideal.
Comment 4 John Baldwin freebsd_committer freebsd_triage 2021-09-28 16:45:20 UTC
I think the switch is claiming to support hot plug but then claiming that no child devices are present.  In particular:

                SltCap: AttnBtn+ PwrCtrl+ MRL+ AttnInd+ PwrInd+ HotPlug+ Surprise-
                        Slot #240, PowerLimit 25.000W; Interlock- NoCompl-
                SltCtl: Enable: AttnBtn+ PwrFlt+ MRL+ PresDet+ CmdCplt+ HPIrq+ LinkChg+
                        Control: AttnInd Off, PwrInd Off, Power+ Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
                        Changed: MRL- PresDet- LinkState-

Notice HotPlug+ in SltCap, but PresDet- (presence detect) in SltSta.  Probably these slots aren't really hot plug and your motherboard failed to write up the pins correctly for this switch to wire PD as always set.  You can try setting 'hw.pci.enable_pcie_hp=0' in loader.conf or at the loader prompt as a workaround to test that theory.

We already have one quirk to ignore the MRL if the DataLink Layer is active.  We could similarly perhaps ignore HotPlug if the DataLink Layer is active bit PD is clear.

If the tunable works, then this patch might work as a quirk:

diff --git a/sys/dev/pci/pci_pci.c b/sys/dev/pci/pci_pci.c
index 2b86f2f50e11..5674bcd69fa7 100644
--- a/sys/dev/pci/pci_pci.c
+++ b/sys/dev/pci/pci_pci.c
@@ -967,21 +967,22 @@ pcib_probe_hotplug(struct pcib_softc *sc)
 		return;
 
 	/*
-	 * Some devices report that they have an MRL when they actually
-	 * do not.  Since they always report that the MRL is open, child
-	 * devices would be ignored.  Try to detect these devices and
-	 * ignore their claim of HotPlug support.
-	 *
-	 * If there is an open MRL but the Data Link Layer is active,
-	 * the MRL is not real.
+	 * Some bridges use HotPlug-capable chips but are not really
+	 * HotPlug bridges.  The bridges can advertise inconsistent
+	 * statuses such as cards not being present or an MRL that is
+	 * always open, but the Data Link Layer is active.  Ignore
+	 * HotPlug on such bridges.
 	 */
-	if ((sc->pcie_slot_cap & PCIEM_SLOT_CAP_MRLSP) != 0) {
-		link_sta = pcie_read_config(dev, PCIER_LINK_STA, 2);
+	link_sta = pcie_read_config(dev, PCIER_LINK_STA, 2);
+	if ((link_sta & PCIEM_LINK_STA_DL_ACTIVE) != 0) {
 		slot_sta = pcie_read_config(dev, PCIER_SLOT_STA, 2);
-		if ((slot_sta & PCIEM_SLOT_STA_MRLSS) != 0 &&
-		    (link_sta & PCIEM_LINK_STA_DL_ACTIVE) != 0) {
+
+		if ((slot_sta & PCIEM_SLOT_STA_PDS) == 0)
+			return;
+
+		if ((sc->pcie_slot_cap & PCIEM_SLOT_CAP_MRLSP) != 0 &&
+		    (slot_sta & PCIEM_SLOT_STA_MRLSS) != 0)
 			return;
-		}
 	}
 
 	/*
Comment 5 Christopher Forgeron 2021-09-30 13:42:53 UTC
(In reply to Mark Johnston from comment #3)
I will be returning attention to this soon, and will post the output.
Comment 6 Christopher Forgeron 2021-09-30 13:43:48 UTC
(In reply to John Baldwin from comment #4)
Thank you, I'll compile this in to my kernel and report back - Probably another few weeks until I'm able to do that.
Comment 7 Christopher Forgeron 2021-09-30 13:44:03 UTC
(In reply to John Baldwin from comment #4)
Thank you, I'll compile this in to my kernel and report back - Probably another few weeks until I'm able to do that.