Hello, SUMMARY: I have not been able to upgrade past FreeBSD 10.3 as since versions 11.0 -> 13.0 devices behind my fileserver's PCIe Switch are not detected. The switch itself is detected, which is a PEX 8664 PCIe Switch, link below. -- I'm using a Dell M610x blade server, which has a PEX/Broadcom 8664 Switch in it to provide 2 PCIe slots, which I use to run SAS controllers to a drive array. This arrangement has worked perfectly since the early days of 7/8.0, and I've pretty much ran each version of FreeBSD since then, until 11.0 came out. This is when I could no longer see/detect devices plugged into the PCIe slots. Things were busy, FreeBSD 10.2 worked well, so it was just left this way for years. Now I do need to upgrade software-wise to 13.0, so I've invested some time tracking down what is happening, with the hopes that a knowledgeable person can fix this up. - I have confirmed with 10.3-RELEASE and a 11.0-RELEASE mini-iso boot that the problem starts in 11.0, and I can confirm it's still an issue in 12.0 and 13.0. I also boot into a 10.2-RELEASE because I happen to have that handy. Everything works/detects fine in 10.3 or lower. - These are GENERIC kernels, fresh from the iso/img I boot from, nothing custom. - I currently have a 10.2 (or 10.3) boot and a 13.0 boot ready to swap back and forth to try and isolate / probe info. I'll attach a few lspci dumps after this message. - The card in question is a LSI 2008 Chipset, mps0 - But it's nothing do do with that card/brand/config. I have also installed an Intel X540 Network card in the PCIe slots and find the same issue - There's nothing behind the PEC 8664 in FreeBSD 13.0, and no problems in 10.2. I have also confirmed that 13.0 boots up and detects the mps0 without issue when I use a different system without the PEX 8664 - At some point in my research I came across a similar issue where the developer mentioned that for some reason they were not scanning behind the switch. It was a different chipset and issue, and I don't have that thread handy at the moment, I will post if I come across it again. Ref: https://www.broadcom.com/products/pcie-switches-bridges/pcie-switches/pex8664 I'm rusty, but fairly knowledgeable and useful if you need me to compile something, install patches, etc. I could give SSH access, but sending me a patch to test may be quicker. lspci dumps in the next message. Thanks.
Some lspci commands I used to try and narrow this down: 1) This is the 10.2 System. You can see both the Intel 82599ES and the LSI 2008 cards. Ignore the MegaRAID, that's a built in PERC card that's always detected (different bus): lspci -vvnnt -[0000:00]-+-00.0 Intel Corporation 5500 I/O Hub to ESI Port [8086:3403] +-01.0-[01]--+-00.0 Broadcom Corporation NetXtreme II BCM5709S Gigabit Ethernet [14e4:163a] | \-00.1 Broadcom Corporation NetXtreme II BCM5709S Gigabit Ethernet [14e4:163a] +-07.0-[03-09]----00.0-[04-09]--+-00.0-[05]-- | +-01.0-[06]-- | +-02.0-[07]-- | +-04.0-[08]--+-00.0 Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] | | \-00.1 Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] | \-10.0-[09]----00.0 LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] +-14.0 Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers [8086:342e] +-14.1 Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers [8086:3422] +-14.2 Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers [8086:3423] +-16.0 Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3430] +-16.1 Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3431] +-16.2 Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3432] +-16.3 Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3433] +-16.4 Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3429] +-16.5 Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:342a] +-16.6 Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:342b] +-16.7 Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:342c] +-1a.0 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 [8086:2937] +-1a.1 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 [8086:2938] +-1a.7 Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 [8086:293c] +-1c.0-[02]----00.0 LSI Logic / Symbios Logic MegaRAID SAS 1078 [1000:0060] +-1d.0 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 [8086:2934] +-1d.1 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 [8086:2935] +-1d.2 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 [8086:2936] +-1d.7 Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 [8086:293a] +-1e.0-[0a]----03.0 Matrox Electronics Systems Ltd. MGA G200eW WPCM450 [102b:0532] +-1f.0 Intel Corporation 82801IB (ICH9) LPC Interface Controller [8086:2918] \-1f.5 Intel Corporation 82801I (ICH9 Family) 2 port SATA Controller [IDE mode] [8086:2926] 2) This is the same command in 13.0, same hardware: lspci -vvnnt -[0000:00]-+-00.0 Intel Corporation 5500 I/O Hub to ESI Port [8086:3403] +-01.0-[01]--+-00.0 Broadcom Inc. and subsidiaries NetXtreme II BCM5709S Gigabit Ethernet [14e4:163a] | \-00.1 Broadcom Inc. and subsidiaries NetXtreme II BCM5709S Gigabit Ethernet [14e4:163a] +-07.0-[03-09]----00.0-[04-09]--+-00.0-[05]-- | +-01.0-[06]-- | +-02.0-[07]-- | +-04.0-[08]-- | \-10.0-[09]-- +-14.0 Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers [8086:342e] +-14.1 Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers [8086:3422] +-14.2 Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers [8086:3423] +-16.0 Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3430] +-16.1 Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3431] +-16.2 Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3432] +-16.3 Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3433] +-16.4 Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3429] +-16.5 Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:342a] +-16.6 Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:342b] +-16.7 Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:342c] +-1a.0 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 [8086:2937] +-1a.1 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 [8086:2938] +-1a.7 Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 [8086:293c] +-1c.0-[02]----00.0 Broadcom / LSI MegaRAID SAS 1078 [1000:0060] +-1d.0 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 [8086:2934] +-1d.1 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 [8086:2935] +-1d.2 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 [8086:2936] +-1d.7 Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 [8086:293a] +-1e.0-[0a]----03.0 Matrox Electronics Systems Ltd. MGA G200eW WPCM450 [102b:0532] +-1f.0 Intel Corporation 82801IB (ICH9) LPC Interface Controller [8086:2918] \-1f.5 Intel Corporation 82801I (ICH9 Family) 2 port SATA Controller [IDE mode] [8086:2926] 3) on 13.0 a quick summary: lspci -M 00:00.0 Host bridge: Intel Corporation 5500 I/O Hub to ESI Port (rev 13) 00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13) ## 00:01.0 is a bridge from 00 to 01-01 00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13) ## 00:07.0 is a bridge from 00 to 03-09 00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 13) 00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13) 00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13) 00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13) 00:16.1 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13) 00:16.2 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13) 00:16.3 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13) 00:16.4 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13) 00:16.5 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13) 00:16.6 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13) 00:16.7 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13) 00:1a.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 02) 00:1a.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 02) 00:1a.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 02) 00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 (rev 02) ## 00:1c.0 is a bridge from 00 to 02-02 00:1d.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 02) 00:1d.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 02) 00:1d.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 02) 00:1d.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 02) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) ## 00:1e.0 is a bridge from 00 to 0a-0a 00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02) 00:1f.5 IDE interface: Intel Corporation 82801I (ICH9 Family) 2 port SATA Controller [IDE mode] (rev 02) 01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709S Gigabit Ethernet (rev 20) 01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709S Gigabit Ethernet (rev 20) 02:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS 1078 (rev 04) 03:00.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa) ## 03:00.0 is a bridge from 03 to 04-09 04:00.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa) ## 04:00.0 is a bridge from 04 to 05-05 04:01.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa) ## 04:01.0 is a bridge from 04 to 06-06 04:02.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa) ## 04:02.0 is a bridge from 04 to 07-07 04:04.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa) ## 04:04.0 is a bridge from 04 to 08-08 04:10.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa) ## 04:10.0 is a bridge from 04 to 09-09 0a:03.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a) Summary of buses: 00: Primary host bus 1e.0 Bridge to 0a-0a 1c.0 Bridge to 02-02 07.0 Bridge to 03-09 01.0 Bridge to 01-01 01: Entered via 00:01.0 02: Entered via 00:1c.0 03: Entered via 00:07.0 00.0 Bridge to 04-09 04: Entered via 03:00.0 10.0 Bridge to 09-09 04.0 Bridge to 08-08 02.0 Bridge to 07-07 01.0 Bridge to 06-06 00.0 Bridge to 05-05 0a: Entered via 00:1e.0 4) Now a dump of the switch I think is in front of my devices in question: lspci -vv -s :04:10.0 04:10.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 255 Bus: primary=04, secondary=09, subordinate=09, sec-latency=0 I/O behind bridge: 0000e000-0000efff [size=4K] Memory behind bridge: df000000-df1fffff [size=2M] Prefetchable memory behind bridge: [disabled] Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity+ SERR+ NoISA+ VGA- VGA16- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+ Address: 00000000fee22000 Data: 0030 Masking: 00000000 Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 2048 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq+ RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 128 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- LnkCap: Port #16, Speed 5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <2us, L1 <2us ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp- LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s (ok), Width x8 (downgraded) TrErr- Train- SlotClk- DLActive+ BWMgmt- ABWMgmt- SltCap: AttnBtn+ PwrCtrl+ MRL+ AttnInd+ PwrInd+ HotPlug+ Surprise- Slot #240, PowerLimit 25.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn+ PwrFlt+ MRL+ PresDet+ CmdCplt+ HPIrq+ LinkChg+ Control: AttnInd Off, PwrInd Off, Power+ Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock- Changed: MRL- PresDet- LinkState- DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR- 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- ARIFwd+ AtomicOpsCap: Routing- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled, ARIFwd- AtomicOpsCtl: EgressBlck- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1- EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch Capabilities: [100 v1] Device Serial Number aa-86-00-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol+ UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ AdvNonFatalErr+ AERCap: First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 00000000 00000000 00000000 00000000 Capabilities: [138 v1] Power Budgeting <?> Capabilities: [148 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=8 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=03 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff Status: NegoPending- InProgress- Port Arbitration Table <?> Capabilities: [e00 v1] Multicast McastCap: MaxGroups 64, ECRCRegen+ McastCtl: NumGroups 1, Enable- McastBAR: IndexPos 0, BaseAddr 0000000000000000 McastReceiveVec: 0000000000000000 McastBlockAllVec: 0000000000000000 McastBlockUntransVec: 0000000000000000 McastOverlayBAR: OverlaySize 0 (disabled), BaseAddr 0000000000000000 Capabilities: [f24 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> 5) Back to 10.2 to dump all the bridges: root@pool31:~ # lspci -vv | grep bridge 00:00.0 Host bridge: Intel Corporation 5500 I/O Hub to ESI Port (rev 13) 00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13) (prog-if 00 [Normal decode]) Memory behind bridge: da000000-ddffffff 00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13) (prog-if 00 [Normal decode]) I/O behind bridge: 0000e000-0000efff Memory behind bridge: df000000-df2fffff 00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 (rev 02) (prog-if 00 [Normal decode]) I/O behind bridge: 0000f000-0000ffff Memory behind bridge: df300000-df3fffff 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) (prog-if 01 [Subtractive decode]) Memory behind bridge: de000000-deffffff Prefetchable memory behind bridge: 00000000d9800000-00000000d9ffffff 00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02) 03:00.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode]) I/O behind bridge: 0000e000-0000efff Memory behind bridge: df000000-df1fffff 04:00.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode]) 04:01.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode]) 04:02.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode]) 04:04.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode]) 04:10.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode]) I/O behind bridge: 0000e000-0000efff Memory behind bridge: df000000-df1fffff After this I think I'm just generating noise, as I'm not sure what info is needed. I await instruction. Thank you for reading this far.
Hello, this is still an issue, although I don't know a polite way to escalate this to the attention of the proper people.
I suspect a verbose dmesg would be useful. "boot -v" from the loader prompt will cause the kernel to generate lots of debug output when enumerating devices. One dmesg each from a working kernel and a non-working kernel would be ideal.
I think the switch is claiming to support hot plug but then claiming that no child devices are present. In particular: SltCap: AttnBtn+ PwrCtrl+ MRL+ AttnInd+ PwrInd+ HotPlug+ Surprise- Slot #240, PowerLimit 25.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn+ PwrFlt+ MRL+ PresDet+ CmdCplt+ HPIrq+ LinkChg+ Control: AttnInd Off, PwrInd Off, Power+ Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock- Changed: MRL- PresDet- LinkState- Notice HotPlug+ in SltCap, but PresDet- (presence detect) in SltSta. Probably these slots aren't really hot plug and your motherboard failed to write up the pins correctly for this switch to wire PD as always set. You can try setting 'hw.pci.enable_pcie_hp=0' in loader.conf or at the loader prompt as a workaround to test that theory. We already have one quirk to ignore the MRL if the DataLink Layer is active. We could similarly perhaps ignore HotPlug if the DataLink Layer is active bit PD is clear. If the tunable works, then this patch might work as a quirk: diff --git a/sys/dev/pci/pci_pci.c b/sys/dev/pci/pci_pci.c index 2b86f2f50e11..5674bcd69fa7 100644 --- a/sys/dev/pci/pci_pci.c +++ b/sys/dev/pci/pci_pci.c @@ -967,21 +967,22 @@ pcib_probe_hotplug(struct pcib_softc *sc) return; /* - * Some devices report that they have an MRL when they actually - * do not. Since they always report that the MRL is open, child - * devices would be ignored. Try to detect these devices and - * ignore their claim of HotPlug support. - * - * If there is an open MRL but the Data Link Layer is active, - * the MRL is not real. + * Some bridges use HotPlug-capable chips but are not really + * HotPlug bridges. The bridges can advertise inconsistent + * statuses such as cards not being present or an MRL that is + * always open, but the Data Link Layer is active. Ignore + * HotPlug on such bridges. */ - if ((sc->pcie_slot_cap & PCIEM_SLOT_CAP_MRLSP) != 0) { - link_sta = pcie_read_config(dev, PCIER_LINK_STA, 2); + link_sta = pcie_read_config(dev, PCIER_LINK_STA, 2); + if ((link_sta & PCIEM_LINK_STA_DL_ACTIVE) != 0) { slot_sta = pcie_read_config(dev, PCIER_SLOT_STA, 2); - if ((slot_sta & PCIEM_SLOT_STA_MRLSS) != 0 && - (link_sta & PCIEM_LINK_STA_DL_ACTIVE) != 0) { + + if ((slot_sta & PCIEM_SLOT_STA_PDS) == 0) + return; + + if ((sc->pcie_slot_cap & PCIEM_SLOT_CAP_MRLSP) != 0 && + (slot_sta & PCIEM_SLOT_STA_MRLSS) != 0) return; - } } /*
(In reply to Mark Johnston from comment #3) I will be returning attention to this soon, and will post the output.
(In reply to John Baldwin from comment #4) Thank you, I'll compile this in to my kernel and report back - Probably another few weeks until I'm able to do that.
(In reply to John Baldwin from comment #4) John, 'hw.pci.enable_pcie_hp=0' This works, thank you very much. I expect your assumption on why is correct, or at least in the right direction. I will be testing the patch at sometime in the future, there's some problems I have to deal with first, but I wanted to report back in case anyone else was stumped by this one. Thanks again, this was a major hurdle to moving off 10.2.