Bug 260127 - pci: Panic enumerating PI7C9X2G304SV PCIe Switch on Ten64 board (NXP QorIQ LS1088A): generic_bs_r_2() at generic_bs_r_2+0x4
Summary: pci: Panic enumerating PI7C9X2G304SV PCIe Switch on Ten64 board (NXP QorIQ LS...
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: arm (show other bugs)
Version: CURRENT
Hardware: arm64 Any
: --- Affects Some People
Assignee: freebsd-arm (Nobody)
URL:
Keywords: crash, needs-qa
Depends on:
Blocks:
 
Reported: 2021-11-29 23:56 UTC by Mathew McBride
Modified: 2021-11-30 20:58 UTC (History)
6 users (show)

See Also:
koobs: maintainer-feedback? (jhb)
koobs: maintainer-feedback? (Andrew)
koobs: mfc-stable13?
koobs: mfc-stable12?


Attachments
dmesg kernel panic with PI7C9X2G304SV enabled (15.66 KB, text/plain)
2021-11-29 23:56 UTC, Mathew McBride
no flags Details
dmesg successful boot when pcie@3500000 disabled (24.48 KB, text/plain)
2021-11-29 23:57 UTC, Mathew McBride
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Mathew McBride 2021-11-29 23:56:07 UTC
Created attachment 229803 [details]
dmesg kernel panic with PI7C9X2G304SV enabled

Thanks to the recent addition of various NXP/Freescale QorIQ drivers, particularly qoriq_dw_pci, as well as the upcoming DPAA2 driver by dsl@mcusim.org, FreeBSD is very close to being usable on the Traverse Technologies Ten64 board.

Unfortunately there appears to be an issue enumerating the Diodes (Pericom) PI7C9X2G304SV PCIe switch. This is used on this board to create two PCIe slots from one PCIe 2.0 lane (specifically, the MiniPCIe slots, see https://ten64doc.traverse.com.au/hardware/architecture/)

The panic occurs even when there are no downstream devices connected to the PI7C9X2G304SV.

When the PCIe controller that the PI7C9X2G304SV is connected to is disabled in the FDT, FreeBSD is able to boot and use the PCIe controller without issue, including using an NVMe SSD as the boot device.

Here is the PCIe topology from Linux:
# lspci -nn
0000:00:00.0 PCI bridge [0604]: Freescale Semiconductor Inc Device [1957:80c0] (rev 10)
0001:00:00.0 PCI bridge [0604]: Freescale Semiconductor Inc Device [1957:80c0] (rev 10)
0001:01:00.0 PCI bridge [0604]: Pericom Semiconductor Device [12d8:b304] (rev 01)
0001:02:01.0 PCI bridge [0604]: Pericom Semiconductor Device [12d8:b304] (rev 01)
0001:02:02.0 PCI bridge [0604]: Pericom Semiconductor Device [12d8:b304] (rev 01)
0002:00:00.0 PCI bridge [0604]: Freescale Semiconductor Inc Device [1957:80c0] (rev 10)
0002:01:00.0 Non-Volatile memory controller [0108]: Silicon Motion, Inc. SM2263EN/SM2263XT SSD Controller [126f:2263] (rev 03)

# lspci -t
-+-[0002:00]---00.0-[01-ff]----00.0
 +-[0001:00]---00.0-[01-ff]----00.0-[02-04]--+-01.0-[03]--
 |                                           \-02.0-[04]--
 \-[0000:00]---00.0-[01-ff]--

Note: The PCIe controller instance that the switch is connected to is at 0x3500000. The NVMe SSD is connected to pcie@3600000 (bus 0002:00:00.0 in Linux), pcie@3400000 is connected to the M.2 Key B ("cellular" slot) which is empty in this example.

By manipulating the FDT in U-Boot pcie@3500000 can be disabled allowing FreeBSD to boot:

setenv setup_distroboot_efi 'sf read 0x80001000 0x580000 0x40000 && fsl_mc lazyapply dpl 0x80001000; sf read $fdt_addr_r 0x600000 0x40000 && fdt addr $fdt_addr_r && fdt resize 10000 && fdt boardsetup && fdt set "/soc/spi@20c0000" "status" "disabled" && fdt set "/soc/pcie@3500000" "status" "disabled"'
saveenv
reboot
Comment 1 Mathew McBride 2021-11-29 23:57:50 UTC
Created attachment 229804 [details]
dmesg successful boot when pcie@3500000 disabled
Comment 2 Kornel Dulęba 2021-11-30 07:53:17 UTC
Can you try running the latest HEAD?
Your backtrace looks fairly similar to a bug I recently stumbled upon on a LS1028A board.
It should be fixed with https://cgit.freebsd.org/src/commit/?id=68cbe189fdd3c572476f8af9219a5d335f05b51a.
Comment 3 Dmitry Salychev 2021-11-30 08:30:58 UTC
(In reply to Kornel Dulęba from comment #2)

I'll give it a try today evening.
Comment 4 Mathew McBride 2021-11-30 09:49:41 UTC
(In reply to Kornel Dulęba from comment #2)
Wow, great timing!
Confirming this fixes the panic for me, the kernel is able to boot correctly after this change.

Devices behind the switch work as well, e.g I can boot from an NVMe attached to the switch (with a MiniPCIe<->M.2 adaptor)
Comment 5 Bjoern A. Zeeb freebsd_committer 2021-11-30 11:28:54 UTC
Fixed already in HEAD.
Comment 6 Dmitry Salychev 2021-11-30 20:58:39 UTC
Works for me on Ten64 as well.