Bug 277605
| Summary: | FreeBSD 14-Release crashes on the RPI CM4 with PCIE to PCI bridge | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Base System | Reporter: | HP van Braam <hp> | ||||
| Component: | kern | Assignee: | freebsd-arm (Nobody) <freebsd-arm> | ||||
| Status: | Closed FIXED | ||||||
| Severity: | Affects Some People | Keywords: | crash | ||||
| Priority: | --- | ||||||
| Version: | 14.0-RELEASE | ||||||
| Hardware: | arm64 | ||||||
| OS: | Any | ||||||
| Attachments: |
|
||||||
I made a typo in the start of the messages: They should be: pcib0: <BCM2838-compatible PCI-express controller> mem 0x7d500000-0x7d50930f irq 80,81 on simplebus2 pcib0: hardware identifies as revision 0x304. I have since build CURRENT for the CM4, and that boots after some help on #freebsd-dev on irc. I managed to get a slightly more useful dump out of it now, I also uploaded the cores and kernel.debug here: https://cloud.tmm.cx/s/HxPeQw28g7N74RR pcib1: <PCI-PCI bridge> irq 91 at device 0.0 on pci0 pcib0: pci_host_generic_core_alloc_resource FAIL: type=3, rid=36, start=0000000000000000, end=00000000000fffff, count=0000000000100000, flags=142 pcib1: failed to allocate initial prefetch window: 0-0xfffff pci1: <OFW PCI bus> on pcib1 x0: 0xffffa000014393d8 x1: 0x0000000000000008 x2: 0xffff0000008843e2 (console_pausestr + 0x305a9) x3: 0x0000000000000153 x4: 0xffffa000fbd96480 x5: 0x0000000000000149 x6: 0xffffa00001dce500 x7: 0x0000000000000000 x8: 0xffff0000c0ee67b0 (ucom_cons_softc + 0xbfe16e30) x9: 0xffff000000a2cde8 (lock_class_mtx_sleep + 0x0) x10: 0x0000000000000001 x11: 0xffff000000d70100 (w_locklistdata + 0x410b8) x12: 0x0000000000000001 x13: 0xffff000000d70134 (w_locklistdata + 0x410ec) x14: 0x0000000000010000 x15: 0x0000000000000001 x16: 0x0000000000010000 x17: 0x0000000000000000 x18: 0xffff00009a9e7240 (ucom_cons_softc + 0x999178c0) x19: 0xffffa000014393f0 x20: 0xffffa000014393d8 x21: 0xffff000000d70100 (w_locklistdata + 0x410b8) x22: 0x0000000000008000 x23: 0x0000000000000000 x24: 0x0000000000000000 x25: 0x000000000000dead x26: 0x0000000000000001 x27: 0xffffa00001eaf680 x28: 0x000000006097de00 x29: 0xffff00009a9e7240 (ucom_cons_softc + 0x999178c0) sp: 0xffff00009a9e7240 lr: 0xffff000000404e5c (__mtx_unlock_flags + 0x58) elr: 0xffff0000004a1364 (witness_unlock + 0xf0) spsr: 0x0000000060000045 far: 0x0000000000000000 esr: 0x00000000bf000002 panic: Unhandled System Error cpuid = 0 time = 1710193245 KDB: stack backtrace: db_trace_self() at db_trace_self db_trace_self_wrapper() at db_trace_self_wrapper+0x38 vpanic() at vpanic+0x1a4 panic() at panic+0x48 do_serror() at do_serror+0x40 handle_serror() at handle_serror+0x40 --- system error, esr 0xbf000002 witness_unlock() at witness_unlock+0xf0 __mtx_unlock_flags() at __mtx_unlock_flags+0x54 bcm_pcib_read_config() at bcm_pcib_read_config+0x154 pci_read_device() at pci_read_device+0x88 pci_add_children() at pci_add_children+0x48 pci_attach() at pci_attach+0xe4 device_attach() at device_attach+0x3fc device_probe_and_attach() at device_probe_and_attach+0x80 bus_generic_attach() at bus_generic_attach+0x1c device_attach() at device_attach+0x3fc devctl2_ioctl() at devctl2_ioctl+0x4cc devfs_ioctl() at devfs_ioctl+0xd8 vn_ioctl() at vn_ioctl+0xc4 devfs_ioctl_f() at devfs_ioctl_f+0x24 kern_ioctl() at kern_ioctl+0x2e0 sys_ioctl() at sys_ioctl+0x120 do_el0_sync() at do_el0_sync+0x59c handle_el0_sync() at handle_el0_sync+0x48 --- exception, esr 0x56000000 Uptime: 46s A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=10e0c34bf842885b4bd78adbbdbd7fb00f133cb5 commit 10e0c34bf842885b4bd78adbbdbd7fb00f133cb5 Author: HP van Braam <hp@tmm.cx> AuthorDate: 2024-04-16 23:01:20 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2024-04-21 22:34:05 +0000 Enable L1SS handling on RPI4 pcib Thanks to @kevans91 for pointing me in the right direction. FreeBSD had the same bug as Linux (see https://bugzilla.kernel.org/show_bug.cgi?id=217276) where the ultimate solution was to honor the brcm,enable-l1ss FDT property. In current versions of the dtb files this property has been added by default. Without this on many, many pcie addin cards the pcib will Serror when trying to assert the clreq# pin on the pcie bus. Many cards do not have these hooked up. PR: 260131, 277638, 277605 Reviewed-by: emaste Signed-off-by: HP van Braam <hp@tmm.cx> Pull-request: https://github.com/freebsd/freebsd-src/pull/1179 sys/arm/broadcom/bcm2835/bcm2838_pci.c | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) |
Created attachment 249060 [details] Screenshot of the crash (captured over HDMI, slightly difficult to read) Hello, I have the following setup: * Raspberry Pi I/O board with a Compute Module 4 * The PCIe Slot has a PCIe to PCI bridge (ASMedia ASM1083) * PCI card in a slot connected to the bridge At first I just wrote FreeBSD 14-Release (aarch64-RPI) to the internal eMMC but this didn't boot at all. I then replaced the dtb files with the ones from the Raspberry Pi foundation, then I managed to boot into FreeBSD14, but got a crash. I got around this crash by adding devmatch_enable="NO" to rc.conf. I made some further edits to the config.txt to enable USB to work, enabling otg_mode=1. At this point the RPI CM4 boots into FreeBSD 14 and everything appears to work. I then added a PCI card to the bridge, at which point u-boot failed. I then rebuilt u-boot from git master, and installed that. Which allowed FreeBSD itself to boot. However, with the ASM1083 connected to the PCIe bus, even with no PCI card attached, the system crashes with the following: pcib0: <BCM2838-compatible PCI-express controller> mem 0x7d500000-0x7d50930f irq 00.01 on simplebus2 pcib0: hardware identifies as revision 0x384. pci0: <OFW PCI bus> on pcib0 pcib1: <PCI-PCI bridge> irq 91 at device 0.0 in pci0 pci1: <OFW PCI bus> on pcib1 x0: 0x000000000000dead x1: 0xffff000000f24840 x2: 0x0000000000000000 .... panic: Unhandled System Error cpuid = 0 time = 1 KDB: stack backtrace: #0 0xffff00000058d4b4 at kdb_backtrace+0x58 #1 0xffff0000004b8fcc at vpanic+0x19c #2 0xffff0000004b8e2c at panic+0x44 #3 0xffff000000896e8c at do_serror+0x3c #4 0xffff0000008742f8 at handle_serror+0x3c #5 0xffff000000a5826e at console_pausestr+0xa72 #6 0xffff0000009f47f3 at digits+0x4385 Uptime: 1s Resetting system... FWIW: I tried the same thing with Linux, and there it works. I'm also able to load a driver for a PCI card (an Adaptec 7980 based adapter in this case) and have it work. So in theory it at least could work? Thanks!