Bug 243118 - sysutils/rpi-firmware: rpi4 updates for rpi3-psci-monitor severely slows down booting on 3B+
Summary: sysutils/rpi-firmware: rpi4 updates for rpi3-psci-monitor severely slows down...
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: arm64 Any
: --- Affects Only Me
Assignee: Kyle Evans
Depends on:
Reported: 2020-01-05 18:21 UTC by Tom Yan
Modified: 2020-08-27 17:21 UTC (History)
3 users (show)

See Also:

test set of rpi3-psci-monitor sources and binaries (50.43 KB, application/gzip)
2020-01-05 18:21 UTC, Tom Yan
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Tom Yan 2020-01-05 18:21:15 UTC
Created attachment 210471 [details]
test set of rpi3-psci-monitor sources and binaries

Booting on a Raspberry Pi 3B+ is extremely slow (could take 30-40 minutes) before the kernel kicks in.

Here's the situation:
- `12.1-RELEASE` works fine
- `12.1-STABLE` / `13.0-CURRENT` snapshots (since at least / as of `20191226`) are broken

It is caused by armstub8.bin updates and I've done some dissecting. The differences between the one in 12.1-RELEASE (2b8890a) and the one in the snapshots (6bfbaff) are:
- new OSC_FREQ
- L2 read/write cache latency setting
Either of the first two changes alone can slow down booting to different severity at different point. Combining them would make booting intolerably slow. The third change doesn't seem to be problematic so far.

Attached is a set of rpi3-psci-monitor sources and binaries I built with gcc-arm-9.2-2019.12-aarch64-aarch64-none-elf from https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-a/downloads:
- rpi3-psci-monitor-2b8890a is equivalent to the one in 12.1-RELEASE
- rpi3-psci-monitor-6bfbaff is equivalent to the one in the snapshots
- rpi3-psci-monitor-2b8890a-{a,b} can show that either of the new values of LOCAL_CONTROL and LOCAL_PRESCALER or OSC_FREQ could slow down booting / time to different severity at different point
- rpi3-psci-monitor-6bfbaff-c can show that L2 read/write cache latency setting, the only remaining difference of the armstub8.bin in 12.1-RELEASE and the one in the snapshots, is not the cause of the issue; although this does not mean it will not be potentially problematic on other aspects if done "generally".

I've already created PRs on https://github.com/gonzoua/rpi3-psci-monitor/pull/6 and https://github.com/raspberrypi/tools/pull/105.

(@kevans91, the armstub8.bin you sent me is the one in 12.1-RELEASE with the first two changes)
Comment 1 Bugzilla Automation freebsd_committer 2020-01-05 18:21:15 UTC
Maintainer informed via mail
Comment 2 Kyle Evans freebsd_committer 2020-01-05 18:50:26 UTC
(In reply to Tom Yan from comment #0)

A-ha. Try this stub: https://people.freebsd.org/~kevans/armstub8.bin (same location as the last one, different binary; sha512 follows)

SHA512 (armstub8.bin) = 643d66ad22d45cac568cf58861959c046a325c9081c41ea59915b7ce8deae6f0bc9234b5c9d9c81a5a2361a0f1c13cefc5c2de891a1c4f8e956759ea989eb7cf

The changes we made reflect upstream, but if this improves things then I'll shoot them a pull request.
Comment 3 Kyle Evans freebsd_committer 2020-01-05 18:52:31 UTC
Take, since I almost certainly caused it.
Comment 4 Tom Yan 2020-01-05 19:45:58 UTC
(In reply to Kyle Evans from comment #2)
I already did. This one is byte-to-byte identical to the one in 12.1-RELEASE. I don't know why I had problems with it when I place it into one of the snapshots (without also replacing u-boot.bin). I specifically re-tested this one (in addition to 2b8890a I built) with both STABLE and CURRENT 20191226 snapshots and couldn't reproduce it anymore. Maybe it was just an incidental pi haywire.

In case you missed it, I've already created an PR upstream. I reported here just to see if maybe you want to merge it downstream beforehand. (You know, things are pretty bad with the snapshots. I plan on bisecting the device tree starting from tomorrow btw.)

Or maybe we should wait until your 3A+ arrive and see, just to make sure it isn't merely my 3B+ happen to be broken, but indeed a BCM2837B0 thing (which kind of explains why the original author of the rpi4 updates didn't think the problematic changes need to be "isolated").
Comment 5 Kyle Evans freebsd_committer 2020-08-27 17:21:06 UTC
This was addressed in ports r528896 after Tom's excellent work was accepted; thanks!