Bug 243118

Summary: sysutils/rpi-firmware: rpi4 updates for rpi3-psci-monitor severely slows down booting on 3B+
Product: Ports & Packages Reporter: Tom Yan <tom.ty89>
Component: Individual Port(s)Assignee: Kyle Evans <kevans>
Status: Closed FIXED    
Severity: Affects Only Me CC: gonzo, kevans, uboot
Priority: ---    
Version: Latest   
Hardware: arm64   
OS: Any   
Attachments:
Description Flags
test set of rpi3-psci-monitor sources and binaries none

Description Tom Yan 2020-01-05 18:21:15 UTC
Created attachment 210471 [details]
test set of rpi3-psci-monitor sources and binaries

Booting on a Raspberry Pi 3B+ is extremely slow (could take 30-40 minutes) before the kernel kicks in.

Here's the situation:
- `12.1-RELEASE` works fine
- `12.1-STABLE` / `13.0-CURRENT` snapshots (since at least / as of `20191226`) are broken

It is caused by armstub8.bin updates and I've done some dissecting. The differences between the one in 12.1-RELEASE (2b8890a) and the one in the snapshots (6bfbaff) are:
- new LOCAL_CONTROL and LOCAL_PRESCALER
- new OSC_FREQ
- L2 read/write cache latency setting
Either of the first two changes alone can slow down booting to different severity at different point. Combining them would make booting intolerably slow. The third change doesn't seem to be problematic so far.

Attached is a set of rpi3-psci-monitor sources and binaries I built with gcc-arm-9.2-2019.12-aarch64-aarch64-none-elf from https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-a/downloads:
- rpi3-psci-monitor-2b8890a is equivalent to the one in 12.1-RELEASE
- rpi3-psci-monitor-6bfbaff is equivalent to the one in the snapshots
- rpi3-psci-monitor-2b8890a-{a,b} can show that either of the new values of LOCAL_CONTROL and LOCAL_PRESCALER or OSC_FREQ could slow down booting / time to different severity at different point
- rpi3-psci-monitor-6bfbaff-c can show that L2 read/write cache latency setting, the only remaining difference of the armstub8.bin in 12.1-RELEASE and the one in the snapshots, is not the cause of the issue; although this does not mean it will not be potentially problematic on other aspects if done "generally".

I've already created PRs on https://github.com/gonzoua/rpi3-psci-monitor/pull/6 and https://github.com/raspberrypi/tools/pull/105.

(@kevans91, the armstub8.bin you sent me is the one in 12.1-RELEASE with the first two changes)
Comment 1 Bugzilla Automation freebsd_committer 2020-01-05 18:21:15 UTC
Maintainer informed via mail
Comment 2 Kyle Evans freebsd_committer 2020-01-05 18:50:26 UTC
(In reply to Tom Yan from comment #0)

A-ha. Try this stub: https://people.freebsd.org/~kevans/armstub8.bin (same location as the last one, different binary; sha512 follows)

SHA512 (armstub8.bin) = 643d66ad22d45cac568cf58861959c046a325c9081c41ea59915b7ce8deae6f0bc9234b5c9d9c81a5a2361a0f1c13cefc5c2de891a1c4f8e956759ea989eb7cf

The changes we made reflect upstream, but if this improves things then I'll shoot them a pull request.
Comment 3 Kyle Evans freebsd_committer 2020-01-05 18:52:31 UTC
Take, since I almost certainly caused it.
Comment 4 Tom Yan 2020-01-05 19:45:58 UTC
(In reply to Kyle Evans from comment #2)
I already did. This one is byte-to-byte identical to the one in 12.1-RELEASE. I don't know why I had problems with it when I place it into one of the snapshots (without also replacing u-boot.bin). I specifically re-tested this one (in addition to 2b8890a I built) with both STABLE and CURRENT 20191226 snapshots and couldn't reproduce it anymore. Maybe it was just an incidental pi haywire.

In case you missed it, I've already created an PR upstream. I reported here just to see if maybe you want to merge it downstream beforehand. (You know, things are pretty bad with the snapshots. I plan on bisecting the device tree starting from tomorrow btw.)

Or maybe we should wait until your 3A+ arrive and see, just to make sure it isn't merely my 3B+ happen to be broken, but indeed a BCM2837B0 thing (which kind of explains why the original author of the rpi4 updates didn't think the problematic changes need to be "isolated").
Comment 5 Kyle Evans freebsd_committer 2020-08-27 17:21:06 UTC
This was addressed in ports r528896 after Tom's excellent work was accepted; thanks!