Bug 252971 - RPI stable/13 after 20210107 crashes during boot
Summary: RPI stable/13 after 20210107 crashes during boot
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: arm (show other bugs)
Version: 13.0-STABLE
Hardware: arm64 Any
: --- Affects Only Me
Assignee: freebsd-arm (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-01-24 11:03 UTC by Helge Oldach
Modified: 2021-03-05 15:44 UTC (History)
5 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Helge Oldach 2021-01-24 11:03:30 UTC
Using snapshot FreeBSD-13.0-ALPHA2-arm64-aarch64-RPI-20210122-02611ef8ee9-256201.img.xz on a RPI4/8GB yields:

ð

U-Boot 2020.10 (Jan 22 2021 - 04:23:32 +0000)

DRAM:  7.9 GiB
RPI 4 Model B (0xd03114)
MMC:   mmc@7e300000: 1, emmc2@7e340000: 0
Loading Environment from FAT... In:    serial
Out:   serial
Err:   serial
Net:   eth0: ethernet@7d580000
PCIe BRCM: link up, 5.0 Gbps x1 (SSC)
starting USB...
Bus xhci_pci: probe failed, error -110
No working controllers found
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc0 is current device
** Bad device specification -bootable devplist **
** Bad device specification :1 bootfstype **
"Synchronous Abort" handler, esr 0x96000004
elr: 000000000009ba50 lr : 000000000009199c (reloc)
elr: 000000003b378a50 lr : 000000003b36e99c
x0 : d51e402010000060 x1 : 0000000000002383
x2 : 000000003b3d34b0 x3 : 000000003afe9d80
x4 : 000000003b3d30b0 x5 : 000000003b3d34b0
x6 : 000000003b3d30c0 x7 : 000000003afeaf70
x8 : 0000000000000000 x9 : 0000000000000008
x10: 000000003b3ce0b6 x11: 000000003af64e70
x12: 0000000000000000 x13: 0000000000000004
x14: 000000003af4cbb8 x15: 00000000ffffffff
x16: 0000000000004110 x17: 1a64509500040200
x18: 000000003af58d90 x19: 000000003afead70
x20: 0000000000000060 x21: 0000000000000060
x22: 0000000000000000 x23: 0000000000000000
x24: 0000000000000000 x25: 0000000000000000
x26: 0000000000000028 x27: 0000000000000003
x28: 000000003b3e4e94 x29: 000000003af4c120

Code: 17ffffdb f9400641 364809a1 b4000980 (f85f8006)
Resetting CPU ...

resetting ...


However, FreeBSD-13.0-CURRENT-arm64-aarch64-RPI-20210107-f2b794e1e90-255641.img.xz boots fine. The main difference (apart from commits) is that the 20210122 snapshot is based on rpi-firmware-1.20210111.g20210111_1 while the older is based on rpi-firmware-1.20201201.g20201201_1. It seems ports r561841 ("Fixes booting on RPI4-8GB") broke it.

13.0-ALPHA2 with manually crafted rpi-firmware-1.20201201.g20201201_1 boots fine as well.
Comment 1 Klaus Küchemann 2021-01-24 15:58:07 UTC
(In reply to Helge Oldach from comment #0)

It's not a BUG , manually crafting rpi-firmware is a FEATURE,
after very long discussions about these issues I changed to positive-thinking :-) lol
if 
you want to get rid of :
<<...starting USB...
Bus xhci_pci: probe failed, error -110
No working controllers found...>>
then 
read:
https://sourceforge.net/projects/d26853-bcm2711-rpi-4-b-dtb/
I have "heard" that some related bugs will be fixed by rpi-firmware in their next release(no warranty of course).
I propose to stop treating bugs in rpi-firmware as freebsd-bugs and instead fix it ourselves 
as long as rpi-org-releases are buggy .
Best Regards
K.
Comment 2 Emmanuel Vadot freebsd_committer 2021-01-24 16:14:42 UTC
(In reply to Helge Oldach from comment #0)

I think that the ALPHA2 build is now using the quarterly ports branch and it didn't had the update to the latest rpi-firmware.
I've now merge it to the 2021Q1 branch so next week snapshot should be ok.
Thanks for reporting.
Comment 3 Helge Oldach 2021-01-24 16:35:47 UTC
(In reply to Emmanuel Vadot from comment #2)
Perhaps my description was misleading...

Snapshot FreeBSD-13.0-ALPHA2-arm64-aarch64-RPI-20210122-02611ef8ee9-256201.img.xz contains rpi-firmware files (fixup.dat & a lot of friends) dating 9th January. That is actually what is installed by sysutils/rpi-firmware as of now (rpi-firmware-1.20210111.g20210111_1).

IOW, this image is broken.

The same applies to snapshot FreeBSD-13.0-ALPHA1-arm64-aarch64-RPI-20210114-7ae27c2d6c4-255938.img.xz

In line with the suggestion to "fix it ourselves" I suggest to remove both snapshots from the ftp mirrors.

Also we might consider reverting ports r561841 until it's fixed upstream.

The snapshot FreeBSD-13.0-CURRENT-arm64-aarch64-RPI-20210107-f2b794e1e90-255641.img.xz has the older rpi-firmware dating 1st December 2020. This one works.
Comment 4 Emmanuel Vadot freebsd_committer 2021-01-24 16:39:38 UTC
Well ok but I needed this update myself to be able to boot on my 8GB model so I don't know what to think then.
Comment 5 Helge Oldach 2021-01-24 16:50:41 UTC
(In reply to Emmanuel Vadot from comment #4)
Not sure why you need that update... This is 13.0-ALPHA2 built yesterday:

FreeBSD p48.local 13.0-ALPHA2 FreeBSD 13.0-ALPHA2 #1 stable/13-c256208-gf76393a6305-dirty: Sat Jan 23 14:59:01 CET 2021     root@p48.local:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC  arm64

Boots just fine with the *old* (1st December) rpi-firmware.

hmo@p48 /boot/msdos $ l bcm2711-rpi-4-b.dtb fixup4.dat start4.elf u-boot.bin
-rwxr-xr-x  1 root  wheel    47484 Dec  1 11:43 bcm2711-rpi-4-b.dtb*
-rwxr-xr-x  1 root  wheel     5428 Dec  1 11:43 fixup4.dat*
-rwxr-xr-x  1 root  wheel  2213312 Dec  1 11:43 start4.elf*
-rwxr-xr-x  1 root  wheel   555264 Jan  7 05:32 u-boot.bin*
hmo@p48 /boot/msdos $

(FTR, u-boot.bin is identical to freshly built u-boot-rpi-arm64, except where compile/link dates are in the file.)
Comment 6 Klaus Küchemann 2021-01-24 16:56:13 UTC
(In reply to Helge Oldach from comment #3)
you are absolutely right. but rpi-firmware`s latest still contains the next trap , there is currently no clean release which could be merged into the port without patching. so keeping an eye on their next release is the way. until then cherry-picking files (depending on what version is on your msdos-partiton) is the only way to get a fully working firmware(including xhci-boot).
Comment 7 Klaus Küchemann 2021-01-24 17:16:39 UTC
(In reply to Klaus Küchemann from comment #6)
in detail: 
"older" fbsd-releases(e.g. end of last year) can be patched by bcm2711-rpi-4-b.dtb, fixup4.dat & start4.elf are "clean".
while 
"newer" release has patched bcm2711-rpi-4-b.dtb to xhci-compatible but fixup4.dat start4.elf are broken for xhci-boot.

related files are available on the above sourceforge-link if someone doesn`t know what version sits on the msdos-partition. of course manu@ can feel free to patch the port to a consistent state NOW. but if he wants to stay pristine with the upstream , we have to patch(overwrite) manually.
K.
Comment 8 Helge Oldach 2021-01-24 17:21:10 UTC
(In reply to Klaus Küchemann from comment #7)
Agreed, my only concern is that manual cherrypicking should reflect into the snapshot build process so that the images distributed on ftp.freebsd.org are working.

For me (I'm on mmc boot, not xhci) the two latest images are broken but the preceding image works. I suppose it's different with xhci boot.
Comment 9 Klaus Küchemann 2021-01-24 17:39:01 UTC
(In reply to Helge Oldach from comment #8)

you are completely right,
I also think that we should not stay with an unpredictable upstream.
e.g. patching and hard linking bcm2711-rpi-4-b.dtb in one of the endOfLastYear-releases would have stopped this issue months ago.
we should ONLY merge an upstream which is clean and not broken.
details to the xhci-issue: 
https://github.com/raspberrypi/firmware/issues/1518
(while edkII-related also affects u-boot) , hopefully fixed in their next release .
Regards
K.
Comment 10 Emmanuel Vadot freebsd_committer 2021-01-27 10:27:52 UTC
So now that I'm home I could test.
FreeBSD-13.0-ALPHA2-arm64-aarch64-RPI-20210122-02611ef8ee9-256201.img does boot on my RPI4 8GB.
I only have one board to test so if there is differences between some revision I'm sorry but can't help.
Comment 11 Helge Oldach 2021-01-27 10:39:57 UTC
(In reply to Emmanuel Vadot from comment #10)
USB boot or MMC boot?
Comment 12 MaShaun Jones 2021-01-31 18:26:43 UTC
I have also been able to successfully boot an ALPHA2 image to a microsd on a RPI4 8gb unit here with the Sat 16 Jan 2021 02:10:13 PM UTC (1610806213) eeprom. It successfully booted, increased the root filesystem and presented the login prompt.
Comment 13 Helge Oldach 2021-02-01 08:37:12 UTC
(In reply to MaShaun Jones from comment #12)
Ok, I did some more testing. With HDMI connected (EFI console) I can confirm that ALPHA2 boots. However it still fails uttering the messages above if HDMI is not connected (serial console). Can you re-do your exercise with HDMI disconnected?

It seems we are seeing different behaviour with HDMI vs. serial boot and with MMC vs. XHCI boot.
Comment 14 Helge Oldach 2021-02-01 08:42:57 UTC
(In reply to Helge Oldach from comment #13)
One more observation: With the older rpi-firmware (1st December) behaviour is exactly opposite: With HDMI connected I getting above message cycle, with serial boot (HDMI disconnected) it boots perfectly.
Comment 15 MaShaun Jones 2021-02-01 12:39:53 UTC
I can also confirm the error:
U-Boot 2020.10 (Jan 22 2021 - 04:23:32 +0000)

DRAM:  7.9 GiB
RPI 4 Model B (0xd03114)
MMC:   mmc@7e300000: 1, emmc2@7e340000: 0
Loading Environment from FAT... In:    serial
Out:   serial
Err:   serial
Net:   eth0: ethernet@7d580000
PCIe BRCM: link up, 5.0 Gbps x1 (SSC)
starting USB...
Bus xhci_pci: probe failed, error -110
No working controllers found
Hit any key to stop autoboot:  0 
switch to partitions #0, OK
mmc0 is current device
** Bad device specification -bootable devplist **
** Bad device specification :1 bootfstype **
"Synchronous Abort" handler, esr 0x96000004
elr: 000000000009ba50 lr : 000000000009199c (reloc)
elr: 000000003b378a50 lr : 000000003b36e99c
x0 : d51e402010000060 x1 : 0000000000002383
x2 : 000000003b3d34b0 x3 : 000000003afe9d80
x4 : 000000003b3d30b0 x5 : 000000003b3d34b0
x6 : 000000003b3d30c0 x7 : 000000003afeaf70
x8 : 0000000000000000 x9 : 0000000000000008
x10: 000000003b3ce0b6 x11: 000000003af64e70
x12: 0000000000000000 x13: 0000000000000004
x14: 000000003af4cbb8 x15: 00000000ffffffff
x16: 0000000000004110 x17: 696504c0eda0a0c1
x18: 000000003af58d90 x19: 000000003afead70
x20: 0000000000000060 x21: 0000000000000060
x22: 0000000000000000 x23: 0000000000000000
x24: 0000000000000000 x25: 0000000000000000
x26: 0000000000000028 x27: 0000000000000003
x28: 000000003b3e4e94 x29: 000000003af4c120

Code: 17ffffdb f9400641 364809a1 b4000980 (f85f8006) 
Resetting CPU ...

resetting ...
Comment 16 commit-hook freebsd_committer 2021-02-08 16:34:29 UTC
A commit references this bug:

Author: manu
Date: Mon Feb  8 16:34:13 UTC 2021
New revision: 564719
URL: https://svnweb.freebsd.org/changeset/ports/564719

Log:
  sysutils/rpi-firmware: Update to 1.20210111

  While here also:
  Replace the deprecated arm_control by arm_64bit for 64bits configuration
  Add hdmi_safe for rpi4, recent updates of rpi-firmware break something related
  to hdmi. Version 1.20201201 will reset if hdmi monitor is connected while later
  version will only work if an hdmi monitor is connected. [1]

  PR:		252971 [1]
  MFH:		2021Q1

Changes:
  head/sysutils/rpi-firmware/Makefile
  head/sysutils/rpi-firmware/distinfo
  head/sysutils/rpi-firmware/files/config_arm64.txt
  head/sysutils/rpi-firmware/files/config_rpi3.txt
  head/sysutils/rpi-firmware/files/config_rpi4.txt
  head/sysutils/rpi-firmware/pkg-plist
Comment 17 commit-hook freebsd_committer 2021-02-08 16:36:31 UTC
A commit references this bug:

Author: manu
Date: Mon Feb  8 16:35:36 UTC 2021
New revision: 564722
URL: https://svnweb.freebsd.org/changeset/ports/564722

Log:
  MFH: r564719

  sysutils/rpi-firmware: Update to 1.20210111

  While here also:
  Replace the deprecated arm_control by arm_64bit for 64bits configuration
  Add hdmi_safe for rpi4, recent updates of rpi-firmware break something related
  to hdmi. Version 1.20201201 will reset if hdmi monitor is connected while later
  version will only work if an hdmi monitor is connected. [1]

  PR:		252971 [1]

Changes:
_U  branches/2021Q1/
  branches/2021Q1/sysutils/rpi-firmware/Makefile
  branches/2021Q1/sysutils/rpi-firmware/distinfo
  branches/2021Q1/sysutils/rpi-firmware/files/config_arm64.txt
  branches/2021Q1/sysutils/rpi-firmware/files/config_rpi3.txt
  branches/2021Q1/sysutils/rpi-firmware/files/config_rpi4.txt
  branches/2021Q1/sysutils/rpi-firmware/pkg-plist
Comment 18 Emmanuel Vadot freebsd_committer 2021-02-08 16:39:57 UTC
The last commit should fix the next BETA image.
Tested on my RPI4-8GB
Be sure to update config.txt on the msdos partition.
Comment 19 Helge Oldach 2021-02-08 16:56:42 UTC
(In reply to Emmanuel Vadot from comment #18)
I assume the log should read
  sysutils/rpi-firmware: Update to 1.20210201
Comment 20 Emmanuel Vadot freebsd_committer 2021-02-08 17:06:42 UTC
(In reply to Helge Oldach from comment #19)
Yes ... sorry.
Comment 21 Helge Oldach 2021-02-10 16:24:21 UTC
OK tested with 1.20210201 firmware and it's indeed working, with MMC boot and serial/HDMI dual console (thanks to hdmi_safe). I did not try XHCI nor network booting.

Regarding XHCI this is new upon device attach:

bcm_xhci0: <VL805 USB 3.0 controller (on the Raspberry Pi 4b)> irq 81 at device 0.0 on pci2
bcm_xhci0: warning: xhci firmware not found.
bcm_xhci0: 32 bytes context size, 64-bit DMA
bcm_xhci0: Controller reset timeout.
bcm_xhci0: XHCI halt/start/probe failed err=18
bcm_xhci0: Controller reset timeout.

Also the RPi is shouting loudly a couple of times during device attach:

gpioregulator0: <GPIO controlled regulator> on ofwbus0
gpioregulator0: cannot get pin 0
gpioregulator0: cannot parse parameters
device_attach: gpioregulator0 attach returned 6

But well, no crash.
Comment 22 Klaus Küchemann 2021-02-10 19:18:29 UTC
(In reply to Helge Oldach from comment #21)
<<...Also the RPi is shouting loudly a couple of times during device attach..>>

lol... cool new audio feature ...
be glad you know how to get rid of it ...
Comment 23 Emmanuel Vadot freebsd_committer 2021-02-10 19:28:31 UTC
(In reply to Helge Oldach from comment #21)

Yeah I'm not really interested in warning or unsupported stuff on RPI4 :)
Closing this bug now.
Comment 24 Helge Oldach 2021-02-10 19:56:48 UTC
(In reply to Emmanuel Vadot from comment #23)
Well. The 2020-12-01 firmware die not exhibit this behaviour.

No issue with XHCI:

Feb  9 08:08:48 p48 kernel: bcm_xhci0: <VL805 USB 3.0 controller (on the Raspberry Pi 4b)> irq 82 at device 0.0 on pci1
Feb  9 08:08:48 p48 kernel: bcm_xhci0: 32 bytes context size, 64-bit DMA
Feb  9 08:08:48 p48 kernel: usbus0 on bcm_xhci0

No issue with gpio:

Feb  9 08:08:48 p48 kernel: gpio0: <Raspberry Pi Firmware GPIO controller> on bcm2835_firmware0
Feb  9 08:08:48 p48 kernel: gpiobus0: <GPIO bus> on gpio0
Feb  9 08:08:48 p48 kernel: gpioregulator0: <GPIO controlled regulator> on ofwbus0
Feb  9 08:08:48 p48 kernel: gpioc0: <GPIO controller> on gpio0
Comment 25 Emmanuel Vadot freebsd_committer 2021-02-17 11:42:00 UTC
Using files from https://github.com/raspberrypi/firmware/issues/1518#issuecomment-779355320 fixes the issue.
I've re-open the bug and will update the port as soon as they do a release.

I'm looking at the gpio problem.
Comment 26 commit-hook freebsd_committer 2021-02-17 12:19:11 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=1cf282363101f5d99b1dadfb0d3250bbe6f482a5

commit 1cf282363101f5d99b1dadfb0d3250bbe6f482a5
Author:     Emmanuel Vadot <manu@FreeBSD.org>
AuthorDate: 2021-02-17 12:11:36 +0000
Commit:     Emmanuel Vadot <manu@FreeBSD.org>
CommitDate: 2021-02-17 12:18:21 +0000

    arm64: rpi4: firmware: Attach at BUS_PASS_BUS + BUS_PASS_ORDER_LATE

    The node have now a compatible with simple-mfd so we need to attach
    at the same pass so the specific driver will be used.

    MFC after:      3 days
    PR:             252971

 sys/arm/broadcom/bcm2835/bcm2835_firmware.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 27 MaShaun Jones 2021-02-17 14:24:16 UTC
I can confirm after patching my 13.0-CURRENT source from last night with this fix and recompiling the kernel now the similar problems I was seeing with the gpio, cpufreq, and temperature sensor messages have been corrected.

[na@bsdbox ~]$ sudo dmesg | grep -i gpio | grep parse
[na@bsdbox ~]$ 

[na@bsdbox ~]$ sudo sysctl -a | grep temperatu
hw.cpufreq.temperature: 57416
dev.cpu.0.temperature: 57.4C

[na@bsdbox ~]$ sudo sysctl dev.cpu.0.freq_levels 
dev.cpu.0.freq_levels: 1500/-1 600/-1
Comment 28 MaShaun Jones 2021-02-17 14:36:51 UTC
Also this was USB booted.

bcm_xhci0: <VL805 USB 3.0 controller (on the Raspberry Pi 4b)> irq 81 at device 0.0 on pci2
bcm_xhci0: 32 bytes context size, 64-bit DMA
usbus0 on bcm_xhci0
ugen0.1: <0x1106 XHCI root HUB> at usbus0
uhub0: <0x1106 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Comment 29 commit-hook freebsd_committer 2021-02-20 19:19:35 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=944f4316558055a2bb6481048386e94b523ab48c

commit 944f4316558055a2bb6481048386e94b523ab48c
Author:     Emmanuel Vadot <manu@FreeBSD.org>
AuthorDate: 2021-02-17 12:11:36 +0000
Commit:     Emmanuel Vadot <manu@FreeBSD.org>
CommitDate: 2021-02-20 19:17:40 +0000

    arm64: rpi4: firmware: Attach at BUS_PASS_BUS + BUS_PASS_ORDER_LATE

    The node have now a compatible with simple-mfd so we need to attach
    at the same pass so the specific driver will be used.

    MFC after:      3 days
    PR:             252971

    (cherry picked from commit 1cf282363101f5d99b1dadfb0d3250bbe6f482a5)

 sys/arm/broadcom/bcm2835/bcm2835_firmware.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 30 commit-hook freebsd_committer 2021-02-21 21:04:42 UTC
A commit in branch releng/13.0 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=07efb4cef7be8336b272869cf9a68c817445983a

commit 07efb4cef7be8336b272869cf9a68c817445983a
Author:     Emmanuel Vadot <manu@FreeBSD.org>
AuthorDate: 2021-02-17 12:11:36 +0000
Commit:     Emmanuel Vadot <manu@FreeBSD.org>
CommitDate: 2021-02-21 21:02:47 +0000

    arm64: rpi4: firmware: Attach at BUS_PASS_BUS + BUS_PASS_ORDER_LATE

    The node have now a compatible with simple-mfd so we need to attach
    at the same pass so the specific driver will be used.

    Approved by:    re (gjb)
    MFC after:      3 days
    PR:             252971

    (cherry picked from commit 1cf282363101f5d99b1dadfb0d3250bbe6f482a5)
    (cherry picked from commit 944f4316558055a2bb6481048386e94b523ab48c)

 sys/arm/broadcom/bcm2835/bcm2835_firmware.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 31 commit-hook freebsd_committer 2021-03-05 10:33:08 UTC
A commit references this bug:

Author: manu
Date: Fri Mar  5 10:32:15 UTC 2021
New revision: 567376
URL: https://svnweb.freebsd.org/changeset/ports/567376

Log:
  sysutils/rpi-firmware: Update to 20210303

  Fix USB on RPI4

  PR:	252971, 253981, 253983
  MFH:		2021Q1

Changes:
  head/sysutils/rpi-firmware/Makefile
  head/sysutils/rpi-firmware/distinfo
  head/sysutils/rpi-firmware/pkg-plist
Comment 32 commit-hook freebsd_committer 2021-03-05 10:35:12 UTC
A commit references this bug:

Author: manu
Date: Fri Mar  5 10:33:54 UTC 2021
New revision: 567377
URL: https://svnweb.freebsd.org/changeset/ports/567377

Log:
  MFH: r567376

  sysutils/rpi-firmware: Update to 20210303

  Fix USB on RPI4

  PR:	252971, 253981, 253983

Changes:
_U  branches/2021Q1/
  branches/2021Q1/sysutils/rpi-firmware/Makefile
  branches/2021Q1/sysutils/rpi-firmware/distinfo
  branches/2021Q1/sysutils/rpi-firmware/pkg-plist
Comment 33 Helge Oldach 2021-03-05 15:44:30 UTC
Yep, the firmware update sorted it out (MMC boot):

bcm_xhci0: <VL805 USB 3.0 controller (on the Raspberry Pi 4b)> irq 92 at device 0.0 on pci2
usbus0 on bcm_xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
ugen0.1: <0x1106 XHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <0x1106 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
uhub0: 5 ports with 4 removable, self powered
ugen0.2: <vendor 0x2109 USB2.0 Hub> at usbus0
uhub1 on uhub0
uhub1: <vendor 0x2109 USB2.0 Hub, class 9/0, rev 2.10/4.21, addr 1> on usbus0