Bug 253983 - RPI4 boot fails when geekworm x829 is attached over USB
Summary: RPI4 boot fails when geekworm x829 is attached over USB
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: arm (show other bugs)
Version: 13.0-STABLE
Hardware: arm64 Any
: --- Affects Only Me
Assignee: freebsd-arm (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-03-03 16:10 UTC by eimar.koort
Modified: 2021-06-07 04:55 UTC (History)
3 users (show)

See Also:


Attachments
RPI4 boot failure with attached Geekworm X829 module (846.16 KB, image/png)
2021-03-03 16:10 UTC, eimar.koort
no flags Details
Freebsd 14-current 20210225-bf667f282a7-256946.img dmesg (16.52 KB, text/plain)
2021-03-04 12:58 UTC, eimar.koort
no flags Details
gpart show (380 bytes, text/plain)
2021-03-04 12:59 UTC, eimar.koort
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description eimar.koort 2021-03-03 16:10:08 UTC
Created attachment 222940 [details]
RPI4 boot failure with attached Geekworm X829 module

Hi,
I'm having a problem with RPI4 booting from SD-card while 2 HDD-s are attached over USB bridge from Geekworkm X829 (1) module. When USB bridge is present during boot, then it skips SD card for unknown reason and fails (see attached image). When i remove the USB bridge (2), then RPI boots fine. SD card is working properly. 
All worked with snapshot installed around october 2020. The problem appeared, when installed fresh 13-BETA1. 

"Screenshot" is taken from 13-BETA (sorry about screenshot quality).
Is there some way to manually force to boot from sd card?

Links:
1) https://raspberrypiwiki.com/index.php/X829
2) https://raspberrypiwiki.com/images/thumb/e/eb/X829-IMG-8042.jpg/300px-X829-IMG-8042.jpg
Comment 1 Klaus Küchemann 2021-03-03 16:30:49 UTC
(In reply to eimar.koort from comment #0)
I see "cannot initialize UEFI-subsystem"....
do you boot from rpi4UEFIdev(not directly supported by FreeBSD)?
if you're booting from u-boot(what I assume), please take a look :
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253981
, you can upgrade the msdos-partition-files(starup4.dat etc.) and directly boot the root-filesystem from USB, so you won't need an uSD-card.
"unrecognized filesystem type" in your screenshot normally happens with incompatible u-boot-versions . there is also a u-boot-file in the source forge-link in show_bug.cgi?id=253981 , `hope that helps for you.

by the way: instead of screenshots you can post dmesg(ideally from the UART-console) 
to: https://dmesgd.nycbug.org
Comment 2 eimar.koort 2021-03-03 19:43:57 UTC
Hi,

Dumb as i am, i don't know how to save console output from u-boot prompt. I'm not able to boot up os to provide failed boot dmesg. 
Sorry about that.

And i almost don't undestand what do You mean by "do you boot from rpi4UEFIdev".
What i did is i took sd card, did a 'dd' RPI Beta4 image to it, inserted to RPI and booted up. So yes, i'm using u-boot. That is coming with image. 

Now i removed USB bridge, booted FBSD, replaced files (start4.elf, fixup4.dat and bcm2711-rpi-4-b.dtb) mentioned in "show_bug.cgi?id=253981". I'm unable to find u-boot.bin file in sourceforge page or link to it. And the problem still persists - when USB bridge is connected, then it skips sd-card and eventually tries to boot from network until it ends on u-boot prompt.

Also i'd like to boot from sd-card instead of usb because it makes me easier to handle this machine in case of failure. I'm not sure rpi is able to boot from mirrored usb disk(s).
Comment 3 Mark Millard 2021-03-03 19:50:29 UTC
"When USB bridge is present during boot, then it skips SD card for unknown reason":

See the following for controlling which order boot devices
are tried in for RPi4B's and such. This is not under FreeBSD's
control so the references are to www.raspberrypi.org pages and
that is where questions about the specific type of issue should
go.

https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/bootflow_2711.md

and:

https://www.raspberrypi.org/documentation/hardware/raspberrypi/bcm2711_bootloader_config.md

The status of this bug report should be changed to indicate
that this is not a FreeBSD bug. But I can not do that for
something that you submit.
Comment 4 eimar.koort 2021-03-03 20:10:00 UTC
Hi, 

I did that couple a days ago with linux and set  eeprom values especially to boot from SD card. Here they are:

root@rpi4:~# rpi-eeprom-config
BOOT_UART=0
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
FREEZE_VERSION=0                                                       
[all]
BOOT_ORDER=0xf1
SD_BOOT_MAX_RETRIES=-1

Also it is worth to mention that linux (rasberry pi os) is able to boot up from SD card when USB bridge is connected. If eeprom values are somehow messed up then shouldn't linux boot fail too?

And why did 13-CURRENT snapshot version from october 2020 boot fine?
Comment 5 Mark Millard 2021-03-03 20:33:23 UTC
(In reply to eimar.koort from comment #4)

Intersting . . .

Your pictures do not cover enough material to see
the full sequence. But it sounds like you can:

A) Boot without the USB drives
B) connect the USB drives
C) use gpart show and report what is listed
D) mount file systems on the various drives
   and see what is listed in ones possibly
   involved in early booting
E) report on what the various file systems
   contain

As stands, to avoid your error messages about UEFI not
being able to initialize and the like, I'd be looking
for whatever might try to set up and start an UEFI
style boot. Unless you have deliberately tried to set
up such a boot, no such attempt should be made by the
FreeBSD boot sequecne on a RPi4B. Something extra
or non-standard has to be present to cause the activity.

Note: efi/boot/bootaa64.efi does not start a UEFI boot,
it is something an already-started uefi style boot tries
to find and then use.
Comment 6 Mark Millard 2021-03-03 20:34:09 UTC
Some background notes . . .

The evidence in your screen image is that the USB drive(s)
are set up to boot in a UEFI style where something found a:

efi/boot/bootaa64.efi

(It might have found a EFI/BOOT/BOOTaa64.efi or other
capitalization variation.)

Currently a modern RPI image such as:

FreeBSD-13.0-BETA4-arm64-aarch64-RPI.img
(listed only because I've previously checked the
contents of this particular example image)

does contain such a file. I can even use the microsd
card produced from the image to boot systems that are
designed/configured for UEFI booting, such as an
OverDrive 1000, via it using that file. The files in
the image that are RPi* specific are simply ignored
when booting a OverDrive 1000 or other such.

But, despite containing the file, FreeBSD is not acively
supporting booting RPi4B's via a method that uses that
file (i.e., via UEFI style booting). It is possible to
set up UEFI booting from software from outside the FreeBSD
project. I'm only familiar with one way but there might
be others. I have no direct clue how your context got a
configuration that would attempt a UEFI boot.
Comment 7 Klaus Küchemann 2021-03-03 20:42:14 UTC
(In reply to eimar.koort from comment #4)
(In reply to eimar.koort from comment #4)
<<And why did 13-CURRENT snapshot version from october 2020 boot fine?>>

because of the firmware. RPI4 4GB-model, right?

so now you have the easiest bugfix ever seen :-) :
simply overwrite the whole October 2020-msdos-partition to your current  disk 
if you felt comfortable with it.
Comment 8 Mark Millard 2021-03-03 21:12:27 UTC
(In reply to Klaus Küchemann from comment #7)

Not likely to work without also deleting/disabling
whatever is providing the UEFI-style-boot
infrastructure that is looking for efi/boot/bootaa64.efi .

Given the BOOT_ORDER and such, I expect the UEFI
infrastructure is on the microsd card in a way
that the RPi4B firmware finds, loads, and executes
it. This means that looking at config.txt content
is part of what should be examined. It likely has
to be involved in making teh firmware load and start
the UEFI infrastrcture.

It appears that the UEFI infrastructure is looking
at the USB drives after the infrastructure is started.
But the infrastructure should not be there in the first
place based on what the desired behavior is reported to
be.

FreeBSD does not implement or provide such infrastrcture,
just the efi/boot/bootaa64.efi that the infrastrcture
looks for. Similarly The RPi4B firmware does not implement
a UEFI infrastructure. The UEFI infrastructure is from
outside both of those places ( but might involve material
from the RPi4B firmware, such as config.txt ).
Comment 9 Klaus Küchemann 2021-03-03 21:22:12 UTC
(In reply to Mark Millard from comment #8)
<<Not likely to work without also deleting/disabling
whatever is providing the UEFI-style-boot...>

eimar.koort@tutamail.com  swore to us :-) that he doesn't even know uefiDev.
So we can assume that the crap is in the new firmware files.
By overwriting the whole msdos partition with October 2020 release,
the issue should be fixed for him.
Comment 10 Mark Millard 2021-03-03 21:50:14 UTC
(In reply to Klaus Küchemann from comment #9)

I am using the newest published firmware files and
do not end up with an UEFI-based boot. The RPi*
folks have not published a UEFI implementation
in their firmware.

I do not claim that there is only one UEFI
implementation around for the RPi4B. (Although,
that is a possibility.)

The context needs to be examined independent of
any claims by any of us. Things are too odd to
be running on unchecked assumptions, at least
if understanding is intended.
Comment 11 Klaus Küchemann 2021-03-03 22:00:47 UTC
(In reply to Mark Millard from comment #10)
<< Things are too odd to
be running on unchecked assumptions, at least
if understanding is intended......>>

 you're right, really odd.... do you remember : "GUESSING" ,:-) ,
my GUESS No.1 is that October 2020-release will fix it for him, if not,
something odd happens in eeprom... or in the src-tree??
let's wait if 2020-release will fix it or not  for eimar.koort@tutamail.com
Comment 12 Klaus Küchemann 2021-03-03 22:06:59 UTC
(In reply to eimar.koort from comment #0)
ah, looking closer here I see two mentioned versions from you:
BETA1 / BETA4 , perhaps you could try 14-current, there were issues mentioned in the ML with some 13-beta-versions....
Comment 13 Mark Millard 2021-03-03 22:17:37 UTC
(In reply to Klaus Küchemann from comment #11)

If the context is cleanedout without looking for the UEFI
implementation and finding it first, we will never know
what was actually going on.

Clearly it is not for me to specify what is done but I
hope for having the implementation found and reported
so we know what to look for in the future.

But this is likely not the fastest route to eimar having
things operational. Isolating what is happening when odd
things are going on tends to be time consuming. So eimar
could naturally decide to avoid the activity, such as by
a clean-out and start over sequence.
Comment 14 Klaus Küchemann 2021-03-03 22:32:24 UTC
(In reply to Mark Millard from comment #13)
yes, I'm also interested in finding the cause of course...
for that we have to know the exact version ( Beta1, Beta4 or whatever)...
in other context there were bugs reported in the ML which appeared in Beta1 and were fixed in Beta4.
to be honest: I`m always on current(14 here) and didn't track all beta-versions.
if 14-current works for Eimar we could ignore the cause(because it's fixed then)
Comment 15 eimar.koort 2021-03-04 10:28:24 UTC
Thank You for responces!

FreeBSD-14.0-CURRENT-arm64-aarch64-RPI-20210225-bf667f282a7-256946.img has same behavior as 13-Betas (boots up w/o usb bridge and is not able to boot with bridge).

I've been looking around trying to find 13-current snapshot image without any luck. Also copied fixup4.dat, start4.elf, bcm2711-rpi-4-b.dtb from linux msdos partition. Tried to different older available u-boot.bin versions. Even tried whole 12.2 release msdos content also without luck.
One thing is: if i rename EFI directory to EFI_orig or whatever, then boot also fails and RPI tries to boot from network (with or w/o usb bridge does not matter).
So i start to thinking, that the culprit is "bootaa64.efi". 

"config.txt" is unchanged from what comes from *.img.

[all]
arm_64bit=1
dtparam=audio=on,i2c_arm=on,spi=on
dtoverlay=mmc
dtoverlay=disable-bt
device_tree_address=0x4000
kernel=u-boot.bin

[pi4]
hdmi_safe=1
armstub=armstub8-gic.bin

Disk layout is also unchanged. 
USB disks are empty: dd if=/dev/zero of=/dev/da[0|1] ...etc.
One thing i can try out is to remove hard disks completely and see what happends.

Is there a way to get msdos content from now disappeared 13-current snapshot releases?
Comment 16 eimar.koort 2021-03-04 11:17:05 UTC
Hi,
Removed both hard disks, connected bridge and it boots up just fine.
Inserted one disk x829 module (does not matter is hdd1 slot or hdd2 slot) - it boots up from sd card.
Both disks added and it's not able to boot up.
Comment 17 Klaus Küchemann 2021-03-04 12:29:31 UTC
(In reply to eimar.koort from comment #16)
ah, now we have an idea where the UEFI-message comes from:
there seems to be a chip on the X829 which is activated if 2 disks are connected.
that chip is probably detected by ACPI/UEFI from the OS...
is the UEFI-message gone when only 1 disk is connected?
I would suggest to boot from USB-disk(or are that SATA-disks on the X829)?
since we don't have the X829 available here `guessing`is the only thing we can do(although don't like guessing:-)..
you can send screenshots of course if you don't have a UART-console.
Comment 18 Klaus Küchemann 2021-03-04 12:48:16 UTC
ASMT USB bridge chipset, ASMedia ASM235CM-Chipset....
should be supported(since it is detected on boot)... hmmm..guessing ;-).. is the disk in one-disk-connection detected by FreeBSD? if you can boot up and login: you can type: 'dmesg' and attach dmesg here
Comment 19 Klaus Küchemann 2021-03-04 12:52:00 UTC
...also 'gpart show' if you're logged in...
Comment 20 eimar.koort 2021-03-04 12:58:38 UTC
Created attachment 222971 [details]
Freebsd 14-current 20210225-bf667f282a7-256946.img dmesg
Comment 21 eimar.koort 2021-03-04 12:59:26 UTC
Created attachment 222972 [details]
gpart show
Comment 22 Klaus Küchemann 2021-03-04 13:11:29 UTC
umass1: <SupTronics X829, class 0/0, rev 3.00/1.00, addr 6> on usbus0
umass1:  SCSI over Bulk-Only; quirks = 0x0100
umass1:1:1: Attached to scbus1
da1 at umass-sim1 bus 1 scbus1 target 0 lun 0
da1: <ASMT 2115 0> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number 20201000000A
da1: 400.000MB/s transfers
da1: 953869MB (1953525168 512 byte sectors)
da1: quirks=0x2<NO_6_BYTE>
----
detected but not available in gpart, normally I would say :
'gpart create -s GPT da1`. .... ' gpart add -t freebsd-ufs da1 ...'newfs -U /dev/da1p1`(or da1s1 or whatever shows up in `ls /dev`)...

but that doesn't answer the 2-disk-setup-issue for now...
perhaps 1st format 1 disk, then connect&format the 2nd? .. 
hard to remote-debug by guessing...
but all in all good news that the phy is detected...
Comment 23 Klaus Küchemann 2021-03-04 13:24:13 UTC
well... in Germany we say:
"I eat a broom" if the UEFI-message reappears when you connect the 2nd data-disk now..
this could perhaps be an fbsd-bug since you boot from fdt...
there is a specialist for that , called HPS@, I'll add him to this bug-report.
Comment 24 eimar.koort 2021-03-04 13:41:00 UTC
I removed disk from second bay, booted up, gpart create -s GPT da1, gpart add -t freebsd-ufs da1; newfs -U /dev/da1p1. 
# camcontrol devlist
<ASMT 2115 0>                      at scbus0 target 0 lun 0 (da0,pass0)
<ASMT 2115 0>                      at scbus1 target 0 lun 0 (da1,pass1)

# ls -la /dev/da*
crw-r-----  1 root  operator  0x73 Feb 25 09:13 /dev/da0
crw-r-----  1 root  operator  0x74 Feb 25 09:13 /dev/da0p1
crw-r-----  1 root  operator  0x81 Feb 25 09:13 /dev/da1

# gpart show
=>      63  62333889  mmcsd0  MBR  (30G)
        63      2016          - free -  (1.0M)
      2079    102312       1  fat32lba  [active]  (50M)
    104391  62228537       2  freebsd  (30G)
  62332928      1024          - free -  (512K)

=>       0  62228537  mmcsd0s2  BSD  (30G)
         0        57            - free -  (29K)
        57  62228480         1  freebsd-ufs  (30G)

=>        40  1953525088  da0  GPT  (932G)
          40  1953525088    1  freebsd-ufs  (932G)

=>        40  1953525088  diskid/DISK-202010000009  GPT  (932G)
          40  1953525088                         1  freebsd-ufs  (932G)


Power down. Added second disk to hdd2 bay. Poweron and unable to boot (as it can be seen from very first attacment/screenshot).
Comment 25 Klaus Küchemann 2021-03-04 14:21:14 UTC
does this mean I have to eat a broom? :-) Ha Ha ..
O.K., clearly a thing for HPS@, if he doesn't know, we all do not know what happens 
with your chipset on 2-disk-boot-setup..at least not by "remote-guessing"..
Comment 26 Klaus Küchemann 2021-03-04 14:32:41 UTC
although unprofessional one last guess:
do the same thing with the second disk(gpart) and try again... **unrecognized filesystem** on your screenshot made me think of this... 
and one very last guess: only 10 percent that this will fix it.. but who knows...
otherwise you can hope hope for an HPS-idea...
Comment 27 eimar.koort 2021-03-04 15:08:34 UTC
Hi,

Now both HDD-s are gparted/ufs'ed. It's able to boot up with only one hdd disk connected. 

Current layout is like that (Booted up without usb bridge. When up, added bridge and did "gpart show"):

=>      63  62333889  mmcsd0  MBR  (30G)
        63      2016          - free -  (1.0M)
      2079    102312       1  fat32lba  [active]  (50M)
    104391  62228537       2  freebsd  (30G)
  62332928      1024          - free -  (512K)

=>       0  62228537  mmcsd0s2  BSD  (30G)
         0        57            - free -  (29K)
        57  62228480         1  freebsd-ufs  (30G)

=>        40  1953525088  da0  GPT  (932G)
          40  1953525088    1  freebsd-ufs  (932G)

=>        40  1953525088  diskid/DISK-202010000009  GPT  (932G)
          40  1953525088                         1  freebsd-ufs  (932G)

=>        40  1953525088  da1  GPT  (932G)
          40  1953525088    1  freebsd-ufs  (932G)

=>        40  1953525088  diskid/DISK-20201000000A  GPT  (932G)
          40  1953525088                         1  freebsd-ufs  (932G)


And just for clarification: disks are regular 2.5" sata hdd-s, which are connected over usb bridge to RPI.

Regards,
Comment 28 Klaus Küchemann 2021-03-04 15:15:47 UTC
yeah, that was expected...
well, I think again of your screenshot....
to force the boot from uSD you can interrupt to the u-boot-prompt...

commands like :
setenv devnum 1 (Enter)
'run mmc_boot' ( or simply : 'boot') (Enter)
.. depending on the devnum of the uSD...

saveenv often fails to write uSD but if you're lucky,
'saveenv' would make that setting persistent...

hope that will help...
Comment 29 Klaus Küchemann 2021-03-04 15:22:41 UTC
and.... :
according to your screenshot the issue consists of the bootdevice (which is one the 
sata-disks)... I would consider to prepare one sata-disk as the boot disk...
which should fix the bootup if you can't 'setenv devnum xy' for the uSD...
Comment 30 eimar.koort 2021-03-04 15:37:53 UTC
Hi,

"Screenshot":

U-Boot> setenv devnum 0 
U-Boot> run mmc_boot
switch to partition #0, OK
mmc0 is current device
Scanning mmc 0:1 ...
Found EFI removable media binary efi/boot/bootaa64.efi
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Scanning disk mmc@7e300000.blk...
Disk disk mmc@7e300000.blk not ready.
Scanning emmc2@7e340000.blk...
** Unrecognized filesystem type **
Scanning disk usb_mass_storage.lun0...
** Unrecognized filesystem type **
Scanning disk usb_mass_storage.lun0...
ERROR: failure to add disk device usb_mass_storage.lun0, r = 20
Eeror: Cannot initialize UEFI sub-system, r = 20
1258028 bytes read in 72 ms (16.7 MiB/s)
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Error: Cannot initialize UEFI sub-system, r = 20
EFI LOAD FAILED: continuing...
U-Boot>

U-Boot> setenv devnum 1
ends with booting from network
Comment 31 Klaus Küchemann 2021-03-04 16:12:43 UTC
yeah, of course the controller hangs on FreeBSD-boot in bootaa64.efi,  not in u-boot...but was worth a try...
thanks for testing everything by "remote-guessing" ;-) ... out of ideas for now ...

HPS has developed a tool called hw.usb.template... where you can set e.g. the voltage( if the second e.g disk causes undervoltage)..
perhaps he'll find the time to take a look into this bug here...
Regards
K.
Comment 32 Mark Millard 2021-03-04 17:27:36 UTC
(In reply to eimar.koort from comment #30)

#30 answered the question of where UEFI came from:
It is an implementation in the u-boot that is in
use.

I've still not clue why u-boot engages its UEFI
implementation or how to disable it --other than
building and installing a u-boot that does not
include a UEFI implementation.

But the direction forward seems to be finding a
way to not have a u-boot based UEFI-style-boot
involved since FreeBSD does not actively support
such on the RPi4B.

Note that UEFI use of itself does not indicate
if ACPI is in use vs. a Device Tree. The way
to boot with UEFI that I know of is just ACPI
based in the only mode that I've tested. (It
is not a u-boot based UEFI implementation.)

One possibility is seeing if UEFI with ACPI can
be enabled in u-boot (and the Device Tree
disabled?).
Comment 33 Mark Millard 2021-03-04 18:54:03 UTC
(In reply to Mark Millard from comment #32)

I did some reading and I got this wrong: apparently,
for armv8, u-boot is always set up and used as a UEFI
implementation (generally not ACPI style, so EBBR
UEFI or a useful subset).

That appears to mean that what we have run into is
a UEFI implementation limitation of some kind in the
u-boot, one tied to handling 2 drives instead of
one in the context, resulting in the explicit
ERROR/Error/FAILED messages listed in the below
sequence, for example:

Scanning disk usb_mass_storage.lun0...
ERROR: failure to add disk device usb_mass_storage.lun0, r = 20
Error: Cannot initialize UEFI sub-system, r = 20
1258028 bytes read in 72 ms (16.7 MiB/s)
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Error: Cannot initialize UEFI sub-system, r = 20
EFI LOAD FAILED: continuing...

So it looks like the issue would require a u-boot
change/fix, not a RPi* firwmare or FreeBSD fix.
In other words, u-boot needs to be changed to manage
to initialize its "UEFI sub-system" successfully
for the type of context.

This would mean that the proper place for a bug
report is upstream for u-boot and possibly one
in the FreeBSD port indicating to pick up the fix
if one is made. Relative to the FreeBSD system
it is "not a bug" because FreeBSD is not the
upstream for the u-boot FreeBSD is trying to use.
Comment 34 Klaus Küchemann 2021-03-04 23:41:03 UTC
(In reply to Mark Millard from comment #33)
if u-boot were the cause that would probably mean that every Linux using u-boot (except Raspberry Pi OS(comment #4)) would also fail to boot in 2-disk-setup.
I normally really don't like to refer to Linux in FreeBSD-topics, but if it were mine controller, I would test e.g. ubuntu with u-boot. Then we could also be sure it's not a fbsd-bug and the issue could be reported to e.g. the manufacturer of the controller.
Comment 35 Mark Millard 2021-03-04 23:57:12 UTC
(In reply to Klaus Küchemann from comment #34)

That presumes FreeBSD and Linux build the exact same
u-boot sources with the exact same options and the
like.

I do not expect that such is the case.

It might be how u-boot was built or some such,
but the UEFI implementation is contained in
u-boot, not elsewhere. u-boot does not make
use of FreeBSD when initializing its UEFI
implementation as far as I can tell. u-boot
produced the error messages, not FreeBSD.

But I'm not aware of a good substitution test
sequence (no alternatives to substitute). May
be something that would execute far enough to
tell if the UEFI initialization failed or not
but that might not be able to deal with later
activities?

Debug builds of u-boot might report something
about why its UEFI initialization fails.
Comment 36 Mark Millard 2021-03-05 00:15:16 UTC
(In reply to Mark Millard from comment #35)

FYI:

# pwd
/wrkdirs/usr/ports/sysutils/u-boot-rpi4/work/u-boot-2020.04

# grep -r 'UEFI sub-system' ./
./cmd/bootefi.c:                printf("Error: Cannot initialize UEFI sub-system, r = %lu\n",
./cmd/efidebug.c:               printf("Error: Cannot initialize UEFI sub-system, r = %lu\n",
./cmd/nvedit_efi.c:             printf("Error: Cannot initialize UEFI sub-system, r = %lu\n",
./cmd/nvedit_efi.c:             printf("Error: Cannot initialize UEFI sub-system, r = %lu\n",
./common/bootm_os.c:            printf("## Failed to initialize UEFI sub-system: r = %lu\n",
./doc/api/efi.rst:Initialization of the UEFI sub-system
./test/py/tests/test_efi_selftest.py:    """Test the device tree support in the UEFI sub-system

Each instance of the Error message code looks like:

        /* Initialize EFI drivers */
        ret = efi_init_obj_list();
        if (ret != EFI_SUCCESS) {
                printf("Error: Cannot initialize UEFI sub-system, r = %lu\n",
                       ret & ~EFI_ERROR_MASK);
                return CMD_RET_FAILURE;
        }
Comment 37 Mark Millard 2021-03-05 00:36:40 UTC
(In reply to Mark Millard from comment #36)

(I should note that I'm not looking at the exact
u-boot sources but the 2020.04 sources I had around
from a prior investigation of some behavior.)

The earlier ERROR message is from:

efi_status_t efi_disk_register(void)
{
. . .
        for (uclass_first_device_check(UCLASS_BLK, &dev); dev;
             uclass_next_device_check(&dev)) {
                struct blk_desc *desc = dev_get_uclass_platdata(dev);
                const char *if_typename = blk_get_if_type_name(desc->if_type);

                /* Add block device for the full device */
                printf("Scanning disk %s...\n", dev->name);
                ret = efi_disk_add_dev(NULL, NULL, if_typename,
                                        desc, desc->devnum, 0, 0, &disk);
                if (ret == EFI_NOT_READY) {
                        printf("Disk %s not ready\n", dev->name);
                        continue;
                }
                if (ret) {
                        printf("ERROR: failure to add disk device %s, r = %lu\n",
                               dev->name, ret & ~EFI_ERROR_MASK);
                        return ret;
                }
. . .
        }

(The "Scanning disk" message matches the above.)

There is seaprate code in the routine based on
CONFIG_BLK being defined (above case) vs. not
(below case) that looks like:

        for (if_type = 0; if_type < IF_TYPE_COUNT; if_type++) {
                const struct blk_driver *cur_drvr;
                const char *if_typename;

                cur_drvr = blk_driver_lookup_type(if_type);
                if (!cur_drvr)
                        continue;

                if_typename = cur_drvr->if_typename;
                printf("Scanning disks on %s...\n", if_typename);
                for (i = 0; i < 4; i++) {
                        struct blk_desc *desc;
                        char devname[32] = { 0 }; /* dp->str is u16[32] long */

                        desc = blk_get_devnum_by_type(if_type, i);
                        if (!desc)
                                continue;
                        if (desc->type == DEV_TYPE_UNKNOWN)
                                continue;

                        snprintf(devname, sizeof(devname), "%s%d",
                                 if_typename, i);

                        /* Add block device for the full device */
                        ret = efi_disk_add_dev(NULL, NULL, if_typename, desc,
                                               i, 0, 0, &disk);
                        if (ret == EFI_NOT_READY) {
                                printf("Disk %s not ready\n", devname);
                                continue;
                        }
                        if (ret) {
                                printf("ERROR: failure to add disk device %s, r = %lu\n",
                                       devname, ret & ~EFI_ERROR_MASK);
                                return ret;
                        }
 . . .

The distinction might get into if more than one
"LUN" is not handled vs. handled. (Not clear to
me at this point.)

Separately, it looks to me like the "20" figure
from the ERROR and Error messages are from:

#define EFI_ALREADY_STARTED             (EFI_ERROR_MASK | 20)

Based on the lack of another type of message it appears that
the code that returned that value was:

static efi_status_t efi_protocol_open(
                        struct efi_handler *handler,
                        void **protocol_interface, void *agent_handle,
                        void *controller_handle, uint32_t attributes)
{
        struct efi_open_protocol_info_item *item;
        struct efi_open_protocol_info_entry *match = NULL;
        bool opened_by_driver = false;
        bool opened_exclusive = false;

        /* If there is no agent, only return the interface */
        if (!agent_handle)
                goto out;

        /* For TEST_PROTOCOL ignore interface attribute */
        if (attributes != EFI_OPEN_PROTOCOL_TEST_PROTOCOL)
                *protocol_interface = NULL;

        /*
         * Check if the protocol is already opened by a driver with the same
         * attributes or opened exclusively
         */
        list_for_each_entry(item, &handler->open_infos, link) {
                if (item->info.agent_handle == agent_handle) {
                        if ((attributes & EFI_OPEN_PROTOCOL_BY_DRIVER) &&
                            (item->info.attributes == attributes))
                                return EFI_ALREADY_STARTED;
                } else {
. . .

Unfortunately, I do nto have the background to
readily/rapidly put all these pieces together
into an explanation or an idea of what would
need to change.
Comment 38 Klaus Küchemann 2021-03-05 00:46:06 UTC
(In reply to Mark Millard from comment #35)
yes, u-boot produces the error-message , so it's the trigger,
but is it the cause?? I doubt that.
 while of course possible,
that there's a patch in uboot-linux-versions that isn't connected to a FreeBSD-driver(or voltage-setting or bootefi or whatever unknown)(what would mean that the cause is in FreeBSD-boot-mechanism)... but that's all wild guessing without real knowledge about what odd thing happens here... 
if possible I would test a working linux-u-boot with FreeBSD(if exists and if possible) to prove there is no special-u-boot-implementation by a company which didn't made it to u-boot-upstream.


(In reply to Mark Millard from comment #37)
<<Unfortunately, I do not have the background to....>>

-> HPS
Comment 39 Mark Millard 2021-03-05 00:50:51 UTC
(In reply to Mark Millard from comment #37)

I got the lack-of-message evidence wrong: the
message would only show up for debug output
enabled contexts.

So here is the other place where EFI_ALREADY_STARTED
ends up being introduced:

efi_status_t EFIAPI efi_install_multiple_protocol_interfaces
                                (efi_handle_t *handle, ...)
{
        EFI_ENTRY("%p", handle);

        efi_va_list argptr;
        const efi_guid_t *protocol;
        void *protocol_interface;
        efi_handle_t old_handle;
        efi_status_t r = EFI_SUCCESS;
        int i = 0;

        if (!handle)
                return EFI_EXIT(EFI_INVALID_PARAMETER);

        efi_va_start(argptr, handle);
        for (;;) {
                protocol = efi_va_arg(argptr, efi_guid_t*);
                if (!protocol)
                        break;
                protocol_interface = efi_va_arg(argptr, void*);
                /* Check that a device path has not been installed before */
                if (!guidcmp(protocol, &efi_guid_device_path)) {
                        struct efi_device_path *dp = protocol_interface;

                        r = EFI_CALL(efi_locate_device_path(protocol, &dp,
                                                            &old_handle));
                        if (r == EFI_SUCCESS &&
                            dp->type == DEVICE_PATH_TYPE_END) {
                                EFI_PRINT("Path %pD already installed\n",
                                          protocol_interface);
                                r = EFI_ALREADY_STARTED;
                                break;
                        }
                }
Comment 40 Mark Millard 2021-03-05 01:02:18 UTC
(In reply to Klaus Küchemann from comment #38)

There is another test: eliminate FreeBSD from a
copy of the media, including not having a
copy of FreeBSD's loader.efi.

U-boot should still start and if this activity
is before loading FreeBSD's loader.efi copy
(bootaa64.efi) then likely still reports the
same error for the failed drive configuration.

Of course, this media would not complete a boot,
but we would only be checking if:

ERROR: failure to add disk device usb_mass_storage.lun0, r = 20
Eeror: Cannot initialize UEFI sub-system, r = 20
1258028 bytes read in 72 ms (16.7 MiB/s)
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Error: Cannot initialize UEFI sub-system, r = 20
EFI LOAD FAILED: continuing...

happens or not or if it does not even get that
far in the message sequence.

If the messages still occur, FreeBSD would then
have been confirmed as not involved.
Comment 41 Klaus Küchemann 2021-03-05 01:13:15 UTC
(In reply to Mark Millard from comment #40)
<<There is another test:..........If the messages still occur, FreeBSD would then
have been confirmed as not involved.>>

while I like the idea to make such a test :

that wouldn't be a real confirmation because:
we would know that it fails with&without a current unchanged fbsd-ufs-partition,
what we wouldn't know is:
would a change in fbsd-boot-mechanism(or driver or elsewhere)
fix the issue?  ;-)

--
@eimar.koort@tutamail.com  : there's a command in u-boot prompt:
'usb start' as far as I remember ...
won`t probably fix the issue but perhaps puts out information...

I don't think we can remote-debug this issue without having the hardware,
except a hardcore-specialist can enlighten us.
Comment 42 Mark Millard 2021-03-05 01:29:17 UTC
(In reply to Klaus Küchemann from comment #41)

I've replicated the failure using a reader for
multimedia cards that has 2 cards plugged in
(and there is the boot media in the normal
microsd card slot):

Scanning usb 0:1...
Found EFI removable media binary efi/boot/bootaa64.efi
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Error: Cannot initialize UEFI sub-system, r = 20
695648 bytes read in 22 ms (30.2 MiB/s)
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Error: Cannot initialize UEFI sub-system, r = 20
EFI LOAD FAILED: continuing...

So I can now experiment. The above and the original
report seem to both be examples of not handling
multiple LUNs in a single USB device.

I will note that the messages following the above are:

BOOTP broadcast 1
BOOTP broadcast 2
DHCP client bound to address 192.168.1.161 (435 ms)
*** ERROR: `serverip' not set
Cannot autoload with TFTPGET
missing environment variable: pxeuuid
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/01-dc-a6-32-b2-e2-aa
*** ERROR: `serverip' not set
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/C0A801A1
. . .

If I gather right, that is not from u-boot
any more but is from the RPi4B firmware: it appears
that u-boot quit when it could not initialize
UEFI. If so, nothing else that requires more u-boot
activity than that will happen.
Comment 43 Mark Millard 2021-03-05 01:37:42 UTC
(In reply to Mark Millard from comment #42)

Note: The experiment was with 13.0-BETA4 without removing
anything. The RPi4B has the default firmware ordering.

I should have looked to the end of the message sequence
(it is still in u-boot):

BOOTP broadcast 1
BOOTP broadcast 2
DHCP client bound to address 192.168.1.161 (435 ms)
*** ERROR: `serverip' not set
Cannot autoload with TFTPGET
missing environment variable: pxeuuid
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/01-dc-a6-32-b2-e2-aa
*** ERROR: `serverip' not set
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/C0A801A1
. . .
*** ERROR: `serverip' not set
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/default
*** ERROR: `serverip' not set
Config file not found
BOOTP broadcast 1
DHCP client bound to address 192.168.1.161 (8 ms)
*** ERROR: `serverip' not set
Cannot autoload with TFTPGET
BOOTP broadcast 1
DHCP client bound to address 192.168.1.161 (9 ms)
*** ERROR: `serverip' not set
Cannot autoload with TFTPGET
U-Boot> 

So it is still in u-boot but it automatically
did not try any more than the above, just going
to its prompt after the network alternative
did not make progress.
Comment 44 Klaus Küchemann 2021-03-05 01:38:23 UTC
(In reply to Mark Millard from comment #42)
(In reply to Mark Millard from comment #42)

good work to replicate the issue !

while off-topic :
the BOOTP/DHCP-messages are from u-boot.
it switches to try to netboot after failing disk-detection.
when it says : ``serverip' not set" 
that means that you can e.g. :
`setenv serverip 10.0.0.23`
tftp file.xy
boot
(presumed you have a  pxe-setup @youserverip)
Comment 45 Klaus Küchemann 2021-03-05 02:03:23 UTC
(In reply to Mark Millard from comment #42)
<<<<So I can now experiment. The above and the original
report seem to both be examples of not handling
multiple LUNs in a single USB device.>>>>

that was a real fu**ing good idea ! :-)

since we are on rpi4 and have a special dedicated pcie/xhci-driver ,
I would be interested in a test with e.g. amd64-driver or RPI3-USB or similar to get closer to the root cause(if it's in FreeBSD instead of u-boot)...
but I will not test tonight, working on other things...
Comment 46 Mark Millard 2021-03-05 02:24:54 UTC
(In reply to Mark Millard from comment #43)

All the experiments so far have the same result:

Both of the following lead to the u-boot attempting
a network boot instead of other alternatives in my
configuration:

A) There is not EFI/BOOT/bootaa64.efi or the like to find
   on a/the microsd card or via a USB device/LUN.

B) There is one or more EFI/BOOT/bootaa64.efi to find but
   there are 2 LUNs on one device. Which device(s) have
   a EFI/BOOT/bootaa64.efi does not seem to matter.

So u-boot gives up and tries a network boot, giving
no opportunity to do more from teh microsd card media
or USB media in both types of context.

Note: I do not have a network boot environment set up,
never have. But I've not done anything to force the
RPi4B from attempting such booting. The network boot
attempts are just default behavior, not something I've
explicitly configured.

Unless u-boot is changed so (B) does not lead to a UEFI
initialization error, it appears that network booting
would need to be used or some other form of u-boot
configuration control would need to be used.

I've no evidence that u-boot can be told to retry UEFI
initialization after its first failure. But I'm no
u-boot expert.
Comment 47 Mark Millard 2021-03-05 02:33:36 UTC
(In reply to Klaus Küchemann from comment #45)

u-boot never starts FreeBSD or its loader.efi
copy in the failing examples (no output from
the bootaa64.efi at all). FreeBSD is not
involved yet at the time of u-boot switching
to attempting a network boot: u-boot is what
is still active.
Comment 48 Klaus Küchemann 2021-03-05 02:40:00 UTC
(In reply to Mark Millard from comment #46)
<<<that network booting
would need to be used or some other form of u-boot
configuration control would need to be used.>>>>

if you are willing to test whether  your genius  2-disk-setup-replication is 
detected on a network boot I have something for you so that you can boot to pxe-server 
without knowing/typing any u-boot- or fbsd-prompt-command :

https://sourceforge.net/projects/freebsdonlyrpi3and4netbootimg/

as said: awesome idea you had !
Comment 49 Klaus Küchemann 2021-03-05 02:56:58 UTC
(In reply to Mark Millard from comment #46)
<<<I've no evidence that u-boot can be told to retry UEFI
initialization after its first failure. But I'm no
u-boot expert.>>>

hint:
at first trial you can interrupt to the u-boot-prompt 
and type :
`usb start`

give a question mark ( ? ) for further command options...

... 'printenv' shows current setup...
'setenv xyz' to overwrite settings....
just for a first hint...
Comment 50 Klaus Küchemann 2021-03-05 02:59:50 UTC
.. ah, forgot to say :
`mmcinfo` should be an interesting request to u-boot-prompt for the first trials..

mmc devices are written as e.g :
`mmc 0:1`
Comment 51 Klaus Küchemann 2021-03-05 03:09:50 UTC
sorry for triple posting... very tired and multiple things tonight here ;-)....

e.g. 'setenv devnum 1` ... for the case that you can detect any uSD by 'usb start` before...
`boot` or `run mmc_boot`after 'setenv devnum xy'.
I doubt that it will succeed but perhaps gives information...

good night
Comment 52 Mark Millard 2021-03-05 03:19:59 UTC
(In reply to Klaus Küchemann from comment #48)

I do not expect going this direction would help
identify a fix to the problem eimar ran into.
(No indication of a desire for network booting.)

And I've no interest in setting up a network
boot environment for myself at this time.
Network configuration is not something I'm
knowledgable about. I provide no outward
bound services and have a mostly default
setup for very basic operation.

But, looking, I've no clue how to address the
lack of handling of multiple populated LUNs
in a USB device for u-boot. Identifying the
multiple-populated-LUNs context looks to be
about my limit for eimar's problem.

It may well be that a u-boot build configuration
could be specified that would work already but
FreeBSD's port just does not do such. I've
no clue if this is just a configuration problem
for building u-boot vs. more being involved.
Comment 53 commit-hook freebsd_committer 2021-03-05 10:33:10 UTC
A commit references this bug:

Author: manu
Date: Fri Mar  5 10:32:15 UTC 2021
New revision: 567376
URL: https://svnweb.freebsd.org/changeset/ports/567376

Log:
  sysutils/rpi-firmware: Update to 20210303

  Fix USB on RPI4

  PR:	252971, 253981, 253983
  MFH:		2021Q1

Changes:
  head/sysutils/rpi-firmware/Makefile
  head/sysutils/rpi-firmware/distinfo
  head/sysutils/rpi-firmware/pkg-plist
Comment 54 commit-hook freebsd_committer 2021-03-05 10:35:13 UTC
A commit references this bug:

Author: manu
Date: Fri Mar  5 10:33:54 UTC 2021
New revision: 567377
URL: https://svnweb.freebsd.org/changeset/ports/567377

Log:
  MFH: r567376

  sysutils/rpi-firmware: Update to 20210303

  Fix USB on RPI4

  PR:	252971, 253981, 253983

Changes:
_U  branches/2021Q1/
  branches/2021Q1/sysutils/rpi-firmware/Makefile
  branches/2021Q1/sysutils/rpi-firmware/distinfo
  branches/2021Q1/sysutils/rpi-firmware/pkg-plist
Comment 55 Mark Millard 2021-03-06 03:07:41 UTC
So far I've not found any u-boot-2020.10 from
the port's code supporting UEFI's :

#define MSG_DEVICE_LOGICAL_UNIT_DP  0x11
typedef struct {
  EFI_DEVICE_PATH_PROTOCOL      Header;
  ///
  /// Logical Unit Number for the interface.
  ///
  UINT8                         Lun;
} DEVICE_LOGICAL_UNIT_DEVICE_PATH;

It appears that for USB the above is used in
UEFI for LUNs. (Some other things like SCSI
have LUN fields built in instead of in a separate.)

[Clearly: I grabbed some notation from something
else that does support such, in this case from
a EDK2 file.]

I only see in u-boot-2020.10/include/efi_api.h :

#define DEVICE_PATH_TYPE_MESSAGING_DEVICE       0x03
#  define DEVICE_PATH_SUB_TYPE_MSG_ATAPI        0x01
#  define DEVICE_PATH_SUB_TYPE_MSG_SCSI         0x02
#  define DEVICE_PATH_SUB_TYPE_MSG_USB          0x05
#  define DEVICE_PATH_SUB_TYPE_MSG_MAC_ADDR     0x0b
#  define DEVICE_PATH_SUB_TYPE_MSG_USB_CLASS    0x0f
#  define DEVICE_PATH_SUB_TYPE_MSG_SATA         0x12
#  define DEVICE_PATH_SUB_TYPE_MSG_NVME         0x17
#  define DEVICE_PATH_SUB_TYPE_MSG_SD           0x1a
#  define DEVICE_PATH_SUB_TYPE_MSG_MMC          0x1d

with matching struct types. Nothing for 0x11.

May be some u-boot's have been modified to ignore
USB devices with multiple populated storage LUNs
as far as UEFI support is concerned --instead of
rejecting the whole UEFI initialization. But I see
no evidence of even a start at handling an
equivalent of DEVICE_LOGICAL_UNIT_DEVICE_PATH for
USB.

That results in the conflicting context (via an
incomplete path specification).

Of course, ignoring the USB LUNs would not make
the multi-drive context work, though it could
let the microsd card boot occur for the example
context that cause the bugzilla entry.

It looks like providing USB storage multi-LUN
support would be a fairly significant upstream
effort for active u-boot developers that are
familiar with working on the code.

I have not checked u-boot code from after what
the sysutils/u-boot-rpi* get.
Comment 56 Klaus Küchemann 2021-03-06 03:15:35 UTC
(In reply to commit-hook from comment #54)
I have meanwhile replicated & successfully booted your genius multi-LUN-2-disk-replication-setup :-) Ha Ha
great catch you did, to say that again !
I`m in the middle of compiling a patch into u-boot and will upload that u-boot-compilation if succeeds...
if I can't patch it tonight I will give some hints for eimar.koort@tutamail.com 
how he can boot his setup...  no warranties that  I can patch it tonight..
Comment 57 Mark Millard 2021-03-06 03:55:50 UTC
(In reply to Mark Millard from comment #55)

In:

QUOTE
Of course, ignoring the USB LUNs would not make
the multi-drive context work, though it could
let the microsd card boot occur for the example
context that cause the bugzilla entry.
END QUOTE

"work" was meant to be "as the boot drive". I
was not trying to imply anything about FreeBSD
being unable to use the drives.

Ignoring the multiple-storage-LUNs on a a device
likely would make things operate if those LUNs
were never to be boot media.
Comment 58 Mark Millard 2021-03-06 04:21:41 UTC
(In reply to Klaus Küchemann from comment #56)

Note: I added a second multimedia reader with
media on it as well to make a more complete test
case.

All the media are simply formatted as one msdsofs
on the MBR media as stands, no files or directories
added.

On the readers I used, the LUNs that I populated
are 1 and 3 on each. (There are a bunch more LUNs
for various kinds of media. I doubt the readers
would work if all were populated at once.)



As far as I can tell, an existing workaround is
to connect the device(s) that have multiple storage
LUNs only after booting if once's context allows for
such a manual operation. (Presumes not booting via
one of those LUNs.)

Have fun getting it going in whatever way you are
working on.
Comment 59 Klaus Küchemann 2021-03-06 04:40:12 UTC
(In reply to Mark Millard from comment #58)

first u-boot-source-patch trials (of course) failed for me tonight, no real surprise,  :-) Ha Ha

O.K., 
here`s a workaround I hope that will work for your test-setup and the bug-reporter`s geekworm-board:

interrupt to get into the u-boot-prompt(e.g. by holding the TAB-key).
--
type exactly following one-liner :
usb stop; boot
(then press Enter)
..... this dirty workaround is ONLY useful for this exact multi-lun-usb-setup when booting from the uSD-slot... it stops the failed controller in u-boot and Rob Crowston`s xhci-driver will restart USB later through another xhci-reset
Comment 61 Mark Millard 2021-03-06 05:03:41 UTC
(In reply to Klaus Küchemann from comment #59)

I use a serial console but others may not.

"usb stop" might stop USB keyboard input and such
until something happens to start up usb again. I'm
not sure when that would be in various cases.

RPi4B's 4GB and less (v1.1, v1.2 at least): I've
never checked if "Rob Crowston`s xhci-driver will
restart USB later through another xhci-reset" or
not. Nor do I have a means of checking models
newer than the RPi4B 8GB model. In fact, no means
of checking any v1.2 RPi4B either.
Comment 62 Klaus Küchemann 2021-03-06 05:11:52 UTC
(In reply to Mark Millard from comment #61)
<<<I use a serial console but others may not.
"usb stop" might stop USB keyboard input>>>>>

you can do it without serial console directly in the glass-console ;-)(HDMI) 
if you want :
the exact one-liner :
usb stop; boot
(then press Enter)

I would be interested if it works for you (because than it will probably work for the geek worm-board as a workaround until we have an idea how to patch in u-boot)
Comment 63 Mark Millard 2021-03-06 05:16:22 UTC
(In reply to Klaus Küchemann from comment #60)

I'm not sure what the point of the two references is
but I'll note that I've had no u-boot command problems
looking at and dealing with the multi-LUN-storage
devices: that context is not UEFI handling of it at
all. The only type of context for which I know of
multi-LUN storage devices having problems is for
UEFI support, where u-boot's USB handling is
incomplete for the issue and ends up generating UEFI
references that do not include an LUN identification
at all.

I probably do not need to know what the two references
are for. (This bugzilla is already large just from trying
to get very basic information about the problem.)
Comment 64 Mark Millard 2021-03-06 05:17:38 UTC
(In reply to Klaus Küchemann from comment #62)

I'll have to get HDMI set up first. My context is
normally headless.
Comment 65 Klaus Küchemann 2021-03-06 05:25:11 UTC
(In reply to Mark Millard from comment #63)
<<This bugzilla is already large just from trying
to get very basic information about the problem.>>

yeah, We always do that way until nobody knows what happens :-) Ha Ha, ... but as long as it leads to a bugfix or workaround ..
  and you were right:
it's a u-boot bug .
Comment 66 Mark Millard 2021-03-06 05:50:18 UTC
(In reply to Mark Millard from comment #64)

I am able to login at the login prompt via the
USB keyboard afterwards on the RPi4B 8GB.

But I'm unable do any loader input. So, for
example, no ability to do a "boot -s" via
loader input: no control of the boot at
that stage (beyond cutting power or the like).

Similarly for a RPi4B 4GB v1.1 that I tried.

I could imagine a HDMI that was slow displaying
from powerup, leading to need to type to get
into u-boot before being able to see the prompt.

(For the most part, the RPi4B*'s that I have access
to have BOOT_UART=1 in the eeprom and config.txt
material to enable debug output for that stage as
well. This makes the early activity take a little
longer, giving a little more time before u-boot
shows up compared to otherwise.)
Comment 67 Klaus Küchemann 2021-03-06 06:09:23 UTC
(In reply to Mark Millard from comment #66)
<<no ability to do a "boot -s" via
loader input:>>
yes , of course, because xhci will be reset(reinitialized) in  later boot-stage by Rob`s driver .
 professionals like you have anyway headless-context or both serial&glass-console ;-)
so you can interrupt to get to loader-input by serial-console even if the useless glass is connected :-) Ha Ha 

so what we have reached:
a workaround for this issue , so that eimar.koort@tutamail.com hopefully 
doesn't have to plug/unplug his Geekworm-board 20 times a day..

so again for readers :
one-liner @uboot-prompt :
usb stop; boot
(then press Enter)

`guess we can close this issue  because detected by you as u-boot-bug..


thanks for testing and your great replication-idea !
Comment 68 eimar.koort 2021-03-06 06:34:03 UTC
Hi,

Installed 13-RC1 and it workaround (usb stop; boot) works for now.

Thank you all!
Comment 69 Klaus Küchemann 2021-03-06 06:40:19 UTC
(In reply to eimar.koort from comment #68)
glad that it works ! ...
all credits to Mark ...
wouldn't have been a solution to unplug/replug the Geekworm-board if power is on... some boards are extremely sensitive.. 2 weeks ago I toasted my RockPro64 only by one-touch-hotplug(although I guess the RPI4 is a bit less sensitive in that discipline).
Comment 70 Mark Millard 2021-03-06 07:04:41 UTC
(In reply to eimar.koort from comment #68)

I'll note that 13.0-RC1 does not have the recent
sysutils/rpi-firwmare update that fixes some
USB related issues, among other things. It will
be 13.0-RC2 before 13.0 gets such. You might
want to update once you get ahold of an updated
sysutils/rpi-firwmare . (None of this deals with
the u-boot multi-storage-LUN device handling
issue(s).)

Future:
Also, I'm not sure that snapshot builds and
release builds automatically track
sysutils/u-boot-rpi* updates if any happen. Even
now, I think the official builds might not be
using the latest sysutils/u-boot-rpi-arm64. (I
got some observable differences in my test
builds that I would not otherwise expect.)
Comment 71 eimar.koort 2021-03-06 11:52:36 UTC
Hi,
Since there is some clarity in this, i'm closing this PR.
Thanks again!
Comment 72 Mark Millard 2021-03-07 07:18:20 UTC
FYI, from a fedora f33 context with the bootable
media on unit 1 and the empty file system on
unit 3 of the multi-media card reader, nothing
in the RPi4B's microsd card slot:

Scanning disk usb_mass_storage.lun1...
** Unrecognized filesystem type **
** Unrecognized filesystem type **
Scanning disk usb_mass_storage.lun3...
ERROR: failure to add disk device usb_mass_storage.lun3, r = 20
Error: Cannot initialize UEFI sub-system, r = 20
2676208 bytes read in 41 ms (62.2 MiB/s)
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Error: Cannot initialize UEFI sub-system, r = 20
EFI LOAD FAILED: continuing...
BOOTP broadcast 1
DHCP client bound to address 192.168.1.171 (121 ms)
*** ERROR: `serverip' not set

If the bootable media was on unit 3 and the empty
file system on unit 1, then it directly started
attempting a network boot, not trying unit 3 at
all, just unit 1.

So FreeBSD's u-boot-rpi4-arm64 configuration's
properties for such multi-storage-LUN USB devices
need not be unusual.
Comment 73 Mark Millard 2021-06-06 21:52:20 UTC
(In reply to Mark Millard from comment #72)

See:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256441

for other examples of getting the:

ERROR: failure to add disk device usb_mass_storage.lun0 . . .

type of error messages. An example was via a powered hub
and the 2 storage devices being plugged into the powered
hub instead of directly pluggged into the RPi4B. This is
sort of analogous to a single device with multiple storage
LUNs populated.

I'm unsure at this point if the submitter's context is
analogous or not. So there may be more ways to get this.
Comment 74 Mark Millard 2021-06-06 22:44:33 UTC
(In reply to Mark Millard from comment #73)

Turns out part of my initial testing context was
getting some behavior from the RPi4B not being
able to well power both USB3 SSDs that I was using
in the tests.

So I switched to a matched pair of lower-power
thumb drives. In that context, direct connection
to RPi4B USB3 ports and use via a powered USB3
hub got the same U-Boot failures, even for the
boot media being a microsd card. Getting:

ERROR: failure to add disk device usb_mass_storage.lun0

in U-Boot and the consequence behavior is not as
special-case as we thought earlier: having more than
one storage device/LUN on the same RPi4B port is not
required to see the problem in the U-Boot behavior.

I will note that when I had a mix of one thumb drive
and one USB3 SSD, things worked, both directly connected
and via the powered hub. Somehow having sufficiently
well matched device types looks to be involved in getting
the problem. (This context looked to be well powered, all
devices showing up at the expected stages and the like.)
Comment 75 Mark Millard 2021-06-06 23:13:48 UTC
(In reply to Mark Millard from comment #74)

I have replicated the problem with Fedora 34 Server
on a RPi4B. Fedora 34 also uses U-Boot. I used the
powered hub context to attach the thumb drives
because I do not have microsd card media around
for Fedora 34, just a USB3 SSD.

So both RPi4B USB3 ports were in use, one for the
USB3 SSD boot media and one for the powered hub
with the 2 thumb drives.

U-Boot 2021.04-rc3 (Mar 13 2021 - 00:00:00 +0000)

DRAM:  7.9 GiB
RPI 4 Model B (0xd03114)
MMC:   mmcnr@7e300000: 1, emmc2@7e340000: 0
Loading Environment from FAT... Card did not respond to voltage select! : -110
In:    serial
Out:   serial
Err:   serial
Net:   eth0: ethernet@7d580000
PCIe BRCM: link up, 5.0 Gbps x1 (SSC)
starting USB...
Bus xhci_pci: Register 5000420 NbrPorts 5
Starting the controller
USB XHCI 1.00
scanning bus xhci_pci for devices... 7 USB Device(s) found
       scanning usb for storage devices... 3 Storage Device(s) found
Hit any key to stop autoboot:  0 
Card did not respond to voltage select! : -110
Card did not respond to voltage select! : -110

Device 0: Vendor: OWC      Rev: 0    Prod: Envoy Pro mini  
            Type: Hard Disk
            Capacity: 228936.5 MB = 223.5 GB (468862128 x 512)
... is now current device
Scanning usb 0:1...
Found EFI removable media binary efi/fedora/grubaa64.efi
** Unrecognized filesystem type **
** Unrecognized filesystem type **
** Unrecognized filesystem type **
** Unrecognized filesystem type **
** Unrecognized filesystem type **
** Unrecognized filesystem type **
2692592 bytes read in 11 ms (233.4 MiB/s)
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Card did not respond to voltage select! : -110
Scanning disk mmcnr@7e300000.blk...
Disk mmcnr@7e300000.blk not ready
Card did not respond to voltage select! : -110
Scanning disk emmc2@7e340000.blk...
Disk emmc2@7e340000.blk not ready
Scanning disk usb_mass_storage.lun0...
** Unrecognized filesystem type **
** Unrecognized filesystem type **
Scanning disk usb_mass_storage.lun0...
** Unrecognized filesystem type **
Scanning disk usb_mass_storage.lun0...
ERROR: failure to add disk device usb_mass_storage.lun0, r = 20
Error: Cannot initialize UEFI sub-system, r = 20
EFI LOAD FAILED: continuing...
BOOTP broadcast 1
DHCP client bound to address 192.168.1.191 (64 ms)
*** ERROR: `serverip' not set
Cannot autoload with TFTPGET
missing environment variable: pxeuuid
missing environment variable: bootfile
Retrieving file: . . .