Using 13.0-RELEASE image on a 16GB SD card, written using the RPi imager tool. With 0 or 1 USB storage devices inserted, the system boots fine. With 2-4 USB storage devices inserted, the system will not boot. Looks like U-Boot is getting stuck in this situation and not able to read the second stage bootloader from the SD card mmc0 storage. May be related to #255080 I have transcribed the console output below (may have some minor typos). With 1 USB storage device inserted: ``` Net: No ethernet found starting USB... Bus usb@7e990000: USB DWC2 scanning bus usb@7e980000 for devices... 5 USB Device(s) found scanning usb for storage devices... 1 Storage Device(s) found Hit any key to stop autoboot: 0 [boot continues successfully] ``` With 2 USB storage device inserted: ``` Net: No ethernet found starting USB... Bus usb@7e990000: USB DWC2 scanning bus usb@7e980000 for devices... 6 USB Device(s) found scanning usb for storage devices... 2 Storage Device(s) found Hit any key to stop autoboot: 0 switch to partitions #0, OK mmc0 is current device Scanning mmc 0:1... Found EFI removable media binary efi/boot/bootaa64.efi libfdt fdt_check_header(): FDT_ERR_BADMAGIC Scanning disk mmc@7e300000.blk... ** Unrecognized filesystem type ** Scanning disk usb_mass_storage.lun0... ** Unrecognized filesystem type ** Scanning disk usb_mass_storage.lun0... ERROR: Failure to add disk device usb_mass_storage.lun0, r=20 Error: Cannot initialize UEFI sub-system, r=20 125912 bytes read in 125 ms (9.6 MiB/s) libfdt fdt_check_header(): FDT_ERR_BADMAGIC Error: Cannot initialize UEFI sub-system, r=20 EFI LOAD FAILED: continuing... MMC Device 1 not found no mmc device at slot 1 Device 0: Vendor: Generic Rev: 2.00 Prod: Flash Disk Type: Removable Hard Disk Capacity: 7680.0 MB = 7.5 GB (15728640 x 512) ... is now current device ** Unrecognized filesystem type ** Waiting for Ethernet connection... done. BOOTP broadcast 1 ... [boot never succeeds] ```
(In reply to Roger Leigh from comment #0) This is not a duplicate of 255080: 255080 does not involve the quoted message at all. This looks to be a duplicate of: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253983 which also involves the error message text: ERROR: Failure to add disk device usb_mass_storage.lun0 . . . The same error is observable on other OS's that use U-Boot, such as Fedora 33 attempting to boot a RPi in a certain type of context. So it is not a FreeBSD issue, but a U-Boot issue, for that type of context. For 253983 the context was a single USB device with multiple storage Logic Units. A multimedia reader with multiple media present at the same time was sufficient to show the problem. But the original report was not for a multi-media reader but some other type of device with multiple storage LUNs. If you have a context where no individual USB device has more than one storage LUN involved, that would be new information. But you do not provide a description of the device(s) involved, the LUNs involved, what ports the devices are plugged into, what partitions have what kinds of file systems with what kinds of content, etc. Still, based on the error message, I expect that this is U-Boot mishandling having multiple populated storage LUNs and is not something FreeBSD can fix. The proper place for submittal would then be upstream-U-Boot.
Hi Mark, This is a representative sample of the USB pendrives used. All are 9.5GB identical parts. Geom name: da0 modified: false state: OK fwheads: 255 fwsectors: 63 last: 15728599 first: 40 entries: 128 scheme: GPT Providers: 1. Name: da0p1 Mediasize: 8053022720 (7.5G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 20480 Mode: r1w1e2 efimedia: HD(1,GPT,82a1cdfa-9901-11eb-b20f-b827ebda9d55,0x28,0xefffb0) rawuuid: 82a1cdfa-9901-11eb-b20f-b827ebda9d55 rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b label: gitlab length: 8053022720 offset: 20480 type: freebsd-ufs index: 1 end: 15728599 start: 40 Consumers: 1. Name: da0 Mediasize: 8053063680 (7.5G) Sectorsize: 512 Mode: r1w1e3 I have tried with: a) no partition table b) GPT partition table (no entries) c) GPT partition table (single entry as above, ufs or zfs partition type) d) geom_stripe with (a) and (c) In all cases: * zero or one USB pendrives boots * two or three USB pendrives fails
(In reply to Roger Leigh from comment #2) Well, I just tried a USB based boot for which 2 USB storage devices were plugged into RPi USB ports but no microsd card in use. It got: . . . U-Boot 2021.04 (Apr 08 2021 - 10:46:22 +0000) DRAM: 7.9 GiB RPI 4 Model B (0xd03114) . . . Bus xhci_pci: Register 5000420 NbrPorts 5 Starting the controller USB XHCI 1.00 scanning bus xhci_pci for devices... Device NOT ready Request Sense returned 02 3A 00 Device NOT ready Request Sense returned 02 3A 00 Device NOT ready Request Sense returned 02 3A 00 5 USB Device(s) found scanning usb for storage devices... 2 Storage Device(s) found Hit any key to stop autoboot: 2 1 0 Card did not respond to voltage select! : -110 Device 0: Vendor: OWC Rev: 0 Prod: Envoy Pro mini Type: Hard Disk Capacity: 228936.5 MB = 223.5 GB (468862128 x 512) ... is now current device Scanning usb 0:1... libfdt fdt_check_header(): FDT_ERR_BADMAGIC Scanning disk mmc@7e300000.blk... Disk mmc@7e300000.blk not ready Card did not respond to voltage select! : -110 Scanning disk emmc2@7e340000.blk... Disk emmc2@7e340000.blk not ready Scanning disk usb_mass_storage.lun0... ** Unrecognized filesystem type ** ** Unrecognized filesystem type ** ** Unrecognized filesystem type ** Scanning disk usb_mass_storage.lun1... Found 7 disks ** Unable to read file ubootefi.var ** Failed to load EFI variables BootOrder not defined EFI boot manager: Cannot load any image Found EFI removable media binary efi/boot/bootaa64.efi . . . and it completed booting just fine. So it looks like I'd need to better approximate your boot context in order to see the problem. I'll see about using a 13.0-RELEASE microsd card boot media with the 2 devices still plugged in. Another possibility is that swapping the ports usage might make a difference in my context.
(In reply to Mark Millard from comment #3) Booting a 13.0-RELEASE microsd card had no problem with the 2 USB storage media already plugged into the RPi ports at power up. But it turns out that the media that I already had around had the same u-boot.bin vintage on it as my normal boot. It might not be the same as on a from-scratch 13.0-RELEASE media would have. What does your U-Boot report compared to mine: U-Boot 2021.04 (Apr 08 2021 - 10:46:22 +0000) ?
My U-boot version: U-Boot> version U-Boot> 2020.10 (Apr 09 2021 - 03:55:54 +0000) aarch64-none-elf-gcc (FreeBSD Ports Collection for aarch64noneelf) 8.4.0 GNU ld (GNU Binutils) 2.33.1 This is what was provided on the 13.0-RELEASE image.
(In reply to Mark Millard from comment #3) I've repeated the problem with a powered hub involved: Powerd hub connected to a RPi USB3 port and 2 storage devices plugged into the powered hub, both USB3 SSDs. (Previously I had one USB3 SSD and one multi-media reader with only one media in it and I was direct connecting those to RPi USB ports. I'll try direct connecting the two USB3 SSDs later.) scanning bus xhci_pci for devices... Device not responding to set address. USB device not accepting new address (error=80000000) 6 USB Device(s) found scanning usb for storage devices... 2 Storage Device(s) found Hit any key to stop autoboot: 2 1 0 switch to partitions #0, OK mmc0 is current device Scanning mmc 0:1... libfdt fdt_check_header(): FDT_ERR_BADMAGIC 7[r[999;999H[6n8Scanning disk mmc@7e300000.blk... Disk mmc@7e300000.blk not ready Scanning disk emmc2@7e340000.blk... ** Unrecognized filesystem type ** Scanning disk usb_mass_storage.lun0... ** Unrecognized filesystem type ** ** Unrecognized filesystem type ** ** Unrecognized filesystem type ** Scanning disk usb_mass_storage.lun0... ERROR: failure to add disk device usb_mass_storage.lun0, r = 20 Error: Cannot initialize UEFI sub-system, r = 20 Found EFI removable media binary efi/boot/bootaa64.efi 1259212 bytes read in 72 ms (16.7 MiB/s) libfdt fdt_check_header(): FDT_ERR_BADMAGIC Error: Cannot initialize UEFI sub-system, r = 20 EFI LOAD FAILED: continuing... Device 0: Vendor: OWC Rev: 0 Prod: Envoy Pro mini Type: Hard Disk Capacity: 228936.5 MB = 223.5 GB (468862128 x 512) ... is now current device Scanning usb 0:1... libfdt fdt_check_header(): FDT_ERR_BADMAGIC Error: Cannot initialize UEFI sub-system, r = 20 Found EFI removable media binary efi/boot/bootaa64.efi 1289100 bytes read in 7 ms (175.6 MiB/s) libfdt fdt_check_header(): FDT_ERR_BADMAGIC Error: Cannot initialize UEFI sub-system, r = 20 EFI LOAD FAILED: continuing... ethernet@7d580000 Waiting for PHY auto negotiation to complete......... TIMEOUT ! bcmgenet: PHY startup failed: -110 missing environment variable: pxeuuid missing environment variable: bootfile Retrieving file: . . .
(In reply to Mark Millard from comment #6) [It appears that the RPi4B is marginal for powering both USB3 SSDs and the other two USB devices I normally have plugged in. So I've unplugged the USB keyboard but still have the EtherNet dongle on a USB2 port.] It has booted with the two USB3 SSDs in the USB3 ports that previously were on the powered hub (so previously using only one USB3 port with storage devices on the RPi4B). In other words, for what I've done to get the type of problem, the powered-hub with devices plugged in is analogus to the previously identified case of a single device with multiple storage LUNs populated. Absent such an anlogy, I've not gotten the problem so far. What I do not know is if your context is also analogus or not. Also: My context suggests that an updated U-Boot is not going to help at this point, at least for an "analogous" way of setting up the problem.
(In reply to Mark Millard from comment #7) Ignore comment 6: looks like there are still power issues and only one of the 2 USB3 SSDs was found. I'll try without the EtherNet dongle.
(In reply to Mark Millard from comment #8) Unfortunately, even just the 2 USB3 SSDs seems to require too much power from the RPi4B. So I found a flash drive to plug in instead of the 2nd USB3 SSD. But it turns out that the combination works both for direct plug-in and for the same USB3 powered hub type of test. May be 2 devices of the same "type" via one RPi USB(3?) port is part of the context for getting the problem?
(In reply to Mark Millard from comment #9) Sure enough, replacing the USB3 SSD with another flash drive of the same type as the other one on the USB3 powered hub got back to: USB XHCI 1.00 scanning bus xhci_pci for devices... 7 USB Device(s) found scanning usb for storage devices... 2 Storage Device(s) found Hit any key to stop autoboot: 0 switch to partitions #0, OK mmc0 is current device Scanning mmc 0:1... libfdt fdt_check_header(): FDT_ERR_BADMAGIC Scanning disk mmc@7e300000.blk... Disk mmc@7e300000.blk not ready Scanning disk emmc2@7e340000.blk... ** Unrecognized filesystem type ** Scanning disk usb_mass_storage.lun0... Scanning disk usb_mass_storage.lun0... ERROR: failure to add disk device usb_mass_storage.lun0, r = 20 Error: Cannot initialize UEFI sub-system, r = 20 Found EFI removable media binary efi/boot/bootaa64.efi 1259212 bytes read in 72 ms (16.7 MiB/s) libfdt fdt_check_header(): FDT_ERR_BADMAGIC Error: Cannot initialize UEFI sub-system, r = 20 EFI LOAD FAILED: continuing... Device 0: Vendor: SanDisk Rev: 0001 Prod: Extreme Type: Removable Hard Disk Capacity: 15272.0 MB = 14.9 GB (31277232 x 512) ... is now current device Scanning usb 0:1... libfdt fdt_check_header(): FDT_ERR_BADMAGIC Error: Cannot initialize UEFI sub-system, r = 20 ethernet@7d580000 Waiting for PHY auto negotiation to complete......... TIMEOUT ! bcmgenet: PHY startup failed: -110 missing environment variable: pxeuuid missing environment variable: bootfile Retrieving file: . . . And it turns out that directly plugging them in to the 2 RPi4B ports (no powered hub) also gets that behavior: USB XHCI 1.00 scanning bus xhci_pci for devices... 5 USB Device(s) found scanning usb for storage devices... 2 Storage Device(s) found Hit any key to stop autoboot: 2 1 0 switch to partitions #0, OK mmc0 is current device Scanning mmc 0:1... libfdt fdt_check_header(): FDT_ERR_BADMAGIC Scanning disk mmc@7e300000.blk... Disk mmc@7e300000.blk not ready Scanning disk emmc2@7e340000.blk... ** Unrecognized filesystem type ** Scanning disk usb_mass_storage.lun0... ** Unrecognized filesystem type ** Scanning disk usb_mass_storage.lun0... ERROR: failure to add disk device usb_mass_storage.lun0, r = 20 Error: Cannot initialize UEFI sub-system, r = 20 Found EFI removable media binary efi/boot/bootaa64.efi 1259212 bytes read in 72 ms (16.7 MiB/s) libfdt fdt_check_header(): FDT_ERR_BADMAGIC Error: Cannot initialize UEFI sub-system, r = 20 EFI LOAD FAILED: continuing... Device 0: Vendor: SanDisk Rev: 0001 Prod: Extreme Type: Removable Hard Disk Capacity: 15272.0 MB = 14.9 GB (31277232 x 512) ... is now current device Scanning usb 0:1... libfdt fdt_check_header(): FDT_ERR_BADMAGIC Error: Cannot initialize UEFI sub-system, r = 20 ethernet@7d580000 Waiting for PHY auto negotiation to complete......... TIMEOUT ! bcmgenet: PHY startup failed: -110 missing environment variable: pxeuuid missing environment variable: bootfile Retrieving file: So no external configuration of multiple storage devices accessed via one RPi4B port is required.
(In reply to Mark Millard from comment #10) I have replicated the problem with Fedora 34 Server on a RPi4B. Fedora 34 also uses U-Boot. I used the powered hub context to attach the thumb drives because I do not have microsd card media around for Fedora 34, just a USB3 SSD. So both RPi4B USB3 ports were in use, one for the USB3 SSD boot media and one for the powered hub with the 2 thumb drives. U-Boot 2021.04-rc3 (Mar 13 2021 - 00:00:00 +0000) DRAM: 7.9 GiB RPI 4 Model B (0xd03114) MMC: mmcnr@7e300000: 1, emmc2@7e340000: 0 Loading Environment from FAT... Card did not respond to voltage select! : -110 In: serial Out: serial Err: serial Net: eth0: ethernet@7d580000 PCIe BRCM: link up, 5.0 Gbps x1 (SSC) starting USB... Bus xhci_pci: Register 5000420 NbrPorts 5 Starting the controller USB XHCI 1.00 scanning bus xhci_pci for devices... 7 USB Device(s) found scanning usb for storage devices... 3 Storage Device(s) found Hit any key to stop autoboot: 0 Card did not respond to voltage select! : -110 Card did not respond to voltage select! : -110 Device 0: Vendor: OWC Rev: 0 Prod: Envoy Pro mini Type: Hard Disk Capacity: 228936.5 MB = 223.5 GB (468862128 x 512) ... is now current device Scanning usb 0:1... Found EFI removable media binary efi/fedora/grubaa64.efi ** Unrecognized filesystem type ** ** Unrecognized filesystem type ** ** Unrecognized filesystem type ** ** Unrecognized filesystem type ** ** Unrecognized filesystem type ** ** Unrecognized filesystem type ** 2692592 bytes read in 11 ms (233.4 MiB/s) libfdt fdt_check_header(): FDT_ERR_BADMAGIC Card did not respond to voltage select! : -110 Scanning disk mmcnr@7e300000.blk... Disk mmcnr@7e300000.blk not ready Card did not respond to voltage select! : -110 Scanning disk emmc2@7e340000.blk... Disk emmc2@7e340000.blk not ready Scanning disk usb_mass_storage.lun0... ** Unrecognized filesystem type ** ** Unrecognized filesystem type ** Scanning disk usb_mass_storage.lun0... ** Unrecognized filesystem type ** Scanning disk usb_mass_storage.lun0... ERROR: failure to add disk device usb_mass_storage.lun0, r = 20 Error: Cannot initialize UEFI sub-system, r = 20 EFI LOAD FAILED: continuing... BOOTP broadcast 1 DHCP client bound to address 192.168.1.191 (64 ms) *** ERROR: `serverip' not set Cannot autoload with TFTPGET missing environment variable: pxeuuid missing environment variable: bootfile Retrieving file: . . .
I should have explicitly noted up front: I'm testing and reporting on RPi4B contexts, not any RPi3 contexts. The existing subject lines's "Pi 3" reference happens to be an example but is over specific as far as the general problem goes.
(In reply to Mark Millard from comment #12) At this point it might be better to consider the older: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253983 as a duplicate if this bugzilla entry because this one identifies a broader range contexts for the type of problem. But, the only way to make this a FreeBSD problem would be to challenge the choice to be U-Boot based. Otherwise, it is an upstream problem for the U-Boot ports used by FreeBSD and not something for FreeBSD to directly fix.
I have tried to reproduce using two different powered USB hubs (standalone and built into a computer monitor). In both cases pendrives were all plugged into the hub, which was connected directly to the Pi. They all behaved identically to the USB pendrives plugged directly into the Pi (which is to say, I see exactly the same behaviour as originally reported). I'll test on a Pi 4B as well for comparison.
Testing with a RPi 4B (8GB model), I get exactly the same U-Boot failure as for the 3B. Exactly the same failure condition and error messages for both directly plugging in the pendrives and with a powered USB hub containing the pendrives.
(In reply to Roger Leigh from comment #14 and #15) That matches what I reported for my test cases (that had sufficient power). But the problem is not in any way specific to FreeBSD. For example, Fedora 34 also has it via its U-Boot. I expect that at some point someone will reclassify the bugzilla report because of that. I expect that the problem can be repeated on some non-RPi U-Boot contexts that the U-Boot has USB storage access for. But I've not tried such: I do not expect that I have access to a system that fits that description. (But there is one I might check the modern U-Boot usb access status on at some point.)
(In reply to Mark Millard from comment #16) The problem is repeatable on a Rock64 by using the USB2 ports. (U-Boot does not deal with the USB3 port.) The context is a little messy for the setup, at least vs. my normal setup. The below is for using a powered hub and 2 thumb drive storage devices. I first looked around in U-Boot, figuring out that it dealt with the USB2 ports but not USB3 port. Part of this was doing a usb reset explicitly (that noticed the drives). Thus the explicit boot command after learning that much: => boot mmc0(part 0) is current device Scanning mmc 0:1... 50580 bytes read in 5 ms (9.6 MiB/s) Card did not respond to voltage select! : -110 Scanning disk mmc@ff500000.blk... Disk mmc@ff500000.blk not ready Scanning disk mmc@ff520000.blk... ** Unrecognized filesystem type ** Scanning disk usb_mass_storage.lun0... Scanning disk usb_mass_storage.lun0... ERROR: failure to add disk device usb_mass_storage.lun0, r = 20 Error: Cannot initialize UEFI sub-system, r = 20 Found EFI removable media binary efi/boot/bootaa64.efi 1288172 bytes read in 33 ms (37.2 MiB/s) Error: Cannot initialize UEFI sub-system, r = 20 EFI LOAD FAILED: continuing... Card did not respond to voltage select! : -110 Device 0: Vendor: SanDisk Rev: 0001 Prod: Extreme Type: Removable Hard Disk Capacity: 15272.0 MB = 14.9 GB (31277232 x 512) ... is now current device Scanning usb 0:1... Error: Cannot initialize UEFI sub-system, r = 20 Speed: 1000, full duplex BOOTP broadcast 1 BOOTP broadcast 2 BOOTP broadcast 3 DHCP client bound to address 192.168.1.171 (805 ms) *** ERROR: `serverip' not set Cannot autoload with TFTPGET missing environment variable: pxeuuid missing environment variable: bootfile Retrieving file: . . . But my explorations lead to another discovery: if I use a command that avoids trying USB at all then it avoids the issue and works. An example is: run bootcmd_mmc0 There might be an explicit U-Boot command that would boot a RPi3B or RPi4B from a microsd card, despite any USB devices that are already plugged in. But, it is still not a FreeBSD source code problem to solve. It is not RPi* specific either. It is much more general than such contexts.
(In reply to Mark Millard from comment #17) I realized that I was not explicit about something and could easily be misinterpreted on the point: Use of the likes of: run bootcmd_mmc0 presumes that one has not already done a usb start or usb reset that sets up the problem. That is exactly what is being avoided. After a usb reset with the drives present, even run bootcmd_mmc0 gets the problem. Changing the environment to not attempt any usb reset or usb start at all looks to be a way of having a configuration that avoids the problem (but will not boot from USB).
(In reply to Mark Millard from comment #18) Looks like for the default U-Boot context on a RPi4B that U-Boot already has done a "usb start" before you get to the U-Boot prompt. "usb storage" and "usb part" already list the information. That may be associated with the environment having: preboot=pci enum; usb start; by default. I do not plan to explore any adjustments to the default U-Boot environments on any of the U-Boot based systems that I have access to.
With U-Boot 2021.07 my RPI 3 Model B (0xa02082) does not boot from mmc at all (no external usb storage devices connected). I've also tested the following snapshots: FreeBSD-13.0-STABLE-arm64-aarch64-RPI-20210722-df674da44ef-246411 FreeBSD-14.0-CURRENT-arm64-aarch64-RPI-20210722-27ea55fc655-248140 I have to apply the following patch to u-boot: --- include/configs/rpi.h.orig 2021-07-18 15:37:55.743031000 +0200 +++ include/configs/rpi.h 2021-07-18 15:38:51.159286000 +0200 @@ -173,7 +173,8 @@ #if CONFIG_IS_ENABLED(CMD_MMC) #define BOOT_TARGET_MMC(func) \ func(MMC, mmc, 0) \ - func(MMC, mmc, 1) + func(MMC, mmc, 1) \ + func(MMC, mmc, 2) #else
(In reply to Herbert J. Skuhra from comment #20) If your context does not involve having multiple USB storage devices present during boot, then your context is not what this bugzilla is intended to cover and what might fix the multiple-USB issue would be unlikely to fix your issue. Also, the issue that this bugzilla is about applies to 2020.10 of U-Boot and the like and is not specific to more recent versions. Please avoid adding comments about different problems than such a pre-existing bugzilla is about. If needed, create your own bugzilla submittal for your issue. Your patch will not fix the issue that was reported by the creation of this bugzilla submittal. (It may fix some other issue.)
Whatever... Just ignore or delete my comment(s) ... if the latter is not possible replace Bugzilla with a platform that allows you delete/censor comments... and creating a new PR is often a total waste of time...
Thanks Mark Millard for pointing preboot var. I run into the exact same problem on openbsd installation. These steps worked for me: - Unplug USBs - (skip if no changes have been made to u-boot environment) boot to u-boot and restore env defaults, env default -a saveenv - Reboot into u-boot again and clear preboot env: setenv preboot saveenv - boot to os with no USB memsticks inserted - insert the USBs and reboot into os again