Bug 256441 - U-Boot build for Raspberry Pi 3B and 4B (arm64) does not boot from MicroSD card slot when >1 USB storage devices are present
Summary: U-Boot build for Raspberry Pi 3B and 4B (arm64) does not boot from MicroSD ca...
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: arm (show other bugs)
Version: 13.0-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-arm (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-06-06 14:50 UTC by Roger Leigh
Modified: 2021-06-08 01:15 UTC (History)
4 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Roger Leigh 2021-06-06 14:50:43 UTC
Using 13.0-RELEASE image on a 16GB SD card, written using the RPi imager tool.

With 0 or 1 USB storage devices inserted, the system boots fine.  With 2-4 USB storage devices inserted, the system will not boot.  Looks like U-Boot is getting stuck in this situation and not able to read the second stage bootloader from the SD card mmc0 storage.  May be related to #255080

I have transcribed the console output below (may have some minor typos).

With 1 USB storage device inserted:

```
Net:  No ethernet found
starting USB...
Bus usb@7e990000: USB DWC2
scanning bus usb@7e980000 for devices... 5 USB Device(s) found
       scanning usb for storage devices... 1 Storage Device(s) found
Hit any key to stop autoboot:    0
[boot continues successfully]
```

With 2 USB storage device inserted:

```
Net:  No ethernet found
starting USB...
Bus usb@7e990000: USB DWC2
scanning bus usb@7e980000 for devices... 6 USB Device(s) found
       scanning usb for storage devices... 2 Storage Device(s) found
Hit any key to stop autoboot:    0
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found EFI removable media binary efi/boot/bootaa64.efi
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Scanning disk mmc@7e300000.blk...
** Unrecognized filesystem type **
Scanning disk usb_mass_storage.lun0...
** Unrecognized filesystem type **
Scanning disk usb_mass_storage.lun0...
ERROR: Failure to add disk device usb_mass_storage.lun0, r=20
Error: Cannot initialize UEFI sub-system, r=20
125912 bytes read in 125 ms (9.6 MiB/s)
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Error: Cannot initialize UEFI sub-system, r=20
EFI LOAD FAILED: continuing...
MMC Device 1 not found
no mmc device at slot 1

Device 0: Vendor: Generic  Rev: 2.00 Prod: Flash Disk
            Type: Removable Hard Disk
            Capacity: 7680.0 MB = 7.5 GB (15728640 x 512)
... is now current device
** Unrecognized filesystem type **
Waiting for Ethernet connection... done.
BOOTP broadcast 1
...
[boot never succeeds]
```
Comment 1 Mark Millard 2021-06-06 17:11:45 UTC
(In reply to Roger Leigh from comment #0)

This is not a duplicate of 255080: 255080 does not involve the
quoted message at all.

This looks to be a duplicate of:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253983

which also involves the error message text:

ERROR: Failure to add disk device usb_mass_storage.lun0 . . .

The same error is observable on other OS's that use U-Boot,
such as Fedora 33 attempting to boot a RPi in a certain type
of context. So it is not a FreeBSD issue, but a U-Boot issue,
for that type of context.

For 253983 the context was a single USB device with multiple
storage Logic Units. A multimedia reader with multiple media
present at the same time was sufficient to show the problem.
But the original report was not for a multi-media reader but
some other type of device with multiple storage LUNs.

If you have a context where no individual USB device has more
than one storage LUN involved, that would be new information.
But you do not provide a description of the device(s) involved,
the LUNs involved, what ports the devices are plugged into,
what partitions have what kinds of file systems with what
kinds of content, etc.

Still, based on the error message, I expect that this is U-Boot
mishandling having multiple populated storage LUNs and is not
something FreeBSD can fix. The proper place for submittal would
then be upstream-U-Boot.
Comment 2 Roger Leigh 2021-06-06 20:03:35 UTC
Hi Mark,

This is a representative sample of the USB pendrives used.  All are 9.5GB identical parts.

Geom name: da0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 15728599
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: da0p1
   Mediasize: 8053022720 (7.5G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r1w1e2
   efimedia: HD(1,GPT,82a1cdfa-9901-11eb-b20f-b827ebda9d55,0x28,0xefffb0)
   rawuuid: 82a1cdfa-9901-11eb-b20f-b827ebda9d55
   rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b
   label: gitlab
   length: 8053022720
   offset: 20480
   type: freebsd-ufs
   index: 1
   end: 15728599
   start: 40
Consumers:
1. Name: da0
   Mediasize: 8053063680 (7.5G)
   Sectorsize: 512
   Mode: r1w1e3


I have tried with:

a) no partition table
b) GPT partition table (no entries)
c) GPT partition table (single entry as above, ufs or zfs partition type)
d) geom_stripe with (a) and (c)

In all cases:

* zero or one USB pendrives boots
* two or three USB pendrives fails
Comment 3 Mark Millard 2021-06-06 20:53:53 UTC
(In reply to Roger Leigh from comment #2)

Well, I just tried a USB based boot for which 2
USB storage devices were plugged into RPi
USB ports but no microsd card in use. It got:

. . .
U-Boot 2021.04 (Apr 08 2021 - 10:46:22 +0000)

DRAM:  7.9 GiB
RPI 4 Model B (0xd03114)
. . .
Bus xhci_pci: Register 5000420 NbrPorts 5
Starting the controller
USB XHCI 1.00
scanning bus xhci_pci for devices... Device NOT ready
   Request Sense returned 02 3A 00
Device NOT ready
   Request Sense returned 02 3A 00
Device NOT ready
   Request Sense returned 02 3A 00
5 USB Device(s) found
       scanning usb for storage devices... 2 Storage Device(s) found
Hit any key to stop autoboot:  2  1  0 
Card did not respond to voltage select! : -110

Device 0: Vendor: OWC      Rev: 0    Prod: Envoy Pro mini  
            Type: Hard Disk
            Capacity: 228936.5 MB = 223.5 GB (468862128 x 512)
... is now current device
Scanning usb 0:1...
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Scanning disk mmc@7e300000.blk...
Disk mmc@7e300000.blk not ready
Card did not respond to voltage select! : -110
Scanning disk emmc2@7e340000.blk...
Disk emmc2@7e340000.blk not ready
Scanning disk usb_mass_storage.lun0...
** Unrecognized filesystem type **
** Unrecognized filesystem type **
** Unrecognized filesystem type **
Scanning disk usb_mass_storage.lun1...
Found 7 disks
** Unable to read file ubootefi.var **
Failed to load EFI variables
BootOrder not defined
EFI boot manager: Cannot load any image
Found EFI removable media binary efi/boot/bootaa64.efi
. . .

and it completed booting just fine.

So it looks like I'd need to better approximate your
boot context in order to see the problem. I'll see
about using a 13.0-RELEASE microsd card boot media
with the 2 devices still plugged in.

Another possibility is that swapping the ports usage
might make a difference in my context.
Comment 4 Mark Millard 2021-06-06 21:04:59 UTC
(In reply to Mark Millard from comment #3)

Booting a 13.0-RELEASE microsd card had no problem
with the 2 USB storage media already plugged into
the RPi ports at power up.

But it turns out that the media that I already had
around had the same u-boot.bin vintage on it as
my normal boot. It might not be the same as on a
from-scratch 13.0-RELEASE media would have.

What does your U-Boot report compared to mine:

U-Boot 2021.04 (Apr 08 2021 - 10:46:22 +0000)

?
Comment 5 Roger Leigh 2021-06-06 21:20:25 UTC
My U-boot version:

U-Boot> version
U-Boot> 2020.10 (Apr 09 2021 - 03:55:54 +0000)

aarch64-none-elf-gcc (FreeBSD Ports Collection for aarch64noneelf) 8.4.0
GNU ld (GNU Binutils) 2.33.1

This is what was provided on the 13.0-RELEASE image.
Comment 6 Mark Millard 2021-06-06 21:25:26 UTC
(In reply to Mark Millard from comment #3)

I've repeated the problem with a powered hub involved:

Powerd hub connected to a RPi USB3 port and
2 storage devices plugged into the powered
hub, both USB3 SSDs.

(Previously I had one USB3 SSD and one multi-media
reader with only one media in it and I was direct
connecting those to RPi USB ports. I'll try direct
connecting the two USB3 SSDs later.)

scanning bus xhci_pci for devices... Device not responding to set address.

      USB device not accepting new address (error=80000000)
6 USB Device(s) found
       scanning usb for storage devices... 2 Storage Device(s) found
Hit any key to stop autoboot:  2  1  0 
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
7[r[999;999H[6n8Scanning disk mmc@7e300000.blk...
Disk mmc@7e300000.blk not ready
Scanning disk emmc2@7e340000.blk...
** Unrecognized filesystem type **
Scanning disk usb_mass_storage.lun0...
** Unrecognized filesystem type **
** Unrecognized filesystem type **
** Unrecognized filesystem type **
Scanning disk usb_mass_storage.lun0...
ERROR: failure to add disk device usb_mass_storage.lun0, r = 20
Error: Cannot initialize UEFI sub-system, r = 20
Found EFI removable media binary efi/boot/bootaa64.efi
1259212 bytes read in 72 ms (16.7 MiB/s)
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Error: Cannot initialize UEFI sub-system, r = 20
EFI LOAD FAILED: continuing...

Device 0: Vendor: OWC      Rev: 0    Prod: Envoy Pro mini  
            Type: Hard Disk
            Capacity: 228936.5 MB = 223.5 GB (468862128 x 512)
... is now current device
Scanning usb 0:1...
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Error: Cannot initialize UEFI sub-system, r = 20
Found EFI removable media binary efi/boot/bootaa64.efi
1289100 bytes read in 7 ms (175.6 MiB/s)
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Error: Cannot initialize UEFI sub-system, r = 20
EFI LOAD FAILED: continuing...
ethernet@7d580000 Waiting for PHY auto negotiation to complete......... TIMEOUT !
bcmgenet: PHY startup failed: -110
missing environment variable: pxeuuid
missing environment variable: bootfile
Retrieving file:  . . .
Comment 7 Mark Millard 2021-06-06 21:46:50 UTC
(In reply to Mark Millard from comment #6)

[It appears that the RPi4B is marginal for powering
both USB3 SSDs and the other two USB devices I normally
have plugged in. So I've unplugged the USB keyboard
but still have the EtherNet dongle on a USB2 port.]

It has booted with the two USB3 SSDs in the USB3 ports
that previously were on the powered hub (so previously
using only one USB3 port with storage devices on the
RPi4B).

In other words, for what I've done to get the type of
problem, the powered-hub with devices plugged in is
analogus to the previously identified case of a single
device with multiple storage LUNs populated. Absent
such an anlogy, I've not gotten the problem so far.

What I do not know is if your context is also analogus
or not.

Also: My context suggests that an updated U-Boot is not
going to help at this point, at least for an "analogous"
way of setting up the problem.
Comment 8 Mark Millard 2021-06-06 22:01:01 UTC
(In reply to Mark Millard from comment #7)

Ignore comment 6: looks like there are still power
issues and only one of the 2 USB3 SSDs was found.

I'll try without the EtherNet dongle.
Comment 9 Mark Millard 2021-06-06 22:17:23 UTC
(In reply to Mark Millard from comment #8)

Unfortunately, even just the 2 USB3 SSDs seems to
require too much power from the RPi4B.

So I found a flash drive to plug in instead of the
2nd USB3 SSD. But it turns out that the combination
works both for direct plug-in and for the same USB3
powered hub type of test.

May be 2 devices of the same "type" via one RPi USB(3?)
port is part of the context for getting the problem?
Comment 10 Mark Millard 2021-06-06 22:31:36 UTC
(In reply to Mark Millard from comment #9)

Sure enough, replacing the USB3 SSD with another flash
drive of the same type as the other one on the USB3
powered hub got back to:

USB XHCI 1.00
scanning bus xhci_pci for devices... 7 USB Device(s) found
       scanning usb for storage devices... 2 Storage Device(s) found
Hit any key to stop autoboot:  0 
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Scanning disk mmc@7e300000.blk...
Disk mmc@7e300000.blk not ready
Scanning disk emmc2@7e340000.blk...
** Unrecognized filesystem type **
Scanning disk usb_mass_storage.lun0...
Scanning disk usb_mass_storage.lun0...
ERROR: failure to add disk device usb_mass_storage.lun0, r = 20
Error: Cannot initialize UEFI sub-system, r = 20
Found EFI removable media binary efi/boot/bootaa64.efi
1259212 bytes read in 72 ms (16.7 MiB/s)
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Error: Cannot initialize UEFI sub-system, r = 20
EFI LOAD FAILED: continuing...

Device 0: Vendor: SanDisk  Rev: 0001 Prod: Extreme         
            Type: Removable Hard Disk
            Capacity: 15272.0 MB = 14.9 GB (31277232 x 512)
... is now current device
Scanning usb 0:1...
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Error: Cannot initialize UEFI sub-system, r = 20
ethernet@7d580000 Waiting for PHY auto negotiation to complete......... TIMEOUT !
bcmgenet: PHY startup failed: -110
missing environment variable: pxeuuid
missing environment variable: bootfile
Retrieving file: . . .

And it turns out that directly plugging them in
to the 2 RPi4B ports (no powered hub) also gets
that behavior:

USB XHCI 1.00
scanning bus xhci_pci for devices... 5 USB Device(s) found
       scanning usb for storage devices... 2 Storage Device(s) found
Hit any key to stop autoboot:  2  1  0 
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Scanning disk mmc@7e300000.blk...
Disk mmc@7e300000.blk not ready
Scanning disk emmc2@7e340000.blk...
** Unrecognized filesystem type **
Scanning disk usb_mass_storage.lun0...
** Unrecognized filesystem type **
Scanning disk usb_mass_storage.lun0...
ERROR: failure to add disk device usb_mass_storage.lun0, r = 20
Error: Cannot initialize UEFI sub-system, r = 20
Found EFI removable media binary efi/boot/bootaa64.efi
1259212 bytes read in 72 ms (16.7 MiB/s)
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Error: Cannot initialize UEFI sub-system, r = 20
EFI LOAD FAILED: continuing...

Device 0: Vendor: SanDisk  Rev: 0001 Prod: Extreme         
            Type: Removable Hard Disk
            Capacity: 15272.0 MB = 14.9 GB (31277232 x 512)
... is now current device
Scanning usb 0:1...
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Error: Cannot initialize UEFI sub-system, r = 20
ethernet@7d580000 Waiting for PHY auto negotiation to complete......... TIMEOUT !
bcmgenet: PHY startup failed: -110
missing environment variable: pxeuuid
missing environment variable: bootfile
Retrieving file: 

So no external configuration of multiple storage devices
accessed via one RPi4B port is required.
Comment 11 Mark Millard 2021-06-06 23:12:58 UTC
(In reply to Mark Millard from comment #10)

I have replicated the problem with Fedora 34 Server
on a RPi4B. Fedora 34 also uses U-Boot. I used the
powered hub context to attach the thumb drives
because I do not have microsd card media around
for Fedora 34, just a USB3 SSD.

So both RPi4B USB3 ports were in use, one for the
USB3 SSD boot media and one for the powered hub
with the 2 thumb drives.

U-Boot 2021.04-rc3 (Mar 13 2021 - 00:00:00 +0000)

DRAM:  7.9 GiB
RPI 4 Model B (0xd03114)
MMC:   mmcnr@7e300000: 1, emmc2@7e340000: 0
Loading Environment from FAT... Card did not respond to voltage select! : -110
In:    serial
Out:   serial
Err:   serial
Net:   eth0: ethernet@7d580000
PCIe BRCM: link up, 5.0 Gbps x1 (SSC)
starting USB...
Bus xhci_pci: Register 5000420 NbrPorts 5
Starting the controller
USB XHCI 1.00
scanning bus xhci_pci for devices... 7 USB Device(s) found
       scanning usb for storage devices... 3 Storage Device(s) found
Hit any key to stop autoboot:  0 
Card did not respond to voltage select! : -110
Card did not respond to voltage select! : -110

Device 0: Vendor: OWC      Rev: 0    Prod: Envoy Pro mini  
            Type: Hard Disk
            Capacity: 228936.5 MB = 223.5 GB (468862128 x 512)
... is now current device
Scanning usb 0:1...
Found EFI removable media binary efi/fedora/grubaa64.efi
** Unrecognized filesystem type **
** Unrecognized filesystem type **
** Unrecognized filesystem type **
** Unrecognized filesystem type **
** Unrecognized filesystem type **
** Unrecognized filesystem type **
2692592 bytes read in 11 ms (233.4 MiB/s)
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Card did not respond to voltage select! : -110
Scanning disk mmcnr@7e300000.blk...
Disk mmcnr@7e300000.blk not ready
Card did not respond to voltage select! : -110
Scanning disk emmc2@7e340000.blk...
Disk emmc2@7e340000.blk not ready
Scanning disk usb_mass_storage.lun0...
** Unrecognized filesystem type **
** Unrecognized filesystem type **
Scanning disk usb_mass_storage.lun0...
** Unrecognized filesystem type **
Scanning disk usb_mass_storage.lun0...
ERROR: failure to add disk device usb_mass_storage.lun0, r = 20
Error: Cannot initialize UEFI sub-system, r = 20
EFI LOAD FAILED: continuing...
BOOTP broadcast 1
DHCP client bound to address 192.168.1.191 (64 ms)
*** ERROR: `serverip' not set
Cannot autoload with TFTPGET
missing environment variable: pxeuuid
missing environment variable: bootfile
Retrieving file: . . .
Comment 12 Mark Millard 2021-06-06 23:19:41 UTC
I should have explicitly noted up front: I'm
testing and reporting on RPi4B contexts, not
any RPi3 contexts.

The existing subject lines's "Pi 3" reference
happens to be an example but is over specific
as far as the general problem goes.
Comment 13 Mark Millard 2021-06-06 23:46:16 UTC
(In reply to Mark Millard from comment #12)

At this point it might be better to consider the older:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253983

as a duplicate if this bugzilla entry because this one
identifies a broader range contexts for the type of
problem.

But, the only way to make this a FreeBSD problem would
be to challenge the choice to be U-Boot based. Otherwise,
it is an upstream problem for the U-Boot ports used by
FreeBSD and not something for FreeBSD to directly fix.
Comment 14 Roger Leigh 2021-06-07 21:54:16 UTC
I have tried to reproduce using two different powered USB hubs (standalone and built into a computer monitor).  In both cases pendrives were all plugged into the hub, which was connected directly to the Pi.  They all behaved identically to the USB pendrives plugged directly into the Pi (which is to say, I see exactly the same behaviour as originally reported).

I'll test on a Pi 4B as well for comparison.
Comment 15 Roger Leigh 2021-06-07 22:12:58 UTC
Testing with a RPi 4B (8GB model), I get exactly the same U-Boot failure as for the 3B.  Exactly the same failure condition and error messages for both directly plugging in the pendrives and with a powered USB hub containing the pendrives.
Comment 16 Mark Millard 2021-06-07 23:00:33 UTC
(In reply to Roger Leigh from comment #14 and #15)

That  matches what I reported for my test cases
(that had sufficient power).

But the problem is not in any way specific to
FreeBSD. For example, Fedora 34 also has it
via its U-Boot.

I expect that at some point someone will reclassify
the bugzilla report because of that.

I expect that the problem can be repeated on some
non-RPi U-Boot contexts that the U-Boot has USB
storage access for. But I've not tried such: I do
not expect that I have access to a system that fits
that description. (But there is one I might check
the modern U-Boot usb access status on at some
point.)
Comment 17 Mark Millard 2021-06-08 00:46:10 UTC
(In reply to Mark Millard from comment #16)

The problem is repeatable on a Rock64 by using the
USB2 ports. (U-Boot does not deal with the USB3 port.)
The context is a little messy for the setup, at least
vs. my normal setup.

The below is for using a powered hub and 2 thumb
drive storage devices. I first looked around in
U-Boot, figuring out that it dealt with the USB2
ports but not USB3 port. Part of this was doing a
usb reset explicitly (that noticed the drives).
Thus the explicit boot command after learning that
much:

=> boot
mmc0(part 0) is current device
Scanning mmc 0:1...
50580 bytes read in 5 ms (9.6 MiB/s)
Card did not respond to voltage select! : -110
Scanning disk mmc@ff500000.blk...
Disk mmc@ff500000.blk not ready
Scanning disk mmc@ff520000.blk...
** Unrecognized filesystem type **
Scanning disk usb_mass_storage.lun0...
Scanning disk usb_mass_storage.lun0...
ERROR: failure to add disk device usb_mass_storage.lun0, r = 20
Error: Cannot initialize UEFI sub-system, r = 20
Found EFI removable media binary efi/boot/bootaa64.efi
1288172 bytes read in 33 ms (37.2 MiB/s)
Error: Cannot initialize UEFI sub-system, r = 20
EFI LOAD FAILED: continuing...
Card did not respond to voltage select! : -110

Device 0: Vendor: SanDisk  Rev: 0001 Prod: Extreme         
            Type: Removable Hard Disk
            Capacity: 15272.0 MB = 14.9 GB (31277232 x 512)
... is now current device
Scanning usb 0:1...
Error: Cannot initialize UEFI sub-system, r = 20
Speed: 1000, full duplex
BOOTP broadcast 1
BOOTP broadcast 2
BOOTP broadcast 3
DHCP client bound to address 192.168.1.171 (805 ms)
*** ERROR: `serverip' not set
Cannot autoload with TFTPGET
missing environment variable: pxeuuid
missing environment variable: bootfile
Retrieving file: . . .



But my explorations lead to another discovery: if
I use a command that avoids trying USB at all
then it avoids the issue and works. An example
is:

run bootcmd_mmc0

There might be an explicit U-Boot command that would
boot a RPi3B or RPi4B from a microsd card, despite
any USB devices that are already plugged in.

But, it is still not a FreeBSD source code problem
to solve. It is not RPi* specific either. It is much
more general than such contexts.
Comment 18 Mark Millard 2021-06-08 00:57:11 UTC
(In reply to Mark Millard from comment #17)

I realized that I was not explicit about
something and could easily be misinterpreted
on the point:

Use of the likes of:

run bootcmd_mmc0

presumes that one has not already done a
usb start or usb reset that sets up the
problem. That is exactly what is being
avoided.

After a usb reset with the drives present,
even run bootcmd_mmc0 gets the problem.

Changing the environment to not attempt
any usb reset or usb start at all looks
to be a way of having a configuration
that avoids the problem (but will not
boot from USB).
Comment 19 Mark Millard 2021-06-08 01:15:09 UTC
(In reply to Mark Millard from comment #18)

Looks like for the default U-Boot context on a RPi4B
that U-Boot already has done a "usb start" before you
get to the U-Boot prompt. "usb storage" and "usb part"
already list the information.

That may be associated with the environment having:

preboot=pci enum; usb start;

by default.

I do not plan to explore any adjustments to the default
U-Boot environments on any of the U-Boot based systems
that I have access to.