Bug 249045 - Loader still gets confused with a CD-ROM and prevents booing from fixed disk.
Summary: Loader still gets confused with a CD-ROM and prevents booing from fixed disk.
Status: Closed Feedback Timeout
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.1-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-09-01 02:29 UTC by rallenh
Modified: 2021-02-23 06:45 UTC (History)
2 users (show)

See Also:

Attachments
No 12.1-RELEASE CD boot from RAID6 (2.32 KB, text/plain)
2020-09-12 03:16 UTC, rallenh 		no flags Details
12.1-RELEASE CD boot (1.32 KB, text/plain)
2020-09-12 03:17 UTC, rallenh 		no flags Details
12-STABLE boot from RAID6 (2.10 KB, text/plain)
2020-09-12 06:12 UTC, rallenh 		no flags Details
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description rallenh 2020-09-01 02:29:23 UTC 
Using the 12.1-RELEASE ISO, I am able to install FreeBSD on my UEFI Dell T610 w/24TB (8x3TB SAS) RAID6 on a PERC H700.

I choose to use the entire disk and GPT. The install goes fine. Then a reboot and I get this:

>>  FreeBSD EFI boot block
    Loader path: /boot/loader.efi

    Initializing modules: ZFS UFS
    Load Path: \EFI\BOOT\BOOTX64.EFI
    Load Device: Acpi(PNP0A08,0x0)/Pci(0xA,0x0)/Pci(0x0,0x0)/Ctrl(0x0)/Scsi(0x0,0x0)/HD(1,GPT,F7080726-EBDE-11EA-90E6-002219BF8EC3,0x28,0x64000)
    BootCurrent: 0001
    BootOrder: 0000 0002 0001[*]
    Probing 5 block devices...*.. done
      ZFS found no pools
      UFS found 1 partition
Consoles: EFI console
FreeBSD/amd64 EFI loader, Revision 1.1

    Command line arguments: loader.efi
    EFI version: 2.10
    EFI Firmware: Dell Inc. (rev 6.1537)
    Console: efi (0x20000000)
    Load Path: HD(2,GPT,F7090ABE-EBDE-11EA-90E6-002219BF8Ec3,0x64028,0x7FFF9C000)
    
    Load Device: Acpi(PNP0A08,0X0)/Pci(0XA,0X0)/Pci(0x0,0x0)/Ctrl(0x0)/Scsi(0x0,0x0)/HD(2,GPT,F7090ABE-EBDE-11EA-90E6-002219BF8EC3,0X64028,0X7FFF9C000)
    BootCurrent: 0001
    BootOrder: 0000 0002 0001[*]
    BootInfo Path: Acpi(PNP0A08,0x0)/Pci(0xA,0x0)/Pci(0x0,0x0)/Ctrl(0x0)/Scsi(0x0,0x0)/HD(1,GPT,F7080726-EBDE-11EA-90E6-002219BF8EC3,0x28,0x64000)
Ignoring Boot0001: Only one DP found
Trying ESP: Acpi(PNP0A08,0X0)/pCI(0XA,0X0)/Pci(0x0,0x0)/Ctrl(0x0)/Scsi(0x0,0x0)/HD(2,GPT,F7090ABE-EBDE-11EA-90E6-002219BF8EC3,0X64028,0X7FFF9C000)
Setting currdev to disk0p2:
Loading /boot/defaults/loader.conf
Loading /boot/device.hints
Loading /boot/loader.conf
Loading /boot/loader.conf.local
Startup error in /boot/lua/loader.lua:
LUA ERROR: /boot/lua/menu.lua:37: module 'drawer' not found:
    no field package.preload['drawer']
    no file '/boot/lua/drawer.lua'
    no file '/boot/lua/5.3/lib/drawer.so'
    no file '/boot/lua/5.3/lib/loadall.so'
    no file './drawer.so'.

can't load 'kernel'

Type '?' for a list of commands. 'help' for more detailed help.
OK

I then re-boot off the CD-ROM and load a shell. I can mount the slice and see the /boot/lua/drawer.lua file is present, but not the /boot/lua/5.3/lib/drawer.so, /boot/lua/5.3/lib/loadall.so or ./drawer.so. I have checked my other FreeBSD 12-STABLE server (BIOS boot) and it doesn't have those files either.

The only way I have been able to get a reboot is to restrict the install to use less than the entire disk, say 1TB. Then reboot after install works as expected.
Comment 1 rallenh 2020-09-09 04:41:30 UTC 
After a fresh install and during the loader menu, I hit "3" and:

OK lsdev
cd devices:
    cd0:        0 blocks (no media)
disk devices:
    disk0:        35156656128 x 512 blocks
      disk0p1:  EFI
      disk0p2:  FreeBSD UFS
      disk0p3:  FreeBSD swap
net devices:
    net0:
    net1:
OK ls disk0p2:
open 'disk0p2:/' failed: no such file or directory
OK


I then reboot off the CD and at the loader menu hit "3" again:

OK ls disk0p2:
disk0p2:/
 d  .snap
 d  dev
 d  user
     COPYRIGHT
 d  boot
{...}
--more--  <space> page down <enter> line down <q> quit


I then set:

OK set currdev=disk0p2:
OK autoboot

Then every boots up as expected.
Comment 2 rallenh 2020-09-10 03:51:10 UTC 
I physically disconnected and disable the SATA port that my CD-ROM was connected to. This permitted reboot-after-install and all other subsequent boots/reboots.

I tested this with 12.1-RELEASE and make world/kernel for 12-STABLE.
Comment 3 rallenh 2020-09-10 03:52:58 UTC 
The fixes for the loader in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=234031 don't cover all/most configurations.
Comment 4 Toomas Soome freebsd_committer freebsd_triage 2020-09-10 09:35:58 UTC 
(In reply to rallenh from comment #1)

My first idea is that on this system the device handles and corresponding device paths are not properly sorted; The problem with UEFI is, we only can access devices by handles, and we only do get list of handles. From this list, we are attempting to identify devices like disks and partitions, and if this identification will go wrong and we assign wrong handle to device, we obviously can not access the data... 

Could you post output from: lsdev -v

I guess we would need to build loader.efi with debug code to be able to extract information about device paths and to see what goes wrong there. The confusing part is the fact that it does not go wrong while booting from CD, but both cases are using the same loader.efi.
Comment 5 rallenh 2020-09-11 03:54:12 UTC 
Hi Toomas,

Here's my lsdev -v (less CD-ROM):
OK lsdev -v
disk devices:
  Acpi(PNP0A08,0x0)/Pci(0xA,0x0)/Pci(0x0,0x0)/Ctrl(0x0)/Scsi(0x0,0x0)
    disk0:    35156656128 X 512 blocks
      disk0p1: EFI               200MB
      disk0p2: FreeBSD UFS        16TB
      disk0p3: FreeBSD swap       63GB
http: (unknown)
net devices:
    net0:    Acpi(PNP0A08,0x0)/Pci(0x1,0x0)/Pci(0x0,0x0)/MAC(002219BF8EC3,0x0)
    net1:    Acpi(PNP0A08,0x0)/Pci(0x1,0x0)/Pci(0x0,0x1)/MAC(002219BF8EC5,0x0)
OK

I have read boot(8) and I can't see how I can "fix" this with a parameter/setting.

Is there a way for me to remove or fix the sorting such that FreeBSD boots?

What's odd is that if I only use two (of the 8) 3TB disks in RAID1, it works. I am lost as to understanding why a 24TB UFS GPT is causing so much trouble. I also have Chelsio T5 card, which can also be bootable but I am pretty sure I have the BIOS disabled. I think those netX: cards are the Broadcom's on the Dell systemboard. They are also disabled, so I don't know why they are showing up...

Also, I updated to 12-STABLE and now it's back to not booting.

Any suggestions to get FreeBSD booting are much appreciated.
Comment 6 Toomas Soome freebsd_committer freebsd_triage 2020-09-11 08:05:47 UTC 
(In reply to rallenh from comment #5)
If you do not mind, can you run lsdev -v from not working setup - with all devices connected?

You can identify network ports via MAC address, but those are not our concern.

once I have the basic information about device paths, I can try to build some test binary, is it ok if I'll mail it to you directly? I also have to warn, it probably will take some number of attempts to try to identif what is really going on there.
Comment 7 rallenh 2020-09-12 03:13:35 UTC 
Hi Toomas,

I re-connected the CD-ROM and re-enabled all the devices in the BIOS I had disabled. The NIC BIOSes are off but the NICs are enabled.

Here's the lsdev -v from an F11 UEFI boot menu choosing the CD-ROM:

OK lsdev -v
cd devices:
  Acpi(PNP0A08,0x0)/Pci(0x1F,0x2)/Ata(0x0)
    cd0:    444340 X 2048 blocks (removable)
      cd0: ISO9660         867MB
disk devices:
  Acpi(PNP0A08,0x0)/Pci(0xA,0x0)/Pci(0x0,0x0)/Ctrl(0x0)/Scsi(0x0,0x0)
    disk0:    35156656128 X 512 blocks
      disk0p1: EFI         200MB
      disk0p2: FreeBSD UFS          16TB
      disk0p3: FreeBSD swap          63GB
net devices:
    net0:    Acpi(PNP0A08,0x0)/Pci(0x1,0x0)/Pci(0x0,0x0)/MAC(002219BF8EC3,0x0)
    net1:    Acpi(PNP0A08,0x0)/Pci(0x1,0x0)/Pci(0x0,0x1)/MAC(002219BF8EC5,0x0)


Here's the lsdev -v from the "FreeBSD" UEFI boot menu I added via the Dell UEFI manager and without the 12.1-RELEASE CD in the CD-ROM:
OK lsdev -v
cd devices:
  Acpi(PNP0A08,0x0)/Pci(0x1F,0x2)/Ata(0x0)
    cd0:    0 blocks (no media)
disk devices:
  Acpi(PNP0A08,0x0)/Pci(0xA,0x0)/Pci(0x0,0x0)/Ctrl(0x0)/Scsi(0x0,0x0)
    disk0:    35156656128 X 512 blocks
      disk0p1: EFI         200MB
      disk0p2: FreeBSD UFS          16TB
      disk0p3: FreeBSD swap          63GB
net devices:
    net0:    Acpi(PNP0A08,0x0)/Pci(0x1,0x0)/Pci(0x0,0x0)/MAC(002219BF8EC3,0x0)
    net1:    Acpi(PNP0A08,0x0)/Pci(0x1,0x0)/Pci(0x0,0x1)/MAC(002219BF8EC5,0x0)

Sure, I don't mind using e-mail for the loader.efi tests. I can build from source as well.

Thank you for offering to help me. This was supposed to be an easy peasy (piece of cake) upgrade.
Comment 8 rallenh 2020-09-12 03:16:58 UTC 
Created attachment 217899 [details]
No 12.1-RELEASE CD boot from RAID6

with loader.efi output from RAID6 disk
Comment 9 rallenh 2020-09-12 03:17:51 UTC 
Created attachment 217900 [details]
12.1-RELEASE CD boot

with loader output
Comment 10 rallenh 2020-09-12 06:12:33 UTC 
Created attachment 217904 [details]
12-STABLE boot from RAID6

with loader.efi output, 12-STABLE r365651
Comment 11 rallenh 2020-09-12 06:19:09 UTC 
12.2-STABLE FreeBSD 12.2-STABLE r365651 GENERIC  amd64:

OK lsdev -v
cd devices:
  Acpi(PNP0A08,0x0)/Pci(0x1F,0x2)/Ata(0x0)
    cd0:    0 blocks (no media)
disk devices:
  Acpi(PNP0A08,0x0)/Pci(0xA,0x0)/Pci(0x0,0x0)/Ctrl(0x0)/Scsi(0x0,0x0)
    disk0:    35156656128 X 512 blocks
      disk0p1: EFI               200MB
      disk0p2: FreeBSD UFS        16TB
      disk0p3: FreeBSD swap       63GB
http: (unknown)
net devices:
    net0:    Acpi(PNP0A08,0x0)/Pci(0x1,0x0)/Pci(0x0,0x0)/MAC(002219BF8EC3,0x0)
    net1:    Acpi(PNP0A08,0x0)/Pci(0x1,0x0)/Pci(0x0,0x1)/MAC(002219BF8EC5,0x0)


12-STABLE makes it past 3rd stage and gets to the menu. With 12.1-RELEASE, it wasn't making it past loader.efi, crashing on loading the loader configuration files.
Comment 12 rallenh 2020-09-14 06:23:16 UTC 
I wasn't updating the EFI partition after rebuilding and installing. I updated my EFI boot with:
sudo gpart bootcode -p /boot/boot1.efifat -i 1 mfid0

It looks like EFI_DEBUG is being turned on? Here's an output from 12-STABLE build and updated boot block:
>> FreeBSD EFI boot block
   Loader path: /boot/loader.efi

   Initializing modules: ZFS UFS
   Load Path: \EFI\BOOT\BOOTX64.EFI
   Load Device: Acpi(PNP0A08,0x0)/Pci(0xA,0x0)/Pci(0x0,0x0)/Ctrl(0x0)/Scsi(0x0,0x0)/HD(1,GPT,24DF125F-F561-11EA-8E3D-002219BF8EC3,0x28,0x64000)
   BootCurrent: 0000
   BootOrder: 0000[*] 0003
   Probing 8 block devices...not supported
not supported
better
not supported
not supported
not supported
not supported
not supported
 done
    ZFS found no pools
    UFS found 1 partition
    command args: -Dh

Consoles: EFI console
    Reading loader env vars from /efi/freebsd/loader.env
Setting currdev to disk0p2:
FreeBSD/amd64 EFI loader, Revision 1.1

   Command line arguments: loader.efi -Dh
   EFI version: 2.10
   EFI Firmware: Dell Inc. (rev 6.1537)
   Console: efi (0x20001000)
   Load Path: HD(2,GPT,39C9EC31-F561-11EA-8E3D-002219BF8EC3,0x64028,0x827800000)

   Load Device: Acpi(PNP0A08,0x0)/Pci(0xA,0x0)/Pci(0x0,0x0)/Ctrl(0x0)/Scsi(0x0,0x0)/HD(2,GPT,39C9EC31-F561-11EA-8E3D-002219BF8EC3,0x64028,0x827800000)
   BootCurrent: 0000
   BootOrder: 0000[*] 0003
   BootInfo Path: Acpi(PNP0A08,0x0)/Pci(0xA,0x0)/Pci(0x0,0x0)/Ctrl(0x0)/Scsi(0x0,0x0)/HD(1,GPT,24DF125F-F561-11EA-8E3D-002219BF8EC3,0x28,0x64000)
Ignoring Boot0000: Only one DP found
Trying ESP: Acpi(PNP0A08,0x0)/Pci(0xA,0x0)/Pci(0x0,0x0)/Ctrl(0x0)/Scsi(0x0,0x0)/HD(2,GPT,39C9EC31-F561-11EA-8E3D-002219BF8EC3,0x64028,0x827800000)
Setting currdev to disk0p2:
Loading /boot/defaults/loader.conf
Loading /boot/defaults/loader.conf
Loading /boot/device.hints
Loading /boot/loader.conf
Loading /boot/loader.conf.local
Loading kernel...

I am now able to boot on 12-STABLE with these two settings my /boot/loader.conf:
beastie_disable="YES"
vfs.root.mountfrom="ufs:/dev/mfid0p2"

I can tell that something still isn't right even though I can boot to multi-user with those settings. I get this after loader.efi (I think) hand-off:

/boot/kernel/kernel text=0x16bc864 data=0x140 data=0x75fe80 syms=[0x8+0x17e098+0x8+0x19bdd7]
Loading configured modules...
/etc/hostid size=0x25
can't find '/boot/entropy'

Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [/boot/kernel/kernel]...
Start @ 0xffffffff80373000 ...
EFI framebuffer information:
addr, size     0xd7800000, 0x800000
dimensions     640 x 480
stride         640
masks          0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000


There is a '/boot/entropy' in my /boot:
117 1566758595 -rw------- 1 root wheel 3128999143 4096 "Sep 12 21:48:11 2020" "Sep 14 01:57:16 2020" "Sep 14 01:57:16 2020" "Sep 12 21:48:11 2020" 32768 8 0x1 /boot/entropy

Sometimes I will get a similar error for /etc/hostid. That file exists as well:
117 1221175408 -rw-r--r-- 1 root wheel 2438831983 37 "Sep 14 01:57:14 2020" "Sep 12 21:51:56 2020" "Sep 12 21:51:56 2020" "Sep 12 21:51:56 2020" 32768 8 0 /etc/hostid


I have removed the EFI boot menu I manually added. I also can't boot into the Beastie Menu but at least I can boot without alot of ceremony.