Bug 209821

I observe the same symptoms (UEFI installer gets stuck immediately after printing framebuffer info; system is frozen, keyboard doesn't react) with hardware:

MB: Asus P8Z68-V PRO GEN3, BIOS 3802 01/15/2015 (latest BIOS)
EFI v2.31 by American Megatrends

Tested with UEFI USB stick installation FreeBSD 10.3 (FreeBSD-10.3-RELEASE-amd64-uefi-memstick.img) and PCBSD 11-Current-May (PCBSD11.0-CURRENTMAY2016-05-14-2016-x64-netinstall-USB.img), both freeze.

Non-UEFI image boots.

I confirm this situation with FreeBSD11-BETA2

Same issue on UX21E

same issue on ASUS P6P67 LE

Confirmed on Asus P8P67 PRO with Bios version 3602 x64, EC version MBECE-0018 and ME version 8.0.2.1410. The problem persisted after setting SATA mode to IDE as well as enabling legacy compatibility. 

It gets as far as printing some frame buffer info on the screen then hangs ignoring all input. BIOS boot works fine.

There is an error in comment 4
The motherboard is ASUS P8P67 LE not ASUS P6P67 LE

I have the same issue on an AMD Sabertooth 990FX Motherboard. 

The system hangs after the loading of the kernel with the EFI console information. It didn't change anything as i tried to change the GOP mode to 640x480/800x600/1024x768. 

I use currently FreeBSD 11.1 amd64. HDD is SSD and a Hardware Raid 5 3ware 9650SE-4LP DISK.
SSD contains the EFI partition. On the same SSD is space for the BSD ROOT ZFS. 

Booting from the installer in UEFI mode hangs the system after

EFI framebuffer information:
addr, size          0xc0000000, 0x10000000
dimensions          800 x 600
stride              832
masks               0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000
_

after i booted the installer in legacy mode, the kernel started. I installed the BSD manually on the SSD ZFS partiton and added the boot1.efi in the UEFI partition and added an entry in my EFI boot menu. The same procedure worked flawless in multiple VMs installed in UEFI mode. 

All other OS can run my system in UEFI mode. Windows 10, Windows 7 and Linux.


Some more information can be found on
https://www.bsdforen.de/threads/freebsd-11-1-auf-3ware-9650se-4lp-disk.34334/

I forgot, the manual installation hangs at the same step.

I unstalled FreeBSD on an SSD partiton with UFS2. As i started the freebsd kernel with grub2 instead of the boot1.efi loader, the graphic output was totally garbled. So maybe it's an issue with the EFI framebuffer handling.

With boot1.efi from 11.2-BETA2 on the EFI partiton i could boot 11.2-BETA2 and 11.1-RELEASE installation.

So i suspect the bug was fixed in the history of stand/efi/boot1/boot1.c between 11.1-RELEASE and 11.2-BETA2.

Good boot1.efi (from 11.2-BETA2 in use)
SHA256 (/tmp/efi/EFI/FreeBSD/boot1.efi) = eb2c3d4c2a5881973e22ecf7bebf9f07d4d9d12c232185d71f9caababb613650

Bad boot1.efi (from 11.1-RELEASE just on the root partiton but not used)
SHA256 (/boot/loader.efi) = f77a59198529a9b220fcc8fc9dd91c0bfbf76519240692527ebc1ca60f1817ee

I can reproduce this bug with bhyve and the FreeBSD 11.1-bootonly iso.

bhyve -AHP \
    -s 0:0,hostbridge \
    -s 1:0,lpc \
    -s 2:0,virtio-net,tap0 \
    -s 3:0,virtio-blk,/dev/zvol/data/VM/byhve/test-vm \
    -s 4:0,ahci-cd,/usr/home/georg/Downloads/FreeBSD-11.1-RELEASE-amd64-bootonly.iso \
    -m 2048M \
    -s 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600,wait \
    -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
    test-vm

Same issue on Asus Sabertooth P67 UEFI Version 3602 (Latest available)

Happens with 11.2-RELEASE but also with 12.0-ALPHA8 isos. Both images boot fine on legacy mode.

Gets stuck at:

Loading kernel...
/boot/kernel/kernel text=0x169d318 data=0x1ceac8+0x80d270 syms=[0x8+0x180c00+0x8+0x19d837]
Loading configured modules...
Could not load one or more modules!
Start @ 0xffffffff80343000 ...
EFI framebuffer information:
addr, size       0xd0000000, 0x30000000
dimensions       800 x 600
stride           832
masks            0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000

Loader variables:

autoboot_delay=NO
bootenv_autolist=YES
bootfile=kernel
comconsole_pcidev=
comconsole_port=1016
comconsole_speed=9600
console=efi
currdev=cd0:
efi-version=2.31
efi_max_resolution=1x1
hint.acpi_throttle.0.disabled=1
hint.atkbd.0.at=atkbdc
hint.atkbd.0.irq=1
hint.atkbdc.0.at=isa
hint.atkbdc.0.port=0x060
hint.atrtc.0.at=isa
hint.atrtc.0.irq=8
hint.atrtc.0.port=0x70
hint.attimer.0.at=isa
hint.attimer.0.irq=0
hint.attimer.0.port=0x40
hint.fd.0.at=fdc0
hint.fd.0.drive=0
hint.fd.1.at=fdc0
hint.fd.1.drive=1
hint.fdc.0.at=isa
hint.fdc.0.drq=2
hint.fdc.0.irq=6
hint.fdc.0.port=0x3F0
hint.p4tcc.0.disabled=1
hint.ppc.0.at=isa
hint.ppc.0.irq=7
hint.psm.0.at=atkbdc
hint.psm.0.irq=12
hint.sc.0.at=isa
hint.sc.0.flags=0x100
hint.smbios.0.mem=0xcec28e98
hint.uart.0.at=isa
hint.uart.0.flags=0x10
hint.uart.0.irq=4
hint.uart.0.port=0x3F8
hint.uart.1.at=isa
hint.uart.1.irq=3
hint.uart.1.port=0x2F8
interpret=OK
kernel=kernel
kernel_options=
loaddev=cd0:
loader_conf_files=/boot/device.hints /boot/loader.conf /boot/loader.conf.local
module_path=/boot/modules;/boot/dtb;/boot/dtb/overlays
nextboot_conf=/boot/nextboot.conf
nextboot_enable=NO
prompt=${interpret}
script.lang=lua
smbios.bios.reldate=11/01/2012
smbios.bios.vendor=American Megatrends Inc.
smbios.bios.version=3602
smbios.chassis.maker=Chassis Manufacture
smbios.chassis.serial=Chassis Serial Number
smbios.chassis.tag=Asset-1234567890
smbios.chassis.version=Chassis Version
smbios.memory.enabled=8388608
smbios.planar.location=To be filled by O.E.M.
smbios.planar.maker=ASUSTek Computer INC.
smbios.planar.product=SABERTOOTH P67
smbios.planar.serial= *omitted*
smbios.planar.tag=To be filled by O.E.M.
smbios.planar.version=Rev 1.xx
smbios.socket.enabled=1
smbios.socket.populated=1
smbios.system.family=To be filled by O.E.M.
smbios.system.maker=System manufacturer
smbios.system.product=System Product Name
smbios.system.serial=System Serial Number
smbios.system.sku=SKU
smbios.system.uuid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
smbios.system.version=System Version
smbios.version=2.6
twiddle_divisor=1
verbose_loading=NO
vfs.mountroot.timeout=10

I'm new to BSD so I still don't know my way around here but I'll be glad to provide any more information given instructions on how to.

This happens to me as well. P8P67 Pro motherboard with 3602 BIOS.

I also have this issue. Asus P8P67 rev 1.02

Same here. ASUS P6P67 Rev. 3.1, BIOS 3602, 12-RELEASE a,d64

(In reply to oz42 from comment #16)
Funny is: I dis a secure erase on my SSD and installed FreeBSD as the first system. No problems.

I see the same hung with FreeBSD 12.1 (XigmaNAS ^2.1.0.4.7091) on an older Supermicro X9SCV-Q with firmware AMI Bios 2.10.1208 from 2012.

The problem is occurs eratic: most cases the box hang at 

EFI framebuffer information:
addr, size          0xe0000000, 0x3ff0000
dimensions          800 x 600
stride              800
masks               0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000

in some rare cases the box boots.

Comment 19 Stefan Eßer 2020-02-05 20:25:52 UTC

Just a me-too, but on the latest -CURRENT:

I tried to switch from legacy boot to UEFI boot, and the EFI loader is started and lets me change options or select a kernel to boot from.

When the kernel is loaded and started, the system locks up with identical information about the EFI frame buffer as shown in comment 18 by O. Hartmann.

System hardware:

ASUS P8H67-M EVO with American Megatrends BIOS 3703 and with i7-2600K CPU (Sandy Bridge)

Comment 20 Stefan Eßer 2020-02-05 20:29:28 UTC

I have updated the affected version to -CURRENT (which as of now is 13-CURRENT), in the hope that this PR gets more visibility that way.

It seems I have the same with a HP Compaq 8200 Elite SFF PC. When disabling in the AMI Bios the EFI mode and using legacy it works fine. Also it hung on a similar message as posted earlier in this ticket:

EFI framebuffer information:
addr, size          0xc0000000, 0x10000000
dimensions          800 x 600
stride              832
masks               0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000

If more info is necessary i'm willing to provide.

I'm having a similar problem, with the slight difference that after printing the EFI framebuffer information, the system does not hang but rather reboots, causing an endless reboot loop.  This is booting FreeBSD-12.1-RELEASE-amd64-memstick.img from a USB flash drive on a HP Compaq 6300 Pro SFF PC.

Disabling the EFI boot devices in the BIOS works around the issue.

Greetings!

I do have the halting-problem.

Hardware 1:
* ASUS P8Z68-V PRO   (not a PRO/GEN3)
* AMIBIOS 080010 ALASKA Ver.36.03
* CPU: Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz (3110.22-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206a7  Family=0x6  Model=0x2a  Stepping=7
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x1fbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
* Onboard Graphics (I assume intel i915)
* tested with and without: nVidia GeForce GTX 770 in PCIe

Hardware 2:
* ASUS P8H67-M Rev.3.0
* SMBIOS v.2.6 Ver.39.01
* CPU: an i5 as well

I was not successful, but you might want to know what I have tried.

To sum it up: Booting successfully on another PC in UEFI mode I have
arrived at the speculation that it might be that it has nothing to do
with a problem related to the graphic chips etc—although switching to
VT(efifb) or similar might be the problem.

Obviously we are stuck at what ever comes after reporting the EFI
framebuffer information including the masks AND clearing the screen
followed by the to be reported lines:
...8<---
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
---<<BOOT>>---
--->8...
 

Using USB-Stick with following images:
http://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/12.1/FreeBSD-12.1-RELEASE-amd64-memstick.img
http://ftp.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/13.0/FreeBSD-13.0-CURRENT-amd64-20200409-r359731-memstick.img

One test was as follows: I did install—booting in UEFI mode—a
12.1-RELEASE onto an empty GPT disk in another PC.
That disk I did attach to my two problematic PCs and booted.
Strangely both do not think the hard disk to be bootable.


In the 12.1-RELEASE version I have tried several modifications:

* ...8<--- /boot/loader.conf on the USB-Stick
  hint.apic.0.disabled="1"
  --->8...
  Did not change the problem :-(

* ...8<--- /boot/loader.conf on the USB-Stick
  kern.vty="sc"
  # as well as    kern.vty=sc
  --->8...
  Did not change the problem :-(

* ...8<--- /boot/loader.conf on the USB-Stick
  exec="set mode 0"
  # as well as 3, 12, ...
  # I could see the effect
  --->8...
  Did not change the problem :-(

* ...8<--- /boot/loader.conf on the USB-Stick
  exec="gop set 0"
  # as well as 3, 12, ...
  # I could see the effect
  --->8...
  Did not change the problem :-(

* ...8<--- /boot/loader.conf on the USB-Stick
  efi_max_resolution="480p"
  # as well as "720p", "1080p", "800x600" 
  # I could see the effect
  --->8...
  Did not change the problem :-(

* loading graphics/drm-kmod for the i915 (5th generation as CPU i5-*)
  https://forums.freebsd.org/threads/how-to-use-the-old-or-the-new-i915kms-driver-for-intel-integrated-graphics-with-xorg.66732/
  mentions something about that topic (using port from 5th generation
  onward instead of /boot/kernel/i915kms.ko)
  I did install using BIOS boot. In that system I did build and install
  the mentioned port, than mounted the USB stick and copied
  /boot/modules/ to it. I did add a couple of lines on the USB stick:
  ...8<--- /boot/loader.conf on the USB-Stick
  verbose_loading="YES"
  newintelgraka_name="/boot/modules/i915kms.ko"
  newintelgraka_load="YES"
  --->8...
  I could observe that it did load the correct kernel module including
  the correct dependencies out of /boot/modules/.
  It did not help (I did this with my nVidia GeForce GTX 770 installed)
  Did not change the problem :-(

I too am at a loss.

Regards,
 Kalten

The symptoms are reproducable here if I boost efiloader's EFI_STAGING_SIZE to 512M or more.
Lowering to less than 512M (which was recently increased to 100M on amd64) allows normal boot.  Maybe someone with deeper UEFI knowledge can find a correlation between EFI_STAGING_SIZE in stand/efi/loader/copy.c and maybe vendor dependent implementation bugs?

-harry

Comment 25 Conrad Meyer 2020-04-28 16:10:23 UTC

Re: EFI_STAGING_SIZE, IIRC loader assumes it can get 512 MB of contiguous memory, and/or doesn't consult the memory map to check for conflicts, or something like that.  With a slightly older 11.x version of efiloader, we had some hardware that couldn't boot with 128 MB memory (OOM) and some hardware that couldn't boot with 512 MB memory (crash).  The sweet spot we found was something like 380 MB for our hardware (which has custom BIOS/firmware anyway), but that's a very limited subset of the real hardware in existence.

This problem exists !
Every new laptop/Ultrabook like my asus vivobook can't switch from efi to legacy there is no options for that, so I had to boot in efi 
The amd graphics card needs amdgpu and these two have problems combined.
I get the framebuffer info and stuck , I can't do anything else 
tried to disable syscons but that doesn't work it even made me unable to switch to any terminal or do anything at all after displaying the framebuffer messages,same as this bug reports.
I don't know if there is a solution for that.
The strange is that with FreeBSD 13 I succeed to start xorg but only two times and by pressing alt+f2 alt+f3 I don't know what is the problem

I'm having the same issue as well.

Z77X-UD5H
Version: F7 (5/11/2012)

I will try and update the bios but the exact same thing is happening.  I was having issues with my upgraded install and now am trying to reinstall.

At least for my issue, upgrading to the newest F16j version of the BIOS for my motherboard has gotten me to the next stage of the install.

Okay, I'm feeling way out of my depths here and starting to get both frustrated and annoyed at myself for not being able to figure this out better. I picked up a cheap PC the other day, an HP Pavilion p7-1147c. The bios interface is extremely limited on this unit and no option for enabling/disable legacy boot but does have support for UEFI. With UEFI and a USB installer, I can get to the installer screen and choose which type of install but the only one that seems to do anything is option #1, then from there it gets to the masks line and halts completely. 

Here's where I'm lost after reading all the comments here. where and which version do I need to download, can I get a legacy boot version of the installer. I think that is my largest missing piece. I found a server link and may have found what I need but it's so much data, so many versions and no differentiators I feel lost there. 

Any direct links or assistance on finding what I'm needing would be stellar!

I am having the same problem with my HP Z2 mini G4 (BIOS version is updated to the latest available version). 

The boot hangs after printing out the EFI framebuffer information. Booting in legacy mode and trying several boot parameters did not work either. 

It would be really great if somebody could look into this issue.

Tested with 12.2-RC3 on a HP Z210 and a HP Z220, both using latest BIOS version.

The HP Z210 halts after EFI framebuffer information output and the HP Z220 works just fine. 

Is there anything I can do to help solving this issue by comparing the EFI environments of these two computers?

I tested with my ASUS K73SD + Freebsd 12.2 (DVD install): EFI boot failed.

However, if I escape to loader prompt and do a "boot -m" or "boot -q" the installation begins.
After installation, boot -q -m still works

I am hitting the same bug:

Asus P8H67-V with Intel i5 (2nd Gen, Sandy Bridge)

- Reproduced with
  - FreeBSD 11.4, 12.0, 12.1, 12.2 and 13-CURRENT

(In reply to RichardG from comment #32)
My previous comment is wrong. Mute or quiet do not prevent hangs during boot.
I also experienced hangs during shutdowns. in that case, the messages "syncing disks vnodes remaining... etc" are'nt displayed.

Comment 35 Graham Perrin 2021-01-21 19:50:39 UTC

Reading this alongside more recent bug 251866 …

(In reply to Graham Perrin from comment #35)
I tested this, lowering EFI_STAGING_SIZE from 100 to 64 : I still got a freeze

I fixed the problem on my machine! The boot process always got stuck after it printed the EFI framebuffer information. This is just before the kernel is started.
The EFI bootloader first loads the kernel to a staging area (which is 64MB large). Then, just before the kernel is started, the bootloader copies the contents of the staging area to the address 0x200000 and jumps to the kernel entry point that is now located at the expected location.
In my case, the LoaderCode and LoaderData memory pages of the EFI bootloader happened to be located at addresses 0x2d1e000 and 0x2e01000, which were overwritten by the contents of the staging area.
I made some changes to the code to avoid this issue, and now my system boots just fine.

These bug reports also seem to be similar to this one: Bug 255073, Bug 219957

I am attaching a patch against the main git branch, please test if it fixes the problem for you too.

Created attachment 225333 [details]
Fix a boot hang on some systems

There is also another scenario when the system can hang after displaying the EFI framebuffer information. This happens when EFI_STAGING_SIZE is too small (e.g., 32MB) and there is no available memory surrounding the staging area. In that case, efi_check_space correctly returns FALSE. The functions that called it (efi_copyin, efi_readin) also return an error. The return value of efi_readin seems to be checked by all of its callers. However, the return value of efi_copyin is never checked.

In a KVM virtual machine with OVMF, FreeBSD boots with no apparent issues even if EFI_STAGING_SIZE is set to 32 and efi_check_space does not expand the allocated memory and returns errors.

On my physical machine, with EFI_STAGING_SIZE set to 32, the bootloader prints a bunch of "efi_check_space: Unable to expand staging area" messages followed by the EFI framebuffer information, and then it hangs. Some graphical corruption was also displayed on the screen.

(In reply to David Sebek from comment #37)

Hello,

On my ASUS K73SD, the patch works. 
Without the patch I never managed to do a cold boot (ie just after a power on) on UEFI. Every boot with the patch succeeded (power on, reboot).

patch applied on 13 current :
FreeBSD 13.0-RELEASE-p1 #0 releng/13.0-n244744-8023e729a52

Richard

Created attachment 225480 [details]
[PATCH] efi: loader: Fix a boot freeze on some amd64 systems

Fix an issue when the boot process could freeze shortly after
displaying EFI framebuffer information on some amd64 systems.

Instead of calling the efi_copy_finish function from amd64_tramp,
include the copy instructions in the trampoline code itself.
This avoids boot hangs and restarts in the cases when
the EFI loader code and/or data segments happen to be located
at a memory location that is overwritten during the copy process.

Also, add missing return value checks. Add a new function that
prints a warning message if there is a danger that the bootloader
may attempt to overwrite memory pages of a type that should not
be overwritten.

similar problem on an ASUS PRIME Z590M-PLUS visually, but it differs in that graphics (gop) is not seen while in UEFI mode, and loader.efi thinks everything must be serial console (boot info reports addr, size 0x0, 0x0) 

The system displays apparently no further activity past that point

I've tried the June 1st patch (though it needs some cleanup op 13.0-RELEASE-p2). It does continue the boot, but the screen is all garbled.

booting openbsd's install69.img in UEFI mode shows a working efifb though.

Booting in CSM mode, with an external video card, works

A 12.2-RELEASE install image does boot with UEFI, and VT(efifb) is available, which means there is a regression between 12.2 and 13.0 handling of UEFI. After removing the PCI gfx card the IGP is detected and works

(In reply to David Sebek from comment #41)

@David Sebek We're using a rather large mfsroot (mfsbsd) and have been seeing this issue on Vmware and bare metal OpenCompute Leopard. I built the loader.efi out of FreeBSD-current using your nice patch and this resolved the issue on VMware but not the OCP Leopard. 

The OCP Leopard previously would not even complete loading the mfs before it hung, but now we get past the mfsroot to: 

Loading kernel...
/boot/kernel/kernel text=0x16bdcc4 data=0x140 data=0x75fe80 syms=[0x8+0x17e098+0x8+0x19bdd3]
Loading configured modules...
/mfsroot size=0x57c0000
can't find '/etc/hostid'
can't find '/boot/entropy'
Start @ 0xffffffff80373000 ...
EFI framebuffer information:
addr, size     0x0, 0x0
dimensions     0 x 0
stride         0
masks          0x00000000, 0x00000000, 0x00000000, 0x00000000

(In reply to James Thomason from comment #44)

Is your monitor connected to a graphics card, or are you using a serial console? I can reproduce a similar output in a virtual machine if I remove the GPU, use serial console instead, and then on the FreeBSD bootloader screen select "5. Cons: Video" instead of "5. Cons: Serial". In that case, the system boots but does not print any further information to the console. Other than that, I have not been able to reproduce your issue yet.

(In reply to David Sebek from comment #45)

I am able to reproduce the problem when a GPU is used that doesn't support EFI GOP. I will take a look at it.

(In reply to David Sebek from comment #46)

In this case the system does not have a framebuffer at all. They are OpenCompute servers which support serial console through ipmi redirect or otherwise through a debug header only. 

We are attempting to boot the host through ipxe sanboot of a UEFI compliant iso image containing a vanilla 12.2-RELEASE mfsbsd kernel+mfs and the loader.efi from FreeBSD-current with your patch. 

If you want I could set you up with ssh access into a tesbed with this type of host and the ipxe environment.

(In reply to ruben from comment #43)

I can confirm that this seems to be a regression for 13.0-RELEASE.

I also suspect this is some kind of "special behaviour" with ASUS UEFI, as the affected system here is also an ASUS board (Pro M560M-C)

I just received a new workstation today with said ASUS Pro M560MC/CSM board and a FreeBSD 13.0-RELEASE memstick isn't able to boot, whereas the 12.2-RELEASE memstick boots and I'm able to install.

(In reply to Sebastian from comment #48)

Did you apply the patch to see if your system booted? How do you know without the proper diagnostics on this problem.

I've tried Releases 11.4, 12.x, 13.0 and the test still fails. So it can't be a regression. NB. OEM/brand is almost an irrelevant quantity as any given vendor may use firmware from a variety of different sources.

Comment 50 Konstantin Belousov 2021-06-20 04:57:10 UTC

(In reply to David Sebek from comment #41)
Please put the patch on reviews.freebsd.org (and add me to the reviewers)

(In reply to Konstantin Belousov from comment #50)
Here it is: https://reviews.freebsd.org/D30828

Comment 52 Konstantin Belousov 2021-07-09 16:11:40 UTC

Early version of the patch to avoid staging copy is available at
https://reviews.freebsd.org/D31121

You need to have a plan to restore both loader and kernel if you decided to
test and the patch causes problems for you.

Comment 53 Graham Perrin 2021-08-13 06:31:59 UTC

(In reply to Konstantin Belousov from comment #52)

> … https://reviews.freebsd.org/D31121 …

▶ <https://cgit.freebsd.org/src/commit/?id=f75caed644a5c8c342a1ea5e7a6d5251f82ed0b1>

Comment 54 Ed Maste 2021-10-04 15:11:08 UTC

This issue should be resolved by kib's commit. Anyone who was able to reproduce this, can you please test with a -CURRENT snapshot (20210930 or later) and report whether it works for you?

Comment 55 Mikael Urankar 2021-10-04 15:33:30 UTC

(In reply to Ed Maste from comment #54)
It works for me on a gigabyte motherboard (it failed to boot prior this work).
Thanks everyone involved.

This works for me... Shuttle XH61V.  This is good work. Thank you!

Comment 57 Ed Maste 2021-10-05 14:12:41 UTC

It would be great if someone with an Asus P8P67 can confirm that this now works.

It was merged to stable/13 in:

commit 1b33aa1f5f99e1270d526ffa5b652250ec80a7ef
Author: Konstantin Belousov <kib@FreeBSD.org>
Date:   Sat Jul 10 22:55:56 2021 +0300

    amd64 UEFI loader: stop copying staging area to 2M physical
    
    (cherry picked from commit f75caed644a5c8c342a1ea5e7a6d5251f82ed0b1)

I can confirm this is now working on Asus P8P67 LE systemboard.

Comment 59 Ed Maste 2021-10-05 16:59:13 UTC

(In reply to rdunkle from comment #58)
> I can confirm this is now working on Asus P8P67 LE systemboard.

Thank you!

Kostik's change is now in stable/13 and will be in FreeBSD 13.0. The patch does not apply (without significant additional work) to stable/12, and a merge there is not currently planned.

If anyone encounters a similar issue in the future please submit a new PR with details.

@kib @emaste Hey guys :) First of all thanks for your work. Ed, i highlight you because you commented on bug #209821 and bug #256722, also tagging Konstantin as author.

I saw that you merged the patches to stable/13 and aren't planning to merge them to stable/12. We currently use 12.2 and don't plan to upgrade to 13 soon, since we have a policy to always wait for at least a .2- or even better a .3-release for stability reasons.

We currently use the 12.1 efi-bootloader as a workaround to have our hardware (HPE Blades) working with 12.2 (debugged back then with @tsoome). Sadly we can't use that fix for our network-boot-images. Is there any chance to get the fixes in stable/12 and maybe even in the 12.3 release? I think there are quite a bunch of users that would be happy if this would be possible. Thanks for taking a look at it.

Comment 61 Ed Maste 2021-10-14 14:02:11 UTC

(In reply to Dani from comment #60)

It's a significant amount of work to port this to 12.x -- it would basically be a reimplementation.  Unfortunately we don't expect to have time to look at this in the near future.

I would highly recommend that affected users plan to migrate to 13.x, and ideally test workloads on stable/13 snapshots to ensure that 13.1 will meet their needs.

Comment 62 Konstantin Belousov 2021-10-14 14:24:51 UTC

There is obvious contradiction in the request to port this thing to stable/12
branch: the backport is high-risk [*] but the reason that it is asked for is
because stable/12 is perceived as more stable.  In other words, you are asking
to destabilize more stable branch.

* Reason for high-risk is that code in stable/12 diverged enough that I need
to re-read at least the pmap bootstrap code anew and identify all places that
depend on the fixed kernel physical load address from scratch.

Thanks for your answers - highly appreciated.

I can see the points in your answers. Well for us FBSD 12 isn't stable, because we can't even boot our hardware with it. So if there is no way to get this into 12, we'll see what we are going to do. Thanks anyway.

Comment 64 Ed Maste 2021-11-29 20:04:56 UTC

(In reply to Dani from comment #63)
Note that David Sebek's patch in comment #37 may be a feasible way to address this in stable/12.

(In reply to Ed Maste from comment #64)

Hey Ed, thanks for the hint. I guess you mean the patches from comment #38 or comment #41 ? Both don't apply against a clean 12.3 src, so i'm not sure how much sense that makes?

(In reply to Dani from comment #65)
I think there were two reasons why UEFI boot froze on some systems. Before the changes in 13-STABLE and 14-CURRENT, the kernel needed to be loaded to a specific physical memory location. Copying the kernel to a specific location in memory could cause problems:

1. The target location could contain the boot loader code. When the boot loader was copying the kernel to the expected address in memory, it accidentally overwrote the boot loader’s code that was responsible for the copy operation, which resulted in a freeze of the system.

2. The target location could contain hardware-reserved addresses. Copying the kernel over those memory addresses could freeze the system.

The patch from #41 fixes only the problem (1.) by not using the boot loader code that may get overwritten.

FreeBSD 14-CURRENT and 13-STABLE fix both problems (1.) and (2.) by not requiring the kernel to be copied to a specific memory address.

The patch #41 needed some minor modifications in order to be applied to FreeBSD 12. I can take a look at it and update for FreeBSD 12.

Created attachment 230798 [details]
[PATCH] [releng/12.3] efi: loader: Inline copy_finish function in amd64 trampoline

This is a simplified version of the previous patch (only includes essential changes). Rebased to FreeBSD 12.3 (releng/12.3).

Patch description:

Instead of calling the efi_copy_finish function from amd64_tramp,
include the copy instructions in the trampoline code itself.
This avoids boot hangs and restarts in the cases when
the efi_copy_finish code happens to be located at a memory
location that is overwritten during the copy process.

(In reply to Dani from comment #65)

See patch in comment #67. It is a reduced version of the patch from comment #41 rebased against FreeBSD 12.3. I tested the patch with FreeBSD 12.3, and it fixes the boot freeze problem on my old laptop.

Comment 69 Konstantin Belousov 2022-01-08 09:12:41 UTC

(In reply to David Sebek from comment #68)
It looks fine to me.  I did some minimal editings, mostly:
- added your (David) copyright to the file
- removed old trampoline code, I do not see why it is needed
- reorganized registers usage mostly to avoid confusing arg renames
- minimal style update for comments

Can anybody test this and confirm that everything works fine?

Comment 70 Konstantin Belousov 2022-01-08 09:13:27 UTC

Created attachment 230812 [details]
Edited version of the David' patch

(In reply to Konstantin Belousov from comment #69)

I tested the updated patch on my system, and it works. I made some minor modifications to the patch:
- Corrected the copyright year
- Fixed one comment
- Fixed indentation of one comment

I think the efi_copy_finish function is also no longer needed besides the fact that I referenced it in a comment in the trampoline code.

Created attachment 230823 [details]
Updated edited version of the David's patch

Comment 73 commit-hook 2022-01-09 01:29:34 UTC

A commit in branch stable/12 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=f131d68e21fc360f5c8e63377f25cf60706d9afa

commit f131d68e21fc360f5c8e63377f25cf60706d9afa
Author:     David Sebek <dasebek@gmail.com>
AuthorDate: 2022-01-07 20:18:49 +0000
Commit:     Konstantin Belousov <kib@FreeBSD.org>
CommitDate: 2022-01-09 01:28:01 +0000

    efi: loader: Inline copy_finish function in amd64 trampoline

    Instead of calling the efi_copy_finish function from amd64_tramp,
    include the copy instructions in the trampoline code itself.
    This avoids boot hangs and restarts in the cases when
    the efi_copy_finish code happens to be located at a memory
    location that is overwritten during the copy process.

    This is a direct commit to stable/12 since no-copy loader mode requires
    the kernel update not suitable for the branch.

    PR:     209821

 stand/efi/loader/arch/amd64/amd64_tramp.S   | 59 +++++++++++++++++++----------
 stand/efi/loader/arch/amd64/elf64_freebsd.c | 18 +++++----
 stand/efi/loader/copy.c                     |  8 ++++
 stand/efi/loader/loader_efi.h               |  1 +
 4 files changed, 60 insertions(+), 26 deletions(-)

Comment 74 Konstantin Belousov 2022-01-09 01:33:16 UTC

I am closing this because we cannot do more for stable/12.  For HEAD and stable/13,
the proper fix was implemented some time ago.

Comment 75 Graham Perrin 2022-01-09 03:55:56 UTC

*** Bug 255073 has been marked as a duplicate of this bug. ***

Hey David & Konstantin

First of all thanks a lot for your work and also for taking a look at 12.3 and providing a patch for it! That is really much appreciated.

I've patched our FBSD 12.3, rebuilt and installed everything. Sadly it didn't fix the problem for us. But while testing we found out the following behavior:

Our Hardware: HPE ProLiant BL460c Gen10

We did the following steps:
1) Install the patched version of 12.3 and shutdown afterwards
2) Attach to the virtual serial port (COM1) via "Integrated LightsOut" to monitor boot process
3) "Cold Boot" the server
-> Boot OK, no hang / error
2) Normal Reboot with "shutdown -r now"
-> Boot OK, no Hang / Error
3) Detach from virtual serial port (COM1)
4) Normal Reboot with "shutdown -r now" 
-> Broken. Hangs after "EFI Framebuffer" is printed (see below)
5) "Cold Boot" the server
-> Still Broken.
6) Reatach to virtual serial port (COM1)
7) Reset the server via "ILO"
-> Boot OK, no hang / error

This only happens when using the "loader.efi" from 12.3 (or 12.2). When we copy the loader.efi from 12.1 to the EFI-Partition (/efi/boot/BOOTx64.efi) the server boots and works totally fine in any configuration.

Here's the last printed output befor the hang:
EFI framebuffer information:
addr, size     0xd8000000, 0x300000
dimensions     1024 x 768
stride         1024
masks          0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000


To be honest: I'm not sure if this has the same root cause. Also i'm not quite sure on how to debug this any further.. If you have any input or idea - we'd be very happy for your reply!

(In reply to Dani from comment #76)

Hello Dani,

The patch doesn't address all scenarios that may result in a system freeze. The proper fix is in 13.0-STABLE and 14.0-CURRENT. See comment #66 for a short explanation.

There is one thing that came to my mind after seeing your results. I remember that when I was debugging the problem, I created a KVM virtual machine with multiple serial consoles connected to it. I found that:

- The boot loader uses the existing UEFI frame buffer to print all its output until the "EFI framebuffer information" ending with the "masks ..." line. This information was printed to all serial consoles.

- The boot loader then gives control over to the kernel, which uses its own serial console configuration. It uses COM1 by default for the serial console. The kernel's serial console output was therefore printed only to COM1.

In the case of my virtual machine, all information until the "EFI framebuffer information" block was printed to all serial consoles, and the kernel output that followed was printed only to COM1. Consoles other than COM1 didn't advance past the "EFI framebuffer information", even though the system was running.

In your case, is the video/serial output properly set in the boot loader when you boot the system? It may be possible that the kernel prints the output to a wrong serial console or stops booting because it could not detect a serial console on COM1? Did you try FreeBSD 13-STABLE to see if it boots?

I am running in the same issue on an Apple iMac 7,1 (Core 2 Duo, 2.0 GHz, 4 GB RAM). The device simply hangs while booting with the following text:

EFI framebuffer information:
addr, size          0x0, 0x0
dimensions          0 x 0
stride              0
masks               0x00000000, 0x00000000, 0x00000000, 0x00000000

I have tried booting 13.0-RELEASE, 13.0-STABLE (build 20220113-972796d007c-248936) as well as 12.2-STABLE. All of those builds result in a hang.
This message follows a similar one I've posted on the helloSystem bug tracker (https://github.com/helloSystem/ISO/issues/362), but I'm reporting it here as the issue is present with the "pure" FreeBSD images. Please do let me know if opening a new bug report, instead of continuing the discussion here, would be better.
Thank you!

(In reply to Riccardo Robecchi from comment #78)

It looks like a different problem to me. In your case, the UEFI framebuffer is probably not recognized by the bootloader (the dimensions are all zero). So the kernel doesn't know where to print the output.