Bug 251866 - Because the loader.efi modified the size of EFI_STAGING_SIZE, vmware could not start the system above FreeBSD 12.2
Summary: Because the loader.efi modified the size of EFI_STAGING_SIZE, vmware could no...
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: standards (show other bugs)
Version: 12.2-RELEASE
Hardware: amd64 Any
: Normal Affects Many People
Assignee: Warner Losh
URL:
Keywords: regression
: 244906 (view as bug list)
Depends on:
Blocks:
 
Reported: 2020-12-15 10:40 UTC by YUAN RUI
Modified: 2021-07-08 18:03 UTC (History)
15 users (show)

See Also:
koobs: mfc-stable12?
koobs: mfc-stable11-


Attachments
trigger bug (49.81 KB, image/jpeg)
2020-12-15 10:40 UTC, YUAN RUI
no flags Details
13.0-RELEASE not booting – UEFI – HP ProBook 440 G7 (67.57 KB, image/jpeg)
2021-04-14 19:20 UTC, Graham Perrin
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description YUAN RUI 2020-12-15 10:40:07 UTC
Created attachment 220575 [details]
trigger bug

After debugging, if you modify EFI_STAGING_SIZE to be greater than 64, vmware cannot start the operating system.

#ifndef EFI_STAGING_SIZE
#if defined(__amd64__)
#define	EFI_STAGING_SIZE	100
#elif defined(__arm__)
#define	EFI_STAGING_SIZE	32
#else
#define	EFI_STAGING_SIZE	64
#endif
#endif

Must be reduced to 64 to ensure that vmware can boot correctly.
Comment 1 YUAN RUI 2020-12-15 10:42:37 UTC
The file is located in freebsd-src/stand/efi/loader/copy.c
Comment 2 YUAN RUI 2020-12-15 11:27:23 UTC
It seems that efi_check_space also has a problem, it will reallocate staging_base, which may cause unknown problems.
Comment 3 Dimitry Andric freebsd_committer 2020-12-15 19:40:34 UTC
I can confirm. Using VMware Workstation 16, and a FreeBSD 12.2-RELEASE amd64 iso, if I install it using UEFI, it fails to boot to completion, and VMware shows:

[The firmware encountered an unexpected exception. The virtual machine cannot boot.]

Last few lines in vmware.log for this VM are:

2020-12-15T20:08:49.485+01:00| vcpu-0| I005: SCSI0: RESET BUS
2020-12-15T20:08:49.485+01:00| vcpu-0| I005: SCSI0: RESET BUS
2020-12-15T20:08:49.486+01:00| vcpu-0| I005: DISKUTIL: scsi0:0 : capacity=209715200 logical sector size=512
2020-12-15T20:08:49.505+01:00| vcpu-0| I005: Guest: Status upon boot failure: No Media
2020-12-15T20:08:49.506+01:00| vcpu-0| I005: DISKUTIL: scsi0:1 : capacity=20971520 logical sector size=512
2020-12-15T20:08:49.508+01:00| vcpu-0| I005: Guest: Status upon boot failure: No Media
2020-12-15T20:08:49.745+01:00| vcpu-0| I005: Guest: About to do EFI boot: EFI VMware Virtual IDE CDROM Drive (IDE 1:0)
2020-12-15T20:08:58.704+01:00| vcpu-0| I005: UHCI: HCReset
2020-12-15T20:08:58.716+01:00| vcpu-1| I005: CPU reset: soft (mode Interp)
2020-12-15T20:08:58.716+01:00| vcpu-2| I005: CPU reset: soft (mode Interp)
2020-12-15T20:08:58.716+01:00| vcpu-3| I005: CPU reset: soft (mode Interp)
2020-12-15T20:08:58.716+01:00| vcpu-0| I005: Guest: Firmware has transitioned to runtime.
2020-12-15T20:08:59.140+01:00| vcpu-0| I005: Msg_Post: Error
2020-12-15T20:08:59.140+01:00| vcpu-0| I005: [msg.efi.exception] The firmware encountered an unexpected exception. The virtual machine cannot boot.
2020-12-15T20:08:59.140+01:00| vcpu-0| I005: ----------------------------------------
Comment 4 Dimitry Andric freebsd_committer 2020-12-15 22:10:04 UTC
(In reply to YUAN RUI from comment #0)
> Must be reduced to 64 to ensure that vmware can boot correctly.

I can at least confirm that building /usr/src/stand/ (from the latest stable/12 svn checkout) with EFI_STAGING_SIZE set to 64, then installing loader_lua.efi to /boot/loader.efi appears to work.
Comment 5 Yuri Pankov freebsd_committer 2020-12-17 04:40:40 UTC
Reverting EFI_STAGING_SIZE back to 64 allows built "cdrom" image to successfully boot on 64GB VM on ESXi 7.  Thanks a lot for finding what broke it!
Comment 6 Andrew Turner freebsd_committer 2020-12-17 12:17:46 UTC
What do you get from the memmap loader command? I'm guessing there is reserved memory that conflicts.
Comment 7 Dimitry Andric freebsd_committer 2020-12-17 13:08:39 UTC
Note that this is with a loader.efi rebuilt with EFI_STAGING_SIZE=64:

OK memmap
                   Type     Physical      Virtual   #Pages Attr
          ACPIMemoryNVS 000000000000 000000000000 00000001 UC WC WT WB
     ConventionalMemory 000000001000 000000000000 0000009f UC WC WT WB
     ConventionalMemory 000000100000 000000000000 000046de UC WC WT WB
             LoaderData 0000047de000 000000000000 00008000 UC WC WT WB
             LoaderCode 00000c7de000 000000000000 00000081 UC WC WT WB
             LoaderData 00000c85f000 000000000000 0000217d UC WC WT WB
             LoaderCode 00000e9dc000 000000000000 0000001a UC WC WT WB
       BootServicesCode 00000e9f6000 000000000000 000001ba UC WC WT WB
       BootServicesData 00000ebb0000 000000000000 00000301 UC WC WT WB
       BootServicesCode 00000eeb1000 000000000000 00000040 UC WC WT WB
    RuntimeServicesCode 00000eef1000 000000000000 00000009 UC WC WT WB
       BootServicesCode 00000eefa000 000000000000 00000013 UC WC WT WB
    RuntimeServicesCode 00000ef0d000 000000000000 00000005 UC WC WT WB
       BootServicesCode 00000ef12000 000000000000 0000001a UC WC WT WB
    RuntimeServicesCode 00000ef2c000 000000000000 00000005 UC WC WT WB
       BootServicesCode 00000ef31000 000000000000 00000037 UC WC WT WB
       BootServicesData 00000ef68000 000000000000 0000048f UC WC WT WB
     ConventionalMemory 00000f3f7000 000000000000 0000006a UC WC WT WB
       BootServicesData 00000f461000 000000000000 000007a0 UC WC WT WB
     ConventionalMemory 00000fc01000 000000000000 00000002 UC WC WT WB
       BootServicesData 00000fc03000 000000000000 000000fa UC WC WT WB
     ConventionalMemory 00000fcfd000 000000000000 0000000c UC WC WT WB
       BootServicesData 00000fd09000 000000000000 000000ee UC WC WT WB
     ConventionalMemory 00000fdf7000 000000000000 00000001 UC WC WT WB
       BootServicesCode 00000fdf8000 000000000000 000000ef UC WC WT WB
    RuntimeServicesCode 00000fee7000 000000000000 00000020 UC WC WT WB
    RuntimeServicesData 00000ff07000 000000000000 00000050 UC WC WT WB
      ACPIReclaimMemory 00000ff57000 000000000000 0000001c UC WC WT WB
          ACPIMemoryNVS 00000ff73000 000000000000 00000004 UC WC WT WB
       BootServicesData 00000ff77000 000000000000 00000048 UC WC WT WB
       BootServicesCode 00000ffbf000 000000000000 00000041 UC WC WT WB
     ConventionalMemory 000010000000 000000000000 000b0000 UC WC WT WB
     ConventionalMemory 000100000000 000000000000 00040000 UC WC WT WB
               Reserved 0000000c0000 000000000000 00000040 WP RP
         MemoryMappedIO 0000ffc00000 000000000000 0000002a UC
Comment 8 Andrew Turner freebsd_committer 2020-12-17 13:31:03 UTC
It looks like the issue is the first LoaderData memory. Can you try commenting out the if(running_on_hyperv()) check in efi_copy_init in
Comment 9 Andrew Turner freebsd_committer 2020-12-17 13:32:18 UTC
(In reply to Andrew Turner from comment #8)
It looks like the issue is the first LoaderData memory. Can you try commenting out the if(running_on_hyperv()) check in efi_copy_init in copy.c so efi_verify_staging_size always runs. It should limit the memory to just the number of pages available.
Comment 10 YUAN RUI 2020-12-17 13:40:27 UTC
If (running_on_hyperv()) is commented out, this problem still occurs.
Comment 11 Andrew Turner freebsd_committer 2020-12-17 13:53:42 UTC
Did it print the line starting "Staging area's size is reduced:" and if so what did it print?
Comment 12 Dimitry Andric freebsd_committer 2020-12-17 14:35:23 UTC
Would it print "Staging area's size is reduced:" before or after the initial messages? If it prints it very early, I cannot see it because the output scrolls off the screen, and the loader's prompt does not seem to support any form of scrollback. (I tried scroll lock, and shift-pageup/down, but neither worked.)

In any case, commenting out the if, but not the call to efi_verify_staging_size(), and going with the default EFI_STAGING_SIZE of 100 still does not work:

OK boot -s
Start @ 0xffffffff8036b000 ...
EFI framebuffer information:
addr, size     0xf0000000, 0x300000
dimensions     1024 x 768
stride         1024
masks          0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000
!!!! X64 Exception Type - 0E(#PF - Page-Fault)  CPU Apic ID - 00000000 !!!!
ExceptionData - 0000000000000000  I:0 R:0 U:0 W:0 P:0 PK:0 S:0
RIP  - FFFFFFFF81885F90, CS  - 0000000000000018, RFLAGS - 0000000000010046
RAX  - 00000000087DF000, RCX - 0000000006600000, RDX - 00000000087DF000
RBX  - 00000000023DEFF8, RSP - 00000000023DEFF8, RBP - 000B0001000B8781
RSI  - 0000000000000000, RDI - 00000000023DEFF8
R8   - 00000000023DB000, R9  - FFFFFFFF8036B000, R10 - 000000000A9706D0
R11  - 00000000000005D0, R12 - 0000000002655000, R13 - 000000000264B000
R14  - 00000000023DB000, R15 - FFFFFFFF8036B000
DS   - 0000000000000008, ES  - 0000000000000008, FS  - 0000000000000008
GS   - 0000000000000008, SS  - 0000000000000008
CR0  - 0000000080010033, CR2 - FFFFFFFF81885F90, CR3 - 000000000FF78000
CR4  - 0000000000000668, CR8 - 0000000000000000
DR0  - 0000000000000000, DR1 - 0000000000000000, DR2 - 0000000000000000
DR3  - 0000000000000000, DR6 - 00000000FFFF0FF0, DR7 - 0000000000000400
GDTR - 00000000FFFFFEA0 000000000000002F, LDTR - 0000000000000000
IDTR - 000000000FEBF3E0 0000000000000FFF,   TR - 0000000000000000
FXSAVE_STATE - 00000000023DEC50
!!!! Can't find image information. !!!!

and then VMware shows its "fatal exception" dialog.
Comment 13 Andrew Turner freebsd_committer 2020-12-17 14:49:59 UTC
It's early in the boot process. It looks like you can use /efi/freebsd/loader.env on the same fs as loader.efi (probably the FAT fs) to set early variables, e.g. to use the serial console.
Comment 14 commit-hook freebsd_committer 2020-12-17 17:02:27 UTC
A commit references this bug:

Author: imp
Date: Thu Dec 17 17:02:09 UTC 2020
New revision: 368721
URL: https://svnweb.freebsd.org/changeset/base/368721

Log:
  Drop EFI_STAGING_SIZE back down to 64M

  vmware can't cope with anything larger than 64MB. Drop this back to
  64MB everywhere but arm.

  PR: 251866
  MFC After: 1 week

Changes:
  head/stand/efi/loader/copy.c
Comment 15 YUAN RUI 2020-12-17 17:43:15 UTC
It seems to increase the size for compatibility with the nvidia driver, but if it is reduced back, the nvidia driver cannot be loaded?
Comment 16 Yuri Pankov freebsd_committer 2020-12-17 17:48:12 UTC
*** Bug 244906 has been marked as a duplicate of this bug. ***
Comment 17 Graham Perrin 2020-12-19 14:28:25 UTC
Thank you. 

I should have the opportunity to test a 13.0-CURRENT snapshot on affected hardware – an HP ProBook 440 G7 – on, or very soon after, Monday 2021-01-04.
Comment 18 Kubilay Kocak freebsd_committer freebsd_triage 2020-12-22 00:29:51 UTC
^Triage: Assign to committer resolving

@Warner Is an EN and/or a RELEASE/images re-roll possible and warranted for this?
Comment 19 Glen Barber freebsd_committer 2020-12-22 01:01:31 UTC
(In reply to Kubilay Kocak from comment #18)

I can answer the second part - a re-roll is not possible for release builds at the  moment.
Comment 20 YUAN RUI 2020-12-22 08:46:35 UTC
(In reply to Glen Barber from comment #19)

But is it possible to remake the installation image (iso)?
Comment 21 YUAN RUI 2020-12-22 08:49:13 UTC
(In reply to commit-hook from comment #14)

It seems that efi_check_space will dynamically expand this size. Does this cause unknown problems?
Comment 22 Glen Barber freebsd_committer 2020-12-22 14:41:20 UTC
(In reply to YUAN RUI from comment #20)

Unfortunately, no.  There is no precident for doing so, however this is something that has been on my list of things to look into for quite some time.
Comment 23 Graham Perrin 2020-12-23 08:15:17 UTC
Thank you, 

(In reply to Glen Barber from comment #22)

Reading this alongside (duplicate) bug 244906 comment 2

Where 12.2 can not be installed on a physical or virtual computer:

* should we advise installation of 12.1 followed by an 
  upgrade to 12.2-RELEASE-p2 or greater?

<https://download.freebsd.org/ftp/releases/ISO-IMAGES/12.1/>

<https://download.freebsd.org/ftp/releases/VM-IMAGES/12.1-RELEASE/>
Comment 24 Andrew Turner freebsd_committer 2020-12-23 10:54:24 UTC
(In reply to YUAN RUI from comment #21)
I would like to add a maximum size for this memory to grow on x86 to fix this. I think the code in efi_verify_staging_size could be used to check this, but will need to get an x86 UEFI test environment set up first.
Comment 25 Graham Perrin 2020-12-25 14:44:42 UTC
FreeBSD-13.0-CURRENT-amd64-20201224-3cc0c0d66a0-255241-memstick.img

> still not working on HP EliteBook 830 G7.

– <https://lists.freebsd.org/pipermail/freebsd-current/2020-December/077970.html>
Comment 26 Graham Perrin 2021-01-21 19:48:57 UTC
Any overlap with some of the more recent comments under bug 209821?
Comment 27 Graham Perrin 2021-03-09 07:11:44 UTC
(In reply to Graham Perrin from comment #17)

> HP ProBook 440 G7

– confirmed, this type of new hardware can not boot from a USB flash drive that was written from: 

FreeBSD-14.0-CURRENT-amd64-20210304-483c6da3a20-257149-disc1.iso

----

If I install EOL FreeBSD 12.1-RELEASE, then upgrade (to 13.0-RC1† for example): 

* where the .iso files for 12.2-RELEASE (and greater) will not boot, might I find that an _installed_ system will boot?

If I recall correctly, the answer is "No" …

----

<http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/amd64/ISO-IMAGES/> 

– the .iso files for 12.1-RELEASE are not there. Please, where are they? 


† https://gist.github.com/grahamperrin/6ae91f3cb559c7cd65f04cee1be13159
Comment 28 commit-hook freebsd_committer 2021-03-22 21:49:04 UTC
A commit in branch stable/12 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=b304cd9789ca7ff3df629af42a976450e8660a11

commit b304cd9789ca7ff3df629af42a976450e8660a11
Author:     Warner Losh <imp@FreeBSD.org>
AuthorDate: 2020-12-17 17:02:09 +0000
Commit:     Ryan Moeller <freqlabs@FreeBSD.org>
CommitDate: 2021-03-22 20:30:23 +0000

    Drop EFI_STAGING_SIZE back down to 64M

    vmware can't cope with anything larger than 64MB. Drop this back to
    64MB everywhere but arm.

    PR: 251866
    MFC After: 1 week

    (cherry picked from commit 4d6047edb675e52b8fad57135ab3ded8e66d0dac)

 stand/efi/loader/copy.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
Comment 29 Graham Perrin 2021-04-14 19:20:01 UTC
Created attachment 224114 [details]
13.0-RELEASE not booting – UEFI – HP ProBook 440 G7

Comparative test results: 

<https://gist.github.com/grahamperrin/5eca8231fa7e6a94a1f55991bcd7f3c4>
Comment 30 Warner Losh freebsd_committer 2021-04-14 20:18:39 UTC
(In reply to Graham Perrin from comment #29)
There's been a lot of chance since 12.x. this may be unrelated to this size change and should have a new bug assigned to it, I think, so we don't conflate the two issues if they are indeed separate.
Comment 31 Graham Perrin 2021-04-15 03:03:31 UTC
Thank you, 

(In reply to Warner Losh from comment #30)

continuation here was at the suggestion of Toomas Soome (tsoome in IRC). 

The hardware cases are now spun off: 

* bug 255072 for legacy boot
* bug 255073 for UEFI boot.
Comment 32 Warner Losh freebsd_committer 2021-04-15 03:10:06 UTC
(In reply to Graham Perrin from comment #31)
thanks!
Comment 33 Warner Losh freebsd_committer 2021-07-08 18:03:08 UTC
I'm not actively working on a fix, so move this back to 'open'