Bug 254490 - REGRESSION: Hybrid disc1.iso fails to boot from USB on BIOS/Legacy ThinkPads and other systems
Summary: REGRESSION: Hybrid disc1.iso fails to boot from USB on BIOS/Legacy ThinkPads ...
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: 13.0-STABLE
Hardware: amd64 Any
: --- Affects Many People
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-03-22 23:44 UTC by Michael Dexter
Modified: 2023-02-02 05:11 UTC (History)
8 users (show)

See Also:


Attachments
BETA4-RC1-oneline.log (5.24 KB, text/plain)
2021-03-24 08:55 UTC, Lewis Cook
no flags Details
BETA4-RC1-long.log (31.37 KB, text/plain)
2021-03-24 08:57 UTC, Lewis Cook
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Dexter freebsd_triage 2021-03-22 23:44:57 UTC
FreeBSD 13.0-BETA4 introduced a regression that prevents the use of a disc1.iso hybrid image from being dd'd to a USB or similar device and booting, when the system is in BIOS/Legacy mode. It produces the error:

CD Loader 1.2

Building the boot loader arguments
Looking up /BOOT/LOADER... File not found
Looking up /boot/Loader... File not found
Boot failed
_

Experienced on:

ThinkPad X230 (multiple)
ThinkPad T430
ThinkPad T400
ACER 1410
Dell e5470

Confirmed on 13.0-RC1 through RC3.
Comment 1 Toomas Soome freebsd_committer freebsd_triage 2021-03-23 06:54:28 UTC
(In reply to Michael Dexter from comment #0)


This output is from cdboot boot program, which is supposed to run while booting from cd/dvd device and it does not know how to handle hard disk.

It means, the isoboot boot program installation into iso image did not happen or is somehow broken. We would need to analyze iso file and its creation process.
Comment 2 Michael Dexter freebsd_triage 2021-03-23 22:46:20 UTC
Thank you Toomas.
I have manually confirmed that BETA1, 2, and 3 are working as expected, to help narrow the search. Might there be a cgit-web-thing page that shows history by directory like svnweb did?
Comment 3 Vincent Milum Jr 2021-03-24 07:30:41 UTC
I can confirm that I can 100% reproduce this issue as well on my Acer Aspire One laptop.

13-BETA4 works exactly as expected for me, while 13-RC1, 13-RC2, and 13-RC3 all fail. This narrows down the commits to about a single week.

Michael Dexter can you confirm if your issue was actually in BETA4? And if so, which exact ISO were you using?

In my case, I download the ISO (linked below), and then flash it to a USB drive using Rufus on Windows 10. Multiple USB drives have been used, and multiple laptops as well. UEFI consistently works, Legacy BIOS consistently fails.

The last ISO that works for me is:
https://download.freebsd.org/ftp/releases/amd64/amd64/ISO-IMAGES/13.0/FreeBSD-13.0-BETA4-amd64-disc1.iso

The first broken ISO for me is:
https://download.freebsd.org/ftp/releases/amd64/amd64/ISO-IMAGES/13.0/FreeBSD-13.0-RC1-amd64-disc1.iso

https://twitter.com/DarkainMX/status/1374582571451916290
Comment 4 Lewis Cook freebsd_committer freebsd_triage 2021-03-24 08:55:42 UTC
Created attachment 223543 [details]
BETA4-RC1-oneline.log

Likewise here.

Tested amd64-{disk1,bootonly} on my ThinkPad X230 in Legacy BIOS mode with RC{1,2,3}, and anything prior works fine BETA{1,2,3,4}.  I've attached commit logs between BETA4 and RC1 to hopefully bisect the commit causing this new behaviour. For anyone interested, there are 46 commits between BETA4 to RC1, so thankfully nothing *too* extreme.

As for for the commit hashes in releng/13.0:
 
BETA1... 638e531019fd360ec43d2662c89d12ae632f168f
BETA2... 6461715f1ae19d028fcae45768cfa91f88b31f32
BETA3... 150b4388d3b5654d02071618dcdae48d6f11161f
BETA4... e32bc25362975db9896eeffc2933e078930a5383

RC1...  60e8939aa85b07f3271dcc01730727dda04c8bf6
RC2...  13c22f7495305f5b92874128b088ab47d9512c20
RC3...  8f731a397ad4dc7b17622c0e69ac045f4a7b9d5b
Comment 5 Lewis Cook freebsd_committer freebsd_triage 2021-03-24 08:57:16 UTC
Created attachment 223544 [details]
BETA4-RC1-long.log

Included long-form logs that contain full commit messages and context.
Comment 6 Michael Dexter freebsd_triage 2021-03-24 18:39:11 UTC
(In reply to Vincent Milum Jr from comment #3)
"Michael Dexter can you confirm if your issue was actually in BETA4? And if so, which exact ISO were you using?"

I have re-confirmed with BETA3 and BETA4 disc1 images from ftp.freebsd.org, which I trust are in sync.
Comment 7 Michael Dexter freebsd_triage 2021-03-25 18:18:02 UTC
All, I suspect the issue is in one of these two components but cgit is not providing a history by date:

https://cgit.freebsd.org/src/tree/release
https://cgit.freebsd.org/src/tree/usr.sbin/makefs

Please share how to show dates with this as one can do with svnweb.freebsd.org, or re-run your above commit summary with additional information, paths, descriptions, --stat etc.

Thank you!
Comment 8 Mark Millard 2021-03-25 21:40:51 UTC
(In reply to Michael Dexter from comment #7)

You seem to possibly be working at the wrong level
for:

https://cgit.freebsd.org/src/tree/release
https://cgit.freebsd.org/src/tree/usr.sbin/makefs

For example going to:

https://cgit.freebsd.org/src/tree/

and selecting "log" on the "release" line shows
a dated list of the commits in release/ (one
line summaries are shown).

https://cgit.freebsd.org/src/tree/usr.sbin/

and selecting "log" on the "makefs" line does
similarly for makefs/ .

But to see what file(s) a commit changed one
has to select the commit line's link and look
at the commit's details.
Comment 9 Michael Dexter freebsd_triage 2021-03-25 22:30:46 UTC
Thank you Mark, mzar and Allan Jude for pointing out the Log view! This is a new tool to all of us.

These emerge as the "usual suspects" for incorrectly-built ISOs but are not guaranteed to be responsible for the regression:

https://cgit.freebsd.org/src/log/usr.sbin/makefs
https://cgit.freebsd.org/src/log/release

If I am not mistaken, this is the date range where the regression was introduced:
Update to BETA4 2021-02-26
Update to RC1   2021-03-05

Within that range these commits stand out for me as possible causes but I cannot say for sure without bisecting and building:

efibootmgr: Check for efi supported after parsing args	2021-03-01
mkimg: We always want the last block of the last inserted partition 2021-03-02
mkimg: Add support for offset if the source is an image 2021-03-02

Michael
Comment 10 Vincent Milum Jr 2021-03-26 04:28:57 UTC
I've narrowed it down to this commit. Running a couple more tests to validate it right now

"mkimg: We always want the last block of the last inserted partition"

https://github.com/freebsd/freebsd-src/commit/f3054d18bfbd4629cb695922f3cd1fab36b1dc5b
Comment 11 Vincent Milum Jr 2021-03-26 07:59:18 UTC
I now think that both mkimg commits are related. I just tried to do a release build from releng/13.0@c9b4e5e9ae398f3bb8ba85507d9710d8a73a1a04 with the commit I previous mention reverted, but ended up with this build error... Its now 1am, and I'll play with this more in the morning, running a test with releng/13.0@head with both mkimg commits reverted to see if it builds and properly boots under legacy BIOS.


--------------


touch disc1
sh /usr/src/release/amd64/mkisoimages.sh -b 13_0_RC3_p1_amd64_CD disc1.iso disc1
Creating `/tmp/efiboot.rQheG5'
/tmp/efiboot.rQheG5: 4039 sectors in 4039 FAT12 clusters (512 bytes/cluster)
BytesPerSec=512 SecPerClust=1 ResSectors=1 FATs=2 RootDirEnts=512 Sectors=4096 Media=0xf0 FATsecs=12 SecPerTrack=63 Heads=255 HiddenSecs=0
Populating `/tmp/efiboot.rQheG5'
Image `/tmp/efiboot.rQheG5' complete
mkimg: partition 0 overlaps partition 1
*** Error code 1

Stop.
make[1]: stopped in /usr/src/release
*** Error code 1

Stop.
make: stopped in /usr/src/release
Comment 12 Emmanuel Vadot freebsd_committer freebsd_triage 2021-03-26 13:37:38 UTC
Could you test this :
diff --git a/release/amd64/mkisoimages.sh b/release/amd64/mkisoimages.sh
index 1a1440fa198..a9e8a2c0439 100644
--- a/release/amd64/mkisoimages.sh
+++ b/release/amd64/mkisoimages.sh
@@ -89,8 +89,8 @@ if [ "$bootable" != "" ]; then
        $MKIMG -s gpt \
            --capacity $imgsize \
            -b "$BASEBITSDIR/boot/pmbr" \
-           $espparam \
            -p freebsd-boot:="$BASEBITSDIR/boot/isoboot" \
+           $espparam \
            -o hybrid.img
 
        # Drop the PMBR, GPT, and boot code into the System Area of the ISO.

I think I've changed the logic that partition must now be passed in the intented order in case we specify an offset at some point.
Comment 13 Emmanuel Vadot freebsd_committer freebsd_triage 2021-03-26 13:40:30 UTC
Adding -v to mkimg in the script gives me :

Before my change
partition 1: starting block 80 ... size 2097152 bytes (4096 blocks)
    location 40960 bytes (80 blocks)
partition 2: starting block 3 ... size 12288 bytes (24 blocks)

After my change
partition 1: starting block 80 ... size 2097152 bytes (4096 blocks)
    location 40960 bytes (80 blocks)
partition 2: starting block 4176 ... size 12288 bytes (24 blocks)

I also think that it's logical that the partition are given in the indented order so if that fix it I'll just document this behavior.
Comment 14 Vincent Milum Jr 2021-03-26 17:57:39 UTC
With that patch, I'm officially booted into the installer now! :D

Successfully booted into 13.0-RC3-p1 from the latest commit in releng/13.0 and this patch applied.
Comment 15 commit-hook freebsd_committer freebsd_triage 2021-03-27 11:09:56 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=90d2f7c413f9fc4ac479fa5e91ba1de6d4ea8d45

commit 90d2f7c413f9fc4ac479fa5e91ba1de6d4ea8d45
Author:     Emmanuel Vadot <manu@FreeBSD.org>
AuthorDate: 2021-03-27 11:04:51 +0000
Commit:     Emmanuel Vadot <manu@FreeBSD.org>
CommitDate: 2021-03-27 11:04:51 +0000

    release: amd64: Fix ISO/USB hybrid image

    Recent mkimg changes forces to have partitions given in explicit order.
    This is so we can have the first partition starting at a specific offset
    and the next ones starting after without having to specify an offset.
    Switch the partition in the mkisoimage.sh script so the first one created
    is the isoboot one.

    PR:    254490
    Reported by:    Michael Dexter <editor@callfortesting.org
    Tested by:      Vincent Milum Jr <freebsd@darkain.com>
    MFC after:      Right now

 release/amd64/mkisoimages.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 16 commit-hook freebsd_committer freebsd_triage 2021-03-27 11:09:58 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=08639983e0384556a37d19814f55417f604964a1

commit 08639983e0384556a37d19814f55417f604964a1
Author:     Emmanuel Vadot <manu@FreeBSD.org>
AuthorDate: 2021-03-27 11:04:51 +0000
Commit:     Emmanuel Vadot <manu@FreeBSD.org>
CommitDate: 2021-03-27 11:09:22 +0000

    release: amd64: Fix ISO/USB hybrid image

    Recent mkimg changes forces to have partitions given in explicit order.
    This is so we can have the first partition starting at a specific offset
    and the next ones starting after without having to specify an offset.
    Switch the partition in the mkisoimage.sh script so the first one created
    is the isoboot one.

    PR:    254490
    Reported by:    Michael Dexter <editor@callfortesting.org
    Tested by:      Vincent Milum Jr <freebsd@darkain.com>
    MFC after:      Right now

    (cherry picked from commit 90d2f7c413f9fc4ac479fa5e91ba1de6d4ea8d45)

 release/amd64/mkisoimages.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 17 commit-hook freebsd_committer freebsd_triage 2021-03-28 18:55:44 UTC
A commit in branch releng/13.0 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=d6616e3dbaeeb0cf336dda3eec1c5fa7faf8a5c5

commit d6616e3dbaeeb0cf336dda3eec1c5fa7faf8a5c5
Author:     Emmanuel Vadot <manu@FreeBSD.org>
AuthorDate: 2021-03-27 11:04:51 +0000
Commit:     Emmanuel Vadot <manu@FreeBSD.org>
CommitDate: 2021-03-28 18:54:28 +0000

    release: amd64: Fix ISO/USB hybrid image

    Recent mkimg changes forces to have partitions given in explicit order.
    This is so we can have the first partition starting at a specific offset
    and the next ones starting after without having to specify an offset.
    Switch the partition in the mkisoimage.sh script so the first one created
    is the isoboot one.

    Approved by:    re(gjb)
    PR:    254490
    Reported by:    Michael Dexter <editor@callfortesting.org
    Tested by:      Vincent Milum Jr <freebsd@darkain.com>
    MFC after:      Right now

    (cherry picked from commit 90d2f7c413f9fc4ac479fa5e91ba1de6d4ea8d45)
    (cherry picked from commit 08639983e0384556a37d19814f55417f604964a1)

 release/amd64/mkisoimages.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 18 Emmanuel Vadot freebsd_committer freebsd_triage 2021-03-28 18:59:15 UTC
Fixed in every affected branches.
Closing now.
Thanks for the report and sorry again for the breakage.
Comment 19 Ian Smith 2023-02-02 05:11:49 UTC
(In reply to Emmanuel Vadot from comment #18)
Not fixed in 12.x unfortunately.  12.3-R was ok but 12.4 suffered the same regression.  I reported it to freebsd-questions@ and re@ in this post:

Boot fail: 12.4-R-{dvd1,disc1}.iso via USB
https://lists.freebsd.org/archives/freebsd-questions/2023-January/002750.html

The following two messages are my replies to Xin LI who kindly pointed me to this bug, and Colin Percival as re@, saying he can't imagine a fixed "patch level release" in this case.

I've not been able to find it, but I suspect that this fix likely preceeded the first 12.x version to include the ability to write the .iso to an USB stick - 12.1 going by release announcements.