Bug 254395 - bsdinstall: fail script install after BETA3
Summary: bsdinstall: fail script install after BETA3
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 13.0-STABLE
Hardware: Any Any
: Normal Affects Some People
Assignee: freebsd-bugs (Nobody)
URL:
Keywords: regression
Depends on:
Blocks: 231027
  Show dependency treegraph
 
Reported: 2021-03-19 03:35 UTC by Andrey Fesenko
Modified: 2021-03-24 14:19 UTC (History)
5 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Andrey Fesenko 2021-03-19 03:35:26 UTC
After BETA3 or RC1 (CURRENT add efi partition /boot/efi mount) script install UEFI system fail checksum/extract base.txz

Old working `PARTITIONS="$DISKSLICE GPT { 260M efi , auto freebsd-ufs / }"`
if edit `PARTITIONS="$DISKSLICE GPT { 260M efi /boot/efi, auto freebsd-ufs / }"` not fix

root /mnt and efi /mnt/boot/efi mount succesfull

bsdinstall_log

 * DEBUG: f_variable_set_defaults: Initializing defaults...
 * DEBUG: f_getvar: var=[debug] value=[1] r=0
 * DEBUG: f_getvar: var=[editor] value=[/usr/bin/ee] r=0
 * DEBUG: f_getvar: var=[ftpState] value=[auto] r=0
 * DEBUG: f_getvar: var=[ftpUser] value=[ftp] r=0
 * DEBUG: f_getvar: var=[hostname] value=[] r=0
 * DEBUG: f_getvar: var=[MEDIA_TIMEOUT] value=[300] r=0
 * DEBUG: f_getvar: var=[nfs_reserved_port_only] value=[NO] r=0
 * DEBUG: f_getvar: var=[nfs_use_tcp] value=[NO] r=0
 * DEBUG: f_getvar: var=[nfs_use_v3] value=[YES] r=0
 * DEBUG: f_getvar: var=[PKG_TMPDIR] value=[/var/tmp] r=0
 * DEBUG: f_getvar: var=[releaseName] value=[13.0-RC2] r=0
 * DEBUG: f_variable_set_defaults: Defaults initialized.
 * DEBUG: variable.subr: Successfully loaded.
 * DEBUG: f_include_lang: file=[/usr/libexec/bsdconfig/include/messages.subr] lang=[]
 * DEBUG: dialog.subr: DIALOG_SELF_INITIALIZE=[1]
 * DEBUG: f_dialog_init: ARGV=[/tmp/installerconfig] GETOPTS_STDARGS=[dD:SX]
 * DEBUG: f_dialog_init: SECURE=[] USE_XDIALOG=[]
 * DEBUG: f_dialog_init: dialog(1) API initialized.
 * DEBUG: dialog.subr: Successfully loaded.
 * DEBUG: f_include: file=[/usr/share/bsdconfig/variable.subr]
 * DEBUG: Began Installation at Fri Mar 19 02:37:31 UTC 2021
 * ./boot/efi/: Can't restore time
 * tar: Error exit delayed from previous errors.
Comment 1 Nathan Whitehorn freebsd_committer 2021-03-19 11:54:26 UTC
You should not include the EFI partition in the PARTITIONS variable in general, but that won't cause this issue. Does the installer actually fail, resulting in a non-bootable system? Or do you just see this message in the output? Any more detail about your install script? What are you untarring?
Comment 2 Ryan Moeller freebsd_committer 2021-03-19 12:50:39 UTC
This affects me as well. The install completely fails. I'm even doing EFI installs: PARTITIONS="zvol/pool/dataset/vol gpt"

I've used this same script quite a lot and it has only started to fail recently. I noticed yesterday and started looking into it a bit before I got pulled into something else.

The tarball that fails to be extracted is base.txz (it's the one that contains ./boot/efi). The output on the console shows this as well:

DEBUG: dialog.subr: DEBUG_SELF_INITIALIZE=[]
DEBUG: UNAME_S=[FreeBSD] UNAME_P=[amd64] UNAME_R=[14.0-CURRENT]
DEBUG: common.subr: Successfully loaded.
DEBUG: f_debug_init: ARGV=[mount] GETOPTS_STDARGS=[dD:]
DEBUG: f_debug_init: debug=[1] debugFile=[]
DEBUG: Running installation step: mount
DEBUG: dialog.subr: DEBUG_SELF_INITIALIZE=[]
DEBUG: UNAME_S=[FreeBSD] UNAME_P=[amd64] UNAME_R=[14.0-CURRENT]
DEBUG: common.subr: Successfully loaded.
DEBUG: f_debug_init: ARGV=[checksum] GETOPTS_STDARGS=[dD:]
DEBUG: f_debug_init: debug=[1] debugFile=[]
DEBUG: Running installation step: checksum
DEBUG: Extracting /storage/bhyve/14.0-CURRENT/a771bf748f9/kernel.txz
DEBUG: Extracting /storage/bhyve/14.0-CURRENT/a771bf748f9/kernel-dbg.txz
DEBUG: Extracting /storage/bhyve/14.0-CURRENT/a771bf748f9/base.txz
ERROR: bsdinstall script failed

At this point the script exits, leaving the filesystems mounted even:

/dev/zvol/storage/bhyve/14.0-CURRENT/a771bf748f9/templatep2 on /mnt (ufs, local, journaled soft-updates)
/dev/zvol/storage/bhyve/14.0-CURRENT/a771bf748f9/templatep1 on /mnt/boot/efi (msdosfs, local)
devfs on /mnt/dev (devfs)

# cat /tmp/bsdinstall-tmp-fstab
/dev/zvol/storage/bhyve/14.0-CURRENT/a771bf748f9/templatep1     /mnt/boot/efi   msdosfs rw      2       2
/dev/zvol/storage/bhyve/14.0-CURRENT/a771bf748f9/templatep2     /mnt    ufs     rw      1       1
# cat /tmp/bsdinstall_etc/fstab
# Device        Mountpoint      FStype  Options Dump    Pass#
/dev/zvol/storage/bhyve/14.0-CURRENT/a771bf748f9/templatep1     /boot/efi               msdosfs rw      2       2
/dev/zvol/storage/bhyve/14.0-CURRENT/a771bf748f9/templatep2     /               ufs     rw      1       1
/dev/zvol/storage/bhyve/14.0-CURRENT/a771bf748f9/templatep3     none            swap    sw      0       0
Comment 3 Nathan Whitehorn freebsd_committer 2021-03-19 13:13:59 UTC
Well, I found the issue and there's even a nice comment in the relevant code that an older, smarter version of me put in in 2018 describing exactly why this is going to break:

--------
# Unpack distributions
bsdinstall checksum
for set in $DISTRIBUTIONS; do
        f_dprintf "Extracting $BSDINSTALL_DISTDIR/$set"
        # XXX: this will fail if any mountpoints are FAT, due to inability to
        # set ctime/mtime on the root of FAT partitions. tar has no option to
        # ignore this. We probably need to switch back to distextract here
        # to properly support EFI.
        tar -xf "$BSDINSTALL_DISTDIR/$set" -C $BSDINSTALL_CHROOT
done

--------

I'll try to get a patch in today. Apologies for the breakage, and thanks for the report.
Comment 4 Andrey Fesenko 2021-03-19 13:34:45 UTC
(In reply to Nathan Whitehorn from comment #1)

Installer fail, if set PARTITIONS="$DISKSLICE GPT { auto freebsd-ufs / }" fail too
https://64.media.tumblr.com/2c95d2ddeed5cc2bd7ffb8fc68f34436/f8abb8c6d81ea645-e4/s1280x1920/6c4753c8bfdd3af75f597371189c4dcb1cbf0880.png

build packer
"qemu_binary": "/usr/libexec/qemu-kvm",
"-bios",
"/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd"

Check virtualbox builder BETA4 installer ok. RC2 fail, similar error
Comment 5 Andrey Fesenko 2021-03-19 13:37:23 UTC
(In reply to Nathan Whitehorn from comment #3)
If possible, it is also worth mentioning in the release notes that you no longer need to specify the efi and boot partitions, but is it possible to make the efi partition smaller, since 260M is a lot for a virtual machine, where there will be only 1 system
Comment 6 Nathan Whitehorn freebsd_committer 2021-03-19 20:06:03 UTC
Thanks for the suggestion about the documentation -- I've updated the man page.

The core problem here is that our tar can't extract archives to FAT32 with default options, since it treats inability to set modification time as a fatal error and FAT32 doesn't let you do that on the root directory. As such, any file in the release tarballs can't be extracted to FAT32. For interactive installations, the bsdinstall distextract tool, a CURSES-y frontend to libarchive, solves this by ignoring ctime/mtime errors.

Some extra commentary on solutions, so it can be in one place. Possibilities are:

1. We drop /boot/efi from mtree. That will result in it not existing in base.txz, solving this issue, but will result in it not being in mtree. It will also leave in place an identical bug that will break scripted installation on bare-metal POWER8 and POWER9 systems, although that is a tier-2 platform.

2. We add an option to tar to ignore failure in setting ctime/mtime, like the interactive installer uses. This has the difficulty that the patch is hacky and would have to go through upstream.

3. We go back to using distextract for scripted installations as well as interactive ones, reverting d7640440fb644fde697f62fdff0b55aa3a4d5ef7. This fixes this issue but will result in installation failures for scripted installs without a controlling tty. (It will also add nice progress bars to scripted installs).

4. We do --exclude /boot/efi when running tar, then mkdir -p it by hand afterward. This is incredibly hacky and otherwise essentially functionally equivalent to #1. Like #1, it will fix this issue and has no obvious functional downside, but leaves scripted installs bare-metal POWER8 and POWER9 broken.

5. We patch the file system driver to (pretend to) allow setting times on the mount point. I don't want to do this, since I don't want to solve this in the kernel at RC3 and I don't like it pretending to do things it can't do.

--------

Of these, 1, 3, and 4 are quite easy to implement, but all have some downside. My temptation is to do 4 for 13.0, since it will definitely work but is just lame, then either do #2 or a variant on #3 where distextract notices there is no tty and doesn't try to set up a dialog as a longer-term fix in HEAD. Any thoughts?
Comment 7 Warner Losh freebsd_committer 2021-03-19 20:09:40 UTC
#4 sounds good for 13.0. Of the hacks, it's the most localized and least hacky, imho.

I'd prefer #5, honestly, longer term but I've not thought through all the implictions. We should loop in cem@ since he's been touching that code most recently. Or delphij@ as he has too...
Comment 8 Ryan Moeller freebsd_committer 2021-03-19 20:55:35 UTC
Would it be possible to defer mounting the ESP until after the tarballs have been extracted? We don't actually have anything in /boot/efi in the base archive. My understanding is that it's just there to ensure the mountpoint exists.
Comment 9 Nathan Whitehorn freebsd_committer 2021-03-19 21:15:24 UTC
(In reply to Ryan Moeller from comment #8)

We could delay it, but it's harder and less-localized than the other solutions. It also completely breaks PowerPC and other systems with analagous but non-ESP boot partions.
Comment 10 Warner Losh freebsd_committer 2021-03-19 21:15:55 UTC
(In reply to Ryan Moeller from comment #8)
Oh! I like this idea better, I think, but I don't know how hard it is to do.
Comment 11 Nathan Whitehorn freebsd_committer 2021-03-19 21:29:37 UTC
(In reply to Warner Losh from comment #10)

It's pretty tricky, since it touches code in a lot of places, has to be conditional on platform, and runs a risk of breaking interactive installs just because it is pretty invasive.
Comment 12 Nathan Whitehorn freebsd_committer 2021-03-22 19:31:13 UTC
Related things:

https://reviews.freebsd.org/D29380 <- patch in review here
https://github.com/libarchive/libarchive/issues/1516 <- libarchive issue here
Comment 13 commit-hook freebsd_committer 2021-03-23 13:30:41 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=c2f16c595eb51c6e0cb6ece3f6f078d738019059

commit c2f16c595eb51c6e0cb6ece3f6f078d738019059
Author:     Nathan Whitehorn <nwhitehorn@FreeBSD.org>
AuthorDate: 2021-03-23 13:19:42 +0000
Commit:     Nathan Whitehorn <nwhitehorn@FreeBSD.org>
CommitDate: 2021-03-23 13:29:54 +0000

    Fix scripted installs on EFI systems after default mounting of the ESP.

    Because the ESP mount point (/boot/efi) is in mtree, tar will attempt to
    extract a directory at that point post-mount when the system is installed.
    Normally, this is fine, since tar can happily set whatever properties it
    wants. For FAT32 file systems, however, like the ESP, tar will attempt to
    set mtime on the root directory, which FAT does not support, and tar will
    interpret this as a fatal error, breaking the install (see
    https://github.com/libarchive/libarchive/issues/1516). This issue would
    also break scripted installs on bare-metal POWER8, POWER9, and PS3
    systems, as well as some ARM systems.

    This patch solves the problem in two ways:
    - If stdout is a TTY, use the distextract stage instead of tar, as in
      interactive installs. distextract solves this problem internally and
      provides a nicer UI to boot, but requires a TTY.
    - If stdout is not a TTY, use tar but, as a stopgap for 13.0, exclude
      boot/efi from tarball extraction and then add it by hand. This is a
      hack, and better solutions (as in the libarchive ticket above) will
      obsolete it, but it solves the most common case, leaving only
      unattended TTY-less installs on a few tier-2 platforms broken.

    In addition, fix a bug with fstab generation uncovered once the tar issue
    is fixed that umount(8) can depend on the ordering of lines in fstab in a
    way that mount(8) does not. The partition editor now writes out fstab in
    mount order, making sure umount (run at the end of scripted, but not
    interactive, installs) succeeds.

    PR:             254395
    Reviewed by:    gjb, imp
    MFC after:      3 days
    Differential Revision:  https://reviews.freebsd.org/D29380

 usr.sbin/bsdinstall/partedit/partedit.c | 39 +++++++++++++++++++++++++++++++++
 usr.sbin/bsdinstall/scripts/script      | 30 ++++++++++++++++++-------
 2 files changed, 61 insertions(+), 8 deletions(-)
Comment 14 commit-hook freebsd_committer 2021-03-23 19:22:47 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=4601382e1362352f17a33e4ed38db5dcfe3f6be5

commit 4601382e1362352f17a33e4ed38db5dcfe3f6be5
Author:     Nathan Whitehorn <nwhitehorn@FreeBSD.org>
AuthorDate: 2021-03-23 13:19:42 +0000
Commit:     Nathan Whitehorn <nwhitehorn@FreeBSD.org>
CommitDate: 2021-03-23 19:21:33 +0000

    Fix scripted installs on EFI systems after default mounting of the ESP.

    Because the ESP mount point (/boot/efi) is in mtree, tar will attempt to
    extract a directory at that point post-mount when the system is installed.
    Normally, this is fine, since tar can happily set whatever properties it
    wants. For FAT32 file systems, however, like the ESP, tar will attempt to
    set mtime on the root directory, which FAT does not support, and tar will
    interpret this as a fatal error, breaking the install (see
    https://github.com/libarchive/libarchive/issues/1516). This issue would
    also break scripted installs on bare-metal POWER8, POWER9, and PS3
    systems, as well as some ARM systems.

    This patch solves the problem in two ways:
    - If stdout is a TTY, use the distextract stage instead of tar, as in
      interactive installs. distextract solves this problem internally and
      provides a nicer UI to boot, but requires a TTY.
    - If stdout is not a TTY, use tar but, as a stopgap for 13.0, exclude
      boot/efi from tarball extraction and then add it by hand. This is a
      hack, and better solutions (as in the libarchive ticket above) will
      obsolete it, but it solves the most common case, leaving only
      unattended TTY-less installs on a few tier-2 platforms broken.

    In addition, fix a bug with fstab generation uncovered once the tar issue
    is fixed that umount(8) can depend on the ordering of lines in fstab in a
    way that mount(8) does not. The partition editor now writes out fstab in
    mount order, making sure umount (run at the end of scripted, but not
    interactive, installs) succeeds.

    PR:             254395
    Approved by:    re (gjb)
    Reviewed by:    gjb, imp
    MFC after:      3 days
    Differential Revision:  https://reviews.freebsd.org/D29380

    (cherry picked from commit c2f16c595eb51c6e0cb6ece3f6f078d738019059)

 usr.sbin/bsdinstall/partedit/partedit.c | 39 +++++++++++++++++++++++++++++++++
 usr.sbin/bsdinstall/scripts/script      | 30 ++++++++++++++++++-------
 2 files changed, 61 insertions(+), 8 deletions(-)
Comment 15 commit-hook freebsd_committer 2021-03-23 21:15:14 UTC
A commit in branch releng/13.0 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=3ef46634cd5a6ba0e156d5da095bbee5f63ec1cd

commit 3ef46634cd5a6ba0e156d5da095bbee5f63ec1cd
Author:     Nathan Whitehorn <nwhitehorn@FreeBSD.org>
AuthorDate: 2021-03-23 13:19:42 +0000
Commit:     Nathan Whitehorn <nwhitehorn@FreeBSD.org>
CommitDate: 2021-03-23 19:22:20 +0000

    Fix scripted installs on EFI systems after default mounting of the ESP.

    Because the ESP mount point (/boot/efi) is in mtree, tar will attempt to
    extract a directory at that point post-mount when the system is installed.
    Normally, this is fine, since tar can happily set whatever properties it
    wants. For FAT32 file systems, however, like the ESP, tar will attempt to
    set mtime on the root directory, which FAT does not support, and tar will
    interpret this as a fatal error, breaking the install (see
    https://github.com/libarchive/libarchive/issues/1516). This issue would
    also break scripted installs on bare-metal POWER8, POWER9, and PS3
    systems, as well as some ARM systems.

    This patch solves the problem in two ways:
    - If stdout is a TTY, use the distextract stage instead of tar, as in
      interactive installs. distextract solves this problem internally and
      provides a nicer UI to boot, but requires a TTY.
    - If stdout is not a TTY, use tar but, as a stopgap for 13.0, exclude
      boot/efi from tarball extraction and then add it by hand. This is a
      hack, and better solutions (as in the libarchive ticket above) will
      obsolete it, but it solves the most common case, leaving only
      unattended TTY-less installs on a few tier-2 platforms broken.

    In addition, fix a bug with fstab generation uncovered once the tar issue
    is fixed that umount(8) can depend on the ordering of lines in fstab in a
    way that mount(8) does not. The partition editor now writes out fstab in
    mount order, making sure umount (run at the end of scripted, but not
    interactive, installs) succeeds.

    PR:             254395
    Approved by:    re (gjb)
    Reviewed by:    gjb, imp
    MFC after:      3 days
    Differential Revision:  https://reviews.freebsd.org/D29380

    (cherry picked from commit c2f16c595eb51c6e0cb6ece3f6f078d738019059)
    (cherry picked from commit 4601382e1362352f17a33e4ed38db5dcfe3f6be5)

 usr.sbin/bsdinstall/partedit/partedit.c | 39 +++++++++++++++++++++++++++++++++
 usr.sbin/bsdinstall/scripts/script      | 30 ++++++++++++++++++-------
 2 files changed, 61 insertions(+), 8 deletions(-)