Bug 227808 - FreeBSD-11.1-STABLE-amd64-bootonly.iso fails to mount root w/ error 19 on HP DL360 G8 and G9 via PXE
Summary: FreeBSD-11.1-STABLE-amd64-bootonly.iso fails to mount root w/ error 19 on HP ...
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 11.1-STABLE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-bugs mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-04-27 15:21 UTC by Rick
Modified: 2018-09-07 14:28 UTC (History)
4 users (show)

See Also:


Attachments
gems listed after failure, shows gems from previous installation (26.00 KB, image/jpeg)
2018-04-27 15:22 UTC, Rick
no flags Details
mountroot fails with error 19 (938.95 KB, image/jpeg)
2018-04-27 15:29 UTC, Rick
no flags Details
FreeBSD 11.2-PRERELEASE hang on initialize cd0 (33.95 KB, image/png)
2018-04-27 17:30 UTC, Rick
no flags Details
FreeBSD 11.2-PRERELEASE moonroot error 19 (31.76 KB, image/png)
2018-04-27 17:31 UTC, Rick
no flags Details
geoms on disk (115.22 KB, image/jpeg)
2018-04-30 12:57 UTC, Rick
no flags Details
11.2-PRERELEASE mountroot failure w/ geoms listed (40.45 KB, image/png)
2018-05-02 10:37 UTC, Rick
no flags Details
11.2-PRERELEASE mount root failure on DL360G9 w/ geoms (52.13 KB, image/png)
2018-05-02 11:30 UTC, Rick
no flags Details
11.2-BETA2-amd64-UEFI: Media panics loading the kernel and root fs citing non-existent init (61.62 KB, image/png)
2018-06-05 13:54 UTC, Rick
no flags Details
11.2-BETA2-amd64-BIOS: Media panics loading the kernel and root fs citing non-existent init (38.57 KB, image/png)
2018-06-05 13:56 UTC, Rick
no flags Details
11.2-PRERELEASE-amd64-UEFI: mountroot error 19 (46.02 KB, image/png)
2018-06-05 13:57 UTC, Rick
no flags Details
11.2-PRERELEASE-amd64-BIOS: mountroot error 19 (36.04 KB, image/png)
2018-06-05 13:58 UTC, Rick
no flags Details
12-CURRENT-amd64-UEFI: mountroot error 19 (50.93 KB, image/png)
2018-06-05 13:59 UTC, Rick
no flags Details
12-CURRENT-amd64-BIOS: mountroot error 19 (37.01 KB, image/png)
2018-06-05 14:00 UTC, Rick
no flags Details
11.2-PRERELEASE-i386-UEFI: Could not boot image (via iPXE) (44.34 KB, image/png)
2018-06-05 14:01 UTC, Rick
no flags Details
12-CURRENT-i386-BIOS: too many address bits (42.28 KB, image/png)
2018-06-05 14:02 UTC, Rick
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Rick 2018-04-27 15:21:06 UTC
FreeBSD-11.1-STABLE-amd64-bootonly.iso fails to boot citing mountroot error 19. ISO is PXE booted from a network server. Hardware is HP DL360G8, dual E5-2640 2.5GHz, 128GB RAM, P420i Smart Array RAID controller.

Reproduced by:

* Power on system
* Press F12 to boot to network, which sends ISO to host
* Allow Beastie countdown to expire

Expected results are that ISO continues to boot permitting manual OS installation. Error first noted on April 27, 2018.
Comment 1 Rick 2018-04-27 15:22:18 UTC
Created attachment 192847 [details]
gems listed after failure, shows gems from previous installation
Comment 2 Glen Barber freebsd_committer 2018-04-27 15:25:22 UTC
Could you try the most recent development snapshot of what is now 11.2-PRERELEASE?  New images from yesterday are available at: https://download.freebsd.org/ftp/snapshots/ISO-IMAGES/11.2/
Comment 3 Rick 2018-04-27 15:29:06 UTC
Created attachment 192848 [details]
mountroot fails with error 19
Comment 4 Rick 2018-04-27 17:30:03 UTC
I repeated the exercise 3x with FreeBSD-11.2-PRERELEASE-amd64-20180420-r332802-bootonly.iso and observed the following results in order of occurrence:

1) system hung initializing cd0 (forced cold reset) \
   attached image: freebsd-11.2-prerelease-bootonly_hang
2) mountroot error 19 (attached image: freebsd-11.2-prerelease_moutroot_error)
3) mountroot error 19 (attached image: freebsd-11.2-prerelease_moutroot_error)

FWIW, firmware is configured to boot legacy BIOS mode.
Comment 5 Rick 2018-04-27 17:30:39 UTC
Created attachment 192854 [details]
FreeBSD 11.2-PRERELEASE hang on initialize cd0
Comment 6 Rick 2018-04-27 17:31:12 UTC
Created attachment 192855 [details]
FreeBSD 11.2-PRERELEASE moonroot error 19
Comment 7 Glen Barber freebsd_committer 2018-04-30 12:11:12 UTC
(In reply to vmiller from comment #6)
> Created attachment 192855 [details]
> FreeBSD 11.2-PRERELEASE moonroot error 19

What is printed if you type '? [enter]' at the mountroot prompt?
Comment 8 Rick 2018-04-30 12:57:17 UTC
Created attachment 192929 [details]
geoms on disk

The list of available geoms when receiving mountroot error 19. This particular screenshot was taken when booted to stable/11 as of March and not with the recent prerelease ISO, which I suspect to return the same result. These geoms are from a previous working 10.3 installation.
Comment 9 Rick 2018-05-02 10:37:53 UTC
Created attachment 192988 [details]
11.2-PRERELEASE mountroot failure w/ geoms listed

This image demonstrates output from "?" immediately following the mountroot error 19 usingg the 11.2-PRERELEASE snapshot bootonly ISO.
Comment 10 Rick 2018-05-02 11:30:34 UTC
Created attachment 192991 [details]
11.2-PRERELEASE mount root failure on DL360G9 w/ geoms

This image demonstrates the mountroot failure on the DP DL360G9 as well. Many of the geoms listed are from a current on disk 10.3 installation.
Comment 11 Rick 2018-06-05 13:52:04 UTC
In an attempt to boot HP DL360 G9 into bootable media, which has proven problematic beyond FreeBSD 10.3, tests were performed on a total of 5 ISO files (from the most recent snapshots) in UEFI and legacy BIOS boot modes to ascertain the exhibited behavior. For the amd64 bootonly ISOs, the expected behavior is media suitable to boot and install FreeBSD of that version. All ISOs, with the exception of the 11.2 amd64 mfs, are project released ISOs. The 11.2 BETA2 amd64 ISO is custom built with a manually generated root.part from /rescue specified as MFS_IMAGE to be injected into the kernel. Described here are the behaviors exhibited in each boot mode.

FreeBSD-11.2-PRERELEASE-amd64-bootonly (test1)

* UEFI: mountroot error 19 (screenshot: test1_FreeBSD-11.2-PRERELEASE-bootonly_fail)
* BIOS: mountroot error 19 (screenshot: test1_bios_FreeBSD-11.2-PRERELEASE-bootonly_fail)

FreeBSD-11.2-PRERELEASE-i386-bootonly (test3)

* UEFI: Could not boot image error via iPXE (screenshot: test3_FreeBSD-11.2-PRERELASE_i386-fail)
* BIOS: Too many address bits (screenshot: test3_bios_FreeBSD-11.2-PRERELASE_i386-fail)

FreeBSD-12-CURRENT-amd64-bootonly (test2)

* UEFI: mountroot error 19 (screenshot: test2_FreeBSD-12-CURRENT-bootonly_fail)
* BIOS: mountroot error 19 (screenshot: test2_bios_FreeBSD-12-CURRENT-bootonly_fail)

FreeBSD-12-CURRENT-i386-bootonly (test4)

* UEFI: Could not boot image error via iPXE (screenshot: test4_FreeBSD-12-CURRENT-i386-fail)
* BIOS: Too many address bits (screenshot: test4_bios_FreeBSD-12-CURRENT-i386-fail)

FreeBSD-11.2-BETA2-amd64-bootonly-mfs (test0)

* UEFI: Media panics loading the kernel and root fs citing non-existent init
  (screenshot: test0_FreeBSD-11.2-md_root-fail)
* BIOS: Media panics loading the kernel and root fs citing non-existent init
  (screenshot: test0_bios_FreeBSD-11.2-md_root-fail)

This is custom built media including a MFS_IMAGE generated from /rescue. It is suspected that the error here is a result of an incomplete root.part rather than faulty ISO image.

This problem is suspected to have existed with FreeBSD 10.3 also, which can be worked around by building an MFS-enabled ISO. The process that created the 10.3 image differs from that used to create the same for 11.2 suggesting discrepancies in the build of MFS-enabled ISOs.
Comment 12 Rick 2018-06-05 13:54:44 UTC
Created attachment 194019 [details]
11.2-BETA2-amd64-UEFI: Media panics loading the kernel and root fs citing non-existent init
Comment 13 Rick 2018-06-05 13:56:03 UTC
Created attachment 194021 [details]
11.2-BETA2-amd64-BIOS: Media panics loading the kernel and root fs citing non-existent init
Comment 14 Rick 2018-06-05 13:57:51 UTC
Created attachment 194022 [details]
11.2-PRERELEASE-amd64-UEFI: mountroot error 19
Comment 15 Rick 2018-06-05 13:58:35 UTC
Created attachment 194023 [details]
11.2-PRERELEASE-amd64-BIOS: mountroot error 19
Comment 16 Rick 2018-06-05 13:59:36 UTC
Created attachment 194024 [details]
12-CURRENT-amd64-UEFI: mountroot error 19
Comment 17 Rick 2018-06-05 14:00:10 UTC
Created attachment 194025 [details]
12-CURRENT-amd64-BIOS: mountroot error 19
Comment 18 Rick 2018-06-05 14:01:28 UTC
Created attachment 194026 [details]
11.2-PRERELEASE-i386-UEFI: Could not boot image (via iPXE)
Comment 19 Rick 2018-06-05 14:02:20 UTC
Created attachment 194027 [details]
12-CURRENT-i386-BIOS: too many address bits
Comment 20 Rick 2018-07-19 17:36:49 UTC
This functionality is tested by booting the HP DL360g9 into iPXE under UEFI and booting the project built bootonly ISO:

1) Boot iPXE
2) ctrl-b to break into iPXE prompt
3) ifconf or dhcp (to configure network)
4) sanboot --drive 0x80 --no-describe http://$url_to_iso

This iPXE config reference seems to illustrate ISOs are loaded via correct syntax of the sanboot command: https://gist.github.com/robinsmidsrod/2234639

The above procedure on the project ISO fails w/ mountroot error 19. However, the ISO boots correctly when mounted as virtual media on the platform accomplished with:

1) Login to iLO
2) attach virtual media CD/DVD (using URL from above)
3) Reboot system into virtual media

From iPXE, the ISO can be loaded where we drop to the loader prompt and inspect the environment. Here are some seemingly relevant environment bits:

...
bootfile=kernel
comconsole_pcidev=
comconsole_port=1016
comconsole_speed=9600
console=efi
currdev=cd0:
efi-version=2.40
...
loaddev=cd0:

vfs.root.mountfrom and vfs.root.mountfrom.options are unset when breaking out to the loader prompt. Setting manually and executing boot exhibits the same behavior (mountroot() error 19).
Comment 21 doktornotor 2018-08-01 20:51:15 UTC
(In reply to Glen Barber from comment #2)

This has been broken for *years*, cf. Bug #199197, Bug #179312 or Bug #169919. Never worked since 9.0 was released.

Bug #199197 has perfectly good info about what's screwed here in case someone finally cares enough to fix it.
Comment 22 Rick 2018-09-07 14:28:25 UTC
FWIW, one can workaround this problem by deploying a bootonly ISO configured with an MFS_IMAGE. EFI_STAGING_SIZE may need adjusting to accommodate larger objects; It defaults to 64. EFI_STAGING_SIZE can be defined within make.conf, release.conf, or on the command line. While not illustrated below, EFI_STAGING_SIZE is implicitly defined specifying a make.conf that includes the knob in the environment via MAKE_CONF.

MFS_IMAGE is necessary because the project’s bootonly ISO fails to PXE boot at
mountroot() w/ error 19 indicating the boot media cannot be located. The hardware platform is HP DL360g9 in UEFI boot mode. MFS_IMAGE implies a MD_ROOT root filesystem.

The build system doesn't seem provide a mechanism to build a default MFS_IMAGE and is therefore done outside the context of a build. The following procedure builds a binary distribution from the source tree in it's default form (including the default bootonly.iso). It rebuilds the bootonly.iso after generating a root partition and MDROOT kernel.

# Create CHROOTDIR w/ GENERIC kernel (also builds binary release)
root # cd ~/projects/freebsd
root # git checkout releng/11.2
root # release/release.sh -c release/release.conf

# Create root.part from ${OBJDIR} bootonly
root # cd ${CHROOTDIR}/usr/obj/usr/src/release
root # rm bootonly/boot/kernel/kernel
root # rm bootonly/boot/geom_md.ko
root # echo "/dev/ufs/FreeBSD_Install / ufs rw 0 0" > bootonly/etc/fstab
root # makefs -B little -o label=FreeBSD_Install root.part bootonly/
root # cd .. && mkdir -p sys/MDROOT
root # mv release/root.part sys/MDROOT/

# Build MDROOT kernel
root # cat ${CHROOTDIR}/usr/src/sys/amd64/conf/MDROOT
include GENERIC
ident MDROOT

makeoptions     MFS_IMAGE=root.part
root # eval chroot ${CHROOTDIR} env KERNCONF=MDROOT make \ 
    -C /usr/src buildkernel

# Build bootonly ISO w MDROOT kernel
root # mount -t devfs devs ${CHROOTDIR}/dev
root # chflags -R noschg ${CHROOTDIR}/usr/obj/usr/src/release/bootonly && \
    rm -rf ${CHROOTDIR}/usr/obj/usr/src/bootonly
root # eval chroot ${CHROOTDIR} env KERNCONF=MDROOT \ 
    NOPKG=1 NOSRC=1 NODOC=1 NOPORTS=1 make -C /usr/src/release bootonly.iso