Bug 211186 - Cannot boot mirrored zpool under 11.0 BETA1
Summary: Cannot boot mirrored zpool under 11.0 BETA1
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: 11.0-BETA1
Hardware: amd64 Any
: --- Affects Many People
Assignee: freebsd-bugs (Nobody)
Depends on:
Reported: 2016-07-18 08:42 UTC by Dan MacDonald
Modified: 2016-07-20 22:39 UTC (History)
2 users (show)

See Also:

FreeBSD 11.0 beta1 gptzfsboot errors (221.27 KB, image/jpeg)
2016-07-18 08:42 UTC, Dan MacDonald
no flags Details
error trying to boot 10.3 amd64 in a mirrored ZFS config (175.17 KB, image/jpeg)
2016-07-19 17:07 UTC, Dan MacDonald
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Dan MacDonald 2016-07-18 08:42:11 UTC
Created attachment 172631 [details]
FreeBSD 11.0 beta1 gptzfsboot errors

I installed FreeBSD-11.0-BETA1-amd64-memstick.img on my Gigabyte N3150N D3V via a USB memory stick. When it came to partitioning I choose to install to a mirrored ZFS pool. FBSD seemed to install fine but when I rebooted out of the installer and removed the memory stick I installed from I got the error shown in the attached screenshot. Boot doesn't even get as far as the Beastie bootloader menu. Note that I only used standard partitioning and installation options that are available within bsdinstall - I didn't drop to the terminal to run extra zfs commands etc.

I noticed that there were three gptzfsboot errors being highlighted so I wondered what would happen if I re-inserted my installation USB drive and tried booting again? Lo and behold, FreeBSD 11 boots from my mirrored ZFS pool but only if I have my USB drive inserted.
Comment 1 Steven Hartland freebsd_committer 2016-07-18 11:15:58 UTC
That sounds like the boot blocks weren't written.

Can you try installing the boot block on all devices e.g.
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1
Comment 2 Dan MacDonald 2016-07-18 18:43:04 UTC
Hi Steven

I ran:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1

and that fixed booting w/o having my USB drive inserted for the next boot but unfortunately it was for the next boot only.

After one successful boot, I thought it was fixed so I powered off and unplugged one drive from the board but I got a very similar (prob the same) error as the one in the screenshot when I powered on. I powered down again and tried just having the other drive attached only to get the same result/errors.

I expected that when I plugged both the HDs back in it would boot but it didn't and now I can't even boot when I have both HDs and the USB drive attached so my original workaround has stopped working too.

The error screen in the screenshot and the one I'm seeing now both seem to be expecting to have three drives attached for some reason - it looks like it is scanning for three to me anyway and I presume that's because of my config during installation?

When I had Arch installed to a mirrored ZFS pool and booting with grub on this same machine I was able to unplug either drive and have it successfully boot so its not a hardware prob.
Comment 3 Steven Hartland freebsd_committer 2016-07-18 19:24:26 UTC
If that worked are you sure that both disks are configured as the boot devices in the BIOS?

I've seen it before that attaching a USB boot device messed with the BIOS boot options rendering it unable to boot the next time.

Another potential issue is if the BIOS has a RAID option that's enabled it could be corrupting the boot blocks somehow, but that's a long shot.

On final interesting thing would be to compare the boot blocks to see if something has corrupted them, that and output from gpart show.
Comment 4 Dan MacDonald 2016-07-19 07:03:26 UTC
Hi Steven

Yes, my HDs are configured as the first two boot devices in the BIOS.

My BIOS doesn't have any RAID features. I can basically enable/disable individual ports, choose the SATA gen (1,2 or 3) or disable onboard SATA entirely.

To clarify the errors a bit, the BIOS drives bit of the error screen varies between 3 and 4 drives depending on if I have the USB drive inserted but it shows 3 gptzfsboot lba errors every time. I'd imagine the 4th 'BIOS drive' is the memory card reader that's integrated into the case. That's attached to  USB port on the mobo but I've not had anything inserted into it whilst installing or doing my tests.

The drives are 2x 4 TB 3.5" Samsung SATAIII drives that have barely been used yet. I performed extended/full tests on them both with smartctl under Linux recently and they both showed no errors.
Comment 5 Dan MacDonald 2016-07-19 17:06:00 UTC
I've tried installing FreeBSD-10.3-RELEASE-amd64-memstick.img on the same machine in a mirrored ZFS config now but I'm getting the same or very similar error - see the attached screenshot. The difference being that it is only showing 2x gptzfsboot errors instead of the three I see when trying to boot 11beta.

I tried running those two gpart commands from the installer shell and they claimed to have installed the botloader to both disks but I've not been able to get 10.3 to boot at all - not when leaving the USB drive in or anything.
Comment 6 Dan MacDonald 2016-07-19 17:07:33 UTC
Created attachment 172730 [details]
error trying to boot 10.3 amd64 in a mirrored ZFS config
Comment 7 Andriy Gapon freebsd_committer 2016-07-19 18:30:35 UTC
Just an observation that error 49 (0x31 aka 31h) seems to mean "no media in drive":
So it could be a benign error from ZFS probing of the card reader, but it could also be a contributor to the later BTX crash.  I think that the latter is not very likely though.

Debugging such early crashes is quite hard.
Comment 8 Steven Hartland freebsd_committer 2016-07-19 19:50:45 UTC
Does the machine support EFI? If so have you tried an EFI install?
Comment 9 Dan MacDonald 2016-07-20 19:41:57 UTC
Yes it supports EFI and BIOS booting.

I've just tried installing 11b again to a mirrored ZFS pool, this time booting in UEFI mode and using the UEFI+BIOS GPT option and it works. I have tested removing one drive at a time and it boots fine from either (albeit with a degraded pool status, of course).

I know 11 is scheduled to be released in a couple of months time so it seems silly for me to install 10.3 now so if I install 11b will I be able to easily upgrade in-place to 11.0 stable? I'm new to FBSD.
Comment 10 Steven Hartland freebsd_committer 2016-07-20 22:39:53 UTC
Yep that will be easy, either by the binary updater or via source if you want to do that.