On June 29, I updated an amd64 system with a GPT ZFS root, to FreeBSD 8.4-RELEASE-p13 #0 r268016: Sun Jun 29 12:58:11 UTC 2014 I rebooted and this worked without issue. On Sep 9, 2014, I upgraded both pools to the latest for 8.4-RELEASE and so the version property no longer applies. $ zpool get version NAME PROPERTY VALUE SOURCE pool0 version - default pool1 version - default $ zpool status pool: pool0 state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM pool0 ONLINE 0 0 0 da0p3 ONLINE 0 0 0 errors: No known data errors pool: pool1 state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM pool1 ONLINE 0 0 0 da1p1 ONLINE 0 0 0 errors: No known data errors $ gpart show => 34 312477629 da0 GPT (149G) 34 128 1 freebsd-boot (64k) 162 67108864 2 freebsd-swap (32G) 67109026 245368637 3 freebsd-zfs (117G) => 34 312477629 da1 GPT (149G) 34 312477629 1 freebsd-zfs (149G) I applied the recommended update to gptzfsboot and the pmbr as follows gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0 On Sep. 16, I rebooted the system to find that it no longer booted and the system froze right after the first "/" symbol. I.e. the "spinner" did not spin. Even CTRL-ALT-DEL was insufficient to break it out of the frozen state, a reset or power cycle was required. We used a bootable 8.4 USB image to get access to the system after it became clear there was no way to boot from the internal drives. Using the fixit shell, we determined the pools were intact and undamaged and reapplied the bootcode again from the USB image as a speculative measure, all to no effect. The underlying block devices are 3ware RAID controller volumes as follows: $ egrep da[01] /var/run/dmesg.boot da0 at twa0 bus 0 scbus0 target 0 lun 0 da0: <AMCC 9650SE-4LP DISK 3.08> Fixed Direct Access SCSI-5 device da0: 100.000MB/s transfers da0: 152577MB (312477696 512 byte sectors: 255H 63S/T 19450C) da1 at twa0 bus 0 scbus0 target 1 lun 0 da1: <AMCC 9650SE-4LP DISK 3.08> Fixed Direct Access SCSI-5 device da1: 100.000MB/s transfers da1: 152577MB (312477696 512 byte sectors: 255H 63S/T 19450C) At this point, we've configured a USB stick to handle the boot phase and so that's the workaround, but this seems like quite an extreme failure mode for gptzfsboot that could probably do with some attention. I've not yet attempted to replicate this on any other system and here's the CPU/RAM for this one: CPU: Intel(R) Xeon(R) CPU E5410 @ 2.33GHz (2327.51-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x10676 Family = 6 Model = 17 Stepping = 6 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0xce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1> AMD Features=0x20100800<SYSCALL,NX,LM> AMD Features2=0x1<LAHF> TSC: P-state invariant real memory = 17716740096 (16896 MB) avail memory = 16535756800 (15769 MB) It's a TYAN Tempest i5100X S5375 motherboard with the following dmidecode details for the BIOS: Vendor: American Megatrends Inc. Version: 080014 Release Date: 01/30/2008 So, where is the boot process likely to have hung if it was so early and what does it mean if CTRL-ALT-DEL is ineffective?
This forum posting sounds very close to what I found. https://forums.freebsd.org/viewtopic.php?&t=42419 So I'll also point out I have the following in /boot.config -hD 115200 Obviously the serial console worked fine with the prior version of gptzfsboot (from 8.2)
I've someone e-mail me indicating they had the same issue and that using the gptzfsboot code from the 9.2 release resolved the issue. I've not yet had a chance to try that, but as far as I can tell the only difference in zfsboot between 8.4 and 9.2 were a couple of lines relating to serial console handling, suggesting like the forum comment that gptzfsboot is hanging in some serial console related code. So, if this is the root of the issue I would guess that some serial console code for boot loaders changed between 8.2 and 8.4, but that zfsboot, at least wasn't updated until 9.2 or earlier.
http://freebsd.1045724.n5.nabble.com/Upgrade-of-RELENG-8-ZFS-boot-pool-leads-to-unbootable-system-td5774091.html also talks about the /boot.config line interfering with booting.
Following a suggestion from Matt Reimer, I've updated the bootcode gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0 but using a gptzfsboot from FreeBSD 9.2-release. So my immediate problem is resolved, but it does mean there's a bug in the gptzfsboot for FreeBSD 8.4 at least.
So there's a bug here still.
(In reply to mark from comment #2) > chance to try that, but as far as I can tell the only difference in zfsboot > between 8.4 and 9.2 were a couple of lines relating to serial console > handling, suggesting like the forum comment that gptzfsboot is hanging in > some serial console related code. The difference between 8.x and 9.x+ is very big. 9.x includes this commit and many other fixes https://svnweb.freebsd.org/base?view=revision&revision=243243
Ok, I was looking at the difference between 8.4 and 9.2 just for the zfsboot directory. There is more to gptzfsboot than just zfsboot and I'd didn't look at all the contributions separately. $ svn diff https://svn0.eu.freebsd.org/base/releng/8.4/sys/boot/i386/zfsboot https://svn0.eu.freebsd.org/base/releng/9.2/sys/boot/i386/zfsboot Index: zfsboot.c =================================================================== --- zfsboot.c (.../8.4/sys/boot/i386/zfsboot) (revision 273837) +++ zfsboot.c (.../9.2/sys/boot/i386/zfsboot) (revision 273837) @@ -54,7 +54,7 @@ #define NOPT 14 #define NDEV 3 -#define BIOS_NUMDRIVES 0x475 +#define BIOS_NUMDRIVES 0x475 #define DRV_HARD 0x80 #define DRV_MASK 0x7f @@ -489,7 +489,12 @@ * will find any other available pools and it may fill in missing * vdevs for the boot pool. */ - for (i = 0; i < *(unsigned char *)PTOV(BIOS_NUMDRIVES); i++) { +#ifndef VIRTUALBOX + for (i = 0; i < *(unsigned char *)PTOV(BIOS_NUMDRIVES); i++) +#else + for (i = 0; i < MAXBDDEV; i++) +#endif + { if ((i | DRV_HARD) == *(uint8_t *)PTOV(ARGS)) continue; @@ -780,8 +785,10 @@ } ioctrl = OPT_CHECK(RBX_DUAL) ? (IO_SERIAL|IO_KEYBOARD) : OPT_CHECK(RBX_SERIAL) ? IO_SERIAL : IO_KEYBOARD; - if (ioctrl & IO_SERIAL) - sio_init(115200 / comspeed); + if (ioctrl & IO_SERIAL) { + if (sio_init(115200 / comspeed) != 0) + ioctrl &= ~IO_SERIAL; + } } if (c == '?') { dnode_phys_t dn; Index: Makefile =================================================================== --- Makefile (.../8.4/sys/boot/i386/zfsboot) (revision 273837) +++ Makefile (.../9.2/sys/boot/i386/zfsboot) (revision 273837) @@ -16,7 +16,6 @@ CFLAGS= -DBOOTPROG=\"zfsboot\" \ -O1 \ - -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3 \ -DBOOT2 \ -DSIOPRT=${BOOT_COMCONSOLE_PORT} \ -DSIOFMT=${B2SIOFMT} \ @@ -78,7 +77,7 @@ SRCS= zfsboot.c -.if ${MACHINE_ARCH} == "amd64" +.if ${MACHINE_CPUARCH} == "amd64" beforedepend zfsboot.o: machine CLEANFILES+= machine machine: @@ -86,3 +85,7 @@ .endif .include <bsd.prog.mk> + +# XXX: clang integrated-as doesn't grok .codeNN directives yet +CFLAGS.zfsldr.S= ${CLANG_NO_IAS} +CFLAGS+= ${CFLAGS.${.IMPSRC:T}}
With OpenZFS and FreeBSD 13.0-RELEASE or greater, is this still an issue?