The story: A disk faulted. It was the first disk in the chassis (so it is the one the controller selects to boot). I added a spare (as the 2nd disk in the chassis), and replaced the faulted disk with the spare. Then I got the system to hang for unrelated reasons, removed the bad disk, and rebooted. (unrelated reasons: I added some zpool v16 zfs v4 disks I created on FreeBSD, tried mouting in Linux with zfs-fuse, which told me a device was missing, and then put them back in the FreeBSD and imported, which also said a device was missing, and then "zpool import -m -F -N" or something similar made it hang. parted on Linux said the disks were ext3, which may be related to the problem.) And on boot, to my surprise, it complained saying I can only boot from mirrors, raidz, etc. (a list which did not include spares). The second disk was defective in a different way, so I didn't try putting it first in the chassis, but that is a workaround, not a bug fix. Fix: Workaround was to boot the DVD, load the zfs modules, and then "zpool detach" the bad disk (converting the spare to a normal mirror), and detach the other root disk, which I wanted to remove anyway (to upgrade firmware). How-To-Repeat: Create a virtual machine with disks in this order: boot mirror disk 1 spare disk boot mirror disk 2 Boot it up. (using gpt slices, matching what I set up before). Replace the boot mirror disk 1 with the spare with: # gpart create -s gpt da1 # gpart add -b 34 -s 128 -t freebsd-boot da1 # gpart add -b 129024 -s 32768 -t freebsd-swap da1 # gpart add -b 258048 -s 8386560 -t freebsd-zfs -l sp_root0 da1 # zpool add zroot spare gpt/sp_root0 # zpool replace zroot gpt/root0 gpt/sp_root0 # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1 Wait for resilver. $ zpool status zroot ------------------------------------------------- pool: zroot state: ONLINE scan: resilvered 1.33G in 0h5m with 0 errors on Wed Apr 18 19:42:19 2012 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 spare-0 ONLINE 0 0 0 gpt/root0 ONLINE 0 0 0 gpt/sp_root0 ONLINE 0 0 0 gpt/root1 ONLINE 0 0 0 spares 665168513187118073 INUSE was /dev/gpt/sp_root0 errors: No known data errors ------------------------------------------------- # shutdown -p now Remove disk 1 and 3 (the old boot disks). Start up again. During boot, you get this output: ------------------------------------------------- ZFS: can only boot from disk, mirror, raidz1, raidz2 and raidz3 vdevs ZFS: i/o error - all block copies unavailable ZFS: can't read MOS ZFS: unexpected object set type 0 ZFS: unexpected object set type 0 FreeBSD/x86 boot Default: zroot:/boot/kernel/kernel boot: ZFS: unexpected object set type 0 FreeBSD/x86 boot Default: zroot:/boot/kernel/kernel boot: -------------------------------------------------
Responsible Changed From-To: freebsd-bugs->freebsd-fs Over to maintainer(s).
Actually, I tried booting with all disks attached, and it still failed the same way. And then tried again with the spare removed, which was the same.
For bugs matching the following criteria: Status: In Progress Changed: (is less than) 2014-06-01 Reset to default assignee and clear in-progress tags. Mail being skipped