Created attachment 223341 [details] unit test After I removed a vdev from my root pool, I can no longer set the bootfs property. This makes Boot Environments unusable. I'm attaching a unit test script to demonstrate the problem. My output is: ==================================================================================== # ./zfs_set_bootfs_after_vdev_removal.sh cannot set property for 'test_remove': operation not supported on this type of pool FAIL NAME PROPERTY VALUE SOURCE test_remove bootfs test_remove/root0 local pool: test_remove state: ONLINE remove: Removal of vdev 1 copied 276K in 0h0m, completed on Tue Mar 16 10:18:08 2021 216 memory used for removed device mappings config: NAME STATE READ WRITE CKSUM test_remove ONLINE 0 0 0 md0 ONLINE 0 0 0 errors: No known data errors
Maybe that's because loader does not support booting from such a pool? Just a speculation on my part. Removing a vdev means that now there is vdev indirection and other complications that require explicit handling.
mm@ filed an upstream issue for this. Thanks, Martin! https://github.com/openzfs/zfs/issues/11762 Andriy: Fair question. My workstation, running main, boots from such a pool. That's how I found the issue. My old SSD was failing, so I migrated to a new one. For historical reasons, my pool was split across two partitions on the old disk. I wanted to use only one on the new disk, so I didn't use mirroring for the migration. I simply added and removed vdevs.
It looks like VDEV_TYPE_INDIRECT is handled at stand/libsa/zfs/zfsimpl.c The code originates in the device removal feature: https://github.com/openzfs/zfs/commit/a1d477c24c7badc89c60955995fd84d311938486 By modifying the vdev_is_bootable() function, setting bootfs is not allowed anymore. I think this is an easy fix and really not necessary on FreeBSD. Thomas Soome has been dealing with support of indirect vdevs at end of 2019, see: https://svnweb.freebsd.org/base?view=revision&revision=355786 It would be great if Thomas would tell us some more about the support of removed vdevs (=indirect vdevs).
This is the most important commit I am referring to: https://svnweb.freebsd.org/base?view=revision&revision=350772