Hello again. First let me link an old issue from long time ago: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=144214 I have an issue booting host due to gang blocks again. In this case however we are talking about feature, though. In OpenZFS 2.4.0 (introduced in FreeBSD 15) new feature was added to allow bigger gang blocks - dynamic_gang_header. When this feature becomes active booting from pool is impossible with our loader, as it refuses to work on pool with that feature enabled. So: 1. Create a pool with 15-STABLE or later (feature is enabled by default). 2. Do some fragmentation. In my case pool was 60% full, though I used zfs rewrite a lot. 3. Can't boot. I don't think any feasible recovery is possible this time, as once feature becomes active pool becomes unbootable, but there's no way to check which files are actually need to be rewritten to eliminate gang blocks, any fragmented file, anywhere on fs can cause this.
We need to either disable this feature by default, or teach the loader about this. Otherwise pools created with the 15.0 installer will become unbootable.
(In reply to Mark Johnston from comment #1) A middle-ground would be to disallow enabling the feature if bootfs is set and to disallow setting bootfs if the feature is enabled. But I am not sure if every boot pool actually has bootfs set; I recall some people using, against best practices, of course, the root dataset as the boot dataset. P.S. I long wanted to disqualify pools without explicitly set bootfs from booting, but that's harder to do than I initially imagined and it's a different topic altogether.
A tentative patch is here: https://reviews.freebsd.org/D53578 I am not sure whether to include it in 15.0. If not, we have to 1) disable the feature in newly created pools 2) warn users who have already created installations using 15.0-BETA etc.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=4d6801a6b5bdd4d055a00484a743cb4ada659669 commit 4d6801a6b5bdd4d055a00484a743cb4ada659669 Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2025-11-06 16:00:50 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2025-11-06 16:02:33 +0000 stand: Teach the zfs loader about dynamic gang headers There is a pool feature, dynamic_gang_header, that is enabled by default in new pools. When this feature is active, gang headers may be larger than 512 bytes. The loader needs to be taught to cope with that. Try using the vdev ashift to pick the gang block header size. If the checksum fails, fall back to the old gang block header size. This is based on a patch by Paul Dagnelie, with testing, bug-fixing and some simplifications from me. PR: 289690 Co-authored by: Paul Dagnelie <paul.dagnelie@klarasystems.com> Reviewed by: imp MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D53578 stand/libsa/zfs/zfsimpl.c | 67 ++++++++++++++++++++++++++++++++++++--------- sys/cddl/boot/zfs/zfsimpl.h | 15 ++-------- 2 files changed, 56 insertions(+), 26 deletions(-)
Still undecided about whether to include this patch in 15.0. If anyone is testing the patch, please report your findings.
A commit in branch stable/15 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=ae17780c717547a7b52cc593dca8867c0e14f9ef commit ae17780c717547a7b52cc593dca8867c0e14f9ef Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2025-11-06 16:00:50 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2025-11-13 13:48:59 +0000 stand: Teach the zfs loader about dynamic gang headers There is a pool feature, dynamic_gang_header, that is enabled by default in new pools. When this feature is active, gang headers may be larger than 512 bytes. The loader needs to be taught to cope with that. Try using the vdev ashift to pick the gang block header size. If the checksum fails, fall back to the old gang block header size. This is based on a patch by Paul Dagnelie, with testing, bug-fixing and some simplifications from me. PR: 289690 Co-authored by: Paul Dagnelie <paul.dagnelie@klarasystems.com> Reviewed by: imp MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D53578 (cherry picked from commit 4d6801a6b5bdd4d055a00484a743cb4ada659669) stand/libsa/zfs/zfsimpl.c | 67 ++++++++++++++++++++++++++++++++++++--------- sys/cddl/boot/zfs/zfsimpl.h | 15 ++-------- 2 files changed, 56 insertions(+), 26 deletions(-)
A commit in branch releng/15.0 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=c70f6e4f4b71c57984c7b0ce5bc7d04a7a0fceab commit c70f6e4f4b71c57984c7b0ce5bc7d04a7a0fceab Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2025-11-06 16:00:50 +0000 Commit: Colin Percival <cperciva@FreeBSD.org> CommitDate: 2025-11-13 20:12:21 +0000 stand: Teach the zfs loader about dynamic gang headers There is a pool feature, dynamic_gang_header, that is enabled by default in new pools. When this feature is active, gang headers may be larger than 512 bytes. The loader needs to be taught to cope with that. Try using the vdev ashift to pick the gang block header size. If the checksum fails, fall back to the old gang block header size. This is based on a patch by Paul Dagnelie, with testing, bug-fixing and some simplifications from me. Approved by: re (cperciva) PR: 289690 Co-authored by: Paul Dagnelie <paul.dagnelie@klarasystems.com> Reviewed by: imp MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D53578 (cherry picked from commit 4d6801a6b5bdd4d055a00484a743cb4ada659669) (cherry picked from commit ae17780c717547a7b52cc593dca8867c0e14f9ef) stand/libsa/zfs/zfsimpl.c | 67 ++++++++++++++++++++++++++++++++++++--------- sys/cddl/boot/zfs/zfsimpl.h | 15 ++-------- 2 files changed, 56 insertions(+), 26 deletions(-)