Description Michael Osipov 2026-04-08 17:08:20 UTC

I have upgraded several servers from 13 to 14 via /usr/src and wanted to install new boot loaders as recommended by the release notes. Made several observations where the documentation is lacking and/or problematic and needs to be updated.

Constellations:
* ZFS + BIOS boot
* UFS + UEFI boot

ZFS + BIOS boot:
> root@deblndw014x:~
> # zpool status zroot
>   pool: zroot
>  state: ONLINE
> status: Some supported and requested features are not enabled on the pool.
>         The pool can still be used, but some features are unavailable.
> action: Enable all features using 'zpool upgrade'. Once this is done,
>         the pool may no longer be accessible by software that does not support
>         the features. See zpool-features(7) for details.
> config:
> 
>         NAME          STATE     READ WRITE CKSUM
>         zroot         ONLINE       0     0     0
>           mirror-0    ONLINE       0     0     0
>             gpt/zfs0  ONLINE       0     0     0
>             gpt/zfs1  ONLINE       0     0     0
> 
> errors: No known data errors
> root@deblndw014x:~
> # gpart show
> =>       40  585937424  da11  GPT  (279G)
>          40       1024     1  freebsd-boot  (512K)
>        1064        984        - free -  (492K)
>        2048   33554432     2  freebsd-swap  (16G)
>    33556480  552380416     3  freebsd-zfs  (263G)
>   585936896        568        - free -  (284K)
> 
> =>       40  585937424  da3  GPT  (279G)
>          40       1024    1  freebsd-boot  (512K)
>        1064        984       - free -  (492K)
>        2048   33554432    2  freebsd-swap  (16G)
>    33556480  552380416    3  freebsd-zfs  (263G)
>   585936896        568       - free -  (284K)
> 
> root@deblndw014x:~
> # gpart show -l
> =>       40  585937424  da11  GPT  (279G)
>          40       1024     1  gptboot0  (512K)
>        1064        984        - free -  (492K)
>        2048   33554432     2  swap0  (16G)
>    33556480  552380416     3  zfs0  (263G)
>   585936896        568        - free -  (284K)
> 
> =>       40  585937424  da3  GPT  (279G)
>          40       1024    1  gptboot1  (512K)
>        1064        984       - free -  (492K)
>        2048   33554432    2  swap1  (16G)
>    33556480  552380416    3  zfs1  (263G)
>   585936896        568       - free -  (284K)

The notes do NOT tell that I have to update the boot code on all drives in the ZFS mirror, for novices likely not obvious because ZFS seems like magic to many.

This would of course apply to ZFS + UEFI in a mirror as well.

UFS + UEFI boot:
I have mounted the ESP to /mnt to update the default loader BOOTX64.efi and FREEBSD/loader.efi, but constantly received errors from cp(1) with broken files. Started digging...
> root@deblndw011x:~
> # gpart show da0
> =>       40  585871888  da0  GPT  (279G)
>          40     409600    1  efi  (200M)
>      409640    4194304    2  freebsd-ufs  (2.0G)
>     4603944   16777216    3  freebsd-swap  (8.0G)
>    21381160    8388608    4  freebsd-ufs  (4.0G)
>    29769768    8388608    5  freebsd-ufs  (4.0G)
>    38158376   16777216    6  freebsd-ufs  (8.0G)
>    54935592   25165824    7  freebsd-ufs  (12G)
>    80101416  505413632    8  freebsd-vinum  (241G)
>   585515048     356880       - free -  (174M)
> 
> root@deblndw011x:~
> # mount -t msdosfs /dev/da0p1 /mnt/
> root@deblndw011x:~
> # df
> Filesystem                                1K-blocks     Used    Avail Capacity  Mounted on
> ...
> /dev/da0p1                                      778      385      393    50%    /mnt
> root@deblndw011x:~
> # df /mnt/
> Filesystem 1K-blocks Used Avail Capacity  Mounted on
> /dev/da0p1       778  385   393    50%    /mnt
> root@deblndw011x:~
> # file -s /dev/da0p1
> /dev/da0p1: DOS/MBR boot sector, code offset 0x3c+2, OEM-ID "BSD4.4  ", root entries 512, sectors 1600 (volumes <=32 MB), sectors/FAT 5, sectors/track 63, heads 1, serial number 0xbd4111ee, label: "EFISYS     ", FAT (12 bit), followed by FAT
> root@deblndw011x:~
> # tree -D /mnt/
> [Dec 31  1979]  /mnt/
> └── [Apr 16  2018]  efi
>     └── [Apr 16  2018]  boot
>         ├── [Apr 16  2018]  BOOTx64.efi
>         └── [Apr 16  2018]  startup.nsh
> 
> 3 directories, 2 files

This does not make any sense, but explains why I was not able to fix /boot/loader.efi twice onto the ESP.

It turned out that there was a period where the FAT12 image of the ESP was copied via dd on to the 200 MiB ESP. This was "fixed" in db8b56134506840832bec2d1ce07b9e00d4d6d71.

The notes MUST put such systems into consideration when recommending adding both loader images into this tiny FS. I was sweating blood and tears because it didn't add up and I thought I messed up with the ESP leaving the system unbootable.

Loader before upgrade:
> # ll /mnt/efi/boot/BOOTx64.efi
> -rwxr-xr-x  1 root  wheel  393216 2018-04-16 10:12 /mnt/efi/boot/BOOTx64.efi*

Loader after upgrade:
> # ll /mnt/efi/boot/BOOTx64.efi
> -rwxr-xr-x  1 root root 662528 2026-04-08 13:01 /mnt/efi/boot/BOOTx64.efi*

As a cherry on top maybe a paragraph could describe how to convert this fauly partition into a real FAT32 ESP with 200 MiB size.