Hi all, while assisting an OPNsense user with SSD replacement in their root zpool, he did something like zpool attach zroot ada1p4 ada0 after replacing a failed ada0. Of course I told them that this is absolutely not the way to do it, that their system will be unbootable in case ada1 fails, etc. We went through the procedure in the forum discussion and the system is up and healthy with a mirrored pool and EFI and legacy boot partitions on both devices, now. For reference: https://forum.opnsense.org/index.php?topic=47333.0 I then asked what gave them the idea to use the whole disk and they claimed about every piece of documentation they could find said to do exactly that. Now I was certain that at least FreeBSD's own docs would of course show how to create a proper GPT partition to use for ZFS etc. and wanted to drop them a link to the handbook. To my great surprise and dismay the handbook uses procedures like this: zpool create example /dev/da0 And in the next code example it references devices like /dev/ad0s1a. - we don't have adN since ATA CAM became the default (in FreeBSD 8?) - nobody is supposed to use MBR partitioning on current hardware in 2025 - nobody is supposed to use BSD partition tables on current hardware on 2025 - even with UFS This entire chapter is misleading, downright dangerous, and in conflict with the structures our installer actually produces. So how can we fix that? Summer of Code? I do not have the capacity to rewrite that chapter myself. Kind regards, take care, Patrick
I'll take this. The chapter originated from my lecture slides from "way back when" we had such disk device names. After it was published, other people contributed to it (naturally). There was no unified device naming scheme applied, which is probably why it looks so stitched together. I think a couple of well placed search&replace actions ought to fix it.
I created this review to work on the patch: https://reviews.freebsd.org/D50603
Great! Thank you, Benedict. Michael Dexter independently started working on it, maybe that can be helpful? https://docs.google.com/document/d/1l95HyrZLCxCbq1Qnm4FVen0a3yQsBUwXSwezsJbuDM8 Kind regards, Patrick
Hi Benedict, I looked at your proposed patch. My point is not only that the device names are historical. My point is that even with your proposed patch e.g. zpool create example /dev/ada1 is something you should never do! Plain and simple. What you should do instead: gpart create -s gpt ada1 gpart add -t freebsd-zfs ada1 zpool create example /dev/ada1p1 Always use partitions of type freebsd-zfs to create ZFS vdevs.
I've added the GPT partition creation to the Quickstart section in my new diff. I've also used the GPT partitions in all zpool status outputs.
(In reply to punkt.de Hosting Team from comment #4) I disagree with the comment that you "never" use a full device and always use a partition. It is the opposite of the guidance ZFS provides. https://openzfs.github.io/openzfs-docs/man/master/7/zpoolconcepts.7.html "ZFS can use individual slices or partitions, though the recommended mode of operation is to use whole disks." I assume you're referring to a bootable devices. In that specific context, it's worth clarifying. Maybe the ZFS section needs a "Partition Considerations" section, e.g., if booting from the device, or doing full-disk encryption, the consider partitions. If doing a 2-dev mirror for storage, it's not necessary. It's touched on in the zpool admin section, but should be more prominent and earlier.
I'd argue that this is a legacy from Sun/Oracle documentation where indeed the recommendation was to use raw disks. That never applied to FreeBSD. But ok, maybe a more general explanation of the topic and specifically the layout for bootable devices is what is really called for.
This also came up recently on an OpenZFS Production User call when the group tried to send a new user to introductory documentation, only to find that the Handbook page was indeed using "ad" devices, not suggesting partitions or labels, giving examples from 2008, repeating itself, and many more shortcomings. This drove me to paste the chapter into a Google doc (not an endorsement) as Patrick pointed out, to perform some cleanup. I have rewritten the first few sections and have a suggested outline. I would prefer this be a good introduction rather than a deep-dive, given that ZFS takes years to understand. I will attach a PDF snapshot and welcome your feedback on this approach, stylistically etc. The doc as mentioned: https://docs.google.com/document/d/1l95HyrZLCxCbq1Qnm4FVen0a3yQsBUwXSwezsJbuDM8/edit
Created attachment 260808 [details] Draft ZFS Chapter Snapshot A snapshot of my draft for those who prefer avoid Google Docs.
I've looked at it and there are definitely good portions to add to the chapter in there. We should avoid trying to integrate them all at once in a big super-patch. Rather, an incremental series of smaller patcher are easier to review and work on. Starting with the device naming seems reasonable to me, followed by other nuggets from the PDF. I can also see Marks argument in that the quickstart section should get people started quickly to graps the concepts. It's true that ZFS is complex, but for people who want to get started, using whole disks to get a pool started in one command has its merits. The question is how big such a quickstart section should be before it turns into its own thing. And starting with something easy (quick wins) only to find later in the chapter that you need GPT partitions, label them properly, (oh and ashift, this that and the other thing...) may make people shy away from the complexity. I think this is not what we want. We want people to get started with ZFS on FreeBSD: either by dumbing it down to the bare essentials or describing to them the whole complexity to do it right from the beginning. I'm good with both and a handbook should contain a lot of details. Hence I'm thinking of maybe reducing the quickstart section to only a few examples or removing it altogether. But that should be it's own patch. What about the patch in Pahbricator, is that a reasonable start or does it need more refinement in itself (again, more patches for other things can follow it)?
Consider defining a scope and index before changing a single word. Is it a quick-start guide or a comprehensive guide? I have gone with the bare minimum of using a single disk and introduce terminology as it is employed. From there basic maintenance commands that would be followed by export/re-import. That would be followed by snapshotting and replication, multi-disk configurations, and disk replacement. What would a quick-start guide include? Perhaps the commands needed to maintain a root-on-ZFS system created by bsdinstall(8)? Regarding device naming, the installer labels partitions by default as a best practice. The devices should only be visible during partitioning, not with zpool(8) and zfs(8) operations. Of course they can, but is there a reason to avoid the best practice?
Regarding the need for an index: Do a search for 'zpool replace' Disk replacement is covered several times and called "recovery" in one place, which has a very specific meaning in storage. I can copy the original version of my doc if you like which simply marked problems with the chapter. Shortcomings include: Sequence of the information Redundancy of the information i.e. "zpool replace" Dated examples such as manually enabling compression Few references to the related manual pages Inconsistent naming in the examples The strange 22.3.8. Self-Healing section The strange 22.3.14. Splitting a Storage Pool section, without commands 22.4.1. Creating and Destroying Datasets though the content is covered in 22.2.1. Single Disk Pool, with a strange focus on compression (lz4 is enabled by default), with compression further discussed in 22.4.10. Compression ARC limits in tuning that have probably changed sysctl syntax and do not give the allowable input, which sometimes needs to be calculated There is a lot of good content there and yes, some can be addressed piece by piece, but a revised index of the desired sections would greatly simplify that process.
Maybe our presumed audiences are different. Benedict wrote > people who want to get started with ZFS ... I don't know anyone who wants to get started with ZFS. ZFS root is simply the default and the only reasonable choice for a server in 2025. I run > 100 productive machines (hardware) - guess, how many have anything but ZFS! Zero! It's the default for OPNsense now for 2 or 3 releases at least. Our installer offers ZFS as the first and default option. So I hope we can agree that the vast majority of people who install FreeBSD for the first time will end up with a ZFS based installation. This includes (by default) a legacy boot and an EFI partition. The handbook should help a system administrator running productive systems. Unless I am completely missing the target audience. So what happens? Administrator installs FreeBSD on a single disk and then wants to add a second one in a mirror. Or installs with a mirrored setup and then needs to replace one disk because it broke. See the linked OPNsense thread. Repeating myself: every single appliance Deciso sells and ships is running ZFS. It's 2025, folks :-) That was my intention when I created this bug report. We do not need a "this is the novelty ZFS filesystem and these are the fundamental concepts - grab a bunch of USB drives and toy with them". We need "this is the FreeBSD default installation, this is how FreeBSD boots, this is how the disks are laid out, this is how you replace a disk, this is how you upgrade the boot loader when you upgrade your zpool". That's my view of reality as far as FreeBSD and ZFS are concerned. Of course yours might be different. Kind regards, Patrick
(In reply to punkt.de Hosting Team from comment #4) > Always use partitions of type freebsd-zfs to create ZFS vdevs. It's not always beneficial to create partitions, ZFS itself is a volume manager and does not really need partitioning if you are using the full disk (although the cost for the partition is de minimis nowadays). I personally use partitions primarily because I use the drives as redundant boot devices, and in previous work (FreeNAS) we created a small swap partition, which was useful because in case the administrator acquired a replacement drive (a lot of them are using consumer grade SATA hard drives and there are non-zero chance that different vendor have a slight but meaningful different opinion on how big "4TB" is, for example), and these swap partitions can be deleted or shrunk in such cases to make space for the replacement device instead of stressing the administrator in the panic (one drive failed, and purchased a slightly smaller drive) situation. However, I have never seen such difference in SAS devices, and I'd argue that if one do not need to split the drive into smaller pieces (for example to use them as boot device or swap, plus full-drive encrypt the data storage) and do not need to make the available space smaller (e.g. dedicate 2GB of space just in case a future replacement was smaller), then creating GPT partition is not going to buy them much.
In my experience it is very beneficial to have gpart show show you that there is a ZFS vdev or a part thereof on that disk even if you use the whole disk and so have only a single partition. YMMV. Also I do not quite parse the "volume manager" argument. Of course I am not suggesting to slice your disk into arbitrary volumes. Just place your ZFS into the form of "container" that is standard in at least the amd64 architecture. And I'll bet everyone a beer at the next EurBSDCon that the most frequent setup of ZFS in total is a bootable ZFS install. Which means we will have - efi - freebsd-boot - freebsd-swap - freebsd-zfs on that drive. And the handbook should reflect what people are most likely to encounter. Again: where do I go to answer questions like: "One of my ZFS mirror SSDs failed - how do I replace it?" And clearly zpool attach zroot ada1p4 ada0 is not the way to go as that unfortunate OPNsense user learned. The handbook needs to be of use for real life scenarios. Real life scenario is a system booting from ZFS.
(In reply to punkt.de Hosting Team from comment #15) > And I'll bet everyone a beer at the next EurBSDCon that the most frequent setup of ZFS in total is a bootable ZFS install. I think you're right, most users of ZFS will be using it on their bootable device. I think the thing we have to be structured about is how to document this as ZFS inherently has nothing to do with boot. My suggestion is that somewhere in the handbook booting is described with callouts for what the installer does and shows the common result with ZFS. It should link to the ZFS section for additional ZFS details. Likewise, I'd suggest the ZFS section reference the boot section with a callout at the very beginning, "ZFS can use whole devices, but before manually configuring a device consider if it will be used for booting the system [link to booting] or for full disk encryption [link to geli]. Both of these common use cases recommend partitioning the device before doing anything with ZFS." Any further discussion of partitions in the ZFS section is mixing topics and potentially confusing. Plus, like this, if there are other partition use cases identified they can be called out in their context rather than adding them to the ZFS section.
Another reason for partitions with a swap partition is/was that replacing that "4TB" drive with another "4TB" drives but is in fact a few blocks smaller allows you to steal the swap to make up the difference. Given that FreeBSD is the only operating system to ship with OpenZFS, it sounds like the end goal, based on these last few comments, is to focus on production-ready use and complexities like booting FreeBSD from ZFS... which goes full circle to what the OPNsense user ran into and started this discussion. Perhaps add your "must have" topics to this ticket? I will keep emphasizing the value of an index and scope before more writing is done.
(In reply to Michael Dexter from comment #11) " Regarding device naming, the installer labels partitions by default as a best practice. The devices should only be visible during partitioning, not with zpool(8) and zfs(8) operations. " Wrt a recent install of 14.2-RELEASE, I see that the installer ('GPT') labels its partitions, however, for the selected ZFS install no such labels are used: [1-0] # gpart show -l => 40 1953525088 nda0 GPT (932G) 40 532480 1 efiboot0 (260M) 532520 1024 2 gptboot0 (512K) 533544 984 - free - (492K) 534528 138412032 3 swap0 (66G) 138946560 1814577152 4 zfs0 (865G) 1953523712 1416 - free - (708K) => 40 1953525088 nda1 GPT (932G) 40 532480 1 efiboot1 (260M) 532520 1024 2 gptboot1 (512K) 533544 984 - free - (492K) 534528 138412032 3 swap1 (66G) 138946560 1814577152 4 zfs1 (865G) 1953523712 1416 - free - (708K) [2-0] # zpool status pool: zroot state: ONLINE scan: scrub repaired 0B in 00:00:15 with 0 errors on Tue May 13 16:09:31 2025 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 nda0p4 ONLINE 0 0 0 nda1p4 ONLINE 0 0 0 errors: No known data errors [3-0] # Is this different in development versions?
Thanks for verifying the behavior and yes, we should verify 15.0-CURRENT and the obvious broader question is: Should the installer use labels when creating the zpool? (Note that I am using "zpool" in this context rather than just "pool" to (hopefully) avoid confusion.
(In reply to Michael Dexter from comment #17) +1 on scope and index One has to weight the merits of trying to create (1) a (very) short ZFS getting started guide (2) a (less short) ZFS introduction text (3) a comprehensive (_FreeBSD_) ZFS guide A 'getting started guide' (1) may result in proposing shortcuts without very much foundation. One such example concerns the use of a non-redundant single disk/vdev as a basis for a ZFS pool. IMO this is fine _for educational_ purposes but, for real world usage* the importance of a redundant pool should at least be emphasized (even issued as a warning); this is one of the foundations on which ZFS was built on. A comprehensive guide (3) will likely expand into a very large chapter or several chapters when divided. An introduction text (2) leaves room for, at least highlighting some ZFS foundational aspects together with a more educational emphasis, creating a somewhat more solid basis of knowledge to build on. IMO a clear view of what the Handbook intends to offer the reader should be established beforehand, and worded as such in the text! Although I have no experience in creating and maintaining ZFS (server) installations at scale, I think sysadmins for such installations may expect useful (starting) information about ZFS in the handbook but IMO, running production servers at scale requires more fundamental knowledge for such a complex topic as ZFS. I imagine this is not unlike setting up and maintaining a network: Handbook readers may expect some guidance but fundamental and practical knowledge has to be sourced elsewhere. Reference information should be found and understood at the man page level. Further information should be sourced and understood from reliable sources such as other FreeBSD resources (forums, mailing lists), OpenZFS github/website, ZFS articles from Klara systems, and practical books such as 'Absolutely FreeBSD' and the 'two ZFS books', etc. ___ * Unless one is limited to only one storage device such as with most laptops.
Blocks, or see also: bug 261212 > Update the ZFS chapter of the FreeBSD Handbook, and other OpenZFS-related pages
Let's put a notice at the top of the ZFS chapter saying that the content is outdated and keep it until the new content is written and integrated. Readers of the chapter deserve this basic level of respect. My personal assessment is that achieving a consensus on the new content will take a few months.
Unassign me for now.
Affects: <https://docs.freebsd.org/en/books/handbook/zfs/> (currently chapter 22).
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/doc/commit/?id=653598764d3cfc7d0ee9b93a24dfe80756cfe4ca commit 653598764d3cfc7d0ee9b93a24dfe80756cfe4ca Author: Sergio Carlavilla Delgado <carlavilla@FreeBSD.org> AuthorDate: 2026-07-18 13:23:23 +0000 Commit: Sergio Carlavilla Delgado <carlavilla@FreeBSD.org> CommitDate: 2026-07-18 13:23:23 +0000 Handbook - ZFS: Rework * Add RAID-Z expansion, dRAID, special vdevs, hot spares with zfsd, TRIM, pool checkpoints, block cloning, fast dedup, bookmarks, holds, resumable/corrective send, project quotas, direct I/O, and boot environments * Add Jails and ZFS crossref target, encryption key loading at boot (zfskeys), and a Pool Properties section * Rewrite Tuning to drop removed sysctls (scrub_delay, vdev.cache.size, arc.meta_limit, …); delete the obsolete ZFS-on-i386 section * Switch examples to ada device names and GPT partitions * Fix the inverted reservation terminology entry, zfs scrub: zpool scrub, stale checksum/compression algorithm lists, and EOL version framing Approved by: arrowd@, bcr@, michaelo@ Differential Revision: D58175 PR: 287148, 261212, 263321 .../content/en/books/handbook/zfs/_index.adoc | 2206 +++++++++++++++----- 1 file changed, 1681 insertions(+), 525 deletions(-)
Fixed!