Bug 204484 - ZFS caches on 4K GELI devices accumulate inordinate write errors
Summary: ZFS caches on 4K GELI devices accumulate inordinate write errors
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 10.2-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-fs mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-11-12 00:08 UTC by Christian Campbell
Modified: 2016-01-17 20:02 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Christian Campbell 2015-11-12 00:08:18 UTC
When zpool creates a cache on a GELI device providing 4K sectors the pool immediately reports message ZFS-8000-9P. The cache device experiences an inordinate number of write errors -- over a million write errors a day (about 11.8 per second).

Using a GELI device with 512b sectors does not exhibit this problem but, of course, degrades encryption performance.

This appears to be the same bug as FreeNAS Bugs #6734 (https://bugs.freenas.org/issues/6734) and #7092 (https://bugs.freenas.org/issues/7092), although I'm unclear why they consider the former resolved.

I don't imagine it matters, but the underlying drive sector size is 512 logical / 512 physical.
Comment 1 nowak 2015-11-12 10:25:59 UTC
L2ARC issues physical writes of size possibly not aligned to device block size. zio_vdev_io_start would later align by padding with zeroes. This changed with http://svnweb.freebsd.org/base/head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c?r1=268123&r2=268855&pathrev=269093 where IO tagged as physical would not be padded (without fixing L2ARC).
Comment 2 Matthew Dresden 2016-01-17 20:02:40 UTC
I have not tried to reproduce this with 512 alignment, but it is still an current issue affecting the stable release of 10.2. Is this bug getting any priority currently.

Using mfsbsd boot and installing from ftp.freebsd.org with the script below, these errors appear on ever cache immediately.

Example was a machine with:

ada0 - 120GB ssd
ada1 - 2tb
ada2 - 2tb
ada3 - 2tb
ada4 - 2tb
add5 - 2tb
ada6 - 127GB ssd

This occurs for me on virtual and physical machines:

Here is a semi automated install script to reproduce my scenario.

Instructions to use:

Boot from mfsbsd usb imaged.
cd /tmp
fetch http://<This Script>
sh install-script.sh
Enter encryption password.
Install completes
take zpool status
reboot
take zpool status

Script Follows:

#!/bin/sh 

cd /tmp

read -p "Enter a disk encryption password: " password </dev/tty

echo "${password}" > passfile.txt

PASSFILE="/tmp/passfile.txt"

fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/10.2-RELEASE/kernel.txz

tar xvfz kernel.txz

sysctl kern.geom.debugflags=0x10

dd bs=512 if=/dev/zero of=/dev/ada0 count=10

dd bs=512 if=/dev/zero of=/dev/ada1 count=10

dd bs=512 if=/dev/zero of=/dev/ada2 count=10

dd bs=512 if=/dev/zero of=/dev/ada3 count=10

dd bs=512 if=/dev/zero of=/dev/ada4 count=10

dd bs=512 if=/dev/zero of=/dev/ada5 count=10

dd bs=512 if=/dev/zero of=/dev/ada6 count=10

dd bs=1m if=/dev/zero of=/dev/ada0 &
pid0=$(echo $!)

dd bs=1m if=/dev/zero of=/dev/ada1 &
pid1=$(echo $!)

dd bs=1m if=/dev/zero of=/dev/ada2 &
pid2=$(echo $!)

dd bs=1m if=/dev/zero of=/dev/ada3 &
pid3=$(echo $!)

dd bs=1m if=/dev/zero of=/dev/ada4 &
pid4=$(echo $!)

dd bs=1m if=/dev/zero of=/dev/ada5 &
pid5=$(echo $!)

dd bs=1m if=/dev/zero of=/dev/ada6 &
pid6=$(echo $!)

printf "\nLetting dd do some work!\n"
printf "\n6 minutes to go"
sleep 60
printf "\n5 minutes to go"
sleep 60
printf "\n4 minutes to go"
sleep 60
printf "\n3 minutes to go"
sleep 60
printf "\n2 minutes to go"
sleep 60
printf "\n1 minutes to go"
sleep 60

for i in "$pid0" "$pid1" "$pid2" "$pid3" "$pid4" "$pid5" "$pid6" 
do
   printf "\nWe don't need to wait any longer -- Killing pid ${i}\n"
   kill "$i"
done

kern_mod="/tmp/boot/kernel"

kldload ${kern_mod}/aesni.ko
kldload ${kern_mod}/geom_eli.ko
kldload ${kern_mod}/zfs.ko

rm -Rf "/tmp/boot"

gpart create -s gpt ada0
gpart create -s gpt ada1
gpart create -s gpt ada2
gpart create -s gpt ada3
gpart create -s gpt ada4
gpart create -s gpt ada5
gpart create -s gpt ada6

# Add p1 - non encrpyted
gpart add -s 64K -a 4k -t freebsd-boot ada1 #p1
gpart add -s 64K -a 4k -t freebsd-boot ada2 #p1
gpart add -s 64K -a 4k -t freebsd-boot ada3 #p1
gpart add -s 64K -a 4k -t freebsd-boot ada4 #p1
gpart add -s 64K -a 4k -t freebsd-boot ada5 #p1

# Add p2 - non encrypted boot
gpart add -s 4g -a 4k -t freebsd-zfs -l boot-zfs1 ada1 #p2
gpart add -s 4g -a 4k -t freebsd-zfs -l boot-zfs2 ada2 #p2
gpart add -s 4g -a 4k -t freebsd-zfs -l boot-zfs3 ada3 #p2
gpart add -s 4g -a 4k -t freebsd-zfs -l boot-zfs4 ada4 #p2
gpart add -s 4g -a 4k -t freebsd-zfs -l boot-zfs5 ada5 #p2

# Add p1 - encrypted swap
gpart add -s 20g -a 4k -t freebsd-swap -l swap1 ada0 #p1
gpart add -s 13g -a 4k -t freebsd-swap -l swap2 ada6 #p1

# Add p3 - encrypted zroot
gpart add -s 16g -a 4k -t freebsd-zfs -l zroot-zfs1 ada1 #p3
gpart add -s 16g -a 4k -t freebsd-zfs -l zroot-zfs2 ada2 #p3
gpart add -s 16g -a 4k -t freebsd-zfs -l zroot-zfs3 ada3 #p3
gpart add -s 16g -a 4k -t freebsd-zfs -l zroot-zfs4 ada4 #p3
gpart add -s 16g -a 4k -t freebsd-zfs -l zroot-zfs5 ada5 #p3

# Add p2 - encrypted zroot logzil
gpart add -s 8g -a 4k -t freebsd-zfs -l zroot-log1 ada0 #p2
gpart add -s 8g -a 4k -t freebsd-zfs -l zroot-log2 ada6 #p2

# Add p3 - encrypted zroot cache
gpart add -s 8g -a 4k -t freebsd-zfs -l zroot-cache1 ada0 #p3
gpart add -s 8g -a 4k -t freebsd-zfs -l zroot-cache2 ada6 #p3

# Add p4 - encrypted datastore
gpart add -a 4k -t freebsd-zfs -l datastore-zfs1 ada1 #p4
gpart add -a 4k -t freebsd-zfs -l datastore-zfs2 ada2 #p4
gpart add -a 4k -t freebsd-zfs -l datastore-zfs3 ada3 #p4
gpart add -a 4k -t freebsd-zfs -l datastore-zfs4 ada4 #p4
gpart add -a 4k -t freebsd-zfs -l datastore-zfs5 ada5 #p4

# Add p4 - encrypted datastore logzil
gpart add -s 8g -a 4k -t freebsd-zfs -l datastore-log1 ada0 #p4
gpart add -s 8g -a 4k -t freebsd-zfs -l datastore-log2 ada6 #p4

# Add p5 - encrypted datastore cache
gpart add -a 4k -t freebsd-zfs -l datastore-cache1 ada0 #p5
gpart add -a 4k -t freebsd-zfs -l datastore-cache2 ada6 #p5

gpart set -a bootme -i 2 ada1
gpart set -a bootme -i 2 ada2
gpart set -a bootme -i 2 ada3
gpart set -a bootme -i 2 ada4
gpart set -a bootme -i 2 ada5

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada2
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada3
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada4
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada5

mkdir /root/keys

dd if=/dev/random of=/root/keys/boot.key bs=128k count=1

geli init -J $PASSFILE -b -K /root/keys/boot.key -s 4096 -l 256 /dev/gpt/swap1
geli init -J $PASSFILE -b -K /root/keys/boot.key -s 4096 -l 256 /dev/gpt/swap2

geli init -J $PASSFILE -b -K /root/keys/boot.key -s 4096 -l 256 /dev/gpt/zroot-zfs1
geli init -J $PASSFILE -b -K /root/keys/boot.key -s 4096 -l 256 /dev/gpt/zroot-zfs2
geli init -J $PASSFILE -b -K /root/keys/boot.key -s 4096 -l 256 /dev/gpt/zroot-zfs3
geli init -J $PASSFILE -b -K /root/keys/boot.key -s 4096 -l 256 /dev/gpt/zroot-zfs4
geli init -J $PASSFILE -b -K /root/keys/boot.key -s 4096 -l 256 /dev/gpt/zroot-zfs5

geli init -J $PASSFILE -b -K /root/keys/boot.key -s 4096 -l 256 /dev/gpt/zroot-log1
geli init -J $PASSFILE -b -K /root/keys/boot.key -s 4096 -l 256 /dev/gpt/zroot-log2

geli init -J $PASSFILE -b -K /root/keys/boot.key -s 4096 -l 256 /dev/gpt/zroot-cache1
geli init -J $PASSFILE -b -K /root/keys/boot.key -s 4096 -l 256 /dev/gpt/zroot-cache2

geli init -J $PASSFILE -b -K /root/keys/boot.key -s 4096 -l 256 /dev/gpt/datastore-zfs1
geli init -J $PASSFILE -b -K /root/keys/boot.key -s 4096 -l 256 /dev/gpt/datastore-zfs2
geli init -J $PASSFILE -b -K /root/keys/boot.key -s 4096 -l 256 /dev/gpt/datastore-zfs3
geli init -J $PASSFILE -b -K /root/keys/boot.key -s 4096 -l 256 /dev/gpt/datastore-zfs4
geli init -J $PASSFILE -b -K /root/keys/boot.key -s 4096 -l 256 /dev/gpt/datastore-zfs5

geli init -J $PASSFILE -b -K /root/keys/boot.key -s 4096 -l 256 /dev/gpt/datastore-log1
geli init -J $PASSFILE -b -K /root/keys/boot.key -s 4096 -l 256 /dev/gpt/datastore-log2

geli init -J $PASSFILE -b -K /root/keys/boot.key -s 4096 -l 256 /dev/gpt/datastore-cache1
geli init -J $PASSFILE -b -K /root/keys/boot.key -s 4096 -l 256 /dev/gpt/datastore-cache2

geli attach -j $PASSFILE -k /root/keys/boot.key /dev/gpt/swap1
geli attach -j $PASSFILE -k /root/keys/boot.key /dev/gpt/swap2

geli attach -j $PASSFILE -k /root/keys/boot.key /dev/gpt/zroot-zfs1
geli attach -j $PASSFILE -k /root/keys/boot.key /dev/gpt/zroot-zfs2
geli attach -j $PASSFILE -k /root/keys/boot.key /dev/gpt/zroot-zfs3
geli attach -j $PASSFILE -k /root/keys/boot.key /dev/gpt/zroot-zfs4
geli attach -j $PASSFILE -k /root/keys/boot.key /dev/gpt/zroot-zfs5

geli attach -j $PASSFILE -k /root/keys/boot.key /dev/gpt/zroot-log1
geli attach -j $PASSFILE -k /root/keys/boot.key /dev/gpt/zroot-log2

geli attach -j $PASSFILE -k /root/keys/boot.key /dev/gpt/zroot-cache1
geli attach -j $PASSFILE -k /root/keys/boot.key /dev/gpt/zroot-cache2

geli attach -j $PASSFILE -k /root/keys/boot.key /dev/gpt/datastore-zfs1
geli attach -j $PASSFILE -k /root/keys/boot.key /dev/gpt/datastore-zfs2
geli attach -j $PASSFILE -k /root/keys/boot.key /dev/gpt/datastore-zfs3
geli attach -j $PASSFILE -k /root/keys/boot.key /dev/gpt/datastore-zfs4
geli attach -j $PASSFILE -k /root/keys/boot.key /dev/gpt/datastore-zfs5

geli attach -j $PASSFILE -k /root/keys/boot.key /dev/gpt/datastore-log1
geli attach -j $PASSFILE -k /root/keys/boot.key /dev/gpt/datastore-log2

geli attach -j $PASSFILE -k /root/keys/boot.key /dev/gpt/datastore-cache1
geli attach -j $PASSFILE -k /root/keys/boot.key /dev/gpt/datastore-cache2

zpool create zboot raidz1 gpt/boot-zfs1 gpt/boot-zfs2 gpt/boot-zfs3 gpt/boot-zfs4 gpt/boot-zfs5
zfs set checksum=fletcher4 zboot
zfs set atime=off zboot

zpool create -f \
-o altroot=/mnt \
-O canmount=off \
-m none \
zroot raidz1 \
/dev/gpt/zroot-zfs1.eli \
/dev/gpt/zroot-zfs2.eli \
/dev/gpt/zroot-zfs3.eli \
/dev/gpt/zroot-zfs4.eli \
/dev/gpt/zroot-zfs5.eli

zpool add zroot log mirror gpt/zroot-log1.eli gpt/zroot-log2.eli
zpool add zroot cache gpt/zroot-cache1.eli gpt/zroot-cache2.eli

zfs set checksum=fletcher4 zroot
zfs set atime=off zroot

zfs create -o mountpoint=none zroot/ROOT
zfs create -o mountpoint=/ zroot/ROOT/default
zfs create -o mountpoint=/tmp -o compression=lzjb -o setuid=off zroot/tmp
chmod 1777 /mnt/tmp

zfs create -o mountpoint=/usr zroot/usr
zfs create zroot/usr/local
zfs create -o compression=lzjb -o setuid=off zroot/usr/ports
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/distfiles
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/packages
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/usr/src
zfs create zroot/usr/obj
zfs create -o mountpoint=/var zroot/var
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/crash
zfs create -o exec=off -o setuid=off zroot/var/db
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/db/pkg
zfs create -o exec=off -o setuid=off zroot/var/empty
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
zfs create -o compression=gzip -o exec=off -o setuid=off zroot/var/mail
zfs create -o exec=off -o setuid=off zroot/var/run
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/tmp

chmod 1777 /mnt/var/tmp

zpool create -m /mnt/datastore datastore raidz1 gpt/datastore-zfs1.eli gpt/datastore-zfs2.eli gpt/datastore-zfs3.eli gpt/datastore-zfs4.eli gpt/datastore-zfs5.eli

zpool add datastore log mirror gpt/datastore-log1.eli gpt/datastore-log2.eli
zpool add datastore cache gpt/datastore-cache1.eli gpt/datastore-cache2.eli

zfs set checksum=fletcher4 datastore
zfs set atime=off datastore
zfs set compression=lz4
zfs create -o mountpoint=/home -o setuid=off datastore/home

cd /tmp

fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/10.2-RELEASE/base.txz
fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/10.2-RELEASE/doc.txz
fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/10.2-RELEASE/lib32.txz
fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/10.2-RELEASE/ports.txz
fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/10.2-RELEASE/src.txz

for i in base doc kernel lib32 ports src; do \
	xz -d -c $i.txz | tar -C /mnt/ -xf - ; \
done

cat << EOF > /mnt/etc/fstab
/dev/ada0p1.eli none swap sw 0 0
/dev/ada6p1.eli none swap sw 0 0
EOF

cat << EOF > /boot/loader.conf
# enable AHCI driver
ahci_load="YES"

# fs modules
zfs_load="YES"
fdescfs_load="YES"
nullfs_load="YES"

# CTL ISCSI
ctl_load="YES"

# Lagg support
if_lagg_load="YES"

# Crypto stuff
geom_eli_load="YES"
crypto_load="YES"
aesni_load="YES"
cryptodev_load="YES"

# pf stuff
pf_load="YES"
pflog_load="YES"

# zfs tuning
vm.kmem_size="8192M"
vm.kmem_size_max="8192M"
vfs.zfs.arc_max="8192M"
vfs.zfs.prefetch_disable="1"
vfs.zfs.zio.use_uma="0"
vfs.zfs.txg.timeout="5"
vfs.zfs.txg.write_limit_override=1073741824

# disk encryption key load
geli_ada0p1_keyfile0_load="YES"
geli_ada0p1_keyfile0_type="ada0p1:geli_keyfile0"
geli_ada0p1_keyfile0_name="/boot/keys/boot.key"

geli_ada0p2_keyfile0_load="YES"
geli_ada0p2_keyfile0_type="ada0p2:geli_keyfile0"
geli_ada0p2_keyfile0_name="/boot/keys/boot.key"

geli_ada0p3_keyfile0_load="YES"
geli_ada0p3_keyfile0_type="ada0p3:geli_keyfile0"
geli_ada0p3_keyfile0_name="/boot/keys/boot.key"

geli_ada0p4_keyfile0_load="YES"
geli_ada0p4_keyfile0_type="ada0p4:geli_keyfile0"
geli_ada0p4_keyfile0_name="/boot/keys/boot.key"

geli_ada0p5_keyfile0_load="YES"
geli_ada0p5_keyfile0_type="ada0p5:geli_keyfile0"
geli_ada0p5_keyfile0_name="/boot/keys/boot.key"

geli_ada6p1_keyfile0_load="YES"
geli_ada6p1_keyfile0_type="ada6p1:geli_keyfile0"
geli_ada6p1_keyfile0_name="/boot/keys/boot.key"

geli_ada6p2_keyfile0_load="YES"
geli_ada6p2_keyfile0_type="ada6p2:geli_keyfile0"
geli_ada6p2_keyfile0_name="/boot/keys/boot.key"

geli_ada6p3_keyfile0_load="YES"
geli_ada6p3_keyfile0_type="ada6p3:geli_keyfile0"
geli_ada6p3_keyfile0_name="/boot/keys/boot.key"

geli_ada6p4_keyfile0_load="YES"
geli_ada6p4_keyfile0_type="ada6p4:geli_keyfile0"
geli_ada6p4_keyfile0_name="/boot/keys/boot.key"

geli_ada6p5_keyfile0_load="YES"
geli_ada6p5_keyfile0_type="ada6p5:geli_keyfile0"
geli_ada6p5_keyfile0_name="/boot/keys/boot.key"

geli_ada1p2_keyfile0_load="YES"
geli_ada1p2_keyfile0_type="ada1p2:geli_keyfile0"
geli_ada1p2_keyfile0_name="/boot/keys/boot.key"

geli_ada1p3_keyfile0_load="YES"
geli_ada1p3_keyfile0_type="ada1p3:geli_keyfile0"
geli_ada1p3_keyfile0_name="/boot/keys/boot.key"

geli_ada1p4_keyfile0_load="YES"
geli_ada1p4_keyfile0_type="ada1p4:geli_keyfile0"
geli_ada1p4_keyfile0_name="/boot/keys/boot.key"

geli_ada2p2_keyfile0_load="YES"
geli_ada2p2_keyfile0_type="ada2p2:geli_keyfile0"
geli_ada2p2_keyfile0_name="/boot/keys/boot.key"

geli_ada2p3_keyfile0_load="YES"
geli_ada2p3_keyfile0_type="ada2p3:geli_keyfile0"
geli_ada2p3_keyfile0_name="/boot/keys/boot.key"

geli_ada2p4_keyfile0_load="YES"
geli_ada2p4_keyfile0_type="ada2p4:geli_keyfile0"
geli_ada2p4_keyfile0_name="/boot/keys/boot.key"

geli_ada3p2_keyfile0_load="YES"
geli_ada3p2_keyfile0_type="ada3p2:geli_keyfile0"
geli_ada3p2_keyfile0_name="/boot/keys/boot.key"

geli_ada3p3_keyfile0_load="YES"
geli_ada3p3_keyfile0_type="ada3p3:geli_keyfile0"
geli_ada3p3_keyfile0_name="/boot/keys/boot.key"

geli_ada3p4_keyfile0_load="YES"
geli_ada3p4_keyfile0_type="ada3p4:geli_keyfile0"
geli_ada3p4_keyfile0_name="/boot/keys/boot.key"

geli_ada4p2_keyfile0_load="YES"
geli_ada4p2_keyfile0_type="ada4p2:geli_keyfile0"
geli_ada4p2_keyfile0_name="/boot/keys/boot.key"

geli_ada4p3_keyfile0_load="YES"
geli_ada4p3_keyfile0_type="ada4p3:geli_keyfile0"
geli_ada4p3_keyfile0_name="/boot/keys/boot.key"

geli_ada4p4_keyfile0_load="YES"
geli_ada4p4_keyfile0_type="ada4p4:geli_keyfile0"
geli_ada4p4_keyfile0_name="/boot/keys/boot.key"

geli_ada5p2_keyfile0_load="YES"
geli_ada5p2_keyfile0_type="ada5p2:geli_keyfile0"
geli_ada5p2_keyfile0_name="/boot/keys/boot.key"

geli_ada5p3_keyfile0_load="YES"
geli_ada5p3_keyfile0_type="ada5p3:geli_keyfile0"
geli_ada5p3_keyfile0_name="/boot/keys/boot.key"

geli_ada5p4_keyfile0_load="YES"
geli_ada5p4_keyfile0_type="ada5p4:geli_keyfile0"
geli_ada5p4_keyfile0_name="/boot/keys/boot.key"

# Mount root
vfs.root.mountfrom="zfs:zroot/ROOT/default"
EOF

cat << EOF > /mnt/etc/rc.conf
# Network
hostname="example.example.com"
ifconfig_em0="DHCP"

# zfs
zfs_enable="YES"

# ssh server
sshd_enable="YES"

# ntp client
ntpd_enable="YES"
ntpd_sync_on_start="YES"

# NFS Server
rpcbind_enable="YES"
rpc_lockd_enable="YES"
rpc_statd_enable="YES"
nfs_server_enable="YES"
# the number of nfsds to run [-n] should be:  (Number-of-clients + 1) * 4
nfs_server_flags="-u -t -n 100"
mountd_enable="YES"
mountd_flags="-r -p 100"

#ISCSI Target Native Service
ctld_enable="YES"

# disable Sendmail
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"

# powerd
#powerd_enable="YES"

# motd
motd_enable="yes"
EOF

cat << EOF >> /mnt/etc/sysctl.conf
kern.maxvnodes=260000
EOF

touch /mnt/etc/ctl.conf
touch /mnt/etc/exports

# Final work

cp /boot/zfs/zpool.cache /mnt/boot/zfs/

cp -pR /root/keys /mnt/boot/

mkdir -p /zboot/boot
cp -pR /mnt/boot/* /zboot/boot/

cp /boot/loader.conf /zboot/boot/

# Command to run following an freebsd update
# rsync -alv --human-readable --delete -c -i /boot/ /zboot/boot/ --exclude loader.conf --exclude keys --exclude boot.key