The following shows /usr/bin/install running very slowly when copying the same small file 10 times: $ df /tmp Filesystem 1K-blocks Used Avail Capacity Mounted on syspool/system/tmp 1579986920 2916 1579984004 0% /tmp $ zpool status syspool pool: syspool state: ONLINE scan: scrub repaired 0B in 00:22:36 with 0 errors on Mon Jun 22 12:24:49 2026 config: NAME STATE READ WRITE CKSUM syspool ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/zsysdisk0 ONLINE 0 0 0 gpt/zsysdisk1 ONLINE 0 0 0 errors: No known data errors $ seq 1 10000 | tee /tmp/hh | wc -c 48894 $ /usr/bin/time csh -c 'repeat 10 /usr/bin/install /tmp/hh /tmp/hh2' 1.52 real 0.00 user 0.04 sys $ /usr/bin/time csh -c 'repeat 10 /usr/bin/install /tmp/hh /tmp/hh2' 1.63 real 0.01 user 0.02 sys $ truss -fc csh -c 'repeat 10 /usr/bin/install /tmp/hh /tmp/hh2' syscall seconds calls errors sigreturn 0.000210349 10 10 fchmod 0.000451316 10 0 rename 0.002175952 10 0 copy_file_range 1.406585436 20 0 getrandom 0.000238228 10 0 minherit 0.000193221 10 0 execve 0.003492236 10 10 sigsuspend 1.471121187 10 10 umask 0.000017425 1 0 lseek 0.000472312 26 0 getpgrp 0.000018428 1 0 wait4 0.000570728 21 10 write 0.000028104 1 0 recvfrom 0.000035205 2 0 poll 0.000278623 2 0 sendto 0.000062127 2 0 connect 0.000048705 2 0 socket 0.000046869 2 0 getpid 0.000239035 13 0 getpeername 0.000016303 1 1 setitimer 0.000617037 32 0 fork 0.002382183 11 0 dup 0.001035424 56 0 pipe2 0.000026176 1 0 __getcwd 0.000020123 1 0 sigprocmask 0.001605172 87 0 sigaction 0.000847859 45 0 ioctl 0.000699790 35 6 dup2 0.000074504 4 0 getegid 0.000018361 1 0 geteuid 0.000036304 2 0 getgid 0.000019093 1 0 getuid 0.000017192 1 0 openat 0.001696058 35 1 getrusage 0.000443794 22 0 __sysctl 0.000128361 6 0 getdirentries 0.001201559 27 0 fcntl 0.000956225 52 0 fstatat 0.001112990 50 5 cpuset_getaffinity 0.000222640 11 0 readlink 0.000239186 11 11 sysarch 0.000200259 11 0 munmap 0.000825598 34 0 pread 0.000444490 13 0 close 0.004672428 202 33 read 0.004842528 40 0 fstat 0.002653986 133 56 open 0.002064127 87 16 sigfastblock 0.000192375 11 0 issetugid 0.000485598 28 0 mprotect 0.001482952 70 0 mmap 0.005792957 229 0 ------------- ------- ------- 2.923359118 1513 169 $ One interesting thing to note is that files containing random data are fast: $ dd if=/dev/random of=/tmp/hh bs=1 count=48894 48894+0 records in 48894+0 records out 48894 bytes transferred in 0.614548 secs (79561 bytes/sec) $ /usr/bin/time csh -c 'repeat 10 /usr/bin/install /tmp/hh /tmp/hh2' 0.03 real 0.00 user 0.03 sys $ /usr/bin/time csh -c 'repeat 10 /usr/bin/install /tmp/hh /tmp/hh2' 0.03 real 0.03 user 0.02 sys $ I don't remember seeing this problem in FreeBSD 13. I suspect it is somehow related to the fact that zfs in FreeBSD 14 defaults to compression=on, which even affects all old datasets that didn't set it off specifically. I first noticed this when running freebsd-update install, which took over 10 minutes to update 8850 files from 14.4-RELEASE-p5 to 14.4-RELEASE-p6.
I can't reproduce it at my pool (mirror of two ssd disks): % /usr/bin/time csh -c 'repeat 10 /usr/bin/install /tmp/hh /tmp/hh2' 0.03 real 0.01 user 0.02 sys Which disks are you using? Please also attach output from: zdb -C syspool | grep ashift zpool get all syspool zfs get all syspool/system/tmp
Also try to set atime=off on your dataset and repeat tests.
(In reply to Oleksandr Kryvulia from comment #1) The disks are Seagate Constellation ES.3 model ST2000NM0033-9ZM175. The ashift is 12.
Created attachment 272452 [details] The output of zpool get all syspool
Created attachment 272453 [details] The output of zfs get all syspool/system/tmp
(In reply to Oleksandr Kryvulia from comment #2) It is fast if I set atime=off. It is also fast if I set compression=off and recreate the test file (so it's not compressed). The output of procstat -kk looks like this when it is slow: PID TID COMM TDNAME KSTACK 9951 101982 install - mi_switch+0xbd _cv_wait+0xfb txg_wait_synced_impl+0xb9 txg_wait_synced+0xb dmu_offset_next+0x80 zfs_holey+0x11a zfs_freebsd_ioctl+0x4b vn_generic_copy_file_range+0x67d vn_copy_file_range+0x25d kern_copy_file_range+0x3be sys_copy_file_range+0x78 amd64_syscall+0x117 fast_syscall_common+0xf8
(In reply to martin from comment #6) Try setting vfs.zfs.dmu_offset_next_sync=0 This makes SEEK_DATA/SEEK_HOLE run much faster, but can miss a hole on recently written data. (For copy_file_range(), that means it might not maintain the hole and write bytes of zeros to the copied file.)
(In reply to Rick Macklem from comment #7) Yes, that makes it fast. I'm not sure how frequent holes are, but maybe adding a "hole was created since the last sync" flag would allow it to decide whether to force a sync before looking for holes? Is there an explanation why setting compression=off makes it fast?
(In reply to martin from comment #8) I'm not sure how frequent holes are, but maybe adding a "hole was created since the last sync" flag would allow it to decide whether to force a sync before looking for holes? Is there an explanation why setting compression=off makes it fast? You've mistaken me for someone that knows how ZFS works.;-) You might try asking on OpenZFS? (I am aware that they know there are issues with performance for SEEK_DATA/SEEK_HOLE and related things.)
(In reply to Oleksandr Kryvulia from comment #1) Are you running a newer system than 14.4? I ask because once block cloning is enabled, the copy should be handled with that. # sysctl vfs.zfs.bclone_enabled should show you if block cloning is enabled.
(In reply to Rick Macklem from comment #10) I have only one system with these symptoms and it is running 14.4 (vfs.zfs.bclone_enabled=0)
(In reply to Rick Macklem from comment #10) No, like Oleksandr, I am only running 14.4 and vfs.zfs.bclone_enabled=0. I tried setting it to 1 while running the test and that also makes it fast (hopefully that hasn't damaged the pool!).