I am running the stable version of Freebsd 13. For a while now, when using the ccache port, the cache size is incorrect. FreeBSD fbsdnvme 13.0-STABLE FreeBSD 13.0-STABLE #3 stable/13-n249224-f6e755d1b26: Thu Feb 3 11:28:42 PST 2022 root@fbsdnvme:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 brian@fbsdnvme:~ $ pkg info | grep ccache ccache-3.7.12_2 Tool to minimize the compile time of C/C++ programs root@fbsdnvme:~ # du -h -d1 .ccache/ && ccache -s 118M .ccache/e 140M .ccache/5 126M .ccache/8 133M .ccache/2 127M .ccache/b 126M .ccache/f 128M .ccache/6 122M .ccache/1 132M .ccache/a 133M .ccache/c 128M .ccache/3 118M .ccache/9 135M .ccache/4 118M .ccache/d 512B .ccache/tmp 118M .ccache/0 130M .ccache/7 2.0G .ccache/ cache directory /root/.ccache primary config /root/.ccache/ccache.conf secondary config (readonly) /usr/local/etc/ccache.conf stats updated Thu Feb 3 16:05:01 2022 stats zeroed Thu Feb 3 15:55:19 2022 cache hit (direct) 11976 cache hit (preprocessed) 30 cache miss 0 cache hit rate 100.00 % cleanups performed 0 files in cache 138624 cache size 438.3 kB max cache size 5.0 GB 438KB is nowhere near 2.0GB
Turns out this manifests itself if one does not change the CCACHE_DIR. If I do not modify /etc/profile or /etc/csh.cshrc then ccache uses /root/.ccache. If I specify these vars, to /usr/.ccache for example, then the cache size is displayed properly.
I do have this even with CCACHE_DIR *is* changed.
OK my repro steps are as follows. I am doing this on an esxi VM typically. Build freebsd 13 VM with 4 cores and 8gb ram Install OS, not too much atypical here, I select a 50GB thin disk, select ntpdate and ntpd on boot and random pid in the install options. do all the below as root. freebsd-update fetch && freebsd-update install reboot pkg install git ccache git bash open-vm-tools-nox11 reboot git clone for stable/13 in /usr/src add WITH_CCACHE_BUILD=yes in /etc/make.conf add export CCACHE_DIR=/usr/.ccache in /etc/profile add setenv CCACHE_DIR "/usr/.ccache" in /etc/csh.cshrc run a preboot script I have which does this. ccache -z && cd /usr/src && git pull && make -j2 buildworld && make -j2 kernel In another window run bash and then watch the output of the below. while true;do ccache -s;sleep 1;done
In response to this not working with CCACHE_DIR set, what I saw is this. If I use UFS, which I often do, then the stats are reported properly. If I use ZFS, then the stats do not work, even if CCACHE_DIR is set.
I now see that another build on the zfs 5x80gb 5 drive z3 setup does properly show stats in subsequent rebuilds. I will install the port and see what else I can figure out.
discovered that running ccache -c fixes this.
To reporter: please change Importance field to "Affects some people". I'm 3rd here with issue: user@freebsdvm:~$ freebsd-version -ku; pkg info ccache | head -1; sudo -u pbuild env CCACHE_DIR=/var/cache/ccache/poudriere ccache -s; du -hs /var/cache/ccache/poudriere/ 13.1-RELEASE 13.1-RELEASE ccache-3.7.12_3 cache directory /var/cache/ccache/poudriere primary config /var/cache/ccache/poudriere/ccache.conf secondary config (readonly) /usr/local/etc/ccache.conf stats updated Thu Jun 23 16:35:58 2022 cache hit (direct) 134 cache hit (preprocessed) 458 cache miss 6270 cache hit rate 8.63 % called for link 933 called for preprocessing 738 multiple source files 2 compiler produced empty output 3 compile failed 474 preprocessor error 137 bad compiler arguments 52 unsupported source language 8 autoconf compile/link 1585 unsupported compiler option 7 no input file 289 cleanups performed 0 files in cache 14284 cache size 3.2 MB max cache size 10.7 GB 130M /var/cache/ccache/poudriere/ user@freebsdvm:~$ 130MiB (compressed by ZFS) is a bit bigger that 3.2MB reported by ccache.
(In reply to Brian Whalen from comment #6) > discovered that running ccache -c fixes this. Permanently? BTW, `-c` should not be normally used: https://ccache.dev/manual/3.7.12.html#_cache_statistics -c, --cleanup Clean up the cache by removing old cached files until the specified file number and cache size limits are not exceeded. This also recalculates the cache file count and size totals. Normally, there is no need to initiate cleanup manually as ccache keeps the cache below the specified limits at runtime and *keeps statistics up to date on each compilation. Forcing a cleanup is mostly useful if you manually modify the cache contents or believe that the cache size statistics may be inaccurate.* Yes, we actually believe that stats are inaccurate, but I don't want to run cleanup manually each time I compile something. Should just work.
My ccache settings: user@freebsdvm:~$ sudo -u pbuild env CCACHE_DIR=/var/cache/ccache/poudriere ccache -p (default) base_dir = (environment) cache_dir = /var/cache/ccache/poudriere (default) cache_dir_levels = 2 (default) compiler = (default) compiler_check = mtime (/var/cache/ccache/poudriere/ccache.conf) compression = false (default) compression_level = 6 (default) cpp_extension = (default) debug = false (default) depend_mode = false (default) direct_mode = true (default) disable = false (default) extra_files_to_hash = (default) hard_link = false (default) hash_dir = true (default) ignore_headers_in_manifest = (default) keep_comments_cpp = false (default) limit_multiple = 0.8 (default) log_file = (default) max_files = 0 (/var/cache/ccache/poudriere/ccache.conf) max_size = 10.7G (default) path = (default) pch_external_checksum = false (default) prefix_command = (default) prefix_command_cpp = (default) read_only = false (default) read_only_direct = false (default) recache = false (default) run_second_cpp = true (default) sloppiness = (default) stats = true (default) temporary_dir = (default) umask =
(In reply to Anton Saietskii from comment #7) Add one more (ccache on ZFS)
(In reply to Anton Saietskii from comment #8) >> discovered that running ccache -c fixes this. > Permanently? No, it doesn't.
This affects me, running 14-CURRENT, with CCACHE_DIR on ZFS with feature lz4_compress active.
(In reply to Michael Grimm from comment #10) > Add one more (ccache on ZFS) Didn't get that. Could you please provide more details on what do you mean?
BTW, works as expected on releng/11.4, ccache 3.7.1: freebsd-version -ku; pkg info ccache | head -1; zfs get compression,mountpoint zroot/ccache; setenv CCACHE_DIR /var/cache/ccache/poudriere/; ccache -s; ccache -p; find /var/cache/ccache/poudriere/ -type f -ls | awk '{sum+=$7;} END {print sum/1024^2" MiB";}' 11.4-RELEASE-p6 11.4-RELEASE-p6 ccache-3.7.1_1 NAME PROPERTY VALUE SOURCE zroot/ccache compression lz4 inherited from zroot zroot/ccache mountpoint /var/cache/ccache local cache directory /var/cache/ccache/poudriere/ primary config /var/cache/ccache/poudriere//ccache.conf secondary config (readonly) /usr/local/etc/ccache.conf stats updated Thu Apr 8 16:10:13 2021 cache hit (direct) 28958 cache hit (preprocessed) 13275 cache miss 34461 cache hit rate 55.07 % called for link 14196 called for preprocessing 1386 multiple source files 17 compiler produced stdout 2 compiler produced empty output 24 compile failed 4392 preprocessor error 3401 couldn't find the compiler 7 cache file missing 88 bad compiler arguments 423 unsupported source language 23 autoconf compile/link 17361 no input file 2160 cleanups performed 268 files in cache 17986 cache size 202.0 MB max cache size 220.2 MB (default) base_dir = (environment) cache_dir = /var/cache/ccache/poudriere/ (default) cache_dir_levels = 2 (default) compiler = (default) compiler_check = mtime (default) compression = false (default) compression_level = 6 (default) cpp_extension = (default) debug = false (default) depend_mode = false (default) direct_mode = true (default) disable = false (default) extra_files_to_hash = (default) hard_link = false (default) hash_dir = true (default) ignore_headers_in_manifest = (default) keep_comments_cpp = false (default) limit_multiple = 0.8 (default) log_file = (default) max_files = 0 (/var/cache/ccache/poudriere//ccache.conf) max_size = 220.2M (default) path = (default) pch_external_checksum = false (default) prefix_command = (default) prefix_command_cpp = (default) read_only = false (default) read_only_direct = false (default) recache = false (default) run_second_cpp = true (default) sloppiness = (default) stats = true (default) temporary_dir = (default) umask = (default) unify = false 165,385 MiB
looking at this again, running ccache -c is a momentary fix. If you want me to poke at or try things I can.
from a 12.0 system I see it as well. root@fcctest:~ # du -sh ; echo " ";ccache -s 29K . cache directory /usr/.ccache primary config /usr/.ccache/ccache.conf secondary config (readonly) /usr/local/etc/ccache.conf stats updated Fri Aug 5 22:08:21 2022 cache hit (direct) 0 cache hit (preprocessed) 0 cache miss 4247 cache hit rate 0.00 % ccache internal error 1 unsupported code directive 1 cleanups performed 0 files in cache 12708 cache size 0.0 kB max cache size 5.0 GB
These results are incorrect in 13.1 with CCACHE_DIR either left at default or set to /usr/.ccache
(In reply to Brian Whalen from comment #18) I'm afraid decription is incorrect. As you can see in my comment #7, stats incorrect with /var/cache/ccache as well. I believe it doesn't depend on CCACHE_DIR at all.
Experimenting with older versions to better ID when this broke. I installed 12.0 with ports and src and was able to make install clean in /usr/ports/devel/ccache. I have only had the /usr/src compile running for a little bit but it does seem to work. The setup text file speaks of modifying /etc/src/conf whereas the newer versions of the file do not say that. 12.0: ccache ver 3.4.2_1 To use ccache for ports, just add WITH_CCACHE_BUILD=yes to /etc/make.conf. The rest of this guide is for building /usr/src and other checkouts. To use ccache for base (11.0+) just add WITH_CCACHE_BUILD=yes to /etc/src.conf. Refer to src.conf(5) for more information. 13-stable with a recent package says To use ccache for ports and base, just add WITH_CCACHE_BUILD=yes to /etc/make.conf. The rest of this guide is for building /usr/src and other checkouts. Refer to src.conf(5) for more information specific to base builds. Modifying src.conf in 13 did not help me but the change is noteworthy.
In a 12.3 release system again with src and ports installed, I did a make install clean of the ccache port. This got me 3.7.12_2. The read me file for ccache leads off with the below. To use ccache for ports and base, just add WITH_CCACHE_BUILD=yes to /etc/make.conf. The rest of this guide is for building /usr/src and other checkouts. Refer to src.conf(5) for more information specific to base builds. I tried adding a src.conf with identical text and it does not work.
12.1 got me ver 3.7.1 and 12.2 got me ver 3.7.1_1. Both of these worked with just a src.conf add. So it would seem that somewhere between 3.7.1_1 and 3.7.12_2 breakage occurred.
As expected, 13.0, which comes with 3.7.1_1. works. 13.1 fails. I am not a developer but wonder about the change from src.conf to make.conf which brings failure along with it. Adding a src.conf file with content specified for make.conf still fails in 13.1.
(In reply to Brian Whalen from comment #23) > As expected, 13.0, which comes with 3.7.1_1. works. > 13.1 fails. I went though git log briefly between 3.7.1_1 and 3.7.12. I believe there's no functional changes, no custom patches changes, nothing except ccache version update. > I am not a developer but wonder about the change from src.conf to make.conf > which brings failure along with it. Adding a src.conf file with content > specified for make.conf still fails in 13.1. You actually just proved that issue doesn't depend on "where WITH_CCACHE_BUILD is defined". And it should not, since all that moving definition to make.conf does -- makes variable visible for make(1) regardless of directory where it called. So, looks like the issue is in ccache 3.7.12 itself. I tried to google this, but didn't find anything (issue is FreeBSD-specific or Linux guys don't care about it?) Wanted to try ccache 3.7.1 (both with PORTREVISIONs _1 and _2) on releng/13.1, but lacking of time for that currently...
git bisect identified the commit "Fix CCACHE_MAXSIZE with filesystem compression (#444)": https://github.com/ccache/ccache/commit/14d4371b50206fe9dfba45222b63a3d01605e16c It seems that st->st_blocks is always 1 on ZFS when called from ccache (at least with compression enabled). If I call stat(2) in a test program after compilation finished, st_blocks values look correct. My best guess is that ZFS updates st_blocks asynchronously. That would explain why manual cache cleanup reports the correct sizes.
This would appear to be a ZFS issue. I see that with UFS this works as expected. I built a 13.1 system with UFS and DOS selected at install and with a make -j2 buildworld I see the size incrementing in ccache -s.
Thanx Anton, I did more testing based on your comments. Confirming yesterdays findings: I built a system with 13.1 with ufs and used current packages to install git, bash, ccache, and open-vm-tools-nox11. I did an initial world and kernel build on the 13.1 release. After completion, the stats were as below. root@f13ufs:~ # ccache -s cache directory /root/.ccache primary config /root/.ccache/ccache.conf secondary config (readonly) /usr/local/etc/ccache.conf stats updated Fri Sep 2 18:25:50 2022 cache hit (direct) 3 cache hit (preprocessed) 83 cache miss 49053 cache hit rate 0.18 % called for link 113 unsupported code directive 6 no input file 1 cleanups performed 0 files in cache 146126 cache size 5.2 GB max cache size 10.0 GB I didn't do a du there but the fact that the cache size was not zero or very small was encouraging. I rebooted, did the installworld bits and rebooted. After the first build and boot my system was at the below. [root@f13ufs ~]# uname -a FreeBSD f13ufs 13.1-STABLE FreeBSD 13.1-STABLE #0 stable/13-n252237-5d45f99a1ed: Fri Sep 2 17:57:48 PDT 2022 root@f13ufs:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 I did a ccache -z and a 2nd world and kernel build. Afterward the stats were: [root@f13ufs ~]# ccache -s cache directory /root/.ccache primary config /root/.ccache/ccache.conf secondary config (readonly) /usr/local/etc/ccache.conf stats updated Fri Sep 2 19:59:38 2022 stats zeroed Fri Sep 2 18:34:42 2022 cache hit (direct) 3 cache hit (preprocessed) 83 cache miss 46864 cache hit rate 0.18 % called for link 113 unsupported code directive 4 no input file 1 cleanups performed 7 files in cache 275201 cache size 9.0 GB max cache size 10.0 GB This is a close to reality value for the size. [root@f13ufs ~]# du -sh ~/.ccache/ 8.4G /root/.ccache/ I did a 2nd cycle of reboot, installworld, and reboot. My system was at: brian@f13ufs:~ $ uname -a FreeBSD f13ufs 13.1-STABLE FreeBSD 13.1-STABLE #1 stable/13-n252244-5ae39c02f22: Fri Sep 2 19:59:28 PDT 2022 root@f13ufs:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 I did a 3rd build world and kernel with the below stats afterward. [root@f13ufs ~]# ccache -s cache directory /root/.ccache primary config /root/.ccache/ccache.conf secondary config (readonly) /usr/local/etc/ccache.conf stats updated Fri Sep 2 20:44:05 2022 stats zeroed Fri Sep 2 20:11:50 2022 cache hit (direct) 45427 cache hit (preprocessed) 1498 cache miss 2 cache hit rate 100.00 % called for link 113 unsupported code directive 4 no input file 1 cleanups performed 0 files in cache 275493 cache size 9.0 GB max cache size 10.0 GB [root@f13ufs ~]# du -sh ~/.ccache/ 8.4G /root/.ccache/
Created attachment 236356 [details] test program printing st_blocks The attached test program demonstrates that it takes several seconds until ZFS reports the correct value of st_blocks via fstat(2). $ cc stat-test.c -o stat-test $ ./stat-test [0] st_blocks=1 [1] st_blocks=1 [2] st_blocks=1 [3] st_blocks=9 [4] st_blocks=9 [5] st_blocks=9 [6] st_blocks=9 [7] st_blocks=9 [8] st_blocks=9 [9] st_blocks=9 I wonder if this a FreeBSD-specific issue or ZFS in general - I don't have a Linux+ZFS setup. If the latter is true, I guess it's just the way ZFS works and hard to fix. We can restore the old behavior where uncompressed size is used. Are there any file systems (other than ZFS) in FreeBSD that support compression?
I will report more on final results but with a 13.1 zfs based system I see the below. After about 48 minutes the real usage is 875M but the size has barely budged off of its zero starting point. root@f13z:~ # date && ps auxwww | grep buildworld && du -sh ~/.ccache/ && ccache -s Sun Sep 4 22:20:35 PDT 2022 root 864 0.0 0.0 12948 2812 v0 S+ 21:31 0:02.68 make -j6 buildworld root 901 0.0 0.0 12948 3484 v0 S 21:31 0:01.21 make -m /usr/src/share/mk -f Makefile.inc1 TARGET=amd64 TARGET_ARCH=amd64 buildworld root 45044 0.0 0.0 12840 2392 0 S+ 22:20 0:00.00 grep buildworld 875M /root/.ccache/ cache directory /root/.ccache primary config /root/.ccache/ccache.conf secondary config (readonly) /usr/local/etc/ccache.conf stats updated Sun Sep 4 22:20:35 2022 cache hit (direct) 0 cache hit (preprocessed) 0 cache miss 19047 cache hit rate 0.00 % unsupported code directive 2 cleanups performed 0 files in cache 56855 cache size 28.7 kB max cache size 5.0 GB
(In reply to Stefan Ehmann from comment #28) >> Are there any file systems (other than ZFS) in FreeBSD that support compression? I'm afraid no. At least with write access.