Bug 246245 - Mk/bsd.ccache.mk: Increase ccache hit ratio
Summary: Mk/bsd.ccache.mk: Increase ccache hit ratio
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Ports Framework (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Many People
Assignee: Port Management Team
URL: https://github.com/ccache/ccache/issu...
Keywords: needs-qa, performance
Depends on:
Blocks:
 
Reported: 2020-05-06 03:53 UTC by Ivan Rozhuk
Modified: 2024-04-20 17:15 UTC (History)
10 users (show)

See Also:
koobs: exp-run?


Attachments
patch (721 bytes, patch)
2020-05-06 03:53 UTC, Ivan Rozhuk
no flags Details | Diff
patch (1.11 KB, patch)
2023-12-29 21:38 UTC, Ivan Rozhuk
rozhuk.im: maintainer-approval?
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Ivan Rozhuk 2020-05-06 03:53:24 UTC
Created attachment 214185 [details]
patch

Make ccache more efficient.
https://ccache.dev/manual/3.7.9.html#_compiling_in_different_directories

Port system builds every port in folders with different names, this broke ccache hit ratio.
With this patch last chromium update take ~20 minutes, not 1 hour 30 minutes as usual.
Comment 1 Po-Chuan Hsieh freebsd_committer freebsd_triage 2020-05-06 15:44:42 UTC
Assign PR to the maintainer.
Comment 2 Bryan Drewery freebsd_committer freebsd_triage 2020-06-15 17:07:04 UTC
Yes this patch makes sense. I haven't tested it though.
Comment 3 Mathieu Arnold freebsd_committer freebsd_triage 2020-06-15 17:35:06 UTC
Do we need an exp-run because of the CFLAGS change?
Comment 4 Bryan Drewery freebsd_committer freebsd_triage 2020-06-15 17:40:47 UTC
(In reply to Mathieu Arnold from comment #3)

It wouldn't hurt. I do worry that it will regress some ports that somehow see
the MAKE_ENV but don't care about CFLAGS passed in. Not worth blocking it on those though as they are already losing all caching on version bumps due to changed workdir name.
Comment 5 dewayne 2020-06-16 02:15:13 UTC
(In reply to Bryan Drewery from comment #4)
The proposed CFLAG change option isn't available in the man pages of clang nor gcc9, so the begging question is: how will this change affect those that have a development/debugging environment.  (Hopefully none).  

Conversely for those that turn off debugging in their CFLAGS (-g0), does this option become a noop?  (I'm afraid that I'm overly cautious when something may be added to binaries)
Comment 6 Ivan Rozhuk 2020-06-16 03:44:15 UTC
(In reply to dewayne from comment #5)

-fdebug-prefix-map=${WRKSRC}=. - documented option.
https://gcc.gnu.org/install/configure.html
https://manpages.debian.org/experimental/binutils-common/as.1.en.html

Then some one make debug build in backtrace or whatever then will see main.c not /tmp/ports/***/***/someapp-1.33.55/main.c.
This allow ccache to cache files even then path to file changed.
In our port system path changed on app version update.
Comment 7 dewayne 2020-06-16 04:10:41 UTC
(In reply to rozhuk.im from comment #6)
Thank-you.  After reviewing your references and poking around in 
# find /usr/src/ | xargs grep debug-prefix-map
files I can see little to be concerned. Cheers.
Comment 8 Ivan Rozhuk 2020-12-20 20:41:47 UTC
Problem a more complex, current patch may improve cache hit ratio but does not work after port version change.
Comment 9 Bryan Drewery freebsd_committer freebsd_triage 2021-07-12 19:27:26 UTC
(In reply to Ivan Rozhuk from comment #8)

Can you elaborate? Which port version? What breaks?
Comment 10 Ivan Rozhuk 2021-07-13 13:26:16 UTC
(In reply to Bryan Drewery from comment #9)

https://github.com/ccache/ccache/issues/751

Then you build port version 1.2.3 it builds some where, and if this is out of source build than 1.2.3 go to the ccache metainfo as part of path to sources.
In next time then you will build version 1.2.4 ccache will cache miss because in meta stored 1.2.3 path to sources.

This issue affects cmake ports with outofsource build and may be some other ports.

Current version of patch may improve cache hit ratio for insource build ports.
Comment 11 Ivan Rozhuk 2021-07-13 13:48:01 UTC
IMHO this part of work is done.


To improve ccache hit ratio there is 2 ways:

1. Always build at same path, this mean do not use port version/hash for dir naming.

2. Step-by-step change outofsource build ports.
For cmake this means that in build dir we need create symlink to sources dir and use this symlink as path to sources.
Comment 12 Ivan Rozhuk 2023-12-29 21:38:27 UTC
Created attachment 247340 [details]
patch

Add CXXFLAGS.
Comment 13 Anton Saietskii 2024-01-03 11:48:00 UTC
(In reply to Ivan Rozhuk from comment #12)

According to ccache documentation [1], one must _either_ set hash_dir to false or pass -fdebug-prefix-map= to CFLAGS. Perhaps there's specific reason to use both options?

[1]: https://ccache.dev/manual/4.9.html#_compiling_in_different_directories
Comment 14 Ivan Rozhuk 2024-01-04 03:41:00 UTC
(In reply to Anton Saietskii from comment #13)

I do not see any disadvantages to set both same time in "hash_dir" description.