Bug 246245 - Mk/bsd.ccache.mk: Increase ccache hit ratio
Summary: Mk/bsd.ccache.mk: Increase ccache hit ratio
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Ports Framework (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Many People
Assignee: Port Management Team
URL: https://github.com/ccache/ccache/issu...
Keywords: needs-qa, performance
Depends on:
Reported: 2020-05-06 03:53 UTC by Ivan Rozhuk
Modified: 2021-07-13 13:48 UTC (History)
6 users (show)

See Also:
koobs: exp-run?

patch (721 bytes, patch)
2020-05-06 03:53 UTC, Ivan Rozhuk
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Ivan Rozhuk 2020-05-06 03:53:24 UTC
Created attachment 214185 [details]

Make ccache more efficient.

Port system builds every port in folders with different names, this broke ccache hit ratio.
With this patch last chromium update take ~20 minutes, not 1 hour 30 minutes as usual.
Comment 1 Po-Chuan Hsieh freebsd_committer 2020-05-06 15:44:42 UTC
Assign PR to the maintainer.
Comment 2 Bryan Drewery freebsd_committer 2020-06-15 17:07:04 UTC
Yes this patch makes sense. I haven't tested it though.
Comment 3 Mathieu Arnold freebsd_committer 2020-06-15 17:35:06 UTC
Do we need an exp-run because of the CFLAGS change?
Comment 4 Bryan Drewery freebsd_committer 2020-06-15 17:40:47 UTC
(In reply to Mathieu Arnold from comment #3)

It wouldn't hurt. I do worry that it will regress some ports that somehow see
the MAKE_ENV but don't care about CFLAGS passed in. Not worth blocking it on those though as they are already losing all caching on version bumps due to changed workdir name.
Comment 5 dewayne 2020-06-16 02:15:13 UTC
(In reply to Bryan Drewery from comment #4)
The proposed CFLAG change option isn't available in the man pages of clang nor gcc9, so the begging question is: how will this change affect those that have a development/debugging environment.  (Hopefully none).  

Conversely for those that turn off debugging in their CFLAGS (-g0), does this option become a noop?  (I'm afraid that I'm overly cautious when something may be added to binaries)
Comment 6 Ivan Rozhuk 2020-06-16 03:44:15 UTC
(In reply to dewayne from comment #5)

-fdebug-prefix-map=${WRKSRC}=. - documented option.

Then some one make debug build in backtrace or whatever then will see main.c not /tmp/ports/***/***/someapp-1.33.55/main.c.
This allow ccache to cache files even then path to file changed.
In our port system path changed on app version update.
Comment 7 dewayne 2020-06-16 04:10:41 UTC
(In reply to rozhuk.im from comment #6)
Thank-you.  After reviewing your references and poking around in 
# find /usr/src/ | xargs grep debug-prefix-map
files I can see little to be concerned. Cheers.
Comment 8 Ivan Rozhuk 2020-12-20 20:41:47 UTC
Problem a more complex, current patch may improve cache hit ratio but does not work after port version change.
Comment 9 Bryan Drewery freebsd_committer 2021-07-12 19:27:26 UTC
(In reply to Ivan Rozhuk from comment #8)

Can you elaborate? Which port version? What breaks?
Comment 10 Ivan Rozhuk 2021-07-13 13:26:16 UTC
(In reply to Bryan Drewery from comment #9)


Then you build port version 1.2.3 it builds some where, and if this is out of source build than 1.2.3 go to the ccache metainfo as part of path to sources.
In next time then you will build version 1.2.4 ccache will cache miss because in meta stored 1.2.3 path to sources.

This issue affects cmake ports with outofsource build and may be some other ports.

Current version of patch may improve cache hit ratio for insource build ports.
Comment 11 Ivan Rozhuk 2021-07-13 13:48:01 UTC
IMHO this part of work is done.

To improve ccache hit ratio there is 2 ways:

1. Always build at same path, this mean do not use port version/hash for dir naming.

2. Step-by-step change outofsource build ports.
For cmake this means that in build dir we need create symlink to sources dir and use this symlink as path to sources.