ci: ccache does not work for macOS cross-compiling builds #21552

https://cirrus-ci.com/task/6268895175639040?logs=ci#L3869:

ccache version 3.7.7
cache directory                     /tmp/ccache_dir
primary config                      /tmp/ccache_dir/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
stats updated                       Tue Mar 30 01:48:58 2021
stats zeroed                        Tue Mar 30 01:21:25 2021
cache hit (direct)                     0
cache hit (preprocessed)               0
cache miss                             0
cache hit rate                      0.00 %
called for link                       10
multiple source files                636
cleanups performed                     0
files in cache                         0
cache size                           0.0 kB
max cache size                     200.0 MB

UPD 2023-08-25. See https://github.com/ccache/ccache/issues/1326.

This line

multiple source files                636

describes a little. According to ccache docs:

The compiler was called to compile multiple source files in one go. This is not supported by ccache.

@dongcarl

ccache behavior was broken in 300733921863c176535806c40afdc813b99e7459 (#19683).

Was this not the case before 300733921863c176535806c40afdc813b99e7459? I'm not entirely sure how that commit makes it so that "the compiler was called to compile multiple source files in one go"

I'm not entirely sure how that commit makes it so that "the compiler was called to compile multiple source files in one go"

I believe this is because ccache do not support invoking a compiler that prefixed with env command.

FWIW, if I inject ccache between env and clang, it works fine.

FWIW, if I inject ccache between env and clang, it works fine.

What is the ccache stats output if you inject ccache between env and clang?

@dongcarl

With the following patch

diff --git a/depends/hosts/darwin.mk b/depends/hosts/darwin.mk
index dd71697f0..8a7c3a791 100644
--- a/depends/hosts/darwin.mk
+++ b/depends/hosts/darwin.mk
@@ -98,6 +98,7 @@ $(foreach TOOL,$(cctools_TOOLS),$(eval darwin_$(TOOL) = $$(build_prefix)/bin/$$(
 darwin_CC=env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH \
               -u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH \
               -u LIBRARY_PATH \
+            /usr/bin/ccache \
             $(clang_prog) --target=$(host) -mmacosx-version-min=$(OSX_MIN_VERSION) \
               -B$(build_prefix)/bin -mlinker-version=$(LD64_VERSION) \
               --sysroot=$(OSX_SDK) \
@@ -106,6 +107,7 @@ darwin_CC=env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH \
 darwin_CXX=env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH \
                -u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH \
                -u LIBRARY_PATH \
+            /usr/bin/ccache \
              $(clangxx_prog) --target=$(host) -mmacosx-version-min=$(OSX_MIN_VERSION) \
                -B$(build_prefix)/bin -mlinker-version=$(LD64_VERSION) \
                --sysroot=$(OSX_SDK) \

• the first run

$ CONFIG_SITE=$PWD/depends/x86_64-apple-darwin18/share/config.site ./configure --disable-ccache
$ ccache --zero-stats
$ make
$ ccache --show-stats 
cache directory                     /home/hebasto/.ccache
primary config                      /home/hebasto/.ccache/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
stats updated                       Fri Apr  9 14:03:35 2021
stats zeroed                        Fri Apr  9 13:59:19 2021
cache hit (direct)                   521
cache hit (preprocessed)               1
cache miss                           647
cache hit rate                     44.65 %
called for link                       12
cleanups performed                     2
files in cache                     24070
cache size                          13.0 GB
max cache size                      15.0 GB

• the second run

$ make clean
$ ccache --zero-stats
$ make
$ ccache --show-stats 
cache directory                     /home/hebasto/.ccache
primary config                      /home/hebasto/.ccache/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
stats updated                       Fri Apr  9 14:04:47 2021
stats zeroed                        Fri Apr  9 14:04:15 2021
cache hit (direct)                   644
cache hit (preprocessed)               2
cache miss                             1
cache hit rate                     99.85 %
called for link                       11
cleanups performed                     0
files in cache                     24073
cache size                          13.0 GB
max cache size                      15.0 GB

Urgh... That kinda sucks... I think the easiest solution is perhaps to wait until ccache 4.1 is available on all distros and set compiler_type/CCACHE_COMPILERTYPE?

to wait until ccache 4.1 is available on all distros @dongcarl What do you mean by "all distros"?

Sorry, probably better rephrased as "available on stable versions of major distros"... Sometimes I think we might need a configure script for depends lol

"available on stable versions of major distros"

• Debian 10 (old stable) has ccache 3.6
• Debian 11 (stable) has ccache 4.2
• Ubuntu 20.04 (latest LTS) has ccache 3.7.7
• Ubuntu 21.04 has ccache 4.2
• Ubuntu 21.10 has ccache 4.2.1
• Ubuntu 22.04 (upcoming LTS) has ccache 4.5.1

src: https://repology.org/project/ccache/versions

@dongcarl

Urgh... That kinda sucks... I think the easiest solution is perhaps to wait until ccache 4.1 is available on all distros and set compiler_type/CCACHE_COMPILERTYPE?

I've tested this approach on Ubuntu 22.04 with ccache 4.5.1, and it doesn't work.

OTOH, another approach has been implemented in #24620.

Not just when cross-compiling. I made a separate issue #25387, because I'm not sure if it's the same (ccache 4.6.1).

(nvm: it does work as far as I can tell, but doesn't use it by default)

Given it's not in the description, can someone summarize the current state of this issue?

• What doesn't work?
• What ccache options have been tried (and don't work)? Taking into account recent ccache releases.
• Have we opened an issue upstream to either report that ccache is broken (for us), or that we have a usecase which isn't satified by it's current feature/option set, to see if they are open to adding a new feature/option?
• If there is any (related) upstream discussions/issues, can somebody link to them here?

Given it's not in the description, can someone summarize the current state of this issue?

• What doesn't work?

• Have we opened an issue upstream to either report that ccache is broken (for us), or that we have a usecase which isn't satified by it's current feature/option set, to see if they are open to adding a new feature/option?

• If there is any (related) upstream discussions/issues, can somebody link to them here?

See https://github.com/ccache/ccache/issues/1326.

Urgh... That kinda sucks... I think the easiest solution is perhaps to wait until ccache 4.1 is available on all distros and set compiler_type/CCACHE_COMPILERTYPE?

Tested with ccache 4.8.2. It does not work.