ci: ccache does not work for macOS cross-compiling builds

hebasto commented at 4:25 am on March 30, 2021: member

https://cirrus-ci.com/task/6268895175639040?logs=ci#L3869:

 0ccache version 3.7.7
 1cache directory                     /tmp/ccache_dir
 2primary config                      /tmp/ccache_dir/ccache.conf
 3secondary config      (readonly)    /etc/ccache.conf
 4stats updated                       Tue Mar 30 01:48:58 2021
 5stats zeroed                        Tue Mar 30 01:21:25 2021
 6cache hit (direct)                     0
 7cache hit (preprocessed)               0
 8cache miss                             0
 9cache hit rate                      0.00 %
10called for link                       10
11multiple source files                636
12cleanups performed                     0
13files in cache                         0
14cache size                           0.0 kB
15max cache size                     200.0 MB

UPD 2023-08-25. See https://github.com/ccache/ccache/issues/1326.

hebasto added the label Bug on Mar 30, 2021

hebasto added the label Tests on Mar 30, 2021

hebasto added the label macOS on Mar 30, 2021

hebasto commented at 10:23 am on April 2, 2021: member

This line

0multiple source files                636

describes a little. According to ccache docs:

The compiler was called to compile multiple source files in one go. This is not supported by ccache.

hebasto commented at 10:39 am on April 2, 2021: member

@dongcarl

ccache behavior was broken in 300733921863c176535806c40afdc813b99e7459 (#19683).

dongcarl commented at 8:02 pm on April 5, 2021: contributor

Was this not the case before 300733921863c176535806c40afdc813b99e7459? I’m not entirely sure how that commit makes it so that “the compiler was called to compile multiple source files in one go”

hebasto commented at 11:33 pm on April 5, 2021: member

I’m not entirely sure how that commit makes it so that “the compiler was called to compile multiple source files in one go”

I believe this is because ccache do not support invoking a compiler that prefixed with env command.

FWIW, if I inject ccache between env and clang, it works fine.

dongcarl commented at 0:33 am on April 9, 2021: contributor

FWIW, if I inject ccache between env and clang, it works fine.

What is the ccache stats output if you inject ccache between env and clang?

hebasto commented at 11:07 am on April 9, 2021: member

@dongcarl

With the following patch

 0diff --git a/depends/hosts/darwin.mk b/depends/hosts/darwin.mk
 1index dd71697f0..8a7c3a791 100644
 2--- a/depends/hosts/darwin.mk
 3+++ b/depends/hosts/darwin.mk
 4@@ -98,6 +98,7 @@ $(foreach TOOL,$(cctools_TOOLS),$(eval darwin_$(TOOL) = $$(build_prefix)/bin/$$(
 5 darwin_CC=env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH \
 6               -u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH \
 7               -u LIBRARY_PATH \
 8+            /usr/bin/ccache \
 9             $(clang_prog) --target=$(host) -mmacosx-version-min=$(OSX_MIN_VERSION) \
10               -B$(build_prefix)/bin -mlinker-version=$(LD64_VERSION) \
11               --sysroot=$(OSX_SDK) \
12@@ -106,6 +107,7 @@ darwin_CC=env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH \
13 darwin_CXX=env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH \
14                -u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH \
15                -u LIBRARY_PATH \
16+            /usr/bin/ccache \
17              $(clangxx_prog) --target=$(host) -mmacosx-version-min=$(OSX_MIN_VERSION) \
18                -B$(build_prefix)/bin -mlinker-version=$(LD64_VERSION) \
19                --sysroot=$(OSX_SDK) \

the first run

 0$ CONFIG_SITE=$PWD/depends/x86_64-apple-darwin18/share/config.site ./configure --disable-ccache
 1$ ccache --zero-stats
 2$ make
 3$ ccache --show-stats 
 4cache directory                     /home/hebasto/.ccache
 5primary config                      /home/hebasto/.ccache/ccache.conf
 6secondary config      (readonly)    /etc/ccache.conf
 7stats updated                       Fri Apr  9 14:03:35 2021
 8stats zeroed                        Fri Apr  9 13:59:19 2021
 9cache hit (direct)                   521
10cache hit (preprocessed)               1
11cache miss                           647
12cache hit rate                     44.65 %
13called for link                       12
14cleanups performed                     2
15files in cache                     24070
16cache size                          13.0 GB
17max cache size                      15.0 GB

the second run

 0$ make clean
 1$ ccache --zero-stats
 2$ make
 3$ ccache --show-stats 
 4cache directory                     /home/hebasto/.ccache
 5primary config                      /home/hebasto/.ccache/ccache.conf
 6secondary config      (readonly)    /etc/ccache.conf
 7stats updated                       Fri Apr  9 14:04:47 2021
 8stats zeroed                        Fri Apr  9 14:04:15 2021
 9cache hit (direct)                   644
10cache hit (preprocessed)               2
11cache miss                             1
12cache hit rate                     99.85 %
13called for link                       11
14cleanups performed                     0
15files in cache                     24073
16cache size                          13.0 GB
17max cache size                      15.0 GB

dongcarl commented at 6:41 pm on May 13, 2021: contributor

Urgh… That kinda sucks… I think the easiest solution is perhaps to wait until ccache 4.1 is available on all distros and set compiler_type/CCACHE_COMPILERTYPE?

prusnak commented at 7:59 pm on March 18, 2022: contributor

to wait until ccache 4.1 is available on all distros @dongcarl What do you mean by “all distros”?

dongcarl commented at 8:04 pm on March 18, 2022: contributor

Sorry, probably better rephrased as “available on stable versions of major distros”… Sometimes I think we might need a configure script for depends lol

prusnak commented at 8:14 pm on March 18, 2022: contributor

“available on stable versions of major distros”

Debian 10 (old stable) has ccache 3.6
Debian 11 (stable) has ccache 4.2
Ubuntu 20.04 (latest LTS) has ccache 3.7.7
Ubuntu 21.04 has ccache 4.2
Ubuntu 21.10 has ccache 4.2.1
Ubuntu 22.04 (upcoming LTS) has ccache 4.5.1

src: https://repology.org/project/ccache/versions

hebasto commented at 12:12 pm on March 20, 2022: member

@dongcarl

Urgh… That kinda sucks… I think the easiest solution is perhaps to wait until ccache 4.1 is available on all distros and set compiler_type/CCACHE_COMPILERTYPE?

I’ve tested this approach on Ubuntu 22.04 with ccache 4.5.1, and it doesn’t work.

OTOH, another approach has been implemented in #24620.

Sjors commented at 8:39 am on June 16, 2022: member

Not just when cross-compiling. I made a separate issue #25387, because I’m not sure if it’s the same (ccache 4.6.1).

(nvm: it does work as far as I can tell, but doesn’t use it by default)

fanquake commented at 9:06 am on August 16, 2023: member

Given it’s not in the description, can someone summarize the current state of this issue?

What doesn’t work?
What ccache options have been tried (and don’t work)? Taking into account recent ccache releases.
Have we opened an issue upstream to either report that ccache is broken (for us), or that we have a usecase which isn’t satified by it’s current feature/option set, to see if they are open to adding a new feature/option?
If there is any (related) upstream discussions/issues, can somebody link to them here?

hebasto commented at 11:16 am on August 25, 2023: member

Given it’s not in the description, can someone summarize the current state of this issue?

What doesn’t work?

Have we opened an issue upstream to either report that ccache is broken (for us), or that we have a usecase which isn’t satified by it’s current feature/option set, to see if they are open to adding a new feature/option?

If there is any (related) upstream discussions/issues, can somebody link to them here?

See https://github.com/ccache/ccache/issues/1326.

hebasto commented at 12:15 pm on August 25, 2023: member

Urgh… That kinda sucks… I think the easiest solution is perhaps to wait until ccache 4.1 is available on all distros and set compiler_type/CCACHE_COMPILERTYPE?

Tested with ccache 4.8.2. It does not work.

hebasto referenced this in commit ee4dc7d928 on Sep 11, 2023

fanquake referenced this in commit ba76fc8bda on Jul 15, 2024

fanquake referenced this in commit b9673bfd52 on Jul 15, 2024

fanquake referenced this in commit 4ad14a83c6 on Jul 15, 2024

fanquake referenced this in commit feeefcf2a3 on Jul 15, 2024

fanquake referenced this in commit 0d25a263c8 on Jul 15, 2024

fanquake referenced this in commit aff4a88fd5 on Jul 17, 2024

fanquake referenced this in commit 18609bf2ca on Jul 17, 2024

fanquake referenced this in commit 23c0b07656 on Jul 17, 2024

fanquake referenced this in commit 298c8192a7 on Jul 20, 2024

fanquake referenced this in commit bc78b59c51 on Jul 22, 2024

fanquake referenced this in commit ee269f2d96 on Jul 24, 2024

fanquake referenced this in commit bda537f7c4 on Jul 25, 2024

fanquake referenced this in commit 36cc1097c1 on Jul 25, 2024

fanquake closed this on Jul 30, 2024

fanquake referenced this in commit 4c62f4b535 on Jul 30, 2024

ci: ccache does not work for macOS cross-compiling builds #21552