build: ccache doesn’t hit across build dirs #31994

issue fanquake openend this issue on March 5, 2025
  1. fanquake commented at 11:30 am on March 5, 2025: member
    This has regressed since 28.x. Opening an issue to track given #30861 has been closed. Tagged for 29.x because ideally this would be fixed, and any fix could be backported.
  2. fanquake added this to the milestone 29.0 on Mar 5, 2025
  3. hebasto commented at 1:04 pm on March 9, 2025: member

    TL;DR

    To enable ccache hits across different build directories on Linux or macOS, configure the ccache options as follows:

    0base_dir = <path_to_your_home_directory>
1hash_dir = false

    or via environment variables:

    0$ export CCACHE_BASEDIR=$HOME
1$ export CCACHE_NOHASHDIR=1

    Detailed Solution

    All following commands were executed on Ubuntu 24.04.2 LTS with ccache 4.9.1 installed.

    Consider building in two separate build directories with ccache debugging enabled:

     0$ export CCACHE_DEBUG=1
 1$ printenv | grep CCACHE
 2CCACHE_DEBUG=1
 3$ export CCACHE_DIR=$(mktemp -d)
 4$ cmake -B build_first -DWITH_CCACHE=ON
 5$ cmake --build build_first -t bitcoin_crypto -j $(nproc)
 6$ cmake -B build_second -DWITH_CCACHE=ON
 7$ ccache -z
 8$ cmake --build build_second -t bitcoin_crypto -j $(nproc)
 9$ ccache -s
10Cacheable calls:     20 /  20 (100.0%)
11  Hits:               0 /  20 ( 0.00%)
12    Direct:           0
13    Preprocessed:     0
14  Misses:            20 /  20 (100.0%)
15Local storage:
16  Cache size (GiB): 0.0 / 5.0 ( 0.09%)
17  Hits:               0 /  20 ( 0.00%)
18  Misses:            20 /  20 (100.0%)

    There were 100% misses. A comparison of the combined input used for hashing for an object file provides some clues:

     0$ diff -u build_first/src/crypto/CMakeFiles/bitcoin_crypto.dir/aes.cpp.o.*.ccache-input-text build_second/src/crypto/CMakeFiles/bitcoin_crypto.dir/aes.cpp.o.*.ccache-input-text
 1--- build_first/src/crypto/CMakeFiles/bitcoin_crypto.dir/aes.cpp.o.20250309_113307_205745.ccache-input-text	2025-03-09 11:33:07.669373196 +0000
 2+++ build_second/src/crypto/CMakeFiles/bitcoin_crypto.dir/aes.cpp.o.20250309_113435_261959.ccache-input-text	2025-03-09 11:34:35.664968636 +0000
 3@@ -10,7 +10,7 @@
 4 ### LANG
 5 en_US.UTF-8
 6 ### cwd
 7-/home/hebasto/dev/bitcoin/build_first
 8+/home/hebasto/dev/bitcoin/build_second
 9 === DIRECT MODE ===
10 ### cache entry version
11 1
12@@ -83,7 +83,7 @@
13 ### arg
14 -DENABLE_X86_SHANI
15 ### arg
16--I/home/hebasto/dev/bitcoin/build_first/src
17+-I/home/hebasto/dev/bitcoin/build_second/src
18 ### arg
19 -I/home/hebasto/dev/bitcoin/src
20 ### arg
21@@ -170,7 +170,7 @@
22 /home/hebasto/dev/bitcoin/src/crypto/aes.cpp
23 "
24 # 1 "
25-/home/hebasto/dev/bitcoin/build_first
26+/home/hebasto/dev/bitcoin/build_second
27 "
28 # 0 "
29 <built-in>

    Hence, we need to address:

    1. Absolute paths in compiler invocation commands, and
    2. Hashing of the current working directory.

    The documented method to resolve the first issue is the base_dir ccache configuration option. The current recommendation from our Productivity Notes remains valid:

    base_dir is required for ccache to share cached compiles of the same file across different … paths; it will only do this for paths under base_dir.

    Additionally, ccache docs state:

    The CWD will not be included in the hash if base_dir is set (and matches the CWD) and the compiler option -fdebug-prefix-map is used.

    The latter is always true:https://github.com/bitcoin/bitcoin/blob/4637cb1eec48d1af8d23eeae1bb4c6f8de55eed9/CMakeLists.txt#L473-L475

    However, in CMake builds the CWD is a build directory. Therefore, none of the meaningful values of base_dir will trigger the skipping of CWD hashing. We must explicitly use the hash_dir option to disable this behaviour.

    Now, let’s set base_dir and disable hash_dir:

     0$ export CCACHE_BASEDIR=$HOME
 1$ export CCACHE_NOHASHDIR=1
 2$ export CCACHE_DIR=$(mktemp -d)
 3$ cmake -B build_first -DWITH_CCACHE=ON
 4$ cmake --build build_first -t bitcoin_crypto -j $(nproc)
 5$ cmake -B build_second -DWITH_CCACHE=ON
 6$ ccache -z
 7$ cmake --build build_second -t bitcoin_crypto -j $(nproc)
 8$ ccache -s
 9Cacheable calls:     20 /  20 (100.0%)
10  Hits:              20 /  20 (100.0%)
11    Direct:          20 /  20 (100.0%)
12    Preprocessed:     0 /  20 ( 0.00%)
13  Misses:             0 /  20 ( 0.00%)
14Local storage:
15  Cache size (GiB): 0.0 / 5.0 ( 0.04%)
16  Hits:              20 /  20 (100.0%)
17  Misses:             0 /  20 ( 0.00%)

    We’ve got 100% hits.

    Ccache docs warn about disabling CWD hashing:

    The reason for including the CWD in the hash by default is to prevent a problem with the storage of the current working directory in the debug info of an object file, which can lead ccache to return a cached object file that has the working directory in the debug info set incorrectly.

    However, I do not see how this affects us, as all paths in the debug information of object files are mapped as mentioned above.

  4. hebasto commented at 1:09 pm on March 9, 2025: member
    Amending our docs, as demonstrated in https://github.com/davidgumberg/bitcoin/commit/1c6ae1043d59d01cbf52f353e50a63b0afa883c4, could be an alternative to #30861.
  5. ryanofsky commented at 2:39 pm on March 11, 2025: contributor

    Thanks for writing this up so clearly. It would be good to add this to the documentation.

    re: #31994 (comment)

    Additionally, ccache docs state:

    The CWD will not be included in the hash if base_dir is set (and matches the CWD) and the compiler option -fdebug-prefix-map is used.

    I still don’t understand why setting hash_dir = false or CCACHE_NOHASHDIR=1 should be necessary. Both conditions listed would seem to be true. Both CWD’s (/home/hebasto/dev/bitcoin/build_first and /home/hebasto/dev/bitcoin/build_second) should match the base_dir (/home/hebasto) because the according to the base_dir documentation it matches “absolute paths that begin with base_dir”.

    However, in CMake builds the CWD is a build directory. Therefore, none of the meaningful values of base_dir will trigger the skipping of CWD hashing. We must explicitly use the hash_dir option to disable this behaviour.

    This doesn’t seem true. How do /home/hebasto/dev/bitcoin/build_first and /home/hebasto/dev/bitcoin/build_second not match /home/hebasto?

  6. hebasto commented at 2:50 pm on March 11, 2025: member

    Thanks for writing this up so clearly. It would be good to add this to the documentation.

    re: #31994 (comment)

    Additionally, ccache docs state:

    The CWD will not be included in the hash if base_dir is set (and matches the CWD) and the compiler option -fdebug-prefix-map is used.

    I still don’t understand why setting hash_dir = false or CCACHE_NOHASHDIR=1 should be necessary. Both conditions listed would seem to be true. Both CWD’s (/home/hebasto/dev/bitcoin/build_first and /home/hebasto/dev/bitcoin/build_second) should match the base_dir (/home/hebasto) because the according to the base_dir documentation it matches “absolute paths that begin with base_dir”.

    This interpretation of the Ccache documentation does not describe the actual behaviour on my different systems. I read “base_dir matches the CWD” as “base_dir equals the CWD”. However, I might be wrong.

  7. ryanofsky commented at 3:21 pm on March 11, 2025: contributor

    This interpretation of the Ccache documentation does not describe the actual behaviour on my different systems. I read “base_dir matches the CWD” as “base_dir equals the CWD”. However, I might be wrong.

    You are probably right but this behavior does not seem to make sense or correspond to documentation of base_dir. It is probably ok for us to force CCACHE_NOHASHDIR, but it seems like it would be safer if ccache just detected conditions it should and shouldn’t hash CWD correctly itself. Ccache’s behavior here seems nonsensical.

  8. hebasto commented at 4:49 pm on March 11, 2025: member

    This interpretation of the Ccache documentation does not describe the actual behaviour on my different systems. I read “base_dir matches the CWD” as “base_dir equals the CWD”. However, I might be wrong.

    You are probably right but this behavior does not seem to make sense or correspond to documentation of base_dir. It is probably ok for us to force CCACHE_NOHASHDIR, but it seems like it would be safer if ccache just detected conditions it should and shouldn’t hash CWD correctly itself. Ccache’s behavior here seems nonsensical.

    It looks like this code is responsible for that behaviour:

    0      if (inc_path != ctx.apparent_cwd || ctx.config.hash_dir()) {
1        hash.hash(inc_path);
2      }
  9. willcl-ark added the label Build system on Mar 19, 2025


fanquake hebasto ryanofsky

Labels
Build system

Milestone
29.0

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-03-31 09:12 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me