build: Fix ccache behavior when cross-compiling for darwin hosts #24620

hebasto commented at 12:08 PM on March 20, 2022: member

For darwin hosts we use the env command to run a compiler in a modified environment which in turn makes ccache effectively no-op. On master (ae005a647ffa1d457c5c7a0528cb29f3b1937b96):

$ cat /etc/os-release | grep VERSION=
VERSION="22.04 (Jammy Jellyfish)"
$ ccache --version | head -1
ccache version 4.5.1
$ make -C depends HOST=x86_64-apple-darwin
$ ./autogen.sh
$ CONFIG_SITE=$PWD/depends/x86_64-apple-darwin/share/config.site ./configure 
$ make clean
$ ccache --zero-stats
$ make
$ ccache --show-stats -v
Summary:
  Cache directory:       /home/hebasto/.cache/ccache
  Primary config:        /home/hebasto/.config/ccache/ccache.conf
  Secondary config:      /etc/ccache.conf
  Stats updated:         Sun Mar 20 12:33:01 2022
  Hits:                      0 /    0
    Direct:                  0 /    0
    Preprocessed:            0 /    0
  Misses:                    0
    Direct:                  0
    Preprocessed:            0
  Uncacheable:             734
Primary storage:
  Hits:                      0 /    0
  Misses:                    0
  Cache size (GB):        1.69 / 5.00 (33.87 %)
  Files:                 11790
Uncacheable:
  Called for linking:       14
  Multiple source files:   720

With this PR:

$ make -C depends HOST=x86_64-apple-darwin
$ ./autogen.sh
$ CONFIG_SITE=$PWD/depends/x86_64-apple-darwin/share/config.site ./configure 
$ make clean
$ make
$ make clean
$ ccache --zero-stats
$ make
$ ccache --show-stats 
Summary:
  Hits:             720 /  720 (100.0 %)
    Direct:         720 /  720 (100.0 %)
    Preprocessed:     0 /    0
  Misses:             0
    Direct:           0
    Preprocessed:     0
  Uncacheable:       14
Primary storage:
  Hits:            1440 / 1440 (100.0 %)
  Misses:             0
  Cache size (GB): 1.74 / 5.00 (34.80 %)

Use the -v/--verbose option for more details.

Fixes bitcoin/bitcoin#21552.

Also this PR improves efficiency of the "macOS 10.15" CI task:

master (e09cf64c48286176e7d080edfe901f47baa0a418), https://cirrus-ci.com/task/6266307669655552:
this PR, https://cirrus-ci.com/task/6033477492539392:

hebasto marked this as a draft on Mar 20, 2022

DrahtBot added the label Build system on Mar 20, 2022

hebasto marked this as ready for review on Mar 20, 2022

prusnak commented at 1:25 PM on March 20, 2022: contributor

Concept ACK

jonatack commented at 12:26 PM on March 21, 2022: contributor

Concept ACK

fanquake commented at 10:13 AM on March 22, 2022: member

~0. I'm not convinced this change is a good way to fix this problem. Wont this also mean that the compiler invocation we use to build darwin packages in depends is now different from the one we use to build bitcoind?

fanquake commented at 10:56 AM on March 22, 2022: member

This change also does not Guix build:

HOSTS="x86_64-apple-darwin" ./contrib/guix/guix-build
...
Building libevent...
make[1]: Entering directory '/bitcoin/depends/work/build/x86_64-apple-darwin/libevent/2.1.12-stable-a581391610c'
  GEN      include/event2/event-config.h
make  all-am
make[2]: Entering directory '/bitcoin/depends/work/build/x86_64-apple-darwin/libevent/2.1.12-stable-a581391610c'
  CC       buffer.lo
  CC       bufferevent.lo
  CC       bufferevent_filter.lo
  CC       bufferevent_pair.lo
  CC       bufferevent_ratelim.lo
  CC       bufferevent_sock.lo
  CC       evmap.lo
  CC       event.lo
buffer.c:2487:60: error: too many arguments to function call, expected 4, have 6
        res = sendfile(source_fd, dest_fd, chain->misalign, &len, NULL, 0);
              ~~~~~~~~                                            ^~~~~~~
/gnu/store/v770rvqs8q21mbzwb3gkihr2glgn80am-clang-10.0.1/lib/clang/10.0.1/include/stddef.h:89:16: note: expanded from macro 'NULL'
#  define NULL ((void*)0)
               ^
/gnu/store/bxh206gz379wkn8cvb2ghlkvpqgwfd2v-gcc-toolchain-10.3.0/include/sys/sendfile.h:33:1: note: 'sendfile' declared here
extern ssize_t sendfile (int __out_fd, int __in_fd, off_t *__offset,
^
1 error generated.
  CC       evthread.lo
make[2]: *** [Makefile:1866: buffer.lo] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory '/bitcoin/depends/work/build/x86_64-apple-darwin/libevent/2.1.12-stable-a581391610c'
make[1]: *** [Makefile:1385: all] Error 2
make[1]: Leaving directory '/bitcoin/depends/work/build/x86_64-apple-darwin/libevent/2.1.12-stable-a581391610c'
make: *** [funcs.mk:282: /bitcoin/depends/work/build/x86_64-apple-darwin/libevent/2.1.12-stable-a581391610c/./.stamp_built] Error 2
make: Leaving directory '/bitcoin/depends'

hebasto marked this as a draft on Mar 27, 2022

build: Fix ccache behavior when cross-compiling for darwin hosts 6f4ff381c2

hebasto force-pushed on Mar 28, 2022

hebasto marked this as ready for review on Mar 28, 2022

hebasto commented at 4:02 PM on March 28, 2022: member

Wont this also mean that the compiler invocation we use to build darwin packages in depends is now different from the one we use to build bitcoind?

Fixed.

Guix builds:

$ find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
f8c0f480ba70b4ba6aac4a93d76f0281043ba8a37ca3b9b116ec3aea6e7851ca  guix-build-6f4ff381c2f9/output/aarch64-linux-gnu/SHA256SUMS.part
84fc763a787f7116a00d0e1115705142caaea9868e51487f088519bb20ac84cd  guix-build-6f4ff381c2f9/output/aarch64-linux-gnu/bitcoin-6f4ff381c2f9-aarch64-linux-gnu-debug.tar.gz
d6f6b6565cc29203aa404861aea6a5bd8e2998d0ae9bb16f65e58c299d06f3ff  guix-build-6f4ff381c2f9/output/aarch64-linux-gnu/bitcoin-6f4ff381c2f9-aarch64-linux-gnu.tar.gz
1103780ab5798c7319b7ae04e47958d3280ab43970e2a0f561415b3943e302d3  guix-build-6f4ff381c2f9/output/arm-linux-gnueabihf/SHA256SUMS.part
793adbaf01daa878e54aee0fec2057121295a66bd17567175a69b7c3eefdcd7c  guix-build-6f4ff381c2f9/output/arm-linux-gnueabihf/bitcoin-6f4ff381c2f9-arm-linux-gnueabihf-debug.tar.gz
86f3e3fd8ee7e6b27def40cf1252cec271fbb558d1e9518aeb60916b0f3f4202  guix-build-6f4ff381c2f9/output/arm-linux-gnueabihf/bitcoin-6f4ff381c2f9-arm-linux-gnueabihf.tar.gz
e9ba180d604c5463dce085a60dc92e5c49cb83b194d7f727548a9e4bef3819e2  guix-build-6f4ff381c2f9/output/arm64-apple-darwin/SHA256SUMS.part
774d7bc88abeaf3455bd44c664666965022c49ad0c6fde197f6409eb1889dcca  guix-build-6f4ff381c2f9/output/arm64-apple-darwin/bitcoin-6f4ff381c2f9-arm64-apple-darwin-unsigned.dmg
a3c7bfc99fee0c8483e44dd02faf3aaeb14f4b886858d4b3be166ad3750d9b70  guix-build-6f4ff381c2f9/output/arm64-apple-darwin/bitcoin-6f4ff381c2f9-arm64-apple-darwin-unsigned.tar.gz
53c2d83e258302740f47c9f83a00bfe995c785dad5b10679162a2da6fe0206c5  guix-build-6f4ff381c2f9/output/arm64-apple-darwin/bitcoin-6f4ff381c2f9-arm64-apple-darwin.tar.gz
05c4b27315846e994f94a359862c9b5ac6c4113e1404cb6020142b40d2d25d32  guix-build-6f4ff381c2f9/output/dist-archive/bitcoin-6f4ff381c2f9.tar.gz
b6414b295482c705d113c6707e6ceca1c1664d20f271a2b8d391ad3c0d783ee5  guix-build-6f4ff381c2f9/output/powerpc64-linux-gnu/SHA256SUMS.part
37300b4c9d2dc9ae751fed8bdaa482607db2225a9e21e8301a5b3be6194fb57d  guix-build-6f4ff381c2f9/output/powerpc64-linux-gnu/bitcoin-6f4ff381c2f9-powerpc64-linux-gnu-debug.tar.gz
c334c66631138bb2ebae8f862131de013c2a30ba81dafab5a1abde68a98321c0  guix-build-6f4ff381c2f9/output/powerpc64-linux-gnu/bitcoin-6f4ff381c2f9-powerpc64-linux-gnu.tar.gz
3a68731fafa61d64a40a4744a739620b9a99d3a793847bb6f5ab4e8871d7f6ac  guix-build-6f4ff381c2f9/output/powerpc64le-linux-gnu/SHA256SUMS.part
6476a34cb4ab3cd99cedab8b377363592d3d116f3f8ae518a7b9768dc245131b  guix-build-6f4ff381c2f9/output/powerpc64le-linux-gnu/bitcoin-6f4ff381c2f9-powerpc64le-linux-gnu-debug.tar.gz
0ad92d09521f558a4d804e6d4f736f5201e2f8f429e6f291b0a9fa56601dfe27  guix-build-6f4ff381c2f9/output/powerpc64le-linux-gnu/bitcoin-6f4ff381c2f9-powerpc64le-linux-gnu.tar.gz
d374f271f8eee3c3c03522ac1397ab9c13976a93efd24fbb1f9dcf4d4a9e9939  guix-build-6f4ff381c2f9/output/riscv64-linux-gnu/SHA256SUMS.part
92141d5c096fd057a9414709002025390003ab3573abb7c4b137743ae9fa1f60  guix-build-6f4ff381c2f9/output/riscv64-linux-gnu/bitcoin-6f4ff381c2f9-riscv64-linux-gnu-debug.tar.gz
b49f0423aa330e8eec01b2d8ed36dc7a444e91c38b67f072931f9ad1f6de6160  guix-build-6f4ff381c2f9/output/riscv64-linux-gnu/bitcoin-6f4ff381c2f9-riscv64-linux-gnu.tar.gz
990cafb070b25951ed0d2ad9edffb6b92022ed918fb9903a6fe65b9ab7364e82  guix-build-6f4ff381c2f9/output/x86_64-apple-darwin/SHA256SUMS.part
ddc3d209f9e27f6ece333618d433c4a670a252e069fc6b17259348b72eedc494  guix-build-6f4ff381c2f9/output/x86_64-apple-darwin/bitcoin-6f4ff381c2f9-x86_64-apple-darwin-unsigned.dmg
d736d7246360523a70d982916700122f6766251f045549b34272937f35b8e355  guix-build-6f4ff381c2f9/output/x86_64-apple-darwin/bitcoin-6f4ff381c2f9-x86_64-apple-darwin-unsigned.tar.gz
df786b1d05b74edf342513685d0b0d148d0f3bc51ee0bd3835cecffe0a9ce671  guix-build-6f4ff381c2f9/output/x86_64-apple-darwin/bitcoin-6f4ff381c2f9-x86_64-apple-darwin.tar.gz
4a3967f7c4381cb299833bf63e4d869c1b033d63997df6064c18e9da79e44ba8  guix-build-6f4ff381c2f9/output/x86_64-linux-gnu/SHA256SUMS.part
f272fcacbd564016376bc932df4cc5264ce2ddd47335fd22e07611675110ff14  guix-build-6f4ff381c2f9/output/x86_64-linux-gnu/bitcoin-6f4ff381c2f9-x86_64-linux-gnu-debug.tar.gz
c7995ffd6072b65b0c69ab5bc344f4a1324e0d2307e4bf4551fb6d72913157c6  guix-build-6f4ff381c2f9/output/x86_64-linux-gnu/bitcoin-6f4ff381c2f9-x86_64-linux-gnu.tar.gz
0059851989d59d2f068d5c93710b5228866314d87ea7a9f8f4f198b4916abb08  guix-build-6f4ff381c2f9/output/x86_64-w64-mingw32/SHA256SUMS.part
a33f19278ce0476dc6db48bdd0ccfff292d20b7ed32e103758834049f4de0633  guix-build-6f4ff381c2f9/output/x86_64-w64-mingw32/bitcoin-6f4ff381c2f9-win64-debug.zip
e9c00c22e65f0d9a2962d1540ff4e877bb02e72ef65131c19992782a29c01da1  guix-build-6f4ff381c2f9/output/x86_64-w64-mingw32/bitcoin-6f4ff381c2f9-win64-setup-unsigned.exe
79601aec47d44960cd116fbcbb04c1eaaaa9dcb431f14bc23be8c5afb7e3d633  guix-build-6f4ff381c2f9/output/x86_64-w64-mingw32/bitcoin-6f4ff381c2f9-win64-unsigned.tar.gz
2fbc90d9921f1a9525f44bee9716955d1092b52d703287c8786e3b5c566be6d8  guix-build-6f4ff381c2f9/output/x86_64-w64-mingw32/bitcoin-6f4ff381c2f9-win64.zip

hebasto commented at 4:04 PM on March 28, 2022: member

Friendly ping @dongcarl.

DrahtBot commented at 5:36 AM on March 31, 2022: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

No conflicts as of last run.

jarolrod commented at 12:13 AM on April 7, 2022: member

guix hashes (x86), mine match @hebasto

find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum

f8c0f480ba70b4ba6aac4a93d76f0281043ba8a37ca3b9b116ec3aea6e7851ca  guix-build-6f4ff381c2f9/output/aarch64-linux-gnu/SHA256SUMS.part
84fc763a787f7116a00d0e1115705142caaea9868e51487f088519bb20ac84cd  guix-build-6f4ff381c2f9/output/aarch64-linux-gnu/bitcoin-6f4ff381c2f9-aarch64-linux-gnu-debug.tar.gz
d6f6b6565cc29203aa404861aea6a5bd8e2998d0ae9bb16f65e58c299d06f3ff  guix-build-6f4ff381c2f9/output/aarch64-linux-gnu/bitcoin-6f4ff381c2f9-aarch64-linux-gnu.tar.gz
1103780ab5798c7319b7ae04e47958d3280ab43970e2a0f561415b3943e302d3  guix-build-6f4ff381c2f9/output/arm-linux-gnueabihf/SHA256SUMS.part
793adbaf01daa878e54aee0fec2057121295a66bd17567175a69b7c3eefdcd7c  guix-build-6f4ff381c2f9/output/arm-linux-gnueabihf/bitcoin-6f4ff381c2f9-arm-linux-gnueabihf-debug.tar.gz
86f3e3fd8ee7e6b27def40cf1252cec271fbb558d1e9518aeb60916b0f3f4202  guix-build-6f4ff381c2f9/output/arm-linux-gnueabihf/bitcoin-6f4ff381c2f9-arm-linux-gnueabihf.tar.gz
e9ba180d604c5463dce085a60dc92e5c49cb83b194d7f727548a9e4bef3819e2  guix-build-6f4ff381c2f9/output/arm64-apple-darwin/SHA256SUMS.part
774d7bc88abeaf3455bd44c664666965022c49ad0c6fde197f6409eb1889dcca  guix-build-6f4ff381c2f9/output/arm64-apple-darwin/bitcoin-6f4ff381c2f9-arm64-apple-darwin-unsigned.dmg
a3c7bfc99fee0c8483e44dd02faf3aaeb14f4b886858d4b3be166ad3750d9b70  guix-build-6f4ff381c2f9/output/arm64-apple-darwin/bitcoin-6f4ff381c2f9-arm64-apple-darwin-unsigned.tar.gz
53c2d83e258302740f47c9f83a00bfe995c785dad5b10679162a2da6fe0206c5  guix-build-6f4ff381c2f9/output/arm64-apple-darwin/bitcoin-6f4ff381c2f9-arm64-apple-darwin.tar.gz
05c4b27315846e994f94a359862c9b5ac6c4113e1404cb6020142b40d2d25d32  guix-build-6f4ff381c2f9/output/dist-archive/bitcoin-6f4ff381c2f9.tar.gz
b6414b295482c705d113c6707e6ceca1c1664d20f271a2b8d391ad3c0d783ee5  guix-build-6f4ff381c2f9/output/powerpc64-linux-gnu/SHA256SUMS.part
37300b4c9d2dc9ae751fed8bdaa482607db2225a9e21e8301a5b3be6194fb57d  guix-build-6f4ff381c2f9/output/powerpc64-linux-gnu/bitcoin-6f4ff381c2f9-powerpc64-linux-gnu-debug.tar.gz
c334c66631138bb2ebae8f862131de013c2a30ba81dafab5a1abde68a98321c0  guix-build-6f4ff381c2f9/output/powerpc64-linux-gnu/bitcoin-6f4ff381c2f9-powerpc64-linux-gnu.tar.gz
3a68731fafa61d64a40a4744a739620b9a99d3a793847bb6f5ab4e8871d7f6ac  guix-build-6f4ff381c2f9/output/powerpc64le-linux-gnu/SHA256SUMS.part
6476a34cb4ab3cd99cedab8b377363592d3d116f3f8ae518a7b9768dc245131b  guix-build-6f4ff381c2f9/output/powerpc64le-linux-gnu/bitcoin-6f4ff381c2f9-powerpc64le-linux-gnu-debug.tar.gz
0ad92d09521f558a4d804e6d4f736f5201e2f8f429e6f291b0a9fa56601dfe27  guix-build-6f4ff381c2f9/output/powerpc64le-linux-gnu/bitcoin-6f4ff381c2f9-powerpc64le-linux-gnu.tar.gz
d374f271f8eee3c3c03522ac1397ab9c13976a93efd24fbb1f9dcf4d4a9e9939  guix-build-6f4ff381c2f9/output/riscv64-linux-gnu/SHA256SUMS.part
92141d5c096fd057a9414709002025390003ab3573abb7c4b137743ae9fa1f60  guix-build-6f4ff381c2f9/output/riscv64-linux-gnu/bitcoin-6f4ff381c2f9-riscv64-linux-gnu-debug.tar.gz
b49f0423aa330e8eec01b2d8ed36dc7a444e91c38b67f072931f9ad1f6de6160  guix-build-6f4ff381c2f9/output/riscv64-linux-gnu/bitcoin-6f4ff381c2f9-riscv64-linux-gnu.tar.gz
990cafb070b25951ed0d2ad9edffb6b92022ed918fb9903a6fe65b9ab7364e82  guix-build-6f4ff381c2f9/output/x86_64-apple-darwin/SHA256SUMS.part
ddc3d209f9e27f6ece333618d433c4a670a252e069fc6b17259348b72eedc494  guix-build-6f4ff381c2f9/output/x86_64-apple-darwin/bitcoin-6f4ff381c2f9-x86_64-apple-darwin-unsigned.dmg
d736d7246360523a70d982916700122f6766251f045549b34272937f35b8e355  guix-build-6f4ff381c2f9/output/x86_64-apple-darwin/bitcoin-6f4ff381c2f9-x86_64-apple-darwin-unsigned.tar.gz
df786b1d05b74edf342513685d0b0d148d0f3bc51ee0bd3835cecffe0a9ce671  guix-build-6f4ff381c2f9/output/x86_64-apple-darwin/bitcoin-6f4ff381c2f9-x86_64-apple-darwin.tar.gz
4a3967f7c4381cb299833bf63e4d869c1b033d63997df6064c18e9da79e44ba8  guix-build-6f4ff381c2f9/output/x86_64-linux-gnu/SHA256SUMS.part
f272fcacbd564016376bc932df4cc5264ce2ddd47335fd22e07611675110ff14  guix-build-6f4ff381c2f9/output/x86_64-linux-gnu/bitcoin-6f4ff381c2f9-x86_64-linux-gnu-debug.tar.gz
c7995ffd6072b65b0c69ab5bc344f4a1324e0d2307e4bf4551fb6d72913157c6  guix-build-6f4ff381c2f9/output/x86_64-linux-gnu/bitcoin-6f4ff381c2f9-x86_64-linux-gnu.tar.gz
0059851989d59d2f068d5c93710b5228866314d87ea7a9f8f4f198b4916abb08  guix-build-6f4ff381c2f9/output/x86_64-w64-mingw32/SHA256SUMS.part
a33f19278ce0476dc6db48bdd0ccfff292d20b7ed32e103758834049f4de0633  guix-build-6f4ff381c2f9/output/x86_64-w64-mingw32/bitcoin-6f4ff381c2f9-win64-debug.zip
e9c00c22e65f0d9a2962d1540ff4e877bb02e72ef65131c19992782a29c01da1  guix-build-6f4ff381c2f9/output/x86_64-w64-mingw32/bitcoin-6f4ff381c2f9-win64-setup-unsigned.exe
79601aec47d44960cd116fbcbb04c1eaaaa9dcb431f14bc23be8c5afb7e3d633  guix-build-6f4ff381c2f9/output/x86_64-w64-mingw32/bitcoin-6f4ff381c2f9-win64-unsigned.tar.gz
2fbc90d9921f1a9525f44bee9716955d1092b52d703287c8786e3b5c566be6d8  guix-build-6f4ff381c2f9/output/x86_64-w64-mingw32/bitcoin-6f4ff381c2f9-win64.zip

fanquake commented at 10:42 AM on April 17, 2022: member

~0. I'm still not convinced this is a good change, or even something that needs to be solved at the cost of introducing more platform specific hacks / complication into our build system. It's surprising that there is no way for us to fix this for our use case given the multitude of ccache configuration options / settings, and it's support for additional compiler wrappers. Have we opened an issue upstream to document our use case, and why ccache doesn't currently work with it?

hebasto closed this on Apr 17, 2022

bitcoin locked this on Apr 17, 2023

hebasto commented at 8:51 AM on August 16, 2023: member

~0. I'm still not convinced this is a good change, or even something that needs to be solved at the cost of introducing more platform specific hacks / complication into our build system. It's surprising that there is no way for us to fix this for our use case given the multitude of ccache configuration options / settings, and it's support for additional compiler wrappers. Have we opened an issue upstream to document our use case, and why ccache doesn't currently work with it?

Considering some recent comments:

should I re-open this PR?

fanquake commented at 8:53 AM on August 16, 2023: member

It's surprising that there is no way for us to fix this for our use case given the multitude of ccache configuration options / settings, and it's support for additional compiler wrappers.

Have we opened an issue upstream to document our use case, and why ccache doesn't currently work with it?

hebasto commented at 1:27 PM on August 26, 2023: member

It's surprising that there is no way for us to fix this for our use case given the multitude of ccache configuration options / settings, and it's support for additional compiler wrappers.

From https://github.com/ccache/ccache/issues/1326#issuecomment-1693774827:

In summary: Use the form env -u VAR ccache clang ... instead.

That is exactly what this PR does.