ci: Improve `ccache` cache hit rates #27083

pull hebasto wants to merge 2 commits into bitcoin:master from hebasto:230211-ci-ccache changing 5 files +6 −4

hebasto commented at 8:46 AM on February 11, 2023: member
Changes that are not affecting binaries, e.g., #27070 or #27072, are expected to show up to 100% ccache hit rate. But that is not the case:
- https://cirrus-ci.com/build/4720733276864512
- https://cirrus-ci.com/build/5915469295648768
Zero hit rate in the "macOS 10.15" is well known. But a fix was considered not "a good change" and closed.

For another task with nearly-zero hit rate--"ARM64 Android APK"--a fix suggested in #27077.

Here are cache hit rates for other problematic tasks:
- "previous releases": 33.47 % and 40.68 %
- "TSan": 77.94 % and 81.07 %
- "MSan": 47.24 % and 49.29 %
- "ASan + LSan + UBSan + integer": 54.96 % and 59.79 %
- "multiprocess": 12.05 % and 15.16 %
- "macOS 13 native": 65.87 % and 67.86 %
In my experiments, making a ccache_cache folder task-specific did not help.

Apparently, the problem is related to https://github.com/cirruslabs/cirrus-ci-docs/pull/896.

This PR fixes the issue. Also CCACHE_SIZE values have been adjusted for some tasks to avoid cache misses due to "cleanups performed".

ci: Move `ccache_cache` folder into `$CIRRUS_WORKING_DIR` one

In Cirrus CI the `/tmp` is a special volume to be mounted into instances
running in a Kubernetes cluster, which makes usage of `ccache` cache
much less effective.

301ab348b4

ci: Adjust `CCACHE_SIZE` for some task

This change improves cache hit rates for the touched CI tasks.

8efa268401

DrahtBot commented at 8:46 AM on February 11, 2023: member



The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.



Reviews

See the guideline for information on the review process. A summary of reviews will appear here.
DrahtBot added the label Tests on Feb 11, 2023
hebasto commented at 8:48 AM on February 11, 2023: member

Drafted for now as I'm going to demonstrate benefits of this PR by re-running CI twice. First time with cleaned caches, and second one to ensure higher hit rates.
hebasto closed this on Feb 11, 2023
fanquake locked this on Feb 19, 2023

Contributors

hebasto

DrahtBot

Labels

Tests

Linked (view graph)

#21552 ci: ccache does not work for macOS cross-compiling builds #24620 build: Fix ccache behavior when cross-compiling for darwin hosts #27070 ci: Fix `fingerprint_script` for `depends` subdir caches #27072 doc: Remove unused REVIEWERS file #27077 ci: Make `ccache` hashing content of compiler binary #27084 ci: A few fixes of `ccache` issues

ci: Improve `ccache` cache hit rates #27083

Reviews