Storage is cheap, so there was little value in extracting build layers in the CI images.
However, the GHA cache is limited to 10GB, so extracting a shared base layer could help to reduce the overall footprint. Possibly, it could even speed up image building, because installation is only done once.
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33620.
See the guideline for information on the review process. A summary of reviews will appear here.
Reviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.
Also, it won’t work anyway, due to --cache-to: https://github.com/maflcko/bitcoin-core-with-ci/actions/runs/18490105374/job/52681542962#step:6:149:
0+ docker buildx ls --format '{{.DriverEndpoint}} {{.Name}}'
1Using existing docker based buildx: default
2Building ci_native_base image layer
3+ docker buildx build --file=/home/runner/work/bitcoin-core-with-ci/bitcoin-core-with-ci/ci/test_imagefile_base --platform=linux --label=bitcoin-ci-test --tag=ci_native_base --cache-from type=gha,scope=ci_win64 --cache-to type=gha,mode=max,ignore-error=true,scope=ci_win64 --load /home/runner/work/bitcoin-core-with-ci/bitcoin-core-with-ci
4ERROR: failed to build: Cache export is not supported for the docker driver.
5Switch to a different driver, or turn on the containerd image store, and try again.
6Learn more at https://docs.docker.com/go/build-cache-backends/
7Command '['docker', 'buildx', 'build', '--file=/home/runner/work/bitcoin-core-with-ci/bitcoin-core-with-ci/ci/test_imagefile_base', '--platform=linux', '--label=bitcoin-ci-test', '--tag=ci_native_base', '--cache-from', 'type=gha,scope=ci_win64', '--cache-to', 'type=gha,mode=max,ignore-error=true,scope=ci_win64', '--load', '/home/runner/work/bitcoin-core-with-ci/bitcoin-core-with-ci']' returned non-zero exit status 1.
8Error: Process completed with exit code 1.
I think if we want something like this to work it could be tricky without using a registry to host the base image which all other jobs pulled from (on a cache hit).
It may be possible without one, but as each job is scoped to itself, which is what keeps the caches seperate,: https://github.com/bitcoin/bitcoin/actions/runs/18489967967/job/52681101678?pr=33620#step:9:151 … I think current approach would end up with one base_image per job, which is not quite what we are after here.
One possible approach I have in mind which could work:
scope) to build the base_image/imports it into the docker storecc @m3dwards in case you have any other ideas for approach here?
have a new (second) docker builder (with a fixed/shared
scope) to build the base_image/imports it into the docker storeuse a second build (with current ci config) which builds based on this base_image (which is now in the local docker store)
yeah, I think this makes sense. I just can’t even figure out how to both support the gha cache backend and also the local storage. edit: Upstream bug: https://github.com/moby/buildkit/issues/2343
We want to support:
Maybe all of this is too complicated and not worth it?
We want to support:
* GHA "read-only" on branch pushes or pull requests, using the cache if available, and building all layers locally (optionally falling back to the local storage for the base layer) * GHA "write-push" on main pushes, using the cache if available, and pushing to it (the base layer and the final image) * Cirrus (both of the above) * Fully local docker buildx (ideally, but optionally falling back to the local store for the base layer) * Fully local podman build (I think this should work trivially)
So the approach we have used currently is to have CI-only config here: https://github.com/bitcoin/bitcoin/blob/6c4fe401e908cff1b67d80035b117aae15fe7db6/.github/actions/configure-docker/action.yml#L35-L59
which appends flags to the docker build only in CI. We always add --cache-from with the correct scope of the job, then --cache-to if we are on master branch, along with --load to always load build images into the local store.
If we are on Cirrus we also update the required URLs.
Maybe all of this is too complicated and not worth it?
I agree this could be tricky to support all configurations you list though.
This has a few benefits:
* The raw python3 -c "..." command to calculate --cpuset-cpus lives in a
Python context.
* The shellcheck SC2086 warning is disabled for the whole command, but
is only needed for the DOCKER_BUILD_CACHE_ARG env var. So in Python,
only pass this one env var to shlex.split() for proper word splitting.
The comments are moved, which can be checked via the git options:
--color-moved=dimmed-zebra --color-moved-ws=ignore-all-space
Hmm, then we should probably revert 6c4fe401e908cff1b67d80035b117aae15fe7db6 (or disable docker caching for GHA explicitly), because with the ~20 tasks here, and each task having a ccache size of 500MB, the GHA limit of 10GB should already be reached.
With the 10GB limit, it seems better to use if for ccache+depends than to cache installed packages?
With the 10GB limit, it seems better to use if for ccache+depends than to cache installed packages?
Done that in #33639
For reference, we are probably better off using the limited cache for ccache, as there are several tasks, which run out of the 500MB ccache: (musl, and the previous_releases one)
0# du -sh /var/lib/containers/storage/volumes/ci_native_*_ccache/
1886M /var/lib/containers/storage/volumes/ci_native_alpine_musl_ccache/
2293M /var/lib/containers/storage/volumes/ci_native_asan_ccache/
3314M /var/lib/containers/storage/volumes/ci_native_msan_ccache/
4201M /var/lib/containers/storage/volumes/ci_native_nowallet_libbitcoinkernel_ccache/
5603M /var/lib/containers/storage/volumes/ci_native_previous_releases_ccache/
Also musl build on the master branch here: https://github.com/bitcoin/bitcoin/actions/runs/18531491383/job/52821862345#step:9:2286 shows hits of just 8%:
0
1bash -c 'ccache --version | head -n 1 && ccache --show-stats'
2ccache version 4.11.3
3Cacheable calls: 794 / 794 (100.0%)
4 Hits: 68 / 794 ( 8.56%)
5 Direct: 9 / 68 (13.24%)
6 Preprocessed: 59 / 68 (86.76%)
7 Misses: 726 / 794 (91.44%)
8Local storage:
9 Cache size (GB): 0.7 / 0.5 (134.4%)
10 Cleanups: 683
11 Hits: 68 / 794 ( 8.56%)
12 Misses: 726 / 794 (91.44%)
Labels
Tests