ci: optionally use local docker build cache #31545

pull 0xB10C wants to merge 1 commits into bitcoin:master from 0xB10C:2024-11-ci-optional-local-docker-build-cache changing 1 files +33 −0
  1. 0xB10C commented at 7:59 AM on December 20, 2024: contributor

    By setting DANGER_DOCKER_BUILD_CACHE_HOST_DIR, the task-specific docker images built during the CI run can be cached. This allows, for example, ephemeral CI runners to reuse the docker images (or layers of it) from earlier runs, by persisting the image cache before the ephemeral CI runner is shut down. The cache keyed by CONTAINER_NAME.

    As --cache-to doesn't remove old cache files, the existing cache is removed after a successful docker build and the newly cached image is moved to it's location to avoid the cache from growing indefinitely with old, unused layers.

    When --cache-from doesn't find the directory, the cached version is a cache-miss, or the cache can't be imported for whatever other reason, it warns and docker build continues by building the docker image.

    This feature is opt-in. The documentation for the docker build cache of type=local can be found on https://docs.docker.com/build/cache/backends/local/

    This replaces #31377 - some of the discussion there might provide more context.

  2. DrahtBot commented at 7:59 AM on December 20, 2024: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31545.

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK maflcko, TheCharlatan, willcl-ark, achow101

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

  3. DrahtBot added the label Tests on Dec 20, 2024
  4. 0xB10C commented at 8:41 AM on December 20, 2024: contributor

    Some additional context:

    I've been testing and using this with ephemeral cirrus runners I've been working on. See a run of the initial commit of this PR here: https://cirrus-ci.com/build/6017527349772288. If this PR is merged, these could be used on bitcoin/bitcoin after a while - once people have rebased their PRs to include this change. Old branches, for example, 28.x backports won't be able to use this (unless backported) but the docker images on old branches might generally be different and will be needed to rebuild.

    The cache for all of our self-hosted x86_64 Cirrus tasks (all but the ARM task) is about 5.2 GB. I don't expect this to grow or shrink significantly.

    $ du -h -d1 .
    477M	./ci_macos_cross
    837M	./ci_win64
    356M	./ci_native_nowallet_libbitcoinkernel
    205M	./ci_native_previous_releases
    1.4G	./ci_native_msan
    314M	./ci_i686_centos
    338M	./ci_native_tsan
    693M	./ci_native_tidy
    359M	./ci_i686_multiprocess
    351M	./ci_native_fuzz
    5.2G	.
    

    Loading, extracting and importing the cache artifacts on a cache-hit takes a few seconds. This is slower compared to the current non-ephemeral self-hosted runners if they already have a fitting image in their docker image store. Here's the log output for a cache-hit on my runner's macos-cross task.

    <details> <summary> `docker build` log section

    </summary>

    [07:49:56.064] + echo 'Creating docker.io/ubuntu:24.04 container to run in'
    [07:49:56.064] Creating docker.io/ubuntu:24.04 container to run in
    [07:49:56.064] + DOCKER_BUILD_CACHE_ARG=
    [07:49:56.064] + DOCKER_BUILD_CACHE_TEMPDIR=
    [07:49:56.064] + DOCKER_BUILD_CACHE_OLD_DIR=
    [07:49:56.064] + DOCKER_BUILD_CACHE_NEW_DIR=
    [07:49:56.064] + '[' /cache/docker/ci-imgs ']'
    [07:49:56.064] + DOCKER_BUILD_CACHE_OLD_DIR=/cache/docker/ci-imgs/ci_macos_cross
    [07:49:56.064] ++ mktemp --directory
    [07:49:56.074] + DOCKER_BUILD_CACHE_TEMPDIR=/var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh
    [07:49:56.074] + DOCKER_BUILD_CACHE_NEW_DIR=/var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh/ci_macos_cross
    [07:49:56.074] + DOCKER_BUILD_CACHE_ARG='--cache-from type=local,src=/cache/docker/ci-imgs/ci_macos_cross --cache-to type=local,dest=/var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh/ci_macos_cross,mode=max'
    [07:49:56.074] + DOCKER_BUILDKIT=1
    [07:49:56.074] + docker build --file /var/lib/cirrus-worker/tmp/cirrus-build/ci/test_imagefile --build-arg CI_IMAGE_NAME_TAG=docker.io/ubuntu:24.04 --build-arg FILE_ENV=./ci/test/00_setup_env_mac_cross.sh --label=bitcoin-ci-test --tag=ci_macos_cross --cache-from type=local,src=/cache/docker/ci-imgs/ci_macos_cross --cache-to type=local,dest=/var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh/ci_macos_cross,mode=max /var/lib/cirrus-worker/tmp/cirrus-build
    [07:49:56.738] [#0](/bitcoin-bitcoin/0/) building with "default" instance using docker driver
    [07:49:56.738] 
    [07:49:56.738] [#1](/bitcoin-bitcoin/1/) [internal] load build definition from test_imagefile
    [07:49:56.738] [#1](/bitcoin-bitcoin/1/) transferring dockerfile: 600B done
    [07:49:56.738] [#1](/bitcoin-bitcoin/1/) WARN: InvalidDefaultArgInFrom: Default value for ARG ${CI_IMAGE_NAME_TAG} results in empty or invalid base image name (line 8)
    [07:49:56.738] [#1](/bitcoin-bitcoin/1/) DONE 0.0s
    [07:49:56.738] 
    [07:49:56.738] [#2](/bitcoin-bitcoin/2/) [internal] load metadata for docker.io/library/ubuntu:24.04
    [07:49:56.738] [#2](/bitcoin-bitcoin/2/) DONE 0.0s
    [07:49:56.738] 
    [07:49:56.738] [#3](/bitcoin-bitcoin/3/) [internal] load .dockerignore
    [07:49:56.738] [#3](/bitcoin-bitcoin/3/) transferring context: 2B done
    [07:49:56.738] [#3](/bitcoin-bitcoin/3/) DONE 0.0s
    [07:49:56.738] 
    [07:49:56.738] [#4](/bitcoin-bitcoin/4/) [internal] load build context
    [07:49:56.738] [#4](/bitcoin-bitcoin/4/) DONE 0.0s
    [07:49:56.738] 
    [07:49:56.738] [#5](/bitcoin-bitcoin/5/) importing cache manifest from local:13157433447571386053
    [07:49:56.738] [#5](/bitcoin-bitcoin/5/) inferred cache manifest type: application/vnd.oci.image.index.v1+json done
    [07:49:56.738] [#5](/bitcoin-bitcoin/5/) DONE 0.0s
    [07:49:56.738] 
    [07:49:56.738] [#6](/bitcoin-bitcoin/6/) [1/4] FROM docker.io/library/ubuntu:24.04@sha256:80dd3c3b9c6cecb9f1667e9290b3bc61b78c2678c02cbdae5f0fea92cc6734ab
    [07:49:56.738] [#6](/bitcoin-bitcoin/6/) resolve docker.io/library/ubuntu:24.04@sha256:80dd3c3b9c6cecb9f1667e9290b3bc61b78c2678c02cbdae5f0fea92cc6734ab
    [07:49:57.299] [#6](/bitcoin-bitcoin/6/) resolve docker.io/library/ubuntu:24.04@sha256:80dd3c3b9c6cecb9f1667e9290b3bc61b78c2678c02cbdae5f0fea92cc6734ab 0.7s done
    [07:49:57.408] [#6](/bitcoin-bitcoin/6/) DONE 0.7s
    [07:49:57.408] 
    [07:49:57.408] [#4](/bitcoin-bitcoin/4/) [internal] load build context
    [07:49:57.408] [#4](/bitcoin-bitcoin/4/) transferring context: 12.88kB done
    [07:49:57.408] [#4](/bitcoin-bitcoin/4/) DONE 0.0s
    [07:49:57.408] 
    [07:49:57.408] [#7](/bitcoin-bitcoin/7/) [2/4] COPY ./ci/retry/retry /usr/bin/retry
    [07:49:57.408] [#7](/bitcoin-bitcoin/7/) CACHED
    [07:49:57.408] 
    [07:49:57.408] [#8](/bitcoin-bitcoin/8/) [3/4] COPY ./ci/test/00_setup_env.sh ././ci/test/00_setup_env_mac_cross.sh ./ci/test/01_base_install.sh /ci_container_base/ci/test/
    [07:49:57.408] [#8](/bitcoin-bitcoin/8/) CACHED
    [07:49:57.408] 
    [07:49:57.408] [#9](/bitcoin-bitcoin/9/) [4/4] RUN ["bash", "-c", "cd /ci_container_base/ && set -o errexit && source ./ci/test/00_setup_env.sh && ./ci/test/01_base_install.sh"]
    [07:49:57.408] [#9](/bitcoin-bitcoin/9/) sha256:febe602dc08a9a790bd092b613b049003ba7c75651d565cee664ecbebbe12bfc 1.85kB / 1.85kB 0.0s done
    [07:49:57.408] [#9](/bitcoin-bitcoin/9/) extracting sha256:febe602dc08a9a790bd092b613b049003ba7c75651d565cee664ecbebbe12bfc done
    [07:49:57.408] [#9](/bitcoin-bitcoin/9/) sha256:cfdd814cf5471383f503651cd4c6f1a1bcd432e92911dac06cff317cdbc6b305 3.38kB / 3.38kB 0.0s done
    [07:49:57.581] [#9](/bitcoin-bitcoin/9/) extracting sha256:cfdd814cf5471383f503651cd4c6f1a1bcd432e92911dac06cff317cdbc6b305 0.0s done
    [07:49:57.588] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 8.39MB / 470.35MB 0.2s
    [07:49:58.038] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 38.80MB / 470.35MB 0.6s
    [07:49:58.488] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 70.25MB / 470.35MB 1.1s
    [07:49:58.938] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 102.76MB / 470.35MB 1.5s
    [07:49:59.388] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 134.22MB / 470.35MB 2.0s
    [07:49:59.839] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 164.63MB / 470.35MB 2.4s
    [07:50:00.292] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 193.99MB / 470.35MB 2.9s
    [07:50:00.738] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 223.35MB / 470.35MB 3.3s
    [07:50:01.191] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 254.80MB / 470.35MB 3.8s
    [07:50:01.638] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 286.26MB / 470.35MB 4.2s
    [07:50:02.089] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 315.62MB / 470.35MB 4.7s
    [07:50:02.538] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 346.03MB / 470.35MB 5.1s
    [07:50:02.988] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 376.44MB / 470.35MB 5.6s
    [07:50:03.438] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 406.85MB / 470.35MB 6.0s
    [07:50:03.888] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 436.21MB / 470.35MB 6.5s
    [07:50:04.338] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 468.71MB / 470.35MB 6.9s
    [07:50:04.811] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 470.35MB / 470.35MB 7.2s done
    [07:50:04.811] [#9](/bitcoin-bitcoin/9/) extracting sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8
    [07:50:30.213] [#9](/bitcoin-bitcoin/9/) extracting sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 25.6s done
    [07:50:30.453] [#9](/bitcoin-bitcoin/9/) CACHED
    [07:50:30.453] 
    [07:50:30.453] [#10](/bitcoin-bitcoin/10/) exporting to image
    [07:50:30.453] [#10](/bitcoin-bitcoin/10/) exporting layers done
    [07:50:30.453] [#10](/bitcoin-bitcoin/10/) exporting manifest sha256:35b82073fb170cde1c1f340217048fa5fdca7bf710eb9836e1e75ae8f632c18a done
    [07:50:30.453] [#10](/bitcoin-bitcoin/10/) exporting config sha256:040ba6df0bbe7839075a62de0a03179747c82ddff2a33a9a11d72202cd6ed3c7 done
    [07:50:30.453] [#10](/bitcoin-bitcoin/10/) exporting attestation manifest sha256:b354c939c9e7630028a036a105ea52f56a74b91c54148db376b51477b6e47668 done
    [07:50:30.453] [#10](/bitcoin-bitcoin/10/) exporting manifest list sha256:550407c300a249757f8f145ac481f361b7e7f22de8b62639006b7f2d202369d5 done
    [07:50:30.453] [#10](/bitcoin-bitcoin/10/) naming to docker.io/library/ci_macos_cross:latest done
    [07:50:30.453] [#10](/bitcoin-bitcoin/10/) unpacking to docker.io/library/ci_macos_cross:latest done
    [07:50:30.453] [#10](/bitcoin-bitcoin/10/) DONE 0.1s
    [07:50:30.453] 
    [07:50:30.453] [#11](/bitcoin-bitcoin/11/) exporting cache to client directory
    [07:50:30.453] [#11](/bitcoin-bitcoin/11/) preparing build cache for export
    [07:50:30.453] [#11](/bitcoin-bitcoin/11/) writing layer sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8
    [07:50:37.448] [#11](/bitcoin-bitcoin/11/) writing layer sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 7.1s done
    [07:50:37.448] [#11](/bitcoin-bitcoin/11/) writing layer sha256:cfdd814cf5471383f503651cd4c6f1a1bcd432e92911dac06cff317cdbc6b305
    [07:50:37.624] [#11](/bitcoin-bitcoin/11/) writing layer sha256:cfdd814cf5471383f503651cd4c6f1a1bcd432e92911dac06cff317cdbc6b305 0.0s done
    [07:50:37.624] [#11](/bitcoin-bitcoin/11/) writing layer sha256:de44b265507ae44b212defcb50694d666f136b35c1090d9709068bc861bb2d64
    [07:50:37.963] [#11](/bitcoin-bitcoin/11/) writing layer sha256:de44b265507ae44b212defcb50694d666f136b35c1090d9709068bc861bb2d64 0.5s done
    [07:50:37.963] [#11](/bitcoin-bitcoin/11/) writing layer sha256:febe602dc08a9a790bd092b613b049003ba7c75651d565cee664ecbebbe12bfc
    [07:50:38.044] [#11](/bitcoin-bitcoin/11/) preparing build cache for export 7.7s done
    [07:50:38.044] [#11](/bitcoin-bitcoin/11/) writing layer sha256:febe602dc08a9a790bd092b613b049003ba7c75651d565cee664ecbebbe12bfc 0.0s done
    [07:50:38.044] [#11](/bitcoin-bitcoin/11/) writing config sha256:16c21cc25bb3008714faa399494f419192dd76aa601d10f5c1216e218acc8d16 done
    [07:50:38.044] [#11](/bitcoin-bitcoin/11/) writing cache manifest sha256:9e819d3419d1888e5ee2d5514592865fe2f9693029640942d852d7e3ae0e347f done
    [07:50:38.044] [#11](/bitcoin-bitcoin/11/) DONE 7.7s
    [07:50:38.044] 
    [07:50:38.044]  
    1 warning found (use docker --debug to expand):
    [07:50:38.044] 
     - InvalidDefaultArgInFrom: Default value for ARG ${CI_IMAGE_NAME_TAG} results in empty or invalid base image name (line 8)
    [07:51:01.292] + '[' /cache/docker/ci-imgs ']'
    [07:51:01.292] + '[' -e /var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh/ci_macos_cross/index.json ']'
    [07:51:01.292] + echo 'Removing the existing docker build cache in /cache/docker/ci-imgs/ci_macos_cross'
    [07:51:01.292] Removing the existing docker build cache in /cache/docker/ci-imgs/ci_macos_cross
    [07:51:01.292] + rm -rf /cache/docker/ci-imgs/ci_macos_cross
    [07:51:01.321] + echo 'Moving the contents of /var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh/ci_macos_cross to /cache/docker/ci-imgs/ci_macos_cross'
    [07:51:01.321] Moving the contents of /var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh/ci_macos_cross to /cache/docker/ci-imgs/ci_macos_cross
    [07:51:01.321] + mv /var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh/ci_macos_cross /cache/docker/ci-imgs/ci_macos_cross
    [07:51:02.272] + rm -rf /var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh
    

    </details>

    It's important to note that this doesn't cache the base images (i.e. ubuntu:24.04, debian:bookworm, centos/amd:stream9, ...). On an ephemeral runner these can be cached with docker load and docker save. Otherwise, you might run into docker-hub rate-limits or connectivity problems to the registries.

    It would be nice to have, in the medium term, pre-build docker images for the CI tasks that can be pulled (and cached) to avoid building them in CI at all (unless there's a change in the underlying docker image of course). My understanding is, that this needs a bit of restructuring of the current apt package installation and similar. @willcl-ark showed me something in the past that looked promising.

  5. in ci/test/00_setup_env.sh:75 in 4149b00ae5 outdated
      67 | @@ -68,3 +68,8 @@ export CI_BASE_PACKAGES=${CI_BASE_PACKAGES:-build-essential pkg-config curl ca-c
      68 |  export GOAL=${GOAL:-install}
      69 |  export DIR_QA_ASSETS=${DIR_QA_ASSETS:-${BASE_SCRATCH_DIR}/qa-assets}
      70 |  export CI_RETRY_EXE=${CI_RETRY_EXE:-"retry --"}
      71 | +# The (optional) `docker build` cache directory.
      72 | +# If set, the folder exists only on the CI host and is used
      73 | +# as a build cache for the docker build of CI images. This
      74 | +# cache can be multiple GB in size.
      75 | +export DOCKER_BUILD_CACHE_HOST_DIR=${DOCKER_BUILD_CACHE_HOST_DIR:-}
    


    maflcko commented at 9:04 AM on December 20, 2024:

    Is this required? It seems like a no-op, apart from the docs.

    Though, docker is already caching by default, and this is normally not needed. So I think it is fine to omit the docs, or inline them in 02_run_container.

    I am even thinking this should have the DANGER_ prefix, as it calls rm -rf on the host for the folder.


    0xB10C commented at 10:37 AM on December 20, 2024:

    Good point. Will move docs to 02_run_container and prefix with DANGER_.

  6. 0xB10C commented at 10:37 AM on December 20, 2024: contributor

    A consideration might also to only specify --cache-to when CIRRUS_BRANCH=master. This makes PR runs slightly faster as the cache isn't overwritten. Also, most PRs don't introduce changes that cause a cache-miss, and if they do we want to not cache it until it's merged. Obviously, someone can fake this by overwriting CIRRUS_BRANCH to forcefully overwrite the cache for a while.

  7. in ci/test/02_run_container.sh:63 in 4149b00ae5 outdated
      58 | +      echo "Removing the existing docker build cache in ${DOCKER_BUILD_CACHE_OLD_DIR}"
      59 | +      rm -rf "${DOCKER_BUILD_CACHE_OLD_DIR}"
      60 | +      echo "Moving the contents of ${DOCKER_BUILD_CACHE_NEW_DIR} to ${DOCKER_BUILD_CACHE_OLD_DIR}"
      61 | +      mv "${DOCKER_BUILD_CACHE_NEW_DIR}" "${DOCKER_BUILD_CACHE_OLD_DIR}"
      62 | +    fi
      63 | +    rm -rf "${DOCKER_BUILD_CACHE_TEMPDIR}"
    


    maflcko commented at 11:09 AM on December 20, 2024:

    This will be left dangling and cause out-of-storage, if the build fails?


    maflcko commented at 11:11 AM on December 20, 2024:

    If this is fine, you could just drop the line (and leave it to the host to clean up). If not, I am not sure how to fix it.


    0xB10C commented at 12:59 PM on December 21, 2024:

    Right, on an ephemeral runner, this won't persist across runs - it's gone when the host shuts down. I'll remove it and clarify in the docs that DOCKER_BUILD_CACHE_HOST_DIR is for ephemeral runners.


    0xB10C commented at 1:17 PM on December 21, 2024:

    Using mktemp --directory ci-docker-build-cache-XXXXXXXXXX to allow a non-ephemeral host have a chance to identify them based on the prefix and clean them up.

  8. ci: optionally use local docker build cache
    By setting DANGER_DOCKER_BUILD_CACHE_HOST_DIR, the task-specific
    docker images built during the CI run can be cached. This allows,
    for example, ephemeral CI runners to reuse the docker images (or
    layers of it) from earlier runs, by persisting the image cache
    before the ephemeral CI runner is shut down. The cache keyed by
    `CONTAINER_NAME`.
    
    As --cache-to doesn't remove old cache files, the existing cache
    is removed after a successful `docker build` and the newly cached
    image is moved to it's location to avoid the cache from growing
    indefinitly with old, unused layers.
    
    When --cache-from doesn't find the directory, the cached version is
    a cache-miss, or the cache can't be imported for whatever other reason,
    it warns and `docker build` continues by building the docker image.
    
    This feature is opt-in. The documentation for the cache type=local
    can be found https://docs.docker.com/build/cache/backends/local/
    
    This replaces https://github.com/bitcoin/bitcoin/pull/31377
    e87429a2d0
  9. 0xB10C force-pushed on Dec 21, 2024
  10. 0xB10C commented at 3:20 PM on December 21, 2024: contributor

    Renamed to DANGER_DOCKER_BUILD_CACHE_HOST_DIR, moved the docs to ci/test/02_run_container.sh and not touching ci/test/00_setup_env.sh anymore, dropped rm -rf "${DOCKER_BUILD_CACHE_TEMPDIR}" but documented that non-ephemeral hosts would need to take care of cleaning up the temp dirs, and gave the temp dirs a ci-docker-build-cache- prefix. Thanks @maflcko.

  11. maflcko commented at 12:51 PM on January 9, 2025: member

    I haven't tested this, and it looks harmless and is easy to revert, if needed. So lgtm ACK e87429a2d0f23eb59526d335844fa5ff5b50b21f

  12. sedited approved
  13. sedited commented at 4:51 PM on January 17, 2025: contributor

    tACK e87429a2d0f23eb59526d335844fa5ff5b50b21f

  14. willcl-ark commented at 2:05 PM on January 20, 2025: member

    Concept ACK.

    Rather than removing the previous local cache and replacing with the newly-created cache, would it make sense to use dockers build cache pruning mechanism, docker builder prune, where we can specify removing either objects older than x, or to a certain storage (size) requirement?

    This way it seems like we could always append to the cache with --cache-to and prune to size or age as makes sense for the runner.

  15. 0xB10C commented at 3:38 PM on January 20, 2025: contributor

    Rather than removing the previous local cache and replacing with the newly-created cache, would it make sense to use dockers build cache pruning mechanism, docker builder prune, where we can specify removing either objects older than x, or to a certain storage (size) requirement?

    I wasn't aware of docker builder prune - though I don't think it would help use here: If we keep dangling image layers around, they will remain in the blob directory, but aren't referenced in the index.json file anymore. So even if they are available in theory, docker build --cache-from won't use them as they aren't indexed.

    See also #31377 (comment) for an example index.json file.

  16. willcl-ark commented at 4:11 PM on January 20, 2025: member

    I wasn't aware of docker builder prune - though I don't think it would help use here: If we keep dangling image layers around, they will remain in the blob directory, but aren't referenced in the index.json file anymore. So even if they are available in theory, docker build --cache-from won't use them as they aren't indexed.

    See also #31377 (comment) for an example index.json file.

    Reading https://github.com/moby/buildkit/issues/1896 it appears that you are correct, as the index.json isn't updated as I expected by --cache-from, or pruned as I expected by docker builder prune. Therefore I agree that the changes here make the most sense for us.

  17. willcl-ark approved
  18. willcl-ark commented at 4:12 PM on January 20, 2025: member

    ACK e87429a2d0f23eb59526d335844fa5ff5b50b21f

  19. achow101 commented at 9:43 PM on January 29, 2025: member

    ACK e87429a2d0f23eb59526d335844fa5ff5b50b21f

  20. achow101 merged this on Jan 29, 2025
  21. achow101 closed this on Jan 29, 2025

  22. sedited referenced this in commit e1405325c9 on Feb 3, 2025
  23. 0xB10C deleted the branch on Feb 11, 2025
  24. in ci/test/02_run_container.sh:67 in e87429a2d0
      62 | +  if [ "$DANGER_DOCKER_BUILD_CACHE_HOST_DIR" ]; then
      63 | +    if [ -e "${DOCKER_BUILD_CACHE_NEW_DIR}/index.json" ]; then
      64 | +      echo "Removing the existing docker build cache in ${DOCKER_BUILD_CACHE_OLD_DIR}"
      65 | +      rm -rf "${DOCKER_BUILD_CACHE_OLD_DIR}"
      66 | +      echo "Moving the contents of ${DOCKER_BUILD_CACHE_NEW_DIR} to ${DOCKER_BUILD_CACHE_OLD_DIR}"
      67 | +      mv "${DOCKER_BUILD_CACHE_NEW_DIR}" "${DOCKER_BUILD_CACHE_OLD_DIR}"
    


    maflcko commented at 2:13 PM on February 19, 2025:

    I wonder if any of this could fail? While you claimed that docker can handle an parallel rm -rf "${DOCKER_BUILD_CACHE_OLD_DIR}", I wonder if the mv and rm could fail when done concurrently. I guess rm -rf never fails, and mv is atomic. However, running two mv at the same time seems like one should fail. And running mv and rm at the same time, seem like it could corrupt the cache.


    0xB10C commented at 4:23 PM on February 19, 2025:

    However, running two mv at the same time seems like one should fail. And running mv and rm at the same time, seem like it could corrupt the cache.

    If multiple jobs operate on the same build cache folder at the same time, then probably yes.

  25. stickies-v referenced this in commit d760fd3dda on Mar 17, 2025
  26. stickies-v referenced this in commit cc83553352 on Mar 17, 2025
  27. stickies-v referenced this in commit 2614933f06 on Mar 17, 2025
  28. stickies-v referenced this in commit b70418c5fc on Mar 17, 2025
  29. stickies-v referenced this in commit 69f8a1fe50 on Mar 17, 2025
  30. willcl-ark referenced this in commit d0d1f4f9eb on Jul 15, 2025
  31. willcl-ark referenced this in commit 02f5ce602a on Jul 15, 2025
  32. willcl-ark referenced this in commit 579e0381b1 on Jul 15, 2025
  33. willcl-ark referenced this in commit b896710c6b on Jul 16, 2025
  34. willcl-ark referenced this in commit 6915e6f8e4 on Jul 16, 2025
  35. willcl-ark referenced this in commit 3c42538755 on Jul 16, 2025
  36. m3dwards referenced this in commit 43561062f7 on Jul 16, 2025
  37. m3dwards referenced this in commit d17a1c0eda on Jul 16, 2025
  38. willcl-ark referenced this in commit 0059530be6 on Jul 17, 2025
  39. willcl-ark referenced this in commit eb7615fe39 on Jul 17, 2025
  40. willcl-ark referenced this in commit 9774246353 on Jul 17, 2025
  41. willcl-ark referenced this in commit e5295f7ad6 on Jul 18, 2025
  42. willcl-ark referenced this in commit 0ba36bb9f9 on Jul 25, 2025
  43. willcl-ark referenced this in commit 9825ac93eb on Jul 25, 2025
  44. willcl-ark referenced this in commit 03ee46598b on Jul 25, 2025
  45. willcl-ark referenced this in commit 1ae84714b6 on Jul 25, 2025
  46. willcl-ark referenced this in commit d1f1d32aee on Jul 25, 2025
  47. willcl-ark referenced this in commit 59f8302917 on Jul 25, 2025
  48. willcl-ark referenced this in commit 4f4e6cb69f on Jul 25, 2025
  49. willcl-ark referenced this in commit 7974865f13 on Jul 25, 2025
  50. willcl-ark referenced this in commit 91f3c02da6 on Jul 25, 2025
  51. willcl-ark referenced this in commit 0989aa5125 on Jul 25, 2025
  52. willcl-ark referenced this in commit e83c3718f2 on Jul 25, 2025
  53. willcl-ark referenced this in commit c906af75a5 on Jul 26, 2025
  54. willcl-ark referenced this in commit 303b36e7e5 on Jul 28, 2025
  55. willcl-ark referenced this in commit 5d6c930894 on Jul 28, 2025
  56. willcl-ark referenced this in commit b82df4ee33 on Jul 28, 2025
  57. willcl-ark referenced this in commit a77cc65e8f on Jul 28, 2025
  58. willcl-ark referenced this in commit 2a11f80746 on Jul 28, 2025
  59. willcl-ark referenced this in commit e2c24b2f81 on Jul 28, 2025
  60. willcl-ark referenced this in commit 2919f3017d on Jul 28, 2025
  61. willcl-ark referenced this in commit 9965727a45 on Jul 30, 2025
  62. willcl-ark referenced this in commit b2a0f12a43 on Jul 31, 2025
  63. m3dwards referenced this in commit b8c87f283f on Aug 5, 2025
  64. willcl-ark referenced this in commit 41bf74af9c on Aug 6, 2025
  65. willcl-ark referenced this in commit 7744c4951e on Aug 6, 2025
  66. willcl-ark referenced this in commit db0b451171 on Aug 6, 2025
  67. willcl-ark referenced this in commit 52458e1d9e on Aug 6, 2025
  68. willcl-ark referenced this in commit 7438d53a51 on Aug 6, 2025
  69. willcl-ark referenced this in commit 2cbe9658f0 on Aug 7, 2025
  70. willcl-ark referenced this in commit 923b5a7373 on Aug 8, 2025
  71. willcl-ark referenced this in commit 7750bab903 on Aug 13, 2025
  72. willcl-ark referenced this in commit 94751796c6 on Aug 13, 2025
  73. willcl-ark referenced this in commit d4c9550dd0 on Aug 13, 2025
  74. willcl-ark referenced this in commit e73f2ad6fa on Aug 13, 2025
  75. willcl-ark referenced this in commit 18f6be09d0 on Sep 1, 2025
  76. willcl-ark referenced this in commit 4d06628243 on Sep 2, 2025
  77. willcl-ark referenced this in commit 713e903521 on Sep 2, 2025
  78. Sjors referenced this in commit c89e501e45 on Sep 3, 2025
  79. Sjors referenced this in commit eacbe19792 on Sep 3, 2025
  80. Sjors referenced this in commit 482659dbb8 on Sep 12, 2025
  81. willcl-ark referenced this in commit a1865b64c6 on Sep 16, 2025
  82. willcl-ark referenced this in commit 1fa22669e5 on Sep 16, 2025
  83. willcl-ark referenced this in commit b82b4f736c on Sep 16, 2025
  84. willcl-ark referenced this in commit 67a9945293 on Sep 19, 2025
  85. willcl-ark referenced this in commit 0a649d07c9 on Sep 23, 2025
  86. bug-castercv502 referenced this in commit bf2e6853b0 on Sep 28, 2025
  87. ivanlele referenced this in commit 15cc9c4e7d on Apr 14, 2026
  88. morozow referenced this in commit eff3184589 on May 8, 2026
  89. Kino1994 referenced this in commit aa30d9b8d5 on Jun 28, 2026
  90. Kino1994 referenced this in commit f2f4f04a5f on Jun 28, 2026
  91. BigcoinBGC referenced this in commit 8d4e7d684c on Jun 30, 2026
  92. BigcoinBGC referenced this in commit 4626d7598e on Jun 30, 2026
  93. bitcoin locked this on Jul 30, 2026

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-08-14 17:51 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me