ci: optionally use local docker build cache #31545

pull 0xB10C wants to merge 1 commits into bitcoin:master from 0xB10C:2024-11-ci-optional-local-docker-build-cache changing 1 files +33 −0
  1. 0xB10C commented at 7:59 AM on December 20, 2024: contributor

    By setting DANGER_DOCKER_BUILD_CACHE_HOST_DIR, the task-specific docker images built during the CI run can be cached. This allows, for example, ephemeral CI runners to reuse the docker images (or layers of it) from earlier runs, by persisting the image cache before the ephemeral CI runner is shut down. The cache keyed by CONTAINER_NAME.

    As --cache-to doesn't remove old cache files, the existing cache is removed after a successful docker build and the newly cached image is moved to it's location to avoid the cache from growing indefinitely with old, unused layers.

    When --cache-from doesn't find the directory, the cached version is a cache-miss, or the cache can't be imported for whatever other reason, it warns and docker build continues by building the docker image.

    This feature is opt-in. The documentation for the docker build cache of type=local can be found on https://docs.docker.com/build/cache/backends/local/

    This replaces #31377 - some of the discussion there might provide more context.

  2. DrahtBot commented at 7:59 AM on December 20, 2024: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31545.

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK maflcko, TheCharlatan, willcl-ark, achow101

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

  3. DrahtBot added the label Tests on Dec 20, 2024
  4. 0xB10C commented at 8:41 AM on December 20, 2024: contributor

    Some additional context:

    I've been testing and using this with ephemeral cirrus runners I've been working on. See a run of the initial commit of this PR here: https://cirrus-ci.com/build/6017527349772288. If this PR is merged, these could be used on bitcoin/bitcoin after a while - once people have rebased their PRs to include this change. Old branches, for example, 28.x backports won't be able to use this (unless backported) but the docker images on old branches might generally be different and will be needed to rebuild.

    The cache for all of our self-hosted x86_64 Cirrus tasks (all but the ARM task) is about 5.2 GB. I don't expect this to grow or shrink significantly.

    $ du -h -d1 .
    477M	./ci_macos_cross
    837M	./ci_win64
    356M	./ci_native_nowallet_libbitcoinkernel
    205M	./ci_native_previous_releases
    1.4G	./ci_native_msan
    314M	./ci_i686_centos
    338M	./ci_native_tsan
    693M	./ci_native_tidy
    359M	./ci_i686_multiprocess
    351M	./ci_native_fuzz
    5.2G	.
    

    Loading, extracting and importing the cache artifacts on a cache-hit takes a few seconds. This is slower compared to the current non-ephemeral self-hosted runners if they already have a fitting image in their docker image store. Here's the log output for a cache-hit on my runner's macos-cross task.

    <details> <summary> `docker build` log section

    </summary>

    [07:49:56.064] + echo 'Creating docker.io/ubuntu:24.04 container to run in'
    [07:49:56.064] Creating docker.io/ubuntu:24.04 container to run in
    [07:49:56.064] + DOCKER_BUILD_CACHE_ARG=
    [07:49:56.064] + DOCKER_BUILD_CACHE_TEMPDIR=
    [07:49:56.064] + DOCKER_BUILD_CACHE_OLD_DIR=
    [07:49:56.064] + DOCKER_BUILD_CACHE_NEW_DIR=
    [07:49:56.064] + '[' /cache/docker/ci-imgs ']'
    [07:49:56.064] + DOCKER_BUILD_CACHE_OLD_DIR=/cache/docker/ci-imgs/ci_macos_cross
    [07:49:56.064] ++ mktemp --directory
    [07:49:56.074] + DOCKER_BUILD_CACHE_TEMPDIR=/var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh
    [07:49:56.074] + DOCKER_BUILD_CACHE_NEW_DIR=/var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh/ci_macos_cross
    [07:49:56.074] + DOCKER_BUILD_CACHE_ARG='--cache-from type=local,src=/cache/docker/ci-imgs/ci_macos_cross --cache-to type=local,dest=/var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh/ci_macos_cross,mode=max'
    [07:49:56.074] + DOCKER_BUILDKIT=1
    [07:49:56.074] + docker build --file /var/lib/cirrus-worker/tmp/cirrus-build/ci/test_imagefile --build-arg CI_IMAGE_NAME_TAG=docker.io/ubuntu:24.04 --build-arg FILE_ENV=./ci/test/00_setup_env_mac_cross.sh --label=bitcoin-ci-test --tag=ci_macos_cross --cache-from type=local,src=/cache/docker/ci-imgs/ci_macos_cross --cache-to type=local,dest=/var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh/ci_macos_cross,mode=max /var/lib/cirrus-worker/tmp/cirrus-build
    [07:49:56.738] [#0](/bitcoin-bitcoin/0/) building with "default" instance using docker driver
    [07:49:56.738] 
    [07:49:56.738] [#1](/bitcoin-bitcoin/1/) [internal] load build definition from test_imagefile
    [07:49:56.738] [#1](/bitcoin-bitcoin/1/) transferring dockerfile: 600B done
    [07:49:56.738] [#1](/bitcoin-bitcoin/1/) WARN: InvalidDefaultArgInFrom: Default value for ARG ${CI_IMAGE_NAME_TAG} results in empty or invalid base image name (line 8)
    [07:49:56.738] [#1](/bitcoin-bitcoin/1/) DONE 0.0s
    [07:49:56.738] 
    [07:49:56.738] [#2](/bitcoin-bitcoin/2/) [internal] load metadata for docker.io/library/ubuntu:24.04
    [07:49:56.738] [#2](/bitcoin-bitcoin/2/) DONE 0.0s
    [07:49:56.738] 
    [07:49:56.738] [#3](/bitcoin-bitcoin/3/) [internal] load .dockerignore
    [07:49:56.738] [#3](/bitcoin-bitcoin/3/) transferring context: 2B done
    [07:49:56.738] [#3](/bitcoin-bitcoin/3/) DONE 0.0s
    [07:49:56.738] 
    [07:49:56.738] [#4](/bitcoin-bitcoin/4/) [internal] load build context
    [07:49:56.738] [#4](/bitcoin-bitcoin/4/) DONE 0.0s
    [07:49:56.738] 
    [07:49:56.738] [#5](/bitcoin-bitcoin/5/) importing cache manifest from local:13157433447571386053
    [07:49:56.738] [#5](/bitcoin-bitcoin/5/) inferred cache manifest type: application/vnd.oci.image.index.v1+json done
    [07:49:56.738] [#5](/bitcoin-bitcoin/5/) DONE 0.0s
    [07:49:56.738] 
    [07:49:56.738] [#6](/bitcoin-bitcoin/6/) [1/4] FROM docker.io/library/ubuntu:24.04@sha256:80dd3c3b9c6cecb9f1667e9290b3bc61b78c2678c02cbdae5f0fea92cc6734ab
    [07:49:56.738] [#6](/bitcoin-bitcoin/6/) resolve docker.io/library/ubuntu:24.04@sha256:80dd3c3b9c6cecb9f1667e9290b3bc61b78c2678c02cbdae5f0fea92cc6734ab
    [07:49:57.299] [#6](/bitcoin-bitcoin/6/) resolve docker.io/library/ubuntu:24.04@sha256:80dd3c3b9c6cecb9f1667e9290b3bc61b78c2678c02cbdae5f0fea92cc6734ab 0.7s done
    [07:49:57.408] [#6](/bitcoin-bitcoin/6/) DONE 0.7s
    [07:49:57.408] 
    [07:49:57.408] [#4](/bitcoin-bitcoin/4/) [internal] load build context
    [07:49:57.408] [#4](/bitcoin-bitcoin/4/) transferring context: 12.88kB done
    [07:49:57.408] [#4](/bitcoin-bitcoin/4/) DONE 0.0s
    [07:49:57.408] 
    [07:49:57.408] [#7](/bitcoin-bitcoin/7/) [2/4] COPY ./ci/retry/retry /usr/bin/retry
    [07:49:57.408] [#7](/bitcoin-bitcoin/7/) CACHED
    [07:49:57.408] 
    [07:49:57.408] [#8](/bitcoin-bitcoin/8/) [3/4] COPY ./ci/test/00_setup_env.sh ././ci/test/00_setup_env_mac_cross.sh ./ci/test/01_base_install.sh /ci_container_base/ci/test/
    [07:49:57.408] [#8](/bitcoin-bitcoin/8/) CACHED
    [07:49:57.408] 
    [07:49:57.408] [#9](/bitcoin-bitcoin/9/) [4/4] RUN ["bash", "-c", "cd /ci_container_base/ && set -o errexit && source ./ci/test/00_setup_env.sh && ./ci/test/01_base_install.sh"]
    [07:49:57.408] [#9](/bitcoin-bitcoin/9/) sha256:febe602dc08a9a790bd092b613b049003ba7c75651d565cee664ecbebbe12bfc 1.85kB / 1.85kB 0.0s done
    [07:49:57.408] [#9](/bitcoin-bitcoin/9/) extracting sha256:febe602dc08a9a790bd092b613b049003ba7c75651d565cee664ecbebbe12bfc done
    [07:49:57.408] [#9](/bitcoin-bitcoin/9/) sha256:cfdd814cf5471383f503651cd4c6f1a1bcd432e92911dac06cff317cdbc6b305 3.38kB / 3.38kB 0.0s done
    [07:49:57.581] [#9](/bitcoin-bitcoin/9/) extracting sha256:cfdd814cf5471383f503651cd4c6f1a1bcd432e92911dac06cff317cdbc6b305 0.0s done
    [07:49:57.588] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 8.39MB / 470.35MB 0.2s
    [07:49:58.038] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 38.80MB / 470.35MB 0.6s
    [07:49:58.488] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 70.25MB / 470.35MB 1.1s
    [07:49:58.938] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 102.76MB / 470.35MB 1.5s
    [07:49:59.388] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 134.22MB / 470.35MB 2.0s
    [07:49:59.839] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 164.63MB / 470.35MB 2.4s
    [07:50:00.292] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 193.99MB / 470.35MB 2.9s
    [07:50:00.738] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 223.35MB / 470.35MB 3.3s
    [07:50:01.191] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 254.80MB / 470.35MB 3.8s
    [07:50:01.638] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 286.26MB / 470.35MB 4.2s
    [07:50:02.089] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 315.62MB / 470.35MB 4.7s
    [07:50:02.538] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 346.03MB / 470.35MB 5.1s
    [07:50:02.988] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 376.44MB / 470.35MB 5.6s
    [07:50:03.438] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 406.85MB / 470.35MB 6.0s
    [07:50:03.888] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 436.21MB / 470.35MB 6.5s
    [07:50:04.338] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 468.71MB / 470.35MB 6.9s
    [07:50:04.811] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 470.35MB / 470.35MB 7.2s done
    [07:50:04.811] [#9](/bitcoin-bitcoin/9/) extracting sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8
    [07:50:30.213] [#9](/bitcoin-bitcoin/9/) extracting sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 25.6s done
    [07:50:30.453] [#9](/bitcoin-bitcoin/9/) CACHED
    [07:50:30.453] 
    [07:50:30.453] [#10](/bitcoin-bitcoin/10/) exporting to image
    [07:50:30.453] [#10](/bitcoin-bitcoin/10/) exporting layers done
    [07:50:30.453] [#10](/bitcoin-bitcoin/10/) exporting manifest sha256:35b82073fb170cde1c1f340217048fa5fdca7bf710eb9836e1e75ae8f632c18a done
    [07:50:30.453] [#10](/bitcoin-bitcoin/10/) exporting config sha256:040ba6df0bbe7839075a62de0a03179747c82ddff2a33a9a11d72202cd6ed3c7 done
    [07:50:30.453] [#10](/bitcoin-bitcoin/10/) exporting attestation manifest sha256:b354c939c9e7630028a036a105ea52f56a74b91c54148db376b51477b6e47668 done
    [07:50:30.453] [#10](/bitcoin-bitcoin/10/) exporting manifest list sha256:550407c300a249757f8f145ac481f361b7e7f22de8b62639006b7f2d202369d5 done
    [07:50:30.453] [#10](/bitcoin-bitcoin/10/) naming to docker.io/library/ci_macos_cross:latest done
    [07:50:30.453] [#10](/bitcoin-bitcoin/10/) unpacking to docker.io/library/ci_macos_cross:latest done
    [07:50:30.453] [#10](/bitcoin-bitcoin/10/) DONE 0.1s
    [07:50:30.453] 
    [07:50:30.453] [#11](/bitcoin-bitcoin/11/) exporting cache to client directory
    [07:50:30.453] [#11](/bitcoin-bitcoin/11/) preparing build cache for export
    [07:50:30.453] [#11](/bitcoin-bitcoin/11/) writing layer sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8
    [07:50:37.448] [#11](/bitcoin-bitcoin/11/) writing layer sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 7.1s done
    [07:50:37.448] [#11](/bitcoin-bitcoin/11/) writing layer sha256:cfdd814cf5471383f503651cd4c6f1a1bcd432e92911dac06cff317cdbc6b305
    [07:50:37.624] [#11](/bitcoin-bitcoin/11/) writing layer sha256:cfdd814cf5471383f503651cd4c6f1a1bcd432e92911dac06cff317cdbc6b305 0.0s done
    [07:50:37.624] [#11](/bitcoin-bitcoin/11/) writing layer sha256:de44b265507ae44b212defcb50694d666f136b35c1090d9709068bc861bb2d64
    [07:50:37.963] [#11](/bitcoin-bitcoin/11/) writing layer sha256:de44b265507ae44b212defcb50694d666f136b35c1090d9709068bc861bb2d64 0.5s done
    [07:50:37.963] [#11](/bitcoin-bitcoin/11/) writing layer sha256:febe602dc08a9a790bd092b613b049003ba7c75651d565cee664ecbebbe12bfc
    [07:50:38.044] [#11](/bitcoin-bitcoin/11/) preparing build cache for export 7.7s done
    [07:50:38.044] [#11](/bitcoin-bitcoin/11/) writing layer sha256:febe602dc08a9a790bd092b613b049003ba7c75651d565cee664ecbebbe12bfc 0.0s done
    [07:50:38.044] [#11](/bitcoin-bitcoin/11/) writing config sha256:16c21cc25bb3008714faa399494f419192dd76aa601d10f5c1216e218acc8d16 done
    [07:50:38.044] [#11](/bitcoin-bitcoin/11/) writing cache manifest sha256:9e819d3419d1888e5ee2d5514592865fe2f9693029640942d852d7e3ae0e347f done
    [07:50:38.044] [#11](/bitcoin-bitcoin/11/) DONE 7.7s
    [07:50:38.044] 
    [07:50:38.044]  
    1 warning found (use docker --debug to expand):
    [07:50:38.044] 
     - InvalidDefaultArgInFrom: Default value for ARG ${CI_IMAGE_NAME_TAG} results in empty or invalid base image name (line 8)
    [07:51:01.292] + '[' /cache/docker/ci-imgs ']'
    [07:51:01.292] + '[' -e /var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh/ci_macos_cross/index.json ']'
    [07:51:01.292] + echo 'Removing the existing docker build cache in /cache/docker/ci-imgs/ci_macos_cross'
    [07:51:01.292] Removing the existing docker build cache in /cache/docker/ci-imgs/ci_macos_cross
    [07:51:01.292] + rm -rf /cache/docker/ci-imgs/ci_macos_cross
    [07:51:01.321] + echo 'Moving the contents of /var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh/ci_macos_cross to /cache/docker/ci-imgs/ci_macos_cross'
    [07:51:01.321] Moving the contents of /var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh/ci_macos_cross to /cache/docker/ci-imgs/ci_macos_cross
    [07:51:01.321] + mv /var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh/ci_macos_cross /cache/docker/ci-imgs/ci_macos_cross
    [07:51:02.272] + rm -rf /var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh
    

    </details>

    It's important to note that this doesn't cache the base images (i.e. ubuntu:24.04, debian:bookworm, centos/amd:stream9, ...). On an ephemeral runner these can be cached with docker load and docker save. Otherwise, you might run into docker-hub rate-limits or connectivity problems to the registries.

    It would be nice to have, in the medium term, pre-build docker images for the CI tasks that can be pulled (and cached) to avoid building them in CI at all (unless there's a change in the underlying docker image of course). My understanding is, that this needs a bit of restructuring of the current apt package installation and similar. @willcl-ark showed me something in the past that looked promising.

  5. in ci/test/00_setup_env.sh:75 in 4149b00ae5 outdated
      67 | @@ -68,3 +68,8 @@ export CI_BASE_PACKAGES=${CI_BASE_PACKAGES:-build-essential pkg-config curl ca-c
      68 |  export GOAL=${GOAL:-install}
      69 |  export DIR_QA_ASSETS=${DIR_QA_ASSETS:-${BASE_SCRATCH_DIR}/qa-assets}
      70 |  export CI_RETRY_EXE=${CI_RETRY_EXE:-"retry --"}
      71 | +# The (optional) `docker build` cache directory.
      72 | +# If set, the folder exists only on the CI host and is used
      73 | +# as a build cache for the docker build of CI images. This
      74 | +# cache can be multiple GB in size.
      75 | +export DOCKER_BUILD_CACHE_HOST_DIR=${DOCKER_BUILD_CACHE_HOST_DIR:-}
    


    maflcko commented at 9:04 AM on December 20, 2024:

    Is this required? It seems like a no-op, apart from the docs.

    Though, docker is already caching by default, and this is normally not needed. So I think it is fine to omit the docs, or inline them in 02_run_container.

    I am even thinking this should have the DANGER_ prefix, as it calls rm -rf on the host for the folder.


    0xB10C commented at 10:37 AM on December 20, 2024:

    Good point. Will move docs to 02_run_container and prefix with DANGER_.

  6. 0xB10C commented at 10:37 AM on December 20, 2024: contributor

    A consideration might also to only specify --cache-to when CIRRUS_BRANCH=master. This makes PR runs slightly faster as the cache isn't overwritten. Also, most PRs don't introduce changes that cause a cache-miss, and if they do we want to not cache it until it's merged. Obviously, someone can fake this by overwriting CIRRUS_BRANCH to forcefully overwrite the cache for a while.

  7. in ci/test/02_run_container.sh:63 in 4149b00ae5 outdated
      58 | +      echo "Removing the existing docker build cache in ${DOCKER_BUILD_CACHE_OLD_DIR}"
      59 | +      rm -rf "${DOCKER_BUILD_CACHE_OLD_DIR}"
      60 | +      echo "Moving the contents of ${DOCKER_BUILD_CACHE_NEW_DIR} to ${DOCKER_BUILD_CACHE_OLD_DIR}"
      61 | +      mv "${DOCKER_BUILD_CACHE_NEW_DIR}" "${DOCKER_BUILD_CACHE_OLD_DIR}"
      62 | +    fi
      63 | +    rm -rf "${DOCKER_BUILD_CACHE_TEMPDIR}"
    


    maflcko commented at 11:09 AM on December 20, 2024:

    This will be left dangling and cause out-of-storage, if the build fails?


    maflcko commented at 11:11 AM on December 20, 2024:

    If this is fine, you could just drop the line (and leave it to the host to clean up). If not, I am not sure how to fix it.


    0xB10C commented at 12:59 PM on December 21, 2024:

    Right, on an ephemeral runner, this won't persist across runs - it's gone when the host shuts down. I'll remove it and clarify in the docs that DOCKER_BUILD_CACHE_HOST_DIR is for ephemeral runners.


    0xB10C commented at 1:17 PM on December 21, 2024:

    Using mktemp --directory ci-docker-build-cache-XXXXXXXXXX to allow a non-ephemeral host have a chance to identify them based on the prefix and clean them up.

  8. ci: optionally use local docker build cache
    By setting DANGER_DOCKER_BUILD_CACHE_HOST_DIR, the task-specific
    docker images built during the CI run can be cached. This allows,
    for example, ephemeral CI runners to reuse the docker images (or
    layers of it) from earlier runs, by persisting the image cache
    before the ephemeral CI runner is shut down. The cache keyed by
    `CONTAINER_NAME`.
    
    As --cache-to doesn't remove old cache files, the existing cache
    is removed after a successful `docker build` and the newly cached
    image is moved to it's location to avoid the cache from growing
    indefinitly with old, unused layers.
    
    When --cache-from doesn't find the directory, the cached version is
    a cache-miss, or the cache can't be imported for whatever other reason,
    it warns and `docker build` continues by building the docker image.
    
    This feature is opt-in. The documentation for the cache type=local
    can be found https://docs.docker.com/build/cache/backends/local/
    
    This replaces https://github.com/bitcoin/bitcoin/pull/31377
    e87429a2d0
  9. 0xB10C force-pushed on Dec 21, 2024
  10. 0xB10C commented at 3:20 PM on December 21, 2024: contributor

    Renamed to DANGER_DOCKER_BUILD_CACHE_HOST_DIR, moved the docs to ci/test/02_run_container.sh and not touching ci/test/00_setup_env.sh anymore, dropped rm -rf "${DOCKER_BUILD_CACHE_TEMPDIR}" but documented that non-ephemeral hosts would need to take care of cleaning up the temp dirs, and gave the temp dirs a ci-docker-build-cache- prefix. Thanks @maflcko.

  11. maflcko commented at 12:51 PM on January 9, 2025: member

    I haven't tested this, and it looks harmless and is easy to revert, if needed. So lgtm ACK e87429a2d0f23eb59526d335844fa5ff5b50b21f

  12. TheCharlatan approved
  13. TheCharlatan commented at 4:51 PM on January 17, 2025: contributor

    tACK e87429a2d0f23eb59526d335844fa5ff5b50b21f

  14. willcl-ark commented at 2:05 PM on January 20, 2025: member

    Concept ACK.

    Rather than removing the previous local cache and replacing with the newly-created cache, would it make sense to use dockers build cache pruning mechanism, docker builder prune, where we can specify removing either objects older than x, or to a certain storage (size) requirement?

    This way it seems like we could always append to the cache with --cache-to and prune to size or age as makes sense for the runner.

  15. 0xB10C commented at 3:38 PM on January 20, 2025: contributor

    Rather than removing the previous local cache and replacing with the newly-created cache, would it make sense to use dockers build cache pruning mechanism, docker builder prune, where we can specify removing either objects older than x, or to a certain storage (size) requirement?

    I wasn't aware of docker builder prune - though I don't think it would help use here: If we keep dangling image layers around, they will remain in the blob directory, but aren't referenced in the index.json file anymore. So even if they are available in theory, docker build --cache-from won't use them as they aren't indexed.

    See also #31377 (comment) for an example index.json file.

  16. willcl-ark commented at 4:11 PM on January 20, 2025: member

    I wasn't aware of docker builder prune - though I don't think it would help use here: If we keep dangling image layers around, they will remain in the blob directory, but aren't referenced in the index.json file anymore. So even if they are available in theory, docker build --cache-from won't use them as they aren't indexed.

    See also #31377 (comment) for an example index.json file.

    Reading https://github.com/moby/buildkit/issues/1896 it appears that you are correct, as the index.json isn't updated as I expected by --cache-from, or pruned as I expected by docker builder prune. Therefore I agree that the changes here make the most sense for us.

  17. willcl-ark approved
  18. willcl-ark commented at 4:12 PM on January 20, 2025: member

    ACK e87429a2d0f23eb59526d335844fa5ff5b50b21f

  19. achow101 commented at 9:43 PM on January 29, 2025: member

    ACK e87429a2d0f23eb59526d335844fa5ff5b50b21f

  20. achow101 merged this on Jan 29, 2025
  21. achow101 closed this on Jan 29, 2025

  22. TheCharlatan referenced this in commit e1405325c9 on Feb 3, 2025
  23. 0xB10C deleted the branch on Feb 11, 2025
  24. in ci/test/02_run_container.sh:67 in e87429a2d0
      62 | +  if [ "$DANGER_DOCKER_BUILD_CACHE_HOST_DIR" ]; then
      63 | +    if [ -e "${DOCKER_BUILD_CACHE_NEW_DIR}/index.json" ]; then
      64 | +      echo "Removing the existing docker build cache in ${DOCKER_BUILD_CACHE_OLD_DIR}"
      65 | +      rm -rf "${DOCKER_BUILD_CACHE_OLD_DIR}"
      66 | +      echo "Moving the contents of ${DOCKER_BUILD_CACHE_NEW_DIR} to ${DOCKER_BUILD_CACHE_OLD_DIR}"
      67 | +      mv "${DOCKER_BUILD_CACHE_NEW_DIR}" "${DOCKER_BUILD_CACHE_OLD_DIR}"
    


    maflcko commented at 2:13 PM on February 19, 2025:

    I wonder if any of this could fail? While you claimed that docker can handle an parallel rm -rf "${DOCKER_BUILD_CACHE_OLD_DIR}", I wonder if the mv and rm could fail when done concurrently. I guess rm -rf never fails, and mv is atomic. However, running two mv at the same time seems like one should fail. And running mv and rm at the same time, seem like it could corrupt the cache.


    0xB10C commented at 4:23 PM on February 19, 2025:

    However, running two mv at the same time seems like one should fail. And running mv and rm at the same time, seem like it could corrupt the cache.

    If multiple jobs operate on the same build cache folder at the same time, then probably yes.


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-05-01 03:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me