ci: optionally use local docker build cache #31545

pull 0xB10C wants to merge 1 commits into bitcoin:master from 0xB10C:2024-11-ci-optional-local-docker-build-cache changing 2 files +33 −0
  1. 0xB10C commented at 7:59 am on December 20, 2024: contributor

    By setting DOCKER_BUILD_CACHE_HOST_DIR, the task-specific docker images built during the CI run can be cached. This allows, for example, ephemeral CI runners to reuse the docker images (or layers of it) from earlier runs, by persisting the image cache before the ephemeral CI runner is shut down. The cache keyed by CONTAINER_NAME.

    As --cache-to doesn’t remove old cache files, the existing cache is removed after a successful docker build and the newly cached image is moved to it’s location to avoid the cache from growing indefinitely with old, unused layers.

    When --cache-from doesn’t find the directory, the cached version is a cache-miss, or the cache can’t be imported for whatever other reason, it warns and docker build continues by building the docker image.

    This feature is opt-in. The documentation for the docker build cache of type=local can be found on https://docs.docker.com/build/cache/backends/local/

    This replaces #31377 - some of the discussion there might provide more context.

  2. ci: optionally use local docker build cache
    By setting DOCKER_BUILD_CACHE_HOST_DIR, the task-specific docker
    images built during the CI run can be cached. This allows, for
    example, ephemeral CI runners to reuse the docker images (or
    layers of it) from earlier runs, by persisting the image cache
    before the ephemeral CI runner is shut down. The cache keyed by
    `CONTAINER_NAME`.
    
    As --cache-to doesn't remove old cache files, the existing cache
    is removed after a successful `docker build` and the newly cached
    image is moved to it's location to avoid the cache from growing
    indefinitly with old, unused layers.
    
    When --cache-from doesn't find the directory, the cached version is
    a cache-miss, or the cache can't be imported for whatever other reason,
    it warns and `docker build` continues by building the docker image.
    
    This feature is opt-in. The documentation for the cache type=local
    can be found https://docs.docker.com/build/cache/backends/local/
    
    This replaces https://github.com/bitcoin/bitcoin/pull/31377
    4149b00ae5
  3. DrahtBot commented at 7:59 am on December 20, 2024: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31545.

    Reviews

    See the guideline for information on the review process. A summary of reviews will appear here.

  4. DrahtBot added the label Tests on Dec 20, 2024
  5. 0xB10C commented at 8:41 am on December 20, 2024: contributor

    Some additional context:

    I’ve been testing and using this with ephemeral cirrus runners I’ve been working on. See a run of the initial commit of this PR here: https://cirrus-ci.com/build/6017527349772288. If this PR is merged, these could be used on bitcoin/bitcoin after a while - once people have rebased their PRs to include this change. Old branches, for example, 28.x backports won’t be able to use this (unless backported) but the docker images on old branches might generally be different and will be needed to rebuild.

    The cache for all of our self-hosted x86_64 Cirrus tasks (all but the ARM task) is about 5.2 GB. I don’t expect this to grow or shrink significantly.

     0$ du -h -d1 .
     1477M	./ci_macos_cross
     2837M	./ci_win64
     3356M	./ci_native_nowallet_libbitcoinkernel
     4205M	./ci_native_previous_releases
     51.4G	./ci_native_msan
     6314M	./ci_i686_centos
     7338M	./ci_native_tsan
     8693M	./ci_native_tidy
     9359M	./ci_i686_multiprocess
    10351M	./ci_native_fuzz
    115.2G	.
    

    Loading, extracting and importing the cache artifacts on a cache-hit takes a few seconds. This is slower compared to the current non-ephemeral self-hosted runners if they already have a fitting image in their docker image store. Here’s the log output for a cache-hit on my runner’s macos-cross task.

      0[07:49:56.064] + echo 'Creating docker.io/ubuntu:24.04 container to run in'
      1[07:49:56.064] Creating docker.io/ubuntu:24.04 container to run in
      2[07:49:56.064] + DOCKER_BUILD_CACHE_ARG=
      3[07:49:56.064] + DOCKER_BUILD_CACHE_TEMPDIR=
      4[07:49:56.064] + DOCKER_BUILD_CACHE_OLD_DIR=
      5[07:49:56.064] + DOCKER_BUILD_CACHE_NEW_DIR=
      6[07:49:56.064] + '[' /cache/docker/ci-imgs ']'
      7[07:49:56.064] + DOCKER_BUILD_CACHE_OLD_DIR=/cache/docker/ci-imgs/ci_macos_cross
      8[07:49:56.064] ++ mktemp --directory
      9[07:49:56.074] + DOCKER_BUILD_CACHE_TEMPDIR=/var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh
     10[07:49:56.074] + DOCKER_BUILD_CACHE_NEW_DIR=/var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh/ci_macos_cross
     11[07:49:56.074] + DOCKER_BUILD_CACHE_ARG='--cache-from type=local,src=/cache/docker/ci-imgs/ci_macos_cross --cache-to type=local,dest=/var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh/ci_macos_cross,mode=max'
     12[07:49:56.074] + DOCKER_BUILDKIT=1
     13[07:49:56.074] + docker build --file /var/lib/cirrus-worker/tmp/cirrus-build/ci/test_imagefile --build-arg CI_IMAGE_NAME_TAG=docker.io/ubuntu:24.04 --build-arg FILE_ENV=./ci/test/00_setup_env_mac_cross.sh --label=bitcoin-ci-test --tag=ci_macos_cross --cache-from type=local,src=/cache/docker/ci-imgs/ci_macos_cross --cache-to type=local,dest=/var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh/ci_macos_cross,mode=max /var/lib/cirrus-worker/tmp/cirrus-build
     14[07:49:56.738] [#0](/bitcoin-bitcoin/0/) building with "default" instance using docker driver
     15[07:49:56.738] 
     16[07:49:56.738] [#1](/bitcoin-bitcoin/1/) [internal] load build definition from test_imagefile
     17[07:49:56.738] [#1](/bitcoin-bitcoin/1/) transferring dockerfile: 600B done
     18[07:49:56.738] [#1](/bitcoin-bitcoin/1/) WARN: InvalidDefaultArgInFrom: Default value for ARG ${CI_IMAGE_NAME_TAG} results in empty or invalid base image name (line 8)
     19[07:49:56.738] [#1](/bitcoin-bitcoin/1/) DONE 0.0s
     20[07:49:56.738] 
     21[07:49:56.738] [#2](/bitcoin-bitcoin/2/) [internal] load metadata for docker.io/library/ubuntu:24.04
     22[07:49:56.738] [#2](/bitcoin-bitcoin/2/) DONE 0.0s
     23[07:49:56.738] 
     24[07:49:56.738] [#3](/bitcoin-bitcoin/3/) [internal] load .dockerignore
     25[07:49:56.738] [#3](/bitcoin-bitcoin/3/) transferring context: 2B done
     26[07:49:56.738] [#3](/bitcoin-bitcoin/3/) DONE 0.0s
     27[07:49:56.738] 
     28[07:49:56.738] [#4](/bitcoin-bitcoin/4/) [internal] load build context
     29[07:49:56.738] [#4](/bitcoin-bitcoin/4/) DONE 0.0s
     30[07:49:56.738] 
     31[07:49:56.738] [#5](/bitcoin-bitcoin/5/) importing cache manifest from local:13157433447571386053
     32[07:49:56.738] [#5](/bitcoin-bitcoin/5/) inferred cache manifest type: application/vnd.oci.image.index.v1+json done
     33[07:49:56.738] [#5](/bitcoin-bitcoin/5/) DONE 0.0s
     34[07:49:56.738] 
     35[07:49:56.738] [#6](/bitcoin-bitcoin/6/) [1/4] FROM docker.io/library/ubuntu:24.04@sha256:80dd3c3b9c6cecb9f1667e9290b3bc61b78c2678c02cbdae5f0fea92cc6734ab
     36[07:49:56.738] [#6](/bitcoin-bitcoin/6/) resolve docker.io/library/ubuntu:24.04@sha256:80dd3c3b9c6cecb9f1667e9290b3bc61b78c2678c02cbdae5f0fea92cc6734ab
     37[07:49:57.299] [#6](/bitcoin-bitcoin/6/) resolve docker.io/library/ubuntu:24.04@sha256:80dd3c3b9c6cecb9f1667e9290b3bc61b78c2678c02cbdae5f0fea92cc6734ab 0.7s done
     38[07:49:57.408] [#6](/bitcoin-bitcoin/6/) DONE 0.7s
     39[07:49:57.408] 
     40[07:49:57.408] [#4](/bitcoin-bitcoin/4/) [internal] load build context
     41[07:49:57.408] [#4](/bitcoin-bitcoin/4/) transferring context: 12.88kB done
     42[07:49:57.408] [#4](/bitcoin-bitcoin/4/) DONE 0.0s
     43[07:49:57.408] 
     44[07:49:57.408] [#7](/bitcoin-bitcoin/7/) [2/4] COPY ./ci/retry/retry /usr/bin/retry
     45[07:49:57.408] [#7](/bitcoin-bitcoin/7/) CACHED
     46[07:49:57.408] 
     47[07:49:57.408] [#8](/bitcoin-bitcoin/8/) [3/4] COPY ./ci/test/00_setup_env.sh ././ci/test/00_setup_env_mac_cross.sh ./ci/test/01_base_install.sh /ci_container_base/ci/test/
     48[07:49:57.408] [#8](/bitcoin-bitcoin/8/) CACHED
     49[07:49:57.408] 
     50[07:49:57.408] [#9](/bitcoin-bitcoin/9/) [4/4] RUN ["bash", "-c", "cd /ci_container_base/ && set -o errexit && source ./ci/test/00_setup_env.sh && ./ci/test/01_base_install.sh"]
     51[07:49:57.408] [#9](/bitcoin-bitcoin/9/) sha256:febe602dc08a9a790bd092b613b049003ba7c75651d565cee664ecbebbe12bfc 1.85kB / 1.85kB 0.0s done
     52[07:49:57.408] [#9](/bitcoin-bitcoin/9/) extracting sha256:febe602dc08a9a790bd092b613b049003ba7c75651d565cee664ecbebbe12bfc done
     53[07:49:57.408] [#9](/bitcoin-bitcoin/9/) sha256:cfdd814cf5471383f503651cd4c6f1a1bcd432e92911dac06cff317cdbc6b305 3.38kB / 3.38kB 0.0s done
     54[07:49:57.581] [#9](/bitcoin-bitcoin/9/) extracting sha256:cfdd814cf5471383f503651cd4c6f1a1bcd432e92911dac06cff317cdbc6b305 0.0s done
     55[07:49:57.588] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 8.39MB / 470.35MB 0.2s
     56[07:49:58.038] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 38.80MB / 470.35MB 0.6s
     57[07:49:58.488] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 70.25MB / 470.35MB 1.1s
     58[07:49:58.938] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 102.76MB / 470.35MB 1.5s
     59[07:49:59.388] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 134.22MB / 470.35MB 2.0s
     60[07:49:59.839] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 164.63MB / 470.35MB 2.4s
     61[07:50:00.292] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 193.99MB / 470.35MB 2.9s
     62[07:50:00.738] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 223.35MB / 470.35MB 3.3s
     63[07:50:01.191] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 254.80MB / 470.35MB 3.8s
     64[07:50:01.638] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 286.26MB / 470.35MB 4.2s
     65[07:50:02.089] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 315.62MB / 470.35MB 4.7s
     66[07:50:02.538] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 346.03MB / 470.35MB 5.1s
     67[07:50:02.988] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 376.44MB / 470.35MB 5.6s
     68[07:50:03.438] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 406.85MB / 470.35MB 6.0s
     69[07:50:03.888] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 436.21MB / 470.35MB 6.5s
     70[07:50:04.338] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 468.71MB / 470.35MB 6.9s
     71[07:50:04.811] [#9](/bitcoin-bitcoin/9/) sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 470.35MB / 470.35MB 7.2s done
     72[07:50:04.811] [#9](/bitcoin-bitcoin/9/) extracting sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8
     73[07:50:30.213] [#9](/bitcoin-bitcoin/9/) extracting sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 25.6s done
     74[07:50:30.453] [#9](/bitcoin-bitcoin/9/) CACHED
     75[07:50:30.453] 
     76[07:50:30.453] [#10](/bitcoin-bitcoin/10/) exporting to image
     77[07:50:30.453] [#10](/bitcoin-bitcoin/10/) exporting layers done
     78[07:50:30.453] [#10](/bitcoin-bitcoin/10/) exporting manifest sha256:35b82073fb170cde1c1f340217048fa5fdca7bf710eb9836e1e75ae8f632c18a done
     79[07:50:30.453] [#10](/bitcoin-bitcoin/10/) exporting config sha256:040ba6df0bbe7839075a62de0a03179747c82ddff2a33a9a11d72202cd6ed3c7 done
     80[07:50:30.453] [#10](/bitcoin-bitcoin/10/) exporting attestation manifest sha256:b354c939c9e7630028a036a105ea52f56a74b91c54148db376b51477b6e47668 done
     81[07:50:30.453] [#10](/bitcoin-bitcoin/10/) exporting manifest list sha256:550407c300a249757f8f145ac481f361b7e7f22de8b62639006b7f2d202369d5 done
     82[07:50:30.453] [#10](/bitcoin-bitcoin/10/) naming to docker.io/library/ci_macos_cross:latest done
     83[07:50:30.453] [#10](/bitcoin-bitcoin/10/) unpacking to docker.io/library/ci_macos_cross:latest done
     84[07:50:30.453] [#10](/bitcoin-bitcoin/10/) DONE 0.1s
     85[07:50:30.453] 
     86[07:50:30.453] [#11](/bitcoin-bitcoin/11/) exporting cache to client directory
     87[07:50:30.453] [#11](/bitcoin-bitcoin/11/) preparing build cache for export
     88[07:50:30.453] [#11](/bitcoin-bitcoin/11/) writing layer sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8
     89[07:50:37.448] [#11](/bitcoin-bitcoin/11/) writing layer sha256:357d2c66fb559e3c323ff1fd5acd1db22cc7052af624ac52c48f0f34d1c1d0c8 7.1s done
     90[07:50:37.448] [#11](/bitcoin-bitcoin/11/) writing layer sha256:cfdd814cf5471383f503651cd4c6f1a1bcd432e92911dac06cff317cdbc6b305
     91[07:50:37.624] [#11](/bitcoin-bitcoin/11/) writing layer sha256:cfdd814cf5471383f503651cd4c6f1a1bcd432e92911dac06cff317cdbc6b305 0.0s done
     92[07:50:37.624] [#11](/bitcoin-bitcoin/11/) writing layer sha256:de44b265507ae44b212defcb50694d666f136b35c1090d9709068bc861bb2d64
     93[07:50:37.963] [#11](/bitcoin-bitcoin/11/) writing layer sha256:de44b265507ae44b212defcb50694d666f136b35c1090d9709068bc861bb2d64 0.5s done
     94[07:50:37.963] [#11](/bitcoin-bitcoin/11/) writing layer sha256:febe602dc08a9a790bd092b613b049003ba7c75651d565cee664ecbebbe12bfc
     95[07:50:38.044] [#11](/bitcoin-bitcoin/11/) preparing build cache for export 7.7s done
     96[07:50:38.044] [#11](/bitcoin-bitcoin/11/) writing layer sha256:febe602dc08a9a790bd092b613b049003ba7c75651d565cee664ecbebbe12bfc 0.0s done
     97[07:50:38.044] [#11](/bitcoin-bitcoin/11/) writing config sha256:16c21cc25bb3008714faa399494f419192dd76aa601d10f5c1216e218acc8d16 done
     98[07:50:38.044] [#11](/bitcoin-bitcoin/11/) writing cache manifest sha256:9e819d3419d1888e5ee2d5514592865fe2f9693029640942d852d7e3ae0e347f done
     99[07:50:38.044] [#11](/bitcoin-bitcoin/11/) DONE 7.7s
    100[07:50:38.044] 
    101[07:50:38.044]  
    1021 warning found (use docker --debug to expand):
    103[07:50:38.044] 
    104 - InvalidDefaultArgInFrom: Default value for ARG ${CI_IMAGE_NAME_TAG} results in empty or invalid base image name (line 8)
    105[07:51:01.292] + '[' /cache/docker/ci-imgs ']'
    106[07:51:01.292] + '[' -e /var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh/ci_macos_cross/index.json ']'
    107[07:51:01.292] + echo 'Removing the existing docker build cache in /cache/docker/ci-imgs/ci_macos_cross'
    108[07:51:01.292] Removing the existing docker build cache in /cache/docker/ci-imgs/ci_macos_cross
    109[07:51:01.292] + rm -rf /cache/docker/ci-imgs/ci_macos_cross
    110[07:51:01.321] + echo 'Moving the contents of /var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh/ci_macos_cross to /cache/docker/ci-imgs/ci_macos_cross'
    111[07:51:01.321] Moving the contents of /var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh/ci_macos_cross to /cache/docker/ci-imgs/ci_macos_cross
    112[07:51:01.321] + mv /var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh/ci_macos_cross /cache/docker/ci-imgs/ci_macos_cross
    113[07:51:02.272] + rm -rf /var/lib/cirrus-worker/tmp/tmp.RYI5XdZ1Eh
    

    It’s important to note that this doesn’t cache the base images (i.e. ubuntu:24.04, debian:bookworm, centos/amd:stream9, …). On an ephemeral runner these can be cached with docker load and docker save. Otherwise, you might run into docker-hub rate-limits or connectivity problems to the registries.

    It would be nice to have, in the medium term, pre-build docker images for the CI tasks that can be pulled (and cached) to avoid building them in CI at all (unless there’s a change in the underlying docker image of course). My understanding is, that this needs a bit of restructuring of the current apt package installation and similar. @willcl-ark showed me something in the past that looked promising.

  6. in ci/test/00_setup_env.sh:75 in 4149b00ae5
    67@@ -68,3 +68,8 @@ export CI_BASE_PACKAGES=${CI_BASE_PACKAGES:-build-essential pkg-config curl ca-c
    68 export GOAL=${GOAL:-install}
    69 export DIR_QA_ASSETS=${DIR_QA_ASSETS:-${BASE_SCRATCH_DIR}/qa-assets}
    70 export CI_RETRY_EXE=${CI_RETRY_EXE:-"retry --"}
    71+# The (optional) `docker build` cache directory.
    72+# If set, the folder exists only on the CI host and is used
    73+# as a build cache for the docker build of CI images. This
    74+# cache can be multiple GB in size.
    75+export DOCKER_BUILD_CACHE_HOST_DIR=${DOCKER_BUILD_CACHE_HOST_DIR:-}
    


    maflcko commented at 9:04 am on December 20, 2024:

    Is this required? It seems like a no-op, apart from the docs.

    Though, docker is already caching by default, and this is normally not needed. So I think it is fine to omit the docs, or inline them in 02_run_container.

    I am even thinking this should have the DANGER_ prefix, as it calls rm -rf on the host for the folder.


    0xB10C commented at 10:37 am on December 20, 2024:
    Good point. Will move docs to 02_run_container and prefix with DANGER_.
  7. 0xB10C commented at 10:37 am on December 20, 2024: contributor
    A consideration might also to only specify --cache-to when CIRRUS_BRANCH=master. This makes PR runs slightly faster as the cache isn’t overwritten. Also, most PRs don’t introduce changes that cause a cache-miss, and if they do we want to not cache it until it’s merged. Obviously, someone can fake this by overwriting CIRRUS_BRANCH to forcefully overwrite the cache for a while.
  8. in ci/test/02_run_container.sh:63 in 4149b00ae5
    58+      echo "Removing the existing docker build cache in ${DOCKER_BUILD_CACHE_OLD_DIR}"
    59+      rm -rf "${DOCKER_BUILD_CACHE_OLD_DIR}"
    60+      echo "Moving the contents of ${DOCKER_BUILD_CACHE_NEW_DIR} to ${DOCKER_BUILD_CACHE_OLD_DIR}"
    61+      mv "${DOCKER_BUILD_CACHE_NEW_DIR}" "${DOCKER_BUILD_CACHE_OLD_DIR}"
    62+    fi
    63+    rm -rf "${DOCKER_BUILD_CACHE_TEMPDIR}"
    


    maflcko commented at 11:09 am on December 20, 2024:
    This will be left dangling and cause out-of-storage, if the build fails?

    maflcko commented at 11:11 am on December 20, 2024:
    If this is fine, you could just drop the line (and leave it to the host to clean up). If not, I am not sure how to fix it.

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-12-21 12:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me