ci: Cache package manager install step

maflcko commented at 11:39 am on January 27, 2023: member

Use the local podman or docker image cache to skip the slow apt step

DrahtBot commented at 11:39 am on January 27, 2023: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	jamesob

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

DrahtBot added the label Tests on Jan 27, 2023

maflcko force-pushed on Jan 27, 2023

maflcko commented at 2:44 pm on January 27, 2023: member

This isn’t used by Cirrus, because I want to test/settle this a bit more.

However, by the one task that runs on my infra, it gives a ~1min speedup: https://cirrus-ci.com/task/5552889800687616

Also, locally it seems to work:

 0podman image ls --all | grep localhost
 1localhost/ci_native_fuzz_valgrind              latest        5867fa661f2d  About an hour ago  1.08 GB
 2localhost/ci_native_tidy                       latest        2196c8dd7b51  2 hours ago        2.03 GB
 3localhost/ci_native_fuzz_msan                  latest        01047aa4c1ba  7 hours ago        818 MB
 4localhost/ci_win64                             latest        a21797b76dc0  7 hours ago        2.11 GB
 5localhost/ci_native_msan                       latest        621b2eb71e9c  8 hours ago        818 MB
 6localhost/ci_native_qt5                        latest        e8b0920ad207  9 hours ago        910 MB
 7localhost/ci_native_nowallet_libbitcoinkernel  latest        2af70303446f  10 hours ago       693 MB
 8localhost/ci_native_fuzz                       latest        31668cc3f135  12 hours ago       989 MB
 9localhost/ci_macos_cross                       latest        8bf17e6b9640  12 hours ago       526 MB
10localhost/ci_i686_centos                       latest        f200c770cd17  13 hours ago       913 MB
11localhost/ci_arm_linux                         latest        e8f8098d02d7  14 hours ago       1.34 GB
12localhost/ci_native_tsan                       latest        d85e2a3cecb5  15 hours ago       843 MB
13localhost/ci_i686_multiprocess                 latest        18f976e245b7  16 hours ago       881 MB
14localhost/ci_s390x                             latest        008df12fecb7  17 hours ago       939 MB

ci: Cache package manager install step fa486de212

maflcko force-pushed on Jan 28, 2023

in ci/test_imagefile:10 in fa486de212

 5+ENV FILE_ENV=${FILE_ENV}
 6+
 7+COPY ./ci/retry/retry /usr/bin/retry
 8+COPY ./ci/test/00_setup_env.sh ./${FILE_ENV} ./ci/test/01_base_install.sh /ci_base_install/ci/test/
 9+
10+RUN ["bash", "-c", "cd /ci_base_install/ && set -o errexit && source ./ci/test/00_setup_env.sh && ./ci/test/01_base_install.sh"]

jamesob commented at 2:31 pm on January 31, 2023:

To get even more of a speedup, wouldn’t we want to move these steps into RUN commands earlier in the Dockerfile, and then just generate a few separate images on the basis of CI_IMAGE_NAME_TAG/FILE_ENV?

maflcko commented at 2:41 pm on January 31, 2023:

The packages are defined in FILE_ENV, so RUN needs to be after FILE_ENV

jamesob commented at 2:49 pm on January 31, 2023:

I’m sorry, shouldn’t be reading this before the caffeine has hit; I misread RUN here for ENTRYPOINT.

jamesob commented at 2:31 pm on January 31, 2023: member

Concept ACK

jamesob commented at 2:37 pm on January 31, 2023: member

(For what it’s worth, I’m an ACK as-is because this PR represents a strict improvement over what we have now.)

in ci/test/04_install.sh:41 in fa486de212

37+  DOCKER_BUILDKIT=1 ${CI_RETRY_EXE} docker build \
38+      --file "${BASE_ROOT_DIR}/ci/test_imagefile" \
39+      --build-arg "CI_IMAGE_NAME_TAG=${CI_IMAGE_NAME_TAG}" \
40+      --build-arg "FILE_ENV=${FILE_ENV}" \
41+      --tag="${CONTAINER_NAME}" \
42+      "${BASE_ROOT_DIR}"

jamesob commented at 2:55 pm on January 31, 2023:

Do we have good confidence that in Cirrus environments, local build caches will persist? If not, we’ll hit slowdowns because pulling from registries is about as fast as it gets.

maflcko commented at 2:59 pm on January 31, 2023:

Based on the docs, probably yes. Though, this can be investigated in the follow-up that actually switches over Cirrus. For now I’ve decided to postpone it to have this settle first.

jamesob approved

jamesob commented at 3:03 pm on January 31, 2023: member

ACK fa486de212108b4609d7c247d2a578f0b4df9703 (jamesob/ackr/26976.1.MarcoFalke.ci_cache_package_manager)

Ran ./ci/test_run_all.sh, container built successfully. DOCKER_BUILDKIT=1 is cool.

fanquake commented at 2:55 pm on February 2, 2023: member

Testing this locally, and it seems to be working as expected. Building on the first run takes ~70s to build the container:

0Creating ubuntu:22.04 container to run in
1[+] Building 73.3s (9/9) FINISHED                                                                                                                                                                                   
2 => [internal] load build definition from test_imagefile

Subsequent runs take ~0s:

0Creating ubuntu:22.04 container to run in
1[+] Building 0.1s (9/9) FINISHED                                                                                                                                                                                    
2 => [internal] load build definition from test_imagefile

maflcko merged this on Feb 2, 2023

maflcko closed this on Feb 2, 2023

maflcko deleted the branch on Feb 2, 2023

sidhujag referenced this in commit 4bc64b9061 on Feb 3, 2023

bitcoin locked this on Feb 2, 2024

ci: Cache package manager install step #26976

Reviews