ci: cache BSD sdk sources separately #35722

pull willcl-ark wants to merge 3 commits into bitcoin:master from willcl-ark:depends-sdk-sources changing 6 files +120 −69

willcl-ark commented at 1:38 PM on July 14, 2026: member

Sometimes CI jobs need to (re)download *BSD SDK archives, and these downloads can delay or fail the job, as has been happening recently.

The SDKs are currently downloaded and extracted while building the CI image itself; this means that when the relevant image-build cache is missed or invalidated, the SDK archives are downloaded again, which is unnecessary.

Instead, cache the BSD SDK archives separately and mount them into the runtime container. Move the BSD SDK setup out of 01_base_install.sh and run it after the container starts, so it is no longer part of the image build and the BSD SDKs are no longer part of the image build cache/layers.

This means rebuilding the BSD images no longer relies on pulling the SDK archives, reducing reliance and load we have on archive servers in many cases.

While touching 02_run_container.py and adding new paths to it, unify path handling for all caches a bit more.
DrahtBot commented at 1:39 PM on July 14, 2026: contributor


The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.



Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35722.



Reviews

See the guideline and AI policy for information on the review process. A summary of reviews will appear here.



Conflicts

Reviewers, this pull request conflicts with the following ones:
- #35754 (ci: update and verify SDKs, lock lint dependencies by l0rinc)
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.
willcl-ark force-pushed on Jul 14, 2026
DrahtBot added the label CI failed on Jul 14, 2026

ci: move BSD SDK setup out of image build

The base install script runs while constructing the CI image, but the BSD
SDKs are only needed later when compiling the source in the running
container.

Moving them out of base_install.sh means we can cache them similarly to
other sources/caches.

aa040360ce

ci: cache BSD SDK sources

Restore and save the NetBSD, FreeBSD, and OpenBSD SDK archives through the
existing cache provider.

Mount the shared sdk-sources directory in the container so cache hits
avoid downloading the archives again.

732997364b

ci: unify cache path handling

The cache directories need to be created and mounted in both host and Docker
modes. Keeping separate lists lets the paths drift.

Define each cache's source and destination once and derive the volume and bind
mounts from that definition.

cd4b34c58d

willcl-ark force-pushed on Jul 14, 2026
willcl-ark renamed this:
~~[NOMERGE] ci: cache BSD sdk sources separately~~
ci: cache BSD sdk sources separately
on Jul 14, 2026
willcl-ark marked this as ready for review on Jul 14, 2026
DrahtBot removed the label CI failed on Jul 14, 2026
DrahtBot added the label Tests on Jul 14, 2026
DrahtBot added the label Needs rebase on Aug 4, 2026
DrahtBot commented at 9:13 AM on August 4, 2026: contributor



🐙 This pull request conflicts with the target branch and needs rebase.
maflcko commented at 10:00 AM on August 4, 2026: member

An alternative would be to add fallback mirrors (like in depends), but this may be fine as well (haven't looked closely)
willcl-ark commented at 11:12 AM on August 4, 2026: member

Yeah I'm open to either option too. The nice thing about caching them in CI is that nobody has to update the fallback mirror on each SDK update (or forget to!). But IMO it's the "cleaner" approach, and the fallbacks also work for contributors local development; could be superior depending on how flakey the original hosts are...

I think it might be @achow101 who could add fallbacks of these, if we want to go that route?
maflcko commented at 11:18 AM on August 4, 2026: member

Do you recall any CI logs where the download failed?
willcl-ark commented at 11:19 AM on August 4, 2026: member

Do you recall any CI logs where the download failed?

I do not, it was just the re-downloading them needlessly which prompted this change.

Contributors

Labels

ci: cache BSD sdk sources separately #35722

Code Coverage & Benchmarks

Reviews

Conflicts