ci: pin and verify external inputs #35754

l0rinc commented at 3:30 AM on July 20, 2026: contributor

Problem: CI fetches SDK archives, tool binaries, Python packages, Git sources, OCI images, and GitHub Actions from mutable references or without checking their content. These inputs can change between runs without a repository change.

Fix: Verify archives and binaries against committed hashes, clone Git sources at reviewed commits, install Python packages from locked or fully hashed requirements, and pin OCI images and GitHub Actions to multi-architecture index digests and full commit SHAs. These references and lock files require manual updates. APT, APK, Homebrew, hosted runner images, and LLVM PPA packages remain trusted; pinning the PPA key would not constrain published packages.

<details><summary>Verification</summary>

Container tags remain beside their added digests for direct comparison in the diff, and Python package hashes can be regenerated locally. The remaining pins can be checked with:

# SDK archives
grep 9600fa93644df674ee916b5e2c8a6ba8dacf631996a65dc922d003b98b5ea3b1 contrib/macdeploy/README.md
curl -fsSL https://download.freebsd.org/releases/amd64/amd64/15.1-RELEASE/MANIFEST | grep 3768988b151c20f965679062b065c63a977d6bbb9f47fd83695ec2c40790c18f
curl -fsSL https://cdn.netbsd.org/pub/NetBSD/NetBSD-11.0_RC6/amd64/binary/sets/SHA512 | grep 0323dbb284913cae27f1e67a41c395dc6c7a4160dc65e79866ca8a69e12bad5fd782cc3f5da81a9d718020a857ccbc764e007f6b2696d88d7b9df83baa86f45b
curl -fsSL https://cdn.netbsd.org/pub/NetBSD/NetBSD-11.0_RC6/amd64/binary/sets/SHA512 | grep a33f9cb6886713a2cc9d3e7bfbecbcb77b82d86cd43fa688531e06b389ad1820f1f63e501a37f899aee700270618b0865aacbf569b65a91fdb77ddd48f23d940
curl -fsSL https://cdn.openbsd.org/pub/OpenBSD/7.9/amd64/SHA256 | grep 923d2e03f06408d50d4848334398c6d04b5514dcac7917badfc178a0eef248de
curl -fsSL https://cdn.openbsd.org/pub/OpenBSD/7.9/amd64/SHA256 | grep 21a67af20aebcabf85b09f4206fc95b4cae0a35d42b154b976f0159f457724f9

# Git sources
git ls-remote https://github.com/llvm/llvm-project 'refs/tags/llvmorg-22.1.7^{}' | grep a255c1ed36a1d06f79bd2633ba9f8d900153007c
git ls-remote https://github.com/riscv-collab/riscv-gnu-toolchain refs/tags/2026.06.06 | grep 81bb1f89664aad156df3d2773195177c92dedc3a
git ls-remote https://github.com/include-what-you-use/include-what-you-use refs/heads/clang_22 | grep 01a091d16b3dedb808db21f32ed3e761737a3691

# Lint binaries
curl -fsSL https://github.com/koalaman/shellcheck/releases/download/v0.11.0/shellcheck-v0.11.0.linux.x86_64.tar.xz | sha256sum | grep 8c3be12b05d5c177a04c29e3c78ce89ac86f1595681cab149b65b97c4e227198
curl -fsSL https://github.com/koalaman/shellcheck/releases/download/v0.11.0/shellcheck-v0.11.0.linux.aarch64.tar.xz | sha256sum | grep 12b331c1d2db6b9eb13cfca64306b1b157a86eb69db83023e261eaa7e7c14588
curl -fsSL https://github.com/becheran/mlc/releases/download/v1.2.0/mlc-x86_64-linux | sha256sum | grep 7a72a93d5b3ee8a554cb840abdfe90aefb709418f225461b52021e3a058238a2
curl -fsSL https://github.com/becheran/mlc/releases/download/v1.2.0/mlc-aarch64-linux | sha256sum | grep 01ec8e086f3b625616d461b63451be9175a02557de6b591bac7cde6791ab074b

# GitHub Actions
git ls-remote https://github.com/actions/checkout refs/tags/v6 | grep d23441a48e516b6c34aea4fa41551a30e30af803
git ls-remote https://github.com/actions/cache refs/tags/v5 | grep caa296126883cff596d87d8935842f9db880ef25
git ls-remote https://github.com/actions/upload-artifact refs/tags/v7 | grep 043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
git ls-remote https://github.com/actions/download-artifact refs/tags/v8 | grep 3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c
git ls-remote https://github.com/actions/github-script refs/tags/v8 | grep ed597411d8f924073f98dfc5c65a23a2325f34cd
git ls-remote https://github.com/docker/setup-buildx-action refs/tags/v4 | grep bb05f3f5519dd87d3ba754cc423b652a5edd6d2c
git ls-remote https://github.com/WarpBuilds/cache refs/tags/v1 | grep 6ab98e7fdd65255995acec1de229a0bf2a40fe37

</details>

DrahtBot added the label Tests on Jul 20, 2026

DrahtBot commented at 3:30 AM on July 20, 2026: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35754.

Reviews

See the guideline and AI policy for information on the review process. A summary of reviews will appear here.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#35722 (ci: cache BSD sdk sources separately by willcl-ark)
#31349 (ci: detect outbound internet traffic generated while running tests by vasild)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

in ci/lint/01_install.sh:35 in 8e8572e3dd

  30 | @@ -31,13 +31,22 @@ python3 --version
  31 |  
  32 |  uv pip install --python /python_env --requirements /ci/lint/requirements.txt
  33 |  
  34 | +case "$(uname --machine)" in
  35 | +  x86_64)  SHELLCHECK_SHA256="8c3be12b05d5c177a04c29e3c78ce89ac86f1595681cab149b65b97c4e227198" MLC_SHA256="7a72a93d5b3ee8a554cb840abdfe90aefb709418f225461b52021e3a058238a2" ;;

maflcko commented at 5:19 AM on July 20, 2026:

The pip deps aren't pinned, so I wonder why those two should be pinned?

l0rinc commented at 5:59 AM on July 20, 2026:

I'm not familiar with how to pin the pip requirements but I'll investigate.

willcl-ark commented at 10:18 AM on July 20, 2026:

I can provide a patch to pin the dependencies using uv.lock, which would be the recommended approach to take when using uv.

willcl-ark commented at 10:58 AM on July 20, 2026:

This commit shows how to use uv.lock to lock pip deps, feel free to cherry-pick here, or I can open seperately:

https://github.com/willcl-ark/bitcoin/commit/9b86a2969118693a5511042ef524f424c7c0a0ee

by using a pyproject "group", we can also make this ~ natively compatible with pip >= 25.1, as this knows how to install "groups" of dependencies outside of having a main application to install, as we are configured.

in ci/test/01_base_install.sh:33 in feb5270399

  29 | @@ -30,6 +30,7 @@ if [ -n "${APT_LLVM_V}" ]; then
  30 |    ${CI_RETRY_EXE} apt-get update
  31 |    ${CI_RETRY_EXE} apt-get install curl -y
  32 |    curl "https://apt.llvm.org/llvm-snapshot.gpg.key" | tee "/etc/apt/trusted.gpg.d/apt.llvm.org.asc"
  33 | +  sha256sum -c <<<"8b2a587ffd672c4687e7581dad4b2f6c1bb2ad6b480cd9771ba2ff48e0b8c75d /etc/apt/trusted.gpg.d/apt.llvm.org.asc"

maflcko commented at 5:20 AM on July 20, 2026:

Is this meant to be const? The CI will break the next time a signature is added or the expiry is extended?

l0rinc commented at 6:00 AM on July 20, 2026:

You suggest dropping it? How often do these change? Shouldn't we know about it?

maflcko commented at 6:18 AM on July 20, 2026:

Well, what is the goal here? If someone controlled this key, they could push arbitrarily different llvm code to the ppa, even when the key is pinned to a hash.

Sjors commented at 9:06 AM on July 20, 2026:

We also don't know how that key is managed, so if llvm.org is compromised, we don't know if the key is still safe. And if we trust the domain, we should just trust whatever key it gives us.

l0rinc commented at 5:19 PM on July 20, 2026:

My goal was to know when any of these change so that we at least have the possibility to review and approve them. With LLMs we can even do a quick check to see if we agree and update the hash if we do (sure, that's also not 100% safe, but a lot safer than what we have now). What I don't like is when these things just suddenly change between runs like #35202

And if we trust the domain, we should just trust whatever key it gives us

Maybe, but I still think it should be a deliberate choice where we have the option to intervene. We should decide when the CI state should change, not outsiders.

maflcko commented at 5:36 AM on July 21, 2026:

What I don't like is when these things just suddenly change between runs like #35202

The CI generally assumes that any container engine should work. Sure, it is annoying when a failure comes when the underlying engine changes, but I don't think this can be avoided: It is up to the host system (e.g. GHA) to provide the container engine and I don't think we can or want to pin it.

If we really wanted to pin the dependencies in the CI, we'd probably have to use nix and compile clang from LLVM ourselves?

l0rinc commented at 5:41 AM on July 21, 2026:

we'd probably have to use nix and compile clang from LLVM ourselves?

I accepted your reasoning here and removed the unpinnable dependency.

But the rest provide explicit pinning capability, so I think we should harden them. As you can see it's quite cheap and can avoid some surprises.

Sjors commented at 9:10 AM on July 20, 2026: member

@l0rinc can you elaborate in the PR description what you're actually checking and trusting?

I'm skeptical this is going to very useful if we simply set these hashes based on the current value and blindly update them every time CI breaks due to a mismatch. In the case of #35754 (review) the trust assumption seems to be circular. But there might be a subset of checks that are useful.

in ci/test/01_base_install.sh:119 in 49553283a2

 115 | @@ -116,6 +116,7 @@ if [ -n "$XCODE_VERSION" ] && [ ! -d "${DEPENDS_DIR}/SDKs/${OSX_SDK_BASENAME}" ]
 116 |    if [ ! -f "$OSX_SDK_PATH" ]; then
 117 |      ${CI_RETRY_EXE} curl --location --fail "${SDK_URL}/${OSX_SDK_FILENAME}" -o "$OSX_SDK_PATH"
 118 |    fi
 119 | +  sha256sum -c <<<"9600fa93644df674ee916b5e2c8a6ba8dacf631996a65dc922d003b98b5ea3b1 ${OSX_SDK_PATH}"

Sjors commented at 9:13 AM on July 20, 2026:

49553283a25c5ddb7ae778ff44a567ce40d06275: this matches contrib/macdeploy/README.md so it makes sense to check and should be easy to keep in sync.

ci: verify cross-build SDK archives

The macOS and BSD cross-build jobs extract SDK archives fetched at runtime without checking their content.
Keep each expected digest beside the corresponding SDK version and verify every archive before extraction.
Create the OpenBSD library symlinks once after both archives are extracted, when all link targets are present.

3751aa3028

ci: pin Git sources used to build tools

Sanitizer jobs build `libc++` from an LLVM tag, the IWYU job builds from a branch, and the bare-metal job builds the RISC-V toolchain from a tag.
Clone each source at the commit selected by that reference so later tag or branch changes cannot alter CI.
Keep the IWYU commit beside `IWYU_LLVM_V` so version bumps expose both values in the same job configuration.

b4efe448a9

lint: lock Python dependencies with uv

`uv.lock` hash-locks Python dependencies.

Running `uv sync --project /ci/lint --locked --only-group lint` in the container will fail if `pyproject.toml` does not match `uv.lock`.

`.python-version` sets the Python version canonically, and `uv sync --locked` will fail if it does not satisfy `requires-python` in `pyproject.toml`.

Pin the `uv` and `ruff` source images to reviewed multi-architecture OCI index digests while retaining their release tags.

8ba28acb87

ci: verify downloaded lint tool binaries

The lint image executes architecture-specific ShellCheck and MLC release binaries without checking their content.
Download each artifact to disk, verify it with `sha256sum`, and reject architectures without a reviewed digest.

5671b32614

in ci/test/01_base_install.sh:140 in 49553283a2

 136 | @@ -136,6 +137,7 @@ if [ -n "$FREEBSD_VERSION" ] && [ ! -d "${DEPENDS_DIR}/SDKs/${FREEBSD_SDK_BASENA
 137 |    if [ ! -f "$FREEBSD_SDK_PATH" ]; then
 138 |      ${CI_RETRY_EXE} curl --location --fail "https://download.freebsd.org/releases/amd64/${FREEBSD_VERSION}-RELEASE/base.txz" -o "$FREEBSD_SDK_PATH"
 139 |    fi
 140 | +  sha256sum -c <<<"3768988b151c20f965679062b065c63a977d6bbb9f47fd83695ec2c40790c18f ${FREEBSD_SDK_PATH}"

Sjors commented at 9:15 AM on July 20, 2026:

4955328: I would either:

download the signed hashes file, pgp verify against a hardcoded key, and then check the file hash; or
at least put this FREEBSD_SDK_HASH instead of in the middle of the CI file

l0rinc renamed this:
~~ci: verify downloaded CI dependencies~~
ci: pin and verify external inputs
on Jul 20, 2026

l0rinc force-pushed on Jul 20, 2026

l0rinc commented at 11:14 PM on July 20, 2026: contributor

can you elaborate in the PR description what you're actually checking and trusting?

Updated the description to distinguish the pinned inputs from the package repositories, LLVM PPA, and hosted runner images that CI still trusts.

And if we trust the domain, we should just trust whatever key it gives us.

Dropped the LLVM signing-key hash.

at least put this FREEBSD_SDK_HASH instead of in the middle of the CI file

Moved each SDK digest beside its version, extended verification to NetBSD and OpenBSD, and moved the OpenBSD symlink creation after both archives are extracted, when all link targets are present.

The pip deps aren't pinned

I can provide a patch to pin the dependencies using uv.lock, which would be the recommended approach to take when using uv.

Cherry-picked Will's uv lock commit and added index digests to its existing uv and ruff image tags.

<details><summary>Will's commit compared with the local version</summary>

git range-diff 9b86a2969118693a5511042ef524f424c7c0a0ee^! 8ba28acb872a8c7e880924a03006ac69067e74a5^!

</details>

Also added hash-checked requirements for pycapnp, pyzmq, and pycapnp's transitive dependencies, then copied them into test images so PIP_PACKAGES can read them during image construction. Kept the IWYU commit hash beside its LLVM version so version bumps expose both values together. Added hashes for the remaining container images in one commit and pinned GitHub Actions to full commit SHAs in another, using YAML anchors for repeated references.

DrahtBot added the label CI failed on Jul 21, 2026

ci: lock test Python dependencies

CI jobs install `pycapnp` and `pyzmq` from PyPI without pinning their versions or checking downloaded artifacts.
Pin the direct and transitive packages, record every artifact hash published for those releases, and enable pip's `--require-hashes` mode on Linux, macOS, and Windows.
Copy the requirements into test images so `PIP_PACKAGES` can read them during image construction.

9bf68d70bc

ci: pin container images

Test and lint jobs build their containers from mutable distribution image tags.
Retain the tags for update visibility and bind each image to a reviewed multi-architecture OCI index digest.

0f0eb35c8b

ci: pin GitHub Actions

Top-level workflows execute external Actions through mutable major-version tags.
Replace each tag with its reviewed full commit SHA and retain the tag in a comment for update visibility.
Define repeated action references once with YAML anchors so every use resolves to the same commit.

49cc4e8cab

l0rinc force-pushed on Jul 21, 2026

in ci/test/00_setup_env_freebsd_cross.sh:10 in 49cc4e8cab

   6 | @@ -7,11 +7,12 @@
   7 |  export LC_ALL=C.UTF-8
   8 |  
   9 |  export CONTAINER_NAME=ci_freebsd_cross
  10 | -export CI_IMAGE_NAME_TAG="mirror.gcr.io/ubuntu:26.04"
  11 | +export CI_IMAGE_NAME_TAG="mirror.gcr.io/ubuntu:26.04@sha256:3131b4cc82a783df6c9df078f86e01819a13594b865c2cad47bd1bca2b7063bb"

maflcko commented at 5:39 AM on July 21, 2026:

I don't see the point here. Sure, the container image is pinned, but this is Ubuntu, so the apt install will just pick the latest version anyway, in which case this pin here is just a verbose no-op.

l0rinc commented at 5:44 AM on July 21, 2026:

I remember having a surprise at another company when we found out Docker image tags weren't deterministic. The point of this PR is to minimize surprises - sure, we can follow up with pinning apt install if you want, but first let's make sure the OS is what we think it is.

maflcko commented at 6:48 AM on July 21, 2026:

Yes, it may be surprising if unknown, but I don't think this change is reducing the surprise. In fact, it may be increasing it, because the line seems to imply this is somehow pinned to a deterministic hash, but the outcome will still be non-deterministic.

Also, I don't think it is possible to pin apt install, or at least, if we'd try, it would be easier to just use nix.

Finally, apart from being a verbose no-op, and possibly increasing surprises, this is also tedious to maintain.

I suspect this will attract a flood of dependabot-style PRs to bump the hash to "fix CVEs", so before this is done, we should have a crew of reviewers to sign up to review those. (I won't be signing up)

in .github/actions/cache/restore/internal/action.yml:30 in 49cc4e8cab

  26 | @@ -27,7 +27,7 @@ runs:
  27 |      - name: Restore cache with WarpBuild
  28 |        id: warp
  29 |        if: ${{ inputs.provider == 'warp' }}
  30 | -      uses: WarpBuilds/cache/restore@v1
  31 | +      uses: WarpBuilds/cache/restore@6ab98e7fdd65255995acec1de229a0bf2a40fe37 # v1

maflcko commented at 5:44 AM on July 21, 2026:

Same here: Adding hashes seems mostly like a corporate check-list item. Since, the hash doesn't cover transitive actions, nor does it cover the warpbuild infra. If the cache provider was compromised, they could still ship wrong stuff via transitive actions or by corrupting the cache or the cache logic in their infra, even with a pinned hash here.

No objection, just leaving a comment.

l0rinc commented at 5:46 AM on July 21, 2026:

This isn't about correctness necessarily, but about it covertly changing without our knowledge.

DrahtBot removed the label CI failed on Jul 21, 2026

maflcko commented at 6:48 AM on July 21, 2026: member

It is good to have a large pull to cover all topics here and explain what is changed and what is not changed.

It seems there are some good changes, like checking the hash after a raw curl. Sure, we try to set curl --fail to catch I/O errors, but those won't catch corrupt server hardware. So adding a hash to the various SDKs could make sense.

However, for other stuff, it is unclear what the goal is, or rather the stated goal is not achieved and will also lead to other problems. (https://github.com/bitcoin/bitcoin/pull/35754#discussion_r3619714965).