Backport Cirrus runners to 29.x

willcl-ark commented at 11:43 am on September 16, 2025: member

Backports #32989 to the 29.x branch

DrahtBot added the label Backport on Sep 16, 2025

DrahtBot commented at 11:43 am on September 16, 2025: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33403.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	m3dwards

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

LLM Linter (✨ experimental)

Possible typos and grammar issues:

Neither build, nor BUILDKIT=1 currently do this properly -> Neither build nor BUILDKIT=1 currently does this properly [subject–verb agreement: “Neither … nor …” takes a singular verb here]

drahtbot_id_5_m

fanquake added this to the milestone 29.2 on Sep 16, 2025

willcl-ark force-pushed on Sep 16, 2025

fanquake commented at 1:28 pm on September 16, 2025: member

Looking at the bottom of https://github.com/bitcoin/bitcoin/actions/runs/17765745116?pr=33403, I see

0 x86_64-w64-mingw32-executables-17765745116 ...  4.12 KB

Looking in the tarball, all the executables are 200 bytes of shell script:

0cat x86_64-w64-mingw32-executables-17765745116/bin/bench_bitcoin.exe         
1#!/usr/bin/env bash
2( wine "/home/admin/actions-runner/_work/_temp/build/bin/bench_bitcoin.exe_orig" "$@" ) || ( sleep 1 && wine "/home/admin/actions-runner/_work/_temp/build/bin/bench_bitcoin.exe_orig" "$@" )

willcl-ark force-pushed on Sep 16, 2025

willcl-ark commented at 1:56 pm on September 16, 2025: member

Looking at the bottom of bitcoin/bitcoin/actions/runs/17765745116?pr=33403, I see

0 x86_64-w64-mingw32-executables-17765745116 ...  4.12 KB

Looking in the tarball, all the executables are 200 bytes of shell script:

0cat x86_64-w64-mingw32-executables-17765745116/bin/bench_bitcoin.exe         
1#!/usr/bin/env bash
2( wine "/home/admin/actions-runner/_work/_temp/build/bin/bench_bitcoin.exe_orig" "$@" ) || ( sleep 1 && wine "/home/admin/actions-runner/_work/_temp/build/bin/bench_bitcoin.exe_orig" "$@" )

Thanks, this was a path issue with the new build directory structure. In this latest push da1d3dda83f7fdb68a887c0f81a9db566666b60e I have refactored this job into a matrix element. In the 29.x (and 28.x) branch(es) we are not testing the built binaries, so don’t need to upload them at all, and therefore don’t need a standalone job for this test.

The diff (excluding the typo in the first force-push) is https://github.com/bitcoin/bitcoin/compare/939e1e18ce152efb57e03bae2e7b8cf5a11d01f4..da1d3dda83f7fdb68a887c0f81a9db566666b60e

willcl-ark marked this as ready for review on Sep 16, 2025

willcl-ark renamed this:
~~WIP: Backport Cirrus runners to 29.x~~
Backport Cirrus runners to 29.x
on Sep 16, 2025

fanquake commented at 8:51 am on September 17, 2025: member

cc @m3dwards

willcl-ark force-pushed on Sep 19, 2025

fanquake commented at 2:13 pm on September 23, 2025: member

Can you also add #33425 here?

willcl-ark commented at 2:13 pm on September 23, 2025: member

Can you also add #33425 here?

Sure

ci: add configure environment action

Github-Pull: #32989
Rebased-From: b8fcc9fcbcd

f63b8e960d

ci: add caching actions

Github-Pull: #32989
Rebased-From: b232b0fa5e9

Add "Restore" and "Save" caching actions.

These actions reduce boilerplate in the main ci.yml configuration file.

These actions are implemented so that caches will be saved on `push`
only.

When a pull request is opened it will cache hit on the caches from the
lastest push, or in the case of depends will hit on any matching depends
hash, falling back to partial matches.

Depends caches are hashed using
`$(git ls-tree HEAD depends "ci/test/$FILE_ENV" | sha256sum | cut -d' ' -f1)`
and this hash is passed in as an input to the actions. This means we
direct cache hit in cases where depends would not be re-built, otherwise
falling back to a partial match.

Previous releases cache is hashed similarly to depends, but using the
test/get_previous_releases.py file.

The cirruslabs cache action will fallback transparently to GitHub's
cache in the case that the job is not being run on a Cirrus Runner,
making these compatible with running on forks (on free GH hardware).

301aa5d814

ci: add REPO_USE_CIRRUS_RUNNERS

Github-Pull: #32989
Rebased-From: 33ba073df7a

If set, Cirrus runners will be used on pushes to, and pull requests
against, this repository.

Forks can set this if they have their own cirrus runners.

954c1a55e4

ci: add configure-docker action

Github-Pull: #32989
Rebased-From: fdf64e55324

Another action to reduce boilerplate in the main ci.yml file.

This action will set up a docker builder compatible with caching build
layers to a container registry using the `gha` build driver.

It will then configure the docker build cache args.

1faf918a16

ci: use buildx in ci

Github-Pull: #32989
Rebased-From: 94a09325475

Using buildx is required to properly load the correct driver, for use
with registry caching. Neither build, nor BUILDKIT=1 currently do this
properly.

Use of `docker buildx build` is compatible with podman.

f3089fb2cf

ci: use docker build cache arg directly

Github-Pull: #32989
Rebased-From: 18f6be09d02

Reverts: e87429a2d0f23eb59526d335844fa5ff5b50b21f

This was added in PR #31545 with the intention that self-hosted runners
might use it to save build cache.

As we are not using hosted runners with a registry build cache, the bulk
of this commit can be reverted, simply using the value of
$DOCKER_BUILD_CACHE_ARG in the script.

link: https://github.com/bitcoin/bitcoin/pull/31545

0a649d07c9

ci: have base install run in right dir

Github-Pull: #32989
Rebased-From: 9c2b96e0d03

This sets the build dir at build time so that Apple SDK gets installed
in the correct/expected location for the runtime to find it.

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

af086431e8

ci: add Cirrus cache host

Github-Pull: #32989
Rebased-From: 020069e6b71

Whilst the action cirruslabs/actions/cache will automatically set this
host, the docker `gha` build cache backend will not be aware of it.

Set the value here, which will later be used in the docker build args to
enable docker build cache on the cirrus cache.

f9f3e8b686

ci: add job to determine runner type

Github-Pull: #32989
Rebased-From: cc1735d7771

To remove multiple occurances of the respository name, against which we
compare `${{ github.repository }}` to check if we should use Cirrus
Runners, introduce a helper job which can check a single environment
variable and output this as an input to subsequent jobs.

Forks can maintain a trivial patch of their repo name against the
`REPO_USE_CIRRUS_RUNNERS` variable in ci.yml if they have Cirrus Runners
of their own, which will then enable cache actions and docker build
cache to use Cirrus Cache.

It's not possible to use `${{ env.USE_CIRRUS_RUNNERS }}` in the
`runs-on:` directive as the context is not supported by GitHub.

If it was, this job would no longer be necessary.

849993377d

ci: port arm 32-bit job

Github-Pull: #32989
Rebased-From: f253031cb8e

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

82c60a3151

ci: update windows-cross job

Github-Pull: #32989
Rebased-From: 04e7bfbceb0

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

894a3cbe42

ci: update asan-lsan-ubsan

Github-Pull: #32989
Rebased-From: 884251441bb

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

819ee09af3

ci: force reinstall of kernel headers in asan

Github-Pull: #32989
Rebased-From: 2c990d84a3d

When using hosted runners in combination with cached docker images,
there is the possibility that the host runner image is updated,
rendering the linux-headers package (stored in the cached docker image)
incompatible.

Fix this by doing a re-install of the headers package in
03_test_script.sh.

If the underlying runner kernel has not changed thie has no effect, but
prevents the job from failing if it has.

a91567a980

ci: port mac-cross-gui-notests

Github-Pull: #32989
Rebased-From: 9c2514de534

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

835b5b8bb1

ci: port nowallet-libbitcoinkernel

Github-Pull: #32989
Rebased-From: 2a00b12d73b

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

e826c3daa5

ci: port i686-multiprocess-DEBUG

Github-Pull: #32989
Rebased-From: f2068f26c12

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

544f902b2a

ci: port fuzzer-address-undefined-integer-nodepends

Github-Pull: #32989
Rebased-From: 341196d75c3

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

85ec6c6882

ci: port previous-releases-depends-debug

Github-Pull: #32989
Rebased-From: 58e38c3a042

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

5057b9a6ff

ci: port centos-depends-gui

Github-Pull: #32989
Rebased-From: 549074bc643

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

b4286cf354

ci: port tidy

Github-Pull: #32989
Rebased-From: bf7d5364527

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

3b2dcc8b9a

ci: port tsan-depends

Github-Pull: #32989
Rebased-From: 9bbae61e3b4

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

643385b22d

ci: port msan-depends

Github-Pull: #32989
Rebased-From: d290a8e6eab

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

0f0378fe3c

ci: port lint

Github-Pull: #32989
Rebased-From: bc41848d00f

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

06424fb004

ci: remove .cirrus.yml

Github-Pull: #32989
Rebased-From: 4393ffdd837

Removed as unused.

a08c3cc51c

ci: dynamically match makejobs with cores

Github-Pull: #32989
Rebased-From: 3f339e99e00

Previously jobs were running on a large multi-core server where 10 jobs
as default made sense (or may even have been on the low side).

Using hosted runners with fixed (and lower) numbers of vCPUs we should
adapt compilation to match the number of cpus we have dynamically.

This is cross-platform compatible with macos and linux only.

c7f290b826

doc: Detail configuration of hosted CI runners

Github-Pull: #32989
Rebased-From: f4272844833dd660c2b9db587856baa408889302

4339787379

ci: add ccache hit-rate warning when < 75%

Github-Pull: #32989
Rebased-From: dd1c5903e8d

Print the ccache hit-rate for the job using a GitHub annotation if it
was below 75%.

773e4cda94

ci: fix annoying docker warning

Github-Pull: #32989
Rebased-From: 2aa288efdda

Docker currently warns that we are missing a default value.

Set this to scratch which will error if an appropriate image tag is not
passed in to silence the warning.

4e8b64b181

ci: remove un-needed lint_run*.sh files

Github-Pull: #32989
Rebased-From: 3c5da69a232

ci/lint_run_all.sh: Only used in .cirrus.yml. Refer to test/lint/README.md on how to run locally.

6ded1fe117

ci: reduce runner sizes on various jobs

Github-Pull: #33319
Rebased-From: 5eeb2facbbbbf68a2c30ef9e6747e39c85d7b116

These jobs can use reduced runner size to avoid wasting CPU, as much of
the long-running part of the job is single-threaded.

Suggested in: https://github.com/bitcoin/bitcoin/pull/32989#discussion_r2321775620

Co-authored-by: MarcoFalke <*~=`'#}+{/-|&$^_@721217.xyz>

4a034cbeb4

ci: Checkout latest merged pulls

Github-Pull: #33303
Rebased-From: fa8f081af31

78d93effd0

ci: link against -lstdc++ in native fuzz with msan job

Github-Pull: #33425
Rebased-From: b77137a5644e09a08442aed7d8a4a9290fb53526

5750355139

willcl-ark force-pushed on Sep 23, 2025

m3dwards commented at 12:39 pm on September 24, 2025: contributor

ACK 5750355139eb7fc2bd11124adf46bf053be6b690

So here are all the (innocent) changed that I’ve found compared with PR #32989

Windows job uses cmd rather than powershell
i686 job file is different in 29.x
Macos test script is different in 29.x
Adds decreasing runner sizes
Adds checking latest merged pulls
Adds ci: link against -lstdc++

Tools used to compare: Will’s backport fish function and the following git range-diff:

0git range-diff ba0b4304eceeec52eb3ffd2e7f8bb831ba1ec278^..ba0b4304eceeec52eb3ffd2e7f8bb831ba1ec278 f63b8e960d5d06cdbbc360aaf781c13fd5aca172^..5750355139eb7fc2bd11124adf46bf053be6b690

fanquake merged this on Sep 24, 2025

fanquake closed this on Sep 24, 2025

Backport Cirrus runners to 29.x #33403

Code Coverage & Benchmarks

Reviews

LLM Linter (✨ experimental)