ci: Migrate CI to hosted Cirrus Runners

willcl-ark commented at 8:39 am on July 16, 2025: member

This changeset migrates all current self-hosted CI jobs over to hosted Cirrus Runners.

These runners cost a flat rate of $150/month, and we qualify for an open source discount of 50%. Therefore they are $75/month/runner.

One “runner” should more accurately be thought of in terms of the number of vCPU you are purchasing: https://cirrus-runners.app/pricing/ or in terms of “concurrency”, where 1 runners gets you 1.0 concurrency. e.g. a Linux x86 Runner gets you 16 vCPU (1.0 concurrency) and 64GB RAM to be provisioned as you choose, amongst one or more jobs.

Cirrus Runners currently only support Linux (x86 and Arm64) and MacOS (Arm64). This changeset does not move the existing Github Actions native MacOS runners away from being run on Github’s infrastructure. This could be a follow up optimisation.

Runs from this changeset using Cirrus Runners can be found at: https://github.com/testing-cirrus-runners/bitcoin2/actions which shows an uncached run on master (CI#1), an outside pull request (CI#3) and an updated push to master (CI#4).

These workflows were run on 10 runners, and we would recommend purchasing a similar number for our CI in this repo to achieve the speed and concurrency we expect.

We include some optional performance commits, but these could be split out and made into followups or dropped entirely.

Benefits

Maintenance

As we are not self-hosting, nobody needs to maintain servers, disks etc.

Bus factor

Currently we have a very small number of people with the know-how working on server setup and maintenance. This setup fixes that so that “anyone” familiar with GitHub-style CI systems can work on it.

Scaling

These do not “auto-scale”/have “unlimited concurrency” like some solutions, but if we want more workers/cpu to increase parallism or increase the runner size of certain jobs for a speed-up we can simply buy more concurrency using the web interface.

Speed

Runtimes aproximate current runtimes pretty well, with some jobs being faster. Caching improvements on pull request (re-runs) are left as future optimisations from the current changeset (see below).

GitHub workflow syntax

With a migration to the more-commonly-used GitHub workflow syntax, migration to other providers in the future is often as simple as a one-line change (and installing a new GitHub app to the repo).

If we decide to self-host again, then we can also self-host GitHub runners (using https://github.com/actions/runner) and maintain new GH-style CI syntax.

Reporting

GitHub workflows provide nicer built-in reporting directly on the “Checks” page of a pr. This includes more-detailed action reporting, and a host of pretty nice integrated features, such as Workflow Commands for creating annotations that can print messages during runs. See for example at the bottom of this window where we report ccache hitrate, if it was below 90%: https://github.com/testing-cirrus-runners/bitcoin/actions/runs/16163449125?pr=1

These could be added conditionally into our CI scripts to report interesting or other information.

Costs

Financial

Relative to competitors Cirrus runners are cheap for the hosted CI-world. However these are likely more expensive than our current setup, or a well-configured (new) self-hosted setup.

If we started with 10 runners to be shared amongst all migrated jobs, this would total $750/mo = $9000/yr.

Note that we are not trying to comptete here on cost directly.

Dependencies

We would be dependent on Cirrus infra.

Forks

Forks should be able to run CI without paid Cirrus runners. This behaviour is achieved through a rather verbose runs-on: directive.
- This directive hardcodes the main repo (unfortunately you cannot use the env github context in this field in particular, for some reason).
- This directive also allows for a fork to patch the runs-on: field in the ci.yml file if they want to use Cirrus Runners too.
- The workflow otherwise will fallback to the GitHub free runners on forks.
This cirrus cache action transparently falls back to github actions cache when not running on cirrus, so forks will get some free github caching (10GB per repo).

All jobs work on forks, but will run (slowly) on GitHub native free hosted runners, instead of Cirrus runners. They will also suffer from poor cache hit-rates, but there’s nothing that can be done about that, and the situtation is an improvement on today.

Migration process

The main org should also, in addition to pulling code changes:

Permit the actions docker/setup-buildx-action@v3 and docker/login-action@v3 to be run in this repo.

Caching

For the number of CI jobs we have, cache usage on GitHub would be an issue as GH only provides 10GB of cache space, per repo. However cirrus provides 10 GB per runner, which scales better with the number of runners.

The cirruslabs/action/[restore|save] action we use here redirects this to Cirrus’ own cache and is both faster and larger.

In the case that user is running CI on a fork, the cirrus cache falls back transparently to GitHub default cache without error.

ccache, depends-sources, built-depends

Cached as blobs via cirruslabs/actions/cache action.
Current implementation:
- On push: restores and saves caches.
- On pull_request: restores but does not save caches.

This means a new pull request should hit a pretty relevant cache. Old pull requests which are not being rebased on master may suffer from lower cache hit-rate.

If we save caches on all pull request runs we run the risk of evicting recent (and more relevant) cache blobs. It may be possible in a future optimisation to widen this to save on pull request runs too, but it will also depend on how many runners we provision and what cache churn rates are like in the main repo.

Docker build layer caching

Cached using the gha cache backend
These cache blobs compete for space with ccache, depends-sources and depends-built caches
gha cache allows --cache-from to be used from pull requests, which does not work using a registry cache type (technically we could use a public read-only token to get this working, but that feels wrong)

This backend does network i/o and so are marginally slower than our current disk i/o cache.

But what about… `x`?

We have tested many other providers, including Runs-on, Buildjet, WarpBuild, and GitHub hosted runners (and investigated even more). But they all fall short in one-way or another.

Runs-On and Buildjet (and others) require installing GH apps with much too-liberal permissions (e.g. Administration: Read|Write) for our use-case.
GitHub hosted runners suffer from all of high costs, lower speed, small cache, and the requirement for a GitHub Teams subscription.
WarpBuild seems to be simply too expensive.

TODO:

To complete migration from self-hosted to hosted for this repo, the backport branches 27.x, 28.x and 29.x would also need their CI ported, but these are left for followups to this change (and pending review/changes here first).

Work and experimentation undertaken with m3dwards

DrahtBot commented at 8:39 am on July 16, 2025: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/32989.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	janb84, maflcko, m3dwards, achow101
Concept ACK	fanquake, hebasto, 0xB10C, stickies-v

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#33145 (CI: silent merge check by m3dwards)
#33051 (Don’t fix Python patch version by Sjors)
#31425 (RFC: Riscv bare metal CI job by TheCharlatan)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

in ci/test_imagefile:7 in c0ad2b6aa8 outdated

3@@ -4,12 +4,16 @@
4 
5 # See ci/README.md for usage.
6 
7-ARG CI_IMAGE_NAME_TAG
8+# We never want scratch, but default arg silences a Warning

fanquake commented at 9:39 am on July 16, 2025:

Can this be specific? I think either document the warning, or drop this comment, or just drop the workaround, if it’s only a warning.

willcl-ark commented at 10:24 am on July 16, 2025:

It is only a warning.

It’s not present current as we are using podman, but docker does this build check: https://docs.docker.com/reference/build-checks/invalid-default-arg-in-from/

If we are happy with a warning on every docker buildx build ... invocation we can drop it, but this hack silences the warning and will always fail the build if scratch ends up being used (and should be pretty easy to diagnose the failure).

Preference between adding a link to the specific check for more context, or should we drop it and live with the warning?

maflcko commented at 7:38 am on August 21, 2025:

Can be resolved?

in ci/test/02_run_container.sh:29 in c0ad2b6aa8 outdated

45-    DOCKER_BUILD_CACHE_NEW_DIR="${DOCKER_BUILD_CACHE_TEMPDIR}/${CONTAINER_NAME}"
46-    DOCKER_BUILD_CACHE_ARG="--cache-from type=local,src=${DOCKER_BUILD_CACHE_OLD_DIR} --cache-to type=local,dest=${DOCKER_BUILD_CACHE_NEW_DIR},mode=max"
47+  echo "DOCKER_BUILD_CACHE_ARG=$DOCKER_BUILD_CACHE_ARG"
48+
49+  # Use buildx directly
50+  # This is useful in CI where docker is used (not podman) and can help achieve better caching using registry cache

fanquake commented at 9:42 am on July 16, 2025:

and can help achieve

This is a bit vauge; have we seen this not work properly? It’s also not completely clear to me that we need buildx to use the registry. Seems like it also works with buildkit, which is now just the default docker build, it any recent Docker version.

willcl-ark commented at 10:30 am on July 16, 2025:

Yes, we have tested that even if we configure the docker builder using the setup-buildx-action where use is set to true by default (which should enable that driver in future docker build invocations IIUC), that the incorrect builder is used unless docker buildx build command specifically is used.

We have tried with BUILDKIT=1 docker build and plain docker build, but both used the incorrect driver (contrary to what we expected).

fanquake commented at 10:47 am on July 16, 2025:

Ok. Could we change this to something like:

Using buildx is required to properly load the correct driver, for use with registry caching. Neither build, nor BUILDKIT=1 currently do this properly

. Just want to avoid phrases like “can help” or “is useful”, and instead be specific about what we are doing & why.

maflcko commented at 11:02 am on July 16, 2025:

why not switch to buildx unconditionally?

willcl-ark commented at 12:19 pm on July 16, 2025:

Does podman accept docker buildx build as a command? I will check in a min with it.

but this was why not originally…

willcl-ark commented at 12:43 pm on July 16, 2025:

OK it seems docker buildx build does work with podman, so I have changed buildx to be used unconditionally in this commit now. Thanks!

maflcko commented at 12:47 pm on July 16, 2025:

Yeah, if buildx is missing on some platforms, it could also use auto-detection to use it when available and not, when not.

fanquake commented at 9:43 am on July 16, 2025: member

Concept ACK. This will also need to go back to 27.x.

in .github/workflows/ci.yml:520 in c0ad2b6aa8 outdated

545         run: sed -i "s|\${INSTALL_BCC_TRACING_TOOLS}|true|g" ./ci/test/00_setup_env_native_asan.sh
546 
547+      - name: Set mmap_rnd_bits
548+        if: ${{ matrix.name == 'TSan, depends, gui' || matrix.name == 'MSan, depends' }}
549+        # Prevents crashes due to high ASLR entropy
550+        run: sudo sysctl -w vm.mmap_rnd_bits=28

fanquake commented at 9:46 am on July 16, 2025:

Wonder if we should just move this into the scripts, rather than set it here, so that it’ll also be done for local usage (where it hasn’t been set already).

willcl-ark commented at 10:28 am on July 16, 2025:

That would make sense to me too. Happy to make that change here, or in a followup.

willcl-ark commented at 1:10 pm on July 16, 2025:

Actually I feel a bit less convinced that dropping a sudo ... command into the CI scripts is ideal for this actually. I see comments that 32 bits should now be “fixed”: https://github.com/google/sanitizers/issues/1614#issuecomment-1885369007

Testing this now locally…

willcl-ark commented at 4:19 pm on July 16, 2025:

Hmmm my msan ci job is not working (due to something unrelated) for me on my main dev machine, and tsan is segfaulting with vm.mmap_rnd_bits=32, so I’m going to leave this in one way or another for now.

maflcko commented at 6:24 am on July 17, 2025:

Actually I feel a bit less convinced that dropping a sudo ... command into the CI scripts is ideal

Yeah, they should be fine to run as a user normally. The only dependencies are Python, Bash, and a container engine.

maflcko commented at 7:32 am on August 21, 2025:

can be resolved?

willcl-ark commented at 10:29 am on July 16, 2025: member

Concept ACK. This will also need to go back to 27.x.

Testing a backport to 29.x here: https://github.com/testing-cirrus-runners/bitcoin2/actions/runs/16320536543

I think the best course of action could be to look for a little more conceptual review here, and after that squash the “ci: port x” commits in this changeset down to a single one, to make backporting to the multiple supported branches easier.

achow101 commented at 6:07 pm on July 16, 2025: member

Concept ACK

in ci/README.md:74 in c0ad2b6aa8 outdated

70+   3. Create a new "robot account" with read/write access to the previously-created Docker repository.
71+4. Set repository variables:
72+   1. `USE_CIRRUS_RUNNERS = true`
73+   2. `REGISTRY_USERNAME = <quay robot account name>`
74+5. Set repo secrets:
75+   1. `RESISTRY_TOKEN = <quay robot token>`

DrahtBot commented at 5:21 am on July 17, 2025:

RESISTRY_TOKEN -> REGISTRY_TOKEN [secret name typo]
GitHubs -> GitHub's [missing apostrophe and letter ‘s’ for possessive]

maflcko commented at 6:51 am on July 17, 2025: member

This means a new pull request should hit a pretty relevant cache. Old pull requests which are not being rebased on master may suffer from lower cache hit-rate.

I don’t think this is true. A pull request that modifies a core header (like serialize.h) will now always start from a cold cache. The current persistent workers have a high ccache hit rate for pulls that are (force) pushed for minor fixups (https://0xb10c.github.io/bitcoin-core-ci-stats/graph/ccache/). Also, before CI runs, pull requests are rebased/merged with master, so the age of a pull request alone shouldn’t affect cache hit rate.

However, the trade-offs here are probably worth to go forward and try to optimize the ccache hit rate later. Concept ACK.

Give the robot account read/write access to this repo

This seems a bit scary. Are you saying that a proprietary third party outside of our control can now push directly to the repo? My assumption was that the tokens would be added to CI in this repo and CI had write access to the registry, not the other way round. Why would the registry need write access here?

Edit: We may reconsider #31850 and drop container image caching, and just accept the intermittent network IO errors or network speed issues.

willcl-ark commented at 7:02 am on July 17, 2025: member

This means a new pull request should hit a pretty relevant cache. Old pull requests which are not being rebased on master may suffer from lower cache hit-rate.

I don’t think this is true. A pull request that modifies a core header (like serialize.h) will now always start from a cold cache. The current persistent workers have a high ccache hit rate for pulls that are (force) pushed for minor fixups (0xb10c.github.io/bitcoin-core-ci-stats/graph/ccache). Also, before CI runs, pull requests are rebased/merged with master, so the age of a pull request alone shouldn’t affect cache hit rate.

However, the trade-offs here are probably worth to go forward and try to optimize the ccache hit rate later. Concept ACK.

Yes, force pushes for minor fixups is the tradeoff we have in the current implementation. As you say, we can set the ccache to save on pull requests too (in the future) if necessary.

Give the robot account read/write access to this repo

This seems a bit scary. Are you saying that a proprietary third party outside of our control can now push directly to the repo? My assumption was that the tokens would be added to CI in this repo and CI had write access to the registry, not the other way round. Why would the registry need write access here?

Sorry for not being clearer! The robot account gets read/write access to the Quay.io (docker) repo, not this code repository!

Edit: We may reconsider #31850 and drop container image caching, and just accept the intermittent network IO errors or network speed issues.

Yes, I would love to get #31850 working in any case, as it would simply avoid long rebuilds in the worst cases; most docker images are rebuilding in < 2 minutes, except MSAN…

in .github/actions/restore-caches/action.yml:4 in 2d0ac71899 outdated

0@@ -0,0 +1,49 @@
1+name: 'Restore Caches'
2+description: 'Restore ccache, depends sources, and built depends caches'
3+inputs:
4+  job-name:

maflcko commented at 8:02 am on July 17, 2025:

nit in 2d0ac71899eadcbff8252b1704d2affb73f0160d: Should be named job-id, not the name?

willcl-ark commented at 10:35 am on July 18, 2025:

switched to job-id in 9458cd0e66d

in .github/actions/restore-caches/action.yml:17 in 2d0ac71899 outdated

12+      run: |
13+        echo "CCACHE_DIR=${{ runner.temp }}/ccache_dir" >> $GITHUB_ENV
14+        echo "DEPENDS_DIR=${{ runner.temp }}/depends" >> $GITHUB_ENV
15+        echo "SOURCES_PATH=${{ runner.temp }}/depends/sources" >> $GITHUB_ENV
16+        echo "BASE_CACHE=${{ runner.temp }}/depends/built" >> $GITHUB_ENV
17+        echo "DEPENDS_HASH=${{ hashFiles('depends/packages/*.mk', 'depends/Makefile', 'depends/config.guess', 'depends/config.sub', 'depends/funcs.mk', 'depends/builders/*.mk', 'depends/hosts/*.mk') }}" >> $GITHUB_ENV

maflcko commented at 8:11 am on July 17, 2025:

nit in https://github.com/bitcoin/bitcoin/commit/2d0ac71899eadcbff8252b1704d2affb73f0160d: Why not just git rev-parse HEAD:depends, like it was done for cirrus? (See git log -S 'git rev-parse HEAD:depends')

willcl-ark commented at 10:35 am on July 18, 2025:

This now uses git rev-parse HEAD:depends in both 868ddf55718 and 1b689cf7811

in .github/actions/restore-caches/action.yml:25 in 2d0ac71899 outdated

20+      id: ccache-cache
21+      uses: cirruslabs/cache/restore@v4
22+      with:
23+        path: ${{ env.CCACHE_DIR }}
24+        # A single ccache per job
25+        key: ccache-${{ inputs.job-name }}-${{ github.ref_name }}-${{ github.run_id }}

maflcko commented at 8:15 am on July 17, 2025:

nit in https://github.com/bitcoin/bitcoin/commit/2d0ac71899eadcbff8252b1704d2affb73f0160d: Comment seems wrong? It is a cache for each job-id + ref_name + run_id. Could just remove the comment?

willcl-ark commented at 10:36 am on July 18, 2025:

comment removed in 9458cd0e66d

in .github/actions/restore-caches/action.yml:27 in 2d0ac71899 outdated

22+      with:
23+        path: ${{ env.CCACHE_DIR }}
24+        # A single ccache per job
25+        key: ccache-${{ inputs.job-name }}-${{ github.ref_name }}-${{ github.run_id }}
26+        restore-keys: |
27+          ccache-${{ inputs.job-name }}-master-

maflcko commented at 8:22 am on July 17, 2025:

nit in https://github.com/bitcoin/bitcoin/commit/2d0ac71899eadcbff8252b1704d2affb73f0160d: why hardcode master. This makes it impossible to cache in pull requests, if this is ever done in the future. Also, only master caches are created, so this doesn’t guard against anything.

Also, the others don’t have it.

willcl-ark commented at 10:37 am on July 18, 2025:

All references to master have been switched to github.event.repository.default_branch where appropriate. Removed the ref_name from the save/restore keys, as it is indeed unnecessary.

in .github/actions/save-caches/action.yml:12 in 2d0ac71899 outdated

 7+runs:
 8+  using: 'composite'
 9+  steps:
10+    - name: Save Ccache cache
11+      uses: cirruslabs/cache/save@v4
12+      if: ${{ (github.event_name == 'push') && (github.ref_name == 'master') }}

maflcko commented at 8:30 am on July 17, 2025:

nit in the same commit: Could use github.event.repository.default_branch instead of master, so that caches can be saved for other repos, such as inquisition?

(same below)

in .github/actions/restore-caches/action.yml:36 in 2d0ac71899 outdated

31+      id: depends-sources
32+      uses: cirruslabs/cache/restore@v4
33+      with:
34+        path: ${{ env.SOURCES_PATH }}
35+        # A single shared depends sources for all jobs per hash of package definitions and meta_depends
36+        key: depends-sources-${{ env.DEPENDS_HASH }}-${{ github.run_id }}

maflcko commented at 8:32 am on July 17, 2025:

same commit: This seems broken? A job-id with NO_QT=1 populating this would corrupt the cache for other job-id’s that build qt?

willcl-ark commented at 10:38 am on July 18, 2025:

9458cd0e66d now uses a depends sources cache per job_id to avoid this pitfall (thanks!):

0key: depends-sources-${{ inputs.job-id }}-${{ env.DEPENDS_HASH }}

in .github/actions/restore-caches/action.yml:14 in 2d0ac71899 outdated

 9+  steps:
10+    - name: Set cache paths
11+      shell: bash
12+      run: |
13+        echo "CCACHE_DIR=${{ runner.temp }}/ccache_dir" >> $GITHUB_ENV
14+        echo "DEPENDS_DIR=${{ runner.temp }}/depends" >> $GITHUB_ENV

maflcko commented at 8:36 am on July 17, 2025:

same commit: Unused?

willcl-ark commented at 10:39 am on July 18, 2025:

Moved this code into the main ci.yml file, so that the cache actions are only performing caching (rather than being responsible for setting env vars), both in 868ddf55718 and 1b689cf7811

in .github/actions/configure-docker/action.yml:38 in b1a4adc814 outdated

33+        password: ${{ inputs.registry-password }}
34+
35+    - name: Set docker build args
36+      shell: bash
37+      run: |
38+        source ${{ env.FILE_ENV }}

maflcko commented at 8:44 am on July 17, 2025:

nit in b1a4adc814369aff21b8c12775270cfe648db486: Could add a comment that this is only used to extract CONTAINER_NAME.

willcl-ark commented at 10:39 am on July 18, 2025:

Comment added in db78efac768

in .github/workflows/ci.yml:25 in 9002faff0d outdated

19@@ -20,6 +20,9 @@ concurrency:
20 env:
21   CI_FAILFAST_TEST_LEAVE_DANGLING: 1  # GHA does not care about dangling processes and setting this variable avoids killing the CI script itself on error
22   MAKEJOBS: '-j10'
23+  REGISTRY: ${{ vars.REGISTRY || 'quay.io' }}
24+  REGISTRY_ORG: ${{ vars.REGISTRY_ORG || 'bitcoin-core' }} # TODO: Set correct org
25+  REGISTRY_REPO: ${{ vars.REGISTRY_REPO || 'core-ci' }} # TODO: Set correct repo

maflcko commented at 8:46 am on July 17, 2025:

9002faff0d7518ebafe05d8d43d6558d238c2659: adjust for https://quay.io/repository/bitcoincore/bitcoin-ci ?

willcl-ark commented at 10:40 am on July 18, 2025:

Registry env vars updated in 7272f56a755

maflcko approved

maflcko commented at 8:50 am on July 17, 2025: member

I guess you want review here and then address it, as it comes in? Once review is finished, the app will be installed and reviewers can also look at a “real” run in this repo?

looked at c0ad2b6aa8e8c31c9f9c9ea2b35ca86f7985c490~23 🎬

Signature:

0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
2trusted comment: looked at c0ad2b6aa8e8c31c9f9c9ea2b35ca86f7985c490~23 🎬
3gylJtm++jv0E+65SRoLFPC+ef+fwpVJftiMQ+ziB1uRZAF2MwE7TW3JaEv8iJIpZsRmForPR0jik8/6QvUs+BQ==

hebasto commented at 12:20 pm on July 17, 2025: member

Concept ACK.

we qualify for an open source discount of 50%.

We would be dependent on Cirrus infra…

We shouldn’t be surprised when Cirrus suddenly changes its modus operandi, including its advertised open-source discount or general availability.

willcl-ark commented at 12:35 pm on July 17, 2025: member

Concept ACK.

we qualify for an open source discount of 50%.

We would be dependent on Cirrus infra…

We shouldn’t be surprised when Cirrus suddenly changes its modus operandi, including its advertised open-source discount or general availability.

Certainly, it is good to be wary of that. I think this is equally true for all cloud providers though.

It’s my belief that if we complete this migration we are resonably well protected against this risk for the following reasons:

As we use the common GitHub workflow yaml format, changing to another provider can be as simple as amending the runs-on: line in the yaml (and installing a different provider’s GH app, and changing the cache action to that provider’s own).
We could always revert back to a self-hosted solution utilizing the new workflow yaml via https://github.com/actions/runner, which for example can be trivially configured on one or more self-hosted servers using services.github.runner with Nix (I have a demo of such a configuration here).

We seem to have a good working relationship so far with Cirrus, @m3dwards has a responsive and helpful contact there. Of course, we are in the tendering stage so there is perhaps extra impetus to be helpful to us, but I don’t see any reason that a historical precedence of limiting free runners (which are allegedly being abused for crypto mining), should appear any more risky to paid/premium customers (than any other provider).

maflcko commented at 12:42 pm on July 17, 2025: member

2. We could always revert back to a self-hosted solution

Agree that this may happen with any third party (including GitHub itself). If we want to switch back to the self-hosted runners, it should be as trivial as git revert $the_merge_commit_of_this_pull. Alternatively, switch to GHA-based self-hosted runners. Though, it would be good to create a proof-of-concept pull request to switch to self-hosted runners in GHA, or a different hosted alternative (e.g. warp-build), after this pull is merged. This makes it easier to see that (1) it works and (2) how easily it is possible.

willcl-ark force-pushed on Jul 18, 2025

willcl-ark commented at 10:34 am on July 18, 2025: member

Pushed c126475ed7a17ec9030066056e31846c7124dcf3 with a CI run on master branch at https://github.com/testing-cirrus-runners/bitcoin2/actions/runs/16368410249

willcl-ark commented at 10:40 am on July 18, 2025: member

Thanks for the review @maflcko & @fanquake , I hope I addressed all your current review comments.

0xB10C commented at 11:27 am on July 18, 2025: contributor

Concept ACK!

I think finishing the migration would close https://github.com/bitcoin/bitcoin/issues/31965

in ci/test/02_run_container.sh:52 in 20a258310e outdated

45@@ -46,8 +46,10 @@ if [ -z "$DANGER_RUN_CI_ON_HOST" ]; then
46     DOCKER_BUILD_CACHE_ARG="--cache-from type=local,src=${DOCKER_BUILD_CACHE_OLD_DIR} --cache-to type=local,dest=${DOCKER_BUILD_CACHE_NEW_DIR},mode=max"
47   fi
48 
49+  # Use buildx unconditionally
50+  # Using buildx is required to properly load the correct driver, for use with registry caching. Neither build, nor BUILDKIT=1 currently do this properly
51   # shellcheck disable=SC2086
52-  DOCKER_BUILDKIT=1 docker build \
53+  DOCKER_BUILDKIT=1 docker buildx build \

maflcko commented at 1:49 pm on July 18, 2025:

nit in 20a258310ea4fe1c4cd280d26047aee063215454: Buildx always uses BuildKit. Can remove buildkit stuff.

maflcko commented at 8:19 pm on July 24, 2025:

(tested podman buildx on a vanilla Ubuntu 24.04, fwiw)

willcl-ark commented at 9:33 pm on July 25, 2025:

Thanks! Removed in a4084cb7554

in ci/test/02_run_container.sh:27 in e5295f7ad6 outdated

44-    DOCKER_BUILD_CACHE_TEMPDIR="$(mktemp --directory ci-docker-build-cache-XXXXXXXXXX)"
45-    DOCKER_BUILD_CACHE_NEW_DIR="${DOCKER_BUILD_CACHE_TEMPDIR}/${CONTAINER_NAME}"
46-    DOCKER_BUILD_CACHE_ARG="--cache-from type=local,src=${DOCKER_BUILD_CACHE_OLD_DIR} --cache-to type=local,dest=${DOCKER_BUILD_CACHE_NEW_DIR},mode=max"
47-  fi
48+  echo "DOCKER_BUILD_CACHE_ARG=$DOCKER_BUILD_CACHE_ARG"
49

maflcko commented at 1:53 pm on July 18, 2025:

nit in e5295f7ad64be1d834d3e0c2726edfcc67debfff: What about DANGER_DOCKER_BUILD_CACHE_HOST_DIR below?

Could link to the reverted commit in the commit description instead: e87429a2d0f23eb59526d335844fa5ff5b50b21f?

willcl-ark commented at 9:33 pm on July 25, 2025:

I have linked to the commit in e83c3718f2c and removed the dangling DANGER_DOCKER_BUILD_CACHE_HOST_DIR below

in ci/test/02_run_container.sh:26 in e5295f7ad6 outdated

43-    # ephemeral, it has to take care of cleaning old TEMPDIR's up.
44-    DOCKER_BUILD_CACHE_TEMPDIR="$(mktemp --directory ci-docker-build-cache-XXXXXXXXXX)"
45-    DOCKER_BUILD_CACHE_NEW_DIR="${DOCKER_BUILD_CACHE_TEMPDIR}/${CONTAINER_NAME}"
46-    DOCKER_BUILD_CACHE_ARG="--cache-from type=local,src=${DOCKER_BUILD_CACHE_OLD_DIR} --cache-to type=local,dest=${DOCKER_BUILD_CACHE_NEW_DIR},mode=max"
47-  fi
48+  echo "DOCKER_BUILD_CACHE_ARG=$DOCKER_BUILD_CACHE_ARG"

maflcko commented at 1:56 pm on July 18, 2025:

nit in e5295f7ad64be1d834d3e0c2726edfcc67debfff: The script has tracing enabled, so i don’t think this is needed?

willcl-ark commented at 9:34 pm on July 25, 2025:

echo also removed in e83c3718f2c

in .github/workflows/ci.yml:291 in 868ddf5571 outdated

286@@ -287,7 +287,8 @@ jobs:
287 
288   windows-cross:
289     name: 'Linux->Windows cross, no tests'
290-    runs-on: ubuntu-latest
291+    # We hardcode the repo name here, otherwise PRs, which originate "from a fork", will NOT use cirrus runners.
292+    runs-on: ${{ (github.repository == 'bitcoin/bitcoin' || vars.USE_CIRRUS_RUNNERS == 'true') && 'ghcr.io/cirruslabs/ubuntu-runner-amd64:24.04-md' || 'ubuntu-24.04' }}

maflcko commented at 2:15 pm on July 18, 2025:

nit in 868ddf557180ffbc7a60ed539a5df1e1cd3e757c: this seems a bit fragile and won’t work for monotree repos ( bitcoin-core/gui), and won’t work for forks that accept pull requests.

Seems better to just hard-code this to cirrus, possibly add a comment: # Run on cirrus by default, but in forks this can be replaced by 'ubuntu-latest' to run on GHA

maflcko commented at 2:48 pm on July 18, 2025:

If you want it to be easy to edit, I wonder if this can be a separate yaml file so that the edit only has to be done in one place?

maflcko commented at 11:02 am on July 21, 2025:

Unrelatedly, the USE_CIRRUS_RUNNERS can probably be removed:

It is only there for forks, but the comment says it doesn’t work for forks anyway in the expected way.
If someone maintains a fork (presumably to add patches), it should be trivial for them to make the required patch to the CI as well.

m3dwards commented at 12:52 pm on July 25, 2025:

Ideally CI will run without a patch in as many places as possible.

By checking the repository name it will run for: pushes/merges to main repo, PRs from forks and pushes to forks (where it would run on a github runner).

The USE_CIRRUS_RUNNERS enabled a fork to also run on Cirrus if they have bought some runners but would NOT work for a PR from a third fork to that fork. I think this is confusing and so I agree to remove this var. Forks would have to patch if they want to run on cirrus runners.

Where I think I disagree is on removing the repository name check as then pushes to forks will not run on github’s infrastructure which I think is a really nice to have. The more that people’s personal forks run CI the less strain it will put on the main repo or on Cirrus runners. I know when I work on things it’s nice to see CI running on my fork before creating a PR.

I’m not sure why the GUI repo should be running bitcoin/bitcoin CI, shouldn’t it have it’s own GUI related CI? But regardless, by keeping the repo name check the GUI repo will be able to run CI but just slower on github’s runners.

willcl-ark commented at 9:35 pm on July 25, 2025:

6d252266e40 and 3e757e511a3 have had the USE_CIRRUS_RUNNERS variable removed as it is confusing and doesn’t work on PR’s to downstream forks as expected (which is partially what it is for).

760497ac393 gives some hints on what a fork should patch, should they wish to use Cirrus Runners on their own CI.

in .github/workflows/ci.yml:318 in c126475ed7 outdated

316+          echo "BASE_CACHE=${{ runner.temp }}/depends/built" >> $GITHUB_ENV
317+          echo "DEPENDS_HASH=$(git rev-parse HEAD:depends)" >> $GITHUB_ENV
318+
319+      - name: Restore caches
320+        id: restore-cache
321+        # TODO: previously this also hashed "ci/test/00_setup_env_win64.sh"

maflcko commented at 2:20 pm on July 18, 2025:

nit in 868ddf5: Yeah, seems required. Same for the other caches?

willcl-ark commented at 9:36 pm on July 25, 2025:

6d252266e40 and 3e757e511a3 now hash FILE_ENV too for the caches, using:

0DEPENDS_HASH=$(git ls-tree HEAD depends "ci/test/$FILE_ENV" | sha256sum | cut -d' ' -f1)

maflcko commented at 2:23 pm on July 18, 2025: member

looked at c126475ed7a17ec9030066056e31846c7124dcf~20 💇

Signature:

0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
2trusted comment: looked at c126475ed7a17ec9030066056e31846c7124dcf~20 💇
3kep8ZK4UJEamaLijXtMFwgjSmf1fhSJuF49dbZ/NHDe/5jmIZR2EzJa0ewjjGov4n3xWZMN5f3LKRekrMZ6HAw==

in .github/workflows/ci.yml:518 in c126475ed7 outdated

518+
519+          - name: 'MSan, depends'
520+            id: 'msan-depends'
521+            cirrus-runner: 'ghcr.io/cirruslabs/ubuntu-runner-amd64:24.04-lg'
522+            fallback-runner: 'ubuntu-24.04'
523+            timeout-minutes: 300

fanquake commented at 10:46 am on July 21, 2025:

Note to remember to drop the timeout post #32999.

m3dwards commented at 1:22 pm on July 21, 2025:

Done

m3dwards force-pushed on Jul 21, 2025

willcl-ark force-pushed on Jul 25, 2025

willcl-ark commented at 9:45 pm on July 25, 2025: member

Thanks for the review on this so far.

Whilst we had tested the docker registry caching on PRs successfully, because I was opening them myself (and was owner of the parent repo) repo-level variables were available to me which were not available to 3rd party pull requests.

The short of this is that this meant the docker registry cache setup could not pull from the registry on pull requests and we didn’t think it was therefore suitable for our purposes.

We have switched to the gha cache backend in the latest force-push with the good news that we finally have it working for all images correctly; caching to/from on pushes, and caching from on pulls.

A push to master can be found here: https://github.com/testing-cirrus-runners/bitcoin2/actions/runs/16529929752

And a pull request (from a 3rd party account) here: https://github.com/testing-cirrus-runners/bitcoin2/actions/runs/16531270443?pr=3

A new commit has been added, 0bd758e6c54, to fix a (new?) issue we experienced with the asan job where the runner host appeared to update it’s image and kernel. The cached docker image for the ASAN job then had the incorrect linux-headers-<version> package installed, causing the job to fail. There are likely multiple ways to fix this (we could input uname as a docker build arg for example), but simply forcing a reinstall of the correct headers in 03_*.sh seemed easiest. Open to suggestions here however…

Marking as ready for review now, as I think this is conceptually ready.

willcl-ark marked this as ready for review on Jul 25, 2025

willcl-ark force-pushed on Jul 28, 2025

willcl-ark commented at 9:07 am on July 28, 2025: member

This push extracts the repeated set CI directories and Set cache paths logic into a configure-environment action

0      - name: Set CI directories
1        run: |
2          echo "BASE_ROOT_DIR=${{ runner.temp }}" >> "$GITHUB_ENV"
3          echo "BASE_BUILD_DIR=${{ runner.temp }}/build-${{ matrix.build-dir-suffix }}" >> "$GITHUB_ENV"
4      - name: Set cache paths
5        run: |
6          ...

and simplifies the configure-docker action string building to avoid repeated ie/else string-building which is prone to errors when updating.

willcl-ark commented at 9:25 am on July 28, 2025: member

Some noteworthy observations from the last two weeks of runs:

We have seen variance in network speeds occasionally which has been raised with Cirrus. e.g. in this run we can see a cache restore hanging for 5 minutes:

It seems to manifest intermittently, and Cirrus are aware and working on it.

We have seen (very rarely) recently jobs take a while to start even when runners are available. This also is known to them and they are workign on fixing, and may even be fixed already:

There seems to be a slight change in webhook creativeness on GitHub side. We have a potential fix that we plan to deploy this night and keep a close attention over the weekend.

fkorotkov commented at 10:41 am on July 28, 2025: none

Hey folks,

Could someone with admin permissions please check that Allow public repositories checkbox is set on this page:

https://github.com/organizations/bitcoiin/settings/actions/runner-groups/1

Seems this is the reason runners are not picking jobs at the moment.

fanquake commented at 10:42 am on July 28, 2025: member

@fkorotkov. Should be fixed now:

willcl-ark force-pushed on Jul 28, 2025

willcl-ark commented at 12:19 pm on July 28, 2025: member

FWIW the CI run taking place here in this PR is using Cirrus Runners on the migrated jobs (thanks @fkorotkov !) and demonstrates an absolute worst-case runtimeof these jobs with:

No docker build cache
No ccache cache
No depends-sources caches
No depends-build caches

in .github/actions/configure-docker/action.yml:15 in 1f5eb93de4 outdated

 5+  steps:
 6+    - name: Set up Docker Buildx
 7+      uses: docker/setup-buildx-action@v3
 8+      with:
 9+        driver-opts: |
10+          network=host

maflcko commented at 12:36 pm on July 28, 2025:

nit in 1f5eb93de45f6a530ba54d799f637c0f1b45bc1b: Could add a comment why this is needed? I guess due to GHA not supporting ipv6 in $current_year?

willcl-ark commented at 9:35 pm on July 28, 2025:

I am actually not 100% sure why this is needed. Warp have it documented in their docs, but IIRC Cirrus also advised us to set it.

My understanding is that it was necessary for the gha caching using custom URL, but perhaps @m3dwards or @fkorotkov could clarify/confirm?

m3dwards commented at 1:43 pm on August 5, 2025:

Added a comment. The docker build container uses gha style cache which is available on the host at http://127.0.0.1:12321/

maflcko commented at 2:12 pm on August 5, 2025:

Thx, but this reminds me that the setting the min port is missing? #32989 (review)

m3dwards commented at 6:05 pm on August 13, 2025:

replied in: #32989 (review)

maflcko commented at 7:26 am on August 21, 2025:

(resolving thread for now) edit:

“Failed to resolve thread”

in .github/actions/configure-docker/action.yml:30 in 1f5eb93de4 outdated

25+        # On forks the gha cache will work but will use Github's cache backend.
26+        # Docker will check for variables $ACTIONS_CACHE_URL, $ACTIONS_RESULTS_URL and $ACTIONS_RUNTIME_TOKEN
27+        # which are set automatically when running on GitHub infra: https://docs.docker.com/build/cache/backends/gha/#synopsis
28+
29+        # Use cirrus cache host in main repo
30+        if [[ "$GITHUB_REPOSITORY" == "bitcoin/bitcoin" ]]; then

maflcko commented at 12:59 pm on July 28, 2025:

nit in 1f5eb93de45f6a530ba54d799f637c0f1b45bc1b: Seems like there are several places that hard-code the main repo, which makes it harder for forks to use cirrus runners.

Maybe a global env next to CIRRUS_CACHE_HOST could be used to make that simpler?

Maybe REPO_USE_CIRRUS_RUNNERS: 'bitcoin/bitcoin' # Use cirrus runners and cache for this repo, instead of falling back to the slow GHA runners?

willcl-ark commented at 3:42 pm on July 28, 2025:

We did try this originally, but (very annoyingly) it does not seem to be possible without a primary workflow to call this workflow. The reason is that the runs-on field only has access to the following “contexts”:

0github, needs, strategy, matrix, vars, inputs

See https://docs.github.com/en/actions/reference/workflows-and-actions/contexts

Whilst vars sounds like it would do the job, the problem is that in workflows of the type on: pull_request that vars (which refer here to “repo vars”, not “workflow level env vars”) are not available to pull requests. They are available to on: pull_request_target wofklows, but we don’t want to go down that route…

Sadly, I can’t even think of a nice-enough hack around this without doing something like:

Create a preliminary workflow which will determine whether to use cirrus runners or not based on an env var.
Change our main workflow type to be on: workflow_call:, where it can then be called from the preliminary workflow, passing with it the information using inputs on whether or not to use cirrus runners.

I hope you agree with me such spaghettic is not worth it.

I think I can minimise the usage of the hardcoding to a single file (ci.yml) though so that a patch is easy to make and maintain…

willcl-ark commented at 7:35 pm on July 28, 2025:

I hope you agree with me such spaghetti is not worth it.

Actually, this turns out to be a lot less invasive than I thought, and the job only takes ~4 seconds to run: https://github.com/testing-cirrus-runners/bitcoin/actions/runs/16578429043

The job is pretty simple: https://github.com/testing-cirrus-runners/bitcoin/commit/9f3fdf420ea56d32db3df98bef2026177afb4067

Perhaps we could use this approach after all?

willcl-ark commented at 9:36 pm on July 28, 2025:

Added a helper job (runtime of ~4 seconds) which has an output whether or not to use cirrus to the subsequent jobs in 8758b6ec333 ci: add job to determine runner type

This reduces the hardcoding to a single env var in the main ci.yml file, which can trivially be patched by forks if desired.

maflcko commented at 9:41 am on July 30, 2025:

Heh, yeah. Somewhat it seems plausible that the environment may not be available before the vm boots up. A ~4 second delay should be acceptable. And it would be trivial to revert to hardcoding thrice, if there is need to.

(Feel free to resolve)

in .github/workflows/ci.yml:289 in e9cf1c9005 outdated

284@@ -285,7 +285,8 @@ jobs:
285 
286   windows-cross:
287     name: 'Linux->Windows cross, no tests'
288-    runs-on: ubuntu-latest
289+    # We hardcode the repo name here, otherwise PRs, which originate "from a fork", will NOT use cirrus runners.
290+    runs-on: ${{ github.repository == 'bitcoin/bitcoin' && 'ghcr.io/cirruslabs/ubuntu-runner-amd64:24.04-md' || 'ubuntu-24.04' }}

maflcko commented at 1:00 pm on July 28, 2025:

nit in e9cf1c90052620b23745ca0d3fdccda2cf5fa0fc: Same, about REPO_USE_CIRRUS_RUNNERS

in .github/workflows/ci.yml:446 in 35a9dd7352 outdated

439@@ -440,3 +440,52 @@ jobs:
440           path: ${{ env.CCACHE_DIR }}
441           # https://github.com/actions/cache/blob/main/tips-and-workarounds.md#update-a-cache
442           key: ${{ github.job }}-ccache-${{ github.run_id }}
443+
444+  ci-matrix:
445+    name: ${{ matrix.name }}
446+    runs-on: ${{ github.repository == 'bitcoin/bitcoin' && matrix.cirrus-runner || matrix.fallback-runner }}

maflcko commented at 1:38 pm on July 28, 2025:

nit in 35a9dd7352d84292fbc2dcf2e544b380be49b4c3: Same (REPO_USE_CIRRUS_RUNNERS)

in .github/workflows/ci.yml:422 in 35a9dd7352 outdated

447+
448+    if: ${{ vars.SKIP_BRANCH_PUSH != 'true' || github.event_name == 'pull_request' }}
449+    timeout-minutes: ${{ matrix.timeout-minutes }}
450+
451+    env:
452+      DANGER_CI_ON_HOST_FOLDERS: 1

maflcko commented at 1:41 pm on July 28, 2025:

nit in 35a9dd7352d84292fbc2dcf2e544b380be49b4c3: Could just be a global env var, instead of repeating it thrice?

in .github/workflows/ci.yml:465 in 35a9dd7352 outdated

460+            id: '32bit-arm'
461+            cirrus-runner: 'ghcr.io/cirruslabs/ubuntu-runner-arm64:24.04-md'
462+            fallback-runner: 'ubuntu-24.04-arm'
463+            timeout-minutes: 120
464+            file-env: './ci/test/00_setup_env_arm.sh'
465+            build-dir-suffix: 'arm'

maflcko commented at 1:45 pm on July 28, 2025:

nit in 35a9dd7352d84292fbc2dcf2e544b380be49b4c3: Unused? Maybe you wanted this to be the container name, or the job id instead?

willcl-ark commented at 9:37 pm on July 28, 2025:

Removed dead code no-longer needed in 3825f903585 ci: port arm 32-bit job (and subsequent job commits)

in .github/actions/restore-caches/action.yml:6 in c7aa954564 outdated

0@@ -0,0 +1,47 @@
1+name: 'Restore Caches'
2+description: 'Restore ccache, depends sources, and built depends caches'
3+inputs:
4+  job-id:
5+    description: 'job id'
6+    required: true

maflcko commented at 1:55 pm on July 28, 2025:

nit in c7aa954564ceaf0800be461115a568d3c919228a: This seems somewhat redundant with the container name now. It would be less yaml code to use container name instead of job id here?

willcl-ark commented at 9:37 pm on July 28, 2025:

Thanks, I agree.

Removed the input entirely and use $CONTAINER_NAME in d6ed832034b ci: add caching actions

in ci/test/03_test_script.sh:30 in 88816a4ed3 outdated

23@@ -24,6 +24,12 @@ fi
24 echo "Free disk space:"
25 df -h
26 
27+# We force an install of linux-headers again here to fix any kernel mismatch
28+# between a cached docker image and the underlying host
29+if [[ "$CONTAINER_NAME" == "ci_native_asan" ]]; then
30+  $CI_RETRY_EXE apt-get install -y bpfcc-tools linux-headers-"$(uname --kernel-release)"

maflcko commented at 2:01 pm on July 28, 2025:

nit in 88816a4ed3f5d528ed7b997241df96fb1f51df44: apt-get install will fail with a stale Ubuntu IP list from a cached image. You’ll have to call apt-get update before every install for each image layer.

maflcko commented at 2:03 pm on July 28, 2025:

Also, I wonder if this can use $PACKAGES?

willcl-ark commented at 10:11 pm on July 28, 2025:

Fixed, and now using $PACKAGES in 8bb213c64c3 ci: force reinstall of kernel headers in asan

in .github/workflows/ci.yml:495 in a3789f2056 outdated

490+            id: 'tsan-depends-gui'
491+            cirrus-runner: 'ghcr.io/cirruslabs/ubuntu-runner-amd64:24.04-lg'
492+            fallback-runner: 'ubuntu-24.04'
493+            timeout-minutes: 120
494+            file-env: './ci/test/00_setup_env_native_tsan.sh'
495+            build-dir-suffix: 'tidy'

maflcko commented at 2:07 pm on July 28, 2025:

nit in a3789f2056a76e72fcafbae7396c35f62fb93f56: wrong (unused) suffix

willcl-ark commented at 9:38 pm on July 28, 2025:

Removed in e62bd1b24b6 ci: port tsan-depends-gui

in .github/workflows/ci.yml:561 in 67429ff9ae outdated

556+      - name: Configure Docker
557+        uses: ./.github/actions/configure-docker
558+
559+      - name: CI script
560+        run: |
561+          set -x

maflcko commented at 2:11 pm on July 28, 2025:

nit in 67429ff9ae0ad7236d01fee7d25dceb965741b2f: Please use -o xtrace, so that non-Bash experts can also understand the intention, without having to ask an LLM read the manpage

willcl-ark commented at 9:58 pm on July 28, 2025:

Fixed in fd32aa85b29 ci: port lint

in .cirrus.yml:5 in 72d82211fc outdated

-1@@ -1,199 +0,0 @@
0-env:  # Global defaults
1-  CIRRUS_CLONE_DEPTH: 1
2-  CIRRUS_LOG_TIMESTAMP: true
3-  MAKEJOBS: "-j10"
4-  TEST_RUNNER_PORT_MIN: "14000"  # Must be larger than 12321, which is used for the http cache. See https://cirrus-ci.org/guide/writing-tasks/#http-cache

maflcko commented at 2:13 pm on July 28, 2025:

Just as a note: This is fine to remove without replacement, because the cirrus cache now runs outside the container. Here, it ran inside it.

maflcko commented at 2:12 pm on August 5, 2025:

Actually, with network=host, this is needed, no?

m3dwards commented at 6:04 pm on August 13, 2025:

I’m not sure it’s needed still, the network=host is applied to the buildx container which is only used to build the images and isn’t actually used to run the CI images.

The main CI container mount cache directories on the host and then separate steps restore and save from those directories on the host without docker.

The buildx container needs to access the http cache to look for cached build layers.

maflcko commented at 6:42 am on August 21, 2025:

Ah, ok. I am not familiar with the GitHub Actions Docker Caching Buildx setup interactions with the docker containers that are run later. But given that this is documented as a buildx option (https://docs.docker.com/reference/cli/docker/buildx/create/#driver-opt) it seems plausible that it may not apply to the docker container.

maflcko commented at 7:11 am on August 21, 2025:

Also, it should be fine to re-apply once the CI intermittently fails due to this. So this pull looks good as-is and no change is needed here.

Unrelatedly, if we want to keep/use the CI setting in the future, it must be documented as a hidden option:

https://github.com/bitcoin/bitcoin/blob/7d9789401be4c91a9eb3c1112564a6524bdc4f70/ci/test/02_run_container.py#L29-L34

in .github/workflows/ci.yml:528 in 67429ff9ae outdated

538@@ -539,3 +539,25 @@ jobs:
539           job-id: ${{ matrix.id }}
540           depends-sources-cache-hit: ${{ steps.restore-cache.outputs.depends-sources-cache-hit }}
541           depends-built-cache-hit: ${{ steps.restore-cache.outputs.depends-built-cache-hit }}
542+
543+  lint:

maflcko commented at 2:22 pm on July 28, 2025:

67429ff9ae0ad7236d01fee7d25dceb965741b2f: This is wrong and will silently pass on any input. You must set COMMIT_RANGE=, like it is done in the script today.

willcl-ark commented at 10:09 pm on July 28, 2025:

Set this to run on PRs in fd32aa85b29 ci: port lint by setting CIRRUS_PR=1 via if [ ${{ github.event_name }}" = "pull_request" ]; then which I think should fix this?

maflcko commented at 9:37 am on July 30, 2025:

Set this to run on PRs in fd32aa8 ci: port lint by setting CIRRUS_PR=1 via if [ ${{ github.event_name }}" = "pull_request" ]; then which I think should fix this?

Yes, this should fix it. This will correctly set the COMMIT_RANGE, just like it did previously. Thx.

in ci/test/00_setup_env.sh:38 in 693d3615cd outdated

34@@ -35,7 +35,7 @@ fi
35 
36 echo "Fallback to default values in env (if not yet set)"
37 # The number of parallel jobs to pass down to make and test_runner.py
38-export MAKEJOBS=${MAKEJOBS:--j4}
39+export MAKEJOBS=${MAKEJOBS:--j$(if command -v nproc > /dev/null 2>&1; then nproc; else sysctl -n hw.logicalcpu; fi)}

maflcko commented at 2:31 pm on July 28, 2025:

693d3615cdb25c5f185c1e6097a6e647eca7153b: This commit isn’t doing anything, because the MAKEJOBS are set globally in the env in GHA (same for the other “perf” commits below).

willcl-ark commented at 10:42 pm on July 28, 2025:

Removed the global makejobs in 89f177406c9 ci: dynamically match makejobs with cores

maflcko commented at 2:32 pm on July 28, 2025: member

there is a bug in the lint config. Also, left some nits.

looked at 72d82211fce78a18bce27a90726d30c85518955c~6 📊

Signature:

0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
2trusted comment: looked at 72d82211fce78a18bce27a90726d30c85518955c~6 📊
3bU4/CudCrcvil8aKqUlXXpLa97DY3Kj7FWx5lSSAhICRDONbCHC2zwPtEQusEF8btAqpICvOG7VJLZX+6A5wCg==

willcl-ark force-pushed on Jul 28, 2025

in ci/test/03_test_script.sh:27 in 3c7b462fa9 outdated

23@@ -24,6 +24,14 @@ fi
24 echo "Free disk space:"
25 df -h
26 
27+# We force an install of linux-headers again here via $PACAKGES to fix any

maflcko commented at 6:16 am on July 29, 2025:

nit (from the llm):

PACAKGES -> PACKAGES [env var name typo]

willcl-ark commented at 10:04 am on July 30, 2025:

Fixed in 8dd7476d496

willcl-ark force-pushed on Jul 29, 2025

willcl-ark commented at 2:46 pm on July 29, 2025: member

I pushed a commit to make a cache save on pull requests, and then a force push dropping that commit. This worked, but I do see some unexpected ccache hit rates… not sure if anyone here might have an idea about it; I’m at a bit of loss about what could be going wrong here to be honest.

In e.g. the previous releases job, the cache was saved here: https://github.com/bitcoin/bitcoin/actions/runs/16589926678/job/46922852038?pr=32989#step:9:77

Sent 426321843 of 426321843 (100.0%), 81.3 MBs/sec Cache saved with key: ccache-ci_native_previous_releases-16589926678

This is the correct size for this cache; the CI run build ended with:

Primary storage: Hits: 0 / 1494 (0.00 %) Misses: 1494 Cache size (GB): 0.43 / 0.50 (86.19 %) Cleanups: 20

We then in the next run, where I only dropped the temporary cache-save commit, restored this exact cache: https://github.com/bitcoin/bitcoin/actions/runs/16595726917/job/46941763412?pr=32989#step:4:49 (note that we don’t hit on primary key, but hit on a fallback, final line, which matches exactly the cache save above):

Cache hit for: ccache-ci_native_previous_releases-16595726917 Cache Size: ~407 MB (426321843 B) Cache restored successfully Cache restored from key: ccache-ci_native_previous_releases-16589926678

In fact we know this is the correct cache as this is currently the only cache that has ever been saved for this job. The problem is that later on in this re-run our ccache hit-rate was a meagre 19%: https://github.com/bitcoin/bitcoin/actions/runs/16595726917/job/46941763412?pr=32989#step:8:2986

ccache version 4.5.1 Summary: Hits: 148 / 747 (19.81 %) Direct: 54 / 747 (7.23 %) Preprocessed: 94 / 693 (13.56 %) Misses: 599 Direct: 693 Preprocessed: 599 Primary storage: Hits: 360 / 1624 (22.17 %) Misses: 1264 Cache size (GB): 0.43 / 0.50 (85.79 %) Cleanups: 64

I’m not sure exactly what is causing the misses here; I would expect this to be 100% (or 99.x%) which I have seen previously.

maflcko commented at 3:02 pm on July 29, 2025: member

I’m not sure exactly what is causing the misses here; I would expect this to be 100% (or 99.x%) which I have seen previously.

ccache does not work when the code changes, so you’ll have to make sure to compile the same code (usually commit id). Ref:

https://github.com/bitcoin/bitcoin/actions/runs/16589926678/job/46922852038?pr=32989#step:2:89 [e115ce1e8089037e2bde27600fb092425b0c461d] vs
https://github.com/bitcoin/bitcoin/actions/runs/16595726917/job/46941763412?pr=32989#step:2:89 [af35901de6ed53cfdb7d60f2b49607d1d187e780]

This means the code was https://github.com/bitcoin/bitcoin/commit/953c90d7649c076e6e43418f50c33b3ae5f82608 vs https://github.com/bitcoin/bitcoin/commit/321984705dbcc5982013c8cfb52389cf3f2e75f6

willcl-ark commented at 3:29 pm on July 29, 2025: member

This means the code was 953c90d vs 3219847

Dang, I must have rebased on upstream/master (out of habit) to drop that top commit. Thanks for catching.

maflcko commented at 6:08 pm on July 29, 2025: member

The checkout action by default picks the “rebase”/merge with master, so this should happen even without you rebasing. In fact, it should happen even when not pushing at all, but just re-running the task. However, I just noticed this does not happen (https://github.com/bitcoin/bitcoin/actions/runs/16521447783/job/46969566419?pr=29641#step:2:90 picks a 4-day old merge). This breaks the use-case of being able to detect silent merge conflicts before a maintainer detects them (or misses them).

Not a blocker, but I think it would be nice to fix this, if possible.

Edit: for more context, on cirrus the merge_base works correctly: https://cirrus-ci.com/task/6351223885922304?logs=merge_base#L7

in .cirrus.yml:107 in 4f4d22a150 outdated

101-  # For faster CI feedback, immediately schedule the linters
102-  << : *CREDITS_TEMPLATE
103-  unshallow_script:
104-    - git fetch --unshallow --no-tags
105-  lint_script:
106-    - ./ci/lint_run.sh

maflcko commented at 7:32 am on July 30, 2025:

You can also remove the now unused stuff:

0ci/lint_run.sh:# Only used in .cirrus.yml. Refer to test/lint/README.md on how to run locally.
1ci/lint_run_all.sh:# Only used in .cirrus.yml for stale re-runs of old pull request tasks. This

in .github/workflows/ci.yml:54 in 01746e60eb outdated

50@@ -52,6 +51,7 @@ jobs:
51     if: github.event_name == 'pull_request' && github.event.pull_request.commits != 1
52     timeout-minutes: 360  # Use maximum time, see https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#jobsjob_idtimeout-minutes. Assuming a worst case time of 1 hour per commit, this leads to a --max-count=6 below.
53     env:
54+      MAKEJOBS: '-j10'

maflcko commented at 7:39 am on July 30, 2025:

nit in 01746e60eb82b9bedb3afa1e791b5abce75ebf75: Unused? This task is using nproc?

willcl-ark commented at 9:56 am on July 30, 2025:

Correct, this is dropped in a81bf5c25a8

in .github/workflows/ci.yml:134 in 01746e60eb outdated

130@@ -131,6 +131,7 @@ jobs:
131     env:
132       DANGER_RUN_CI_ON_HOST: 1
133       BASE_ROOT_DIR: ${{ github.workspace }}
134+      MAKEJOBS: '-j10'

maflcko commented at 7:41 am on July 30, 2025:

nit in https://github.com/bitcoin/bitcoin/commit/01746e60eb82b9bedb3afa1e791b5abce75ebf75: Any reason to change this setting for the other tasks, but not for this one? For reference, the task has 3 mac-arm cpu threads.

willcl-ark commented at 9:57 am on July 30, 2025:

Also dropped in a81bf5c

in ci/README.md:79 in 4a1c5b0469 outdated

75+### Forked repositories
76+
77+When used in a fork the CI will run on GitHub's free hosted runners by default.
78+In this case, due to GitHub's 10GB-per-repo cache size limitations caches will be frequently evicted and missed, but the workflows will run (slowly).
79+
80+It is also possible to use your own Cirrus Runners in your own fork with an appropriate patch to the ../.github/workflows/ci.yml file and the actions in ../.github/actions/**/*.yml.

maflcko commented at 7:48 am on July 30, 2025:

nit in 4a1c5b04691266f1441fd966a4be65140f9a5cc4: Outdated doc?

willcl-ark commented at 9:57 am on July 30, 2025:

Thanks, updated the doc in c7c87775768

maflcko approved

maflcko commented at 8:01 am on July 30, 2025: member

everything looks good and this is probably ready for final review before merge? I just found some nits, which can be done along with follow-ups like #32989 (review). Also, the merge_base issue mentioned above can be addressed in a follow-up.

However, I am not familiar with the GitHub-style CI, including the (docker) caching, but the pull request motivation claims that many are familiar with this, so it should be easy to find reviewers for it, before merging it.

review ACK b6cf0b325793184e7824f7dc488a68f4908ed0c6 📦

Signature:

0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
2trusted comment: review ACK b6cf0b325793184e7824f7dc488a68f4908ed0c6 📦
34CWuGfE/k0O4gs/FcFN2Btaa/mALDAFjwg+EWCP2/wwWeM+xx/1ZmsAOtqxTrTh9V6NvX78Gw+wzGvFrDG65Dg==

DrahtBot requested review from fanquake on Jul 30, 2025

DrahtBot requested review from achow101 on Jul 30, 2025

DrahtBot requested review from 0xB10C on Jul 30, 2025

DrahtBot requested review from hebasto on Jul 30, 2025

willcl-ark force-pushed on Jul 30, 2025

in .github/workflows/ci.yml:133 in a81bf5c25a outdated

129@@ -131,6 +130,7 @@ jobs:
130     env:
131       DANGER_RUN_CI_ON_HOST: 1
132       BASE_ROOT_DIR: ${{ github.workspace }}
133+      CONTAINER_NAME: ${{ matrix.job-name }}

maflcko commented at 10:03 am on July 30, 2025:

nit in a81bf5c25a8dc0b3a2f8458380921b5026726d64: Looks like a unrelated and unneeded change?

willcl-ark commented at 10:08 am on July 30, 2025:

Oh, this has sneaked into the wrong commit!

It is “needed” by b49dd9890b9 to print the container name for these jobs when we have a low ccache hitrate warning in the CI annotations (bottom of the page on the run summary). Otherwise they are printed as:

“low ccache hitrate in ”

Will fix it up

maflcko commented at 10:11 am on July 30, 2025:

Ah, but then the yaml seems like the wrong place. Would be better to put in in the sh:

 0diff --git a/ci/test/00_setup_env_mac_native_fuzz.sh b/ci/test/00_setup_env_mac_native_fuzz.sh
 1index cacf2423ac..c79b48c8ed 100755
 2--- a/ci/test/00_setup_env_mac_native_fuzz.sh
 3+++ b/ci/test/00_setup_env_mac_native_fuzz.sh
 4@@ -6,6 +6,7 @@
 5 
 6 export LC_ALL=C.UTF-8
 7 
 8+bla...insert...here  # macos does not use a container, but the env var is needed for logging
 9 export CMAKE_GENERATOR="Ninja"
10 export BITCOIN_CONFIG="-DBUILD_FOR_FUZZING=ON"
11 export CI_OS_NAME="macos"

willcl-ark commented at 10:14 am on July 30, 2025:

But in this case we won’t get:

‘macOS 14 native, arm64, no depends, sqlite only, gui’
‘macOS 14 native, arm64, fuzz’

But will fix to a single name for both jobs (not a massive issue perhaps, it’s only an informational annotation?)

maflcko commented at 10:16 am on July 30, 2025:

Yes, container names must be unique over all configs and there are two macos configs (ci/test/00_setup_env_mac_native.sh) and (ci/test/00_setup_env_mac_native_fuzz.sh), so it should be possible. But just a nit. Anything is fine here.

willcl-ark commented at 2:32 pm on July 31, 2025:

Fixed in 1587a7baa96 ci: dynamically match makejobs with cores

in .github/workflows/ci.yml:22 in b6cf0b3257 outdated

18@@ -19,7 +19,8 @@ concurrency:
19 
20 env:
21   CI_FAILFAST_TEST_LEAVE_DANGLING: 1  # GHA does not care about dangling processes and setting this variable avoids killing the CI script itself on error
22-  MAKEJOBS: '-j10'
23+  CIRRUS_CACHE_HOST: http://127.0.0.1:12321/ # When using Cirrus Runners this host can be used as with the docker `gha` build cache type.

maflcko commented at 10:21 am on July 30, 2025:

from the llm:

used as with -> used with [“used as with” is redundant and makes the phrase unclear]

willcl-ark commented at 11:07 am on July 30, 2025:

Will get with the above if I re-touch

willcl-ark commented at 2:32 pm on July 31, 2025:

fixed in 7a33a8bea13 ci: add Cirrus cache host

DrahtBot renamed this:
~~Migrate CI to hosted Cirrus Runners~~
ci: Migrate CI to hosted Cirrus Runners
on Jul 30, 2025

DrahtBot added the label Tests on Jul 30, 2025

willcl-ark force-pushed on Jul 31, 2025

willcl-ark commented at 2:32 pm on July 31, 2025: member

Pushed to fix the merge conflict, and addressed nits

in .github/workflows/ci.yml:486 in fd5e807fca outdated

486+            cirrus-runner: 'ghcr.io/cirruslabs/ubuntu-runner-amd64:24.04-md'
487+            fallback-runner: 'ubuntu-24.04'
488+            timeout-minutes: 120
489+            file-env: './ci/test/00_setup_env_native_tidy.sh'
490+
491+          - name: 'TSan, depends, gui'

fanquake commented at 2:35 pm on July 31, 2025:

Should be no gui after #33099

willcl-ark commented at 10:49 am on August 4, 2025:

Removed in dc1613169db

fanquake commented at 2:47 pm on July 31, 2025: member

TSAN failure (https://github.com/bitcoin/bitcoin/actions/runs/16651908244/job/47126650397?pr=32989#step:8:6056):

  02025-07-31T14:39:40.396402Z [test] [net.cpp:2419] [void CConnman::SetTryNewOutboundPeer(bool)] [net] setting try another outbound peer=false
  12025-07-31T14:39:40.396438Z [test] [net.cpp:3192] [void==================
  2WARNING: ThreadSanitizer: data race (pid=11584)
  3  Write of size 8 at 0x7234000016f0 by main thread:
  4    [#0](/bitcoin-bitcoin/0/) blockmanager_tests::blockmanager_readblock_hash_mismatch::test_method() /home/admin/actions-runner/_work/_temp/build/src/test/./test/blockmanager_tests.cpp:143:28 (test_bitcoin+0x43dce3) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
  5    [#1](/bitcoin-bitcoin/1/) blockmanager_tests::blockmanager_readblock_hash_mismatch_invoker() /home/admin/actions-runner/_work/_temp/build/src/test/./test/blockmanager_tests.cpp:140:1 (test_bitcoin+0x43d6eb) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
  6    [#2](/bitcoin-bitcoin/2/) boost::detail::function::void_function_invoker<void (*)(), void>::invoke(boost::detail::function::function_buffer&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:59:11 (test_bitcoin+0x3275cd) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
  7    [#3](/bitcoin-bitcoin/3/) boost::function_n<void>::operator()() const /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:789:14 (test_bitcoin+0x2a2618) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
  8    [#4](/bitcoin-bitcoin/4/) boost::detail::forward::operator()() /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1416:32 (test_bitcoin+0x2a2618)
  9    [#5](/bitcoin-bitcoin/5/) boost::detail::function::function_obj_invoker<boost::detail::forward, int>::invoke(boost::detail::function::function_buffer&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:79:18 (test_bitcoin+0x2a2618)
 10    [#6](/bitcoin-bitcoin/6/) boost::function_n<int>::operator()() const /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:789:14 (test_bitcoin+0x226873) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 11    [#7](/bitcoin-bitcoin/7/) int boost::detail::do_invoke<boost::shared_ptr<boost::detail::translator_holder_base>, boost::function<int ()>>(boost::shared_ptr<boost::detail::translator_holder_base> const&, boost::function<int ()> const&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:329:30 (test_bitcoin+0x226873)
 12    [#8](/bitcoin-bitcoin/8/) boost::execution_monitor::catch_signals(boost::function<int ()> const&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:931:16 (test_bitcoin+0x226873)
 13    [#9](/bitcoin-bitcoin/9/) boost::execution_monitor::execute(boost::function<int ()> const&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1329:16 (test_bitcoin+0x226bea) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 14    [#10](/bitcoin-bitcoin/10/) boost::execution_monitor::vexecute(boost::function<void ()> const&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1425:5 (test_bitcoin+0x2222cc) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 15    [#11](/bitcoin-bitcoin/11/) boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned long) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/unit_test_monitor.ipp:49:9 (test_bitcoin+0x2222cc)
 16    [#12](/bitcoin-bitcoin/12/) boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:815:44 (test_bitcoin+0x255e8d) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 17    [#13](/bitcoin-bitcoin/13/) boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:784:58 (test_bitcoin+0x2563eb) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 18    [#14](/bitcoin-bitcoin/14/) boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:784:58 (test_bitcoin+0x2563eb) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 19    [#15](/bitcoin-bitcoin/15/) boost::unit_test::framework::run(unsigned long, bool) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:1722:29 (test_bitcoin+0x220dc1) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 20    [#16](/bitcoin-bitcoin/16/) boost::unit_test::unit_test_main(boost::unit_test::test_suite* (*)(int, char**), int, char**) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/unit_test_main.ipp:250:9 (test_bitcoin+0x23bd3c) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 21    [#17](/bitcoin-bitcoin/17/) main /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/unit_test_main.ipp:306:12 (test_bitcoin+0x23c5c3) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 22
 23  Previous read of size 8 at 0x7234000016f0 by thread T1:
 24    [#0](/bitcoin-bitcoin/0/) CBlockIndex::GetBlockHash() const /home/admin/actions-runner/_work/_temp/build/src/./chain.h:245:9 (test_bitcoin+0x13700f1) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 25    [#1](/bitcoin-bitcoin/1/) ValidationSignals::UpdatedBlockTip(CBlockIndex const*, CBlockIndex const*, bool)::$_0::operator()() const /home/admin/actions-runner/_work/_temp/build/src/./validationinterface.cpp:179:5 (test_bitcoin+0x13700f1)
 26    [#2](/bitcoin-bitcoin/2/) decltype(std::declval<ValidationSignals::UpdatedBlockTip(CBlockIndex const*, CBlockIndex const*, bool)::$_0&>()()) std::__1::__invoke[abi:de200100]<ValidationSignals::UpdatedBlockTip(CBlockIndex const*, CBlockIndex const*, bool)::$_0&>(ValidationSignals::UpdatedBlockTip(CBlockIndex const*, CBlockIndex const*, bool)::$_0&) /cxx_build/include/c++/v1/__type_traits/invoke.h:179:25 (test_bitcoin+0x13700f1)
 27    [#3](/bitcoin-bitcoin/3/) void std::__1::__invoke_void_return_wrapper<void, true>::__call[abi:de200100]<ValidationSignals::UpdatedBlockTip(CBlockIndex const*, CBlockIndex const*, bool)::$_0&>(ValidationSignals::UpdatedBlockTip(CBlockIndex const*, CBlockIndex const*, bool)::$_0&) /cxx_build/include/c++/v1/__type_traits/invoke.h:251:5 (test_bitcoin+0x13700f1)
 28    [#4](/bitcoin-bitcoin/4/) void std::__1::__invoke_r[abi:de200100]<void, ValidationSignals::UpdatedBlockTip(CBlockIndex const*, CBlockIndex const*, bool)::$_0&>(ValidationSignals::UpdatedBlockTip(CBlockIndex const*, CBlockIndex const*, bool)::$_0&) /cxx_build/include/c++/v1/__type_traits/invoke.h:273:10 (test_bitcoin+0x13700f1)
 29    [#5](/bitcoin-bitcoin/5/) std::__1::__function::__alloc_func<ValidationSignals::UpdatedBlockTip(CBlockIndex const*, CBlockIndex const*, bool)::$_0, std::__1::allocator<ValidationSignals::UpdatedBlockTip(CBlockIndex const*, CBlockIndex const*, bool)::$_0>, void ()>::operator()[abi:de200100]() /cxx_build/include/c++/v1/__functional/function.h:167:12 (test_bitcoin+0x13700f1)
 30    [#6](/bitcoin-bitcoin/6/) std::__1::__function::__func<ValidationSignals::UpdatedBlockTip(CBlockIndex const*, CBlockIndex const*, bool)::$_0, std::__1::allocator<ValidationSignals::UpdatedBlockTip(CBlockIndex const*, CBlockIndex const*, bool)::$_0>, void ()>::operator()() /cxx_build/include/c++/v1/__functional/function.h:319:10 (test_bitcoin+0x13700f1)
 31    [#7](/bitcoin-bitcoin/7/) std::__1::__function::__value_func<void ()>::operator()[abi:de200100]() const /cxx_build/include/c++/v1/__functional/function.h:436:12 (test_bitcoin+0x183b982) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 32    [#8](/bitcoin-bitcoin/8/) std::__1::function<void ()>::operator()() const /cxx_build/include/c++/v1/__functional/function.h:995:10 (test_bitcoin+0x183b982)
 33    [#9](/bitcoin-bitcoin/9/) SerialTaskRunner::ProcessQueue() /home/admin/actions-runner/_work/_temp/build/src/./scheduler.cpp:173:5 (test_bitcoin+0x183b982)
 34    [#10](/bitcoin-bitcoin/10/) SerialTaskRunner::MaybeScheduleProcessQueue()::$_0::operator()() const /home/admin/actions-runner/_work/_temp/build/src/./scheduler.cpp:142:41 (test_bitcoin+0x183d4e5) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 35    [#11](/bitcoin-bitcoin/11/) decltype(std::declval<SerialTaskRunner::MaybeScheduleProcessQueue()::$_0&>()()) std::__1::__invoke[abi:de200100]<SerialTaskRunner::MaybeScheduleProcessQueue()::$_0&>(SerialTaskRunner::MaybeScheduleProcessQueue()::$_0&) /cxx_build/include/c++/v1/__type_traits/invoke.h:179:25 (test_bitcoin+0x183d4e5)
 36    [#12](/bitcoin-bitcoin/12/) void std::__1::__invoke_void_return_wrapper<void, true>::__call[abi:de200100]<SerialTaskRunner::MaybeScheduleProcessQueue()::$_0&>(SerialTaskRunner::MaybeScheduleProcessQueue()::$_0&) /cxx_build/include/c++/v1/__type_traits/invoke.h:251:5 (test_bitcoin+0x183d4e5)
 37    [#13](/bitcoin-bitcoin/13/) void std::__1::__invoke_r[abi:de200100]<void, SerialTaskRunner::MaybeScheduleProcessQueue()::$_0&>(SerialTaskRunner::MaybeScheduleProcessQueue()::$_0&) /cxx_build/include/c++/v1/__type_traits/invoke.h:273:10 (test_bitcoin+0x183d4e5)
 38    [#14](/bitcoin-bitcoin/14/) std::__1::__function::__alloc_func<SerialTaskRunner::MaybeScheduleProcessQueue()::$_0, std::__1::allocator<SerialTaskRunner::MaybeScheduleProcessQueue()::$_0>, void ()>::operator()[abi:de200100]() /cxx_build/include/c++/v1/__functional/function.h:167:12 (test_bitcoin+0x183d4e5)
 39    [#15](/bitcoin-bitcoin/15/) std::__1::__function::__func<SerialTaskRunner::MaybeScheduleProcessQueue()::$_0, std::__1::allocator<SerialTaskRunner::MaybeScheduleProcessQueue()::$_0>, void ()>::operator()() /cxx_build/include/c++/v1/__functional/function.h:319:10 (test_bitcoin+0x183d4e5)
 40    [#16](/bitcoin-bitcoin/16/) std::__1::__function::__value_func<void ()>::operator()[abi:de200100]() const /cxx_build/include/c++/v1/__functional/function.h:436:12 (test_bitcoin+0x183a8ef) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 41    [#17](/bitcoin-bitcoin/17/) std::__1::function<void ()>::operator()() const /cxx_build/include/c++/v1/__functional/function.h:995:10 (test_bitcoin+0x183a8ef)
 42    [#18](/bitcoin-bitcoin/18/) CScheduler::serviceQueue() /home/admin/actions-runner/_work/_temp/build/src/./scheduler.cpp:60:17 (test_bitcoin+0x183a8ef)
 43    [#19](/bitcoin-bitcoin/19/) ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2::operator()() const /home/admin/actions-runner/_work/_temp/build/src/test/util/./test/util/setup_common.cpp:229:114 (test_bitcoin+0xe5c228) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 44    [#20](/bitcoin-bitcoin/20/) decltype(std::declval<ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2&>()()) std::__1::__invoke[abi:de200100]<ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2&>(ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2&) /cxx_build/include/c++/v1/__type_traits/invoke.h:179:25 (test_bitcoin+0xe5c228)
 45    [#21](/bitcoin-bitcoin/21/) void std::__1::__invoke_void_return_wrapper<void, true>::__call[abi:de200100]<ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2&>(ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2&) /cxx_build/include/c++/v1/__type_traits/invoke.h:251:5 (test_bitcoin+0xe5c228)
 46    [#22](/bitcoin-bitcoin/22/) void std::__1::__invoke_r[abi:de200100]<void, ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2&>(ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2&) /cxx_build/include/c++/v1/__type_traits/invoke.h:273:10 (test_bitcoin+0xe5c228)
 47    [#23](/bitcoin-bitcoin/23/) std::__1::__function::__alloc_func<ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2, std::__1::allocator<ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2>, void ()>::operator()[abi:de200100]() /cxx_build/include/c++/v1/__functional/function.h:167:12 (test_bitcoin+0xe5c228)
 48    [#24](/bitcoin-bitcoin/24/) std::__1::__function::__func<ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2, std::__1::allocator<ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2>, void ()>::operator()() /cxx_build/include/c++/v1/__functional/function.h:319:10 (test_bitcoin+0xe5c228)
 49    [#25](/bitcoin-bitcoin/25/) std::__1::__function::__value_func<void ()>::operator()[abi:de200100]() const /cxx_build/include/c++/v1/__functional/function.h:436:12 (test_bitcoin+0x1916058) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 50    [#26](/bitcoin-bitcoin/26/) std::__1::function<void ()>::operator()() const /cxx_build/include/c++/v1/__functional/function.h:995:10 (test_bitcoin+0x1916058)
 51    [#27](/bitcoin-bitcoin/27/) util::TraceThread(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>) /home/admin/actions-runner/_work/_temp/build/src/util/./util/thread.cpp:21:9 (test_bitcoin+0x1916058)
 52    [#28](/bitcoin-bitcoin/28/) decltype(std::declval<void (*)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>)>()(std::declval<char const*>(), std::declval<ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2>())) std::__1::__invoke[abi:de200100]<void (*)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), char const*, ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2>(void (*&&)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), char const*&&, ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2&&) /cxx_build/include/c++/v1/__type_traits/invoke.h:179:25 (test_bitcoin+0xe5bddb) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 53    [#29](/bitcoin-bitcoin/29/) void std::__1::__thread_execute[abi:de200100]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (*)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), char const*, ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2, 2ul, 3ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (*)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), char const*, ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2>&, std::__1::__tuple_indices<2ul, 3ul>) /cxx_build/include/c++/v1/__thread/thread.h:199:3 (test_bitcoin+0xe5bddb)
 54    [#30](/bitcoin-bitcoin/30/) void* std::__1::__thread_proxy[abi:de200100]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (*)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), char const*, ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2>>(void*) /cxx_build/include/c++/v1/__thread/thread.h:208:3 (test_bitcoin+0xe5bddb)
 55
 56  Location is heap block of size 200 at 0x7234000016c0 allocated by main thread:
 57    [#0](/bitcoin-bitcoin/0/) operator new(unsigned long) <null> (test_bitcoin+0x20a6e6) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 58    [#1](/bitcoin-bitcoin/1/) void* std::__1::__libcpp_operator_new[abi:de200100]<unsigned long>(unsigned long) /cxx_build/include/c++/v1/__new/allocate.h:37:10 (test_bitcoin+0xfe0560) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 59    [#2](/bitcoin-bitcoin/2/) std::__1::__hash_node<std::__1::__hash_value_type<uint256, CBlockIndex>, void*>* std::__1::__libcpp_allocate[abi:de200100]<std::__1::__hash_node<std::__1::__hash_value_type<uint256, CBlockIndex>, void*>>(std::__1::__element_count, unsigned long) /cxx_build/include/c++/v1/__new/allocate.h:64:28 (test_bitcoin+0xfe0560)
 60    [#3](/bitcoin-bitcoin/3/) std::__1::allocator<std::__1::__hash_node<std::__1::__hash_value_type<uint256, CBlockIndex>, void*>>::allocate[abi:de200100](unsigned long) /cxx_build/include/c++/v1/__memory/allocator.h:105:14 (test_bitcoin+0xfe0560)
 61    [#4](/bitcoin-bitcoin/4/) std::__1::allocator_traits<std::__1::allocator<std::__1::__hash_node<std::__1::__hash_value_type<uint256, CBlockIndex>, void*>>>::allocate[abi:de200100](std::__1::allocator<std::__1::__hash_node<std::__1::__hash_value_type<uint256, CBlockIndex>, void*>>&, unsigned long) /cxx_build/include/c++/v1/__memory/allocator_traits.h:270:16 (test_bitcoin+0xfe0560)
 62    [#5](/bitcoin-bitcoin/5/) std::__1::unique_ptr<std::__1::__hash_node<std::__1::__hash_value_type<uint256, CBlockIndex>, void*>, std::__1::__hash_node_destructor<std::__1::allocator<std::__1::__hash_node<std::__1::__hash_value_type<uint256, CBlockIndex>, void*>>>> std::__1::__hash_table<std::__1::__hash_value_type<uint256, CBlockIndex>, std::__1::__unordered_map_hasher<uint256, std::__1::__hash_value_type<uint256, CBlockIndex>, BlockHasher, std::__1::equal_to<uint256>, true>, std::__1::__unordered_map_equal<uint256, std::__1::__hash_value_type<uint256, CBlockIndex>, std::__1::equal_to<uint256>, BlockHasher, true>, std::__1::allocator<std::__1::__hash_value_type<uint256, CBlockIndex>>>::__construct_node_hash<std::__1::piecewise_construct_t const&, std::__1::tuple<uint256&&>, std::__1::tuple<CBlockHeader const&>>(unsigned long, std::__1::piecewise_construct_t const&, std::__1::tuple<uint256&&>&&, std::__1::tuple<CBlockHeader const&>&&) /cxx_build/include/c++/v1/__hash_table:1840:21 (test_bitcoin+0xfe0560)
 63    [#6](/bitcoin-bitcoin/6/) std::__1::pair<std::__1::__hash_iterator<std::__1::__hash_node<std::__1::__hash_value_type<uint256, CBlockIndex>, void*>*>, bool> std::__1::__hash_table<std::__1::__hash_value_type<uint256, CBlockIndex>, std::__1::__unordered_map_hasher<uint256, std::__1::__hash_value_type<uint256, CBlockIndex>, BlockHasher, std::__1::equal_to<uint256>, true>, std::__1::__unordered_map_equal<uint256, std::__1::__hash_value_type<uint256, CBlockIndex>, std::__1::equal_to<uint256>, BlockHasher, true>, std::__1::allocator<std::__1::__hash_value_type<uint256, CBlockIndex>>>::__emplace_unique_key_args<uint256, std::__1::piecewise_construct_t const&, std::__1::tuple<uint256&&>, std::__1::tuple<CBlockHeader const&>>(uint256 const&, std::__1::piecewise_construct_t const&, std::__1::tuple<uint256&&>&&, std::__1::tuple<CBlockHeader const&>&&) /cxx_build/include/c++/v1/__hash_table:1547:25 (test_bitcoin+0xfe0560)
 64    [#7](/bitcoin-bitcoin/7/) std::__1::pair<std::__1::__hash_map_iterator<std::__1::__hash_iterator<std::__1::__hash_node<std::__1::__hash_value_type<uint256, CBlockIndex>, void*>*>>, bool> std::__1::unordered_map<uint256, CBlockIndex, BlockHasher, std::__1::equal_to<uint256>, std::__1::allocator<std::__1::pair<uint256 const, CBlockIndex>>>::try_emplace[abi:de200100]<CBlockHeader const&>(uint256&&, CBlockHeader const&) /cxx_build/include/c++/v1/unordered_map:1288:21 (test_bitcoin+0xfc9637) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 65    [#8](/bitcoin-bitcoin/8/) node::BlockManager::AddToBlockIndex(CBlockHeader const&, CBlockIndex*&) /home/admin/actions-runner/_work/_temp/build/src/./node/blockstorage.cpp:211:41 (test_bitcoin+0xfc9637)
 66    [#9](/bitcoin-bitcoin/9/) Chainstate::LoadGenesisBlock() /home/admin/actions-runner/_work/_temp/build/src/./validation.cpp:5070:42 (test_bitcoin+0x1334783) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 67    [#10](/bitcoin-bitcoin/10/) node::CompleteChainstateInitialization(ChainstateManager&, node::ChainstateLoadOptions const&) /home/admin/actions-runner/_work/_temp/build/src/./node/chainstate.cpp:65:82 (test_bitcoin+0xfe9b61) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 68    [#11](/bitcoin-bitcoin/11/) node::LoadChainstate(ChainstateManager&, kernel::CacheSizes const&, node::ChainstateLoadOptions const&) /home/admin/actions-runner/_work/_temp/build/src/./node/chainstate.cpp:181:38 (test_bitcoin+0xfe9232) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 69    [#12](/bitcoin-bitcoin/12/) ChainTestingSetup::LoadVerifyActivateChainstate() /home/admin/actions-runner/_work/_temp/build/src/test/util/./test/util/setup_common.cpp:308:28 (test_bitcoin+0xe53a70) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 70    [#13](/bitcoin-bitcoin/13/) TestingSetup::TestingSetup(ChainType, TestOpts) /home/admin/actions-runner/_work/_temp/build/src/test/util/./test/util/setup_common.cpp:331:5 (test_bitcoin+0xe540f7) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 71    [#14](/bitcoin-bitcoin/14/) blockmanager_tests::blockmanager_readblock_hash_mismatch::blockmanager_readblock_hash_mismatch() /home/admin/actions-runner/_work/_temp/build/src/test/./test/blockmanager_tests.cpp:140:1 (test_bitcoin+0x43d30a) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 72    [#15](/bitcoin-bitcoin/15/) blockmanager_tests::blockmanager_readblock_hash_mismatch_invoker() /home/admin/actions-runner/_work/_temp/build/src/test/./test/blockmanager_tests.cpp:140:1 (test_bitcoin+0x43d30a)
 73    [#16](/bitcoin-bitcoin/16/) boost::detail::function::void_function_invoker<void (*)(), void>::invoke(boost::detail::function::function_buffer&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:59:11 (test_bitcoin+0x3275cd) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 74    [#17](/bitcoin-bitcoin/17/) boost::function_n<void>::operator()() const /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:789:14 (test_bitcoin+0x2a2618) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 75    [#18](/bitcoin-bitcoin/18/) boost::detail::forward::operator()() /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1416:32 (test_bitcoin+0x2a2618)
 76    [#19](/bitcoin-bitcoin/19/) boost::detail::function::function_obj_invoker<boost::detail::forward, int>::invoke(boost::detail::function::function_buffer&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:79:18 (test_bitcoin+0x2a2618)
 77    [#20](/bitcoin-bitcoin/20/) boost::function_n<int>::operator()() const /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:789:14 (test_bitcoin+0x226873) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 78    [#21](/bitcoin-bitcoin/21/) int boost::detail::do_invoke<boost::shared_ptr<boost::detail::translator_holder_base>, boost::function<int ()>>(boost::shared_ptr<boost::detail::translator_holder_base> const&, boost::function<int ()> const&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:329:30 (test_bitcoin+0x226873)
 79    [#22](/bitcoin-bitcoin/22/) boost::execution_monitor::catch_signals(boost::function<int ()> const&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:931:16 (test_bitcoin+0x226873)
 80    [#23](/bitcoin-bitcoin/23/) boost::execution_monitor::execute(boost::function<int ()> const&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1329:16 (test_bitcoin+0x226bea) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 81    [#24](/bitcoin-bitcoin/24/) boost::execution_monitor::vexecute(boost::function<void ()> const&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1425:5 (test_bitcoin+0x2222cc) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 82    [#25](/bitcoin-bitcoin/25/) boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned long) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/unit_test_monitor.ipp:49:9 (test_bitcoin+0x2222cc)
 83    [#26](/bitcoin-bitcoin/26/) boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:815:44 (test_bitcoin+0x255e8d) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 84    [#27](/bitcoin-bitcoin/27/) boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:784:58 (test_bitcoin+0x2563eb) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 85    [#28](/bitcoin-bitcoin/28/) boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:784:58 (test_bitcoin+0x2563eb) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 86    [#29](/bitcoin-bitcoin/29/) boost::unit_test::framework::run(unsigned long, bool) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:1722:29 (test_bitcoin+0x220dc1) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 87    [#30](/bitcoin-bitcoin/30/) boost::unit_test::unit_test_main(boost::unit_test::test_suite* (*)(int, char**), int, char**) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/unit_test_main.ipp:250:9 (test_bitcoin+0x23bd3c) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 88    [#31](/bitcoin-bitcoin/31/) main /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/unit_test_main.ipp:306:12 (test_bitcoin+0x23c5c3) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 89
 90  Thread T1 'b-scheduler' (tid=11645, running) created by main thread at:
 91    [#0](/bitcoin-bitcoin/0/) pthread_create <null> (test_bitcoin+0x18677e) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 92    [#1](/bitcoin-bitcoin/1/) std::__1::__libcpp_thread_create[abi:de200100](unsigned long*, void* (*)(void*), void*) /cxx_build/include/c++/v1/__thread/support/pthread.h:182:10 (test_bitcoin+0xe52b07) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 93    [#2](/bitcoin-bitcoin/2/) std::__1::thread::thread<void (&)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), char const (&) [10], ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2, 0>(void (&)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), char const (&) [10], ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2&&) /cxx_build/include/c++/v1/__thread/thread.h:218:14 (test_bitcoin+0xe52b07)
 94    [#3](/bitcoin-bitcoin/3/) ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts) /home/admin/actions-runner/_work/_temp/build/src/test/util/./test/util/setup_common.cpp:229:46 (test_bitcoin+0xe52b07)
 95    [#4](/bitcoin-bitcoin/4/) TestingSetup::TestingSetup(ChainType, TestOpts) /home/admin/actions-runner/_work/_temp/build/src/test/util/./test/util/setup_common.cpp:323:7 (test_bitcoin+0xe5400f) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 96    [#5](/bitcoin-bitcoin/5/) blockmanager_tests::blockmanager_readblock_hash_mismatch::blockmanager_readblock_hash_mismatch() /home/admin/actions-runner/_work/_temp/build/src/test/./test/blockmanager_tests.cpp:140:1 (test_bitcoin+0x43d30a) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 97    [#6](/bitcoin-bitcoin/6/) blockmanager_tests::blockmanager_readblock_hash_mismatch_invoker() /home/admin/actions-runner/_work/_temp/build/src/test/./test/blockmanager_tests.cpp:140:1 (test_bitcoin+0x43d30a)
 98    [#7](/bitcoin-bitcoin/7/) boost::detail::function::void_function_invoker<void (*)(), void>::invoke(boost::detail::function::function_buffer&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:59:11 (test_bitcoin+0x3275cd) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
 99    [#8](/bitcoin-bitcoin/8/) boost::function_n<void>::operator()() const /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:789:14 (test_bitcoin+0x2a2618) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
100    [#9](/bitcoin-bitcoin/9/) boost::detail::forward::operator()() /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1416:32 (test_bitcoin+0x2a2618)
101    [#10](/bitcoin-bitcoin/10/) boost::detail::function::function_obj_invoker<boost::detail::forward, int>::invoke(boost::detail::function::function_buffer&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:79:18 (test_bitcoin+0x2a2618)
102    [#11](/bitcoin-bitcoin/11/) boost::function_n<int>::operator()() const /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:789:14 (test_bitcoin+0x226873) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
103    [#12](/bitcoin-bitcoin/12/) int boost::detail::do_invoke<boost::shared_ptr<boost::detail::translator_holder_base>, boost::function<int ()>>(boost::shared_ptr<boost::detail::translator_holder_base> const&, boost::function<int ()> const&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:329:30 (test_bitcoin+0x226873)
104    [#13](/bitcoin-bitcoin/13/) boost::execution_monitor::catch_signals(boost::function<int ()> const&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:931:16 (test_bitcoin+0x226873)
105    [#14](/bitcoin-bitcoin/14/) boost::execution_monitor::execute(boost::function<int ()> const&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1329:16 (test_bitcoin+0x226bea) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
106    [#15](/bitcoin-bitcoin/15/) boost::execution_monitor::vexecute(boost::function<void ()> const&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1425:5 (test_bitcoin+0x2222cc) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
107    [#16](/bitcoin-bitcoin/16/) boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned long) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/unit_test_monitor.ipp:49:9 (test_bitcoin+0x2222cc)
108    [#17](/bitcoin-bitcoin/17/) boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:815:44 (test_bitcoin+0x255e8d) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
109    [#18](/bitcoin-bitcoin/18/) boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:784:58 (test_bitcoin+0x2563eb) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
110    [#19](/bitcoin-bitcoin/19/) boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:784:58 (test_bitcoin+0x2563eb) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
111    [#20](/bitcoin-bitcoin/20/) boost::unit_test::framework::run(unsigned long, bool) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:1722:29 (test_bitcoin+0x220dc1) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
112    [#21](/bitcoin-bitcoin/21/) boost::unit_test::unit_test_main(boost::unit_test::test_suite* (*)(int, char**), int, char**) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/unit_test_main.ipp:250:9 (test_bitcoin+0x23bd3c) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
113    [#22](/bitcoin-bitcoin/22/) main /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/unit_test_main.ipp:306:12 (test_bitcoin+0x23c5c3) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
114
115SUMMARY: ThreadSanitizer: data race /home/admin/actions-runner/_work/_temp/build/src/test/./test/blockmanager_tests.cpp:143:28 in blockmanager_tests::blockmanager_readblock_hash_mismatch::test_method()

fanquake commented at 4:06 pm on July 31, 2025: member

TSAN failure

Didn’t fail with a re-run: https://github.com/bitcoin/bitcoin/actions/runs/16651908244/job/47133023619.

willcl-ark force-pushed on Aug 4, 2025

willcl-ark commented at 10:50 am on August 4, 2025: member

Pushed another run with caches saving to re-test cache hit/restore later.

willcl-ark force-pushed on Aug 4, 2025

m3dwards force-pushed on Aug 5, 2025

willcl-ark force-pushed on Aug 6, 2025

willcl-ark commented at 1:28 pm on August 6, 2025: member

Force pushed to pick up changes in master (particularly #33002).

Other notable changes:

Added a remote ccache host in 2adbe2caec. Previously we used actions/cache blobs for ccache. This works ~ ok, but Cirrus restores partial matches in reverse-age order (oldest first), which was resulting in bad cache hitrates for ccache when multiple caches existed for a job (say, “old” and “new”). The HTTP ccache backend is used on Cirrus infra and shared between all jobs. This requires using docker -host networking as we are using ccache inside the container. Visible (all misses on first run) here in logs
Set TEST_RUNNER_PORT_MIN in d57b291736 to avoid conflicting with the above remote ccache host
Reduced ccache hitrate warning level to 75% in 0e768d18bd

willcl-ark force-pushed on Aug 6, 2025

fanquake commented at 3:57 pm on August 6, 2025: member

TSAN failure, same as the one above (https://github.com/bitcoin/bitcoin/actions/runs/16781531680/job/47521224560?pr=32989#step:8:5924):

0 2025-08-06T15:41:21.507392Z [test] [net.cpp:3192] [void==================
1WARNING: ThreadSanitizer: data race (pid=10235)
2  Write of size 8 at 0x7234000016f0 by main thread:
3    [#0](/bitcoin-bitcoin/0/) blockmanager_tests::blockmanager_readblock_hash_mismatch::test_method() /home/admin/actions-runner/_work/_temp/build/src/test/./test/blockmanager_tests.cpp:143:28 (test_bitcoin+0x43dce3) (BuildId: 93c9590d6a842e462ee1d55a9ba83c69fb598193)
4    [#1](/bitcoin-bitcoin/1/) blockmanager_tests::blockmanager_readblock_hash_mismatch_invoker() /home/admin/actions-runner/_work/_temp/build/src/test/./test/blockmanager_tests.cpp:140:1 (test_bitcoin+0x43d6eb) (BuildId: 93c9590d6a842e462ee1d55a9ba83c69fb598193)

willcl-ark force-pushed on Aug 7, 2025

in .github/actions/restore-caches/action.yml:18 in 64d38b8e81 outdated

13+    description: 'A boolean value to indicate an exact match was found for depends built'
14+runs:
15+  using: 'composite'
16+  steps:
17+    - name: Restore Ccache cache from blob
18+      if: inputs.use-cirrus != 'true'

maflcko commented at 8:36 am on August 7, 2025:

64d38b8e81d4fbbfac02f4e3e734d8ecd61dd1dc: i don’t understand this. This reads as if the cache is only restored when not run on cirrus, which seems odd and unwanted?

edit: Oh, I see it is due to #32989 (comment)

willcl-ark commented at 2:04 pm on August 7, 2025:

Correct, I should add a comment perhaps, but choosing only a single cache type (blob storage or http endpoint) was done becuase I think we are just otherwise doubling the space the ccache storage will take per job:

1 x storage local blob 1x storage http endpoint

Both will store the same cached compilation, and both will use cache quota.

As the restore policy of the blob storage is currently backwards (IMO), use only http endpoint for ccache, and let ccache handle LRU evictions itself.

maflcko commented at 2:45 pm on August 7, 2025:

how do evictions happen, if export CCACHE_MAXSIZE=${CCACHE_MAXSIZE:-500M} is set to 500M in total for all tasks? It seems they’d happen too much and too early?

willcl-ark commented at 3:12 pm on August 7, 2025:

Good question; remote backends are not governed by client-side CCACHE_MAXSIZE settings, see https://ccache.dev/manual/4.11.3.html#_http_storage_backend for a bit more info.

Basically the onus is on the server to periodically run ccache --trim-dir to keep it under control (we have asked for more details on this process cc @m3dwards )

maflcko commented at 3:20 pm on August 7, 2025:

Would it not be easier for Cirrus to just fix the reverse-age order to pick the latest instead? Or somehow work around this by embedding the epoch seconds in the ccache blob name?

willcl-ark commented at 8:59 pm on August 7, 2025:

Yes I agree, the blob cache makes the action simpler, and the whole process easier to reason about (and doesnt require a carve out for the fallback if not using cirrus runners). But without it restoring newest first we are going to see worse cache hit-rates than with this approach.

Perhaps @fkorotkov could clarify here if oldest match is correct, and if there is a way to restore newest first like (as I understand) GHA does natively? My understanding, which may be incorrect, is that epoch seconds would not help (its analagous to monotonically-increasing run_id), and on the wildcard Cirrus will still restore oldest match.

fkorotkov commented at 9:44 pm on August 7, 2025:

I actually email @m3dwards couple hours ago that we changed behaviour:

“restore-keys” now should return the newest entry within the last 1000 cache entries in lexicographical order. There might be an edge case where one can accumulate more than 1000 keys before the eviction happens. We added some metrics to track this case and thought of lazy evicting entries based on some heuristic.

So it should now work as expected and return the freshest entity among first 1000 in lexicographical order (hence this S3 API quirk).

willcl-ark commented at 8:27 am on August 8, 2025:

Great, thanks Fedor!

in .github/actions/restore-caches/action.yml:30 in 64d38b8e81 outdated

25+          ccache-${{ env.CONTAINER_NAME }}-
26+
27+    - name: Configure remote ccache http host
28+      if: inputs.use-cirrus == 'true'
29+      shell: bash
30+      run: echo "CCACHE_REMOTE_STORAGE=http://$CIRRUS_HTTP_CACHE_HOST" >> $GITHUB_ENV

maflcko commented at 8:38 am on August 7, 2025:

https://github.com/bitcoin/bitcoin/commit/64d38b8e81d4fbbfac02f4e3e734d8ecd61dd1dc: why is this needed? Shouldn’t cirrus take care of this itself in its own action?

edit: Oh, I see it is due to #32989 (comment)

in ci/test/00_setup_env_mac_native.sh:11 in d8aa07772f outdated

 7@@ -8,6 +8,7 @@ export LC_ALL=C.UTF-8
 8 
 9 # Homebrew's python@3.12 is marked as externally managed (PEP 668).
10 # Therefore, `--break-system-packages` is needed.
11+export CONTAINER_NAME="macOS 14 native, arm64, no depends, sqlite only, gui"  # macos does not use a container, but the env var is needed for logging

maflcko commented at 8:59 am on August 7, 2025:

nit in 993e59ca0c: Would be good to use naming consistent with all other CONTAINER_NAME?

For ci/test/00_setup_env_mac_native.sh:

export CONTAINER_NAME=ci_mac_native

For ci/test/00_setup_env_mac_native_fuzz.sh:

export CONTAINER_NAME=ci_mac_native_fuzz

maflcko commented at 9:23 am on August 7, 2025: member

There are still some leftover files, which could be deleted: #32989 (review)

Also, I am not familiar with the github-cirrus-ccache-remote caching, so I can’t review that one.

review ACK 36d3042512ed9e638b413866b5a36eec48895bcc 🐬

Signature:

0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
2trusted comment: review ACK 36d3042512ed9e638b413866b5a36eec48895bcc 🐬
34yyXnlBx7f31chJrv9654xosAKktz56Z9xXrCrKYw1aCnbWlBuJSiszVRRFAOduhG1Icnok9Jit9L+RiraxkDg==

DrahtBot requested review from fanquake on Aug 7, 2025

in ci/test/02_run_container.sh:89 in 36d3042512 outdated

84@@ -114,6 +85,17 @@ if [ -z "$DANGER_RUN_CI_ON_HOST" ]; then
85   # When detecting podman-docker, `--external` should be added.
86   docker image prune --force --filter "label=$CI_IMAGE_LABEL"
87 
88+  # Check if CCACHE_REMOTE_STORAGE is set and configure container networking
89+  CCACHE_EXTRA_ARGS=""

maflcko commented at 10:15 am on August 7, 2025:

unused variable?

willcl-ark force-pushed on Aug 8, 2025

maflcko commented at 3:32 pm on August 8, 2025: member

not sure why you dropped bbdf75db37? c18b0939da also seems fine to keep, but can be cherry-picked in a follow-up.

review ACK 739aef1023 🥐

Signature:

0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
2trusted comment: review ACK 739aef1023 🥐
3yatePmaUEq1mRxa3IrcM8HaRLs7UFnMq63a1gs2AjBktsmSyDRQZ7HrlwCV+HQvukziiZ8Ghb7nZxf/8z0YgCw==

willcl-ark commented at 4:12 pm on August 8, 2025: member

review ACK 739aef1 🥐 Show signature

Thank you so much for the (re)-review on this one @maflcko!

I was waiting for CI to go green again after a push-with-forced-cache-save and re-push-without before explaining:

not sure why you dropped bbdf75d?

This was added specifically to accomodate the http-based ccache endpoint (port 12345), so dropped along with that change. Currently it defaults to 11000 which seems to not conflict with anything currently:

0❯ rg TEST_RUNNER_PORT_MIN
1test/functional/test_framework/util.py
2354:PORT_MIN = int(os.getenv('TEST_RUNNER_PORT_MIN', default=11000))

Is there a reason you’d prefer to have this added back in; just to be more explicit?

c18b093 also seems fine to keep, but can be cherry-picked in a follow-up.

Despite this gaining quite some decent-enough runtime performance, I’d prefer to add this in a followup after having gotten TSAN failures twice in this PR. @fanquake also noted that he saw some issues when using (2 * nproc) in local runs, so I think safer to leave for now.

in .github/actions/restore-caches/action.yml:43 in 739aef1023 outdated

38+    - name: Restore previous releases cache
39+      id: previous-releases
40+      uses: cirruslabs/cache/restore@v4
41+      with:
42+        path: ${{ env.PREVIOUS_RELEASES_DIR }}
43+        key: previous-releases-${{ env.CONTAINER_NAME }}-${{ hashFiles('./test/get_previous_releases.py') }}

m3dwards commented at 5:29 pm on August 13, 2025:

nit: perhaps hash of get_previous_releases.py should be an environment variable set in configure environment just like DEPENDS_HASH is so that it’s not repeated in save/restore cache files.

willcl-ark commented at 8:58 am on August 14, 2025:

Added in 5d652ca1ef9 and used in b57bb7c0fbd

in .github/actions/restore-caches/action.yml:52 in 739aef1023 outdated

47+    - name: export cache hit
48+      id: export-cache-hit
49+      shell: bash
50+      run: |
51+        echo "depends-sources-cache-hit=${{ steps.depends-sources.outputs.cache-hit }}" >> $GITHUB_OUTPUT
52+        echo "depends-built-cache-hit=${{ steps.depends-built.outputs.cache-hit }}" >> $GITHUB_OUTPUT

m3dwards commented at 5:38 pm on August 13, 2025:

Is this missing outputting previous-releases-cache-hit?

willcl-ark commented at 8:58 am on August 14, 2025:

Added debug output in b57bb7c0fbd

in .github/actions/restore-caches/action.yml:7 in 739aef1023 outdated

0@@ -0,0 +1,52 @@
1+name: 'Restore Caches'
2+description: 'Restore ccache, depends sources, and built depends caches'
3+outputs:
4+  depends-sources-cache-hit:
5+    description: 'A boolean value to indicate an exact match was found for depends sources'
6+  depends-built-cache-hit:
7+    description: 'A boolean value to indicate an exact match was found for depends built'

m3dwards commented at 5:39 pm on August 13, 2025:

Also missing previous-releases-cache-hit?

willcl-ark commented at 9:00 am on August 14, 2025:

Reworked cache inputs/outputs in b57bb7c0fbd to use env vars.

Inputs and outputs are more explicit but only really needed when passing info between jobs. For us, env vars work fine and let us remove the inputs from the main ci.yml file.

in .github/actions/configure-environment/action.yml:27 in 39e9c27679 outdated

22+
23+    - name: Get container name
24+      shell: bash
25+      run: |
26+        source $FILE_ENV
27+        echo "CONTAINER_NAME=$CONTAINER_NAME" >> "$GITHUB_ENV"

m3dwards commented at 6:15 pm on August 13, 2025:

potential tiny nit: Are we using this as a replacement for job id / matrix id? And if so is container name the correct name for the environment variable? AFAICT it’s used to split caches per job giving them a unique name. The fact it’s actually a docker container name isn’t relevant where it’s used. So two questions, could this be job.id / matrix.id instead and if not is this the right name?

willcl-ark commented at 9:03 am on August 14, 2025:

We don’t have matrix.id in our CI yml currently so to avoid creating another field existing $CONTAINER_NAME has been used.

matrix.name would not be suitable as the names have spaces in.

maflcko commented at 6:49 am on August 21, 2025:

When running locally, the image, the container and the volumes are named after it:

0ci/test/02_run_container.sh:  docker volume create "${CONTAINER_NAME}_ccache" || true
1ci/test/02_run_container.sh:  docker volume create "${CONTAINER_NAME}_depends" || true
2ci/test/02_run_container.sh:  docker volume create "${CONTAINER_NAME}_depends_sources" || true
3ci/test/02_run_container.sh:  docker volume create "${CONTAINER_NAME}_previous_releases" || true

So I think it is fine to use the same naming scheme for GHA as well.

in .github/workflows/ci.yml:512 in c762d2d5ea outdated

458@@ -495,6 +459,12 @@ jobs:
459         with:
460           use-cirrus: ${{ needs.runners.outputs.use-cirrus-runners }}
461 
462+      - name: Enable bpfcc script
463+        if: ${{ env.CONTAINER_NAME == 'ci_native_asan' }}

m3dwards commented at 6:35 pm on August 13, 2025:

Why not matrix.id?

willcl-ark commented at 9:03 am on August 14, 2025:

Same comment as above regarding matrix.id and matrix.name

willcl-ark force-pushed on Aug 14, 2025

DrahtBot added the label Needs rebase on Aug 20, 2025

in .github/actions/configure-environment/action.yml:18 in 5d652ca1ef outdated

13+
14+    - name: Set cache paths
15+      shell: bash
16+      run: |
17+        echo "BASE_CACHE=${{ runner.temp }}/depends/built" >> $GITHUB_ENV
18+        echo "PREVIOUS_RELEASES_DIR=${{ runner.temp }}/previous_releases" >> $GITHUB_ENV

maflcko commented at 7:00 am on August 21, 2025:

style nit in 5d652ca1ef996cbe7ad9e94a4c6d29b11a7f35e9: Not sure about the distinction, but PREVIOUS_RELEASES_DIR is also a CI dir (config). See the output in https://github.com/bitcoin/bitcoin/actions/runs/16960628990/job/48072349653?pr=32989#step:9:114:

0+ ./ci/test/02_run_container.py
1Export only allowed settings:
2+ bash -c 'grep export ./ci/test/00_setup_env*.sh'
3+ cat /tmp/env-admin-ci_native_previous_releases
4CCACHE_TEMPDIR=/tmp/.ccache-temp
5CCACHE_MAXSIZE=500M
6 ...
7PREVIOUS_RELEASES_DIR=/home/admin/actions-runner/_work/_temp/previous_releases

willcl-ark commented at 9:36 am on September 1, 2025:

Merged these into a single step in fdf64e55324, as there was no clear distinction between the two steps.

maflcko added this to the milestone 30.0 on Aug 21, 2025

maflcko commented at 7:03 am on August 21, 2025: member

only change is some env var shuffling

re-ACK c2d18db38de6d5f71d48437422b93b8a962d3935 🛄

Signature:

0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
2trusted comment: re-ACK c2d18db38de6d5f71d48437422b93b8a962d3935 🛄
3bYgBtprVMXfUk5BgAVvp+8lLUjPrKQfBmFfyJQyWnDxsJYbT2dBKu4IbR6dTg7fEycNYRXVDQYayCZO/ZJkeBw==

maflcko commented at 7:13 am on August 21, 2025: member

c18b093 also seems fine to keep, but can be cherry-picked in a follow-up.

Despite this gaining quite some decent-enough runtime performance, I’d prefer to add this in a followup after having gotten TSAN failures twice in this PR.

@fanquake also noted that he saw some issues when using (2 * nproc) in local runs, so I think safer to leave for now.

Ah, I see. I guess “some issues” refers to the symptoms also seen in #31132 (comment) ?

maflcko commented at 7:23 am on August 21, 2025: member

I’ve added the 30 milestone, because there seems to be conceptual agreement to do this. This is not an end-user feature, so the feature freeze does not apply, but it would be good to get some more review on this. One of the goals in the pull description is so that "anyone" familiar with GitHub-style CI systems can work on it. It would be nice if one or two of the GitHub-style CI persons could review this :sweat_smile:

in .github/workflows/ci.yml:34 in c2d18db38d outdated

28@@ -28,6 +29,22 @@ defaults:
29     shell: bash
30 
31 jobs:
32+  runners:
33+    name: 'determine runners'
34+    runs-on: ubuntu-24.04

maflcko commented at 7:30 am on August 21, 2025:

syle nit: Could be ubuntu-latest, to avoid having to touch this again in the future?

willcl-ark commented at 9:36 am on September 1, 2025:

amended in cc1735d7771

in .github/workflows/ci.yml:538 in c2d18db38d outdated

552+    timeout-minutes: 20
553+    env:
554+      CONTAINER_NAME: "bitcoin-linter"
555+    steps:
556+      - name: Checkout
557+        uses: actions/checkout@v4

maflcko commented at 7:36 am on August 21, 2025:

nit: IIUC the checkout merge_base issue mentioned in #32989 (comment) still remains and would be nice to fix. But this can be done in a follow-up.

0xB10C commented at 6:49 am on August 22, 2025:

0        uses: actions/checkout@v5

willcl-ark commented at 9:36 am on September 1, 2025:

Yes I’d like to keep this for a followup for now.

maflcko commented at 7:38 am on August 21, 2025: member

(just nits)

in .github/workflows/ci.yml:314 in c2d18db38d outdated

312@@ -295,36 +313,23 @@ jobs:
313       - name: Checkout
314         uses: actions/checkout@v4

0xB10C commented at 6:48 am on August 22, 2025:

nit:

0        uses: actions/checkout@v5

See https://github.com/bitcoin/bitcoin/pull/33171

willcl-ark commented at 9:36 am on September 1, 2025:

Done, here and elsewhere

DrahtBot requested review from 0xB10C on Aug 22, 2025

in .github/workflows/ci.yml:497 in c2d18db38d outdated

497+            timeout-minutes: 120
498+            file-env: './ci/test/00_setup_env_native_msan.sh'
499+
500     steps:
501       - name: Checkout
502         uses: actions/checkout@v4

0xB10C commented at 6:49 am on August 22, 2025:

0        uses: actions/checkout@v5

DrahtBot requested review from 0xB10C on Aug 22, 2025

maflcko commented at 6:19 am on August 28, 2025: member

Are you still working on this?

willcl-ark commented at 6:52 am on August 28, 2025: member

Yep. But been on holiday this week and last.

stickies-v commented at 4:42 pm on August 28, 2025: contributor

Concept ACK. Price seems reasonable for the benefits, and we we’re not locking ourselves in too much.

maflcko commented at 5:31 pm on August 28, 2025: member

Also, would be nice to have a pull request for https://github.com/bitcoin-core/qa-assets prepared. Possibly without caching is fine, if the msan-fuzz task fits in the GHA timeout.

ci: add configure environment action b8fcc9fcbc

ci: add caching actions

Add "Restore" and "Save" caching actions.

These actions reduce boilerplate in the main ci.yml configuration file.

These actions are implemented so that caches will be saved on `push`
only.

When a pull request is opened it will cache hit on the caches from the
lastest push, or in the case of depends will hit on any matching depends
hash, falling back to partial matches.

Depends caches are hashed using
`$(git ls-tree HEAD depends "ci/test/$FILE_ENV" | sha256sum | cut -d' ' -f1)`
and this hash is passed in as an input to the actions. This means we
direct cache hit in cases where depends would not be re-built, otherwise
falling back to a partial match.

Previous releases cache is hashed similarly to depends, but using the
test/get_previous_releases.py file.

The cirruslabs cache action will fallback transparently to GitHub's
cache in the case that the job is not being run on a Cirrus Runner,
making these compatible with running on forks (on free GH hardware).

b232b0fa5e

ci: add REPO_USE_CIRRUS_RUNNERS

If set, Cirrus runners will be used on pushes to, and pull requests
against, this repository.

Forks can set this if they have their own cirrus runners.

33ba073df7

ci: add configure-docker action

Another action to reduce boilerplate in the main ci.yml file.

This action will set up a docker builder compatible with caching build
layers to a container registry using the `gha` build driver.

It will then configure the docker build cache args.

fdf64e5532

ci: use buildx in ci

Using buildx is required to properly load the correct driver, for use
with registry caching. Neither build, nor BUILDKIT=1 currently do this
properly.

Use of `docker buildx build` is compatible with podman.

94a0932547

ci: use docker build cache arg directly

Reverts: e87429a2d0f23eb59526d335844fa5ff5b50b21f

This was added in PR #31545 with the intention that self-hosted runners
might use it to save build cache.

As we are not using hosted runners with a registry build cache, the bulk
of this commit can be reverted, simply using the value of
$DOCKER_BUILD_CACHE_ARG in the script.

link: https://github.com/bitcoin/bitcoin/pull/31545

18f6be09d0

ci: have base install run in right dir

This sets the build dir at build time so that Apple SDK gets installed
in the correct/expected location for the runtime to find it.

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

9c2b96e0d0

ci: add Cirrus cache host

Whilst the action cirruslabs/actions/cache will automatically set this
host, the docker `gha` build cache backend will not be aware of it.

Set the value here, which will later be used in the docker build args to
enable docker build cache on the cirrus cache.

020069e6b7

ci: add job to determine runner type

To remove multiple occurances of the respository name, against which we
compare `${{ github.repository }}` to check if we should use Cirrus
Runners, introduce a helper job which can check a single environment
variable and output this as an input to subsequent jobs.

Forks can maintain a trivial patch of their repo name against the
`REPO_USE_CIRRUS_RUNNERS` variable in ci.yml if they have Cirrus Runners
of their own, which will then enable cache actions and docker build
cache to use Cirrus Cache.

It's not possible to use `${{ env.USE_CIRRUS_RUNNERS }}` in the
`runs-on:` directive as the context is not supported by GitHub.

If it was, this job would no longer be necessary.

cc1735d777

ci: update windows-cross job

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

04e7bfbceb

willcl-ark force-pushed on Sep 1, 2025

willcl-ark commented at 9:38 am on September 1, 2025: member

Rebased on master to catch upstream changes, manually added changes to align with #31802 and #33099 which were merged while I was away. Hopefully I didn’t miss any others.

maflcko commented at 9:54 am on September 1, 2025: member

Just some style fixups, but it looks like out-of-storage errors are happening:

0/home/admin/actions-runner/_work/_temp/depends/work/build/arm-linux-gnueabihf/qt/6.7.3-634ec665c7f/qtbase/src/gui/util/qvalidator.h:163:2: fatal error: cannot write PCH file: No space left on device

re-ACK 8a7eb6ef2c226459739bdfe050b735a5a0d0b771 🕧

Signature:

0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
2trusted comment: re-ACK 8a7eb6ef2c226459739bdfe050b735a5a0d0b771 🕧
3UW0pp2ycXnHD1XOEPIkrXvZU9Yk4ma/cp6mh+VKLxb/jU9Ryw9kg5dli95zhThJX+jxKqUUr07tQPNLMxv1eDQ==

DrahtBot requested review from stickies-v on Sep 1, 2025

DrahtBot removed the label Needs rebase on Sep 1, 2025

in .github/workflows/ci.yml:416 in 8a7eb6ef2c outdated

410@@ -406,44 +411,145 @@ jobs:
411           TEST_RUNNER_EXTRA: ${{ github.event_name != 'pull_request' && '--extended' || '' }}
412         run: py -3 test/functional/test_runner.py --jobs $NUMBER_OF_PROCESSORS --ci --quiet --tmpdirprefix="$RUNNER_TEMP" --combinedlogslen=99999999 --timeout-factor=$TEST_RUNNER_TIMEOUT_FACTOR $EXCLUDE $TEST_RUNNER_EXTRA
413 
414-  asan-lsan-ubsan-integer-no-depends-usdt:
415-    name: 'ASan + LSan + UBSan + integer, no depends, USDT'
416-    runs-on: ubuntu-24.04 # has to match container in ci/test/00_setup_env_native_asan.sh for tracing tools
417+  ci-matrix:
418+    name: ${{ matrix.name }}
419+    needs: runners

janb84 commented at 12:23 pm on September 1, 2025:

0    needs: [runners, lint]

Nit / Question: Why not wait on successful run of linter before running more heavier / costlier runs ?

m3dwards commented at 2:57 pm on September 2, 2025:

Lint job looks at GPG keys on commits and AFAIK rarely fails. It’s not like a “normal” lint that checks code style and therefore often fails.

maflcko commented at 3:09 pm on September 2, 2025:

It’s not like a “normal” lint that checks code style and therefore often fails.

It is, and it fails occasionally: https://github.com/bitcoin/bitcoin/pulls?q=is%3Apr+%22task+lint%3A+https%3A%2F%2Fgithub.com%2Fbitcoin%2Fbitcoin%2Fruns%2F%22+is%3Aclosed

No opinion on the dep graph. The benefit of putting it in needs is that the other tasks are skipped (as they are possibly not needed). The downside of putting it in needs is that the other tasks are skipped (as the failure is intended/ignored and the other tasks are desired).

Either seems fine.

janb84 commented at 1:05 pm on September 1, 2025: contributor

ACK 8a7eb6ef2c226459739bdfe050b735a5a0d0b771

PR changes CI from self hosted instances to hosted Cirrus Runners. In addition to migrating to a more-commonly-used GitHub workflow syntax.

The migration to the more-commonly-used GitHub workflow syntax makes the whole CI proces easier more structured, easier to understand, this will benefit future maintenance. In addition to be less reliant on (centralised) self-hosting with the ability to easily switch-out Cirrus in the future, will be a benefit (in the long term).

Small non-blocking NIT / question added.

extensive code review ✅
research into Cirrus runners ✅
coming form Azure Pipelines did some read up on Github action best practices. ✅

maflcko commented at 1:11 pm on September 1, 2025: member

Just some style fixups, but it looks like out-of-storage errors are happening:

0/home/admin/actions-runner/_work/_temp/depends/work/build/arm-linux-gnueabihf/qt/6.7.3-634ec665c7f/qtbase/src/gui/util/qvalidator.h:163:2: fatal error: cannot write PCH file: No space left on device

For reference, https://github.com/bitcoin/bitcoin/actions/runs/17373684837/job/49315124575?pr=32989#step:9:1483:

 0+ echo 'Free disk space:'
 1+ df -h
 2Free disk space:
 3+ [[ ci_arm_linux == \c\i\_\n\a\t\i\v\e\_\a\s\a\n ]]
 4Filesystem      Size  Used Avail Use% Mounted on
 5overlay          17G   15G  3.0G  83% /
 6tmpfs            64M     0   64M   0% /dev
 7shm              64M     0   64M   0% /dev/shm
 8/dev/vda1        17G   15G  3.0G  83% /etc/hosts
 9tmpfs           5.9G     0  5.9G   0% /proc/acpi
10tmpfs           5.9G     0  5.9G   0% /proc/scsi
11tmpfs           5.9G     0  5.9G   0% /sys/firmware

But https://cirrus-runners.app/setup/#__tabbed_3_3 claims at least 50 GB?

fkorotkov commented at 1:54 pm on September 1, 2025: none

@maflcko apologies for the disk size on Arm runners issue. It should be fixed. Please try to re-run the job.

fanquake commented at 1:56 pm on September 1, 2025: member

@maflcko It should be fixed. Please try to re-run the job.

Thanks. Kicked the job.

maflcko commented at 3:04 pm on September 1, 2025: member

Looks like Cirrus switched from Neoverse N1 to Apple (with virtual Linux aarch64), which doesn’t support 32-bit mode, so the CI here fails expectedly.

Maybe you can use GHA ubuntu-24.04-arm for now, which has:

 0VM Image
 1  - OS: Linux (arm64)
 2  - Source: Partner
 3  - Name: Ubuntu 24.04 by Arm Limited
 4  - Version: 20250728.24.1
 5  - Included Software: https://github.com/actions/partner-runner-images/blob/main/images/arm-ubuntu-24-image.md
 6
 7
 8Run lscpu
 9Architecture:                         aarch64
10CPU op-mode(s):                       32-bit, 64-bit
11Byte Order:                           Little Endian
12CPU(s):                               4
13On-line CPU(s) list:                  0-3
14Vendor ID:                            ARM
15Model name:                           Neoverse-N2
16....

ci: port arm 32-bit job

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

f253031cb8

ci: update asan-lsan-ubsan

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

884251441b

ci: force reinstall of kernel headers in asan

When using hosted runners in combination with cached docker images,
there is the possibility that the host runner image is updated,
rendering the linux-headers package (stored in the cached docker image)
incompatible.

Fix this by doing a re-install of the headers package in
03_test_script.sh.

If the underlying runner kernel has not changed thie has no effect, but
prevents the job from failing if it has.

2c990d84a3

ci: port mac-cross-gui-notests

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

9c2514de53

ci: port nowallet-libbitcoinkernel

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

2a00b12d73

ci: port no-IPC-i686-DEBUG

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

f2068f26c1

ci: port fuzzer-address-undefined-integer-nodepends

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

341196d75c

ci: port previous-releases-depends-debug

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

58e38c3a04

ci: port centos-depends-gui

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

549074bc64

ci: port tidy

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

bf7d536452

ci: port tsan-depends

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

9bbae61e3b

ci: port msan-depends

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

d290a8e6ea

ci: port lint

Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>

bc41848d00

ci: remove .cirrus.yml

Removed as unused.

4393ffdd83

ci: dynamically match makejobs with cores

Previously jobs were running on a large multi-core server where 10 jobs
as default made sense (or may even have been on the low side).

Using hosted runners with fixed (and lower) numbers of vCPUs we should
adapt compilation to match the number of cpus we have dynamically.

This is cross-platform compatible with macos and linux only.

3f339e99e0

doc: Detail configuration of hosted CI runners f427284483

ci: add ccache hit-rate warning when < 75%

Print the ccache hit-rate for the job using a GitHub annotation if it
was below 75%.

dd1c5903e8

ci: fix annoying docker warning

Docker currently warns that we are missing a default value.

Set this to scratch which will error if an appropriate image tag is not
passed in to silence the warning.

2aa288efdd

ci: remove un-needed lint_run*.sh files

ci/lint_run.sh: Only used in .cirrus.yml. Refer to test/lint/README.md on how to run locally.
ci/lint_run_all.sh: Only used in .cirrus.yml for stale re-runs of old pull request tasks.

3c5da69a23

willcl-ark force-pushed on Sep 1, 2025

janb84 commented at 4:37 pm on September 1, 2025: contributor

re ACK 3c5da69a232ba1cfb935012aa53e57002efe0d77

changes since last ACK:

changed cirrus-runner for 32 bit ARM

DrahtBot requested review from maflcko on Sep 1, 2025

maflcko commented at 8:21 am on September 2, 2025: member

re-ACK 3c5da69a232ba1cfb935012aa53e57002efe0d77 🏗

Signature:

0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
2trusted comment: re-ACK 3c5da69a232ba1cfb935012aa53e57002efe0d77 🏗
3iCG/VfhhPRwVifYiSBwp/fjsB601ab+yeyoFMfCXr+dpHw+QGtigbkvFGKi5UaJAdp4S/BpA9wYVL4yXf6gfBA==

m3dwards commented at 2:58 pm on September 2, 2025: contributor

ACK 3c5da69a232ba1cfb935012aa53e57002efe0d77

achow101 commented at 11:25 pm on September 2, 2025: member

ACK 3c5da69a232ba1cfb935012aa53e57002efe0d77

Not particularly familiar with github actions or the cirrus, but it seems like all of the tasks that we were running on cirrus previously are correctly ported over and working.

fanquake merged this on Sep 3, 2025

fanquake closed this on Sep 3, 2025

in ci/test/02_run_container.sh:29 in 9c2b96e0d0 outdated

25@@ -26,6 +26,7 @@ if [ -z "$DANGER_RUN_CI_ON_HOST" ]; then
26       --file "${BASE_READ_ONLY_DIR}/ci/test_imagefile" \
27       --build-arg "CI_IMAGE_NAME_TAG=${CI_IMAGE_NAME_TAG}" \
28       --build-arg "FILE_ENV=${FILE_ENV}" \
29+      --build-arg "BASE_ROOT_DIR=${BASE_ROOT_DIR}" \

maflcko commented at 10:43 am on September 3, 2025:

nit in 9c2b96e0d0303708710f9fbca3f9fdcb2fff21c1: Actually, on a second thought, this seems a bit fragile. It seems this would break the use-case of using several different local folders to run the CI, because the CI images have different, non-deterministic, paths embedded?

I presume the correct fix would be to download the sdk to a hard-coded path in the container image, and then adjust the pre-existing rsync call to copy it at runtime in the container, if needed.

maflcko commented at 3:41 pm on September 3, 2025: member

To complete migration from self-hosted to hosted for this repo, the backport branches 27.x, 28.x and 29.x would also need their CI ported, but these are left for followups to this change (and pending review/changes here first).

27.x looks close to EOL in one month, according to https://bitcoincore.org/en/lifecycle/, so I guess it could just die with the old CI?

fanquake commented at 3:42 pm on September 3, 2025: member

Yea. I think it’s fine to leave 27.x as is, this just needs backporting to 29.x & 28.x.

ci: Migrate CI to hosted Cirrus Runners #32989

Benefits

Maintenance

Bus factor

Scaling

Speed

GitHub workflow syntax

Reporting

Costs

Financial

Dependencies

Forks

Migration process

Caching

ccache, depends-sources, built-depends

Docker build layer caching

But what about… x?

TODO:

Code Coverage & Benchmarks

Reviews

Conflicts

But what about… `x`?