ci: Migrate CI to hosted Cirrus Runners #32989

pull willcl-ark wants to merge 29 commits into bitcoin:master from willcl-ark:cirrus-runners changing 15 files +372 −315
  1. willcl-ark commented at 8:39 am on July 16, 2025: member

    This changeset migrates all current self-hosted CI jobs over to hosted Cirrus Runners.

    These runners cost a flat rate of $150/month, and we qualify for an open source discount of 50%. Therefore they are $75/month/runner.

    One “runner” should more accurately be thought of in terms of the number of vCPU you are purchasing: https://cirrus-runners.app/pricing/ or in terms of “concurrency”, where 1 runners gets you 1.0 concurrency. e.g. a Linux x86 Runner gets you 16 vCPU (1.0 concurrency) and 64GB RAM to be provisioned as you choose, amongst one or more jobs.

    Cirrus Runners currently only support Linux (x86 and Arm64) and MacOS (Arm64). This changeset does not move the existing Github Actions native MacOS runners away from being run on Github’s infrastructure. This could be a follow up optimisation.

    Runs from this changeset using Cirrus Runners can be found at: https://github.com/testing-cirrus-runners/bitcoin2/actions which shows an uncached run on master (CI#1), an outside pull request (CI#3) and an updated push to master (CI#4).

    These workflows were run on 10 runners, and we would recommend purchasing a similar number for our CI in this repo to achieve the speed and concurrency we expect.

    We include some optional performance commits, but these could be split out and made into followups or dropped entirely.

    Benefits

    Maintenance

    As we are not self-hosting, nobody needs to maintain servers, disks etc.

    Bus factor

    Currently we have a very small number of people with the know-how working on server setup and maintenance. This setup fixes that so that “anyone” familiar with GitHub-style CI systems can work on it.

    Scaling

    These do not “auto-scale”/have “unlimited concurrency” like some solutions, but if we want more workers/cpu to increase parallism or increase the runner size of certain jobs for a speed-up we can simply buy more concurrency using the web interface.

    Speed

    Runtimes aproximate current runtimes pretty well, with some jobs being faster. Caching improvements on pull request (re-runs) are left as future optimisations from the current changeset (see below).

    GitHub workflow syntax

    With a migration to the more-commonly-used GitHub workflow syntax, migration to other providers in the future is often as simple as a one-line change (and installing a new GitHub app to the repo).

    If we decide to self-host again, then we can also self-host GitHub runners (using https://github.com/actions/runner) and maintain new GH-style CI syntax.

    Reporting

    GitHub workflows provide nicer built-in reporting directly on the “Checks” page of a pr. This includes more-detailed action reporting, and a host of pretty nice integrated features, such as Workflow Commands for creating annotations that can print messages during runs. See for example at the bottom of this window where we report ccache hitrate, if it was below 90%: https://github.com/testing-cirrus-runners/bitcoin/actions/runs/16163449125?pr=1

    These could be added conditionally into our CI scripts to report interesting or other information.

    Costs

    Financial

    Relative to competitors Cirrus runners are cheap for the hosted CI-world. However these are likely more expensive than our current setup, or a well-configured (new) self-hosted setup.

    If we started with 10 runners to be shared amongst all migrated jobs, this would total $750/mo = $9000/yr.

    Note that we are not trying to comptete here on cost directly.

    Dependencies

    We would be dependent on Cirrus infra.

    Forks

    • Forks should be able to run CI without paid Cirrus runners. This behaviour is achieved through a rather verbose runs-on: directive.
      • This directive hardcodes the main repo (unfortunately you cannot use the env github context in this field in particular, for some reason).
      • This directive also allows for a fork to patch the runs-on: field in the ci.yml file if they want to use Cirrus Runners too.
      • The workflow otherwise will fallback to the GitHub free runners on forks.
    • This cirrus cache action transparently falls back to github actions cache when not running on cirrus, so forks will get some free github caching (10GB per repo).

    All jobs work on forks, but will run (slowly) on GitHub native free hosted runners, instead of Cirrus runners. They will also suffer from poor cache hit-rates, but there’s nothing that can be done about that, and the situtation is an improvement on today.

    Migration process

    The main org should also, in addition to pulling code changes:

    1. Permit the actions docker/setup-buildx-action@v3 and docker/login-action@v3 to be run in this repo.

    Caching

    For the number of CI jobs we have, cache usage on GitHub would be an issue as GH only provides 10GB of cache space, per repo. However cirrus provides 10 GB per runner, which scales better with the number of runners.

    The cirruslabs/action/[restore|save] action we use here redirects this to Cirrus’ own cache and is both faster and larger.

    In the case that user is running CI on a fork, the cirrus cache falls back transparently to GitHub default cache without error.

    ccache, depends-sources, built-depends

    • Cached as blobs via cirruslabs/actions/cache action.
    • Current implementation:
      • On push: restores and saves caches.
      • On pull_request: restores but does not save caches.

    This means a new pull request should hit a pretty relevant cache. Old pull requests which are not being rebased on master may suffer from lower cache hit-rate.

    If we save caches on all pull request runs we run the risk of evicting recent (and more relevant) cache blobs. It may be possible in a future optimisation to widen this to save on pull request runs too, but it will also depend on how many runners we provision and what cache churn rates are like in the main repo.

    Docker build layer caching

    • Cached using the gha cache backend
    • These cache blobs compete for space with ccache, depends-sources and depends-built caches
    • gha cache allows --cache-from to be used from pull requests, which does not work using a registry cache type (technically we could use a public read-only token to get this working, but that feels wrong)

    This backend does network i/o and so are marginally slower than our current disk i/o cache.

    But what about… x?

    We have tested many other providers, including Runs-on, Buildjet, WarpBuild, and GitHub hosted runners (and investigated even more). But they all fall short in one-way or another.

    • Runs-On and Buildjet (and others) require installing GH apps with much too-liberal permissions (e.g. Administration: Read|Write) for our use-case.
    • GitHub hosted runners suffer from all of high costs, lower speed, small cache, and the requirement for a GitHub Teams subscription.
    • WarpBuild seems to be simply too expensive.

    TODO:

    To complete migration from self-hosted to hosted for this repo, the backport branches 27.x, 28.x and 29.x would also need their CI ported, but these are left for followups to this change (and pending review/changes here first).


    Work and experimentation undertaken with @m3dwards

  2. DrahtBot commented at 8:39 am on July 16, 2025: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/32989.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK janb84
    Concept ACK fanquake, achow101, hebasto, 0xB10C, stickies-v
    Stale ACK maflcko

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #33201 (Add functional test for IPC interface by sipa)
    • #33145 (CI: silent merge check by m3dwards)
    • #33051 (Don’t fix Python patch version by Sjors)
    • #31802 (Add bitcoin-{node,gui} to release binaries for IPC by Sjors)
    • #31425 (RFC: Riscv bare metal CI job by TheCharlatan)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  3. in ci/test_imagefile:7 in c0ad2b6aa8 outdated
    3@@ -4,12 +4,16 @@
    4 
    5 # See ci/README.md for usage.
    6 
    7-ARG CI_IMAGE_NAME_TAG
    8+# We never want scratch, but default arg silences a Warning
    


    fanquake commented at 9:39 am on July 16, 2025:
    Can this be specific? I think either document the warning, or drop this comment, or just drop the workaround, if it’s only a warning.

    willcl-ark commented at 10:24 am on July 16, 2025:

    It is only a warning.

    It’s not present current as we are using podman, but docker does this build check: https://docs.docker.com/reference/build-checks/invalid-default-arg-in-from/

    If we are happy with a warning on every docker buildx build ... invocation we can drop it, but this hack silences the warning and will always fail the build if scratch ends up being used (and should be pretty easy to diagnose the failure).

    Preference between adding a link to the specific check for more context, or should we drop it and live with the warning?


    maflcko commented at 7:38 am on August 21, 2025:
    Can be resolved?
  4. in ci/test/02_run_container.sh:29 in c0ad2b6aa8 outdated
    45-    DOCKER_BUILD_CACHE_NEW_DIR="${DOCKER_BUILD_CACHE_TEMPDIR}/${CONTAINER_NAME}"
    46-    DOCKER_BUILD_CACHE_ARG="--cache-from type=local,src=${DOCKER_BUILD_CACHE_OLD_DIR} --cache-to type=local,dest=${DOCKER_BUILD_CACHE_NEW_DIR},mode=max"
    47+  echo "DOCKER_BUILD_CACHE_ARG=$DOCKER_BUILD_CACHE_ARG"
    48+
    49+  # Use buildx directly
    50+  # This is useful in CI where docker is used (not podman) and can help achieve better caching using registry cache
    


    fanquake commented at 9:42 am on July 16, 2025:

    and can help achieve

    This is a bit vauge; have we seen this not work properly? It’s also not completely clear to me that we need buildx to use the registry. Seems like it also works with buildkit, which is now just the default docker build, it any recent Docker version.


    willcl-ark commented at 10:30 am on July 16, 2025:

    Yes, we have tested that even if we configure the docker builder using the setup-buildx-action where use is set to true by default (which should enable that driver in future docker build invocations IIUC), that the incorrect builder is used unless docker buildx build command specifically is used.

    We have tried with BUILDKIT=1 docker build and plain docker build, but both used the incorrect driver (contrary to what we expected).


    fanquake commented at 10:47 am on July 16, 2025:
    Ok. Could we change this to something like: Using buildx is required to properly load the correct driver, for use with registry caching. Neither build, nor BUILDKIT=1 currently do this properly. Just want to avoid phrases like “can help” or “is useful”, and instead be specific about what we are doing & why.

    maflcko commented at 11:02 am on July 16, 2025:
    why not switch to buildx unconditionally?

    willcl-ark commented at 12:19 pm on July 16, 2025:

    Does podman accept docker buildx build as a command? I will check in a min with it.

    but this was why not originally…


    willcl-ark commented at 12:43 pm on July 16, 2025:
    OK it seems docker buildx build does work with podman, so I have changed buildx to be used unconditionally in this commit now. Thanks!

    maflcko commented at 12:47 pm on July 16, 2025:
    Yeah, if buildx is missing on some platforms, it could also use auto-detection to use it when available and not, when not.
  5. fanquake commented at 9:43 am on July 16, 2025: member
    Concept ACK. This will also need to go back to 27.x.
  6. in .github/workflows/ci.yml:520 in c0ad2b6aa8 outdated
    545         run: sed -i "s|\${INSTALL_BCC_TRACING_TOOLS}|true|g" ./ci/test/00_setup_env_native_asan.sh
    546 
    547+      - name: Set mmap_rnd_bits
    548+        if: ${{ matrix.name == 'TSan, depends, gui' || matrix.name == 'MSan, depends' }}
    549+        # Prevents crashes due to high ASLR entropy
    550+        run: sudo sysctl -w vm.mmap_rnd_bits=28
    


    fanquake commented at 9:46 am on July 16, 2025:
    Wonder if we should just move this into the scripts, rather than set it here, so that it’ll also be done for local usage (where it hasn’t been set already).

    willcl-ark commented at 10:28 am on July 16, 2025:
    That would make sense to me too. Happy to make that change here, or in a followup.

    willcl-ark commented at 1:10 pm on July 16, 2025:

    Actually I feel a bit less convinced that dropping a sudo ... command into the CI scripts is ideal for this actually. I see comments that 32 bits should now be “fixed”: https://github.com/google/sanitizers/issues/1614#issuecomment-1885369007

    Testing this now locally…


    willcl-ark commented at 4:19 pm on July 16, 2025:
    Hmmm my msan ci job is not working (due to something unrelated) for me on my main dev machine, and tsan is segfaulting with vm.mmap_rnd_bits=32, so I’m going to leave this in one way or another for now.

    maflcko commented at 6:24 am on July 17, 2025:

    Actually I feel a bit less convinced that dropping a sudo ... command into the CI scripts is ideal

    Yeah, they should be fine to run as a user normally. The only dependencies are Python, Bash, and a container engine.


    maflcko commented at 7:32 am on August 21, 2025:
    can be resolved?
  7. willcl-ark commented at 10:29 am on July 16, 2025: member

    Concept ACK. This will also need to go back to 27.x.

    Testing a backport to 29.x here: https://github.com/testing-cirrus-runners/bitcoin2/actions/runs/16320536543

    I think the best course of action could be to look for a little more conceptual review here, and after that squash the “ci: port x” commits in this changeset down to a single one, to make backporting to the multiple supported branches easier.

  8. achow101 commented at 6:07 pm on July 16, 2025: member
    Concept ACK
  9. in ci/README.md:74 in c0ad2b6aa8 outdated
    70+   3. Create a new "robot account" with read/write access to the previously-created Docker repository.
    71+4. Set repository variables:
    72+   1. `USE_CIRRUS_RUNNERS = true`
    73+   2. `REGISTRY_USERNAME = <quay robot account name>`
    74+5. Set repo secrets:
    75+   1. `RESISTRY_TOKEN = <quay robot token>`
    


    DrahtBot commented at 5:21 am on July 17, 2025:
    RESISTRY_TOKEN -> REGISTRY_TOKEN [secret name typo]
    GitHubs -> GitHub's [missing apostrophe and letter ‘s’ for possessive]
    
  10. maflcko commented at 6:51 am on July 17, 2025: member

    This means a new pull request should hit a pretty relevant cache. Old pull requests which are not being rebased on master may suffer from lower cache hit-rate.

    I don’t think this is true. A pull request that modifies a core header (like serialize.h) will now always start from a cold cache. The current persistent workers have a high ccache hit rate for pulls that are (force) pushed for minor fixups (https://0xb10c.github.io/bitcoin-core-ci-stats/graph/ccache/). Also, before CI runs, pull requests are rebased/merged with master, so the age of a pull request alone shouldn’t affect cache hit rate.

    However, the trade-offs here are probably worth to go forward and try to optimize the ccache hit rate later. Concept ACK.

    Give the robot account read/write access to this repo

    This seems a bit scary. Are you saying that a proprietary third party outside of our control can now push directly to the repo? My assumption was that the tokens would be added to CI in this repo and CI had write access to the registry, not the other way round. Why would the registry need write access here?

    Edit: We may reconsider #31850 and drop container image caching, and just accept the intermittent network IO errors or network speed issues.

  11. willcl-ark commented at 7:02 am on July 17, 2025: member

    This means a new pull request should hit a pretty relevant cache. Old pull requests which are not being rebased on master may suffer from lower cache hit-rate.

    I don’t think this is true. A pull request that modifies a core header (like serialize.h) will now always start from a cold cache. The current persistent workers have a high ccache hit rate for pulls that are (force) pushed for minor fixups (0xb10c.github.io/bitcoin-core-ci-stats/graph/ccache). Also, before CI runs, pull requests are rebased/merged with master, so the age of a pull request alone shouldn’t affect cache hit rate.

    However, the trade-offs here are probably worth to go forward and try to optimize the ccache hit rate later. Concept ACK.

    Yes, force pushes for minor fixups is the tradeoff we have in the current implementation. As you say, we can set the ccache to save on pull requests too (in the future) if necessary.

    Give the robot account read/write access to this repo

    This seems a bit scary. Are you saying that a proprietary third party outside of our control can now push directly to the repo? My assumption was that the tokens would be added to CI in this repo and CI had write access to the registry, not the other way round. Why would the registry need write access here?

    Sorry for not being clearer! The robot account gets read/write access to the Quay.io (docker) repo, not this code repository!

    Edit: We may reconsider #31850 and drop container image caching, and just accept the intermittent network IO errors or network speed issues.

    Yes, I would love to get #31850 working in any case, as it would simply avoid long rebuilds in the worst cases; most docker images are rebuilding in < 2 minutes, except MSAN…

  12. in .github/actions/restore-caches/action.yml:4 in 2d0ac71899 outdated
    0@@ -0,0 +1,49 @@
    1+name: 'Restore Caches'
    2+description: 'Restore ccache, depends sources, and built depends caches'
    3+inputs:
    4+  job-name:
    


    maflcko commented at 8:02 am on July 17, 2025:
    nit in 2d0ac71899eadcbff8252b1704d2affb73f0160d: Should be named job-id, not the name?

    willcl-ark commented at 10:35 am on July 18, 2025:
    switched to job-id in 9458cd0e66d
  13. in .github/actions/restore-caches/action.yml:17 in 2d0ac71899 outdated
    12+      run: |
    13+        echo "CCACHE_DIR=${{ runner.temp }}/ccache_dir" >> $GITHUB_ENV
    14+        echo "DEPENDS_DIR=${{ runner.temp }}/depends" >> $GITHUB_ENV
    15+        echo "SOURCES_PATH=${{ runner.temp }}/depends/sources" >> $GITHUB_ENV
    16+        echo "BASE_CACHE=${{ runner.temp }}/depends/built" >> $GITHUB_ENV
    17+        echo "DEPENDS_HASH=${{ hashFiles('depends/packages/*.mk', 'depends/Makefile', 'depends/config.guess', 'depends/config.sub', 'depends/funcs.mk', 'depends/builders/*.mk', 'depends/hosts/*.mk') }}" >> $GITHUB_ENV
    


    maflcko commented at 8:11 am on July 17, 2025:
    nit in https://github.com/bitcoin/bitcoin/commit/2d0ac71899eadcbff8252b1704d2affb73f0160d: Why not just git rev-parse HEAD:depends, like it was done for cirrus? (See git log -S 'git rev-parse HEAD:depends')

    willcl-ark commented at 10:35 am on July 18, 2025:
    This now uses git rev-parse HEAD:depends in both 868ddf55718 and 1b689cf7811
  14. in .github/actions/restore-caches/action.yml:25 in 2d0ac71899 outdated
    20+      id: ccache-cache
    21+      uses: cirruslabs/cache/restore@v4
    22+      with:
    23+        path: ${{ env.CCACHE_DIR }}
    24+        # A single ccache per job
    25+        key: ccache-${{ inputs.job-name }}-${{ github.ref_name }}-${{ github.run_id }}
    


    maflcko commented at 8:15 am on July 17, 2025:
    nit in https://github.com/bitcoin/bitcoin/commit/2d0ac71899eadcbff8252b1704d2affb73f0160d: Comment seems wrong? It is a cache for each job-id + ref_name + run_id. Could just remove the comment?

    willcl-ark commented at 10:36 am on July 18, 2025:
    comment removed in 9458cd0e66d
  15. in .github/actions/restore-caches/action.yml:27 in 2d0ac71899 outdated
    22+      with:
    23+        path: ${{ env.CCACHE_DIR }}
    24+        # A single ccache per job
    25+        key: ccache-${{ inputs.job-name }}-${{ github.ref_name }}-${{ github.run_id }}
    26+        restore-keys: |
    27+          ccache-${{ inputs.job-name }}-master-
    


    maflcko commented at 8:22 am on July 17, 2025:

    nit in https://github.com/bitcoin/bitcoin/commit/2d0ac71899eadcbff8252b1704d2affb73f0160d: why hardcode master. This makes it impossible to cache in pull requests, if this is ever done in the future. Also, only master caches are created, so this doesn’t guard against anything.

    Also, the others don’t have it.


    willcl-ark commented at 10:37 am on July 18, 2025:
    All references to master have been switched to github.event.repository.default_branch where appropriate. Removed the ref_name from the save/restore keys, as it is indeed unnecessary.
  16. in .github/actions/save-caches/action.yml:12 in 2d0ac71899 outdated
     7+runs:
     8+  using: 'composite'
     9+  steps:
    10+    - name: Save Ccache cache
    11+      uses: cirruslabs/cache/save@v4
    12+      if: ${{ (github.event_name == 'push') && (github.ref_name == 'master') }}
    


    maflcko commented at 8:30 am on July 17, 2025:

    nit in the same commit: Could use github.event.repository.default_branch instead of master, so that caches can be saved for other repos, such as inquisition?

    (same below)

  17. in .github/actions/restore-caches/action.yml:36 in 2d0ac71899 outdated
    31+      id: depends-sources
    32+      uses: cirruslabs/cache/restore@v4
    33+      with:
    34+        path: ${{ env.SOURCES_PATH }}
    35+        # A single shared depends sources for all jobs per hash of package definitions and meta_depends
    36+        key: depends-sources-${{ env.DEPENDS_HASH }}-${{ github.run_id }}
    


    maflcko commented at 8:32 am on July 17, 2025:
    same commit: This seems broken? A job-id with NO_QT=1 populating this would corrupt the cache for other job-id’s that build qt?

    willcl-ark commented at 10:38 am on July 18, 2025:

    9458cd0e66d now uses a depends sources cache per job_id to avoid this pitfall (thanks!):

    0key: depends-sources-${{ inputs.job-id }}-${{ env.DEPENDS_HASH }}
    
  18. in .github/actions/restore-caches/action.yml:14 in 2d0ac71899 outdated
     9+  steps:
    10+    - name: Set cache paths
    11+      shell: bash
    12+      run: |
    13+        echo "CCACHE_DIR=${{ runner.temp }}/ccache_dir" >> $GITHUB_ENV
    14+        echo "DEPENDS_DIR=${{ runner.temp }}/depends" >> $GITHUB_ENV
    


    maflcko commented at 8:36 am on July 17, 2025:
    same commit: Unused?

    willcl-ark commented at 10:39 am on July 18, 2025:
    Moved this code into the main ci.yml file, so that the cache actions are only performing caching (rather than being responsible for setting env vars), both in 868ddf55718 and 1b689cf7811
  19. in .github/actions/configure-docker/action.yml:38 in b1a4adc814 outdated
    33+        password: ${{ inputs.registry-password }}
    34+
    35+    - name: Set docker build args
    36+      shell: bash
    37+      run: |
    38+        source ${{ env.FILE_ENV }}
    


    maflcko commented at 8:44 am on July 17, 2025:
    nit in b1a4adc814369aff21b8c12775270cfe648db486: Could add a comment that this is only used to extract CONTAINER_NAME.

    willcl-ark commented at 10:39 am on July 18, 2025:
    Comment added in db78efac768
  20. in .github/workflows/ci.yml:25 in 9002faff0d outdated
    19@@ -20,6 +20,9 @@ concurrency:
    20 env:
    21   CI_FAILFAST_TEST_LEAVE_DANGLING: 1  # GHA does not care about dangling processes and setting this variable avoids killing the CI script itself on error
    22   MAKEJOBS: '-j10'
    23+  REGISTRY: ${{ vars.REGISTRY || 'quay.io' }}
    24+  REGISTRY_ORG: ${{ vars.REGISTRY_ORG || 'bitcoin-core' }} # TODO: Set correct org
    25+  REGISTRY_REPO: ${{ vars.REGISTRY_REPO || 'core-ci' }} # TODO: Set correct repo
    


    maflcko commented at 8:46 am on July 17, 2025:
    9002faff0d7518ebafe05d8d43d6558d238c2659: adjust for https://quay.io/repository/bitcoincore/bitcoin-ci ?

    willcl-ark commented at 10:40 am on July 18, 2025:
    Registry env vars updated in 7272f56a755
  21. maflcko approved
  22. maflcko commented at 8:50 am on July 17, 2025: member

    I guess you want review here and then address it, as it comes in? Once review is finished, the app will be installed and reviewers can also look at a “real” run in this repo?

    looked at c0ad2b6aa8e8c31c9f9c9ea2b35ca86f7985c490~23 🎬

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: looked at c0ad2b6aa8e8c31c9f9c9ea2b35ca86f7985c490~23 🎬
    3gylJtm++jv0E+65SRoLFPC+ef+fwpVJftiMQ+ziB1uRZAF2MwE7TW3JaEv8iJIpZsRmForPR0jik8/6QvUs+BQ==
    
  23. hebasto commented at 12:20 pm on July 17, 2025: member

    Concept ACK.

    we qualify for an open source discount of 50%.

    We would be dependent on Cirrus infra…

    We shouldn’t be surprised when Cirrus suddenly changes its modus operandi, including its advertised open-source discount or general availability.

  24. willcl-ark commented at 12:35 pm on July 17, 2025: member

    Concept ACK.

    we qualify for an open source discount of 50%.

    We would be dependent on Cirrus infra…

    We shouldn’t be surprised when Cirrus suddenly changes its modus operandi, including its advertised open-source discount or general availability.

    Certainly, it is good to be wary of that. I think this is equally true for all cloud providers though.

    It’s my belief that if we complete this migration we are resonably well protected against this risk for the following reasons:

    1. As we use the common GitHub workflow yaml format, changing to another provider can be as simple as amending the runs-on: line in the yaml (and installing a different provider’s GH app, and changing the cache action to that provider’s own).
    2. We could always revert back to a self-hosted solution utilizing the new workflow yaml via https://github.com/actions/runner, which for example can be trivially configured on one or more self-hosted servers using services.github.runner with Nix (I have a demo of such a configuration here).

    We seem to have a good working relationship so far with Cirrus, @m3dwards has a responsive and helpful contact there. Of course, we are in the tendering stage so there is perhaps extra impetus to be helpful to us, but I don’t see any reason that a historical precedence of limiting free runners (which are allegedly being abused for crypto mining), should appear any more risky to paid/premium customers (than any other provider).

  25. maflcko commented at 12:42 pm on July 17, 2025: member

    2. We could always revert back to a self-hosted solution

    Agree that this may happen with any third party (including GitHub itself). If we want to switch back to the self-hosted runners, it should be as trivial as git revert $the_merge_commit_of_this_pull. Alternatively, switch to GHA-based self-hosted runners. Though, it would be good to create a proof-of-concept pull request to switch to self-hosted runners in GHA, or a different hosted alternative (e.g. warp-build), after this pull is merged. This makes it easier to see that (1) it works and (2) how easily it is possible.

  26. willcl-ark force-pushed on Jul 18, 2025
  27. willcl-ark commented at 10:34 am on July 18, 2025: member
    Pushed c126475ed7a17ec9030066056e31846c7124dcf3 with a CI run on master branch at https://github.com/testing-cirrus-runners/bitcoin2/actions/runs/16368410249
  28. willcl-ark commented at 10:40 am on July 18, 2025: member
    Thanks for the review @maflcko & @fanquake , I hope I addressed all your current review comments.
  29. 0xB10C commented at 11:27 am on July 18, 2025: contributor

    Concept ACK!

    I think finishing the migration would close https://github.com/bitcoin/bitcoin/issues/31965

  30. in ci/test/02_run_container.sh:52 in 20a258310e outdated
    45@@ -46,8 +46,10 @@ if [ -z "$DANGER_RUN_CI_ON_HOST" ]; then
    46     DOCKER_BUILD_CACHE_ARG="--cache-from type=local,src=${DOCKER_BUILD_CACHE_OLD_DIR} --cache-to type=local,dest=${DOCKER_BUILD_CACHE_NEW_DIR},mode=max"
    47   fi
    48 
    49+  # Use buildx unconditionally
    50+  # Using buildx is required to properly load the correct driver, for use with registry caching. Neither build, nor BUILDKIT=1 currently do this properly
    51   # shellcheck disable=SC2086
    52-  DOCKER_BUILDKIT=1 docker build \
    53+  DOCKER_BUILDKIT=1 docker buildx build \
    


    maflcko commented at 1:49 pm on July 18, 2025:
    nit in 20a258310ea4fe1c4cd280d26047aee063215454: Buildx always uses BuildKit. Can remove buildkit stuff.

    maflcko commented at 8:19 pm on July 24, 2025:
    (tested podman buildx on a vanilla Ubuntu 24.04, fwiw)

    willcl-ark commented at 9:33 pm on July 25, 2025:
    Thanks! Removed in a4084cb7554
  31. in ci/test/02_run_container.sh:27 in e5295f7ad6 outdated
    44-    DOCKER_BUILD_CACHE_TEMPDIR="$(mktemp --directory ci-docker-build-cache-XXXXXXXXXX)"
    45-    DOCKER_BUILD_CACHE_NEW_DIR="${DOCKER_BUILD_CACHE_TEMPDIR}/${CONTAINER_NAME}"
    46-    DOCKER_BUILD_CACHE_ARG="--cache-from type=local,src=${DOCKER_BUILD_CACHE_OLD_DIR} --cache-to type=local,dest=${DOCKER_BUILD_CACHE_NEW_DIR},mode=max"
    47-  fi
    48+  echo "DOCKER_BUILD_CACHE_ARG=$DOCKER_BUILD_CACHE_ARG"
    49 
    


    maflcko commented at 1:53 pm on July 18, 2025:

    nit in e5295f7ad64be1d834d3e0c2726edfcc67debfff: What about DANGER_DOCKER_BUILD_CACHE_HOST_DIR below?

    Could link to the reverted commit in the commit description instead: e87429a2d0f23eb59526d335844fa5ff5b50b21f?


    willcl-ark commented at 9:33 pm on July 25, 2025:
    I have linked to the commit in e83c3718f2c and removed the dangling DANGER_DOCKER_BUILD_CACHE_HOST_DIR below
  32. in ci/test/02_run_container.sh:26 in e5295f7ad6 outdated
    43-    # ephemeral, it has to take care of cleaning old TEMPDIR's up.
    44-    DOCKER_BUILD_CACHE_TEMPDIR="$(mktemp --directory ci-docker-build-cache-XXXXXXXXXX)"
    45-    DOCKER_BUILD_CACHE_NEW_DIR="${DOCKER_BUILD_CACHE_TEMPDIR}/${CONTAINER_NAME}"
    46-    DOCKER_BUILD_CACHE_ARG="--cache-from type=local,src=${DOCKER_BUILD_CACHE_OLD_DIR} --cache-to type=local,dest=${DOCKER_BUILD_CACHE_NEW_DIR},mode=max"
    47-  fi
    48+  echo "DOCKER_BUILD_CACHE_ARG=$DOCKER_BUILD_CACHE_ARG"
    


    maflcko commented at 1:56 pm on July 18, 2025:
    nit in e5295f7ad64be1d834d3e0c2726edfcc67debfff: The script has tracing enabled, so i don’t think this is needed?

    willcl-ark commented at 9:34 pm on July 25, 2025:
    echo also removed in e83c3718f2c
  33. in .github/workflows/ci.yml:291 in 868ddf5571 outdated
    286@@ -287,7 +287,8 @@ jobs:
    287 
    288   windows-cross:
    289     name: 'Linux->Windows cross, no tests'
    290-    runs-on: ubuntu-latest
    291+    # We hardcode the repo name here, otherwise PRs, which originate "from a fork", will NOT use cirrus runners.
    292+    runs-on: ${{ (github.repository == 'bitcoin/bitcoin' || vars.USE_CIRRUS_RUNNERS == 'true') && 'ghcr.io/cirruslabs/ubuntu-runner-amd64:24.04-md' || 'ubuntu-24.04' }}
    


    maflcko commented at 2:15 pm on July 18, 2025:

    nit in 868ddf557180ffbc7a60ed539a5df1e1cd3e757c: this seems a bit fragile and won’t work for monotree repos ( bitcoin-core/gui), and won’t work for forks that accept pull requests.

    Seems better to just hard-code this to cirrus, possibly add a comment: # Run on cirrus by default, but in forks this can be replaced by 'ubuntu-latest' to run on GHA


    maflcko commented at 2:48 pm on July 18, 2025:
    If you want it to be easy to edit, I wonder if this can be a separate yaml file so that the edit only has to be done in one place?

    maflcko commented at 11:02 am on July 21, 2025:

    Unrelatedly, the USE_CIRRUS_RUNNERS can probably be removed:

    • It is only there for forks, but the comment says it doesn’t work for forks anyway in the expected way.
    • If someone maintains a fork (presumably to add patches), it should be trivial for them to make the required patch to the CI as well.

    m3dwards commented at 12:52 pm on July 25, 2025:

    Ideally CI will run without a patch in as many places as possible.

    By checking the repository name it will run for: pushes/merges to main repo, PRs from forks and pushes to forks (where it would run on a github runner).

    The USE_CIRRUS_RUNNERS enabled a fork to also run on Cirrus if they have bought some runners but would NOT work for a PR from a third fork to that fork. I think this is confusing and so I agree to remove this var. Forks would have to patch if they want to run on cirrus runners.

    Where I think I disagree is on removing the repository name check as then pushes to forks will not run on github’s infrastructure which I think is a really nice to have. The more that people’s personal forks run CI the less strain it will put on the main repo or on Cirrus runners. I know when I work on things it’s nice to see CI running on my fork before creating a PR.

    I’m not sure why the GUI repo should be running bitcoin/bitcoin CI, shouldn’t it have it’s own GUI related CI? But regardless, by keeping the repo name check the GUI repo will be able to run CI but just slower on github’s runners.


    willcl-ark commented at 9:35 pm on July 25, 2025:

    6d252266e40 and 3e757e511a3 have had the USE_CIRRUS_RUNNERS variable removed as it is confusing and doesn’t work on PR’s to downstream forks as expected (which is partially what it is for).

    760497ac393 gives some hints on what a fork should patch, should they wish to use Cirrus Runners on their own CI.

  34. in .github/workflows/ci.yml:318 in c126475ed7 outdated
    316+          echo "BASE_CACHE=${{ runner.temp }}/depends/built" >> $GITHUB_ENV
    317+          echo "DEPENDS_HASH=$(git rev-parse HEAD:depends)" >> $GITHUB_ENV
    318+
    319+      - name: Restore caches
    320+        id: restore-cache
    321+        # TODO: previously this also hashed "ci/test/00_setup_env_win64.sh"
    


    maflcko commented at 2:20 pm on July 18, 2025:
    nit in 868ddf5: Yeah, seems required. Same for the other caches?

    willcl-ark commented at 9:36 pm on July 25, 2025:

    6d252266e40 and 3e757e511a3 now hash FILE_ENV too for the caches, using:

    0DEPENDS_HASH=$(git ls-tree HEAD depends "ci/test/$FILE_ENV" | sha256sum | cut -d' ' -f1)
    
  35. maflcko commented at 2:23 pm on July 18, 2025: member

    looked at c126475ed7a17ec9030066056e31846c7124dcf~20 💇

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: looked at c126475ed7a17ec9030066056e31846c7124dcf~20 💇
    3kep8ZK4UJEamaLijXtMFwgjSmf1fhSJuF49dbZ/NHDe/5jmIZR2EzJa0ewjjGov4n3xWZMN5f3LKRekrMZ6HAw==
    
  36. in .github/workflows/ci.yml:518 in c126475ed7 outdated
    518+
    519+          - name: 'MSan, depends'
    520+            id: 'msan-depends'
    521+            cirrus-runner: 'ghcr.io/cirruslabs/ubuntu-runner-amd64:24.04-lg'
    522+            fallback-runner: 'ubuntu-24.04'
    523+            timeout-minutes: 300
    


    fanquake commented at 10:46 am on July 21, 2025:
    Note to remember to drop the timeout post #32999.

    m3dwards commented at 1:22 pm on July 21, 2025:
    Done
  37. m3dwards force-pushed on Jul 21, 2025
  38. willcl-ark force-pushed on Jul 25, 2025
  39. willcl-ark commented at 9:45 pm on July 25, 2025: member

    Thanks for the review on this so far.

    Whilst we had tested the docker registry caching on PRs successfully, because I was opening them myself (and was owner of the parent repo) repo-level variables were available to me which were not available to 3rd party pull requests.

    The short of this is that this meant the docker registry cache setup could not pull from the registry on pull requests and we didn’t think it was therefore suitable for our purposes.

    We have switched to the gha cache backend in the latest force-push with the good news that we finally have it working for all images correctly; caching to/from on pushes, and caching from on pulls.

    A push to master can be found here: https://github.com/testing-cirrus-runners/bitcoin2/actions/runs/16529929752

    And a pull request (from a 3rd party account) here: https://github.com/testing-cirrus-runners/bitcoin2/actions/runs/16531270443?pr=3

    A new commit has been added, 0bd758e6c54, to fix a (new?) issue we experienced with the asan job where the runner host appeared to update it’s image and kernel. The cached docker image for the ASAN job then had the incorrect linux-headers-<version> package installed, causing the job to fail. There are likely multiple ways to fix this (we could input uname as a docker build arg for example), but simply forcing a reinstall of the correct headers in 03_*.sh seemed easiest. Open to suggestions here however…

    Marking as ready for review now, as I think this is conceptually ready.

  40. willcl-ark marked this as ready for review on Jul 25, 2025
  41. willcl-ark force-pushed on Jul 28, 2025
  42. willcl-ark commented at 9:07 am on July 28, 2025: member

    This push extracts the repeated set CI directories and Set cache paths logic into a configure-environment action

    0      - name: Set CI directories
    1        run: |
    2          echo "BASE_ROOT_DIR=${{ runner.temp }}" >> "$GITHUB_ENV"
    3          echo "BASE_BUILD_DIR=${{ runner.temp }}/build-${{ matrix.build-dir-suffix }}" >> "$GITHUB_ENV"
    4      - name: Set cache paths
    5        run: |
    6          ...
    

    and simplifies the configure-docker action string building to avoid repeated ie/else string-building which is prone to errors when updating.

  43. willcl-ark commented at 9:25 am on July 28, 2025: member

    Some noteworthy observations from the last two weeks of runs:

    1. We have seen variance in network speeds occasionally which has been raised with Cirrus. e.g. in this run we can see a cache restore hanging for 5 minutes:

    It seems to manifest intermittently, and Cirrus are aware and working on it.

    1. We have seen (very rarely) recently jobs take a while to start even when runners are available. This also is known to them and they are workign on fixing, and may even be fixed already:

    There seems to be a slight change in webhook creativeness on GitHub side. We have a potential fix that we plan to deploy this night and keep a close attention over the weekend.

  44. fkorotkov commented at 10:41 am on July 28, 2025: none

    Hey folks,

    Could someone with admin permissions please check that Allow public repositories checkbox is set on this page:

    https://github.com/organizations/bitcoiin/settings/actions/runner-groups/1

    Seems this is the reason runners are not picking jobs at the moment.

  45. fanquake commented at 10:42 am on July 28, 2025: member

    @fkorotkov. Should be fixed now:

  46. willcl-ark force-pushed on Jul 28, 2025
  47. willcl-ark commented at 12:19 pm on July 28, 2025: member

    FWIW the CI run taking place here in this PR is using Cirrus Runners on the migrated jobs (thanks @fkorotkov !) and demonstrates an absolute worst-case runtimeof these jobs with:

    • No docker build cache
    • No ccache cache
    • No depends-sources caches
    • No depends-build caches
  48. in .github/actions/configure-docker/action.yml:15 in 1f5eb93de4 outdated
     5+  steps:
     6+    - name: Set up Docker Buildx
     7+      uses: docker/setup-buildx-action@v3
     8+      with:
     9+        driver-opts: |
    10+          network=host
    


    maflcko commented at 12:36 pm on July 28, 2025:
    nit in 1f5eb93de45f6a530ba54d799f637c0f1b45bc1b: Could add a comment why this is needed? I guess due to GHA not supporting ipv6 in $current_year?

    willcl-ark commented at 9:35 pm on July 28, 2025:

    I am actually not 100% sure why this is needed. Warp have it documented in their docs, but IIRC Cirrus also advised us to set it.

    My understanding is that it was necessary for the gha caching using custom URL, but perhaps @m3dwards or @fkorotkov could clarify/confirm?


    m3dwards commented at 1:43 pm on August 5, 2025:
    Added a comment. The docker build container uses gha style cache which is available on the host at http://127.0.0.1:12321/

    maflcko commented at 2:12 pm on August 5, 2025:
    Thx, but this reminds me that the setting the min port is missing? #32989 (review)

    m3dwards commented at 6:05 pm on August 13, 2025:
    replied in: #32989 (review)

    maflcko commented at 7:26 am on August 21, 2025:

    (resolving thread for now) edit:

    “Failed to resolve thread”

  49. in .github/actions/configure-docker/action.yml:30 in 1f5eb93de4 outdated
    25+        # On forks the gha cache will work but will use Github's cache backend.
    26+        # Docker will check for variables $ACTIONS_CACHE_URL, $ACTIONS_RESULTS_URL and $ACTIONS_RUNTIME_TOKEN
    27+        # which are set automatically when running on GitHub infra: https://docs.docker.com/build/cache/backends/gha/#synopsis
    28+
    29+        # Use cirrus cache host in main repo
    30+        if [[ "$GITHUB_REPOSITORY" == "bitcoin/bitcoin" ]]; then
    


    maflcko commented at 12:59 pm on July 28, 2025:

    nit in 1f5eb93de45f6a530ba54d799f637c0f1b45bc1b: Seems like there are several places that hard-code the main repo, which makes it harder for forks to use cirrus runners.

    Maybe a global env next to CIRRUS_CACHE_HOST could be used to make that simpler?

    Maybe REPO_USE_CIRRUS_RUNNERS: 'bitcoin/bitcoin' # Use cirrus runners and cache for this repo, instead of falling back to the slow GHA runners?


    willcl-ark commented at 3:42 pm on July 28, 2025:

    We did try this originally, but (very annoyingly) it does not seem to be possible without a primary workflow to call this workflow. The reason is that the runs-on field only has access to the following “contexts”:

    0github, needs, strategy, matrix, vars, inputs
    

    See https://docs.github.com/en/actions/reference/workflows-and-actions/contexts

    Whilst vars sounds like it would do the job, the problem is that in workflows of the type on: pull_request that vars (which refer here to “repo vars”, not “workflow level env vars”) are not available to pull requests. They are available to on: pull_request_target wofklows, but we don’t want to go down that route…

    Sadly, I can’t even think of a nice-enough hack around this without doing something like:

    1. Create a preliminary workflow which will determine whether to use cirrus runners or not based on an env var.
    2. Change our main workflow type to be on: workflow_call:, where it can then be called from the preliminary workflow, passing with it the information using inputs on whether or not to use cirrus runners.

    I hope you agree with me such spaghettic is not worth it.

    I think I can minimise the usage of the hardcoding to a single file (ci.yml) though so that a patch is easy to make and maintain…


    willcl-ark commented at 7:35 pm on July 28, 2025:

    I hope you agree with me such spaghetti is not worth it.

    Actually, this turns out to be a lot less invasive than I thought, and the job only takes ~4 seconds to run: https://github.com/testing-cirrus-runners/bitcoin/actions/runs/16578429043

    The job is pretty simple: https://github.com/testing-cirrus-runners/bitcoin/commit/9f3fdf420ea56d32db3df98bef2026177afb4067

    Perhaps we could use this approach after all?


    willcl-ark commented at 9:36 pm on July 28, 2025:

    Added a helper job (runtime of ~4 seconds) which has an output whether or not to use cirrus to the subsequent jobs in 8758b6ec333 ci: add job to determine runner type

    This reduces the hardcoding to a single env var in the main ci.yml file, which can trivially be patched by forks if desired.


    maflcko commented at 9:41 am on July 30, 2025:

    Heh, yeah. Somewhat it seems plausible that the environment may not be available before the vm boots up. A ~4 second delay should be acceptable. And it would be trivial to revert to hardcoding thrice, if there is need to.

    (Feel free to resolve)

  50. in .github/workflows/ci.yml:289 in e9cf1c9005 outdated
    284@@ -285,7 +285,8 @@ jobs:
    285 
    286   windows-cross:
    287     name: 'Linux->Windows cross, no tests'
    288-    runs-on: ubuntu-latest
    289+    # We hardcode the repo name here, otherwise PRs, which originate "from a fork", will NOT use cirrus runners.
    290+    runs-on: ${{ github.repository == 'bitcoin/bitcoin' && 'ghcr.io/cirruslabs/ubuntu-runner-amd64:24.04-md' || 'ubuntu-24.04' }}
    


    maflcko commented at 1:00 pm on July 28, 2025:
    nit in e9cf1c90052620b23745ca0d3fdccda2cf5fa0fc: Same, about REPO_USE_CIRRUS_RUNNERS
  51. in .github/workflows/ci.yml:446 in 35a9dd7352 outdated
    439@@ -440,3 +440,52 @@ jobs:
    440           path: ${{ env.CCACHE_DIR }}
    441           # https://github.com/actions/cache/blob/main/tips-and-workarounds.md#update-a-cache
    442           key: ${{ github.job }}-ccache-${{ github.run_id }}
    443+
    444+  ci-matrix:
    445+    name: ${{ matrix.name }}
    446+    runs-on: ${{ github.repository == 'bitcoin/bitcoin' && matrix.cirrus-runner || matrix.fallback-runner }}
    


    maflcko commented at 1:38 pm on July 28, 2025:
    nit in 35a9dd7352d84292fbc2dcf2e544b380be49b4c3: Same (REPO_USE_CIRRUS_RUNNERS)
  52. in .github/workflows/ci.yml:422 in 35a9dd7352 outdated
    447+
    448+    if: ${{ vars.SKIP_BRANCH_PUSH != 'true' || github.event_name == 'pull_request' }}
    449+    timeout-minutes: ${{ matrix.timeout-minutes }}
    450+
    451+    env:
    452+      DANGER_CI_ON_HOST_FOLDERS: 1
    


    maflcko commented at 1:41 pm on July 28, 2025:
    nit in 35a9dd7352d84292fbc2dcf2e544b380be49b4c3: Could just be a global env var, instead of repeating it thrice?
  53. in .github/workflows/ci.yml:465 in 35a9dd7352 outdated
    460+            id: '32bit-arm'
    461+            cirrus-runner: 'ghcr.io/cirruslabs/ubuntu-runner-arm64:24.04-md'
    462+            fallback-runner: 'ubuntu-24.04-arm'
    463+            timeout-minutes: 120
    464+            file-env: './ci/test/00_setup_env_arm.sh'
    465+            build-dir-suffix: 'arm'
    


    maflcko commented at 1:45 pm on July 28, 2025:
    nit in 35a9dd7352d84292fbc2dcf2e544b380be49b4c3: Unused? Maybe you wanted this to be the container name, or the job id instead?

    willcl-ark commented at 9:37 pm on July 28, 2025:
    Removed dead code no-longer needed in 3825f903585 ci: port arm 32-bit job (and subsequent job commits)
  54. in .github/actions/restore-caches/action.yml:6 in c7aa954564 outdated
    0@@ -0,0 +1,47 @@
    1+name: 'Restore Caches'
    2+description: 'Restore ccache, depends sources, and built depends caches'
    3+inputs:
    4+  job-id:
    5+    description: 'job id'
    6+    required: true
    


    maflcko commented at 1:55 pm on July 28, 2025:
    nit in c7aa954564ceaf0800be461115a568d3c919228a: This seems somewhat redundant with the container name now. It would be less yaml code to use container name instead of job id here?

    willcl-ark commented at 9:37 pm on July 28, 2025:

    Thanks, I agree.

    Removed the input entirely and use $CONTAINER_NAME in d6ed832034b ci: add caching actions

  55. in ci/test/03_test_script.sh:30 in 88816a4ed3 outdated
    23@@ -24,6 +24,12 @@ fi
    24 echo "Free disk space:"
    25 df -h
    26 
    27+# We force an install of linux-headers again here to fix any kernel mismatch
    28+# between a cached docker image and the underlying host
    29+if [[ "$CONTAINER_NAME" == "ci_native_asan" ]]; then
    30+  $CI_RETRY_EXE apt-get install -y bpfcc-tools linux-headers-"$(uname --kernel-release)"
    


    maflcko commented at 2:01 pm on July 28, 2025:
    nit in 88816a4ed3f5d528ed7b997241df96fb1f51df44: apt-get install will fail with a stale Ubuntu IP list from a cached image. You’ll have to call apt-get update before every install for each image layer.

    maflcko commented at 2:03 pm on July 28, 2025:
    Also, I wonder if this can use $PACKAGES?

    willcl-ark commented at 10:11 pm on July 28, 2025:
    Fixed, and now using $PACKAGES in 8bb213c64c3 ci: force reinstall of kernel headers in asan
  56. in .github/workflows/ci.yml:495 in a3789f2056 outdated
    490+            id: 'tsan-depends-gui'
    491+            cirrus-runner: 'ghcr.io/cirruslabs/ubuntu-runner-amd64:24.04-lg'
    492+            fallback-runner: 'ubuntu-24.04'
    493+            timeout-minutes: 120
    494+            file-env: './ci/test/00_setup_env_native_tsan.sh'
    495+            build-dir-suffix: 'tidy'
    


    maflcko commented at 2:07 pm on July 28, 2025:
    nit in a3789f2056a76e72fcafbae7396c35f62fb93f56: wrong (unused) suffix

    willcl-ark commented at 9:38 pm on July 28, 2025:
    Removed in e62bd1b24b6 ci: port tsan-depends-gui
  57. in .github/workflows/ci.yml:561 in 67429ff9ae outdated
    556+      - name: Configure Docker
    557+        uses: ./.github/actions/configure-docker
    558+
    559+      - name: CI script
    560+        run: |
    561+          set -x
    


    maflcko commented at 2:11 pm on July 28, 2025:
    nit in 67429ff9ae0ad7236d01fee7d25dceb965741b2f: Please use -o xtrace, so that non-Bash experts can also understand the intention, without having to ask an LLM read the manpage

    willcl-ark commented at 9:58 pm on July 28, 2025:
    Fixed in fd32aa85b29 ci: port lint
  58. in .cirrus.yml:5 in 72d82211fc outdated
    -1@@ -1,199 +0,0 @@
    0-env:  # Global defaults
    1-  CIRRUS_CLONE_DEPTH: 1
    2-  CIRRUS_LOG_TIMESTAMP: true
    3-  MAKEJOBS: "-j10"
    4-  TEST_RUNNER_PORT_MIN: "14000"  # Must be larger than 12321, which is used for the http cache. See https://cirrus-ci.org/guide/writing-tasks/#http-cache
    


    maflcko commented at 2:13 pm on July 28, 2025:
    Just as a note: This is fine to remove without replacement, because the cirrus cache now runs outside the container. Here, it ran inside it.

    maflcko commented at 2:12 pm on August 5, 2025:
    Actually, with network=host, this is needed, no?

    m3dwards commented at 6:04 pm on August 13, 2025:

    I’m not sure it’s needed still, the network=host is applied to the buildx container which is only used to build the images and isn’t actually used to run the CI images.

    The main CI container mount cache directories on the host and then separate steps restore and save from those directories on the host without docker.

    The buildx container needs to access the http cache to look for cached build layers.


    maflcko commented at 6:42 am on August 21, 2025:
    Ah, ok. I am not familiar with the GitHub Actions Docker Caching Buildx setup interactions with the docker containers that are run later. But given that this is documented as a buildx option (https://docs.docker.com/reference/cli/docker/buildx/create/#driver-opt) it seems plausible that it may not apply to the docker container.

    maflcko commented at 7:11 am on August 21, 2025:

    Also, it should be fine to re-apply once the CI intermittently fails due to this. So this pull looks good as-is and no change is needed here.

    Unrelatedly, if we want to keep/use the CI setting in the future, it must be documented as a hidden option:

    https://github.com/bitcoin/bitcoin/blob/7d9789401be4c91a9eb3c1112564a6524bdc4f70/ci/test/02_run_container.py#L29-L34

  59. in .github/workflows/ci.yml:528 in 67429ff9ae outdated
    538@@ -539,3 +539,25 @@ jobs:
    539           job-id: ${{ matrix.id }}
    540           depends-sources-cache-hit: ${{ steps.restore-cache.outputs.depends-sources-cache-hit }}
    541           depends-built-cache-hit: ${{ steps.restore-cache.outputs.depends-built-cache-hit }}
    542+
    543+  lint:
    


    maflcko commented at 2:22 pm on July 28, 2025:
    67429ff9ae0ad7236d01fee7d25dceb965741b2f: This is wrong and will silently pass on any input. You must set COMMIT_RANGE=, like it is done in the script today.

    willcl-ark commented at 10:09 pm on July 28, 2025:
    Set this to run on PRs in fd32aa85b29 ci: port lint by setting CIRRUS_PR=1 via if [ ${{ github.event_name }}" = "pull_request" ]; then which I think should fix this?

    maflcko commented at 9:37 am on July 30, 2025:

    Set this to run on PRs in fd32aa8 ci: port lint by setting CIRRUS_PR=1 via if [ ${{ github.event_name }}" = "pull_request" ]; then which I think should fix this?

    Yes, this should fix it. This will correctly set the COMMIT_RANGE, just like it did previously. Thx.

  60. in ci/test/00_setup_env.sh:38 in 693d3615cd outdated
    34@@ -35,7 +35,7 @@ fi
    35 
    36 echo "Fallback to default values in env (if not yet set)"
    37 # The number of parallel jobs to pass down to make and test_runner.py
    38-export MAKEJOBS=${MAKEJOBS:--j4}
    39+export MAKEJOBS=${MAKEJOBS:--j$(if command -v nproc > /dev/null 2>&1; then nproc; else sysctl -n hw.logicalcpu; fi)}
    


    maflcko commented at 2:31 pm on July 28, 2025:
    693d3615cdb25c5f185c1e6097a6e647eca7153b: This commit isn’t doing anything, because the MAKEJOBS are set globally in the env in GHA (same for the other “perf” commits below).

    willcl-ark commented at 10:42 pm on July 28, 2025:
    Removed the global makejobs in 89f177406c9 ci: dynamically match makejobs with cores
  61. maflcko commented at 2:32 pm on July 28, 2025: member

    there is a bug in the lint config. Also, left some nits.

    looked at 72d82211fce78a18bce27a90726d30c85518955c~6 📊

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: looked at 72d82211fce78a18bce27a90726d30c85518955c~6 📊
    3bU4/CudCrcvil8aKqUlXXpLa97DY3Kj7FWx5lSSAhICRDONbCHC2zwPtEQusEF8btAqpICvOG7VJLZX+6A5wCg==
    
  62. willcl-ark force-pushed on Jul 28, 2025
  63. willcl-ark force-pushed on Jul 28, 2025
  64. in ci/test/03_test_script.sh:27 in 3c7b462fa9 outdated
    23@@ -24,6 +24,14 @@ fi
    24 echo "Free disk space:"
    25 df -h
    26 
    27+# We force an install of linux-headers again here via $PACAKGES to fix any
    


    maflcko commented at 6:16 am on July 29, 2025:

    nit (from the llm):

    PACAKGES -> PACKAGES [env var name typo]
    

    willcl-ark commented at 10:04 am on July 30, 2025:
    Fixed in 8dd7476d496
  65. willcl-ark force-pushed on Jul 29, 2025
  66. willcl-ark force-pushed on Jul 29, 2025
  67. willcl-ark commented at 2:46 pm on July 29, 2025: member

    I pushed a commit to make a cache save on pull requests, and then a force push dropping that commit. This worked, but I do see some unexpected ccache hit rates… not sure if anyone here might have an idea about it; I’m at a bit of loss about what could be going wrong here to be honest.

    In e.g. the previous releases job, the cache was saved here: https://github.com/bitcoin/bitcoin/actions/runs/16589926678/job/46922852038?pr=32989#step:9:77

    Sent 426321843 of 426321843 (100.0%), 81.3 MBs/sec Cache saved with key: ccache-ci_native_previous_releases-16589926678

    This is the correct size for this cache; the CI run build ended with:

    Primary storage: Hits: 0 / 1494 (0.00 %) Misses: 1494 Cache size (GB): 0.43 / 0.50 (86.19 %) Cleanups: 20

    We then in the next run, where I only dropped the temporary cache-save commit, restored this exact cache: https://github.com/bitcoin/bitcoin/actions/runs/16595726917/job/46941763412?pr=32989#step:4:49 (note that we don’t hit on primary key, but hit on a fallback, final line, which matches exactly the cache save above):

    Cache hit for: ccache-ci_native_previous_releases-16595726917 Cache Size: ~407 MB (426321843 B) Cache restored successfully Cache restored from key: ccache-ci_native_previous_releases-16589926678

    In fact we know this is the correct cache as this is currently the only cache that has ever been saved for this job. The problem is that later on in this re-run our ccache hit-rate was a meagre 19%: https://github.com/bitcoin/bitcoin/actions/runs/16595726917/job/46941763412?pr=32989#step:8:2986

    ccache version 4.5.1 Summary: Hits: 148 / 747 (19.81 %) Direct: 54 / 747 (7.23 %) Preprocessed: 94 / 693 (13.56 %) Misses: 599 Direct: 693 Preprocessed: 599 Primary storage: Hits: 360 / 1624 (22.17 %) Misses: 1264 Cache size (GB): 0.43 / 0.50 (85.79 %) Cleanups: 64

    I’m not sure exactly what is causing the misses here; I would expect this to be 100% (or 99.x%) which I have seen previously.

  68. maflcko commented at 3:02 pm on July 29, 2025: member

    I’m not sure exactly what is causing the misses here; I would expect this to be 100% (or 99.x%) which I have seen previously.

    ccache does not work when the code changes, so you’ll have to make sure to compile the same code (usually commit id). Ref:

    This means the code was https://github.com/bitcoin/bitcoin/commit/953c90d7649c076e6e43418f50c33b3ae5f82608 vs https://github.com/bitcoin/bitcoin/commit/321984705dbcc5982013c8cfb52389cf3f2e75f6

  69. willcl-ark commented at 3:29 pm on July 29, 2025: member

    This means the code was 953c90d vs 3219847

    Dang, I must have rebased on upstream/master (out of habit) to drop that top commit. Thanks for catching.

  70. maflcko commented at 6:08 pm on July 29, 2025: member

    The checkout action by default picks the “rebase”/merge with master, so this should happen even without you rebasing. In fact, it should happen even when not pushing at all, but just re-running the task. However, I just noticed this does not happen (https://github.com/bitcoin/bitcoin/actions/runs/16521447783/job/46969566419?pr=29641#step:2:90 picks a 4-day old merge). This breaks the use-case of being able to detect silent merge conflicts before a maintainer detects them (or misses them).

    Not a blocker, but I think it would be nice to fix this, if possible.

    Edit: for more context, on cirrus the merge_base works correctly: https://cirrus-ci.com/task/6351223885922304?logs=merge_base#L7

  71. in .cirrus.yml:107 in 4f4d22a150 outdated
    101-  # For faster CI feedback, immediately schedule the linters
    102-  << : *CREDITS_TEMPLATE
    103-  unshallow_script:
    104-    - git fetch --unshallow --no-tags
    105-  lint_script:
    106-    - ./ci/lint_run.sh
    


    maflcko commented at 7:32 am on July 30, 2025:

    You can also remove the now unused stuff:

    0ci/lint_run.sh:# Only used in .cirrus.yml. Refer to test/lint/README.md on how to run locally.
    1ci/lint_run_all.sh:# Only used in .cirrus.yml for stale re-runs of old pull request tasks. This
    
  72. in .github/workflows/ci.yml:54 in 01746e60eb outdated
    50@@ -52,6 +51,7 @@ jobs:
    51     if: github.event_name == 'pull_request' && github.event.pull_request.commits != 1
    52     timeout-minutes: 360  # Use maximum time, see https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#jobsjob_idtimeout-minutes. Assuming a worst case time of 1 hour per commit, this leads to a --max-count=6 below.
    53     env:
    54+      MAKEJOBS: '-j10'
    


    maflcko commented at 7:39 am on July 30, 2025:
    nit in 01746e60eb82b9bedb3afa1e791b5abce75ebf75: Unused? This task is using nproc?

    willcl-ark commented at 9:56 am on July 30, 2025:
    Correct, this is dropped in a81bf5c25a8
  73. in .github/workflows/ci.yml:134 in 01746e60eb outdated
    130@@ -131,6 +131,7 @@ jobs:
    131     env:
    132       DANGER_RUN_CI_ON_HOST: 1
    133       BASE_ROOT_DIR: ${{ github.workspace }}
    134+      MAKEJOBS: '-j10'
    


    maflcko commented at 7:41 am on July 30, 2025:
    nit in https://github.com/bitcoin/bitcoin/commit/01746e60eb82b9bedb3afa1e791b5abce75ebf75: Any reason to change this setting for the other tasks, but not for this one? For reference, the task has 3 mac-arm cpu threads.

    willcl-ark commented at 9:57 am on July 30, 2025:
    Also dropped in a81bf5c
  74. in ci/README.md:79 in 4a1c5b0469 outdated
    75+### Forked repositories
    76+
    77+When used in a fork the CI will run on GitHub's free hosted runners by default.
    78+In this case, due to GitHub's 10GB-per-repo cache size limitations caches will be frequently evicted and missed, but the workflows will run (slowly).
    79+
    80+It is also possible to use your own Cirrus Runners in your own fork with an appropriate patch to the ../.github/workflows/ci.yml file and the actions in ../.github/actions/**/*.yml.
    


    maflcko commented at 7:48 am on July 30, 2025:
    nit in 4a1c5b04691266f1441fd966a4be65140f9a5cc4: Outdated doc?

    willcl-ark commented at 9:57 am on July 30, 2025:
    Thanks, updated the doc in c7c87775768
  75. maflcko approved
  76. maflcko commented at 8:01 am on July 30, 2025: member

    everything looks good and this is probably ready for final review before merge? I just found some nits, which can be done along with follow-ups like #32989 (review). Also, the merge_base issue mentioned above can be addressed in a follow-up.

    However, I am not familiar with the GitHub-style CI, including the (docker) caching, but the pull request motivation claims that many are familiar with this, so it should be easy to find reviewers for it, before merging it.

    review ACK b6cf0b325793184e7824f7dc488a68f4908ed0c6 📦

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: review ACK b6cf0b325793184e7824f7dc488a68f4908ed0c6 📦
    34CWuGfE/k0O4gs/FcFN2Btaa/mALDAFjwg+EWCP2/wwWeM+xx/1ZmsAOtqxTrTh9V6NvX78Gw+wzGvFrDG65Dg==
    
  77. DrahtBot requested review from fanquake on Jul 30, 2025
  78. DrahtBot requested review from achow101 on Jul 30, 2025
  79. DrahtBot requested review from 0xB10C on Jul 30, 2025
  80. DrahtBot requested review from hebasto on Jul 30, 2025
  81. willcl-ark force-pushed on Jul 30, 2025
  82. in .github/workflows/ci.yml:133 in a81bf5c25a outdated
    129@@ -131,6 +130,7 @@ jobs:
    130     env:
    131       DANGER_RUN_CI_ON_HOST: 1
    132       BASE_ROOT_DIR: ${{ github.workspace }}
    133+      CONTAINER_NAME: ${{ matrix.job-name }}
    


    maflcko commented at 10:03 am on July 30, 2025:
    nit in a81bf5c25a8dc0b3a2f8458380921b5026726d64: Looks like a unrelated and unneeded change?

    willcl-ark commented at 10:08 am on July 30, 2025:

    Oh, this has sneaked into the wrong commit!

    It is “needed” by b49dd9890b9 to print the container name for these jobs when we have a low ccache hitrate warning in the CI annotations (bottom of the page on the run summary). Otherwise they are printed as:

    “low ccache hitrate in

    Will fix it up


    maflcko commented at 10:11 am on July 30, 2025:

    Ah, but then the yaml seems like the wrong place. Would be better to put in in the sh:

     0diff --git a/ci/test/00_setup_env_mac_native_fuzz.sh b/ci/test/00_setup_env_mac_native_fuzz.sh
     1index cacf2423ac..c79b48c8ed 100755
     2--- a/ci/test/00_setup_env_mac_native_fuzz.sh
     3+++ b/ci/test/00_setup_env_mac_native_fuzz.sh
     4@@ -6,6 +6,7 @@
     5 
     6 export LC_ALL=C.UTF-8
     7 
     8+bla...insert...here  # macos does not use a container, but the env var is needed for logging
     9 export CMAKE_GENERATOR="Ninja"
    10 export BITCOIN_CONFIG="-DBUILD_FOR_FUZZING=ON"
    11 export CI_OS_NAME="macos"
    

    willcl-ark commented at 10:14 am on July 30, 2025:

    But in this case we won’t get:

    1. ‘macOS 14 native, arm64, no depends, sqlite only, gui’
    2. ‘macOS 14 native, arm64, fuzz’

    But will fix to a single name for both jobs (not a massive issue perhaps, it’s only an informational annotation?)


    maflcko commented at 10:16 am on July 30, 2025:
    Yes, container names must be unique over all configs and there are two macos configs (ci/test/00_setup_env_mac_native.sh) and (ci/test/00_setup_env_mac_native_fuzz.sh), so it should be possible. But just a nit. Anything is fine here.

    willcl-ark commented at 2:32 pm on July 31, 2025:
    Fixed in 1587a7baa96 ci: dynamically match makejobs with cores
  83. in .github/workflows/ci.yml:22 in b6cf0b3257 outdated
    18@@ -19,7 +19,8 @@ concurrency:
    19 
    20 env:
    21   CI_FAILFAST_TEST_LEAVE_DANGLING: 1  # GHA does not care about dangling processes and setting this variable avoids killing the CI script itself on error
    22-  MAKEJOBS: '-j10'
    23+  CIRRUS_CACHE_HOST: http://127.0.0.1:12321/ # When using Cirrus Runners this host can be used as with the docker `gha` build cache type.
    


    maflcko commented at 10:21 am on July 30, 2025:

    from the llm:

    used as with -> used with [“used as with” is redundant and makes the phrase unclear]
    

    willcl-ark commented at 11:07 am on July 30, 2025:
    Will get with the above if I re-touch

    willcl-ark commented at 2:32 pm on July 31, 2025:
    fixed in 7a33a8bea13 ci: add Cirrus cache host
  84. DrahtBot renamed this:
    Migrate CI to hosted Cirrus Runners
    ci: Migrate CI to hosted Cirrus Runners
    on Jul 30, 2025
  85. DrahtBot added the label Tests on Jul 30, 2025
  86. willcl-ark force-pushed on Jul 31, 2025
  87. willcl-ark force-pushed on Jul 31, 2025
  88. willcl-ark commented at 2:32 pm on July 31, 2025: member
    Pushed to fix the merge conflict, and addressed nits
  89. in .github/workflows/ci.yml:486 in fd5e807fca outdated
    486+            cirrus-runner: 'ghcr.io/cirruslabs/ubuntu-runner-amd64:24.04-md'
    487+            fallback-runner: 'ubuntu-24.04'
    488+            timeout-minutes: 120
    489+            file-env: './ci/test/00_setup_env_native_tidy.sh'
    490+
    491+          - name: 'TSan, depends, gui'
    


    fanquake commented at 2:35 pm on July 31, 2025:
    Should be no gui after #33099

    willcl-ark commented at 10:49 am on August 4, 2025:
    Removed in dc1613169db
  90. fanquake commented at 2:47 pm on July 31, 2025: member

    TSAN failure (https://github.com/bitcoin/bitcoin/actions/runs/16651908244/job/47126650397?pr=32989#step:8:6056):

      02025-07-31T14:39:40.396402Z [test] [net.cpp:2419] [void CConnman::SetTryNewOutboundPeer(bool)] [net] setting try another outbound peer=false
      12025-07-31T14:39:40.396438Z [test] [net.cpp:3192] [void==================
      2WARNING: ThreadSanitizer: data race (pid=11584)
      3  Write of size 8 at 0x7234000016f0 by main thread:
      4    [#0](/bitcoin-bitcoin/0/) blockmanager_tests::blockmanager_readblock_hash_mismatch::test_method() /home/admin/actions-runner/_work/_temp/build/src/test/./test/blockmanager_tests.cpp:143:28 (test_bitcoin+0x43dce3) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
      5    [#1](/bitcoin-bitcoin/1/) blockmanager_tests::blockmanager_readblock_hash_mismatch_invoker() /home/admin/actions-runner/_work/_temp/build/src/test/./test/blockmanager_tests.cpp:140:1 (test_bitcoin+0x43d6eb) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
      6    [#2](/bitcoin-bitcoin/2/) boost::detail::function::void_function_invoker<void (*)(), void>::invoke(boost::detail::function::function_buffer&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:59:11 (test_bitcoin+0x3275cd) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
      7    [#3](/bitcoin-bitcoin/3/) boost::function_n<void>::operator()() const /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:789:14 (test_bitcoin+0x2a2618) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
      8    [#4](/bitcoin-bitcoin/4/) boost::detail::forward::operator()() /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1416:32 (test_bitcoin+0x2a2618)
      9    [#5](/bitcoin-bitcoin/5/) boost::detail::function::function_obj_invoker<boost::detail::forward, int>::invoke(boost::detail::function::function_buffer&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:79:18 (test_bitcoin+0x2a2618)
     10    [#6](/bitcoin-bitcoin/6/) boost::function_n<int>::operator()() const /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:789:14 (test_bitcoin+0x226873) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     11    [#7](/bitcoin-bitcoin/7/) int boost::detail::do_invoke<boost::shared_ptr<boost::detail::translator_holder_base>, boost::function<int ()>>(boost::shared_ptr<boost::detail::translator_holder_base> const&, boost::function<int ()> const&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:329:30 (test_bitcoin+0x226873)
     12    [#8](/bitcoin-bitcoin/8/) boost::execution_monitor::catch_signals(boost::function<int ()> const&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:931:16 (test_bitcoin+0x226873)
     13    [#9](/bitcoin-bitcoin/9/) boost::execution_monitor::execute(boost::function<int ()> const&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1329:16 (test_bitcoin+0x226bea) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     14    [#10](/bitcoin-bitcoin/10/) boost::execution_monitor::vexecute(boost::function<void ()> const&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1425:5 (test_bitcoin+0x2222cc) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     15    [#11](/bitcoin-bitcoin/11/) boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned long) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/unit_test_monitor.ipp:49:9 (test_bitcoin+0x2222cc)
     16    [#12](/bitcoin-bitcoin/12/) boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:815:44 (test_bitcoin+0x255e8d) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     17    [#13](/bitcoin-bitcoin/13/) boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:784:58 (test_bitcoin+0x2563eb) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     18    [#14](/bitcoin-bitcoin/14/) boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:784:58 (test_bitcoin+0x2563eb) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     19    [#15](/bitcoin-bitcoin/15/) boost::unit_test::framework::run(unsigned long, bool) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:1722:29 (test_bitcoin+0x220dc1) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     20    [#16](/bitcoin-bitcoin/16/) boost::unit_test::unit_test_main(boost::unit_test::test_suite* (*)(int, char**), int, char**) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/unit_test_main.ipp:250:9 (test_bitcoin+0x23bd3c) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     21    [#17](/bitcoin-bitcoin/17/) main /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/unit_test_main.ipp:306:12 (test_bitcoin+0x23c5c3) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     22
     23  Previous read of size 8 at 0x7234000016f0 by thread T1:
     24    [#0](/bitcoin-bitcoin/0/) CBlockIndex::GetBlockHash() const /home/admin/actions-runner/_work/_temp/build/src/./chain.h:245:9 (test_bitcoin+0x13700f1) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     25    [#1](/bitcoin-bitcoin/1/) ValidationSignals::UpdatedBlockTip(CBlockIndex const*, CBlockIndex const*, bool)::$_0::operator()() const /home/admin/actions-runner/_work/_temp/build/src/./validationinterface.cpp:179:5 (test_bitcoin+0x13700f1)
     26    [#2](/bitcoin-bitcoin/2/) decltype(std::declval<ValidationSignals::UpdatedBlockTip(CBlockIndex const*, CBlockIndex const*, bool)::$_0&>()()) std::__1::__invoke[abi:de200100]<ValidationSignals::UpdatedBlockTip(CBlockIndex const*, CBlockIndex const*, bool)::$_0&>(ValidationSignals::UpdatedBlockTip(CBlockIndex const*, CBlockIndex const*, bool)::$_0&) /cxx_build/include/c++/v1/__type_traits/invoke.h:179:25 (test_bitcoin+0x13700f1)
     27    [#3](/bitcoin-bitcoin/3/) void std::__1::__invoke_void_return_wrapper<void, true>::__call[abi:de200100]<ValidationSignals::UpdatedBlockTip(CBlockIndex const*, CBlockIndex const*, bool)::$_0&>(ValidationSignals::UpdatedBlockTip(CBlockIndex const*, CBlockIndex const*, bool)::$_0&) /cxx_build/include/c++/v1/__type_traits/invoke.h:251:5 (test_bitcoin+0x13700f1)
     28    [#4](/bitcoin-bitcoin/4/) void std::__1::__invoke_r[abi:de200100]<void, ValidationSignals::UpdatedBlockTip(CBlockIndex const*, CBlockIndex const*, bool)::$_0&>(ValidationSignals::UpdatedBlockTip(CBlockIndex const*, CBlockIndex const*, bool)::$_0&) /cxx_build/include/c++/v1/__type_traits/invoke.h:273:10 (test_bitcoin+0x13700f1)
     29    [#5](/bitcoin-bitcoin/5/) std::__1::__function::__alloc_func<ValidationSignals::UpdatedBlockTip(CBlockIndex const*, CBlockIndex const*, bool)::$_0, std::__1::allocator<ValidationSignals::UpdatedBlockTip(CBlockIndex const*, CBlockIndex const*, bool)::$_0>, void ()>::operator()[abi:de200100]() /cxx_build/include/c++/v1/__functional/function.h:167:12 (test_bitcoin+0x13700f1)
     30    [#6](/bitcoin-bitcoin/6/) std::__1::__function::__func<ValidationSignals::UpdatedBlockTip(CBlockIndex const*, CBlockIndex const*, bool)::$_0, std::__1::allocator<ValidationSignals::UpdatedBlockTip(CBlockIndex const*, CBlockIndex const*, bool)::$_0>, void ()>::operator()() /cxx_build/include/c++/v1/__functional/function.h:319:10 (test_bitcoin+0x13700f1)
     31    [#7](/bitcoin-bitcoin/7/) std::__1::__function::__value_func<void ()>::operator()[abi:de200100]() const /cxx_build/include/c++/v1/__functional/function.h:436:12 (test_bitcoin+0x183b982) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     32    [#8](/bitcoin-bitcoin/8/) std::__1::function<void ()>::operator()() const /cxx_build/include/c++/v1/__functional/function.h:995:10 (test_bitcoin+0x183b982)
     33    [#9](/bitcoin-bitcoin/9/) SerialTaskRunner::ProcessQueue() /home/admin/actions-runner/_work/_temp/build/src/./scheduler.cpp:173:5 (test_bitcoin+0x183b982)
     34    [#10](/bitcoin-bitcoin/10/) SerialTaskRunner::MaybeScheduleProcessQueue()::$_0::operator()() const /home/admin/actions-runner/_work/_temp/build/src/./scheduler.cpp:142:41 (test_bitcoin+0x183d4e5) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     35    [#11](/bitcoin-bitcoin/11/) decltype(std::declval<SerialTaskRunner::MaybeScheduleProcessQueue()::$_0&>()()) std::__1::__invoke[abi:de200100]<SerialTaskRunner::MaybeScheduleProcessQueue()::$_0&>(SerialTaskRunner::MaybeScheduleProcessQueue()::$_0&) /cxx_build/include/c++/v1/__type_traits/invoke.h:179:25 (test_bitcoin+0x183d4e5)
     36    [#12](/bitcoin-bitcoin/12/) void std::__1::__invoke_void_return_wrapper<void, true>::__call[abi:de200100]<SerialTaskRunner::MaybeScheduleProcessQueue()::$_0&>(SerialTaskRunner::MaybeScheduleProcessQueue()::$_0&) /cxx_build/include/c++/v1/__type_traits/invoke.h:251:5 (test_bitcoin+0x183d4e5)
     37    [#13](/bitcoin-bitcoin/13/) void std::__1::__invoke_r[abi:de200100]<void, SerialTaskRunner::MaybeScheduleProcessQueue()::$_0&>(SerialTaskRunner::MaybeScheduleProcessQueue()::$_0&) /cxx_build/include/c++/v1/__type_traits/invoke.h:273:10 (test_bitcoin+0x183d4e5)
     38    [#14](/bitcoin-bitcoin/14/) std::__1::__function::__alloc_func<SerialTaskRunner::MaybeScheduleProcessQueue()::$_0, std::__1::allocator<SerialTaskRunner::MaybeScheduleProcessQueue()::$_0>, void ()>::operator()[abi:de200100]() /cxx_build/include/c++/v1/__functional/function.h:167:12 (test_bitcoin+0x183d4e5)
     39    [#15](/bitcoin-bitcoin/15/) std::__1::__function::__func<SerialTaskRunner::MaybeScheduleProcessQueue()::$_0, std::__1::allocator<SerialTaskRunner::MaybeScheduleProcessQueue()::$_0>, void ()>::operator()() /cxx_build/include/c++/v1/__functional/function.h:319:10 (test_bitcoin+0x183d4e5)
     40    [#16](/bitcoin-bitcoin/16/) std::__1::__function::__value_func<void ()>::operator()[abi:de200100]() const /cxx_build/include/c++/v1/__functional/function.h:436:12 (test_bitcoin+0x183a8ef) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     41    [#17](/bitcoin-bitcoin/17/) std::__1::function<void ()>::operator()() const /cxx_build/include/c++/v1/__functional/function.h:995:10 (test_bitcoin+0x183a8ef)
     42    [#18](/bitcoin-bitcoin/18/) CScheduler::serviceQueue() /home/admin/actions-runner/_work/_temp/build/src/./scheduler.cpp:60:17 (test_bitcoin+0x183a8ef)
     43    [#19](/bitcoin-bitcoin/19/) ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2::operator()() const /home/admin/actions-runner/_work/_temp/build/src/test/util/./test/util/setup_common.cpp:229:114 (test_bitcoin+0xe5c228) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     44    [#20](/bitcoin-bitcoin/20/) decltype(std::declval<ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2&>()()) std::__1::__invoke[abi:de200100]<ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2&>(ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2&) /cxx_build/include/c++/v1/__type_traits/invoke.h:179:25 (test_bitcoin+0xe5c228)
     45    [#21](/bitcoin-bitcoin/21/) void std::__1::__invoke_void_return_wrapper<void, true>::__call[abi:de200100]<ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2&>(ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2&) /cxx_build/include/c++/v1/__type_traits/invoke.h:251:5 (test_bitcoin+0xe5c228)
     46    [#22](/bitcoin-bitcoin/22/) void std::__1::__invoke_r[abi:de200100]<void, ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2&>(ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2&) /cxx_build/include/c++/v1/__type_traits/invoke.h:273:10 (test_bitcoin+0xe5c228)
     47    [#23](/bitcoin-bitcoin/23/) std::__1::__function::__alloc_func<ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2, std::__1::allocator<ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2>, void ()>::operator()[abi:de200100]() /cxx_build/include/c++/v1/__functional/function.h:167:12 (test_bitcoin+0xe5c228)
     48    [#24](/bitcoin-bitcoin/24/) std::__1::__function::__func<ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2, std::__1::allocator<ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2>, void ()>::operator()() /cxx_build/include/c++/v1/__functional/function.h:319:10 (test_bitcoin+0xe5c228)
     49    [#25](/bitcoin-bitcoin/25/) std::__1::__function::__value_func<void ()>::operator()[abi:de200100]() const /cxx_build/include/c++/v1/__functional/function.h:436:12 (test_bitcoin+0x1916058) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     50    [#26](/bitcoin-bitcoin/26/) std::__1::function<void ()>::operator()() const /cxx_build/include/c++/v1/__functional/function.h:995:10 (test_bitcoin+0x1916058)
     51    [#27](/bitcoin-bitcoin/27/) util::TraceThread(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>) /home/admin/actions-runner/_work/_temp/build/src/util/./util/thread.cpp:21:9 (test_bitcoin+0x1916058)
     52    [#28](/bitcoin-bitcoin/28/) decltype(std::declval<void (*)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>)>()(std::declval<char const*>(), std::declval<ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2>())) std::__1::__invoke[abi:de200100]<void (*)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), char const*, ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2>(void (*&&)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), char const*&&, ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2&&) /cxx_build/include/c++/v1/__type_traits/invoke.h:179:25 (test_bitcoin+0xe5bddb) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     53    [#29](/bitcoin-bitcoin/29/) void std::__1::__thread_execute[abi:de200100]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (*)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), char const*, ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2, 2ul, 3ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (*)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), char const*, ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2>&, std::__1::__tuple_indices<2ul, 3ul>) /cxx_build/include/c++/v1/__thread/thread.h:199:3 (test_bitcoin+0xe5bddb)
     54    [#30](/bitcoin-bitcoin/30/) void* std::__1::__thread_proxy[abi:de200100]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (*)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), char const*, ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2>>(void*) /cxx_build/include/c++/v1/__thread/thread.h:208:3 (test_bitcoin+0xe5bddb)
     55
     56  Location is heap block of size 200 at 0x7234000016c0 allocated by main thread:
     57    [#0](/bitcoin-bitcoin/0/) operator new(unsigned long) <null> (test_bitcoin+0x20a6e6) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     58    [#1](/bitcoin-bitcoin/1/) void* std::__1::__libcpp_operator_new[abi:de200100]<unsigned long>(unsigned long) /cxx_build/include/c++/v1/__new/allocate.h:37:10 (test_bitcoin+0xfe0560) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     59    [#2](/bitcoin-bitcoin/2/) std::__1::__hash_node<std::__1::__hash_value_type<uint256, CBlockIndex>, void*>* std::__1::__libcpp_allocate[abi:de200100]<std::__1::__hash_node<std::__1::__hash_value_type<uint256, CBlockIndex>, void*>>(std::__1::__element_count, unsigned long) /cxx_build/include/c++/v1/__new/allocate.h:64:28 (test_bitcoin+0xfe0560)
     60    [#3](/bitcoin-bitcoin/3/) std::__1::allocator<std::__1::__hash_node<std::__1::__hash_value_type<uint256, CBlockIndex>, void*>>::allocate[abi:de200100](unsigned long) /cxx_build/include/c++/v1/__memory/allocator.h:105:14 (test_bitcoin+0xfe0560)
     61    [#4](/bitcoin-bitcoin/4/) std::__1::allocator_traits<std::__1::allocator<std::__1::__hash_node<std::__1::__hash_value_type<uint256, CBlockIndex>, void*>>>::allocate[abi:de200100](std::__1::allocator<std::__1::__hash_node<std::__1::__hash_value_type<uint256, CBlockIndex>, void*>>&, unsigned long) /cxx_build/include/c++/v1/__memory/allocator_traits.h:270:16 (test_bitcoin+0xfe0560)
     62    [#5](/bitcoin-bitcoin/5/) std::__1::unique_ptr<std::__1::__hash_node<std::__1::__hash_value_type<uint256, CBlockIndex>, void*>, std::__1::__hash_node_destructor<std::__1::allocator<std::__1::__hash_node<std::__1::__hash_value_type<uint256, CBlockIndex>, void*>>>> std::__1::__hash_table<std::__1::__hash_value_type<uint256, CBlockIndex>, std::__1::__unordered_map_hasher<uint256, std::__1::__hash_value_type<uint256, CBlockIndex>, BlockHasher, std::__1::equal_to<uint256>, true>, std::__1::__unordered_map_equal<uint256, std::__1::__hash_value_type<uint256, CBlockIndex>, std::__1::equal_to<uint256>, BlockHasher, true>, std::__1::allocator<std::__1::__hash_value_type<uint256, CBlockIndex>>>::__construct_node_hash<std::__1::piecewise_construct_t const&, std::__1::tuple<uint256&&>, std::__1::tuple<CBlockHeader const&>>(unsigned long, std::__1::piecewise_construct_t const&, std::__1::tuple<uint256&&>&&, std::__1::tuple<CBlockHeader const&>&&) /cxx_build/include/c++/v1/__hash_table:1840:21 (test_bitcoin+0xfe0560)
     63    [#6](/bitcoin-bitcoin/6/) std::__1::pair<std::__1::__hash_iterator<std::__1::__hash_node<std::__1::__hash_value_type<uint256, CBlockIndex>, void*>*>, bool> std::__1::__hash_table<std::__1::__hash_value_type<uint256, CBlockIndex>, std::__1::__unordered_map_hasher<uint256, std::__1::__hash_value_type<uint256, CBlockIndex>, BlockHasher, std::__1::equal_to<uint256>, true>, std::__1::__unordered_map_equal<uint256, std::__1::__hash_value_type<uint256, CBlockIndex>, std::__1::equal_to<uint256>, BlockHasher, true>, std::__1::allocator<std::__1::__hash_value_type<uint256, CBlockIndex>>>::__emplace_unique_key_args<uint256, std::__1::piecewise_construct_t const&, std::__1::tuple<uint256&&>, std::__1::tuple<CBlockHeader const&>>(uint256 const&, std::__1::piecewise_construct_t const&, std::__1::tuple<uint256&&>&&, std::__1::tuple<CBlockHeader const&>&&) /cxx_build/include/c++/v1/__hash_table:1547:25 (test_bitcoin+0xfe0560)
     64    [#7](/bitcoin-bitcoin/7/) std::__1::pair<std::__1::__hash_map_iterator<std::__1::__hash_iterator<std::__1::__hash_node<std::__1::__hash_value_type<uint256, CBlockIndex>, void*>*>>, bool> std::__1::unordered_map<uint256, CBlockIndex, BlockHasher, std::__1::equal_to<uint256>, std::__1::allocator<std::__1::pair<uint256 const, CBlockIndex>>>::try_emplace[abi:de200100]<CBlockHeader const&>(uint256&&, CBlockHeader const&) /cxx_build/include/c++/v1/unordered_map:1288:21 (test_bitcoin+0xfc9637) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     65    [#8](/bitcoin-bitcoin/8/) node::BlockManager::AddToBlockIndex(CBlockHeader const&, CBlockIndex*&) /home/admin/actions-runner/_work/_temp/build/src/./node/blockstorage.cpp:211:41 (test_bitcoin+0xfc9637)
     66    [#9](/bitcoin-bitcoin/9/) Chainstate::LoadGenesisBlock() /home/admin/actions-runner/_work/_temp/build/src/./validation.cpp:5070:42 (test_bitcoin+0x1334783) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     67    [#10](/bitcoin-bitcoin/10/) node::CompleteChainstateInitialization(ChainstateManager&, node::ChainstateLoadOptions const&) /home/admin/actions-runner/_work/_temp/build/src/./node/chainstate.cpp:65:82 (test_bitcoin+0xfe9b61) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     68    [#11](/bitcoin-bitcoin/11/) node::LoadChainstate(ChainstateManager&, kernel::CacheSizes const&, node::ChainstateLoadOptions const&) /home/admin/actions-runner/_work/_temp/build/src/./node/chainstate.cpp:181:38 (test_bitcoin+0xfe9232) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     69    [#12](/bitcoin-bitcoin/12/) ChainTestingSetup::LoadVerifyActivateChainstate() /home/admin/actions-runner/_work/_temp/build/src/test/util/./test/util/setup_common.cpp:308:28 (test_bitcoin+0xe53a70) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     70    [#13](/bitcoin-bitcoin/13/) TestingSetup::TestingSetup(ChainType, TestOpts) /home/admin/actions-runner/_work/_temp/build/src/test/util/./test/util/setup_common.cpp:331:5 (test_bitcoin+0xe540f7) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     71    [#14](/bitcoin-bitcoin/14/) blockmanager_tests::blockmanager_readblock_hash_mismatch::blockmanager_readblock_hash_mismatch() /home/admin/actions-runner/_work/_temp/build/src/test/./test/blockmanager_tests.cpp:140:1 (test_bitcoin+0x43d30a) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     72    [#15](/bitcoin-bitcoin/15/) blockmanager_tests::blockmanager_readblock_hash_mismatch_invoker() /home/admin/actions-runner/_work/_temp/build/src/test/./test/blockmanager_tests.cpp:140:1 (test_bitcoin+0x43d30a)
     73    [#16](/bitcoin-bitcoin/16/) boost::detail::function::void_function_invoker<void (*)(), void>::invoke(boost::detail::function::function_buffer&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:59:11 (test_bitcoin+0x3275cd) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     74    [#17](/bitcoin-bitcoin/17/) boost::function_n<void>::operator()() const /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:789:14 (test_bitcoin+0x2a2618) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     75    [#18](/bitcoin-bitcoin/18/) boost::detail::forward::operator()() /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1416:32 (test_bitcoin+0x2a2618)
     76    [#19](/bitcoin-bitcoin/19/) boost::detail::function::function_obj_invoker<boost::detail::forward, int>::invoke(boost::detail::function::function_buffer&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:79:18 (test_bitcoin+0x2a2618)
     77    [#20](/bitcoin-bitcoin/20/) boost::function_n<int>::operator()() const /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:789:14 (test_bitcoin+0x226873) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     78    [#21](/bitcoin-bitcoin/21/) int boost::detail::do_invoke<boost::shared_ptr<boost::detail::translator_holder_base>, boost::function<int ()>>(boost::shared_ptr<boost::detail::translator_holder_base> const&, boost::function<int ()> const&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:329:30 (test_bitcoin+0x226873)
     79    [#22](/bitcoin-bitcoin/22/) boost::execution_monitor::catch_signals(boost::function<int ()> const&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:931:16 (test_bitcoin+0x226873)
     80    [#23](/bitcoin-bitcoin/23/) boost::execution_monitor::execute(boost::function<int ()> const&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1329:16 (test_bitcoin+0x226bea) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     81    [#24](/bitcoin-bitcoin/24/) boost::execution_monitor::vexecute(boost::function<void ()> const&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1425:5 (test_bitcoin+0x2222cc) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     82    [#25](/bitcoin-bitcoin/25/) boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned long) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/unit_test_monitor.ipp:49:9 (test_bitcoin+0x2222cc)
     83    [#26](/bitcoin-bitcoin/26/) boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:815:44 (test_bitcoin+0x255e8d) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     84    [#27](/bitcoin-bitcoin/27/) boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:784:58 (test_bitcoin+0x2563eb) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     85    [#28](/bitcoin-bitcoin/28/) boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:784:58 (test_bitcoin+0x2563eb) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     86    [#29](/bitcoin-bitcoin/29/) boost::unit_test::framework::run(unsigned long, bool) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:1722:29 (test_bitcoin+0x220dc1) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     87    [#30](/bitcoin-bitcoin/30/) boost::unit_test::unit_test_main(boost::unit_test::test_suite* (*)(int, char**), int, char**) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/unit_test_main.ipp:250:9 (test_bitcoin+0x23bd3c) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     88    [#31](/bitcoin-bitcoin/31/) main /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/unit_test_main.ipp:306:12 (test_bitcoin+0x23c5c3) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     89
     90  Thread T1 'b-scheduler' (tid=11645, running) created by main thread at:
     91    [#0](/bitcoin-bitcoin/0/) pthread_create <null> (test_bitcoin+0x18677e) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     92    [#1](/bitcoin-bitcoin/1/) std::__1::__libcpp_thread_create[abi:de200100](unsigned long*, void* (*)(void*), void*) /cxx_build/include/c++/v1/__thread/support/pthread.h:182:10 (test_bitcoin+0xe52b07) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     93    [#2](/bitcoin-bitcoin/2/) std::__1::thread::thread<void (&)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), char const (&) [10], ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2, 0>(void (&)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), char const (&) [10], ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts)::$_2&&) /cxx_build/include/c++/v1/__thread/thread.h:218:14 (test_bitcoin+0xe52b07)
     94    [#3](/bitcoin-bitcoin/3/) ChainTestingSetup::ChainTestingSetup(ChainType, TestOpts) /home/admin/actions-runner/_work/_temp/build/src/test/util/./test/util/setup_common.cpp:229:46 (test_bitcoin+0xe52b07)
     95    [#4](/bitcoin-bitcoin/4/) TestingSetup::TestingSetup(ChainType, TestOpts) /home/admin/actions-runner/_work/_temp/build/src/test/util/./test/util/setup_common.cpp:323:7 (test_bitcoin+0xe5400f) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     96    [#5](/bitcoin-bitcoin/5/) blockmanager_tests::blockmanager_readblock_hash_mismatch::blockmanager_readblock_hash_mismatch() /home/admin/actions-runner/_work/_temp/build/src/test/./test/blockmanager_tests.cpp:140:1 (test_bitcoin+0x43d30a) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     97    [#6](/bitcoin-bitcoin/6/) blockmanager_tests::blockmanager_readblock_hash_mismatch_invoker() /home/admin/actions-runner/_work/_temp/build/src/test/./test/blockmanager_tests.cpp:140:1 (test_bitcoin+0x43d30a)
     98    [#7](/bitcoin-bitcoin/7/) boost::detail::function::void_function_invoker<void (*)(), void>::invoke(boost::detail::function::function_buffer&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:59:11 (test_bitcoin+0x3275cd) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
     99    [#8](/bitcoin-bitcoin/8/) boost::function_n<void>::operator()() const /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:789:14 (test_bitcoin+0x2a2618) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
    100    [#9](/bitcoin-bitcoin/9/) boost::detail::forward::operator()() /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1416:32 (test_bitcoin+0x2a2618)
    101    [#10](/bitcoin-bitcoin/10/) boost::detail::function::function_obj_invoker<boost::detail::forward, int>::invoke(boost::detail::function::function_buffer&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:79:18 (test_bitcoin+0x2a2618)
    102    [#11](/bitcoin-bitcoin/11/) boost::function_n<int>::operator()() const /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:789:14 (test_bitcoin+0x226873) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
    103    [#12](/bitcoin-bitcoin/12/) int boost::detail::do_invoke<boost::shared_ptr<boost::detail::translator_holder_base>, boost::function<int ()>>(boost::shared_ptr<boost::detail::translator_holder_base> const&, boost::function<int ()> const&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:329:30 (test_bitcoin+0x226873)
    104    [#13](/bitcoin-bitcoin/13/) boost::execution_monitor::catch_signals(boost::function<int ()> const&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:931:16 (test_bitcoin+0x226873)
    105    [#14](/bitcoin-bitcoin/14/) boost::execution_monitor::execute(boost::function<int ()> const&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1329:16 (test_bitcoin+0x226bea) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
    106    [#15](/bitcoin-bitcoin/15/) boost::execution_monitor::vexecute(boost::function<void ()> const&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1425:5 (test_bitcoin+0x2222cc) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
    107    [#16](/bitcoin-bitcoin/16/) boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned long) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/unit_test_monitor.ipp:49:9 (test_bitcoin+0x2222cc)
    108    [#17](/bitcoin-bitcoin/17/) boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:815:44 (test_bitcoin+0x255e8d) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
    109    [#18](/bitcoin-bitcoin/18/) boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:784:58 (test_bitcoin+0x2563eb) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
    110    [#19](/bitcoin-bitcoin/19/) boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:784:58 (test_bitcoin+0x2563eb) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
    111    [#20](/bitcoin-bitcoin/20/) boost::unit_test::framework::run(unsigned long, bool) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:1722:29 (test_bitcoin+0x220dc1) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
    112    [#21](/bitcoin-bitcoin/21/) boost::unit_test::unit_test_main(boost::unit_test::test_suite* (*)(int, char**), int, char**) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/unit_test_main.ipp:250:9 (test_bitcoin+0x23bd3c) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
    113    [#22](/bitcoin-bitcoin/22/) main /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/boost/test/impl/unit_test_main.ipp:306:12 (test_bitcoin+0x23c5c3) (BuildId: 8e805892ada5ea62ad577c3e5653487da52099c4)
    114
    115SUMMARY: ThreadSanitizer: data race /home/admin/actions-runner/_work/_temp/build/src/test/./test/blockmanager_tests.cpp:143:28 in blockmanager_tests::blockmanager_readblock_hash_mismatch::test_method()
    
  91. fanquake commented at 4:06 pm on July 31, 2025: member
  92. willcl-ark force-pushed on Aug 4, 2025
  93. willcl-ark commented at 10:50 am on August 4, 2025: member
    Pushed another run with caches saving to re-test cache hit/restore later.
  94. willcl-ark force-pushed on Aug 4, 2025
  95. m3dwards force-pushed on Aug 5, 2025
  96. willcl-ark force-pushed on Aug 6, 2025
  97. willcl-ark commented at 1:28 pm on August 6, 2025: member

    Force pushed to pick up changes in master (particularly #33002).

    Other notable changes:

    • Added a remote ccache host in 2adbe2caec. Previously we used actions/cache blobs for ccache. This works ~ ok, but Cirrus restores partial matches in reverse-age order (oldest first), which was resulting in bad cache hitrates for ccache when multiple caches existed for a job (say, “old” and “new”). The HTTP ccache backend is used on Cirrus infra and shared between all jobs. This requires using docker -host networking as we are using ccache inside the container. Visible (all misses on first run) here in logs
    • Set TEST_RUNNER_PORT_MIN in d57b291736 to avoid conflicting with the above remote ccache host
    • Reduced ccache hitrate warning level to 75% in 0e768d18bd
  98. willcl-ark force-pushed on Aug 6, 2025
  99. fanquake commented at 3:57 pm on August 6, 2025: member

    TSAN failure, same as the one above (https://github.com/bitcoin/bitcoin/actions/runs/16781531680/job/47521224560?pr=32989#step:8:5924):

    0 2025-08-06T15:41:21.507392Z [test] [net.cpp:3192] [void==================
    1WARNING: ThreadSanitizer: data race (pid=10235)
    2  Write of size 8 at 0x7234000016f0 by main thread:
    3    [#0](/bitcoin-bitcoin/0/) blockmanager_tests::blockmanager_readblock_hash_mismatch::test_method() /home/admin/actions-runner/_work/_temp/build/src/test/./test/blockmanager_tests.cpp:143:28 (test_bitcoin+0x43dce3) (BuildId: 93c9590d6a842e462ee1d55a9ba83c69fb598193)
    4    [#1](/bitcoin-bitcoin/1/) blockmanager_tests::blockmanager_readblock_hash_mismatch_invoker() /home/admin/actions-runner/_work/_temp/build/src/test/./test/blockmanager_tests.cpp:140:1 (test_bitcoin+0x43d6eb) (BuildId: 93c9590d6a842e462ee1d55a9ba83c69fb598193)
    
  100. willcl-ark force-pushed on Aug 7, 2025
  101. in .github/actions/restore-caches/action.yml:18 in 64d38b8e81 outdated
    13+    description: 'A boolean value to indicate an exact match was found for depends built'
    14+runs:
    15+  using: 'composite'
    16+  steps:
    17+    - name: Restore Ccache cache from blob
    18+      if: inputs.use-cirrus != 'true'
    


    maflcko commented at 8:36 am on August 7, 2025:

    64d38b8e81d4fbbfac02f4e3e734d8ecd61dd1dc: i don’t understand this. This reads as if the cache is only restored when not run on cirrus, which seems odd and unwanted?

    edit: Oh, I see it is due to #32989 (comment)


    willcl-ark commented at 2:04 pm on August 7, 2025:

    Correct, I should add a comment perhaps, but choosing only a single cache type (blob storage or http endpoint) was done becuase I think we are just otherwise doubling the space the ccache storage will take per job:

    1 x storage local blob 1x storage http endpoint

    Both will store the same cached compilation, and both will use cache quota.

    As the restore policy of the blob storage is currently backwards (IMO), use only http endpoint for ccache, and let ccache handle LRU evictions itself.


    maflcko commented at 2:45 pm on August 7, 2025:
    how do evictions happen, if export CCACHE_MAXSIZE=${CCACHE_MAXSIZE:-500M} is set to 500M in total for all tasks? It seems they’d happen too much and too early?

    willcl-ark commented at 3:12 pm on August 7, 2025:

    Good question; remote backends are not governed by client-side CCACHE_MAXSIZE settings, see https://ccache.dev/manual/4.11.3.html#_http_storage_backend for a bit more info.

    Basically the onus is on the server to periodically run ccache --trim-dir to keep it under control (we have asked for more details on this process cc @m3dwards )


    maflcko commented at 3:20 pm on August 7, 2025:
    Would it not be easier for Cirrus to just fix the reverse-age order to pick the latest instead? Or somehow work around this by embedding the epoch seconds in the ccache blob name?

    willcl-ark commented at 8:59 pm on August 7, 2025:

    Yes I agree, the blob cache makes the action simpler, and the whole process easier to reason about (and doesnt require a carve out for the fallback if not using cirrus runners). But without it restoring newest first we are going to see worse cache hit-rates than with this approach.

    Perhaps @fkorotkov could clarify here if oldest match is correct, and if there is a way to restore newest first like (as I understand) GHA does natively? My understanding, which may be incorrect, is that epoch seconds would not help (its analagous to monotonically-increasing run_id), and on the wildcard Cirrus will still restore oldest match.


    fkorotkov commented at 9:44 pm on August 7, 2025:

    I actually email @m3dwards couple hours ago that we changed behaviour:

    “restore-keys” now should return the newest entry within the last 1000 cache entries in lexicographical order. There might be an edge case where one can accumulate more than 1000 keys before the eviction happens. We added some metrics to track this case and thought of lazy evicting entries based on some heuristic.

    So it should now work as expected and return the freshest entity among first 1000 in lexicographical order (hence this S3 API quirk).


    willcl-ark commented at 8:27 am on August 8, 2025:
    Great, thanks Fedor!
  102. in .github/actions/restore-caches/action.yml:30 in 64d38b8e81 outdated
    25+          ccache-${{ env.CONTAINER_NAME }}-
    26+
    27+    - name: Configure remote ccache http host
    28+      if: inputs.use-cirrus == 'true'
    29+      shell: bash
    30+      run: echo "CCACHE_REMOTE_STORAGE=http://$CIRRUS_HTTP_CACHE_HOST" >> $GITHUB_ENV
    


    maflcko commented at 8:38 am on August 7, 2025:

    https://github.com/bitcoin/bitcoin/commit/64d38b8e81d4fbbfac02f4e3e734d8ecd61dd1dc: why is this needed? Shouldn’t cirrus take care of this itself in its own action?

    edit: Oh, I see it is due to #32989 (comment)

  103. in ci/test/00_setup_env_mac_native.sh:11 in d8aa07772f outdated
     7@@ -8,6 +8,7 @@ export LC_ALL=C.UTF-8
     8 
     9 # Homebrew's python@3.12 is marked as externally managed (PEP 668).
    10 # Therefore, `--break-system-packages` is needed.
    11+export CONTAINER_NAME="macOS 14 native, arm64, no depends, sqlite only, gui"  # macos does not use a container, but the env var is needed for logging
    


    maflcko commented at 8:59 am on August 7, 2025:

    nit in 993e59ca0c: Would be good to use naming consistent with all other CONTAINER_NAME?

    For ci/test/00_setup_env_mac_native.sh:

    export CONTAINER_NAME=ci_mac_native

    For ci/test/00_setup_env_mac_native_fuzz.sh:

    export CONTAINER_NAME=ci_mac_native_fuzz

  104. maflcko commented at 9:23 am on August 7, 2025: member

    There are still some leftover files, which could be deleted: #32989 (review)

    Also, I am not familiar with the github-cirrus-ccache-remote caching, so I can’t review that one.

    review ACK 36d3042512ed9e638b413866b5a36eec48895bcc 🐬

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: review ACK 36d3042512ed9e638b413866b5a36eec48895bcc 🐬
    34yyXnlBx7f31chJrv9654xosAKktz56Z9xXrCrKYw1aCnbWlBuJSiszVRRFAOduhG1Icnok9Jit9L+RiraxkDg==
    
  105. DrahtBot requested review from fanquake on Aug 7, 2025
  106. in ci/test/02_run_container.sh:89 in 36d3042512 outdated
    84@@ -114,6 +85,17 @@ if [ -z "$DANGER_RUN_CI_ON_HOST" ]; then
    85   # When detecting podman-docker, `--external` should be added.
    86   docker image prune --force --filter "label=$CI_IMAGE_LABEL"
    87 
    88+  # Check if CCACHE_REMOTE_STORAGE is set and configure container networking
    89+  CCACHE_EXTRA_ARGS=""
    


    maflcko commented at 10:15 am on August 7, 2025:
    unused variable?
  107. willcl-ark force-pushed on Aug 8, 2025
  108. willcl-ark force-pushed on Aug 8, 2025
  109. maflcko commented at 3:32 pm on August 8, 2025: member

    not sure why you dropped bbdf75db37? c18b0939da also seems fine to keep, but can be cherry-picked in a follow-up.

    review ACK 739aef1023 🥐

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: review ACK 739aef1023 🥐
    3yatePmaUEq1mRxa3IrcM8HaRLs7UFnMq63a1gs2AjBktsmSyDRQZ7HrlwCV+HQvukziiZ8Ghb7nZxf/8z0YgCw==
    
  110. willcl-ark commented at 4:12 pm on August 8, 2025: member

    review ACK 739aef1 🥐 Show signature

    Thank you so much for the (re)-review on this one @maflcko!

    I was waiting for CI to go green again after a push-with-forced-cache-save and re-push-without before explaining:

    not sure why you dropped bbdf75d?

    This was added specifically to accomodate the http-based ccache endpoint (port 12345), so dropped along with that change. Currently it defaults to 11000 which seems to not conflict with anything currently:

    0❯ rg TEST_RUNNER_PORT_MIN
    1test/functional/test_framework/util.py
    2354:PORT_MIN = int(os.getenv('TEST_RUNNER_PORT_MIN', default=11000))
    

    Is there a reason you’d prefer to have this added back in; just to be more explicit?

    c18b093 also seems fine to keep, but can be cherry-picked in a follow-up.

    Despite this gaining quite some decent-enough runtime performance, I’d prefer to add this in a followup after having gotten TSAN failures twice in this PR. @fanquake also noted that he saw some issues when using (2 * nproc) in local runs, so I think safer to leave for now.

  111. in .github/actions/restore-caches/action.yml:43 in 739aef1023 outdated
    38+    - name: Restore previous releases cache
    39+      id: previous-releases
    40+      uses: cirruslabs/cache/restore@v4
    41+      with:
    42+        path: ${{ env.PREVIOUS_RELEASES_DIR }}
    43+        key: previous-releases-${{ env.CONTAINER_NAME }}-${{ hashFiles('./test/get_previous_releases.py') }}
    


    m3dwards commented at 5:29 pm on August 13, 2025:
    nit: perhaps hash of get_previous_releases.py should be an environment variable set in configure environment just like DEPENDS_HASH is so that it’s not repeated in save/restore cache files.

    willcl-ark commented at 8:58 am on August 14, 2025:
    Added in 5d652ca1ef9 and used in b57bb7c0fbd
  112. in .github/actions/restore-caches/action.yml:52 in 739aef1023 outdated
    47+    - name: export cache hit
    48+      id: export-cache-hit
    49+      shell: bash
    50+      run: |
    51+        echo "depends-sources-cache-hit=${{ steps.depends-sources.outputs.cache-hit }}" >> $GITHUB_OUTPUT
    52+        echo "depends-built-cache-hit=${{ steps.depends-built.outputs.cache-hit }}" >> $GITHUB_OUTPUT
    


    m3dwards commented at 5:38 pm on August 13, 2025:
    Is this missing outputting previous-releases-cache-hit?

    willcl-ark commented at 8:58 am on August 14, 2025:
    Added debug output in b57bb7c0fbd
  113. in .github/actions/restore-caches/action.yml:7 in 739aef1023 outdated
    0@@ -0,0 +1,52 @@
    1+name: 'Restore Caches'
    2+description: 'Restore ccache, depends sources, and built depends caches'
    3+outputs:
    4+  depends-sources-cache-hit:
    5+    description: 'A boolean value to indicate an exact match was found for depends sources'
    6+  depends-built-cache-hit:
    7+    description: 'A boolean value to indicate an exact match was found for depends built'
    


    m3dwards commented at 5:39 pm on August 13, 2025:
    Also missing previous-releases-cache-hit?

    willcl-ark commented at 9:00 am on August 14, 2025:

    Reworked cache inputs/outputs in b57bb7c0fbd to use env vars.

    Inputs and outputs are more explicit but only really needed when passing info between jobs. For us, env vars work fine and let us remove the inputs from the main ci.yml file.

  114. in .github/actions/configure-environment/action.yml:27 in 39e9c27679 outdated
    22+
    23+    - name: Get container name
    24+      shell: bash
    25+      run: |
    26+        source $FILE_ENV
    27+        echo "CONTAINER_NAME=$CONTAINER_NAME" >> "$GITHUB_ENV"
    


    m3dwards commented at 6:15 pm on August 13, 2025:
    potential tiny nit: Are we using this as a replacement for job id / matrix id? And if so is container name the correct name for the environment variable? AFAICT it’s used to split caches per job giving them a unique name. The fact it’s actually a docker container name isn’t relevant where it’s used. So two questions, could this be job.id / matrix.id instead and if not is this the right name?

    willcl-ark commented at 9:03 am on August 14, 2025:

    We don’t have matrix.id in our CI yml currently so to avoid creating another field existing $CONTAINER_NAME has been used.

    matrix.name would not be suitable as the names have spaces in.


    maflcko commented at 6:49 am on August 21, 2025:

    When running locally, the image, the container and the volumes are named after it:

    0ci/test/02_run_container.sh:  docker volume create "${CONTAINER_NAME}_ccache" || true
    1ci/test/02_run_container.sh:  docker volume create "${CONTAINER_NAME}_depends" || true
    2ci/test/02_run_container.sh:  docker volume create "${CONTAINER_NAME}_depends_sources" || true
    3ci/test/02_run_container.sh:  docker volume create "${CONTAINER_NAME}_previous_releases" || true
    

    So I think it is fine to use the same naming scheme for GHA as well.

  115. in .github/workflows/ci.yml:512 in c762d2d5ea outdated
    458@@ -495,6 +459,12 @@ jobs:
    459         with:
    460           use-cirrus: ${{ needs.runners.outputs.use-cirrus-runners }}
    461 
    462+      - name: Enable bpfcc script
    463+        if: ${{ env.CONTAINER_NAME == 'ci_native_asan' }}
    


    m3dwards commented at 6:35 pm on August 13, 2025:
    Why not matrix.id?

    willcl-ark commented at 9:03 am on August 14, 2025:
    Same comment as above regarding matrix.id and matrix.name
  116. willcl-ark force-pushed on Aug 14, 2025
  117. willcl-ark force-pushed on Aug 14, 2025
  118. willcl-ark force-pushed on Aug 14, 2025
  119. DrahtBot added the label Needs rebase on Aug 20, 2025
  120. in .github/actions/configure-environment/action.yml:18 in 5d652ca1ef outdated
    13+
    14+    - name: Set cache paths
    15+      shell: bash
    16+      run: |
    17+        echo "BASE_CACHE=${{ runner.temp }}/depends/built" >> $GITHUB_ENV
    18+        echo "PREVIOUS_RELEASES_DIR=${{ runner.temp }}/previous_releases" >> $GITHUB_ENV
    


    maflcko commented at 7:00 am on August 21, 2025:

    style nit in 5d652ca1ef996cbe7ad9e94a4c6d29b11a7f35e9: Not sure about the distinction, but PREVIOUS_RELEASES_DIR is also a CI dir (config). See the output in https://github.com/bitcoin/bitcoin/actions/runs/16960628990/job/48072349653?pr=32989#step:9:114:

    0+ ./ci/test/02_run_container.py
    1Export only allowed settings:
    2+ bash -c 'grep export ./ci/test/00_setup_env*.sh'
    3+ cat /tmp/env-admin-ci_native_previous_releases
    4CCACHE_TEMPDIR=/tmp/.ccache-temp
    5CCACHE_MAXSIZE=500M
    6 ...
    7PREVIOUS_RELEASES_DIR=/home/admin/actions-runner/_work/_temp/previous_releases
    

    willcl-ark commented at 9:36 am on September 1, 2025:
    Merged these into a single step in fdf64e55324, as there was no clear distinction between the two steps.
  121. maflcko added this to the milestone 30.0 on Aug 21, 2025
  122. maflcko commented at 7:03 am on August 21, 2025: member

    only change is some env var shuffling

    re-ACK c2d18db38de6d5f71d48437422b93b8a962d3935 🛄

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: re-ACK c2d18db38de6d5f71d48437422b93b8a962d3935 🛄
    3bYgBtprVMXfUk5BgAVvp+8lLUjPrKQfBmFfyJQyWnDxsJYbT2dBKu4IbR6dTg7fEycNYRXVDQYayCZO/ZJkeBw==
    
  123. maflcko commented at 7:13 am on August 21, 2025: member

    c18b093 also seems fine to keep, but can be cherry-picked in a follow-up.

    Despite this gaining quite some decent-enough runtime performance, I’d prefer to add this in a followup after having gotten TSAN failures twice in this PR.

    @fanquake also noted that he saw some issues when using (2 * nproc) in local runs, so I think safer to leave for now.

    Ah, I see. I guess “some issues” refers to the symptoms also seen in #31132 (comment) ?

  124. maflcko commented at 7:23 am on August 21, 2025: member
    I’ve added the 30 milestone, because there seems to be conceptual agreement to do this. This is not an end-user feature, so the feature freeze does not apply, but it would be good to get some more review on this. One of the goals in the pull description is so that "anyone" familiar with GitHub-style CI systems can work on it. It would be nice if one or two of the GitHub-style CI persons could review this :sweat_smile:
  125. in .github/workflows/ci.yml:34 in c2d18db38d outdated
    28@@ -28,6 +29,22 @@ defaults:
    29     shell: bash
    30 
    31 jobs:
    32+  runners:
    33+    name: 'determine runners'
    34+    runs-on: ubuntu-24.04
    


    maflcko commented at 7:30 am on August 21, 2025:
    syle nit: Could be ubuntu-latest, to avoid having to touch this again in the future?

    willcl-ark commented at 9:36 am on September 1, 2025:
    amended in cc1735d7771
  126. in .github/workflows/ci.yml:538 in c2d18db38d outdated
    552+    timeout-minutes: 20
    553+    env:
    554+      CONTAINER_NAME: "bitcoin-linter"
    555+    steps:
    556+      - name: Checkout
    557+        uses: actions/checkout@v4
    


    maflcko commented at 7:36 am on August 21, 2025:
    nit: IIUC the checkout merge_base issue mentioned in #32989 (comment) still remains and would be nice to fix. But this can be done in a follow-up.

    0xB10C commented at 6:49 am on August 22, 2025:
    0        uses: actions/checkout@v5
    

    willcl-ark commented at 9:36 am on September 1, 2025:
    Yes I’d like to keep this for a followup for now.
  127. maflcko commented at 7:38 am on August 21, 2025: member
    (just nits)
  128. in .github/workflows/ci.yml:314 in c2d18db38d outdated
    312@@ -295,36 +313,23 @@ jobs:
    313       - name: Checkout
    314         uses: actions/checkout@v4
    


    0xB10C commented at 6:48 am on August 22, 2025:

    nit:

    0        uses: actions/checkout@v5
    

    See https://github.com/bitcoin/bitcoin/pull/33171


    willcl-ark commented at 9:36 am on September 1, 2025:
    Done, here and elsewhere
  129. DrahtBot requested review from 0xB10C on Aug 22, 2025
  130. in .github/workflows/ci.yml:497 in c2d18db38d outdated
    497+            timeout-minutes: 120
    498+            file-env: './ci/test/00_setup_env_native_msan.sh'
    499+
    500     steps:
    501       - name: Checkout
    502         uses: actions/checkout@v4
    


    0xB10C commented at 6:49 am on August 22, 2025:
    0        uses: actions/checkout@v5
    
  131. DrahtBot requested review from 0xB10C on Aug 22, 2025
  132. DrahtBot requested review from 0xB10C on Aug 22, 2025
  133. maflcko commented at 6:19 am on August 28, 2025: member
    Are you still working on this?
  134. willcl-ark commented at 6:52 am on August 28, 2025: member
    Yep. But been on holiday this week and last.
  135. stickies-v commented at 4:42 pm on August 28, 2025: contributor
    Concept ACK. Price seems reasonable for the benefits, and we we’re not locking ourselves in too much.
  136. maflcko commented at 5:31 pm on August 28, 2025: member
    Also, would be nice to have a pull request for https://github.com/bitcoin-core/qa-assets prepared. Possibly without caching is fine, if the msan-fuzz task fits in the GHA timeout.
  137. ci: add configure environment action b8fcc9fcbc
  138. ci: add caching actions
    Add "Restore" and "Save" caching actions.
    
    These actions reduce boilerplate in the main ci.yml configuration file.
    
    These actions are implemented so that caches will be saved on `push`
    only.
    
    When a pull request is opened it will cache hit on the caches from the
    lastest push, or in the case of depends will hit on any matching depends
    hash, falling back to partial matches.
    
    Depends caches are hashed using
    `$(git ls-tree HEAD depends "ci/test/$FILE_ENV" | sha256sum | cut -d' ' -f1)`
    and this hash is passed in as an input to the actions. This means we
    direct cache hit in cases where depends would not be re-built, otherwise
    falling back to a partial match.
    
    Previous releases cache is hashed similarly to depends, but using the
    test/get_previous_releases.py file.
    
    The cirruslabs cache action will fallback transparently to GitHub's
    cache in the case that the job is not being run on a Cirrus Runner,
    making these compatible with running on forks (on free GH hardware).
    b232b0fa5e
  139. ci: add REPO_USE_CIRRUS_RUNNERS
    If set, Cirrus runners will be used on pushes to, and pull requests
    against, this repository.
    
    Forks can set this if they have their own cirrus runners.
    33ba073df7
  140. ci: add configure-docker action
    Another action to reduce boilerplate in the main ci.yml file.
    
    This action will set up a docker builder compatible with caching build
    layers to a container registry using the `gha` build driver.
    
    It will then configure the docker build cache args.
    fdf64e5532
  141. ci: use buildx in ci
    Using buildx is required to properly load the correct driver, for use
    with registry caching. Neither build, nor BUILDKIT=1 currently do this
    properly.
    
    Use of `docker buildx build` is compatible with podman.
    94a0932547
  142. ci: use docker build cache arg directly
    Reverts: e87429a2d0f23eb59526d335844fa5ff5b50b21f
    
    This was added in PR #31545 with the intention that self-hosted runners
    might use it to save build cache.
    
    As we are not using hosted runners with a registry build cache, the bulk
    of this commit can be reverted, simply using the value of
    $DOCKER_BUILD_CACHE_ARG in the script.
    
    link: https://github.com/bitcoin/bitcoin/pull/31545
    18f6be09d0
  143. ci: have base install run in right dir
    This sets the build dir at build time so that Apple SDK gets installed
    in the correct/expected location for the runtime to find it.
    
    Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>
    9c2b96e0d0
  144. ci: add Cirrus cache host
    Whilst the action cirruslabs/actions/cache will automatically set this
    host, the docker `gha` build cache backend will not be aware of it.
    
    Set the value here, which will later be used in the docker build args to
    enable docker build cache on the cirrus cache.
    020069e6b7
  145. ci: add job to determine runner type
    To remove multiple occurances of the respository name, against which we
    compare `${{ github.repository }}` to check if we should use Cirrus
    Runners, introduce a helper job which can check a single environment
    variable and output this as an input to subsequent jobs.
    
    Forks can maintain a trivial patch of their repo name against the
    `REPO_USE_CIRRUS_RUNNERS` variable in ci.yml if they have Cirrus Runners
    of their own, which will then enable cache actions and docker build
    cache to use Cirrus Cache.
    
    It's not possible to use `${{ env.USE_CIRRUS_RUNNERS }}` in the
    `runs-on:` directive as the context is not supported by GitHub.
    
    If it was, this job would no longer be necessary.
    cc1735d777
  146. ci: update windows-cross job
    Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>
    04e7bfbceb
  147. willcl-ark force-pushed on Sep 1, 2025
  148. willcl-ark commented at 9:38 am on September 1, 2025: member
    Rebased on master to catch upstream changes, manually added changes to align with #31802 and #33099 which were merged while I was away. Hopefully I didn’t miss any others.
  149. maflcko commented at 9:54 am on September 1, 2025: member

    Just some style fixups, but it looks like out-of-storage errors are happening:

    0/home/admin/actions-runner/_work/_temp/depends/work/build/arm-linux-gnueabihf/qt/6.7.3-634ec665c7f/qtbase/src/gui/util/qvalidator.h:163:2: fatal error: cannot write PCH file: No space left on device
    

    re-ACK 8a7eb6ef2c226459739bdfe050b735a5a0d0b771 🕧

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: re-ACK 8a7eb6ef2c226459739bdfe050b735a5a0d0b771 🕧
    3UW0pp2ycXnHD1XOEPIkrXvZU9Yk4ma/cp6mh+VKLxb/jU9Ryw9kg5dli95zhThJX+jxKqUUr07tQPNLMxv1eDQ==
    
  150. DrahtBot requested review from stickies-v on Sep 1, 2025
  151. DrahtBot removed the label Needs rebase on Sep 1, 2025
  152. in .github/workflows/ci.yml:416 in 8a7eb6ef2c outdated
    410@@ -406,44 +411,145 @@ jobs:
    411           TEST_RUNNER_EXTRA: ${{ github.event_name != 'pull_request' && '--extended' || '' }}
    412         run: py -3 test/functional/test_runner.py --jobs $NUMBER_OF_PROCESSORS --ci --quiet --tmpdirprefix="$RUNNER_TEMP" --combinedlogslen=99999999 --timeout-factor=$TEST_RUNNER_TIMEOUT_FACTOR $EXCLUDE $TEST_RUNNER_EXTRA
    413 
    414-  asan-lsan-ubsan-integer-no-depends-usdt:
    415-    name: 'ASan + LSan + UBSan + integer, no depends, USDT'
    416-    runs-on: ubuntu-24.04 # has to match container in ci/test/00_setup_env_native_asan.sh for tracing tools
    417+  ci-matrix:
    418+    name: ${{ matrix.name }}
    419+    needs: runners
    


    janb84 commented at 12:23 pm on September 1, 2025:
    0    needs: [runners, lint]
    

    Nit / Question: Why not wait on successful run of linter before running more heavier / costlier runs ?

  153. janb84 commented at 1:05 pm on September 1, 2025: contributor

    ACK 8a7eb6ef2c226459739bdfe050b735a5a0d0b771

    PR changes CI from self hosted instances to hosted Cirrus Runners. In addition to migrating to a more-commonly-used GitHub workflow syntax.

    The migration to the more-commonly-used GitHub workflow syntax makes the whole CI proces easier more structured, easier to understand, this will benefit future maintenance. In addition to be less reliant on (centralised) self-hosting with the ability to easily switch-out Cirrus in the future, will be a benefit (in the long term).

    Small non-blocking NIT / question added.

    • extensive code review ✅
    • research into Cirrus runners ✅
    • coming form Azure Pipelines did some read up on Github action best practices. ✅
  154. maflcko commented at 1:11 pm on September 1, 2025: member

    Just some style fixups, but it looks like out-of-storage errors are happening:

    0/home/admin/actions-runner/_work/_temp/depends/work/build/arm-linux-gnueabihf/qt/6.7.3-634ec665c7f/qtbase/src/gui/util/qvalidator.h:163:2: fatal error: cannot write PCH file: No space left on device
    

    For reference, https://github.com/bitcoin/bitcoin/actions/runs/17373684837/job/49315124575?pr=32989#step:9:1483:

     0+ echo 'Free disk space:'
     1+ df -h
     2Free disk space:
     3+ [[ ci_arm_linux == \c\i\_\n\a\t\i\v\e\_\a\s\a\n ]]
     4Filesystem      Size  Used Avail Use% Mounted on
     5overlay          17G   15G  3.0G  83% /
     6tmpfs            64M     0   64M   0% /dev
     7shm              64M     0   64M   0% /dev/shm
     8/dev/vda1        17G   15G  3.0G  83% /etc/hosts
     9tmpfs           5.9G     0  5.9G   0% /proc/acpi
    10tmpfs           5.9G     0  5.9G   0% /proc/scsi
    11tmpfs           5.9G     0  5.9G   0% /sys/firmware
    

    But https://cirrus-runners.app/setup/#__tabbed_3_3 claims at least 50 GB?

  155. fkorotkov commented at 1:54 pm on September 1, 2025: none
    @maflcko apologies for the disk size on Arm runners issue. It should be fixed. Please try to re-run the job.
  156. fanquake commented at 1:56 pm on September 1, 2025: member

    @maflcko It should be fixed. Please try to re-run the job.

    Thanks. Kicked the job.

  157. maflcko commented at 3:04 pm on September 1, 2025: member

    Looks like Cirrus switched from Neoverse N1 to Apple (with virtual Linux aarch64), which doesn’t support 32-bit mode, so the CI here fails expectedly.

    Maybe you can use GHA ubuntu-24.04-arm for now, which has:

     0VM Image
     1  - OS: Linux (arm64)
     2  - Source: Partner
     3  - Name: Ubuntu 24.04 by Arm Limited
     4  - Version: 20250728.24.1
     5  - Included Software: https://github.com/actions/partner-runner-images/blob/main/images/arm-ubuntu-24-image.md
     6
     7
     8Run lscpu
     9Architecture:                         aarch64
    10CPU op-mode(s):                       32-bit, 64-bit
    11Byte Order:                           Little Endian
    12CPU(s):                               4
    13On-line CPU(s) list:                  0-3
    14Vendor ID:                            ARM
    15Model name:                           Neoverse-N2
    16....
    
  158. ci: port arm 32-bit job
    Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>
    f253031cb8
  159. ci: update asan-lsan-ubsan
    Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>
    884251441b
  160. ci: force reinstall of kernel headers in asan
    When using hosted runners in combination with cached docker images,
    there is the possibility that the host runner image is updated,
    rendering the linux-headers package (stored in the cached docker image)
    incompatible.
    
    Fix this by doing a re-install of the headers package in
    03_test_script.sh.
    
    If the underlying runner kernel has not changed thie has no effect, but
    prevents the job from failing if it has.
    2c990d84a3
  161. ci: port mac-cross-gui-notests
    Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>
    9c2514de53
  162. ci: port nowallet-libbitcoinkernel
    Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>
    2a00b12d73
  163. ci: port no-IPC-i686-DEBUG
    Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>
    f2068f26c1
  164. ci: port fuzzer-address-undefined-integer-nodepends
    Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>
    341196d75c
  165. ci: port previous-releases-depends-debug
    Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>
    58e38c3a04
  166. ci: port centos-depends-gui
    Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>
    549074bc64
  167. ci: port tidy
    Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>
    bf7d536452
  168. ci: port tsan-depends
    Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>
    9bbae61e3b
  169. ci: port msan-depends
    Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>
    d290a8e6ea
  170. ci: port lint
    Co-authored-by: Max Edwards <youwontforgetthis@gmail.com>
    bc41848d00
  171. ci: remove .cirrus.yml
    Removed as unused.
    4393ffdd83
  172. ci: dynamically match makejobs with cores
    Previously jobs were running on a large multi-core server where 10 jobs
    as default made sense (or may even have been on the low side).
    
    Using hosted runners with fixed (and lower) numbers of vCPUs we should
    adapt compilation to match the number of cpus we have dynamically.
    
    This is cross-platform compatible with macos and linux only.
    3f339e99e0
  173. doc: Detail configuration of hosted CI runners f427284483
  174. ci: add ccache hit-rate warning when < 75%
    Print the ccache hit-rate for the job using a GitHub annotation if it
    was below 75%.
    dd1c5903e8
  175. ci: fix annoying docker warning
    Docker currently warns that we are missing a default value.
    
    Set this to scratch which will error if an appropriate image tag is not
    passed in to silence the warning.
    2aa288efdd
  176. ci: remove un-needed lint_run*.sh files
    ci/lint_run.sh: Only used in .cirrus.yml. Refer to test/lint/README.md on how to run locally.
    ci/lint_run_all.sh: Only used in .cirrus.yml for stale re-runs of old pull request tasks.
    3c5da69a23
  177. willcl-ark force-pushed on Sep 1, 2025
  178. janb84 commented at 4:37 pm on September 1, 2025: contributor

    re ACK 3c5da69a232ba1cfb935012aa53e57002efe0d77

    changes since last ACK:

    • changed cirrus-runner for 32 bit ARM
  179. DrahtBot requested review from maflcko on Sep 1, 2025

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-09-02 06:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me