ci: add Guix builds to CI #36082

Currently there is limited continuous testing of our exact release binaries. This is acceptable today as we use many similar toolchain configurations in CI which has most/everything we use in Guix pretty much covered. Many PRs also see Guix builds, although few of those probably run tests against the Guix-built binaries.

This PR would continuously exercise the Guix building process itself, but also, in the event that we start shipping some fully-static Linux builds (x64 and aarch64) as per current revision of #25573, our entire custom-built static glibc and compiler toolchain would be relatively untested by current CI jobs, and that would also be tested here.

This adds Guix builds for the following platforms:

• arm64-apple-darwin
• x86_64-linux-gnu
• x86_64-w64-mingw32

The resulting binaries are then tested in a subsequent job.

<!--e57a25ab6845829454e8d69fc972939a-->

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

<!--006a51241073e994b41acfe9ec718e94-->

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/36082.

<!--021abf342d371248e50ceaed478a90ca-->

Reviews

See the guideline and AI policy for information on the review process.

If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

<!--174a7506f384e20aa4161008e828411d-->

Conflicts

Reviewers, this pull request conflicts with the following ones:

• #35855 _{<img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/35855.svg">} (guix: update time-machine by fanquake)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

<!--5faf32d7da4f0f540f40219e4f7537a3-->

#36006 got auto-closed by a linked PR.

More info on the infra side of things:

I have configured two physically separate bitcoin core substitute servers for better availability (using this module. One server evaluates the guix closure and builds it. Then uses guix copy to copy store items to the second server via ssh.

We could include the default substitute servers (https://bordeaux.guix.gnu.org https://ci.guix.gnu.org), but this makes the CI runs noticeably slower as these (slow) servers are queried even when not used/selected. These servers obviously will never include our custom compilers, which are the slowest thing to build, too.

I tested a run with a single substitute server, shutting the server off part way through, and Guix didn't like that it thought a substitute should be available but then wasn't and so failed. I think our use of --fallback protects us in the case that no substitute servers are available before the run begins; everything is just rebuilt, but have not tested this yet.

<!--85328a0da195eb286784d51f73fa0af9-->

🚧 At least one of the CI tasks failed. _{Task macOS native, fuzz: https://github.com/bitcoin/bitcoin/actions/runs/32941900565/job/98094394757} _{LLM reason (✨ experimental): CI failed because the rpc fuzz target returned exit code 1 due to an “Error processing input” during fuzzing.}

<details><summary>Hints</summary>

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

• Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

• A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

• An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

</details>

@hebasto I think that efbb91102107dd9e758c46b9cd51d7a1de28c38a (the first commit here) and #35929 are duplicating solutions in different ways.

I believe that #35929 is the neater solution, and still covers the multi-toolchain (dual Guix build) scenario via embedding the toolchain info in the build ID as part of:

BASE_CACHE/<host>/<package>/<package>-<version>-<build_id>.tar.gz

Do you agree? If so I think I'll drop efbb91102107dd9e758c46b9cd51d7a1de28c38a from here, and rebase on/review #35929.

@hebasto I think that efbb911 (the first commit here) and #35929 are duplicating solutions in different ways.

I've reviewed and tested efbb91102107dd9e758c46b9cd51d7a1de28c38a. Compared to #35929, it manages to fix the first subissue: "1. All depends are rebuilt when switching to another branch with identical Guix scripts and the depends subdirectory". Unfortunately, it fails to resolve the second one: "2. All non-GUI packages are built twice per host (mostly relevant for non-Linux hosts)".

Another point worth mentioning: for your commit to work as expected, the BASE_CACHE environment variable must be set explicitly.

I believe that #35929 is the neater solution, and still covers the multi-toolchain (dual Guix build) scenario via embedding the toolchain info in the build ID as part of:

BASE_CACHE/<host>/<package>/<package>-<version>-<build_id>.tar.gz

Do you agree? If so I think I'll drop efbb911 from here, and rebase on/review #35929.

That sounds good to me.

@willcl-ark

Please also consider #35929 (comment).

Thanks hebasto, that sounds reasonable to me

Concept ACK

Concept ACK

Added a commit to run true inside the guix-codesign container to try and catch things like #36256 (review)

I don't think there's any more we can do when we don't have signatures.

Marking this as ready for review now as I'd be interested in comments on its current state.

For me the main open question to be answered is:

• Are we happy to rely on the two (physically distinct) Guix substitute servers I'm operating as the only enabled ones in these jobs?

If we add in other upstream servers (the defaults), the job runtime increases a fair amount (about 5 minutes longer to "check substitutes" in all jobs, not to mention that they'll be missing our compiler toolchains as subs so we need to rebuild those anyway which takes ages) but the jobs would become slightly more robust against my two servers being down, or in the case I disappear or whatever.

Another alternative is to have someone else operate another server. This would be my personal preference, if it were possible.

The motivation/goal isn't listed, but right now it seems to catch build errors such as #36006 (comment). Though, from the perspective of end-users a build error is harmless (there are no bugs if there is no binary :sweat_smile:), so it could make more sense to actually run the tests. See also #36271#pullrequestreview-5283937055. However, I am not sure what the best way to implement this with GHA is (see related #35774 (review)).

Though, from the perspective of end-users a build error is harmless (there are no bugs if there is no binary 😅)

We might have to agree to disagree on this point 😋

However, I am not sure what the best way to implement this with GHA is (see related #35774 (review))

Would it be ok to just run x64 tests? I think that case would be a lot easier... in 35744 we need to move binaries between machines.

Right, this is a bit subjective. I am not sure how much we want to squeeze here. This is open for a month and there hasn't been any code/approach review yet.

The guix build issues happen maybe once in a year. Personally I think it is fine to revert a pull request temporarily once in a year, or apply a workaround. Recall that it will never be possible to catch all build/platform issues with CI. E.g. #36281 also required a revert/workaround recently, but I don't think we are going to add a macOS+nix task to the matrix, especially considering that the build issue is intermittent.

Would it be ok to just run x64 tests?

Ideally all the release binaries are tested.I think we can cover all the Linux ones with qemu and the Windows/macOS ones as well (possibly with Rosetta 2, until that is removed as well).

In the past Canonical was running the release unit tests (via Launchpad while building the snap), but it didn't run the functional tests and I think they dropped support for ppc64el...

But then I don't know how much of this should be here in the "hot" matrix, or if some of it should be in a nightly matrix in another repo.

Just me wondering, feel free to ignore.

OK most of that seems reasonable.

What I will do then, is push here, to run my updated branch on our Warp CI runners, just to see if the "test the build binaries" commit I've added works. But then I will close the PR, until (or if) interest picks up.

FWIW I do have a continuous guix build running https://my.cdash.org/index.php?project=bitcoin-core#!#Continuous which doesn't hit every merge, or test any built binaries, but is "doing builds".

I think the chances of tests passing our CI but failing on a guix build are low, and now really sure what more that adds, but perhaps we can see. My intention here was to test the building process itself, more regularly.

Though, from the perspective of end-users a build error is harmless

I think issues like #36006 (comment) are exactly the kind of thing the CI should be catching, so that build/release time issues don't appear randomly, after non-build related changes are made (#32750 in this case).

Personally I think it is fine to revert a pull request temporarily once in a year, or apply a workaround. Recall that it will never be possible to catch all build/platform issues with CI.

These reversions aren't necessarily trivial, because they can also require CI/src/other changes outside of Guix, i.e if rolling back a compiler update, or dependency change.

I think "Does the code compile in a release context" is decent requirement (for merging). A number of CI jobs are setup to emulate building the code in a release context (i.e matching compiler versions), so using the actual release build seems like an improvement, which catches the issues the CI currently misses?

Sure, I am not objected to this pull. However, this is just adding a smaller subset (x86_64-linux-gnu, ...), so any build failure in the armhf release bin (etc) won't be caught. Also, stuff at runtime isn't caught (like bugs in the libstc++ std::fs impl, or so).

Thinking about end-users they'd probably benefit most from a full build+test pipeline for all release bins.

These reversions aren't necessarily trivial

The fix in #33178 looks small and on the easier end. Are there any others that were more involved?

Again, no objection to this pull, but I think it would be more useful to have "full" build+test coverage for all guix release bins (possibly in a nightly repo or in bitcoin-core/packaging), than just a select subset for guix build-only in CI here.

<!--85328a0da195eb286784d51f73fa0af9-->

🚧 At least one of the CI tasks failed. _{Task lint: https://github.com/bitcoin/bitcoin/actions/runs/36594516478/job/109495857711} _{LLM reason (✨ experimental): CI failed because the lint-files check reported an executable-permission issue: .github/ci-guix-test.py has mode 644 despite a shebang, instead of 755.}

<details><summary>Hints</summary>

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

• Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

• A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

• An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

</details>

If we start shipping fully static (glibc-based) linux binaries it would seem prudent to have a CI job testing them.

This PR would cover x64 linux in that regard. We could probably add aarch64 too, or even start with just those two jobs.

An alternative would be to add a non-Guix job which built the static toolchain(s) and binary (re-inventing the Guix build in the process though).

If we start shipping fully static (glibc-based) linux binaries it would seem prudent to have a CI job testing them.

I think that is a good reason. Mind including the goal/motivation in the pull description, so that it is easier to follow? Currently it just repeats what the code changes are, but this is visible from looking at the commits already.

Thanks I've updated the description as you suggest.

I'd gone for linux + windows + mac here to cover as many bases as possible in the fewest number of jobs, but perhaps aarch64-linux is worth covering too post-#25573, or could replace the windows or macos job?

I'll rebase and push a few tidy-ups to the top two commits which were LLM-written as a PoC/test.