RFC: Rethinking depends native package build strategy #35048

When building release binaries in the Guix environment, the depends native packages, namely native_capnp, native_libmultiprocess, and native_qt, are currently built for every target host. This results in a needless expenditure of Guix builders' compute resources.

One possible improvement is to build these native packages as part of a Guix profile instead.

Pros:

1. Speeds up the depends build process across all hosts by roughly 20%, significantly reducing resource consumption for builders.

2. Removes the need to expose the native build toolchain to the cross-compilation build environment, which improves security and build isolation.

Cons:

1. The Guix manifest file will need to be updated to maintain three new package definitions.

Alternative Approach:

As an alternative, the depends build subsystem could be restructured to share and reuse native packages across hosts where possible. This would eliminate the need to modify the Guix manifest and might provide some benefit to developers building depends locally. However, local builders already benefit from an effective built-package cache.

Additionally, the effectiveness of this alternative could be undermined when using LLVM-only build profiles for macOS targets.

Conclusion:

There is clear room to optimize the release build process, which relies heavily on volunteer computing resources. I believe the performance gains and isolation improvements are well worth the extra complexity and maintenance burden introduced to the Guix manifest.

I am posting this RFC to gather conceptual feedback and opinions on this approach.

native_capnp, native_libmultiprocess, and native_qt, are currently built for every target host

I was confused by this. May help to clarify the problem IIUC is not that these packages are being built for every host but that they are being rebuilt for every host and more ideally would be cached?

native_capnp, native_libmultiprocess, and native_qt, are currently built for every target host

I was confused by this. May help to clarify the problem IIUC is not that these packages are being built for every host but that they are being rebuilt for every host and more ideally would be cached?

That's correct. Currently, built native packages are cached within host-specific subdirectories, meaning they aren't shared or reused across different hosts.

If we decide this caching behavior is the preferred direction for an improvement, I've mentioned it as an alternative approach in the OP.

Concept ACK. I think I slightly prefer the guix package approach.

I don't want to derail this RFC, but on a related note I had also toyed with enabling ccache inside of guix: https://github.com/bitcoin/bitcoin/compare/master...willcl-ark:bitcoin:guix-ccache

I wanted to do this as I am doing Continuous Guix building on various platforms over on this cdash site, and ccache would speed up the rebuilds here a lot!

The branch uses ccache for both depends and main build.

I did do some testing on whether use of ccache (or not) would affect final hashes, but don't remember the outcome :( In any case, I wasn't going to recommend this for release building.

Additionally, the effectiveness of this alternative could be undermined when using #30206 build profiles for macOS targets.

I don't understand this concern, can you ellaborate on this a bit? Are you proposing to remove the native packages entirely? If not, would we instruct the guix depends build step to skip the native packages? I'm guessing most of the speedup comes from the native qt build? Might there be some more stuff we can skip on it?

Additionally, the effectiveness of this alternative could be undermined when using #30206 build profiles for macOS targets.

I don't understand this concern, can you ellaborate on this a bit?

Let's assume that we have caching for native packages that works across different hosts (the alternative approach). Their build IDs will commit to the native toolchain used to build them. When using an LLVM-only / sans-GCC environment, as in #32764, which implements #30206, those IDs will differ, effectively invalidating the cache.

Are you proposing to remove the native packages entirely?

No, I'm not. The depends build subsystem should be able to cross-compile.

If not, would we instruct the guix depends build step to skip the native packages?

Yes.

I'm guessing most of the speedup comes from the native qt build? Might there be some more stuff we can skip on it?

The current native_qt package has already been stripped down. When migrating to Qt 6, I specifically aimed to build only necessary stuff.

I believe the performance gains and isolation improvements are well worth the extra complexity and maintenance burden introduced to the Guix manifest.

Not sure I agree with the tradeoff here; introducing more complexity / maintenance effort to save a few (generally already cached) CPU cycles.

I'm guessing most of the speedup comes from the native qt build?

I would think so too. @hebasto what percentage of the total native package build time is currently Qt?

Removes the need to expose the native build toolchain to the cross-compilation build environment, which improves security and build isolation.

Can you explain this further? Hasn't the native toolchain been used to build the environment, (or parts of it), so removing the compiler/toolchain binaries from the environment doesn't seem like much of a security improvement (if the assumption is the binary is malicious)?

I believe the performance gains and isolation improvements are well worth the extra complexity and maintenance burden introduced to the Guix manifest.

Not sure I agree with the tradeoff here; introducing more complexity / maintenance effort to save a few (generally already cached) CPU cycles.

While the native packages are cached, they are cached per host across the current 8 hosts. This means that every time a cache is invalidated or needs an initial population, either of the suggested approaches will save significantly more than just a few CPU cycles.

As for complexity, we actually use three native Qt packages, not just one: qtbase, qttools and qttranslations.

I'm guessing most of the speedup comes from the native qt build?

I would think so too. @hebasto what percentage of the total native package build time is currently Qt?

Roughly 90%.

Removes the need to expose the native build toolchain to the cross-compilation build environment, which improves security and build isolation.

Can you explain this further? Hasn't the native toolchain been used to build the environment, (or parts of it), so removing the compiler/toolchain binaries from the environment doesn't seem like much of a security improvement (if the assumption is the binary is malicious)?

You're right. From a strict security standpoint against a compromised native toolchain, the improvement is marginal.

Hey there, is there a reason this issue is still opened?. Guess the discussion should have closed it, no? @hebasto

Hey there, is there a reason this issue is still opened?.

The reason remains the same as stated in the OP. It's a "needless expenditure of Guix builders' compute resources".

Guess the discussion should have closed it, no?

What conclusion was reached that would warrant closing it?

Hey there, is there a reason this issue is still opened?.

The reason remains the same as stated in the OP. It's a "needless expenditure of Guix builders' compute resources".

Guess the discussion should have closed it, no?

What conclusion was reached that would warrant closing it?

Yeah, you're right

either of the suggested approaches will save significantly more than just a few CPU cycles.

It is some amount, but not a lot in the context of a full guix build, and less so in the case of --no-substitutes; if the goal is to try and save volunteer compute, is this the lowest hanging/highest impact fruit?

I would think so too. @hebasto what percentage of the total native package build time is currently Qt?

Roughly 90%.

This discussion could almost be reframed from "Rethink native depends packages" to "How can we build Qt less times", given we currently perform 16 compiles of Qts Core library, per Guix build, which I would assume is > 60% of total compilation in depends, maybe more?

Something not covered yet is how we'd be testing things if we made this split; separating our release build from depends, means we'd no longer be quasi testing release builds (by using depends) in CI. Would we add Guix builds to CI, to cover that?

Might also want to update the op in regards to security/isolation benefits from doing this, as it's not clear there are any, given the compiler being removed from the environment, was used to build the environment?

either of the suggested approaches will save significantly more than just a few CPU cycles.

It is some amount, but not a lot in the context of a full guix build, and less so in the case of --no-substitutes;

While considering the wider context is interesting, it doesn't change the absolute number of wasted CPU cycles.

if the goal is to try and save volunteer compute, is this the lowest hanging/highest impact fruit?

As a Guix builder using RISC-V hardware, which isn't known for its high performance right now, I'd like to increase the overall efficiency of a full Guix build. I've analyzed only the depends-related part of the process and noticed two actionable issues: this one and #34228.

I would think so too. @hebasto what percentage of the total native package build time is currently Qt?

Roughly 90%.

This discussion could almost be reframed from "Rethink native depends packages" to "How can we build Qt less times", given we currently perform 16 compiles of Qts Core library, per Guix build, which I would assume is > 60% of total compilation in depends, maybe more?

We could certainly focus on Qt first. However, if we choose the alternative approach, it should work for all native packages by default.

Overall, I estimate a 20% savings in compile time in depends.

Something not covered yet is how we'd be testing things if we made this split; separating our release build from depends, means we'd no longer be quasi testing release builds (by using depends) in CI. Would we add Guix builds to CI, to cover that?

I've been thinking about this as well. Choosing the same versions of binary packages in a Guix profile as we use in depends should address this concern, no?