libmultiprocess cross-compiling to Linux hosts
#25046
To successfully call the capnp_generate_cpp() function, the libmultiprocess build system must be provided with paths to the native capnp and capnpc-c++ tools.
This comment points the same:
I think
packages/libmultiprocess.mkprobably needs to be passing a-DCAPNP_EXECUTABLE=.../depends/arm-linux-gnueabihf/native/bin/capnpargument to cmake. Also the package should have dependencies on bothcapnpandnative_capnp.
Fixes bitcoin/bitcoin#24387.
To successfully call the `capnp_generate_cpp()` function, the
`libmultiprocess` build system must be provided with paths to the native
`capnp` and `capnpc-c++` tools.
https://github.com/bitcoin/bitcoin/pull/25046/checks?check_run_id=6246362180:
0make[2]: Entering directory '/tmp/cirrus-ci-build/ci/scratch/build/src'
1Makefile:20771: /tmp/cirrus-ci-build/depends/i686-pc-linux-gnu/native/include/mpgen.mk: No such file or directory
2make[2]: *** No rule to make target '/tmp/cirrus-ci-build/depends/i686-pc-linux-gnu/native/include/mpgen.mk'. Stop.
3make[2]: Leaving directory '/tmp/cirrus-ci-build/ci/scratch/build/src'
4make[1]: *** [Makefile:922: distdir-am] Error 1
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
Reviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.
Thanks for picking this up @hebasto. I spend a day looking into this issue #24387 (comment) but got too confused by what I was seeing.
Taking a quick look at this, first commit passing executable paths to libmultiprocess makes sense. Second commit dropping native_libmultiprocess at first thought does not seem to make sense. Third commit optimizing native builds seems ok, but perhaps not worth added complexity.
For the second commit, I wouldn’t think you could drop the native_libmultiprocess package. The libmultiprocess & capnp packages are both fat packages producing multiple outputs which are sometimes unnecessary. A less wasteful capnp packaging would be divided in two parts, with one package to build command-line tools capnp, etc which only need to be natively compiled, and separate package to build libraries libcapnp.a, etc which need to be cross and native. Similarly a less wasteful libmultiprocess packaging would define one package to build the mpgen codegen tool, which only needs to be built natively, and another package to build the libmultiprocess.a library. When I wrote these packages, I wrote them as fat packages building codegen executables and runtime libraries together, instead smaller packages building them separately, just for simplicity to avoid repeating the similar build definitions. I also didn’t try to trim unnecessary outputs because they didn’t seem very expensive to build.
So I don’t understand how the second commit dropping native_libmultiprocess could work, given that bitcoin build needs it to provide a native mpgen codegen executable. The comment “I don’t understand why it currently has a dependency” was just trying to say that libmultiprocess package should not depend on native_libmultiprocess package, not that native_libmultiprocess should not exist at all.
Questions / Recommendations for this PR
I wonder if first commit is sufficient to fix problems, and maybe second and third commits are unnecessary? I think I would drop them and maybe make them into separate followups if they provide advantages.
This is more of a long term question, and maybe an ignorant one since I’m not familiar with the Guix build, but why do even need the depends system anymore? I’d expect the Guix build to define guix packages instead of using depends packages. I don’t personally don’t use depends since I use Nix and write a shell.nix to bring in developer dependencies. I’d expect other developers to use their own preferred package managers. Do people actually use the depends system, or is there a thought of dropping it and just switching to guix packages for QA and releases and standard packages for developing?
Thank you for your review.
I wonder if first commit is sufficient to fix problems, and maybe second and third commits are unnecessary?
Dropped second and third commits.
9+endif
10+
11+define $(package)_set_vars :=
12+ifneq ($(host),$(build))
13+$(package)_cmake_opts := -DCAPNP_EXECUTABLE="$$(native_capnp_prefixbin)/capnp"
14+$(package)_cmake_opts += -DCAPNPC_CXX_EXECUTABLE="$$(native_capnp_prefixbin)/capnpc-c++"
In commit “build: Fix libmultiprocess cross-compiling to Linux hosts” (c0f5cc14ef9fae2b2de4222ee061729629ebb6b4)
Note (just for understanding): This overrides the capnp tool paths used by the cross-compiled libmultiprocess build, so it will run native capnp tools, instead of trying to run cross compiled ones found by find_package(CapnProto). The find_package line in the libmultiprocess build is only intended to find the cross-compiled capnp library paths, but it also finds the cross-compiled tool paths and will use them if they are not overridden.
There’s some documentation about overriding these two variables: https://github.com/capnproto/capnproto/blob/55a368beb987abf9eeb9b3843e9c5423ad37ab29/c%2B%2B/cmake/CapnProtoConfig.cmake.in#L18-L22
3@@ -4,6 +4,16 @@ $(package)_download_path=$(native_$(package)_download_path)
4 $(package)_file_name=$(native_$(package)_file_name)
5 $(package)_sha256_hash=$(native_$(package)_sha256_hash)
6 $(package)_dependencies=native_$(package) capnp
In commit “build: Fix libmultiprocess cross-compiling to Linux hosts” (c0f5cc14ef9fae2b2de4222ee061729629ebb6b4)
In a different PR, could probably drop native_$(package) as a dependency here. The cross compiled libmultiprocess packages doesn’t actually depend on native libmultiprocess package for anything. (This is different from the cross compiled capnp which does depend on the native capnp for codegen tools used during cross-compilation).
Code review ACK c0f5cc14ef9fae2b2de4222ee061729629ebb6b4
Thanks for fixing this
This is more of a long term question, and maybe an ignorant one since I’m not familiar with the Guix build, but why do even need the depends system anymore? I’d expect the Guix build to define guix packages instead of using depends packages. I don’t personally don’t use depends since I use Nix and write a shell.nix to bring in developer dependencies. I’d expect other developers to use their own preferred package managers. Do people actually use the depends system, or is there a thought of dropping it and just switching to guix packages for QA and releases and standard packages for developing?
The depends system exists as a middle ground between requiring Guix to build Bitcoin core, and being completely reliant on system packages, neither of which are ideal / usable for everyone scenarios.
I’d expect other developers to use their own preferred package managers.
This is fine unless your package manger has versions of our dependencies that are too old, broken in some way, or just not packaged. Package managers also have a tendency to patch their sources, and those patches may undermine some assumption we make, or subtly break things. Having the ability to compile using unpatched (or only our patches) source is nice. Package manger vendored libraries are also no-doubt configured / compiled (for better or worse) differently to how we build libraries in depends.
I think for self-compiling end users, who may want to customize their builds slightly (i.e no port forwarding libs, no wallet, additional compiler flags etc), but otherwise want to compile a Bitcoin Core as close as possible to how we build releases, depends is very useful.
Depends is also very convenient for cross-compilation. If we delete depends tomorrow, and I’m using a Debian machine, and want to cross-compile a deployable Bitcoin Core for macOS (with all optional features + gui). What would be your recommended way of doing so? It’s probably going to end up being something like what depends would have been doing anyways, but I’m just going to have to put everything together manually, rather than running make HOST=x and plugging a CONFIG_SITE into configure. Slightly less involved (no SDK and less convoluted compiler setup), but a similar scenario, is if I’m cross-compiling for Windows.
Having a single source of truth, that defines the dependencies required to build Bitcoin Core, and how they are patched / compiled, which isn’t reliant on other systems (i.e guix / nix) or tooling (other than, fairly basic compiler + utls + make etc), I think, is important, and, if for some reason we had to move away from Guix tomorrow, we could take depends, plug it into a different setup/environment that provides reproducibility, and just carry on.
Thanks @fanquake! I didn’t realize is that GUIX isn’t actually available on macos. I assumed guix was like nix could easily be installed everywhere. I also didn’t realize our GUIX build doesn’t really define packages. It just calls make -C depends from a shell script. So my comment was written with false assumption that there was no need for depends/packages/boost.mk and similar package definitions because there were GUIX versions of these packages.
If we delete depends tomorrow, and I’m using a Debian machine, and want to cross-compile a deployable Bitcoin Core for macOS (with all optional features + gui). What would be your recommended way of doing so?
Presumably guix has an option to cross compile packages. In nix it is something like --arg crossSystem '{ config = "aarch64-unknown-linux-gnu"; }'
The depends system exists as a middle ground between requiring Guix to build Bitcoin core, and being completely reliant on system packages, neither of which are ideal / usable for everyone scenarios.
The depends system just seems like a bootleg version of nix or guix to me with nonstandard package definitions that are fragile painful to debug[*]. So I don’t know why anybody would use depends if there were guix package definitions easily available somewhere. This seems like it is not the case, and guix itself is not very portable, and guix may be generally less useful as a package manager for development than nix (https://nix.dev/).
[*] My main problem debugging depends packages is It seems to love immediately deleting stuff after building it. So if anything isn’t built correctly I have to wait for a slow rebuild and figure out some incantation that will do a partial build keeping the output I want to look at. I was waiting for GUIX to solve this but I guess the scope of our GUIX build is just to provide a build environment, not be a development package manager.
Presumably guix has an option to cross compile packages. In nix it is something like
--arg crossSystem '{ config = "aarch64-unknown-linux-gnu"; }'
Yes, that is possible. Although just like NixOS, it would produce libraries and executables that expects a loader and libs under /gnu/store just like /nix/store.
This seems like it is not the case, and guix itself is not very portable, and guix may be generally less useful as a package manager for development than nix (https://nix.dev/).
Agreed! Although to be clear there’s nothing about Guix that makes it inherently that makes it less portable, it’s just that no one has done the work yet.
Our usecase for Guix is specifically has a bootstrappable and reproducible release builder. And it works very well for that. NixOS doesn’t have a good bootstrapping story so far (although nothing about NixOS makes it inherently unable to be bootstrapped), and we will see how that evolves.
[*] My main problem debugging depends packages is It seems to love immediately deleting stuff after building it. So if anything isn’t built correctly I have to wait for a slow rebuild and figure out some incantation that will do a partial build keeping the output I want to look at. I was waiting for GUIX to solve this but I guess the scope of our GUIX build is just to provide a build environment, not be a development package manager.
Have you tried: make -C depends qt_{configured,built,staged}?
Presumably guix has an option to cross compile packages. In nix it is something like
--arg crossSystem '{ config = "aarch64-unknown-linux-gnu"; }'Yes, that is possible. Although just like NixOS, it would produce libraries and executables that expects a loader and libs under /gnu/store just like /nix/store.
This is up to the package definitions, no? The autoconf/cmake wrappers nix and guix provide would build everything with /nix/store or /gnu/store prefixes by default, but we wouldn’t call these. The guix packages should be able to perform the same build steps as depends packages, calling ./configure manually with all same arguments and outputs. The package outputs would physically be stored in /gnu/store, but that is no different than depends system physically storing outputs in depends/x86_64-pc-linux-gnu. The place where outputs are stored is not the place executables and libraries need to run from.
This seems like it is not the case, and guix itself is not very portable, and guix may be generally less useful as a package manager for development than nix (https://nix.dev/).
Agreed! Although to be clear there’s nothing about Guix that makes it inherently that makes it less portable, it’s just that no one has done the work yet.
Fair enough. Nix definitely has its warts as well. But I was surprised to see guix wasn’t trying to compete as as a portable package manager and development tool the same way https://nix.dev/ wants to (not to mention other package managers like vcpkg, conan, spack, etc)
But in general it does seem not feasible to get rid of depends packages and replace them with guix packages if guix doesn’t run on a major platform like macos.
[*] My main problem debugging depends packages is It seems to love immediately deleting stuff after building it. So if anything isn’t built correctly I have to wait for a slow rebuild and figure out some incantation that will do a partial build keeping the output I want to look at. I was waiting for GUIX to solve this but I guess the scope of our GUIX build is just to provide a build environment, not be a development package manager.
Have you tried:
make -C depends qt_{configured,built,staged}?
Yes, but I forgot to mention my other pain point with the depends system, which is that every time I made a change to funcs.mk, it would rebuild the everything from scratch. I’m sure depends system is ok if you work with it regularly, but my experience trying to write simple packages for it in #16367 was very painful between it deleting build outputs and constantly rebuilding boost for no reason, having a long list of phases I don’t understand, and just generally being grouchy package manager. I also had problems with the way it downloaded tarballs. Like it would download a tarball, and the hash wouldn’t match, and then it would delete the result, or it would fail to recover after you updated the hash or something like that. I can’t remember the details. Plus config.site stuff passing dependencies between packages is fragile and sucks. I don’t know. I do know I was perfectly happy developing with ubuntu packages before, and I am perfectly happy developing with nix packages now. But every time a depends issue like this one comes up, I dread having to debug it and wish I were dealing with a normal packaging system.
I guess on the plus side, if we want to minimize dependencies, maybe having the depends system make it painful to add dependencies is a good thing!
I’m all for minimizing dependencies to the point where needing a package manager becomes unnecessary.