guix: build multiprocess

Sjors commented at 7:22 am on July 19, 2024: member

In order to run a Stratum v2 Template Provider as a “sidecar” via IPC, and to have bitcoin-node be reproducibly built, libmultiprocess needs to be built with guix.

Presumably this would happen as part of #10102 or #30437 eventually.

However if the TP “sidecar” is built on top of the Bitcoin Core codebase, like https://github.com/Sjors/bitcoin/pull/48 attempts, then that binary should also be Guix built. So it makes sense to track this independently.

I made a naive attempt in https://github.com/Sjors/bitcoin/pull/48/commits/33d22b9276e10ce8e0cfabe2a5b81d3cb23d8609 by simply adding MULTIPROCESS=1 to the depends build in contrib/guix/libexec/build.sh. That didn’t work because curl is not accessible.

A workaround along those lines would be fine for https://github.com/Sjors/bitcoin/pull/48 for the time being though.

cc @ryanofsky

fanquake commented at 9:27 am on July 19, 2024: member

That didn’t work because curl is not accessible.

Can you elaborate? I did a build with MULTIPROCESS=1 (https://github.com/fanquake/bitcoin/tree/guix_multiprocess) and it worked fine:

0# time BASE_CACHE="/base_cache" SOURCES_PATH="/sources" SDK_PATH="/SDKs" HOSTS="x86_64-linux-gnu" ./contrib/guix/guix-build 
1e4e606e3a7076fe0e1f93b98ecdfbbfcd552a3c25f368684f3e94aab919058ce  guix-build-e0e38abd03a0/output/dist-archive/bitcoin-e0e38abd03a0.tar.gz
2e8b4c9f20abfe29fa1f0fa47515a28d169fcc8c09077dfec816eb0b3ba95f834  guix-build-e0e38abd03a0/output/x86_64-linux-gnu/SHA256SUMS.part
37c3d1af1872ba0c2dfb6d0f7f6eb617974814eb3c8cfd1f137839b62a776452c  guix-build-e0e38abd03a0/output/x86_64-linux-gnu/bitcoin-e0e38abd03a0-x86_64-linux-gnu-debug.tar.gz
4456fde063956f820c443e45b58a9303b51ac1b648852aa3d22db29191c60b704  guix-build-e0e38abd03a0/output/x86_64-linux-gnu/bitcoin-e0e38abd03a0-x86_64-linux-gnu.tar.gz

Sjors commented at 10:34 am on July 19, 2024: member

0make: Entering directory '/bitcoin/depends'
1Checksum missing or mismatched for native_capnp source. Forcing re-download.
2Fetching capnproto-c++-1.0.1.tar.gz from https://capnproto.org/
3/gnu/store/v9p25q9l5nnaixkhpap5rnymmwbhf9rp-bash-minimal-5.1.16/bin/sh: line 1: curl: command not found
4Fetching capnproto-c++-1.0.1.tar.gz from https://bitcoincore.org/depends-sources
5/gnu/store/v9p25q9l5nnaixkhpap5rnymmwbhf9rp-bash-minimal-5.1.16/bin/sh: line 1: curl: command not found
6make: *** [funcs.mk:302: /home/guix/depends-sources-cache/download-stamps/.stamp_fetched-native_capnp-capnproto-cxx-1.0.1.tar.gz.hash] Error 127
7make: Leaving directory '/bitcoin/depends'

But let me try putting MULTIPROCESS=1 at the bottom like you did.

fanquake commented at 10:37 am on July 19, 2024: member

You need to have curl installed to download things with curl. The downloading happens outside guix, so this is a local issue.

Sjors commented at 10:41 am on July 19, 2024: member

The downloading happens outside guix, so this is a local issue.

Ah

I do have curl installed though.

0curl --version
1curl 8.5.0 (x86_64-pc-linux-gnu) libcurl/8.5.0 OpenSSL/3.0.13 zlib/1.3 brotli/1.1.0 zstd/1.5.5 libidn2/2.3.7 libpsl/0.21.2 (+libidn2/2.3.7) libssh/0.10.6/openssl/zlib nghttp2/1.59.0 librtmp/2.3 OpenLDAP/2.6.7
2Release-Date: 2023-12-06, security patched: 8.5.0-2ubuntu10.1
3Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
4Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

Sjors commented at 10:45 am on July 19, 2024: member

I tried with your branch as well, and get the same failure. So I agree it must be some difference between our guix setups. But I’ve signed plenty releases with it, and never ran into this with any of the build change PR’s for which I made guix builds.

fanquake commented at 10:48 am on July 19, 2024: member

You should also add MULTIPROCESS=1 to your guix build command, so that your multiprocess tarballs are redownloaded outside guix.

The issue is that the mismatch is detected at build time in the container, where redownloading will fail due to the lack of curl.

Sjors commented at 10:52 am on July 19, 2024: member

Ah that worked, thanks!

In that case I don’t think there’s anything specific that needs to happen.

Sjors closed this on Jul 19, 2024

guix: build multiprocess #30483