Add bitcoin-{node,gui} to release binaries for IPC

Sjors commented at 4:04 pm on February 5, 2025: member

Have depends make libmultiprocess by default, which has the effect of including it in the release binaries. Except for Windows and OpenBSD which are not supported yet, the latter due to a fairly trivial upstream issue.

The initial main use case is to enable experimental support for the Mining IPC interface. A working example of a Stratum v2 Template Provider client using this interface can be found here: https://github.com/Sjors/bitcoin/pull/48.

Additionally the bitcoin-node and bitcoin-gui are added to Maintenance.cmake.

Builds on #30975 (which originally tried to do this) and #31741.

See #31756 for discussion of when this should happen. See also #31098 for any remaining issues specific to the Mining interface. Will keep this draft for now.

Guix hashes:

 0f59b9d013a55e786940364f8fb543809474a87ce31d318706553a75244f10943  guix-build-7b632b77c928/output/aarch64-linux-gnu/SHA256SUMS.part
 1719b3221a8ad7bb54dfa1fcd399da88d63aa673364b9a350f258fe3c9896ff51  guix-build-7b632b77c928/output/aarch64-linux-gnu/bitcoin-7b632b77c928-aarch64-linux-gnu-debug.tar.gz
 2c6d69b2c82fc1cbdf97b80460c28838a71d8f851631a29cbea54088bc53f9ace  guix-build-7b632b77c928/output/aarch64-linux-gnu/bitcoin-7b632b77c928-aarch64-linux-gnu.tar.gz
 38b4e359efae5dc8760e85cd7b49f42bcef1d0a49fd5e3958e370aaa3fa54eceb  guix-build-7b632b77c928/output/arm-linux-gnueabihf/SHA256SUMS.part
 46eca404744b564aa85aab70c8c06b12b96adae2e8421244f6b0512eb27f17d43  guix-build-7b632b77c928/output/arm-linux-gnueabihf/bitcoin-7b632b77c928-arm-linux-gnueabihf-debug.tar.gz
 557a0537af734427987e27f466ce71486bb491983b45b57cb5627b3d730d80fe9  guix-build-7b632b77c928/output/arm-linux-gnueabihf/bitcoin-7b632b77c928-arm-linux-gnueabihf.tar.gz
 6e7479069c3dbacee72bdb8644d4a725390982eaa5e7504af484afc6653edd0b5  guix-build-7b632b77c928/output/arm64-apple-darwin/SHA256SUMS.part
 7e00d43ab179ab380bd0565fa2770b6830c3a931cf134ccc83b08b56278c18f16  guix-build-7b632b77c928/output/arm64-apple-darwin/bitcoin-7b632b77c928-arm64-apple-darwin-unsigned.tar.gz
 89440db1cc618da37a822b89f2583d4897eafaf32316c4c069d73d133d4f2fac1  guix-build-7b632b77c928/output/arm64-apple-darwin/bitcoin-7b632b77c928-arm64-apple-darwin-unsigned.zip
 92462593777d882663d10a03ae55c94ba27d9bf4cace8f5b4c5a6cf3d644abfbe  guix-build-7b632b77c928/output/arm64-apple-darwin/bitcoin-7b632b77c928-arm64-apple-darwin.tar.gz
109b4b24daaf0d1ed58348bd649de25669410e17b8370d9bc3c3ff205646145f95  guix-build-7b632b77c928/output/dist-archive/bitcoin-7b632b77c928.tar.gz
11cff99ea14fb4a83a7d7d90eb5259c0604cd4024f629180c5b9004a02fbeeeeb6  guix-build-7b632b77c928/output/powerpc64-linux-gnu/SHA256SUMS.part
12ba60b5af2d909f4d85803286b5bfb70ee2b8d9fe37d4b47fbf2652ca9ef9f5a5  guix-build-7b632b77c928/output/powerpc64-linux-gnu/bitcoin-7b632b77c928-powerpc64-linux-gnu-debug.tar.gz
137b46707cc79a6b00d70b8ceb5040dd228fe5bd5e55051efc7f000015d9fd409f  guix-build-7b632b77c928/output/powerpc64-linux-gnu/bitcoin-7b632b77c928-powerpc64-linux-gnu.tar.gz
14403c66f87a80a3e0eb9356fba2ecb012ee6e894a6268ec9d93986652793ca5f9  guix-build-7b632b77c928/output/riscv64-linux-gnu/SHA256SUMS.part
156518077f268e8b2e99758d7debb4ab45268ec144ff42de72cd810c020baa15c6  guix-build-7b632b77c928/output/riscv64-linux-gnu/bitcoin-7b632b77c928-riscv64-linux-gnu-debug.tar.gz
16082768355971e67f5b6179e285e628bb992c169f0d0cc25f31b45d40e1db1075  guix-build-7b632b77c928/output/riscv64-linux-gnu/bitcoin-7b632b77c928-riscv64-linux-gnu.tar.gz
17e2e9c1d0187b95e95d54ba066d73978847675957f7d112d362e5484641025464  guix-build-7b632b77c928/output/x86_64-apple-darwin/SHA256SUMS.part
185823cc70d455d7eec4663565cbf7eb4bcb5e71cdcdaca5c627fa6bc5d0d16ccc  guix-build-7b632b77c928/output/x86_64-apple-darwin/bitcoin-7b632b77c928-x86_64-apple-darwin-unsigned.tar.gz
19a10c49b7cbb6fd1c37b9a667347eb775e5c24d6b4971485e2fac604d131c6946  guix-build-7b632b77c928/output/x86_64-apple-darwin/bitcoin-7b632b77c928-x86_64-apple-darwin-unsigned.zip
20c7d7d6b498acca964a552b7411a03a3697bd26270dde943ded51cab21b3520c7  guix-build-7b632b77c928/output/x86_64-apple-darwin/bitcoin-7b632b77c928-x86_64-apple-darwin.tar.gz
2126479789047c7e2e385684a580cafa411c082aa3926779e3a172166ea3d92425  guix-build-7b632b77c928/output/x86_64-linux-gnu/SHA256SUMS.part
22e354f3364147edbca514d2c03835826a5934d65a45c5c38be21e03df8c06a616  guix-build-7b632b77c928/output/x86_64-linux-gnu/bitcoin-7b632b77c928-x86_64-linux-gnu-debug.tar.gz
2373978916ae3a23a8732308d7058647829ab755fabd346da3c8a223a19ded0d2d  guix-build-7b632b77c928/output/x86_64-linux-gnu/bitcoin-7b632b77c928-x86_64-linux-gnu.tar.gz
24c5de6037faa7140c1ca563280e42aa34deba1ca1d90b56453f3eeca043da3431  guix-build-7b632b77c928/output/x86_64-w64-mingw32/SHA256SUMS.part
251fc7d801069c258001545a3b890c15bd7ed07a1f1b3d6d7efa3bd61ac95bef77  guix-build-7b632b77c928/output/x86_64-w64-mingw32/bitcoin-7b632b77c928-win64-debug.zip
2694eff72ebc49877e7637dd2fe99619010d5ac5cd1e9ed95e4cbdd296b7fce0e5  guix-build-7b632b77c928/output/x86_64-w64-mingw32/bitcoin-7b632b77c928-win64-setup-unsigned.exe
2715c323f6115ca2b03722edc738d96f9f1ceb13fd1391f009c93f88a660ae57c8  guix-build-7b632b77c928/output/x86_64-w64-mingw32/bitcoin-7b632b77c928-win64-unsigned.tar.gz
28746fb9257f964b124a9446809633e00bf216a7c4945e0bd81ae552ebca48e0cf  guix-build-7b632b77c928/output/x86_64-w64-mingw32/bitcoin-7b632b77c928-win64.zip

DrahtBot commented at 4:04 pm on February 5, 2025: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31802.

Reviews

See the guideline for information on the review process. A summary of reviews will appear here.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#31982 (scripted-diff: rename libmultiprocess repository by fanquake)
#31665 (build: Make config warnings fatal if -DWCONFIGURE_ERROR=ON by davidgumberg)
#31375 (multiprocess: Add bitcoin wrapper executable by ryanofsky)
#31349 (ci: detect outbound internet traffic generated while running tests by vasild)
#31282 (refactor: Make node_id a const& in RemoveBlockRequest by maflcko)
#31161 (cmake: Set top-level target output locations by hebasto)
#30595 (kernel: Introduce initial C header API by TheCharlatan)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

DrahtBot added the label Needs rebase on Feb 6, 2025

Sjors force-pushed on Feb 7, 2025

sipa commented at 9:03 pm on February 7, 2025: member

Some chatter from IRC:

 017:21:41 < darosior> It might be confusing to release both a bitcoin-wallet utility and a bitcoin-wallet binary as part of multiprocess?
 117:24:08 < darosior> We could rename the utility, but then it would be nice to at least have one deprecation cycle. Given recent momentum i estimate it's possible we might release multiprocess in 
 2                     30.0, which means if we want to deprecate the bitcoin-wallet utility name we should do it.. now?
 317:33:31 < sipa> bitcoin-wallet-util ?
 4...
 523:38:57 < _aj_> darosior: bitcoin-wallet-process, bitcoin-gui-process, etc? it's multi *-process!
 603:04:34 < Sjors[m]> darosior: at the moment there is no wallet binary if were to enable multiprocess. That won't happen until [#19460](/bitcoin-bitcoin/19460/).
 7...
 803:05:18 < Sjors[m]> Or maybe already in [#10102](/bitcoin-bitcoin/10102/)
 9...
1003:05:50 < Sjors[m]> In any case [#31802](/bitcoin-bitcoin/31802/) only adds bitcoin-node and bitcoin-gui.
11...
1203:07:19 < Sjors[m]> Though if we want to rename the utility eventually, it's always better to do it early.

Sjors commented at 10:58 am on February 8, 2025: member

@sipa I opened #31827

onlinesipahimithu commented at 1:26 am on February 9, 2025: none

GOOD lesson from

bitcoin deleted a comment on Feb 9, 2025

Sjors force-pushed on Feb 10, 2025

Sjors force-pushed on Feb 13, 2025

DrahtBot removed the label Needs rebase on Feb 13, 2025

Sjors force-pushed on Feb 17, 2025

DrahtBot added the label Needs rebase on Feb 20, 2025

Squashed 'src/ipc/libmultiprocess/' content from commit 1954f7f65661

git-subtree-dir: src/ipc/libmultiprocess
git-subtree-split: 1954f7f65661d49e700c344eae0fc8092decf975

db6b95d53a

Merge commit 'db6b95d53a69a009330dd69e3cc25b40bf121d7f' as 'src/ipc/libmultiprocess' 76b5b4310e

scripted-diff: s/WITH_MULTIPROCESS/ENABLE_IPC/ in cmake

Rename WITH_MULTIPROCESS to ENABLE_IPC, because ENABLE_IPC is a more accurate
name for the feature. It controls whether the src/ipc/ directory is built and
whether IPC features like -ipcbind, -ipcconnect, and -ipcfd are available. It
does NOT currently enable multiprocess features which are implemented in #10102
building on top of the IPC features. It will also no longer (as of the next
commit), control whether a find_package call is made so the "WITH_" prefix is
also inappropriate.

-BEGIN VERIFY SCRIPT-
git grep -l WITH_MULTIPROCESS | xargs sed -i s/WITH_MULTIPROCESS/ENABLE_IPC/g
-END VERIFY SCRIPT-

d61d7e8ddf

cmake: Support building with libmultiprocess subtree

When ENABLE_IPC option is on, build with libmultiprocess subtree and
`add_subdirectory(src/ipc/libmultiprocess)` instead of external package
and `find_package(Libmultiprocess)` by default.

Behavior can be toggled with `WITH_LIBMULTIPROCESS` option. Using a subtree
should be more convenient for most bitcoin developers, but using an external
package is more convenient for developing in the libmultiprocess repository.

The `WITH_LIBMULTIPROCESS` option is also used to avoid needing to changing the
depends build here. But in later commits, the depends build is switched to use
the add_subdirectory build as well.

Co-authored-by: Cory Fields <cory-nospam-@coryfields.com>

24f53bef69

cmake: Fix ctest mptest "Unable to find executable" errors

This change is technically not needed to add libmultiprocess as a subtree, but
it avoids a CI failure in followup PR #30975 which enables multiprocess build
option in more CI jobs. In that PR, several jobs fail due to the mptest
executable not being built by default, as reported
https://github.com/bitcoin/bitcoin/pull/30975#issuecomment-2623801480

95bbdd3523

cmake: Fix warnings from boost headers

This change is technically not needed to add libmultiprocess as a subtree, but
it avoids a CI failure in followup PR #30975 which enables multiprocess build
option in more CI jobs. In that PR, the "macOS 14 native no depends job" fails
due to warnings in boost headers treated as errors, reported in
https://github.com/bitcoin/bitcoin/pull/30975#issuecomment-2623801480 and
https://github.com/chaincodelabs/libmultiprocess/issues/138

a36912427a

cmake: Fix clang-tidy "no input files" errors

This change is technically not needed to add libmultiprocess as a subtree, but
it avoids a CI failure in followup PR #30975 which enables multiprocess build
option in more CI jobs. In that PR, clang-tidy job fails due to missing
generated example files as reported
https://github.com/bitcoin/bitcoin/pull/30975#issuecomment-2627116832

Different fixes were suggested
https://github.com/bitcoin/bitcoin/pull/30975#issuecomment-2627152623 and
https://github.com/bitcoin/bitcoin/pull/30975#issuecomment-2627403382

Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>

00b1406430

cmake: Fix fuzzer "multiple definition of `main'" errors

This change is technically not needed to add libmultiprocess as a subtree, but
it avoids a CI failure in followup PR #30975 which enables multiprocess build
option in more CI jobs. Specifically this makes it possible to use the
BUILD_FOR_FUZZING and ENABLE_IPC options together easily.

The change just updates the cmake build so when -DSANITIZERS=fuzzer is
specified, the fuzz test binary is built with -fsanitize=fuzzer (so it can use
libFuzzer's main function), and libraries are built with
-fsanitize=fuzzer-no-link (so they can be linked into other executables with
their own main functions).

Previously when -DSANITIZERS=fuzzer was specified, -fsanitize=fuzzer was
applied to ALL libraries and executables.  This was inappropriate because it
made it impossible to build any executables other than the fuzz test executable
without triggering link errors:

- "multiple definition of `main'"
- "undefined reference to `LLVMFuzzerTestOneInput'"

if they depended on any libraries instrumented for fuzzing.

This was especially a problem when the ENABLE_IPC option was set because the
mpgen code generator could not be built, so nothing else that depended on
generated sources including the fuzz test binary could be built either.

6f1875c8ae

doc: Update documentation to explain libmultiprocess subtree

Co-authored-by: Cory Fields <cory-nospam-@coryfields.com>

9394be1ad4

lint: Add exclusions for libmultiprocess subtree

Without this change linter produces errors about:

- Use of std::filesystem the libmultiprocess example program.
- Use of locale-dependent functions in example program, in the build time code
  generator, and in the runtime library for debug logging.
- Include guards not beginning with BITCOIN_

7b58c3f47b

depends, moveonly: split up int_get_build_id function

Move parts of the int_get_build_id into a new int_get_build_properties
function. There is no change in behavior. This just organizes assignments
better so some build properties can be used to help compute build ids in the
next commit.

e9259f537c

depends: Switch libmultiprocess packages to use local git subtree

With newly introduced libmultiprocess subtree, there's no need for depends
system to download and track changes to the upstream repository.

Note that adding the libmultiprocess subtree does not allow dropping
libmultiprocess packages from the depends build, because libmultiprocess
includes a code generation tool called mpgen, and in cross-compiled builds,
bitcoin core's cmake build system doesn't have access to a native toolchain and
can't build mpgen itself, so the depends system (or the native environment if
not using depends) needs to supply it.

61e4a1ee9d

depends: remove non-native libmultiprocess build

Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>

c2c5a0f492

ryanofsky referenced this in commit 83e40d3b52 on Feb 24, 2025

ryanofsky referenced this in commit 8619f03ec2 on Feb 24, 2025

ryanofsky referenced this in commit cbb7b41c20 on Feb 24, 2025

fanquake referenced this in commit 01f7715766 on Feb 25, 2025

fanquake referenced this in commit ba0a4391ff on Feb 25, 2025

ci: build multiprocess for all depends jobs

Except:
1. i686, DEBUG (changed to "no multiprocess")
2. Windows due to lack of support

aa1aae96e2

ci: enable multiprocess for non-depends jobs

Install capnp where needed.

The bitcoin-node binary is built on all platforms which have
multiprocess enabled, but for functional tests it's only used in
the macOS native (no depends) and CentOS native (depends) jobs.

6f0b192275

pablomartin4btc referenced this in commit bd2453d134 on Feb 26, 2025

pablomartin4btc referenced this in commit 75d5d235a6 on Feb 26, 2025

build: add bitcoin-{node,gui} to Maintenance.cmake 7e05d8871b

build: depends makes libmultiprocess by default

Except for Windows and OpenBSD.

7b632b77c9

Sjors force-pushed on Feb 27, 2025

DrahtBot removed the label Needs rebase on Feb 27, 2025

ngn999 referenced this in commit fd12bfb763 on Feb 27, 2025

DrahtBot added the label Needs rebase on Mar 5, 2025

DrahtBot commented at 2:54 pm on March 5, 2025: contributor

🐙 This pull request conflicts with the target branch and needs rebase.

Add bitcoin-{node,gui} to release binaries for IPC #31802

Code Coverage & Benchmarks

Reviews

Conflicts