Multiprocess build support

ryanofsky commented at 8:44 pm on April 16, 2020: member

This PR is part of the process separation project.

This PR consists of build changes only. It adds an --enable-multiprocess autoconf option (off by default and marked experimental), that builds new bitcoin-node and bitcoin-gui binaries. These currently function the same as existing bitcoind and bitcoin-qt binaries, but are extended in #10102 with IPC features to execute node, wallet, and gui functions in separate processes.

In addition to adding the --enable-multiprocess config flag, it also adds a depends package and autoconf rules to build with the libmultiprocess library, and it adds new travis configuration to exercise the build code and run functional tests with the new binaries.

The changes in this PR were originally part of #10102 but were moved into #16367 to be able to develop and review the multiprocess build changes independently of the code changes. #16367 was briefly merged and then reverted in #18588. Only change since #16367 has been dropping the native_boost.mk depends package which was pointed out to be no longer necessary in #16367 (comment) and #18588#pullrequestreview-391765649

MarcoFalke commented at 8:56 pm on April 16, 2020: member

re-ACK c8672dfeb7443d83fbae02c1b1cd867f83b972b1, only change since my last review is dropping boost and small doc fixup. This review does not cover libmultiprocess 🚕

Signature:

 0-----BEGIN PGP SIGNED MESSAGE-----
 1Hash: SHA512
 2
 3re-ACK c8672dfeb7443d83fbae02c1b1cd867f83b972b1, only change since my last review is dropping boost and small doc fixup. This review does not cover libmultiprocess 🚕
 4-----BEGIN PGP SIGNATURE-----
 5
 6iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
 7pUgnPQwAntHVvhb+Ic5V+GQI8wMKrY52J9CGmWu9dLo/0dH9zC3oO+dyFk4BRcNT
 8HZ5Hai5yy7+OxfWvqf88isAqlUfNa7z6jmT/YZfa3OviLufzS00kazMn1CFhAOvJ
 9O2as6oAZUifG6YrQmtSCTw1fN79oCq/NP5VkufNauSvHDtRhYIiK7p3kZqo4Jm8w
10hs5WzlEqcRpxpkbbLYXRh4Be6Q7UL5cRGhYz9U8V79uo0Cp3klo+onXwofrYc/jq
11tBMVAS5Qu7tcLEAIsr/NLlp+OSjy1/gkeM6vlR3gmvzj86b7gKnHpBQeQZI1PcNo
12eZ/cN2kDgB5/fLIT+KX9e25qiSJm0XCAmN7fI64oTI85PRpQcVALEkf+jJmgGh8r
13urLfBT6WUesJMHLcQLrOrdvX2aFMiLE/vSjuxpRqKJhSGvFLCg86R6t0SflXB6WF
14FTp9VW76Ofi8ZWcA0+cU1bHjF2GUP33TddGE+O9posORfibm2FQ4n43gfp90izIE
15/YYDPbeG
16=IWHw
17-----END PGP SIGNATURE-----

Timestamp of file with hash 5ffc2ebe9acc3666c3cfa34fc843d6447defa592787071c5cc86bf728ace8075 -

laanwj added the label Build system on Apr 16, 2020

DrahtBot commented at 10:41 pm on April 16, 2020: member

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#19014 (test: Replace TEST_PREVIOUS_RELEASES env var with test_framework option by MarcoFalke)
#18980 (build: Decouple clientversion.cpp from the git repo by hebasto)
#18912 (ci: Run fuzz testing test cases (bitcoin-core/qa-assets) under valgrind to catch memory errors by practicalswift)
#18077 (net: Add NAT-PMP port forwarding support by hebasto)
#16549 (UI external signer support (e.g. hardware wallet) by Sjors)
#16546 (External signer support - Wallet Box edition by Sjors)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

in depends/packages/boost.mk:4 in c8672dfeb7 outdated

0@@ -1,7 +1,7 @@
1 package=boost
2 $(package)_version=1_70_0
3 $(package)_download_path=https://dl.bintray.com/boostorg/release/1.70.0/source/
4-$(package)_file_name=$(package)_$($(package)_version).tar.bz2
5+$(package)_file_name=boost_$($(package)_version).tar.bz2

fanquake commented at 11:55 pm on April 16, 2020:

This seems leftover from the boost changes. I’d suggest dropping as it would likely just be changed back. i.e: #17928

ryanofsky commented at 8:23 pm on April 17, 2020:

re: #18677 (review)

This seems leftover from the boost changes. I’d suggest dropping as it would likely just be changed back. i.e: #17928

Substituting $(package) in variable names seems pretty clearly correct and helps with package inheritance while substituting $(package) in external URLs seems pretty clearly incorrect and breaks package inheritance. But I can make a separate commit or PR for this change if it would be better

in doc/multiprocess.md:29 in c8672dfeb7 outdated

24+```
25+cd <BITCOIN_SOURCE_DIRECTORY>
26+make -C depends NO_QT=1 MULTIPROCESS=1
27+./configure --prefix=$PWD/depends/x86_64-pc-linux-gnu
28+make
29+src/bitcoin-node -regtest -printtoconsole -debug=ipc

fanquake commented at 0:26 am on April 17, 2020:

-debug=ipc doesn’t currently do anything right now, as it needs the changes in #10102?

ryanofsky commented at 8:23 pm on April 17, 2020:

re: #18677 (review)

-debug=ipc doesn’t currently do anything right now, as it needs the changes in #10102?

Yes, the comments above refer to 10102, explain the debug option and mention this

in src/Makefile.am:568 in c8672dfeb7 outdated

571   $(LIBBITCOIN_SERVER) \
572   $(LIBBITCOIN_WALLET) \
573   $(LIBBITCOIN_COMMON) \
574-  $(LIBUNIVALUE) \
575   $(LIBBITCOIN_UTIL) \
576+  $(LIBUNIVALUE) \

fanquake commented at 0:27 am on April 17, 2020:

Why is this reordering needed?

ryanofsky commented at 8:24 pm on April 17, 2020:

re: #18677 (review)

Why is this reordering needed?

Util library depends on univalue, and dependencies need to be listed after libraries that depend on them, so this is more correct. Without this change the error (on plain linux build) is:

0  CXXLD    bitcoin-wallet
1libbitcoin_util.a(libbitcoin_util_a-request.o): In function `JSONRPCReply[abi:cxx11](UniValue const&, UniValue const&, UniValue const&)':
2src/rpc/request.cpp:48: undefined reference to `UniValue::write[abi:cxx11](unsigned int, unsigned int) const'
3libbitcoin_util.a(libbitcoin_util_a-system.o): In function `ArgsManager::logArgsPrefix(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<UniValue, std::allocator<UniValue> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<UniValue, std::allocator<UniValue> > > > > const&) const':
4src/util/system.cpp:875: undefined reference to `UniValue::write[abi:cxx11](unsigned int, unsigned int) const'
5clang: error: linker command failed with exit code 1 (use -v to see invocation)

in depends/builders/darwin.mk:2 in c8672dfeb7 outdated

0@@ -1,5 +1,5 @@
1-build_darwin_CC:=$(shell xcrun -f clang)
2-build_darwin_CXX:=$(shell xcrun -f clang++)
3+build_darwin_CC:=clang
4+build_darwin_CXX:=clang++

fanquake commented at 1:05 am on April 17, 2020:

Can you clarify why this is needed? I did look at the commit comments about passing --sysroot and missing includes, however from what I can tell, there should be no change in behaviour with this diff. I did run a few compilation tests and didn’t see any obvious difference in compile flags/includes etc.

My assumption is that the problems seen (https://github.com/chaincodelabs/libmultiprocess/issues/5, #16367 (comment)) might have been due to an issue with the development environments i.e missing SDK headers, not installing the CommandLineTools etc.

Mostly curious because if this was an issue it seems like it shouldn’t be multiprocess specific? It also seems strange to drop the xcrun call only for clang/clang++, and not for the rest of the tools.

ryanofsky commented at 8:23 pm on April 17, 2020:

re: #18677 (review)

Can you clarify why this is needed? I did look at the commit comments about passing --sysroot and missing includes, however from what I can tell, there should be no change in behaviour with this diff. I did run a few compilation tests and didn’t see any obvious difference in compile flags/includes etc.

The comment the PR description links to is #16367 (comment). If you follow the instructions there creating test.cpp echo '#include <stdlib.h>' > test.cpp and running /Library/Developer/CommandLineTools/usr/bin/clang++ -c test.cpp, then you can see if the problem affects your system.

It has something to do with the SDK version:

My original fix from those comments was https://github.com/ryanofsky/bitcoin/commit/d6921b176186fd8304b6d6a13772fd0fc32fe726, but Cory’s suggestion https://github.com/ryanofsky/bitcoin/commit/04bec8d2844e0cd2e09a62b21b006292f33f8093 seemed better: just use native compiler for native packages, not the sdk compiler.

My assumption is that the problems seen (chaincodelabs/libmultiprocess#5, #16367 (comment)) might have been due to an issue with the development environments i.e missing SDK headers, not installing the CommandLineTools etc.

Not unless running the shell xcrun -f clang compiler without a sysroot option is supposed to work.

Mostly curious because if this was an issue it seems like it shouldn’t be multiprocess specific?

It’s not, just a bad interaction with native packages that don’t specify –sysroot, which we didn’t have previously.

It also seems strange to drop the xcrun call only for clang/clang++, and not for the rest of the tools.

That’s a good point. It didn’t occur to me to test changing all the other tools paths when I adopted Cory’s fix, and at the moment I don’t have a convenient way to retest. My original fix https://github.com/ryanofsky/bitcoin/commit/d6921b176186fd8304b6d6a13772fd0fc32fe726 doesn’t have this inconsistency, and I’d be happy to switch back to it if it’s preferable.

It’d be nice to have automated test coverage for this. It seems curious that the one osx job we have on travis does not test depends, IIUC?

fanquake commented at 1:36 am on April 17, 2020: member

Thanks for reopening and the responses here. I also read through the IRC logs from last night.

Few comments other than on the code. Nothing blocking while this is all still experimental.

Cross-compiling for macOS (on Debian 10.3) with MULTIPROCESS=1 currently fails while building capnp:

0make -C depends/ HOST=x86_64-apple-darwin16 MULTIPROCESS=1
1...
2libtool: link: clang++ -target x86_64-apple-darwin16 -mmacosx-version-min=10.12 --sysroot /bitcoin/depends/SDKs/MacOSX10.14.sdk -stdlib=libc++ -I./src -I./src -DKJ_HEADER_WARNINGS -DCAPNP_HEADER_WARNINGS -DCAPNP_INCLUDE_DIR=\"/bitcoin/depends/x86_64-apple-darwin16/include\" -D_THREAD_SAFE -pipe -O2 -D_THREAD_SAFE -D_THREAD_SAFE -o .libs/capnp src/capnp/compiler/module-loader.o src/capnp/compiler/capnp.o -Wl,-bind_at_load  -L/bitcoin/depends/x86_64-apple-darwin16/lib ./.libs/libcapnpc.dylib ./.libs/libcapnp-json.dylib /bitcoin/depends/work/build/x86_64-apple-darwin16/capnp/0.7.0-87ee694a224/.libs/libcapnp.dylib ./.libs/libcapnp.dylib /bitcoin/depends/work/build/x86_64-apple-darwin16/capnp/0.7.0-87ee694a224/.libs/libkj.dylib ./.libs/libkj.dylib
3echo capnp capnpc-c++ src/capnp/test.capnp src/capnp/test-import.capnp src/capnp/test-import2.capnp src/capnp/compat/json-test.capnp | (read CAPNP CAPNPC_CXX SOURCES && ./$CAPNP compile --src-prefix=./src -o./$CAPNPC_CXX:src -I./src $SOURCES)
4./capnp: line 117: /bitcoin/depends/work/build/x86_64-apple-darwin16/capnp/0.7.0-87ee694a224/.libs/capnp: cannot execute binary file: Exec format error
5./capnp: line 117: /bitcoin/depends/work/build/x86_64-apple-darwin16/capnp/0.7.0-87ee694a224/.libs/capnp: Success
6make[1]: *** [Makefile:4326: test_capnpc_middleman] Error 126
7make[1]: Leaving directory '/bitcoin/depends/work/build/x86_64-apple-darwin16/capnp/0.7.0-87ee694a224'
8make: *** [funcs.mk:254: /bitcoin/depends/work/build/x86_64-apple-darwin16/capnp/0.7.0-87ee694a224/./.stamp_built] Error 2
9make: Leaving directory '/bitcoin/depends'

Would you prefer these issues opened here or the libmultiprocess repo?

Can we pass --with-openssl=no (+ any other relevant flags) to the capnp configure? I assume we aren’t using any parts of it that require OpenSSL, and we might as well opt out of any autodetection and/or additional compilation.

Also, I didn’t notice before, but just wanted to note that building capnp currently requires gnu extensions. This is at odds with our current compiler requirements.

in src/Makefile.am:554 in c8672dfeb7 outdated

554-bitcoind_SOURCES = bitcoind.cpp
555-bitcoind_CPPFLAGS = $(AM_CPPFLAGS) $(BITCOIN_INCLUDES)
556-bitcoind_CXXFLAGS = $(AM_CXXFLAGS) $(PIE_FLAGS)
557-bitcoind_LDFLAGS = $(RELDFLAGS) $(AM_LDFLAGS) $(LIBTOOL_APP_LDFLAGS)
558+# bitcoind & bitcoin-node binaries #
559+bitcoin_common_sources = bitcoind.cpp

MarcoFalke commented at 11:48 am on April 17, 2020:

Sorry to be bikeshedding this again, but what about naming this node_sources or daemon_sources? As of now it sounds confusingly similar to libbitcoin_common, which is an utility library for all Bitcoin Core util binaries

ryanofsky commented at 5:08 pm on April 20, 2020:

re: #18677 (review)

Sorry to be bikeshedding this again, but what about naming this node_sources or daemon_sources? As of now it sounds confusingly similar to libbitcoin_common, which is an utility library for all Bitcoin Core util binaries

Switched to daemon_sources as suggested. Possible this is just bikeshedding with past-Marco #16367 (review), but I think all the suggestions have been good, and the confusion with libbitcoin_common didn’t occur to me before this

ryanofsky force-pushed on Apr 20, 2020

ryanofsky commented at 6:39 pm on April 20, 2020: member

Updated c8672dfeb7443d83fbae02c1b1cd867f83b972b1 -> f80a689013710a695370e207db0048373d252043 (pr/ipc-build2.1 -> pr/ipc-build2.2, compare) with x86_64-apple-darwin16 fixes and suggested automake renames

re: #18677#pullrequestreview-395075571

Cross-compiling for macOS (on Debian 10.3) with MULTIPROCESS=1 currently fails while building capnp:
0make -C depends/ HOST=x86_64-apple-darwin16 MULTIPROCESS=1
1...

Good catch, I hadn’t tested this configuration. I fixed this and more problems with it in the latest push.

Would you prefer these issues opened here or the libmultiprocess repo?

This was a depends system issue so it made sense here, but in general it can be hard to pin down where issues arise so either place should be fine

Can we pass --with-openssl=no (+ any other relevant flags) to the capnp configure? I assume we aren’t using any parts of it that require OpenSSL, and we might as well opt out of any autodetection and/or additional compilation.

I think you’d have to write an openssl depends package and add it to $(package)_dependencies in capnp.mk for there to be anything to detect. But maybe requesting to be built without openssl when openssl isn’t available would be a good way to demonstrate not liking openssl

Also, I didn’t notice before, but just wanted to note that building capnp currently requires gnu extensions. This is at odds with our current compiler requirements.

I don’t think gnu extensions are required, and I don’t see any problem building without them (following regular build steps but passing CXXFLAGS=-std=c++14 to configure). Digging into the history of this check it looks like it was added to work around a cygwin bug where snprintf wasn’t declared with -std=c++0x, but was with -std=gnu++0x

https://groups.google.com/d/msg/capnproto/l46nHozrhCY/vJTHMkJC-ZEJ https://groups.google.com/d/msg/capnproto/l46nHozrhCY/UG__SqiVbywJ https://stackoverflow.com/questions/20149633/how-to-use-snprintf-in-g-std-c11-version-4-8-2

ryanofsky referenced this in commit 1e94a2bcbc on Apr 22, 2020

ryanofsky force-pushed on Apr 22, 2020

ryanofsky force-pushed on Apr 23, 2020

luke-jr commented at 6:45 pm on April 24, 2020: member

I thought we were going to pivot this to just add a -process=node/wallet/gui option to the same bitcoin-qt binary?

ryanofsky commented at 7:02 pm on April 27, 2020: member

Updated f80a689013710a695370e207db0048373d252043 -> fba09de775f1eff68d2f72650bccc418cbe87cf8 (pr/ipc-build2.2 -> pr/ipc-build2.3, compare) to try to fix travis error https://travis-ci.org/github/bitcoin/bitcoin/jobs/677378707 missing pthread.h Updated fba09de775f1eff68d2f72650bccc418cbe87cf8 -> 474f4f1d02267c158001c5af0a558d24ef5f971d (pr/ipc-build2.3 -> pr/ipc-build2.4, compare) fixing travis error and moving mac os builder workaround to #18743

re: #18677 (comment)

I thought we were going to pivot this

I think the current naming is better, but I pointed out in the IRC meeting it is easy to change. If it’s important, would suggest opening a followup PR and considering this as usability change to discuss ideally after having some experience using the feature

ryanofsky referenced this in commit 114b2103ad on May 1, 2020

in configure.ac:1476 in 474f4f1d02 outdated

1414+  build_multiprocess=$libmultiprocess_found
1415+else
1416+  build_multiprocess=no
1417+fi
1418+
1419+AM_CONDITIONAL([BUILD_MULTIPROCESS],[test "x$build_multiprocess" = xyes])

hebasto commented at 5:20 am on May 2, 2020:

The BUILD_MULTIPROCESS conditional seems not used in Makefiles.

ryanofsky commented at 3:16 pm on May 6, 2020:

re: #18677 (review)

The BUILD_MULTIPROCESS conditional seems not used in Makefiles.

It’s used in #10102 to conditionally build a new libbitcoin_ipc.a library. This could be moved to #10102, but #10102 does not otherwise touch the configure script, and I’m trying to keep as much as possible of build changes in this PR so they can be understood as a unit, and don’t get lost in c++ changes in the other PR

in src/Makefile.am:636 in 4f359eb256 outdated

634-bitcoin_wallet_CXXFLAGS = $(AM_CXXFLAGS) $(PIE_FLAGS)
635-bitcoin_wallet_LDFLAGS = $(RELDFLAGS) $(AM_LDFLAGS) $(LIBTOOL_APP_LDFLAGS)
636+bitcoin_wallet_CPPFLAGS = $(bitcoin_bin_cppflags)
637+bitcoin_wallet_CXXFLAGS = $(bitcoin_bin_cxxflags)
638+bitcoin_wallet_LDFLAGS = $(bitcoin_bin_ldflags)
639+bitcoin_wallet_LDADD = $(LIBBITCOIN_WALLET_TOOL) $(bitcoin_bin_ldadd)

hebasto commented at 6:05 am on May 2, 2020:

With this PR bitcoin_wallet_LDADD contains $(LIBBITCOIN_SERVER) comparing to the master. Is this needed?

ryanofsky commented at 3:17 pm on May 6, 2020:

re: #18677 (review)

With this PR bitcoin_wallet_LDADD contains $(LIBBITCOIN_SERVER) comparing to the master. Is this needed?

Good catch, removed. It shouldn’t effect build output, but this way there are still link errors if wallet tries to access a node symbol

hebasto commented at 6:51 am on May 2, 2020: member

7c99f2c45e23ae1ce8c5b539d14109497ac4322c: Maybe add fallback to default value for the TEST_RUNNER_ENV variable in the 00_setup_env.sh script for consistency?

ryanofsky force-pushed on May 6, 2020

ryanofsky commented at 4:19 pm on May 6, 2020: member

Updated 474f4f1d02267c158001c5af0a558d24ef5f971d -> abf920e460b72f43daef1b006cb13c1f1ee2e256 (pr/ipc-build2.4 -> pr/ipc-build2.5, compare) with suggestions

Thanks for review!

re: #18677 (comment)

7c99f2c: Maybe add fallback to default value for the TEST_RUNNER_ENV variable in the 00_setup_env.sh script for consistency?

Added

DrahtBot added the label Needs rebase on May 7, 2020

fanquake referenced this in commit 56611b0e24 on May 7, 2020

ryanofsky force-pushed on May 7, 2020

ryanofsky commented at 1:25 pm on May 7, 2020: member

Rebased abf920e460b72f43daef1b006cb13c1f1ee2e256 -> e1a8c361c26ae3251b21f505f9538336fdb14155 (pr/ipc-build2.5 -> pr/ipc-build2.6, compare) due to conflict with #18899

DrahtBot removed the label Needs rebase on May 7, 2020

in configure.ac:1461 in e1a8c361c2 outdated

1457+  fi
1458+elif test "x$with_libmultiprocess" != xno; then
1459+  AC_MSG_ERROR([--with-libmultiprocess=$with_libmultiprocess value is not yes, auto, or no])
1460+fi
1461+AC_SUBST(LIBMULTIPROCESS_CFLAGS)
1462+AC_SUBST(LIBMULTIPROCESS_LIBS)

hebasto commented at 7:12 am on May 8, 2020:

In this PR these variables are not used in Makefiles.

But in #10102 one can see: https://github.com/bitcoin/bitcoin/blob/aa38570fa5f1ceee5883a8170da60206051e05fd/src/Makefile.am#L597 https://github.com/bitcoin/bitcoin/blob/aa38570fa5f1ceee5883a8170da60206051e05fd/src/Makefile.am#L649

Is it intended?

ryanofsky commented at 2:47 pm on May 8, 2020:

re: #18677 (review)

In this PR these variables are not used in Makefiles.

But in #10102 one can see:

Is it intended?

What’s unexpected here? LIBMULTIPROCESS_CFLAGS and LIBMULTIPROCESS_LIBS are set by the configure script and used in the makefile.

If the question is why are the variables set in this PR when they are not used until #10102, the answer is that I would like all the configure changes to be reviewed here as a complete unit so they make more sense, and so #10102 reviewers can just review c++ code without being forced to look at m4 macros in configure code.

If the question is why is the makefile not specifying LIBMULTIPROCESS_LIBS in this PR, the answer is I’d be happy to add it in this PR, but am a little concerned that if I add it, someone will come back 3 weeks from now asking me why it is added when it is not currently needed.

I do appreciate these reviews, but would also appreciate more context for the questions. If there is a concern about a bug, what kind of bug is it? If it’s a concern about clarity, is there something beyond these two lines of code that is confusing? If the question is for curiosity, is there something in particular about this which is interesting?

hebasto commented at 6:29 pm on May 8, 2020:

If it’s a concern about clarity, is there something beyond these two lines of code that is confusing?

Yes, it is about clarity. Thank you for your patient clarification regarding future changes.

ryanofsky commented at 6:43 pm on May 8, 2020:

Yes, it is about clarity. Thank you for your patient clarification regarding future changes.

Thank you for reviewing and testing and finding things to fix! I’d very much like to add comments to the build files to document things which are surprising. But I can’t do that without knowing which specific things are surprising, so the more context provided in review comments, the better!

hebasto commented at 7:15 am on May 8, 2020: member

Testing e1a8c361c26ae3251b21f505f9538336fdb14155 on Linux Mint 19.3 (x86_64):

 0$ make -C depends NO_QT=1 MULTIPROCESS=1
 1...
 2Extracting libmultiprocess...
 3/home/hebasto/GitHub/bitcoin/depends/sources/5741d750a04e644a03336090d8979c6d033e32c0.tar.gz: OK
 4Preprocessing libmultiprocess...
 5Configuring libmultiprocess...
 6-- The CXX compiler identification is GNU 7.5.0
 7-- Check for working CXX compiler: /usr/bin/g++
 8-- Check for working CXX compiler: /usr/bin/g++ -- works
 9-- Detecting CXX compiler ABI info
10-- Detecting CXX compiler ABI info - done
11-- Detecting CXX compile features
12-- Detecting CXX compile features - done
13-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1") 
14-- Checking for module 'capnp'
15--   Found capnp, version 0.7.0
16-- Checking for module 'capnp-rpc'
17--   Found capnp-rpc, version 0.7.0
18-- Checking for module 'capnp-json'
19--   Found capnp-json, version 0.7.0
20-- Checking for module 'kj'
21--   Found kj, version 0.7.0
22-- Checking for module 'kj-async'
23--   Found kj-async, version 0.7.0
24-- Checking for module 'kj-http'
25--   Found kj-http, version 0.7.0
26-- Checking for module 'kj-test'
27--   Found kj-test, version 0.7.0
28-- Looking for C++ include pthread.h
29-- Looking for C++ include pthread.h - not found
30CMake Error at /usr/share/cmake-3.10/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
31  Could NOT find Threads (missing: Threads_FOUND)
32Call Stack (most recent call first):
33  /usr/share/cmake-3.10/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
34  /usr/share/cmake-3.10/Modules/FindThreads.cmake:205 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
35  CMakeLists.txt:11 (find_package)
36
37
38-- Configuring incomplete, errors occurred!
39See also "/home/hebasto/GitHub/bitcoin/depends/work/build/x86_64-pc-linux-gnu/libmultiprocess/5741d750a04e644a03336090d8979c6d033e32c0-037d09711be/CMakeFiles/CMakeOutput.log".
40See also "/home/hebasto/GitHub/bitcoin/depends/work/build/x86_64-pc-linux-gnu/libmultiprocess/5741d750a04e644a03336090d8979c6d033e32c0-037d09711be/CMakeFiles/CMakeError.log".
41funcs.mk:258: recipe for target '/home/hebasto/GitHub/bitcoin/depends/work/build/x86_64-pc-linux-gnu/libmultiprocess/5741d750a04e644a03336090d8979c6d033e32c0-037d09711be/./.stamp_configured' failed
42make: *** [/home/hebasto/GitHub/bitcoin/depends/work/build/x86_64-pc-linux-gnu/libmultiprocess/5741d750a04e644a03336090d8979c6d033e32c0-037d09711be/./.stamp_configured] Error 1
43make: Leaving directory '/home/hebasto/GitHub/bitcoin/depends'

ryanofsky commented at 3:24 pm on May 8, 2020: member

re: #18677 (comment)

Testing e1a8c36 on Linux Mint 19.3 (x86_64):

Thank you for reporting this. I fixed the a similar problem on travis by installing the libc6-dev-amd64-cross package, and maybe there is a similar package for mint. Either way, I’m going to try to reproduce the problem and eliminate the need for that dependency (details in #18915). IMO, it does not need to block this PR, but I will try to fix it soon anyway.

Sjors commented at 7:48 pm on May 8, 2020: member

Concept ACK

Changes are simpler than the last time I reviewed #16367, but I didn’t study the current build code in much detail.

I was unable to build depends for e1a8c361c26ae3251b21f505f9538336fdb14155 on macOS 10.15.4,:

0make -C depends MULTIPROCESS=1
1Building libmultiprocess...
2[ 14%] Linking CXX static library libmultiprocess.a
3Error running link command: No such file or directory
4make[3]: *** [libmultiprocess.a] Error 2
5make[2]: *** [CMakeFiles/multiprocess.dir/all] Error 2
6make[1]: *** [all] Error 2
7make: *** [/Users/sjors/dev/bitcoin-ipc/depends/work/build/x86_64-apple-darwin19.4.0/libmultiprocess/5741d750a04e644a03336090d8979c6d033e32c0-332cf4b0b73/./.stamp_built] Error 2

Compiling without depends works fine (installed the latest master version of libmultiprocess).

Starting tests with BITCOIND breaks the backwards compatibility test: BITCOIND=bitcoin-node test/functional/feature_backwards_compatibility.py

0FileNotFoundError: [Errno 2] No such file or directory: '/Users/sjors/dev/bitcoin-ipc/releases/v0.19.0.1/bin/bitcoind'

ryanofsky commented at 8:08 pm on May 8, 2020: member

I was unable to build depends for e1a8c36 on macOS 10.15.4,:

Boy, everything is broken after the changes to support linux->mac cross compile. Googling I think I might need to manually pass the host_AR variable, or just entirely give up on passing cross-compile options when the builder platform is the same as the host platform.

Thanks for testing.

Starting tests with BITCOIND breaks the backwards compatibility test: BITCOIND=bitcoin-node test/functional/feature_backwards_compatibility.py
0FileNotFoundError: [Errno 2] No such file or directory: '/Users/sjors/dev/bitcoin-ipc/releases/v0.19.0.1/bin/bitcoind'

Will try to reproduce but if you happen to know more about the failure, it’d be helpful. Not clear why setting BITCOIND=bitcoin-node would cause bitcoin-ipc/releases/v0.19.0.1/bin/bitcoind not to exist. I wouldn’t expect the BITCOIND variable to affect contents of a download

hebasto commented at 6:49 am on May 9, 2020: member

re: #18677 (comment)

Testing e1a8c36 on Linux Mint 19.3 (x86_64):

Thank you for reporting this. I fixed the a similar problem on travis by installing the libc6-dev-amd64-cross package, and maybe there is a similar package for mint. Either way, I’m going to try to reproduce the problem and eliminate the need for that dependency (details in #18915). IMO, it does not need to block this PR, but I will try to fix it soon anyway.

Thanks, it works. Btw, Linux Mint is based on Ubuntu’s code base.

hebasto commented at 6:59 am on May 9, 2020: member

Testing e1a8c361c26ae3251b21f505f9538336fdb14155 on Linux Mint 19.3 (x86_64):

 0$ make -C depends NO_QT=1 MULTIPROCESS=1
 1$ CONFIG_SITE=$PWD/depends/x86_64-pc-linux-gnu/share/config.site ./configure
 2$ make check
 3$ BITCOIND=bitcoin-node test/functional/wallet_multiwallet.py
 42020-05-09T06:57:09.915000Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test_p9ahj4dt
 52020-05-09T06:57:12.472000Z TestFramework (ERROR): Assertion failed
 6Traceback (most recent call last):
 7  File "/home/hebasto/GitHub/bitcoin/test/functional/test_framework/test_node.py", line 475, in assert_start_raises_init_error
 8    self.wait_for_rpc_connection()
 9  File "/home/hebasto/GitHub/bitcoin/test/functional/test_framework/test_node.py", line 220, in wait_for_rpc_connection
10    'bitcoind exited with status {} during initialization'.format(self.process.returncode)))
11test_framework.test_node.FailedToStartError: [node 0] bitcoind exited with status -11 during initialization
12
13During handling of the above exception, another exception occurred:
14
15Traceback (most recent call last):
16  File "/home/hebasto/GitHub/bitcoin/test/functional/test_framework/test_framework.py", line 114, in main
17    self.run_test()
18  File "test/functional/wallet_multiwallet.py", line 95, in run_test
19    self.nodes[0].assert_start_raises_init_error(['-wallet=wallet.dat/bad'], exp_stderr, match=ErrorMatch.PARTIAL_REGEX)
20  File "/home/hebasto/GitHub/bitcoin/test/functional/test_framework/test_node.py", line 489, in assert_start_raises_init_error
21    'Expected message "{}" does not partially match stderr:\n"{}"'.format(expected_msg, stderr))
22  File "/home/hebasto/GitHub/bitcoin/test/functional/test_framework/test_node.py", line 161, in _raise_assertion_error
23    raise AssertionError(self._node_msg(msg))
24AssertionError: [node 0] Expected message "boost::filesystem::create_directory:" does not partially match stderr:
25""
262020-05-09T06:57:12.524000Z TestFramework (INFO): Stopping nodes
272020-05-09T06:57:12.525000Z TestFramework (WARNING): Not cleaning up dir /tmp/bitcoin_func_test_p9ahj4dt
282020-05-09T06:57:12.525000Z TestFramework (ERROR): Test failed. Test logging available at /tmp/bitcoin_func_test_p9ahj4dt/test_framework.log
292020-05-09T06:57:12.525000Z TestFramework (ERROR): Hint: Call /home/hebasto/GitHub/bitcoin/test/functional/combine_logs.py '/tmp/bitcoin_func_test_p9ahj4dt' to consolidate all logs

ryanofsky force-pushed on May 11, 2020

ryanofsky commented at 11:40 pm on May 11, 2020: member

Updated e1a8c361c26ae3251b21f505f9538336fdb14155 -> 952ac003aba664204979579e6f0363033a07cff2 (pr/ipc-build2.6 -> pr/ipc-build2.7, compare) building natively without sysroot when possible in light of #18915

I responded to all issues reported above but I couldn’t reproduce some problems. It’s difficult to follow the threads in this PR and know in advance which problems are related, but I want to suggest if anybody sees any problems or errors at all with this PR please just go to https://github.com/chaincodelabs/libmultiprocess/issues/new and make a new issue pasting the error and build commands, and please open different issues if there are different problems.

re: #18677#pullrequestreview-408292513

I fixed the a similar problem on travis by installing the libc6-dev-amd64-cross package, and maybe there is a similar package for mint

Now with the new update, it should no longer be necessary to install this package;

re: #18677 (comment)

I was unable to build depends for e1a8c36 on macOS 10.15.4,:

I think this should be fixed in the latest update which switches back to a native build.

re: #18677 (comment)

Starting tests with BITCOIND breaks the backwards compatibility test: BITCOIND=bitcoin-node test/functional/feature_backwards_compatibility.py

I tried to reproduce this, but didn’t couldn’t get the problem to happen. As mentioned in the earlier reply #18677 (comment), it’s difficult to see how changes from this PR or from setting the BITCOIND variable could cause v0.19.0.1/bin/bitcoind binary not to be found. My attempt to reproduce looked like:

0git checkout e1a8c361c26ae3251b21f505f9538336fdb14155
1./autogen.sh
2export PKG_CONFIG_PATH=/home/russ/src/libmultiprocess/prefix/lib/pkgconfig
3./configure --disable-ccache --enable-multiprocess
4make
5contrib/devtools/previous_release.sh -b v0.15.2 v0.16.3 v0.17.1 v0.18.1 v0.19.0.1
6BITCOIND=bitcoin-node test/functional/feature_backwards_compatibility.py

Would definitely encourage creating an issue so we can get to the bottom of this at https://github.com/chaincodelabs/libmultiprocess/issues/new if you still see this error.

re: #18677 (comment)

0test_framework.test_node.FailedToStartError: [node 0] bitcoind exited with status -11 during initialization

I tried to follow the same steps to reproduce but couldn’t get this issue to happen. There’s a chance this could be fixed in the latest update if there was some inconsitency in the depends sysroots causing this error. For for this error to happen at this point in the test is pretty mysterious given node would have had to start successfully before this. Would encourage creating an issue at https://github.com/chaincodelabs/libmultiprocess/issues/new if this error still happens or if you see any new errors from this PR. Here are steps I tried to reproduce this with:

0git checkout e1a8c361c26ae3251b21f505f9538336fdb14155
1make -C depends MULTIPROCESS=1 NO_QT=1 V=1
2./autogen.sh
3CONFIG_SITE=$PWD/depends/x86_64-pc-linux-gnu/share/config.site ./configure --disable-ccache
4make
5BITCOIND=bitcoin-node test/functional/wallet_multiwallet.py

in .travis.yml:130 in 952ac003ab outdated

147@@ -148,6 +148,11 @@ jobs:
148       env: >-
149         FILE_ENV="./ci/test/00_setup_env_native_fuzz.sh"
150 
151+    - stage: test

MarcoFalke commented at 12:22 pm on May 12, 2020:

not sure if this is useful to run on travis. This is an experimental feature (currently not even a feature). So the inconvenience in the unlikely case that it breaks is probably not worth having it run on every pull request. Please keep the ci config, though. If you want you can add it to .cirrus.ci here in the repo or people can run it manually on their machines.

ryanofsky commented at 1:01 pm on May 12, 2020:

re: #18677 (review)

not sure if this is useful to run on travis

What if I added an if: commit_message !~ /depends:|multiprocess:/ condition? This travis config has uncovered a few build and runtime bugs that weren’t happening for me locally, so I think it’s been pretty useful. The idea behind the regex would be not run on normal pull requests but catch regressions on depends and multiprocess pull requests.

MarcoFalke commented at 1:09 pm on May 12, 2020:

Absolutely, if you can get the condition to work. Travis is running the build on a temporary merge commit, so the commit message might not might not be the one you pushed.

DrahtBot added the label Needs rebase on May 12, 2020

DrahtBot commented at 1:34 pm on May 12, 2020: member

🐙 This pull request conflicts with the target branch and needs rebase.

build: multiprocess autotools changes

autoconf and automake changes to support multiprocess gui/node/wallet execution.

This adds a new --enable-multiprocess flag, and build configuration code to
detect libraries needed for multiprocess support. The --enable-multiprocess
flag builds new bitcoin-node and bitcoin-gui executables, which are updated in
https://github.com/bitcoin/bitcoin/pull/10102 to communicate across processes.
But for now they are functionally equivalent to existing bitcoind and
bitcoin-qt executables.

5d1377b52b

depends: add MULTIPROCESS depends option

Builds required packages and passes --enable-multiprocess option to bitcoin build

603fd6a2e7

multiprocess: add multiprocess travis configuration e2bab2aa16

sidhujag referenced this in commit d51641ba68 on May 12, 2020

ryanofsky force-pushed on May 13, 2020

ryanofsky commented at 0:31 am on May 13, 2020: member

Rebased 952ac003aba664204979579e6f0363033a07cff2 -> c4378e6bafcf1d749293519ab9dde10d8b6d2230 (pr/ipc-build2.7 -> pr/ipc-build2.8, compare) due to conflict with #18929. Also added condition to skip travis multiprocess depends build on non-depends, non-multiprocess PRs Updated c4378e6bafcf1d749293519ab9dde10d8b6d2230 -> e2bab2aa162ae38b2bf8195b577c982402fbee9d (pr/ipc-build2.8 -> pr/ipc-build2.9, compare) just updating commit messages to try to trigger travis build

fanquake removed the label Needs rebase on May 13, 2020

ryanofsky force-pushed on May 13, 2020

hebasto approved

hebasto commented at 6:00 am on May 16, 2020: member

ACK e2bab2aa162ae38b2bf8195b577c982402fbee9d, tested on Linux Mint 19.3 (x86_64):

0$ make -C depends NO_QT=1 MULTIPROCESS=1
1$ ./autogen.sh
2$ CONFIG_SITE=$PWD/depends/x86_64-pc-linux-gnu/share/config.site ./configure
3$ make check
4$ BITCOIND=bitcoin-node ./test/functional/test_runner.py

Also built depends without NO_QT=1 and tested ./src/bitcoin-gui :+1:

in depends/packages/native_libmultiprocess.mk:2 in 603fd6a2e7 outdated

0@@ -0,0 +1,18 @@
1+package=native_libmultiprocess
2+$(package)_version=5741d750a04e644a03336090d8979c6d033e32c0

Sjors commented at 11:21 am on May 19, 2020:

If you need to change anything else: maybe bump to 9f5b8355649aca94b4b529d89ba439aa3b2f9642 for consistency with building from master (but the new commits shouldn’t be relevant here)

in depends/packages/native_capnp.mk:7 in e2bab2aa16

0@@ -0,0 +1,18 @@
1+package=native_capnp
2+$(package)_version=0.7.0
3+$(package)_download_path=https://capnproto.org/
4+$(package)_download_file=capnproto-c++-$($(package)_version).tar.gz
5+$(package)_file_name=capnproto-cxx-$($(package)_version).tar.gz
6+$(package)_sha256_hash=c9a4c0bd88123064d483ab46ecee777f14d933359e23bff6fb4f4dbd28b4cd41
7+

Sjors commented at 12:14 pm on May 19, 2020:

As @fanquake suggested elsewhere, you could add this:

0define $(package)_set_vars
1	$(package)_config_opts=--without-openssl
2endef

Note that at least on macOS it won’t pick up my homebrew installed openssl@1.1: stable 1.1.1g even when I use --with-openssl, but I think it’s worth explicitly opting out anyway.

You mentioned:

But maybe requesting to be built without openssl when openssl isn’t available would be a good way to demonstrate not liking openssl

It’s also consistent with the practice in other dependencies to actively opt out of stuff (see e.g. #17730, #16370)

ryanofsky commented at 1:41 pm on May 19, 2020:

re: #18677 (review)

It’s also consistent with the practice in other dependencies to actively opt out of stuff (see e.g. #17730, #16370)

Thanks, I will make that change in a followup, or here if another update is required (reluctant to make changes now it seems like all the build problems so far have been ironed out).

At the time I made that comment about OpenSSL I mistakenly thought the depends build was more hermetic, and that it wouldn’t pick up system dependencies in host package builds. This was before I knew about #18915

Sjors approved

Sjors commented at 1:01 pm on May 19, 2020: member

tACK e2bab2aa162ae38b2bf8195b577c982402fbee9d on macOS 10.15.4

Depends builds now. Without depends still works fine too, using capnp 0.7.0 from homebrew, and the latest libmultiprocess self compiled. Ideally you should tag a libmultiprocess (pre-)release after this lands in master, or right before 0.21 branches off.

I’m no longer able to reproduce a problem with BITCOIND=bitcoin-node test/functional/feature_backwards_compatibility.py

Direct link to IRC meeting log for April 16th: http://www.erisian.com.au/meetbot/bitcoin-core-dev/2020/bitcoin-core-dev.2020-04-16-19.03.log.html#l-243

I thought we were going to pivot this to just add a -process=node/wallet/gui option to the same bitcoin-qt binary?

think the current naming is better, but I pointed out in the IRC meeting it is easy to change.

I think separate binaries is a better approach for now. For development I’d like to have them both, so I don’t have to reconfigure to see if a bug is related to the multiprocess stuff. I imagine at some point we’ll feel comfortable shipping the multi-process binaries, with their dependencies, but not comfortable enough to make them the default.

ryanofsky commented at 1:52 pm on May 19, 2020: member

I think it might be a good time to merge this PR. It seems to be at a point where it is working for the people who’ve tested it on different platforms. There are definitely improvements to be made going forward, but we know build changes are fragile and a fix for one bug tends to lead to other bugs, so it would be great to be done with this big PR when it’s in a pretty good state, and start to track future issues in smaller more focused PRs.

Sjors commented at 3:24 pm on May 19, 2020: member

Maybe a Gitian build to top it off?

01d9040e1d0e33a0aeebc2ee3a3be56b20df9fe985acc90be8d7d4aa7e3909356  bitcoin-3aec27edc9d4-aarch64-linux-gnu.tar.gz
13cb8c621421daf7de5cf02004a88eb462d0dcc855f0235cc40f01e65ea9dd115  bitcoin-3aec27edc9d4-win64-setup-unsigned.exe
2baed7424d10644a1ff7443801a1aaaa7326ad37fdf587a098950e05c0fbc7ad1  bitcoin-3aec27edc9d4-osx-unsigned.dmg

MarcoFalke added the label Needs Guix build on May 19, 2020

MarcoFalke added the label Needs gitian build on May 19, 2020

fanquake commented at 11:12 am on May 20, 2020: member

I think it might be a good time to merge this PR.

Thanks. I’m planning on merging this shortly.

practicalswift commented at 3:21 pm on May 20, 2020: contributor

ACK e2bab2aa162ae38b2bf8195b577c982402fbee9d

Thanks for doing this excellent architectural work: a dream come true (even if optional)!

With this dream off the table I think I only have Sandboxed API integration, control-flow integrity and killing the UUM bug class for good using -ftrivial-auto-var-init=zero left to dream about from a security perspective :)

Thanks @ryanofsky: you’re great, and so is your patience – this has been a very long road :)

DrahtBot commented at 4:55 am on May 21, 2020: member

Gitian builds

File	commit 0aa2ff0f660ec7fec532b5f33850219bad5104dd(master)	commit b8eebc5ebd395bde044b609c67ef59e002ac8505(master and this pull)
*-aarch64-linux-gnu-debug.tar.gz	`f4e97320d67e9f9d...`	`ae39b2293d95e03f...`
*-aarch64-linux-gnu.tar.gz	`708dd8c7998c93bb...`	`0f85de77dcb78b72...`
*-arm-linux-gnueabihf-debug.tar.gz	`8306690022f814f2...`	`67e97aae8e4cf93b...`
*-arm-linux-gnueabihf.tar.gz	`027f84bd9489b511...`	`044ea5779bc42b8d...`
*-osx-unsigned.dmg	`02a0b8f2ba11dfa4...`	`0e12b3b08e9a993f...`
*-osx64.tar.gz	`52e97780af471514...`	`3d3a0cb246ec2fce...`
*-riscv64-linux-gnu-debug.tar.gz	`cba762eed7d969a3...`	`2801f933cf057b6b...`
*-riscv64-linux-gnu.tar.gz	`2c732808ce42f68c...`	`04e4d5ab4627cc1f...`
*-win64-debug.zip	`f947a4d958558143...`	`40ed56707d48b50e...`
*-win64-setup-unsigned.exe	`cda69226dcaaa83c...`	`bd1b04bf38fce1d0...`
*-win64.zip	`b5f95d502e377110...`	`274344dd485b3896...`
*-x86_64-linux-gnu-debug.tar.gz	`9a26cbd755166033...`	`3d1b6e10e5228a55...`
*-x86_64-linux-gnu.tar.gz	`d2d737f786ee7a7b...`	`d1e526a4024cf79b...`
*.tar.gz	`d0b818b8368fb371...`	`57e4c4fb9899e323...`
bitcoin-core-linux-0.21-res.yml	`b6f91507a347a74f...`	`7cd851b20104fca5...`
bitcoin-core-osx-0.21-res.yml	`e2fe3f4cfb0c2efe...`	`30368228d2d82424...`
bitcoin-core-win-0.21-res.yml	`a587bff12db575a7...`	`9a1db49db80d364d...`
linux-build.log	`bc5a88fd2de120fe...`	`af81c60b53c76c59...`
osx-build.log	`d36c40220263504e...`	`9c0599a713ab0521...`
win-build.log	`d3d4007978e2e2b2...`	`3825b58385bba7d1...`
bitcoin-core-linux-0.21-res.yml.diff		`857dd69b4ceed69b...`
bitcoin-core-osx-0.21-res.yml.diff		`245076a96f4893ee...`
bitcoin-core-win-0.21-res.yml.diff		`62b1aeb2ec857d8d...`
linux-build.log.diff		`7fd8b3b45ed65662...`
osx-build.log.diff		`e3f930c3cc1a04cd...`
win-build.log.diff		`b7e5e7f383c24859...`

DrahtBot removed the label Needs gitian build on May 21, 2020

fanquake merged this on May 21, 2020

fanquake closed this on May 21, 2020

MarcoFalke commented at 1:23 pm on May 21, 2020: member

Checked that the ci config file is indeed native and runs on aarch64:

 0Options used to compile and link:
 1  multiprocess  = yes
 2  with wallet   = yes
 3  with gui / qt = yes
 4    with qr     = yes
 5  with zmq      = yes
 6  with test     = yes
 7    with fuzz   = no
 8  with bench    = yes
 9  with upnp     = yes
10  use asm       = yes
11  sanitizers    = 
12  debug enabled = no
13  gprof enabled = no
14  werror        = no
15
16  target os     = linux
17  build os      = 
18
19  CC            = /usr/bin/ccache gcc
20  CFLAGS        = -pipe -O2 
21  CPPFLAGS      =   -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -I/bitcoin-core/depends/aarch64-unknown-linux-gnu/share/../include/  -DHAVE_BUILD_INFO -D__STDC_FORMAT_MACROS
22  CXX           = /usr/bin/ccache g++ -std=c++11
23  CXXFLAGS      =   -fstack-reuse=none -Wstack-protector -fstack-protector-all     -pipe -O2 
24  LDFLAGS       = -pthread  -Wl,-z,relro -Wl,-z,now -pie  -L/bitcoin-core/depends/aarch64-unknown-linux-gnu/share/../lib 
25  ARFLAGS       = cr

sidhujag referenced this in commit 6a4320e7a9 on May 21, 2020

MarcoFalke removed the label Needs Guix build on May 31, 2020

ryanofsky added this to the "Done" column in a project

jasonbcox referenced this in commit ce5bf0a880 on Sep 23, 2020

janus referenced this in commit d9bbecfcf9 on Nov 8, 2020

maaku referenced this in commit 59aef72c88 on Jan 23, 2021

maaku referenced this in commit dd6d81a504 on Jan 23, 2021

furszy referenced this in commit 424ea0949e on Jun 30, 2021

UdjinM6 referenced this in commit 66d80b6a30 on Jul 19, 2021

kittywhiskers referenced this in commit a003178978 on Aug 27, 2021

DrahtBot locked this on Feb 15, 2022

Multiprocess build support #18677

Conflicts

Gitian builds