multiprocess: Add bitcoin wrapper executable #31375

Intended to make bitcoin command line features more discoverable and allow installing new multiprocess binaries in libexec/ instead of bin/ so they don’t cause confusion.

Idea and implementation of this were discussed in #30983.

Initial implementation of this feature is deliberately minimal so the UX can evolve in response to feedback and there are not too many details to debate and discuss in a single PR. But many improvements are possible or planned:

• Adding manpage and bash completions.
• Showing nicer error messages that detect if an executable isn’t installed and suggest how to fix (comment)
• Showing wrapper command lines in subcommand in help output (comment). This could be done conditionally as suggested in the comment or be unconditional.
• Showing wrapper command lines in subcommand error output. There is a bitcoin-cli error pointed out in (comment) that is needlessly confusing.
• Integrating help so bitcoin help subcommand invokes bitcoin subcommand -h. bitcoin -h subcommand should also be supported and be equivalent (comment)
• Adding support for bitcoin-util subcommands. Ideal interface would probably be more like bitcoin grind not bitcoin util grind but this has been punted for now. Supporting subcommands directly would require some ArgsManager modifications
• Adding a dedicated python functional test for the wrapper. Right now there is some CI coverage by setting the BITCOIN_CMD variable, but this doesn’t cover things like the help output and version output, and support for different directory layouts.
• Better --multiprocess (-m) / --monolithic (-M) default selection. Right now, default is monolithic but it probably makes sense to chose more intelligently depending on whether -ipc options are enabled and what binaries are available.
• Maybe parsing bitcoin.conf and supporting options to control wrapper behavior like custom locations or preferences or aliases.
• Better command command line usability. Allow combining short options like (-ah). Allow fuzzy matching of subcommands or suggestions if you misspell. (suggested by stickies in review club)
• Not directly related to this PR but bitcoin-cli named implementation used by the wrapper should do a better job disambiguating named arguments from base64 arguments ending in = as pointed out in (comment)

This PR is part of the process separation project. A review club meeting for it took place in https://bitcoincore.reviews/31375

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31375.

Reviews

See the guideline for information on the review process.

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

• #32297 (bitcoin-cli: Add -ipcconnect option by ryanofsky)
• #30437 (ipc: add bitcoin-mine test program by ryanofsky)
• #28690 (build: Introduce internal kernel library by TheCharlatan)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/33557375133

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

• Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

• A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

• An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

Concept ACK

I think it would be more clear to move build/src/bitcoin-{node,gui} to build/src/libexec, rather than use a different file organization for CMake builds than for installs.

The “Win64 native, VS 2022” job still seems unhappy.

The “Win64 native, VS 2022” job still seems unhappy.

https://learn.microsoft.com/en-us/cpp/error-messages/compiler-warnings/compiler-warning-level-3-c4996?view=msvc-170#posix-function-names:

To turn off deprecation warnings for these functions, define the preprocessor macro _CRT_NONSTDC_NO_WARNINGS. You can define this macro at the command line by including the option /D_CRT_NONSTDC_NO_WARNINGS.

We have already used this macro:https://github.com/bitcoin/bitcoin/blob/7590e93bc73b3bbac641f05d490fd5c984156b33/cmake/leveldb.cmake#L87-L89

Thanks for the reviews!

Updated da108a6e5be220654a65b6613ee7eb2c4ddc8677 -> 02567bf2bef5997ea5f0765780d196f36d3053e8 (pr/wrap.5 -> pr/wrap.6, compare) to fix windows build warning and making a change to avoid a potentially confusing behavior #31375 (review)

re: #31375#pullrequestreview-2467341532

I think it would be more clear to move build/src/bitcoin-{node,gui} to build/src/libexec, rather than use a different file organization for CMake builds than for installs.

Agree and I think #31161 should allow this to be simplified.

re: #31375 (comment)

We have already used this macro:

https://github.com/bitcoin/bitcoin/blob/7590e93bc73b3bbac641f05d490fd5c984156b33/cmake/leveldb.cmake#L87-L89

It seems like it would be better if this code could be compiled without disabling warnings, especially since if the old names are being deprecated. For now I just added a #define to switch to the recommended name. For leveldb it probably does make sense to disable the warnings to avoid needing to change the code too much.

For reference the CI failure is:

0                                   subprocess.CalledProcessError: Command '['C:\\hostedtoolcache\\windows\\Python\\3.12.7\\x64\\python.exe', 'D:/a/bitcoin/bitcoin\\contrib\\signet\\miner', "--cli='D:\\a\\bitcoin\\bitcoin\\build\\src\\Release\\bitcoin-cli.exe' -nonamed '-datadir=D:\\a\\_temp\\test_runner_₿_🏃_20241203_223434\\tool_signet_miner_211\\node0'", 'generate', '--address=tb1q2ndfasp67k5wp30fkt63tw9gf465lcjf7rm5fc', "--grind-cmd='D:\\a\\bitcoin\\bitcoin\\build\\src\\Release\\bitcoin-util.exe' grind", '--nbits=1d00ffff', '--set-block-time=1733265821', '--poolnum=99']' returned non-zero exit status 1.

Earlier it says:

0FileNotFoundError: [WinError 2] The system cannot find the file specified

I cherry-picked the first commit onto #30975 in order to try a (arm64-apple-darwin) Guix build: https://github.com/Sjors/bitcoin/tree/2024/12/multiprocess-guix

This also required adding it to installable targets and Maintenance.cmake (maybe not needed for this PR).

There was a brief discussion on IRC yesterday about whether anything special needs to happen in terms of macOS code signing. The conclusion was: probably not.

https://bitcoin-irc.chaincode.com/bitcoin-core-dev/2024-12-12#1074526

On my M4 I had to codesign -s - ... the binaries from my Guix build just like before, which suggests nothing special is required.

I wrote:

I think it would be more clear to move build/src/bitcoin-{node,gui} to build/src/libexec, rather than use a different file organization for CMake builds than for installs.

But I just realized that the bitcoin binary would be installed to /usr/bin whereas new binaries like bitcoin-node and bitcoin-mine would go to /usr/libexec.

To match this behavior in development mode we could use build/bin and build/libexec. As you mentioned #31161 should allow this. But there’s no need to wait for that.

I sometimes install a user-specific node using -DCMAKE_INSTALL_PREFIX=$HOME. By default $HOME/bin is in the PATH on Ubuntu, but $HOME/libexec isn’t. Looking at f17453ff13ebd0be842396c17bcc99b0c903abb8 it handles this nicely; if the bitcoin executable lives under bin it will try ../libexec.

Mostly happy with 365ac72dc7629765afb85bb9cc7c53d153c7c034.

I didn’t test and review Windows code. I also don’t really understand the test changes in a3b92c1f46c14c5c8db1f2833fd869d8a6a7ad4f, hopefully @maflcko does. But it’s useful that we use the wrapper in 365ac72dc7629765afb85bb9cc7c53d153c7c034.

Thanks for the review and testing, really helpful feedback!

Updated 365ac72dc7629765afb85bb9cc7c53d153c7c034 -> 128e110ed7af5f95d2bd16c36eca93c7c9b03573 (pr/wrap.9 -> pr/wrap.10, compare) fixing version and help formatting and weakly_canonical error handling. Updated 128e110ed7af5f95d2bd16c36eca93c7c9b03573 -> 044c1129db06983da598f427dff85513d8480b3a (pr/wrap.10 -> pr/wrap.11, compare) to fix lint failure https://cirrus-ci.com/task/5251623682834432

🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/34561550110

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

• Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

• A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

• An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

I guess bitcoin is intentionally not added to installable_targets because it makes reference to the multiprocess binaries that aren’t shipped?

I made a Windows guix build for 044c1129db06983da598f427dff85513d8480b3a. It got misidentified as Trojan:Script/Wavatac.B1ml again. But since the bitcoin binary isn’t included yet, it can’t be because of the execvp call.

Just to be sure, I also tried a guix build for master @ 62bd61de110b057cbfd6e31e4d0b727d93119c72 that this PR is based on. That didn’t trigger the anti-virus. But maybe it’s just random.

In order to better test on Windows I used this: 26d3a069f94e2cabcc8d1f73fd2cfa3e7c320497. The installer puts bitcoin.exe in C:\Program Files\Bitcoin\daemon. Running it without arguments will show the help. Starting the daemon fails with “Error: Command line contains unexpected token ‘Files\Bitcoin\daemon\bitcoind’” so it seems to trip over the space. It also can’t find bitcoin-qt.exe, which lives in Bitcoin\ instead of daemon\. If I move it to a directory without space, I’m able to start the daemon and I’m able to stop the daemon using bitcoin.exe rpc stop. If I move the bitcoin-qt executable I’m also able to start the GUI. No complaints from virus scanners this time.

Thanks for the reviews and testing!

Rebased 044c1129db06983da598f427dff85513d8480b3a -> 17f25149c680c1fdb2f79dbe373e8cf820e62488 (pr/wrap.11 -> pr/wrap.12, compare) to fix conflicts, also made many suggested updates and some fixes, particularly for windows. Updated 17f25149c680c1fdb2f79dbe373e8cf820e62488 -> 3d46df16e9cf4a960fedde9e515667030b39bb03 (pr/wrap.12 -> pr/wrap.13, compare) to fix python test bugs after bad rebase Updated 3d46df16e9cf4a960fedde9e515667030b39bb03 -> c11f7e7336b4a5ea6a8f22dc40dcae1b3ddd81e2 (pr/wrap.13 -> pr/wrap.14, compare) with documentation cleanups

re: #31375 (comment)

I guess bitcoin is intentionally not added to installable_targets because it makes reference to the multiprocess binaries that aren’t shipped?

No that was just an oversight, it should be added now. The wrapper tool is meant to be useful in developer builds and source builds not just official binaries releases, so it doesn’t need to be tied to current options we are building binary releases with.

Starting the daemon fails with “Error: Command line contains unexpected token ‘Files\Bitcoin\daemon\bitcoind’” so it seems to trip over the space. It also can’t find bitcoin-qt.exe, which lives in Bitcoin\ instead of daemon\. If I move it to a directory without space, I’m able to start the daemon and I’m able to stop the daemon using bitcoin.exe rpc stop. If I move the bitcoin-qt executable I’m also able to start the GUI. No complaints from virus scanners this time.

Thanks for all the windows testing, this is very helpful. I made attempted fixes for all these issues so we should be able to iterate and resolve them. I added your windows installer commit to this PR with a small tweak to put bitcoin.exe in program files directory instead of the daemon directory, then changed bitcoin.cpp to look for executables in the daemon directory and implemented command line argument escaping for execvp on windows since the windows execvp function doesn’t provide that

On Windows it’s a bit ugly to show the full path:

None of the commands work anymore though, except version and help. The commands daemon and gui do not start anything. When I start the GUI manually through the desktop item, the command bitcoin rpc help doesn’t return anything.

This also doesn’t return anything: .\bitcoin.exe daemon --help

I would expect trying to run the multiprocess versions to throw an error, but nothing happens when I do .\bitcoin.exe daemon -m.

Everything seems to be in place though:

 0PS C:\Program Files\Bitcoin> ls
 1
 2
 3    Directory: C:\Program Files\Bitcoin
 4
 5
 6Mode                 LastWriteTime         Length Name
 7----                 -------------         ------ ----
 8d-----         1/20/2025  11:10 AM                daemon
 9d-----         1/20/2025  11:10 AM                share
10-a----         1/20/2025  11:10 AM       39362576 bitcoin-qt.exe
11-a----         1/20/2025  11:10 AM            126 bitcoin.conf
12-a----         1/20/2025  11:10 AM        1221120 bitcoin.exe
13-a----         1/20/2025  11:10 AM           1142 COPYING.txt
14-a----         1/20/2025  11:10 AM            846 readme.txt
15-a----         1/20/2025  11:10 AM         175560 uninstall.exe
16
17ls daemon
18    Directory: C:\Program Files\Bitcoin\daemon
19
20Mode                 LastWriteTime         Length Name
21----                 -------------         ------ ----
22-a----         1/20/2025  11:10 AM        2249728 bitcoin-cli.exe
23-a----         1/20/2025  11:10 AM        4061184 bitcoin-tx.exe
24-a----         1/20/2025  11:10 AM        9435648 bitcoin-wallet.exe
25-a----         1/20/2025  11:10 AM       15302656 bitcoind.exe
26-a----         1/20/2025  11:10 AM       26467840 test_bitcoin.exe

Updated c11f7e7336b4a5ea6a8f22dc40dcae1b3ddd81e2 -> 0b503d792f0b6255f6dc5b0e4ee7ce2b7b667e90 (pr/wrap.14 -> pr/wrap.15, compare) with fixes for windows.

re: #31375 (comment)

None of the commands work anymore though, except version and help

Thanks, should be fixed now. After struggling with cross compilation and qemu a bit I was able to reproduce this. There was a bug in the ExecVp argv loop that caused string_view to be constructed with a null pointer and for the program to crash. I was a little surprised it would crash without an error, but maybe this is what happens by default if you deference a null pointer. There was also another bug in the same loop that caused arguments to be dropped. Both bugs should be fixed now and now it seems to work as expected and handle arguments with spaces correctly on windows.

On Windows it’s a bit ugly to show the full path:

I could reproduce this with powershell, but I was not seeing this when I tested with cmd.exe. So this behavior seems to vary with the shell, and it looks like this only happens with powershell, not with the classic windows shell or other shells like bash. For these other shells, I think it is probably better to show unmodified argv[0] in usage information instead of some modified or static alternative. But we could do something different in the future, like maybe show original argv[0] path in the Usage: string, and basename of that path in the command overview list.

After struggling with cross compilation and qemu a bit I was able to reproduce this.

I just have a dual boot system and configured sshd on it so I can quickly scp the windows installer from my guix machine.

it looks like this only happens with powershell, not with the classic windows shell

In that case I wouldn’t worry about it.

Just tested: 0b503d792f0b6255f6dc5b0e4ee7ce2b7b667e90 works!

tACK 0b503d792f0b6255f6dc5b0e4ee7ce2b7b667e90

Also tested on macOS.

The logic in ExecCommand() is easier to follow compared to my previous review.

I did not study the Windows ExecVp implementation in f006565ad220154c6e402c99074d326699d62d3e. I don’t expect anyone to use it until Windows multiprocess works (and is in a release). It would be good to call the wrapper from the Windows CI.

I tried Guix building this branch:

0HOSTS="x86_64-linux-gnu" ./contrib/guix/guix-build
1Consolidate compiler generated dependencies of target bitcoin-qt
2[100%] Built target bitcoin-qt
3Checking binary security...
4/distsrc-base/distsrc-0b503d792f0b-x86_64-linux-gnu/build/src/bitcoin: failed FORTIFY
5make[3]: *** [CMakeFiles/check-security.dir/build.make:71: CMakeFiles/check-security] Error 1
6make[2]: *** [CMakeFiles/Makefile2:565: CMakeFiles/check-security.dir/all] Error 2
7make[1]: *** [CMakeFiles/Makefile2:572: CMakeFiles/check-security.dir/rule] Error 2
8make: *** [Makefile:270: check-security] Error 2

I had only tried the Windows guix build. Getting the same error as @fanquake with x86_64-linux-gnu.

Can also be reproduced on Ubuntu 24.10 with:

0pip install lief==0.13.2
1contrib/devtools/security-check.py build/src/bitcoin

Adding this to security-check.py works, similar to bitcoin-util:

0    # bitcoin does not currently contain any fortified functions
1    if '--monolithic' in binary.strings:
2        return True

Not sure if it’s correct though.

re: #31375 (comment)

Adding this to security-check.py works, similar to bitcoin-util:

I think that makes sense. The new executable is only using c++ vector/string functions and calling execvp so it is probably just not calling any fortified functions, and won’t have any fortified function symbols, IIUC.

fortified functions

I have no idea what those look like.

Rebased 0b503d792f0b6255f6dc5b0e4ee7ce2b7b667e90 -> 171ffdd10e41df9e06ec3f6ef52f1cbbfcaf1748 (pr/wrap.15 -> pr/wrap.16, compare) to avoid conflict. Also added suggestions and security check fix. Additionally made some tweaks like using path::filename() method to avoid long paths #31375 (comment), and adding documentation for internal commands not shown by default. Updated 171ffdd10e41df9e06ec3f6ef52f1cbbfcaf1748 -> 5e0196c8e0d1687bcf59a9569bb8b7268d1f195d (pr/wrap.16 -> pr/wrap.17, compare) to fix windows compile error https://cirrus-ci.com/task/5239580141551616

fortified functions

I have no idea what those look like.

I know very little about this but I believe they just have a _chk suffix: https://www.gnu.org/software/libc/manual/html_node/Source-Fortification.html

🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/36101379316

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

• Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

• A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

• An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

I wonder if python test framework changes here are scaring reviewers from this PR? I potentially could drop the framework integration here and move it into a different PR. Maybe add just a more limited test here instead.

Would be good to know either way.

Approach ACK 0e4ee158ca.

I wonder if python test framework changes here are scaring reviewers from this PR?

Moving the python changes to another PR would probably speed the review of this one and probably delay the review of the other one (with the python changes).

Thanks for the reviews!

Rebased 0e4ee158cadc3eb8f6af1b33440ae9a95fe19487 -> edb8a72226b362ee60e4184fd8f56d1e588ce5b6 (pr/wrap.18 -> pr/wrap.19, compare) due to conflict with #31818

re: #31375#pullrequestreview-2612338457

Moving the python changes to another PR would probably speed the review of this one and probably delay the review of the other one (with the python changes).

Thanks, I guess for now it should be easy enough for reviewers not review the python changes if they aren’t interested and only ack the C++ ones.

re-ACK 6f0405177f89377ed272bae42becec1c75b8943c.

Lightly reviewed the code changes since my previous review. It’s nice that ExecCommand is now a bit shorter, as a result of moving some helper functionality to src/util/exec.h.

The Python changes are also easier to follow now thanks to splitting the commit.

I made a guix build for 6f0405, but didn’t try including the IPC binaries with #31802 since there’s too many build system changes there. That will be easier to test after #31741 lands.

Retested a bit on Apple Silicon macOS and on Windows.

Thanks for the review!

Updated 185c6a416df679ddc7b38750aa4d181c0b043d5b -> 623c0f0df878968dabbaa271a7654c638603f369 (pr/wrap.22 -> pr/wrap.23, compare) cleaning up comments and help with suggestions

Couldn’t much of this be done by using a just file? I saw a PR a while that tried to add something about a justfile which is a way to wrap command to make them easier and more discoverable.

Yes it could also be done as a shell script or python script. If we could count on a justfile interpreter, or a portable shell, or python being installed on all platforms which bitcoin runs on, the changes in the two commits here a47cf680619cae5892d72355f56f5c5051fd5434 and cbc0bdd73ed0fd672307cf0f80bf5f22f5321ccd would be simpler, though other 5 commits would not really be changed.

re-ACK a9935d4dae8847d0549ec51ddc4349c7e8c61763

Briefly tested another guix build on Windows.

re-ACK 418236c106e32abd7357551d309f8e6d1e494f72

Same as my last review, but the two commits shared with #31866 are merged.

Also tested (ad hoc signed) guix builds on macOS (M4 with 15.3.2, Intel with 13.7.4).

There was some discussion about this in the IRC meeting today: https://achow101.com/ircmeetings/2025/bitcoin-core-dev.2025-04-10_16_00.log.html

One thing that came up was whether to update documentation to suggest using bitcoin daemon instead of bitcoind and bitcoin gui instead of bitcoin-qt. Seems like the most important places in doc/ to update are the README.md and README_windows.md files, but also build-{unix,osd,windows,etc}.md files, multiprocess.md, tor.md and zmq.md files.

I think it would be better to update documentation in a separate PR since existing command lines are still correct and I think it might be good for the wrapper to get a little more testing before fully recommending it everywhere. So I think I will open a separate draft PR to update the documentation, but can also do it here if there is a preference for changing the documentation at the same time.

I’m thinking it might also be good to add an “Installed Files” section to https://github.com/bitcoin/bitcoin/blob/master/doc/files.md to describe files and directories installed by cmake and included in binary releases.

Curious for :+1::-1: if this would be useful, or just a maintenance burden. Right now it seems like we don’t have documentation describing binaries & installed files, unless I missed it somewhere. It also seems like ideally the information in files.md would be in a manpage, but that would be a separate issue.

ACK 155977ae8b4280100577455d34afa4d26ce89c58

I think it would be better to update documentation in a separate PR since existing command lines are still correct and I think it might be good for the wrapper to get a little more testing before fully recommending it everywhere

:+1:

I’m thinking it might also be good to add an “Installed Files” section to https://github.com/bitcoin/bitcoin/blob/master/doc/files.md to describe files and directories installed by cmake and included in binary releases.

Added this section in #31679 now, looks like: https://github.com/ryanofsky/bitcoin/blob/pr/libexec/doc/files.md#installed-files

Am still planning to make a separate PR to mention the wrapper executable in other documentation, and bitcoin daemon / bitcoin gui / bitcoin rpc as alternatives to calling bitcoind / bitcoin-qt / bitcoin-cli directly.

re-ACK 155977ae8b4280100577455d34afa4d26ce89c58

Since my last review, this adds bitcoin chainstate. I didn’t retest.

Thanks for the reviews!

Added 2 commits 155977ae8b4280100577455d34afa4d26ce89c58 -> 86eedd3c1243d4fdbe1c0f44920340ccacbac13e (pr/wrap.26 -> pr/wrap.27, compare) just adding documentation and release notes

re: #31375 (comment)

So I think I will open a separate draft PR to update the documentation

Changed my mind about this and decided just to update documentation here recommending bitcoin tool as an alternative to other command lines. If it gains use and testing and maybe has some convenience features added like options parsing and aliases, it probably will make sense to recommend it more widely, but these changes should make it easier to get started with.