RFC: Multiprocess binaries and packaging options

ryanofsky commented at 7:06 pm on September 26, 2024: contributor

Issue for discussion about ways multiprocess functionality could be packaged and released. Trying to figure out which of the 3 release options described below makes the most sense.

Binaries

One goal of the multiprocess project has been to provide minimal binaries that only include node, wallet, or gui code, which spawn or connect to other processes as needed to provide other functionality.

The idea implemented in #10102 is to have 3 binaries:

bitcoin-node - contains node code and libraries (leveldb), and no wallet or gui code
bitcoin-wallet - contains wallet code and libraries (sqlite), and no node or gui code
bitcoin-gui - contains gui code and libraries (qt), and no node or wallet code

Which are built differently than current binaries:

bitcoind - contains node code+libraries (leveldb) and wallet code+libraries (sqlite)
bitcoin-qt - contains node code+libraries (leveldb), wallet code+libraries (sqlite), and gui code+libraries (qt)

Original release plan

My original idea implementing this was for there to be a separate multiprocess bitcoin release with the multiprocess binaries.

The main release would be unchanged, including existing bitcoind and bitcoin-qt binaries, but there would be a separate release containing bitcoin-node, bitcoin-wallet and bitcoin-gui binaries. The main release would be available in the normal place like bitcoincore.org/bin/bitcoin-core-27.1/, and the multiprocess release would be available alongside, at bitcoincore.org/bin/bitcoin-core-multiprocess-27.1/ or someplace similar. Then, after a release cycle or two, the multiprocess release could replace the main release if it was working well.

But this approach has some drawbacks, namely that it would be a burden for maintainers to create separate releases and make them accessible, and that it could be confusing for users to encounter two releases with different binaries. So it doesn’t seem like this approach is optimal (at least not for general users, maybe it would be ok for miners using the IPC mining interface).

Release options overview

Taking a step back, it seems like there are 3 possible options for releasing multiprocess functionality:

Side-release: create a separate multiprocess release for multiprocess binaries, as described above.
Side-binaries: add multiprocess binaries (bitcoin-node, bitcoin-wallet, bitcoin-gui) to existing releases alongside existing binaries (bitcoind and bitcoin-qt).
Combined binaries: add multiprocess functionality to existing bitcoind and bitcoin-qt binaries that can be enabled/disabled with runtime arguments.
Minimally combined binaries: add limited multiprocess functionality to existing bitcoind and bitcoin-qt binaries that can be enabled/disabled with runtime arguments. Difference from option 3 is that IPC side-binaries would also exist and bitcoind and bitcoin-qt would not contain IPC code, but automatically exec() IPC binaries when -ipcbind or other IPC options were enabled.

Release options comparison

From discussion in recent PR’s and offline, it seems like approach (3) might be favored, but it is worth considering pros and cons of the three approaches.

Side-release
- Pros:
  - Opt-in: Keeps original binaries unchanged for users who don’t want multiprocess functionality.
  - Minimal binaries: Node binary contains only node code not wallet code, gui binary contains no node or wallet code.
- Cons:
  - Extra work: It’s more work for maintainers and signers to create an extra set of packages and make them accessible.
  - Complexity: Posting a separate release could be confusing for users.
Side-binaries:
- Pros:
  - Opt-in and Minimal binaries: same as previous
  - Easy to implement: Implemented in #30975 and basically requires no changes other than flipping cmake options.
- Cons:
  - Confusing: This would probably be the most confusing approach for users. They would see one release with bitcoind, bitcoin-qt, bitcoin-node, bitcoin-wallet, bitcoin-gui binaries and not be able to easily understand how they are intended to be used.
- Potential mitigations:
  - We would probably put multiprocess binaries in another folder (like multiprocess/bin/ instead of bin/) so they are clearly labeled and aren’t confused with the default binaries.
  - We might want to rethink our approach to packaging binaries to begin with. We are already shipping a substantial collection of binaries: bitcoin-cli, bitcoind, bitcoin-qt, bitcoin-tx, bitcoin-util, bitcoin-wallet, and test_bitcoin which users are expected to call individually, and are probably already somewhat confusing. By contrast, git ships dozens of binaries and nobody is confused by them because they live in a libexec directory and are wrapped with single git command that finds and executes the right ones. Maybe taking git as inspiration we could have add a unified bitcoin command that looks like:
```
0bitcoin gui [OPTIONS]             # Start a gui
1bitcoin daemon [OPTIONS]          # Start a daemon
2bitcoin rpc [OPTIONS] COMMAND     # Call an rpc
3bitcoin wallet [OPTIONS] COMMAND  # Access a wallet
4bitcoin test [OPTIONS]            # Run tests
5bitcoin help
6# ... and more for bitcoin-util, bitcoin-tx ...
```
    Update: wrapper command is implemented in #31375
Combined binaries:
- Pros:
  - Compatibility: Users could download and invoke bitcoind and bitcoin-qt binaries as before, not know they support IPC, not use IPC unless it’s switched on by default, and there would be runtime options to turn it on or off.
- Cons:
  - Bigger binaries and more dependencies: Binaries will both contain node, wallet, and IPC code. There will be no non-IPC binaries users can deploy and run separately if they don’t use IPC, and no node binary users can deploy and run separately if they don’t use the wallet, or want to run the wallet and node on different hosts.
  - Requires code changes: Will require some thinking about how to add runtime options to #10102 to control whether bitcoind should spawn separate wallet process or run wallet code internally, and similarly whether bitcoin-gui should use a separate node process or run node code in the same process. Will also require updating the build system.
Minimally combined binaries:
- Pros:
  - Combines pros listed in option 1 with pros listed in option 3.
- Cons:
  - Some implementation messiness: A downside of this approach is options will need to be parsed twice. For example bitcoind will need to read arguments and config file to detect if any IPC features were requested, then it will need to exec() bitcoin-node which will read the same options and configuration again. In general this doesn’t seem as nice and clean as having a single wrapper binary like #31375 that calls all the other binaries, and can show help displaying what subcommands are available.

As mentioned, approach (3) seems favored so far, though personally I like approach (2), especially if we could have a unified bitcoin command.

Creating this issue to get feedback and document any decisions we make.

This issue is part of the process separation project.

Sjors commented at 9:15 am on September 27, 2024: member

I like the idea of a unified bitcoin command.

We could start with seperate binaries for multiprocess, e.g.:

bitcoin daemon (for bitcoind)
bitcoin daemon multiprocess for bitcoin-node
bitcoin gui (for bitcoin-qt)
bitcoin gui multiprocess (for bitcoin-gui)

And perhaps not worry about just including multiprocess in the utilities:

bitcoin wallet (for bitcoin-wallet)

Then later on the multiprocess option could be made default and we drop the non-multiprocess binary.

The existing bitcoind, bitcoin-qt and bitcoin-cli and binaries should probably stay where they are. Later on they can be changed to emit a deprecation warning “use bitcoin daemon instead”, and then eventually deleted.

hebasto commented at 11:37 am on September 27, 2024: member

I like the idea of a unified bitcoin command.

So do I.

Would it be beneficial for users if we started shipping DEB and RPM packages? The package manager would handle the proper installation of multiple executables.

maflcko commented at 11:55 am on September 27, 2024: member

Would it be beneficial for users if we started shipping DEB and RPM packages? The package manager would handle the proper installation of multiple executables.

IIRC there were some in the packagin repo, but they were both removed due to being unmaintained, see https://github.com/bitcoin-core/packaging?tab=readme-ov-file#bitcoin-core-packaging

ryanofsky commented at 12:48 pm on September 27, 2024: contributor

re: #30983 (comment)

We could start with seperate binaries for multiprocess, e.g.:

This makes sense. Initially I was thinking all the binaries had to live in the same place, but this is not true. We could leave all existing binaries where they are in bin/ and just add 3 new files to releases when multiprocess option is turned on:

bin/bitcoin
libexec/bitcoin-node
libexec/bitcoin-gui

Implementing this might be as simple as changing installation path for bitcoin-node and bitcoin-gui from BINDIR to LIBEXECDIR in cmake, and then writing the bitcoin wrapper as described above. The wrapper could just look for whatever binary it was trying to start in the libexec/ directory first and bin/ directory second.

On command lines you suggested, I think they make sense but might suggest:

bitcoin daemon as synonym for bitcoind
bitcoin gui as synonym for bitcoin-qt
bitcoin rpc as synonym for bitcoin-cli
bitcoin -m daemon as synonym for bitcoin-node
bitcoin -m gui as synonym for bitcoin-gui

etc

re: #30983 (comment)

On DEB and RPM packages, I’m sure they could be useful for their own reasons, but I don’t know if they offer things that would specifically help with this issue, and I think a bitcoin wrapper should be able to work well regardless of the release format.

Sjors commented at 12:56 pm on September 27, 2024: member

bitcoin -m daemon -debug=net as a format indeed makes sense. It makes it more clear which arguments are passed into bitcoin-node.

While you’re add it, -v could switch versions if we use libexec/version/ as the directory. Though outside the functional tests there’s probably no use for that, and it’s potentially dangerous to switch between versions.

MrSuddenJoy commented at 1:29 pm on September 27, 2024: none

@ryanofsky bin/bitcoin should (or could) be meta-process of all others that you mentioned. Just symlink maybe?

Or just use dickary dependencies.

ryanofsky commented at 3:44 pm on September 30, 2024: contributor

@ryanofsky bin/bitcoin should (or could) be meta-process of all others that you mentioned. Just symlink maybe?

It could be a symlink to existing executable, which could interpret its arguments as needed based on argv[0], instead of being a new executable. But I think having a new executable that calls exec https://man7.org/linux/man-pages/man3/exec.3.html would be more straightforward

willcl-ark commented at 9:03 am on October 1, 2024: member

I very much like the idea of the wrapper approach described in #30983 (comment), it feels to me like the correct way to structure and package an application consisting of multiple binaries rather than overloading existing binaries with even more options.

My only reservation with the approach is the potential magnitude of the effects downstream. If executed poorly this could break many scripts, programs and general infra. However, so long as we maintain the current binary names in the current directory, as currently proposed, and only introduce a new wrapper around them, then it seems like it should mitigate any downstream impact (completely).

ryanofsky referenced this in commit 65e49f4579 on Oct 1, 2024

ryanofsky commented at 8:29 pm on October 1, 2024: contributor

Here’s an initial implementation of a bitcoin wrapper executable: commit 65e49f4579ee823284bf04f1295b59116acc727b (branch)

This should be purely additive, and to willcl-ark’s point, not break backwards compatibility. It just lets us add new binaries to a libexec/ directory instead of bin/.

MrSuddenJoy commented at 10:49 am on October 2, 2024: none

@ryanofsky Not entirely sure this is way to go imho.

As long as everything works well ( = as expected ) its all ok, but as soon as things go south, problems (especially ones that are hard to solve/complex) start to accumulate/pile up. And thats exactly whats root cause of many bad things, namely frustration, burn-out and things like this.

Having meta-package has proven (to me) to be perfect cure for problems described above. If something breaks, its dead simple to: localise where (in code) fault is, fix it, and prevent it from happening again.

josibake commented at 1:15 pm on October 2, 2024: member

Echoing others, I like option 2 (side-binaries). A bitcoin <cmd> wrapper on top of multiple binaries also seems quite nice. While this might cause some confusion for end users, I think this will be far less confusing than having a side release (and much less work for maintainers and guix builders).

Regarding option 3, one of the value propositions of multiprocess in my mind is being able to run a binary for a specific purpose, with only the code needed for that purpose in the binary, e.g., running bitcoin-node on a server without wanting/needing a wallet or gui. Option 3 seems to undermine that, while also introducing complexity (as mentioned in the cons).

ryanofsky referenced this in commit 713ca1ea5e on Nov 26, 2024

ryanofsky referenced this in commit c34c719ecd on Nov 26, 2024

ryanofsky referenced this in commit e0c8158b28 on Nov 27, 2024

ryanofsky referenced this in commit b06c7ad0ae on Nov 27, 2024

ryanofsky referenced this in commit c68a225861 on Dec 3, 2024

ryanofsky commented at 8:57 pm on December 3, 2024: contributor

The bitcoin wrapper discussed above is added in PR #31375

ryanofsky referenced this in commit 255b946df9 on Dec 11, 2024

ryanofsky referenced this in commit 3ad329b3ac on Dec 11, 2024

Sjors referenced this in commit 7c305f51e6 on Dec 13, 2024

ryanofsky referenced this in commit f17453ff13 on Dec 13, 2024

ryanofsky referenced this in commit 79c9a8ffd6 on Dec 17, 2024

ryanofsky referenced this in commit ce3b0ca72d on Dec 17, 2024

ryanofsky referenced this in commit 2e5806a26b on Jan 18, 2025

ryanofsky referenced this in commit d7a87a4a55 on Jan 18, 2025

ryanofsky referenced this in commit 390bd18924 on Jan 19, 2025

ryanofsky referenced this in commit 56135b3e78 on Jan 20, 2025

ryanofsky referenced this in commit 689e44a32a on Jan 24, 2025

ryanofsky referenced this in commit a6b755d00e on Jan 24, 2025

ryanofsky referenced this in commit 2378952e0a on Jan 24, 2025

darosior commented at 10:08 pm on January 28, 2025: member

I don’t think the cons of option 3 are necessary to get its pros. How about simply keeping bitcoind and bitcoin-qt which would start -node, -wallet and -gui binaries from libexec? The bigger binaries are a consequence of wanting to be able to disable IPC through a runtime option. It’s an internal, why is it something we would want to make possible for the user to change? And why should we keep releasing the large binaries? If we do, when do we stop? I think that once we are confident of shipping multiprocess-enabled binaries we should just do it and not stay in a halfway situation.

ryanofsky commented at 4:50 pm on January 29, 2025: contributor

re: #30983 (comment)

I don’t think the cons of option 3 are necessary to get its pros. How about simply keeping bitcoind and bitcoin-qt which would start -node, -wallet and -gui binaries from libexec?

Nice idea! This is an interesting hybrid of approaches 2 & 3 and I’ll add it as a 4th option in the description.

Minimally combined binaries: add limited multiprocess functionality to existing bitcoind and bitcoin-qt binaries that can be enabled/disabled with runtime arguments. Difference from option 3 is that IPC side-binaries would also exist and bitcoind and bitcoin-qt would not contain IPC code, but automatically exec() IPC binaries them -ipcbind or other IPC options were enabled.
Minimally combined binaries:
- Pros:
  - Combines pros listed in option 1 with pros listed in option 3.
- Cons:
  - Some implementation messiness: A downside of this approach is options will need to be parsed twice. For example bitcoind will need to read arguments and config file to detect if any ipc features were requested, then it will need to exec bitcoin-node which will read the same options and configuration again. In general this doesn’t seem as nice and clean as having a single wrapper binary like #31375 that calls all the other binaries, and can show help displaying what subcommands are available.

The bigger binaries are a consequence of wanting to be able to disable IPC through a runtime option. It’s an internal, why is it something we would want to make possible for the user to change? And why should we keep releasing the large binaries? If we do, when do we stop? I think that once we are confident of shipping multiprocess-enabled binaries we should just do it and not stay in a halfway situation.

I think my brain might be a little fried, but I am not following this part. Maybe you can be more concrete and say what negative outcome(s) you would like to avoid?

darosior commented at 5:52 pm on January 29, 2025: member

Maybe you can be more concrete and say what negative outcome(s) you would like to avoid?

Sure, i meant i don’t see why we should release non-IPC binaries once we have decided to enable multiprocess. It seems it would make us be in this halfway situation of keeping around the bigger binaries with no clear reason nor timeline about when we’d get rid of them.

ryanofsky commented at 6:09 pm on January 29, 2025: contributor

Sure, i meant i don’t see why we should release non-IPC binaries once we have decided to enable multiprocess.

Got it. I honestly don’t have an opinion about that. I think existing non-ipc binaries should be kept as long as features like adding a Chain IPC interface (#29409) or removing wallet code from bitcoin-node process (#10102) are planned. But after that point you could easily drop them, especially if #31375 gained adoption. You could also remove them in phases, like (1) move them to libexec (2) change bitcoin command default to run multirprocess instead of monolithic binaries (3) remove monolithic binaries.

darosior commented at 6:14 pm on January 29, 2025: member

I agree the multiprocess project should be finished before we consider enabling it and shipping binaries. I guess that ties into the discussion in #31756 so i’ll expand there.

ryanofsky referenced this in commit c06f29e3a3 on Jan 30, 2025

ryanofsky referenced this in commit bde036879d on Jan 30, 2025

RFC: Multiprocess binaries and packaging options #30983

Binaries

Original release plan

Release options overview

Release options comparison