cmake: Move internal binaries from bin/ to libexec/

ryanofsky commented at 5:01 pm on January 17, 2025: contributor

This moves binaries not intended to be called directly by users from the bin/ directory to the libexec/ directory in cmake installs and binary releases. After this change, the binaries remaining in bin/ are:

bitcoin
bitcoin-cli
bitcoind
bitcoin-qt
bitcoin-tx
bitcoin-util
bitcoin-wallet

And the binaries that are moved to libexec/ are:

bench_bitcoin
bitcoin-chainstate
bitcoin-gui
bitcoin-node
test_bitcoin
test_bitcoin-qt

Removing these esoteric and not generally useful binaries from bin/ and from the system PATH should make it easier for typical users to find and identify the binaries which are actually useful.

Advanced users can still run all the binaries in libexec/ and the bitcoin wrapper executable introduced in #31375 is also able to locate binaries regardless of whether they are installed in bin/ or libexec/.

This PR is part of the process separation project.

DrahtBot commented at 5:01 pm on January 17, 2025: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31679.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	Sjors, ismaelsadeeq
Concept ACK	TheCharlatan, jonatack, hodlinator

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#31349 (ci: detect outbound internet traffic generated while running tests by vasild)
#30437 (ipc: add bitcoin-mine test program by ryanofsky)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

LLM Linter (✨ experimental)

Possible typos and grammar issues:

noticable -> noticeable [correct spelling]

drahtbot_id_4_m

DrahtBot added the label Build system on Jan 17, 2025

DrahtBot added the label CI failed on Jan 17, 2025

DrahtBot commented at 7:52 am on January 18, 2025: contributor

CI failed with

[17:52:38.033] /ci_container_base/ci/test/03_test_script.sh: line 154: /ci_container_base/ci/scratch/out/bin/test_bitcoin: No such file or directory

luke-jr commented at 3:25 pm on January 23, 2025: member

libexec is usually used for binaries run by programs rather than people. I don’t think most of the ones you’re moving fit that, and should remain in bin.

Sjors commented at 7:46 am on January 24, 2025: member

Run by people is a higher bar than “not typically useful”, but could make sense.

If there are no binaries for which it currently makes sense to put them in libexec, since they’re all called by people, then it seems this PR has be based on #31375.

ryanofsky commented at 5:57 pm on January 28, 2025: contributor

The idea that an executable placed in libexec/ should be called by other executables is a good rule of thumb but not a requirement of some kind. (And if it were a requirement, it would be satisfied by #31375.)

The technical difference between binaries installed to bin/ and binaries installed to libexec/ is that binaries in bin/ get exposed on the PATH while binaries in libexec/ don’t.

There is nothing wrong with binaries in libexec/ being documented and called used by users and packagers and other tools and scripts. Not every executable that a user could conceivably call needs to be present in PATH.

Packages like postfix have manpages for internal executables installed in libexec/ (anvil, bounce, smtpd, etc). Git subcommands can be called manually from libexec and have manpages of their own, too. I would say postfix and git are pretty much archetypes of packages designed around unix principles and there is nothing novel about installing low-level executables in libexec/.

With this PR, two sets of binaries are installed to libexec/ if they have been enabled in cmake:

multiprocess binaries bitcoin-node and bitcoin-gui
testing binaries bench_bitcoin test_bitcoin test_bitcoin-qt.

I can’t think of a reason why multiprocess binaries should be installed on PATH, but if someone wants to make that case I’m ready to hear it. As an alternative to this PR, they could be installed in bin/ with different names like bitoin-node-multiprocess or to a completely new folder.

For the test and bench binaries I just think it’s confusing and not useful to install these on the PATH. I think this is true today and will be especially true after #31375.

ryanofsky force-pushed on Jan 30, 2025

ryanofsky commented at 3:05 am on January 30, 2025: contributor

Updated 2a952710661a1990ed1c24b0ebd16de8ac0df87c -> 47a872236e070814ad74922d3f8a653e1c6af968 (pr/libexec.1 -> pr/libexec.2, compare) to fix CI error

DrahtBot removed the label CI failed on Jan 30, 2025

DrahtBot added the label Needs rebase on Feb 12, 2025

ryanofsky force-pushed on Feb 13, 2025

ryanofsky commented at 6:48 pm on February 13, 2025: contributor

Rebased 47a872236e070814ad74922d3f8a653e1c6af968 -> d2ceb2e0735a2c8343f8316b55fac55323aba62c (pr/libexec.2 -> pr/libexec.3, compare) due to conflict with #31834

DrahtBot removed the label Needs rebase on Feb 13, 2025

DrahtBot added the label Needs rebase on Feb 14, 2025

ryanofsky force-pushed on Feb 14, 2025

ryanofsky commented at 3:19 pm on February 14, 2025: contributor

Rebased d2ceb2e0735a2c8343f8316b55fac55323aba62c -> 45bfd97ec7c9991f41673d79b01277bfd940e64e (pr/libexec.3 -> pr/libexec.4, compare) due to conflict with #31844

DrahtBot removed the label Needs rebase on Feb 14, 2025

TheCharlatan commented at 7:14 pm on April 8, 2025: contributor

Concept ACK

jonatack commented at 4:37 am on April 9, 2025: member

Concept ACK

Would need a release note?

ryanofsky commented at 1:34 pm on April 14, 2025: contributor

Would need a release note?

Thanks! Added release note and files.md documention in new commits.

Added 2 commits 45bfd97ec7c9991f41673d79b01277bfd940e64e -> bda7626e987ce3d7287f72e6989dedde2c4ab666 (pr/libexec.4 -> pr/libexec.5, compare) with release notes and documentation

TheCharlatan commented at 9:16 pm on April 15, 2025: contributor

I think some installation packagers run test_bitcoin as part of their installation recipes, or at least I found one such example: https://github.com/epiccurious/bitcoin-core-node-builder/pull/164/files. @hebasto also polled usage of test_bitcoin on social media: https://x.com/hhebasto/status/1729879307768738070. Clearly there aren’t many people doing this, but are we ok with breaking the workflows of the few people that are? You mention that:

For the test and bench binaries I just think it’s confusing and not useful to install these on the PATH. I think this is true today and will be especially true after #31375.

What is confusing about keeping them where they are for now? Maybe there is some utility in testing your platform before actually running bitcoin on it.

DrahtBot added the label Needs rebase on May 7, 2025

cmake: Move internal binaries from bin/ to libexec/

This moves binaries not intended to be called directly by users from the `bin/`
directory to the `libexec/` directory in cmake installs and binary releases.
After this change, the binaries remaining in `bin/` are:

- bitcoin
- bitcoin-cli
- bitcoind
- bitcoin-qt
- bitcoin-tx
- bitcoin-util
- bitcoin-wallet

And the binaries that are moved to `libexec/` are:

- bench_bitcoin
- bitcoin-chainstate
- bitcoin-gui
- bitcoin-node
- test_bitcoin
- test_bitcoin-qt

Removing these esoteric and not generally useful binaries from `bin/` and from
the system PATH should make it easier for typical users to find and identify the
binaries which are actually useful.

Advanced users can still run all the binaries in `libexec/` and the `bitcoin`
wrapper executable introduced in #31375 is also able to locate binaries
regardless of whether they are installed in `bin/` or `libexec/`.

eded935be7

doc: Add release notes describing libexec/ binaries 5e89320949

doc: Add description of installed files to files.md d253fa9ebe

ryanofsky force-pushed on May 29, 2025

ryanofsky commented at 9:45 pm on May 29, 2025: contributor

Rebased bda7626e987ce3d7287f72e6989dedde2c4ab666 -> d253fa9ebe4305118417a02aa72cc06810662ac6 (pr/libexec.5 -> pr/libexec.6, compare) due to conflict with #28710

DrahtBot removed the label Needs rebase on May 29, 2025

Sjors commented at 5:49 am on May 30, 2025: member

Just #28710, no conflict with #31375?

Concept ACK

ismaelsadeeq commented at 11:17 am on May 30, 2025: member

Concept ACK

Sjors commented at 4:04 pm on May 30, 2025: member

tACK d253fa9ebe4305118417a02aa72cc06810662ac6

I tested on top of af227a0e7d9754623d36925c56599f9b921f5cb3 from #31802 on macOS 15.5 and 13.7.6, as well as Windows 11 (not impacted).

The bitcoin test command is perhaps a bit too well hidden. Maybe it’s better to drop the help command and put a bit more emphasis on the --all option:

0Commands:
1  gui [ARGS]     Start GUI, equivalent to running 'bitcoin-qt [ARGS]' or 'bitcoin-gui [ARGS]'
2  ...
3  tx ...
4
5Use 'bitcoin --help --all' for additional commands, e.g. for running tests.

That last “e.g.” might address @TheCharlatan’s concern:

Maybe there is some utility in testing your platform before actually running bitcoin on it.

DrahtBot requested review from TheCharlatan on May 30, 2025

DrahtBot requested review from jonatack on May 30, 2025

DrahtBot requested review from ismaelsadeeq on May 30, 2025

in doc/release-notes-31679.md:8 in d253fa9ebe

0@@ -0,0 +1,11 @@
1+Install changes
2+---------------
3+
4+The `test_bitcoin`, `test_bitcoin-qt`, `bench_bitcoin`, `bitcoin-chainstate`,
5+`bitcoin-node`, and `bitcoin-wallet` binaries are now installed in
6+`$PREFIX/libexec` rather than `$PREFIX/bin`. If you are using a binary release
7+or building from source with default build options, most of these programs are
8+not built by default anyway so these changes may not be noticable.

maflcko commented at 7:25 am on June 2, 2025:

noticable -> noticeable [correct spelling]

in doc/files.md:161 in d253fa9ebe

157+| share/rpcauth/                      | Documentation and scripts for RPC authentication setup |
158+
159+### Notes
160+
161+- *Italicized* files are installed in source builds but not included in binary releases.
162+- README and bitcoin.conf files included in binary releases but not installed in source builds.

hodlinator commented at 2:44 pm on June 3, 2025:

nit:

0- README and bitcoin.conf files are included in binary releases but not installed in source builds.

in src/CMakeLists.txt:472 in d253fa9ebe

468@@ -469,6 +469,7 @@ if(BUILD_UTIL_CHAINSTATE)
469       core_interface
470       bitcoinkernel
471   )
472+  install_binary_component(bitcoin-chainstate INTERNAL)

hodlinator commented at 7:52 pm on June 4, 2025:

Do we want to introduce installation of this binary? Should be called out in PR description and/or commit message if kept.

Sjors commented at 8:30 am on June 5, 2025:

I think it’s fine, but indeed should be in the description.

TheCharlatan commented at 8:45 am on June 5, 2025:

If so, it should also get a note that its command line interface is not expected to be stable.

hodlinator commented at 7:59 pm on June 4, 2025: contributor

Concept ACK d253fa9ebe4305118417a02aa72cc06810662ac6

Concept

Would be nice (but not critical) to get this done in the same release where we introduce the now merged wrapper, bitcoin(.exe).

Tested

 0₿ cmake --install build --prefix $PWD/install |sort
 1-- Install configuration: "RelWithDebInfo"
 2-- Installing: /home/hodlinator/bc/b2/install/bin/bitcoin
 3-- Installing: /home/hodlinator/bc/b2/install/bin/bitcoin-cli
 4-- Installing: /home/hodlinator/bc/b2/install/bin/bitcoind
 5-- Installing: /home/hodlinator/bc/b2/install/bin/bitcoin-qt
 6-- Installing: /home/hodlinator/bc/b2/install/bin/bitcoin-tx
 7-- Installing: /home/hodlinator/bc/b2/install/bin/bitcoin-util
 8-- Installing: /home/hodlinator/bc/b2/install/bin/bitcoin-wallet
 9-- Installing: /home/hodlinator/bc/b2/install/lib64/libbitcoinkernel.so
10-- Installing: /home/hodlinator/bc/b2/install/lib64/pkgconfig/libbitcoinkernel.pc
11-- Installing: /home/hodlinator/bc/b2/install/libexec/bench_bitcoin
12-- Installing: /home/hodlinator/bc/b2/install/libexec/bitcoin-chainstate
13-- Installing: /home/hodlinator/bc/b2/install/libexec/bitcoin-gui
14-- Installing: /home/hodlinator/bc/b2/install/libexec/bitcoin-node
15-- Installing: /home/hodlinator/bc/b2/install/libexec/test_bitcoin
16-- Installing: /home/hodlinator/bc/b2/install/libexec/test_bitcoin-qt
17-- Installing: /home/hodlinator/bc/b2/install/share/man/man1/bitcoin-cli.1
18-- Installing: /home/hodlinator/bc/b2/install/share/man/man1/bitcoind.1
19-- Installing: /home/hodlinator/bc/b2/install/share/man/man1/bitcoin-qt.1
20-- Installing: /home/hodlinator/bc/b2/install/share/man/man1/bitcoin-tx.1
21-- Installing: /home/hodlinator/bc/b2/install/share/man/man1/bitcoin-util.1
22-- Installing: /home/hodlinator/bc/b2/install/share/man/man1/bitcoin-wallet.1
23-- Set non-toolchain portion of runtime path of "/home/hodlinator/bc/b2/install/libexec/bitcoin-chainstate" to ""

0./install/bin/bitcoin node (bin)
1./install/bin/bitcoin test (libexec)

DrahtBot added the label CI failed on Jun 15, 2025

DrahtBot removed the label CI failed on Jun 16, 2025

ismaelsadeeq commented at 2:40 pm on June 16, 2025: member

Tested ACK d253fa9ebe4305118417a02aa72cc06810662ac6

By building with bench in a release build

0-- Installing: /Users/abubakarismail/Desktop/Work/bitcoin-dev/bitcoin/install/bin/bitcoin
1-- Installing: /Users/abubakarismail/Desktop/Work/bitcoin-dev/bitcoin/install/bin/bitcoin-cli
2-- Installing: /Users/abubakarismail/Desktop/Work/bitcoin-dev/bitcoin/install/bin/bitcoin-tx
3-- Installing: /Users/abubakarismail/Desktop/Work/bitcoin-dev/bitcoin/install/bin/bitcoin-util
4-- Installing: /Users/abubakarismail/Desktop/Work/bitcoin-dev/bitcoin/install/bin/bitcoin-wallet
5-- Installing: /Users/abubakarismail/Desktop/Work/bitcoin-dev/bitcoin/install/bin/bitcoind
6-- Installing: /Users/abubakarismail/Desktop/Work/bitcoin-dev/bitcoin/install/libexec/bench_bitcoin
7-- Installing: /Users/abubakarismail/Desktop/Work/bitcoin-dev/bitcoin/install/libexec/test_bitcoin

DrahtBot requested review from hodlinator on Jun 16, 2025

cmake: Move internal binaries from bin/ to libexec/ #31679

Code Coverage & Benchmarks

Reviews

Conflicts

LLM Linter (✨ experimental)

Concept

Tested