This means that the “util library” will be compiled twice?
Yes, as it stands currently at least these few files of the util library will be compiled twice.
Maybe it could make sense to split libutil out from libcommon and make libkernel depend on libutil?
Unfortunately, there are plenty of util/*.cpp that libbitcoinkernel doesn’t use, and I wish to prevent any futher coupling in that regard especially since the files under util/*.cpp are largely unrelated.
Maybe it could make sense to remove consensus stuff from libnode and instead make libnode depend on libkernel?
Unfortunately bitcoin-chainstate or libbitcoinkernel requires quite a lot of .cpp files that libnode doesn’t, and also vice versa (both ends of the venn diagram are non-empty). So there’s no clear hierarchy that we can establish without linking in a large number of unnecessary .cpp files which I want to prevent.
The following invocation (in a shell that supports process substitution) will print all the .cpp that bitcoin-chainstate/libbitcoinkernel requires but libnode doesn’t:
0comm -23 <(make --no-print-directory -C src print-bitcoin_chainstate_SOURCES | cut -d= -f2 | tr ' ' '\n' | grep -v '\.h$' | sort -u) \
1 <(make --no-print-directory -C src print-libbitcoin_node_a_SOURCES | cut -d= -f2 | tr ' ' '\n' | grep -v '\.h$' | sort -u)
0arith_uint256.cpp
1bitcoin-chainstate.cpp
2chainparamsbase.cpp
3chainparams.cpp
4clientversion.cpp
5coins.cpp
6compat/glibcxx_sanity.cpp
7compressor.cpp
8consensus/merkle.cpp
9consensus/tx_check.cpp
10core_read.cpp
11deploymentinfo.cpp
12fs.cpp
13hash.cpp
14init/common.cpp
15key.cpp
16logging.cpp
17netaddress.cpp
18policy/feerate.cpp
19policy/policy.cpp
20primitives/block.cpp
21primitives/transaction.cpp
22pubkey.cpp
23random.cpp
24randomenv.cpp
25scheduler.cpp
26script/interpreter.cpp
27script/script.cpp
28script/script_error.cpp
29script/standard.cpp
30support/cleanse.cpp
31support/lockedpool.cpp
32sync.cpp
33threadinterrupt.cpp
34uint256.cpp
35util/asmap.cpp
36util/bytevectorhash.cpp
37util/getuniquepath.cpp
38util/hasher.cpp
39util/moneystr.cpp
40util/rbf.cpp
41util/serfloat.cpp
42util/settings.cpp
43util/strencodings.cpp
44util/syscall_sandbox.cpp
45util/system.cpp
46util/thread.cpp
47util/threadnames.cpp
48util/time.cpp
49util/tokenpipe.cpp
50warnings.cpp
All the .cpp that libnode requires but bitcoin-chainstate/libbitcoinkernel doesn’t:
0comm -13 <(make --no-print-directory -C src print-bitcoin_chainstate_SOURCES | cut -d= -f2 | tr ' ' '\n' | grep -v '\.h$' | sort -u) \
1 <(make --no-print-directory -C src print-libbitcoin_node_a_SOURCES | cut -d= -f2 | tr ' ' '\n' | grep -v '\.h$' | sort -u)
0addrdb.cpp
1addrman.cpp
2banman.cpp
3blockencodings.cpp
4httprpc.cpp
5httpserver.cpp
6i2p.cpp
7index/txindex.cpp
8init.cpp
9mapport.cpp
10net.cpp
11net_processing.cpp
12node/caches.cpp
13node/coin.cpp
14node/context.cpp
15node/interfaces.cpp
16node/miner.cpp
17node/minisketchwrapper.cpp
18node/psbt.cpp
19node/transaction.cpp
20noui.cpp
21rest.cpp
22rpc/blockchain.cpp
23rpc/mining.cpp
24rpc/misc.cpp
25rpc/net.cpp
26rpc/rawtransaction.cpp
27rpc/server.cpp
28rpc/server_util.cpp
29torcontrol.cpp
30txorphanage.cpp
31txrequest.cpp
32wallet/init.cpp
However, if we perform a similar analysis on libbitcoinconsensus.la we’ll see that it compiles the crypto/*.cpp files, which can be eliminated by https://github.com/bitcoin/bitcoin/pull/24322/commits/7c2b41e9f81e52cb30efc2d90ebb4b414b97617f, after that change there’s just script/bitcoinconsensus.cpp left.
Potential followup PR: Extract out a libbitcoinconensus_internal.la which doesn’t include script/bitcoinconsensus.cpp that libbitcoinkernel can link against and avoid unnecessary compilation.
0comm -13 <(make --no-print-directory -C src print-bitcoin_chainstate_SOURCES | cut -d= -f2 | tr ' ' '\n' | grep -v '\.h$' | sort -u) \
1 <(make --no-print-directory -C src print-libbitcoinconsensus_la_SOURCES | cut -d= -f2 | tr ' ' '\n' | grep -v '\.h$' | sort -u)
0crypto/aes.cpp
1crypto/chacha20.cpp
2crypto/chacha_poly_aead.cpp
3crypto/hkdf_sha256_32.cpp
4crypto/hmac_sha256.cpp
5crypto/hmac_sha512.cpp
6crypto/muhash.cpp
7crypto/poly1305.cpp
8crypto/ripemd160.cpp
9crypto/sha1.cpp
10crypto/sha256.cpp
11crypto/sha256_sse4.cpp
12crypto/sha3.cpp
13crypto/sha512.cpp
14crypto/siphash.cpp
15script/bitcoinconsensus.cpp
Appendix
For reference, here’s the .cpp comparison for libcommon:
All the .cpp that bitcoin-chainstate/libbitcoinkernel requires but libcommon doesn’t:
0comm -23 <(make --no-print-directory -C src print-bitcoin_chainstate_SOURCES | cut -d= -f2 | tr ' ' '\n' | grep -v '\.h$' | sort -u) \
1 <(make --no-print-directory -C src print-libbitcoin_common_a_SOURCES | cut -d= -f2 | tr ' ' '\n' | grep -v '\.h$' | sort -u)
0arith_uint256.cpp
1bitcoin-chainstate.cpp
2blockfilter.cpp
3chain.cpp
4chainparamsbase.cpp
5clientversion.cpp
6compat/glibcxx_sanity.cpp
7consensus/merkle.cpp
8consensus/tx_check.cpp
9consensus/tx_verify.cpp
10dbwrapper.cpp
11deploymentstatus.cpp
12flatfile.cpp
13fs.cpp
14hash.cpp
15index/base.cpp
16index/blockfilterindex.cpp
17index/coinstatsindex.cpp
18logging.cpp
19node/blockstorage.cpp
20node/chainstate.cpp
21node/coinstats.cpp
22node/ui_interface.cpp
23policy/fees.cpp
24policy/packages.cpp
25policy/rbf.cpp
26policy/settings.cpp
27pow.cpp
28primitives/block.cpp
29primitives/transaction.cpp
30pubkey.cpp
31random.cpp
32randomenv.cpp
33script/interpreter.cpp
34script/script.cpp
35script/script_error.cpp
36script/sigcache.cpp
37shutdown.cpp
38signet.cpp
39support/cleanse.cpp
40support/lockedpool.cpp
41sync.cpp
42threadinterrupt.cpp
43timedata.cpp
44txdb.cpp
45txmempool.cpp
46uint256.cpp
47util/asmap.cpp
48util/bytevectorhash.cpp
49util/getuniquepath.cpp
50util/hasher.cpp
51util/moneystr.cpp
52util/rbf.cpp
53util/serfloat.cpp
54util/settings.cpp
55util/strencodings.cpp
56util/syscall_sandbox.cpp
57util/system.cpp
58util/thread.cpp
59util/threadnames.cpp
60util/time.cpp
61util/tokenpipe.cpp
62validation.cpp
63validationinterface.cpp
64versionbits.cpp
All the .cpp that libcommon requires but bitcoin-chainstate/libbitcoinkernel doesn’t:
0comm -13 <(make --no-print-directory -C src print-bitcoin_chainstate_SOURCES | cut -d= -f2 | tr ' ' '\n' | grep -v '\.h$' | sort -u) \
1 <(make --no-print-directory -C src print-libbitcoin_common_a_SOURCES | cut -d= -f2 | tr ' ' '\n' | grep -v '\.h$' | sort -u)
0base58.cpp
1bech32.cpp
2common/bloom.cpp
3core_write.cpp
4external_signer.cpp
5key_io.cpp
6merkleblock.cpp
7netbase.cpp
8net_permissions.cpp
9net_types.cpp
10outputtype.cpp
11protocol.cpp
12psbt.cpp
13rpc/external_signer.cpp
14rpc/rawtransaction_util.cpp
15rpc/util.cpp
16script/descriptor.cpp
17script/sign.cpp
18script/signingprovider.cpp
