refactor: Remove unused includes

practicalswift commented at 3:04 pm on May 31, 2019: contributor

Make reasoning about dependencies easier by not including unused dependencies.

Please note that the removed headers are not “transitively included” by other still included headers. Thus the removals are real.

As an added bonus this change means less work for the preprocessor/compiler. At least 51 393 lines of code no longer needs to be processed:

0$ git diff -u HEAD~1 | grep -E '^\-#include ' | cut -f2 -d"<" | cut -f1 -d">" | \
1      sed 's%^%src/%g' | xargs cat | wc -l
251393

Note that 51 393 is the lower bound: the real number is likely much higher when taking into account transitively included headers :-)

fanquake added the label Refactoring on May 31, 2019

DrahtBot commented at 3:49 pm on May 31, 2019: member

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#16083 ([WIP] transaction fees in getblock by FelixWeis)
#16069 (test: move-only: Split large tests into smaller compile units by MarcoFalke)
#15987 (Wallet, GUI: Warn when sending to already-used Bitcoin addresses (also RPC: include such information in getaddressinfo) by luke-jr)
#15810 ([WIP] Remove nAbsurdFee fee from AcceptToMemoryPool by jnewbery)
#15382 ([util] add runCommandParseJSON by Sjors)
#14912 ([WIP] External signer support (e.g. hardware wallet) by Sjors)
#14384 (Resolve validationinterface circular dependencies by 251Labs)
#13949 (Introduce MempoolObserver interface to break “policy/fees -> txmempool -> policy/fees” circular dependency by Empact)
#13751 (Utils and libraries: Drops the boost/algorithm/string/split.hpp dependency by 251Labs)
#13062 (Make script interpreter independent from storage type CScript by sipa)
#10443 (Add fee_est tool for debugging fee estimation code by ryanofsky)
#9152 (Wallet/RPC: sweepprivkeys method to scan UTXO set and send to local wallet by luke-jr)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

MarcoFalke commented at 4:05 pm on May 31, 2019: member

Concept ACK, but is this complete? It seems that for example wallet.cpp should not depend on net.h and txmempool.h (txs are submitted via the chain interface).

dongcarl commented at 5:55 pm on May 31, 2019: member

@practicalswift Are you finding these manually or is there a static analysis tool that you use?

practicalswift force-pushed on Jun 1, 2019

practicalswift renamed this:
~~Make reasoning about dependencies easier by not including unused dependencies~~
Remove seven circular dependencies. Make reasoning about dependencies easier by not including unused dependencies.
on Jun 2, 2019

practicalswift commented at 8:29 am on June 2, 2019: contributor

@MarcoFalke Oh, good catch! My analysis script was a bit too conservative which made it miss the mentioned removal opportunities. I re-ran the analysis and removed quite a few more unused includes. Thanks for noticing! The removals should be complete now: let me know if you find any completeness counterexamples (false negatives) or removals that look incorrect (false positives) in the updated version! @dongcarl It’s a combination of my own #include analysis script, lots of test compilations/test suite runs and manual review of the candidate removals :-)

MarcoFalke renamed this:
~~Remove seven circular dependencies. Make reasoning about dependencies easier by not including unused dependencies.~~
refactor: Remove unused includes
on Jun 2, 2019

practicalswift force-pushed on Jun 2, 2019

Make reasoning about dependencies easier by not including unused dependencies eca9767673

practicalswift force-pushed on Jun 2, 2019

promag commented at 9:31 pm on June 2, 2019: member

I think this is great even if incomplete (considering we are in the begin of 0.19). Say goodbye to some fake circular dependencies.

utACK eca9767.

fanquake commented at 1:01 pm on June 3, 2019: member

utACK https://github.com/bitcoin/bitcoin/pull/16129/commits/eca97676737174ac6ff775c60e3798c69fb67f4a

If there’s a way to automate away this checking (which I think @dongcarl is alluding too), that’d be great.

I probably see a slight speedup during compiling. A few iterations of make clean; time make -j6

master (c7cfd20a77ce57d200b3b9e5e0dfb0d63818abdc)

0real	0m20.960s
1real	0m20.761s
2real	0m20.580s
3real	0m21.375s
4real	0m20.287s

This PR (https://github.com/bitcoin/bitcoin/commit/eca97676737174ac6ff775c60e3798c69fb67f4a):

0real	0m20.914s
1real	0m20.643s
2real	0m20.321s
3real	0m20.049s
4real	0m19.920s

MarcoFalke added the label Needs gitian build on Jun 5, 2019

MarcoFalke commented at 8:59 am on June 5, 2019: member

@fanquake It seems you are using ccache?

in src/core_read.cpp:6 in eca9767673 outdated

1@@ -2,16 +2,13 @@
2 // Distributed under the MIT software license, see the accompanying
3 // file COPYING or http://www.opensource.org/licenses/mit-license.php.
4 
5-#include <core_io.h>
6-

MarcoFalke commented at 9:06 am on June 5, 2019:

We always include the corresponding header file into the cpp file, because they are one module. For some reason core_io is the header file for both core_read and core_write…

practicalswift commented at 6:04 am on June 6, 2019:

Oh, yes of course! Thanks for noticing. The unexpected file naming fooled me. Now fixed by adding a commit – didn’t squash in order to not invalidate the many utACK:s. Let me know if you prefer a squash anyway.

MarcoFalke commented at 9:16 am on June 5, 2019: member

utACK eca97676737174ac6ff775c60e3798c69fb67f4a

Good to see some heavy unused includes such as net and validation go from wallet et al.

Signature:

 0-----BEGIN PGP SIGNED MESSAGE-----
 1Hash: SHA512
 2
 3utACK eca97676737174ac6ff775c60e3798c69fb67f4a
 4
 5Good to see some heavy unused includes such as net and validation go from wallet et al.
 6-----BEGIN PGP SIGNATURE-----
 7
 8iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
 9pUjDHwv+Lr0cxXrLCdXoWTbCZqE+LNvbCr9skT123fqixJAynCioxlhRmRTG8h5H
10grrJN6NcBx+7j62FdsD7i63gN43NB8v39E76n7srDx3PJhMjAnMY7eE3lSNahB8r
11M883ppvD11FIDBHFt99lBSRjLssS0GvJkcQOPCDujwhQkTAEk3RIJlRZSkBJuW6e
12SNFLazqgGwijf7ctnIWXjyEBXL/k2joz01iNi3ANrz+iIwlV13bRnxUbDromYIZo
13GS3yTi/0q18ai5B3cvcniR331DE1XNv/9P937RXxkxvnItwPVkC2bOm46MHnW0jH
14N/45QgVyVSzlOHlNoJ71cufllTnaUXYibfZq8vicTu9CUOqHyE3wnS8iW43KCx3z
15rSmav/NNqTuUAlCL0Xcb/DpsSLPy9CW3oPUZqNF4G6s0LTfltNAHhtziQBhPeeje
16aOxy+NUQzV7aa6O/O+6WT1zK8cdJYY7cdt8mjUjtTIYEyhARD2N2zen0HSlmbpiP
17C3KdAu2x
18=JgzO
19-----END PGP SIGNATURE-----

Timestamp of file with hash 8038fe8e9c52a24ec081a494a241e5a2fe735738b384a4c2d3eb1250a80ba885 -

MarcoFalke commented at 9:39 am on June 5, 2019: member

Unless there are objections, this will be merged within the next week or so.

sdaftuar commented at 9:51 am on June 5, 2019: member

Unless there are objections, this will be merged within the next week or so.

Not an objection, but I do think that in general we should consider whether any of the conflicting PRs should have priority over pure refactor PRs like this. (I don’t have an opinion myself, as I have not reviewed any of the PRs that conflict to see how important they are or how close they are to being merged.)

MarcoFalke commented at 9:56 am on June 5, 2019: member

I did check them and they are all WIP, test-only or refactoring themselves.

Include core_io.h from core_read.cpp 67f4e9c522

MarcoFalke commented at 2:31 pm on June 6, 2019: member

File name	RSS (kB)
addrdb.cpp	339432
addrman.cpp	308212
arith_uint256.cpp	72260
banman.cpp	268376
base58.cpp	96548
bech32.cpp	69608
blockencodings.cpp	485452
blockfilter.cpp	179224
bloom.cpp	163444
chain.cpp	185280
chainparamsbase.cpp	235344
chainparams.cpp	417012
clientversion.cpp	87868
coins.cpp	137128
compat/glibc_sanity.cpp	25712
compat/glibcxx_sanity.cpp	68404
compat/strnlen.cpp	17112
compressor.cpp	138056
consensus/merkle.cpp	101224
consensus/tx_check.cpp	119508
consensus/tx_verify.cpp	208472
core_read.cpp	413892
core_write.cpp	375424
crypto/aes.cpp	36584
crypto/chacha20.cpp	32984
crypto/hkdf_sha256_32.cpp	47640
crypto/hmac_sha256.cpp	48616
crypto/hmac_sha512.cpp	27264
crypto/poly1305.cpp	31088
crypto/ripemd160.cpp	66860
crypto/sha1.cpp	48756
crypto/sha256_avx2.cpp	370964
crypto/sha256.cpp	129324
crypto/sha256_shani.cpp	118748
crypto/sha256_sse41.cpp	380744
crypto/sha256_sse4.cpp	25328
crypto/sha512.cpp	57012
crypto/siphash.cpp	58372
dbwrapper.cpp	302036
flatfile.cpp	255008
fs.cpp	155368
hash.cpp	81216
httprpc.cpp	419268
httpserver.cpp	344028
index/base.cpp	336248
index/blockfilterindex.cpp	361868
index/txindex.cpp	471132
init.cpp	924328
interfaces/chain.cpp	520804
interfaces/handler.cpp	215672
interfaces/node.cpp	619612
interfaces/wallet.cpp	845216
key.cpp	122092
key_io.cpp	228028
keystore.cpp	441392
leveldb/db/builder.cc	80432
leveldb/db/c.cc	69276
leveldb/db/dbformat.cc	62416
leveldb/db/db_impl.cc	176980
leveldb/db/db_iter.cc	82104
leveldb/db/dumpfile.cc	96772
leveldb/db/filename.cc	69704
leveldb/db/log_reader.cc	62776
leveldb/db/log_writer.cc	60488
leveldb/db/memtable.cc	68064
leveldb/db/repair.cc	140544
leveldb/db/table_cache.cc	63348
leveldb/db/version_edit.cc	100996
leveldb/db/version_set.cc	176228
leveldb/db/write_batch.cc	64000
leveldb/helpers/memenv/memenv.cc	104628
leveldb/port/port_posix.cc	55404
leveldb/port/port_posix_sse.cc	58788
leveldb/table/block_builder.cc	72256
leveldb/table/block.cc	74708
leveldb/table/filter_block.cc	72000
leveldb/table/format.cc	62508
leveldb/table/iterator.cc	49556
leveldb/table/merger.cc	54980
leveldb/table/table_builder.cc	71380
leveldb/table/table.cc	67104
leveldb/table/two_level_iterator.cc	66664
leveldb/util/arena.cc	62708
leveldb/util/bloom.cc	50956
leveldb/util/cache.cc	61448
leveldb/util/coding.cc	57616
leveldb/util/comparator.cc	60948
leveldb/util/crc32c.cc	57148
leveldb/util/env.cc	56380
leveldb/util/env_posix.cc	110268
leveldb/util/filter_policy.cc	46772
leveldb/util/hash.cc	55120
leveldb/util/histogram.cc	64004
leveldb/util/logging.cc	61076
leveldb/util/options.cc	50752
leveldb/util/status.cc	57752
lib/univalue.cpp	2712
lib/univalue_get.cpp	2644
lib/univalue_read.cpp	2692
lib/univalue_write.cpp	2700
logging.cpp	188492
merkleblock.cpp	123228
miner.cpp	545048
netaddress.cpp	131020
netbase.cpp	303412
net.cpp	641752
net_processing.cpp	903684
node/coin.cpp	359000
node/psbt.cpp	229924
node/transaction.cpp	501876
noui.cpp	323392
outputtype.cpp	323616
policy/feerate.cpp	96472
policy/fees.cpp	521436
policy/policy.cpp	281476
policy/rbf.cpp	330760
policy/settings.cpp	126552
pow.cpp	173568
primitives/block.cpp	118776
primitives/transaction.cpp	151108
protocol.cpp	252616
psbt.cpp	323128
-pthread	992596
pubkey.cpp	86164
random.cpp	210588
rest.cpp	553628
rpc/blockchain.cpp	847204
rpc/client.cpp	280928
rpc/mining.cpp	676556
rpc/misc.cpp	509676
rpc/net.cpp	470636
rpc/protocol.cpp	279744
rpc/rawtransaction.cpp	840376
rpc/rawtransaction_util.cpp	449240
rpc/server.cpp	538228
rpc/util.cpp	441840
scheduler.cpp	363208
script/bitcoinconsensus.cpp	145580
script/descriptor.cpp	488564
script/interpreter.cpp	210904
script/ismine.cpp	318740
script/script.cpp	100584
script/script_error.cpp	24064
script/sigcache.cpp	418992
script/sign.cpp	307888
script/standard.cpp	333964
shutdown.cpp	33240
src/secp256k1.c	15296
support/cleanse.cpp	25672
support/lockedpool.cpp	124324
sync.cpp	162008
test/addrman_tests.cpp	652328
test/allocator_tests.cpp	553304
test/amount_tests.cpp	551544
test/arith_uint256_tests.cpp	692628
test/base32_tests.cpp	547868
test/base58_tests.cpp	537788
test/base64_tests.cpp	547844
test/bech32_tests.cpp	536752
test/bip32_tests.cpp	559484
test/blockchain_tests.cpp	539524
test/blockencodings_tests.cpp	647392
test/blockfilter_index_tests.cpp	644636
test/blockfilter_tests.cpp	595292
test/bloom_tests.cpp	672124
test/bswap_tests.cpp	548016
test/checkqueue_tests.cpp	668592
test/coins_tests.cpp	684284
test/compress_tests.cpp	538112
test/crypto_tests.cpp	641656
test/cuckoocache_tests.cpp	569404
test/dbwrapper_tests.cpp	571232
test/denialofservice_tests.cpp	677568
test/descriptor_tests.cpp	686608
test/flatfile_tests.cpp	554992
test/fs_tests.cpp	537424
test/getarg_tests.cpp	635464
test/hash_tests.cpp	554424
test/key_io_tests.cpp	596568
test/key_tests.cpp	597832
test/limitedmap_tests.cpp	536476
test/main.cpp	142840
test/mempool_tests.cpp	648792
test/merkleblock_tests.cpp	543556
test/merkle_tests.cpp	536332
test/miner_tests.cpp	683808
test/multisig_tests.cpp	587852
test/netbase_tests.cpp	624632
test/net_tests.cpp	632696
test/pmt_tests.cpp	559956
test/policyestimator_tests.cpp	563212
test/pow_tests.cpp	560040
test/prevector_tests.cpp	546660
test/raii_event_tests.cpp	546828
test/random_tests.cpp	584428
test/reverselock_tests.cpp	536732
test/rpc_tests.cpp	808252
test/sanity_tests.cpp	547536
test/scheduler_tests.cpp	562272
test/scriptnum_tests.cpp	539224
test/script_p2sh_tests.cpp	644396
test/script_standard_tests.cpp	728936
test/script_tests.cpp	1422668
test/serialize_tests.cpp	651156
test/setup_common.cpp	625408
test/sighash_tests.cpp	598748
test/sigopcount_tests.cpp	589084
test/skiplist_tests.cpp	559568
test/streams_tests.cpp	566088
test/sync_tests.cpp	536888
test/timedata_tests.cpp	538180
test/torcontrol_tests.cpp	585648
test/transaction_tests.cpp	834816
test/txindex_tests.cpp	574132
test/txvalidationcache_tests.cpp	639960
test/txvalidation_tests.cpp	546328
test/uint256_tests.cpp	593688
test/util_tests.cpp	897316
test/util_threadnames_tests.cpp	536448
test/validation_block_tests.cpp	594424
test/validation_tests.cpp	618512
test/versionbits_tests.cpp	575776
threadinterrupt.cpp	79676
timedata.cpp	276120
torcontrol.cpp	625500
txdb.cpp	485644
txmempool.cpp	653824
ui_interface.cpp	634304
uint256.cpp	66424
util/bip32.cpp	90048
util/bytevectorhash.cpp	54548
util/error.cpp	229948
util/fees.cpp	120932
util/moneystr.cpp	108296
util/rbf.cpp	90536
util/strencodings.cpp	106212
util/strencodings.cpp	2532
util/system.cpp	335012
util/threadnames.cpp	73768
util/time.cpp	446380
util/url.cpp	51316
util/validation.cpp	113864
validation.cpp	971296
validationinterface.cpp	851868
versionbits.cpp	192660
versionbitsinfo.cpp	54724
wallet/coincontrol.cpp	479884
wallet/coinselection.cpp	329532
wallet/crypter.cpp	451652
wallet/db.cpp	511924
wallet/feebumper.cpp	612368
wallet/fees.cpp	485752
wallet/init.cpp	637808
wallet/load.cpp	560320
wallet/psbtwallet.cpp	475836
wallet/rpcdump.cpp	861276
wallet/rpcwallet.cpp	1332508
wallet/test/coinselector_tests.cpp	876888
wallet/test/db_tests.cpp	564976
wallet/test/init_test_fixture.cpp	545212
wallet/test/init_tests.cpp	674808
wallet/test/psbt_wallet_tests.cpp	855056
wallet/test/wallet_crypto_tests.cpp	585340
wallet/test/wallet_test_fixture.cpp	686740
wallet/test/wallet_tests.cpp	875692
wallet/wallet.cpp	1284744
wallet/walletdb.cpp	764024
wallet/walletutil.cpp	264060
warnings.cpp	224448

File name	RSS (kB)
addrdb.cpp	339432
addrman.cpp	300868
arith_uint256.cpp	62044
banman.cpp	268376
base58.cpp	96548
bech32.cpp	69608
blockencodings.cpp	485452
blockfilter.cpp	179224
bloom.cpp	163444
chain.cpp	185280
chainparamsbase.cpp	235344
chainparams.cpp	417012
clientversion.cpp	87612
coins.cpp	137128
compat/glibc_sanity.cpp	25712
compat/glibcxx_sanity.cpp	68404
compat/strnlen.cpp	17112
compressor.cpp	138056
consensus/merkle.cpp	101216
consensus/tx_check.cpp	119508
consensus/tx_verify.cpp	208472
core_read.cpp	331212
core_write.cpp	375372
crypto/aes.cpp	36304
crypto/chacha20.cpp	32984
crypto/hkdf_sha256_32.cpp	47640
crypto/hmac_sha256.cpp	48616
crypto/hmac_sha512.cpp	27264
crypto/poly1305.cpp	31088
crypto/ripemd160.cpp	66860
crypto/sha1.cpp	48756
crypto/sha256_avx2.cpp	350716
crypto/sha256.cpp	129324
crypto/sha256_shani.cpp	118748
crypto/sha256_sse41.cpp	360612
crypto/sha256_sse4.cpp	25328
crypto/sha512.cpp	57012
crypto/siphash.cpp	58372
dbwrapper.cpp	302036
flatfile.cpp	255008
fs.cpp	155368
hash.cpp	81216
httprpc.cpp	419240
httpserver.cpp	344028
index/base.cpp	336248
index/blockfilterindex.cpp	361868
index/txindex.cpp	471132
init.cpp	924352
interfaces/chain.cpp	520804
interfaces/handler.cpp	215672
interfaces/node.cpp	533112
interfaces/wallet.cpp	708204
key.cpp	121412
key_io.cpp	228028
keystore.cpp	441392
leveldb/db/builder.cc	80432
leveldb/db/c.cc	69276
leveldb/db/dbformat.cc	62416
leveldb/db/db_impl.cc	176980
leveldb/db/db_iter.cc	82104
leveldb/db/dumpfile.cc	96772
leveldb/db/filename.cc	69704
leveldb/db/log_reader.cc	62776
leveldb/db/log_writer.cc	60488
leveldb/db/memtable.cc	68064
leveldb/db/repair.cc	140544
leveldb/db/table_cache.cc	63348
leveldb/db/version_edit.cc	100996
leveldb/db/version_set.cc	176228
leveldb/db/write_batch.cc	64000
leveldb/helpers/memenv/memenv.cc	104628
leveldb/port/port_posix.cc	55404
leveldb/port/port_posix_sse.cc	58788
leveldb/table/block_builder.cc	72256
leveldb/table/block.cc	74708
leveldb/table/filter_block.cc	72000
leveldb/table/format.cc	62508
leveldb/table/iterator.cc	49556
leveldb/table/merger.cc	54980
leveldb/table/table_builder.cc	71380
leveldb/table/table.cc	67104
leveldb/table/two_level_iterator.cc	66664
leveldb/util/arena.cc	62708
leveldb/util/bloom.cc	50956
leveldb/util/cache.cc	61448
leveldb/util/coding.cc	57616
leveldb/util/comparator.cc	60948
leveldb/util/crc32c.cc	57148
leveldb/util/env.cc	56380
leveldb/util/env_posix.cc	110268
leveldb/util/filter_policy.cc	46772
leveldb/util/hash.cc	55120
leveldb/util/histogram.cc	64004
leveldb/util/logging.cc	61076
leveldb/util/options.cc	50752
leveldb/util/status.cc	57752
lib/univalue.cpp	144548
lib/univalue_get.cpp	79124
lib/univalue_read.cpp	123936
lib/univalue_write.cpp	83164
logging.cpp	188492
merkleblock.cpp	123220
miner.cpp	521796
netaddress.cpp	131020
netbase.cpp	302472
net.cpp	641752
net_processing.cpp	903388
node/coin.cpp	359000
node/psbt.cpp	229924
node/transaction.cpp	501876
noui.cpp	323392
outputtype.cpp	323616
policy/feerate.cpp	96472
policy/fees.cpp	510808
policy/policy.cpp	160232
policy/rbf.cpp	330760
policy/settings.cpp	126552
pow.cpp	173568
primitives/block.cpp	118412
primitives/transaction.cpp	151108
protocol.cpp	252616
psbt.cpp	303900
-pthread	979916
pubkey.cpp	86164
random.cpp	210588
rest.cpp	553628
rpc/blockchain.cpp	845848
rpc/client.cpp	280928
rpc/mining.cpp	676556
rpc/misc.cpp	515044
rpc/net.cpp	468504
rpc/protocol.cpp	279596
rpc/rawtransaction.cpp	839388
rpc/rawtransaction_util.cpp	447992
rpc/server.cpp	537812
rpc/util.cpp	441840
scheduler.cpp	363208
script/bitcoinconsensus.cpp	145580
script/descriptor.cpp	488564
script/interpreter.cpp	210904
script/ismine.cpp	318740
script/script.cpp	97876
script/script_error.cpp	24064
script/sigcache.cpp	418276
script/sign.cpp	307888
script/standard.cpp	247636
shutdown.cpp	33240
src/secp256k1.c	90624
support/cleanse.cpp	25672
support/lockedpool.cpp	124324
sync.cpp	162008
test/addrman_tests.cpp	652328
test/allocator_tests.cpp	553304
test/amount_tests.cpp	551544
test/arith_uint256_tests.cpp	692628
test/base32_tests.cpp	547868
test/base58_tests.cpp	537788
test/base64_tests.cpp	547844
test/bech32_tests.cpp	536752
test/bip32_tests.cpp	559484
test/blockchain_tests.cpp	539524
test/blockencodings_tests.cpp	647392
test/blockfilter_index_tests.cpp	644636
test/blockfilter_tests.cpp	595292
test/bloom_tests.cpp	672124
test/bswap_tests.cpp	548016
test/checkqueue_tests.cpp	668592
test/coins_tests.cpp	674912
test/compress_tests.cpp	538112
test/crypto_tests.cpp	641656
test/cuckoocache_tests.cpp	569404
test/dbwrapper_tests.cpp	571232
test/denialofservice_tests.cpp	677660
test/descriptor_tests.cpp	686608
test/flatfile_tests.cpp	554992
test/fs_tests.cpp	537424
test/getarg_tests.cpp	635464
test/hash_tests.cpp	554424
test/key_io_tests.cpp	596568
test/key_tests.cpp	597832
test/limitedmap_tests.cpp	536476
test/main.cpp	142840
test/mempool_tests.cpp	648792
test/merkleblock_tests.cpp	543556
test/merkle_tests.cpp	536332
test/miner_tests.cpp	683644
test/multisig_tests.cpp	586904
test/netbase_tests.cpp	624632
test/net_tests.cpp	632696
test/pmt_tests.cpp	559956
test/policyestimator_tests.cpp	563212
test/pow_tests.cpp	560040
test/prevector_tests.cpp	546660
test/raii_event_tests.cpp	546828
test/random_tests.cpp	584428
test/reverselock_tests.cpp	536732
test/rpc_tests.cpp	782644
test/sanity_tests.cpp	547536
test/scheduler_tests.cpp	562272
test/scriptnum_tests.cpp	539224
test/script_p2sh_tests.cpp	644272
test/script_standard_tests.cpp	728936
test/script_tests.cpp	1422668
test/serialize_tests.cpp	651156
test/setup_common.cpp	625256
test/sighash_tests.cpp	598748
test/sigopcount_tests.cpp	589084
test/skiplist_tests.cpp	559568
test/streams_tests.cpp	566088
test/sync_tests.cpp	536888
test/timedata_tests.cpp	538180
test/torcontrol_tests.cpp	585644
test/transaction_tests.cpp	834816
test/txindex_tests.cpp	568128
test/txvalidationcache_tests.cpp	640032
test/txvalidation_tests.cpp	546328
test/uint256_tests.cpp	593688
test/util_tests.cpp	897316
test/util_threadnames_tests.cpp	536448
test/validation_block_tests.cpp	594424
test/validation_tests.cpp	618512
test/versionbits_tests.cpp	575776
threadinterrupt.cpp	79676
timedata.cpp	275988
torcontrol.cpp	625500
txdb.cpp	480384
txmempool.cpp	649156
ui_interface.cpp	493492
uint256.cpp	66424
util/bip32.cpp	90048
util/bytevectorhash.cpp	54548
util/error.cpp	229948
util/fees.cpp	120932
util/moneystr.cpp	108296
util/rbf.cpp	90536
util/strencodings.cpp	106212
util/strencodings.cpp	2532
util/system.cpp	334932
util/threadnames.cpp	73768
util/time.cpp	446380
util/url.cpp	51316
util/validation.cpp	113864
validation.cpp	971080
validationinterface.cpp	840984
versionbits.cpp	192660
versionbitsinfo.cpp	54724
wallet/coincontrol.cpp	479884
wallet/coinselection.cpp	329532
wallet/crypter.cpp	451652
wallet/db.cpp	491732
wallet/feebumper.cpp	506332
wallet/fees.cpp	467316
wallet/init.cpp	528692
wallet/load.cpp	560320
wallet/psbtwallet.cpp	475836
wallet/rpcdump.cpp	857432
wallet/rpcwallet.cpp	1297072
wallet/test/coinselector_tests.cpp	876888
wallet/test/db_tests.cpp	564976
wallet/test/init_test_fixture.cpp	545212
wallet/test/init_tests.cpp	562384
wallet/test/psbt_wallet_tests.cpp	854972
wallet/test/wallet_crypto_tests.cpp	585340
wallet/test/wallet_test_fixture.cpp	680460
wallet/test/wallet_tests.cpp	875692
wallet/wallet.cpp	1162820
wallet/walletdb.cpp	764024
wallet/walletutil.cpp	264060
warnings.cpp	224348

  0diff --git a/tmp/a b/tmp/b
  1index 71c592b9a5..d3402ea013 100644
  2--- a/tmp/a
  3+++ b/tmp/b
  4@@ -2,2 +2,2 @@
  5-| addrman.cpp | 308212 |
  6-| arith_uint256.cpp | 72260 |
  7+| addrman.cpp | 300868 |
  8+| arith_uint256.cpp | 62044 |
  9@@ -13 +13 @@
 10-| clientversion.cpp | 87868 |
 11+| clientversion.cpp | 87612 |
 12@@ -19 +19 @@
 13-| consensus/merkle.cpp | 101224 |
 14+| consensus/merkle.cpp | 101216 |
 15@@ -22,3 +22,3 @@
 16-| core_read.cpp | 413892 |
 17-| core_write.cpp | 375424 |
 18-| crypto/aes.cpp | 36584 |
 19+| core_read.cpp | 331212 |
 20+| core_write.cpp | 375372 |
 21+| crypto/aes.cpp | 36304 |
 22@@ -32 +32 @@
 23-| crypto/sha256_avx2.cpp | 370964 |
 24+| crypto/sha256_avx2.cpp | 350716 |
 25@@ -35 +35 @@
 26-| crypto/sha256_sse41.cpp | 380744 |
 27+| crypto/sha256_sse41.cpp | 360612 |
 28@@ -43 +43 @@
 29-| httprpc.cpp | 419268 |
 30+| httprpc.cpp | 419240 |
 31@@ -48 +48 @@
 32-| init.cpp | 924328 |
 33+| init.cpp | 924352 |
 34@@ -51,3 +51,3 @@
 35-| interfaces/node.cpp | 619612 |
 36-| interfaces/wallet.cpp | 845216 |
 37-| key.cpp | 122092 |
 38+| interfaces/node.cpp | 533112 |
 39+| interfaces/wallet.cpp | 708204 |
 40+| key.cpp | 121412 |
 41@@ -97,4 +97,4 @@
 42-| lib/univalue.cpp | 2712 |
 43-| lib/univalue_get.cpp | 2644 |
 44-| lib/univalue_read.cpp | 2692 |
 45-| lib/univalue_write.cpp | 2700 |
 46+| lib/univalue.cpp | 144548 |
 47+| lib/univalue_get.cpp | 79124 |
 48+| lib/univalue_read.cpp | 123936 |
 49+| lib/univalue_write.cpp | 83164 |
 50@@ -102,2 +102,2 @@
 51-| merkleblock.cpp | 123228 |
 52-| miner.cpp | 545048 |
 53+| merkleblock.cpp | 123220 |
 54+| miner.cpp | 521796 |
 55@@ -105 +105 @@
 56-| netbase.cpp | 303412 |
 57+| netbase.cpp | 302472 |
 58@@ -107 +107 @@
 59-| net_processing.cpp | 903684 |
 60+| net_processing.cpp | 903388 |
 61@@ -114,2 +114,2 @@
 62-| policy/fees.cpp | 521436 |
 63-| policy/policy.cpp | 281476 |
 64+| policy/fees.cpp | 510808 |
 65+| policy/policy.cpp | 160232 |
 66@@ -119 +119 @@
 67-| primitives/block.cpp | 118776 |
 68+| primitives/block.cpp | 118412 |
 69@@ -122,2 +122,2 @@
 70-| psbt.cpp | 323128 |
 71-| -pthread | 992596 |
 72+| psbt.cpp | 303900 |
 73+| -pthread | 979916 |
 74@@ -127 +127 @@
 75-| rpc/blockchain.cpp | 847204 |
 76+| rpc/blockchain.cpp | 845848 |
 77@@ -130,6 +130,6 @@
 78-| rpc/misc.cpp | 509676 |
 79-| rpc/net.cpp | 470636 |
 80-| rpc/protocol.cpp | 279744 |
 81-| rpc/rawtransaction.cpp | 840376 |
 82-| rpc/rawtransaction_util.cpp | 449240 |
 83-| rpc/server.cpp | 538228 |
 84+| rpc/misc.cpp | 515044 |
 85+| rpc/net.cpp | 468504 |
 86+| rpc/protocol.cpp | 279596 |
 87+| rpc/rawtransaction.cpp | 839388 |
 88+| rpc/rawtransaction_util.cpp | 447992 |
 89+| rpc/server.cpp | 537812 |
 90@@ -142 +142 @@
 91-| script/script.cpp | 100584 |
 92+| script/script.cpp | 97876 |
 93@@ -144 +144 @@
 94-| script/sigcache.cpp | 418992 |
 95+| script/sigcache.cpp | 418276 |
 96@@ -146 +146 @@
 97-| script/standard.cpp | 333964 |
 98+| script/standard.cpp | 247636 |
 99@@ -148 +148 @@
100-| src/secp256k1.c | 15296 |
101+| src/secp256k1.c | 90624 |
102@@ -168 +168 @@
103-| test/coins_tests.cpp | 684284 |
104+| test/coins_tests.cpp | 674912 |
105@@ -173 +173 @@
106-| test/denialofservice_tests.cpp | 677568 |
107+| test/denialofservice_tests.cpp | 677660 |
108@@ -186,2 +186,2 @@
109-| test/miner_tests.cpp | 683808 |
110-| test/multisig_tests.cpp | 587852 |
111+| test/miner_tests.cpp | 683644 |
112+| test/multisig_tests.cpp | 586904 |
113@@ -197 +197 @@
114-| test/rpc_tests.cpp | 808252 |
115+| test/rpc_tests.cpp | 782644 |
116@@ -201 +201 @@
117-| test/script_p2sh_tests.cpp | 644396 |
118+| test/script_p2sh_tests.cpp | 644272 |
119@@ -205 +205 @@
120-| test/setup_common.cpp | 625408 |
121+| test/setup_common.cpp | 625256 |
122@@ -212 +212 @@
123-| test/torcontrol_tests.cpp | 585648 |
124+| test/torcontrol_tests.cpp | 585644 |
125@@ -214,2 +214,2 @@
126-| test/txindex_tests.cpp | 574132 |
127-| test/txvalidationcache_tests.cpp | 639960 |
128+| test/txindex_tests.cpp | 568128 |
129+| test/txvalidationcache_tests.cpp | 640032 |
130@@ -224 +224 @@
131-| timedata.cpp | 276120 |
132+| timedata.cpp | 275988 |
133@@ -226,3 +226,3 @@
134-| txdb.cpp | 485644 |
135-| txmempool.cpp | 653824 |
136-| ui_interface.cpp | 634304 |
137+| txdb.cpp | 480384 |
138+| txmempool.cpp | 649156 |
139+| ui_interface.cpp | 493492 |
140@@ -238 +238 @@
141-| util/system.cpp | 335012 |
142+| util/system.cpp | 334932 |
143@@ -243,2 +243,2 @@
144-| validation.cpp | 971296 |
145-| validationinterface.cpp | 851868 |
146+| validation.cpp | 971080 |
147+| validationinterface.cpp | 840984 |
148@@ -250,4 +250,4 @@
149-| wallet/db.cpp | 511924 |
150-| wallet/feebumper.cpp | 612368 |
151-| wallet/fees.cpp | 485752 |
152-| wallet/init.cpp | 637808 |
153+| wallet/db.cpp | 491732 |
154+| wallet/feebumper.cpp | 506332 |
155+| wallet/fees.cpp | 467316 |
156+| wallet/init.cpp | 528692 |
157@@ -256,2 +256,2 @@
158-| wallet/rpcdump.cpp | 861276 |
159-| wallet/rpcwallet.cpp | 1332508 |
160+| wallet/rpcdump.cpp | 857432 |
161+| wallet/rpcwallet.cpp | 1297072 |
162@@ -261,2 +261,2 @@
163-| wallet/test/init_tests.cpp | 674808 |
164-| wallet/test/psbt_wallet_tests.cpp | 855056 |
165+| wallet/test/init_tests.cpp | 562384 |
166+| wallet/test/psbt_wallet_tests.cpp | 854972 |
167@@ -264 +264 @@
168-| wallet/test/wallet_test_fixture.cpp | 686740 |
169+| wallet/test/wallet_test_fixture.cpp | 680460 |
170@@ -266 +266 @@
171-| wallet/wallet.cpp | 1284744 |
172+| wallet/wallet.cpp | 1162820 |
173@@ -269 +269 @@
174-| warnings.cpp | 224448 |
175+| warnings.cpp | 224348 |

MarcoFalke commented at 2:44 pm on June 6, 2019: member

Thanks for working on this. See related issues:

Building requires >1GB memory #6658
Cleanup code by using forward declarations and other methods… #2767

MarcoFalke removed the label Needs gitian build on Jun 6, 2019

MarcoFalke merged this on Jun 6, 2019

MarcoFalke closed this on Jun 6, 2019

MarcoFalke referenced this in commit d0f81a96d9 on Jun 6, 2019

sidhujag referenced this in commit 14681afec2 on Jun 6, 2019

practicalswift commented at 2:21 pm on June 7, 2019: contributor

@MarcoFalke Thanks for quantifying the results. I want to replicate your numbers: how did you perform the measurements - using time or something else? :-)

MarcoFalke commented at 8:13 am on June 8, 2019: member

Yeah, I run gcc through /usr/bin/time.

practicalswift commented at 10:31 pm on June 17, 2019: contributor

Post-merge “before vs after” statistics for this PR using a tool I’ve written to measure the impact a PR has on memory usage and compile times. It outputs Markdown tables for easy use here on GitHub :-)

Seems like this PR cut max memory usage (RSS) and total compile time by roughly 2 %.

Comparing max memory usage (RSS) between old revision 58e291cfad12fa85af87d093acfa7b44702e3521 (branch HEAD) and new revision eca97676737174ac6ff775c60e3798c69fb67f4a (branch HEAD):

File	Old	New	Diff	Percent
`arith_uint256.cpp`	69 MB	57 MB	-11 MB	-17 %
`bench/bench_bench_bitcoin-base58.o`	196 MB	72 MB	-125 MB	-64 %
`bench/bench_bench_bitcoin-bech32.o`	199 MB	85 MB	-114 MB	-57 %
`bench/bench_bench_bitcoin-bench_bitcoin.o`	252 MB	236 MB	-16 MB	-6 %
`bench/bench_bench_bitcoin-checkqueue.o`	315 MB	283 MB	-32 MB	-10 %
`bench/bench_bench_bitcoin-examples.o`	197 MB	71 MB	-126 MB	-64 %
`bench/bench_bench_bitcoin-verify_script.o`	232 MB	218 MB	-14 MB	-6 %
`crypto/crypto_libbitcoin_crypto_avx2_a-sha256_avx2.o`	244 MB	221 MB	-23 MB	-9 %
`crypto/crypto_libbitcoin_crypto_sse41_a-sha256_sse41.o`	253 MB	230 MB	-23 MB	-9 %
`interfaces/libbitcoin_server_a-node.o`	600 MB	524 MB	-76 MB	-13 %
`interfaces/libbitcoin_wallet_a-wallet.o`	787 MB	706 MB	-81 MB	-10 %
`libbitcoin_common_a-core_read.o`	387 MB	313 MB	-75 MB	-19 %
`libbitcoin_common_a-psbt.o`	310 MB	274 MB	-35 MB	-11 %
`libbitcoin_consensus_a-arith_uint256.o`	68 MB	58 MB	-10 MB	-15 %
`libbitcoin_server_a-ui_interface.o`	603 MB	509 MB	-94 MB	-16 %
`policy/libbitcoin_common_a-policy.o`	283 MB	165 MB	-118 MB	-42 %
`qt/qt_libbitcoinqt_a-addresstablemodel.o`	617 MB	580 MB	-38 MB	-6 %
`qt/qt_libbitcoinqt_a-bitcoinunits.o`	162 MB	142 MB	-20 MB	-12 %
`qt/qt_libbitcoinqt_a-clientmodel.o`	529 MB	383 MB	-146 MB	-28 %
`qt/qt_libbitcoinqt_a-coincontroldialog.o`	766 MB	668 MB	-99 MB	-13 %
`qt/qt_libbitcoinqt_a-transactiondesc.o`	650 MB	514 MB	-136 MB	-21 %
`qt/qt_libbitcoinqt_a-transactiontablemodel.o`	493 MB	408 MB	-86 MB	-17 %
`qt/qt_libbitcoinqt_a-utilitydialog.o`	385 MB	346 MB	-39 MB	-10 %
`qt/qt_libbitcoinqt_a-walletmodeltransaction.o`	386 MB	291 MB	-95 MB	-25 %
`qt/test/qt_test_test_bitcoin_qt-apptests.o`	430 MB	397 MB	-33 MB	-8 %
`script/libbitcoin_common_a-standard.o`	334 MB	246 MB	-88 MB	-26 %
`wallet/libbitcoin_server_a-init.o`	604 MB	511 MB	-92 MB	-15 %
`wallet/libbitcoin_wallet_a-feebumper.o`	582 MB	502 MB	-80 MB	-14 %
`wallet/libbitcoin_wallet_a-wallet.o`	1187 MB	1084 MB	-102 MB	-9 %
`wallet/test/test_test_bitcoin-init_tests.o`	677 MB	558 MB	-119 MB	-18 %
`zmq/libbitcoin_zmq_a-zmqabstractnotifier.o`	224 MB	94 MB	-130 MB	-58 %
… suppressing 415 unchanged measurements …	…	…	…	…
Average (N=446)	324 MB	318 MB	-6 MB	-2 %
Sum (N=446)	144641 MB	142009 MB	-2632 MB	-2 %

Comparing build time (user CPU time) between old revision 58e291cfad12fa85af87d093acfa7b44702e3521 (branch HEAD) and new revision eca97676737174ac6ff775c60e3798c69fb67f4a (branch HEAD) for files with a measured change in max memory usage (RSS):

File	Old	New	Diff	Percent
`arith_uint256.cpp`	1 s.	1 s.	-0 s.	-12 %
`bench/bench_bench_bitcoin-base58.o`	2 s.	1 s.	-2 s.	-72 %
`bench/bench_bench_bitcoin-bech32.o`	3 s.	1 s.	-2 s.	-63 %
`bench/bench_bench_bitcoin-bench_bitcoin.o`	4 s.	3 s.	-0 s.	-7 %
`bench/bench_bench_bitcoin-checkqueue.o`	6 s.	5 s.	-0 s.	-8 %
`bench/bench_bench_bitcoin-examples.o`	3 s.	1 s.	-2 s.	-72 %
`bench/bench_bench_bitcoin-verify_script.o`	4 s.	4 s.	-0 s.	-3 %
`crypto/crypto_libbitcoin_crypto_avx2_a-sha256_avx2.o`	5 s.	5 s.	-0 s.	-6 %
`crypto/crypto_libbitcoin_crypto_sse41_a-sha256_sse41.o`	6 s.	5 s.	-0 s.	-6 %
`interfaces/libbitcoin_server_a-node.o`	14 s.	10 s.	-3 s.	-24 %
`interfaces/libbitcoin_wallet_a-wallet.o`	22 s.	18 s.	-4 s.	-19 %
`libbitcoin_common_a-core_read.o`	9 s.	7 s.	-2 s.	-18 %
`libbitcoin_common_a-psbt.o`	8 s.	7 s.	-1 s.	-11 %
`libbitcoin_consensus_a-arith_uint256.o`	1 s.	1 s.	-0 s.	-17 %
`libbitcoin_server_a-ui_interface.o`	19 s.	17 s.	-2 s.	-8 %
`policy/libbitcoin_common_a-policy.o`	4 s.	2 s.	-2 s.	-40 %
`qt/qt_libbitcoinqt_a-addresstablemodel.o`	11 s.	11 s.	-0 s.	-3 %
`qt/qt_libbitcoinqt_a-bitcoinunits.o`	2 s.	2 s.	-0 s.	-16 %
`qt/qt_libbitcoinqt_a-clientmodel.o`	9 s.	7 s.	-3 s.	-30 %
`qt/qt_libbitcoinqt_a-coincontroldialog.o`	14 s.	13 s.	-1 s.	-7 %
`qt/qt_libbitcoinqt_a-transactiondesc.o`	13 s.	10 s.	-3 s.	-21 %
`qt/qt_libbitcoinqt_a-transactiontablemodel.o`	10 s.	8 s.	-2 s.	-23 %
`qt/qt_libbitcoinqt_a-utilitydialog.o`	7 s.	6 s.	-1 s.	-9 %
`qt/qt_libbitcoinqt_a-walletmodeltransaction.o`	6 s.	4 s.	-2 s.	-38 %
`qt/test/qt_test_test_bitcoin_qt-apptests.o`	7 s.	7 s.	-1 s.	-8 %
`script/libbitcoin_common_a-standard.o`	6 s.	4 s.	-2 s.	-29 %
`wallet/libbitcoin_server_a-init.o`	13 s.	10 s.	-3 s.	-24 %
`wallet/libbitcoin_wallet_a-feebumper.o`	11 s.	10 s.	-1 s.	-10 %
`wallet/libbitcoin_wallet_a-wallet.o`	38 s.	37 s.	-2 s.	-4 %
`wallet/test/test_test_bitcoin-init_tests.o`	14 s.	12 s.	-2 s.	-12 %
`zmq/libbitcoin_zmq_a-zmqabstractnotifier.o`	3 s.	1 s.	-2 s.	-64 %
… suppressing 415 measurements …	…	…	…	…
Average (N=446)	7 s.	7 s.	-0 s.	-2 %
Sum (N=446)	2985 s.	2922 s.	-63 s.	-2 %

practicalswift commented at 10:15 am on June 20, 2019: contributor

@fanquake @promag @MarcoFalke As reviewers of this PR you might be interested in reviewing and/or concept ACK:ing the follow-up PR #16238 which does essentially the same thing but for C++ standard library headers and C compatibility headers.

That PR brings another reduction (-838 MB) in accumulated max memory usage (RSS) during a build :-)

deadalnix referenced this in commit a66c800178 on Jun 2, 2020

deadalnix referenced this in commit a4f8e383fd on Jun 2, 2020

practicalswift deleted the branch on Apr 10, 2021

Munkybooty referenced this in commit febf61f5ac on Oct 20, 2021

vijaydasmp referenced this in commit c51633413c on Oct 21, 2021

vijaydasmp referenced this in commit 60500d0589 on Dec 6, 2021

vijaydasmp referenced this in commit 359d82039d on Dec 6, 2021

vijaydasmp referenced this in commit 1185debc03 on Dec 11, 2021

vijaydasmp referenced this in commit dcbc0d4e3d on Dec 11, 2021

vijaydasmp referenced this in commit 1350bcb46d on Dec 11, 2021

vijaydasmp referenced this in commit de2fbb2d0a on Dec 11, 2021

vijaydasmp referenced this in commit 228e4e3c97 on Dec 13, 2021

vijaydasmp referenced this in commit d9e694abdd on Dec 13, 2021

vijaydasmp referenced this in commit 90b9a5b784 on Dec 13, 2021

vijaydasmp referenced this in commit 7675e3ac1d on Dec 13, 2021

vijaydasmp referenced this in commit 5695202a80 on Dec 14, 2021

vijaydasmp referenced this in commit 8023da61e1 on Dec 18, 2021

vijaydasmp referenced this in commit 0b20ff2caa on Dec 19, 2021

vijaydasmp referenced this in commit adfe89cca8 on Dec 19, 2021

vijaydasmp referenced this in commit 9667a8fd91 on Dec 19, 2021

vijaydasmp referenced this in commit 0f2679f560 on Dec 20, 2021

vijaydasmp referenced this in commit d3804888b9 on Dec 20, 2021

vijaydasmp referenced this in commit 6f047e8da5 on Dec 20, 2021

vijaydasmp referenced this in commit e43ffec414 on Dec 21, 2021

vijaydasmp referenced this in commit a2724130d5 on Dec 24, 2021

vijaydasmp referenced this in commit bddede90e8 on Dec 25, 2021

vijaydasmp referenced this in commit fb44deccd0 on Dec 27, 2021

PastaPastaPasta referenced this in commit 3148450032 on Jan 3, 2022

DrahtBot locked this on Aug 16, 2022

refactor: Remove unused includes #16129

Conflicts