test: move-only: Split large tests into smaller compile units #16069

pull MarcoFalke wants to merge 1 commits into bitcoin:master from MarcoFalke:1905-testCompFast changing 6 files +1484 −1419
  1. MarcoFalke commented at 8:32 PM on May 21, 2019: member

    Building the tests (even with wallet disabled) might take a lot of memory and time for a single cpp file. Fix that by splitting up the tests into smaller units.

    See "Building requires >1GB memory #6658" for some background discussion.

    Can be reviewed with the --color-moved=dimmed-zebra git diff option

  2. MarcoFalke added the label Tests on May 21, 2019
  3. DrahtBot commented at 10:22 PM on May 21, 2019: member

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #16278 (tests: Reduce compile-time memory usage, compilation time and unneccessary recompiles by removing unused includes in tests by practicalswift)
    • #16273 (refactor: Reduce compile-time memory usage by 2%, compilation time by 2% and avoid unnecessary recompiles by removing unused includes by practicalswift)
    • #16227 (Refactor CWallet's inheritance chain by achow101)
    • #15935 (WIP: Add <datadir>/settings.json persistent settings storage by ryanofsky)
    • #15934 (Separate settings merging from parsing by ryanofsky)
    • #14045 (refactor: Fix the chainparamsbase -> util circular dependency by Empact)
    • #13751 (Utils and libraries: Drops the boost/algorithm/string/split.hpp dependency by l2a5b1)
    • #13525 (policy: Report reason inputs are nonstandard from AreInputsStandard by Empact)
    • #8994 (Testchains: Introduce custom chain whose constructor... by jtimon)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  4. MarcoFalke renamed this:
    test: Split large tests into smaller compile units
    test: move-only: Split large tests into smaller compile units
    on May 21, 2019
  5. MarcoFalke commented at 11:00 PM on May 21, 2019: member

    <details><summary>Maximum resident set size before</summary>

    rss file
    142888 test/main.cpp
    625172 test/setup_common.cpp
    686472 wallet/test/wallet_test_fixture.cpp
    544948 wallet/test/init_test_fixture.cpp
    692280 test/arith_uint256_tests.cpp
    652108 test/addrman_tests.cpp
    551336 test/amount_tests.cpp
    552988 test/allocator_tests.cpp
    548280 test/base32_tests.cpp
    537832 test/base58_tests.cpp
    548164 test/base64_tests.cpp
    536968 test/bech32_tests.cpp
    559472 test/bip32_tests.cpp
    539684 test/blockchain_tests.cpp
    647268 test/blockencodings_tests.cpp
    595196 test/blockfilter_tests.cpp
    644400 test/blockfilter_index_tests.cpp
    672252 test/bloom_tests.cpp
    548044 test/bswap_tests.cpp
    668488 test/checkqueue_tests.cpp
    684164 test/coins_tests.cpp
    537948 test/compress_tests.cpp
    641656 test/crypto_tests.cpp
    570004 test/cuckoocache_tests.cpp
    677508 test/denialofservice_tests.cpp
    686380 test/descriptor_tests.cpp
    554816 test/flatfile_tests.cpp
    537476 test/fs_tests.cpp
    635492 test/getarg_tests.cpp
    554092 test/hash_tests.cpp
    597024 test/key_io_tests.cpp
    597764 test/key_tests.cpp
    536468 test/limitedmap_tests.cpp
    571140 test/dbwrapper_tests.cpp
    618428 test/validation_tests.cpp
    648752 test/mempool_tests.cpp
    536232 test/merkle_tests.cpp
    543296 test/merkleblock_tests.cpp
    683988 test/miner_tests.cpp
    587696 test/multisig_tests.cpp
    632672 test/net_tests.cpp
    624584 test/netbase_tests.cpp
    559684 test/pmt_tests.cpp
    563288 test/policyestimator_tests.cpp
    560168 test/pow_tests.cpp
    546892 test/prevector_tests.cpp
    546664 test/raii_event_tests.cpp
    584624 test/random_tests.cpp
    547256 test/reverselock_tests.cpp
    808148 test/rpc_tests.cpp
    547576 test/sanity_tests.cpp
    562048 test/scheduler_tests.cpp
    644692 test/script_p2sh_tests.cpp
    1422800 test/script_tests.cpp
    728832 test/script_standard_tests.cpp
    538900 test/scriptnum_tests.cpp
    651236 test/serialize_tests.cpp
    598284 test/sighash_tests.cpp
    588984 test/sigopcount_tests.cpp
    559588 test/skiplist_tests.cpp
    565804 test/streams_tests.cpp
    536700 test/sync_tests.cpp
    536540 test/util_threadnames_tests.cpp
    538048 test/timedata_tests.cpp
    585368 test/torcontrol_tests.cpp
    834948 test/transaction_tests.cpp
    574084 test/txindex_tests.cpp
    546160 test/txvalidation_tests.cpp
    639888 test/txvalidationcache_tests.cpp
    593356 test/uint256_tests.cpp
    896684 test/util_tests.cpp
    594316 test/validation_block_tests.cpp
    575288 test/versionbits_tests.cpp
    564908 wallet/test/db_tests.cpp
    854708 wallet/test/psbt_wallet_tests.cpp
    875644 wallet/test/wallet_tests.cpp
    585204 wallet/test/wallet_crypto_tests.cpp
    876104 wallet/test/coinselector_tests.cpp
    674376 wallet/test/init_tests.cpp
    845440 interfaces/wallet.cpp
    479572 wallet/coincontrol.cpp
    451484 wallet/crypter.cpp
    511860 wallet/db.cpp
    612428 wallet/feebumper.cpp
    485684 wallet/fees.cpp
    556188 wallet/load.cpp
    475780 wallet/psbtwallet.cpp
    861792 wallet/rpcdump.cpp
    1332236 wallet/rpcwallet.cpp
    1281028 wallet/wallet.cpp
    762568 wallet/walletdb.cpp
    263872 wallet/walletutil.cpp
    330020 wallet/coinselection.cpp
    339432 addrdb.cpp
    308116 addrman.cpp
    268308 banman.cpp
    485424 blockencodings.cpp
    179068 blockfilter.cpp
    185244 chain.cpp
    208372 consensus/tx_verify.cpp
    254780 flatfile.cpp
    419480 httprpc.cpp
    343828 httpserver.cpp
    336228 index/base.cpp
    361940 index/blockfilterindex.cpp
    471024 index/txindex.cpp
    520764 interfaces/chain.cpp
    619468 interfaces/node.cpp
    924180 init.cpp
    302076 dbwrapper.cpp
    544704 miner.cpp
    641832 net.cpp
    903264 net_processing.cpp
    358976 node/coin.cpp
    229912 node/psbt.cpp
    501888 node/transaction.cpp
    323884 noui.cpp
    521168 policy/fees.cpp
    330612 policy/rbf.cpp
    126556 policy/settings.cpp
    173804 pow.cpp
    553416 rest.cpp
    847036 rpc/blockchain.cpp
    676636 rpc/mining.cpp
    509404 rpc/misc.cpp
    470480 rpc/net.cpp
    840408 rpc/rawtransaction.cpp
    538172 rpc/server.cpp
    418808 script/sigcache.cpp
    33152 shutdown.cpp
    276024 timedata.cpp
    625324 torcontrol.cpp
    485608 txdb.cpp
    653780 txmempool.cpp
    634352 ui_interface.cpp
    971112 validation.cpp
    851716 validationinterface.cpp
    192428 versionbits.cpp
    637688 wallet/init.cpp
    281000 rpc/client.cpp
    96560 base58.cpp
    69428 bech32.cpp
    163168 bloom.cpp
    416976 chainparams.cpp
    136964 coins.cpp
    138040 compressor.cpp
    413548 core_read.cpp
    375208 core_write.cpp
    122000 key.cpp
    227804 key_io.cpp
    441208 keystore.cpp
    123032 merkleblock.cpp
    131080 netaddress.cpp
    303380 netbase.cpp
    323464 outputtype.cpp
    96632 policy/feerate.cpp
    281516 policy/policy.cpp
    252320 protocol.cpp
    322676 psbt.cpp
    449220 rpc/rawtransaction_util.cpp
    441612 rpc/util.cpp
    363292 scheduler.cpp
    488440 script/descriptor.cpp
    318996 script/ismine.cpp
    307684 script/sign.cpp
    334212 script/standard.cpp
    54380 versionbitsinfo.cpp
    224324 warnings.cpp
    124168 support/lockedpool.cpp
    235096 chainparamsbase.cpp
    87436 clientversion.cpp
    25696 compat/glibc_sanity.cpp
    68020 compat/glibcxx_sanity.cpp
    17080 compat/strnlen.cpp
    155372 fs.cpp
    215620 interfaces/handler.cpp
    188612 logging.cpp
    210444 random.cpp
    279612 rpc/protocol.cpp
    25620 support/cleanse.cpp
    161668 sync.cpp
    79660 threadinterrupt.cpp
    89828 util/bip32.cpp
    54568 util/bytevectorhash.cpp
    230148 util/error.cpp
    120564 util/fees.cpp
    335328 util/system.cpp
    108312 util/moneystr.cpp
    90484 util/rbf.cpp
    73596 util/threadnames.cpp
    2664 util/strencodings.cpp
    442260 util/time.cpp
    51268 util/url.cpp
    114116 util/validation.cpp
    72340 arith_uint256.cpp
    101188 consensus/merkle.cpp
    119668 consensus/tx_check.cpp
    81204 hash.cpp
    118492 primitives/block.cpp
    151388 primitives/transaction.cpp
    86228 pubkey.cpp
    145560 script/bitcoinconsensus.cpp
    211024 script/interpreter.cpp
    100572 script/script.cpp
    24088 script/script_error.cpp
    66428 uint256.cpp
    106076 util/strencodings.cpp
    36372 crypto/aes.cpp
    33016 crypto/chacha20.cpp
    47732 crypto/hkdf_sha256_32.cpp
    48728 crypto/hmac_sha256.cpp
    27220 crypto/hmac_sha512.cpp
    30864 crypto/poly1305.cpp
    66676 crypto/ripemd160.cpp
    48804 crypto/sha1.cpp
    129300 crypto/sha256.cpp
    57124 crypto/sha512.cpp
    58328 crypto/siphash.cpp
    25280 crypto/sha256_sse4.cpp
    380596 crypto/sha256_sse41.cpp
    370976 crypto/sha256_avx2.cpp
    118640 crypto/sha256_shani.cpp
    2664 lib/univalue.cpp
    2644 lib/univalue_get.cpp
    2636 lib/univalue_read.cpp
    2672 lib/univalue_write.cpp
    80248 leveldb/db/builder.cc
    69432 leveldb/db/c.cc
    62316 leveldb/db/dbformat.cc
    177068 leveldb/db/db_impl.cc
    81864 leveldb/db/db_iter.cc
    96676 leveldb/db/dumpfile.cc
    69900 leveldb/db/filename.cc
    63128 leveldb/db/log_reader.cc
    60824 leveldb/db/log_writer.cc
    68068 leveldb/db/memtable.cc
    140740 leveldb/db/repair.cc
    63396 leveldb/db/table_cache.cc
    100876 leveldb/db/version_edit.cc
    175984 leveldb/db/version_set.cc
    64132 leveldb/db/write_batch.cc
    72380 leveldb/table/block_builder.cc
    74412 leveldb/table/block.cc
    71848 leveldb/table/filter_block.cc
    62496 leveldb/table/format.cc
    49552 leveldb/table/iterator.cc
    55320 leveldb/table/merger.cc
    70904 leveldb/table/table_builder.cc
    67520 leveldb/table/table.cc
    66692 leveldb/table/two_level_iterator.cc
    62608 leveldb/util/arena.cc
    51080 leveldb/util/bloom.cc
    61568 leveldb/util/cache.cc
    57560 leveldb/util/coding.cc
    61116 leveldb/util/comparator.cc
    57092 leveldb/util/crc32c.cc
    55984 leveldb/util/env.cc
    110316 leveldb/util/env_posix.cc
    46436 leveldb/util/filter_policy.cc
    55140 leveldb/util/hash.cc
    64320 leveldb/util/histogram.cc
    61076 leveldb/util/logging.cc
    51012 leveldb/util/options.cc
    57592 leveldb/util/status.cc
    54976 leveldb/port/port_posix.cc
    58672 leveldb/port/port_posix_sse.cc
    104404 leveldb/helpers/memenv/memenv.cc
    15144 src/secp256k1.c
    991952 -pthread

    </details>

    <details><summary>Maximum resident set size after</summary>

    rss file
    142888 test/main.cpp
    625172 test/setup_common.cpp
    686472 wallet/test/wallet_test_fixture.cpp
    544948 wallet/test/init_test_fixture.cpp
    692280 test/arith_uint256_tests.cpp
    652108 test/addrman_tests.cpp
    551336 test/amount_tests.cpp
    552988 test/allocator_tests.cpp
    779156 test/args_tests.cpp
    548280 test/base32_tests.cpp
    537832 test/base58_tests.cpp
    548164 test/base64_tests.cpp
    536968 test/bech32_tests.cpp
    559472 test/bip32_tests.cpp
    539684 test/blockchain_tests.cpp
    647268 test/blockencodings_tests.cpp
    595196 test/blockfilter_tests.cpp
    644400 test/blockfilter_index_tests.cpp
    672252 test/bloom_tests.cpp
    548044 test/bswap_tests.cpp
    668488 test/checkqueue_tests.cpp
    684164 test/coins_tests.cpp
    537948 test/compress_tests.cpp
    641656 test/crypto_tests.cpp
    570004 test/cuckoocache_tests.cpp
    677508 test/denialofservice_tests.cpp
    686380 test/descriptor_tests.cpp
    554816 test/flatfile_tests.cpp
    537476 test/fs_tests.cpp
    635492 test/getarg_tests.cpp
    554092 test/hash_tests.cpp
    597024 test/key_io_tests.cpp
    597764 test/key_tests.cpp
    536468 test/limitedmap_tests.cpp
    571140 test/dbwrapper_tests.cpp
    618428 test/validation_tests.cpp
    648752 test/mempool_tests.cpp
    536232 test/merkle_tests.cpp
    543296 test/merkleblock_tests.cpp
    683988 test/miner_tests.cpp
    587696 test/multisig_tests.cpp
    632672 test/net_tests.cpp
    624584 test/netbase_tests.cpp
    559684 test/pmt_tests.cpp
    563288 test/policyestimator_tests.cpp
    560168 test/pow_tests.cpp
    546892 test/prevector_tests.cpp
    546664 test/raii_event_tests.cpp
    584624 test/random_tests.cpp
    547256 test/reverselock_tests.cpp
    783084 test/rpc_tests.cpp
    547576 test/sanity_tests.cpp
    562048 test/scheduler_tests.cpp
    790696 test/script_eval_tests.cpp
    644692 test/script_p2sh_tests.cpp
    1308364 test/script_tests.cpp
    728832 test/script_standard_tests.cpp
    538900 test/scriptnum_tests.cpp
    651236 test/serialize_tests.cpp
    598284 test/sighash_tests.cpp
    588984 test/sigopcount_tests.cpp
    559588 test/skiplist_tests.cpp
    565804 test/streams_tests.cpp
    536700 test/sync_tests.cpp
    536540 test/util_threadnames_tests.cpp
    538048 test/timedata_tests.cpp
    585368 test/torcontrol_tests.cpp
    834948 test/transaction_tests.cpp
    574084 test/txindex_tests.cpp
    546160 test/txvalidation_tests.cpp
    639888 test/txvalidationcache_tests.cpp
    593356 test/uint256_tests.cpp
    807744 test/util_tests.cpp
    594316 test/validation_block_tests.cpp
    575288 test/versionbits_tests.cpp
    564908 wallet/test/db_tests.cpp
    854708 wallet/test/psbt_wallet_tests.cpp
    875644 wallet/test/wallet_tests.cpp
    585204 wallet/test/wallet_crypto_tests.cpp
    876104 wallet/test/coinselector_tests.cpp
    674376 wallet/test/init_tests.cpp
    720816 interfaces/wallet.cpp
    479572 wallet/coincontrol.cpp
    451484 wallet/crypter.cpp
    511860 wallet/db.cpp
    612428 wallet/feebumper.cpp
    485684 wallet/fees.cpp
    556188 wallet/load.cpp
    475780 wallet/psbtwallet.cpp
    861792 wallet/rpcdump.cpp
    1301264 wallet/rpcwallet.cpp
    1255100 wallet/wallet.cpp
    762568 wallet/walletdb.cpp
    263872 wallet/walletutil.cpp
    330020 wallet/coinselection.cpp
    339432 addrdb.cpp
    308116 addrman.cpp
    268308 banman.cpp
    485424 blockencodings.cpp
    179068 blockfilter.cpp
    185244 chain.cpp
    208372 consensus/tx_verify.cpp
    254780 flatfile.cpp
    419480 httprpc.cpp
    343828 httpserver.cpp
    336228 index/base.cpp
    361940 index/blockfilterindex.cpp
    471024 index/txindex.cpp
    520764 interfaces/chain.cpp
    619468 interfaces/node.cpp
    924180 init.cpp
    302076 dbwrapper.cpp
    544704 miner.cpp
    641832 net.cpp
    903264 net_processing.cpp
    358976 node/coin.cpp
    229912 node/psbt.cpp
    501888 node/transaction.cpp
    323884 noui.cpp
    521168 policy/fees.cpp
    330612 policy/rbf.cpp
    126556 policy/settings.cpp
    173804 pow.cpp
    553416 rest.cpp
    847036 rpc/blockchain.cpp
    676636 rpc/mining.cpp
    509404 rpc/misc.cpp
    470480 rpc/net.cpp
    838908 rpc/rawtransaction.cpp
    538172 rpc/server.cpp
    418808 script/sigcache.cpp
    33152 shutdown.cpp
    276024 timedata.cpp
    625324 torcontrol.cpp
    485608 txdb.cpp
    653780 txmempool.cpp
    634352 ui_interface.cpp
    971112 validation.cpp
    851716 validationinterface.cpp
    192428 versionbits.cpp
    637688 wallet/init.cpp
    281000 rpc/client.cpp
    96560 base58.cpp
    69428 bech32.cpp
    163168 bloom.cpp
    416976 chainparams.cpp
    136964 coins.cpp
    138040 compressor.cpp
    413548 core_read.cpp
    375208 core_write.cpp
    122000 key.cpp
    227804 key_io.cpp
    441208 keystore.cpp
    123032 merkleblock.cpp
    131080 netaddress.cpp
    303380 netbase.cpp
    323464 outputtype.cpp
    96632 policy/feerate.cpp
    281516 policy/policy.cpp
    252320 protocol.cpp
    322676 psbt.cpp
    449220 rpc/rawtransaction_util.cpp
    441612 rpc/util.cpp
    363292 scheduler.cpp
    488440 script/descriptor.cpp
    318996 script/ismine.cpp
    307684 script/sign.cpp
    334212 script/standard.cpp
    54380 versionbitsinfo.cpp
    224324 warnings.cpp
    124168 support/lockedpool.cpp
    235096 chainparamsbase.cpp
    87688 clientversion.cpp
    25696 compat/glibc_sanity.cpp
    68020 compat/glibcxx_sanity.cpp
    17080 compat/strnlen.cpp
    155372 fs.cpp
    215620 interfaces/handler.cpp
    188612 logging.cpp
    210444 random.cpp
    279612 rpc/protocol.cpp
    25620 support/cleanse.cpp
    161668 sync.cpp
    79660 threadinterrupt.cpp
    89828 util/bip32.cpp
    54568 util/bytevectorhash.cpp
    230148 util/error.cpp
    120564 util/fees.cpp
    335328 util/system.cpp
    108312 util/moneystr.cpp
    90484 util/rbf.cpp
    73596 util/threadnames.cpp
    2664 util/strencodings.cpp
    442260 util/time.cpp
    51268 util/url.cpp
    114116 util/validation.cpp
    72340 arith_uint256.cpp
    101188 consensus/merkle.cpp
    119668 consensus/tx_check.cpp
    81204 hash.cpp
    118492 primitives/block.cpp
    151388 primitives/transaction.cpp
    86228 pubkey.cpp
    145560 script/bitcoinconsensus.cpp
    211024 script/interpreter.cpp
    100572 script/script.cpp
    24088 script/script_error.cpp
    66428 uint256.cpp
    106076 util/strencodings.cpp
    36372 crypto/aes.cpp
    33016 crypto/chacha20.cpp
    47732 crypto/hkdf_sha256_32.cpp
    48728 crypto/hmac_sha256.cpp
    27220 crypto/hmac_sha512.cpp
    30864 crypto/poly1305.cpp
    66676 crypto/ripemd160.cpp
    48804 crypto/sha1.cpp
    129300 crypto/sha256.cpp
    57124 crypto/sha512.cpp
    58328 crypto/siphash.cpp
    25280 crypto/sha256_sse4.cpp
    380596 crypto/sha256_sse41.cpp
    370976 crypto/sha256_avx2.cpp
    118640 crypto/sha256_shani.cpp
    2648 lib/univalue.cpp
    2644 lib/univalue_get.cpp
    2692 lib/univalue_read.cpp
    2704 lib/univalue_write.cpp
    80248 leveldb/db/builder.cc
    69432 leveldb/db/c.cc
    62316 leveldb/db/dbformat.cc
    177068 leveldb/db/db_impl.cc
    81864 leveldb/db/db_iter.cc
    96676 leveldb/db/dumpfile.cc
    69900 leveldb/db/filename.cc
    63128 leveldb/db/log_reader.cc
    60824 leveldb/db/log_writer.cc
    68068 leveldb/db/memtable.cc
    140740 leveldb/db/repair.cc
    63396 leveldb/db/table_cache.cc
    100876 leveldb/db/version_edit.cc
    175984 leveldb/db/version_set.cc
    64132 leveldb/db/write_batch.cc
    72380 leveldb/table/block_builder.cc
    74412 leveldb/table/block.cc
    71848 leveldb/table/filter_block.cc
    62496 leveldb/table/format.cc
    49552 leveldb/table/iterator.cc
    55320 leveldb/table/merger.cc
    70904 leveldb/table/table_builder.cc
    67520 leveldb/table/table.cc
    66692 leveldb/table/two_level_iterator.cc
    62608 leveldb/util/arena.cc
    51080 leveldb/util/bloom.cc
    61568 leveldb/util/cache.cc
    57560 leveldb/util/coding.cc
    61116 leveldb/util/comparator.cc
    57092 leveldb/util/crc32c.cc
    55984 leveldb/util/env.cc
    110316 leveldb/util/env_posix.cc
    46436 leveldb/util/filter_policy.cc
    55140 leveldb/util/hash.cc
    64320 leveldb/util/histogram.cc
    61076 leveldb/util/logging.cc
    51012 leveldb/util/options.cc
    57592 leveldb/util/status.cc
    54976 leveldb/port/port_posix.cc
    58672 leveldb/port/port_posix_sse.cc
    104404 leveldb/helpers/memenv/memenv.cc
    15184 src/secp256k1.c
    993432 -pthread

    </details>

    Diff:

    @@ -6,6 +6,7 @@
 | 652108 | test/addrman_tests.cpp |
 | 551336 | test/amount_tests.cpp |
 | 552988 | test/allocator_tests.cpp |
+| 779156 | test/args_tests.cpp |
 | 548280 | test/base32_tests.cpp |
 | 537832 | test/base58_tests.cpp |
 | 548164 | test/base64_tests.cpp |
@@ -47,11 +48,12 @@
 | 546664 | test/raii_event_tests.cpp |
 | 584624 | test/random_tests.cpp |
 | 547256 | test/reverselock_tests.cpp |
-| 808148 | test/rpc_tests.cpp |
+| 783084 | test/rpc_tests.cpp |
 | 547576 | test/sanity_tests.cpp |
 | 562048 | test/scheduler_tests.cpp |
+| 790696 | test/script_eval_tests.cpp |
 | 644692 | test/script_p2sh_tests.cpp |
-| 1422800 | test/script_tests.cpp |
+| 1308364 | test/script_tests.cpp |
 | 728832 | test/script_standard_tests.cpp |
 | 538900 | test/scriptnum_tests.cpp |
 | 651236 | test/serialize_tests.cpp |
@@ -68,7 +70,7 @@
 | 546160 | test/txvalidation_tests.cpp |
 | 639888 | test/txvalidationcache_tests.cpp |
 | 593356 | test/uint256_tests.cpp |
-| 896684 | test/util_tests.cpp |
+| 807744 | test/util_tests.cpp |
 | 594316 | test/validation_block_tests.cpp |
 | 575288 | test/versionbits_tests.cpp |
 | 564908 | wallet/test/db_tests.cpp |
@@ -77,7 +79,7 @@
 | 585204 | wallet/test/wallet_crypto_tests.cpp |
 | 876104 | wallet/test/coinselector_tests.cpp |
 | 674376 | wallet/test/init_tests.cpp |
-| 845440 | interfaces/wallet.cpp |
+| 720816 | interfaces/wallet.cpp |
 | 479572 | wallet/coincontrol.cpp |
 | 451484 | wallet/crypter.cpp |
 | 511860 | wallet/db.cpp |
@@ -86,8 +88,8 @@
 | 556188 | wallet/load.cpp |
 | 475780 | wallet/psbtwallet.cpp |
 | 861792 | wallet/rpcdump.cpp |
-| 1332236 | wallet/rpcwallet.cpp |
-| 1281028 | wallet/wallet.cpp |
+| 1301264 | wallet/rpcwallet.cpp |
+| 1255100 | wallet/wallet.cpp |
 | 762568 | wallet/walletdb.cpp |
 | 263872 | wallet/walletutil.cpp |
 | 330020 | wallet/coinselection.cpp |
@@ -124,7 +126,7 @@
 | 676636 | rpc/mining.cpp |
 | 509404 | rpc/misc.cpp |
 | 470480 | rpc/net.cpp |
-| 840408 | rpc/rawtransaction.cpp |
+| 838908 | rpc/rawtransaction.cpp |
 | 538172 | rpc/server.cpp |
 | 418808 | script/sigcache.cpp |
 | 33152 | shutdown.cpp |
@@ -168,7 +170,7 @@
 | 224324 | warnings.cpp |
 | 124168 | support/lockedpool.cpp |
 | 235096 | chainparamsbase.cpp |
-| 87436 | clientversion.cpp |
+| 87688 | clientversion.cpp |
 | 25696 | compat/glibc_sanity.cpp |
 | 68020 | compat/glibcxx_sanity.cpp |
 | 17080 | compat/strnlen.cpp |
@@ -220,10 +222,10 @@
 | 380596 | crypto/sha256_sse41.cpp |
 | 370976 | crypto/sha256_avx2.cpp |
 | 118640 | crypto/sha256_shani.cpp |
-| 2664 | lib/univalue.cpp |
+| 2648 | lib/univalue.cpp |
 | 2644 | lib/univalue_get.cpp |
-| 2636 | lib/univalue_read.cpp |
-| 2672 | lib/univalue_write.cpp |
+| 2692 | lib/univalue_read.cpp |
+| 2704 | lib/univalue_write.cpp |
 | 80248 | leveldb/db/builder.cc |
 | 69432 | leveldb/db/c.cc |
 | 62316 | leveldb/db/dbformat.cc |
@@ -265,5 +267,5 @@
 | 54976 | leveldb/port/port_posix.cc |
 | 58672 | leveldb/port/port_posix_sse.cc |
 | 104404 | leveldb/helpers/memenv/memenv.cc |
-| 15144 | src/secp256k1.c |
-| 991952 | -pthread |
+| 15184 | src/secp256k1.c |
+| 993432 | -pthread |
  6. gmaxwell commented at 1:57 AM on May 24, 2019: contributor

    Concept ACK splitting tests up to reduce compile time memory usage.

    If I'm not mistaken, script_tests.cpp is the peak?

    -| 1422800 | test/script_tests.cpp | +| 1308364 | test/script_tests.cpp |

    This isn't a really huge reduction? Is there a different strategy which might make a more meaningful difference in terms of what systems can compile the tests?

  7. laanwj commented at 10:10 AM on May 29, 2019: member

    This isn't a really huge reduction? Is there a different strategy which might make a more meaningful difference in terms of what systems can compile the tests?

    Most of that is hardcoded JSON data, I think. If parsing large hardcoded data structures are a run-time memory sink, there's are other ways of embedding data by converting directly to a compilation unit like:

    • ld -r -b binary embedded_data.dat -o embedded_data.o
    • objcopy -I binary -O elf64-littleriscv -B riscv --rename-section .data=.rodata embedded_data.dat embedded_data.o (requires knowing architecture)

    I think the first option is most portable (works on every platform with GNU linker, so not OSX). Might still want to have C++ code generation as a fallback as there's no beating that in portability.

    Edit: this article abut embedding binary data is useful too.

  8. MarcoFalke force-pushed on May 29, 2019
  9. DrahtBot added the label Needs rebase on Jun 6, 2019
  10. test: Split large tests into smaller compile units fabf534b2d
  11. MarcoFalke force-pushed on Jun 11, 2019
  12. MarcoFalke commented at 3:21 PM on June 11, 2019: member

    @laanwj I think the JSON doesn't matter much, see

    • test: Compile json to hex separately, at most once #16120

    Rebased and updated RSS diff

    @@ -3,0 +4 @@
+| test/args_tests.cpp | 778888 |
@@ -47 +48 @@
-| test/rpc_tests.cpp | 782820 |
+| test/rpc_tests.cpp | 783044 |
@@ -49,0 +51 @@
+| test/script_eval_tests.cpp | 791044 |
@@ -53 +55 @@
-| test/script_tests.cpp | 1423008 |
+| test/script_tests.cpp | 1308532 |
@@ -68 +70 @@
-| test/util_tests.cpp | 896896 |
+| test/util_tests.cpp | 814368 |

    So there is still no real difference.

  13. MarcoFalke added the label Up for grabs on Jun 11, 2019
  14. DrahtBot removed the label Needs rebase on Jun 11, 2019
  15. DrahtBot commented at 3:05 PM on June 27, 2019: member

    <!--cf906140f33d8803c4a75a2196329ecb-->Needs rebase

  16. DrahtBot added the label Needs rebase on Jun 27, 2019
  17. practicalswift commented at 6:58 PM on June 27, 2019: contributor

    @MarcoFalke You're correct that the absurd memory usage in the case of script_tests.cpp is not due to the inclusion of test/data/script_tests.json.h :-)

    Try dropping the inclusion by using a dummy json_tests::script_tests array instead and you'll see that the drop in memory usage is insignificant.

    With the inclusion dropped this is how memory consumption varies between compiler configurations:

    Configuration Memory usage (max RSS)
    g++ -O0 620 MB
    g++ -O0 -g 697 MB
    g++ -O1 657 MB
    g++ -O1 -g 991 MB
    g++ -O2 752 MB
    g++ -O2 -g 1225 MB
    clang++ -O0 323 MB
    clang++ -O0 -g 348 MB
    clang++ -O1 342 MB
    clang++ -O1 -g 370 MB
    clang++ -O2 383 MB
    clang++ -O2 -g 470 MB

    And these are the relative memory usages:

    Configuration Relative memory usage
    -O0 gcc = 1.9 * clang
    -O0 -g gcc = 2.0 * clang
    -O1 gcc = 1.9 * clang
    -O1 -g gcc = 2.6 * clang
    -O2 gcc = 1.9 * clang
    -O2 -g gcc = 2.6 * clang 
    $ g++ --version | head -1
g++ (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
$ clang++ --version | head -1
clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
  18. fanquake commented at 2:26 AM on July 4, 2019: member

    Closing this, as it's "Up for grabs" and the issue seems to be being addressed in other PRs.

  19. fanquake closed this on Jul 4, 2019
  20. MarcoFalke deleted the branch on Oct 12, 2019
  21. MarcoFalke removed the label Up for grabs on Oct 12, 2019
  22. laanwj removed the label Needs rebase on Oct 24, 2019
  23. DrahtBot locked this on Dec 16, 2021
Contributors
MarcoFalkeDrahtBotgmaxwelllaanwjpracticalswiftfanquake
Labels
Tests
Linked (view graph)
#6658 Building requires >1GB memory
github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-04-17 06:14 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me