build: Add configure option –enable-lowmem to allow for building in low-memory environments #16331

pull practicalswift wants to merge 1 commits into bitcoin:master from practicalswift:low-mem changing 1 files +11 −0
  1. practicalswift commented at 11:13 am on July 3, 2019: contributor

    Addresses issue #6658 (“Building requires >1GB memory”). Alternative to PR #16312.

    With --enable-lowmem the compile-time minimum memory requirement drops from 1.4 GB (1409 GB) to 1.0 GB (1046 MB).

    Additionally the total compile time for a full build drops ~11 percent.

    Background

    The aburd memory usage for script_tests is due to how the unit test script_build is handled by gcc -g -fvar-tracking (note: -fvar-tracking is a subset of -O2).

    Compile-time memory use, hall of shame (before)

    # File Max memory usage (RSS)
    1 test/test_test_bitcoin-script_tests.o 1409 MB
    2 wallet/libbitcoin_wallet_a-wallet.o 1178 MB
    3 wallet/libbitcoin_wallet_a-rpcwallet.o 1058 MB
    4 libbitcoin_server_a-validation.o 950 MB
    5 test/test_test_bitcoin-util_tests.o 902 MB
    … … …
    455 compat/libbitcoin_util_a-strnlen.o 18 MB

    Accumulated memory use during a full build (455 compiler invocations): 141 GB (max RSS)

    Compile-time memory use, hall of shame (after)

    # File Max memory usage (RSS)
    1 wallet/libbitcoin_wallet_a-wallet.o 1046 MB
    2 test/test_test_bitcoin-script_tests.o 925 MB
    3 wallet/test/test_test_bitcoin-wallet_tests.o 893 MB
    4 wallet/libbitcoin_wallet_a-rpcwallet.o 878 MB
    5 wallet/test/test_test_bitcoin-coinselector_tests.o 877 MB
    … … …
    455 compat/libbitcoin_util_a-strnlen.o 18 MB

    Accumulated memory use during a full build (455 compiler invocations): 135 GB (max RSS)

    Compile-time memory use before and after this PR

    Comparing max memory usage (RSS) between old revision (branch master) and new revision (branch low-mem):

    File Old New Diff Percent
    bench/bench_bench_bitcoin-bench.o 619 MB 586 MB -33 MB -5 %
    bench/bench_bench_bitcoin-checkblock.o 899 MB 638 MB -261 MB -29 %
    bench/bench_bench_bitcoin-verify_script.o 218 MB 205 MB -13 MB -6 %
    bench/bench_bitcoin 440 MB 359 MB -80 MB -18 %
    bitcoin-wallet 180 MB 148 MB -32 MB -18 %
    bitcoin_cli-bitcoin-cli.o 322 MB 294 MB -29 MB -9 %
    bitcoind 389 MB 314 MB -74 MB -19 %
    crypto/crypto_libbitcoin_crypto_avx2_a-sha256_avx2.o 221 MB 160 MB -61 MB -28 %
    crypto/crypto_libbitcoin_crypto_base_a-ripemd160.o 58 MB 37 MB -20 MB -35 %
    crypto/crypto_libbitcoin_crypto_base_a-sha256.o 116 MB 100 MB -16 MB -14 %
    crypto/crypto_libbitcoin_crypto_sse41_a-sha256_sse41.o 230 MB 177 MB -53 MB -23 %
    crypto/ripemd160.cpp 58 MB 38 MB -20 MB -35 %
    crypto/sha256.cpp 115 MB 99 MB -16 MB -14 %
    index/libbitcoin_server_a-base.o 487 MB 457 MB -30 MB -6 %
    interfaces/libbitcoin_wallet_a-wallet.o 715 MB 674 MB -41 MB -6 %
    leveldb/db/leveldb_libleveldb_a-db_impl.o 172 MB 156 MB -16 MB -9 %
    leveldb/db/leveldb_libleveldb_a-repair.o 134 MB 121 MB -13 MB -10 %
    leveldb/db/leveldb_libleveldb_a-version_set.o 170 MB 152 MB -18 MB -11 %
    libbitcoin_common_a-chainparams.o 377 MB 354 MB -23 MB -6 %
    libbitcoin_common_a-core_write.o 370 MB 337 MB -33 MB -9 %
    libbitcoin_common_a-netaddress.o 128 MB 117 MB -11 MB -9 %
    libbitcoin_common_a-netbase.o 266 MB 244 MB -22 MB -8 %
    libbitcoin_common_a-psbt.o 274 MB 250 MB -24 MB -9 %
    libbitcoin_server_a-addrman.o 273 MB 258 MB -15 MB -5 %
    libbitcoin_server_a-dbwrapper.o 280 MB 258 MB -22 MB -8 %
    libbitcoin_server_a-httprpc.o 398 MB 372 MB -26 MB -7 %
    libbitcoin_server_a-net.o 608 MB 560 MB -48 MB -8 %
    libbitcoin_server_a-net_processing.o 860 MB 807 MB -54 MB -6 %
    libbitcoin_server_a-timedata.o 243 MB 230 MB -12 MB -5 %
    libbitcoin_server_a-torcontrol.o 597 MB 562 MB -35 MB -6 %
    libbitcoin_server_a-txdb.o 442 MB 413 MB -30 MB -7 %
    libbitcoin_server_a-txmempool.o 595 MB 558 MB -36 MB -6 %
    libbitcoin_server_a-ui_interface.o 509 MB 459 MB -50 MB -10 %
    libbitcoin_server_a-validation.o 951 MB 817 MB -134 MB -14 %
    policy/libbitcoin_server_a-fees.o 484 MB 459 MB -25 MB -5 %
    primitives/libbitcoin_consensus_a-transaction.o 150 MB 135 MB -15 MB -10 %
    primitives/transaction.cpp 150 MB 135 MB -16 MB -10 %
    qt/bitcoin-qt 568 MB 482 MB -86 MB -15 %
    qt/test/test_bitcoin-qt 622 MB 531 MB -91 MB -15 %
    rpc/libbitcoin_common_a-rawtransaction_util.o 424 MB 395 MB -29 MB -7 %
    rpc/libbitcoin_common_a-util.o 414 MB 388 MB -26 MB -6 %
    rpc/libbitcoin_server_a-blockchain.o 741 MB 636 MB -104 MB -14 %
    rpc/libbitcoin_server_a-mining.o 599 MB 556 MB -43 MB -7 %
    rpc/libbitcoin_server_a-misc.o 472 MB 440 MB -32 MB -7 %
    rpc/libbitcoin_server_a-net.o 545 MB 513 MB -33 MB -6 %
    rpc/libbitcoin_server_a-rawtransaction.o 726 MB 638 MB -87 MB -12 %
    rpc/libbitcoin_server_a-server.o 532 MB 500 MB -32 MB -6 %
    rpc/libbitcoin_util_a-protocol.o 243 MB 228 MB -16 MB -6 %
    script/bitcoinconsensus.cpp 139 MB 127 MB -11 MB -8 %
    script/interpreter.cpp 194 MB 167 MB -27 MB -14 %
    script/libbitcoin_common_a-descriptor.o 459 MB 431 MB -27 MB -6 %
    script/libbitcoin_common_a-sign.o 296 MB 272 MB -25 MB -8 %
    script/libbitcoin_common_a-standard.o 246 MB 230 MB -16 MB -6 %
    script/libbitcoin_consensus_a-bitcoinconsensus.o 138 MB 127 MB -11 MB -8 %
    script/libbitcoin_consensus_a-interpreter.o 196 MB 168 MB -28 MB -14 %
    test/test_bitcoin 747 MB 604 MB -143 MB -19 %
    test/test_test_bitcoin-arith_uint256_tests.o 675 MB 629 MB -46 MB -7 %
    test/test_test_bitcoin-blockfilter_tests.o 584 MB 548 MB -36 MB -6 %
    test/test_test_bitcoin-bloom_tests.o 626 MB 586 MB -40 MB -6 %
    test/test_test_bitcoin-crypto_tests.o 562 MB 527 MB -35 MB -6 %
    test/test_test_bitcoin-descriptor_tests.o 615 MB 570 MB -46 MB -7 %
    test/test_test_bitcoin-key_io_tests.o 574 MB 544 MB -31 MB -5 %
    test/test_test_bitcoin-mempool_tests.o 663 MB 630 MB -33 MB -5 %
    test/test_test_bitcoin-miner_tests.o 636 MB 594 MB -42 MB -7 %
    test/test_test_bitcoin-netbase_tests.o 636 MB 601 MB -35 MB -5 %
    test/test_test_bitcoin-rpc_tests.o 783 MB 732 MB -51 MB -7 %
    test/test_test_bitcoin-script_tests.o 1410 MB 925 MB -484 MB -34 %
    test/test_test_bitcoin-sighash_tests.o 733 MB 570 MB -163 MB -22 %
    test/test_test_bitcoin-torcontrol_tests.o 569 MB 527 MB -42 MB -7 %
    test/test_test_bitcoin-transaction_tests.o 802 MB 707 MB -95 MB -12 %
    test/test_test_bitcoin-uint256_tests.o 563 MB 533 MB -30 MB -5 %
    test/test_test_bitcoin-util_tests.o 903 MB 824 MB -79 MB -9 %
    wallet/libbitcoin_wallet_a-coinselection.o 294 MB 275 MB -19 MB -6 %
    wallet/libbitcoin_wallet_a-db.o 454 MB 424 MB -30 MB -7 %
    wallet/libbitcoin_wallet_a-rpcwallet.o 1059 MB 879 MB -180 MB -17 %
    wallet/libbitcoin_wallet_a-wallet.o 1178 MB 1047 MB -132 MB -11 %
    wallet/libbitcoin_wallet_a-walletdb.o 720 MB 668 MB -52 MB -7 %
    wallet/libbitcoin_wallet_a-walletutil.o 235 MB 222 MB -13 MB -6 %
    wallet/test/test_test_bitcoin-ismine_tests.o 868 MB 806 MB -62 MB -7 %
    wallet/test/test_test_bitcoin-psbt_wallet_tests.o 839 MB 791 MB -49 MB -6 %
    … suppressing 375 unchanged measurements … … … … …
    Average (N=455) 320 MB 305 MB -15 MB -5 %
    Sum (N=455) 145389 MB 138721 MB -6669 MB -5 %
  2. build: Add configure option --enable-lowmem for building in low-memory environments 3b45986b16
  3. fanquake added the label Build system on Jul 3, 2019
  5. laanwj commented at 11:52 am on July 3, 2019: member
    I think the name of this option (--enable-lowmem) is confusing. Given that, I’d assume it lowers run time memory requirements.
  6. practicalswift commented at 11:53 am on July 3, 2019: contributor
    @laanwj Good point! What would be a more appropriate name?
  7. MarcoFalke commented at 12:18 pm on July 3, 2019: member
    What is the point in compiling releases with -g by default. Has this ever helped to debug an issue, considering that they are optimized with -O2?
  8. laanwj commented at 12:40 pm on July 3, 2019: member

    What is the point in compiling releases with -g by default. Has this ever helped to debug an issue, considering that they are optimized with -O2?

    The debug info for releases has been useful for me at times to get backtraces.

  9. practicalswift commented at 1:26 pm on July 3, 2019: contributor

    How is this different from https://github.com/bitcoin/bitcoin/blob/7f985d6c8159f377b685ec2f404f733bdae6ac58/doc/build-unix.md#memory-requirements

    The end result in terms of max RSS for the troublesome files is roughly the same (on my machine at least!), but this PR speeds up the total build time by -11% whereas the linked GCC garbage collection tweak slows down the total build time by several hundred percent.

    The qualitative difference is that this PR reduces the total amount of work performed during a build whereas the linked work-around simply trades memory use for compile time by making the GCC garbage collector hyperactive :-)

  10. MarcoFalke commented at 1:31 pm on July 3, 2019: member
    Thanks for the explanation. Do you think it makes sense to update the documention to mention fvar-tracking instead of adding a “configure wrapper” around fvar-tracking?
  11. practicalswift commented at 3:44 pm on July 5, 2019: contributor

    @laanwj

    What is the point in compiling releases with -g by default. Has this ever helped to debug an issue, considering that they are optimized with -O2?

    The debug info for releases has been useful for me at times to get backtraces.

    Instead of switching the default from -g -O2 to -O2: what about going with -g -O2 -fno-var-tracking as the default? (-fvar-tracking would be opt-in via --enable-debug .)

    That would combine the best of both world: 1.) nice backtraces in releases, 2.) out-of-the-box compile-time memory requirement reduced from 1.4 GB to 1.0 GB and 3.) out-of-the-box compile times reduced by 11%.

    A win win win? :-) @MarcoFalke

    Do you think it makes sense to update the documention to mention fvar-tracking instead of adding a “configure wrapper” around fvar-tracking?

    That is an option. I guess we can choose from one of three levels of user-friendliness:

    1. Build with low memory requirement by default.
    2. Build with low memory requirement if enabled using a configure option discoverable by ./configure --help. Perhaps with the configure script suggesting the option if a low memory environment is detected.
    3. Build with low memory requirement if enabled using a CXXFLAGS environment variable discoverable via the GitHub documentation

    In the issue #6658 I think @gmaxwell makes a good case for low compile-time memory use in also the out-of-the-box case:

    Y’all being overly pedantic. Matt identified a usability problem: User obtains and attempts to compile the software with normal instructions on a common and reasonable platform type, and it crashes.

    If you want to suggest that the docs should have instructions with low mem cflags or whatnot, OKAY, but without that it doesn’t stop being an issue because just us experts can work around it. Our memory usage is kind of obnoxious during compile time IMO, and if we could do some minor tweaks to reduce the peak that would be nice too– but at the very least we should provide some instructions.

    And @TheBlueMatt (earlier) in the same issue:

    This is a default gcc on a relatively recent Debian. I really don’t think the response to “I can’t build Bitcoin Core on a vps with 1GB of memory with default parameters” should be “oh, but if you set these flags you can!”

  12. practicalswift closed this on Aug 11, 2019
  13. practicalswift deleted the branch on Apr 10, 2021
  14. DrahtBot locked this on Aug 18, 2022


practicalswift MarcoFalke laanwj

Labels
Build system

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-12-18 21:12 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me