Avoid integer overflow in CheckDiskSpace #27235

pull dergoegge wants to merge 2 commits into bitcoin:master from dergoegge:2023-03-system-ub changing 2 files +11 −2
  1. dergoegge commented at 5:33 PM on March 9, 2023: member

    Starting a fresh node with -prune=1 causes an integer overflow to happen in CheckDiskSpace (here) because nPruneTarget is to the max uint64_t value.

     node1 stderr util/system.cpp:138:51: runtime error: unsigned integer overflow: 52428800 + 18446744073709551615 cannot be represented in type 'unsigned long'
        [#0](/bitcoin-bitcoin/0/) 0x564a482b5088 in CheckDiskSpace(fs::path const&, unsigned long) src/./src/util/system.cpp:138:51
        [#1](/bitcoin-bitcoin/1/) 0x564a4728dc59 in AppInitMain(node::NodeContext&, interfaces::BlockAndHeaderTipInfo*) src/./src/init.cpp:1639:14
        [#2](/bitcoin-bitcoin/2/) 0x564a47256e6a in AppInit(node::NodeContext&, int, char**) src/./src/bitcoind.cpp:221:43
        [#3](/bitcoin-bitcoin/3/) 0x564a47256087 in main src/./src/bitcoind.cpp:265:13
        [#4](/bitcoin-bitcoin/4/) 0x7fcb7cbffd8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 69389d485a9793dbe873f0ea2c93e02efaa9aa3d)
        [#5](/bitcoin-bitcoin/5/) 0x7fcb7cbffe3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: 69389d485a9793dbe873f0ea2c93e02efaa9aa3d)
        [#6](/bitcoin-bitcoin/6/) 0x564a471957f4 in _start (/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/bitcoind+0xca07f4) (BuildId: 035cb22302d37317a630900a15a26ecb326d395c)
    SUMMARY: UndefinedBehaviorSanitizer: unsigned-integer-overflow util/system.cpp:138:51 in 
    

    I think side stepping the overflow for this specific case, is better than adding an exception to the UB suppresions file.

  2. DrahtBot commented at 5:33 PM on March 9, 2023: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK MarcoFalke, john-moffett

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

  3. martinus commented at 5:42 PM on March 9, 2023: contributor

    How about setting nPruneTarget to something smaller, like std::numeric_limits<uint64_t>::max() / 2? That's still large enough

  4. in src/util/system.cpp:137 in e3ec6d7d49 outdated
     133 | @@ -134,6 +134,10 @@ bool CheckDiskSpace(const fs::path& dir, uint64_t additional_bytes)
     134 |  {
     135 |      constexpr uint64_t min_disk_space = 52428800; // 50 MiB
     136 |  
     137 | +    if (additional_bytes == std::numeric_limits<uint64_t>::max()) {
    


    vasild commented at 5:54 PM on March 9, 2023:

    A few issues:

    1. This check uses the knowledge that the special magic constant std::numeric_limits<uint64_t>::max() is returned by GetPruneTarget() and that in such cases additional_bytes does not make sense. That's not for CheckDiskSpace() to know - the caller should choose a better additional_bytes when calling CheckDiskSpace().

    2. This magic constant is used in two other places - when setting nPruneTarget and in LoadChainstate(). It is better to give it a name instead of repeating it for a 3rd time (e.g. PRUNE_TARGET_UNLIMITED);

    3. In the case of prune=1 I think we should use additional_bytes=chainparams.AssumedBlockchainSize() * 1024 * 1024 * 1024, not 0.

    To address all three what about introducing

    BlockManager::IsPruneModeAutomatic()
    {
        return fPruneMode && nPruneTarget != PRUNE_TARGET_UNLIMITED;
    }
    

    and then

    diff --git i/src/init.cpp w/src/init.cpp
    index f296c16208..3336b66c06 100644
    --- i/src/init.cpp
    +++ w/src/init.cpp
    @@ -1628,13 +1628,13 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
     
         int chain_active_height = WITH_LOCK(cs_main, return chainman.ActiveChain().Height());
     
         // On first startup, warn on low block storage space
         if (!fReindex && !fReindexChainState && chain_active_height <= 1) {
             uint64_t additional_bytes_needed{
    -            chainman.m_blockman.IsPruneMode() ?
    +            chainman.m_blockman.IsPruneModeAutomatic() ?
                     chainman.m_blockman.GetPruneTarget() :
                     chainparams.AssumedBlockchainSize() * 1024 * 1024 * 1024};
     
             if (!CheckDiskSpace(args.GetBlocksDirPath(), additional_bytes_needed)) {
                 InitWarning(strprintf(_(
                         "Disk space for %s may not accommodate the block files. " \
    diff --git i/src/node/chainstate.cpp w/src/node/chainstate.cpp
    index 4741c4c421..3dc24c438c 100644
    --- i/src/node/chainstate.cpp
    +++ w/src/node/chainstate.cpp
    @@ -41,13 +41,13 @@ ChainstateLoadResult LoadChainstate(ChainstateManager& chainman, const CacheSize
             LogPrintf("Validating signatures for all blocks.\n");
         }
         LogPrintf("Setting nMinimumChainWork=%s\n", chainman.MinimumChainWork().GetHex());
         if (chainman.MinimumChainWork() < UintToArith256(chainman.GetConsensus().nMinimumChainWork)) {
             LogPrintf("Warning: nMinimumChainWork set below default value of %s\n", chainman.GetConsensus().nMinimumChainWork.GetHex());
         }
    -    if (chainman.m_blockman.GetPruneTarget() == std::numeric_limits<uint64_t>::max()) {
    +    if (!chainman.m_blockman.IsPruneModeAutomatic()) {
             LogPrintf("Block pruning enabled.  Use RPC call pruneblockchain(height) to manually prune block and undo files.\n");
         } else if (chainman.m_blockman.GetPruneTarget()) {
             LogPrintf("Prune configured to target %u MiB on disk for block and undo files.\n", chainman.m_blockman.GetPruneTarget() / 1024 / 1024);
         }
     
         LOCK(cs_main);
    diff --git i/src/rpc/blockchain.cpp w/src/rpc/blockchain.cpp
    index 8bee066ab8..c20b312e3d 100644
    --- i/src/rpc/blockchain.cpp
    +++ w/src/rpc/blockchain.cpp
    @@ -1269,13 +1269,13 @@ RPCHelpMan getblockchaininfo()
         obj.pushKV("size_on_disk", chainman.m_blockman.CalculateCurrentUsage());
         obj.pushKV("pruned", chainman.m_blockman.IsPruneMode());
         if (chainman.m_blockman.IsPruneMode()) {
             obj.pushKV("pruneheight", chainman.m_blockman.GetFirstStoredBlock(tip)->nHeight);
     
             // if 0, execution bypasses the whole if block.
    -        bool automatic_pruning{args.GetIntArg("-prune", 0) != 1};
    +        const bool automatic_pruning{chainman.m_blockman.IsPruneModeAutomatic()};
             obj.pushKV("automatic_pruning",  automatic_pruning);
             if (automatic_pruning) {
                 obj.pushKV("prune_target_size", chainman.m_blockman.GetPruneTarget());
             }
         }
    
  5. dergoegge marked this as a draft on Mar 9, 2023
  6. in src/util/system.cpp:138 in e3ec6d7d49 outdated
     133 | @@ -134,6 +134,10 @@ bool CheckDiskSpace(const fs::path& dir, uint64_t additional_bytes)
     134 |  {
     135 |      constexpr uint64_t min_disk_space = 52428800; // 50 MiB
     136 |  
     137 | +    if (additional_bytes == std::numeric_limits<uint64_t>::max()) {
     138 | +        additional_bytes = 0;
    


    maflcko commented at 10:42 AM on March 10, 2023:

    Seems an odd way to write this when you can just use SaturatingAdd?


    vasild commented at 11:08 AM on March 10, 2023:

    Do you mean SaturatingAdd(min_disk_space, additional_bytes)? If yes, then in this case it would end up with UINT64_MAX and this function will always report that there is not enough disk space (because that number is so big, nobody has so much disk space).

    In this case, I think neither 0 or UINT64_MAX is suitable. In manual pruning mode, I guess, it makes sense to assume that the user will delete some blocks but not much and thus the required disk space is as if in non-pruned mode - AssumedBlockchainSize(). That would better be handled by the caller.


    maflcko commented at 11:16 AM on March 10, 2023:

    Ah, correct. Sorry for my wrong review comment.


    maflcko commented at 11:25 AM on March 10, 2023:

    In that case, additional_bytes_needed can be clamped by AssumedBlockchainSize?

    [Or maybe by twice the AssumedBlockchainSize to accommodate for an imaginary user that just bought a 1TB drive (intending to use it for a few years) and set pruning to 999GB, but doesn't have that much space available?]

  7. maflcko commented at 10:43 AM on March 10, 2023: member

    https://cirrus-ci.com/task/5059018708746240

    Not sure about referring to an URL that will auto-delete in 90 days. It would be better to at least copy-paste the relevant lines, or even better, add a test case.

  8. vasild commented at 10:59 AM on March 10, 2023: contributor

    Just for reference, like @MarcoFalke pointed above:

     node1 stderr util/system.cpp:138:51: runtime error: unsigned integer overflow: 52428800 + 18446744073709551615 cannot be represented in type 'unsigned long'
        [#0](/bitcoin-bitcoin/0/) 0x564a482b5088 in CheckDiskSpace(fs::path const&, unsigned long) src/./src/util/system.cpp:138:51
        [#1](/bitcoin-bitcoin/1/) 0x564a4728dc59 in AppInitMain(node::NodeContext&, interfaces::BlockAndHeaderTipInfo*) src/./src/init.cpp:1639:14
        [#2](/bitcoin-bitcoin/2/) 0x564a47256e6a in AppInit(node::NodeContext&, int, char**) src/./src/bitcoind.cpp:221:43
        [#3](/bitcoin-bitcoin/3/) 0x564a47256087 in main src/./src/bitcoind.cpp:265:13
        [#4](/bitcoin-bitcoin/4/) 0x7fcb7cbffd8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 69389d485a9793dbe873f0ea2c93e02efaa9aa3d)
        [#5](/bitcoin-bitcoin/5/) 0x7fcb7cbffe3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: 69389d485a9793dbe873f0ea2c93e02efaa9aa3d)
        [#6](/bitcoin-bitcoin/6/) 0x564a471957f4 in _start (/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/bitcoind+0xca07f4) (BuildId: 035cb22302d37317a630900a15a26ecb326d395c)
    SUMMARY: UndefinedBehaviorSanitizer: unsigned-integer-overflow util/system.cpp:138:51 in 
    
  9. dergoegge force-pushed on Mar 10, 2023
  10. dergoegge marked this as ready for review on Mar 10, 2023
  11. in src/init.cpp:1634 in 4164b1f18f outdated
    1630 | @@ -1631,10 +1631,11 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
    1631 |  
    1632 |      // On first startup, warn on low block storage space
    1633 |      if (!fReindex && !fReindexChainState && chain_active_height <= 1) {
    1634 | +        uint64_t assumed_chain_size{chainparams.AssumedBlockchainSize() * 1024 * 1024 * 1024};
    


    maflcko commented at 2:48 PM on March 10, 2023:
            uint64_t assumed_chain_bytes{chainparams.AssumedBlockchainSize() * 1024 * 1024 * 1024};
    

    nit: Instead of "size", use "bytes", like in other places, for example additional_bytes_needed?

  12. [util] Avoid integer overflow in CheckDiskSpace 4517419628
  13. dergoegge force-pushed on Mar 10, 2023
  14. maflcko commented at 2:51 PM on March 10, 2023: member

    Should be trivial to add a test, no?

  15. maflcko commented at 3:27 PM on March 10, 2023: member

    Introduced in 6630a1e8448c633e4abaa8f5903f11cac6f433a7 so I guess no backport needed?

  16. dergoegge commented at 9:46 AM on March 13, 2023: member

    Should be trivial to add a test, no?

    • I don't want to add a new functional test just for this.
    • I tried to add a unit test but all our testing setups don't use the actual init logic.
    • Maybe it can be added to one of the existing functional tests that do pruning, but I don't know which one. I tried but it always ends up being a bigger change to the test than I think would be justified for this trivial change.

    If you know an easy way to add a test let me know, happy to add it.

  17. dergoegge commented at 10:21 AM on March 13, 2023: member

    How about setting nPruneTarget to something smaller, like std::numeric_limits<uint64_t>::max() / 2? That's still large enough @martinus That wouldn't quite work because CheckDiskSpace would return false then, given that std::numeric_limits<uint64_t>::max() / 2 + 50MB would likely not be available in free disk space on any system (https://github.com/bitcoin/bitcoin/blob/1884b71b1dc1fe463a2f00e61bcc56e4720cbc69/src/util/system.cpp#LL138). Currently the overflow works out because it just ends up checking that 50MB are available.

  18. maflcko commented at 11:10 AM on March 13, 2023: member

    All you need to do to add a test is adding a call to start a node with the prune=1 arg. For example:

    diff --git a/test/functional/rpc_blockchain.py b/test/functional/rpc_blockchain.py
    index 7a0cedb1f5..8e086cf268 100755
    --- a/test/functional/rpc_blockchain.py
    +++ b/test/functional/rpc_blockchain.py
    @@ -69,6 +69,7 @@ class BlockchainTest(BitcoinTestFramework):
     
         def run_test(self):
             self.wallet = MiniWallet(self.nodes[0])
    +        self._test_prune_disk_space()
             self.mine_chain()
             self._test_max_future_block_time()
             self.restart_node(
    @@ -100,6 +101,20 @@ class BlockchainTest(BitcoinTestFramework):
                 self.generate(self.wallet, 1)
             assert_equal(self.nodes[0].getblockchaininfo()['blocks'], HEIGHT)
     
    +    def _test_prune_disk_space(self):
    +        self.stop_node(0)
    +        self.log.info("Avoid integer overflow")  
    +        from test_framework.test_node import ErrorMatch 
    +        self.nodes[0].assert_start_raises_init_error(
    +            extra_args=["-prune=1"],
    +            match=ErrorMatch.PARTIAL_REGEX,
    +            expected_msg="runtime error: unsigned integer overflow: 52428800 \+ 18446744073709551615 cannot be represented in type 'unsigned long'",
    +        )
    +        self.log.info("Avoid warning when assumed chain size is enough")  
    +        self.start_node(0, extra_args=["-prune=123456789"])
    +        self.stop_node(0, expected_stderr="may not accommodate the block files. Approximately 0 GB of data will be stored in this directory.")
    +        assert False
    +
         def _test_max_future_block_time(self):
             self.stop_node(0)
             self.log.info("A block tip of more than MAX_FUTURE_BLOCK_TIME in the future raises an error")
    
  19. in test/functional/rpc_blockchain.py:105 in 7225dcbb15 outdated
     100 | @@ -100,6 +101,11 @@ def mine_chain(self):
     101 |              self.generate(self.wallet, 1)
     102 |          assert_equal(self.nodes[0].getblockchaininfo()['blocks'], HEIGHT)
     103 |  
     104 | +    def _test_prune_disk_space(self):
     105 | +        self.log.info("Test that a manually pruned node can start up")
    


    maflcko commented at 12:01 PM on March 13, 2023:
            self.log.info("Test that a manually pruned node does not run into integer overflow on first start")
    

    nit: Not sure if more context is helpful. Otherwise, I suspect this test will be removed in the future due to being redundant.

  20. in test/functional/rpc_blockchain.py:110 in 7225dcbb15 outdated
     100 | @@ -100,6 +101,11 @@ def mine_chain(self):
     101 |              self.generate(self.wallet, 1)
     102 |          assert_equal(self.nodes[0].getblockchaininfo()['blocks'], HEIGHT)
     103 |  
     104 | +    def _test_prune_disk_space(self):
     105 | +        self.log.info("Test that a manually pruned node can start up")
     106 | +        self.stop_node(0)
     107 | +        self.start_node(0, ["-prune=1"])
     108 | +
    


    maflcko commented at 12:02 PM on March 13, 2023:

    Can also add my test to check that no false warning will be issued anymore? Integer overflow isn't the only issue you are fixing here.

  21. maflcko approved
  22. maflcko commented at 12:02 PM on March 13, 2023: member

    lgtm

  23. [test] Add manual prune startup test case 05eeba2c5f
  24. dergoegge force-pushed on Mar 13, 2023
  25. in test/functional/rpc_blockchain.py:108 in 05eeba2c5f
     100 | @@ -100,6 +101,13 @@ def mine_chain(self):
     101 |              self.generate(self.wallet, 1)
     102 |          assert_equal(self.nodes[0].getblockchaininfo()['blocks'], HEIGHT)
     103 |  
     104 | +    def _test_prune_disk_space(self):
     105 | +        self.log.info("Test that a manually pruned node does not run into "
     106 | +                      "integer overflow on first start up")
     107 | +        self.restart_node(0, extra_args=["-prune=1"])
     108 | +        self.log.info("Avoid warning when assumed chain size is enough")
    


    maflcko commented at 12:27 PM on March 13, 2023:
            self.log.info("Test that no warning appears when assumed chain size fits on disk, but the prune target would not")
    

    nit (if you retouch)

  26. maflcko approved
  27. maflcko commented at 1:31 PM on March 13, 2023: member

    ACK 05eeba2c5fb312e0e6a730b01eb7d1b422d75dbb 🥝

    <details><summary>Show signature</summary>

    Signature:

    untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    trusted comment: ACK 05eeba2c5fb312e0e6a730b01eb7d1b422d75dbb 🥝
    ldqnydo8QCYst0rz5DHKllo5bSa1wtQwO0yxrHDUxZFuBdkdTde53ru7wlcgYhRrGK2WIJyKmNEgbQMF7paMCg==
    

    </details>

  28. john-moffett approved
  29. john-moffett commented at 3:19 PM on March 13, 2023: contributor

    ACK 05eeba2c5fb312e0e6a730b01eb7d1b422d75dbb

  30. glozow merged this on Mar 13, 2023
  31. glozow closed this on Mar 13, 2023

  32. sidhujag referenced this in commit 13db9a4b18 on Mar 14, 2023
  33. sidhujag referenced this in commit ad18c043bb on Mar 15, 2023
  34. vasild commented at 1:27 PM on March 16, 2023: contributor

    ACK 05eeba2c5fb312e0e6a730b01eb7d1b422d75dbb

  35. bitcoin locked this on Mar 15, 2024

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-04-13 21:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me