Avoid signed integer overflow and invalid integer negation when loading malformed mempool.dat files #20383

pull practicalswift wants to merge 5 commits into bitcoin:master from practicalswift:signed-integer-malformed-mempool-dat-and-rpc changing 11 files +66 −24
  1. practicalswift commented at 3:11 PM on November 13, 2020: contributor

    Closes #19278.

    Avoid signed integer overflow when loading malformed mempool.dat files.

    Avoid invalid integer negation when loading malformed mempool.dat files (or when processing prioritisetransaction RPC calls).

    Add note about the valid range of inputs for FormatMoney(...).

    Add test.

    Before this patch:

    $ xxd -p -r > mempool-signed-integer-overflow.dat << "EOF"
    01000000000000003f2d3f3f21f800000000000000000000000000000000
    6d697464657363656e64616e00000001000000ec000000003d6a6c000000
    000000000000ec9bf601000000000000000000ec9b0001000000000001ff
    fffef900000001000000ec0000000000ec9b000001000000000101000100
    00000001000000ec000000003d6a6a000000000000000020ec9b000000fa
    00
    EOF
    $ cp mempool-signed-integer-overflow.dat ~/.bitcoin/regtest/mempool.dat
    $ UBSAN_OPTIONS="print_stacktrace=1:halt_on_error=1:report_error_type=1" src/bitcoind -regtest
    …
    txmempool.cpp:839:15: runtime error: signed integer overflow: -7211388903327006720 + -7211353718954917888 cannot be represented in type 'long'
    …
    
    $ xxd -p -r > mempool-invalid-negation.dat << "EOF"
    0100000000000000002e000000005d2d000d020000000000000000000000
    200000000000000000000080fc0000002d
    EOF
    $ cp mempool-invalid-negation.dat ~/.bitcoin/regtest/mempool.dat
    $ UBSAN_OPTIONS="print_stacktrace=1:halt_on_error=1:report_error_type=1" src/bitcoind -regtest
    …
    util/moneystr.cpp:16:34: runtime error: negation of -9223372036854775808 cannot be represented in type 'CAmount' (aka 'long'); cast to an unsigned type to negate this value to itself
    …
    

    After this patch:

    $ xxd -p -r > mempool-signed-integer-overflow.dat << "EOF"
    01000000000000003f2d3f3f21f800000000000000000000000000000000
    6d697464657363656e64616e00000001000000ec000000003d6a6c000000
    000000000000ec9bf601000000000000000000ec9b0001000000000001ff
    fffef900000001000000ec0000000000ec9b000001000000000101000100
    00000001000000ec000000003d6a6a000000000000000020ec9b000000fa
    00
    EOF
    $ cp mempool-signed-integer-overflow.dat ~/.bitcoin/regtest/mempool.dat
    $ UBSAN_OPTIONS="print_stacktrace=1:halt_on_error=1:report_error_type=1" src/bitcoind -regtest
    …
    2020-11-13T12:34:56Z PrioritiseTransaction(...) failed. Invalid fee delta?
    …
    
    $ xxd -p -r > mempool-invalid-negation.dat << "EOF"
    0100000000000000002e000000005d2d000d020000000000000000000000
    200000000000000000000080fc0000002d
    EOF
    $ cp mempool-invalid-negation.dat ~/.bitcoin/regtest/mempool.dat
    $ UBSAN_OPTIONS="print_stacktrace=1:halt_on_error=1:report_error_type=1" src/bitcoind -regtest
    …
    2020-11-13T12:34:56Z PrioritiseTransaction(...) failed. Invalid fee delta?
    …
    
  2. DrahtBot added the label Build system on Nov 13, 2020
  3. DrahtBot added the label Mempool on Nov 13, 2020
  4. DrahtBot added the label Mining on Nov 13, 2020
  5. DrahtBot added the label RPC/REST/ZMQ on Nov 13, 2020
  6. DrahtBot added the label Utils/log/libs on Nov 13, 2020
  7. DrahtBot added the label Validation on Nov 13, 2020
  8. in src/txmempool.cpp:841 in 38c331b124 outdated
     831 | @@ -831,11 +832,17 @@ TxMempoolInfo CTxMemPool::info(const GenTxid& gtxid) const
     832 |  
     833 |  TxMempoolInfo CTxMemPool::info(const uint256& txid) const { return info(GenTxid{false, txid}); }
     834 |  
     835 | -void CTxMemPool::PrioritiseTransaction(const uint256& hash, const CAmount& nFeeDelta)
     836 | +bool CTxMemPool::PrioritiseTransaction(const uint256& hash, const CAmount& nFeeDelta)
     837 |  {
     838 | +    if (nFeeDelta == std::numeric_limits<CAmount>::min()) {
     839 | +        return false;
     840 | +    }
    


    luke-jr commented at 6:42 PM on November 13, 2020:

    Why?


    practicalswift commented at 9:50 PM on November 13, 2020:

    Otherwise the following invalid integer negation will take place:

    util/moneystr.cpp:16:34: runtime error: negation of -9223372036854775808 cannot be represented in type 'CAmount' (aka 'long'); cast to an unsigned type to negate this value to itself
        [#0](/bitcoin-bitcoin/0/) 0x55b386a14eac in FormatMoney[abi:cxx11](long const&) src/util/moneystr.cpp:16:34
        [#1](/bitcoin-bitcoin/1/) 0x55b3861962f3 in CTxMemPool::PrioritiseTransaction(uint256 const&, long const&) src/txmempool.cpp:861:77
    

    FormatMoney(CAmount n) is not defined for n == std::numeric_limits<CAmount>::min().

    The negation of std::numeric_limits<CAmount>::min() (-9223372036854775808) cannot be represented in the type CAmount (int64_t).

    To reproduce:

    $ git checkout master
    $ make -C src/ bitcoind
    $ xxd -p -r > mempool-invalid-negation.dat << "EOF"
    0100000000000000002e000000005d2d000d020000000000000000000000
    200000000000000000000080fc0000002d
    EOF
    $ cp mempool-invalid-negation.dat ~/.bitcoin/regtest/mempool.dat
    $ UBSAN_OPTIONS="print_stacktrace=1:halt_on_error=1:report_error_type=1" src/bitcoind -regtest
    …
    util/moneystr.cpp:16:34: runtime error: negation of -9223372036854775808 cannot be represented in type 'CAmount' (aka 'long'); cast to an unsigned type to negate this value to itself
    …
    

    practicalswift commented at 4:09 PM on November 19, 2020:

    If the fix to FormayMoney in #20406 is applied we don't have to have the restriction. I think that would be even better :)

  9. DrahtBot commented at 8:11 PM on November 13, 2020: member

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #21007 (bitcoind: Add -daemonwait option to wait for initialization by laanwj)
    • #20406 (util: Avoid invalid integer negation in FormatMoney and ValueFromAmount by practicalswift)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  10. in src/test/fuzz/util.h:29 in 6ab2080bd8 outdated
      24 | @@ -25,6 +25,7 @@
      25 |  #include <test/util/setup_common.h>
      26 |  #include <txmempool.h>
      27 |  #include <uint256.h>
      28 | +#include <util/overflow.h>
    


    luke-jr commented at 9:45 PM on November 13, 2020:

    This needs to be in the parent commit


    practicalswift commented at 10:14 PM on November 13, 2020:

    Good point. Now fixed.

  11. in src/rpc/mining.cpp:476 in 38c331b124 outdated
     471 | @@ -472,7 +472,9 @@ static RPCHelpMan prioritisetransaction()
     472 |          throw JSONRPCError(RPC_INVALID_PARAMETER, "Priority is no longer supported, dummy argument to prioritisetransaction must be 0.");
     473 |      }
     474 |  
     475 | -    EnsureMemPool(request.context).PrioritiseTransaction(hash, nAmount);
     476 | +    if (!EnsureMemPool(request.context).PrioritiseTransaction(hash, nAmount)) {
     477 | +        throw JSONRPCError(RPC_MISC_ERROR, "PrioritiseTransaction failed.");
    


    luke-jr commented at 9:48 PM on November 13, 2020:

    Maybe a bit more verbosity here would be good.


    practicalswift commented at 10:24 PM on November 13, 2020:

    Now using a slightly more verbose error message which hints that an invalid fee_delta argument is the most likely cause. Let me know if you have any other suggestion on wording.

  12. in src/validation.cpp:5031 in 38c331b124 outdated
    5080 | @@ -5081,7 +5081,9 @@ bool LoadMempool(CTxMemPool& pool)
    5081 |  
    5082 |              CAmount amountdelta = nFeeDelta;
    5083 |              if (amountdelta) {
    5084 | -                pool.PrioritiseTransaction(tx->GetHash(), amountdelta);
    5085 | +                if (!pool.PrioritiseTransaction(tx->GetHash(), amountdelta)) {
    5086 | +                    LogPrintf("PrioritiseTransaction(...) failed. Invalid fee delta?\n");
    


    luke-jr commented at 9:50 PM on November 13, 2020:

    This should be literally impossible to hit based on invalid data...?

    AFAICT, the only possibility is if the user manages to call prioritisetransaction RPC before it's loaded the transaction in question


    practicalswift commented at 10:12 PM on November 13, 2020:

    It is possible, but don't take my word for it. Take my PoC for it! :)

    Proof of concept:

    $ git checkout master
    $ make -C src/ bitcoind
    $ xxd -p -r > mempool-signed-integer-overflow.dat << "EOF"
    01000000000000003f2d3f3f21f800000000000000000000000000000000
    6d697464657363656e64616e00000001000000ec000000003d6a6c000000
    000000000000ec9bf601000000000000000000ec9b0001000000000001ff
    fffef900000001000000ec0000000000ec9b000001000000000101000100
    00000001000000ec000000003d6a6a000000000000000020ec9b000000fa
    00
    EOF
    $ cp mempool-signed-integer-overflow.dat ~/.bitcoin/regtest/mempool.dat
    $ UBSAN_OPTIONS="print_stacktrace=1:halt_on_error=1:report_error_type=1" src/bitcoind -regtest
    …
    txmempool.cpp:839:15: runtime error: signed integer overflow: -7211388903327006720 + -7211353718954917888 cannot be represented in type 'long'
        [#0](/bitcoin-bitcoin/0/) 0x556807f763ec in CTxMemPool::PrioritiseTransaction(uint256 const&, long const&) src/txmempool.cpp:839:15
        [#1](/bitcoin-bitcoin/1/) 0x556807ff3c61 in LoadMempool(CTxMemPool&) src/validation.cpp:5084:22
    …
    
  13. practicalswift force-pushed on Nov 13, 2020
  14. practicalswift force-pushed on Nov 13, 2020
  15. in src/validation.cpp:5064 in bde46e38fd outdated
    5113 | @@ -5112,7 +5114,9 @@ bool LoadMempool(CTxMemPool& pool)
    5114 |          file >> mapDeltas;
    5115 |  
    5116 |          for (const auto& i : mapDeltas) {
    5117 | -            pool.PrioritiseTransaction(i.first, i.second);
    5118 | +            if (!pool.PrioritiseTransaction(i.first, i.second)) {
    5119 | +                LogPrintf("PrioritiseTransaction(...) failed. Invalid fee delta?\n");
    


    Crypt-iQ commented at 2:46 PM on November 14, 2020:

    @practicalswift do you have a PoC for this one too? Just curious what the input looks like.


    practicalswift commented at 11:43 AM on November 15, 2020:

    @Crypt-iQ Absolutely, your wish is my command: here you go! :)

    Proof of concept:

    $ git checkout master
    $ make -C src/ bitcoind
    $ xxd -p -r > mempool-poc-for-second-case-on-line-5115.dat << EOF
    0100000000000000010000000000000000000000000000003f0020202020
    202020202020202020ff20ff016e2682a8e0af181ae4ffa4ce2fc42b1e3d
    8ce88019b194a45cd7c476b88ef1e220202020202020802020ff2020
    EOF
    $ cp mempool-poc-for-second-case-on-line-5115.dat ~/.bitcoin/regtest/mempool.dat
    $ UBSAN_OPTIONS="print_stacktrace=1:halt_on_error=1:report_error_type=1" src/bitcoind -regtest
    …
    txmempool.cpp:839:15: runtime error: signed integer overflow: -62769881340174304 + -9214329515250016224 cannot be represented in type 'long'
        [#0](/bitcoin-bitcoin/0/) 0x5652ab2203ec in CTxMemPool::PrioritiseTransaction(uint256 const&, long const&) src/txmempool.cpp:839:15
        [#1](/bitcoin-bitcoin/1/) 0x5652ab29e1ad in LoadMempool(CTxMemPool&) src/validation.cpp:5115:18
    …
    

    :)


    promag commented at 11:11 PM on November 15, 2020:

    Add as a test?


    practicalswift commented at 7:12 AM on November 16, 2020:

    @promag A fuzz test is added in #19259 ("tests: Add fuzzing harness for LoadMempool(...) and DumpMempool(...)"). Review welcome! :)

  16. Crypt-iQ commented at 2:51 PM on November 14, 2020: contributor

    crACK 2c61f90f047097fe372beb8bc8b4d779c66cd505

    Will run my seeds against this patch.

  17. promag commented at 11:12 PM on November 15, 2020: member

    Concept ACK, change looks good to me.

  18. practicalswift commented at 7:13 AM on November 16, 2020: contributor

    @promag A fuzz test is added in #19259 ("tests: Add fuzzing harness for LoadMempool(...) and DumpMempool(...)"). Review welcome! :)

  19. DrahtBot added the label Needs rebase on Nov 30, 2020
  20. practicalswift force-pushed on Nov 30, 2020
  21. DrahtBot removed the label Needs rebase on Nov 30, 2020
  22. in src/txmempool.cpp:845 in 1977f9d946 outdated
     839 | +        return false;
     840 | +    }
     841 |      {
     842 |          LOCK(cs);
     843 |          CAmount &delta = mapDeltas[hash];
     844 | +        if (AdditionOverflow(delta, nFeeDelta)) {
    


    MarcoFalke commented at 9:22 AM on January 14, 2021:

    This check isn't complete. You'd have to check all ancestors and descandants as well. Also, I don't understand why this can't be FeeDeltaRange https://github.com/bitcoin/bitcoin/pull/20089/files#r557249710


    MarcoFalke commented at 7:52 PM on November 2, 2021:

    Implemented an alternative that also checks all ancestors and descendants here: #23418

  23. in src/rpc/mining.cpp:471 in 1977f9d946 outdated
     466 | @@ -467,7 +467,9 @@ static RPCHelpMan prioritisetransaction()
     467 |          throw JSONRPCError(RPC_INVALID_PARAMETER, "Priority is no longer supported, dummy argument to prioritisetransaction must be 0.");
     468 |      }
     469 |  
     470 | -    EnsureMemPool(request.context).PrioritiseTransaction(hash, nAmount);
     471 | +    if (!EnsureMemPool(request.context).PrioritiseTransaction(hash, nAmount)) {
     472 | +        throw JSONRPCError(RPC_MISC_ERROR, "PrioritiseTransaction(...) failed. Invalid fee_delta argument?");
    


    MarcoFalke commented at 9:25 AM on January 14, 2021:

    might be good to update the docs and release notes. Something like

    diff --git a/src/amount.h b/src/amount.h
    index 47968e80b1..9c770a8fe3 100644
    --- a/src/amount.h
    +++ b/src/amount.h
    @@ -24,5 +24,6 @@ static const CAmount COIN = 100000000;
      * */
     static const CAmount MAX_MONEY = 21000000 * COIN;
     inline bool MoneyRange(const CAmount& nValue) { return (nValue >= 0 && nValue <= MAX_MONEY); }
    +inline bool FeeDeltaRange(const CAmount& fee_delta) { return -MAX_MONEY <= fee_delta && fee_delta <= MAX_MONEY; }
     
     #endif //  BITCOIN_AMOUNT_H
    diff --git a/src/rpc/mining.cpp b/src/rpc/mining.cpp
    index 0e96beb6db..181570807c 100644
    --- a/src/rpc/mining.cpp
    +++ b/src/rpc/mining.cpp
    @@ -444,14 +444,14 @@ static RPCHelpMan prioritisetransaction()
                     {
                         {"txid", RPCArg::Type::STR_HEX, RPCArg::Optional::NO, "The transaction id."},
                         {"dummy", RPCArg::Type::NUM, RPCArg::Optional::OMITTED_NAMED_ARG, "API-Compatibility for previous API. Must be zero or null.\n"
    -            "                  DEPRECATED. For forward compatibility use named arguments and omit this parameter."},
    -                    {"fee_delta", RPCArg::Type::NUM, RPCArg::Optional::NO, "The fee value (in satoshis) to add (or subtract, if negative).\n"
    -            "                  Note, that this value is not a fee rate. It is a value to modify absolute fee of the TX.\n"
    -            "                  The fee is not actually paid, only the algorithm for selecting transactions into a block\n"
    -            "                  considers the transaction as it would have paid a higher (or lower) fee."},
    +                              "DEPRECATED. For forward compatibility use named arguments and omit this parameter."},
    +                    {"fee_delta", RPCArg::Type::NUM, RPCArg::Optional::NO, "The fee value (in satoshis) to add (or subtract, if negative). Must be in the range +-" + ToString(MAX_MONEY) + " sat.\n"
    +                              "Note, that this value is not a fee rate. It is a value to modify absolute fee of the TX.\n"
    +                              "The fee is not actually paid, only the algorithm for selecting transactions into a block\n"
    +                              "considers the transaction as it would have paid a higher (or lower) fee."},
                     },
                     RPCResult{
    -                    RPCResult::Type::BOOL, "", "Returns true"},
    +                    RPCResult::Type::BOOL, "", "Returns true or throws on error"},
                     RPCExamples{
                         HelpExampleCli("prioritisetransaction", "\"txid\" 0.0 10000")
                 + HelpExampleRpc("prioritisetransaction", "\"txid\", 0.0, 10000")
    @@ -460,14 +460,17 @@ static RPCHelpMan prioritisetransaction()
     {
         LOCK(cs_main);
     
    -    uint256 hash(ParseHashV(request.params[0], "txid"));
    -    CAmount nAmount = request.params[2].get_int64();
    +    const uint256 hash(ParseHashV(request.params[0], "txid"));
    +    const CAmount nAmount = request.params[2].get_int64();
     
         if (!(request.params[1].isNull() || request.params[1].get_real() == 0)) {
             throw JSONRPCError(RPC_INVALID_PARAMETER, "Priority is no longer supported, dummy argument to prioritisetransaction must be 0.");
         }
     
    -    EnsureMemPool(request.context).PrioritiseTransaction(hash, nAmount);
    +    CTxMemPool& mempool = EnsureMemPool(request.context);
    +    if (!mempool.PrioritiseTransaction(hash, nAmount)) {
    +        throw JSONRPCError(RPC_INVALID_PARAMETER, "Invalid fee_delta passed");
    +    }
         return true;
     },
         };
    
  24. MarcoFalke commented at 9:26 AM on January 14, 2021: member

    If you want to eliminate this issue completely, the diff would be quite invasive. So I am not sure if we want this. Maybe a note "pls don't fuzz your rpc interface in production" is good enough?

  25. practicalswift force-pushed on Jan 14, 2021
  26. practicalswift commented at 2:06 PM on January 15, 2021: contributor

    Maybe a note "pls don't fuzz your rpc interface in production" is good enough?

    I think we should try to make all our interfaces robust against crazy inputs. That includes our RPC interface :)

    Where "robustness" includes being able to handle crazy inputs without triggering ASAN, MSAN, TSAN or UBSAN warnings.

    In other words I think we should strive for handling crazy user input (UI robustness), crazy programmatic input (RPC robustness) and crazy network input (P2P robustness) without triggering sanitizer warnings.

    Note that even if we didn't care at all about robustness against crazy inputs in our RPC interface it is still makes perfect sense to fuzz the RPC interface since it is the broadest possible general "entry point" in to our application. By fuzzing the RPC interface we can reach large parts of our code base and uncover also non-RPC specific bugs.

  27. MarcoFalke commented at 2:16 PM on January 15, 2021: member

    Sure, when it is possible to add a simple check if the input is invalid, but here it requires changing the whole mempool package tracking code

  28. felipsoarez commented at 11:39 PM on January 15, 2021: none

    Concept ACK

  29. DrahtBot added the label Needs rebase on Feb 8, 2021
  30. util: Move AdditionOverflow(...) to new file src/util/overflow.h bbd79180d5
  31. mempool: Avoid signed integer overflow in CTxMemPool::PrioritiseTransaction(...) c4258488c9
  32. util: Add note about the valid range of inputs for FormatMoney(...) 7df8dbf9ba
  33. tests: Test invalid fee delta check 1db76e1d86
  34. mempool: Avoid signed integer overflow in CTxMemPool::PrioritiseTransaction(...) 7244d852d2
  35. practicalswift force-pushed on Feb 11, 2021
  36. DrahtBot removed the label Needs rebase on Feb 11, 2021
  37. in src/amount.h:27 in 7244d852d2
      23 | @@ -24,5 +24,6 @@ static const CAmount COIN = 100000000;
      24 |   * */
      25 |  static const CAmount MAX_MONEY = 21000000 * COIN;
      26 |  inline bool MoneyRange(const CAmount& nValue) { return (nValue >= 0 && nValue <= MAX_MONEY); }
      27 | +inline bool FeeDeltaRange(const CAmount& fee_delta) { return -MAX_MONEY <= fee_delta && fee_delta <= MAX_MONEY; }
    


    luke-jr commented at 4:56 PM on February 28, 2021:

    Fee deltas have no such range limit.


    practicalswift commented at 6:15 PM on February 28, 2021:
  38. practicalswift closed this on Feb 28, 2021

  39. practicalswift deleted the branch on Apr 10, 2021
  40. MarcoFalke referenced this in commit dde7205c57 on Jun 27, 2022
  41. sidhujag referenced this in commit 758481e22d on Jun 27, 2022
  42. DrahtBot locked this on Nov 2, 2022

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-05-02 12:14 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me