Detect transaction problems early during deserialization #36906

pull sipa wants to merge 6 commits into bitcoin:master from sipa:202611_txdeser_failfast changing 16 files +588 −54
  1. sipa commented at 10:08 PM on November 17, 2026: member

    This is split off from #36521 (and not included in #36895), as it's effectively an independent improvement.

    This introduces a number of additional checks that are applied during transaction deserialization, which reduce various worst case deserialization time scenarios:

    • Reject transactions with more txins or more txouts than can possibly fit in a block (applied always)
    • Reject transactions with 0 txins or 0 txouts (applied in P2P code only; there are RPCs like fundrawtransaction that explicitly support transactions with 0 inputs)
    • Count weight accurately during transaction or block deserialization, and bail out as soon as it exceeds 4M (applied in P2P code).

    This is a mild behavior change, as detecting these problems early causes them to be treated as deserialization errors, which in P2P logic cause messages to be ignored instead of peer disconnection.

  2. DrahtBot commented at 10:09 PM on November 17, 2026: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    External sites

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline and AI policy for information on the review process. A summary of reviews will appear here.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

  3. sipa commented at 10:11 PM on November 17, 2026: member

    @ajtowns I considered your suggestion in #36521 (review), but don't like that it's not explicit what the lifetime of the budget counting is. I use a more strongly typed approach here to add mutable state to streams, but still keep the budget object scoped in P2P code. LMK what you think.

  4. in src/net_processing.cpp:5069 in aaea704057
    5062 | @@ -5054,7 +5063,10 @@ void PeerManagerImpl::ProcessMessage(Peer& peer, CNode& pfrom, const std::string
    5063 |          }
    5064 |  
    5065 |          BlockTransactions resp;
    5066 | -        vRecv >> resp;
    5067 | +        // The transactions are part of the block, so stop deserializing as soon as they exceed the weight that the
    5068 | +        // transactions of a valid block can have (not accounting for those of the block that are already known).
    5069 | +        TxWeightBudget budget{MAX_BLOCK_TXS_WEIGHT};
    5070 | +        vRecv >> WithState{budget}(TX_WITH_WITNESS_P2P(resp));
    


    ajtowns commented at 11:51 PM on November 17, 2026:

    @ajtowns I considered your suggestion in #36521 (review), but don't like that it's not explicit what the lifetime of the budget counting is. I use a more strongly typed approach here to add mutable state to streams, but still keep the budget object scoped in P2P code. LMK what you think.

    Hmm, the lifetime of the budget there is just the wrapper, which seems fairly explicit to me?

    So:

    vRecv >> TX_WITH_WITNESS_BLOCK_LIMIT(block_a) >> TX_WITH_WITNESS_BLOCK_LIMT(tx_b);
    

    starts the budget at "(" and ends it at ")", so it would be the same as:

    TxWeightBudget budget{MAX_BLOCK_TXS_WEIGHT};
    vRecv >> WithState{budget}(TX_WITH_WITNESS_P2P(block_a));
    budget = MAX_BLOCK_TXS_WEIGHT;
    vRecv >> WithState{budget}(TX_WITH_WITNESS_P2P(tx_b));
    

    To get the same as:

    vRecv >> WithState{budget}(TX_WITH_WITNESS_P2P(block_a)) >> WithState{budget}(TX_WITH_WITNESS_P2P(tx_b));
    

    you'd need to write:

    const TransactionSerParams TxWitLimit{.allow_witness = true, .reject_empty = false, .max_weight = MAX_BLOCK_TXS_WEIGHT};
    ParamsStream{vrecv, TxWitLimit} << block_a << tx_b;
    

    at which point the budget is scoped to the params stream.


    sipa commented at 9:34 PM on November 18, 2026:

    My concern is/was mostly with something like a change READWRITE(block.vtx); into READWRITE(TX_WITH_WITNESS(block.vtx));, which would introduce a new scope without limiting, or a scope with limiting but a different variable (this exact example might not work, but you see what I mean).

    I've switched to your approach, but added a feature for parameters to declare they cannot be overridden, which would satisfy that concern and doesn't cause any compilation problems.

  5. in src/test/transaction_tests.cpp:1437 in 039f5ce69d
    1430 | @@ -1431,4 +1431,28 @@ BOOST_AUTO_TEST_CASE(tx_malformed_serialization)
    1431 |      BOOST_CHECK_EXCEPTION(bad_flag >> TX_WITH_WITNESS(tx), std::ios_base::failure, HasReason{"Unknown transaction optional data"});
    1432 |  }
    1433 |  
    1434 | +BOOST_AUTO_TEST_CASE(too_many_inputs_or_outputs)
    1435 | +{
    1436 | +    // Deserialization fails as soon as a count of inputs or outputs is read that is more than fit in a block, before
    1437 | +    // reading them (so here, rather than for running out of data), in either format.
    


    maflcko commented at 10:04 AM on November 18, 2026:

    nit in the first commit: It took me two passes to parse the sentence. Maybe split it in two and repeat the verb?

        // reading them. So it fails here, rather than for running out of data, in either format.
    

    sipa commented at 9:29 PM on November 18, 2026:

    Done.

  6. in test/functional/mempool_accept.py:237 in 039f5ce69d
     231 | @@ -232,12 +232,21 @@ def run_test(self):
     232 |  
     233 |          self.log.info('A really large transaction')
     234 |          tx = tx_from_hex(raw_tx_reference)
     235 | +        # Use inputs with a non-empty scriptSig, so that there are not more of them than fit in a block (which would
     236 | +        # fail decoding).
     237 | +        tx.vin[0].scriptSig = CScript([OP_TRUE])
    


    maflcko commented at 10:25 AM on November 18, 2026:

    nit in the first commit: I mostly only wrote this test for fun in b55555da3e25a47f1e7fced7f09d4f0bf8198624. There is no need to retain the code exactly. It would also be fine to just use:

            self.log.info('A really large transaction')
            tx = tx_from_hex(raw_tx_reference)
            assert_equal(len(tx.vin[0].serialize()), 41)
            tx.vin = [tx.vin[0]] * (MAX_BLOCK_WEIGHT // (WITNESS_SCALE_FACTOR * 41))
            self.check_mempool_result(
                result_expected=[{'txid': tx.txid_hex, 'allowed': False, 'reject-reason': 'bad-txns-oversize'}],
                rawtxs=[tx.serialize().hex()],
            )
    

    to keep it brief and in symmetry with the below check.


    sipa commented at 9:29 PM on November 18, 2026:

    Done.

  7. in src/test/blockencodings_tests.cpp:478 in 9de186b527
     473 | +        block_txn.txn = block.vtx;
     474 | +
     475 | +        const auto check{[&](const auto& obj, auto deser) {
     476 | +            const auto data{DataStream{} << obj};
     477 | +            const auto reason = with_input ? "Transaction without outputs" : "Transaction without inputs";
     478 | +            BOOST_CHECK_EXCEPTION(DataStream{data} >> deser, std::ios_base::failure, HasReason{reason});
    


    maflcko commented at 10:53 AM on November 18, 2026:

    nit in 9de186b5271fe3f7a9edad1740d86ea4a5db7998: Avoid the copy and make data mutable or use a SpanReader?

    Also, I think the deser type can be inferred in C++20, to avoid having to type it again:

      const auto check{[&]<typename T>(const T& obj) {
          const auto data{DataStream{} << obj};
          const auto reason = with_input ? "Transaction without outputs" : "Transaction without inputs";
          BOOST_CHECK_EXCEPTION(SpanReader{data} >> T{}, std::ios_base::failure, HasReason{reason});
      }};
    

    sipa commented at 9:30 PM on November 18, 2026:

    Done.

  8. maflcko commented at 11:40 AM on November 18, 2026: member

    partial review

    lgtm aaea7040575171c1d12887fa15e64bc201a3f67c~2 📚

    <details><summary>Show signature</summary>

    Signature:

    untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    trusted comment: lgtm aaea7040575171c1d12887fa15e64bc201a3f67c~2 📚
    nLvjeeAlv+OVV17u/LiU6qKTHXSKQVslEz44pNdPili+F59XLnmtzwm412Oqz9INKt3W2RLaQ3ySHRG1hsCCAA==
    

    </details>

  9. primitives: reject txs with more inputs or outputs than fit in a block
    When deserializing a transaction, read the input and output counts
    first, and fail before reading the inputs or outputs if more of them
    are claimed than can fit in a block, given their minimum size (41 bytes
    for an input, 9 for an output). Such a transaction can never be valid.
    
    This allows us to allocate the vin and vout vectors immediately, rather
    than sequentially in batches of 5 MB as the std::vector deserializer
    does, because that memory becomes bounded: to 2.7 MB for 24390 inputs,
    and 5.3 MB for 111111 outputs. It is a preparation for further changes
    that rely on this structuring of conditionals.
    
    This affects all places where transactions are deserialized. Through
    RPC, such transactions can no longer be decoded. From P2P, a transaction
    or block containing one is ignored like other malformed messages, rather
    than rejected by validation.
    093c7d7c4f
  10. p2p: reject transactions without inputs or outputs when deserializing
    Add a reject_empty flag to TransactionSerParams, which makes
    deserialization fail as soon as a transaction is found to have no
    inputs or no outputs. Such transactions are never valid, but are
    currently fully deserialized (along with the rest of the message) before
    validation rejects them.
    
    This reduces the memory a peer can make us allocate by sending a block
    full of such transactions. At 10 bytes each, about 400000 of them fit in
    a block message, each of which gets its own CTransaction. With this
    change, the smallest transaction that gets deserialized has one input
    and one output, which takes 60 bytes.
    
    The flag is enabled (as TX_WITH_WITNESS_P2P) only where transactions are
    received from P2P peers: tx and block messages, and the transactions in
    cmpctblock and blocktxn messages. Like other malformed messages, these
    are now ignored, rather than, for a block, causing the peer to be
    disconnected, or, for a transaction, being added to the reject filters.
    The functional tests are updated accordingly.
    79ac5541b0
  11. refactor: use the stream's parameters for compact block transactions
    BlockTransactions and PrefilledTransaction (de)serialized their
    transactions with TX_WITH_WITNESS_P2P, overriding any parameters of the
    stream. Use the stream's parameters instead, like CBlock does, and pass
    TX_WITH_WITNESS where they are serialized and TX_WITH_WITNESS_P2P where
    data from peers is deserialized. This does not change behavior
    (serialization only depends on allow_witness), but makes it possible to
    deserialize them with other parameters. Tests that are not about data
    from peers do so, with TX_WITH_WITNESS.
    482aba265c
  12. serialize: add StreamHasParams concept
    Constrain ParamsStream::GetParams<P>() to the parameter types that the
    stream, or a stream it wraps, has. Before, calling it for another type
    failed inside the function body, which cannot be tested for.
    
    With that, add a StreamHasParams<Stream, P> concept, which tells whether
    GetParams<P>() can be called on a stream of type Stream.
    
    This does not change what compiles. Calling GetParams<P>() for a type
    the stream does not have is now reported as an unsatisfied constraint
    at the call site.
    8517aabbb9
  13. serialize: add support for mutable state when (de)serializing with params
    Let serialization parameters come with state: if a params type has a
    State type and an InitState() method, a ParamsStream with such params
    holds a State, initialized with InitState(). (De)serialization code can
    read and modify it through GetState<Params>(), like it reads the params
    through GetParams<Params>(). Whether a stream holds the state of a
    params type can be checked with the StreamHasState concept.
    
    This makes it possible to keep track of something across everything
    that is (de)serialized through one stream, for example to limit the
    total size of what is deserialized, while the params themselves stay
    immutable.
    
    As a copy of a ParamsStream would have a state of its own, make it
    non-copyable.
    
    Giving params to a stream that already has params of that type would
    hide the state which the stream holds for them behind a new one. Let
    params types forbid this by having a NO_OVERRIDE member, which makes
    doing so a compile-time error.
    
    Co-authored-by: Pieter Wuille <bitcoin-dev@wuille.net>
    c0fa7a7fd7
  14. p2p: limit the weight of transactions and blocks while deserializing
    Add an optional weight limit to TransactionSerParams. When it is set,
    the stream's state for these params holds the weight that may still be
    deserialized. It is decremented while transactions are being
    deserialized (charging the minimum weight of inputs and outputs before
    allocating memory for them), causing failure as soon as it is exceeded.
    The limit thus applies to all transactions that are deserialized
    through one stream together, such as those of a block.
    
    As giving TransactionSerParams to a stream that already has them would
    start a new state, with a limit of its own, for what is deserialized
    with them, mark them as NO_OVERRIDE. For the same reason, do not allow
    the deserializing constructor that takes params to be used with such a
    stream.
    
    Set the limit to MAX_BLOCK_WEIGHT minus the header in
    TX_WITH_WITNESS_P2P, which is what tx, block, cmpctblock and blocktxn
    messages are deserialized with: the transactions in a block, cmpctblock
    or blocktxn message are part of one block (for blocktxn, this does not
    account for those of the block that are already known), and a
    transaction that does not fit in a block cannot be valid.
    
    The P2P message size limit allows blocks of up to 4 MB, which is up to
    16M weight, all of which is currently deserialized before validation
    rejects it (unless a transaction has more inputs or outputs than fit in
    a block). With this change, the weight deserialized from a message is
    at most MAX_BLOCK_WEIGHT minus the header.
    
    Messages exceeding the limit are ignored, like other malformed
    messages, rather than causing the peer to be disconnected for sending an
    invalid block.
    
    Co-authored-by: Anthony Towns <aj@erisian.com.au>
    57a26d1a63
  15. sipa force-pushed on Nov 18, 2026
  16. DrahtBot added the label CI failed on Nov 18, 2026
  17. DrahtBot removed the label CI failed on Nov 19, 2026
  18. in src/net_processing.cpp:5112 in 57a26d1a63
    5108 | @@ -5109,7 +5109,7 @@ void PeerManagerImpl::ProcessMessage(Peer& peer, CNode& pfrom, const std::string
    5109 |          }
    5110 |  
    5111 |          std::shared_ptr<CBlock> pblock = std::make_shared<CBlock>();
    5112 | -        vRecv >> TX_WITH_WITNESS(*pblock);
    5113 | +        vRecv >> TX_WITH_WITNESS_P2P(*pblock);
    


    Yudis-bit commented at 1:10 PM on November 19, 2026:

    Can this stall block download when a requested block fails deserialization?

    If an oversized block throws std::ios_base::failure in vRecv >> TX_WITH_WITNESS_P2P, the exception unwinds out of ProcessMessage before RemoveBlockRequest(hash, pfrom.GetId()) runs. Since the peer isn't disconnected, the hash stays in mapBlocksInFlight until the 10-minute download timeout expires.


    sipa commented at 1:50 PM on November 19, 2026:

    That's a fair concern. It's no different from how we treat a peer that never responds to a getdata requests at all, but extending the definition of serialization-invalid means that there are potentially broken (but not malicious) peers out there that aren't disconnected quickly anymore.

    I'll investigate (separate from this PR) whether we can just blanket disconnect any peers that gives us undeserializable messages.


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-11-19 19:51 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me