Keep witness stacks in serialized form in CScriptWitness #36895

pull sipa wants to merge 13 commits into bitcoin:master from sipa:202611_witpack_minimal changing 62 files +631 −241
  1. sipa commented at 7:11 PM on November 16, 2026: member

    This is split off from #36521. This PR focuses on reducing memory usage of transactions in memory, leaving performance improvements for other PRs.

    CScriptWitness stores a witness stack as a std::vector<std::vector<unsigned char>>, which costs a 24-byte vector object and an allocation for every non-empty element. This makes it possible to construct transactions that take a large amount of memory after deserialization.

    This PR first introduces encapsulation for CScriptWitness, hiding its representation. The last commit then changes the representation to be single std::unique_ptr<byte[]> which for non-empty witness stacks stores their serialization prefixed by a small length header. Witnesses can no longer be modified in place; they can only be replaced in their entirety. Access to the individual witness stack elements is done by on-the-fly deserialization instead.

    CScriptWitness shrinks from 24 to 8 bytes, and CTxIn from 112 to 96 bytes. Worst-case allocated size for transactions kept in memory drops from roughly 28 bytes/WU to 1.6 bytes/WU.

    This worsens deserialization and validation time, adding some 50-70 ms per block to the worst case for me. A follow-up #36918 with optimizations improves the worst case deserialization time significantly instead. Other minor degradations:

    • libbitcoinkernel's witness stack element accessor becomes $\mathcal{O}(n)$ in the index of the witness element; using it to iterate a whole witness may therefore become $\mathcal{O}(n^2)$.
    • CScriptWitness::back() is $\mathcal{O}(n)$ in the length of the witness stack.

    Issue reported by furszy and 0xB10C.

  2. script: give CScriptWitness an interface that hides its representation
    Add element access through spans (forward iteration, front(), back(), and
    indexing), conversion to and construction from a vector of elements,
    serialization, and memory usage accounting, so that users no longer need to
    access the stack vector directly. This prepares for a future commit which
    will replace the representation with a more compact & efficient one.
    
    The stack vector itself stays public until the next commits have converted
    its users.
    18403ab754
  3. test: check CScriptWitness against the vector of vectors it represents
    Add a unit test and a fuzz test that compare CScriptWitness's interface
    (accessors, construction, and serialization) with the
    std::vector<std::vector<unsigned char>> it represents. The unit test
    focuses on element counts and sizes around CompactSize encoding
    boundaries.
    49c0f21971
  4. scripted-diff: use CScriptWitness's size(), empty() and back()
    Convert the uses of CScriptWitness::stack that only need the number of
    elements, whether there are any, or the last element, which the witness
    provides itself. Other uses are converted in the next commits.
    
    -BEGIN VERIFY SCRIPT-
    # Lines that need the stack itself (to compute the size of its
    # serialization without the element count, to pass the last element
    # where a vector is expected, or to index all elements in a loop) are
    # left for the manual changes.
    sed -i -E -e '/GetSizeOfCompactSize|TAPROOT_LEAF_TAPSCRIPT|i < (txin\.)?scriptWitness\.stack/b' \
        -e 's/\.stack\.(size|empty|back)\(\)/.\1()/g' \
        $(git grep -l -E '\.stack\.(size|empty|back)\(\)' -- src ':!src/script/miniscript.*' ':!src/script/script.h')
    -END VERIFY SCRIPT-
    a271d6b773
  5. scripted-diff: read and compare CScriptWitness through its interface
    Convert the uses of CScriptWitness::stack that read an element by index,
    bind a const reference to the stack, or compare two stacks, and the
    uses of IsNull().
    
    -BEGIN VERIFY SCRIPT-
    FILES=$(git grep -l -E '\.stack\b' -- src ':!src/script/miniscript.*' ':!src/script/script.h')
    
    # Reading an element by index: the witness has operator[] itself. Writes
    # to elements, and loops indexing all elements (which can be quadratic
    # with indexing that is not constant-time), are left for the manual
    # changes.
    sed -i -E -e '/MAX_STANDARD_P2WSH_STACK_ITEM_SIZE|HexStr\((txin\.)?scriptWitness\.stack\[|\.stack\[[^]]*\]( =|\.resize)|^\s*\+\+/b' \
        -e 's/\.stack\[/[/g' $FILES
    
    # Const references to the stack: refer to the witness itself, which
    # provides the same read-only accessors.
    sed -i -E 's/^(\s*const auto& \w+( = |\{).*)\.stack(\}?;)$/\1\3/' $FILES
    
    # Comparisons of two stacks: compare the witnesses, which are equal
    # exactly when their stacks are.
    sed -i -E '/\.stack [!=]= /s/\.stack\b//g' $FILES
    
    # IsNull(): use empty(), which the witness provides along with size()
    # and its other container-like accessors.
    sed -i -E -e 's/(scriptWitness|final_script_witness)\.IsNull\(\)/\1.empty()/g' \
        -e 's/witness->IsNull\(\)/witness->empty()/g' \
        $(git grep -l -E '(scriptWitness|final_script_witness)\.IsNull\(\)|witness->IsNull\(\)' -- src)
    -END VERIFY SCRIPT-
    9a7fdee51d
  6. scripted-diff: serialize CScriptWitness rather than its stack
    Convert the uses of CScriptWitness::stack that serialize or deserialize
    it, or compute the size of its serialization. The witness itself has the
    same serialization as its stack.
    
    -BEGIN VERIFY SCRIPT-
    sed -i -E -e 's/ (>>|<<) ([][A-Za-z0-9_.]+)\.stack( <<|;)/ \1 \2\3/' \
        -e 's/(SerializeToVector|UnserializeFromVector)\(s, (\w+)\.stack\)/\1(s, \2)/' \
        -e 's/::GetSerializeSize\(([][A-Za-z0-9_.]+)\.stack\)/::GetSerializeSize(\1)/' \
        $(git grep -l -E '\.stack\b' -- src ':!src/script/miniscript.*' ':!src/script/script.h')
    -END VERIFY SCRIPT-
    940d5dc167
  7. scripted-diff: replace witnesses as a whole instead of their stack
    Convert the uses of CScriptWitness::stack (and SetNull()) that clear the
    stack, assign a new one, or push a single element onto an empty one, to
    assigning a new witness. The remaining uses are converted manually in
    the next commits.
    
    -BEGIN VERIFY SCRIPT-
    FILES=$(git grep -l -E '\.stack\b|(scriptWitness|final_script_witness)\.SetNull\(\)' -- src ':!src/script/miniscript.*' ':!src/script/script.h')
    
    # Clearing the stack (SetNull(), clear() or resize(0)): assign an empty
    # witness. A clear() followed by pushing elements in a loop is left for
    # the manual changes.
    sed -i -E -e '/txin\.scriptWitness\.stack\.clear/b' \
        -e 's/(scriptWitness|final_script_witness)\.SetNull\(\)/\1 = {}/' \
        -e 's/(scriptWitness)\.stack\.(clear\(\)|resize\(0\));$/\1 = {};/' $FILES
    
    # Assigning a vector of elements to the stack: assign a witness
    # constructed from it (moving from the vector is pointless, as the
    # witness is constructed from a const reference). An assignment that
    # spells out the vector's type is left for the manual changes.
    sed -i -E -e '/\.stack = std::vector</b' \
        -e 's/(\w+)\.stack = (std::move\((\w+)\)|(.*));$/\1 = CScriptWitness{\3\4};/' $FILES
    
    # Pushing a single element onto an empty witness (in these benchmarks
    # and tests): assign a witness constructed from that element. The one
    # place that pushes onto a non-empty witness (to make a different
    # witness) is left for the manual changes, as are pushes in loops.
    sed -i -E '/ tx\.vin\[0\]\.scriptWitness\.stack\.push_back\(\{5\}\)|Consume/!s/(\w+)\.stack\.push_back\((.*)\);$/\1 = CScriptWitness{{\2}};/' \
        src/bench/blockencodings.cpp src/bench/mempool_eviction.cpp src/bench/mempool_stress.cpp src/bench/rpc_mempool.cpp \
        src/test/fuzz/connect_block.cpp src/test/fuzz/txdownloadman.cpp src/test/orphanage_tests.cpp src/test/txdownload_tests.cpp \
        src/test/validation_block_tests.cpp
    -END VERIFY SCRIPT-
    3c57d948d5
  8. script: let ExecuteWitnessScript take its initial stack by value
    This lets callers move the stack into it, rather than having it be
    copied from a span. This will be used in a future commit.
    fa5d9491b5
  9. script: build the witness execution stack with ToVectors()
    VerifyWitnessProgram used a span over CScriptWitness::stack, popped the
    elements with other roles (the witness script, the annex, the control
    block, and the leaf script) off that span, and copied the remaining
    elements into the initial execution stack.
    
    Instead, convert the witness to a vector of vectors once, using
    ToVectors(), pop those elements off that vector, and move what remains
    into the execution stack.
    d55243249a
  10. refactor: use CScriptWitness's interface where that needs more than a rename
    Convert the remaining non-test uses of CScriptWitness::stack, which need
    more than a substitution.
    
    This includes the range-based for loops over the stack. Iterating over a
    CScriptWitness yields its elements as spans, by value, where the stack's
    elements were vectors bound by reference. The spans refer to memory
    owned by the witness, so they stay valid as long as it is not modified.
    0a82e05e34
  11. test, bench, fuzz: build witness stacks before constructing a CScriptWitness
    Convert the remaining uses of CScriptWitness::stack in tests, benchmarks,
    and fuzz tests, by constructing witnesses from a separately-built vector
    of elements, rather than modifying the stack in place.
    1a0e6223f5
  12. script: remove CScriptWitness's old interface
    All users of CScriptWitness now go through the interface that hides its
    representation. Make the stack vector private, turning the struct into a
    class, and remove IsNull() and SetNull(), which have no callers left.
    
    The representation itself does not change here.
    e6e42947d0
  13. span: add SameSpan
    Add a helper that tells whether two spans refer to the same range of
    memory, rather than whether their contents are equal. A future commit
    uses it to compare iterators that consist of a span.
    2366d35e07
  14. DrahtBot commented at 7:11 PM on November 16, 2026: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    External sites

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline and AI policy for information on the review process.

    Type Reviewers
    ACK instagibbs
    Concept ACK darosior, w0xlt, xyzconstant, brunoerg
    Stale ACK optout21

    If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #36776 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/36776.svg"></sub> (bip352: match P2SH-P2WPKH instead of evaluating it by Eunovo)
    • #36721 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/36721.svg"></sub> (script: store interpreter stack elements as reference-counted objects by conduition)
    • #36548 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/36548.svg"></sub> (net: serve MSG_BLOCK by stripping witnesses in place by l0rinc)
    • #36466 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/36466.svg"></sub> (policy: make unstructured Taproot annexes standard by RobinLinus)
    • #36122 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/36122.svg"></sub> (BIP460: CISA for Taproot key path spends by fjahr)
    • #36109 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/36109.svg"></sub> (test: fix coinbases built on arbitrary parents by l0rinc)
    • #35788 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/35788.svg"></sub> (bitcoin-util: Add evalscript subcommand by ajtowns)
    • #35716 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/35716.svg"></sub> (wallet: Replace mapWallet and wtxOrdered with a boost::multi_index by achow101)
    • #35662 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/35662.svg"></sub> (script: prevent partial reinitialization of sighash data by l0rinc)
    • #31252 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/31252.svg"></sub> (rpc: print P2WSH and P2SH redem Script in getrawtransaction and getblock by polespinasa)
    • #29247 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/29247.svg"></sub> (CAT in Tapscript (BIP-347) by arminsabouri)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

  15. darosior commented at 7:48 PM on November 16, 2026: member

    Concept ACK

  16. w0xlt commented at 8:26 PM on November 16, 2026: contributor

    Concept ACK

  17. fanquake added this to the milestone 33.0 on Nov 16, 2026
  18. sipa force-pushed on Nov 16, 2026
  19. xyzconstant commented at 11:08 PM on November 16, 2026: contributor

    Concept ACK

  20. brunoerg commented at 1:22 PM on November 17, 2026: contributor

    Concept ACK

  21. pzafonte commented at 8:36 AM on November 18, 2026: contributor
    • libbitcoinkernel's witness stack element accessor becomes O(n) in the index of the witness element; using it to iterate a whole witness may therefore become O(n^2)

    This should be able to be addressed by an API function that walks the stack once and hands each item's bytes to the caller. Testing it out.

    Edit: See: #36909

  22. sedited requested review from optout21 on Nov 18, 2026
  23. sedited requested review from alexanderwiederin on Nov 18, 2026
  24. sedited requested review from furszy on Nov 18, 2026
  25. instagibbs commented at 6:34 PM on November 18, 2026: member

    Did another clanker deep dive to see its performance effects on memory and verification speed. On the historical blockchain it seems like noise. Under adversarial conditions slightly slower verification speeds when dealing with related issues, but nothing remotely close to close to worst case blocks, even post-bip54.

    Worst case regression I'm getting is +149ms for a block that is just 4M of witness data elements, which is a bit north of "50-70 ms per block" but might be due to measurement differences, and is largely irrelevant anyways.

    Can share longer form slop on request, but nothing concerning.

    concept ACK on focusing on the persistent memory usage first

  26. sipa commented at 10:06 PM on November 18, 2026: member

    @instagibbs FWIW, these are the benchmarks I based this on. "minimal" is this PR. The "lazy conversion" and "direction deserialization" are follow-up optimizations not included in this PR. Hardware is a high-end Zen5 machine.


    Each row is one 4M WU block, timed over deserialize, CheckBlock and ConnectBlock, in ms. These are medians of 11 interleaved rounds on the now-idle machine, gcc Release.

    block master minimal + lazy conversion + direct deserialization
    valid: unknown witness version, 2M one-byte items 99 169 (+70) 108 76
    invalid: P2WPKH with 2M one-byte items 99 169 (+70) 108 77
    valid: unknown tapleaf version, 2M one-byte items 99 151 (+52) 151 70
    invalid: P2WSH script mismatch, 2M one-byte items 99 147 (+48) 147 66
    invalid: P2WSH script mismatch, 4M empty items 249 293 (+44) 294 59
    invalid: P2WSH with 4M empty items (master's slowest) 307 296 (−11) 294 61
    valid: 5,169 P2WSH spends of 403 items 22.2 23.3 (+1.1) 23.4 21.8
  27. in src/script/script.cpp:306 in a6a4df2770
     306 | +    for (const auto element : *this) {
     307 | +        if (!first) ret += ", ";
     308 | +        first = false;
     309 | +        ret += HexStr(element);
     310 |      }
     311 |      return ret + ")";
    


    optout21 commented at 10:32 AM on November 19, 2026:

    a6a4df2 script: store witness stacks as their serialization:

    Nit: Could be simplified to: (requires #include <util/string.h>)

        return "CScriptWitness(" + util::Join(*this, ", ", [](auto element) { return HexStr(element); }) + ")";
    

    sipa commented at 6:29 PM on November 19, 2026:

    Done.

  28. in src/policy/policy.cpp:316 in 0a82e05e34
     310 | @@ -312,8 +311,10 @@ bool IsWitnessStandard(const CTransaction& tx, const CCoinsViewCache& mapInputs)
     311 |              size_t sizeWitnessStack = tx.vin[i].scriptWitness.size() - 1;
     312 |              if (sizeWitnessStack > MAX_STANDARD_P2WSH_STACK_ITEMS)
     313 |                  return false;
     314 | +            // The elements as a vector of spans, for indexed access in the loop below.
     315 | +            const auto witness_stack = tx.vin[i].scriptWitness.ToSpans();
     316 |              for (unsigned int j = 0; j < sizeWitnessStack; j++) {
    


    optout21 commented at 12:01 PM on November 19, 2026:

    0a82e05 refactor: use CScriptWitness's interface where that needs more than a rename:

    Minor: Here back() is called, then ToSpans(). If the MAX_STANDARD_P2WSH_SCRIPT_SIZE check was moved after ToSpans(), the back() call could be avoided (back is potentially expensive).


    sipa commented at 6:37 PM on November 19, 2026:

    #36918 instead has a lazy creation of the stack (and no more ToSpans), which avoids this issue entirely I think.

  29. optout21 commented at 12:14 PM on November 19, 2026: contributor

    crACK a6a4df2770eb5f79f4e2dc58dee12889ca0be700

    This change reduces the memory usage of CScriptWitness. Instead of the parsed representation it stores the serialized version. Memory footprint is reduced especially for empty witnesses, or for empty or small witness elements. Considering that the parsing is relatively low complexity, and the usage pattern, the tradeoff seems to be worth it.

    Instead of storing a byte vector (with a footprint of 3 values, 24 bytes on 64-bit systems) it pushes the optimization further: it stores a byte pointer, no allocated size / capacity, and the length in 4 bytes inside the data buffer. The memory needed for the fields is thus 8 bytes inline plus 4 extra bytes on the heap, but only for non-empty witnesses.

    The commits are granular and well structured (nice usage of scripted diffs for usage switch).

    Two minor issues left as comments. No other issues found.

    An alternative considered: Store the length as a separate field, not part of the buffer. The advantages would be: simpler access to the serialized form, and no header read on each access. However, the memory usage improvement -- the biggest win of this PR -- would take a hit (8 extra bytes for every witness, including empty ones; CTxIn size from 96 to 104).

    A note on the range of counts and lengths: uint64_t ReadCompactSize() is implicitly converted to size_t in a few places. This is fine: ReadCompactSize rejects values above MAX_SIZE (2^25), so anything it returns fits in 26 bits. For deserialized witnesses this always holds; for witnesses built from vectors it holds in practice. It's worth mentioning that it's possible to construct an artificially large witness with an element larger than MAX_SIZE. In this case std::terminate would be hit on first access, since the accessors are noexcept. That's not reachable from the network, but checking MAX_SIZE in the constructor would make the invariant hold by construction.

    Further (followup) ideas:

    • I wonder if this "light read-only byte vector" construct would be worthwhile to be separated on its own into utils.

    • A potential optimization: Since the class wrote the serialization itself, lengths can be decoded without ReadCompactSize's internal checks, with a fast path for one-byte lengths. This also removes the terminate case above, as decoding can no longer throw. In addition, operator[] can read the header and the element count only once instead of twice. In a quick micro-benchmark this made iteration, back() and [] on a typical 3-element witness about 5× faster, and 1.4–2× faster for 10k one-byte elements; see branch pr36895-oo-optim1.

  30. DrahtBot requested review from darosior on Nov 19, 2026
  31. DrahtBot requested review from instagibbs on Nov 19, 2026
  32. DrahtBot requested review from brunoerg on Nov 19, 2026
  33. script: store witness stacks as their serialization
    Replace CScriptWitness's vector of vectors with a single allocation,
    containing the size of the stack's serialization (4 bytes, little
    endian), followed by that serialization. An empty stack uses no
    allocation at all.
    
    This means a witness stack no longer needs one allocation per non-empty
    element plus one for the outer vector, and no longer has a 24-byte
    overhead per element. A block whose witnesses consist of millions of
    tiny elements thus no longer costs tens of bytes of memory and an
    allocation per byte. It also shrinks CScriptWitness from 24 to 8 bytes,
    and thus CTxIn from 112 to 96 bytes. The downside is that indexing
    becomes O(n) in the distance from the bottom of the stack.
    
    This is the simplest implementation of this representation:
    deserialization still goes through a vector of vectors, and elements
    are found by parsing the serialization with SpanReader and
    ReadCompactSize. Both can be made faster in follow-up changes.
    0c981e75a9
  34. in src/test/script_tests.cpp:1737 in 49c0f21971 outdated
    1732 | @@ -1733,4 +1733,67 @@ BOOST_AUTO_TEST_CASE(formatscriptflags)
    1733 |      BOOST_CHECK_EQUAL(FormatScriptFlags(script_verify_flags::from_int(1u<<26)), "0x04000000");
    1734 |  }
    1735 |  
    1736 | +BOOST_AUTO_TEST_CASE(script_witness_representation)
    1737 | +{
    


    instagibbs commented at 2:39 PM on November 19, 2026:

    49c0f21971050b24cecde56bdb42653af3910fc6

    A couple suggestions for tests here. 3 to 5 compactsize boundary which IIUC is consensus legal in a block, and every way of making an empty witness giving the same object, since HasWitness() and so the witness flag and wtxid depend on it.

    diff --git a/src/test/script_tests.cpp b/src/test/script_tests.cpp
    index cc6fa3843a..3b46f468d4 100644
    --- a/src/test/script_tests.cpp
    +++ b/src/test/script_tests.cpp
    @@ -1774,26 +1774,110 @@ BOOST_AUTO_TEST_CASE(script_witness_representation)
                 if (!stack.empty()) BOOST_CHECK(std::ranges::equal(deser.back(), stack.back()));
     
                 // Moving leaves the source empty.
                 CScriptWitness moved{std::move(deser)};
                 BOOST_CHECK(moved == witness);
                 BOOST_CHECK(deser.empty() && deser.size() == 0 && deser.begin() == deser.end()); // NOLINT(bugprone-use-after-move)
     
                 // Copy and move assignment replace the stack that was there.
                 CScriptWitness copy{witness};
                 CScriptWitness assigned{{{0xff}}};
                 assigned = copy;
                 copy = {};
                 BOOST_CHECK(copy.empty());
                 BOOST_CHECK(copy.begin() == copy.end());
                 BOOST_CHECK(assigned == witness);
     
                 CScriptWitness move_assigned{{{0xff}}};
                 move_assigned = std::move(assigned);
                 BOOST_CHECK(move_assigned == witness);
                 BOOST_CHECK(assigned.empty() && assigned.begin() == assigned.end()); // NOLINT(bugprone-use-after-move)
                 if (!stack.empty()) BOOST_CHECK(std::ranges::equal(move_assigned.back(), stack.back()));
             }
         }
     }
     
    +BOOST_AUTO_TEST_CASE(script_witness_large_count)
    +{
    +    // Element counts on both sides of the boundary between the 3-byte and 5-byte CompactSize encodings. Elements are
    +    // small, and only a few are accessed by index, as indexing may take time linear in the index.
    +    for (const size_t count : {size_t{0xffff}, size_t{0x10000}}) {
    +        std::vector<std::vector<unsigned char>> stack(count);
    +        for (auto& element : stack) element = m_rng.randbytes(m_rng.randrange(3));
    +        const CScriptWitness witness{stack};
    +
    +        BOOST_CHECK_EQUAL(witness.size(), count);
    +        BOOST_CHECK_EQUAL(std::distance(witness.begin(), witness.end()), (std::ptrdiff_t)count);
    +        BOOST_CHECK(witness.ToVectors() == stack);
    +        BOOST_CHECK(std::ranges::equal(witness.front(), stack.front()));
    +        BOOST_CHECK(std::ranges::equal(witness.back(), stack.back()));
    +        for (const size_t i : {size_t{1}, count / 2, count - 2}) BOOST_CHECK(std::ranges::equal(witness[i], stack[i]));
    +
    +        std::vector<unsigned char> ser_vec, ser_wit;
    +        VectorWriter{ser_vec, 0, stack};
    +        VectorWriter{ser_wit, 0, witness};
    +        BOOST_CHECK(ser_vec == ser_wit);
    +        // The count is encoded as 0xfd followed by 2 bytes, or 0xfe followed by 4.
    +        BOOST_CHECK_EQUAL(int{ser_wit.at(0)}, count <= 0xffff ? 0xfd : 0xfe);
    +        CScriptWitness deser;
    +        SpanReader{ser_vec} >> deser;
    +        BOOST_CHECK(deser == witness);
    +        BOOST_CHECK_EQUAL(deser.size(), count);
    +        BOOST_CHECK(std::ranges::equal(deser.back(), stack.back()));
    +    }
    +}
    +
    +BOOST_AUTO_TEST_CASE(script_witness_empty)
    +{
    +    // Every way of producing a witness without elements gives the same witness. A transaction is serialized with
    +    // witness data exactly when one of its inputs has a non-empty witness, so this determines its wtxid and whether its
    +    // serialization round-trips.
    +    using Stack = std::vector<std::vector<unsigned char>>;
    +    const std::vector<unsigned char> ser_empty{0x00};
    +    const CScriptWitness from_default;
    +    const CScriptWitness from_vector{Stack{}};
    +    CScriptWitness from_deser{{{0x01}}};
    +    SpanReader{ser_empty} >> from_deser;
    +    CScriptWitness moved_from{{{0x01}}};
    +    const CScriptWitness moved_to{std::move(moved_from)};
    +    BOOST_CHECK_EQUAL(moved_to.size(), 1U);
    +    CScriptWitness reset{{{0x01}}};
    +    reset = {};
    +    const CScriptWitness copied{from_default};
    +    CScriptWitness copy_assigned{{{0x01}}};
    +    copy_assigned = from_default;
    +
    +    const std::vector<const CScriptWitness*> empties{&from_default, &from_vector, &from_deser, &moved_from, &reset, &copied, &copy_assigned}; // NOLINT(bugprone-use-after-move)
    +    for (const CScriptWitness* witness : empties) {
    +        BOOST_CHECK(witness->empty());
    +        BOOST_CHECK_EQUAL(witness->size(), 0U);
    +        BOOST_CHECK(witness->begin() == witness->end());
    +        BOOST_CHECK(witness->ToVectors().empty());
    +        BOOST_CHECK(*witness == from_default);
    +        std::vector<unsigned char> ser;
    +        VectorWriter{ser, 0, *witness};
    +        BOOST_CHECK(ser == ser_empty);
    +        CMutableTransaction tx;
    +        tx.vin.resize(1);
    +        tx.vin[0].scriptWitness = *witness;
    +        BOOST_CHECK(!tx.HasWitness());
    +    }
    +
    +    // A single empty element is a different stack.
    +    const CScriptWitness one_empty{Stack(1)};
    +    BOOST_CHECK(!one_empty.empty());
    +    BOOST_CHECK_EQUAL(one_empty.size(), 1U);
    +    BOOST_CHECK(one_empty.back().empty());
    +    BOOST_CHECK(one_empty != from_default);
    +    std::vector<unsigned char> ser;
    +    VectorWriter{ser, 0, one_empty};
    +    BOOST_CHECK(ser == (std::vector<unsigned char>{0x01, 0x00}));
    +    CScriptWitness deser;
    +    SpanReader{ser} >> deser;
    +    BOOST_CHECK(deser == one_empty);
    +    CMutableTransaction tx;
    +    tx.vin.resize(1);
    +    tx.vin[0].scriptWitness = one_empty;
    +    BOOST_CHECK(tx.HasWitness());
    +}
    +
     BOOST_AUTO_TEST_SUITE_END()
    
  35. sipa force-pushed on Nov 19, 2026
  36. pzafonte commented at 4:42 PM on November 19, 2026: contributor
    • A potential optimization: Since the class wrote the serialization itself, lengths can be decoded without ReadCompactSize's internal checks, with a fast path for one-byte lengths.

    I like this, a natural consequence of this is that it would speed up the kernel.

  37. instagibbs approved
  38. instagibbs commented at 5:02 PM on November 19, 2026: member

    ACK https://github.com/bitcoin/bitcoin/pull/36895/commits/0c981e75a9e7e352d1c0f08118596ea91a9fa102

    Ran quite a few validation passes over worst-case at-rest memory usage for various places like orphanage and other vectors. Does what it says on the tin with no slowdown on average case transaction/block patterns, the biggest exception being single transaction NFT blocks which are a few hundred mics slower due to double copying of the exec_script.

    Also, looks like mempool will fit ~13-15% more transactions in it, based on my historical simulations.

    This change also doesn't seem make future updates to interface any more difficult, so we're not being boxed in with this bugfix change.

    CI failures seem unrelated

  39. DrahtBot requested review from optout21 on Nov 19, 2026
  40. DrahtBot added the label CI failed on Nov 19, 2026
  41. sipa commented at 6:31 PM on November 19, 2026: member

    @optout21 I had a series of several optimizations in my old PR #36521, some of which do similar things to what you suggest. I've opened a draft PR #36918 with those optimization to avoid duplicate work.

    This PR is deliberately avoiding performance optimizations, to focus review on the more important issue of memory usage.


    An alternative considered: Store the length as a separate field, not part of the buffer. The advantages would be: simpler access to the serialized form, and no header read on each access. However, the memory usage improvement -- the biggest win of this PR -- would take a hit (8 extra bytes for every witness, including empty ones; CTxIn size from 96 to 104).

    I'd rather not do that. Memory usage is the priority here I think. In the follow-up PR the header is also expanded with the position of the last element, so that CScriptWitness::back() runs is $\mathcal{O}(1)$ time.

    In this case std::terminate would be hit on first access, since the accessors are noexcept. That's not reachable from the network, but checking MAX_SIZE in the constructor would make the invariant hold by construction.

    The follow-up uses an optimized decoder that never throws (but relies on the fact that the bytes are self-encoded and thus guaranteed to be correct).

    I wonder if this "light read-only byte vector" construct would be worthwhile to be separated on its own into utils.

    If there is a second use case for it, that seems reasonable, but I'm not sure where that'd be.

    Since the class wrote the serialization itself, lengths can be decoded without ReadCompactSize's internal checks, with a fast path for one-byte lengths.

    I believe the old PR already had that; it's in the follow-up PR now.


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-11-19 19:51 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me