rpc: avoid quadratic `gettxspendingprevout` work and preserve order #35889

pull l0rinc wants to merge 4 commits into bitcoin:master from l0rinc:l0rinc/rpc-gettxspendingprevout-linear changing 2 files +29 −24
  1. l0rinc commented at 10:57 PM on August 4, 2026: contributor

    Problem: gettxspendingprevout erases each mempool result from a vector while holding mempool.cs, shifting the remaining requests every time and making large calls quadratic in the critical section. For 10,000 mempool matches, an operation-count model reaches nearly 50 million moves. For mixed requests, the RPC returns mempool results before txospenderindex results instead of following request order. #34749 introduced both regressions.

    Fix: gettxspendingprevout stores each result at its request position and collects unresolved requests in a reserved worklist for the txospenderindex lookup. The mempool pass is linear, the response follows request order, and Clang can verify the lock requirement on GetConflictTx.

    Benchmark: The functional benchmark sends mempool-only requests ranging from 8,000 to 128,000 entries ten times per size. Using the same settings for the unfixed and fixed commits:

    AMD Ryzen 7 3700X (8 cores)
    unfixed  ██████████████████████████████  90 s
    fixed    ███▒░░░░░░░░░░░░░░░░░░░░░░░░░░  10 s  (-80 s, 9.0x faster)
    
    Raspberry Pi 5 (4 cores)
    unfixed  ██████████████████████████████  685 s
    fixed    ▓░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  22 s  (-663 s, 31.1x faster)
    

    The unfixed run timed out after ~9 minutes on a Raspberry Pi 4 with 1 GB RAM.

    <details><summary>Benchmark command</summary>

    for commit in 963b061358b489b2ff4ff64895f2b895b3b89844 46e7173550a93cbe9d4e8ea28cfe7216286d8197; do \
      git fetch origin "$commit" && git checkout --detach "$commit" && \
      rm -rfd build && cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTS=OFF -DENABLE_WALLET=OFF >/dev/null 2>&1 && \
      ninja -C build -j1 bitcoind >/dev/null 2>&1 && \
      build/test/functional/test_runner.py rpc_gettxspendingprevout_quadratic.py --repeats=10 || break; \
    done
    

    </details>

  2. DrahtBot added the label RPC/REST/ZMQ on Aug 4, 2026
  3. DrahtBot commented at 10:57 PM on August 4, 2026: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35889.

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline and AI policy for information on the review process.

    Type Reviewers
    Concept ACK andrewtoth

    If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

  4. in src/rpc/mempool.cpp:991 in 5bae16cb85
     988 | +                size_t request_index;
     989 |              };
     990 |              std::vector<Entry> prevouts_to_process;
     991 |              prevouts_to_process.reserve(output_params.size());
     992 | -            for (unsigned int idx = 0; idx < output_params.size(); idx++) {
     993 | +            for (size_t idx{0}; idx < output_params.size(); ++idx) {
    


    andrewtoth commented at 1:02 AM on August 5, 2026:

    nit: if we're touching this anyways, we can make idx const:

                for (const size_t idx : std::views::iota(size_t{0}, output_params.size())) {
    

    l0rinc commented at 2:11 AM on August 5, 2026:

    I deliberately remove short-scoped consts, they're just noise - but I don't mind the std::views::iota


    andrewtoth commented at 7:25 PM on August 5, 2026:

    What is your rationale for const declarations being noise? Everything should be const by default IMO. Modern languages like Rust do this intentionally, because it avoids large classes of software bugs that can be caught at compile time.


    l0rinc commented at 7:49 PM on August 5, 2026:

    when the scope is narrow I find it just a distraction


    andrewtoth commented at 8:58 PM on August 5, 2026:

    I don't think you should be deliberately removing consts. Using const by default is specifically called out in the CppCoreGuidelines which we reference in our doc/developer-notes.md:

    Immutable objects are easier to reason about, so make objects non-const only when there is a need to change their value. Prevents accidental or hard-to-notice change of value.

    Anyways my nit was about being able to make idx const by doing this. I don't see the point of the refactor if you'll just leave it non-const.


    l0rinc commented at 9:06 PM on August 5, 2026:

    I don't see the point of the refactor if you'll just leave it non-const.

    It's a simple for loop with a narrow scope, we rarely const those, it's how I've always structured these. If you insist I can add it back, but I did it deliberately.

  5. in src/rpc/mempool.cpp:1064 in 21c198f8d8
    1061 | +                    results[prevout.request_index] = make_output(prevout);
    1062 |                  }
    1063 |              }
    1064 |  
    1065 | +            UniValue result{UniValue::VARR};
    1066 | +            for (auto& output : results) result.push_back(std::move(output));
    


    andrewtoth commented at 1:09 AM on August 5, 2026:

    We should probably check the invariant that we have assigned something to every entry in results.

                for (auto& output : results) result.push_back(std::move(CHECK_NONFATAL(!output.isNull(), output)));
    

    maflcko commented at 5:58 AM on August 5, 2026:

    Aren't type checks done by the RPC server? If not, it should be fixed there. Otherwise, the code is filled with redundant and possibly inconsistent checks wrt the docs, which should be the ground truth.


    l0rinc commented at 7:07 AM on August 5, 2026:

    I missed your comment @maflcko, will investigate tomorrow


    andrewtoth commented at 2:04 PM on August 5, 2026:

    I would argue this isn't really type checking, but making sure the procedural code above has made an assignment to every entry in the vector. A null value is just the unassigned default. We could also do Assume(!output.isNull()) to be very sure, or have the vector be std::vector<std::option<UniValue>> and check for std::nullopt.


    l0rinc commented at 7:22 PM on August 5, 2026:

    Reverted since this does seem to be guarded, though I sympathize with the assignment check (it's why I accepted it originally). Can be revisited later.

  6. in src/rpc/mempool.cpp:1027 in 46e7173550 outdated
    1023 | @@ -1024,19 +1024,16 @@ static RPCMethod gettxspendingprevout()
    1024 |                  LOCK(mempool.cs);
    1025 |  
    1026 |                  // Make the result if the spending tx appears in the mempool or this is a mempool_only request
    1027 | -                for (auto it = prevouts_to_process.begin(); it != prevouts_to_process.end(); ) {
    1028 | -                    const CTransaction* spending_tx{mempool.GetConflictTx(it->outpoint)};
    1029 | +                std::erase_if(prevouts_to_process, [&](const Entry& prevout) EXCLUSIVE_LOCKS_REQUIRED(mempool.cs) {
    


    andrewtoth commented at 1:18 AM on August 5, 2026:

    This might be a little easier to reason about if it's more procedural and additive. We won't need EXCLUSIVE_LOCKS_REQUIRED this way either.

    std::vector<Entry> unresolved;
    {
        const CTxMemPool& mempool = EnsureAnyMemPool(request.context);
        LOCK(mempool.cs);
        for (const Entry& prevout : prevouts_to_process) {
            const CTransaction* spending_tx{mempool.GetConflictTx(prevout.outpoint)};
            if (!spending_tx && !mempool_only) {
                unresolved.push_back(prevout); 
            } else {
                results[prevout.request_index] = make_output(prevout, spending_tx);
            }
        }
    }
    

    Then we can loop over unresolved below instead of prevouts_to_process?


    l0rinc commented at 5:45 AM on August 5, 2026:

    Yeah, I like the more procedural part, but we would be allocating the worst case twice, even when few entries remain.


    andrewtoth commented at 2:05 PM on August 5, 2026:

    If allocations is the concern, then we can just do unresolved.reserve(prevouts_to_process.size()); and we are guaranteed to only have 1 allocation. That should be negligible.


    l0rinc commented at 4:54 PM on August 5, 2026:

    Yes, that's my point, we're allocating the worst case yet again


    andrewtoth commented at 5:50 PM on August 5, 2026:

    Right, but why do we care about a single allocation not on the hot path here? This is fixing a quadratic algorithm issue, so this extra allocation should be negligible?

    One other issue is that using the lambda here is bypassing our lock checking infrastructure. The reason we need the EXCLUSIVE_LOCKS_REQUIRED is to suppress the missing lock warning inside the lambda. We can't actually statically analyze that the lock is held in there.

    The implication of this is that the LOCK(mempool.cs); line above can be commented out, and this would compile without a warning. The same is not true for the procedural loop style.


    l0rinc commented at 7:21 PM on August 5, 2026:

    Sure, done, added you as coauthor

  7. andrewtoth commented at 1:20 AM on August 5, 2026: contributor

    Concept ACK

  8. l0rinc force-pushed on Aug 5, 2026
  9. maflcko added this to the milestone 32.0 on Aug 5, 2026
  10. test: cover mixed `gettxspendingprevout` order
    Record that `gettxspendingprevout` currently returns mempool results before `txospenderindex` results for mixed requests.
    8c39cc7a1a
  11. refactor: identify prevouts by request index
    Replace `Entry`'s pointer into `output_params` with the request index, which identifies both the input and its response slot.
    c31e8dd8de
  12. rpc: preserve `gettxspendingprevout` order
    Store each `gettxspendingprevout` result at its request position so mixed mempool and `txospenderindex` results preserve request order.
    a49fc36cbb
  13. rpc: avoid quadratic prevout resolution
    `gettxspendingprevout` erases each mempool result from its worklist while holding `mempool.cs`, shifting the remaining requests every time and making the pass quadratic when it resolves many requests.
    Collect unresolved requests in a reserved worklist so the mempool pass is linear and the compiler can verify the lock requirement on `GetConflictTx`.
    
    Co-authored-by: Andrew Toth <andrewstoth@gmail.com>
    04e0701cd9
  14. l0rinc force-pushed on Aug 5, 2026

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-08-08 03:51 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me