rpc: avoid quadratic `gettxspendingprevout` work and preserve order #35889

pull l0rinc wants to merge 4 commits into bitcoin:master from l0rinc:l0rinc/rpc-gettxspendingprevout-linear changing 2 files +30 −24
  1. l0rinc commented at 10:57 PM on August 4, 2026: contributor

    Problem: gettxspendingprevout erases each mempool result from a vector while holding mempool.cs, shifting the remaining requests every time and making large calls quadratic in the critical section. For 10,000 mempool matches, an operation-count model reaches nearly 50 million moves. For mixed requests, the RPC returns mempool results before txospenderindex results instead of following request order. #34749 introduced both regressions.

    Fix: gettxspendingprevout stores each result at its request position and collects unresolved requests in a reserved worklist for the txospenderindex lookup. The mempool pass is linear, the response follows request order, and Clang can verify the lock requirement on GetConflictTx.

    Benchmark: The functional benchmark sends mempool-only requests ranging from 8,000 to 128,000 entries ten times per size. Using the same settings for the unfixed and fixed commits:

    AMD Ryzen 7 3700X (8 cores)
    unfixed  ██████████████████████████████  90 s
    fixed    ███▒░░░░░░░░░░░░░░░░░░░░░░░░░░  10 s  (-80 s, 9.0x faster)
    
    Raspberry Pi 5 (4 cores)
    unfixed  ██████████████████████████████  685 s
    fixed    ▓░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  22 s  (-663 s, 31.1x faster)
    

    The unfixed run timed out after ~9 minutes on a Raspberry Pi 4 with 1 GB RAM.

    <details><summary>Benchmark command</summary>

    for commit in 963b061358b489b2ff4ff64895f2b895b3b89844 46e7173550a93cbe9d4e8ea28cfe7216286d8197; do \
      git fetch origin "$commit" && git checkout --detach "$commit" && \
      rm -rfd build && cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTS=OFF -DENABLE_WALLET=OFF >/dev/null 2>&1 && \
      ninja -C build -j1 bitcoind >/dev/null 2>&1 && \
      build/test/functional/test_runner.py rpc_gettxspendingprevout_quadratic.py --repeats=10 || break; \
    done
    

    </details>

  2. DrahtBot added the label RPC/REST/ZMQ on Aug 4, 2026
  3. DrahtBot commented at 10:57 PM on August 4, 2026: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35889.

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline and AI policy for information on the review process.

    Type Reviewers
    ACK sedited, hodlinator
    Concept ACK andrewtoth

    If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

  4. in src/rpc/mempool.cpp:991 in 5bae16cb85
     988 | +                size_t request_index;
     989 |              };
     990 |              std::vector<Entry> prevouts_to_process;
     991 |              prevouts_to_process.reserve(output_params.size());
     992 | -            for (unsigned int idx = 0; idx < output_params.size(); idx++) {
     993 | +            for (size_t idx{0}; idx < output_params.size(); ++idx) {
    


    andrewtoth commented at 1:02 AM on August 5, 2026:

    nit: if we're touching this anyways, we can make idx const:

                for (const size_t idx : std::views::iota(size_t{0}, output_params.size())) {
    

    l0rinc commented at 2:11 AM on August 5, 2026:

    I deliberately remove short-scoped consts, they're just noise - but I don't mind the std::views::iota


    andrewtoth commented at 7:25 PM on August 5, 2026:

    What is your rationale for const declarations being noise? Everything should be const by default IMO. Modern languages like Rust do this intentionally, because it avoids large classes of software bugs that can be caught at compile time.


    l0rinc commented at 7:49 PM on August 5, 2026:

    when the scope is narrow I find it just a distraction


    andrewtoth commented at 8:58 PM on August 5, 2026:

    I don't think you should be deliberately removing consts. Using const by default is specifically called out in the CppCoreGuidelines which we reference in our doc/developer-notes.md:

    Immutable objects are easier to reason about, so make objects non-const only when there is a need to change their value. Prevents accidental or hard-to-notice change of value.

    Anyways my nit was about being able to make idx const by doing this. I don't see the point of the refactor if you'll just leave it non-const.


    l0rinc commented at 9:06 PM on August 5, 2026:

    I don't see the point of the refactor if you'll just leave it non-const.

    It's a simple for loop with a narrow scope, we rarely const those, it's how I've always structured these. If you insist I can add it back, but I did it deliberately.


    l0rinc commented at 7:57 PM on August 13, 2026:

    Added const, I don't really mind either way

  5. in src/rpc/mempool.cpp:1064 in 21c198f8d8
    1061 | +                    results[prevout.request_index] = make_output(prevout);
    1062 |                  }
    1063 |              }
    1064 |  
    1065 | +            UniValue result{UniValue::VARR};
    1066 | +            for (auto& output : results) result.push_back(std::move(output));
    


    andrewtoth commented at 1:09 AM on August 5, 2026:

    We should probably check the invariant that we have assigned something to every entry in results.

                for (auto& output : results) result.push_back(std::move(CHECK_NONFATAL(!output.isNull(), output)));
    

    maflcko commented at 5:58 AM on August 5, 2026:

    Aren't type checks done by the RPC server? If not, it should be fixed there. Otherwise, the code is filled with redundant and possibly inconsistent checks wrt the docs, which should be the ground truth.


    l0rinc commented at 7:07 AM on August 5, 2026:

    I missed your comment @maflcko, will investigate tomorrow


    andrewtoth commented at 2:04 PM on August 5, 2026:

    I would argue this isn't really type checking, but making sure the procedural code above has made an assignment to every entry in the vector. A null value is just the unassigned default. We could also do Assume(!output.isNull()) to be very sure, or have the vector be std::vector<std::option<UniValue>> and check for std::nullopt.


    l0rinc commented at 7:22 PM on August 5, 2026:

    Reverted since this does seem to be guarded, though I sympathize with the assignment check (it's why I accepted it originally). Can be revisited later.

  6. in src/rpc/mempool.cpp:1027 in 46e7173550 outdated
    1023 | @@ -1024,19 +1024,16 @@ static RPCMethod gettxspendingprevout()
    1024 |                  LOCK(mempool.cs);
    1025 |  
    1026 |                  // Make the result if the spending tx appears in the mempool or this is a mempool_only request
    1027 | -                for (auto it = prevouts_to_process.begin(); it != prevouts_to_process.end(); ) {
    1028 | -                    const CTransaction* spending_tx{mempool.GetConflictTx(it->outpoint)};
    1029 | +                std::erase_if(prevouts_to_process, [&](const Entry& prevout) EXCLUSIVE_LOCKS_REQUIRED(mempool.cs) {
    


    andrewtoth commented at 1:18 AM on August 5, 2026:

    This might be a little easier to reason about if it's more procedural and additive. We won't need EXCLUSIVE_LOCKS_REQUIRED this way either.

    std::vector<Entry> unresolved;
    {
        const CTxMemPool& mempool = EnsureAnyMemPool(request.context);
        LOCK(mempool.cs);
        for (const Entry& prevout : prevouts_to_process) {
            const CTransaction* spending_tx{mempool.GetConflictTx(prevout.outpoint)};
            if (!spending_tx && !mempool_only) {
                unresolved.push_back(prevout); 
            } else {
                results[prevout.request_index] = make_output(prevout, spending_tx);
            }
        }
    }
    

    Then we can loop over unresolved below instead of prevouts_to_process?


    l0rinc commented at 5:45 AM on August 5, 2026:

    Yeah, I like the more procedural part, but we would be allocating the worst case twice, even when few entries remain.


    andrewtoth commented at 2:05 PM on August 5, 2026:

    If allocations is the concern, then we can just do unresolved.reserve(prevouts_to_process.size()); and we are guaranteed to only have 1 allocation. That should be negligible.


    l0rinc commented at 4:54 PM on August 5, 2026:

    Yes, that's my point, we're allocating the worst case yet again


    andrewtoth commented at 5:50 PM on August 5, 2026:

    Right, but why do we care about a single allocation not on the hot path here? This is fixing a quadratic algorithm issue, so this extra allocation should be negligible?

    One other issue is that using the lambda here is bypassing our lock checking infrastructure. The reason we need the EXCLUSIVE_LOCKS_REQUIRED is to suppress the missing lock warning inside the lambda. We can't actually statically analyze that the lock is held in there.

    The implication of this is that the LOCK(mempool.cs); line above can be commented out, and this would compile without a warning. The same is not true for the procedural loop style.


    l0rinc commented at 7:21 PM on August 5, 2026:

    Sure, done, added you as coauthor

  7. andrewtoth commented at 1:20 AM on August 5, 2026: contributor

    Concept ACK

  8. l0rinc force-pushed on Aug 5, 2026
  9. maflcko added this to the milestone 32.0 on Aug 5, 2026
  10. l0rinc force-pushed on Aug 5, 2026
  11. in src/rpc/mempool.cpp:1065 in 04e0701cd9 outdated
    1062 | +                    results[prevout.request_index] = make_output(prevout);
    1063 |                  }
    1064 |              }
    1065 |  
    1066 | +            UniValue result{UniValue::VARR};
    1067 | +            for (auto& output : results) result.push_back(std::move(output));
    


    hodlinator commented at 12:28 PM on August 13, 2026:

    Could we either do result.reserve(results.size()) or add an overload for UniValue::push_backV(std::vector<UniValue>&& vec)?


    l0rinc commented at 7:56 PM on August 13, 2026:

    Thanks, took the reserve, seems the simplest

  12. hodlinator approved
  13. hodlinator commented at 1:37 PM on August 13, 2026: contributor

    ACK 04e0701cd930644a88de8c8d06f3f0186b32294a

    Thanks for catching the quadratic and ordering issues! I did catch the ordering issue in #34635 (review) but downplayed it, though I agree it's worth enforcing in order to be nice to API users.

    cc @sstone who introduced the index and may be interested in both properties getting restored for his use case.

  14. DrahtBot requested review from andrewtoth on Aug 13, 2026
  15. test: cover mixed `gettxspendingprevout` order
    Record that `gettxspendingprevout` currently returns mempool results before `txospenderindex` results for mixed requests.
    0af025e2e8
  16. refactor: identify prevouts by request index
    Replace `Entry`'s pointer into `output_params` with the request index, which identifies both the input and its response slot.
    74ab882c13
  17. rpc: preserve `gettxspendingprevout` order
    Store each `gettxspendingprevout` result at its request position so mixed mempool and `txospenderindex` results preserve request order.
    ad08d31c15
  18. rpc: avoid quadratic prevout resolution
    `gettxspendingprevout` erases each mempool result from its worklist while holding `mempool.cs`, shifting the remaining requests every time and making the pass quadratic when it resolves many requests.
    Collect unresolved requests in a reserved worklist so the mempool pass is linear and the compiler can verify the lock requirement on `GetConflictTx`.
    
    Co-authored-by: Andrew Toth <andrewstoth@gmail.com>
    4cae112628
  19. l0rinc force-pushed on Aug 13, 2026
  20. in src/rpc/mempool.cpp:1044 in 4cae112628
    1047 |                  }
    1048 |              }
    1049 |  
    1050 |              // mempool_only requests resolve every outpoint above, so only other requests reach the index.
    1051 | -            if (!prevouts_to_process.empty() && (!g_txospenderindex || !g_txospenderindex->BlockUntilSyncedToCurrentChain())) {
    1052 | +            if (unresolved.size() && (!g_txospenderindex || !g_txospenderindex->BlockUntilSyncedToCurrentChain())) {
    


    sedited commented at 11:36 AM on August 14, 2026:

    Nit: Why not keep using empty here?

  21. in src/rpc/mempool.cpp:1048 in 4cae112628
    1052 | +            if (unresolved.size() && (!g_txospenderindex || !g_txospenderindex->BlockUntilSyncedToCurrentChain())) {
    1053 |                  throw JSONRPCError(RPC_MISC_ERROR, "Mempool lacks a relevant spend, and txospenderindex is unavailable.");
    1054 |              }
    1055 |  
    1056 | -            for (const auto& prevout : prevouts_to_process) {
    1057 | +            for (auto& prevout : unresolved) {
    


    sedited commented at 11:37 AM on August 14, 2026:

    Nit: Can this be kept const?

  22. in src/rpc/mempool.cpp:1030 in 4cae112628
    1027 |                  LOCK(mempool.cs);
    1028 |  
    1029 |                  // Make the result if the spending tx appears in the mempool or this is a mempool_only request
    1030 | -                for (auto it = prevouts_to_process.begin(); it != prevouts_to_process.end(); ) {
    1031 | -                    const CTransaction* spending_tx{mempool.GetConflictTx(it->outpoint)};
    1032 | +                for (auto& prevout : prevouts_to_process) {
    


    sedited commented at 11:40 AM on August 14, 2026:

    Nit: Can this be made const?

  23. sedited approved
  24. sedited commented at 11:41 AM on August 14, 2026: contributor

    ACK 4cae112628a96b5889bba7341e4f06f474ada9db

  25. DrahtBot requested review from hodlinator on Aug 14, 2026
  26. in src/rpc/mempool.cpp:1038 in 4cae112628
    1042 | +                    } else {
    1043 | +                        results[prevout.request_index] = make_output(prevout, spending_tx);
    1044 |                      }
    1045 | -
    1046 | -                    result.push_back(make_output(*it, spending_tx));
    1047 | -                    it = prevouts_to_process.erase(it);
    


    hodlinator commented at 12:04 PM on August 14, 2026:

    (Just realized that I didn't react to the quadratic nature of this because I'm somewhat used to a pattern of swapping the element to be erased with the last element of the vector and then popping the vector, making it O(1). The drawback is that it decreases order even more, so not that interesting for this case).

  27. hodlinator approved
  28. hodlinator commented at 12:10 PM on August 14, 2026: contributor

    re-ACK 4cae112628a96b5889bba7341e4f06f474ada9db

    Agree with #35889#pullrequestreview-4936755569 on making the range-for reference elements const. Also have a very slight preference for !v.empty() over v.size() but no big deal.. wish there was something like container::nonempty() or container::any().


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-08-14 17:51 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me