wallet: parallel fast rescan (approx 5x speed up with 16 threads) #34400

pull Eunovo wants to merge 10 commits into bitcoin:master from Eunovo:new-rescan changing 15 files +533 −275
  1. Eunovo commented at 7:46 pm on January 24, 2026: contributor

    EDIT: Some parts of this PR have been split into #34667 and #34681 to ease review. This PR will be put in draft until the previous PRs have been merged.

    This PR uses the ThreadPool to implement parallel fast rescan.

    This PR:

    • Adds a ThreadPool to the WalletContext to be shared by all the wallets
    • Adds -walletpar parameter to configure the number of threads to be used for parallel scanning
    • Updates the wallet_fast_rescan.py test to ensure it catches cases where the FastRescan filter wasn’t properly updated. This is crucial to ensure that changes in the PR do not cause newly added output scripts to be missed.
    • Refactors ScanForWalletTransactions to make the implementation of parallel scanning easier.
    • Implements parallel scanning

    Benchmarks: NOTE: to reproduce, please tune your system with pyperf system tune

    EDIT Set up your node to use block filters by setting blockfilterindex=1 in your bitcoin.conf file and ensure your blockfilterindex is synced to tip before attempting to reproduce.

    Using the following command on mainnet with a wallet with no scripts and hyperfine version 1.20.0:

    0hyperfine --show-output --export-markdown results.md --export-json results.json  \
    1--sort command \
    2--runs 3 \
    3-L commit ef847e8,37d356b \
    4-L num_threads 1,2,4,8,16 \
    5--prepare 'git checkout {commit} && cmake --build build -j 20 && build/bin/bitcoind -blockfilterindex=1 -walletpar={num_threads} && sleep 10 && build/bin/bitcoin-cli loadwalllet <wallet-name>' \
    6--conclude 'build/bin/bitcoin-cli stop && sleep 10' \
    7'build/bin/bitcoin-cli rescanblockchain 500000 900000'
    

    I obtained the following results:

    Command Mean [s] Min [s] Max [s] Relative
    build/bin/bitcoin-cli rescanblockchain 500000 900000 (commit = baseline, num_threads = ..) 536.996 ± 0.722 536.257 537.701 4.64 ± 0.01
    build/bin/bitcoin-cli rescanblockchain 500000 900000 (commit = parallel_scan, num_threads = 1) 540.210 ± 2.696 537.172 542.320 4.67 ± 0.03
    build/bin/bitcoin-cli rescanblockchain 500000 900000 (commit = parallel_scan, num_threads = 2) 358.190 ± 0.515 357.675 358.706 3.10 ± 0.01
    build/bin/bitcoin-cli rescanblockchain 500000 900000 (commit = parallel_scan, num_threads = 4) 230.217 ± 2.321 228.814 232.896 1.99 ± 0.02
    build/bin/bitcoin-cli rescanblockchain 500000 900000 (commit = parallel_scan, num_threads = 8) 151.144 ± 1.748 149.506 152.984 1.31 ± 0.02
    build/bin/bitcoin-cli rescanblockchain 500000 900000 (commit = parallel_scan, num_threads = 16) 115.642 ± 0.305 115.390 115.982 1.00

    System information:

     0Architecture:             x86_64
     1  CPU op-mode(s):         32-bit, 64-bit
     2  Address sizes:          46 bits physical, 48 bits virtual
     3  Byte Order:             Little Endian
     4CPU(s):                   20
     5  On-line CPU(s) list:    0-19
     6Vendor ID:                GenuineIntel
     7  Model name:             Intel(R) Core(TM) Ultra 7 265
     8    CPU family:           6
     9    Model:                198
    10    Thread(s) per core:   1
    11    Core(s) per socket:   1
    12    Socket(s):            20
    13    Stepping:             2
    14    CPU(s) scaling MHz:   41%
    15    CPU max MHz:          4800.0000
    16    CPU min MHz:          800.0000
    17    BogoMIPS:             4761.60
    

    Further benchmarks were performed using a python script with custom chains designed with payments at specified intervals, and the following graph was produced:

    This graph was produced from a laptop with the following CPU specifications:

     0Architecture:                x86_64
     1  CPU op-mode(s):            32-bit, 64-bit
     2  Address sizes:             48 bits physical, 48 bits virtual
     3  Byte Order:                Little Endian
     4CPU(s):                      16
     5  On-line CPU(s) list:       0-15
     6Vendor ID:                   AuthenticAMD
     7  Model name:                AMD Ryzen 9 8945HS w/ Radeon 780M Graphics
     8    CPU family:              25
     9    Model:                   117
    10    Thread(s) per core:      2
    11    Core(s) per socket:      8
    12    Socket(s):               1
    13    Stepping:                2
    14    Frequency boost:         enabled
    15    CPU(s) scaling MHz:      63%
    16    CPU max MHz:             5263.0000
    17    CPU min MHz:             400.0000
    

    All materials for the custom benchmarks can be found here.

    Although not explicitly checked with Valgrind, hyperfine reported that memory usage stayed the same across all runs. I’m not sure to what degree hyperfine’s memory usage report can be trusted, but the PR limits the amount of block hashes that can be held in memory for processing to 1000 (not configurable by the user).

    All benchmarks were performed against https://github.com/bitcoin/bitcoin/pull/34400/commits/37d356bbbe3efed3c7c9e64fae1bac3f4d0ec6eb instead of master because -walletpar is implemented here, and the benchmark scripts would otherwise break, or more complicated scripts would be required to accommodate master.

  2. DrahtBot commented at 7:46 pm on January 24, 2026: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/34400.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    Concept ACK w0xlt, ismaelsadeeq

    If your review is incorrectly listed, please copy-paste <!–meta-tag:bot-skip–> into the comment that the bot should ignore.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #34861 (wallet: Add importdescriptors interface by polespinasa)
    • #34681 (wallet: refactor ScanForWalletTransactions by Eunovo)
    • #34667 (test: ensure FastWalletRescanFilter is correctly updated during scanning by Eunovo)
    • #33008 (wallet: support bip388 policy with external signer by Sjors)
    • #32489 (wallet: Add exportwatchonlywallet RPC to export a watchonly version of a wallet by achow101)
    • #27865 (wallet: Track no-longer-spendable TXOs separately by achow101)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  3. DrahtBot added the label CI failed on Jan 24, 2026
  4. DrahtBot commented at 8:44 pm on January 24, 2026: contributor

    🚧 At least one of the CI tasks failed. Task test max 6 ancestor commits: https://github.com/bitcoin/bitcoin/actions/runs/21320629356/job/61369934184 LLM reason (✨ experimental): Compilation failed due to an unused private member (m_thread_pool) in wallet.h being treated as an error under -Werror.

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  5. Eunovo force-pushed on Jan 25, 2026
  6. DrahtBot removed the label CI failed on Jan 25, 2026
  7. Eunovo renamed this:
    Parallel Fast Rescan (approx 5x speed up with 16 threads)
    wallet: parallel fast rescan (approx 5x speed up with 16 threads)
    on Jan 27, 2026
  8. DrahtBot added the label Wallet on Jan 27, 2026
  9. luke-jr commented at 4:50 am on January 29, 2026: member
    I would have expected rescanning to be I/O bound rather than CPU, in which case parallelization could make things worse (more random seeking). Have you benchmarked this on a non-SSD?
  10. Eunovo commented at 8:42 am on January 29, 2026: contributor

    I would have expected rescanning to be I/O bound rather than CPU, in which case parallelization could make things worse (more random seeking). Have you benchmarked this on a non-SSD?

    Fast rescan checks block filters, which involves considerable hashing. This PR parallelises the checking of block filters, and my benchmarks show considerable improvements in rescan speeds with block filters. Slow rescan, which is I/O bound, remains the same. I expect the speedup to be transferable to non-SSD machines, but I haven’t benchmarked this.

  11. DrahtBot added the label Needs rebase on Feb 4, 2026
  12. in src/wallet/scan.cpp:203 in ef847e8bce outdated
    198+        // If m_max_blockqueue_size blocks have been filtered,
    199+        // stop reading more blocks for now, to give the
    200+        // main scanning loop a chance to update progress
    201+        // and erase some blocks from the queue.
    202+        if (m_continue && completed < m_max_blockqueue_size) m_continue = ReadBlockHash(result);
    203+        else if (!futures.empty()) thread_pool->ProcessTask();
    


    bvbfan commented at 8:59 am on February 8, 2026:
    This slows down the scanning no? All workers already process submit task in its own WorkThread just randomly trying to acquire mutex from scanning thread is non sense to me.

    Eunovo commented at 11:40 am on February 8, 2026:
    Are you referring to the ThreadPool::m_mutex? This mutex is not held during task processing. It is only briefly held to access the work queue. Calling ProcessTask() from the main thread does not slow down scanning; it gives the main thread work to do instead of wasting cycles waiting for results.

    bvbfan commented at 6:29 pm on February 9, 2026:
    Yep it’s not held during task execution, but if main thread do a task, it cannot put new tasks to queue i.e. workers “fight” itself to read something and do nothing. The idea is main thread submit tasks faster than workers could finish to keep all of them busy otherwise there is no difference between 3 and 16 thread (~13 threads do nothing).

    Eunovo commented at 9:34 am on February 10, 2026:
    The main thread intentionally submits only up to WORKERS_COUNT tasks before waiting, rather than continuously submitting. This allows it to pause and update filters whenever a payment is found, preventing unnecessary work on wallets with many transactions packed into a short block range.
  13. Eunovo force-pushed on Feb 8, 2026
  14. DrahtBot removed the label Needs rebase on Feb 8, 2026
  15. Eunovo force-pushed on Feb 11, 2026
  16. Eunovo commented at 5:20 pm on February 11, 2026: contributor
    #33689 has been merged; the cherry-picked Threadpool commit has been removed.
  17. furszy commented at 8:43 pm on February 11, 2026: member

    I like the PR conceptually but I think it would be nice to first improve the current scanning code structure, then land the parallelization feature. The current code mixes a lot responsibilities. Similar to what you did in 633531614f69de49733642fd19cc9eba830fbdea, but into a separate PR so we can first land some good building blocks for this to happen.

    Some quick pseudo-code structuring how I imagine it, which is similar to yours:

     0Scan(wallet, start_block_hash, end_block_hash, fn_filter_block, fn_process_block, interrupt) {
     1     it_current_hash = start_block_hash;
     2
     3     while (it_current_hash != end_block_hash || interrupt) {
     4         // Skip block if needed (this function contains the BlockFilterIndex check if enabled)
     5         if (fn_filter_block(it_current_hash)) continue;
     6    
     7         // (this is more or less how we currently do it, we fetch the block and the next block hash at the same time)
     8         block = chain.find_block(it_current_hash).next_block(it_current_hash);
     9    
    10         // (inside this function the wallet will digest the block update the filter and save progress if needed)
    11         fn_process_block(block);
    12      }
    13}
    
  18. w0xlt commented at 8:12 am on February 21, 2026: contributor
    Concept ACK
  19. test/wallet: ensure FastWalletRescanFilter is updated during scanning
    The fixed non-range descriptor address ensured that the FastWalletRescanFilter would match all Blocks even if the filter wasn't properly updated.
    This commit moves the non-range descriptor tx to a different block, so that the filters must be updated after each TopUp for the test to pass.
    90d347b40b
  20. wallet: move scanning logic to wallet/scan.cpp
    Move rescan logic to new class to allow the scanning loop
    to be simplified by delegating some logic to member
    functions in future commits.
    
    CWallet::ScanForWalletTransactions impl is moved to scan.cpp
    to prevent circular dependency of the form
    "wallet/wallet -> wallet/scan -> wallet/wallet".
    81ec69cbbf
  21. wallet/scan: extract block filtering logic to ShouldFetchBlock method
    Pure extraction of block filter matching logic into a dedicated method.
    This prepares for further refactoring of the scanning loop.
    6e662fe9e1
  22. wallet/scan: extract block scanning logic to ScanBlock method
    Pure extraction of block transaction processing logic into a dedicated
    method. This isolates the logic for fetching blocks and syncing their
    transactions.
    34d4cd27e3
  23. wallet/scan: extract block iteration logic into ReadNextBlock
    Extract block reading logic into ReadNextBlock method which returns
    std::optional<pair<hash, height>>.
    Introduce m_next_block member to track iteration state, consolidating
    the loop termination logic into ReadNextBlock.
    6909533e4a
  24. wallet/scan: extract progress tracking helper methods
    Extract UpdateProgress, UpdateTipIfChanged, and LogScanStatus methods.
    Move progress tracking variables to class members. This simplifies the
    main scanning loop and groups related progress tracking logic together.
    9f02e8b067
  25. wallet/scan: simplify ChainScanner::Scan main loop
    Extract ProcessBlock method and simplify the main scanning loop. This
    final refactoring demonstrates the clean separation of concerns with
    each helper method handling a specific aspect of the scanning process.
    e392eb5f93
  26. wallet: setup wallet threadpool
    All wallets will use the same ThreadPool owned by the WalletContext.
    ThreadPool::Submit() is threadsafe so there's no need for the use of
    external synchronization primitives when submiting tasks.
    All threads started by the ThreadPool will be destroyed with the
    ThreadPool and WalletContext.
    881ebc4730
  27. wallet/scan: combine block iteration and filtering in ReadNextBlocks
    This commit refactors the block filtering logic from ShouldFetchBlock
    into a new ReadNextBlocks method that works with a queue of blocks
    (m_blocks). This prepares the code for parallel block filter checking
    while keeping the current single-threaded behaviour.
    555ea0148e
  28. Eunovo force-pushed on Feb 26, 2026
  29. Eunovo commented at 4:21 pm on February 26, 2026: contributor

    I like the PR conceptually but I think it would be nice to first improve the current scanning code structure, then land the parallelization feature. The current code mixes a lot responsibilities. Similar to what you did in 6335316, but into a separate PR so we can first land some good building blocks for this to happen.

    I moved the test change into #34667 and the ScanForWalletTransactions refactor into #34681. I’ll be putting this PR in draft while those PRs are open.

  30. Eunovo marked this as a draft on Feb 26, 2026
  31. ismaelsadeeq commented at 2:34 pm on March 13, 2026: member

    Concept ACK

    I attempted to reproduce the benchmarks for this But I did not use hyperfine, I used time

    System: AMD Ryzen 7 7700, 16 cores, 64GB RAM, mainnet, blocks 500000–900000, 3 runs averaged.

    Baseline (881ebc4730ad15bd26e3e32ee3c9ba9d6e05552d): single-threaded fast rescan, -walletpar has no effect — consistently ~283s regardless of thread count.

    Parallel scan (48154b87e2fb303ca0f3d46da29a3fbbf8758a06): fast rescan with threadpool parallelism enabled via -walletpar.

    num_threads baseline parallel speedup
    1 282.618s 288.214s 0.98x (slight overhead)
    2 282.618s 187.485s 1.51x
    4 282.618s 119.575s 2.37x
    8 282.618s 81.565s 3.47x
    16 282.618s 62.965s 4.53x
    • Speedup scales well up to 16 threads, going from ~283s down to ~63s — a 4.5x improvement.
    • Results are very consistent across runs (low variance), because the machine is bare metal and no other running processes apart from bitcoind are present during the benchmark runs.

    Steps to reproduce

    1. Restart bitcoind with -blockfilterindex=1 till it’s done.
    2. create a new wallet test
    3. Stop the node
    4. Save the script below as bench_script.sh
     0#!/usr/bin/env bash
     1set -euo pipefail
     2
     3COMMITS=(881ebc4730ad15bd26e3e32ee3c9ba9d6e05552d 48154b87e2fb303ca0f3d46da29a3fbbf8758a06)
     4THREADS=(1 2 4 8 16)
     5RUNS=3
     6WALLET_NAME="test"
     7DATADIR="$HOME/.bitcoin"
     8RESULTS_CSV="results.csv"
     9RESULTS_MD="results.md"
    10
    11echo "commit,num_threads,run,seconds" > "$RESULTS_CSV"
    12
    13for commit in "${COMMITS[@]}"; do
    14  short="${commit:0:7}"
    15  git checkout "$commit"
    16  cmake --build build -j 20
    17
    18  for num_threads in "${THREADS[@]}"; do
    19    echo "=== commit=$short num_threads=$num_threads ==="
    20
    21    for run in $(seq 1 $RUNS); do
    22      echo "  run $run/$RUNS"
    23
    24      # start node
    25      build/bin/bitcoind -blockfilterindex=1 -walletpar="$num_threads" -daemonwait
    26      build/bin/bitcoin-cli loadwallet "$WALLET_NAME"
    27
    28      build/bin/bitcoin-cli rescanblockchain 500000 900000
    29
    30      # parse timing from debug log: "Rescan completed in 284737ms"
    31      ms=$(grep "Rescan completed in" "$DATADIR/debug.log" | tail -1 | grep -oP '\d+(?=ms)')
    32      seconds=$(python3 -c "print(f'{$ms / 1000:.3f}')")
    33
    34      echo "$short,$num_threads,$run,$seconds" >> "$RESULTS_CSV"
    35      echo "  -> ${seconds}s"
    36
    37      build/bin/bitcoin-cli stop
    38      # wait for clean shutdown
    39      while build/bin/bitcoin-cli ping 2>/dev/null; do sleep 1; done
    40      sleep 5
    41    done
    42  done
    43done
    44
    45# markdown table with averages
    46python3 - "$RESULTS_CSV" "$RESULTS_MD" <<'EOF'
    47import sys, csv
    48from collections import defaultdict
    49
    50infile, outfile = sys.argv[1], sys.argv[2]
    51
    52rows = list(csv.DictReader(open(infile)))
    53groups = defaultdict(list)
    54for r in rows:
    55    groups[(r['commit'], r['num_threads'])].append(float(r['seconds']))
    56
    57with open(outfile, 'w') as f:
    58    f.write("| commit | num_threads | run1 | run2 | run3 | mean |\n")
    59    f.write("|--------|-------------|------|------|------|------|\n")
    60    for (commit, threads), times in sorted(groups.items()):
    61        mean = sum(times) / len(times)
    62        runs = " | ".join(f"{t:.3f}" for t in times)
    63        f.write(f"| {commit} | {threads} | {runs} | {mean:.3f} |\n")
    64
    65print(f"Written {outfile}")
    66EOF
    67
    68echo "Done. Results in $RESULTS_CSV and $RESULTS_MD"
    
    1. Make the script executable chmod +x bench_script.sh
    2. Execute the script ./bench_script.sh
    3. You can go a step further by using a top like btop https://github.com/aristocratos/btop to monitor the resource usage and how it will be well utilized when rescanning in parallel.

    The current steps to reproduce in the description are stale because the commit hashes have changed since your force pushes.

  32. in src/wallet/scan.cpp:191 in 48154b87e2
    195+        // Submit jobs to the threadpool in batches of at most `workers_count` size.
    196+        // This prevents over-submission: if we queued all jobs upfront and the filtered
    197+        // block range is smaller than expected, worker threads would process blocks
    198+        // that get discarded, wasting CPU cycles.
    199+        const size_t job_gap = workers_count - futures.size();
    200+        if (job_gap > 0 && i < m_blocks.size()) {
    


    rkrux commented at 2:03 pm on March 16, 2026:

    In 48154b87e2fb303ca0f3d46da29a3fbbf8758a06 “wallet: check blockfilters in parallel”

    These two conditions in this check seem redundant with the same two conditions in the following for loop.


    Eunovo commented at 1:43 pm on March 18, 2026:
    Fixed.
  33. in src/wallet/scan.cpp:194 in 48154b87e2
    198+        // that get discarded, wasting CPU cycles.
    199+        const size_t job_gap = workers_count - futures.size();
    200+        if (job_gap > 0 && i < m_blocks.size()) {
    201+            for (size_t j = 0; j < job_gap && i < m_blocks.size(); ++j, ++i) {
    202+                auto block = m_blocks[i];
    203+                futures.emplace_back(*thread_pool->Submit([&filter, block = std::move(block)]() {
    


    rkrux commented at 2:08 pm on March 16, 2026:

    In 48154b87e2fb303ca0f3d46da29a3fbbf8758a06 “wallet: check blockfilters in parallel”

    So it appears this is a flow where multiple tasks can be submitted in one go. There is an overload method of Submit that accepts a range of tasks & pushes all of them in the queue within one acquisition of queue lock, while notifying all the waiting workers. I think this workflow can be benefitted with this method, an untested code snippet is below because this branch is not rebased over master that contains the ranged overload.

    https://github.com/bitcoin/bitcoin/blob/ff7cdf633e375f151cccbcc78c7add161b3d29b8/src/util/threadpool.h#L199-L220

     0diff --git a/src/wallet/scan.cpp b/src/wallet/scan.cpp
     1index 6b776930f1..2295a242a1 100644
     2--- a/src/wallet/scan.cpp
     3+++ b/src/wallet/scan.cpp
     4@@ -147,6 +147,20 @@ std::optional<std::pair<size_t, size_t>> ChainScanner::ReadNextBlocks(const std:
     5         return std::make_pair<size_t, size_t>(0, m_blocks.size());
     6     }
     7     filter->UpdateIfNeeded();
     8+
     9+    auto block_matcher = [&filter](uint256 block_hash) {
    10+        const auto matches_block{filter->MatchesBlock(block_hash)};
    11+        if (matches_block.has_value()) {
    12+            if (*matches_block) {
    13+                return FilterRes::FILTER_MATCH;
    14+            } else {
    15+                return FilterRes::FILTER_NO_MATCH;
    16+            }
    17+        } else {
    18+            return FilterRes::FILTER_NO_FILTER;
    19+        }
    20+    }
    21+
    22     auto* thread_pool = m_wallet.m_thread_pool;
    23     // ThreadPool pointer should never be null here
    24     // during normal operation because it should
    25@@ -187,23 +201,13 @@ std::optional<std::pair<size_t, size_t>> ChainScanner::ReadNextBlocks(const std:
    26         // This prevents over-submission: if we queued all jobs upfront and the filtered
    27         // block range is smaller than expected, worker threads would process blocks
    28         // that get discarded, wasting CPU cycles.
    29-        const size_t job_gap = workers_count - futures.size();
    30-        if (job_gap > 0 && i < m_blocks.size()) {
    31-            for (size_t j = 0; j < job_gap && i < m_blocks.size(); ++j, ++i) {
    32-                auto block = m_blocks[i];
    33-                futures.emplace_back(*thread_pool->Submit([&filter, block = std::move(block)]() {
    34-                    const auto matches_block{filter->MatchesBlock(block.first)};
    35-                    if (matches_block.has_value()) {
    36-                        if (*matches_block) {
    37-                            return FilterRes::FILTER_MATCH;
    38-                        } else {
    39-                            return FilterRes::FILTER_NO_MATCH;
    40-                        }
    41-                    } else {
    42-                        return FilterRes::FILTER_NO_FILTER;
    43-                    }
    44-                }));
    45+        auto to_submit_jobs_count = std::min(workers_count - futures.size(), m_blocks - i);
    46+        if (to_submit_jobs_count) {
    47+            std::vector<std::function<FilterRes(uint256)>> block_matchers;
    48+            for (; i < to_submit_jobs_count; ++i) {
    49+                block_matchers.emplace_back(block_matcher(m_blocks[i].first));
    50             }
    51+            futures.emplace_back(*thread_pool->Submit(std::move(block_matchers)));
    52         }
    53 
    54         // If m_max_blockqueue_size blocks have been filtered,
    

    Eunovo commented at 12:18 pm on March 18, 2026:
    I will check this when I rebase on master.
  34. in src/wallet/scan.cpp:220 in 48154b87e2 outdated
    224+            if (next_block) m_blocks.emplace_back(*next_block);
    225+        }
    226+        else if (!futures.empty()) {
    227+            // Join work processing instead of waiting idly.
    228+            thread_pool->ProcessTask();
    229+        }
    


    rkrux commented at 2:19 pm on March 16, 2026:

    In 48154b8 “wallet: check blockfilters in parallel”

    I’m doubtful that putting the controller (non-worker) thread to process the threadpool tasks is helpful.

    This threadpool is shared across wallets. It could be the case that multiple RPCs of different wallets could be called simultaneously. ProcessTask picks the first item from a shared queue in the threadpool. Can’t it happen that this controller thread picks up the task of another RPC of another wallet, thereby distorting results (from a RPC latency point of view) of this one?


    Eunovo commented at 12:05 pm on March 18, 2026:

    I’m doubtful that putting the controller (non-worker) thread to process the threadpool tasks is helpful.

    It is helpful. I had better results when I put the thread to work vs when I didn’t.

    Can’t it happen that this controller thread picks up the task of another RPC of another wallet, thereby distorting results (from a RPC latency point of view) of this one?

    True, this creates an argument to ditch the shared threadpool and create one at the begining of the scan process; the same way we initialise script threads in ConnectBlock


    rkrux commented at 12:16 pm on March 18, 2026:

    I was doubtful of the helpfulness of using processtask because of this cross-wallet/rpc operation.

    this creates an argument to ditch the shared threadpool and create one at the begining of the scan process

    Interesting, I had thought of not using the processtask function in the current thread pool setup as a way. But an intra-wallet scan operation specific threadpool seems like a good alternative to consider. I wll think of its implications.

  35. in src/wallet/scan.cpp:183 in 48154b87e2
    187+            }
    188 
    189-    const auto& [block_hash, block_height] = m_blocks[0];
    190-    auto matches_block{filter->MatchesBlock(block_hash)};
    191+            if (!range.has_value()) range = std::make_pair(current_block_index, current_block_index + 1);
    192+            else range->second = current_block_index + 1;
    


    rkrux commented at 2:57 pm on March 16, 2026:

    In 48154b8 “wallet: check blockfilters in parallel”

    It appears that current_block_index + 1 is equal to completed due to a int current_block_index = completed - 1; above.


    Eunovo commented at 1:43 pm on March 18, 2026:
    Fixed.
  36. rkrux commented at 3:05 pm on March 16, 2026: contributor
    I’ve looked only at the 48154b8 “wallet: check blockfilters in parallel” commit partially.
  37. wallet: check blockfilters in parallel
    This commit implements parallel block filter checking using the wallet
    threadpool. The main thread reads block hashes and queues them while
    worker threads check filters in parallel.
    
    Synchronization:
    - Operations requiring cs_wallet (GetLastBlockHeight, SyncTransaction)
      remain on the main thread since cs_wallet is a RecursiveMutex and
      ScanForWalletTransactions is called from AttachChain which locks cs_wallet
    - Main thread uses ThreadPool::ProcessTask() to join workers when the
      block queue is full, avoiding busy-waiting
    
    Batching:
    - Up to m_max_blockqueue_size (1000) blocks are queued for filtering
    - When queue is full, main loop processes filtered blocks before reading more
    Thread safety:
    - All futures (at most `workers_count`)  are waited on before returning to
      avoid data races on `FastWalletRescanFilter::m_filter_set`.
    
    Benchmarks show considerable improvement (approx 5x with 16 threads).
    32609391d5
  38. Eunovo force-pushed on Mar 18, 2026
  39. achow101 referenced this in commit 696b5457c5 on Mar 24, 2026
  40. DrahtBot commented at 1:40 am on March 24, 2026: contributor
    🐙 This pull request conflicts with the target branch and needs rebase.
  41. DrahtBot added the label Needs rebase on Mar 24, 2026

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-03-24 03:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me