validation: fetch block inputs on parallel threads >40% faster IBD

andrewtoth commented at 2:40 pm on October 22, 2024: contributor

This PR parallelizes fetching all input prevouts of a block right before block connection, achieving up to 46% faster IBD performance¹²³⁴.

Problem

Currently, when fetching inputs in ConnectBlock, each input is fetched from the cache sequentially. A cache miss requires a round trip to the disk database to fetch the outpoint and insert it into the cache. Since the database is read-only during ConnectBlock, we can fetch all inputs of a block in parallel on multiple threads while connecting.

Solution

We introduce a new CoinsViewCacheAsync CCoinsViewCache subclass that manages worker threads to fetch block inputs in parallel. The block is passed to the CoinsViewCacheAsync view before entering ConnectBlock, which kicks off the worker threads to begin fetching the inputs. The view is then passed to ConnectBlock since it provides the same API as CCoinsViewCache. The cache returns fetched coins as they become available via the overridden FetchCoin method. If not available yet, the main thread also fetches coins as it waits.

Implementation Details

The CoinsViewCacheAsync implements a lock-free MPSC (Multiple Producer, Single Consumer) queue design:

Work Distribution: Collects all input prevouts into a queue and uses a barrier to start all worker threads simultaneously
Synchronization: Worker threads use an atomic counter to claim which inputs to fetch, and each input has an atomic flag to signal completion to the main thread
Main Thread Processing: The main thread waits for inputs in order and moves their Coin into the cacheCoins map as they become available.
Work Stealing: If the main thread catches up to the workers, it assists with fetching to maximize parallelism
Completion: All threads synchronize on a barrier via a Reset method, which ensures all threads are parked before proceeding

Safety and Correctness

The CoinsViewCacheAsync works on a block that has not been fully validated, but it does not interfere or modify any of the validation during ConnectBlock
It simply fetches inputs in parallel, which must be fetched before a transaction is validated anyways
Invalid blocks: If an invalid block is mined, the temporary cache is discarded without being flushed. This is an improvement over the current behavior, where inputs are inserted into the main CoinsTip() cache when pulled through for an invalid block

Performance

Benchmarks show up to 46% faster IBD performance¹²³⁴. The parallelization of expensive disk lookups provides significant speedup.

Credits

Inspired by this comment.

#31132#pullrequestreview-3515011880 ↩︎ ↩︎
#31132 (comment) ↩︎ ↩︎
#31132#pullrequestreview-3347436866 ↩︎ ↩︎
#31132 (comment) ↩︎ ↩︎

DrahtBot commented at 2:40 pm on October 22, 2024: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31132.

Reviews

See the guideline for information on the review process.

Type	Reviewers
Concept ACK	ismaelsadeeq, Raimo33, l0rinc

If your review is incorrectly listed, please copy-paste <!–meta-tag:bot-skip–> into the comment that the bot should ignore.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#33866 (refactor: Let CCoinsViewCache::BatchWrite return void by TheCharlatan)
#32317 (kernel: Separate UTXO set access from validation functions by TheCharlatan)
#30342 (kernel, logging: Pass Logger instances to kernel objects by ryanofsky)
#29700 (kernel, refactor: return error status on all fatal errors by ryanofsky)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

DrahtBot added the label Validation on Oct 22, 2024

andrewtoth force-pushed on Oct 22, 2024

DrahtBot commented at 2:45 pm on October 22, 2024: contributor

🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/31894441286

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

DrahtBot added the label CI failed on Oct 22, 2024

andrewtoth renamed this:
~~validation: fetch block inputs parallel threads ~17% faster IBD~~
validation: fetch block inputs on parallel threads ~17% faster IBD
on Oct 22, 2024

andrewtoth force-pushed on Oct 22, 2024

DrahtBot removed the label CI failed on Oct 22, 2024

in src/inputfetcher.h:169 in e9e23b59f8 outdated

146+        : m_batch_size(batch_size)
147+    {
148+        m_worker_threads.reserve(worker_thread_count);
149+        for (size_t n = 0; n < worker_thread_count; ++n) {
150+            m_worker_threads.emplace_back([this, n]() {
151+                util::ThreadRename(strprintf("inputfetch.%i", n));

l0rinc commented at 10:26 am on October 23, 2024:

Q: Is this a leftover a hack for non-owning LevelDB threads, or is this really the best way to name threads in a cross-platform way?

andrewtoth commented at 1:49 pm on October 23, 2024:

Unsure, copied from CScriptCheck. If the state of the art of thread naming has advanced since that was written, please let me know!

sipa commented at 1:54 pm on October 23, 2024:

The C++ standard library does as far as I know have no way of renaming threads at all. src/util/threadnames.{h,cpp} is our wrapper around the various platform-dependent ways of doing so on supported systems.

l0rinc commented at 2:03 pm on October 23, 2024:

Thank you, please resolve the comment.

in src/inputfetcher.h:124 in e9e23b59f8 outdated

184+                    continue;
185+                }
186+
187+                buffer.emplace_back(outpoint);
188+                if (buffer.size() == m_batch_size) {
189+                    Add(std::move(buffer));

l0rinc commented at 11:29 am on October 23, 2024:

We’re mostly creating the buckets randomly here, so each thread will need access to basically all of the keys. Since we have an idea of how LevelDB works here (i.e. Sorted String Table), we could likely improve cache locality (would likely be most beneficial on HDDs) and minimize lock contention by splitting the reads by sorted transactions instead.

andrewtoth commented at 1:46 pm on October 23, 2024:

I don’t think there is any lock contention here if we are doing multithreaded reading?

I also think what you’re suggesting would add a lot more complexity to this PR, when this is “good enough”.

l0rinc commented at 2:02 pm on October 23, 2024:

This might be as simple as sorting by tx before we create the buckets.

andrewtoth commented at 2:12 pm on October 23, 2024:

If a benchmark shows that it is better, then great!

in src/inputfetcher.h:90 in e9e23b59f8 outdated

68+        std::vector<std::pair<COutPoint, Coin>> pairs{};
69+        do {
70+            std::vector<COutPoint> outpoints{};
71+            outpoints.reserve(m_batch_size);
72+            {
73+                WAIT_LOCK(m_mutex, lock);

l0rinc commented at 11:30 am on October 23, 2024:

I’m wondering if we really need to (b)lock here or whether we could we create a read-only snapshot instead and avoid stalling?

andrewtoth commented at 1:34 pm on October 23, 2024:

This is blocking so we can access the queue of shared outpoints that we need to fetch from. It is not blocking for LevelDB, we access the db once we are out of the critical section.

l0rinc commented at 2:04 pm on October 23, 2024:

As mentioned before, why do we need shared outpoints here?

andrewtoth commented at 2:15 pm on October 23, 2024:

The main thread adds all outpoints to a global vector, which all workers will fetch their work from.

andrewtoth commented at 0:56 am on December 4, 2024:

We no longer need to block on the shared outpoints vector. We write to it once in the main thread before notifying the other threads and then only read from it afterwards.

in src/inputfetcher.h:39 in e9e23b59f8 outdated

24+ * onto the queue, where they are fetched by N worker threads. The resulting
25+ * coins are pushed onto another queue after they are read from disk. When
26+ * the main is done adding outpoints, it starts writing the results of the read
27+ * queue to the cache.
28+ */
29+class InputFetcher

l0rinc commented at 11:39 am on October 23, 2024:

I know it’s not trivial request, but can we add a test for this class which fetches everything in parallel and sequentially and assert that the result is equivalent? And preferably also a benchmark, like we have it for https://github.com/bitcoin/bitcoin/blob/master/src/bench/checkqueue.cpp. I would gladly help here, if needed.

andrewtoth commented at 1:35 pm on October 23, 2024:

Yes, I can add these but I am waiting for some more conceptual support.

andrewtoth commented at 3:06 pm on November 7, 2024:

Added tests and benchmark. The test has random parameters, one of which would be end up having a single worker thread.

andrewtoth commented at 8:59 pm on November 16, 2024:

Also added fuzz harness

in src/inputfetcher.h:145 in e9e23b59f8 outdated

140+    }
141+
142+
143+public:
144+    //! Create a new input fetcher
145+    explicit InputFetcher(size_t batch_size, size_t worker_thread_count) noexcept

l0rinc commented at 11:46 am on October 23, 2024:

For consistency (see: explicit CCheckQueue(unsigned int batch_size, int worker_threads_num)) and simplicity (m_input_fetcher{/*batch_size=*/128, static_cast<size_t>(options.worker_threads_num)}, and to follow modern C++ directions where sizes seem to be preferred as signed values, see: #30927 (review)), please consider making these int(s) instead.

andrewtoth commented at 3:06 pm on November 7, 2024:

Done.

in src/validation.cpp:6257 in e9e23b59f8 outdated

6247@@ -6243,6 +6248,7 @@ static ChainstateManager::Options&& Flatten(ChainstateManager::Options&& opts)
6248 
6249 ChainstateManager::ChainstateManager(const util::SignalInterrupt& interrupt, Options options, node::BlockManager::Options blockman_options)
6250     : m_script_check_queue{/*batch_size=*/128, options.worker_threads_num},
6251+      m_input_fetcher{/*batch_size=*/128, static_cast<size_t>(options.worker_threads_num)},

l0rinc commented at 12:07 pm on October 23, 2024:

Unlike the script checks, these fetches aren’t CPU bound, there is no reason to provide the number of CPUs as the number of parallels threads. I don’t know if we care about HDD performance here or not, but we can likely find a multiplier that makes this better for both SSD and HDD.

Quoting from https://pkolaczk.github.io/disk-parallelism:

It was surprising to me that even 64 threads, which are far more than the number of CPU cores (4 physical, 8 virtual), still improved the performance. I guess that with requests of such a small size to such a fast storage, you need to submit really many of them to keep the SSD busy.

If we can provide a benchmark for this usecase we can likely find an optimal multiplier here - I won’t nack but this part is very important for me.

andrewtoth commented at 1:43 pm on October 23, 2024:

Adding more threads will require more memory, which is one reason to not use many more.

I did a benchmark using 64 threads on the same 16 vcore machine, and it was slightly slower :/

l0rinc commented at 2:00 pm on October 23, 2024:

4x may be too much to begin with, but 1.5-2x sounds plausible, I’ll help with benchmarking this once my current batches finish.

andrewtoth commented at 3:06 pm on November 7, 2024:

Added a benchmark to experiment with these.

in src/inputfetcher.h:123 in e9e23b59f8 outdated

183+                if (cache.HaveCoinInCache(outpoint)) {
184+                    continue;
185+                }
186+
187+                buffer.emplace_back(outpoint);
188+                if (buffer.size() == m_batch_size) {

l0rinc commented at 12:10 pm on October 23, 2024:

Would it be possible to create the batch sizes dynamically? Since the number of missing values differs for every block (and every dbcache size), it may not make more sense to calculate the optimal split instead of using the random value of 128. Coroutines might alleviate this problem.

andrewtoth commented at 1:42 pm on October 23, 2024:

I’m not sure it would warrant the complexity I think this batch size is “good enough” for now. In a follow up we could maybe add ways to set this with configs to experiment if there really is more optimal settings.

andrewtoth commented at 3:05 pm on November 7, 2024:

I changed the batch size to be number of workers.

in src/inputfetcher.h:132 in e9e23b59f8 outdated

192+                }
193+            }
194+            txids.insert(tx->GetHash());
195+        }
196+
197+        Add(std::move(buffer));

l0rinc commented at 12:11 pm on October 23, 2024:

Do we always have leftovers or will this process the last batch twice (or process an empty one) if the batch happens to be divisible by batch_size?

andrewtoth commented at 1:37 pm on October 23, 2024:

It won’t process twice, but it could pass in an empty vector, which is ignored if you look at Add implementation.

in src/inputfetcher.h:70 in e9e23b59f8 outdated

60+
61+    std::vector<std::thread> m_worker_threads;
62+    bool m_request_stop GUARDED_BY(m_mutex){false};
63+
64+    /** Internal function that does the fetching from disk. */
65+    void Loop() noexcept EXCLUSIVE_LOCKS_REQUIRED(!m_mutex)

l0rinc commented at 12:35 pm on October 23, 2024:

We’re basically mimicking RocksDB’s MultiGet here - but prewarming the cache instead in separate get requests, since we can’t really access LevelDB’s internals.

Since splitting into buckets isn’t trivial and since MultiGet seems to rely on C++20 coroutines (which wasn’t available in 2012 when CCheckQueue was written), I’m wondering how much simpler this fetching would be if we had lightweight suspendible threads instead: https://rocksdb.org/blog/2022/10/07/asynchronous-io-in-rocksdb.html#multiget

andrewtoth commented at 1:40 pm on October 23, 2024:

I think it would be similar in complexity, we would still need all the locking mechanisms to prevent multithreaded access.

What would really be great is if we had a similar construction to Rust’s std::sync::mpsc.

l0rinc commented at 1:58 pm on October 23, 2024:

Can you tell me why we need to prevent multithreaded access exactly? We could collect the values to different vectors, each one accessed only by a single thread and merge them into the cache at the end on a single thread, right?

How would mpsc solve this better? Do you think we need work stealing to make it perfectly parallel? Wouldn’t coroutines already achieve the same?

sipa commented at 2:07 pm on October 23, 2024:

I haven’t yet experimented with them, but as far as I understand it, coroutines are just programming paradigm, not magic; they don’t do anything of their own, besides making things that were already possible easier to write. In particular, you still need a thread pool or some mechanism for scheduling how to run them,

andrewtoth commented at 2:09 pm on October 23, 2024:

We could collect the values to different vectors, each one accessed only by a single thread and merge them into the cache at the end on a single thread

If the vectors are thread local, then how can the main thread access them at the end to write them? We also want to be writing throughout while the workers are fetching, not just at the end.

How would mpsc solve this better?

Instead of each worker thread having a local queue of results, which they then append to the global results queue, they could just push each result to the channel individually. The main thread could just pull results off the channel as they arrive, instead of waiting to be awoken by a worker thread that appended all its results to the global queue.

work stealing

That is a concept for async rust, or std::async::mpsc. We can do all this without introducing an async runtime. But, this is getting off topic.

l0rinc commented at 3:15 pm on October 23, 2024:

coroutines are just programming paradigm, not magic

That’s also what I was counting on! :D

In RocksDB they have high and low priority work (I assume that’s just added to the front or the back of a background work deque) – this could align well with @furszy’s suggestion for mixing different kinds of background work units.

I haven’t used the C++ variant of coroutines either, but my thinking was that since they can theoretically yield execution when waiting for IO (and resume later), this would allow threads to focus on other tasks in the meantime. Combined with an appropriate scheduling mechanism (such as a thread pool), we could maximize both CPU and IO usage, if I’m not mistaken. Instead of each thread handling just one task, it could suspend a coroutine while waiting on IO (e.g., a database fetch) and resume it later, effectively maximizing CPU and IO work without needing to know the exact details of the work.

If the vectors are thread local

The vector would still be global, but each thread would only access a single bucket (i.e. global vector of vectors, with each thread from the pool writing only to vector[thread_id], which contains a vector of fetched coins). When all the work is finished, we’d iterate over the global vector and merge the results into the cache on a single thread. As mentioned, sorting the outpoints before fetching could help improve data locality and reduce lock contention, and the coroutines above would help with work stealing, ensuring that all threads finish roughly at the same time.

Is there anything prohibiting us from doing something like this to minimize synchronization and lock contention during the fetch phase? I understand some synchronization would still be needed during the merge, but this could help reduce global locks and unnecessary synchronization throughout the process.

sipa commented at 3:28 pm on October 23, 2024:

I haven’t used the C++ variant of coroutines either, but my thinking was that since they can theoretically yield execution when waiting for IO (and resume later), this would allow threads to focus on other tasks in the meantime.

That needs async I/O, and is unrelated to coroutines, as far as I understand it. Coroutines just help with keeping track of what to do when the reads come back inside rocksdb.

As long as LevelDB (or whatever database engine we use) internally does not use async I/O, there will be one (waiting) thread per parallel outstanding read request from the database.

HowHsu commented at 4:06 pm on June 28, 2025:

Is there anything prohibiting us from doing something like this to minimize synchronization and lock contention during the fetch phase? I understand some synchronization would still be needed during the merge, but this could help reduce global locks and unnecessary synchronization throughout the process.

As far as I know, the advantage of coroutines over threads is faster context switching, since it doesn’t go through the operating system kernel. This advantage only becomes apparent under extremely high concurrency, such as hundreds of thousands of concurrent tasks. Using coroutines does not eliminate the need for synchronization mechanisms where they are inherently required.

l0rinc changes_requested

l0rinc commented at 12:44 pm on October 23, 2024: contributor

Concept ACK

I’m still missing tests and benchmarks here and I think we need to find better default values for SSD and HDD parallelism, and I’d be interested in how coroutines would perform here instead of trying to find the best batching size manually.

furszy commented at 2:25 pm on October 23, 2024: member

Cool idea.

Since the inputs fetcher call is blocking, instead of creating a new set of worker threads, what do you think about re-using the existing script validation ones (or any other unused worker threads) by implementing a general-purpose thread pool shared among the validation checks? The script validation checks and the inputs fetching mechanism are never done concurrently because you need the inputs in order to verify the scripts. So, they could share workers.

This should be benchmarked because it might add some overhead but, #26966 introduces such structure inside 401f21bfd72f32a28147677af542887518a4dbff, which we could pull off and use for validation.

andrewtoth commented at 2:48 pm on October 23, 2024: contributor

implementing a general-purpose thread pool shared among the validation checks?

Nice, yes that would be great! That would simplify this PR a lot if it could just schedule tasks on worker threads and receive the responses, instead of implementing all the sync code itself.

#26966 introduces such structure inside https://github.com/bitcoin/bitcoin/commit/401f21bfd72f32a28147677af542887518a4dbff, which we could pull off and use for validation.

Concept ACK!

l0rinc commented at 6:12 pm on October 24, 2024: contributor

Finished benching on a HDD until 860k on Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz, CPU = 8:

0Summary
1'COMMIT=f278ca4ec3f0a90c285e640f1a270869ca594d20 ./build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=860000 -dbcache=10000 -printtoconsole=0' ran
2 1.02 times faster than 'COMMIT=e9e23b59f8eedb8dfae75aa660328299fba92b50 ./build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=860000 -dbcache=10000 -printtoconsole=
30'

f278ca4ec3 coins: allow emplacing non-dirty coins internally (39993.343777768874 seconds = 11.1 hours) e9e23b59f8 validation: fetch block inputs in parallel (40929.84310861388 seconds = 11.3 hours)

~~So likely on HDD we shouldn’t use so many threads, apparently it slows down IBD.~~ Maybe we could add a new config option (iothreads or iothreadmultiplier or something). The defaults should likely depend on whether it’s an SSD or HDD.

Edit:

0"command": "COMMIT=f278ca4ec3f0a90c285e640f1a270869ca594d20 ./build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=860000 -dbcache=10000 -printtoconsole=0",
1"times": [39993.343777768874],
2
3"command": "COMMIT=e9e23b59f8eedb8dfae75aa660328299fba92b50 ./build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=860000 -dbcache=10000 -printtoconsole=0",
4"times": [40929.84310861388],

I have retried the same with half the parallelism (rebased, but no other change in the end, otherwise the results would be hard to interpret):

0"command": "COMMIT=8207d372b2fac24af0f8999b30e71e88d40b3a13 ./build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=860000 -dbcache=10000 -printtoconsole=0",
1"times": [40579.00445769842],

So it’s a tiny bit faster than before (surprisingly stable for an actual IBD with real peers), but still slower-than/same-as before, so not sure why it’s not faster.

Edit:

Running it on a HDD with a low dbcache value reproduces the original result:

0hyperfine --runs 1 --show-output --export-json /mnt/my_storage/ibd_full-threaded-inputs-3.json --parameter-list COMMIT 92fc718592be55812b2c73a3bf57599fc81425fa,8207d372b2fac24af0f8999b30e71e88d40b3a13 --prepare 'rm -rf /mnt/my_storage/BitcoinData/* && git checkout {COMMIT} && git clean -fxd && git reset --hard && cmake -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_UTIL=OFF -DBUILD_TX=OFF -DBUILD_TESTS=OFF -DENABLE_WALLET=OFF -DINSTALL_MAN=OFF && cmake --build build -j$(nproc)' 'COMMIT={COMMIT} ./build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=860000 -dbcache=1000 -printtoconsole=0'

08207d372b2 validation: fetch block inputs in parallel
192fc718592 coins: allow emplacing non-dirty coins internally
2Summary
3  'COMMIT=8207d372b2fac24af0f8999b30e71e88d40b3a13 ./build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=860000 -dbcache=1000 -printtoconsole=0' ran
4    1.16 times faster than 'COMMIT=92fc718592be55812b2c73a3bf57599fc81425fa ./build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=860000 -dbcache=1000 -printtoconsole=0'

andrewtoth commented at 6:31 pm on October 24, 2024: contributor

So likely on HDD we shouldn’t use so many threads, apparently it slows down IBD.

I’m not sure we can conclude that from your benchmark. It used a very high dbcache setting, which makes the effect of this change less important. It also is syncing from untrusted network peers, so there is some variance which could also account for the 2% difference.

andrewtoth force-pushed on Oct 24, 2024

DrahtBot commented at 7:25 pm on October 24, 2024: contributor

🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/32027275494

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

DrahtBot added the label CI failed on Oct 24, 2024

DrahtBot removed the label CI failed on Oct 25, 2024

andrewtoth force-pushed on Oct 26, 2024

DrahtBot added the label CI failed on Oct 26, 2024

DrahtBot commented at 6:52 pm on October 26, 2024: contributor

🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/32107893176

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

andrewtoth force-pushed on Oct 26, 2024

andrewtoth force-pushed on Oct 27, 2024

andrewtoth marked this as a draft on Oct 27, 2024

andrewtoth force-pushed on Oct 27, 2024

andrewtoth force-pushed on Nov 7, 2024

DrahtBot removed the label CI failed on Nov 7, 2024

andrewtoth force-pushed on Nov 7, 2024

andrewtoth marked this as ready for review on Nov 7, 2024

andrewtoth commented at 3:05 pm on November 7, 2024: contributor

@furszy I tried to switch to using a shared threadpool, but it is much slower that way. We need a way to have shared state between threads for this, instead of just scheduling tasks. I suppose the generic threadpool is great for scheduling independent tasks like indexing an individual block, but for quickly pulling outpoints off a shared vector it is not optimized well.

From #29386:

I just noticed the comment in the code:

For each thread a thread stack needs to be allocated. By default on Linux, threads take up 8MiB for the thread stack on a 64-bit system, and 4MiB in a 32-bit system.

Only 8MiB of Virtual Memory is allocated, which doesn’t really mean anything. Due to CoW mechanism, only the parts of stack that are being used will be allocated as Physical Memory which is the one that actually matters.

So, I don’t think it matters much to have an extra threadpool owned by the input fetcher.

I think this is ready for more review. I also added tests and a benchmark.

andrewtoth force-pushed on Nov 7, 2024

andrewtoth force-pushed on Nov 9, 2024

andrewtoth commented at 3:28 pm on November 13, 2024: contributor

For later blocks where cache misses are much more common, this change has an even bigger impact. This benchmark report shows a 40% speedup measuring from blocks 840k to 850k. Also, compare flamegraphs of master and this branch, where the latter has 15 worker threads fetching coins from disk. https://bitcoin-dev-tools.github.io/benchcoin/results/pr-19/11798124132/index.html

andrewtoth commented at 3:35 am on November 16, 2024: contributor

Even with just 2 worker threads, there is significant (~30%) speed improvement for syncing recent blocks. https://bitcoin-dev-tools.github.io/benchcoin/results/pr-19/11865650166/index.html

andrewtoth force-pushed on Nov 16, 2024

DrahtBot commented at 7:45 pm on November 16, 2024: contributor

🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/33086747731

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

DrahtBot added the label CI failed on Nov 16, 2024

andrewtoth force-pushed on Nov 16, 2024

DrahtBot removed the label CI failed on Nov 16, 2024

andrewtoth force-pushed on Nov 17, 2024

in src/test/fuzz/inputfetcher.cpp:36 in 2bd5f0f03b outdated

27+    CBlock block;
28+    Txid prevhash{Txid::FromUint256(ConsumeUInt256(fuzzed_data_provider))};
29+
30+    const auto txs{fuzzed_data_provider.ConsumeIntegralInRange<uint32_t>(1,
31+        std::numeric_limits<uint32_t>::max())};
32+    for (uint32_t i{0}; i < txs; ++i) {

dergoegge commented at 2:16 pm on November 18, 2024:

This will create very long running inputs (e.g. txs = std::numeric_limits<uint32_t>::max()).

0    LIMITED_WHILE(fuzzed_data_provider.ConsumeBool(), N) {

or

0    LIMITED_WHILE(fuzzed_data_provider.remaining_bytes(), N) {

andrewtoth commented at 5:04 pm on November 18, 2024:

Thanks, done!

in src/test/fuzz/inputfetcher.cpp:41 in 2bd5f0f03b outdated

31+        std::numeric_limits<uint32_t>::max())};
32+    for (uint32_t i{0}; i < txs; ++i) {
33+        CMutableTransaction tx;
34+
35+        const auto inputs{fuzzed_data_provider.ConsumeIntegral<uint32_t>()};
36+        for (uint32_t j{0}; j < inputs; ++j) {

dergoegge commented at 2:17 pm on November 18, 2024:

Same as above, this will create long running inputs and maybe even run out of memory?

andrewtoth force-pushed on Nov 18, 2024

DrahtBot commented at 3:34 pm on November 18, 2024: contributor

🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/33143571653

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

DrahtBot added the label CI failed on Nov 18, 2024

andrewtoth force-pushed on Nov 18, 2024

DrahtBot removed the label CI failed on Nov 18, 2024

andrewtoth force-pushed on Nov 19, 2024

in src/coins.h:420 in 3a4af55071 outdated

415@@ -416,13 +416,14 @@ class CCoinsViewCache : public CCoinsViewBacked
416     void AddCoin(const COutPoint& outpoint, Coin&& coin, bool possible_overwrite);
417 
418     /**
419-     * Emplace a coin into cacheCoins without performing any checks, marking
420-     * the emplaced coin as dirty.
421+     * Emplace a coin into cacheCoins without performing any checks, optionally
422+     * marking the emplaced coin as dirty.

sedited commented at 2:38 pm on November 19, 2024:

Should this rather say “optionally marking the emplaced coin as not dirty”, since the default is always dirty?

andrewtoth commented at 6:53 pm on November 20, 2024:

I’m not sure that’s the best though, since we do not mark a coin as not dirty. That is the default state.

What about “marking the coin as dirty unless set_dirty is set to false”?

sedited commented at 7:15 pm on November 20, 2024:

That sounds good to me :+1:

andrewtoth commented at 9:56 pm on November 20, 2024:

Done.

andrewtoth force-pushed on Nov 19, 2024

andrewtoth force-pushed on Nov 20, 2024

DrahtBot added the label CI failed on Nov 20, 2024

DrahtBot commented at 6:46 pm on November 20, 2024: contributor

🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/33279820062

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

andrewtoth force-pushed on Nov 20, 2024

DrahtBot removed the label CI failed on Nov 20, 2024

andrewtoth force-pushed on Nov 21, 2024

DrahtBot commented at 5:12 pm on November 21, 2024: contributor

🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/33335042693

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

DrahtBot added the label CI failed on Nov 21, 2024

andrewtoth force-pushed on Nov 21, 2024

DrahtBot removed the label CI failed on Nov 21, 2024

andrewtoth force-pushed on Nov 22, 2024

andrewtoth force-pushed on Nov 24, 2024

in src/test/inputfetcher_tests.cpp:55 in 3c201bcffc outdated

50+    const auto cores{GetNumCores()};
51+    const auto num_txs{m_rng.randrange(cores * 10)};
52+    const auto block{CreateBlock(num_txs)};
53+    const auto batch_size{m_rng.randrange<int32_t>(block.vtx.size() * 2)};
54+    const auto worker_threads{m_rng.randrange(cores * 2)};
55+    InputFetcher fetcher{batch_size, worker_threads};

ismaelsadeeq commented at 8:31 pm on November 26, 2024:

In 3c201bcffc1d7e382e8afa9a88750a4c261c1cf8 “tests: add inputfetcher tests” You can set this up in InputFetcherTest so that you don’t have to repeat it in the rest of the tests.

andrewtoth commented at 0:58 am on December 4, 2024:

Done.

in src/bench/inputfetcher.cpp:15 in 2349ac7d60 outdated

10+#include <primitives/block.h>
11+#include <serialize.h>
12+#include <streams.h>
13+#include <util/time.h>
14+
15+static constexpr auto QUEUE_BATCH_SIZE{128};

ismaelsadeeq commented at 8:33 pm on November 26, 2024:

In 2349ac7d6071746a80223358bce0d5e556b277d7 “bench: add inputfetcher bench” How did you select this batch size?

andrewtoth commented at 0:58 am on December 4, 2024:

This is the hardcoded batch size used in CheckQueue. Not sure why that was selected, but I deferred to previous choices.

l0rinc commented at 3:45 pm on September 29, 2025:

I would prefer retesting those assumptions (I don’t even think we need a batch here)

andrewtoth commented at 7:06 pm on October 3, 2025:

Removed the batch size 🎉

in src/bench/inputfetcher.cpp:18 in 2349ac7d60 outdated

14+
15+static constexpr auto QUEUE_BATCH_SIZE{128};
16+static constexpr auto DELAY{2ms};
17+
18+//! Simulates a DB by adding a delay when calling GetCoin
19+class DelayedCoinsView : public CCoinsView

ismaelsadeeq commented at 8:47 pm on November 26, 2024:

In 2349ac7d6071746a80223358bce0d5e556b277d7 “bench: add inputfetcher bench” nit: will be nice if we have block413567 input’s data that we can read so that we dont have to simulate this.

andrewtoth commented at 0:59 am on December 4, 2024:

We’re reading the previous outpoints of that block’s inputs, which are in many other previous blocks. So, not sure this is feasible.

ismaelsadeeq commented at 8:55 pm on November 26, 2024: member

Concept ACK This is nice. Although I have not yet benchmarked this branch, I also like @furszy’s idea of having a general-purpose thread pool.

I just have one test improvement comment, question and a nit after first pass of the PR

DrahtBot added the label Needs rebase on Dec 4, 2024

andrewtoth force-pushed on Dec 4, 2024

andrewtoth commented at 1:02 am on December 4, 2024: contributor

Rebased. Since #30039 reading inputs is much faster, so the effect of this is somewhat less significant (17% -> 10%). It’s still a significant speedup though so still worth it. Especially for worst case where the cache is completely empty, like on startup or right after it gets flushed due to size.

It is also refactored significantly. The main thread now writes everything before notifying threads, and then joins in working. This lets us do significantly less work in the critical section and parallelize more checks.

andrewtoth renamed this:
~~validation: fetch block inputs on parallel threads ~17% faster IBD~~
validation: fetch block inputs on parallel threads 10% faster IBD
on Dec 4, 2024

andrewtoth force-pushed on Dec 4, 2024

DrahtBot commented at 1:29 am on December 4, 2024: contributor

🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/33884531020

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

DrahtBot added the label CI failed on Dec 4, 2024

andrewtoth force-pushed on Dec 4, 2024

DrahtBot removed the label Needs rebase on Dec 4, 2024

DrahtBot removed the label CI failed on Dec 4, 2024

DrahtBot added the label Needs rebase on Dec 5, 2024

andrewtoth force-pushed on Dec 5, 2024

DrahtBot removed the label Needs rebase on Dec 5, 2024

in src/validation.cpp:3176 in b2da764446 outdated

3194@@ -3195,6 +3195,8 @@ bool Chainstate::ConnectTip(BlockValidationState& state, CBlockIndex* pindexNew,
3195     LogDebug(BCLog::BENCH, "  - Load block from disk: %.2fms\n",
3196              Ticks<MillisecondsDouble>(time_2 - time_1));
3197     {
3198+        m_chainman.GetInputFetcher().FetchInputs(CoinsTip(), CoinsDB(), blockConnecting);

l0rinc commented at 9:26 am on May 11, 2025:

Can we let the objects do the job instead of querying their internals and doing it ourselves (“tell, don’t ask”):

0        m_chainman.FetchInputs(CoinsTip(), CoinsDB(), blockConnecting);

andrewtoth commented at 2:20 pm on October 2, 2025:

I tried to mimic the script validation like GetCheckQueue. But, I guess this is different enough. Will change next time I push.

andrewtoth commented at 0:00 am on October 4, 2025:

Done.

DrahtBot added the label CI failed on Jun 2, 2025

maflcko commented at 7:02 am on June 10, 2025: member

Looks like the CI started failing, due to too many threads being launched in the functional tests with that parallelism? As the threads may open files, this could be hitting the max open files limit? Or maybe it is a different limit hit?

HowHsu commented at 4:42 pm on June 26, 2025: none

Hi folks, this looks great, since if all the prevout coins of all transactions of a block are loaded in advance, then the optimization in #32791 makes sense.

sedited commented at 9:09 am on September 17, 2025: contributor

What’s the status here?

andrewtoth force-pushed on Sep 17, 2025

andrewtoth commented at 8:38 pm on September 17, 2025: contributor

Looks like the CI started failing, due to too many threads being launched in the functional tests with that parallelism? As the threads may open files, this could be hitting the max open files limit? Or maybe it is a different limit hit?

Thanks, I added -par=1 to all nodes spawned in features_proxy.py in 6980852416040bdddf111df3cea3ec50639f010a. That test spawns lots of nodes and block validation is not relevant to it.

What’s the status here?

Rebased to fix silent conflicts and added the fix for features_proxy.py.

andrewtoth commented at 10:57 pm on September 21, 2025: contributor

I benchmarked the latest branch with default dbcache up to 912683. Results are a speedup of 14% - 5:07 vs 5:49.

Command	Mean [s]	Min [s]	Max [s]	Relative
`echo 688c03597afb0b76077f1ffc4608eef19481056e && /usr/bin/time ./build/bin/bitcoind -printtoconsole=0 -connect=192.168.2.171 -stopatheight=912683`	18430.672 ± 19.856	18416.631	18444.712	1.00
`echo 1444ed855f438f1270104fca259ce61b99ed5cdb && /usr/bin/time ./build/bin/bitcoind -printtoconsole=0 -connect=192.168.2.171 -stopatheight=912683`	20937.219 ± 62.635	20892.929	20981.508	1.14 ± 0.00

andrewtoth renamed this:
~~validation: fetch block inputs on parallel threads 10% faster IBD~~
validation: fetch block inputs on parallel threads >10% faster IBD
on Sep 21, 2025

maflcko removed the label CI failed on Sep 22, 2025

andrewtoth commented at 5:18 pm on September 23, 2025: contributor

Did the same benchmark with 5000 dbcache and there is a 6% speedup :rocket: - 4:27 vs 4:44. Even with far fewer cache misses this change is still a benefit, and will continue to improve block connection speed as the blockchain and utxo set get bigger.

Command	Mean [s]	Min [s]	Max [s]	Relative
`echo 688c03597afb0b76077f1ffc4608eef19481056e && /usr/bin/time ./build/bin/bitcoind -printtoconsole=0 -connect=192.168.2.171 -stopatheight=912683 -dbcache=5000`	16021.047 ± 5.892	16016.881	16025.213	1.00
`echo 1444ed855f438f1270104fca259ce61b99ed5cdb && /usr/bin/time ./build/bin/bitcoind -printtoconsole=0 -connect=192.168.2.171 -stopatheight=912683 -dbcache=5000`	17057.947 ± 42.032	17028.226	17087.668	1.06 ± 0.00

in test/functional/feature_proxy.py:141 in 688c03597a outdated

135@@ -136,6 +136,9 @@ def setup_nodes(self):
136         if self.have_unix_sockets:
137             args[5] = ['-listen', f'-proxy=unix:{socket_path}']
138             args[6] = ['-listen', f'-onion=unix:{socket_path}']
139+        # Keep validation threads low to avoid CI thread/pid limits.
140+        # Ensure even empty arg lists get '-par=1'.
141+        args = [a + ['-par=1'] if a else ['-par=1'] for a in args]

maflcko commented at 8:16 pm on September 24, 2025:

seems a bit odd to have the number of nodes in a test influence whether or not the test has to be edited to remove or add -par=1 everywhere. Would it not be easier to just globally set -par=2 for all funtional tests?

 0diff --git a/test/functional/test_framework/util.py b/test/functional/test_framework/util.py
 1index e5a5938f07..42bb213dd3 100644
 2--- a/test/functional/test_framework/util.py
 3+++ b/test/functional/test_framework/util.py
 4@@ -459,6 +459,7 @@ def write_config(config_path, *, n, chain, extra_config="", disable_autoconnect=
 5         f.write("printtoconsole=0\n")
 6         f.write("natpmp=0\n")
 7         f.write("shrinkdebugfile=0\n")
 8+        f.write("par=2\n")
 9         # To improve SQLite wallet performance so that the tests don't timeout, use -unsafesqlitesync
10         f.write("unsafesqlitesync=1\n")
11         if disable_autoconnect:

andrewtoth commented at 9:01 pm on September 24, 2025:

Yes, I wondered if that would be more invasive to other tests though.

maflcko commented at 7:42 am on September 25, 2025:

It disables the auto-detection for all functional tests by default, which I can’t really find a downside to. Also, it removes idle “spam” threads while debugging (gdb and other tools will display less script check threads), which also seems beneficial to have?

andrewtoth commented at 4:09 pm on September 27, 2025:

Makes sense. Done here #33485.

in src/inputfetcher.h:134 in 688c03597a outdated

140+                        m_in_flight_outpoints_count -= m_last_outpoint_index;
141+                        m_last_outpoint_index = 0;
142+                        break;
143+                    }
144+                }
145+            } catch (const std::runtime_error&) {

l0rinc commented at 2:31 pm on September 29, 2025:

nit: is there anything in the error that we may want to log?

andrewtoth commented at 6:04 pm on October 11, 2025:

Added a log.

in src/inputfetcher.h:107 in 688c03597a

102+                while (m_last_outpoint_index == 0) {
103+                    if ((is_main_thread && m_in_flight_outpoints_count == 0) || m_request_stop) {
104+                        return;
105+                    }
106+                    ++m_idle_worker_count;
107+                    cond.wait(lock);

l0rinc commented at 2:33 pm on September 29, 2025:

I haven’t reviewed it in detail but was wondering why we need locking here, it should be possible to do most of this lock free (especially if we sort the keys first so that threads are more likely to access different regions). I have started reviewing and testing it in detail, but to have some progress I’m sharing my observations as I go along

andrewtoth commented at 2:21 pm on October 2, 2025:

I’ve updated to use semaphores instead of mutex. That should be more efficient.

especially if we sort the keys first so that threads are more likely to access different regions

I don’t understand what this has to do with being lock free.

l0rinc commented at 2:40 pm on October 2, 2025:

I don’t understand what this has to do with being lock free.

We may have fewer file system locks if the threads are accessing different regions

I’ve updated to use semaphores instead of mutex

I will review that in more detail soon, probably next week.

andrewtoth commented at 2:47 pm on October 2, 2025:

We may have fewer file system locks if the threads are accessing different regions

Ok, but that is not the same as this InputFetcher construction being lock free.

l0rinc commented at 2:54 pm on October 2, 2025:

No, that’s orthogonal, it’s another area where we could possibly reduce contention.

in src/inputfetcher.h:82 in 688c03597a

77+    const CCoinsViewCache* m_cache{nullptr};
78+
79+    std::vector<std::thread> m_worker_threads;
80+    bool m_request_stop GUARDED_BY(m_mutex){false};
81+
82+    //! Internal function that does the fetching from disk.

l0rinc commented at 2:35 pm on September 29, 2025:

instead of the comment, can we express this in the method name?

andrewtoth commented at 2:22 pm on October 2, 2025:

I updated the thread name to ThreadLoop, which just does the loop. There is another function now, FetchInputsOnThread, that fetches for each block until finished.

in src/inputfetcher.h:84 in 688c03597a outdated

74+    //! DB coins view to fetch from.
75+    const CCoinsView* m_db{nullptr};
76+    //! The cache to check if we already have this input.
77+    const CCoinsViewCache* m_cache{nullptr};
78+
79+    std::vector<std::thread> m_worker_threads;

l0rinc commented at 2:38 pm on September 29, 2025:

I have tried std::jthread in l0rinc/bitcoin@6afe2e8 (#40) but it seems the CI’s libc++ doesn’t provide it

Q: it’s just the second commit and we’re already doing the fetching on multiple threads. Can we add a single-threaded input fetcher first and add multithreading only as the very last step?

andrewtoth commented at 3:55 pm on October 4, 2025:

Can we add a single-threaded input fetcher first and add multithreading only as the very last step?

Done.

andrewtoth commented at 2:09 pm on October 14, 2025:

I have tried std::jthread

I looked at this, but it doesn’t really add anything to this implementation. We could have a std::stop_token for each thread, but we would have to request_stop() each jthread before releasing the semaphore in the destructor anyway. So it doesn’t let us remove the destructor, and saves a line for not having to declare m_request_stop. I don’t think it’s worth it to use jthreads here.

l0rinc commented at 2:49 pm on October 14, 2025:

I couldn’t get it to work on CI anyway

in src/inputfetcher.h:73 in 688c03597a

68+     */
69+    int32_t m_in_flight_outpoints_count GUARDED_BY(m_mutex){0};
70+    //! The number of worker threads that are waiting on m_worker_cv
71+    int32_t m_idle_worker_count GUARDED_BY(m_mutex){0};
72+    //! The maximum number of outpoints to be assigned in one batch
73+    const int32_t m_batch_size;

l0rinc commented at 2:42 pm on September 29, 2025:

what if instead of locking we just iterate every nth element (where n is the thread index), implicitly dividing the input into n buckets without locking. Each thread would work on a distinct set of values - we can pre-filter for existing values on a single thread before forking off. This won’t have work stealing, but we can likely assume uniform distribution and the solution would be trivial and lock free.

andrewtoth commented at 2:24 pm on October 2, 2025:

Prefiltering on the main thread is too slow, it’s faster if we do the filtering in parallel. So, we still need to have a smaller batch size because then work will not be divided evenly. One thread could get all cache misses while the others all have cached inputs.

l0rinc commented at 2:40 pm on October 2, 2025:

Not sure why that’s problematic, we don’t have to have perfect parallelism, it seems to me we can assume uniform distribution - it’s fine if there are outliers if that makes the code simpler (which I think it should, it could even eliminate most locks, since the jobs are basically completely independent)

in src/inputfetcher.h:75 in 688c03597a outdated

72+    //! The maximum number of outpoints to be assigned in one batch
73+    const int32_t m_batch_size;
74+    //! DB coins view to fetch from.
75+    const CCoinsView* m_db{nullptr};
76+    //! The cache to check if we already have this input.
77+    const CCoinsViewCache* m_cache{nullptr};

l0rinc commented at 2:44 pm on September 29, 2025:

could we pre-filter on a single tread and send the results to the fetcher instead? That way we can also decide not to do multi-threaded access for small sets (we can experiment with the values, but we can probably start with set size < nproc should be done on a single thread).

andrewtoth commented at 2:25 pm on October 2, 2025:

Prefiltering on the main thread is too slow. It is several milliseconds to check every input in large blocks whether they exist in the cache.

in src/inputfetcher.h:1 in 688c03597a outdated

0@@ -0,0 +1,246 @@
1+// Copyright (c) 2024-present The Bitcoin Core developers

l0rinc commented at 2:45 pm on September 29, 2025:

nit: the curse of long review queues

0// Copyright (c) 2025-present The Bitcoin Core developers

andrewtoth commented at 2:25 pm on October 2, 2025:

Done.

in src/coins.h:432 in 688c03597a outdated

430+     * NOT FOR GENERAL USE. Used when loading coins from a UTXO snapshot, and
431+     * in the InputFetcher.
432      * @sa ChainstateManager::PopulateAndValidateSnapshot()
433      */
434-    void EmplaceCoinInternalDANGER(COutPoint&& outpoint, Coin&& coin);
435+    void EmplaceCoinInternalDANGER(COutPoint&& outpoint, Coin&& coin, bool set_dirty = true);

l0rinc commented at 2:48 pm on September 29, 2025:

to peel away the preparatory commits, it would simplify review to extract these into tiny, focused PRs - to have some progress, since this PR is in review for some time, but it’s a very good change that I’d like to have some progress on.

nit: I understand the default param is meant to make the diff smaller, but it doesn’t help with understanding the effect of the change, to see where this is used and what we’re changing

andrewtoth commented at 4:09 pm on October 15, 2025:

I think I can just drop this first commit entirely. We don’t actually care to not set the coins we fetch as dirty. In the happy path, all these coins will be spent immediately after ConnectBlock, so they will be set to dirty anyways. In the unhappy path where the valid proof-of-work block is found to be invalid, the dirty coins we added will just cause the coins to be overwritten by the same data in the db at the next flush.

in src/coins.cpp:114 in 688c03597a outdated

109@@ -110,10 +110,15 @@ void CCoinsViewCache::AddCoin(const COutPoint &outpoint, Coin&& coin, bool possi
110            (bool)it->second.coin.IsCoinBase());
111 }
112 
113-void CCoinsViewCache::EmplaceCoinInternalDANGER(COutPoint&& outpoint, Coin&& coin) {
114-    cachedCoinsUsage += coin.DynamicMemoryUsage();
115+void CCoinsViewCache::EmplaceCoinInternalDANGER(COutPoint&& outpoint, Coin&& coin, bool set_dirty) {
116+    const auto mem_usage{coin.DynamicMemoryUsage()};

l0rinc commented at 2:49 pm on September 29, 2025:

af8a366bd6a08d9362e69a89b0b89b5c94eb63ca I had something similar in https://github.com/bitcoin/bitcoin/pull/32313/files#diff-f0ed73d62dae6ca28ebd3045e5fc0d5d02eaaacadb4c2a292985a3fbd7e1c77cR254

Can you please explain in the commit message why this change is necessary?

andrewtoth commented at 3:59 pm on October 3, 2025:

Added some explanation in the commit message. Please let me know if it makes it more clear.

in src/validation.cpp:6271 in 688c03597a outdated

6267@@ -6266,6 +6268,7 @@ static ChainstateManager::Options&& Flatten(ChainstateManager::Options&& opts)
6268 
6269 ChainstateManager::ChainstateManager(const util::SignalInterrupt& interrupt, Options options, node::BlockManager::Options blockman_options)
6270     : m_script_check_queue{/*batch_size=*/128, std::clamp(options.worker_threads_num, 0, MAX_SCRIPTCHECK_THREADS)},
6271+      m_input_fetcher{/*batch_size=*/128, std::clamp(options.worker_threads_num, 0, MAX_SCRIPTCHECK_THREADS)},

l0rinc commented at 2:50 pm on September 29, 2025:

I have tested this with different par values and surprisingly it barely had any effect. Is it because of the locking?

andrewtoth commented at 2:27 pm on October 2, 2025:

I believe this is resolved.

in src/coins.cpp:118 in af8a366bd6 outdated

115+void CCoinsViewCache::EmplaceCoinInternalDANGER(COutPoint&& outpoint, Coin&& coin, bool set_dirty) {
116+    const auto mem_usage{coin.DynamicMemoryUsage()};
117     auto [it, inserted] = cacheCoins.try_emplace(std::move(outpoint), std::move(coin));
118-    if (inserted) CCoinsCacheEntry::SetDirty(*it, m_sentinel);
119+    if (inserted) {
120+        cachedCoinsUsage += mem_usage;

l0rinc commented at 2:57 pm on September 29, 2025:

this seems like a change in behavior, but it assumed that the coin was never in the cache - though if (inserted) didn’t help with understanding this, so it likely isn’t.

Is that still something that we can assume - hence the “danger”, right? And if it’s always inserted, does the insertion guard still make sense? It’s a bit even more confusing now :/

andrewtoth commented at 2:33 pm on October 2, 2025:

This is dangerous because it doesn’t check for freshness or if already inserted. It is meant to bulk load new utxos from the assume utxo set. Since assume utxo assumes the utxo set is currently empty, the coins would always be inserted. This is repurposed here to bulk load utxos from the db directly into the cache. However, an invalid block could be mined which spends an already spent utxo that is in the cache but has not been synced to the db yet. In that case, the insertion will fail here. There is a unit test specifically for this scenario.

in src/inputfetcher.h:25 in 912f26b81e outdated

19+#include <unordered_set>
20+#include <vector>
21+
22+/**
23+ * Input fetcher for fetching inputs from the CoinsDB and inserting
24+ * into the CoinsTip.

l0rinc commented at 2:59 pm on September 29, 2025:

0 * Helper for fetching inputs from the CoinsDB and inserting into the CoinsTip.

andrewtoth commented at 3:59 pm on October 3, 2025:

Done.

in src/inputfetcher.h:27 in 912f26b81e outdated

22+/**
23+ * Input fetcher for fetching inputs from the CoinsDB and inserting
24+ * into the CoinsTip.
25+ *
26+ * The main thread loops through the block and writes all input prevouts to a
27+ * global vector. It then wakes all workers and starts working as well. Each

l0rinc commented at 3:00 pm on September 29, 2025:

do we need to write to a global vector or can we safely iterate the prevouts directly from each thread?

andrewtoth commented at 2:36 pm on October 2, 2025:

We iterate the prevouts directly from the block now. However, we store the tx index and vin index in a global vector now. This way we can flatten the inputs instead of having to scan the txs to see how many inputs they have.

in src/inputfetcher.h:83 in 912f26b81e outdated

78+
79+    std::vector<std::thread> m_worker_threads;
80+    bool m_request_stop GUARDED_BY(m_mutex){false};
81+
82+    //! Internal function that does the fetching from disk.
83+    void Loop(int32_t index, bool is_main_thread = false) noexcept EXCLUSIVE_LOCKS_REQUIRED(!m_mutex)

l0rinc commented at 3:04 pm on September 29, 2025:

Q: do we really need the main thread to be part of this? I expect this to be disk bound and not CPU restricted, we should be able to go beyond nproc, so it should be safe to leave the main thread out of this as far as I can tell…

andrewtoth commented at 2:37 pm on October 2, 2025:

Not sure, would have to benchmark this. I have updated the functions though to make the main thread’s entrance clearer.

l0rinc commented at 2:43 pm on October 2, 2025:

My benchmarks so far indicate the opposite: after 3-4 threads there is no benefit to the parallelization (either on SSD or HDD). I will remeasure your new changes after you give me the 👍

in src/inputfetcher.h:121 in 912f26b81e outdated

125+                    // block, it won't be in the cache yet but it also won't be
126+                    // in the db either.
127+                    if (m_txids.contains(outpoint.hash)) {
128+                        continue;
129+                    }
130+                    if (m_cache->HaveCoinInCache(outpoint)) {

l0rinc commented at 3:28 pm on September 29, 2025:

as mentioned before I think it should be safe to pre-filter on a single thread instead

andrewtoth commented at 2:38 pm on October 2, 2025:

It is definitely safe to do, since all access would be on main thread. It is also safe to do from parallel threads if we don’t write until all threads are done reading, which is what this PR does. Prefiltering is slow though (several milliseconds) so is better to do in parallel.

l0rinc commented at 2:45 pm on October 2, 2025:

But prefiltering would allow sorting, which should untangle the threads. The threads will access the same files (which are more likely to be different from the files the other threads are requesting), so they may profit from cache locality if the OS supports it - that’s why I suggested giving it a try.

in src/inputfetcher.h:118 in 912f26b81e outdated

122+                for (auto i{end_index - local_batch_size}; i < end_index; ++i) {
123+                    const auto& outpoint{m_outpoints[i]};
124+                    // If an input spends an outpoint from earlier in the
125+                    // block, it won't be in the cache yet but it also won't be
126+                    // in the db either.
127+                    if (m_txids.contains(outpoint.hash)) {

l0rinc commented at 3:31 pm on September 29, 2025:

what if this ends up on different threads, i.e. a spend from an earlier outpoint processed on a different thread? Wouldn’t we take care of those automatically? We can likely skip all values that are not found, since we will revalidate everything after this cache warming call - we just have to document that it’s theoretically possible that some values won’t be in the cache after this call (though the internal spends should be added, just in a different thread, right?).

andrewtoth commented at 2:40 pm on October 2, 2025:

I’m not sure I understand this. The m_txids set is computed on the main thread, and is only read from multiple threads. If we didn’t do this we would try to fetch non-existent outputs from the db, which would be much slower.

in src/inputfetcher.h:127 in 912f26b81e outdated

131+                        continue;
132+                    }
133+                    if (auto coin{m_db->GetCoin(outpoint)}; coin) {
134+                        local_pairs.emplace_back(outpoint, std::move(*coin));
135+                    } else {
136+                        // Missing an input. This block will fail validation.

l0rinc commented at 3:32 pm on September 29, 2025:

do we really care about this, it’s not our job here to validate, just fetch whatever we can, the validation will happen after this pre-warming.

andrewtoth commented at 2:41 pm on October 2, 2025:

We don’t really care, but it would be good to not continue doing work here if we know it’s pointless. This just exits early. No validation is happening.

l0rinc commented at 2:50 pm on October 2, 2025:

I think I would prefer a less opinionated version, as long as it’s still correct. No need to optimize for the consensus failure speed in my opinion, I would prefer simpler code for a change as risky as this one.

in src/inputfetcher.h:151 in 912f26b81e outdated

157+
158+    //! Create a new input fetcher
159+    explicit InputFetcher(int32_t batch_size, int32_t worker_thread_count) noexcept
160+        : m_batch_size(batch_size)
161+    {
162+        if (worker_thread_count < 1) {

l0rinc commented at 3:34 pm on September 29, 2025:

what’s the reason for allowing negative worker_thread_count? In other cases I think it was used to signal how many CPUs to reserve, but that doesn’t seem to be the case here, and since we’re claming to min of 0, consider:

0        if (worker_thread_count == 0) {

andrewtoth commented at 3:58 pm on October 3, 2025:

Done.

in src/inputfetcher.h:172 in 912f26b81e outdated

187+    void FetchInputs(CCoinsViewCache& cache,
188+                     const CCoinsView& db,
189+                     const CBlock& block) noexcept
190+        EXCLUSIVE_LOCKS_REQUIRED(!m_mutex)
191+    {
192+        if (m_worker_threads.empty() || block.vtx.size() <= 1) {

l0rinc commented at 3:37 pm on September 29, 2025:

Can we maybe do something like this instead?

0        if (block.vtx.size() < m_worker_threads.size()) {

andrewtoth commented at 2:42 pm on October 2, 2025:

This is to not enter if there is only a coinbase tx, since it has no inputs to fetch. If there were 2 txs, and the second has 1000 inputs, we would still want to enter here.

in src/inputfetcher.h:135 in 912f26b81e outdated

193+            return;
194+        }
195+
196+        // Set the db and cache to use for this block.
197+        m_db = &db;
198+        m_cache = &cache;

l0rinc commented at 3:38 pm on September 29, 2025:

can we avoid mutating the state in a multithreaded class for safety? It’s easier to follow along knowing that the class is immutable and the state is passed along…

andrewtoth commented at 2:43 pm on October 2, 2025:

I don’t think we can do that. We need to set these here for other threads to read. These are only read from other threads, never written to. We also only read from other threads after the main thread has released the counting_semaphore, so we know the pointers are synced across the threads.

l0rinc commented at 2:48 pm on October 2, 2025:

I really dislike that, will try to come up with a lock-free version later (maybe next week)

in src/test/inputfetcher_tests.cpp:50 in c705c6f1f1 outdated

46+
47+        return block;
48+    }
49+
50+public:
51+    explicit InputFetcherTest(const ChainType chainType = ChainType::MAIN,

l0rinc commented at 3:40 pm on September 29, 2025:

can we add these tests before the multithreading change - having a single-threaded InputFetcher first, adding tests and benchmarks after and doing the actual multithreading as a very last step. That would construct the whole scenario in smaller steps, proving that every change is safe

andrewtoth commented at 3:57 pm on October 4, 2025:

Done.

in src/test/inputfetcher_tests.cpp:55 in c705c6f1f1 outdated

50+public:
51+    explicit InputFetcherTest(const ChainType chainType = ChainType::MAIN,
52+                             TestOpts opts = {})
53+        : BasicTestingSetup{chainType, opts}
54+    {
55+        SeedRandomForTest(SeedRand::ZEROS);

l0rinc commented at 3:41 pm on September 29, 2025:

I understand why benchmarks need predictability, but wouldn’t we want variance for tests?

andrewtoth commented at 6:04 pm on October 11, 2025:

Changed to FIXED_SEED.

in src/test/inputfetcher_tests.cpp:162 in c705c6f1f1 outdated

166+
167+class ThrowCoinsView : public CCoinsView
168+{
169+    std::optional<Coin> GetCoin(const COutPoint& outpoint) const override
170+    {
171+        throw std::runtime_error("database error");

l0rinc commented at 3:44 pm on September 29, 2025:

consider std::terminate

andrewtoth commented at 2:44 pm on October 2, 2025:

Err, we want to throw a runtime error here to test the try/catch in the inputfetcher.

in src/test/fuzz/inputfetcher.cpp:148 in 1faf0595a5 outdated

145+            // Check any newly added coins in the cache are the same as the db
146+            const auto& coin{cache.AccessCoin(outpoint)};
147+            assert(!coin.IsSpent());
148+            assert(coin.fCoinBase == (*maybe_coin).fCoinBase);
149+            assert(coin.nHeight == (*maybe_coin).nHeight);
150+            assert(coin.out == (*maybe_coin).out);

l0rinc commented at 3:47 pm on September 29, 2025:

0            assert(coin.fCoinBase == maybe_coin->fCoinBase);
1            assert(coin.nHeight == maybe_coin->nHeight);
2            assert(coin.out == maybe_coin->out);

andrewtoth commented at 6:04 pm on October 11, 2025:

Done!

l0rinc commented at 4:16 pm on September 29, 2025: contributor

I have re-reviewed the changes again lightly and did quite a few benchmarks on different platforms. There were a lot of surprises, see my measurements:

 0COMMITS="688c03597afb0b76077f1ffc4608eef19481056e af8a366bd6a08d9362e69a89b0b89b5c94eb63ca"; \
 1STOP=915961; DBCACHE=450; \
 2CC=gcc; CXX=g++; \
 3BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
 4(echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done; echo "") && \
 5hyperfine \
 6  --sort command \
 7  --runs 1 \
 8  --export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \
 9  --parameter-list COMMIT ${COMMITS// /,} \
10  --prepare "killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
11    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \
12    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20" \
13  --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
14  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0"
15
16688c03597a validation: fetch block inputs in parallel
17af8a366bd6 coins: allow emplacing non-dirty coins internally
18
19Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=915961 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 688c03597afb0b76077f1ffc4608eef19481056e)
20  Time (abs ≡):        29732.695 s               [User: 60441.083 s, System: 5856.247 s]
21
22Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=915961 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = af8a366bd6a08d9362e69a89b0b89b5c94eb63ca)
23  Time (abs ≡):        37896.082 s               [User: 60968.810 s, System: 7062.414 s]
24
25Relative speed comparison
26        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=915961 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 688c03597afb0b76077f1ffc4608eef19481056e)
27        1.27          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=915961 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = af8a366bd6a08d9362e69a89b0b89b5c94eb63ca)

Retested it separately with:

0# cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && time ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=916000 -connect=rpi5-16-3.local
1
2cat ../BitcoinData/debug.log | egrep 'height=0|height=916000'
32025-09-25T17:03:06Z UpdateTip: new best=000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f height=0 version=0x00000001 log2_work=32.000022 tx=1 date='2009-01-03T18:15:05Z' progress=0.000000 cache=0.3MiB(0txo)
42025-09-26T01:02:56Z UpdateTip: new best=000000000000000000003ca9748080f4c3d1230ba9fa4bed66be6ded05f9b6e6 height=916000 version=0x2000e000 log2_work=95.840381 tx=1246369867 date='2025-09-23T07:22:08Z' progress=0.998966 cache=367.7MiB(2821413txo)
57h 59m 50s

Doing the same on an Intel i9 with SSD shows similar results

 0COMMITS="688c03597afb0b76077f1ffc4608eef19481056e af8a366bd6a08d9362e69a89b0b89b5c94eb63ca"; \
 1STOP=915961; DBCACHE=450; \
 2CC=gcc; CXX=g++; \
 3BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
 4(echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done; echo "") && \
 5hyperfine \
 6  --sort command \
 7  --runs 1 \
 8  --export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \
 9  --parameter-list COMMIT ${COMMITS// /,} \
10  --prepare "killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
11    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \
12    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20" \
13  --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
14  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0"
15
16688c03597a validation: fetch block inputs in parallel
17af8a366bd6 coins: allow emplacing non-dirty coins internally
18
19Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=915961 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 688c03597afb0b76077f1ffc4608eef19481056e)
20  Time (abs ≡):        12698.166 s               [User: 33794.242 s, System: 3015.471 s]
21
22Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=915961 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = af8a366bd6a08d9362e69a89b0b89b5c94eb63ca)
23  Time (abs ≡):        15928.708 s               [User: 28382.232 s, System: 2308.299 s]
24
25Relative speed comparison
26        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=915961 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 688c03597afb0b76077f1ffc4608eef19481056e)
27        1.25          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=915961 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = af8a366bd6a08d9362e69a89b0b89b5c94eb63ca)

and

 0COMMITS="688c03597afb0b76077f1ffc4608eef19481056e af8a366bd6a08d9362e69a89b0b89b5c94eb63ca"; \
 1STOP=916000; DBCACHE=450; \
 2CC=gcc; CXX=g++; \
 3BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
 4(echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done; echo "") && \
 5hyperfine \
 6  --sort command \
 7  --runs 3 \
 8  --export-json "$BASE_DIR/ibd-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \
 9  --parameter-list COMMIT ${COMMITS// /,} \
10  --prepare "killall bitcoind 2>/dev/null; rm -rf $DATA_DIR/*; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
11    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \
12    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=1 -printtoconsole=0; sleep 20" \
13  --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log && \
14             grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log" \
15  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -blocksonly -printtoconsole=0"
16
17688c03597a validation: fetch block inputs in parallel
18af8a366bd6 coins: allow emplacing non-dirty coins internally
19
20Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=916000 -dbcache=450 -blocksonly -printtoconsole=0 (COMMIT = 688c03597afb0b76077f1ffc4608eef19481056e)
21  Time (mean ± σ):     21484.108 s ± 1187.956 s    [User: 42976.944 s, System: 4356.289 s]
22  Range (min … max):   20112.390 s … 22175.559 s    3 runs
23
24Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=916000 -dbcache=450 -blocksonly -printtoconsole=0 (COMMIT = af8a366bd6a08d9362e69a89b0b89b5c94eb63ca)
25  Time (mean ± σ):     26589.393 s ± 1171.370 s    [User: 36011.245 s, System: 3193.496 s]
26  Range (min … max):   25607.731 s … 27886.055 s    3 runs
27
28Relative speed comparison
29        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=916000 -dbcache=450 -blocksonly -printtoconsole=0 (COMMIT = 688c03597afb0b76077f1ffc4608eef19481056e)
30        1.24 ±  0.09  COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=916000 -dbcache=450 -blocksonly -printtoconsole=0 (COMMIT = af8a366bd6a08d9362e69a89b0b89b5c94eb63ca)

Increasing the memory decreases the difference:

COMMITS=“688c03597afb0b76077f1ffc4608eef19481056e af8a366bd6a08d9362e69a89b0b89b5c94eb63ca”; STOP=915961; DBCACHE=4500; CC=gcc; CXX=g++; BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; (echo “”; for c in $COMMITS; do git fetch -q origin $c && git log -1 –pretty=’%h %s’ $c || exit 1; done; echo “”) && hyperfine –sort command –runs 1 –export-json “$BASE_DIR/rdx-$(sed -E ’s/(\w{8})\w+ ?/\1-/g;s/-$//’«<"$COMMITS”)-$STOP-$DBCACHE-$CC.json" –parameter-list COMMIT ${COMMITS// /,} –prepare “killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset –hard &&
cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 &&
./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20” –cleanup “cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log” “COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0”

688c03597a validation: fetch block inputs in parallel af8a366bd6 coins: allow emplacing non-dirty coins internally

Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=915961 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 688c03597afb0b76077f1ffc4608eef19481056e) Time (abs ≡): 11801.704 s [User: 20216.598 s, System: 1181.879 s]

Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=915961 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = af8a366bd6a08d9362e69a89b0b89b5c94eb63ca) Time (abs ≡): 12916.432 s [User: 17150.579 s, System: 747.711 s]

Relative speed comparison 1.00 COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=915961 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 688c03597afb0b76077f1ffc4608eef19481056e) 1.09 COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=915961 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = af8a366bd6a08d9362e69a89b0b89b5c94eb63ca)

0</details>
1
2Note that the difference between small and big dbcache is also shrunk from 23% to 7.5%!
3
4Checked the same on an i7 with hdd, it seems the speedup is best on non-rotating disks, maybe we could consider reducing the parallelism in those cases:
5
6<details>
7<summary>i7 with HDD, IBD/reindex-chainstate seem is ~16% faster for default memory</summary>

COMMITS=“688c03597afb0b76077f1ffc4608eef19481056e af8a366bd6a08d9362e69a89b0b89b5c94eb63ca”;
STOP=916000; DBCACHE=450;
CC=gcc; CXX=g++;
BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs";
(echo “”; for c in $COMMITS; do git fetch -q origin $c && git log -1 –pretty=’%h %s’ $c || exit 1; done; echo “”) &&
hyperfine
–sort command
–runs 1
–export-json “$BASE_DIR/rdx-$(sed -E ’s/(\w{8})\w+ ?/\1-/g;s/-$//’«<"$COMMITS”)-$STOP-$DBCACHE-$CC.json"
–parameter-list COMMIT ${COMMITS// /,}
–prepare “killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset –hard &&
cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 &&
./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20”
–cleanup “cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log &&
grep -q ‘height=0’ $DATA_DIR/debug.log && grep -q ‘height=$STOP’ $DATA_DIR/debug.log”
“COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0”

688c03597a validation: fetch block inputs in parallel af8a366bd6 coins: allow emplacing non-dirty coins internally

Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=916000 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 688c03597afb0b76077f1ffc4608eef19481056e) Time (abs ≡): 35766.853 s [User: 39688.514 s, System: 2853.808 s]

Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=916000 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = af8a366bd6a08d9362e69a89b0b89b5c94eb63ca) Time (abs ≡): 41355.517 s [User: 35667.321 s, System: 2872.506 s]

Relative speed comparison 1.00 COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=916000 -dbcache=450 -reindex-chainstate -blockso$ly -connect=0 -printtoconsole=0 (COMMIT = 688c03597afb0b76077f1ffc4608eef19481056e) 1.16 COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=916000 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = af8a366bd6a08d9362e69a89b0b89b5c94eb63ca)

0and

COMMITS=“688c03597afb0b76077f1ffc4608eef19481056e af8a366bd6a08d9362e69a89b0b89b5c94eb63ca”;
STOP=916000; DBCACHE=450;
CC=gcc; CXX=g++;
BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs";
(echo “”; for c in $COMMITS; do git fetch -q origin $c && git log -1 –pretty=’%h %s’ $c || exit 1; done; echo “”) &&
hyperfine
–sort command
–runs 1
–export-json “$BASE_DIR/rdx-$(sed -E ’s/(\w{8})\w+ ?/\1-/g;s/-$//’«<"$COMMITS”)-$STOP-$DBCACHE-$CC.json"
–parameter-list COMMIT ${COMMITS// /,}
–prepare “killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset –hard &&
cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 &&
./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20”
–cleanup “cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log &&
grep -q ‘height=0’ $DATA_DIR/debug.log && grep -q ‘height=$STOP’ $DATA_DIR/debug.log”
“COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0”

688c03597a validation: fetch block inputs in parallel af8a366bd6 coins: allow emplacing non-dirty coins internally

Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=916000 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 688c03597afb0b76077f1ffc4608eef19481056e) Time (abs ≡): 35766.853 s [User: 39688.514 s, System: 2853.808 s]

Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=916000 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = af8a366bd6a08d9362e69a89b0b89b5c94eb63ca) Time (abs ≡): 41355.517 s [User: 35667.321 s, System: 2872.506 s]

Relative speed comparison 1.00 COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=916000 -dbcache=450 -reindex-chainstate -blockso$ly -connect=0 -printtoconsole=0 (COMMIT = 688c03597afb0b76077f1ffc4608eef19481056e) 1.16 COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=916000 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = af8a366bd6a08d9362e69a89b0b89b5c94eb63ca)

0
1</details>
2
3Checking the same on my M4 max laptop was the most surprising:
4<details>
5<summary>M4 max with SSD, IBD/reindex-chainstate seem is 290% faster for default memory</summary>

STOP=916000; DBCACHE=450;
DATA_DIR="/Users/lorinc/Library/Application\ Support/Bitcoin";
hyperfine
–sort command
–runs 1
–parameter-list COMMIT 688c03597afb0b76077f1ffc4608eef19481056e
–prepare “killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset –hard &&
cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 &&
./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20”
–cleanup “grep -q ‘height=0’ $DATA_DIR/debug.log && grep -q ‘height=$STOP’ $DATA_DIR/debug.log”
“./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0”

Benchmark 1: ./build/bin/bitcoind -datadir=/Users/lorinc/Library/Application\ Support/Bitcoin -stopatheight=916000 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = af8a366bd6a08d9362e69a89b0b89b5c94eb63ca) Time (abs ≡): 20658.825 s [User: 19679.918 s, System: 4763.490 s] Benchmark 1b: ./build/bin/bitcoind -datadir=/Users/lorinc/Library/Application\ Support/Bitcoin -stopatheight=916000 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = af8a366bd6a08d9362e69a89b0b89b5c94eb63ca) Time (abs ≡): 20186.312 s [User: 19481.126 s, System: 4716.728 s]

Benchmark 2: ./build/bin/bitcoind -datadir=/Users/lorinc/Library/Application\ Support/Bitcoin -stopatheight=916000 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 688c03597afb0b76077f1ffc4608eef19481056e) Time (abs ≡): 7131.178 s [User: 17244.133 s, System: 12850.427 s] Benchmark 2b: ./build/bin/bitcoind -datadir=/Users/lorinc/Library/Application\ Support/Bitcoin -stopatheight=916000 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 688c03597afb0b76077f1ffc4608eef19481056e) Time (abs ≡): 7180.762 s [User: 17430.360 s, System: 12949.731 s

Relative speed comparison 2.90 ./build/bin/bitcoind -datadir=/Users/lorinc/Library/Application\ Support/Bitcoin -stopatheight=916000 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = af8a366bd6a08d9362e69a89b0b89b5c94eb63ca) 1.00 ./build/bin/bitcoind -datadir=/Users/lorinc/Library/Application\ Support/Bitcoin -stopatheight=916000 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 688c03597afb0b76077f1ffc4608eef19481056e)

0
1It was hard to believe this was true, so I re-ran it a few times, and it was consistent.
2
3</details>
4
5I have tried -par=32 on my laptop as well - exactly the same speed:
6<details>
7<summary>-par=32</summary>

STOP=916000; DBCACHE=450;
DATA_DIR="/Users/lorinc/Library/Application\ Support/Bitcoin";
hyperfine
–sort command
–runs 1
–parameter-list COMMIT 688c03597afb0b76077f1ffc4608eef19481056e
–prepare “killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset –hard &&
cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 &&
./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20”
–cleanup “grep -q ‘height=0’ $DATA_DIR/debug.log && grep -q ‘height=$STOP’ $DATA_DIR/debug.log”
“./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 -par=32”

Benchmark 1: ./build/bin/bitcoind -datadir=/Users/lorinc/Library/Application\ Support/Bitcoin -stopatheight=916000 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 -par=32 (COMMIT = 688c03597afb0b76077f1ffc4608eef19481056e) Time (abs ≡): 7109.626 s [User: 17210.848 s, System: 12938.964 s]

0note, the commit had:

  m_input_fetcher{/*batch_size=*/128, std::clamp(options.worker_threads_num, 0, 10 * MAX_SCRIPTCHECK_THREADS)},

0
1</details>

l0rinc changes_requested

l0rinc commented at 4:17 pm on September 29, 2025: contributor

in src/inputfetcher.h:86 in 688c03597a

81+
82+    //! Internal function that does the fetching from disk.
83+    void Loop(int32_t index, bool is_main_thread = false) noexcept EXCLUSIVE_LOCKS_REQUIRED(!m_mutex)
84+    {
85+        auto local_batch_size{0};
86+        auto end_index{0};

l0rinc commented at 5:41 pm on September 29, 2025:

I think we should add exact types here to make sure calculations like end_index - local_batch_size can’t underflow

andrewtoth commented at 2:44 pm on October 2, 2025:

I rewrote this part, these are gone now.

in src/bench/inputfetcher.cpp:38 in 688c03597a outdated

42+    DelayedCoinsView db(DELAY);
43+    CCoinsViewCache cache(&db);
44+
45+    // The main thread should be counted to prevent thread oversubscription, and
46+    // to decrease the variance of benchmark results.
47+    const auto worker_threads_num{GetNumCores() - 1};

l0rinc commented at 5:44 pm on September 29, 2025:

I’m a bit conflicted here: this way we’re all measuring something slightly different - which is especially problematic since the work here isn’t even CPU bound. What if we did a min of npcu and 4?

Raimo33 commented at 2:23 pm on October 1, 2025: contributor

Concept ACK

andrewtoth force-pushed on Oct 2, 2025

andrewtoth commented at 3:47 pm on October 2, 2025: contributor

Updated the input fetcher significantly:

uses counting_semaphores to synchronize threads instead of mutex + condvar.
stores tx + vin indexes in global vector instead of copying the COutPoints. The COutPoints are read from a global CBlock pointer.
The fetch queue counter is an atomic int instead of a mutex guarded int.

andrewtoth force-pushed on Oct 3, 2025

andrewtoth commented at 5:33 pm on October 3, 2025: contributor

Removed m_batch_size. Each thread now increments the atomic counter by 1.

andrewtoth force-pushed on Oct 3, 2025

l0rinc commented at 0:53 am on October 4, 2025: contributor

The latest version seems very promising, I like that the algorithms is getting simpler. I noticed that for small dbcache it has a very noticeable effect, but for very high dbcache this seems to add an extra cost - since we already have everything in the cache, so it just does useless work. I wonder if we could enable this fetching only after the very first time we Flush and erase, since it cannot help in any way before that.

andrewtoth force-pushed on Oct 4, 2025

l0rinc commented at 6:52 pm on October 7, 2025: contributor

Compared it against master on a Raspberry Pi 5 synchronizing from real peers for realism, ran it twice for good measure until 917000 blocks with dbcache 450:

This isn’t the latest version of the PR, but should likely be representative anyway.

 0COMMITS="a8f9a806751b5755bdec5b096186f70c0bfddcfa f0dc19f16826f68ef482acfb7b24e8bb7168fc51"; \
 1STOP=917000; DBCACHE=450; \
 2CC=gcc; CXX=g++; \
 3BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
 4(echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done) && \
 5(echo "" && echo "IBD | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(df -T $BASE_DIR | awk 'NR==2{print $2}') | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 0 && echo SSD || echo HDD)"; echo "") &&\
 6hyperfine \
 7  --sort command \
 8  --runs 1 \
 9  --export-json "$BASE_DIR/ibd-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \
10  --parameter-list COMMIT ${COMMITS// /,} \
11  --prepare "killall bitcoind 2>/dev/null; rm -rf $DATA_DIR/*; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
12    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \
13    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=1 -printtoconsole=0; sleep 20" \
14  --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log && \
15             grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log" \
16  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -blocksonly -printtoconsole=0"
17
18a8f9a80675 validation: fetch block inputs in parallel
19f0dc19f168 coins: allow emplacing non-dirty coins internally
20
21IBD | 917000 blocks | dbcache 450 | rpi5-16-2 | aarch64 | Cortex-A76 | 4 cores | 15Gi RAM | ext4 | SSD
22
23Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=917000 -dbcache=450 -blocksonly -printtoconsole=0 (COMMIT = a8f9a806751b5755bdec5b096186f70c0bfddcfa)
24  Time (abs ≡):        47485.682 s               [User: 79615.847 s, System: 9374.261 s]
25 
26Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=917000 -dbcache=450 -blocksonly -printtoconsole=0 (COMMIT = f0dc19f16826f68ef482acfb7b24e8bb7168fc51)
27  Time (abs ≡):        56374.354 s               [User: 78807.079 s, System: 10196.290 s]
28 
29Relative speed comparison
30        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=917000 -dbcache=450 -blocksonly -printtoconsole=0 (COMMIT = a8f9a806751b5755bdec5b096186f70c0bfddcfa)
31        1.19          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=917000 -dbcache=450 -blocksonly -printtoconsole=0 (COMMIT = f0dc19f16826f68ef482acfb7b24e8bb7168fc51)

 0COMMITS="a8f9a806751b5755bdec5b096186f70c0bfddcfa f0dc19f16826f68ef482acfb7b24e8bb7168fc51"; \
 1STOP=917000; DBCACHE=450; \
 2CC=gcc; CXX=g++; \
 3BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
 4(echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done) && \
 5(echo "" && echo "IBD | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(df -T $BASE_DIR | awk 'NR==2{print $2}') | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 0 && echo SSD || echo HDD)"; echo "") &&\
 6hyperfine \
 7  --sort command \
 8  --runs 1 \
 9  --export-json "$BASE_DIR/ibd-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \
10  --parameter-list COMMIT ${COMMITS// /,} \
11  --prepare "killall bitcoind 2>/dev/null; rm -rf $DATA_DIR/*; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
12    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \
13    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=1 -printtoconsole=0; sleep 20" \
14  --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log && \
15             grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log" \
16  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -blocksonly -printtoconsole=0"
17
18a8f9a80675 validation: fetch block inputs in parallel
19f0dc19f168 coins: allow emplacing non-dirty coins internally
20
21IBD | 917000 blocks | dbcache 450 | rpi5-16-2 | aarch64 | Cortex-A76 | 4 cores | 15Gi RAM | ext4 | SSD
22
23Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=917000 -dbcache=450 -blocksonly -printtoconsole=0 (COMMIT = a8f9a806751b5755bdec5b096186f70c0bfddcfa)
24  Time (abs ≡):        45907.874 s               [User: 81006.258 s, System: 10039.919 s]
25 
26Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=917000 -dbcache=450 -blocksonly -printtoconsole=0 (COMMIT = f0dc19f16826f68ef482acfb7b24e8bb7168fc51)
27  Time (abs ≡):        55612.464 s               [User: 81830.349 s, System: 11913.754 s]
28 
29Relative speed comparison
30        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=917000 -dbcache=450 -blocksonly -printtoconsole=0 (COMMIT = a8f9a806751b5755bdec5b096186f70c0bfddcfa)
31        1.21          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=917000 -dbcache=450 -blocksonly -printtoconsole=0 (COMMIT = f0dc19f16826f68ef482acfb7b24e8bb7168fc51)

The variance between the runs shows 1% for master and 3% difference for the PR indicating that we’re likely nearing the network bandwidth limitations.

andrewtoth force-pushed on Oct 11, 2025

andrewtoth force-pushed on Oct 14, 2025

andrewtoth commented at 8:28 pm on October 14, 2025: contributor

I noticed that for small dbcache it has a very noticeable effect, but for very high dbcache this seems to add an extra cost - since we already have everything in the cache, so it just does useless work. I wonder if we could enable this fetching only after the very first time we Flush and erase, since it cannot help in any way before that. @l0rinc There is already quite a lot to review here, and your benchmarks (and mine) show very promising results. So, I would prefer to keep this idea as a follow-up. We can do isolated benchmarks with your suggested change afterwards and propose an improvement accordingly.

l0rinc commented at 9:04 pm on October 14, 2025: contributor

I’m fine with doing that in a follow-up if you think it’s too complicated (though it’s likely quite simple, we can just track the very first cache miss and always prefetch after that - that heuristic would even survive node restarts. Maybe we need to skip the Bip30 values though, but it’s just a heuristic anyway). But I don’t yet see this PR as close to being final yet - do you? I still want to review it thoroughly, I don’t think we should ossify yet :)

DrahtBot added the label Needs rebase on Oct 15, 2025

andrewtoth force-pushed on Oct 16, 2025

DrahtBot removed the label Needs rebase on Oct 16, 2025

andrewtoth commented at 2:07 am on October 16, 2025: contributor

Rebased due to conflicts. Removed the first commit. We don’t need to modify EmplaceCoinInternalDANGER, we can just insert the coins as dirty into the cache. They will be set dirty when they are spent anyways. If a block fails validation inside ConnectBlock, then the dirty coins will just rewrite the same value to the db on the next flush or sync.

in src/inputfetcher.h:35 in 063946d6bd outdated

30+private:
31+    /**
32+     * The flattened indexes to each input in the block. The first item in the
33+     * pair is the index of the tx, and the second is the index of the vin.
34+     */
35+    std::vector<std::pair<size_t, size_t>> m_inputs{};

l0rinc commented at 11:56 pm on October 16, 2025:

As far as I understood the tx index and the vin index cannot exceed the limits of a uint32_t, see https://github.com/bitcoin/bitcoin/blob/e744fd1249bf9577274614eaf3997bf4bbb612ff/src/primitives/transaction.h#L32

Please consider std::vector<std::pair<uint32_t, uint32_t>> instead, which would likely halve the memory footprint. Based on https://godbolt.org/z/Wb918bWaM it seems to me this layout allows modern compilers to coalesce the two member accesses into a single, optimal 64-bit load from memory.

Packing it into a single uint64_t seems to be the best, but that’s completely unreadable, so I’d go with the above std::pair<uint32_t, uint32_t>.

andrewtoth commented at 1:59 pm on October 28, 2025:

Done, using uint32_t in the Input struct now, not sure if it will have any effect in the struct though.

in src/validation.cpp:3142 in 64de911053 outdated

3136@@ -3137,6 +3137,8 @@ bool Chainstate::ConnectTip(
3137     LogDebug(BCLog::BENCH, "  - Load block from disk: %.2fms\n",
3138              Ticks<MillisecondsDouble>(time_2 - time_1));
3139     {
3140+        m_chainman.FetchInputs(CoinsTip(), CoinsDB(), *block_to_connect);
3141+
3142         CCoinsViewCache view(&CoinsTip());

l0rinc commented at 2:02 pm on October 18, 2025:

I have played with this to see if we can construct the new cache layer before the fetcher, so that it populates the new layer instead of reading & writing to the old one - seemed like a cleaner separation.

But unfortunately we would need access to both in-memory cache layers inside FetchInputs in that case - to read for presence from the old cache and to write the newly fetched values to the new one (which will be flushed together with the new outputs when existing this scope).

But given that we’re adding these missing entries only to spend them a moment later in the other cache layer (and to avoid having all of the missing ones require two-hop-lookups) we should still try reading from the stable cache and writing to the temporary one.

Collecting the missing inputs to a separate cache would also help with benchmarking and testing since the underlying cache would only be modified once block connection finishes - while the complete diff would be in the top cache layer. We also shouldn’t add the entries to the cache if the block fails validation.

andrewtoth commented at 1:30 pm on October 28, 2025:

Done.

in src/test/coinsviewcacheasync_tests.cpp:44 in 64de911053 outdated

36+
37+        Txid prevhash{Txid::FromUint256(uint256(1))};
38+
39+        for (auto i{1}; i < num_txs; ++i) {
40+            CMutableTransaction tx;
41+            const auto txid{m_rng.randbool() ? Txid::FromUint256(uint256(i)) : prevhash};

l0rinc commented at 4:14 pm on October 18, 2025:

Does this mean the spent tx is never processed on the same thread currently?

Maybe we can mix it up a bit by something like

0if (m_rng.randbool()) {
1    prevhash = tx.GetHash(); // TODO This can theoretically simulate double spends
2}

andrewtoth commented at 2:04 pm on October 28, 2025:

Does this mean the spent tx is never processed on the same thread currently?

I don’t think that’s what it means. The same thread could fetch two inputs in a row.

Maybe we can mix it up a bit by something like

So we can use the same prevhash in difference txs? What is the benefit of this?

in src/test/inputfetcher_tests.cpp:150 in 64de911053 outdated

148+}
149+
150+BOOST_FIXTURE_TEST_CASE(fetch_no_inputs, InputFetcherTest)
151+{
152+    const auto& block{getBlock()};
153+    for (auto i{0}; i < 3; ++i) {

l0rinc commented at 4:34 pm on October 18, 2025:

What’s the point of the loop in these, is there a state we’re changing in each iteration?

andrewtoth commented at 1:31 pm on October 28, 2025:

The InputFetcher is stateful, so this is making sure previous state does not leak into the next fetch phase.

in src/test/fuzz/inputfetcher.cpp:49 in 64de911053 outdated

42+{
43+public:
44+    std::optional<Coin> GetCoin(const COutPoint&) const override
45+    {
46+        abort();
47+    }

l0rinc commented at 4:52 pm on October 18, 2025:

nit:

0    std::optional<Coin> GetCoin(const COutPoint&) const override { std::abort(); }

andrewtoth commented at 2:00 pm on October 28, 2025:

Done.

in src/test/fuzz/inputfetcher.cpp:26 in 64de911053 outdated

21+class DbCoinsView : public CCoinsView
22+{
23+private:
24+    DbMap& m_map;
25+
26+public:

l0rinc commented at 4:53 pm on October 18, 2025:

nit: in test code I’d strive for simpler code instead of needlessly “safe”

0struct DbCoinsView : CCoinsView
1{
2    DbMap& m_map;

andrewtoth commented at 2:00 pm on October 28, 2025:

Done.

in src/bench/inputfetcher.cpp:29 in 64de911053 outdated

24+    DelayedCoinsView(std::chrono::milliseconds delay) : m_delay(delay) {}
25+
26+    std::optional<Coin> GetCoin(const COutPoint&) const override
27+    {
28+        UninterruptibleSleep(m_delay);
29+        return Coin{};

l0rinc commented at 4:54 pm on October 18, 2025:

GetCoin shouldn’t return spent entries

andrewtoth commented at 2:00 pm on October 28, 2025:

Done.

in src/bench/inputfetcher.cpp:24 in 64de911053 outdated

19+{
20+private:
21+    std::chrono::milliseconds m_delay;
22+
23+public:
24+    DelayedCoinsView(std::chrono::milliseconds delay) : m_delay(delay) {}

l0rinc commented at 4:54 pm on October 18, 2025:

this seems overly general to me, I think we can inline the delay for now

andrewtoth commented at 2:00 pm on October 28, 2025:

Done.

in src/inputfetcher.h:153 in 64de911053 outdated

148+            return;
149+        }
150+
151+        m_db = &db;
152+        m_cache = &cache;
153+        m_block = &block;

l0rinc commented at 4:55 pm on October 18, 2025:

As mentioned before, I really dislike these lines, a fetcher shouldn’t change the internal state (especially since they’re const). Since we need a continuously running threads, can we package these to avoid state mutations? This would likely be solved by sending these off to a ThreadPool instance.

andrewtoth commented at 1:36 pm on October 28, 2025:

Indeed, this would be gracefully solved with #33689. We could pass all state into the worker threads via lambda capture.

in src/test/fuzz/inputfetcher.cpp:120 in 64de911053 outdated

115+
116+            prevhash = tx.GetHash();
117+            block.vtx.push_back(MakeTransactionRef(tx));
118+        }
119+
120+        fetcher.FetchInputs(cache, db, block);

l0rinc commented at 5:00 pm on October 18, 2025:

We shouldn’t test this with invalid (empty) blocks:

0        if (block.vtx.empty()) continue;
1        fetcher.FetchInputs(cache, db, block);

andrewtoth commented at 1:37 pm on October 28, 2025:

Why not? The InputFetcher should not make assumptions about the structure of the block being passed in.

l0rinc commented at 4:30 pm on October 28, 2025:

Even consensus invalid ones? Or did I misunderstand the context here?

andrewtoth commented at 4:38 pm on October 28, 2025:

Yeah, InputFetcher doesn’t know if a block is consensus valid or not yet. It hasn’t passed ConnectBlock yet before entering here.

in src/bench/inputfetcher.cpp:39 in 64de911053 outdated

34+
35+static void InputFetcherBenchmark(benchmark::Bench& bench)
36+{
37+    DataStream stream{benchmark::data::block413567};
38+    CBlock block;
39+    stream >> TX_WITH_WITNESS(block);

l0rinc commented at 5:08 pm on October 18, 2025:

nit:

0    CBlock block;
1    DataStream{benchmark::data::block413567} >> TX_WITH_WITNESS(block);

andrewtoth commented at 2:00 pm on October 28, 2025:

Done.

in src/bench/inputfetcher.cpp:37 in 64de911053 outdated

40+
41+    DelayedCoinsView db(DELAY);
42+    CCoinsViewCache cache(&db);
43+
44+    // The main thread should be counted to prevent thread oversubscription, and
45+    // to decrease the variance of benchmark results.

l0rinc commented at 5:09 pm on October 18, 2025:

“should be counted” is the reason for the “- 1”?

in src/bench/inputfetcher.cpp:42 in 64de911053 outdated

42+    CCoinsViewCache cache(&db);
43+
44+    // The main thread should be counted to prevent thread oversubscription, and
45+    // to decrease the variance of benchmark results.
46+    const auto worker_threads_num{GetNumCores() - 1};
47+    InputFetcher fetcher{static_cast<size_t>(worker_threads_num)};

l0rinc commented at 5:12 pm on October 18, 2025:

nit: if we keep the processor count here (which kinda’ makes the benchmark measure different things on different platforms, but I don’t really have a better idea, unless we assume that even the simplest machines where this matters (e.g. rpi4) already have at least 4 threads - and since this isn’t even CPU bound, we shouldn’t pretend that it does):

0    const auto worker_threads_num{size_t(GetNumCores() - 1)};
1    const InputFetcher fetcher{worker_threads_num};

or

0    const InputFetcher fetcher{/*max_thread_count=*/4};

andrewtoth commented at 1:38 pm on October 28, 2025:

Why don’t we want different benchmarks on different machines? All benchmarks are subtly different depending on the host machine.

l0rinc commented at 4:32 pm on October 28, 2025:

Yeah, but we want to understand where the differences are coming from, otherwise we’d have the “faster-on-my-machine” syndrome. If you disagree, just resolve the issue.

in src/bench/inputfetcher.cpp:50 in 64de911053 outdated

45+    // to decrease the variance of benchmark results.
46+    const auto worker_threads_num{GetNumCores() - 1};
47+    InputFetcher fetcher{static_cast<size_t>(worker_threads_num)};
48+
49+    bench.run([&] {
50+        const auto ok{cache.Flush()};

l0rinc commented at 5:17 pm on October 18, 2025:

Doesn’t this change the behavior of the benchmark after the first iteration?

andrewtoth commented at 1:39 pm on October 28, 2025:

Fixed in latest iteration.

in src/validation.cpp:6302 in 64de911053 outdated

6266@@ -6265,6 +6267,7 @@ static ChainstateManager::Options&& Flatten(ChainstateManager::Options&& opts)
6267 
6268 ChainstateManager::ChainstateManager(const util::SignalInterrupt& interrupt, Options options, node::BlockManager::Options blockman_options)
6269     : m_script_check_queue{/*batch_size=*/128, std::clamp(options.worker_threads_num, 0, MAX_SCRIPTCHECK_THREADS)},
6270+      m_input_fetcher{std::clamp<size_t>(options.worker_threads_num, 0, MAX_SCRIPTCHECK_THREADS)},

l0rinc commented at 5:25 pm on October 18, 2025:

I don’t understand what 0 threads means. It likely means turn off prefetching.

I would consider it to be a lot more intuitive if this started with 1 (and not be bound by the number of unrelated MAX_SCRIPTCHECK_THREADS since it’s not script related and not even CPU bound).

Note: chainstatemanager_snapshot_init creates it with 0 workers by default, not sure it’s intended

andrewtoth commented at 1:41 pm on October 28, 2025:

0 threads means it is turned off, yes. If -par=1 is configured, we want to pass 0 to input fetcher to disable prefetching. Also if options.worker_threads_num was negative for some reason. Do you have a suggestion of how this can be made more clear?

in src/inputfetcher.h:119 in 64de911053 outdated

108+                }
109+                if (m_cache->HaveCoinInCache(outpoint)) {
110+                    continue;
111+                }
112+                if (auto coin{m_db->GetCoin(outpoint)}; coin) {
113+                    m_coins[thread_index].emplace_back(outpoint, std::move(*coin));

l0rinc commented at 5:38 pm on October 18, 2025:

It seems to me that since the original m_coins is never written by different threads, we shouldn’t have a false sharing problem here - right?

andrewtoth commented at 1:43 pm on October 28, 2025:

Possibly. This is no longer relevant in the current implementation.

in src/inputfetcher.h:84 in 64de911053 outdated

73+    const CBlock* m_block{nullptr};
74+
75+    std::vector<std::thread> m_worker_threads;
76+    std::counting_semaphore<> m_start_semaphore{0};
77+    std::counting_semaphore<> m_complete_semaphore{0};
78+    std::atomic<bool> m_request_stop{false};

l0rinc commented at 5:41 pm on October 18, 2025:

This style of dynamic work-stealing seems too complicated for such a uniform problem - static slicing would likely be a lot simpler and I would expect it to perform equally well.

I also realize that some of these are needed to avoid recreating the threads for every call. Currently the parallelism and the thread recreation are done in a single commit - could we first implement multithreading with thread recreation, and do the thread reuse as a separate concern (though a ThreadPool with dedicated test)?

andrewtoth commented at 2:05 pm on October 28, 2025:

It should be simpler now. I’m not sure if this comment is still valid though with the current approach. I agree if we already had a ThreadPool this would be much cleaner.

in src/test/inputfetcher_tests.cpp:131 in 64de911053 outdated

129+
130+        // Add all inputs as spent already in cache
131+        for (const auto& tx : block.vtx) {
132+            for (const auto& in : tx->vin) {
133+                auto outpoint{in.prevout};
134+                Coin coin{}; // Not setting nValue implies spent

l0rinc commented at 6:09 pm on October 18, 2025:

0                Coin coin{};
1                assert(coin.IsSpent());

andrewtoth commented at 2:01 pm on October 28, 2025:

Done.

andrewtoth force-pushed on Oct 19, 2025

andrewtoth commented at 3:26 pm on October 19, 2025: contributor

Updated to use std::barrier for the completion synchronization instead of acquiring a semaphore for each thread, as suggested by @l0rinc .

andrewtoth force-pushed on Oct 19, 2025

in src/inputfetcher.h:96 in a07936a62c

91+            m_start_semaphore.acquire();
92+            if (m_request_stop.load(std::memory_order_relaxed)) {
93+                return;
94+            }
95+            Work(thread_index);
96+            [[maybe_unused]] const auto arrival_token{m_complete_barrier.arrive()};

l0rinc commented at 1:23 am on October 21, 2025:

Would this suffice?

0            (void)m_complete_barrier.arrive();

in src/inputfetcher.h:190 in a07936a62c

185+        m_input_counter.store(0, std::memory_order_relaxed);
186+        m_start_semaphore.release(m_worker_threads.size());
187+
188+        // Have the main thread work too before we wait for other threads
189+        Work(m_worker_threads.size());
190+        m_complete_barrier.arrive_and_wait();

l0rinc commented at 1:29 am on October 21, 2025:

What’s the reason for not doing the completion here instead of the OnCompletion workaround?

0        m_complete_barrier.arrive_and_wait();
1        for (auto& coins : m_coins) {
2            for (auto& [outpoint, coin] : coins) {
3                m_cache->EmplaceCoinInternalDANGER(std::move(outpoint), std::move(coin));
4            }
5            coins.clear();
6        }

in src/inputfetcher.h:149 in a07936a62c

144+        }
145+    }
146+
147+public:
148+    explicit InputFetcher(size_t worker_thread_count) noexcept
149+        : m_complete_barrier{static_cast<int32_t>(worker_thread_count + 1), OnCompletionWrapper{this}}

l0rinc commented at 1:30 am on October 21, 2025:

std::ptrdiff_t seems more appropriate here: https://en.cppreference.com/w/cpp/thread/barrier/barrier.html

0    explicit InputFetcher(size_t worker_thread_count) noexcept : m_complete_barrier{std::ptrdiff_t(worker_thread_count + 1)}

in src/inputfetcher.h:110 in a07936a62c outdated

160+    }
161+
162+    //! Fetch all block inputs from db, and insert into cache.
163+    void FetchInputs(CCoinsViewCache& cache, const CCoinsView& db, const CBlock& block) noexcept
164+    {
165+        if (block.vtx.size() <= 1 || m_worker_threads.size() == 0) {

l0rinc commented at 2:21 pm on October 21, 2025:

wouldn’t m_worker_threads.size() == 0 be an error?

andrewtoth commented at 1:44 pm on October 28, 2025:

No, we can have no worker threads, for instance if started with -par=1. In that case just disable prefetching.

l0rinc commented at 5:58 pm on November 2, 2025:

Not a biggy, but this also seems uncovered by the unit tests.

in src/bench/inputfetcher.cpp:18 in a07936a62c outdated

13+#include <util/time.h>
14+
15+static constexpr auto DELAY{2ms};
16+
17+//! Simulates a DB by adding a delay when calling GetCoin
18+class DelayedCoinsView : public CCoinsView

l0rinc commented at 2:24 pm on October 21, 2025:

I personally would favor going for simpler code as opposed to going for theoretically better encapsulation, similarly to current inputfetcher_tests.cpp:

0struct DelayedCoinsView : CCoinsView

andrewtoth commented at 2:01 pm on October 28, 2025:

Done.

in src/bench/inputfetcher.cpp:28 in a07936a62c outdated

27+    {
28+        UninterruptibleSleep(m_delay);
29+        return Coin{};
30+    }
31+
32+    bool BatchWrite(CoinsViewCacheCursor&, const uint256&) override { return true; }

l0rinc commented at 2:25 pm on October 21, 2025:

We can add better assertions if we count the iterator size here:

0bool BatchWrite(CoinsViewCacheCursor& cursor, const uint256&) override
1{
2    for (auto it{cursor.Begin()}; it != cursor.End(); it = cursor.NextAndMaybeErase(*it)) {
3        m_write_count++;
4    }
5    return true;
6}

So the bench could be (without flush which makes the bench runs equivalent):

0bench.run([&] {
1    CCoinsViewCache block_cache{&cache};
2    fetcher.FetchInputs(cache, block_cache, db, block);
3    assert(db.m_write_count == 0 && cache.GetCacheSize() == 0 && block_cache.GetCacheSize() == 4599);
4});

andrewtoth commented at 1:49 pm on October 28, 2025:

When do we do a batch write though in this example?

l0rinc commented at 4:28 pm on October 28, 2025:

Originally we flushed, so if we still want to, we may want to extend this to assert that behavior. Or avoid flushing and simplify the bench. Or add the flushing behavior above to a test, etc.

andrewtoth commented at 4:37 pm on October 28, 2025:

We avoid flushing now.

l0rinc commented at 6:08 pm on November 2, 2025:

I think we could still add

0        ankerl::nanobench::doNotOptimizeAway(&temp_cache);
1        Assert(temp_cache.GetCacheSize() == 4599);

to document that the benchmark can loop now (i.e. every iteration should be the same)

in src/validation.cpp:3140 in a07936a62c

3136@@ -3137,6 +3137,8 @@ bool Chainstate::ConnectTip(
3137     LogDebug(BCLog::BENCH, "  - Load block from disk: %.2fms\n",
3138              Ticks<MillisecondsDouble>(time_2 - time_1));
3139     {
3140+        m_chainman.FetchInputs(CoinsTip(), CoinsDB(), *block_to_connect);

l0rinc commented at 2:30 pm on October 21, 2025:

Given that we already create the actual temporary top cache here, it would be great if we could separate the reading and writing (read from old/big in-memory cache and write to the temporary small one):

0CCoinsViewCache* cache{&CoinsTip()};
1CCoinsViewCache new_cache{cache};
2m_chainman.FetchInputs(*cache, new_cache, CoinsDB(), *block_to_connect);

This would also mean that the actually read changes are read after that from the top-layer cache instead of always requiring two hops (assuming many missing ones). It also makes sense to not add inputs from blocks that fail validation - which we’d be getting for free this way.

andrewtoth commented at 1:50 pm on October 28, 2025:

Done.

l0rinc commented at 3:32 pm on October 21, 2025: contributor

I love the new structure, it’s a lot easier to track the progress compared to previous versions. It’s also measurably faster than previous versions, we seem to be nearing ~20% - sweeeet! It also pairs well with other Siphash related changes that are in the pipeline.

I have reimplemented most of it locally to make sure I can do a meaningful review, see https://github.com/l0rinc/bitcoin/pull/47 for my attempt. It’s not finished, I’m still experimenting with different alternatives to make sure we can make this as simple and useful as possible (e.g. using barriers, filtering and sorting before fetch, adding dedicated ThreadPool, splitting reading and writing caches etc.), but wanted to publish my observations so far.

My remaining biggest concern is that the threadpool should definitely be untangled from the InputFetcher logic (I hate concurrent mutability), it’s an independent concern mixed with non-trivial fetcher logic. We also shouldn’t parallelize based on CPU in the first place, I don’t see why we’d do that. And the ThreadPool part still needs independent tests.

I have tried fetching everything on a single thread and the same with sorted UTXOs and it does seem to be ~6% faster on an SSD (I’d expected even bigger difference on HDD, still measuring that) - but I don’t have a final solution that’s faster than everything else (since I could only solve it by single-threaded filtering which is slow).

0b6ccd542fd single-threaded NO sort
1c50a6bd981 single-threaded + sorted fetch
2
3reindex-chainstate | 700000 blocks | dbcache 100 | i9-ssd | x86_64 | Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz | 16 cores | 62Gi RAM | xfs | SSD
4 
5Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=700000 -dbcache=100 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = b6ccd542fdc371d3ecd90164169e3d5d7c60e82d)
6  Time (abs ≡):        11322.014 s               [User: 20431.572 s, System: 1375.995 s]
7 
8Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=700000 -dbcache=100 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = c50a6bd9815ebb2e0d86d5e6b679da5ac021b796)
9  Time (abs ≡):        10646.049 s               [User: 19231.516 s, System: 1268.395 s]

andrewtoth commented at 4:55 pm on October 21, 2025: contributor

Thank you for your review @l0rinc!

I love the new structure, it’s a lot easier to track the progress compared to previous versions. It’s also measurably faster than previous versions, we seem to be nearing ~20% - sweeeet!

:rocket:

we can make this as simple and useful as possible

What else do you have in mind for this being useful? It has a very focused purpose in my view. I would love to make it as simple as possible.

using barriers

Done :)

filtering and sorting before fetch

Can you expand on why we would want this? Prefiltering on the main thread was always slower than filtering in parallel when I was testing this idea.

adding dedicated ThreadPool

Yes, see #26966. Hopefully we can pull that out and it can be used here. See https://github.com/andrewtoth/bitcoin/tree/test-thread-pool.

splitting reading and writing caches

I’m not sure what you mean by this? There is only one cache, and we read from it concurrently, then write to it on a single thread. Can you expand on how we can split the cache into two? Also, why would we want separate caches?

My remaining biggest concern is that the threadpool should definitely be untangled from the InputFetcher logic

Is this not accomplished by ThreadLoop, Work and OnCompletion functions? The Work function has no thread pool logic at all, it is completely independent and can be run by one thread or many. Do you have any concrete suggestions, or specific codepaths that are tangled that are concerning?

I hate concurrent mutability

I’m not sure I understand this. There is no concurrent mutability in my implementation. That would be undefined behavior. Can you point out the data members that are being concurrently mutated and how? Perhaps we have different definitions of concurrent mutability.

We also shouldn’t parallelize based on CPU in the first place, I don’t see why we’d do that.

This is following the logic of the check queue. Ideally we could reuse both threads in the same threadpool in the future. Do you have any concrete recommendations on how we many threads we should run? Using the number of threads per CPU is yielding a 20% speed boost, so it seems like a sane enough choice.

l0rinc commented at 6:02 pm on October 24, 2025: contributor

Let me summarize our offline discussions:

Cache hierarchy

During block connection we’re adding an extra temporary in-memory dbcache layer on top so that whatever happens during block connection doesn’t end up polluting the big dbcache or leveldb. I think we should take advantage of this and use the temporary top layer to collect the missing inputs there:

if block validation fails we can just throw it out
it’s easier to test and benchmark, we can rerun the same operation without changing the underlying state
the missing values will all be fetched from the top layer now, avoiding two-hop lookups
since the threads aren’t reading from the temp cache and aren’t writing to the big cache, we can copy the needed inputs from the big in-memory cache contents on the main thread as well to the temporary top layer (without locking) while the other threads are fetching from the DB (since those are IO bound, this being CPU bound). That would create a dedicated dbcache per blocks which we could eventually flush independently.

This might need some workarounds, but it does enable new cache invalidation opportunities.

Sorted fetch

Since LevelDB writes (via the CDBBatch and MemTable) are already sorted and result in a significant speedup compared to inserting the values one-by-one (though likely this isn’t just the effect of sorting), I thought it would make sense to experiment with sorted fetches as well. I have added an extra fetch (similarly to this PR, see https://github.com/l0rinc/bitcoin/commit/b72f67d4a88495a0222cbb9ae825daa6ea38e4df) before calling ConnectBlock so that the cache is pre-warmed, but on a single thread. This is the baseline, in the next commit after gathering the missing values I’m sorting them and doing the fetch from the db one-by-one, but in a sorted order.

The results indicate that sorted fetching is a lot faster, especially on lower-end devices (i7 with HDD and Rpi5 with SSD).

 0COMMITS="7d27af98c7cf858b5ab5a02e64f89a857cc53172 ccc748b05858a9ebeb375cc1f3e7426698394470 ead8da2b33117807e27a66f814cf11cdf676d194"; \
 1STOP=919191; DBCACHE=450; \
 2CC=gcc; CXX=g++; \
 3BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
 4(echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done) && \
 5(echo "" && echo "reindex-chainstate | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(df -T $BASE_DIR | awk 'NR==2{print $2}') | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 0 && echo SSD || echo HDD)"; echo "") &&\
 6hyperfine \
 7  --sort command \
 8  --runs 1 \
 9  --export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \
10  --parameter-list COMMIT ${COMMITS// /,} \
11  --prepare "killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
12    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \
13    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20" \
14  --conclude "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log && \
15             grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log" \
16  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0"
17
187d27af98c7 Merge bitcoin/bitcoin#33461: ci: add Valgrind fuzz
19ccc748b058 coins: prefetch inputs on single thread
20ead8da2b33 coins: sorted input prefetch
21
22reindex-chainstate | 919191 blocks | dbcache 450 | rpi5-16-2 | aarch64 | Cortex-A76 | 4 cores | 15Gi RAM | ext4 | SSD
23
24Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=919191 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 7d27af98c7cf858b5ab5a02e64f89a857cc53172)
25  Time (abs ≡):        42271.083 s               [User: 66842.872 s, System: 8003.776 s]
26 
27Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=919191 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = ccc748b05858a9ebeb375cc1f3e7426698394470)
28  Time (abs ≡):        40776.988 s               [User: 67394.771 s, System: 7130.144 s]
29 
30Benchmark 3: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=919191 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = ead8da2b33117807e27a66f814cf11cdf676d194)
31  Time (abs ≡):        38435.516 s               [User: 64537.357 s, System: 6716.634 s]
32 
33Relative speed comparison
34        1.10          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=919191 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 7d27af98c7cf858b5ab5a02e64f89a857cc53172)
35        1.06          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=919191 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = ccc748b05858a9ebeb375cc1f3e7426698394470)
36        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=919191 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = ead8da2b33117807e27a66f814cf11cdf676d194)

(interestingly it seems this is even faster than master by a lot, we’re not yet sure what’s causing that since the fetches are still single-threaded)

Similar speedup on HDD:

 0STOP=919191; DBCACHE=450; \                                                                                                                   
 1CC=gcc; CXX=g++; \                                                                                                                                                                  
 2BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \                                                                                           
 3(echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done) && \                                                                      
 4(echo "" && echo "reindex-chainstate | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cor
 5es | $(free -h | awk '/^Mem:/{print $2}') RAM | $(df -T $BASE_DIR | awk 'NR==2{print $2}') | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 0 && echo SSD || e
 6cho HDD)"; echo "") &&\                                                                                                                                                             
 7hyperfine \                                  
 8  --sort command \                           
 9  --runs 1 \                                 
10  --export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \                                                                      
11  --parameter-list COMMIT ${COMMITS// /,} \                                               
12  --prepare "killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \                                                  
13    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \                                                                  
14    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20" \                                                                        
15  --conclude "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log && \                                                                                                   
16             grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log" \                                                                                
17  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0"                         
187d27af98c7 Merge bitcoin/bitcoin#33461: ci: add Valgrind fuzz                             
19ccc748b058 coins: prefetch inputs on single thread                                        
20ead8da2b33 coins: sorted input prefetch                                                   
21
22reindex-chainstate | 919191 blocks | dbcache 450 | i7-hdd | x86_64 | Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz | 8 cores | 62Gi RAM | ext4 | HDD                                      
23
24Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=919191 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 7d27af98c7cf858b5ab5a02e64f89a857cc53172)                                        
25  Time (abs ≡):        42897.195 s               [User: 38613.661 s, System: 3154.561 s]                                                                                            
26                                             
27Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=919191 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = ccc748b05858a9ebeb375cc1f3e7426698394470)                                               
28  Time (abs ≡):        42015.404 s               [User: 40096.242 s, System: 3180.038 s]                                                                                            
29                                             
30Benchmark 3: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=919191 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = ead8da2b33117807e27a66f814cf11cdf676d194)                                               
31  Time (abs ≡):        40176.361 s               [User: 38614.897 s, System: 3047.461 s]                                                                                            
32                                             
33Relative speed comparison                    
34        1.07          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=919191 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 7d27af98c7cf858b5ab5a02e64f89a857cc53172)                               
35        1.05          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=919191 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = ccc748b05858a9ebeb375cc1f3e7426698394470)                               
36        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=919191 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = ead8da2b33117807e27a66f814cf11cdf676d194)

On a very powerful i9 with very performant SSD sorting isn’t faster than master, but it’s still a lot faster than random fetching:

 0STOP=919191; DBCACHE=450; \                                                                                                      
 1CC=gcc; CXX=g++; \                                                                                                                                                     
 2BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \                                                                              
 3(echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done) && \                                                         
 4(echo "" && echo "reindex-chainstate | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(df -T $BASE_DIR | awk 'NR==2{print $2}') | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 0 && echo SSD || echo HDD)"; echo "") &&\                                  
 5hyperfine \                              
 6  --sort command \                       
 7  --runs 1 \                             
 8  --export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \                                                         
 9  --parameter-list COMMIT ${COMMITS// /,} \                                        
10  --prepare "killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \                                     
11    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \                                                     
12    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20" \                                                           
13  --conclude "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log && \                                                                                      
14             grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log" \                                                                   
15  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0"            
16
177d27af98c7 Merge bitcoin/bitcoin#33461: ci: add Valgrind fuzz                      
18ccc748b058 coins: prefetch inputs on single thread                                 
19ead8da2b33 coins: sorted input prefetch                                            
20
21reindex-chainstate | 919191 blocks | dbcache 450 | i9-ssd | x86_64 | Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz | 16 cores | 62Gi RAM | xfs | SSD                        
22
23Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=919191 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 7d27af98c7cf858b5ab5a02e64f89a857cc53172)                    
24  Time (abs ≡):        20383.488 s               [User: 38259.799 s, System: 2702.188 s]                                                                               
25                                         
26Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=919191 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = ccc748b05858a9ebeb375cc1f3e7426698394470)                    
27  Time (abs ≡):        21213.645 s               [User: 39728.945 s, System: 2709.999 s]                                                                               
28                                         
29Benchmark 3: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=919191 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = ead8da2b33117807e27a66f814cf11cdf676d194)                    
30  Time (abs ≡):        20476.751 s               [User: 38448.254 s, System: 2597.600 s]                                                                               
31                                         
32Relative speed comparison                
33        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=919191 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 7d27af98c7cf858b5ab5a02e64f89a857cc53172)           
34        1.04          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=919191 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = ccc748b05858a9ebeb375cc1f3e7426698394470)           
35        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=919191 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = ead8da2b33117807e27a66f814cf11cdf676d194)

Theoretically this can also be affected by LevelDB’s internal options.block_cache and options.block_size and iteroptions.fill_cache (currently experimenting with different values to see if it makes any difference). This is why I have suggested trying a read-only snapshot per thread, if they’re reading sorted values the previous path could theoretically be caches which can help avoid a few lookups.

Separate ThreadPool

I think we should be able to use a single barrier for starting and stopping, assuming that we either want to use all of them or none of them, something like: https://github.com/l0rinc/bitcoin/commit/67e79e041b524eb1b0039ff39d4fc5b235d89b8f#diff-7ad6c5646dfd3749d2634861dfaf98e8c6fbc041a8e8eded973ff39929acffd7 or #33689 This would also help with completely separating the multithreaded part from the Inputfetcher which would also allow testing that critical behavior separately. This would also help with minimizing the mutable state which could theoretically allow concurrent calls to swap out the work from under the started threads (so likely the m_task in my example should be atomic, haven’t given it enough thought).

Since sorted-fetching currently needs a preprocessing step, we could use these available threads for filtering operations as well (since single-threaded filtering is slow, as proven above) - we could filter on all threads for missing, sort the missing on main thread and spread out to all threads for db fetching - leaving the main thread to do other things (since db fetching isn’t really CPU bound, no need to involve the main thread), such as the mentioned outer-layer warming from the main in-memory dbcache. And alternative to the “fetch-missing & sort & get” on a single thread is to try “fetch-all & sort & filter & get” instead which is more parallelizable. Since everything in the outer-layer will be spent anyway, in the future maybe we could even flush them on a background thread to db (given that we’ve just copied the items to the temp cache, after successful flush we could just remove them from the main cache) - which would likely simplify the main dbcache’s dirty and spent behaviors, but that’s outside of the scope of this PR but could provide extra motivation for these changes).

andrewtoth force-pushed on Oct 27, 2025

andrewtoth commented at 1:58 am on October 27, 2025: contributor

Thank you @l0rinc for your detailed review and suggestions! I have taken some of them. The input fetcher has now been redesigned.

Coins are written to the ephemeral cache that is created just to be used in ConnectBlock, instead of the main cache. This requires a new method in CCoinsViewCache - GetPossiblySpentCoinFromCache. Since we write to an empty cache instead of CoinsTip(), we could insert a Coin that exists in the db but is spent in CoinsTip(). Previously that insertion would fail since we were inserting again into CoinsTip() which would not overwrite the spent coin.
Because of this we can safely write to the ephemeral cache on the main thread, since the worker threads will be reading from a different cache. So the main thread does not do any fetching, but it writes fetched Coins to the cache in parallel as workers fetch them. It is a lock-free MPSC queue. The workers and main thread synchronize on an atomic Status member of each input. The workers set it to READY while the main thread spins on it until it is no longer WAITING. This is a substantial speed improvement over the previous version, and it also lets us insert Coins from the main cache into the ephemeral cache faster. So this change speeds up block connection even when there are no cache misses, and makes the workers utilize more CPU rather than just IO.
A single barrier is used to synchronize the threads. Both the main and worker threads call arrive_and_wait before they begin their work and after completion.

I removed the last commit, and instead add the InputFetcher as multi threaded initially. It didn’t make sense to split it since the worker threads and main threads do different things now.

I haven’t been able to effectively utilize a sorting strategy. Sorting the inputs before fetching doesn’t seem to have a benefit in the multi threaded approach. The overhead of copying the COutPoints and sorting them was always slower. I think the parallel fetching dominates any speedup achieved by sorted fetching anyways.

andrewtoth force-pushed on Oct 27, 2025

l0rinc commented at 9:48 pm on October 27, 2025: contributor

Looks like I forgot to push the comments last time - that explains your surprise. I pushed all of them, many are out of date, please just resolve them. Sorry for the confusion.

Since I’m reposting these old comments (since commented from the main GitHub view), lemme’ post a recently finished Rpi4 benchmark showing a 31% speedup for an older push \:D/

 0COMMITS="063946d6bd78035276d12e070a208d84492ac5cd 64de91105312d36dadb5f71ec01fc6af9b14da69"; \
 1STOP=919191; DBCACHE=450; \
 2CC=gcc; CXX=g++; \
 3BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
 4(echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done) && \
 5(echo "" && echo "reindex-chainstate | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(df -T $BASE_DIR | awk 'NR==2{print $2}') | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 0 && echo SSD || echo HDD)"; echo "") &&\
 6hyperfine \
 7  --sort command \
 8  --runs 1 \
 9  --export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \
10  --parameter-list COMMIT ${COMMITS// /,} \
11  --prepare "killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
12    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \
13    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20" \
14  --conclude "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log && \
15             grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log" \
16  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0"
17
18063946d6bd coins: add inputfetcher
1964de911053 coins: fetch coins on parallel threads
20
21reindex-chainstate | 919191 blocks | dbcache 450 | rpi4-8-1 | aarch64 | Cortex-A72 | 4 cores | 7.6Gi RAM | ext4 | SSD
22
23Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=919191 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 063946d6bd78035276d12e070a208d84492ac5cd)
24  Time (abs ≡):        329713.086 s               [User: 172834.637 s, System: 87207.916 s]
25 
26Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=919191 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 64de91105312d36dadb5f71ec01fc6af9b14da69)
27  Time (abs ≡):        251976.600 s               [User: 177923.005 s, System: 116804.580 s]
28 
29Relative speed comparison
30        1.31          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=919191 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 063946d6bd78035276d12e070a208d84492ac5cd)
31        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=919191 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 64de91105312d36dadb5f71ec01fc6af9b14da69)

andrewtoth force-pushed on Oct 28, 2025

in src/bench/inputfetcher.cpp:28 in 2aa5103481 outdated

23+        Coin coin{};
24+        coin.out.nValue = 1;
25+        return coin;
26+    }
27+
28+    bool BatchWrite(CoinsViewCacheCursor&, const uint256&) override { return true; }

l0rinc commented at 4:40 pm on October 28, 2025:

nit: we could throw now to indicate we don’t have unrepeatable side-effects

andrewtoth commented at 11:08 pm on November 2, 2025:

We could just get rid of this line now.

andrewtoth commented at 11:31 pm on November 2, 2025:

Got rid of it.

in src/bench/inputfetcher.cpp:18 in 2aa5103481 outdated

13+#include <util/time.h>
14+
15+static constexpr auto DELAY{2ms};
16+
17+//! Simulates a DB by adding a delay when calling GetCoin
18+struct DelayedCoinsView : CCoinsView

l0rinc commented at 4:40 pm on October 28, 2025:

Nit: structs are now formatted consistently on the same line

andrewtoth commented at 11:32 pm on November 2, 2025:

Done.

l0rinc commented at 8:44 am on November 3, 2025:

There are other ones that I didn’t mention, reformatted the change, please take the ones that make sense:

  0diff --git a/src/coins.cpp b/src/coins.cpp
  1index 2ef2e36ccc..baac1a32b5 100644
  2--- a/src/coins.cpp
  3+++ b/src/coins.cpp
  4@@ -173,7 +173,8 @@ bool CCoinsViewCache::HaveCoinInCache(const COutPoint &outpoint) const {
  5     return (it != cacheCoins.end() && !it->second.coin.IsSpent());
  6 }
  7 
  8-std::optional<Coin> CCoinsViewCache::GetPossiblySpentCoinFromCache(const COutPoint &outpoint) const noexcept {
  9+std::optional<Coin> CCoinsViewCache::GetPossiblySpentCoinFromCache(const COutPoint& outpoint) const noexcept
 10+{
 11     if (auto it{cacheCoins.find(outpoint)}; it != cacheCoins.end()) return it->second.coin;
 12     return std::nullopt;
 13 }
 14diff --git a/src/coins.h b/src/coins.h
 15index e7d28ace97..02c3ea9e15 100644
 16--- a/src/coins.h
 17+++ b/src/coins.h
 18@@ -407,7 +407,7 @@ public:
 19      * Used in InputFetcher to make sure we do not add a coin from the backing
 20      * view when it is spent in the cache but not yet flushed to the parent.
 21      */
 22-    std::optional<Coin> GetPossiblySpentCoinFromCache(const COutPoint &outpoint) const noexcept;
 23+    std::optional<Coin> GetPossiblySpentCoinFromCache(const COutPoint& outpoint) const noexcept;
 24 
 25     /**
 26      * Return a reference to Coin in the cache, or coinEmpty if not found. This is
 27diff --git a/src/inputfetcher.h b/src/inputfetcher.h
 28index a8a3f4d1ad..74c655caf1 100644
 29--- a/src/inputfetcher.h
 30+++ b/src/inputfetcher.h
 31@@ -46,8 +46,8 @@ private:
 32     struct Input {
 33         enum class Status : uint8_t {
 34             WAITING, // The coin has not been fetched yet
 35-            READY, // The coin has been fetched and is ready to be inserted into the cache
 36-            FAILED, // The coin failed to be fetched
 37+            READY,   // The coin has been fetched and is ready to be inserted into the cache
 38+            FAILED,  // The coin failed to be fetched
 39             SKIPPED, // The coin is created and spent in the same block so cannot be fetched
 40         };
 41 
 42diff --git a/src/test/fuzz/inputfetcher.cpp b/src/test/fuzz/inputfetcher.cpp
 43index cd2a0f5c68..70f5153912 100644
 44--- a/src/test/fuzz/inputfetcher.cpp
 45+++ b/src/test/fuzz/inputfetcher.cpp
 46@@ -18,8 +18,7 @@
 47 
 48 using DbMap = std::map<const COutPoint, std::pair<std::optional<const Coin>, bool>>;
 49 
 50-struct DbCoinsView : CCoinsView
 51-{
 52+struct DbCoinsView : CCoinsView {
 53     DbMap& m_map;
 54     DbCoinsView(DbMap& map) noexcept : m_map(map) {}
 55 
 56@@ -35,8 +34,7 @@ struct DbCoinsView : CCoinsView
 57     }
 58 };
 59 
 60-struct NoAccessCoinsView : CCoinsView
 61-{
 62+struct NoAccessCoinsView : CCoinsView {
 63     std::optional<Coin> GetCoin(const COutPoint&) const override { abort(); }
 64 };
 65 
 66@@ -49,7 +47,8 @@ FUZZ_TARGET(inputfetcher)
 67         fuzzed_data_provider.ConsumeIntegralInRange<int32_t>(2, 4)};
 68     InputFetcher fetcher{worker_threads};
 69 
 70-    LIMITED_WHILE(fuzzed_data_provider.ConsumeBool(), 10000) {
 71+    LIMITED_WHILE(fuzzed_data_provider.ConsumeBool(), 10000)
 72+    {
 73         CBlock block;
 74         Txid prevhash{Txid::FromUint256(ConsumeUInt256(fuzzed_data_provider))};
 75 
 76@@ -61,13 +60,13 @@ FUZZ_TARGET(inputfetcher)
 77         NoAccessCoinsView back;
 78         CCoinsViewCache main_cache(&back);
 79 
 80-        LIMITED_WHILE(fuzzed_data_provider.ConsumeBool(), 10000) {
 81+        LIMITED_WHILE(fuzzed_data_provider.ConsumeBool(), 10000)
 82+        {
 83             CMutableTransaction tx;
 84 
 85-            LIMITED_WHILE(fuzzed_data_provider.ConsumeBool(), 10) {
 86-                const auto txid{fuzzed_data_provider.ConsumeBool()
 87-                    ? Txid::FromUint256(ConsumeUInt256(fuzzed_data_provider))
 88-                    : prevhash};
 89+            LIMITED_WHILE(fuzzed_data_provider.ConsumeBool(), 10)
 90+            {
 91+                const auto txid{fuzzed_data_provider.ConsumeBool() ? Txid::FromUint256(ConsumeUInt256(fuzzed_data_provider)) : prevhash};
 92                 const auto index{fuzzed_data_provider.ConsumeIntegral<uint32_t>()};
 93                 const COutPoint outpoint(txid, index);
 94 
 95@@ -87,8 +86,8 @@ FUZZ_TARGET(inputfetcher)
 96                     maybe_coin = std::nullopt;
 97                 }
 98                 db_map.try_emplace(outpoint, std::make_pair(
 99-                    maybe_coin,
100-                    fuzzed_data_provider.ConsumeBool()));
101+                                                 maybe_coin,
102+                                                 fuzzed_data_provider.ConsumeBool()));
103 
104                 // Add the coin to the cache
105                 if (fuzzed_data_provider.ConsumeBool()) {
106diff --git a/src/test/inputfetcher_tests.cpp b/src/test/inputfetcher_tests.cpp
107index 33fb8c6cb0..83f3d19432 100644
108--- a/src/test/inputfetcher_tests.cpp
109+++ b/src/test/inputfetcher_tests.cpp
110@@ -48,7 +48,7 @@ private:
111 
112 public:
113     explicit InputFetcherTest(const ChainType chainType = ChainType::MAIN,
114-                             TestOpts opts = {})
115+                              TestOpts opts = {})
116         : BasicTestingSetup{chainType, opts}
117     {
118         SeedRandomForTest(SeedRand::FIXED_SEED);
119@@ -200,8 +200,7 @@ BOOST_FIXTURE_TEST_CASE(fetch_no_inputs, InputFetcherTest)
120     }
121 }
122 
123-struct ThrowCoinsView : CCoinsView
124-{
125+struct ThrowCoinsView : CCoinsView {
126     std::optional<Coin> GetCoin(const COutPoint&) const override
127     {
128         throw std::runtime_error("database error");
129diff --git a/src/validation.h b/src/validation.h
130index 26141305cb..2ff7c5ef6d 100644
131--- a/src/validation.h
132+++ b/src/validation.h
133@@ -1341,7 +1341,8 @@ public:
134     void RecalculateBestHeader() EXCLUSIVE_LOCKS_REQUIRED(::cs_main);
135 
136     CCheckQueue<CScriptCheck>& GetCheckQueue() { return m_script_check_queue; }
137-    void FetchInputs(CCoinsViewCache& temp_cache, const CCoinsViewCache& main_cache, const CCoinsView& db, const CBlock& block) noexcept {
138+    void FetchInputs(CCoinsViewCache& temp_cache, const CCoinsViewCache& main_cache, const CCoinsView& db, const CBlock& block) noexcept
139+    {
140         m_input_fetcher.FetchInputs(temp_cache, main_cache, db, block);
141     }

andrewtoth commented at 10:15 pm on November 4, 2025:

Took the formats, thanks!

l0rinc commented at 8:40 pm on October 29, 2025: contributor

I’d say it’s time to update the PR description:

 0COMMITS="cb0fdfdf3704d5ffe6ccc634de6fdba6b7b57a85 2aa510348143521a14146e41b5cf87cb3e60b29e"; \
 1STOP=921129; DBCACHE=450; \
 2CC=gcc; CXX=g++; \
 3BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
 4(echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done) && \
 5(echo "" && echo "reindex-chainstate | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(df -T $BASE_DIR | awk 'NR==2{print $2}') | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 0 && echo SSD || echo HDD)"; echo "") &&\
 6hyperfine \
 7  --sort command \
 8  --runs 1 \
 9  --export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \
10  --parameter-list COMMIT ${COMMITS// /,} \
11  --prepare "killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
12    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \
13    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20" \
14  --conclude "killall bitcoind || true; sleep 5; cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log; \
15              grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log" \
16  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0"
17
18cb0fdfdf37 coins: add inputfetcher
192aa5103481 validation: fetch block inputs via InputFetcher before connecting
20
21reindex-chainstate | 921129 blocks | dbcache 450 | rpi5-16-2 | aarch64 | Cortex-A76 | 4 cores | 15Gi RAM | ext4 | SSD
22
23Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = cb0fdfdf3704d5ffe6ccc634de6fdba6b7b57a85)
24  Time (abs ≡):        40539.887 s               [User: 69358.879 s, System: 7393.185 s]
25 
26Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 2aa510348143521a14146e41b5cf87cb3e60b29e)
27  Time (abs ≡):        32768.672 s               [User: 69495.022 s, System: 11880.553 s]
28 
29Relative speed comparison
30        1.24          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = cb0fdfdf3704d5ffe6ccc634de6fdba6b7b57a85)
31        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 2aa510348143521a14146e41b5cf87cb3e60b29e)

With this change we’re getting a 9 hours full reindex-chainstate on an rpi5 with default 450MB memory. Wow!

andrewtoth renamed this:
~~validation: fetch block inputs on parallel threads >10% faster IBD~~
validation: fetch block inputs on parallel threads >20% faster IBD
on Oct 29, 2025

andrewtoth force-pushed on Oct 31, 2025

l0rinc commented at 8:20 am on November 1, 2025: contributor

Rebased the latest version of the PR now that #31645 was merged and measured a reindex on my M4 laptop: it finished all 3 runs with dbcache 450, 4500 and 45000 overnight.

0time ./build/bin/bitcoind -stopatheight=921129 -dbcache=45000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 \
1&& time ./build/bin/bitcoind -stopatheight=921129 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 \
2&& time ./build/bin/bitcoind -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 
3
4./build/bin/bitcoind -stopatheight=921129 -dbcache=45000 -reindex-chainstate   8805.12s user 1456.47s system 134% cpu 2:07:22.36 total
5./build/bin/bitcoind -stopatheight=921129 -dbcache=4500 -reindex-chainstate    12201.80s user 3698.94s system 197% cpu 2:13:54.50 total
6./build/bin/bitcoind -stopatheight=921129 -dbcache=450 -reindex-chainstate     20676.11s user 10582.81s system 358% cpu 2:25:26.52 total

repeated the same later to check for stability:

0./build/bin/bitcoind -stopatheight=921129 -dbcache=45000 -reindex-chainstate   8873.87s user 1468.17s system 134% cpu 2:07:47.62 total
1./build/bin/bitcoind -stopatheight=921129 -dbcache=4500 -reindex-chainstate    12178.09s user 3621.39s system 197% cpu 2:13:11.48 total
2./build/bin/bitcoind -stopatheight=921129 -dbcache=450 -reindex-chainstate     20725.97s user 10666.85s system 359% cpu 2:25:33.12 total

Note: I did get a bitcoind(70369,0x16f5ff000) malloc: Failed to allocate segment from range group - out of space warning above, maybe 45 GB memory was a bit too much, but I can continue the block connection after the reindexes, so the measurements are likely representative. Edit: reran the 45 GB case, it did complete successfully in a similar time without errors.

reindex-chainstate seems correct, it can continue after:

0./build/bin/bitcoind
12025-11-01T08:16:25Z Bitcoin Core version v30.99.0-45fe0c0e5bed (release build)
2...
32025-11-01T08:16:29Z nBestHeight = 921129
4...
52025-11-01T08:16:29Z UpdateTip: new best=00000000000000000000a216ce3209897114b757dbac3b651c72484829637c0e height=921130 version=0x343ba000 log2_work=95.904905 tx=1262937761 date='2025-10-28T06:06:11Z' progress=0.998475 cache=0.5MiB(3812txo)
62025-11-01T08:16:29Z UpdateTip: new best=00000000000000000001965d9ff2ef639dd6829da2ad19c3f5d46691475f0df1 height=921131 version=0x25472000 log2_work=95.904918 tx=1262938885 date='2025-10-28T06:13:50Z' progress=0.998477 cache=1.6MiB(9726txo)

andrewtoth force-pushed on Nov 1, 2025

andrewtoth commented at 9:47 pm on November 1, 2025: contributor

Rebased to test behavior with #31645. Some other touch-ups include:

Use const COutPoint& in Input struct instead of vin and vtx indexes
Cleanup shared vectors and pointers at the end of loop
Refactor inner work loop to do fewer existence checks at the expense of some duplicated code
Use std::atomic_usize_t instead of std::atomic<usize_t>
Use GetPossiblySpentCoinFromCache in tests and fuzz harness for better clarity and correctness

in src/coins.h:489 in 62868c8846

481@@ -474,6 +482,14 @@ class CCoinsViewCache : public CCoinsViewBacked
482     //! See: https://stackoverflow.com/questions/42114044/how-to-release-unordered-map-memory
483     void ReallocateCache();
484 
485+    /**
486+     * Reserve enough space in the cache so the underlying unordered_map will
487+     * not have to rehash unless capacity is exceeded.
488+     */
489+    void Reserve(size_t capacity) {

l0rinc commented at 2:39 pm on November 2, 2025:

Nit: can you please reformat the change with latest clang-format after rebase? Nit2: this seems to belong closer to GetCacheSize, one reserves, the other returns the actual size

andrewtoth commented at 11:32 pm on November 2, 2025:

I couldn’t get clang-format to work, but I made this a one-liner and moved under GetCacheSize.

in src/inputfetcher.h:165 in 62868c8846

160+        m_cache = &cache;
161+        m_input_head.store(0, std::memory_order_relaxed);
162+        m_barrier.arrive_and_wait();
163+
164+        // Insert fetched coins into the temp_cache as they are set to READY.
165+        temp_cache.Reserve(m_inputs.size() + outputs_count);

l0rinc commented at 2:50 pm on November 2, 2025:

I like that we’re doing this, we’ve usually avoided preallocating our maps - we should do it more often!

I know we’re providing empty caches via the parameters, but for completeness either assert that they’re empty or:

0        temp_cache.Reserve(temp_cache.GetCacheSize() + m_inputs.size() + outputs_count);

I have added

0        Assert(temp_cache.GetCacheSize() <= temp_cache.GetCacheSize() + m_inputs.size() + outputs_count);

after the for (auto& input : m_inputs) { loop to validate that the allocation achieves - the tests pass! 👍

andrewtoth commented at 11:32 pm on November 2, 2025:

Done.

in src/inputfetcher.h:117 in 62868c8846 outdated

144+        }
145+
146+        // Loop through the inputs of the block and set them in the queue.
147+        // Construct the set of txids to filter, and count the outputs to reserve for temp_cache.
148+        auto outputs_count{block.vtx[0]->vout.size()};
149+        for (size_t i{1}; i < block.vtx.size(); ++i) {

l0rinc commented at 2:57 pm on November 2, 2025:

we’re also just adding stuff to the m_inputs without any reservation, the first few times we will have some needless copying - we could likely alleviate that by doing a rough reservation

0        m_txids.reserve(block.vtx.size());
1        m_inputs.reserve(2 * block.vtx.size()); // rough guess

andrewtoth commented at 11:07 pm on November 2, 2025:

This won’t have a measurable effect if we are connecting lots of blocks though.

andrewtoth commented at 11:33 pm on November 2, 2025:

Did it anyways.

in src/inputfetcher.h:64 in 62868c8846 outdated

66+    /**
67+     * The set of txids of all txs in the block being fetched.
68+     * Used to filter out inputs that are created and spent in the same block,
69+     * since they will not be in the db or the cache.
70+     */
71+    std::unordered_set<Txid, SaltedTxidHasher> m_txids{};

l0rinc commented at 3:06 pm on November 2, 2025:

I understand if you don’t want to bother with this, but how many of these do we expect per block? I wonder if we want to incur the hashing cost here or if using a sorted set or a sorted vector with std::binary_search (or even (unsorted?) SIMD-enabled linear scan) would be faster or simpler here.

I also thought of whether we could just add these to temp_cache directly, but that would likely pollute the up-coming validation (unless we can differentiate these from existing inputs).

Taking block413567, we can assuming for the benchmark that 5% of the txs are internal spends (we need to measure this properly before we decide) and assuming 1556 txs (4886 inputs, 3581 outputs), 287 internal spends, we can compare the alternatives, here’s a bench for motivation.

  0// Copyright (c) 2022-present The Bitcoin Core developers
  1// Distributed under the MIT software license, see the accompanying
  2// file COPYING or https://www.opensource.org/licenses/mit-license.php.
  3
  4#include <bench/bench.h>
  5#include <bench/nanobench.h>
  6#include <primitives/transaction_identifier.h>
  7#include <random.h>
  8#include <util/check.h>
  9#include <util/hasher.h>
 10
 11#include <algorithm>
 12#include <ranges>
 13#include <set>
 14#include <unordered_set>
 15#include <vector>
 16
 17namespace {
 18
 19constexpr size_t iterations{100}; // since the inputs of the benchmarks are mutated by sorting, we can't rerun the benchmarks
 20constexpr size_t hits_count{275}; // assuming ~5% of blocks contain internal spends
 21constexpr size_t tx_count{5500};
 22
 23struct Dataset {
 24    std::set<Txid> sorted_set;
 25    std::unordered_set<Txid, SaltedTxidHasher> unsorted_set;
 26    std::vector<Txid> vec_sorted;
 27    std::vector<Txid> vec_unsorted;
 28
 29    std::vector<Txid> queries;
 30};
 31
 32std::vector<Dataset> BuildDatasets()
 33{
 34    FastRandomContext rng(/*fDeterministic=*/true);
 35
 36    std::vector<Dataset> datasets;
 37    datasets.reserve(iterations);
 38
 39    for (size_t d{0}; d < iterations; ++d) {
 40        Dataset ds;
 41        ds.queries.reserve(tx_count);
 42        ds.unsorted_set.reserve(tx_count);
 43        ds.vec_sorted.reserve(tx_count);
 44        ds.vec_unsorted.reserve(tx_count);
 45
 46        for (size_t i{0}; i < tx_count; ++i) {
 47            Txid t{Txid::FromUint256(rng.rand256())};
 48            ds.sorted_set.emplace(t);
 49            ds.unsorted_set.emplace(t);
 50            ds.vec_sorted.emplace_back(t);
 51            ds.vec_unsorted.emplace_back(t);
 52
 53            ds.queries.emplace_back(i < hits_count ? t : Txid::FromUint256(rng.rand256()));
 54        }
 55
 56        std::ranges::shuffle(ds.queries, rng);
 57        std::ranges::shuffle(ds.vec_unsorted, rng);
 58        std::sort(ds.vec_sorted.begin(), ds.vec_sorted.end());
 59
 60        datasets.emplace_back(std::move(ds));
 61    }
 62    return datasets;
 63}
 64
 65} // namespace
 66
 67static void Txid_UnorderedSalted(benchmark::Bench& bench)
 68{
 69    static auto ds{BuildDatasets()};
 70    bench.epochs(1).epochIterations(1).batch(iterations).run([&] {
 71        size_t sum{0};
 72        for (const auto& s : ds) {
 73            for (const auto& q : s.queries) {
 74                sum += s.unsorted_set.contains(q);
 75            }
 76        }
 77        ankerl::nanobench::doNotOptimizeAway(sum);
 78        Assert(sum == iterations * hits_count);
 79    });
 80}
 81
 82static void Txid_SetOrdered(benchmark::Bench& bench)
 83{
 84    static auto ds{BuildDatasets()};
 85    bench.epochs(1).epochIterations(1).batch(iterations).run([&] {
 86        size_t sum{0};
 87        for (const auto& s : ds) {
 88            for (const auto& q : s.queries) {
 89                sum += s.sorted_set.contains(q);
 90            }
 91        }
 92        ankerl::nanobench::doNotOptimizeAway(sum);
 93        Assert(sum == iterations * hits_count);
 94    });
 95}
 96
 97static void Txid_VectorBinarySearch(benchmark::Bench& bench)
 98{
 99    static auto ds{BuildDatasets()};
100    bench.epochs(1).epochIterations(1).batch(iterations).run([&] {
101        size_t sum{0};
102        for (const auto& s : ds) {
103            for (const auto& q : s.queries) {
104                sum += std::binary_search(s.vec_sorted.begin(), s.vec_sorted.end(), q);
105            }
106        }
107        ankerl::nanobench::doNotOptimizeAway(sum);
108        Assert(sum == iterations * hits_count);
109    });
110}
111
112static void Txid_VectorLinearScan(benchmark::Bench& bench)
113{
114    static auto ds{BuildDatasets()};
115    const auto contains_linear{[](const std::vector<Txid>& v, const Txid& x) noexcept {
116        return std::ranges::find(v, x) != v.end();
117    }};
118    bench.epochs(1).epochIterations(1).batch(iterations).run([&] {
119        size_t sum{0};
120        for (const auto& s : ds) {
121            for (const auto& q : s.queries) {
122                sum += contains_linear(s.vec_unsorted, q);
123            }
124        }
125        ankerl::nanobench::doNotOptimizeAway(sum);
126        Assert(sum == iterations * hits_count);
127    });
128}
129
130BENCHMARK(Txid_UnorderedSalted, benchmark::PriorityLevel::LOW);
131BENCHMARK(Txid_SetOrdered, benchmark::PriorityLevel::LOW);
132BENCHMARK(Txid_VectorBinarySearch, benchmark::PriorityLevel::LOW);
133BENCHMARK(Txid_VectorLinearScan, benchmark::PriorityLevel::LOW);

0cmake -B build -DBUILD_BENCH=ON -DENABLE_IPC=OFF -DCMAKE_BUILD_TYPE=Release && cmake --build build -j$(nproc) && build/bin/bench_bitcoin -filter='Txid_.*'

ns/op	op/s	err%	total	benchmark
217,247.09	4,603.05	0.0%	0.02	`Txid_SetOrdered`
202,607.50	4,935.65	0.0%	0.02	`Txid_UnorderedSalted`
194,347.91	5,145.41	0.0%	0.02	`Txid_VectorBinarySearch`
12,449,880.83	80.32	0.0%	1.24	`Txid_VectorLinearScan`

ns/op	op/s	err%	ins/op	cyc/op	IPC	bra/op	miss%	total	benchmark
1,179,326.18	847.94	0.0%	2,361,313.36	2,820,742.15	0.837	475,693.50	1.2%	0.12	`Txid_SetOrdered`
912,335.48	1,096.09	0.0%	1,346,377.49	2,183,167.97	0.617	84,874.38	5.7%	0.09	`Txid_UnorderedSalted`
732,493.03	1,365.20	0.0%	2,411,291.90	1,752,928.05	1.376	531,672.95	8.1%	0.07	`Txid_VectorBinarySearch`
25,449,442.73	39.29	0.0%	235,989,723.07	60,929,976.87	3.873	58,998,462.96	0.0%	2.54	`Txid_VectorLinearScan`

So the benchmark isn’t as revealing as I was hoping, please verify my assumptions and we can still test these with some macro-benchmark to see if there’s any performance or memory advantage to any of these.

andrewtoth commented at 9:38 pm on November 2, 2025:

I did try and use a sorted vector and do binary search in the workers, but it was not a measurable performance difference. In theory it should be faster since txid comparison is much faster than siphash. I can do this if you want, but I think it makes the code clearer using the unordered_set.

andrewtoth commented at 1:42 am on November 3, 2025:

I wrote benchmarks to check how fast it was to construct, since this is the work that is done not in parallel:

 0static void SortedVectorBenchmark(benchmark::Bench& bench)
 1{
 2    CBlock block;
 3    DataStream{benchmark::data::block413567} >> TX_WITH_WITNESS(block);
 4    std::vector<Txid> v{};
 5    v.reserve(block.vtx.size());
 6
 7    bench.run([&] {
 8        for (const auto& tx : block.vtx) {
 9            v.emplace_back(tx->GetHash());
10        }
11        std::sort(v.begin(), v.end());
12    });
13}
14
15static void UnorderedSetBenchmark(benchmark::Bench& bench)
16{
17    CBlock block;
18    DataStream{benchmark::data::block413567} >> TX_WITH_WITNESS(block);
19    std::unordered_set<Txid, SaltedTxidHasher> u{};
20    u.reserve(block.vtx.size());
21
22    bench.run([&] {
23        for (const auto& tx : block.vtx) {
24            u.emplace(tx->GetHash());
25        }
26    });
27}
28
29static void SetBenchmark(benchmark::Bench& bench)
30{
31    CBlock block;
32    DataStream{benchmark::data::block413567} >> TX_WITH_WITNESS(block);
33    std::set<Txid> s{};
34
35    bench.run([&] {
36        for (const auto& tx : block.vtx) {
37            s.insert(tx->GetHash());
38        }
39    });
40}

Results:

0|               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |         bra/op |   miss% |     total | benchmark
1|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
2|           45,794.65 |           21,836.61 |    1.6% |      777,851.91 |      110,515.70 |  7.038 |      88,000.15 |    0.5% |      0.01 | `UnorderedSetBenchmark`
3|           96,598.09 |           10,352.17 |    1.8% |      574,593.10 |      233,289.00 |  2.463 |     129,942.30 |    1.5% |      0.01 | `SetBenchmark`
4|        1,037,787.00 |              963.59 |    5.1% |   12,280,496.00 |    2,472,876.00 |  4.966 |   2,863,217.00 |    1.3% |      0.02 | :wavy_dash: `SortedVectorBenchmark` (Unstable with ~1.9 iters. Increase `minEpochIterations` to e.g. 19)

Obviously we don’t want to use the sorted vector. unordered_set is roughly twice as fast as set. I don’t think it’s worth pursuing a different filter container.

l0rinc commented at 9:09 am on November 3, 2025:

Hmm, I’m not sure this is correct.

Obviously we don’t want to use the sorted vector

That’s not what I’m getting. You were inserting to the same collection over and over, I’m not sure what we were measuring there. Adding the collection creation and an assertion to not optimize it away (to make sure we’re measuring the same thing in every iteration) reveals something completely different.

 0#include <bench/bench.h>
 1#include <bench/data/block413567.raw.h>
 2#include <coins.h>
 3#include <inputfetcher.h>
 4#include <primitives/block.h>
 5#include <serialize.h>
 6#include <streams.h>
 7
 8static void InputFetcher_SortedVectorBenchmark(benchmark::Bench& bench)
 9{
10    CBlock block;
11    DataStream{benchmark::data::block413567} >> TX_WITH_WITNESS(block);
12
13    bench.run([&] {
14        std::vector<Txid> v{};
15        v.reserve(block.vtx.size());
16        for (const auto& tx : block.vtx) {
17            v.emplace_back(tx->GetHash());
18        }
19        std::sort(v.begin(), v.end());
20        ankerl::nanobench::doNotOptimizeAway(v);
21    });
22}
23
24static void InputFetcher_UnorderedSetBenchmark(benchmark::Bench& bench)
25{
26    CBlock block;
27    DataStream{benchmark::data::block413567} >> TX_WITH_WITNESS(block);
28
29    bench.run([&] {
30        std::unordered_set<Txid, SaltedTxidHasher> u{};
31        u.reserve(block.vtx.size());
32        for (const auto& tx : block.vtx) {
33            u.emplace(tx->GetHash());
34        }
35        ankerl::nanobench::doNotOptimizeAway(u);
36    });
37}
38
39static void InputFetcher_SetBenchmark(benchmark::Bench& bench)
40{
41    CBlock block;
42    DataStream{benchmark::data::block413567} >> TX_WITH_WITNESS(block);
43
44    bench.run([&] {
45        std::set<Txid> s{};
46        for (const auto& tx : block.vtx) {
47            s.insert(tx->GetHash());
48        }
49        ankerl::nanobench::doNotOptimizeAway(s);
50    });
51}
52
53BENCHMARK(InputFetcher_SortedVectorBenchmark, benchmark::PriorityLevel::HIGH);
54BENCHMARK(InputFetcher_UnorderedSetBenchmark, benchmark::PriorityLevel::HIGH);
55BENCHMARK(InputFetcher_SetBenchmark, benchmark::PriorityLevel::HIGH);

ns/op	op/s	err%	ins/op	cyc/op	IPC	bra/op	miss%	total	benchmark
246,121.59	4,063.03	0.0%	978,180.25	883,481.68	1.107	224,954.25	2.8%	11.00	`InputFetcher_SetBenchmark`
130,850.86	7,642.29	0.2%	608,319.13	469,743.71	1.295	135,588.13	4.4%	11.04	`InputFetcher_SortedVectorBenchmark`
171,092.09	5,844.81	0.0%	1,207,752.53	614,142.35	1.967	174,278.58	1.1%	11.00	`InputFetcher_UnorderedSetBenchmark`

andrewtoth commented at 1:30 pm on November 3, 2025:

Hmm right I was not clearing the containers, so the sorting was dominating. I retried with clearing and the unordered_map is still slightly faster. In your benchmark you are creating and reserving the map inside the benchmark instead of before. In this implementation the reserved memory is kept in between blocks, so reserving and creating outside makes more sense.

l0rinc commented at 10:48 am on November 5, 2025:

In this implementation the reserved memory is kept in between blocks

K, so let’s reserve outside and clear inside. Now that missing values aren’t failures, we can experiment with shortids - since a missing value isn’t a failure anymore (even though I wouldn’t expect any collisions in 64 bits either, assuming uniform distribution. But even if the distribution isn’t uniform, we can likely store it safely). 64 bit ids for internal spends would mean that in case of some collision we will attempt to fetch something from disk that was actually in the current block - so the attacker can at best slow down block validation by a few milliseconds.

  0// Copyright (c) 2022-present The Bitcoin Core developers
  1// Distributed under the MIT software license, see the accompanying
  2// file COPYING or https://www.opensource.org/licenses/mit-license.php.
  3
  4#include <algorithm>
  5#include <bench/bench.h>
  6#include <bench/data/block413567.raw.h>
  7#include <bench/nanobench.h>
  8#include <coins.h>
  9#include <functional>
 10#include <inputfetcher.h>
 11#include <primitives/block.h>
 12#include <primitives/transaction_identifier.h>
 13#include <random.h>
 14#include <ranges>
 15#include <serialize.h>
 16#include <set>
 17#include <streams.h>
 18#include <unordered_set>
 19#include <util/check.h>
 20#include <util/hasher.h>
 21#include <vector>
 22
 23namespace {
 24constexpr size_t iterations{100}; // since the inputs of the benchmarks are mutated by sorting, we can't rerun the benchmarks
 25constexpr size_t hits_count{275}; // assuming ~5% of blocks contain internal spends
 26constexpr size_t tx_count{5500};
 27
 28uint64_t GetShortID(const Txid& txid)
 29{
 30    return txid.ToUint256().GetUint64(0);
 31}
 32
 33template <typename T>
 34struct Dataset {
 35    std::set<T> sorted_set;
 36    std::unordered_set<T, std::conditional_t<std::is_same_v<T, Txid>, SaltedTxidHasher, std::identity>> unsorted_set;
 37    std::vector<T> vec_sorted;
 38    std::vector<T> vec_unsorted;
 39    std::vector<T> queries;
 40
 41    static T Convert(const Txid& txid)
 42    {
 43        if constexpr (std::is_same_v<T, Txid>) {
 44            return txid;
 45        } else {
 46            static_assert(std::is_same_v<T, uint64_t>);
 47            return GetShortID(txid);
 48        }
 49    }
 50};
 51
 52template <typename T>
 53std::vector<Dataset<T>> BuildDatasets()
 54{
 55    FastRandomContext rng(/*fDeterministic=*/true);
 56
 57    std::vector<Dataset<T>> datasets;
 58    datasets.reserve(iterations);
 59
 60    for (size_t d{0}; d < iterations; ++d) {
 61        Dataset<T> ds;
 62        ds.queries.reserve(tx_count);
 63        ds.unsorted_set.reserve(tx_count);
 64        ds.vec_sorted.reserve(tx_count);
 65        ds.vec_unsorted.reserve(tx_count);
 66
 67        for (size_t i{0}; i < tx_count; ++i) {
 68            T t1{Dataset<T>::Convert(Txid::FromUint256(rng.rand256()))};
 69            ds.sorted_set.emplace(t1);
 70            ds.unsorted_set.emplace(t1);
 71            ds.vec_sorted.emplace_back(t1);
 72            ds.vec_unsorted.emplace_back(t1);
 73
 74            T t2{Dataset<T>::Convert(Txid::FromUint256(rng.rand256()))};
 75            ds.queries.emplace_back(i < hits_count ? t1 : t2);
 76        }
 77
 78        std::ranges::shuffle(ds.queries, rng);
 79        std::ranges::shuffle(ds.vec_unsorted, rng);
 80        std::sort(ds.vec_sorted.begin(), ds.vec_sorted.end());
 81
 82        datasets.emplace_back(std::move(ds));
 83    }
 84    return datasets;
 85}
 86} // namespace
 87
 88static void Txid_UnorderedSalted(benchmark::Bench& bench)
 89{
 90    static auto ds{BuildDatasets<Txid>()};
 91    bench.epochs(1).epochIterations(1).batch(iterations).run([&] {
 92        size_t sum{0};
 93        for (const auto& s : ds) {
 94            for (const auto& q : s.queries) {
 95                sum += s.unsorted_set.contains(q);
 96            }
 97        }
 98        ankerl::nanobench::doNotOptimizeAway(sum);
 99        Assert(sum == iterations * hits_count);
100    });
101}
102
103static void Txid_SetOrdered(benchmark::Bench& bench)
104{
105    static auto ds{BuildDatasets<Txid>()};
106    bench.epochs(1).epochIterations(1).batch(iterations).run([&] {
107        size_t sum{0};
108        for (const auto& s : ds) {
109            for (const auto& q : s.queries) {
110                sum += s.sorted_set.contains(q);
111            }
112        }
113        ankerl::nanobench::doNotOptimizeAway(sum);
114        Assert(sum == iterations * hits_count);
115    });
116}
117
118static void Txid_VectorBinarySearch(benchmark::Bench& bench)
119{
120    static auto ds{BuildDatasets<Txid>()};
121    bench.epochs(1).epochIterations(1).batch(iterations).run([&] {
122        size_t sum{0};
123        for (const auto& s : ds) {
124            for (const auto& q : s.queries) {
125                sum += std::binary_search(s.vec_sorted.begin(), s.vec_sorted.end(), q);
126            }
127        }
128        ankerl::nanobench::doNotOptimizeAway(sum);
129        Assert(sum == iterations * hits_count);
130    });
131}
132
133static void Txid_VectorLinearScan(benchmark::Bench& bench)
134{
135    static auto ds{BuildDatasets<Txid>()};
136    const auto contains_linear{[](const std::vector<Txid>& v, const Txid& x) noexcept {
137        return std::ranges::find(v, x) != v.end();
138    }};
139    bench.epochs(1).epochIterations(1).batch(iterations).run([&] {
140        size_t sum{0};
141        for (const auto& s : ds) {
142            for (const auto& q : s.queries) {
143                sum += contains_linear(s.vec_unsorted, q);
144            }
145        }
146        ankerl::nanobench::doNotOptimizeAway(sum);
147        Assert(sum == iterations * hits_count);
148    });
149}
150
151static void Txid_UnorderedSalted_shortid(benchmark::Bench& bench)
152{
153    static auto ds{BuildDatasets<uint64_t>()};
154    bench.epochs(1).epochIterations(1).batch(iterations).run([&] {
155        size_t sum{0};
156        for (const auto& s : ds) {
157            for (const auto& q : s.queries) {
158                sum += s.unsorted_set.contains(q);
159            }
160        }
161        ankerl::nanobench::doNotOptimizeAway(sum);
162        Assert(sum == iterations * hits_count);
163    });
164}
165
166static void Txid_SetOrdered_shortid(benchmark::Bench& bench)
167{
168    static auto ds{BuildDatasets<uint64_t>()};
169    bench.epochs(1).epochIterations(1).batch(iterations).run([&] {
170        size_t sum{0};
171        for (const auto& s : ds) {
172            for (const auto& q : s.queries) {
173                sum += s.sorted_set.contains(q);
174            }
175        }
176        ankerl::nanobench::doNotOptimizeAway(sum);
177        Assert(sum == iterations * hits_count);
178    });
179}
180
181static void Txid_VectorBinarySearch_shortid(benchmark::Bench& bench)
182{
183    static auto ds{BuildDatasets<uint64_t>()};
184    bench.epochs(1).epochIterations(1).batch(iterations).run([&] {
185        size_t sum{0};
186        for (const auto& s : ds) {
187            for (const auto& q : s.queries) {
188                sum += std::ranges::binary_search(s.vec_sorted, q);
189            }
190        }
191        ankerl::nanobench::doNotOptimizeAway(sum);
192        Assert(sum == iterations * hits_count);
193    });
194}
195
196static void Txid_VectorLinearScan_shortid(benchmark::Bench& bench)
197{
198    static auto ds{BuildDatasets<uint64_t>()};
199    const auto contains_linear{[](const std::vector<uint64_t>& v, uint64_t x) noexcept {
200        return std::ranges::find(v, x) != v.end();
201    }};
202    bench.epochs(1).epochIterations(1).batch(iterations).run([&] {
203        size_t sum{0};
204        for (const auto& s : ds) {
205            for (const auto& q : s.queries) {
206                sum += contains_linear(s.vec_unsorted, q);
207            }
208        }
209        ankerl::nanobench::doNotOptimizeAway(sum);
210        Assert(sum == iterations * hits_count);
211    });
212}
213
214static void InputFetcher_SortedVectorBenchmark(benchmark::Bench& bench)
215{
216    CBlock block;
217    DataStream{benchmark::data::block413567} >> TX_WITH_WITNESS(block);
218
219    std::vector<Txid> v{};
220    v.reserve(block.vtx.size());
221    bench.run([&] {
222        for (const auto& tx : block.vtx) {
223            v.emplace_back(tx->GetHash());
224        }
225        std::sort(v.begin(), v.end());
226        ankerl::nanobench::doNotOptimizeAway(v);
227        v.clear();
228    });
229}
230
231static void InputFetcher_UnorderedSetBenchmark(benchmark::Bench& bench)
232{
233    CBlock block;
234    DataStream{benchmark::data::block413567} >> TX_WITH_WITNESS(block);
235
236    std::unordered_set<Txid, SaltedTxidHasher> u{};
237    u.reserve(block.vtx.size());
238    bench.run([&] {
239        for (const auto& tx : block.vtx) {
240            u.emplace(tx->GetHash());
241        }
242        ankerl::nanobench::doNotOptimizeAway(u);
243        u.clear();
244    });
245}
246
247static void InputFetcher_SetBenchmark(benchmark::Bench& bench)
248{
249    CBlock block;
250    DataStream{benchmark::data::block413567} >> TX_WITH_WITNESS(block);
251
252    std::set<Txid> s{};
253    bench.run([&] {
254        for (const auto& tx : block.vtx) {
255            s.insert(tx->GetHash());
256        }
257        ankerl::nanobench::doNotOptimizeAway(s);
258        s.clear();
259    });
260}
261
262static void InputFetcher_SortedVectorBenchmark_shortid(benchmark::Bench& bench)
263{
264    CBlock block;
265    DataStream{benchmark::data::block413567} >> TX_WITH_WITNESS(block);
266
267    std::vector<uint64_t> v{};
268    v.reserve(block.vtx.size());
269    bench.run([&] {
270        for (const auto& tx : block.vtx) {
271            v.emplace_back(GetShortID(tx->GetHash()));
272        }
273        std::ranges::sort(v);
274        ankerl::nanobench::doNotOptimizeAway(v);
275        v.clear();
276    });
277}
278
279static void InputFetcher_UnorderedSetBenchmark_shortid(benchmark::Bench& bench)
280{
281    CBlock block;
282    DataStream{benchmark::data::block413567} >> TX_WITH_WITNESS(block);
283
284    std::unordered_set<uint64_t> u{};
285    u.reserve(block.vtx.size());
286    bench.run([&] {
287        for (const auto& tx : block.vtx) {
288            u.emplace(GetShortID(tx->GetHash()));
289        }
290        ankerl::nanobench::doNotOptimizeAway(u);
291        u.clear();
292    });
293}
294
295static void InputFetcher_SetBenchmark_shortid(benchmark::Bench& bench)
296{
297    CBlock block;
298    DataStream{benchmark::data::block413567} >> TX_WITH_WITNESS(block);
299
300    std::set<uint64_t> s{};
301    bench.run([&] {
302        for (const auto& tx : block.vtx) {
303            s.insert(GetShortID(tx->GetHash()));
304        }
305        ankerl::nanobench::doNotOptimizeAway(s);
306        s.clear();
307    });
308}
309
310BENCHMARK(InputFetcher_SortedVectorBenchmark, benchmark::PriorityLevel::LOW);
311BENCHMARK(InputFetcher_UnorderedSetBenchmark, benchmark::PriorityLevel::LOW);
312BENCHMARK(InputFetcher_SetBenchmark, benchmark::PriorityLevel::LOW);
313
314BENCHMARK(InputFetcher_SortedVectorBenchmark_shortid, benchmark::PriorityLevel::LOW);
315BENCHMARK(InputFetcher_UnorderedSetBenchmark_shortid, benchmark::PriorityLevel::LOW);
316BENCHMARK(InputFetcher_SetBenchmark_shortid, benchmark::PriorityLevel::LOW);
317
318BENCHMARK(Txid_UnorderedSalted, benchmark::PriorityLevel::LOW);
319BENCHMARK(Txid_SetOrdered, benchmark::PriorityLevel::LOW);
320BENCHMARK(Txid_VectorBinarySearch, benchmark::PriorityLevel::LOW);
321BENCHMARK(Txid_VectorLinearScan, benchmark::PriorityLevel::LOW);
322
323BENCHMARK(Txid_UnorderedSalted_shortid, benchmark::PriorityLevel::LOW);
324BENCHMARK(Txid_SetOrdered_shortid, benchmark::PriorityLevel::LOW);
325BENCHMARK(Txid_VectorBinarySearch_shortid, benchmark::PriorityLevel::LOW);
326BENCHMARK(Txid_VectorLinearScan_shortid, benchmark::PriorityLevel::LOW);

The new measurements indicate that the single-threaded preparation is 4x faster with a sorted vector, while multithreaded search is more than 2x faster with small ids compared to the current approach. Worth investigating further, I’d say :)

in src/coins.cpp:176 in aeec2e421d outdated

172@@ -173,6 +173,11 @@ bool CCoinsViewCache::HaveCoinInCache(const COutPoint &outpoint) const {
173     return (it != cacheCoins.end() && !it->second.coin.IsSpent());
174 }
175 
176+std::optional<Coin> CCoinsViewCache::GetPossiblySpentCoinFromCache(const COutPoint &outpoint) const {

l0rinc commented at 4:40 pm on November 2, 2025:

This seems like a code smell that we should pay attention to. I have already brought this up a year ago in #30673 (review), seems it keeps biting us. Instead of adding a new method that basically does the same, can we keep GetCoin to simply return the coin and move the spentness checks to the call sites? I understand that it would likely require another PR, but it would make this one cleaner - I don’t like that we need a workaround for a scenario that isn’t out of the ordinary.

andrewtoth commented at 9:52 pm on November 2, 2025:

This is a consequence of skipping the main cache and writing directly to the temp cache.

I don’t think we can modify GetCoin to not return spent coins, it’s part of the definition Retrieve the Coin (unspent transaction output) for a given outpoint.. This also goes backwards from your change #30849 where you added TODOs to not return spent coins where our test GetCoins do. I don’t think a PR to return spent coins from GetCoin would get enough support to be merged.

If GetCoin could return spent coins, we would also not be able to use it until we call HaveCoinInCache. This is because even though GetCoin is const, it modifies cacheCoins internally (which has the mutable modifier for this purpose). But, HaveCoinInCache also returns false if the coin is spent, so we would need to modify that. So, we would have 2 methods change and we would also then have to do 2 lookups. This method is very simple and very much defined for this special purpose, similar to EmplaceCoinInCacheDANGER.

I’m open to other suggestions, but modifying GetCoin (and then HaveCoinInCache) doesn’t seem like the way.

in src/inputfetcher.h:132 in aeec2e421d outdated

127+public:
128+    explicit InputFetcher(size_t worker_thread_count) noexcept
129+        : m_barrier{static_cast<int32_t>(worker_thread_count + 1)}
130+    {
131+        for (size_t n{0}; n < worker_thread_count; ++n) {
132+            m_worker_threads.emplace_back([this, n]() {

l0rinc commented at 4:54 pm on November 2, 2025:

nit:

0            m_worker_threads.emplace_back([this, n] {

andrewtoth commented at 11:33 pm on November 2, 2025:

Done.

in src/inputfetcher.h:128 in aeec2e421d outdated

123+            m_barrier.arrive_and_wait();
124+        }
125+    }
126+
127+public:
128+    explicit InputFetcher(size_t worker_thread_count) noexcept

l0rinc commented at 4:56 pm on November 2, 2025:

0    explicit InputFetcher(int32_t worker_thread_count) noexcept : m_barrier{(worker_thread_count + 1)}

this would remove a few static casts

andrewtoth commented at 11:33 pm on November 2, 2025:

Done.

in src/inputfetcher.h:62 in aeec2e421d outdated

57+        const COutPoint& outpoint;
58+        //! The coin that workers will fetch and main thread will insert into cache.
59+        Coin coin{};
60+
61+        Input(Input&& other) noexcept : outpoint{other.outpoint} {} // Only moved in setup for reallocation.
62+        Input(const COutPoint& o LIFETIMEBOUND) noexcept : outpoint{o} {}

l0rinc commented at 4:58 pm on November 2, 2025:

nit:

0        explicit Input(const COutPoint& o LIFETIMEBOUND) noexcept : outpoint{o} {}

andrewtoth commented at 11:33 pm on November 2, 2025:

Done.

in src/inputfetcher.h:43 in aeec2e421d outdated

38+ */
39+class InputFetcher
40+{
41+private:
42+    //! The latest input being fetched. Workers atomically increment this when fetching.
43+    alignas(64) std::atomic_size_t m_input_head{0};

l0rinc commented at 5:01 pm on November 2, 2025:

is the alignas a false sharing guard? Does it have a measurable effect?

andrewtoth commented at 9:54 pm on November 2, 2025:

It was for false sharing. I don’t think it has a measurable effect, but it might on some systems? I think it’s harmless to keep, since there’s only one InputFetcher.

andrewtoth commented at 11:33 pm on November 2, 2025:

Removed it.

in src/inputfetcher.h:182 in aeec2e421d outdated

184+        m_cache = nullptr;
185+    }
186+
187+    ~InputFetcher()
188+    {
189+        m_request_stop = true;

l0rinc commented at 5:05 pm on November 2, 2025:

I don’t like that we need a separate field for this only, but I guess without std::jthread we have to do this manually.

Since this field isn’t checked on every iteration, only once per thread per job as far as I can tell, we could repurpose m_input_head and use it as

0alignas(64) std::atomic_int32_t m_input_head{0};
1...
2if (m_input_head.load(std::memory_order_acquire) < 0) [[unlikely]] {
3...
4m_input_head.store(-1, std::memory_order_relaxed);

andrewtoth commented at 9:55 pm on November 2, 2025:

I don’t think that’s actually preferable. The current way is more readable IMO.

in src/inputfetcher.h:190 in aeec2e421d outdated

185+    }
186+
187+    ~InputFetcher()
188+    {
189+        m_request_stop = true;
190+        m_barrier.arrive_and_wait();

l0rinc commented at 5:10 pm on November 2, 2025:

not certain, but I think this should likely be:

0        m_barrier.arrive_and_drop();

andrewtoth commented at 11:33 pm on November 2, 2025:

Done.

in src/inputfetcher.h:120 in aeec2e421d outdated

115+                } catch (const std::runtime_error& e) {
116+                    LogPrintLevel(BCLog::VALIDATION, BCLog::Level::Warning, "InputFetcher failed to fetch input: %s.\n", e.what());
117+                }
118+                // Input missing or spent. This block will fail validation.
119+                // Skip remaining inputs.
120+                m_input_head.store(m_inputs.size(), std::memory_order_relaxed);

l0rinc commented at 5:13 pm on November 2, 2025:

I understand that this poison pill broadcast is needed to stop execution when errors occur - but I’m not sure we should care here about fetching error, I think we should just try to continue if that makes the code simpler

andrewtoth commented at 10:30 pm on November 2, 2025:

In ConnectBlock if we encounter a missing input we abort validation immediately. It is wasted work to continue. Why should we do it here?

l0rinc commented at 7:44 am on November 3, 2025:

I don’t think we should abort, it’s not the fetcher’s job to validate. If we didn’t, we could even use short ids for the intra-block spends (64 bit likely won’t even result in a single duplicate, and when they do collide we would just do a db check)

andrewtoth commented at 1:31 pm on November 3, 2025:

Why would short ids matter here though? Isn’t that for keeping the bandwidth smaller for compact blocks?

l0rinc commented at 1:36 pm on November 3, 2025:

no, I mean, we don’t actually need 256 bits of precision here, just a probabilistic check, so taking the first 32/64 bits of the hash should suffice, since the worst case is just going to disk, so it’s not a tragedy if there are false positives, as long as in the average case checks are lot faster

andrewtoth commented at 2:25 pm on November 3, 2025:

I don’t see why aborting early would prevent us from using less precision. It’s doubtful there would be a collision, and if there were that block will just be a little slower to connect. I don’t think it would have a measurable effect though, siphashing 64 bits vs 256 here?

andrewtoth commented at 10:15 pm on November 4, 2025:

I removed the abort early logic, so we keep going if we don’t find an input. It makes the logic much simpler, but we will do some more work if we get a block mined that is double spending.

in src/inputfetcher.h:85 in aeec2e421d outdated

86+            if (m_request_stop) [[unlikely]] {
87+                return;
88+            }
89+            while (true) {
90+                const size_t i{m_input_head.fetch_add(1, std::memory_order_relaxed)};
91+                if (i >= m_inputs.size()) [[unlikely]] {

l0rinc commented at 5:47 pm on November 2, 2025:

I don’t mind the unlikely parts but we’re barely using them in the code, they have some weird side-effects when merged - which may not be the case here, so I’m fine wither way, if nothing else, it documents the usage

in src/inputfetcher.h:33 in aeec2e421d outdated

28+ * into the ephemeral cache used in ConnectBlock.
29+ *
30+ * It spawns a fixed set of worker threads that fetch Coins for each input
31+ * in a block. The Coin is moved into the Input struct and then the status is
32+ * atomically updated to READY. The main thread spin loops on the status field
33+ * until it is READY and then inserts it into the temporary cache.

l0rinc commented at 5:49 pm on November 2, 2025:

I’m surprised synchronized insertion is faster than collecting them in a lock-free way - guess it’s all the copying, but we should be able to avoid that, I don’t like that we’re locking again

andrewtoth commented at 10:00 pm on November 2, 2025:

There are no locks… This is a lock free implementation. It is synchronized with atomics per input, and the threads are work stealing via the m_input_head. Or do you mean something else?

l0rinc commented at 7:46 am on November 3, 2025:

atomics work via CAS, they need to retry in case of high contention. the previous solution didn’t have contention, the threads all knew beforehand what to work on.

andrewtoth commented at 2:20 pm on November 3, 2025:

Some contention on shared resources is unavoidable. The threads need to synchronize on atomics. There are no locks in this implementation that all other threads need to wait on.

The main thread may have to wait here if there is a slow fetch, but it would be reading uncontested memory right up until one worker thread flips this bit. None of the other threads are trying to write to this same bit, so there is no contention between the main thread and any other worker threads.

andrewtoth commented at 10:16 pm on November 4, 2025:

I’ve updated this to be an atomic_bool instead of this enum. So, since there is only a false value that is set to true, we have the main thread call input.ready.wait(false, std::memory_order_acquire);. This way we don’t spin.

in src/inputfetcher.h:50 in aeec2e421d outdated

45+    //! The inputs of the block which is being fetched.
46+    struct Input {
47+        enum class Status : uint8_t {
48+            WAITING, // The coin has not been fetched yet
49+            READY, // The coin has been fetched and is ready to be inserted into the cache
50+            FAILED, // The coin failed to be fetched

l0rinc commented at 5:50 pm on November 2, 2025:

as mentioned before, is FAILED an important state for the fetcher? Why not just debug/warn log and continue?

andrewtoth commented at 10:03 pm on November 2, 2025:

We can’t have the main thread insert a failed fetch. The coin will not be updated. I suppose the main thread could check if the coin is unspent. We want to exit ASAP as well if we can’t find an input, so we need to signal the main thread that they should exit the loop.

l0rinc commented at 7:47 am on November 3, 2025:

that’s my point, I don’t think we should validate at all, it would simplify the code if we didn’t

andrewtoth commented at 10:19 pm on November 4, 2025:

Done, this Status enum has been replaced. It is now just an atomic_bool ready.

in src/inputfetcher.h:104 in aeec2e421d outdated

 99+                    continue;
100+                }
101+                try {
102+                    if (auto coin{m_cache->GetPossiblySpentCoinFromCache(input.outpoint)}) {
103+                        input.coin = std::move(*coin);
104+                        if (!input.coin.IsSpent()) [[likely]] { // Coin from cache could be spent

l0rinc commented at 5:57 pm on November 2, 2025:

hmm, code states it’s likely, but the new tests are passing with:

0if (!input.coin.IsSpent()) {
1    throw "";
2}

andrewtoth commented at 10:42 pm on November 2, 2025:

Right, didn’t cover this new happy path in tests. Unlikely case is covered though. Will add. Thanks.

in src/test/inputfetcher_tests.cpp:24 in e3045d2237 outdated

20+#include <string>
21+#include <unordered_set>
22+
23+BOOST_AUTO_TEST_SUITE(inputfetcher_tests)
24+
25+struct InputFetcherTest : BasicTestingSetup {

l0rinc commented at 5:59 pm on November 2, 2025:

I think the test belongs with the implementation, I need it to be able to review the first commit (otherwise it’s just dead code, with the tests it has at least a test user)

andrewtoth commented at 11:34 pm on November 2, 2025:

Done.

in src/test/inputfetcher_tests.cpp:20 in e3045d2237 outdated

16+
17+#include <cstdint>
18+#include <memory>
19+#include <stdexcept>
20+#include <string>
21+#include <unordered_set>

l0rinc commented at 6:02 pm on November 2, 2025:

nit: some of these seem unused:

0#include <memory>
1#include <stdexcept>
2#include <unordered_set>

andrewtoth commented at 11:04 pm on November 2, 2025:

<cstdint> we need because we use int32_t.

in src/test/inputfetcher_tests.cpp:85 in 62868c8846

80+                db.EmplaceCoinInternalDANGER(std::move(outpoint), std::move(coin));
81+            }
82+        }
83+
84+        CCoinsViewCache main_cache(&db);
85+        CCoinsViewCache cache(&cache);

l0rinc commented at 8:03 pm on November 2, 2025:

hmm, how does this even compile?

andrewtoth commented at 11:34 pm on November 2, 2025:

Fixed. But, it wouldn’t affect the correctness of the test.

l0rinc commented at 7:49 am on November 3, 2025:

how so? Can we assert the behavior of the main cache as well so that the previous version doesn’t pass?

andrewtoth commented at 2:43 pm on November 5, 2025:

Can we assert the behavior of the main cache as well so that the previous version doesn’t pass?

Neither the temp cache or the main cache touch their backing cache in the input fetcher. So, we can’t assert a failure if the backing cache is something else. We can assert that the backing view is not touched during FetchInputs, which is done in the fuzz harness.

in src/test/fuzz/inputfetcher.cpp:43 in 62868c8846 outdated

38+struct NoAccessCoinsView : CCoinsView
39+{
40+    std::optional<Coin> GetCoin(const COutPoint&) const override { abort(); }
41+};
42+
43+FUZZ_TARGET(inputfetcher)

l0rinc commented at 8:24 pm on November 2, 2025:

I don’t have a good fuzzer locally - does this cover all the new code?

andrewtoth commented at 11:05 pm on November 2, 2025:

Yes, I need to fuzz it though myself. The CI is fuzzing it a little bit.

in src/inputfetcher.h:105 in 62868c8846

100+                }
101+                try {
102+                    if (auto coin{m_cache->GetPossiblySpentCoinFromCache(input.outpoint)}) {
103+                        input.coin = std::move(*coin);
104+                        if (!input.coin.IsSpent()) [[likely]] { // Coin from cache could be spent
105+                            // We need release here, so setting coin 2 lines above happens before the main thread loads.

l0rinc commented at 8:28 pm on November 2, 2025:

The comments indicate that you also don’t think this code is very intuitive - I’m also a bit lost here, can we simplify this somehow? I don’t even understand what “release” means here or why both branches result in Status::READY (can we unify them) or what happens if the first internal if isn’t fulfilled, or the outer one or the else, etc. The branching + continue + try/catch doesn’t help

andrewtoth commented at 11:35 pm on November 2, 2025:

Rewrote this part to make it more clear. I was trying to be clever by avoiding some extra checks, but they are probably meaningless and clarity is better here.

l0rinc changes_requested

l0rinc commented at 8:31 pm on November 2, 2025: contributor

I like how we’re progressing here! I think we need a few more things and have to try out a few alternatives (I haven’t given up on sorting yet, especially now with bigger dbcache) and want to see how this combines with the other optimizations (threadpool, SipHash13, map hash caching, etc), but I’m definitely getting closer and closer to an ACK :D

I’m testing full IBD locally on my servers, but those are always slower than reindex-chainstes since the nodes can’t send the blocks fast enough - I don’t yet have a seeding node yet, but working on it.

“It simply inserts inputs into the temporary cache, which must be fetched before a transaction is validated anyways.” beautiful, I think some of this should be added to the commit messages as well. commit message nit: “coins: add InputFetcher” (commit message content and formatting is a bit sloppy)

I’d merge fuzz: add inputfetcher fuzz harness and tests: add inputfetcher tests and coins: add inputfetcher since the clients are needed for the review of InputFetcher, otherwise it’s just dead code added…

  0diff --git a/src/bench/CMakeLists.txt b/src/bench/CMakeLists.txt
  1index 9d03f075a7..dcb6281699 100644
  2--- a/src/bench/CMakeLists.txt
  3+++ b/src/bench/CMakeLists.txt
  4@@ -52,6 +52,7 @@ add_executable(bench_bitcoin
  5   streams_findbyte.cpp
  6   strencodings.cpp
  7   txgraph.cpp
  8+  txid_membership.cpp
  9   txorphanage.cpp
 10   util_time.cpp
 11   verify_script.cpp
 12diff --git a/src/bench/inputfetcher.cpp b/src/bench/inputfetcher.cpp
 13index c10fcc5b5e..c1660c3ccf 100644
 14--- a/src/bench/inputfetcher.cpp
 15+++ b/src/bench/inputfetcher.cpp
 16@@ -12,20 +12,18 @@
 17 #include <streams.h>
 18 #include <util/time.h>
 19 
 20-static constexpr auto DELAY{2ms};
 21-
 22 //! Simulates a DB by adding a delay when calling GetCoin
 23 struct DelayedCoinsView : CCoinsView
 24 {
 25     std::optional<Coin> GetCoin(const COutPoint&) const override
 26     {
 27-        UninterruptibleSleep(DELAY);
 28+        UninterruptibleSleep(2ms);
 29         Coin coin{};
 30         coin.out.nValue = 1;
 31         return coin;
 32     }
 33 
 34-    bool BatchWrite(CoinsViewCacheCursor&, const uint256&) override { return true; }
 35+    bool BatchWrite(CoinsViewCacheCursor&, const uint256&) override { throw std::logic_error{"unused"}; }
 36 };
 37 
 38 static void InputFetcherBenchmark(benchmark::Bench& bench)
 39@@ -39,11 +37,13 @@ static void InputFetcherBenchmark(benchmark::Bench& bench)
 40     // The main thread should be counted to prevent thread oversubscription, and
 41     // to decrease the variance of benchmark results.
 42     const auto worker_threads_num{GetNumCores() - 1};
 43-    InputFetcher fetcher{static_cast<size_t>(worker_threads_num)};
 44+    InputFetcher fetcher{worker_threads_num};
 45 
 46     bench.run([&] {
 47         CCoinsViewCache temp_cache(&main_cache);
 48         fetcher.FetchInputs(temp_cache, main_cache, db, block);
 49+        ankerl::nanobench::doNotOptimizeAway(&temp_cache);
 50+        Assert(temp_cache.GetCacheSize() == 4599);
 51     });
 52 }
 53 
 54diff --git a/src/bench/txid_membership.cpp b/src/bench/txid_membership.cpp
 55new file mode 100644
 56index 0000000000..e646bb2a4a
 57--- /dev/null
 58+++ b/src/bench/txid_membership.cpp
 59@@ -0,0 +1,134 @@
 60+// Copyright (c) 2022-present The Bitcoin Core developers
 61+// Distributed under the MIT software license, see the accompanying
 62+// file COPYING or https://www.opensource.org/licenses/mit-license.php.
 63+
 64+#include <bench/bench.h>
 65+#include <bench/nanobench.h>
 66+#include <primitives/transaction_identifier.h>
 67+#include <random.h>
 68+#include <util/check.h>
 69+#include <util/hasher.h>
 70+
 71+#include <algorithm>
 72+#include <ranges>
 73+#include <set>
 74+#include <unordered_set>
 75+#include <vector>
 76+
 77+namespace {
 78+
 79+constexpr size_t iterations{100}; // since the inputs of the benchmarks are mutated by sorting, we can't rerun the benchmarks
 80+constexpr size_t hits_count{275}; // assuming ~5% of blocks contain internal spends
 81+constexpr size_t tx_count{5500};
 82+
 83+struct Dataset {
 84+    std::set<Txid> sorted_set;
 85+    std::unordered_set<Txid, SaltedTxidHasher> unsorted_set;
 86+    std::vector<Txid> vec_sorted;
 87+    std::vector<Txid> vec_unsorted;
 88+
 89+    std::vector<Txid> queries;
 90+};
 91+
 92+std::vector<Dataset> BuildDatasets()
 93+{
 94+    FastRandomContext rng(/*fDeterministic=*/true);
 95+
 96+    std::vector<Dataset> datasets;
 97+    datasets.reserve(iterations);
 98+
 99+    for (size_t d{0}; d < iterations; ++d) {
100+        Dataset ds;
101+        ds.queries.reserve(tx_count);
102+        ds.unsorted_set.reserve(tx_count);
103+        ds.vec_sorted.reserve(tx_count);
104+        ds.vec_unsorted.reserve(tx_count);
105+
106+        for (size_t i{0}; i < tx_count; ++i) {
107+            Txid t{Txid::FromUint256(rng.rand256())};
108+            ds.sorted_set.emplace(t);
109+            ds.unsorted_set.emplace(t);
110+            ds.vec_sorted.emplace_back(t);
111+            ds.vec_unsorted.emplace_back(t);
112+
113+            ds.queries.emplace_back(i < hits_count ? t : Txid::FromUint256(rng.rand256()));
114+        }
115+
116+        std::ranges::shuffle(ds.queries, rng);
117+        std::ranges::shuffle(ds.vec_unsorted, rng);
118+        std::sort(ds.vec_sorted.begin(), ds.vec_sorted.end());
119+
120+        datasets.emplace_back(std::move(ds));
121+    }
122+    return datasets;
123+}
124+
125+} // namespace
126+
127+static void Txid_UnorderedSalted(benchmark::Bench& bench)
128+{
129+    static auto ds{BuildDatasets()};
130+    bench.epochs(1).epochIterations(1).batch(iterations).run([&] {
131+        size_t sum{0};
132+        for (const auto& s : ds) {
133+            for (const auto& q : s.queries) {
134+                sum += s.unsorted_set.contains(q);
135+            }
136+        }
137+        ankerl::nanobench::doNotOptimizeAway(sum);
138+        Assert(sum == iterations * hits_count);
139+    });
140+}
141+
142+static void Txid_SetOrdered(benchmark::Bench& bench)
143+{
144+    static auto ds{BuildDatasets()};
145+    bench.epochs(1).epochIterations(1).batch(iterations).run([&] {
146+        size_t sum{0};
147+        for (const auto& s : ds) {
148+            for (const auto& q : s.queries) {
149+                sum += s.sorted_set.contains(q);
150+            }
151+        }
152+        ankerl::nanobench::doNotOptimizeAway(sum);
153+        Assert(sum == iterations * hits_count);
154+    });
155+}
156+
157+static void Txid_VectorBinarySearch(benchmark::Bench& bench)
158+{
159+    static auto ds{BuildDatasets()};
160+    bench.epochs(1).epochIterations(1).batch(iterations).run([&] {
161+        size_t sum{0};
162+        for (const auto& s : ds) {
163+            for (const auto& q : s.queries) {
164+                sum += std::binary_search(s.vec_sorted.begin(), s.vec_sorted.end(), q);
165+            }
166+        }
167+        ankerl::nanobench::doNotOptimizeAway(sum);
168+        Assert(sum == iterations * hits_count);
169+    });
170+}
171+
172+static void Txid_VectorLinearScan(benchmark::Bench& bench)
173+{
174+    static auto ds{BuildDatasets()};
175+    const auto contains_linear{[](const std::vector<Txid>& v, const Txid& x) noexcept {
176+        return std::ranges::find(v, x) != v.end();
177+    }};
178+    bench.epochs(1).epochIterations(1).batch(iterations).run([&] {
179+        size_t sum{0};
180+        for (const auto& s : ds) {
181+            for (const auto& q : s.queries) {
182+                sum += contains_linear(s.vec_unsorted, q);
183+            }
184+        }
185+        ankerl::nanobench::doNotOptimizeAway(sum);
186+        Assert(sum == iterations * hits_count);
187+    });
188+}
189+
190+BENCHMARK(Txid_UnorderedSalted, benchmark::PriorityLevel::LOW);
191+BENCHMARK(Txid_SetOrdered, benchmark::PriorityLevel::LOW);
192+BENCHMARK(Txid_VectorBinarySearch, benchmark::PriorityLevel::LOW);
193+BENCHMARK(Txid_VectorLinearScan, benchmark::PriorityLevel::LOW);
194diff --git a/src/coins.h b/src/coins.h
195index 1b3dcfc309..95c3bfe2f5 100644
196--- a/src/coins.h
197+++ b/src/coins.h
198@@ -486,7 +486,8 @@ public:
199      * Reserve enough space in the cache so the underlying unordered_map will
200      * not have to rehash unless capacity is exceeded.
201      */
202-    void Reserve(size_t capacity) {
203+    void Reserve(size_t capacity)
204+    {
205         cacheCoins.reserve(capacity);
206     }
207 
208diff --git a/src/inputfetcher.h b/src/inputfetcher.h
209index cd9eaed6ea..d15c4874cc 100644
210--- a/src/inputfetcher.h
211+++ b/src/inputfetcher.h
212@@ -17,6 +17,7 @@
213 #include <atomic>
214 #include <barrier>
215 #include <cstdint>
216+#include <set>
217 #include <stdexcept>
218 #include <thread>
219 #include <unordered_set>
220@@ -38,9 +39,8 @@
221  */
222 class InputFetcher
223 {
224-private:
225     //! The latest input being fetched. Workers atomically increment this when fetching.
226-    alignas(64) std::atomic_size_t m_input_head{0};
227+    alignas(64) std::atomic_int32_t m_input_head{0};
228 
229     //! The inputs of the block which is being fetched.
230     struct Input {
231@@ -59,7 +59,7 @@ private:
232         Coin coin{};
233 
234         Input(Input&& other) noexcept : outpoint{other.outpoint} {} // Only moved in setup for reallocation.
235-        Input(const COutPoint& o LIFETIMEBOUND) noexcept : outpoint{o} {}
236+        explicit Input(const COutPoint& o LIFETIMEBOUND) noexcept : outpoint{o} {}
237     };
238     std::vector<Input> m_inputs{};
239 
240@@ -68,7 +68,7 @@ private:
241      * Used to filter out inputs that are created and spent in the same block,
242      * since they will not be in the db or the cache.
243      */
244-    std::unordered_set<Txid, SaltedTxidHasher> m_txids{};
245+    std::set<Txid> m_txids{};
246 
247     //! DB coins view to fetch from.
248     const CCoinsView* m_db{nullptr};
249@@ -77,18 +77,15 @@ private:
250 
251     std::vector<std::thread> m_worker_threads{};
252     std::barrier<> m_barrier;
253-    bool m_request_stop{false};
254 
255     void WorkLoop() noexcept
256     {
257         while (true) {
258             m_barrier.arrive_and_wait();
259-            if (m_request_stop) [[unlikely]] {
260-                return;
261-            }
262+            if (m_input_head.load(std::memory_order_relaxed) < 0) [[unlikely]] return;
263             while (true) {
264-                const size_t i{m_input_head.fetch_add(1, std::memory_order_relaxed)};
265-                if (i >= m_inputs.size()) [[unlikely]] {
266+                const auto i{m_input_head.fetch_add(1, std::memory_order_relaxed)};
267+                if (i >= int32_t(m_inputs.size())) [[unlikely]] {
268                     break;
269                 }
270                 auto& input{m_inputs[i]};
271@@ -125,11 +122,10 @@ private:
272     }
273 
274 public:
275-    explicit InputFetcher(size_t worker_thread_count) noexcept
276-        : m_barrier{static_cast<int32_t>(worker_thread_count + 1)}
277+    explicit InputFetcher(int32_t worker_thread_count) noexcept : m_barrier{(worker_thread_count + 1)}
278     {
279-        for (size_t n{0}; n < worker_thread_count; ++n) {
280-            m_worker_threads.emplace_back([this, n]() {
281+        for (int32_t n{0}; n < worker_thread_count; ++n) {
282+            m_worker_threads.emplace_back([this, n] {
283                 util::ThreadRename(strprintf("inputfetch.%i", n));
284                 WorkLoop();
285             });
286@@ -145,6 +141,8 @@ public:
287 
288         // Loop through the inputs of the block and set them in the queue.
289         // Construct the set of txids to filter, and count the outputs to reserve for temp_cache.
290+        //m_txids.reserve(block.vtx.size());
291+        m_inputs.reserve(2 * block.vtx.size()); // rough guess
292         auto outputs_count{block.vtx[0]->vout.size()};
293         for (size_t i{1}; i < block.vtx.size(); ++i) {
294             const auto& tx{block.vtx[i]};
295@@ -162,7 +160,7 @@ public:
296         m_barrier.arrive_and_wait();
297 
298         // Insert fetched coins into the temp_cache as they are set to READY.
299-        temp_cache.Reserve(m_inputs.size() + outputs_count);
300+        temp_cache.Reserve(temp_cache.GetCacheSize() + m_inputs.size() + outputs_count);
301         for (auto& input : m_inputs) {
302             auto status{input.status.load(std::memory_order_acquire)};
303             while (status == Input::Status::WAITING) {
304@@ -175,6 +173,7 @@ public:
305                 break;
306             }
307         }
308+        Assert(temp_cache.GetCacheSize() <= temp_cache.GetCacheSize() + m_inputs.size() + outputs_count); // TODO remove
309 
310         m_barrier.arrive_and_wait();
311         // Cleanup after all worker threads have exited the inner loop.
312@@ -186,11 +185,9 @@ public:
313 
314     ~InputFetcher()
315     {
316-        m_request_stop = true;
317-        m_barrier.arrive_and_wait();
318-        for (auto& t : m_worker_threads) {
319-            t.join();
320-        }
321+        m_input_head.store(-1, std::memory_order_relaxed);
322+        m_barrier.arrive_and_drop();
323+        for (auto& t : m_worker_threads) t.join();
324     }
325 };
326 
327diff --git a/src/test/inputfetcher_tests.cpp b/src/test/inputfetcher_tests.cpp
328index b92a15d291..3d085bc843 100644
329--- a/src/test/inputfetcher_tests.cpp
330+++ b/src/test/inputfetcher_tests.cpp
331@@ -14,10 +14,8 @@
332 
333 #include <boost/test/unit_test.hpp>
334 
335-#include <cstdint>
336 #include <memory>
337 #include <stdexcept>
338-#include <string>
339 #include <unordered_set>
340 
341 BOOST_AUTO_TEST_SUITE(inputfetcher_tests)
342@@ -82,7 +80,7 @@ BOOST_FIXTURE_TEST_CASE(fetch_inputs, InputFetcherTest)
343         }
344 
345         CCoinsViewCache main_cache(&db);
346-        CCoinsViewCache cache(&cache);
347+        CCoinsViewCache cache(&main_cache);
348         getFetcher().FetchInputs(cache, main_cache, db, block);
349 
350         std::unordered_set<Txid, SaltedTxidHasher> txids{};
351diff --git a/src/validation.cpp b/src/validation.cpp
352index 7564b97a07..07ab71852f 100644
353--- a/src/validation.cpp
354+++ b/src/validation.cpp
355@@ -6299,7 +6299,7 @@ static ChainstateManager::Options&& Flatten(ChainstateManager::Options&& opts)
356 
357 ChainstateManager::ChainstateManager(const util::SignalInterrupt& interrupt, Options options, node::BlockManager::Options blockman_options)
358     : m_script_check_queue{/*batch_size=*/128, std::clamp(options.worker_threads_num, 0, MAX_SCRIPTCHECK_THREADS)},
359-      m_input_fetcher{std::clamp<size_t>(options.worker_threads_num, 0, MAX_SCRIPTCHECK_THREADS)},
360+      m_input_fetcher{std::clamp<int32_t>(options.worker_threads_num, 0, MAX_SCRIPTCHECK_THREADS)},
361       m_interrupt{interrupt},
362       m_options{Flatten(std::move(options))},
363       m_blockman{interrupt, std::move(blockman_options)},

andrewtoth force-pushed on Nov 2, 2025

l0rinc commented at 9:34 am on November 3, 2025: contributor

I have the IBD numbers for the i7-hdd and i9-ssd server. They’re not as glorious as our reindex-chainstate measurements, most likely since I don’t yet have a way to test IBD from extremely fast peers. But as a sanity-check I think it’s fine, we’re still bandwidth bound - which is a good problem to have.

 0COMMITS="bf07cf0adf19889727cb6bea24ebfbbfcc231a0c 45fe0c0e5beddce1c9e836ab5d97aa064069c192"; \                        
 1STOP=921129; DBCACHE=4500; \                                                                                                                                
 2CC=gcc; CXX=g++; \                                                                                                                                          
 3BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \                                                                   (echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done) && \                                              
 4(echo "" && echo "IBD | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(df -T $BASE_DIR | awk 'NR==2{print $2}') | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 0 && echo SSD || echo HDD)"; echo "") &&\
 5hyperfine \
 6  --sort command \
 7  --runs 2 \
 8  --export-json "$BASE_DIR/ibd-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \
 9  --parameter-list COMMIT ${COMMITS// /,} \
10  --prepare "killall bitcoind 2>/dev/null; rm -rf $DATA_DIR/*; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
11    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \
12    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=1 -printtoconsole=0; sleep 20" \
13  --conclude "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log && \
14             grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log" \
15  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -blocksonly -printtoconsole=0"
16
17bf07cf0adf coins: add inputfetcher
1845fe0c0e5b validation: fetch block inputs via InputFetcher before connecting
19
20Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=4500 -blocksonly -printtoconsole=0 (COMMIT = bf07cf0adf19889727cb6bea24ebfbbfcc231a0c)      
21  Time (mean ± σ):     36470.995 s ± 113.187 s    [User: 38024.880 s, System: 2035.545 s]
22  Range (min … max):   36390.960 s … 36551.030 s    2 runs
23  
24Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=4500 -blocksonly -printtoconsole=0 (COMMIT = 45fe0c0e5beddce1c9e836ab5d97aa064069c192)                                                                         
25  Time (mean ± σ):     33962.782 s ± 375.163 s    [User: 41686.176 s, System: 2832.686 s]
26  Range (min … max):   33697.502 s … 34228.062 s    2 runs
27  
28Relative speed comparison
29        1.07 ±  0.01  COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=4500 -blocksonly -printtoconsole=0 (COMMIT = bf07cf0adf19889727cb6bea24ebfbbfcc231a0c)
30        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=4500 -blocksonly -printtoconsole=0 (COMMIT = 45fe0c0e5beddce1c9e836ab5d97aa064069c192)

and

 0COMMITS="aeec2e421d2ba102d905633d474f0fb88f91a9bf 62868c8846f043477d128788eadced3e71522417"; \
 1STOP=921129; DBCACHE=450; \
 2CC=gcc; CXX=g++; \
 3BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
 4(echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done) && \
 5(echo "" && echo "IBD | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(df -T $BASE_DIR | awk 'NR==2{print $2}') | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 0 && echo SSD || echo HDD)"; echo "") &&\
 6hyperfine \
 7  --sort command \
 8  --runs 2 \
 9  --export-json "$BASE_DIR/ibd-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \
10  --parameter-list COMMIT ${COMMITS// /,} \
11  --prepare "killall bitcoind 2>/dev/null; rm -rf $DATA_DIR/*; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
12    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \
13    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=1 -printtoconsole=0; sleep 20" \
14  --conclude "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log && \
15             grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log" \
16  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -blocksonly -printtoconsole=0"
17
18aeec2e421d coins: add inputfetcher
1962868c8846 validation: fetch block inputs via InputFetcher before connecting
20
21Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -blocksonly -printtoconsole=0 (COMMIT = aeec2e421d2ba102d905633d474f0fb88f91a9bf)
22  Time (mean ± σ):     30907.958 s ± 1761.510 s    [User: 51503.334 s, System: 3833.881 s]
23  Range (min … max):   29662.383 s … 32153.534 s    2 runs
24 
25Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -blocksonly -printtoconsole=0 (COMMIT = 62868c8846f043477d128788eadced3e71522417)
26  Time (mean ± σ):     27504.900 s ± 2529.652 s    [User: 59336.996 s, System: 5796.247 s]
27  Range (min … max):   25716.165 s … 29293.634 s    2 runs
28 
29Relative speed comparison
30        1.12 ±  0.12  COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -blocksonly -printtoconsole=0 (COMMIT = aeec2e421d2ba102d905633d474f0fb88f91a9bf)
31        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -blocksonly -printtoconsole=0 (COMMIT = 62868c8846f043477d128788eadced3e71522417)

and

 0COMMITS="2aa510348143521a14146e41b5cf87cb3e60b29e cb0fdfdf3704d5ffe6ccc634de6fdba6b7b57a85"; \
 1STOP=921129; DBCACHE=450; \
 2CC=gcc; CXX=g++; \
 3BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
 4(echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done) && \
 5(echo "" && echo "IBD | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(df -T $BASE_DIR | awk 'NR==2{print $2}') | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 0 && echo SSD || echo HDD)"; echo "") &&\
 6hyperfine \
 7  --sort command \
 8  --runs 2 \
 9  --export-json "$BASE_DIR/ibd-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \
10  --parameter-list COMMIT ${COMMITS// /,} \
11  --prepare "killall bitcoind 2>/dev/null; rm -rf $DATA_DIR/*; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
12    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \
13    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=1 -printtoconsole=0; sleep 20" \
14  --conclude "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log && \
15             grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log" \
16  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -blocksonly -printtoconsole=0"
17
182aa5103481 validation: fetch block inputs via InputFetcher before connecting
19cb0fdfdf37 coins: add inputfetcher
20
21Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -blocksonly -printtoconsole=0 (COMMIT = 2aa510348143521a14146e41b5cf87cb3e60b29e)
22  Time (mean ± σ):     61239.351 s ± 4942.104 s    [User: 90457.852 s, System: 16836.057 s]
23  Range (min … max):   57744.756 s … 64733.946 s    2 runs
24
25Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -blocksonly -printtoconsole=0 (COMMIT = cb0fdfdf3704d5ffe6ccc634de6fdba6b7b57a85)
26  Time (mean ± σ):     66848.997 s ± 1122.176 s    [User: 88025.800 s, System: 11057.384 s]
27  Range (min … max):   66055.499 s … 67642.496 s    2 runs
28  
29Relative speed comparison
30        1.09 ±  0.09  COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -blocksonly -printtoconsole=0 (COMMIT = cb0fdfdf3704d5ffe6ccc634de6fdba6b7b57a85)
31        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -blocksonly -printtoconsole=0 (COMMIT = 2aa510348143521a14146e41b5cf87cb3e60b29e)

The reindex-chainstate cases (which we can look at as a more stable way of testing offline-IBD) show very good results even for the max-memory usecase (45 GB dbcache) - and confirm @andrewtoth’s claim that we may be able to deprecate the -dbcache argument after this since it has barely any effect after this change!

 0COMMITS="bf07cf0adf19889727cb6bea24ebfbbfcc231a0c 45fe0c0e5beddce1c9e836ab5d97aa064069c192"; \
 1STOP=921129; DBCACHE=45000; \
 2CC=gcc; CXX=g++; \
 3BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
 4(echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done) && \
 5(echo "" && echo "reindex-chainstate | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(df -T $BASE_DIR | awk 'NR==2{print $2}') | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 0 && echo SSD || echo HDD)"; echo "") &&\
 6hyperfine \
 7  --sort command \
 8  --runs 1 \
 9  --export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \
10  --parameter-list COMMIT ${COMMITS// /,} \
11  --prepare "killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
12    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \
13    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20" \
14  --conclude "killall bitcoind || true; sleep 5; grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log; \
15              cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
16  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0"
17
18bf07cf0adf coins: add inputfetcher
1945fe0c0e5b validation: fetch block inputs via InputFetcher before connecting
20 
21Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=45000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = bf07cf0adf19889727cb6bea24ebfbbfcc231a0c)
22  Time (abs ≡):        16044.026 s               [User: 23421.874 s, System: 695.027 s]
23
24Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=45000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 45fe0c0e5beddce1c9e836ab5d97aa064069c192)
25  Time (abs ≡):        15643.115 s               [User: 26237.588 s, System: 984.588 s]
26 
27Relative speed comparison
28        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=45000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 45fe0c0e5beddce1c9e836ab5d97aa064069c192)
29        1.03          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=45000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = bf07cf0adf19889727cb6bea24ebfbbfcc231a0c)

the same measurement with default memory is basically exactly the same speed as with max memory (450 MB -> 15818 seconds vs 45 GB -> 15643 seconds)

 0COMMITS="bf07cf0adf19889727cb6bea24ebfbbfcc231a0c 45fe0c0e5beddce1c9e836ab5d97aa064069c192"; STOP=921129; DBCACHE=450; CC=gcc; CXX=g++; BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; (echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done) && (echo "" && echo "reindex-chainstate | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(df -T $BASE_DIR | awk 'NR==2{print $2}') | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 0 && echo SSD || echo HDD)"; echo "") &&hyperfine   --sort command   --runs 1   --export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json"   --parameter-list COMMIT ${COMMITS// /,}   --prepare "killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
 1    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \
 2    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20"   --conclude "killall bitcoind || true; sleep 5; grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log; \
 3              cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log"   "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0"
 4
 5bf07cf0adf coins: add inputfetcher
 645fe0c0e5b validation: fetch block inputs via InputFetcher before connecting
 7 
 8Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = bf07cf0adf19889727cb6bea24ebfbbfcc231a0c)
 9  Time (abs ≡):        20500.654 s               [User: 40766.355 s, System: 2845.314 s]
10
11Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 45fe0c0e5beddce1c9e836ab5d97aa064069c192)
12  Time (abs ≡):        15818.604 s               [User: 45952.420 s, System: 4127.137 s]
13 
14Relative speed comparison
15        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 45fe0c0e5beddce1c9e836ab5d97aa064069c192)
16        1.30          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = bf07cf0adf19889727cb6bea24ebfbbfcc231a0c)

 0COMMITS="cb0fdfdf3704d5ffe6ccc634de6fdba6b7b57a85 2aa510348143521a14146e41b5cf87cb3e60b29e"; STOP=921129; DBCACHE=450;
 1 CC=gcc; CXX=g++; BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; (echo ""; for c in $COMMITS; do git fetch -q origi
 2n $c && git log -1 --pretty='%h %s' $c || exit 1; done) && (echo "" && echo "reindex-chainstate | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(unam
 3e -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(df -T $BASE_DIR | awk 'NR==2{print $2}') | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 0 && echo SSD || echo HDD)"; echo "") &&hyperfine   --sort command   --runs 1   --export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json"   --parameter-list COMMIT ${COMMITS// /,}   --prepare "killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
 4    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \
 5    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20"   --conclude "killall bitcoind || true; sleep 5; cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log; \
 6              grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log"   "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0"
 7
 8cb0fdfdf37 coins: add inputfetcher                                            
 92aa5103481 validation: fetch block inputs via InputFetcher before connecting                                                                                
10
11Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = cb0fdfdf3704d5ffe6ccc634de6fdba6b7b57a85)
12  Time (abs ≡):        43407.876 s               [User: 40230.765 s, System: 3077.358 s]
13                                                                              
14Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 2aa510348143521a14146e41b5cf87cb3e60b29e)                       
15  Time (abs ≡):        37189.669 s               [User: 45706.002 s, System: 4452.708 s]
16  
17Relative speed comparison
18        1.17          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = cb0fdfdf3704d5ffe6ccc634de6fdba6b7b57a85)
19        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 2aa510348143521a14146e41b5cf87cb3e60b29e)

in src/test/fuzz/inputfetcher.cpp:50 in 21e5e10bf3 outdated

45+    SeedRandomStateForTest(SeedRand::ZEROS);
46+    FuzzedDataProvider fuzzed_data_provider(buffer.data(), buffer.size());
47+
48+    const auto worker_threads{
49+        fuzzed_data_provider.ConsumeIntegralInRange<int32_t>(2, 4)};
50+    InputFetcher fetcher{worker_threads};

sedited commented at 11:19 am on November 3, 2025:

I’m observing a memory leak in this fuzz test similar to the one we had for the thread pool. Over there, we disabled logging and instantiate only a single pool instance: https://github.com/bitcoin/bitcoin/pull/33689/files#diff-68602d972fe2b027e3987eff0042c27f3f00fc161b7fe871bdb147571f348298R49-R58 . Maybe similar things can be done here?

l0rinc commented at 11:22 am on November 3, 2025:

Thanks for reporting, that explains why I was seeing

0bench_bitcoin(10874,0x207c84800) malloc: Failed to allocate segment from range group - out of space

and

0bitcoind(70369,0x16f5ff000) malloc: Failed to allocate segment from range group - out of space

recently (cc: @maflcko)

maflcko commented at 11:44 am on November 3, 2025:

Interesting, how would a fuzz target lead to a crash in bench or bitcoind? Did you run it in parallel?

Though, the suggestion to disable logging is correct, because while fuzzing, we probably don’t want to spend cycles on log formatting. I guess those logs only end up in the buffer which causes the memory to grow? (There should be a DEFAULT_MAX_LOG_BUFFER, so if buffering is the issue, the memory should be limited)

sedited commented at 12:23 pm on November 3, 2025:

The following patch seems to stabilize memory consumption:

 0diff --git a/src/test/fuzz/inputfetcher.cpp b/src/test/fuzz/inputfetcher.cpp
 1index cd2a0f5c68..609b8e1191 100644
 2--- a/src/test/fuzz/inputfetcher.cpp
 3+++ b/src/test/fuzz/inputfetcher.cpp
 4@@ -43 +43,10 @@ struct NoAccessCoinsView : CCoinsView
 5-FUZZ_TARGET(inputfetcher)
 6+std::optional<InputFetcher> g_fetcher{};
 7+
 8+static void setup_threadpool_test()
 9+{
10+    LogInstance().DisableLogging();
11+    g_fetcher.emplace(3);
12+}
13+
14+FUZZ_TARGET(inputfetcher, .init = setup_threadpool_test)
15@@ -48,4 +56,0 @@ FUZZ_TARGET(inputfetcher)
16-    const auto worker_threads{
17-        fuzzed_data_provider.ConsumeIntegralInRange<int32_t>(2, 4)};
18-    InputFetcher fetcher{worker_threads};
19-
20@@ -115 +120 @@ FUZZ_TARGET(inputfetcher)
21-        fetcher.FetchInputs(cache, main_cache, db, block);
22+        g_fetcher->FetchInputs(cache, main_cache, db, block);

l0rinc commented at 12:27 pm on November 3, 2025:

Are we sure we’re not just masking a real problem with the disabled logger?

sedited commented at 12:33 pm on November 3, 2025:

The logger is way less problematic in terms of its effect on the memory growing, and I find it difficult to really pin its effect. Having a global input fetcher that does not get instantiated with every fuzzer iteration has an immediate and clear effect. It is not clear to me if we are are actually leaking anything through the threads, or if creating and destroying thousands of threads per second puts too much pressure on the os (same for the threadpool).

andrewtoth commented at 10:18 pm on November 4, 2025:

@TheCharlatan thanks for fuzzing, and the diff for the fuzzer! I have taken it, and added you as a co-author :heart_hands:. @l0rinc it is concerning that you are getting malloc errors. Are there any other details you can share about this?

l0rinc commented at 7:49 am on November 6, 2025:

Did the same on master:

0git log -1
1commit 5c5704e730796c6f31e2d7891bf6334674a04219 (HEAD, upstream/master, upstream/HEAD, origin/master, origin/HEAD)

and unfortunately I’m getting the same:

0time ./build/bin/bitcoind -stopatheight=921129 -dbcache=45000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0\
1&& time ./build/bin/bitcoind -stopatheight=921129 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0\
2&& time ./build/bin/bitcoind -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0
3bitcoind(71239,0x170f2f000) malloc: Failed to allocate segment from range group - out of space
4...

so it’s not related to this PR

Also fuzzed for almost a day, no problems came up:

 0...
 1[#12935791](/bitcoin-bitcoin/12935791/)       REDUCE cov: 1485 ft: 9496 corp: 917/727Kb lim: 4096 exec/s: 475 rss: 189Mb L: 1640/4082 MS: 3 EraseBytes-PersAutoDict-InsertRepeatedBytes- DE: "\001\\"-
 2[#12941278](/bitcoin-bitcoin/12941278/)       REDUCE cov: 1485 ft: 9496 corp: 917/727Kb lim: 4096 exec/s: 475 rss: 189Mb L: 1734/4082 MS: 2 PersAutoDict-EraseBytes- DE: "\377\031"-
 3[#12943860](/bitcoin-bitcoin/12943860/)       REDUCE cov: 1485 ft: 9496 corp: 917/727Kb lim: 4096 exec/s: 475 rss: 189Mb L: 1303/4082 MS: 2 InsertByte-EraseBytes-
 4[#12956441](/bitcoin-bitcoin/12956441/)       REDUCE cov: 1485 ft: 9496 corp: 917/727Kb lim: 4096 exec/s: 475 rss: 189Mb L: 2407/4082 MS: 1 EraseBytes-
 5[#12973964](/bitcoin-bitcoin/12973964/)       REDUCE cov: 1485 ft: 9496 corp: 917/726Kb lim: 4096 exec/s: 475 rss: 189Mb L: 2400/4082 MS: 3 ChangeByte-ChangeBinInt-EraseBytes-
 6[#12980710](/bitcoin-bitcoin/12980710/)       REDUCE cov: 1485 ft: 9496 corp: 917/726Kb lim: 4096 exec/s: 475 rss: 189Mb L: 1835/4082 MS: 1 EraseBytes-
 7[#12986931](/bitcoin-bitcoin/12986931/)       REDUCE cov: 1485 ft: 9496 corp: 917/726Kb lim: 4096 exec/s: 475 rss: 189Mb L: 471/4082 MS: 1 EraseBytes-
 8[#12987148](/bitcoin-bitcoin/12987148/)       REDUCE cov: 1485 ft: 9496 corp: 917/726Kb lim: 4096 exec/s: 475 rss: 189Mb L: 1295/4082 MS: 2 PersAutoDict-EraseBytes- DE: "\377\377\377\377"-
 9[#13001660](/bitcoin-bitcoin/13001660/)       REDUCE cov: 1485 ft: 9496 corp: 917/726Kb lim: 4096 exec/s: 475 rss: 189Mb L: 2399/4082 MS: 2 PersAutoDict-EraseBytes- DE: "\332\377\377\377"-
10[#13008530](/bitcoin-bitcoin/13008530/)       REDUCE cov: 1485 ft: 9496 corp: 917/726Kb lim: 4096 exec/s: 475 rss: 189Mb L: 3079/4082 MS: 5 ChangeByte-PersAutoDict-EraseBytes-ChangeASCIIInt-InsertRepeatedBytes- DE: "\363\006\000\000\000\000\000\000"-
11[#13015051](/bitcoin-bitcoin/13015051/)       REDUCE cov: 1485 ft: 9496 corp: 917/726Kb lim: 4096 exec/s: 475 rss: 189Mb L: 3063/4082 MS: 1 EraseBytes-
12[#13026498](/bitcoin-bitcoin/13026498/)       REDUCE cov: 1485 ft: 9496 corp: 917/726Kb lim: 4096 exec/s: 475 rss: 189Mb L: 2374/4082 MS: 2 ChangeBinInt-EraseBytes-

andrewtoth force-pushed on Nov 4, 2025

DrahtBot added the label CI failed on Nov 4, 2025

andrewtoth force-pushed on Nov 4, 2025

DrahtBot removed the label CI failed on Nov 4, 2025

andrewtoth commented at 1:46 am on November 5, 2025: contributor

Benchmarked the latest up to block 921129 and it’s 16% faster :rocket:. Not as fast as some of @l0rinc’s numbers but it’s on a laptop with an internal NVMe SSD. This change will see the most benefit for disk IO with higher latency, like network connected storage.

Command	Mean [s]	Min [s]	Max [s]	Relative
`echo d606c36a13ca2a055d1a4eb4c623fb6aa45405b2 && /usr/bin/time ./build/bin/bitcoind -printtoconsole=0 -connect=192.168.2.171 -stopatheight=921129`	18498.670 ± 16.716	18486.850	18510.490	1.00
`echo 25c45bb0d0bd6618ec9296a1a43605657124e5de && /usr/bin/time ./build/bin/bitcoind -printtoconsole=0 -connect=192.168.2.171 -stopatheight=921129`	21537.077 ± 123.626	21449.660	21624.494	1.16 ± 0.01

Also refactored to not stop early if an input is missing. This let’s us simplify the logic. We can get rid of the different status flags and just synchronize each input on an atomic bool.

andrewtoth force-pushed on Nov 5, 2025

andrewtoth force-pushed on Nov 6, 2025

in src/inputfetcher.h:90 in 63dde36c1d outdated

76+     * @return false if there are no more inputs in the queue to fetch
77+     */
78+    bool FetchCoin() noexcept
79+    {
80+        const size_t i{m_input_head.fetch_add(1, std::memory_order_relaxed)};
81+        if (i >= m_inputs.size()) [[unlikely]] return false;

l0rinc commented at 3:22 pm on November 6, 2025:

when can this be true?

andrewtoth commented at 5:28 pm on November 6, 2025:

This is true when all inputs have been fetched from the block. We want the compiler to optimize for the case where we have work.

in src/inputfetcher.h:83 in 63dde36c1d

78+    bool FetchCoin() noexcept
79+    {
80+        const size_t i{m_input_head.fetch_add(1, std::memory_order_relaxed)};
81+        if (i >= m_inputs.size()) [[unlikely]] return false;
82+        auto& input{m_inputs[i]};
83+        if (std::binary_search(m_txids.begin(), m_txids.end(), input.outpoint.hash.ToUint256().GetUint64(0))) {

l0rinc commented at 3:22 pm on November 6, 2025:

0        if (std::ranges::binary_search(m_txids, input.outpoint.hash.ToUint256().GetUint64(0))) {

in src/inputfetcher.h:93 in 63dde36c1d

88+        auto coin{m_cache->GetPossiblySpentCoinFromCache(input.outpoint)};
89+        if (!coin) {
90+            try {
91+                coin = m_db->GetCoin(input.outpoint);
92+            } catch (const std::runtime_error& e) {
93+                LogPrintLevel(BCLog::VALIDATION, BCLog::Level::Warning, "InputFetcher failed to fetch input: %s.\n", e.what());

l0rinc commented at 3:24 pm on November 6, 2025:

nit: trailing newline shouldn’t be needed anymore

0                LogPrintLevel(BCLog::VALIDATION, BCLog::Level::Warning, "InputFetcher failed to fetch input: %s.", e.what());

in src/inputfetcher.h:120 in 63dde36c1d

115+            const auto& tx{block.vtx[i]};
116+            outputs_count += tx->vout.size();
117+            m_txids.emplace_back(tx->GetHash().ToUint256().GetUint64(0));
118+            for (const auto& input : tx->vin) m_inputs.emplace_back(input.prevout);
119+        }
120+        std::sort(m_txids.begin(), m_txids.end());

l0rinc commented at 3:24 pm on November 6, 2025:

0        std::ranges::sort(m_txids);

DrahtBot added the label CI failed on Nov 6, 2025

DrahtBot commented at 3:25 pm on November 6, 2025: contributor

🚧 At least one of the CI tasks failed. Task lint: https://github.com/bitcoin/bitcoin/actions/runs/19139133943/job/54698986986 LLM reason (✨ experimental): Trailing whitespace detected in src/inputfetcher.h caused the lint check to fail.

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

in src/inputfetcher.h:134 in 63dde36c1d

129+        temp_cache.Reserve(temp_cache.GetCacheSize() + m_inputs.size() + outputs_count);
130+        for (auto& input : m_inputs) {
131+            while (!input.ready.test(std::memory_order_acquire)) {
132+                // Work too while we wait
133+                if (!FetchCoin()) {
134+                    input.ready.wait(false, std::memory_order_acquire);

l0rinc commented at 3:26 pm on November 6, 2025:

based on https://en.cppreference.com/w/cpp/atomic/atomic_flag/wait.html

0                    input.ready.wait(/*old*/false, std::memory_order_acquire);

in src/inputfetcher.h:132 in 63dde36c1d

127+
128+        // Insert fetched coins into the temp_cache as they are set to ready.
129+        temp_cache.Reserve(temp_cache.GetCacheSize() + m_inputs.size() + outputs_count);
130+        for (auto& input : m_inputs) {
131+            while (!input.ready.test(std::memory_order_acquire)) {
132+                // Work too while we wait

l0rinc commented at 3:27 pm on November 6, 2025:

not yet sure I fully understand why this is needed - it’s more complex, but does it result in at least a theoretic speedup?

andrewtoth commented at 5:31 pm on November 6, 2025:

The work done on the other threads is much slower than what the main thread is doing. Fetching inputs possibly from disk vs inserting coins into the temporary cache. So, in cases where there are fewer worker threads the main thread will likely be waiting on the work to be done. In these cases we can just start fetching as well. I think this would have a large impact in cases where there are few worker threads (like rpis), or if using a low but > 1 -par value. For instance, using -par=2 this would in theory double the efficiency of fetching inputs from disk. It is likely not a measurable effect for setups that have 16 or more vcpus.

l0rinc commented at 6:09 pm on November 6, 2025:

I’m not sure I understand why, what’s the difference between two threads working while main is waiting vs one thread working and main also working? Where does the difference come from (especially given that the work is not fully cpu-bound)?

andrewtoth commented at 6:29 pm on November 6, 2025:

two threads working while main is waiting vs one thread working and main also working?

It would be two threads working while main is waiting vs two threads working and main also working? So the latter has 3 threads working vs the former’s 2 threads working. If using -par=3 this is the case. 50% more work is done in parallel.

l0rinc commented at 6:52 pm on November 6, 2025:

So why not just do the work that par defined and leave main asleep, wouldn’t that be simpler while basically achieving exactly the same?

andrewtoth commented at 7:02 pm on November 6, 2025:

No because we would have to insert all entries into temp cache at the end, instead of parallelizing that work as well. That was the previous implementation where we waited for every thread to be done and then inserted everything in series before exiting. Now, the main thread does both. It inserts while others are fetching, but if it inserts fast enough where it is waiting for new entries it will also fetch entries.

andrewtoth commented at 7:10 pm on November 6, 2025:

On a system with 15 worker threads + main, it is likely that main will not be waiting much. The other 15 threads are busy setting newly fetched coins to ready so that the main thread can continuously read true for the ready flags. On a system with only 3 worker threads + main, it is likely that the 3 worker threads will not be able to fetch and set ready coins fast enough where the main thread does not have to wait. For instance all 3 threads are fetching from disk, and the main thread reads the next input and ready is still false. So now it can either wait until one of the 3 worker threads fetches an input from disk, or it can start helping out the 3 workers and fetch from disk itself. This will increase parallel throughput by 33%, since 4 workers are fetching instead of just 3. Then when main returns with its fetched coin, main can insert the rest of the coins that the workers have fetched and set ready while main was busy fetching. Then it will catch up again to the latest input that is not yet ready, and fetch another coin.

l0rinc commented at 8:18 pm on November 6, 2025:

We discussed this out of band, here’s the summary:

the main thread is special because it has access to the dbcache, it can insert there without locking the cache.
the threads each have their inputs now, each of which have a switch to signal when they’ve fetched the input and move on to the next
the threads each compete for which input to fetch, marking them one-by-one as ready
the main thread goes in order, spins until the next one is available, if it needs to wait, it does some fetching itself, rechecks later if the given value is available and if so, it inserts to the dbcache, after which it checks the next value in order.

This way the IO and CPU bound work is parallelized, so we don’t need to do the heavy rehashing at the end, it’s done while the other threads are doing IO work - that’s why it’s faster.

I have to think about this, it sounds like we can simplify this further, but this is already an improvement over previous solutions.

And given that the worst thing that can happen for an internal spend is that if there’s another input with the same prefix in the same block, we wouldn’t fetch it here and it would be fetched during block connection like it was done before. This means that we likely don’t even need 64 bits for that, likely 32 are enough. Back of the napkin calculations indicate that would mean that 1/1000 blocks would contain transactions that won’t be fetched by InputFetcher and would need to be fetched during block connection instead on a single thread. As long as 32 bit checks are faster than 64 (which should definitely be the case for the sorted-vector case), this should likely result in an overall speedup. We definitely need to add a test case for that.

in src/inputfetcher.h:139 in 63dde36c1d

134+                    input.ready.wait(false, std::memory_order_acquire);
135+                    break;
136+                }
137+            }
138+            if (input.coin.IsSpent()) continue;
139+            temp_cache.EmplaceCoinInternalDANGER(COutPoint{input.outpoint}, std::move(input.coin));

l0rinc commented at 3:27 pm on November 6, 2025:

continue effectively skipping the last line is a bit confusing an dangerous - it should only skip the next line:

0            if (!input.coin.IsSpent()) {
1                temp_cache.EmplaceCoinInternalDANGER(COutPoint{input.outpoint}, std::move(input.coin));
2            }

in src/inputfetcher.h:70 in 63dde36c1d outdated

56+    /**
57+     * The set of first 8 bytes of txids of all txs in the block being fetched.
58+     * Used to filter out inputs that are created and spent in the same block,
59+     * since they will not be in the db or the cache.
60+     */
61+    std::vector<uint64_t> m_txids{};

l0rinc commented at 3:29 pm on November 6, 2025:

should we document here what happens in case of a cache miss or collision?

in src/inputfetcher.h:76 in 63dde36c1d

71+
72+    /**
73+     * Fetches the next input in the queue. Safe to call from any thread once inside the barrier.
74+     * 
75+     * @return true if there are more inputs in the queue to fetch
76+     * @return false if there are no more inputs in the queue to fetch

l0rinc commented at 3:33 pm on November 6, 2025:

this will fix the linter failure as well (whitespace after *):

0     * 
1     * [@return](/bitcoin-bitcoin/contributor/return/) whether there are more inputs in the queue to fetch

l0rinc changes_requested

in src/inputfetcher.h:78 in 63dde36c1d

73+     * Fetches the next input in the queue. Safe to call from any thread once inside the barrier.
74+     * 
75+     * @return true if there are more inputs in the queue to fetch
76+     * @return false if there are no more inputs in the queue to fetch
77+     */
78+    bool FetchCoin() noexcept

l0rinc commented at 5:00 pm on November 6, 2025:

can we call it something else to not coincide with CCoinsViewCache::FetchCoin

andrewtoth force-pushed on Nov 6, 2025

DrahtBot removed the label CI failed on Nov 6, 2025

andrewtoth force-pushed on Nov 8, 2025

andrewtoth force-pushed on Nov 12, 2025

andrewtoth force-pushed on Nov 14, 2025

l0rinc commented at 10:29 am on November 14, 2025: contributor

I was still wondering how number of parallel threads affects this, given that it’s not a cpu-bound task.

The measurements were done on i9, m4 and rpi4. First two have 16 threads, rpi has 4. The results still indicate to me that it doesn’t make sense to set parallelism based on number of cpus directly. Beyond 4-8 threads the systems didn’t really perform any better.

I will continue measuring it on other systems as well, but wanted to share preliminary results since these measurements take a lot of time.

 0commit=c2b0239001629a43d50cb8eb00e884423db89b38 && git log -1 --pretty='%h %s' $commit && git checkout $commit >/dev/null 2>&1 && rm -rf build && cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF >/dev/null 2>&1 && ninja -C build bitcoind -j$(nproc) >/dev/null 2>&1 && for par in 2 4 8 16 32 64; do   time ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=600000 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 -par=$par; done
 1c2b0239001 validation: fetch block inputs via InputFetcher before connecting
 2
 3real    94m1.512s
 4user    169m2.995s
 5sys     11m59.017s
 6
 7real    86m23.533s
 8user    171m32.112s
 9sys     12m42.929s
10
11real    82m58.013s
12user    179m20.981s
13sys     14m25.288s
14
15real    82m38.540s
16user    197m24.427s
17sys     20m31.753s
18
19real    82m38.442s
20user    197m16.954s
21sys     20m32.770s
22
23real    82m46.060s
24user    197m44.609s
25sys     20m37.426s

 0commit=c2b0239001629a43d50cb8eb00e884423db89b38 && git log -1 --pretty='%h %s' $commit && \
 1git checkout $commit >/dev/null 2>&1 && rm -rf build && cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DENABLE_IPC=OFF >/dev/null 2>&1 && ninja -C build bitcoind -j$(nproc) >/dev/null 2>&1 && \
 2for par in 2 4 8 16 32 64; do
 3  time ./build/bin/bitcoind -stopatheight=800000 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 -par=$par
 4done
 5
 6c2b0239001 validation: fetch block inputs via InputFetcher before connecting
 7./build/bin/bitcoind -stopatheight=800000 -dbcache=450 -reindex-chainstate     10711.00s user 1495.61s system 181% cpu 1:51:52.59 total
 8./build/bin/bitcoind -stopatheight=800000 -dbcache=450 -reindex-chainstate     10601.27s user 1311.29s system 207% cpu 1:35:49.25 total
 9./build/bin/bitcoind -stopatheight=800000 -dbcache=450 -reindex-chainstate     11362.34s user 2558.23s system 242% cpu 1:35:29.27 total
10./build/bin/bitcoind -stopatheight=800000 -dbcache=450 -reindex-chainstate     12203.63s user 5783.56s system 299% cpu 1:40:07.62 total
11./build/bin/bitcoind -stopatheight=800000 -dbcache=450 -reindex-chainstate     12124.30s user 5764.53s system 298% cpu 1:40:01.07 total
12./build/bin/bitcoind -stopatheight=800000 -dbcache=450 -reindex-chainstate     12276.36s user 5816.05s system 300% cpu 1:40:13.27 total

 0root@rpi4-8-1:/mnt/my_storage/bitcoin# for par in 4 8; do \
 1    COMMITS="c2b0239001629a43d50cb8eb00e884423db89b38"; \
 2    STOP=700000; DBCACHE=450; \
 3    CC=gcc; CXX=g++; \
 4    BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
 5    (echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done) && \
 6    (echo "" && echo "reindex-chainstate | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(df -T $BASE_DIR | awk 'NR==2{print $2}') | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 0 && echo SSD || echo HDD)"; echo "") &&\
 7    hyperfine \
 8      --sort command \
 9      --runs 1 \
10      --export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \
11      --parameter-list COMMIT ${COMMITS// /,} \
12      --prepare "killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
13        cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \
14        ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20" \
15      --conclude "killall bitcoind || true; sleep 5; grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'Disabling script verification at block [#1](/bitcoin-bitcoin/1/)' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log; \
16                  cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
17      "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 -par=$par"; \
18done
19
20c2b0239001 validation: fetch block inputs via InputFetcher before connecting
21
22reindex-chainstate | 700000 blocks | dbcache 450 | rpi4-8-1 | aarch64 | Cortex-A72 | 4 cores | 7.6Gi RAM | ext4 | SSD
23
24Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=700000 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 -par=4 (COMMIT = c2b0239001629a43d50cb8eb00e884423db89b38)
25  Time (abs ≡):        36057.895 s               [User: 61089.585 s, System: 12571.590 s]
26 
27
28c2b0239001 validation: fetch block inputs via InputFetcher before connecting
29
30reindex-chainstate | 700000 blocks | dbcache 450 | rpi4-8-1 | aarch64 | Cortex-A72 | 4 cores | 7.6Gi RAM | ext4 | SSD
31
32Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=700000 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 -par=8 (COMMIT = c2b0239001629a43d50cb8eb00e884423db89b38)
33  Time (abs ≡):        35682.583 s               [User: 61828.582 s, System: 14162.402 s]

Edit: updated times:

Edit2: update with even more fine-grained measurements

  0
  1i9-ssd | x86_64 | Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz | 16 cores | 62Gi RAM | xfs | SSD
  2
  3commit=c2b0239001629a43d50cb8eb00e884423db89b38 && git log -1 --pretty='%h %s' $commit && git checkout $commit >/dev/null 2>&1 && rm -rf build && cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF >/dev/null 2>&1 && ninja -C build bitcoind -j$(nproc) >/dev/null 2>&1 && for par in 2 4 8 16 32 64; do   time ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=600000 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 -par=$par; done
  4c2b0239001 validation: fetch block inputs via InputFetcher before connecting
  5
  6real    94m1.512s
  7user    169m2.995s
  8sys     11m59.017s
  9
 10real    86m23.533s
 11user    171m32.112s
 12sys     12m42.929s
 13
 14real    82m58.013s
 15user    179m20.981s
 16sys     14m25.288s
 17
 18real    82m38.540s
 19user    197m24.427s
 20sys     20m31.753s
 21
 22real    82m38.442s
 23user    197m16.954s
 24sys     20m32.770s
 25
 26real    82m46.060s
 27user    197m44.609s
 28sys     20m37.426s
 29
 30
 31
 32root@rpi4-8-1:/mnt/my_storage/bitcoin# for par in 2 4 8 16; do \
 33    COMMITS="c2b0239001629a43d50cb8eb00e884423db89b38"; \
 34    STOP=700000; DBCACHE=450; \
 35    CC=gcc; CXX=g++; \
 36    BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
 37    (echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done) && \
 38    (echo "" && echo "reindex-chainstate | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(df -T $BASE_DIR | awk 'NR==2{print $2}') | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 0 && echo SSD || echo HDD)"; echo "") &&\
 39    hyperfine \
 40      --sort command \
 41      --runs 1 \
 42      --export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \
 43      --parameter-list COMMIT ${COMMITS// /,} \
 44      --prepare "killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
 45        cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \
 46        ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20" \
 47      --conclude "killall bitcoind || true; sleep 5; grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'Disabling script verification at block [#1](/bitcoin-bitcoin/1/)' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log; \
 48                  cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
 49      "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 -par=$par"; \
 50done
 51
 52c2b0239001 validation: fetch block inputs via InputFetcher before connecting
 53
 54reindex-chainstate | 700000 blocks | dbcache 450 | rpi4-8-1 | aarch64 | Cortex-A72 | 4 cores | 7.6Gi RAM | ext4 | SSD
 55
 56Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=700000 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 -par=2 (COMMIT = c2b0239001629a43d50cb8eb00e884423db89b38)
 57  Time (abs ≡):        39770.114 s               [User: 62016.921 s, System: 12176.118 s]
 58
 59c2b0239001 validation: fetch block inputs via InputFetcher before connecting
 60
 61reindex-chainstate | 700000 blocks | dbcache 450 | rpi4-8-1 | aarch64 | Cortex-A72 | 4 cores | 7.6Gi RAM | ext4 | SSD
 62
 63Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=700000 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 -par=4 (COMMIT = c2b0239001629a43d50cb8eb00e884423db89b38)
 64  Time (abs ≡):        36057.895 s               [User: 61089.585 s, System: 12571.590 s]
 65
 66
 67c2b0239001 validation: fetch block inputs via InputFetcher before connecting
 68
 69reindex-chainstate | 700000 blocks | dbcache 450 | rpi4-8-1 | aarch64 | Cortex-A72 | 4 cores | 7.6Gi RAM | ext4 | SSD
 70
 71Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=700000 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 -par=8 (COMMIT = c2b0239001629a43d50cb8eb00e884423db89b38)
 72  Time (abs ≡):        35682.583 s               [User: 61828.582 s, System: 14162.402 s]
 73
 74c2b0239001 validation: fetch block inputs via InputFetcher before connecting
 75
 76reindex-chainstate | 700000 blocks | dbcache 450 | rpi4-8-1 | aarch64 | Cortex-A72 | 4 cores | 7.6Gi RAM | ext4 | SSD
 77
 78Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=700000 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 -par=16 (COMMIT = c2b0239001629a43d50cb8eb00e884423db89b38)
 79  Time (abs ≡):        36414.980 s               [User: 63043.265 s, System: 16780.513 s]
 80
 81
 82
 83
 84
 85rpi5-16-1:
 86commit=d6fac85ee4465cce8e81e36cdfd46636d34725fa && git log -1 --pretty='%h %s' $commit && git fetch origin $commit >/dev/null 2>&1 && git checkout $commit >/dev/null 2>&1 && rm -rf build && cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DENABLE_IPC=OFF >/dev/null 2>&1 && ninja -C build bitcoind -j$(nproc) >/dev/null 2>&1 && for par in 1 2 3 4 5 6 7 8 9 10; do   time ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -par=$par -stopatheight=800000 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0; done
 87d6fac85ee4 validation: fetch block inputs via InputFetcher before connecting
 88
 89real    393m14.069s
 90user    611m13.972s
 91sys     64m44.860s
 92
 93real    335m41.301s
 94user    615m9.967s
 95sys     61m58.381s
 96
 97real    314m37.969s
 98user    617m42.106s
 99sys     60m47.302s
100
101real    313m26.580s
102user    619m26.740s
103sys     63m59.801s
104
105real    314m35.267s
106user    622m30.314s
107sys     66m53.456s
108
109real    314m14.335s
110user    621m22.819s
111sys     68m30.831s
112
113real    314m49.454s
114user    621m3.552s
115sys     70m1.733s
116
117real    316m10.270s
118user    624m1.233s
119sys     70m48.074s
120
121real    315m57.060s
122user    619m48.948s
123sys     72m10.586s
124
125real    316m27.926s
126user    622m56.166s
127sys     73m16.170s
128
129
130
131i9:
132commit=d6fac85ee4465cce8e81e36cdfd46636d34725fa && git log -1 --pretty='%h %s' $commit && git fetch origin $commit >/dev/null 2>&1 && git checkout $commit >/dev/null 2>&1 && rm -rf build && cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DENABLE_IPC=OFF >/dev/null 2>&1 && ninja -C build bitcoind -j$(nproc) >/dev/null 2>&1 && for par in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18; do   time ./build/bin/bitcoind -par=$par -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0; done
133
134d6fac85ee4 validation: fetch block inputs via InputFetcher before connecting
135
136real    395m38.371s
137user    624m54.760s
138sys     50m2.017s
139
140real    321m54.706s
141user    638m0.914s
142sys     45m16.018s
143
144real    295m13.108s
145user    635m21.111s
146sys     44m41.847s
147
148real    281m53.289s
149user    636m26.743s
150sys     45m4.569s
151
152real    274m35.436s
153user    638m44.171s
154sys     46m25.320s
155
156real    270m4.622s
157user    641m52.896s
158sys     45m44.507s
159
160real    266m44.501s
161user    647m12.988s
162sys     47m8.929s
163
164real    265m9.994s
165user    661m8.192s
166sys     48m54.941s
167
168real    263m49.009s
169user    675m41.091s
170sys     49m52.024s
171
172real    262m13.541s
173user    687m23.237s
174sys     51m29.839s
175
176real    262m29.564s
177user    701m19.813s
178sys     52m20.824s
179
180real    261m29.692s
181user    717m48.727s
182sys     53m54.110s
183
184real    260m36.882s
185user    727m35.223s
186sys     56m30.900s
187
188real    259m58.230s
189user    740m17.961s
190sys     57m37.615s
191
192real    259m46.496s
193user    750m59.419s
194sys     59m55.428s
195
196real    262m3.069s
197user    756m49.146s
198sys     63m51.639s
199
200real    262m29.892s
201user    755m7.876s
202sys     63m29.261s
203
204real    262m54.519s
205user    759m4.919s
206sys     62m51.652s
207
208
209
210i7:
211commit=d6fac85ee4465cce8e81e36cdfd46636d34725fa && git log -1 --pretty='%h %s' $commit && git fetch origin $commit >/dev/null 2>&1 && git checkout $commit >/dev/null 2>&1 && rm -rf build && cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DENABLE_IPC=OFF >/dev/null 2>&1 && ninja -C build bitcoind -j$(nproc) >/dev/null 2>&1 && for par in 1 2 3 4 5 6 7 8 9 10 11 12 13; do   time ./build/bin/bitcoind -par=$par -datadir=/mnt/my_storage/BitcoinData -stopatheight=900000 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0; done
212
213d6fac85ee4 validation: fetch block inputs via InputFetcher before connecting
214
215real    677m53.522s
216user    540m13.974s
217sys     48m0.685s
218
219real    616m58.994s
220user    547m24.056s
221sys     46m0.808s
222
223real    582m52.884s
224user    554m55.162s
225sys     45m21.949s
226
227real    577m5.937s
228user    563m47.629s
229sys     45m28.084s
230
231real    568m25.225s
232user    576m19.153s
233sys     46m34.582s
234
235real    566m31.568s
236user    586m8.162s
237sys     46m41.220s
238
239real    564m43.096s
240user    594m42.382s
241sys     49m17.091s
242
243real    558m40.218s
244user    600m10.625s
245sys     52m24.030s
246
247real    556m23.944s
248user    606m36.724s
249sys     55m27.147s
250
251real    565m47.020s
252user    607m41.449s
253sys     56m21.510s
254
255real    563m37.784s
256user    609m40.123s
257sys     57m35.573s
258
259real    567m43.207s
260user    608m53.189s
261sys     57m43.208s
262
263real    563m14.968s
264user    611m1.819s
265sys     58m43.991s
266
267
268Macbook Pro M4 Max
269commit=c2b0239001629a43d50cb8eb00e884423db89b38 && git log -1 --pretty='%h %s' $commit && \
270git checkout $commit >/dev/null 2>&1 && rm -rf build && cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DENABLE_IPC=OFF >/dev/null 2>&1 && ninja -C build bitcoind -j$(nproc) >/dev/null 2>&1 && \
271for par in 2 4 8 16 32 64; do
272  time ./build/bin/bitcoind -stopatheight=800000 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 -par=$par
273done
274
275c2b0239001 validation: fetch block inputs via InputFetcher before connecting
276./build/bin/bitcoind -stopatheight=800000 -dbcache=450 -reindex-chainstate     10711.00s user 1495.61s system 181% cpu 1:51:52.59 total
277./build/bin/bitcoind -stopatheight=800000 -dbcache=450 -reindex-chainstate     10601.27s user 1311.29s system 207% cpu 1:35:49.25 total
278./build/bin/bitcoind -stopatheight=800000 -dbcache=450 -reindex-chainstate     11362.34s user 2558.23s system 242% cpu 1:35:29.27 total
279./build/bin/bitcoind -stopatheight=800000 -dbcache=450 -reindex-chainstate     12203.63s user 5783.56s system 299% cpu 1:40:07.62 total
280./build/bin/bitcoind -stopatheight=800000 -dbcache=450 -reindex-chainstate     12124.30s user 5764.53s system 298% cpu 1:40:01.07 total
281./build/bin/bitcoind -stopatheight=800000 -dbcache=450 -reindex-chainstate     12276.36s user 5816.05s system 300% cpu 1:40:13.27 total

andrewtoth force-pushed on Nov 23, 2025

DrahtBot added the label CI failed on Nov 23, 2025

DrahtBot commented at 1:55 am on November 23, 2025: contributor

🚧 At least one of the CI tasks failed. Task fuzzer,address,undefined,integer, no depends: https://github.com/bitcoin/bitcoin/actions/runs/19603900050/job/56139454218 LLM reason (✨ experimental): Fuzz run crashes with UndefinedBehaviorSanitizer: null-pointer-use (null CTransaction dereference in CoinsViewCacheAsync).

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

andrewtoth marked this as a draft on Nov 23, 2025

andrewtoth force-pushed on Nov 23, 2025

DrahtBot removed the label CI failed on Nov 23, 2025

andrewtoth marked this as ready for review on Nov 23, 2025

andrewtoth commented at 5:13 pm on November 23, 2025: contributor

I’ve updated the PR to make the InputFetcher a subclass of CCoinsViewCache. Instead of waiting on the MPSC queue to be finished before connecting the block, the queue can be processed inside CCoinsViewCache::FetchCoin during ConnectBlock. This makes the fetching non-blocking, which is a significant performance improvement. It’s been renamed to CoinsViewCacheAsync. @l0rinc thank you for your very thorough benchmarks. I’ve updated this to use 4 worker threads, which yields the same speed as with 15 on my benchmark machine. I’ve also made this non-configurable, as I don’t see a reason why a user would want to change it. It should help performance on single core machines as well since the parallel work is IO bound.

This new non-blocking version using 4 threads is significantly faster. On the same machine that yielded 16% IBD speedup until block 921129, this version is now 21% faster :rocket:. I’m curious to see benchmarks on other machines.

Command	Mean [s]	Min [s]	Max [s]	Relative
`echo 42e68d48b4da838361b045a977242d3262f8b351 && /usr/bin/time ./build/bin/bitcoind -printtoconsole=0 -connect=192.168.2.171 -stopatheight=921129`	17842.357 ± 131.143	17749.624	17935.089	1.00
`echo 25c45bb0d0bd6618ec9296a1a43605657124e5de && /usr/bin/time ./build/bin/bitcoind -printtoconsole=0 -connect=192.168.2.171 -stopatheight=921129`	21537.077 ± 123.626	21449.660	21624.494	1.21

andrewtoth force-pushed on Nov 24, 2025

DrahtBot added the label CI failed on Nov 24, 2025

DrahtBot commented at 1:28 am on November 24, 2025: contributor

🚧 At least one of the CI tasks failed. Task fuzzer,address,undefined,integer, no depends: https://github.com/bitcoin/bitcoin/actions/runs/19620057148/job/56178655244 LLM reason (✨ experimental): libFuzzer crash (deadly signal) from an assertion in DbCoinsView::GetCoin during fuzzing.

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

andrewtoth force-pushed on Nov 24, 2025

DrahtBot removed the label CI failed on Nov 25, 2025

andrewtoth force-pushed on Nov 25, 2025

andrewtoth force-pushed on Nov 26, 2025

coins: add CoinsViewCacheAsync for parallel input fetching

Introduce CoinsViewCacheAsync utility class that manages worker threads to
fetch block inputs in parallel. Uses a lock-free MPSC queue design
with an atomic counter for work distribution and per-input atomic flags
for completion signaling.

Provides the same API as CCoinsViewCache, and adds StartFetching and Reset
methods to control background fetching.

The main thread processes inputs sequentially as they become available,
returning fetched coins from the FetchCoin method. If the main thread
catches up to the workers, it assists with fetching to maximize
parallelism.

Co-authored-by: l0rinc <pap.lorinc@gmail.com>
Co-authored-by: TheCharlatan <seb.kung@gmail.com>

fc4346fe05

bench: add CoinsViewCacheAsync benchmark

Add benchmark to measure CoinsViewCacheAsync performance.
This helps validate performance improvements and identify potential regressions.

58ff45a641

validation: fetch block inputs via CCoinsViewCacheAsync during connection

Pass CCoinsViewCacheAsync as the view into ConnectBlock. Inputs are now fetched
in parallel during block connection, speeding up connection by over 30%.

0f3778fbfb

andrewtoth force-pushed on Nov 27, 2025

in src/coins.h:405 in 0f3778fbfb

400@@ -401,6 +401,14 @@ class CCoinsViewCache : public CCoinsViewBacked
401      */
402     bool HaveCoinInCache(const COutPoint &outpoint) const;
403 
404+    /**
405+     * Retrieves the coin from the cache even if it is spent, without calling

l0rinc commented at 12:21 pm on November 27, 2025:

nit: other comments use the bare verb form

0     * Retrieve the coin from the cache even if it is spent, without calling

in src/coinsviewcacheasync.h:85 in 0f3778fbfb

80+     * Similar to CCoinsViewCache::GetCoin, but it does not mutate internally.
81+     * Therefore safe to call from any thread once inside the barrier.
82+     */
83+    std::optional<Coin> GetCoinWithoutMutating(const COutPoint& outpoint) const
84+    {
85+        auto coin{static_cast<CCoinsViewCache*>(base)->GetPossiblySpentCoinFromCache(outpoint)};

l0rinc commented at 12:28 pm on November 27, 2025:

This highlights that our dbcache layering is a bit messy – having to static_cast the base view to CCoinsViewCache here is quite ugly. It would be good to clean this up in a follow-up.

in src/test/fuzz/coinsviewcacheasync.cpp:164 in 0f3778fbfb

159+                assert(coin->nHeight == db_coin.nHeight);
160+                assert(coin->out == db_coin.out);
161+            }
162+        }
163+        assert(cache.GetCacheSize() == outpoints_in_cache.size());
164+        fuzzed_data_provider.ConsumeBool() ? (void)cache.Flush() : cache.Reset();

l0rinc commented at 1:05 pm on November 27, 2025:

We’re moving towards Flush not returning a value – can we avoid adding new code that relies on its return? Not sure it’s possible before that other PR lands, but worth keeping in mind.

andrewtoth commented at 4:53 pm on November 27, 2025:

The (void) here does not rely on the return value. Or do you mean something else?

l0rinc commented at 5:09 pm on November 27, 2025:

we wouldn’t need it if Flush were a void itself, like Reset, right?

andrewtoth commented at 5:15 pm on November 27, 2025:

Yes, but if the other PR is merged, then this will still compile with no errors or warnings right?

l0rinc commented at 5:25 pm on November 27, 2025:

you will have a conflict where you’re making Flush abstract and they’re making it void

in src/bench/coinsviewcacheasync.cpp:21 in 0f3778fbfb

16+
17+//! Simulates a DB by adding a delay when calling GetCoin
18+struct DelayedCoinsView : CCoinsView {
19+    std::optional<Coin> GetCoin(const COutPoint&) const override
20+    {
21+        UninterruptibleSleep(DELAY);

l0rinc commented at 1:06 pm on November 27, 2025:

I wonder if we could change this benchmark to use an actual LevelDB-backed view instead of a synthetic delay – and maybe reuse the setup from #32554 for a more realistic measurement?

in src/coinsviewcacheasync.h:208 in fc4346fe05 outdated

203+    explicit CoinsViewCacheAsync(CCoinsViewCache& cache, const CCoinsView& db, bool deterministic = false) noexcept
204+        : CCoinsViewCache{&cache, deterministic}, m_db{db}, m_barrier{WORKER_THREADS + 1}
205+    {
206+        for (uint32_t n{0}; n < WORKER_THREADS; ++n) {
207+            m_worker_threads.emplace_back([this, n] {
208+                util::ThreadRename(strprintf("inputfetcher.%i", n));

l0rinc commented at 1:08 pm on November 27, 2025:

nit: now that this is CoinsViewCacheAsync, maybe rename the thread prefix from “inputfetcher.%i” to something like “coinsviewcacheasync.%i” or similar, to reflect the current design.

in src/bench/coinsviewcacheasync.cpp:43 in 0f3778fbfb

38+        async_cache.StartFetching(block);
39+        async_cache.Reset();
40+    });
41+}
42+
43+BENCHMARK(CoinsViewCacheAsyncBenchmark, benchmark::PriorityLevel::HIGH);

l0rinc commented at 1:15 pm on November 27, 2025:

nit: the formatter should enforce a trailing newline at the end of C++ source files; could you add one here?

in src/bench/coinsviewcacheasync.cpp:38 in 0f3778fbfb

33+    DelayedCoinsView db{};
34+    CCoinsViewCache main_cache(&db);
35+    CoinsViewCacheAsync async_cache{main_cache, db};
36+
37+    bench.run([&] {
38+        async_cache.StartFetching(block);

l0rinc commented at 1:16 pm on November 27, 2025:

What if we added a second run inside the same benchmark that calls StartFetching again or a series of AccessCoin calls - to exercise the “everything is already in the cache” path (similar to a large -dbcache scenario)?

andrewtoth commented at 4:54 pm on November 27, 2025:

Ah, yes the Reset now just short circuits it. So this is not a very good benchmark anymore. Will fix to access all coins.

in src/test/fuzz/coinsviewcacheasync.cpp:39 in 0f3778fbfb

34+        cacheCoins.clear();
35+        ReallocateCache();
36+        cachedCoinsUsage = 0;
37+    }
38+
39+    CoinsViewDb() : CCoinsViewCache(nullptr, /*deterministic=*/true) {}

l0rinc commented at 1:18 pm on November 27, 2025:

Same idea as in the bench: would it be feasible to fuzz this against an actual LevelDB-backed view instead of a synthetic cache, or is that too expensive / complicated for now?

in src/test/fuzz/coinsviewcacheasync.cpp:67 in 0f3778fbfb

62+
63+        std::map<const COutPoint, const Coin> db_map{};
64+        std::map<const COutPoint, const Coin> cache_map{};
65+        std::vector<COutPoint> input_outpoints{};
66+
67+        CCoinsViewCache main_cache(&*g_db, /*deterministic=*/true);

l0rinc commented at 1:18 pm on November 27, 2025:

Why do we need this cache to be constructed with deterministic=true here? Is the deterministic ordering actually required for the fuzz harness, or could we drop that?

in src/coins.h:410 in 0f3778fbfb

405+     * Retrieves the coin from the cache even if it is spent, without calling
406+     * the backing CCoinsView if no coin exists.
407+     * Used in CoinsViewCacheAsync to make sure we do not add a coin from the backing
408+     * view when it is spent in the cache but not yet flushed to the parent.
409+     */
410+    std::optional<Coin> GetPossiblySpentCoinFromCache(const COutPoint& outpoint) const noexcept;

l0rinc commented at 1:20 pm on November 27, 2025:

As mentioned before, I’m not a fan of adding a separate “get-possibly-spent” helper – it feels like spentness is a separate concern and we’re encoding it into the accessor. I still think long term it’d be cleaner if GetCoin always returned the raw coin and call sites handled spentness explicitly, but I understand that’s probably a separate cleanup.

in src/coinsviewcacheasync.h:47 in 0f3778fbfb

42+{
43+private:
44+    //! The latest input not yet being fetched. Workers atomically increment this when fetching.
45+    mutable std::atomic_uint32_t m_input_head{0};
46+    //! The latest input not yet accessed by a consumer. Only the main thread increments this.
47+    mutable uint32_t m_input_tail{0};

l0rinc commented at 1:21 pm on November 27, 2025:

Hmm, can the main thread keep track of this locally instead of storing m_input_tail as a member? It looks like we’re effectively simulating a queue here (claiming work from the “head” and consuming from the “tail”). Could we use an actual std::deque or similar, or would that invalidate indices / references that the workers rely on?

andrewtoth commented at 4:57 pm on November 27, 2025:

Yes, this is a queue! An MPSC queue to be exact. I don’t think it’s possible to not store this as an instance member. I don’t think we can use a std::deque though because we can’t actually mutate the container. The benefit of a deque would be to actually push and pop, but here we set up the container before kicking off the worker threads.

in src/coinsviewcacheasync.h:70 in 0f3778fbfb

65+    mutable std::vector<InputToFetch> m_inputs{};
66+
67+    /**
68+     * The first 8 bytes of txids of all txs in the block being fetched. This is used to filter out inputs that
69+     * are created and spent in the same block, since they will not be in the db or the cache.
70+     * Using only the first 8 bytes is a performance improvement, versus storing the entire 32 bytes. In case of a

l0rinc commented at 1:23 pm on November 27, 2025:

Do we have tests that explicitly exercise the 8-byte prefix collision case (i.e. two txids in the same block sharing the same first 64 bits)? If we used 32-bit prefixes instead, collisions would be much more frequent, but the structure would be smaller; did we benchmark 32-bit vs 64-bit prefixes, or is 64-bit a conservative choice?

andrewtoth commented at 4:59 pm on November 27, 2025:

did we benchmark 32-bit vs 64-bit prefixes

I did do microbenchmarks with 32-bit, and it was not more performant. Also, there is no uint256::GetUint32, so we would either just cast the 64 bits to 32 or have to write a new method on uint256. I don’t think it’s worth exploring this more.

in src/coinsviewcacheasync.h:122 in 0f3778fbfb

117+    {
118+        const auto [ret, inserted] = cacheCoins.try_emplace(outpoint);
119+        if (inserted) {
120+            for (auto i{m_input_tail}; i < m_inputs.size(); ++i) {
121+                auto& input{m_inputs[i]};
122+                if (input.outpoint != outpoint) continue;

l0rinc commented at 1:28 pm on November 27, 2025:

I find this iteration quite hard to follow. Could we extract the search into a helper and then act on the found index, something like:

 0    std::optional<uint32_t> FindInputIndex(const COutPoint& outpoint) const
 1    {
 2        for (size_t i{m_input_tail}; i < m_inputs.size(); ++i) {
 3            if (m_inputs[i].outpoint == outpoint) {
 4                return i;
 5            }
 6        }
 7        return std::nullopt;
 8    }
 9
10
11    CCoinsMap::iterator FetchCoin(const COutPoint& outpoint) const override
12    {
13        const auto [ret, inserted] = cacheCoins.try_emplace(outpoint);
14        if (!inserted) return ret;
15
16        if (const auto idx_opt{FindInputIndex(outpoint)}) {
17            auto& input{m_inputs[*idx_opt]};
18
19            // Wait until this input is ready. Acquire matches the worker's release.
20            while (!input.ready.test(std::memory_order_acquire)) {
21                // Try to process other inputs instead of just waiting
22                if (!ProcessInputInBackground()) {
23                    // No more work; just wait on this one
24                    input.ready.wait(/*old=*/false, std::memory_order_acquire);
25                    break;
26                }
27            }
28
29            if (input.coin) [[likely]]
30                ret->second.coin = std::move(*input.coin);
31            m_input_tail = *idx_opt + 1; // We will never need to scan earlier entries again
32        }
33
34        if (ret->second.coin.IsSpent()) [[unlikely]] {
35            // We only get here for BIP30 checks, txid collisions, or missing/spent inputs.
36            if (auto coin{FetchCoinFromParent(outpoint)}) {
37                ret->second.coin = std::move(*coin);
38            } else {
39                cacheCoins.erase(ret);
40                return cacheCoins.end();
41            }
42        }
43
44        cachedCoinsUsage += ret->second.coin.DynamicMemoryUsage();
45        return ret;
46    }

in src/coinsviewcacheasync.h:177 in 0f3778fbfb

172+            const auto& tx{block.vtx[i]};
173+            m_txids.emplace_back(tx->GetHash().ToUint256().GetUint64(0));
174+            for (const auto& input : tx->vin) m_inputs.emplace_back(input.prevout);
175+        }
176+        if (m_inputs.size() == 0) return;
177+        std::ranges::sort(m_txids);

l0rinc commented at 2:46 pm on November 27, 2025:

It might be worth mentioning (either in a comment here or in the PR description) that benchmarks indicated this sorted std::vector<uint64_t> + binary_search approach is significantly faster than a std::unordered_set<Txid> or std::set<Txid> for the expected hit/miss mix.

in src/coinsviewcacheasync.h:212 in 0f3778fbfb

207+            m_worker_threads.emplace_back([this, n] {
208+                util::ThreadRename(strprintf("inputfetcher.%i", n));
209+                while (true) {
210+                    m_barrier.arrive_and_wait();
211+                    if (m_request_stop) [[unlikely]] return;
212+                    while (ProcessInputInBackground()) {}

l0rinc commented at 2:49 pm on November 27, 2025:

nit: this is very compact, but maybe a bit hard to see, consider:

0for (;;) {
1    if (!ProcessInputInBackground()) break;
2}

in src/validation.cpp:3100 in 0f3778fbfb

3096@@ -3095,6 +3097,7 @@ bool Chainstate::ConnectTip(
3097             if (state.IsInvalid())
3098                 InvalidBlockFound(pindexNew, state);
3099             LogError("%s: ConnectBlock %s failed, %s\n", __func__, pindexNew->GetBlockHash().ToString(), state.ToString());
3100+            view.Reset();

l0rinc commented at 2:49 pm on November 27, 2025:

Is Reset() strictly necessary here given that a failed ConnectBlock will discard the ephemeral view anyway? Can we carry any async state across to the next block?

andrewtoth commented at 5:01 pm on November 27, 2025:

We don’t actually throw away the ephemeral view anymore. We just reset the state each time. This is a big performance improvement since we don’t have to reallocate anything. If we didn’t do this we would need an external thread pool as well, since the threads are owned by the CoinsViewCacheAsync. I think it’s a fair tradeoff for one extra line here. Perhaps calling it ephemeral_view is a misnomer though. I’m not sure what a better name would be right now.

l0rinc commented at 5:13 pm on November 27, 2025:

We don’t actually throw away the ephemeral view anymore

Hah, I missed that in the latest push, thanks. 👍

calling it ephemeral_view is a misnomer though

Yeah :)

in src/coinsviewcacheasync.h:87 in 0f3778fbfb

82+     */
83+    std::optional<Coin> GetCoinWithoutMutating(const COutPoint& outpoint) const
84+    {
85+        auto coin{static_cast<CCoinsViewCache*>(base)->GetPossiblySpentCoinFromCache(outpoint)};
86+        if (!coin) coin = m_db.GetCoin(outpoint);
87+        if (coin && !coin->IsSpent()) [[likely]] return coin;

l0rinc commented at 2:51 pm on November 27, 2025:

I don’t mind the [[likely]] hints (they can help document expectations), but I know others are opposed to using them in our codebase.

andrewtoth commented at 5:02 pm on November 27, 2025:

There are some already in the codebase. They are added in C++20, why not use them? Maybe in future compiler versions the hints will become more useful for optimization? I only use them when we know a path is always happy or unhappy for a valid block (except for BIP30 checks, which we only do at the very beginning). An invalid block with valid proof of work is very rare, so we can optimize for the case where we don’t get one. If we get an invalid block, we care less about speed to validate.

l0rinc commented at 5:23 pm on November 27, 2025:

why not use them

#30535 (review) #31682 (review) #33657 (review)

andrewtoth commented at 5:40 pm on November 27, 2025:

The first link shows a benefit to using them, and that is now in our codebase. The second and third links reference this blog post which has crazy convoluted usages. I don’t find that blog post convincing. The usages here are straightforward and in very hot paths. They are only used in paths where we know a valid block will always or never go into (aside from BIP30).

in src/coinsviewcacheasync.h:176 in 0f3778fbfb

171+        for (size_t i{1}; i < block.vtx.size(); ++i) {
172+            const auto& tx{block.vtx[i]};
173+            m_txids.emplace_back(tx->GetHash().ToUint256().GetUint64(0));
174+            for (const auto& input : tx->vin) m_inputs.emplace_back(input.prevout);
175+        }
176+        if (m_inputs.size() == 0) return;

l0rinc commented at 2:54 pm on November 27, 2025:

What does the if (m_inputs.size() == 0) guard against exactly?

andrewtoth commented at 5:06 pm on November 27, 2025:

I use this instead of an m_is_fetching or equivalent boolean state. We don’t want to exit the barrier if we haven’t entered it. We don’t enter above if the block has less than 2 txs, and we don’t enter if it is an invalid block with 2 or more txs that have zero inputs. After we stop the threads and exit the barrier, we clear the inputs as well so we don’t exit again.

l0rinc commented at 3:26 pm on November 27, 2025: contributor

This new design achieves the best results I’ve seen so far across all platforms measured, excellent job!!

It evolved from an InputFetcher helper (where the parallelism happened before ConnectBlock was called) to CoinsViewCacheAsync, a multithreaded CCoinsViewCache subclass whose worker threads run in parallel with ConnectBlock itself.

Internal spends are still filtered using short txid prefixes stored in a sorted std::vector and checked via std::binary_search. In the rare case of a prefix collision, the async fetcher will simply skip that input, and it will be fetched later by the normal synchronous path.

For now the async view uses a fixed worker thread count of 4. The workload is primarily I/O-bound on DB latency rather than CPU-bound, so 4 workers already hide most of the latency and it simplifies the implementation. If needed we can make this configurable or tie it to -par later.

This way the I/O-bound work runs in parallel with the CPU-bound validation work, and the preliminary results are very encouraging: on a Raspberry Pi 5 the best -reindex-chainstate so far is about 7.3 hours with -dbcache=4500 and about 7.7 hours with the default 450 MB, roughly 36% and 46% faster than the current single-threaded baseline.

The new implementation has been fuzzed for several days - it would be good to get some more eyes on it.

(some of these measurements are suspiciously good, especially the M4 versions (maybe it switches from the energy efficient cores to the performance ones), I will try to replicate the results in followups)

  0> M4 @ dbcache=450:
  1for commit in dfde31f2ec1f90976f3ba6b06f2b38a1307c01ab 32de0ff1a97fc2880ba2f507dd00082727badf3f; do
  2  git fetch origin $commit >/dev/null 2>&1 && git checkout $commit >/dev/null 2>&1 && git log -1 --pretty='%h %s' && \
  3  rm -rf build && cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DENABLE_IPC=OFF >/dev/null 2>&1 && ninja -C build bitcoind -j$(nproc) >/dev/null 2>&1 && \
  4  time ./build/bin/bitcoind -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 || exit 1
  5done
  6dfde31f2ec Merge bitcoin/bitcoin#33864: scripted-diff: fix leftover references to `policy/fees.h`
  7./build/bin/bitcoind -stopatheight=921129 -dbcache=450 -reindex-chainstate     20994.00s user 5348.08s system 120% cpu 6:03:05.28 total
  8
  932de0ff1a9 validation: fetch block inputs via CCoinsViewCacheAsync during connection
 10./build/bin/bitcoind -stopatheight=921129 -dbcache=450 -reindex-chainstate     16748.42s user 1830.34s system 276% cpu 1:51:50.94 total
 11
 12
 13> M4 @ dbcache=4500:
 14d5ed4ba9d8 Merge bitcoin/bitcoin#33906: depends: Add patch for Windows11Style plugin
 15./build/bin/bitcoind -stopatheight=921129 -dbcache=4500 -blocksonly    8895.93s user 751.51s system 133% cpu 2:00:08.82 total
 16b1a791db1c validation: fetch block inputs via CCoinsViewCacheAsync during connection
 17./build/bin/bitcoind -stopatheight=921129 -dbcache=4500 -blocksonly    10327.49s user 940.30s system 186% cpu 1:40:57.44 total
 18
 19> M4 @ dbcache=45000:
 20for commit in dfde31f2ec1f90976f3ba6b06f2b38a1307c01ab b1a791db1c75a47569b690baf7b074b78e08ca5a; do
 21  git fetch origin $commit >/dev/null 2>&1 && git checkout $commit >/dev/null 2>&1 && git log -1 --pretty='%h %s' && \
 22  rm -rf build && cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DENABLE_IPC=OFF >/dev/null 2>&1 && ninja -C build bitcoind -j$(nproc) >/dev/null 2>&1 && \
 23  time ./build/bin/bitcoind -stopatheight=921129 -dbcache=45000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 || exit 1
 24done
 25dfde31f2ec Merge bitcoin/bitcoin#33864: scripted-diff: fix leftover references to `policy/fees.h`
 26./build/bin/bitcoind -stopatheight=921129 -dbcache=45000 -reindex-chainstate   6338.71s user 332.32s system 131% cpu 1:24:22.56 total
 27
 28b1a791db1c validation: fetch block inputs via CCoinsViewCacheAsync during connection
 29./build/bin/bitcoind -stopatheight=921129 -dbcache=45000 -reindex-chainstate   7301.13s user 471.62s system 162% cpu 1:19:35.33 total
 30
 31
 32> i9 @ dbcache=450:
 33dfde31f2ec Merge bitcoin/bitcoin#33864: scripted-diff: fix leftover references to `policy/fees.h`
 34e86d485271 validation: fetch block inputs via CCoinsViewCacheAsync during connection
 35
 36reindex-chainstate | 921129 blocks | dbcache 450 | i9-ssd | x86_64 | Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz | 16 cores | 62Gi RAM | xfs | SSD
 37
 38Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = dfde31f2ec1f90976f3ba6b06f2b38a1307c01ab)
 39  Time (abs ≡):        20659.626 s               [User: 40733.602 s, System: 2871.904 s]
 40
 41Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = e86d48527122c803a58d7bfecffd43a0e373c756)
 42  Time (abs ≡):        14729.736 s               [User: 39566.674 s, System: 2313.959 s]
 43
 44Relative speed comparison
 45        1.40          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = dfde31f2ec1f90976f3ba6b06f2b38a1307c01ab)
 46        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = e86d48527122c803a58d7bfecffd43a0e373c756)
 47
 48
 49> i9  @ dbcache=4500:
 50COMMITS="dfde31f2ec1f90976f3ba6b06f2b38a1307c01ab 32de0ff1a97fc2880ba2f507dd00082727badf3f"; STOP=921129; DBCACHE=4500; CC=gcc; CXX=g++; BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; (echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done) && (echo "" && echo "reindex-chainstate | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(df -T $BASE_DIR | awk 'NR==2{print $2}') | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 0 && echo SSD || echo HDD)"; echo "") &&hyperfine   --sort command   --runs 1   --export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json"   --parameter-list COMMIT ${COMMITS// /,}   --prepare "killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
 51    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \
 52    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20"   --conclude "killall bitcoind || true; sleep 5; grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'Disabling script verification at block [#1](/bitcoin-bitcoin/1/)' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log; \
 53              cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log"   "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0";
 54
 55dfde31f2ec Merge bitcoin/bitcoin#33864: scripted-diff: fix leftover references to `policy/fees.h`
 5632de0ff1a9 validation: fetch block inputs via CCoinsViewCacheAsync during connection
 57
 58reindex-chainstate | 921129 blocks | dbcache 4500 | i9-ssd | x86_64 | Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz | 16 cores | 62Gi RAM | xfs | SSD
 59
 60Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = dfde31f2ec1f90976f3ba6b06f2b38a1307c01ab)
 61  Time (abs ≡):        16615.768 s               [User: 25458.915 s, System: 859.662 s]
 62
 63Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 32de0ff1a97fc2880ba2f507dd00082727badf3f)
 64  Time (abs ≡):        13689.366 s               [User: 26290.581 s, System: 991.037 s]
 65
 66Relative speed comparison
 67        1.21          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = dfde31f2ec1f90976f3ba6b06f2b38a1307c01ab)
 68        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 32de0ff1a97fc2880ba2f507dd00082727badf3f)
 69
 70
 71> i9  @ dbcache=45000:
 72COMMITS="dfde31f2ec1f90976f3ba6b06f2b38a1307c01ab 32de0ff1a97fc2880ba2f507dd00082727badf3f"; STOP=921129; DBCACHE=45000; CC=gcc; CXX=g++; BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; (echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done) && (echo "" && echo "reindex-chainstate | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(df -T $BASE_DIR | awk 'NR==2{print $2}') | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 0 && echo SSD || echo HDD)"; echo "") &&hyperfine   --sort command   --runs 1   --export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json"   --parameter-list COMMIT ${COMMITS// /,}   --prepare "killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
 73    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \
 74    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20"   --conclude "killall bitcoind || true; sleep 5; grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'Disabling script verification at block [#1](/bitcoin-bitcoin/1/)' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log; \
 75              cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log"   "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0";
 76
 77dfde31f2ec Merge bitcoin/bitcoin#33864: scripted-diff: fix leftover references to `policy/fees.h`
 7832de0ff1a9 validation: fetch block inputs via CCoinsViewCacheAsync during connection
 79
 80reindex-chainstate | 921129 blocks | dbcache 45000 | i9-ssd | x86_64 | Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz | 16 cores | 62Gi RAM | xfs | SSD
 81
 82Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=45000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = dfde31f2ec1f90976f3ba6b06f2b38a1307c01ab)
 83  Time (abs ≡):        16118.775 s               [User: 23433.898 s, System: 725.843 s]
 84
 85Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=45000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 32de0ff1a97fc2880ba2f507dd00082727badf3f)
 86  Time (abs ≡):        14429.306 s               [User: 23850.818 s, System: 792.987 s]
 87
 88Relative speed comparison
 89        1.12          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=45000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = dfde31f2ec1f90976f3ba6b06f2b38a1307c01ab)
 90        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=45000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 32de0ff1a97fc2880ba2f507dd00082727badf3f)
 91
 92
 93> i7 @ dbcache=450:
 94COMMITS="dfde31f2ec1f90976f3ba6b06f2b38a1307c01ab 32de0ff1a97fc2880ba2f507dd00082727badf3f"; STOP=921129; DBCACHE=450; CC=gcc; CXX=g++; BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; (echo ""; for c in $COMMITS; do git fetch -q origi
 95n $c && git log -1 --pretty='%h %s' $c || exit 1; done) && (echo "" && echo "reindex-chainstate | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(unam
 96e -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(df -T $BASE_DIR | awk 'NR==2{print $2}') | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 0 && echo SSD || echo HDD)"; echo "") &&hyperfine   --sort command   --runs 1   --export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json"   --parameter-list COMMIT ${COMMITS/
 97/ /,}   --prepare "killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \
 98    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20"   --conclude "killall bitcoind || true; sleep 5;
 99grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'Disabling script verification at block [#1](/bitcoin-bitcoin/1/)' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.
100log; \
101              cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log"   "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -d
102bcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0";
103
104dfde31f2ec Merge bitcoin/bitcoin#33864: scripted-diff: fix leftover references to `policy/fees.h`
10532de0ff1a9 validation: fetch block inputs via CCoinsViewCacheAsync during connection
106
107reindex-chainstate | 921129 blocks | dbcache 450 | i7-hdd | x86_64 | Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz | 8 cores | 62Gi RAM | ext4 | HDD
108
109Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -conne
110ct=0 -printtoconsole=0 (COMMIT = dfde31f2ec1f90976f3ba6b06f2b38a1307c01ab)
111  Time (abs ≡):        42473.571 s               [User: 40584.287 s, System: 3012.074 s]
112
113Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -conne
114ct=0 -printtoconsole=0 (COMMIT = 32de0ff1a97fc2880ba2f507dd00082727badf3f)
115  Time (abs ≡):        34193.205 s               [User: 42326.030 s, System: 2778.267 s]
116
117Relative speed comparison
118        1.24          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -reindex-chainstate -blockson
119ly -connect=0 -printtoconsole=0 (COMMIT = dfde31f2ec1f90976f3ba6b06f2b38a1307c01ab)
120        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -reindex-chainstate -blockson
121ly -connect=0 -printtoconsole=0 (COMMIT = 32de0ff1a97fc2880ba2f507dd00082727badf3f)
122
123
124> i7 @ dbcache=4500:
125COMMITS="d5ed4ba9d8627f1897322ce7eb5b34e08e4f73ac b1a791db1c75a47569b690baf7b074b78e08ca5a"; STOP=921129; DBCACHE=4500; CC=gcc; CXX=g++; BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; (echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done) && (echo "" && echo "reindex-chainstate | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(df -T $BASE_DIR | awk 'NR==2{print $2}') | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 0 && echo SSD || echo HDD)"; echo "") &&hyperfine   --sort command   --runs 1   --export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json"   --parameter-list COMMIT ${COMMITS// /,}   --prepare "killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
126cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \
127./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20"   --conclude "killall bitcoind || true; sleep 5; grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'Disabling script verification at block [#1](/bitcoin-bitcoin/1/)' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log; \
128cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log"   "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0";
129
130d5ed4ba9d8 Merge bitcoin/bitcoin#33906: depends: Add patch for Windows11Style plugin
131b1a791db1c validation: fetch block inputs via CCoinsViewCacheAsync during connection
132
133reindex-chainstate | 921129 blocks | dbcache 4500 | i7-hdd | x86_64 | Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz | 8 cores | 62Gi RAM | ext4 | HDD
134
135Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = d5ed4ba9d8627f1897322ce7eb5b34e08e4f73ac)
136  Time (abs ≡):        27190.152 s               [User: 33685.961 s, System: 1842.096 s]
137
138Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = b1a791db1c75a47569b690baf7b074b78e08ca5a)
139  Time (abs ≡):        23873.513 s               [User: 27793.779 s, System: 1036.030 s]
140
141Relative speed comparison
142        1.14          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = d5ed4ba9d8627f1897322ce7eb5b34e08e4f73ac)
143        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = b1a791db1c75a47569b690baf7b074b78e08ca5a)
144
145
146> i7 @ dbcache=45000:
147COMMITS="dfde31f2ec1f90976f3ba6b06f2b38a1307c01ab e86d48527122c803a58d7bfecffd43a0e373c756"; \
148STOP=921129; DBCACHE=45000; \
149CC=gcc; CXX=g++; \
150BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
151(echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done) && \
152(echo "" && echo "reindex-chainstate | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(df -T $BASE_DIR | awk 'NR==2{print $2}') | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 0 && echo SSD || echo HDD)"; echo "") &&\
153hyperfine \
154  --sort command \
155  --runs 2 \
156  --export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \
157  --parameter-list COMMIT ${COMMITS// /,} \
158  --prepare "killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
159    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \
160    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20" \
161  --conclude "killall bitcoind || true; sleep 5; grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'Disabling script verification at block [#1](/bitcoin-bitcoin/1/)' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log; \
162              cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
163  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0";
164
165dfde31f2ec Merge bitcoin/bitcoin#33864: scripted-diff: fix leftover references to `policy/fees.h`
166e86d485271 validation: fetch block inputs via CCoinsViewCacheAsync during connection
167
168reindex-chainstate | 921129 blocks | dbcache 45000 | i7-hdd | x86_64 | Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz | 8 cores | 62Gi RAM | ext4 | HDD
169
170Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=45000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = dfde31f2ec1f90976f3ba6b06f2b38a1307c01ab)
171  Time (mean ± σ):     22133.846 s ± 42.629 s    [User: 24498.825 s, System: 634.139 s]
172  Range (min … max):   22103.703 s … 22163.990 s    2 runs
173
174Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=45000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = e86d48527122c803a58d7bfecffd43a0e373c756)
175  Time (mean ± σ):     20547.518 s ±  8.809 s    [User: 25074.730 s, System: 695.076 s]
176  Range (min … max):   20541.289 s … 20553.747 s    2 runs
177
178Relative speed comparison
179        1.08 ±  0.00  COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=45000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = dfde31f2ec1f90976f3ba6b06f2b38a1307c01ab)
180        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=45000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = e86d48527122c803a58d7bfecffd43a0e373c756)
181
182
183> rpi5 @ dbcache=450:
184COMMITS="dfde31f2ec1f90976f3ba6b06f2b38a1307c01ab e86d48527122c803a58d7bfecffd43a0e373c756"; \
185STOP=921129; DBCACHE=450; \
186CC=gcc; CXX=g++; \
187BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
188(echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done) && \
189(echo "" && echo "reindex-chainstate | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(df -T $BASE_DIR | awk 'NR==2{print $2}') | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 0 && echo SSD || echo HDD)"; echo "") &&\
190hyperfine \
191  --sort command \
192  --runs 2 \
193  --export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \
194  --parameter-list COMMIT ${COMMITS// /,} \
195  --prepare "killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
196    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \
197    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20" \
198  --conclude "killall bitcoind || true; sleep 5; grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'Disabling script verification at block [#1](/bitcoin-bitcoin/1/)' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log; \
199              cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
200  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0";
201
202dfde31f2ec Merge bitcoin/bitcoin#33864: scripted-diff: fix leftover references to `policy/fees.h`
203e86d485271 validation: fetch block inputs via CCoinsViewCacheAsync during connection
204
205reindex-chainstate | 921129 blocks | dbcache 450 | rpi5-16-1 | aarch64 | Cortex-A76 | 4 cores | 15Gi RAM | ext4 | SSD
206
207Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = dfde31f2ec1f90976f3ba6b06f2b38a1307c01ab)
208  Time (mean ± σ):     41084.236 s ± 701.352 s    [User: 68642.573 s, System: 7256.334 s]
209  Range (min … max):   40588.305 s … 41580.166 s    2 runs
210
211Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = e86d48527122c803a58d7bfecffd43a0e373c756)
212  Time (mean ± σ):     28200.555 s ± 297.983 s    [User: 66305.959 s, System: 5678.536 s]
213  Range (min … max):   27989.850 s … 28411.261 s    2 runs
214
215Relative speed comparison
216        1.46 ±  0.03  COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = dfde31f2ec1f90976f3ba6b06f2b38a1307c01ab)
217        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=450 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = e86d48527122c803a58d7bfecffd43a0e373c756)
218
219
220> rpi5 @ dbcache=4500:
221COMMITS="dfde31f2ec1f90976f3ba6b06f2b38a1307c01ab e86d48527122c803a58d7bfecffd43a0e373c756"; STOP=921129; DBCACHE=4500; CC14:18:21 [10/1679]
222E_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; (echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exi
223t 1; done) && (echo "" && echo "reindex-chainstate | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs)
224| $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(df -T $BASE_DIR | awk 'NR==2{print $2}') | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 0 &&
225 echo SSD || echo HDD)"; echo "") &&hyperfine   --sort command   --runs 1   --export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.jso
226n"   --parameter-list COMMIT ${COMMITS// /,}   --prepare "killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
227
228    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \
229    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20"   --conclude "killall bitcoind || true; sleep 5; grep -q 'height=0' $DATA
230_DIR/debug.log && grep -q 'Disabling script verification at block [#1](/bitcoin-bitcoin/1/)' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log; \
231              cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log"   "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0";
232root@rpi5-16-1:/mnt/my_storage/bitcoin# COMMITS="dfde31f2ec1f90976f3ba6b06f2b38a1307c01ab e86d48527122c803a58d7bfecffd43a0e373c756"; STOP=921129; DBCACHE=4500; CC=gcc; CXX=g++; BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; (echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done) && (echo "" && echo "reindex-chainstate | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(df -T $BASE_DIR | awk 'NR==2{print $2}') | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 0 && echo SSD || echo HDD)"; echo "") &&hyperfine   --sort command   --runs 1   --export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json"   --parameter-list COMMIT ${COMMITS// /,}   --prepare "killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
233    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \
234    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20"   --conclude "killall bitcoind || true; sleep 5; grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'Disabling script verification at block [#1](/bitcoin-bitcoin/1/)' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log; \
235              cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log"   "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0";
236
237dfde31f2ec Merge bitcoin/bitcoin#33864: scripted-diff: fix leftover references to `policy/fees.h`
238e86d485271 validation: fetch block inputs via CCoinsViewCacheAsync during connection
239
240reindex-chainstate | 921129 blocks | dbcache 4500 | rpi5-16-1 | aarch64 | Cortex-A76 | 4 cores | 15Gi RAM | ext4 | SSD
241
242Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = dfde31f2ec1f90976f3ba6b06f2b38a1307c01ab)
243  Time (abs ≡):        35867.389 s               [User: 41695.508 s, System: 3281.868 s]
244
245Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = e86d48527122c803a58d7bfecffd43a0e373c756)
246  Time (abs ≡):        26440.662 s               [User: 43495.688 s, System: 3743.419 s]
247
248Relative speed comparison
249        1.36          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = dfde31f2ec1f90976f3ba6b06f2b38a1307c01ab)
250        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=921129 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = e86d48527122c803a58d7bfecffd43a0e373c756)

andrewtoth renamed this:
~~validation: fetch block inputs on parallel threads >20% faster IBD~~
validation: fetch block inputs on parallel threads >40% faster IBD
on Nov 27, 2025

validation: fetch block inputs on parallel threads >40% faster IBD #31132

Problem

Solution

Implementation Details

Safety and Correctness

Performance

Credits

Code Coverage & Benchmarks

Reviews

Conflicts

Cache hierarchy

Sorted fetch

Separate ThreadPool