Significantly improves loadtxoutset performance via a number of changes:
Also adds a subcommand to bitcoin-util for getting a snapshot's hash_serialized_3, muhash, coin count and total amount values without importing to leveldb first.
On my systems, performance ends up bottlenecked by leveldb CPU usage.
<!--e57a25ab6845829454e8d69fc972939a-->
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
<!--006a51241073e994b41acfe9ec718e94-->
<!--021abf342d371248e50ceaed478a90ca-->
See the guideline and AI policy for information on the review process. A summary of reviews will appear here.
<!--5faf32d7da4f0f540f40219e4f7537a3-->
The performance change I see on a Ryzen 9 3900X system with RAID HDD storage is ~11minutes with to ~2m for a load, consisting of 1m40s of CPU time followed by 20s of disk flushing; with SSD and a slightly faster CPU, total time is reduced from ~5m30s to ~1m (no significant flush delay after the CPU work is completed here). These numbers are with the block 965000 utxo set.
<!--85328a0da195eb286784d51f73fa0af9-->
🚧 At least one of the CI tasks failed.
<sub>Task tidy: https://github.com/bitcoin/bitcoin/actions/runs/39762952091/job/119771350806</sub>
<sub>LLM reason (✨ experimental): CI failed because clang-tidy flagged a mismatched argument comment (moveout instead of moveto) in test/coinsviewoverlay_tests.cpp; warnings are treated as errors.</sub>
<details><summary>Hints</summary>
Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:
Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.
Leave a comment here, if you need help tracking down a confusing failure.
</details>
The snapshot file format is not node state: reading a snapshot needs no
chainstate, so move it out. This prepares the way for adding snapshot
parsing to bitcoin-util in a future commit.
Mostly move-only. The commit can be reviewed with the
--color-moved=dimmed-zebra option for an easier review.
SnapshotMetadata took the expected network in its constructor and compared
it while parsing, throwing on mismatch. Split that check out as a separate
`CheckNetwork` method, so that callers have the option to read snapshots
for arbitrary networks.
Also replace the per-object `const std::set` for checking supported
versions with an inline function.
Normally, we want updates to the database to be fsynced to disk so that
the database is in a consistent state even after a crash. This adds an
option to drop that guarantee, in favour of faster bulk writes, relying
on the caller to make other arrangements to ensure the resulting database
is flushed to disk.
This populates a chainstate leveldb directly from byte buffers (generated
from Coin objects via RawCoinDataDBWriter::Batch). This avoids the
overhead of going through a CoinsView map, and allows splitting the Coin
serialization step and the leveldb writes across different threads.
Multi-producer, multi-consumer thread-safe queue with maximum capacity
for handing out jobs to threads.
Allows importing snapshots directly into leveldb (dbwrapper), without
passing through Chainstate or CoinsView first. Supports multithreaded
processing, with one thread reading, one thread hashing, one thread
serializing for the database, and one thread populating the database.
Adds an additional check that the coins in the snapshot are sorted in
txid order.
Nothing calls it yet, so behaviour is unchanged.
ActivateSnapshot and, therefore, the loadtxoutset RPC now use the new
utxo_snapshot.cpp pipeline.
The dump is no longer staged in a CCoinsViewCache and then hashed and
flushed out of it, so `-dbcache` worth of cached coins during a mainnet
load are gone along with the flush cycles. In addition, the hashing
of the dump takes place in parallel with the import, rather than after
it completes.
The old loader is left in place until the next commit removes it;
nothing calls it any more.
After the previous two commits, this code is no longer used, so remove it.
This also removes the circular dependency between node/utxo_snapshot
and validation.
Adds a subcommand for quickly generating the summary of a serialized
utxo set for verification purposes (including various hashes, coin count
and value).
Can you please give us more context? Why do you consider an 11-minute load too slow? Is there a reason to think this is what's preventing people from using it - enough to justify yet more complexity in the most "load-bearing" parts of the code? Or is this just a demo to explore the possibilities?
Over the past years we've discussed how relying on someone else's UTXO set makes us uncomfortable (see #32377 and #36598). This change adds even more AssumeUTXO-specific complexity, which is exactly why we're considering removing it completely. I wouldn't object to adding it back later through IPC and the Kernel if there's still demand and doesn't add complexity - but there's no indication of that, especially since #36476 (possibly combined with SwiftSync) would already speed up actual UTXO set construction by an order of magnitude on resource-constrained nodes. Quickly loading someone else's UTXO set could still be a useful academic pursuit, but many of us are not comfortable polluting the code with multiple chainstates, which make consensus-critical paths harder to reason about.