validation: assumeutxo params for testnet and signet

Sjors commented at 2:12 pm on September 21, 2023: member

Based on #27596. This adds snapshot parameters for testnet block 2,500,000 and signet block 160,000.

You can generate the snapshots yourself using ./contrib/devtools/utxo_snapshot.sh or download my torrent:

testnet: magnet:?xt=urn:btih:511e09f4bf853aefab00de5c070b1e031f0ecbe9&dn=utxo-testnet-2500000.dat&tr=udp%3A%2F%2Ftracker.bitcoin.sprovoost.nl%3A6969
signet: magnet:?xt=urn:btih:9da986cb27b3980ea7fd06b21e199b148d486880&dn=utxo-signet-160000.dat&tr=udp%3A%2F%2Ftracker.bitcoin.sprovoost.nl%3A6969

It would be a good idea to test:

That you can produce the same snapshot files, sha256 sums:

0eeeca845385ba91e84ef58c09d38f98f246a24feadaad57fe1e5874f3f92ef8c utxo-signet-160000.dat
179db4b025448cc0ac388d8589a28eab02de53055d181e34eb47391717aa16388 utxo-testnet-2500000.dat

That the snapshots work

Not part of this PR:

I also have a branch with a mainnet snapshot at height 800,000. It can be used for testing, but I’m not proposing it for merge just yet. I think more testing and conceptual discussion should happen first.

branch: 2023/09/assume_mainnet
torrent: magnet:?xt=urn:btih:50ee955bef37f5ec3e5b0df4cf0288af3d715a2e&dn=utxo-800000.dat&tr=udp%3A%2F%2Ftracker.bitcoin.sprovoost.nl%3A6969

net_processing: Request assumeutxo background chain blocks

Add new PeerManagerImpl::TryDownloadingHistoricalBlocks method and use it to
request background chain blocks in addition to blocks normally requested by
FindNextBlocksToDownload.

Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
Co-authored-by: James O'Beirne <james.obeirne@gmail.com>

a9ea542e8a

bugfix: correct is_snapshot_cs in VerifyDB e9aa43db7e

assumeutxo: remove snapshot during -reindex{-chainstate}

Removing a snapshot chainstate from disk (and memory) is consistent with
existing reindex operations.

00129799d9

chainparams: add blockhash to AssumeutxoData

This allows us to reference assumeutxo configuration by blockhash as
well as height; this is helpful in future changes when we want to
reference assumeutxo configurations before the block index is loaded.

4ac7dd25c3

validation: MaybeRebalanceCaches when chain leaves IBD

Check to see if we need to rebalance caches across chainstates when
a chain leaves IBD.

3661b4b085

validation: add ChainstateRole 6d5aca2340

validation: only call UpdatedBlockTip for active chainstate

This notification isn't needed for background chainstates.

`kernel::Notifications::blockTip` are also skipped.

03a348c64f

validation: pass ChainstateRole for validationinterface calls

This allows consumers to decide how to handle events from background or
assumedvalid chainstates.

ac53194df7

validationinterface: only send zmq notifications for active b2117a1fa9

wallet: validationinterface: only handle active chain notifications e2c5c5dfdb

net_processing: validationinterface: ignore some events for bg chain 40d5ca3082

validation: indexing changes for assumeutxo

When using an assumedvalid chainstate, only process validationinterface
callbacks from the background chainstate within indexes. This ensures
that all indexes are built in-order.

Later, we can possibly designate indexes which can be built out of order
and continue their operation during snapshot use.

Once the background sync has completed, restart the indexes so that
they continue to index the now-validated snapshot chainstate.

e144e6b811

validation: pruning for multiple chainstates

Introduces ChainstateManager::GetPruneRange().

The prune budget is split evenly between the number of chainstates,
however the prune budget may be exceeded if the resulting shares are
beneath `MIN_DISK_SPACE_FOR_BLOCK_FILES`.

8cfd27186d

test: adjust chainstate tests to use recognized snapshot base

In future commits, loading the block index while making use of a
snapshot is contingent on the snapshot being recognized by chainparams.

Ensure all existing unittests that use snapshots use a recognized
snapshot (at height 110).

Co-authored-by: Ryan Ofsky <ryan@ofsky.org>

14fcb1b1ee

validation: populate nChainTx value for assumedvalid chainstates

Use the expected AssumeutxoData in order to bootstrap nChainTx values
for assumedvalid blockindex entries in the snapshot chainstate. This
is necessary because nChainTx is normally built up from nTx values,
which are populated using blockdata which the snapshot chainstate
does not yet have.

a7a5debb6a

blockstorage: segment normal/assumedvalid blockfiles

When using an assumedvalid (snapshot) chainstate along with a background
chainstate, we are syncing two very different regions of the chain
simultaneously. If we use the same blockfile space for both of these
syncs, wildly different height blocks will be stored alongside one
another, making pruning ineffective.

This change implements a separate blockfile cursor for the assumedvalid
chainstate when one is in use.

bd770ff590

validation: assumeutxo: swap m_mempool on snapshot activation

Otherwise we will not receive transactions during background sync until
restart.

0f34fc74a7

validation: do not activate snapshot if behind active chain

Most easily reviewed with

  git show --color-moved=dimmed-zebra --color-moved-ws=ignore-all-space

Co-authored-by: Ryan Ofsky <ryan@ofsky.org>

126d7171a7

rpc: add loadtxoutset

Co-authored-by: Sebastian Falbesoner <sebastian.falbesoner@gmail.com>

4193ef8fe1

refuse to activate a UTXO snapshot if mempool not empty

This ensures that we avoid any unexpected conditions inherent in
transferring non-empty mempools across chainstates.

Note that this should never happen in practice given that snapshot
activation will not occur outside of IBD, based upon the height checks
in `loadtxoutset`.

89bbb2221d

rpc: add getchainstates

Co-authored-by: Ryan Ofsky <ryan@ofsky.org>

fb87f83c34

test: add feature_assumeutxo functional test

Most ideas for test improvements (TODOs) provided by Russ Yanofsky.

afc909337b

contrib: add script to demo/test assumeutxo

Add the script to the shellcheck exception list since the
quoted variables rule needs to be violated in order to get
bitcoind to pick up on $CHAIN_HACK_FLAGS.

ef55d7b2e6

doc: add note about confusing HaveTxsDownloaded name 32b797f37e

chainparams: add testnet assumeutxo param at height 2_500_000 d55cc7fd53

chainparams: add signet assumeutxo param at height 160_000 d5f83a7181

DrahtBot commented at 2:12 pm on September 21, 2023: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Reviews

See the guideline for information on the review process. A summary of reviews will appear here.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#28550 (Covenant tools softfork by jamesob)
#28453 (wallet: Receive silent payment transactions by achow101)
#28368 (Fee Estimator updates from Validation Interface/CScheduler thread by ismaelsadeeq)
#28120 (p2p: make block download logic aware of limited peers threshold by furszy)
#28031 (Package Relay 1/3: Introduce TxDownloadManager and improve orphan-handling by glozow)
#27866 (blockstorage: Return on fatal flush errors by TheCharlatan)
#27837 (net: introduce block tracker to retry to download blocks after failure by furszy)
#27770 (Introduce ‘getblockfileinfo’ RPC command by furszy)
#27039 (blockstorage: do not flush block to disk if it is already there by pinheadmz)
#26966 (index: blockfilter initial sync speedup, parallelize process by furszy)
#26762 (bugfix: Make CCheckQueue RAII-styled (attempt 2) by hebasto)
#26045 (rpc: Optimize serialization disk space of dumptxoutset by aureleoules)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

DrahtBot added the label Validation on Sep 21, 2023

DrahtBot added the label Needs rebase on Sep 29, 2023

DrahtBot commented at 6:53 pm on September 29, 2023: contributor

🐙 This pull request conflicts with the target branch and needs rebase.

Sjors commented at 12:53 pm on September 30, 2023: member

Testnet and signet params have been absorbed into #27596.

I opened a draft PR for those who want to test mainnet: #28553

Sjors closed this on Sep 30, 2023

validation: assumeutxo params for testnet and signet #28516

Reviews

Conflicts