assumeutxo (2) #27596

• Background and FAQ: https://github.com/jamesob/assumeutxo-docs/tree/2019-04-proposal/proposal
• Prior progress/project: https://github.com/bitcoin/bitcoin/projects/11
• Replaces #15606, which was closed due to Github slowness. Original description and commentary can be found there.

This changeset finishes the first phase of the assumeutxo project. It makes UTXO snapshots loadable via RPC (loadtxoutset) and adds assumeutxo parameters to chainparams. It contains all the remaining changes necessary to both use an assumedvalid snapshot chainstate and do a full validation sync in the background.

This may look like a lot to review, but note that

• ~200 lines are a (non-essential) demo shell script
• Many lines are functional test, documentation, and relatively dilute RPC code.

So it shouldn’t be as burdensome to review as the linecount might suggest.

• P2P: minor changes are made to init.cpp and net_processing.cpp to make simultaneous IBD across multiple chainstates work.
• Pruning: implement correct pruning behavior when using a background chainstate
• Blockfile separation: to prevent “fragmentation” in blockfile storage, have background chainstates use separate blockfiles from active snapshot chainstates to avoid interleaving heights and impairing pruning.
• Indexing: some CValidationInterface events are given with an additional parameter, ChainstateRole, and all indexers ignore events from ChainstateRole::ASSUMEDVALID so that indexation only happens sequentially.
• Have -reindex properly wipe snapshot chainstates.
• RPC: introduce RPC commands loadtxoutset and (hidden) getchainstates.
• Release docs & first assumeutxo commitment: add notes and a particular assumeutxo hash value for first AU-enabled release.
  • This will complete the project and allow use of UTXO snapshots for faster node bootstrap.

The next phase, if it were to be pursued, would be coming up with a way to distribute the UTXO snapshots over the P2P network.

UTXO snapshots

Create your own with ./contrib/devtools/utxo_snapshot.sh, e.g.

0./contrib/devtools/utxo_snapshot.sh 788000 utxo.dat ./src/bitcoin-cli -datadir=$(pwd)/testdata`)

or use the pre-generated ones listed below.

• Testnet: 2'500'000 (Sjors):
  • torrent: magnet:?xt=urn:btih:511e09f4bf853aefab00de5c070b1e031f0ecbe9&dn=utxo-testnet-2500000.dat&tr=udp%3A%2F%2Ftracker.bitcoin.sprovoost.nl%3A6969
  • sha256: 79db4b025448cc0ac388d8589a28eab02de53055d181e34eb47391717aa16388
• Signet: 160'000 (Sjors):
  • torrent: magnet:?xt=urn:btih:9da986cb27b3980ea7fd06b21e199b148d486880&dn=utxo-signet-160000.dat&tr=udp%3A%2F%2Ftracker.bitcoin.sprovoost.nl%3A6969
  • sha256: eeeca845385ba91e84ef58c09d38f98f246a24feadaad57fe1e5874f3f92ef8c
• Mainnet: 800'000 (Sjors):
  • Note: this needs the following commit cherry-picked in: https://github.com/Sjors/bitcoin/commit/24deb2022b822f22fba9fcbee201e37a83225eb2
  • torrent: magnet:?xt=urn:btih:50ee955bef37f5ec3e5b0df4cf0288af3d715a2e&dn=utxo-800000.dat&tr=udp%3A%2F%2Ftracker.bitcoin.sprovoost.nl%3A6969

Testing

For fun (~5min)

If you want to do a quick test, you can run ./contrib/devtools/test_utxo_snapshots.sh and follow the instructions. This is mostly obviated by the functional tests, though.

For real (longer)

If you’d like to experience a real usage of assumeutxo, you can do that too. I’ve cut a new snapshot at height 788'000 (http://img.jameso.be/utxo-788000.dat - but you can do it yourself with ./contrib/devtools/utxo_snapshot.sh if you want). Download that, and then create a datadir for testing:

 0$ cd ~/src/bitcoin  # or whatever
 1
 2# get the snapshot
 3$ curl http://img.jameso.be/utxo-788000.dat > utxo-788000.dat
 4
 5# you'll want to do this if you like copy/pasting 
 6$ export AU_DATADIR=/home/${USER}/au-test # or wherever
 7
 8$ mkdir ${AU_DATADIR}
 9$ vim ${AU_DATADIR}/bitcoin.conf
10
11dbcache=8000  # or, you know, something high
12blockfilterindex=1
13coinstatsindex=1
14prune=3000
15logthreadnames=1

Obtain this branch, build it, and then start bitcoind:

0$ git remote add jamesob https://github.com/jamesob/bitcoin
1$ git fetch jamesob assumeutxo
2$ git checkout jamesob/assumeutxo
3
4$ ./configure $conf_args && make  # (whatever you like to do here)
5
6# start 'er up and watch the logs
7$ ./src/bitcoind -datadir=${AU_DATADIR}

Then, in some other window, load the snapshot

0$ ./src/bitcoin-cli -datadir=${AU_DATADIR} loadtxoutset $(pwd)/utxo-788000.dat

You’ll see some log messages about headers retrieval and waiting to see the snapshot in the headers chain. Once you get the full headers chain, you’ll spend a decent amount of time (~10min) loading the snapshot, checking it, and flushing it to disk. After all that happens, you should be syncing to tip in pretty short order, and you’ll see the occasional [background validation] log message go by.

In yet another window, you can check out chainstate status with

0$ ./src/bitcoin-cli -datadir=${AU_DATADIR} getchainstates

as well as usual favorites like getblockchaininfo.

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Reviews

See the guideline for information on the review process.

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

• #28550 (Covenant tools softfork by jamesob)
• #28453 (wallet: Receive silent payment transactions by achow101)
• #28368 (Fee Estimator updates from Validation Interface/CScheduler thread by ismaelsadeeq)
• #28120 (p2p: make block download logic aware of limited peers threshold by furszy)
• #28031 (Package Relay 1/3: Introduce TxDownloadManager and improve orphan-handling by glozow)
• #27837 (net: introduce block tracker to retry to download blocks after failure by furszy)
• #27770 (Introduce ‘getblockfileinfo’ RPC command by furszy)
• #26966 (index: blockfilter initial sync speedup, parallelize process by furszy)
• #26762 (bugfix: Make CCheckQueue RAII-styled (attempt 2) by hebasto)
• #26045 (rpc: Optimize serialization disk space of dumptxoutset by aureleoules)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

CI’s passing after a silent conflict in the rebase. I’ve added a link to @Sjors’ snapshot torrent in the PR description.

No known outstanding issues here; ready for testing!

What’s your feeling on how important figuring out the fix for the pruning issue is at the moment? People don’t consider it blocking #24008, right?

Right, pruning issues shouldn’t block #24008 - those changes need to happen regardless of how we manage pruning with snapshots.

I’ll reproduce/investigate the pruning issues over the coming days. But it’s worth noting that @Sjors didn’t see any regressions in pruning on this branch before loading the snapshot, meaning the degraded pruning behavior only kicks in if the user loads a snapshot. And even though, the code just doesn’t prune as aggressively as it seems it should.

Hm, for what it’s worth, pruning is working as expected for me on this branch.

 0Every 2.0s: cat <(./src/bitcoin-cli -datadir=/home/james/tmp/bitcoin-au-testing getmempoolinfo) <( ./src/bitcoin-cli -datadir=/home/james/tmp/bitcoin-au-testing getchainstates)                                                                                                                                                                         fido: Wed May 10 11:16:49 2023
 1
 2{
 3  "loaded": true,
 4  "size": 108146,
 5  "bytes": 42192499,
 6  "usage": 234882016,
 7  "total_fee": 23.04459003,
 8  "maxmempool": 300000000,
 9  "mempoolminfee": 0.00013381,
10  "minrelaytxfee": 0.00001000,
11  "incrementalrelayfee": 0.00001000,
12  "unbroadcastcount": 0,
13  "fullrbf": false
14}
15{
16  "active_chain_type": "validated_snapshot",
17  "validated_snapshot": {
18    "blocks": 789094,
19    "bestblockhash": "00000000000000000002dad6b3cd82fc00075978eb0d63dcbf6bce3bdc2c3f32",
20    "difficulty": 48005534313578.78,
21    "verificationprogress": 0.9999931560421257,
22    "snapshot_blockhash": "00000000000000000001f3fa1b4c03c877740778f56b0d5456b18dd88f7f695e",
23    "initialblockdownload": false,
24    "coins_db_cache_bytes": 8388608,
25    "coins_tip_cache_bytes": 11498684416,
26    "has_mempool": true
27  },
28  "headers": 789094
29}

Will attempt to repro with a fresh run, I guess?

I’ve done a number of runs here with some combination of indexing and pruning, and I think this is in about as good a shape as it can be, with the following caveat:

When using assumeutxo with pruning and indexing, the snapshot chainstate cannot be pruned because of how we handle indexing during bg sync. Since the snapshot chainstate isn’t indexed until after the background sync completes (in order to avoid out-of-order indexation), we aren’t able to prune the snapshot chainstate based upon the prune blocks that the indices configure.

So the upshot of all this is that pruning, when used with assumeutxo and indexing, is basically best-effort. I think that’s an okay state as long as we’re clear about it to end users. Probably merits some kind of GUI alert which I think can be done later.

After bg sync completes, we’re in a pretty good state (-blockfilterindex + -coinstatsindex + -prune=550):

 0777M    /home/james/tmp/bitcoin-au-testing/blocks
 1total 667M
 2drwx------ 3 james users 4.0K May 12 07:02 .
 3drwx------ 5 james users 4.0K May 12 10:34 ..
 4-rw------- 1 james users 128M May 11 21:24 blk00014.dat
 5-rw------- 1 james users 128M May 11 21:24 blk00015.dat
 6-rw------- 1 james users 127M May 11 21:24 blk00016.dat
 7-rw------- 1 james users 128M May 12 05:04 blk00017.dat
 8-rw------- 1 james users  32M May 12 06:41 blk02910.dat
 9-rw------- 1 james users  48M May 12 10:31 blk03593.dat
10drwx------ 2 james users 4.0K May 12 10:34 index
11-rw------- 1 james users  18M May 11 21:24 rev00014.dat
12-rw------- 1 james users  18M May 11 21:24 rev00015.dat
13-rw------- 1 james users  18M May 11 21:24 rev00016.dat
14-rw------- 1 james users  16M May 12 05:04 rev00017.dat
15-rw------- 1 james users 3.3M May 12 06:41 rev02910.dat
16-rw------- 1 james users 6.0M May 12 10:31 rev03593.dat

Through the last few days of testing, I did find that I forgot to the divide the prune target by the number of chainstates outstanding, so that may have created some of the weirdness @Sjors was seeing. But otherwise I just think that enabling all these features together results in the necessary trade-off of blowing out the pruning budget.

Also worth noting that based on the current size of blocks over the last few weeks, the 244 block trailing prune window is potentially going to result in over 550MB of block data, which is our lowest allowable target.

I started testing this by creating / loading a snapshot on signet at height 100016. I haven’t really looked at the code yet and also didn’t read the provided docs so that I could do things by trial and error, and hopefully get to the desired result anyway but have a better chance to encounter some edge cases / bugs along the way.

Things worked really well! Just some observations / minor issues:

• Didn’t encounter any problems with indexes, pruning, -reindex or reindex-chainstate
• In one run, I successfully loaded the chainstate from disk, and then restarted bitcoind (without loading any additional blocks via p2p). Then, init would fail the verification checks with the next startup because the blocks aren’t there: (Verification error: ReadBlockFromDisk failed at 100016, hash=000001301bada810147463d5b1d72caf0807ad1fdaf015d084f5c474b8cc44c6)
• I am allowed to do invalidateblock on a block smaller than the assumeUTXO block before the background state has been loaded and corrupt my block database that way. That’s obviously not an issue in practice, so probably no need to fix - just wanted to mention it.
• could add an empty m_assumeutxo_data entry in chainparams for signet, like testnet has
• in the getchainstates rpc, the field verificationprogress has some numerical fluctuations - maybe round this to some reasonable precision
• after the background chainstate has been fully validated, I observed some warnings in the log while running with indexes, not sure if this is something serious: 2023-05-12T22:06:22Z ChainStateFlushed: WARNING: Locator contains block (hash=000001301bada810147463d5b1d72caf0807ad1fdaf015d084f5c474b8cc44c6) not on known best chain (tip=000000f04acd86089bf33802cfa46e31fa9f99918c32d5a4e7535ae7de57f805); not writing index locator

Here’s a signet snapshot torrent: magnet:?xt=urn:btih:0e1a61b4ed9a57c41d98f1ddd5061bc088c857b6&dn=utxo-signet-140000.dat&tr=udp%3A%2F%2Ftracker.bitcoin.sprovoost.nl%3A6969

Can be added to kernel/chainparams.cpp with:

0m_assumeutxo_data = MapAssumeutxo{
1            {
2                140'000,
3                {AssumeutxoHash{uint256S("0x91412d7018c8564beef1938eef2bfe18feee53674fd26bea425daa4280437641")}, 2233882},
4            },
5        };

This basically gets you to the signet tip faster than you can hit enter :-)

Did the following more or less random test scenario (build at commit b8fd765a06998a79ef6efef5355c956a4679df6f) over the last days:

• start up bitcoind -prune=800 -blockfilterindex=1 (-dbcache intentionally left at default)
• load snapshot from height 788000 via bitcoind loadtxoutset /path/to/utxo-788000.dat (thanks for providing the file!)
• wait until snapshot chainstate reaches network tip
• shut down the node and wait 1-2 days (with the idea to again provoke both chainstates being in IBD again)
• restart bitcoind -prune=800 -blockfilterindex=1
• wait until snapshot chainstate reaches network tip
• watch background validation proceed

So far so good, everything went quite smooth and nothing unexpected happened! The only disappointment was finding that apparently the provided prune target of 800MB was not respected:

0$ du -h ./datadir/blocks/ 
1116M    ./datadir/blocks/index
24.8G    ./datadir/blocks/

(background validation was at block ~230000 at that point)

Looking at the block files (being in ranges blk00001-blk00030 and blk00084-blk00086), it seems that in my case the prune target is respected only for the background chainstate, but the block files blk00001.dat-blk00030.dat (which were obviously created in the course of snapshot chainstate validation, catching up to the network tip for the second time, looking at the timestamp) are just sitting there and didn’t get pruned anymore for some reason. Any idea what’s going on here?

Thanks for testing, @theStack!

So far so good, everything went quite smooth and nothing unexpected happened! The only disappointment was finding that apparently the provided prune target of 800MB was not respected:

Yes, this is unfortunately expected behavior. Since you have -blockfilterindex enabled and the background sync hasn’t yet completed, prune locks are preventing any pruning from happening on the snapshot chainstate. I mentioned this obliquely here: #27596 (comment)

The only way around this would be to mark indexes which can be built out of order (i.e. everything except coinstats) and only halt snapshot indexation for those that can’t. But this is something that could be done after the initial assumeutxo feature.

So far so good, everything went quite smooth and nothing unexpected happened! The only disappointment was finding that apparently the provided prune target of 800MB was not respected:

Yes, this is unfortunately expected behavior. Since you have -blockfilterindex enabled and the background sync hasn’t yet completed, prune locks are preventing any pruning from happening on the snapshot chainstate. I mentioned this obliquely here: #27596 (comment) @jamesob: Thanks for clarifying (admittedly I didn’t read your comment just a few posts above before posting my test results), it makes sense that we have to keep the blocks from the snapshot chainstate for the indexer at least until background sync completes. I’d agree that this is not a problem as long as it is clearly communicated to the user, e.g. by putting a well-visible warning into the loadtxoutset RPC help. The ultra-conservative approach here would be to simply throw an error if a snapshot is loaded on a node that was started with both pruning and an index enabled (if keeping the -prune target promise has absolute highest priority), but that seems to be too limiting.

Cross-linking to some discussion on how we load the block index that @sdaftuar has surfaced: https://github.com/bitcoin/bitcoin/pull/23174/files#r1198450499

If anyone has encountered an issue like this while testing this branch, a comment would be welcome.

New bug, albeit pretty minor:

• if you restart the node after activating the snapshot but before connecting any blocks to the snapshot chainstate, ReadBlockFromDisk checks fail when calling VerifyDB() during chainstate initialization. I’m not sure when this would happen in practice, but it’s an annoying edge case.

Edit: fixed this.

Rebased.

I’ve made a few changes/improvements that I think fixes the bug @sdaftuar found (not adding all potential tips to setBlockIndexCandidates), albeit I’m not yet integrating the changes in #27746.

• More comprehensive functional tests: the tests now stop and start at more points throughout the sync process, including immediately after snapshot load and before bg validation completes.
• We now include the blockhash of the snapshot base in m_assumeutxo_data; this allows us to move the nChainTx reconstruction where it belongs, into BlockManager::LoadBlockIndex().
• All blocks for which we have data (or are assumedvalid) are added to all chainstates’ setBlockIndexCandidates.
  • Some related changes in CheckBlockIndex() needed to be made to update some outdated assertions.

I’m going to be taking a close look at @sdaftuar’s proposed changes over the next few days to see if I can integrate them here.

All blocks for which we have data (or are assumedvalid) are added to all chainstates’ setBlockIndexCandidates.

Being assumedvalid is unrelated to whether we should add an entry to setBlockIndexCandidates; setBlockIndexCandidates should have the current tip and any block for which we HAVE_DATA and for which we HAVE_DATA for all the blocks that we’d need to connect, in order to get from our current tip to that target block.

The assumedvalid bit is really just something we’re using to help with adjusting the invariants in CheckBlockIndex(); it should have no bearing on any of the actual validation logic outside of that.

Rebased

Also fixed the minor issues with the test_utxo_snapshots.sh demo script.

Since last push, I’ve completed a full test of the mainnet snapshot; after a few days having finished the background sync, -prune=550 (and everything else) working as expected.

 0759M    /home/james/tmp/bitcoin-au-testing/blocks
 1total 650M
 2drwx------ 3 james users 4.0K May 31 10:54 .
 3drwx------ 5 james users 4.0K May 31 11:09 ..
 4-rw------- 1 james users 127M May 29 15:33 blk03626.dat
 5-rw------- 1 james users 128M May 30 02:38 blk03627.dat
 6-rw------- 1 james users 128M May 30 17:29 blk03628.dat
 7-rw------- 1 james users 128M May 31 06:05 blk03629.dat
 8-rw------- 1 james users  64M May 31 10:57 blk03630.dat
 9drwx------ 2 james users 4.0K May 31 10:54 index
10-rw------- 1 james users  17M May 29 15:30 rev03626.dat
11-rw------- 1 james users  15M May 30 02:36 rev03627.dat
12-rw------- 1 james users  19M May 30 17:23 rev03628.dat
13-rw------- 1 james users  18M May 31 06:01 rev03629.dat
14-rw------- 1 james users 8.0M May 31 10:57 rev03630.dat

This may look like a lot to review, but note that

* 200 lines are duplicated in [#24008](/bitcoin-bitcoin/24008/)

This should refer to #27746 now, shouldn’t it?

While testing.

Once you get the full headers chain, you’ll spend a decent amount of time (~10min) loading the snapshot, checking it, and flushing it to disk

FlushSnapshotToDisk: saving snapshot chain state (10588 MB) started takes about 5 hours to complete on pop-OS 22. During the FlushSnapshotToDisk process, I believe there should be some progress indication, similar to what occurs during the snapshot loading process. Currently, it appears to be like it is stuck.

When I attempted to shut down bitcoind while FlushSnapshotToDisk was in progress, it resulted in some sort of a deadlock.

Below are the logs after I stop bitcoind while saving the snapshot.

02023-06-27T17:45:52Z [httpworker.0] [snapshot] 90000000 coins loaded (99.72%, 10561 MB)
12023-06-27T17:45:52Z [httpworker.0] [snapshot] loaded 90249783 (10588 MB) coins from snapshot 00000000000000000001f3fa1b4c03c877740778f56b0d5456b18dd88f7f695e
22023-06-27T17:45:52Z [httpworker.0] FlushSnapshotToDisk: saving snapshot chainstate (10588 MB) started
32023-06-27T18:00:33Z [httpworker.1] [snapshot] waiting to see blockheader 00000000000000000001f3fa1b4c03c877740778f56b0d5456b18dd88f7f695e in headers chain before snapshot activation
42023-06-27T18:15:54Z [init] tor: Thread interrupt
52023-06-27T18:15:54Z [addcon] addcon thread exit
62023-06-27T18:15:54Z [torcontrol] torcontrol thread exit
72023-06-27T18:15:54Z [init] Shutdown: In progress...
82023-06-27T18:15:54Z [net] net thread exit

Tried to pkill bitcoind but it’s still running possibly waiting for the flush to finish.

Tested ACK built with and tested on WSL Ubuntu 22.04

you’ll spend a decent amount of time (~10min) loading the snapshot

this took closer to 30 min with an Intel(R) Core(TM) i9-10980HK CPU @ 2.40GHz 3.10 GHz

getchainstates returns:

 0{
 1  "active_chain_type": "snapshot",
 2  "ibd": {
 3    "blocks": 896,
 4    "bestblockhash": "00000000dabe35441efb615ad072e22ece58326cf0a43b990fc00aab808e8589",
 5    "difficulty": 1,
 6    "verificationprogress": 1.076164026301743e-06,
 7    "snapshot_blockhash": "0000000000000000000000000000000000000000000000000000000000000000",
 8    "initialblockdownload": true,
 9    "coins_db_cache_bytes": 419430,
10    "coins_tip_cache_bytes": 366490419,
11    "has_mempool": false
12  },
13  "snapshot": {
14    "blocks": 789045,
15    "bestblockhash": "000000000000000000030939094d57c25369fbed9b6bafe94896de37f47d77f5",
16    "difficulty": 48005534313578.78,
17    "verificationprogress": 0.9786754966337523,
18    "snapshot_blockhash": "00000000000000000001f3fa1b4c03c877740778f56b0d5456b18dd88f7f695e",
19    "initialblockdownload": true,
20    "coins_db_cache_bytes": 7969177,
21    "coins_tip_cache_bytes": 6963317964,
22    "has_mempool": true
23  },
24  "headers": 797452
25}

And getblockchaininfo returns:

 0{
 1  "chain": "main",
 2  "blocks": 789140,
 3  "headers": 797452,
 4  "bestblockhash": "000000000000000000049973648ea7a15bf2c7f3585eb8700f889e9aca5af73c",
 5  "difficulty": 48005534313578.78,
 6  "time": 1683751586,
 7  "mediantime": 1683748352,
 8  "verificationprogress": 0.9788762180334918,
 9  "initialblockdownload": true,
10  "chainwork": "0000000000000000000000000000000000000000481500309c4ab2d789f587f6",
11  "size_on_disk": 2225580771,
12  "pruned": true,
13  "pruneheight": 788001,
14  "automatic_pruning": true,
15  "prune_target_size": 3145728000,
16  "warnings": "This is a pre-release test build - use at your own risk - do not use for mining or merchant applications"
17}

Rebased au.027 -> au.029

The merge of #27607 really made that difficult.

This rebase fixes merge conflicts as well as incorporates @sdaftuar’s changes from #27746. His net changes are in the first commit here.

I’m very tired of this project and hope we can focus effort on getting this changeset reviewed and merged to be done with the development of this feature.

Test failure for a66790c662d6516b9fec9ca2cc97e410a72fe899 on Ubuntu 23.04 (gcc 12.3.0)

 0$ src/test/test_bitcoin --run_test=validation_chainstatemanager_tests
 1Running 7 test cases...
 2node/blockstorage.cpp:307 LoadBlockIndex: Assertion `GetParams().AssumeutxoForBlockhash(*snapshot_blockhash)' failed.
 3unknown location(0): fatal error: in "validation_chainstatemanager_tests/chainstatemanager_loadblockindex": signal: SIGABRT (application abort requested)
 4test/validation_chainstatemanager_tests.cpp(441): last checkpoint
 5test_bitcoin: common/args.cpp:577: void ArgsManager::AddArg(const std::string&, const std::string&, unsigned int, const OptionsCategory&): Assertion `ret.second' failed.
 6unknown location(0): fatal error: in "validation_chainstatemanager_tests/chainstatemanager_snapshot_init": signal: SIGABRT (application abort requested)
 7test/validation_chainstatemanager_tests.cpp(508): last checkpoint: "chainstatemanager_snapshot_init" fixture ctor
 8test_bitcoin: common/args.cpp:577: void ArgsManager::AddArg(const std::string&, const std::string&, unsigned int, const OptionsCategory&): Assertion `ret.second' failed.
 9unknown location(0): fatal error: in "validation_chainstatemanager_tests/chainstatemanager_snapshot_completion": signal: SIGABRT (application abort requested)
10test/validation_chainstatemanager_tests.cpp(577): last checkpoint: "chainstatemanager_snapshot_completion" fixture ctor
11test_bitcoin: common/args.cpp:577: void ArgsManager::AddArg(const std::string&, const std::string&, unsigned int, const OptionsCategory&): Assertion `ret.second' failed.
12unknown location(0): fatal error: in "validation_chainstatemanager_tests/chainstatemanager_snapshot_completion_hash_mismatch": signal: SIGABRT (application abort requested)
13test/validation_chainstatemanager_tests.cpp(660): last checkpoint: "chainstatemanager_snapshot_completion_hash_mismatch" fixture ctor

It’s happy again the next commit fe735572d05b95569e9c82fa1b1767c7da23b423.

— Here’s a torrent for 800,000: magnet:?xt=urn:btih:50ee955bef37f5ec3e5b0df4cf0288af3d715a2e&dn=utxo-800000.dat&tr=udp%3A%2F%2Ftracker.bitcoin.sprovoost.nl%3A6969

You’ll need this:

0{
1  "coins_written": 111535121,
2  "base_hash": "00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054",
3  "base_height": 800000,
4  "txoutset_hash": "7d69b87512db3d13b9758ea32b93ce468d18cf7456fb5d250c9e1bed9339e4d2",
5  "nchaintx": 868965226
6}

I’ll try a sync with this shortly. Update: ran successful IBD on Intel macOS and on Ubuntu 23. With the latter I first had it sync until 2013 so the dbcache would be 1 GB, before loading the snapshot.

—

On macOS the shutdown (a day) after completing background sync seems gets stuck:

0023-08-16T07:48:43Z [msghand] [net] Saw new cmpctblock header hash=0000000000000000000229118fcb007fc035045c3a3394233116aa25d496129e peer=398
12023-08-16T07:48:44Z [msghand] UpdateTip: new best=0000000000000000000229118fcb007fc035045c3a3394233116aa25d496129e height=803421 version=0x20000000 log2_work=94.362393 tx=879900343 date='2023-08-16T07:48:37Z' progress=1.000000 cache=114.9MiB(924996txo)
22023-08-16T07:49:00Z [msghand] New outbound peer connected: version: 70016, blocks=803421, peer=491 (block-relay-only)
3^C2023-08-16T07:50:33Z [addcon] addcon thread exit
42023-08-16T07:50:33Z [opencon] opencon thread exit
52023-08-16T07:50:33Z [init] Shutdown: In progress...
62023-08-16T07:50:33Z [net] net thread exit
72023-08-16T07:50:33Z [msghand] msghand thread exit
82023-08-16T07:55:12Z [scheduler] Flushed fee estimates to fee_estimates.dat.
92023-08-16T08:23:36Z [scheduler] Potential stale tip detected, will try using extra outbound peer (last tip update: 2092 seconds ago)

I didn’t run with useful debug flags unfortunately. It was a pruned node. Despite kill -9 it was able to restart in good shape though:

02023-08-16T09:03:57Z [init] [snapshot] cleaning up unneeded background chainstate, then reinitializing

Started review (will update list below with progress).

• 6be9fd22ee2b9759cdd30a7bca4d07a9346eb57b First draft of background sync logic (1/23)
• a62ed3267d69ae8f4e9edf68f4a0f5348fdefe2d bugfix: correct is_snapshot_cs in VerifyDB (2/23)
• 3b9d3d63d9ccb609ec08da34776f7c5f29d21a20 assumeutxo: disallow -reindex when background syncing (3/23)
• cdb73e44b7f0d4ea186ba657ddf7809434f0a12a chainparams: add blockhash to AssumeutxoData (4/23)
• 5ea9abfbac3543f2d2cef90a1528abed35623822 validation: MaybeRebalanceCaches when chain leaves IBD (5/23)
• d9afcd70910f30fe9c99afa4dbbcdd7ccaf29d6e validation: add ChainstateRole (6/23)
• 2856710077abbc8f07a97adfff5f6829358155be validation: pass ChainstateRole for validationinterface calls (7/23)
• f2a15428277090d6ad387de6d517303b9bc4e2af validationinterface: only send zmq notifications for active (8/23)
• 550ac6248d1a596207c357181edbec79236d576d wallet: validationinterface: only handle active chain notifications (9/23)
• ddbd0639e0af4a56cd5f647e11e0d38f06d20a68 net_processing: validationinterface: ignore some events for bg chain (10/23)
• c177b8f442892a9b865b0b771f0a4823f797322d index: cache indexer references on ChainstateManager (11/23)
• f64eeaeeb83d1c21f2d9e65c630eee2a1a5d48ba validation: indexing changes for assumeutxo (12/23)
• 9e2b8ef836aeb1dc2e4b57449fc56badebea6772 validation: pruning for multiple chainstates (13/23)
• a66790c662d6516b9fec9ca2cc97e410a72fe899 validation: populate nChainTx value for assumedvalid chainstates (14/23)
• fe735572d05b95569e9c82fa1b1767c7da23b423 blockstorage: segment normal/assumedvalid blockfiles (15/23)
• d342085067306231a2a454c39d276495dbd13cd3 validation: assumeutxo: swap m_mempool on snapshot activation (16/23)
• 8ac7427b41fdec1dbc7e06d0aa2c0e469b879ba2 rpc: add loadtxoutset (17/23)
• bc3042436e4e4a84328a2e56ca021a617d37c27a rpc: add getchainstates (18/23)
• 24497a881ebf699b92c01342a24f1c4b35b79590 test: add feature_assumeutxo functional test (19/23)
• c5ce27802be60d81f909a8f0f6a3827f956d88e3 contrib: add script to demo/test assumeutxo (20/23)
• 9e46cf539533076d6206964e8d74e504030fc4a5 doc: update release-process.md for assumeutxo (21/23)
• 9e62b849852bfea64eda18f51137a4a53e80f8c0 chainparams: add assumeutxo param at height 788_000 (22/23)
• 253bcd5fd392f70de82ec7c675f9777d1e098d50 assumeutxo: swap mempool on existing activation (23/23)

Rebased, but only part of the way addressing all feedback so far.

• Took @ryanofsky’s net_processing cleanup commit
• Better error handling in loadtxoutset (@theStack)
• Cleaned up unnecessary chainstate roles (@dergoegge, @ryanofsky)
  • Refactoring and behavior changes are now separate commits

Okay, the latest rebase is pushed, and I believe I’ve addressed all outstanding feedback (with the exception of two marginal comments from @ryanofsky relating to the first commit here, which perhaps I’ll leave to @sdaftuar… unless he doesn’t get back in some amount of time, in which case I’ll take a look myself).

I’ve iterated through all commits to ensure they build and pass the chainstate unittests. I’ve also fixed the intermediate-commit test failure that @Sjors discovered, which was based on not updating the unittest to use a “recognized” snapshot early enough. I’ve introduced a commit that fixes that sequencing.

For anyone curious, the tag updates here are au.029 -> au.031 (GitHub range diff) but the diff is kind of useless because of the rebase.

Much more useful is doing the range-diff locally yourself:

0git range-diff master jamesob/au.029 jamesob/au.031

Thanks again for continued review and testing @ryanofsky @dergoegge @fjahr @Sjors @theStack @achow101 @D33r-Gee @ismaelsadeeq @alexanderwiederin @mzumsande @hazeycode and anyone else I forgot.

This all looks very good and I feel like I got through the most complicated changes, but I’m still working through the rest. Most of my comments are not too important but I do think the indexing and pruning changes can be simplified a bit.

Progress so far:

• 08a34e56ba7aa8714982d352238f1cb6eba43e29 net_processing: Request assumeutxo background chain blocks (1/24)
• 37af5f47541e77753d20bfbb7c105b55c0caf305 bugfix: correct is_snapshot_cs in VerifyDB (2/24)
• 1755c53365d53693213a721138a44bdd889d8794 assumeutxo: remove snapshot during -reindex{-chainstate} (3/24)
• 3d103bbf266bf3c8f66b3aff2237cc0a95f96a75 chainparams: add blockhash to AssumeutxoData (4/24)
• 3e7d3537fc06c8aff8a48edc8f179a70af46e04c validation: MaybeRebalanceCaches when chain leaves IBD (5/24)
• 95f45e38b5e47eaef56468909d04418b93e9c962 validation: add ChainstateRole (6/24)
• 9eda8224119450b8614837e0c8decfc81a6b9915 validation: only call UpdatedBlockTip for active chainstate (7/24)
• 2873d91ada1c240de280a35bbf169ed0c708e057 validation: pass ChainstateRole for validationinterface calls (8/24)
• b48729144fba1dbc64d603b2644a76c3f00d44eb validationinterface: only send zmq notifications for active (9/24)
• ba2f21a9ef88ce9631a2143f16569ee231953a91 wallet: validationinterface: only handle active chain notifications (10/24)
• 5d096add65e84530c907b5ee934af144067d5ef1 net_processing: validationinterface: ignore some events for bg chain (11/24)
• 688b68f34fe061c3064a516cdf77d4c155bfcafd index: cache indexer references on ChainstateManager (12/24)
• 5d01151b41b244dabeda54122400df17a3dfc4d2 validation: indexing changes for assumeutxo (13/24)
• b9ac4f7925e2d9dec67be06fdc2a381532581136 validation: pruning for multiple chainstates (14/24)
• abaf61487bd5eb2c33ed62dae602b5f479f757f9 test: adjust chainstate tests to use recognized snapshot base (15/24)
• a2a49ea7f1ed2931ab388bac677196dedc10714e validation: populate nChainTx value for assumedvalid chainstates (16/24)
• 07517a2311c410dd107c4feb4e113bb7c629da8e blockstorage: segment normal/assumedvalid blockfiles (17/24)
• 0e507021932f72aa7345bf84b805c7cda68eea65 validation: assumeutxo: swap m_mempool on snapshot activation (18/24)
• 03de0d5ac9457b89e577a11e146cc394a8494b7b rpc: add loadtxoutset (19/24)
• 05aba13ce27bbe7346ffb72a619638b6f84d3679 rpc: add getchainstates (20/24)
• 60d6f6bbe91879d542439bec05ef406eb9954eb8 test: add feature_assumeutxo functional test (21/24)
• 5394f2e9b7fea36d3cba43af82143f5c72c08ae5 contrib: add script to demo/test assumeutxo (22/24)
• a3f96fbee900fdf23a160a461164a65a0c0cfe84 doc: update release-process.md for assumeutxo (23/24)
• e44815e9a754c967415402b34807633625393084 chainparams: add assumeutxo param at height 788_000 (24/24)

Did code review at e44815e9a754c967415402b34807633625393084. Most of the @ryanofsky I think are worth addressing. Once these are addressed or resolved I will do my own clean path.

I also spotted a couple of typos along the way.

Okay, all feedback addressed - some good simplifications there. Thanks for the comprehensive review, @ryanofsky @naumenkogs.

You can check the diff with

0git range-diff upstream/master jamesob/au.031 jamesob/au.033

Had to push a small commit to fix underflow in pruning calculations.

0git range-diff upstream/master jamesob/au.033 jamesob/au.034