validation: assumeutxo params mainnet

Sjors commented at 12:53 pm on September 30, 2023: member

This adds snapshot parameters for mainnet block 840,000.

You can generate the snapshot yourself using ./contrib/devtools/utxo_snapshot.sh or download my torrent:

torrent: magnet:?xt=urn:btih:596c26cc709e213fdfec997183ff67067241440c&dn=utxo-840000.dat&tr=udp%3A%2F%2Ftracker.bitcoin.sprovoost.nl%3A6969

It would be a good idea to test:

That you can produce the same snapshot file, sha256 sum:

0dc4bb43d58d6a25e91eae93eb052d72e3318bd98ec62a5d0c11817cefbba177b utxo-840000.dat

That the snapshot works

DrahtBot commented at 12:53 pm on September 30, 2023: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	fjahr, achow101, theStack, mzumsande, willcl-ark
Stale ACK	pablomartin4btc

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

No conflicts as of last run.

DrahtBot added the label Needs rebase on Oct 2, 2023

Sjors force-pushed on Oct 3, 2023

DrahtBot removed the label Needs rebase on Oct 3, 2023

in src/kernel/chainparams.cpp:180 in 27a8739795 outdated

172@@ -173,7 +173,12 @@ class CMainParams : public CChainParams {
173         };
174 
175         m_assumeutxo_data = {
176-            // TODO to be specified in a future patch.
177+            {
178+                .height = 800'000,
179+                .hash_serialized = AssumeutxoHash{uint256S("0x7d69b87512db3d13b9758ea32b93ce468d18cf7456fb5d250c9e1bed9339e4d2")},
180+                .nChainTx = 868965226,
181+                .blockhash = uint256S("0x00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054")

maflcko commented at 12:28 pm on October 3, 2023:

0                .blockhash = uint256S("0x00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054"),

style-nit, if you re-touch.

Sjors force-pushed on Oct 9, 2023

DrahtBot added the label CI failed on Oct 9, 2023

Sjors force-pushed on Oct 25, 2023

Sjors commented at 9:02 am on October 25, 2023: member

Updated hash_serialized after #28685.

I only tested that the loadtxoutset command finishes. @pablomartin4btc I found a different hash than you… https://github.com/Sjors/bitcoin/commit/24deb2022b822f22fba9fcbee201e37a83225eb2#commitcomment-130847775

DrahtBot removed the label CI failed on Oct 25, 2023

pablomartin4btc approved

pablomartin4btc commented at 2:02 am on November 1, 2023: member

tACK 0800be24bc6de4154653be75b8fa9902a4a8c3a7

 0./contrib/devtools/utxo_snapshot.sh 800000 - ./src/bitcoin-cli -datadir=${AU_DATADIR}
 1Rewinding chain back to height 800000 (by invalidating 00000000000000000000e26b239cf19ec7ace5edd9694d51a3f6933247720947); this may take a while
 2Generating txoutset info...
 3{
 4  "height": 800000,
 5  "bestblock": "00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054",
 6  "txouts": 111535121,
 7  "bogosize": 8443325590,
 8  "hash_serialized_3": "6e63799cac0ab45f94789f97692caf691ee66625e920cbf2525fceb24509adda",
 9  "total_amount": 19437286.83643360,
10  "transactions": 72936026,
11  "disk_size": 7264997650
12}
13Restoring chain to original height; this may take a while

I incorrectly took the muhash value from gettxoutsetinfo (without running the utxo_snapshot.sh which also performs invalidateblock right before).

pablomartin4btc referenced this in commit 24deb2022b on Nov 1, 2023

Sjors commented at 2:08 am on November 1, 2023: member

@pablomartin4btc thanks for checking that you get the same snapshot. In addition it would be good to test if you’re able to load the snapshot and complete the background sync.

pablomartin4btc commented at 2:24 am on November 1, 2023: member

In addition it would be good to test if you’re able to load the snapshot and complete the background sync.

I’ve already checked IBD completed fine with[background validation]finished, but with the previoushash_serialized, as I wanted to verify that #28698 was only happening before the IBD completion.

I’ll run all the process again with the newhash_serializedvalue.

pablomartin4btc commented at 7:49 pm on November 9, 2023: member

Tested loadtxoutset a few times (pending IBD to finish) with coinstatsindex & blockfilterindex, and without them. I’ve also validated that the sha256sum corresponds to the one obtained from the downloaded file.

Now, having a pruned node with IBD completed, from which I got the m_assumeutxo_data mentioned in my previous review, when I run ./contrib/devtools/utxo_snapshot.sh with a filename in order to perform the actual dumptxoutsetwithin the script, I got a node crash with ERROR: ReadBlockFromDisk: OpenBlockFile failed for FlatFilePos(nFile=-1, nPos=0), which instance I had to kill since it was unresponsive. I’m not sure if this situation was due to the invalidateblock performed twice by runnning the utxo_snapshot script (one to get the info from gettxoutsetinfo and the second for the actual dump), the pruned node or a combination of all three. Starting the node again without blockfilterindex (option given by the node before doing the suggested -reindex or -reindex-chainstate which doesn’t work on pruned nodes), made it worse till getting !setBlockIndexCandidates.empty(). I see there are a few open issues regarding invalidateblock & setBlockIndexCandidates so before investigating further I’d like to know first if the dumptxoutset is expected to work from a fully synced pruned node?

Sjors commented at 1:39 am on November 10, 2023: member

As discussed offline, the utxo_snapshot.sh script should have a sanity check to make the node isn’t pruned below the snapshot height. On a pruned node you can only make a snapshot for a block that hasn’t been pruned yet.

pablomartin4btc commented at 1:20 pm on November 10, 2023: member

As discussed offline, the utxo_snapshot.sh script should have a sanity check to make the node isn’t pruned below the snapshot height. On a pruned node you can only make a snapshot for a block that hasn’t been pruned yet.

I agree with it, mentioned on a #27845 that’s touching the script ATM.

In the testing I performed and mentioned above the pruned node was above the snapshot height (800'000):

0  "pruned": true,
1  "pruneheight": 600510,
2  "automatic_pruning": true,
3  "prune_target_size": 20971520000,

ryanofsky referenced this in commit 1c8893bd1c on Dec 4, 2023

DrahtBot added the label CI failed on Jan 17, 2024

Sjors commented at 2:16 pm on February 13, 2024: member

I plan to make a fresh snapshot after #26045 lands.

fjahr commented at 5:20 pm on March 4, 2024: contributor

Didn’t see this one until the conflict popped up. I think we can move to 830,000 for v27.0 and I am unsure if we need to wait for #26045. I will bring it up in the next meeting.

Sjors force-pushed on Apr 29, 2024

Sjors commented at 11:38 am on April 29, 2024: member

I switched to the halving block! Also dropped all prerequisites but #26045 from the description, but see conceptual discussion in #29616.

Sjors renamed this:
~~[do not merge] validation: assumeutxo params mainnet~~
validation: assumeutxo params mainnet
on Apr 29, 2024

DrahtBot removed the label CI failed on Apr 29, 2024

DrahtBot added the label Validation on Apr 29, 2024

Sjors force-pushed on May 28, 2024

Sjors commented at 11:09 am on May 28, 2024: member

Rebased now that #29612 landed. Will do some final testing before marking this ready to review.

Sjors marked this as ready for review on May 28, 2024

fjahr commented at 8:00 am on July 22, 2024: contributor

tACK f193b0fd853cb9e2f7cb4ed9dd6fb5d2662a04fe

I have re-tested the full assumeutxo flow using this snapshot: Downloaded the snapshot from the torrent link, verified the hash, successfully loaded the snapshot in a fresh node, verified that the snapshot chainstate was able to sync to the tip and the node was usable. Then waited until the background chainstate caught up to the snapshot and that was also successful. Restarted the node and verified the cleanup of the second chainstate. Also checked that I get the exact same dump using dumptxoutset with #29553 on a fully synced node that did not use the snapshot to sync.

DrahtBot requested review from pablomartin4btc on Jul 22, 2024

DrahtBot commented at 7:53 pm on August 7, 2024: contributor

🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/25496351407

Make sure to run all tests locally, according to the documentation.

The failure may happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

DrahtBot added the label CI failed on Aug 7, 2024

achow101 added this to the milestone 28.0 on Aug 8, 2024

achow101 commented at 5:44 pm on August 8, 2024: member

Looks like there’s a silent merge conflict:

 0kernel/chainparams.cpp: In constructor ‘CMainParams::CMainParams()’:
 1kernel/chainparams.cpp:194:9: error: no match for ‘operator=’ (operand types are ‘std::__debug::vector<AssumeutxoData>’ and ‘<brace-enclosed initializer list>’)
 2  194 |         };
 3      |         ^
 4In file included from /usr/include/c++/11/vector:76,
 5                 from /usr/include/c++/11/functional:62,
 6                 from /usr/include/c++/11/pstl/glue_algorithm_defs.h:13,
 7                 from /usr/include/c++/11/algorithm:74,
 8                 from ./span.h:8,
 9                 from ./uint256.h:10,
10                 from ./consensus/params.h:9,
11                 from ./kernel/chainparams.h:9,
12                 from kernel/chainparams.cpp:6:
13/usr/include/c++/11/debug/vector:249:7: note: candidate: ‘std::__debug::vector<_Tp, _Allocator>& std::__debug::vector<_Tp, _Allocator>::operator=(const std::__debug::vector<_Tp, _Allocator>&) [with _Tp = AssumeutxoData; _Allocator = std::allocator<AssumeutxoData>]’
14  249 |       operator=(const vector&) = default;
15      |       ^~~~~~~~
16/usr/include/c++/11/debug/vector:249:17: note:   no known conversion for argument 1 from ‘<brace-enclosed initializer list>’ to ‘const std::__debug::vector<AssumeutxoData>&’
17  249 |       operator=(const vector&) = default;
18      |                 ^~~~~~~~~~~~~
19/usr/include/c++/11/debug/vector:252:7: note: candidate: ‘std::__debug::vector<_Tp, _Allocator>& std::__debug::vector<_Tp, _Allocator>::operator=(std::__debug::vector<_Tp, _Allocator>&&) [with _Tp = AssumeutxoData; _Allocator = std::allocator<AssumeutxoData>]’
20  252 |       operator=(vector&&) = default;
21      |       ^~~~~~~~
22/usr/include/c++/11/debug/vector:252:17: note:   no known conversion for argument 1 from ‘<brace-enclosed initializer list>’ to ‘std::__debug::vector<AssumeutxoData>&&’
23  252 |       operator=(vector&&) = default;
24      |                 ^~~~~~~~
25/usr/include/c++/11/debug/vector:255:7: note: candidate: ‘std::__debug::vector<_Tp, _Allocator>& std::__debug::vector<_Tp, _Allocator>::operator=(std::initializer_list<_Tp>) [with _Tp = AssumeutxoData; _Allocator = std::allocator<AssumeutxoData>]’
26  255 |       operator=(initializer_list<value_type> __l)
27      |       ^~~~~~~~
28/usr/include/c++/11/debug/vector:255:46: note:   no known conversion for argument 1 from ‘<brace-enclosed initializer list>’ to ‘std::initializer_list<AssumeutxoData>’
29  255 |       operator=(initializer_list<value_type> __l)
30      |                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~

Sjors force-pushed on Aug 8, 2024

Sjors commented at 6:01 pm on August 8, 2024: member

Rebased due to #30560 and replaced uint256S("0x... with uint256{"..., as well as using consteval_ctor for the block hash, and m_chain_tx_count.

I did not test that the update to contrib/devtools/test_utxo_snapshots.sh in c06f2368e2346b087b72064a4f1e9817a680ef5f actually produces the code in this commit.

If #30598 lands I’ll bake a new torrent, update the PR description and probably rebase. I’ll hold off on testing this PR myself until that’s done.

DrahtBot removed the label CI failed on Aug 8, 2024

fjahr commented at 8:50 pm on August 8, 2024: contributor

re-ACK fb543c9f733610b8fadbfd6f1c71f1bb520de985

Only the merge conflict was addressed, the actual values did not change, per git range-diff master f193b0fd853cb9e2f7cb4ed9dd6fb5d2662a04fe fb543c9f733610b8fadbfd6f1c71f1bb520de985. Also confirmed that the dump can still be loaded without error.

chainparams: add mainnet assumeutxo param at height 840_000 1610643c8b

Sjors force-pushed on Aug 10, 2024

Sjors commented at 6:57 am on August 10, 2024: member

Rebased after the (tiny) format change in #30598, for easier testing. Updated torrent link and snapshot file hash in the description. The chainparams themselves are unchanged.

I tested myself that another produced an identical snapshot.

By the way #28358 would speed up loading the snapshot for anyone with enough RAM.

fjahr commented at 8:58 pm on August 10, 2024: contributor

re-ACK 1610643c8b37a9f674b236cfa79abf8f8aaf1410

Verified only rebase via git range-diff master fb543c9f733610b8fadbfd6f1c71f1bb520de985 1610643c8b37a9f674b236cfa79abf8f8aaf1410 and re-tested loading and syncing with the new snapshot torrent.

Sjors commented at 12:17 pm on August 11, 2024: member

02024-08-10T19:39:49Z [snapshot] computing UTXO stats for background chainstate to validate snapshot - this could take a few minutes
1Killed

I think this happened because I used a RAM drive for pruned sync and it ran out of space (a Guix build I ran at the same time failed too). Would be nice to fail more gracefully, but I don’t think it’s related to the specific snapshot here.

Once I freed up space and started the node again, it repeated that last log entry and continued normally:

02024-08-11T12:18:43Z [snapshot] computing UTXO stats for background chainstate to validate snapshot - this could take a few minutes
12024-08-11T12:19:31Z [snapshot] snapshot beginning at 0000000000000000000320283a032748cef8227873ff4872689bf23f1cda83a5 has been fully validated
22024-08-11T12:19:31Z [snapshot] allocating all cache to the snapshot chainstate

achow101 commented at 1:57 am on August 13, 2024: member

ACK 1610643c8b37a9f674b236cfa79abf8f8aaf1410

If a few more people can review this in the next day or so, I believe we can have this in for 28.0.

fjahr commented at 9:29 am on August 13, 2024: contributor

If a few more people can review this in the next day or so, I believe we can have this in for 28.0.

I have pinged a few more people for (re-)reviews and testing but please give it until the end of the week because testing the full flow can take a while depending on what machine you are doing it on or what connection speed you have.

theStack approved

theStack commented at 10:08 am on August 13, 2024: contributor

Tested ACK 1610643c8b37a9f674b236cfa79abf8f8aaf1410

Loaded the provided snapshot on a freshly created pruned node (-prune=20000), and waited first until the snapshot chainstate left IBD, then until the background validation was finished successfully. Right after loading, I also checked that UTXO-querying RPCs like gettxoutsetinfo / scantxoutset / gettxout use the chainstate snapshot (see also PR #30636). Also created a dump on another node on height 840000 and verified that the AssumeUtxo hash ("txoutsethash" result of dumptxoutset) and the snapshot file sha256 hashes match. ✔️ Didn’t test any wallet functionality though.

Btw, if someone needs a simple torrent downloading tool for the CLI, I can recommend https://aria2.github.io/ (just don’t forget to put the magnet URI between single quotes, i.e. $ aria2c 'magnet:...'; without them the link in the PR description is still accepted, but the download won’t start).

mzumsande commented at 6:07 pm on August 13, 2024: contributor

tested ACK 1610643c8b37a9f674b236cfa79abf8f8aaf1410

I re-created the snapshot at height 840k and verified that the file is identical to the one I downloaded from the torrent. I also successfully loaded the snapshot into a new node (though I didn’t complete the sync).

willcl-ark approved

willcl-ark commented at 9:48 pm on August 13, 2024: member

tACK 1610643c8b37a9f674b236cfa79abf8f8aaf1410

Checked that the snapshot was created and had the same sha256sum

 0$ ./contrib/devtools/utxo_snapshot.sh 840000 snapshot.dat ./src/bitcoin-cli
 1Do you want to disable network activity (setnetworkactive false) before running invalidateblock? (Y/n): y
 2Disabling network activity
 3false
 4Rewinding chain back to height 840000 (by invalidating 00000000000000000001b48a75d5a3077913f3f441eb7e08c13c43f768db2463); this may take a while
 5Generating UTXO snapshot...
 6{
 7  "coins_written": 176948713,
 8  "base_hash": "0000000000000000000320283a032748cef8227873ff4872689bf23f1cda83a5",
 9  "base_height": 840000,
10  "path": "/home/will/.bitcoin/snapshot.dat",
11  "txoutset_hash": "a2a5521b1b5ab65f67818e5e8eccabb7171a517f9e2382208f77687310768f96",
12  "nchaintx": 991032194
13}
14Restoring chain to original height; this may take a while
15# ....
16
17$ sha256sum snapshot.dat
18dc4bb43d58d6a25e91eae93eb052d72e3318bd98ec62a5d0c11817cefbba177b  snapshot.dat

I also loaded the snapshot into a fresh node which worked (but didn’t yet complete the sync).

achow101 merged this on Aug 14, 2024

achow101 closed this on Aug 14, 2024

pablomartin4btc commented at 4:40 pm on August 14, 2024: member

tACK 1610643c8b37a9f674b236cfa79abf8f8aaf1410

Checksum matches downloaded file and performed loadtxoutset on a new node.

I’m still performing some other tests.

Sjors deleted the branch on Aug 15, 2024

pablomartin4btc commented at 2:47 pm on August 21, 2024: member

 0./src/bitcoin-cli -datadir=${AU_DATADIR} getchainstates
 1{
 2  "headers": 857301,
 3  "chainstates": [
 4    {
 5      "blocks": 857301,
 6      "bestblockhash": "000000000000000000017a865975aac40cdb7d05f4010a6e915ffe7af260a151",
 7      "difficulty": 86871474313761.95,
 8      "verificationprogress": 0.9999994926046412,
 9      "coins_db_cache_bytes": 8388608,
10      "coins_tip_cache_bytes": 17169383424,
11      "snapshot_blockhash": "0000000000000000000320283a032748cef8227873ff4872689bf23f1cda83a5",
12      "validated": true
13    }
14  ]
15}

Replicated behaviour as described by @Sjors and after freeing the necessary space the node continued running/ syncing successfully.

Also tested the UTXO-querying RPCs as @theStack above.

 0./contrib/devtools/utxo_snapshot.sh 840000 snapshot.dat ./src/bitcoin-cli -datadir=${AU_DATADIR}
 1Do you want to disable network activity (setnetworkactive false) before running invalidateblock? (Y/n): 
 2Disabling network activity
 3false
 4Rewinding chain back to height 840000 (by invalidating 00000000000000000001b48a75d5a3077913f3f441eb7e08c13c43f768db2463); this may take a while
 5Generating UTXO snapshot...
 6{
 7  "coins_written": 176948713,
 8  "base_hash": "0000000000000000000320283a032748cef8227873ff4872689bf23f1cda83a5",
 9  "base_height": 840000,
10  "path": "/home/pablo/.test_utxo_840/snapshot.dat",
11  "txoutset_hash": "a2a5521b1b5ab65f67818e5e8eccabb7171a517f9e2382208f77687310768f96",
12  "nchaintx": 991032194
13}
14Restoring chain to original height; this may take a while
15Restoring network activity
16true

0./src/bitcoin-cli -datadir=${AU_DATADIR} loadtxoutset /home/pablo/.test_utxo_2/utxo-840000.dat 
1{
2  "coins_loaded": 176948713,
3  "tip_hash": "0000000000000000000320283a032748cef8227873ff4872689bf23f1cda83a5",
4  "base_height": 840000,
5  "path": "/home/pablo/.test_utxo_2/utxo-840000.dat"
6}

0/contrib/devtools/utxo_snapshot.sh 840000 snapshot2.dat ./src/bitcoin-cli -datadir=${AU_DATADIR}
1Do you want to disable network activity (setnetworkactive false) before running invalidateblock? (Y/n): 
2Disabling network activity
3false
4error code: -8
5error message:
6Block height out of range

And the user will see the following in the node and it would stay there if not reset:

02024-08-21T14:44:13Z UpdateTip: new best=00000000000000afa0cd000a16e244f56032735d41acd32ac00337aceb2a5240 height=235382 version=0x00000002 log2_work=69.987697 tx=17492185 date='2013-05-09T23:54:32Z' progress=0.016219 cache=71.0MiB(571085txo)
12024-08-21T14:44:13Z UpdateTip: new best=0000000000000087c5e0b820afff496b95ba44ad64640c73b234d3261d3f99d2 height=235383 version=0x00000002 log2_work=69.987750 tx=17492341 date='2013-05-09T23:54:47Z' progress=0.016219 cache=71.0MiB(571291txo)
22024-08-21T14:44:13Z UpdateTip: new best=000000000000014a4b5fddf3c8abb6209247255ca9e8df786b271dd1b2ac82a6 height=235384 version=0x00000002 log2_work=69.987804 tx=17492344 date='2013-05-10T00:20:18Z' progress=0.016219 cache=71.0MiB(571297txo)
32024-08-21T14:44:13Z SetNetworkActive: false

0/contrib/devtools/utxo_snapshot.sh 840000 snapshot2.dat ./src/bitcoin-cli -datadir=${AU_DATADIR}
1Error: The node has not yet synchronized to block height 840001.
2Please wait until the node has synchronized past this block height and try again.

achow101 referenced this in commit bc87ad9854 on Aug 21, 2024

achow101 referenced this in commit a1f2b5bbb5 on Aug 26, 2024

Theghost256 approved

validation: assumeutxo params mainnet #28553

Code Coverage

Reviews

Conflicts