Synchronization Checkpoints #9885

Describe the issue

It takes a long time and consumes a lot of resources to synchronize beginning at the Genesis block. Many people don't want to spend that opportunity cost. The Genesis block is a checkpoint with an empty UTXO set. Lets let people start from (Periodic UTXO Merkle Tree + Checkpoint Block) Checkpoints instead of the Genesis block.

Expected behaviour

1. Close Bitcoin
2. Delete the blocks, chainstate, and database folders.
3. Start Bitcoin
4. Send and receive transactions

Actual behaviour

1. Close Bitcoin
2. Delete the blocks, chainstate, and database folders.
3. Start Bitcoin
4. Watch the grass grow, paint dry, etc.
5. Use Electrum. *Comic relief, come on guys, such a hard audience!

What version of bitcoin-core are you using?

0.13.2

Suggested Solution

Every CheckpointPeriod number of blocks, maybe every 1 week, 1 month, or 3 months:

1. Block is determined by client to be a Checkpoint Block.
2. UTXO set is atomically: sorted by txid, placed into an independently reproducible Merlke tree
3. Resulting hash from Step 2 Merkle tree is hashed with Checkpoint Block Hash. This is called the Checkpoint Hash.
4. Look through Checkpoint Hash messages from other nodes. Puts into a Trustworthy Checkpoint Hash List any signed by a Set of Public Keys Considered as *Trustworthy (SPKCT). SPKCT is loaded with data from a json file, which can be updated by the user, maybe client restart required.
5. Client verifies Checkpoint Hash signatures in the SPKCT.
6. For each Trustworthy Checkpoint Hash, compare with Checkpoint Hash computed at Step 3. a. All match: life as normal b. Discrepancy: Alarm user: Trustworthy node is compromised/mistaken c. List empty: Alarm user: Trustworthy nodes unreachable, suspect serious network problems

• Trustworthy here means as trustworthy as the signature used to sign the Genesis block.

Given the above is standard practice...

• When a pruning client is first started, it can ask the network for the latest (or some CheckpointVerifyBlockWindow blocks back) Checkpoint, and start from there instead of the Genesis block.
• Also, if a pruning client hasn't connected to the network in a long time, it could potentially again start synchronizing from the latest Checkpoint instead of from where it disconnected. This might require the wallet support utxo-only coins better. I forget, do blocks contain a Merkel tree hash for the transaction hashes?
• This could also be used to jump start a client to get it working immediately, and then if the user wants build up from the Genesis block in the background.

Security/Reliability Improvement

My proposed solution actually helps identify numerous problems. For example:

1. If a portion of the network gets disconnected, it can now be detected
2. If a computer is compromised or there is some computational error in the UTXO state, it can now be detected

Security Model Alert

If a user decides to start synchronization from a Checkpoint, such requires more trust than starting from the Genesis block. This is because a UTXO set cannot be self-verified without beginning synchronization at the Genesis block. But... I think my proposal does a pretty good job of identifying if someone lies. See "Security/Reliability Improvement", think about instead of loading the latest Checkpoint, load say 10 Checkpoints back or something. Ultimately I think my suggestion results in almost as good of security as starting synchronization at the Genesis block.

This bitcoin.stackexchange.com question might be of use to you.

dcousens: isn't that link just for regular old pruning? I don't see anything there about making checkpoints and being able to start from a checkpoint.

==============

Also I forgot to mention that when starting from a checkpoint, the user should probably use some other method other than the bitcoin network to verify that the highest block's hash & height & difficulty match what others have. It might be worthwhile to verify block headers starting from the Genesis block to the checkpoint block, but I don't think this actually increases security, probably a waste.

More on security: I guess you could point out that the SPKCT could collude, and you could just be connected to them, and they might lie and give you false utxo data, and shield you from getting correct utxo data. But then I could point out that if they are able to do that, then they could even do that with today's start from Genesis method.

@praxeology-guy my point was that you could use the chain state of a pruning node as your checkpoint, provided you trust it.

Oh, you put an @ symbol. @dcousens. cool.

Well, there are numerous issues w/ doing that:

1. The software using the chain state has to be shut down/stop processing blocks, its all manual, etc. I want to make bitcoin easy to use for everyone. Start it up and it works fast and easy!
2. "Provided you trust it", sure if it was signed by a Trustworthy that would help, but it would not be nearly as good as my suggestion. See Step 6 in my solution. See "Security/Reliability Improvement".

I think a better place to discuss that would be the bitcoin-core or the bitcoin-dev mailing list.

There is ongoing discussion about UTXO and TXO commitments, which would respectively allow full nodes to synchronize history with SPV security assumption, or run without ever-growing UTXO set entirely.

@jonasschnelli Sorry I'm new here and my last attempts to use a mailing list either were censored or not responded to so... I went here with this one. @sipa thanks, I'll read through it. I guess my idea isn't so novel after all, unless maybe my idea of having a CheckpointPeriod is new... well and also I did not propose the UTXO commitment to be included in a block. Man, I saw subjects about "UTXO commitment" in the mailing list but that didn't mean anything to me until you responded now.

Maybe it could be included in the Checkpoint Block if the UTXO was split into an "old" and a "new", where "new" spans the last "NewSpan-1" blocks before the Checkpoint Block. Then the "old" UTXO hash could be calculated in the background while the new blocks are being built, and the Checkpoint Block would only be delayed by calculating the "new" UTXO hash.

Sorry just wanted to throw that out there now before I go to bed. Will look at the mailing list tomorrow.

I appreciate that you're trying to help by coming up with ideas, but the idea of an issue tracker like this is not to open brainstorming topics about far-out features.

The issue tracker in principle is for a) bugs b) sometimes as a a kind of dev TODO list. If you're not actively contributing to the code though, there's no use in opening an issue for every one of your ideas.

Also all of these things have been discussed in some form before, and need technical development instead of just talking about it. The bottleneck is contributors working on them and actually fleshing out the ideas, not ideas.

@praxeology-guy I don't know how to get in touch with you, but I share your interest in quickening the process of starting bitcoin when you don't already have the full blockchain. I described an idea at https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-September/016430.html.

My apologies for using this issue to attempt to contact praxeology-guy. If I knew how to contact a Github user without doing it here, I'd do that instead.