Skip verifying transaction signatures during initial block-chain download #492

This change skips ECDSA signature verification for transactions during the initial block-chain download, which makes downloading the block chain much faster.

"Initial block chain download" is all blocks up to 120 blocks before the last blockchain lock-in point.

Reasoning for why this is safe:

If an attacker tries to feed a client bad transactions during the initial block-chain download (transactions with invalid signatures), then they change the merkle tree, and at the first blockchain lock-in the bad chain will be rejected.

Transactions are still checked for orphan blocks that come in during initial block-chain-download (ConnectInputs will be called with fBlock=FALSE) and for mined blocks (ConnectInputs called with fMiner=TRUE).

It seems to me that its sort of weird that I could generate a bunch of completely insane blocks, and feed you them and get you to accept them— only to drop them when you hit the first checkpoint. Perhaps I could use this to pollute your wallet.dat with megabytes of junk transactions? I suppose I could still do this without this change though.

Perhaps it should validate up to block 75k or so, there are so few transactions that this shouldn't change the speed much but by then the difficulty starts getting high enough to make mining insane blocks just for disruptive purposes less attractive.

I like that the stopping points is 120 blocks before the last lock-in, this seems prudent.

It might be prudent to rig the RPC/UI to never show anything as confirmed until the client reaches the last checkpoint. I can imagine an attack where I can somehow trigger a remote node to flush and rebuild its blockchain, and then sneak some crap transactions past it while its between checkpoints. Though I admit the attack there is a bit far out and this really would make bitcoin no more vulnerable to it.

Thanks for the thoughtful comments.

RE: polluting the wallet with megabytes of junk blocks/transactions: I'm working on another fix for that (see my DoSprevention branch for the start, punishing peers that send lots of obviously bogus orphan blocks is on my TODO list). This change doesn't make that attack any easier to mount. By far the bottleneck to that attack would be generating bogus blocks with valid proof-of-work, and if you were doing that they'd have only a single coinbase transaction which doesn't have a signature to check in it anyway.

RE: rigging the RPC/UI to show everything as unconfirmed until initial download is complete: good idea, but as you say this proposed change doesn't affect that.

I'm not sure I really like this. This means giving the maintainer of the software control over which block-chain is accepted by the client. Of course, he can already do this by changing the source code, so there may not be a real reason for my dislike for it. However, maybe we can still have a configuration option "Verify entire chain", for paranoid people?

On the other hand, if we are going to trust the locked-in chain data in the client anyway, why not do it for each and every block? There is not even a need for storing the complete hash of each block. For example, one could argue that 72-bit security is enough for blocks (would require 10 years to break with the current network power). In that case, store for each block the lower 9-N bytes of the hash, where N is the number of zero bytes at the front implied by its difficulty. The entire current chain would require 617725 bytes, in that case.

We already control which block-chain is accepted with the blockchain checkpoint mechanism.

RE: adding a 'verify the whole thing' option: if you're that paranoid, you should run a separate piece of code to do the verification, in my humble opinion. In fact, a standalone verify-bklindex/blk000 tool would be way spiffy.

I don't understand your last point; the client needs all (unspent and not-involving-my-keys) transactions in the chain, and transactions are what makes the chain large.

1. There is a subtle difference: currently the maintainer can prevent certain chains from being accepted by clients, but the accepted chain does need to be valid. If you're going to skip verification of everything beneath a certain checkpoint, an "evil maintainer" could make "his" client accept a fake chain. Again, I realize there is little difference with the maintainer just changing the code, so this argument is probably meaningless.

2. Verification in a separate tool: I like that idea, actually, as it also allows detection of corruption of the database afterwards.

3. Sure, after downloading it has the entire chain. I guess it wasn't entirely clear, but I was talking about a 600 kB file that would be shipped with/inside the binary. That would suffice to lock each and every block, instead of just a few checkpoints.

I'm testing the speed on this just by timing how long it takes to download a few thousand blocks. Anything else you want me to test for this one? And - are there better ways of speed testing optimizations?

You'll need to test on blocks that actually have a bunch of transactions in them, so don't test with the first few thousand blocks in the main chain (because they're all mostly empty-- a "generate bitcoins" transaction doesn't have a signature to verify).

I'd suggest:

• Download up to block 50,000 or so
• Shutdown, then save the blkindex.dat and blk0001.dat files
• Time how long it takes to download blocks 50,000 to 60,000
• Shutdown, reset blkindex.dat/blk0001.dat, and apply the patch
• Repeat test

You could eliminate network variance by removing the node's addr.dat and then running with: -noirc -nodnsseed -nolisten -connect=...ip address...

If you have two machines, then -connecting the test machine to a machine on your local network for the block download.

There was a 10% gain for the first 70k blocks. At 99k, the patched client was 27m faster. Testing with Gavin's method now to verify.

...it took 1:17 for the unpatched client to get to 99k, and :50 for the patched.

27m faster? This sounds broken. Even with 4000 transactions per block there is no way it should be taking that long.

By using Gavin's method to avoid network variance - the original client took 13m to get from 65-75k, and the patched client took 10m. Giving a 23% improvement. I'm assuming that percentage would be higher with the recent, heavier blocks.

There weren't that many transactions at 75k either yet. Maybe you can benchmark the entire chain verification? That is what every new node installation has to go through anyway.

Use a local node that is synced up, and use -connect to connect to it from an empty install, both patched and unpatched. Maybe it's possible to run bitcoind in a shell using "time", to measure how much CPU time it uses as well as how much clock time? This may take quite some time though...

Sipa, I can do that now. I think that knowing these timings will be good for future tests as well. I had been just eyeballing it, but I will try to write some kind of script that does the speed test. I imagine this could be used on several current and future commits.

Cheers for reminding me about time, that will make things a lot less tedious =P

OK I have tested this with the full blockchain. I went from 0 to 145,686 and had the following results:

0.4.0rc1 - 10H:21M

0.4.0rc1 + pull 492 - 7H:14M

So that's an improvement of 3H:7M or 30%

These numbers are back to sounding suspect to me again. My desktop does over a thousand ECDSA verifies a second. That doesn't really jive with the speedup here.

gmaxwell: can you test this commit to see if you get different results from my test?

Sure, I will. FWIW, I'm not questioning your methodology so much as suggesting that if its true then there is perhaps something else wrong with bitcoin which out to be fixed regardless of this change.

I'm running this in virtualbox, so maybe my system's speed is a contributing factor - I will try it tomorrow on a different machine.

As for something else being wrong; in general I think major changes are needed to the blockchain download. Having a GB+ requirement is fairly undesirable (esp for mobile).

I don't remember who, but someone brought up an interesting idea at the conference (poss. Gavin?). It would be an opt-in system for running a wallet, with optional trusted centralized verification. I'm not sure what that entailed but I envisioned the following:

I could run bitcoin with -server, which would download the blockchain like the current system. I could then connect to that node to verify my transactions from another bitcoin client (where I wouldn't have to download the chain).

Does anyone know if someone is working on this, and/or if it's viable? So far, it's the cleverest solution to the download woes IMO.

I'm also wondering if this is possible with dropbox?

I have also read a little about the headers-only version, I can't find a reference to how it's used - so I'm assuming it isn't fully implemented yet.