New Feature: Faster --reindex speed optimisation #21113

Hi,

If a user uses the --reindex command to download the block chain this works fine. However if you stop it midway for what ever reason for example:

• Computer Crash
• Not leaving computer on for days
• Running out of disk space

Then in this case you need to use the --reindex command again. However it takes forever to recheck each already downloaded blk000*.dat file before continuing to reindex.

Currently on my machine my blocks are reindexing between 5-26 seconds per block. (My Machine: Mac Mini with External HDD)

Do you think it would be a good idea to use file hashing checks to speed up this process for example on ubuntu you can run this command:

sha256sum blk00000.dat

which returns this hash

be88bbfc0c09b3527e71f38fe14ba8693d35271d15bfdca57567429e06671003

Would it be possible to create another .dat file containing all blk file hashes. Then this file can be cross referenced against your file system to ensure it matches. As long as each node contains a copy of the same file, then this block-hashes.dat type file can be redownloaded from a peer or a hash check to compare it with your current block-hashes.dat file.

Then your computer can cross reference each file sha256sum with the block-hashes.dat file to speed up the process.

Then i think this would speed up the duplicate --reindex by up to 8-10x faster depending on your machine

Currently my blocks are reindexing between 5-26 seconds per block. If this was using the sha256sum it would take about 3 seconds per block on a standard HDD Im am sure SSD would probably be even faster.

I believe the change would need to be made in this file https://github.com/bitcoin/bitcoin/blob/f1239b70d116ea28b65e60993a6e4ac82cc6c2b1/src/validation.cpp

near line 4611

I would do this myself yet i am not a C programmer.

• Does anybody think this would be an improvement.
• would anybody like to implement this update?
• I think this would take about 15 minutes to add this feature for a fluent c programmer.

See #15605

Hi @MarcoFalke I see a lot of comments and merged commits from that issue number #15605 you mentioned Yet the --reindex is still very slow.

1. Is that an incomplete issue? #15605
2. Would my suggestion lower security of bitcoin or would it remain secure
3. Alternatively would it be possible to add a param / option like this:

bitcoind --reindex {starting-block-id}

so then you can just tell the --reindex command which block number to start from if needed like in my case. Currently i will probably have to wait about 2 hours before it continues as i had about 100 blocks out of approx 2000 to go before my HDD became full. I switched HDD and copied all data across, yet now the reindex command is taking forever.

It has been running for 2 hours so far, and i estimate 2 more hours to go to reach the height i was last at on my other HDD.

Reindex should never be needed in normal operation.

Computer Crash

If your computer crashes regularly, you should fix your Computer, not use reindex.

Not leaving computer on for days

I fail to see how that requires reindex

Running out of disk space

I think Bitcoin Core could be smarter to not corrupt the datadir when running out of disk space. We shouldn't work on making reindex faster, but rather avoid the reindex in the first place.

block-hashes.dat

This idea doesn't work because each node stores blocks in different files, so the hashes wouldn't match. Assumeutxo is a similar idea, which is why I liked to that.

bitcoind --reindex {starting-block-id}

Again, a user having to use reindex is either due to a hardware corruption on their side (in which case we can't do anything about it) or due to a bug in Bitcoin Core (in which case we should fix the bug, not make reindex faster).

Hi @MarcoFalke, Ok thats sounds fine, my actual issue is i ran out of disk space, when i had about 100 blocks left to be downloaded to be up to date.

• I have now moved all files to a bigger HDD 1TB in size. In this case what command should i run to bring it all up to date with the other nodes without using the ---reindex command in order to download the final 100 Blocks?

Note: Originally i pruned the index. Yet i realised you cant use RPC to collect data correctly if you do that. So currently i have all blks from blk00001 up to blk01686 then i have a gap in numbers and it continues at blk01787 to blk02433 where the pruned files start from and end.

Is there a better solution to bring my node up to date other than using --reindex in this case?

p.s who came up with the idea to make the blk files not match numbers between nodes. That idea sounds very silly to me. No wonder reindexing is so slow. My last block took 55 seconds. I guess this might take a lot longer than i thought.

That idea sounds very silly to me

Usually the issue tracker is used to track technical issues related to the Bitcoin Core code base. General bitcoin questions and/or support requests are best directed to the Bitcoin StackExchange or the #bitcoin IRC channel on freenode.

• Do you have a better solution than --reindex or not?
• This is a technical issue this software is slow as hell.

• Never mind, i guess you was probably just lying to sound good lol....
• There is no better solution than --reindex is there?

There is no better solution than --reindex is there?

At this point no. Though people are working on various related projects such as assumeutxo.

who came up with the idea to make the blk files not match numbers between nodes.

There are some suboptimal decisions made by Satoshi back in 2009 that current day contributors and maintainers bear no responsibility for. This may be one of those examples.

Never mind, i guess you was probably just lying to sound good lol....

Please don't accuse people of lying when they are attempting to help you in good faith.

General bitcoin questions and/or support requests are best directed to the Bitcoin StackExchange or the #bitcoin IRC channel on freenode.

I would emphasize this.

Ok thats great thanks for your help. Sorry im just a little stressed and tired, the machine has be running for 48 hours+ non stop on this subject and i have been awake for most of that time.

Yes you should sack Satoshi for bad coding he probably thought it would be a good encryption effect by randomising the blks yet on the long run it just makes it slower to synchronise files / transactions.

One more question before i go. Who started github.com/bitcoin if satoshi has been awol since 2009.

I'll repeat :)

General bitcoin questions and/or support requests are best directed to the Bitcoin StackExchange or the #bitcoin IRC channel on freenode.

e.g. https://bitcoin.stackexchange.com/questions/28108/satoshis-final-statement

Blocks are saves to block files in the order they are downloaded. Some downloaded blocks may later turn out to be "stale" ( a branch with less total work than the main chain). Removing those blocks from the files after they have been written to complicates the code for no reason, but this is getting off-topic.