When Finding a corrupt block during re-index, download the corrupt block, continue reindex from volume #19274

<!-- Describe the issue -->

I recently had a corrupt bitcoin database. I've been trying to re-index the blockchain as I have a full node and the entire blockchain downloaded to my volume.

I'm seeing errors in the debug.log that I'm assuming are corrupt blocks. LoadExternalBlockFile: Deserialize or I/O error

LoadExternalBlockFile: Deserialize or I/O error - CBufferedFile::Fill: end of file: unspecified iostream_category error

LoadExternalBlockFile: Deserialize or I/O error - non-canonical ReadCompactSize(): unspecified iostream_category error

When the re-index reaches one of those blocks, it stops trying to re-index the blockchain and begins to sync over the network. It doesn't always stop immediately after finding the error. Sometimes it will continue to re-index, but when looking for the best tip, it will begin syncing with the network after getting to that block.

I'll delete the corrupt block, restart Bitcoin to re-index, hoping it will download the missing block and continue to re-index from my drive. I haven't seen that happen.

<!--- What behavior did you expect? -->

When finding a corrupt or missing block, replace that one block and continue re-indexing from the volume. Take priority for using blocks already on the volume rather than syncing over the network.

<!-- What version of Bitcoin Core are you using, where did you get it (website, self-compiled, etc)? -->

0.20

<!-- What type of machine are you observing the error on (OS/CPU and disk type)? -->

2010 Mac 5,1 10.13.6 os 2.8 GHz Quad-Core Intel Xeon 24 GB 1066 MHz DDR3 ATI Radeon HD 5870 1024 MB blockchain stored on an attached DroboPro with 5 TB free storage. The Drobo had a drive failure, I can lose another 2 drives before data loss. Drobo is rebuilding.

<!-- Any extra information that might be useful in the debugging process. -->

To test, delete an early block and start Bitcoin.

<!--- This is normally the contents of a `debug.log` or `config.log` file. Raw text or a link to a pastebin type site are preferred. -->

2020-06-14T12:50:20Z Reindexing block file blk00163.dat... 2020-06-14T12:51:03Z Loaded 518 blocks from external file in 42614ms 2020-06-14T12:51:03Z Reindexing block file blk00164.dat... 2020-06-14T12:51:26Z Loaded 606 blocks from external file in 22425ms 2020-06-14T12:51:26Z Reindexing block file blk00165.dat... 2020-06-14T12:52:04Z Loaded 492 blocks from external file in 38732ms 2020-06-14T12:52:04Z Reindexing block file blk00166.dat... 2020-06-14T12:52:29Z Loaded 518 blocks from external file in 24488ms 2020-06-14T12:52:29Z Reindexing block file blk00167.dat... 2020-06-14T12:53:10Z Loaded 595 blocks from external file in 41404ms 2020-06-14T12:53:10Z Reindexing block file blk00168.dat... 2020-06-14T12:53:32Z Loaded 573 blocks from external file in 22214ms 2020-06-14T12:53:32Z Reindexing block file blk00169.dat... 2020-06-14T12:53:37Z LoadExternalBlockFile: Deserialize or I/O error - CBufferedFile::Fill: end of file: unspecified iostream_category error 2020-06-14T12:53:37Z Loaded 42 blocks from external file in 5048ms 2020-06-14T12:53:37Z Reindexing block file blk00170.dat... 2020-06-14T12:53:49Z Reindexing block file blk00171.dat... 2020-06-14T12:54:05Z Reindexing block file blk00172.dat... 2020-06-14T12:54:17Z Reindexing block file blk00173.dat... 2020-06-14T12:54:29Z Reindexing block file blk00174.dat... 2020-06-14T12:54:45Z Reindexing block file blk00175.dat... 2020-06-14T12:54:57Z Reindexing block file blk00176.dat... 2020-06-14T12:55:10Z Reindexing block file blk00177.dat... 2020-06-14T12:55:25Z Reindexing block file blk00178.dat... 2020-06-14T12:55:37Z Reindexing block file blk00179.dat... 2020-06-14T12:55:54Z Reindexing block file blk00180.dat... 2020-06-14T12:56:13Z Reindexing block file blk00181.dat... 2020-06-14T12:56:31Z Reindexing block file blk00182.dat... 2020-06-14T12:56:47Z Reindexing block file blk00183.dat... 2020-06-14T12:57:01Z Reindexing block file blk00184.dat... 2020-06-14T12:57:15Z Reindexing block file blk00185.dat... 2020-06-14T12:57:32Z Reindexing block file blk00186.dat... 2020-06-14T12:57:46Z Reindexing block file blk00187.dat...

When performing a re-index, Bitcoin stops loading block files where it will begin to sync with network. And still spends time reindexing each block file that I already have downloaded. Example: During a re-index Bitcoin 1st reindexes all block files. There is a confirmation message, Loaded x blocks from external file in x ms. In my last -reindex pass at block file 865, I stopped seeing those confirmations. And yet, bitcoin will continue to look at each file down to 2108, which is the last block file I have downloaded. Then it begins to updateTip starting from the beginning blocks. When it gets to file 865, it stops updateTip, and begins to re-download blocks from the network.

Ideally, Bitcoin would see all my local blocks, and only download if a block in the middle is not complete.
Check local for block, if error, request network for download, check next local block.

Instead of, check local for block, error, download rest of blocks from network. Or at least stop reindexing if it's going to ignore all those other blocks.

2020-06-22T14:05:20Z Reindexing block file blk00862.dat... 2020-06-22T14:05:26Z Loaded 112 blocks from external file in 6682ms 2020-06-22T14:05:26Z Reindexing block file blk00863.dat... 2020-06-22T14:05:34Z Loaded 146 blocks from external file in 7456ms 2020-06-22T14:05:34Z Reindexing block file blk00864.dat... 2020-06-22T14:05:40Z Loaded 119 blocks from external file in 6144ms 2020-06-22T14:05:40Z Reindexing block file blk00865.dat... 2020-06-22T14:05:45Z Reindexing block file blk00866.dat... 2020-06-22T14:05:50Z Reindexing block file blk00867.dat...

The problem is not easily reproducible.

Please open a new issue (or leave a comment in here if you want this re-opened) if you experience the problem again.

seems similar suggestion as #17341. I still think that this might make sense, and with the getblockfrompeer rpc this should be possible without adding much code, especially after #27652 would be resolved.

Not sure if this is worth it. Block corruption should never happen or only rarely happen, so optimizing re-index doesn't seem worth it.

It seems better to optimize the hardware you are running on.

Not sure if this is worth it. Block corruption should never happen or only rarely happen, so optimizing re-index doesn't seem worth it.

Does it really happen less today than corruptions for which -reindex or -reindex-chainstate would help? Also, it seems that the data integrity requirements for block storage are lower than for other types of data, since the data is mostly used to help others sync: For example, it's not uncommon to store the /blocks directory on a external drive which may be slower and of less quality than the drive the chainstate and wallet are on.

The point of this request is not how a block was corrupted or the hardware it runs on. The frustrating part is when bitcoin is reindexing and finds a bad block somewhere in the middle of the stored blocks, it doesn't just download the one block and then try the next block on the disk. It redownloads all the blocks after the bad block. Regardless if those blocks are good. This was a couple versions ago. Maybe this has been fixed. You can test this by deleting a block in the stored blockchain and then running a reindex to see if it only replaces the lost block and then moves on to the next stored block on the local disk. Or does it count all the rest of the blocks on the blockchain as bad and request those blocks again from the network.

mostly used to help others sync: For example, it's not uncommon to store the /blocks directory on a external drive which may be slower and of less quality than the drive the chainstate and wallet are on.

Right, so what help is it to others to provide them with blocks slowly or not at all (when they are corrupted). It may be more helpful to peers to not store and offer any blocks in this case, so that fast and high quality peers can be used instead.

The point is decentralization.