Add -reindex, to perform in-place reindexing of block chain files #1943

sipa commented at 10:13 PM on October 21, 2012: member

Flushes the blktree/ and coins/ databases, and reindexes the block chain files, as if their contents was loaded via -loadblock.

Based on earlier work by Jeff Garzik.

This is a step towards automatic upgrade of pre-ultraprune databases as well.

jgarzik commented at 10:28 PM on October 21, 2012: contributor

Supercedes #1870

jgarzik commented at 10:43 PM on October 21, 2012: contributor

ACK

It would be nice if the code movement was pre-pended as a separate, first commit, but that's nit picking.

BitcoinPullTester commented at 12:06 AM on October 22, 2012: none

Automatic sanity-testing: FAILED BUILD/TEST, see http://jenkins.bluematt.me/pull-tester/fb917a38cc17d6484d2ffa93c6cdde6861d30ee2 for binaries and test log.

This could happen for one of several reasons:

It chanages paths in makefile.linux-mingw or otherwise changes build scripts in a way that made them incompatible with the automated testing scripts
It does not build on either Linux i386 or Win32 (via MinGW cross compile)
The test suite fails on either Linux i386 or Win32
The block test-cases failed (lookup the first bNN identifier which failed in https://github.com/TheBlueMatt/test-scripts/blob/master/FullBlockTestGenerator.java)

gmaxwell commented at 12:14 AM on October 22, 2012: contributor

Works good for me.. I abused this a little bit (running in valgrind, aborted runs, concurrent RPC activity) and haven't yet been able to break it. I haven't yet reviewed the code. (I have a lot of reading to do to fully catch up with ultraprune)

gmaxwell commented at 12:22 AM on October 22, 2012: contributor

Er. Perhaps I spoke a bit too fast. the reindex appears to have been aborted by recieving a new block from the network, which caused a wad of orpan blocks and then sync continued very slowly from a remote peer. Perhaps while reindex is active we should ignore new blocks from the network?

sipa commented at 8:48 PM on October 22, 2012: member

@jgarzik split the commit in two

sipa commented at 10:07 PM on October 23, 2012: member

@gmaxwell That's exactly what happens, by the way. While reindexing, fImporting is true, and the "block" message gets ignored entirely, and no IBD is initiated.

sipa commented at 6:43 PM on October 28, 2012: member

Rebased on top of #1962

sipa commented at 6:47 PM on October 28, 2012: member

By the way, this also swaps the order of -loadblock's and bootstrap.dat. Bootstrap.dat should be first.

BitcoinPullTester commented at 5:56 PM on October 30, 2012: none

Automatic sanity-testing: FAILED BUILD/TEST, see http://jenkins.bluematt.me/pull-tester/a31ed6aeece0b0201eb52fc566001f3f15dd1b98 for binaries and test log.

This could happen for one of several reasons:

It chanages paths in makefile.linux-mingw or otherwise changes build scripts in a way that made them incompatible with the automated testing scripts
It does not build on either Linux i386 or Win32 (via MinGW cross compile)
The test suite fails on either Linux i386 or Win32
The block test-cases failed (lookup the first bNN identifier which failed in https://github.com/TheBlueMatt/test-scripts/blob/master/FullBlockTestGenerator.java)

sipa commented at 8:46 PM on October 30, 2012: member

Rebased and tests fixed

gmaxwell commented at 5:41 PM on October 31, 2012: contributor

'Works' for me, though I'm still seeing it get disrupted by blocks on the network and end up taking forever.

sipa commented at 8:22 PM on October 31, 2012: member

Added support for continuing reindex across restarts, and improved output.

BitcoinPullTester commented at 1:04 AM on November 2, 2012: none

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/d515540a18935cd8c9a1901bf1c3dcf559fcf568 for binaries and test log.

sipa commented at 11:42 AM on November 4, 2012: member

Improved GUI feedback during reindex. @laanwj Can you also have a look?

laanwj commented at 12:07 PM on November 4, 2012: member

GUI changes look good

BitcoinPullTester commented at 11:52 PM on November 5, 2012: none

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/70b6ec491f6589dc66ee1ef11089527d9f3e75f9 for binaries and test log.

gavinandresen commented at 7:15 PM on November 8, 2012: contributor

ACK code changes (haven't had time to test thoroughly, but that shouldn't hold up the pull).

Add CBufferedFile b5d5f44c95

LoadExternalBlockFile switched to CBufferedFile 05d9726805

Move ThreadImport to init.cpp 7a5b7535bf

Add -reindex, to perform in-place reindexing of block chain files

Flushes the blktree/ and coins/ databases, and reindexes the
block chain files, as if their contents was loaded via -loadblock.

Based on earlier work by Jeff Garzik.

7fea484674

sipa referenced this in commit 485cf044ba on Nov 9, 2012

sipa merged this on Nov 9, 2012

sipa closed this on Nov 9, 2012

laudney referenced this in commit 17c0f123cb on Mar 19, 2014

in src/serialize.h:1320 in 7fea484674

1315 | +    }
1316 | +
1317 | +    // rewind to a given reading position
1318 | +    bool SetPos(uint64 nPos) {
1319 | +        nReadPos = nPos;
1320 | +        if (nReadPos + nRewind < nSrcPos) {

promag commented at 10:45 PM on May 29, 2019:

I think there's a bug here that happens If the buffer size is greater than the file size.

Change 2* to 40* in https://github.com/bitcoin/bitcoin/blob/c7cfd20a77ce57d200b3b9e5e0dfb0d63818abdc/src/validation.cpp#L4421

And then run with -reindex

KolbyML referenced this in commit 1b7d182bb5 on Dec 5, 2020

KolbyML referenced this in commit 46fad30bbb on Dec 5, 2020

DrahtBot locked this on Sep 8, 2021