Add -reindex, to perform in-place reindexing of blockchain data files #1870

1. 
   jgarzik commented at 9:01 PM on September 26, 2012: contributor

   Hopefully this is done in a database-independent manner, with minimal interference to the upcoming ultraprune.

2. 
   BitcoinPullTester commented at 2:56 AM on September 27, 2012: none

   Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/68e760d2656716cc9d538567c0735c77d8e5749e for binaries and test log.

3. in src/main.cpp:None in 68e760d265 outdated

   2329 | +    }
   2330 | +}
   2331 | +
   2332 | +void ReindexBlockchainPrep()
   2333 | +{
   2334 | +    filesystem::path pathIdx = GetDataDir() / "blkindex.dat";
   

   ---

   ---

   

   sipa commented at 2:01 PM on September 28, 2012:

   I really wouldn't delete bdb database files yourself. Use the dbenv methods to do so.

   ---

   

   gavinandresen commented at 9:21 PM on October 1, 2012:

   I agree with sipa, dbenv seemed to get very confused when I renamed database files behind its back for my corrupt wallet.dat handling code.

4. 
   sipa commented at 2:07 PM on September 28, 2012: member

   Surprising how little code changes are necessary for this. Ultraprune already has a CDiskBlockPos however, which is also used inside CTxBlockPos, so maybe you can use that instead? I think you can just copy it, and remove the IMPLEMENT_SERIALIZE.

5. 
   sipa commented at 10:05 PM on October 1, 2012: member

   I also wonder about this: what if the current best chain happens to be known in the new/repained blockchain? There should be no need to rebuild the entire index in that case, only the block positions.

   EDIT: right, you can't, as in the current database structure, transactions also contain disk positions. It's something that can be done post-ultraprune though.

6. 
   jgarzik commented at 7:41 PM on October 3, 2012: contributor

   Updated with the following changes:

   • uses CDiskBlockPos from sipa's ultraprune branch
   • calls our CDBEnv to perform database removal
7. 
   sipa commented at 1:50 PM on October 7, 2012: member

   Have you tested this?

   $ ./bitcoind -reindex
   DB_ENV->dbremove: method not permitted before handle's open method
   

8. 
   gavinandresen commented at 4:37 PM on October 7, 2012: contributor

   I wrote up a 4-step test plan that I've been working through.

   I also have a branch (not published) to reconcile this with my corrupt wallet handling pull; @jgarzik see https://github.com/gavinandresen/bitcoin-git/commits/BDB_RECOVER

   The testplan I was working through:

   1. Run new testnet3-in-a-box instance, let it IBD, shut it down. Remove testnet3/database/ directory. EXPECT: automatic re-index of blk0001.dat on startup.
   2. Same a (1), but run with -detachdb. Corrupt blkindex.dat by truncating it by a few bytes, then restart. EXPECT: automatic re-index of blk0001.dat on startup.
   3. Same as (2), but corrupt some bytes instead of truncating the file. EXPECT: automatic re-index.
   4. Compile a bitcoind against a later version of BDB (e.g. BDB 5 instead of 4). Run without -detachdb. Then startup a bitcoind compiled against the earlier BDB version. EXPECT: Something Reasonable-- not a BDB_RUNRECOVERY uncaught exception/crash.
   5. Same as (4), run with -detachdb. EXPECT: Same something reasonable as (4).

   I haven't tested 4/5 yet, and don't know enough about BDB major version compatibility to know what is reasonable to expect.

9. 
   jgarzik commented at 5:02 AM on October 9, 2012: contributor

   Doing the database remove in the environment definitely creates a super-large removal transactions, yet does not actually remove blkindex.dat. blk0002.dat is growing here, which should not be the case. Further debugging is needed.

10. 
    gavinandresen commented at 3:20 PM on October 9, 2012: contributor

    Instead of: dbenv.dbremove(NULL, strFile.c_str(), "main", DB_AUTO_COMMIT); you might try dbenv.dbremove(NULL, strFile.c_str(), NULL, DB_AUTO_COMMIT); ... then re-create blkindex.dat using TxDb("cr")

    I think passing NULL as the third argument removes the file.

11. Add -reindex, to perform in-place reindexing of blockchain data files

    Caveat: a new genesis block is added to blkNNNN.dat with each run,
    thanks to LoadBlockIndex() creating one, when it creates a blank block index.

    f49f263185
12. 
    jgarzik commented at 3:45 AM on October 10, 2012: contributor

    Fixed a few bugs, and rebased on top of @gavinandresen 's ::RemoveDb(). blkindex.dat is now properly removed, and reindex proceeds as expected. Things appear to be working now.

    Caveat: a genesis block is newly stored to blkNNNN.dat, for each -reindex invocation. LoadBlockIndex() adds a new one to the (it thinks) newly created block file. All other blocks are properly read and processed in-place.

    -reindex highlights the poor speed of BDB indexing alone... and should provide a useful apples-to-apples basis for comparing the old BDB system with ultraprune.

13. 
    sipa commented at 9:58 PM on October 20, 2012: member

    Needs rebase.

14. 
    BitcoinPullTester commented at 10:33 PM on October 20, 2012: none

    Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/f49f2631856a4a233a957ce8361ffab3b08561cc for binaries and test log.

15. 
    jgarzik commented at 10:28 PM on October 21, 2012: contributor

    Superceded by #1943, closing

16. jgarzik closed this on Oct 21, 2012

    ---

17. jgarzik deleted the branch on Aug 24, 2014
18. KolbyML referenced this in commit 696e328d8c on Dec 5, 2020
19. KolbyML referenced this in commit 3c05a7bd2d on Dec 5, 2020
20. DrahtBot locked this on Sep 8, 2021

Contributors

jgarzikBitcoinPullTestersipagavinandresen

Milestone
0.8.0