Support pruning from txindex.dat. #1405

pull TheBlueMatt wants to merge 2 commits into bitcoin:master from TheBlueMatt:prune changing 6 files +183 −0

TheBlueMatt commented at 10:26 PM on May 31, 2012: member

I did this to see how much it would actually prune, but, as it pruned just over half of the CTxIndexs in blkindex.dat, I thought I'd pull request it. It only prunes up to the latest checkpoint as there is no undoing pruning, so chain reorgs cant undo transactions that have CTxIns that were pruned.

Note that the pruning process with -prune takes a lot of time with -prune (an order of magnitude greater than loading chain on my tmpfs-backed system) and eats around 400MB of memory to load the needed information into memory. -autoprune is much less resource-intensive, but the checkpoint has to exist before the given block is downloaded, so users upgrading will not be effected unless they use -prune once.

You can also run IBD with -autoprune to prune at each checkpoint.
Diapolo commented at 6:15 PM on June 7, 2012: none

The "Add UI InitMessage for -loadblock." commit should get in asap, but be seperate from this one.

Support pruning from txindex.dat.

By launching with -prune to prune from genisis block to latest
checkpoint.
Or running initial block download with -autoprune to prune each
time you get a new checkpoint.

1a3a3c66f5

Add UI InitMessage for -loadblock. 24f4c50f29
TheBlueMatt commented at 10:17 PM on June 7, 2012: member

Finally got around to benchmarking IBD using -autoprune, it was a tiny sliver faster than master (just under 5%) -loadblock'ing to the current block, but its blkindex.dat was only 351MB instead of 554MB.
Thus, I made -autoprune default and rebased.
in src/db.cpp:None in 24f4c50f29
```
 653 | +        nTxsPruned++;
 654 | +    }
 655 | +
 656 | +    WriteHashBestCheckpoint(hashPruneTo);
 657 | +
 658 | +    //if (!TxnCommit())
```
Diapolo commented at 11:21 PM on June 7, 2012:

That Commented out code should be removed, if not used :).

TheBlueMatt commented at 11:55 PM on June 7, 2012:

Meh, it would be nice to have that, so it serves as a TODO.

laanwj commented at 8:36 AM on June 13, 2012:

If you add TODO: in the comment, many code editors will show it in an overview. Might be useful, and more clear to readers that it isn't just commented out code (otherwise, someone that doesn't know might remove it).

laanwj commented at 8:36 AM on June 13, 2012:

Oh, right, you did this above. Never mind.
sipa commented at 4:23 PM on June 8, 2012: member

No noticeable speedup here, but ACK on the idea. I'll have a look at the code later.
jgarzik commented at 6:22 PM on June 8, 2012: contributor

idea ACK. I like this!

in src/db.cpp:None in 24f4c50f29

 550 | @@ -535,6 +551,118 @@ CBlockIndex static * InsertBlockIndex(uint256 hash)
 551 |      return pindexNew;
 552 |  }
 553 |  
 554 | +bool CTxDB::PruneBlockIndex(uint256 hashPruneFrom, uint256 hashPruneTo)
 555 | +{
 556 | +//    TODO: assert here, but cant #include main.h

laanwj commented at 8:38 AM on June 13, 2012:

Why main.h? For assert you need the header <cassert> (which seems to be included in serialize.h)

TheBlueMatt commented at 2:23 PM on June 13, 2012:

main.h has hashBestBlock.

TheBlueMatt commented at 8:17 PM on July 5, 2012: member

Closing in favor of an (eventual) ultraprune. No point in merging a stop-gap that conflicts with an actual solution.
TheBlueMatt closed this on Jul 5, 2012
suprnurd referenced this in commit 27f3218de9 on Dec 5, 2017
suprnurd referenced this in commit b6804678f2 on Dec 5, 2017
lateminer referenced this in commit 9e7eaf205f on Jan 22, 2019
lateminer referenced this in commit d8663a8410 on May 6, 2020
DrahtBot locked this on Sep 8, 2021

Contributors

Linked (view graph)

#1486 Skip tons of duplicate GetHash() calls in Block Commit Thread