Prune-to mode (removable blockchain storage)

luke-jr commented at 6:38 am on July 26, 2017: member

It would be nice if pruning could be configured to move data rather than delete it, such that rescanning/etc would work when the removable (or perhaps simply slower) storage is available.

jonasschnelli commented at 6:46 am on July 26, 2017: contributor

Move would mean the block(files) would still be accessible from Bitcoin-Core but just moved to a different path (maybe a different filesystem, NAS, etc.)?

jonasschnelli added the label Block storage on Jul 26, 2017

jonasschnelli added the label Feature on Jul 26, 2017

luke-jr commented at 7:54 am on July 26, 2017: member

It seems it would be useful to support removable drives, eg for laptops.

rodentrabies commented at 7:51 pm on September 10, 2017: contributor

Started working on this and have some questions:

Should this be global (set in init.cpp alongside with -prune option or even -prune=/path/to/reserve/location) or per-command (for example bitcoin-cli pruneblockchain 1000 --to=/path/to/reserve/location)?
Should logic in validation.cpp be aware of this option, i.e. prunePath be passed alongside pruneHeight as parameter?

rodentrabies commented at 7:19 am on September 11, 2017: contributor

Assuming we should be able to work with blocks on the removable storage when it’s available, some validation logic should be aware of it, so one way of doing it is adding reserve storage location to the set of global flags in validation.cpp.

jnewbery commented at 9:00 pm on September 14, 2017: contributor

I have doubts about this concept. What exactly is the use case here?

a user who has limited disk space so they want to prune
but they have cold storage to back up the blocks that they prune (perhaps an external disk drive)
and that removable storage is attached whenever they’re running their node, so the blocks can be pruned to it
and when they want to rescan, the removable storage is attached so they can reload the blocks

so the removable storage is pretty much attached whenever they’re running bitcoind, which begs the question: why not just put the datadir on the removable storage?

Implementing this seems like a bunch of complication for not much benefit, or am I totally missing the point?

meshcollider commented at 10:18 pm on September 14, 2017: contributor

@jnewbery I guess the blocks would not be pruned unless it was attached, so that you could run the node without it and it would not prune anything until you attach the storage and pruneto it

luke-jr commented at 10:22 pm on September 14, 2017: member

What @MeshCollider said is one use case. Another would be archiving old blocks to non-SSD while keeping recent ones on SSD.

rodentrabies commented at 9:01 am on September 15, 2017: contributor

My personal usecase is that I have a home storage mounted on my laptop, so it would be great to prune blocks there and have the whole blockchain when I’m at home without needing to resync. Not that it’s a general problem, but still, such a feature would be pretty usefull.

TheBlueMatt commented at 6:16 pm on September 15, 2017: contributor

Hmm, better question, then, is what utility you gain from having the full blocks? If you dont need to rescan, you hopefully don’t need to access them, and you cant run with -txindex with pruning, so rescan is about the only thing you could even do with them. It might be cool to create a bootstrap.dat over time as your node prunes blocks, but reindex isn’t all that much faster than network sync, and hopefully we can continue to improve that anyway.

luke-jr commented at 6:20 pm on September 15, 2017: member

@TheBlueMatt In prune-to mode, you would be able to use txindex…

andronoob commented at 12:13 pm on February 18, 2019: none

I’m curious that if BitTorrent-like partial block storage is possible?

Currently, pruned nodes cannot serve historical blocks to a node which is bootstrapping.

Also, there’s something like spruned which allows a pruned node to provide more services.

andronoob commented at 7:27 am on July 28, 2020: none

I recently realized that this idea is actually “pluggable block storage”.

Hmm, better question, then, is what utility you gain from having the full blocks? If you dont need to rescan, you hopefully don’t need to access them

I think this might provide utility just like a peer serving historical blocks (in the case that local blockfilterindex enables P2P-based rescanning) - oh, it’s still rescanning.

Nobody likes blockchain rescanning. Users just want their wallet to work, especially after loading/importing wallets/keys/addresses - then the annoying blockchain rescanning process starts, so that the user has no choice but to wait.

andronoob commented at 7:34 am on July 28, 2020: none

What’s more, some users also want to store their blocks on multiple partitions of a disk (or multiple disks), in a case that one partition exhausted its free space - just like a “storage pool” or “spanned volume”.

willcl-ark commented at 2:31 pm on April 10, 2024: contributor

The feature request didn’t seem to attract much attention in the past. Also, the issue seems not important enough right now to keep it sitting around idle in the list of open issues.

Pull requests with improvements are always welcome.

willcl-ark closed this on Apr 10, 2024

Prune-to mode (removable blockchain storage) #10929