prune shall not delete blocks it did not download

zefir-k commented at 6:26 pm on May 23, 2024: none

Is there an existing issue for this?

I have searched the existing issues

Current behaviour

I operate a full node on an embedded system with datadir being on a local SSD but the blocks are stored on an external USB-HDD, which is often referred to be a well balanced compromise of speedy chainstate tracking and heavy blockchain data.

When setting up new nodes I usually build bitcoind from source and for a quick upstart I use to attach the USB-HDD containing the blocks to it, which saves the system from the initial blockchain download.

This particular time I needed an UI version and after running into dependency issues for compiling, I installed the current (v27.0.0) snap for Ubuntu. To reuse the existing blocks, I symlinked ~/snap/bitcoin-core/common/.bitcoin/blocks to the mounted blocks directory and allowed snap to use external mounts with snap connect bitcoin-core:removable-media.

bitcoin-core.qt starts, re-scans the blocks and then prunes my mounted blocks directory.

If bitcoin-core assumes pruning to be enabled by default, at least it should have asked for confirmation to delete blocks it did not download.

Expected behaviour

bitcoin-core should track which blocks it actually downloaded and only prune those. Blocks that were provided externally should not be deleted or at least should require a user’s confirmation.

Steps to reproduce

have an external HDD containing full blockchain
install snap of latest bitcoin-core (here: v27.0.0)
allow snap using external HDD with snap connect bitcoin-core:removable-media
symlink ~/snap/bitcoin-core/common/.bitcoin/blocks to the mounted blocks directory
run bitcoin-core.qt
observe that blocks are pruned on external HDD

Relevant log output

No response

How did you obtain Bitcoin Core

Package manager

What version of Bitcoin Core are you using?

v27.0.0

Operating system and version

Ubuntu 24.04 LTS

Machine specifications

No response

maflcko commented at 6:40 pm on May 23, 2024: member

Using the same blocksdir for two different nodes is not supported. Nodes may download blocks in a different order and save them to different locations in the blocksfiles. This will lead to an error at some point, latest when one of the nodes can’t find a block where it believes to be one.

Currently, I don’t think what you are trying to achieve is possible without copying blocks.

If bitcoin-core assumes pruning to be enabled by default, at least it should have asked for confirmation to delete blocks it did not download.

I don’t think pruning is enabled by default. It would be a bug if it was.

EDIT: Looks like prune is enabled by default in the GUI

maflcko added the label Block storage on May 23, 2024

maflcko added the label Questions and Help on May 23, 2024

zefir-k commented at 9:09 am on May 24, 2024: none

Using the same blocksdir for two different nodes is not supported. Nodes may download blocks in a different order and save them to different locations in the blocksfiles. This will lead to an error at some point, latest when one of the nodes can’t find a block where it believes to be one.

Currently, I don’t think what you are trying to achieve is possible without copying blocks.

Hm, my experience differs: using this regularly and never ran into issues. Since the external HDD contains fully downloaded and validated blockchain, there is not much to dissent between different nodes besides the very last potentially incomplete block. There is this tool (https://github.com/bitcoin-core/bitcoin-maintainer-tools/blob/main/fastcopy-chaindata.py) which works with hardlinks but can’t be used over filesystem boundaries - but generally it is possible.

If bitcoin-core assumes pruning to be enabled by default, at least it should have asked for confirmation to delete blocks it did not download.

I don’t think pruning is enabled by default. It would be a bug if it was.

Maybe problem is, I started bitcoin-core.qt first, which found my local SSD has insufficient space for the full blockchain and hence wrote a prune-enabled config, which I did not check when softlinking to my external blockchain. So I learned something for the price of a 2-day-IBD.

Proposal: don’t prune if blocksdir is defined - or at least document that if pruning is enabled your externally kept blockchain will get deleted.

maflcko commented at 11:55 am on May 24, 2024: member

Using the same blocksdir for two different nodes is not supported. Nodes may download blocks in a different order and save them to different locations in the blocksfiles. This will lead to an error at some point, latest when one of the nodes can’t find a block where it believes to be one. Currently, I don’t think what you are trying to achieve is possible without copying blocks.

Hm, my experience differs: using this regularly and never ran into issues. Since the external HDD contains fully downloaded and validated blockchain, there is not much to dissent between different nodes besides the very last potentially incomplete block. There is this tool (https://github.com/bitcoin-core/bitcoin-maintainer-tools/blob/main/fastcopy-chaindata.py) which works with hardlinks but can’t be used over filesystem boundaries - but generally it is possible.

Sure, but the script also intends to correctly handle blocks/index, as well as the last block file. I don’t see where your approach does it.