Prune Node Rescan Project Tracking #29183

1. 
   furszy commented at 8:31 pm on January 4, 2024: member

   This issue will be edited frequently to reflect the current status of the project.

   What should I review now?


   👇 👇 👇 👇 – #27837 – ☝️ ☝️ ☝️ ☝️

   The goal of this project is to enable complete blockchain rescan capability on prune nodes, allowing these storage-limited nodes to import new wallet descriptors and update existing ones at any time, as well as loading external and un-synced wallets with a birth-time older than 48 hours. This, in turn, enables the monitoring of new addresses/scripts and tracking balance changes without the need to repeatedly download the entire blockchain.

   The first project milestone introduces the feature for prune nodes that construct and maintain a local block filters set database. The node will request and keep track of the missing historical blocks that match the wallet scripts filters set to subsequently perform the rescan.

   The following sections are ordered by their PRs dependencies:

   Net: Single Block Requests Tracking System and On Download Failure Retry Mechanism

   • #28170
   • #28120
   • #30118
   • #27837

   Indexes Stability and Performance Improvements

   • #27607
   • #27720
   • #28955
   • #26966

   Wallet Rescan (depends on the net section)

   • #27469
   • Cache wallet filters’ set and connect it to the current wallet rescan functionality. (Pending)
   • Wallet: implement “rescan status” class to track the progress and retrieve it via an RPC command. (Pending)
   • Wallet: introduce prune node rescan functionality. (Pending - depends on net changes)

   Issues Solved By This Work

   • #21267
   • #23727
   • [ ] #27188
   • [ ] #27652
   • [ ] #27939
   • https://github.com/sparrowwallet/sparrow/issues/1111#issuecomment-2054204911
2. 
   chrisguida commented at 8:49 pm on January 23, 2024: none

   Does scanblocks work on pruned nodes yet? If not, does this project aim to add that capability?
3. 
   furszy commented at 10:04 pm on January 23, 2024: member

   Does scanblocks work on pruned nodes yet? If not, does this project aim to add that capability?

   scanblocks always worked on pruned nodes. Just need to enable -blockfilterindex since the node’s inception.

4. 
   chrisguida commented at 8:10 pm on January 24, 2024: none

   Excellent, thanks!!
5. 
   chrisguida commented at 8:11 pm on January 24, 2024: none

   What’s the process for building the blockfilterindex on an already-synced pruned node? Enable blockfilterindex, reindex?
6. 
   furszy commented at 10:30 pm on January 24, 2024: member

   What’s the process for building the blockfilterindex on an already-synced pruned node? Enable blockfilterindex, reindex?

   Yes. For further node support, please come to IRC or use bitcoin.stackexchange. This is a project tracking issue.

7. 
   shovas commented at 11:11 pm on February 8, 2024: none

   Here from #27188. I’m presently experimenting with wallets for bitcoin, litecoin, dogecoin, and bitcoin cash using linux cli tools so if I can help I’ll try.
8. 
   AndySchroder commented at 2:20 am on August 1, 2024: none

   Wondering if it makes sense to have a prune witness data only mode? Can’t #25957 work without witness data? Seems like pruning all witness data after the last 2,000 blocks would be a technically easy option with a high reward in disk space savings. I thought that was the reason we had the segwit discount (to encourage more prunable data be placed in the witness), but is that discount doesn’t even seem to be used by bitcoin core in a meaningful way in terms of pruning? Is there any open issue for pruning witness data?
9. 
   furszy commented at 3:06 pm on August 29, 2024: member

   Wondering if it makes sense to have a prune witness data only mode? Can’t #25957 work without witness data? Seems like pruning all witness data after the last 2,000 blocks would be a technically easy option with a high reward in disk space savings.

   Something like that wouldn’t be easy to implement with the current storage model. The witness data is tied to the transaction on disk in its serialized form, which is then tied to the block on disk in its serialized form. So, in order to discard the witness data, we would need to create a process that rewrites the block files entirely.

10. 
    murchandamus commented at 2:59 pm on January 23, 2025: member

    Rescan functionality for the wallet on the basis of client-side block filters would be an enormous win. Concept ACK
11. 
    AndySchroder commented at 3:50 pm on January 23, 2025: none

    One thing I have been thinking about this approach: Since we do need to selectively re-download specific blocks, it might be prudent to have an option only re-download those specific blocks from peers only over TOR. Since the total amount of data that needs to be downloaded is substantially lower than doing an IBD, this may not be that slow. Also, can the re-downloaded blocks be re-downloaded in parallel since we will know what blocks they are?
12. 
    furszy commented at 8:15 pm on February 10, 2025: member

    Since we do need to selectively re-download specific blocks, it might be prudent to have an option only re-download those specific blocks from peers only over TOR

    Thats possible for sure. Could also request a few extra blocks to make the guess harder.

    Also, can the re-downloaded blocks be re-downloaded in parallel since we will know what blocks they are?

    Yes. Thats how #27837 was implemented (blocks are requested in parallel - block processing is sync due to current net processing structure).

13. glozow added the label Tracking Issue on Oct 15, 2025
14. 
    darosior commented at 2:50 pm on February 18, 2026: member

    I’m curious if a first step of simply downloading blocks for the desired interval, instead of using block filters, was considered. We already have access to a validated tip and a header chain, so we could download past blocks and scan them without re-validating them again. A second step could weigh whether introducing the complexity of client side filtering is worth the reduction in bandwidth usage. Finally, because this method would indiscriminately download blocks like in IBD, it would not be necessary to try and use an anonymization network.
15. 
    Sjors commented at 4:50 pm on February 18, 2026: member

    I’m curious if a first step of simply downloading blocks for the desired interval

    Essentially just unpruning all the way to the block where the wallet was last synced? That seems reasonable, but you’d want some user interaction if it’s a big download.

    Perhaps a simple unprune HEIGHT RPC could be added? And then when wallet loading fails, it could suggest something like Try the unprune HEIGHT rpc if you have X GB of free space (anything is better than the current reindex suggestion).

    The unprune RPC could prevent pruning while the node is running, so the user can do their wallet rescan. At the next restart things get pruned back, unless they change the -prune setting.

16. 
    darosior commented at 5:34 pm on February 18, 2026: member

    Not really, what i had in mind was rather to only temporarily download those blocks. That could be done in chunks of a day worth of blocks for instance (144 blocks), or anything that would be minimal disk space overhead anyways compared to the current size of the chainstate.
17. 
    Sjors commented at 5:40 pm on February 18, 2026: member

    Oh you mean a rolling window of ephemeral downloads during the rescan? That makes sense too, though we also wouldn’t want to do that without prompting the user, since it can be a long and slow download.
18. 
    furszy commented at 10:00 pm on February 18, 2026: member

    I’m curious if a first step of simply downloading blocks for the desired interval, instead of using block filters, was considered. We already have access to a validated tip and a header chain, so we could download past blocks and scan them without re-validating them again.

    The next unmerged step along this path adds the ability to request, track, and re-attempt block downloads (#27837), which is essentially what you are asking for, just extended with an interval locator on top. The idea there was to separate the logic responsible for downloading best-chain blocks up to the tip from the logic responsible for downloading specific blocks, with the goal of having two structures answering the same question “what should download next” in different ways and with different priorities.

    Re “simply download blocks from the desired interval” approach: I have always felt this wasn’t really a complete feature, but more of a half-baked implementation of the real one, mainly because of the excessive bandwidth it would consume for older wallets. That said, it is doable.

    The rolling/ephemeral window approach wouldn’t work as you imagine. The wallet isn’t loaded (we fail during load on prune nodes), so we can’t notify and discard the blocks.

    There is a good way we can implement this that is way more space-efficient than that. Basically, irrelevant blocks can be discarded immediately by storing the wallet’s elements set (scripts hash-mapped into the filter space) within the interval tracking mechanism. This doesn’t require a wallet to be loaded at all, and it enables additional use cases like running scanblocks on prune nodes and letting users query for txs relevant to given script(s) on the fly on prune nodes too.

    If we have enough review power for this, I can update+rework #27837 (or open a new PR) to move us one step closer to the requested functionality, but we should coordinate to make sure we are using our time wisely.

19. 
    AndySchroder commented at 5:01 am on February 19, 2026: none

    A second step could weigh whether introducing the complexity of client side filtering is worth the reduction in bandwidth usage

    With blocks already downloaded, I can do a a full chain re-scan in a fairly short amount of time. See https://github.com/sparrowwallet/sparrow/issues/1368#issuecomment-2050047410 for some data. It seems to me if we are just re-scanning a specific wallet, the time would be much shorter. If a wallet only has 200 transactions and does need to download some blocks, I would think that would mean only 200 blocks need to be downloaded at most, right?

Contributors
furszy chrisguida shovas AndySchroder murchandamus darosior Sjors

Labels
Tracking Issue