Prune Node Rescan Project Tracking #29183

1. 
   furszy commented at 8:31 pm on January 4, 2024: member

   This issue will be edited frequently to reflect the current status of the project.

   What should I review now?


   👇 👇 👇 👇 – #27837 – ☝️ ☝️ ☝️ ☝️

   The goal of this project is to enable complete blockchain rescan capability on prune nodes, allowing these storage-limited nodes to import new wallet descriptors and update existing ones at any time, as well as loading external and un-synced wallets with a birth-time older than 48 hours. This, in turn, enables the monitoring of new addresses/scripts and tracking balance changes without the need to repeatedly download the entire blockchain.

   The first project milestone introduces the feature for prune nodes that construct and maintain a local block filters set database. The node will request and keep track of the missing historical blocks that match the wallet scripts filters set to subsequently perform the rescan.

   The following sections are ordered by their PRs dependencies:

   Net: Single Block Requests Tracking System and On Download Failure Retry Mechanism

   • #28170
   • #28120
   • #30118
   • #27837

   Indexes Stability and Performance Improvements

   • #27607
   • #27720
   • #28955
   • #26966

   Wallet Rescan (depends on the net section)

   • #27469
   • Cache wallet filters’ set and connect it to the current wallet rescan functionality. (Pending)
   • Wallet: implement “rescan status” class to track the progress and retrieve it via an RPC command. (Pending)
   • Wallet: introduce prune node rescan functionality. (Pending - depends on net changes)

   Issues Solved By This Work

   • #21267
   • #23727
   • [ ] #27188
   • [ ] #27652
   • [ ] #27939
   • https://github.com/sparrowwallet/sparrow/issues/1111#issuecomment-2054204911
2. 
   chrisguida commented at 8:49 pm on January 23, 2024: none

   Does scanblocks work on pruned nodes yet? If not, does this project aim to add that capability?
3. 
   furszy commented at 10:04 pm on January 23, 2024: member

   Does scanblocks work on pruned nodes yet? If not, does this project aim to add that capability?

   scanblocks always worked on pruned nodes. Just need to enable -blockfilterindex since the node’s inception.

4. 
   chrisguida commented at 8:10 pm on January 24, 2024: none

   Excellent, thanks!!
5. 
   chrisguida commented at 8:11 pm on January 24, 2024: none

   What’s the process for building the blockfilterindex on an already-synced pruned node? Enable blockfilterindex, reindex?
6. 
   furszy commented at 10:30 pm on January 24, 2024: member

   What’s the process for building the blockfilterindex on an already-synced pruned node? Enable blockfilterindex, reindex?

   Yes. For further node support, please come to IRC or use bitcoin.stackexchange. This is a project tracking issue.

7. 
   shovas commented at 11:11 pm on February 8, 2024: none

   Here from #27188. I’m presently experimenting with wallets for bitcoin, litecoin, dogecoin, and bitcoin cash using linux cli tools so if I can help I’ll try.
8. 
   AndySchroder commented at 2:20 am on August 1, 2024: none

   Wondering if it makes sense to have a prune witness data only mode? Can’t #25957 work without witness data? Seems like pruning all witness data after the last 2,000 blocks would be a technically easy option with a high reward in disk space savings. I thought that was the reason we had the segwit discount (to encourage more prunable data be placed in the witness), but is that discount doesn’t even seem to be used by bitcoin core in a meaningful way in terms of pruning? Is there any open issue for pruning witness data?
9. 
   furszy commented at 3:06 pm on August 29, 2024: member

   Wondering if it makes sense to have a prune witness data only mode? Can’t #25957 work without witness data? Seems like pruning all witness data after the last 2,000 blocks would be a technically easy option with a high reward in disk space savings.

   Something like that wouldn’t be easy to implement with the current storage model. The witness data is tied to the transaction on disk in its serialized form, which is then tied to the block on disk in its serialized form. So, in order to discard the witness data, we would need to create a process that rewrites the block files entirely.

10. 
    murchandamus commented at 2:59 pm on January 23, 2025: contributor

    Rescan functionality for the wallet on the basis of client-side block filters would be an enormous win. Concept ACK
11. 
    AndySchroder commented at 3:50 pm on January 23, 2025: none

    One thing I have been thinking about this approach: Since we do need to selectively re-download specific blocks, it might be prudent to have an option only re-download those specific blocks from peers only over TOR. Since the total amount of data that needs to be downloaded is substantially lower than doing an IBD, this may not be that slow. Also, can the re-downloaded blocks be re-downloaded in parallel since we will know what blocks they are?
12. 
    furszy commented at 8:15 pm on February 10, 2025: member

    Since we do need to selectively re-download specific blocks, it might be prudent to have an option only re-download those specific blocks from peers only over TOR

    Thats possible for sure. Could also request a few extra blocks to make the guess harder.

    Also, can the re-downloaded blocks be re-downloaded in parallel since we will know what blocks they are?

    Yes. Thats how #27837 was implemented (blocks are requested in parallel - block processing is sync due to current net processing structure).

Contributors
furszy chrisguida shovas AndySchroder murchandamus