Allow for rescan using block filters with pruned node #21267

1. 
   benthecarman commented at 1:19 pm on February 22, 2021: contributor

   After #15946 it should be possible to do a rescan using block filters with a pruned node. Blocks that are not stored could be requested from peers.
2. jonasschnelli added the label Wallet on Feb 27, 2021
3. jonasschnelli added the label Feature on Feb 27, 2021
4. 
   andronoob commented at 1:25 pm on July 11, 2021: none

   Related: #19463
5. 
   chrisguida commented at 4:05 pm on July 21, 2022: none

   Also related: #23727 and https://github.com/bitcoin/bitcoin/pull/23549
6. 
   Xekyo commented at 1:38 am on November 8, 2022: member

   Also related: https://github.com/bitcoin/bitcoin/pull/25957
7. 
   aureleoules commented at 3:32 pm on November 23, 2022: member

   I’m not sure how the wallet could initiate the download of the missing blocks considering the libbitcoin_wallet is not linked with any node related code, which is expected. Maybe an alternative could be to print the missing blocks after a rescanblockchain or importdescriptors and let the user manually run getblockfrompeer? (with an option to make it precious?)

   Such as:

    0./src/bitcoin-cli rescanblockchain 149000 170000
    1{
    2  "status": "failure",
    3  "start_height": 149000,
    4  "stop_height": 170000,
    5  "missing_blocks": [
    6    "000000000004b2eefb71b225f425b99266c81acee1dbe720400086f0bfa38674",
    7    "0000000000005e5fd51f764d230441092f1b69d1a1eeab334c5bb32412e8dc51",
    8    "0000000073cd77cb31cc16969808eb2cc426cfce6388ccb883de25671d410d87"
    9  ]
   10}
   

   That might be troublesome if there are a lot of missing blocks though.

8. 
   andronoob commented at 3:51 am on November 25, 2022: none

   That might be troublesome if there are a lot of missing blocks though.

   By design there will be false positives so that some privacy can be retained. Those possibly related (not necessarily missing) blocks should then be fetched from different peers to enhance that privacy protection, or even better, fetched through different Tor circuits.

9. 
   furszy commented at 2:30 pm on November 25, 2022: member

   I’m actually half-way (almost ready with the PoC) through implementing a sync mode that fetches the missing (pruned) blocks from peers based on whether they match or not the filters (by first syncing-up the filters-chain from the network if it was not previously stored on the node). Started it ~2 months ago. Will push the project PR in few weeks with all the details etc :). Then start decoupling it into smaller PRs.
10. 
    darosior commented at 10:57 am on March 22, 2023: member

    @furszy are you still working on this?
11. 
    furszy commented at 2:33 pm on March 22, 2023: member

    Yeah @darosior, thanks for the ping. Let’s update this issue.

    I do have a running version of this. And the complete project extends beyond this use case actually.

    As is often the case, the working path hasn’t been as straight as expected. I have been improving orthogonal components in parallel to make the end result cleaner and more usable. For example, to make the mainnet live tests less painful, have created #26966 which reduces the time it takes for the block filter index to sync from ~7 hours down to less than 45 minutes. Although this sidetracked me from the project’s end goal, it was ultimately worth the effort.

    Also have to add that, while the ‘all-in-one’ project branch is functional, it still could benefit from some refactoring and consolidation rounds. And to address it, have started decoupling parts of it into sub-projects. The mentioned #26966 is one of them, all the refactoring commits were extracted from that branch (so it would be great if that PR starts getting reviewed).

    The plan is to keep pushing sub-projects forward so that the overall picture becomes clearer. Just matter of staying on track and continue making progress.

12. 
    furszy commented at 9:01 pm on June 18, 2023: member

    Two more PRs added to the public ground-work queue:

    • #27837: This work will allow us to keep track of the filter matching requested blocks. It is dependent on #27836, which also contain parts of the blocks fetching process in the all-in-one branch like the random peer selection method and the FetchBlock error handling for network limited peers.
    • #27607: This work, along with #27720 and #27708 (both of which have already been merged), aims to simplify #24230. The focus here is to prepare the ground for #26966 and the storage of block filters coming from the wire.

    Will keep pushing progress.

13. 
    GregTonoski commented at 8:56 pm on June 22, 2023: none

    Is there an initial version for testing, perhaps?
14. 
    furszy commented at 10:26 pm on June 22, 2023: member

    Not public yet mostly because some of the public PRs decoupled from the all-in-one branch have led to additional improvements and bug fixes throughout the sources. Which has been great for the software as a whole but they have been time consuming. Additionally, I’m not satisfied with the high false-positive rate of the filters (our wallet produces 8k scripts from the beginning), which was one of the motivations behind #27469 and want to do more before sharing it.

    Essentially, I’m preferring this incremental approach as it sums value to the release cycle, and lets me solidify the bases of the work while incorporate new improvements that arise during the review process. It might not be that “visible” but, if you check all the linked PRs, it has been quite fruitful so far.

    Promise to ping here once the long branch is available.

Contributors
benthecarman andronoob chrisguida Xekyo aureleoules furszy darosior GregTonoski

Labels
Feature Wallet