Make rescans faster

jamesob commented at 8:20 pm on December 9, 2021: member

This issue aggregates ideas for making wallet rescans substantially faster.

Using compact block filters, when available, could substantially speed up wallet rescans although it is not clear to me how this might work for non-descriptor-based (legacy) wallets.

Currently, #20664 proposes to add an RPC that will use compact block filters to scan for blocks relevant to a given script descriptor. This gives a listing of blocks that correspond to the descriptor, but the user then has to do something with that list of blocks to find transactions relevant to their wallet.

It would be nice to integrate scanblocks’ use of block filters opaquely into rescanblockchain (wallet rescanning) so that if block filters are available, the faster method of consulting them could be used instead of scanning the entire chain, or forcing the user to do something more roundabout with the output of scanblocks.

From @sipa:

regarding faster rescanning, i think there is some lower-hanging fruit, like (a) rescanning all wallets at startup in parallel rather than one by one and (b) building a set of scriptPubKeys to look for and using that for (legacy wallet) rescanning, instead of calling IsMine for each (especially with achow101’s migrate-to-descriptor code, i think there should be a way of computing that set for legacy wallets too)

Possible changes

integrate compact block filter usage into rescanblockchain to allow faster rescans when the compact block index is available, otherwise falling back to the standard scanning method. A first approach could possibly use filters only when dealing with a non-legacy descriptor wallet
- Currently proposed in #23549
allow pruned nodes with the block filter index to rescan quickly by getting a list of relevant blocks to the wallet, requesting the download of blocks, and then parsing the blocks for relevant transactions
rescanning all wallets at startup in parallel (instead of one-by-one): #11756
building a set of scriptPubKeys to look for when using a legacy wallet based on @achow101’s migrate-to-descriptor code

jamesob added the label Feature on Dec 9, 2021

ryanofsky commented at 9:51 pm on December 9, 2021: contributor

[ ] rescanning all wallets at startup in parallel (instead of one-by-one)

This is #11756, and I’m working on it now, but a little indirectly. Specifically I’m:

Porting BaseIndex::ThreadSync and other index code to sync with interfaces::Chain instead of calling validation code directly. The motivation is to let indexes be used with #10102 and run in different processes, but this also adds a new chain sync feature that can be used by wallets.
Introducing a block reader class to help with parallel rescans and pass same block data to multiple indexes.
Going to rebase #15719 on top of this rescan code so wallets can use the same chain sync interface as the indexes

There’s some uncertainty and performance tradeoffs in step 2. Right now my block reader implementation just reads 2 blocks at a time so previous block can be processed while next block is read. But it might be better to let indexes and wallets read blocks more freely without trying to schedule things.

maflcko commented at 11:25 am on December 10, 2021: member

although it is not clear to me how this might work for non-descriptor-based (legacy) wallets.

Something like #15845 (comment) should do that

maflcko added the label Wallet on Dec 10, 2021

maflcko commented at 11:44 am on December 15, 2021: member

Make rescans faster #23727

Possible changes