Multiwallet rescan sequentially scans multiple wallets instead of in parallel

gmaxwell commented at 1:32 am on November 23, 2017: contributor

If you start up with a collection of N out of sync wallets bitcoin will perform N interdependent rescans; this can be rather slow e.g. if they’re all at height 200k.

This should be fixed, or we should offer suicide counseling at a minimum.

fanquake added the label Wallet on Nov 23, 2017

promag commented at 11:28 pm on February 21, 2018: member

Wouldn’t be better to rescan the chain (load blocks, etc) only once and give it to each wallet to concurrently check for transactions?

luke-jr commented at 11:48 pm on February 21, 2018: member

Indeed, it’s been on my todo list (for after multiwallet is completely merged) for a while…

ryanofsky commented at 6:46 pm on June 1, 2020: member

@ariard has an experimental implementation of this which not only rescans wallets in parallel, but indexes as well

https://gist.github.com/ariard/89f9bcc3a7ab9576fc6d15d251032cfa https://github.com/bitcoin/bitcoin/compare/master...ariard:2019-07-rework-rescan https://github.com/bitcoin/bitcoin/compare/master...ariard:2019-08-rescan-index-refactor

ryanofsky referenced this in commit f8d600c3e4 on Jun 1, 2020

ryanofsky referenced this in commit c0ea9e57d2 on Jun 5, 2020

ryanofsky referenced this in commit 175a412cc6 on Jun 5, 2020

ryanofsky referenced this in commit 47b3b7a4f8 on Jun 5, 2020

ryanofsky referenced this in commit 72a825d3ef on Jun 5, 2020

ryanofsky commented at 3:33 pm on June 9, 2020: member

@pstratem Just catching on some of your wallet rescan comments in IRC. Want to make sure you’re aware we’d like to remove ScanForWalletTransactions and replace it with parallel scanning code here:

https://github.com/ariard/bitcoin/blob/2019-08-rescan-index-refactor/src/node/rescan.h https://github.com/ariard/bitcoin/blob/2019-08-rescan-index-refactor/src/node/rescan.cpp

Some more links in #11756 (comment)

ariard commented at 11:31 pm on June 15, 2020: member

@ryanofsky You may be interested by https://github.com/ariard/bitcoin/tree/2020-02-wallet-standalone, which was a starting clean-up of linked above branches.

I was aiming to go further and split the server (a.k.a indexes) in its own process, like bitcoin-server but was waiting for Core dev meetup before going further as it could have been subject to discussion in the approach. Generally I think we can dissociate headers servicing from addresses syncing, like the wallet would register a list of address and that would be the job of the server to notify the wallet back. Instead of sending whole blocks to wallet.

That would be a trade-off between more storage for maintaining addresses index compare to performance but maybe worthy of it.

With regards to threading model, you may have 1) one thread syncing in-parallel multiple wallets, based on lower registration or 2) a thread pool, each assigns to a wallet for sync. An issue with 1) is if Wallet A is at block 100 and Wallet B and C are sync to block 10000, they will have to starve until A has catched-up with them instead of making progress on their own.

ryanofsky referenced this in commit 8125ed13fc on Jul 10, 2020

ryanofsky referenced this in commit ae5d16ac7a on Jul 11, 2020

ryanofsky referenced this in commit feb4c159e2 on Jul 11, 2020

ryanofsky referenced this in commit 80715b397f on Jul 13, 2020

ryanofsky referenced this in commit eae4592ac4 on Jul 14, 2020

fscemama commented at 1:27 am on July 24, 2020: none

…

With regards to threading model, you may have 1) one thread syncing in-parallel multiple wallets, based on lower registration or 2) a thread pool, each assigns to a wallet for sync. An issue with 1) is if Wallet A is at block 100 and Wallet B and C are sync to block 10000, they will have to starve until A has catched-up with them instead of making progress on their own.

And an issue if new addresses are created. Could be worked around, yet. I like your solution, because it can be made in short term. Threads would come later (and really help).

Thanks all for your efforts. The topic is getting critical to us. Currently, any wallet rescan freezes the use of bitcoind for all other wallets. It can take 30 minutes to re-sync a wallet that has not been loaded for 6 months.

ryanofsky referenced this in commit ba3cdcc8a0 on Aug 7, 2020

ryanofsky referenced this in commit b4b381b422 on Aug 7, 2020

ryanofsky referenced this in commit ad8d69fd6e on Aug 17, 2020

ryanofsky referenced this in commit 097168853e on Aug 17, 2020

ryanofsky referenced this in commit 3013f972bc on Sep 2, 2020

ryanofsky referenced this in commit efab0d88b2 on Sep 2, 2020

ryanofsky referenced this in commit 70ed802e90 on Sep 25, 2020

ryanofsky referenced this in commit 35205452b5 on Sep 25, 2020

ryanofsky referenced this in commit 9e1abdbdfb on Apr 11, 2021

ryanofsky referenced this in commit 59cd6f8e6a on Apr 11, 2021

ryanofsky referenced this in commit 08d2529787 on May 19, 2021

ryanofsky referenced this in commit 9d7c3881f3 on May 19, 2021

ryanofsky referenced this in commit 53e94024b8 on Jun 14, 2021

ryanofsky referenced this in commit 96c2246bdf on Jun 14, 2021

ryanofsky referenced this in commit 554c3edef8 on Jun 14, 2021

ryanofsky referenced this in commit 90bf033d71 on Jun 15, 2021

ryanofsky referenced this in commit 7bbec573f6 on Aug 19, 2021

ryanofsky referenced this in commit 2ef0b64f42 on Aug 19, 2021

ryanofsky referenced this in commit 8c4d9fe797 on Sep 3, 2021

ryanofsky referenced this in commit 128132b0a8 on Sep 3, 2021

ryanofsky referenced this in commit bab7a0a52e on Oct 5, 2021

ryanofsky referenced this in commit 0d5d3370e0 on Oct 5, 2021

ryanofsky referenced this in commit 11c22a6cbb on Oct 13, 2021

ryanofsky referenced this in commit 9facba601c on Oct 13, 2021

ryanofsky referenced this in commit 9273bdeee1 on Nov 29, 2021

ryanofsky referenced this in commit d6b8f93fb7 on Nov 29, 2021

ryanofsky referenced this in commit 6245cdadc9 on Nov 29, 2021

Multiwallet rescan sequentially scans multiple wallets instead of in parallel #11756