wallet, rpc: add UTXO set check and incremental rescan to importdescriptors

musaHaruna commented at 4:17 pm on September 15, 2025: none

When importing descriptors, users may accidentally provide an incorrect birthdate (timestamp). This can cause the wallet to miss relevant historical transactions, leading to incorrect or incomplete balances. Currently, the wallet only relies on rescans starting from the provided timestamp.

This PR extends the importdescriptors RPC with a new optional argument: scan_utxoset (bool, default=false): If enabled, the wallet will compare its calculated trusted balance against UTXO set balance (by generating the scriptpukeys of the wallet and comparing it with the chains UTXO set scriptpubkeys to get the accurate balance belonging to the wallet and comparing it with the wallet trusted balance).

If the balances match, import continues as normal. If a discrepancy is detected, the wallet will attempt incremental rescans in chunks of recent blocks until the missing history is found. If the wallet is pruned, incremental rescans will not go earlier than the prune boundary.

Additional information is returned in the RPC response under an "info" object when scan_utxoset is used, such as: utxo_check: whether the UTXO set matched the wallet balance scanned_chunks: number of incremental rescan chunks attempted scanned_blocks: approximate number of blocks scanned during incremental rescans

This helps detect and fix balance mismatches caused by wrong descriptor timestamps.

A big thanks to fjahr for suggesting this approach comment

node: add and implement FindCoinsByScript for UTXO-set scan by script

Add node::FindCoinsByScript and wire it into the node interface as
Node::getCoinsByScript(), then implement the DB-backed scan that iterates
the on-disk UTXO DB with a CCoinsViewCursor.

ba9f134734

DrahtBot added the label RPC/REST/ZMQ on Sep 15, 2025

DrahtBot commented at 4:17 pm on September 15, 2025: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33392.

Reviews

See the guideline for information on the review process. A summary of reviews will appear here.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#33135 (wallet: warn against accidental unsafe older() import by Sjors)
#32861 (Have createwalletdescriptor auto-detect an unused(KEY) by Sjors)
#32652 (wallet: add codex32 argument to addhdkey by roconnor-blockstream)
#29136 (wallet: addhdkey RPC to add just keys to wallets via new unused(KEY) descriptor by achow101)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

LLM Linter (✨ experimental)

Possible typos and grammar issues:

“If true, scans the UTXO set balance and compare with wallet balance and triggers incremental rescan if discrepency is found.” -> “If true, scans the UTXO set balance and compares with wallet balance and triggers incremental rescan if discrepancy is found.” [subject-verb agreement (“compare” -> “compares”) and spelling (“discrepency” -> “discrepancy”)]
“# Mine 1000 total 1000 more blocksthen send the second tx” -> “# Mine 1000 more blocks, then send the second tx” [typo/duplication (“total 1000 1000”) and missing space in “blocksthen”; rewritten for clarity]

drahtbot_id_5_m

musaHaruna force-pushed on Sep 16, 2025

fjahr commented at 2:17 pm on September 16, 2025: contributor

It seems kind of weird to me to add this as an option to getbalance. The problem is in the importdescriptors call, which is using the wrong birthdate. Have you looked into making this an option of importdescriptors? If the performance isn’t too bad, this could even be on by default. If that is possible that would seem preferred, since you also write “However, users may not realize that their wallet balance is incomplete.”, if they don’t know they likely also won’t call the getbalance with this option. So ideally we could “force” this on users in a way that they don’t feel (much) negative impact from it. If we can’t do that then I am not sure this is of much use since the users could just call rescanblockchain directly if they are aware that there is a problem with their balances.

musaHaruna commented at 4:13 pm on September 16, 2025: none

Have you looked into making this an option of importdescriptors?

I have not looked into making it an option for importdecriptors, but I will look into it, and get back to you on that.

Just thinking on a high level and somewhat naively because am a new contributor with little knowledge about the codebase, what if we can atomatically cross check the balance by scanning the utxoset in importdescriptor before triggering the rescan, incase of wrong birthtime, seeing that even if they use the new flags in getbalance, they might/will eventually have to run rescanblockchain.

Thanks for the suggestion. I really appreciate it.

musaHaruna commented at 7:45 pm on September 16, 2025: none

I have not looked into making it an option for importdecriptors, but I will look into it, and get back to you on that.

Yes, I have looked into adding the option to importdescriptors by introducing a scan_utxoset flag and it’s possible. The idea is that when enabled, the wallet would scan the UTXO set immediately after import to verify balances against chainstate, and if a discrepancy is detected (for example due to an incorrect birthdate), it could automatically trigger a full rescan from height 0 to restore missing history. This way, users wouldn’t have to manually diagnose incomplete balances — the import flow itself would handle it. What Do you think this approach?.

I don’t yet know the full impact on performance, but I’m thinking of making the trade-offs very clear in the RPC docs so users can decide whether to enable it.

fjahr commented at 8:53 pm on September 16, 2025: contributor

Yes, I have looked into adding the option to importdescriptors by introducing a scan_utxoset flag and it’s possible. The idea is that when enabled, the wallet would scan the UTXO set immediately after import to verify balances against chainstate, and if a discrepancy is detected (for example due to an incorrect birthdate), it could automatically trigger a full rescan from height 0 to restore missing history. This way, users wouldn’t have to manually diagnose incomplete balances — the import flow itself would handle it. What Do you think this approach?.

Cool, this is pretty much what I had in mind, I guess I would slightly prefer that full rescan isn’t started by default and rather the user receives a clear hint that there might be some funds missing and if they want to make sure to get them they should run a full rescan, basically just like what you are doing now in the return from getbalance. But I don’t have a strong preference there, either sound fine, maybe wait for some more conceptual feedback from other reviewers.

I don’t yet know the full impact on performance, but I’m thinking of making the trade-offs very clear in the RPC docs so users can decide whether to enable it.

Maybe run some benchmarks on the utxo set scan and full rescan and add them to the PR here. I don’t have a good feeling for what the relation is and this might influence what reviewers think is better concerning running a full rescan automatically or just returning a warning/hint from importdescriptors.

Another idea concerning performance: It seems likely that if the birth date is wrong the user might be off by just a few days or weeks, rather than they have a wallet with the satoshi coins. So instead of a full rescan from the start of the chain the rescan could move backwards from the originally supplied birth date and scan the chain in 1000 (or so) block increments and it could stop once the balance matches the one from the utxo set scan. A nice side effect of this is that it means the rescan is also pruning compatible (for as many blocks that are available) but I guess this could be also achieved by starting from pruneheight instead of 0 if pruning is enabled. It’s something you need to keep in mind either way. This would also need benchmarks but I can’t imagine moving backwards with somewhat large increments would make the process much slower. And ideally the process would exit early in the most common scenario.

musaHaruna commented at 9:27 am on September 17, 2025: none

Maybe run some benchmarks on the utxo set scan and full rescan and add them to the PR here. I don’t have a good feeling for what the relation is and this might influence what reviewers think is better concerning running a full rescan automatically or just returning a warning/hint from importdescriptors.

I will run some benchmarks on both utxo set scan and full rescan, update the whole PR to use the new approach you suggested i.e to add the feature on importdescriptors directly, which I honestly think is better.

Another idea concerning performance: It seems likely that if the birth date is wrong the user might be off by just a few days or weeks, rather than they have a wallet with the satoshi coins. So instead of a full rescan from the start of the chain the rescan could move backwards from the originally supplied birth date and scan the chain in 1000 (or so) block increments and it could stop once the balance matches the one from the utxo set scan. A nice side effect of this is that it means the rescan is also pruning compatible (for as many blocks that are available) but I guess this could be also achieved by starting from pruneheight instead of 0 if pruning is enabled. It’s something you need to keep in mind either way. This would also need benchmarks but I can’t imagine moving backwards with somewhat large increments would make the process much slower. And ideally the process would exit early in the most common scenario.

Just to make sure we’re on the same page — my understanding is that the idea is to:

Use chunked backward rescans starting from the supplied birthdate (e.g. 1000-block increments), and stop as soon as the wallet balance matches the UTXO-set scan. This handles the common case where the birthdate is only off by a few days or weeks, so we don’t need to rescan the entire chain.

At the same time, respect the pruneheight as the lower bound if pruning is enabled. That means we only scan back as far as blocks are available, and if the discrepancy still isn’t resolved at that point, we’d warn the user that a reindex is required.

musaHaruna renamed this:
~~rpc: add scan_utxoset option to getbalance(s) to verify wallet balance accuracy~~
wallet/rpc: add scan_utxoset option to getbalance(s) to verify wallet balance accuracy
on Sep 17, 2025

fjahr commented at 12:58 pm on September 18, 2025: contributor

Just to make sure we’re on the same page — my understanding is that the idea is to:

Use chunked backward rescans starting from the supplied birthdate (e.g. 1000-block increments), and stop as soon as the wallet balance matches the UTXO-set scan. This handles the common case where the birthdate is only off by a few days or weeks, so we don’t need to rescan the entire chain.

Right, so for example if the supplied birthdate was blockheight 800,000 but there is a discrepancy, then, instead rescanning from 0 to 800k right away, scan 799k - 800k and check the balance, exit if it’s a match, continue with 798k - 799k and so on. This shouldn’t be too complex to implement since the rescanblockchain RPC already takes a start and an end height.

At the same time, respect the pruneheight as the lower bound if pruning is enabled. That means we only scan back as far as blocks are available, and if the discrepancy still isn’t resolved at that point, we’d warn the user that a reindex is required.

Yeah, there would be some kind of safe abort if the approach for walking backwards runs into unavailable blocks, just like rescanblockchain does as well.

in src/wallet/rpc/coins.cpp:234 in 93baccaa35 outdated

231                 {
232                     {"dummy", RPCArg::Type::STR, RPCArg::Optional::OMITTED, "Remains for backward compatibility. Must be excluded or set to \"*\"."},
233                     {"minconf", RPCArg::Type::NUM, RPCArg::Default{0}, "Only include transactions confirmed at least this many times."},
234                     {"include_watchonly", RPCArg::Type::BOOL, RPCArg::Default{false}, "No longer used"},
235                     {"avoid_reuse", RPCArg::Type::BOOL, RPCArg::Default{true}, "(only available if avoid_reuse wallet flag is set) Do not include balance in dirty outputs; addresses are considered dirty if they have previously been used in a transaction."},
236+                    {"scan_utxoset", RPCArg::Type::BOOL, RPCArg::Default{false}, "If true, scan the UTXO set and return scanned UTXO balance alongside the trusted balance."},

luke-jr commented at 10:31 pm on September 18, 2025:

This shouldn’t be a positional parameter, at least.

luke-jr commented at 10:32 pm on September 18, 2025: member

I agree getbalance* is the wrong place for this kind of check.

But it also will fail to detect incorrect birthdates if the TXOs are spent, so it can’t be relied on either…

musaHaruna force-pushed on Sep 19, 2025

musaHaruna renamed this:
~~wallet/rpc: add scan_utxoset option to getbalance(s) to verify wallet balance accuracy~~
wallet, rpc: add UTXO set check and incremental rescan to importdescriptors
on Sep 19, 2025

wallet: add GetWalletUTXOSetBalance to calculate balance from UTXO set

Introduce a new helper `GetWalletUTXOSetBalance(const CWallet&)` that
derives the wallet’s balance directly from the UTXO set rather than from
the wallet’s transaction history.

Usefull for verifying wallet balance correctness against the
chainstate without requiring a full rescan.

ac06ddb8b9

wallet/rpc: add scan_utxoset arg & docs for importdescriptors

Add a new optional boolean RPC argument `scan_utxoset` to
importdescriptors and document the response fields used when the
scan_utxoset check is enabled.

fa3ac7378c

wallet: extend RescanFromTime with optional endTime to limit rescan range

Add a new overload of CWallet::RescanFromTime that accepts an optional
endTime parameter. When provided, the method finds the approximate end
height and calls ScanForWalletTransactions with a bounded [start, end]
range. This makes incremental or windowed rescans possible while preserving
existing behavior when no endTime is specified.

37719609f1

musaHaruna force-pushed on Sep 19, 2025

rpc: extend importdescriptors with UTXO check and incremental rescan

Extend the importdescriptors RPC with an optional `scan_utxoset` parameter.
When enabled, the wallet balance is compared against a balance derived from
the UTXO set. If a discrepancy is found, the wallet attempts incremental,
non-overlapping rescans in fixed-size block chunks until the balances match,
allowing for faster recovery in many cases. When a match is detected, the
RPC returns early with an `info` object describing the UTXO check and rescan
progress. If no match is found, the code falls back to the existing full
rescan behavior starting from the earliest descriptor timestamp (or the
prune boundary if applicable). Responses are updated to include optional
`info` fields with `utxo_check`, `scanned_chunks`, and `scanned_blocks`
metadata.

82a222b00d

musaHaruna force-pushed on Sep 19, 2025

test: add functional test for importdescriptors scan_utxo flag

Add a functional test that verifies `importdescriptors`
behavior with and without `scan_utxoset`, ensuring wallets correctly detect
UTXOs and transactions depending on the timestamp used. The test covers both
`timestamp="now"` (no history vs. UTXO set scan) and accurate historical timestamps
(both imports discover full history), and also asserts returned metadata fields
(`success`, `utxo_check`, `scanned_chunks`, `scanned_blocks`).

1ed236e3c6

wallet, rpc: add UTXO set check and incremental rescan to importdescriptors #33392

Code Coverage & Benchmarks

Reviews

Conflicts

LLM Linter (✨ experimental)