wallet: Track no-longer-spendable TXOs separately

achow101 commented at 6:40 pm on June 12, 2023: member

In #27286, the wallet keeps track of all of its transaction outputs, even if they are already spent or are otherwise unspendable. This TXO set is iterated for balance checking and coin selection preparation, which can still be slow for wallets that have had a lot of activity. This PR aims to improve the performance of such wallets by moving UTXOs that are definitely no longer spendable to a different map in the wallet so that far fewer TXOs need to be iterated for the aforementioned functions.

Unspendable TXOs (not to be confused with Unspent TXOs) are those which have a spending transaction that has been confirmed, or are no longer valid due to reorgs. TXOs that are spent in unconfirmed transactions remain in the primary TXO set, and are filtered out of balance and coin selection as before.

Depends on #27286

DrahtBot commented at 6:40 pm on June 12, 2023: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/27865.

Reviews

See the guideline for information on the review process.

Type	Reviewers
Concept ACK	remyers, murchandamus, jonatack

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#31757 (wallet: fix crash on double block disconnection by furszy)
#31250 (wallet: Disable creating and loading legacy wallets by achow101)
#30221 (wallet: Ensure best block matches wallet scan state by achow101)
#29278 (Wallet: Add maxfeerate wallet startup option by ismaelsadeeq)
#29062 (Wallet: (Refactor) GetBalance to calculate used balance by BrandonOdiwuor)
#28710 (Remove the legacy wallet and BDB dependency by achow101)
#25722 (refactor: Use util::Result class for wallet loading by ryanofsky)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

DrahtBot added the label Wallet on Jun 12, 2023

DrahtBot added the label CI failed on Jun 12, 2023

DrahtBot added the label Needs rebase on Jun 28, 2023

achow101 force-pushed on Jun 28, 2023

DrahtBot removed the label Needs rebase on Jun 28, 2023

achow101 force-pushed on Jun 28, 2023

DrahtBot added the label Needs rebase on Sep 6, 2023

achow101 force-pushed on Sep 7, 2023

DrahtBot removed the label Needs rebase on Sep 7, 2023

achow101 commented at 9:32 pm on September 7, 2023: member

@t-bast You may be interested in this for very large wallets.

t-bast commented at 12:26 pm on September 11, 2023: contributor

Neat, thanks @achow101, @remyers will give that a try and will review the related PRs!

remyers commented at 7:49 pm on September 12, 2023: contributor

Concept ACK - on a large synthetic test wallet I’m seeing a 250x speed up.

I created a large wallet in regtest with many fanout style transactions - 100,000 blocks each creating 1000 utxos. There are only 117 utxos currently, but the wallet.dat file is 370M.

Using the same wallet.dat file and (transaction history) fundrawtransaction saw a substantial speedup: ~5 seconds using v25.99.0-a36134fcc7b4 ~20 milliseconds using v25.99.0-d9c4dcfe68c5 (latest wallet-unspent-txos)

I will try to recreate my large test wallet.dat file tomorrow so I can confirm the steps to reproduce it. Is there anything in particular I can do to help? Would you like more testing? code review?

NB: I had to export my original wallet descriptors and re-import them to work around the issue fixed in #27920. Before the export/import the wallet.dat file was 1.1G, but the speed of fundrawtransaction was 5 sec both in the original file and the tested one created via importing descriptors.

achow101 commented at 8:55 pm on September 12, 2023: member

Is there anything in particular I can do to help? Would you like more testing? code review?

The prerequisite PR (#27286) will need review and testing before this can be merged. That should also provide some speed up.

remyers commented at 9:53 am on September 13, 2023: contributor

The prerequisite PR (#27286) will need review and testing before this can be merged. That should also provide some speed up.

Will do. I’m getting 400-500ms for the same test using just #27286, so indeed also ~100x speed-up.

DrahtBot added the label Needs rebase on Sep 14, 2023

achow101 force-pushed on Sep 14, 2023

DrahtBot removed the label Needs rebase on Sep 14, 2023

achow101 force-pushed on Sep 15, 2023

DrahtBot added the label Needs rebase on Sep 19, 2023

achow101 force-pushed on Sep 27, 2023

DrahtBot removed the label Needs rebase on Sep 27, 2023

DrahtBot added the label Needs rebase on Oct 16, 2023

achow101 force-pushed on Oct 16, 2023

DrahtBot removed the label Needs rebase on Oct 16, 2023

DrahtBot added the label Needs rebase on Oct 23, 2023

achow101 force-pushed on Oct 24, 2023

DrahtBot removed the label Needs rebase on Oct 24, 2023

achow101 force-pushed on Oct 24, 2023

DrahtBot added the label Needs rebase on Oct 29, 2023

achow101 force-pushed on Nov 13, 2023

DrahtBot removed the label Needs rebase on Nov 13, 2023

DrahtBot added the label Needs rebase on Nov 15, 2023

achow101 force-pushed on Nov 15, 2023

DrahtBot removed the label Needs rebase on Nov 15, 2023

DrahtBot added the label Needs rebase on Nov 24, 2023

achow101 force-pushed on Nov 28, 2023

DrahtBot removed the label Needs rebase on Nov 28, 2023

DrahtBot added the label Needs rebase on Dec 8, 2023

achow101 force-pushed on Dec 8, 2023

DrahtBot removed the label Needs rebase on Dec 8, 2023

DrahtBot added the label Needs rebase on Dec 11, 2023

achow101 force-pushed on Dec 11, 2023

DrahtBot removed the label Needs rebase on Dec 11, 2023

DrahtBot added the label Needs rebase on Dec 12, 2023

achow101 force-pushed on Dec 19, 2023

DrahtBot removed the label Needs rebase on Dec 19, 2023

murchandamus commented at 7:47 pm on December 28, 2023: contributor

Concept ACK

DrahtBot added the label Needs rebase on Feb 3, 2024

achow101 force-pushed on Feb 3, 2024

DrahtBot removed the label Needs rebase on Feb 3, 2024

DrahtBot added the label Needs rebase on Feb 20, 2024

achow101 force-pushed on Feb 20, 2024

DrahtBot removed the label Needs rebase on Feb 21, 2024

DrahtBot added the label Needs rebase on Mar 27, 2024

achow101 force-pushed on Apr 1, 2024

DrahtBot removed the label Needs rebase on Apr 1, 2024

DrahtBot added the label Needs rebase on Apr 8, 2024

achow101 force-pushed on Apr 25, 2024

DrahtBot removed the label Needs rebase on Apr 25, 2024

achow101 force-pushed on Jun 6, 2024

DrahtBot added the label Needs rebase on Aug 27, 2024

achow101 force-pushed on Aug 29, 2024

DrahtBot removed the label Needs rebase on Aug 29, 2024

achow101 force-pushed on Sep 10, 2024

DrahtBot added the label Needs rebase on Oct 24, 2024

walletdb: Load Txs last

Need to load txs last so that IsMine works.

9848611d10

wallet: Keep track of transaction outputs owned by the wallet

When loading transactions to the wallet, check the outputs, and keep
track of the ones that are IsMine.

af8e6221c6

wallet: Exit IsTrustedTx early if wtx is already in trusted_parents 326b7c59e7

wallet: Change balance calculation to use m_txos

Since we track the outputs owned by the wallet with m_txos, we can now
calculate the balance of the wallet by iterating m_txos and summing up
the amounts of the unspent txos.

As ISMINE_USED is not an actual isminetype that we attach to outputs and
was just passed into `CachedTxGetAvailableCredit` for convenience, we
pull the same determining logic from that function into `GetBalances` in
order to preserve existing behavior.

c088e9b699

wallet: Recalculate the wallet's txos after any imports 1b2dc9cec2

test: Test for balance update due to watchonly becoming spendable 016f08cf67

wallet: Use wallet's TXO set in AvailableCoins

Instead of iterating every transaction and every output stored in wallet
when trying to figure out what outputs can be spent, iterate the TXO set
which should be a lot smaller.

e2e0fcd5db

wallet: Retrieve TXO directly in FetchSelectedInputs

Instead of searching mapWallet for the preselected inputs, search
m_txos.

wallet_fundrawtransaction.py spends external inputs and needs the change
output to also belong to the test wallet for the oversized tx test.

961d0f4845

wallet: Recompute wallet TXOs after descriptor migration

When a legacy wallet has been migrated to contain descriptors, but
before the transactions have been updated to match, we need to recompute
the wallet TXOs so that the transaction update will work correctly.

79a35663f0

wallet: Have GetDebit use the wallet's TXO set

Instead of looking up the previous tx in mapWallet, and then calling
IsMine on the specified output, use the TXO set and its cached IsMine
value.

933a550af5

wallet: Remove unused CachedTxGet{Available,Immature}Credit

These two functions are no longer used as GetBalances now uses the TXO
set rather than per-tx cached balances

36743a5ee1

bench: Have AvailableCoins benchmark include a lot of unrelated utxos

One of the main issues with AvailableCoins is its performance when txs
have unrelated outputs, so update the benchmark to check the performance
of that.

48545bdb3b

achow101 force-pushed on Oct 24, 2024

DrahtBot removed the label Needs rebase on Oct 25, 2024

achow101 force-pushed on Oct 28, 2024

wallet: Store the TxState in each WalletTXO as well 3481c945fb

wallet: Get the depth in main chain for a TxState 0904a7d078

wallet: Get the coinbase maturity state for a TXO and TxState a74ef9bfed

wallet: Use WalletTXO stored state and coinbase rather than wtx 7d2bcc7458

wallet: Replace CWalletTx::fFromMe with m_from_me for "from me" status

fFromMe was an unused variable in CWalletTx that did not accurately
track the "from me" status of the transaction as the name would imply.
m_from_me is a replacement which does track this status and can be used
to determine whether a transaction is "from me" with the different
ismine filters that we use.

36f2c6a6bd

wallet: Replace CachedTxIsFromMe with CheckIsFromMeMap

Instead of looking at the cached amounts or searching every input of a
transaction each time we want to determine whether it is "from me", use
the m_from_me map which stores this value for us.

ba24bf61cd

wallet: Store a copy of m_from_me in WalletTXOs and use for "from me"

Since we need to know whether the transaction that creates a WalletTXO
is "from me", we should store this state in the WalletTXO too, copied
from its parent CWalletTx.

78d97b0aed

wallet: Have WalletTXOs also store parent tx time

WalletTXOs need to know their parent tx's timestamp for AvailableCoins
to work.

c52d419846

wallet: Make CWalletTx::m_state private with {get,set}ters fa51521e95

wallet: have CWalletTx also be aware of it's WalletTXOs 204e49e373

wallet: Also update a CWalletTx's WalletTXOs states

When the state of a CWalletTx changes, we need to change the state in
the WalletTXOs too.

531aa39036

wallet: Remove unused WalletTXO::GetWalletTx() bc1c12cb29

wallet: Have IsSpent take a min_conf

A min_conf parameter is added to IsSpent so that it can set a
confirmation threshold for whether something is considered spent.

710a59dcc4

wallet: Iterate block txs in reverse on blockDisconnected

When a block is disconnected, we need to process the transactions in
reverse order so that the wallet's TXO set is updated in the correct
order.

f8a18da0fb

wallet, tests: Have CreateSyncedWallet use CWallet::Create

CWallet::Create will properly connect the wallet to the chain, so we
should be doing that rather than ad-hoc chain connection.

27b432f524

wallet: Move definintely unusable TXOs to a separate container

Definitely unusable TXOs are those that are spent by a confirmed
transaction or were produced by a now conflicted transaction. However,
we still need them for GetDebit, so we store them in a separate
m_unusable_txos container. MarkConflicted, AbandonTransaction, and
loading (via PruneSpentTXOs) will ensure that these unusable TXOs are
properly moved.

09fb29b82c

achow101 force-pushed on Oct 28, 2024

DrahtBot removed the label CI failed on Oct 28, 2024

jonatack commented at 12:35 pm on November 26, 2024: member

Concept ACK, pending #27286, if this improves performance for wallets with many transactions.

DrahtBot commented at 8:54 pm on March 13, 2025: contributor

🐙 This pull request conflicts with the target branch and needs rebase.

DrahtBot added the label Needs rebase on Mar 13, 2025

wallet: Track no-longer-spendable TXOs separately #27865

Code Coverage & Benchmarks

Reviews

Conflicts