wallet: Track no-longer-spendable TXOs separately #27865

pull achow101 wants to merge 29 commits into bitcoin:master from achow101:wallet-unspent-txos changing 17 files +642 −370
  1. achow101 commented at 6:40 pm on June 12, 2023: member

    In #27286, the wallet keeps track of all of its transaction outputs, even if they are already spent or are otherwise unspendable. This TXO set is iterated for balance checking and coin selection preparation, which can still be slow for wallets that have had a lot of activity. This PR aims to improve the performance of such wallets by moving UTXOs that are definitely no longer spendable to a different map in the wallet so that far fewer TXOs need to be iterated for the aforementioned functions.

    Unspendable TXOs (not to be confused with Unspent TXOs) are those which have a spending transaction that has been confirmed, or are no longer valid due to reorgs. TXOs that are spent in unconfirmed transactions remain in the primary TXO set, and are filtered out of balance and coin selection as before.

    Depends on #27286

  2. DrahtBot commented at 6:40 pm on June 12, 2023: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/27865.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    Concept ACK remyers, murchandamus, jonatack

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #bitcoin-core/gui/872 (Menu action to export a watchonly wallet by achow101)
    • #32618 (wallet: Remove ISMINE_WATCHONLY and watchonly from RPCs by achow101)
    • #32523 (wallet: Remove watchonly behavior and isminetypes by achow101)
    • #32489 (wallet: Add exportwatchonlywallet RPC to export a watchonly version of a wallet by achow101)
    • #29415 (Broadcast own transactions only via short-lived Tor or I2P connections by vasild)
    • #29278 (Wallet: Add maxfeerate wallet startup option by ismaelsadeeq)
    • #29062 (Wallet: (Refactor) GetBalance to calculate used balance by BrandonOdiwuor)
    • #27286 (wallet: Keep track of the wallet’s own transaction outputs in memory by achow101)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

    LLM Linter (✨ experimental)

    Possible typos and grammar issues:

    • in wallet/wallet.h comment “definitely no longer usuable” -> “definitely no longer usable” [“usuable” is a misspelling of “usable”]

    drahtbot_id_4_m

  3. DrahtBot added the label Wallet on Jun 12, 2023
  4. DrahtBot added the label CI failed on Jun 12, 2023
  5. DrahtBot added the label Needs rebase on Jun 28, 2023
  6. achow101 force-pushed on Jun 28, 2023
  7. DrahtBot removed the label Needs rebase on Jun 28, 2023
  8. achow101 force-pushed on Jun 28, 2023
  9. achow101 force-pushed on Jun 28, 2023
  10. DrahtBot added the label Needs rebase on Sep 6, 2023
  11. achow101 force-pushed on Sep 7, 2023
  12. DrahtBot removed the label Needs rebase on Sep 7, 2023
  13. achow101 commented at 9:32 pm on September 7, 2023: member
    @t-bast You may be interested in this for very large wallets.
  14. t-bast commented at 12:26 pm on September 11, 2023: contributor
    Neat, thanks @achow101, @remyers will give that a try and will review the related PRs!
  15. remyers commented at 7:49 pm on September 12, 2023: contributor

    Concept ACK - on a large synthetic test wallet I’m seeing a 250x speed up.

    I created a large wallet in regtest with many fanout style transactions - 100,000 blocks each creating 1000 utxos. There are only 117 utxos currently, but the wallet.dat file is 370M.

    Using the same wallet.dat file and (transaction history) fundrawtransaction saw a substantial speedup: ~5 seconds using v25.99.0-a36134fcc7b4 ~20 milliseconds using v25.99.0-d9c4dcfe68c5 (latest wallet-unspent-txos)

    I will try to recreate my large test wallet.dat file tomorrow so I can confirm the steps to reproduce it. Is there anything in particular I can do to help? Would you like more testing? code review?

    NB: I had to export my original wallet descriptors and re-import them to work around the issue fixed in #27920. Before the export/import the wallet.dat file was 1.1G, but the speed of fundrawtransaction was 5 sec both in the original file and the tested one created via importing descriptors.

  16. achow101 commented at 8:55 pm on September 12, 2023: member

    Is there anything in particular I can do to help? Would you like more testing? code review?

    The prerequisite PR (#27286) will need review and testing before this can be merged. That should also provide some speed up.

  17. remyers commented at 9:53 am on September 13, 2023: contributor

    The prerequisite PR (#27286) will need review and testing before this can be merged. That should also provide some speed up.

    Will do. I’m getting 400-500ms for the same test using just #27286, so indeed also ~100x speed-up.

  18. DrahtBot added the label Needs rebase on Sep 14, 2023
  19. achow101 force-pushed on Sep 14, 2023
  20. DrahtBot removed the label Needs rebase on Sep 14, 2023
  21. achow101 force-pushed on Sep 15, 2023
  22. achow101 force-pushed on Sep 15, 2023
  23. DrahtBot added the label Needs rebase on Sep 19, 2023
  24. achow101 force-pushed on Sep 27, 2023
  25. DrahtBot removed the label Needs rebase on Sep 27, 2023
  26. DrahtBot added the label Needs rebase on Oct 16, 2023
  27. achow101 force-pushed on Oct 16, 2023
  28. DrahtBot removed the label Needs rebase on Oct 16, 2023
  29. DrahtBot added the label Needs rebase on Oct 23, 2023
  30. achow101 force-pushed on Oct 24, 2023
  31. DrahtBot removed the label Needs rebase on Oct 24, 2023
  32. achow101 force-pushed on Oct 24, 2023
  33. DrahtBot added the label Needs rebase on Oct 29, 2023
  34. achow101 force-pushed on Nov 13, 2023
  35. DrahtBot removed the label Needs rebase on Nov 13, 2023
  36. DrahtBot added the label Needs rebase on Nov 15, 2023
  37. achow101 force-pushed on Nov 15, 2023
  38. DrahtBot removed the label Needs rebase on Nov 15, 2023
  39. DrahtBot added the label Needs rebase on Nov 24, 2023
  40. achow101 force-pushed on Nov 28, 2023
  41. DrahtBot removed the label Needs rebase on Nov 28, 2023
  42. DrahtBot added the label Needs rebase on Dec 8, 2023
  43. achow101 force-pushed on Dec 8, 2023
  44. DrahtBot removed the label Needs rebase on Dec 8, 2023
  45. DrahtBot added the label Needs rebase on Dec 11, 2023
  46. achow101 force-pushed on Dec 11, 2023
  47. DrahtBot removed the label Needs rebase on Dec 11, 2023
  48. DrahtBot added the label Needs rebase on Dec 12, 2023
  49. achow101 force-pushed on Dec 19, 2023
  50. DrahtBot removed the label Needs rebase on Dec 19, 2023
  51. murchandamus commented at 7:47 pm on December 28, 2023: contributor
    Concept ACK
  52. DrahtBot added the label Needs rebase on Feb 3, 2024
  53. achow101 force-pushed on Feb 3, 2024
  54. DrahtBot removed the label Needs rebase on Feb 3, 2024
  55. DrahtBot added the label Needs rebase on Feb 20, 2024
  56. achow101 force-pushed on Feb 20, 2024
  57. DrahtBot removed the label Needs rebase on Feb 21, 2024
  58. DrahtBot added the label Needs rebase on Mar 27, 2024
  59. achow101 force-pushed on Apr 1, 2024
  60. DrahtBot removed the label Needs rebase on Apr 1, 2024
  61. DrahtBot added the label Needs rebase on Apr 8, 2024
  62. achow101 force-pushed on Apr 25, 2024
  63. DrahtBot removed the label Needs rebase on Apr 25, 2024
  64. achow101 force-pushed on Jun 6, 2024
  65. DrahtBot added the label Needs rebase on Aug 27, 2024
  66. achow101 force-pushed on Aug 29, 2024
  67. DrahtBot removed the label Needs rebase on Aug 29, 2024
  68. achow101 force-pushed on Sep 10, 2024
  69. DrahtBot added the label Needs rebase on Oct 24, 2024
  70. achow101 force-pushed on Oct 24, 2024
  71. DrahtBot removed the label Needs rebase on Oct 25, 2024
  72. achow101 force-pushed on Oct 28, 2024
  73. achow101 force-pushed on Oct 28, 2024
  74. achow101 force-pushed on Oct 28, 2024
  75. DrahtBot removed the label CI failed on Oct 28, 2024
  76. jonatack commented at 12:35 pm on November 26, 2024: member
    Concept ACK, pending #27286, if this improves performance for wallets with many transactions.
  77. DrahtBot added the label Needs rebase on Mar 13, 2025
  78. achow101 force-pushed on Apr 10, 2025
  79. DrahtBot removed the label Needs rebase on Apr 10, 2025
  80. DrahtBot added the label Needs rebase on Apr 25, 2025
  81. achow101 force-pushed on Apr 25, 2025
  82. DrahtBot removed the label Needs rebase on Apr 25, 2025
  83. DrahtBot added the label Needs rebase on May 7, 2025
  84. achow101 force-pushed on May 7, 2025
  85. DrahtBot removed the label Needs rebase on May 7, 2025
  86. achow101 force-pushed on May 7, 2025
  87. DrahtBot added the label Needs rebase on May 14, 2025
  88. achow101 force-pushed on May 15, 2025
  89. DrahtBot removed the label Needs rebase on May 15, 2025
  90. achow101 force-pushed on May 16, 2025
  91. DrahtBot added the label Needs rebase on May 16, 2025
  92. achow101 force-pushed on May 16, 2025
  93. DrahtBot removed the label Needs rebase on May 16, 2025
  94. achow101 force-pushed on May 16, 2025
  95. DrahtBot added the label CI failed on May 19, 2025
  96. achow101 force-pushed on May 19, 2025
  97. DrahtBot removed the label CI failed on May 19, 2025
  98. DrahtBot added the label Needs rebase on May 19, 2025
  99. achow101 force-pushed on May 20, 2025
  100. DrahtBot removed the label Needs rebase on May 20, 2025
  101. DrahtBot added the label Needs rebase on May 21, 2025
  102. achow101 force-pushed on May 21, 2025
  103. DrahtBot removed the label Needs rebase on May 21, 2025
  104. DrahtBot added the label Needs rebase on May 30, 2025
  105. wallet: MarkDirty after AddWalletDescriptor
    After adding a wallet descriptor (typically by import), mark all balance
    caches dirty. This allows transactions that the wallet already knows
    about that have outputs that are now ISMINE_SPENDABLE after the import
    to actually be shown in balance calculations. Legacy wallet imports
    would do this, but importdescriptors did not.
    1d1064bb27
  106. test: Test for balance update due to untracked output becoming spendable b3182f311f
  107. walletdb: Load Txs last
    Need to load txs last so that IsMine works.
    98921911f3
  108. wallet: Keep track of transaction outputs owned by the wallet
    When loading transactions to the wallet, check the outputs, and keep
    track of the ones that are IsMine.
    41c82afdfc
  109. wallet: Exit IsTrustedTx early if wtx is already in trusted_parents 72378b2bf4
  110. wallet: Recalculate the wallet's txos after any imports 8d9c7c3461
  111. wallet: Change balance calculation to use m_txos
    Since we track the outputs owned by the wallet with m_txos, we can now
    calculate the balance of the wallet by iterating m_txos and summing up
    the amounts of the unspent txos.
    
    As ISMINE_USED is not an actual isminetype that we attach to outputs and
    was just passed into `CachedTxGetAvailableCredit` for convenience, we
    pull the same determining logic from that function into `GetBalances` in
    order to preserve existing behavior.
    ae50dcb3a4
  112. wallet: Use wallet's TXO set in AvailableCoins
    Instead of iterating every transaction and every output stored in wallet
    when trying to figure out what outputs can be spent, iterate the TXO set
    which should be a lot smaller.
    8542412980
  113. wallet: Retrieve TXO directly in FetchSelectedInputs
    Instead of searching mapWallet for the preselected inputs, search
    m_txos.
    
    wallet_fundrawtransaction.py spends external inputs and needs the change
    output to also belong to the test wallet for the oversized tx test.
    935ad8999e
  114. wallet: Recompute wallet TXOs after descriptor migration
    When a legacy wallet has been migrated to contain descriptors, but
    before the transactions have been updated to match, we need to recompute
    the wallet TXOs so that the transaction update will work correctly.
    ad76a44c3d
  115. wallet: Have GetDebit use the wallet's TXO set
    Instead of looking up the previous tx in mapWallet, and then calling
    IsMine on the specified output, use the TXO set and its cached IsMine
    value.
    1047e8d1dc
  116. wallet: Remove unused CachedTxGet{Available,Immature}Credit
    These two functions are no longer used as GetBalances now uses the TXO
    set rather than per-tx cached balances
    45e8bd91fe
  117. bench: Have AvailableCoins benchmark include a lot of unrelated utxos
    One of the main issues with AvailableCoins is its performance when txs
    have unrelated outputs, so update the benchmark to check the performance
    of that.
    39734259fc
  118. wallet: Store the TxState in each WalletTXO as well 8298994de5
  119. wallet: Get the depth in main chain for a TxState 74c9442043
  120. wallet: Get the coinbase maturity state for a TXO and TxState 0e06c6a78f
  121. wallet: Use WalletTXO stored state and coinbase rather than wtx db0c38154b
  122. wallet: Replace CWalletTx::fFromMe with m_from_me for "from me" status
    fFromMe was an unused variable in CWalletTx that did not accurately
    track the "from me" status of the transaction as the name would imply.
    m_from_me is a replacement which does track this status and can be used
    to determine whether a transaction is "from me" with the different
    ismine filters that we use.
    82ac657ff5
  123. wallet: Replace CachedTxIsFromMe with CheckIsFromMeMap
    Instead of looking at the cached amounts or searching every input of a
    transaction each time we want to determine whether it is "from me", use
    the m_from_me map which stores this value for us.
    0d615f17ef
  124. wallet: Store a copy of m_from_me in WalletTXOs and use for "from me"
    Since we need to know whether the transaction that creates a WalletTXO
    is "from me", we should store this state in the WalletTXO too, copied
    from its parent CWalletTx.
    3c8187f2b2
  125. wallet: Have WalletTXOs also store parent tx time
    WalletTXOs need to know their parent tx's timestamp for AvailableCoins
    to work.
    ed0731a00a
  126. wallet: Make CWalletTx::m_state private with {get,set}ters dceee325a2
  127. wallet: have CWalletTx also be aware of it's WalletTXOs 2d49f12515
  128. wallet: Also update a CWalletTx's WalletTXOs states
    When the state of a CWalletTx changes, we need to change the state in
    the WalletTXOs too.
    1ead3a60a7
  129. wallet: Remove unused WalletTXO::GetWalletTx() a05fd159c7
  130. wallet: Have IsSpent take a min_conf
    A min_conf parameter is added to IsSpent so that it can set a
    confirmation threshold for whether something is considered spent.
    d50d7e5d1a
  131. wallet: Iterate block txs in reverse on blockDisconnected
    When a block is disconnected, we need to process the transactions in
    reverse order so that the wallet's TXO set is updated in the correct
    order.
    afed0e986e
  132. achow101 force-pushed on May 30, 2025
  133. DrahtBot removed the label Needs rebase on May 30, 2025
  134. wallet, tests: Have CreateSyncedWallet use CWallet::Create
    CWallet::Create will properly connect the wallet to the chain, so we
    should be doing that rather than ad-hoc chain connection.
    98974312fb
  135. wallet: Move definintely unusable TXOs to a separate container
    Definitely unusable TXOs are those that are spent by a confirmed
    transaction or were produced by a now conflicted transaction. However,
    we still need them for GetDebit, so we store them in a separate
    m_unusable_txos container. MarkConflicted, AbandonTransaction, and
    loading (via PruneSpentTXOs) will ensure that these unusable TXOs are
    properly moved.
    072e2835c4
  136. in src/wallet/test/util.cpp:26 in 9554a4d79a outdated
    27-    }
    28+    bilingual_str error;
    29+    std::vector<bilingual_str> warnings;
    30+    auto wallet = CWallet::Create(context, "", CreateMockableWalletDatabase(), WALLET_FLAG_DESCRIPTORS, error, warnings);
    31+
    32+    // Allow the fallback fee with it's default
    


    maflcko commented at 7:17 am on June 2, 2025:
    “with it’s default” -> “with its default” [“it’s” is a contraction of “it is,” but here the possessive “its” is required]

    achow101 commented at 7:37 pm on June 2, 2025:
    Fixed
  137. achow101 force-pushed on Jun 2, 2025

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-06-08 21:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me