Currently, the wallet is not actually aware about its own transaction outputs. Instead, it will iterate all of the transactions stored in mapWallet, and then all of the outputs of those transactions, in order to figure out what belongs to it for the purposes of coin selection and balance calculation. For balance calculation, there is caching that results in it only iterating all of the transactions, but not all of the outputs. However when the cache is dirty, everything is iterated. This is especially problematic for wallets that have a lot of transactions, or transactions that have a lot of unrelated outputs (as may occur with coinjoins or batched payments).
This PR helps to resolve this issue by making the wallet track all of the outputs that belong to it in a new member m_txos. Note that this includes outputs that may have already been spent. Both balance calculation (GetBalance) and coin selection (AvailableCoins) are updated to iterate m_txos. This is generally faster since it ignores all of the unrelated outputs, and it is not slower as in the worst case of wallets containing only single output transactions, it’s exactly the same number of outputs iterated.
m_txos is memory only, and it is populated during wallet loading. When each transaction is loaded, all of its outputs are checked to see if it is IsMine, and if so, an entry added to m_txos. When new transactions are received, the same procedure is done.
Since imports can change the IsMine status of a transaction (although they can only be “promoted” from watchonly to spendable), all of the import RPCs will be a bit slower as they re-iterate all transactions and all outputs to update m_txos.
Each output in m_txos is stored in a new WalletTXO class. This class contains references to the parent CWalletTx and the CTxOut itself. It also caches the IsMine value of the txout. This should be safe as IsMine should not change unless there are imports. This allows us to have additional performance improvements in places that use these WalletTXOs as they can use the cached IsMine rather than repeatedly calling IsMine which can be expensive.
The existing WalletBalance benchmark demonstrates the performance improvement that this PR makes. The existing WalletAvailableCoins benchmark doesn’t as all of the outputs used in that benchmark belong to the test wallet. I’ve updated that benchmark to have a bunch of unrelated outputs in each transaction so that the difference is demonstrated.
This is part of a larger project to have the wallet actually track and store a set of its UTXOs.
Built on #24914 as it requires loading the txs last in order for m_txos to be built correctly.
Benchmarks:
Master:
| ns/op | op/s | err% | ins/op | cyc/op | IPC | bra/op | miss% | total | benchmark |
|---|---|---|---|---|---|---|---|---|---|
| 34,590,013.00 | 28.91 | 0.0% | 812,669,269.00 | 148,360,642.50 | 5.478 | 18,356,853.00 | 0.2% | 0.76 | WalletAvailableCoins |
| 3,193.46 | 313,139.91 | 0.4% | 96,868.06 | 13,731.82 | 7.054 | 26,238.01 | 0.1% | 0.01 | WalletBalanceClean |
| 26,871.18 | 37,214.59 | 3.3% | 768,179.50 | 115,544.39 | 6.648 | 154,171.09 | 0.1% | 0.01 | WalletBalanceDirty |
| 3,177.30 | 314,732.47 | 0.2% | 96,868.06 | 13,646.20 | 7.099 | 26,238.01 | 0.1% | 0.01 | WalletBalanceMine |
| 10.73 | 93,186,952.53 | 0.1% | 157.00 | 46.14 | 3.403 | 36.00 | 0.0% | 0.01 | WalletBalanceWatch |
| 590,497,920.00 | 1.69 | 0.1% | 12,761,692,005.00 | 2,536,899,595.00 | 5.030 | 129,124,399.00 | 0.7% | 6.50 | WalletCreateEncrypted |
| 182,929,529.00 | 5.47 | 0.0% | 4,199,271,397.00 | 785,477,302.00 | 5.346 | 75,363,377.00 | 1.1% | 2.01 | WalletCreatePlain |
| 699,337.20 | 1,429.93 | 0.7% | 18,054,294.00 | 3,005,072.20 | 6.008 | 387,756.60 | 0.3% | 0.04 | WalletCreateTxUseOnlyPresetInputs |
| 32,068,583.80 | 31.18 | 0.5% | 562,026,110.00 | 137,457,635.60 | 4.089 | 90,667,459.40 | 0.3% | 1.78 | WalletCreateTxUsePresetInputsAndCoinSelection |
| 36.62 | 27,306,578.40 | 0.5% | 951.00 | 157.05 | 6.056 | 133.00 | 0.0% | 0.01 | WalletIsMineDescriptors |
| 35.00 | 28,569,989.42 | 0.7% | 937.00 | 150.33 | 6.233 | 129.00 | 0.0% | 0.01 | WalletIsMineMigratedDescriptors |
| 203,284,889.00 | 4.92 | 0.0% | 4,622,691,895.00 | 872,875,275.00 | 5.296 | 90,345,002.00 | 1.2% | 1.02 | WalletLoadingDescriptors |
| 1,165,766,084.00 | 0.86 | 0.0% | 24,139,316,211.00 | 5,005,218,705.00 | 4.823 | 2,664,455,775.00 | 0.1% | 1.17 | WalletMigration |
PR:
| ns/op | op/s | err% | ins/op | cyc/op | IPC | bra/op | miss% | total | benchmark |
|---|---|---|---|---|---|---|---|---|---|
| 33,975,750.50 | 29.43 | 0.1% | 794,719,150.50 | 145,763,550.00 | 5.452 | 16,036,630.50 | 0.2% | 0.75 | WalletAvailableCoins |
| 2,442.01 | 409,498.46 | 0.2% | 60,782.04 | 10,500.60 | 5.788 | 9,492.01 | 0.3% | 0.01 | WalletBalanceClean |
| 2,763.12 | 361,909.21 | 0.2% | 61,493.05 | 11,859.48 | 5.185 | 9,625.01 | 0.2% | 0.01 | WalletBalanceDirty |
| 2,347.98 | 425,898.72 | 0.3% | 60,782.04 | 10,082.73 | 6.028 | 9,492.01 | 0.2% | 0.01 | WalletBalanceMine |
| 11.67 | 85,654,630.36 | 0.2% | 176.00 | 50.18 | 3.508 | 40.00 | 0.0% | 0.01 | WalletBalanceWatch |
| 590,119,519.00 | 1.69 | 0.1% | 12,754,398,258.00 | 2,534,998,522.00 | 5.031 | 129,078,027.00 | 0.7% | 6.50 | WalletCreateEncrypted |
| 183,124,790.00 | 5.46 | 0.1% | 4,199,212,926.00 | 786,323,886.00 | 5.340 | 75,354,437.00 | 1.1% | 2.02 | WalletCreatePlain |
| 669,643.00 | 1,493.33 | 0.1% | 17,213,904.20 | 2,877,336.40 | 5.983 | 394,292.80 | 0.3% | 0.04 | WalletCreateTxUseOnlyPresetInputs |
| 26,205,987.80 | 38.16 | 0.8% | 365,551,340.80 | 112,376,905.20 | 3.253 | 65,684,276.20 | 0.4% | 1.44 | WalletCreateTxUsePresetInputsAndCoinSelection |
| 34.75 | 28,778,846.38 | 0.1% | 937.00 | 149.41 | 6.271 | 129.00 | 0.0% | 0.01 | WalletIsMineDescriptors |
| 29.91 | 33,428,072.85 | 0.2% | 920.00 | 128.63 | 7.152 | 126.00 | 0.0% | 0.01 | WalletIsMineMigratedDescriptors |
| 202,437,985.00 | 4.94 | 0.1% | 4,626,686,256.00 | 869,439,274.00 | 5.321 | 90,961,305.00 | 1.1% | 1.02 | WalletLoadingDescriptors |
| 1,158,394,152.00 | 0.86 | 0.0% | 24,143,589,972.00 | 4,971,946,380.00 | 4.856 | 2,665,355,654.00 | 0.1% | 1.16 | WalletMigration |
