wallet: track mempool conflicts with wallet transactions #27307

pull ishaanam wants to merge 5 commits into bitcoin:master from ishaanam:track_mempool_wallet_conflicts changing 10 files +383 −34
  1. ishaanam commented at 9:30 pm on March 22, 2023: contributor

    The mempool_conflicts variable is added to CWalletTx, it is a set of txids of txs in the mempool conflicting with the wallet tx or a wallet tx’s parent. This PR only changes how mempool-conflicted txs are dealt with in memory.

    IsSpent now returns false for an output being spent by a mempool conflicted transaction where it previously returned true.

    A txid is added to mempool_conflicts during transactionAddedToMempool. A txid is removed from mempool_conflicts during transactionRemovedFromMempool.

    This PR also adds a mempoolconflicts field to the gettransaction wallet RPC result.

    Builds on #27145 Second attempt at #18600

  2. DrahtBot commented at 9:30 pm on March 22, 2023: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage

    For detailed information about the code coverage, see the test coverage report.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK achow101, ryanofsky, furszy
    Approach ACK glozow
    Stale ACK murchandamus

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #29680 (wallet: fix unrelated parent conflict doesn’t cause child tx to be marked as conflict by Eunovo)
    • #28616 (Show transactions as not fully confirmed during background validation by Sjors)
    • #27865 (wallet: Track no-longer-spendable TXOs separately by achow101)
    • #27286 (wallet: Keep track of the wallet’s own transaction outputs in memory by achow101)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  3. DrahtBot added the label Wallet on Mar 22, 2023
  4. ishaanam force-pushed on Apr 13, 2023
  5. ishaanam force-pushed on Apr 13, 2023
  6. ishaanam force-pushed on May 3, 2023
  7. DrahtBot added the label CI failed on May 3, 2023
  8. ishaanam force-pushed on May 14, 2023
  9. ishaanam force-pushed on May 14, 2023
  10. ishaanam force-pushed on May 15, 2023
  11. DrahtBot removed the label CI failed on May 15, 2023
  12. ishaanam force-pushed on May 18, 2023
  13. DrahtBot added the label CI failed on May 18, 2023
  14. DrahtBot added the label Needs rebase on May 27, 2023
  15. ishaanam force-pushed on May 28, 2023
  16. ishaanam marked this as ready for review on May 28, 2023
  17. DrahtBot removed the label CI failed on May 28, 2023
  18. DrahtBot removed the label Needs rebase on May 28, 2023
  19. ishaanam force-pushed on Jun 15, 2023
  20. DrahtBot added the label CI failed on Jun 15, 2023
  21. ishaanam force-pushed on Jun 15, 2023
  22. ishaanam force-pushed on Jun 18, 2023
  23. ishaanam force-pushed on Jun 18, 2023
  24. DrahtBot removed the label CI failed on Jun 18, 2023
  25. in src/wallet/transaction.h:326 in 0538ad7d4c outdated
    320@@ -315,7 +321,9 @@ class CWalletTx
    321     template<typename T> T* state() { return std::get_if<T>(&m_state); }
    322 
    323     bool isAbandoned() const { return state<TxStateInactive>() && state<TxStateInactive>()->abandoned; }
    324-    bool isConflicted() const { return state<TxStateConflicted>(); }
    325+    bool isConflicted() const { return state<TxStateConflicted>() || state<TxStateMempoolConflicted>(); }
    326+    bool isMempoolConflicted() const { return state<TxStateMempoolConflicted>(); }
    327+    bool isBlockConflicted() const { return state<TxStateConflicted>(); }
    


    evansmj commented at 4:26 am on June 28, 2023:
    What do you think of renaming the TxStateConflicted struct to TxStateBlockConflicted, since that one is used for block conflicts and there is now a difference here between block and mempool conflicts?

    ishaanam commented at 2:15 pm on June 28, 2023:
    That makes sense, I’ll do that using a scripted diff.
  26. ishaanam force-pushed on Jun 29, 2023
  27. ishaanam force-pushed on Jun 29, 2023
  28. DrahtBot added the label CI failed on Jun 29, 2023
  29. ishaanam force-pushed on Jun 30, 2023
  30. DrahtBot removed the label CI failed on Jun 30, 2023
  31. ishaanam force-pushed on Jul 17, 2023
  32. ishaanam force-pushed on Jul 17, 2023
  33. DrahtBot added the label CI failed on Jul 17, 2023
  34. ishaanam force-pushed on Jul 17, 2023
  35. in test/functional/wallet_conflicts.py:255 in 42412c592e outdated
    214+        tx3 = bob.sendrawtransaction(raw_tx3)
    215+
    216+        self.disconnect_nodes(0, 1)
    217+
    218+        # alice has all 0 txs, bob has 3
    219+        assert_equal(len(alice.getrawmempool()), 0)
    


    maflcko commented at 8:46 am on July 21, 2023:

    https://cirrus-ci.com/task/6496120092753920?logs=ci#L3132

     0 test  2023-07-17T05:13:02.641000Z TestFramework (ERROR): Assertion failed 
     1                                   Traceback (most recent call last):
     2                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 131, in main
     3                                       self.run_test()
     4                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/wallet_conflicts.py", line 34, in run_test
     5                                       self.test_mempool_and_block_conflicts()
     6                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/wallet_conflicts.py", line 219, in test_mempool_and_block_conflicts
     7                                       assert_equal(len(alice.getrawmempool()), 0)
     8                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/util.py", line 57, in assert_equal
     9                                       raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args))
    10                                   AssertionError: not(1 == 0)
    

    ishaanam commented at 1:24 pm on July 23, 2023:
    Fixed
  36. ishaanam force-pushed on Jul 23, 2023
  37. DrahtBot removed the label CI failed on Jul 23, 2023
  38. achow101 requested review from achow101 on Sep 20, 2023
  39. achow101 requested review from murchandamus on Sep 20, 2023
  40. achow101 requested review from furszy on Sep 20, 2023
  41. achow101 requested review from glozow on Sep 20, 2023
  42. in src/wallet/wallet.cpp:1427 in 20d58f2554 outdated
    1384@@ -1386,6 +1385,36 @@ void CWallet::transactionAddedToMempool(const CTransactionRef& tx) {
    1385     if (it != mapWallet.end()) {
    1386         RefreshMempoolStatus(it->second, chain());
    1387     }
    1388+
    1389+    const uint256& conflict_hash = tx->GetHash();
    1390+
    1391+    for (const CTxIn& tx_in : tx->vin) {
    


    glozow commented at 10:57 am on September 26, 2023:
    Question: What’s the expected state of a transaction if its parent is replaced? My (shallow) understanding suggests it becomes inactive? And if the parent is re-added to mempool, it stays inactive?

    ishaanam commented at 5:19 pm on January 30, 2024:
    With this PR, if a transaction’s parent is replaced, then it should become TxStateMempoolConflicted. If the conflict is removed, then it becomes inactive.

    glozow commented at 10:04 am on January 31, 2024:
    Ah makes sense! I think it would be good to add a test for this case?

    ishaanam commented at 5:45 am on February 5, 2024:
    I have now added a test for descendants of conflicted transactions.
  43. DrahtBot added the label Needs rebase on Oct 17, 2023
  44. ishaanam force-pushed on Oct 17, 2023
  45. DrahtBot removed the label Needs rebase on Oct 17, 2023
  46. in src/wallet/wallet.h:341 in 770a73c66a outdated
    336@@ -337,6 +337,9 @@ class CWallet final : public WalletStorage, public interfaces::Chain::Notificati
    337     void AddToSpends(const COutPoint& outpoint, const uint256& wtxid, WalletBatch* batch = nullptr) EXCLUSIVE_LOCKS_REQUIRED(cs_wallet);
    338     void AddToSpends(const CWalletTx& wtx, WalletBatch* batch = nullptr) EXCLUSIVE_LOCKS_REQUIRED(cs_wallet);
    339 
    340+    // Map of wtx hash to a set of mempool conflict hashes
    341+    std::map<uint256, std::set<uint256>> MempoolConflicts GUARDED_BY(cs_wallet);
    


    achow101 commented at 7:52 pm on November 3, 2023:

    In 770a73c66a48434fa48f594709dbbf5831dc1d5d “wallet: track mempool conflicts”

    nit: snake_case for member variables.

    Additionally, this is tracking these conflicts at the CWallet level. However I wonder if it might be better to track it on a transaction level, i.e. have CWalletTx have the set of txids.


    ishaanam commented at 11:48 pm on November 4, 2023:
    I agree it is better to track conflicts on a transaction level. I’ve implemented it this way now and it looks much better.
  47. ishaanam force-pushed on Nov 4, 2023
  48. in test/functional/wallet_conflicts.py:184 in 003efbbe45 outdated
    179+        self.generate(self.nodes[0], 3)
    180+        assert_equal(alice.getbalance(), 75)
    181+
    182+    def test_mempool_and_block_conflicts(self):
    183+        alice = self.nodes[0].get_wallet_rpc("alice")
    184+        bob = self.nodes[1].get_wallet_rpc("bob")
    


    achow101 commented at 8:53 pm on November 6, 2023:

    In 003efbbe45079c4416810a025b2bc372559dff15 “test: Add tests for wallet mempool conflicts”

    nit: When doing separate test cases like this, we should also use different wallets to prevent state from one test cases from effecting another. So this should be making its own wallets once again.


    murchandamus commented at 8:11 pm on November 9, 2023:
    Yeah, this could be a problem if you only run one of the tests separately instead of the whole suite. Then that test would fail, since it’s setup depends on the prior test having been run before.

    ishaanam commented at 4:47 am on November 11, 2023:
    Done
  49. in test/functional/wallet_conflicts.py:137 in 003efbbe45 outdated
    128@@ -123,5 +129,148 @@ def run_test(self):
    129         assert_equal(former_conflicted["confirmations"], 1)
    130         assert_equal(former_conflicted["blockheight"], 217)
    131 
    132+    def test_mempool_conflict(self):
    133+        self.nodes[0].createwallet("alice")
    134+        alice = self.nodes[0].get_wallet_rpc("alice")
    135+
    136+        self.nodes[1].createwallet("bob")
    137+        bob = self.nodes[1].get_wallet_rpc("bob")
    


    achow101 commented at 8:54 pm on November 6, 2023:

    In 003efbbe45079c4416810a025b2bc372559dff15 “test: Add tests for wallet mempool conflicts”

    nit: No need to make another wallet for the receiver, just use the default wallet.


    ishaanam commented at 4:47 am on November 11, 2023:
    Done
  50. in src/wallet/wallet.cpp:1509 in 28a7e883c2 outdated
    1544@@ -1482,8 +1545,8 @@ void CWallet::blockConnected(ChainstateRole role, const interfaces::BlockInfo& b
    1545 
    1546     // Scan block
    1547     for (size_t index = 0; index < block.data->vtx.size(); index++) {
    1548-        SyncTransaction(block.data->vtx[index], TxStateConfirmed{block.hash, block.height, static_cast<int>(index)});
    1549         transactionRemovedFromMempool(block.data->vtx[index], MemPoolRemovalReason::BLOCK);
    1550+        SyncTransaction(block.data->vtx[index], TxStateConfirmed{block.hash, block.height, static_cast<int>(index)});
    


    achow101 commented at 9:30 pm on November 6, 2023:

    In 28a7e883c2f9249213fddbe24948ec5bb90b0fad “wallet: track mempool conflicts”

    It’s not clear to me that this change has any effect.


    ishaanam commented at 2:33 pm on November 7, 2023:
    I think before this was to fix a bug, but I fixed that bug in a different way. Regardless, doesn’t it make more sense to run transactionRemovedFromMempool before syncing the transaction as confirmed?

    glozow commented at 4:14 pm on January 30, 2024:
    fwiw, it seems better to keep it the way it was if it’s not doing anything. What was the bug?

    ishaanam commented at 10:26 pm on February 1, 2024:
    I don’t remember exactly what the bug was, but I know that it had something to do with assuming that all transactions that we call transactionRemovedFromMempool on are TxStateInMempool. That being said, I will change it back to the way it was before because it is not causing any problems right now.

    achow101 commented at 10:18 pm on February 2, 2024:
    I think this hasn’t been reverted yet?

    ishaanam commented at 5:55 am on February 5, 2024:
    Should be fixed now.
  51. in test/functional/wallet_conflicts.py:228 in 003efbbe45 outdated
    189+        assert_equal(bob.getbalances()["mine"]["untrusted_pending"], 0)
    190+
    191+        self.disconnect_nodes(0, 1)
    192+
    193+        raw_tx = alice.createrawtransaction(inputs=[unspents[0]], outputs=[{bob.getnewaddress() : 24.99999}])
    194+        raw_tx1 = alice.signrawtransactionwithwallet(raw_tx)['hex']
    


    achow101 commented at 9:36 pm on November 6, 2023:

    In 003efbbe45079c4416810a025b2bc372559dff15 “test: Add tests for wallet mempool conflicts”

    This can be compressed into one line using the send RPC.

    0        raw_tx1 = alice.send(outputs=[{bob.getnewaddress(): 24.9999}], inputs=[unspents[0]], add_to_wallet=False)["hex"]
    

    ishaanam commented at 0:20 am on November 11, 2023:
    These two things are not technically equivalent, because send creates a change output here, which results in intermittent failures for the rest of the test. Is there any way to tell send not to create a change output?
  52. in test/functional/wallet_conflicts.py:280 in 003efbbe45 outdated
    237+        assert_equal(bob.getbalances()["mine"]["untrusted_pending"], Decimal("24.99990000"))
    238+
    239+        # we will be disconnecting this block in the future
    240+        tx2_conflict = alice.sendrawtransaction(tx2_conflict)
    241+        assert_equal(len(alice.getrawmempool()), 1)
    242+        blk = self.generate(self.nodes[0], 11, sync_fun=self.no_op)[0]
    


    achow101 commented at 9:39 pm on November 6, 2023:

    In 003efbbe45079c4416810a025b2bc372559dff15 “test: Add tests for wallet mempool conflicts”

    It would be useful to have a comment that explains that 11 blocks are mined so that when they are invalidated, tx2_conflict does not get put back into the mempool.


    ishaanam commented at 4:48 am on November 11, 2023:
    I’ve added a comment.
  53. in test/functional/wallet_conflicts.py:143 in 003efbbe45 outdated
    138+
    139+        self.generatetoaddress(self.nodes[0], COINBASE_MATURITY + 3, alice.getnewaddress())
    140+
    141+        self.log.info("Test a scenario where a transaction has a mempool conflict")
    142+
    143+        unspents = [{"txid" : element["txid"], "vout" : element["vout"]} for element in alice.listunspent()]
    


    achow101 commented at 9:39 pm on November 6, 2023:

    In 003efbbe45079c4416810a025b2bc372559dff15 “test: Add tests for wallet mempool conflicts”

    This isn’t actually necessary since listunspent has these fields already and the extra ones are just ignored.


    ishaanam commented at 4:48 am on November 11, 2023:
    Fixed
  54. DrahtBot added the label Needs rebase on Nov 7, 2023
  55. in src/wallet/wallet.cpp:757 in 5107e4e730 outdated
    751@@ -752,8 +752,7 @@ bool CWallet::IsSpent(const COutPoint& outpoint) const
    752         const uint256& wtxid = it->second;
    753         const auto mit = mapWallet.find(wtxid);
    754         if (mit != mapWallet.end()) {
    755-            int depth = GetTxDepthInMainChain(mit->second);
    756-            if (depth > 0  || (depth == 0 && !mit->second.isAbandoned()))
    757+            if (mit->second.isConfirmed() || mit->second.InMempool() || (mit->second.isInactive() && !mit->second.isAbandoned()))
    


    murchandamus commented at 10:40 pm on November 8, 2023:

    In 5107e4e730a150068c5a214020a6393d2ba33aa8 “wallet: use CWalletTx member functions to determine tx state”:

    While I see why TxStateConflicted would be excluded here, it is not quite obvious to me why your replacement is equivalent. Could you perhaps expand a bit on this in a comment or the commit message?


    ishaanam commented at 10:05 pm on November 10, 2023:

    depth > 0 || (depth == 0 && !mit->second.isAbandoned()) is saying that the spending transaction should be confirmed, or: must not be (block) conflicted, and it must not be abandoned, for the corresponding output to be considered spent.

    mit->second.isConfirmed() || mit->second.InMempool() || (mit->second.isInactive() && !mit->second.isAbandoned()) is saying that the spending transaction should be confirmed, or in the mempool, or: must be inactive and not abandoned for the corresponding output to be considered spent.

    In this context, saying that a transaction must not be conflicted is logically equivalent to saying that the transaction must be inactive and not abandoned. I will add a comment to clarify this.


    ryanofsky commented at 2:44 pm on January 31, 2024:

    In commit “wallet: use CWalletTx member functions to determine tx state” (86aaf81f81b48e82358db6700a6c9d46c5b5b0df)

    Code comment here is just repeating the boolean expression and not explaining anything. I think the simplest explanation of this is “An output is considered spent by a spending transaction, unless the spending transaction is conflicted or abandoned.” Could also add “But the check below is written to check for valid spending transactions instead of invalid ones so it isn’t tied to wallet conflict tracking code which is imperfect and could change.”

  56. in src/wallet/interfaces.cpp:95 in 920f16c9a1 outdated
    90@@ -91,7 +91,7 @@ WalletTxStatus MakeWalletTxStatus(const CWallet& wallet, const CWalletTx& wtx)
    91     WalletTxStatus result;
    92     result.block_height =
    93         wtx.state<TxStateConfirmed>() ? wtx.state<TxStateConfirmed>()->confirmed_block_height :
    94-        wtx.state<TxStateConflicted>() ? wtx.state<TxStateConflicted>()->conflicting_block_height :
    


    murchandamus commented at 10:43 pm on November 8, 2023:

    In 920f16c9a11d167444f509a9fd2f0244f1f635eb “scripted-diff: wallet: s/TxStateConflicted/TxStateBlockConflicted”:

    I was considering whether there might be a way to better express that the transaction is in conflict with a confirmed transaction. How about:

    0        wtx.state<TxStateConfirmedConflict>() ? wtx.state<TxStateConfirmedConflict>()->conflicting_block_height :
    

    ishaanam commented at 1:00 am on November 11, 2023:

    I think that TxStateBlockConflicted and TxStateMempoolConflicted are a bit more clear, because with something like TxStateUnconfirmedConflict instead of TxStateMempoolConflicted, it is unclear whether the conflict is in the mempool, or just hasn’t been broadcasted yet. Additionally, I felt that it was more uniform with the other transaction states to be describing the state of the transaction instead of the state of a conflict. Eg. TxStateBlockConflicted indicates that the transaction is block-conflicted, and similarly TxStateConfirmed indicates that the transaction is confirmed. On the other hand, TxStateConfirmedConflict indicates the state of the conflict.

    That being said, if you or others think that TxStateConfirmedConflict or something similar is better, I would be open to changing it.


    murchandamus commented at 9:19 pm on January 29, 2024:
    I don’t feel strongly about it, please feel free to mark this comment as resolved.
  57. in src/wallet/wallet.cpp:1431 in 28a7e883c2 outdated
    1432+    for (const CTxIn& tx_in : tx->vin) {
    1433+        // If the wallet contains no transactions that spend the same prevout, continue
    1434+        if (mapTxSpends.count(tx_in.prevout) == 0) continue;
    1435+
    1436+        std::pair<TxSpends::const_iterator, TxSpends::const_iterator> range;
    1437+        range = mapTxSpends.equal_range(tx_in.prevout);
    


    murchandamus commented at 7:30 pm on November 9, 2023:

    In 28a7e883c2f9249213fddbe24948ec5bb90b0fad “wallet: track mempool conflicts”:

    Note to self and other reviewers: TxSpends is an unordered_multimap which permits multiple entries to have the same key, and equal_range(…) returns the range of iterators that match the given key denominated in first iterator to first iterator after the range.

  58. in test/functional/wallet_abandonconflict.py:238 in 28a7e883c2 outdated
    231@@ -232,7 +232,9 @@ def run_test(self):
    232         balance = newbalance
    233 
    234         # Invalidate the block with the double spend. B & C's 10 BTC outputs should no longer be available
    235-        self.nodes[0].invalidateblock(self.nodes[0].getbestblockhash())
    236+        blk = self.nodes[0].getbestblockhash()
    237+        self.generate(self.nodes[1], 10)
    


    murchandamus commented at 7:47 pm on November 9, 2023:
    In 28a7e883c2f9249213fddbe24948ec5bb90b0fad “wallet: track mempool conflicts”: Could you perhaps explain what the purpose of mining ten blocks here is?

    ishaanam commented at 4:51 am on November 11, 2023:

    I’ve added this comment explaining why 10 blocks need to be mined:

    0# mine 10 blocks so that when the blk is invalidated, the transactions are not
    1# returned to the mempool
    
  59. in test/functional/wallet_conflicts.py:155 in 003efbbe45 outdated
    150+        tx2 = alice.signrawtransactionwithwallet(raw_tx)['hex']
    151+
    152+        raw_tx = alice.createrawtransaction(inputs=[unspents[2]], outputs=[{bob.getnewaddress() : 24.9899}])
    153+        tx3 = alice.signrawtransactionwithwallet(raw_tx)['hex']
    154+
    155+        tx1 = alice.sendrawtransaction(tx1)
    


    murchandamus commented at 8:08 pm on November 9, 2023:
    In 003efbbe45079c4416810a025b2bc372559dff15 “test: Add tests for wallet mempool conflicts”: It’s a bit confusing that you take an input tx1 here which is the ‘hex’ string of the raw transaction, but then assign the resulting ’txid’ back to tx1.

    ishaanam commented at 4:53 am on November 11, 2023:
    I’ve replaced tx1 with tx1_txid here and for all other instances of this.
  60. in test/functional/wallet_conflicts.py:195 in 003efbbe45 outdated
    169+
    170+        tx3 = alice.sendrawtransaction(tx3)
    171+
    172+        assert_equal(alice.gettransaction(tx1)["mempoolconflicts"], [])
    173+
    174+        # now all of alice's outputs should be considered spent
    


    murchandamus commented at 8:09 pm on November 9, 2023:
    In 003efbbe45079c4416810a025b2bc372559dff15 “test: Add tests for wallet mempool conflicts”: Since tx2 replaced tx1, why would unspents[0] be considered spent here?

    ishaanam commented at 10:49 pm on November 10, 2023:
    Because tx3 replaced tx2, so tx1 is now TxStateInactive instead of TxStateMempoolConflicted, making unspents[0] spent.

    furszy commented at 3:08 pm on February 7, 2024:
    Would be good to document the behavior inside the code. It is not immediately evident only by reading the code.

    ishaanam commented at 5:48 pm on February 9, 2024:
    I have added a comment before all of the tests in this file explaining how these tests create conflicts.
  61. in test/functional/wallet_conflicts.py:218 in 003efbbe45 outdated
    213+
    214+        raw_tx = bob.createrawtransaction(inputs=[bob_unspents[0], bob_unspents[1]], outputs=[{bob.getnewaddress() : 49.999}])
    215+        raw_tx3 = bob.signrawtransactionwithwallet(raw_tx)['hex']
    216+        tx3 = bob.sendrawtransaction(raw_tx3)
    217+
    218+        # alice has 0 txs, bob has 3
    


    murchandamus commented at 8:24 pm on November 9, 2023:
    In 003efbbe45079c4416810a025b2bc372559dff15 “test: Add tests for wallet mempool conflicts”: I’m having trouble keeping track of all the things going on here. Perhaps you could add in these comments which transactions each node now knows about

    ishaanam commented at 4:37 am on November 15, 2023:
    I agree that it gets a bit confusing, so I’ve added a lot more comments.
  62. in test/functional/wallet_conflicts.py:231 in 003efbbe45 outdated
    222+        assert_equal(bob.getbalances()["mine"]["untrusted_pending"], Decimal("49.99900000"))
    223+
    224+        # bob broadcasts tx_1 conflict
    225+        tx1_conflict = bob.sendrawtransaction(tx1_conflict)
    226+        assert_equal(len(alice.getrawmempool()), 0)
    227+        assert_equal(len(bob.getrawmempool()), 2)
    


    murchandamus commented at 8:27 pm on November 9, 2023:
    In 003efbbe45079c4416810a025b2bc372559dff15 “test: Add tests for wallet mempool conflicts”: tx1_conflict should displace tx1, but why does Bob not have 3 again now? Shouldn’t Bob know about tx1, tx2, and tx3 before, and then tx1_conflict, tx2, and tx3 after?

    ishaanam commented at 4:35 am on November 15, 2023:
    tx1_conflict also displaces tx3 because tx3 spends tx1, so it is also considered mempool-conflicted. So instead tx1_conflict and tx2 are the only transactions left in the mempool. I’ve added a comment indicating this.
  63. ishaanam force-pushed on Nov 11, 2023
  64. ishaanam force-pushed on Nov 11, 2023
  65. DrahtBot added the label CI failed on Nov 11, 2023
  66. DrahtBot removed the label Needs rebase on Nov 11, 2023
  67. ishaanam force-pushed on Nov 11, 2023
  68. ishaanam force-pushed on Nov 11, 2023
  69. ishaanam commented at 4:58 am on November 11, 2023: contributor
    Rebased and addressed most of the review comments (thanks @achow101 and @murchandamus). I will address the remaining comments soon.
  70. ishaanam force-pushed on Nov 15, 2023
  71. ishaanam commented at 4:40 am on November 15, 2023: contributor
    I have addressed the remaining questions, the only changes are the addition of comments to the functional test.
  72. DrahtBot removed the label CI failed on Nov 15, 2023
  73. in src/wallet/wallet.cpp:755 in 80ba1d97f1 outdated
    751@@ -752,8 +752,9 @@ bool CWallet::IsSpent(const COutPoint& outpoint) const
    752         const uint256& wtxid = it->second;
    753         const auto mit = mapWallet.find(wtxid);
    754         if (mit != mapWallet.end()) {
    755-            int depth = GetTxDepthInMainChain(mit->second);
    756-            if (depth > 0  || (depth == 0 && !mit->second.isAbandoned()))
    757+            // A transaction is considered spent if a spending transaction
    


    murchandamus commented at 6:48 pm on December 27, 2023:

    In “wallet: use CWalletTx member functions to determine tx state” (80ba1d97f179121736d702b2d54bd3938cb57d96):

    Did you mean “An output is considered spent…”?


    ishaanam commented at 8:28 pm on December 27, 2023:
    Fixed
  74. murchandamus commented at 7:18 pm on December 27, 2023: contributor
    ACK 22778ac98f1bcb695de8ccf0b02fe9ebe39b01aa
  75. ishaanam force-pushed on Dec 27, 2023
  76. murchandamus commented at 9:19 pm on December 27, 2023: contributor
  77. DrahtBot added the label CI failed on Jan 13, 2024
  78. DrahtBot added the label Needs rebase on Jan 24, 2024
  79. ishaanam force-pushed on Jan 29, 2024
  80. DrahtBot removed the label Needs rebase on Jan 29, 2024
  81. in src/wallet/transaction.h:58 in a6ae5b23b1 outdated
    52+    std::string toString() const { return strprintf("BlockConflicted (block=%s, height=%i)", conflicting_block_hash.ToString(), conflicting_block_height); }
    53+};
    54+
    55+//! State of transaction with one or more mempool conflicts.
    56+struct TxStateMempoolConflicted {
    57+    std::string toString() const { return strprintf("MempoolConflicted"); }
    


    murchandamus commented at 9:24 pm on January 29, 2024:

    In a6ae5b23b1497ab6f4899db49348db623700a2d8 “wallet: track mempool conflicts”:

    Nit: Should this perhaps also inform about the conflicting transactions like the BlockConflicted state informs about the block?


    ishaanam commented at 2:48 am on February 2, 2024:
    This could be done in a follow-up if people want it, because currently the conflicting transactions are stored in the CWalletTx, and not the transaction state, so this function is not aware of the txids of the mempool conflicts.
  82. murchandamus commented at 9:45 pm on January 29, 2024: contributor
    ACK 71059cf4267d80d2934b85a7046fc7b64900378b
  83. DrahtBot removed the label CI failed on Jan 29, 2024
  84. in src/wallet/transaction.h:268 in a6ae5b23b1 outdated
    264@@ -258,6 +265,8 @@ class CWalletTx
    265     CTransactionRef tx;
    266     TxState m_state;
    267 
    268+    std::set<uint256> mempool_conflicts;
    


    glozow commented at 3:03 pm on January 30, 2024:

    in a6ae5b23b1497ab6f4899db49348db623700a2d8:

    It’s best to use Txid or Wtxid for this. And maybe add a doc stating whether these are direct conflicts only, or if they might be descendants of a conflicting transaction?


    ishaanam commented at 2:50 am on February 2, 2024:
    Done
  85. in src/wallet/wallet.cpp:1423 in a6ae5b23b1 outdated
    1418@@ -1419,6 +1419,36 @@ void CWallet::transactionAddedToMempool(const CTransactionRef& tx) {
    1419     if (it != mapWallet.end()) {
    1420         RefreshMempoolStatus(it->second, chain());
    1421     }
    1422+
    1423+    const uint256& conflict_hash = tx->GetHash();
    


    glozow commented at 3:04 pm on January 30, 2024:

    in a6ae5b23b1497ab6f4899db49348db623700a2d8

    same, auto or Txid would be better


    ishaanam commented at 2:50 am on February 2, 2024:
    Done
  86. in src/wallet/wallet.cpp:1435 in a6ae5b23b1 outdated
    1430+        range = mapTxSpends.equal_range(tx_in.prevout);
    1431+
    1432+        // For each wallet transaction spending this prevout...
    1433+        for (TxSpends::const_iterator _it = range.first; _it != range.second; ++_it) {
    1434+            // if this tx is the same as the tx which was just added to the mempool, continue
    1435+            if (_it->second == conflict_hash) continue;
    


    glozow commented at 4:15 pm on January 30, 2024:

    in 067189eddee1a2209c6c42741a91601c5514e170

    Got a little nervous seeing the txid comparison and “same tx.” Is it possible to run into a same-txid-different-witness situation here?


    ishaanam commented at 5:45 pm on January 30, 2024:
    Good catch, I think that is possible. Would just using wtxids everywhere here fix this?

    glozow commented at 9:57 am on January 31, 2024:
    Yes I’d expect so

    ishaanam commented at 2:51 am on February 2, 2024:
    Thanks, that should be fixed now

    achow101 commented at 10:14 pm on February 2, 2024:
    If we get the same txid but different wtxid, I’m not sure that the result is necessarily one that makes sense. Most things in the wallet are keyed and listed by txid, so with this change, we’d end up with a transaction that’s listed as conflicted by it’s own txid. Also, we won’t ever update the wallet to store a transaction with a different wtxid.

    ishaanam commented at 9:04 pm on February 9, 2024:

    If we get the same txid but different wtxid, I’m not sure that the result is necessarily one that makes sense. Most things in the wallet are keyed and listed by txid, so with this change, we’d end up with a transaction that’s listed as conflicted by it’s own txid. Also, we won’t ever update the wallet to store a transaction with a different wtxid.

    Sorry I didn’t see this earlier. After looking at this again I don’t think that same-txid-different-witness will be a problem here. Please see: #27307 (review)

  87. in src/wallet/rpc/transactions.cpp:424 in c71736d372 outdated
    420@@ -417,6 +421,10 @@ static std::vector<RPCResult> TransactionDescriptionString()
    421            }},
    422            {RPCResult::Type::STR_HEX, "replaced_by_txid", /*optional=*/true, "Only if 'category' is 'send'. The txid if this tx was replaced."},
    423            {RPCResult::Type::STR_HEX, "replaces_txid", /*optional=*/true, "Only if 'category' is 'send'. The txid if this tx replaces another."},
    424+           {RPCResult::Type::ARR, "mempoolconflicts", "Conflicting transaction currently in the mempool.",
    


    glozow commented at 4:16 pm on January 30, 2024:

    c71736d372c65b0978671501029e202574c6bbef

    Also good to document - is it just directly conflicting, or descendants as well?


    ishaanam commented at 2:51 am on February 2, 2024:
    Done
  88. in test/functional/wallet_conflicts.py:153 in 71059cf426 outdated
    140+        self.generate(self.nodes[2], 1)
    141+
    142+        self.log.info("Test a scenario where a transaction has a mempool conflict")
    143+
    144+        unspents = alice.listunspent()
    145+        assert_equal(len(unspents), 3)
    


    glozow commented at 4:34 pm on January 30, 2024:

    71059cf4267d80d2934b85a7046fc7b64900378b

    There’s not anything preventing us from checking the exact contents of listunspent right? Here and in the rest of the test.


    ishaanam commented at 2:51 am on February 2, 2024:
    Good point, the tests now check the exact contents.
  89. in test/functional/wallet_conflicts.py:192 in 71059cf426 outdated
    168+
    169+        self.log.info("Test scenario where a mempool conflict is removed")
    170+
    171+        alice.sendrawtransaction(tx3)
    172+
    173+        assert_equal(alice.gettransaction(tx1_txid)["mempoolconflicts"], [])
    


    glozow commented at 4:38 pm on January 30, 2024:

    71059cf4267d80d2934b85a7046fc7b64900378b

    We don’t expect to see tx1 in mempool, right? It’s not being rebroadcast, as it’s inactive?

    0        assert tx1_txid not in self.nodes[0].getrawmempool()
    

    Might be good to have a check for this behavior.


    ishaanam commented at 2:52 am on February 2, 2024:
    I’ve added an assertion that the tx isn’t in the mempool.
  90. in test/functional/wallet_conflicts.py:141 in 71059cf426 outdated
    129@@ -123,5 +130,165 @@ def run_test(self):
    130         assert_equal(former_conflicted["confirmations"], 1)
    131         assert_equal(former_conflicted["blockheight"], 217)
    132 
    133+    def test_mempool_conflict(self):
    


    glozow commented at 4:40 pm on January 30, 2024:
    Not sure if in scope for this PR, but it would be nice to have a test for how the wallet sees descendants of conflicting transactions (e.g. if Bob creates a child off of tx1 or tx2).

    ishaanam commented at 2:55 am on February 2, 2024:
    Yes, I will add these in a follow-up PR.

    ishaanam commented at 5:50 am on February 5, 2024:
    Actually, I have just added a test in this PR.
  91. glozow commented at 4:40 pm on January 30, 2024: member
    Approach ACK
  92. bitcoin deleted a comment on Jan 30, 2024
  93. in src/wallet/transaction.h:54 in a6ae5b23b1 outdated
    47@@ -48,7 +48,12 @@ struct TxStateBlockConflicted {
    48     int conflicting_block_height;
    49 
    50     explicit TxStateBlockConflicted(const uint256& block_hash, int height) : conflicting_block_hash(block_hash), conflicting_block_height(height) {}
    51-    std::string toString() const { return strprintf("Conflicted (block=%s, height=%i)", conflicting_block_hash.ToString(), conflicting_block_height); }
    52+    std::string toString() const { return strprintf("BlockConflicted (block=%s, height=%i)", conflicting_block_hash.ToString(), conflicting_block_height); }
    53+};
    54+
    55+//! State of transaction with one or more mempool conflicts.
    


    ryanofsky commented at 3:01 pm on January 31, 2024:

    In commit “wallet: track mempool conflicts” (a6ae5b23b1497ab6f4899db49348db623700a2d8)

    I think it would be useful to add that mempool conflicted transactions are serialized as inactive transactions, so the distinction between TxStateMempoolConflicted and TxStateInactive only exists at runtime.


    ishaanam commented at 5:50 am on February 5, 2024:
    I have added a comment indicating this.
  94. ryanofsky commented at 3:34 pm on January 31, 2024: contributor

    Code review 71059cf4267d80d2934b85a7046fc7b64900378b

    I’m starting to review this, but I don’t understand how it interacts with abandoned transactions. Is it still possible to abandon a transaction with conflicts with the mempool? If a transaction is already abandoned and we know it conflicts with the mempool does isMempoolConflicted still return false?

    It seems to me like a more straightforward way to implement this feature would be to add a bool mempool_conflicts field to TxStateInactive rather than adding a new TxStateMempoolConflicted state. The description of TxStateInactive says “May conflict with the mempool, or with an unknown block, or be abandoned, never broadcast, or rejected from the mempool for another reason.” So the inactive state was meant to handle transactions that could have conflicts or be inactive for a variety of reasons.

    I think I would recommend removing the new MempoolConflicted state here, but otherwise keeping this PR unchanged. The mempool tracking feature, renames, and other cleanups here do seem very nice, so it is good to see this work.

  95. ishaanam force-pushed on Feb 2, 2024
  96. in src/wallet/wallet.cpp:1511 in 5fbf18b89d outdated
    1506+
    1507+                    // Remove the previously conflicting transaction from this wtx's entry
    1508+                    wtx.mempool_conflicts.erase(conflict_txid);
    1509+
    1510+                    if (wtx.mempool_conflicts.empty()) {
    1511+                        // If this wtx has no other mempool conflicts, erase this wtx entry
    


    achow101 commented at 10:16 pm on February 2, 2024:

    In 5fbf18b89d6768bed1be95dcbc98797df5788dfc “wallet: track mempool conflicts”

    I’m a little bit confused by the “erase this wtx entry” part of the comment. AFAICT, no entries are being erased, Perhaps this should say:

    0                        // If this wtx has no other mempool conflicts, mark it as inactive
    

    ishaanam commented at 5:53 am on February 5, 2024:
    Good point, this comment is from a previous version of the PR. I have fixed it.
  97. achow101 commented at 10:37 pm on February 2, 2024: member

    It seems to me like a more straightforward way to implement this feature would be to add a bool mempool_conflicts field to TxStateInactive rather than adding a new TxStateMempoolConflicted state. The description of TxStateInactive says “May conflict with the mempool, or with an unknown block, or be abandoned, never broadcast, or rejected from the mempool for another reason.” So the inactive state was meant to handle transactions that could have conflicts or be inactive for a variety of reasons.

    Personally, I don’t like how TxStateInactive is so broad. It’s kind of a catch-all “transaction is neither in a block nor in the mempool, and we don’t know why” (except for abandoned, and even then, I don’t like how abandoned is a part of it and not a separate state). This PR introduces a distinct state of “transaction currently has things in the mempool that conflict with it” which is specific and we know the reason that something is not in the mempool, so I think it makes sense to be a separate state. I think it’s more useful to carve out from TxStateInactive more specific states with reasons that we know for sure why something is in that state.

  98. ryanofsky commented at 0:43 am on February 3, 2024: contributor

    Personally, I don’t like how TxStateInactive is so broad. […]

    Sure, but then I’m pretty sure we need to add a bool abandoned; field to the TxStateMempoolConflicted struct so we don’t lose track of whether the user has abandoned the transaction.

    Alternately, we could go more in the other direction, getting rid of the inactive catch-all and dividing TxStateInactive into 4 states:

    • TxStateAbandonedAndConflictsWithMempool
    • TxStateAbandonedAndProbablyDoesntConflictWithMempool
    • TxStateNotAbandonedAndConflictsWithMempool
    • TxStateNotAbandonedAndProbablyDoesntConflictWithMempool

    Personally I think just keeping the TxStateInactive state with bool abandoned; bool mempool_conflicts; is better than subdividing it, because most wallet code should just need to consider whether transactions are active or not, without reverse-engineering the causes of transaction becoming inactive and making special cases out of them. But any of these approaches would be fine as long as we aren’t losing track of whether a transaction is abandoned after a conflict is detected.

  99. ryanofsky commented at 0:56 am on February 3, 2024: contributor

    Personally, I don’t like how TxStateInactive is so broad. […]

    I will also add that I think adding a new top-level state instead of just adding a new field to TxStateInactive makes this PR considerably harder to review because you have to manually search for every line of code that is accessing the m_state field directly or indirectly and verify that it is either:

    1. Treating the TxStateInactive and TxStateMempoolConflicted states equivalently and not changing behavior
    2. Treating the TxStateInactive and TxStateMempoolConflicted states differently and changing behavior in a useful way.

    If can avoid adding a new top-level state it would make this PR easier to review, because you could focus on the second case, and easily see where behavior is changing and not changing, which I don’t think is possible with the current approach.

    This is not a fundamental objection, but might be something to consider.

  100. achow101 commented at 1:25 am on February 3, 2024: member

    Sure, but then I’m pretty sure we need to add a bool abandoned; field to the TxStateMempoolConflicted struct so we don’t lose track of whether the user has abandoned the transaction.

    We already lose track of that if a conflict is created and that conflict confirms. Abandoned is not a permanent state - as soon as it shows back up in the mempool, it’ll no longer be abandoned. As soon as a conflict appears in a block, it’ll no longer be abandoned. Likewise, if a transaction shows up in the mempool that conflicts with it, it should reflect that it is now conflicted and no longer needs to be marked as abandoned. Abandoned has always been a transient state since someone else could rebroadcast the transaction to the network. Once there’s something else that makes the transaction invalid, I don’t think it’s useful to keep track of whether it is abandoned, so we wouldn’t need to have more states to track that.

  101. ryanofsky commented at 2:11 am on February 3, 2024: contributor

    Abandoned is not a permanent state - as soon as it shows back up in the mempool, it’ll no longer be abandoned.

    It makes sense that a transaction in the mempool can’t be treated as abandoned. But I don’t see why the abandoned state be should be cleared just because a mempool heuristic happened to detect a conflict. That just seems like a bad UX and a confusion of two orthogonal concepts. The abandoned state is transient in the sense that it can be overriden by mempool events and block confirmations, but it should be less transient than the mempool conflicted state because it (1) reflects user intent and (2) is actually serialized to disk, so it doesn’t seem to me something that should be thrown away unnecessarily.

    Also, the PR description doesn’t mention anything about discarding the abandoned state so this seems to me like an unintentional bug than a sensible design choice.

    I would just recommend adding the conflict tracking feature without adding a new top level state and without changing the behavior of abandoned transactions (especially unintentionally). It would shrink this PR, make it safer and easier to review, and provide all the benefits of the current implementation. IMO, if we really do want to change the transaction state diagram and add new top level states, I think it would be better to do in a separate PR not tied to adding a new tracking feature.

  102. ishaanam force-pushed on Feb 5, 2024
  103. ishaanam commented at 7:08 am on February 5, 2024: contributor

    Is it still possible to abandon a transaction with conflicts with the mempool?

    Yes, because of this change.

    If a transaction is already abandoned and we know it conflicts with the mempool does isMempoolConflicted still return false?

    Yes

    Sure, but then I’m pretty sure we need to add a bool abandoned; field to the TxStateMempoolConflicted struct so we don’t lose track of whether the user has abandoned the transaction.

    If the transaction has been abandoned, then it won’t be marked as TxStateMempoolConflicted because of these lines of code. Although the conflicting transactions are kept track of regardless, the actual transaction state only changes if the transaction is inactive (not abandoned).

    I would just recommend adding the conflict tracking feature without adding a new top level state and without changing the behavior of abandoned transactions (especially unintentionally). It would shrink this PR, make it safer and easier to review, and provide all the benefits of the current implementation. IMO, if we really do want to change the transaction state diagram and add new top level states, I think it would be better to do in a separate PR not tied to adding a new tracking feature.

    From what I can tell, this PR is not changing the behavior of abandoned transactions. That being said, if anybody still thinks that TxStateMempoolConflicted should be removed in favor of just adding a boolean to TxStateInactive, I would be willing to implement that. I just think that TxStateMempoolConflicted makes more of a transition towards having more specific transaction states, which seems to be something we want.

  104. achow101 commented at 10:18 pm on February 6, 2024: member
    ACK 600fc562f815941b4a95c9f4f357dc6eed04be6c
  105. DrahtBot requested review from murchandamus on Feb 6, 2024
  106. DrahtBot requested review from glozow on Feb 6, 2024
  107. in src/wallet/wallet.cpp:757 in 86aaf81f81 outdated
    751@@ -752,8 +752,9 @@ bool CWallet::IsSpent(const COutPoint& outpoint) const
    752         const uint256& wtxid = it->second;
    753         const auto mit = mapWallet.find(wtxid);
    754         if (mit != mapWallet.end()) {
    755-            int depth = GetTxDepthInMainChain(mit->second);
    756-            if (depth > 0  || (depth == 0 && !mit->second.isAbandoned()))
    757+            // An output is considered spent if a spending transaction
    758+            // is either confirmed, or in the mempool, or inactive (but not abandoned)
    759+            if (mit->second.isConfirmed() || mit->second.InMempool() || (mit->second.isInactive() && !mit->second.isAbandoned()))
    


    furszy commented at 2:36 pm on February 7, 2024:

    In 86aaf81f8:

    If you touch this again; could pull mit->second into its own variable to improve readability.

    0const auto& wtx = mit->second;
    1// An output is considered spent by a spending transaction,
    2// unless the spending transaction is conflicted or abandoned.
    3if (wtx.isConfirmed() || wtx.InMempool() || (wtx.isInactive() && !wtx.isAbandoned()))
    4    return true;
    

    ishaanam commented at 5:39 pm on February 9, 2024:
    Done
  108. in src/wallet/wallet.cpp:1437 in 3de89342dc outdated
    1432+
    1433+        // For each wallet transaction spending this prevout...
    1434+        for (TxSpends::const_iterator _it = range.first; _it != range.second; ++_it) {
    1435+            // if this tx is the same as the tx which was just added to the mempool, continue
    1436+            auto it = mapWallet.find(_it->second);
    1437+            if (it->second.tx->GetWitnessHash() == conflict_wtxid) continue;
    


    furszy commented at 2:54 pm on February 7, 2024:

    Would be good to explain that you are doing the mapWallet lookup because the wtx could have been updated. Because otherwise, this could just be a if (_it->second == conflict_txid) continue; right?.

    Also, as you already know that the output belongs to one of the wallet’s txs, could:

    0CWalletTx& wtx = mapWallet.at(_it->second);
    1if (wtx.tx->GetWitnessHash() == conflict_wtxid) continue;
    

    ishaanam commented at 8:59 pm on February 9, 2024:

    Would be good to explain that you are doing the mapWallet lookup because the wtx could have been updated.

    I just realized that the wtx could not have been updated, because for it to be updated, transactionAddedToMempool would need to be called, which would be after this call, right? So I have reverted the change which compares witness hashes.

    Also, as you already know that the output belongs to one of the wallet’s txs, could:

    Yes, I have implemented this.

  109. in src/wallet/wallet.cpp:1514 in 3de89342dc outdated
    1512+                        if (wtx.isMempoolConflicted()) {
    1513+                            wtx.m_state = TxStateInactive{};
    1514+                            return TxUpdate::CHANGED;
    1515+                        }
    1516+                    }
    1517+                }
    


    furszy commented at 3:00 pm on February 7, 2024:

    Can remove the initial wtx.mempool_conflicts.empty() call and directly call erase.

     0auto try_updating_state = [&](CWalletTx& wtx) EXCLUSIVE_LOCKS_REQUIRED(cs_wallet) {
     1    // Remove the previously conflicting transaction from this wtx's entry
     2    wtx.mempool_conflicts.erase(conflict_txid);
     3    if (wtx.mempool_conflicts.empty()) {
     4        // If this wtx has no other mempool conflicts, it is now considered inactive
     5        if (wtx.isMempoolConflicted()) {
     6            wtx.m_state = TxStateInactive{};
     7            return TxUpdate::CHANGED;
     8        }
     9    }
    10    return TxUpdate::UNCHANGED;
    11};
    

    ishaanam commented at 5:48 pm on February 9, 2024:
    Done
  110. in test/functional/wallet_conflicts.py:179 in 600fc562f8 outdated
    162+
    163+        tx2_txid = alice.sendrawtransaction(tx2)
    164+
    165+        # Check that the 0th unspent is now available because the transaction spending it has been replaced in the mempool
    166+        assert_equal(alice.listunspent(), [unspents[0]])
    167+        assert_equal(alice.getbalance(), 25)
    


    furszy commented at 3:04 pm on February 7, 2024:
    Could document why it has been replaced here. It is not immediately evident only by reading the code.

    ishaanam commented at 5:42 pm on February 9, 2024:
    I have added a lot more comments to the test which should clarify this.
  111. in test/functional/wallet_conflicts.py:204 in 600fc562f8 outdated
    178+        # now all of alice's outputs should be considered spent
    179+        assert_equal(alice.listunspent(), [])
    180+        assert_equal(alice.getbalance(), 0)
    181+
    182+        bob.sendall([self.nodes[2].getnewaddress()])
    183+        self.generate(self.nodes[2], 1)
    


    furszy commented at 3:12 pm on February 7, 2024:
    Would be good to document why you are doing this. I guess that you are cleaning up the mempool and wallet for the next test?

    ishaanam commented at 5:40 pm on February 9, 2024:
    Yes, I’ve added a comment for that.
  112. furszy commented at 3:14 pm on February 7, 2024: member
    light review, left few comments, nothing big.
  113. ishaanam force-pushed on Feb 8, 2024
  114. DrahtBot commented at 3:30 am on February 9, 2024: contributor

    🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the documentation.

    Possibly this is due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    Leave a comment here, if you need help tracking down a confusing failure.

    Debug: https://github.com/bitcoin/bitcoin/runs/21385308940

  115. DrahtBot added the label CI failed on Feb 9, 2024
  116. ishaanam force-pushed on Feb 9, 2024
  117. ryanofsky commented at 5:21 pm on February 9, 2024: contributor

    re: #27307 (comment)

    Thanks ishaanam for pointing out the lines of code that answer my questions. Since original links got broken in rebase, here are stable links that point to the two lines of code which prioritize the abandoned state over the conflicted state:

    This is better than what I was afraid of, since the abandoned state consistently takes priority over the mempoolconflicted state. But this behavior is fragile and unnecessarily complicates our internal model and potential RPC and GUI interfaces, because there’s no logical reason for the abandoned and mempool conflicted states to contradict each other, when both can be true at the same time.

    If we take the current approach in 3471ebc8c6ee268ad501525121ea3f3a221fb5ad, we will have to be careful going forward, whenever we do any state change, to remember that the abandoned state is supposed to take priority over the mempool conflicted state (this is not even documented anywhere), and decide how we want to deal with potentially odd states like mempool_conflicts set being nonempty when the transaction is not in the MempoolConflicted state.

    For background, I already think it’s bad that abandoned and block-conflicted states exclude each other, and that the block conflicted state takes priority over the abandoned state and resets it. But the block conflicted state is pretty permanent, so it’s basically just a UI quirk if a block conflict causes the abandoned state to be forgotten about. There are also backwards compatibility benefits to storing the block conflict and abandoned bit exclusively in the same field.

    In the current case, though, there are no backwards compatibility concerns, so it seems wrong and unnecessary to go out our way to create an unenforced, undocumented, and confusing hierarchy of states: MempoolConflicted < Abandoned < BlockConflicted, when there is a simpler alternative of just straightforwardly representing the information we have about transactions directly.

    It seems pretty clear to me that it would make this change simpler, safer and easier to review, make the code more straightforward and maintainable long term, and avoid potential bugs and confusion in the UI if it just represented abandoned and mempool conflicted states independently and didn’t try to establish a hierarchy between them.

    I think a good approach would avoid adding a TxStateMempoolConflicted top-level state, and just add a bool mempool_conflicts or std::set<Txid> mempool_conflicts member to the TxStateInactive struct. And it is good to rename TxStateConflicted to TxStateBlockConflicted too for clarity. If you’re skeptical of this approach, I’d be curious to know why, but we can also experiment in a separate PR if you are concerned there are benefits in this PR which would be lost.

    If it would be helpful, I could make another PR next week to compare the two approaches. But I’d be curious to know what you think. There is a lot of a complexity and a long history of bugs in wallet transaction state tracking (https://github.com/bitcoin-core/bitcoin-devwiki/wiki/Wallet-Transaction-Conflict-Tracking) so I think think it is worth taking a little more time to get this right.

  118. DrahtBot removed the label CI failed on Feb 9, 2024
  119. ishaanam force-pushed on Feb 11, 2024
  120. ishaanam commented at 5:19 am on February 11, 2024: contributor

    I think a good approach would avoid adding a TxStateMempoolConflicted top-level state, and just add a bool mempool_conflicts or std::set mempool_conflicts member to the TxStateInactive struct.

    Thanks for your feedback, I have implemented this.

  121. in src/wallet/transaction.h:338 in df0130d733 outdated
    334@@ -335,7 +335,7 @@ class CWalletTx
    335     void updateState(interfaces::Chain& chain);
    336 
    337     bool isAbandoned() const { return state<TxStateInactive>() && state<TxStateInactive>()->abandoned; }
    338-    bool isConflicted() const { return state<TxStateConflicted>(); }
    339+    bool isConflicted() const { return state<TxStateBlockConflicted>(); }
    


    ryanofsky commented at 6:01 pm on February 15, 2024:

    In commit “scripted-diff: wallet: s/TxStateConflicted/TxStateBlockConflicted” (df0130d73341bb2784b23cd2e8307f30da09ca37)

    Not very important, but I think it would be good if this scripted-diff commit renamed CWalletTx::isConflicted to isBlockConflicted up front, so isConflicted could be added as a new method later without affecting existing callers. Otherwise it takes extra effort for reviewers to look up all the existing callers of isConflicted recursively and making sure their behavior is not changing unintentionally.


    ishaanam commented at 10:48 pm on February 22, 2024:
    Done
  122. in src/wallet/wallet.cpp:756 in 523a48b718 outdated
    751@@ -752,8 +752,10 @@ bool CWallet::IsSpent(const COutPoint& outpoint) const
    752         const uint256& wtxid = it->second;
    753         const auto mit = mapWallet.find(wtxid);
    754         if (mit != mapWallet.end()) {
    755-            int depth = GetTxDepthInMainChain(mit->second);
    756-            if (depth > 0  || (depth == 0 && !mit->second.isAbandoned()))
    757+            // An output is considered spent if a spending transaction
    758+            // is either confirmed, or in the mempool, or inactive (but not abandoned)
    


    ryanofsky commented at 6:22 pm on February 15, 2024:

    In commit “wallet: use CWalletTx member functions to determine tx state” (523a48b7180409970cec286f0aef6afbbacd2ffa)

    This code comment is just repeating the boolean expression below without explaining it. Would suggest a comment explaining what it is trying to do like “An output is considered spent by a spending transaction unless the spending transaction is conflicted or abandoned. The check below is written conservatively to check for the inverse condition and only consider spending transactions in known, potentially active states.”

    (Made a similar suggestion earlier #27307 (review))


    ishaanam commented at 10:48 pm on February 22, 2024:
    Done
  123. in src/wallet/transaction.h:267 in 2c867784b2 outdated
    261@@ -258,6 +262,10 @@ class CWalletTx
    262     CTransactionRef tx;
    263     TxState m_state;
    264 
    265+    // Transactions that conflict directly with the tx, or
    266+    // that conflict with an ancestor transaction
    267+    std::set<Txid> mempool_conflicts;
    


    ryanofsky commented at 6:28 pm on February 15, 2024:

    In commit “wallet: track mempool conflicts” (2c867784b2a719c9e701e19e389c3adfa7d7faae)

    Would suggest expanding comment to something like: “Set of mempool transactions that conflict directly with the tx or an ancestor transaction. This set will be empty if the transaction is confirmed, but can be nonempty in other states.”

    I was initially surprised to see that this set was populated not just for inactive transactions, but also for block-conflicted transactions, so think it is worth mentioning the relationship to the transaction state.


    ishaanam commented at 10:48 pm on February 22, 2024:
    Done
  124. in src/wallet/wallet.cpp:761 in 2c867784b2 outdated
    754@@ -755,7 +755,7 @@ bool CWallet::IsSpent(const COutPoint& outpoint) const
    755             // An output is considered spent if a spending transaction
    756             // is either confirmed, or in the mempool, or inactive (but not abandoned)
    757             const auto& wtx = mit->second;
    758-            if (wtx.isConfirmed() || wtx.InMempool() || (wtx.isInactive() && !wtx.isAbandoned()))
    759+            if (wtx.isConfirmed() || wtx.InMempool() || (wtx.isInactive() && !wtx.isAbandoned() && !wtx.isMempoolConflicted()))
    


    ryanofsky commented at 6:35 pm on February 15, 2024:

    In commit “wallet: track mempool conflicts” (2c867784b2a719c9e701e19e389c3adfa7d7faae)

    I was surprised to see this change, since I thought intent of the preceding commit was to preserve current behavior and consider spending transactions active even they conflicted with mempool transactions. Both approaches seem ok, but if this behavior is intended, it seems like it would be simpler to write this condition as:

    0if (!wtx.IsAbandoned() && !wtx.isConflicted()) {
    1    return true; // spent
    2}
    

    ishaanam commented at 10:49 pm on February 22, 2024:
    The preceding was more of a refactor, while this commit aims to have IsSpent return false for txos spent by mempool-conflicted transactions.

    ryanofsky commented at 3:45 pm on February 23, 2024:

    In commit “wallet: track mempool conflicts” (a650caab9f11be7f5927f9aa556bb7d2a8ebed33)

    The preceding was more of a refactor, while this commit aims to have IsSpent return false for txos spent by mempool-conflicted transactions.

    Sounds good. I guess I’m still curious why you prefer over:

    0if (wtx.isConfirmed() || wtx.InMempool() || (wtx.isInactive() && !wtx.isAbandoned() && !wtx.isMempoolConflicted())
    

    over

    0if (!wtx.IsAbandoned() && !wtx.isConflicted())
    

    since both checks should equivalent. Either approach seems fine to me, though.

  125. in src/wallet/transaction.h:345 in 2c867784b2 outdated
    342@@ -335,7 +343,9 @@ class CWalletTx
    343     void updateState(interfaces::Chain& chain);
    344 
    345     bool isAbandoned() const { return state<TxStateInactive>() && state<TxStateInactive>()->abandoned; }
    346-    bool isConflicted() const { return state<TxStateBlockConflicted>(); }
    347+    bool isConflicted() const { return isBlockConflicted() || isMempoolConflicted(); }
    


    ryanofsky commented at 6:51 pm on February 15, 2024:

    In commit “wallet: track mempool conflicts” (2c867784b2a719c9e701e19e389c3adfa7d7faae)

    Note: effects of changing this definition seem to be:

    • CachedTxIsTrusted is changed, so if a mempool conflict with a transaction is detected, the transaction will no longer be trusted, and its outputs will show up in the unconfirmed balance and no longer be spent from.
    • isUnconfirmed is changed, so ResubmitWalletTransactions is changed, so if a mempool conflict with a transaction is detected, the wallet will not attempt to rebroadcast it.

    Would be good to mention these behavior changes and any others in the commit message. Also, if isConflicted were renamed to isBlockConflicted in the first commit, and then introduced as a new function here, these changes would be a lot more obvious because the relevant code would have visible diffs here.


    ishaanam commented at 10:50 pm on February 22, 2024:
    I have updated the commit message.
  126. in src/wallet/transaction.h:66 in 2c867784b2 outdated
    63-    std::string toString() const { return strprintf("Inactive (abandoned=%i)", abandoned); }
    64+    explicit TxStateInactive(bool abandoned = false, bool mempool_conflicted = false)
    65+      : abandoned(abandoned),
    66+        mempool_conflicted(mempool_conflicted)
    67+        {}
    68+    std::string toString() const { return strprintf("Inactive (abandoned=%i, mempool_conflicted=%i)", abandoned, mempool_conflicted); }
    


    ryanofsky commented at 7:22 pm on February 15, 2024:

    In commit “wallet: track mempool conflicts” (2c867784b2a719c9e701e19e389c3adfa7d7faae)

    It doesn’t seem actually seem useful to add this TxStateInactive::mempool_conflicted member, so would suggest dropping it and changing definition of isMempoolConflicted() to just return !mempool_conflicts.empty().

    When I made earlier suggestion to add this member in #27307 (comment) I didn’t dig into the relationship between the mempool conflicted state and mempool_conflicts variable, so I thought it might be needed to keep both, but it looks like mempool_conflicted is entirely redundant, so it would be confusing to have separate mempool_conflicts and mempool_conflicted variables for no reason.


    ishaanam commented at 10:50 pm on February 22, 2024:
    I have removed the mempool_conflicted variable.
  127. in src/wallet/wallet.cpp:1447 in 2c867784b2 outdated
    1442+
    1443+                if (wtx.isInactive()) {
    1444+                    wtx.m_state = TxStateInactive{wtx.isAbandoned(), /*mempool_conflicted=*/ true};
    1445+                    return TxUpdate::CHANGED;
    1446+                } else {
    1447+                    return TxUpdate::UNCHANGED;
    


    ryanofsky commented at 7:49 pm on February 15, 2024:

    In commit “wallet: track mempool conflicts” (2c867784b2a719c9e701e19e389c3adfa7d7faae)

    Seems like a minor bug here. If this returns TxUpdate::UNCHANGED it means RecursiveUpdateTxState will stop recursing, so the parent transaction will have its mempool_conflicts list updated, but its descendants will not.

    In practice, this should be a minor bug because the only time TxUpdate::UNCHANGED is actually returned here is when the transaction is in BlockConflicted state, and it shouldn’t be a big deal if mempool conflicts for child transactions aren’t added when we already know there is a block conflict. But it would be better to track mempool conflicts consistently, not just at the top level in this case.

    Taken with above suggestion do drop the mempool_conflicted variable, it seems like it would make sense to replace this logic with just:

    0auto inserted = wtx.mempool_conflicts.insert(conflict_txid);
    1if (inserted == wtx.mempool_conflicts.end()) TxUpdate::UNCHANGED;
    2return TxUpdate::CHANGED;
    

    ishaanam commented at 10:51 pm on February 22, 2024:
    Good point, this bug should be fixed now.
  128. in src/wallet/wallet.cpp:1451 in 2c867784b2 outdated
    1446+                } else {
    1447+                    return TxUpdate::UNCHANGED;
    1448+                }
    1449+            };
    1450+
    1451+            RecursiveUpdateTxState(_it->second, try_updating_state);
    


    ryanofsky commented at 7:56 pm on February 15, 2024:

    In commit “wallet: track mempool conflicts” (2c867784b2a719c9e701e19e389c3adfa7d7faae)

    It seems wasteful and potentially slow to use RecursiveUpdateTxState in its current form because it will be calling WalletBatch::WriteTx for these transactions and updating the database even though the transaction data will not be changing.

    Would suggest fixing this by adding a CWallet::RecursiveUpdateTxState overload that takes a WalletBatch* parameter and does not call WriteTx when it is null. E.g.:

     0void CWallet::RecursiveUpdateTxState(const uint256& tx_hash, const TryUpdatingStateFn& try_updating_state)
     1{
     2    WalletBatch batch(GetDatabase(), false);
     3    RecursiveUpdateTxState(&batch, tx_hash, try_updating_state);
     4}
     5
     6void CWallet::RecursiveUpdateTxState(WalletBatch* batch, const uint256& tx_hash, const TryUpdatingStateFn& try_updating_state)
     7{
     8    // ...
     9    if (batch) batch->WriteTx(wtx);
    10    // ...
    11}
    

    ishaanam commented at 10:51 pm on February 22, 2024:
    Done.
  129. in src/wallet/wallet.cpp:1444 in 2c867784b2 outdated
    1439+            auto try_updating_state = [&](CWalletTx& wtx) EXCLUSIVE_LOCKS_REQUIRED(cs_wallet) {
    1440+                // Add this conflict to the wtx's entry (create one if one does not already exist)
    1441+                wtx.mempool_conflicts.insert(conflict_txid);
    1442+
    1443+                if (wtx.isInactive()) {
    1444+                    wtx.m_state = TxStateInactive{wtx.isAbandoned(), /*mempool_conflicted=*/ true};
    


    ryanofsky commented at 8:03 pm on February 15, 2024:

    In commit “wallet: track mempool conflicts” (2c867784b2a719c9e701e19e389c3adfa7d7faae)

    Can drop this code if TxStateInactive::mempool_conflicted member is dropped as suggested above.


    ishaanam commented at 10:51 pm on February 22, 2024:
    Done
  130. in src/wallet/wallet.cpp:1501 in 2c867784b2 outdated
    1510+                    if (wtx.isMempoolConflicted()) {
    1511+                        wtx.m_state = TxStateInactive{};
    1512+                        return TxUpdate::CHANGED;
    1513+                    }
    1514+                }
    1515+                return TxUpdate::UNCHANGED;
    


    ryanofsky commented at 8:07 pm on February 15, 2024:

    In commit “wallet: track mempool conflicts” (2c867784b2a719c9e701e19e389c3adfa7d7faae)

    This has the same problems as the new transactionAddedToMempool code above. Would suggest avoiding the need for state transitions, and always updating the mempool_conflicts set recursively for BlockConflicted as well as Inactive transactions:

    0if (!wtx.mempool_conflicts.erase(conflict_txid)) return TxUpdate::UNCHANGED;
    1return TxUpdate::CHANGED;
    

    ishaanam commented at 10:52 pm on February 22, 2024:
    Done
  131. in src/wallet/wallet.cpp:1434 in 2c867784b2 outdated
    1429+
    1430+        std::pair<TxSpends::const_iterator, TxSpends::const_iterator> range;
    1431+        range = mapTxSpends.equal_range(tx_in.prevout);
    1432+
    1433+        // For each wallet transaction spending this prevout...
    1434+        for (TxSpends::const_iterator _it = range.first; _it != range.second; ++_it) {
    


    ryanofsky commented at 8:10 pm on February 15, 2024:

    In commit “wallet: track mempool conflicts” (2c867784b2a719c9e701e19e389c3adfa7d7faae)

    Just IMO, but

    0auto range =  mapTxSpends.equal_range(tx_in.prevout);
    1for (auto it = range.first; it != range.second; ++it)
    

    would make core more readable, dropping the long type declarations and extra underscore. But feel free to keep current style if that is your preference.


    ishaanam commented at 10:53 pm on February 22, 2024:
    I have implemented most of this, except for removing the extra underscore, because it was already defined above.
  132. in src/wallet/wallet.cpp:1428 in 2c867784b2 outdated
    1419@@ -1420,6 +1420,37 @@ void CWallet::transactionAddedToMempool(const CTransactionRef& tx) {
    1420     if (it != mapWallet.end()) {
    1421         RefreshMempoolStatus(it->second, chain());
    1422     }
    1423+
    1424+    const Txid& conflict_txid = tx->GetHash();
    1425+
    1426+    for (const CTxIn& tx_in : tx->vin) {
    1427+        // If the wallet contains no transactions that spend the same prevout, continue
    1428+        if (mapTxSpends.count(tx_in.prevout) == 0) continue;
    


    ryanofsky commented at 8:12 pm on February 15, 2024:

    In commit “wallet: track mempool conflicts” (2c867784b2a719c9e701e19e389c3adfa7d7faae)

    There’s no reason for this line. If the count is 0, equal_range will return an empty range and the result will be the same.


    ishaanam commented at 10:54 pm on February 22, 2024:
    Fixed here and for transactionRemovedFromMempool.
  133. in test/functional/wallet_conflicts.py:33 in c891a65067 outdated
    28@@ -28,6 +29,20 @@ def get_utxo_of_value(self, from_tx_id, search_value):
    29         return next(tx_out["vout"] for tx_out in self.nodes[0].gettransaction(from_tx_id)["details"] if tx_out["amount"] == Decimal(f"{search_value}"))
    30 
    31     def run_test(self):
    32+        """
    33+        The following tests check the behavior of the wallet when
    


    ryanofsky commented at 8:22 pm on February 15, 2024:

    In commit “test: Add tests for wallet mempool conflicts” (c891a65067f57ba87e443d0cdc4f7e6b434c436f)

    Ignore this suggestion if it requires too much work, but IMO it would be really helpful for review if as much of the new test code as possible was added in the first commit before the C++ code changes, and the tests were updated alongside the code changes in subsequent commits.

    Independently of this PR, the fact that a commit like 2c867784b2a719c9e701e19e389c3adfa7d7faae can make a number of behavior changes and not need any significant test updates shows that we are lacking a lot of test coverage for these code paths presently. So it would be very reassuring to add the test coverage before making the changes and see the changes having the expected effects on tests, instead of making the changes blindly and adding the test coverage later.


    ishaanam commented at 10:55 pm on February 22, 2024:
    I have made the first commit the commit with tests, and they are updated to reflect the changes in the following commits.
  134. ryanofsky commented at 8:49 pm on February 15, 2024: contributor

    Code review c891a65067f57ba87e443d0cdc4f7e6b434c436f.

    Thanks for the updates! A lot of nice changes and test coverage here, and I think the PR looks close to ready to ACK, though I did leave a number of suggestions below. Most of these should be very simple or simplify code, so should not create too much work.

  135. ishaanam force-pushed on Feb 22, 2024
  136. DrahtBot added the label CI failed on Feb 22, 2024
  137. DrahtBot commented at 11:52 pm on February 22, 2024: contributor

    🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the documentation.

    Possibly this is due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    Leave a comment here, if you need help tracking down a confusing failure.

    Debug: https://github.com/bitcoin/bitcoin/runs/21886408495

  138. in src/wallet/transaction.h:265 in a650caab9f outdated
    257@@ -258,6 +258,13 @@ class CWalletTx
    258     CTransactionRef tx;
    259     TxState m_state;
    260 
    261+    // Set of mempool transactions that conflict
    262+    // directly with the transaction, or that conflict
    263+    // with an ancestor transaction. This set is empty
    264+    // if the transaction is confirmed, but can be
    265+    // non-empty in other states.
    


    ryanofsky commented at 3:28 pm on February 23, 2024:

    In commit “wallet: track mempool conflicts” (a650caab9f11be7f5927f9aa556bb7d2a8ebed33)

    Would maybe tweak comment to be more specific. “This set will be empty if state is InMempool or Confirmed, but can be nonempty if state is Inactive or BlockConflicted.”


    ishaanam commented at 6:05 am on February 28, 2024:
    Done
  139. in src/wallet/receive.cpp:260 in a650caab9f outdated
    256@@ -257,7 +257,7 @@ bool CachedTxIsTrusted(const CWallet& wallet, const CWalletTx& wtx, std::set<uin
    257 {
    258     AssertLockHeld(wallet.cs_wallet);
    259     if (wtx.isConfirmed()) return true;
    260-    if (wtx.isBlockConflicted()) return false;
    261+    if (wtx.isConflicted()) return false;
    


    ryanofsky commented at 3:36 pm on February 23, 2024:

    In commit “wallet: track mempool conflicts” (a650caab9f11be7f5927f9aa556bb7d2a8ebed33)

    Can this change be reverted? This change seems like it will have no effect, because of the if (!wtx.InMempool()) return false; line below. If the transaction conflicts with the mempool, it should already not be in in the mempool and already not be trusted.

    Would suggest reverting this change and dropping “if a tx has a mempool conflict, the tx will no longer be trusted” from the commit description. Also might suggest dropping the CWalletTx::isConflicted method since it is only used one other place.


    ishaanam commented at 6:05 am on February 28, 2024:
    Done
  140. in src/wallet/wallet.cpp:1440 in a650caab9f outdated
    1435+
    1436+        // For each wallet transaction spending this prevout...
    1437+        for (auto _it = range.first; _it != range.second; ++_it) {
    1438+            // if this tx is the same as the tx which was just added to the mempool, continue
    1439+            CWalletTx& wallet_tx = mapWallet.at(_it->second);
    1440+            if (wallet_tx.tx->GetHash() == conflict_txid) continue;
    


    ryanofsky commented at 3:55 pm on February 23, 2024:

    In commit “wallet: use CWalletTx member functions to determine tx state” (feb702a615cf4d9d2d4aa825b46951ae0f389183)

    The mapWallet lookup appears to not do anything and it seems like could be simplified

    0-            CWalletTx& wallet_tx = mapWallet.at(_it->second);
    1-            if (wallet_tx.tx->GetHash() == conflict_txid) continue;
    2+            if (_it->second == conflict_txid) continue;
    

    ishaanam commented at 6:05 am on February 28, 2024:
    Done
  141. in src/wallet/wallet.cpp:1444 in a650caab9f outdated
    1440+            if (wallet_tx.tx->GetHash() == conflict_txid) continue;
    1441+
    1442+            auto try_updating_state = [&](CWalletTx& wtx) EXCLUSIVE_LOCKS_REQUIRED(cs_wallet) {
    1443+                wtx.mempool_conflicts.insert(conflict_txid);
    1444+
    1445+                return TxUpdate::CHANGED;
    


    ryanofsky commented at 4:08 pm on February 23, 2024:

    In commit “wallet: track mempool conflicts” (a650caab9f11be7f5927f9aa556bb7d2a8ebed33)

    Would be nice to add a check that conflict txid is actually inserted as assumed:

    0-                wtx.mempool_conflicts.insert(conflict_txid);
    1-
    2+                Assume(wtx.mempool_conflicts.insert(conflict_txid).second);
    

    ishaanam commented at 6:09 am on February 28, 2024:
    It seems like that is not always true, so I have changed this code to do something similar to transactionRemovedFromMempool where it only returns TxUpdate::CHANGED if the txid is actually inserted.

    ryanofsky commented at 6:58 pm on February 29, 2024:

    re: #27307 (review)

    It seems like that is not always true, so I have changed this code to do something similar to transactionRemovedFromMempool where it only returns TxUpdate::CHANGED if the txid is actually inserted.

    Actually that’s not surprising since a transaction could depend on another transaction through multiple paths. So the new code will now be a little more efficient.

  142. in src/wallet/wallet.cpp:1501 in a650caab9f outdated
    1496+
    1497+            auto try_updating_state = [&](CWalletTx& wtx) EXCLUSIVE_LOCKS_REQUIRED(cs_wallet) {
    1498+                // Remove the previously conflicting transaction from this wtx's entry
    1499+                if (wtx.mempool_conflicts.erase(conflict_txid)) return TxUpdate::CHANGED;
    1500+
    1501+                return TxUpdate::UNCHANGED;
    


    ryanofsky commented at 4:15 pm on February 23, 2024:

    In commit “wallet: track mempool conflicts” (a650caab9f11be7f5927f9aa556bb7d2a8ebed33)

    Unimportant style point, but usually it makes more sense to return early when state is not changing than when it is changing:

    0-                if (wtx.mempool_conflicts.erase(conflict_txid)) return TxUpdate::CHANGED;
    1-
    2-                return TxUpdate::UNCHANGED;
    3+                if (!wtx.mempool_conflicts.erase(conflict_txid)) return TxUpdate::UNCHANGED;
    4+                return TxUpdate::CHANGED;
    

    So if someone wants to add extra code logging or dealing with the state change, there is a place to insert it after the early return.

    Alternately this could be simplified to

    0return wtx.mempool_conflicts.erase(conflict_txid)) ? TxUpdate::CHANGED : TxUpdate::UNCHANGED;
    

    ishaanam commented at 6:06 am on February 28, 2024:
    Done
  143. in test/functional/wallet_conflicts.py:349 in a650caab9f outdated
    345@@ -346,7 +346,6 @@ def test_descendants_with_mempool_conflicts(self):
    346         tx1_child = bob.signrawtransactionwithwallet(raw_tx)['hex']
    347         tx1_child_txid = bob.sendrawtransaction(tx1_child)
    348 
    349-        # Currently neither tx1 nor tx1_child should have any conflicts
    


    ryanofsky commented at 4:18 pm on February 23, 2024:

    In commit “wallet: track mempool conflicts” (a650caab9f11be7f5927f9aa556bb7d2a8ebed33)

    Should this comment be restored? Unclear why it is being removed here


    ishaanam commented at 6:06 am on February 28, 2024:
    That was a mistake, should be fixed now.
  144. ryanofsky approved
  145. ryanofsky commented at 4:38 pm on February 23, 2024: contributor

    Code review ACK 923734ef6528e5d4e13748db7387f44d02237d4d. Seems like a very nice change that improves the wallet balance calculation and ability to spend, and also returns useful information about the mempool.

    Thanks for the updates, too. I think comparing current a650caab9f11be7f5927f9aa556bb7d2a8ebed33 to previous a6ae5b23b1497ab6f4899db49348db623700a2d8, the resulting code is simpler and the charges are more direct and easier to review.

    I left some more suggestions below, but most are not important and none are critical so feel free to ignore them.

  146. DrahtBot requested review from achow101 on Feb 23, 2024
  147. ishaanam force-pushed on Feb 28, 2024
  148. DrahtBot removed the label CI failed on Feb 28, 2024
  149. in src/wallet/wallet.cpp:761 in 9fa7f81328 outdated
    757@@ -758,7 +758,7 @@ bool CWallet::IsSpent(const COutPoint& outpoint) const
    758             // check for the inverse condition and only consider spending
    759             // transactions in known, potentially active states.
    760             const auto& wtx = mit->second;
    761-            if (wtx.isConfirmed() || wtx.InMempool() || (wtx.isInactive() && !wtx.isAbandoned()))
    762+            if (!wtx.isAbandoned() && !wtx.isMempoolConflicted() && !wtx.isBlockConflicted())
    


    ryanofsky commented at 7:14 pm on February 29, 2024:

    In commit “wallet: track mempool conflicts” (9fa7f813281d8520a5eb5adbd29fe3a0d2ab5a4f)

    Would suggest deleting the “An output is considered spent…” comment above now that the code is simpler and it no longer adds any new information. The part of the comment about checking for the “inverse condition” is also no longer true, because the code is now checking directly if wtx is conflicted or abandoned, instead of checking for the inverse condition (that it is confirmed, or in mempool, or inactive and not abandoned).

  150. in src/wallet/wallet.cpp:761 in 92b3204b69 outdated
    758+            // unless the spending transaction is conflicted or
    759+            // abandoned. The check below is written conservatively to
    760+            // check for the inverse condition and only consider spending
    761+            // transactions in known, potentially active states.
    762+            const auto& wtx = mit->second;
    763+            if (wtx.isConfirmed() || wtx.InMempool() || (wtx.isInactive() && !wtx.isAbandoned()))
    


    ryanofsky commented at 7:18 pm on February 29, 2024:

    In commit “wallet: use CWalletTx member functions to determine tx state” (92b3204b69e5ec25cbb29bd158f25d2b07e48cf6)

    Would suggest simplifying this code and deleting the comment so this is just:

    0const auto& wtx = mit->second;
    1if (!wtx.isAbandoned() && !wtx.isBlockConflicted())
    

    Using the simpler condition would make this commit easier to understand, and make the change in the next commit more obvious where the line becomes:

    0if (!wtx.isAbandoned() && !wtx.isMempoolConflicted() && !wtx.isBlockConflicted())
    
  151. ryanofsky approved
  152. ryanofsky commented at 7:28 pm on February 29, 2024: contributor

    Code review ACK b853688c92d3745d5ec2ec5119584148084e7c1c. Just more simplifications since that last review. This PR keeps getting smaller and simpler, and seems like a very obvious improvement now.

    I left another suggestion (really 2 related suggestions), but it is not very important. This looks good in its current form.

  153. in test/functional/wallet_conflicts.py:141 in d3f7764c80 outdated
    137@@ -123,5 +138,249 @@ def run_test(self):
    138         assert_equal(former_conflicted["confirmations"], 1)
    139         assert_equal(former_conflicted["blockheight"], 217)
    140 
    141+    def test_mempool_conflict(self):
    


    furszy commented at 10:38 pm on March 6, 2024:
    In d3f7764c802fe: It would be good to unload the created wallet/s at the end of each test case (call unloadwallet()). Otherwise we keep them active for the entire test lifetime.
  154. in test/functional/wallet_conflicts.py:244 in d3f7764c80 outdated
    238+
    239+        # create a conflict to previous tx (also spends unspents[1]), but don't broadcast, sends funds to alice
    240+        raw_tx = alice.createrawtransaction(inputs=[unspents[1]], outputs=[{alice.getnewaddress() : 24.9999}])
    241+        tx2_conflict = alice.signrawtransactionwithwallet(raw_tx)['hex']
    242+
    243+        bob_unspents = [{"txid" : element, "vout" : 0} for element in [tx1_txid, tx2_txid]]
    


    furszy commented at 10:42 pm on March 6, 2024:

    In d3f7764c802fe:

    Tiny note for reviewers: vout is always 0 because “alice only contain 3 utxo of 25 btc each. So, tx1 and tx2 are changeless transactions. (could be good to mention this in the code too)

  155. in src/wallet/wallet.cpp:1436 in 9fa7f81328 outdated
    1443+
    1444+                return TxUpdate::CHANGED;
    1445+            };
    1446+
    1447+            RecursiveUpdateTxState(nullptr, _it->second, try_updating_state);
    1448+        }
    


    furszy commented at 11:03 pm on March 6, 2024:

    styling tiny nit:

    In case you like it:

    0// Mark all already spent inputs, and all txs that consumes them, as conflicted
    1for (auto range = mapTxSpends.equal_range(tx_in.prevout); range.first != range.second; range.first++) {
    2    const uint256& spent_id = range.first->second;
    3    // Skip the recently added tx
    4    if (spent_id == conflict_txid) continue;
    5    RecursiveUpdateTxState(/*batch=*/nullptr, spent_id, [&conflict_txid](CWalletTx& wtx) EXCLUSIVE_LOCKS_REQUIRED(cs_wallet) {
    6    	return wtx.mempool_conflicts.insert(conflict_txid).second ? TxUpdate::CHANGED : TxUpdate::UNCHANGED;
    7    });
    8}
    

    (same for transactionRemovedFromMempool)

  156. in test/functional/wallet_conflicts.py:310 in d3f7764c80 outdated
    296+
    297+        bob.sendrawtransaction(raw_tx2)
    298+        assert_equal(bob.getbalances()["mine"]["untrusted_pending"], 0)
    299+
    300+        bob.sendrawtransaction(tx1_conflict_conflict) # kick tx1_conflict out of the mempool
    301+        bob.sendrawtransaction(raw_tx1) #re-broadcast tx1 because it is no longer conflicted
    


    furszy commented at 2:32 am on March 7, 2024:

    In d3f7764c802fe:

    nit: Could move the tx1_conflict_conflict creation here. Basically because it adds unneeded information at the beginning of the test that reviewers/readers have to carry over. The tx only exist to kick tx1_conflict out of the mempool so tx1 can be re-accepted at this point.

  157. in test/functional/wallet_conflicts.py:327 in d3f7764c80 outdated
    311+        bob.reconsiderblock(blk)
    312+        assert_equal(alice.getbestblockhash(), bob.getbestblockhash())
    313+        self.sync_mempools()
    314+        self.generate(self.nodes[2], 1)
    315+
    316+    def test_descendants_with_mempool_conflicts(self):
    


    furszy commented at 2:45 am on March 7, 2024:

    In d3f7764c802fe: Could add wallet balance checks along this test.

    Also, could test the scenario where the parent and the child tx are conflicted by different txs, then only the parent gets unconflicted -> this shouldn’t remove the “conflicting” status from the child.

  158. furszy commented at 2:48 am on March 7, 2024: member
    left few small things. Will get deeper.
  159. in src/wallet/wallet.h:522 in 92b3204b69 outdated
    517@@ -518,7 +518,7 @@ class CWallet final : public WalletStorage, public interfaces::Chain::Notificati
    518     bool IsTxInMainChain(const CWalletTx& wtx) const EXCLUSIVE_LOCKS_REQUIRED(cs_wallet)
    519     {
    520         AssertLockHeld(cs_wallet);
    521-        return GetTxDepthInMainChain(wtx) > 0;
    522+        return wtx.isConfirmed();
    523     }
    


    furszy commented at 12:50 pm on March 7, 2024:
    In 92b3204b69e: Could just remove this method and replace the two places that uses it for a simpler wtx.isConfirmed().
  160. in src/wallet/wallet.cpp:1371 in 9fa7f81328 outdated
    1367     WalletBatch batch(GetDatabase(), false);
    1368+    RecursiveUpdateTxState(&batch, tx_hash, try_updating_state);
    1369+}
    1370+
    1371+void CWallet::RecursiveUpdateTxState(WalletBatch* batch, const uint256& tx_hash, const TryUpdatingStateFn& try_updating_state) EXCLUSIVE_LOCKS_REQUIRED(cs_wallet) {
    1372+    // Do not flush the wallet here for performance reasons
    


    furszy commented at 12:56 pm on March 7, 2024:

    In 9fa7f81328:

    This comment relates to the second argument of the WalletBatch constructor. Need to move it to the caller side, it does not mean anything inside RecursiveUpdateTxState.


    ryanofsky commented at 2:44 pm on March 11, 2024:

    re: #27307 (review)

    This comment relates to the second argument of the WalletBatch constructor. Need to move it to the caller side, it does not mean anything inside RecursiveUpdateTxState.

    Agree the comment makes sense where it was previously above WalletBatch, and shouldn’t be moved in this commit

  161. test: Add tests for wallet mempool conflicts 180973a941
  162. scripted-diff: wallet: s/TxStateConflicted/TxStateBlockConflicted
    -BEGIN VERIFY SCRIPT-
    sed -i 's/TxStateConflicted/TxStateBlockConflicted/g' src/wallet/wallet.cpp src/wallet/interfaces.cpp src/wallet/transaction.h src/wallet/transaction.cpp
    sed -i 's/isConflicted/isBlockConflicted/g' src/wallet/transaction.h src/wallet/wallet.cpp
    -END VERIFY SCRIPT-
    ffe5ff1fb6
  163. ishaanam force-pushed on Mar 14, 2024
  164. in src/wallet/interfaces.cpp:104 in 8fd1c8894b outdated
    100@@ -101,7 +101,7 @@ WalletTxStatus MakeWalletTxStatus(const CWallet& wallet, const CWalletTx& wtx)
    101     result.is_trusted = CachedTxIsTrusted(wallet, wtx);
    102     result.is_abandoned = wtx.isAbandoned();
    103     result.is_coinbase = wtx.IsCoinBase();
    104-    result.is_in_main_chain = wallet.IsTxInMainChain(wtx);
    105+    result.is_in_main_chain = wtx.isConfirmed();
    


    ryanofsky commented at 6:40 pm on March 15, 2024:

    In commit “wallet: use CWalletTx member functions to determine tx state” (8fd1c8894b24d73bd2769363503c7464101af04f)

    Would be good to make clear that this commit is intended to be a refactoring, and shouldn’t change behavior. (Could say this in commit message or put “wallet refactor:” in the commit subject or do both).


    ishaanam commented at 9:22 pm on March 20, 2024:
    Done
  165. in src/wallet/wallet.cpp:1429 in 3fa20dfb6a outdated
    1420@@ -1418,6 +1421,20 @@ void CWallet::transactionAddedToMempool(const CTransactionRef& tx) {
    1421     if (it != mapWallet.end()) {
    1422         RefreshMempoolStatus(it->second, chain());
    1423     }
    1424+
    1425+    const Txid& conflict_txid = tx->GetHash();
    1426+
    1427+    for (const CTxIn& tx_in : tx->vin) {
    1428+        // Mark all already spent inputs, and all txs that consumes them, as conflicted
    1429+        for (auto range = mapTxSpends.equal_range(tx_in.prevout); range.first != range.second; range.first++) {
    


    ryanofsky commented at 6:43 pm on March 15, 2024:

    In commit “wallet: track mempool conflicts” (3fa20dfb6a8a92a9b9f00248d546a21dbf6821d6)

    Would be helpful IMO to keep previous comment explaining the for loop “// For each wallet transaction spending this prevout..”


    ishaanam commented at 9:25 pm on March 20, 2024:
    Done
  166. in src/wallet/wallet.cpp:1500 in 3fa20dfb6a outdated
    1495+
    1496+        for (auto range = mapTxSpends.equal_range(tx_in.prevout); range.first != range.second; range.first++) {
    1497+            const uint256& spent_id = range.first->second;
    1498+
    1499+            RecursiveUpdateTxState(/*batch=*/nullptr, spent_id, [&conflict_txid](CWalletTx& wtx) EXCLUSIVE_LOCKS_REQUIRED(cs_wallet) {
    1500+                return !wtx.mempool_conflicts.erase(conflict_txid) ? TxUpdate::UNCHANGED : TxUpdate::CHANGED;
    


    ryanofsky commented at 6:48 pm on March 15, 2024:

    In commit “wallet: track mempool conflicts” (3fa20dfb6a8a92a9b9f00248d546a21dbf6821d6)

    Negated logic is a little odd. Would suggest simplifying to:

    0return wtx.mempool_conflicts.erase(conflict_txid) ? TxUpdate::CHANGED : TxUpdate::UNCHANGED;
    

    ishaanam commented at 9:25 pm on March 20, 2024:
    Done
  167. in src/wallet/wallet.cpp:1495 in 3fa20dfb6a outdated
    1490+            };
    1491+
    1492+            RecursiveUpdateTxState(nullptr, _it->second, try_updating_state);
    1493+        }
    1494+        */
    1495+
    


    ryanofsky commented at 6:52 pm on March 15, 2024:

    In commit “wallet: track mempool conflicts” (3fa20dfb6a8a92a9b9f00248d546a21dbf6821d6)

    Probably keeping this commented out code is unintentional, but I would suggest replacing with a real comment. Maybe “Iterate over all wallet transactions spending txin.prev and recursively mark them as no longer conflicting with conflict_txid”


    ishaanam commented at 9:25 pm on March 20, 2024:
    Done
  168. in src/wallet/wallet.cpp:1425 in 3fa20dfb6a outdated
    1420@@ -1418,6 +1421,20 @@ void CWallet::transactionAddedToMempool(const CTransactionRef& tx) {
    1421     if (it != mapWallet.end()) {
    1422         RefreshMempoolStatus(it->second, chain());
    1423     }
    1424+
    1425+    const Txid& conflict_txid = tx->GetHash();
    


    ryanofsky commented at 6:56 pm on March 15, 2024:

    In commit “wallet: track mempool conflicts” (3fa20dfb6a8a92a9b9f00248d546a21dbf6821d6)

    IMO conflict_txid is not the best name for this variable because it is inconsistent with other variable names in this function and we don’t know if this transaction actually conflicts with anything. Would suggest renaming conflict_txid to txid or tx_hash to be consistent with other variables referring to the same transaction (tx and tx_in).

    Would also do same rename in the transactionRemovedFromMempool function below.


    ishaanam commented at 9:25 pm on March 20, 2024:
    Done
  169. ryanofsky approved
  170. ryanofsky commented at 7:02 pm on March 15, 2024: contributor
    Code review ACK dbcc4a75e4a1108f4a04c22cfae363848994c24d. Changes since last review were a few test improvements and some more simplifications. This looks very good. Left suggestions below, all minor, none important. This PR seems like a clear improvement and should be merged if it gets more ACKs.
  171. wallet refactor: use CWalletTx member functions to determine tx state d64922b590
  172. wallet: track mempool conflicts
    Behavior changes are:
    - if a tx has a mempool conflict, the wallet will not attempt to
      rebroadcast it
    - if a txo is spent by a mempool-conflicted tx, that txo is no
      longer considered spent
    54e07ee22f
  173. wallet, rpc: show mempool conflicts in `gettransaction` result 5952292133
  174. ishaanam force-pushed on Mar 20, 2024
  175. achow101 commented at 7:36 pm on March 26, 2024: member
    ACK 5952292133d6cc889f51ae771f2e0557311e1efe
  176. DrahtBot requested review from ryanofsky on Mar 26, 2024
  177. ryanofsky approved
  178. ryanofsky commented at 9:04 pm on March 26, 2024: contributor
    Code review ACK 5952292133d6cc889f51ae771f2e0557311e1efe. Just small suggested changes since last review
  179. in test/functional/wallet_conflicts.py:40 in 180973a941 outdated
    35+        using raw transaction RPCs that double-spend UTXOs and have more
    36+        fees, replacing the original transaction.
    37+        """
    38+
    39+        self.test_block_conflicts()
    40+        self.generatetoaddress(self.nodes[0], COINBASE_MATURITY + 7, self.nodes[2].getnewaddress())
    


    furszy commented at 12:56 pm on March 27, 2024:
    nit: Would be good to mention why this generatetoaddress is needed. I removed it locally and the test still passes.
  180. in src/wallet/rpc/transactions.cpp:424 in 5952292133
    420@@ -417,6 +421,10 @@ static std::vector<RPCResult> TransactionDescriptionString()
    421            }},
    422            {RPCResult::Type::STR_HEX, "replaced_by_txid", /*optional=*/true, "Only if 'category' is 'send'. The txid if this tx was replaced."},
    423            {RPCResult::Type::STR_HEX, "replaces_txid", /*optional=*/true, "Only if 'category' is 'send'. The txid if this tx replaces another."},
    424+           {RPCResult::Type::ARR, "mempoolconflicts", "Transactions that directly conflict with either this transaction or an ancestor transaction",
    


    furszy commented at 1:06 pm on March 27, 2024:
    Not for this PR just so you don’t have to re-touch it but it would be good to describe the difference between walletconflicts and mempoolconflicts inside the help text. walletconflicts description is quite vague.
  181. furszy commented at 1:07 pm on March 27, 2024: member
    ACK 59522921
  182. ryanofsky merged this on Mar 27, 2024
  183. ryanofsky closed this on Mar 27, 2024

  184. ryanofsky commented at 5:00 pm on March 27, 2024: contributor

    This is merged but it would be good to followup with documentation / test improvements from #27307#pullrequestreview-1963245186.

    Might also be useful to add release notes saying the wallet has a new heuristic to detect when wallet transactions conflict with the mempool, that conflicting mempool transactions are shown in a new gettransaction "mempoolconflicts" field, that this allows inputs to the conflicted transactions to be respent without manually abandoning the transactions, and can cause wallet balances to appear higher.


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-07-05 22:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me