test: wallet rescan with reorged parent + IsFromMe child in mempool #29179

pull glozow wants to merge 2 commits into bitcoin:master from glozow:2024-01-test-reorg-rescan changing 3 files +131 −9
  1. glozow commented at 4:26 pm on January 4, 2024: member

    Originally motivated by #29019, which reverts back to having requestMempoolTransactions emit transactionAddedToMempool in mapTx default order instead of GetSortedDepthAndScore order.

    It’s important that these notifications happen in topological order, otherwise the wallet rescan may miss transactions that belong to it. Notably, checking whether a transaction IsFromMe requires knowing its inputs, which may be from a mempool parent.

    When using mapTx order, a parent may come later than its child if it was added from a block disconnected in a reorg.

    This PR adds a test for this case.

  2. glozow added the label Tests on Jan 4, 2024
  3. DrahtBot commented at 4:26 pm on January 4, 2024: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage

    For detailed information about the code coverage, see the test coverage report.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK furszy, achow101
    Stale ACK Eunovo, stickies-v

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #28710 (Remove the legacy wallet and BDB dependency by achow101)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  4. achow101 commented at 11:02 pm on January 4, 2024: member
    Huh, looks like this behavior has been broken since it was introduced in #15652
  5. achow101 commented at 11:16 pm on January 4, 2024: member
    ACK 1a52ca7619eeb1baafa5a32b364381126862b29d
  6. in test/functional/wallet_import_rescan.py:290 in 1a52ca7619 outdated
    286+        # For each variant, create an unconfirmed child transaction from initial_txid, sending half of the balance to
    287+        # itself and the other half to unspendable/fees. This transaction is only recognizable as
    288+        # belonging to this wallet when the inputs are known. That means the rescan must process the
    289+        # parent before the child.
    290+        for variant in mempool_variants:
    291+            inputs_to_spend = [txin for txin in self.nodes[1].listunspent() if txin["txid"] == variant.initial_txid]
    


    stickies-v commented at 12:18 pm on January 5, 2024:

    nit: could be a bit more efficient by only calling and iterating over listunspent once (does not test for txid collisions, but I don’t think we need that?)

     0diff --git a/test/functional/wallet_import_rescan.py b/test/functional/wallet_import_rescan.py
     1index 1abc9c0c79..928455c355 100755
     2--- a/test/functional/wallet_import_rescan.py
     3+++ b/test/functional/wallet_import_rescan.py
     4@@ -286,13 +286,12 @@ class ImportRescanTest(BitcoinTestFramework):
     5         # itself and the other half to unspendable/fees. This transaction is only recognizable as
     6         # belonging to this wallet when the inputs are known. That means the rescan must process the
     7         # parent before the child.
     8+        unspent_txid_map = {txin["txid"] : txin for txin in self.nodes[1].listunspent()}
     9         for variant in mempool_variants:
    10-            inputs_to_spend = [txin for txin in self.nodes[1].listunspent() if txin["txid"] == variant.initial_txid]
    11-            assert_equal(len(inputs_to_spend), 1)
    12             # Send full amount, subtracting fee from outputs, to ensure no change is created.
    13             child = self.nodes[1].send(
    14                 add_to_wallet=False,
    15-                inputs=inputs_to_spend,
    16+                inputs=[unspent_txid_map[variant.initial_txid]],
    17                 outputs=[{ADDRESS_BCRT1_UNSPENDABLE : variant.initial_amount}],
    18                 subtract_fee_from_outputs=[0]
    19             )
    

    glozow commented at 6:12 pm on January 5, 2024:
    taken
  7. in test/functional/wallet_import_rescan.py:282 in 1a52ca7619 outdated
    278             variant.timestamp = timestamp
    279 
    280+        # Mine a block so these parents are confirmed
    281+        assert_equal(len(self.nodes[0].getrawmempool()), len(mempool_variants))
    282+        self.sync_mempools()
    283+        block_to_disconnect = self.generate(self.nodes[0], 1)
    


    stickies-v commented at 1:55 pm on January 5, 2024:

    nit: makes more sense to unpack here imo

    0        block_to_disconnect = self.generate(self.nodes[0], 1)[0]
    

    glozow commented at 6:12 pm on January 5, 2024:
    I agree, taken
  8. in test/functional/wallet_import_rescan.py:323 in 1a52ca7619 outdated
    317@@ -283,9 +318,11 @@ def run_test(self):
    318             variant.node = self.nodes[2 + IMPORT_NODES.index(ImportNode(variant.prune, expect_rescan))]
    319             variant.do_import(variant.timestamp)
    320             if expect_rescan:
    321-                variant.expected_balance = variant.initial_amount
    322+                # Ensure both transactions were rescanned
    323+                assert_equal(variant.node.gettransaction(variant.initial_txid)['confirmations'], 0)
    324+                assert_equal(variant.node.gettransaction(variant.child_txid)['confirmations'], 0)
    


    stickies-v commented at 4:06 pm on January 5, 2024:

    nit: I don’t think we’re actually checking the confirmations values, so for future maintenance, would this be more straightforward?

    0                variant.node.gettransaction(variant.initial_txid)  # raises if tx wasn't rescanned
    1                variant.node.gettransaction(variant.child_txid)    # raises if tx wasn't rescanned
    

    glozow commented at 6:13 pm on January 5, 2024:
    Added a similar comment.
  9. in test/functional/wallet_import_rescan.py:85 in 1a52ca7619 outdated
    81@@ -79,7 +82,7 @@ def do_import(self, timestamp):
    82             )
    83             assert_equal(response, [{"success": True}])
    84 
    85-    def check(self, txid=None, amount=None, confirmation_height=None):
    86+    def check(self, txid=None, amount=None, confirmation_height=None, amount_is_balance=True):
    


    stickies-v commented at 5:19 pm on January 5, 2024:

    It doesn’t seem like we’re actually testing the expected balance at any point, rather the amount received, but I may be missing nuance. Would this diff make sense? I don’t like adding an extra parameter to an already long function for something so specific, if we can avoid it.

     0diff --git a/test/functional/wallet_import_rescan.py b/test/functional/wallet_import_rescan.py
     1index 1abc9c0c79..fa9f43f66c 100755
     2--- a/test/functional/wallet_import_rescan.py
     3+++ b/test/functional/wallet_import_rescan.py
     4@@ -82,7 +82,7 @@ class Variant(collections.namedtuple("Variant", "call data address_type rescan p
     5             )
     6             assert_equal(response, [{"success": True}])
     7 
     8-    def check(self, txid=None, amount=None, confirmation_height=None, amount_is_balance=True):
     9+    def check(self, txid=None, amount=None, confirmation_height=None):
    10         """Verify that listtransactions/listreceivedbyaddress return expected values."""
    11 
    12         txs = self.node.listtransactions(label=self.label, count=10000, include_watchonly=True)
    13@@ -112,8 +112,7 @@ class Variant(collections.namedtuple("Variant", "call data address_type rescan p
    14 
    15             address, = [ad for ad in addresses if txid in ad["txids"]]
    16             assert_equal(address["address"], self.address["address"])
    17-            if amount_is_balance:
    18-                assert_equal(address["amount"], self.expected_balance)
    19+            assert_equal(address["amount"], self.amount_received)
    20             assert_equal(address["confirmations"], confirmations)
    21             # Verify the transaction is correctly marked watchonly depending on
    22             # whether the transaction pays to an imported public key or
    23@@ -227,11 +226,11 @@ class ImportRescanTest(BitcoinTestFramework):
    24             variant.node = self.nodes[2 + IMPORT_NODES.index(ImportNode(variant.prune, expect_rescan))]
    25             variant.do_import(variant.timestamp)
    26             if expect_rescan:
    27-                variant.expected_balance = variant.initial_amount
    28+                variant.amount_received = variant.initial_amount
    29                 variant.expected_txs = 1
    30                 variant.check(variant.initial_txid, variant.initial_amount, variant.confirmation_height)
    31             else:
    32-                variant.expected_balance = 0
    33+                variant.amount_received = 0
    34                 variant.expected_txs = 0
    35                 variant.check()
    36 
    37@@ -251,7 +250,7 @@ class ImportRescanTest(BitcoinTestFramework):
    38         # Check the latest results from getbalance and listtransactions.
    39         for variant in IMPORT_VARIANTS:
    40             self.log.info('Run check for variant {}'.format(variant))
    41-            variant.expected_balance += variant.sent_amount
    42+            variant.amount_received += variant.sent_amount
    43             variant.expected_txs += 1
    44             variant.check(variant.sent_txid, variant.sent_amount, variant.confirmation_height)
    45 
    46@@ -297,7 +296,6 @@ class ImportRescanTest(BitcoinTestFramework):
    47                 subtract_fee_from_outputs=[0]
    48             )
    49             variant.child_txid = child["txid"]
    50-            variant.expected_balance = 0
    51             self.nodes[0].sendrawtransaction(child["hex"])
    52 
    53         # Mempools should contain the child transactions for each variant.
    54@@ -321,10 +319,11 @@ class ImportRescanTest(BitcoinTestFramework):
    55                 # Ensure both transactions were rescanned
    56                 assert_equal(variant.node.gettransaction(variant.initial_txid)['confirmations'], 0)
    57                 assert_equal(variant.node.gettransaction(variant.child_txid)['confirmations'], 0)
    58+                variant.amount_received = variant.initial_amount
    59                 variant.expected_txs = 1
    60-                variant.check(variant.initial_txid, variant.initial_amount, 0, False)
    61+                variant.check(variant.initial_txid, variant.initial_amount, 0)
    62             else:
    63-                variant.expected_balance = 0
    64+                variant.amount_received = 0
    65                 variant.expected_txs = 0
    66                 variant.check()
    67 
    

    glozow commented at 6:11 pm on January 5, 2024:
    Thanks, taken.
  10. stickies-v commented at 5:25 pm on January 5, 2024: contributor

    Approach ACK, nice one finding this test case! 1a52ca7619eeb1baafa5a32b364381126862b29d looks good to me but not super familiar with this part of the code. Left some suggestions about maintainability and can be ignored.

    I verified that the tests fail on #29019 as well as on 453b4813ebc74859864803e9972b58e4be76a4d6~1, as is necessary.

  11. glozow force-pushed on Jan 5, 2024
  12. furszy commented at 10:18 pm on January 5, 2024: member
    Pre-review note: have you considered that wallet_import_rescan.py is a legacy wallet only test? If it’s placed here due to a lack of alternatives, I think this could be moved to a separate wallet_mempool.py file, where we could continue adding more cases related to the wallet-mempool interaction (or.. we could upgrade this file to run on a descriptors wallet).
  13. glozow commented at 8:46 am on January 8, 2024: member

    Pre-review note: have you considered that wallet_import_rescan.py is a legacy wallet only test?

    Ah good point. ~I added a commit to run this test with descriptors as I couldn’t find a reason why we don’t do this. Lmk if this seems sufficient / ok to do?~ EDIT: oh descriptors can’t use dumpprivkey, nvm

  14. glozow force-pushed on Jan 8, 2024
  15. DrahtBot added the label CI failed on Jan 8, 2024
  16. DrahtBot removed the label CI failed on Jan 8, 2024
  17. glozow commented at 9:04 am on January 9, 2024: member
    I’m working on adding a descriptors version of this test so we can have both.
  18. glozow force-pushed on Jan 9, 2024
  19. glozow commented at 10:39 am on January 9, 2024: member
    I’ve added another commit which has pretty much the same test but using descriptor wallets i.e. a rescan through importdescriptors. Just like the other test, you can reproduce the bug by cherry-picking from #29019. You should get JSONRPCEXCEPTION: Invalid or non-wallet transaction id (-5) for the child tx.
  20. glozow force-pushed on Jan 9, 2024
  21. DrahtBot added the label CI failed on Jan 9, 2024
  22. glozow force-pushed on Jan 9, 2024
  23. in test/functional/wallet_rescan_unconfirmed.py:31 in fa2b8d0e30 outdated
    26+    def skip_test_if_missing_module(self):
    27+        self.skip_if_no_wallet()
    28+        self.skip_if_no_sqlite()
    29+
    30+    def run_test(self):
    31+        # Node with which we will use to send funds to the wallet and mine blocks
    


    stickies-v commented at 11:56 am on January 9, 2024:

    grammar nit

    0        # Node which we will use to send funds to the wallet and mine blocks
    

    glozow commented at 3:48 pm on January 9, 2024:
    taken
  24. in test/functional/wallet_rescan_unconfirmed.py:41 in fa2b8d0e30 outdated
    36+        import_wallet_node = self.nodes[2]
    37+
    38+
    39+        self.log.info("Create wallets and mine initial chain")
    40+        tester_wallet = MiniWallet(tester_node)
    41+        self.generate(tester_wallet, 110)
    


    stickies-v commented at 12:00 pm on January 9, 2024:
    Is there a reason it’s 110 and not COINBASE_MATURITY?

    glozow commented at 3:48 pm on January 9, 2024:
    changed
  25. in test/functional/wallet_rescan_unconfirmed.py:80 in fa2b8d0e30 outdated
    89+        self.log.info("Check that the importing node has properly rescanned mempool transactions")
    90+        # Check that parent address is correctly determined as ismine
    91+        test_address(w1, parent_address, solvable=True, ismine=True)
    92+        # This would raise a JSONRPCError if the transactions were not identified as belonging to the wallet.
    93+        assert_equal(w1.gettransaction(tx_parent_to_reorg["txid"])["confirmations"], 0)
    94+        assert_equal(w1.gettransaction(tx_child_unconfirmed_sweep["txid"])["confirmations"], 0)
    


    stickies-v commented at 12:06 pm on January 9, 2024:
    nit: I still think it’s unnecessarily confusing for (current and future) reviewers to figure out why confirmations needs to equal 0 when afaiu this is completely unrelated to the test. I would either remove the value comparison or specifically state in the docs that we don’t care about the confirmations value, just that the response is a JSON object with a field we recognize?

    glozow commented at 2:30 pm on January 9, 2024:
    We are testing that they are marked as in-mempool wallet transactions. If the reorg didn’t happen, the value would be 1.

    stickies-v commented at 9:51 pm on January 11, 2024:
    Makes sense, thank you! I failed to appreciate that we’re calling the wallet gettransaction RPC, and not the not getrawtransaction RPC.

    glozow commented at 9:57 am on January 12, 2024:
    Ah yeah, it’s a wallet-specific RPC. I’ve added a check that the “confirmations” value is 1 on wallet 0 a few lines up, prior to the reorg. Hopefully that also helps make it clear why we’re testing this way.
  26. stickies-v commented at 12:09 pm on January 9, 2024: contributor
    Hooray for adding descriptor wallet support. Is there a reason both legacy and descriptor wallet tests aren’t in the new wallet_rescan_unconfirmed test, though? That seems like a more logical grouping with less duplication? If there are technical objections, I think it would be good to add some extra documentation to both tests stating that the other wallet type is tested in file xxx.py, making it easier to clean up these tests e.g. when we drop legacy wallets or make other significant changes?
  27. DrahtBot removed the label CI failed on Jan 9, 2024
  28. glozow commented at 2:30 pm on January 9, 2024: member

    Is there a reason both legacy and descriptor wallet tests aren’t in the new wallet_rescan_unconfirmed test, though? That seems like a more logical grouping with less duplication?

    They use different RPCs. I think it’s more readable for them to be separate (having most the test as if is_sqlite_compiled() elif is_bdb_compiled() is pretty annoying). I think it’s much easier to delete legacy without touching the descriptor one if they’re in different files.

  29. glozow force-pushed on Jan 9, 2024
  30. glozow requested review from furszy on Jan 10, 2024
  31. glozow requested review from stickies-v on Jan 10, 2024
  32. glozow requested review from achow101 on Jan 10, 2024
  33. Eunovo commented at 10:20 pm on January 10, 2024: none

    Tested ACK d9df438

    Ran the tests locally and got test_framework.authproxy.JSONRPCException: Invalid or non-wallet transaction id (-5) for both wallet_import_rescan.py and wallet_rescan_unconfirmed.py when insertion order is used in CTxMemPool::entryAll().

  34. in test/functional/wallet_rescan_unconfirmed.py:82 in d9df438c6e outdated
    77+            assert tx_parent_to_reorg["txid"] in node.getrawmempool()
    78+
    79+        self.log.info("Import descriptor wallet on another node")
    80+        descriptors_to_import = []
    81+        for item in w0.listdescriptors()["descriptors"]:
    82+            # descriptors_to_import.append(item)
    


    stickies-v commented at 9:38 pm on January 11, 2024:
    probably should be removed?

    glozow commented at 9:55 am on January 12, 2024:
    Oops! Removed.
  35. in test/functional/wallet_import_rescan.py:287 in d9df438c6e outdated
    283+        assert_equal(len(self.nodes[0].getrawmempool()), 0)
    284+
    285+        # For each variant, create an unconfirmed child transaction from initial_txid, sending all
    286+        # the funds to an unspendable address. Importantly, no change output is created so the
    287+        # transaction can't be recognized using its outputs. The wallet rescan needs to know the
    288+        # inputs of the transaction to detect it, so the parent must be processed before the child.
    


    stickies-v commented at 9:47 pm on January 11, 2024:

    What do you think about adding one line to these docs in each file to reference the other file? Might be useful to e.g. keep tests in sync if problems are found, or when people wonder why this is (at first sight) only tested for legacy or descriptor wallets.

    0        # inputs of the transaction to detect it, so the parent must be processed before the child.
    1        # This behaviour is tested for descriptor wallets in wallet_rescan_unconfirmed.py.
    

    glozow commented at 9:56 am on January 12, 2024:
    Added this comment in the legacy test as I agree it’ll be convenient to know what coverage we are/aren’t losing when deleting a legacy test.
  36. stickies-v approved
  37. stickies-v commented at 9:52 pm on January 11, 2024: contributor

    ACK d9df438c6e581dae0c818a4c2f5fe95627ae26bc

    I think it’s much easier to delete legacy without touching the descriptor one if they’re in different files.

    Thanks, you’re right. There’s less overlap than I thought there would be.

  38. [test] rescan legacy wallet with reorged parent + IsFromMe child in mempool
    Test that wallet rescans process transactions topologically, even if a
    parent's entry into the mempool is later than that of its child.
    This behavior is important because IsFromMe requires the ability to look
    up a transaction's inputs.
    c3d02be536
  39. glozow force-pushed on Jan 12, 2024
  40. stickies-v approved
  41. stickies-v commented at 10:03 am on January 12, 2024: contributor
    re-ACK c39e82926a3740642f8f1bb72c6b10b67a6dc724
  42. glozow commented at 10:35 am on January 12, 2024: member
    CI failure is unrelated, #29234
  43. in test/functional/wallet_rescan_unconfirmed.py:27 in c39e82926a outdated
    24+        # Immediate tx relay
    25+        self.extra_args = [['-whitelist=noban@127.0.0.1']] * self.num_nodes
    26+
    27+    def skip_test_if_missing_module(self):
    28+        self.skip_if_no_wallet()
    29+        self.skip_if_no_sqlite()
    


    furszy commented at 12:08 pm on January 12, 2024:

    in c39e82926a:

    tiny nit: looks redundant to call add_wallet_options(legacy=False) and then call skip_if_no_sqlite(). If the wallet is enabled and sqlite is not supported, then the only option is a legacy wallet, which is blocked by the add_wallet_options call.


    glozow commented at 2:17 pm on January 12, 2024:
    Seems pretty common to specify both. They are technically different; one says no legacy and the other says need descriptors. I know that’s effectively the same thing and we don’t have 3 types, but I think it’s harmless to keep this.
  44. in test/functional/wallet_rescan_unconfirmed.py:44 in c39e82926a outdated
    39+
    40+        self.log.info("Create wallets and mine initial chain")
    41+        tester_wallet = MiniWallet(tester_node)
    42+        self.generate(tester_wallet, COINBASE_MATURITY)
    43+
    44+        orig_wallet_node.createwallet(wallet_name='w0', disable_private_keys=False, descriptors=True)
    


    furszy commented at 12:15 pm on January 12, 2024:

    in https://github.com/bitcoin/bitcoin/commit/c39e82926a3740642f8f1bb72c6b10b67a6dc724:

    nit: disable_private_keys is false by default. And, because this is a descriptors-only test (per self.add_wallet_options), the descriptors flag is also redundant.


    stickies-v commented at 1:30 pm on January 12, 2024:
    I initially left the same comment about disable_private_keys but think it is helpful to contrast the difference with further down the test where we set disable_private_keys=True. Likewise, I think keeping descriptors=True is helpful self-documentation given the contrast with the test in wallet_import_rescan.py. No strong opinion on either, though.

    glozow commented at 2:19 pm on January 12, 2024:
    dropped the extra args (though I tend to prefer being a bit more verbose and self-documenting)
  45. in test/functional/wallet_rescan_unconfirmed.py:71 in c39e82926a outdated
    66+        self.log.info("Create a child tx and wait for it to propagate to all mempools")
    67+        # The only UTXO available to spend is tx_parent_to_reorg.
    68+        assert_equal(len(w0_utxos), 1)
    69+        assert_equal(w0_utxos[0]["txid"], tx_parent_to_reorg["txid"])
    70+        tx_child_unconfirmed_sweep = w0.sendall([ADDRESS_BCRT1_UNSPENDABLE])
    71+        self.sync_all()
    


    furszy commented at 12:21 pm on January 12, 2024:
    tiny nit: this could be sync_mempools instead.

    glozow commented at 2:17 pm on January 12, 2024:
    done
  46. in test/functional/wallet_rescan_unconfirmed.py:86 in c39e82926a outdated
    81+        descriptors_to_import = []
    82+        for item in w0.listdescriptors()["descriptors"]:
    83+            # use timestamp 0 to tell wallet to rescan entire chain
    84+            if not item["internal"]:
    85+                descriptors_to_import.append({"desc": item["desc"], "timestamp": 0, "label": "w0 import"})
    86+        import_wallet_node.createwallet(wallet_name="w1", disable_private_keys=True, blank=True, descriptors=True)
    


    furszy commented at 12:26 pm on January 12, 2024:

    In https://github.com/bitcoin/bitcoin/commit/c39e82926a3740642f8f1bb72c6b10b67a6dc724:

    same as above, the descriptors flag isn’t needed. Also, if you set disable_private_keys=True, there is no need to set blank=True.


    glozow commented at 2:18 pm on January 12, 2024:
    dropped the args
  47. in test/functional/wallet_rescan_unconfirmed.py:85 in c39e82926a outdated
    80+        self.log.info("Import descriptor wallet on another node")
    81+        descriptors_to_import = []
    82+        for item in w0.listdescriptors()["descriptors"]:
    83+            # use timestamp 0 to tell wallet to rescan entire chain
    84+            if not item["internal"]:
    85+                descriptors_to_import.append({"desc": item["desc"], "timestamp": 0, "label": "w0 import"})
    


    furszy commented at 12:34 pm on January 12, 2024:

    In c39e82926a3:

    Can simplify this by only importing the used address descriptor.

    0descriptors_to_import = [{"desc": w0.getaddressinfo(parent_address)['parent_desc'], "timestamp": 0, "label": "w0 import"}]
    

    glozow commented at 2:18 pm on January 12, 2024:
    nice, taken
  48. in test/functional/wallet_rescan_unconfirmed.py:42 in c39e82926a outdated
    37+        import_wallet_node = self.nodes[2]
    38+
    39+
    40+        self.log.info("Create wallets and mine initial chain")
    41+        tester_wallet = MiniWallet(tester_node)
    42+        self.generate(tester_wallet, COINBASE_MATURITY)
    


    furszy commented at 1:10 pm on January 12, 2024:

    In https://github.com/bitcoin/bitcoin/commit/c39e82926a3740642f8f1bb72c6b10b67a6dc724:

    Unless you specify a clean chain (which this test does not), the test will have an existing matured one. Which means, you could modify this for

    0        tester_wallet = MiniWallet(tester_node)
    1        self.generate(tester_wallet, 1)
    

    glozow commented at 2:21 pm on January 12, 2024:
    Keeping as is because I prefer the peace of mind of knowing tester_wallet has a mature coinbase.

    furszy commented at 2:35 pm on January 12, 2024:

    Keeping as is because I prefer the peace of mind of knowing tester_wallet has a mature coinbase.

    You shouldn’t be concerned about that. See code. If the wallet wouldn’t be having mature coins, the test would be always failing due a lack of funds.

    or well, np. Its fine.


    glozow commented at 2:47 pm on January 12, 2024:
    Deleted this line altogether, there’s no reason to generate 1 either.
  49. furszy commented at 1:15 pm on January 12, 2024: member

    Code reviewed, looking good. Left few comments, mostly testing time improvements. Nothing blocking.

    As the scenario being covered does not require verifying p2p mempool relay, you could speed up the new wallet_rescan_unconfirmed.py test by ~2.5x (tested locally, ~3 secs versus ~8 secs) by using a single node instead of creating three of them. See commit f8273512 (feel free to pull it).

  50. glozow force-pushed on Jan 12, 2024
  51. stickies-v approved
  52. stickies-v commented at 2:39 pm on January 12, 2024: contributor
    re-ACK 9db0d42509e0e223b45238fd8087fc98ef32c091
  53. glozow force-pushed on Jan 12, 2024
  54. DrahtBot added the label CI failed on Jan 12, 2024
  55. [test] import descriptor wallet with reorged parent + IsFromMe child in mempool
    Test that wallet rescans process transactions topologically, even if a
    parent's entry into the mempool is later than that of its child.
    This behavior is important because IsFromMe requires the ability to look
    up a transaction's inputs.
    
    Co-authored-by: furszy <matiasfurszyfer@protonmail.com>
    df30247705
  56. glozow force-pushed on Jan 12, 2024
  57. glozow commented at 2:53 pm on January 12, 2024: member

    See commit https://github.com/bitcoin/bitcoin/commit/f8273512a50d9ed03bc48803634fb8780b9023c7 (feel free to pull it).

    Sorry I hadn’t seen this. Thanks, taken with a couple of edits.

  58. DrahtBot removed the label CI failed on Jan 12, 2024
  59. in test/functional/wallet_rescan_unconfirmed.py:56 in df30247705
    51+        # the funds to an unspendable address. Importantly, no change output is created so the
    52+        # transaction can't be recognized using its outputs. The wallet rescan needs to know the
    53+        # inputs of the transaction to detect it, so the parent must be processed before the child.
    54+        w0_utxos = w0.listunspent()
    55+
    56+        self.log.info("Create a child tx and wait for it to propagate to all mempools")
    


    furszy commented at 7:03 pm on January 12, 2024:

    nit: leftover

    0        self.log.info("Create a child tx and wait until all wallets are notified")
    
  60. in test/functional/wallet_rescan_unconfirmed.py:68 in df30247705
    63+
    64+        self.log.info("Mock a reorg, causing parent to re-enter mempools after its child")
    65+        node.invalidateblock(block_to_reorg)
    66+        assert tx_parent_to_reorg["txid"] in node.getrawmempool()
    67+
    68+        self.log.info("Import descriptor wallet on another node")
    


    furszy commented at 7:04 pm on January 12, 2024:
    nit: leftover; should be “import descriptor on another wallet”
  61. furszy commented at 7:07 pm on January 12, 2024: member
    Code review ACK df30247705, nits can be disregarded.
  62. DrahtBot requested review from stickies-v on Jan 12, 2024
  63. achow101 commented at 5:38 pm on January 16, 2024: member
    ACK df30247705940c50c5eaafd74e2abbeb8b0cec07
  64. DrahtBot removed review request from achow101 on Jan 16, 2024
  65. achow101 merged this on Jan 16, 2024
  66. achow101 closed this on Jan 16, 2024

  67. glozow referenced this in commit d055adad98 on Jan 18, 2024
  68. stickies-v commented at 1:30 pm on January 18, 2024: contributor

    Post-merge re-ACK df30247705940c50c5eaafd74e2abbeb8b0cec07, I was still investigating the non-deterministic behaviour in wallet_import_rescan but I’ve now found the issue and it doesn’t seem problematic enough to revert or immediately fix.

    When rebasing these tests on top of e.g. #29019 as well as on https://github.com/bitcoin/bitcoin/commit/453b4813ebc74859864803e9972b58e4be76a4d6~1, the mempool variant tests should fail for every variant where expect_rescan == True because in each case, only the parent is re-imported in the mempool (after re-org). However, when running the test multiple times one can observe this is not the case, and it sometimes takes 2 or even more rescan variants before the test fails.

    Expected failure logs:

    0...
    12024-01-18T13:20:15.028000Z TestFramework (INFO): Test that the mempool is rescanned as well if the rescan parameter is set to true
    22024-01-18T13:20:30.271000Z TestFramework (INFO): Run import for mempool variant Variant(call=<Call.single: 1>, data=<Data.address: 1>, address_type=<AddressType.bech32: 'bech32'>, rescan=<Rescan.no: 1>, prune=False)
    32024-01-18T13:20:30.317000Z TestFramework (INFO): Run import for mempool variant Variant(call=<Call.single: 1>, data=<Data.address: 1>, address_type=<AddressType.bech32: 'bech32'>, rescan=<Rescan.yes: 2>, prune=False)
    42024-01-18T13:20:30.388000Z TestFramework (ERROR): JSONRPC error
    5...
    

    Actual failure logs, sometimes:

     0...
     12024-01-18T13:25:50.799000Z TestFramework (INFO): Test that the mempool is rescanned as well if the rescan parameter is set to true
     22024-01-18T13:26:06.108000Z TestFramework (INFO): Run import for mempool variant Variant(call=<Call.single: 1>, data=<Data.address: 1>, address_type=<AddressType.bech32: 'bech32'>, rescan=<Rescan.no: 1>, prune=False)
     32024-01-18T13:26:06.164000Z TestFramework (INFO): Run import for mempool variant Variant(call=<Call.single: 1>, data=<Data.address: 1>, address_type=<AddressType.bech32: 'bech32'>, rescan=<Rescan.yes: 2>, prune=False)
     42024-01-18T13:26:06.235000Z TestFramework (INFO): Run import for mempool variant Variant(call=<Call.single: 1>, data=<Data.address: 1>, address_type=<AddressType.p2sh_segwit: 'p2sh-segwit'>, rescan=<Rescan.no: 1>, prune=False)
     52024-01-18T13:26:06.268000Z TestFramework (INFO): Run import for mempool variant Variant(call=<Call.single: 1>, data=<Data.address: 1>, address_type=<AddressType.p2sh_segwit: 'p2sh-segwit'>, rescan=<Rescan.yes: 2>, prune=False)
     62024-01-18T13:26:06.341000Z TestFramework (INFO): Run import for mempool variant Variant(call=<Call.single: 1>, data=<Data.address: 1>, address_type=<AddressType.legacy: 'legacy'>, rescan=<Rescan.no: 1>, prune=False)
     72024-01-18T13:26:06.374000Z TestFramework (INFO): Run import for mempool variant Variant(call=<Call.single: 1>, data=<Data.address: 1>, address_type=<AddressType.legacy: 'legacy'>, rescan=<Rescan.yes: 2>, prune=False)
     82024-01-18T13:26:06.449000Z TestFramework (INFO): Run import for mempool variant Variant(call=<Call.single: 1>, data=<Data.pub: 2>, address_type=<AddressType.bech32: 'bech32'>, rescan=<Rescan.no: 1>, prune=False)
     92024-01-18T13:26:06.494000Z TestFramework (INFO): Run import for mempool variant Variant(call=<Call.single: 1>, data=<Data.pub: 2>, address_type=<AddressType.bech32: 'bech32'>, rescan=<Rescan.yes: 2>, prune=False)
    102024-01-18T13:26:06.578000Z TestFramework (ERROR): JSONRPC error
    11...
    

    This happens because a chain of transactions can be formed, where variant_a.child_tx == variant_b.initial_txid, causing variant_a’s child tx to be re-imported into the mempool when variant b’s initial tx is re-orged.

    I don’t think this needs to be fixed or reverted, because if such a chain is formed, the last variant in the chain is still guaranteed to fail. The descriptor test is unaffected because we create only a single parent/child combo.

  69. glozow referenced this in commit 2e76e87cef on Jan 18, 2024
  70. glozow referenced this in commit 62694d727e on Jan 18, 2024
  71. stickies-v commented at 2:26 pm on January 19, 2024: contributor
    Reviewers might be interested in #29283, fixing a rare intermittency issue
  72. glozow referenced this in commit ecb8ebc660 on Jan 19, 2024
  73. glozow referenced this in commit ac1b9a51db on Jan 19, 2024

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-07-03 10:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me