rpc: Enable wallet import on pruned nodes and add test #24865

pull aureleoules wants to merge 3 commits into bitcoin:master from aureleoules:2022-04-importwallet-pruned changing 5 files +189 −31
  1. aureleoules commented at 3:38 pm on April 15, 2022: member

    Reopens #16037

    I have rebased the PR, addressed the comments of the original PR and added a functional test.

    Before this change importwallet fails if any block is pruned. This PR makes it possible to importwallet if all required blocks aren’t pruned. This is possible because the dump format includes key timestamps.

    For reviewers: python test/functional/wallet_pruning.py --nocleanup will generate a large blockchain (~700MB) that can be used to manually test wallet imports on a pruned node. Node0 is not pruned, while node1 is.

  2. DrahtBot added the label RPC/REST/ZMQ on Apr 15, 2022
  3. DrahtBot added the label Wallet on Apr 15, 2022
  4. in test/functional/wallet_pruning.py:13 in 35432901b4 outdated
     8+
     9+from test_framework.util import (
    10+    assert_raises_rpc_error,
    11+)
    12+
    13+from test_framework.blocktools import (create_coinbase)
    


    vincenzopalazzo commented at 9:13 pm on April 15, 2022:
    0from test_framework.blocktools import create_coinbase
    
  5. in test/functional/wallet_pruning.py:23 in 35432901b4 outdated
    20+from test_framework.wallet_util import (
    21+    get_key,
    22+)
    23+from test_framework.test_framework import BitcoinTestFramework
    24+
    25+class WalletPruningTest(BitcoinTestFramework):
    


    vincenzopalazzo commented at 9:14 pm on April 15, 2022:
    0
    1
    2class WalletPruningTest(BitcoinTestFramework):
    

    Usualy in python there are two spaces between class and the last import declaration

  6. vincenzopalazzo changes_requested
  7. DrahtBot commented at 6:33 am on April 16, 2022: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK kouloumos, furszy, w0xlt, achow101
    Concept ACK Xekyo, theStack
    Stale ACK ryanofsky

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    No conflicts as of last run.

  8. aureleoules force-pushed on Apr 16, 2022
  9. DrahtBot added the label Needs rebase on Apr 26, 2022
  10. aureleoules force-pushed on Apr 26, 2022
  11. DrahtBot removed the label Needs rebase on Apr 26, 2022
  12. aureleoules force-pushed on May 14, 2022
  13. aureleoules force-pushed on May 14, 2022
  14. in test/functional/wallet_pruning.py:23 in 46b7f731b1 outdated
    19+
    20+
    21+class WalletPruningTest(BitcoinTestFramework):
    22+    nTime = 0
    23+    def mine_large_blocks(self, node, n):
    24+        # Taken from test/functional/feature_pruning.py
    


    murchandamus commented at 3:47 pm on May 31, 2022:
    If we’re reusing helper functions, perhaps we should extract them to a single place in a follow-up PR.
  15. in test/functional/wallet_pruning.py:108 in 46b7f731b1 outdated
    131+        self.nodes[1].dumpwallet(os.path.join(self.nodes[1].datadir, wname))
    132+
    133+        # mine large blocks, blocks will be pruned automatically
    134+        self.mine_large_blocks(self.nodes[0], 150)
    135+        self.sync_all(self.nodes)
    136+
    


    murchandamus commented at 3:59 pm on May 31, 2022:
    This looks good to me, but perhaps it would be more comprehensive to learn the birthheight or first transaction’s height from the wallet, and check/assert here that the corresponding block has been pruned.
  16. murchandamus commented at 4:00 pm on May 31, 2022: contributor
    Did a light code-review, concept ACK
  17. in src/wallet/rpc/backup.cpp:1392 in 46b7f731b1 outdated
    1387@@ -1383,6 +1388,8 @@ RPCHelpMan importmulti()
    1388                 nLowestTimestamp = timestamp;
    1389             }
    1390         }
    1391+
    1392+        EnsureBlockDataFromTime(pwallet->chain(), nLowestTimestamp);
    


    achow101 commented at 9:06 pm on June 20, 2022:

    In 46b7f731b1e98b0836c5b97cd99c0887d6173a17 “rpc: Enable wallet import on pruned nodes and add test”

    We don’t want to do this for importmulti. importmulti can have some imports succeed, and others fail. Just because one import has an out of range timestamp does not mean the entire import should fail the RPC. This check would result in a JSONRPCError which is not what we expect from importmulti - rather there should be an error per import that has a too old timestamp.

    Furthermore, we already handle the pruned node case. If the rescan reports a time that is more recent than the timestamp for a given import, we will give an error for that import with more details about what happened, maybe why it happened, and how the user can resolve it.

  18. aureleoules force-pushed on Jul 5, 2022
  19. aureleoules force-pushed on Jul 5, 2022
  20. aureleoules force-pushed on Jul 5, 2022
  21. aureleoules force-pushed on Jul 5, 2022
  22. aureleoules force-pushed on Jul 5, 2022
  23. aureleoules force-pushed on Jul 5, 2022
  24. aureleoules force-pushed on Jul 5, 2022
  25. aureleoules force-pushed on Jul 5, 2022
  26. in test/functional/wallet_pruning.py:54 in b45e54bf51 outdated
    49+        self.skip_if_no_wallet()
    50+
    51+    def setup_network(self):
    52+        self.setup_nodes()
    53+        self.connect_nodes(0, 1)
    54+        self.sync_all(self.nodes)
    


    kouloumos commented at 4:22 pm on July 12, 2022:
    Do you actually need this? The only difference in behaviour that I see from the default https://github.com/bitcoin/bitcoin/blob/1d89fc695a3aeb3e3dcadf371b7667572b38c836/test/functional/test_framework/test_framework.py#L387 is the 0 -> 1 instead of 1 -> 0 nodes’ connection which seems to make no difference for the test.
  27. in test/functional/wallet_pruning.py:59 in b45e54bf51 outdated
    54+        self.sync_all(self.nodes)
    55+
    56+    def create_big_chain(self):
    57+        self.log.info("Generating a long chain of blocks...")
    58+        # Generate 288 light blocks, the minimum required to stay on disk with prune enabled
    59+        self.generate(self.nodes[0], 288, sync_fun=lambda: self.sync_all(self.nodes))
    


    kouloumos commented at 4:28 pm on July 12, 2022:
    I think that if sync_fun is not defined you will still have the desired behaviour here
  28. in test/functional/wallet_pruning.py:63 in b45e54bf51 outdated
    58+        # Generate 288 light blocks, the minimum required to stay on disk with prune enabled
    59+        self.generate(self.nodes[0], 288, sync_fun=lambda: self.sync_all(self.nodes))
    60+
    61+        # Generate large blocks to make sure we have enough to test chain pruning
    62+        self.mine_large_blocks(self.nodes[0], 610)
    63+        self.sync_all(self.nodes)
    


  29. in test/functional/wallet_pruning.py:77 in b45e54bf51 outdated
    72+        # import wallet
    73+        self.nodes[1].importwallet(os.path.join(self.nodes[1].datadir, wname))
    74+
    75+        # mine some blocks, pruning should not have removed the block
    76+        self.mine_large_blocks(self.nodes[0], 5)
    77+        self.sync_all(self.nodes)
    


    kouloumos commented at 4:39 pm on July 12, 2022:
    nit: for how it is used in this test, I think that sync_all could be part of the mine_large_blocks method
  30. in test/functional/wallet_pruning.py:112 in b45e54bf51 outdated
    107+        self.log.info("Warning! This test requires ~1.5GB of disk space")
    108+
    109+        self.create_big_chain()
    110+
    111+        self.wallet_import_pruned_test()
    112+        self.wallet_import_pruned_missing_blocks()
    


    kouloumos commented at 9:04 am on July 13, 2022:
    Maybe it would be better to include a test prefix as in most tests? e.g test_wallet_import_on_pruned_node{_with_missing_blocks}
  31. in test/functional/wallet_pruning.py:65 in b45e54bf51 outdated
    60+
    61+        # Generate large blocks to make sure we have enough to test chain pruning
    62+        self.mine_large_blocks(self.nodes[0], 610)
    63+        self.sync_all(self.nodes)
    64+
    65+    def wallet_import_pruned_test(self):
    


    kouloumos commented at 10:06 am on July 13, 2022:

    I think that this test is not testing what you are expecting it to test, or its scope is just not clear to me.

    Even if I revert your changes to backup.cpp (git checkout 9fb2a2bc6768ab03fcada9155d52a16ce6f6a0cc -- src/wallet/rpc/backup.cpp) this test passes. My understanding is that this never reaches the prune state because no blocks are pruned until we are above PruneAfterHeight https://github.com/bitcoin/bitcoin/blob/1d89fc695a3aeb3e3dcadf371b7667572b38c836/src/chainparams.cpp#L429 The logic in feature_pruning.py also helped me to understand this behaviour.

  32. kouloumos commented at 2:21 pm on July 13, 2022: contributor

    I think that wallet_pruning.py is not testing properly this added feature. This is due to the testing logic but mainly because of the default setup of the testing framework that is being used in this test.

    This is possible because the dump format includes key timestamps.

    The importwallet rpc calculates the min nTimeBegin timestamp for which findFirstBlockWithTimeAndHeight in the newly created function EnsureBlockDataFromTime will find the earliest block with timestamp equal or greater to.

    As you can verify from the wallet_*.dat dumps of this test

    0# node1 wallet dump
    1...
    2cUxsWyKyZ9MAQTaAhUQWJmBbSvHMwSmuv59KgxQV7oZQU3PXN3KE 1970-01-01T00:00:01Z label=coinbase # addr=msX6jQXvxiNhx3Q62PKeLPrhrqZQdSimTg,2MtBq31Chq2pntJ81PQLR4q4sMEG7K46wsQ,bcrt1qsw5g6ehh439vurfyhdh93d66hw0kf9085wy7ma
    3...
    

    the deterministic private keys that are imported to each test node as part of the testing framework setup https://github.com/bitcoin/bitcoin/blob/1d89fc695a3aeb3e3dcadf371b7667572b38c836/test/functional/test_framework/test_framework.py#L441

    have a 1970-01-01T00:00:01Z timestamp which means that EnsureBlockDataFromTime will always check for the existence of all blocks between genesis and the current height.

    Even if that was not the case, because the nodes’ wallets are initialized before we start generating blocks, the timestamp in the wallet dump is still before the first created block thus EnsureBlockDataFromTime checks for the existence of all blocks between the first block and the current height.

    This behaviour will always result to missing blocks as soon as we start pruning. The reason that the first part of the test (wallet_import_pruned_test) is not failing is because it never reaches the prune state (height < PruntAfterHeight).

    I believe that you can better test for the new feature by overriding setup_nodes and initializing node1 wallet mid-test

  33. aureleoules force-pushed on Jul 19, 2022
  34. aureleoules force-pushed on Jul 19, 2022
  35. aureleoules commented at 7:25 am on July 20, 2022: member
    Thank you for your thorough review @kouloumos ! I believe I have addressed all of your comments.
  36. in src/wallet/rpc/backup.cpp:106 in 45435bc688 outdated
    101+    int height{0};
    102+    const bool found{chain.findFirstBlockWithTimeAndHeight(timestamp - TIMESTAMP_WINDOW, 0, FoundBlock().height(height))};
    103+    const std::optional<int> chainHeight{chain.getHeight()};
    104+
    105+    if (found && chainHeight && !chain.hasBlocks(chain.getBlockHash(chainHeight.value()), height)) {
    106+        throw JSONRPCError(RPC_WALLET_ERROR, "Pruned blocks required to import keys");
    


    kouloumos commented at 9:37 am on September 1, 2022:
    I think that at this point we have the required height for which we need blocks. Could this change to a more descriptive error message stating that required weight? And maybe “[…]. Use RPC call getblockchaininfo to determine your pruned height” as here?

    aureleoules commented at 10:11 am on September 5, 2022:
    thanks, added it
  37. aureleoules force-pushed on Sep 5, 2022
  38. aureleoules force-pushed on Sep 6, 2022
  39. in src/wallet/rpc/backup.cpp:105 in 0aedf6f3cb outdated
    100+
    101+    int height{0};
    102+    const bool found{chain.findFirstBlockWithTimeAndHeight(timestamp - TIMESTAMP_WINDOW, 0, FoundBlock().height(height))};
    103+    const std::optional<int> chainHeight{chain.getHeight()};
    104+
    105+    if (found && chainHeight && !chain.hasBlocks(chain.getBlockHash(chainHeight.value()), height)) {
    


    ryanofsky commented at 5:42 pm on September 7, 2022:

    In commit “rpc: Enable wallet import on pruned nodes and add test” (0aedf6f3cb5a513f9e42458362261d4afa2ea17f)

    I think it would be slightly better to pass in pwallet->m_last_block_processed instead of chain.getBlockHash(chainHeight.value()) as the first hasBlocks argument. In general to avoid race conditions wallet just tries to query node based on it’s own chain tip and not rely on current node tip. To be more race free, could add a Chain::hasBlocksFromTime(int64_t start_time, uint256 end_block) method and pass m_last_block_processed as the end block. But just passing m_last_block_processed to hasBlocks here and dropping the chainHeight variable should already be an improvement.


    aureleoules commented at 11:24 pm on September 7, 2022:
    Thanks for the review, fixed it!
  40. in test/functional/feature_index_prune.py:33 in 0aedf6f3cb outdated
    29@@ -30,10 +30,10 @@ def sync_index(self, height):
    30         expected_stats = {
    31             'coinstatsindex': {'synced': True, 'best_block_height': height}
    32         }
    33-        self.wait_until(lambda: self.nodes[1].getindexinfo() == expected_stats)
    34+        self.wait_until(lambda: self.nodes[1].getindexinfo() == expected_stats, timeout=120)
    


    ryanofsky commented at 5:52 pm on September 7, 2022:

    In commit “rpc: Enable wallet import on pruned nodes and add test” (0aedf6f3cb5a513f9e42458362261d4afa2ea17f)

    Seems like an unrelated change? Would be good to drop this if possible, or maybe comment why overriding timeout is necessary.


    aureleoules commented at 11:23 pm on September 7, 2022:
    oops, fixed
  41. in test/functional/test_framework/blocktools.py:90 in 0aedf6f3cb outdated
    86@@ -87,6 +87,13 @@ def create_block(hashprev=None, coinbase=None, ntime=None, *, version=None, tmpl
    87     block.calc_sha256()
    88     return block
    89 
    90+def create_large_block(hashprev=None, ntime=None, height=None, scriptPubKey=CScript([OP_RETURN] + [OP_TRUE] * 950000)):
    


    ryanofsky commented at 5:59 pm on September 7, 2022:

    In commit “rpc: Enable wallet import on pruned nodes and add test” (0aedf6f3cb5a513f9e42458362261d4afa2ea17f)

    This CScript(...) value should just be generated locally inside the function, not be a python default parameter. Otherwise the cscript object is created and lives in memory even if this function is never called.

    Can just say scriptPubKey=None in parameter list and if scriptPubKey is None: scriptPubKey = CScript(...) inside the function. Or just make scriptPubKey a local variable since the argument is never specified anyway.


    aureleoules commented at 6:13 pm on September 7, 2022:
    This is a large object that takes time to construct so I thought making it static would avoid constructing it multiple times, but I had not thought of the fact that this lives in memory even if the function is unused. Will fix though, thanks.
  42. ryanofsky approved
  43. ryanofsky commented at 6:04 pm on September 7, 2022: contributor
    Code review ACK 0aedf6f3cb5a513f9e42458362261d4afa2ea17f. Nice change and tests!
  44. aureleoules force-pushed on Sep 7, 2022
  45. aureleoules force-pushed on Sep 7, 2022
  46. aureleoules force-pushed on Sep 8, 2022
  47. aureleoules force-pushed on Sep 8, 2022
  48. in src/wallet/rpc/backup.cpp:103 in f42c077a8f outdated
     98+    if (!chain.havePruned())
     99+        return;
    100+
    101+    int height{0};
    102+    const bool found{chain.findFirstBlockWithTimeAndHeight(timestamp - TIMESTAMP_WINDOW, 0, FoundBlock().height(height))};
    103+    const std::optional<int> chainHeight{chain.getHeight()};
    


    ryanofsky commented at 3:06 pm on September 8, 2022:

    In commit “rpc: Enable wallet import on pruned nodes and add test” (f42c077a8f948f5c93b68eb1b7fa246b4f6475ef)

    I think there’s no need for the chainHeight variable anymore and it would be better to drop it.

  49. in test/functional/feature_pruning.py:31 in f42c077a8f outdated
    33 # Rescans start at the earliest block up to 2 hours before a key timestamp, so
    34 # the manual prune RPC avoids pruning blocks in the same window to be
    35 # compatible with pruning based on key creation time.
    36 TIMESTAMP_WINDOW = 2 * 60 * 60
    37 
    38+scriptPubKey = CScript([OP_RETURN] + [OP_TRUE] * 950000)
    


    ryanofsky commented at 3:10 pm on September 8, 2022:

    In commit “rpc: Enable wallet import on pruned nodes and add test” (f42c077a8f948f5c93b68eb1b7fa246b4f6475ef)

    Just IMO, but I think it would be better to keep the existing big_script variable below and delete this new variable (smaller diff, more descriptive variable name, more straightforward code avoiding global variables)

  50. in test/functional/test_framework/blocktools.py:90 in f42c077a8f outdated
    86@@ -87,6 +87,13 @@ def create_block(hashprev=None, coinbase=None, ntime=None, *, version=None, tmpl
    87     block.calc_sha256()
    88     return block
    89 
    90+def create_large_block(hashprev, ntime, height, scriptPubKey):
    


    ryanofsky commented at 3:32 pm on September 8, 2022:

    In commit “rpc: Enable wallet import on pruned nodes and add test” (f42c077a8f948f5c93b68eb1b7fa246b4f6475ef)

    This new create_large_block seems very inflexible and also has a misleading name. Would drop it and instead just add a new option to create_coinbase like:

     0diff --git a/test/functional/feature_pruning.py b/test/functional/feature_pruning.py
     1index 8d72d11baf5..d14970c9463 100755
     2--- a/test/functional/feature_pruning.py
     3+++ b/test/functional/feature_pruning.py
     4@@ -10,7 +10,8 @@ This test takes 30 mins or more (up to 2 hours)
     5 """
     6 import os
     7 
     8-from test_framework.blocktools import create_large_block
     9+from test_framework.blocktools import create_block
    10+from test_framework.blocktools import create_coinbase
    11 from test_framework.test_framework import BitcoinTestFramework
    12 from test_framework.util import (
    13     assert_equal,
    14@@ -47,7 +48,7 @@ def mine_large_blocks(node, n):
    15     previousblockhash = int(best_block["hash"], 16)
    16 
    17     for _ in range(n):
    18-        block = create_large_block(hashprev=previousblockhash, height=height, ntime=mine_large_blocks.nTime, scriptPubKey=scriptPubKey)
    19+        block = create_block(hashprev=previousblockhash, ntime=mine_large_blocks.nTime, coinbase=create_coinbase(height, script_pubkey=scriptPubKey))
    20         block.solve()
    21 
    22         # Submit to the node
    23diff --git a/test/functional/test_framework/blocktools.py b/test/functional/test_framework/blocktools.py
    24index 5916ffc2b17..77653df96e9 100644
    25--- a/test/functional/test_framework/blocktools.py
    26+++ b/test/functional/test_framework/blocktools.py
    27@@ -87,13 +87,6 @@ def create_block(hashprev=None, coinbase=None, ntime=None, *, version=None, tmpl
    28     block.calc_sha256()
    29     return block
    30 
    31-def create_large_block(hashprev, ntime, height, scriptPubKey):
    32-    coinbase_tx = create_coinbase(height)
    33-    coinbase_tx.vout[0].scriptPubKey = scriptPubKey
    34-    coinbase_tx.rehash()
    35-
    36-    return create_block(coinbase=coinbase_tx, ntime=ntime, hashprev=hashprev)
    37-
    38 def get_witness_script(witness_root, witness_nonce):
    39     witness_commitment = uint256_from_str(hash256(ser_uint256(witness_root) + ser_uint256(witness_nonce)))
    40     output_data = WITNESS_COMMITMENT_HEADER + ser_uint256(witness_commitment)
    41@@ -127,7 +120,7 @@ def script_BIP34_coinbase_height(height):
    42     return CScript([CScriptNum(height)])
    43 
    44 
    45-def create_coinbase(height, pubkey=None, extra_output_script=None, fees=0, nValue=50):
    46+def create_coinbase(height, pubkey=None, *, script_pubkey=None, extra_output_script=None, fees=0, nValue=50):
    47     """Create a coinbase transaction.
    48 
    49     If pubkey is passed in, the coinbase output will be a P2PK output;
    50@@ -145,6 +138,8 @@ def create_coinbase(height, pubkey=None, extra_output_script=None, fees=0, nValu
    51         coinbaseoutput.nValue += fees
    52     if pubkey is not None:
    53         coinbaseoutput.scriptPubKey = key_to_p2pk_script(pubkey)
    54+    elif script_pubkey is not None:
    55+        coinbaseoutput.scriptPubKey = script_pubkey
    56     else:
    57         coinbaseoutput.scriptPubKey = CScript([OP_TRUE])
    58     coinbase.vout = [coinbaseoutput]
    59diff --git a/test/functional/wallet_pruning.py b/test/functional/wallet_pruning.py
    60index 9b8160a4490..e42994422e0 100755
    61--- a/test/functional/wallet_pruning.py
    62+++ b/test/functional/wallet_pruning.py
    63@@ -7,7 +7,8 @@
    64 import os
    65 
    66 from test_framework.util import assert_raises_rpc_error
    67-from test_framework.blocktools import create_large_block
    68+from test_framework.blocktools import create_block
    69+from test_framework.blocktools import create_coinbase
    70 from test_framework.test_framework import BitcoinTestFramework
    71 
    72 from test_framework.script import (
    73@@ -26,7 +27,7 @@ class WalletPruningTest(BitcoinTestFramework):
    74         self.nTime = max(self.nTime, int(best_block["time"])) + 1
    75         previousblockhash = int(best_block["hash"], 16)
    76         for _ in range(n):
    77-            block = create_large_block(hashprev=previousblockhash, height=height, ntime=self.nTime, scriptPubKey=self.scriptPubKey)
    78+            block = create_block(hashprev=previousblockhash, ntime=self.nTime, coinbase=create_coinbase(height, script_pubkey=self.scriptPubKey))
    79             block.solve()
    80 
    81             # Submit to the node
    
  51. in src/wallet/rpc/backup.cpp:105 in f42c077a8f outdated
    100+
    101+    int height{0};
    102+    const bool found{chain.findFirstBlockWithTimeAndHeight(timestamp - TIMESTAMP_WINDOW, 0, FoundBlock().height(height))};
    103+    const std::optional<int> chainHeight{chain.getHeight()};
    104+
    105+    LOCK(wallet.cs_wallet);
    


    ryanofsky commented at 3:37 pm on September 8, 2022:

    In commit “rpc: Enable wallet import on pruned nodes and add test” (f42c077a8f948f5c93b68eb1b7fa246b4f6475ef)

    I don’t think we need to keep the wallet locked while running all the code below. Would suggest using a shorter lock:

    0uint256 tip_hash = WITH_LOCK(wallet.cs_wallet, return wallet.GetLastBlockHash());
    
  52. ryanofsky approved
  53. ryanofsky commented at 3:42 pm on September 8, 2022: contributor
    Code review ACK f42c077a8f948f5c93b68eb1b7fa246b4f6475ef. Just suggested changes since last review, and increasing sync_blocks timeout in the new test. Thanks for the updates!
  54. aureleoules force-pushed on Sep 8, 2022
  55. aureleoules commented at 6:25 pm on September 8, 2022: member
    Thanks for the review again @ryanofsky, I think I’ve addressed all your comments.
  56. aureleoules force-pushed on Sep 8, 2022
  57. in test/functional/feature_pruning.py:48 in 05463d4001 outdated
    40@@ -41,28 +41,14 @@ def mine_large_blocks(node, n):
    41         mine_large_blocks.nTime = 0
    42 
    43     # Get the block parameters for the first block
    44-    big_script = CScript([OP_RETURN] + [OP_NOP] * 950000)
    45     best_block = node.getblock(node.getbestblockhash())
    46     height = int(best_block["height"]) + 1
    47     mine_large_blocks.nTime = max(mine_large_blocks.nTime, int(best_block["time"])) + 1
    48     previousblockhash = int(best_block["hash"], 16)
    49+    bigScript = CScript([OP_RETURN] + [OP_TRUE] * 950000)
    


    ryanofsky commented at 6:38 pm on September 12, 2022:

    In commit “rpc: Enable wallet import on pruned nodes and add test” (05463d4001001213c17672c1607a3a493927ddd7)

    Can just keep existing big_script variable, no need to replace it with new bigScript variable. This would make the diff small and make the naming convention more convenient


    aureleoules commented at 10:02 am on September 13, 2022:
    my bad, fixed
  58. ryanofsky approved
  59. ryanofsky commented at 6:39 pm on September 12, 2022: contributor
    Code review ACK 05463d4001001213c17672c1607a3a493927ddd7. Only changes since last review were some suggested cleanups & simplifications
  60. aureleoules force-pushed on Sep 13, 2022
  61. aureleoules force-pushed on Sep 13, 2022
  62. ryanofsky approved
  63. ryanofsky commented at 5:22 pm on September 13, 2022: contributor
    Code review ACK 6dc2e7141de83022f0ca0c48afd5ea7e5c4684d0. Only change since last review is changing bigScript back to big_script
  64. w0xlt approved
  65. aureleoules commented at 12:58 pm on September 16, 2022: member
    Fixes #9409.
  66. in test/functional/wallet_pruning.py:119 in 6dc2e7141d outdated
    114+    def run_test(self):
    115+        self.log.info("Warning! This test requires ~1.5GB of disk space")
    116+
    117+        self.create_big_chain()
    118+
    119+        # connect mid-test node1
    


    kouloumos commented at 1:05 pm on September 30, 2022:
    nit: maybe add some comments here to explain the rationale behind this mid-test connection? I think it will not be very clear for someone reading this test in the future.
  67. in test/functional/wallet_pruning.py:81 in 6dc2e7141d outdated
    76+        wname = "wallet_pruned.dat"
    77+
    78+        # export wallet
    79+        self.nodes[1].dumpwallet(os.path.join(self.nodes[1].datadir, wname))
    80+        # import wallet
    81+        self.nodes[1].importwallet(os.path.join(self.nodes[1].datadir, wname))
    


    kouloumos commented at 1:34 pm on September 30, 2022:

    This dump&import pattern in the same wallet might be ok for checking against the RPC assertion of the newly introduced logic, but in reality the keys and scripts in the dump file are not imported in the wallet because they already exist.

    I don’t consider this as an issue because that behavior is also tested elsewhere(wallet_dump.py) but I think changing this part makes for a better usage scenario.

  68. kouloumos commented at 1:34 pm on September 30, 2022: contributor

    reACK 6dc2e7141de83022f0ca0c48afd5ea7e5c4684d0

    I’ve also verified the expected behavior on a pruned signet node by dumping the wallet and then manually changing the timestamps in the file to simulate older keys for which the block data do not exist (pruned).

    Does it make sense to also add a check (in the functional tests), that after restoring, the wallet correctly accounts for balances and maybe addresses? Or that’s redundant because a similar check exists at wallet_dump.py? https://github.com/bitcoin/bitcoin/blob/33eef562a321ce772a9a88073a78a85894cb3fe8/test/functional/wallet_dump.py#L198-L206

  69. aureleoules force-pushed on Oct 2, 2022
  70. aureleoules commented at 4:22 pm on October 2, 2022: member
    Thank you @kouloumos for your review. I have reworked the test to take your comments into account and also reduce the time it takes to run. It took 1min24 to run and reduced it to 54 seconds (on my machine). Also added more comments to explain each step of the test and split the PR into two commits.
  71. aureleoules force-pushed on Oct 2, 2022
  72. aureleoules force-pushed on Oct 2, 2022
  73. aureleoules commented at 4:51 pm on October 2, 2022: member
    Added a test case to check that the wallet balance is correct.
  74. andrewtoth commented at 1:44 pm on October 19, 2022: contributor
    Sorry, I didn’t check that #26206 will actually conflict with this PR. This needs a rebase on top of it and then revert that commit in this PR.
  75. aureleoules force-pushed on Oct 19, 2022
  76. aureleoules force-pushed on Oct 19, 2022
  77. aureleoules commented at 2:53 pm on October 19, 2022: member
    Rebased, reverted 4aff7a48a4e0f1075306f181a276b8a74c857022 and added @ryanofsky and @kouloumos as co-authors.
  78. in src/wallet/rpc/backup.cpp:96 in 661e45300a outdated
    92@@ -93,6 +93,20 @@ static void RescanWallet(CWallet& wallet, const WalletRescanReserver& reserver,
    93     }
    94 }
    95 
    96+static void EnsureBlockDataFromTime(interfaces::Chain& chain, const CWallet& wallet, int64_t timestamp)
    


    andrewtoth commented at 2:46 am on October 20, 2022:
    Why create a function for this? Would this be easier to reason about if it was inline?

    aureleoules commented at 7:53 am on October 20, 2022:
    I’ve seen many instances of Ensure(.*) functions and it also helps to reduce the length of the parent function I suppose.
  79. kouloumos commented at 5:20 pm on October 24, 2022: contributor
    Does this need a rebase after #26380?
  80. aureleoules closed this on Oct 24, 2022

  81. aureleoules reopened this on Oct 24, 2022

  82. in src/wallet/rpc/backup.cpp:98 in 661e45300a outdated
    92@@ -93,6 +93,20 @@ static void RescanWallet(CWallet& wallet, const WalletRescanReserver& reserver,
    93     }
    94 }
    95 
    96+static void EnsureBlockDataFromTime(interfaces::Chain& chain, const CWallet& wallet, int64_t timestamp)
    97+{
    98+    if (!chain.havePruned())
    


    maflcko commented at 7:20 am on October 25, 2022:
    0    if (!chain.havePruned()) {
    

    nit (for new code)

  83. aureleoules force-pushed on Oct 25, 2022
  84. aureleoules force-pushed on Oct 25, 2022
  85. aureleoules commented at 8:40 am on October 25, 2022: member
    Dropped e1eadaa72d6831d1d0a53ba97c215dc4cdb64436, rebased for #26380 and addressed MarcoFalke’s nit.
  86. in test/functional/wallet_pruning.py:126 in 6abd81b11d outdated
    121+        self.mine_large_blocks(self.nodes[0], 128)
    122+
    123+        # Generate 50MB worth of large blocks in the second blk*.dat file
    124+        self.mine_large_blocks(self.nodes[0], 50)
    125+        # Create a wallet and a backup of it which birth's block is in the second blk*.dat file
    126+        self.nodes[0].createwallet(wallet_name="wallet_init", descriptors=self.options.descriptors, load_on_startup=True)
    


    achow101 commented at 6:42 pm on October 31, 2022:

    In 6abd81b11dd1a45bdffcc85ba90591474a3f48b0 “test: Wallet imports on pruned nodes”

    Since this is testing legacy wallet only RPCs, we should not be looking at the --legacy-wallet/--descriptors options to determine the wallet type to create, for all wallets created in this test.

    0        self.nodes[0].createwallet(wallet_name="wallet_init", descriptors=False, load_on_startup=True)
    
  87. in test/functional/test_runner.py:88 in 6abd81b11d outdated
    82@@ -83,6 +83,7 @@
    83     'feature_pruning.py',
    84     'feature_dbcrash.py',
    85     'feature_index_prune.py',
    86+    'wallet_pruning.py',
    


    achow101 commented at 6:48 pm on October 31, 2022:

    In 6abd81b11dd1a45bdffcc85ba90591474a3f48b0 “test: Wallet imports on pruned nodes”

    Should have --legacy-wallet to indicate that this test is a legacy wallet only test. Also, the default is --descriptors.

    0    'wallet_pruning.py --legacy-wallet',
    
  88. in test/functional/feature_pruning.py:14 in 6abd81b11d outdated
     9@@ -10,8 +10,8 @@
    10 """
    11 import os
    12 
    13+from test_framework.blocktools import create_block
    14 from test_framework.blocktools import create_coinbase
    


    w0xlt commented at 4:54 am on November 1, 2022:
    0from test_framework.blocktools import (
    1    create_block,
    2    create_coinbase,
    3)
    
  89. w0xlt approved
  90. aureleoules force-pushed on Nov 1, 2022
  91. aureleoules commented at 11:32 am on November 1, 2022: member
    Addressed the suggestions, thanks @achow101 @w0xlt.
  92. in test/functional/wallet_pruning.py:71 in a0762cf6ec outdated
    66+        self.log.info("Make sure we can import wallet when pruned and required blocks are still available")
    67+
    68+        wname = "wallet_pruned.dat"
    69+
    70+        # Export wallet from node 0
    71+        self.nodes[0].createwallet(wallet_name="wallet_pruned", descriptors=self.options.descriptors, load_on_startup=True)
    


    achow101 commented at 7:28 pm on November 1, 2022:

    In a0762cf6ec0bbc5eb770912a1ab917862f5515ca “test: Wallet imports on pruned nodes”

    0        self.nodes[0].createwallet(wallet_name="wallet_pruned", descriptors=False, load_on_startup=True)
    

    aureleoules commented at 10:54 am on November 2, 2022:
    Oops, pushed.
  93. in test/functional/wallet_pruning.py:79 in a0762cf6ec outdated
    74+        # Add mature coins to the wallet
    75+        self.generatetoaddress(self.nodes[0], 1, self.nodes[0].getnewaddress(), sync_fun=lambda: None)
    76+        self.generate(self.nodes[0], 100)
    77+
    78+        # Import wallet into node 1
    79+        self.nodes[1].createwallet(wallet_name="wallet_pruned", descriptors=self.options.descriptors, load_on_startup=True)
    


    achow101 commented at 7:28 pm on November 1, 2022:

    In a0762cf6ec0bbc5eb770912a1ab917862f5515ca “test: Wallet imports on pruned nodes”

    0        self.nodes[1].createwallet(wallet_name="wallet_pruned", descriptors=False, load_on_startup=True)
    
  94. aureleoules force-pushed on Nov 2, 2022
  95. andrewtoth commented at 3:29 pm on November 2, 2022: contributor
    You should rebase to fix ci test failure for p2p_sendtxrcncl.py.
  96. aureleoules force-pushed on Nov 2, 2022
  97. aureleoules commented at 3:53 pm on November 2, 2022: member

    You should rebase to fix ci test failure for p2p_sendtxrcncl.

    :heavy_check_mark:

  98. in test/functional/wallet_pruning.py:96 in c46f5fe58e outdated
    91+        self.log.info("Wallet successfully imported on pruned node")
    92+
    93+    def test_wallet_import_pruned_with_missing_blocks(self):
    94+        self.log.info("Make sure we cannot import wallet when pruned and required blocks are not available")
    95+
    96+        wname = "wallet_init.dat"
    


    furszy commented at 3:03 pm on November 4, 2022:

    In c46f5fe5:

    Tiny nit, this could be passed as an argument so we avoid hardcoding file names more than once.

  99. furszy approved
  100. furszy commented at 3:04 pm on November 4, 2022: member
    Code ACK c46f5fe5
  101. in test/functional/wallet_pruning.py:120 in c46f5fe58e outdated
    114+
    115+        self.log.info("Generate a long chain of blocks...")
    116+
    117+        # A blk*.dat file is 128MB
    118+        # Generate 288 light blocks, the minimum required to stay on disk with prune enabled
    119+        self.generate(self.nodes[0], 288, sync_fun=lambda: None)
    


    theStack commented at 3:24 pm on November 10, 2022:
    nit: not directly related to this PR, but as a follow-up I think it would make sense to introduce the constant MIN_BLOCKS_TO_KEEP into the functional test framework, in order to the replace this magic number here and in other tests (e.g. feature_pruning.py)
  102. in test/functional/wallet_pruning.py:63 in c46f5fe58e outdated
    58+
    59+    def setup_network(self):
    60+        self.setup_nodes()
    61+
    62+    def skip_test_if_missing_module(self):
    63+        self.skip_if_no_wallet()
    


    theStack commented at 3:32 pm on November 10, 2022:

    Should ensure here that the test is only run if legacy wallet support (BDB) is compiled in:

    0        self.skip_if_no_wallet()
    1        self.skip_if_no_bdb()
    
  103. theStack commented at 3:37 pm on November 10, 2022: contributor
    Concept ACK
  104. aureleoules force-pushed on Nov 11, 2022
  105. aureleoules force-pushed on Nov 11, 2022
  106. aureleoules commented at 10:59 am on November 11, 2022: member
  107. aureleoules force-pushed on Nov 16, 2022
  108. aureleoules commented at 12:09 pm on November 16, 2022: member
    Reduced the number of blocks generated to speed up the test (from 1:40 to 1:24).
  109. in src/wallet/rpc/backup.cpp:587 in c594dbc756 outdated
    582+                if (birth_time > 0) nTimeBegin = std::min(nTimeBegin, birth_time);
    583                 scripts.push_back(std::pair<CScript, int64_t>(script, birth_time));
    584             }
    585         }
    586         file.close();
    587+        EnsureBlockDataFromTime(pwallet->chain(), *pwallet, nTimeBegin);
    


    kouloumos commented at 3:26 pm on November 24, 2022:
    Is there a reason that you are passing the chain interface as a separate argument instead of accessing it inside the function with wallet.chain().*?

    aureleoules commented at 10:07 am on November 27, 2022:
    Removed chain argument, thanks.
  110. in test/functional/wallet_pruning.py:120 in c594dbc756 outdated
    115+
    116+        self.log.info("Generate a long chain of blocks...")
    117+
    118+        # A blk*.dat file is 128MB
    119+        # Generate 288 light blocks, the minimum required to stay on disk with prune enabled
    120+        self.generate(self.nodes[0], MIN_BLOCKS_TO_KEEP, sync_fun=lambda: None)
    


    kouloumos commented at 3:30 pm on November 24, 2022:

    Is this different than

    0        self.generate(self.nodes[0], MIN_BLOCKS_TO_KEEP, sync_fun=self.no_op)
    

    If not, I think it’s better to keep it consistent across tests.


    kouloumos commented at 8:27 am on November 25, 2022:

    I understand why you need some light blocks in the beginning. What I don’t understand is the comment here referring to specifically 288 light blocks. Looking at the c++ code I read this about that number https://github.com/bitcoin/bitcoin/blob/38d06e1561013f4ca845fd5ba6ffcc64de67f9c0/src/validation.h#L68-L69

    How is that related to the logic of this test?


    aureleoules commented at 10:08 am on November 27, 2022:
    Updated thanks.

    aureleoules commented at 10:09 am on November 27, 2022:
    I think this was based on a different version of this test but you are correct 288 is not needed. I changed it to 250.
  111. in test/functional/wallet_pruning.py:127 in c594dbc756 outdated
    118+        # A blk*.dat file is 128MB
    119+        # Generate 288 light blocks, the minimum required to stay on disk with prune enabled
    120+        self.generate(self.nodes[0], MIN_BLOCKS_TO_KEEP, sync_fun=lambda: None)
    121+
    122+        # Generate 50MB worth of large blocks in the second blk*.dat file
    123+        self.mine_large_blocks(self.nodes[0], 50)
    


    kouloumos commented at 7:29 am on November 25, 2022:

    Are you sure that the files generated here are part of the second blk*.data file? Using an adaptation of this function says otherwise https://github.com/bitcoin/bitcoin/blob/38d06e1561013f4ca845fd5ba6ffcc64de67f9c0/test/functional/feature_pruning.py#L296-L297

    Your latest push decreasing the number of generated large blocks invalidate this statement. I calculated that you need min 142 blocks to be generated here in order for the the blk00001.dat file to be created, resulting to the wallet that you create below to actually have a birth height in this file.


    aureleoules commented at 10:08 am on November 27, 2022:
    Yes the comment was wrong.
  112. in test/functional/wallet_pruning.py:60 in c594dbc756 outdated
    55+    def setup_nodes(self):
    56+        self.add_nodes(self.num_nodes, extra_args=self.extra_args)
    57+        self.start_node(0)
    58+
    59+    def setup_network(self):
    60+        self.setup_nodes()
    


    kouloumos commented at 8:08 am on November 25, 2022:

    The custom setup logic that I implied on my initial review was because you were using the default created wallets. But since then, you are testing on manually created wallets, therefore this is not needed.

    Additionally, I’ve later realize that you can skip the creation of the default wallets with self.wallet_names=[] which can help to simplify the logic. https://github.com/bitcoin/bitcoin/blob/38d06e1561013f4ca845fd5ba6ffcc64de67f9c0/test/functional/test_framework/test_framework.py#L108-L113

  113. in test/functional/wallet_pruning.py:133 in c594dbc756 outdated
    128+        self.nodes[0].createwallet(wallet_name=wallet_name, descriptors=False, load_on_startup=True)
    129+        self.nodes[0].dumpwallet(os.path.join(self.nodes[0].datadir, wallet_name + ".dat"))
    130+        self.nodes[0].unloadwallet(wallet_name)
    131+
    132+        # Generate large blocks to make sure we have enough to test chain pruning
    133+        # The first blk*.dat file should be pruned from the node 1
    


    kouloumos commented at 8:42 am on November 25, 2022:
    This comment is not true. That file might be later pruned, but at this point is not and it makes reading the code confusing.
  114. in test/functional/wallet_pruning.py:83 in c594dbc756 outdated
    78+
    79+        # Import wallet into node 1
    80+        self.nodes[1].createwallet(wallet_name="wallet_pruned", descriptors=False, load_on_startup=True)
    81+        self.nodes[1].importwallet(os.path.join(self.nodes[0].datadir, wname))
    82+
    83+        # Make sure the wallets have the same balance
    


    kouloumos commented at 9:55 am on November 25, 2022:

    nit

    0        # Make sure that prune node's wallet correctly accounts for balances
    
  115. in test/functional/wallet_pruning.py:91 in c594dbc756 outdated
    86+        # Mine some blocks, pruning should not have removed the blocks required for the wallet
    87+        self.mine_large_blocks(self.nodes[0], 5)
    88+        self.sync_all()
    89+
    90+        # Import wallet, should still work
    91+        self.nodes[1].importwallet(os.path.join(self.nodes[0].datadir, wname))
    


    kouloumos commented at 9:59 am on November 25, 2022:

    ~You already imported the wallet in the pruned node above. I don’t understand why you need to repeat this here. What do those newly mined 5 large blocks achieve?~

    Further down me messing with the code, I understood that those 5 blocks are needed to actually get into prune state and prune the blk00000.dat file. But for someone reading the code this is not clear. And if we are now getting into prune mode, then what’s the above importwallet for?

  116. in test/functional/wallet_pruning.py:21 in c594dbc756 outdated
    16+    OP_RETURN,
    17+    OP_TRUE,
    18+)
    19+
    20+class WalletPruningTest(BitcoinTestFramework):
    21+    nTime = 0
    


    kouloumos commented at 11:29 am on November 25, 2022:
    Is this the right place for this variable? I think that I haven’t seen any other tests initializing self variables in such a way.

    aureleoules commented at 10:12 am on November 27, 2022:
    Thanks I moved it.
  117. in test/functional/wallet_pruning.py:40 in c594dbc756 outdated
    17+    OP_TRUE,
    18+)
    19+
    20+class WalletPruningTest(BitcoinTestFramework):
    21+    nTime = 0
    22+    def mine_large_blocks(self, node, n):
    


    kouloumos commented at 11:31 am on November 25, 2022:
    From what I’ve seen, custom functions are often declared after the setup override functions.

    aureleoules commented at 10:12 am on November 27, 2022:
    Thanks, I moved it.
  118. kouloumos commented at 1:56 pm on November 25, 2022: contributor

    The functional test got me really confused on what it is testing and how it is testing it. It might be because I am re-reviewing this after 55 days, but I think that some simplification (left comments inline) as well as commenting on the basis of the change and not on what the code does could help with that.

    What the test does (afaiu) is simulating 2 different wallets with 2 different birth heights wallet_birthheight_1 < wallet_birthheight_2 It then prunes below birthheight_2 to:

    • check with test_wallet_import_pruned_test that wallet_birthheight_2 is imported correctly.
    • check with test_wallet_import_pruned_with_missing_blocks that wallet_birthheight_1 cannot imported correctly.

    Based on the test overview above and some comments that I left inline, I did some implementation-logic restructuring and added a few comments that I think can help with comprehension. You can find my changes at 69b9b6247305d06168f58e9e156807e09a925bf6 (diff), feel free to pick whatever you like.

  119. aureleoules force-pushed on Nov 27, 2022
  120. aureleoules force-pushed on Nov 27, 2022
  121. aureleoules commented at 10:16 am on November 27, 2022: member
    Thanks for your detailed review @kouloumos. I cherry-picked your commit and squashed it. I also addressed some of your additional suggestions.
  122. kouloumos commented at 10:54 am on November 28, 2022: contributor
    ACK 8e87433a4460088d15275623f0b6f1893af28d82 nit: The MIN_BLOCKS_TO_KEEP constant is not used anymore in the test, which makes 8e87433a4460088d15275623f0b6f1893af28d82 out of scope.
  123. maflcko referenced this in commit 3be21060d6 on Nov 30, 2022
  124. maflcko commented at 4:37 pm on December 7, 2022: member
    This is adding a new test wallet_pruning.py, but I can’t find it in the CI output at all. Does anyone see what is going on here?
  125. fanquake commented at 4:45 pm on December 7, 2022: member
    Is it becuase it’s in EXTENDED_SCRIPTS? That is the job that’s failing, and didn’t make it to the functional tests: https://github.com/bitcoin/bitcoin/pull/24865/checks?check_run_id=9728080972.
  126. kouloumos commented at 4:48 pm on December 7, 2022: contributor
    It also runs as part of Win64 native [vs2022]. I can see that the extended tests run there and wallet_pruning.py runs successfully.
  127. maflcko commented at 4:50 pm on December 7, 2022: member

    Ah thx. Maybe there should be a skip_if_not_running_extended :sweat_smile: ?

    Also, I think this needs a rebase, see upcoming CI failure in https://cirrus-ci.com/task/6513158118965248

  128. aureleoules force-pushed on Dec 7, 2022
  129. rpc: Enable wallet import on pruned nodes
    Co-authored-by: João Barbosa <joao.paulo.barbosa@gmail.com>
    Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
    e6906fcf9e
  130. aureleoules force-pushed on Dec 8, 2022
  131. aureleoules commented at 12:53 pm on December 8, 2022: member
    Rebased to address silent conflicts.
  132. achow101 commented at 10:10 pm on December 14, 2022: member
    ACK 63493d778f443554059d77bd8c9b1a926685a892
  133. in test/functional/wallet_pruning.py:25 in 63493d778f outdated
    20+    OP_TRUE,
    21+)
    22+
    23+class WalletPruningTest(BitcoinTestFramework):
    24+    def add_options(self, parser):
    25+        self.add_wallet_options(parser)
    


    kouloumos commented at 7:35 am on December 15, 2022:

    If this is a legacy wallet only test, this should be

    0        self.add_wallet_options(parser, descriptors=False)
    
  134. in test/functional/wallet_pruning.py:119 in 63493d778f outdated
    114+        if (unload):
    115+            self.nodes[0].unloadwallet(wallet_name)
    116+
    117+    def run_test(self):
    118+        self.nTime = 0
    119+        self.log.info("Warning! This test requires ~1.1GB of disk space")
    


    kouloumos commented at 7:37 am on December 15, 2022:
    nit I haven’t calculate it properly, but with some quick monitoring during the test run, it seems that this might be closer to 1.3GB
  135. test: Wallet imports on pruned nodes
    Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
    Co-authored-by: Andreas Kouloumos <kouloumosa@gmail.com>
    71d9a7c03b
  136. test: Introduce MIN_BLOCKS_TO_KEEP constant 564b580bf0
  137. aureleoules force-pushed on Dec 15, 2022
  138. aureleoules commented at 8:54 am on December 15, 2022: member
    Thanks @kouloumos I addressed your comments.
  139. aureleoules requested review from ryanofsky on Dec 15, 2022
  140. aureleoules removed review request from ryanofsky on Dec 15, 2022
  141. aureleoules requested review from w0xlt on Dec 15, 2022
  142. aureleoules removed review request from w0xlt on Dec 15, 2022
  143. aureleoules requested review from furszy on Dec 15, 2022
  144. aureleoules removed review request from furszy on Dec 15, 2022
  145. aureleoules requested review from kouloumos on Dec 15, 2022
  146. aureleoules removed review request from kouloumos on Dec 15, 2022
  147. aureleoules requested review from achow101 on Dec 15, 2022
  148. kouloumos commented at 8:58 am on December 15, 2022: contributor
    ACK 564b580bf07742483a140c7c095b896a6d5d6cad
  149. in src/wallet/rpc/backup.cpp:104 in e6906fcf9e outdated
     99+    if (!chain.havePruned()) {
    100+        return;
    101+    }
    102+
    103+    int height{0};
    104+    const bool found{chain.findFirstBlockWithTimeAndHeight(timestamp - TIMESTAMP_WINDOW, 0, FoundBlock().height(height))};
    


    furszy commented at 6:54 pm on December 15, 2022:

    In e6906fcf:

    tiny nit: (only if you have to re-touch)

    0    const bool found{chain.findFirstBlockWithTimeAndHeight(timestamp - TIMESTAMP_WINDOW, /*min_height=*/ 0, FoundBlock().height(height))};
    
  150. in test/functional/test_framework/blocktools.py:142 in 71d9a7c03b outdated
    137@@ -138,6 +138,8 @@ def create_coinbase(height, pubkey=None, extra_output_script=None, fees=0, nValu
    138         coinbaseoutput.nValue += fees
    139     if pubkey is not None:
    140         coinbaseoutput.scriptPubKey = key_to_p2pk_script(pubkey)
    141+    elif script_pubkey is not None:
    142+        coinbaseoutput.scriptPubKey = script_pubkey
    


    furszy commented at 7:06 pm on December 15, 2022:

    In 71d9a7c0:

    atm is just a non-blocking nit:

    it’s weird to add this new ´script_pubkey´ arg that is only set if the ´pubkey´ arg is not provided. It is not stated anywhere.

    Ideally, should unify both into a single arg. So there is no confusion over which of them take precedence over the other.

  151. in test/functional/wallet_pruning.py:61 in 71d9a7c03b outdated
    56+
    57+            # Simulate 10 minutes of work time per block
    58+            # Important for matching a timestamp with a block +- some window
    59+            self.nTime += 600
    60+            for n in self.nodes:
    61+                if n.running:
    


    furszy commented at 7:09 pm on December 15, 2022:

    aureleoules commented at 9:35 am on December 16, 2022:
    Right, in a previous version of the test it could crash without this check.
  152. furszy approved
  153. furszy commented at 7:11 pm on December 15, 2022: member
    ACK 564b580 left few non-blocking nits.
  154. w0xlt approved
  155. achow101 commented at 10:19 pm on December 16, 2022: member
    reACK 564b580bf07742483a140c7c095b896a6d5d6cad
  156. achow101 merged this on Dec 16, 2022
  157. achow101 closed this on Dec 16, 2022

  158. sidhujag referenced this in commit 59bb1c8365 on Dec 17, 2022
  159. in test/functional/wallet_pruning.py:48 in 564b580bf0
    43+        height = int(best_block["height"]) + 1
    44+        self.nTime = max(self.nTime, int(best_block["time"])) + 1
    45+        previousblockhash = int(best_block["hash"], 16)
    46+        big_script = CScript([OP_RETURN] + [OP_TRUE] * 950000)
    47+        for _ in range(n):
    48+            block = create_block(hashprev=previousblockhash, ntime=self.nTime, coinbase=create_coinbase(height, script_pubkey=big_script))
    


    maflcko commented at 5:06 pm on December 19, 2022:

    nit: It is possible to do this without blocktools. See

     0diff --git a/test/functional/wallet_pruning.py b/test/functional/wallet_pruning.py
     1index 6d8475ce8d..d768130776 100755
     2--- a/test/functional/wallet_pruning.py
     3+++ b/test/functional/wallet_pruning.py
     4@@ -40,20 +40,10 @@ class WalletPruningTest(BitcoinTestFramework):
     5     def mine_large_blocks(self, node, n):
     6         # Get the block parameters for the first block
     7         best_block = node.getblock(node.getbestblockhash())
     8-        height = int(best_block["height"]) + 1
     9         self.nTime = max(self.nTime, int(best_block["time"])) + 1
    10-        previousblockhash = int(best_block["hash"], 16)
    11-        big_script = CScript([OP_RETURN] + [OP_TRUE] * 950000)
    12+        big_script = "raw({})".format(bytes([OP_RETURN] + [OP_TRUE] * 950000).hex())
    13         for _ in range(n):
    14-            block = create_block(hashprev=previousblockhash, ntime=self.nTime, coinbase=create_coinbase(height, script_pubkey=big_script))
    15-            block.solve()
    16-
    17-            # Submit to the node
    18-            node.submitblock(block.serialize().hex())
    19-
    20-            previousblockhash = block.sha256
    21-            height += 1
    22-
    23+            self.generateblock(node, big_script, [])
    24             # Simulate 10 minutes of work time per block
    25             # Important for matching a timestamp with a block +- some window
    26             self.nTime += 600
    

    Also, might be good to double check the runtime in light of https://github.com/bitcoin/bitcoin/commit/6c872d5e656a7117bbdf19a0220572b93de16f31

  160. aureleoules deleted the branch on Jan 12, 2023
  161. bitcoin locked this on Jan 12, 2024

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-07-05 19:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me