rpc: Enable wallet import on pruned nodes and add test #24865

pull aureleoules wants to merge 3 commits into bitcoin:master from aureleoules:2022-04-importwallet-pruned changing 5 files +189 −31
  1. aureleoules commented at 3:38 PM on April 15, 2022: member

    Reopens #16037

    I have rebased the PR, addressed the comments of the original PR and added a functional test.

    Before this change importwallet fails if any block is pruned. This PR makes it possible to importwallet if all required blocks aren't pruned. This is possible because the dump format includes key timestamps.

    For reviewers: python test/functional/wallet_pruning.py --nocleanup will generate a large blockchain (~700MB) that can be used to manually test wallet imports on a pruned node. Node0 is not pruned, while node1 is.

  2. DrahtBot added the label RPC/REST/ZMQ on Apr 15, 2022
  3. DrahtBot added the label Wallet on Apr 15, 2022
  4. in test/functional/wallet_pruning.py:13 in 35432901b4 outdated
       8 | +
       9 | +from test_framework.util import (
      10 | +    assert_raises_rpc_error,
      11 | +)
      12 | +
      13 | +from test_framework.blocktools import (create_coinbase)
    


    vincenzopalazzo commented at 9:13 PM on April 15, 2022:
    from test_framework.blocktools import create_coinbase
    
  5. in test/functional/wallet_pruning.py:23 in 35432901b4 outdated
      20 | +from test_framework.wallet_util import (
      21 | +    get_key,
      22 | +)
      23 | +from test_framework.test_framework import BitcoinTestFramework
      24 | +
      25 | +class WalletPruningTest(BitcoinTestFramework):
    


    vincenzopalazzo commented at 9:14 PM on April 15, 2022:
    
    
    class WalletPruningTest(BitcoinTestFramework):
    

    Usualy in python there are two spaces between class and the last import declaration

  6. vincenzopalazzo changes_requested
  7. DrahtBot commented at 6:33 AM on April 16, 2022: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK kouloumos, furszy, w0xlt, achow101
    Concept ACK Xekyo, theStack
    Stale ACK ryanofsky

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    No conflicts as of last run.

  8. aureleoules force-pushed on Apr 16, 2022
  9. DrahtBot added the label Needs rebase on Apr 26, 2022
  10. aureleoules force-pushed on Apr 26, 2022
  11. DrahtBot removed the label Needs rebase on Apr 26, 2022
  12. aureleoules force-pushed on May 14, 2022
  13. aureleoules force-pushed on May 14, 2022
  14. in test/functional/wallet_pruning.py:23 in 46b7f731b1 outdated
      19 | +
      20 | +
      21 | +class WalletPruningTest(BitcoinTestFramework):
      22 | +    nTime = 0
      23 | +    def mine_large_blocks(self, node, n):
      24 | +        # Taken from test/functional/feature_pruning.py
    


    murchandamus commented at 3:47 PM on May 31, 2022:

    If we're reusing helper functions, perhaps we should extract them to a single place in a follow-up PR.

  15. in test/functional/wallet_pruning.py:108 in 46b7f731b1 outdated
     131 | +        self.nodes[1].dumpwallet(os.path.join(self.nodes[1].datadir, wname))
     132 | +
     133 | +        # mine large blocks, blocks will be pruned automatically
     134 | +        self.mine_large_blocks(self.nodes[0], 150)
     135 | +        self.sync_all(self.nodes)
     136 | +
    


    murchandamus commented at 3:59 PM on May 31, 2022:

    This looks good to me, but perhaps it would be more comprehensive to learn the birthheight or first transaction's height from the wallet, and check/assert here that the corresponding block has been pruned.

  16. murchandamus commented at 4:00 PM on May 31, 2022: contributor

    Did a light code-review, concept ACK

  17. in src/wallet/rpc/backup.cpp:1392 in 46b7f731b1 outdated
    1387 | @@ -1383,6 +1388,8 @@ RPCHelpMan importmulti()
    1388 |                  nLowestTimestamp = timestamp;
    1389 |              }
    1390 |          }
    1391 | +
    1392 | +        EnsureBlockDataFromTime(pwallet->chain(), nLowestTimestamp);
    


    achow101 commented at 9:06 PM on June 20, 2022:

    In 46b7f731b1e98b0836c5b97cd99c0887d6173a17 "rpc: Enable wallet import on pruned nodes and add test"

    We don't want to do this for importmulti. importmulti can have some imports succeed, and others fail. Just because one import has an out of range timestamp does not mean the entire import should fail the RPC. This check would result in a JSONRPCError which is not what we expect from importmulti - rather there should be an error per import that has a too old timestamp.

    Furthermore, we already handle the pruned node case. If the rescan reports a time that is more recent than the timestamp for a given import, we will give an error for that import with more details about what happened, maybe why it happened, and how the user can resolve it.

  18. aureleoules force-pushed on Jul 5, 2022
  19. aureleoules force-pushed on Jul 5, 2022
  20. aureleoules force-pushed on Jul 5, 2022
  21. aureleoules force-pushed on Jul 5, 2022
  22. aureleoules force-pushed on Jul 5, 2022
  23. aureleoules force-pushed on Jul 5, 2022
  24. aureleoules force-pushed on Jul 5, 2022
  25. aureleoules force-pushed on Jul 5, 2022
  26. in test/functional/wallet_pruning.py:54 in b45e54bf51 outdated
      49 | +        self.skip_if_no_wallet()
      50 | +
      51 | +    def setup_network(self):
      52 | +        self.setup_nodes()
      53 | +        self.connect_nodes(0, 1)
      54 | +        self.sync_all(self.nodes)
    


    kouloumos commented at 4:22 PM on July 12, 2022:

    Do you actually need this? The only difference in behaviour that I see from the default https://github.com/bitcoin/bitcoin/blob/1d89fc695a3aeb3e3dcadf371b7667572b38c836/test/functional/test_framework/test_framework.py#L387 is the 0 -> 1 instead of 1 -> 0 nodes' connection which seems to make no difference for the test.

  27. in test/functional/wallet_pruning.py:59 in b45e54bf51 outdated
      54 | +        self.sync_all(self.nodes)
      55 | +
      56 | +    def create_big_chain(self):
      57 | +        self.log.info("Generating a long chain of blocks...")
      58 | +        # Generate 288 light blocks, the minimum required to stay on disk with prune enabled
      59 | +        self.generate(self.nodes[0], 288, sync_fun=lambda: self.sync_all(self.nodes))
    


    kouloumos commented at 4:28 PM on July 12, 2022:

    I think that if sync_fun is not defined you will still have the desired behaviour here

  28. in test/functional/wallet_pruning.py:63 in b45e54bf51 outdated
      58 | +        # Generate 288 light blocks, the minimum required to stay on disk with prune enabled
      59 | +        self.generate(self.nodes[0], 288, sync_fun=lambda: self.sync_all(self.nodes))
      60 | +
      61 | +        # Generate large blocks to make sure we have enough to test chain pruning
      62 | +        self.mine_large_blocks(self.nodes[0], 610)
      63 | +        self.sync_all(self.nodes)
    


  29. in test/functional/wallet_pruning.py:77 in b45e54bf51 outdated
      72 | +        # import wallet
      73 | +        self.nodes[1].importwallet(os.path.join(self.nodes[1].datadir, wname))
      74 | +
      75 | +        # mine some blocks, pruning should not have removed the block
      76 | +        self.mine_large_blocks(self.nodes[0], 5)
      77 | +        self.sync_all(self.nodes)
    


    kouloumos commented at 4:39 PM on July 12, 2022:

    nit: for how it is used in this test, I think that sync_all could be part of the mine_large_blocks method

  30. in test/functional/wallet_pruning.py:112 in b45e54bf51 outdated
     107 | +        self.log.info("Warning! This test requires ~1.5GB of disk space")
     108 | +
     109 | +        self.create_big_chain()
     110 | +
     111 | +        self.wallet_import_pruned_test()
     112 | +        self.wallet_import_pruned_missing_blocks()
    


    kouloumos commented at 9:04 AM on July 13, 2022:

    Maybe it would be better to include a test prefix as in most tests? e.g test_wallet_import_on_pruned_node{_with_missing_blocks}

  31. in test/functional/wallet_pruning.py:65 in b45e54bf51 outdated
      60 | +
      61 | +        # Generate large blocks to make sure we have enough to test chain pruning
      62 | +        self.mine_large_blocks(self.nodes[0], 610)
      63 | +        self.sync_all(self.nodes)
      64 | +
      65 | +    def wallet_import_pruned_test(self):
    


    kouloumos commented at 10:06 AM on July 13, 2022:

    I think that this test is not testing what you are expecting it to test, or its scope is just not clear to me.

    Even if I revert your changes to backup.cpp (git checkout 9fb2a2bc6768ab03fcada9155d52a16ce6f6a0cc -- src/wallet/rpc/backup.cpp) this test passes. My understanding is that this never reaches the prune state because no blocks are pruned until we are above PruneAfterHeight https://github.com/bitcoin/bitcoin/blob/1d89fc695a3aeb3e3dcadf371b7667572b38c836/src/chainparams.cpp#L429 The logic in feature_pruning.py also helped me to understand this behaviour.

  32. kouloumos commented at 2:21 PM on July 13, 2022: contributor

    I think that wallet_pruning.py is not testing properly this added feature. This is due to the testing logic but mainly because of the default setup of the testing framework that is being used in this test.

    This is possible because the dump format includes key timestamps.

    The importwallet rpc calculates the min nTimeBegin timestamp for which findFirstBlockWithTimeAndHeight in the newly created function EnsureBlockDataFromTime will find the earliest block with timestamp equal or greater to.

    As you can verify from the wallet_*.dat dumps of this test

    # node1 wallet dump
    ...
    cUxsWyKyZ9MAQTaAhUQWJmBbSvHMwSmuv59KgxQV7oZQU3PXN3KE 1970-01-01T00:00:01Z label=coinbase # addr=msX6jQXvxiNhx3Q62PKeLPrhrqZQdSimTg,2MtBq31Chq2pntJ81PQLR4q4sMEG7K46wsQ,bcrt1qsw5g6ehh439vurfyhdh93d66hw0kf9085wy7ma
    ...
    

    the deterministic private keys that are imported to each test node as part of the testing framework setup https://github.com/bitcoin/bitcoin/blob/1d89fc695a3aeb3e3dcadf371b7667572b38c836/test/functional/test_framework/test_framework.py#L441

    have a 1970-01-01T00:00:01Z timestamp which means that EnsureBlockDataFromTime will always check for the existence of all blocks between genesis and the current height.

    Even if that was not the case, because the nodes' wallets are initialized before we start generating blocks, the timestamp in the wallet dump is still before the first created block thus EnsureBlockDataFromTime checks for the existence of all blocks between the first block and the current height.

    This behaviour will always result to missing blocks as soon as we start pruning. The reason that the first part of the test (wallet_import_pruned_test) is not failing is because it never reaches the prune state (height < PruntAfterHeight).

    I believe that you can better test for the new feature by overriding setup_nodes and initializing node1 wallet mid-test

  33. aureleoules force-pushed on Jul 19, 2022
  34. aureleoules force-pushed on Jul 19, 2022
  35. aureleoules commented at 7:25 AM on July 20, 2022: member

    Thank you for your thorough review @kouloumos ! I believe I have addressed all of your comments.

  36. in src/wallet/rpc/backup.cpp:106 in 45435bc688 outdated
     101 | +    int height{0};
     102 | +    const bool found{chain.findFirstBlockWithTimeAndHeight(timestamp - TIMESTAMP_WINDOW, 0, FoundBlock().height(height))};
     103 | +    const std::optional<int> chainHeight{chain.getHeight()};
     104 | +
     105 | +    if (found && chainHeight && !chain.hasBlocks(chain.getBlockHash(chainHeight.value()), height)) {
     106 | +        throw JSONRPCError(RPC_WALLET_ERROR, "Pruned blocks required to import keys");
    


    kouloumos commented at 9:37 AM on September 1, 2022:

    I think that at this point we have the required height for which we need blocks. Could this change to a more descriptive error message stating that required weight? And maybe "[...]. Use RPC call getblockchaininfo to determine your pruned height" as here?


    aureleoules commented at 10:11 AM on September 5, 2022:

    thanks, added it

  37. aureleoules force-pushed on Sep 5, 2022
  38. aureleoules force-pushed on Sep 6, 2022
  39. in src/wallet/rpc/backup.cpp:105 in 0aedf6f3cb outdated
     100 | +
     101 | +    int height{0};
     102 | +    const bool found{chain.findFirstBlockWithTimeAndHeight(timestamp - TIMESTAMP_WINDOW, 0, FoundBlock().height(height))};
     103 | +    const std::optional<int> chainHeight{chain.getHeight()};
     104 | +
     105 | +    if (found && chainHeight && !chain.hasBlocks(chain.getBlockHash(chainHeight.value()), height)) {
    


    ryanofsky commented at 5:42 PM on September 7, 2022:

    In commit "rpc: Enable wallet import on pruned nodes and add test" (0aedf6f3cb5a513f9e42458362261d4afa2ea17f)

    I think it would be slightly better to pass in pwallet->m_last_block_processed instead of chain.getBlockHash(chainHeight.value()) as the first hasBlocks argument. In general to avoid race conditions wallet just tries to query node based on it's own chain tip and not rely on current node tip. To be more race free, could add a Chain::hasBlocksFromTime(int64_t start_time, uint256 end_block) method and pass m_last_block_processed as the end block. But just passing m_last_block_processed to hasBlocks here and dropping the chainHeight variable should already be an improvement.


    aureleoules commented at 11:24 PM on September 7, 2022:

    Thanks for the review, fixed it!

  40. in test/functional/feature_index_prune.py:33 in 0aedf6f3cb outdated
      29 | @@ -30,10 +30,10 @@ def sync_index(self, height):
      30 |          expected_stats = {
      31 |              'coinstatsindex': {'synced': True, 'best_block_height': height}
      32 |          }
      33 | -        self.wait_until(lambda: self.nodes[1].getindexinfo() == expected_stats)
      34 | +        self.wait_until(lambda: self.nodes[1].getindexinfo() == expected_stats, timeout=120)
    


    ryanofsky commented at 5:52 PM on September 7, 2022:

    In commit "rpc: Enable wallet import on pruned nodes and add test" (0aedf6f3cb5a513f9e42458362261d4afa2ea17f)

    Seems like an unrelated change? Would be good to drop this if possible, or maybe comment why overriding timeout is necessary.


    aureleoules commented at 11:23 PM on September 7, 2022:

    oops, fixed

  41. in test/functional/test_framework/blocktools.py:90 in 0aedf6f3cb outdated
      86 | @@ -87,6 +87,13 @@ def create_block(hashprev=None, coinbase=None, ntime=None, *, version=None, tmpl
      87 |      block.calc_sha256()
      88 |      return block
      89 |  
      90 | +def create_large_block(hashprev=None, ntime=None, height=None, scriptPubKey=CScript([OP_RETURN] + [OP_TRUE] * 950000)):
    


    ryanofsky commented at 5:59 PM on September 7, 2022:

    In commit "rpc: Enable wallet import on pruned nodes and add test" (0aedf6f3cb5a513f9e42458362261d4afa2ea17f)

    This CScript(...) value should just be generated locally inside the function, not be a python default parameter. Otherwise the cscript object is created and lives in memory even if this function is never called.

    Can just say scriptPubKey=None in parameter list and if scriptPubKey is None: scriptPubKey = CScript(...) inside the function. Or just make scriptPubKey a local variable since the argument is never specified anyway.


    aureleoules commented at 6:13 PM on September 7, 2022:

    This is a large object that takes time to construct so I thought making it static would avoid constructing it multiple times, but I had not thought of the fact that this lives in memory even if the function is unused. Will fix though, thanks.

  42. ryanofsky approved
  43. ryanofsky commented at 6:04 PM on September 7, 2022: contributor

    Code review ACK 0aedf6f3cb5a513f9e42458362261d4afa2ea17f. Nice change and tests!

  44. aureleoules force-pushed on Sep 7, 2022
  45. aureleoules force-pushed on Sep 7, 2022
  46. aureleoules force-pushed on Sep 8, 2022
  47. aureleoules force-pushed on Sep 8, 2022
  48. in src/wallet/rpc/backup.cpp:103 in f42c077a8f outdated
      98 | +    if (!chain.havePruned())
      99 | +        return;
     100 | +
     101 | +    int height{0};
     102 | +    const bool found{chain.findFirstBlockWithTimeAndHeight(timestamp - TIMESTAMP_WINDOW, 0, FoundBlock().height(height))};
     103 | +    const std::optional<int> chainHeight{chain.getHeight()};
    


    ryanofsky commented at 3:06 PM on September 8, 2022:

    In commit "rpc: Enable wallet import on pruned nodes and add test" (f42c077a8f948f5c93b68eb1b7fa246b4f6475ef)

    I think there's no need for the chainHeight variable anymore and it would be better to drop it.

  49. in test/functional/feature_pruning.py:31 in f42c077a8f outdated
      33 |  # Rescans start at the earliest block up to 2 hours before a key timestamp, so
      34 |  # the manual prune RPC avoids pruning blocks in the same window to be
      35 |  # compatible with pruning based on key creation time.
      36 |  TIMESTAMP_WINDOW = 2 * 60 * 60
      37 |  
      38 | +scriptPubKey = CScript([OP_RETURN] + [OP_TRUE] * 950000)
    


    ryanofsky commented at 3:10 PM on September 8, 2022:

    In commit "rpc: Enable wallet import on pruned nodes and add test" (f42c077a8f948f5c93b68eb1b7fa246b4f6475ef)

    Just IMO, but I think it would be better to keep the existing big_script variable below and delete this new variable (smaller diff, more descriptive variable name, more straightforward code avoiding global variables)

  50. in test/functional/test_framework/blocktools.py:90 in f42c077a8f outdated
      86 | @@ -87,6 +87,13 @@ def create_block(hashprev=None, coinbase=None, ntime=None, *, version=None, tmpl
      87 |      block.calc_sha256()
      88 |      return block
      89 |  
      90 | +def create_large_block(hashprev, ntime, height, scriptPubKey):
    


    ryanofsky commented at 3:32 PM on September 8, 2022:

    In commit "rpc: Enable wallet import on pruned nodes and add test" (f42c077a8f948f5c93b68eb1b7fa246b4f6475ef)

    This new create_large_block seems very inflexible and also has a misleading name. Would drop it and instead just add a new option to create_coinbase like:

    diff --git a/test/functional/feature_pruning.py b/test/functional/feature_pruning.py
    index 8d72d11baf5..d14970c9463 100755
    --- a/test/functional/feature_pruning.py
    +++ b/test/functional/feature_pruning.py
    @@ -10,7 +10,8 @@ This test takes 30 mins or more (up to 2 hours)
     """
     import os
     
    -from test_framework.blocktools import create_large_block
    +from test_framework.blocktools import create_block
    +from test_framework.blocktools import create_coinbase
     from test_framework.test_framework import BitcoinTestFramework
     from test_framework.util import (
         assert_equal,
    @@ -47,7 +48,7 @@ def mine_large_blocks(node, n):
         previousblockhash = int(best_block["hash"], 16)
     
         for _ in range(n):
    -        block = create_large_block(hashprev=previousblockhash, height=height, ntime=mine_large_blocks.nTime, scriptPubKey=scriptPubKey)
    +        block = create_block(hashprev=previousblockhash, ntime=mine_large_blocks.nTime, coinbase=create_coinbase(height, script_pubkey=scriptPubKey))
             block.solve()
     
             # Submit to the node
    diff --git a/test/functional/test_framework/blocktools.py b/test/functional/test_framework/blocktools.py
    index 5916ffc2b17..77653df96e9 100644
    --- a/test/functional/test_framework/blocktools.py
    +++ b/test/functional/test_framework/blocktools.py
    @@ -87,13 +87,6 @@ def create_block(hashprev=None, coinbase=None, ntime=None, *, version=None, tmpl
         block.calc_sha256()
         return block
     
    -def create_large_block(hashprev, ntime, height, scriptPubKey):
    -    coinbase_tx = create_coinbase(height)
    -    coinbase_tx.vout[0].scriptPubKey = scriptPubKey
    -    coinbase_tx.rehash()
    -
    -    return create_block(coinbase=coinbase_tx, ntime=ntime, hashprev=hashprev)
    -
     def get_witness_script(witness_root, witness_nonce):
         witness_commitment = uint256_from_str(hash256(ser_uint256(witness_root) + ser_uint256(witness_nonce)))
         output_data = WITNESS_COMMITMENT_HEADER + ser_uint256(witness_commitment)
    @@ -127,7 +120,7 @@ def script_BIP34_coinbase_height(height):
         return CScript([CScriptNum(height)])
     
     
    -def create_coinbase(height, pubkey=None, extra_output_script=None, fees=0, nValue=50):
    +def create_coinbase(height, pubkey=None, *, script_pubkey=None, extra_output_script=None, fees=0, nValue=50):
         """Create a coinbase transaction.
     
         If pubkey is passed in, the coinbase output will be a P2PK output;
    @@ -145,6 +138,8 @@ def create_coinbase(height, pubkey=None, extra_output_script=None, fees=0, nValu
             coinbaseoutput.nValue += fees
         if pubkey is not None:
             coinbaseoutput.scriptPubKey = key_to_p2pk_script(pubkey)
    +    elif script_pubkey is not None:
    +        coinbaseoutput.scriptPubKey = script_pubkey
         else:
             coinbaseoutput.scriptPubKey = CScript([OP_TRUE])
         coinbase.vout = [coinbaseoutput]
    diff --git a/test/functional/wallet_pruning.py b/test/functional/wallet_pruning.py
    index 9b8160a4490..e42994422e0 100755
    --- a/test/functional/wallet_pruning.py
    +++ b/test/functional/wallet_pruning.py
    @@ -7,7 +7,8 @@
     import os
     
     from test_framework.util import assert_raises_rpc_error
    -from test_framework.blocktools import create_large_block
    +from test_framework.blocktools import create_block
    +from test_framework.blocktools import create_coinbase
     from test_framework.test_framework import BitcoinTestFramework
     
     from test_framework.script import (
    @@ -26,7 +27,7 @@ class WalletPruningTest(BitcoinTestFramework):
             self.nTime = max(self.nTime, int(best_block["time"])) + 1
             previousblockhash = int(best_block["hash"], 16)
             for _ in range(n):
    -            block = create_large_block(hashprev=previousblockhash, height=height, ntime=self.nTime, scriptPubKey=self.scriptPubKey)
    +            block = create_block(hashprev=previousblockhash, ntime=self.nTime, coinbase=create_coinbase(height, script_pubkey=self.scriptPubKey))
                 block.solve()
     
                 # Submit to the node
    
    
  51. in src/wallet/rpc/backup.cpp:105 in f42c077a8f outdated
     100 | +
     101 | +    int height{0};
     102 | +    const bool found{chain.findFirstBlockWithTimeAndHeight(timestamp - TIMESTAMP_WINDOW, 0, FoundBlock().height(height))};
     103 | +    const std::optional<int> chainHeight{chain.getHeight()};
     104 | +
     105 | +    LOCK(wallet.cs_wallet);
    


    ryanofsky commented at 3:37 PM on September 8, 2022:

    In commit "rpc: Enable wallet import on pruned nodes and add test" (f42c077a8f948f5c93b68eb1b7fa246b4f6475ef)

    I don't think we need to keep the wallet locked while running all the code below. Would suggest using a shorter lock:

    uint256 tip_hash = WITH_LOCK(wallet.cs_wallet, return wallet.GetLastBlockHash());
    
  52. ryanofsky approved
  53. ryanofsky commented at 3:42 PM on September 8, 2022: contributor

    Code review ACK f42c077a8f948f5c93b68eb1b7fa246b4f6475ef. Just suggested changes since last review, and increasing sync_blocks timeout in the new test. Thanks for the updates!

  54. aureleoules force-pushed on Sep 8, 2022
  55. aureleoules commented at 6:25 PM on September 8, 2022: member

    Thanks for the review again @ryanofsky, I think I've addressed all your comments.

  56. aureleoules force-pushed on Sep 8, 2022
  57. in test/functional/feature_pruning.py:48 in 05463d4001 outdated
      40 | @@ -41,28 +41,14 @@ def mine_large_blocks(node, n):
      41 |          mine_large_blocks.nTime = 0
      42 |  
      43 |      # Get the block parameters for the first block
      44 | -    big_script = CScript([OP_RETURN] + [OP_NOP] * 950000)
      45 |      best_block = node.getblock(node.getbestblockhash())
      46 |      height = int(best_block["height"]) + 1
      47 |      mine_large_blocks.nTime = max(mine_large_blocks.nTime, int(best_block["time"])) + 1
      48 |      previousblockhash = int(best_block["hash"], 16)
      49 | +    bigScript = CScript([OP_RETURN] + [OP_TRUE] * 950000)
    


    ryanofsky commented at 6:38 PM on September 12, 2022:

    In commit "rpc: Enable wallet import on pruned nodes and add test" (05463d4001001213c17672c1607a3a493927ddd7)

    Can just keep existing big_script variable, no need to replace it with new bigScript variable. This would make the diff small and make the naming convention more convenient


    aureleoules commented at 10:02 AM on September 13, 2022:

    my bad, fixed

  58. ryanofsky approved
  59. ryanofsky commented at 6:39 PM on September 12, 2022: contributor

    Code review ACK 05463d4001001213c17672c1607a3a493927ddd7. Only changes since last review were some suggested cleanups & simplifications

  60. aureleoules force-pushed on Sep 13, 2022
  61. aureleoules force-pushed on Sep 13, 2022
  62. ryanofsky approved
  63. ryanofsky commented at 5:22 PM on September 13, 2022: contributor

    Code review ACK 6dc2e7141de83022f0ca0c48afd5ea7e5c4684d0. Only change since last review is changing bigScript back to big_script

  64. w0xlt approved
  65. aureleoules commented at 12:58 PM on September 16, 2022: member

    Fixes #9409.

  66. in test/functional/wallet_pruning.py:119 in 6dc2e7141d outdated
     114 | +    def run_test(self):
     115 | +        self.log.info("Warning! This test requires ~1.5GB of disk space")
     116 | +
     117 | +        self.create_big_chain()
     118 | +
     119 | +        # connect mid-test node1
    


    kouloumos commented at 1:05 PM on September 30, 2022:

    nit: maybe add some comments here to explain the rationale behind this mid-test connection? I think it will not be very clear for someone reading this test in the future.

  67. in test/functional/wallet_pruning.py:81 in 6dc2e7141d outdated
      76 | +        wname = "wallet_pruned.dat"
      77 | +
      78 | +        # export wallet
      79 | +        self.nodes[1].dumpwallet(os.path.join(self.nodes[1].datadir, wname))
      80 | +        # import wallet
      81 | +        self.nodes[1].importwallet(os.path.join(self.nodes[1].datadir, wname))
    


    kouloumos commented at 1:34 PM on September 30, 2022:

    This dump&import pattern in the same wallet might be ok for checking against the RPC assertion of the newly introduced logic, but in reality the keys and scripts in the dump file are not imported in the wallet because they already exist.

    I don't consider this as an issue because that behavior is also tested elsewhere(wallet_dump.py) but I think changing this part makes for a better usage scenario.

  68. kouloumos commented at 1:34 PM on September 30, 2022: contributor

    reACK 6dc2e7141de83022f0ca0c48afd5ea7e5c4684d0

    I've also verified the expected behavior on a pruned signet node by dumping the wallet and then manually changing the timestamps in the file to simulate older keys for which the block data do not exist (pruned).

    Does it make sense to also add a check (in the functional tests), that after restoring, the wallet correctly accounts for balances and maybe addresses? Or that's redundant because a similar check exists at wallet_dump.py? https://github.com/bitcoin/bitcoin/blob/33eef562a321ce772a9a88073a78a85894cb3fe8/test/functional/wallet_dump.py#L198-L206

  69. aureleoules force-pushed on Oct 2, 2022
  70. aureleoules commented at 4:22 PM on October 2, 2022: member

    Thank you @kouloumos for your review. I have reworked the test to take your comments into account and also reduce the time it takes to run. It took 1min24 to run and reduced it to 54 seconds (on my machine). Also added more comments to explain each step of the test and split the PR into two commits.

  71. aureleoules force-pushed on Oct 2, 2022
  72. aureleoules force-pushed on Oct 2, 2022
  73. aureleoules commented at 4:51 PM on October 2, 2022: member

    Added a test case to check that the wallet balance is correct.

  74. andrewtoth commented at 1:44 PM on October 19, 2022: contributor

    Sorry, I didn't check that #26206 will actually conflict with this PR. This needs a rebase on top of it and then revert that commit in this PR.

  75. aureleoules force-pushed on Oct 19, 2022
  76. aureleoules force-pushed on Oct 19, 2022
  77. aureleoules commented at 2:53 PM on October 19, 2022: member

    Rebased, reverted 4aff7a48a4e0f1075306f181a276b8a74c857022 and added @ryanofsky and @kouloumos as co-authors.

  78. in src/wallet/rpc/backup.cpp:96 in 661e45300a outdated
      92 | @@ -93,6 +93,20 @@ static void RescanWallet(CWallet& wallet, const WalletRescanReserver& reserver,
      93 |      }
      94 |  }
      95 |  
      96 | +static void EnsureBlockDataFromTime(interfaces::Chain& chain, const CWallet& wallet, int64_t timestamp)
    


    andrewtoth commented at 2:46 AM on October 20, 2022:

    Why create a function for this? Would this be easier to reason about if it was inline?


    aureleoules commented at 7:53 AM on October 20, 2022:

    I've seen many instances of Ensure(.*) functions and it also helps to reduce the length of the parent function I suppose.

  79. kouloumos commented at 5:20 PM on October 24, 2022: contributor

    Does this need a rebase after #26380?

  80. aureleoules closed this on Oct 24, 2022

  81. aureleoules reopened this on Oct 24, 2022

  82. in src/wallet/rpc/backup.cpp:98 in 661e45300a outdated
      92 | @@ -93,6 +93,20 @@ static void RescanWallet(CWallet& wallet, const WalletRescanReserver& reserver,
      93 |      }
      94 |  }
      95 |  
      96 | +static void EnsureBlockDataFromTime(interfaces::Chain& chain, const CWallet& wallet, int64_t timestamp)
      97 | +{
      98 | +    if (!chain.havePruned())
    


    maflcko commented at 7:20 AM on October 25, 2022:
        if (!chain.havePruned()) {
    

    nit (for new code)

  83. aureleoules force-pushed on Oct 25, 2022
  84. aureleoules force-pushed on Oct 25, 2022
  85. aureleoules commented at 8:40 AM on October 25, 2022: member

    Dropped e1eadaa72d6831d1d0a53ba97c215dc4cdb64436, rebased for #26380 and addressed MarcoFalke's nit.

  86. in test/functional/wallet_pruning.py:126 in 6abd81b11d outdated
     121 | +        self.mine_large_blocks(self.nodes[0], 128)
     122 | +
     123 | +        # Generate 50MB worth of large blocks in the second blk*.dat file
     124 | +        self.mine_large_blocks(self.nodes[0], 50)
     125 | +        # Create a wallet and a backup of it which birth's block is in the second blk*.dat file
     126 | +        self.nodes[0].createwallet(wallet_name="wallet_init", descriptors=self.options.descriptors, load_on_startup=True)
    


    achow101 commented at 6:42 PM on October 31, 2022:

    In 6abd81b11dd1a45bdffcc85ba90591474a3f48b0 "test: Wallet imports on pruned nodes"

    Since this is testing legacy wallet only RPCs, we should not be looking at the --legacy-wallet/--descriptors options to determine the wallet type to create, for all wallets created in this test.

            self.nodes[0].createwallet(wallet_name="wallet_init", descriptors=False, load_on_startup=True)
    
  87. in test/functional/test_runner.py:88 in 6abd81b11d outdated
      82 | @@ -83,6 +83,7 @@
      83 |      'feature_pruning.py',
      84 |      'feature_dbcrash.py',
      85 |      'feature_index_prune.py',
      86 | +    'wallet_pruning.py',
    


    achow101 commented at 6:48 PM on October 31, 2022:

    In 6abd81b11dd1a45bdffcc85ba90591474a3f48b0 "test: Wallet imports on pruned nodes"

    Should have --legacy-wallet to indicate that this test is a legacy wallet only test. Also, the default is --descriptors.

        'wallet_pruning.py --legacy-wallet',
    
  88. in test/functional/feature_pruning.py:14 in 6abd81b11d outdated
       9 | @@ -10,8 +10,8 @@
      10 |  """
      11 |  import os
      12 |  
      13 | +from test_framework.blocktools import create_block
      14 |  from test_framework.blocktools import create_coinbase
    


    w0xlt commented at 4:54 AM on November 1, 2022:
    from test_framework.blocktools import (
        create_block,
        create_coinbase,
    )
    
  89. w0xlt approved
  90. aureleoules force-pushed on Nov 1, 2022
  91. aureleoules commented at 11:32 AM on November 1, 2022: member

    Addressed the suggestions, thanks @achow101 @w0xlt.

  92. in test/functional/wallet_pruning.py:71 in a0762cf6ec outdated
      66 | +        self.log.info("Make sure we can import wallet when pruned and required blocks are still available")
      67 | +
      68 | +        wname = "wallet_pruned.dat"
      69 | +
      70 | +        # Export wallet from node 0
      71 | +        self.nodes[0].createwallet(wallet_name="wallet_pruned", descriptors=self.options.descriptors, load_on_startup=True)
    


    achow101 commented at 7:28 PM on November 1, 2022:

    In a0762cf6ec0bbc5eb770912a1ab917862f5515ca "test: Wallet imports on pruned nodes"

            self.nodes[0].createwallet(wallet_name="wallet_pruned", descriptors=False, load_on_startup=True)
    

    aureleoules commented at 10:54 AM on November 2, 2022:

    Oops, pushed.

  93. in test/functional/wallet_pruning.py:79 in a0762cf6ec outdated
      74 | +        # Add mature coins to the wallet
      75 | +        self.generatetoaddress(self.nodes[0], 1, self.nodes[0].getnewaddress(), sync_fun=lambda: None)
      76 | +        self.generate(self.nodes[0], 100)
      77 | +
      78 | +        # Import wallet into node 1
      79 | +        self.nodes[1].createwallet(wallet_name="wallet_pruned", descriptors=self.options.descriptors, load_on_startup=True)
    


    achow101 commented at 7:28 PM on November 1, 2022:

    In a0762cf6ec0bbc5eb770912a1ab917862f5515ca "test: Wallet imports on pruned nodes"

            self.nodes[1].createwallet(wallet_name="wallet_pruned", descriptors=False, load_on_startup=True)
    
  94. aureleoules force-pushed on Nov 2, 2022
  95. andrewtoth commented at 3:29 PM on November 2, 2022: contributor

    You should rebase to fix ci test failure for p2p_sendtxrcncl.py.

  96. aureleoules force-pushed on Nov 2, 2022
  97. aureleoules commented at 3:53 PM on November 2, 2022: member

    You should rebase to fix ci test failure for p2p_sendtxrcncl.

    :heavy_check_mark:

  98. in test/functional/wallet_pruning.py:96 in c46f5fe58e outdated
      91 | +        self.log.info("Wallet successfully imported on pruned node")
      92 | +
      93 | +    def test_wallet_import_pruned_with_missing_blocks(self):
      94 | +        self.log.info("Make sure we cannot import wallet when pruned and required blocks are not available")
      95 | +
      96 | +        wname = "wallet_init.dat"
    


    furszy commented at 3:03 PM on November 4, 2022:

    In c46f5fe5:

    Tiny nit, this could be passed as an argument so we avoid hardcoding file names more than once.

  99. furszy approved
  100. furszy commented at 3:04 PM on November 4, 2022: member

    Code ACK c46f5fe5

  101. in test/functional/wallet_pruning.py:120 in c46f5fe58e outdated
     114 | +
     115 | +        self.log.info("Generate a long chain of blocks...")
     116 | +
     117 | +        # A blk*.dat file is 128MB
     118 | +        # Generate 288 light blocks, the minimum required to stay on disk with prune enabled
     119 | +        self.generate(self.nodes[0], 288, sync_fun=lambda: None)
    


    theStack commented at 3:24 PM on November 10, 2022:

    nit: not directly related to this PR, but as a follow-up I think it would make sense to introduce the constant MIN_BLOCKS_TO_KEEP into the functional test framework, in order to the replace this magic number here and in other tests (e.g. feature_pruning.py)

  102. in test/functional/wallet_pruning.py:63 in c46f5fe58e outdated
      58 | +
      59 | +    def setup_network(self):
      60 | +        self.setup_nodes()
      61 | +
      62 | +    def skip_test_if_missing_module(self):
      63 | +        self.skip_if_no_wallet()
    


    theStack commented at 3:32 PM on November 10, 2022:

    Should ensure here that the test is only run if legacy wallet support (BDB) is compiled in:

            self.skip_if_no_wallet()
            self.skip_if_no_bdb()
    
  103. theStack commented at 3:37 PM on November 10, 2022: contributor

    Concept ACK

  104. aureleoules force-pushed on Nov 11, 2022
  105. aureleoules force-pushed on Nov 11, 2022
  106. aureleoules commented at 10:59 AM on November 11, 2022: member
  107. aureleoules force-pushed on Nov 16, 2022
  108. aureleoules commented at 12:09 PM on November 16, 2022: member

    Reduced the number of blocks generated to speed up the test (from 1:40 to 1:24).

  109. in src/wallet/rpc/backup.cpp:587 in c594dbc756 outdated
     582 | +                if (birth_time > 0) nTimeBegin = std::min(nTimeBegin, birth_time);
     583 |                  scripts.push_back(std::pair<CScript, int64_t>(script, birth_time));
     584 |              }
     585 |          }
     586 |          file.close();
     587 | +        EnsureBlockDataFromTime(pwallet->chain(), *pwallet, nTimeBegin);
    


    kouloumos commented at 3:26 PM on November 24, 2022:

    Is there a reason that you are passing the chain interface as a separate argument instead of accessing it inside the function with wallet.chain().*?


    aureleoules commented at 10:07 AM on November 27, 2022:

    Removed chain argument, thanks.

  110. in test/functional/wallet_pruning.py:120 in c594dbc756 outdated
     115 | +
     116 | +        self.log.info("Generate a long chain of blocks...")
     117 | +
     118 | +        # A blk*.dat file is 128MB
     119 | +        # Generate 288 light blocks, the minimum required to stay on disk with prune enabled
     120 | +        self.generate(self.nodes[0], MIN_BLOCKS_TO_KEEP, sync_fun=lambda: None)
    


    kouloumos commented at 3:30 PM on November 24, 2022:

    Is this different than

            self.generate(self.nodes[0], MIN_BLOCKS_TO_KEEP, sync_fun=self.no_op)
    

    If not, I think it's better to keep it consistent across tests.


    kouloumos commented at 8:27 AM on November 25, 2022:

    I understand why you need some light blocks in the beginning. What I don't understand is the comment here referring to specifically 288 light blocks. Looking at the c++ code I read this about that number https://github.com/bitcoin/bitcoin/blob/38d06e1561013f4ca845fd5ba6ffcc64de67f9c0/src/validation.h#L68-L69

    How is that related to the logic of this test?


    aureleoules commented at 10:08 AM on November 27, 2022:

    Updated thanks.


    aureleoules commented at 10:09 AM on November 27, 2022:

    I think this was based on a different version of this test but you are correct 288 is not needed. I changed it to 250.

  111. in test/functional/wallet_pruning.py:127 in c594dbc756 outdated
     118 | +        # A blk*.dat file is 128MB
     119 | +        # Generate 288 light blocks, the minimum required to stay on disk with prune enabled
     120 | +        self.generate(self.nodes[0], MIN_BLOCKS_TO_KEEP, sync_fun=lambda: None)
     121 | +
     122 | +        # Generate 50MB worth of large blocks in the second blk*.dat file
     123 | +        self.mine_large_blocks(self.nodes[0], 50)
    


    kouloumos commented at 7:29 AM on November 25, 2022:

    Are you sure that the files generated here are part of the second blk*.data file? Using an adaptation of this function says otherwise https://github.com/bitcoin/bitcoin/blob/38d06e1561013f4ca845fd5ba6ffcc64de67f9c0/test/functional/feature_pruning.py#L296-L297

    Your latest push decreasing the number of generated large blocks invalidate this statement. I calculated that you need min 142 blocks to be generated here in order for the the blk00001.dat file to be created, resulting to the wallet that you create below to actually have a birth height in this file.


    aureleoules commented at 10:08 AM on November 27, 2022:

    Yes the comment was wrong.

  112. in test/functional/wallet_pruning.py:60 in c594dbc756 outdated
      55 | +    def setup_nodes(self):
      56 | +        self.add_nodes(self.num_nodes, extra_args=self.extra_args)
      57 | +        self.start_node(0)
      58 | +
      59 | +    def setup_network(self):
      60 | +        self.setup_nodes()
    


    kouloumos commented at 8:08 AM on November 25, 2022:

    The custom setup logic that I implied on my initial review was because you were using the default created wallets. But since then, you are testing on manually created wallets, therefore this is not needed.

    Additionally, I've later realize that you can skip the creation of the default wallets with self.wallet_names=[] which can help to simplify the logic. https://github.com/bitcoin/bitcoin/blob/38d06e1561013f4ca845fd5ba6ffcc64de67f9c0/test/functional/test_framework/test_framework.py#L108-L113

  113. in test/functional/wallet_pruning.py:133 in c594dbc756 outdated
     128 | +        self.nodes[0].createwallet(wallet_name=wallet_name, descriptors=False, load_on_startup=True)
     129 | +        self.nodes[0].dumpwallet(os.path.join(self.nodes[0].datadir, wallet_name + ".dat"))
     130 | +        self.nodes[0].unloadwallet(wallet_name)
     131 | +
     132 | +        # Generate large blocks to make sure we have enough to test chain pruning
     133 | +        # The first blk*.dat file should be pruned from the node 1
    


    kouloumos commented at 8:42 AM on November 25, 2022:

    This comment is not true. That file might be later pruned, but at this point is not and it makes reading the code confusing.

  114. in test/functional/wallet_pruning.py:83 in c594dbc756 outdated
      78 | +
      79 | +        # Import wallet into node 1
      80 | +        self.nodes[1].createwallet(wallet_name="wallet_pruned", descriptors=False, load_on_startup=True)
      81 | +        self.nodes[1].importwallet(os.path.join(self.nodes[0].datadir, wname))
      82 | +
      83 | +        # Make sure the wallets have the same balance
    


    kouloumos commented at 9:55 AM on November 25, 2022:

    nit

            # Make sure that prune node's wallet correctly accounts for balances
    
  115. in test/functional/wallet_pruning.py:91 in c594dbc756 outdated
      86 | +        # Mine some blocks, pruning should not have removed the blocks required for the wallet
      87 | +        self.mine_large_blocks(self.nodes[0], 5)
      88 | +        self.sync_all()
      89 | +
      90 | +        # Import wallet, should still work
      91 | +        self.nodes[1].importwallet(os.path.join(self.nodes[0].datadir, wname))
    


    kouloumos commented at 9:59 AM on November 25, 2022:

    ~You already imported the wallet in the pruned node above. I don't understand why you need to repeat this here. What do those newly mined 5 large blocks achieve?~

    Further down me messing with the code, I understood that those 5 blocks are needed to actually get into prune state and prune the blk00000.dat file. But for someone reading the code this is not clear. And if we are now getting into prune mode, then what's the above importwallet for?

  116. in test/functional/wallet_pruning.py:21 in c594dbc756 outdated
      16 | +    OP_RETURN,
      17 | +    OP_TRUE,
      18 | +)
      19 | +
      20 | +class WalletPruningTest(BitcoinTestFramework):
      21 | +    nTime = 0
    


    kouloumos commented at 11:29 AM on November 25, 2022:

    Is this the right place for this variable? I think that I haven't seen any other tests initializing self variables in such a way.


    aureleoules commented at 10:12 AM on November 27, 2022:

    Thanks I moved it.

  117. in test/functional/wallet_pruning.py:40 in c594dbc756 outdated
      17 | +    OP_TRUE,
      18 | +)
      19 | +
      20 | +class WalletPruningTest(BitcoinTestFramework):
      21 | +    nTime = 0
      22 | +    def mine_large_blocks(self, node, n):
    


    kouloumos commented at 11:31 AM on November 25, 2022:

    From what I've seen, custom functions are often declared after the setup override functions.


    aureleoules commented at 10:12 AM on November 27, 2022:

    Thanks, I moved it.

  118. kouloumos commented at 1:56 PM on November 25, 2022: contributor

    The functional test got me really confused on what it is testing and how it is testing it. It might be because I am re-reviewing this after 55 days, but I think that some simplification (left comments inline) as well as commenting on the basis of the change and not on what the code does could help with that.

    What the test does (afaiu) is simulating 2 different wallets with 2 different birth heights wallet_birthheight_1 < wallet_birthheight_2 It then prunes below birthheight_2 to:

    • check with test_wallet_import_pruned_test that wallet_birthheight_2 is imported correctly.
    • check with test_wallet_import_pruned_with_missing_blocks that wallet_birthheight_1 cannot imported correctly.

    Based on the test overview above and some comments that I left inline, I did some implementation-logic restructuring and added a few comments that I think can help with comprehension. You can find my changes at 69b9b6247305d06168f58e9e156807e09a925bf6 (diff), feel free to pick whatever you like.

  119. aureleoules force-pushed on Nov 27, 2022
  120. aureleoules force-pushed on Nov 27, 2022
  121. aureleoules commented at 10:16 AM on November 27, 2022: member

    Thanks for your detailed review @kouloumos. I cherry-picked your commit and squashed it. I also addressed some of your additional suggestions.

  122. kouloumos commented at 10:54 AM on November 28, 2022: contributor

    ACK 8e87433a4460088d15275623f0b6f1893af28d82 nit: The MIN_BLOCKS_TO_KEEP constant is not used anymore in the test, which makes 8e87433a4460088d15275623f0b6f1893af28d82 out of scope.

  123. maflcko referenced this in commit 3be21060d6 on Nov 30, 2022
  124. maflcko commented at 4:37 PM on December 7, 2022: member

    This is adding a new test wallet_pruning.py, but I can't find it in the CI output at all. Does anyone see what is going on here?

  125. fanquake commented at 4:45 PM on December 7, 2022: member

    Is it becuase it's in EXTENDED_SCRIPTS? That is the job that's failing, and didn't make it to the functional tests: https://github.com/bitcoin/bitcoin/pull/24865/checks?check_run_id=9728080972.

  126. kouloumos commented at 4:48 PM on December 7, 2022: contributor

    It also runs as part of Win64 native [vs2022]. I can see that the extended tests run there and wallet_pruning.py runs successfully.

  127. maflcko commented at 4:50 PM on December 7, 2022: member

    Ah thx. Maybe there should be a skip_if_not_running_extended :sweat_smile: ?

    Also, I think this needs a rebase, see upcoming CI failure in https://cirrus-ci.com/task/6513158118965248

  128. aureleoules force-pushed on Dec 7, 2022
  129. rpc: Enable wallet import on pruned nodes
    Co-authored-by: João Barbosa <joao.paulo.barbosa@gmail.com>
    Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
    e6906fcf9e
  130. aureleoules force-pushed on Dec 8, 2022
  131. aureleoules commented at 12:53 PM on December 8, 2022: member

    Rebased to address silent conflicts.

  132. achow101 commented at 10:10 PM on December 14, 2022: member

    ACK 63493d778f443554059d77bd8c9b1a926685a892

  133. in test/functional/wallet_pruning.py:25 in 63493d778f outdated
      20 | +    OP_TRUE,
      21 | +)
      22 | +
      23 | +class WalletPruningTest(BitcoinTestFramework):
      24 | +    def add_options(self, parser):
      25 | +        self.add_wallet_options(parser)
    


    kouloumos commented at 7:35 AM on December 15, 2022:

    If this is a legacy wallet only test, this should be

            self.add_wallet_options(parser, descriptors=False)
    
  134. in test/functional/wallet_pruning.py:119 in 63493d778f outdated
     114 | +        if (unload):
     115 | +            self.nodes[0].unloadwallet(wallet_name)
     116 | +
     117 | +    def run_test(self):
     118 | +        self.nTime = 0
     119 | +        self.log.info("Warning! This test requires ~1.1GB of disk space")
    


    kouloumos commented at 7:37 AM on December 15, 2022:

    nit I haven't calculate it properly, but with some quick monitoring during the test run, it seems that this might be closer to 1.3GB

  135. test: Wallet imports on pruned nodes
    Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
    Co-authored-by: Andreas Kouloumos <kouloumosa@gmail.com>
    71d9a7c03b
  136. test: Introduce MIN_BLOCKS_TO_KEEP constant 564b580bf0
  137. aureleoules force-pushed on Dec 15, 2022
  138. aureleoules commented at 8:54 AM on December 15, 2022: member

    Thanks @kouloumos I addressed your comments.

  139. aureleoules requested review from ryanofsky on Dec 15, 2022
  140. aureleoules removed review request from ryanofsky on Dec 15, 2022
  141. aureleoules requested review from w0xlt on Dec 15, 2022
  142. aureleoules removed review request from w0xlt on Dec 15, 2022
  143. aureleoules requested review from furszy on Dec 15, 2022
  144. aureleoules removed review request from furszy on Dec 15, 2022
  145. aureleoules requested review from kouloumos on Dec 15, 2022
  146. aureleoules removed review request from kouloumos on Dec 15, 2022
  147. aureleoules requested review from achow101 on Dec 15, 2022
  148. kouloumos commented at 8:58 AM on December 15, 2022: contributor

    ACK 564b580bf07742483a140c7c095b896a6d5d6cad

  149. in src/wallet/rpc/backup.cpp:104 in e6906fcf9e outdated
      99 | +    if (!chain.havePruned()) {
     100 | +        return;
     101 | +    }
     102 | +
     103 | +    int height{0};
     104 | +    const bool found{chain.findFirstBlockWithTimeAndHeight(timestamp - TIMESTAMP_WINDOW, 0, FoundBlock().height(height))};
    


    furszy commented at 6:54 PM on December 15, 2022:

    In e6906fcf:

    tiny nit: (only if you have to re-touch)

        const bool found{chain.findFirstBlockWithTimeAndHeight(timestamp - TIMESTAMP_WINDOW, /*min_height=*/ 0, FoundBlock().height(height))};
    
  150. in test/functional/test_framework/blocktools.py:142 in 71d9a7c03b outdated
     137 | @@ -138,6 +138,8 @@ def create_coinbase(height, pubkey=None, extra_output_script=None, fees=0, nValu
     138 |          coinbaseoutput.nValue += fees
     139 |      if pubkey is not None:
     140 |          coinbaseoutput.scriptPubKey = key_to_p2pk_script(pubkey)
     141 | +    elif script_pubkey is not None:
     142 | +        coinbaseoutput.scriptPubKey = script_pubkey
    


    furszy commented at 7:06 PM on December 15, 2022:

    In 71d9a7c0:

    atm is just a non-blocking nit:

    it's weird to add this new ´script_pubkey´ arg that is only set if the ´pubkey´ arg is not provided. It is not stated anywhere.

    Ideally, should unify both into a single arg. So there is no confusion over which of them take precedence over the other.

  151. in test/functional/wallet_pruning.py:61 in 71d9a7c03b outdated
      56 | +
      57 | +            # Simulate 10 minutes of work time per block
      58 | +            # Important for matching a timestamp with a block +- some window
      59 | +            self.nTime += 600
      60 | +            for n in self.nodes:
      61 | +                if n.running:
    


    furszy commented at 7:09 PM on December 15, 2022:

    aureleoules commented at 9:35 AM on December 16, 2022:

    Right, in a previous version of the test it could crash without this check.

  152. furszy approved
  153. furszy commented at 7:11 PM on December 15, 2022: member

    ACK 564b580 left few non-blocking nits.

  154. w0xlt approved
  155. achow101 commented at 10:19 PM on December 16, 2022: member

    reACK 564b580bf07742483a140c7c095b896a6d5d6cad

  156. achow101 merged this on Dec 16, 2022
  157. achow101 closed this on Dec 16, 2022

  158. sidhujag referenced this in commit 59bb1c8365 on Dec 17, 2022
  159. in test/functional/wallet_pruning.py:48 in 564b580bf0
      43 | +        height = int(best_block["height"]) + 1
      44 | +        self.nTime = max(self.nTime, int(best_block["time"])) + 1
      45 | +        previousblockhash = int(best_block["hash"], 16)
      46 | +        big_script = CScript([OP_RETURN] + [OP_TRUE] * 950000)
      47 | +        for _ in range(n):
      48 | +            block = create_block(hashprev=previousblockhash, ntime=self.nTime, coinbase=create_coinbase(height, script_pubkey=big_script))
    


    maflcko commented at 5:06 PM on December 19, 2022:

    nit: It is possible to do this without blocktools. See

    diff --git a/test/functional/wallet_pruning.py b/test/functional/wallet_pruning.py
    index 6d8475ce8d..d768130776 100755
    --- a/test/functional/wallet_pruning.py
    +++ b/test/functional/wallet_pruning.py
    @@ -40,20 +40,10 @@ class WalletPruningTest(BitcoinTestFramework):
         def mine_large_blocks(self, node, n):
             # Get the block parameters for the first block
             best_block = node.getblock(node.getbestblockhash())
    -        height = int(best_block["height"]) + 1
             self.nTime = max(self.nTime, int(best_block["time"])) + 1
    -        previousblockhash = int(best_block["hash"], 16)
    -        big_script = CScript([OP_RETURN] + [OP_TRUE] * 950000)
    +        big_script = "raw({})".format(bytes([OP_RETURN] + [OP_TRUE] * 950000).hex())
             for _ in range(n):
    -            block = create_block(hashprev=previousblockhash, ntime=self.nTime, coinbase=create_coinbase(height, script_pubkey=big_script))
    -            block.solve()
    -
    -            # Submit to the node
    -            node.submitblock(block.serialize().hex())
    -
    -            previousblockhash = block.sha256
    -            height += 1
    -
    +            self.generateblock(node, big_script, [])
                 # Simulate 10 minutes of work time per block
                 # Important for matching a timestamp with a block +- some window
                 self.nTime += 600
    

    Also, might be good to double check the runtime in light of https://github.com/bitcoin/bitcoin/commit/6c872d5e656a7117bbdf19a0220572b93de16f31

  160. aureleoules deleted the branch on Jan 12, 2023
  161. bitcoin locked this on Jan 12, 2024

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-04-21 18:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me