test: add MAX_DISCONNECTED_TX_POOL_BYTES, chainlimits coverage #32516

pull instagibbs wants to merge 3 commits into bitcoin:master from instagibbs:2025-05-reorg_mempool_trimming changing 1 files +113 −12
  1. instagibbs commented at 3:19 pm on May 15, 2025: member

    DisconnectedBlockTransactions::LimitMemoryUsage() has unit test coverage, but the default value end to end doesn’t have coverage.

    This test adds exercised coverage of memory limiting of the disconnect pool, and some basic behavior sanity checks.

    Another test added is making sure chainlimits are being respected on reorg, and the expected transactions pruned.

    Lastly, fix the existing test case which was using a deficient test via directly inducing reorgs with invalidateblock

  2. DrahtBot commented at 3:19 pm on May 15, 2025: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/32516.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK mzumsande, i-am-yuvi

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    LLM Linter (✨ experimental)

    Possible typos and grammar issues:

    • all of the mined the old fork -> all of the transactions mined in the old fork [Grammatically incorrect phrase “of the mined the”]
  3. in test/functional/mempool_updatefromblock.py:106 in 7d53373d0e outdated
     97@@ -97,6 +98,53 @@ def transaction_graph_test(self, size, n_tx_to_mine=None, fee=100_000):
     98             assert_equal(entry['ancestorcount'], k + 1)
     99             assert_equal(entry['ancestorsize'], sum(tx_size[0:(k + 1)]))
    100 
    101+        self.generate(self.nodes[0], 1)
    102+        assert_equal(self.nodes[0].getrawmempool(), [])
    103+        wallet.rescan_utxos()
    104+
    105+        # Make 200 outputs to spend via wallet
    106+        wallet.send_self_transfer_multi(from_node=self.nodes[0], num_outputs=200)
    


    instagibbs commented at 3:20 pm on May 15, 2025:
    MiniWallet doesn’t appear to support more than 49 utxos at a time? Unclear to me what’s going on so I did this instead.

    theStack commented at 5:05 pm on May 15, 2025:

    Not exactly sure if this is what you are asking, but: new blocks with coinbase transaction outputs for MiniWallet can be generated by passing an instance of the latter to self.generate (instead of the node object), i.e. in this case:

    0        self.generate(wallet, 300)
    

    (added +100 to let the coins mature)


    instagibbs commented at 5:06 pm on May 15, 2025:
    Oh, mine to wallet, not to node which has attached wallet :facepalm:
  4. fanquake renamed this:
    functional test: add MAX_DISCONNECTED_TX_POOL_BYTES coverage
    test: add MAX_DISCONNECTED_TX_POOL_BYTES coverage
    on May 15, 2025
  5. DrahtBot added the label Tests on May 15, 2025
  6. instagibbs marked this as a draft on May 15, 2025
  7. instagibbs commented at 4:39 pm on May 15, 2025: member
    I don’t think I’m actually hitting that particular limit, getting weird results when looking closer, draft for now while I look
  8. instagibbs force-pushed on May 15, 2025
  9. instagibbs marked this as ready for review on May 15, 2025
  10. instagibbs force-pushed on May 15, 2025
  11. DrahtBot added the label CI failed on May 15, 2025
  12. DrahtBot commented at 6:51 pm on May 15, 2025: contributor

    🚧 At least one of the CI tasks failed. Task lint: https://github.com/bitcoin/bitcoin/runs/42311652764 LLM reason (✨ experimental): The CI failure is due to lint errors identified by ruff.

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  13. instagibbs force-pushed on May 15, 2025
  14. instagibbs force-pushed on May 15, 2025
  15. instagibbs renamed this:
    test: add MAX_DISCONNECTED_TX_POOL_BYTES coverage
    test: add MAX_DISCONNECTED_TX_POOL_BYTES, chainlimits coverage
    on May 15, 2025
  16. instagibbs force-pushed on May 16, 2025
  17. DrahtBot removed the label CI failed on May 16, 2025
  18. in test/functional/mempool_updatefromblock.py:133 in 8f256663d1 outdated
    128+        large_std_txs = []
    129+        # Add children to ensure they're recursively removed if disconnectpool trimming of parent occurs
    130+        small_child_txs = []
    131+        aggregate_serialized_size = 0
    132+        while aggregate_serialized_size < MAX_DISCONNECTED_TX_POOL_BYTES:
    133+            # Mine parents in FIFO order via priority (priority map wiped during reorg)
    


    mzumsande commented at 5:48 pm on May 19, 2025:
    I don’t understand this comment. Did you use prioritisetransaction in an older version, but dropped that?

    instagibbs commented at 8:40 pm on May 19, 2025:
    oops, yep, removed
  19. in test/functional/mempool_updatefromblock.py:199 in 8f256663d1 outdated
    196@@ -190,7 +197,7 @@ def test_chainlimits_exceeded(self):
    197 
    198     def run_test(self):
    199         # Use batch size limited by DEFAULT_ANCESTOR_LIMIT = 25 to not fire "too many unconfirmed parents" error.
    


    mzumsande commented at 7:45 pm on May 19, 2025:

    This comment is outdated, because DEFAULT_ANCESTOR_LIMIT = 25 is no longer after the last commit. Not sure about the reason why it was used before, is it no longer valid?

    Also, changing DEFAULT_ANCESTOR_LIMIT to 100 could be mentioned in the commit message, it also affects the other subtests.


    instagibbs commented at 8:40 pm on May 19, 2025:
    Kept the test setup as is, since it tests a meaningful 3 block reorg. Changed the comment because the motivation is different.
  20. mzumsande commented at 7:49 pm on May 19, 2025: contributor
    Concept ACK
  21. functional test: add MAX_DISCONNECTED_TX_POOL_BYTES coverage 1aeb1838ca
  22. test: check chainlimits respects on reorg 05abcc8699
  23. instagibbs force-pushed on May 19, 2025
  24. test: fix transaction_graph_test reorg test
    The current test directly uses invalidatblock to trigger
    mempool re-entry of transactions. Unfortunately, the
    behavior doesn't match what a real reorg would look like. As
    a result you get surprising behavior such as the mempool
    descendant chain limits being exceeded, or if a fork is
    greater than 10 blocks deep, evicted block transactions stop
    being submitted back into in the mempool.
    
    Fix this by preparing an empty fork chain, invalidating that
    via rpc, and then continuing with the logic, finally
    reconsidering the fork chain once the rest of the test is
    prepared. This triggers a more typical codepath.
    
    Also, extend the descendant limit to 100, like ancestor
    limit.
    2b202e9db5
  25. instagibbs force-pushed on May 19, 2025
  26. mzumsande commented at 9:30 pm on May 20, 2025: contributor
    Code Review ACK 2b202e9db56487e658fc41089178f31ef4259a0d
  27. i-am-yuvi commented at 4:56 pm on May 21, 2025: contributor
    Concept ACK
  28. i-am-yuvi commented at 5:11 pm on May 21, 2025: contributor

    tACK 2b202e9db56487e658fc41089178f31ef4259a0d

    • DisconnectTip() & reconnect via invalidateblock and reconsiderblock
    • LimitMemoryUsage(): exercised by flooding > 20 MB
    • test_chainlimits_exceeded

    Logs

     02025-05-21T16:57:42.023000Z TestFramework (INFO): PRNG seed is: 4085491027579870526
     12025-05-21T16:57:42.024000Z TestFramework (INFO): Initializing test directory /var/folders/jb/wlbrz0t95vl58wzxjt75wqmh0000gn/T/bitcoin_func_test_jb27si_k
     22025-05-21T16:57:42.337000Z TestFramework (INFO): Creating 100 transactions...
     32025-05-21T16:57:42.717000Z TestFramework (INFO): The batch of 25 transactions has been accepted into the mempool.
     42025-05-21T16:57:42.722000Z TestFramework (INFO): All of the transactions from the current batch have been mined into a block.
     52025-05-21T16:57:44.161000Z TestFramework (INFO): The batch of 25 transactions has been accepted into the mempool.
     62025-05-21T16:57:44.166000Z TestFramework (INFO): All of the transactions from the current batch have been mined into a block.
     72025-05-21T16:57:46.466000Z TestFramework (INFO): The batch of 25 transactions has been accepted into the mempool.
     82025-05-21T16:57:46.472000Z TestFramework (INFO): All of the transactions from the current batch have been mined into a block.
     92025-05-21T16:57:48.423000Z TestFramework (INFO): The last batch of 25 transactions has been accepted into the mempool.
    102025-05-21T16:57:48.571000Z TestFramework (INFO): All of the recently mined transactions have been re-added into the mempool in 0.1478879451751709 seconds.
    112025-05-21T16:57:48.571000Z TestFramework (INFO): Checking descendants/ancestors properties of all of the in-mempool transactions...
    122025-05-21T16:57:48.633000Z TestFramework (INFO): Creating independent transactions to test MAX_DISCONNECTED_TX_POOL_BYTES limit during reorg
    132025-05-21T16:57:53.509000Z TestFramework (INFO): Check that too long chains on reorg are handled
    142025-05-21T16:57:55.389000Z TestFramework (INFO): Stopping nodes
    152025-05-21T16:57:55.494000Z TestFramework (INFO): Cleaning up /var/folders/jb/wlbrz0t95vl58wzxjt75wqmh0000gn/T/bitcoin_func_test_jb27si_k on exit
    162025-05-21T16:57:55.494000Z TestFramework (INFO): Tests successful
    

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-05-25 18:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me