[kernel 2d/n] Reduce CTxMemPool constructor call sites #25215

pull dongcarl wants to merge 6 commits into bitcoin:master from dongcarl:2022-03-libbitcoinkernel-reduce-mempool-ctor changing 9 files +29 −26
  1. dongcarl commented at 9:31 pm on May 25, 2022: member

    This is part of the libbitcoinkernel project: #24303, https://github.com/bitcoin/bitcoin/projects/18

    This PR reduces the number of call sites where we explicitly construct CTxMemPool. This is done in preparation for later PRs which decouple the mempool module from ArgsManager, eventually all of libbitcoinkernel will be decoupled from ArgsManager.

    The changes in this PR:

    • Allows us to have less code churn as we modify CTxMemPool’s constructor in later PRs
    • In many cases, we can make use of existing CTxMemPool instances, getting rid of extraneous constructions
    • In other cases, we construct a ChainTestingSetup and use the CTxMemPool there, so that we can rely on the logic in setup_common to set things up correctly

    Notes for Reviewers

    A note on using existing mempools

    When evaluating whether or not it’s appropriate to use an existing mempool in a *TestingSetup struct, the key is to make sure that the mempool has the same lifetime as the *TestingSetup struct.

    Example 1: In src/fuzz/tx_pool.cpp, the TestingSetup is initialized in initialize_tx_pool and lives as a static global, while the CTxMemPool is in the tx_pool_standard fuzz target, meaning that each time the tx_pool_standard fuzz target gets run, a new CTxMemPool is created. If we were to use the static global TestingSetup’s CTxMemPool we might run into problems since its CTxMemPool will carry state between subsequent runs. This is why we don’t modify src/fuzz/tx_pool.cpp in this PR.

    Example 2: In src/bench/mempool_eviction.cpp, we see that the TestingSetup is in the same scope as the constructed CTxMemPool, so it is safe to use its CTxMemPool.

    A note on checking CTxMemPool ctor call sites

    After the “tree-wide: clang-format CTxMemPool references” commit, you can find all CTxMemPool ctor call sites with the following command:

    0git grep -E -e 'make_unique<CTxMemPool>' \
    1            -e '\bCTxMemPool\s+[^({;]+[({]' \
    2            -e '\bCTxMemPool\s+[^;]+;' \
    3            -e '\bnew\s+CTxMemPool\b'
    

    At the end of the PR, you will find that there are still quite a few call sites that we can seemingly get rid of:

     0$ git grep -E -e 'make_unique<CTxMemPool>' -e '\bCTxMemPool\s+[^({;]+[({]' -e '\bCTxMemPool\s+[^;]+;' -e '\bnew\s+CTxMemPool\b'
     1# rearranged for easier explication
     2src/init.cpp:        node.mempool = std::make_unique<CTxMemPool>(node.fee_estimator.get(), mempool_check_ratio);
     3src/test/util/setup_common.cpp:    m_node.mempool = std::make_unique<CTxMemPool>(m_node.fee_estimator.get(), 1);
     4src/rpc/mining.cpp:        CTxMemPool empty_mempool;
     5src/test/util/setup_common.cpp:    CTxMemPool empty_pool;
     6src/bench/mempool_stress.cpp:    CTxMemPool pool;
     7src/bench/mempool_stress.cpp:    CTxMemPool pool;
     8src/test/fuzz/rbf.cpp:    CTxMemPool pool;
     9src/test/fuzz/tx_pool.cpp:    CTxMemPool tx_pool_{/*estimator=*/nullptr, /*check_ratio=*/1};
    10src/test/fuzz/tx_pool.cpp:    CTxMemPool tx_pool_{/*estimator=*/nullptr, /*check_ratio=*/1};
    11src/test/fuzz/validation_load_mempool.cpp:    CTxMemPool pool{};
    12src/txmempool.h:    /** Create a new CTxMemPool.
    

    Let’s break them down one by one:

    0src/init.cpp:        node.mempool = std::make_unique<CTxMemPool>(node.fee_estimator.get(), mempool_check_ratio);
    1src/test/util/setup_common.cpp:    m_node.mempool = std::make_unique<CTxMemPool>(m_node.fee_estimator.get(), 1);
    

    Necessary


    0src/rpc/mining.cpp:        CTxMemPool empty_mempool;
    1src/test/util/setup_common.cpp:    CTxMemPool empty_pool;
    

    These are fixed in #25223 where we stop requiring the BlockAssembler to have a CTxMemPool if it’s not going to consult it anyway (as is the case in these two call sites)


    0src/bench/mempool_stress.cpp:    CTxMemPool pool;
    1src/bench/mempool_stress.cpp:    CTxMemPool pool;
    

    Fixed in #24927.


    0src/test/fuzz/rbf.cpp:    CTxMemPool pool;
    1src/test/fuzz/tx_pool.cpp:    CTxMemPool tx_pool_{/*estimator=*/nullptr, /*check_ratio=*/1};
    2src/test/fuzz/tx_pool.cpp:    CTxMemPool tx_pool_{/*estimator=*/nullptr, /*check_ratio=*/1};
    3src/test/fuzz/validation_load_mempool.cpp:    CTxMemPool pool{};
    

    These are all cases where we don’t want the CTxMemPool state to persist between runs, see the previous section “A note on using existing mempools”


    0src/txmempool.h:    /** Create a new CTxMemPool.
    

    It’s a comment (someone link me to a grep that understands syntax plz thx)

  2. tree-wide: clang-format CTxMemPool references
    [META] Do this so that we can more easily grep for all actual instances
           of CTxMemPool construction.
    03574b956a
  3. dongcarl added this to the "WIP PRs" column in a project

  4. dongcarl moved this from the "WIP PRs" to the "Ready for Review PRs" column in a project

  5. DrahtBot added the label P2P on May 25, 2022
  6. DrahtBot added the label RPC/REST/ZMQ on May 25, 2022
  7. DrahtBot commented at 9:11 am on May 26, 2022: member

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #25290 ([kernel 3a/n] Decouple CTxMemPool from ArgsManager by dongcarl)
    • #25077 (Fix chain tip data race and corrupt rest response by MarcoFalke)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  8. MarcoFalke commented at 10:04 am on May 27, 2022: member
    It looks like you are mostly modifying tests. If the goal it to avoid future code churn, why not add a test-only function that constructs a test-only mempool?
  9. dongcarl commented at 6:50 pm on May 27, 2022: member

    @MarcoFalke

    It looks like you are mostly modifying tests. If the goal it to avoid future code churn, why not add a test-only function that constructs a test-only mempool?

    I agree, though in my mind ChainTestingSetup::ChainTestingSetup is that “test-only function that constructs a test-only mempool”, and I’m just making more tests use it! 😄

  10. MarcoFalke commented at 7:35 am on May 28, 2022: member
    Well, it also sets up a full chain, so it is not possible to call it twice. I am mostly asking in the context of #25073 , which I did after your suggestion (see #19909 (comment) ).
  11. dongcarl commented at 8:13 pm on May 31, 2022: member

    @MarcoFalke

    I am mostly asking in the context of #25073 , which I did after your suggestion (see #19909 (comment) ).

    Ah I see! I consider the cases in #25073 as ones where we don’t want the CTxMemPool state to persist for the lifetime of the TestingSetup, and therefore we wouldn’t have changed it in this PR at all. See “A note on using existing mempools” in the original description. I’m also okay with adding these new invocations of the CTxMemPool constructor in #25073 since they are logically necessary and avoids clear().

    I do now see the benefit to having a standalone function that constructs a test-only mempool… It would coalesce all of the cases not addressed by this PR and 2e (mostly the src/fuzz ones and the new miner_test ones in #25073) into just one call site… I’ll give it some thought.

  12. laanwj added this to the "Blockers" column in a project

  13. dongcarl commented at 4:17 pm on June 13, 2022: member

    @MarcoFalke

    I’ll give it some thought.

    Given it some thought. I think “making a standalone function that constructs a test-only mempool” is not relevant to this PR and can be done in the future. In fact, it’s likely easier to do with the CTxMemPool::Options introduced in #25290.

    I don’t think anything we’re doing here or in #25290 is mutually exclusive with your #25073, since we’re not outright eliminating all CTxMemPool callsites, just eliminating them where they’re not necessary. So if you’re okay with it, I think we can table this discussion for now.

  14. MarcoFalke removed the label RPC/REST/ZMQ on Jun 13, 2022
  15. MarcoFalke removed the label P2P on Jun 13, 2022
  16. MarcoFalke added the label Refactoring on Jun 13, 2022
  17. in src/test/blockencodings_tests.cpp:210 in bf65ab9088 outdated
    206@@ -207,7 +207,7 @@ BOOST_AUTO_TEST_CASE(NonCoinbasePreforwardRTTest)
    207 
    208 BOOST_AUTO_TEST_CASE(SufficientPreforwardRTTest)
    209 {
    210-    CTxMemPool pool;
    211+    CTxMemPool& pool = *Assert(m_node.mempool);
    


    laanwj commented at 12:53 pm on June 15, 2022:

    I’m somewhat divided about this change. From a “testing hygiene” point of view it seems preferable to me to construct a new mempool instead of use an existing one from the global state.

    I also like the idea of a function to construct a default mempool for testing. But maybe I’m missing something.


    dongcarl commented at 3:59 pm on June 15, 2022:

    From a “testing hygiene” point of view it seems preferable to me to construct a new mempool instead of use an existing one from the global state.

    In cases where we only need one mempool that does not outlive TestingSetup (the cases addressed in this PR), it is not a problem to use TestingSetup’s mempool since it’s reconstructed each time. In many cases it is much more preferable to use TestingSetup’s mempool:

    • TestingSetup’s mempool will respect gArgs flags given to it by the TestingSetup constructor, local mempools will not. This is an easier footgun to shoot than one would think, and has already led to a case where we were fuzzing a noop: #24634
    • TestingSetup’s mempool is registered in TestingSetup’s CChainStates’ m_mempool and perhaps in other places. Using a local mempool means that there will be a discrepancy between the mempool the test operates on and the mempool that CChainStates operate on.

    To be clear, I’m not saying that all of our tests all have to use TestingSetup’s mempool, there are many places where that doesn’t make sense (see this PR’s original description). I’m only changing places where it’s obviously correct (and a coalescing of codepaths) to use TestingSetup’s mempool.

    I also like the idea of a function to construct a default mempool for testing.

    Sure! I like it too. It’s great for cases of CTxMemPool construction that this PR deliberately leaves out. It might even be able to use the DummyChainstate technique in src/test/fuzz/tx_pool.cpp to replace the m_mempool pointer in CChainState. It’s just done easier after we introduce CTxMemPool::Options in #25290 so I think we can table it for now.


    laanwj commented at 7:11 pm on June 15, 2022:
    OK, agree. I was afraid this would introduce errors the other way around (by accidentally reusing mempool between tests), but if that’s not the case, and can even avoid unexpected behavior, great.
  18. fanquake referenced this in commit a7a36590f5 on Jun 15, 2022
  19. in src/rest.cpp:820 in 29d1084245 outdated
    814@@ -815,10 +815,10 @@ static bool rest_getutxos(const std::any& context, HTTPRequest* req, const std::
    815             LOCK2(cs_main, mempool->cs);
    816             CCoinsViewCache& viewChain = chainman.ActiveChainstate().CoinsTip();
    817             CCoinsViewMemPool viewMempool(&viewChain, *mempool);
    818-            process_utxos(viewMempool, *mempool);
    819+            process_utxos(viewMempool, mempool);
    820         } else {
    821             LOCK(cs_main);  // no need to lock mempool!
    


    theuni commented at 6:17 pm on June 15, 2022:
    Comment no longer applies.

    dongcarl commented at 9:32 pm on June 15, 2022:
    Fixed as of 319f0ceeeb25f28e027fc41be2755092dc5365b4
  20. theuni commented at 7:14 pm on June 15, 2022: member

    Quick concept/code review ACK, looks like no functional change.

    I have no thoughts on the test setup changes, so I’ll defer to @laanwj’s and @MarcoFalke’s reviews on those.

    In case it’s helpful for future work to use clang-query to find instances rather than these nasty greps, the incantation for matching CTxMemPool constructors would be: cxxConstructExpr(hasType(cxxRecordDecl(hasName("CTxMemPool"))))

  21. rest/getutxos: Don't construct empty mempool
    ...just don't try to consult it at all when fCheckMemPool is false
    319f0ceeeb
  22. test/policyestimator: Use ChainTestingSetup's CTxMemPool 213457e170
  23. scripted-diff: test: Use CTxMemPool in TestingSetup
    After this commit, there should be no explicit instantiation of
    CTxMemPool in src/test other than those in fuzz/ and setup_common
    
    -BEGIN VERIFY SCRIPT-
    find_regex="CTxMemPool\s+([^;({]+)(|\(\)|\{\});" \
        && git grep -l -E "$find_regex" -- src/test \
            | grep -v -e "^src/test/util/setup_common.cpp$" \
                      -e "^src/test/fuzz/" \
            | xargs sed -i -E "s@$find_regex@CTxMemPool\& \1 = *Assert(m_node.mempool);@g"
    -END VERIFY SCRIPT-
    86e732def3
  24. bench: Use existing CTxMemPool in TestingSetup 020caba3df
  25. bench/rpc_mempool: Create ChainTestingSetup, use its CTxMemPool
    This is correct because:
    
    - The ChainTestingSetup is constructed before the call to bench.run(...)
    - All the runs are performed on the same mempool
    d273e53b6e
  26. dongcarl force-pushed on Jun 15, 2022
  27. dongcarl commented at 9:31 pm on June 15, 2022: member

    Pushed bf65ab9088df6f7e49030f3539b51d75023a6572 -> d273e53b6e2cabd91a83f0ff0f9b6cfe1815b637

  28. laanwj commented at 5:43 pm on June 16, 2022: member
    Code review ACK d273e53b6e2cabd91a83f0ff0f9b6cfe1815b637
  29. laanwj merged this on Jun 16, 2022
  30. laanwj closed this on Jun 16, 2022

  31. laanwj moved this from the "Ready for Review PRs" to the "Done or Closed or Rethinking" column in a project

  32. laanwj removed this from the "Blockers" column in a project

  33. sidhujag referenced this in commit c763f11790 on Jun 17, 2022
  34. DrahtBot locked this on Jun 16, 2023

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-07-01 13:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me