refactor: Simplify CTxMempool/BlockAssembler fields, remove some external mapTx access #28391

pull TheCharlatan wants to merge 15 commits into bitcoin:master from TheCharlatan:simplifyMemPoolInteractions changing 14 files +113 −94
  1. TheCharlatan commented at 9:47 am on September 2, 2023: contributor

    Motivation

    • It seems preferable to use stdlib data structures instead of boost if they can achieve close to the same thing.
    • Code external to mempool should ideally use its public helper methods instead of accessing mapTx or its iterators directly.
    • Reduce the number of complex boost multi index type interactions
    • Also see #28335 for further context/motivation. This PR together with #28385 simplifies that one.

    Overview of things done in this PR:

    • Make vTxHashes a vector of transaction references instead of a pair of transaction hash and iterator. The trade off here is that the data is retrieved on the fly with GetEntry instead of being cached in vTxHashes.
    • Introduce GetEntry helper method to replace the more involved GetIter where applicable
    • Replace mapTx access with CTxMemPool helper methods
    • Simplify checkChainLimits call in node/interfaces.cpp
    • Make CTxMemPoolEntrys lockPointsmutable such that they can be changed with a const iterator directly instead of going through mapTx
    • Make BlockAssembler’s inBlock and failedTx sets of transaction hashes.
  2. DrahtBot commented at 9:47 am on September 2, 2023: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage

    For detailed information about the code coverage, see the test coverage report.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK stickies-v, glozow, maflcko
    Concept ACK Sjors
    Stale ACK ismaelsadeeq

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #28368 (Fee Estimator updates from Validation Interface/CScheduler thread by ismaelsadeeq)
    • #28335 (RFC: Remove boost usage from kernel api / headers by TheCharlatan)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  3. DrahtBot added the label Refactoring on Sep 2, 2023
  4. TheCharlatan renamed this:
    refactor: Simplify CTxMempool / Miner fields, remove some external mapTx usage
    refactor: Simplify CTxMempool/BlockAssembler fields, remove some external mapTx usage
    on Sep 2, 2023
  5. TheCharlatan force-pushed on Sep 2, 2023
  6. DrahtBot added the label CI failed on Sep 2, 2023
  7. TheCharlatan renamed this:
    refactor: Simplify CTxMempool/BlockAssembler fields, remove some external mapTx usage
    refactor: Simplify CTxMempool/BlockAssembler fields, remove some external mapTx access
    on Sep 3, 2023
  8. DrahtBot removed the label CI failed on Sep 4, 2023
  9. TheCharlatan marked this as ready for review on Sep 5, 2023
  10. glozow commented at 1:32 pm on September 5, 2023: member
    Concept ACK fwiw As additional motivation, I imagine this would also make it easier to change the internal implementation of CTxMemPool without touching all these files.
  11. TheCharlatan force-pushed on Sep 6, 2023
  12. DrahtBot added the label Needs rebase on Sep 14, 2023
  13. TheCharlatan force-pushed on Sep 15, 2023
  14. DrahtBot removed the label Needs rebase on Sep 15, 2023
  15. DrahtBot added the label CI failed on Sep 21, 2023
  16. in src/node/interfaces.cpp:704 in a5bdd7eace outdated
    701-        CTxMemPoolEntry entry(tx, 0, 0, 0, 0, false, 0, lp);
    702-        const CTxMemPool::Limits& limits{m_node.mempool->m_limits};
    703         LOCK(m_node.mempool->cs);
    704-        return m_node.mempool->CalculateMemPoolAncestors(entry, limits).has_value();
    705+        std::string err_string;
    706+        return m_node.mempool->CheckPackageLimits({tx}, m_node.mempool->m_limits, err_string);
    


    ismaelsadeeq commented at 5:55 pm on September 24, 2023:

    CI failure is due to merge conflict can be fixed with

    0        return m_node.mempool->CheckPackageLimits({tx}, GetVirtualTransactionSize(*tx), err_string);
    
  17. in src/test/blockencodings_tests.cpp:54 in b0dda173e1 outdated
    50@@ -51,8 +51,8 @@ static CBlock BuildBlockTestCase() {
    51 }
    52 
    53 // Number of shared use_counts we expect for a tx we haven't touched
    54-// (block + mempool + our copy from the GetSharedTx call)
    55-constexpr long SHARED_TX_OFFSET{3};
    56+// (block + mempool entry + mempool txns_randomized + our copy from the GetSharedTx call)
    


    ismaelsadeeq commented at 5:56 pm on September 24, 2023:

    In b0dda173e1784f177152349ff7002345f054f888 [refactor] rewrite vTxHashes as a vector of CTransactionRef txns_randomized is introduced in next commit d7eec401ba7c7b4c8fc58e39e358ba4518da97d7

    Does this belongs to this commit?

  18. ismaelsadeeq commented at 5:59 pm on September 24, 2023: member

    Concept ACK on the changes on in this PR.

    This refactor all seems relevant. The commits are really atomic and easier to review. 💯

  19. TheCharlatan force-pushed on Sep 25, 2023
  20. TheCharlatan commented at 10:56 am on September 25, 2023: contributor
    Rebased a5bdd7eacea7be11e064aa7a9a4f12624dfe53fd -> f0577ea4c428421a85e98bfeb2fbd5e617a4622a (simplifyMemPoolInteractions_0 -> simplifyMemPoolInteractions_1, compare)
  21. DrahtBot removed the label CI failed on Sep 25, 2023
  22. glozow requested review from stickies-v on Oct 3, 2023
  23. DrahtBot added the label Needs rebase on Oct 30, 2023
  24. TheCharlatan force-pushed on Oct 30, 2023
  25. TheCharlatan commented at 4:19 pm on October 30, 2023: contributor
    Rebased f0577ea4c428421a85e98bfeb2fbd5e617a4622a -> 5fb9ddf0ab423dae5c0056eda10b02817927fcc4 (simplifyMemPoolInteractions_1 -> simplifyMemPoolInteractions_2, compare)
  26. DrahtBot removed the label Needs rebase on Oct 30, 2023
  27. DrahtBot added the label CI failed on Oct 30, 2023
  28. in src/node/interfaces.cpp:706 in 5fb9ddf0ab outdated
    698@@ -700,11 +699,9 @@ class ChainImpl : public Chain
    699     bool checkChainLimits(const CTransactionRef& tx) override
    700     {
    701         if (!m_node.mempool) return true;
    702-        LockPoints lp;
    703-        CTxMemPoolEntry entry(tx, 0, 0, 0, 0, false, 0, lp);
    704-        const CTxMemPool::Limits& limits{m_node.mempool->m_limits};
    705         LOCK(m_node.mempool->cs);
    


    ismaelsadeeq commented at 7:26 pm on October 30, 2023:
  29. TheCharlatan force-pushed on Oct 30, 2023
  30. TheCharlatan commented at 9:39 pm on October 30, 2023: contributor

    Updated 5fb9ddf0ab423dae5c0056eda10b02817927fcc4 -> e579bb98ba8977af284ba6914ffb2b1da7f34cdd (simplifyMemPoolInteractions_2 -> simplifyMemPoolInteractions_3, compare)

    • Addressed @ismaelsadeeq’s comment, fixed change ordering in commit (no net change to the code).
    • Fixed reverted change by cherry-picking commit e7e462eadc66e484ccb2f51967a0ad5f0d3bd41a [refactor] use CheckPackageLimits for checkChainLimits.
  31. DrahtBot removed the label CI failed on Oct 30, 2023
  32. in src/rpc/mempool.cpp:319 in 8e41f6555f outdated
    314@@ -315,8 +315,8 @@ static void entryToJSON(const CTxMemPool& pool, UniValue& info, const CTxMemPool
    315     info.pushKV("depends", depends);
    316 
    317     UniValue spent(UniValue::VARR);
    318-    const CTxMemPool::txiter& it = pool.mapTx.find(tx.GetHash());
    319-    const CTxMemPoolEntry::Children& children = it->GetMemPoolChildrenConst();
    320+    const auto it = pool.GetIter(tx.GetHash());
    321+    const CTxMemPoolEntry::Children& children = (*CHECK_NONFATAL(it))->GetMemPoolChildrenConst();
    


    ismaelsadeeq commented at 12:14 pm on November 1, 2023:

    why not just?

    0    const CTxMemPoolEntry::Children& children = e.GetMemPoolChildrenConst();
    

    is there a reason why you are getting the iterator of the txid?


    TheCharlatan commented at 7:30 pm on November 1, 2023:
    Huh, yeah. That is more straight forward.
  33. in src/test/mempool_tests.cpp:191 in 3c7543cc98 outdated
    190@@ -191,7 +191,7 @@ BOOST_AUTO_TEST_CASE(MempoolIndexingTest)
    191     CheckSort<descendant_score>(pool, sortedOrder);
    


    ismaelsadeeq commented at 12:38 pm on November 1, 2023:

    In 3c7543cc98efb8d5e38b8c18f2f184c9272cd092 " [refactor] remove access to mapTx in mempool_tests.cpp "

    nit: In the commit message there is still mapTx access in mempool_tests.cpp CheckSort test maybe the commit message should be [refactor] use CTxMemPool helper method to get a transaction mempool iterator.

  34. in src/policy/rbf.cpp:39 in 896684ec10 outdated
    35@@ -35,7 +36,7 @@ RBFTransactionState IsRBFOptIn(const CTransaction& tx, const CTxMemPool& pool)
    36 
    37     // If all the inputs have nSequence >= maxint-1, it still might be
    38     // signaled for RBF if any unconfirmed parents have signaled.
    39-    const CTxMemPoolEntry entry{*pool.mapTx.find(tx.GetHash())};
    40+    const CTxMemPoolEntry entry{**Assert(pool.GetIter(tx.GetHash()))};
    


    ismaelsadeeq commented at 1:36 pm on November 1, 2023:
    Are you assuming that the callers of IsRBFOptIn must pass a mempool transaction with the Assert?

    TheCharlatan commented at 7:29 pm on November 1, 2023:
    Yes, as I understand it that is what is checked in the code block above, so this should always hold.
  35. ismaelsadeeq commented at 1:50 pm on November 1, 2023: member

    Light Code review ACK e579bb98ba8977af284ba6914ffb2b1da7f34cdd Non expert look at the changes here, there are no any external usage of CTxMemPool mapTx.find. The changes here looks good to me

    Just a question and nits below. I see there are still some external access of mapTx in the codebase like mapTx.size, mapTx.get and mapTx.project.

  36. DrahtBot requested review from glozow on Nov 1, 2023
  37. TheCharlatan force-pushed on Nov 1, 2023
  38. TheCharlatan commented at 7:48 pm on November 1, 2023: contributor

    Thank you for the review @ismaelsadeeq!

    Updated e579bb98ba8977af284ba6914ffb2b1da7f34cdd -> 105a0f4db4ffdc25d3ad30300c949d46d5d8e647 (simplifyMemPoolInteractions_3 -> simplifyMemPoolInteractions_4, compare)

    • Added commit 105a0f4db4ffdc25d3ad30300c949d46d5d8e647 for removing access to mapTx in validation_block_tests.
    • Addressed @ismaelsadeeq’s comment, simplifying code by just using mempool entry directly.
    • Addressed @ismaelsadeeq’s comment, making commit message more precise.
  39. ismaelsadeeq commented at 12:11 pm on November 2, 2023: member
    Re ACK 105a0f4db4ffdc25d3ad30300c949d46d5d8e647
  40. in src/node/mini_miner.cpp:77 in bc21629f66 outdated
    70@@ -71,8 +71,11 @@ MiniMiner::MiniMiner(const CTxMemPool& mempool, const std::vector<COutPoint>& ou
    71 
    72     // Add every entry to m_entries_by_txid and m_entries, except the ones that will be replaced.
    


    ismaelsadeeq commented at 12:33 pm on November 2, 2023:
    Changes in this commit bc21629f66b261c42f9e08ff8925c39d14874653 is identical to the one in https://github.com/bitcoin/bitcoin/pull/28762/commits/d525998b07be066c97a36536b58f0b98d816be39 , but here the txmempool.h dependency from mini_miner.h, and the imports are not updated?
  41. in src/node/interfaces.cpp:809 in 105a0f4db4 outdated
    804@@ -806,8 +805,8 @@ class ChainImpl : public Chain
    805     {
    806         if (!m_node.mempool) return;
    807         LOCK2(::cs_main, m_node.mempool->cs);
    


    stickies-v commented at 3:30 pm on November 2, 2023:
    infoAll() holds its own CTxMempool::cs lock, and I think we don’t need that to add stuff to notifications? I’m also not quite sure if we need to be holding ::cs_main here?

    TheCharlatan commented at 10:54 am on November 10, 2023:
    This seems a bit out of scope of this PR, but I think you are right, I don’t see the need for ::cs_main here either. Maybe open a separate PR for this?
  42. TheCharlatan force-pushed on Nov 3, 2023
  43. TheCharlatan commented at 10:24 am on November 3, 2023: contributor

    Updated 105a0f4db4ffdc25d3ad30300c949d46d5d8e647 -> 65839d6267af1f5e0e04aded72cbfa23b56a1237 (simplifyMemPoolInteractions_4 -> simplifyMemPoolInteractions_5, compare)

    • Addressed @ismaelsadeeq’s comment, pulling in the improved commit from #28762
    • Added a new first commit to introduce a GetEntry helper for retrieving mempool entries. This allows avoiding the usage of GetIter just for retrieving a mempool entry.
    • Use this new GetEntry helper over GetIter where applicable.
    • Refactored the blockencodings_tests commit to use a simpler helper function.
    • Dropped commit 5633e6770538c83cc5a75cf926347a709ae7e2c0, since GetEntry is now used instead.
  44. TheCharlatan force-pushed on Nov 3, 2023
  45. TheCharlatan commented at 3:37 pm on November 3, 2023: contributor

    Rebased 65839d6267af1f5e0e04aded72cbfa23b56a1237 -> fc5e4dea6ffa21191e852a79103a2dfc9c241df1 (simplifyMemPoolInteractions_5 -> simplifyMemPoolInteractions_6, compare)

    • Dropped the commit that was integrated in #28762 e3b2e630b219ca15fe0b2640ca422712c86ac33d
  46. in src/txmempool.h:687 in c93f52a881 outdated
    683@@ -684,6 +684,8 @@ class CTxMemPool
    684         return (mapTx.count(gtxid.GetHash()) != 0);
    685     }
    686 
    687+    std::optional<CTxMemPoolEntry::CTxMemPoolEntryRef> GetEntry(const uint256& hash) const LIFETIMEBOUND;
    


    glozow commented at 4:50 pm on November 3, 2023:

    Concept ACK to handing out references to the entry instead of iterators into mapTx

    ISTM that this should require holding CTxMemPool::cs?


    TheCharlatan commented at 5:29 pm on November 3, 2023:
    Yeah, missed that. Will add.
  47. TheCharlatan force-pushed on Nov 3, 2023
  48. TheCharlatan commented at 6:20 pm on November 3, 2023: contributor

    Updated fc5e4dea6ffa21191e852a79103a2dfc9c241df1 -> a20c1d2b81a702f97267d4f098f13ed80e9320cf (simplifyMemPoolInteractions_6 -> simplifyMemPoolInteractions_7, compare)

    • Addressed @glozow’s comment, making the locked mutex a requirement for calling GetEntry.
  49. DrahtBot added the label CI failed on Nov 3, 2023
  50. Sjors commented at 7:52 am on November 6, 2023: member

    The first commit ee79f05f1fef57bfbea24333a46e525385d02319 introduces a dangling reference warning for me. gcc 13.1.0 on Ubuntu 23.04.

     0./configure --enable-suppress-external-warnings --disable-fuzz-binary --without-gui
     1make
     2...
     3
     4validation.cpp: In member function PackageMempoolAcceptResult {anonymous}::MemPoolAccept::AcceptPackage(const Package&, ATMPArgs&):
     5validation.cpp:1488:36: warning: possibly dangling reference to a temporary [-Wdangling-reference]
     6 1488 |             const CTxMemPoolEntry& entry = Assert(m_pool.GetEntry(txid)).value();
     7      |                                    ^~~~~
     8validation.cpp:1488:80: note: the temporary was destroyed at the end of the full expression (&(& inline_assertion_check<true, std::optional<std::reference_wrapper<const CTxMemPoolEntry> > >((&(({anonymous}::MemPoolAccept*)this)->{anonymous}::MemPoolAccept::m_pool)->CTxMemPool::GetEntry((* &(& txid)->transaction_identifier<false>::operator const uint256&())), ((const char*)"validation.cpp"), 1488, ((const char*)(& __func__)), ((const char*)"m_pool.GetEntry(txid)")))->std::optional<std::reference_wrapper<const CTxMemPoolEntry> >::value())->std::reference_wrapper<const CTxMemPoolEntry>::operator const CTxMemPoolEntry&()
     9 1488 |             const CTxMemPoolEntry& entry = Assert(m_pool.GetEntry(txid)).value();
    10      |                                                                                ^
    
  51. in src/txmempool.cpp:856 in ee79f05f1f outdated
    849@@ -850,6 +850,14 @@ std::vector<TxMempoolInfo> CTxMemPool::infoAll() const
    850     return ret;
    851 }
    852 
    853+std::optional<CTxMemPoolEntry::CTxMemPoolEntryRef> CTxMemPool::GetEntry(const uint256& hash) const
    854+{
    855+    indexed_transaction_set::const_iterator i = mapTx.find(hash);
    856+    if (i == mapTx.end())
    


    Sjors commented at 9:02 am on November 6, 2023:
    ee79f05f1fef57bfbea24333a46e525385d02319: {
  52. in src/node/interfaces.cpp:708 in 6b6793e683 outdated
    701@@ -702,9 +702,9 @@ class ChainImpl : public Chain
    702         if (!m_node.mempool) return true;
    703         LockPoints lp;
    704         CTxMemPoolEntry entry(tx, 0, 0, 0, 0, false, 0, lp);
    705-        const CTxMemPool::Limits& limits{m_node.mempool->m_limits};
    706         LOCK(m_node.mempool->cs);
    707-        return m_node.mempool->CalculateMemPoolAncestors(entry, limits).has_value();
    708+        std::string err_string;
    709+        return m_node.mempool->CheckPackageLimits({tx}, entry.GetTxSize(), err_string);
    


    Sjors commented at 9:27 am on November 6, 2023:

    6b6793e6835dd2ddf367490362042bf34eb260a3: The only caller checkChainLimits of is the wallet’s CreateTransactionInternal, which will say “Transaction has too long of a mempool chain” regardless of which limit is hit.

    A (wallet) followup should propagate err_string up.

  53. Sjors commented at 9:33 am on November 6, 2023: member
    Concept ACK. Lightly reviewed the refactoring (as of a20c1d2b81a702f97267d4f098f13ed80e9320cf) which all looks reasonable, but I’m not very familiar with mempool code.
  54. in src/validation.cpp:1488 in ee79f05f1f outdated
    1484@@ -1485,9 +1485,8 @@ PackageMempoolAcceptResult MemPoolAccept::AcceptPackage(const Package& package,
    1485             // transactions that are already in the mempool, and only call AcceptMultipleTransactions() with
    1486             // the new transactions. This ensures we don't double-count transaction counts and sizes when
    1487             // checking ancestor/descendant limits, or double-count transaction fees for fee-related policy.
    1488-            auto iter = m_pool.GetIter(txid);
    1489-            assert(iter != std::nullopt);
    1490-            results_final.emplace(wtxid, MempoolAcceptResult::MempoolTx(iter.value()->GetTxSize(), iter.value()->GetFee()));
    1491+            const CTxMemPoolEntry& entry = Assert(m_pool.GetEntry(txid)).value();
    


    Sjors commented at 9:42 am on November 6, 2023:

    ee79f05f1fef57bfbea24333a46e525385d02319: this fixes the dangling warning:

    0const CTxMemPoolEntry& entry{Assert(m_pool.GetEntry(txid)).value()};
    

    TheCharlatan commented at 9:44 am on November 6, 2023:
    Thanks, might be a good idea to bracket initialize in other places too.
  55. TheCharlatan force-pushed on Nov 6, 2023
  56. TheCharlatan commented at 10:29 am on November 6, 2023: contributor

    Updated a20c1d2b81a702f97267d4f098f13ed80e9320cf -> 836a0a7566e166556f80303a9b3dd9560587159c (simplifyMemPoolInteractions_7 -> simplifyMemPoolInteractions_8, compare)

    • Addressed @Sjors’s comment, adding brackets to if statement
    • Applied @Sjors’s patch and also bracket initialize some other vars.
  57. TheCharlatan commented at 11:04 am on November 6, 2023: contributor

    Updated 836a0a7566e166556f80303a9b3dd9560587159c -> 11b8735852bd7439c4a07c88d07d7dfb13a3c7b7 (simplifyMemPoolInteractions_8 -> simplifyMemPoolInteractions_9, compare)

    • Fixed msvc build error by using auto instead of CTxMemPoolEntry&, which led to implicit conversions.
  58. TheCharlatan force-pushed on Nov 6, 2023
  59. in src/txmempool.h:687 in 11b8735852 outdated
    683@@ -684,6 +684,8 @@ class CTxMemPool
    684         return (mapTx.count(gtxid.GetHash()) != 0);
    685     }
    686 
    687+    std::optional<CTxMemPoolEntry::CTxMemPoolEntryRef> GetEntry(const uint256& hash) const LIFETIMEBOUND EXCLUSIVE_LOCKS_REQUIRED(cs);
    


    maflcko commented at 12:16 pm on November 6, 2023:
    0    CTxMemPoolEntry* GetEntry(const uint256& hash) const LIFETIMEBOUND EXCLUSIVE_LOCKS_REQUIRED(cs);
    

    Instead of optional ref, could just use a pointer?


    TheCharlatan commented at 12:40 pm on November 6, 2023:
    Yeah, I thought this would better convey its semantics, but with all the force pushes over the past two hours, it seems like it is not intuitive to work with, so I’ll revert to using a pointer.
  60. TheCharlatan force-pushed on Nov 6, 2023
  61. DrahtBot removed the label CI failed on Nov 6, 2023
  62. TheCharlatan force-pushed on Nov 6, 2023
  63. TheCharlatan commented at 1:59 pm on November 6, 2023: contributor

    Updated 66a4c881e3387b55b26cd2a890bd0d5d522c6727 -> 995aa6b9cb5d1ea685a5d2ac6767d21b373e4c84 (simplifyMemPoolInteractions_9 -> simplifyMemPoolInteractions_10, compare)

    • Addressed @maflcko’s comment, changed toCTxMemPoolEntry* GetEntry(...).
  64. in src/txmempool.cpp:855 in 6903d8aa37 outdated
    849@@ -850,6 +850,15 @@ std::vector<TxMempoolInfo> CTxMemPool::infoAll() const
    850     return ret;
    851 }
    852 
    853+const CTxMemPoolEntry* CTxMemPool::GetEntry(const uint256& hash) const
    854+{
    855+    indexed_transaction_set::const_iterator i = mapTx.find(hash);
    


    ismaelsadeeq commented at 12:39 pm on November 8, 2023:

    Assert?

    0    AssertLockHeld(cs);
    1    indexed_transaction_set::const_iterator i = mapTx.find(hash);
    
  65. in src/test/miniminer_tests.cpp:179 in 6903d8aa37 outdated
    178@@ -179,9 +179,9 @@ BOOST_FIXTURE_TEST_CASE(miniminer_1p1c, TestChain100Setup)
    179     };
    


    ismaelsadeeq commented at 1:19 pm on November 8, 2023:

    nit https://github.com/bitcoin/bitcoin/blob/6903d8aa37e01d16aea0b53c3729f8f92c7d2bf0/src/test/miniminer_tests.cpp#L97 Can also be

    0BOOST_CHECK(negative_modified_fees < feerate_zero.GetFee(pool.GetEntry(tx_mod_negative->GetHash())->GetTxSize()));
    
  66. ismaelsadeeq approved
  67. ismaelsadeeq commented at 1:25 pm on November 8, 2023: member
    re ACK 995aa6b9cb5d1ea685a5d2ac6767d21b373e4c84
  68. DrahtBot requested review from Sjors on Nov 8, 2023
  69. DrahtBot requested review from glozow on Nov 8, 2023
  70. in src/txmempool.h:687 in 6903d8aa37 outdated
    683@@ -684,6 +684,8 @@ class CTxMemPool
    684         return (mapTx.count(gtxid.GetHash()) != 0);
    685     }
    686 
    687+    const CTxMemPoolEntry* GetEntry(const uint256& hash) const LIFETIMEBOUND EXCLUSIVE_LOCKS_REQUIRED(cs);
    


    glozow commented at 3:29 pm on November 8, 2023:
    6903d8aa37e01d16aea0b53c3729f8f92c7d2bf0 Could you use Txid instead? or GenTxid
  71. in src/test/blockencodings_tests.cpp:65 in b7fa64593e outdated
    61@@ -62,7 +62,7 @@ BOOST_AUTO_TEST_CASE(SimpleRoundTripTest)
    62 
    63     LOCK2(cs_main, pool.cs);
    64     pool.addUnchecked(entry.FromTx(block.vtx[2]));
    65-    BOOST_CHECK_EQUAL(pool.mapTx.find(block.vtx[2]->GetHash())->GetSharedTx().use_count(), SHARED_TX_OFFSET + 0);
    66+    BOOST_CHECK_EQUAL(pool.get(block.vtx[2]->GetHash()).use_count(), SHARED_TX_OFFSET + 0);
    


    glozow commented at 3:34 pm on November 8, 2023:
    Wow TIL there’s a get!
  72. in src/node/miner.h:145 in aedfef28af outdated
    141@@ -142,7 +142,7 @@ class BlockAssembler
    142     uint64_t nBlockTx;
    143     uint64_t nBlockSigOpsCost;
    144     CAmount nFees;
    145-    CTxMemPool::setEntries inBlock;
    146+    std::unordered_set<uint256, SaltedTxidHasher> inBlock;
    


    glozow commented at 3:37 pm on November 8, 2023:
    aedfef28af6e8c5bd8833fb63e2ef4f3264f934c could use Txid for this as well
  73. in src/node/miner.cpp:301 in aedfef28af outdated
    297@@ -298,7 +298,7 @@ void BlockAssembler::addPackageTxs(const CTxMemPool& mempool, int& nPackagesSele
    298     // because some of their txs are already in the block
    299     indexed_modified_transaction_set mapModifiedTx;
    300     // Keep track of entries that failed inclusion, to avoid duplicate work
    301-    CTxMemPool::setEntries failedTx;
    302+    std::set<uint256> failedTx;
    


    glozow commented at 3:37 pm on November 8, 2023:
    same, should be Txid
  74. glozow commented at 3:38 pm on November 8, 2023: member
    LGTM 995aa6b9cb5d1ea685a5d2ac6767d21b373e4c84, just a couple introductions of uint256 that should probably be the type-safe ones instead
  75. DrahtBot requested review from glozow on Nov 8, 2023
  76. glozow commented at 3:40 pm on November 8, 2023: member
    drahty pong
  77. DrahtBot requested review from glozow on Nov 8, 2023
  78. TheCharlatan force-pushed on Nov 8, 2023
  79. TheCharlatan commented at 6:25 pm on November 8, 2023: contributor

    Updated 995aa6b9cb5d1ea685a5d2ac6767d21b373e4c84 -> 3e7595b11bdad260efb39adc42677ed0beae186d (simplifyMemPoolInteractions_10 -> simplifyMemPoolInteractions_11, compare)

  80. in src/rpc/mempool.cpp:348 in 65c4b3bac0 outdated
    342@@ -344,10 +343,12 @@ UniValue MempoolToJSON(const CTxMemPool& pool, bool verbose, bool include_mempoo
    343         }
    344         LOCK(pool.cs);
    345         UniValue o(UniValue::VOBJ);
    346-        for (const CTxMemPoolEntry& e : pool.mapTx) {
    347-            const uint256& hash = e.GetTx().GetHash();
    348+        for (const auto& txinfo : pool.infoAll()) {
    349+            const uint256& hash = txinfo.tx->GetHash();
    350+            const auto entry{pool.GetEntry(Txid::FromUint256(hash))};
    


    glozow commented at 6:54 pm on November 8, 2023:

    nit 65c4b3bac0894dfdf7e1c756697f9dd0da1f97e7: don’t need to go to uint256 and back again

    0            const Txid& hash = txinfo.tx->GetHash();
    1            const auto entry{pool.GetEntry(hash)};
    
  81. glozow commented at 6:57 pm on November 8, 2023: member
    ACK 3e7595b11bdad260efb39adc42677ed0beae186d
  82. DrahtBot requested review from ismaelsadeeq on Nov 8, 2023
  83. TheCharlatan force-pushed on Nov 8, 2023
  84. TheCharlatan commented at 10:29 pm on November 8, 2023: contributor

    Updated 3e7595b11bdad260efb39adc42677ed0beae186d -> 2be5e799ba623f969f5ffc59667a5bc6799df290 (simplifyMemPoolInteractions_11 -> simplifyMemPoolInteractions_12, compare)

  85. glozow commented at 9:27 am on November 9, 2023: member
    reACK 2be5e799ba623f969f5ffc59667a5bc6799df290
  86. in src/rpc/mempool.cpp:349 in 2be5e799ba outdated
    342@@ -344,10 +343,12 @@ UniValue MempoolToJSON(const CTxMemPool& pool, bool verbose, bool include_mempoo
    343         }
    344         LOCK(pool.cs);
    345         UniValue o(UniValue::VOBJ);
    346-        for (const CTxMemPoolEntry& e : pool.mapTx) {
    347-            const uint256& hash = e.GetTx().GetHash();
    348+        for (const auto& txinfo : pool.infoAll()) {
    349+            const Txid& hash = txinfo.tx->GetHash();
    350+            const auto entry{pool.GetEntry(hash)};
    351+            CHECK_NONFATAL(entry != nullptr);
    


    stickies-v commented at 12:00 pm on November 9, 2023:

    nit: could simplify

    0            const auto entry{CHECK_NONFATAL(pool.GetEntry(hash))};
    
  87. in src/policy/rbf.cpp:40 in 2be5e799ba outdated
    35@@ -35,8 +36,9 @@ RBFTransactionState IsRBFOptIn(const CTransaction& tx, const CTxMemPool& pool)
    36 
    37     // If all the inputs have nSequence >= maxint-1, it still might be
    38     // signaled for RBF if any unconfirmed parents have signaled.
    39-    const CTxMemPoolEntry entry{*pool.mapTx.find(tx.GetHash())};
    40-    auto ancestors{pool.AssumeCalculateMemPoolAncestors(__func__, entry, CTxMemPool::Limits::NoLimits(),
    41+    const auto entry{pool.GetEntry(tx.GetHash())};
    42+    Assert(entry != nullptr);
    


    stickies-v commented at 12:04 pm on November 9, 2023:

    nit: one-liners seem appropriate? (+ here, here)

    0    const auto entry{Assert(pool.GetEntry(tx.GetHash()))};
    
  88. in src/txmempool.h:395 in 2be5e799ba outdated
    391@@ -392,7 +392,7 @@ class CTxMemPool
    392     indexed_transaction_set mapTx GUARDED_BY(cs);
    393 
    394     using txiter = indexed_transaction_set::nth_index<0>::type::const_iterator;
    395-    std::vector<std::pair<uint256, txiter>> vTxHashes GUARDED_BY(cs); //!< All tx witness hashes/entries in mapTx, in random order
    396+    std::vector<CTransactionRef> txns_randomized GUARDED_BY(cs); //!< All transactions in mapTx, in random order
    


    stickies-v commented at 12:10 pm on November 9, 2023:
    nit: I think a brief rationale behind why we need this change would be good to add to the commit message of “[refactor] rewrite vTxHashes as a vector of CTransactionRef”
  89. in src/txmempool.cpp:862 in 2be5e799ba outdated
    857+    AssertLockHeld(cs);
    858+    indexed_transaction_set::const_iterator i = mapTx.find(txid);
    859+    if (i == mapTx.end()) {
    860+        return nullptr;
    861+    }
    862+    return &*i;
    


    stickies-v commented at 12:16 pm on November 9, 2023:

    nit: Given that we’re doing not any meaninful operations with i I think using auto makes sense, and the ternary operator also seems appropriate here, simplifying this to:

    0    const auto i{mapTx.find(txid)};
    1    return i == mapTx.end() ? nullptr : &(*i);
    
  90. in src/test/miniminer_tests.cpp:451 in 2be5e799ba outdated
    446@@ -447,15 +447,15 @@ BOOST_FIXTURE_TEST_CASE(miniminer_overlap, TestChain100Setup)
    447     // tx3's feerate is lower than tx2's. same fee, different weight.
    448     BOOST_CHECK(tx2_feerate > tx3_feerate);
    449     const auto tx3_anc_feerate = CFeeRate(low_fee + med_fee + high_fee + high_fee, tx_vsizes[0] + tx_vsizes[1] + tx_vsizes[2] + tx_vsizes[3]);
    450-    const auto tx3_iter = pool.GetIter(tx3->GetHash());
    451-    BOOST_CHECK(tx3_anc_feerate == CFeeRate(tx3_iter.value()->GetModFeesWithAncestors(), tx3_iter.value()->GetSizeWithAncestors()));
    452+    const auto tx3_entry = pool.GetEntry(tx3->GetHash());
    453+    BOOST_CHECK(tx3_anc_feerate == CFeeRate(tx3_entry->GetModFeesWithAncestors(), tx3_entry->GetSizeWithAncestors()));
    


    stickies-v commented at 12:23 pm on November 9, 2023:

    This seems like a slight regression: tx3_iter.value() would throw if the hash was not found, whereas tx3_entry->GetSizeWithAncestors() is UB. I think it would be good to add inline Asserts to the entry initialization (and add list initialization while you’re at it)?

    0const auto tx3_entry{Assert(pool.GetEntry(tx3->GetHash()))};
    

    Similar concerns here and here.

  91. in src/node/interfaces.cpp:811 in 2be5e799ba outdated
    806@@ -806,8 +807,8 @@ class ChainImpl : public Chain
    807     {
    808         if (!m_node.mempool) return;
    809         LOCK2(::cs_main, m_node.mempool->cs);
    810-        for (const CTxMemPoolEntry& entry : m_node.mempool->mapTx) {
    811-            notifications.transactionAddedToMempool(entry.GetSharedTx());
    812+        for (const auto& txinfo : m_node.mempool->infoAll()) {
    813+            notifications.transactionAddedToMempool(txinfo.tx);
    


    stickies-v commented at 12:48 pm on November 9, 2023:

    It’s probably not too important, but this is a bit of a performance and memory regression to construct the vector of TxMempoolInfo when really we only need the underlying txref.

    I think it could make sense to add std::vector<const CTxMemPoolEntry*> entryAll() const EXCLUSIVE_LOCKS_REQUIRED(cs); to reduce memory and computational overhead and as a bonus make the code a bit easier to read when we just want to iterate over the mempool entries?

     0diff --git a/src/node/interfaces.cpp b/src/node/interfaces.cpp
     1index a73530c142..155546c3a1 100644
     2--- a/src/node/interfaces.cpp
     3+++ b/src/node/interfaces.cpp
     4@@ -807,8 +807,8 @@ public:
     5     {
     6         if (!m_node.mempool) return;
     7         LOCK2(::cs_main, m_node.mempool->cs);
     8-        for (const auto& txinfo : m_node.mempool->infoAll()) {
     9-            notifications.transactionAddedToMempool(txinfo.tx);
    10+        for (const auto& entry : m_node.mempool->entryAll()) {
    11+            notifications.transactionAddedToMempool(Assert(entry)->GetSharedTx());
    12         }
    13     }
    14     bool hasAssumedValidChain() override
    15diff --git a/src/rpc/mempool.cpp b/src/rpc/mempool.cpp
    16index 4093eb99af..82fe394b45 100644
    17--- a/src/rpc/mempool.cpp
    18+++ b/src/rpc/mempool.cpp
    19@@ -343,16 +343,13 @@ UniValue MempoolToJSON(const CTxMemPool& pool, bool verbose, bool include_mempoo
    20         }
    21         LOCK(pool.cs);
    22         UniValue o(UniValue::VOBJ);
    23-        for (const auto& txinfo : pool.infoAll()) {
    24-            const Txid& hash = txinfo.tx->GetHash();
    25-            const auto entry{pool.GetEntry(hash)};
    26-            CHECK_NONFATAL(entry != nullptr);
    27+        for (const auto& entry : pool.entryAll()) {
    28             UniValue info(UniValue::VOBJ);
    29-            entryToJSON(pool, info, *entry);
    30+            entryToJSON(pool, info, *CHECK_NONFATAL(entry));
    31             // Mempool has unique entries so there is no advantage in using
    32             // UniValue::pushKV, which checks if the key already exists in O(N).
    33             // UniValue::pushKVEnd is used instead which currently is O(1).
    34-            o.pushKVEnd(hash.ToString(), info);
    35+            o.pushKVEnd(entry->GetTx().ToString(), info);
    36         }
    37         return o;
    38     } else {
    39diff --git a/src/txmempool.cpp b/src/txmempool.cpp
    40index 684116c011..542c904a1a 100644
    41--- a/src/txmempool.cpp
    42+++ b/src/txmempool.cpp
    43@@ -838,6 +838,20 @@ static TxMempoolInfo GetInfo(CTxMemPool::indexed_transaction_set::const_iterator
    44     return TxMempoolInfo{it->GetSharedTx(), it->GetTime(), it->GetFee(), it->GetTxSize(), it->GetModifiedFee() - it->GetFee()};
    45 }
    46 
    47+std::vector<const CTxMemPoolEntry*> CTxMemPool::entryAll() const
    48+{
    49+    AssertLockHeld(cs);
    50+    auto iters{GetSortedDepthAndScore()};
    51+
    52+    std::vector<const CTxMemPoolEntry*> ret;
    53+    ret.reserve(mapTx.size());
    54+    for (const auto& it : iters) {
    55+        ret.push_back(&(*it));
    56+    }
    57+
    58+    return ret;
    59+}
    60+
    61 std::vector<TxMempoolInfo> CTxMemPool::infoAll() const
    62 {
    63     LOCK(cs);
    64diff --git a/src/txmempool.h b/src/txmempool.h
    65index e15b2972a4..7eb205b2df 100644
    66--- a/src/txmempool.h
    67+++ b/src/txmempool.h
    68@@ -697,6 +697,7 @@ public:
    69     /** Returns info for a transaction if its entry_sequence < last_sequence */
    70     TxMempoolInfo info_for_relay(const GenTxid& gtxid, uint64_t last_sequence) const;
    71 
    72+    std::vector<const CTxMemPoolEntry*> entryAll() const EXCLUSIVE_LOCKS_REQUIRED(cs);
    73     std::vector<TxMempoolInfo> infoAll() const;
    74 
    75     size_t DynamicMemoryUsage() const;
    

    TheCharlatan commented at 5:01 pm on November 9, 2023:
    Helpers are good, so I’ll add this patch as a separate commit.
  92. ismaelsadeeq approved
  93. ismaelsadeeq commented at 1:11 pm on November 9, 2023: member
    ACK 2be5e799ba623f969f5ffc59667a5bc6799df290 🥘
  94. in src/rpc/mempool.cpp:319 in 2be5e799ba outdated
    314@@ -315,8 +315,7 @@ static void entryToJSON(const CTxMemPool& pool, UniValue& info, const CTxMemPool
    315     info.pushKV("depends", depends);
    316 
    317     UniValue spent(UniValue::VARR);
    318-    const CTxMemPool::txiter& it = pool.mapTx.find(tx.GetHash());
    319-    const CTxMemPoolEntry::Children& children = it->GetMemPoolChildrenConst();
    320+    const CTxMemPoolEntry::Children& children{e.GetMemPoolChildrenConst()};
    321     for (const CTxMemPoolEntry& child : children) {
    


    stickies-v commented at 1:36 pm on November 9, 2023:

    nit: can simplify

    0    for (const auto& child : e.GetMemPoolChildrenConst()) {
    
  95. in src/test/mempool_tests.cpp:194 in 2be5e799ba outdated
    190@@ -191,7 +191,7 @@ BOOST_AUTO_TEST_CASE(MempoolIndexingTest)
    191     CheckSort<descendant_score>(pool, sortedOrder);
    192 
    193     CTxMemPool::setEntries setAncestors;
    194-    setAncestors.insert(pool.mapTx.find(tx6.GetHash()));
    195+    setAncestors.insert(*pool.GetIter(tx6.GetHash()));
    


    stickies-v commented at 1:43 pm on November 9, 2023:

    nit: not a regression, but using .value() seems safer (also for tx7, tx8, tx9 further down)

    0    setAncestors.insert(pool.GetIter(tx6.GetHash()).value());
    
  96. in src/txmempool.cpp:522 in 2be5e799ba outdated
    523-        vTxHashes.pop_back();
    524-        if (vTxHashes.size() * 2 < vTxHashes.capacity())
    525-            vTxHashes.shrink_to_fit();
    526+    if (txns_randomized.size() > 1) {
    527+        // Update idx_randomized of the to-be-moved entry.
    528+        get_iter_from_wtxid(txns_randomized.back()->GetWitnessHash())->idx_randomized = it->idx_randomized;
    


    stickies-v commented at 1:57 pm on November 9, 2023:

    nit: I think we can just directly use GetIter?

    0        GetIter(txns_randomized.back()->GetHash()).value()->idx_randomized = it->idx_randomized;
    
  97. in src/blockencodings.cpp:111 in 2be5e799ba outdated
    106@@ -107,12 +107,13 @@ ReadStatus PartiallyDownloadedBlock::InitData(const CBlockHeaderAndShortTxIDs& c
    107     std::vector<bool> have_txn(txn_available.size());
    108     {
    109     LOCK(pool->cs);
    110-    for (size_t i = 0; i < pool->vTxHashes.size(); i++) {
    111-        uint64_t shortid = cmpctblock.GetShortID(pool->vTxHashes[i].first);
    112+    for (size_t i = 0; i < pool->txns_randomized.size(); i++) {
    113+        const auto& tx = pool->txns_randomized[i];
    


    stickies-v commented at 2:00 pm on November 9, 2023:

    nit (although I realize this diff is from a scripted-diff, so may not be worth updating here)

    0    for (const auto& tx : pool->txns_randomized) {
    
  98. stickies-v commented at 2:10 pm on November 9, 2023: contributor

    Code Review ACK 2be5e799ba623f969f5ffc59667a5bc6799df290

    Left a number of suggestions: some about performance improvements, some about avoiding UB, and some general style/concise nits, but they can all be done in a follow-up too.

  99. TheCharlatan force-pushed on Nov 9, 2023
  100. TheCharlatan commented at 8:57 pm on November 9, 2023: contributor

    Thank you for the review @stickies-v!

    Updated 2be5e799ba623f969f5ffc59667a5bc6799df290 -> f2d8e17851da215d0b85ffad9d80e50eb7b2b363 (simplifyMemPoolInteractions_12 -> simplifyMemPoolInteractions_13, compare)

    • Addressed @stickies-v’s comment, adding a new first commit implementing the suggested patch for a more efficient helper method for iterating through mempool transactions. The affected lines now retrieve a vector of CTxMemPoolEntry* through this new entryAll() instead of calling infoAll().
    • Addressed @stickies-v’s comment_1, comment_2, comment_3, making assertion checks more compact and adding some in the tests for clarity where required.
    • Addressed @stickies-v’s comment, adding a brief rationale to the message of the commit introducing vTxHashes as a vector of CTransactionRef.
    • Addressed @stickies-v’s comment, making GetEntry method body more concise.
    • Addressed @stickies-v’s comment, simplifying looping over GetMemPoolChildrenConst().
    • Addressed @stickies-v’s comment, explicitly calling value() on the option returned by GetIter(...).
    • Addressed @stickies-v’s comment, replacing the get_iter_from_wtxid call in removeUnchecked with the more widely used GetIter.
    • Addressed @stickies-v’s comment, replacing the for loop iterating over txns_randomized in blockencodings.cpp with a range based for loop.
  101. DrahtBot added the label CI failed on Nov 9, 2023
  102. TheCharlatan force-pushed on Nov 9, 2023
  103. DrahtBot removed the label CI failed on Nov 9, 2023
  104. in src/txmempool.h:700 in f2d8e17851 outdated
    696@@ -695,6 +697,7 @@ class CTxMemPool
    697     /** Returns info for a transaction if its entry_sequence < last_sequence */
    698     TxMempoolInfo info_for_relay(const GenTxid& gtxid, uint64_t last_sequence) const;
    699 
    700+    std::vector<const CTxMemPoolEntry*> entryAll() const EXCLUSIVE_LOCKS_REQUIRED(cs);
    


    maflcko commented at 10:34 am on November 10, 2023:

    first commit: I think here the result can not be nullptr, so a reference makes more sense, no?

    diff:

     0diff --git a/src/node/interfaces.cpp b/src/node/interfaces.cpp
     1index e019ab9595..3930280797 100644
     2--- a/src/node/interfaces.cpp
     3+++ b/src/node/interfaces.cpp
     4@@ -806,8 +806,8 @@ public:
     5     {
     6         if (!m_node.mempool) return;
     7         LOCK2(::cs_main, m_node.mempool->cs);
     8-        for (const auto& entry : m_node.mempool->entryAll()) {
     9-            notifications.transactionAddedToMempool(Assert(entry)->GetSharedTx());
    10+        for (const CTxMemPoolEntry& entry : m_node.mempool->entryAll()) {
    11+            notifications.transactionAddedToMempool(entry.GetSharedTx());
    12         }
    13     }
    14     bool hasAssumedValidChain() override
    15diff --git a/src/rpc/mempool.cpp b/src/rpc/mempool.cpp
    16index 3767ae0b5d..dc21385a3d 100644
    17--- a/src/rpc/mempool.cpp
    18+++ b/src/rpc/mempool.cpp
    19@@ -345,13 +345,13 @@ UniValue MempoolToJSON(const CTxMemPool& pool, bool verbose, bool include_mempoo
    20         }
    21         LOCK(pool.cs);
    22         UniValue o(UniValue::VOBJ);
    23-        for (const auto& e : pool.entryAll()) {
    24+        for (const CTxMemPoolEntry& e : pool.entryAll()) {
    25             UniValue info(UniValue::VOBJ);
    26-            entryToJSON(pool, info, *CHECK_NONFATAL(e));
    27+            entryToJSON(pool, info, e);
    28             // Mempool has unique entries so there is no advantage in using
    29             // UniValue::pushKV, which checks if the key already exists in O(N).
    30             // UniValue::pushKVEnd is used instead which currently is O(1).
    31-            o.pushKVEnd(e->GetTx().GetHash().ToString(), info);
    32+            o.pushKVEnd(e.GetTx().GetHash().ToString(), info);
    33         }
    34         return o;
    35     } else {
    36diff --git a/src/txmempool.cpp b/src/txmempool.cpp
    37index 3698374c26..e5347cbf06 100644
    38--- a/src/txmempool.cpp
    39+++ b/src/txmempool.cpp
    40@@ -836,16 +836,15 @@ static TxMempoolInfo GetInfo(CTxMemPool::indexed_transaction_set::const_iterator
    41     return TxMempoolInfo{it->GetSharedTx(), it->GetTime(), it->GetFee(), it->GetTxSize(), it->GetModifiedFee() - it->GetFee()};
    42 }
    43 
    44-std::vector<const CTxMemPoolEntry*> CTxMemPool::entryAll() const
    45+std::vector<std::reference_wrapper<const CTxMemPoolEntry>> CTxMemPool::entryAll() const
    46 {
    47     AssertLockHeld(cs);
    48 
    49-    std::vector<const CTxMemPoolEntry*> ret;
    50+    std::vector<std::reference_wrapper<const CTxMemPoolEntry>> ret;
    51     ret.reserve(mapTx.size());
    52     for (const auto& it : GetSortedDepthAndScore()) {
    53-        ret.push_back(&(*it));
    54+        ret.push_back(*it);
    55     }
    56-
    57     return ret;
    58 }
    59 
    60diff --git a/src/txmempool.h b/src/txmempool.h
    61index a4ba2c4db2..c58dbadd83 100644
    62--- a/src/txmempool.h
    63+++ b/src/txmempool.h
    64@@ -695,7 +695,7 @@ public:
    65     /** Returns info for a transaction if its entry_sequence < last_sequence */
    66     TxMempoolInfo info_for_relay(const GenTxid& gtxid, uint64_t last_sequence) const;
    67 
    68-    std::vector<const CTxMemPoolEntry*> entryAll() const EXCLUSIVE_LOCKS_REQUIRED(cs);
    69+    std::vector<std::reference_wrapper<const CTxMemPoolEntry>> entryAll() const EXCLUSIVE_LOCKS_REQUIRED(cs);
    70     std::vector<TxMempoolInfo> infoAll() const;
    71 
    72     size_t DynamicMemoryUsage() const;
    

    TheCharlatan commented at 10:38 am on November 10, 2023:
    Thanks, will apply.
  105. in src/test/miniminer_tests.cpp:182 in c14e589817 outdated
    178@@ -179,9 +179,9 @@ BOOST_FIXTURE_TEST_CASE(miniminer_1p1c, TestChain100Setup)
    179     };
    180     std::map<uint256, TxDimensions> tx_dims;
    181     for (const auto& tx : all_transactions) {
    182-        const auto it = pool.GetIter(tx->GetHash()).value();
    183-        tx_dims.emplace(tx->GetHash(), TxDimensions{it->GetTxSize(), it->GetModifiedFee(),
    184-                                              CFeeRate(it->GetModifiedFee(), it->GetTxSize())});
    185+        const auto entry{Assert(pool.GetEntry(tx->GetHash()))};
    


    maflcko commented at 10:50 am on November 10, 2023:
    nit in c14e589817035e76e2707fbcd5cb99689d42de77: If you call Assert before an assignment, it could make sense to turn it into a reference by calling *Assert(). This makes later code smaller and easier to read.
  106. in src/validation.cpp:1486 in c14e589817 outdated
    1484@@ -1485,9 +1485,8 @@ PackageMempoolAcceptResult MemPoolAccept::AcceptPackage(const Package& package,
    1485             // transactions that are already in the mempool, and only call AcceptMultipleTransactions() with
    1486             // the new transactions. This ensures we don't double-count transaction counts and sizes when
    1487             // checking ancestor/descendant limits, or double-count transaction fees for fee-related policy.
    1488-            auto iter = m_pool.GetIter(txid);
    1489-            assert(iter != std::nullopt);
    1490-            results_final.emplace(wtxid, MempoolAcceptResult::MempoolTx(iter.value()->GetTxSize(), iter.value()->GetFee()));
    1491+            const auto entry{Assert(m_pool.GetEntry(txid))};
    


    maflcko commented at 10:51 am on November 10, 2023:
    same
  107. TheCharlatan force-pushed on Nov 10, 2023
  108. TheCharlatan commented at 10:52 am on November 10, 2023: contributor

    Thank you for the patch @maflcko,

    Updated f2d8e17851da215d0b85ffad9d80e50eb7b2b363 -> 91df77fde5adc7e2e785aa0d60b82a8515c9b538 (simplifyMemPoolInteractions_13 -> simplifyMemPoolInteractions_14, compare)

    • Applied @maflcko’s patch, making entryAll() return references to CTxMemPoolEntry instead of pointers.
  109. in src/policy/rbf.cpp:39 in 5f72a579de outdated
    35@@ -35,8 +36,8 @@ RBFTransactionState IsRBFOptIn(const CTransaction& tx, const CTxMemPool& pool)
    36 
    37     // If all the inputs have nSequence >= maxint-1, it still might be
    38     // signaled for RBF if any unconfirmed parents have signaled.
    39-    const CTxMemPoolEntry entry{*pool.mapTx.find(tx.GetHash())};
    40-    auto ancestors{pool.AssumeCalculateMemPoolAncestors(__func__, entry, CTxMemPool::Limits::NoLimits(),
    41+    const auto entry{Assert(pool.GetEntry(tx.GetHash()))};
    


    maflcko commented at 10:56 am on November 10, 2023:

    5f72a579de74a6d911ca8529fe8754995025d61a: Same?

    Also, could squash into the previous GetEntry commit?


    TheCharlatan commented at 1:14 pm on November 10, 2023:
    Addressed this, but did not squash, since the change is a bit different to the others.
  110. in src/test/mempool_tests.cpp:298 in 6f8af55a42 outdated
    296     CheckSort<descendant_score>(pool, snapshotOrder);
    297 
    298-    pool.removeRecursive(pool.mapTx.find(tx9.GetHash())->GetTx(), REMOVAL_REASON_DUMMY);
    299-    pool.removeRecursive(pool.mapTx.find(tx8.GetHash())->GetTx(), REMOVAL_REASON_DUMMY);
    300+    pool.removeRecursive(pool.GetIter(tx9.GetHash()).value()->GetTx(), REMOVAL_REASON_DUMMY);
    301+    pool.removeRecursive(pool.GetIter(tx8.GetHash()).value()->GetTx(), REMOVAL_REASON_DUMMY);
    


    maflcko commented at 11:23 am on November 10, 2023:
    nit in 1598b906a57bcd5fc50c1945d8420f8ace1d9c34: Could use get or GetEntry, when only access to the tx pointer is needed, and not full access to the internal iterator?
  111. in src/txmempool.cpp:522 in 6635cffb1e outdated
    517@@ -518,8 +518,10 @@ void CTxMemPool::removeUnchecked(txiter it, MemPoolRemovalReason reason)
    518     RemoveUnbroadcastTx(hash, true /* add logging because unchecked */ );
    519 
    520     if (vTxHashes.size() > 1) {
    521+        // Update vTxHashesIdx of the to-be-moved entry.
    522+        GetIter(vTxHashes.back()->GetHash()).value()->vTxHashesIdx = it->vTxHashesIdx;
    


    maflcko commented at 11:37 am on November 10, 2023:

    6635cffb1e17ad5fdfdabe68cbdcaf1c68d8070d:

    Not sure about turning (theoretical) UB into an exception throw. Might be better to turn it into an assert fail by using Assert(GetEntry())->?

  112. in src/test/validation_block_tests.cpp:286 in 91df77fde5 outdated
    283@@ -284,7 +284,7 @@ BOOST_AUTO_TEST_CASE(mempool_locks_reorg)
    284         // Check that all txs are in the pool
    285         {
    286             LOCK(m_node.mempool->cs);
    


    maflcko commented at 11:42 am on November 10, 2023:
    91df77fde5adc7e2e785aa0d60b82a8515c9b538: Is the lock needed?

    TheCharlatan commented at 12:30 pm on November 10, 2023:
    No, but does the other one in the while loop make sense?
  113. in src/txmempool.cpp:848 in 91df77fde5 outdated
    843+    AssertLockHeld(cs);
    844+
    845+    std::vector<std::reference_wrapper<const CTxMemPoolEntry>> ret;
    846+    ret.reserve(mapTx.size());
    847+    for (const auto& it : GetSortedDepthAndScore()) {
    848+        ret.push_back(*it);
    


    maflcko commented at 11:44 am on November 10, 2023:

    nit in the first commit:

    0clang-tidy-17 -p=/ci_container_base/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu -quiet -load=/tidy-build/libbitcoin-tidy.so /ci_container_base/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/txmempool.cpp
    1txmempool.cpp:848:13: error: use emplace_back instead of push_back [modernize-use-emplace,-warnings-as-errors]
    2  848 |         ret.push_back(*it);
    3      |             ^~~~~~~~~~
    4      |             emplace_back(
    
  114. maflcko commented at 11:47 am on November 10, 2023: member

    ACK 91df77fde5adc7e2e785aa0d60b82a8515c9b538 🕐

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: ACK 91df77fde5adc7e2e785aa0d60b82a8515c9b538   🕐
    3wkACAJP7/zp07cuu9nzpQskOL3zFkoacxerM58ShUGLJjAQioR6qpm1Tmjz0iuk+Z9brI9ayv/TjkNPKZgMJCw==
    
  115. DrahtBot requested review from ismaelsadeeq on Nov 10, 2023
  116. DrahtBot requested review from glozow on Nov 10, 2023
  117. DrahtBot requested review from stickies-v on Nov 10, 2023
  118. DrahtBot added the label CI failed on Nov 10, 2023
  119. TheCharlatan commented at 12:45 pm on November 10, 2023: contributor
    Will address the outstanding comments shortly.
  120. TheCharlatan force-pushed on Nov 10, 2023
  121. TheCharlatan commented at 1:13 pm on November 10, 2023: contributor

    Thank you for the review @maflcko,

    Updated 91df77fde5adc7e2e785aa0d60b82a8515c9b538 -> 3b73acb3ec678f5099aa0d77178a0b0d50670c2b (simplifyMemPoolInteractions_14 -> simplifyMemPoolInteractions_15, compare)

    • Addressed @maflcko’s comment_1, comment_2, comment_3, using Assert on assignment to const reference.
    • Addressed @maflcko’s comment, using get to access transactions in mempool_tests instead of GetIter.
    • Addressed @maflcko’s comment, applying Assert(GetEntry(...)), instead of GetIter(...).value() to turn the potentially thrown exception into an assertion failure.
    • Addressed @maflcko’s comment, removing the redundant mempool.cs lock.
    • Fixed tidy failure.
  122. in src/txmempool.cpp:841 in 3b73acb3ec outdated
    837@@ -836,6 +838,18 @@ static TxMempoolInfo GetInfo(CTxMemPool::indexed_transaction_set::const_iterator
    838     return TxMempoolInfo{it->GetSharedTx(), it->GetTime(), it->GetFee(), it->GetTxSize(), it->GetModifiedFee() - it->GetFee()};
    839 }
    840 
    841+std::vector<std::reference_wrapper<const CTxMemPoolEntry>> CTxMemPool::entryAll() const
    


    stickies-v commented at 1:51 pm on November 10, 2023:

    nit: can use CTxMemPoolEntryRef

     0diff --git a/src/kernel/mempool_entry.h b/src/kernel/mempool_entry.h
     1index 85c2195b13..7c905ca4f4 100644
     2--- a/src/kernel/mempool_entry.h
     3+++ b/src/kernel/mempool_entry.h
     4@@ -176,4 +176,6 @@ public:
     5     mutable Epoch::Marker m_epoch_marker; //!< epoch when last touched, useful for graph algorithms
     6 };
     7 
     8+using CTxMemPoolEntryRef = CTxMemPoolEntry::CTxMemPoolEntryRef;
     9+
    10 #endif // BITCOIN_KERNEL_MEMPOOL_ENTRY_H
    11diff --git a/src/txmempool.cpp b/src/txmempool.cpp
    12index 666b43a633..efcb77f47c 100644
    13--- a/src/txmempool.cpp
    14+++ b/src/txmempool.cpp
    15@@ -838,11 +838,11 @@ static TxMempoolInfo GetInfo(CTxMemPool::indexed_transaction_set::const_iterator
    16     return TxMempoolInfo{it->GetSharedTx(), it->GetTime(), it->GetFee(), it->GetTxSize(), it->GetModifiedFee() - it->GetFee()};
    17 }
    18 
    19-std::vector<std::reference_wrapper<const CTxMemPoolEntry>> CTxMemPool::entryAll() const
    20+std::vector<CTxMemPoolEntryRef> CTxMemPool::entryAll() const
    21 {
    22     AssertLockHeld(cs);
    23 
    24-    std::vector<std::reference_wrapper<const CTxMemPoolEntry>> ret;
    25+    std::vector<CTxMemPoolEntryRef> ret;
    26     ret.reserve(mapTx.size());
    27     for (const auto& it : GetSortedDepthAndScore()) {
    28         ret.emplace_back(*it);
    29diff --git a/src/txmempool.h b/src/txmempool.h
    30index f95f5f1653..3b0b8cf519 100644
    31--- a/src/txmempool.h
    32+++ b/src/txmempool.h
    33@@ -697,7 +697,7 @@ public:
    34     /** Returns info for a transaction if its entry_sequence < last_sequence */
    35     TxMempoolInfo info_for_relay(const GenTxid& gtxid, uint64_t last_sequence) const;
    36 
    37-    std::vector<std::reference_wrapper<const CTxMemPoolEntry>> entryAll() const EXCLUSIVE_LOCKS_REQUIRED(cs);
    38+    std::vector<CTxMemPoolEntryRef> entryAll() const EXCLUSIVE_LOCKS_REQUIRED(cs);
    39     std::vector<TxMempoolInfo> infoAll() const;
    40 
    41     size_t DynamicMemoryUsage() const;
    

    TheCharlatan commented at 4:01 pm on November 10, 2023:
    Will push this, seems easy enough to re-ACK.
  123. DrahtBot removed the label CI failed on Nov 10, 2023
  124. maflcko commented at 2:30 pm on November 10, 2023: member

    re-ACK 3b73acb3ec678f5099aa0d77178a0b0d50670c2b 🏎

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: re-ACK 3b73acb3ec678f5099aa0d77178a0b0d50670c2b 🏎
    3fLkSmbWBYVbxmpiJUroAwcUY3HCXE0ZLjid8zq00VVdauTIWHAS6uxJFY23WBkqE+FE+CV05dbN3zzW0PJ3oDA==
    
  125. stickies-v approved
  126. stickies-v commented at 3:36 pm on November 10, 2023: contributor
    ACK 3b73acb3ec678f5099aa0d77178a0b0d50670c2b
  127. [refactor] Add helper for iterating through mempool entries
    Instead of reaching into the mapTx data structure, use a helper method
    that provides the required vector of CTxMemPoolEntry pointers.
    453b4813eb
  128. [refactor] Add helper for retrieving mempool entry
    In places where the iterator is only needed for accessing the actual
    entry, it should not be required to first retrieve the iterator.
    1c6a73abbd
  129. [refactor] remove access to mapTx from policy/rbf.cpp 14804699e5
  130. [refactor] use exists() instead of mapTx.find() 9cd8cafb77
  131. [refactor] get wtxid from entry instead of vTxHashes fad61aa561
  132. [refactor] remove access to mapTx from rpc/mempool.cpp 8892d6b744
  133. [refactor] remove access to mapTx in blockencodings_tests.cpp f80909e7a3
  134. [refactor] remove access to mapTx.find in mempool_tests.cpp dbc5bdbf59
  135. [refactor] use CheckPackageLimits for checkChainLimits
    The behavior is the same as CalculateMemPoolAncestors. The only
    difference is the string returned, and the string is discarded anyway
    since checkChainLimits only cares about pass/fail.
    1bf4855016
  136. [txmempool] make CTxMemPoolEntry::lockPoints mutable
    Allows calling UpdateLockPoints() with a (const) txiter. Note that this
    was already possible for caller using mapTx.modify(txiter). The point
    here is to not be accessing mapTx when doing so.
    333367a940
  137. [refactor] remove access to mapTx in validation.cpp 938643c3b2
  138. [refactor] rewrite vTxHashes as a vector of CTransactionRef
    vTxHashes exposes a complex mapTx iterator type that its external users
    don't need. Directly populate it with CTransactionRef instead.
    a03aef9cec
  139. scripted-diff: rename vTxHashes to txns_randomized
    -BEGIN VERIFY SCRIPT-
    git grep -l "vTxHashesIdx" src | xargs sed -i "s/vTxHashesIdx/idx_randomized/g"
    git grep -l "vTxHashes" src | xargs sed -i "s/vTxHashes/txns_randomized/g"
    -END VERIFY SCRIPT-
    55b0939cab
  140. [refactor] rewrite BlockAssembler inBlock and failedTx as sets of txids d0cd2e804e
  141. [refactor] remove access to mapTx in validation_block_tests
    Use the helper function instead of reaching into the mapTx member
    object.
    4dd94ca18f
  142. TheCharlatan force-pushed on Nov 10, 2023
  143. TheCharlatan commented at 4:06 pm on November 10, 2023: contributor

    Updated 3b73acb3ec678f5099aa0d77178a0b0d50670c2b -> 4dd94ca18f6fc843137ffca3e6d3e97e4f19377b (simplifyMemPoolInteractions_15 -> simplifyMemPoolInteractions_16, compare)

  144. in src/kernel/mempool_entry.h:179 in 4dd94ca18f
    171@@ -172,8 +172,10 @@ class CTxMemPoolEntry
    172     Parents& GetMemPoolParents() const { return m_parents; }
    173     Children& GetMemPoolChildren() const { return m_children; }
    174 
    175-    mutable size_t vTxHashesIdx; //!< Index in mempool's vTxHashes
    176+    mutable size_t idx_randomized; //!< Index in mempool's txns_randomized
    177     mutable Epoch::Marker m_epoch_marker; //!< epoch when last touched, useful for graph algorithms
    178 };
    179 
    180+using CTxMemPoolEntryRef = CTxMemPoolEntry::CTxMemPoolEntryRef;
    


    maflcko commented at 4:12 pm on November 10, 2023:
    If you put this in the global namespace, might as well remove the duplicate from the inner scope?

    TheCharlatan commented at 4:20 pm on November 10, 2023:
    Can it be replaced without declaration order issues?

    stickies-v commented at 4:45 pm on November 10, 2023:
    I tried that at first but couldn’t figure a more concise way given the Parents and Children typedefs.

    maflcko commented at 9:24 am on November 13, 2023:

    Not sure why that’d be difficult. In C++, it is possible to forward declare a struct:

     0diff --git a/src/kernel/mempool_entry.h b/src/kernel/mempool_entry.h
     1index 7c905ca4f4..199fe948a3 100644
     2--- a/src/kernel/mempool_entry.h
     3+++ b/src/kernel/mempool_entry.h
     4@@ -22,6 +22,9 @@
     5 #include <stdint.h>
     6 
     7 class CBlockIndex;
     8+class CTxMemPoolEntry;
     9+
    10+using CTxMemPoolEntryRef = std::reference_wrapper<const CTxMemPoolEntry>;
    11 
    12 struct LockPoints {
    13     // Will be set to the blockchain height and median time past
    14@@ -61,11 +64,9 @@ struct CompareIteratorByHash {
    15  * all ancestors of the newly added transaction.
    16  *
    17  */
    18-
    19 class CTxMemPoolEntry
    20 {
    21 public:
    22-    typedef std::reference_wrapper<const CTxMemPoolEntry> CTxMemPoolEntryRef;
    23     // two aliases, should the types ever diverge
    24     typedef std::set<CTxMemPoolEntryRef, CompareIteratorByHash> Parents;
    25     typedef std::set<CTxMemPoolEntryRef, CompareIteratorByHash> Children;
    26@@ -176,6 +177,4 @@ public:
    27     mutable Epoch::Marker m_epoch_marker; //!< epoch when last touched, useful for graph algorithms
    28 };
    29 
    30-using CTxMemPoolEntryRef = CTxMemPoolEntry::CTxMemPoolEntryRef;
    31-
    32 #endif // BITCOIN_KERNEL_MEMPOOL_ENTRY_H
    

    TheCharlatan commented at 10:13 am on November 13, 2023:
    Will push this if there are further things that come up.
  145. stickies-v approved
  146. stickies-v commented at 4:48 pm on November 10, 2023: contributor
    re-ACK 4dd94ca18f6fc843137ffca3e6d3e97e4f19377b
  147. DrahtBot requested review from maflcko on Nov 10, 2023
  148. glozow commented at 5:00 pm on November 10, 2023: member
    reACK 4dd94ca
  149. DrahtBot removed review request from glozow on Nov 10, 2023
  150. maflcko commented at 4:22 pm on November 11, 2023: member

    re-ACK 4dd94ca18f6fc843137ffca3e6d3e97e4f19377b 👝

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: re-ACK 4dd94ca18f6fc843137ffca3e6d3e97e4f19377b 👝
    3bfX5G2mlfoBXb55qisEcFZjk5n7/wUwHbMjlDI8yW0PZkGts+ABxmKmEYnMVoT8uNoolCpfL0MahaEHSXhgUCQ==
    
  151. DrahtBot removed review request from maflcko on Nov 11, 2023
  152. fanquake merged this on Nov 13, 2023
  153. fanquake closed this on Nov 13, 2023

  154. in src/txmempool.cpp:845 in 453b4813eb outdated
    840+{
    841+    AssertLockHeld(cs);
    842+
    843+    std::vector<CTxMemPoolEntryRef> ret;
    844+    ret.reserve(mapTx.size());
    845+    for (const auto& it : GetSortedDepthAndScore()) {
    


    sdaftuar commented at 3:19 am on December 7, 2023:

    Is there a reason that we are invoking GetSortedDepthAndScore() here, ie do any of the callers rely on a particular sort order?

    (I’m planning to eliminate this function in #28676, so just want to understand if this behavior needs to be preserved for some reason.)


    TheCharlatan commented at 9:26 am on December 7, 2023:
    This function is supposed to mimic infoAll, which was originally used in this patchset to retrieve mempool information instead of directly accessing mapTx. Looking at its usage you are right, the way I read this too there were no order promises made prior to this patch. I opened #29019 to instead just iterate through mapTx.
  155. glozow referenced this in commit dd391944dc on Dec 18, 2023
  156. glozow commented at 4:42 pm on January 5, 2024: member
    Should we backport 453b4813ebc74859864803e9972b58e4be76a4d6?

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-06-29 07:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me