These are the non-feeestimator-specific refactoring commits from #11775, mostly focusing on splitting the validationinterface into two parlell interfaces.
There were no changes except reordering commits and rebasing on latest master.
These are the non-feeestimator-specific refactoring commits from #11775, mostly focusing on splitting the validationinterface into two parlell interfaces.
There were no changes except reordering commits and rebasing on latest master.
106@@ -93,12 +107,22 @@ class CValidationInterface {
107 /**
108 * Notifies listeners of a block being disconnected
109 *
110+ * The ordering of BlockDisconnected and TransactionRemovedFromMempool
111+ * (for transactions removed due to memory constraints or lock time/
112+ * coinbase maturity chenges during the disconnection/reorg) is undefined,
587 txiter it = mapTx.find(tx->GetHash());
588 if (it != mapTx.end()) {
589 setEntries stage;
590 stage.insert(it);
591 RemoveStaged(stage, true, MemPoolRemovalReason::BLOCK);
592+ txn_removed_in_block.push_back(tx); // Use the block's copy
in commit ff936aeea569a2ae56a3d9096a486fbd94b6dde7: I don’t understand what this comment means. Mind to elaborate or remove?
Edit: I presume instead of “copy” you mean “witness”, since the witness might be different and it is no longer a copy?
417@@ -418,8 +418,10 @@ bool CTxMemPool::addUnchecked(const uint256& hash, const CTxMemPoolEntry &entry,
418
419 void CTxMemPool::removeUnchecked(txiter it, MemPoolRemovalReason reason)
420 {
421- if (reason != MemPoolRemovalReason::BLOCK && reason != MemPoolRemovalReason::CONFLICT) {
422- // BLOCK and CONFLICT callbacks are generated in removeForBlock
423+ if (reason != MemPoolRemovalReason::BLOCK && reason != MemPoolRemovalReason::CONFLICT &&
424+ reason != MemPoolRemovalReason::REPLACED) {
425+ // BLOCK and CONFLICT callbacks are generated in removeForBlock REPLACED
164@@ -165,18 +165,22 @@ void CZMQNotificationInterface::TransactionAddedToMempool(const CTransactionRef&
165 }
166 }
167
168+void CZMQNotificationInterface::TransactionAddedToMempool(const CTransactionRef& ptx, const std::vector<CTransactionRef>& txn_replaced) {
169+ NotifyTransaction(ptx);
170+}
txn_replaced
is dropped?
598@@ -599,7 +599,7 @@ void CTxMemPool::removeForBlock(const std::vector<CTransactionRef>& vtx, unsigne
599 removeConflicts(*tx, txn_conflicts);
600 ClearPrioritisation(tx->GetHash());
601 }
602- GetMainSignals().MempoolUpdatedForBlockConnect(std::move(txn_removed_in_block), std::move(txn_conflicts));
603+ GetMainSignals().MempoolUpdatedForBlockConnect(std::move(txn_removed_in_block), std::move(txn_conflicts), nBlockHeight);
1204@@ -1205,8 +1205,11 @@ void CWallet::SyncTransaction(const CTransactionRef& ptx, const CBlockIndex *pin
1205 }
1206 }
1207
1208-void CWallet::TransactionAddedToMempool(const CTransactionRef& ptx) {
1209+void CWallet::TransactionAddedToMempool(const CTransactionRef& ptx, const std::vector<CTransactionRef>& txn_replaced) {
1210 LOCK2(cs_main, cs_wallet);
1211+
1212+ for (const CTransactionRef& txit : txn_replaced) TransactionRemovedFromMempool(txit, MemPoolRemovalReason::REPLACED);
Still a big set of commits.
It seems like it would have been simpler to do:
0+void CTxMemPool::removeRecursive(const CTransaction &origTx, MemPoolRemovalReason reason, std::vector<CTransactionRef>* txn_removed) {
1 // Remove transaction from memory pool
2 LOCK(cs);
3 setEntries setAllRemoves;
4 calculateRemoveRecursive(origTx, setAllRemoves);
5+ if (txn_removed != nullptr) {
6+ for (const txiter& it : setAllRemoves) {
7+ txn_removed->push_back(it->GetSharedTx());
8+ }
9+ }
10 RemoveStaged(setAllRemoves, false, reason);
11 }
and pass in &txn_removed
rather than duplicate the code. That would also avoid the need to split out calculateRemoveRecursive()
from what I can see, though maybe another PR needs it? I’m not sure that UpdateMempoolForReorg()
shouldn’t also be tracking the conflicting txs it removes and issuing a notification anyway?
Otherwise:
all look fine and straightforward (and easily reviewable/mergeable if they were all there was).
I haven’t verified that the comments in
are actually correct. For what it’s worth, a commit that documented what the ordering ideally would be (after all these commits, eg) and noted any deviations from that as TODOs (which then get removed in subsequent commits as they’re fixed) might be/have been better?
That leaves these commits I’m not confident about:
EDIT: In particular, there’s a couple of changes in the “Add txn_replaced” commit that deal with (un)registering MempoolInterface that seem like they should be in an earlier commit, possibly the “parallel validation interface for mempool events” one?
I’m not familiar with what ConnectTrace
does or the impact of changing that area of code to be able to be confident in the BlockConnected… commit. Having one commit that just adds MempoolUpdatedForBlockConnect
followed by one that removes ConnectTrace might be simpler to review? The MempoolUpdateForBlockConnect parts look good at least. Seems like the switch from BasicTestingSetup to TestingSetup for policyestimator_tests.cpp should be in one of the commits in “Move fee estimator thread” PR?
66+ *
67+ * Is called after a series of BlockConnected/BlockDisconnected events once
68+ * the chain has made forward progress and is now at the best-known-tip.
69+ *
70+ * If a block is found to be invalid, this event may trigger without
71+ * forward-progress, only to trigger again soon thereafter.
In commit “Clarify validationinterface notification ordering” (e7d7edc898c31ef37d6d64e61c72979c32253a43)
I couldn’t understand what “without forward-progress” means, and would appreciate any clarification. Does it just mean this can be called repeatedly with the same tip? (I asked about this previously in #11856 (review))
Started review (will update this comment with progress).
93+ * This only fires for transactions which leave mempool because of expiry,
94+ * size limiting, or reorg (changes in lock times/coinbase maturity). This
95+ * does not include any transactions which are included in
96+ * MempoolUpdatedForBlockConnect or TransactionAddedToMempool(txn_replaced)
97+ *
98+ * reason == REORG is not ordered with BlockDisconnected/BlockDisconnected!
BlockDisconnected/BlockConnected
?
98+ * reason == REORG is not ordered with BlockDisconnected/BlockDisconnected!
99+ *
100+ * Note that in some rare cases (eg mempool limiting) a
101+ * TransactionRemovedFromMempool event may fire with no corresponding
102+ * TransactionAddedToMempool event.
103+ * (TODO: remove this edge case)
TransactionRemovedFromMempool
no longer has a corresponding TransactionAddedToMempool
event (TransactionAddedToMempool
carries its own replaced txs).
Ah, I understand now. Suggested wording:
Note that a transaction may fire a TransactionRemovedFromMempool notification without having previously fired a TransactionAddedToMempool notification (for example if it passes all AcceptToMemoryPool checks, but isn't added to the mempool due to a LimitMempoolSize() step).
or similar
22@@ -23,15 +23,20 @@ class uint256;
23 class CScheduler;
24 class CTxMemPool;
25 enum class MemPoolRemovalReason;
26+class MempoolInterface;
27
28 // These functions dispatch to one or all registered wallets
29
30 /** Register a wallet to receive updates from core */
592 txiter it = mapTx.find(tx->GetHash());
593 if (it != mapTx.end()) {
594 setEntries stage;
595 stage.insert(it);
596 RemoveStaged(stage, true, MemPoolRemovalReason::BLOCK);
597+ txn_removed_in_block.push_back(tx); // Use the block's copy (as per spec)
as per spec
is not very meaningful for me). Suggested wording: Use the tx as it appears in the block. See comment for MempoolUpdatedForBlockConnect() for details.
or similar
2386- * The block is added to connectTrace if connection succeeds.
2387 */
2388-bool CChainState::ConnectTip(CValidationState& state, const CChainParams& chainparams, CBlockIndex* pindexNew, const std::shared_ptr<const CBlock>& pblock, ConnectTrace& connectTrace, DisconnectedBlockTransactions &disconnectpool)
2389+bool CChainState::ConnectTip(CValidationState& state, const CChainParams& chainparams, CBlockIndex* pindexNew, const std::shared_ptr<const CBlock>& pblock, DisconnectedBlockTransactions &disconnectpool)
2390 {
2391 assert(pindexNew->pprev == chainActive.Tip());
Can you add an AssertLockHeld(cs_main);
to this function? ConnectTip()
is only called by ActivateBestChainStep()
, which holds cs_main
.
This would make it clearer that MempoolUpdatedForBlockConnect is called immediately before its BlockConnected (ie we can’t have two BlockConnected calls racing each other).
2368@@ -2441,7 +2369,7 @@ bool CChainState::ConnectTip(CValidationState& state, const CChainParams& chainp
2369 LogPrint(BCLog::BENCH, " - Connect postprocess: %.2fms [%.2fs (%.2fms/blk)]\n", (nTime6 - nTime5) * MILLI, nTimePostConnect * MICRO, nTimePostConnect * MILLI / nBlocksTotal);
2370 LogPrint(BCLog::BENCH, "- Connect block: %.2fms [%.2fs (%.2fms/blk)]\n", (nTime6 - nTime1) * MILLI, nTimeTotal * MICRO, nTimeTotal * MILLI / nBlocksTotal);
2371
2372- connectTrace.BlockConnected(pindexNew, std::move(pthisBlock));
2373+ GetMainSignals().BlockConnected(std::move(pthisBlock), pindexNew);
I can’t comment on the exact line above (currently L2361), but can you update:
0 // Remove conflicting transactions from the mempool.;
to:
0 // Remove conflicting transactions from the mempool. This will fire the MempoolUpdatedForBlockConnect() notification.
or similar
91+ * Notifies listeners of a transaction leaving mempool.
92+ *
93+ * This only fires for transactions which leave mempool because of expiry,
94+ * size limiting, or reorg (changes in lock times/coinbase maturity). This
95+ * does not include any transactions which are included in
96+ * MempoolUpdatedForBlockConnect or TransactionAddedToMempool(txn_replaced)
Because many listeners (eg wallet) will want both types of events
to be well-ordered, they are parallel and connected on the backend.
However, they are exposed separately to clients to separate the
logic (and because, hopefully, eventually, they can be exposed to
external clients of Bitcoin Core via libconsensus or similar).
This removes the whole ConnectTrace object, which may make it
slightly harder to remove the unbounded-memory-during-reorg bug
by throwing blocks out of memory and re-loading them from disk
later. Comments are added to validationinterface to note where
this should likely happen instead of ConnectTrace.
37-void UnregisterAllValidationInterfaces();
38+/** Register a listener to receive updates from mempool */
39+void RegisterMempoolInterface(MempoolInterface* listener);
40+/** Unregister a listener from mempool */
41+void UnregisterMempoolInterface(MempoolInterface* listener);
42+/** Unregister all listeners from core and mempool */
98+ * reason == REORG is not ordered with BlockConnected/BlockDisconnected!
99+ *
100+ * Note that in some rare cases (eg mempool limiting) a
101+ * TransactionRemovedFromMempool event may fire with no corresponding
102+ * TransactionAddedToMempool event for the same transaction.
103+ * (TODO: remove this edge case)
646@@ -648,9 +647,6 @@ class CTxMemPool
647
648 size_t DynamicMemoryUsage() const;
649
650- boost::signals2::signal<void (CTransactionRef)> NotifyEntryAdded;
I think it could make sense either way, with mempool signals living inside and coming from the mempool itself, or completely pulled out and coming from validation.cpp. However it does seem suboptimal to have the call-sites be split, with some coming from validation.cpp and others coming from the mempool.
Is it a design goal to eventually move all the call-sites to validation.cpp? If so then I think this intermediate state is fine for now, as this is still overall an improvement (and it’s the existing logic that is already split between ATMP living in validation.cpp, but removeForBlock living in txmempool.cpp), but I would suggest that we add some comments indicating that these invocations of GetMainSignals() should be moved out eventually.
-debug=mempool
, sending to wallet and checking unconfirmed balance, spending from wallet and checking balance. Did not attempt to test conflicts/double spends.
587 txiter it = mapTx.find(tx->GetHash());
588 if (it != mapTx.end()) {
589 setEntries stage;
590 stage.insert(it);
591 RemoveStaged(stage, true, MemPoolRemovalReason::BLOCK);
592+ txn_removed_in_block.push_back(tx); // Use the block's copy as witness may be different
I think it would make sense to return both the version removed from the mempool and the version in the block, in the event that a malleated witness is mined.
My thinking is that listeners should either be expected to track everything they care about, in which case we should only return wtxid’s to them when referencing something they’ve already seen, or they should not – in which case we should give them full transactions everywhere. Right now we’re giving full transactions for the added to mempool and removed from mempool cases, so I think we should also give the full transaction in this last edge case, for completeness.
2324@@ -2318,9 +2325,6 @@ bool static ProcessMessage(CNode* pfrom, const std::string& strCommand, CDataStr
2325 }
2326 }
2327
2328- for (const CTransactionRef& removedTx : lRemovedTxn)
80@@ -81,14 +81,14 @@ class MempoolInterface {
81 *
82 * Called on a background thread.
83 */
84- virtual void TransactionAddedToMempool(const CTransactionRef &ptxn) {}
85+ virtual void TransactionAddedToMempool(const CTransactionRef &ptxn, const std::vector<CTransactionRef>& txn_replaced) {}
164@@ -165,18 +165,22 @@ void CZMQNotificationInterface::TransactionAddedToMempool(const CTransactionRef&
165 }
166 }
167
168-void CZMQNotificationInterface::BlockConnected(const std::shared_ptr<const CBlock>& pblock, const CBlockIndex* pindexConnected, const std::vector<CTransactionRef>& vtxConflicted)
169+void CZMQNotificationInterface::TransactionAddedToMempool(const CTransactionRef& ptx, const std::vector<CTransactionRef>& txn_replaced) {
170+ NotifyTransaction(ptx);
171+}