Next step toward #19556
Instead of relying on the mempool global, each chainstate is given an optional mempool to keep up to date with the tip (block connections, disconnections, reorgs, …)
Next step toward #19556
Instead of relying on the mempool global, each chainstate is given an optional mempool to keep up to date with the tip (block connections, disconnections, reorgs, …)
Concept ACK, and code change looks good.
Would be nice to have a case with no mempool, even to ease review. Do you have a followup with this?
Hmm, I think it is hard to write a meaningful test case right now. Locally I tested with
0diff --git a/src/init.cpp b/src/init.cpp
1index 50d25e672e..7fb8b3aaa7 100644
2--- a/src/init.cpp
3+++ b/src/init.cpp
4@@ -1563,7 +1563,7 @@ bool AppInitMain(const util::Ref& context, NodeContext& node)
5 const int64_t load_block_index_start_time = GetTimeMillis();
6 try {
7 LOCK(cs_main);
8- chainman.InitializeChainstate(node.mempool);
9+ chainman.InitializeChainstate(nullptr);
10 chainman.m_total_coinstip_cache = nCoinCacheUsage;
11 chainman.m_total_coinsdb_cache = nCoinDBCache;
12
Though, if you have a test commit, I am happy to cherry-pick it here.
blocksonly
or withoutmempool
PR.
git blame
stack for those lines? See also #19556 (review)
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
Reviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.
2511@@ -2508,22 +2512,23 @@ bool CChainState::DisconnectTip(BlockValidationState& state, const CChainParams&
2512 if (!FlushStateToDisk(chainparams, state, FlushStateMode::IF_NEEDED))
2513 return false;
2514
2515- if (disconnectpool) {
2516+ if (disconnectpool && m_mempool) {
2517+ AssertLockHeld(m_mempool->cs);
AssertLockHeld
here is silencing the compile time thread safety check, and will need to be LockAssertion lock(m_mempool->cs)
after #19668 . It’s not redundant with the earlier assertion, because the silencing is scoped to the block, which ends immediately for the if (m_mempool) AssertLockHeld(m_mempool->cs);
line.
That was my initial draft, but it would also mean that all changed lines will need to change again. Is there any reason to increase the git blame stack for those lines? See also #19556 (comment)
I totally agree with this. If the intention is to eventually allow CChainState
to not have a mempool, then it makes sense to have the constructor take a pointer, and then assert that it’s not nullptr. To make mempool optional, you just need to change the internal implementation of CChainState rather than the interface and all call sites.
@jnewbery would you drop if (m_mempool)
checks?
edit: @MarcoFalke sorry for being pedantic, feel free to ignore.
2877@@ -2866,7 +2878,8 @@ bool CChainState::ActivateBestChain(BlockValidationState &state, const CChainPar
2878 LimitValidationInterfaceQueue();
2879
2880 {
2881- LOCK2(cs_main, ::mempool.cs); // Lock transaction pool for at least as long as it takes for connectTrace to be consumed
2882+ LOCK(cs_main);
2883+ LOCK(m_mempool ? &m_mempool->cs : nullptr); // Lock transaction pool for at least as long as it takes for connectTrace to be consumed
For those scratching their heads over whether LOCK(nullptr)
is ok, I think it’s fine. It constructs a UniqueLock
using this constructor:
https://github.com/bitcoin/bitcoin/blob/a78742830aa35bf57bcb0a4730977a1e5a1876bc/src/sync.h#L163
Which calls the default constructor (1 in https://en.cppreference.com/w/cpp/thread/unique_lock/unique_lock) for the base std::unique_lock<Mutex>
class and then exits:
https://github.com/bitcoin/bitcoin/blob/a78742830aa35bf57bcb0a4730977a1e5a1876bc/src/sync.h#L165
In the destructor, the call to owns_lock()
returns false (since the unique_lock
doesn’t have a locked mutex):
and the base class destructor does nothing.
Code review ACK faab637301a336c318ea8ee0187a74a2c7a92ef3
@jnewbery would you drop if (m_mempool) checks?
I think it’s fine like this. My alternative approach would have been to leave those out and keep the logic the same, but add an assert m_mempool != nullptr
to the constructor, but it’s not a hard review, so I think it’s fine to change the logic now.
My alternative approach would have been to leave those out and keep the logic the same, but add an
assert m_mempool != nullptr
to the constructor
Same thing here. But it’s fine as it is.
Concept ACK.
Could conditional locking stuff hinder effective usage of Thread Safety Analysis (especially when CTxMemPool::cs
will transit from RecursiveMutex
to Mutex
)?
https://clang.llvm.org/docs/ThreadSafetyAnalysis.html#no-conditionally-held-locks
cs
to a non-recursive mutex or lock annotations outside of validation should be unaffected.
Without a mempool, the mempool lock won’t exist and thus can not be taken. Though, this shouldn’t affect anything outside of validation. E.g. changing
cs
to a non-recursive mutex or lock annotations outside of validation should be unaffected.
I guess it won’t work. For example, this patch
0--- a/src/validation.h
1+++ b/src/validation.h
2@@ -645,7 +645,7 @@ public:
3 CCoinsViewCache& view, const CChainParams& chainparams, bool fJustCheck = false) EXCLUSIVE_LOCKS_REQUIRED(cs_main);
4
5 // Apply the effects of a block disconnection on the UTXO set.
6- bool DisconnectTip(BlockValidationState& state, const CChainParams& chainparams, DisconnectedBlockTransactions* disconnectpool) EXCLUSIVE_LOCKS_REQUIRED(cs_main);
7+ bool DisconnectTip(BlockValidationState& state, const CChainParams& chainparams, DisconnectedBlockTransactions* disconnectpool) EXCLUSIVE_LOCKS_REQUIRED(cs_main, m_mempool->cs);
8
9 // Manual block validity manipulation:
10 bool PreciousBlock(BlockValidationState& state, const CChainParams& params, CBlockIndex* pindex) LOCKS_EXCLUDED(cs_main);
causes -Wthread-safety-analysis
warnings regardless of if (m_mempool) AssertLockHeld(m_mempool->cs);
and LOCK(m_mempool ? &m_mempool->cs : nullptr);
before the DisconnectTip()
calls.
I think a such disadvantage should be avoided.
0--- a/src/validation.h 1+++ b/src/validation.h 2@@ -645,7 +645,7 @@ public: 3 CCoinsViewCache& view, const CChainParams& chainparams, bool fJustCheck = false) EXCLUSIVE_LOCKS_REQUIRED(cs_main); 4 5 // Apply the effects of a block disconnection on the UTXO set. 6- bool DisconnectTip(BlockValidationState& state, const CChainParams& chainparams, DisconnectedBlockTransactions* disconnectpool) EXCLUSIVE_LOCKS_REQUIRED(cs_main); 7+ bool DisconnectTip(BlockValidationState& state, const CChainParams& chainparams, DisconnectedBlockTransactions* disconnectpool) EXCLUSIVE_LOCKS_REQUIRED(cs_main, m_mempool->cs); 8 9 // Manual block validity manipulation: 10 bool PreciousBlock(BlockValidationState& state, const CChainParams& params, CBlockIndex* pindex) LOCKS_EXCLUDED(cs_main);
causes
-Wthread-safety-analysis
warnings regardless ofif (m_mempool) AssertLockHeld(m_mempool->cs);
andLOCK(m_mempool ? &m_mempool->cs : nullptr);
before theDisconnectTip()
calls.I think a such disadvantage should be avoided.
Is such an annotation in the header file dereferencing a pointer that may be NULL
valid ?
Is such an annotation in the header file dereferencing a pointer that may be
NULL
valid ?
Not sure about annotation validity, but Clang accepts it.
I don’t want to lose ability to use EXCLUSIVE_LOCKS_REQUIRED(m_mempool->cs)
annotations.
369@@ -370,9 +370,10 @@ static bool IsCurrentForFeeEstimation() EXCLUSIVE_LOCKS_REQUIRED(cs_main)
370 * and instead just erase from the mempool as needed.
371 */
372
373-static void UpdateMempoolForReorg(DisconnectedBlockTransactions& disconnectpool, bool fAddToMempool) EXCLUSIVE_LOCKS_REQUIRED(cs_main, ::mempool.cs)
374+static void UpdateMempoolForReorg(CTxMemPool& mempool, DisconnectedBlockTransactions& disconnectpool, bool fAddToMempool) EXCLUSIVE_LOCKS_REQUIRED(cs_main)
::
but still requiring mempool.cs
to already be locked. Keeping the annotation here means adding LockAnnotation lock(mempool->cs);
in a bunch of places that call UpdateMempoolForReorg
.
If we want to support not having a mempool, it might be better to virtualize CTxMemPool, so that validation expects a TxMemPoolInterface
that has a cs
lock to let thread safety annotations all make sense, and a bunch of virtual functions that are either implemented by CTxMemPool as is, or by dummy functions in a NoMemPool
class.
If this PR just had m_mempool
as a reference rather than a pointer, that would be a bunch of changes when supporting an optional mempool, though I think it would mostly be limited to the class definitions and the function arguments, which is maybe slightly less than every call site.
The conditional locking in the current patch seems pretty fragile to me, but not wrong at least.
If we want to support not having a mempool, it might be better to virtualize CTxMemPool, so that validation expects a
TxMemPoolInterface
that has acs
lock to let thread safety annotations all make sense, and a bunch of virtual functions that are either implemented by CTxMemPool as is, or by dummy functions in aNoMemPool
class.
Sounds like a nice approach.
51+ void check(const CCoinsViewCache& coins) override { invalidCall(); }
52+ void removeForBlock(const std::vector<CTransactionRef>& vtx, unsigned int block_height) override { invalidCall(); }
53+
54+private:
55+ void invalidCall() { Assert(false); }
56+};
50+ void transactionUpdated() override { invalidCall(); }
51+ void check(const CCoinsViewCache& coins) override { invalidCall(); }
52+ void removeForBlock(const std::vector<CTransactionRef>& vtx, unsigned int block_height) override { invalidCall(); }
53+
54+private:
55+ void invalidCall() { Assert(false); }
face16170176c83d04c68454c91eaf42c80e6ec7
nit: custom Assert
has no benefits in this case
0 void invalidCall() { assert(false); }
I prefer Assert
over assert
because it comes with the following benefits:
Assert
, which are not part of assert
Trying to test thread safety annotations with the following patch:
0--- a/src/interfaces/txmempool.cpp
1+++ b/src/interfaces/txmempool.cpp
2@@ -34,7 +34,6 @@ public:
3
4 void removeForBlock(const std::vector<CTransactionRef>& vtx, unsigned int block_height) override EXCLUSIVE_LOCKS_REQUIRED(m_mutex)
5 {
6- AssertLockHeld(m_txmempool.cs);
7 m_txmempool.removeForBlock(vtx, block_height);
8 }
and got:
0interfaces/txmempool.cpp:37:21: warning: calling function 'removeForBlock' requires holding mutex 'm_txmempool.cs' exclusively [-Wthread-safety-analysis]
1 m_txmempool.removeForBlock(vtx, block_height);
2 ^
31 warning generated.
Could be related to https://clang.llvm.org/docs/ThreadSafetyAnalysis.html#no-alias-analysis
30+ void check(const CCoinsViewCache& coins) override
31+ {
32+ m_txmempool.check(&coins);
33+ }
34+
35+ void removeForBlock(const std::vector<CTransactionRef>& vtx, unsigned int block_height) override EXCLUSIVE_LOCKS_REQUIRED(m_mutex)
0 void removeForBlock(const std::vector<CTransactionRef>& vtx, unsigned int block_height) override EXCLUSIVE_LOCKS_REQUIRED(m_txmempool.cs)
Annotation can not be added to cpp files (or at least they will be ignored).
I don’t think so. https://clang.llvm.org/docs/ThreadSafetyAnalysis.html:
Attributes must be attached to named declarations, such as classes, methods, and data members.
Just logically, if validation does not include the txmempool.cpp file, how would it know about the annotation?
And practically you observed that it didn’t work: #19629 (comment)
Another test of thread safety annotations with the following patch:
0--- a/src/validation.cpp
1+++ b/src/validation.cpp
2@@ -2595,7 +2595,6 @@ public:
3 bool CChainState::ConnectTip(BlockValidationState& state, const CChainParams& chainparams, CBlockIndex* pindexNew, const std::shared_ptr<const CBlock>& pblock, ConnectTrace& connectTrace, DisconnectedBlockTransactions &disconnectpool)
4 {
5 AssertLockHeld(cs_main);
6- AssertLockHeld(m_txmempool->m_mutex);
7
8 assert(pindexNew->pprev == m_chain.Tip());
9 // Read block from disk.
10--- a/src/validation.h
11+++ b/src/validation.h
12@@ -689,7 +689,7 @@ public:
13
14 private:
15 bool ActivateBestChainStep(BlockValidationState& state, const CChainParams& chainparams, CBlockIndex* pindexMostWork, const std::shared_ptr<const CBlock>& pblock, bool& fInvalidFound, ConnectTrace& connectTrace) EXCLUSIVE_LOCKS_REQUIRED(cs_main, m_txmempool->m_mutex);
16- bool ConnectTip(BlockValidationState& state, const CChainParams& chainparams, CBlockIndex* pindexNew, const std::shared_ptr<const CBlock>& pblock, ConnectTrace& connectTrace, DisconnectedBlockTransactions& disconnectpool) EXCLUSIVE_LOCKS_REQUIRED(cs_main, m_txmempool->m_mutex);
17+ bool ConnectTip(BlockValidationState& state, const CChainParams& chainparams, CBlockIndex* pindexNew, const std::shared_ptr<const CBlock>& pblock, ConnectTrace& connectTrace, DisconnectedBlockTransactions& disconnectpool) EXCLUSIVE_LOCKS_REQUIRED(cs_main);
18
19 void InvalidBlockFound(CBlockIndex* pindex, const BlockValidationState& state) EXCLUSIVE_LOCKS_REQUIRED(cs_main);
20 CBlockIndex* FindMostWorkChain() EXCLUSIVE_LOCKS_REQUIRED(cs_main);
and this line https://github.com/bitcoin/bitcoin/blob/face16170176c83d04c68454c91eaf42c80e6ec7/src/validation.cpp#L2640
does not raise a thread safety warning as one could expect due to this annotation: https://github.com/bitcoin/bitcoin/blob/face16170176c83d04c68454c91eaf42c80e6ec7/src/interfaces/txmempool.cpp#L35
m_txmempool
from the implementation class to the base one, changing its type to a pointer, and providing EXCLUSIVE_LOCKS_REQUIRED(m_txmempool->cs)
annotation.
Tested fa1d3da12e6d0712bf6934d6df9e6d5d0e43801d with #19629 (comment). Looks ok:
0validation.cpp:2639:18: warning: calling function 'removeForBlock' requires holding mutex 'm_txmempool->m_mutex' exclusively [-Wthread-safety-analysis]
1 m_txmempool->removeForBlock(blockConnecting.vtx, pindexNew->nHeight);
2 ^
31 warning generated.
FWIW, I found a patch set from #19668 very useful to test correctness of thread safety annotations.
Currently, fa1d3da12e6d0712bf6934d6df9e6d5d0e43801d, a bunch of -Wthread-safety-analysis
warnings are generated.
0interfaces/txmempool.cpp:38:9: warning: calling function 'AssertLockHeldInternal<AnnotatedMixin<std::recursive_mutex> >' requires holding mutex 'm_txmempool.cs' exclusively [-Wthread-safety-analysis]
1 AssertLockHeld(m_txmempool.cs);
2 ^
3./sync.h:79:28: note: expanded from macro 'AssertLockHeld'
4#define AssertLockHeld(cs) AssertLockHeldInternal(#cs, __FILE__, __LINE__, cs)
5 ^
6interfaces/txmempool.cpp:39:21: warning: calling function 'removeForBlock' requires holding mutex 'm_txmempool.cs' exclusively [-Wthread-safety-analysis]
7 m_txmempool.removeForBlock(vtx, block_height);
8 ^
92 warnings generated.
10validation.cpp:379:5: warning: calling function 'AssertLockHeldInternal<AnnotatedMixin<std::recursive_mutex> >' requires holding mutex 'mempool.cs' exclusively [-Wthread-safety-analysis]
11 AssertLockHeld(mempool.cs);
12 ^
13./sync.h:79:28: note: expanded from macro 'AssertLockHeld'
14#define AssertLockHeld(cs) AssertLockHeldInternal(#cs, __FILE__, __LINE__, cs)
15 ^
16validation.cpp:397:21: warning: calling function 'removeRecursive' requires holding mutex 'mempool.cs' exclusively [-Wthread-safety-analysis]
17 mempool.removeRecursive(**it, MemPoolRemovalReason::REORG);
18 ^
19validation.cpp:409:13: warning: calling function 'UpdateTransactionsFromBlock' requires holding mutex 'mempool.cs' exclusively [-Wthread-safety-analysis]
20 mempool.UpdateTransactionsFromBlock(vHashUpdate);
21 ^
22validation.cpp:412:13: warning: calling function 'removeForReorg' requires holding mutex 'mempool.cs' exclusively [-Wthread-safety-analysis]
23 mempool.removeForReorg(&::ChainstateActive().CoinsTip(), ::ChainActive().Tip()->nHeight + 1, STANDARD_LOCKTIME_VERIFY_FLAGS);
24 ^
25validation.cpp:414:5: warning: calling function 'LimitMempoolSize' requires holding mutex 'mempool.cs' exclusively [-Wthread-safety-analysis]
26 LimitMempoolSize(mempool, gArgs.GetArg("-maxmempool", DEFAULT_MAX_MEMPOOL_SIZE) * 1000000, std::chrono::hours{gArgs.GetArg("-mempoolexpiry", DEFAULT_MEMPOOL_EXPIRY)});
27 ^
28validation.cpp:2521:9: warning: calling function 'AssertLockHeldInternal<AnnotatedMixin<std::recursive_mutex> >' requires holding mutex 'm_txmempool->txmempool().cs' exclusively [-Wthread-safety-analysis]
29 AssertLockHeld(m_txmempool->txmempool()->cs);
30 ^
31./sync.h:79:28: note: expanded from macro 'AssertLockHeld'
32#define AssertLockHeld(cs) AssertLockHeldInternal(#cs, __FILE__, __LINE__, cs)
33 ^
34validation.cpp:2529:39: warning: calling function 'removeRecursive' requires holding mutex 'm_txmempool->txmempool().cs' exclusively [-Wthread-safety-analysis]
35 m_txmempool->txmempool()->removeRecursive(**it, MemPoolRemovalReason::REORG);
36 ^
377 warnings generated.
Yes, if #19668 is merged, the AsserLockHeld in interfaces/txmempool
must be replaced by LockAssertion
changing its type to a pointer
I don’t think this works. Or at least it would lead to the initial solution I had (keep a mempool pointer in validation). I am happy to revert to the initial solution or take over a patch, if you have one that compiles on clang with and without –enable-debug
I don’t think this works. … if you have one that compiles on clang with and without –enable-debug
Mind looking at https://github.com/hebasto/bitcoin/commits/pr19629-0816-FORK ?
The same branch, rebased on top of the master (1bc8e8eae2dc4b47ef67afc6880ea48b8e84a086) + #19668:
LimitMempoolSize()
is a bit messed, but it is resolvable, I think.
UPDATE: the improved patch version is available.
@MarcoFalke Please have a look at the updated branch: https://github.com/hebasto/bitcoin/commits/pr19629-0816-FORKv2:
LimitMempoolSize()
is fixedTxMempool
is a pure interface nowThe same branch, rebased on top of the master (e6e277f9ed4da7aff9b7b39a7838bada0c3e572a) + #19668::