Add CBlockIndex lock annotations, guard nStatus/nFile/nDataPos/nUndoPos by cs_main #22932

pull jonatack wants to merge 9 commits into bitcoin:master from jonatack:require-GetBlockPos-to-hold-cs_main changing 13 files +82 −52
  1. jonatack commented at 4:00 pm on September 9, 2021: member

    Issues:

    • CBlockIndex member functions GetBlockPos(), GetUndoPos(), IsAssumedValid(), RaiseValidity(), and IsValid() and block storage functions WriteUndoDataForBlock() and IsBlockPruned() are missing thread safety lock annotations to help ensure that they are called with mutex cs_main to avoid bugs like #22895. Doing this also enables the next step:

    • CBlockIndex::nStatus may be racy, i.e. potentially accessed by multiple threads, see #17161. A solution is to guard it by cs_main, along with fellow data members nFile, nDataPos and nUndoPos.

    This pull:

    • adds thread safety lock annotations for the functions listed above
    • guards CBlockIndex::nStatus, nFile, nDataPos and nUndoPos by cs_main

    How to review and test:

    Mitigates/potentially closes #17161.

  2. jonatack force-pushed on Sep 9, 2021
  3. DrahtBot added the label Block storage on Sep 9, 2021
  4. DrahtBot added the label Consensus on Sep 9, 2021
  5. DrahtBot added the label UTXO Db and Indexes on Sep 9, 2021
  6. DrahtBot added the label Validation on Sep 9, 2021
  7. DrahtBot added the label Wallet on Sep 9, 2021
  8. jonatack force-pushed on Sep 9, 2021
  9. fanquake removed the label Wallet on Sep 10, 2021
  10. fanquake removed the label UTXO Db and Indexes on Sep 10, 2021
  11. fanquake removed the label Validation on Sep 10, 2021
  12. fanquake removed the label Consensus on Sep 10, 2021
  13. DrahtBot commented at 4:00 pm on September 12, 2021: member

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #23927 (rpc: Pruning nodes can not fetch blocks before syncing past their height by fjahr)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  14. jonatack force-pushed on Sep 16, 2021
  15. jonatack commented at 8:04 pm on September 16, 2021: member
    Rebased following the merge of #22895 that this was based on.
  16. in src/chain.h:229 in f6c2fdb0a5 outdated
    214@@ -212,7 +215,8 @@ class CBlockIndex
    215     {
    216     }
    217 
    218-    FlatFilePos GetBlockPos() const {
    219+    FlatFilePos GetBlockPos() const EXCLUSIVE_LOCKS_REQUIRED(cs_main)
    


    MarcoFalke commented at 10:17 am on September 17, 2021:

    This is annotating the function, but not the underlying data member that needs the annotation.

    Not sure how involved it will be to annotate everything properly here. Also, given that cs_main is the validation lock, we should (eventually/ideally) have a separate node_storage lock. Though, this again requires a major validation/node_storage rewrite.

    However an argument against adding the lock annotation to the data member could be that chain is supposed to be a data-structure-only primitive, like transaction or block. In that case only annotating the getter and setter functions (and making them standalone) would seem appropriate.


    jonatack commented at 4:38 pm on September 17, 2021:
    Having a quick look at annotating nStatus to assess.

    jonatack commented at 5:17 pm on September 17, 2021:
    Appended a small commit to guard CBlockIndex::nStatus by cs_main as a starting point.
  17. MarcoFalke commented at 10:17 am on September 17, 2021: member
    Concept ACK. Feel free to ignore the meta comment.
  18. in src/test/util/blockfilter.cpp:15 in f6c2fdb0a5 outdated
    11@@ -12,7 +12,7 @@
    12 bool ComputeFilter(BlockFilterType filter_type, const CBlockIndex* block_index, BlockFilter& filter)
    13 {
    14     CBlock block;
    15-    if (!ReadBlockFromDisk(block, block_index->GetBlockPos(), Params().GetConsensus())) {
    16+    if (!ReadBlockFromDisk(block, WITH_LOCK(cs_main, return block_index->GetBlockPos()), Params().GetConsensus())) {
    


    MarcoFalke commented at 10:29 am on September 17, 2021:
    nit: unit tests are single-threaded, so you can just take the lock for the whole function. It is likely that there are other lock violations anyway: See #17161 (comment)

    jonatack commented at 4:07 pm on September 17, 2021:
    done
  19. in src/wallet/test/wallet_tests.cpp:90 in f6c2fdb0a5 outdated
    85@@ -86,7 +86,8 @@ BOOST_FIXTURE_TEST_CASE(scan_for_wallet_transactions, TestChain100Setup)
    86 {
    87     // Cap last block file size, and mine new block in a new block file.
    88     CBlockIndex* oldTip = m_node.chainman->ActiveChain().Tip();
    89-    GetBlockFileInfo(oldTip->GetBlockPos().nFile)->nSize = MAX_BLOCKFILE_SIZE;
    90+    const auto pos{WITH_LOCK(cs_main, return oldTip->GetBlockPos())};
    91+    GetBlockFileInfo(pos.nFile)->nSize = MAX_BLOCKFILE_SIZE;
    


    MarcoFalke commented at 10:30 am on September 17, 2021:
    Same: Can include this in the cs_main scope as well, I think.

    jonatack commented at 4:15 pm on September 17, 2021:
    done, thank you
  20. jonatack force-pushed on Sep 17, 2021
  21. jonatack commented at 4:22 pm on September 17, 2021: member

    Thanks @MarcoFalke, updated with your suggestions per git diff f6c2fdb e950f0e

     0diff --git a/src/test/util/blockfilter.cpp b/src/test/util/blockfilter.cpp
     1index 311913b5db..1a663bd0b3 100644
     2--- a/src/test/util/blockfilter.cpp
     3+++ b/src/test/util/blockfilter.cpp
     4@@ -11,8 +11,10 @@
     5 
     6 bool ComputeFilter(BlockFilterType filter_type, const CBlockIndex* block_index, BlockFilter& filter)
     7 {
     8+    LOCK(cs_main);
     9+
    10     CBlock block;
    11-    if (!ReadBlockFromDisk(block, WITH_LOCK(cs_main, return block_index->GetBlockPos()), Params().GetConsensus())) {
    12+    if (!ReadBlockFromDisk(block, block_index->GetBlockPos(), Params().GetConsensus())) {
    13         return false;
    14     }
    15 
    16diff --git a/src/wallet/test/wallet_tests.cpp b/src/wallet/test/wallet_tests.cpp
    17index 37fd52a5e7..4ac8936ee7 100644
    18--- a/src/wallet/test/wallet_tests.cpp
    19+++ b/src/wallet/test/wallet_tests.cpp
    20@@ -86,8 +86,10 @@ BOOST_FIXTURE_TEST_CASE(scan_for_wallet_transactions, TestChain100Setup)
    21 {
    22     // Cap last block file size, and mine new block in a new block file.
    23     CBlockIndex* oldTip = m_node.chainman->ActiveChain().Tip();
    24-    const auto pos{WITH_LOCK(cs_main, return oldTip->GetBlockPos())};
    25-    GetBlockFileInfo(pos.nFile)->nSize = MAX_BLOCKFILE_SIZE;
    26+    {
    27+        LOCK(cs_main);
    28+        GetBlockFileInfo(oldTip->GetBlockPos().nFile)->nSize = MAX_BLOCKFILE_SIZE;
    29+    }
    30     CreateAndProcessBlock({}, GetScriptForRawPubKey(coinbaseKey.GetPubKey()));
    31     CBlockIndex* newTip = m_node.chainman->ActiveChain().Tip();
    
  22. jonatack force-pushed on Sep 17, 2021
  23. DrahtBot added the label Needs rebase on Sep 23, 2021
  24. jonatack force-pushed on Sep 23, 2021
  25. jonatack commented at 9:29 pm on September 23, 2021: member
    Rebased after the merge of #21526.
  26. DrahtBot removed the label Needs rebase on Sep 23, 2021
  27. jonatack force-pushed on Sep 24, 2021
  28. laanwj added this to the "Blockers" column in a project

  29. jonatack renamed this:
    consensus: require CBlockIndex::GetBlockPos() to hold mutex cs_main
    Guard CBlockIndex::nStatus by cs_main, require GetBlockPos/GetUndoPos to hold cs_main
    on Oct 6, 2021
  30. in src/chain.h:230 in e8cc6987f3 outdated
    222@@ -220,7 +223,8 @@ class CBlockIndex
    223     {
    224     }
    225 
    226-    FlatFilePos GetBlockPos() const {
    227+    FlatFilePos GetBlockPos() const EXCLUSIVE_LOCKS_REQUIRED(cs_main)
    228+    {
    


    promag commented at 9:59 am on October 16, 2021:

    e8cc6987f34d19f88de8c461af1efcdd8c3825ec

    Missing

    0AssertLockHeld(cs_main);
    

    jonatack commented at 5:26 pm on November 21, 2021:
    Thanks @promag, done.
  31. in src/chain.h:241 in 9bec673716 outdated
    232@@ -233,7 +233,8 @@ class CBlockIndex
    233         return ret;
    234     }
    235 
    236-    FlatFilePos GetUndoPos() const {
    237+    FlatFilePos GetUndoPos() const EXCLUSIVE_LOCKS_REQUIRED(cs_main)
    238+    {
    


    promag commented at 9:59 am on October 16, 2021:

    9bec673716e68b1854562e4e48a6cb26d8949a59

    Missing

    0AssertLockHeld(cs_main);
    

    jonatack commented at 5:27 pm on November 21, 2021:
    Done.
  32. in src/node/blockstorage.cpp:340 in 9bec673716 outdated
    336@@ -336,7 +337,7 @@ static bool WriteBlockToDisk(const CBlock& block, FlatFilePos& pos, const CMessa
    337 bool WriteUndoDataForBlock(const CBlockUndo& blockundo, BlockValidationState& state, CBlockIndex* pindex, const CChainParams& chainparams)
    338 {
    339     // Write undo information to disk
    340-    if (pindex->GetUndoPos().IsNull()) {
    341+    if (WITH_LOCK(cs_main, return pindex->GetUndoPos()).IsNull()) {
    


    promag commented at 10:03 am on October 16, 2021:

    9bec673716e68b1854562e4e48a6cb26d8949a59

    Why is this needed? WriteUndoDataForBlock is only called from CChainState::ConnectBlock which already locks cs_main.


    jonatack commented at 5:39 pm on November 21, 2021:
    @promag You’re right; Clang 13 seems unable to detect this on its own, but the unneeded lock can be avoided by requiring WriteUndoDataForBlock() to hold cs_main instead. Done in 686ae2d97.
  33. in src/node/blockstorage.cpp:360 in 9cacc50225 outdated
    355@@ -355,9 +356,12 @@ bool WriteUndoDataForBlock(const CBlockUndo& blockundo, BlockValidationState& st
    356         }
    357 
    358         // update nUndoPos in block index
    359-        pindex->nUndoPos = _pos.nPos;
    360-        pindex->nStatus |= BLOCK_HAVE_UNDO;
    361-        setDirtyBlockIndex.insert(pindex);
    362+        {
    363+            LOCK(cs_main);
    


    promag commented at 10:05 am on October 16, 2021:

    9cacc50225f5d8990aeb4bd3cb6d03a18d776ff0

    Again, cs_main is already held here right?


    jonatack commented at 5:41 pm on November 21, 2021:
    Thanks! Removed in both places in this function as described in #22932 (review).
  34. in src/rpc/rawtransaction.cpp:199 in 9cacc50225 outdated
    195@@ -196,7 +196,8 @@ static RPCHelpMan getrawtransaction()
    196     if (!tx) {
    197         std::string errmsg;
    198         if (blockindex) {
    199-            if (!(blockindex->nStatus & BLOCK_HAVE_DATA)) {
    200+            const auto status{WITH_LOCK(cs_main, return blockindex->nStatus)};
    


    promag commented at 10:08 am on October 16, 2021:

    9cacc50225f5d8990aeb4bd3cb6d03a18d776ff0

    nit,

    0const bool block_have_data{WITH_LOCK(cs_main, return blockindex->nStatus & BLOCK_HAVE_DATA)};
    

    jonatack commented at 5:30 pm on November 21, 2021:
    Done.
  35. promag commented at 10:10 am on October 16, 2021: member
    Concept ACK.
  36. jonatack force-pushed on Nov 21, 2021
  37. jonatack force-pushed on Nov 21, 2021
  38. jonatack commented at 10:29 am on November 22, 2021: member
    Thank you for reviewing, @promag. Updated based on your feedback per git diff 9cacc50 b1c859a.
  39. fjahr commented at 5:24 pm on November 28, 2021: member
    Hm, the description only says “Potentially related to issue #17161.”, otherwise I don’t really see a description of what problem this PR is actually fixing/improving?
  40. DrahtBot added the label Needs rebase on Dec 15, 2021
  41. laanwj commented at 8:08 pm on December 15, 2021: member

    otherwise I don’t really see a description of what problem this PR is actually fixing/improving?

    Is the nStatus field potentially accessed by multiple threads at once? #17161 seems to indicate it is. If so, it needs to be either an atomic value or protected by a mutex.

  42. jonatack commented at 11:31 am on December 16, 2021: member
    Rebased and updated per git range-diff df6e961 b1c859a d588248. @fjahr I improved the OP. @laanwj: thanks!
  43. jonatack force-pushed on Dec 16, 2021
  44. DrahtBot removed the label Needs rebase on Dec 16, 2021
  45. fanquake commented at 3:37 am on December 17, 2021: member
    While you’re making changes, can you also update the commits to drop the consensus: prefix, as I don’t think that’s correct for these changes.
  46. hebasto commented at 2:04 pm on December 18, 2021: member
    Concept ACK.
  47. in src/node/blockstorage.cpp:451 in d588248ecf outdated
    452-    {
    453-        LOCK(cs_main);
    454-        block_pos = pindex->GetBlockPos();
    455-    }
    456-
    457+    const FlatFilePos block_pos{WITH_LOCK(cs_main, return pindex->GetBlockPos())};
    


    hebasto commented at 2:31 pm on December 18, 2021:

    How about getting rid of this function entirely:

     0diff --git a/src/net_processing.cpp b/src/net_processing.cpp
     1index 2a30afdb5..030a8c736 100644
     2--- a/src/net_processing.cpp
     3+++ b/src/net_processing.cpp
     4@@ -1848,7 +1848,7 @@ void PeerManagerImpl::ProcessGetBlockData(CNode& pfrom, Peer& peer, const CInv&
     5         // Fast-path: in this case it is possible to serve the block directly from disk,
     6         // as the network format matches the format on disk
     7         std::vector<uint8_t> block_data;
     8-        if (!ReadRawBlockFromDisk(block_data, pindex, m_chainparams.MessageStart())) {
     9+        if (!ReadRawBlockFromDisk(block_data, pindex->GetBlockPos(), m_chainparams.MessageStart())) {
    10             assert(!"cannot load block from disk");
    11         }
    12         m_connman.PushMessage(&pfrom, msgMaker.Make(NetMsgType::BLOCK, Span{block_data}));
    13diff --git a/src/node/blockstorage.cpp b/src/node/blockstorage.cpp
    14index 1af24ecf6..bf47ab351 100644
    15--- a/src/node/blockstorage.cpp
    16+++ b/src/node/blockstorage.cpp
    17@@ -446,12 +446,6 @@ bool ReadRawBlockFromDisk(std::vector<uint8_t>& block, const FlatFilePos& pos, c
    18     return true;
    19 }
    20 
    21-bool ReadRawBlockFromDisk(std::vector<uint8_t>& block, const CBlockIndex* pindex, const CMessageHeader::MessageStartChars& message_start)
    22-{
    23-    const FlatFilePos block_pos{WITH_LOCK(cs_main, return pindex->GetBlockPos())};
    24-    return ReadRawBlockFromDisk(block, block_pos, message_start);
    25-}
    26-
    27 /** Store block on disk. If dbp is non-nullptr, the file is known to already reside on disk */
    28 FlatFilePos SaveBlockToDisk(const CBlock& block, int nHeight, CChain& active_chain, const CChainParams& chainparams, const FlatFilePos* dbp)
    29 {
    30diff --git a/src/node/blockstorage.h b/src/node/blockstorage.h
    31index 93e3b1e93..6872848f2 100644
    32--- a/src/node/blockstorage.h
    33+++ b/src/node/blockstorage.h
    34@@ -73,7 +73,6 @@ void UnlinkPrunedFiles(const std::set<int>& setFilesToPrune);
    35 bool ReadBlockFromDisk(CBlock& block, const FlatFilePos& pos, const Consensus::Params& consensusParams);
    36 bool ReadBlockFromDisk(CBlock& block, const CBlockIndex* pindex, const Consensus::Params& consensusParams);
    37 bool ReadRawBlockFromDisk(std::vector<uint8_t>& block, const FlatFilePos& pos, const CMessageHeader::MessageStartChars& message_start);
    38-bool ReadRawBlockFromDisk(std::vector<uint8_t>& block, const CBlockIndex* pindex, const CMessageHeader::MessageStartChars& message_start);
    39 
    40 bool UndoReadFromDisk(CBlockUndo& blockundo, const CBlockIndex* pindex);
    41 bool WriteUndoDataForBlock(const CBlockUndo& blockundo, BlockValidationState& state, CBlockIndex* pindex, const CChainParams& chainparams) EXCLUSIVE_LOCKS_REQUIRED(cs_main);
    

    ?


    jonatack commented at 4:44 pm on December 21, 2021:
    Done
  48. in src/chain.h:41 in d588248ecf outdated
    37@@ -37,6 +38,8 @@ static constexpr int64_t TIMESTAMP_WINDOW = MAX_FUTURE_BLOCK_TIME;
    38  */
    39 static constexpr int64_t MAX_BLOCK_TIME_GAP = 90 * 60;
    40 
    41+extern RecursiveMutex cs_main;
    


    hebasto commented at 2:33 pm on December 18, 2021:

    style nit, here and after: For new code maybe be explicit about the global namespace:

    0extern RecursiveMutex ::cs_main;
    

    ?


    jonatack commented at 4:44 pm on December 21, 2021:
    Done.
  49. in src/node/blockstorage.h:79 in d588248ecf outdated
    75@@ -73,7 +76,7 @@ bool ReadRawBlockFromDisk(std::vector<uint8_t>& block, const FlatFilePos& pos, c
    76 bool ReadRawBlockFromDisk(std::vector<uint8_t>& block, const CBlockIndex* pindex, const CMessageHeader::MessageStartChars& message_start);
    77 
    78 bool UndoReadFromDisk(CBlockUndo& blockundo, const CBlockIndex* pindex);
    79-bool WriteUndoDataForBlock(const CBlockUndo& blockundo, BlockValidationState& state, CBlockIndex* pindex, const CChainParams& chainparams);
    80+bool WriteUndoDataForBlock(const CBlockUndo& blockundo, BlockValidationState& state, CBlockIndex* pindex, const CChainParams& chainparams) EXCLUSIVE_LOCKS_REQUIRED(cs_main);
    


    hebasto commented at 2:35 pm on December 18, 2021:

    style nit: Is it more readable:

    0bool WriteUndoDataForBlock(const CBlockUndo& blockundo, BlockValidationState& state, CBlockIndex* pindex, const CChainParams& chainparams)
    1    EXCLUSIVE_LOCKS_REQUIRED(cs_main);
    

    ?


    jonatack commented at 4:45 pm on December 21, 2021:
    Not to me ;) but ok, done
  50. jonatack force-pushed on Dec 21, 2021
  51. jonatack commented at 5:44 pm on December 21, 2021: member
    Updated with feedback by @fanquake and @hebasto (thanks!)
  52. in src/index/txindex.cpp:63 in 29cf830d15 outdated
    56@@ -57,7 +57,10 @@ bool TxIndex::WriteBlock(const CBlock& block, const CBlockIndex* pindex)
    57     // Exclude genesis block transaction because outputs are not spendable.
    58     if (pindex->nHeight == 0) return true;
    59 
    60-    CDiskTxPos pos(pindex->GetBlockPos(), GetSizeOfCompactSize(block.vtx.size()));
    61+    CDiskTxPos pos{
    62+        WITH_LOCK(::cs_main, return pindex->GetBlockPos()),
    63+        GetSizeOfCompactSize(block.vtx.size())
    64+    };
    


    hebasto commented at 7:24 pm on December 29, 2021:

    0bfa32b131ed2da8799e1d1239bf390dd612b7bb style nit: clang-format-diff.py insists on:

    0        GetSizeOfCompactSize(block.vtx.size())};
    

    jonatack commented at 4:51 pm on January 5, 2022:
    done
  53. in src/wallet/test/wallet_tests.cpp:147 in 29cf830d15 outdated
    139@@ -137,10 +140,10 @@ BOOST_FIXTURE_TEST_CASE(scan_for_wallet_transactions, TestChain100Setup)
    140 
    141     // Prune the older block file.
    142     {
    143-        LOCK(cs_main);
    144+        LOCK(::cs_main);
    145         Assert(m_node.chainman)->m_blockman.PruneOneBlockFile(oldTip->GetBlockPos().nFile);
    146+        UnlinkPrunedFiles({oldTip->GetBlockPos().nFile});
    147     }
    148-    UnlinkPrunedFiles({oldTip->GetBlockPos().nFile});
    


    hebasto commented at 7:47 pm on December 29, 2021:
    0bfa32b131ed2da8799e1d1239bf390dd612b7bb (here and after) UnlinkPrunedFiles includes a logging operation. Could we keep logging out of the ::cs_main lock?

    jonatack commented at 4:51 pm on January 5, 2022:
    done
  54. hebasto commented at 7:48 pm on December 29, 2021: member

    The WriteUndoDataForBlock contains disk operations. It seems suboptimal to hold ::cs_main for the whole function. Maybe drop 44037646c8b5e0ee9dcd31094eb6d0a8aefc75cd, and enhance b41973ddc239e47c33562639b9d51a4daf5fef19 with

     0--- a/src/node/blockstorage.cpp
     1+++ b/src/node/blockstorage.cpp
     2@@ -339,7 +339,7 @@ static bool WriteBlockToDisk(const CBlock& block, FlatFilePos& pos, const CMessa
     3 bool WriteUndoDataForBlock(const CBlockUndo& blockundo, BlockValidationState& state, CBlockIndex* pindex, const CChainParams& chainparams)
     4 {
     5     // Write undo information to disk
     6-    if (pindex->GetUndoPos().IsNull()) {
     7+    if (WITH_LOCK(::cs_main, return pindex->GetUndoPos()).IsNull()) {
     8         FlatFilePos _pos;
     9         if (!FindUndoPos(state, pindex->nFile, _pos, ::GetSerializeSize(blockundo, CLIENT_VERSION) + 40)) {
    10             return error("ConnectBlock(): FindUndoPos failed");
    

    ?

  55. DrahtBot added the label Needs rebase on Jan 4, 2022
  56. jonatack commented at 4:53 pm on January 5, 2022: member

    Thanks for reviewing @hebasto!

    The WriteUndoDataForBlock contains disk operations. It seems suboptimal to hold ::cs_main for the whole function.

    The lock is already held by the function’s caller (and is also needed further down in the function for pindex->nStatus) so it seemed best to verify the lock is already held rather than taking it twice unnecessarily. I updated the commit description of d1200d67830e777fb96737e31ec5d85b3d095348 to make this clearer.

    Rebased and updated per git range-diff e31cdb0 29cf830 3b9ea0b .

  57. jonatack force-pushed on Jan 5, 2022
  58. DrahtBot removed the label Needs rebase on Jan 5, 2022
  59. DrahtBot added the label Needs rebase on Jan 7, 2022
  60. jonatack force-pushed on Jan 7, 2022
  61. jonatack commented at 12:58 pm on January 7, 2022: member
    Rebased.
  62. DrahtBot removed the label Needs rebase on Jan 7, 2022
  63. in src/txdb.cpp:311 in 71bfaa1c36 outdated
    307@@ -308,6 +308,7 @@ bool CBlockTreeDB::LoadBlockIndexGuts(const Consensus::Params& consensusParams,
    308             CDiskBlockIndex diskindex;
    309             if (pcursor->GetValue(diskindex)) {
    310                 // Construct block index object
    311+                LOCK(::cs_main);
    


    hebasto commented at 10:03 am on January 10, 2022:
    Why a scope of this lock is so wide?

    jonatack commented at 1:53 pm on January 11, 2022:

    Scope narrowed per git diff 71bfaa1 e9560c6

     0--- a/src/txdb.cpp
     1+++ b/src/txdb.cpp
     2@@ -308,7 +308,6 @@ bool CBlockTreeDB::LoadBlockIndexGuts(const Consensus::Params& consensusParams,
     3             CDiskBlockIndex diskindex;
     4             if (pcursor->GetValue(diskindex)) {
     5                 // Construct block index object
     6-                LOCK(::cs_main);
     7                 CBlockIndex* pindexNew = insertBlockIndex(diskindex.GetBlockHash());
     8                 pindexNew->pprev          = insertBlockIndex(diskindex.hashPrev);
     9                 pindexNew->nHeight        = diskindex.nHeight;
    10@@ -320,11 +319,15 @@ bool CBlockTreeDB::LoadBlockIndexGuts(const Consensus::Params& consensusParams,
    11                 pindexNew->nTime          = diskindex.nTime;
    12                 pindexNew->nBits          = diskindex.nBits;
    13                 pindexNew->nNonce         = diskindex.nNonce;
    14-                pindexNew->nStatus        = diskindex.nStatus;
    15                 pindexNew->nTx            = diskindex.nTx;
    16+                {
    17+                    LOCK(::cs_main);
    18+                    pindexNew->nStatus = diskindex.nStatus;
    19+                }
    
  64. hebasto commented at 10:04 am on January 10, 2022: member
    Approach ACK 71bfaa1c367ea15d73a027dad1fe4601df0c73d2.
  65. achow101 commented at 9:42 pm on January 10, 2022: member
    ACK 71bfaa1c367ea15d73a027dad1fe4601df0c73d2
  66. jonatack force-pushed on Jan 11, 2022
  67. jonatack commented at 1:55 pm on January 11, 2022: member
    Thank you @hebasto and @achow101 for the reviews; updated with @hebasto’s feedback per git diff 71bfaa1 e9560c6
  68. hebasto approved
  69. hebasto commented at 2:04 pm on January 11, 2022: member
    ACK e9560c6a448b4bcf144b0c6fa81ce59ef29863ee
  70. achow101 commented at 7:42 pm on January 11, 2022: member
    ACK e9560c6a448b4bcf144b0c6fa81ce59ef29863ee
  71. in src/chain.h:316 in e9560c6a44 outdated
    312@@ -308,20 +313,27 @@ class CBlockIndex
    313     bool IsValid(enum BlockStatus nUpTo = BLOCK_VALID_TRANSACTIONS) const
    314     {
    315         assert(!(nUpTo & ~BLOCK_VALID_MASK)); // Only validity flags allowed.
    316-        if (nStatus & BLOCK_FAILED_MASK)
    317+        LOCK(::cs_main);
    


    vasild commented at 3:54 pm on January 17, 2022:

    All places that call IsValid() already own cs_main. Thus I think it is better just to mandate that it is required when calling this function and avoid locking it recursively:

    0     //! Check whether this block index entry is valid up to the passed validity level.
    1     bool IsValid(enum BlockStatus nUpTo = BLOCK_VALID_TRANSACTIONS) const
    2+        EXCLUSIVE_LOCKS_REQUIRED(cs_main)
    3     {
    4+        AssertLockHeld(cs_main);
    5         assert(!(nUpTo & ~BLOCK_VALID_MASK)); // Only validity flags allowed.
    6-        LOCK(::cs_main);
    7         if (nStatus & BLOCK_FAILED_MASK) {
    

    jonatack commented at 1:58 pm on January 19, 2022:
    That is indeed better (thanks!). ACK, tested, done.
  72. in src/chain.h:327 in e9560c6a44 outdated
    324     //! @returns true if the block is assumed-valid; this means it is queued to be
    325     //!   validated by a background chainstate.
    326-    bool IsAssumedValid() const { return nStatus & BLOCK_ASSUMED_VALID; }
    327+    bool IsAssumedValid() const
    328+    {
    329+        LOCK(::cs_main);
    


    vasild commented at 4:03 pm on January 17, 2022:

    Same as IsValid() (except some tests):

     0diff --git i/src/chain.h w/src/chain.h
     1index 2a7e420c31..291e684eef 100644
     2--- i/src/chain.h
     3+++ w/src/chain.h
     4
     5[...]
     6     //! [@returns](/bitcoin-bitcoin/contributor/returns/) true if the block is assumed-valid; this means it is queued to be
     7     //!   validated by a background chainstate.
     8-    bool IsAssumedValid() const
     9+    bool IsAssumedValid() const EXCLUSIVE_LOCKS_REQUIRED(cs_main)
    10     {
    11-        LOCK(::cs_main);
    12+        AssertLockHeld(cs_main);
    13         return nStatus & BLOCK_ASSUMED_VALID;
    14     }
    15[...]
    16
    17diff --git i/src/test/validation_chainstatemanager_tests.cpp w/src/test/validation_chainstatemanager_tests.cpp
    18index 47d9a77126..e3fd1f203a 100644
    19--- i/src/test/validation_chainstatemanager_tests.cpp
    20+++ w/src/test/validation_chainstatemanager_tests.cpp
    21@@ -230,13 +230,16 @@ BOOST_FIXTURE_TEST_CASE(chainstatemanager_activate_snapshot, TestChain100Setup)
    22     BOOST_CHECK(!chainman.ActiveChainstate().m_from_snapshot_blockhash->IsNull());
    23     BOOST_CHECK_EQUAL(
    24         *chainman.ActiveChainstate().m_from_snapshot_blockhash,
    25         *chainman.SnapshotBlockhash());
    26 
    27     // Ensure that the genesis block was not marked assumed-valid.
    28-    BOOST_CHECK(!chainman.ActiveChain().Genesis()->IsAssumedValid());
    29+    {
    30+        LOCK(cs_main);
    31+        BOOST_CHECK(!chainman.ActiveChain().Genesis()->IsAssumedValid());
    32+    }
    33 
    34     const AssumeutxoData& au_data = *ExpectedAssumeutxo(snapshot_height, ::Params());
    35     const CBlockIndex* tip = chainman.ActiveTip();
    36 
    37     BOOST_CHECK_EQUAL(tip->nChainTx, au_data.nChainTx);
    38 
    39@@ -351,16 +354,16 @@ BOOST_FIXTURE_TEST_CASE(chainstatemanager_loadblockindex, TestChain100Setup)
    40     // tip candidates.
    41     reload_all_block_indexes();
    42     BOOST_CHECK_EQUAL(cs1.setBlockIndexCandidates.size(), cs1.m_chain.Height() + 1);
    43 
    44     // Mark some region of the chain assumed-valid.
    45     for (int i = 0; i <= cs1.m_chain.Height(); ++i) {
    46+        LOCK(::cs_main);
    47         auto index = cs1.m_chain[i];
    48 
    49         if (i < last_assumed_valid_idx && i >= assumed_valid_start_idx) {
    50-            LOCK(::cs_main);
    51             index->nStatus = BlockStatus::BLOCK_VALID_TREE | BlockStatus::BLOCK_ASSUMED_VALID;
    52         }
    53 
    54         ++num_indexes;
    55         if (index->IsAssumedValid()) ++num_assumed_valid;
    

    jonatack commented at 1:59 pm on January 19, 2022:
    ACK, tested, done.
  73. in src/chain.h:336 in e9560c6a44 outdated
    333     //! Raise the validity level of this block index entry.
    334     //! Returns true if the validity was changed.
    335     bool RaiseValidity(enum BlockStatus nUpTo)
    336     {
    337         assert(!(nUpTo & ~BLOCK_VALID_MASK)); // Only validity flags allowed.
    338+        LOCK(::cs_main);
    


    vasild commented at 4:06 pm on January 17, 2022:

    Same as IsValid() (except some tests):

     0diff --git i/src/chain.h w/src/chain.h
     1index 2a7e420c31..291e684eef 100644
     2--- i/src/chain.h
     3+++ w/src/chain.h
     4
     5[...]
     6     //! Raise the validity level of this block index entry.
     7     //! Returns true if the validity was changed.
     8-    bool RaiseValidity(enum BlockStatus nUpTo)
     9+    bool RaiseValidity(enum BlockStatus nUpTo) EXCLUSIVE_LOCKS_REQUIRED(cs_main)
    10     {
    11+        AssertLockHeld(cs_main);
    12         assert(!(nUpTo & ~BLOCK_VALID_MASK)); // Only validity flags allowed.
    13-        LOCK(::cs_main);
    14         if (nStatus & BLOCK_FAILED_MASK) return false;
    15[...]
    16
    17diff --git i/src/test/fuzz/chain.cpp w/src/test/fuzz/chain.cpp
    18index 4f97e1ebb1..e4fd2b32f6 100644
    19--- i/src/test/fuzz/chain.cpp
    20+++ w/src/test/fuzz/chain.cpp
    21@@ -55,13 +55,13 @@ FUZZ_TARGET(chain)
    22             BlockStatus::BLOCK_FAILED_MASK,
    23             BlockStatus::BLOCK_OPT_WITNESS,
    24         });
    25         if (block_status & ~BLOCK_VALID_MASK) {
    26             continue;
    27         }
    28-        (void)disk_block_index->RaiseValidity(block_status);
    29+        WITH_LOCK(cs_main, (void)disk_block_index->RaiseValidity(block_status));
    30     }
    31 
    32     CBlockIndex block_index{block_header};
    33     block_index.phashBlock = &zero;
    34     (void)block_index.GetBlockHash();
    35     (void)block_index.ToString();
    36diff --git i/src/test/validation_chainstatemanager_tests.cpp w/src/test/validation_chainstatemanager_tests.cpp
    37index 47d9a77126..e3fd1f203a 100644
    38--- i/src/test/validation_chainstatemanager_tests.cpp
    39+++ w/src/test/validation_chainstatemanager_tests.cpp
    40@@ -230,13 +230,16 @@ BOOST_FIXTURE_TEST_CASE(chainstatemanager_activate_snapshot, TestChain100Setup)
    41     BOOST_CHECK(!chainman.ActiveChainstate().m_from_snapshot_blockhash->IsNull());
    42     BOOST_CHECK_EQUAL(
    43         *chainman.ActiveChainstate().m_from_snapshot_blockhash,
    44         *chainman.SnapshotBlockhash());
    45 
    46     // Ensure that the genesis block was not marked assumed-valid.
    47-    BOOST_CHECK(!chainman.ActiveChain().Genesis()->IsAssumedValid());
    48+    {
    49+        LOCK(cs_main);
    50+        BOOST_CHECK(!chainman.ActiveChain().Genesis()->IsAssumedValid());
    51+    }
    52 
    53     const AssumeutxoData& au_data = *ExpectedAssumeutxo(snapshot_height, ::Params());
    54     const CBlockIndex* tip = chainman.ActiveTip();
    55 
    56     BOOST_CHECK_EQUAL(tip->nChainTx, au_data.nChainTx);
    57 
    58@@ -351,16 +354,16 @@ BOOST_FIXTURE_TEST_CASE(chainstatemanager_loadblockindex, TestChain100Setup)
    59     // tip candidates.
    60     reload_all_block_indexes();
    61     BOOST_CHECK_EQUAL(cs1.setBlockIndexCandidates.size(), cs1.m_chain.Height() + 1);
    62 
    63     // Mark some region of the chain assumed-valid.
    64     for (int i = 0; i <= cs1.m_chain.Height(); ++i) {
    65+        LOCK(::cs_main);
    66         auto index = cs1.m_chain[i];
    67 
    68         if (i < last_assumed_valid_idx && i >= assumed_valid_start_idx) {
    69-            LOCK(::cs_main);
    70             index->nStatus = BlockStatus::BLOCK_VALID_TREE | BlockStatus::BLOCK_ASSUMED_VALID;
    71         }
    72 
    73         ++num_indexes;
    74         if (index->IsAssumedValid()) ++num_assumed_valid;
    

    jonatack commented at 1:59 pm on January 19, 2022:
    Done.
  74. in src/node/blockstorage.cpp:430 in e9560c6a44 outdated
    426@@ -427,6 +427,7 @@ CBlockIndex* BlockManager::GetLastCheckpoint(const CCheckpointData& data)
    427 
    428 bool IsBlockPruned(const CBlockIndex* pblockindex)
    429 {
    430+    LOCK(::cs_main);
    


    vasild commented at 4:30 pm on January 17, 2022:

    Same as IsValid(), except 3 of 4 callers of IsBlockPruned() already own cs_main. The 4th caller, blockToJSON() could be changed to acquire the mutex.

     0diff --git i/src/node/blockstorage.cpp w/src/node/blockstorage.cpp
     1index 5074080ed1..2217a4d6c8 100644
     2--- i/src/node/blockstorage.cpp
     3+++ w/src/node/blockstorage.cpp
     4@@ -424,13 +424,13 @@ CBlockIndex* BlockManager::GetLastCheckpoint(const CCheckpointData& data)
     5     }
     6     return nullptr;
     7 }
     8 
     9 bool IsBlockPruned(const CBlockIndex* pblockindex)
    10 {
    11-    LOCK(::cs_main);
    12+    AssertLockHeld(cs_main);
    13     return (fHavePruned && !(pblockindex->nStatus & BLOCK_HAVE_DATA) && pblockindex->nTx > 0);
    14 }
    15 
    16 // If we're using -prune with -reindex, then delete block files that will be ignored by the
    17 // reindex.  Since reindexing works by starting at block file 0 and looping until a blockfile
    18 // is missing, do the same here to delete any later block files after a gap.  Also delete all
    19diff --git i/src/node/blockstorage.h w/src/node/blockstorage.h
    20index 7efa7e3b72..0cdab404da 100644
    21--- i/src/node/blockstorage.h
    22+++ w/src/node/blockstorage.h
    23@@ -163,13 +163,13 @@ public:
    24     {
    25         Unload();
    26     }
    27 };
    28 
    29 //! Check whether the block associated with this index entry is pruned or not.
    30-bool IsBlockPruned(const CBlockIndex* pblockindex);
    31+bool IsBlockPruned(const CBlockIndex* pblockindex) EXCLUSIVE_LOCKS_REQUIRED(cs_main);
    32 
    33 void CleanupBlockRevFiles();
    34 
    35 /** Open a block file (blk?????.dat) */
    36 FILE* OpenBlockFile(const FlatFilePos& pos, bool fReadOnly = false);
    37 /** Translation to a filesystem path */
    38diff --git i/src/rpc/blockchain.cpp w/src/rpc/blockchain.cpp
    39index 455c32c57a..d7c1572099 100644
    40--- i/src/rpc/blockchain.cpp
    41+++ w/src/rpc/blockchain.cpp
    42@@ -172,13 +172,17 @@ UniValue blockToJSON(const CBlock& block, const CBlockIndex* tip, const CBlockIn
    43             }
    44             break;
    45 
    46         case TxVerbosity::SHOW_DETAILS:
    47         case TxVerbosity::SHOW_DETAILS_AND_PREVOUT:
    48             CBlockUndo blockUndo;
    49-            const bool have_undo = !IsBlockPruned(blockindex) && UndoReadFromDisk(blockUndo, blockindex);
    50+            bool have_undo;
    51+            {
    52+                LOCK(cs_main);
    53+                have_undo = !IsBlockPruned(blockindex) && UndoReadFromDisk(blockUndo, blockindex);
    54+            }
    55 
    56             for (size_t i = 0; i < block.vtx.size(); ++i) {
    57                 const CTransactionRef& tx = block.vtx.at(i);
    58                 // coinbase transaction (i.e. i == 0) doesn't have undo data
    59                 const CTxUndo* txundo = (have_undo && i > 0) ? &blockUndo.vtxundo.at(i - 1) : nullptr;
    60                 UniValue objTx(UniValue::VOBJ);
    61@@ -925,14 +929,15 @@ static RPCHelpMan getblockheader()
    62 
    63     return blockheaderToJSON(tip, pblockindex);
    64 },
    65     };
    66 }
    67 
    68-static CBlock GetBlockChecked(const CBlockIndex* pblockindex)
    69+static CBlock GetBlockChecked(const CBlockIndex* pblockindex) EXCLUSIVE_LOCKS_REQUIRED(cs_main)
    70 {
    71+    AssertLockHeld(cs_main);
    72     CBlock block;
    73     if (IsBlockPruned(pblockindex)) {
    74         throw JSONRPCError(RPC_MISC_ERROR, "Block not available (pruned data)");
    75     }
    76 
    77     if (!ReadBlockFromDisk(block, pblockindex, Params().GetConsensus())) {
    78@@ -942,14 +947,15 @@ static CBlock GetBlockChecked(const CBlockIndex* pblockindex)
    79         throw JSONRPCError(RPC_MISC_ERROR, "Block not found on disk");
    80     }
    81 
    82     return block;
    83 }
    84 
    85-static CBlockUndo GetUndoChecked(const CBlockIndex* pblockindex)
    86+static CBlockUndo GetUndoChecked(const CBlockIndex* pblockindex) EXCLUSIVE_LOCKS_REQUIRED(cs_main)
    87 {
    88+    AssertLockHeld(cs_main);
    89     CBlockUndo blockUndo;
    90     if (IsBlockPruned(pblockindex)) {
    91         throw JSONRPCError(RPC_MISC_ERROR, "Undo data not available (pruned data)");
    92     }
    93 
    94     if (!UndoReadFromDisk(blockUndo, pblockindex)) {
    

    I think it is better to call UndoReadFromDisk() also under the same acquisition of cs_main as IsBlockPruned() (as in the patch above) because that would ensure nStatus remains unchanged until after the disk read has completed (i.e. we don’t end up with a status that indicates no file on disk and still trying to read the file, if nStatus is changed after we read it and before we read the file from disk).


    jonatack commented at 2:03 pm on January 19, 2022:

    I had planned to add lock annotations to IsBlockPruned() in a follow-up, but I may as well do it here with the other changes. Done.

    If this pull is now too large to attract re-ACKs, in that case it could be reverted to its original version with the first 3 commits only, and the nStatus preparation and change could be proposed separately. We’ll see.

  75. in src/chain.h:234 in e9560c6a44 outdated
    230+    FlatFilePos GetBlockPos() const EXCLUSIVE_LOCKS_REQUIRED(::cs_main)
    231     {
    232+        AssertLockHeld(::cs_main);
    233         FlatFilePos ret;
    234         if (nStatus & BLOCK_HAVE_DATA) {
    235             ret.nFile = nFile;
    


    vasild commented at 4:48 pm on January 17, 2022:

    This access pattern requires that nFile stays in sync with nStatus - we don’t want nStatus to be changed concurrently by another thread after we have read it and entered this if but before we have read nFile. That is to say - nFile, nDataPos and nUndoPos should be protected by the same mechanism as nStatus. This is easy:

     0diff --git i/src/chain.h w/src/chain.h
     1index 2a7e420c31..d8f108b466 100644
     2--- i/src/chain.h
     3+++ w/src/chain.h
     4@@ -161,19 +161,19 @@ public:
     5     CBlockIndex* pskip{nullptr};
     6 
     7     //! height of the entry in the chain. The genesis block has height 0
     8     int nHeight{0};
     9 
    10     //! Which # file this block is stored in (blk?????.dat)
    11-    int nFile{0};
    12+    int nFile GUARDED_BY(::cs_main){0};
    13 
    14     //! Byte offset within blk?????.dat where this block's data is stored
    15-    unsigned int nDataPos{0};
    16+    unsigned int nDataPos GUARDED_BY(::cs_main){0};
    17 
    18     //! Byte offset within rev?????.dat where this block's undo data is stored
    19-    unsigned int nUndoPos{0};
    20+    unsigned int nUndoPos GUARDED_BY(::cs_main){0};
    21 
    22     //! (memory only) Total amount of work (expected number of hashes) in the chain up to and including this block
    23     arith_uint256 nChainWork{};
    24 
    25     //! Number of transactions in this block.
    26     //! Note: in a potential headers-first mode, this number cannot be relied upon
    27diff --git i/src/txdb.cpp w/src/txdb.cpp
    28index 3fd5ca5b19..e2aed1da7c 100644
    29--- i/src/txdb.cpp
    30+++ w/src/txdb.cpp
    31@@ -308,23 +308,23 @@ bool CBlockTreeDB::LoadBlockIndexGuts(const Consensus::Params& consensusParams,
    32             CDiskBlockIndex diskindex;
    33             if (pcursor->GetValue(diskindex)) {
    34                 // Construct block index object
    35                 CBlockIndex* pindexNew = insertBlockIndex(diskindex.GetBlockHash());
    36                 pindexNew->pprev          = insertBlockIndex(diskindex.hashPrev);
    37                 pindexNew->nHeight        = diskindex.nHeight;
    38-                pindexNew->nFile          = diskindex.nFile;
    39-                pindexNew->nDataPos       = diskindex.nDataPos;
    40-                pindexNew->nUndoPos       = diskindex.nUndoPos;
    41                 pindexNew->nVersion       = diskindex.nVersion;
    42                 pindexNew->hashMerkleRoot = diskindex.hashMerkleRoot;
    43                 pindexNew->nTime          = diskindex.nTime;
    44                 pindexNew->nBits          = diskindex.nBits;
    45                 pindexNew->nNonce         = diskindex.nNonce;
    46                 pindexNew->nTx            = diskindex.nTx;
    47                 {
    48                     LOCK(::cs_main);
    49+                    pindexNew->nFile = diskindex.nFile;
    50+                    pindexNew->nDataPos = diskindex.nDataPos;
    51+                    pindexNew->nUndoPos = diskindex.nUndoPos;
    52                     pindexNew->nStatus = diskindex.nStatus;
    53                 }
    54 
    55                 if (!CheckProofOfWork(pindexNew->GetBlockHash(), pindexNew->nBits, consensusParams)) {
    56                     return error("%s: CheckProofOfWork failed: %s", __func__, pindexNew->ToString());
    57                 }
    

    jonatack commented at 2:03 pm on January 19, 2022:
    Done in the nStatus commit (thanks!)
  76. vasild commented at 4:55 pm on January 17, 2022: member

    Approach ACK e9560c6a448b4bcf144b0c6fa81ce59ef29863ee

    I think this does not fully resolve #17161 but is an improvement that can go in as is. I think the issue is not fully resolved because we can still have the nStatus variable go out of sync with the disk contents - we can read nStatus & BLOCK_HAVE_DATA to true under cs_main, but immediately after the read, before cs_main is released, prune may delete the files from disk:

    https://github.com/bitcoin/bitcoin/blob/d0bf9bb6a539f151ec92725d20a2b6c22cb095a5/src/validation.cpp#L2296

  77. jonatack force-pushed on Jan 19, 2022
  78. jonatack renamed this:
    Guard CBlockIndex::nStatus by cs_main, require GetBlockPos/GetUndoPos to hold cs_main
    Add CBlockIndex lock annotations, guard nStatus/nFile/nDataPos/nUndoPos by cs_main
    on Jan 19, 2022
  79. jonatack commented at 2:23 pm on January 19, 2022: member
    Addressed review feedback by @vasild and updated the pull title/description. Invalidates previous ACKs by @hebasto, @achow101 and @vasild.
  80. in src/test/util/blockfilter.cpp:16 in cf2145beb2 outdated
    10@@ -11,6 +11,8 @@
    11 
    12 bool ComputeFilter(BlockFilterType filter_type, const CBlockIndex* block_index, BlockFilter& filter)
    13 {
    14+    LOCK(::cs_main);
    


    vasild commented at 10:08 am on January 21, 2022:
    I am just curious - why prefix with ::? There is no conflict with a local symbol under the same name, so the :: shouldn’t be necessary? Right now in the codebase there are 154 occurrences of ::cs_main and 421 occurrences of cs_main (without the prefix).

    hebasto commented at 10:12 am on January 21, 2022:
    I do prefer to specify the global namespace for the sake being explicit.

    jonatack commented at 11:21 am on January 25, 2022:
    Right, my understanding from review feedback is that it is preferred for new or updated code.

    vasild commented at 12:25 pm on January 27, 2022:
    For all global variables or just for cs_main? Global variables are already supposed to start with g_, so prefixing them with :: is kind of redundant if the purpose is to show to the reader that this is a global variable.

    MarcoFalke commented at 12:40 pm on January 28, 2022:
    cs_main doesn’t start with g_
  81. vasild approved
  82. vasild commented at 10:09 am on January 21, 2022: member
    ACK cf2145beb26f0cf3016709b2159bcb4053b58eb9
  83. DrahtBot added the label Needs rebase on Jan 25, 2022
  84. Require CBlockIndex::GetBlockPos() to hold mutex cs_main 6fd4341c10
  85. Require WriteUndoDataForBlock() to hold mutex cs_main
    Mutex cs_main is already held by the caller of WriteUndoDataForBlock().
    This change is needed to require CBlockIndex::GetUndoPos() to hold
    cs_main and CBlockIndex::nStatus to be guarded by cs_main in the
    following commits without adding 2 unnecessary cs_main locks to
    WriteUndoDataForBlock().
    2e557ced28
  86. Require CBlockIndex::GetUndoPos() to hold mutex cs_main 572393448b
  87. Require CBlockIndex::IsAssumedValid() to hold cs_main 8ef457cb83
  88. Require CBlockIndex::RaiseValidity() to hold cs_main e9f3aa5f6a
  89. Require CBlockIndex::IsValid() to hold cs_main ca47b00577
  90. Require IsBlockPruned() to hold mutex cs_main
    Co-authored-by: Vasil Dimov <vd@FreeBSD.org>
    eaeeb88768
  91. Remove/inline ReadRawBlockFromDisk(block_data, pindex, message_start) 5d59ae0ba8
  92. Guard CBlockIndex::nStatus/nFile/nDataPos/nUndoPos by cs_main
    Co-authored-by: Vasil Dimov <vd@FreeBSD.org>
    6ea5682784
  93. jonatack force-pushed on Jan 25, 2022
  94. jonatack commented at 8:06 pm on January 25, 2022: member

    Rebased.

    This has concept ACKs by @MarcoFalke and @promag and ACKs by @hebasto, @achow101 and @vasild.

    Review/re-ACKs welcome.

  95. DrahtBot removed the label Needs rebase on Jan 25, 2022
  96. jamesob commented at 1:50 am on January 26, 2022: member

    Benched 6ea56827842b9b2bd730edc38f3a7b1f46f6247b; no performance impact :+1:.

    bench name command
    ibd.local.range.500000.540000 bitcoind -dbcache=10000 -debug=coindb -debug=bench -listen=0 -connect=0 -addnode=127.0.0.1:8888 -prune=9999999 -printtoconsole=0 -assumevalid=000000000000000000176c192f42ad13ab159fdb20198b87e7ba3c001e47b876

    #22932 vs. $mergebase (absolute)

    bench name x #22932 $mergebase
    ibd.local.range.500000.540000.total_secs 2 3093.6633 (± 17.1804) 3102.2931 (± 0.4422)
    ibd.local.range.500000.540000.peak_rss_KiB 2 6355854.0000 (± 1646.0000) 6355764.0000 (± 1492.0000)
    ibd.local.range.500000.540000.cpu_kernel_secs 2 263.2300 (± 3.3200) 264.0900 (± 0.1700)
    ibd.local.range.500000.540000.cpu_user_secs 2 13020.2400 (± 4.7400) 13015.7250 (± 8.2850)

    #22932 vs. $mergebase (relative)

    bench name x #22932 $mergebase
    ibd.local.range.500000.540000.total_secs 2 1.000 1.003
    ibd.local.range.500000.540000.peak_rss_KiB 2 1.000 1.000
    ibd.local.range.500000.540000.cpu_kernel_secs 2 1.000 1.003
    ibd.local.range.500000.540000.cpu_user_secs 2 1.000 1.000
  97. laanwj commented at 9:53 am on January 27, 2022: member
    Code review ACK 6ea56827842b9b2bd730edc38f3a7b1f46f6247b
  98. laanwj merged this on Jan 27, 2022
  99. laanwj closed this on Jan 27, 2022

  100. jonatack deleted the branch on Jan 27, 2022
  101. vasild commented at 11:40 am on January 27, 2022: member
    ACK 6ea56827842b9b2bd730edc38f3a7b1f46f6247b
  102. laanwj removed this from the "Blockers" column in a project

  103. in src/chain.h:385 in 6ea5682784
    381@@ -382,6 +382,7 @@ class CDiskBlockIndex : public CBlockIndex
    382 
    383     SERIALIZE_METHODS(CDiskBlockIndex, obj)
    384     {
    385+        LOCK(::cs_main);
    


    MarcoFalke commented at 1:14 pm on January 28, 2022:

    can you explain why this is needed?

    CDiskBlockIndex creates a copy of the data and it isn’t used in validation or blockstorage code. It is only used to serialize a struct and then discarded.


    jonatack commented at 0:13 am on January 29, 2022:

    can you explain why this is needed?

    CDiskBlockIndex creates a copy of the data and it isn’t used in validation or blockstorage code. It is only used to serialize a struct and then discarded.

    Good point. Opened #24199 as an approach to address this.

  104. in src/txdb.cpp:321 in 6ea5682784
    320                 pindexNew->nBits          = diskindex.nBits;
    321                 pindexNew->nNonce         = diskindex.nNonce;
    322-                pindexNew->nStatus        = diskindex.nStatus;
    323                 pindexNew->nTx            = diskindex.nTx;
    324+                {
    325+                    LOCK(::cs_main);
    


    MarcoFalke commented at 1:18 pm on January 28, 2022:
    Can you explain why this is needed? The calling function should already hold cs_main?

    jonatack commented at 8:03 pm on January 28, 2022:
    Good catch–we can drop the lock in favor of an annotation; done in #24197.
  105. sidhujag referenced this in commit 0ac526ba19 on Jan 28, 2022
  106. MarcoFalke referenced this in commit 4efdbabd70 on Jan 31, 2022
  107. sidhujag referenced this in commit 776e028f39 on Feb 1, 2022
  108. Fabcien referenced this in commit 1e299c9763 on Jan 24, 2023
  109. Fabcien referenced this in commit 4d9bd4f787 on Jan 24, 2023
  110. Fabcien referenced this in commit 99d5b83546 on Jan 24, 2023
  111. Fabcien referenced this in commit fb7c3b85e0 on Jan 24, 2023
  112. Fabcien referenced this in commit 7b92966d1c on Jan 24, 2023
  113. Fabcien referenced this in commit 20237dec6f on Jan 24, 2023
  114. Fabcien referenced this in commit bd5235a5a8 on Jan 24, 2023
  115. Fabcien referenced this in commit dcac626d18 on Jan 24, 2023
  116. DrahtBot locked this on Jan 29, 2023

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-07-03 10:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me