indexes: Stop using node internal types #25494

pull ryanofsky wants to merge 7 commits into bitcoin:master from ryanofsky:pr/ind changing 21 files +273 −142
  1. ryanofsky commented at 8:14 pm on June 28, 2022: contributor

    Start transitioning index code away from using internal node types like CBlockIndex and CChain so index code is less coupled to node code and index code will later be able to stop locking cs_main and sync without having to deal with validationinterface race conditions, and so new indexes are easier to write and can run as plugins or separate processes.

    This PR contains the first 7 commits from #24230 (comment) which have been split off for easier review. Previous review comments can be found in #24230

  2. DrahtBot added the label Refactoring on Jun 28, 2022
  3. DrahtBot commented at 0:33 am on June 29, 2022: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #25623 ([kernel 3e/n] Decouple CDBWrapper and CBlockTreeDB from ArgsManager by dongcarl)
    • #25297 (WIP, wallet: speedup transactions sync, rescan and load not flushing to db constantly by furszy)
    • #25222 (refactor: Pass reference to LookUpStats by MarcoFalke)
    • #24150 (refactor: move index class members from protected to private by jonatack)
    • #19888 (rpc, test: Improve getblockstats for unspendables by fjahr)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  4. in src/node/chain.cpp:17 in 3c82fa533a outdated
    12+    interfaces::BlockInfo info{index ? *index->phashBlock : uint256::ZERO};
    13+    if (index) {
    14+        info.prev_hash = index->pprev ? index->pprev->phashBlock : nullptr;
    15+        info.height = index->nHeight;
    16+        LOCK(::cs_main);
    17+        info.file_number = index->nFile;
    


    maflcko commented at 6:25 am on June 29, 2022:
    It seems that this will only copy/alias the internal memory to a different type. I wonder if this is actually useful, because it disables lock annotations for the memory where they may actually still be needed? Also, I wonder how useful an interface is that returns a data structure that points into memory that is owned by a different process (I am assuming each blockindex runs in a different multiprocess)

    ryanofsky commented at 4:52 pm on July 5, 2022:

    re: #25494 (review)

    In commit “interfaces, refactor: Add more block information to block connected notifications” (0a93a16614659a7c12e7e95478a9bb564c1a13cb)

    It seems that this will only copy/alias the internal memory to a different type. I wonder if this is actually useful, because it disables lock annotations for the memory where they may actually still be needed?

    I don’t think there’s actually a concern here, and this is what current code already does. cs_main is locked while reading nFile and nDataPos values, but not when ReadBlockFromDisk is called. Values are copied first and then used.

    Also, I wonder how useful an interface is that returns a data structure that points into memory that is owned by a different process (I am assuming each blockindex runs in a different multiprocess)

    The change in this commit is to replace individual block & height parameters with a BlockInfo struct parameter that can hold more information about the block:

    0-        virtual void blockConnected(const CBlock& block, int height) {}
    1-        virtual void blockDisconnected(const CBlock& block, int height) {}
    2+        virtual void blockConnected(const BlockInfo& block) {}
    3+        virtual void blockDisconnected(const BlockInfo& block) {}
    

    The point of using the struct is to make code more readable and avoid the need to update blockConnected/blockDisconnected declarations when more block information is needed.

    Semantically there’s no difference between passing a const uint256& hash parameter vs adding a const uint256& hash; struct member, or passing an int file_number parameter vs. adding an int file_number; struct member. All the copy and reference and lifetime concerns are the same.

    It is true that because referencing memory in other processes is not possible, multiprocess code will create local copies of values when references are passed, and the local copies may have shorter lifetimes than original references. But this is true wherever references and pointers are used in src/interface/ classes, so not a new thing.

  5. maflcko removed the label Refactoring on Jul 1, 2022
  6. DrahtBot added the label UTXO Db and Indexes on Jul 1, 2022
  7. fjahr commented at 11:15 pm on July 2, 2022: contributor

    Linter warns of a circular dependency

    0A new circular dependency in the form of "index/base -> node/context -> net_processing -> index/blockfilterindex -> index/base" appears to have been introduced.
    
  8. Onyeali approved
  9. ryanofsky force-pushed on Jul 5, 2022
  10. ryanofsky commented at 5:30 pm on July 5, 2022: contributor

    re: #25494 (comment)

    Linter warns of a circular dependency

    Thanks! Should be fixed


    Updated 3c82fa533a31939b867c774fe2c05f7a2e1d1a0c -> ddbbdcf4112c3a84e3287e6bf701e0eb592b24c3 (pr/ind.1 -> pr/ind.2, compare) to fix lint-circular-dependencies.py error

  11. mzumsande commented at 7:48 pm on July 7, 2022: contributor

    ACK ddbbdcf4112c3a84e3287e6bf701e0eb592b24c3

    I reviewed these commits multiple times as part of #24230, and think they are good. I also did some manual testing for this PR on regtest and signet and didn’t see any differences in behavior.

    While it’s a bit unfortunate that this introduces a circular dependency, this is transitory and will get removed with the last commits of #24230 after all internal uses of CChainState are removed.

    nit: clang-format suggests a few minor changes (spacing) over these commits

  12. in src/wallet/test/fuzz/notifications.cpp:165 in 0a93a16614 outdated
    160@@ -155,8 +161,14 @@ FUZZ_TARGET_INIT(wallet_notifications, initialize_setup)
    161                 auto& [coins, block]{chain.back()};
    162                 if (block.vtx.empty()) return; // Can only disconnect if the block was submitted first
    163                 // Disconnect block
    164-                a.wallet->blockDisconnected(block, chain.size() - 1);
    165-                b.wallet->blockDisconnected(block, chain.size() - 1);
    166+                const uint256& hash = block.GetHash();
    167+                const uint256& prev_hash = chain.size() >= 2 ? std::get<1>(chain[chain.size() - 2]).GetHash() : uint256();
    


    fjahr commented at 10:24 pm on July 8, 2022:
    nit: I think this line could be made a little easier to parse but since it’s in fuzzing it’s not so important

    maflcko commented at 2:19 pm on July 13, 2022:
    Same commit: There is an early return above if the size is 1 or less, thus the size must be 2 or more, thus the check here is not needed. Also, could just remove all of this and write &block.hashPrevBlock below?

    ryanofsky commented at 7:31 pm on July 14, 2022:

    re: #25494 (review)

    Same commit: There is an early return above if the size is 1 or less, thus the size must be 2 or more, thus the check here is not needed. Also, could just remove all of this and write &block.hashPrevBlock below?

    Good catch, simplified


    ryanofsky commented at 7:31 pm on July 14, 2022:

    re: #25494 (review)

    nit: I think this line could be made a little easier to parse but since it’s in fuzzing it’s not so important

    Simplified

  13. in src/interfaces/chain.h:88 in 0a93a16614 outdated
    74+    const uint256* prev_hash = nullptr;
    75+    int height = -1;
    76+    int file_number = -1;
    77+    unsigned data_pos = 0;
    78+    const CBlock* data = nullptr;
    79+    const CBlockUndo* undo_data = nullptr;
    


    fjahr commented at 7:21 pm on July 9, 2022:
    note for other reviewers: in this PR this is unused but will be used in follow-ups, see #24230
  14. fjahr commented at 12:06 pm on July 10, 2022: contributor

    tACK ddbbdcf4112c3a84e3287e6bf701e0eb592b24c3

    Reviewed code and ran it, syncing txindex from intermittently synced state to fully synced and fully synced coinstatsindex from scratch.

  15. in src/wallet/test/fuzz/notifications.cpp:142 in 0a93a16614 outdated
    137@@ -138,8 +138,14 @@ FUZZ_TARGET_INIT(wallet_notifications, initialize_setup)
    138                     block.vtx.emplace_back(MakeTransactionRef(tx));
    139                 }
    140                 // Mine block
    141-                a.wallet->blockConnected(block, chain.size());
    142-                b.wallet->blockConnected(block, chain.size());
    143+                const uint256& hash = block.GetHash();
    144+                const uint256& prev_hash = std::get<1>(chain.back()).GetHash();
    


    maflcko commented at 2:16 pm on July 13, 2022:

    first commit: block and chain.back() are both mutable references to the same memory, so I think this is wrong

    Could write &block.hashPrevBlock below?


    ryanofsky commented at 7:31 pm on July 14, 2022:

    re: #25494 (review)

    first commit: block and chain.back() are both mutable references to the same memory, so I think this is wrong

    Could write &block.hashPrevBlock below?

    Good catch. block.hashPrevBlock seems simpler anyway

  16. in src/interfaces/chain.h:81 in 0a93a16614 outdated
    76+    int file_number = -1;
    77+    unsigned data_pos = 0;
    78+    const CBlock* data = nullptr;
    79+    const CBlockUndo* undo_data = nullptr;
    80+
    81+    BlockInfo(const uint256& hash) : hash(hash) {}
    


    maflcko commented at 2:20 pm on July 13, 2022:
    first commit: Missing LIFETIMEBOUND attribute for hash?

    ryanofsky commented at 7:28 pm on July 14, 2022:

    re: #25494 (review)

    first commit: Missing LIFETIMEBOUND attribute for hash?

    Added. This is great! I did not know about this

  17. in src/wallet/wallet.cpp:1349 in 0a93a16614 outdated
    1355     // future with a stickier abandoned state or even removing abandontransaction call.
    1356-    m_last_block_processed_height = height - 1;
    1357-    m_last_block_processed = block.hashPrevBlock;
    1358-    for (const CTransactionRef& ptx : block.vtx) {
    1359+    m_last_block_processed_height = block.height - 1;
    1360+    m_last_block_processed = *block.prev_hash;
    


    maflcko commented at 2:23 pm on July 13, 2022:
    First commit: Why are you changing this from block.data->hashPrevBlock? If this is intentional, missing an assert to avoid UB?

    ryanofsky commented at 7:32 pm on July 14, 2022:

    re: #25494 (review)

    First commit: Why are you changing this from block.data->hashPrevBlock? If this is intentional, missing an assert to avoid UB?

    I just didn’t want to depend on block data unnecessarily here, even it it used below. Maybe the code could be split up or reorganized and it would no longer be necessary. Added asserts here and below, though.

  18. in src/index/coinstatsindex.cpp:105 in 2f8a5307b7 outdated
    101@@ -102,7 +102,7 @@ struct DBHashKey {
    102 
    103 std::unique_ptr<CoinStatsIndex> g_coin_stats_index;
    104 
    105-CoinStatsIndex::CoinStatsIndex(size_t n_cache_size, bool f_memory, bool f_wipe)
    106+CoinStatsIndex::CoinStatsIndex(std::unique_ptr<interfaces::Chain> chain, size_t n_cache_size, bool f_memory, bool f_wipe) : BaseIndex(std::move(chain))
    


    maflcko commented at 2:30 pm on July 13, 2022:
    2f8a5307b7065de08df2edaa369739544248387f: (nit) Start newline before : to avoid excessive line length?

    ryanofsky commented at 7:28 pm on July 14, 2022:

    re: #25494 (review)

    2f8a530: (nit) Start newline before : to avoid excessive line length?

    Now moved to new line

  19. in src/init.cpp:1604 in 2f8a5307b7 outdated
    1588     }
    1589 
    1590     for (const auto& filter_type : g_enabled_filter_types) {
    1591-        InitBlockFilterIndex(filter_type, cache_sizes.filter_index, false, fReindex);
    1592-        if (!GetBlockFilterIndex(filter_type)->Start(chainman.ActiveChainstate())) {
    1593+        InitBlockFilterIndex([&]{ return interfaces::MakeChain(node); }, filter_type, cache_sizes.filter_index, false, fReindex);
    


    maflcko commented at 2:33 pm on July 13, 2022:
    2f8a5307b7065de08df2edaa369739544248387f: Wrapping this into a function to avoid moving a pointer seems overkill?

    ryanofsky commented at 7:28 pm on July 14, 2022:

    re: #25494 (review)

    2f8a530: Wrapping this into a function to avoid moving a pointer seems overkill?

    Reason this is using a std::function is I was looking for a simple way to allow the Init*Index functions to stay where they are in src/index/ without referencing the NodeContext struct. Maybe this isn’t a critical goal though, or there is a better way of achieving it. Since I want to extend #10102 to run indexes separately I will have to revisit this code anyway later, so happy to go with whatever would be suggested now if this is not good.

  20. in src/Makefile.am:881 in 88cad3cb60 outdated
    877@@ -878,6 +878,7 @@ libbitcoinkernel_la_SOURCES = \
    878   key.cpp \
    879   logging.cpp \
    880   node/blockstorage.cpp \
    881+  node/chain.cpp \
    


    maflcko commented at 3:32 pm on July 13, 2022:
    88cad3cb604058d8f26790eeb37abe2dabc0c033: Could explain in the commit message that this is only temporary? If not, move it to the kernel folder?

    ryanofsky commented at 7:29 pm on July 14, 2022:

    re: #25494 (review)

    88cad3c: Could explain in the commit message that this is only temporary? If not, move it to the kernel folder?

    Moved to kernel folder

  21. in src/index/blockfilterindex.cpp:231 in 88cad3cb60 outdated
    230+        if (!m_db->Read(DBHeightKey(block.height - 1), read_out)) {
    231             return false;
    232         }
    233 
    234-        uint256 expected_block_hash = pindex->pprev->GetBlockHash();
    235+        uint256 expected_block_hash = *block.prev_hash;
    


    maflcko commented at 3:32 pm on July 13, 2022:
    88cad3cb604058d8f26790eeb37abe2dabc0c033: I wonder where UB should be avoided with asserts and where not

    ryanofsky commented at 7:29 pm on July 14, 2022:

    re: #25494 (review)

    88cad3c: I wonder where UB should be avoided with asserts and where not

    I think it’s reasonable to add Asserts all these places or add a const uint256& PrevHash() helper method which asserts, if the Assert is too verbose. Just to be clear, there shouldn’t be any UB presently here due to the block.height > 0 check above. The Assert is just an extra check against possible future UB that could occur if a bug is introduced somewhere else in the code.

  22. in src/index/blockfilterindex.cpp:240 in 88cad3cb60 outdated
    236@@ -233,18 +237,18 @@ bool BlockFilterIndex::WriteBlock(const CBlock& block, const CBlockIndex* pindex
    237         prev_header = read_out.second.header;
    238     }
    239 
    240-    BlockFilter filter(m_filter_type, block, block_undo);
    241+    BlockFilter filter(m_filter_type, *block.data, block_undo);
    


    maflcko commented at 3:32 pm on July 13, 2022:
    Same

    ryanofsky commented at 7:51 pm on July 14, 2022:

    re: #25494 (review)

    Same

    Added Assert

  23. in src/index/coinstatsindex.cpp:133 in 88cad3cb60 outdated
    133+        if (!m_db->Read(DBHeightKey(block.height - 1), read_out)) {
    134             return false;
    135         }
    136 
    137-        uint256 expected_block_hash{pindex->pprev->GetBlockHash()};
    138+        uint256 expected_block_hash{*block.prev_hash};
    


    maflcko commented at 3:33 pm on July 13, 2022:
    Same

    ryanofsky commented at 7:29 pm on July 14, 2022:

    re: #25494 (review)

    Same

    Added Assert

  24. in src/index/base.cpp:221 in ddbbdcf411 outdated
    226-{
    227-    LOCK(cs_main);
    228     // Don't commit anything if we haven't indexed any block yet
    229     // (this could happen if init is interrupted).
    230     if (m_best_block_index == nullptr) {
    231         return false;
    


    maflcko commented at 3:50 pm on July 13, 2022:
    ddbbdcf4112c3a84e3287e6bf701e0eb592b24c3: This changes behavior: Previously there was a call to error(), now there is none?

    ryanofsky commented at 7:30 pm on July 14, 2022:

    re: #25494 (review)

    ddbbdcf: This changes behavior: Previously there was a call to error(), now there is none?

    Good catch. The error print was lost because of a bad rebase. The m_best_block_index == nullptr check was added in #24117 after this was written. Fixed now

  25. maflcko commented at 3:55 pm on July 13, 2022: member

    Happy to merge as-is. Let me know what you think.

    review ACK ddbbdcf4112c3a84e3287e6bf701e0eb592b24c 🐯

    Signature:

     0-----BEGIN PGP SIGNED MESSAGE-----
     1Hash: SHA512
     2
     3review ACK ddbbdcf4112c3a84e3287e6bf701e0eb592b24c 🐯
     4-----BEGIN PGP SIGNATURE-----
     5
     6iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
     7pUjssQwAplJ1ZbuRcMBriV0HxqVNf02T4stpYurghGghrlmhm8pidPOD4r+gcr+g
     8ECrmBrsbS9Ka/ycxwoM0nel0hEMsXLZWy6ueY0ufksVY8TFgIdVSQJtSLZ9MNuUE
     97n9/Qz+z1tl2isShx5nbEQ+UcRiCfXVryfV0OuGO4RLPSJV9fqYaPBKh85af6hxt
    10P+q4qD4Ql6kWZLPyrdZ+M3JO3KIgRDdGLjXdvFQvNDVDos68HT+UTRvRod2B3kkM
    115I84K4L6SfO/WW5sGXM/G8t7qyC4Sdv906qBP9zGkxFV/D5jj3n+cpgb8vog1RGr
    126riVrC9pPnCC8kSopDgHdHIEiNdnqxeQJ2eiFutmI2B2Gz6ex7YjvBGyeJeRSka5
    13sNtrD+9EKtUE5Ta/Iby8mpl1+lSpzC9vFGJGuhTQg1kNzOV80MSVw480wDZcUrTM
    14vtZoxXpOH3/CnL+8Y+MvxanIahrvS/UApp60iZ848v8pEXtI4cGzUsysRIekUkeD
    15fmaWQU9d
    16=52YB
    17-----END PGP SIGNATURE-----
    
  26. ryanofsky commented at 7:31 pm on July 13, 2022: contributor

    Happy to merge as-is. Let me know what you think.

    Thanks for review Marco. I definitely want to make some improvements based on your comments, so better not to merge for now

  27. in src/node/chain.cpp:1 in ddbbdcf411 outdated
    0@@ -0,0 +1,23 @@
    1+// Copyright (c) 2022 The Bitcoin Core developers
    


    maflcko commented at 6:41 am on July 14, 2022:
    Maybe add this new file to iwyu?

    ryanofsky commented at 7:28 pm on July 14, 2022:

    re: #25494 (review)

    Maybe add this new file to iwyu?

    Thanks, added



    maflcko commented at 7:41 am on July 15, 2022:
    Looks like an iwyu bug, so maybe undo for now?

    ryanofsky commented at 12:22 pm on July 15, 2022:

    re: #25494 (review)

    https://cirrus-ci.com/task/4504627910541312?logs=ci#L6998

    Looks like an iwyu bug, so maybe undo for now?

    I’m not actually sure what the bug is but I added some missing includes and undid change that adds the file to IWYU for now. The larger PR #24230 this was part of adds more to this file so there is a chance to enable IWYU again later

  28. ryanofsky force-pushed on Jul 14, 2022
  29. ryanofsky commented at 11:51 pm on July 14, 2022: contributor
    Updated ddbbdcf4112c3a84e3287e6bf701e0eb592b24c3 -> fb245ec3d3644cc9cef9b990ba2c8313277f5d41 (pr/ind.2 -> pr/ind.3, compare) with suggested changes like moving the new file from node/ to kernel/ Updated fb245ec3d3644cc9cef9b990ba2c8313277f5d41 -> f678151ff9a0a5934965b1c85ec97a1dfe45a67e (pr/ind.3 -> pr/ind.4, compare) to fix lint error https://cirrus-ci.com/task/6068993915092992
  30. ryanofsky force-pushed on Jul 15, 2022
  31. in src/index/base.cpp:221 in f678151ff9 outdated
    228     // Don't commit anything if we haven't indexed any block yet
    229     // (this could happen if init is interrupted).
    230-    if (m_best_block_index == nullptr) {
    231-        return false;
    232+    bool ok = m_best_block_index != nullptr;
    233+    if (ok) {
    


    maflcko commented at 7:39 am on July 15, 2022:

    This is still wrong?

    Previously the call order was CoinStatsIndex::CustomCommit , then BaseIndex::CommitInternal (which may be skipped).

    Now, everything is skipped?


    ryanofsky commented at 11:55 am on July 15, 2022:

    re: #25494 (review)

    In commit “indexes, refactor: Remove CChainState use in index CommitInternal method” (f678151ff9a0a5934965b1c85ec97a1dfe45a67e)

    This is still wrong?

    Previously the call order was CoinStatsIndex::CustomCommit , then BaseIndex::CommitInternal (which may be skipped).

    Now, everything is skipped?

    You are referring to the m_best_block_index == nullptr case? If so then yes the whole commit is skipped now when previously only half of the commit was skipped (custom part was not skipped, base part was skipped).

    I didn’t notice yesterday that only the base half of the commit was skipped, but it’s definitely better to skip the whole commit. This m_block_index == nullptr case was only added very recently in https://github.com/bitcoin/bitcoin/pull/24117/commits/0243907faee0aa6af09974131d9a46a7f9c3ef38 as part of an attempt to avoid index corruption if state was flushed during startup that would happen because GetLocator can return null on startup. It was an imperfect attempt and according to comments in #24117 (comment), corruption continued to happen after that PR. It’s possible this commit might inadvertently fix this by skipping the whole commit instead of half of it, but it’s not clear. In any case, it should be better behavior and it makes more sense to skip the whole commit.


    mzumsande commented at 6:00 pm on July 19, 2022:

    I think that skipping the error message in the m_block_index == nullptr case would have actually been ok: Right now, everytime one starts to sync a new index with an existing chain, this would appear once in the error log, even though it is expected and not actually an error. But possibly this is fixed anyway by a later commit from #24230 rewriting the init logic anyway into not attempting to call commit before actually doing some indexing?

    I agree that skipping the entire thing custom+base commit makes more sense, but I can’t see how this could possibly avoid additional cases of corruption or be the cause of #24117 (comment), since the custom code in both coinstatsindex and blockfilterindex doesn’t make use of m_block_index and just prepares custom data in the batch without committing.

  32. maflcko commented at 7:39 am on July 15, 2022: member

    f678151ff9a0a5934965b1c85ec97a1dfe45a67e 📭

    Signature:

     0-----BEGIN PGP SIGNED MESSAGE-----
     1Hash: SHA512
     2
     3f678151ff9a0a5934965b1c85ec97a1dfe45a67e 📭
     4-----BEGIN PGP SIGNATURE-----
     5
     6iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
     7pUhL5AwAtnEh0cX8QFjSw7c20rH3WCFif2w8QDJpwnRDHHmc8qSzVamRRmmjGaZ2
     8uT2IvZCVzzLqvcJKWJR6Woan9pS5GJtF/pqBxv1olJd8CBNO1VrL3o5NnBzKIDAN
     96PWq4Tl50TrIwnR5Wnd8pEYfbK77M4ghdSZ5TYtzL+gHTL8wQcYGmJCw7pDAgeAp
    10iMMe82rhI/sYeU9nqo3l5MF2IgJkzEuQ/wpxD4rmkDyf+9yLNtquHCBzR5BSPhDl
    115LoR7v9LR5WqHZzOWyrrY/FhUmnbYDpmub8TAmYGj6cy+in37jLheDY4EDU2mzJG
    12CMtE/6yRGY8dPFohG//yUquEfHG7lVacW+9tMXrRZ0+zcmlBFcobOtwNRJ4Oc4qZ
    13IcmULASpFcVpFbLrOTqS5n/zrIvicQSxH4nXGXNnB7gvZFT4lmpnOin6txgN/Sic
    14sf20wlYEbQpJBWgBtRjSny8SuY5BHJeAypVzIbw1Sc78GpZs2m3bmjHZHuwud97p
    15FKbfDUd0
    16=d7Fu
    17-----END PGP SIGNATURE-----
    
  33. ryanofsky force-pushed on Jul 15, 2022
  34. ryanofsky commented at 12:21 pm on July 15, 2022: contributor
    Updated f678151ff9a0a5934965b1c85ec97a1dfe45a67e -> 8bc1602821202d4863fab05ee1aab3ff3b70a7c0 (pr/ind.4 -> pr/ind.5, compare) updating commit message 8bc1602821202d4863fab05ee1aab3ff3b70a7c0 to describe some changed behavior and no longer enabling IWYU for now in the new file
  35. maflcko commented at 12:47 pm on July 15, 2022: member

    ACK 8bc1602821202d4863fab05ee1aab3ff3b70a7c0 though did not review the last commit 🗡

    Signature:

     0-----BEGIN PGP SIGNED MESSAGE-----
     1Hash: SHA512
     2
     3ACK 8bc1602821202d4863fab05ee1aab3ff3b70a7c0 though did not review the last commit 🗡
     4-----BEGIN PGP SIGNATURE-----
     5
     6iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
     7pUhkpQv9EYKytsqsNKtA9Yte0NOoa/hTFzjqhx34b+kkeB9twTIhnl/8kqLcct6a
     81983DDhe835/ZeKxwAGPV9KTnBma+E7Sr9nAaFuY8GJfA2qsFncXzoeRpZRW8e9m
     9EjZ1Yvi8Yac3AE9RArPayknO0uo7tvlYoL/2mx7k+6V/VvlZSAqrUMgwQY8PvQmJ
    10c6AoUvbUrMSy6HM4HZHi9TEVveQqwMqYwPwKRqy0A5CBAlwIN75m/NKFSDWcjVTS
    11p3m88uM5fZQ1gQ4Xu9FTvXSQavIKr883gUeUXOR9yv5FXgXHCRVGjrbpVEey2nmK
    12jOwdKWHVpVv0+6jI1qzJnkTlKdKvFKcvTdPp9EW/woWW4FdcEAtAG4s0kDfvQrN6
    13b8ZUUpowDJsM+3Uisuqq6dhq48j56u0Ux0RmXBTGsCT4t8hpPSUbsAqqSZnhpWlJ
    14SKw4c9amB3i8kcRLIBkpntEsOgtZnXBaNgdlpv3iWUALC8xqBOF8RDi10qMcS0x2
    151/7likbu
    16=LR6p
    17-----END PGP SIGNATURE-----
    
  36. ryanofsky commented at 2:20 pm on July 18, 2022: contributor
    @fjahr & @mzumsande could you reack? I think only change since previous reviews has been expanding the m_block_index == nullptr case in CommitInternal to keep logging an error and avoid committing anything
  37. DrahtBot added the label Needs rebase on Jul 18, 2022
  38. interfaces, refactor: Add more block information to block connected notifications
    Add new interfaces::BlockInfo struct to be able to pass extra block
    information (file and undo information) to indexes which they are
    updated to use high level interfaces::Chain notifications.
    
    This commit does not change behavior in any way.
    a0b5b4ae5a
  39. indexes, refactor: Pass Chain interface instead of CChainState class to indexes
    Passing abstract Chain interface will let indexes run in separate
    processes.
    
    This commit does not change behavior in any way.
    33b4d48cfc
  40. indexes, refactor: Remove CBlockIndex* uses in coinstatsindex LookUpOne function
    This commit does not change behavior in any way.
    addb4f2af1
  41. indexes, refactor: Remove CBlockIndex* uses in index Init methods
    Replace overriden index Init() methods that use the best block
    CBlockIndex* pointer with pure CustomInit() callbacks that are passed
    the block hash and height.
    
    This gets rid of more CBlockIndex* pointer uses so indexes can work
    outside the bitcoin-node process. It also simplifies the initialization
    call sequence so index implementations are not responsible for
    initializing the base class.
    
    There is a slight change in behavior here since now the best block
    pointer is loaded and checked before the custom index init functions are
    called instead of while they are called.
    bef4e405f3
  42. indexes, refactor: Remove CBlockIndex* uses in index WriteBlock methods
    Replace WriteBlock method with CustomAppend and pass BlockInfo struct
    instead of CBlockIndex* pointer
    
    This commit does not change behavior in any way.
    dc971be083
  43. indexes, refactor: Remove CBlockIndex* uses in index Rewind methods
    Replace Rewind method with CustomRewind and pass block hashes and
    heights instead of CBlockIndex* pointers
    
    This commit does not change behavior in any way.
    ee3a079fab
  44. indexes, refactor: Remove CChainState use in index CommitInternal method
    Replace CommitInternal method with CustomCommit and use interfaces::Chain
    instead of CChainState to generate block locator.
    
    This commit does not change behavior in any way, except in the
    (m_best_block_index == nullptr) case, which was added recently in
    https://github.com/bitcoin/bitcoin/pull/24117 as part of an ongoing attempt to
    prevent index corruption if bitcoind is interrupted during startup. New
    behavior in that case should be slightly better than the old behavior (skipping
    the entire custom+base commit now vs only skipping the base commit previously)
    and this might avoid more cases of corruption.
    7878f97bf1
  45. ryanofsky force-pushed on Jul 18, 2022
  46. ryanofsky commented at 7:06 pm on July 18, 2022: contributor
    Rebased 8bc1602821202d4863fab05ee1aab3ff3b70a7c0 -> 7878f97bf15b6e7c9b47d1c0d96419b97e1bdcbd (pr/ind.5 -> pr/ind.6, compare) due to conflicts with #23997 and #25487
  47. DrahtBot removed the label Needs rebase on Jul 18, 2022
  48. maflcko commented at 8:27 am on July 19, 2022: member

    ACK 7878f97bf15b6e7c9b47d1c0d96419b97e1bdcbd though did not review the last commit 🤼

    Signature:

     0-----BEGIN PGP SIGNED MESSAGE-----
     1Hash: SHA512
     2
     3ACK 7878f97bf15b6e7c9b47d1c0d96419b97e1bdcbd though did not review the last commit 🤼
     4-----BEGIN PGP SIGNATURE-----
     5
     6iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
     7pUgZAAwAg2W4ylK74K+MHoP2XiZFNNZe6tzS3wpYnBUZXoP0ee1rOg/lNGl1o2ER
     8d0zs+tGjHajkpDiQB9DgCqNLQAEq1qOCRmxH1W9nt04gkVqyZPPcOg7GE2da1MNL
     9n8P8gNTHiqmS8FrRSAs40udE/VuhSr9j/4OLaCEjQiRdhWPKB1J4IduIK0OMQdW0
    10LQw2wBXRb0rtG4L0ztwmndFXx9IbBLRNZCFRR6a2IW4/2rZN3FmPSr6kDA7PV9S+
    11bM/+NNPThUga3KTfbPBj5ERivN/4UZXILyOdFVxs+dOcPViZVvtWd8uBLsiH+2QO
    125zSuVeXnWPSujZXcNKpWnXbUJv7Pi8ClPkBv2016w5hzGGYOngV/TzLyeZtihn3S
    13IXQ7afP8zbCPBQvtcVUGB3QLZLvl1Byw88U+6OXSp0Yw/gEXXQLHujyLOI2G2/9P
    14RwNmi+6qDhhylB5TPist30uVk4D84NEDG/cz269cHe0+M+icnh2cpwysdM7fMO1S
    15vhWrJkyl
    16=8s9I
    17-----END PGP SIGNATURE-----
    
  49. mzumsande commented at 6:01 pm on July 19, 2022: contributor
    Code Review ACK 7878f97bf15b6e7c9b47d1c0d96419b97e1bdcbd
  50. fanquake merged this on Jul 19, 2022
  51. fanquake closed this on Jul 19, 2022

  52. sidhujag referenced this in commit d5838f1b8b on Jul 20, 2022
  53. in src/index/base.cpp:365 in bef4e405f3 outdated
    358@@ -359,7 +359,10 @@ bool BaseIndex::Start()
    359     // Need to register this ValidationInterface before running Init(), so that
    360     // callbacks are not missed if Init sets m_synced to true.
    361     RegisterValidationInterface(this);
    362-    if (!Init()) {
    363+    if (!Init()) return false;
    364+
    365+    const CBlockIndex* index = m_best_block_index.load();
    366+    if (!CustomInit(index ? std::make_optional(interfaces::BlockKey{index->GetBlockHash(), index->nHeight}) : std::nullopt)) {
    


    ryanofsky commented at 6:01 pm on May 18, 2023:

    In commit “indexes, refactor: Remove CBlockIndex* uses in index Init methods” (bef4e405f3de2718dfee279a9abff4daf016da26)

    Calling the CustomInit() function after the Init() function here seems to cause a race condition, noticed by @furszy and described in #27607 (comment). The problem is that Init() can set m_synced to true, which enables BlockConnected callbacks. But if BlockConnected callback runs before CustomInit finishes the index may still be only partially loaded, and the CustomAppend call in BlockConnected could fail

  54. achow101 referenced this in commit 3a83d4417b on May 31, 2023
  55. sidhujag referenced this in commit 9ec40ffad2 on May 31, 2023
  56. achow101 referenced this in commit bf1b6383db on Mar 20, 2024
  57. bitcoin locked this on May 17, 2024

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-07-05 16:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me