refactor: prohibit direct flags access in CCoinsCacheEntry and remove invalid tests #30906

pull l0rinc wants to merge 9 commits into bitcoin:master from l0rinc:l0rinc/hide-coin-flags changing 5 files +289 −359
  1. l0rinc commented at 1:39 pm on September 15, 2024: contributor

    Similarly to #30849, this cleanup is intended to de-risk #30673 (review) by simplifying the coin cache public interface.

    CCoinsCacheEntry provided general access to its internal flags state, even though, in reality, it could only be clean, fresh, dirty, or fresh|dirty (in the follow-up, we will remove fresh without dirty).

    Once it was marked as dirty, we couldn’t set the state back to clean with AddFlags(0)—tests explicitly checked against that.

    This PR refines the public interface to make this distinction clearer and to make invalid behavior impossible, rather than just checked by tests. We don’t need extensive access to the internals of CCoinsCacheEntry, as many tests were simply validating invalid combinations in this way.

    The last few commits contain significant test refactorings to make coins_tests easier to change in follow-ups.

  2. DrahtBot commented at 1:39 pm on September 15, 2024: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/30906.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK andrewtoth, ryanofsky
    Concept ACK davidgumberg, hodlinator

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #31308 (ci, iwyu: Treat warnings as errors for specific targets by hebasto)
    • #31306 (ci: Update Clang in “tidy” job by hebasto)
    • #31132 (validation: fetch block inputs on parallel threads ~17% faster IBD by andrewtoth)
    • #30643 (coins: Add move operations to Coin and CCoinsCacheEntry by l0rinc)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  3. DrahtBot added the label UTXO Db and Indexes on Sep 15, 2024
  4. l0rinc renamed this:
    coins: prohibit direct flags access in CCoinsCacheEntry and remove invalid tests
    refactor: prohibit direct flags access in CCoinsCacheEntry and remove invalid tests
    on Sep 15, 2024
  5. in src/test/coins_tests.cpp:623 in 03cf2bd3e1 outdated
    616@@ -616,7 +617,9 @@ void GetCoinsMapEntry(const CCoinsMap& map, CAmount& value, char& flags, const C
    617         } else {
    618             value = it->second.coin.out.nValue;
    619         }
    620-        flags = it->second.GetFlags();
    621+        flags = 0;
    622+        if (it->second.IsDirty()) flags |= DIRTY;
    623+        if (it->second.IsFresh()) flags |= FRESH;
    624         assert(flags != NO_ENTRY);
    


    andrewtoth commented at 1:52 pm on September 15, 2024:
    This assertion is obviated now.

    l0rinc commented at 6:35 pm on September 15, 2024:
    Replaced it with a flag validation
  6. in src/test/coins_tests.cpp:710 in 03cf2bd3e1 outdated
    710@@ -708,7 +711,7 @@ static void CheckSpendCoins(CAmount base_value, CAmount cache_value, CAmount exp
    711     GetCoinsMapEntry(test.cache.map(), result_value, result_flags);
    712     BOOST_CHECK_EQUAL(result_value, expected_value);
    713     BOOST_CHECK_EQUAL(result_flags, expected_flags);
    714-};
    715+}
    


    andrewtoth commented at 1:54 pm on September 15, 2024:
    Why was this changed?

    l0rinc commented at 4:01 pm on September 15, 2024:
    Because class declarations don’t need semicolons, but I’ll revert if you find it distracting
  7. in src/test/coinscachepair_tests.cpp:188 in 03cf2bd3e1 outdated
    213     BOOST_CHECK_EQUAL(sentinel.second.Next(), &n2);
    214     BOOST_CHECK_EQUAL(sentinel.second.Prev(), &n2);
    215     BOOST_CHECK_EQUAL(n2.second.Next(), &sentinel);
    216     BOOST_CHECK_EQUAL(n2.second.Prev(), &sentinel);
    217 
    218     // But adding DIRTY re-inserts it after n2
    


    andrewtoth commented at 1:57 pm on September 15, 2024:
    But can be removed now.

    l0rinc commented at 6:36 pm on September 15, 2024:
    Done

    ryanofsky commented at 6:58 pm on October 30, 2024:

    re: #30906 (review)

    In commit “coins: Split up AddFlags to remove invalid states” (8e1381a991e325395fbc10df2fb091c508efa0f3)

    Done

    Seems like this is not actually done as of 8e1381a991e325395fbc10df2fb091c508efa0f3


    l0rinc commented at 1:17 pm on October 31, 2024:
    Was already removed in coins: Remove direct GetFlags access, but I’ve moved it up to coins, refactor: Split up AddFlags to remove invalid states
  8. andrewtoth commented at 1:58 pm on September 15, 2024: contributor

    Concept ACK

    Can we remove any other tests that are trying to add flags that are not FRESH or DIRTY and so are now useless?

  9. l0rinc force-pushed on Sep 15, 2024
  10. l0rinc commented at 6:39 pm on September 15, 2024: contributor

    Can we remove any other tests that are trying to add flags that are not FRESH or DIRTY and so are now useless?

    I’ve merged NO_ENTRY and char(0) values - this simplifies the cases somewhat. Do you think we can remove any of the lines? There aren’t any duplicates, even after the above change…

    Edit: moved the booleans closer to the edges in later commits.

  11. l0rinc force-pushed on Sep 15, 2024
  12. l0rinc force-pushed on Sep 15, 2024
  13. DrahtBot added the label CI failed on Sep 15, 2024
  14. DrahtBot removed the label CI failed on Sep 15, 2024
  15. in src/test/coins_tests.cpp:570 in 390c670c37 outdated
    570@@ -571,11 +571,19 @@ const static CAmount VALUE2 = 200;
    571 const static CAmount VALUE3 = 300;
    572 const static char DIRTY = CCoinsCacheEntry::DIRTY;
    573 const static char FRESH = CCoinsCacheEntry::FRESH;
    574-const static char NO_ENTRY = -1;
    


    andrewtoth commented at 9:47 pm on September 15, 2024:

    Since we don’t pass a bitmap as flags anymore, does it make sense to get rid of the char type for flags in coins_tests and replace with a more descriptive enum?

    0enum Flags
    1{
    2   Clean,
    3   Dirty,
    4   Fresh,
    5   DirtyAndFresh
    6};
    

    Could also be a std::optional<Flags> so we can handle the NO_ENTRY case cleanly instead of with a -1.


    l0rinc commented at 10:53 pm on September 15, 2024:

    what’s the difference between clean and NO_ENTRY? Does that difference still make sense after this change?

    get rid of the char type for flags in coins_tests and replace with a more descriptive enum

    Since we don’t have flags in the production code anymore I would prefer getting rid of them in the tests as well. Reintroducing an enum would kinda’ defeat that purpose in my opinion.

    To make sure the tests are still checking the same combinations, I have added a few scripted diffs to transition the tests away from the flags - now the tabular data displays dirty and fresh booleans instead (I guess we can group some of the examples into CoinMapEntry as well, if needed). This should help us selectively decide which combinations are still valid. What do you think?


    andrewtoth commented at 10:59 pm on September 15, 2024:
    Clean means not dirty and not fresh. NO_ENTRY is a special value that is not a possible flag but signals to the tests that that entry does not exist. With proper types, that would correspond to a std::nullopt. We don’t have to name the enum Flags, we can name it EntryState. That way we don’t have to validate the flags variable to see if it’s one of the values we will accept. That is better served by using an enum and std::optional.

    andrewtoth commented at 11:03 pm on September 15, 2024:

    I don’t think replacing with true and false is the right approach. Using enums, tests would be like

    0CheckAccessCoin(VALUE1, SPENT , SPENT , DIRTY|FRESH, DIRTY|FRESH);
    1becomes
    2CheckAccessCoin(VALUE1, SPENT , SPENT , DirtyAndFresh, DirtyAndFresh);
    3
    4CheckSpendCoins(ABSENT, SPENT , ABSENT, FRESH      , NO_ENTRY    );
    5becomes
    6CheckSpendCoins(ABSENT, SPENT , ABSENT, Fresh      , std::nullopt);
    

    l0rinc commented at 11:03 pm on September 15, 2024:

    is a special value that is not a possible flag but signals to the tests that that entry does not exist

    That is better served by using an enum and std::optional.

    Thanks, I’ll investigate tomorrow. Please leave some comments about the overall direction as well, do you think this new test layout is more readable or less? Drafting until then.


    l0rinc commented at 4:04 pm on September 17, 2024:
    Did a significant restructure, the tabular tests should be a lot easier to comprehend and modify now. Also testing the exception messages directly, you’ll need this in your new PR. I still need to get rid of remaining flags in the tests, will do that tomorrow - any early feedback until then is appreciated.

    l0rinc commented at 6:43 am on September 18, 2024:
    Done
  16. l0rinc marked this as a draft on Sep 15, 2024
  17. l0rinc force-pushed on Sep 17, 2024
  18. in src/test/coins_tests.cpp:569 in ba52ebd416 outdated
    563@@ -562,6 +564,15 @@ BOOST_AUTO_TEST_CASE(ccoins_serialization)
    564     }
    565 }
    566 
    567+struct CoinStruct {
    568+    const CAmount value;
    569+    const char flags;
    


    andrewtoth commented at 4:29 pm on September 17, 2024:
    I still think this should be an enum named EntryState or something, with all combinations of flags like https://github.com/bitcoin/bitcoin/pull/30906/commits/ba52ebd41673abf63d4495215c84be7244469d30#r1760349038. That’s the proper way to do this instead of defining static vars for each value of a char we care about.

    l0rinc commented at 4:32 pm on September 17, 2024:
    Absolutely, that’s my plan for tomorrow - but needed to clean up the test before doing that

    l0rinc commented at 6:52 am on September 18, 2024:
    Done
  19. in src/test/coins_tests.cpp:702 in c4bf4d76fe outdated
    730+    CheckAccessCoin(VALUE1, {SPENT , DIRTY      }, {SPENT , DIRTY      });
    731+    CheckAccessCoin(VALUE1, {SPENT , DIRTY|FRESH}, {SPENT , DIRTY|FRESH});
    732+    CheckAccessCoin(VALUE1, {VALUE2, UNCHANGED  }, {VALUE2, UNCHANGED  });
    733+    CheckAccessCoin(VALUE1, {VALUE2, FRESH      }, {VALUE2, FRESH      });
    734+    CheckAccessCoin(VALUE1, {VALUE2, DIRTY      }, {VALUE2, DIRTY      });
    735+    CheckAccessCoin(VALUE1, {VALUE2, DIRTY|FRESH}, {VALUE2, DIRTY|FRESH});
    


    davidgumberg commented at 6:45 pm on September 17, 2024:
    Nit: Could be made into a loop as below

    l0rinc commented at 7:01 am on September 18, 2024:
    Done, thanks!
  20. davidgumberg commented at 6:55 pm on September 17, 2024: contributor
    Concept ACK on dropping the bitfield interface for CCoinsCacheEntry flags
  21. l0rinc force-pushed on Sep 18, 2024
  22. l0rinc force-pushed on Sep 18, 2024
  23. l0rinc marked this as ready for review on Sep 18, 2024
  24. in src/coins.h:136 in b2f9df7817 outdated
    127@@ -128,6 +128,22 @@ struct CCoinsCacheEntry
    128     CoinsCachePair* m_next{nullptr};
    129     uint8_t m_flags{0};
    130 
    131+    //! Adding a flag also requires a self reference to the pair that contains
    132+    //! this entry in the CCoinsCache map and a reference to the sentinel of the
    133+    //! flagged pair linked list.
    134+    inline void AddFlags(uint8_t flags, CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
    135+    {
    136+        assert(flags & DIRTY || flags & FRESH);
    


    hodlinator commented at 9:27 am on September 18, 2024:

    nit: terseness++

    0        assert(flags & (DIRTY | FRESH));
    

    l0rinc commented at 8:14 am on September 19, 2024:
    Clever, thanks!
  25. in src/coins.h:186 in b2f9df7817 outdated
    193+    }
    194+    inline void SetFresh(CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
    195+    {
    196+        AddFlags(FRESH, self, sentinel);
    197     }
    198     inline void ClearFlags() noexcept
    


    andrewtoth commented at 1:11 pm on September 18, 2024:
    0    inline void SetClean() noexcept
    


    hodlinator commented at 1:57 pm on September 18, 2024:

    While “clean” is the opposite of “dirty”, it doesn’t involve how fresh something is. It’s on another dimension.

    I actually prefer ClearFlags(), ClearState() or possibly Reset-something().

    Same goes for Caching::CLEAN. Would prefer VOID/EMPTY/INIT/NULL.


    andrewtoth commented at 10:27 pm on September 18, 2024:

    fresh is actually a substate of dirty. Being fresh implies dirty, so cleaning implies no longer fresh.

    See #30673 which inspired this, which intends to remove possibility of FRESH-but-not-DIRTY entries.


    l0rinc commented at 7:13 am on September 19, 2024:

    FRESH is definitely a very confusing concept

    the parent cache does not have this coin or that it is a spent coin in the parent cache

    Maybe after #30673 we can find a better name (and description) for it


    hodlinator commented at 7:39 pm on September 19, 2024:
    Okay, so FRESH and DIRTY_FRESH will merge into one. :+1:

    l0rinc commented at 7:49 pm on September 19, 2024:
    rather FRESH will be removed in the mentioned follow-up, since it’s not actually possible (unless in case of reorgs, which I don’t fully understand yet, but not important for this PR)
  26. in src/test/coins_tests.cpp:651 in b2f9df7817 outdated
    670-        flags = it->second.GetFlags();
    671-        assert(flags != NO_ENTRY);
    672+    if (auto it{map.find(outp)}; it != map.end()) {
    673+        return CoinStruct{
    674+            it->second.coin.IsSpent() ? SPENT : it->second.coin.out.nValue,
    675+            ToCaching(it->second.IsDirty(), it->second.IsFresh())};
    


    andrewtoth commented at 1:16 pm on September 18, 2024:
    Do we need a new function if we’re only calling it the one time here? Just seems like unnecessary indirection.

    l0rinc commented at 1:36 pm on September 18, 2024:
    Converting flags to enums isn’t the responsibility of this method, that’s why I’ve extracted it.
  27. l0rinc force-pushed on Sep 18, 2024
  28. l0rinc force-pushed on Sep 18, 2024
  29. in src/test/coins_tests.cpp:576 in 5d6937c7ce outdated
    571+    DIRTY_FRESH,
    572+};
    573+
    574+struct CoinStruct {
    575+    const CAmount value;
    576+    const Caching caching;
    


    hodlinator commented at 9:52 pm on September 18, 2024:

    Types should for the most part be nouns IMO. Seldom verbs in present participle like Caching. CoinStruct feels a bit WIP too.

    Suggestions:

    0struct CoinState {
    1    const CAmount value;
    2    const CacheState cache;
    
    0struct CoinCache {
    1    enum State {
    2        CLEAN,
    3        DIRTY,
    4        FRESH,
    5        DIRTY_FRESH,
    6    };
    7    const CAmount value;
    8    const State state;
    

    l0rinc commented at 8:13 am on September 19, 2024:
    Had a few versions before, played around with it a bit more, thanks for the pointers, what do you think of the new structure and names.

    hodlinator commented at 7:45 pm on September 19, 2024:
    Like it! Moves related things closer together.

    l0rinc commented at 7:50 pm on September 19, 2024:
    The usages are a lot more verbose, but I’ve extracted everything important now, so the tables are still readable
  30. hodlinator commented at 10:03 pm on September 18, 2024: contributor

    Concept ACK.

    Really like the move to put restrictions on how flags are used.

  31. l0rinc force-pushed on Sep 19, 2024
  32. in src/coins.h:136 in a595328a4a outdated
    127@@ -128,6 +128,22 @@ struct CCoinsCacheEntry
    128     CoinsCachePair* m_next{nullptr};
    129     uint8_t m_flags{0};
    130 
    131+    //! Adding a flag also requires a self reference to the pair that contains
    132+    //! this entry in the CCoinsCache map and a reference to the sentinel of the
    133+    //! flagged pair linked list.
    134+    inline void AddFlags(uint8_t flags, CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
    135+    {
    136+        assert(flags & (DIRTY | FRESH));
    


    andrewtoth commented at 1:52 am on September 20, 2024:
    I think this should be Assume. We should prefer it since it will be compiled out for release builds and this is a hot path.

    l0rinc commented at 3:50 pm on September 21, 2024:
    This was meant to be a temporary transition, since we’re removing the rest of the flags in your PR - but I’m fine with Assume as well - changed.
  33. in src/coins.h:164 in a595328a4a outdated
    176+        SetClean();
    177     }
    178 
    179-    //! Adding a flag also requires a self reference to the pair that contains
    180-    //! this entry in the CCoinsCache map and a reference to the sentinel of the
    181-    //! flagged pair linked list.
    


    andrewtoth commented at 1:56 am on September 20, 2024:
    Hmm not sure we should remove the extra context here about what exactly self and sentinel are and where they come from.

    l0rinc commented at 3:50 pm on September 21, 2024:

    This was moved to the AddFlags already - even though it seems to me the code already makes all of that clear:

    Adding a flag also requires a self reference to the pair that contains this entry in the CCoinsCache map

    AddFlags(uint8_t flags, CoinsCachePair& self

    and a reference to the sentinel of the flagged pair linked list

    , CoinsCachePair& sentinel


    andrewtoth commented at 4:22 pm on September 22, 2024:

    Ah yes I see it was moved up with AddFlags to private. Maybe better to keep it on SetDirty.

    Adding a flag also requires a self reference to the pair that contains this entry in the CCoinsCache map

    There’s no context in the method signature that it is an entry in the CCoinsCache map.

    and a reference to the sentinel of the flagged pair linked list

    There’s no context in the method signature about a flagged linked list.


    l0rinc commented at 4:46 pm on September 22, 2024:

    There’s no context in the method signature that it is an entry in the CCoinsCache map

    isn’t that what CoinsCachePair means?

    method signature about a flagged linked list

    The sentinel hints at it.

    But, again, I kept the methods for AddFlags - and don’t want to duplicate it for SetFresh and SetDirty.

  34. andrewtoth commented at 2:13 am on September 20, 2024: contributor

    I think the commit title test: Migrate GetCoinsMapEntry to return std::optional<CoinState> is no longer accurate.

    I didn’t find splitting the changes in coins_tests up into that many commits to be helpful for review. I ended up reviewing the combined diff for that file. Will review in more depth later.

  35. l0rinc force-pushed on Sep 21, 2024
  36. l0rinc force-pushed on Sep 21, 2024
  37. in src/test/coins_tests.cpp:573 in 1a89a97b1a outdated
    568+    enum class State { CLEAN, DIRTY, FRESH, DIRTY_FRESH };
    569+
    570+    const CAmount value;
    571+    const State state;
    572+
    573+    CoinEntry(CAmount v, State s) : value(v), state(s) {}
    


    andrewtoth commented at 4:31 pm on September 22, 2024:
    0    constexpr CoinEntry(CAmount v, State s) : value(v), state(s) {}
    

    then all const static MaybeCoin declarations below can be made constexpr.


    l0rinc commented at 11:41 am on September 23, 2024:
    Done, thanks
  38. in src/test/coins_tests.cpp:757 in 1a89a97b1a outdated
    912-    CheckAddCoin(VALUE2, VALUE3, FAIL  , DIRTY      , NO_ENTRY   , false);
    913-    CheckAddCoin(VALUE2, VALUE3, VALUE3, DIRTY      , DIRTY      , true );
    914-    CheckAddCoin(VALUE2, VALUE3, FAIL  , DIRTY|FRESH, NO_ENTRY   , false);
    915-    CheckAddCoin(VALUE2, VALUE3, VALUE3, DIRTY|FRESH, DIRTY|FRESH, true );
    916+    for (auto base_value : {ABSENT, SPENT, VALUE1}) {
    917+        CheckAddCoin(base_value, VALUE3, MISSING,            VALUE3_DIRTY_FRESH,   false);
    


    andrewtoth commented at 5:05 pm on September 22, 2024:
    Can we just hardcode value3 for write? It’s the same for every case here.

    l0rinc commented at 5:15 pm on September 22, 2024:
    Wouldn’t that be inconsistent? In every other case, the constant contained both values.

    andrewtoth commented at 5:22 pm on September 22, 2024:
    I mean, if you look at the Write column, each value is VALUE3.

    l0rinc commented at 11:16 am on September 23, 2024:

    I thought of that, but since it has to be different from base_value, seems weird to assume that in a different method instead of the call-site.

    0Base        Write   Cache        Expected      Coinbase
    1base_value, VALUE3, SPENT_DIRTY, VALUE3_DIRTY, false // VALUE3 + SPENT_DIRTY -> VALUE3_DIRTY
    

    makes more sense than

    0Base        Cache        Expected      Coinbase
    1base_value, SPENT_DIRTY, VALUE3_DIRTY, false // SPENT_DIRTY -> VALUE3_DIRTY???
    

    But I don’t mind renaming VALUE3 to VAL3 if you’re worried about verbosity.

  39. l0rinc force-pushed on Sep 23, 2024
  40. in src/test/coins_tests.cpp:616 in 1dd79a32c9 outdated
    624+constexpr MaybeCoin VALUE2_FRESH       = CoinEntry{VALUE2, CoinEntry::State::FRESH};
    625+constexpr MaybeCoin VALUE2_CLEAN       = CoinEntry{VALUE2, CoinEntry::State::CLEAN};
    626+constexpr MaybeCoin VALUE3_DIRTY       = CoinEntry{VALUE3, CoinEntry::State::DIRTY};
    627+constexpr MaybeCoin VALUE3_DIRTY_FRESH = CoinEntry{VALUE3, CoinEntry::State::DIRTY_FRESH};
    628+
    629+const static std::string EX_OVERWRITE_UNSPENT = "Attempted to overwrite an unspent coin (when possible_overwrite is false)";
    


    andrewtoth commented at 11:51 pm on September 27, 2024:
    Since #30921, we can now make these constexpr string_view.

    l0rinc commented at 6:33 am on September 28, 2024:
  41. l0rinc force-pushed on Sep 28, 2024
  42. andrewtoth commented at 0:13 am on September 29, 2024: contributor

    crACK cb9be6729c4e27bf2d6d703c0d454a22cbcdb6e1

    This simplifies the CCoinsCacheEntry interface, which was leaking the flags bitfield as an implementation detail. Now we don’t have to worry about fuzzing that, and it will make removing the FRESH-but-not-DIRTY or FRESH-and-spent cases easier in #30673.

    Also, the tests are much saner now and easier to parse by also removing the flags interface there.

    There are some minor issues with the commits though.

    Commit titles test: Migrate GetCoinsMapEntry to return std::optional<CoinState> and test: Group values and states in tests into CoinState wrappers should reference CoinEntry now.

    test: Validate error messages on fail should add some context in the commit message or description that it is for CCoinsViewCache AddCoin and WriteCoin unit tests.

  43. DrahtBot requested review from hodlinator on Sep 29, 2024
  44. DrahtBot requested review from davidgumberg on Sep 29, 2024
  45. l0rinc force-pushed on Sep 30, 2024
  46. l0rinc force-pushed on Sep 30, 2024
  47. l0rinc commented at 1:08 pm on September 30, 2024: contributor

    No code changes since https://github.com/bitcoin/bitcoin/commit/cb9be6729c4e27bf2d6d703c0d454a22cbcdb6e1, only commit messages (as requested) and rebased on latest master (separately).

    Edit: git range-diff cb9be67~7..cb9be67 48047ce~7..48047ce

  48. bitcoin deleted a comment on Oct 14, 2024
  49. bitcoin deleted a comment on Oct 14, 2024
  50. DrahtBot added the label Needs rebase on Oct 24, 2024
  51. l0rinc force-pushed on Oct 24, 2024
  52. DrahtBot removed the label Needs rebase on Oct 24, 2024
  53. in src/coins.cpp:111 in 8e1381a991 outdated
    107@@ -107,13 +108,8 @@ void CCoinsViewCache::AddCoin(const COutPoint &outpoint, Coin&& coin, bool possi
    108 
    109 void CCoinsViewCache::EmplaceCoinInternalDANGER(COutPoint&& outpoint, Coin&& coin) {
    110     cachedCoinsUsage += coin.DynamicMemoryUsage();
    111-    auto [it, inserted] = cacheCoins.emplace(
    112-        std::piecewise_construct,
    


    laanwj commented at 10:55 am on October 29, 2024:
    Was this verbose code that is replaced here a workaround for lack of try_emplace?

    l0rinc commented at 3:19 pm on October 29, 2024:

    It seems this was introduced in 2017, when the project was still on C++11, but try_emplace was only introduced in C++17.

    When this was split out to EmplaceCoinInternalDANGER, the old impl was copied (on Aug 25, 2020), but C++17 support was only enabled in Nov 18, 2020.

    Note that this was already partially upgraded for FetchCoin.

  54. in src/coins.h:134 in 8e1381a991 outdated
    127@@ -128,6 +128,22 @@ struct CCoinsCacheEntry
    128     CoinsCachePair* m_next{nullptr};
    129     uint8_t m_flags{0};
    130 
    131+    //! Adding a flag also requires a self reference to the pair that contains
    132+    //! this entry in the CCoinsCache map and a reference to the sentinel of the
    133+    //! flagged pair linked list.
    134+    inline void AddFlags(uint8_t flags, CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
    


    ryanofsky commented at 12:20 pm on October 30, 2024:

    In commit “coins: Split up AddFlags to remove invalid states” (8e1381a991e325395fbc10df2fb091c508efa0f3)

    Not very important, but this commit would be easier to understand if AddFlags function were not moving at the same time as it being modified. Specifically it is harder to see the new Assume(flags & (DIRTY | FRESH)) change and related changes because of this.

    Would suggest not moving AddFlags this commit, and instead moving it in the next commit “coins: Remove direct GetFlags access” getting rid of both flags methods at the same time.


    l0rinc commented at 1:17 pm on October 31, 2024:
    Good idea, done!
  55. in src/test/coins_tests.cpp:760 in e8e70a4ae3 outdated
    760-    SingleEntryCacheTest test(base_value, cache_value, cache_flags);
    761-
    762-    CAmount result_value;
    763-    char result_flags;
    764     try {
    765+        SingleEntryCacheTest test(base_value, cache_coin);
    


    ryanofsky commented at 12:23 pm on October 30, 2024:

    In commit “test: Migrate GetCoinsMapEntry to return MaybeCoin” (e8e70a4ae3cbd7a648629c2df4b5a7f6f0686438)

    Would be better to leave this outside of the try loop so it is clearer this is not expected to throw std::logic_error, and if it it did, the test framework would catch it.

    Same for CheckWriteCoins test below.


    l0rinc commented at 1:17 pm on October 31, 2024:
    I was hesitant at first (mostly because this will be removed in test: Validate error messages on fail), but we want to always fail when these calls do, so you’re right, thanks!
  56. in src/coins.h:165 in 8e1381a991 outdated
    177     }
    178 
    179-    //! Adding a flag also requires a self reference to the pair that contains
    180-    //! this entry in the CCoinsCache map and a reference to the sentinel of the
    181-    //! flagged pair linked list.
    182-    inline void AddFlags(uint8_t flags, CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
    


    ryanofsky commented at 12:30 pm on October 30, 2024:

    In commit “coins: Split up AddFlags to remove invalid states” (8e1381a991e325395fbc10df2fb091c508efa0f3)

    For this commit and other commits, it would be helpful if there was an indication about whether they were supposed to change behavior or test coverage. Would suggest including “refactor” in the subject for commits which are not supposed to change behavior like “coins, refactor: Split up AddFlags to remove invalid states” or could just say behavior isn’t changing in the commit description. The PR title does include “refactor:” but this isn’t visible at the commit level where it tends to be most useful (in git log and blame)


    l0rinc commented at 1:17 pm on October 31, 2024:
    added refactor to the coins commits, thanks
  57. in src/coins.h:178 in 8e1381a991 outdated
    178 
    179-    //! Adding a flag also requires a self reference to the pair that contains
    180-    //! this entry in the CCoinsCache map and a reference to the sentinel of the
    181-    //! flagged pair linked list.
    182-    inline void AddFlags(uint8_t flags, CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
    183+    inline void SetDirty(CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
    


    ryanofsky commented at 6:29 pm on October 30, 2024:

    In commit “coins: Split up AddFlags to remove invalid states” (8e1381a991e325395fbc10df2fb091c508efa0f3)

    Maybe wait for other opinions to see if this suggestion is worthwhile, but IMO this commit provides a good opportunity to remove the Assume(&self.second == this); footgun and make calling these functions more straightforward:

      0--- a/src/coins.cpp
      1+++ b/src/coins.cpp
      2@@ -49,7 +49,7 @@ CCoinsMap::iterator CCoinsViewCache::FetchCoin(const COutPoint &outpoint) const
      3             cachedCoinsUsage += ret->second.coin.DynamicMemoryUsage();
      4             if (ret->second.coin.IsSpent()) { // TODO GetCoin cannot return spent coins
      5                 // The parent only has an empty entry for this outpoint; we can consider our version as fresh.
      6-                ret->second.SetFresh(*ret, m_sentinel);
      7+                SetFresh(*ret, m_sentinel);
      8             }
      9         } else {
     10             cacheCoins.erase(ret);
     11@@ -95,8 +95,8 @@ void CCoinsViewCache::AddCoin(const COutPoint &outpoint, Coin&& coin, bool possi
     12         fresh = !it->second.IsDirty();
     13     }
     14     it->second.coin = std::move(coin);
     15-    it->second.SetDirty(*it, m_sentinel);
     16-    if (fresh) it->second.SetFresh(*it, m_sentinel);
     17+    SetDirty(*it, m_sentinel);
     18+    if (fresh) SetFresh(*it, m_sentinel);
     19     cachedCoinsUsage += it->second.coin.DynamicMemoryUsage();
     20     TRACE5(utxocache, add,
     21            outpoint.hash.data(),
     22@@ -109,7 +109,7 @@ void CCoinsViewCache::AddCoin(const COutPoint &outpoint, Coin&& coin, bool possi
     23 void CCoinsViewCache::EmplaceCoinInternalDANGER(COutPoint&& outpoint, Coin&& coin) {
     24     cachedCoinsUsage += coin.DynamicMemoryUsage();
     25     auto [it, inserted] = cacheCoins.try_emplace(std::move(outpoint), std::move(coin));
     26-    if (inserted) it->second.SetDirty(*it, m_sentinel);
     27+    if (inserted) SetDirty(*it, m_sentinel);
     28 }
     29 
     30 void AddCoins(CCoinsViewCache& cache, const CTransaction &tx, int nHeight, bool check_for_overwrite) {
     31@@ -139,7 +139,7 @@ bool CCoinsViewCache::SpendCoin(const COutPoint &outpoint, Coin* moveout) {
     32     if (it->second.IsFresh()) {
     33         cacheCoins.erase(it);
     34     } else {
     35-        it->second.SetDirty(*it, m_sentinel);
     36+        SetDirty(*it, m_sentinel);
     37         it->second.coin.Clear();
     38     }
     39     return true;
     40@@ -199,11 +199,11 @@ bool CCoinsViewCache::BatchWrite(CoinsViewCacheCursor& cursor, const uint256 &ha
     41                     entry.coin = it->second.coin;
     42                 }
     43                 cachedCoinsUsage += entry.coin.DynamicMemoryUsage();
     44-                entry.SetDirty(*itUs, m_sentinel);
     45+                SetDirty(*itUs, m_sentinel);
     46                 // We can mark it FRESH in the parent if it was FRESH in the child
     47                 // Otherwise it might have just been flushed from the parent's cache
     48                 // and already exist in the grandparent
     49-                if (it->second.IsFresh()) entry.SetFresh(*itUs, m_sentinel);
     50+                if (it->second.IsFresh()) SetFresh(*itUs, m_sentinel);
     51             }
     52         } else {
     53             // Found the entry in the parent cache
     54@@ -231,7 +231,7 @@ bool CCoinsViewCache::BatchWrite(CoinsViewCacheCursor& cursor, const uint256 &ha
     55                     itUs->second.coin = it->second.coin;
     56                 }
     57                 cachedCoinsUsage += itUs->second.coin.DynamicMemoryUsage();
     58-                itUs->second.SetDirty(*itUs, m_sentinel);
     59+                SetDirty(*itUs, m_sentinel);
     60                 // NOTE: It isn't safe to mark the coin as FRESH in the parent
     61                 // cache. If it already existed and was spent in the parent
     62                 // cache then marking it FRESH would prevent that spentness
     63--- a/src/coins.h
     64+++ b/src/coins.h
     65@@ -131,17 +131,16 @@ private:
     66     //! Adding a flag also requires a self reference to the pair that contains
     67     //! this entry in the CCoinsCache map and a reference to the sentinel of the
     68     //! flagged pair linked list.
     69-    inline void AddFlags(uint8_t flags, CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
     70+    static inline void AddFlags(uint8_t flags, CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
     71     {
     72         Assume(flags & (DIRTY | FRESH));
     73-        Assume(&self.second == this);
     74-        if (!m_flags) {
     75-            m_prev = sentinel.second.m_prev;
     76-            m_next = &sentinel;
     77+        if (!self.second.m_flags) {
     78+            self.second.m_prev = sentinel.second.m_prev;
     79+            self.second.m_next = &sentinel;
     80             sentinel.second.m_prev = &self;
     81-            m_prev->second.m_next = &self;
     82+            self.second.m_prev->second.m_next = &self;
     83         }
     84-        m_flags |= flags;
     85+        self.second.m_flags |= flags;
     86     }
     87 
     88 public:
     89@@ -175,11 +174,11 @@ public:
     90         SetClean();
     91     }
     92 
     93-    inline void SetDirty(CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
     94+    friend void SetDirty(CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
     95     {
     96         AddFlags(DIRTY, self, sentinel);
     97     }
     98-    inline void SetFresh(CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
     99+    friend void SetFresh(CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
    100     {
    101         AddFlags(FRESH, self, sentinel);
    102     }
    

    l0rinc commented at 1:17 pm on October 31, 2024:
    Or maybe even go one step further and make SetDirty/SetFresh static as well - thanks for the hint, pushed in a separate commit with your coauthorship.

    andrewtoth commented at 4:25 pm on November 6, 2024:
    This is a great idea!
  58. in src/coins.h:183 in 8e1381a991 outdated
    198     }
    199-    inline void ClearFlags() noexcept
    200+    inline void SetClean() noexcept
    201     {
    202         if (!m_flags) return;
    203         m_next->second.m_prev = m_prev;
    


    ryanofsky commented at 7:05 pm on October 30, 2024:

    In commit “coins: Split up AddFlags to remove invalid states” (8e1381a991e325395fbc10df2fb091c508efa0f3)

    Not directly related to this PR, but would it make sense for SetClean to set m_prev and m_next to null here?

    It seems that would be better than keeping dangling or invalid pointers. This would also make it possible to add asserts in AddFlags to confirm that the linked list state is valid when it is called.

    Not suggesting any change for this commit, just wondering if this idea makes sense for a possible followup commit or PR.


    l0rinc commented at 1:17 pm on October 31, 2024:
    Good idea, did it in a separate commit, let’s see what the CI and others think.
  59. in src/test/coinscachepair_tests.cpp:24 in 8df16f23a2 outdated
    20@@ -21,7 +21,7 @@ std::list<CoinsCachePair> CreatePairs(CoinsCachePair& sentinel)
    21         auto node{std::prev(nodes.end())};
    22         node->second.SetDirty(*node, sentinel);
    23 
    24-        BOOST_CHECK_EQUAL(node->second.GetFlags(), CCoinsCacheEntry::DIRTY);
    25+        BOOST_CHECK(node->second.IsDirty());
    


    ryanofsky commented at 7:13 pm on October 30, 2024:

    In commit “coins: Remove direct GetFlags access” (8df16f23a22b2e0b72dd6338ed37c09485f5ede3)

    May should check !IsFresh() here so new check is equivalent to previous check. Alternately, could just mention in commit message that this commit reduces test coverage a little bit by not checking some flags that are not relevant to the tests.

    Same comment applies to other tests below like linked_list_random_deletion


    l0rinc commented at 1:17 pm on October 31, 2024:
    Hmm, so always check both? Ok, I don’t mind, done
  60. in src/test/coinscachepair_tests.cpp:66 in 8df16f23a2 outdated
    62@@ -63,7 +63,7 @@ BOOST_AUTO_TEST_CASE(linked_list_iteration)
    63 
    64     // Delete the nodes from the list to make sure there are no dangling pointers
    65     for (auto it{nodes.begin()}; it != nodes.end(); it = nodes.erase(it)) {
    66-        BOOST_CHECK_EQUAL(it->second.GetFlags(), 0);
    67+        BOOST_CHECK(!it->second.IsDirty() && !it->second.IsFresh());
    


    ryanofsky commented at 7:15 pm on October 30, 2024:

    In commit “coins: Remove direct GetFlags access” (8df16f23a22b2e0b72dd6338ed37c09485f5ede3)

    Not important, but usually test code and output is more readable with CHECK(a); CHECK(b); instead of CHECK(a && b)


    l0rinc commented at 1:17 pm on October 31, 2024:
    Yeah, but I don’t like the format of that. If this fails, it’s easy to debug.
  61. in src/test/coins_tests.cpp:577 in e8e70a4ae3 outdated
    572+
    573+    bool operator==(const CoinEntry& o) const = default;
    574+    friend std::ostream& operator<<(std::ostream& os, const CoinEntry& e) { return os << e.value << e.flags; }
    575+
    576+    constexpr bool IsDirty() const { return flags & DIRTY; }
    577+    constexpr bool IsFresh() const { return flags & FRESH; }
    


    ryanofsky commented at 7:31 pm on October 30, 2024:

    In commit “test: Migrate GetCoinsMapEntry to return MaybeCoin” (e8e70a4ae3cbd7a648629c2df4b5a7f6f0686438)

    It seems like these methods are not called in this commit, but they could be called in InsertCoinsMapEntry below. Maybe call them there or drop them here?


    l0rinc commented at 1:17 pm on October 31, 2024:
    Done
  62. in src/test/coins_tests.cpp:687 in 553db5dec1 outdated
    716-    CheckAccessCoin(VALUE1, SPENT , SPENT , DIRTY|FRESH, DIRTY|FRESH);
    717-    CheckAccessCoin(VALUE1, VALUE2, VALUE2, 0          , 0          );
    718-    CheckAccessCoin(VALUE1, VALUE2, VALUE2, FRESH      , FRESH      );
    719-    CheckAccessCoin(VALUE1, VALUE2, VALUE2, DIRTY      , DIRTY      );
    720-    CheckAccessCoin(VALUE1, VALUE2, VALUE2, DIRTY|FRESH, DIRTY|FRESH);
    721+    CheckAccessCoin(ABSENT, MISSING,            MISSING           );
    


    ryanofsky commented at 9:45 pm on October 30, 2024:

    In commit “test: Group values and states in tests into CoinEntry wrappers” (553db5dec1334ed5e8e07e34306f16de9afc22bb)

    Review note: It’s not really easy to verify by looking at code that new and old test cases are equivalent, but an approach that worked for me was to add extra prints showing the test cases, and then diff the output from test_bitcoin -t coins_tests before and after this commit, and confirm there are were no changes. My change used to test this was:

     0--- a/src/test/coins_tests.cpp
     1+++ b/src/test/coins_tests.cpp
     2@@ -286,6 +286,7 @@ void SimulationTest(CCoinsView* base, bool fake_best_block)
     3 // Run the above simulation for multiple base types.
     4 BOOST_FIXTURE_TEST_CASE(coins_cache_simulation_test, CacheTest)
     5 {
     6+    if ((1)) return;
     7     CCoinsViewTest base{m_rng};
     8     SimulationTest(&base, false);
     9 
    10@@ -318,6 +319,7 @@ UtxoData::iterator FindRandomFrom(const std::set<COutPoint> &utxoSet) {
    11 // has the expected effect (the other duplicate is overwritten at all cache levels)
    12 BOOST_FIXTURE_TEST_CASE(updatecoins_simulation_test, UpdateTest)
    13 {
    14+    if ((1)) return;
    15     SeedRandomForTest(SeedRand::ZEROS);
    16 
    17     bool spent_a_duplicate_coinbase = false;
    18@@ -572,7 +574,7 @@ struct CoinEntry {
    19     CoinEntry(CAmount v, char s) : value(v), flags(s) {}
    20 
    21     bool operator==(const CoinEntry& o) const = default;
    22-    friend std::ostream& operator<<(std::ostream& os, const CoinEntry& e) { return os << e.value << e.flags; }
    23+    friend std::ostream& operator<<(std::ostream& os, const CoinEntry& e) { return os << e.value << " " << int{e.flags}; }
    24 
    25     constexpr bool IsDirty() const { return flags & DIRTY; }
    26     constexpr bool IsFresh() const { return flags & FRESH; }
    27@@ -671,6 +673,7 @@ public:
    28 
    29 static void CheckAccessCoin(CAmount base_value, const CoinEntry& cache_coin, const CoinEntry& expected)
    30 {
    31+    tfm::format(std::cout, "CheckAccessCoin(%s, %s, %s)\n", base_value, cache_coin, expected);
    32     SingleEntryCacheTest test(base_value, cache_coin);
    33     test.cache.AccessCoin(OUTPOINT);
    34     test.cache.SelfTest(/*sanity_check=*/false);
    35@@ -717,6 +720,7 @@ BOOST_AUTO_TEST_CASE(ccoins_access)
    36 
    37 static void CheckSpendCoins(CAmount base_value, const CoinEntry& cache_coin, const CoinEntry& expected)
    38 {
    39+    tfm::format(std::cout, "CheckSpendCoins(%s, %s, %s)\n", base_value, cache_coin, expected);
    40     SingleEntryCacheTest test(base_value, cache_coin);
    41     test.cache.SpendCoin(OUTPOINT);
    42     test.cache.SelfTest();
    43@@ -763,6 +767,7 @@ BOOST_AUTO_TEST_CASE(ccoins_spend)
    44 
    45 static void CheckAddCoin(CAmount base_value, CAmount modify_value, const CoinEntry& cache_coin, const CoinEntry& expected, bool coinbase)
    46 {
    47+    tfm::format(std::cout, "CheckAddCoin(%s, %s, %s, %s, %s)\n", base_value, modify_value, cache_coin, expected, coinbase);
    48     try {
    49         SingleEntryCacheTest test(base_value, cache_coin);
    50         CTxOut output;
    51@@ -809,6 +814,7 @@ BOOST_AUTO_TEST_CASE(ccoins_add)
    52 
    53 void CheckWriteCoins(const CoinEntry& parent, const CoinEntry& child, const CoinEntry& expected)
    54 {
    55+    tfm::format(std::cout, "CheckWriteCoins(%s, %s, %s)\n", parent, child, expected);
    56     try {
    57         SingleEntryCacheTest test(ABSENT, parent);
    58         WriteCoinsViewEntry(test.cache, child);
    

    l0rinc commented at 1:17 pm on October 31, 2024:
    Thank you for checking! Do you need any change here from me?

    ryanofsky commented at 2:21 pm on November 5, 2024:
    Thanks, no this is just a note to help myself and other reviewers.
  63. in src/test/coins_tests.cpp:756 in 553db5dec1 outdated
    820-    CheckAddCoin(VALUE2, VALUE3, VALUE3, FRESH      , DIRTY|FRESH, true );
    821-    CheckAddCoin(VALUE2, VALUE3, FAIL  , DIRTY      , NO_ENTRY   , false);
    822-    CheckAddCoin(VALUE2, VALUE3, VALUE3, DIRTY      , DIRTY      , true );
    823-    CheckAddCoin(VALUE2, VALUE3, FAIL  , DIRTY|FRESH, NO_ENTRY   , false);
    824-    CheckAddCoin(VALUE2, VALUE3, VALUE3, DIRTY|FRESH, DIRTY|FRESH, true );
    825+    for (auto base_value : {ABSENT, SPENT, VALUE1}) {
    


    ryanofsky commented at 10:04 pm on October 30, 2024:

    In commit “test: Group values and states in tests into CoinEntry wrappers” (553db5dec1334ed5e8e07e34306f16de9afc22bb)

    Note: because this for loop is moving from CheckAddCoin function to the top-level test code, this commit changes order of the tests. But tests that are performed should still be the same. It would be good to note in commit message that this commit does affect test order of this test but doesn’t change its coverage.


    l0rinc commented at 1:17 pm on October 31, 2024:
    Done, thanks
  64. in src/test/coins_tests.cpp:883 in 553db5dec1 outdated
    936-    for (const CAmount parent_value : {ABSENT, SPENT, VALUE1})
    937-        for (const CAmount child_value : {ABSENT, SPENT, VALUE2})
    938-            for (const char parent_flags : parent_value == ABSENT ? ABSENT_FLAGS : FLAGS)
    939-                for (const char child_flags : child_value == ABSENT ? ABSENT_FLAGS : CLEAN_FLAGS)
    940-                    CheckWriteCoins(parent_value, child_value, parent_value, parent_flags, child_flags, parent_flags);
    941+    for (const CoinEntry parent : {MISSING,
    


    ryanofsky commented at 10:08 pm on October 30, 2024:

    In commit “test: Group values and states in tests into CoinEntry wrappers” (553db5dec1334ed5e8e07e34306f16de9afc22bb)

    Note: Again in this part of the test, the order cases are run in seems to be changing, but the same set of cases is run. Would be helpful to mention this in commit message.


    l0rinc commented at 1:17 pm on October 31, 2024:
    Done in previous step
  65. ryanofsky approved
  66. ryanofsky commented at 10:42 pm on October 30, 2024: contributor
    Code review ACK 3b5b2457f713014fd5073da34b76a5a8cd796e4a. Nice change that should make the API more safe and also makes tests more complete and easier to maintain.
  67. DrahtBot requested review from andrewtoth on Oct 30, 2024
  68. coins, refactor: Split up AddFlags to remove invalid states
    CCoinsCacheEntry provided general access to its internal flags state, even though in reality it could only be clean, fresh, dirty or fresh|dirty.
    
    After it got dirtied we couldn't set the state back to clean by AddFlags(0) - tests were explicitly checking against that.
    
    This commit cleans up the public interface to make this distinction cleaner and invalid behavior impossible instead of just checked by tests.
    
    Also modernized `EmplaceCoinInternalDANGER` since it was already modified.
    
    Co-authored-by: Andrew Toth <andrewstoth@gmail.com>
    a6921049fc
  69. coins, refactor: Make AddFlags, SetDirty, SetFresh static
    This makes the `Assume(&self.second == this)` check redundant
    
    Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
    d5e3bbf440
  70. l0rinc force-pushed on Oct 31, 2024
  71. l0rinc commented at 1:17 pm on October 31, 2024: contributor

    Thanks a lot @ryanofsky for the review. Applied almost all of your insightful changes, adding two new commits (adding you as coauthor). You can view the diff with Compare or via git range-diff 8e1381a..3b5b2457 a6921049..aa2f3139

    Added the following commits based on feedback:

    • coins, refactor: Assume state after SetClean in AddFlags to prevent dangling pointers
    • coins, refactor: Make AddFlags, SetDirty, SetFresh static

    Updated commit messages to warn about test order changes and method move reasons. Separated method move from method changes and other minor moves between commits for clarity. Asserted both fresh and dirty in tests where flags were changed originally.

  72. in src/coins.h:181 in a6921049fc outdated
    179-    inline void ClearFlags() noexcept
    180+    inline void SetDirty(CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
    181+    {
    182+        AddFlags(DIRTY, self, sentinel);
    183+    }
    184+    inline void SetFresh(CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
    


    ryanofsky commented at 1:58 pm on November 5, 2024:

    In commit “coins, refactor: Split up AddFlags to remove invalid states” (a6921049fcc3de4067dc3f88fef57884450d25a1)

    Not important, but instead of SetFresh and SetClean() are being introduced as non-static methods in this commit and then changing them to static in the next commit, wouldn’t it make more sense to just introduce them as static methods in this commit? That should reduce churn and make the next commit simpler, because only AddFlags will be changing.


    l0rinc commented at 11:59 am on November 10, 2024:
    Wouldn’t I need to make AddFlags static as well in that case? Which would trigger this removal and rewrite of the whole AddFlags method which is done in the next commit. Or you mean squash the first two commits?

    ryanofsky commented at 11:14 am on November 12, 2024:

    re: #30906 (review)

    Wouldn’t I need to make AddFlags static as well in that case?

    No that should not be necessary, and wouldn’t suggest that. The suggestion is to make the new SetFresh and SetClean methods static and have them call self.second.AddFlags instead of AddFlags in this commit so all the new calls to these methods don’t need to change again in commit d5e3bbf440aa948df8159999fd4eb5275c354b93.

  73. in src/coins.h:169 in 2d691b50d3 outdated
    165@@ -166,6 +166,7 @@ struct CCoinsCacheEntry
    166     {
    167         Assume(flags & (DIRTY | FRESH));
    168         if (!self.second.m_flags) {
    169+            Assume(self.second.m_prev == nullptr && self.second.m_next == nullptr);
    


    ryanofsky commented at 2:07 pm on November 5, 2024:

    In commit “coins, refactor: Assume state after SetClean in AddFlags to prevent dangling pointers” (2d691b50d3729445a60898981b78c4239f2f0dd7)

    I think this could also check the converse, that pointers are valid if flags were set. It could simply check that pointers are non-null if flags are set, or go further and check prev->next == self, and next->prev == self


    l0rinc commented at 12:24 pm on November 10, 2024:

    check that pointers are non-null if flags are set, or go further and check prev->next == self, and next->prev == self

    the latter seems redundant, but the first is simple to check - done

  74. in src/test/coins_tests.cpp:563 in a902af6fb1 outdated
    559@@ -558,16 +560,29 @@ BOOST_AUTO_TEST_CASE(ccoins_serialization)
    560     }
    561 }
    562 
    563+const static char DIRTY = CCoinsCacheEntry::DIRTY;
    564+const static char FRESH = CCoinsCacheEntry::FRESH;
    565+const static char NO_ENTRY = -1;
    566+
    567+struct CoinEntry {
    


    ryanofsky commented at 2:16 pm on November 5, 2024:

    In commit “test: Migrate GetCoinsMapEntry to return MaybeCoin” (a902af6fb195fae97a89876b88d5034162ab8366)

    May want to add “refactor” to commit subject or otherwise mention that this commit is not adding or removing any test coverage. Otherwise it’s hard to know looking at the commit what it’s intended to do.

    Same comment also applies to commit “test: Remove remaining unbounded flags from coins_tests” (e4db49949c55470950a206172ee53f6baacf3767) and commit “test: Compact ccoins_access and ccoins_spend” (aa2f3139529c054b011a0f75ff314e6d63f0d977)


    l0rinc commented at 12:24 pm on November 10, 2024:
    Done
  75. in src/test/coins_tests.cpp:574 in a902af6fb1 outdated
    569+    const char flags;
    570+
    571+    CoinEntry(CAmount v, char s) : value(v), flags(s) {}
    572+
    573+    bool operator==(const CoinEntry& o) const = default;
    574+    friend std::ostream& operator<<(std::ostream& os, const CoinEntry& e) { return os << e.value << e.flags; }
    


    ryanofsky commented at 2:18 pm on November 5, 2024:

    In commit “test: Migrate GetCoinsMapEntry to return MaybeCoin” (a902af6fb195fae97a89876b88d5034162ab8366)

    Should this include a space between the value and flags? Otherwise if test fails it looks like amount and flags will run together and might not be possible to distinguish.


    l0rinc commented at 12:24 pm on November 10, 2024:
    Done
  76. ryanofsky approved
  77. ryanofsky commented at 2:29 pm on November 5, 2024: contributor
    Code review ACK aa2f3139529c054b011a0f75ff314e6d63f0d977. Thanks for the updates! Left a few more comments that are not important and you can feel free to ignore.
  78. in src/coins.h:163 in d5e3bbf440 outdated
    161@@ -162,26 +162,19 @@ struct CCoinsCacheEntry
    162     //! Adding a flag also requires a self reference to the pair that contains
    163     //! this entry in the CCoinsCache map and a reference to the sentinel of the
    


    andrewtoth commented at 4:31 pm on November 6, 2024:

    nit: this comment is no longer accurate after d5e3bbf440aa948df8159999fd4eb5275c354b93

    We should just remove the part about a self reference, and only mention the sentinel.


    l0rinc commented at 12:23 pm on November 10, 2024:
    Done
  79. in src/coins.h:165 in d5e3bbf440 outdated
    161@@ -162,26 +162,19 @@ struct CCoinsCacheEntry
    162     //! Adding a flag also requires a self reference to the pair that contains
    163     //! this entry in the CCoinsCache map and a reference to the sentinel of the
    164     //! flagged pair linked list.
    165-    inline void AddFlags(uint8_t flags, CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
    166+    static inline void AddFlags(uint8_t flags, CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
    


    andrewtoth commented at 4:31 pm on November 6, 2024:

    I think self is no longer a good name for this parameter if we make this static. It should just be pair.

    Same for SetDirty and SetFresh.


    l0rinc commented at 12:23 pm on November 10, 2024:
    Done

    ryanofsky commented at 11:27 am on November 12, 2024:

    In commit “coins, refactor: Make AddFlags, SetDirty, SetFresh static” (d5e3bbf440aa948df8159999fd4eb5275c354b93)

    It would make sense to rename self to pair this commit instead of in later commit afb0a3c88691cadc07fc985483f4cae1f486f6be so this code does not have to change twice and so later the change in the later commit can be move-only. Right now the later commit is tedious to review (or at least I don’t know an easy way to review it) because the argument is renamed at the same time as the function is moved.

  80. in src/test/coinscachepair_tests.cpp:173 in f92268456a outdated
    176     BOOST_CHECK_EQUAL(n1.second.Prev(), &sentinel);
    177     BOOST_CHECK_EQUAL(sentinel.second.Next(), &n1);
    178     BOOST_CHECK_EQUAL(n2.second.Prev(), &n1);
    179 
    180-    // Check that we can clear flags then re-add them
    181+    // Check that we can clear state then re-set them
    


    andrewtoth commented at 4:36 pm on November 6, 2024:
    nit: // Check that we can clear state then re-set it

    l0rinc commented at 12:23 pm on November 10, 2024:
    Done
  81. andrewtoth approved
  82. andrewtoth commented at 4:44 pm on November 6, 2024: contributor

    re-crACK aa2f3139529c054b011a0f75ff314e6d63f0d977

    It might be worth updating the AddFlags comment now, but otherwise non-blocking nits.

  83. coins, refactor: Assume state after SetClean in AddFlags to prevent dangling pointers
    Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
    9d61b956d9
  84. coins, refactor: Remove direct GetFlags access
    We don't need so much access to the internals of CCoinsCacheEntry, since many tests are just exercising invalid combinations this way.
    This implies that `AddFlags` has private access now.
    afb0a3c886
  85. test, refactor: Migrate GetCoinsMapEntry to return MaybeCoin 7b3cfcf601
  86. test: Group values and states in tests into CoinEntry wrappers
    Note: that this commit affects the test order, but doesn't change its behavior or coverage otherwise.
    c4eb31be3e
  87. test: Validate error messages on fail
    The `ccoins_add` and `ccoins_write` tests check the actual exception error messages now instead of just that they fail for the given parameters.
    This enables us testing different exceptions in a more fine-grained way in later changes.
    236bbfad67
  88. test, refactor: Remove remaining unbounded flags from coins_tests 141cec4d88
  89. test, refactor: Compact ccoins_access and ccoins_spend 9edbe5a48f
  90. l0rinc force-pushed on Nov 10, 2024
  91. l0rinc commented at 12:24 pm on November 10, 2024: contributor
    Thanks @ryanofsky & @andrewtoth, applied most nits, please re-review: git range-diff a6921049..aa2f3139 a6921049..9edbe5a4 or https://github.com/bitcoin/bitcoin/compare/aa2f3139529c054b011a0f75ff314e6d63f0d977..9edbe5a48fe9a2c0d364882fcb7d44ae6aac6d24
  92. andrewtoth commented at 3:49 pm on November 11, 2024: contributor
    re-crACK 9edbe5a48fe9a2c0d364882fcb7d44ae6aac6d24
  93. DrahtBot requested review from ryanofsky on Nov 11, 2024
  94. in src/coins.h:204 in 9edbe5a48f
    200@@ -197,11 +201,11 @@ struct CCoinsCacheEntry
    201     }
    202 
    203     //! Only use this for initializing the linked list sentinel
    204-    inline void SelfRef(CoinsCachePair& self) noexcept
    205+    inline void SelfRef(CoinsCachePair& pair) noexcept
    


    andrewtoth commented at 3:50 pm on November 11, 2024:
    minor nit: I don’t think this one needed to be renamed to pair. self still makes sense here.

    l0rinc commented at 3:52 pm on November 11, 2024:
    yeah, but it’s more consistent this way (and method name already mentions self)
  95. ryanofsky approved
  96. ryanofsky commented at 11:34 am on November 12, 2024: contributor

    Code review ACK 9edbe5a48fe9a2c0d364882fcb7d44ae6aac6d24. Only small changes since last review like adding an Assume, changing commit messages, renaming self to pair. Thanks for the updates!

    I left two suggestions which are not important and don’t affect the final code, just meant to make review a little easier for the next reviewer. Feel free to ignore them or save for a later update.


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-11-21 09:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me