refactor: prohibit direct flags access in CCoinsCacheEntry and remove invalid tests #30906

pull l0rinc wants to merge 7 commits into bitcoin:master from l0rinc:l0rinc/hide-coin-flags changing 5 files +288 −353
  1. l0rinc commented at 1:39 pm on September 15, 2024: contributor

    Similarly to #30849, this cleanup is intended to de-risk #30673 (review) by simplifying the coin cache public interface.

    CCoinsCacheEntry provided general access to its internal flags state, even though, in reality, it could only be clean, fresh, dirty, or fresh|dirty (in the follow-up, we will remove fresh without dirty).

    Once it was marked as dirty, we couldn’t set the state back to clean with AddFlags(0)—tests explicitly checked against that.

    This PR refines the public interface to make this distinction clearer and to make invalid behavior impossible, rather than just checked by tests. We don’t need extensive access to the internals of CCoinsCacheEntry, as many tests were simply validating invalid combinations in this way.

    The last few commits contain significant test refactorings to make coins_tests easier to change in follow-ups.

  2. DrahtBot commented at 1:39 pm on September 15, 2024: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage

    For detailed information about the code coverage, see the test coverage report.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK ryanofsky
    Concept ACK davidgumberg, hodlinator
    Stale ACK andrewtoth

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #31132 (validation: fetch block inputs on parallel threads ~17% faster IBD by andrewtoth)
    • #30643 (coins: Add move operations to Coin and CCoinsCacheEntry by l0rinc)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  3. DrahtBot added the label UTXO Db and Indexes on Sep 15, 2024
  4. l0rinc renamed this:
    coins: prohibit direct flags access in CCoinsCacheEntry and remove invalid tests
    refactor: prohibit direct flags access in CCoinsCacheEntry and remove invalid tests
    on Sep 15, 2024
  5. in src/test/coins_tests.cpp:623 in 03cf2bd3e1 outdated
    616@@ -616,7 +617,9 @@ void GetCoinsMapEntry(const CCoinsMap& map, CAmount& value, char& flags, const C
    617         } else {
    618             value = it->second.coin.out.nValue;
    619         }
    620-        flags = it->second.GetFlags();
    621+        flags = 0;
    622+        if (it->second.IsDirty()) flags |= DIRTY;
    623+        if (it->second.IsFresh()) flags |= FRESH;
    624         assert(flags != NO_ENTRY);
    


    andrewtoth commented at 1:52 pm on September 15, 2024:
    This assertion is obviated now.

    l0rinc commented at 6:35 pm on September 15, 2024:
    Replaced it with a flag validation
  6. in src/test/coins_tests.cpp:710 in 03cf2bd3e1 outdated
    710@@ -708,7 +711,7 @@ static void CheckSpendCoins(CAmount base_value, CAmount cache_value, CAmount exp
    711     GetCoinsMapEntry(test.cache.map(), result_value, result_flags);
    712     BOOST_CHECK_EQUAL(result_value, expected_value);
    713     BOOST_CHECK_EQUAL(result_flags, expected_flags);
    714-};
    715+}
    


    andrewtoth commented at 1:54 pm on September 15, 2024:
    Why was this changed?

    l0rinc commented at 4:01 pm on September 15, 2024:
    Because class declarations don’t need semicolons, but I’ll revert if you find it distracting
  7. in src/test/coinscachepair_tests.cpp:188 in 03cf2bd3e1 outdated
    213     BOOST_CHECK_EQUAL(sentinel.second.Next(), &n2);
    214     BOOST_CHECK_EQUAL(sentinel.second.Prev(), &n2);
    215     BOOST_CHECK_EQUAL(n2.second.Next(), &sentinel);
    216     BOOST_CHECK_EQUAL(n2.second.Prev(), &sentinel);
    217 
    218     // But adding DIRTY re-inserts it after n2
    


    andrewtoth commented at 1:57 pm on September 15, 2024:
    But can be removed now.

    l0rinc commented at 6:36 pm on September 15, 2024:
    Done

    ryanofsky commented at 6:58 pm on October 30, 2024:

    re: #30906 (review)

    In commit “coins: Split up AddFlags to remove invalid states” (8e1381a991e325395fbc10df2fb091c508efa0f3)

    Done

    Seems like this is not actually done as of 8e1381a991e325395fbc10df2fb091c508efa0f3

  8. andrewtoth commented at 1:58 pm on September 15, 2024: contributor

    Concept ACK

    Can we remove any other tests that are trying to add flags that are not FRESH or DIRTY and so are now useless?

  9. l0rinc force-pushed on Sep 15, 2024
  10. l0rinc commented at 6:39 pm on September 15, 2024: contributor

    Can we remove any other tests that are trying to add flags that are not FRESH or DIRTY and so are now useless?

    I’ve merged NO_ENTRY and char(0) values - this simplifies the cases somewhat. Do you think we can remove any of the lines? There aren’t any duplicates, even after the above change…

    Edit: moved the booleans closer to the edges in later commits.

  11. l0rinc force-pushed on Sep 15, 2024
  12. l0rinc force-pushed on Sep 15, 2024
  13. DrahtBot added the label CI failed on Sep 15, 2024
  14. DrahtBot removed the label CI failed on Sep 15, 2024
  15. in src/test/coins_tests.cpp:570 in 390c670c37 outdated
    570@@ -571,11 +571,19 @@ const static CAmount VALUE2 = 200;
    571 const static CAmount VALUE3 = 300;
    572 const static char DIRTY = CCoinsCacheEntry::DIRTY;
    573 const static char FRESH = CCoinsCacheEntry::FRESH;
    574-const static char NO_ENTRY = -1;
    


    andrewtoth commented at 9:47 pm on September 15, 2024:

    Since we don’t pass a bitmap as flags anymore, does it make sense to get rid of the char type for flags in coins_tests and replace with a more descriptive enum?

    0enum Flags
    1{
    2   Clean,
    3   Dirty,
    4   Fresh,
    5   DirtyAndFresh
    6};
    

    Could also be a std::optional<Flags> so we can handle the NO_ENTRY case cleanly instead of with a -1.


    l0rinc commented at 10:53 pm on September 15, 2024:

    what’s the difference between clean and NO_ENTRY? Does that difference still make sense after this change?

    get rid of the char type for flags in coins_tests and replace with a more descriptive enum

    Since we don’t have flags in the production code anymore I would prefer getting rid of them in the tests as well. Reintroducing an enum would kinda’ defeat that purpose in my opinion.

    To make sure the tests are still checking the same combinations, I have added a few scripted diffs to transition the tests away from the flags - now the tabular data displays dirty and fresh booleans instead (I guess we can group some of the examples into CoinMapEntry as well, if needed). This should help us selectively decide which combinations are still valid. What do you think?


    andrewtoth commented at 10:59 pm on September 15, 2024:
    Clean means not dirty and not fresh. NO_ENTRY is a special value that is not a possible flag but signals to the tests that that entry does not exist. With proper types, that would correspond to a std::nullopt. We don’t have to name the enum Flags, we can name it EntryState. That way we don’t have to validate the flags variable to see if it’s one of the values we will accept. That is better served by using an enum and std::optional.

    andrewtoth commented at 11:03 pm on September 15, 2024:

    I don’t think replacing with true and false is the right approach. Using enums, tests would be like

    0CheckAccessCoin(VALUE1, SPENT , SPENT , DIRTY|FRESH, DIRTY|FRESH);
    1becomes
    2CheckAccessCoin(VALUE1, SPENT , SPENT , DirtyAndFresh, DirtyAndFresh);
    3
    4CheckSpendCoins(ABSENT, SPENT , ABSENT, FRESH      , NO_ENTRY    );
    5becomes
    6CheckSpendCoins(ABSENT, SPENT , ABSENT, Fresh      , std::nullopt);
    

    l0rinc commented at 11:03 pm on September 15, 2024:

    is a special value that is not a possible flag but signals to the tests that that entry does not exist

    That is better served by using an enum and std::optional.

    Thanks, I’ll investigate tomorrow. Please leave some comments about the overall direction as well, do you think this new test layout is more readable or less? Drafting until then.


    l0rinc commented at 4:04 pm on September 17, 2024:
    Did a significant restructure, the tabular tests should be a lot easier to comprehend and modify now. Also testing the exception messages directly, you’ll need this in your new PR. I still need to get rid of remaining flags in the tests, will do that tomorrow - any early feedback until then is appreciated.

    l0rinc commented at 6:43 am on September 18, 2024:
    Done
  16. l0rinc marked this as a draft on Sep 15, 2024
  17. l0rinc force-pushed on Sep 17, 2024
  18. in src/test/coins_tests.cpp:569 in ba52ebd416 outdated
    563@@ -562,6 +564,15 @@ BOOST_AUTO_TEST_CASE(ccoins_serialization)
    564     }
    565 }
    566 
    567+struct CoinStruct {
    568+    const CAmount value;
    569+    const char flags;
    


    andrewtoth commented at 4:29 pm on September 17, 2024:
    I still think this should be an enum named EntryState or something, with all combinations of flags like https://github.com/bitcoin/bitcoin/pull/30906/commits/ba52ebd41673abf63d4495215c84be7244469d30#r1760349038. That’s the proper way to do this instead of defining static vars for each value of a char we care about.

    l0rinc commented at 4:32 pm on September 17, 2024:
    Absolutely, that’s my plan for tomorrow - but needed to clean up the test before doing that

    l0rinc commented at 6:52 am on September 18, 2024:
    Done
  19. in src/test/coins_tests.cpp:702 in c4bf4d76fe outdated
    730+    CheckAccessCoin(VALUE1, {SPENT , DIRTY      }, {SPENT , DIRTY      });
    731+    CheckAccessCoin(VALUE1, {SPENT , DIRTY|FRESH}, {SPENT , DIRTY|FRESH});
    732+    CheckAccessCoin(VALUE1, {VALUE2, UNCHANGED  }, {VALUE2, UNCHANGED  });
    733+    CheckAccessCoin(VALUE1, {VALUE2, FRESH      }, {VALUE2, FRESH      });
    734+    CheckAccessCoin(VALUE1, {VALUE2, DIRTY      }, {VALUE2, DIRTY      });
    735+    CheckAccessCoin(VALUE1, {VALUE2, DIRTY|FRESH}, {VALUE2, DIRTY|FRESH});
    


    davidgumberg commented at 6:45 pm on September 17, 2024:
    Nit: Could be made into a loop as below

    l0rinc commented at 7:01 am on September 18, 2024:
    Done, thanks!
  20. davidgumberg commented at 6:55 pm on September 17, 2024: contributor
    Concept ACK on dropping the bitfield interface for CCoinsCacheEntry flags
  21. l0rinc force-pushed on Sep 18, 2024
  22. l0rinc force-pushed on Sep 18, 2024
  23. l0rinc marked this as ready for review on Sep 18, 2024
  24. in src/coins.h:136 in b2f9df7817 outdated
    127@@ -128,6 +128,22 @@ struct CCoinsCacheEntry
    128     CoinsCachePair* m_next{nullptr};
    129     uint8_t m_flags{0};
    130 
    131+    //! Adding a flag also requires a self reference to the pair that contains
    132+    //! this entry in the CCoinsCache map and a reference to the sentinel of the
    133+    //! flagged pair linked list.
    134+    inline void AddFlags(uint8_t flags, CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
    135+    {
    136+        assert(flags & DIRTY || flags & FRESH);
    


    hodlinator commented at 9:27 am on September 18, 2024:

    nit: terseness++

    0        assert(flags & (DIRTY | FRESH));
    

    l0rinc commented at 8:14 am on September 19, 2024:
    Clever, thanks!
  25. in src/coins.h:186 in b2f9df7817 outdated
    193+    }
    194+    inline void SetFresh(CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
    195+    {
    196+        AddFlags(FRESH, self, sentinel);
    197     }
    198     inline void ClearFlags() noexcept
    


    andrewtoth commented at 1:11 pm on September 18, 2024:
    0    inline void SetClean() noexcept
    


    hodlinator commented at 1:57 pm on September 18, 2024:

    While “clean” is the opposite of “dirty”, it doesn’t involve how fresh something is. It’s on another dimension.

    I actually prefer ClearFlags(), ClearState() or possibly Reset-something().

    Same goes for Caching::CLEAN. Would prefer VOID/EMPTY/INIT/NULL.


    andrewtoth commented at 10:27 pm on September 18, 2024:

    fresh is actually a substate of dirty. Being fresh implies dirty, so cleaning implies no longer fresh.

    See #30673 which inspired this, which intends to remove possibility of FRESH-but-not-DIRTY entries.


    l0rinc commented at 7:13 am on September 19, 2024:

    FRESH is definitely a very confusing concept

    the parent cache does not have this coin or that it is a spent coin in the parent cache

    Maybe after #30673 we can find a better name (and description) for it


    hodlinator commented at 7:39 pm on September 19, 2024:
    Okay, so FRESH and DIRTY_FRESH will merge into one. :+1:

    l0rinc commented at 7:49 pm on September 19, 2024:
    rather FRESH will be removed in the mentioned follow-up, since it’s not actually possible (unless in case of reorgs, which I don’t fully understand yet, but not important for this PR)
  26. in src/test/coins_tests.cpp:651 in b2f9df7817 outdated
    670-        flags = it->second.GetFlags();
    671-        assert(flags != NO_ENTRY);
    672+    if (auto it{map.find(outp)}; it != map.end()) {
    673+        return CoinStruct{
    674+            it->second.coin.IsSpent() ? SPENT : it->second.coin.out.nValue,
    675+            ToCaching(it->second.IsDirty(), it->second.IsFresh())};
    


    andrewtoth commented at 1:16 pm on September 18, 2024:
    Do we need a new function if we’re only calling it the one time here? Just seems like unnecessary indirection.

    l0rinc commented at 1:36 pm on September 18, 2024:
    Converting flags to enums isn’t the responsibility of this method, that’s why I’ve extracted it.
  27. l0rinc force-pushed on Sep 18, 2024
  28. l0rinc force-pushed on Sep 18, 2024
  29. in src/test/coins_tests.cpp:576 in 5d6937c7ce outdated
    571+    DIRTY_FRESH,
    572+};
    573+
    574+struct CoinStruct {
    575+    const CAmount value;
    576+    const Caching caching;
    


    hodlinator commented at 9:52 pm on September 18, 2024:

    Types should for the most part be nouns IMO. Seldom verbs in present participle like Caching. CoinStruct feels a bit WIP too.

    Suggestions:

    0struct CoinState {
    1    const CAmount value;
    2    const CacheState cache;
    
    0struct CoinCache {
    1    enum State {
    2        CLEAN,
    3        DIRTY,
    4        FRESH,
    5        DIRTY_FRESH,
    6    };
    7    const CAmount value;
    8    const State state;
    

    l0rinc commented at 8:13 am on September 19, 2024:
    Had a few versions before, played around with it a bit more, thanks for the pointers, what do you think of the new structure and names.

    hodlinator commented at 7:45 pm on September 19, 2024:
    Like it! Moves related things closer together.

    l0rinc commented at 7:50 pm on September 19, 2024:
    The usages are a lot more verbose, but I’ve extracted everything important now, so the tables are still readable
  30. hodlinator commented at 10:03 pm on September 18, 2024: contributor

    Concept ACK.

    Really like the move to put restrictions on how flags are used.

  31. l0rinc force-pushed on Sep 19, 2024
  32. in src/coins.h:136 in a595328a4a outdated
    127@@ -128,6 +128,22 @@ struct CCoinsCacheEntry
    128     CoinsCachePair* m_next{nullptr};
    129     uint8_t m_flags{0};
    130 
    131+    //! Adding a flag also requires a self reference to the pair that contains
    132+    //! this entry in the CCoinsCache map and a reference to the sentinel of the
    133+    //! flagged pair linked list.
    134+    inline void AddFlags(uint8_t flags, CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
    135+    {
    136+        assert(flags & (DIRTY | FRESH));
    


    andrewtoth commented at 1:52 am on September 20, 2024:
    I think this should be Assume. We should prefer it since it will be compiled out for release builds and this is a hot path.

    l0rinc commented at 3:50 pm on September 21, 2024:
    This was meant to be a temporary transition, since we’re removing the rest of the flags in your PR - but I’m fine with Assume as well - changed.
  33. in src/coins.h:164 in a595328a4a outdated
    176+        SetClean();
    177     }
    178 
    179-    //! Adding a flag also requires a self reference to the pair that contains
    180-    //! this entry in the CCoinsCache map and a reference to the sentinel of the
    181-    //! flagged pair linked list.
    


    andrewtoth commented at 1:56 am on September 20, 2024:
    Hmm not sure we should remove the extra context here about what exactly self and sentinel are and where they come from.

    l0rinc commented at 3:50 pm on September 21, 2024:

    This was moved to the AddFlags already - even though it seems to me the code already makes all of that clear:

    Adding a flag also requires a self reference to the pair that contains this entry in the CCoinsCache map

    AddFlags(uint8_t flags, CoinsCachePair& self

    and a reference to the sentinel of the flagged pair linked list

    , CoinsCachePair& sentinel


    andrewtoth commented at 4:22 pm on September 22, 2024:

    Ah yes I see it was moved up with AddFlags to private. Maybe better to keep it on SetDirty.

    Adding a flag also requires a self reference to the pair that contains this entry in the CCoinsCache map

    There’s no context in the method signature that it is an entry in the CCoinsCache map.

    and a reference to the sentinel of the flagged pair linked list

    There’s no context in the method signature about a flagged linked list.


    l0rinc commented at 4:46 pm on September 22, 2024:

    There’s no context in the method signature that it is an entry in the CCoinsCache map

    isn’t that what CoinsCachePair means?

    method signature about a flagged linked list

    The sentinel hints at it.

    But, again, I kept the methods for AddFlags - and don’t want to duplicate it for SetFresh and SetDirty.

  34. andrewtoth commented at 2:13 am on September 20, 2024: contributor

    I think the commit title test: Migrate GetCoinsMapEntry to return std::optional<CoinState> is no longer accurate.

    I didn’t find splitting the changes in coins_tests up into that many commits to be helpful for review. I ended up reviewing the combined diff for that file. Will review in more depth later.

  35. l0rinc force-pushed on Sep 21, 2024
  36. l0rinc force-pushed on Sep 21, 2024
  37. in src/test/coins_tests.cpp:573 in 1a89a97b1a outdated
    568+    enum class State { CLEAN, DIRTY, FRESH, DIRTY_FRESH };
    569+
    570+    const CAmount value;
    571+    const State state;
    572+
    573+    CoinEntry(CAmount v, State s) : value(v), state(s) {}
    


    andrewtoth commented at 4:31 pm on September 22, 2024:
    0    constexpr CoinEntry(CAmount v, State s) : value(v), state(s) {}
    

    then all const static MaybeCoin declarations below can be made constexpr.


    l0rinc commented at 11:41 am on September 23, 2024:
    Done, thanks
  38. in src/test/coins_tests.cpp:757 in 1a89a97b1a outdated
    912-    CheckAddCoin(VALUE2, VALUE3, FAIL  , DIRTY      , NO_ENTRY   , false);
    913-    CheckAddCoin(VALUE2, VALUE3, VALUE3, DIRTY      , DIRTY      , true );
    914-    CheckAddCoin(VALUE2, VALUE3, FAIL  , DIRTY|FRESH, NO_ENTRY   , false);
    915-    CheckAddCoin(VALUE2, VALUE3, VALUE3, DIRTY|FRESH, DIRTY|FRESH, true );
    916+    for (auto base_value : {ABSENT, SPENT, VALUE1}) {
    917+        CheckAddCoin(base_value, VALUE3, MISSING,            VALUE3_DIRTY_FRESH,   false);
    


    andrewtoth commented at 5:05 pm on September 22, 2024:
    Can we just hardcode value3 for write? It’s the same for every case here.

    l0rinc commented at 5:15 pm on September 22, 2024:
    Wouldn’t that be inconsistent? In every other case, the constant contained both values.

    andrewtoth commented at 5:22 pm on September 22, 2024:
    I mean, if you look at the Write column, each value is VALUE3.

    l0rinc commented at 11:16 am on September 23, 2024:

    I thought of that, but since it has to be different from base_value, seems weird to assume that in a different method instead of the call-site.

    0Base        Write   Cache        Expected      Coinbase
    1base_value, VALUE3, SPENT_DIRTY, VALUE3_DIRTY, false // VALUE3 + SPENT_DIRTY -> VALUE3_DIRTY
    

    makes more sense than

    0Base        Cache        Expected      Coinbase
    1base_value, SPENT_DIRTY, VALUE3_DIRTY, false // SPENT_DIRTY -> VALUE3_DIRTY???
    

    But I don’t mind renaming VALUE3 to VAL3 if you’re worried about verbosity.

  39. l0rinc force-pushed on Sep 23, 2024
  40. in src/test/coins_tests.cpp:616 in 1dd79a32c9 outdated
    624+constexpr MaybeCoin VALUE2_FRESH       = CoinEntry{VALUE2, CoinEntry::State::FRESH};
    625+constexpr MaybeCoin VALUE2_CLEAN       = CoinEntry{VALUE2, CoinEntry::State::CLEAN};
    626+constexpr MaybeCoin VALUE3_DIRTY       = CoinEntry{VALUE3, CoinEntry::State::DIRTY};
    627+constexpr MaybeCoin VALUE3_DIRTY_FRESH = CoinEntry{VALUE3, CoinEntry::State::DIRTY_FRESH};
    628+
    629+const static std::string EX_OVERWRITE_UNSPENT = "Attempted to overwrite an unspent coin (when possible_overwrite is false)";
    


    andrewtoth commented at 11:51 pm on September 27, 2024:
    Since #30921, we can now make these constexpr string_view.

    l0rinc commented at 6:33 am on September 28, 2024:
  41. l0rinc force-pushed on Sep 28, 2024
  42. andrewtoth commented at 0:13 am on September 29, 2024: contributor

    crACK cb9be6729c4e27bf2d6d703c0d454a22cbcdb6e1

    This simplifies the CCoinsCacheEntry interface, which was leaking the flags bitfield as an implementation detail. Now we don’t have to worry about fuzzing that, and it will make removing the FRESH-but-not-DIRTY or FRESH-and-spent cases easier in #30673.

    Also, the tests are much saner now and easier to parse by also removing the flags interface there.

    There are some minor issues with the commits though.

    Commit titles test: Migrate GetCoinsMapEntry to return std::optional<CoinState> and test: Group values and states in tests into CoinState wrappers should reference CoinEntry now.

    test: Validate error messages on fail should add some context in the commit message or description that it is for CCoinsViewCache AddCoin and WriteCoin unit tests.

  43. DrahtBot requested review from hodlinator on Sep 29, 2024
  44. DrahtBot requested review from davidgumberg on Sep 29, 2024
  45. l0rinc force-pushed on Sep 30, 2024
  46. l0rinc force-pushed on Sep 30, 2024
  47. l0rinc commented at 1:08 pm on September 30, 2024: contributor

    No code changes since https://github.com/bitcoin/bitcoin/commit/cb9be6729c4e27bf2d6d703c0d454a22cbcdb6e1, only commit messages (as requested) and rebased on latest master (separately).

    Edit: git range-diff cb9be67~7..cb9be67 48047ce~7..48047ce

  48. bitcoin deleted a comment on Oct 14, 2024
  49. bitcoin deleted a comment on Oct 14, 2024
  50. DrahtBot added the label Needs rebase on Oct 24, 2024
  51. coins: Split up AddFlags to remove invalid states
    CCoinsCacheEntry provided general access to its internal flags state, even though in reality it could only be clean, fresh, dirty or fresh|dirty.
    
    After it got dirtied we couldn't set the state back to clean by AddFlags(0) - tests were explicitly checking against that.
    
    This commit cleans up the public interface to make this distinction cleaner and invalid behavior impossible instead of just checked by tests.
    
    Also modernized `EmplaceCoinInternalDANGER` since it was already modified.
    
    Co-authored-by: Andrew Toth <andrewstoth@gmail.com>
    8e1381a991
  52. coins: Remove direct GetFlags access
    We don't need so much access to the internals of CCoinsCacheEntry, since many tests are just exercising invalid combinations this way.
    8df16f23a2
  53. test: Migrate GetCoinsMapEntry to return MaybeCoin e8e70a4ae3
  54. test: Group values and states in tests into CoinEntry wrappers 553db5dec1
  55. test: Validate error messages on fail
    The `ccoins_add` and `ccoins_write` tests check the actual exception error messages now instead of just that they fail for the given parameters.
    This enables us testing different exceptions in a more fine-grained way in later changes.
    f4099126ca
  56. test: Remove remaining unbounded flags from coins_tests cbf9b23cf5
  57. test: Compact ccoins_access and ccoins_spend 3b5b2457f7
  58. l0rinc force-pushed on Oct 24, 2024
  59. DrahtBot removed the label Needs rebase on Oct 24, 2024
  60. in src/coins.cpp:111 in 8e1381a991 outdated
    107@@ -107,13 +108,8 @@ void CCoinsViewCache::AddCoin(const COutPoint &outpoint, Coin&& coin, bool possi
    108 
    109 void CCoinsViewCache::EmplaceCoinInternalDANGER(COutPoint&& outpoint, Coin&& coin) {
    110     cachedCoinsUsage += coin.DynamicMemoryUsage();
    111-    auto [it, inserted] = cacheCoins.emplace(
    112-        std::piecewise_construct,
    


    laanwj commented at 10:55 am on October 29, 2024:
    Was this verbose code that is replaced here a workaround for lack of try_emplace?

    l0rinc commented at 3:19 pm on October 29, 2024:

    It seems this was introduced in 2017, when the project was still on C++11, but try_emplace was only introduced in C++17.

    When this was split out to EmplaceCoinInternalDANGER, the old impl was copied (on Aug 25, 2020), but C++17 support was only enabled in Nov 18, 2020.

    Note that this was already partially upgraded for FetchCoin.

  61. in src/coins.h:134 in 8e1381a991 outdated
    127@@ -128,6 +128,22 @@ struct CCoinsCacheEntry
    128     CoinsCachePair* m_next{nullptr};
    129     uint8_t m_flags{0};
    130 
    131+    //! Adding a flag also requires a self reference to the pair that contains
    132+    //! this entry in the CCoinsCache map and a reference to the sentinel of the
    133+    //! flagged pair linked list.
    134+    inline void AddFlags(uint8_t flags, CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
    


    ryanofsky commented at 12:20 pm on October 30, 2024:

    In commit “coins: Split up AddFlags to remove invalid states” (8e1381a991e325395fbc10df2fb091c508efa0f3)

    Not very important, but this commit would be easier to understand if AddFlags function were not moving at the same time as it being modified. Specifically it is harder to see the new Assume(flags & (DIRTY | FRESH)) change and related changes because of this.

    Would suggest not moving AddFlags this commit, and instead moving it in the next commit “coins: Remove direct GetFlags access” getting rid of both flags methods at the same time.

  62. in src/test/coins_tests.cpp:760 in e8e70a4ae3 outdated
    760-    SingleEntryCacheTest test(base_value, cache_value, cache_flags);
    761-
    762-    CAmount result_value;
    763-    char result_flags;
    764     try {
    765+        SingleEntryCacheTest test(base_value, cache_coin);
    


    ryanofsky commented at 12:23 pm on October 30, 2024:

    In commit “test: Migrate GetCoinsMapEntry to return MaybeCoin” (e8e70a4ae3cbd7a648629c2df4b5a7f6f0686438)

    Would be better to leave this outside of the try loop so it is clearer this is not expected to throw std::logic_error, and if it it did, the test framework would catch it.

    Same for CheckWriteCoins test below.

  63. in src/coins.h:165 in 8e1381a991 outdated
    177     }
    178 
    179-    //! Adding a flag also requires a self reference to the pair that contains
    180-    //! this entry in the CCoinsCache map and a reference to the sentinel of the
    181-    //! flagged pair linked list.
    182-    inline void AddFlags(uint8_t flags, CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
    


    ryanofsky commented at 12:30 pm on October 30, 2024:

    In commit “coins: Split up AddFlags to remove invalid states” (8e1381a991e325395fbc10df2fb091c508efa0f3)

    For this commit and other commits, it would be helpful if there was an indication about whether they were supposed to change behavior or test coverage. Would suggest including “refactor” in the subject for commits which are not supposed to change behavior like “coins, refactor: Split up AddFlags to remove invalid states” or could just say behavior isn’t changing in the commit description. The PR title does include “refactor:” but this isn’t visible at the commit level where it tends to be most useful (in git log and blame)

  64. in src/coins.h:178 in 8e1381a991 outdated
    178 
    179-    //! Adding a flag also requires a self reference to the pair that contains
    180-    //! this entry in the CCoinsCache map and a reference to the sentinel of the
    181-    //! flagged pair linked list.
    182-    inline void AddFlags(uint8_t flags, CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
    183+    inline void SetDirty(CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
    


    ryanofsky commented at 6:29 pm on October 30, 2024:

    In commit “coins: Split up AddFlags to remove invalid states” (8e1381a991e325395fbc10df2fb091c508efa0f3)

    Maybe wait for other opinions to see if this suggestion is worthwhile, but IMO this commit provides a good opportunity to remove the Assume(&self.second == this); footgun and make calling these functions more straightforward:

      0--- a/src/coins.cpp
      1+++ b/src/coins.cpp
      2@@ -49,7 +49,7 @@ CCoinsMap::iterator CCoinsViewCache::FetchCoin(const COutPoint &outpoint) const
      3             cachedCoinsUsage += ret->second.coin.DynamicMemoryUsage();
      4             if (ret->second.coin.IsSpent()) { // TODO GetCoin cannot return spent coins
      5                 // The parent only has an empty entry for this outpoint; we can consider our version as fresh.
      6-                ret->second.SetFresh(*ret, m_sentinel);
      7+                SetFresh(*ret, m_sentinel);
      8             }
      9         } else {
     10             cacheCoins.erase(ret);
     11@@ -95,8 +95,8 @@ void CCoinsViewCache::AddCoin(const COutPoint &outpoint, Coin&& coin, bool possi
     12         fresh = !it->second.IsDirty();
     13     }
     14     it->second.coin = std::move(coin);
     15-    it->second.SetDirty(*it, m_sentinel);
     16-    if (fresh) it->second.SetFresh(*it, m_sentinel);
     17+    SetDirty(*it, m_sentinel);
     18+    if (fresh) SetFresh(*it, m_sentinel);
     19     cachedCoinsUsage += it->second.coin.DynamicMemoryUsage();
     20     TRACE5(utxocache, add,
     21            outpoint.hash.data(),
     22@@ -109,7 +109,7 @@ void CCoinsViewCache::AddCoin(const COutPoint &outpoint, Coin&& coin, bool possi
     23 void CCoinsViewCache::EmplaceCoinInternalDANGER(COutPoint&& outpoint, Coin&& coin) {
     24     cachedCoinsUsage += coin.DynamicMemoryUsage();
     25     auto [it, inserted] = cacheCoins.try_emplace(std::move(outpoint), std::move(coin));
     26-    if (inserted) it->second.SetDirty(*it, m_sentinel);
     27+    if (inserted) SetDirty(*it, m_sentinel);
     28 }
     29 
     30 void AddCoins(CCoinsViewCache& cache, const CTransaction &tx, int nHeight, bool check_for_overwrite) {
     31@@ -139,7 +139,7 @@ bool CCoinsViewCache::SpendCoin(const COutPoint &outpoint, Coin* moveout) {
     32     if (it->second.IsFresh()) {
     33         cacheCoins.erase(it);
     34     } else {
     35-        it->second.SetDirty(*it, m_sentinel);
     36+        SetDirty(*it, m_sentinel);
     37         it->second.coin.Clear();
     38     }
     39     return true;
     40@@ -199,11 +199,11 @@ bool CCoinsViewCache::BatchWrite(CoinsViewCacheCursor& cursor, const uint256 &ha
     41                     entry.coin = it->second.coin;
     42                 }
     43                 cachedCoinsUsage += entry.coin.DynamicMemoryUsage();
     44-                entry.SetDirty(*itUs, m_sentinel);
     45+                SetDirty(*itUs, m_sentinel);
     46                 // We can mark it FRESH in the parent if it was FRESH in the child
     47                 // Otherwise it might have just been flushed from the parent's cache
     48                 // and already exist in the grandparent
     49-                if (it->second.IsFresh()) entry.SetFresh(*itUs, m_sentinel);
     50+                if (it->second.IsFresh()) SetFresh(*itUs, m_sentinel);
     51             }
     52         } else {
     53             // Found the entry in the parent cache
     54@@ -231,7 +231,7 @@ bool CCoinsViewCache::BatchWrite(CoinsViewCacheCursor& cursor, const uint256 &ha
     55                     itUs->second.coin = it->second.coin;
     56                 }
     57                 cachedCoinsUsage += itUs->second.coin.DynamicMemoryUsage();
     58-                itUs->second.SetDirty(*itUs, m_sentinel);
     59+                SetDirty(*itUs, m_sentinel);
     60                 // NOTE: It isn't safe to mark the coin as FRESH in the parent
     61                 // cache. If it already existed and was spent in the parent
     62                 // cache then marking it FRESH would prevent that spentness
     63--- a/src/coins.h
     64+++ b/src/coins.h
     65@@ -131,17 +131,16 @@ private:
     66     //! Adding a flag also requires a self reference to the pair that contains
     67     //! this entry in the CCoinsCache map and a reference to the sentinel of the
     68     //! flagged pair linked list.
     69-    inline void AddFlags(uint8_t flags, CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
     70+    static inline void AddFlags(uint8_t flags, CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
     71     {
     72         Assume(flags & (DIRTY | FRESH));
     73-        Assume(&self.second == this);
     74-        if (!m_flags) {
     75-            m_prev = sentinel.second.m_prev;
     76-            m_next = &sentinel;
     77+        if (!self.second.m_flags) {
     78+            self.second.m_prev = sentinel.second.m_prev;
     79+            self.second.m_next = &sentinel;
     80             sentinel.second.m_prev = &self;
     81-            m_prev->second.m_next = &self;
     82+            self.second.m_prev->second.m_next = &self;
     83         }
     84-        m_flags |= flags;
     85+        self.second.m_flags |= flags;
     86     }
     87 
     88 public:
     89@@ -175,11 +174,11 @@ public:
     90         SetClean();
     91     }
     92 
     93-    inline void SetDirty(CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
     94+    friend void SetDirty(CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
     95     {
     96         AddFlags(DIRTY, self, sentinel);
     97     }
     98-    inline void SetFresh(CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
     99+    friend void SetFresh(CoinsCachePair& self, CoinsCachePair& sentinel) noexcept
    100     {
    101         AddFlags(FRESH, self, sentinel);
    102     }
    
  65. in src/coins.h:189 in 8e1381a991 outdated
    198     }
    199-    inline void ClearFlags() noexcept
    200+    inline void SetClean() noexcept
    201     {
    202         if (!m_flags) return;
    203         m_next->second.m_prev = m_prev;
    


    ryanofsky commented at 7:05 pm on October 30, 2024:

    In commit “coins: Split up AddFlags to remove invalid states” (8e1381a991e325395fbc10df2fb091c508efa0f3)

    Not directly related to this PR, but would it make sense for SetClean to set m_prev and m_next to null here?

    It seems that would be better than keeping dangling or invalid pointers. This would also make it possible to add asserts in AddFlags to confirm that the linked list state is valid when it is called.

    Not suggesting any change for this commit, just wondering if this idea makes sense for a possible followup commit or PR.

  66. in src/test/coinscachepair_tests.cpp:24 in 8df16f23a2 outdated
    20@@ -21,7 +21,7 @@ std::list<CoinsCachePair> CreatePairs(CoinsCachePair& sentinel)
    21         auto node{std::prev(nodes.end())};
    22         node->second.SetDirty(*node, sentinel);
    23 
    24-        BOOST_CHECK_EQUAL(node->second.GetFlags(), CCoinsCacheEntry::DIRTY);
    25+        BOOST_CHECK(node->second.IsDirty());
    


    ryanofsky commented at 7:13 pm on October 30, 2024:

    In commit “coins: Remove direct GetFlags access” (8df16f23a22b2e0b72dd6338ed37c09485f5ede3)

    May should check !IsFresh() here so new check is equivalent to previous check. Alternately, could just mention in commit message that this commit reduces test coverage a little bit by not checking some flags that are not relevant to the tests.

    Same comment applies to other tests below like linked_list_random_deletion

  67. in src/test/coinscachepair_tests.cpp:66 in 8df16f23a2 outdated
    62@@ -63,7 +63,7 @@ BOOST_AUTO_TEST_CASE(linked_list_iteration)
    63 
    64     // Delete the nodes from the list to make sure there are no dangling pointers
    65     for (auto it{nodes.begin()}; it != nodes.end(); it = nodes.erase(it)) {
    66-        BOOST_CHECK_EQUAL(it->second.GetFlags(), 0);
    67+        BOOST_CHECK(!it->second.IsDirty() && !it->second.IsFresh());
    


    ryanofsky commented at 7:15 pm on October 30, 2024:

    In commit “coins: Remove direct GetFlags access” (8df16f23a22b2e0b72dd6338ed37c09485f5ede3)

    Not important, but usually test code and output is more readable with CHECK(a); CHECK(b); instead of CHECK(a && b)

  68. in src/test/coins_tests.cpp:577 in e8e70a4ae3 outdated
    572+
    573+    bool operator==(const CoinEntry& o) const = default;
    574+    friend std::ostream& operator<<(std::ostream& os, const CoinEntry& e) { return os << e.value << e.flags; }
    575+
    576+    constexpr bool IsDirty() const { return flags & DIRTY; }
    577+    constexpr bool IsFresh() const { return flags & FRESH; }
    


    ryanofsky commented at 7:31 pm on October 30, 2024:

    In commit “test: Migrate GetCoinsMapEntry to return MaybeCoin” (e8e70a4ae3cbd7a648629c2df4b5a7f6f0686438)

    It seems like these methods are not called in this commit, but they could be called in InsertCoinsMapEntry below. Maybe call them there or drop them here?

  69. in src/test/coins_tests.cpp:687 in 553db5dec1 outdated
    716-    CheckAccessCoin(VALUE1, SPENT , SPENT , DIRTY|FRESH, DIRTY|FRESH);
    717-    CheckAccessCoin(VALUE1, VALUE2, VALUE2, 0          , 0          );
    718-    CheckAccessCoin(VALUE1, VALUE2, VALUE2, FRESH      , FRESH      );
    719-    CheckAccessCoin(VALUE1, VALUE2, VALUE2, DIRTY      , DIRTY      );
    720-    CheckAccessCoin(VALUE1, VALUE2, VALUE2, DIRTY|FRESH, DIRTY|FRESH);
    721+    CheckAccessCoin(ABSENT, MISSING,            MISSING           );
    


    ryanofsky commented at 9:45 pm on October 30, 2024:

    In commit “test: Group values and states in tests into CoinEntry wrappers” (553db5dec1334ed5e8e07e34306f16de9afc22bb)

    Review note: It’s not really easy to verify by looking at code that new and old test cases are equivalent, but an approach that worked for me was to add extra prints showing the test cases, and then diff the output from test_bitcoin -t coins_tests before and after this commit, and confirm there are were no changes. My change used to test this was:

     0--- a/src/test/coins_tests.cpp
     1+++ b/src/test/coins_tests.cpp
     2@@ -286,6 +286,7 @@ void SimulationTest(CCoinsView* base, bool fake_best_block)
     3 // Run the above simulation for multiple base types.
     4 BOOST_FIXTURE_TEST_CASE(coins_cache_simulation_test, CacheTest)
     5 {
     6+    if ((1)) return;
     7     CCoinsViewTest base{m_rng};
     8     SimulationTest(&base, false);
     9 
    10@@ -318,6 +319,7 @@ UtxoData::iterator FindRandomFrom(const std::set<COutPoint> &utxoSet) {
    11 // has the expected effect (the other duplicate is overwritten at all cache levels)
    12 BOOST_FIXTURE_TEST_CASE(updatecoins_simulation_test, UpdateTest)
    13 {
    14+    if ((1)) return;
    15     SeedRandomForTest(SeedRand::ZEROS);
    16 
    17     bool spent_a_duplicate_coinbase = false;
    18@@ -572,7 +574,7 @@ struct CoinEntry {
    19     CoinEntry(CAmount v, char s) : value(v), flags(s) {}
    20 
    21     bool operator==(const CoinEntry& o) const = default;
    22-    friend std::ostream& operator<<(std::ostream& os, const CoinEntry& e) { return os << e.value << e.flags; }
    23+    friend std::ostream& operator<<(std::ostream& os, const CoinEntry& e) { return os << e.value << " " << int{e.flags}; }
    24 
    25     constexpr bool IsDirty() const { return flags & DIRTY; }
    26     constexpr bool IsFresh() const { return flags & FRESH; }
    27@@ -671,6 +673,7 @@ public:
    28 
    29 static void CheckAccessCoin(CAmount base_value, const CoinEntry& cache_coin, const CoinEntry& expected)
    30 {
    31+    tfm::format(std::cout, "CheckAccessCoin(%s, %s, %s)\n", base_value, cache_coin, expected);
    32     SingleEntryCacheTest test(base_value, cache_coin);
    33     test.cache.AccessCoin(OUTPOINT);
    34     test.cache.SelfTest(/*sanity_check=*/false);
    35@@ -717,6 +720,7 @@ BOOST_AUTO_TEST_CASE(ccoins_access)
    36 
    37 static void CheckSpendCoins(CAmount base_value, const CoinEntry& cache_coin, const CoinEntry& expected)
    38 {
    39+    tfm::format(std::cout, "CheckSpendCoins(%s, %s, %s)\n", base_value, cache_coin, expected);
    40     SingleEntryCacheTest test(base_value, cache_coin);
    41     test.cache.SpendCoin(OUTPOINT);
    42     test.cache.SelfTest();
    43@@ -763,6 +767,7 @@ BOOST_AUTO_TEST_CASE(ccoins_spend)
    44 
    45 static void CheckAddCoin(CAmount base_value, CAmount modify_value, const CoinEntry& cache_coin, const CoinEntry& expected, bool coinbase)
    46 {
    47+    tfm::format(std::cout, "CheckAddCoin(%s, %s, %s, %s, %s)\n", base_value, modify_value, cache_coin, expected, coinbase);
    48     try {
    49         SingleEntryCacheTest test(base_value, cache_coin);
    50         CTxOut output;
    51@@ -809,6 +814,7 @@ BOOST_AUTO_TEST_CASE(ccoins_add)
    52 
    53 void CheckWriteCoins(const CoinEntry& parent, const CoinEntry& child, const CoinEntry& expected)
    54 {
    55+    tfm::format(std::cout, "CheckWriteCoins(%s, %s, %s)\n", parent, child, expected);
    56     try {
    57         SingleEntryCacheTest test(ABSENT, parent);
    58         WriteCoinsViewEntry(test.cache, child);
    
  70. in src/test/coins_tests.cpp:786 in 553db5dec1 outdated
    820-    CheckAddCoin(VALUE2, VALUE3, VALUE3, FRESH      , DIRTY|FRESH, true );
    821-    CheckAddCoin(VALUE2, VALUE3, FAIL  , DIRTY      , NO_ENTRY   , false);
    822-    CheckAddCoin(VALUE2, VALUE3, VALUE3, DIRTY      , DIRTY      , true );
    823-    CheckAddCoin(VALUE2, VALUE3, FAIL  , DIRTY|FRESH, NO_ENTRY   , false);
    824-    CheckAddCoin(VALUE2, VALUE3, VALUE3, DIRTY|FRESH, DIRTY|FRESH, true );
    825+    for (auto base_value : {ABSENT, SPENT, VALUE1}) {
    


    ryanofsky commented at 10:04 pm on October 30, 2024:

    In commit “test: Group values and states in tests into CoinEntry wrappers” (553db5dec1334ed5e8e07e34306f16de9afc22bb)

    Note: because this for loop is moving from CheckAddCoin function to the top-level test code, this commit changes order of the tests. But tests that are performed should still be the same. It would be good to note in commit message that this commit does affect test order of this test but doesn’t change its coverage.

  71. in src/test/coins_tests.cpp:883 in 553db5dec1 outdated
    936-    for (const CAmount parent_value : {ABSENT, SPENT, VALUE1})
    937-        for (const CAmount child_value : {ABSENT, SPENT, VALUE2})
    938-            for (const char parent_flags : parent_value == ABSENT ? ABSENT_FLAGS : FLAGS)
    939-                for (const char child_flags : child_value == ABSENT ? ABSENT_FLAGS : CLEAN_FLAGS)
    940-                    CheckWriteCoins(parent_value, child_value, parent_value, parent_flags, child_flags, parent_flags);
    941+    for (const CoinEntry parent : {MISSING,
    


    ryanofsky commented at 10:08 pm on October 30, 2024:

    In commit “test: Group values and states in tests into CoinEntry wrappers” (553db5dec1334ed5e8e07e34306f16de9afc22bb)

    Note: Again in this part of the test, the order cases are run in seems to be changing, but the same set of cases is run. Would be helpful to mention this in commit message.

  72. ryanofsky approved
  73. ryanofsky commented at 10:42 pm on October 30, 2024: contributor
    Code review ACK 3b5b2457f713014fd5073da34b76a5a8cd796e4a. Nice change that should make the API more safe and also makes tests more complete and easier to maintain.
  74. DrahtBot requested review from andrewtoth on Oct 30, 2024

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-10-31 03:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me