kernel: De-globalize validation caches #30141

pull TheCharlatan wants to merge 6 commits into bitcoin:master from TheCharlatan:noGlobalScriptCache changing 19 files +199 −262
  1. TheCharlatan commented at 2:17 pm on May 19, 2024: contributor

    The validation caches are currently setup independently from where the rest of the validation code is initialized. This makes their ownership semantics unclear. There is also no clear enforcement on when and in what order they need to be initialized. The caches are always initialized in the BasicTestingSetup although a number of tests don’t actually need them.

    Solve this by moving the caches from global scope into the ChainstateManager class. This simplifies the usage of the kernel library by no longer requiring manual setup of the caches prior to using the ChainstateManager. Tests that need to access the caches can instantiate them independently.


    This pull request is part of the libbitcoinkernel project.

  2. DrahtBot commented at 2:17 pm on May 19, 2024: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage

    For detailed information about the code coverage, see the test coverage report.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK theuni
    Concept ACK ajtowns
    Stale ACK maflcko, stickies-v

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #30344 (kernel: remove mempool_persist by theuni)
    • #30342 (kernel, logging: Pass Logger instances to kernel objects by ryanofsky)
    • #30332 (Stratum v2 connectivity by Sjors)
    • #29745 (bench: Adds a benchmark for CheckInputScripts by kevkevinpal)
    • #29641 (scripted-diff: Use LogInfo/LogDebug over LogPrintf/LogPrint by maflcko)
    • #29520 (add -limitdummyscriptdatasize option by Retropex)
    • #29432 (Stratum v2 Template Provider (take 3) by Sjors)
    • #28676 ([WIP] Cluster mempool implementation by sdaftuar)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  3. DrahtBot added the label Validation on May 19, 2024
  4. TheCharlatan marked this as ready for review on May 19, 2024
  5. in src/validation.h:933 in 353ba0a19e outdated
    929@@ -924,7 +930,7 @@ class ChainstateManager
    930 public:
    931     using Options = kernel::ChainstateManagerOpts;
    932 
    933-    explicit ChainstateManager(const util::SignalInterrupt& interrupt, Options options, node::BlockManager::Options blockman_options);
    934+    explicit ChainstateManager(const util::SignalInterrupt& interrupt, Options options, node::BlockManager::Options blockman_options, bilingual_str& error);
    


    ajtowns commented at 6:04 pm on May 20, 2024:

    Rather than adding an error return in the constructor here, wouldn’t it be better to change our cuckoo cache to clamp its size when given too large a value? ie, InitScriptExecutionCache(135000000) is treated as 134,217,727 instead? The value passed in is only used as an approximate target in the first place. Historical context is #25527.

     0--- a/src/cuckoocache.h
     1+++ b/src/cuckoocache.h
     2@@ -360,16 +360,16 @@ public:
     3      * structure
     4      * [@returns](/bitcoin-bitcoin/contributor/returns/) A pair of the maximum number of elements storable (see setup()
     5      * documentation for more detail) and the approximate total size of these
     6-     * elements in bytes or std::nullopt if the size requested is too large.
     7+     * elements in bytes.
     8      */
     9-    std::optional<std::pair<uint32_t, size_t>> setup_bytes(size_t bytes)
    10+    std::pair<uint32_t, size_t> setup_bytes(size_t bytes)
    11     {
    12         size_t requested_num_elems = bytes / sizeof(Element);
    13         if (std::numeric_limits<uint32_t>::max() < requested_num_elems) {
    14-            return std::nullopt;
    15+            requested_num_elems = std::numeric_limits<uint32_t>::max();
    16         }
    17 
    18-        auto num_elems = setup(bytes/sizeof(Element));
    19+        auto num_elems = setup(requested_num_elems);
    20 
    21         size_t approx_size_bytes = num_elems * sizeof(Element);
    22         return std::make_pair(num_elems, approx_size_bytes);
    23--- a/src/script/sigcache.cpp
    24+++ b/src/script/sigcache.cpp
    25@@ -77,7 +77,7 @@ public:
    26         std::unique_lock<std::shared_mutex> lock(cs_sigcache);
    27         setValid.insert(entry);
    28     }
    29-    std::optional<std::pair<uint32_t, size_t>> setup_bytes(size_t n)
    30+    std::pair<uint32_t, size_t> setup_bytes(size_t n)
    31     {
    32         return setValid.setup_bytes(n);
    33     }
    34@@ -96,10 +96,7 @@ static CSignatureCache signatureCache;
    35 // signatureCache.
    36 bool InitSignatureCache(size_t max_size_bytes)
    37 {
    38-    auto setup_results = signatureCache.setup_bytes(max_size_bytes);
    39-    if (!setup_results) return false;
    40-
    41-    const auto [num_elems, approx_size_bytes] = *setup_results;
    42+    const auto [num_elems, approx_size_bytes] = signatureCache.setup_bytes(max_size_bytes);
    43     LogPrintf("Using %zu MiB out of %zu MiB requested for signature cache, able to store %zu elements\n",
    44               approx_size_bytes >> 20, max_size_bytes >> 20, num_elems);
    45     return true;
    46--- a/src/validation.cpp
    47+++ b/src/validation.cpp
    48@@ -1939,10 +1939,8 @@ bool InitScriptExecutionCache(size_t max_size_bytes)
    49     g_scriptExecutionCacheHasher.Write(nonce.begin(), 32);
    50     g_scriptExecutionCacheHasher.Write(nonce.begin(), 32);
    51 
    52-    auto setup_results = g_scriptExecutionCache.setup_bytes(max_size_bytes);
    53-    if (!setup_results) return false;
    54+    const auto [num_elems, approx_size_bytes] = g_scriptExecutionCache.setup_bytes(max_size_bytes);
    55 
    56-    const auto [num_elems, approx_size_bytes] = *setup_results;
    57     LogPrintf("Using %zu MiB out of %zu MiB requested for script execution cache, able to store %zu elements\n",
    58               approx_size_bytes >> 20, max_size_bytes >> 20, num_elems);
    59     return true;
    

    TheCharlatan commented at 8:38 am on May 21, 2024:
    I like this, no need to introduce all this error handling infrastructure for a debug-only option. Applied.
  6. ajtowns commented at 6:09 pm on May 20, 2024: contributor
    Concept ACK
  7. theuni commented at 7:09 pm on May 20, 2024: member
    Concept ACK. Nice.
  8. in src/script/sigcache.cpp:67 in 79c9c55b27 outdated
    71-static CSignatureCache signatureCache;
    72-
    73 // To be called once in AppInitMain/BasicTestingSetup to initialize the
    74 // signatureCache.
    75-bool InitSignatureCache(size_t max_size_bytes)
    76+bool InitSignatureCache(size_t max_size_bytes, CSignatureCache& signature_cache)
    


    theuni commented at 7:19 pm on May 20, 2024:
    Any reason to not make max_size_bytes a CSignatureCache constructor param? That way InitSignatureCache could go away.
  9. TheCharlatan force-pushed on May 21, 2024
  10. TheCharlatan commented at 8:47 am on May 21, 2024: contributor

    Updated 79c9c55b27bd113885da2c36ad76e5ed145027b3 -> 4a1df97a016935348e50f384ed52a6671990835f (noGlobalScriptCache_0 -> noGlobalScriptCache_1, compare)

    • Addressed @theuni’s comment, made CSignatureCache more RAII styled.
    • Addressed @ajtowns’s comment, applied his suggestion of dropping the error handling if the cache size exceeds the uint32_t maximum value.
  11. in src/node/chainstatemanager_args.cpp:58 in 4a1df97a01 outdated
    52@@ -53,6 +53,16 @@ util::Result<void> ApplyArgsManOptions(const ArgsManager& args, ChainstateManage
    53     opts.worker_threads_num = std::clamp(script_threads - 1, 0, MAX_SCRIPTCHECK_THREADS);
    54     LogPrintf("Script verification uses %d additional threads\n", opts.worker_threads_num);
    55 
    56+    if (auto max_size = args.GetIntArg("-maxsigcachesize")) {
    57+        // 1. When supplied with a max_size of 0, both InitSignatureCache and
    58+        //    InitScriptExecutionCache create the minimum possible cache (2
    


    theuni commented at 9:06 pm on May 21, 2024:
    Comment is now out of date.

    TheCharlatan commented at 9:21 pm on May 21, 2024:
    Thanks, fixed.
  12. TheCharlatan force-pushed on May 21, 2024
  13. TheCharlatan commented at 9:20 pm on May 21, 2024: contributor

    Updated 4a1df97a016935348e50f384ed52a6671990835f -> 5b1576bbc664a07e00b5528e13c1928df9bc9b0c (noGlobalScriptCache_1 -> noGlobalScriptCache_2, compare)

  14. DrahtBot added the label CI failed on May 22, 2024
  15. DrahtBot removed the label CI failed on May 22, 2024
  16. DrahtBot added the label Needs rebase on Jun 17, 2024
  17. TheCharlatan force-pushed on Jun 18, 2024
  18. TheCharlatan commented at 8:40 am on June 18, 2024: contributor

    Rebased 5b1576bbc664a07e00b5528e13c1928df9bc9b0c -> 63923c8da686da42f522771f338ea8f2a4f4e568 (noGlobalScriptCache_2 -> noGlobalScriptCache_3, compare)

  19. DrahtBot removed the label Needs rebase on Jun 18, 2024
  20. in src/cuckoocache.h:370 in 8960bc986b outdated
    368     {
    369         size_t requested_num_elems = bytes / sizeof(Element);
    370         if (std::numeric_limits<uint32_t>::max() < requested_num_elems) {
    371-            return std::nullopt;
    372+            requested_num_elems = std::numeric_limits<uint32_t>::max();
    373         }
    


    stickies-v commented at 10:17 am on June 18, 2024:

    nit: declaring requested_num_elems as a uint32_t would seem more intuitive:

     0diff --git a/src/cuckoocache.h b/src/cuckoocache.h
     1index c9bd45befc..85556a4da5 100644
     2--- a/src/cuckoocache.h
     3+++ b/src/cuckoocache.h
     4@@ -364,10 +364,10 @@ public:
     5      */
     6     std::pair<uint32_t, size_t> setup_bytes(size_t bytes)
     7     {
     8-        size_t requested_num_elems = bytes / sizeof(Element);
     9-        if (std::numeric_limits<uint32_t>::max() < requested_num_elems) {
    10-            requested_num_elems = std::numeric_limits<uint32_t>::max();
    11-        }
    12+        uint32_t requested_num_elems{(uint32_t)std::min(
    13+            bytes / sizeof(Element),
    14+            (size_t)std::numeric_limits<uint32_t>::max()
    15+        )};
    16 
    17         auto num_elems = setup(requested_num_elems);
    18 
    

    TheCharlatan commented at 8:57 am on June 19, 2024:
    Mmh, this does not feel more readable to me, because of the two casts.

    stickies-v commented at 10:10 am on June 19, 2024:
    It’s not really a readability suggestion, I just prefer having requested_nums_elems be the integer type for which we’re doing the bounds checking. But it’s just a nit so happy to leave as is.

    stickies-v commented at 2:53 pm on June 19, 2024:
    nit: std::optional no longer needed as an include
  21. in src/validation.h:371 in 1409fa4f41 outdated
    368+/**
    369+ * Convenience struct for initializing and passing the script execution cache.
    370+ */
    371+struct ValidationCache {
    372+    ScriptCache m_script_execution_cache;
    373+    ValidationCache(size_t script_execution_cache_bytes);
    


    stickies-v commented at 12:53 pm on June 18, 2024:

    To minimize future footguns by accidentally passing a cache by value, perhaps best to disable the copy constructor and copy assignment operator? (+ same for CSignatureCache in 63923c8da686da42f522771f338ea8f2a4f4e568)

     0diff --git a/src/validation.h b/src/validation.h
     1index 9f858bff1d..90f484bc9f 100644
     2--- a/src/validation.h
     3+++ b/src/validation.h
     4@@ -369,6 +369,10 @@ static_assert(std::is_nothrow_destructible_v<CScriptCheck>);
     5 struct ValidationCache {
     6     ScriptCache m_script_execution_cache;
     7     ValidationCache(size_t script_execution_cache_bytes);
     8+
     9+    // ValidationCache should only be passed by reference or pointer
    10+    ValidationCache(const ValidationCache&) = delete;
    11+    ValidationCache& operator=(const ValidationCache&) = delete;
    12 };
    13 
    14 /** Functions for validating blocks and updating the block tree */
    
  22. in src/validation.h:374 in 63923c8da6 outdated
    371+ * Convenience struct for initializing and passing the script execution cache
    372+ * and signature cache.
    373+ */
    374+struct ValidationCache {
    375+    ScriptCache m_script_execution_cache;
    376+    CSHA256 m_script_execution_cache_hasher;
    


    stickies-v commented at 8:20 pm on June 18, 2024:

    If I understand correctly, m_script_execution_cache_hasher is a pre-initialized hasher that we want to copy every time we use it, and we definitely don’t want to modify in-place. I think having it as a public struct member is not very robust to enforce that, perhaps better to use a getter here? E.g. something like:

     0diff --git a/src/validation.cpp b/src/validation.cpp
     1index 39ab7b56ab..600c427c93 100644
     2--- a/src/validation.cpp
     3+++ b/src/validation.cpp
     4@@ -2095,17 +2095,22 @@ bool CScriptCheck::operator()() {
     5     return VerifyScript(scriptSig, m_tx_out.scriptPubKey, witness, nFlags, CachingTransactionSignatureChecker(ptxTo, nIn, m_tx_out.nValue, cacheStore, *m_signature_cache, *txdata), &error);
     6 }
     7 
     8-ValidationCache::ValidationCache(size_t script_execution_cache_bytes, size_t signature_cache_bytes)
     9-    : m_signature_cache{signature_cache_bytes / 2}
    10+CSHA256 ValidationCache::CreateScriptExecutionCacheHasher()
    11 {
    12     // Setup the salted hasher
    13     uint256 nonce = GetRandHash();
    14     // We want the nonce to be 64 bytes long to force the hasher to process
    15     // this chunk, which makes later hash computations more efficient. We
    16     // just write our 32-byte entropy twice to fill the 64 bytes.
    17-    m_script_execution_cache_hasher.Write(nonce.begin(), 32);
    18-    m_script_execution_cache_hasher.Write(nonce.begin(), 32);
    19+    CSHA256 hasher;
    20+    hasher.Write(nonce.begin(), 32);
    21+    hasher.Write(nonce.begin(), 32);
    22+    return hasher;
    23+}
    24 
    25+ValidationCache::ValidationCache(size_t script_execution_cache_bytes, size_t signature_cache_bytes)
    26+    : m_signature_cache{signature_cache_bytes / 2}
    27+{
    28     const auto [num_elems, approx_size_bytes] = m_script_execution_cache.setup_bytes(script_execution_cache_bytes);
    29     LogPrintf("Using %zu MiB out of %zu MiB requested for script execution cache, able to store %zu elements\n",
    30               approx_size_bytes >> 20, script_execution_cache_bytes >> 20, num_elems);
    31@@ -2148,7 +2153,7 @@ bool CheckInputScripts(const CTransaction& tx, TxValidationState& state,
    32     // properly commits to the scriptPubKey in the inputs view of that
    33     // transaction).
    34     uint256 hashCacheEntry;
    35-    CSHA256 hasher = validation_cache.m_script_execution_cache_hasher;
    36+    auto hasher{validation_cache.ScriptExecutionCacheHasher()};
    37     hasher.Write(UCharCast(tx.GetWitnessHash().begin()), 32).Write((unsigned char*)&flags, sizeof(flags)).Finalize(hashCacheEntry.begin());
    38     AssertLockHeld(cs_main); //TODO: Remove this requirement by making CuckooCache not require external locks
    39     if (validation_cache.m_script_execution_cache.contains(hashCacheEntry, !cacheFullScriptStore)) {
    40diff --git a/src/validation.h b/src/validation.h
    41index c4b49ec3e8..1e48f6b4a4 100644
    42--- a/src/validation.h
    43+++ b/src/validation.h
    44@@ -369,11 +369,16 @@ static_assert(std::is_nothrow_destructible_v<CScriptCheck>);
    45  * Convenience struct for initializing and passing the script execution cache
    46  * and signature cache.
    47  */
    48-struct ValidationCache {
    49+class ValidationCache {
    50+public:
    51     ScriptCache m_script_execution_cache;
    52-    CSHA256 m_script_execution_cache_hasher;
    53+    CSHA256 ScriptExecutionCacheHasher() { return m_script_execution_cache_hasher; }
    54     CSignatureCache m_signature_cache;
    55     ValidationCache(size_t script_execution_cache_bytes, size_t signature_cache_bytes);
    56+private:
    57+    static CSHA256 CreateScriptExecutionCacheHasher();
    58+    //! Pre-initialized hasher to avoid having to recreate it for every hash calculation
    59+    const CSHA256 m_script_execution_cache_hasher{CreateScriptExecutionCacheHasher()};
    60 };
    61 
    62 /** Functions for validating blocks and updating the block tree */
    

    It seems this is quite brittle on master too, so not a regression, but probably something to address here?


    TheCharlatan commented at 8:57 am on June 19, 2024:
    Is there a reason you added a separate initialization function for the hasher?

    stickies-v commented at 10:22 am on June 19, 2024:

    Two (soft) reasons:

    1. to make m_script_execution_cache_hasher const
    2. having a separate construction function for the hasher seems more composable (e.g. testing the hasher) and makes the ValidationCache constructor more readable

    I’m fine with your approach too, though. It addresses my main concern, everything else is more of a nit.

  23. stickies-v commented at 8:47 pm on June 18, 2024: contributor
    Approach ACK 63923c8da686da42f522771f338ea8f2a4f4e568
  24. TheCharlatan force-pushed on Jun 19, 2024
  25. TheCharlatan commented at 8:57 am on June 19, 2024: contributor

    Thank you for the review @stickies-v,

    63923c8da686da42f522771f338ea8f2a4f4e568 -> 6ad4aa82056030a8a72b1315676e0703c1500d0d (noGlobalScriptCache_3 -> noGlobalScriptCache_4, compare)

    • Addressed @stickies-v’s comment, deleting copy and move constructor of the newly introduced ValidationCache class.
    • Addressed @stickies-v’s comment, applying the suggestion of making the pre-initialized hasher private and adding a getter for it.
  26. in src/test/txvalidationcache_tests.cpp:20 in f015a501e4 outdated
    15 #include <validation.h>
    16 
    17 #include <boost/test/unit_test.hpp>
    18 
    19 struct Dersig100Setup : public TestChain100Setup {
    20+    ValidationCache m_validation_cache;
    


    maflcko commented at 2:17 pm on June 19, 2024:
    May be good to explain why this is needed, when m_node.chainman->m_validation_cache already exists?

    TheCharlatan commented at 3:11 pm on June 19, 2024:
    I was thinking it might be useful to control the caches directly for the tests, but on second thought that does not seem like a good thing. Will revert.
  27. TheCharlatan force-pushed on Jun 19, 2024
  28. TheCharlatan commented at 3:15 pm on June 19, 2024: contributor

    6ad4aa82056030a8a72b1315676e0703c1500d0d -> fa660a266b3aa03f40c47a9330e3bce8be89bb54 (noGlobalScriptCache_4 -> noGlobalScriptCache_5, compare)

    • Addressed @stickies-v’s comment, changing integer type of requested_num_elems.
    • Addressed @maflcko’s comment, reverting introduction of a separate m_validation_cache member in the Dersig100Setup testing class and instead using m_node.chainman->m_validation_cache.
  29. in src/cuckoocache.h:367 in f1dad94b9d outdated
    368     {
    369-        size_t requested_num_elems = bytes / sizeof(Element);
    370-        if (std::numeric_limits<uint32_t>::max() < requested_num_elems) {
    371-            return std::nullopt;
    372-        }
    373+        uint32_t requested_num_elems{(uint32_t)std::min(
    


    maflcko commented at 3:19 pm on June 19, 2024:

    nit: You can combine the cast by using () over {}. Also you can specify the template to avoid the size_t cast, and ensure the right template is always picked:

    0uint32_t requested_num_elems(std::min<size_t>(
    
  30. in src/init.cpp:618 in f1dad94b9d outdated
    617@@ -618,7 +618,7 @@ void SetupServerArgs(ArgsManager& argsman)
    618     argsman.AddArg("-test=<option>", "Pass a test-only option. Options include : " + Join(TEST_OPTIONS_DOC, ", ") + ".", ArgsManager::ALLOW_ANY | ArgsManager::DEBUG_ONLY, OptionsCategory::DEBUG_TEST);
    619     argsman.AddArg("-capturemessages", "Capture all P2P messages to disk", ArgsManager::ALLOW_ANY | ArgsManager::DEBUG_ONLY, OptionsCategory::DEBUG_TEST);
    620     argsman.AddArg("-mocktime=<n>", "Replace actual time with " + UNIX_EPOCH_TIME + " (default: 0)", ArgsManager::ALLOW_ANY | ArgsManager::DEBUG_ONLY, OptionsCategory::DEBUG_TEST);
    621-    argsman.AddArg("-maxsigcachesize=<n>", strprintf("Limit sum of signature cache and script execution cache sizes to <n> MiB (default: %u)", DEFAULT_MAX_SIG_CACHE_BYTES >> 20), ArgsManager::ALLOW_ANY | ArgsManager::DEBUG_ONLY, OptionsCategory::DEBUG_TEST);
    622+    argsman.AddArg("-maxsigcachesize=<n>", strprintf("Limit sum of signature cache and script execution cache sizes to <n> MiB (default: %u). Will clamp to %u MiB if exceeded.", DEFAULT_MAX_SIG_CACHE_BYTES >> 20, std::numeric_limits<uint32_t>::max() >> 20), ArgsManager::ALLOW_ANY | ArgsManager::DEBUG_ONLY, OptionsCategory::DEBUG_TEST);
    


    stickies-v commented at 4:09 pm on June 19, 2024:
    This seems incorrect, we clamp the number of elements, not the number of bytes. I’d prefer not documenting the clamping behaviour here, given how generous the limit is.

    TheCharlatan commented at 9:16 am on June 28, 2024:
    Good catch, this is indeed wrong.
  31. in src/validation.h:375 in fa660a266b outdated
    372+ * and signature cache.
    373+ */
    374+class ValidationCache
    375+{
    376+private:
    377+    //! Pre-initialized hasher for to avoid having to recreate it for every hash calculation.
    


    stickies-v commented at 4:13 pm on June 19, 2024:

    typo nit

    0    //! Pre-initialized hasher to avoid having to recreate it for every hash calculation.
    
  32. in src/script/sigcache.h:20 in 2412ec2863 outdated
    15 #include <util/hasher.h>
    16 
    17+#include <cstddef>
    18+#include <cstdint>
    19+#include <shared_mutex>
    20+#include <utility>
    


    maflcko commented at 7:49 am on June 20, 2024:

    nit in 594189567956063fa1a44bef9e86e72c14207027:

    0script/sigcache.h should remove these lines:
    1- #include <cstdint>  // lines 18-18
    2- #include <utility>  // lines 20-20
    

    maflcko commented at 2:44 pm on June 20, 2024:
    nit: You added the headers in one commit, and then removed them in the next. Wrong order?

    TheCharlatan commented at 2:49 pm on June 20, 2024:
    They are required in 201b29e468f621a1fb055060920231ef46192414 and then no-longer in 974934b3c23ce7f34ef46f3dfc13468cd46f4b8a.
  33. TheCharlatan force-pushed on Jun 20, 2024
  34. TheCharlatan commented at 10:47 am on June 20, 2024: contributor

    fa660a266b3aa03f40c47a9330e3bce8be89bb54 -> f946b083fe1431fb8d5d2c080ed3c2938116c37a (noGlobalScriptCache_5 -> noGlobalScriptCache_6, compare)

    • Addressed @maflcko’s nit, removing unnecessary casts by using a template param and allowing type narrowing instead.
  35. in src/validation.h:374 in f946b083fe outdated
    372  */
    373 class ValidationCache
    374 {
    375+private:
    376+    //! Pre-initialized hasher for to avoid having to recreate it for every hash calculation.
    377+    CSHA256 m_script_execution_cache_hasher;
    


    maflcko commented at 12:42 pm on June 20, 2024:
    nit in /f946b083fe1431fb8d5d2c080ed3c2938116c37a: Unrelated change? The commit message is “De-globalize signature cache”, but this hunk makes the script execution cache private.
  36. in src/validation.h:347 in f946b083fe outdated
    343@@ -343,10 +344,11 @@ class CScriptCheck
    344     bool cacheStore;
    345     ScriptError error{SCRIPT_ERR_UNKNOWN_ERROR};
    346     PrecomputedTransactionData *txdata;
    347+    CSignatureCache* m_signature_cache;
    


    maflcko commented at 12:45 pm on June 20, 2024:
    nit in /f946b083fe1431fb8d5d2c080ed3c2938116c37a: Why use a pointer when a reference is better?

    TheCharlatan commented at 2:18 pm on June 20, 2024:
    I think we similarly make CTransaction a pointer to keep the CScriptCheck class is_nothrow_move_assignable_v.
  37. maflcko commented at 1:53 pm on June 20, 2024: member

    lgtm ACK f946b083fe1431fb8d5d2c080ed3c2938116c37a 📪

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: lgtm ACK f946b083fe1431fb8d5d2c080ed3c2938116c37a 📪
    3+e/Gr2EKw0EMF23Rn1I/M82DRXfwFyZ4Z4OTj9mkYl5Ts/2bHRLeFTINRyp5/NELtGmmoWQsNpJJtGXbXGW2BA==
    
  38. DrahtBot requested review from ajtowns on Jun 20, 2024
  39. DrahtBot requested review from stickies-v on Jun 20, 2024
  40. DrahtBot requested review from theuni on Jun 20, 2024
  41. TheCharlatan force-pushed on Jun 20, 2024
  42. TheCharlatan commented at 2:36 pm on June 20, 2024: contributor

    f946b083fe1431fb8d5d2c080ed3c2938116c37a -> 974934b3c23ce7f34ef46f3dfc13468cd46f4b8a (noGlobalScriptCache_6 -> noGlobalScriptCache_7, compare)

  43. maflcko commented at 2:46 pm on June 20, 2024: member

    ACK 974934b3c23ce7f34ef46f3dfc13468cd46f4b8a 🌤

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: ACK 974934b3c23ce7f34ef46f3dfc13468cd46f4b8a 🌤
    3t0ljkEzhaLlyhqM3Wen2M5qYseMJ8IvP7zBw8W314vzPwSnjzO8HfmrSCiuIpIyFKH9EAxgkpqPBONPeTrvwAg==
    
  44. in src/validation.h:92 in 56247bbdd0 outdated
    88@@ -88,6 +89,8 @@ extern std::condition_variable g_best_block_cv;
    89 /** Used to notify getblocktemplate RPC of new tips. */
    90 extern uint256 g_best_block;
    91 
    92+using ScriptCache = CuckooCache::cache<uint256, SignatureCacheHasher>;
    


    theuni commented at 3:12 pm on June 27, 2024:
    Move this into ValidationCache? It seems unused anywhere else.
  45. in src/validation.h:385 in a4fc935bf9 outdated
    380     ValidationCache(const ValidationCache&) = delete;
    381     ValidationCache(ValidationCache&&) = delete;
    382     ValidationCache& operator=(const ValidationCache&) = delete;
    383     ValidationCache& operator=(ValidationCache&&) = delete;
    384+
    385+    CSHA256 ScriptExecutionCacheHasher() { return m_script_execution_cache_hasher; }
    


    theuni commented at 3:17 pm on June 27, 2024:
    This function can be const.
  46. in src/validation.cpp:2104 in a4fc935bf9 outdated
    2104     // We want the nonce to be 64 bytes long to force the hasher to process
    2105     // this chunk, which makes later hash computations more efficient. We
    2106     // just write our 32-byte entropy twice to fill the 64 bytes.
    2107-    g_scriptExecutionCacheHasher.Write(nonce.begin(), 32);
    2108-    g_scriptExecutionCacheHasher.Write(nonce.begin(), 32);
    2109+    m_script_execution_cache_hasher.Write(nonce.begin(), 32);
    


    theuni commented at 3:21 pm on June 27, 2024:

    Might be useful to pass the nonce in rather than generating it here to make the order of rng/ValidationCache instantiation explicit.

    Presumably it’d also make for easier testing with a deterministic seed.


    TheCharlatan commented at 10:11 am on June 28, 2024:
    Does this even need to be a fresh random value every time? Couldn’t it just be any nothing up my sleeve number?

    theuni commented at 1:46 pm on June 28, 2024:
    I believe the idea behind these nonces is to introduce diversity so that everyone has a different cache. That way there’s no possibility of poisoning the “global” cache.

    TheCharlatan commented at 3:39 pm on June 28, 2024:
    I did this now, but was not completely sure where to put the nonce generation.
  47. in src/script/sigcache.cpp:22 in 201b29e468 outdated
    35-    CSHA256 m_salted_hasher_ecdsa;
    36-    CSHA256 m_salted_hasher_schnorr;
    37-    typedef CuckooCache::cache<uint256, SignatureCacheHasher> map_type;
    38-    map_type setValid;
    39-    std::shared_mutex cs_sigcache;
    40+    uint256 nonce = GetRandHash();
    


    theuni commented at 3:33 pm on June 27, 2024:
    Same comment here, would be nice to pass this in. Perhaps one nonce to rule them all? I don’t think there’d be any loss of security.
  48. in src/script/sigcache.h:49 in 201b29e468 outdated
    44+    typedef CuckooCache::cache<uint256, SignatureCacheHasher> map_type;
    45+    map_type setValid;
    46+    std::shared_mutex cs_sigcache;
    47+
    48+public:
    49+    CSignatureCache();
    


    theuni commented at 3:46 pm on June 27, 2024:
    Similar to @stickies-v’s comment in a previous commit, since we’re going to be passing around references and pointers to this, it’d be a good idea to make it non-copyable to avoid accidental copies.

    theuni commented at 3:55 pm on June 27, 2024:
    Oh, I guess the mutex takes care of that for us. Still, wouldn’t hurt to be explicit.

    stickies-v commented at 4:38 pm on June 27, 2024:
    I didn’t realize the mutex has the same practical effect when I mentioned it it in my original comment) - but I too would still prefer to be more explicit here. If for some reason that’s not a good idea, can we at least add a comment to the mutex to indicate that it also serves this purpose in case it is updated in the future?
  49. theuni approved
  50. theuni commented at 3:52 pm on June 27, 2024: member

    utACK modulo the non-copyable comment.

    Looks good. Left some comments with a few nice-to-have’s. I’m not sure how much trouble it’d be to pass in the nonces (and whether it’s 100% safe to reuse them), so that might make more sense as a follow-up.

  51. in src/node/chainstatemanager_args.cpp:64 in 56247bbdd0 outdated
    55@@ -56,6 +56,15 @@ util::Result<void> ApplyArgsManOptions(const ArgsManager& args, ChainstateManage
    56     opts.worker_threads_num = std::clamp(script_threads - 1, 0, MAX_SCRIPTCHECK_THREADS);
    57     LogPrintf("Script verification uses %d additional threads\n", opts.worker_threads_num);
    58 
    59+    if (auto max_size = args.GetIntArg("-maxsigcachesize")) {
    60+        // 1. When supplied with a max_size of 0, both the signature cache and
    61+        //    script execution cache create the minimum possible cache (2
    62+        //    elements). Therefore, we can use 0 as a floor here.
    63+        // 2. Multiply first, divide after to avoid integer truncation.
    64+        size_t clamped_size_each = std::max<int64_t>(*max_size, 0) * (1 << 20) / 2;
    


    stickies-v commented at 4:25 pm on June 27, 2024:
    reviewer note: orthogonal to this move-only change, but this has all kinds of under- and overflow issues. Since it’s a debug option, i suppose it’s not the end of the world though
  52. in src/script/sigcache.h:36 in 974934b3c2 outdated
    32+/**
    33+ * Valid signature cache, to avoid doing expensive ECDSA signature checking
    34+ * twice for every transaction (once when accepted into memory pool, and
    35+ * again when accepted into the block chain)
    36+ */
    37+class CSignatureCache
    


    stickies-v commented at 6:56 pm on June 27, 2024:
    nit: *ducks* it seems like now would be the perfect time to rename this to SignatureCache?
  53. in src/test/script_p2sh_tests.cpp:61 in 974934b3c2 outdated
    57@@ -58,6 +58,7 @@ BOOST_FIXTURE_TEST_SUITE(script_p2sh_tests, BasicTestingSetup)
    58 
    59 BOOST_AUTO_TEST_CASE(sign)
    60 {
    61+    CSignatureCache signature_cache{DEFAULT_MAX_SIG_CACHE_BYTES / 2};
    


    stickies-v commented at 7:05 pm on June 27, 2024:
    nit: any reason this isn’t initialized where it’s used?
  54. stickies-v approved
  55. stickies-v commented at 7:33 pm on June 27, 2024: contributor

    ACK 974934b3c23ce7f34ef46f3dfc13468cd46f4b8a

    Reduces dependencies on globals, cleans up the code, and generally makes it more robust through more self-contained code using an RAII-like pattern for ValidationCache, not requiring manual initialization of the caches preventing footguns.

    I think is good as-is but would prefer addressing various outstanding nits from reviewers as I think some are important enough - will quickly re-ACK myself.

  56. DrahtBot requested review from theuni on Jun 27, 2024
  57. validation: Don't error if maxsigcachesize exceeds uint32::max
    Instead clamp it to uint32::max if it exceeds it.
    
    Co-authored-by: Anthony Towns <aj@erisian.com.au>
    f369684efa
  58. TheCharlatan force-pushed on Jun 28, 2024
  59. TheCharlatan commented at 11:37 am on June 28, 2024: contributor

    Thank you for the reviews @stickies-v and @theuni!

    Updated 974934b3c23ce7f34ef46f3dfc13468cd46f4b8a -> 475c57ba482abd8675d0d3ccde98452808be5536 (noGlobalScriptCache_7 -> noGlobalScriptCache_8, compare)

    • Addressed @theuni’s comment, removed unneeded using declaration.
    • Addressed @stickies-v’s comment, removed incorrect description of cache size clamping behavior.
    • Addressed @stickies-v’s comment, removed unneeded optional include.
    • Addressed @theuni’s comment, marking getter for m_script_execution_cache_hasher as const.
    • Addressed @stickies-v’s comment, fixing docstring typo for the script execution cache hasher.
    • Addressed @theuni’s and @stickies-v’s comments, removing the copy constructor of the SignatureCache class. Also removed the move constructor, since I don’t see a good reason why that should be kept around either.
    • Addressed @stickies-v’s comment, renaming CSignatureCache to SignatureCache.
    • Addressed @stickies-v’s comment, moving signature_cache initialization closer to where it is used in script_p2sh_tests.cpp.
  60. TheCharlatan force-pushed on Jun 28, 2024
  61. stickies-v commented at 12:57 pm on June 28, 2024: contributor
    re-ACK 475c57ba482abd8675d0d3ccde98452808be5536, addressing outstanding review nits / touchups, no fundamental changes.
  62. DrahtBot requested review from maflcko on Jun 28, 2024
  63. in src/script/sigcache.h:49 in 003ce1a4d8 outdated
    44+    typedef CuckooCache::cache<uint256, SignatureCacheHasher> map_type;
    45+    map_type setValid;
    46+    std::shared_mutex cs_sigcache;
    47+
    48+public:
    49+    SignatureCache();
    


    theuni commented at 1:52 pm on June 28, 2024:
    Nit: Commit message still says CSignatureCache :)
  64. in src/validation.h:375 in a17bd08d59 outdated
    372+    CuckooCache::cache<uint256, SignatureCacheHasher> m_script_execution_cache;
    373+
    374+    ValidationCache(size_t script_execution_cache_bytes);
    375+
    376+    ValidationCache(const ValidationCache&) = delete;
    377+    ValidationCache(ValidationCache&&) = delete;
    


    theuni commented at 1:55 pm on June 28, 2024:

    Nit: I don’t see the harm in keeping it movable even if we don’t currently need to move it. As a reader of the code, if I needed to move it in the future, I’d be afraid to remove these because “they’re probably there for a reason”.

    Unless there actually is a reason?

  65. in src/script/sigcache.h:50 in 475c57ba48 outdated
    43@@ -46,7 +44,12 @@ class SignatureCache
    44     std::shared_mutex cs_sigcache;
    45 
    46 public:
    47-    SignatureCache();
    48+    SignatureCache(size_t max_size_bytes);
    49+
    50+    SignatureCache(const SignatureCache&) = delete;
    51+    SignatureCache(SignatureCache&&) = delete;
    


    theuni commented at 2:00 pm on June 28, 2024:
    Same comment here about deleted moves.
  66. DrahtBot requested review from theuni on Jun 28, 2024
  67. kernel: De-globalize script execution cache
    Move its ownership to the ChainstateManager class.
    
    Next to simplifying usage of the kernel library by no longer requiring
    manual setup of the cache prior to using validation code, it also slims
    down the amount of memory allocated by BasicTestingSetup.
    b3b0ced291
  68. kernel: De-globalize script execution cache hasher
    Move it to the ChainstateManager class.
    f2743cd26c
  69. Expose CSignatureCache class in header
    This is done in preparation for the following commit. Also rename it to
    SignatureCache.
    2c2d9c7ab6
  70. kernel: De-globalize signature cache
    Move its ownership to the ChainstateManager class.
    
    Next to simplifying usage of the kernel library by no longer requiring
    manual setup of the cache prior to using validation code, it also slims
    down the amount of memory allocated by BasicTestingSetup.
    
    Use this opportunity to make SignatureCache RAII styled
    25b84f0387
  71. TheCharlatan force-pushed on Jun 28, 2024
  72. TheCharlatan commented at 3:38 pm on June 28, 2024: contributor

    Updated 475c57ba482abd8675d0d3ccde98452808be5536 -> 064a401e10f2b2da8f31f682929431d06c671d93 (noGlobalScriptCache_8 -> noGlobalScriptCache_9, compare)

    • Addressed @theuni’s comment_1 and comment_2, reverting deletion of the move constructor in the caches.
    • Addressed @theuni’s comment, fixup in the commit message.
    • Applied suggestion in @theuni’s comment_1 and comment_2, making the nonce used to initialize the caches a constructor argument, as well as re-using the same nonce for both caches. I did not expose it as an argument to the chainstate manager, but I’m also not completely sure if it is generated in a better spot now.
  73. validation: Use a single nonce to initialize caches
    Potentially allowing a deterministic nonce to be passed in could also be
    useful for future testing.
    064a401e10
  74. TheCharlatan force-pushed on Jun 28, 2024
  75. theuni commented at 3:55 pm on June 28, 2024: member
    • Applied suggestion in @theuni’s comment_1 and comment_2, making the nonce used to initialize the caches a constructor argument, as well as re-using the same nonce for both caches. I did not expose it as an argument to the chainstate manager, but I’m also not completely sure if it is generated in a better spot now.

    Paging @sipa for a quick concept ACK that use of a single nonce doesn’t change any assumptions. See the comments above.

  76. theuni approved
  77. theuni commented at 5:37 pm on June 28, 2024: member
    utACK 064a401e10f2b2da8f31f682929431d06c671d93 assuming sipa doesn’t have a problem with the shared nonce.
  78. DrahtBot requested review from stickies-v on Jun 28, 2024

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-07-01 10:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me