kernel: Move kernel-related cache constants to kernel cache #31483

pull TheCharlatan wants to merge 8 commits into bitcoin:master from TheCharlatan:kernel_cache_sizes changing 18 files +321 −75
  1. TheCharlatan commented at 12:17 pm on December 12, 2024: contributor

    Carrying non-kernel related fields in the cache sizes for the indexes is confusing for kernel library users. The cache sizes are set currently with magic numbers in bitcoin-chainstate. The comments for the cache size calculations are not completely clear. The constants for the cache sizes are also currently in txdb.h, which is not an ideal place for holding all cache size related constants.

    Solve these things by moving the kernel-specific cache size fields to their own struct and moving the constants to either the node or the kernel cache sizes.

    This slightly changes the way the cache is allocated if (and only if) the txindex and/or blockfilterindex is used. Since they are now given precedence over the block tree db cache, this results in a bit less cache being allocated to the block tree db, coinsdb and coins caches. The effect is negligible though, i.e. cache sizes with default dbcache reported through the logs are:

    master:

    0Cache configuration:
    1* Using 2.0 MiB for block index database
    2* Using 56.0 MiB for transaction index database
    3* Using 49.0 MiB for basic block filter index database
    4* Using 8.0 MiB for chain state database
    5* Using 335.0 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
    

    this PR:

    0Cache configuration:
    1* Using 2.0 MiB for block index database
    2* Using 56.2 MiB for transaction index database
    3* Using 49.2 MiB for basic block filter index database
    4* Using 8.0 MiB for chain state database
    5* Using 334.5 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
    

    This PR is part of the libbitcoinkernel project.

  2. DrahtBot commented at 12:17 pm on December 12, 2024: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31483.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK stickies-v, ryanofsky, hodlinator

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #31650 (refactor: Enforce readability-avoid-const-params-in-decls by maflcko)
    • #31645 (optimization: increase default LevelDB write batch size to 64 MiB by l0rinc)
    • #30965 (kernel: Move block tree db open to block manager by TheCharlatan)
    • #29641 (scripted-diff: Use LogInfo over LogPrintf [WIP, NOMERGE, DRAFT] by maflcko)
    • #24539 (Add a “tx output spender” index by sstone)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  3. DrahtBot added the label Validation on Dec 12, 2024
  4. in src/kernel/caches.h:15 in 62da6a0b1e outdated
    10+
    11+//! min. -dbcache (MiB)
    12+static constexpr int64_t MIN_DB_CACHE{4};
    13+//! -dbcache default (MiB)
    14+static constexpr int64_t DEFAULT_DB_CACHE{450};
    15+//! Max memory allocated to block tree DB specific cache
    


    ajtowns commented at 12:40 pm on December 12, 2024:
    Missing the (MiB) comment.

    TheCharlatan commented at 12:58 pm on December 12, 2024:
    Thanks, fixed.
  5. TheCharlatan force-pushed on Dec 12, 2024
  6. in src/node/caches.h:21 in 173477b5ef outdated
    16 namespace node {
    17-struct CacheSizes {
    18-    int64_t block_tree_db;
    19-    int64_t coins_db;
    20-    int64_t coins;
    21+struct IndexCacheSizes {
    


    stickies-v commented at 8:18 am on December 13, 2024:

    nit: will we have separate structs for different node cache sizes? I think just keeping it CacheSizes and adding the node:: namespace in ambiguous places makes more sense for now?

     0diff --git a/src/node/caches.cpp b/src/node/caches.cpp
     1index 50e41b8d1e..41bd8c95d0 100644
     2--- a/src/node/caches.cpp
     3+++ b/src/node/caches.cpp
     4@@ -13,11 +13,11 @@
     5 #include <string>
     6 
     7 namespace node {
     8-std::tuple<IndexCacheSizes, kernel::CacheSizes> CalculateCacheSizes(const ArgsManager& args, size_t n_indexes)
     9+std::tuple<node::CacheSizes, kernel::CacheSizes> CalculateCacheSizes(const ArgsManager& args, size_t n_indexes)
    10 {
    11     int64_t nTotalCache = (args.GetIntArg("-dbcache", nDefaultDbCache) << 20);
    12     nTotalCache = std::max(nTotalCache, nMinDbCache << 20); // total cache cannot be less than nMinDbCache
    13-    IndexCacheSizes sizes;
    14+    CacheSizes sizes;
    15     sizes.tx_index = std::min(nTotalCache / 8, args.GetBoolArg("-txindex", DEFAULT_TXINDEX) ? nMaxTxIndexCache << 20 : 0);
    16     nTotalCache -= sizes.tx_index;
    17     sizes.filter_index = 0;
    18diff --git a/src/node/caches.h b/src/node/caches.h
    19index f4cc14bb23..c47134539b 100644
    20--- a/src/node/caches.h
    21+++ b/src/node/caches.h
    22@@ -14,11 +14,11 @@
    23 class ArgsManager;
    24 
    25 namespace node {
    26-struct IndexCacheSizes {
    27+struct CacheSizes {
    28     int64_t tx_index;
    29     int64_t filter_index;
    30 };
    31-std::tuple<IndexCacheSizes, kernel::CacheSizes> CalculateCacheSizes(const ArgsManager& args, size_t n_indexes = 0);
    32+std::tuple<node::CacheSizes, kernel::CacheSizes> CalculateCacheSizes(const ArgsManager& args, size_t n_indexes = 0);
    33 } // namespace node
    34 
    35 #endif // BITCOIN_NODE_CACHES_H
    36diff --git a/src/test/util/setup_common.h b/src/test/util/setup_common.h
    37index 46035772f5..fbea90a20b 100644
    38--- a/src/test/util/setup_common.h
    39+++ b/src/test/util/setup_common.h
    40@@ -104,7 +104,7 @@ struct BasicTestingSetup {
    41  * initialization behaviour.
    42  */
    43 struct ChainTestingSetup : public BasicTestingSetup {
    44-    node::IndexCacheSizes m_index_cache_sizes{std::get<0>(node::CalculateCacheSizes(m_args))};
    45+    node::CacheSizes m_index_cache_sizes{std::get<0>(node::CalculateCacheSizes(m_args))};
    46     kernel::CacheSizes m_kernel_cache_sizes{std::get<1>(node::CalculateCacheSizes(m_args))};
    47     bool m_coins_db_in_memory{true};
    48     bool m_block_tree_db_in_memory{true};
    

    TheCharlatan commented at 9:45 am on December 14, 2024:
    My expectation would be that if other non-index caches are added later, they would get their own struct too. This also might make it a bit easier in case the indexes ever get their own process?

    stickies-v commented at 7:33 pm on December 16, 2024:
    Works for me, thanks!
  7. in src/init.cpp:123 in 173477b5ef outdated
    119@@ -119,9 +120,10 @@ using common::AmountErrMsg;
    120 using common::InvalidPortErrMsg;
    121 using common::ResolveErrMsg;
    122 
    123+using kernel::CacheSizes;
    


    stickies-v commented at 10:01 am on December 13, 2024:

    nit: since it doesn’t meaningfully change the diff, I would much prefer not to use using here to immediately disambiguate with node::CacheSizes (+ in chainstate.cpp)

     0diff --git a/src/init.cpp b/src/init.cpp
     1index dd273cfe61..f52933b658 100644
     2--- a/src/init.cpp
     3+++ b/src/init.cpp
     4@@ -120,8 +120,6 @@ using common::AmountErrMsg;
     5 using common::InvalidPortErrMsg;
     6 using common::ResolveErrMsg;
     7 
     8-using kernel::CacheSizes;
     9-
    10 using node::ApplyArgsManOptions;
    11 using node::BlockManager;
    12 using node::CalculateCacheSizes;
    13@@ -1179,7 +1177,7 @@ static ChainstateLoadResult InitAndLoadChainstate(
    14     NodeContext& node,
    15     bool do_reindex,
    16     const bool do_reindex_chainstate,
    17-    CacheSizes& cache_sizes,
    18+    kernel::CacheSizes& cache_sizes,
    19     const ArgsManager& args)
    20 {
    21     const CChainParams& chainparams = Params();
    22diff --git a/src/node/chainstate.cpp b/src/node/chainstate.cpp
    23index 660d2c3289..d4d8828f73 100644
    24--- a/src/node/chainstate.cpp
    25+++ b/src/node/chainstate.cpp
    26@@ -29,14 +29,12 @@
    27 #include <memory>
    28 #include <vector>
    29 
    30-using kernel::CacheSizes;
    31-
    32 namespace node {
    33 // Complete initialization of chainstates after the initial call has been made
    34 // to ChainstateManager::InitializeChainstate().
    35 static ChainstateLoadResult CompleteChainstateInitialization(
    36     ChainstateManager& chainman,
    37-    const CacheSizes& cache_sizes,
    38+    const kernel::CacheSizes& cache_sizes,
    39     const ChainstateLoadOptions& options) EXCLUSIVE_LOCKS_REQUIRED(::cs_main)
    40 {
    41     auto& pblocktree{chainman.m_blockman.m_block_tree_db};
    42@@ -172,7 +170,7 @@ static ChainstateLoadResult CompleteChainstateInitialization(
    43     return {ChainstateLoadStatus::SUCCESS, {}};
    44 }
    45 
    46-ChainstateLoadResult LoadChainstate(ChainstateManager& chainman, const CacheSizes& cache_sizes,
    47+ChainstateLoadResult LoadChainstate(ChainstateManager& chainman, const kernel::CacheSizes& cache_sizes,
    48                                     const ChainstateLoadOptions& options)
    49 {
    50     if (!chainman.AssumedValidBlock().IsNull()) {
    

    edit: oh, that’s less true for the current node::IndexCacheSizes name of course. I personally still prefer keeping the namespace for clarity, but it also fits within the init.cpp coding style.

  8. in src/qt/optionsmodel.cpp:15 in 2e6c43f9c3 outdated
    11@@ -12,6 +12,7 @@
    12 
    13 #include <common/args.h>
    14 #include <interfaces/node.h>
    15+#include <kernel/caches.h>
    


    stickies-v commented at 10:14 am on December 13, 2024:
    nit: this is in the wrong commit (2e6c43f9c3aed577678a537b5456adaa31a2cb04)
  9. in src/kernel/caches.h:14 in 173477b5ef outdated
     9+#include <cstdint>
    10+
    11+//! min. -dbcache (MiB)
    12+static constexpr int64_t MIN_DB_CACHE{4};
    13+//! -dbcache default (MiB)
    14+static constexpr int64_t DEFAULT_DB_CACHE{450};
    


    stickies-v commented at 2:57 pm on December 13, 2024:
    I’m not sure what the best solution is, but aren’t MIN_DB_CACHE and DEFAULT_DB_CACHE node attributes, instead of kernel? Then again, we do need a default total cache size in kernel, so maybe the best approach is to just rename to e.g. MIN_KERNEL_CACHE and DEFAULT_KERNEL_CACHE - or redefine DEFAULT_DB_CACHE{DEFAULT_KERNEL_CACHE};?

    TheCharlatan commented at 9:43 am on December 14, 2024:
    Mmh, is there a scenario where DEFAULT_DB_CACHE and DEFAULT_KERNEL_CACHE would ever diverge? The current naming does not seem too confusing to me.

    stickies-v commented at 7:31 pm on December 16, 2024:

    Mmh, is there a scenario where DEFAULT_DB_CACHE and DEFAULT_KERNEL_CACHE would ever diverge?

    node could decide to increase the default total cache size (e.g. because it enables certain indexes by default), and that shouldn’t affect the kernel default cache size.

    The current naming does not seem too confusing to me.

    It’s not necessarily confusing, but -dbcache is a node setting, not a kernel setting. So I think DEFAULT_KERNEL_CACHE is both more accurate (in kernel/caches.h) and - as per the previous paragraph - more futureproof.

    Either way, it’s easy to fix in the future. My suggestion feels more logical to me but either is fine, so can be resolved if you disagree.

  10. in src/test/util/setup_common.h:107 in 173477b5ef outdated
    103@@ -103,7 +104,8 @@ struct BasicTestingSetup {
    104  * initialization behaviour.
    105  */
    106 struct ChainTestingSetup : public BasicTestingSetup {
    107-    node::CacheSizes m_cache_sizes{};
    108+    node::IndexCacheSizes m_index_cache_sizes{std::get<0>(node::CalculateCacheSizes(m_args))};
    


    stickies-v commented at 3:09 pm on December 13, 2024:
    This member is not used, best to remove until it is?
  11. stickies-v commented at 3:12 pm on December 13, 2024: contributor
    Concept ACK. Makes sense to separate this.
  12. TheCharlatan force-pushed on Dec 14, 2024
  13. TheCharlatan commented at 10:10 am on December 14, 2024: contributor

    Thanks for the review @stickies-v,

    Updated 173477b5efad4723863b38fe51cd1156620d01df -> f454987be6f140aca6c13ab3abf94c3afa183fdf (kernel_cache_sizes_0 -> kernel_cache_sizes_1, compare)

    • Addressed @stickies-v’s comment, removed unnecassary new index size member in the tests.
    • Addressed @stickies-v’s comment, fixed commit order for include addition and optionsmodel and removed the no longer needed txdb include.
  14. TheCharlatan force-pushed on Dec 16, 2024
  15. TheCharlatan commented at 9:28 pm on December 16, 2024: contributor

    Thanks for the comments @stickies-v,

    Updated f454987be6f140aca6c13ab3abf94c3afa183fdf -> 2c49e0bfd295dc0326d02cd0a80853d7dee885ac (kernel_cache_sizes_1 -> kernel_cache_sizes_2, compare)

    • Addressed @stickies-v’s comment, added a kernel-specific cache size constant and moved the minimum size out of the kernel.
  16. in src/node/caches.cpp:24 in 2c49e0bfd2 outdated
    27-    CacheSizes sizes;
    28-    sizes.block_tree_db = std::min(nTotalCache / 8, nMaxBlockDBCache << 20);
    29-    nTotalCache -= sizes.block_tree_db;
    30-    sizes.tx_index = std::min(nTotalCache / 8, args.GetBoolArg("-txindex", DEFAULT_TXINDEX) ? nMaxTxIndexCache << 20 : 0);
    31+    int64_t nTotalCache = (args.GetIntArg("-dbcache", DEFAULT_DB_CACHE) << 20);
    32+    nTotalCache = std::max(nTotalCache, MIN_DB_CACHE << 20); // total cache cannot be less than nMinDbCache
    


    stickies-v commented at 1:53 pm on December 17, 2024:
    0    nTotalCache = std::max(nTotalCache, MIN_DB_CACHE << 20); // total cache cannot be less than MIN_DB_CACHE
    

    edit: actually, the comment is pretty redundant, might as well just remove it too.


    TheCharlatan commented at 3:37 pm on December 17, 2024:
    thanks for the suggestion, I removed it.
  17. stickies-v approved
  18. stickies-v commented at 2:31 pm on December 17, 2024: contributor

    ACK 2c49e0bfd295dc0326d02cd0a80853d7dee885ac

    The only behaviour change is a very slight difference in cache allocation by moving block_tree_db allocation after index allocation. Because block_tree_db is capped at 2MiB, the maximum impact this change can have is very limited. Even at the most extreme case with -dbcache=4 -txindex -blockfilterindex=1, the results are not meaningfully different:

    On master:

    02024-12-17T14:25:35Z Cache configuration:
    12024-12-17T14:25:35Z * Using 0.5 MiB for block index database
    22024-12-17T14:25:35Z * Using 0.4 MiB for transaction index database
    32024-12-17T14:25:35Z * Using 0.4 MiB for basic block filter index database
    42024-12-17T14:25:35Z * Using 1.3 MiB for chain state database
    52024-12-17T14:25:35Z * Using 1.3 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
    

    With this PR:

    02024-12-17T14:27:38Z Cache configuration:
    12024-12-17T14:27:38Z * Using 0.4 MiB for block index database
    22024-12-17T14:27:38Z * Using 0.5 MiB for transaction index database
    32024-12-17T14:27:38Z * Using 0.4 MiB for basic block filter index database
    42024-12-17T14:27:38Z * Using 1.3 MiB for chain state database
    52024-12-17T14:27:38Z * Using 1.3 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
    
  19. TheCharlatan force-pushed on Dec 17, 2024
  20. TheCharlatan commented at 3:37 pm on December 17, 2024: contributor

    Updated 2c49e0bfd295dc0326d02cd0a80853d7dee885ac -> 8e929e343e14b0228f6f4841ecf2d85a73afda3b (kernel_cache_sizes_2 -> kernel_cache_sizes_3, compare)

  21. stickies-v approved
  22. stickies-v commented at 5:34 pm on December 17, 2024: contributor
    re-ACK 8e929e343e14b0228f6f4841ecf2d85a73afda3b - just a small doc change.
  23. in src/kernel/caches.h:29 in 8e929e343e outdated
    22@@ -18,10 +23,10 @@ struct CacheSizes {
    23 
    24     CacheSizes(int64_t total_cache)
    25     {
    26-        block_tree_db = std::min(total_cache / 8, nMaxBlockDBCache << 20);
    27+        block_tree_db = std::min(total_cache / 8, MAX_BLOCK_DB_CACHE << 20);
    28         total_cache -= block_tree_db;
    29         // use 25%-50% of the remainder for disk cache, capped by the maximum.
    30-        coins_db = std::min({total_cache / 2, (total_cache / 4) + (1 << 23), nMaxCoinsDBCache << 20});
    31+        coins_db = std::min({total_cache / 2, (total_cache / 4) + (1 << 23), MAX_COINS_DB_CACHE << 20});
    


    ryanofsky commented at 6:09 pm on December 17, 2024:

    re: In commit “kernel: Move kernel-related cache constants to kernel cache” (8e929e343e14b0228f6f4841ecf2d85a73afda3b)

    Precedes this PR, but this code does not seem to make any sense. (1 « 23) is 8MiB which is the same as MAX_COINS_DB_CACHE. So this is effectively min(total / 2, total / 4 + 8MiB, 8Mib) with the middle argument not doing anything. Could be simplified to std::min(total_cache / 2, MAX_COINS_DB_CACHE << 20) with no change in behavior AFAICT.


    TheCharlatan commented at 8:31 pm on December 17, 2024:
    Heh, I guess I should have actually done the math here. Will remove.
  24. ryanofsky approved
  25. ryanofsky commented at 6:24 pm on December 17, 2024: contributor

    Code review ACK 8e929e343e14b0228f6f4841ecf2d85a73afda3b. All these changes make sense and definitely make the kernel API nicer.

    I do also think it would be an improvement to change all the int64_t fields to size_t since they just get casted to size_t in the end anyway, and because size_t is more self-documenting and makes it clearer the sizes are in bytes.

  26. TheCharlatan force-pushed on Dec 17, 2024
  27. TheCharlatan commented at 9:14 pm on December 17, 2024: contributor

    Thanks for the review @ryanofsky! Updated 8e929e343e14b0228f6f4841ecf2d85a73afda3b -> 7517126d9fd2c3e50dc95ce831bac6895b950e24 (kernel_cache_sizes_3 -> kernel_cache_sizes_4, compare)

  28. in src/kernel/caches.h:26 in 7517126d9f outdated
    21+    size_t coins_db;
    22+    size_t coins;
    23+
    24+    CacheSizes(int64_t total_cache)
    25+    {
    26+        block_tree_db = std::min(total_cache / 8, MAX_BLOCK_DB_CACHE << 20);
    


    stickies-v commented at 12:12 pm on December 18, 2024:

    Would be prudent to ensure no negative values here

    0        assert(total_cache >= 0);
    1        block_tree_db = std::min(total_cache / 8, MAX_BLOCK_DB_CACHE << 20);
    

    hodlinator commented at 3:05 pm on December 21, 2024:

    re: #31483 (review)

    Or make total_cache size_t, removing the need for an assert, and also helps signal the unit being bytes? (May require adding casts though).


    We also have some implicit narrowing going on here, especially on 32-bit platforms, from int64_t to size_t. Adding curlies trigger an error.

    0        block_tree_db = {std::min(total_cache / 8, MAX_BLOCK_DB_CACHE << 20)};
    
  29. in src/kernel/caches.h:23 in 7517126d9f outdated
    17+
    18+namespace kernel {
    19+struct CacheSizes {
    20+    size_t block_tree_db;
    21+    size_t coins_db;
    22+    size_t coins;
    


    stickies-v commented at 12:23 pm on December 18, 2024:
    I think I’m ~0 on this change. I follow @ryanofsky’s reasoning, but it also goes against the work done e.g. in #23962. If kernel::CacheSizes becomes part of the public kernel interface, then keeping these members an int64_t might be more prudent, as per e.g. the link in this comment? But of course, size_t is a pretty natural fit for a size type.

    ryanofsky commented at 5:03 pm on December 18, 2024:

    re: #31483 (review)

    I agree with changes in the issues you linked to, but I don’t think the reasoning in those cases carries over to this one. int32_t is a good choice for transaction sizes and weights, because we know the sizes of transactions are bounded and can’t get very big, so it makes sense to use a small type, and we want the bounds to be platform independent so it makes to use fixed width types, and because they are numbers we do arithmetic with, and signed types tend to be safer than unsigned types for arithmetic.

    I think size_t is a better choice for the cache size variables in this PR, because these values are eventually used to allocate memory so making them size_t from the beginning is clearer and avoids the need for casts and conversions later on. Also size_t is the standard type to use for this purpose so it’s more idiomatic c++. Also maximum cache sizes do depend on the platform, so size_t a safe choice because it is guaranteed to be able to represent the maximum size of any allocated object on a platform. I do think a downside of choosing size_t is that it is unsigned and unsigned types have less intuitive overflow behavior, but if the values are just going to get casted to size_t later anyway, potentially bad overflow behavior is there no matter what.


    stickies-v commented at 6:14 pm on December 18, 2024:

    Yeah, thinking about it more I agree with all of your points. I was a bit alarmed by all the comparison operations we were doing with these (now) size_t types, but they almost all happen inside the constructor. Once the sizes are determined, they don’t seem likely to be used for arithmetic operations or comparisons, just for allocating memory.

    I think the benefits you outline outweigh the potential downsides, so I’m happy with the new approach.


    hodlinator commented at 3:03 pm on December 21, 2024:

    re: #31483 (review)

    My intuition was “why not keep using 64-bit types in case the platform is 32-bit”, but 32-bit implies it will not be able to make use of larger than size_t, so size_t is preferable.


    Added:

    0static_assert(sizeof(size_t) == 4);
    

    and built on a 32-bit container (i686-centos).


    What is worrying is that I also tested adding an extra zero here:

    0//! Suggested default amount of cache reserved for the kernel (MiB)
    1static constexpr int64_t DEFAULT_KERNEL_CACHE{4500};
    

    Shifted left 20 bits, that value should have become greater than 4294967295/max of size_t, but didn’t trigger any errors. :warning: See my suggestion in another comment about adding curlies to trigger narrowing conversion errors.

    This 10x tweak of that constant is caught by asserts I added to my branch.

  30. stickies-v approved
  31. stickies-v commented at 12:24 pm on December 18, 2024: contributor

    re-ACK 7517126d9fd2c3e50dc95ce831bac6895b950e24

    I think c71fc6ac6e62e5eea96e8aa4e0f43260afc81723 is an absolute improvement, but I’m ~0 on 7517126d9fd2c3e50dc95ce831bac6895b950e24. (edit: resolved)

  32. DrahtBot requested review from ryanofsky on Dec 18, 2024
  33. in src/node/caches.cpp:28 in c71fc6ac6e outdated
    23@@ -24,8 +24,7 @@ CacheSizes CalculateCacheSizes(const ArgsManager& args, size_t n_indexes)
    24         sizes.filter_index = max_cache / n_indexes;
    25         nTotalCache -= sizes.filter_index * n_indexes;
    26     }
    27-    sizes.coins_db = std::min(nTotalCache / 2, (nTotalCache / 4) + (1 << 23)); // use 25%-50% of the remainder for disk cache
    28-    sizes.coins_db = std::min(sizes.coins_db, nMaxCoinsDBCache << 20); // cap total coins db cache
    


    ryanofsky commented at 4:40 pm on December 18, 2024:

    In commit “init: Simplify coinsdb cache calculation” (c71fc6ac6e62e5eea96e8aa4e0f43260afc81723)

    For background on why previous code was written this way, apparently the first line coins_db = min(total / 2, total / 4 + 8MiB) made sense when it was added in b3ed4236beb7f68e1720ceb3da15e0c3682ef629 from #6102. But when the second line coins_db = min(coins_db, 8MiB) was added later in 32cab91278651d07a11132b7636dc3d21144e616 from #8273, the total / 4 term in the first line became redundant.

    It seems like the second change was probably intentional (even though the implementation is confusing) because the commit message says “This avoids that the extra cache memory goes to the much less effective leveldb cache instead of our application-level cache.”


    hodlinator commented at 10:10 am on December 21, 2024:

    re: #31483 (review)

    Agree with the simplification mostly. Only way the old calculation was useful was if someone was to tweak nMaxCoinsDBCache to be lower than (1 << 23)/8MiB, which is very unlikely.


    ryanofsky commented at 6:16 pm on January 8, 2025:

    To be pedantic, old calculation could only be useful if nMaxCoinsDBCache was higher not lower than 8 MiB. As long as it is 8MiB or lower this commit does not change behavior.

    In #6102 the coins_db cache size was set to min(50% total cache size, 8MiB + 25% total cache size). But then in #8273 it was capped to nMaxCoinsDBCache, so as long as nMaxCoinsDBCache <= 8 MiB, this is just equivalent to min(50% total cache size, nMaxCoinsDBCache)

  34. ryanofsky approved
  35. ryanofsky commented at 5:09 pm on December 18, 2024: contributor
    Code review ACK 7517126d9fd2c3e50dc95ce831bac6895b950e24. Just added two new commits since last review simplifying coinsdb cache calculation and switching to size_t to avoid casts. Thanks for the updates!
  36. in src/node/caches.h:26 in 7517126d9f outdated
    28+struct IndexCacheSizes {
    29+    size_t tx_index;
    30+    size_t filter_index;
    31 };
    32-CacheSizes CalculateCacheSizes(const ArgsManager& args, size_t n_indexes = 0);
    33+std::tuple<IndexCacheSizes, kernel::CacheSizes> CalculateCacheSizes(const ArgsManager& args, size_t n_indexes = 0);
    


    hodlinator commented at 9:27 am on December 21, 2024:

    nit: Me and the compilers would prefer a contrived name over std::tuple<...>. It still allows for the structured bindings in init.cpp.

    0struct IndexAndKernelCacheSizes {
    1    IndexCacheSizes index;
    2    kernel::CacheSizes kernel;
    3};
    4IndexAndKernelCacheSizes CalculateCacheSizes(const ArgsManager& args, size_t n_indexes = 0);
    
     0diff --git a/src/node/caches.cpp b/src/node/caches.cpp
     1index bd0c93d76f..c7197b6ef1 100644
     2--- a/src/node/caches.cpp
     3+++ b/src/node/caches.cpp
     4@@ -18,7 +18,7 @@ static constexpr int64_t MAX_TX_INDEX_CACHE{1024};
     5 static constexpr int64_t MAX_FILTER_INDEX_CACHE{1024};
     6 
     7 namespace node {
     8-std::tuple<IndexCacheSizes, kernel::CacheSizes> CalculateCacheSizes(const ArgsManager& args, size_t n_indexes)
     9+IndexAndKernelCacheSizes CalculateCacheSizes(const ArgsManager& args, size_t n_indexes)
    10 {
    11     int64_t nTotalCache = (args.GetIntArg("-dbcache", DEFAULT_DB_CACHE) << 20);
    12     nTotalCache = std::max(nTotalCache, MIN_DB_CACHE << 20);
    13diff --git a/src/node/caches.h b/src/node/caches.h
    14index 02b91cee87..afa0946d8b 100644
    15--- a/src/node/caches.h
    16+++ b/src/node/caches.h
    17@@ -9,7 +9,6 @@
    18 
    19 #include <cstddef>
    20 #include <cstdint>
    21-#include <tuple>
    22 
    23 class ArgsManager;
    24 
    25@@ -23,7 +22,11 @@ struct IndexCacheSizes {
    26     size_t tx_index;
    27     size_t filter_index;
    28 };
    29-std::tuple<IndexCacheSizes, kernel::CacheSizes> CalculateCacheSizes(const ArgsManager& args, size_t n_indexes = 0);
    30+struct IndexAndKernelCacheSizes {
    31+    IndexCacheSizes index;
    32+    kernel::CacheSizes kernel;
    33+};
    34+IndexAndKernelCacheSizes CalculateCacheSizes(const ArgsManager& args, size_t n_indexes = 0);
    35 } // namespace node
    36 
    37 #endif // BITCOIN_NODE_CACHES_H
    38diff --git a/src/test/util/setup_common.h b/src/test/util/setup_common.h
    39index ca9e2111a4..33ad258457 100644
    40--- a/src/test/util/setup_common.h
    41+++ b/src/test/util/setup_common.h
    42@@ -104,7 +104,7 @@ struct BasicTestingSetup {
    43  * initialization behaviour.
    44  */
    45 struct ChainTestingSetup : public BasicTestingSetup {
    46-    kernel::CacheSizes m_kernel_cache_sizes{std::get<1>(node::CalculateCacheSizes(m_args))};
    47+    kernel::CacheSizes m_kernel_cache_sizes{node::CalculateCacheSizes(m_args).kernel};
    48     bool m_coins_db_in_memory{true};
    49     bool m_block_tree_db_in_memory{true};
    50     std::function<void()> m_make_chainman{};
    

    The compilation speed difference is probably not measurable, more a case of death by a thousand eventual cuts.

  37. in src/txdb.h:32 in 7517126d9f outdated
    27 static const int64_t nDefaultDbBatchSize = 16 << 20;
    28-//! min. -dbcache (MiB)
    29-static const int64_t nMinDbCache = 4;
    30-//! Max memory allocated to block tree DB specific cache, if no -txindex (MiB)
    31-static const int64_t nMaxBlockDBCache = 2;
    32-//! Max memory allocated to block tree DB specific cache, if -txindex (MiB)
    


    hodlinator commented at 9:48 am on December 21, 2024:

    nit: The message of commit 69159326810822c804ab53a009ef57fb5f28fac9 belonging to this PR states:

    It always applied in the same way, no matter how the txindex is setup.

    There is some nuance here, the comment was correct in 32cab91278651d07a11132b7636dc3d21144e616 (laanwj 2016 #8273), but was no longer valid after 8181db88f6e0ed96654951e18b1558cd8f78765b (jimpo 2017 #13033). Could add nuance to the commit message.

  38. in src/node/caches.cpp:25 in 7517126d9f outdated
    28-    sizes.block_tree_db = std::min(nTotalCache / 8, nMaxBlockDBCache << 20);
    29-    nTotalCache -= sizes.block_tree_db;
    30-    sizes.tx_index = std::min(nTotalCache / 8, args.GetBoolArg("-txindex", DEFAULT_TXINDEX) ? nMaxTxIndexCache << 20 : 0);
    31+    int64_t nTotalCache = (args.GetIntArg("-dbcache", DEFAULT_DB_CACHE) << 20);
    32+    nTotalCache = std::max(nTotalCache, MIN_DB_CACHE << 20);
    33+    IndexCacheSizes sizes;
    


    hodlinator commented at 10:15 am on December 21, 2024:

    nit: Arguably clearer:

    0    IndexCacheSizes index_sizes;
    
  39. in src/node/caches.cpp:19 in 7517126d9f outdated
    12+#include <algorithm>
    13+#include <string>
    14+
    15+// Unlike for the UTXO database, for the txindex scenario the leveldb cache make
    16+// a meaningful difference: https://github.com/bitcoin/bitcoin/pull/8273#issuecomment-229601991
    17+static constexpr int64_t MAX_TX_INDEX_CACHE{1024};
    


    hodlinator commented at 3:05 pm on December 21, 2024:
    No longer documents unit (MiB) due to commit 69159326810822c804ab53a009ef57fb5f28fac9.
  40. hodlinator commented at 10:25 pm on December 21, 2024: contributor

    Code review 7517126d9fd2c3e50dc95ce831bac6895b950e24

    Some concern over implicit narrowing on 32-bit platforms (see inline comments).

    Rough suggestions: https://github.com/TheCharlatan/bitcoin/compare/kernel_cache_sizes...hodlinator:bitcoin:pr/31483_suggestions + Use of curly braces to catch narrowing. + The above encourages asserts, leading to finding cases where truncation would eventually happen (increasing DEFAULT_KERNEL_CACHE by 10x triggers assert). - Not so happy about my strategy to avoid repeated asserts and casts, that part is definitely WIP. Edit: Ended up replacing user defined literal solution with simple free function (previous fancy & ugly solution).

  41. TheCharlatan commented at 10:53 pm on December 22, 2024: contributor
    Thank you for the suggestions @hodlinator, I appreciate the time you took for this. I was originally intent on tackling the 32bit case in a follow-up, mainly because I was thinking whether we should trap the max amount of cache at numeric_limits<size_t>::max(). That might be a bit friendlier than just running into an assert, some kind of allocation failure, or much less cache being allocated than requested. With your suggestions, I agree that this should be handled in this PR now.
  42. doc: Correct docstring describing max block tree db cache
    It is always applied in the same way, no matter how the txindex is
    setup. This was no longer accurate after 8181db8, where their
    initialization was made independent.
    5db7d4d3d2
  43. TheCharlatan force-pushed on Jan 3, 2025
  44. TheCharlatan commented at 5:18 pm on January 3, 2025: contributor

    Thank you for the review and code suggestions @hodlinator,

    Updated 7517126d9fd2c3e50dc95ce831bac6895b950e24 -> ad80e410b3bf8814cb5b997715ee8cb28f2a2a66 (kernel_cache_sizes_4 -> kernel_cache_sizes_5, compare)

    • Add handling for dbcache sizes exceeding size_t size. Previously the value would just overflow if larger than 4096 on 32 bit platforms. Now it returns an error on startup. Add a release note for it, since it changes behaviour.
    • Addressed @hodlinator’s comment, made total_cache a size_t type.
    • Addressed @hodlinator’s comment and @stickies-v’s comment, replacing std::tuple with a new struct.
    • Addressed @hodlinator’s comment, renaming sizes to index_sizes in node/caches.cpp.
    • Addressed @hodlinator’s comment, document unit for MAX_TX_INDEX_CACHE.
    • Addressed @hodlinator’s comment, add the suggested MiBToBytes function.
    • Addressed @hodlinator’s comment, added a hint to the commit message on when the docstring describing the block tree db cache initialization became inaccurate.
  45. in src/kernel/caches.h:26 in ad80e410b3 outdated
    21+constexpr size_t MiBToBytes(int64_t mib)
    22+{
    23+    Assert(mib >= 0);
    24+    const int64_t bytes{mib << 20};
    25+    Assert(bytes >= 0);
    26+    Assert(static_cast<uint64_t>(bytes) <= static_cast<uint64_t>(std::numeric_limits<size_t>::max()));
    


    hodlinator commented at 10:48 am on January 6, 2025:

    (Sorry to be correcting my own suggestion, ran out of steam a bit by the time I landed on the approach of using this free function).

    • The 2 first asserts could be improved through replacing them with this more complex one.
    • size_t is always unsigned, it was probably unnecessary to cast the max value in the last assert.
    0    Assert(std::countl_zero(static_cast<uint64_t>(mib)) >= 21); // Ensure signed bit is unset + enough zeros to shift.
    1    const int64_t bytes{mib << 20};
    2    Assert(static_cast<uint64_t>(bytes) <= std::numeric_limits<size_t>::max());
    

    countl_zero requires:

    0+ #include <bit>
    
  46. in src/kernel/caches.h:23 in ad80e410b3 outdated
    16+//! Max memory allocated to block tree DB specific cache (MiB)
    17+static constexpr int64_t MAX_BLOCK_DB_CACHE{2};
    18+//! Max memory allocated to coin DB specific cache (MiB)
    19+static constexpr int64_t MAX_COINS_DB_CACHE{8};
    20+
    21+constexpr size_t MiBToBytes(int64_t mib)
    


    hodlinator commented at 11:00 am on January 6, 2025:

    nit: Could deserve a brief comment?

    0//! Guards against truncation of values before converting.
    1constexpr size_t MiBToBytes(int64_t mib)
    
  47. in src/node/caches.cpp:27 in ad80e410b3 outdated
    32-    nTotalCache -= sizes.block_tree_db;
    33-    sizes.tx_index = std::min(nTotalCache / 8, args.GetBoolArg("-txindex", DEFAULT_TXINDEX) ? nMaxTxIndexCache << 20 : 0);
    34-    nTotalCache -= sizes.tx_index;
    35-    sizes.filter_index = 0;
    36+    int64_t db_cache = args.GetIntArg("-dbcache", DEFAULT_DB_CACHE);
    37+    if (static_cast<uint64_t>(db_cache << 20) > static_cast<uint64_t>(std::numeric_limits<size_t>::max())) {
    


    hodlinator commented at 11:16 am on January 6, 2025:

    Good to have this for user supplied data instead of triggering asserts!

    nit: Might be able to skip one cast:

    0    if (static_cast<uint64_t>(db_cache << 20) > std::numeric_limits<size_t>::max()) {
    

    hodlinator commented at 1:12 pm on January 6, 2025:
    Would be good to check against negative -dbcache values as well.

    TheCharlatan commented at 4:02 pm on January 6, 2025:
    Ugh, must have switched to the old branch by mistake when I was testing this. I think it would be better to clamp the negative values to zero, which preserves the current behaviour.
  48. in src/init.cpp:1615 in ad80e410b3 outdated
    1612+    }
    1613+    auto [index_cache_sizes, kernel_cache_sizes] = *cache_sizes;
    1614 
    1615     LogPrintf("Cache configuration:\n");
    1616-    LogPrintf("* Using %.1f MiB for block index database\n", cache_sizes.block_tree_db * (1.0 / 1024 / 1024));
    1617+    LogPrintf("* Using %.1f MiB for block index database\n", kernel_cache_sizes.block_tree_db * (1.0 / 1024 / 1024));
    


    hodlinator commented at 2:06 pm on January 6, 2025:
    nit: Could use more modern LogInfo() instead.
  49. in doc/release-notes-31483.md:5 in ad80e410b3 outdated
    0@@ -0,0 +1,6 @@
    1+Updated settings
    2+------
    3+
    4+- On 32-bit systems an error is returned on startup if the user passes a
    5+  `-dbcache` value exceeding 4096MiB.
    


    hodlinator commented at 2:25 pm on January 6, 2025:

    nit: Simpler to say 4GiB?

    I know 32-bit Windows has issues using more than 2GB unless one sets /LARGEADDRESSAWARE, not sure whether we do that. See https://learn.microsoft.com/en-us/windows/win32/memory/memory-limits-for-windows-releases

    Edit: I guess we don’t ship 32-bit Windows builds any longer, so the 2GB limit should not be a concern.

  50. hodlinator commented at 2:29 pm on January 6, 2025: contributor

    Code review ad80e410b3bf8814cb5b997715ee8cb28f2a2a66

    Thanks for incorporating so much of my feedback!

    Verified that documented cache configuration output matches PR summary. (Also double-checked that allocated memory budget without -txindex and -blockfilterindex=basic is unchanged).

    Built for 32-bit (i686 centos).


    ad80e410b3bf8814cb5b997715ee8cb28f2a2a66 - commit message typo: s/not/note

  51. TheCharlatan force-pushed on Jan 6, 2025
  52. TheCharlatan commented at 4:31 pm on January 6, 2025: contributor

    Updated ad80e410b3bf8814cb5b997715ee8cb28f2a2a66 -> 54bdaaefbc683a61f3051dd90bbaf69de5d6cac5 (kernel_cache_sizes_5 -> kernel_cache_sizes_6, compare)

    • Addressed @hodlinator’s comment, took suggested simplification for the number range check and dropped unneeded cast.
    • Addressed @hodlinator’s comment, added a brief comment describing the purpose of MiBToBytes.
    • Addressed @hodlinator’s comment, added a check for negative -dbcache values. They are now just converted to 0, which ends up not changing the current behaviour where negative values are permitted, but the minimum dbcache value is allocated.
    • Addressed @hodlinator’s comment, use LogInfo() where applicable.
    • Addressed @hodlinator’s comment, mention 4GiB instead of 4096MiB for easier reading in the release notes.
  53. in src/kernel/caches.h:25 in 54bdaaefbc outdated
    20+static constexpr int64_t MAX_COINS_DB_CACHE{8};
    21+
    22+//! Guard against truncation of values before converting.
    23+constexpr size_t MiBToBytes(int64_t mib)
    24+{
    25+    Assert(std::countl_zero(static_cast<uint64_t>(mib)) >= 21); // Ensure signed bit is unset + enough zeros to shift.
    


    hodlinator commented at 11:31 am on January 7, 2025:

    nit: If you re-touch, slightly more correct:

    0    Assert(std::countl_zero(static_cast<uint64_t>(mib)) >= 21); // Ensure sign bit is unset + enough zeros to shift.
    

    hodlinator commented at 1:07 pm on January 7, 2025:
    Should probably fix this typo in commit it was introduced (a5742b7c09f1b4204abf4a1992018fb7be2fef27), instead of in final commit (224bfeb07f7561470451c95dc0df1db959bc187e).
  54. in src/kernel/caches.h:13 in 54bdaaefbc outdated
     8+#include <util/check.h>
     9+
    10+#include <algorithm>
    11+#include <bit>
    12+#include <cstdint>
    13+#include <limits>
    


    hodlinator commented at 11:33 am on January 7, 2025:
    nit: <limits> should be added in same commit as <bit> and MiBToBytes().
  55. hodlinator approved
  56. hodlinator commented at 11:43 am on January 7, 2025: contributor

    crACK 54bdaaefbc683a61f3051dd90bbaf69de5d6cac5

    Thanks again for incorporating my suggestions. Preserving previous behavior for negative -dbcache values seems good.

    3 nits remaining:

    54bdaaefbc683a61f3051dd90bbaf69de5d6cac5 still has “not”-typo in commit message:

    Add a release not for the change in behaviour on 32-bit systems.

  57. DrahtBot requested review from ryanofsky on Jan 7, 2025
  58. DrahtBot requested review from stickies-v on Jan 7, 2025
  59. TheCharlatan force-pushed on Jan 7, 2025
  60. TheCharlatan commented at 12:59 pm on January 7, 2025: contributor

    Thanks for the reviews @hodlinator, just cleaning up your nits:

    Updated 54bdaaefbc683a61f3051dd90bbaf69de5d6cac5 -> 224bfeb07f7561470451c95dc0df1db959bc187e (kernel_cache_sizes_6 -> kernel_cache_sizes_7, compare)

  61. TheCharlatan force-pushed on Jan 7, 2025
  62. TheCharlatan commented at 2:52 pm on January 7, 2025: contributor

    Updated 224bfeb07f7561470451c95dc0df1db959bc187e -> 9f61eb074438f85ed1a23fb3da94e4c68855d3fb (kernel_cache_sizes_7 -> kernel_cache_sizes_8, compare)

    • Fixed comment in the correct commit now.
  63. hodlinator approved
  64. hodlinator commented at 3:21 pm on January 7, 2025: contributor

    re-ACK 9f61eb074438f85ed1a23fb3da94e4c68855d3fb

    Fixed all my remaining nits.

    Tested a few -dbcache values

    0₿ build/src/bitcoind -dbcache=-1
    1...
    22025-01-07T15:13:09Z Cache configuration:
    32025-01-07T15:13:09Z * Using 0.5 MiB for block index database
    42025-01-07T15:13:09Z * Using 1.8 MiB for chain state database
    52025-01-07T15:13:09Z * Using 1.8 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
    

    Corresponds to

    0//! min. -dbcache (MiB)
    1static constexpr int64_t MIN_DB_CACHE{4};
    

    0₿ build/src/bitcoind 
    1...
    22025-01-07T15:14:48Z Cache configuration:
    32025-01-07T15:14:48Z * Using 2.0 MiB for block index database
    42025-01-07T15:14:48Z * Using 8.0 MiB for chain state database
    52025-01-07T15:14:48Z * Using 440.0 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
    

    0₿ build/src/bitcoind -dbcache=8000 -txindex=1 -blockfilterindex=1
    1...
    22025-01-07T15:18:09Z Cache configuration:
    32025-01-07T15:18:09Z * Using 2.0 MiB for block index database
    42025-01-07T15:18:09Z * Using 1000.0 MiB for transaction index database
    52025-01-07T15:18:09Z * Using 875.0 MiB for basic block filter index database
    62025-01-07T15:18:09Z * Using 8.0 MiB for chain state database
    72025-01-07T15:18:09Z * Using 6115.0 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
    

    Ship it! :rocket:

  65. TheCharlatan force-pushed on Jan 7, 2025
  66. TheCharlatan commented at 3:53 pm on January 7, 2025: contributor

    Sorry to invalidate the ACK again, Updated 9f61eb074438f85ed1a23fb3da94e4c68855d3fb -> 578654c686e394d08bac7c3bcddbfd90b46eaa62 (kernel_cache_sizes_8 -> kernel_cache_sizes_9, compare)

    • Forgot to drop the newlines in LogInfo, again. Fixed that now. -_-
  67. hodlinator approved
  68. hodlinator commented at 3:56 pm on January 7, 2025: contributor
    re-ACK 578654c686e394d08bac7c3bcddbfd90b46eaa62
  69. in src/kernel/caches.h:25 in 578654c686 outdated
    20+static constexpr int64_t MAX_COINS_DB_CACHE{8};
    21+
    22+//! Guard against truncation of values before converting.
    23+constexpr size_t MiBToBytes(int64_t mib)
    24+{
    25+    Assert(std::countl_zero(static_cast<uint64_t>(mib)) >= 21); // Ensure sign bit is unset + enough zeros to shift.
    


    stickies-v commented at 11:44 am on January 8, 2025:
    I think the docstring is misleading: we’re casting to uint64_t, so the sign bit will always be unset. Relatedly, this function does not deal with negative input well, so I think we should first assert that mib is positive and update the docstring to remove the signedness reference.

    hodlinator commented at 9:18 am on January 9, 2025:

    Tried compiling a static_assert with mib replaced by -1. Got the expected result from the sign bit:

    0/home/hodlinator/bitcoin/src/bitcoind.cpp:164:63: error: static assertion failed
    1  164 |     static_assert(std::countl_zero(static_cast<uint64_t>(-1)) >= 21);
    2      |                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~
    3/home/hodlinator/bitcoin/src/bitcoind.cpp:164:63: note: the comparison reduces to ‘(0 >= 21)’
    

    Edit: The sign bit is referring to the mib variable, and that bit position does not become unset just from casting.


    stickies-v commented at 10:03 am on January 9, 2025:
    Did you hard-code the -1 literal? I don’t think that holds for a variable mib at runtime?

    hodlinator commented at 10:35 am on January 9, 2025:
    Yes, I hard-coded the literal into the static_assert above. What part of integer arithmetic and casting changes when done in a runtime context?

    hodlinator commented at 10:39 am on January 9, 2025:
    Even if the assert is correct, maybe the fact that this thread exists is justification enough for replacing it with something closer to what it was originally. Like #31483 (review)

    stickies-v commented at 11:22 am on January 9, 2025:

    Sorry, maybe I’m misunderstanding you but I think for a run-time invocation of MiBToBytes() with mib==-1 this assertion Assert(std::countl_zero(static_cast<uint64_t>(mib)) >= 21); would not be hit, which is different to your hard-coded static_assert expression.

    But I agree that @ryanofsky’s right-shifting the max approach preferable (and also what I suggested in my inlined approach in #31483 (review))

  70. in src/kernel/caches.h:23 in 578654c686 outdated
    18+static constexpr int64_t MAX_BLOCK_DB_CACHE{2};
    19+//! Max memory allocated to coin DB specific cache (MiB)
    20+static constexpr int64_t MAX_COINS_DB_CACHE{8};
    21+
    22+//! Guard against truncation of values before converting.
    23+constexpr size_t MiBToBytes(int64_t mib)
    


    stickies-v commented at 11:54 am on January 8, 2025:

    I think a std::optional<size_t> return type is more appropriate for this function, the assertions make this function hard to test. Below diff updates the return type, adds a positiveness check (as per my other comment), and adds a test case:

     0diff --git a/src/bitcoin-chainstate.cpp b/src/bitcoin-chainstate.cpp
     1index 46ca35ea90..623aec5aa0 100644
     2--- a/src/bitcoin-chainstate.cpp
     3+++ b/src/bitcoin-chainstate.cpp
     4@@ -123,7 +123,7 @@ int main(int argc, char* argv[])
     5     util::SignalInterrupt interrupt;
     6     ChainstateManager chainman{interrupt, chainman_opts, blockman_opts};
     7 
     8-    kernel::CacheSizes cache_sizes{MiBToBytes(DEFAULT_KERNEL_CACHE)};
     9+    kernel::CacheSizes cache_sizes{*Assert(MiBToBytes(DEFAULT_KERNEL_CACHE))};
    10     node::ChainstateLoadOptions options;
    11     auto [status, error] = node::LoadChainstate(chainman, cache_sizes, options);
    12     if (status != node::ChainstateLoadStatus::SUCCESS) {
    13diff --git a/src/kernel/caches.h b/src/kernel/caches.h
    14index 2b5ba9eefc..298ebf8d43 100644
    15--- a/src/kernel/caches.h
    16+++ b/src/kernel/caches.h
    17@@ -11,6 +11,7 @@
    18 #include <bit>
    19 #include <cstdint>
    20 #include <limits>
    21+#include <optional>
    22 
    23 //! Suggested default amount of cache reserved for the kernel (MiB)
    24 static constexpr int64_t DEFAULT_KERNEL_CACHE{450};
    25@@ -20,11 +21,15 @@ static constexpr int64_t MAX_BLOCK_DB_CACHE{2};
    26 static constexpr int64_t MAX_COINS_DB_CACHE{8};
    27 
    28 //! Guard against truncation of values before converting.
    29-constexpr size_t MiBToBytes(int64_t mib)
    30+constexpr std::optional<size_t> MiBToBytes(int64_t mib)
    31 {
    32-    Assert(std::countl_zero(static_cast<uint64_t>(mib)) >= 21); // Ensure sign bit is unset + enough zeros to shift.
    33+    if (mib < 0 || std::countl_zero(static_cast<uint64_t>(mib)) < 21) {
    34+        return std::nullopt;
    35+    }
    36     const int64_t bytes{mib << 20};
    37-    Assert(static_cast<uint64_t>(bytes) <= std::numeric_limits<size_t>::max());
    38+    if (static_cast<uint64_t>(bytes) > std::numeric_limits<size_t>::max()) {
    39+        return std::nullopt;
    40+    }
    41     return static_cast<size_t>(bytes);
    42 }
    43 
    44@@ -36,9 +41,9 @@ struct CacheSizes {
    45 
    46     CacheSizes(size_t total_cache)
    47     {
    48-        block_tree_db = std::min(total_cache / 8, MiBToBytes(MAX_BLOCK_DB_CACHE));
    49+        block_tree_db = std::min(total_cache / 8, *Assert(MiBToBytes(MAX_BLOCK_DB_CACHE)));
    50         total_cache -= block_tree_db;
    51-        coins_db = std::min(total_cache / 2, MiBToBytes(MAX_COINS_DB_CACHE));
    52+        coins_db = std::min(total_cache / 2, *Assert(MiBToBytes(MAX_COINS_DB_CACHE)));
    53         total_cache -= coins_db;
    54         coins = total_cache; // the rest goes to the coins cache
    55     }
    56diff --git a/src/node/caches.cpp b/src/node/caches.cpp
    57index 9dbbccd304..f59748a222 100644
    58--- a/src/node/caches.cpp
    59+++ b/src/node/caches.cpp
    60@@ -8,6 +8,7 @@
    61 #include <index/txindex.h>
    62 #include <kernel/caches.h>
    63 #include <logging.h>
    64+#include <util/check.h>
    65 
    66 #include <algorithm>
    67 #include <optional>
    68@@ -31,14 +32,14 @@ std::optional<CacheSizes> CalculateCacheSizes(const ArgsManager& args, size_t n_
    69 
    70     // negative values are permitted, but interpreted as zero.
    71     db_cache = std::max(int64_t{0}, db_cache);
    72-    size_t total_cache = std::max(MiBToBytes(db_cache), MiBToBytes(MIN_DB_CACHE));
    73+    size_t total_cache = std::max(*Assert(MiBToBytes(db_cache)), *Assert(MiBToBytes(MIN_DB_CACHE)));
    74 
    75     IndexCacheSizes index_sizes;
    76-    index_sizes.tx_index = std::min(total_cache / 8, args.GetBoolArg("-txindex", DEFAULT_TXINDEX) ? MiBToBytes(MAX_TX_INDEX_CACHE) : 0);
    77+    index_sizes.tx_index = std::min(total_cache / 8, args.GetBoolArg("-txindex", DEFAULT_TXINDEX) ? *Assert(MiBToBytes(MAX_TX_INDEX_CACHE)) : 0);
    78     total_cache -= index_sizes.tx_index;
    79     index_sizes.filter_index = 0;
    80     if (n_indexes > 0) {
    81-        int64_t max_cache = std::min(total_cache / 8, MiBToBytes(MAX_FILTER_INDEX_CACHE));
    82+        int64_t max_cache = std::min(total_cache / 8, *Assert(MiBToBytes(MAX_FILTER_INDEX_CACHE)));
    83         index_sizes.filter_index = max_cache / n_indexes;
    84         total_cache -= index_sizes.filter_index * n_indexes;
    85     }
    86diff --git a/src/test/CMakeLists.txt b/src/test/CMakeLists.txt
    87index ed73e44901..7cf42c2aa9 100644
    88--- a/src/test/CMakeLists.txt
    89+++ b/src/test/CMakeLists.txt
    90@@ -71,6 +71,7 @@ add_executable(test_bitcoin
    91   httpserver_tests.cpp
    92   i2p_tests.cpp
    93   interfaces_tests.cpp
    94+  kernel_caches_tests.cpp
    95   key_io_tests.cpp
    96   key_tests.cpp
    97   logging_tests.cpp
    

    stickies-v commented at 2:25 pm on January 8, 2025:

    I’m not sure we need MiBToBytes() at all. All but one callsite can be eliminated by just storing the constants as size_t byte amounts, allowing compile-time guarantees that we’re not overflowing the system’s size_t size. It adds a few more right shift operations, but those aren’t prone to overflowing, and are only required for viewing purposes.

    The only place where we’re dealing with run-time values (in CalculateCacheSizes(), we already partially (and buggily, as per my other comment) implemented the checks, so I’d suggest inlining the function there.

      0diff --git a/src/bitcoin-chainstate.cpp b/src/bitcoin-chainstate.cpp
      1index 46ca35ea90..51c482607a 100644
      2--- a/src/bitcoin-chainstate.cpp
      3+++ b/src/bitcoin-chainstate.cpp
      4@@ -123,7 +123,7 @@ int main(int argc, char* argv[])
      5     util::SignalInterrupt interrupt;
      6     ChainstateManager chainman{interrupt, chainman_opts, blockman_opts};
      7 
      8-    kernel::CacheSizes cache_sizes{MiBToBytes(DEFAULT_KERNEL_CACHE)};
      9+    kernel::CacheSizes cache_sizes{DEFAULT_KERNEL_CACHE};
     10     node::ChainstateLoadOptions options;
     11     auto [status, error] = node::LoadChainstate(chainman, cache_sizes, options);
     12     if (status != node::ChainstateLoadStatus::SUCCESS) {
     13diff --git a/src/init.cpp b/src/init.cpp
     14index 1ae968760d..300b46a3d2 100644
     15--- a/src/init.cpp
     16+++ b/src/init.cpp
     17@@ -489,7 +489,7 @@ void SetupServerArgs(ArgsManager& argsman, bool can_listen_ipc)
     18     argsman.AddArg("-conf=<file>", strprintf("Specify path to read-only configuration file. Relative paths will be prefixed by datadir location (only useable from command line, not configuration file) (default: %s)", BITCOIN_CONF_FILENAME), ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);
     19     argsman.AddArg("-datadir=<dir>", "Specify data directory", ArgsManager::ALLOW_ANY | ArgsManager::DISALLOW_NEGATION, OptionsCategory::OPTIONS);
     20     argsman.AddArg("-dbbatchsize", strprintf("Maximum database write batch size in bytes (default: %u)", nDefaultDbBatchSize), ArgsManager::ALLOW_ANY | ArgsManager::DEBUG_ONLY, OptionsCategory::OPTIONS);
     21-    argsman.AddArg("-dbcache=<n>", strprintf("Maximum database cache size <n> MiB (minimum %d, default: %d). Make sure you have enough RAM. In addition, unused memory allocated to the mempool is shared with this cache (see -maxmempool).", MIN_DB_CACHE, DEFAULT_DB_CACHE), ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);
     22+    argsman.AddArg("-dbcache=<n>", strprintf("Maximum database cache size <n> MiB (minimum %d, default: %d). Make sure you have enough RAM. In addition, unused memory allocated to the mempool is shared with this cache (see -maxmempool).", MIN_DB_CACHE >> 20, DEFAULT_DB_CACHE >> 20), ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);
     23     argsman.AddArg("-includeconf=<file>", "Specify additional configuration file, relative to the -datadir path (only useable from configuration file, not command line)", ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);
     24     argsman.AddArg("-allowignoredconf", strprintf("For backwards compatibility, treat an unused %s file in the datadir as a warning, not an error.", BITCOIN_CONF_FILENAME), ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);
     25     argsman.AddArg("-loadblock=<file>", "Imports blocks from external file on startup", ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);
     26diff --git a/src/kernel/caches.h b/src/kernel/caches.h
     27index 2b5ba9eefc..d30813d284 100644
     28--- a/src/kernel/caches.h
     29+++ b/src/kernel/caches.h
     30@@ -8,25 +8,14 @@
     31 #include <util/check.h>
     32 
     33 #include <algorithm>
     34-#include <bit>
     35 #include <cstdint>
     36-#include <limits>
     37 
     38-//! Suggested default amount of cache reserved for the kernel (MiB)
     39-static constexpr int64_t DEFAULT_KERNEL_CACHE{450};
     40-//! Max memory allocated to block tree DB specific cache (MiB)
     41-static constexpr int64_t MAX_BLOCK_DB_CACHE{2};
     42-//! Max memory allocated to coin DB specific cache (MiB)
     43-static constexpr int64_t MAX_COINS_DB_CACHE{8};
     44-
     45-//! Guard against truncation of values before converting.
     46-constexpr size_t MiBToBytes(int64_t mib)
     47-{
     48-    Assert(std::countl_zero(static_cast<uint64_t>(mib)) >= 21); // Ensure sign bit is unset + enough zeros to shift.
     49-    const int64_t bytes{mib << 20};
     50-    Assert(static_cast<uint64_t>(bytes) <= std::numeric_limits<size_t>::max());
     51-    return static_cast<size_t>(bytes);
     52-}
     53+//! Suggested default amount of cache reserved for the kernel (bytes)
     54+static constexpr size_t DEFAULT_KERNEL_CACHE{450 << 20};
     55+//! Max memory allocated to block tree DB specific cache (bytes)
     56+static constexpr size_t MAX_BLOCK_DB_CACHE{2 << 20};
     57+//! Max memory allocated to coin DB specific cache (bytes)
     58+static constexpr size_t MAX_COINS_DB_CACHE{8 << 20};
     59 
     60 namespace kernel {
     61 struct CacheSizes {
     62@@ -36,9 +25,9 @@ struct CacheSizes {
     63 
     64     CacheSizes(size_t total_cache)
     65     {
     66-        block_tree_db = std::min(total_cache / 8, MiBToBytes(MAX_BLOCK_DB_CACHE));
     67+        block_tree_db = std::min(total_cache / 8, MAX_BLOCK_DB_CACHE);
     68         total_cache -= block_tree_db;
     69-        coins_db = std::min(total_cache / 2, MiBToBytes(MAX_COINS_DB_CACHE));
     70+        coins_db = std::min(total_cache / 2, MAX_COINS_DB_CACHE);
     71         total_cache -= coins_db;
     72         coins = total_cache; // the rest goes to the coins cache
     73     }
     74diff --git a/src/node/caches.cpp b/src/node/caches.cpp
     75index 9dbbccd304..af27d9af0f 100644
     76--- a/src/node/caches.cpp
     77+++ b/src/node/caches.cpp
     78@@ -10,35 +10,35 @@
     79 #include <logging.h>
     80 
     81 #include <algorithm>
     82+#include <bit>
     83 #include <optional>
     84 #include <string>
     85 
     86 // Unlike for the UTXO database, for the txindex scenario the leveldb cache make
     87 // a meaningful difference: [#8273 (comment)](/bitcoin-bitcoin/8273/#issuecomment-229601991)
     88-//! Max memory allocated to tx index DB specific cache in MiB.
     89-static constexpr int64_t MAX_TX_INDEX_CACHE{1024};
     90-//! Max memory allocated to all block filter index caches combined in MiB.
     91-static constexpr int64_t MAX_FILTER_INDEX_CACHE{1024};
     92+//! Max memory allocated to tx index DB specific cache in bytes.
     93+static constexpr size_t MAX_TX_INDEX_CACHE{1024 << 20};
     94+//! Max memory allocated to all block filter index caches combined in bytes.
     95+static constexpr size_t MAX_FILTER_INDEX_CACHE{1024 << 20};
     96 
     97 namespace node {
     98 std::optional<CacheSizes> CalculateCacheSizes(const ArgsManager& args, size_t n_indexes)
     99 {
    100-    int64_t db_cache = args.GetIntArg("-dbcache", DEFAULT_DB_CACHE);
    101-    if (static_cast<uint64_t>(db_cache << 20) > std::numeric_limits<size_t>::max()) {
    102-        LogWarning("Cannot allocate more than %d MiB in total for db caches.", static_cast<double>(std::numeric_limits<size_t>::max()) * (1.0 / 1024 / 1024));
    103+    // negative values are permitted, but interpreted as zero.
    104+    uint64_t db_cache_mib = std::max<int64_t>(0, args.GetIntArg("-dbcache", DEFAULT_DB_CACHE >> 20));
    105+    constexpr size_t max_mib = std::numeric_limits<size_t>::max() >> 20;
    106+    if (db_cache_mib > max_mib) {
    107+        LogWarning("Cannot allocate more than %d MiB for db caches.", max_mib);
    108         return std::nullopt;
    109     }
    110-
    111-    // negative values are permitted, but interpreted as zero.
    112-    db_cache = std::max(int64_t{0}, db_cache);
    113-    size_t total_cache = std::max(MiBToBytes(db_cache), MiBToBytes(MIN_DB_CACHE));
    114+    size_t total_cache{std::max(static_cast<size_t>(db_cache_mib << 20), MIN_DB_CACHE)};
    115 
    116     IndexCacheSizes index_sizes;
    117-    index_sizes.tx_index = std::min(total_cache / 8, args.GetBoolArg("-txindex", DEFAULT_TXINDEX) ? MiBToBytes(MAX_TX_INDEX_CACHE) : 0);
    118+    index_sizes.tx_index = std::min(total_cache / 8, args.GetBoolArg("-txindex", DEFAULT_TXINDEX) ? MAX_TX_INDEX_CACHE : 0);
    119     total_cache -= index_sizes.tx_index;
    120     index_sizes.filter_index = 0;
    121     if (n_indexes > 0) {
    122-        int64_t max_cache = std::min(total_cache / 8, MiBToBytes(MAX_FILTER_INDEX_CACHE));
    123+        int64_t max_cache = std::min(total_cache / 8, MAX_FILTER_INDEX_CACHE);
    124         index_sizes.filter_index = max_cache / n_indexes;
    125         total_cache -= index_sizes.filter_index * n_indexes;
    126     }
    127diff --git a/src/node/caches.h b/src/node/caches.h
    128index fd6b8b4ab8..f52fefd766 100644
    129--- a/src/node/caches.h
    130+++ b/src/node/caches.h
    131@@ -13,10 +13,10 @@
    132 
    133 class ArgsManager;
    134 
    135-//! min. -dbcache (MiB)
    136-static constexpr int64_t MIN_DB_CACHE{4};
    137-//! -dbcache default (MiB)
    138-static constexpr int64_t DEFAULT_DB_CACHE{DEFAULT_KERNEL_CACHE};
    139+//! min. -dbcache (bytes)
    140+static constexpr size_t MIN_DB_CACHE{4 << 20};
    141+//! -dbcache default (bytes)
    142+static constexpr size_t DEFAULT_DB_CACHE{DEFAULT_KERNEL_CACHE};
    143 
    144 namespace node {
    145 struct IndexCacheSizes {
    146diff --git a/src/qt/optionsdialog.cpp b/src/qt/optionsdialog.cpp
    147index 76b7852109..2cea1bf6ab 100644
    148--- a/src/qt/optionsdialog.cpp
    149+++ b/src/qt/optionsdialog.cpp
    150@@ -95,7 +95,7 @@ OptionsDialog::OptionsDialog(QWidget* parent, bool enableWallet)
    151     ui->verticalLayout->setStretchFactor(ui->tabWidget, 1);
    152 
    153     /* Main elements init */
    154-    ui->databaseCache->setRange(MIN_DB_CACHE, std::numeric_limits<int>::max());
    155+    ui->databaseCache->setRange(MIN_DB_CACHE >> 20, std::numeric_limits<int>::max());
    156     ui->threadsScriptVerif->setMinimum(-GetNumCores());
    157     ui->threadsScriptVerif->setMaximum(MAX_SCRIPTCHECK_THREADS);
    158     ui->pruneWarning->setVisible(false);
    159diff --git a/src/qt/optionsmodel.cpp b/src/qt/optionsmodel.cpp
    160index 2c6f13dc6d..5ff22b0f4c 100644
    161--- a/src/qt/optionsmodel.cpp
    162+++ b/src/qt/optionsmodel.cpp
    163@@ -470,7 +470,7 @@ QVariant OptionsModel::getOption(OptionID option, const std::string& suffix) con
    164                suffix.empty()          ? getOption(option, "-prev") :
    165                                          DEFAULT_PRUNE_TARGET_GB;
    166     case DatabaseCache:
    167-        return qlonglong(SettingToInt(setting(), DEFAULT_DB_CACHE));
    168+        return qlonglong(SettingToInt(setting(), DEFAULT_DB_CACHE >> 20));
    169     case ThreadsScriptVerif:
    170         return qlonglong(SettingToInt(setting(), DEFAULT_SCRIPTCHECK_THREADS));
    171     case Listen:
    172@@ -733,7 +733,7 @@ void OptionsModel::checkAndMigrate()
    173         // see [#8273](/bitcoin-bitcoin/8273/)
    174         // force people to upgrade to the new value if they are using 100MB
    175         if (settingsVersion < 130000 && settings.contains("nDatabaseCache") && settings.value("nDatabaseCache").toLongLong() == 100)
    176-            settings.setValue("nDatabaseCache", (qint64)DEFAULT_DB_CACHE);
    177+            settings.setValue("nDatabaseCache", (qint64)DEFAULT_DB_CACHE >> 20);
    178 
    179         settings.setValue(strSettingsVersionKey, CLIENT_VERSION);
    180     }
    

    Note: implementing something like this diff should also address all the other comments in this review.


    TheCharlatan commented at 3:28 pm on January 8, 2025:

    allowing compile-time guarantees that we’re not overflowing the system’s size_t size.

    I’m not sure that is actually true. Does this actually guard against any possible overflow?


    stickies-v commented at 3:51 pm on January 8, 2025:

    I’m not sure what the guarantees are, but at least on my machine, with static constexpr size_t MAX_FILTER_INDEX_CACHE{18446744073709551616}; (i.e. max + 1 on 64 bit) compilation fails with:

    0../../src/node/caches.cpp:22:48: error: integer literal is too large to be represented in any integer type
    1static constexpr size_t MAX_FILTER_INDEX_CACHE{18446744073709551616};
    2                                               ^
    31 error generated.
    

    Similarly, for static constexpr uint8_t TEST_8{266}; it fails with:

     0../../src/node/caches.cpp:23:33: error: constant expression evaluates to 266 which cannot be narrowed to type 'uint8_t' (aka 'unsigned char') [-Wc++11-narrowing]
     1static constexpr uint8_t TEST_8{266};
     2                                ^~~
     3../../src/node/caches.cpp:23:33: note: insert an explicit cast to silence this issue
     4static constexpr uint8_t TEST_8{266};
     5                                ^~~
     6                                static_cast<uint8_t>( )
     7../../src/node/caches.cpp:23:33: warning: implicit conversion from 'int' to 'const uint8_t' (aka 'const unsigned char') changes value from 266 to 10 [-Wconstant-conversion]
     8static constexpr uint8_t TEST_8{266};
     9                               ~^~~
    101 warning and 1 error generated.
    

    TheCharlatan commented at 5:29 pm on January 8, 2025:

    I’m more concerned how the shift will be cast. E.g. a potential future bump to 3000 MiB would error:

    0error: constant expression evaluates to -1149239296 which cannot be narrowed to type 'size_t' (aka 'unsigned long') [-Wc++11-narrowing]
    1   16 | static constexpr size_t DEFAULT_KERNEL_CACHE{3000 << 20};
    

    so to make this future proof, I think a function interpreting the values properly, or more ugly casts are required.


    ryanofsky commented at 6:42 pm on January 8, 2025:

    re: #31483 (review)

    and adds a test case

    I don’t think I see a new test case added in the diff. Is it missing?

    Another way to make this testable without adding interface complexity would just be to throw std::overflow_error


    stickies-v commented at 11:11 am on January 9, 2025:

    Hmm, you raise a good point. I see two alternatives:

    1. The low-code solution: when values get problematic (which is pretty infrequent, and constant), write them as one of follows:
    0static constexpr size_t DEFAULT_KERNEL_CACHE{3145728000};
    1static_assert(DEFAULT_KERNEL_CACHE >> 20 == 3000);
    

    or

    0static constexpr size_t DEFAULT_KERNEL_CACHE{size_t{3000} << 20};
    1static_assert(DEFAULT_KERNEL_CACHE >> 20 == 3000);
    
    1. instead of adding a MiBToBytes function, add a template <typename T> constexpr T CheckedLeftShift(T value, unsigned shift) function to util/overflow.h which can more generally be re-used?
     0static constexpr uint32_t DEFAULT_KERNEL_CACHE{CheckedLeftShift<uint32_t>(3000, 20)};
     1
     2...
     3
     4template <typename T>
     5constexpr T CheckedLeftShift(T value, unsigned shift)
     6{
     7    static_assert(std::is_unsigned_v<T>, 
     8                  "CheckedLeftShift requires an unsigned type");
     9
    10    if (shift >= std::numeric_limits<T>::digits) {
    11        throw std::overflow_error("shift out of range");
    12    }
    13
    14    if (value > (std::numeric_limits<T>::max() >> shift)) {
    15        throw std::overflow_error("shift would overflow");
    16    }
    17
    18    return static_cast<T>(value << shift);
    19}
    

    stickies-v commented at 11:13 am on January 9, 2025:

    I don’t think I see a new test case added in the diff. Is it missing?

    Snap, they weren’t included in my git diff because they were added in a new file, and I didn’t notice, sorry. I see @TheCharlatan already added a test case in his latest push.


    TheCharlatan commented at 1:01 pm on January 9, 2025:
    Are there places where this would be re-used? I’d be happy to add something more generic if there is demand for it.

    ryanofsky commented at 3:26 pm on January 9, 2025:

    re: #31483 (review)

    Are there places where this would be re-used? I’d be happy to add something more generic if there is demand for it.

    FWIW, all of the approaches discussed in this thread seem fine to me. If I were writing this PR, I think I would drop the MiBToBytes function and just write the constants as byte counts:

    0static constexpr size_t MAX_BLOCK_DB_CACHE{2 << 20};
    1static constexpr size_t MAX_COINS_DB_CACHE{8 << 20};
    

    Then when converting the -dbcache value from MiB to bytes I would just cap the value if it would overflow. (I might consider adding a SaturatingLeftShift function to overflow.h to do that, but that would probably be overkill).


    TheCharlatan commented at 7:51 am on January 10, 2025:
    @hodlinator since you initially proposed the current solution, do you have any fresh opinions here?

    stickies-v commented at 11:15 am on January 10, 2025:

    Are there places where this would be re-used?

    I can’t quite find one - it’s also a bit difficult to search for given how much << is used for streams too. I agree there’s no need to generalize too early, but given how similar the functionality is I still have a preference for the more generic CheckedLeftShift() and/or SaturatingLeftShift() as suggested by @ryanofsky. But, I’m okay with either approach.

    I think I would drop the MiBToBytes function and just write the constants as byte counts:

    I agree (+ with everything in your comment).


    stickies-v commented at 8:14 pm on January 10, 2025:

    Okay I’m sorry I can’t seem to just let it go but here’s another approach. It adds SaturatingLeftShift and CheckedLeftShift() util/overflow.h functionality (+tests, pinky promise), a _MiB string literal to allow remaining sizes to be converted to size_t bytes with concise notation and compile-time overflow guarantees (🤞). It also incorporates @ryanofsky’s suggestion to just saturate -dbcache values instead of throwing. It’s not necessary for my suggestion but I think it’s a good approach and aligns well with the SaturatingLeftShift() helper fn.

      0diff --git a/src/bitcoin-chainstate.cpp b/src/bitcoin-chainstate.cpp
      1index 82bda4b22b..51c482607a 100644
      2--- a/src/bitcoin-chainstate.cpp
      3+++ b/src/bitcoin-chainstate.cpp
      4@@ -26,7 +26,6 @@
      5 #include <random.h>
      6 #include <script/sigcache.h>
      7 #include <util/chaintype.h>
      8-#include <util/byte_conversion.h>
      9 #include <util/fs.h>
     10 #include <util/signalinterrupt.h>
     11 #include <util/task_runner.h>
     12@@ -124,7 +123,7 @@ int main(int argc, char* argv[])
     13     util::SignalInterrupt interrupt;
     14     ChainstateManager chainman{interrupt, chainman_opts, blockman_opts};
     15 
     16-    kernel::CacheSizes cache_sizes{MiBToBytes(DEFAULT_KERNEL_CACHE)};
     17+    kernel::CacheSizes cache_sizes{DEFAULT_KERNEL_CACHE};
     18     node::ChainstateLoadOptions options;
     19     auto [status, error] = node::LoadChainstate(chainman, cache_sizes, options);
     20     if (status != node::ChainstateLoadStatus::SUCCESS) {
     21diff --git a/src/init.cpp b/src/init.cpp
     22index 1ae968760d..847facad87 100644
     23--- a/src/init.cpp
     24+++ b/src/init.cpp
     25@@ -489,7 +489,7 @@ void SetupServerArgs(ArgsManager& argsman, bool can_listen_ipc)
     26     argsman.AddArg("-conf=<file>", strprintf("Specify path to read-only configuration file. Relative paths will be prefixed by datadir location (only useable from command line, not configuration file) (default: %s)", BITCOIN_CONF_FILENAME), ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);
     27     argsman.AddArg("-datadir=<dir>", "Specify data directory", ArgsManager::ALLOW_ANY | ArgsManager::DISALLOW_NEGATION, OptionsCategory::OPTIONS);
     28     argsman.AddArg("-dbbatchsize", strprintf("Maximum database write batch size in bytes (default: %u)", nDefaultDbBatchSize), ArgsManager::ALLOW_ANY | ArgsManager::DEBUG_ONLY, OptionsCategory::OPTIONS);
     29-    argsman.AddArg("-dbcache=<n>", strprintf("Maximum database cache size <n> MiB (minimum %d, default: %d). Make sure you have enough RAM. In addition, unused memory allocated to the mempool is shared with this cache (see -maxmempool).", MIN_DB_CACHE, DEFAULT_DB_CACHE), ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);
     30+    argsman.AddArg("-dbcache=<n>", strprintf("Maximum database cache size <n> MiB (minimum %d, default: %d). Make sure you have enough RAM. In addition, unused memory allocated to the mempool is shared with this cache (see -maxmempool).", MIN_DB_CACHE >> 20, DEFAULT_DB_CACHE >> 20), ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);
     31     argsman.AddArg("-includeconf=<file>", "Specify additional configuration file, relative to the -datadir path (only useable from configuration file, not command line)", ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);
     32     argsman.AddArg("-allowignoredconf", strprintf("For backwards compatibility, treat an unused %s file in the datadir as a warning, not an error.", BITCOIN_CONF_FILENAME), ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);
     33     argsman.AddArg("-loadblock=<file>", "Imports blocks from external file on startup", ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);
     34@@ -1179,7 +1179,7 @@ static ChainstateLoadResult InitAndLoadChainstate(
     35     NodeContext& node,
     36     bool do_reindex,
     37     const bool do_reindex_chainstate,
     38-    CacheSizes& cache_sizes,
     39+    const CacheSizes& cache_sizes,
     40     const ArgsManager& args)
     41 {
     42     const CChainParams& chainparams = Params();
     43@@ -1605,11 +1605,7 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
     44     ReadNotificationArgs(args, kernel_notifications);
     45 
     46     // cache size calculations
     47-    const auto cache_sizes = CalculateCacheSizes(args, g_enabled_filter_types.size());
     48-    if (!cache_sizes) {
     49-        return InitError(_("Failed to calculate cache sizes. Try lowering the -dbcache value."));
     50-    }
     51-    auto [index_cache_sizes, kernel_cache_sizes] = *cache_sizes;
     52+    const auto [index_cache_sizes, kernel_cache_sizes] = CalculateCacheSizes(args, g_enabled_filter_types.size());
     53 
     54     LogInfo("Cache configuration:");
     55     LogInfo("* Using %.1f MiB for block index database", kernel_cache_sizes.block_tree_db * (1.0 / 1024 / 1024));
     56diff --git a/src/kernel/caches.h b/src/kernel/caches.h
     57index 9ee5a31e0a..4414d1a123 100644
     58--- a/src/kernel/caches.h
     59+++ b/src/kernel/caches.h
     60@@ -5,16 +5,16 @@
     61 #ifndef BITCOIN_KERNEL_CACHES_H
     62 #define BITCOIN_KERNEL_CACHES_H
     63 
     64-#include <util/byte_conversion.h>
     65+#include <util/storage.h>
     66 
     67 #include <algorithm>
     68 
     69-//! Suggested default amount of cache reserved for the kernel (MiB)
     70-static constexpr int64_t DEFAULT_KERNEL_CACHE{450};
     71+//! Suggested default amount of cache reserved for the kernel (bytes)
     72+static constexpr size_t DEFAULT_KERNEL_CACHE{450_MiB};
     73 //! Max memory allocated to block tree DB specific cache (bytes)
     74-static constexpr size_t MAX_BLOCK_DB_CACHE{MiBToBytes(2)};
     75+static constexpr size_t MAX_BLOCK_DB_CACHE{2_MiB};
     76 //! Max memory allocated to coin DB specific cache (bytes)
     77-static constexpr size_t MAX_COINS_DB_CACHE{MiBToBytes(8)};
     78+static constexpr size_t MAX_COINS_DB_CACHE{8_MiB};
     79 
     80 namespace kernel {
     81 struct CacheSizes {
     82diff --git a/src/node/caches.cpp b/src/node/caches.cpp
     83index 41a8971665..94ad5bbe61 100644
     84--- a/src/node/caches.cpp
     85+++ b/src/node/caches.cpp
     86@@ -8,34 +8,27 @@
     87 #include <index/txindex.h>
     88 #include <kernel/caches.h>
     89 #include <logging.h>
     90-#include <util/byte_conversion.h>
     91+#include <util/overflow.h>
     92+#include <util/storage.h>
     93 
     94 #include <algorithm>
     95-#include <optional>
     96 #include <stdexcept>
     97 #include <string>
     98 
     99 // Unlike for the UTXO database, for the txindex scenario the leveldb cache make
    100 // a meaningful difference: [#8273 (comment)](/bitcoin-bitcoin/8273/#issuecomment-229601991)
    101 //! Max memory allocated to tx index DB specific cache in bytes.
    102-static constexpr size_t MAX_TX_INDEX_CACHE{MiBToBytes(1024)};
    103+static constexpr size_t MAX_TX_INDEX_CACHE{1024_MiB};
    104 //! Max memory allocated to all block filter index caches combined in bytes.
    105-static constexpr size_t MAX_FILTER_INDEX_CACHE{MiBToBytes(1024)};
    106+static constexpr size_t MAX_FILTER_INDEX_CACHE{1024_MiB};
    107 
    108 namespace node {
    109-std::optional<CacheSizes> CalculateCacheSizes(const ArgsManager& args, size_t n_indexes)
    110+CacheSizes CalculateCacheSizes(const ArgsManager& args, size_t n_indexes)
    111 {
    112-    int64_t db_cache = args.GetIntArg("-dbcache", DEFAULT_DB_CACHE);
    113-
    114-    // negative values are permitted, but interpreted as zero.
    115-    db_cache = std::max(int64_t{0}, db_cache);
    116-
    117-    size_t total_cache = 0;
    118-    try {
    119-        total_cache = std::max(MiBToBytes(db_cache), MiBToBytes(MIN_DB_CACHE));
    120-    } catch (const std::out_of_range&) {
    121-        LogError("Cannot allocate more than %d MiB in total for db caches.", std::numeric_limits<size_t>::max() >> 20);
    122-        return std::nullopt;
    123+    // Total cache size in MiB, floored by MIN_DB_CACHE and capped by max size_t value
    124+    size_t total_cache{DEFAULT_DB_CACHE};
    125+    if (auto db_cache = args.GetIntArg("-dbcache")) {
    126+        total_cache = std::max(MIN_DB_CACHE, SaturatingLeftShift<size_t>(*db_cache, 20));
    127     }
    128 
    129     IndexCacheSizes index_sizes;
    130@@ -47,6 +40,6 @@ std::optional<CacheSizes> CalculateCacheSizes(const ArgsManager& args, size_t n_
    131         index_sizes.filter_index = max_cache / n_indexes;
    132         total_cache -= index_sizes.filter_index * n_indexes;
    133     }
    134-    return {{index_sizes, kernel::CacheSizes{total_cache}}};
    135+    return {index_sizes, kernel::CacheSizes{total_cache}};
    136 }
    137 } // namespace node
    138diff --git a/src/node/caches.h b/src/node/caches.h
    139index b4f2e3d516..aab7871e93 100644
    140--- a/src/node/caches.h
    141+++ b/src/node/caches.h
    142@@ -6,17 +6,16 @@
    143 #define BITCOIN_NODE_CACHES_H
    144 
    145 #include <kernel/caches.h>
    146-#include <util/byte_conversion.h>
    147+#include <util/storage.h>
    148 
    149 #include <cstddef>
    150-#include <optional>
    151 
    152 class ArgsManager;
    153 
    154-//! min. -dbcache (MiB)
    155-static constexpr int64_t MIN_DB_CACHE{4};
    156-//! -dbcache default (MiB)
    157-static constexpr int64_t DEFAULT_DB_CACHE{DEFAULT_KERNEL_CACHE};
    158+//! min. -dbcache (bytes)
    159+static constexpr size_t MIN_DB_CACHE{4_MiB};
    160+//! -dbcache default (bytes)
    161+static constexpr size_t DEFAULT_DB_CACHE{DEFAULT_KERNEL_CACHE};
    162 
    163 namespace node {
    164 struct IndexCacheSizes {
    165@@ -27,7 +26,7 @@ struct CacheSizes {
    166     IndexCacheSizes index;
    167     kernel::CacheSizes kernel;
    168 };
    169-std::optional<CacheSizes> CalculateCacheSizes(const ArgsManager& args, size_t n_indexes = 0);
    170+CacheSizes CalculateCacheSizes(const ArgsManager& args, size_t n_indexes = 0);
    171 } // namespace node
    172 
    173 #endif // BITCOIN_NODE_CACHES_H
    174diff --git a/src/qt/optionsdialog.cpp b/src/qt/optionsdialog.cpp
    175index 76b7852109..2cea1bf6ab 100644
    176--- a/src/qt/optionsdialog.cpp
    177+++ b/src/qt/optionsdialog.cpp
    178@@ -95,7 +95,7 @@ OptionsDialog::OptionsDialog(QWidget* parent, bool enableWallet)
    179     ui->verticalLayout->setStretchFactor(ui->tabWidget, 1);
    180 
    181     /* Main elements init */
    182-    ui->databaseCache->setRange(MIN_DB_CACHE, std::numeric_limits<int>::max());
    183+    ui->databaseCache->setRange(MIN_DB_CACHE >> 20, std::numeric_limits<int>::max());
    184     ui->threadsScriptVerif->setMinimum(-GetNumCores());
    185     ui->threadsScriptVerif->setMaximum(MAX_SCRIPTCHECK_THREADS);
    186     ui->pruneWarning->setVisible(false);
    187diff --git a/src/qt/optionsmodel.cpp b/src/qt/optionsmodel.cpp
    188index 2c6f13dc6d..5ff22b0f4c 100644
    189--- a/src/qt/optionsmodel.cpp
    190+++ b/src/qt/optionsmodel.cpp
    191@@ -470,7 +470,7 @@ QVariant OptionsModel::getOption(OptionID option, const std::string& suffix) con
    192                suffix.empty()          ? getOption(option, "-prev") :
    193                                          DEFAULT_PRUNE_TARGET_GB;
    194     case DatabaseCache:
    195-        return qlonglong(SettingToInt(setting(), DEFAULT_DB_CACHE));
    196+        return qlonglong(SettingToInt(setting(), DEFAULT_DB_CACHE >> 20));
    197     case ThreadsScriptVerif:
    198         return qlonglong(SettingToInt(setting(), DEFAULT_SCRIPTCHECK_THREADS));
    199     case Listen:
    200@@ -733,7 +733,7 @@ void OptionsModel::checkAndMigrate()
    201         // see [#8273](/bitcoin-bitcoin/8273/)
    202         // force people to upgrade to the new value if they are using 100MB
    203         if (settingsVersion < 130000 && settings.contains("nDatabaseCache") && settings.value("nDatabaseCache").toLongLong() == 100)
    204-            settings.setValue("nDatabaseCache", (qint64)DEFAULT_DB_CACHE);
    205+            settings.setValue("nDatabaseCache", (qint64)DEFAULT_DB_CACHE >> 20);
    206 
    207         settings.setValue(strSettingsVersionKey, CLIENT_VERSION);
    208     }
    209diff --git a/src/test/util/setup_common.h b/src/test/util/setup_common.h
    210index 7d7c7b7cd4..33ad258457 100644
    211--- a/src/test/util/setup_common.h
    212+++ b/src/test/util/setup_common.h
    213@@ -104,7 +104,7 @@ struct BasicTestingSetup {
    214  * initialization behaviour.
    215  */
    216 struct ChainTestingSetup : public BasicTestingSetup {
    217-    kernel::CacheSizes m_kernel_cache_sizes{node::CalculateCacheSizes(m_args).value().kernel};
    218+    kernel::CacheSizes m_kernel_cache_sizes{node::CalculateCacheSizes(m_args).kernel};
    219     bool m_coins_db_in_memory{true};
    220     bool m_block_tree_db_in_memory{true};
    221     std::function<void()> m_make_chainman{};
    222diff --git a/src/test/util_tests.cpp b/src/test/util_tests.cpp
    223index e8e0abbda6..dbe1d76ca8 100644
    224--- a/src/test/util_tests.cpp
    225+++ b/src/test/util_tests.cpp
    226@@ -13,12 +13,12 @@
    227 #include <test/util/setup_common.h>
    228 #include <uint256.h>
    229 #include <util/bitdeque.h>
    230-#include <util/byte_conversion.h>
    231 #include <util/fs.h>
    232 #include <util/fs_helpers.h>
    233 #include <util/moneystr.h>
    234 #include <util/overflow.h>
    235 #include <util/readwritefile.h>
    236+#include <util/storage.h>
    237 #include <util/strencodings.h>
    238 #include <util/string.h>
    239 #include <util/time.h>
    240@@ -141,19 +141,6 @@ BOOST_AUTO_TEST_CASE(util_criticalsection)
    241     } while(0);
    242 }
    243 
    244-BOOST_AUTO_TEST_CASE(byte_conversion)
    245-{
    246-    // maximum allowed value in MiB
    247-    const int64_t max_cache = std::numeric_limits<size_t>::max() >> 20;
    248-
    249-    BOOST_CHECK_EXCEPTION(MiBToBytes(-1), std::out_of_range, HasReason("Value may not be negative."));
    250-    BOOST_CHECK_EXCEPTION(MiBToBytes(std::numeric_limits<int64_t>::max()), std::out_of_range, HasReason("Conversion to bytes of"));
    251-    BOOST_CHECK_EXCEPTION(MiBToBytes(max_cache + 1), std::out_of_range, HasReason("Conversion to bytes of"));
    252-
    253-    BOOST_CHECK_EQUAL(MiBToBytes(0), 0);
    254-    BOOST_CHECK_EQUAL(MiBToBytes(max_cache), max_cache << 20);
    255-}
    256-
    257 constexpr char HEX_PARSE_INPUT[] = "04678afdb0fe5548271967f1a67130b7105cd6a828e03909a67962e0ea1f61deb649f6bc3f4cef38c4f35504e51ec112de5c384df7ba0b8d578a4c702b6bf11d5f";
    258 constexpr uint8_t HEX_PARSE_OUTPUT[] = {
    259     0x04, 0x67, 0x8a, 0xfd, 0xb0, 0xfe, 0x55, 0x48, 0x27, 0x19, 0x67, 0xf1, 0xa6, 0x71, 0x30, 0xb7,
    260@@ -1892,4 +1879,63 @@ BOOST_AUTO_TEST_CASE(clearshrink_test)
    261     }
    262 }
    263 
    264+template <typename T>
    265+void TestCheckedLeftShift()
    266+{
    267+    BOOST_TEST_CONTEXT("Testing CheckedLeftShift with " << typeid(T).name()) {
    268+        // Basic operations
    269+        BOOST_CHECK_EQUAL(CheckedLeftShift<T>(0, 1), 0);
    270+        BOOST_CHECK_EQUAL(CheckedLeftShift<T>(1, 1), 2);
    271+        BOOST_CHECK_EQUAL(CheckedLeftShift<T>(2, 2), 8);
    272+
    273+        // Negative input
    274+        BOOST_CHECK(!CheckedLeftShift<T>(-1, 1));
    275+
    276+        // Overflow cases
    277+        BOOST_CHECK(!CheckedLeftShift<T>((std::numeric_limits<T>::max() >> 1) + 1, 1));
    278+        BOOST_CHECK(!CheckedLeftShift<T>(std::numeric_limits<T>::max(), 1));
    279+    }
    280+}
    281+
    282+template <typename T>
    283+void TestSaturatingLeftShift()
    284+{
    285+    BOOST_TEST_CONTEXT("Testing SaturatingLeftShift with " << typeid(T).name()) {
    286+        // Basic operations
    287+        BOOST_CHECK_EQUAL(SaturatingLeftShift<T>(0, 1), 0);
    288+        BOOST_CHECK_EQUAL(SaturatingLeftShift<T>(1, 1), 2);
    289+        BOOST_CHECK_EQUAL(SaturatingLeftShift<T>(2, 2), 8);
    290+
    291+        // Negative input
    292+        BOOST_CHECK_EQUAL(SaturatingLeftShift<T>(-1, 1), 0);
    293+
    294+        // Saturation cases
    295+        BOOST_CHECK_EQUAL(SaturatingLeftShift<T>((std::numeric_limits<T>::max() >> 1) + 1, 1), std::numeric_limits<T>::max());
    296+        BOOST_CHECK_EQUAL(SaturatingLeftShift<T>(std::numeric_limits<T>::max(), 1), std::numeric_limits<T>::max());
    297+    }
    298+}
    299+
    300+BOOST_AUTO_TEST_CASE(checked_left_shift_test)
    301+{
    302+    TestCheckedLeftShift<uint8_t>();
    303+    TestCheckedLeftShift<size_t>();
    304+    TestCheckedLeftShift<uint64_t>();
    305+}
    306+
    307+BOOST_AUTO_TEST_CASE(saturating_left_shift_test)
    308+{
    309+    TestSaturatingLeftShift<uint8_t>();
    310+    TestSaturatingLeftShift<size_t>();
    311+    TestSaturatingLeftShift<uint64_t>();
    312+}
    313+
    314+BOOST_AUTO_TEST_CASE(mib_string_literal_test)
    315+{
    316+    BOOST_CHECK_EQUAL(0_MiB, 0);
    317+    BOOST_CHECK_EQUAL(1_MiB, 1024 * 1024);
    318+
    319+    constexpr size_t max_safe_mib = std::numeric_limits<size_t>::max() / (1024 * 1024);
    320+    BOOST_CHECK_EQUAL(max_safe_mib * (1024 * 1024), max_safe_mib * 1_MiB);
    321+}
    322+
    323 BOOST_AUTO_TEST_SUITE_END()
    324diff --git a/src/util/byte_conversion.h b/src/util/byte_conversion.h
    325deleted file mode 100644
    326index bbdad04bff..0000000000
    327--- a/src/util/byte_conversion.h
    328+++ /dev/null
    329@@ -1,26 +0,0 @@
    330-// Copyright (c) 2025-present The Bitcoin Core developers
    331-// Distributed under the MIT software license, see the accompanying
    332-// file COPYING or http://www.opensource.org/licenses/mit-license.php.
    333-
    334-#ifndef BITCOIN_UTIL_BYTE_CONVERSION_H
    335-#define BITCOIN_UTIL_BYTE_CONVERSION_H
    336-
    337-#include <tinyformat.h>
    338-
    339-#include <cstdint>
    340-#include <limits>
    341-#include <stdexcept>
    342-
    343-//! Guard against truncation of values before converting.
    344-constexpr size_t MiBToBytes(int64_t mib)
    345-{
    346-    if (mib < 0) {
    347-        throw std::out_of_range("Value may not be negative.");
    348-    }
    349-    if (static_cast<uint64_t>(mib) > std::numeric_limits<size_t>::max() >> 20) {
    350-        throw std::out_of_range(strprintf("Conversion to bytes of %d does not fit into size_t with maximum value %d.", mib, std::numeric_limits<size_t>::max()));
    351-    }
    352-    return static_cast<size_t>(mib) << 20;
    353-}
    354-
    355-#endif // BITCOIN_UTIL_BYTE_CONVERSION_H
    356diff --git a/src/util/overflow.h b/src/util/overflow.h
    357index 7e0cce6c27..bb5277db28 100644
    358--- a/src/util/overflow.h
    359+++ b/src/util/overflow.h
    360@@ -47,4 +47,42 @@ template <class T>
    361     return i + j;
    362 }
    363 
    364+/**
    365+ * [@brief](/bitcoin-bitcoin/contributor/brief/) Left bit shift with overflow checking.
    366+ * [@param](/bitcoin-bitcoin/contributor/param/) i The input value to be left shifted.
    367+ * [@param](/bitcoin-bitcoin/contributor/param/) shift The number of bits to left shift.
    368+ * [@return](/bitcoin-bitcoin/contributor/return/) The result of the left shift, or std::nullopt in case of
    369+ *         overflow or negative input value.
    370+ */
    371+template <class Output, class Input>
    372+constexpr std::optional<Output> CheckedLeftShift(Input i, unsigned shift) noexcept
    373+    requires std::is_unsigned_v<Output>
    374+{
    375+    if constexpr (std::is_signed_v<Input>) {
    376+        if (i < 0) return std::nullopt;
    377+    }
    378+    if (std::make_unsigned_t<Input>(i) > (std::numeric_limits<Output>::max() >> shift)) {
    379+        return std::nullopt;
    380+    }
    381+    return i << shift;
    382+}
    383+
    384+/**
    385+ * [@brief](/bitcoin-bitcoin/contributor/brief/) Left bit shift with safe minimum and maximum values.
    386+ * [@param](/bitcoin-bitcoin/contributor/param/) i The input value to be left shifted.
    387+ * [@param](/bitcoin-bitcoin/contributor/param/) shift The number of bits to left shift.
    388+ * [@return](/bitcoin-bitcoin/contributor/return/) The result of the left shift, with the return value clamped
    389+ *         between zero and the maximum Output value if overflow occurs.
    390+ */
    391+template <class Output, class Input>
    392+constexpr Output SaturatingLeftShift(Input i, unsigned shift) noexcept
    393+    requires std::is_unsigned_v<Output>
    394+{
    395+    auto default_value{std::numeric_limits<Output>::max()};
    396+    if constexpr (std::is_signed_v<Input>) {
    397+        if (i < 0) default_value = 0;
    398+    }
    399+    return CheckedLeftShift<Output>(i, shift).value_or(default_value);
    400+}
    401+
    402 #endif // BITCOIN_UTIL_OVERFLOW_H
    403diff --git a/src/util/storage.h b/src/util/storage.h
    404new file mode 100644
    405index 0000000000..042875510b
    406--- /dev/null
    407+++ b/src/util/storage.h
    408@@ -0,0 +1,24 @@
    409+// Copyright (c) 2025-present The Bitcoin Core developers
    410+// Distributed under the MIT software license, see the accompanying
    411+// file COPYING or http://www.opensource.org/licenses/mit-license.php.
    412+
    413+#ifndef BITCOIN_UTIL_STORAGE_H
    414+#define BITCOIN_UTIL_STORAGE_H
    415+
    416+#include <util/overflow.h>
    417+
    418+#include <cstdint>
    419+#include <limits>
    420+#include <stdexcept>
    421+
    422+//! Overflow-safe conversion of MiB to bytes.
    423+constexpr size_t operator"" _MiB(unsigned long long mebibytes)
    424+{
    425+    auto bytes{CheckedLeftShift<size_t>(static_cast<size_t>(mebibytes), 20)};
    426+    if (!bytes) {
    427+        throw std::overflow_error("mebibytes cannot be converted to bytes");
    428+    }
    429+    return *bytes;
    430+}
    431+
    432+#endif // BITCOIN_UTIL_STORAGE_H
    

    hodlinator commented at 8:42 pm on January 10, 2025:

    @hodlinator since you initially proposed the current solution, do you have any fresh opinions here?

    When used with size_t constants, I get helpful GCC compiler errors when increasing the value of the constants (I guess this in part comes from C++ integer literals being signed 32-bit by default?):

    0/home/hodlinator/bitcoin/src/node/caches.cpp:23:53: error: narrowing conversion of -176160768 from int to size_t {aka long unsigned int} [-Wnarrowing]
    1   23 | static constexpr size_t MAX_FILTER_INDEX_CACHE{8024 << 20};
    2      |                                                ~~~~~^~~~~
    

    I don’t get such errors when shifting an int64_t constant and then truncating it into a size_t function argument. 10x-ed DEFAULT_KERNEL_CACHE:

    0static constexpr int64_t DEFAULT_KERNEL_CACHE{4500};
    

    Added this into caches.cpp/CalculateCacheSizes(), sending a too large value into size_t function argument:

    0kernel::CacheSizes cache_sizes{DEFAULT_KERNEL_CACHE << 20};
    

    Got no error/warning when compiling on CentOS (32-bit).


    The second block above makes me lean towards recommending keeping some kind of free function with checks for non-size_t constants. @stickies-v’s latest diff seems overall like a good direction, building on the pre-existing overflow.h.

  71. in src/node/caches.cpp:32 in 578654c686 outdated
    35-    sizes.filter_index = 0;
    36+    int64_t db_cache = args.GetIntArg("-dbcache", DEFAULT_DB_CACHE);
    37+    if (static_cast<uint64_t>(db_cache << 20) > std::numeric_limits<size_t>::max()) {
    38+        LogWarning("Cannot allocate more than %d MiB in total for db caches.", static_cast<double>(std::numeric_limits<size_t>::max()) * (1.0 / 1024 / 1024));
    39+        return std::nullopt;
    40+    }
    


    stickies-v commented at 1:50 pm on January 8, 2025:

    This check seems pretty broken to me. Left shifting a negative value is UB, and if db_cache is indeed too large it would already overflow in static_cast<uint64_t>(db_cache << 20).

    On my machine, it seems this whole block is compiled away entirely, I’m not getting any warnings for either negative or too large values.


    TheCharlatan commented at 2:45 pm on January 8, 2025:

    This is only relevant on machines where size_t is not a uint64_t, so I’m not surprised this is optimized away. I tested this by compiling for arm-linux-gnueabihf and running with:

    0qemu-arm -L /usr/arm-linux-gnueabihf build_arm_32/src/bitcoind -nowallet -dbcache=10000 -signet
    

    +1 for clamping the value range to zero before doing that check though.


    stickies-v commented at 3:36 pm on January 8, 2025:

    This is only relevant on machines where size_t is not a uint64_t, so I’m not surprised this is optimized away.

    It seems to me like this check is necessary on all platforms? E.g. on this branch, when I run with -dbcache=10000000000000000 on arm64 I get

    0../../src/kernel/caches.h:25 MiBToBytes: Assertion `std::countl_zero(static_cast<uint64_t>(mib)) >= 21' failed.
    

    In static_cast<uint64_t>(db_cache << 20) we’re left-shifting db_cache without checking that it’ll fit the uint64_t. I think my approach suggested in #31483 (review) addresses all of this quite nicely?


    ryanofsky commented at 6:49 pm on January 8, 2025:

    re: #31483 (review)

    Agree with stickies overflow should be fixed. I think an easy way would just be to check if db_cache > std::numeric_limits<size_t>::max() >> 20 though.

  72. stickies-v commented at 2:34 pm on January 8, 2025: contributor
    I like the increased checks for overflows on 32-bit (or other platforms), but I don’t think we need the new MiBToBytes() helper function for that and some of the new logic seems buggy to me. More detail on both in the comments.
  73. DrahtBot requested review from stickies-v on Jan 8, 2025
  74. in src/node/caches.cpp:1 in 19316eccaf outdated


    ryanofsky commented at 6:02 pm on January 8, 2025:

    In commit “init: Simplify coinsdb cache calculation” (19316eccaf93776c49b1870f8aa9a5fe57ec5f33)

    Didn’t notice this first time I reviewed this, but it would be good if commit subject included “refactor” and make it obvious this is not changing behavior. This simplification is only changing the way the calculation is performed, not changing the result of the calculation.

  75. in src/kernel/caches.h:22 in 41bc716488 outdated
    17+constexpr size_t MiBToBytes(int64_t mib)
    18+{
    19+    Assert(std::countl_zero(static_cast<uint64_t>(mib)) >= 21); // Ensure sign bit is unset + enough zeros to shift.
    20+    const int64_t bytes{mib << 20};
    21+    Assert(static_cast<uint64_t>(bytes) <= std::numeric_limits<size_t>::max());
    22+    return static_cast<size_t>(bytes);
    


    ryanofsky commented at 6:27 pm on January 8, 2025:

    In commit “kernel: Move kernel-specific cache size options to kernel” (41bc7164880ffd0782ff5882211c4fcadd656022)

    This seems like it is overthinking things. I think this should just be:

    0Assert(mib >= 0);
    1Assert(mib <= std::numeric_limits<size_t>::max() >> 20);
    2return mib << 20;
    
  76. ryanofsky commented at 6:51 pm on January 8, 2025: contributor
    Code review 578654c686e394d08bac7c3bcddbfd90b46eaa62. Looks pretty good, but there does appear to be a possible overflow bug #31483 (review) found by stickies that should be fixed
  77. DrahtBot requested review from ryanofsky on Jan 8, 2025
  78. [refactor] init: Simplify coinsdb cache calculation
    (total_cache / 4) + (1 << 23) is at least 8 MiB and nMaxCoinsDBCache is
    also 8 MiB, so the minimum between the two will always be
    nMaxCoinsDBCache. This is just a simplification and not changing the
    result of the calculation.
    
    Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
    8bd5f8a38c
  79. TheCharlatan force-pushed on Jan 9, 2025
  80. TheCharlatan force-pushed on Jan 9, 2025
  81. TheCharlatan force-pushed on Jan 9, 2025
  82. TheCharlatan marked this as a draft on Jan 9, 2025
  83. TheCharlatan force-pushed on Jan 9, 2025
  84. TheCharlatan marked this as ready for review on Jan 9, 2025
  85. DrahtBot added the label CI failed on Jan 9, 2025
  86. DrahtBot commented at 11:25 am on January 9, 2025: contributor

    🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/35365058175

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  87. TheCharlatan force-pushed on Jan 9, 2025
  88. TheCharlatan commented at 12:11 pm on January 9, 2025: contributor

    Thank you for the reviews @ryanofsky and @stickies-v , sorry for the many pushes here, pushed a stale branch by mistake.

    Updated 578654c686e394d08bac7c3bcddbfd90b46eaa62 -> 941a194c944826049f823858cb15c41963e4619a (kernel_cache_sizes_9 -> kernel_cache_sizes_10, compare)

    • Addressed @stickies-v’s comment, moved some MiB to bytes conversions to the constants themselves so we can do the conversion and bounds check at compile time.
    • Addressed @ryanofsky’s comment, added refactor clarification in the message of the commit cleaning up the cache calculation.
    • Addressed @ryanofsky’s comment, using the suggested simplified method for converting and checking the range of of the passed in value in MiBToBytes.
    • Added a separate commit for introducing the MiBToBytes utility in its own header and with its own unit test.
  89. TheCharlatan force-pushed on Jan 9, 2025
  90. TheCharlatan commented at 12:55 pm on January 9, 2025: contributor

    Updated 941a194c944826049f823858cb15c41963e4619a -> 82706e217f34d1f09cbd30dde6b8ae5ac0385f0a (kernel_cache_sizes_10 -> kernel_cache_sizes_11, compare)

    • Fix UBSan failure by avoiding undefined behaviour by casting to size_t.
  91. DrahtBot removed the label CI failed on Jan 9, 2025
  92. in src/node/caches.cpp:32 in 82706e217f outdated
    45+    try {
    46+        total_cache = std::max(MiBToBytes(db_cache), MiBToBytes(MIN_DB_CACHE));
    47+    } catch (const std::out_of_range&) {
    48+        LogError("Cannot allocate more than %d MiB in total for db caches.", std::numeric_limits<size_t>::max() >> 20);
    49+        return std::nullopt;
    50+    }
    


    ryanofsky commented at 3:01 pm on January 9, 2025:

    In commit “init: Use size_t consistently for cache sizes” (82706e217f34d1f09cbd30dde6b8ae5ac0385f0a)

    Code could be simplified to make it clear what is causing the new error:

     0--- a/src/node/caches.cpp
     1+++ b/src/node/caches.cpp
     2@@ -25,14 +25,10 @@ static constexpr size_t MAX_FILTER_INDEX_CACHE{MiBToBytes(1024)};
     3 namespace node {
     4 std::optional<CacheSizes> CalculateCacheSizes(const ArgsManager& args, size_t n_indexes)
     5 {
     6-    int64_t db_cache = args.GetIntArg("-dbcache", DEFAULT_DB_CACHE);
     7-
     8-    // negative values are permitted, but interpreted as zero.
     9-    db_cache = std::max(int64_t{0}, db_cache);
    10-
    11+    const int64_t db_cache = std::max(MIN_DB_CACHE, args.GetIntArg("-dbcache", DEFAULT_DB_CACHE));
    12     size_t total_cache = 0;
    13     try {
    14-        total_cache = std::max(MiBToBytes(db_cache), MiBToBytes(MIN_DB_CACHE));
    15+        total_cache = MiBToBytes(db_cache);
    16     } catch (const std::out_of_range&) {
    17         LogError("Cannot allocate more than %d MiB in total for db caches.", std::numeric_limits<size_t>::max() >> 20);
    18         return std::nullopt;
    

    Though I think we don’t need this error, could do something even simpler like:

    0// Total cache size in MiB, floored by MIN_DB_CACHE
    1const int64_t db_cache = std::max(MIN_DB_CACHE, args.GetIntArg("-dbcache", DEFAULT_DB_CACHE));
    2// Total cache size in bytes, capped by max size_t value
    3size_t total_cache = std::min<int64_t>(db_cache, std::numeric_limits<size_t>::max() >> 20) << 20;
    

    Current code or whatever you prefer seems fine though.


    hodlinator commented at 10:08 am on January 10, 2025:
    Thread #31483 (review): Agree it’s a bit verbose right now. I especially like the first suggestion in the diff.
  93. ryanofsky approved
  94. ryanofsky commented at 3:34 pm on January 9, 2025: contributor
    Code review ACK 82706e217f34d1f09cbd30dde6b8ae5ac0385f0a. Since last review avoided undefined -dbcache overflow behavior and changed the warning into an error. Also simplified and moved MiB conversion function and added test, and converted most constants to use byte sizes instead of MiB sizes (MIN_DB_CACHE still uses MiB not bytes, I think because that constant is not used by the kernel and is used by qt)
  95. DrahtBot requested review from hodlinator on Jan 9, 2025
  96. hodlinator approved
  97. hodlinator commented at 10:14 am on January 10, 2025: contributor

    re-ACK 82706e217f34d1f09cbd30dde6b8ae5ac0385f0a

    Mixing of MiB and bytes in constants is tolerable, and at least the units are documented.

    New unit tests look good to me.


    Nit: commit message in 82706e217f34d1f09cbd30dde6b8ae5ac0385f0a contains 3 ways of describing the number of bits:

    • “memory was allocated on 32bit platforms.”
    • “maximum size of a 32 bit unsigned”
    • “in behaviour on 32-bit systems.”
  98. in src/init.cpp:1612 in 82706e217f outdated
    1604@@ -1603,18 +1605,22 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
    1605     ReadNotificationArgs(args, kernel_notifications);
    1606 
    1607     // cache size calculations
    1608-    CacheSizes cache_sizes = CalculateCacheSizes(args, g_enabled_filter_types.size());
    1609+    const auto cache_sizes = CalculateCacheSizes(args, g_enabled_filter_types.size());
    1610+    if (!cache_sizes) {
    1611+        return InitError(_("Failed to calculate cache sizes. Try lowering the -dbcache value."));
    1612+    }
    1613+    auto [index_cache_sizes, kernel_cache_sizes] = *cache_sizes;
    


    stickies-v commented at 5:14 pm on January 10, 2025:
    nit: this could be const if you update InitAndLoadChainstate()’s cache_sizes parameter to const
  99. DrahtBot requested review from stickies-v on Jan 10, 2025
  100. TheCharlatan force-pushed on Jan 11, 2025
  101. TheCharlatan commented at 2:21 pm on January 11, 2025: contributor

    Thank you for the re-newed reviews and your suggestions @ryanofsky and @stickies-v. I hope we can settle on something this time round:

    Updated 82706e217f34d1f09cbd30dde6b8ae5ac0385f0a -> 45fe603f8fd94c171f5ce1a38bc20d273baa9b1b (kernel_cache_sizes_11 -> kernel_cache_sizes_12, compare)

    • Replaced MiBToBytes with the patch @stickies-v suggested, spread out over its relevant commits.
    • Added CheckedLeftShift, SaturatingLeftShift and _MiB string literal functions with relevant tests.
    • Addressed @ryanofsky’s comment, dropped the hard error if the requested amount of -dbcache does not fit in a size_t. Instead the maximum value is now capped at the maximum value of a size_t.
    • Removed the release note, since there is no significant change in behaviour anymore.
  102. TheCharlatan force-pushed on Jan 11, 2025
  103. DrahtBot added the label CI failed on Jan 11, 2025
  104. DrahtBot commented at 3:21 pm on January 11, 2025: contributor

    🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/35471739098

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  105. TheCharlatan force-pushed on Jan 11, 2025
  106. DrahtBot removed the label CI failed on Jan 11, 2025
  107. in src/util/byte_units.h:17 in cf20c1b9e0 outdated
    12+//! Overflow-safe conversion of MiB to bytes.
    13+constexpr size_t operator"" _MiB(unsigned long long mebibytes)
    14+{
    15+    auto bytes{CheckedLeftShift<size_t>(mebibytes, 20)};
    16+    if (!bytes) {
    17+        throw std::overflow_error("mebibytes could not be converted to bytes");
    


    hodlinator commented at 7:17 pm on January 11, 2025:

    nit: Give a hint at why?

    0        throw std::overflow_error("Too many mebibytes to fit into size_t bytes");
    
  108. in src/node/caches.cpp:27 in cf20c1b9e0 outdated
    31-    sizes.block_tree_db = std::min(nTotalCache / 8, nMaxBlockDBCache << 20);
    32-    nTotalCache -= sizes.block_tree_db;
    33-    sizes.tx_index = std::min(nTotalCache / 8, args.GetBoolArg("-txindex", DEFAULT_TXINDEX) ? nMaxTxIndexCache << 20 : 0);
    34-    nTotalCache -= sizes.tx_index;
    35-    sizes.filter_index = 0;
    36+    // Convert -dbcache from MiB units to bytes. The total chache is floored by MIN_DB_CACHE and capped by max size_t value.
    


    hodlinator commented at 7:24 pm on January 11, 2025:

    nit: Typo

    0    // Convert -dbcache from MiB units to bytes. The total cache is floored by MIN_DB_CACHE and capped by max size_t value.
    
  109. in src/init.cpp:1606 in cf20c1b9e0 outdated
    1604@@ -1603,18 +1605,18 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
    1605     ReadNotificationArgs(args, kernel_notifications);
    1606 
    1607     // cache size calculations
    1608-    CacheSizes cache_sizes = CalculateCacheSizes(args, g_enabled_filter_types.size());
    1609+    const auto [index_cache_sizes, kernel_cache_sizes] = CalculateCacheSizes(args, g_enabled_filter_types.size());
    


    hodlinator commented at 7:27 pm on January 11, 2025:
    nit: Could have less churn on the line by making InitAndLoadChainstate() take const in the same commit where this line is first touched.
  110. in src/util/overflow.h:64 in cf20c1b9e0 outdated
    59+constexpr std::optional<Output> CheckedLeftShift(Input i, unsigned shift) noexcept
    60+{
    61+    if constexpr (std::is_signed_v<Input>) {
    62+        if (i < 0) return std::nullopt;
    63+    }
    64+    if (std::make_unsigned_t<Input>(i) > Output(std::numeric_limits<Output>::max() >> shift)) {
    


    hodlinator commented at 7:34 pm on January 11, 2025:

    nit: Could modernize cast.

    0    if (std::make_unsigned_t<Input>(i) > Output{std::numeric_limits<Output>::max() >> shift}) {
    

    TheCharlatan commented at 8:58 pm on January 11, 2025:
    Using non-narrowing conversion should give you a compiler error. Allowing type narrowing here should be safe.

    hodlinator commented at 9:10 pm on January 11, 2025:
    In thread #31483 (review): What the.. could have sworn my suggestion compiled without warnings, but now I get them. Sorry.

    stickies-v commented at 10:57 am on January 13, 2025:

    in e0a541702def3a4c6cce8e44b6ebe8d608edad4e:

    Right-shifting by more bits than the width of Output is UB, so we should probably add an extra check for that. I’ve also added some test cases, but interestingly they don’t seem to trigger UBSan (without the util/overflow.h patch) on my machine:

     0diff --git a/src/test/util_tests.cpp b/src/test/util_tests.cpp
     1index 09ba278d3a..ae292d7761 100644
     2--- a/src/test/util_tests.cpp
     3+++ b/src/test/util_tests.cpp
     4@@ -1897,6 +1897,7 @@ void TestCheckedLeftShift()
     5         // Overflow cases
     6         BOOST_CHECK(!CheckedLeftShift<T>((MAX >> 1) + 1, 1));
     7         BOOST_CHECK(!CheckedLeftShift<T>(MAX, 1));
     8+        BOOST_CHECK(!CheckedLeftShift<T>(1, std::numeric_limits<T>::digits + 1));
     9     }
    10 }
    11 
    12@@ -1919,6 +1920,8 @@ void TestSaturatingLeftShift()
    13         // Saturation cases
    14         BOOST_CHECK_EQUAL(SaturatingLeftShift<T>((MAX >> 1) + 1, 1), MAX);
    15         BOOST_CHECK_EQUAL(SaturatingLeftShift<T>(MAX, 1), MAX);
    16+        BOOST_CHECK_EQUAL(SaturatingLeftShift<T>(-1, std::numeric_limits<T>::digits + 1), 0);
    17+        BOOST_CHECK_EQUAL(SaturatingLeftShift<T>(1, std::numeric_limits<T>::digits + 1), MAX);
    18     }
    19 }
    20 
    21diff --git a/src/util/overflow.h b/src/util/overflow.h
    22index a0698f5c69..d85f5b9c9d 100644
    23--- a/src/util/overflow.h
    24+++ b/src/util/overflow.h
    25@@ -61,7 +61,8 @@ constexpr std::optional<Output> CheckedLeftShift(Input i, unsigned shift) noexce
    26     if constexpr (std::is_signed_v<Input>) {
    27         if (i < 0) return std::nullopt;
    28     }
    29-    if (std::make_unsigned_t<Input>(i) > Output(std::numeric_limits<Output>::max() >> shift)) {
    30+    if (shift >= std::numeric_limits<Input>::digits ||
    31+        std::make_unsigned_t<Input>(i) > Output(std::numeric_limits<Output>::max() >> shift)) {
    32         return std::nullopt;
    33     }
    34     return i << shift;
    
  111. TheCharlatan commented at 8:07 pm on January 11, 2025: contributor

    Updated 45fe603f8fd94c171f5ce1a38bc20d273baa9b1b -> cf20c1b9e0b46cb8c4d4f93a054afd9c1f85e4e0 (kernel_cache_sizes_12 -> kernel_cache_sizes_13, compare)

    • Fixed an issue with an include, a cast, and a signed/unsigned comparison.
  112. in src/kernel/caches.h:27 in 4eebb63891 outdated
    22+
    23+    CacheSizes(size_t total_cache)
    24+    {
    25+        block_tree_db = std::min<size_t>(total_cache / 8, MAX_BLOCK_DB_CACHE);
    26+        total_cache -= block_tree_db;
    27+        coins_db = std::min<size_t>(total_cache / 2, MAX_COINS_DB_CACHE);
    


    hodlinator commented at 8:18 pm on January 11, 2025:
    nit in commit 4eebb638916ecf43cfd9237c7e8bb8da7bc397d7: Seems like we could use std::min without the template parameter here from the start (it’s removed later without parameter types changing, resulting in line churn).
  113. hodlinator approved
  114. hodlinator commented at 8:25 pm on January 11, 2025: contributor

    re-ACK cf20c1b9e0b46cb8c4d4f93a054afd9c1f85e4e0

    Happy with the _MiB user-defined literals. :)

    Nothing blocking. Should probably at least fix the typo.


    mega-nit: Could possibly have used CheckedLeftShift() or SaturatingLeftShift() in intermediate commits. But end state doesn’t have naked shifts on touched lines, which is fine.

    0- kernel::CacheSizes cache_sizes{nDefaultDbCache << 20};
    1+ kernel::CacheSizes cache_sizes{CheckedLeftShift<size_t>(nDefaultDbCache, 20).value()};
    
  115. DrahtBot requested review from ryanofsky on Jan 11, 2025
  116. TheCharlatan force-pushed on Jan 11, 2025
  117. TheCharlatan commented at 9:01 pm on January 11, 2025: contributor

    Thanks for the ACK @hodlinator :)

    Updated cf20c1b9e0b46cb8c4d4f93a054afd9c1f85e4e0 -> 0e2887262e1a3a4a5ab0382f9e8713b135408a77 (kernel_cache_sizes_13 -> kernel_cache_sizes_14, compare)

    mega-nit: Could possibly have used CheckedLeftShift() or SaturatingLeftShift() in intermediate commits.

    Skipping this one for now, unless another reviewer thinks it’s useful.

  118. hodlinator approved
  119. hodlinator commented at 9:11 pm on January 11, 2025: contributor

    re-ACK 0e2887262e1a3a4a5ab0382f9e8713b135408a77

    Thanks for addressing my comments. :rocket:

  120. DrahtBot added the label CI failed on Jan 11, 2025
  121. DrahtBot removed the label CI failed on Jan 11, 2025
  122. in src/util/overflow.h:59 in e0a541702d outdated
    54+ * @param shift The number of bits to left shift.
    55+ * @return The result of the left shift, or std::nullopt in case of
    56+ *         overflow or negative input value.
    57+ */
    58+template <std::unsigned_integral Output, std::integral Input>
    59+constexpr std::optional<Output> CheckedLeftShift(Input i, unsigned shift) noexcept
    


    stickies-v commented at 10:29 am on January 13, 2025:

    in e0a541702def3a4c6cce8e44b6ebe8d608edad4e: nit: would prefer replacing i with input (I’m sorry for not doing this in my initial suggestion)

     0diff --git a/src/util/overflow.h b/src/util/overflow.h
     1index a0698f5c69..c35e44690a 100644
     2--- a/src/util/overflow.h
     3+++ b/src/util/overflow.h
     4@@ -50,38 +50,38 @@ template <class T>
     5 
     6 /**
     7  * [@brief](/bitcoin-bitcoin/contributor/brief/) Left bit shift with overflow checking.
     8- * [@param](/bitcoin-bitcoin/contributor/param/) i The input value to be left shifted.
     9+ * [@param](/bitcoin-bitcoin/contributor/param/) input The input value to be left shifted.
    10  * [@param](/bitcoin-bitcoin/contributor/param/) shift The number of bits to left shift.
    11  * [@return](/bitcoin-bitcoin/contributor/return/) The result of the left shift, or std::nullopt in case of
    12  *         overflow or negative input value.
    13  */
    14 template <std::unsigned_integral Output, std::integral Input>
    15-constexpr std::optional<Output> CheckedLeftShift(Input i, unsigned shift) noexcept
    16+constexpr std::optional<Output> CheckedLeftShift(Input input, unsigned shift) noexcept
    17 {
    18     if constexpr (std::is_signed_v<Input>) {
    19-        if (i < 0) return std::nullopt;
    20+        if (input < 0) return std::nullopt;
    21     }
    22-    if (std::make_unsigned_t<Input>(i) > Output(std::numeric_limits<Output>::max() >> shift)) {
    23+    if (std::make_unsigned_t<Input>(input) > Output(std::numeric_limits<Output>::max() >> shift)) {
    24         return std::nullopt;
    25     }
    26-    return i << shift;
    27+    return input << shift;
    28 }
    29 
    30 /**
    31  * [@brief](/bitcoin-bitcoin/contributor/brief/) Left bit shift with safe minimum and maximum values.
    32- * [@param](/bitcoin-bitcoin/contributor/param/) i The input value to be left shifted.
    33+ * [@param](/bitcoin-bitcoin/contributor/param/) input The input value to be left shifted.
    34  * [@param](/bitcoin-bitcoin/contributor/param/) shift The number of bits to left shift.
    35  * [@return](/bitcoin-bitcoin/contributor/return/) The result of the left shift, with the return value clamped
    36  *         between zero and the maximum Output value if overflow occurs.
    37  */
    38 template <std::unsigned_integral Output, std::integral Input>
    39-constexpr Output SaturatingLeftShift(Input i, unsigned shift) noexcept
    40+constexpr Output SaturatingLeftShift(Input input, unsigned shift) noexcept
    41 {
    42     auto default_value{std::numeric_limits<Output>::max()};
    43     if constexpr (std::is_signed_v<Input>) {
    44-        if (i < 0) default_value = 0;
    45+        if (input < 0) default_value = 0;
    46     }
    47-    return CheckedLeftShift<Output>(i, shift).value_or(default_value);
    48+    return CheckedLeftShift<Output>(input, shift).value_or(default_value);
    49 }
    50 
    51 #endif // BITCOIN_UTIL_OVERFLOW_H
    
  123. in src/util/overflow.h:83 in e0a541702d outdated
    78+constexpr Output SaturatingLeftShift(Input i, unsigned shift) noexcept
    79+{
    80+    auto default_value{std::numeric_limits<Output>::max()};
    81+    if constexpr (std::is_signed_v<Input>) {
    82+        if (i < 0) default_value = 0;
    83+    }
    


    stickies-v commented at 11:50 am on January 13, 2025:

    in e0a541702def3a4c6cce8e44b6ebe8d608edad4e:

    Saturated left shifting a 0-input should probably always return a 0-output, so suggested update (+ tests, one depending on my other comment re right-shift UB):

     0diff --git a/src/test/util_tests.cpp b/src/test/util_tests.cpp
     1index 09ba278d3a..07b98be446 100644
     2--- a/src/test/util_tests.cpp
     3+++ b/src/test/util_tests.cpp
     4@@ -1897,6 +1897,7 @@ void TestCheckedLeftShift()
     5         // Overflow cases
     6         BOOST_CHECK(!CheckedLeftShift<T>((MAX >> 1) + 1, 1));
     7         BOOST_CHECK(!CheckedLeftShift<T>(MAX, 1));
     8+        // BOOST_CHECK(!CheckedLeftShift<T>(0, std::numeric_limits<T>::digits + 1));  // TODO: requires right-shift overflow check to pass
     9     }
    10 }
    11 
    12@@ -1919,6 +1920,7 @@ void TestSaturatingLeftShift()
    13         // Saturation cases
    14         BOOST_CHECK_EQUAL(SaturatingLeftShift<T>((MAX >> 1) + 1, 1), MAX);
    15         BOOST_CHECK_EQUAL(SaturatingLeftShift<T>(MAX, 1), MAX);
    16+        BOOST_CHECK_EQUAL(SaturatingLeftShift<T>(0, std::numeric_limits<T>::digits + 1), 0);
    17     }
    18 }
    19 
    20diff --git a/src/util/overflow.h b/src/util/overflow.h
    21index a0698f5c69..f9e1fe4e82 100644
    22--- a/src/util/overflow.h
    23+++ b/src/util/overflow.h
    24@@ -77,10 +77,7 @@ constexpr std::optional<Output> CheckedLeftShift(Input i, unsigned shift) noexce
    25 template <std::unsigned_integral Output, std::integral Input>
    26 constexpr Output SaturatingLeftShift(Input i, unsigned shift) noexcept
    27 {
    28-    auto default_value{std::numeric_limits<Output>::max()};
    29-    if constexpr (std::is_signed_v<Input>) {
    30-        if (i < 0) default_value = 0;
    31-    }
    32+    auto default_value{i <= 0 ? 0 : std::numeric_limits<Output>::max()};
    33     return CheckedLeftShift<Output>(i, shift).value_or(default_value);
    34 }
    35 
    
  124. in src/node/caches.cpp:29 in ccf8caa443 outdated
    25@@ -24,9 +26,6 @@ CacheSizes CalculateCacheSizes(const ArgsManager& args, size_t n_indexes)
    26         sizes.filter_index = max_cache / n_indexes;
    27         nTotalCache -= sizes.filter_index * n_indexes;
    28     }
    29-    sizes.coins_db = std::min(nTotalCache / 2, nMaxCoinsDBCache << 20);
    30-    nTotalCache -= sizes.coins_db;
    31-    sizes.coins = nTotalCache; // the rest goes to in-memory cache
    32-    return sizes;
    33+    return {sizes, kernel::CacheSizes{static_cast<size_t>(nTotalCache)}};
    


    stickies-v commented at 12:09 pm on January 13, 2025:

    in ccf8caa4430f57b8f148e96ac2a241221977bcab

    note/nit: this can overflow on 32 bit machines. It is fixed in later commits, and I think fixing it in this commit would be overly verbose, but I’m mentioning it in case there are further code changes that could make this problematic again.

  125. in src/node/chainstate.cpp:32 in ccf8caa443 outdated
    28@@ -29,6 +29,8 @@
    29 #include <memory>
    30 #include <vector>
    31 
    32+using kernel::CacheSizes;
    


    stickies-v commented at 12:13 pm on January 13, 2025:

    in ccf8caa4430f57b8f148e96ac2a241221977bcab

    nit: I’ve raised this before, but since we now have a kernel::CacheSizes and node::CacheSizes, I think using a kernel::CacheSizes in a node file is unnecessarily confusing, so I would prefer making this explicit (as in the chainstate.h file):

     0diff --git a/src/node/chainstate.cpp b/src/node/chainstate.cpp
     1index 660d2c3289..d4d8828f73 100644
     2--- a/src/node/chainstate.cpp
     3+++ b/src/node/chainstate.cpp
     4@@ -29,14 +29,12 @@
     5 #include <memory>
     6 #include <vector>
     7 
     8-using kernel::CacheSizes;
     9-
    10 namespace node {
    11 // Complete initialization of chainstates after the initial call has been made
    12 // to ChainstateManager::InitializeChainstate().
    13 static ChainstateLoadResult CompleteChainstateInitialization(
    14     ChainstateManager& chainman,
    15-    const CacheSizes& cache_sizes,
    16+    const kernel::CacheSizes& cache_sizes,
    17     const ChainstateLoadOptions& options) EXCLUSIVE_LOCKS_REQUIRED(::cs_main)
    18 {
    19     auto& pblocktree{chainman.m_blockman.m_block_tree_db};
    20@@ -172,7 +170,7 @@ static ChainstateLoadResult CompleteChainstateInitialization(
    21     return {ChainstateLoadStatus::SUCCESS, {}};
    22 }
    23 
    24-ChainstateLoadResult LoadChainstate(ChainstateManager& chainman, const CacheSizes& cache_sizes,
    25+ChainstateLoadResult LoadChainstate(ChainstateManager& chainman, const kernel::CacheSizes& cache_sizes,
    26                                     const ChainstateLoadOptions& options)
    27 {
    28     if (!chainman.AssumedValidBlock().IsNull()) {
    
  126. in src/node/caches.cpp:14 in 0e2887262e outdated
     6@@ -7,31 +7,38 @@
     7 #include <common/args.h>
     8 #include <index/txindex.h>
     9 #include <kernel/caches.h>
    10+#include <logging.h>
    11+#include <util/byte_units.h>
    12 
    13 #include <algorithm>
    14+#include <optional>
    


    stickies-v commented at 2:35 pm on January 13, 2025:

    in 0e2887262e1a3a4a5ab0382f9e8713b135408a77

    nit: optional header no longer necessary

  127. in src/node/caches.cpp:36 in 0e2887262e outdated
    41+    }
    42+
    43+    IndexCacheSizes index_sizes;
    44+    index_sizes.tx_index = std::min(total_cache / 8, args.GetBoolArg("-txindex", DEFAULT_TXINDEX) ? MAX_TX_INDEX_CACHE : 0);
    45+    total_cache -= index_sizes.tx_index;
    46+    index_sizes.filter_index = 0;
    


    stickies-v commented at 2:45 pm on January 13, 2025:

    in 0e2887262e

    nit: I would default-initialize IndexCacheSizes members to 0, avoiding potential issues of accessing uninitialized members (and removing the need for this line):

     0diff --git a/src/node/caches.cpp b/src/node/caches.cpp
     1index 0e0305cd19..f4fbb7ead9 100644
     2--- a/src/node/caches.cpp
     3+++ b/src/node/caches.cpp
     4@@ -33,7 +33,6 @@ CacheSizes CalculateCacheSizes(const ArgsManager& args, size_t n_indexes)
     5     IndexCacheSizes index_sizes;
     6     index_sizes.tx_index = std::min(total_cache / 8, args.GetBoolArg("-txindex", DEFAULT_TXINDEX) ? MAX_TX_INDEX_CACHE : 0);
     7     total_cache -= index_sizes.tx_index;
     8-    index_sizes.filter_index = 0;
     9     if (n_indexes > 0) {
    10         int64_t max_cache = std::min(total_cache / 8, MAX_FILTER_INDEX_CACHE);
    11         index_sizes.filter_index = max_cache / n_indexes;
    12diff --git a/src/node/caches.h b/src/node/caches.h
    13index 0f916c7e5b..f24e9cc910 100644
    14--- a/src/node/caches.h
    15+++ b/src/node/caches.h
    16@@ -19,8 +19,8 @@ static constexpr size_t DEFAULT_DB_CACHE{DEFAULT_KERNEL_CACHE};
    17 
    18 namespace node {
    19 struct IndexCacheSizes {
    20-    size_t tx_index;
    21-    size_t filter_index;
    22+    size_t tx_index{0};
    23+    size_t filter_index{0};
    24 };
    25 struct CacheSizes {
    26     IndexCacheSizes index;
    
  128. in src/node/caches.cpp:38 in 0e2887262e outdated
    45+    index_sizes.filter_index = 0;
    46     if (n_indexes > 0) {
    47-        int64_t max_cache = std::min(nTotalCache / 8, max_filter_index_cache << 20);
    48-        sizes.filter_index = max_cache / n_indexes;
    49-        nTotalCache -= sizes.filter_index * n_indexes;
    50+        int64_t max_cache = std::min(total_cache / 8, MAX_FILTER_INDEX_CACHE);
    


    stickies-v commented at 2:51 pm on January 13, 2025:

    in 0e2887262e1a3a4a5ab0382f9e8713b135408a77

    nit: this should now also be a size_t

    0        size_t max_cache{std::min(total_cache / 8, MAX_FILTER_INDEX_CACHE)};
    
  129. stickies-v commented at 3:06 pm on January 13, 2025: contributor

    Did a full re-review on 0e2887262e1a3a4a5ab0382f9e8713b135408a77. Thank you for incorporating my util/overflow.h and util/byte_units.h suggestions, I really like the current approach.

    Nits can be ignored, but I think my 2 util/overflow.h UB comments(1, 2) would be best to fix before merge. I probably won’t be having any further comments after this.

  130. DrahtBot requested review from stickies-v on Jan 13, 2025
  131. in src/util/overflow.h:78 in e0a541702d outdated
    73+ * @param shift The number of bits to left shift.
    74+ * @return The result of the left shift, with the return value clamped
    75+ *         between zero and the maximum Output value if overflow occurs.
    76+ */
    77+template <std::unsigned_integral Output, std::integral Input>
    78+constexpr Output SaturatingLeftShift(Input i, unsigned shift) noexcept
    


    ryanofsky commented at 5:04 pm on January 13, 2025:

    In commit “util: Add integer left shift helpers” (e0a541702def3a4c6cce8e44b6ebe8d608edad4e)

    It seems wrong that calling SaturatingLeftShift on a negative input can wrap around to 0. The point of saturating operations should be that they don’t wrap around, but effectively perform operations with full precision and just clamp the result values. In c++ left shift i << j is defined as i * 2j, so CheckedLeftShift and SaturatingLeftShift should be able to compute the same thing, but without undefined behavior or wrapping.

    It also seems like a footgun that these left shift functions are using different input and output types instead of just using the one type like other functions in this file. Would suggest making new functions more similar to existing functions and writing like:

     0template <std::integral T>
     1constexpr std::optional<T> CheckedLeftShift(T i, unsigned shift) noexcept
     2{
     3    if (shift == 0 || i == 0) return i;
     4    // Avoid undefined c++ behavior if shift is >= number of bits in T.
     5    if (shift >= sizeof(T) * CHAR_BIT) return std::nullopt;
     6    // If i << shift is too big to fit in T, return nullopt.
     7    const auto max{std::numeric_limits<T>::max() >> shift};
     8    if (i > 0 && i > max) return std::nullopt;
     9    if (i < 0 && i < -max - 1) return std::nullopt;
    10    return i << shift;
    11}
    12
    13template <std::integral T>
    14constexpr T SaturatingLeftShift(T i, unsigned shift) noexcept
    15{
    16    if (auto result{CheckedLeftShift(i, shift)}) return *result;
    17    // If i << shift is to big to fit in T, return biggest positive or negative
    18    // number that fits.
    19    return i < 0 ? std::numeric_limits<T>::min() : std::numeric_limits<T>::max();
    20}
    

    I didn’t test these extensively but did sanity check with:

     0    static_assert(SaturatingLeftShift<signed char>(0, 100) ==  0);
     1    static_assert(SaturatingLeftShift<signed char>(1, 100) ==  127);
     2    static_assert(SaturatingLeftShift<signed char>(-1, 100) ==  -128);
     3
     4    static_assert(SaturatingLeftShift<signed char>(3, 2) ==  12);
     5    static_assert(SaturatingLeftShift<signed char>(-3, 2) ==  -12);
     6
     7    static_assert(SaturatingLeftShift<signed char>(6, 4) ==  96);
     8    static_assert(SaturatingLeftShift<signed char>(-6, 4) ==  -96);
     9
    10    static_assert(SaturatingLeftShift<signed char>(63, 1) ==  126);
    11    static_assert(SaturatingLeftShift<signed char>(64, 1) ==  127);
    12    static_assert(SaturatingLeftShift<signed char>(-63, 1) ==  -126);
    13    static_assert(SaturatingLeftShift<signed char>(-64, 1) ==  -128);
    14    static_assert(SaturatingLeftShift<signed char>(-65, 1) ==  -128);
    15
    16
    17    static_assert(SaturatingLeftShift<signed char>(31, 2) ==  124);
    18    static_assert(SaturatingLeftShift<signed char>(32, 2) ==  127);
    19    static_assert(SaturatingLeftShift<signed char>(-31, 2) ==  -124);
    20    static_assert(SaturatingLeftShift<signed char>(-32, 2) ==  -128);
    21    static_assert(SaturatingLeftShift<signed char>(-33, 2) ==  -128);
    

    stickies-v commented at 5:59 pm on January 13, 2025:

    It seems wrong that calling SaturatingLeftShift on a negative input can wrap around to 0.

    My understanding was that left shift a << b with negative a was UB in C++, which is why I thought saturating negative inputs to 0 was the most sensible noexcept approach. Thanks for pointing out (as per your cppreference link) that since C++20 this is indeed defined, which definitely means our implementation can (and should) too, nice!

    It also seems like a footgun that these left shift functions are using different input and output types instead of just using the one type like other functions in this file.

    That was my initial approach, but I eventually decided on separate types as to be able to safely catch negative input values instead of silently wrapping around (e.g. when calling SaturatingLeftShift<size_t>(*db_cache, 20) with a negative db_cache). For my understanding, if it’s not too much off-topic, I’d be curious to hear where the footgun potential is - since the point of my approach was to exactly avoid that?


    I agree that since C++20, left shift helper functions with negative input and result values, a single type T is preferable.

    Your suggested implementation LGTM but I think keeping std::numeric_limits<T>::digits instead of sizeof(T) * CHAR_BIT is preferable since it takes into account the sign bit?

    edit: I still think, to prevent accidental overflow/wrap-around, that having separate Input and Output types is safer, even when allowing for negative inputs and outputs.


    TheCharlatan commented at 9:10 am on January 14, 2025:

    Your suggested implementation LGTM but I think keeping std::numeric_limits::digits instead of sizeof(T) * CHAR_BIT is preferable since it takes into account the sign bit?

    As far as I understand it is legal to shift e.g. int32_t{1} << 31 to get the minimum int32_t value. But I think in this case we should prohibit this quirk of the underlying bit representation and use std::numeric_limits<T>::digits.


    ryanofsky commented at 3:27 pm on January 14, 2025:

    re: #31483 (review)

    I didn’t know about digits() and agree that it would be better to use than sizeof() so thanks for suggesting that.

    For my understanding, if it’s not too much off-topic, I’d be curious to hear where the footgun potential is - since the point of my approach was to exactly avoid that?

    My concern with using different types for input and output is that code like auto kbps = SaturatingLeftShift(mbps, 10) would be misleading because you could reasonably assume that multiplying a number by 1024 should not change its type from signed to unsigned (and set it to 0 if the input is negative), but that is what the current implementation in e0a541702def3a4c6cce8e44b6ebe8d608edad4e) would do.


    stickies-v commented at 3:40 pm on January 14, 2025:

    but that is what the current implementation in e0a5417) would do.

    Are you sure it would? I don’t see how the Output return type could be deduced with auto. That’s kind of the point of the separate Input and Output types: it force the user to specify the return type, because especially with left shift the chances of the return type being much bigger than the input are high. Using (and potentially wrapping around) the input type, for that reason, seems footgunny for a util/overflow.h function?

    When testing with:

    0    int32_t mbps{20};
    1    auto kbps = SaturatingLeftShift(mbps, 10);
    

    I get, as expected:

    0../../src/node/caches.cpp:30:17: error: no matching function for call to 'SaturatingLeftShift'
    1    auto kbps = SaturatingLeftShift(mbps, 10);
    2                ^~~~~~~~~~~~~~~~~~~
    3../../src/util/overflow.h:78:18: note: candidate template ignored: couldn't infer template argument 'Output'
    4constexpr Output SaturatingLeftShift(Input i, unsigned shift) noexcept
    5                 ^
    61 error generated.
    

    would be misleading

    With the user being forced to specify the return type, I don’t think there’s any room for ambiguity?


    ryanofsky commented at 4:02 pm on January 14, 2025:

    Are you sure it would? I don’t see how the Output return type could be deduced with auto.

    You’re right. I didn’t realize e0a541702def3a4c6cce8e44b6ebe8d608edad4e required specifying what type to cast the output to. In that case, I take back the comment that it has a footgun.

    But I would say the implementation does not seem consistent with other functions in this file, or consistent with what I would expect a function that is doing a saturating multiplication operation to do. I think the fact that it casts output to a different type is not ideal, and the wrapping to 0 doesn’t make much sense to me. I think SaturatingLeftShift(i, shift) should just be a saturating implementation of (i x 2shift) like SaturatingAdd(i, j) is a saturating implementation of (i + j).


    TheCharlatan commented at 4:31 pm on January 14, 2025:

    But I would say the implementation does not seem consistent with other functions in this file, or consistent with what I would expect a function that is doing a saturating multiplication operation to do.

    I tend to agree with this, and rolled with your suggestion for now, but still open for other suggestions.


    stickies-v commented at 1:56 pm on January 15, 2025:
    Alright, I’m on board too. I liked how the dual-type implementation made using the functions without casting and clamping easier (and as such perhaps a bit safer), but I agree that from a single-responsibility principle the current approach makes more sense, and it is consistent.
  132. DrahtBot requested review from ryanofsky on Jan 13, 2025
  133. TheCharlatan commented at 4:26 pm on January 14, 2025: contributor

    Updated 0e2887262e1a3a4a5ab0382f9e8713b135408a77 -> 922aa7a25650bdfd8428198f171f88915af1ffa0 (kernel_cache_sizes_14 -> kernel_cache_sizes_15, compare)

    • Addressed @stickies-v’s comment, renaming i to input.
    • Addressed @stickies-v’s comment, avoid UB when shifting by more bits than the shifted type is wide.
    • Addressed @stickies-v’s comment, return 0 early in CheckedLeftShift when the input is 0.
    • Addressed @stickies-v’s comment, get rid of potentially confusing using kernel::CachSizes statement in init.cpp.
    • Addressed @stickies-v’s comment, removed optional include where no longer required from a prior state of this pull request.
    • Addressed @stickies-v’s comment, default-initialize IndexCacheSizes fields with 0.
    • Addressed @stickies-v’s comment, correct type of max_cache to size_t.
    • Took @ryanofsky’s suggestion, make CheckedLeftShift and SaturatingLeftShift use the same type for input and output, as well as allowing negative input.
  134. TheCharlatan force-pushed on Jan 14, 2025
  135. in src/test/util_tests.cpp:1882 in a85c07a891 outdated
    1877@@ -1877,4 +1878,99 @@ BOOST_AUTO_TEST_CASE(clearshrink_test)
    1878     }
    1879 }
    1880 
    1881+template <typename T>
    1882+void TestCheckedLeftShift()
    


    ryanofsky commented at 5:56 pm on January 14, 2025:

    In commit “util: Add integer left shift helpers” (a85c07a891f76cd9e606aa83c0671506b1173f30)

    Could consider adding a fuzz test too. Maybe:

     0namespace {
     1//! Test overflow operations for type T using a wider type, W, to verify results.
     2template <typename T, typename W>
     3void TestOverflow(FuzzedDataProvider& fuzzed_data_provider)
     4{
     5    constexpr auto min{std::numeric_limits<T>::min()};
     6    constexpr auto max{std::numeric_limits<T>::max()};
     7    static_assert(min >= std::numeric_limits<W>::min() / 2);
     8    static_assert(max <= std::numeric_limits<W>::max() / 2);
     9
    10    auto widen = [](T value) -> W { return value; };
    11    auto clamp = [](W value) -> W { return std::clamp<W>(value, min, max); };
    12    auto check = [](W value) -> std::optional<W> { if (value >= min && value <= max) return value; else return std::nullopt; };
    13
    14    const T i = fuzzed_data_provider.ConsumeIntegral<T>();
    15    const T j = fuzzed_data_provider.ConsumeIntegral<T>();
    16    const unsigned shift = fuzzed_data_provider.ConsumeIntegralInRange<unsigned>(0, std::numeric_limits<W>::digits - 1);
    17
    18    Assert(clamp(widen(i) + widen(j)) == SaturatingAdd(i, j));
    19    Assert(check(widen(i) + widen(j)) == CheckedAdd(i, j));
    20
    21    Assert(clamp(widen(i) << shift) == SaturatingLeftShift(i, j));
    22    Assert(check(widen(i) << shift) == CheckedLeftShift(i, j));
    23}
    24} // namespace
    25
    26FUZZ_TARGET(overflow)
    27{
    28    FuzzedDataProvider fuzzed_data_provider(buffer.data(), buffer.size());
    29    TestOverflow<int8_t, int16_t>(fuzzed_data_provider);
    30    TestOverflow<int16_t, int32_t>(fuzzed_data_provider);
    31    TestOverflow<int32_t, int64_t>(fuzzed_data_provider);
    32    TestOverflow<uint8_t, uint16_t>(fuzzed_data_provider);
    33    TestOverflow<uint16_t, uint32_t>(fuzzed_data_provider);
    34    TestOverflow<uint32_t, uint64_t>(fuzzed_data_provider);
    35}
    

    (This compiles but I didn’t actually test it)


    TheCharlatan commented at 9:24 pm on January 14, 2025:

    Thanks, but I get a crash immediately:

    0test/fuzz/overflow.cpp:36 TestOverflow: Assertion `check(widen(i) << shift) == CheckedLeftShift(i, j)' failed.
    

    Does this correction make sense to you?

     0diff --git a/src/test/fuzz/overflow.cpp b/src/test/fuzz/overflow.cpp
     1index defaf9926d..d29273eb6e 100644
     2--- a/src/test/fuzz/overflow.cpp
     3+++ b/src/test/fuzz/overflow.cpp
     4@@ -27,13 +27,13 @@ void TestOverflow(FuzzedDataProvider& fuzzed_data_provider)
     5 
     6     const T i = fuzzed_data_provider.ConsumeIntegral<T>();
     7     const T j = fuzzed_data_provider.ConsumeIntegral<T>();
     8-    const unsigned shift = fuzzed_data_provider.ConsumeIntegralInRange<unsigned>(0, std::numeric_limits<W>::digits - 1);
     9+    const unsigned shift = fuzzed_data_provider.ConsumeIntegralInRange<unsigned>(0, std::numeric_limits<T>::digits - 1);
    10 
    11     Assert(clamp(widen(i) + widen(j)) == SaturatingAdd(i, j));
    12     Assert(check(widen(i) + widen(j)) == CheckedAdd(i, j));
    13 
    14-    Assert(clamp(widen(i) << shift) == SaturatingLeftShift(i, j));
    15-    Assert(check(widen(i) << shift) == CheckedLeftShift(i, j));
    16+    Assert(clamp(widen(i) << shift) == SaturatingLeftShift(i, shift));
    17+    Assert(check(widen(i) << shift) == CheckedLeftShift(i, shift));
    18 }
    19 } // namespace
    

    Edit: The test seems fine otherwise and I don’t find new paths after just a few seconds.


    ryanofsky commented at 9:31 pm on January 14, 2025:
    Thanks, the bottom part of that correction (shifting by shift instead of j) makes sense but the top part changing std::numeric_limits<W>::digits to std::numeric_limits<T>::digits reduces the coverage of the fuzz test by reducing the range of shift so I think would not be a good change.

    TheCharlatan commented at 10:29 pm on January 14, 2025:
    Ah, of course. The actual problem is that we are shifting even out of the widened type’s range, so it wraps around. Not sure if there is a nice solution to that without re-implementing the logic we are trying to test. Do you have a better solution besides changing the shift value to get the digits of T?

    ryanofsky commented at 4:02 am on January 15, 2025:

    Ah, of course. The actual problem is that we are shifting even out of the widened type’s range, so it wraps around.

    I see. So the max amount you can safely shift is without overflowing is not std::numeric_limits<W>::digits - 1 but actually std::numeric_limits<W>::digits - std::numeric_limits<T>::digits since that is how much free space there is to the right of T’s bits in W.

    So maybe if you switch to that amount and change the test cases to:

    TestOverflow<int8_t, int64_t>(fuzzed_data_provider);
    TestOverflow<int16_t, int64_t>(fuzzed_data_provider);
    TestOverflow<int32_t, int64_t>(fuzzed_data_provider);
    TestOverflow<uint8_t, uint64_t>(fuzzed_data_provider);
    TestOverflow<uint16_t, uint64_t>(fuzzed_data_provider);
    TestOverflow<uint32_t, uint64_t>(fuzzed_data_provider);
    

    It could provide good coverage without overflowing


    TheCharlatan commented at 8:23 am on January 15, 2025:
    Yeah, that should work. I’ll fuzz with this for some time.

    hodlinator commented at 9:56 am on January 16, 2025:

    In thread #31483 (review):

    I see. So the max amount you can safely shift is without overflowing is not std::numeric_limits<W>::digits - 1 but actually std::numeric_limits<W>::digits - std::numeric_limits<T>::digits since that is how much free space there is to the right of T’s bits in W.

    Would appreciate if someone were to ELI5 this explanation. I’m thinking it must be clamp(widen(i) << shift) that doesn’t tolerate it somehow (but why?), while SaturatingLeftShift(i, shift) works okay? Tried fuzzing with std::numeric_limits<W>::digits - (std::numeric_limits<T>::digits - 1) which fails, and it’s not intuitive to me.

    (This does not compile (for unsigned types) due to shifting as many bits as are in the type:

    0static_assert((W{1} << std::numeric_limits<W>::digits) == 0);
    

    Tried to trigger some UB-complaint through the following, but did not succeed:

    0static_assert((W{1 << 8} << (std::numeric_limits<W>::digits - 1)) == 0);
    

    )


    ryanofsky commented at 3:21 pm on January 16, 2025:

    In thread #31483 (comment):

    I see. So the max amount you can safely shift is without overflowing is not std::numeric_limits<W>::digits - 1 but actually std::numeric_limits<W>::digits - std::numeric_limits<T>::digits since that is how much free space there is to the right of T’s bits in W.

    Would appreciate if someone were to ELI5 this explanation. I’m thinking it must be clamp(widen(i) << shift) that doesn’t tolerate it somehow (but why?), while SaturatingLeftShift(i, shift) works okay?

    That’s right, the case I was thinking of is T=uint16_t, W=uint32_t, shift=31, i=0x8000, where 0x8000 << 31 is too big to fit in an uint32_t. The result will be well-defined but it will be 0 before it is seen by clamp(). The biggest shift that is possible without wrapping to 0 is 0x8000 << 16


    hodlinator commented at 11:29 am on January 17, 2025:
    Ah, so widen(i) << shift => 0 for the left side even before clamp(), and SaturatingLeftShift(i, shift) => std::numeric_limits<uint16_t>::max() for the right side. Thanks!
  136. in src/util/overflow.h:67 in a85c07a891 outdated
    63+    // Avoid undefined c++ behaviour if shift is >= number of bits in T.
    64+    if (shift >= std::numeric_limits<T>::digits) return std::nullopt;
    65+    // If input << shift is too big to fit in T, return nullopt.
    66+    const auto max_allowed_input{std::numeric_limits<T>::max() >> shift};
    67+    if (input > 0 && input > max_allowed_input) return std::nullopt;
    68+    if constexpr (std::is_signed_v<T>) {
    


    ryanofsky commented at 6:02 pm on January 14, 2025:

    In commit “util: Add integer left shift helpers” (https://github.com/bitcoin/bitcoin/commit/a85c07a891f76cd9e606aa83c0671506b1173f30)

    Could consider dropping this line. Code is correct without it and it seems like a distraction. Or, if it is important to check type of T for some reason, presumably that reason would also apply in the SaturatingLeftShift function below and this same check should be done there before checking for input < 0.


    TheCharlatan commented at 8:42 pm on January 14, 2025:
    I had a compiler complain about signed to unsigned integer comparison without it.
  137. in src/util/overflow.h:57 in a85c07a891 outdated
    48@@ -47,4 +49,44 @@ template <class T>
    49     return i + j;
    50 }
    51 
    52+/**
    53+ * @brief Left bit shift with overflow checking.
    54+ * @param input The input value to be left shifted.
    55+ * @param shift The number of bits to left shift.
    56+ * @return The result of the left shift, or std::nullopt in case of
    57+ *         overflow or too high shift value.
    


    ryanofsky commented at 6:13 pm on January 14, 2025:

    In commit “util: Add integer left shift helpers” (https://github.com/bitcoin/bitcoin/commit/a85c07a891f76cd9e606aa83c0671506b1173f30)

    “overflow or too high shift value” seem like the same thing so it is confusing to refer to them as different cases. Phrasing is also potentially ambiguous. Would maybe suggest @return (input * 2^shift) or nullopt if it would not fit in the return type.

  138. in src/util/overflow.h:81 in a85c07a891 outdated
    76+ * @param input The input value to be left shifted.
    77+ * @param shift The number of bits to left shift.
    78+ * @return The result of the left shift, with the return value saturated
    79+ *         to the maximum value of T if the input is positive and the
    80+ *         minimum value of T if the input is negative, on left shift
    81+ *         overflow.
    


    ryanofsky commented at 6:18 pm on January 14, 2025:

    In commit “util: Add integer left shift helpers” (https://github.com/bitcoin/bitcoin/commit/a85c07a891f76cd9e606aa83c0671506b1173f30)

    Again would suggest something shorter and less ambiguous about meaning of left shift like @return (input * 2^shift) clamped to fit between the lowest and highest representable values of the type T

  139. in src/node/caches.cpp:31 in 922aa7a256 outdated
    35+    // Convert -dbcache from MiB units to bytes. The total cache is floored by MIN_DB_CACHE and capped by max size_t value.
    36+    size_t total_cache{DEFAULT_DB_CACHE};
    37+    if (std::optional<int64_t> db_cache = args.GetIntArg("-dbcache")) {
    38+        if (*db_cache < 0) db_cache = 0;
    39+        uint64_t db_cache_bytes = SaturatingLeftShift<uint64_t>(*db_cache, 20);
    40+        total_cache = std::max<size_t>(MIN_DB_CACHE, std::min<uint64_t>(db_cache_bytes, std::numeric_limits<size_t>::max()));
    


    ryanofsky commented at 8:01 pm on January 14, 2025:

    In commit “init: Use size_t consistently for cache sizes” (922aa7a25650bdfd8428198f171f88915af1ffa0)

    Note: Passing max size_t value to min<uint64_t> could be broken in the theoretical case where size_t type is wider than uint64_t. If we had a saturate_cast function the code be simplified to avoid this type of problem:

    0    if (std::optional<int64_t> db_cache = args.GetIntArg("-dbcache")) {
    1        total_cache = std::max(MIN_DB_CACHE, saturate_cast<size_t>(SaturatingLeftShift(*db_cache, 20)));
    2    }
    

    But I think it is fine to keep current code because for practical purposes size_t should not be wider than uint64_t.

  140. ryanofsky approved
  141. ryanofsky commented at 8:27 pm on January 14, 2025: contributor
    Code review ACK 922aa7a25650bdfd8428198f171f88915af1ffa0. Changes since last review were clamping -dbcache cache value instead of returning an error if it is too large, making more constants use bytes instead of mib, replacing MiBToBytes with SaturatingLeftShift, adding MiB literal suffix. Would maybe consider squashing the last two commits as it seems a little to safer to rename the constants at the same time as changing their units instead of separately.
  142. DrahtBot requested review from hodlinator on Jan 14, 2025
  143. TheCharlatan force-pushed on Jan 15, 2025
  144. TheCharlatan commented at 9:06 am on January 15, 2025: contributor

    Updated 922aa7a25650bdfd8428198f171f88915af1ffa0 -> 430d5feee9b8be3a85e85696c449753783301c2b (kernel_cache_sizes_15 -> kernel_cache_sizes_16, compare)

    • Addressed @ryanofsky’s comment_1, comment_2, simplifying left shift helpers doc string.
    • Added @ryanofsky’s suggested fuzz test.
    • Reverted bit width check to if (shift >= sizeof(T) * CHAR_BIT) return std::nullopt; instead of counting digits, since it actually does as advertised (count bits to avoid UB). Any potential shift into negative value ranges is still caught by the max_allowed_input check.
  145. in src/util/overflow.h:69 in 430d5feee9 outdated
    64+    // If input << shift is too big to fit in T, return nullopt.
    65+    const auto max_allowed_input{std::numeric_limits<T>::max() >> shift};
    66+    if (input > 0 && input > max_allowed_input) return std::nullopt;
    67+    if constexpr (std::is_signed_v<T>) {
    68+        if (input < 0 && input < -max_allowed_input - 1) return std::nullopt;
    69+    }
    


    stickies-v commented at 12:24 pm on January 15, 2025:

    in 7993f26173d911d3773cb2580938ac9aad8ad31b:

    nit: I think the runtime signedness checks are unnecessary, and find std::numeric_limits<T>::min() a bit more readable than -max_allowed_input - 1, so putting it all together this can just be simplified to

    0    if (input > (std::numeric_limits<T>::max() >> shift)) return std::nullopt;
    1    if (input < (std::numeric_limits<T>::min() >> shift)) return std::nullopt;
    

    ryanofsky commented at 2:16 pm on January 15, 2025:

    re: #31483 (review)

    nit: I think the runtime signedness checks are unnecessary, and find std::numeric_limits<T>::min() a bit more readable than -max_allowed_input - 1, so putting it all together this can just be simplified to

    Agree with this suggestion. The only reason for writing code that way was that I read “For negative a, the value of a » b is implementation-defined” in the c++ reference and didn’t realize that sentence was in the “until c++20” section. After c++ 20, right shifting negative values is well defined so this should be ok.


    TheCharlatan commented at 2:26 pm on January 15, 2025:
    Mmh, yeah, I think that should work.
  146. in src/test/util_tests.cpp:1973 in 7993f26173 outdated
    1968+
    1969+BOOST_AUTO_TEST_CASE(mib_string_literal_test)
    1970+{
    1971+    BOOST_CHECK_EQUAL(0_MiB, 0);
    1972+    BOOST_CHECK_EQUAL(1_MiB, 1024 * 1024);
    1973+    BOOST_CHECK_EXCEPTION(17592186044416_MiB, std::overflow_error, HasReason("MiB value too large for size_t byte conversion"));
    


    stickies-v commented at 12:34 pm on January 15, 2025:

    in 7993f26173d911d3773cb2580938ac9aad8ad31b

    You could make this test platform independent with:

     0diff --git a/src/test/util_tests.cpp b/src/test/util_tests.cpp
     1index 9991b1dda5..5bde29d0bb 100644
     2--- a/src/test/util_tests.cpp
     3+++ b/src/test/util_tests.cpp
     4@@ -1970,7 +1970,9 @@ BOOST_AUTO_TEST_CASE(mib_string_literal_test)
     5 {
     6     BOOST_CHECK_EQUAL(0_MiB, 0);
     7     BOOST_CHECK_EQUAL(1_MiB, 1024 * 1024);
     8-    BOOST_CHECK_EXCEPTION(17592186044416_MiB, std::overflow_error, HasReason("MiB value too large for size_t byte conversion"));
     9+    const auto max_mib{std::numeric_limits<size_t>::max() >> 20};
    10+    BOOST_CHECK_EXCEPTION(operator"" _MiB(static_cast<unsigned long long>(max_mib) + 1),
    11+                          std::overflow_error, HasReason("MiB value too large for size_t byte conversion"));
    12 }
    13 
    14 BOOST_AUTO_TEST_SUITE_END()
    
  147. in src/node/caches.h:23 in 430d5feee9 outdated
    19+static constexpr size_t DEFAULT_DB_CACHE{DEFAULT_KERNEL_CACHE};
    20+
    21 namespace node {
    22+struct IndexCacheSizes {
    23+    int64_t tx_index{0};
    24+    int64_t filter_index{0};
    


    stickies-v commented at 1:50 pm on January 15, 2025:
    I think these should be size_t too?

    TheCharlatan commented at 2:14 pm on January 15, 2025:
    Must have dropped this by mistake during one of the rebases.
  148. stickies-v approved
  149. stickies-v commented at 1:59 pm on January 15, 2025: contributor
    ACK 430d5feee9b8be3a85e85696c449753783301c2b
  150. DrahtBot requested review from ryanofsky on Jan 15, 2025
  151. in src/util/overflow.h:56 in 7993f26173 outdated
    48@@ -47,4 +49,41 @@ template <class T>
    49     return i + j;
    50 }
    51 
    52+/**
    53+ * @brief Left bit shift with overflow checking.
    54+ * @param input The input value to be left shifted.
    55+ * @param shift The number of bits to left shift.
    56+ * @return (intput * 2^shift) or nullopt if it would not fit in the return type.
    


    ryanofsky commented at 2:19 pm on January 15, 2025:

    In commit “util: Add integer left shift helpers” (7993f26173d911d3773cb2580938ac9aad8ad31b)

    intput (unless you want to coin a phrase)

  152. in src/test/fuzz/overflow.cpp:22 in 430d5feee9 outdated
    16+template <typename T, typename W>
    17+void TestOverflow(FuzzedDataProvider& fuzzed_data_provider)
    18+{
    19+    constexpr auto min{std::numeric_limits<T>::min()};
    20+    constexpr auto max{std::numeric_limits<T>::max()};
    21+    static_assert(min >= std::numeric_limits<W>::min() / 2);
    


    ryanofsky commented at 2:23 pm on January 15, 2025:

    In commit “fuzz: Add fuzz test for checked and saturating add and left shift” (b7c8d88ec2d299d2b19f5c622ee5080cfdf8b2a0)

    Could add comment to explain divide by two: “// Range needs to be at least twice as big to allow two numbers to be added without overflowing.” Apparently I have goldfish memory, I wrote this code yesterday and already forgot why it was there.

  153. ryanofsky approved
  154. ryanofsky commented at 2:40 pm on January 15, 2025: contributor

    Code review ACK 430d5feee9b8be3a85e85696c449753783301c2b. Changes since last review were adding fuzz test and changing comments as suggested and switching back from digits() to sizeof().

    This all looks good, though I was curious about two things. (No need to investigate these, just curious if there is any more information)

    • This is moot but I was wondering about the “I had a compiler complain about signed to unsigned integer comparison without it.” #31483 (review) comment because I actually don’t know what signed to unsigned comparison would have been taking place there or why I wasn’t seeing it.

    • Curious what motivated change from digits() back to sizeof(). Both ways seem fine. I guess sizeof() matches c++ documentation more closely, but digits() looked a little cleaner and could maybe be better in a theoretical case where a type had some padding or custom integer types were supported.

  155. util: Add integer left shift helpers
    The helpers are used in the following commits to increase the safety of
    conversions during cache size calculations.
    
    Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
    Co-authored-by: stickies-v <stickies-v@protonmail.com>
    c03a2795a8
  156. fuzz: Add fuzz test for checked and saturating add and left shift
    Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
    d5e2c4a409
  157. kernel: Move kernel-specific cache size options to kernel
    Carrying non-kernel related fields in the cache sizes for the indexes is
    confusing for kernel library users. The cache sizes also are set
    currently with magic numbers in bitcoin-chainstate. The comments for the
    cache size calculations are also not completely clear.
    
    Solve these things by moving the kernel-specific cache size fields to
    their own struct.
    
    This slightly changes the way the cache is allocated if the txindex
    and/or blockfilterindex is used. Since they are now given precedence
    over the block tree db cache, this results in a bit less cache being
    allocated to the block tree db, coinsdb and coins caches. The effect is
    negligible though, i.e. cache sizes with default dbcache reported
    through the logs are:
    
    master:
    Cache configuration:
    * Using 2.0 MiB for block index database
    * Using 56.0 MiB for transaction index database
    * Using 49.0 MiB for basic block filter index database
    * Using 8.0 MiB for chain state database
    * Using 335.0 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
    
    this branch:
    Cache configuration:
    * Using 2.0 MiB for block index database
    * Using 56.2 MiB for transaction index database
    * Using 49.2 MiB for basic block filter index database
    * Using 8.0 MiB for chain state database
    * Using 334.5 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
    e758b26b85
  158. kernel: Move non-kernel db cache size constants
    These have nothing to do with the txdb, so move them out and into the
    node caches.
    8826cae285
  159. kernel: Move default cache constants to caches
    They are not related to the txdb, so a better place for them is the
    new kernel and node cache file. Re-use the default amount of kernel
    cache for the default node cache.
    65cde3621d
  160. init: Use size_t consistently for cache sizes
    This avoids having to rely on implicit casts when passing them to the
    various functions allocating the caches.
    
    This also ensures that if the requested amount of db_cache does not fit
    in a size_t, it is clamped to the maximum value of a size_t.
    
    Also take this opportunity to make the total amounts of cache in the
    chainstate manager a size_t too.
    2a92702baf
  161. TheCharlatan force-pushed on Jan 15, 2025
  162. TheCharlatan commented at 3:32 pm on January 15, 2025: contributor

    Updated 430d5feee9b8be3a85e85696c449753783301c2b -> 2a92702bafca5c78b270a9502a22cb9deac02cfc (kernel_cache_sizes_16 -> kernel_cache_sizes_17, compare)

    Curious what motivated change from digits() back to sizeof(). Both ways seem fine. I guess sizeof() matches c++ documentation more closely

    Yes, that was my motivation for the change back to sizeof().

  163. stickies-v commented at 3:42 pm on January 15, 2025: contributor

    re-ACK 2a92702bafca5c78b270a9502a22cb9deac02cfc

    No changes except for outstanding review comments to simplify CheckedLeftShift()’s implementation wrt range checks, making the _MiB test platform-agnostic, making IndexCacheSizes members size_t and doc/typo improvements.

  164. DrahtBot requested review from ryanofsky on Jan 15, 2025
  165. ryanofsky approved
  166. ryanofsky commented at 8:40 pm on January 15, 2025: contributor
    Code review ACK 2a92702bafca5c78b270a9502a22cb9deac02cfc. Changes since last review are fixing size options to use size_t instead of int64_t again, simplifying CheckedLeftShift more, and making other minor suggested cleanups
  167. in src/util/overflow.h:62 in 2a92702baf
    57+ */
    58+template <std::integral T>
    59+constexpr std::optional<T> CheckedLeftShift(T input, unsigned shift) noexcept
    60+{
    61+    if (shift == 0 || input == 0) return input;
    62+    // Avoid undefined c++ behaviour if shift is >= number of bits in T.
    


    hodlinator commented at 8:43 am on January 16, 2025:

    mega-nit in case you re-touch:

    0    // Avoid undefined C++ behaviour if shift is >= number of bits in T.
    
  168. in src/test/fuzz/overflow.cpp:27 in 2a92702baf
    22+    static_assert(min >= std::numeric_limits<W>::min() / 2);
    23+    static_assert(max <= std::numeric_limits<W>::max() / 2);
    24+
    25+    auto widen = [](T value) -> W { return value; };
    26+    auto clamp = [](W value) -> W { return std::clamp<W>(value, min, max); };
    27+    auto check = [](W value) -> std::optional<W> { if (value >= min && value <= max) return value; else return std::nullopt; };
    


    hodlinator commented at 9:03 am on January 16, 2025:

    nit: Naked else without braces is contrary to developer-notes.md:

    https://github.com/bitcoin/bitcoin/blob/98939ce7b7441cc95b8fda4d502f182aab1d6a32/doc/developer-notes.md#L76-L79

    Consider using ternary operator:

    0    auto check = [](W value) -> std::optional<W> { return (value >= min && value <= max) ? std::optional{value} : std::nullopt; };
    
  169. in src/node/caches.cpp:32 in 8826cae285 outdated
    29+    sizes.tx_index = std::min(nTotalCache / 8, args.GetBoolArg("-txindex", DEFAULT_TXINDEX) ? MAX_TX_INDEX_CACHE << 20 : 0);
    30     nTotalCache -= sizes.tx_index;
    31     if (n_indexes > 0) {
    32-        int64_t max_cache = std::min(nTotalCache / 8, max_filter_index_cache << 20);
    33+        int64_t max_cache = std::min(nTotalCache / 8, MAX_FILTER_INDEX_CACHE << 20);
    34         sizes.filter_index = max_cache / n_indexes;
    


    hodlinator commented at 11:03 am on January 16, 2025:
    nit in commit 8826cae285490439dc1f19b25fa70b2b9e62dfe8: (We’re assigning what seems to be an int64_t to size_t filter_index here. I tried adding curlies on the right side of the assignment and compiling for 32-bit but was unable to trigger narrowing errors. Maybe the compiler is clever enough to see that (MAX_FILTER_INDEX_CACHE << 20) < std::numeric_limits<size_t>::max()).
  170. hodlinator approved
  171. hodlinator commented at 1:13 pm on January 16, 2025: contributor

    re-ACK 2a92702bafca5c78b270a9502a22cb9deac02cfc

    • Excellent with added fuzz test.
    • Added support for shifting negative numbers.
    • Adopting size_t earlier for some cases, reducing churn.

    Succeeded building on 32-bit (CentOS).


    Nit: commit message 65cde3621dbb9ac7d210d4926e7601c4adf5f498

    0- kernel: Move default cache constants to caches
    1+ kernel: Move default cache constants to caches.h
    
  172. fanquake merged this on Jan 16, 2025
  173. fanquake closed this on Jan 16, 2025

  174. TheCharlatan referenced this in commit 230a439a4a on Jan 17, 2025

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-01-21 06:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me