node: smooth oversized `dbcache` warnings #35200

l0rinc commented at 12:56 PM on May 3, 2026: contributor

Problem: The oversized -dbcache warning threshold has a sharp formula cliff when detected RAM crosses the cutoff used by the warning logic.

This was reported during review of #34641, where an earlier version could jump from the auto default at 4095 MiB RAM to 75% of RAM at 4096 MiB. That made 1 MiB of extra detected RAM raise the warning threshold from about 511 MiB to 3072 MiB.

The surviving warning-only code has the same shape at a different boundary. Below 2 GiB RAM the cap is DEFAULT_DB_CACHE (450 MiB), but at 2 GiB it switches to 75% of total RAM, so a tiny increase in detected RAM can suddenly raise the warning threshold from 450 MiB to about 1536 MiB.

Fix: Base the warning on a reserved non-dbcache memory budget instead:

\text{warn if } \mathit{dbcache} > \max\left(\mathit{DEFAULT\_DB\_CACHE}, 0.75 \cdot \max(\text{total RAM} - \mathit{DBCACHE\_WARNING\_RESERVED\_RAM}, 0)\right)

DBCACHE_WARNING_RESERVED_RAM is 2 GiB, so the fixed DEFAULT_DB_CACHE cap remains the floor below that reserve and the warning threshold grows monotonically above it.

This keeps the warning conservative around low-memory boundaries and avoids treating a boundary-crossing 1 MiB RAM difference as a reason to allow a much larger explicit -dbcache.

Quick reference:

System RAM	Previous warning cap	New warning cap
1 GiB	450 MiB	450 MiB
2 GiB	1536 MiB	450 MiB
3 GiB	2304 MiB	768 MiB
4 GiB	3072 MiB	1536 MiB
8 GiB	6144 MiB	4608 MiB
16 GiB	12288 MiB	10752 MiB
32 GiB	24576 MiB	23040 MiB

On 32-bit builds, effective -dbcache values are still capped to 1024 MiB before the warning check, so thresholds above that cap are not reachable there.

DrahtBot commented at 12:56 PM on May 3, 2026: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35200.

Reviews

See the guideline and AI policy for information on the review process.

Type	Reviewers
ACK	sedited, w0xlt, optout21

If your review is incorrectly listed, please copy-paste <code></code> into the comment that the bot should ignore.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#35205 (kernel,node: clean up dbcache helpers and add kernel API by l0rinc)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

DrahtBot added the label CI failed on May 3, 2026

DrahtBot commented at 1:45 PM on May 3, 2026: contributor

🚧 At least one of the CI tasks failed. <sub>Task i686, no IPC: https://github.com/bitcoin/bitcoin/actions/runs/25279789848/job/74115183233</sub> <sub>LLM reason (✨ experimental): CI failed because the CTest “Bitcoin Core Test Suite” had failing tests, specifically sock_tests (1 test failed).</sub>

<details><summary>Hints</summary>

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

</details>

l0rinc closed this on May 5, 2026

l0rinc reopened this on May 5, 2026

DrahtBot removed the label CI failed on May 5, 2026

l0rinc force-pushed on May 5, 2026

sedited commented at 2:34 PM on June 7, 2026: contributor

Is this really problem given that RAM size more or less follows a discrete distribution across deployments? I'm guessing the next common RAM configuration is probably 3GiB, which seems a more justifiable jump.

optout21 commented at 8:10 AM on June 18, 2026: contributor

reACK d164a043426e395a0b11feea2094c5f00c528202

Re-review after minor changes, no significant changes.

Prev: ACK b4f7fc3d92d21e3a58ccad67de2af8c56ca772f3 This change streamlines the formula used for warning about too large DB cache value set. The old formula with a discontinuity is changed to a linear formula, which is more intuitive, and has less risk of unintended consequences.

l0rinc commented at 12:11 PM on June 18, 2026: contributor

Is this really problem given that RAM size more or less follows a discrete distribution across deployments

On k8s clusters, people can set any weird memory value, so it kind of makes sense to make it more predictable. Also, we've encountered a few surprising cases with swapping when dbcache was too high (e.g. https://x.com/caesrcd/status/2062766249038250356). As it turns out, after a while, IBD just gets slower and slower with bigger dbcache values - many people reported that the ideal was 2-3 GB, after which validation just got slower. Also, running reindex-chainstate with 2 GB of total memory still works with roughly -dbcache=100 (the previous warning was at 1536 MiB; the new 450 MiB is closer).

in src/node/caches.h:38 in b4f7fc3d92

  34 | @@ -31,7 +35,8 @@ struct CacheSizes {
  35 |  CacheSizes CalculateCacheSizes(const ArgsManager& args, size_t n_indexes = 0);
  36 |  constexpr bool ShouldWarnOversizedDbCache(uint64_t dbcache, uint64_t total_ram) noexcept
  37 |  {
  38 | -    const uint64_t cap{(total_ram < 2_GiB) ? DEFAULT_DB_CACHE : (total_ram / 100) * 75};
  39 | +    const uint64_t headroom{total_ram > DBCACHE_WARNING_RESERVED_RAM ? total_ram - DBCACHE_WARNING_RESERVED_RAM : 0};

sedited commented at 1:25 PM on June 18, 2026:

Nit: headroom would imply this is the space left free, but it's the opposite, right? If so, how about calling this available or something along the lines?

w0xlt commented at 11:53 PM on July 8, 2026:

Yes, available_ram, available_for_dbcache, or ram_above_reserve would be clearer.

l0rinc commented at 6:11 PM on July 9, 2026:

Thanks, taken!

sedited approved

sedited commented at 1:42 PM on June 18, 2026: contributor

ACK b4f7fc3d92d21e3a58ccad67de2af8c56ca772f3

w0xlt commented at 12:00 AM on July 9, 2026: contributor

ACK b4f7fc3d92d21e3a58ccad67de2af8c56ca772f3

The old rule:

if RAM < 2 GiB: warn above 450 MiB if RAM >= 2 GiB: warn above 75% of RAM

That means crossing from just under 2 GiB to exactly 2 GiB raises the warning threshold from 450 MiB to 1536 MiB. One extra MiB of detected RAM should not make a 1.5 GiB dbcache suddenly look safe.

The new model is more coherent:

warn if dbcache > max(DEFAULT_DB_CACHE, 75% * (total_ram - 2 GiB))

nit: headroom can be renamed.

l0rinc commented at 12:23 AM on July 9, 2026: contributor

rfm?

DrahtBot added the label Needs rebase on Jul 9, 2026

node: smooth oversized `-dbcache` warnings

The oversized `-dbcache` warning currently switches from a fixed `450 MiB` threshold below `2 GiB` of RAM to `75%` of total RAM at `2 GiB`.
This creates a cliff where a small increase in RAM can raise the warning threshold to about `1536 MiB`.

Apply the `75%` factor only to RAM above a `2 GiB` reserve while keeping `DEFAULT_DB_CACHE` as the minimum threshold.
This removes the cliff: the threshold stays at the default until the percentage term exceeds it, then grows by `0.75 MiB` per additional MiB of RAM.

This also aligns better with the recently merged parallel input prevout fetcher which performs better with slightly lower dbcache memory.

Co-authored-by: Bortlesboat <Bortlesboat@users.noreply.github.com>

d164a04342

l0rinc force-pushed on Jul 9, 2026

l0rinc commented at 6:18 PM on July 9, 2026: contributor

Rebased to resolve the size_t -> uint64_t conflicts, applied rename suggestion, amended commit message to highlight the lower dbcache preference after #35295, ready for review again.

DrahtBot removed the label Needs rebase on Jul 9, 2026

sedited approved

sedited commented at 3:31 PM on July 10, 2026: contributor

Re-ACK d164a043426e395a0b11feea2094c5f00c528202

DrahtBot requested review from w0xlt on Jul 10, 2026

w0xlt commented at 11:36 PM on July 10, 2026: contributor

reACK d164a043426e395a0b11feea2094c5f00c528202

sedited commented at 7:51 AM on July 14, 2026: contributor

@optout21 please leave a new comment when you are re-ACKing. It's fine here, but in other pull requests it might be difficult to search for the correct comment. It also makes the notifications for me a bit easier to handle.

sedited merged this on Jul 14, 2026

sedited closed this on Jul 14, 2026

optout21 commented at 8:00 AM on July 14, 2026: contributor

please leave a new comment when you are re-ACKing

Noted, thanks for the heads-up. I know the PR author prefers in-place edit for minimal info re-a-c-k's, to not have notifications, but I assume notifications are more important for maintainers.

fanquake commented at 8:17 AM on July 14, 2026: member

prefers in-place edit for minimal info re-a-c-k's,

Yea, please don't do this. All it does it hide useful information.

maflcko commented at 8:17 AM on July 14, 2026: member

I have a really hard time reading the pull description, because it seems to show a graph about a completely different code change in a completely different pull request (https://github.com/bitcoin/bitcoin/pull/34641#discussion_r2900769756)

I think a pull description should briefly motivate and explain the changes in the pull request itself, not expand and recap threads unrelated to the topic.

I keep running into those (https://github.com/bitcoin/bitcoin/pull/35688#pullrequestreview-4661456996), and walk away from review more confused than before. Is it just me?

The change here seems trivial enough that a human could type a short 3-line (or so) self-contained pull description. I suspect, this may have been reviewed/merged faster this way.

alexanderwiederin referenced this in commit f8c77960a3 on Jul 14, 2026

l0rinc deleted the branch on Jul 14, 2026

l0rinc commented at 3:57 PM on July 14, 2026: contributor

seems trivial enough that a human could type a short 3-line (or so) self-contained pull description

Thanks for the feedback, I have started compressing my PR descriptions lately. I usually start off by writing out every detail that I found interesting or surprising, and as time passes I remove the parts that turn out not to be important after all. Keep the feedback coming.