Raise maximum -dbcache setting #28249

1. 
   Sjors commented at 10:43 am on August 10, 2023: member

   Please describe the feature you’d like to see added.

   nMaxDbCache was set to 16 GB back in 2015: https://github.com/bitcoin/bitcoin/commit/b3ed4236beb7f68e1720ceb3da15e0c3682ef629#diff-d102b6032635ce90158c1e6e614f03b50e4449aa46ce23370da5387a658342fd

   The UTXO set has been growing significantly over the past few months so we should probably raise it.

   Is your feature related to a problem, if so please describe it.

   Doing IBD without flushing dbcache is faster. That said, I’m not sure how much one single flush matters.

   Describe the solution you’d like

   Make nMaxDbCache bigger.

   Describe any alternatives you’ve considered

   No response

   Please leave any additional context

   Much growth recently: https://statoshi.info/d/000000009/unspent-transaction-output-set?orgId=1&from=now%2Fy&to=now&refresh=5s&viewPanel=8

2. Sjors added the label Feature on Aug 10, 2023
3. 
   Sjors commented at 10:43 am on August 10, 2023: member

   cc @sipa
4. 
   jonatack commented at 3:24 pm on August 10, 2023: member

   There is some related discussion and testing in #6102 containing the commit you linked to and more recently in #25325.
5. 
   russeree commented at 10:58 pm on August 15, 2023: contributor

   Would the eBPF USDT tracepoints for the UTXO cache be of use here as a way to measure the overall impact of these flush operations during an IBD or reindex operation?

   https://github.com/bitcoin/bitcoin/tree/master/contrib/tracing#log_utxocache_flushpy

6. 
   sipa commented at 3:50 am on August 16, 2023: member

   @Sjors When doing a full IBD from genesis with dbcache set to 16 GiB, is the limit ever reached (or gotten close to)? If not, it’s probably unnecessary to raise it.
7. 
   Sjors commented at 8:27 am on August 16, 2023: member

   @sipa I ran an IBD as part of testing #27596. Since I did a restart after the cache initially hit 1 GB, the data is not accurate, but the cache reached 13,334 MiB at block at block 798,000. A month earlier at 794,000 the log shows it as 12,766 MiB. So the memory footprint is growing ~7 GB per year (and that’s not even the worst month, see chart I linked to above).
8. 
   russeree commented at 8:58 am on August 16, 2023: contributor

   I am re running an -reindex=1 with tracepoints watching for flushes.

   Here is my last log entry for cache size height 800673 size 14,045 MiB

   2023-08-11T12:56:23Z UpdateTip: new best=00000000000000000002b6e22bae3b1f28339b6275c02ec9cbb6f19ffd9c7e9c height=800673 version=0x3017a000 log2_work=94.326952 tx=871155372 date='2023-07-28T20:42:50Z' progress=0.995044 cache=14045.6MiB(115054304txo)

   Ran Again 2023-08-16T15:41:07Z UpdateTip: new best=000000000000000000000ff0f3d17f4d92a0534cb63ba8e7293d21ea09932997 height=803380 version=0x20006000 log2_work=94.361871 tx=879733788 date='2023-08-16T01:52:09Z' progress=0.999792 cache=14590.0MiB(119818570txo)

9. 
   sipa commented at 12:01 pm on August 28, 2023: member

   Seems reasonable to increase it to 32 GB in that case. I’m slightly worried about people setting it to the max “because they can, and higher is better” and then getting hit with an unexpectedly slow flush time at the end, but if we’re close to hitting the maximum, then it’s clearly not crazy to want to set it higher.
10. Sjors referenced this in commit d65e467b94 on Aug 28, 2023
11. 
    Sjors commented at 4:05 pm on August 28, 2023: member

    Opened a PR #28358 to ~double~ quadruple it.
12. Sjors referenced this in commit 2da750ab50 on Aug 28, 2023
13. Sjors referenced this in commit 2a6c94841a on Feb 13, 2024
14. Sjors referenced this in commit 77b129bf8e on Feb 22, 2024
15. Sjors referenced this in commit c016405dad on Feb 22, 2024
16. Sjors referenced this in commit 5f313495ee on Mar 12, 2024
17. Sjors referenced this in commit 2a71819af6 on May 14, 2024
18. Sjors referenced this in commit bb3b980dfd on May 28, 2024
19. achow101 closed this on Sep 16, 2024

    ---

20. achow101 referenced this in commit fce9e065c1 on Sep 16, 2024

Contributors
Sjors jonatack russeree sipa

Labels
Feature