Node stuck with repeated “Cache size exceeds total space” log message #27599

1. 
   fjahr commented at 5:17 pm on May 8, 2023: contributor

   Is there an existing issue for this?

   • I have searched the existing issues

   Current behaviour

   Bug report from IRC bitcoin-core-dev: The node stopped updating the chain tip while printing “Cache size exceeds total space” multiple times per second. Node had to be restarted after “stale tip” messages, then resumed to work normally.

   Start of conversation: https://bitcoin-irc.chaincode.com/bitcoin-core-dev/2023-05-08#920380; User logs: https://jb55.com/s/58818ccccfb21d95.txt

   Expected behaviour

   Node should not require restart

   Steps to reproduce

   Unclear

   Relevant log output

   See link in description

   How did you obtain Bitcoin Core

   Compiled from source

   What version of Bitcoin Core are you using?

   tbd

   Operating system and version

   tbd

   Machine specifications

   HDD disk

2. 
   fjahr commented at 5:22 pm on May 8, 2023: contributor

   @jb55 just repeating the open questions from IRC here:

   • Do you have a custom maxmempool setting?
   • Do you have the last UpdateTip line your node saw before the cache size messages? the end of those UpdateTip lines shows how big your coins cache was at the end of block validation (@sdaftuar )
   • Did this happen around the time when we had the 2 block reorg ~20h ago, i.e. around 788685-788688 (not sure what timezone you are in with those logs)?
   • Can you say which version and OS you are on as well?

   Thanks!

3. 
   fjahr commented at 5:48 pm on May 8, 2023: contributor

   @jb55 just repeating the open questions from IRC here:

   • Do you have a custom maxmempool setting?
   • Do you have the last UpdateTip line your node saw before the cache size messages? the end of those UpdateTip lines shows how big your coins cache was at the end of block validation (@sdaftuar )
   • Did this happen around the time when we had the 2 block reorg ~20h ago, i.e. around 788685-788688 (not sure what timezone you are in with those logs)?
   • Can you say which version and OS you are on as well?

   Thanks!

   From IRC: “sdaftuar: unfortunately no this is just a tmux backbuffer since I was rebuilding my utxo due to some kind of leveldb corruption. wasn’t my main systemd logs. @fjahr: I don’t have any custom maxmempool settings. This started happening 20hrs ago yes when mempool was hot”

4. 
   jaonoctus commented at 11:43 pm on October 4, 2023: none

   same here, not sure if it’s related.

   I just changed the dbcache to 1000, and I do have 8GB on this machine

5. 
   BebeSparkelSparkel commented at 6:45 pm on October 26, 2023: none

   I am having the same problem. When I start bitcoind it logs UpdateTip continuously and uses a huge amount of cache. Currently, progress=0.09 and the cache is already 2.5GB. It seems that when the tip is updated the cache memory is not released.
6. 
   BebeSparkelSparkel commented at 8:56 pm on October 26, 2023: none

   The cache uses seems to have have stabilized now that the progress has reached 0.3 at around 7.3GB. Seems that with the earlier blocks it does not empty the cache properly but with later blocks it does.
7. 
   sipa commented at 8:57 pm on October 26, 2023: member

   The cache is flushed whenever it grows too large. Earlier blocks are smaller and thus don’t grow the cache as fast.
8. 
   BebeSparkelSparkel commented at 9:58 pm on October 26, 2023: none

   The cache is flushed whenever it grows too large. Earlier blocks are smaller and thus don’t grow the cache as fast.

   You are right. Makes sense now.

   0best=000000000000000000005a92fbb18bc22d75d3a1c3c620c63c4af381c0a02072 height=774031 version=0x2e76c000 log2_work=93.972720 tx=800236955 date='2023-01-28T14:36:22Z' progress=0.997035 cache=726.0MiB(5447153txo)
   12023-02-05T10:59:56Z Cache size (762055408) exceeds total space (761373440)
   22023-02-05T10:59:56Z FlushStateToDisk: write coins cache to disk (5452863 coins, 762055kB) started
   32023-02-05T11:00:22Z connect() to 192.168.1.218:8333 failed: Permission denied (13)
   42023-02-05T11:00:35Z FlushStateToDisk: write coins cache to disk (5452863 coins, 762055kB) completed (39.36s)
   52023-02-05T11:00:35Z UpdateTip: new best=000000000000000000046a95c1c13aac123e804590d12be55cce7dc881bc36cf height=774032 version=0x20800000 log2_work=93.972732 tx=800240556 date='2023-01-28T15:10:08Z' progress=0.997044 cache=50.2MiB(0txo)
   

Contributors
fjahr jaonoctus BebeSparkelSparkel sipa