b-msghand invoked oom-killer: Master (v28.99) crashing during IBD #31561

1. 
   pinheadmz commented at 12:57 pm on December 24, 2024: member

   Running master (specifically 318b2a2f90dfd8ae20beca58e55e6d240a7f3d27 which is HEAD of #31415) on Ubuntu 24.10 on a digital ocean droplet with 8GB RAM. Crashing regularly during IBD. Logs always end like this:

   0
   12024-12-23T20:39:07Z [validation] Enqueuing BlockConnected: block hash=000000000000000000008364a7396ba08c5ab141d10136e3ad0d9fa37899d7b0 block height=814565
   22024-12-23T20:39:07Z [validation] Enqueuing UpdatedBlockTip: new block hash=000000000000000000008364a7396ba08c5ab141d10136e3ad0d9fa37899d7b0 fork block hash=000000000000000000023c4f2b1f2457d2f46cd7b6cb27f87ab4788cd54adde1 (in IBD=true)
   32024-12-23T20:39:07Z [validation] ActiveTipChange: new block hash=000000000000000000008364a7396ba08c5ab141d10136e3ad0d9fa37899d7b0 block height=814565
   42024-12-23T20:39:07Z [bench]   - Load block from disk: 27.96ms
   52024-12-23T20:39:07Z [bench]     - Sanity checks: 3.86ms [20.30s (5.18ms/blk)]
   62024-12-23T20:39:07Z [bench]     - Fork checks: 0.13ms [1.19s (0.30ms/blk)]
   

   Configuration is just -txindex=1. Have also added -dbcache=100 and 4 GB swap which did not fix the issue.

   Logs only report Killed but there is OOM details in dmesg

   Seems to be the same issue as https://github.com/bitcoin/bitcoin/issues/30001

2. 
   maflcko commented at 1:27 pm on December 24, 2024: member

   Seems to be the same issue as #30001

   I don’t think they are the same. In fact, there are many differences: You are using bitcoind with -txindex and default settings, the other person was using the GUI with a large dbcache and external HDD for the blocksdir.

   It looks similar to #31041, though.

   Given that it is likely not heap (https://github.com/bitcoin/bitcoin/issues/31041#issuecomment-2514072312), it would be good to debug where the usage is coming from. Memory-mapped files seems unlikely as well, but would be good to rule out.

3. maflcko added the label Resource usage on Dec 24, 2024
4. maflcko added the label Linux/Unix on Dec 24, 2024
5. bitcoin deleted a comment on Dec 25, 2024
6. 
   pinheadmz commented at 7:48 pm on December 26, 2024: member

   The node in question finished sync after I ran it inside heaptrack and is humming along just fine now. I printed out the flamegraph although I have trouble interpreting it:

   

7. 
   davidgumberg commented at 8:23 pm on December 27, 2024: contributor

   The node in question finished sync after I ran it inside heaptrack and is humming along just fine now. I printed out the flamegraph although I have trouble interpreting it:

   [ flamegraph.svg]

   At first glance it seems unusual that so much time (~40% of samples) is spent in the call stack above CScriptCheck::operator() doing allocations.

Contributors
pinheadmz maflcko davidgumberg

Labels
Linux/Unix Resource usage