Ubuntu Full Node Out of Memory Error #9812

1. 
   quentinlesceller commented at 4:22 pm on February 20, 2017: none

   Describe the issue

   After 24 hours running, Bitcoin daemon is killed because out of memory error.

   Can you reliably reproduce the issue?

   Cannot manually reproduce. The daemon is killed every 24 hours.

   Expected behaviour

   Bitcoind should not crash after 24 hours of uptime.

   Actual behaviour

   Bitcoind crashes because of Out of Memory Error

   What version of bitcoin-core are you using?

   Bitcoin 0.13.2, ubuntu PPA.

   Machine specs:

   Amazon AWS. EC2 t2.medium.

   • OS: Ubuntu 16.04
   • CPU: 2 vCPU
   • RAM: 4 GB RAM (and 8 GB of swap)
   • Disk size: 140 GB
   • Disk Type (HD/SDD): SSD

   Any extra information that might be useful in the debugging process.

   Crash happened at (20:32).

   • syslog
   • debug.log

   BitcoinTalk post.

2. laanwj added the label Resource usage on Feb 20, 2017
3. 
   laanwj commented at 4:36 pm on February 20, 2017: member

   4GB ram and 8GB swap should be more than enough to run in the default settings.

   Did you set anything in bitcoin.conf that may affect memory usage?

4. 
   quentinlesceller commented at 4:41 pm on February 20, 2017: none

   Anonymized bitcoin.conf. Only modifications are 3 allowed IPs for RPC and server=1.
5. 
   paveljanik commented at 5:46 pm on February 20, 2017: contributor

   What else is running on the machine? I think the issue is outside to the bitcoind.
6. 
   quentinlesceller commented at 6:32 pm on February 20, 2017: none

   Here is a list of the process running on the machine at the time of the crash.
7. 
   TheBlueMatt commented at 7:05 pm on February 20, 2017: member

   bitcoind isnt using a ton of memory in that list (though, to be fair, nor is anything else). Looks like you might be running in docker - are there any docker-imposed memory limits that might affect this? It looks like bitcoind has an rss of around 400MB, which is reasonable, though VM size much higher, which is expected but shouldn’t be an issue (unless you have overcommit off?). Still, your system log shows that your swap was mostly unused at the time bitcoind was killed, which seems strange unless there was something preventing swap from being used.
8. 
   quentinlesceller commented at 7:18 pm on February 20, 2017: none

   bitcoind is not running in a docker. Only apache2 is. free -m output is the following:

   0              total        used        free      shared  buff/cache   available
   1Mem:           3950         433          41          10        3475        3433
   2Swap:          8191          48        8143
   

   Seems swap is properly configured?

9. 
   TheBlueMatt commented at 7:49 pm on February 20, 2017: member

   Unclear - the fact that oomkiller was invoked when you had “Free swap = 8350600kB” seems strange. If, as you seem to have indicated, this issue happens roughly once every 24 hours, it’d be great if you could get a copy of /proc/${BITCOIND_PID}/smaps just before, or soon before the oom-killer kills bitcoind, as well as possibly bitcoin-cli getmemoryinfo (if you want to try 0.14.0rc1).
10. 
    quentinlesceller commented at 8:28 pm on February 20, 2017: none

    I have setup a python script to log /proc/${BITCOIND_PID}/smaps. Will let you know once I have a crash.
11. 
    quentinlesceller commented at 9:35 pm on March 2, 2017: none

    Issue not happening anymore, closing.
12. quentinlesceller closed this on Mar 2, 2017

    ---

13. DrahtBot locked this on Sep 8, 2021

Contributors
quentinlesceller laanwj paveljanik TheBlueMatt

Labels
Resource usage