Sync problem crashes Bitcoin: EXCEPTION: St9bad

Smiggel commented at 8:15 am on February 25, 2017: none

I have installed core 0.14.0rc1 after running my node (Raspberry Pi 2) successfully on 0.13.2 for weeks .

After upgrading I am getting errors and having issues keeping the node running. Once in a while core crashes with this error in the terminal:

EXCEPTION: St9bad_alloc
std::bad_alloc
bitcoin in scheduler

terminate called after throwing an instance of ‘std::bad_alloc’ what(): std::bad_alloc

This happens when I use previously downloaded blockchain and also when I start to download the whole blockchain all over again.

My system settings: Raspberry Pi 2 4GB SD card NFS share to Synology NAS (blockchain is successfully stored on the nas). Have a backup on an 128GB USB Stick in the pi.

laanwj commented at 8:41 am on February 25, 2017: member

It means your node is out of memory. You need to either add more swap space, and/or reduce the memory requirements of your node by decreasing the -dbcache or the mempool size.

laanwj closed this on Feb 25, 2017

laanwj commented at 8:42 am on February 25, 2017: member

Reopening this as it may be an issue with higher (default) memory usage on 0.14

laanwj reopened this on Feb 25, 2017

laanwj added the label Resource usage on Feb 25, 2017

Smiggel commented at 8:53 am on February 25, 2017: none

@laanwj Would like to give it a try with other settings. Perhaps that can help. My Pi has 1GB memory. It doesn’t use all of it. About 50% when core is syncing. My swapfile is set to 1024 in /etc/dphys-swapfile and my db cache in bitcoin.conf is set to 64.

sipa commented at 9:01 am on February 25, 2017: member

Can you give your full bitcoin.conf (if any), and an excerpt at the end of debug.log?

Smiggel commented at 9:16 am on February 25, 2017: none

@sipa My bitcoin.conf: `listen=1 bind=0.0.0.0 port=8333 maxconnections=64 upnp=1

dbcache=64 par=2 checkblocks=24 checklevel=0

disablewallet=1

rpcbind=127.0.0.1 rpcport=8332 rpcallowip=127.0.0.1

#maxuploadtarget=144

server=1 testnet=0 `

Debug.log: http://gofile.me/2I87y/U8JPyyl2L

sipa commented at 9:19 am on February 25, 2017: member

Can you try reducing the maximum mempool as well? (maxmempool=64, for example)

Smiggel commented at 9:22 am on February 25, 2017: none

@sipa Done. I am starting the node again. Going to keep an eye on it.

[update] Node is syncing now. It’s 3 years and 49 weeks behind now. Lot of work to do. Hopefully it keeps running now.

Smiggel commented at 2:04 pm on February 25, 2017: none

@laanwj @sipa It seems the maxmempool limit fixed the issue. Node is still running, where it used to crash already before. Also memory usage is more stable now.

I’ll keep monitoring it closely, but so far it seems fixed.

theuni commented at 4:18 pm on February 25, 2017: member

@Smiggel Did it always crash with the same error? Specifically this part:

0bitcoin in scheduler

That’s interesting because we’re dumping the addrdb/bandb to disk. I wouldn’t be surprised if we make a few copies of them in the process.

Smiggel commented at 5:12 pm on February 25, 2017: none

@theuni The other error I had before was

EXCEPTION: St9bad_alloc
std::bad_alloc
bitcoin in ProcessMessages()

Other then that I didn’t get errors. Right now it’s still running and still stable.

GameKyuubi commented at 4:16 pm on March 2, 2017: none

0************************
1EXCEPTION: St9bad_alloc
2std::bad_alloc
3bitcoin in ProcessMessages()

bitcoin.conf:

0dbcache=64
1mempool=64
2prune=550

AWS Ubuntu 16.0.4 Micro instance bitcoind v0.14.0rc1 1gb ram 0 swap 15gb HDD space left

It’s also been killed silently because of memory usage with the same config.

Smiggel commented at 4:42 pm on March 2, 2017: none

@GameKyuubi Does it run stable when you set the maxmempool limit? Ever since I set it to 64 my node runs stable. All crashes are gone.

GameKyuubi commented at 5:15 pm on March 2, 2017: none

Whoops, syntax error in my conf

jnewbery commented at 6:45 pm on March 2, 2017: member

Out of interest, what was the syntax error in your conf?

sipa commented at 6:48 pm on March 2, 2017: member

mempool instead of maxmempool

GameKyuubi commented at 4:54 pm on March 5, 2017: none

@Smiggel Alright, so bitocoind is still getting killed even with my change to the config.

0/var/log/syslog.1:Mar  4 16:14:49 ip-172-31-34-248 kernel: [126646.452271] Out of memory: Kill process 6336 (bitcoind) score 779 or sacrifice child
1/var/log/syslog.1:Mar  4 16:14:49 ip-172-31-34-248 kernel: [126646.458536] Killed process 6336 (bitcoind) total-vm:1270232kB, anon-rss:387052kB, file-rss:0kB

GameKyuubi commented at 9:42 am on March 8, 2017: none

Any suggestions for a fix or acknowledgement that this is a real issue yet?

sipa commented at 6:56 pm on March 8, 2017: member

@GameKyuubi It looks like bitcoind was only using 387 MB of memory. That is totally expected. If that’s enough to get it killed on your system, you must be using too much memory for other things.

GameKyuubi commented at 6:11 am on March 10, 2017: none

@sipa is there a clear description of how much memory bitcoind is supposed to use with maxmempool=64 and dbcache=64? This was on a 512mb VPS with nothing else running except tmux and htop.

laanwj commented at 7:01 am on March 10, 2017: member

512MB is not enough for running bitcoind. 1GB is absolutely the lower bound, and even then you may have to tweak settings.

GameKyuubi commented at 12:08 pm on March 12, 2017: none

Noted. Back to 1GB, no problems so far.

Smiggel commented at 11:52 am on May 7, 2017: none

Noted. Back to 1GB, no problems so far.

Is that on a Raspberry Pi?

Leviathn commented at 7:48 am on September 12, 2017: none

This appears to have resolved – unless necessary as a warning re - minimum memory usage.. @gamekyuubi I assume things are still going well? Feel free to report if upgrade to .15 went smoothly as well.

fanquake closed this on Oct 8, 2017

rebroad commented at 2:57 am on May 24, 2020: contributor

out of interest, have any pulls since this issue addressed this issue to handle running out of memory more elegantly?

DrahtBot locked this on Feb 15, 2022

Sync problem crashes Bitcoin: EXCEPTION: St9bad_alloc #9857