ProcessMessages exception on Pine64 #8044

jonasschnelli commented at 8:17 PM on May 11, 2016: contributor

Just had a St9bac_alloc exception in ProcessMessages() during IBD:

Computer: Pine64 2GB Processor: AArch64 rev 4 Storage: USB stick

2016-05-11 19:14.26 ProcessMessages(block, 308961 bytes) FAILED peer=264
bitcoind: main.cpp:2237: bool ConnectBlock(const CBlock&, CValidationState&, CBlockIndex*, CCoinsViewCache&, const CChainParams&, bool): Assertion 'hashPrevBlock == view.GetBestBlock()'  failed

Edit: the debug log output was from a different session (I used -printtoconsole).

jonasschnelli added the label Block storage on May 12, 2016

laanwj commented at 9:42 AM on May 12, 2016: member

bad_alloc means that memory is full. Add swap or reduce the dbcache size :-)

jonasschnelli commented at 9:44 AM on May 12, 2016: contributor

I though 2GB ram, minimal linux and dbcache=1500 should do it. But let me try to use dbcache=1000.

laanwj commented at 10:26 AM on May 12, 2016: member

Ideally a bad_alloc would close the application immediately with a clearer error though See also #7228, which is even worse as there it simply continues, rejecting a block.

jonasschnelli commented at 3:40 PM on May 14, 2016: contributor

The main problem with the bad_alloc crash is that the database was/is corrupted afterwards. Either we should implement something like #8037 or try to graceful close in low-men situations (maybe impossible).

jonasschnelli commented at 7:32 AM on May 26, 2016: contributor

Again a St9bad_alloc at progress=0.901800. RAM: 2GB. DBCache: 800MB.

Node running in a gnu-screen:

ubuntu@pine64:~/bitcoin$ ./src/bitcoind --dbcache=800 --datadir=/disk2/fullnode/ --prune=550


************************
EXCEPTION: St9bad_alloc
std::bad_alloc
bitcoin in ProcessMessages()

bitcoind: main.cpp:2237: bool ConnectBlock(const CBlock&, CValidationState&, CBlockIndex*, CCoinsViewCache&, const CChainParams&, bool): Assertion `hashPrevBlock == view.GetBestBlock()' failed.
Aborted

More details:

ubuntu@pine64:~$ tail -f /disk2/fullnode/debug.log 
2016-05-25 17:30:53 skip verificationUpdateTip: new best=00000000000000000145cb386289ab8f9015246d226431a4b5de6da87f524c79 height=393014 version=0x00000004 log2_work=83.900934 tx=103124904 date='2016-01-12 14:55:45' progress=0.901795 cache=589.5MiB(357746tx)
2016-05-25 17:30:54 skip verificationUpdateTip: new best=0000000000000000075e9a75e8d44ba9e512e96d05a70a3159b13ad62eef1066 height=393015 version=0x00000004 log2_work=83.90097 tx=103126917 date='2016-01-12 15:00:52' progress=0.901800 cache=590.4MiB(359388tx)
2016-05-25 17:30:58 skip verification

************************
EXCEPTION: St9bad_alloc       
std::bad_alloc       
bitcoin in ProcessMessages()       

2016-05-25 17:31:12 ProcessMessages(block, 248665 bytes) FAILED peer=31
^C
ubuntu@pine64:~$ free
              total        used        free      shared  buff/cache   available
Mem:        2036960       49056     1731308       16744      256596     1939369
Swap:             0           0           0
ubuntu@pine64:~$

RAM failure?

paveljanik commented at 7:35 AM on May 26, 2016: contributor

Calling free after the memory has been deallocated by the kernel doesn't provide any information. Call free periodically in other screen...

jonasschnelli commented at 7:56 AM on May 26, 2016: contributor

@paveljanik: I did paste the free output only to prove the system total memory (2GB).

sipa commented at 12:18 PM on May 26, 2016: member

@jonasschnelli It may be useful to monitor memory usage while the application is running too...

MarcoFalke commented at 6:23 PM on December 22, 2018: member

@jonasschnelli Closing for now, since there hasn't been an update.

MarcoFalke closed this on Dec 22, 2018

DrahtBot locked this on Sep 8, 2021