Bitcoind 0.14.0rc1: OOM -> block marked invalid #9854

1. 
   sipa commented at 10:18 pm on February 24, 2017: member

   Describe the issue

   My bitcoind v0.14.0rc1 running on a Raspberry Pi 3, OOMed while processing a block, but didn’t shutdown. Instead, it simply marked the block as invalid, and continued running.

   What version of bitcoin-core are you using?

   02017-02-22 17:08:06 Bitcoin version v0.14.0.0-7d75a5a
   

   Any extra information that might be useful in the debugging process.

   bitcoin.conf:

   0dbcache=500
   1blocksonly=1
   2debug=bench
   

   debug.log:

    0017-02-24 10:39:06 UpdateTip: new best=00000000000000002c54efc91a9c88f030ac8be8b07ccce1fd743769fcee0d7b height=316694 version=0x00000002 log2_work=80.285579 tx=44994328 date='2014-08-20 21:49:12' progress=0.226366 cache=494.0MiB(428034tx)
    12017-02-24 10:39:06   - Connect postprocess: 1.04ms [364.74s]
    22017-02-24 10:39:06 - Connect block: 28.42ms [31745.07s]
    32017-02-24 10:39:06 Pre-allocating up to position 0x8000000 in blk00167.dat
    42017-02-24 10:39:08 Prune: UnlinkPrunedFiles deleted blk/rev (00161)
    52017-02-24 10:39:15
    6
    7************************
    8EXCEPTION: St9bad_alloc
    9std::bad_alloc
   10bitcoin in ProcessMessages()
   11
   122017-02-24 10:39:15 ProcessMessages(block, 300905 bytes) FAILED peer=345
   132017-02-24 10:39:40   - Load block from disk: 0.01ms [3425.64s]
   142017-02-24 10:39:40     - Sanity checks: 0.27ms [554.49s]
   152017-02-24 10:39:40     - Fork checks: 0.45ms [1527.74s]
   162017-02-24 10:39:40 ERROR: ConnectBlock(): inputs missing/spent
   172017-02-24 10:39:40 Misbehaving: 92.27.214.6:8333 peer=377 (0 -> 100) BAN THRESHOLD EXCEEDED
   182017-02-24 10:39:40 InvalidChainFound: invalid block=0000000000000000105e43d3721802e8799ae33a2f940123a1f500de53fc0a53  height=316695 log2_work=80.285679  date=2014-08-20 22:31:56
   192017-02-24 10:39:40 InvalidChainFound:  current best=00000000000000002c54efc91a9c88f030ac8be8b07ccce1fd743769fcee0d7b  height=316694 log2_work=80.285579  date=2014-08-20 21:49:12
   202017-02-24 10:39:40 ERROR: ConnectTip(): ConnectBlock 0000000000000000105e43d3721802e8799ae33a2f940123a1f500de53fc0a53 failed
   212017-02-24 10:39:40 InvalidChainFound: invalid block=0000000000000000105e43d3721802e8799ae33a2f940123a1f500de53fc0a53  height=316695  log2_work=80.285679  date=2014-08-20 22:31:56
   222017-02-24 10:39:40 InvalidChainFound:  current best=00000000000000002c54efc91a9c88f030ac8be8b07ccce1fd743769fcee0d7b  height=316694 log2_work=80.285579  date=2014-08-20 21:49:12
   232017-02-24 10:39:40 connect() to [2a02:810c:83c0:20c0:eb1f:a96b:1b81:813a]:8333 failed: Network is unreachable (101)
   

2. 
   theuni commented at 11:20 pm on February 24, 2017: member

   Here’s one approach: skip dealing with bad_alloc and terminate immediately: https://github.com/theuni/bitcoin/commit/28afe574567ba838d46959047282460dbab39b91

   Edit: I’m uneasy with all approaches that I’ve come up with, but bailing immediately seems like the least likely thing to cause issues. I believe in the case above, at least, this would have worked.

   Edit2: prevector calls malloc/realloc directly. So we’d either need to switch to new[] there, or call std::get_new_handler() directly.

3. fanquake added the label Bug on Feb 25, 2017
4. fanquake added this to the milestone 0.14.0 on Feb 25, 2017
5. 
   TheBlueMatt commented at 0:07 am on February 25, 2017: member

   One possible way to explain this (from IRC):

   “10:39:08 Prune:” is the prune called by AcceptBlock’s FlushStateToDisk call (seems obvious given the Pre-allocating blk just above)….now, that flush call hit fDoFullFlush triggering a pcoinsTip->Flush(). OK, so now in CCoinsViewDB::BatchWrite we iterate over the entries in mapCoins, allocating memory for each one in the leveldb batch, erasing AS WE GO…at some point it ran out of memory, throwing bad_alloc and leaving pcoinsTip in an inconsistent state, but std::bad_alloc is not an std::runtime_error so the catch at the end of FlushStateToDisk didnt catch it. OK, phew, so now we go to connect the next block and fail to connect its inputs (without allocating new disk space). I /think/ that explains your debug.log without hand waving about hardware errors

6. 
   laanwj commented at 11:35 am on February 28, 2017: member

   Worked around by #9856
7. laanwj closed this on Feb 28, 2017

   ---

8. DrahtBot locked this on Sep 8, 2021

Contributors
sipa theuni TheBlueMatt laanwj

Labels
Bug

Milestone
0.14.0