Bitcoin core 0.15.1 started to crash

khelle commented at 3:50 pm on February 19, 2018: none

After about 2 months of running bitcoin core 0.15.1 it crashed, and now don’t want to restart. The logs shows me following info:

  02018-02-19 15:36:39 Bitcoin version v0.15.1.0-g7b57bc998f33
  12018-02-19 15:36:39 InitParameterInteraction: parameter interaction: -whitelistforcerelay=1 -> setting -whitelistrelay=1
  22018-02-19 15:36:39 Assuming ancestors of block 0000000000000000003b9ce759c2a087d52abc4266f8f4ebd6d768b89defa50a have valid signatures.
  32018-02-19 15:36:39 Setting nMinimumChainWork=000000000000000000000000000000000000000000723d3581fe1bd55373540a
  42018-02-19 15:36:39 Using the 'standard' SHA256 implementation
  52018-02-19 15:36:39 Using RdRand as an additional entropy source
  62018-02-19 15:36:39 Default data directory /home/btc/.bitcoin
  72018-02-19 15:36:39 Using data directory /home/btc/.bitcoin
  82018-02-19 15:36:39 Using config file /home/btc/.bitcoin/bitcoin.conf
  92018-02-19 15:36:39 Using at most 800 automatic connections (1024 file descriptors available)
 102018-02-19 15:36:39 Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
 112018-02-19 15:36:39 Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
 122018-02-19 15:36:39 Using 4 threads for script verification
 132018-02-19 15:36:39 scheduler thread start
 142018-02-19 15:36:39 Binding RPC on address 0.0.0.0 port 15501 failed.
 152018-02-19 15:36:39 HTTP: creating work queue of depth 2048
 162018-02-19 15:36:39 Config options rpcuser and rpcpassword will soon be deprecated. Locally-run instances may remove rpcuser to use cookie-based auth, or may be replaced with rpcauth. Please see share/rpcuser for rpcauth auth generation.
 172018-02-19 15:36:39 HTTP: starting 4 worker threads
 182018-02-19 15:36:39 init message: Verifying wallet(s)...
 192018-02-19 15:36:39 Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
 202018-02-19 15:36:39 Using wallet HOTWALLET.dat
 212018-02-19 15:36:39 CDBEnv::Open: LogDir=/home/btc/.bitcoin/database ErrorFile=/home/btc/.bitcoin/db.log
 222018-02-19 15:36:40 Cache configuration:
 232018-02-19 15:36:40 * Using 2.0MiB for block index database
 242018-02-19 15:36:40 * Using 8.0MiB for chain state database
 252018-02-19 15:36:40 * Using 4086.0MiB for in-memory UTXO set (plus up to 286.1MiB of unused mempool space)
 262018-02-19 15:36:40 init message: Loading block index...
 272018-02-19 15:36:40 Opening LevelDB in /home/btc/.bitcoin/blocks/index
 282018-02-19 15:36:40 Opened LevelDB successfully
 292018-02-19 15:36:40 Using obfuscation key for /home/btc/.bitcoin/blocks/index: 0000000000000000
 302018-02-19 15:36:44 LoadBlockIndexDB: last block file = 1184
 312018-02-19 15:36:44 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=33, size=34893824, heights=509914...509946, time=2018-02-19...2018-02-19)
 322018-02-19 15:36:44 Checking all blk files are present...
 332018-02-19 15:36:44 LoadBlockIndexDB: transaction index disabled
 342018-02-19 15:36:44 Opening LevelDB in /home/btc/.bitcoin/chainstate
 352018-02-19 15:36:44 Opened LevelDB successfully
 362018-02-19 15:36:44 Using obfuscation key for /home/btc/.bitcoin/chainstate: b9d65b264bd51a4c
 372018-02-19 15:36:44 Loaded best chain: hashBestChain=0000000000000000003d98b1c37b41fb04fb5ded589e0e3686aadc3d45ea9e22 height=509945 date=2018-02-19 14:38:27 progress=0.999964
 382018-02-19 15:36:44 init message: Rewinding blocks...
 392018-02-19 15:36:46 init message: Verifying blocks...
 402018-02-19 15:36:46 Verifying last 6 blocks at level 3
 412018-02-19 15:36:46 [0%]...[16%]...[33%]...[50%]...[66%]...[83%]...[99%]...[DONE].
 422018-02-19 15:36:47 No coin database inconsistencies in last 7 blocks (10803 transactions)
 432018-02-19 15:36:47  block index            7149ms
 442018-02-19 15:36:47 init message: Loading wallet...
 452018-02-19 15:36:49 nFileVersion = 150100
 462018-02-19 15:36:49 Keys: 22706 plaintext, 0 encrypted, 22706 w/ metadata, 22706 total
 472018-02-19 15:36:49  wallet                 1356ms
 482018-02-19 15:36:49 setKeyPool.size() = 2000
 492018-02-19 15:36:49 mapWallet.size() = 17746
 502018-02-19 15:36:49 mapAddressBook.size() = 25579
 512018-02-19 15:36:49 init message: Loading wallet...
 522018-02-19 15:36:49 nFileVersion = 150100
 532018-02-19 15:36:49 Keys: 1719 plaintext, 0 encrypted, 1719 w/ metadata, 1719 total
 542018-02-19 15:36:49  wallet                   93ms
 552018-02-19 15:36:49 setKeyPool.size() = 1000
 562018-02-19 15:36:49 mapWallet.size() = 572
 572018-02-19 15:36:49 mapAddressBook.size() = 881
 582018-02-19 15:36:49 init message: Loading wallet...
 592018-02-19 15:36:49 nFileVersion = 150100
 602018-02-19 15:36:49 Keys: 2014 plaintext, 0 encrypted, 2014 w/ metadata, 2014 total
 612018-02-19 15:36:49  wallet                   41ms
 622018-02-19 15:36:49 setKeyPool.size() = 2000
 632018-02-19 15:36:49 mapWallet.size() = 1
 642018-02-19 15:36:49 mapAddressBook.size() = 25
 652018-02-19 15:36:49 init message: Loading wallet...
 662018-02-19 15:36:49 nFileVersion = 150100
 672018-02-19 15:36:49 Keys: 2002 plaintext, 0 encrypted, 2002 w/ metadata, 2002 total
 682018-02-19 15:36:49  wallet                   66ms
 692018-02-19 15:36:49 setKeyPool.size() = 2000
 702018-02-19 15:36:49 mapWallet.size() = 0
 712018-02-19 15:36:49 mapAddressBook.size() = 1
 722018-02-19 15:36:49 mapBlockIndex.size() = 509980
 732018-02-19 15:36:49 nBestHeight = 509945
 742018-02-19 15:36:49 torcontrol thread start
 752018-02-19 15:36:49 ERROR: ReadBlockFromDisk: Deserialize or I/O error - ReadCompactSize(): size too large: iostream error at CBlockDiskPos(nFile=1184, nPos=33872952)
 762018-02-19 15:36:49 AddLocal(79.137.70.11:8333,1)
 772018-02-19 15:36:49 *** Failed to read block
 782018-02-19 15:36:49 Discover: IPv4 eth0: 79.137.70.11
 792018-02-19 15:36:49 Error: Error: A fatal internal error occurred, see debug.log for details
 802018-02-19 15:36:49 AddLocal([2001:41d0:602:20b::]:8333,1)
 812018-02-19 15:36:49 Discover: IPv6 eth0: 2001:41d0:602:20b::
 822018-02-19 15:36:49 Failed to connect best blockBound to [::]:8333
 832018-02-19 15:36:49 Imported mempool transactions from disk: 0 successes, 0 failed, 0 expired
 842018-02-19 15:36:49 Bound to 0.0.0.0:8333
 852018-02-19 15:36:49 init message: Loading P2P addresses...
 862018-02-19 15:36:49 Loaded 68266 addresses from peers.dat  327ms
 872018-02-19 15:36:49 init message: Loading banlist...
 882018-02-19 15:36:49 init message: Starting network threads...
 892018-02-19 15:36:49 init message: Done loading
 902018-02-19 15:36:49 net thread start
 912018-02-19 15:36:49 msghand thread start
 922018-02-19 15:36:49 opencon thread start
 932018-02-19 15:36:49 dnsseed thread start
 942018-02-19 15:36:49 addcon thread start
 952018-02-19 15:36:49 Leaving InitialBlockDownload (latching to false)
 962018-02-19 15:36:49 AddToWallet b9d9f1dc053141b30b589524b0663d98ea6926a6be838275d29c30d88eb397cc  
 972018-02-19 15:36:49 tor: Thread interrupt
 982018-02-19 15:36:49 opencon thread exit
 992018-02-19 15:36:49 dnsseed thread exit
1002018-02-19 15:36:49 torcontrol thread exit
1012018-02-19 15:36:49 addcon thread exit
1022018-02-19 15:36:49 scheduler thread interrupt
1032018-02-19 15:36:49 Shutdown: In progress...
1042018-02-19 15:36:49 net thread exit
1052018-02-19 15:36:49 msghand thread exit
1062018-02-19 15:36:50 Shutdown: done

What can the source of the issue?

MarcoFalke commented at 5:16 pm on February 19, 2018: member

You might want to check your hardware for defects just to be safe.

memtest86 to check your RAM
to check the CPU behaviour under load, use linpack or Prime95
to test your storage device use smartctl or CrystalDiskInfo

Source: https://bitcoin.stackexchange.com/a/12206

khelle commented at 10:33 am on February 20, 2018: none

I checked my disk and cpu and seems to be working fine. I also tried to reindex the whole blockchain and node now works without crash but still has tons of following errors:

 02018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
 12018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
 22018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
 32018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
 42018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
 52018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
 62018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
 72018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
 82018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
 92018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
102018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
112018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
122018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
132018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
142018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
152018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
162018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
172018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
182018-02-20 09:55:01 libevent: Error from accept() call: Too many open files

This issue seemed to be referenced in #11368 And being worked on in #12274

After deeper check it seemed the node was being crashed by having too many RPC calls made by exterior service. One of this crashes corrupted its database, so reindex needed to be made. It seems I have to create some kind of requests queue on my side to not flood core again. Issue fixed.

khelle closed this on Feb 20, 2018

MarcoFalke commented at 6:39 pm on February 20, 2018: member

@khelle Thanks for reporting back!

MarcoFalke locked this on Sep 8, 2021

Bitcoin core 0.15.1 started to crash #12483