Bitcoin core 0.15.1 started to crash #12483

After about 2 months of running bitcoin core 0.15.1 it crashed, and now don't want to restart. The logs shows me following info:

2018-02-19 15:36:39 Bitcoin version v0.15.1.0-g7b57bc998f33
2018-02-19 15:36:39 InitParameterInteraction: parameter interaction: -whitelistforcerelay=1 -> setting -whitelistrelay=1
2018-02-19 15:36:39 Assuming ancestors of block 0000000000000000003b9ce759c2a087d52abc4266f8f4ebd6d768b89defa50a have valid signatures.
2018-02-19 15:36:39 Setting nMinimumChainWork=000000000000000000000000000000000000000000723d3581fe1bd55373540a
2018-02-19 15:36:39 Using the 'standard' SHA256 implementation
2018-02-19 15:36:39 Using RdRand as an additional entropy source
2018-02-19 15:36:39 Default data directory /home/btc/.bitcoin
2018-02-19 15:36:39 Using data directory /home/btc/.bitcoin
2018-02-19 15:36:39 Using config file /home/btc/.bitcoin/bitcoin.conf
2018-02-19 15:36:39 Using at most 800 automatic connections (1024 file descriptors available)
2018-02-19 15:36:39 Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
2018-02-19 15:36:39 Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
2018-02-19 15:36:39 Using 4 threads for script verification
2018-02-19 15:36:39 scheduler thread start
2018-02-19 15:36:39 Binding RPC on address 0.0.0.0 port 15501 failed.
2018-02-19 15:36:39 HTTP: creating work queue of depth 2048
2018-02-19 15:36:39 Config options rpcuser and rpcpassword will soon be deprecated. Locally-run instances may remove rpcuser to use cookie-based auth, or may be replaced with rpcauth. Please see share/rpcuser for rpcauth auth generation.
2018-02-19 15:36:39 HTTP: starting 4 worker threads
2018-02-19 15:36:39 init message: Verifying wallet(s)...
2018-02-19 15:36:39 Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2018-02-19 15:36:39 Using wallet HOTWALLET.dat
2018-02-19 15:36:39 CDBEnv::Open: LogDir=/home/btc/.bitcoin/database ErrorFile=/home/btc/.bitcoin/db.log
2018-02-19 15:36:40 Cache configuration:
2018-02-19 15:36:40 * Using 2.0MiB for block index database
2018-02-19 15:36:40 * Using 8.0MiB for chain state database
2018-02-19 15:36:40 * Using 4086.0MiB for in-memory UTXO set (plus up to 286.1MiB of unused mempool space)
2018-02-19 15:36:40 init message: Loading block index...
2018-02-19 15:36:40 Opening LevelDB in /home/btc/.bitcoin/blocks/index
2018-02-19 15:36:40 Opened LevelDB successfully
2018-02-19 15:36:40 Using obfuscation key for /home/btc/.bitcoin/blocks/index: 0000000000000000
2018-02-19 15:36:44 LoadBlockIndexDB: last block file = 1184
2018-02-19 15:36:44 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=33, size=34893824, heights=509914...509946, time=2018-02-19...2018-02-19)
2018-02-19 15:36:44 Checking all blk files are present...
2018-02-19 15:36:44 LoadBlockIndexDB: transaction index disabled
2018-02-19 15:36:44 Opening LevelDB in /home/btc/.bitcoin/chainstate
2018-02-19 15:36:44 Opened LevelDB successfully
2018-02-19 15:36:44 Using obfuscation key for /home/btc/.bitcoin/chainstate: b9d65b264bd51a4c
2018-02-19 15:36:44 Loaded best chain: hashBestChain=0000000000000000003d98b1c37b41fb04fb5ded589e0e3686aadc3d45ea9e22 height=509945 date=2018-02-19 14:38:27 progress=0.999964
2018-02-19 15:36:44 init message: Rewinding blocks...
2018-02-19 15:36:46 init message: Verifying blocks...
2018-02-19 15:36:46 Verifying last 6 blocks at level 3
2018-02-19 15:36:46 [0%]...[16%]...[33%]...[50%]...[66%]...[83%]...[99%]...[DONE].
2018-02-19 15:36:47 No coin database inconsistencies in last 7 blocks (10803 transactions)
2018-02-19 15:36:47  block index            7149ms
2018-02-19 15:36:47 init message: Loading wallet...
2018-02-19 15:36:49 nFileVersion = 150100
2018-02-19 15:36:49 Keys: 22706 plaintext, 0 encrypted, 22706 w/ metadata, 22706 total
2018-02-19 15:36:49  wallet                 1356ms
2018-02-19 15:36:49 setKeyPool.size() = 2000
2018-02-19 15:36:49 mapWallet.size() = 17746
2018-02-19 15:36:49 mapAddressBook.size() = 25579
2018-02-19 15:36:49 init message: Loading wallet...
2018-02-19 15:36:49 nFileVersion = 150100
2018-02-19 15:36:49 Keys: 1719 plaintext, 0 encrypted, 1719 w/ metadata, 1719 total
2018-02-19 15:36:49  wallet                   93ms
2018-02-19 15:36:49 setKeyPool.size() = 1000
2018-02-19 15:36:49 mapWallet.size() = 572
2018-02-19 15:36:49 mapAddressBook.size() = 881
2018-02-19 15:36:49 init message: Loading wallet...
2018-02-19 15:36:49 nFileVersion = 150100
2018-02-19 15:36:49 Keys: 2014 plaintext, 0 encrypted, 2014 w/ metadata, 2014 total
2018-02-19 15:36:49  wallet                   41ms
2018-02-19 15:36:49 setKeyPool.size() = 2000
2018-02-19 15:36:49 mapWallet.size() = 1
2018-02-19 15:36:49 mapAddressBook.size() = 25
2018-02-19 15:36:49 init message: Loading wallet...
2018-02-19 15:36:49 nFileVersion = 150100
2018-02-19 15:36:49 Keys: 2002 plaintext, 0 encrypted, 2002 w/ metadata, 2002 total
2018-02-19 15:36:49  wallet                   66ms
2018-02-19 15:36:49 setKeyPool.size() = 2000
2018-02-19 15:36:49 mapWallet.size() = 0
2018-02-19 15:36:49 mapAddressBook.size() = 1
2018-02-19 15:36:49 mapBlockIndex.size() = 509980
2018-02-19 15:36:49 nBestHeight = 509945
2018-02-19 15:36:49 torcontrol thread start
2018-02-19 15:36:49 ERROR: ReadBlockFromDisk: Deserialize or I/O error - ReadCompactSize(): size too large: iostream error at CBlockDiskPos(nFile=1184, nPos=33872952)
2018-02-19 15:36:49 AddLocal(79.137.70.11:8333,1)
2018-02-19 15:36:49 *** Failed to read block
2018-02-19 15:36:49 Discover: IPv4 eth0: 79.137.70.11
2018-02-19 15:36:49 Error: Error: A fatal internal error occurred, see debug.log for details
2018-02-19 15:36:49 AddLocal([2001:41d0:602:20b::]:8333,1)
2018-02-19 15:36:49 Discover: IPv6 eth0: 2001:41d0:602:20b::
2018-02-19 15:36:49 Failed to connect best blockBound to [::]:8333
2018-02-19 15:36:49 Imported mempool transactions from disk: 0 successes, 0 failed, 0 expired
2018-02-19 15:36:49 Bound to 0.0.0.0:8333
2018-02-19 15:36:49 init message: Loading P2P addresses...
2018-02-19 15:36:49 Loaded 68266 addresses from peers.dat  327ms
2018-02-19 15:36:49 init message: Loading banlist...
2018-02-19 15:36:49 init message: Starting network threads...
2018-02-19 15:36:49 init message: Done loading
2018-02-19 15:36:49 net thread start
2018-02-19 15:36:49 msghand thread start
2018-02-19 15:36:49 opencon thread start
2018-02-19 15:36:49 dnsseed thread start
2018-02-19 15:36:49 addcon thread start
2018-02-19 15:36:49 Leaving InitialBlockDownload (latching to false)
2018-02-19 15:36:49 AddToWallet b9d9f1dc053141b30b589524b0663d98ea6926a6be838275d29c30d88eb397cc  
2018-02-19 15:36:49 tor: Thread interrupt
2018-02-19 15:36:49 opencon thread exit
2018-02-19 15:36:49 dnsseed thread exit
2018-02-19 15:36:49 torcontrol thread exit
2018-02-19 15:36:49 addcon thread exit
2018-02-19 15:36:49 scheduler thread interrupt
2018-02-19 15:36:49 Shutdown: In progress...
2018-02-19 15:36:49 net thread exit
2018-02-19 15:36:49 msghand thread exit
2018-02-19 15:36:50 Shutdown: done

What can the source of the issue?

You might want to check your hardware for defects just to be safe.

• memtest86 to check your RAM
• to check the CPU behaviour under load, use linpack or Prime95
• to test your storage device use smartctl or CrystalDiskInfo

Source: https://bitcoin.stackexchange.com/a/12206

I checked my disk and cpu and seems to be working fine. I also tried to reindex the whole blockchain and node now works without crash but still has tons of following errors:

2018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
2018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
2018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
2018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
2018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
2018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
2018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
2018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
2018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
2018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
2018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
2018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
2018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
2018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
2018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
2018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
2018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
2018-02-20 09:55:01 libevent: Error from accept() call: Too many open files
2018-02-20 09:55:01 libevent: Error from accept() call: Too many open files

This issue seemed to be referenced in #11368 And being worked on in #12274

After deeper check it seemed the node was being crashed by having too many RPC calls made by exterior service. One of this crashes corrupted its database, so reindex needed to be made. It seems I have to create some kind of requests queue on my side to not flood core again. Issue fixed.

@khelle Thanks for reporting back!