Bitcoind crashes with Too Many Files error #14870

issue SanJoseCosta openend this issue on December 4, 2018

SanJoseCosta commented at 5:12 pm on December 4, 2018: none

Stops accepting RCP calls

Every time I start bitcoind

Version 17 downloaded executable

Mac OSX

Matthews-MacBook-Pro:~ matt$ /Applications/bitcoin-0.17.0/bin/bitcoind -server -testnet -txindex -rest -rpcuser=matt -rpcpassword=plexus -rpcport=8332 -debug=leveldb 2018-12-04T17:07:25Z Bitcoin Core version v0.17.0 (release build) 2018-12-04T17:07:25Z InitParameterInteraction: parameter interaction: -whitelistforcerelay=1 -> setting -whitelistrelay=1 2018-12-04T17:07:25Z Assuming ancestors of block 0000000000000037a8cd3e06cd5edbfe9dd1dbcc5dacab279376ef7cfc2b4c75 have valid signatures. 2018-12-04T17:07:25Z Setting nMinimumChainWork=00000000000000000000000000000000000000000000007dbe94253893cbd463 2018-12-04T17:07:25Z Using the ‘sse4(1way),sse41(4way),avx2(8way)’ SHA256 implementation 2018-12-04T17:07:25Z Using RdRand as an additional entropy source 2018-12-04T17:07:25Z Default data directory /Users/matt/Library/Application Support/Bitcoin 2018-12-04T17:07:25Z Using data directory /Users/matt/Library/Application Support/Bitcoin/testnet3 2018-12-04T17:07:25Z Using config file /Users/matt/Library/Application Support/Bitcoin/bitcoin.conf 2018-12-04T17:07:25Z Using at most 125 automatic connections (283 file descriptors available) 2018-12-04T17:07:25Z Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements 2018-12-04T17:07:25Z Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements 2018-12-04T17:07:25Z Using 4 threads for script verification 2018-12-04T17:07:25Z scheduler thread start 2018-12-04T17:07:25Z HTTP: creating work queue of depth 16 2018-12-04T17:07:25Z Config options rpcuser and rpcpassword will soon be deprecated. Locally-run instances may remove rpcuser to use cookie-based auth, or may be replaced with rpcauth. Please see share/rpcauth for rpcauth auth generation. 2018-12-04T17:07:25Z HTTP: starting 4 worker threads 2018-12-04T17:07:25Z Using wallet directory /Users/matt/Library/Application Support/Bitcoin/testnet3/wallets 2018-12-04T17:07:25Z init message: Verifying wallet(s)… 2018-12-04T17:07:25Z Using BerkeleyDB version Berkeley DB 4.8.30: (April 9, 2010) 2018-12-04T17:07:25Z Using wallet wallet.dat 2018-12-04T17:07:25Z BerkeleyEnvironment::Open: LogDir=/Users/matt/Library/Application Support/Bitcoin/testnet3/wallets/database ErrorFile=/Users/matt/Library/Application Support/Bitcoin/testnet3/wallets/db.log 2018-12-04T17:07:26Z Cache configuration: 2018-12-04T17:07:26Z * Using 2.0MiB for block index database 2018-12-04T17:07:26Z * Using 56.0MiB for transaction index database 2018-12-04T17:07:26Z * Using 8.0MiB for chain state database 2018-12-04T17:07:26Z * Using 384.0MiB for in-memory UTXO set (plus up to 286.1MiB of unused mempool space) 2018-12-04T17:07:26Z init message: Loading block index… 2018-12-04T17:07:26Z LevelDB using max_open_files=1000 (default=1000) 2018-12-04T17:07:26Z Opening LevelDB in /Users/matt/Library/Application Support/Bitcoin/testnet3/blocks/index 2018-12-04T17:07:26Z leveldb: Recovering log #791 2018-12-04T17:07:26Z leveldb: Level-0 table #801: started 2018-12-04T17:07:26Z leveldb: Level-0 table #801: 1101 bytes OK 2018-12-04T17:07:26Z leveldb: Delete type=2 #788 2018-12-04T17:07:26Z leveldb: Delete type=2 #785 2018-12-04T17:07:26Z leveldb: Delete type=2 #784 2018-12-04T17:07:26Z leveldb: Delete type=2 #790 2018-12-04T17:07:26Z leveldb: Delete type=2 #786 2018-12-04T17:07:26Z leveldb: Delete type=2 #787 2018-12-04T17:07:26Z leveldb: Delete type=2 #783 2018-12-04T17:07:26Z leveldb: Delete type=2 #782 2018-12-04T17:07:26Z leveldb: Delete type=2 #781 2018-12-04T17:07:26Z leveldb: Delete type=0 #791 2018-12-04T17:07:26Z leveldb: Delete type=3 #789 2018-12-04T17:07:26Z Opened LevelDB successfully 2018-12-04T17:07:26Z Using obfuscation key for /Users/matt/Library/Application Support/Bitcoin/testnet3/blocks/index: 0000000000000000 2018-12-04T17:07:27Z leveldb: Compacting 1@0 + 8@1 files 2018-12-04T17:07:27Z leveldb: Generated table #803@0: 14888 keys, 2140910 bytes 2018-12-04T17:07:27Z leveldb: Generated table #804@0: 5178 keys, 744418 bytes 2018-12-04T17:07:28Z leveldb: Generated table #805@0: 7347 keys, 1056062 bytes 2018-12-04T17:07:28Z leveldb: Generated table #806@0: 14938 keys, 2136593 bytes 2018-12-04T17:07:28Z leveldb: Generated table #807@0: 1256 keys, 179294 bytes 2018-12-04T17:07:28Z leveldb: Generated table #808@0: 14970 keys, 2138641 bytes 2018-12-04T17:07:28Z leveldb: Generated table #809@0: 3608 keys, 517536 bytes 2018-12-04T17:07:28Z leveldb: Generated table #810@0: 2 keys, 221 bytes 2018-12-04T17:07:28Z leveldb: Compacted 1@0 + 8@1 files => 8913675 bytes 2018-12-04T17:07:28Z leveldb: compacted to: files[ 0 8 55 91 0 0 0 ] 2018-12-04T17:07:33Z LoadBlockIndexDB: last block file = 154 2018-12-04T17:07:33Z LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=5075, size=108826844, heights=1441407…1446454, time=2018-10-31…2018-12-04) 2018-12-04T17:07:33Z Checking all blk files are present… 2018-12-04T17:07:33Z LevelDB using max_open_files=1000 (default=1000) 2018-12-04T17:07:33Z Opening LevelDB in /Users/matt/Library/Application Support/Bitcoin/testnet3/chainstate 2018-12-04T17:07:33Z leveldb: Recovering log #9327 2018-12-04T17:07:33Z leveldb: Level-0 table #9332: started 2018-12-04T17:07:33Z leveldb: Level-0 table #9332: 308 bytes OK 2018-12-04T17:07:33Z leveldb: Delete type=2 #9326 2018-12-04T17:07:33Z leveldb: Delete type=0 #9327 2018-12-04T17:07:33Z leveldb: Delete type=2 #9311 2018-12-04T17:07:33Z leveldb: Delete type=2 #9310 2018-12-04T17:07:33Z leveldb: Delete type=2 #9312 2018-12-04T17:07:33Z leveldb: Delete type=3 #9325 2018-12-04T17:07:33Z Opened LevelDB successfully 2018-12-04T17:07:33Z Using obfuscation key for /Users/matt/Library/Application Support/Bitcoin/testnet3/chainstate: fcab38c4f274e80a 2018-12-04T17:07:34Z Loaded best chain: hashBestChain=00000000000000063df115997921bf6856c08a08318b592df576c2f863b7d7c5 height=1446453 date=2018-12-04T04:45:18Z progress=0.999422 2018-12-04T17:07:34Z init message: Rewinding blocks… 2018-12-04T17:07:41Z WriteBatch memory usage: db=index, before=0.0MiB, after=0.0MiB 2018-12-04T17:07:41Z WriteBatch memory usage: db=chainstate, before=0.0MiB, after=0.0MiB 2018-12-04T17:07:41Z init message: Verifying blocks… 2018-12-04T17:07:41Z Verifying last 6 blocks at level 3 2018-12-04T17:07:41Z [0%]…leveldb: Compacting 1@0 + 3@1 files 2018-12-04T17:07:41Z leveldb: Generated table #9334@0: 1 keys, 207 bytes 2018-12-04T17:07:41Z [16%]…leveldb: Generated table #9335@0: 3162 keys, 181793 bytes 2018-12-04T17:07:41Z [33%]…leveldb: Generated table #9336@0: 2846 keys, 181033 bytes 2018-12-04T17:07:41Z leveldb: Compacted 1@0 + 3@1 files => 363033 bytes 2018-12-04T17:07:41Z leveldb: compacted to: files[ 0 3 78 484 0 0 0 ] 2018-12-04T17:07:41Z [50%]…[66%]…[83%]…[99%]…[DONE]. 2018-12-04T17:07:41Z No coin database inconsistencies in last 6 blocks (444 transactions) 2018-12-04T17:07:41Z block index 15119ms 2018-12-04T17:07:41Z LevelDB using max_open_files=1000 (default=1000) 2018-12-04T17:07:41Z Opening LevelDB in /Users/matt/Library/Application Support/Bitcoin/testnet3/indexes/txindex 2018-12-04T17:07:41Z leveldb: Recovering log #12582 2018-12-04T17:07:41Z leveldb: Delete type=3 #12580 2018-12-04T17:07:41Z leveldb: Delete type=0 #12582 2018-12-04T17:07:41Z Opened LevelDB successfully 2018-12-04T17:07:41Z Using obfuscation key for /Users/matt/Library/Application Support/Bitcoin/testnet3/indexes/txindex: 0000000000000000 2018-12-04T17:07:41Z init message: Loading wallet… 2018-12-04T17:07:41Z txindex thread start 2018-12-04T17:07:41Z txindex is enabled at height 1446453 2018-12-04T17:07:41Z txindex thread exit 2018-12-04T17:07:42Z [default wallet] nFileVersion = 170000 2018-12-04T17:07:42Z [default wallet] Keys: 61977 plaintext, 0 encrypted, 61977 w/ metadata, 61977 total. Unknown wallet records: 1 2018-12-04T17:07:42Z [default wallet] Wallet completed loading in 1474ms 2018-12-04T17:07:42Z [default wallet] setKeyPool.size() = 2000 2018-12-04T17:07:42Z [default wallet] mapWallet.size() = 630 2018-12-04T17:07:42Z [default wallet] mapAddressBook.size() = 70700 2018-12-04T17:07:42Z mapBlockIndex.size() = 1446512 2018-12-04T17:07:42Z nBestHeight = 1446453 2018-12-04T17:07:42Z torcontrol thread start 2018-12-04T17:07:42Z AddLocal([2600:1700:57f0:8050:14a2:afde:1894:925c]:18333,1) 2018-12-04T17:07:42Z Discover: IPv6 en0: 2600:1700:57f0:8050:14a2:afde:1894:925c 2018-12-04T17:07:42Z AddLocal([2600:1700:57f0:8050:2556:3391:9481:b0b]:18333,1) 2018-12-04T17:07:42Z Discover: IPv6 en0: 2600:1700:57f0:8050:2556:3391:9481:b0b 2018-12-04T17:07:42Z Bound to [::]:18333 2018-12-04T17:07:42Z Bound to 0.0.0.0:18333 2018-12-04T17:07:42Z init message: Loading P2P addresses… 2018-12-04T17:07:42Z Leaving InitialBlockDownload (latching to false) 2018-12-04T17:07:42Z Imported mempool transactions from disk: 42 succeeded, 0 failed, 0 expired, 0 already there 2018-12-04T17:07:43Z Loaded 62215 addresses from peers.dat 192ms 2018-12-04T17:07:43Z init message: Loading banlist… 2018-12-04T17:07:43Z init message: Starting network threads… 2018-12-04T17:07:43Z net thread start 2018-12-04T17:07:43Z init message: Done loading 2018-12-04T17:07:43Z dnsseed thread start 2018-12-04T17:07:43Z addcon thread start 2018-12-04T17:07:43Z opencon thread start 2018-12-04T17:07:43Z msghand thread start 2018-12-04T17:07:43Z New outbound peer connected: version: 70015, blocks=1446521, peer=0 2018-12-04T17:07:49Z UpdateTip: new best=000000000000002d5ecbdf5e466a286dcce223661ee85fa5947d53861442565a height=1446454 version=0x20000000 log2_work=71.761827 tx=48217097 date=‘2018-12-04T04:52:27Z’ progress=0.999427 cache=0.0MiB(125txo) warning=‘11 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.0MiB, after=0.0MiB 2018-12-04T17:07:49Z UpdateTip: new best=00000000000000a6a191de2c5c2cdc42b64472579911aedf49607852c26e83c6 height=1446455 version=0x20000000 log2_work=71.761848 tx=48217134 date=‘2018-12-04T04:59:11Z’ progress=0.999433 cache=0.0MiB(185txo) warning=‘11 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.0MiB, after=0.0MiB 2018-12-04T17:07:49Z UpdateTip: new best=000000000000b927b40c61133cb79e87434b7724c6473e11cab0b3807496e1eb height=1446456 version=0x20000000 log2_work=71.761848 tx=48217234 date=‘2018-12-04T05:19:19Z’ progress=0.999448 cache=0.1MiB(419txo) warning=‘11 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.0MiB, after=0.0MiB 2018-12-04T17:07:49Z Pre-allocating up to position 0xe00000 in rev00154.dat 2018-12-04T17:07:49Z UpdateTip: new best=00000000000001100433c151cc8eca67aa93f56aa6acbbd23d6be471ac33a946 height=1446457 version=0x2000e000 log2_work=71.761869 tx=48217275 date=‘2018-12-04T05:28:32Z’ progress=0.999456 cache=0.1MiB(460txo) warning=‘12 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.0MiB, after=0.0MiB 2018-12-04T17:07:49Z UpdateTip: new best=000000000017e1ea5199451d3e3541726ed4bd857fcaf9ea3c8edeef105bb689 height=1446458 version=0x20000000 log2_work=71.761869 tx=48217390 date=‘2018-12-04T05:48:40Z’ progress=0.999471 cache=0.1MiB(648txo) warning=‘12 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.0MiB, after=0.0MiB 2018-12-04T17:07:49Z UpdateTip: new best=00000000000000ec28bf31cc9fea23314c03b2d46cfac2b3410d7f14d2544962 height=1446459 version=0x20000000 log2_work=71.76189 tx=48217431 date=‘2018-12-04T05:53:56Z’ progress=0.999475 cache=0.1MiB(717txo) warning=‘12 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.0MiB, after=0.0MiB 2018-12-04T17:07:49Z UpdateTip: new best=00000000000dd9d42b543aae4fc1c1feacb0f08784c28d8fff4e481cdfc1c267 height=1446460 version=0x20000000 log2_work=71.76189 tx=48217559 date=‘2018-12-04T06:14:04Z’ progress=0.999491 cache=0.1MiB(1003txo) warning=‘12 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.0MiB, after=0.0MiB 2018-12-04T17:07:49Z UpdateTip: new best=00000000000c02b40c20b543aa0ff681586ad40f5f7b28ed4320d845fb9a9d33 height=1446461 version=0x20000000 log2_work=71.76189 tx=48217683 date=‘2018-12-04T06:34:08Z’ progress=0.999507 cache=0.2MiB(1166txo) warning=‘12 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.0MiB, after=0.0MiB 2018-12-04T17:07:49Z UpdateTip: new best=000000000002833d8fee2756457a1e8356afa5674f10d97ab711afd0aa5f4213 height=1446462 version=0x20000000 log2_work=71.76189 tx=48217869 date=‘2018-12-04T06:54:16Z’ progress=0.999522 cache=0.2MiB(1439txo) warning=‘12 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.0MiB, after=0.1MiB 2018-12-04T17:07:49Z UpdateTip: new best=00000000000000953b09524644db82230fbd4939532f34dc8ad5ff4a0a6cbe79 height=1446463 version=0x20000000 log2_work=71.761911 tx=48217939 date=‘2018-12-04T07:01:34Z’ progress=0.999528 cache=0.2MiB(1554txo) warning=‘12 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.1MiB, after=0.1MiB 2018-12-04T17:07:49Z UpdateTip: new best=000000000000011ddc97ba7e679b04aecb5dee23a9895a8c59d6c4fc4fce339c height=1446464 version=0x20000000 log2_work=71.761932 tx=48218016 date=‘2018-12-04T07:11:21Z’ progress=0.999536 cache=0.2MiB(1657txo) warning=‘12 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.1MiB, after=0.1MiB 2018-12-04T17:07:49Z UpdateTip: new best=000000000000010c87a927806b19eef5117088aa6f8b30a0062013a342eae3a3 height=1446465 version=0x20000000 log2_work=71.761953 tx=48218151 date=‘2018-12-04T07:29:54Z’ progress=0.999550 cache=0.3MiB(1874txo) warning=‘12 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.1MiB, after=0.1MiB 2018-12-04T17:07:49Z UpdateTip: new best=00000000000000f413b6abf1bb13d47c4caa2b9c68d86861f64c6ea762f96d21 height=1446466 version=0x20000000 log2_work=71.761973 tx=48218226 date=‘2018-12-04T07:39:24Z’ progress=0.999557 cache=0.3MiB(1990txo) warning=‘11 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.1MiB, after=0.1MiB 2018-12-04T17:07:49Z UpdateTip: new best=00000000000000c3876835ae819e6a06b2689784bd5a49dff5a4d04c358bbd9d height=1446467 version=0x2fff8000 log2_work=71.761994 tx=48218249 date=‘2018-12-04T07:42:16Z’ progress=0.999560 cache=0.3MiB(2017txo) warning=‘12 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.1MiB, after=0.1MiB 2018-12-04T17:07:49Z UpdateTip: new best=000000000000004f7fff32ff0255f533f69c6ed204714585b7e24c7294a8c5af height=1446468 version=0x20400000 log2_work=71.762015 tx=48218351 date=‘2018-12-04T07:54:48Z’ progress=0.999569 cache=0.3MiB(2158txo) warning=‘13 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.1MiB, after=0.1MiB 2018-12-04T17:07:49Z UpdateTip: new best=000000000000000df9611365cbefce75de3aecceec9a71c484a851f3d00398b8 height=1446469 version=0x20800000 log2_work=71.762036 tx=48218353 date=‘2018-12-04T07:55:01Z’ progress=0.999570 cache=0.3MiB(2160txo) warning=‘14 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.1MiB, after=0.1MiB 2018-12-04T17:07:49Z UpdateTip: new best=000000000004a260c26fe6b315d5cc1f0a5490d62fbc08689042029f9af69050 height=1446470 version=0x20000000 log2_work=71.762036 tx=48218514 date=‘2018-12-04T08:15:12Z’ progress=0.999585 cache=0.3MiB(2395txo) warning=‘14 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.1MiB, after=0.1MiB 2018-12-04T17:07:49Z UpdateTip: new best=000000000000005ff7940b2a1e7f4816e4a664d2238968739bd5fdafd0f4ecd3 height=1446471 version=0x20000000 log2_work=71.762057 tx=48218515 date=‘2018-12-04T08:15:09Z’ progress=0.999585 cache=0.3MiB(2396txo) warning=‘14 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.1MiB, after=0.1MiB 2018-12-04T17:07:49Z UpdateTip: new best=0000000000000056df79f565bb235683bd8ecb388e4338f22f5c3e30e2f11d3b height=1446472 version=0x20000000 log2_work=71.762078 tx=48218609 date=‘2018-12-04T08:26:31Z’ progress=0.999594 cache=0.3MiB(2537txo) warning=‘14 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.1MiB, after=0.1MiB 2018-12-04T17:07:49Z UpdateTip: new best=000000000000005e817358aa64b8f0cb6fa33fd50bef0d163826eee6dd00a68c height=1446473 version=0x20000000 log2_work=71.762099 tx=48218622 date=‘2018-12-04T08:28:05Z’ progress=0.999595 cache=0.3MiB(2548txo) warning=‘14 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.1MiB, after=0.1MiB 2018-12-04T17:07:49Z UpdateTip: new best=0000000000000012cd1646b81f12739a74f996edd07b574b852c2a8f8d12ea0f height=1446474 version=0x20000000 log2_work=71.76212 tx=48218774 date=‘2018-12-04T08:46:45Z’ progress=0.999610 cache=0.4MiB(2788txo) warning=‘14 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.1MiB, after=0.1MiB 2018-12-04T17:07:49Z UpdateTip: new best=00000000000000725a7cb479398be3bb12858628fbb263f4a6bf15c4241f9606 height=1446475 version=0x20800000 log2_work=71.762141 tx=48218781 date=‘2018-12-04T08:47:31Z’ progress=0.999610 cache=0.4MiB(2800txo) warning=‘15 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.1MiB, after=0.1MiB 2018-12-04T17:07:49Z UpdateTip: new best=0000000000000111c4572babeaf1c9526e8eddaa22beff9c4bf71de473ca3841 height=1446476 version=0x20000000 log2_work=71.762161 tx=48218921 date=‘2018-12-04T09:06:45Z’ progress=0.999625 cache=0.4MiB(2998txo) warning=‘15 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.1MiB, after=0.1MiB 2018-12-04T17:07:49Z UpdateTip: new best=000000000000004b73101db3749c7b060916dbc490855a78daefd2bcd008b21b height=1446477 version=0x20000000 log2_work=71.762182 tx=48218984 date=‘2018-12-04T09:15:17Z’ progress=0.999632 cache=0.4MiB(3139txo) warning=‘15 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.1MiB, after=0.1MiB 2018-12-04T17:07:49Z UpdateTip: new best=00000000000000c6bba9e81651ad471b671f3acd7cd5ed9ae79cb2ac1f045061 height=1446478 version=0x20000000 log2_work=71.762203 tx=48219018 date=‘2018-12-04T09:18:57Z’ progress=0.999635 cache=0.4MiB(3206txo) warning=‘15 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.1MiB, after=0.1MiB 2018-12-04T17:07:49Z UpdateTip: new best=00000000000000900505f898e53cbaaaa446ecc2a0aff020a6f5161b60f9063c height=1446479 version=0x20000000 log2_work=71.762224 tx=48219029 date=‘2018-12-04T09:20:01Z’ progress=0.999636 cache=0.4MiB(3222txo) warning=‘15 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.1MiB, after=0.1MiB 2018-12-04T17:07:49Z New outbound peer connected: version: 70015, blocks=1446521, peer=1 2018-12-04T17:07:49Z UpdateTip: new best=000000000000001865349c72ef373b8211c412af115702e6888b0c6772ee82e3 height=1446480 version=0x20000000 log2_work=71.762245 tx=48219087 date=‘2018-12-04T09:26:42Z’ progress=0.999641 cache=0.5MiB(3322txo) warning=‘15 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.1MiB, after=0.1MiB 2018-12-04T17:07:49Z UpdateTip: new best=00000000000000b433e9fbbd278dc567c90cace86556444e1ae57649741c5612 height=1446481 version=0x20000000 log2_work=71.762266 tx=48219212 date=‘2018-12-04T09:39:26Z’ progress=0.999651 cache=0.5MiB(3521txo) warning=‘15 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.1MiB, after=0.2MiB 2018-12-04T17:07:49Z UpdateTip: new best=00000000000000d5803219f9474245dcbfdc70a44580f25eba34a7d54b599a96 height=1446482 version=0x3fffe000 log2_work=71.762287 tx=48219310 date=‘2018-12-04T09:52:36Z’ progress=0.999661 cache=0.5MiB(3664txo) warning=‘16 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.2MiB, after=0.2MiB 2018-12-04T17:07:49Z UpdateTip: new best=0000000000000044fe30a71fba844f92b427de27b7f6173d684699356d49f073 height=1446483 version=0x20000000 log2_work=71.762308 tx=48219379 date=‘2018-12-04T09:57:19Z’ progress=0.999665 cache=0.5MiB(3784txo) warning=‘16 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.2MiB, after=0.2MiB 2018-12-04T17:07:49Z UpdateTip: new best=0000000000000041704eaee1703fd8ee63a680b430af3098c1d2e8549c77d86e height=1446484 version=0x20000000 log2_work=71.762328 tx=48219428 date=‘2018-12-04T10:02:35Z’ progress=0.999669 cache=0.5MiB(3851txo) warning=‘16 of last 100 blocks have unexpected version’ 2018-12-04T17:07:49Z WriteBatch memory usage: db=txindex, before=0.2MiB, after=0.2MiB 2018-12-04T17:07:50Z UpdateTip: new best=000000000000005ec9c970eaa7953d6dd52532a242583f011d2d73fd94b48ca1 height=1446485 version=0x2000e000 log2_work=71.762349 tx=48219503 date=‘2018-12-04T10:14:32Z’ progress=0.999678 cache=0.5MiB(3983txo) warning=‘17 of last 100 blocks have unexpected version’ 2018-12-04T17:07:50Z WriteBatch memory usage: db=txindex, before=0.2MiB, after=0.2MiB 2018-12-04T17:07:50Z UpdateTip: new best=00000000000000dfb2d5f5c06f1ea6122a12886204164ac4cfd2f7bdc95daf05 height=1446486 version=0x20000000 log2_work=71.76237 tx=48219551 date=‘2018-12-04T10:20:13Z’ progress=0.999683 cache=0.6MiB(4088txo) warning=‘17 of last 100 blocks have unexpected version’ 2018-12-04T17:07:50Z WriteBatch memory usage: db=txindex, before=0.2MiB, after=0.2MiB 2018-12-04T17:07:50Z UpdateTip: new best=0000000000000003d2037b15bef07123e0e7b8750c5a0188fa8f959bf12d1fb2 height=1446487 version=0x20000000 log2_work=71.762391 tx=48219612 date=‘2018-12-04T10:30:08Z’ progress=0.999690 cache=0.6MiB(4177txo) warning=‘17 of last 100 blocks have unexpected version’ 2018-12-04T17:07:50Z WriteBatch memory usage: db=txindex, before=0.2MiB, after=0.2MiB 2018-12-04T17:07:50Z UpdateTip: new best=000000000000013bb1e193cc3e190d1c5683190ffb06a77f39e9fea1ece344b4 height=1446488 version=0x20000000 log2_work=71.762412 tx=48219678 date=‘2018-12-04T10:44:20Z’ progress=0.999701 cache=0.6MiB(4289txo) warning=‘17 of last 100 blocks have unexpected version’ 2018-12-04T17:07:50Z WriteBatch memory usage: db=txindex, before=0.2MiB, after=0.2MiB 2018-12-04T17:07:50Z UpdateTip: new best=0000000000000018435c24b9822bd36fe8372d4afe79add694e53e1df19cbf2c height=1446489 version=0x2000e000 log2_work=71.762433 tx=48219686 date=‘2018-12-04T10:45:19Z’ progress=0.999702 cache=0.6MiB(4307txo) warning=‘18 of last 100 blocks have unexpected version’ 2018-12-04T17:07:50Z WriteBatch memory usage: db=txindex, before=0.2MiB, after=0.2MiB 2018-12-04T17:07:50Z UpdateTip: new best=000000000000004fee26b2474777afeae45ccb401bba82a7b31ec7942e60073c height=1446490 version=0x3fffe000 log2_work=71.762454 tx=48219740 date=‘2018-12-04T10:56:15Z’ progress=0.999711 cache=0.6MiB(4383txo) warning=‘19 of last 100 blocks have unexpected version’ 2018-12-04T17:07:50Z WriteBatch memory usage: db=txindex, before=0.2MiB, after=0.2MiB 2018-12-04T17:07:50Z UpdateTip: new best=000000000010cffca8bdddbbe64cadb0811f263c09e99e129fb7e3c76dd69c7c height=1446491 version=0x20000000 log2_work=71.762454 tx=48219865 date=‘2018-12-04T11:16:24Z’ progress=0.999726 cache=0.6MiB(4564txo) warning=‘19 of last 100 blocks have unexpected version’ 2018-12-04T17:07:50Z WriteBatch memory usage: db=txindex, before=0.2MiB, after=0.2MiB 2018-12-04T17:07:50Z UpdateTip: new best=0000000000127ce377076949417a616cf174d4d921f5c8469f839af6cfecd2a8 height=1446492 version=0x20000000 log2_work=71.762454 tx=48219957 date=‘2018-12-04T11:36:34Z’ progress=0.999742 cache=0.6MiB(4697txo) warning=‘19 of last 100 blocks have unexpected version’ 2018-12-04T17:07:50Z WriteBatch memory usage: db=txindex, before=0.2MiB, after=0.2MiB 2018-12-04T17:07:50Z UpdateTip: new best=000000000000003265773110b43eebc2cf5d218ffcd37fddd34e944d69b830e9 height=1446493 version=0x20000000 log2_work=71.762475 tx=48219973 date=‘2018-12-04T11:38:33Z’ progress=0.999744 cache=0.6MiB(4717txo) warning=‘19 of last 100 blocks have unexpected version’ 2018-12-04T17:07:50Z WriteBatch memory usage: db=txindex, before=0.2MiB, after=0.2MiB 2018-12-04T17:07:50Z UpdateTip: new best=000000000000002f92e46079e95d815adc8dc60ebdad1a3700135bfb7fe21ce0 height=1446494 version=0x2000e000 log2_work=71.762496 tx=48220033 date=‘2018-12-04T11:47:02Z’ progress=0.999750 cache=0.6MiB(4819txo) warning=‘20 of last 100 blocks have unexpected version’ 2018-12-04T17:07:50Z WriteBatch memory usage: db=txindex, before=0.2MiB, after=0.2MiB 2018-12-04T17:07:50Z UpdateTip: new best=000000000000010e437366fc750cd51f763715273813573ee19706bb80fe6feb height=1446495 version=0x20000000 log2_work=71.762516 tx=48220049 date=‘2018-12-04T11:54:05Z’ progress=0.999756 cache=0.6MiB(4862txo) warning=‘20 of last 100 blocks have unexpected version’ 2018-12-04T17:07:50Z WriteBatch memory usage: db=txindex, before=0.2MiB, after=0.2MiB 2018-12-04T17:07:50Z UpdateTip: new best=00000000000000dc8d0f77531f11e78fb0799ba22da26521f71dca56ae448b3b height=1446496 version=0x20000000 log2_work=71.762537 tx=48220084 date=‘2018-12-04T11:58:32Z’ progress=0.999759 cache=0.7MiB(4907txo) warning=‘20 of last 100 blocks have unexpected version’ 2018-12-04T17:07:50Z WriteBatch memory usage: db=txindex, before=0.2MiB, after=0.2MiB 2018-12-04T17:07:50Z UpdateTip: new best=00000000000000fdffea60041b4b78b249816aef47c2b11ebb37f389ebcd5f4d height=1446497 version=0x20000000 log2_work=71.762558 tx=48220115 date=‘2018-12-04T12:02:26Z’ progress=0.999762 cache=0.7MiB(4930txo) warning=‘20 of last 100 blocks have unexpected version’ 2018-12-04T17:07:50Z WriteBatch memory usage: db=txindex, before=0.2MiB, after=0.2MiB 2018-12-04T17:07:50Z UpdateTip: new best=00000000000000aa287e17caa22607fcd3b6d76ea54f7e8a54be831944c9d7bf height=1446498 version=0x20000000 log2_work=71.762579 tx=48220166 date=‘2018-12-04T12:14:10Z’ progress=0.999771 cache=0.7MiB(5006txo) warning=‘20 of last 100 blocks have unexpected version’ 2018-12-04T17:07:50Z WriteBatch memory usage: db=txindex, before=0.2MiB, after=0.2MiB 2018-12-04T17:07:50Z UpdateTip: new best=000000000000005f7fd1441bf4f3d8884b6d35d6cd9bb6069f5d1413b94bd07e height=1446499 version=0x20000000 log2_work=71.7626 tx=48220188 date=‘2018-12-04T12:15:56Z’ progress=0.999773 cache=0.7MiB(5027txo) warning=‘20 of last 100 blocks have unexpected version’ 2018-12-04T17:07:50Z WriteBatch memory usage: db=txindex, before=0.2MiB, after=0.2MiB 2018-12-04T17:07:50Z UpdateTip: new best=000000000000011cd1e6c4c65b781391d5f98afaf5bf583224c9dfd35312cfce height=1446500 version=0x20000000 log2_work=71.762621 tx=48220194 date=‘2018-12-04T12:17:25Z’ progress=0.999774 cache=0.7MiB(5034txo) warning=‘20 of last 100 blocks have unexpected version’ 2018-12-04T17:07:50Z WriteBatch memory usage: db=txindex, before=0.2MiB, after=0.2MiB 2018-12-04T17:07:50Z UpdateTip: new best=00000000000000b9a20c526ed07ab299b85c952d0d290dfcab5e23d78ce91044 height=1446501 version=0x20000000 log2_work=71.762642 tx=48220197 date=‘2018-12-04T12:17:41Z’ progress=0.999774 cache=0.7MiB(5039txo) warning=‘20 of last 100 blocks have unexpected version’ 2018-12-04T17:07:50Z WriteBatch memory usage: db=txindex, before=0.2MiB, after=0.2MiB 2018-12-04T17:07:50Z UpdateTip: new best=00000000b83f71941586cf5c1d61cdb6b1720b5f6c2f089a463f1e7feece25a4 height=1446502 version=0x20000000 log2_work=71.762642 tx=48220206 date=‘2018-12-04T12:37:45Z’ progress=0.999790 cache=0.7MiB(5059txo) warning=‘20 of last 100 blocks have unexpected version’ 2018-12-04T17:07:50Z WriteBatch memory usage: db=txindex, before=0.2MiB, after=0.2MiB 2018-12-04T17:07:50Z UpdateTip: new best=00000000000057f9ad5833bb285426a625eac6aee4b06582a04a8ffd4661753e height=1446503 version=0x20000000 log2_work=71.762642 tx=48220394 date=‘2018-12-04T12:57:46Z’ progress=0.999805 cache=0.7MiB(5353txo) warning=‘20 of last 100 blocks have unexpected version’ 2018-12-04T17:07:50Z WriteBatch memory usage: db=txindex, before=0.2MiB, after=0.2MiB 2018-12-04T17:07:50Z UpdateTip: new best=0000000000004656449376305e090c4cdb5975c3766589270b3c6300a838f4a6 height=1446504 version=0x20000000 log2_work=71.762642 tx=48220479 date=‘2018-12-04T13:17:48Z’ progress=0.999821 cache=0.7MiB(5496txo) warning=‘19 of last 100 blocks have unexpected version’ 2018-12-04T17:07:50Z WriteBatch memory usage: db=txindex, before=0.2MiB, after=0.2MiB 2018-12-04T17:07:50Z UpdateTip: new best=00000000000000bc5445dcc5fe48437651d035dd02b4dc7941e1d7aca89b91af height=1446505 version=0x20000000 log2_work=71.762663 tx=48220556 date=‘2018-12-04T13:35:05Z’ progress=0.999834 cache=0.7MiB(5655txo) warning=‘19 of last 100 blocks have unexpected version’ 2018-12-04T17:07:50Z WriteBatch memory usage: db=txindex, before=0.2MiB, after=0.3MiB 2018-12-04T17:07:50Z UpdateTip: new best=00000000000000c9ad52284a86ce9b3b99c0f6f5d4e8f41032e9119a7981daa6 height=1446506 version=0x3fffe000 log2_work=71.762683 tx=48220586 date=‘2018-12-04T13:41:24Z’ progress=0.999839 cache=0.8MiB(5698txo) warning=‘19 of last 100 blocks have unexpected version’ 2018-12-04T17:07:50Z WriteBatch memory usage: db=txindex, before=0.3MiB, after=0.3MiB 2018-12-04T17:07:50Z UpdateTip: new best=0000000000000126d77f2721cf0961964de348743d572c9dab201e69a24dc29a height=1446507 version=0x20000000 log2_work=71.762704 tx=48220598 date=‘2018-12-04T13:42:32Z’ progress=0.999840 cache=0.8MiB(5719txo) warning=‘19 of last 100 blocks have unexpected version’ 2018-12-04T17:07:50Z WriteBatch memory usage: db=txindex, before=0.3MiB, after=0.3MiB 2018-12-04T17:07:50Z UpdateTip: new best=00000000000e54acb8b3b06a3e4d1c7717922d028173f2e48987bb1c67479b8f height=1446508 version=0x20000000 log2_work=71.762704 tx=48220700 date=‘2018-12-04T14:02:42Z’ progress=0.999856 cache=0.8MiB(5837txo) warning=‘19 of last 100 blocks have unexpected version’ 2018-12-04T17:07:50Z WriteBatch memory usage: db=txindex, before=0.3MiB, after=0.3MiB 2018-12-04T17:07:51Z UpdateTip: new best=00000000000fe82f8eafe6e998a21897f0541e4a8a44ca57db64289fbd37bf33 height=1446509 version=0x20000000 log2_work=71.762704 tx=48220784 date=‘2018-12-04T14:22:51Z’ progress=0.999871 cache=0.8MiB(6035txo) warning=‘19 of last 100 blocks have unexpected version’ 2018-12-04T17:07:51Z WriteBatch memory usage: db=txindex, before=0.3MiB, after=0.3MiB 2018-12-04T17:07:51Z UpdateTip: new best=000000000008874f18591516935c6a9f70ba51e2e595430a45ded60c1243fd64 height=1446510 version=0x20000000 log2_work=71.762704 tx=48220873 date=‘2018-12-04T14:43:07Z’ progress=0.999887 cache=0.8MiB(6169txo) warning=‘19 of last 100 blocks have unexpected version’ 2018-12-04T17:07:51Z WriteBatch memory usage: db=txindex, before=0.3MiB, after=0.3MiB 2018-12-04T17:07:51Z UpdateTip: new best=0000000000042659b84a60782a1c3fe122d819553f4f1a1f8d9d6f11e6e69f6e height=1446511 version=0x20000000 log2_work=71.762704 tx=48220954 date=‘2018-12-04T15:03:18Z’ progress=0.999903 cache=0.8MiB(6306txo) warning=‘19 of last 100 blocks have unexpected version’ 2018-12-04T17:07:51Z WriteBatch memory usage: db=txindex, before=0.3MiB, after=0.3MiB 2018-12-04T17:07:51Z UpdateTip: new best=0000000000000047b77e6797554d88bed4e552516cf36d07c4f2c0fa0f19f2bc height=1446512 version=0x3fffe000 log2_work=71.762725 tx=48220963 date=‘2018-12-04T15:05:29Z’ progress=0.999905 cache=0.8MiB(6320txo) warning=‘20 of last 100 blocks have unexpected version’ 2018-12-04T17:07:51Z WriteBatch memory usage: db=txindex, before=0.3MiB, after=0.3MiB 2018-12-04T17:07:51Z UpdateTip: new best=00000000000c9c09a65437ce5c003b3f4f2cd92571293b9457e24b279e5a57cd height=1446513 version=0x20000000 log2_work=71.762725 tx=48221039 date=‘2018-12-04T15:25:31Z’ progress=0.999920 cache=0.9MiB(6438txo) warning=‘19 of last 100 blocks have unexpected version’ 2018-12-04T17:07:51Z WriteBatch memory usage: db=txindex, before=0.3MiB, after=0.3MiB 2018-12-04T17:07:51Z UpdateTip: new best=000000000000011b9057ea5be9bc46491d4482f4de2cbedd76d95e4fab46a5a6 height=1446514 version=0x20000000 log2_work=71.762746 tx=48221071 date=‘2018-12-04T15:35:22Z’ progress=0.999928 cache=0.9MiB(6489txo) warning=‘19 of last 100 blocks have unexpected version’ 2018-12-04T17:07:51Z WriteBatch memory usage: db=txindex, before=0.3MiB, after=0.3MiB 2018-12-04T17:07:51Z UpdateTip: new best=0000000000000066d74db224a0ee1b2f68093b1f6bb387ef0a6638178317c6aa height=1446515 version=0x20000000 log2_work=71.762767 tx=48221109 date=‘2018-12-04T15:40:49Z’ progress=0.999932 cache=0.9MiB(6554txo) warning=‘19 of last 100 blocks have unexpected version’ 2018-12-04T17:07:51Z WriteBatch memory usage: db=txindex, before=0.3MiB, after=0.3MiB 2018-12-04T17:07:51Z UpdateTip: new best=00000000000000eea4fec653de6e0fefe21f21a22969eac0ae489078b0d84b7c height=1446516 version=0x2000e000 log2_work=71.762788 tx=48221201 date=‘2018-12-04T15:59:17Z’ progress=0.999947 cache=0.9MiB(6686txo) warning=‘20 of last 100 blocks have unexpected version’ 2018-12-04T17:07:51Z WriteBatch memory usage: db=txindex, before=0.3MiB, after=0.3MiB 2018-12-04T17:07:51Z UpdateTip: new best=00000000000000dc81f087c596ab27a61963ba0b614e8cce752bed0fda262363 height=1446517 version=0x20000000 log2_work=71.762809 tx=48221228 date=‘2018-12-04T16:06:21Z’ progress=0.999952 cache=0.9MiB(6713txo) warning=‘19 of last 100 blocks have unexpected version’ 2018-12-04T17:07:51Z [default wallet] AddToWallet c4031ac9ef08555e73295662fcd34ee3a81ce2e7293c5a0c1825a9d5b661cab9
2018-12-04T17:07:51Z [default wallet] AddToWallet 90604551570b0456450c71c776d4f2d338179b8ea5422d76bf69f64227d2c3c7
2018-12-04T17:07:51Z WriteBatch memory usage: db=txindex, before=0.3MiB, after=0.3MiB 2018-12-04T17:07:51Z UpdateTip: new best=0000000000000114892185841fca097ae538cff3b4a2e34f5db795d461f2975f height=1446518 version=0x20000000 log2_work=71.76283 tx=48221244 date=‘2018-12-04T16:12:04Z’ progress=0.999957 cache=0.9MiB(6736txo) warning=‘19 of last 100 blocks have unexpected version’ 2018-12-04T17:07:51Z WriteBatch memory usage: db=txindex, before=0.3MiB, after=0.3MiB 2018-12-04T17:07:51Z UpdateTip: new best=000000000000002738dc0900171dc153d2f05f4fc7e6e7015a6180f16b90bae5 height=1446519 version=0x20000000 log2_work=71.76285 tx=48221245 date=‘2018-12-04T16:12:06Z’ progress=0.999957 cache=0.9MiB(6737txo) warning=‘19 of last 100 blocks have unexpected version’ 2018-12-04T17:07:51Z WriteBatch memory usage: db=txindex, before=0.3MiB, after=0.3MiB 2018-12-04T17:07:51Z UpdateTip: new best=000000000017ea0106e1ada0f470f8d27b41c7ad7d29dfa957bf3580c4b84249 height=1446520 version=0x20000000 log2_work=71.76285 tx=48221308 date=‘2018-12-04T16:32:11Z’ progress=0.999972 cache=0.9MiB(6823txo) warning=‘18 of last 100 blocks have unexpected version’ 2018-12-04T17:07:51Z WriteBatch memory usage: db=txindex, before=0.3MiB, after=0.3MiB 2018-12-04T17:07:51Z UpdateTip: new best=0000000000188a9a88f54251a671f18e553376bebf9f247fe7547e93e37ca1bb height=1446521 version=0x20000000 log2_work=71.76285 tx=48221359 date=‘2018-12-04T16:52:16Z’ progress=0.999988 cache=0.9MiB(6877txo) warning=‘18 of last 100 blocks have unexpected version’ 2018-12-04T17:07:51Z WriteBatch memory usage: db=txindex, before=0.3MiB, after=0.3MiB 2018-12-04T17:07:54Z P2P peers available. Skipped DNS seeding. 2018-12-04T17:07:54Z dnsseed thread exit 2018-12-04T17:08:13Z New outbound peer connected: version: 70015, blocks=1414448, peer=2 2018-12-04T17:08:14Z New outbound peer connected: version: 70015, blocks=1446521, peer=3 2018-12-04T17:08:15Z New outbound peer connected: version: 70015, blocks=1446521, peer=4 2018-12-04T17:08:44Z New outbound peer connected: version: 70015, blocks=1446521, peer=5 2018-12-04T17:08:50Z New outbound peer connected: version: 70015, blocks=1446521, peer=6 2018-12-04T17:08:56Z New outbound peer connected: version: 70015, blocks=1446521, peer=7 2018-12-04T17:09:32Z leveldb: Compacting 1@0 + 2@1 files 2018-12-04T17:09:32Z leveldb: Generated table #12585@0: 3897 keys, 205474 bytes 2018-12-04T17:09:32Z leveldb: Generated table #12586@0: 645 keys, 34023 bytes 2018-12-04T17:09:32Z leveldb: Generated table #12587@0: 287 keys, 15201 bytes 2018-12-04T17:09:32Z leveldb: Compacted 1@0 + 2@1 files => 254698 bytes 2018-12-04T17:09:32Z leveldb: compacted to: files[ 0 3 62 548 658 0 0 ] 2018-12-04T17:09:35Z [default wallet] AddToWallet 60ed8bec64160a4f3dbc7c3ba04ff2856114385da37028180a6848271564c800 new 2018-12-04T17:09:42Z leveldb: Compacting 1@1 + 48@2 files 2018-12-04T17:09:42Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:42Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:42Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:42Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:42Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:42Z leveldb: Generated table #12588@1: 41636 keys, 2170895 bytes 2018-12-04T17:09:42Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:42Z leveldb: compacted to: files[ 0 3 62 548 658 0 0 ] 2018-12-04T17:09:42Z leveldb: Compaction error: IO error: /Users/matt/Library/Application Support/Bitcoin/testnet3/indexes/txindex/012589.ldb: Too many open files 2018-12-04T17:09:42Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:42Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:42Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:42Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:42Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:42Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:42Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:42Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:42Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:42Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:42Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:47Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:47Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:47Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:47Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:47Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:47Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:47Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:47Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:47Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:47Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files 2018-12-04T17:09:52Z libevent: Error from accept() call: Too many open files
jamesob commented at 5:25 pm on December 4, 2018: member
Possibly related to #12495? @RQ732P thanks for the report. Can you rerun this and watch the open file descriptor count and tell us how high it gets? E.g.
0$ watch "lsof -w -p $(pidof bitcoind) | awk 'BEGIN { fd=0; mem=0; } /ldb$/ { if (\$4 == \"mem\") mem++; else fd++ } END { printf \"mem = %s, fd = %s\n\", mem, fd}'"
jamesob commented at 5:29 pm on December 4, 2018: member
@RQ732P can you also post what file descriptors bitcoind has open after you see the error? E.g.
0lsof -w -a -d ^mem -p $(pidof bitcoind)
isghe commented at 11:28 pm on December 4, 2018: contributor

are we sure is it a bug? try to remove the option -debug=leveldb. I am asking, because I run bitcoind (mainnet, testnet and regtest) on macOS from years, and I never had a similar problem. But ok I have to try in testnet with -debug=leveldb. And while is synching is normal it’s not accepting RPC calls.

So, is it crashing, like it is written in the title of this issue “Bitcoind crashes with Too Many Files error”, or is it “Stops accepting RCP calls”?

“Crashing” is not the same of “Stops accepting RCP calls”

ch4ot1c commented at 5:05 am on December 5, 2018: contributor

 02018-12-04T17:07:26Z init message: Loading block index...
 12018-12-04T17:07:26Z LevelDB using max_open_files=1000 (default=1000)
 22018-12-04T17:07:26Z Opening LevelDB in /Users/matt/Library/Application Support/Bitcoin/testnet3/blocks/index
 32018-12-04T17:07:26Z leveldb: Recovering log [#791](/bitcoin-bitcoin/791/)
 42018-12-04T17:07:26Z leveldb: Level-0 table [#801](/bitcoin-bitcoin/801/): started
 52018-12-04T17:07:26Z leveldb: Level-0 table [#801](/bitcoin-bitcoin/801/): 1101 bytes OK
 62018-12-04T17:07:26Z leveldb: Delete type=2 [#788](/bitcoin-bitcoin/788/)
 72018-12-04T17:07:26Z leveldb: Delete type=2 [#785](/bitcoin-bitcoin/785/)
 82018-12-04T17:07:26Z leveldb: Delete type=2 [#784](/bitcoin-bitcoin/784/)
 92018-12-04T17:07:26Z leveldb: Delete type=2 [#790](/bitcoin-bitcoin/790/)
102018-12-04T17:07:26Z leveldb: Delete type=2 [#786](/bitcoin-bitcoin/786/)
112018-12-04T17:07:26Z leveldb: Delete type=2 [#787](/bitcoin-bitcoin/787/)
122018-12-04T17:07:26Z leveldb: Delete type=2 [#783](/bitcoin-bitcoin/783/)
132018-12-04T17:07:26Z leveldb: Delete type=2 [#782](/bitcoin-bitcoin/782/)
142018-12-04T17:07:26Z leveldb: Delete type=2 [#781](/bitcoin-bitcoin/781/)
152018-12-04T17:07:26Z leveldb: Delete type=0 [#791](/bitcoin-bitcoin/791/)
162018-12-04T17:07:26Z leveldb: Delete type=3 [#789](/bitcoin-bitcoin/789/)
172018-12-04T17:07:26Z Opened LevelDB successfully
182018-12-04T17:07:26Z Using obfuscation key for /Users/matt/Library/Application Support/Bitcoin/testnet3/blocks/index: 0000000000000000
192018-12-04T17:07:27Z leveldb: Compacting 1@0 + 8@1 files
202018-12-04T17:07:27Z leveldb: Generated table [#803](/bitcoin-bitcoin/803/)@0: 14888 keys, 2140910 bytes
212018-12-04T17:07:27Z leveldb: Generated table [#804](/bitcoin-bitcoin/804/)@0: 5178 keys, 744418 bytes
222018-12-04T17:07:28Z leveldb: Generated table [#805](/bitcoin-bitcoin/805/)@0: 7347 keys, 1056062 bytes
232018-12-04T17:07:28Z leveldb: Generated table [#806](/bitcoin-bitcoin/806/)@0: 14938 keys, 2136593 bytes
242018-12-04T17:07:28Z leveldb: Generated table [#807](/bitcoin-bitcoin/807/)@0: 1256 keys, 179294 bytes
252018-12-04T17:07:28Z leveldb: Generated table [#808](/bitcoin-bitcoin/808/)@0: 14970 keys, 2138641 bytes
262018-12-04T17:07:28Z leveldb: Generated table [#809](/bitcoin-bitcoin/809/)@0: 3608 keys, 517536 bytes
272018-12-04T17:07:28Z leveldb: Generated table [#810](/bitcoin-bitcoin/810/)@0: 2 keys, 221 bytes
282018-12-04T17:07:28Z leveldb: Compacted 1@0 + 8@1 files => 8913675 bytes
292018-12-04T17:07:28Z leveldb: compacted to: files[ 0 8 55 91 0 0 0 ]
302018-12-04T17:07:33Z LoadBlockIndexDB: last block file = 154
312018-12-04T17:07:33Z LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=5075, size=108826844, heights=1441407...1446454, time=2018-10-31...2018-12-04)
322018-12-04T17:07:33Z Checking all blk files are present...
332018-12-04T17:07:33Z LevelDB using max_open_files=1000 (default=1000)
342018-12-04T17:07:33Z Opening LevelDB in /Users/matt/Library/Application Support/Bitcoin/testnet3/chainstate
352018-12-04T17:07:33Z leveldb: Recovering log [#9327](/bitcoin-bitcoin/9327/)
362018-12-04T17:07:33Z leveldb: Level-0 table [#9332](/bitcoin-bitcoin/9332/): started
372018-12-04T17:07:33Z leveldb: Level-0 table [#9332](/bitcoin-bitcoin/9332/): 308 bytes OK
382018-12-04T17:07:33Z leveldb: Delete type=2 [#9326](/bitcoin-bitcoin/9326/)
392018-12-04T17:07:33Z leveldb: Delete type=0 [#9327](/bitcoin-bitcoin/9327/)
402018-12-04T17:07:33Z leveldb: Delete type=2 [#9311](/bitcoin-bitcoin/9311/)
412018-12-04T17:07:33Z leveldb: Delete type=2 [#9310](/bitcoin-bitcoin/9310/)
422018-12-04T17:07:33Z leveldb: Delete type=2 [#9312](/bitcoin-bitcoin/9312/)
432018-12-04T17:07:33Z leveldb: Delete type=3 [#9325](/bitcoin-bitcoin/9325/)

 02018-12-04T17:07:34Z Loaded best chain: hashBestChain=00000000000000063df115997921bf6856c08a08318b592df576c2f863b7d7c5 height=1446453 date=2018-12-04T04:45:18Z progress=0.999422
 12018-12-04T17:07:34Z init message: Rewinding blocks...
 22018-12-04T17:07:41Z WriteBatch memory usage: db=index, before=0.0MiB, after=0.0MiB
 32018-12-04T17:07:41Z WriteBatch memory usage: db=chainstate, before=0.0MiB, after=0.0MiB
 42018-12-04T17:07:41Z init message: Verifying blocks...
 52018-12-04T17:07:41Z Verifying last 6 blocks at level 3
 62018-12-04T17:07:41Z [0%]...leveldb: Compacting 1@0 + 3@1 files
 72018-12-04T17:07:41Z leveldb: Generated table [#9334](/bitcoin-bitcoin/9334/)@0: 1 keys, 207 bytes
 82018-12-04T17:07:41Z [16%]...leveldb: Generated table [#9335](/bitcoin-bitcoin/9335/)@0: 3162 keys, 181793 bytes
 92018-12-04T17:07:41Z [33%]...leveldb: Generated table [#9336](/bitcoin-bitcoin/9336/)@0: 2846 keys, 181033 bytes
102018-12-04T17:07:41Z leveldb: Compacted 1@0 + 3@1 files => 363033 bytes
112018-12-04T17:07:41Z leveldb: compacted to: files[ 0 3 78 484 0 0 0 ]
122018-12-04T17:07:41Z [50%]...[66%]...[83%]...[99%]...[DONE].
132018-12-04T17:07:41Z No coin database inconsistencies in last 6 blocks (444 transactions)
142018-12-04T17:07:41Z block index 15119ms
152018-12-04T17:07:41Z LevelDB using max_open_files=1000 (default=1000)
162018-12-04T17:07:41Z Opening LevelDB in /Users/matt/Library/Application Support/Bitcoin/testnet3/indexes/txindex
172018-12-04T17:07:41Z leveldb: Recovering log [#12582](/bitcoin-bitcoin/12582/)
182018-12-04T17:07:41Z leveldb: Delete type=3 [#12580](/bitcoin-bitcoin/12580/)
192018-12-04T17:07:41Z leveldb: Delete type=0 [#12582](/bitcoin-bitcoin/12582/)

This part is most suspicious

tryphe commented at 5:50 pm on May 6, 2019: contributor

~Looks like this is caused by #12495, which raises the levelDB maximum file handles from 64 to ~1024~ 1000, except on Windows. Default is here: https://github.com/bitcoin/bitcoin/blob/master/src/leveldb/include/leveldb/options.h#L89~

When i run ulimit -n on Mac Sierra 10.13, I get 256. I think this is typical (for Macs). ~This could cause problems.~
gmaxwell commented at 7:40 pm on May 6, 2019: contributor

@tryphe I’m pretty sure your comment is confused.
tryphe commented at 11:34 pm on May 6, 2019: contributor

Ok, so after some IRC discussion: leveldb utilizes mmap which only uses one file descriptor for a short amount of time while the mapping is obtained synchronously. Therefore, the leveldb options aren’t the issue, but rather the problem most likely stems from the number of RPC threads not counting towards the per-process file descriptor limit being checked during daemon initialization via RaiseFileDescriptorLimit() called here. In other words, because each RPC thread ~could use >=0 file descriptors during a call~ uses 1 file descriptor for accept(), this per-process descriptor limit might be reached, causing mmap to be unable to obtain any descriptors, which would cause the compaction error above and other “Too many open files” errors. Edit: it seems bind() uses one as well, which doesn’t count towards the fd limit requested below. (thanks to gwillen for clarification)

Furthermore, on MacOS, if the descriptor limit has not been modified on the system, RaiseFileDescriptorLimit() should (at least that’s what it looks like to me) raise the current process’ file descriptor limit from 256 to 283 on MacOS 10 with the default arguments (8 + 150 + 125). On most Linux systems, the default is 1024, which is greater than 283, so 1024 is used instead. Seeing that the difference in descriptors between Unix and Unix-like platforms can be quite large, ~I think some further testing is necessary with particularly file-intensive RPC calls in order to see how a more dynamic descriptor limit could possibly be defined. For example: if I reduce the descriptor limit on a Linux install from 1024 to 283, how many RPC threads would I need to have and how many calls to call X would I need to make for this limit to be approached?~ Edit: I’ve created a branch to “test” an improved fd limit, but I cannot reproduce this bug.
tryphe commented at 8:56 am on May 8, 2019: contributor

@RQ732P can you try my branch to see if it fixes your problem?

~git clone https://github.com/tryphe/bitcoin --branch fd-limits~

Edit: New branch to try:

git clone https://github.com/tryphe/bitcoin --branch fd-limits-3
MarcoFalke added the label Resource usage on Apr 28, 2020
MarcoFalke commented at 11:20 pm on May 8, 2020: member

Is this still an issue with a recent version of Bitcoin Core? If yes, what are the steps to reproduce?
MarcoFalke closed this on May 8, 2020

j4ys0n commented at 8:51 pm on June 4, 2021: none

This is still an issue.

02021-06-04T20:01:11Z libevent: Error from accept() call: Too many open files
12021-06-04T20:01:11Z LevelDB read failure: IO error: /Users/user/Library/Application Support/Bitcoin/chainstate/619393.ldb: Too many open files
22021-06-04T20:01:11Z Fatal LevelDB error: IO error: /Users/user/Library/Application Support/Bitcoin/chainstate/619393.ldb: Too many open files
32021-06-04T20:01:11Z You can use -debug=leveldb to get more complete diagnostic messages
42021-06-04T20:01:11Z Error: Error reading from database, shutting down.
52021-06-04T20:01:11Z Error reading from database: Fatal LevelDB error: IO error: /Users/user/Library/Application Support/Bitcoin/chainstate/619393.ldb: Too many open files

Looks like the issue is recursive RPC calls. However, i’m doing them in sequence, not all at once, so it shouldn’t be overwhelming.

tryphe commented at 5:10 am on July 27, 2021: contributor

@j4ys0n thanks for reporting.

Is there a script you have which can consistently reproduce the crash on MacOS? If so, feel free to ping one of the maintainers with an @ in order to re-open the issue.
MarcoFalke reopened this on Jul 27, 2021
MarcoFalke added the label macOS on Jul 27, 2021
MarcoFalke added the label RPC/REST/ZMQ on Jul 27, 2021
sr-gi commented at 6:28 pm on June 21, 2023: none

Are there any steps to reproduce this?
MarcoFalke commented at 6:32 pm on June 21, 2023: member

maybe related #27732?

Contributors
SanJoseCosta jamesob isghe ch4ot1c tryphe gmaxwell MarcoFalke j4ys0n sr-gi

Labels
macOS RPC/REST/ZMQ Resource usage