OSX open file limit is less than LevelDB's assumptions #2517

DarkHelmet433 commented at 1:44 AM on April 13, 2013: none

By default, OSX has a soft RLIMIT_NOFILE of 256. According to IRC: 06:26 <sipa> leveldb assumes 1000

I had the misfortune of having a crash due to a driver bug. Bitcoin-Qt complained about a corrupt chainstate database and prompted to reindex. This was the result:

<pre> SetBestChain: new best=00000000000003e98ba4932e87919aac4bd618f4584e475231fe2834556ff174 height=192473 work=415632761278238416140 tx=5651224 date=2012-08-05 19:57:03 ProcessBlock: ACCEPTED Committing 39050 changed transactions to coin database... SetBestChain: new best=000000000000051905ee767096eee37b163937d7340633fa037089372807c3ac height=192474 work=415641508847434375834 tx=5651608 date=2012-08-05 20:15:11 ProcessBlock: ACCEPTED SetBestChain: new best=00000000000004a983540823a4fda18116ec409ad67913aa37462d2c87e4ea21 height=192475 work=415650256416630335528 tx=5651618 date=2012-08-05 20:34:12 ProcessBlock: ACCEPTED SetBestChain: new best=000000000000042a12b3e291db2b5c9d23b7d1b4d2948ebd6cbd13807cdd0ce2 height=192476 work=415659003985826295222 tx=5651627 date=2012-08-05 20:34:47 ProcessBlock: ACCEPTED SetBestChain: new best=00000000000002017b0005688a315f334d10a014b7118602a4db11ad77bb9e39 height=192477 work=415667751555022254916 tx=5652818 date=2012-08-05 20:44:56 ProcessBlock: ACCEPTED SetBestChain: new best=0000000000000429b8f10461d5bfbff40ea07ad7b1573bcaf04187627e866572 height=192478 work=415676499124218214610 tx=5652953 date=2012-08-05 20:48:33 ProcessBlock: ACCEPTED SetBestChain: new best=000000000000063a28265a2dc91e8e3fedb27517d1e0af995d78aed65d0a5cb4 height=192479 work=415685246693414174304 tx=5653098 date=2012-08-05 20:52:59 ProcessBlock: ACCEPTED SetBestChain: new best=00000000000006994b5e60e0d7883d1252d180b880d333dab461070e49025a38 height=192480 work=415693994262610133998 tx=5653656 date=2012-08-05 21:05:22 ProcessBlock: ACCEPTED SetBestChain: new best=000000000000020e5e4ed55aaaab9cf3aa1bbca102ee7c8577b6c5b15d8e682f height=192481 work=415702741831806093692 tx=5654460 date=2012-08-05 21:19:01 ProcessBlock: ACCEPTED LevelDB write failure: IO error: /Users/peter/Library/Application Support/Bitcoin/blocks/index/003187.log: Too many open files *** System error: Database I/O error ERROR: ProcessBlock() : AcceptBlock FAILED Loaded 3953 blocks from external file in 2270181ms Flush(false) wallet.dat refcount=0 ThreadSocketHandler exited ThreadMessageHandler exited wallet.dat checkpoint ThreadOpenConnections exited wallet.dat detach wallet.dat closed DBFlush(false) ended 731ms StopNode() Flushed 0 addresses to peers.dat 1ms Committing 4858 changed transactions to coin database... Flush(true) DBFlush(true) ended 0ms Bitcoin exited </pre>

I have seen Bitcoin-Qt abort due to running out of fd's during normal operation.

My bitcoin.conf has txindex=1.

I manually raised the fd limit to 4096 and reran it.

I'm tempted to compile from github with a patch loosely like this:

<pre> #include <sys/resource.h> ... struct rlimit lim; ... if (getrlimit(RLIMIT_NOFILE, &lim) != -1) { if (lim.rlim_cur < 4096) lim.rlim_cur = 4096; if (lim.rlim_cur > lim.rlim_max) lim.rlim_cur = lim.rlim_max; setrlimit(RLIMIT_NOFILE, &lim); /* ignore error, or log a warning */ } </pre>

ie: try to raise fd limit to either 4096, or the hard limit.. whichever is lower.

sipa commented at 10:55 PM on April 23, 2013: member

#2557 should improve upon this, but we should at least try increasing the ulimit to 150 + maxconnections or so, and /or decrease maxconnections otherwise.

sipa commented at 10:54 PM on April 25, 2013: member

I've created a pull request from your snippet: #2568

sipa commented at 11:11 PM on April 29, 2013: member

Closed via #2557 and #2568

sipa closed this on Apr 29, 2013

MarcoFalke locked this on Sep 8, 2021