As noted in #11368 if too many connections are made to the RPC interface, then other code will fail on open(2) syscalls with EMFILE. The result can be that the block database gets into an inconsistent state.
On many Linux distributions, by default, each process has 1024 file descriptors; these are shared between open files and network connections. The main init code attempts to apportion them between uses, but neglects to constrain the RPC layer: https://github.com/bitcoin/bitcoin/blob/master/src/init.cpp#L907
Unfortunately, libevent does not allow a natural way to bound the number of file-descriptors used by the evhttp server. Therefore, we have to resort to requesting to stop new connections by disabling the accept listener in the epoll event structure. This is not a good way to control load, and more connections are accepted until the next epoll cycle is triggered, but it does stop an unbounded number of connections from being created, and does prevent a high number of connections to the RPC layer from damaging the rest of the system.
To avoid problems of a similar nature, the second patch additionally raises the rlimit of number of file descriptors as high as it can go.
To repro the database crash and validate the fix, the following node.js fragment:
0var uri = 'http://127.0.0.1:8332/rest/block/000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f.json'
1for (var message = 0; message < 10000; message++) {
2 request(uri)
3}
The messages around the database crash due to open(2) failing due to too many open files
2017-11-26 19:35:55 libevent: Error from accept() call: Too many open files 2017-11-26 19:35:55 ERROR: WriteBlockToDisk: OpenBlockFile failed 2017-11-26 19:35:55 libevent: timeout_next: event: 0x7f59001dcef0, in 15 seconds, 475453 useconds 2017-11-26 19:35:55 *** Failed to write block 2017-11-26 19:35:55 libevent: epoll_dispatch: epoll_wait reports 1 2017-11-26 19:35:55 Error: Error: A fatal internal error occurred, see debug.log for details
