Reduce number of open LevelDB files to 64 #2557

1. 
   sipa commented at 10:14 pm on April 23, 2013: member

   I don’t notice any significant IBD performance degradation using this.
2. Reduce number of open LevelDB files to 64 2233fc5318
3. 
   gmaxwell commented at 10:19 pm on April 23, 2013: contributor

   ACK. Reduces FD usage in my torture test from 500 to 112.

   However, 112 + 125 is awful close to 256, so I’m still concerned about OSX.

4. gavinandresen referenced this in commit fe15aa335c on Apr 24, 2013
5. gavinandresen merged this on Apr 24, 2013
6. gavinandresen closed this on Apr 24, 2013

   ---

7. 
   Diapolo commented at 3:47 pm on April 25, 2013: none

   This seems to cause massive issues for me on Win7 x64, I often get the message that I need to reindex because of a broken database or sometimes only Database I/O error. I now removed that patch and start over to see if that is helping me.
8. 
   sipa commented at 10:04 pm on April 25, 2013: member

   @Diapolo That would be bad and unexpected news. I don’t see how that could cause/worsen that, but we never know of course. Can you confirm it’s related to this?
9. 
   Diapolo commented at 8:06 am on April 26, 2013: none

   I sent @gmaxwell a copy of my coinstate folder, with which I had the problem. Currently I’m 73 weeks behind with a client that doesn’t have this patch active. After it synced without error I’ll restart with this patch applied and a bootstrap (which contains 50% of the current blocks on the net) + re-sync.
10. 
    Diapolo commented at 3:07 pm on April 26, 2013: none

    No crashes without the patch… testing with it again now.
11. 
    Diapolo commented at 4:05 pm on April 26, 2013: none

    Again, Database I/O error with the patch applied… 2013-04-26 16:02:19 LevelDB read failure: IO error: C:\Users\Diapolo\AppData\Roaming\Bitcoin\chainstate\000223.sst: Could not create random access file.
12. 
    Diapolo commented at 5:46 pm on April 26, 2013: none

    Perhaps we should just remove that patch for Windows, as per http://msdn.microsoft.com/en-us/library/6e3b887c.aspx

    “Up to 2,048 files can be open simultaneously at the lowio level (that is, opened and accessed by means of the _open, _read, _write, and so forth family of I/O functions). Up to 512 files can be open simultaneously at the stdio level (that is, opened and accessed by means of the fopen, fgetc, fputc, and so forth family of functions). The limit of 512 open files at the stdio level can be increased to a maximum of 2,048 by means of the _setmaxstdio function.”

13. 
    sipa commented at 6:05 pm on April 26, 2013: member

    @Diapolo I’d rather find the reason for this crash, as this patch should not have any difference at all during IBD.
14. 
    sipa commented at 0:54 am on April 27, 2013: member

    @Diapolo Can you retry with #2580 compiled in?
15. 
    Diapolo commented at 10:06 am on April 27, 2013: none

    @sipa Sure, I’ll take a look later…
16. 
    Diapolo commented at 10:40 am on April 27, 2013: none

    @sipa

    Seems the error was the same as I posted in the chat yesterday: 2013-04-27 10:15:34 LevelDB read failure: IO error: C:\Users\Diapolo\AppData\Roaming\Bitcoin\chainstate\000446.sst: Could not create random access file. 2013-04-27 10:15:34 LevelDB error encountered: IO error: C:\Users\Diapolo\AppData\Roaming\Bitcoin\chainstate\000446.sst: Could not create random access file. 2013-04-27 10:15:34 *** Systemfehler: Database I/O error

17. 
    sipa commented at 6:08 pm on April 27, 2013: member

    @Diapolo Feel like retrying with the commits in https://github.com/sipa/bitcoin/commits/win32ldberr ?
18. sipa deleted the branch on May 7, 2013
19. laudney referenced this in commit 5bb1c85d80 on Mar 19, 2014
20. DrahtBot locked this on Sep 8, 2021

Contributors
sipa gmaxwell Diapolo

Milestone
v0.8.2