Do not flush the cache after every block outside of IBD #4505

1. 
   sipa commented at 3:38 pm on July 10, 2014: member

   It seems the reason why jenkins has been failing for several months is that a 200-pathological-block reorg takes over 20 minutes, at which time a timeout occurs.

   The reason is that since a refactor, we’re now doing reorganizations in steps (and since more recently, even releasing cs_main in between if possible), which means that outside of IBD, we’re dumping the entire coins database (over a million txids in the large-reorg test!) to disk, and removing it from memory after every block.

   This change, in combination with a sufficiently large -dbcache may be enough to bring that time down.

2. sipa renamed this:
   Do not flush the cache after every block during IBD
   Do not flush the cache after every block outside of IBD
   on Jul 10, 2014
3. 
   gmaxwell commented at 3:42 pm on July 10, 2014: contributor

   ACK. Should we also consider a hidden switch to control the TIMEOUT_INTERVAL?
4. 
   TheBlueMatt commented at 3:48 pm on July 10, 2014: member

   would be preferable to monkey-sed’ing during testing
5. 
   jgarzik commented at 3:49 pm on July 10, 2014: contributor

   ut ACK
6. 
   laanwj commented at 3:23 am on July 11, 2014: member

   Oops. Good catch!

   But pulltester seems to have a problem with it:

   Transactions which were/weren’t in memory pool but shouldn’t/should have been: 1

7. 
   sipa commented at 2:04 pm on July 11, 2014: member

   If #4494 improves some performance, it’s likely to make a difference in that large-reorg test too.
8. Do not flush the cache after every block outside of IBD d4d3fbd828
9. 
   sipa commented at 2:48 pm on July 11, 2014: member

   Thanks @BitcoinPullTester! Fixed a bug that was exposed by this change.

   When a reorganization happens, the disconnected blocks’ transaction outputs were wiped from the CCoinsCache, but the empty entries remained there, preventing them from being revived if the same transactions occurred in the other branch too.

   In case a cache flush happens in between the disconnect and connect (as used to be the case, at least outside of IBD), this is not an issue.

10. 
    BitcoinPullTester commented at 3:10 pm on July 11, 2014: none

    Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/p4505_d4d3fbd828a477e8459169a097d5f1bfb69c2781/ for binaries and test log. This test script verifies pulls every time they are updated. It, however, dies sometimes and fails to test properly. If you are waiting on a test, please check timestamps to verify that the test.log is moving at http://jenkins.bluematt.me/pull-tester/current/ Contact BlueMatt on freenode if something looks broken.
11. laanwj merged this on Jul 15, 2014
12. laanwj closed this on Jul 15, 2014

    ---

13. laanwj referenced this in commit 2c0f019bfc on Jul 15, 2014
14. MarcoFalke locked this on Sep 8, 2021

Contributors
sipa gmaxwell TheBlueMatt jgarzik laanwj BitcoinPullTester