Clean up calculations of pcoinsTip memory usage #10133

1. 
   morcos commented at 2:33 pm on March 31, 2017: member

   In #10126 we changed the calculation of when to flush so that the requested dbcache memory limit would be respected even by the peak usage occurring at flush.

   Without doubling the default dbcache limit, this would lead to an effective default of half the previous which I believe is too low.

   First commit doubles the default and applies the DB_PEAK_USAGE_FACTOR in 2 other places. The debug logging of the usage still reports the current usage.

   The second commit lowers the defaults slightly to ensure 0.14.1 has a lower memory footprint than 0.13.2.

   The third commit makes the delayed flushing on large dbcaches a bit less aggressive, so the limit is less likely to be violated in practice.

2. 
   JeremyRubin commented at 3:02 pm on March 31, 2017: contributor

   utack 99b814f.
3. Make pcoinsTip memory calculations consistent

   Since we are more accurately measuring pcoinsTip peak usage at twice the current in dynamic usage, it makes sense to double the default (this will lead to the same effective usage and peak usage as previously).
   We should also double the buffer used to avoid flushing if above 90% but still sufficient space remaining.

   5b95a190e8
4. Lower default memory footprint slightly f33afd3b2b
5. Make threshold for flushing more conservative.

   Always leave a reasonable buffer of 50MB for usage from newly connected block (once over 50%) and increase the high water mark buffer to 200MB.

   1b55e07b7a
6. in src/validation.cpp:3642 in 99b814ff16 outdated

   3638@@ -3639,7 +3639,7 @@ bool CVerifyDB::VerifyDB(const CChainParams& chainparams, CCoinsView *coinsview,
   3639             }
   3640         }
   3641         // check level 3: check for inconsistencies during memory-only disconnect of tip blocks
   3642-        if (nCheckLevel >= 3 && pindex == pindexState && (coins.DynamicMemoryUsage() + pcoinsTip->DynamicMemoryUsage()) <= nCoinCacheUsage) {
   3643+        if (nCheckLevel >= 3 && pindex == pindexState && (coins.DynamicMemoryUsage() + DB_PEAK_USAGE_FACTOR * pcoinsTip->DynamicMemoryUsage()) <= nCoinCacheUsage) {
   

   ---

   ---

   

   sipa commented at 4:54 pm on March 31, 2017:

   The changes in the rollback test are never flushed, so there is no need for compensating for a peak?

   ---

   

   morcos commented at 5:11 pm on March 31, 2017:

   agreed. assumed coins was built on top of pcoinstip, missed that. will change.
7. 
   morcos commented at 6:39 pm on March 31, 2017: member

   Fixed sipa’s issue.

   After discussion I decided to remove the lowering of maxmempool until we are more confident that maxes sense.

   I also made the flushing buffer a bit smarter.

8. morcos force-pushed on Mar 31, 2017
9. 
   sdaftuar commented at 7:06 pm on March 31, 2017: member

   utACK 1b55e07b7a61a9e6c299cf4c40fde80fa715d440

   We should tag this for 0.14.1

10. 
    sipa commented at 7:09 pm on March 31, 2017: member

    Benchmarking.
11. fanquake added this to the milestone 0.14.1 on Mar 31, 2017
12. fanquake added the label Resource usage on Apr 1, 2017
13. laanwj merged this on Apr 5, 2017
14. laanwj closed this on Apr 5, 2017

    ---

15. laanwj referenced this in commit 5fc6a77aa6 on Apr 5, 2017
16. laanwj referenced this in commit ab864d3e49 on Apr 5, 2017
17. laanwj referenced this in commit d6867689d1 on Apr 5, 2017
18. laanwj referenced this in commit 0e5133cb2e on Apr 5, 2017
19. 
    laanwj commented at 6:24 am on April 5, 2017: member

    utACK 1b55e07

    BTW @fanquake if you tag things for backport please also add the “needs backport” tag, this allows for filtering what has not been backported yet.

20. codablock referenced this in commit a80dafff74 on Sep 27, 2017
21. codablock referenced this in commit 10db648901 on Oct 12, 2017
22. codablock referenced this in commit ff9b2967af on Oct 24, 2017
23. UdjinM6 referenced this in commit b0cbebd6d1 on Nov 8, 2017
24. PastaPastaPasta referenced this in commit 587db5c9bc on May 20, 2019
25. PastaPastaPasta referenced this in commit ead476b9a1 on May 20, 2019
26. PastaPastaPasta referenced this in commit bf033dc5e9 on May 21, 2019
27. PastaPastaPasta referenced this in commit 1821673936 on May 22, 2019
28. PastaPastaPasta referenced this in commit bcc215746e on May 22, 2019
29. PastaPastaPasta referenced this in commit bdc4564ca4 on May 22, 2019
30. PastaPastaPasta referenced this in commit 894301291f on May 22, 2019
31. PastaPastaPasta referenced this in commit e5b57f5d3d on May 23, 2019
32. UdjinM6 referenced this in commit 05fb99be9a on May 23, 2019
33. PastaPastaPasta referenced this in commit 03f902367e on May 23, 2019
34. UdjinM6 referenced this in commit 8409de724c on May 27, 2019
35. DrahtBot locked this on Sep 8, 2021

Contributors
morcos JeremyRubin sipa sdaftuar laanwj

Labels
Resource usage

Milestone
0.14.1