During IBD, when doing pruning, prune 10% extra to avoid pruning again soon after #11658

1. 
   luke-jr commented at 9:08 am on November 11, 2017: member

   Pruning forces a chainstate flush, which can defeat the dbcache and harm performance significantly.

   Alternative to #11359

2. During IBD, when doing pruning, prune 10% extra to avoid pruning again soon after

   Pruning forces a chainstate flush, which can defeat the dbcache and harm performance significantly.

   ac51a26bdc
3. fanquake added the label Validation on Nov 11, 2017
4. in src/validation.cpp:3399 in ac51a26bdc

   3390@@ -3391,6 +3391,15 @@ void FindFilesToPrune(std::set<int>& setFilesToPrune, uint64_t nPruneAfterHeight
   3391     int count=0;
   3392 
   3393     if (nCurrentUsage + nBuffer >= nPruneTarget) {
   3394+        // On a prune event, the chainstate DB is flushed.
   3395+        // To avoid excessive prune events negating the benefit of high dbcache
   3396+        // values, we should not prune too rapidly.
   3397+        // So when pruning in IBD, increase the buffer a bit to avoid a re-prune too soon.
   3398+        if (IsInitialBlockDownload()) {
   3399+            // Since this is only relevant during IBD, we use a fixed 10%
   

   ---

   ---

   

   promag commented at 4:14 pm on November 15, 2017:

   BTW, why 10%, in IBD could be even more no?
5. 
   Sjors commented at 11:29 am on December 7, 2017: member

   Some IRC discussion.
6. Sjors referenced this in commit 03c20f3cc5 on Feb 10, 2018
7. Sjors referenced this in commit c3eea611eb on Feb 10, 2018
8. Sjors referenced this in commit a9e8bb4640 on Feb 10, 2018
9. Sjors referenced this in commit a3716db4c4 on Feb 10, 2018
10. Sjors referenced this in commit 132c88e64f on Feb 19, 2018
11. Sjors referenced this in commit d49bab490a on Feb 19, 2018
12. Sjors referenced this in commit 86bef23e65 on Feb 19, 2018
13. Sjors referenced this in commit 22b81deb24 on Mar 12, 2018
14. Sjors referenced this in commit 82efbf1e8a on Mar 26, 2018
15. 
    Sjors commented at 11:06 am on May 28, 2018: member

    tACK ac51a26 (although I rebased it on master).

    This performs better than master in my tests so far: #12404 (comment) @promag wrote:

    BTW, why 10%, in IBD could be even more no?

    I tried this in #12404 and there seems to be negative side effects that I can’t explain. I’d say this PR is a safer choice for now. @luke-jr you may want to borrow the release note I added to my PR, though that can be done later.

16. 
    Sjors commented at 12:06 pm on June 14, 2018: member

    I’m currently racing two Orange Pi’s with 2 GB memory. This PR means cache rarely grows beyond 300 MB, but it’s still a bit faster than master; after 5 days it’s at 2018-01-27 vs. master is still at 2017-11-25.
17. laanwj added this to the "Blockers" column in a project

    ---

18. 
    sipa commented at 3:05 am on July 6, 2018: member

    utACK ac51a26bdc69dc35e1f4f89b62c3134047e93bc1. This should be a safe first improvement. We can later make the overshoot value configurable or smarter, but this should already be a lot better.
19. MarcoFalke added the label Needs release notes on Jul 7, 2018
20. MarcoFalke removed the label Needs release notes on Jul 7, 2018
21. 
    promag commented at 10:04 pm on July 7, 2018: member

    utACK ac51a26. Improvement looks good.
22. 
    ajtowns commented at 6:50 am on July 12, 2018: member

    utACK ac51a26bdc69dc35e1f4f89b62c3134047e93bc1
23. 
    achow101 commented at 10:56 pm on July 13, 2018: member

    utACK ac51a26bdc69dc35e1f4f89b62c3134047e93bc1
24. sipa merged this on Jul 14, 2018
25. sipa closed this on Jul 14, 2018

    ---

26. sipa referenced this in commit 9a1ad2c5cb on Jul 14, 2018
27. fanquake removed this from the "Blockers" column in a project

    ---

28. 
    ajtowns commented at 7:40 am on July 16, 2018: member

    Post-merge tested ACK.

    I ran this on some memory-constrained Linodes (dbcache=1000, prune=5000, and both connect= to a single nearby fully synced node; 2GB memory, 50GB disk), and saw a reasonably consistent 15% performance increase compared to not having this patch.

29. 
    martinpgaunt commented at 7:48 am on July 16, 2018: none

    How use this ?
30. fanquake deleted a comment on Jul 16, 2018
31. 
    Sjors commented at 8:30 am on July 16, 2018: member

    @martinpgaunt it just works, if you download Bitcoin Core 0.17 when it comes out in a couple of weeks (or a bit later).
32. jasonbcox referenced this in commit 966fa58918 on Sep 13, 2019
33. jonspock referenced this in commit 518957562a on Dec 22, 2019
34. proteanx referenced this in commit e8a3e85e10 on Dec 23, 2019
35. PastaPastaPasta referenced this in commit d7b71db2b9 on Jul 19, 2020
36. PastaPastaPasta referenced this in commit 0c7ba8e4c9 on Jul 24, 2020
37. PastaPastaPasta referenced this in commit 474e20ff38 on Jul 27, 2020
38. UdjinM6 referenced this in commit d3f7be5b84 on Jul 27, 2020
39. UdjinM6 referenced this in commit 0eb337028b on Jul 27, 2020
40. DrahtBot locked this on Sep 8, 2021

Contributors
luke-jr promag Sjors sipa ajtowns achow101 martinpgaunt

Labels
Validation