This should reduce pruning flushes even more, speeding up IBD with pruning on systems that have a sufficient dbcache.
Assumes 1 MB per block between tip and best header chain. Simply adds this to the buffer pruning is trying to leave available, which results in pruning almost everything up until we get close to where we need to be keeping blocks.
<!--e57a25ab6845829454e8d69fc972939a-->
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
<!--006a51241073e994b41acfe9ec718e94-->
For detailed information about the code coverage, see the test coverage report.
<!--021abf342d371248e50ceaed478a90ca-->
See the guideline for information on the review process.
| Type | Reviewers |
|---|---|
| ACK | andrewtoth, fjahr, achow101 |
| Concept ACK | jonasschnelli, theStack, 0xB10C, jonatack, kristapsk |
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.
<!--174a7506f384e20aa4161008e828411d-->
No conflicts as of last run.
Interesting, will have a look.
Nice!
Concept ACK.
Could we further reduce flushes by allowing a user parameter (-pruneflushbuffer [or similar])? Some users are probably okay with providing a few GBs of extra space for speeding up pruning (assume hight dbcache) (but wan't the extra space be freed after IBD).
Or we could automatically use he dbcache size as the buffer size.
I also like the idea of temporally assigning more RAM for IBD. This could even be done by default for GUI users (with opt-out in the intro screen).
4023 | - // Since this is only relevant during IBD, we use a fixed 10% 4024 | - nBuffer += nPruneTarget / 10; 4025 | + // So when pruning in IBD, increase the buffer to avoid a re-prune too soon. 4026 | + if (is_ibd && target_sync_height > (uint64_t)chain_tip_height) { 4027 | + // Since this is only relevant during IBD, we assume blocks are at least 1 MB on average 4028 | + static constexpr uint64_t average_block_size = 1000000; /* 1 MB */
AFAIK we don't have access to future block sizes at this point.
Without rebase, compiling fails for me:
$ make clean && ./configure CXXFLAGS="-O0 -g" CFLAGS="-O0 -g" && make -j 9
...
touch src/config/bitcoin-config.h.in
CXX util/libbitcoin_util_a-moneystr.o
CXX util/libbitcoin_util_a-readwritefile.o
CXX util/libbitcoin_util_a-settings.o
CXX util/libbitcoin_util_a-serfloat.o
CXX util/libbitcoin_util_a-spanparsing.o
CXX util/libbitcoin_util_a-strencodings.o
CXX util/libbitcoin_util_a-string.o
CXX util/libbitcoin_util_a-url.o
make[3]: *** No rule to make target `libunivalue.la'. Stop.
make[2]: *** [univalue/libunivalue.la] Error 2
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [all-recursive] Error 1
make: *** [all-recursive] Error 1
After git rebase master, everything runs smoothly again. Will report back shortly with my review results!
@stickies-v If you want to compile older commits of master, you'll need to make distclean first.
Without rebase, compiling fails for me:
Note: It's usually best to merge PRs into master for testing. Sometimes (bugfixes) they can be based on very old commits (when the bug was introduced); and if there's a silent problem with the merge, you'd want to notice that too.
@stickies-v If you want to compile older commits of master, you'll need to make distclean first.
Can we fix this build system bug?
Concept ACK
Concept ACK
We'll be doing a Bitcoin Core PR Review club covering this PR on the 29th: bitcoincore.reviews/20827
Concept ACK, I read the PR Review Club and will review after rebase.
fwiw: #23581 moved BlockManager to node/blockstorage, that's where the conflict with master happens.
I've rebased this PR in https://github.com/0xB10C/bitcoin/tree/2022-01-lukejr-ibd-prune-max-rebased and started benchmarking https://github.com/0xB10C/bitcoin/commit/db2a9e71a748c4806b85f767441fb3f090d62163 (PR) vs https://github.com/0xB10C/bitcoin/commit/2e01b6986099715afa40ed6464da4b321b630e9c (mergebase; MB).
I have 8 different prune and dbcache configurations set-up. Each configuration runs three times per binary (PR and MB). That's 8 configs * 3 runs * 2 binaries = 48. I expect this to take about 6 days. Configurations:
1. -dbcache=300 -prune=550
2. -dbcache=4000 -prune=550
3. -dbcache=300 -prune=1100
4. -dbcache=4000 -prune=1100
5. -dbcache=300 -prune=2200
6. -dbcache=4000 -prune=2200
7. -dbcache=300 -prune=4400
8. -dbcache=4000 -prune=4400
Benchmarks run on a dedicated, properly cooled, and otherwise idle machine with two HDDs in a RAID0. The node syncs from a local node on the same machine and disks (connected via addnode and otherwise connect=0). The sync starts at block height 500_000 and ends at height 600_000. Between each run, the machine idles 5 minutes. The debug.log, with extra debug=coindb debug=prune logging, of each run is saved. I'll post results and graphs next week.
With a prune just before IBD, there is more blk and undo data available when running master than with this change. One potential issue that was mentioned during the PR review club:
On non-mainnet networks, the 1MB per block assumption doesn't hold. When we prune with a large nBuffer just before IBD ends, the prune might cause problems on larger reorgs as not enough undo data is present to revert back to the last shared block. Testnet has seen large reorgs in the past.
Might be worth thinking about potential alternatives to increasing the nBuffer by 1MB * remaining_blocks. There might be cleaner ways to implement "prune everything as long as we are not close to IBD being done". Haven't come up with a practical threshold for "IBD being close to done" yet that works for all networks.
The 1 MB per block assumption would mean at least 550 blocks. Testnet has deeper reorgs than that?
The 1 MB per block assumption would mean at least 550 blocks. Testnet has deeper reorgs than that?
I haven't observed this personally, but yes testnet has been unreliable in regards to deep reorgs in the past. That's one motivation for SigNet in BIP 325. Also:
benchmarking 0xB10C@db2a9e7 (PR) vs 0xB10C@2e01b69 (mergebase; MB).
I have 8 different prune and dbcache configurations set-up. Each configuration runs three times per binary (PR and MB). That's 8 configs * 3 runs * 2 binaries = 48. I expect this to take about 6 days. Configurations:
1. -dbcache=300 -prune=550 2. -dbcache=4000 -prune=550 3. -dbcache=300 -prune=1100 4. -dbcache=4000 -prune=1100 5. -dbcache=300 -prune=2200 6. -dbcache=4000 -prune=2200 7. -dbcache=300 -prune=4400 8. -dbcache=4000 -prune=4400Benchmarks run on a dedicated, properly cooled, and otherwise idle machine with two HDDs in a RAID0. The node syncs from a local node on the same machine and disks (connected via
addnodeand otherwiseconnect=0). The sync starts at block height 500_000 and ends at height 600_000. Between each run, the machine idles 5 minutes. The debug.log, with extradebug=coindb debug=prunelogging, of each run is saved. I'll post results and graphs next week.
Benchmarks done. Happy to share debug.logs and code if anyone wants to look into it or reproduce.
tl;dr: With some configurations, an IBD performance improvement of 15% or more between block 500k and 600k was measured.
This PR makes pruning more aggressive. We prune more blk/rev files in a single prune operation to minimize the total number of prune operations needed. This also reduces the number of expensive dbcache flushes as each prune operation requires a dbcache flush.
| config | MB prunes | PR prunes | prunes | MB pruned files | PR pruned files | pruned files | MB flushes | PR flushes | flushes |
|---|---|---|---|---|---|---|---|---|---|
| 0: prune=550 dbcache=300 | 726 | 670 | 92.29% | 1 | 1.08 | 1.08x | 726 | 670 | 92% |
| 1: prune=550 dbcache=4000 | 726 | 670 | 92.29% | 1 | 1.08 | 1.08x | 726 | 670 | 92% |
| 2: prune=1100 dbcache=300 | 723 | 150 | 20.75% | 1 | 4.83 | 4.83x | 723 | 150 | 21% |
| 3: prune=1100 dbcache=4000 | 723 | 150 | 20.75% | 1 | 4.83 | 4.83x | 723 | 150 | 21% |
| 4: prune=2200 dbcache=300 | 358 | 58 | 16.2% | 2 | 12.47 | 6.23x | 359 | 76 | 21% |
| 5: prune=2200 dbcache=4000 | 358 | 58 | 16.2% | 2 | 12.47 | 6.23x | 358 | 58 | 16% |
| 6: prune=4400 dbcache=300 | 175 | 26 | 14.86% | 4 | 27.69 | 6.92x | 178 | 69 | 39% |
| 7: prune=4400 dbcache=4000 | 175 | 26 | 14.86% | 4 | 27.69 | 6.92x | 175 | 26 | 15% |
In the IBDs for configurations 0 & 1 (prune = 550MB; minimum prune target), we only prune and flush about 8% less in PR than in MB. With MB, every prune operation prunes 1 blk/rev file pairs. PR prune operations sometimes prune 2 blk/rev files, averaging at 1.08 files per prune. Increasing the dbcache doesn't affect the number of flushes. Likely because the max cache size is not reached.
Similarly, a larger dbcache doesn't affect configurations 2 & 3. However, PR only has 21% of the flushes and prune operations of MB. MB prunes 1 file pair per prune operation while PR prunes 4.83 file pairs on average.
We first see a larger dbcache affecting the number of flushes in configurations 4 & 5. Configuration 4 (c4) has a dbcache of 300MB and configuration 5 (c5) a dbcache of 4000MB. With c4, the number of flushes of PR compared to MB is only 21%. With c5, it's 16%. PR prunes on average 12.47 file pairs per prune operation, while MB prunes 2 pairs.
With configurations 6 & 7 PR prunes 27.7 file pairs on average while MB prunes 4 pairs. However, in configuration 6 (c6) the small dbcache (300MB) is a limiting factor. PR flushes 69 times with c6 and 26 times with configuration 7 (dbcache=4000MB) compared to 178 (c6) and 175 (c7) flushes with MB.
Fewer dbcache flushes due to fewer prune operations should yield IBD performance improvements.
| Configuration | MB run 1 | MB run 2 | MB run 3 | MB mean | PR run 1 | PR run 2 | PR run 3 | PR mean | improvement |
|---|---|---|---|---|---|---|---|---|---|
| 0: prune=550 dbcache=300 | 9799s | 10046s | 10003s | 9949s | 9478s | 9531s | 9518s | 9509s | 4.4% |
| 1: prune=550 dbcache=4000 | 10219s | 10047s | 9915s | 10060s | 9649s | 9549s | 9712s | 9636s | 4.2% |
| 2: prune=1100 dbcache=300 | 9970s | 10191s | 10073s | 10078s | 8249s | 8467s | 8310s | 8342s | 17.2% |
| 3: prune=1100 dbcache=4000 | 10084s | 9887s | 9976s | 9982s | 8652s | 8483s | 8329s | 8488s | 15.0% |
| 4: prune=2200 dbcache=300 | 8991s | 9071s | 8817s | 8959s | 7494s | 7687s | 7496s | 7559s | 15.6% |
| 5: prune=2200 dbcache=4000 | 8973s | 8865s | 9063s | 8967s | 7405s | 7169s | 7661s | 7411s | 17.4% |
| 6: prune=4400 dbcache=300 | 8378s | 8102s | 7972s | 8150s | 7645s | 7533s | 7515s | 7564s | 7.2% |
| 7: prune=4400 dbcache=4000 | 8144s | 8263s | 8002s | 8136s | 6833s | 6982s | 6792s | 6869s | 15.6% |
The performance improves for all configurations.
In configurations 0 (c0) and 1 (c1), the increased dbcache had no effect. We saw 8% fewer prunes and flushes with PR than MB for both configurations. We measured a 4.4% (c0) and a 4.2% (c1) improvement in IBD performance with PR.
In configurations 2 (c2) and 3 (c3), PR had only 21% of the flush and prune operations than MB had (see above). We measured a performance improvement of 17.2% for c2 and a 15% for c3. The difference of 2.2% stems likely from PR run 1 of c3, which took about 400s (~5%) longer than PR run 1 of c2.
The smaller dbcache of configuration 4 (c4) was noticeable in the number of flushes and prunes already. The PR showed a 15.6% improvement over MB in c4 and a 17.4% improvement in c5.
With configurations 6 (c6) and 7 (c7), the effect of the smaller dbcache of c6 became more noticeable. We saw a 7.2% improvement for PR with c6 compared to a 15.6% improvement with c7.
PR with a larger prune target does reduce the number of flushes significantly compared to MB by deleting more blk/rev file pairs per prune operation. However, the default dbcache (of 300MB) can fill up with larger prune targets. Here, increasing the dbcache is helpful. Given the right configuration, fewer flushes significantly impact the IBD performance between block 500000 and 600000. Often, a performance improvement of over 15% can be measured.
This PR is worth pursuing.
I'm running full (genesis to 710k) IBD benchmarks now.
utACK https://github.com/bitcoin/bitcoin/pull/20827/commits/24f3936337de3afb4fa56efc83009e2527d22df0
Review by @0xB10C is helpful although I was confused at some places if MB is for Master Branch or Mega Byte.
I ran full IBD benchmarks (block 0 to 710k compared to 500k to 600k in #20827 (comment)) with these three configurations:
1. -dbcache=300 -prune=550
2. -dbcache=4000 -prune=4400
3. -dbcache=4000 -prune=8800
I ran each configuration two times (compared to three times in #20827 (comment)) for both the PR binary and the MB binary. That's 3 configurations * 2 binaries * 2 runs = 12 full IBDs. The setup and hardware were the same as in the 500k-to-600k benchmark. However, I've started with an empty datadir compared to a datadir pre-synced to block 500k in the previous benchmark.
tl;dr: I measured a 3.9% full-IBD speed-up with a default dbcache and the minimum pruning target and a 17.4% full-IBD speed-up with a dbcache of 4000MB and a prune target of 8800MB.
| config | MB prunes | PR prunes | prunes | MB pruned files | PR pruned files | pruned files | MB flushes | PR flushes | flushes |
|---|---|---|---|---|---|---|---|---|---|
| 0: prune=550 dbcache=300 | 2805 | 2403 | 85.67% | 1 | 1.17 | 1.17x | 2805 | 2403 | 86% |
| 1: prune=4400 dbcache=4000 | 695 | 101 | 14.53% | 4 | 27.75 | 6.94x | 695 | 101 | 15% |
| 2: prune=8800 dbcache=4000 | 393 | 48 | 12.21% | 6.99 | 58.17 | 8.32x | 393 | 48 | 12% |
Across the board, it's visible that pruning more files in a single pruning operation causes fewer pruning operations in total. This causes fewer dbcache flushes in total. The PR binary in configuration 0 pruned 1.17 blk/rev file pair per prune operation compared 1 file pair in the MB. This caused the PR binary to flush 24% fewer times. With higher prune targets, we flush even less often. In configuration 1 we flushed 85% fewer times with PR than with MB and in configuration 2 we flushed 88% fewer times. In configuration 2 we flushed 58 blk/rev file pairs on average per flush operation compared to only 7 file pairs in MB.
| Configuration | MB run 1 | MB run 2 | MB mean | PR run 1 | PR run 2 | PR mean | improvement |
|---|---|---|---|---|---|---|---|
| 0: prune=550 dbcache=300 | 40621s | 39884s | 40252s | 38127s | 39208s | 38667s | 3.9% |
| 1: prune=4400 dbcache=4000 | 33346s | 33395s | 33370s | 27597s | 28715s | 28156s | 15.6% |
| 2: prune=8800 dbcache=4000 | 30734s | 32040s | 31387s | 25656s | 26214s | 25935s | 17.4% |
With configuration 0, I measured a 3.9% IBD speed-up with the PR binary. The MB binary averaged at 11:10:52 and the PR binary at 10:44:27 for a full IBD.
With configuration 1 I measured a 15.6% IBD speed-up with the PR binary. The MB binary averaged at 09:16:10 and the PR binary at 07:49:16 for a full IBD.
With configuration 2, I measured a 17.4% IBD speed-up with the PR binary. The MB binary averaged at 08:43:07 and the PR binary at 07:12:15 for a full IBD.
I've looked at the dbcache size right before it's flushed to discuss the chosen configurations. This gives a good indication of the dbcache utilization.
For the three configurations, the blue box plot (for the MB binary) on the left and the orange box plot (for the PR binary) show the distributions of the dbcache size right before flushing it to disk.
For configuration 0, with a default dbcache of 300MB, both MB and PR utilize only about 100MB and 125MB of the dbcache in the median. PR has some outliners just below 500MB, which are larger than the configured dbcache. This is likely the reserved mempool memory being used as dbcache during IBD (where the mempool is empty).
With configuration 1 with a 4000MB dbcache, the median utilization is about 275MB for the MB binary and about 1050MB for PR. PR has outliers up to 2100MB, but limiting it to 1600MB compared to 4000MB would probably be fine too.
With configuration 2 with a 4000MB dbcache, the median utilization is about 450MB for the MB binary and about 1600MB for PR. PR has outliers up to 2700MB, but limiting it to 2600MB compared to 4000MB would probably be fine too.
None of the configurations are limited by dbcache size.
The 4c/8t i7 CPU of the benchmarking machine was under about 25% load without signature verification and 65% load with signature verification (assumevalid). The limiting factor was probably disk IO on the HDD raid.
Again, benchmarks on a system with SDDs would be helpful too. As HDD space is usually cheaper than SSD space, maybe more people enable pruning on systems with an SSD. Additionally, systems with weaker processors might see smaller performance improvements as they are also limited by their CPU and not only by disk IO.
Rebased
Are you still working on this?
Rebased
Concept ACK
Concept ACK
I have a project that syncs bitcoin core on an external drive and this would really help a lot of people testing it with smaller flash drives and higher RAM. So much so that I've made scripts to do manual pruning (back to 550mib) only when disk space is getting low to emulate this.
Are you still working on this?
297 | @@ -298,6 +298,7 @@ void BlockManager::FindFilesToPrune( 298 | // Distribute our -prune budget over all chainstates. 299 | const auto target = std::max( 300 | MIN_DISK_SPACE_FOR_BLOCK_FILES, GetPruneTarget() / chainman.GetAll().size()); 301 | + const uint64_t target_sync_height = chainman.m_best_header->nHeight;
nit: Any reason this declaration is up here and not directly above where it is used below?
320 | @@ -320,10 +321,13 @@ void BlockManager::FindFilesToPrune( 321 | // On a prune event, the chainstate DB is flushed. 322 | // To avoid excessive prune events negating the benefit of high dbcache 323 | // values, we should not prune too rapidly. 324 | - // So when pruning in IBD, increase the buffer a bit to avoid a re-prune too soon. 325 | - if (chainman.IsInitialBlockDownload()) { 326 | - // Since this is only relevant during IBD, we use a fixed 10% 327 | - nBuffer += target / 10; 328 | + // So when pruning in IBD, increase the buffer to avoid a re-prune too soon. 329 | + const auto chain_tip_height = chain.m_chain.Height();
nit: could remove the cast in the next line with
const uint64_t chain_tip_height = chain.m_chain.Height();
ACK d298ff8b62b2624ed390c8a2f905c888ffc956ff
Ran benchmark with hyperfine:
hyperfine --show-output --parameter-list commit 96ec3b67a7a7f968d002e13d6fc227f69b7f07d7,d298ff8b62b2624ed390c8a2f905c888ffc956ff --setup 'git checkout {commit} && make -j$(nproc) src/bitcoind' --prepare 'sync; sudo /sbin/sysctl vm.drop_caches=3; rm -r /home/user/.bitcoin/blocks /home/user/.bitcoin/chainstate' -M 3 './src/bitcoind -dbcache=2000 -prune=2000 -printtoconsole=0 -stopatheight=800000'
Results were 21% faster on this branch :rocket:
Summary
./src/bitcoind -dbcache=2000 -prune=2000 -printtoconsole=0 -stopatheight=800000 (commit = d298ff8b62b2624ed390c8a2f905c888ffc956ff) ran
1.21 ± 0.00 times faster than ./src/bitcoind -dbcache=2000 -prune=2000 -printtoconsole=0 -stopatheight=800000 (commit = 96ec3b67a7a7f968d002e13d6fc227f69b7f07d7)
utACK d298ff8b62b2624ed390c8a2f905c888ffc956ff
While there may be slightly better configurations possible, as @0xB10C pointed out above, I think the 1MB assumption is alright and this is a clear improvement already, so this can be merged and potentially improved later.
ACK d298ff8b62b2624ed390c8a2f905c888ffc956ff
15% improvement in IBD performance in some configurations seems significant enough for a release note, no?