This change is part of [IBD] - Tracking PR for speeding up Initial Block Download
Summary
When the in-memory UTXO set is flushed to LevelDB (after IBD or AssumeUTXO load), it does so in batches to manage memory usage during the flush.
A hidden -dbbatchsize
config option exists to modify this value. This PR only changes the default from 16
MiB to 32
MiB.
Using a larger default reduces the overhead of many small writes and improves I/O efficiency (especially on HDDs). It may also help LevelDB optimize writes more effectively (e.g., via internal ordering).
The change is meant to speed up a critical part of IBD: dumping the accumulated work to disk.
Context
The UTXO set has grown significantly since 2017, when the original fixed 16 MiB batch size was chosen.
With the current multi-gigabyte UTXO set and the common practice of using larger -dbcache
values, the fixed 16 MiB batch size leads to several inefficiencies:
- Flushing the entire UTXO set often requires thousands of separate 16 MiB write operations.
- Particularly on HDDs, the cumulative disk seek time and per-operation overhead from numerous small writes significantly slow down the flushing process.
- Each
WriteBatch
call incurs internal LevelDB overhead (e.g., MemTable handling, compaction triggering logic). More frequent, smaller batches amplify this cumulative overhead.
Flush times of 20-30 minutes are not uncommon, even on capable hardware.
Considerations
As noted by sipa, flushing involves a temporary memory usage increase as the batch is prepared. A larger batch size naturally leads to a larger peak during this phase. Crashing due to OOM during a flush is highly undesirable, but now that #30611 is merged, the most we’d lose is the first hour of IBD.
Increasing the LevelDB write batch size from 16 to 32 MiB raised the measured peaks by ~70 MiB in my tests during UTXO dump. The option remains hidden, and users can always override it.
The increased peak memory usage (detailed below) is primarily attributed to LevelDB’s leveldb::Arena
(backing MemTables) and the temporary storage of serialized batch data (e.g., std::string
in CDBBatch::WriteImpl
).
Performance gains are most pronounced on systems with slower I/O (HDDs), but some SSDs also show measurable improvements.
Measurements:
AssumeUTXO proxy, multiple runs with error bars (flushing time is faster that the measured loading + flushing):
- Raspberry Pi, dbcache=500: ~30% faster with 32 MiB vs 16 MiB, peak +~75 MiB and still < 1 GiB.
- i7 + HDD: results vary by dbcache, but 32 MiB usually beats 16 MiB and tracks close to 64 MiB without the larger peak.
- i9 + fast NVMe: roughly flat across 16/32/64 MiB. The goal here is to avoid regressions, which holds.
Reproducer:
0# Set up a clean demo environment
1rm -rfd demo && mkdir -p demo
2
3# Build Bitcoin Core
4cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j$(nproc)
5
6# Start bitcoind with minimal settings without mempool and internet connection
7build/bin/bitcoind -datadir=demo -stopatheight=1
8build/bin/bitcoind -datadir=demo -blocksonly=1 -connect=0 -dbcache=3000 -daemon
9
10# Load the AssumeUTXO snapshot, making sure the path is correct
11# Expected output includes `"coins_loaded": 184821030`
12build/bin/bitcoin-cli -datadir=demo -rpcclienttimeout=0 loadtxoutset ~/utxo-880000.dat
13
14# Stop the daemon and verify snapshot flushes in the logs
15build/bin/bitcoin-cli -datadir=demo stop
16grep "FlushSnapshotToDisk: completed" demo/debug.log
This PR originally proposed 64 MiB, then a dynamic size, but both were dropped: 64 MiB increased peaks more than desired on low-RAM systems, and the dynamic variant underperformed across mixed hardware. 32 MiB is a simpler default that captures most of the gains with a modest peak increase.
For more details see: #31645 (comment)
While the PR isn’t about IBD in general, rather about a critical section of it, I have measured a reindex-chainstate until 900k blocks, showing a 1% overall speedup:
0COMMITS="e6bfd95d5012fa1d91f83bf4122cb292afd6277f af653f321b135a59e38794b537737ed2f4a0040b"; \
1STOP=900000; DBCACHE=10000; \
2CC=gcc; CXX=g++; \
3BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
4(echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done; echo "") && \
5hyperfine \
6 --sort command \
7 --runs 1 \
8 --export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \
9 --parameter-list COMMIT ${COMMITS// /,} \
10 --prepare "killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
11 cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release && ninja -C build bitcoind && \
12 ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 10" \
13 --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
14 "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0"
15
16e6bfd95d50 Merge bitcoin-core/gui#881: Move `FreespaceChecker` class into its own module
17af653f321b coins: derive `batch_write_bytes` from `-dbcache` when unspecified
18
19Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=900000 -dbcache=10000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = e6bfd95d5012fa1d91f83bf4122cb292afd6277f)
20 Time (abs ≡): 25016.346 s [User: 30333.911 s, System: 826.463 s]
21
22Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=900000 -dbcache=10000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = af653f321b135a59e38794b537737ed2f4a0040b)
23 Time (abs ≡): 24801.283 s [User: 30328.665 s, System: 834.110 s]
24
25Relative speed comparison
26 1.01 COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=900000 -dbcache=10000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = e6bfd95d5012fa1d91f83bf4122cb292afd6277f)
27 1.00 COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=900000 -dbcache=10000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = af653f321b135a59e38794b537737ed2f4a0040b)