[IBD] prevector: store `P2WSH`/`P2TR`/`P2PK` scripts inline

l0rinc commented at 4:20 pm on April 15, 2025: contributor

This change is part of [IBD] - Tracking PR for speeding up Initial Block Download

Summary

The current prevector size of 28 bytes (chosen to fill the sizeof(CScript) aligned size) was introduced in 2015 (https://github.com/bitcoin/bitcoin/pull/6914) before SegWit and TapRoot. However, the increasingly common P2WSH and P2TR scripts are both 34 bytes, and are forced to use heap (re)allocation rather than efficient inline storage.

The core trade-off of this change is to eliminate heap allocations for common 34-36 byte scripts at the cost of increasing the base memory footprint of all CScript objects by 8 bytes (while still respecting peak memory usage defined by -dbcache).

Context

Increasing the prevector size allows these scripts to be stored inline, avoiding heap allocations, reducing potential memory fragmentation, and improving performance during cache flushes. Massif analysis confirms a lower stable memory usage after flushing, suggesting the elimination of heap allocations outweighs the larger base size for common workloads.

Due to memory alignment, increasing the prevector size to 36 bytes doesn’t change the overall sizeof(CScript) compared to an increase to 34 bytes, allowing us to include P2PK scripts as well at no additional memory cost.

dbcache=440

Massif before, with a heap threshold of 28:

 0    MB
 1744.1^#                                                                       
 2     |#: ::::::@: :::::::   :@:: @::::::::::::::@@                            
 3     |#: ::::::@::::: :::   :@:::@:::::: :: ::::@                             
 4     |#: ::::::@::::: :::   :@:::@:::::: :: ::::@                             
 5     |#: ::::::@::::: ::: : :@:::@:::::: :: ::::@                             
 6     |#: ::::::@::::: ::: : :@:::@:::::: :: ::::@                             
 7     |#: ::::::@::::: ::: : :@:::@:::::: :: ::::@                             
 8     |#::::::::@::::: ::: : :@:::@:::::: :: ::::@                             
 9     |#::::::::@::::: ::: :::@:::@:::::: :: ::::@                             
10     |#::::::::@::::: ::: :::@:::@:::::: :: ::::@                             
11     |#::::::::@::::: ::: :::@:::@:::::: :: ::::@                             
12     |#::::::::@::::: ::: :::@:::@:::::: :: ::::@                             
13     |#::::::::@::::: :::::::@:::@:::::: :: ::::@                             
14     |#::::::::@::::: :::::::@:::@:::::: :: ::::@ :::::@:::::@:::::@:::::@::::
15     |#::::::::@::::: :::::::@:::@:::::: :: ::::@ :::::@:::::@:::::@:::::@::::
16     |#::::::::@::::: :::::::@:::@:::::: :: ::::@ :::::@:::::@:::::@:::::@::::
17     |#::::::::@::::: :::::::@:::@:::::: :: ::::@ :::::@:::::@:::::@:::::@::::
18     |#::::::::@::::: :::::::@:::@:::::: :: ::::@ :::::@:::::@:::::@:::::@::::
19     |#::::::::@::::: :::::::@:::@:::::: :: ::::@ :::::@:::::@:::::@:::::@::::
20     |#::::::::@::::: :::::::@:::@:::::: :: ::::@ :::::@:::::@:::::@:::::@::::
21   0 +----------------------------------------------------------------------->h
22     0                                                                   1.805

and after, with a heap threshold of 36:

 0    MB
 1744.2^       :                                                                
 2     |#  :  :::::::::::   : : :: ::: @@:::::: ::  :                           
 3     |#  :  :::: ::::::   : : :: ::: @ :: ::  :   :                           
 4     |#  :  :::: :::::::  : :@:: ::: @ :: ::  : :::                           
 5     |#  :  :::: :::::::  : :@:: ::: @ :: ::  : : :                           
 6     |#  :  :::: :::::::  : :@:: ::: @ :: ::  : : :                           
 7     |#  :  :::: :::::::  : :@:: ::: @ :: ::  : : :                           
 8     |#  :: :::: :::::::  : :@:: ::: @ :: ::  : : :                           
 9     |#  :: :::: :::::::  : :@:: ::::@ :: ::  : : :                           
10     |#:::: :::: :::::::  :::@:: ::::@ :: ::  : : :                           
11     |#: ::::::: :::::::  :::@:: ::::@ :: :: @: : :                           
12     |#: ::::::: :::::::  :::@:::::::@ :: :: @: : :                           
13     |#: ::::::: ::::::::::::@:::::::@ :: :: @: : :                           
14     |#: ::::::: :::::::: :::@:::::::@ :: :: @: : :                           
15     |#: ::::::: :::::::: :::@:::::::@ :: :: @: : :::@:::@::::@::::::@:::::@::
16     |#: ::::::: :::::::: :::@:::::::@ :: :: @: : :::@:::@::::@::::::@:::::@::
17     |#: ::::::: :::::::: :::@:::::::@ :: :: @: : :::@:::@::::@::::::@:::::@::
18     |#: ::::::: :::::::: :::@:::::::@ :: :: @: : :::@:::@::::@::::::@:::::@::
19     |#: ::::::: :::::::: :::@:::::::@ :: :: @: : :::@:::@::::@::::::@:::::@::
20     |#: ::::::: :::::::: :::@:::::::@ :: :: @: : :::@:::@::::@::::::@:::::@::
21   0 +----------------------------------------------------------------------->h
22     0                                                                   1.618

for dbcache=4500:

Massif before, with a heap threshold of 28:

 0    GB
 14.565^   ::                                                                   
 2     | ##:   @@:::  :::: :@::::  :::: ::::                                    
 3     | # :   @ ::   :::  :@: ::  : :: :::                                     
 4     | # :   @ :: :::::  :@: ::  : :: :::                                     
 5     | # :   @ :: : :::  :@: :: @: :: :::                                     
 6     | # :   @ :: : :::  :@: :: @: :: :::                                     
 7     | # :   @ :: : :::  :@: :: @: :: :::                                     
 8     | # :   @ :: : :::  :@: :: @: :: :::                                     
 9     | # : ::@ :: : :::  :@: :: @: :: :::                                     
10     | # : : @ :: : :::  :@: :: @: :: :::                                     
11     | # : : @ :: : :::  :@: :: @: ::::::                                     
12     | # : : @ :: : :::  :@: :: @: ::::::                                     
13     | # : : @ :: : :::  :@: :: @: ::::::                                     
14     | # : : @ :: : ::: ::@: :: @: ::::::                                     
15     | # : : @ :: : ::: ::@: :: @: ::::::                                     
16     | # : : @ :: : ::: ::@: :: @: ::::::                                     
17     | # : : @ :: : ::: ::@: :: @: :::::: @::                                 
18     | # : : @ :: : ::: ::@: :: @: :::::: @:                                  
19     | # : : @ :: : ::: ::@: :::@: :::::: @:                                  
20     | # : : @ :: : ::: ::@: :::@: :::::: @: :::::::::::::::::::::::::::::@:::
21   0 +----------------------------------------------------------------------->h
22     0                                                                   1.500

and after, with a heap threshold of 36:

 0    GB
 14.640^    :                                                                   
 2     | ##::  :::::   ::::  ::::::@  ::::                                      
 3     | # ::  : :::   ::::  :: :::@  ::::                                      
 4     | # :: :: :::   ::::  :: :::@  ::::                                      
 5     | # :: :: :::  :::::  :: :::@  ::::                                      
 6     | # :: :: :::  :::::  :: :::@  ::::                                      
 7     | # :: :: :::  :::::  :: :::@  ::::                                      
 8     | # :: :: :::  :::::  :: :::@  ::::                                      
 9     | # :: :: :::  :::::  :: :::@  ::::  :@@                                 
10     | # :: :: :::  ::::: ::: :::@  :::::::@                                  
11     | # :: :: :::  ::::: ::: :::@  ::::: :@                                  
12     | # :: :: :::  ::::: ::: :::@::::::: :@                                  
13     | # ::::: :::  ::::: ::: :::@: ::::: :@                                  
14     | # ::::: :::  ::::: ::: :::@: ::::: :@                                  
15     | # ::::: :::::::::: ::: :::@: ::::: :@                                  
16     | # ::::: :::: ::::: ::: :::@: ::::: :@                                  
17     | # ::::: :::: ::::: ::: :::@: ::::: :@                                  
18     | # ::::: :::: ::::::::: :::@: ::::: :@                                  
19     | # ::::: :::: ::::::::: :::@: ::::: :@                                  
20     | # ::::: :::: ::::::::: :::@: ::::: :@ ::::::@:::@:::@::::@:::::@::::@::
21   0 +----------------------------------------------------------------------->h
22     0                                                                   1.360

Benchmarks and Memory

Performance benchmarks for AssumeUTXO load and flush show:

Small dbcache (450MB): ~1-3% performance improvement (despite more frequent flushes)
Large dbcache (4500MB): ~6-8% performance improvement due to fewer heap allocations (and basically the number of flushes)
Very large dbcache (4500MB): ~5-6% performance improvement due to fewer heap allocations (and memory limit not being reached, so there’s no memory penalty)

Full IBD and -reindex-chainstate with also show an overall ~3-4% speedup (both for smaller and larger dbcache values).

We haven’t investigated using different prevector sizes based on script type, though this could be explored in the future if needed.

Historical explanation for the speedup (by Anthony Towns)

I think the tradeoff is something like:

spends of p2pk, p2sh, p2pkh coins – these cost 8 more bytes

spends of p2wpkh – these cost 16 more bytes (sPK and scriptSig didn’t need an allocation)

spends of p2wsh and p2tr – these cost ~48 fewer bytes (save 64 byte allocation on 64bit system, lose 8 bytes for both scriptSig and sPK)

spends of nested p2wsh – presumably save ~96 bytes, since the scriptSig would save an allocation, but I’m bundling it in the previous section

Based on mainnet.observer stats for 2025-05-08, p2wpkh is about 55% of txs, p2tr is about 28%, p2pkh about 13%, p2wsh about 4% and the rest is noise, maybe? Those numbers net out to a saving of ~5.5 bytes per input. If p2wpkh rose from 55% to 80% and p2tr dropped to 20%, that would net to wasting ~3.2 bytes per input.

DrahtBot commented at 4:20 pm on April 15, 2025: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/32279.

Reviews

See the guideline for information on the review process.

Type	Reviewers
Approach ACK	ajtowns
Stale ACK	maflcko, achow101

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#33021 (test: revive test verifying that GetCoinsCacheSizeState switches from OK→LARGE→CRITICAL by l0rinc)
#28531 (improve MallocUsage() accuracy by LarryRuane)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

l0rinc renamed this:
~~prevector: allocate `P2WSH` and `P2TR` scripts on stack as well~~
[IBD] prevector: allocate `P2WSH` and `P2TR` scripts on stack as well
on Apr 15, 2025

l0rinc commented at 5:08 pm on April 15, 2025: contributor

Additional info:

Added logging to the original prevector destructor to measure its final size - and printed the values during an assumeutxo load & dump:

I haven’t investigated in detail why there are so many empty prevectors, can very well be just an unoptimized debug run artifact of all the Coin copies.

If we plot the coin cache size vs height, we see that with the original prevector size, the first part of the sync fits more values initially and fewer ones later:

After this change, the occupancy is more predictable:

AssumeUTXO

Ran some AssumeUTXO benchmarks with different values (on SSD):

 0COMMITS="847a891cdfdf9cfd69a77a8688c995e532ea29db dc0c79a46b5663eddc0a4365b9b7de22cf3fefae"; \
 1CC=gcc; CXX=g++; \
 2BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/ShallowBitcoinData"; LOG_DIR="$BASE_DIR/logs"; UTXO_SNAPSHOT_PATH="$BASE_DIR/utxo-880000.dat"; \
 3(for c in $COMMITS; do git fetch origin $c -q && git log -1 --pretty=format:'%h %s' $c || exit 1; done) && \
 4for DBCACHE in 450 4500 45000; do \
 5  hyperfine \
 6  --sort 'command' \
 7  --runs 5 \
 8  --export-json "$BASE_DIR/assumeutxo-${COMMITS// /-}-$DBCACHE-$CC.json" \
 9  --parameter-list COMMIT ${COMMITS// /,} \
10  --prepare "killall bitcoind; rm -rf $DATA_DIR/chainstate $DATA_DIR/chainstate_snapshot $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard; \
11    cmake -B build -DCMAKE_BUILD_TYPE=Release -DENABLE_WALLET=OFF && cmake --build build -j$(nproc) --target bitcoind --target bitcoin-cli && \
12    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=1 -printtoconsole=0; \
13    ./build/bin/bitcoind -datadir=$DATA_DIR -daemon -blocksonly=1 -connect=0 -dbcache=$DBCACHE -printtoconsole=0; sleep 10" \
14  --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-assumeutxo-{COMMIT}-dbcache-$DBCACHE-$(date +%s).log; build/bin/bitcoin-cli -datadir=$DATA_DIR stop; sleep 10; killall bitcoind; sleep 10;" \
15  "COMPILER=$CC DBCACHE=$DBCACHE COMMIT={COMMIT} ./build/bin/bitcoin-cli -datadir=$DATA_DIR -rpcclienttimeout=0 loadtxoutset $UTXO_SNAPSHOT_PATH"; \
16done

847a891cdf test: assert CScript allocation characteristics dc0c79a46b store P2WSH/P2TR/P2PK scripts inline

 0Benchmark 1: COMPILER=gcc DBCACHE=450 COMMIT=847a891cdfdf9cfd69a77a8688c995e532ea29db ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
 1  Time (mean ± σ):     451.840 s ±  2.255 s    [User: 0.001 s, System: 0.001 s]
 2  Range (min … max):   448.790 s … 454.846 s    5 runs
 3
 4Benchmark 2: COMPILER=gcc DBCACHE=450 COMMIT=dc0c79a46b5663eddc0a4365b9b7de22cf3fefae ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
 5  Time (mean ± σ):     451.480 s ±  5.381 s    [User: 0.002 s, System: 0.001 s]
 6  Range (min … max):   447.360 s … 459.782 s    5 runs
 7
 8Relative speed comparison
 9        1.00 ±  0.01  COMPILER=gcc DBCACHE=450 COMMIT=847a891cdfdf9cfd69a77a8688c995e532ea29db ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
10        1.00          COMPILER=gcc DBCACHE=450 COMMIT=dc0c79a46b5663eddc0a4365b9b7de22cf3fefae ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat

847a891cdf test: assert CScript allocation characteristics dc0c79a46b store P2WSH/P2TR/P2PK scripts inline

 0Benchmark 1: COMPILER=gcc DBCACHE=4500 COMMIT=847a891cdfdf9cfd69a77a8688c995e532ea29db ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
 1  Time (mean ± σ):     480.316 s ±  2.640 s    [User: 0.001 s, System: 0.002 s]
 2  Range (min … max):   477.157 s … 483.803 s    5 runs
 3
 4Benchmark 2: COMPILER=gcc DBCACHE=4500 COMMIT=dc0c79a46b5663eddc0a4365b9b7de22cf3fefae ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
 5  Time (mean ± σ):     452.521 s ±  7.533 s    [User: 0.001 s, System: 0.001 s]
 6  Range (min … max):   445.367 s … 460.740 s    5 runs
 7
 8Relative speed comparison
 9        1.06 ±  0.02  COMPILER=gcc DBCACHE=4500 COMMIT=847a891cdfdf9cfd69a77a8688c995e532ea29db ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
10        1.00          COMPILER=gcc DBCACHE=4500 COMMIT=dc0c79a46b5663eddc0a4365b9b7de22cf3fefae ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat

847a891cdf test: assert CScript allocation characteristics dc0c79a46b store P2WSH/P2TR/P2PK scripts inline

 0Benchmark 1: COMPILER=gcc DBCACHE=45000 COMMIT=847a891cdfdf9cfd69a77a8688c995e532ea29db ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
 1  Time (mean ± σ):     469.846 s ±  1.583 s    [User: 0.002 s, System: 0.001 s]
 2  Range (min … max):   467.909 s … 472.053 s    5 runs
 3
 4Benchmark 2: COMPILER=gcc DBCACHE=45000 COMMIT=dc0c79a46b5663eddc0a4365b9b7de22cf3fefae ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
 5  Time (mean ± σ):     448.079 s ±  1.180 s    [User: 0.002 s, System: 0.001 s]
 6  Range (min … max):   447.280 s … 450.165 s    5 runs
 7
 8Relative speed comparison
 9        1.05 ±  0.00  COMPILER=gcc DBCACHE=45000 COMMIT=847a891cdfdf9cfd69a77a8688c995e532ea29db ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
10        1.00          COMPILER=gcc DBCACHE=45000 COMMIT=dc0c79a46b5663eddc0a4365b9b7de22cf3fefae ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat

And on HDD:

 0COMMITS="156f9913a26598009c8356c34548002e6a6aba02 a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7"; \
 1CC=gcc; CXX=g++; \
 2BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/ShallowBitcoinData"; LOG_DIR="$BASE_DIR/logs"; UTXO_SNAPSHOT_PATH="$BASE_DIR/utxo-880000.dat"; \
 3(for c in $COMMITS; do git fetch origin $c -q && git log -1 --pretty=format:'%h %s' $c || exit 1; done) && \
 4for DBCACHE in 450 4500 45000; do \
 5  hyperfine \
 6  --sort 'command' \
 7  --runs 5 \
 8  --export-json "$BASE_DIR/assumeutxo-${COMMITS// /-}-$DBCACHE-$CC.json" \
 9  --parameter-list COMMIT ${COMMITS// /,} \
10  --prepare "killall bitcoind; rm -rf $DATA_DIR/chainstate $DATA_DIR/chainstate_snapshot $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard; \
11    cmake -B build -DCMAKE_BUILD_TYPE=Release -DENABLE_WALLET=OFF && cmake --build build -j$(nproc) --target bitcoind --target bitcoin-cli && \
12    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=1 -printtoconsole=0; \
13    ./build/bin/bitcoind -datadir=$DATA_DIR -daemon -blocksonly=1 -connect=0 -dbcache=$DBCACHE -printtoconsole=0; sleep 10" \
14  --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-assumeutxo-{COMMIT}-dbcache-$DBCACHE-$(date +%s).log; build/bin/bitcoin-cli -datadir=$DATA_DIR stop; sleep 10; killall bitcoind; sleep 10;" \
15  "COMPILER=$CC DBCACHE=$DBCACHE COMMIT={COMMIT} ./build/bin/bitcoin-cli -datadir=$DATA_DIR -rpcclienttimeout=0 loadtxoutset $UTXO_SNAPSHOT_PATH"; \
16done

156f9913a2 test: assert CScript allocation characteristics a2a3e0cfd8 store P2WSH/P2TR/P2PK scripts inline

 0Benchmark 1: COMPILER=gcc DBCACHE=450 COMMIT=156f9913a26598009c8356c34548002e6a6aba02 ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
 1  Time (mean ± σ):     949.219 s ± 24.273 s    [User: 0.002 s, System: 0.001 s]
 2  Range (min … max):   919.208 s … 969.841 s    5 runs
 3
 4Benchmark 2: COMPILER=gcc DBCACHE=450 COMMIT=a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7 ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
 5  Time (mean ± σ):     923.687 s ± 15.431 s    [User: 0.001 s, System: 0.001 s]
 6  Range (min … max):   909.239 s … 948.855 s    5 runs
 7
 8Relative speed comparison
 9        1.03 ±  0.03  COMPILER=gcc DBCACHE=450 COMMIT=156f9913a26598009c8356c34548002e6a6aba02 ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
10        1.00          COMPILER=gcc DBCACHE=450 COMMIT=a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7 ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat

156f9913a2 test: assert CScript allocation characteristics a2a3e0cfd8 store P2WSH/P2TR/P2PK scripts inline

 0Benchmark 1: COMPILER=gcc DBCACHE=4500 COMMIT=156f9913a26598009c8356c34548002e6a6aba02 ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
 1  Time (mean ± σ):     778.101 s ± 14.503 s    [User: 0.002 s, System: 0.001 s]
 2  Range (min … max):   765.670 s … 801.459 s    5 runs
 3
 4Benchmark 2: COMPILER=gcc DBCACHE=4500 COMMIT=a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7 ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
 5  Time (mean ± σ):     718.500 s ± 17.941 s    [User: 0.001 s, System: 0.001 s]
 6  Range (min … max):   693.273 s … 735.929 s    5 runs
 7
 8Relative speed comparison
 9        1.08 ±  0.03  COMPILER=gcc DBCACHE=4500 COMMIT=156f9913a26598009c8356c34548002e6a6aba02 ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
10        1.00          COMPILER=gcc DBCACHE=4500 COMMIT=a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7 ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat

156f9913a2 test: assert CScript allocation characteristics a2a3e0cfd8 store P2WSH/P2TR/P2PK scripts inline

 0Benchmark 1: COMPILER=gcc DBCACHE=45000 COMMIT=156f9913a26598009c8356c34548002e6a6aba02 ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
 1  Time (mean ± σ):     567.985 s ±  9.220 s    [User: 0.001 s, System: 0.001 s]
 2  Range (min … max):   556.441 s … 578.439 s    5 runs
 3
 4Benchmark 2: COMPILER=gcc DBCACHE=45000 COMMIT=a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7 ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
 5  Time (mean ± σ):     535.541 s ± 11.944 s    [User: 0.001 s, System: 0.001 s]
 6  Range (min … max):   520.044 s … 549.775 s    5 runs
 7
 8Relative speed comparison
 9        1.06 ±  0.03  COMPILER=gcc DBCACHE=45000 COMMIT=156f9913a26598009c8356c34548002e6a6aba02 ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
10        1.00          COMPILER=gcc DBCACHE=45000 COMMIT=a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7 ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat

Reindex-chainstate

 0COMMITS="156f9913a26598009c8356c34548002e6a6aba02 a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7"; \
 1STOP_HEIGHT=888888; DBCACHE=450; \
 2CC=gcc; CXX=g++; \
 3BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
 4(for c in $COMMITS; do git fetch origin $c -q && git log -1 --pretty=format:'%h %s' $c || exit 1; done) && \
 5hyperfine \
 6  --sort 'command' \
 7  --runs 2 \
 8  --export-json "$BASE_DIR/rdx-${COMMITS// /-}-$STOP_HEIGHT-$DBCACHE-$CC.json" \
 9  --parameter-list COMMIT ${COMMITS// /,} \
10  --prepare "killall bitcoind; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard; \
11    cmake -B build -DCMAKE_BUILD_TYPE=Release -DENABLE_WALLET=OFF && cmake --build build -j$(nproc) --target bitcoind && \
12    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP_HEIGHT -dbcache=5000 -printtoconsole=0; sleep 100" \
13  --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
14  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP_HEIGHT -dbcache=$DBCACHE -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0"

156f9913a2 test: assert CScript allocation characteristics a2a3e0cfd8 store P2WSH/P2TR/P2PK scripts inline

 0Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=450 -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0 (COMMIT = 156f9913a26598009c8356c34548002e6a6aba02)
 1  Time (mean ± σ):     19748.586 s ±  3.502 s    [User: 35991.245 s, System: 2756.673 s]
 2  Range (min … max):   19746.110 s … 19751.062 s    2 runs
 3
 4Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=450 -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0 (COMMIT = a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7)
 5  Time (mean ± σ):     19333.189 s ±  6.061 s    [User: 35338.722 s, System: 2733.049 s]
 6  Range (min … max):   19328.903 s … 19337.474 s    2 runs
 7
 8Relative speed comparison
 9        1.02 ±  0.00  COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=450 -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0 (COMMIT = 156f9913a26598009c8356c34548002e6a6aba02)
10        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=450 -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0 (COMMIT = a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7)

Full IBD

 0COMMITS="156f9913a26598009c8356c34548002e6a6aba02 a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7"; \
 1STOP_HEIGHT=888888; DBCACHE=450; \
 2CC=gcc; CXX=g++; \
 3BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
 4(for c in $COMMITS; do git fetch origin $c -q && git log -1 --pretty=format:'%h %s' $c || exit 1; done) && \
 5hyperfine \
 6  --sort 'command' \
 7  --runs 2 \
 8  --export-json "$BASE_DIR/rdx-${COMMITS// /-}-$STOP_HEIGHT-$DBCACHE-$CC.json" \
 9  --parameter-list COMMIT ${COMMITS// /,} \
10  --prepare "killall bitcoind; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard; \
11    cmake -B build -DCMAKE_BUILD_TYPE=Release -DENABLE_WALLET=OFF && cmake --build build -j$(nproc) --target bitcoind && \
12    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP_HEIGHT -dbcache=5000 -printtoconsole=0; sleep 100" \
13  --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
14  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP_HEIGHT -dbcache=$DBCACHE -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0"

156f9913a2 test: assert CScript allocation characteristics a2a3e0cfd8 store P2WSH/P2TR/P2PK scripts inline

 0Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=450 -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0 (COMMIT = 156f9913a26598009c8356c34548002e6a6aba02)
 1  Time (mean ± σ):     39873.924 s ± 637.446 s    [User: 35537.079 s, System: 2668.529 s]
 2  Range (min … max):   39423.182 s … 40324.667 s    2 runs
 3
 4Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=450 -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0 (COMMI$ = a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7)
 5  Time (mean ± σ):     38682.594 s ±  7.327 s    [User: 34211.447 s, System: 2618.127 s]
 6  Range (min … max):   38677.412 s … 38687.775 s    2 runs
 7
 8Relative speed comparison
 9        1.03 ±  0.02  COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=450 -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0 (COMMIT = 156f9913a26598009c8356c34548002e6a6aba02)
10        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=450 -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0 (COMMIT = a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7)

 0COMMITS="847a891cdfdf9cfd69a77a8688c995e532ea29db dc0c79a46b5663eddc0a4365b9b7de22cf3fefae"; \
 1STOP_HEIGHT=888888; DBCACHE=45000; \
 2CC=gcc; CXX=g++; \
 3BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
 4(for c in $COMMITS; do git fetch origin $c -q && git log -1 --pretty=format:'%h %s' $c || exit 1; done) && \
 5hyperfine \
 6  --sort 'command' \
 7  --runs 2 \
 8  --export-json "$BASE_DIR/ibd-${COMMITS// /-}-$STOP_HEIGHT-$DBCACHE-$CC.json" \
 9  --parameter-list COMMIT ${COMMITS// /,} \
10  --prepare "killall bitcoind; rm -rf $DATA_DIR/*; git checkout {COMMIT}; git clean -fxd; git reset --hard; \
11    cmake -B build -DCMAKE_BUILD_TYPE=Release -DENABLE_WALLET=OFF && cmake --build build -j$(nproc) --target bitcoind && \
12    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=1 -printtoconsole=0; sleep 100" \
13  --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
14  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP_HEIGHT -dbcache=$DBCACHE -blocksonly -printtoconsole=0"

847a891cdf test: assert CScript allocation characteristics dc0c79a46b store P2WSH/P2TR/P2PK scripts inline

 0Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=45000 -blocksonly -printtoconsole=0 (COMMIT = 847a891cdfdf9cfd69a77a8688c995e532ea29db)
 1  Time (mean ± σ):     39061.975 s ± 366.257 s    [User: 37221.558 s, System: 1941.817 s]
 2  Range (min … max):   38802.992 s … 39320.958 s    2 runs
 3
 4Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=45000 -blocksonly -printtoconsole=0 (COMMIT = dc0c79a46b5663eddc0a4365b9b7de22cf3fefae)
 5  Time (mean ± σ):     38214.234 s ± 65.671 s    [User: 36851.441 s, System: 2103.990 s]
 6  Range (min … max):   38167.797 s … 38260.670 s    2 runs
 7
 8Relative speed comparison
 9        1.02 ±  0.01  COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=45000 -blocksonly -printtoconsole=0 (COMMIT = 847a891cdfdf9cfd69a77a8688c995e532ea29db)
10        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=45000 -blocksonly -printtoconsole=0 (COMMIT = dc0c79a46b5663eddc0a4365b9b7de22cf3fefae)

Memory

Note that this was measured for prevector<34>, but should be basically the same for the current 36 as well.

 0-------------------------------------------------------------------------------
 1  n       time(ms)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
 2--------------------------------------------------------------------------------
 3  0              0                0                0             0            0
 4  1        142,755    4,881,866,480    4,646,778,126   235,088,354            0
 595.18% (4,646,778,126B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 6->73.76% (3,600,809,984B) 0x5502B7: CCoinsViewCache::EmplaceCoinInternalDANGER(COutPoint&&, Coin&&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
 7| ->73.76% (3,600,809,984B) 0x49DDCF: ChainstateManager::PopulateAndValidateSnapshot(Chainstate&, AutoFile&, node::SnapshotMetadata const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
 8|   ->73.76% (3,600,809,984B) 0x49F3CA: ChainstateManager::ActivateSnapshot(AutoFile&, node::SnapshotMetadata const&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
 9|     ->73.76% (3,600,809,984B) 0x367A0F: loadtxoutset()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)
10|       ->73.76% (3,600,809,984B) 0x3680F6: std::_Function_handler<UniValue (RPCHelpMan const&, JSONRPCRequest const&), loadtxoutset()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)
11|         ->73.76% (3,600,809,984B) 0x60403B: RPCHelpMan::HandleRequest(JSONRPCRequest const&) const (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
12|           ->73.76% (3,600,809,984B) 0x373359: CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)
13|             ->73.76% (3,600,809,984B) 0x425F6C: ExecuteCommand(CRPCCommand const&, JSONRPCRequest const&, UniValue&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
14|               ->73.76% (3,600,809,984B) 0x427373: CRPCTable::execute(JSONRPCRequest const&) const (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
15|                 ->73.76% (3,600,809,984B) 0x4274C6: JSONRPCExec(JSONRPCRequest const&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
16|                   ->73.76% (3,600,809,984B) 0x4FF1C3: HTTPReq_JSONRPC(std::any const&, HTTPRequest*) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
17|                     ->73.76% (3,600,809,984B) 0x50CA81: WorkQueue<HTTPClosure>::Run() (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
18|                       ->73.76% (3,600,809,984B) 0x505F06: HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
19|                         ->73.76% (3,600,809,984B) 0x49A6DB3: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33)
20|                           ->73.76% (3,600,809,984B) 0x4CEBAA3: start_thread (pthread_create.c:447)
21|                             ->73.76% (3,600,809,984B) 0x4D78A33: clone (clone.S:100)
22|
23->08.19% (399,758,776B) 0x5522AC: std::__detail::_Hashtable_alloc<PoolAllocator<std::__detail::_Hash_node<std::pair<COutPoint const, CCoinsCacheEntry>, false>, 144ul, 8ul> >::_M_allocate_buckets(unsigned long) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
24| ->08.19% (399,758,776B) 0x5526AD: std::_Hashtable<COutPoint, std::pair<COutPoint const, CCoinsCacheEntry>, PoolAllocator<std::pair<COutPoint const, CCoinsCacheEntry>, 144ul, 8ul>, std::__detail::_Select1st, std::equal_to<COutPoint>, SaltedOutpointHasher, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_rehash(unsigned long, unsigned long const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
25|   ->08.19% (399,758,776B) 0x5528D5: std::_Hashtable<COutPoint, std::pair<COutPoint const, CCoinsCacheEntry>, PoolAllocator<std::pair<COutPoint const, CCoinsCacheEntry>, 144ul, 8ul>, std::__detail::_Select1st, std::equal_to<COutPoint>, SaltedOutpointHasher, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node<std::pair<COutPoint const, CCoinsCacheEntry>, false>*, unsigned long) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
26|     ->08.19% (399,758,776B) 0x550219: CCoinsViewCache::EmplaceCoinInternalDANGER(COutPoint&&, Coin&&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
27|       ->08.19% (399,758,776B) 0x49DDCF: ChainstateManager::PopulateAndValidateSnapshot(Chainstate&, AutoFile&, node::SnapshotMetadata const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
28|         ->08.19% (399,758,776B) 0x49F3CA: ChainstateManager::ActivateSnapshot(AutoFile&, node::SnapshotMetadata const&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
29|           ->08.19% (399,758,776B) 0x367A0F: loadtxoutset()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)
30|             ->08.19% (399,758,776B) 0x3680F6: std::_Function_handler<UniValue (RPCHelpMan const&, JSONRPCRequest const&), loadtxoutset()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)
31|               ->08.19% (399,758,776B) 0x60403B: RPCHelpMan::HandleRequest(JSONRPCRequest const&) const (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
32|                 ->08.19% (399,758,776B) 0x373359: CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)
33|                   ->08.19% (399,758,776B) 0x425F6C: ExecuteCommand(CRPCCommand const&, JSONRPCRequest const&, UniValue&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
34|                     ->08.19% (399,758,776B) 0x427373: CRPCTable::execute(JSONRPCRequest const&) const (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
35|                       ->08.19% (399,758,776B) 0x4274C6: JSONRPCExec(JSONRPCRequest const&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
36|                         ->08.19% (399,758,776B) 0x4FF1C3: HTTPReq_JSONRPC(std::any const&, HTTPRequest*) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
37|                           ->08.19% (399,758,776B) 0x50CA81: WorkQueue<HTTPClosure>::Run() (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
38|                             ->08.19% (399,758,776B) 0x505F06: HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
39|                               ->08.19% (399,758,776B) 0x49A6DB3: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33)
40|                                 ->08.19% (399,758,776B) 0x4CEBAA3: start_thread (pthread_create.c:447)
41|                                   ->08.19% (399,758,776B) 0x4D78A33: clone (clone.S:100)
42|
43->07.53% (367,723,723B) 0x2F508F: prevector<28u, unsigned char, unsigned int, int>::change_capacity(unsigned int) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
44| ->07.53% (367,723,579B) 0x4A5D3F: prevector<28u, unsigned char, unsigned int, int>::resize(unsigned int) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
45| | ->07.52% (367,274,326B) 0x49DF47: ChainstateManager::PopulateAndValidateSnapshot(Chainstate&, AutoFile&, node::SnapshotMetadata const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
46| | | ->07.52% (367,274,326B) 0x49F3CA: ChainstateManager::ActivateSnapshot(AutoFile&, node::SnapshotMetadata const&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
47| | |   ->07.52% (367,274,326B) 0x367A0F: loadtxoutset()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)
48| | |     ->07.52% (367,274,326B) 0x3680F6: std::_Function_handler<UniValue (RPCHelpMan const&, JSONRPCRequest const&), loadtxoutset()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)
49| | |       ->07.52% (367,274,326B) 0x60403B: RPCHelpMan::HandleRequest(JSONRPCRequest const&) const (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
50| | |         ->07.52% (367,274,326B) 0x373359: CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)
51| | |           ->07.52% (367,274,326B) 0x425F6C: ExecuteCommand(CRPCCommand const&, JSONRPCRequest const&, UniValue&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
52| | |             ->07.52% (367,274,326B) 0x427373: CRPCTable::execute(JSONRPCRequest const&) const (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
53| | |               ->07.52% (367,274,326B) 0x4274C6: JSONRPCExec(JSONRPCRequest const&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
54| | |                 ->07.52% (367,274,326B) 0x4FF1C3: HTTPReq_JSONRPC(std::any const&, HTTPRequest*) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
55| | |                   ->07.52% (367,274,326B) 0x50CA81: WorkQueue<HTTPClosure>::Run() (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
56| | |                     ->07.52% (367,274,326B) 0x505F06: HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
57| | |                       ->07.52% (367,274,326B) 0x49A6DB3: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33)
58| | |                         ->07.52% (367,274,326B) 0x4CEBAA3: start_thread (pthread_create.c:447)
59| | |                           ->07.52% (367,274,326B) 0x4D78A33: clone (clone.S:100)
60| | |
61| | ->00.01% (449,253B) in 1+ places, all below ms_print's threshold (01.00%)
62| |
63| ->00.00% (144B) in 1+ places, all below ms_print's threshold (01.00%)
64|
65->03.64% (177,834,800B) 0x286989: node::BlockManager::InsertBlockIndex(uint256 const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
66| ->01.82% (88,980,600B) 0x28ABA8: kernel::BlockTreeDB::LoadBlockIndexGuts(Consensus::Params const&, std::function<CBlockIndex* (uint256 const&)>, util::SignalInterrupt const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
67| | ->01.82% (88,980,600B) 0x28AFA8: node::BlockManager::LoadBlockIndex(std::optional<uint256> const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
68| |   ->01.82% (88,980,600B) 0x28B9FE: node::BlockManager::LoadBlockIndexDB(std::optional<uint256> const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
69| |     ->01.82% (88,980,600B) 0x48329F: ChainstateManager::LoadBlockIndex() (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
70| |       ->01.82% (88,980,600B) 0x29ABA7: node::CompleteChainstateInitialization(ChainstateManager&, node::ChainstateLoadOptions const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
71| |         ->01.82% (88,980,600B) 0x29BB79: node::LoadChainstate(ChainstateManager&, kernel::CacheSizes const&, node::ChainstateLoadOptions const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
72| |           ->01.82% (88,980,600B) 0x1EE0BE: InitAndLoadChainstate(node::NodeContext&, bool, bool, kernel::CacheSizes const&, ArgsManager const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
73| |             ->01.82% (88,980,600B) 0x1F3C54: AppInitMain(node::NodeContext&, interfaces::BlockAndHeaderTipInfo*) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
74| |               ->01.82% (88,980,600B) 0x1BE757: main (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
75| |
76| ->01.82% (88,854,200B) 0x28ABC4: kernel::BlockTreeDB::LoadBlockIndexGuts(Consensus::Params const&, std::function<CBlockIndex* (uint256 const&)>, util::SignalInterrupt const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
77|   ->01.82% (88,854,200B) 0x28AFA8: node::BlockManager::LoadBlockIndex(std::optional<uint256> const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
78|     ->01.82% (88,854,200B) 0x28B9FE: node::BlockManager::LoadBlockIndexDB(std::optional<uint256> const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
79|       ->01.82% (88,854,200B) 0x48329F: ChainstateManager::LoadBlockIndex() (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
80|         ->01.82% (88,854,200B) 0x29ABA7: node::CompleteChainstateInitialization(ChainstateManager&, node::ChainstateLoadOptions const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
81|           ->01.82% (88,854,200B) 0x29BB79: node::LoadChainstate(ChainstateManager&, kernel::CacheSizes const&, node::ChainstateLoadOptions const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
82|             ->01.82% (88,854,200B) 0x1EE0BE: InitAndLoadChainstate(node::NodeContext&, bool, bool, kernel::CacheSizes const&, ArgsManager const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
83|               ->01.82% (88,854,200B) 0x1F3C54: AppInitMain(node::NodeContext&, interfaces::BlockAndHeaderTipInfo*) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
84|                 ->01.82% (88,854,200B) 0x1BE757: main (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
85|
86->02.06% (100,650,843B) in 1084 places, all below massif's threshold (1.00%)

 0--------------------------------------------------------------------------------
 1  n       time(ms)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
 2--------------------------------------------------------------------------------
 3  0              0                0                0             0            0
 4  1         85,674    3,026,857,896    3,008,671,214    18,186,682            0
 599.40% (3,008,671,214B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 6->84.57% (2,559,836,160B) 0x5511A7: CCoinsViewCache::EmplaceCoinInternalDANGER(COutPoint&&, Coin&&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
 7| ->84.57% (2,559,836,160B) 0x49E98F: ChainstateManager::PopulateAndValidateSnapshot(Chainstate&, AutoFile&, node::SnapshotMetadata const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
 8|   ->84.57% (2,559,836,160B) 0x49FF9A: ChainstateManager::ActivateSnapshot(AutoFile&, node::SnapshotMetadata const&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
 9|     ->84.57% (2,559,836,160B) 0x367FAF: loadtxoutset()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)
10|       ->84.57% (2,559,836,160B) 0x368686: std::_Function_handler<UniValue (RPCHelpMan const&, JSONRPCRequest const&), loadtxoutset()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)
11|         ->84.57% (2,559,836,160B) 0x60446B: RPCHelpMan::HandleRequest(JSONRPCRequest const&) const (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
12|           ->84.57% (2,559,836,160B) 0x373939: CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)
13|             ->84.57% (2,559,836,160B) 0x4269CC: ExecuteCommand(CRPCCommand const&, JSONRPCRequest const&, UniValue&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
14|               ->84.57% (2,559,836,160B) 0x427DD3: CRPCTable::execute(JSONRPCRequest const&) const (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
15|                 ->84.57% (2,559,836,160B) 0x427F26: JSONRPCExec(JSONRPCRequest const&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
16|                   ->84.57% (2,559,836,160B) 0x4FFED3: HTTPReq_JSONRPC(std::any const&, HTTPRequest*) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
17|                     ->84.57% (2,559,836,160B) 0x50D791: WorkQueue<HTTPClosure>::Run() (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
18|                       ->84.57% (2,559,836,160B) 0x506C16: HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
19|                         ->84.57% (2,559,836,160B) 0x49A6DB3: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33)
20|                           ->84.57% (2,559,836,160B) 0x4CEBAA3: start_thread (pthread_create.c:447)
21|                             ->84.57% (2,559,836,160B) 0x4D78A33: clone (clone.S:100)
22|
23->06.50% (196,857,944B) 0x5531FC: std::__detail::_Hashtable_alloc<PoolAllocator<std::__detail::_Hash_node<std::pair<COutPoint const, CCoinsCacheEntry>, false>, 152ul, 8ul> >::_M_allocate_buckets(unsigned long) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
24| ->06.50% (196,857,944B) 0x5535FD: std::_Hashtable<COutPoint, std::pair<COutPoint const, CCoinsCacheEntry>, PoolAllocator<std::pair<COutPoint const, CCoinsCacheEntry>, 152ul, 8ul>, std::__detail::_Select1st, std::equal_to<COutPoint>, SaltedOutpointHasher, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_rehash(unsigned long, unsigned long const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
25|   ->06.50% (196,857,944B) 0x553825: std::_Hashtable<COutPoint, std::pair<COutPoint const, CCoinsCacheEntry>, PoolAllocator<std::pair<COutPoint const, CCoinsCacheEntry>, 152ul, 8ul>, std::__detail::_Select1st, std::equal_to<COutPoint>, SaltedOutpointHasher, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node<std::pair<COutPoint const, CCoinsCacheEntry>, false>*, unsigned long) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
26|     ->06.50% (196,857,944B) 0x551107: CCoinsViewCache::EmplaceCoinInternalDANGER(COutPoint&&, Coin&&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
27|       ->06.50% (196,857,944B) 0x49E98F: ChainstateManager::PopulateAndValidateSnapshot(Chainstate&, AutoFile&, node::SnapshotMetadata const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
28|         ->06.50% (196,857,944B) 0x49FF9A: ChainstateManager::ActivateSnapshot(AutoFile&, node::SnapshotMetadata const&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
29|           ->06.50% (196,857,944B) 0x367FAF: loadtxoutset()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)
30|             ->06.50% (196,857,944B) 0x368686: std::_Function_handler<UniValue (RPCHelpMan const&, JSONRPCRequest const&), loadtxoutset()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)
31|               ->06.50% (196,857,944B) 0x60446B: RPCHelpMan::HandleRequest(JSONRPCRequest const&) const (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
32|                 ->06.50% (196,857,944B) 0x373939: CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)
33|                   ->06.50% (196,857,944B) 0x4269CC: ExecuteCommand(CRPCCommand const&, JSONRPCRequest const&, UniValue&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
34|                     ->06.50% (196,857,944B) 0x427DD3: CRPCTable::execute(JSONRPCRequest const&) const (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
35|                       ->06.50% (196,857,944B) 0x427F26: JSONRPCExec(JSONRPCRequest const&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
36|                         ->06.50% (196,857,944B) 0x4FFED3: HTTPReq_JSONRPC(std::any const&, HTTPRequest*) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
37|                           ->06.50% (196,857,944B) 0x50D791: WorkQueue<HTTPClosure>::Run() (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
38|                             ->06.50% (196,857,944B) 0x506C16: HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
39|                               ->06.50% (196,857,944B) 0x49A6DB3: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33)
40|                                 ->06.50% (196,857,944B) 0x4CEBAA3: start_thread (pthread_create.c:447)
41|                                   ->06.50% (196,857,944B) 0x4D78A33: clone (clone.S:100)
42|
43->05.88% (177,834,800B) 0x286859: node::BlockManager::InsertBlockIndex(uint256 const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
44| ->02.94% (88,980,600B) 0x28AA78: kernel::BlockTreeDB::LoadBlockIndexGuts(Consensus::Params const&, std::function<CBlockIndex* (uint256 const&)>, util::SignalInterrupt const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
45| | ->02.94% (88,980,600B) 0x28AE78: node::BlockManager::LoadBlockIndex(std::optional<uint256> const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
46| |   ->02.94% (88,980,600B) 0x28B8CE: node::BlockManager::LoadBlockIndexDB(std::optional<uint256> const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
47| |     ->02.94% (88,980,600B) 0x483E9F: ChainstateManager::LoadBlockIndex() (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
48| |       ->02.94% (88,980,600B) 0x29ABB7: node::CompleteChainstateInitialization(ChainstateManager&, node::ChainstateLoadOptions const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
49| |         ->02.94% (88,980,600B) 0x29BB89: node::LoadChainstate(ChainstateManager&, kernel::CacheSizes const&, node::ChainstateLoadOptions const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
50| |           ->02.94% (88,980,600B) 0x1EE0EE: InitAndLoadChainstate(node::NodeContext&, bool, bool, kernel::CacheSizes const&, ArgsManager const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
51| |             ->02.94% (88,980,600B) 0x1F3C84: AppInitMain(node::NodeContext&, interfaces::BlockAndHeaderTipInfo*) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
52| |               ->02.94% (88,980,600B) 0x1BE717: main (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
53| |
54| ->02.94% (88,854,200B) 0x28AA94: kernel::BlockTreeDB::LoadBlockIndexGuts(Consensus::Params const&, std::function<CBlockIndex* (uint256 const&)>, util::SignalInterrupt const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
55|   ->02.94% (88,854,200B) 0x28AE78: node::BlockManager::LoadBlockIndex(std::optional<uint256> const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
56|     ->02.94% (88,854,200B) 0x28B8CE: node::BlockManager::LoadBlockIndexDB(std::optional<uint256> const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
57|       ->02.94% (88,854,200B) 0x483E9F: ChainstateManager::LoadBlockIndex() (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
58|         ->02.94% (88,854,200B) 0x29ABB7: node::CompleteChainstateInitialization(ChainstateManager&, node::ChainstateLoadOptions const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
59|           ->02.94% (88,854,200B) 0x29BB89: node::LoadChainstate(ChainstateManager&, kernel::CacheSizes const&, node::ChainstateLoadOptions const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
60|             ->02.94% (88,854,200B) 0x1EE0EE: InitAndLoadChainstate(node::NodeContext&, bool, bool, kernel::CacheSizes const&, ArgsManager const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
61|               ->02.94% (88,854,200B) 0x1F3C84: AppInitMain(node::NodeContext&, interfaces::BlockAndHeaderTipInfo*) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
62|                 ->02.94% (88,854,200B) 0x1BE717: main (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
63|

Diffing the two reveals a dramatic difference: prevector<28u, unsigned char, unsigned int, int>::change_capacity is 07.53% of prevector<28>, but is completely missing in prevector<34> since the P2WSH/P2TR scripts are now allocated inline and don’t need any capacity change anymore.

The other difference visible directly in these peak snapshot header lines is the extra-heap(B):

prevector<28> peaks at ~235 MiB of extra-heap.
prevector<34> peaks at only ~21 MiB of extra-heap.

These explain the 7% speedup for large memory use cases: elimination of millions of small, separate heap allocations triggered when CScripts (like P2WSH/P2TR) exceed the 28-byte inline limit, each incurring overhead.

l0rinc renamed this:
~~[IBD] prevector: allocate `P2WSH` and `P2TR` scripts on stack as well~~
[IBD] prevector: allocate `P2WSH`/`P2TR`/`P2PK` scripts on stack
on Apr 15, 2025

l0rinc force-pushed on Apr 15, 2025

DrahtBot added the label CI failed on Apr 15, 2025

DrahtBot commented at 9:18 pm on April 15, 2025: contributor

🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/40612214686

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

DrahtBot removed the label CI failed on Apr 15, 2025

l0rinc marked this as ready for review on Apr 15, 2025

l0rinc force-pushed on Apr 16, 2025

DrahtBot added the label CI failed on Apr 16, 2025

DrahtBot commented at 8:01 am on April 16, 2025: contributor

🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/40633401176

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

l0rinc force-pushed on Apr 16, 2025

DrahtBot removed the label CI failed on Apr 16, 2025

achow101 commented at 6:27 pm on April 16, 2025: member

Since this results in more flushes, I think we should wait for the composition of the UTXO set to change to the point that this provides an improvement for nodes with default dbcache.

l0rinc commented at 7:46 pm on April 16, 2025: contributor

Thanks for the idea and comment @achow101! I’ll measure IBD with default settings after the benchmarking servers free up, but so far it seems that there’s barely any slowdown (if any) for default dbcache and a measurable speedup for larger memory settings. So we might already be at the point where it’s either a neutral or a net positive change. And given that we have a few other PRs in the queue that speed up small dbcache IBDs as well (see #32043), we can cover all important scenarios via multiple focused changes.

l0rinc force-pushed on Apr 17, 2025

l0rinc commented at 7:55 am on April 18, 2025: contributor

A few previous measurements were done on 34 byte prevector, the latest version bumped it to 36 (which included P2PK scripts inline as well). I’ve rerun the AssumeUTXO, reindex-chainstate and IBD benchmarks (both on SSD and HDD, since they have different performance profiles) and couldn’t find a single instance where master was any faster.

this provides an improvement for nodes with default dbcache @achow101, what other usecases do you think I should measure to make sure this is indeed the case?

in src/bench/prevector.cpp:33 in a2a3e0cfd8 outdated

32-        t0.resize(28);
33-        t1.resize(29);
34+        prevector<36, T> t0;
35+        prevector<36, T> t1;
36+        t0.resize(36);
37+        t1.resize(37);

kuegi commented at 12:35 pm on April 18, 2025:

maybe a “stupid” question, but why not use a constant for the 36 in this file too?

l0rinc commented at 7:18 pm on April 19, 2025:

I could, of course, for now I was going for simplifying the diff, not make the code more readable. I can of course do that later, but for now I’m looking for the simplest diff which demonstrates the problem and the proposed solution, and after I’m sure there’s general support for it, we can sneak in code refactors as well.

l0rinc commented at 11:13 am on April 22, 2025:

Thanks, extracted to a separate commit - as suggested by @luke-jr as well - added you both as co-authors.

sipa commented at 7:31 pm on April 19, 2025: member

Nit: in title/description of the PR, I don’t think it’s accurate to say “on the stack”. While there are occasionally CScript objects created on the stack, the vast majority is inside CCoinsCacheDb, where they are still heap-allocated. The benefit of the change here is that some of these would no longer need a separate heap allocation per CScript.

l0rinc renamed this:
~~[IBD] prevector: allocate `P2WSH`/`P2TR`/`P2PK` scripts on stack~~
[IBD] prevector: store `P2WSH`/`P2TR`/`P2PK` scripts inline
on Apr 19, 2025

l0rinc force-pushed on Apr 19, 2025

l0rinc commented at 7:39 pm on April 19, 2025: contributor

Thank you, I have used inline/stack interchangeably - this is a lot better, appreciate the correction.

l0rinc force-pushed on Apr 19, 2025

l0rinc commented at 7:18 am on April 20, 2025: contributor

Finished a full reindex-chainstate with default dbcache=450 on HDD

 0COMMITS="156f9913a26598009c8356c34548002e6a6aba02 a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7"; \
 1STOP_HEIGHT=888888; DBCACHE=450; \
 2CC=gcc; CXX=g++; \
 3BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
 4(for c in $COMMITS; do git fetch origin $c -q && git log -1 --pretty=format:'%h %s' $c || exit 1; done) && \
 5hyperfine \
 6  --sort 'command' \
 7  --runs 2 \
 8  --export-json "$BASE_DIR/rdx-${COMMITS// /-}-$STOP_HEIGHT-$DBCACHE-$CC.json" \
 9  --parameter-list COMMIT ${COMMITS// /,} \
10  --prepare "killall bitcoind; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard; \
11    cmake -B build -DCMAKE_BUILD_TYPE=Release -DENABLE_WALLET=OFF && cmake --build build -j$(nproc) --target bitcoind && \
12    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP_HEIGHT -dbcache=5000 -printtoconsole=0; sleep 100" \
13  --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
14  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP_HEIGHT -dbcache=$DBCACHE -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0"

156f9913a2 test: assert CScript allocation characteristics a2a3e0cfd8 store P2WSH/P2TR/P2PK scripts inline

 0Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=450 -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0 (COMMIT = 156f9913a26598009c8356c34548002e6a6aba02)
 1  Time (mean ± σ):     39873.924 s ± 637.446 s    [User: 35537.079 s, System: 2668.529 s]
 2  Range (min … max):   39423.182 s … 40324.667 s    2 runs
 3
 4Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=450 -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0 (COMMI$ = a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7)
 5  Time (mean ± σ):     38682.594 s ±  7.327 s    [User: 34211.447 s, System: 2618.127 s]
 6  Range (min … max):   38677.412 s … 38687.775 s    2 runs
 7
 8Relative speed comparison
 9        1.03 ±  0.02  COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=450 -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0 (COMMIT = 156f9913a26598009c8356c34548002e6a6aba02)
10        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=450 -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0 (COMMIT = a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7)

in src/bench/checkqueue.cpp:19 in 43e003f893 outdated

15@@ -16,7 +16,7 @@
16 
17 static const size_t BATCHES = 101;
18 static const size_t BATCH_SIZE = 30;
19-static const int PREVECTOR_SIZE = 28;
20+static const int PREVECTOR_SIZE = 36;

luke-jr commented at 10:10 pm on April 21, 2025:

Maybe makes sense to move this to a header and reuse it everywhere else?

l0rinc commented at 11:13 am on April 22, 2025:

Done, thanks

bitcoin deleted a comment on Apr 22, 2025

l0rinc force-pushed on Apr 22, 2025

in src/bench/prevector.cpp:5 in 1abeadee08 outdated

1@@ -2,20 +2,22 @@
2 // Distributed under the MIT software license, see the accompanying
3 // file COPYING or http://www.opensource.org/licenses/mit-license.php.
4 
5-#include <prevector.h>

luke-jr commented at 11:38 pm on April 22, 2025:

nit: why moving this?

l0rinc commented at 9:01 am on April 23, 2025:

Just sorted the imports for consistency - but I see that the corresponding header is sometimes separated, so I’ve reverted this move.

What do you think about the whole change in general, can you give me a conceptual review?

l0rinc force-pushed on Apr 23, 2025

l0rinc requested review from luke-jr on Apr 23, 2025

DrahtBot added the label CI failed on Apr 28, 2025

l0rinc force-pushed on Apr 29, 2025

l0rinc commented at 9:44 am on April 29, 2025: contributor

Rebased to cover new tests to resolve new CI failure. Also adjusted the test comments to mention heap/stack less often.

DrahtBot removed the label CI failed on Apr 29, 2025

luke-jr referenced this in commit 83bb9899c6 on Jun 6, 2025

luke-jr referenced this in commit 5d37f3d52d on Jun 6, 2025

luke-jr referenced this in commit 5ff88009a0 on Jun 6, 2025

in src/script/script.h:410 in a5f2ffa951 outdated

405@@ -406,7 +406,8 @@ class CScriptNum
406  * Tests in October 2015 showed use of this reduced dbcache memory usage by 23%
407  *  and made an initial sync 13% faster.
408  */
409-typedef prevector<28, unsigned char> CScriptBase;
410+static constexpr unsigned int PREVECTOR_SIZE{28};
411+typedef prevector<PREVECTOR_SIZE, unsigned char> CScriptBase;

maflcko commented at 8:45 am on July 21, 2025:

nit in the first commit: could be using using for new code?

l0rinc commented at 5:55 pm on July 21, 2025:

Sure, modernized the definition in a separate commit

in src/script/script.h:409 in a5f2ffa951 outdated

405@@ -406,7 +406,8 @@ class CScriptNum
406  * Tests in October 2015 showed use of this reduced dbcache memory usage by 23%
407  *  and made an initial sync 13% faster.
408  */
409-typedef prevector<28, unsigned char> CScriptBase;
410+static constexpr unsigned int PREVECTOR_SIZE{28};

maflcko commented at 9:03 am on July 21, 2025:

nit in the first commit: Should this be called script_base_size instead? There are other uses of prevector (CompressedScript, netaddr, …), so defining this globally could be confusing.

l0rinc commented at 6:10 pm on July 21, 2025:

Makes sense, thanks

in src/test/script_tests.cpp:1161 in ecc6c07e58 outdated

1163+    BOOST_CHECK_EQUAL(sizeof(prevector<PREVECTOR_SIZE, unsigned char>), 32);
1164+    BOOST_CHECK_EQUAL(sizeof(CScriptBase), 32);
1165+    BOOST_CHECK_EQUAL(sizeof(CScript), 32);
1166+    BOOST_CHECK_EQUAL(sizeof(CTxOut), 40);
1167+
1168+    CKey dummyKey;

maflcko commented at 11:14 am on July 21, 2025:

nit in ecc6c07e583842a2afe1a8b5bb8ec0bb9f997fdb: Use snake_case for new code?

l0rinc commented at 6:02 pm on July 21, 2025:

Done

in src/test/script_tests.cpp:1164 in ecc6c07e58 outdated

1166+    BOOST_CHECK_EQUAL(sizeof(CTxOut), 40);
1167+
1168+    CKey dummyKey;
1169+    dummyKey.MakeNewKey(true);
1170+
1171+    std::vector<std::vector<uint8_t>> dummyVSolutions;

maflcko commented at 11:25 am on July 21, 2025:

nit in https://github.com/bitcoin/bitcoin/commit/ecc6c07e583842a2afe1a8b5bb8ec0bb9f997fdb: Could add and use a small helper: TxoutType GetTxoutType(const CScript& output_script)? This makes the code shorter below and wraps this dummy.

l0rinc commented at 6:08 pm on July 21, 2025:

Done, thanks

l0rinc force-pushed on Jul 21, 2025

l0rinc commented at 6:22 pm on July 21, 2025: contributor

Thanks, applied most of your comments + a few additional cleanups. The first push was a simple rebase, the second contains the changes.

in src/test/script_tests.cpp:1182 in 3a9c80f7b4 outdated

1177+        BOOST_CHECK_EQUAL(script.allocated_memory(), 0);
1178+    }
1179+
1180+    // P2WPKH has direct allocation
1181+    {
1182+        const auto script{GetScriptForDestination(WitnessV0KeyHash{PKHash{CKeyID{CPubKey{dummy_key.GetPubKey()}.GetID()}}})};

maflcko commented at 11:58 am on July 22, 2025:

nit in 3a9c80f7b4807e9abd9f2da36e8ee479ecf00d19:

GetPubKey does not need to be wrapped in CPubKey{}. You could just store a dummy_pubkey for all test cases to re-use.

l0rinc commented at 6:06 pm on July 22, 2025:

Thanks, will do this in the next push

maflcko commented at 8:14 am on July 25, 2025:

Thanks, will do this in the next push

I don’t think you did?

l0rinc commented at 11:37 pm on July 25, 2025:

I quickly pushed a rebase, thinking I’ll follow with applying the recommendations - but ended up in a CompressedScript rabbithole (which I decided to leave to a follow-up in the end).

Looks like the extractions and inlines I did here ended up with lots of unnecessary casts, thanks for noticing, cleaned it up in latest push!

maflcko approved

maflcko commented at 12:13 pm on July 22, 2025: member

I haven’t tested IBD/reindex performance.

review ACK f7d9f4510d537b280808e1e8e203c9445c8ad4df 🍴

Signature:

0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
2trusted comment: review ACK f7d9f4510d537b280808e1e8e203c9445c8ad4df 🍴
3nXDCykVC7pudYdduUKbJSXeg0XwPp/YhNfksikOYMhOyQXyHf4x9gENBZX6FGMcUav92tl9Hna3Z5RXQAOddDQ==

l0rinc commented at 7:06 pm on July 23, 2025: contributor

I have remeasured -reindex-chainstate speed by running the rebased PR (now that other optimizations were merged) until 900k blocks twice before and twice after, and the speedup is still ~4%:

 0COMMITS="7129c9ea8e950f50bdc56d88c57617c66c90bb8a f7d9f4510d537b280808e1e8e203c9445c8ad4df"; \                     
 1STOP=900000; DBCACHE=5000; \                                                                                                                                                   
 2CC=gcc; CXX=g++; \                                                                                                                                                             
 3BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
 4(echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done; echo "") && \
 5hyperfine \
 6  --sort command \
 7  --runs 2 \
 8  --export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \
 9  --parameter-list COMMIT ${COMMITS// /,} \ 
10  --prepare "killall bitcoind; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
11    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release && ninja -C build bitcoind && \
12    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 10" \
13  --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
14  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0"
15
167129c9ea8e Merge bitcoin/bitcoin#32827: mempool: Avoid needless vtx iteration during IBD
17f7d9f4510d prevector: store P2WSH/P2TR/P2PK scripts inline
18
19Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=900000 -dbcache=5000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 7129c9ea8e950f50bdc56d88c57617c66c90bb8a)
20  Time (mean ± σ):     17108.190 s ± 48.793 s    [User: 30728.280 s, System: 987.269 s]                                                                                        
21  Range (min … max):   17073.688 s … 17142.691 s    2 runs                                                                                                                     
22                                                                                       
23Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=900000 -dbcache=5000 -reindex-chainstate -blocksonly -connect=0 -printtocons$le=0 (COMMIT = f7d9f4510d537b280808e1e8e203c9445c8ad4df)
24  Time (mean ± σ):     16510.352 s ± 69.453 s    [User: 29862.737 s, System: 943.983 s]
25  Range (min … max):   16461.241 s … 16559.463 s    2 runs
26  
27Relative speed comparison
28        1.04 ±  0.01  COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=900000 -dbcache=5000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 7129c9ea8e950f50bdc56d88c57617c66c90bb8a)
29        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=900000 -dbcache=5000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = f7d9f4510d537b280808e1e8e203c9445c8ad4df)

Edit: To show the exact behavior, I’ve plotted the height-vs-time (and the coins-vs-time) before and after:

7129c9ea8e is master
f7d9f4510d is this PR

Edit2: Added bar charts by 100k blocks as requested by @ajtowns:

in src/test/script_tests.cpp:1163 in f7d9f4510d outdated

1158+    return Solver(output_script, unused);
1159+}
1160+
1161+BOOST_AUTO_TEST_CASE(script_size_and_capacity_test)
1162+{
1163+    BOOST_CHECK_EQUAL(sizeof(prevector<34, uint8_t>), sizeof(prevector<SCRIPT_BASE_SIZE, uint8_t>));

ajtowns commented at 2:32 am on July 24, 2025:

Could consider adding something like:

0// SCRIPT_BASE_SIZE should be set to avoid wasting space in padding
1BOOST_CHECK(sizeof(CScriptBase) != sizeof(prevector<SCRIPT_BASE_SIZE+1, uint8_t>));

It might be nicer to have prevector include a static constexpr size_t STATIC_SIZE = N; member, and use CScriptBase::STATIC_SIZE rather than SCRIPT_BASE_SIZE.

l0rinc commented at 11:35 pm on July 25, 2025:

This is indeed a lot nicer, thanks!

ajtowns commented at 2:36 am on July 24, 2025: contributor

I haven’t investigated in detail why there are so many empty prevectors, can very well be just an unoptimized debug run artifact of all the Coin copies.

I think it’s just that coins get cleared when they’re spent, so end up as 0 size by the time they’re destructed? Empty scriptSig’s for witness spends might also have some impact.

I think the tradeoff is something like:

spends of p2pk, p2sh, p2pkh coins – these cost 8 more bytes
spends of p2wpkh – these cost 16 more bytes (sPK and scriptSig didn’t need an allocation)
spends of p2wsh and p2tr – these cost ~48 fewer bytes (save 64 byte allocation on 64bit system, lose 8 bytes for both scriptSig and sPK)
spends of nested p2wsh – presumably save ~96 bytes, since the scriptSig would save an allocation, but I’m bundling it in the previous section

Based on mainnet.observer stats for 2025-05-08, p2wpkh is about 55% of txs, p2tr is about 28%, p2pkh about 13%, p2wsh about 4% and the rest is noise, maybe? Those numbers net out to a saving of ~5.5 bytes per input. If p2wpkh rose from 55% to 80% and p2tr dropped to 20%, that would net to wasting ~3.2 bytes per input.

So I think this change seems plausible now, and certainly is in line with wanting to recommend the use of 256-bit scriptPubKeys for any sort of collaboratively held utxos (ie multisig via p2wpkh or taproot scripts, or musig via taproot), and to encourage collaboratively held utxos.

Approach ACK.

ajtowns commented at 2:39 am on July 24, 2025: contributor

To show the exact behavior, I’ve plotted the height-vs-time (and the coins-vs-time) before and after:

Probably would be better to bucket the heights and do bar charts, ie: how long did the 1st, 2nd, 3rd, etc 100k blocks take with both approaches. That way it’s easier to see if a large gain for this PR was temporary (eg, due to lots of inscriptions) and is not providing a benefit with current traffic.

luke-jr referenced this in commit 59ccbb4049 on Jul 24, 2025

luke-jr referenced this in commit 430aae56c6 on Jul 24, 2025

luke-jr referenced this in commit ca2576d8cc on Jul 24, 2025

luke-jr referenced this in commit d02dffbd4b on Jul 24, 2025

l0rinc force-pushed on Jul 24, 2025

achow101 commented at 9:55 pm on July 24, 2025: member

ACK ec0b90e40e21ccb97c57373f7a7665ab9554acec

In a sample size of 1, this seems to provide a significant improvement in IBD time. It’s surprising to me that it looks like this would have had benefit around the time that Taproot activated as previous measurements hadn’t indicated that.

DrahtBot requested review from maflcko on Jul 24, 2025

DrahtBot requested review from ajtowns on Jul 24, 2025

refactor: extract `STATIC_SIZE` constant to prevector

Co-authored-by: Anthony Towns <aj@erisian.com.au>

756da2a994

refactor: modernize `CScriptBase` definition 65ac7f6d4d

l0rinc force-pushed on Jul 25, 2025

l0rinc commented at 11:39 pm on July 25, 2025: contributor

Thanks for the recommendations, took them and extended the tests slightly to document the current behavior of CompressedScript (unchanged) and uncompressed P2PK as well (also unchanged). I have also rebased to make sure all new test cases are run - ready for re-review.

test: assert `CScript` allocation characteristics

Verifies that script types are correctly allocated using prevector's direct or indirect storage based on their size:

Direct allocated script types (size ≤ 28 bytes):
* OP_RETURN (small)
* P2WPKH
* P2SH
* P2PKH

Indirect allocated script types (size > 28 bytes):
* P2WSH
* P2TR
* P2PK
* MULTISIG (small)

This test provides a baseline for verifying changes to prevector's inline capacity.

The `CHECK_SCRIPT_STATIC_SIZE` and `CHECK_SCRIPT_DYNAMIC_SIZE` macros were added to differentiate the two cases - while preserving the correct source code line in case of failure.

52121506b2

prevector: store `P2WSH`/`P2TR`/`P2PK` scripts inline

The current `prevector` size of 28 bytes (chosen to fill the `sizeof(CScript)` aligned size) was introduced in 2015 (https://github.com/bitcoin/bitcoin/pull/6914) before SegWit and TapRoot.
However, the increasingly common `P2WSH` and `P2TR` scripts are both 34 bytes, and are forced to use heap (re)allocation rather than efficient inline storage.

The core trade-off of this change is to eliminate heap allocations for common 34-36 byte scripts at the cost of increasing the base memory footprint of all `CScript` objects by 8 bytes (while still respecting peak memory usage defined by `-dbcache`).

Increasing the `prevector` size allows these scripts to be stored inline, avoiding extra heap allocations, reducing potential memory fragmentation, and improving performance during cache flushes. Massif analysis confirms a lower stable memory usage after flushing, suggesting the elimination of heap allocations outweighs the larger base size for common workloads.

Due to memory alignment, increasing the `prevector` size to 36 bytes doesn't change the overall `sizeof(CScript)` compared to an increase to 34 bytes, allowing us to include `P2PK` scripts as well at no additional memory cost.

Performance benchmarks for AssumeUTXO load and flush show:
* Small dbcache (450MB): ~1-3% performance improvement (despite more frequent flushes)
* Large dbcache (4500MB): ~6-8% performance improvement due to fewer heap allocations (and basically the number of flushes)
* Very large dbcache (4500MB): ~5-6% performance improvement due to fewer heap allocations (and memory limit not being reached, so there's no memory penalty)

Full IBD and reindex-chainstate with larger `dbcache` values also show an overall ~3-4% speedup.

Co-authored-by: Ava Chow <github@achow101.com>
Co-authored-by: Andrew Toth <andrewstoth@gmail.com>
Co-authored-by: maflcko <6399679+maflcko@users.noreply.github.com>

d5104cfbae

in src/test/script_tests.cpp:1215 in 44fc161634 outdated

1213         BOOST_CHECK(script.IsPayToWitnessScriptHash());
1214         BOOST_CHECK_EQUAL(script.size(), 34);
1215-        BOOST_CHECK_EQUAL(script.capacity(), 34);
1216-        BOOST_CHECK_EQUAL(script.allocated_memory(), 34);
1217+        BOOST_CHECK_EQUAL(script.capacity(), CScriptBase::STATIC_SIZE);
1218+        BOOST_CHECK_EQUAL(script.allocated_memory(), 0);

ajtowns commented at 5:22 am on July 26, 2025:

 0static void check_script_has_static_size(CScript& script, unsigned int size)
 1{
 2    BOOST_CHECK_EQUAL(script.size(), size);
 3    BOOST_CHECK_EQUAL(script.capacity(), CScriptBase::STATIC_SIZE);
 4    BOOST_CHECK_EQUAL(script.allocated_memory(), 0);
 5}
 6
 7static void check_script_has_dynamic_size(CScript& script, unsigned int size)
 8{
 9    BOOST_CHECK_EQUAL(script.size(), size);
10    BOOST_CHECK_EQUAL(script.capacity(), size);
11    BOOST_CHECK_EQUAL(script.allocated_memory(), size);
12}

and

0        BOOST_CHECK(script.IsPayToWitnessScriptHash());
1        check_script_has_static_size(script, 34);

?

l0rinc commented at 5:24 am on July 26, 2025:

you think that would be easier to follow? If it fails I would see it logged in check_script_has_dynamic_size - unless I make it a macro.

l0rinc commented at 5:35 am on July 26, 2025:

Thanks, pushed, let me know what you think!

ajtowns commented at 6:35 am on July 26, 2025:

Can get errors like:

0test/script_tests.cpp(1246): error: in "script_tests/script_size_and_capacity_test": check script_dynamic_alloc( script, 64, 64 ) has failed for ( CScript(size=67,cap=67), 64, 64 )
1test/script_tests.cpp(1253): error: in "script_tests/script_size_and_capacity_test": check script_dynamic_alloc( script, 10, 1000 ) has failed for ( CScript(size=71,cap=103), 10, 1000 )

with this approach:

 0--- a/src/test/script_tests.cpp
 1+++ b/src/test/script_tests.cpp
 2@@ -38,6 +38,12 @@
 3
 4 using namespace util::hex_literals;
 5
 6+static std::ostream& operator<<(std::ostream& os, const CScript& script)
 7+{
 8+    os << strprintf("CScript(size=%d,cap=%d)", script.size(), script.capacity());
 9+    return os;
10+}
11+
12 static const unsigned int gFlags = SCRIPT_VERIFY_P2SH | SCRIPT_VERIFY_STRICTENC;
13
14 unsigned int ParseScriptFlags(std::string strFlags);
15@@ -1158,6 +1164,16 @@ static TxoutType GetTxoutType(const CScript& output_script)
16     return Solver(output_script, unused);
17 }
18
19+static bool script_static_alloc(const CScript& script, unsigned int size)
20+{
21+    return script.size() == size && script.capacity() == script.STATIC_SIZE && script.allocated_memory() == 0;
22+}
23+
24+static bool script_dynamic_alloc(const CScript& script, unsigned int size, unsigned int cap)
25+{
26+    return script.size() == size && script.capacity() == cap && script.allocated_memory() == cap;
27+}
28+
29 BOOST_AUTO_TEST_CASE(script_size_and_capacity_test)
30 {
31     BOOST_CHECK_EQUAL(sizeof(CompressedScript), 40);
32
33@@ -1241,18 +1243,14 @@ BOOST_AUTO_TEST_CASE(script_size_and_capacity_test)
34
35         const auto script{GetScriptForRawPubKey(uncompressed_pubkey)};
36         BOOST_CHECK_EQUAL(GetTxoutType(script), TxoutType::PUBKEY);
37-        BOOST_CHECK_EQUAL(script.size(), 67);
38-        BOOST_CHECK_EQUAL(script.capacity(), 67);
39-        BOOST_CHECK_EQUAL(script.allocated_memory(), 67);
40+        BOOST_CHECK_PREDICATE(script_dynamic_alloc, (script)(67)(67));
41     }
42
43     // Bare multisig needs extra allocation
44     {
45         const auto script{GetScriptForMultisig(1, std::vector{2, dummy_pubkey})};
46         BOOST_CHECK_EQUAL(GetTxoutType(script), TxoutType::MULTISIG);
47-        BOOST_CHECK_EQUAL(script.size(), 71);
48-        BOOST_CHECK_EQUAL(script.capacity(), 103);
49-        BOOST_CHECK_EQUAL(script.allocated_memory(), 103);
50+        BOOST_CHECK_PREDICATE(script_dynamic_alloc, (script)(71)(103));
51     }
52 }

l0rinc force-pushed on Jul 26, 2025

[IBD] prevector: store P2WSH/P2TR/P2PK scripts inline #32279

Summary

Context

Benchmarks and Memory

Historical explanation for the speedup (by Anthony Towns)

Code Coverage & Benchmarks

Reviews

Conflicts

Additional info:

AssumeUTXO

Reindex-chainstate

Full IBD

Memory

[IBD] prevector: store `P2WSH`/`P2TR`/`P2PK` scripts inline #32279