[IBD] prevector: store P2WSH/P2TR/P2PK scripts inline #32279

pull l0rinc wants to merge 4 commits into bitcoin:master from l0rinc:l0rinc/prevector-size changing 6 files +131 −28
  1. l0rinc commented at 4:20 pm on April 15, 2025: contributor

    This change is part of [IBD] - Tracking PR for speeding up Initial Block Download

    Summary

    The current prevector size of 28 bytes (chosen to fill the sizeof(CScript) aligned size) was introduced in 2015 (https://github.com/bitcoin/bitcoin/pull/6914) before SegWit and TapRoot. However, the increasingly common P2WSH and P2TR scripts are both 34 bytes, and are forced to use heap (re)allocation rather than efficient inline storage.

    The core trade-off of this change is to eliminate heap allocations for common 34-36 byte scripts at the cost of increasing the base memory footprint of all CScript objects by 8 bytes (while still respecting peak memory usage defined by -dbcache).

    Context

    Increasing the prevector size allows these scripts to be stored inline, avoiding heap allocations, reducing potential memory fragmentation, and improving performance during cache flushes. Massif analysis confirms a lower stable memory usage after flushing, suggesting the elimination of heap allocations outweighs the larger base size for common workloads.

    Due to memory alignment, increasing the prevector size to 36 bytes doesn’t change the overall sizeof(CScript) compared to an increase to 34 bytes, allowing us to include P2PK scripts as well at no additional memory cost.

    dbcache=440

    Massif before, with a heap threshold of 28:

     0    MB
     1744.1^#                                                                       
     2     |#: ::::::@: :::::::   :@:: @::::::::::::::@@                            
     3     |#: ::::::@::::: :::   :@:::@:::::: :: ::::@                             
     4     |#: ::::::@::::: :::   :@:::@:::::: :: ::::@                             
     5     |#: ::::::@::::: ::: : :@:::@:::::: :: ::::@                             
     6     |#: ::::::@::::: ::: : :@:::@:::::: :: ::::@                             
     7     |#: ::::::@::::: ::: : :@:::@:::::: :: ::::@                             
     8     |#::::::::@::::: ::: : :@:::@:::::: :: ::::@                             
     9     |#::::::::@::::: ::: :::@:::@:::::: :: ::::@                             
    10     |#::::::::@::::: ::: :::@:::@:::::: :: ::::@                             
    11     |#::::::::@::::: ::: :::@:::@:::::: :: ::::@                             
    12     |#::::::::@::::: ::: :::@:::@:::::: :: ::::@                             
    13     |#::::::::@::::: :::::::@:::@:::::: :: ::::@                             
    14     |#::::::::@::::: :::::::@:::@:::::: :: ::::@ :::::@:::::@:::::@:::::@::::
    15     |#::::::::@::::: :::::::@:::@:::::: :: ::::@ :::::@:::::@:::::@:::::@::::
    16     |#::::::::@::::: :::::::@:::@:::::: :: ::::@ :::::@:::::@:::::@:::::@::::
    17     |#::::::::@::::: :::::::@:::@:::::: :: ::::@ :::::@:::::@:::::@:::::@::::
    18     |#::::::::@::::: :::::::@:::@:::::: :: ::::@ :::::@:::::@:::::@:::::@::::
    19     |#::::::::@::::: :::::::@:::@:::::: :: ::::@ :::::@:::::@:::::@:::::@::::
    20     |#::::::::@::::: :::::::@:::@:::::: :: ::::@ :::::@:::::@:::::@:::::@::::
    21   0 +----------------------------------------------------------------------->h
    22     0                                                                   1.805
    

    and after, with a heap threshold of 36:

     0    MB
     1744.2^       :                                                                
     2     |#  :  :::::::::::   : : :: ::: @@:::::: ::  :                           
     3     |#  :  :::: ::::::   : : :: ::: @ :: ::  :   :                           
     4     |#  :  :::: :::::::  : :@:: ::: @ :: ::  : :::                           
     5     |#  :  :::: :::::::  : :@:: ::: @ :: ::  : : :                           
     6     |#  :  :::: :::::::  : :@:: ::: @ :: ::  : : :                           
     7     |#  :  :::: :::::::  : :@:: ::: @ :: ::  : : :                           
     8     |#  :: :::: :::::::  : :@:: ::: @ :: ::  : : :                           
     9     |#  :: :::: :::::::  : :@:: ::::@ :: ::  : : :                           
    10     |#:::: :::: :::::::  :::@:: ::::@ :: ::  : : :                           
    11     |#: ::::::: :::::::  :::@:: ::::@ :: :: @: : :                           
    12     |#: ::::::: :::::::  :::@:::::::@ :: :: @: : :                           
    13     |#: ::::::: ::::::::::::@:::::::@ :: :: @: : :                           
    14     |#: ::::::: :::::::: :::@:::::::@ :: :: @: : :                           
    15     |#: ::::::: :::::::: :::@:::::::@ :: :: @: : :::@:::@::::@::::::@:::::@::
    16     |#: ::::::: :::::::: :::@:::::::@ :: :: @: : :::@:::@::::@::::::@:::::@::
    17     |#: ::::::: :::::::: :::@:::::::@ :: :: @: : :::@:::@::::@::::::@:::::@::
    18     |#: ::::::: :::::::: :::@:::::::@ :: :: @: : :::@:::@::::@::::::@:::::@::
    19     |#: ::::::: :::::::: :::@:::::::@ :: :: @: : :::@:::@::::@::::::@:::::@::
    20     |#: ::::::: :::::::: :::@:::::::@ :: :: @: : :::@:::@::::@::::::@:::::@::
    21   0 +----------------------------------------------------------------------->h
    22     0                                                                   1.618
    

    for dbcache=4500:

    Massif before, with a heap threshold of 28:

     0    GB
     14.565^   ::                                                                   
     2     | ##:   @@:::  :::: :@::::  :::: ::::                                    
     3     | # :   @ ::   :::  :@: ::  : :: :::                                     
     4     | # :   @ :: :::::  :@: ::  : :: :::                                     
     5     | # :   @ :: : :::  :@: :: @: :: :::                                     
     6     | # :   @ :: : :::  :@: :: @: :: :::                                     
     7     | # :   @ :: : :::  :@: :: @: :: :::                                     
     8     | # :   @ :: : :::  :@: :: @: :: :::                                     
     9     | # : ::@ :: : :::  :@: :: @: :: :::                                     
    10     | # : : @ :: : :::  :@: :: @: :: :::                                     
    11     | # : : @ :: : :::  :@: :: @: ::::::                                     
    12     | # : : @ :: : :::  :@: :: @: ::::::                                     
    13     | # : : @ :: : :::  :@: :: @: ::::::                                     
    14     | # : : @ :: : ::: ::@: :: @: ::::::                                     
    15     | # : : @ :: : ::: ::@: :: @: ::::::                                     
    16     | # : : @ :: : ::: ::@: :: @: ::::::                                     
    17     | # : : @ :: : ::: ::@: :: @: :::::: @::                                 
    18     | # : : @ :: : ::: ::@: :: @: :::::: @:                                  
    19     | # : : @ :: : ::: ::@: :::@: :::::: @:                                  
    20     | # : : @ :: : ::: ::@: :::@: :::::: @: :::::::::::::::::::::::::::::@:::
    21   0 +----------------------------------------------------------------------->h
    22     0                                                                   1.500
    

    and after, with a heap threshold of 36:

     0    GB
     14.640^    :                                                                   
     2     | ##::  :::::   ::::  ::::::@  ::::                                      
     3     | # ::  : :::   ::::  :: :::@  ::::                                      
     4     | # :: :: :::   ::::  :: :::@  ::::                                      
     5     | # :: :: :::  :::::  :: :::@  ::::                                      
     6     | # :: :: :::  :::::  :: :::@  ::::                                      
     7     | # :: :: :::  :::::  :: :::@  ::::                                      
     8     | # :: :: :::  :::::  :: :::@  ::::                                      
     9     | # :: :: :::  :::::  :: :::@  ::::  :@@                                 
    10     | # :: :: :::  ::::: ::: :::@  :::::::@                                  
    11     | # :: :: :::  ::::: ::: :::@  ::::: :@                                  
    12     | # :: :: :::  ::::: ::: :::@::::::: :@                                  
    13     | # ::::: :::  ::::: ::: :::@: ::::: :@                                  
    14     | # ::::: :::  ::::: ::: :::@: ::::: :@                                  
    15     | # ::::: :::::::::: ::: :::@: ::::: :@                                  
    16     | # ::::: :::: ::::: ::: :::@: ::::: :@                                  
    17     | # ::::: :::: ::::: ::: :::@: ::::: :@                                  
    18     | # ::::: :::: ::::::::: :::@: ::::: :@                                  
    19     | # ::::: :::: ::::::::: :::@: ::::: :@                                  
    20     | # ::::: :::: ::::::::: :::@: ::::: :@ ::::::@:::@:::@::::@:::::@::::@::
    21   0 +----------------------------------------------------------------------->h
    22     0                                                                   1.360
    

    Benchmarks and Memory

    Performance benchmarks for AssumeUTXO load and flush show:

    • Small dbcache (450MB): ~1-3% performance improvement (despite more frequent flushes)
    • Large dbcache (4500MB): ~6-8% performance improvement due to fewer heap allocations (and basically the number of flushes)
    • Very large dbcache (4500MB): ~5-6% performance improvement due to fewer heap allocations (and memory limit not being reached, so there’s no memory penalty)

    Full IBD and -reindex-chainstate with also show an overall ~3-4% speedup (both for smaller and larger dbcache values).

    We haven’t investigated using different prevector sizes based on script type, though this could be explored in the future if needed.

    Historical explanation for the speedup (by Anthony Towns)

    I think the tradeoff is something like:

    • spends of p2pk, p2sh, p2pkh coins – these cost 8 more bytes
    • spends of p2wpkh – these cost 16 more bytes (sPK and scriptSig didn’t need an allocation)
    • spends of p2wsh and p2tr – these cost ~48 fewer bytes (save 64 byte allocation on 64bit system, lose 8 bytes for both scriptSig and sPK)
    • spends of nested p2wsh – presumably save ~96 bytes, since the scriptSig would save an allocation, but I’m bundling it in the previous section

    Based on mainnet.observer stats for 2025-05-08, p2wpkh is about 55% of txs, p2tr is about 28%, p2pkh about 13%, p2wsh about 4% and the rest is noise, maybe? Those numbers net out to a saving of ~5.5 bytes per input. If p2wpkh rose from 55% to 80% and p2tr dropped to 20%, that would net to wasting ~3.2 bytes per input.

  2. DrahtBot commented at 4:20 pm on April 15, 2025: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/32279.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    Approach ACK ajtowns
    Stale ACK maflcko, achow101

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #33021 (test: revive test verifying that GetCoinsCacheSizeState switches from OK→LARGE→CRITICAL by l0rinc)
    • #28531 (improve MallocUsage() accuracy by LarryRuane)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  3. l0rinc renamed this:
    prevector: allocate `P2WSH` and `P2TR` scripts on stack as well
    [IBD] prevector: allocate `P2WSH` and `P2TR` scripts on stack as well
    on Apr 15, 2025
  4. l0rinc commented at 5:08 pm on April 15, 2025: contributor

    Additional info:

    Added logging to the original prevector destructor to measure its final size - and printed the values during an assumeutxo load & dump:

    I haven’t investigated in detail why there are so many empty prevectors, can very well be just an unoptimized debug run artifact of all the Coin copies.


    If we plot the coin cache size vs height, we see that with the original prevector size, the first part of the sync fits more values initially and fewer ones later:

    After this change, the occupancy is more predictable:


    AssumeUTXO

    Ran some AssumeUTXO benchmarks with different values (on SSD):

     0COMMITS="847a891cdfdf9cfd69a77a8688c995e532ea29db dc0c79a46b5663eddc0a4365b9b7de22cf3fefae"; \
     1CC=gcc; CXX=g++; \
     2BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/ShallowBitcoinData"; LOG_DIR="$BASE_DIR/logs"; UTXO_SNAPSHOT_PATH="$BASE_DIR/utxo-880000.dat"; \
     3(for c in $COMMITS; do git fetch origin $c -q && git log -1 --pretty=format:'%h %s' $c || exit 1; done) && \
     4for DBCACHE in 450 4500 45000; do \
     5  hyperfine \
     6  --sort 'command' \
     7  --runs 5 \
     8  --export-json "$BASE_DIR/assumeutxo-${COMMITS// /-}-$DBCACHE-$CC.json" \
     9  --parameter-list COMMIT ${COMMITS// /,} \
    10  --prepare "killall bitcoind; rm -rf $DATA_DIR/chainstate $DATA_DIR/chainstate_snapshot $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard; \
    11    cmake -B build -DCMAKE_BUILD_TYPE=Release -DENABLE_WALLET=OFF && cmake --build build -j$(nproc) --target bitcoind --target bitcoin-cli && \
    12    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=1 -printtoconsole=0; \
    13    ./build/bin/bitcoind -datadir=$DATA_DIR -daemon -blocksonly=1 -connect=0 -dbcache=$DBCACHE -printtoconsole=0; sleep 10" \
    14  --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-assumeutxo-{COMMIT}-dbcache-$DBCACHE-$(date +%s).log; build/bin/bitcoin-cli -datadir=$DATA_DIR stop; sleep 10; killall bitcoind; sleep 10;" \
    15  "COMPILER=$CC DBCACHE=$DBCACHE COMMIT={COMMIT} ./build/bin/bitcoin-cli -datadir=$DATA_DIR -rpcclienttimeout=0 loadtxoutset $UTXO_SNAPSHOT_PATH"; \
    16done
    

    847a891cdf test: assert CScript allocation characteristics dc0c79a46b store P2WSH/P2TR/P2PK scripts inline

     0Benchmark 1: COMPILER=gcc DBCACHE=450 COMMIT=847a891cdfdf9cfd69a77a8688c995e532ea29db ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
     1  Time (mean ± σ):     451.840 s ±  2.255 s    [User: 0.001 s, System: 0.001 s]
     2  Range (min  max):   448.790 s  454.846 s    5 runs
     3
     4Benchmark 2: COMPILER=gcc DBCACHE=450 COMMIT=dc0c79a46b5663eddc0a4365b9b7de22cf3fefae ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
     5  Time (mean ± σ):     451.480 s ±  5.381 s    [User: 0.002 s, System: 0.001 s]
     6  Range (min  max):   447.360 s  459.782 s    5 runs
     7
     8Relative speed comparison
     9        1.00 ±  0.01  COMPILER=gcc DBCACHE=450 COMMIT=847a891cdfdf9cfd69a77a8688c995e532ea29db ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
    10        1.00          COMPILER=gcc DBCACHE=450 COMMIT=dc0c79a46b5663eddc0a4365b9b7de22cf3fefae ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
    

    847a891cdf test: assert CScript allocation characteristics dc0c79a46b store P2WSH/P2TR/P2PK scripts inline

     0Benchmark 1: COMPILER=gcc DBCACHE=4500 COMMIT=847a891cdfdf9cfd69a77a8688c995e532ea29db ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
     1  Time (mean ± σ):     480.316 s ±  2.640 s    [User: 0.001 s, System: 0.002 s]
     2  Range (min  max):   477.157 s  483.803 s    5 runs
     3
     4Benchmark 2: COMPILER=gcc DBCACHE=4500 COMMIT=dc0c79a46b5663eddc0a4365b9b7de22cf3fefae ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
     5  Time (mean ± σ):     452.521 s ±  7.533 s    [User: 0.001 s, System: 0.001 s]
     6  Range (min  max):   445.367 s  460.740 s    5 runs
     7
     8Relative speed comparison
     9        1.06 ±  0.02  COMPILER=gcc DBCACHE=4500 COMMIT=847a891cdfdf9cfd69a77a8688c995e532ea29db ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
    10        1.00          COMPILER=gcc DBCACHE=4500 COMMIT=dc0c79a46b5663eddc0a4365b9b7de22cf3fefae ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
    

    847a891cdf test: assert CScript allocation characteristics dc0c79a46b store P2WSH/P2TR/P2PK scripts inline

     0Benchmark 1: COMPILER=gcc DBCACHE=45000 COMMIT=847a891cdfdf9cfd69a77a8688c995e532ea29db ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
     1  Time (mean ± σ):     469.846 s ±  1.583 s    [User: 0.002 s, System: 0.001 s]
     2  Range (min  max):   467.909 s  472.053 s    5 runs
     3
     4Benchmark 2: COMPILER=gcc DBCACHE=45000 COMMIT=dc0c79a46b5663eddc0a4365b9b7de22cf3fefae ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
     5  Time (mean ± σ):     448.079 s ±  1.180 s    [User: 0.002 s, System: 0.001 s]
     6  Range (min  max):   447.280 s  450.165 s    5 runs
     7
     8Relative speed comparison
     9        1.05 ±  0.00  COMPILER=gcc DBCACHE=45000 COMMIT=847a891cdfdf9cfd69a77a8688c995e532ea29db ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
    10        1.00          COMPILER=gcc DBCACHE=45000 COMMIT=dc0c79a46b5663eddc0a4365b9b7de22cf3fefae ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
    

    And on HDD:

     0COMMITS="156f9913a26598009c8356c34548002e6a6aba02 a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7"; \
     1CC=gcc; CXX=g++; \
     2BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/ShallowBitcoinData"; LOG_DIR="$BASE_DIR/logs"; UTXO_SNAPSHOT_PATH="$BASE_DIR/utxo-880000.dat"; \
     3(for c in $COMMITS; do git fetch origin $c -q && git log -1 --pretty=format:'%h %s' $c || exit 1; done) && \
     4for DBCACHE in 450 4500 45000; do \
     5  hyperfine \
     6  --sort 'command' \
     7  --runs 5 \
     8  --export-json "$BASE_DIR/assumeutxo-${COMMITS// /-}-$DBCACHE-$CC.json" \
     9  --parameter-list COMMIT ${COMMITS// /,} \
    10  --prepare "killall bitcoind; rm -rf $DATA_DIR/chainstate $DATA_DIR/chainstate_snapshot $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard; \
    11    cmake -B build -DCMAKE_BUILD_TYPE=Release -DENABLE_WALLET=OFF && cmake --build build -j$(nproc) --target bitcoind --target bitcoin-cli && \
    12    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=1 -printtoconsole=0; \
    13    ./build/bin/bitcoind -datadir=$DATA_DIR -daemon -blocksonly=1 -connect=0 -dbcache=$DBCACHE -printtoconsole=0; sleep 10" \
    14  --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-assumeutxo-{COMMIT}-dbcache-$DBCACHE-$(date +%s).log; build/bin/bitcoin-cli -datadir=$DATA_DIR stop; sleep 10; killall bitcoind; sleep 10;" \
    15  "COMPILER=$CC DBCACHE=$DBCACHE COMMIT={COMMIT} ./build/bin/bitcoin-cli -datadir=$DATA_DIR -rpcclienttimeout=0 loadtxoutset $UTXO_SNAPSHOT_PATH"; \
    16done
    

    156f9913a2 test: assert CScript allocation characteristics a2a3e0cfd8 store P2WSH/P2TR/P2PK scripts inline

     0Benchmark 1: COMPILER=gcc DBCACHE=450 COMMIT=156f9913a26598009c8356c34548002e6a6aba02 ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
     1  Time (mean ± σ):     949.219 s ± 24.273 s    [User: 0.002 s, System: 0.001 s]
     2  Range (min  max):   919.208 s  969.841 s    5 runs
     3
     4Benchmark 2: COMPILER=gcc DBCACHE=450 COMMIT=a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7 ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
     5  Time (mean ± σ):     923.687 s ± 15.431 s    [User: 0.001 s, System: 0.001 s]
     6  Range (min  max):   909.239 s  948.855 s    5 runs
     7
     8Relative speed comparison
     9        1.03 ±  0.03  COMPILER=gcc DBCACHE=450 COMMIT=156f9913a26598009c8356c34548002e6a6aba02 ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
    10        1.00          COMPILER=gcc DBCACHE=450 COMMIT=a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7 ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
    

    156f9913a2 test: assert CScript allocation characteristics a2a3e0cfd8 store P2WSH/P2TR/P2PK scripts inline

     0Benchmark 1: COMPILER=gcc DBCACHE=4500 COMMIT=156f9913a26598009c8356c34548002e6a6aba02 ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
     1  Time (mean ± σ):     778.101 s ± 14.503 s    [User: 0.002 s, System: 0.001 s]
     2  Range (min  max):   765.670 s  801.459 s    5 runs
     3
     4Benchmark 2: COMPILER=gcc DBCACHE=4500 COMMIT=a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7 ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
     5  Time (mean ± σ):     718.500 s ± 17.941 s    [User: 0.001 s, System: 0.001 s]
     6  Range (min  max):   693.273 s  735.929 s    5 runs
     7
     8Relative speed comparison
     9        1.08 ±  0.03  COMPILER=gcc DBCACHE=4500 COMMIT=156f9913a26598009c8356c34548002e6a6aba02 ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
    10        1.00          COMPILER=gcc DBCACHE=4500 COMMIT=a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7 ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
    

    156f9913a2 test: assert CScript allocation characteristics a2a3e0cfd8 store P2WSH/P2TR/P2PK scripts inline

     0Benchmark 1: COMPILER=gcc DBCACHE=45000 COMMIT=156f9913a26598009c8356c34548002e6a6aba02 ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
     1  Time (mean ± σ):     567.985 s ±  9.220 s    [User: 0.001 s, System: 0.001 s]
     2  Range (min  max):   556.441 s  578.439 s    5 runs
     3
     4Benchmark 2: COMPILER=gcc DBCACHE=45000 COMMIT=a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7 ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
     5  Time (mean ± σ):     535.541 s ± 11.944 s    [User: 0.001 s, System: 0.001 s]
     6  Range (min  max):   520.044 s  549.775 s    5 runs
     7
     8Relative speed comparison
     9        1.06 ±  0.03  COMPILER=gcc DBCACHE=45000 COMMIT=156f9913a26598009c8356c34548002e6a6aba02 ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
    10        1.00          COMPILER=gcc DBCACHE=45000 COMMIT=a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7 ./build/bin/bitcoin-cli -datadir=/mnt/my_storage/ShallowBitcoinData -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat
    

    Reindex-chainstate

     0COMMITS="156f9913a26598009c8356c34548002e6a6aba02 a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7"; \
     1STOP_HEIGHT=888888; DBCACHE=450; \
     2CC=gcc; CXX=g++; \
     3BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
     4(for c in $COMMITS; do git fetch origin $c -q && git log -1 --pretty=format:'%h %s' $c || exit 1; done) && \
     5hyperfine \
     6  --sort 'command' \
     7  --runs 2 \
     8  --export-json "$BASE_DIR/rdx-${COMMITS// /-}-$STOP_HEIGHT-$DBCACHE-$CC.json" \
     9  --parameter-list COMMIT ${COMMITS// /,} \
    10  --prepare "killall bitcoind; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard; \
    11    cmake -B build -DCMAKE_BUILD_TYPE=Release -DENABLE_WALLET=OFF && cmake --build build -j$(nproc) --target bitcoind && \
    12    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP_HEIGHT -dbcache=5000 -printtoconsole=0; sleep 100" \
    13  --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
    14  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP_HEIGHT -dbcache=$DBCACHE -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0"
    

    156f9913a2 test: assert CScript allocation characteristics a2a3e0cfd8 store P2WSH/P2TR/P2PK scripts inline

     0Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=450 -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0 (COMMIT = 156f9913a26598009c8356c34548002e6a6aba02)
     1  Time (mean ± σ):     19748.586 s ±  3.502 s    [User: 35991.245 s, System: 2756.673 s]
     2  Range (min  max):   19746.110 s  19751.062 s    2 runs
     3
     4Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=450 -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0 (COMMIT = a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7)
     5  Time (mean ± σ):     19333.189 s ±  6.061 s    [User: 35338.722 s, System: 2733.049 s]
     6  Range (min  max):   19328.903 s  19337.474 s    2 runs
     7
     8Relative speed comparison
     9        1.02 ±  0.00  COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=450 -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0 (COMMIT = 156f9913a26598009c8356c34548002e6a6aba02)
    10        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=450 -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0 (COMMIT = a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7)
    

    Full IBD

     0COMMITS="156f9913a26598009c8356c34548002e6a6aba02 a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7"; \
     1STOP_HEIGHT=888888; DBCACHE=450; \
     2CC=gcc; CXX=g++; \
     3BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
     4(for c in $COMMITS; do git fetch origin $c -q && git log -1 --pretty=format:'%h %s' $c || exit 1; done) && \
     5hyperfine \
     6  --sort 'command' \
     7  --runs 2 \
     8  --export-json "$BASE_DIR/rdx-${COMMITS// /-}-$STOP_HEIGHT-$DBCACHE-$CC.json" \
     9  --parameter-list COMMIT ${COMMITS// /,} \
    10  --prepare "killall bitcoind; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard; \
    11    cmake -B build -DCMAKE_BUILD_TYPE=Release -DENABLE_WALLET=OFF && cmake --build build -j$(nproc) --target bitcoind && \
    12    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP_HEIGHT -dbcache=5000 -printtoconsole=0; sleep 100" \
    13  --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
    14  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP_HEIGHT -dbcache=$DBCACHE -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0"
    

    156f9913a2 test: assert CScript allocation characteristics a2a3e0cfd8 store P2WSH/P2TR/P2PK scripts inline

     0Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=450 -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0 (COMMIT = 156f9913a26598009c8356c34548002e6a6aba02)
     1  Time (mean ± σ):     39873.924 s ± 637.446 s    [User: 35537.079 s, System: 2668.529 s]
     2  Range (min  max):   39423.182 s  40324.667 s    2 runs
     3
     4Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=450 -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0 (COMMI$ = a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7)
     5  Time (mean ± σ):     38682.594 s ±  7.327 s    [User: 34211.447 s, System: 2618.127 s]
     6  Range (min  max):   38677.412 s  38687.775 s    2 runs
     7
     8Relative speed comparison
     9        1.03 ±  0.02  COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=450 -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0 (COMMIT = 156f9913a26598009c8356c34548002e6a6aba02)
    10        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=450 -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0 (COMMIT = a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7)
    
     0COMMITS="847a891cdfdf9cfd69a77a8688c995e532ea29db dc0c79a46b5663eddc0a4365b9b7de22cf3fefae"; \
     1STOP_HEIGHT=888888; DBCACHE=45000; \
     2CC=gcc; CXX=g++; \
     3BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
     4(for c in $COMMITS; do git fetch origin $c -q && git log -1 --pretty=format:'%h %s' $c || exit 1; done) && \
     5hyperfine \
     6  --sort 'command' \
     7  --runs 2 \
     8  --export-json "$BASE_DIR/ibd-${COMMITS// /-}-$STOP_HEIGHT-$DBCACHE-$CC.json" \
     9  --parameter-list COMMIT ${COMMITS// /,} \
    10  --prepare "killall bitcoind; rm -rf $DATA_DIR/*; git checkout {COMMIT}; git clean -fxd; git reset --hard; \
    11    cmake -B build -DCMAKE_BUILD_TYPE=Release -DENABLE_WALLET=OFF && cmake --build build -j$(nproc) --target bitcoind && \
    12    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=1 -printtoconsole=0; sleep 100" \
    13  --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
    14  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP_HEIGHT -dbcache=$DBCACHE -blocksonly -printtoconsole=0"
    

    847a891cdf test: assert CScript allocation characteristics dc0c79a46b store P2WSH/P2TR/P2PK scripts inline

     0Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=45000 -blocksonly -printtoconsole=0 (COMMIT = 847a891cdfdf9cfd69a77a8688c995e532ea29db)
     1  Time (mean ± σ):     39061.975 s ± 366.257 s    [User: 37221.558 s, System: 1941.817 s]
     2  Range (min  max):   38802.992 s  39320.958 s    2 runs
     3
     4Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=45000 -blocksonly -printtoconsole=0 (COMMIT = dc0c79a46b5663eddc0a4365b9b7de22cf3fefae)
     5  Time (mean ± σ):     38214.234 s ± 65.671 s    [User: 36851.441 s, System: 2103.990 s]
     6  Range (min  max):   38167.797 s  38260.670 s    2 runs
     7
     8Relative speed comparison
     9        1.02 ±  0.01  COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=45000 -blocksonly -printtoconsole=0 (COMMIT = 847a891cdfdf9cfd69a77a8688c995e532ea29db)
    10        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=45000 -blocksonly -printtoconsole=0 (COMMIT = dc0c79a46b5663eddc0a4365b9b7de22cf3fefae)
    

    Memory

    Note that this was measured for prevector<34>, but should be basically the same for the current 36 as well.

     0-------------------------------------------------------------------------------
     1  n       time(ms)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
     2--------------------------------------------------------------------------------
     3  0              0                0                0             0            0
     4  1        142,755    4,881,866,480    4,646,778,126   235,088,354            0
     595.18% (4,646,778,126B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
     6->73.76% (3,600,809,984B) 0x5502B7: CCoinsViewCache::EmplaceCoinInternalDANGER(COutPoint&&, Coin&&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
     7| ->73.76% (3,600,809,984B) 0x49DDCF: ChainstateManager::PopulateAndValidateSnapshot(Chainstate&, AutoFile&, node::SnapshotMetadata const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
     8|   ->73.76% (3,600,809,984B) 0x49F3CA: ChainstateManager::ActivateSnapshot(AutoFile&, node::SnapshotMetadata const&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
     9|     ->73.76% (3,600,809,984B) 0x367A0F: loadtxoutset()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)
    10|       ->73.76% (3,600,809,984B) 0x3680F6: std::_Function_handler<UniValue (RPCHelpMan const&, JSONRPCRequest const&), loadtxoutset()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)
    11|         ->73.76% (3,600,809,984B) 0x60403B: RPCHelpMan::HandleRequest(JSONRPCRequest const&) const (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    12|           ->73.76% (3,600,809,984B) 0x373359: CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)
    13|             ->73.76% (3,600,809,984B) 0x425F6C: ExecuteCommand(CRPCCommand const&, JSONRPCRequest const&, UniValue&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    14|               ->73.76% (3,600,809,984B) 0x427373: CRPCTable::execute(JSONRPCRequest const&) const (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    15|                 ->73.76% (3,600,809,984B) 0x4274C6: JSONRPCExec(JSONRPCRequest const&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    16|                   ->73.76% (3,600,809,984B) 0x4FF1C3: HTTPReq_JSONRPC(std::any const&, HTTPRequest*) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    17|                     ->73.76% (3,600,809,984B) 0x50CA81: WorkQueue<HTTPClosure>::Run() (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    18|                       ->73.76% (3,600,809,984B) 0x505F06: HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    19|                         ->73.76% (3,600,809,984B) 0x49A6DB3: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33)
    20|                           ->73.76% (3,600,809,984B) 0x4CEBAA3: start_thread (pthread_create.c:447)
    21|                             ->73.76% (3,600,809,984B) 0x4D78A33: clone (clone.S:100)
    22|
    23->08.19% (399,758,776B) 0x5522AC: std::__detail::_Hashtable_alloc<PoolAllocator<std::__detail::_Hash_node<std::pair<COutPoint const, CCoinsCacheEntry>, false>, 144ul, 8ul> >::_M_allocate_buckets(unsigned long) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    24| ->08.19% (399,758,776B) 0x5526AD: std::_Hashtable<COutPoint, std::pair<COutPoint const, CCoinsCacheEntry>, PoolAllocator<std::pair<COutPoint const, CCoinsCacheEntry>, 144ul, 8ul>, std::__detail::_Select1st, std::equal_to<COutPoint>, SaltedOutpointHasher, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_rehash(unsigned long, unsigned long const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    25|   ->08.19% (399,758,776B) 0x5528D5: std::_Hashtable<COutPoint, std::pair<COutPoint const, CCoinsCacheEntry>, PoolAllocator<std::pair<COutPoint const, CCoinsCacheEntry>, 144ul, 8ul>, std::__detail::_Select1st, std::equal_to<COutPoint>, SaltedOutpointHasher, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node<std::pair<COutPoint const, CCoinsCacheEntry>, false>*, unsigned long) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    26|     ->08.19% (399,758,776B) 0x550219: CCoinsViewCache::EmplaceCoinInternalDANGER(COutPoint&&, Coin&&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    27|       ->08.19% (399,758,776B) 0x49DDCF: ChainstateManager::PopulateAndValidateSnapshot(Chainstate&, AutoFile&, node::SnapshotMetadata const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    28|         ->08.19% (399,758,776B) 0x49F3CA: ChainstateManager::ActivateSnapshot(AutoFile&, node::SnapshotMetadata const&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    29|           ->08.19% (399,758,776B) 0x367A0F: loadtxoutset()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)
    30|             ->08.19% (399,758,776B) 0x3680F6: std::_Function_handler<UniValue (RPCHelpMan const&, JSONRPCRequest const&), loadtxoutset()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)
    31|               ->08.19% (399,758,776B) 0x60403B: RPCHelpMan::HandleRequest(JSONRPCRequest const&) const (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    32|                 ->08.19% (399,758,776B) 0x373359: CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)
    33|                   ->08.19% (399,758,776B) 0x425F6C: ExecuteCommand(CRPCCommand const&, JSONRPCRequest const&, UniValue&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    34|                     ->08.19% (399,758,776B) 0x427373: CRPCTable::execute(JSONRPCRequest const&) const (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    35|                       ->08.19% (399,758,776B) 0x4274C6: JSONRPCExec(JSONRPCRequest const&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    36|                         ->08.19% (399,758,776B) 0x4FF1C3: HTTPReq_JSONRPC(std::any const&, HTTPRequest*) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    37|                           ->08.19% (399,758,776B) 0x50CA81: WorkQueue<HTTPClosure>::Run() (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    38|                             ->08.19% (399,758,776B) 0x505F06: HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    39|                               ->08.19% (399,758,776B) 0x49A6DB3: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33)
    40|                                 ->08.19% (399,758,776B) 0x4CEBAA3: start_thread (pthread_create.c:447)
    41|                                   ->08.19% (399,758,776B) 0x4D78A33: clone (clone.S:100)
    42|
    43->07.53% (367,723,723B) 0x2F508F: prevector<28u, unsigned char, unsigned int, int>::change_capacity(unsigned int) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    44| ->07.53% (367,723,579B) 0x4A5D3F: prevector<28u, unsigned char, unsigned int, int>::resize(unsigned int) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    45| | ->07.52% (367,274,326B) 0x49DF47: ChainstateManager::PopulateAndValidateSnapshot(Chainstate&, AutoFile&, node::SnapshotMetadata const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    46| | | ->07.52% (367,274,326B) 0x49F3CA: ChainstateManager::ActivateSnapshot(AutoFile&, node::SnapshotMetadata const&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    47| | |   ->07.52% (367,274,326B) 0x367A0F: loadtxoutset()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)
    48| | |     ->07.52% (367,274,326B) 0x3680F6: std::_Function_handler<UniValue (RPCHelpMan const&, JSONRPCRequest const&), loadtxoutset()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)
    49| | |       ->07.52% (367,274,326B) 0x60403B: RPCHelpMan::HandleRequest(JSONRPCRequest const&) const (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    50| | |         ->07.52% (367,274,326B) 0x373359: CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)
    51| | |           ->07.52% (367,274,326B) 0x425F6C: ExecuteCommand(CRPCCommand const&, JSONRPCRequest const&, UniValue&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    52| | |             ->07.52% (367,274,326B) 0x427373: CRPCTable::execute(JSONRPCRequest const&) const (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    53| | |               ->07.52% (367,274,326B) 0x4274C6: JSONRPCExec(JSONRPCRequest const&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    54| | |                 ->07.52% (367,274,326B) 0x4FF1C3: HTTPReq_JSONRPC(std::any const&, HTTPRequest*) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    55| | |                   ->07.52% (367,274,326B) 0x50CA81: WorkQueue<HTTPClosure>::Run() (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    56| | |                     ->07.52% (367,274,326B) 0x505F06: HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    57| | |                       ->07.52% (367,274,326B) 0x49A6DB3: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33)
    58| | |                         ->07.52% (367,274,326B) 0x4CEBAA3: start_thread (pthread_create.c:447)
    59| | |                           ->07.52% (367,274,326B) 0x4D78A33: clone (clone.S:100)
    60| | |
    61| | ->00.01% (449,253B) in 1+ places, all below ms_print's threshold (01.00%)
    62| |
    63| ->00.00% (144B) in 1+ places, all below ms_print's threshold (01.00%)
    64|
    65->03.64% (177,834,800B) 0x286989: node::BlockManager::InsertBlockIndex(uint256 const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    66| ->01.82% (88,980,600B) 0x28ABA8: kernel::BlockTreeDB::LoadBlockIndexGuts(Consensus::Params const&, std::function<CBlockIndex* (uint256 const&)>, util::SignalInterrupt const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    67| | ->01.82% (88,980,600B) 0x28AFA8: node::BlockManager::LoadBlockIndex(std::optional<uint256> const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    68| |   ->01.82% (88,980,600B) 0x28B9FE: node::BlockManager::LoadBlockIndexDB(std::optional<uint256> const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    69| |     ->01.82% (88,980,600B) 0x48329F: ChainstateManager::LoadBlockIndex() (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    70| |       ->01.82% (88,980,600B) 0x29ABA7: node::CompleteChainstateInitialization(ChainstateManager&, node::ChainstateLoadOptions const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    71| |         ->01.82% (88,980,600B) 0x29BB79: node::LoadChainstate(ChainstateManager&, kernel::CacheSizes const&, node::ChainstateLoadOptions const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    72| |           ->01.82% (88,980,600B) 0x1EE0BE: InitAndLoadChainstate(node::NodeContext&, bool, bool, kernel::CacheSizes const&, ArgsManager const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    73| |             ->01.82% (88,980,600B) 0x1F3C54: AppInitMain(node::NodeContext&, interfaces::BlockAndHeaderTipInfo*) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    74| |               ->01.82% (88,980,600B) 0x1BE757: main (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    75| |
    76| ->01.82% (88,854,200B) 0x28ABC4: kernel::BlockTreeDB::LoadBlockIndexGuts(Consensus::Params const&, std::function<CBlockIndex* (uint256 const&)>, util::SignalInterrupt const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    77|   ->01.82% (88,854,200B) 0x28AFA8: node::BlockManager::LoadBlockIndex(std::optional<uint256> const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    78|     ->01.82% (88,854,200B) 0x28B9FE: node::BlockManager::LoadBlockIndexDB(std::optional<uint256> const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    79|       ->01.82% (88,854,200B) 0x48329F: ChainstateManager::LoadBlockIndex() (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    80|         ->01.82% (88,854,200B) 0x29ABA7: node::CompleteChainstateInitialization(ChainstateManager&, node::ChainstateLoadOptions const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    81|           ->01.82% (88,854,200B) 0x29BB79: node::LoadChainstate(ChainstateManager&, kernel::CacheSizes const&, node::ChainstateLoadOptions const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    82|             ->01.82% (88,854,200B) 0x1EE0BE: InitAndLoadChainstate(node::NodeContext&, bool, bool, kernel::CacheSizes const&, ArgsManager const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    83|               ->01.82% (88,854,200B) 0x1F3C54: AppInitMain(node::NodeContext&, interfaces::BlockAndHeaderTipInfo*) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    84|                 ->01.82% (88,854,200B) 0x1BE757: main (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    85|
    86->02.06% (100,650,843B) in 1084 places, all below massif's threshold (1.00%)
    
     0--------------------------------------------------------------------------------
     1  n       time(ms)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
     2--------------------------------------------------------------------------------
     3  0              0                0                0             0            0
     4  1         85,674    3,026,857,896    3,008,671,214    18,186,682            0
     599.40% (3,008,671,214B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
     6->84.57% (2,559,836,160B) 0x5511A7: CCoinsViewCache::EmplaceCoinInternalDANGER(COutPoint&&, Coin&&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
     7| ->84.57% (2,559,836,160B) 0x49E98F: ChainstateManager::PopulateAndValidateSnapshot(Chainstate&, AutoFile&, node::SnapshotMetadata const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
     8|   ->84.57% (2,559,836,160B) 0x49FF9A: ChainstateManager::ActivateSnapshot(AutoFile&, node::SnapshotMetadata const&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
     9|     ->84.57% (2,559,836,160B) 0x367FAF: loadtxoutset()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)
    10|       ->84.57% (2,559,836,160B) 0x368686: std::_Function_handler<UniValue (RPCHelpMan const&, JSONRPCRequest const&), loadtxoutset()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)
    11|         ->84.57% (2,559,836,160B) 0x60446B: RPCHelpMan::HandleRequest(JSONRPCRequest const&) const (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    12|           ->84.57% (2,559,836,160B) 0x373939: CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)
    13|             ->84.57% (2,559,836,160B) 0x4269CC: ExecuteCommand(CRPCCommand const&, JSONRPCRequest const&, UniValue&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    14|               ->84.57% (2,559,836,160B) 0x427DD3: CRPCTable::execute(JSONRPCRequest const&) const (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    15|                 ->84.57% (2,559,836,160B) 0x427F26: JSONRPCExec(JSONRPCRequest const&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    16|                   ->84.57% (2,559,836,160B) 0x4FFED3: HTTPReq_JSONRPC(std::any const&, HTTPRequest*) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    17|                     ->84.57% (2,559,836,160B) 0x50D791: WorkQueue<HTTPClosure>::Run() (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    18|                       ->84.57% (2,559,836,160B) 0x506C16: HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    19|                         ->84.57% (2,559,836,160B) 0x49A6DB3: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33)
    20|                           ->84.57% (2,559,836,160B) 0x4CEBAA3: start_thread (pthread_create.c:447)
    21|                             ->84.57% (2,559,836,160B) 0x4D78A33: clone (clone.S:100)
    22|
    23->06.50% (196,857,944B) 0x5531FC: std::__detail::_Hashtable_alloc<PoolAllocator<std::__detail::_Hash_node<std::pair<COutPoint const, CCoinsCacheEntry>, false>, 152ul, 8ul> >::_M_allocate_buckets(unsigned long) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    24| ->06.50% (196,857,944B) 0x5535FD: std::_Hashtable<COutPoint, std::pair<COutPoint const, CCoinsCacheEntry>, PoolAllocator<std::pair<COutPoint const, CCoinsCacheEntry>, 152ul, 8ul>, std::__detail::_Select1st, std::equal_to<COutPoint>, SaltedOutpointHasher, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_rehash(unsigned long, unsigned long const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    25|   ->06.50% (196,857,944B) 0x553825: std::_Hashtable<COutPoint, std::pair<COutPoint const, CCoinsCacheEntry>, PoolAllocator<std::pair<COutPoint const, CCoinsCacheEntry>, 152ul, 8ul>, std::__detail::_Select1st, std::equal_to<COutPoint>, SaltedOutpointHasher, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node<std::pair<COutPoint const, CCoinsCacheEntry>, false>*, unsigned long) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    26|     ->06.50% (196,857,944B) 0x551107: CCoinsViewCache::EmplaceCoinInternalDANGER(COutPoint&&, Coin&&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    27|       ->06.50% (196,857,944B) 0x49E98F: ChainstateManager::PopulateAndValidateSnapshot(Chainstate&, AutoFile&, node::SnapshotMetadata const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    28|         ->06.50% (196,857,944B) 0x49FF9A: ChainstateManager::ActivateSnapshot(AutoFile&, node::SnapshotMetadata const&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    29|           ->06.50% (196,857,944B) 0x367FAF: loadtxoutset()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)
    30|             ->06.50% (196,857,944B) 0x368686: std::_Function_handler<UniValue (RPCHelpMan const&, JSONRPCRequest const&), loadtxoutset()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)
    31|               ->06.50% (196,857,944B) 0x60446B: RPCHelpMan::HandleRequest(JSONRPCRequest const&) const (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    32|                 ->06.50% (196,857,944B) 0x373939: CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)
    33|                   ->06.50% (196,857,944B) 0x4269CC: ExecuteCommand(CRPCCommand const&, JSONRPCRequest const&, UniValue&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    34|                     ->06.50% (196,857,944B) 0x427DD3: CRPCTable::execute(JSONRPCRequest const&) const (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    35|                       ->06.50% (196,857,944B) 0x427F26: JSONRPCExec(JSONRPCRequest const&, bool) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    36|                         ->06.50% (196,857,944B) 0x4FFED3: HTTPReq_JSONRPC(std::any const&, HTTPRequest*) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    37|                           ->06.50% (196,857,944B) 0x50D791: WorkQueue<HTTPClosure>::Run() (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    38|                             ->06.50% (196,857,944B) 0x506C16: HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    39|                               ->06.50% (196,857,944B) 0x49A6DB3: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33)
    40|                                 ->06.50% (196,857,944B) 0x4CEBAA3: start_thread (pthread_create.c:447)
    41|                                   ->06.50% (196,857,944B) 0x4D78A33: clone (clone.S:100)
    42|
    43->05.88% (177,834,800B) 0x286859: node::BlockManager::InsertBlockIndex(uint256 const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    44| ->02.94% (88,980,600B) 0x28AA78: kernel::BlockTreeDB::LoadBlockIndexGuts(Consensus::Params const&, std::function<CBlockIndex* (uint256 const&)>, util::SignalInterrupt const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    45| | ->02.94% (88,980,600B) 0x28AE78: node::BlockManager::LoadBlockIndex(std::optional<uint256> const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    46| |   ->02.94% (88,980,600B) 0x28B8CE: node::BlockManager::LoadBlockIndexDB(std::optional<uint256> const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    47| |     ->02.94% (88,980,600B) 0x483E9F: ChainstateManager::LoadBlockIndex() (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    48| |       ->02.94% (88,980,600B) 0x29ABB7: node::CompleteChainstateInitialization(ChainstateManager&, node::ChainstateLoadOptions const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    49| |         ->02.94% (88,980,600B) 0x29BB89: node::LoadChainstate(ChainstateManager&, kernel::CacheSizes const&, node::ChainstateLoadOptions const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    50| |           ->02.94% (88,980,600B) 0x1EE0EE: InitAndLoadChainstate(node::NodeContext&, bool, bool, kernel::CacheSizes const&, ArgsManager const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    51| |             ->02.94% (88,980,600B) 0x1F3C84: AppInitMain(node::NodeContext&, interfaces::BlockAndHeaderTipInfo*) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    52| |               ->02.94% (88,980,600B) 0x1BE717: main (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    53| |
    54| ->02.94% (88,854,200B) 0x28AA94: kernel::BlockTreeDB::LoadBlockIndexGuts(Consensus::Params const&, std::function<CBlockIndex* (uint256 const&)>, util::SignalInterrupt const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    55|   ->02.94% (88,854,200B) 0x28AE78: node::BlockManager::LoadBlockIndex(std::optional<uint256> const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    56|     ->02.94% (88,854,200B) 0x28B8CE: node::BlockManager::LoadBlockIndexDB(std::optional<uint256> const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    57|       ->02.94% (88,854,200B) 0x483E9F: ChainstateManager::LoadBlockIndex() (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    58|         ->02.94% (88,854,200B) 0x29ABB7: node::CompleteChainstateInitialization(ChainstateManager&, node::ChainstateLoadOptions const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    59|           ->02.94% (88,854,200B) 0x29BB89: node::LoadChainstate(ChainstateManager&, kernel::CacheSizes const&, node::ChainstateLoadOptions const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    60|             ->02.94% (88,854,200B) 0x1EE0EE: InitAndLoadChainstate(node::NodeContext&, bool, bool, kernel::CacheSizes const&, ArgsManager const&) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    61|               ->02.94% (88,854,200B) 0x1F3C84: AppInitMain(node::NodeContext&, interfaces::BlockAndHeaderTipInfo*) (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    62|                 ->02.94% (88,854,200B) 0x1BE717: main (in /mnt/my_storage/bitcoin/build/bin/bitcoind)
    63|
    

    Diffing the two reveals a dramatic difference: prevector<28u, unsigned char, unsigned int, int>::change_capacity is 07.53% of prevector<28>, but is completely missing in prevector<34> since the P2WSH/P2TR scripts are now allocated inline and don’t need any capacity change anymore.

    The other difference visible directly in these peak snapshot header lines is the extra-heap(B):

    • prevector<28> peaks at ~235 MiB of extra-heap.
    • prevector<34> peaks at only ~21 MiB of extra-heap.

    These explain the 7% speedup for large memory use cases: elimination of millions of small, separate heap allocations triggered when CScripts (like P2WSH/P2TR) exceed the 28-byte inline limit, each incurring overhead.

  5. l0rinc renamed this:
    [IBD] prevector: allocate `P2WSH` and `P2TR` scripts on stack as well
    [IBD] prevector: allocate `P2WSH`/`P2TR`/`P2PK` scripts on stack
    on Apr 15, 2025
  6. l0rinc force-pushed on Apr 15, 2025
  7. l0rinc force-pushed on Apr 15, 2025
  8. DrahtBot added the label CI failed on Apr 15, 2025
  9. DrahtBot commented at 9:18 pm on April 15, 2025: contributor

    🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/40612214686

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  10. DrahtBot removed the label CI failed on Apr 15, 2025
  11. l0rinc marked this as ready for review on Apr 15, 2025
  12. l0rinc force-pushed on Apr 16, 2025
  13. l0rinc force-pushed on Apr 16, 2025
  14. DrahtBot added the label CI failed on Apr 16, 2025
  15. DrahtBot commented at 8:01 am on April 16, 2025: contributor

    🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/40633401176

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  16. l0rinc force-pushed on Apr 16, 2025
  17. DrahtBot removed the label CI failed on Apr 16, 2025
  18. achow101 commented at 6:27 pm on April 16, 2025: member
    Since this results in more flushes, I think we should wait for the composition of the UTXO set to change to the point that this provides an improvement for nodes with default dbcache.
  19. l0rinc commented at 7:46 pm on April 16, 2025: contributor
    Thanks for the idea and comment @achow101! I’ll measure IBD with default settings after the benchmarking servers free up, but so far it seems that there’s barely any slowdown (if any) for default dbcache and a measurable speedup for larger memory settings. So we might already be at the point where it’s either a neutral or a net positive change. And given that we have a few other PRs in the queue that speed up small dbcache IBDs as well (see #32043), we can cover all important scenarios via multiple focused changes.
  20. l0rinc force-pushed on Apr 17, 2025
  21. l0rinc commented at 7:55 am on April 18, 2025: contributor

    A few previous measurements were done on 34 byte prevector, the latest version bumped it to 36 (which included P2PK scripts inline as well). I’ve rerun the AssumeUTXO, reindex-chainstate and IBD benchmarks (both on SSD and HDD, since they have different performance profiles) and couldn’t find a single instance where master was any faster.

    this provides an improvement for nodes with default dbcache @achow101, what other usecases do you think I should measure to make sure this is indeed the case?

  22. in src/bench/prevector.cpp:33 in a2a3e0cfd8 outdated
    32-        t0.resize(28);
    33-        t1.resize(29);
    34+        prevector<36, T> t0;
    35+        prevector<36, T> t1;
    36+        t0.resize(36);
    37+        t1.resize(37);
    


    kuegi commented at 12:35 pm on April 18, 2025:
    maybe a “stupid” question, but why not use a constant for the 36 in this file too?

    l0rinc commented at 7:18 pm on April 19, 2025:
    I could, of course, for now I was going for simplifying the diff, not make the code more readable. I can of course do that later, but for now I’m looking for the simplest diff which demonstrates the problem and the proposed solution, and after I’m sure there’s general support for it, we can sneak in code refactors as well.

    l0rinc commented at 11:13 am on April 22, 2025:
    Thanks, extracted to a separate commit - as suggested by @luke-jr as well - added you both as co-authors.
  23. sipa commented at 7:31 pm on April 19, 2025: member
    Nit: in title/description of the PR, I don’t think it’s accurate to say “on the stack”. While there are occasionally CScript objects created on the stack, the vast majority is inside CCoinsCacheDb, where they are still heap-allocated. The benefit of the change here is that some of these would no longer need a separate heap allocation per CScript.
  24. l0rinc renamed this:
    [IBD] prevector: allocate `P2WSH`/`P2TR`/`P2PK` scripts on stack
    [IBD] prevector: store `P2WSH`/`P2TR`/`P2PK` scripts inline
    on Apr 19, 2025
  25. l0rinc force-pushed on Apr 19, 2025
  26. l0rinc commented at 7:39 pm on April 19, 2025: contributor
    Thank you, I have used inline/stack interchangeably - this is a lot better, appreciate the correction.
  27. l0rinc force-pushed on Apr 19, 2025
  28. l0rinc commented at 7:18 am on April 20, 2025: contributor

    Finished a full reindex-chainstate with default dbcache=450 on HDD

     0COMMITS="156f9913a26598009c8356c34548002e6a6aba02 a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7"; \
     1STOP_HEIGHT=888888; DBCACHE=450; \
     2CC=gcc; CXX=g++; \
     3BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
     4(for c in $COMMITS; do git fetch origin $c -q && git log -1 --pretty=format:'%h %s' $c || exit 1; done) && \
     5hyperfine \
     6  --sort 'command' \
     7  --runs 2 \
     8  --export-json "$BASE_DIR/rdx-${COMMITS// /-}-$STOP_HEIGHT-$DBCACHE-$CC.json" \
     9  --parameter-list COMMIT ${COMMITS// /,} \
    10  --prepare "killall bitcoind; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard; \
    11    cmake -B build -DCMAKE_BUILD_TYPE=Release -DENABLE_WALLET=OFF && cmake --build build -j$(nproc) --target bitcoind && \
    12    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP_HEIGHT -dbcache=5000 -printtoconsole=0; sleep 100" \
    13  --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
    14  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP_HEIGHT -dbcache=$DBCACHE -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0"
    

    156f9913a2 test: assert CScript allocation characteristics a2a3e0cfd8 store P2WSH/P2TR/P2PK scripts inline

     0Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=450 -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0 (COMMIT = 156f9913a26598009c8356c34548002e6a6aba02)
     1  Time (mean ± σ):     39873.924 s ± 637.446 s    [User: 35537.079 s, System: 2668.529 s]
     2  Range (min  max):   39423.182 s  40324.667 s    2 runs
     3
     4Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=450 -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0 (COMMI$ = a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7)
     5  Time (mean ± σ):     38682.594 s ±  7.327 s    [User: 34211.447 s, System: 2618.127 s]
     6  Range (min  max):   38677.412 s  38687.775 s    2 runs
     7
     8Relative speed comparison
     9        1.03 ±  0.02  COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=450 -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0 (COMMIT = 156f9913a26598009c8356c34548002e6a6aba02)
    10        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=450 -blocksonly -reindex-chainstate -connect=0 -printtoconsole=0 (COMMIT = a2a3e0cfd8549e3be03fd81e75ebb7527e9345d7)
    
  29. in src/bench/checkqueue.cpp:19 in 43e003f893 outdated
    15@@ -16,7 +16,7 @@
    16 
    17 static const size_t BATCHES = 101;
    18 static const size_t BATCH_SIZE = 30;
    19-static const int PREVECTOR_SIZE = 28;
    20+static const int PREVECTOR_SIZE = 36;
    


    luke-jr commented at 10:10 pm on April 21, 2025:
    Maybe makes sense to move this to a header and reuse it everywhere else?

    l0rinc commented at 11:13 am on April 22, 2025:
    Done, thanks
  30. bitcoin deleted a comment on Apr 22, 2025
  31. l0rinc force-pushed on Apr 22, 2025
  32. in src/bench/prevector.cpp:5 in 1abeadee08 outdated
    1@@ -2,20 +2,22 @@
    2 // Distributed under the MIT software license, see the accompanying
    3 // file COPYING or http://www.opensource.org/licenses/mit-license.php.
    4 
    5-#include <prevector.h>
    


    luke-jr commented at 11:38 pm on April 22, 2025:
    nit: why moving this?

    l0rinc commented at 9:01 am on April 23, 2025:

    Just sorted the imports for consistency - but I see that the corresponding header is sometimes separated, so I’ve reverted this move.

    What do you think about the whole change in general, can you give me a conceptual review?

  33. l0rinc force-pushed on Apr 23, 2025
  34. l0rinc requested review from luke-jr on Apr 23, 2025
  35. DrahtBot added the label CI failed on Apr 28, 2025
  36. l0rinc force-pushed on Apr 29, 2025
  37. l0rinc commented at 9:44 am on April 29, 2025: contributor
    Rebased to cover new tests to resolve new CI failure. Also adjusted the test comments to mention heap/stack less often.
  38. DrahtBot removed the label CI failed on Apr 29, 2025
  39. luke-jr referenced this in commit 83bb9899c6 on Jun 6, 2025
  40. luke-jr referenced this in commit 5d37f3d52d on Jun 6, 2025
  41. luke-jr referenced this in commit 5ff88009a0 on Jun 6, 2025
  42. in src/script/script.h:410 in a5f2ffa951 outdated
    405@@ -406,7 +406,8 @@ class CScriptNum
    406  * Tests in October 2015 showed use of this reduced dbcache memory usage by 23%
    407  *  and made an initial sync 13% faster.
    408  */
    409-typedef prevector<28, unsigned char> CScriptBase;
    410+static constexpr unsigned int PREVECTOR_SIZE{28};
    411+typedef prevector<PREVECTOR_SIZE, unsigned char> CScriptBase;
    


    maflcko commented at 8:45 am on July 21, 2025:
    nit in the first commit: could be using using for new code?

    l0rinc commented at 5:55 pm on July 21, 2025:
    Sure, modernized the definition in a separate commit
  43. in src/script/script.h:409 in a5f2ffa951 outdated
    405@@ -406,7 +406,8 @@ class CScriptNum
    406  * Tests in October 2015 showed use of this reduced dbcache memory usage by 23%
    407  *  and made an initial sync 13% faster.
    408  */
    409-typedef prevector<28, unsigned char> CScriptBase;
    410+static constexpr unsigned int PREVECTOR_SIZE{28};
    


    maflcko commented at 9:03 am on July 21, 2025:
    nit in the first commit: Should this be called script_base_size instead? There are other uses of prevector (CompressedScript, netaddr, …), so defining this globally could be confusing.

    l0rinc commented at 6:10 pm on July 21, 2025:
    Makes sense, thanks
  44. in src/test/script_tests.cpp:1161 in ecc6c07e58 outdated
    1163+    BOOST_CHECK_EQUAL(sizeof(prevector<PREVECTOR_SIZE, unsigned char>), 32);
    1164+    BOOST_CHECK_EQUAL(sizeof(CScriptBase), 32);
    1165+    BOOST_CHECK_EQUAL(sizeof(CScript), 32);
    1166+    BOOST_CHECK_EQUAL(sizeof(CTxOut), 40);
    1167+
    1168+    CKey dummyKey;
    


    maflcko commented at 11:14 am on July 21, 2025:
    nit in ecc6c07e583842a2afe1a8b5bb8ec0bb9f997fdb: Use snake_case for new code?

    l0rinc commented at 6:02 pm on July 21, 2025:
    Done
  45. in src/test/script_tests.cpp:1164 in ecc6c07e58 outdated
    1166+    BOOST_CHECK_EQUAL(sizeof(CTxOut), 40);
    1167+
    1168+    CKey dummyKey;
    1169+    dummyKey.MakeNewKey(true);
    1170+
    1171+    std::vector<std::vector<uint8_t>> dummyVSolutions;
    


    maflcko commented at 11:25 am on July 21, 2025:
    nit in https://github.com/bitcoin/bitcoin/commit/ecc6c07e583842a2afe1a8b5bb8ec0bb9f997fdb: Could add and use a small helper: TxoutType GetTxoutType(const CScript& output_script)? This makes the code shorter below and wraps this dummy.

    l0rinc commented at 6:08 pm on July 21, 2025:
    Done, thanks
  46. l0rinc force-pushed on Jul 21, 2025
  47. l0rinc force-pushed on Jul 21, 2025
  48. l0rinc commented at 6:22 pm on July 21, 2025: contributor
    Thanks, applied most of your comments + a few additional cleanups. The first push was a simple rebase, the second contains the changes.
  49. in src/test/script_tests.cpp:1182 in 3a9c80f7b4 outdated
    1177+        BOOST_CHECK_EQUAL(script.allocated_memory(), 0);
    1178+    }
    1179+
    1180+    // P2WPKH has direct allocation
    1181+    {
    1182+        const auto script{GetScriptForDestination(WitnessV0KeyHash{PKHash{CKeyID{CPubKey{dummy_key.GetPubKey()}.GetID()}}})};
    


    maflcko commented at 11:58 am on July 22, 2025:

    nit in 3a9c80f7b4807e9abd9f2da36e8ee479ecf00d19:

    GetPubKey does not need to be wrapped in CPubKey{}. You could just store a dummy_pubkey for all test cases to re-use.


    l0rinc commented at 6:06 pm on July 22, 2025:
    Thanks, will do this in the next push

    maflcko commented at 8:14 am on July 25, 2025:

    Thanks, will do this in the next push

    I don’t think you did?


    l0rinc commented at 11:37 pm on July 25, 2025:

    I quickly pushed a rebase, thinking I’ll follow with applying the recommendations - but ended up in a CompressedScript rabbithole (which I decided to leave to a follow-up in the end).

    Looks like the extractions and inlines I did here ended up with lots of unnecessary casts, thanks for noticing, cleaned it up in latest push!

  50. maflcko approved
  51. maflcko commented at 12:13 pm on July 22, 2025: member

    I haven’t tested IBD/reindex performance.

    review ACK f7d9f4510d537b280808e1e8e203c9445c8ad4df 🍴

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: review ACK f7d9f4510d537b280808e1e8e203c9445c8ad4df 🍴
    3nXDCykVC7pudYdduUKbJSXeg0XwPp/YhNfksikOYMhOyQXyHf4x9gENBZX6FGMcUav92tl9Hna3Z5RXQAOddDQ==
    
  52. l0rinc commented at 7:06 pm on July 23, 2025: contributor

    I have remeasured -reindex-chainstate speed by running the rebased PR (now that other optimizations were merged) until 900k blocks twice before and twice after, and the speedup is still ~4%:

     0COMMITS="7129c9ea8e950f50bdc56d88c57617c66c90bb8a f7d9f4510d537b280808e1e8e203c9445c8ad4df"; \                     
     1STOP=900000; DBCACHE=5000; \                                                                                                                                                   
     2CC=gcc; CXX=g++; \                                                                                                                                                             
     3BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
     4(echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done; echo "") && \
     5hyperfine \
     6  --sort command \
     7  --runs 2 \
     8  --export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \
     9  --parameter-list COMMIT ${COMMITS// /,} \ 
    10  --prepare "killall bitcoind; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
    11    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release && ninja -C build bitcoind && \
    12    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 10" \
    13  --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
    14  "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0"
    15
    167129c9ea8e Merge bitcoin/bitcoin#32827: mempool: Avoid needless vtx iteration during IBD
    17f7d9f4510d prevector: store P2WSH/P2TR/P2PK scripts inline
    18
    19Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=900000 -dbcache=5000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 7129c9ea8e950f50bdc56d88c57617c66c90bb8a)
    20  Time (mean ± σ):     17108.190 s ± 48.793 s    [User: 30728.280 s, System: 987.269 s]                                                                                        
    21  Range (min  max):   17073.688 s  17142.691 s    2 runs                                                                                                                     
    22                                                                                       
    23Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=900000 -dbcache=5000 -reindex-chainstate -blocksonly -connect=0 -printtocons$le=0 (COMMIT = f7d9f4510d537b280808e1e8e203c9445c8ad4df)
    24  Time (mean ± σ):     16510.352 s ± 69.453 s    [User: 29862.737 s, System: 943.983 s]
    25  Range (min  max):   16461.241 s  16559.463 s    2 runs
    26  
    27Relative speed comparison
    28        1.04 ±  0.01  COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=900000 -dbcache=5000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 7129c9ea8e950f50bdc56d88c57617c66c90bb8a)
    29        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=900000 -dbcache=5000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = f7d9f4510d537b280808e1e8e203c9445c8ad4df)
    

    Edit: To show the exact behavior, I’ve plotted the height-vs-time (and the coins-vs-time) before and after:

    • 7129c9ea8e is master
    • f7d9f4510d is this PR

    Edit2: Added bar charts by 100k blocks as requested by @ajtowns:

  53. in src/test/script_tests.cpp:1163 in f7d9f4510d outdated
    1158+    return Solver(output_script, unused);
    1159+}
    1160+
    1161+BOOST_AUTO_TEST_CASE(script_size_and_capacity_test)
    1162+{
    1163+    BOOST_CHECK_EQUAL(sizeof(prevector<34, uint8_t>), sizeof(prevector<SCRIPT_BASE_SIZE, uint8_t>));
    


    ajtowns commented at 2:32 am on July 24, 2025:

    Could consider adding something like:

    0// SCRIPT_BASE_SIZE should be set to avoid wasting space in padding
    1BOOST_CHECK(sizeof(CScriptBase) != sizeof(prevector<SCRIPT_BASE_SIZE+1, uint8_t>));
    

    It might be nicer to have prevector include a static constexpr size_t STATIC_SIZE = N; member, and use CScriptBase::STATIC_SIZE rather than SCRIPT_BASE_SIZE.


    l0rinc commented at 11:35 pm on July 25, 2025:
    This is indeed a lot nicer, thanks!
  54. ajtowns commented at 2:36 am on July 24, 2025: contributor

    I haven’t investigated in detail why there are so many empty prevectors, can very well be just an unoptimized debug run artifact of all the Coin copies.

    I think it’s just that coins get cleared when they’re spent, so end up as 0 size by the time they’re destructed? Empty scriptSig’s for witness spends might also have some impact.

    I think the tradeoff is something like:

    • spends of p2pk, p2sh, p2pkh coins – these cost 8 more bytes
    • spends of p2wpkh – these cost 16 more bytes (sPK and scriptSig didn’t need an allocation)
    • spends of p2wsh and p2tr – these cost ~48 fewer bytes (save 64 byte allocation on 64bit system, lose 8 bytes for both scriptSig and sPK)
    • spends of nested p2wsh – presumably save ~96 bytes, since the scriptSig would save an allocation, but I’m bundling it in the previous section

    Based on mainnet.observer stats for 2025-05-08, p2wpkh is about 55% of txs, p2tr is about 28%, p2pkh about 13%, p2wsh about 4% and the rest is noise, maybe? Those numbers net out to a saving of ~5.5 bytes per input. If p2wpkh rose from 55% to 80% and p2tr dropped to 20%, that would net to wasting ~3.2 bytes per input.

    So I think this change seems plausible now, and certainly is in line with wanting to recommend the use of 256-bit scriptPubKeys for any sort of collaboratively held utxos (ie multisig via p2wpkh or taproot scripts, or musig via taproot), and to encourage collaboratively held utxos.

    Approach ACK.

  55. ajtowns commented at 2:39 am on July 24, 2025: contributor

    To show the exact behavior, I’ve plotted the height-vs-time (and the coins-vs-time) before and after:

    Probably would be better to bucket the heights and do bar charts, ie: how long did the 1st, 2nd, 3rd, etc 100k blocks take with both approaches. That way it’s easier to see if a large gain for this PR was temporary (eg, due to lots of inscriptions) and is not providing a benefit with current traffic.

  56. luke-jr referenced this in commit 59ccbb4049 on Jul 24, 2025
  57. luke-jr referenced this in commit 430aae56c6 on Jul 24, 2025
  58. luke-jr referenced this in commit ca2576d8cc on Jul 24, 2025
  59. luke-jr referenced this in commit d02dffbd4b on Jul 24, 2025
  60. l0rinc force-pushed on Jul 24, 2025
  61. achow101 commented at 9:55 pm on July 24, 2025: member

    ACK ec0b90e40e21ccb97c57373f7a7665ab9554acec

    In a sample size of 1, this seems to provide a significant improvement in IBD time. It’s surprising to me that it looks like this would have had benefit around the time that Taproot activated as previous measurements hadn’t indicated that.

  62. DrahtBot requested review from maflcko on Jul 24, 2025
  63. DrahtBot requested review from ajtowns on Jul 24, 2025
  64. refactor: extract `STATIC_SIZE` constant to prevector
    Co-authored-by: Anthony Towns <aj@erisian.com.au>
    756da2a994
  65. refactor: modernize `CScriptBase` definition 65ac7f6d4d
  66. l0rinc force-pushed on Jul 25, 2025
  67. l0rinc commented at 11:39 pm on July 25, 2025: contributor
    Thanks for the recommendations, took them and extended the tests slightly to document the current behavior of CompressedScript (unchanged) and uncompressed P2PK as well (also unchanged). I have also rebased to make sure all new test cases are run - ready for re-review.
  68. test: assert `CScript` allocation characteristics
    Verifies that script types are correctly allocated using prevector's direct or indirect storage based on their size:
    
    Direct allocated script types (size ≤ 28 bytes):
    * OP_RETURN (small)
    * P2WPKH
    * P2SH
    * P2PKH
    
    Indirect allocated script types (size > 28 bytes):
    * P2WSH
    * P2TR
    * P2PK
    * MULTISIG (small)
    
    This test provides a baseline for verifying changes to prevector's inline capacity.
    
    The `CHECK_SCRIPT_STATIC_SIZE` and `CHECK_SCRIPT_DYNAMIC_SIZE` macros were added to differentiate the two cases - while preserving the correct source code line in case of failure.
    52121506b2
  69. prevector: store `P2WSH`/`P2TR`/`P2PK` scripts inline
    The current `prevector` size of 28 bytes (chosen to fill the `sizeof(CScript)` aligned size) was introduced in 2015 (https://github.com/bitcoin/bitcoin/pull/6914) before SegWit and TapRoot.
    However, the increasingly common `P2WSH` and `P2TR` scripts are both 34 bytes, and are forced to use heap (re)allocation rather than efficient inline storage.
    
    The core trade-off of this change is to eliminate heap allocations for common 34-36 byte scripts at the cost of increasing the base memory footprint of all `CScript` objects by 8 bytes (while still respecting peak memory usage defined by `-dbcache`).
    
    Increasing the `prevector` size allows these scripts to be stored inline, avoiding extra heap allocations, reducing potential memory fragmentation, and improving performance during cache flushes. Massif analysis confirms a lower stable memory usage after flushing, suggesting the elimination of heap allocations outweighs the larger base size for common workloads.
    
    Due to memory alignment, increasing the `prevector` size to 36 bytes doesn't change the overall `sizeof(CScript)` compared to an increase to 34 bytes, allowing us to include `P2PK` scripts as well at no additional memory cost.
    
    Performance benchmarks for AssumeUTXO load and flush show:
    * Small dbcache (450MB): ~1-3% performance improvement (despite more frequent flushes)
    * Large dbcache (4500MB): ~6-8% performance improvement due to fewer heap allocations (and basically the number of flushes)
    * Very large dbcache (4500MB): ~5-6% performance improvement due to fewer heap allocations (and memory limit not being reached, so there's no memory penalty)
    
    Full IBD and reindex-chainstate with larger `dbcache` values also show an overall ~3-4% speedup.
    
    Co-authored-by: Ava Chow <github@achow101.com>
    Co-authored-by: Andrew Toth <andrewstoth@gmail.com>
    Co-authored-by: maflcko <6399679+maflcko@users.noreply.github.com>
    d5104cfbae
  70. in src/test/script_tests.cpp:1215 in 44fc161634 outdated
    1213         BOOST_CHECK(script.IsPayToWitnessScriptHash());
    1214         BOOST_CHECK_EQUAL(script.size(), 34);
    1215-        BOOST_CHECK_EQUAL(script.capacity(), 34);
    1216-        BOOST_CHECK_EQUAL(script.allocated_memory(), 34);
    1217+        BOOST_CHECK_EQUAL(script.capacity(), CScriptBase::STATIC_SIZE);
    1218+        BOOST_CHECK_EQUAL(script.allocated_memory(), 0);
    


    ajtowns commented at 5:22 am on July 26, 2025:
     0static void check_script_has_static_size(CScript& script, unsigned int size)
     1{
     2    BOOST_CHECK_EQUAL(script.size(), size);
     3    BOOST_CHECK_EQUAL(script.capacity(), CScriptBase::STATIC_SIZE);
     4    BOOST_CHECK_EQUAL(script.allocated_memory(), 0);
     5}
     6
     7static void check_script_has_dynamic_size(CScript& script, unsigned int size)
     8{
     9    BOOST_CHECK_EQUAL(script.size(), size);
    10    BOOST_CHECK_EQUAL(script.capacity(), size);
    11    BOOST_CHECK_EQUAL(script.allocated_memory(), size);
    12}
    

    and

    0        BOOST_CHECK(script.IsPayToWitnessScriptHash());
    1        check_script_has_static_size(script, 34);
    

    ?


    l0rinc commented at 5:24 am on July 26, 2025:
    you think that would be easier to follow? If it fails I would see it logged in check_script_has_dynamic_size - unless I make it a macro.

    l0rinc commented at 5:35 am on July 26, 2025:
    Thanks, pushed, let me know what you think!

    ajtowns commented at 6:35 am on July 26, 2025:

    Can get errors like:

    0test/script_tests.cpp(1246): error: in "script_tests/script_size_and_capacity_test": check script_dynamic_alloc( script, 64, 64 ) has failed for ( CScript(size=67,cap=67), 64, 64 )
    1test/script_tests.cpp(1253): error: in "script_tests/script_size_and_capacity_test": check script_dynamic_alloc( script, 10, 1000 ) has failed for ( CScript(size=71,cap=103), 10, 1000 )
    

    with this approach:

     0--- a/src/test/script_tests.cpp
     1+++ b/src/test/script_tests.cpp
     2@@ -38,6 +38,12 @@
     3
     4 using namespace util::hex_literals;
     5
     6+static std::ostream& operator<<(std::ostream& os, const CScript& script)
     7+{
     8+    os << strprintf("CScript(size=%d,cap=%d)", script.size(), script.capacity());
     9+    return os;
    10+}
    11+
    12 static const unsigned int gFlags = SCRIPT_VERIFY_P2SH | SCRIPT_VERIFY_STRICTENC;
    13
    14 unsigned int ParseScriptFlags(std::string strFlags);
    15@@ -1158,6 +1164,16 @@ static TxoutType GetTxoutType(const CScript& output_script)
    16     return Solver(output_script, unused);
    17 }
    18
    19+static bool script_static_alloc(const CScript& script, unsigned int size)
    20+{
    21+    return script.size() == size && script.capacity() == script.STATIC_SIZE && script.allocated_memory() == 0;
    22+}
    23+
    24+static bool script_dynamic_alloc(const CScript& script, unsigned int size, unsigned int cap)
    25+{
    26+    return script.size() == size && script.capacity() == cap && script.allocated_memory() == cap;
    27+}
    28+
    29 BOOST_AUTO_TEST_CASE(script_size_and_capacity_test)
    30 {
    31     BOOST_CHECK_EQUAL(sizeof(CompressedScript), 40);
    32
    33@@ -1241,18 +1243,14 @@ BOOST_AUTO_TEST_CASE(script_size_and_capacity_test)
    34
    35         const auto script{GetScriptForRawPubKey(uncompressed_pubkey)};
    36         BOOST_CHECK_EQUAL(GetTxoutType(script), TxoutType::PUBKEY);
    37-        BOOST_CHECK_EQUAL(script.size(), 67);
    38-        BOOST_CHECK_EQUAL(script.capacity(), 67);
    39-        BOOST_CHECK_EQUAL(script.allocated_memory(), 67);
    40+        BOOST_CHECK_PREDICATE(script_dynamic_alloc, (script)(67)(67));
    41     }
    42
    43     // Bare multisig needs extra allocation
    44     {
    45         const auto script{GetScriptForMultisig(1, std::vector{2, dummy_pubkey})};
    46         BOOST_CHECK_EQUAL(GetTxoutType(script), TxoutType::MULTISIG);
    47-        BOOST_CHECK_EQUAL(script.size(), 71);
    48-        BOOST_CHECK_EQUAL(script.capacity(), 103);
    49-        BOOST_CHECK_EQUAL(script.allocated_memory(), 103);
    50+        BOOST_CHECK_PREDICATE(script_dynamic_alloc, (script)(71)(103));
    51     }
    52 }
    
  71. l0rinc force-pushed on Jul 26, 2025

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-07-26 21:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me