coins: use jumboblock SipHash-1-3 for hashing CCoinsMap keys #35215

pull l0rinc wants to merge 4 commits into bitcoin:master from l0rinc:l0rinc/siphash-jumbo changing 5 files +151 −15
  1. l0rinc commented at 2:02 PM on May 5, 2026: contributor

    Problem: Several internal hash-table and index-prefix paths key by values that already contain a 32-byte txid/hash. The most important current user is CCoinsMap, which stores the in-memory dbcache and hashes COutPoint keys as a 32-byte txid plus a 32-bit output index.

    The existing PresaltedSipHasher path uses SipHash-2-4 for these fixed-width shapes. That is conservative, but expensive: the txid is processed as four independent 64-bit message blocks, followed by the fixed-size tail and finalization. The 32-byte txid-only path and 36-byte outpoint-shaped path both take 14 SipRounds today.

    This also matters for hash-prefix index work such as #35531: once a new persisted key format chooses a hash function, changing it later would require reindexing. If we want a faster non-standard keyed hash for txids there, it is better to expose and benchmark the fixed txid-only construction before that format ships.

    SipHash-1-3 & Jumbo blocks: This implementation follows Pieter Wuille's jumboblock suggestion based on the SipHash analysis paper. The main input is already a 256-bit hash, so the hasher processes the four txid limbs together as one block instead of feeding them as four independent 64-bit SipHash message blocks.

    <details> <summary>Pieter Wuille's jumboblock and SipRound sketch</summary>

    we can actually process a full 256-bit hash at once, as one block, rather than having a block per 64-bit.

    SH24=SipHash-2-4, SH13=SipHash-1-3, JB=jumboblock(this idea), UP=unpadded(dropping the last block, which IMO doesn't help given that our inputs are constant length).

    In the UTXO set cache setting, we have:

    • SH24: 16 SipRounds
    • SH24+UP: 14 SipRounds
    • SH24+JB: 10 SipRounds
    • SH24+JB+UP: 8 SipRounds
    • SH13: 9 SipRounds
    • SH13+UP: 8 SipRounds
    • SH13+JB: 6 SipRounds
    • SH13+JB+UP: 5 SipRounds

    Specifically, this is pseudocode that I asked him about:

    (m0-m3 are the 64-bit limbs of the hash input, m4-m5 are other inputs including padding)

# initialization
v0 = c0 ^ k0
v1 = c1 ^ k1
v2 = c2 ^ k0
v3 = c3 ^ k1

# process m0-m3
v0 ^= m0
v1 ^= m1
v2 ^= m2
v3 ^= m3
SIPROUND * c
v0 ^= m3
v1 ^= m0
v2 ^= m1
v3 ^= m2

# process m4
v3 ^= m4
SIPROUND * c
v0 ^= m4

# process m5
v3 ^= m5
SIPROUND * c
v0 ^= m5

# finalize
v2 ^= 0xff
SIPROUND * d
return v0 ^ v1 ^ v2 ^ v3

    </details>

    Design: Add a narrow PresaltedSipHasher13Jumbo specialization for two fixed input shapes whose main input is already a uniformly distributed uint256 hash:

    • uint256: process the four txid/hash limbs as one jumboblock, omit the final length/padding word, and run 3 finalization SipRounds. This is the 4-round txid-only SH13+JB+UP case.
    • uint256 + uint32_t: process the four txid/hash limbs as one jumboblock, run one compression round for the 32-bit index, omit the final length/padding word, and run 3 finalization SipRounds. This is the 5-round outpoint-shaped SH13+JB+UP case.

    These paths intentionally do not try to match a variable-length SipHash API. Their supported input shapes are fixed and unambiguous, so the length/padding word does not add useful separation for these local/index uses.

    Pieter also ran the jumboblock idea by Jean-Philippe Aumasson, one of the SipHash authors; based on a preliminary analysis, Aumasson did not think this made collisions easier to construct. Aumasson also said SipHash-1-3 is fine for this hashmap use case and offered to comment on or review the PR.

    <img width="2100" height="860" alt="siphash_compare_github_aligned_v7_compact min" src="https://github.com/user-attachments/assets/c3bf9297-c4b1-4c96-a47b-dc6b4377758b" />

    • Old 36-byte path: 4 separate 64-bit txid compressions + 1 index/length compression + 4 finalization rounds = 14 SipRounds.
    • New 36-byte path: 1 combined 256-bit txid compression + 1 index compression + 3 finalization rounds = 5 SipRounds.
    • New 32-byte path: 1 combined 256-bit txid compression + 3 finalization rounds = 4 SipRounds.

    Fix: Add PresaltedSipHasher13Jumbo, a fixed-shape SipHash-1-3 jumboblock specialization for hashing an existing uint256 hash, optionally plus a uint32_t index. Then switch the existing SaltedOutpointHasher wrapper to use the 36-byte overload, so existing COutPoint unordered maps and sets keep their public hasher type while getting the faster implementation.

    This covers the coins cache and other in-memory outpoint tables through the existing abstraction, without spreading a variant-specific type name through call sites. The 32-byte overload gives txid-only index-prefix work, such as #35531, a benchmarked way to use the same construction before a new disk format commits to a hash function.

    The regular PresaltedSipHasher path remains for existing txid/wtxid/uint256 hashers, compact-block short IDs, and current persisted/index key derivation unless a specific follow-up deliberately opts into this non-standard construction.

    For the SaltedOutpointHasher users changed in this PR, the salted hash codes are only local in-memory table indexes for the current process; they already vary across normal restarts and are not serialized, persisted, sent over the network, or used for consensus.

    Reproducer: Fixed test vectors document both non-standard jumboblock outputs. The benchmarks now include SipHash-2-4 baselines and jumboblock paths for both supported fixed-width shapes.

    Counting the dbcache buckets indicates the new 36-byte outpoint hasher satisfies the uniformness criteria relied on before: <img width="1200" height="750" alt="ccoinsmap-collisions" src="https://github.com/user-attachments/assets/eeedec81-acdc-4adf-a9c8-bfce089700da" />

    Fresh isolated aarch64 microbenchmarks on a Raspberry Pi 5 show the new 36-byte path about 2x faster with GCC and Clang.

    <details> <summary>Linux reproducer command</summary>

    The command below rebuilds with GCC and Clang and prints only the benchmark output after the build.

    for COMPILER in gcc clang; do \
  if [ "$COMPILER" = gcc ]; then CC=gcc; CXX=g++; else CC=clang; CXX=clang++; fi; \
  cmake -B "build-bench-$COMPILER" -DCMAKE_BUILD_TYPE=Release -DBUILD_BENCH=ON -DBUILD_TESTS=OFF -DBUILD_GUI=OFF -DENABLE_WALLET=OFF -DCMAKE_C_COMPILER="$CC" -DCMAKE_CXX_COMPILER="$CXX" >/dev/null 2>&1 && \
  cmake --build "build-bench-$COMPILER" --target bench_bitcoin -j"$(nproc)" >/dev/null 2>&1 && \
  echo "" && echo "$(date -I) | SipHash fixed-width microbench | $("$CC" --version | head -1) | $("$CXX" --version | head -1) | $(hostname) | $(uname -m) | $(lscpu | awk -F: '/Model name/{print $2; exit}' | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM" && \
  "build-bench-$COMPILER/bin/bench_bitcoin" -filter='SipHash.*32b|SipHash.*36b' -min-time=10000; \
done

    </details>

    <details> <summary>aarch64 SipHash 36-byte microbenchmarks: ~2x faster with GCC and Clang</summary>

    2026-05-03 | SipHash 36-byte microbench | gcc (Ubuntu 14.2.0-19ubuntu2) 14.2.0 | g++ (Ubuntu 14.2.0-19ubuntu2) 14.2.0 | rpi5-16-2 | aarch64 | Cortex-A76 | 4 cores | 15Gi RAM

|               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |         bra/op |   miss% |     total | benchmark
|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
|               16.61 |       60,192,540.58 |    0.0% |           80.00 |           39.78 |  2.011 |           0.00 |   50.0% |     11.00 | `SipHash13Jumbo_36b`
|               33.92 |       29,483,465.40 |    0.0% |          171.00 |           81.23 |  2.105 |           0.00 |   50.0% |     11.00 | `SipHash24_36b`

2026-05-03 | SipHash 36-byte microbench | Ubuntu clang version 22.0.0 (++20250923084147+c890a9050e88-1~exp1~20250923084331.324) | Ubuntu clang version 22.0.0 (++20250923084147+c890a9050e88-1~exp1~20250923084331.324) | rpi5-16-2 | aarch64 | Cortex-A76 | 4 cores | 15Gi RAM

|               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |         bra/op |   miss% |     total | benchmark
|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
|               17.02 |       58,748,229.11 |    0.0% |           81.00 |           40.75 |  1.988 |           0.00 |   50.0% |     11.00 | `SipHash13Jumbo_36b`
|               33.21 |       30,115,556.16 |    0.0% |          172.00 |           79.51 |  2.163 |           0.00 |   50.0% |     11.00 | `SipHash24_36b`

    </details>

    Benchmarks

    The -reindex-chainstate runs below were collected before this final shared-wrapper shape, while the branch still applied the same jumboblock hasher only to CCoinsMap.

    <details><summary>5% faster | reindex-chainstate | 946649 blocks | dbcache 1000 | i9-ssd | x86_64 | Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz | 16 cores | 62Gi RAM | SSD</summary>

    for DBCACHE in 1000; do \
    COMMITS="976985eccd546a95e38973b854ccc6589e8afc74 b16188a906302b7d9e06adf2bc57e2b4f88b942f"; \
    STOP=946649; CC=gcc; CXX=g++; \
    BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
    (echo ""; for c in $COMMITS; do git fetch -q origin "$c" 2>/dev/null || true; git log -1 --pretty='%h %s' $c || exit 1; done) && \
    (echo "" && echo "$(date -I) | reindex-chainstate | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 1 && echo HDD || echo SSD)"; echo "") && \
    hyperfine \
    --sort command \
    --runs 1 \
    --export-json "$BASE_DIR/rdx-$(sed -E 's/[^ ]+/\L&/g;s/[.]/_/g;s/ /-/g'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \
    --parameter-list COMMIT ${COMMITS// /,} \
    --prepare "killall -9 bitcoind 2>/dev/null; rm -f ./build/bin/bitcoind; git clean -fxd; git reset --hard {COMMIT} && \
      cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release && ninja -C build bitcoind -j1 && \
      ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20; rm -f $DATA_DIR/debug.log; rm -rfd $DATA_DIR/indexes;" \
    --conclude "killall bitcoind || true; sleep 5; grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'Disabling script verification at block [#1](/bitcoin-bitcoin/1/)' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log && grep 'Bitcoin Core version' $DATA_DIR/debug.log | grep -q \"\$(git rev-parse --short=12 {COMMIT})\"; \
                cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
    "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0";
done

976985eccd Merge bitcoin/bitcoin#34124: validation: make `CCoinsView` a pure virtual interface
b16188a906 crypto: inline jumboblock SipHash

2026-05-04 | reindex-chainstate | 946649 blocks | dbcache 1000 | i9-ssd | x86_64 | Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz | 16 cores | 62Gi RAM | SSD

Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=946649 -dbcache=1000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 976985eccd546a95e38973b854ccc6589e8afc74)
  Time (abs ≡):        19247.091 s               [User: 33228.894 s, System: 1908.564 s]

Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=946649 -dbcache=1000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = b16188a906302b7d9e06adf2bc57e2b4f88b942f)
  Time (abs ≡):        18357.707 s               [User: 32399.997 s, System: 1935.467 s]

Relative speed comparison
        1.05          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=946649 -dbcache=1000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 976985eccd546a95e38973b854ccc6589e8afc74)
        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=946649 -dbcache=1000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = b16188a906302b7d9e06adf2bc57e2b4f88b942f)

    </details>

    <details><summary>5% faster | reindex-chainstate | 946649 blocks | dbcache 30000 | i9-ssd | x86_64 | Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz | 16 cores | 62Gi RAM | SSD</summary>

    for DBCACHE in 30000; do \
    COMMITS="976985eccd546a95e38973b854ccc6589e8afc74 b16188a906302b7d9e06adf2bc57e2b4f88b942f"; \
    STOP=946649; CC=gcc; CXX=g++; \
    BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
    (echo ""; for c in $COMMITS; do git fetch -q origin "$c" 2>/dev/null || true; git log -1 --pretty='%h %s' $c || exit 1; done) && \
    (echo "" && echo "$(date -I) | reindex-chainstate | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 1 && echo HDD || echo SSD)"; echo "") && \
    hyperfine \
    --sort command \
    --runs 1 \
    --export-json "$BASE_DIR/rdx-$(sed -E 's/[^ ]+/\L&/g;s/[.]/_/g;s/ /-/g'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \
    --parameter-list COMMIT ${COMMITS// /,} \
    --prepare "killall -9 bitcoind 2>/dev/null; rm -f ./build/bin/bitcoind; git clean -fxd; git reset --hard {COMMIT} && \
      cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release && ninja -C build bitcoind -j1 && \
      ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20; rm -f $DATA_DIR/debug.log; rm -rfd $DATA_DIR/indexes;" \
    --conclude "killall bitcoind || true; sleep 5; grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'Disabling script verification at block [#1](/bitcoin-bitcoin/1/)' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log && grep 'Bitcoin Core version' $DATA_DIR/debug.log | grep -q \"\$(git rev-parse --short=12 {COMMIT})\"; \
                cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
    "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0";
done

ffa1b71e87 bench: add SipHash-2-4 36-byte benchmark
42410eda26 coins: use jumboblock SipHash for `CCoinsMap`

2026-05-03 | reindex-chainstate | 946649 blocks | dbcache 30000 | i9-ssd | x86_64 | Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz | 16 cores | 62Gi RAM | SSD

Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=946649 -dbcache=30000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = ffa1b71e870755d2a355fe3c8ed8c084544fabb2)
  Time (abs ≡):        17664.762 s               [User: 24440.658 s, System: 773.072 s]

Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=946649 -dbcache=30000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 42410eda260ed
  Time (abs ≡):        16835.349 s               [User: 23591.362 s, System: 759.491 s]

Relative speed comparison
        1.05          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=946649 -dbcache=30000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = ffa1b71e870755d2a355fe3c8ed8c084544fabb2)
        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=946649 -dbcache=30000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 42410eda260ed60bc2af1e8857e96f348bafdcde)

    </details>

    <details><summary>2% faster | reindex-chainstate | 946649 blocks | dbcache 1000 | rpi5-16-3 | aarch64 | Cortex-A76 | 4 cores | 15Gi RAM | SSD</summary>

    for DBCACHE in 1000; do \
    COMMITS="71728b0c83d6f372406f34549ce8b988fa7e3a1e b16188a906302b7d9e06adf2bc57e2b4f88b942f"; \
    STOP=946649; CC=gcc; CXX=g++; \
    BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
    (echo ""; for c in $COMMITS; do git fetch -q origin "$c" 2>/dev/null || true; git log -1 --pretty='%h %s' $c || exit 1; done) && \
    (echo "" && echo "$(date -I) | reindex-chainstate | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 1 && echo HDD || echo SSD)"; echo "") && \
    hyperfine \
    --sort command \
    --runs 1 \
    --export-json "$BASE_DIR/rdx-$(sed -E 's/[^ ]+/\L&/g;s/[.]/_/g;s/ /-/g'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \
    --parameter-list COMMIT ${COMMITS// /,} \
    --prepare "killall -9 bitcoind 2>/dev/null; rm -f ./build/bin/bitcoind; git clean -fxd; git reset --hard {COMMIT} && \
      cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release && ninja -C build bitcoind -j1 && \
      ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20; rm -f $DATA_DIR/debug.log; rm -rfd $DATA_DIR/indexes;" \
    --conclude "killall bitcoind || true; sleep 5; grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'Disabling script verification at block [#1](/bitcoin-bitcoin/1/)' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log && grep 'Bitcoin Core version' $DATA_DIR/debug.log | grep -q \"\$(git rev-parse --short=12 {COMMIT})\"; \
                cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
    "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0";
done

71728b0c83 bench: add SipHash-2-4 36-byte benchmark
b16188a906 crypto: inline jumboblock SipHash

2026-05-03 | reindex-chainstate | 946649 blocks | dbcache 1000 | rpi5-16-3 | aarch64 | Cortex-A76 | 4 cores | 15Gi RAM | SSD

Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=946649 -dbcache=1000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 71728b0c83d6f372406f34549ce8b988fa7e3a1e)
  Time (abs ≡):        37976.474 s               [User: 55537.835 s, System: 4759.529 s]

Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=946649 -dbcache=1000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = b16188a906302b7d9e06adf2bc57e2b4f88b942f)
  Time (abs ≡):        37076.745 s               [User: 54518.158 s, System: 4766.083 s]

Relative speed comparison
        1.02          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=946649 -dbcache=1000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 71728b0c83d6f372406f34549ce8b988fa7e3a1e)
        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=946649 -dbcache=1000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = b16188a906302b7d9e06adf2bc57e2b4f88b942f)

    </details>

    <details><summary>2% faster | reindex-chainstate | 946649 blocks | dbcache 1000 | umbrel | x86_64 | Intel(R) N150 | 4 cores | 15Gi RAM | SSD</summary>

    for DBCACHE in 1000; do \
    COMMITS="71728b0c83d6f372406f34549ce8b988fa7e3a1e b16188a906302b7d9e06adf2bc57e2b4f88b942f"; \
    STOP=946649; CC=gcc; CXX=g++; \
    BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
    (echo ""; for c in $COMMITS; do git fetch -q origin "$c" 2>/dev/null || true; git log -1 --pretty='%h %s' $c || exit 1; done) && \
    (echo "" && echo "$(date -I) | reindex-chainstate | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 1 && echo HDD || echo SSD)"; echo "") && \
    hyperfine \
    --sort command \
    --runs 1 \
    --export-json "$BASE_DIR/rdx-$(sed -E 's/[^ ]+/\L&/g;s/[.]/_/g;s/ /-/g'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \
    --parameter-list COMMIT ${COMMITS// /,} \
    --prepare "killall -9 bitcoind 2>/dev/null; rm -f ./build/bin/bitcoind; git clean -fxd; git reset --hard {COMMIT} && \
      cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release && ninja -C build bitcoind -j1 && \
      ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20; rm -f $DATA_DIR/debug.log; rm -rfd $DATA_DIR/indexes;" \
    --conclude "killall bitcoind || true; sleep 5; grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'Disabling script verification at block [#1](/bitcoin-bitcoin/1/)' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log && grep 'Bitcoin Core version' $DATA_DIR/debug.log | grep -q \"\$(git rev-parse --short=12 {COMMIT})\"; \
                cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
    "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0";
done

71728b0c83 bench: add SipHash-2-4 36-byte benchmark
b16188a906 crypto: inline jumboblock SipHash

2026-05-03 | reindex-chainstate | 946649 blocks | dbcache 1000 | umbrel | x86_64 | Intel(R) N150 | 4 cores | 15Gi RAM | SSD

Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=946649 -dbcache=1000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 71728b0c83d6f372406f34549ce8b988fa7e3a1e)
  Time (abs ≡):        27194.936 s               [User: 36353.676 s, System: 3494.054 s]

Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=946649 -dbcache=1000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = b16188a906302b7d9e06adf2bc57e2b4f88b942f)
  Time (abs ≡):        26559.372 s               [User: 35396.175 s, System: 3521.104 s]

Relative speed comparison
        1.02          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=946649 -dbcache=1000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 71728b0c83d6f372406f34549ce8b988fa7e3a1e)
        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=946649 -dbcache=1000 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = b16188a906302b7d9e06adf2bc57e2b4f88b942f)

    </details>

    Future Work

    • Evaluate using the 32-byte jumboblock path in txindex hash-prefix work such as #35531 before a new persisted key format ships, because changing the hash function afterward would require reindexing.
    • Add a general SipHash-1-3 implementation, shorter-input specializations, or other variants from Pieter's sketch if benchmarks justify them.
    • Evaluate related ideas for compact-block short IDs (BIP152); this is a protocol surface and would require separate design, BIP discussion, and negotiation.
  2. DrahtBot added the label UTXO Db and Indexes on May 5, 2026
  3. DrahtBot commented at 2:02 PM on May 5, 2026: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35215.

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK andrewtoth
    Concept ACK sedited, optout21, theStack

    If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    No conflicts as of last run.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

  4. l0rinc force-pushed on May 5, 2026
  5. DrahtBot added the label CI failed on May 5, 2026
  6. DrahtBot commented at 7:27 PM on May 5, 2026: contributor

    <!--85328a0da195eb286784d51f73fa0af9-->

    🚧 At least one of the CI tasks failed. <sub>Task 32 bit ARM: https://github.com/bitcoin/bitcoin/actions/runs/25393628377/job/74474475011</sub> <sub>LLM reason (✨ experimental): CI failed due to a C++ build error: hash_tests.cpp couldn’t compile because uint256’s consteval hex parsing wasn’t a constant expression.</sub>

    <details><summary>Hints</summary>

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

    </details>

  7. l0rinc force-pushed on May 5, 2026
  8. DrahtBot removed the label CI failed on May 5, 2026
  9. veorq commented at 5:20 AM on May 6, 2026: none

    FTR I confirm my statements quoted by OP

    Pieter also ran the jumboblock idea by Jean-Philippe Aumasson, one of the SipHash authors; based on a preliminary analysis, Aumasson did not think this made collisions easier to construct. Aumasson also said SipHash-1-3 is fine for this hashmap use case and offered to comment on or review the PR.

  10. ViniciusCestarii commented at 7:57 PM on May 7, 2026: contributor

    Nice. Ran the microbench and it reproduces ~2x on x86_64 as well.

    archlinux | i7-13650HX | 20 cores | 32GB RAM

    gcc 15.2.1: 31.57 ns -> 16.45 ns (1.92x) clang 22.1.3: 32.39 ns -> 16.73 ns (1.94x)

    <details> <summary>Full bench tables</summary>

    2026-05-07 | SipHash 36-byte microbench | gcc (GCC) 15.2.1 20260209 | g++ (GCC) 15.2.1 20260209 | archlinux | x86_64 | 13th Gen Intel(R) Core(TM) i7-13650HX | 20 cores | 31Gi RAM

    | ns/op | op/s | err% | ins/op | cyc/op | IPC | bra/op | miss% | total | benchmark |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:---------- | 16.45 | 60,793,565.44 | 0.1% | 105.00 | 45.97 | 2.284 | 0.00 | 29.9% | 11.00 | SipHash13Jumbo_36b | 31.57 | 31,678,908.80 | 0.2% | 254.00 | 88.23 | 2.879 | 3.00 | 0.0% | 10.98 | SipHash24_36b

    2026-05-07 | SipHash 36-byte microbench | clang version 22.1.3 | clang version 22.1.3 | archlinux | x86_64 | 13th Gen Intel(R) Core(TM) i7-13650HX | 20 cores | 31Gi RAM

    | ns/op | op/s | err% | ins/op | cyc/op | IPC | bra/op | miss% | total | benchmark |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:---------- | 16.73 | 59,756,190.42 | 0.1% | 106.00 | 46.77 | 2.267 | 0.00 | 1.6% | 11.01 | SipHash13Jumbo_36b | 32.39 | 30,871,809.86 | 0.1% | 246.00 | 90.51 | 2.718 | 3.00 | 0.0% | 10.99 | SipHash24_36b

    </details>

  11. optout21 commented at 11:45 AM on May 29, 2026: contributor

    I'd share an additional data point here: pushing the idea further, I've explored a bit using a simpler, non-SipHash XOR-only hasher. My conclusion is that that's not a viable solution, I'm only sharing it here as an additional data point (as a kind of lower bound on the hashing time). If pre-change (PresaltedSipHasher) hashing time is 100%, this PR (PresaltedSipHasher13Jumbo) is 50%, XOR-only (XorHasher6x8_2_36b) is 22% (lower means faster). The XOR-only hasher only XOR's together a salt, the four 8-byte parts of the TXID, and the vout (6 parts, 5 XOR's). While the solution may be usable, there are uncertainties stemming from the fact that it's not a secure hash, while the additional performance gain is rather small. The proposal of this PR is sound, and has a better reward/risk ratio. The code delta and a bit more details are available here: https://github.com/optout21/bitcoin/pull/12

  12. sedited commented at 11:59 AM on May 29, 2026: contributor

    Concept ACK

  13. optout21 commented at 12:03 PM on May 29, 2026: contributor

    Concept ACK

    What I would highlight:

    • TXID is a hash already (making the requirements on the hasher less strict)
    • the presented performance improvement
    • the presented bucket analysis
  14. bench: add SipHash-2-4 36-byte benchmark
    Rename the existing 32-byte benchmark to `SipHash24_32b`.
Add a 36-byte variant for `uint256` plus a 32-bit outpoint index.
This records the current baseline shape before adding outpoint-specific hashers.
    ef1f83a91e
  15. crypto: use jumboblock SipHash for local indexes
    Add `PresaltedSipHasher13Jumbo` for hashing a `uint256`, optionally plus a `uint32_t` outpoint index.
For the supported fixed 32-byte and 36-byte inputs, the implementation processes the four hash limbs as one jumboblock, omits SipHash's final length/padding word, and runs 3 finalization SipRounds.
The 32-byte path is the 4-round `SH13+JB+UP` case from Pieter Wuille's sketch, and the 36-byte path adds one compression round for the extra index.

Switch the existing `SaltedOutpointHasher` wrapper to the new presalted hasher, so existing `COutPoint` unordered containers keep their public hasher type while using the faster implementation.
This is a non-standard local-index specialization, meant for keys that already contain a uniformly distributed hash.
Add fixed test vectors for both non-standard input shapes.

Co-authored-by: Pieter Wuille <pieter@wuille.net>
Co-authored-by: Jean-Philippe Aumasson <jeanphilippe.aumasson@gmail.com>
    8d8f01d1f8
  16. bench: add jumboblock SipHash benchmarks
    Add focused `PresaltedSipHasher13Jumbo` benchmarks for the same fixed-width shapes covered by the existing SipHash-2-4 baselines.
`SipHash13Jumbo_32b` measures the txid-only path that can be used for salted local index prefixes.
`SipHash13Jumbo_36b` measures the outpoint-shaped `uint256` plus `uint32_t` path used by `SaltedOutpointHasher`.

Keep these benchmarks before the inline commit so reviewers can measure the non-inline jumboblock implementation and then compare it with the inlined version in the next commit.
    daa2eccd39
  17. crypto: inline jumboblock SipHash
    Move `siphash_detail::SipRound` and both `PresaltedSipHasher13Jumbo` overloads into `siphash.h`.
This lets the benchmark, the txid-only path, and `COutPoint` hash-table call sites inline the short fixed-width hash body through their presalted hasher wrappers.

Keep this as a separate commit after the benchmark additions so reviewers can measure the non-inline jumboblock implementation and then compare it with the inlined version.
Inlining made the 36-byte benchmark up to about 16% faster locally, while the existing SipHash implementations did not show improvement from the same treatment.
    4d00740e29
  18. l0rinc force-pushed on Jun 16, 2026
  19. l0rinc commented at 10:12 PM on June 16, 2026: contributor

    Added a simpler 32 byte PresaltedSipHasher13Jumbo hasher for #35531 (+ rebase and code simplification + PR description adjustments)

  20. in src/crypto/siphash.h:18 in 4d00740e29 
      13 |  #include <span>
  14 | +#include <uint256.h>
  15 | +
  16 | +namespace siphash_detail {
  17 | +
  18 | +ALWAYS_INLINE void SipRound(uint64_t& v0, uint64_t& v1, uint64_t& v2, uint64_t& v3)
    optout21 commented at 8:26 AM on June 17, 2026:

    4d00740 crypto: inline jumboblock SipHash:

    I haven't seen documented the rationale for switching from preprocessor macro to inline method. Is it just code style or performance? I'm just curious, not opposing.

    Also, "Move siphash_detail::SipRound" in the commit description is not entirely correct, as pre-commit SipRound didn't exist, only SIPROUND. Could you correct it?

    l0rinc commented at 8:41 AM on June 17, 2026:

    The last commit inlines this after the benchmarks to prove that inlining speeds it up - which wasn't the case for previous SipHash implementations. Switching from macros is mostly a modernization attempt now that C++ has proper alternatives.

  21. optout21 commented at 8:27 AM on June 17, 2026: contributor

    Concept ACK (4d00740e2921c09a717bcf1964b94780a64757bb)

    Re-reviewed, including the new 32-byte version of Jumbo. Verified benchmarks locally.

  22. DrahtBot requested review from sedited on Jun 17, 2026
  23. andrewtoth approved
  24. andrewtoth commented at 11:23 PM on June 18, 2026: contributor

    ACK 4d00740e2921c09a717bcf1964b94780a64757bb

    Verified the speedups. I'm not an expert on hashing, but it makes sense to me that if we're hashing a 32-byte cryptographic hash we can do less work since an attacker can't control most of the input either. The attestation of the author of siphash gives me confidence in this change as well.

    variant SipHash-2-4 (ns/op) SipHash13Jumbo (ns/op) speedup
    32-byte 17.46 7.87 2.22x
    36-byte 17.14 8.75 1.96x
  25. DrahtBot requested review from optout21 on Jun 19, 2026
  26. optout21 commented at 7:07 AM on June 22, 2026: contributor

    I've performed 'bit-flip' tests on the new and old SipHasher. The results are good, as expected, virtually identical.

    property SipHasher SipHasher13Jumbo
    mean output-bit flip fraction per input-bit flip (ideal 0.5): 0.5000 0.5000
    bit-independence mean (deviation from 0.5): 0.0040 0.0040
    bit-independence max (deviation from 0.5): 0.0191 0.0193
  27. sedited removed review request from optout21 on Jun 24, 2026
  28. sedited requested review from theStack on Jun 24, 2026
  29. theStack commented at 9:45 PM on June 28, 2026: contributor

    Concept ACK

  30. DrahtBot requested review from optout21 on Jun 28, 2026
Contributors
l0rincDrahtBotveorqViniciusCestariioptout21seditedandrewtoththeStack


theStackseditedoptout21

Labels
UTXO Db and Indexes
Linked (view graph)
#1 JSON-RPC support for mobile devices ("ultra-lightweight" clients)#34280 Coins Cache Cleanup checklist#35531 txindex: hash keys and pack positions to reduce disk usage#35573 build: guard forced inlining
github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-07-07 05:51 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me