bench: add index benchmarks #35827

arejula27 commented at 7:53 PM on July 27, 2026: none

There is a fair amount of work going into the indices at the moment, and no cheap way to tell whether a change actually improves what it set out to improve: checking end to end means a full IBD. The only index benchmark in the tree, BlockFilterIndexSync, syncs coinbase-only blocks, so it cannot show anything that scales with what a block contains, and there is none at all for TxIndex or TxoSpenderIndex.

This adds shared helpers in src/bench/index_sync_util.{h,cpp} that build a chain of blocks with chained, validly-signed transactions (~2 inputs and ~2 outputs each, every output paying a distinct key), and on top of them, for txindex, txospenderindex and blockfilterindex:

<Index>SyncDisk / <Index>SyncMem — full sync with the index database on the filesystem, and in memory.
<Index>Lookup — one query per indexed key (FindTx, FindSpender).

How far apart Disk and Mem land depends on the data directory, which defaults to a tmpfs on many Linux systems. Pointing -testdatadir at real storage is what makes the difference the cost of the write path:

Benchmark	tmpfs (default)	ext4 via `-testdatadir`
`TxIndexSyncDisk`	7.0 ms	18.1 ms
`TxIndexSyncMem`	6.5 ms	6.8 ms
`TxIndexLookup`	13.4 us	13.7 us
`TxoSpenderIndexSyncDisk`	8.3 ms	23.8 ms
`TxoSpenderIndexSyncMem`	7.8 ms	8.1 ms
`TxoSpenderIndexLookup`	13.8 us	14.1 us
`BlockFilterIndexSyncRealisticDisk`	14.7 ms	31.0 ms
`BlockFilterIndexSyncRealisticMem`	13.8 ms	17.8 ms

The existing coinbase-only BlockFilterIndexSync is left untouched. The whole suite runs in well under a minute.

I have used this on those two pull requests, and in both the interesting number was one the existing benchmark could not produce: #35531 (5-byte siphash keys) trades a 49% smaller index on disk for a lookup regression that grows with index size, and #34489 (batch db writes) cuts write syscalls by 91% with CPU unchanged. The numbers are in those threads.

Disclosure: I used claude to help me in some parts, however all code has been verified by me and I ran the bench manually on the scenarios mentioned

DrahtBot added the label Tests on Jul 27, 2026

DrahtBot commented at 7:53 PM on July 27, 2026: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35827.

Reviews

See the guideline and AI policy for information on the review process.

Type	Reviewers
Concept ACK	l0rinc

If your review is incorrectly listed, please copy-paste <code></code> into the comment that the bot should ignore.

arejula27 commented at 8:04 PM on July 27, 2026: none

I think this PR can be splited into two, one for the doc and another for the index benchmarks

in src/bench/index_blockfilter.cpp:70 in 6bf3b13e53 outdated

  65 | +// Returns a fresh, not-yet-initialized BASIC BlockFilterIndex. `f_memory=false`
  66 | +// exercises the real disk write path, true isolates CPU cost from I/O.
  67 | +static std::unique_ptr<BlockFilterIndex> MakeBlockFilterIndex(TestChain100Setup& test_setup, bool f_memory)
  68 | +{
  69 | +    return std::make_unique<BlockFilterIndex>(interfaces::MakeChain(test_setup.m_node), BlockFilterType::BASIC,
  70 | +                                              /*n_cache_size=*/1 << 20, f_memory, /*f_wipe=*/true);

l0rinc commented at 8:10 PM on July 27, 2026:

nit: you can use 1_MiB

in src/bench/index_sync_util.cpp:49 in 6bf3b13e53 outdated

  44 | +    // Coinbase from block 1 matures exactly when spent at height 101 (chain
  45 | +    // tip is at height 100 right after TestChain100Setup).
  46 | +    auto& coinbase_to_spend{test_setup.m_coinbase_txns[0]};
  47 | +    size_t k0{0};
  48 | +    size_t k1{1};
  49 | +    const CMutableTransaction first_tx{test_setup.CreateValidTransaction({coinbase_to_spend}, {COutPoint(coinbase_to_spend->GetHash(), 0)},

l0rinc commented at 8:11 PM on July 27, 2026:

I don't usually mind long lines but formatting is a bit off on GitHub

in src/bench/index_sync_util.h:25 in 6bf3b13e53 outdated

  20 | +/**
  21 | + * Size of the chain built by the index benchmarks. Building it is not timed but
  22 | + * dominates the wall time of a run, so keep these just large enough for the
  23 | + * timed part to stand out of the noise.
  24 | + */
  25 | +static constexpr size_t BENCH_INDEX_NUM_BLOCKS{50};

l0rinc commented at 8:12 PM on July 27, 2026:

we should avoid adding new arch dependent types if possible

arejula27 commented at 5:11 PM on July 28, 2026:

mb

in src/bench/index_sync_util.h:59 in 6bf3b13e53 outdated

  54 | +        assert(!index->BlockUntilSyncedToCurrentChain());
  55 | +        index->Sync();
  56 | +
  57 | +        IndexSummary summary{index->GetSummary()};
  58 | +        assert(summary.synced);
  59 | +        assert(summary.best_block_hash == WITH_LOCK(::cs_main, return test_setup.m_node.chainman->ActiveTip()->GetBlockHash()));

l0rinc commented at 8:13 PM on July 27, 2026:

locking inside the measured block will likely distort the results. Consider precalculating and maybe using setup block

in src/bench/index_sync_util.h:97 in 6bf3b13e53 outdated

  92 | +    assert(!keys.empty());
  93 | +    bench.batch(keys.size()).unit("lookup").run([&] {
  94 | +        for (const Key& key : keys) {
  95 | +            const bool ok{lookup_one(key)};
  96 | +            assert(ok);
  97 | +            ankerl::nanobench::doNotOptimizeAway(ok);

l0rinc commented at 8:14 PM on July 27, 2026:

either assert or doNotOptimizeAway, no need for both. Alternatively have a bool at the begining, and each value and assert at the end

in src/bench/index_txindex.cpp:38 in 6bf3b13e53 outdated

  33 | +}
  34 | +
  35 | +static void TxIndexSyncDisk(benchmark::Bench& bench) { TxIndexSync(bench, /*f_memory=*/false); }
  36 | +static void TxIndexSyncMem(benchmark::Bench& bench) { TxIndexSync(bench, /*f_memory=*/true); }
  37 | +BENCHMARK(TxIndexSyncDisk);
  38 | +BENCHMARK(TxIndexSyncMem);

l0rinc commented at 8:15 PM on July 27, 2026:

I'd prefer having all of these at the very end instead

in src/bench/index_txindex.cpp:49 in 6bf3b13e53 outdated

  44 | +{
  45 | +    const auto test_setup = MakeNoLogFileContext<TestChain100Setup>();
  46 | +    ExtendChainWithSpends(*test_setup, BENCH_INDEX_NUM_BLOCKS, BENCH_INDEX_TXS_PER_BLOCK);
  47 | +
  48 | +    // Build and fully sync a persistent txindex, kept alive for the lookups.
  49 | +    auto index{MakeTxIndex(*test_setup, /*f_memory=*/false)};

l0rinc commented at 8:15 PM on July 27, 2026:

these are likely small, so even if they're not in memory, the OS will likely make sure they're still actually in memory...

arejula27 commented at 5:04 PM on July 28, 2026:

Thats a problem i am having with this PR, based on my results looks like Im not using the disk, not sure if it is only the size or am i doing something wrong elsewhere

l0rinc commented at 5:19 PM on July 28, 2026:

You can try invalidating the OS file caches

arejula27 commented at 5:57 PM on July 28, 2026:

I didn't find a way to do it from inside the benchmark itself, only things the user can do outside the process, but I don't know this area well enough. Do you know a simpler way?

Also I noticed the writes are not fsynced (fSync=false in BaseIndex::Commit) and the index is smaller than the write buffer, so it never leaves the memtable. I'll try with a bigger chain and see what happens

in src/bench/index_txospender.cpp:56 in 6bf3b13e53 outdated

  51 | +
  52 | +    const std::vector<COutPoint> outpoints{CollectChainSpentOutpoints(*test_setup, BENCH_INDEX_FIRST_SPEND_HEIGHT)};
  53 | +
  54 | +    BenchIndexLookup(bench, outpoints, [&](const COutPoint& txo) {
  55 | +        const auto result{index->FindSpender(txo)};
  56 | +        return result && result->has_value();

l0rinc commented at 8:16 PM on July 27, 2026:

aren't these two conditions exactly the same?

arejula27 commented at 5:34 PM on July 28, 2026:

 /**
     * Search the index for a transaction that spends the given outpoint.
     *
     * [@param](/bitcoin-bitcoin/contributor/param/)[in] txo  The outpoint to search for.
     *
     * [@return](/bitcoin-bitcoin/contributor/return/)  std::nullopt               if the outpoint has not been spent on-chain.
     *          std::optional{TxoSpender}  if the output has been spent on-chain. Contains the spending transaction
     *                                     and the block it was confirmed in.
     *          util::Unexpected{error}    if something unexpected happened (i.e. disk or deserialization error).
     */
    util::Expected<std::optional<TxoSpender>, std::string> FindSpender(const COutPoint& txo) const;

FindSpender returns util::Expected<std::optional<TxoSpender>, std::string>, which might be complex read. My understanding:

The first check is "no read error".
The second check is "a spender was found".

How do you understand it?

l0rinc commented at 8:17 PM on July 27, 2026: contributor

Concept ACK

Thanks for taking care of these, I will review them later in more detail, just quickly glanced at the structure.

arejula27 force-pushed on Jul 27, 2026

DrahtBot added the label CI failed on Jul 29, 2026

maflcko removed the label CI failed on Jul 30, 2026

DrahtBot added the label CI failed on Jul 30, 2026

DrahtBot commented at 7:52 PM on July 30, 2026: contributor

🚧 At least one of the CI tasks failed. <sub>Task iwyu: https://github.com/bitcoin/bitcoin/actions/runs/30384384298/job/90985587575</sub> <sub>LLM reason (✨ experimental): CI failed because IWYU detected missing/incorrect #includes and generated a failure (Failure generated from IWYU) after applying include changes.</sub>

<details><summary>Hints</summary>

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

</details>

arejula27 force-pushed on Jul 30, 2026

DrahtBot removed the label CI failed on Aug 17, 2026

arejula27 force-pushed on Sep 6, 2026

arejula27 commented at 4:25 PM on September 6, 2026: none

Invalidating the OS caches wasn't the real issue but the data directory defaults to a tmpfs. So f_memory=false was writing to RAM either way. With -testdatadir on real storage the Disk/Mem split shows up on its own, around 2.7x for txindex. The helper documents that now, since the default hides it.

Digging into that I also found the benchmarks never reached BaseIndex::Commit(), a test setup never flushes the chainstate, so its guard always returned early and the commit path stayed out of the measurement. ExtendChainWithSpends() forces a flush now.

The rest of your comments are applied, thank you for the review @l0rinc

bench: add reusable index sync benchmark helpers

The existing BlockFilterIndexSync syncs coinbase-only blocks, which
cannot exercise anything that scales with the number of inputs or
outputs per block.

Add src/bench/index_sync_util.{h,cpp} with the chain setup and timing
loops shared by index benchmarks. ExtendChainWithSpends() builds blocks
of validly-signed transactions with ~2 inputs each (the average ratio
observed on mainnet), paying every output to a distinct key: block
filter elements are deduplicated, so reusing a couple of scripts would
keep every filter tiny no matter how many transactions the block holds.

It also flushes the chainstate once the chain is built. BaseIndex::Commit
returns early unless the index tip is an ancestor of the last flushed
block, and a test setup never flushes on its own, so without this the
benchmarks would write the index entries but never the best-block
locator, leaving CustomCommit() outside the measurement.

5ef138bd01

bench: add TxoSpenderIndex, TxIndex and BlockFilterIndex sync benchmarks

Add, on top of the shared helpers:
- TxoSpenderIndexSync{Disk,Mem} and TxoSpenderIndexLookup (FindSpender)
- TxIndexSync{Disk,Mem} and TxIndexLookup (FindTx)
- BlockFilterIndexSyncRealistic{Disk,Mem}

The Disk variant exercises the file-backed write path, the Mem variant
keeps the database in memory. Reading the pair as the cost of that path
requires the data directory to be on real storage: it defaults to
fs::temp_directory_path(), which on many Linux systems is a tmpfs, and
there both variants write to RAM and report the same number. Pointing
-testdatadir at a real filesystem is what makes the Disk figures mean
what their name says.

The pre-existing coinbase-only BlockFilterIndexSync is left untouched.
It scales with the number of blocks over near-empty filters, where the
new one scales with what a block contains, so the two are kept as
separate axes.

d02a3de47b

doc: describe measuring I/O and memory cost of benchmarks

The framework reports wall-clock time and, where available, cycles and
instructions, but not syscalls, bytes written, on-disk size or peak
memory. Document how to measure those around bench_bitcoin with strace,
/usr/bin/time and du, and the -testdatadir knob needed to avoid tmpfs.

0205152867

arejula27 force-pushed on Sep 6, 2026