txospenderindex: hash keys and pack positions to reduce disk usage #36480

pull arejula27 wants to merge 6 commits into bitcoin:master from arejula27:txospenderindex-block-keys changing 15 files +789 −264
  1. arejula27 commented at 8:46 PM on October 8, 2026: contributor

    This applies the txindex key scheme from #35531 to txospenderindex. The motivation is to save space and to allow replicating what #36002 does for txindex, by referencing blocks instead of local files.

    The first commits refactor the index code so the #35531 scheme can be shared: BaseIndex::DB can opt in to the versioned sync locator, and the block sequence keys, sequence assignment and salted hasher move from txindex into index/block_seq.h, without changing txindex's on-disk format. The following commits switch txospenderindex to the new keys and add tests. As in txindex, entries of disconnected blocks are kept and lookups only count blocks in the active chain, so CustomRemove is gone. Entries written by v31 are still read as a fallback. The last commit makes FindSpender skip candidates it cannot read and return std::optional, mimicking TxIndex::FindTx, since with 5-byte prefixes an unreadable false positive could otherwise hide the real spender.

    Each entry is about 33% smaller (key bytes):

    • before: 's' (1) + SipHash-2-4 of the outpoint (8) + CDiskTxPos, as varints nFile (2) + nPos (4) + nTxOffset (3) = 18 bytes
    • after: 'x' (1) + SipHash-1-3-UJ prefix (5) + block sequence varint (3) + tx offset (3) = 12 bytes

    I did not have time to do a benchmark (my setup is really cheap, laptop where i work with an external SSD connected), will try to do on the next days. But same as #35531 i hope to reduce disk space and indexing time.

  2. index: let indexes opt in to a versioned sync locator
    Move the best_block_v2 locator handling from TxIndex::DB into
    BaseIndex::DB behind a constructor flag, so other indexes can reuse
    it when they change their on-disk format.
    53566bf296
  3. refactor: move block sequence keys to index/block_seq.h
    Make the key prefix a template parameter so other indexes can reuse
    them.
    
    Not in db_key.h, which holds the height-based scheme of
    blockfilterindex and coinstatsindex.
    f3b1bef792
  4. refactor: move block sequence assignment and hasher salt to block_seq
    Move the logic that assigns a sequence number to a newly indexed block
    and the creation of the salted hasher out of TxIndex::DB into free
    functions in block_seq, so other indexes using the same scheme can
    share them.
    e2a3182e45
  5. txospenderindex: key entries by block sequence instead of disk position
    Use the txindex key scheme: a 5-byte salted hash of the spent outpoint,
    followed by the block sequence and the offset of the spending tx.
    
    As in txindex, entries of disconnected blocks are kept, so CustomRemove
    goes away and FindSpender only returns spends in the active chain.
    
    Fall back to the legacy entries if no spend is found.
    007650b331
  6. test: cover txospenderindex hash collisions, reorgs and legacy entries
    Pin the encodings of the new and legacy keys, and add unit tests for
    hash prefix false positives, for entries of disconnected blocks being
    kept but not returned, and for the fallback to legacy entries. Add a
    functional test that upgrades a v31 datadir, reorgs a block indexed in
    the legacy format and downgrades again.
    21772d62ec
  7. txospenderindex: skip unreadable candidates in FindSpender
    Mimic TxIndex::FindTx: skip a candidate that cannot be read and return
    std::optional. With 5-byte hash prefixes collisions are no longer
    negligible, and failing on an unreadable false positive would hide the
    real spender.
    fadf210822
  8. DrahtBot commented at 8:46 PM on October 8, 2026: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    External sites

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline and AI policy for information on the review process.

    Type Reviewers
    Concept ACK andrewtoth, l0rinc, w0xlt

    If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #36033 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/36033.svg"></sub> ([wip,nomerge,rfc] build: Require C++23 compiler by maflcko)
    • #36002 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/36002.svg"></sub> (txindex: allow running in pruned mode by andrewtoth)
    • #35569 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/35569.svg"></sub> (Encapsulation for CTransaction by purpleKarrot)
    • #34132 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/34132.svg"></sub> (coins, dbwrapper: remove error catcher, make point-read failures fatal by l0rinc)
    • #33324 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/33324.svg"></sub> (blocks: add resumable reobfuscation for existing block files by l0rinc)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

  9. andrewtoth commented at 8:50 PM on October 8, 2026: contributor

    Concept ACK!

    I did not have time to do a benchmark

    I have done this before. IIRC the db size went from ~84GB -> ~62GB. The time to index is also much lower.

    Do we want to also enable pruning in one go like we are with #36002? I don't think it would be controversial so we could do it all in 1 PR.

  10. arejula27 commented at 8:56 PM on October 8, 2026: contributor

    I have done this before.

    Sorry @andrewtoth, i wasnt aware of it :sweat_smile: , should i compare with ur code and add u as co author?

    IIRC the db size went from ~84GB -> ~62GB.

    sounds awesome!

    Do we want to also enable pruning in one go like we are with #36002? I don't think it would be controversial so we could do it all in 1 PR.

    Sure! i will add the commits, but it will take me some time o7

  11. DrahtBot added the label CI failed on Oct 8, 2026
  12. DrahtBot commented at 10:15 PM on October 8, 2026: contributor

    <!--85328a0da195eb286784d51f73fa0af9-->

    🚧 At least one of the CI tasks failed. <sub>Task iwyu: https://github.com/bitcoin/bitcoin/actions/runs/37841926848/job/113533161262</sub> <sub>LLM reason (✨ experimental): CI failed because the IWYU lint check modified three files, leaving uncommitted include changes that caused git diff --exit-code to fail.</sub>

    <details><summary>Hints</summary>

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

    </details>

  13. andrewtoth commented at 10:15 PM on October 8, 2026: contributor

    Sorry @andrewtoth, i wasnt aware of it 😅 , should i compare with ur code and add u as co author?

    No worries, it was nothing I published. Let's just keep going with this. I don't need to be added as co-author.

    it will take me some time

    On second thought, if we merge #36002 first we can reuse the TxLookupResult (sorry I have to push an update there)

  14. arejula27 commented at 10:53 PM on October 8, 2026: contributor

    Sounds good to me, i will wait an then add here TxLookupResult !

  15. l0rinc commented at 5:58 AM on October 9, 2026: contributor

    Concept ACK 👍 for pruning support and also for merging #36002 first - if you agree, please convert this to draft until then so reviewers know which PRs to prioritize.

  16. arejula27 marked this as a draft on Oct 9, 2026
  17. w0xlt commented at 6:07 AM on October 15, 2026: contributor

    Concept ACK


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-10-19 04:51 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me