txindex: hash keys and pack positions to reduce disk usage #35531

pull andrewtoth wants to merge 5 commits into bitcoin:master from andrewtoth:txindex_optimization changing 7 files +364 −24
  1. andrewtoth commented at 10:33 PM on June 14, 2026: contributor

    The current txindex uses the full 32-byte txid as keys, which takes up about 66 GB of disk space today on mainnet. Using a 5-byte key prefix instead drops the disk usage to 27 GB - cutting the size to less than half.

    Using the full 32-bytes is unnecessary since a 5-byte salted siphash will produce collisions in about 1 in 1.1 trillion. Some collisions will occur, but the penalty is just an extra disk read, deserialization and hash. The tx position can be appended to the key instead of used as a value, and a LevelDB iterator can seek to the prefix and then scan for the correct tx. This is an almost identical approach to txospenderindex.

    Also instead of storing the file position of the block, we can store only the block height and direct file position of the transaction. This can be packed into a single integer by multiplying the height by the max serialized block size and then adding the position. The height can be recovered by dividing by the max block size and the position can be recovered by modulo the max block size. This is a more compact representation, and the block file can be recovered by the CBlockIndex that is already in memory.

    However, this requires us to erase tx entries that are no longer part of the best chain. This is a breaking change, since today users can still look up txs in blocks that have been reorged out of the best chain.

    If a tx is not found with this method, we fallback to looking up the legacy entry. With this method a user with an existing db can opt to erase the indexes/txindex folder and reindex, or keep the current index and new entries will be appended with the smaller footprint.

    The time to index was faster on my machine with this method, 1h23m vs current 1h50m. Lookups are roughly the same, around 0.2ms per lookup with getrawtransaction. When testing on mainnet, I got ~860k 2-way collisions, and 1 3-way collision that worst case could cause an extra 2 false positives when reading.

  2. DrahtBot commented at 10:33 PM on June 14, 2026: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35531.

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    Concept ACK optout21, theStack, sedited, l0rinc

    If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #35587 (Remove boost as a unit test runner by rustaceanrob)
    • #35568 (txospenderindex: disable bloom filters to optimize disk usage by andrewtoth)
    • #35474 (node: move index ownership to NodeContext by w0xlt)
    • #34132 (coins: drop error catcher, centralize fatal read handling by l0rinc)
    • #33324 (blocks: add -reobfuscate-blocks argument to enable (de)obfuscating existing blocks by l0rinc)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

  3. andrewtoth renamed this:
    txindex: use siphash keys to optimize disk usage
    txindex: use 8-byte siphash keys to optimize disk usage
    on Jun 14, 2026
  4. sipa commented at 10:43 PM on June 14, 2026: member

    With 1.376e9 transactions in total, the chance of having at least one collision if each is given a 64-bit random identifier, is around 5%.

  5. andrewtoth commented at 10:46 PM on June 14, 2026: contributor

    @sipa Yes, apologies if I was not clear in the PR description, but collisions are handled with this change. When testing I had 2 sets of 2 txs that collided.

  6. sipa commented at 10:51 PM on June 14, 2026: member

    Oh, I see! Sorry, I saw the number of 1 in 18.4 quintillion and jumped to conclusions.

    Neat. You're essentially treating the database as a set of (txid siphash, tx data) pairs, rather than as a (txid siphash) -> (tx data) map, so it functions as a multimap instead.

  7. andrewtoth commented at 10:55 PM on June 14, 2026: contributor

    Thanks! Yes, collisions are rare enough that this will not have a noticeable read penalty. A collision may incur an extra disk read, deserialization and sha256 hash.

  8. sipa commented at 11:53 PM on June 14, 2026: member

    If existence of collisions isn't the criterion to judge this by, but the expected read amplification from those collisions, we can possibly go even lower?

    With 5-byte salted hashes, the expected amplication factor is under 1.002 per read, up to 2 billion txids, and would save a few extra gigabytes. That might even be a bigger speed win than the loss from that amplication. 4-byte hashes would give an amplication that's probably too much.

  9. andrewtoth force-pushed on Jun 15, 2026
  10. andrewtoth renamed this:
    txindex: use 8-byte siphash keys to optimize disk usage
    txindex: use 5-byte siphash keys to optimize disk usage
    on Jun 15, 2026
  11. andrewtoth commented at 2:30 AM on June 15, 2026: contributor

    Updated to use 5-byte siphash. The reindex was faster by 8 minutes, and the db size is now 32 GB. I got 860k 2-way collisions now though, instead of 2 before, and I got 1 3-way collision. The read penalty was not really measurable from my machine though, likely because I am using a fast laptop with fast directly connected storage.

  12. l0rinc commented at 10:40 AM on June 15, 2026: contributor

    Concept ACK, will play with this after we're done with the compactions

  13. in src/index/txindex.cpp:53 in e3f35ee417
      48 | +    return TxHashKeyPrefix{
      49 | +        static_cast<uint8_t>(siphash >> 56),
      50 | +        static_cast<uint8_t>(siphash >> 48),
      51 | +        static_cast<uint8_t>(siphash >> 40),
      52 | +        static_cast<uint8_t>(siphash >> 32),
      53 | +        static_cast<uint8_t>(siphash >> 24),
    


    optout21 commented at 11:05 AM on June 15, 2026:

    e3f35ee txindex: use 5-byte siphash keys to optimize disk usage:

    Can the const offsets be given in hex? Structure shows better through the hex numbers 18, 20, 28, 30, 38.


    optout21 commented at 11:08 AM on June 15, 2026:

    e3f35ee txindex: use 5-byte siphash keys to optimize disk usage:

    I may be shooting in the dark, but could it be that doing the right shifts incrementally (i.e., first by 24, then 4 times by 8; placed in reverse order), is slightly more efficient? (shifts with smaller size; a micro-optimization).


    andrewtoth commented at 1:03 AM on June 16, 2026:

    Done.


    andrewtoth commented at 1:04 AM on June 16, 2026:

    Hmm not sure that could really make a difference that would be visible to a user or to benchmarks here?


    optout21 commented at 11:55 AM on June 16, 2026:

    I was able to measure only minimal differences in performance. I microbenchmarked only the shift operations. The result was 3% speedup, or 0.000486 microsec per iteration (1518827 microsec vs 1470169, for 100000000 iterations). I think this is negligible.

    Please set this thread to Resolved.

    The two versions compared were:

                buf[0] = static_cast<uint8_t>(siphash >> 0x38);
                buf[1] = static_cast<uint8_t>(siphash >> 0x30);
                buf[2] = static_cast<uint8_t>(siphash >> 0x28);
                buf[3] = static_cast<uint8_t>(siphash >> 0x20);
                buf[4] = static_cast<uint8_t>(siphash >> 0x18);
    
                siphash >>= 24;
                buf[4] = static_cast<uint8_t>(siphash);
                siphash >>= 8;
                buf[3] = static_cast<uint8_t>(siphash);
                siphash >>= 8;
                buf[2] = static_cast<uint8_t>(siphash);
                siphash >>= 8;
                buf[1] = static_cast<uint8_t>(siphash);
                siphash >>= 8;
                buf[0] = static_cast<uint8_t>(siphash);
    

    sipa commented at 12:12 PM on June 16, 2026:

    If the CPU performance of constructing the hash is at all relevant (I don't know), we could consider a faster hash function (like #35215).


    andrewtoth commented at 12:36 PM on June 16, 2026:

    we could consider a faster hash function (like #35215).

    The unfortunate thing about this use case is that we would have to decide on this before a release, since changing the hash function after the fact would require a reindex.


    sipa commented at 1:26 PM on June 16, 2026:

    Indeed, changing it is painful once released.

    Just to assess whether that's worth investigating at all, would someone benchmark with and without the simplified siphash there?


    andrewtoth commented at 1:41 PM on June 16, 2026:

    I was looking to do that :). But, the hash function there is optimized for COutPoint, which has an extra 4 bytes after the uint256. @l0rinc is that sufficiently faster if we just pass a constant as the extra, or is there a more optimized version we can run that omits the extra field?


    optout21 commented at 1:48 PM on June 16, 2026:

    Yes, #35215 was optimization for the extra vout 8 bytes, which is not the case here. A speedup could be obtained with a hash that internally works with 5-byte-only values, but on 64-bit architecture that's not really faster than operations with 8-byte values, so I don't see an easy win here.

    What could be measured as a boundary data point is just taking 5 bytes of the TXID without any extra hashing/salting, and if the speedup is significant, considered.


    sipa commented at 1:50 PM on June 16, 2026:

    I think you have it backwards, @optout21.

    We're discussing replacing the SipHash function used to compute the 5-byte value; it's not using it as an input.

    The input we need here is the txid. In #35215 the input is a COutPoint.


    l0rinc commented at 1:52 PM on June 16, 2026:

    @l0rinc is that sufficiently faster if we just pass a constant as the extra, or is there a more optimized version we can run that omits the extra field?

    We can add a versions without the extra of course. I will help with benchmarking this after the compaction work is behind us - unless you think this is more urgent for some reasonn.


    andrewtoth commented at 1:53 PM on June 16, 2026:

    A tangent but this now got me thinking, if it works well for txindex, we could extend this method to chainstate. siphash the CoinEntry as key prefix, append the Coin to the key, and seek to the prefix and scan for the right outpoint.


    sipa commented at 1:54 PM on June 16, 2026:

    @andrewtoth I think taking the same design approach into account, we can drop 1 extra SipHash round from the construction in #35215 if there is no extra uint32_t to add, so 4 instead of 5 rounds (compared to 14 rounds with traditional SipHash-2-4).


    l0rinc commented at 1:56 PM on June 16, 2026:

    Yes, that's what I mean. I don't mind adding it to #35215 if you think it's a good idea.


    andrewtoth commented at 2:06 PM on June 16, 2026:

    seek to the prefix and scan for the right outpoint.

    Actually that won't work, because we don't have the outpoint we can reconstruct like we can the txid from the transaction we read.


    optout21 commented at 3:48 PM on June 16, 2026:

    We're discussing replacing the SipHash function used to compute the 5-byte value

    Yes, I didn't mean otherwise. My point was (maybe not clearly expressed) that internally SipHash works with 8-byte values, which is a waste, if in the end only a 5-byte hash is needed. A custom version working with 5-byte values is conceivable, but since 64-bit CPUs are optimized for 64 bit width, it probably wouldn't be faster.

    My other point was that to get an upper bound on the speedup possible through tweaking the hash, it's possible to measure an oversimplified solution, where no hash is used at all, but 5 bytes are taken directly from the 32-byte TXID (which is itself a hash). I'm not saying that this would be an acceptable solution (thought it might), but it would be faster than any optimized hash (to get the 5 bytes).


    sipa commented at 4:14 PM on June 16, 2026:

    @optout21 I see what you mean now, but that is not what we're talking about.

    SipHash, and all variants of it, internally work with 64-bit values, that's not going to change. It would be an entirely distinct hash function, which needs separate analysis, to change that. All designs just truncate the final 64-bit value to 40-bit.

    What is being discussed now is independent from the 8/5 byte question. The idea is that, since this PR bites the bullet in changing the data layout to be hash-based, we might as well pick an efficient hash function. Right now, this PR uses normal SipHash-2-4. @l0rinc's PR linked above introduces a more experimental (and custom) variant of SipHash-1-3 with jumboblocks and without padding. Using that same variant here would mean a construction that only needs 4 SipHash rounds (all operating on four internal 64-bit values) rather than 14 SipHash rounds. This same change would be possible even if this PR used 8-byte ids.



    andrewtoth commented at 11:33 PM on June 18, 2026:

    Ran some benchmarks, 2 runs each of master, siphash24, jumbo siphash13, and just taking the first 5 bytes (unsafe, but the best we can expect to get).

    Siphash24 is 13 minutes faster than master, and siphash13 is another 90 seconds faster than that. The raw first 5 byte variant was only 11s faster than siphash13, so we're very close to the theoretical limit.

    I have a very fast CPU, so a slower machine might show a bigger speedup with siphash13. An extra 90 seconds though doesn't seem like a deal breaker for me though. If we can get the siphash13 in before this is released we should definitely take it though.

    Variant min sync avg sync max sync avg (h:m:s) size (GiB) vs master
    master (full 32B txid) 6286s 6355s 6424s 1h45m55s 65.85
    normal (5B SipHash-2-4) 5550s 5578s 5607s 1h32m58s 32.18 −12.2% time
    jumbo (5B SipHash-1-3) 5437s 5488s 5538s 1h31m27s 32.07 −13.7% time
    raw5 (first 5B of txid) 5417s 5476s 5536s 1h31m16s 32.04 −13.8% time
  14. optout21 commented at 11:37 AM on June 15, 2026: contributor

    Concept ACK (e3f35ee4171875124104b404464151f9b3da1566)

    The reduction of index size is very positive!

    The approach for graceful DB update is interesting, it's nice that reindex is not forced.

    Is there a benchmark that exposes the effect of this change?

    Out of curiosity, what was the number of bytes used before switching to 5? 8? And the resulting size? (unfortunately the original version & values were not preserved in the description).

  15. andrewtoth force-pushed on Jun 16, 2026
  16. andrewtoth commented at 1:11 AM on June 16, 2026: contributor

    @optout21 the initial run was using an 8-byte siphash. The db was 36 GB and it took 1h40m.

    Is there a benchmark that exposes the effect of this change?

    The goal was to shrink the db size on disk, but of course if performance was negatively affected it might not be worth it. There are two relevant metrics - syncing the index and reading entries. It seems the former gets a nice bump from this as well. I am assuming it's mostly from the fact that we write a lot less data to disk.

    Benchmarking the sync speed can be done by deleting the /indexes/txindex directory in the datadir, and then restarting and waiting for the txindex is enabled at height log line. This can be compared to the txindex thread start log line to get the delta.

    Benchmarking the read speed can be done with the following script using apache bench:

    TXID="<txid>"
    printf '{"jsonrpc":"1.0","id":"ab","method":"getrawtransaction","params":["%s",0]}\n' "$TXID" > /tmp/data.json
    ab -n 10000 -c 1 -k -A user:password -p /tmp/data.json -T application/json http://127.0.0.1:8332/
    
  17. andrewtoth force-pushed on Jun 16, 2026
  18. DrahtBot added the label CI failed on Jun 16, 2026
  19. DrahtBot commented at 1:19 AM on June 16, 2026: contributor

    <!--85328a0da195eb286784d51f73fa0af9-->

    🚧 At least one of the CI tasks failed. <sub>Task iwyu: https://github.com/bitcoin/bitcoin/actions/runs/27586700495/job/81558529020</sub> <sub>LLM reason (✨ experimental): CI failed because the IWYU (include-what-you-use) check detected missing includes and forced a formatting/patch to src/index/txindex.cpp, causing the job to exit non-zero.</sub>

    <details><summary>Hints</summary>

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

    </details>

  20. andrewtoth force-pushed on Jun 16, 2026
  21. DrahtBot removed the label CI failed on Jun 16, 2026
  22. theStack commented at 2:57 PM on June 16, 2026: contributor

    Concept ACK

  23. mzumsande commented at 5:18 PM on June 16, 2026: contributor

    In case of a downgrade, the behavior is not ideal. Nothing will break and there will be no warnings or errors, but some transactions won't be returned after getrawtransaction queries even though they exist, while others will be returned normally - depending on which version indexed them.

    So I think that this would need to be documented well. Alternatively, an upgrade similar to how it was done with coinstatsindex in 30.0 might be cleaner - especially since the txindex syncs very fast in comparison.

  24. andrewtoth commented at 10:11 PM on June 16, 2026: contributor

    @mzumsande good observation.

    We could make ReadBestBlock/WriteBestBlock virtual, and override them in txindex to use a new locator, say Bv2, that is used to track the latest block. If Bv2 isn't found, lookup B and start from there. That way a downgrade will ignore newer hashed entries and resync from where the latest legacy entries are.

  25. in src/index/txindex_key.h:19 in 94d7377eda
      14 | +#include <cstddef>
      15 | +#include <cstdint>
      16 | +#include <ios>
      17 | +
      18 | +namespace txindex {
      19 | +constexpr uint8_t DB_TXINDEX_HASHED{'T'};
    


    optout21 commented at 9:22 AM on June 17, 2026:

    94d7377 txindex: use 5-byte siphash keys to optimize disk usage:

    Nit: The 'T' could be confused with the legacy 't', discussing/debugging/etc. in a mixed legacy-new DB environment, maybe a different letter could be picked, to reduce the risk of confusion.


    andrewtoth commented at 12:19 AM on June 19, 2026:

    Updated to 'x'.

  26. optout21 commented at 9:25 AM on June 17, 2026: contributor

    LGTM! Reviewed code, tested lightly locally, including reindex (upgrade and downgrade). Not ack'ing now, as I can see the pending points:

    • Optimized hash from #35215. This is not a must, can do without (but if it lands earlier, it should be taken; if not, can be done later.)
    • Discussion about downgrade scenario.
  27. sedited commented at 9:51 AM on June 17, 2026: contributor

    Concept ACK

    As for the fallback, I'm curious how much slower this makes transaction querying. Do you think a forced migration to the new DB would be too expensive?

  28. andrewtoth commented at 2:13 PM on June 18, 2026: contributor

    @sedited I don't think the new queries are noticeably slower. It's one more read of a non-existent entry.

    We could do a migration like coinstatsindex in v30, where we keep the old db and reindex in a new directory as @mzumsande suggested. I assumed it would be better to have a graceful upgrade, but I suppose every user will want to reindex to reap the disk savings. In that case, we can write release notes to tell users to wipe their old txindex directory if they are not planning on downgrading? This path will also make the code changes simpler since we don't have to support upgrade and downgrade.

    What does everyone think - support graceful upgrade/downgrade, or just index into a new directory?

  29. l0rinc commented at 2:21 PM on June 18, 2026: contributor

    What does everyone think - support graceful upgrade/downgrade, or just index into a new directory?

    Wouldn't on-demand migration provide the possibility to do both, as you mentioned? Start immediately and the system will fall back to amortized O(1) migration, or delete everything and do an O(n) migration? (note that I still haven't reviewed it in detail, only responding to the question)

  30. mzumsande commented at 2:36 PM on June 18, 2026: contributor

    What does everyone think - support graceful upgrade/downgrade, or just index into a new directory?

    The exact same procedure would probably not be a good idea - if we'd keep the old index with its 66GB around by default, users who don't do anything manually (which are probably most) would experience a increase in disk space because they would have both. That was not a problem for coinstatsindex because it is much smaller.

  31. optout21 commented at 2:53 PM on June 18, 2026: contributor

    The graceful upgrade without forced reindex is very user-friendly (at the price of hybrid data in the DB, and logic to try to read both). The downgrade being the less frequent use case, I think it's acceptable to require a reindex in that case. However, the question is how to prevent the old code from using the existing DB. Maybe at graceful upgrade rename the index DB (but keep its content), so the old code will not find it. Just an idea.

  32. sedited commented at 7:16 PM on June 18, 2026: contributor

    What does everyone think - support graceful upgrade/downgrade, or just index into a new directory?

    I think what you have now is fine tbh. I asked the question before because I was curious to hear some of your reasoning.

  33. andrewtoth force-pushed on Jun 19, 2026
  34. andrewtoth force-pushed on Jun 19, 2026
  35. DrahtBot added the label CI failed on Jun 19, 2026
  36. DrahtBot commented at 12:29 AM on June 19, 2026: contributor

    <!--85328a0da195eb286784d51f73fa0af9-->

    🚧 At least one of the CI tasks failed. <sub>Task No wallet: https://github.com/bitcoin/bitcoin/actions/runs/27797474238/job/82260299474</sub> <sub>LLM reason (✨ experimental): CI failed due to a Clang -Werror compile error: deleting BaseIndex::DB with virtual functions but a non-virtual destructor (-Wdelete-non-abstract-non-virtual-dtor).</sub>

    <details><summary>Hints</summary>

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

    </details>

  37. andrewtoth commented at 12:36 AM on June 19, 2026: contributor

    Thanks for all your responses. Seems we're all in agreement about keeping graceful upgrades and downgrades.

    I pushed a commit 736410a1739fe8fd5c7c4140929d115ace6475ee that updates the block locator for txindex, and reads the legacy block locator if the new version is not yet written. This lets us continue to use the legacy index when upgrading, and an older node will ignore hashed entries when downgrading and resync from where the legacy locator left off.

    If a user wipes their txindex and resyncs with the new hashed entries, then downgrades, they will reindex legacy again and have both indexes in one db. I think that's ok though, we should just mention in the release notes to wipe the db only if you don't plan on downgrading.

  38. DrahtBot removed the label CI failed on Jun 19, 2026
  39. Sjors commented at 1:06 PM on June 19, 2026: member

    You could further shrink the index by letting go of the requirement that that -txindex also indexes the block (via CDiskTxPos.nPos). Pointing directly to the file position in the block file shaves off a few bytes. You could encode the block height, which still saves some bytes compared to the current approach, but then you don't know if a transaction is in the best chain or orphaned. This may not be desirable, so it's probably best to keep CDiskTxPos as is.

  40. andrewtoth commented at 2:51 PM on June 19, 2026: contributor

    @Sjors yeah, if we use the height we could give a wrong block hash back for an orphaned tx. Will leave as is.

    One thing we could do to shave off some more GBs is remove bloom filters, a la #35568, but that will likely slow down the legacy lookups. If we didn't care about graceful upgrading we could do that.

  41. Sjors commented at 3:06 PM on June 19, 2026: member

    @andrewtoth you could conditionally disable it for newly created indexes? The release note could mention that disk space savings can be achieved by deleting the existing index.

  42. optout21 commented at 3:46 PM on June 19, 2026: contributor

    @Sjors, do you mean that the two positions could be merged? There are two uint32_t positions: nPos is the offset of the block within the file, and nTxOffset is the offset of the TX within the block. There are also two seeks to get to the correct position.

  43. andrewtoth commented at 3:52 PM on June 19, 2026: contributor

    @optout21 I believe the suggestion is to just have one position which points to the position of the transaction in the file, not the block. Then, we can also encode the height of the block in 3 bytes and look up the hash in CBlockIndex. But, the nPos and nTxOffset are both encoded as varints, so this might not actually save much space with the extra 3 bytes.

  44. andrewtoth commented at 4:01 PM on June 19, 2026: contributor

    you could conditionally disable it for newly created indexes @Sjors good idea, we could skip bloom filters and legacy lookups if we don't see any 't' entries in the db. We would have to peek inside it before opening it though, since the bloom filters are a startup option. I think I might leave that for a follow-up to add to #35568 if this gets merged.

  45. l0rinc commented at 7:48 PM on June 19, 2026: contributor

    Q: Could this key format change consider future prune compatibility? It's probably out of scope, but maybe worth taking into account here, since if the index could identify the containing block independently of local block files, a pruned node (with the header chain) could potentially try to fetch that block on demand in a follow-up.

  46. Sjors commented at 7:54 PM on June 19, 2026: member

    @l0rinc we could, if we keep the original CDiskTxPos.nPos around for pruned blocks, we can infer which block we're missing.

  47. andrewtoth commented at 7:57 PM on June 19, 2026: contributor

    if the index could identify the containing block independently of local block files

    Interesting idea. If we did store the block height instead of file position, we could do that. But, we would need to solve the problem of finding the right block if the tx requested is in a block not part of the best chain.

    if we keep the original CDiskTxPos.nPos around for pruned blocks, we can infer which block we're missing. @Sjors interesting, can you elaborate?

  48. sipa commented at 8:08 PM on June 19, 2026: member

    We could store a single number, 4000000*block_height + tx_offset.

    It can even use an integer encoding without length prefix, as that can be inferred from the length of the serialized value record.

    Shouls be 6 bytes for the forseeable future on mainnet.

    This does imply erasing entries for reorged/disconnected blocks, as otherwise we'll have nonsensical tx offsets.

  49. andrewtoth commented at 8:29 PM on June 19, 2026: contributor

    erasing entries for reorged/disconnected blocks

    ~hmm that could require us to do manual compactions on the txindex db then...~ Actually, these entries can be overwritten already on reorgs, so it doesn't change current behavior.

  50. andrewtoth force-pushed on Jun 24, 2026
  51. andrewtoth renamed this:
    txindex: use 5-byte siphash keys to optimize disk usage
    txindex: hash keys and pack positions to reduce disk usage
    on Jun 24, 2026
  52. andrewtoth force-pushed on Jun 24, 2026
  53. DrahtBot added the label CI failed on Jun 24, 2026
  54. DrahtBot commented at 1:07 AM on June 24, 2026: contributor

    <!--85328a0da195eb286784d51f73fa0af9-->

    🚧 At least one of the CI tasks failed. <sub>Task iwyu: https://github.com/bitcoin/bitcoin/actions/runs/28066837312/job/83092910222</sub> <sub>LLM reason (✨ experimental): CI failed because IWYU detected and required missing/incorrect #include fixes in src/index/txindex.cpp (triggering “Failure generated from IWYU”).</sub>

    <details><summary>Hints</summary>

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

    </details>

  55. andrewtoth commented at 1:20 AM on June 24, 2026: contributor

    @sipa I updated this with your suggestion to encode the block height and transaction file position using max serialized block size as a mask. Thanks!

    This reduced the db to 29GB, and it synced in 1h27m :rocket:

    So we must also delete entries when blocks are disconnected now. This makes the diff a little bigger to review, but in return it opens the door to allow txindex with pruning. We could fetch blocks JIT from peers when a tx is requested from a missing block. cc @l0rinc

  56. DrahtBot removed the label CI failed on Jun 24, 2026
  57. Sjors commented at 9:26 AM on June 24, 2026: member

    IIUC we lose the ability find transactions in stale blocks (that are not included in the canonical chain). Test that passes before and fails after this PR:

    diff --git a/test/functional/rpc_rawtransaction.py b/test/functional/rpc_rawtransaction.py
    index 78e12139fc..4e4f85d1f8 100755
    --- a/test/functional/rpc_rawtransaction.py
    +++ b/test/functional/rpc_rawtransaction.py
    @@ -172,4 +172,14 @@ class RawTransactionsTest(BitcoinTestFramework):
                 gottx = self.nodes[n].getrawtransaction(txid=tx, verbose=True, blockhash=block1)
                 assert_equal(gottx['in_active_chain'], False)
    +            if n == 0:
    +                self.log.info("Test getrawtransaction with -txindex can find a stale block transaction without blockhash")
    +                coinbase_txid = self.nodes[n].getblock(block1)["tx"][0]
    +                # Mine another block so txindex processes the reorg and calls CustomRemove()
    +                # for the stale block before querying it.
    +                self.generate(self.nodes[n], 1, sync_fun=self.no_op)
    +                sync_txindex(self, self.nodes[n])
    +                raw_tx = self.nodes[n].getrawtransaction(txid=coinbase_txid, verbose=True)
    +                assert_equal(raw_tx["txid"], coinbase_txid)
    +                assert_equal(raw_tx["blockhash"], block1)
                 self.nodes[n].reconsiderblock(block1)
                 assert_equal(self.nodes[n].getbestblockhash(), block2)
    

    It's worth pointing that out in the description. It might be fine, but we could preserve this functionality by switching the key to use the block hash upon disconnect, instead of erasing:

    <details> <summary>patch</summary>

    diff --git a/src/index/txindex.cpp b/src/index/txindex.cpp
    index e1f42d1fa7..b010161667 100644
    --- a/src/index/txindex.cpp
    +++ b/src/index/txindex.cpp
    @@ -40,4 +40,24 @@ const std::string DB_BEST_BLOCK_V2{"best_block_v2"};
     std::unique_ptr<TxIndex> g_txindex;
    
    +namespace txindex {
    +constexpr uint8_t DB_TXINDEX_STALE{'y'};
    +
    +struct StaleDBKey {
    +    TxHashKeyPrefix hash_prefix;
    +    uint256 block_hash{};
    +    uint32_t tx_offset{0};
    +
    +    SERIALIZE_METHODS(StaleDBKey, obj)
    +    {
    +        uint8_t prefix{DB_TXINDEX_STALE};
    +        READWRITE(prefix);
    +        if (prefix != DB_TXINDEX_STALE) {
    +            throw std::ios_base::failure("Invalid format for stale txindex DB key");
    +        }
    +        READWRITE(obj.hash_prefix, obj.block_hash, obj.tx_offset);
    +    }
    +};
    +} // namespace txindex
    +
    
     /** Access to the txindex database (indexes/txindex/) */
    @@ -51,5 +71,5 @@ public:
         bool ReadTxPos(const Txid& txid, CDiskTxPos& pos) const;
    
    -    /// Write or erase a block of transaction positions to the DB.
    +    /// Write active entries or move active entries to stale entries for a block.
         void WriteTxs(const interfaces::BlockInfo& block, bool erase = false);
    
    @@ -99,8 +119,9 @@ void TxIndex::DB::WriteTxs(const interfaces::BlockInfo& block, bool erase)
         uint32_t tx_offset{GetSizeOfCompactSize(block.data->vtx.size())};
         for (const auto& tx : block.data->vtx) {
    -        const txindex::DBKey key{txindex::CreateKeyPrefix(m_hasher, tx->GetHash()),
    -                                 txindex::Position{static_cast<uint32_t>(block.height), tx_offset}};
    +        const txindex::TxHashKeyPrefix hash_prefix{txindex::CreateKeyPrefix(m_hasher, tx->GetHash())};
    +        const txindex::DBKey key{hash_prefix, txindex::Position{static_cast<uint32_t>(block.height), tx_offset}};
             if (erase) {
                 batch.Erase(key);
    +            batch.Write(txindex::StaleDBKey{hash_prefix, block.hash, tx_offset}, "");
             } else {
                 batch.Write(key, "");
    @@ -149,15 +170,6 @@ bool TxIndex::FindTx(const Txid& tx_hash, uint256& block_hash, CTransactionRef&
         txindex::DBKey key{prefix, {}};
         const auto header_offset{static_cast<uint32_t>(GetSerializeSize(CBlockHeader{}))};
    -    for (; it->Valid() && it->GetKey(key) && key.hash_prefix == prefix; it->Next()) {
    -        FlatFilePos tx_pos;
    -        uint256 candidate_block_hash;
    -        {
    -            LOCK(cs_main);
    -            const CBlockIndex* pindex{m_chainstate->m_chain[key.pos.block_height]};
    -            if (!pindex) continue;
    -            tx_pos = FlatFilePos{pindex->nFile, pindex->nDataPos + header_offset + key.pos.tx_offset};
    -            candidate_block_hash = pindex->GetBlockHash();
    -        }
    -        AutoFile file{m_chainstate->m_blockman.OpenBlockFile(tx_pos, true)};
    +    const auto read_tx_at_pos{[&](const FlatFilePos& pos) {
    +        AutoFile file{m_chainstate->m_blockman.OpenBlockFile(pos, true)};
             if (file.IsNull()) {
                 LogError("OpenBlockFile failed");
    @@ -170,4 +182,17 @@ bool TxIndex::FindTx(const Txid& tx_hash, uint256& block_hash, CTransactionRef&
                 return false;
             }
    +        return true;
    +    }};
    +    for (; it->Valid() && it->GetKey(key) && key.hash_prefix == prefix; it->Next()) {
    +        FlatFilePos tx_pos;
    +        uint256 candidate_block_hash;
    +        {
    +            LOCK(cs_main);
    +            const CBlockIndex* pindex{m_chainstate->m_chain[key.pos.block_height]};
    +            if (!pindex) continue;
    +            tx_pos = FlatFilePos{pindex->nFile, pindex->nDataPos + header_offset + key.pos.tx_offset};
    +            candidate_block_hash = pindex->GetBlockHash();
    +        }
    +        if (!read_tx_at_pos(tx_pos)) return false;
             if (tx->GetHash() == tx_hash) {
                 block_hash = candidate_block_hash;
    @@ -176,4 +201,21 @@ bool TxIndex::FindTx(const Txid& tx_hash, uint256& block_hash, CTransactionRef&
         }
    
    +    it->Seek(std::pair{txindex::DB_TXINDEX_STALE, prefix});
    +    txindex::StaleDBKey stale_key{prefix};
    +    for (; it->Valid() && it->GetKey(stale_key) && stale_key.hash_prefix == prefix; it->Next()) {
    +        FlatFilePos tx_pos;
    +        {
    +            LOCK(cs_main);
    +            const CBlockIndex* pindex{m_chainstate->m_blockman.LookupBlockIndex(stale_key.block_hash)};
    +            if (!pindex || !(pindex->nStatus & BLOCK_HAVE_DATA)) continue;
    +            tx_pos = FlatFilePos{pindex->nFile, pindex->nDataPos + header_offset + stale_key.tx_offset};
    +        }
    +        if (!read_tx_at_pos(tx_pos)) return false;
    +        if (tx->GetHash() == tx_hash) {
    +            block_hash = stale_key.block_hash;
    +            return true;
    +        }
    +    }
    +
         // Fallback to legacy if no hashed entry matched.
         CDiskTxPos postx;
    diff --git a/src/test/txindex_tests.cpp b/src/test/txindex_tests.cpp
    index 18913291b8..a16899d631 100644
    --- a/src/test/txindex_tests.cpp
    +++ b/src/test/txindex_tests.cpp
    @@ -180,5 +180,5 @@ BOOST_FIXTURE_TEST_CASE(txindex_collision_scan_path, TestChain100Setup)
     }
    
    -BOOST_FIXTURE_TEST_CASE(txindex_reorg_erases_entries, TestChain100Setup)
    +BOOST_FIXTURE_TEST_CASE(txindex_reorg_keeps_stale_entries, TestChain100Setup)
     {
         TxIndex txindex(interfaces::MakeChain(m_node), 1_MiB, true);
    @@ -205,4 +205,5 @@ BOOST_FIXTURE_TEST_CASE(txindex_reorg_erases_entries, TestChain100Setup)
         BOOST_REQUIRE(txindex.FindTx(unique_txid, block_hash, tx_disk));
         BOOST_CHECK(tx_disk->GetHash() == unique_txid);
    +    const uint256 stale_block_hash{block_hash};
    
         ChainstateManager& chainman{*m_node.chainman};
    @@ -218,6 +219,8 @@ BOOST_FIXTURE_TEST_CASE(txindex_reorg_erases_entries, TestChain100Setup)
         BOOST_REQUIRE(txindex.BlockUntilSyncedToCurrentChain());
    
    -    // The disconnected transaction's entry must have been erased.
    -    BOOST_CHECK(!txindex.FindTx(unique_txid, block_hash, tx_disk));
    +    // The disconnected transaction must still be found through the stale block entry.
    +    BOOST_REQUIRE(txindex.FindTx(unique_txid, block_hash, tx_disk));
    +    BOOST_CHECK(tx_disk->GetHash() == unique_txid);
    +    BOOST_CHECK(block_hash == stale_block_hash);
    
         txindex.Stop();
    

    </details>

    This is arguably wasteful for transactions that are included in the canonical chain, but we could (later) expand the RPC to be able to find them. Stale blocks are pretty rare on mainnet anyway, so it's not much extra data.

  58. andrewtoth commented at 1:52 PM on June 24, 2026: contributor

    we lose the ability find transactions in stale blocks @Sjors I updated the PR description to note that breaking change. There is a unit test for this new functionality.

    It might be fine, but we could preserve this functionality by switching the key to use the block hash upon disconnect, instead of erasing:

    What should we do here? I think (could be wrong) that's not really a feature many users use, and users who are syncing the index from scratch also don't get that feature (since the initial sync won't index non-canonical blocks) so it's not really deterministic anyways. IMO it would be fine to remove that functionality and address it in a release note.

    Users can also work around this by just passing the block hash to getrawtransaction, as you note it only breaks if there's no block hash.

  59. Sjors commented at 2:51 PM on June 24, 2026: member

    updated the PR description to note that breaking change

    Let's also mention it in the release note.

    not really a feature many users use

    Probably not. I could imagine it's useful for external wallet / lightning software, to know that a given transaction now lives in a stale block. But as long as they kept track of the block hash, they don't need the index.

    It might not hurt to (have an agent) scan for projects that use -txindex to see if any rely on this behavior.

    so it's not really deterministic anyways

    That and the extra complexity seem good reasons to not bother supporting it.

  60. andrewtoth force-pushed on Jun 25, 2026
  61. andrewtoth commented at 2:22 AM on June 25, 2026: contributor

    Thanks @Sjors, added release notes.

  62. in src/index/txindex.cpp:67 in 1d5b61c400
      66 |  TxIndex::DB::DB(size_t n_cache_size, bool f_memory, bool f_wipe) :
      67 | -    BaseIndex::DB(gArgs.GetDataDirNet() / "indexes" / "txindex", n_cache_size, f_memory, f_wipe)
      68 | +    BaseIndex::DB(gArgs.GetDataDirNet() / "indexes" / "txindex", n_cache_size, f_memory, f_wipe),
      69 | +    m_hasher{[](CDBWrapper& db) {
      70 | +        std::pair<uint64_t, uint64_t> siphash_key;
      71 | +        if (!db.Read("siphash_key", siphash_key)) {
    


    l0rinc commented at 9:18 PM on June 26, 2026:

    1d5b61c txindex: hash keys and pack positions to reduce disk usage:

    could we make siphash_key named by its role, not the implementation detail? The siphash doc calls them keys but for the dbcache we're calling them salts:

    static constexpr std::string DB_TXID_HASH_SALT{"txid_hash_salt"};
    
  63. in src/index/txindex.cpp:68 in 1d5b61c400 outdated
      67 | -    BaseIndex::DB(gArgs.GetDataDirNet() / "indexes" / "txindex", n_cache_size, f_memory, f_wipe)
      68 | +    BaseIndex::DB(gArgs.GetDataDirNet() / "indexes" / "txindex", n_cache_size, f_memory, f_wipe),
      69 | +    m_hasher{[](CDBWrapper& db) {
      70 | +        std::pair<uint64_t, uint64_t> siphash_key;
      71 | +        if (!db.Read("siphash_key", siphash_key)) {
      72 | +            FastRandomContext rng{};
    


    l0rinc commented at 9:19 PM on June 26, 2026:

    1d5b61c txindex: hash keys and pack positions to reduce disk usage:

    Do we ever need to make this deterministic?


    andrewtoth commented at 9:00 PM on June 30, 2026:

    I don't see a reason to.

  64. in src/index/txindex.cpp:106 in 1d5b61c400
     104 | +        const txindex::DBKey key{txindex::CreateKeyPrefix(m_hasher, tx->GetHash()),
     105 | +                                 txindex::Position{static_cast<uint32_t>(block.height), tx_offset}};
     106 | +        if (erase) {
     107 | +            batch.Erase(key);
     108 | +        } else {
     109 | +            batch.Write(key, "");
    


    l0rinc commented at 9:20 PM on June 26, 2026:

    1d5b61c txindex: hash keys and pack positions to reduce disk usage:

    So basically leveldb is storing a sorted set now instead of a map. I wonder if we could tune LevelDB better for this case. nit:

                batch.Write(key, ""); // The tx position is encoded in the key, so the value is intentionally empty
    

    Actually, it seems that this results in extra data for the "empty" value, which is actually encoded as a one-element \0, see:

    BOOST_AUTO_TEST_CASE(dbwrapper_empty_string_vs_span)
    {
        const auto batch_size{[&](const auto& value) {
            CDBWrapper dbw({.path = m_args.GetDataDirBase() / "empty_string_vs_span", .cache_bytes = 1_MiB, .memory_only = true});
            CDBBatch batch(dbw);
            batch.Write(0, value);
            return batch.ApproximateSize();
        }};
        BOOST_CHECK_EQUAL(batch_size(""), batch_size(std::span<std::byte>{})); // Fails with: 20 != 19
    }
    

    This would likely save us roughly 1.4 GB of logical payload currently. I wonder if we could fix TxoSpenderIndex as well.


    andrewtoth commented at 9:00 PM on June 30, 2026:

    Tested this - 27 GB synced in 1h23m :rocket:

  65. in src/index/txindex.cpp:150 in 1d5b61c400
     153 |  {
     154 | +    const txindex::TxHashKeyPrefix prefix{txindex::CreateKeyPrefix(m_db->m_hasher, tx_hash)};
     155 | +    std::unique_ptr<CDBIterator> it{m_db->NewIterator()};
     156 | +    it->Seek(std::pair{txindex::DB_TXINDEX_HASHED, prefix});
     157 | +    txindex::DBKey key{prefix, {}};
     158 | +    const auto header_offset{static_cast<uint32_t>(GetSerializeSize(CBlockHeader{}))};
    


    l0rinc commented at 9:24 PM on June 26, 2026:

    1d5b61c txindex: hash keys and pack positions to reduce disk usage:

    nit: can we use the fixed header size constant to e.g. CBlockHeader::SERIALIZED_SIZE here instead? Should likely be done in another PR before this.


    andrewtoth commented at 9:02 PM on June 30, 2026:

    I removed the header from the read path. It's computed in the write path at the start of a block instead. Having the position be after the header was done because previously header was read first and then seeked to tx. Now we don't read header and can seek right to tx.

    re: the size constant, can be done before or after, it's not a blocker for this PR IMO.

  66. in src/index/txindex.cpp:177 in 1d5b61c400
     180 | +            block_hash = block_index->GetBlockHash();
     181 | +            return true;
     182 | +        }
     183 | +    }
     184 | +
     185 | +    // Fallback to legacy if no hashed entry matched.
    


    l0rinc commented at 9:25 PM on June 26, 2026:

    1d5b61c txindex: hash keys and pack positions to reduce disk usage:

    A txindex miss now searches the hashed bucket before falling back to the legacy full-txid key. Could we document at the fallback why that extra lookup is intentional? And preferably extract the two independent algorithms to local helpers.

        // Fallback to legacy if no hashed entry matched. This makes misses pay an
        // extra lookup, but keeps existing full-txid entries readable after upgrade.
    

    Q: is it possible to use the https://en.wikipedia.org/wiki/Strategy_pattern here to chose either of the 3 combinations: always use legacy, use new with fallback, always use new. In that case the decision is only taken once, after that we only ever use one of the searches.


    andrewtoth commented at 9:04 PM on June 30, 2026:

    Added the comment.

    re: strategy - we don't need the always legacy case. That's not something that can happen if we're using a node with an advancing chain.

    We can check the db before opening if it contains any legacy keys, and use that to determine if we need to do a fallback. We can also open without bloom filters in that case. I opted to wait and see if #35568 gets merged before doing that though. Can be done safely in a follow-up.

  67. in src/index/txindex_key.h:67 in 1d5b61c400
      62 | +
      63 | +using TxHashKeyPrefix = std::array<uint8_t, 5>;
      64 | +
      65 | +inline TxHashKeyPrefix CreateKeyPrefix(const PresaltedSipHasher& hasher, const Txid& txid)
      66 | +{
      67 | +    const uint64_t siphash{hasher(txid.ToUint256())};
    


    l0rinc commented at 9:27 PM on June 26, 2026:

    1d5b61c txindex: hash keys and pack positions to reduce disk usage:

    Does endianness matter here? Don't we need a htobe64_internal call here? It would be preferable to be able to copy these between architectures. We could copy the normalized version directly into the array:

    using TxHashKeyPrefix = std::array<std::byte, 5>;
    
    inline TxHashKeyPrefix CreateKeyPrefix(const PresaltedSipHasher& hasher, const Txid& txid)
    {
        const uint64_t hash{htobe64_internal(hasher(txid.ToUint256()))};
        TxHashKeyPrefix prefix;
        std::memcpy(prefix.data(), &hash, prefix.size());
        return prefix;
    }
    
  68. in src/index/txindex_key.h:63 in 1d5b61c400
      58 | +        block_height = static_cast<uint32_t>(code / MAX_BLOCK_SERIALIZED_SIZE);
      59 | +        tx_offset = static_cast<uint32_t>(code % MAX_BLOCK_SERIALIZED_SIZE);
      60 | +    }
      61 | +};
      62 | +
      63 | +using TxHashKeyPrefix = std::array<uint8_t, 5>;
    


    l0rinc commented at 9:29 PM on June 26, 2026:

    1d5b61c txindex: hash keys and pack positions to reduce disk usage:

    Can we use std::byte here instead?

  69. in doc/release-notes-35531.md:4 in 8c5562e876
       0 | @@ -0,0 +1,12 @@
       1 | +## Index
       2 | +
       3 | +- The transaction index (`-txindex`) now stores less data on disk: the previous
       4 | +  index used about 66 GB, while the new index uses about 29 GB. The index is
    


    l0rinc commented at 9:33 PM on June 26, 2026:

    8c5562e doc: add release notes:

    By the time this gets released the 66 vs 29 will be outdated - how about roughly half the size or similar?

    - The transaction index (`-txindex`) now stores less data on disk, roughly
      halving the size of a fully rebuilt index. The index is backwards compatible,
      so existing users will not see the space saving unless the index is
      recreated. To do so, stop the node, delete the
    
  70. in doc/release-notes-35531.md:6 in 8c5562e876
       0 | @@ -0,0 +1,12 @@
       1 | +## Index
       2 | +
       3 | +- The transaction index (`-txindex`) now stores less data on disk: the previous
       4 | +  index used about 66 GB, while the new index uses about 29 GB. The index is
       5 | +  backwards compatible, so existing users will not see the space saving unless
       6 | +  the index is erased and rebuilt. To do so, stop the node, delete the
    


    l0rinc commented at 9:34 PM on June 26, 2026:

    8c5562e doc: add release notes:

    erased and rebuilt sounds scary - how about "rebuilt"/"recreated"?

  71. in src/index/txindex_key.h:27 in 8c5562e876
      22 | +//! The location of a transaction: the height of the block that contains it and the
      23 | +//! transaction's byte offset within that block (after the header).
      24 | +//!
      25 | +//! Since the offset must always be less than the max block serialized size, we can
      26 | +//! pack the position into a single integer code = max_block_size * height + offset
      27 | +//! and split apart as (height = code / max_block_size, offset = code % max_block_size).
    


    l0rinc commented at 9:50 PM on June 26, 2026:

    Can we add a test to validate the boundary crossings of 1-7 bytes?

    BOOST_AUTO_TEST_CASE(txindex_position_width_boundaries)
    {
        constexpr std::array<std::pair<txindex::BlockTxPosition, size_t>, 14> boundaries{{
            // block height   tx offset   width
            {{0,              0},         1},
            {{0,              255},       1},
            {{0,              256},       2},
            {{0,              65'535},    2},
            {{0,              65'536},    3},
            {{4,              777'215},   3},
            {{4,              777'216},   4},
            {{1'073,          2'967'295}, 4},
            {{1'073,          2'967'296}, 5},
            {{274'877,        3'627'775}, 5},
            {{274'877,        3'627'776}, 6},
            {{70'368'744,     710'655},   6},
            {{70'368'744,     710'656},   7},
            {{4'294'967'295U, 3'999'999}, 7},
        }};
        for (const auto& [position, expected_width] : boundaries) {
            DataStream stream;
            stream << position;
            BOOST_CHECK_EQUAL(stream.size(), expected_width);
    
            txindex::BlockTxPosition decoded;
            stream >> decoded;
            BOOST_CHECK_EQUAL(decoded.block_height, position.block_height);
            BOOST_CHECK_EQUAL(decoded.tx_offset_in_block, position.tx_offset_in_block);
        }
    }
    

    andrewtoth commented at 9:05 PM on June 30, 2026:

    Not needed anymore with the static 6 byte suffix.

  72. in src/index/txindex_key.h:48 in 8c5562e876
      43 | +    }
      44 | +
      45 | +    template <typename Stream>
      46 | +    void Unserialize(Stream& s)
      47 | +    {
      48 | +        const size_t width{s.size()};
    


    l0rinc commented at 9:51 PM on June 26, 2026:

    Could we avoid architecture-specific types in serialization code?

  73. in src/index/txindex_key.h:86 in 8c5562e876
      81 | +    explicit DBKey(const TxHashKeyPrefix& hash_in, const Position& pos_in) : hash_prefix{hash_in}, pos{pos_in} {}
      82 | +
      83 | +    SERIALIZE_METHODS(DBKey, obj)
      84 | +    {
      85 | +        uint8_t prefix{DB_TXINDEX_HASHED};
      86 | +        READWRITE(prefix);
    


    l0rinc commented at 9:57 PM on June 26, 2026:

    Mixing read/write & validation like this seems confusing to me - could the constant prefix be written during serialization and only validated during deserialization?

    template <typename Stream>
    void Serialize(Stream& s) const
    {
        ser_writedata8(s, DB_TXINDEX_HASHED);
        s << hash_prefix << pos;
    }
    
    template <typename Stream>
    void Unserialize(Stream& s)
    {
        if (ser_readdata8(s) != DB_TXINDEX_HASHED) throw std::ios_base::failure("Invalid format for txindex DB key");
        s >> hash_prefix >> pos;
    }
    
  74. in src/index/txindex.cpp:98 in 8c5562e876 outdated
      98 | +    batch.Write(DB_BEST_BLOCK_V2, locator);
      99 | +}
     100 | +
     101 | +void TxIndex::DB::WriteTxs(const interfaces::BlockInfo& block, bool erase)
     102 |  {
     103 |      CDBBatch batch(*this);
    


    l0rinc commented at 9:59 PM on June 26, 2026:

    https://github.com/bitcoin-core/leveldb-subtree/pull/48 would come in handy here

    diff --git a/src/dbwrapper.cpp b/src/dbwrapper.cpp
    index ffe6f267a6..e23cf2f4f6 100644
    --- a/src/dbwrapper.cpp
    +++ b/src/dbwrapper.cpp
    @@ -175,6 +175,11 @@ void CDBBatch::Clear()
         assert(m_value_scratch.empty());
     }
     
    +void CDBBatch::Reserve(size_t size)
    +{
    +    m_impl_batch->batch.Reserve(size);
    +}
    +
     void CDBBatch::WriteImpl(std::span<const std::byte> key, DataStream& value)
     {
         leveldb::Slice slKey(CharCast(key.data()), key.size());
    diff --git a/src/dbwrapper.h b/src/dbwrapper.h
    index 83da6febe7..0fd243cd6e 100644
    --- a/src/dbwrapper.h
    +++ b/src/dbwrapper.h
    @@ -102,6 +102,7 @@ public:
         explicit CDBBatch(const CDBWrapper& _parent);
         ~CDBBatch();
         void Clear();
    +    void Reserve(size_t size);
     
         template <typename K, typename V>
         void Write(const K& key, const V& value)
    diff --git a/src/index/txindex.cpp b/src/index/txindex.cpp
    index f1756b9120..0f05e0ab1f 100644
    --- a/src/index/txindex.cpp
    +++ b/src/index/txindex.cpp
    @@ -99,6 +99,8 @@ void TxIndex::DB::WriteTxs(const interfaces::BlockInfo& block, bool erase)
     {
         assert(block.data);
         CDBBatch batch(*this);
    +    const auto batch_size{batch.ApproximateSize() + block.data->vtx.size() * (1 + 1 + 1 + txindex::TxHashKeyPrefix{}.size() + 6 + 1)}; // tag + key length + db prefix + hash prefix + compact position + empty value length
    +    batch.Reserve(batch_size);
         uint32_t tx_offset_in_block{GetSizeOfCompactSize(block.data->vtx.size())};
         for (const auto& tx : block.data->vtx) {
             const txindex::DBKey key{txindex::CreateKeyPrefix(m_hasher, tx->GetHash()),
    @@ -110,6 +112,7 @@ void TxIndex::DB::WriteTxs(const interfaces::BlockInfo& block, bool erase)
             }
             tx_offset_in_block += ::GetSerializeSize(TX_WITH_WITNESS(*tx));
         }
    +    assert(batch.ApproximateSize() <= batch_size); // TODO remove
         WriteBatch(batch);
     }
     
    diff --git a/src/leveldb/db/write_batch.cc b/src/leveldb/db/write_batch.cc
    index b54313c35e..b2cb2103d8 100644
    --- a/src/leveldb/db/write_batch.cc
    +++ b/src/leveldb/db/write_batch.cc
    @@ -37,6 +37,8 @@ void WriteBatch::Clear() {
       rep_.resize(kHeader);
     }
     
    +void WriteBatch::Reserve(size_t size) { rep_.reserve(size); }
    +
     size_t WriteBatch::ApproximateSize() const { return rep_.size(); }
     
     Status WriteBatch::Iterate(Handler* handler) const {
    diff --git a/src/leveldb/include/leveldb/write_batch.h b/src/leveldb/include/leveldb/write_batch.h
    index 94d4115fed..e05287e299 100644
    --- a/src/leveldb/include/leveldb/write_batch.h
    +++ b/src/leveldb/include/leveldb/write_batch.h
    @@ -21,6 +21,7 @@
     #ifndef STORAGE_LEVELDB_INCLUDE_WRITE_BATCH_H_
     #define STORAGE_LEVELDB_INCLUDE_WRITE_BATCH_H_
     
    +#include <cstddef>
     #include <string>
     
     #include "leveldb/export.h"
    @@ -56,6 +57,9 @@ class LEVELDB_EXPORT WriteBatch {
       // Clear all updates buffered in this batch.
       void Clear();
     
    +  // Reserve space for updates buffered in this batch.
    +  void Reserve(size_t size);
    +
       // The size of the database changes caused by this batch.
       //
       // This number is tied to implementation details, and may change across
    diff --git a/src/test/dbwrapper_tests.cpp b/src/test/dbwrapper_tests.cpp
    index 185bf491e5..8dd0699a08 100644
    --- a/src/test/dbwrapper_tests.cpp
    +++ b/src/test/dbwrapper_tests.cpp
    @@ -167,6 +167,9 @@ BOOST_AUTO_TEST_CASE(dbwrapper_batch)
     
             uint256 res;
             CDBBatch batch(dbw);
    +        const auto empty_batch_size{batch.ApproximateSize()};
    +        batch.Reserve(1_MiB);
    +        BOOST_CHECK_EQUAL(batch.ApproximateSize(), empty_batch_size);
     
             batch.Write(key, in);
             batch.Write(key2, in2);
    

    andrewtoth commented at 9:05 PM on June 30, 2026:

    Can be added safely after the leveldb change is merged.

  75. in src/index/txindex_key.h:28 in 8c5562e876
      23 | +//! transaction's byte offset within that block (after the header).
      24 | +//!
      25 | +//! Since the offset must always be less than the max block serialized size, we can
      26 | +//! pack the position into a single integer code = max_block_size * height + offset
      27 | +//! and split apart as (height = code / max_block_size, offset = code % max_block_size).
      28 | +struct Position {
    


    l0rinc commented at 10:11 PM on June 26, 2026:

    I originally thought tx_offset was the block-file offset; we might want to clarify that it's the serialized byte offset inside the block. We could rename the type to something less general, maybe BlockTxPosition (which would also make clear that tx_offset is a serialized-block offset).

  76. in src/index/txindex.cpp:166 in 8c5562e876 outdated
     169 | +        if (file.IsNull()) {
     170 | +            LogError("OpenBlockFile failed");
     171 | +            return false;
     172 | +        }
     173 | +        try {
     174 | +            file >> TX_WITH_WITNESS(tx);
    


    l0rinc commented at 11:01 PM on June 29, 2026:

    Should we mutate tx when the hash doesn't match? A hash-prefix false positive can be left in the output tx when the scan misses the requested txid. Given that https://github.com/bitcoin/bitcoin/blob/7a851180058facf7824903cc46a1948beb944ed5/src/rpc/rawtransaction.cpp#L156 ignores the return value, tx could contain the wrong value after the call.


    diff --git a/src/test/txindex_tests.cpp b/src/test/txindex_tests.cpp
    --- a/src/test/txindex_tests.cpp	(revision 5ab57fe7288f9cbec8437317d8e51496900e97f6)
    +++ b/src/test/txindex_tests.cpp	(revision 75a9d1198f25488b64fb02daf437e3fe025fd64c)
    @@ -157,6 +157,7 @@
         it->Next();
         BOOST_REQUIRE(it->Valid() && it->GetKey(key) && key.hash_prefix == target_prefix);
         BOOST_CHECK(read_txid(key.pos) == target_txid);
    +    const txindex::Position target_pos{key.pos};
     
         CTransactionRef tx_disk;
         uint256 block_hash;
    @@ -164,6 +165,12 @@
         BOOST_REQUIRE(tx_disk);
         BOOST_CHECK(tx_disk->GetHash() == target_txid);
     
    +    db.Erase(txindex::DBKey{target_prefix, target_pos});
    +    CTransactionRef missing_tx;
    +    BOOST_CHECK(!txindex.FindTx(target_txid, block_hash, missing_tx));
    +    BOOST_CHECK(!missing_tx);
    +    db.Write(txindex::DBKey{target_prefix, target_pos}, "");
    +
         // Legacy fallback: drop the first coinbase's hashed entry and re-add it under the
         // old 't' + txid schema (a physical CDiskTxPos), then confirm the lookup still
         // finds it via the legacy path.
    

    andrewtoth commented at 1:30 PM on June 30, 2026:

    Nice catch! There were no false positives before, so this bug was never triggered.

  77. in src/index/txindex.cpp:144 in 8c5562e876 outdated
     147 |      return true;
     148 |  }
     149 |  
     150 |  BaseIndex::DB& TxIndex::GetDB() const { return *m_db; }
     151 |  
     152 |  bool TxIndex::FindTx(const Txid& tx_hash, uint256& block_hash, CTransactionRef& tx) const
    


    l0rinc commented at 11:03 PM on June 29, 2026:

    can we make this [[nodiscard]] to avoid the situation below - and maybe mention that @param[out] tx The transaction itself. is undefined if we return false?

  78. in doc/release-notes-35531.md:10 in 8c5562e876 outdated
       5 | +  backwards compatible, so existing users will not see the space saving unless
       6 | +  the index is erased and rebuilt. To do so, stop the node, delete the
       7 | +  `<datadir>/indexes/txindex` directory, and restart; rebuilding can take up to
       8 | +  a few hours depending on hardware. Once rebuilt, the index can no longer be
       9 | +  read by previous releases, so downgrading will require rebuilding it again.
      10 | +  Additionally, transactions that are only in blocks reorged out of the best
    


    l0rinc commented at 11:07 PM on June 29, 2026:

    Not sure it matters but will this return the same historical duplicate transaction IDs as before (i.e. BIP30 duplicates?)


    andrewtoth commented at 8:45 PM on June 30, 2026:

    Interesting! For BIP30 txs both this and legacy indexes will return the same tx data. But, the legacy index would return the later block, and this index will return the earlier block. I don't think it really matters though. Is it worth documenting?


    l0rinc commented at 8:49 PM on June 30, 2026:

    Is it worth documenting?

    it's worth a code comment I'd say...

  79. in src/test/txindex_tests.cpp:152 in 8c5562e876 outdated
     147 | +    BOOST_REQUIRE(it->Valid() && it->GetKey(key) && key.hash_prefix == fake_prefix);
     148 | +    const txindex::Position fake_pos{key.pos};
     149 | +
     150 | +    db.Write(txindex::DBKey{target_prefix, fake_pos}, "");
     151 | +
     152 | +    // The target's bucket now holds the forged false positive first, then the real target.
    


    l0rinc commented at 11:17 PM on June 29, 2026:

    c6d3197 tests: cover txindex hash prefix collisions, legacy lookups and erasure:

    Could the test assert the encoded positions it already controls instead of reimplementing tx reads? I don't fully understand why we're re-reading, wouldn't this suffice?

    // The target's bucket now holds the forged false positive first, then the real target.
    it.reset(db.NewIterator());
    it->Seek(std::pair{txindex::DB_TXINDEX_HASHED, target_prefix});
    BOOST_REQUIRE(it->Valid() && it->GetKey(key) && key.hash_prefix == target_prefix);
    BOOST_CHECK(key.pos.block_height == fake_pos.block_height);
    BOOST_CHECK(key.pos.tx_offset_in_block == fake_pos.tx_offset_in_block);
    it->Next();
    BOOST_REQUIRE(it->Valid() && it->GetKey(key) && key.hash_prefix == target_prefix);
    const txindex::BlockTxPosition target_pos{key.pos};
    BOOST_CHECK(target_pos.block_height != fake_pos.block_height || target_pos.tx_offset_in_block != fake_pos.tx_offset_in_block);
    
  80. in src/index/txindex.cpp:131 in 8c5562e876
     126 |  bool TxIndex::CustomAppend(const interfaces::BlockInfo& block)
     127 |  {
     128 |      // Exclude genesis block transaction because outputs are not spendable.
     129 |      if (block.height == 0) return true;
     130 |  
     131 |      assert(block.data);
    


    l0rinc commented at 11:20 PM on June 29, 2026:

    WriteTxs needs block.data both from CustomAppend and CustomRemove, we could add the assertion inside instead:

    diff --git a/src/index/txindex.cpp b/src/index/txindex.cpp
    --- a/src/index/txindex.cpp	(revision f7bcaef568c2d803961efc379d01facb7efdfec0)
    +++ b/src/index/txindex.cpp	(revision 25de18042050c5f7998265021420c74ab4b84789)
    @@ -97,6 +97,7 @@
     
     void TxIndex::DB::WriteTxs(const interfaces::BlockInfo& block, bool erase)
     {
    +    assert(block.data);
         CDBBatch batch(*this);
         uint32_t tx_offset_in_block{GetSizeOfCompactSize(block.data->vtx.size())};
         for (const auto& tx : block.data->vtx) {
    @@ -131,7 +132,6 @@
         // Exclude genesis block transaction because outputs are not spendable.
         if (block.height == 0) return true;
     
    -    assert(block.data);
         m_db->WriteTxs(block);
         return true;
     }
    
  81. in src/index/txindex.cpp:138 in 8c5562e876 outdated
     141 | +    return true;
     142 | +}
     143 | +
     144 | +bool TxIndex::CustomRemove(const interfaces::BlockInfo& block)
     145 | +{
     146 | +    m_db->WriteTxs(block, /*erase=*/true);
    


    l0rinc commented at 11:50 PM on June 29, 2026:

    was just wondering what happens if we want to undo genesis, but it seems to be explicitly guarded, so maybe we could document it here (nit, just resolve if you disagree):

        assert(block.height > 0);
    
  82. in src/index/txindex.cpp:158 in 8c5562e876
     161 | +        CBlockIndex* block_index;
     162 | +        {
     163 | +            LOCK(cs_main);
     164 | +            block_index = m_chainstate->m_chain[key.pos.block_height];
     165 | +            if (!block_index) continue;
     166 | +            tx_pos = FlatFilePos{block_index->nFile, block_index->nDataPos + header_offset + key.pos.tx_offset};
    


    l0rinc commented at 11:53 PM on June 29, 2026:

    We're copying tx_pos under the lock, but keep the CBlockIndex* alive after the lock only to read the block hash if the candidate transaction matches.

    Could we copy the block hash under the same lock too, so the unlocked file read uses only local values?

    FlatFilePos tx_pos;
    uint256 candidate_block_hash;
    {
        LOCK(cs_main);
        const CBlockIndex* block_index{m_chainstate->m_chain[key.pos.block_height]};
        if (!block_index) continue;
        tx_pos = FlatFilePos{block_index->nFile, block_index->nDataPos + CBlockHeader::SERIALIZED_SIZE + key.pos.tx_offset_in_block};
        candidate_block_hash = block_index->GetBlockHash();
    }
    

    ...

    tx = candidate_tx;
    block_hash = candidate_block_hash;
    

    andrewtoth commented at 1:32 PM on June 30, 2026:

    CBlockIndex* being returned is immutable. There are certain fields on it that are guarded by cs_main (like nFile and nDataPos), but others can be read without the lock (like GetBlockHash()).

    I think keeping the index is the correct pattern here.


    l0rinc commented at 6:06 PM on June 30, 2026:

    Isn't that the case for tx_pos as well, any reason for constructing that inside but not the candidate_block_hash? Even if that's not the case, seems simpler to only expose what's strictly needed after the scope terminated.


    andrewtoth commented at 7:21 PM on June 30, 2026:

    tx_pos needs nFile and nDataPos which are guarded by cs_main.

  83. in src/index/txindex.cpp:72 in 8c5562e876
      71 | +        if (!db.Read("siphash_key", siphash_key)) {
      72 | +            FastRandomContext rng{};
      73 | +            siphash_key = {rng.rand64(), rng.rand64()};
      74 | +            db.Write("siphash_key", siphash_key, /*fSync=*/true);
      75 | +        }
      76 | +        return PresaltedSipHasher{siphash_key.first, siphash_key.second};
    


    l0rinc commented at 12:06 AM on June 30, 2026:

    This looks like an ad-hoc deserialization code for PresaltedSipHasher - could we encapsulate that inside the object itself?

    diff --git a/src/crypto/siphash.h b/src/crypto/siphash.h
    index 2f28473a4f..e1eeeb25c5 100644
    --- a/src/crypto/siphash.h
    +++ b/src/crypto/siphash.h
    @@ -10,10 +10,13 @@
     #include <span>
     
     class uint256;
    +class PresaltedSipHasher;
     
     /** Shared SipHash internal state v[0..3], initialized from (k0, k1). */
     class SipHashState
     {
    +    friend class PresaltedSipHasher;
    +
         static constexpr uint64_t C0{0x736f6d6570736575ULL}, C1{0x646f72616e646f6dULL}, C2{0x6c7967656e657261ULL}, C3{0x7465646279746573ULL};
     
     public:
    @@ -48,17 +51,32 @@ public:
      *
      * This class caches the initial SipHash v[0..3] state derived from (k0, k1)
      * and implements a specialized hashing path for uint256 values, with or
    - * without an extra 32-bit word. The internal state is immutable, so
    - * PresaltedSipHasher instances can be reused for multiple hashes with the
    - * same key.
    + * without an extra 32-bit word. The call operators leave the cached state
    + * unchanged, so PresaltedSipHasher instances can be reused for multiple hashes
    + * with the same key.
      */
     class PresaltedSipHasher
     {
    -    const SipHashState m_state;
    +    SipHashState m_state;
     
     public:
    +    PresaltedSipHasher() noexcept : PresaltedSipHasher{0, 0} {}
         explicit PresaltedSipHasher(uint64_t k0, uint64_t k1) noexcept : m_state{k0, k1} {}
     
    +    template <typename Stream>
    +    void Serialize(Stream& s) const
    +    {
    +        s << (m_state.v[0] ^ SipHashState::C0) << (m_state.v[1] ^ SipHashState::C1);
    +    }
    +
    +    template <typename Stream>
    +    void Unserialize(Stream& s)
    +    {
    +        uint64_t k0, k1;
    +        s >> k0 >> k1;
    +        m_state = SipHashState{k0, k1};
    +    }
    +
         /** Equivalent to CSipHasher(k0, k1).Write(val).Finalize(). */
         uint64_t operator()(const uint256& val) const noexcept;
     
    diff --git a/src/index/txindex.cpp b/src/index/txindex.cpp
    index a71e8046f7..404b57a56a 100644
    --- a/src/index/txindex.cpp
    +++ b/src/index/txindex.cpp
    @@ -21,6 +21,7 @@
     #include <streams.h>
     #include <sync.h>
     #include <uint256.h>
    +#include <util/check.h>
     #include <util/fs.h>
     #include <util/log.h>
     #include <validation.h>
    @@ -63,13 +64,14 @@ public:
     TxIndex::DB::DB(size_t n_cache_size, bool f_memory, bool f_wipe) :
         BaseIndex::DB(gArgs.GetDataDirNet() / "indexes" / "txindex", n_cache_size, f_memory, f_wipe),
         m_hasher{[](CDBWrapper& db) {
    -        std::pair<uint64_t, uint64_t> siphash_key;
    -        if (!db.Read("siphash_key", siphash_key)) {
    +        PresaltedSipHasher hasher;
    +        if (!db.Read(txindex::DB_TXID_HASH_SALT, hasher)) {
    +            // The salt only needs to be generated once and persisted.
                 FastRandomContext rng{};
    -            siphash_key = {rng.rand64(), rng.rand64()};
    -            db.Write("siphash_key", siphash_key, /*fSync=*/true);
    +            db.Write(txindex::DB_TXID_HASH_SALT, PresaltedSipHasher{rng.rand64(), rng.rand64()}, /*fSync=*/true);
    +            Assert(db.Read(txindex::DB_TXID_HASH_SALT, hasher));
             }
    -        return PresaltedSipHasher{siphash_key.first, siphash_key.second};
    +        return hasher;
         }(*this)}
     {}
     
    diff --git a/src/index/txindex_key.h b/src/index/txindex_key.h
    index 025638c36a..8228de8075 100644
    --- a/src/index/txindex_key.h
    +++ b/src/index/txindex_key.h
    @@ -15,9 +15,11 @@
     #include <cstddef>
     #include <cstdint>
     #include <ios>
    +#include <string>
     
     namespace txindex {
     constexpr uint8_t DB_TXINDEX_HASHED{'x'};
    +static const std::string DB_TXID_HASH_SALT{"txid_hash_salt"};
     
     //! The location of a transaction: the height of the block that contains it and the
     //! transaction's byte offset within that block (after the header).
    diff --git a/src/test/hash_tests.cpp b/src/test/hash_tests.cpp
    index a5059a8fe8..d51fbecf9f 100644
    --- a/src/test/hash_tests.cpp
    +++ b/src/test/hash_tests.cpp
    @@ -5,6 +5,7 @@
     #include <clientversion.h>
     #include <crypto/siphash.h>
     #include <hash.h>
    +#include <streams.h>
     #include <test/util/random.h>
     #include <test/util/setup_common.h>
     #include <util/strencodings.h>
    @@ -128,7 +129,14 @@ BOOST_AUTO_TEST_CASE(siphash)
         // and the test would be affected by default tx version bumps if not fixed.
         tx.version = 1;
         ss << TX_WITH_WITNESS(tx);
    -    BOOST_CHECK_EQUAL(PresaltedSipHasher(1, 2)(ss.GetHash()), 0x79751e980c2a0a35ULL);
    +    const uint256 tx_hash{ss.GetHash()};
    +    BOOST_CHECK_EQUAL(PresaltedSipHasher(1, 2)(tx_hash), 0x79751e980c2a0a35ULL);
    +
    +    PresaltedSipHasher roundtrip_hasher;
    +    DataStream serialized_hasher;
    +    serialized_hasher << PresaltedSipHasher{1, 2};
    +    serialized_hasher >> roundtrip_hasher;
    +    BOOST_CHECK_EQUAL(roundtrip_hasher(tx_hash), 0x79751e980c2a0a35ULL);
     
         // Check consistency between CSipHasher and PresaltedSipHasher.
         FastRandomContext ctx;
    diff --git a/src/test/txindex_tests.cpp b/src/test/txindex_tests.cpp
    index 7a01325102..c9a65fe7f7 100644
    --- a/src/test/txindex_tests.cpp
    +++ b/src/test/txindex_tests.cpp
    @@ -99,9 +99,8 @@ BOOST_FIXTURE_TEST_CASE(txindex_collision_scan_path, TestChain100Setup)
     
         CDBWrapper& db{TxIndexTest::GetDB(txindex)};
         ChainstateManager& chainman{*m_node.chainman};
    -    std::pair<uint64_t, uint64_t> siphash_key;
    -    BOOST_REQUIRE(db.Read("siphash_key", siphash_key));
    -    const PresaltedSipHasher hasher{siphash_key.first, siphash_key.second};
    +    PresaltedSipHasher hasher;
    +    BOOST_REQUIRE(db.Read(txindex::DB_TXID_HASH_SALT, hasher));
     
         // Resolve a position to its physical on-disk location via the active chain, the
         // same way TxIndex::FindTx does.
    

    andrewtoth commented at 1:34 PM on June 30, 2026:

    This seems like a pretty invasive change to siphasher code. I'm not sure if it's worth the review effort rather than just serializing 2 uint64_ts?


    l0rinc commented at 6:09 PM on June 30, 2026:

    It simplifies usage and encapsulates the serialization. We can investigate if it's worth doing in a preceding PR instead. I can also accept if you don't think it's a good idea, but I'd like to explore it.


    andrewtoth commented at 7:21 PM on June 30, 2026:

    Doesn't need to precede it, can be done in a follow-up. Based on your diff it should be compatible with the current serialization.


    l0rinc commented at 7:22 PM on June 30, 2026:

    Agree, please resolve

  84. in src/index/txindex.cpp:151 in 8c5562e876 outdated
     154 | +    const txindex::TxHashKeyPrefix prefix{txindex::CreateKeyPrefix(m_db->m_hasher, tx_hash)};
     155 | +    std::unique_ptr<CDBIterator> it{m_db->NewIterator()};
     156 | +    it->Seek(std::pair{txindex::DB_TXINDEX_HASHED, prefix});
     157 | +    txindex::DBKey key{prefix, {}};
     158 | +    const auto header_offset{static_cast<uint32_t>(GetSerializeSize(CBlockHeader{}))};
     159 | +    for (; it->Valid() && it->GetKey(key) && key.hash_prefix == prefix; it->Next()) {
    


    l0rinc commented at 1:41 AM on June 30, 2026:

    It seems to me a malformed new key would just fall through to the legacy read instead of failing:

    diff --git a/src/test/txindex_tests.cpp b/src/test/txindex_tests.cpp
    --- a/src/test/txindex_tests.cpp	(revision 0948a9dfac92cfbe14bee0f6e79ca02975454737)
    +++ b/src/test/txindex_tests.cpp	(date 1782783654685)
    @@ -21,6 +21,7 @@
     #include <sync.h>
     #include <test/util/setup_common.h>
     #include <util/byte_units.h>
    +#include <util/check.h>
     #include <validation.h>
     
     #include <array>
    @@ -187,6 +188,12 @@
         BOOST_CHECK(!missing_tx);
         db.Write(txindex::DBKey{target_prefix, target_pos}, "");
     
    +    db.Write(std::make_pair(uint8_t{'t'}, target_txid.ToUint256()), *Assert(resolve_pos(target_pos))); // Legacy fallback entry.
    +    db.Write(std::pair{txindex::DB_TXINDEX_HASHED, target_prefix}, ""); // Malformed hashed key without position suffix.
    +    tx_disk.reset();
    +    BOOST_CHECK(!txindex.FindTx(target_txid, block_hash, tx_disk));
    +    BOOST_CHECK(!tx_disk);
    +
         // Legacy fallback: drop the first coinbase's hashed entry and re-add it under the
         // old 't' + txid schema (a physical CDiskTxPos), then confirm the lookup still
         // finds it via the legacy path.
    
    

    andrewtoth commented at 7:24 PM on June 30, 2026:

    The legacy read would then return false though, so should be fine?


    l0rinc commented at 8:52 PM on June 30, 2026:

    I don't have the change checked out locally, but regardless of the return value we should likely catch invalid content early (if we can sanity-check cheaply).


    andrewtoth commented at 9:16 PM on June 30, 2026:

    The it->GetKey will return false if it fails to deserialize, so it keeps the same behavior as current. A malformed key just returns false from FindTx. I think that's fine. Malformed keys shouldn't be written to the db.


    l0rinc commented at 9:21 PM on June 30, 2026:

    Malformed keys shouldn't be written to the db.

    Sure, but they can still be read as malformed (e.g. from worn-out SD cards).


    andrewtoth commented at 4:42 PM on July 4, 2026:

    This seems like an issue with CDBIterator::GetKey. All iterators in the codebase return false for a malformed key. This would require a general fix touching all consumers. I think it is out of scope for this PR.

  85. in src/index/txindex_key.h:38 in 8c5562e876
      33 | +    void Serialize(Stream& s) const
      34 | +    {
      35 | +        assert(tx_offset < MAX_BLOCK_SERIALIZED_SIZE);
      36 | +        const uint64_t code{uint64_t{MAX_BLOCK_SERIALIZED_SIZE} * block_height + tx_offset};
      37 | +        size_t width{1};
      38 | +        for (uint64_t v{code >> 8}; v != 0; v >>= 8) ++width;
    


    l0rinc commented at 2:08 AM on June 30, 2026:

    1d5b61c txindex: hash keys and pack positions to reduce disk usage:

    We should be able to use an intrinsic for this. std::bit_width would do the loop, but since you start from 1, we could just always set the lowest bit and do something like:

            const auto width{CeilDiv(unsigned(std::bit_width(code | 1)), 8u)};
    

    It still bothers me that we're doing all of this manually when 6 bytes would already cover heights 274,878 to 70,368,744, so we could simply use a fixed BigEndianFormatter<6> for reading and writing:

    static constexpr int SERIALIZED_SIZE{6}; // Holds packed positions until height 70,368,744
    
    template <typename Stream>
    void Serialize(Stream& s) const
    {
        assert(tx_offset_in_block < MAX_BLOCK_SERIALIZED_SIZE);
        const uint64_t code{uint64_t{MAX_BLOCK_SERIALIZED_SIZE} * block_height + tx_offset_in_block};
        s << Using<BigEndianFormatter<SERIALIZED_SIZE>>(code);
    }
    
    template <typename Stream>
    void Unserialize(Stream& s)
    {
        uint64_t code;
        s >> Using<BigEndianFormatter<SERIALIZED_SIZE>>(code);
        block_height = uint32_t(code / MAX_BLOCK_SERIALIZED_SIZE);
        tx_offset_in_block = uint32_t(code % MAX_BLOCK_SERIALIZED_SIZE);
    }
    
  86. in src/index/txindex.cpp:123 in 8c5562e876
     115 | @@ -71,27 +116,65 @@ TxIndex::TxIndex(std::unique_ptr<interfaces::Chain> chain, size_t n_cache_size,
     116 |  
     117 |  TxIndex::~TxIndex() = default;
     118 |  
     119 | +interfaces::Chain::NotifyOptions TxIndex::CustomOptions()
     120 | +{
     121 | +    interfaces::Chain::NotifyOptions options;
     122 | +    options.disconnect_data = true;
     123 | +    return options;
    


    l0rinc commented at 5:16 AM on June 30, 2026:

    nit:

        return {.disconnect_data = true};
    
  87. l0rinc changes_requested
  88. l0rinc commented at 6:37 AM on June 30, 2026: contributor

    I like the overall approach and the disk-space reduction, but I’m not ready to ACK yet.

    My main concerns are:

    • batch.Write(key, "") appears to serialize a one-byte \0 value instead of a truly empty value, which is costly at txindex scale (we should apply the fix to TxoSpenderIndex as well).
    • FindTx can mutate the output tx with a hash-prefix false positive even when it returns false, and at least one caller ignores the return value.
    • The position encoding is variable-width and hand-rolled, while a fixed 6-byte BigEndianFormatter seems simpler and covers block heights up to ~70M.
    • Malformed hashed keys should fail closed instead of falling through to legacy lookup.

    The rest of my comments are mostly simplifications, naming, documentation, and test coverage around the new persisted format.

  89. txindex: make TxIndex::FindTx [[nodiscard]]
    Co-authored-by: l0rinc <pap.lorinc@gmail.com>
    52d514d194
  90. txindex: use a new block locator for downgrade safety
    The hashed txindex entries cannot be found by older nodes. Record sync
    progress under a new locator key so a downgraded node will not rely on
    entries indexed by upgraded nodes, and instead continue syncing from the
    legacy locator.
    ff11d04f56
  91. txindex: hash keys and pack positions to reduce disk usage
    Use a 5-byte salted siphash to key txindex entries,
    instead of the full 32-byte txid. Store a packed tx position
    after the hash in the key, so an iterator can scan
    through any collisions and return the correct tx.
    
    Fallback to the legacy key lookup if the tx is not found.
    
    Erase entries after a block is disconnected. This removes txs that are not part of the best chain, so we can store the block height instead of block file position.
    
    Co-authored-by: Pieter Wuille <pieter@wuille.net>
    Co-authored-by: l0rinc <pap.lorinc@gmail.com>
    98d672f572
  92. tests: cover txindex hash prefix collisions, legacy lookups and erasure
    
    Co-authored-by: l0rinc <pap.lorinc@gmail.com>
    0cb404ce91
  93. doc: add release notes b433cc0a6b
  94. andrewtoth force-pushed on Jun 30, 2026
  95. sedited referenced this in commit b393985aa0 on Jul 4, 2026
  96. willcl-ark added the label UTXO Db and Indexes on Jul 9, 2026
  97. willcl-ark added the label Resource usage on Jul 9, 2026

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-07-09 22:51 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me