Currently, electrs and other indexers map between an address/scripthash to the list of the relevant transactions.
However, in order to fetch those transactions from bitcoind, electrs relies on reading the whole block and post-filtering for a specific transaction1. Other indexers use a txindex to fetch a transaction using its txid 234.
The above approach has significant storage and CPU overhead, since the txid is a pseudo-random 32-byte value.
This PR is adding support for using the transaction’s position within its block to be able to fetch it directly using REST API, using the following HTTP request (to fetch the N-th transaction from BLOCKHASH):
0GET /rest/txfromblock/BLOCKHASH-N.bin
If binary response format is used, the transaction data will be read directly from the storage and sent back to the client, without any deserialization overhead.
The resulting index is much smaller (allowing it to be cached in RAM):
0$ du -sh indexes/locations/ indexes/txindex/
12.5G indexes/locations/
257G indexes/txindex/
The new index is using the following DB schema:
0struct DBKey {
1 uint256 hash; // blockhash
2 uint32_t row; // allow splitting one block's transactions into multiple DB rows
3};
4
5struct DBValue {
6 FlatFilePos block_pos; // file id + offset of the block
7 std::vector<uint32_t> offsets; // a list of transaction offsets within the block
8};
For example, when fetching the 5000th transaction of [block #90005](https://mempool.space/block/000000000000000000017bfd05b5fa367a424c4a565a4baf7950d9e8605df8ec) using ab -k -c 1 -n 100000, enabled locationsindex improves the performance ~19x (2.574ms → 0.136ms).
I am working on a proof-of-concept indexer (https://github.com/romanz/bindex-rs) which is using #32540 & #32541 - please let me know if there are any questions/comments/concerns :)
