index: store per-block transaction locations for efficient lookups #32541

pull romanz wants to merge 3 commits into bitcoin:master from romanz:locations-index changing 14 files +477 āˆ’5
  1. romanz commented at 7:27 am on May 17, 2025: contributor

    Currently, electrs and other indexers map between an address/scripthash to the list of the relevant transactions.

    However, in order to fetch those transactions from bitcoind, electrs relies on reading the whole block and post-filtering for a specific transaction1. Other indexers use a txindex to fetch a transaction using its txid 234.

    The above approach has significant storage and CPU overhead, since the txid is a pseudo-random 32-byte value.

    This PR is adding support for using the transaction’s position within its block to be able to fetch it directly using REST API, using the following HTTP request (to fetch the N-th transaction from BLOCKHASH):

    0GET /rest/txfromblock/BLOCKHASH-N.bin
    

    If binary response format is used, the transaction data will be read directly from the storage and sent back to the client, without any deserialization overhead.

    The resulting index is much smaller (allowing it to be cached):

    0$ du -sh indexes/locations/ indexes/txindex/
    12.5G	indexes/locations/
    257G	indexes/txindex/
    

    The new index is using the following DB schema:

    0struct DBKey {
    1    uint256 hash;   // blockhash
    2    uint32_t part;  // allow splitting one block's transactions into multiple DB rows
    3};
    4
    5struct DBValue {
    6    FlatFilePos block_pos;          // file id + offset of the block
    7    std::vector<uint32_t> offsets;  // a list of transaction offsets within the block
    8};
    
  2. DrahtBot commented at 7:28 am on May 17, 2025: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/32541.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    Concept ACK TheCharlatan

    If your review is incorrectly listed, please react with šŸ‘Ž to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #32699 (docs: adds correct updated documentation links by Zeegaths)
    • #26966 (index: initial sync speedup, parallelize process by furszy)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

    LLM Linter (✨ experimental)

    Possible typos and grammar issues:

    List each error with minimal context, followed by a very brief rationale:

    • typo: /rest/txfromblox/BLOCKHASH-OFFSET -> /rest/txfromblock/BLOCKHASH-OFFSET [missing ā€˜c’ in endpoint name]

    No other typos were found.

    drahtbot_id_4_m

  3. DrahtBot added the label UTXO Db and Indexes on May 17, 2025
  4. DrahtBot added the label CI failed on May 17, 2025
  5. DrahtBot commented at 7:37 am on May 17, 2025: contributor

    🚧 At least one of the CI tasks failed. Task lint: https://github.com/bitcoin/bitcoin/runs/42402043332 LLM reason (✨ experimental): The CI failure is due to missing include guards in src/index/locationsindex.h.

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  6. romanz force-pushed on May 17, 2025
  7. TheCharlatan commented at 9:37 am on May 17, 2025: contributor

    Concept ACK

    Can you add the schema of the index and the expected arguments for the REST API to the pull request description? I was a bit confused at first if this now exposes the file position, but if I read it correctly now, this just allows querying a transaction by its index in the block.

  8. romanz commented at 10:00 am on May 17, 2025: contributor

    Concept ACK

    Thanks!

    Can you add the schema of the index and the expected arguments for the REST API to the pull request description?

    Sure - updated in #32541#issue-3070502385.

  9. romanz force-pushed on May 17, 2025
  10. romanz commented at 11:58 am on May 17, 2025: contributor
    Fixed a few issues (following SonarQube run).
  11. DrahtBot removed the label CI failed on May 17, 2025
  12. luke-jr commented at 6:15 pm on May 20, 2025: member
    How does this compare to getrawtransaction <txid> 0 <blockhash> without a txindex?
  13. romanz commented at 9:51 am on May 21, 2025: contributor

    I have used ApacheBench 2.3 for benchmarking REST API, and the following Rust client for getrawtransaction RPC:

    fetching using the new index

     0$ ab -k -c 1 -n 100000 http://localhost:8332/rest/txfromblock/0000000000000000000083a0cff38278aae196d6d923a7e8ee7e5a0e371226fe-42.bin
     1
     2Document Path:          /rest/txfromblock/0000000000000000000083a0cff38278aae196d6d923a7e8ee7e5a0e371226fe-42.bin
     3Document Length:        301 bytes
     4
     5Concurrency Level:      1
     6Time taken for tests:   13.760 seconds
     7Complete requests:      100000
     8Failed requests:        0
     9Keep-Alive requests:    100000
    10Total transferred:      40500000 bytes
    11HTML transferred:       30100000 bytes
    12Requests per second:    7267.65 [#/sec] (mean)
    13Time per request:       0.138 [ms] (mean)
    14Time per request:       0.138 [ms] (mean, across all concurrent requests)
    15Transfer rate:          2874.41 [Kbytes/sec] received
    

    fetching using txindex

     0$ ab -k -c 1 -n 100000 http://localhost:8332/rest/tx/4137d0dbad434d68a4f52b7bebcba91ddac3f7f5c92b84130432bd6b5e2ea57a.bin
     1
     2Document Path:          /rest/tx/4137d0dbad434d68a4f52b7bebcba91ddac3f7f5c92b84130432bd6b5e2ea57a.bin
     3Document Length:        301 bytes
     4
     5Concurrency Level:      1
     6Time taken for tests:   14.075 seconds
     7Complete requests:      100000
     8Failed requests:        0
     9Keep-Alive requests:    100000
    10Total transferred:      40500000 bytes
    11HTML transferred:       30100000 bytes
    12Requests per second:    7104.78 [#/sec] (mean)
    13Time per request:       0.141 [ms] (mean)
    14Time per request:       0.141 [ms] (mean, across all concurrent requests)
    15Transfer rate:          2810.00 [Kbytes/sec] received
    

    fetching without txindex

    0time cargo run --release -- 4137d0dbad434d68a4f52b7bebcba91ddac3f7f5c92b84130432bd6b5e2ea57a 0000000000000000000083a0cff38278aae196d6d923a7e8ee7e5a0e371226fe
    1    Finished `release` profile [optimized] target(s) in 0.02s
    2     Running `target/release/bench-getrawtx 4137d0dbad434d68a4f52b7bebcba91ddac3f7f5c92b84130432bd6b5e2ea57a 0000000000000000000083a0cff38278aae196d6d923a7e8ee7e5a0e371226fe`
    3iterations = 1000
    4average RPC duration = 8.563491ms
    5
    6real	0m8.628s
    7user	0m0.070s
    8sys	0m0.052s
    
  14. DrahtBot added the label CI failed on Jun 13, 2025
  15. DrahtBot commented at 5:34 pm on June 13, 2025: contributor

    🚧 At least one of the CI tasks failed. Task previous releases, depends DEBUG: https://github.com/bitcoin/bitcoin/runs/42406243587 LLM reason (✨ experimental): The CI failure is caused by a missing header file test/util/index.h during compilation.

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  16. romanz force-pushed on Jun 13, 2025
  17. romanz commented at 7:02 pm on June 13, 2025: contributor
    Rebased to fix #32541 (comment).
  18. DrahtBot removed the label CI failed on Jun 13, 2025
  19. romanz marked this as a draft on Jun 14, 2025
  20. romanz marked this as ready for review on Jun 14, 2025
  21. TheCharlatan commented at 2:34 pm on June 15, 2025: contributor

    How does this compare to getrawtransaction 0 without a txindex?

    As far as I understand the index makes this operation faster by not requiring to read the entire block and then iterating through the transactions to find the match, which I am guessing is what the last benchmark is showing. romanz, would this new endpoint be used while creating the entire index initially, or to serve certain requests? It is not quite clear to me when an indexing client wouldn’t want to read through the entire block and instead only get its transactions selectively.

  22. romanz commented at 5:20 pm on June 15, 2025: contributor

    As far as I understand the index makes this operation faster by not requiring to read the entire block and then iterating through the transactions to find the match

    Correct - the proposed index improves the performance of fetching a single transaction (similar to txindex), requiring significantly less storage.

    would this new endpoint be used while creating the entire index initially, or to serve certain requests?

    I would like the new index to be used to serve history-related queries. For example, https://electrum-protocol.readthedocs.io/en/latest/protocol-methods.html#blockchain-scripthash-get-history.

    You are right that during the history indexing process, the client doesn’t need the proposed index, since it needs to read both the entire block (and undo) data in order to create a map between a transaction’s location and its ScriptPubKeys.

    BTW, I am working on a proof-of-concept indexer (https://github.com/romanz/bindex-rs) which is using #32540 & #32541 - please let me know if there are any questions/comments/concerns :)

  23. in src/rest.cpp:412 in d962c9a917 outdated
    407+        }
    408+    }
    409+
    410+    const LocationsIndex* locations_index = g_locations_index.get();
    411+    if (!locations_index) {
    412+        return RESTERR(req, HTTP_BAD_REQUEST, "Locations index is not enabled");
    


    maflcko commented at 12:04 pm on June 20, 2025:

    not sure about returning an error here. Should be easy to implement a fallback?

    This should also make it easier to benchmark the benefit of the index.

    (I guess it is clear that RPC with json overhead eats some CPU, but out of curiosity I wonder what the difference is between reading the db value (tx location) + reading the tx versus reading the full block and implementing a raw-tx-seeker to jump to the raw bytes of the tx in the block. This should be possible by just reading the length prefixes and then skipping them. I guess the worst case scenario is when its the last transaction, where the prior transaction(s) are mostly filled with minimally sized length prefixes)


    romanz commented at 3:35 pm on June 20, 2025:

    Thanks!

    This should also make it easier to benchmark the benefit of the index.

    Sounds good - will implement a fallback.


    romanz commented at 1:29 pm on June 21, 2025:
    Implemented in df45388b97.

    romanz commented at 6:39 am on June 22, 2025:

    Tested using the following Testnet4 block:

    -locationsindex=1

     0$ BLOCKHASH=00000000000000047b26e3ef164e30f0475c12408d0ece060f90f722aa90787b
     1$ ab -k -c 1 -n 100000 http://localhost:48332/rest/txfromblock/$BLOCKHASH-0.bin
     2
     3Document Path:          /rest/txfromblock/00000000000000047b26e3ef164e30f0475c12408d0ece060f90f722aa90787b-0.bin
     4Document Length:        273 bytes
     5
     6Concurrency Level:      1
     7Time taken for tests:   2.983 seconds
     8Complete requests:      100000
     9Failed requests:        0
    10Keep-Alive requests:    100000
    11Total transferred:      37700000 bytes
    12HTML transferred:       27300000 bytes
    13Requests per second:    33527.87 [#/sec] (mean)
    14Time per request:       0.030 [ms] (mean)
    15Time per request:       0.030 [ms] (mean, across all concurrent requests)
    16Transfer rate:          12343.76 [Kbytes/sec] received
    17
    18$ ab -k -c 1 -n 100000 http://localhost:48332/rest/txfromblock/$BLOCKHASH-1000.bin
    19
    20Document Path:          /rest/txfromblock/00000000000000047b26e3ef164e30f0475c12408d0ece060f90f722aa90787b-1000.bin
    21Document Length:        308 bytes
    22
    23Concurrency Level:      1
    24Time taken for tests:   3.122 seconds
    25Complete requests:      100000
    26Failed requests:        0
    27Keep-Alive requests:    100000
    28Total transferred:      41200000 bytes
    29HTML transferred:       30800000 bytes
    30Requests per second:    32032.96 [#/sec] (mean)
    31Time per request:       0.031 [ms] (mean)
    32Time per request:       0.031 [ms] (mean, across all concurrent requests)
    33Transfer rate:          12888.26 [Kbytes/sec] received
    34
    35$ ab -k -c 1 -n 100000 http://localhost:48332/rest/txfromblock/$BLOCKHASH-2000.bin
    36
    37Document Path:          /rest/txfromblock/00000000000000047b26e3ef164e30f0475c12408d0ece060f90f722aa90787b-2000.bin
    38Document Length:        308 bytes
    39
    40Concurrency Level:      1
    41Time taken for tests:   3.030 seconds
    42Complete requests:      100000
    43Failed requests:        0
    44Keep-Alive requests:    100000
    45Total transferred:      41200000 bytes
    46HTML transferred:       30800000 bytes
    47Requests per second:    33003.94 [#/sec] (mean)
    48Time per request:       0.030 [ms] (mean)
    49Time per request:       0.030 [ms] (mean, across all concurrent requests)
    50Transfer rate:          13278.93 [Kbytes/sec] received
    51
    52$ ab -k -c 1 -n 100000 http://localhost:48332/rest/txfromblock/$BLOCKHASH-3000.bin
    53
    54Document Path:          /rest/txfromblock/00000000000000047b26e3ef164e30f0475c12408d0ece060f90f722aa90787b-3000.bin
    55Document Length:        309 bytes
    56
    57Concurrency Level:      1
    58Time taken for tests:   3.362 seconds
    59Complete requests:      100000
    60Failed requests:        0
    61Keep-Alive requests:    100000
    62Total transferred:      41300000 bytes
    63HTML transferred:       30900000 bytes
    64Requests per second:    29745.72 [#/sec] (mean)
    65Time per request:       0.034 [ms] (mean)
    66Time per request:       0.034 [ms] (mean, across all concurrent requests)
    67Transfer rate:          11997.05 [Kbytes/sec] received
    68
    69$ ab -k -c 1 -n 100000 http://localhost:48332/rest/txfromblock/$BLOCKHASH-4000.bin
    70
    71Document Path:          /rest/txfromblock/00000000000000047b26e3ef164e30f0475c12408d0ece060f90f722aa90787b-4000.bin
    72Document Length:        309 bytes
    73
    74Concurrency Level:      1
    75Time taken for tests:   2.854 seconds
    76Complete requests:      100000
    77Failed requests:        0
    78Keep-Alive requests:    100000
    79Total transferred:      41300000 bytes
    80HTML transferred:       30900000 bytes
    81Requests per second:    35040.53 [#/sec] (mean)
    82Time per request:       0.029 [ms] (mean)
    83Time per request:       0.029 [ms] (mean, across all concurrent requests)
    84Transfer rate:          14132.56 [Kbytes/sec] received
    

    -locationsindex=0

     0$ ab -k -c 1 -n 100000 http://localhost:48332/rest/txfromblock/$BLOCKHASH-0.bin
     1
     2Concurrency Level:      1
     3Time taken for tests:   3.245 seconds
     4Complete requests:      100000
     5Failed requests:        0
     6Keep-Alive requests:    100000
     7Total transferred:      37700000 bytes
     8HTML transferred:       27300000 bytes
     9Requests per second:    30819.31 [#/sec] (mean)
    10Time per request:       0.032 [ms] (mean)
    11Time per request:       0.032 [ms] (mean, across all concurrent requests)
    12Transfer rate:          11346.56 [Kbytes/sec] received
    13
    14$ ab -k -c 1 -n 10000 http://localhost:48332/rest/txfromblock/$BLOCKHASH-1000.bin
    15
    16Concurrency Level:      1
    17Time taken for tests:   4.588 seconds
    18Complete requests:      10000
    19Failed requests:        0
    20Keep-Alive requests:    10000
    21Total transferred:      4120000 bytes
    22HTML transferred:       3080000 bytes
    23Requests per second:    2179.68 [#/sec] (mean)
    24Time per request:       0.459 [ms] (mean)
    25Time per request:       0.459 [ms] (mean, across all concurrent requests)
    26Transfer rate:          876.98 [Kbytes/sec] received
    27
    28$ ab -k -c 1 -n 10000 http://localhost:48332/rest/txfromblock/$BLOCKHASH-2000.bin
    29
    30Concurrency Level:      1
    31Time taken for tests:   9.388 seconds
    32Complete requests:      10000
    33Failed requests:        0
    34Keep-Alive requests:    10000
    35Total transferred:      4120000 bytes
    36HTML transferred:       3080000 bytes
    37Requests per second:    1065.20 [#/sec] (mean)
    38Time per request:       0.939 [ms] (mean)
    39Time per request:       0.939 [ms] (mean, across all concurrent requests)
    40Transfer rate:          428.58 [Kbytes/sec] received
    41
    42$ ab -k -c 1 -n 10000 http://localhost:48332/rest/txfromblock/$BLOCKHASH-3000.bin
    43
    44Concurrency Level:      1
    45Time taken for tests:   12.819 seconds
    46Complete requests:      10000
    47Failed requests:        0
    48Keep-Alive requests:    10000
    49Total transferred:      4130000 bytes
    50HTML transferred:       3090000 bytes
    51Requests per second:    780.10 [#/sec] (mean)
    52Time per request:       1.282 [ms] (mean)
    53Time per request:       1.282 [ms] (mean, across all concurrent requests)
    54Transfer rate:          314.63 [Kbytes/sec] received
    55
    56$ ab -k -c 1 -n 10000 http://localhost:48332/rest/txfromblock/$BLOCKHASH-4000.bin
    57
    58Concurrency Level:      1
    59Time taken for tests:   17.962 seconds
    60Complete requests:      10000
    61Failed requests:        0
    62Keep-Alive requests:    10000
    63Total transferred:      4130000 bytes
    64HTML transferred:       3090000 bytes
    65Requests per second:    556.73 [#/sec] (mean)
    66Time per request:       1.796 [ms] (mean)
    67Time per request:       1.796 [ms] (mean, across all concurrent requests)
    68Transfer rate:          224.54 [Kbytes/sec] received
    
  24. maflcko commented at 12:05 pm on June 20, 2025: member
    lgtm. I wonder if there should be a fallback?
  25. romanz force-pushed on Jun 21, 2025
  26. romanz commented at 9:58 am on June 21, 2025: contributor
    Rebased over master to use std::vector<std::byte> (following #32743).
  27. romanz marked this as a draft on Jun 21, 2025
  28. DrahtBot added the label CI failed on Jun 21, 2025
  29. DrahtBot commented at 2:43 pm on June 21, 2025: contributor

    🚧 At least one of the CI tasks failed. Task MSan, depends: https://github.com/bitcoin/bitcoin/runs/44528503073 LLM reason (✨ experimental): The CI failure is caused by the abort in the locationsindex_tests subprocess.

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  30. romanz force-pushed on Jun 21, 2025
  31. romanz force-pushed on Jun 21, 2025
  32. romanz marked this as ready for review on Jun 21, 2025
  33. DrahtBot removed the label CI failed on Jun 21, 2025
  34. romanz requested review from maflcko on Jun 21, 2025
  35. romanz commented at 7:32 am on June 22, 2025: contributor
    BTW, if I understand correctly, this index may also improve the performance of SendBlockTransactions during handling of GETBLOCKTXN requests.
  36. in src/node/blockstorage.cpp:1182 in df45388b97 outdated
    1177+    size_t tx_count = ReadCompactSize(filein);
    1178+    if (tx_index >= tx_count) {
    1179+        return false;
    1180+    }
    1181+
    1182+    SkipTransaction skip_tx;
    


    maflcko commented at 1:20 pm on June 23, 2025:

    thanks for implementing this idea. Though, at least locally on my system it looks like it didn’t improve the end-to-end performance for me, at least not in a significant benchmark.

    So it seems fine to use the “normal” deserialization and even go through a redundant ser-deser loop?

     0diff --git a/src/node/blockstorage.cpp b/src/node/blockstorage.cpp
     1index a049122894..64cb914cb1 100644
     2--- a/src/node/blockstorage.cpp
     3+++ b/src/node/blockstorage.cpp
     4@@ -1179,17 +1179,14 @@ bool BlockManager::ReadRawTxFromBlock(std::vector<std::byte>& tx_bytes, const Fl
     5         return false;
     6     }
     7 
     8-    SkipTransaction skip_tx;
     9+    CMutableTransaction skip_tx;
    10     for (size_t i = 0; i < tx_index; ++i) {
    11-        filein >> skip_tx;
    12+        filein >> TX_WITH_WITNESS(skip_tx);
    13     }
    14-    int64_t tx_start = filein.tell();
    15-    filein >> skip_tx;
    16-    int64_t tx_end = filein.tell();
    17+        filein >> TX_WITH_WITNESS(skip_tx);
    18 
    19-    tx_bytes.resize(tx_end - tx_start);
    20-    filein.seek(tx_start, SEEK_SET);
    21-    filein.read(tx_bytes);
    22+    tx_bytes.resize(0);
    23+    VectorWriter {*(std::vector<unsigned char> *)&tx_bytes,tx_bytes.size()}<<TX_WITH_WITNESS(skip_tx);
    24     return true;
    25 }
    26 
    

    maflcko commented at 3:16 pm on June 23, 2025:
    Wrapping the autofile into BufferedReader helped more on my system with reading later transactions in the block via this fallback branch.

    romanz commented at 8:02 pm on June 23, 2025:

    romanz commented at 5:11 pm on June 26, 2025:

    Benchmark results

    Tested 99b48777ee using ab -k -c 1 -n 10000 over block 900005 (having >5000 txs).

    0BLOCKHASH=000000000000000000017bfd05b5fa367a424c4a565a4baf7950d9e8605df8ec
    

    -locationsindex=1

     0Document Path:          /rest/txfromblock/$BLOCKHASH-0.bin
     1Document Length:        384 bytes
     2Requests per second:    7387.93 [#/sec] (mean)
     3Time per request:       0.135 [ms] (mean)
     4
     5Document Path:          /rest/txfromblock/$BLOCKHASH-1000.bin
     6Document Length:        234 bytes
     7Requests per second:    7875.35 [#/sec] (mean)
     8Time per request:       0.127 [ms] (mean)
     9
    10Document Path:          /rest/txfromblock/$BLOCKHASH-2000.bin
    11Document Length:        234 bytes
    12Requests per second:    7470.15 [#/sec] (mean)
    13Time per request:       0.134 [ms] (mean)
    14
    15Document Path:          /rest/txfromblock/$BLOCKHASH-3000.bin
    16Document Length:        220 bytes
    17Requests per second:    7102.34 [#/sec] (mean)
    18Time per request:       0.141 [ms] (mean)
    19
    20Document Path:          /rest/txfromblock/$BLOCKHASH-4000.bin
    21Document Length:        221 bytes
    22Requests per second:    7569.49 [#/sec] (mean)
    23Time per request:       0.132 [ms] (mean)
    24
    25Document Path:          /rest/txfromblock/$BLOCKHASH-5000.bin
    26Document Length:        234 bytes
    27Requests per second:    7333.99 [#/sec] (mean)
    28Time per request:       0.136 [ms] (mean)
    

    -locationsindex=0

     0Document Path:          /rest/txfromblock/$BLOCKHASH-0.bin
     1Document Length:        384 bytes
     2Requests per second:    4030.87 [#/sec] (mean)
     3Time per request:       0.248 [ms] (mean)
     4
     5Document Path:          /rest/txfromblock/$BLOCKHASH-1000.bin
     6Document Length:        234 bytes
     7Requests per second:    764.46 [#/sec] (mean)
     8Time per request:       1.308 [ms] (mean)
     9
    10Document Path:          /rest/txfromblock/$BLOCKHASH-2000.bin
    11Document Length:        234 bytes
    12Requests per second:    526.17 [#/sec] (mean)
    13Time per request:       1.901 [ms] (mean)
    14
    15Document Path:          /rest/txfromblock/$BLOCKHASH-3000.bin
    16Document Length:        220 bytes
    17Requests per second:    391.21 [#/sec] (mean)
    18Time per request:       2.556 [ms] (mean)
    19
    20Document Path:          /rest/txfromblock/$BLOCKHASH-4000.bin
    21Document Length:        221 bytes
    22Requests per second:    261.30 [#/sec] (mean)
    23Time per request:       3.827 [ms] (mean)
    24
    25Document Path:          /rest/txfromblock/$BLOCKHASH-5000.bin
    26Document Length:        234 bytes
    27Requests per second:    208.15 [#/sec] (mean)
    28Time per request:       4.804 [ms] (mean)
    
  37. maflcko approved
  38. maflcko commented at 1:20 pm on June 23, 2025: member
    lgtm
  39. maflcko commented at 1:22 pm on June 23, 2025: member

    BTW, if I understand correctly, this index may also improve the performance of SendBlockTransactions during handling of GETBLOCKTXN requests.

    This is only the fallback where the recent block fell out of memory. If this happens, the performance either shouldn’t matter, or it may be better to keep more recent blocks in memory than just a single one.

  40. romanz force-pushed on Jun 23, 2025
  41. romanz force-pushed on Jun 28, 2025
  42. romanz commented at 6:43 am on June 28, 2025: contributor
    Rebased over master and fixed rest_tx_from_block() argument name case (in addition to #32825).
  43. index: store per-block transaction locations for efficient lookups
    Currently, electrs and other indexers are used to maintain a map between
    an address/scripthash to the list of the relevant transactions.
    
    However, in order to fetch those transactions from bitcoind, electrs
    relies on reading the whole block and post-filtering for a specific
    transaction [1]. Other indexers use a `txindex` to fetch a transaction
    using its txid [2,3,4].
    
    The above approach has significant storage and CPU overhead, since
    the `txid` is a pseudo-random 32-byte value.
    
    This PR is adding support for using the transaction's offset within
    its block to be able to read it directly using REST API.
    
    If binary response format is used, the transaction data will be read
    directly from the storage and sent back to the client, without any
    deserialization overhead.
    
    The resulting index is much smaller (allowing it to be cached):
    
      $ du -sh indexes/locations/ indexes/txindex/
      2.5G	indexes/locations/
      57G	indexes/txindex/
    
    [1] https://github.com/romanz/electrs/blob/master/doc/schema.md
    [2] https://github.com/Blockstream/electrs/blob/new-index/doc/schema.md#txstore
    [3] https://github.com/spesmilo/electrumx/blob/master/docs/HOWTO.rst#prerequisites
    [4] https://github.com/cculianu/Fulcrum/blob/master/README.md#requirements
    91c9d9e745
  44. rest: allow fetching transaction from a block without locationindex 75a38f4928
  45. romanz force-pushed on Jun 30, 2025
  46. romanz commented at 1:13 pm on June 30, 2025: contributor
    Rebased over master after #32825 is merged (to resolve a small conflict).
  47. doc: add release notes and documentation for `/rest/txfromblock/` c695d13468

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-07-05 18:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me