rest: fetch spent transaction outputs by blockhash #32540

pull romanz wants to merge 2 commits into bitcoin:master from romanz:spent-prevouts changing 4 files +149 −0
  1. romanz commented at 7:27 am on May 17, 2025: contributor

    Today, it is possible to fetch a block’s spent prevouts in order to build an external index by using the /rest/block/BLOCKHASH.json endpoint. However, its performance is low due to JSON serialization overhead.

    We can significantly optimize it by adding a new REST API endpoint, using a binary response format (returning a collection of spent txout lists, one per each block transaction):

     0$ BLOCKHASH=00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054
 1
 2$ ab -k -c 1 -n 100 http://localhost:8332/rest/block/$BLOCKHASH.json
 3Document Length:        13278152 bytes
 4Requests per second:    3.53 [#/sec] (mean)
 5Time per request:       283.569 [ms] (mean)
 6
 7$ ab -k -c 1 -n 10000 http://localhost:8332/rest/spenttxouts/$BLOCKHASH.bin
 8Document Length:        195591 bytes
 9Requests per second:    254.47 [#/sec] (mean)
10Time per request:       3.930 [ms] (mean)

    Currently, this PR is being used and tested by Bindex1.

    This PR would allow to improve the performance of external indexers such as electrs2, ElectrumX3, Fulcrum4 and Blockbook5.

    1. https://github.com/romanz/bindex-rs ↩︎

    2. https://github.com/romanz/electrs (also blockstream.info and mempool.space forks) ↩︎

    3. https://github.com/spesmilo/electrumx ↩︎

    4. https://github.com/cculianu/Fulcrum ↩︎

    5. https://github.com/trezor/blockbook ↩︎

  2. DrahtBot commented at 7:27 am on May 17, 2025: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/32540.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK maflcko, TheCharlatan, achow101
    Concept ACK adyshimony
    Stale ACK tapcrafter

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    No conflicts as of last run.

  3. DrahtBot added the label RPC/REST/ZMQ on May 17, 2025
  4. romanz force-pushed on May 17, 2025
  5. romanz commented at 3:47 pm on May 17, 2025: contributor
    Force-pushed to fix a typo in commit description.
  6. yancyribbens commented at 6:42 pm on May 17, 2025: contributor
    Nice. I wonder about just adding a content-type like option so that the api endpoint is the same but you simply say if you want binary or json content returned? Maybe there’s other endpoints that it could then be applied to as well..
  7. romanz commented at 9:06 am on May 18, 2025: contributor

    Nice.

    Thanks :)

    I wonder about just adding a content-type like option so that the api endpoint is the same but you simply say if you want binary or json content returned? Maybe there’s other endpoints that it could then be applied to as well.

    Good idea - would it be OK to implement it in a separate PR?

  8. in src/rest.cpp:324 in ffe571f461 outdated 
    319+    WriteCompactSize(ssSpentResponse, block_undo.vtxundo.size() + 1);
320+    WriteCompactSize(ssSpentResponse, 0); // block_undo.vtxundo doesn't contain coinbase tx
321+    for (const CTxUndo& tx_undo : block_undo.vtxundo) {
322+        WriteCompactSize(ssSpentResponse, tx_undo.vprevout.size());
323+        for (const Coin& coin : tx_undo.vprevout) {
324+            coin.out.Serialize(ssSpentResponse);
    tapcrafter commented at 12:53 pm on May 18, 2025:
    Is there a way to document the output of this REST method? From the name alone I would’ve expected it to return a list of outpoints. But it seems to return a list of transaction outputs (CTxOut or Coin depending on the context). Which absolutely makes sense given the use case. So perhaps a different name would help make that more clear? Perhaps rest/spenttxouts?
    romanz commented at 5:09 am on May 19, 2025:

    So perhaps a different name would help make that more clear? Perhaps rest/spenttxouts?

    Sounds good, thanks! Fixed in 8cb0465def.

  9. tapcrafter commented at 1:00 pm on May 18, 2025: none

    tACK ffe571f461930b7a05a3cf9f7128e843ea9f7e2d

    Running ffe571f461930b7a05a3cf9f7128e843ea9f7e2d:

     0$ ./build/bin/bitcoind -regtest -rpcallowip=::1 -rpcuser=u -rpcpassword=p -rest -txindex
 1
 2# Invalid extension:
 3
 4$ curl -v -g 'u:p@localhost:18443/rest/spentoutputs/4376e2f945afef224981b665778ec45ebe64745e7d768c2937a8e271b69d708a.json'
 5
 6* Host localhost:18443 was resolved.
 7...
 8< HTTP/1.1 404 Not Found
 9< Content-Type: text/plain
10< Date: Sun, 18 May 2025 12:28:13 GMT
11< Content-Length: 47
12< 
13output format not found (available: bin, hex)
14* Connection [#0](/bitcoin-bitcoin/0/) to host localhost left intact
15
16
17# Correct extension:
18
19$ curl -v -g 'u:p@localhost:18443/rest/spentoutputs/4376e2f945afef224981b665778ec45ebe64745e7d768c2937a8e271b69d708a.hex'
20* Host localhost:18443 was resolved.
21...
22< HTTP/1.1 200 OK
23< Content-Type: text/plain
24< Date: Sun, 18 May 2025 12:28:43 GMT
25< Content-Length: 75
26< 
2702000100f2052a010000001976a914090454b16d8fd499365c53a2226373c6d8734b5188ac
28* Connection [#0](/bitcoin-bitcoin/0/) to host localhost left intact
29
30# Which corresponds to the TxOut of the spent transaction:
31
32$ bitcoin-cli -regtest -rpcuser=u -rpcpassword=p decoderawtransaction 020000000001010000000000000000000000000000000000000000000000000000000000000000ffffffff025500feffffff0200f2052a010000001976a914090454b16d8fd499365c53a2226373c6d8734b5188ac0000000000000000266a24aa21a9ede2f61c3f71d1defd3fa999dfa36953755c690689799962b48bebd836974e8cf90120000000000000000000000000000000000000000000000000000000000000000004000000
33{
34  "txid": "38e6ad741857df84af27a709c1d99397b6e33c5078b284d49da6ae74b78d9bc6",
35  "hash": "b254f236b5141afd961bf80918b1417aa1d7f7ced2cbdd7a3bfbd243748e2c6f",
36  "version": 2,
37  "size": 170,
38  "vsize": 143,
39  "weight": 572,
40  "locktime": 4,
41  "vin": [
42    {
43      "coinbase": "5500",
44      "txinwitness": [
45        "0000000000000000000000000000000000000000000000000000000000000000"
46      ],
47      "sequence": 4294967294
48    }
49  ],
50  "vout": [
51    {
52      "value": 50.00000000,
53      "n": 0,
54      "scriptPubKey": {
55        "asm": "OP_DUP OP_HASH160 090454b16d8fd499365c53a2226373c6d8734b51 OP_EQUALVERIFY OP_CHECKSIG",
56        "desc": "addr(mgLdZh69L6Zzs68mfvSkTcnz12XtgSQVJE)#ggfft2ks",
57        "hex": "76a914090454b16d8fd499365c53a2226373c6d8734b5188ac",
58        "address": "mgLdZh69L6Zzs68mfvSkTcnz12XtgSQVJE",
59        "type": "pubkeyhash"
60      }
61    },
62    {
63      "value": 0.00000000,
64      "n": 1,
65      "scriptPubKey": {
66        "asm": "OP_RETURN aa21a9ede2f61c3f71d1defd3fa999dfa36953755c690689799962b48bebd836974e8cf9",
67        "desc": "raw(6a24aa21a9ede2f61c3f71d1defd3fa999dfa36953755c690689799962b48bebd836974e8cf9)#cav96mf3",
68        "hex": "6a24aa21a9ede2f61c3f71d1defd3fa999dfa36953755c690689799962b48bebd836974e8cf9",
69        "type": "nulldata"
70      }
71    }
72  ]
73}
  10. bitcoin deleted a comment on May 18, 2025
  11. yancyribbens commented at 1:50 pm on May 18, 2025: contributor

    Good idea - would it be OK to implement it in a separate PR?

    Sure, as you wish. I’m no authority :)

  12. romanz force-pushed on May 19, 2025
  13. adyshimony commented at 2:26 am on May 22, 2025: none

    tACK

    https://github.com/romanz/bitcoin/commit/8cb0465defdf96a86b7485ff098d2ba70843e942

     0$ ab -k -c 1 -n 100 http://localhost:8332/rest/spenttxouts/00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054.bin
 1This is ApacheBench, Version 2.3 <$Revision: 1903618 $>
 2Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
 3Licensed to The Apache Software Foundation, http://www.apache.org/
 4
 5Benchmarking localhost (be patient).....done
 6
 7
 8Server Software:        
 9Server Hostname:        localhost
10Server Port:            8332
11
12Document Path:          /rest/spenttxouts/00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054.bin
13Document Length:        195591 bytes
14
15Concurrency Level:      1
16Time taken for tests:   0.444 seconds
17Complete requests:      100
18Failed requests:        0
19Keep-Alive requests:    100
20Total transferred:      19569800 bytes
21HTML transferred:       19559100 bytes
22Requests per second:    225.09 [#/sec] (mean)
23Time per request:       4.443 [ms] (mean)
24Time per request:       4.443 [ms] (mean, across all concurrent requests)
25Transfer rate:          43016.55 [Kbytes/sec] received
26
27Connection Times (ms)
28              min  mean[+/-sd] median   max
29Connect:        0    0   0.0      0       0
30Processing:     3    4   1.3      4       7
31Waiting:        3    4   1.3      4       7
32Total:          3    4   1.3      4       7
33
34Percentage of the requests served within a certain time (ms)
35  50%      4
36  66%      5
37  75%      6
38  80%      6
39  90%      7
40  95%      7
41  98%      7
42  99%      7
43 100%      7 (longest request)

    On my machine there is a significant improvement , well done.

     0$ ab -k -c 1 -n 100 http://localhost:8332/rest/block/00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054.json
 1This is ApacheBench, Version 2.3 <$Revision: 1903618 $>
 2Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
 3Licensed to The Apache Software Foundation, http://www.apache.org/
 4
 5Benchmarking localhost (be patient).....done
 6
 7
 8Server Software:        
 9Server Hostname:        localhost
10Server Port:            8332
11
12Document Path:          /rest/block/00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054.json
13Document Length:        13278152 bytes
14
15Concurrency Level:      1
16Time taken for tests:   33.053 seconds
17Complete requests:      100
18Failed requests:        0
19Keep-Alive requests:    100
20Total transferred:      1327825300 bytes
21HTML transferred:       1327815200 bytes
22Requests per second:    3.03 [#/sec] (mean)
23Time per request:       330.530 [ms] (mean)
24Time per request:       330.530 [ms] (mean, across all concurrent requests)
25Transfer rate:          39231.07 [Kbytes/sec] received
26
27Connection Times (ms)
28              min  mean[+/-sd] median   max
29Connect:        0    0   0.0      0       0
30Processing:   308  331  17.5    327     396
31Waiting:      297  315  16.7    310     386
32Total:        308  331  17.5    327     396
33
34Percentage of the requests served within a certain time (ms)
35  50%    327
36  66%    334
37  75%    337
38  80%    339
39  90%    345
40  95%    377
41  98%    396
42  99%    396
43 100%    396 (longest request)
  14. in src/rest.cpp:287 in 8cb0465def outdated 
    281@@ -281,6 +282,69 @@ static bool rest_headers(const std::any& context,
282     }
283 }
284 
285+static bool rest_spent_txouts(const std::any& context, HTTPRequest* req, const std::string& strURIPart)
286+{
287+    if (!CheckWarmup(req))
    luke-jr commented at 9:00 am on May 22, 2025:
    nit: braces required
  15. luke-jr commented at 9:00 am on May 22, 2025: member
    Not sure we should export a custom binary format for undo data like this. At least there should be a JSON option?
  16. romanz marked this as a draft on May 22, 2025
  17. romanz commented at 3:58 pm on May 22, 2025: contributor

    Not sure we should export a custom binary format for undo data like this.

    In my opinion, using a binary format allows us to achieve significantly better performance when building an external index (compared to using /rest/block/HASH.json).

    I have followed the example of the /rest/getutxos/ endpoint, which also uses binary encoding when it returns a std::vector<CCoin> outs: https://github.com/bitcoin/bitcoin/blob/d2c9fc84e17120f186a54ef92bab76ea7e8d31b5/src/rest.cpp#L906

  18. romanz commented at 3:58 pm on May 22, 2025: contributor

    At least there should be a JSON option?

    Sounds good, added in https://github.com/bitcoin/bitcoin/pull/32540/commits/9d7e23e2f505ce6cbe830fc607cf203b1a48ba0d:

    Tested with:

    0curl -s http://localhost:8332/rest/spenttxouts/00000000000034a32d25a4df37018194645cd62bf311b04ce322da9d800d4576.json | jq .

     0[
 1  [],
 2  [
 3    {
 4      "value": 19.9,
 5      "scriptPubKey": {
 6        "asm": "OP_DUP OP_HASH160 9f1c396fc47b5e8127d8fca2cbe0da12fb65b18b OP_EQUALVERIFY OP_CHECKSIG",
 7        "desc": "addr(1FWJFZiX8v1y6Mf14bzT2qTjzKgJ948gTN)#n6pdnxnu",
 8        "hex": "76a9149f1c396fc47b5e8127d8fca2cbe0da12fb65b18b88ac",
 9        "address": "1FWJFZiX8v1y6Mf14bzT2qTjzKgJ948gTN",
10        "type": "pubkeyhash"
11      }
12    }
13  ],
14  [
15    {
16      "value": 1.13,
17      "scriptPubKey": {
18        "asm": "OP_DUP OP_HASH160 63ecfce5e3b086c6b4c9a8f68b64c70eaf4f579e OP_EQUALVERIFY OP_CHECKSIG",
19        "desc": "addr(1A7MoCV1q8L6K2k6sJLnPFDJSR3kmiyubF)#k7amcrpk",
20        "hex": "76a91463ecfce5e3b086c6b4c9a8f68b64c70eaf4f579e88ac",
21        "address": "1A7MoCV1q8L6K2k6sJLnPFDJSR3kmiyubF",
22        "type": "pubkeyhash"
23      }
24    }
25  ],
26  [
27    {
28      "value": 0.129,
29      "scriptPubKey": {
30        "asm": "OP_DUP OP_HASH160 7c78cf144b8bf68372403cf5690aa49560275423 OP_EQUALVERIFY OP_CHECKSIG",
31        "desc": "addr(1CM9WjjPScLSfQaYXq9YY5VgqUh6bP87rw)#nrcg7jla",
32        "hex": "76a9147c78cf144b8bf68372403cf5690aa4956027542388ac",
33        "address": "1CM9WjjPScLSfQaYXq9YY5VgqUh6bP87rw",
34        "type": "pubkeyhash"
35      }
36    }
37  ]
38]
  19. romanz marked this as ready for review on May 22, 2025
  20. yancyribbens commented at 1:10 pm on May 24, 2025: contributor

    would it be OK to implement it in a separate PR?

    I was thinking it might be not ideal to have this endpoint in addition to adding a content-type. Unless the plan is to deprecate this after content type is added? Maybe that’s what is meant by the two conflicting types mentioned #32583 (comment)

  21. yancyribbens commented at 2:12 pm on May 24, 2025: contributor

    I was thinking it might be not ideal to have this endpoint in addition to adding a content-type. Unless the plan is to deprecate this after content type is added? Maybe that’s what is meant by the two conflicting types mentioned #32583 (comment)

    Please disregard

  22. in src/rest.cpp:288 in 9d7e23e2f5 outdated 
    281@@ -281,6 +282,111 @@ static bool rest_headers(const std::any& context,
282     }
283 }
284 
285+/**
286+ * Serialize spent outputs as a list of per-transaction CTxOut lists using binary format.
287+ */
288+static void SerializeBlockUndo(DataStream &stream, const CBlockUndo &block_undo) {
    maflcko commented at 6:21 am on May 26, 2025:
    nit: clang-format new code?
    romanz commented at 6:37 am on May 26, 2025:
    Thanks! Fixed in 1e2b26e4f8.
  23. maflcko commented at 6:23 am on May 26, 2025: member

    Please squash your commits according to https://github.com/bitcoin/bitcoin/blob/master/CONTRIBUTING.md#squashing-commits

    lgtm ACK 9d7e23e2f505ce6cbe830fc607cf203b1a48ba0d

  24. DrahtBot requested review from tapcrafter on May 26, 2025
  25. romanz force-pushed on May 26, 2025
  26. in src/rest.cpp:364 in 1e2b26e4f8 outdated 
    359+    switch (rf) {
360+    case RESTResponseFormat::BINARY: {
361+        DataStream ssSpentResponse{};
362+        SerializeBlockUndo(ssSpentResponse, block_undo);
363+        req->WriteHeader("Content-Type", "application/octet-stream");
364+        req->WriteReply(HTTP_OK, std::as_bytes(std::span{ssSpentResponse}));
    maflcko commented at 6:55 am on May 26, 2025:
    nit: can remove the casts
    romanz commented at 6:21 pm on May 27, 2025:
    Thanks, good catch! Will update on the next rebase.
    maflcko commented at 7:20 pm on May 27, 2025:
    (If you want, you can also remove them in other places in this file. This way, the file doesn’t have to be touched again for that reason.)
    romanz commented at 5:28 pm on June 7, 2025:

    nit: can remove the casts

    Changed std::as_bytes(std::span{ssSpentResponse}) to ssSpentResponse in d4e212e8a6.

    If you want, you can also remove them in other places in this file.

    0$ git grep std::as_bytes src/rest.cpp
1src/rest.cpp:        req->WriteReply(HTTP_OK, std::as_bytes(std::span{block_data}));

    I am not sure that I can remove it here, since block_data is std::vector<uint8_t>, and WriteReply(...) accepts std::span<std::byte>.

    Maybe I can change ReadRawBlock(...) to work with std::vector<std::byte>: https://github.com/bitcoin/bitcoin/commit/7dd0554abf62eab8f17b44d9562630cca265cf73 WDYT?

    maflcko commented at 2:22 pm on June 9, 2025:

    Maybe I can change ReadRawBlock(...) to work with std::vector<std::byte>: 7dd0554

    Sure, but probably a separate pull request. Also to avoid the new cast in the commit, SpanReader could be extended with another constructor, so that the cast is private/internal.

    0explicit SpanReader(std::span<const std::byte> data) : ...
    romanz commented at 8:05 pm on June 9, 2025:
    Thanks - will open a new PR :)
    romanz commented at 5:04 am on June 13, 2025:
    Opened #32743.
  27. maflcko commented at 7:00 am on May 26, 2025: member

    review ACK 1e2b26e4f8498a08072104b12759d91ef8b410db 👤

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
2trusted comment: review ACK 1e2b26e4f8498a08072104b12759d91ef8b410db 👤
3umcSMyJu+op/g4YObmcufkFh2Q/WLwQJSqCZJWjqXxYHLJJ8sJUi5i8pxOkmKaAisZihoMJKWGgqClMx/SwBCg==
  28. in src/rest.cpp:355 in 1e2b26e4f8 outdated 
    350+    if (!pblockindex) {
351+        return RESTERR(req, HTTP_NOT_FOUND, hashStr + " not found");
352+    }
353+
354+    CBlockUndo block_undo;
355+    if (pblockindex->nHeight > 0 && !chainman->m_blockman.ReadBlockUndo(block_undo, *pblockindex)) {
    TheCharlatan commented at 2:51 pm on May 26, 2025:
    Would it be useful if we’d have a ReadRawBlockUndo? Similarly to ReadRawBlock it could just read the data without any hash integrity checks and without an extra serialization roundtrip. Our internal serialization just skips over the coinbase, but I don’t feel like that would be surfacing too much detail either.
    romanz commented at 6:07 pm on May 26, 2025:

    Thanks @TheCharlatan! It should indeed improve the read performance :)

    Would it be OK if I would open a separate PR for adding ReadRawBlockUndo?

    TheCharlatan commented at 6:43 pm on May 26, 2025:
    I think it would be ok, but we’d have to change the binary format here to skip over the empty 0 entry for the coinbase?
    romanz commented at 7:11 pm on May 26, 2025:

    but we’d have to change the binary format here to skip over the empty 0 entry for the coinbase?

    Yes - I’ll adapt this PR to return the undo data from storage (with as little overhead as possible).

    Indeed, the binary format will be vector<vector<Coin>>1, for the all block’s transactions (except the first one).

    1. https://github.com/bitcoin/bitcoin/blob/638a4c0bd8b53766faeb437244b2aae4eed28dcf/src/undo.h#L51-L68 ↩︎

    maflcko commented at 6:22 am on May 27, 2025:

    I wonder if we really want to commit to the storage serialization on the rest interface. Currently in this pull, it will be re-serialized, so there is no dependency. However, with this suggestion, and a future change in the storage serialization, there will be a breaking change on the rest interface, or alternatively a re-serialization again (back to what this pull is doing).

    No strong opinion, just mentioning it here.

    Maybe a benchmark could help to decide if it is worth it?

    romanz commented at 6:07 pm on May 27, 2025:

    Maybe a benchmark could help to decide if it is worth it?

    Reading the undo data directly from storage (i.e. without re-serialization) results in ~3.2x requests per second, and even a bit less data sent over the network:

     0Document Path:          /rest/spenttxouts/00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054.bin
 1Document Length:        184971 bytes
 2
 3Concurrency Level:      1
 4Time taken for tests:   8.060 seconds
 5Complete requests:      10000
 6Failed requests:        0
 7Keep-Alive requests:    10000
 8Total transferred:      1850780000 bytes
 9HTML transferred:       1849710000 bytes
10Requests per second:    1240.71 [#/sec] (mean)
11Time per request:       0.806 [ms] (mean)
12Time per request:       0.806 [ms] (mean, across all concurrent requests)
13Transfer rate:          224245.94 [Kbytes/sec] received

    With re-serialization (1e2b26e4f8):

     0Document Path:          /rest/spenttxouts/00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054.bin
 1Document Length:        195591 bytes
 2
 3Concurrency Level:      1
 4Time taken for tests:   26.052 seconds
 5Complete requests:      10000
 6Failed requests:        0
 7Keep-Alive requests:    10000
 8Total transferred:      1956980000 bytes
 9HTML transferred:       1955910000 bytes
10Requests per second:    383.84 [#/sec] (mean)
11Time per request:       2.605 [ms] (mean)
12Time per request:       2.605 [ms] (mean, across all concurrent requests)
13Transfer rate:          73356.44 [Kbytes/sec] received

    So I am in favor of adding ReadRawBlockUndo :)

    romanz commented at 6:20 pm on May 27, 2025:

    However, with this suggestion, and a future change in the storage serialization, there will be a breaking change on the rest interface, or alternatively a re-serialization again (back to what this pull is doing).

    I think that non-stable REST output format is would be OK, in case it enables performant data retrieval.

    maflcko commented at 8:48 am on May 28, 2025:

    Reading the undo data directly from storage (i.e. without re-serialization) results in ~3.2x requests per second, and even a bit less data sent over the network:

    Hmm. I am not sure if this is representative of the end-to-end performance. Simply fetching the raw (compressed) data is missing the decompression overhead. The end product can probably not do much with the compressed data by itself, so the benchmark should include the decompression as well.

    romanz commented at 3:53 pm on May 29, 2025:

    I have benchmarked both approaches (reading block undo data directly from storage and returning re-serialized spent txouts) using https://github.com/bitcoin/bitcoin/pull/32540/commits/d5828eadc39772bf825eb4141340a281f4a490a7 with https://github.com/romanz/bench-rest/commit/f90aaddf1c3a0c0687f373cc89b0c343e35df739:

    Fetching block undo data (using ReadRawBlockUndo)

     0$ time cargo run --release --bin blockundo
 1    Finished `release` profile [optimized] target(s) in 0.02s
 2     Running `target/release/blockundo`
 3[700000..701000) 1202[us/call] Stats { total: 5117052, value: 4220850331315943, by_type: [1545802, 2028814, 8, 5, 0, 0, 1542423] }
 4[701000..702000) 1248[us/call] Stats { total: 5016564, value: 3238540334293590, by_type: [1485990, 2015971, 4, 7, 0, 0, 1514592] }
 5[702000..703000) 1364[us/call] Stats { total: 5303401, value: 2664566072512831, by_type: [1586091, 2071826, 2, 4, 0, 0, 1645478] }
 6[703000..704000) 1378[us/call] Stats { total: 5666124, value: 2400056525309041, by_type: [1671073, 2216971, 2, 10, 0, 1, 1778067] }
 7[704000..705000) 1373[us/call] Stats { total: 5835815, value: 2765124924840842, by_type: [1607472, 2347087, 4, 17, 2, 1, 1881232] }
 8[705000..706000) 1283[us/call] Stats { total: 5571989, value: 2720571523603325, by_type: [1438779, 2236071, 4, 8, 2, 0, 1897125] }
 9[706000..707000) 1263[us/call] Stats { total: 5371702, value: 2113610858419429, by_type: [1416760, 2128942, 2, 4, 0, 1, 1825993] }
10[707000..708000) 1318[us/call] Stats { total: 5559416, value: 2439623089965085, by_type: [1492894, 2172092, 4, 10, 0, 2, 1894414] }
11[708000..709000) 1330[us/call] Stats { total: 5611331, value: 2946515191901043, by_type: [1406453, 2246277, 4, 11, 0, 1, 1958585] }
12[709000..710000) 1317[us/call] Stats { total: 5589581, value: 3176856863850345, by_type: [1289761, 2315330, 4, 12, 0, 0, 1984474] }
13
14real	0m14.856s
15user	0m8.244s
16sys	0m1.835s

    Fetching re-serialized spent txouts (using ReadBlockUndo)

     0$ time cargo run --release --bin spenttxouts
 1    Finished `release` profile [optimized] target(s) in 0.02s
 2     Running `target/release/spenttxouts`
 3[700000..701000) 3339[us/call] Stats { total: 5117052, value: 4220850331315943 }
 4[701000..702000) 3200[us/call] Stats { total: 5016564, value: 3238540334293590 }
 5[702000..703000) 3484[us/call] Stats { total: 5303401, value: 2664566072512831 }
 6[703000..704000) 3664[us/call] Stats { total: 5666124, value: 2400056525309041 }
 7[704000..705000) 3657[us/call] Stats { total: 5835815, value: 2765124924840842 }
 8[705000..706000) 3564[us/call] Stats { total: 5571989, value: 2720571523603325 }
 9[706000..707000) 3471[us/call] Stats { total: 5371702, value: 2113610858419429 }
10[707000..708000) 3597[us/call] Stats { total: 5559416, value: 2439623089965085 }
11[708000..709000) 3414[us/call] Stats { total: 5611331, value: 2946515191901043 }
12[709000..710000) 3560[us/call] Stats { total: 5589581, value: 3176856863850345 }
13
14real	0m37.479s
15user	0m5.743s
16sys	0m3.111s

    Indeed, decompression requires more work on the client side, but the overall latency improvement is still significant:

    • ab benchmark shows ~3.2x per-call latency improvement
    • this benchmark shows ~2.5x per-call latency improvement
    TheCharlatan commented at 7:31 pm on May 29, 2025:
    I do think it would be fine if the binary format changes, which given the compressor is fairly likely. If we assume users in the worst case can always use the kernel library to deserialize the undo data in the future, that would guarantee that they will have a versioned decoder for it. Would this performance difference when using ReadRawBlockUndo be significant in the grand scheme when it comes to building an electrs index romanz?
    romanz commented at 6:41 am on May 31, 2025:
    Will prepare a benchmark to compare both formats :)
    romanz commented at 1:07 pm on June 1, 2025:

    In https://github.com/bitcoin/bitcoin/pull/32540/commits/62e68bd46726dc17b657c4c334b06351fc761868, I have added /blockundo/ REST endpoint (which is using ReadBlockRawUndo and client-side deserilalization).

    I have used https://github.com/romanz/bench-rest/commit/084e07764fdb76d4bc25d2090769c0ef3f0a44c5 to benchmark it against /spenttxouts/ REST endpoint (which is using ReadBlockUndo and server-side deserialization) by reading 10k blocks from height=800000.

    In addition, I have also benchmarked /block/ REST endpoint which is needed to build an external address index (under development in Bindex, which should improve electrs index performance).

    Each benchmark was run 3 times. The block & undo data was read from Samsung 980 1 TB SSD NVMe M.2.

     0$ echo 3 | sudo tee /proc/sys/vm/drop_caches; bash run.sh --type block-undo --start 800000 --count 10000
 1[2025-06-01T11:58:45Z INFO  bench] fetching 10000 blocks
 2[2025-06-01T11:58:47Z INFO  bench] BlockUndo [@801000](/bitcoin-bitcoin/contributor/801000/) 1523[us/call]
 3[2025-06-01T11:58:48Z INFO  bench] BlockUndo [@802000](/bitcoin-bitcoin/contributor/802000/) 1556[us/call]
 4[2025-06-01T11:58:50Z INFO  bench] BlockUndo [@803000](/bitcoin-bitcoin/contributor/803000/) 1527[us/call]
 5[2025-06-01T11:58:52Z INFO  bench] BlockUndo [@804000](/bitcoin-bitcoin/contributor/804000/) 1945[us/call]
 6[2025-06-01T11:58:53Z INFO  bench] BlockUndo [@805000](/bitcoin-bitcoin/contributor/805000/) 1583[us/call]
 7[2025-06-01T11:58:55Z INFO  bench] BlockUndo [@806000](/bitcoin-bitcoin/contributor/806000/) 1522[us/call]
 8[2025-06-01T11:58:56Z INFO  bench] BlockUndo [@807000](/bitcoin-bitcoin/contributor/807000/) 1477[us/call]
 9[2025-06-01T11:58:58Z INFO  bench] BlockUndo [@808000](/bitcoin-bitcoin/contributor/808000/) 1474[us/call]
10[2025-06-01T11:58:59Z INFO  bench] BlockUndo [@809000](/bitcoin-bitcoin/contributor/809000/) 1472[us/call]
11[2025-06-01T11:59:01Z INFO  bench] BlockUndo [@810000](/bitcoin-bitcoin/contributor/810000/) 1691[us/call]
127.99user 1.42system 0:16.05elapsed 58%CPU (0avgtext+0avgdata 36676maxresident)k
13113416inputs+448outputs (144major+8848minor)pagefaults 0swaps
14
15$ echo 3 | sudo tee /proc/sys/vm/drop_caches; bash run.sh --type block-undo --start 800000 --count 10000
16[2025-06-01T11:59:14Z INFO  bench] fetching 10000 blocks
17[2025-06-01T11:59:16Z INFO  bench] BlockUndo [@801000](/bitcoin-bitcoin/contributor/801000/) 1951[us/call]
18[2025-06-01T11:59:18Z INFO  bench] BlockUndo [@802000](/bitcoin-bitcoin/contributor/802000/) 1670[us/call]
19[2025-06-01T11:59:20Z INFO  bench] BlockUndo [@803000](/bitcoin-bitcoin/contributor/803000/) 1645[us/call]
20[2025-06-01T11:59:21Z INFO  bench] BlockUndo [@804000](/bitcoin-bitcoin/contributor/804000/) 1418[us/call]
21[2025-06-01T11:59:23Z INFO  bench] BlockUndo [@805000](/bitcoin-bitcoin/contributor/805000/) 1549[us/call]
22[2025-06-01T11:59:24Z INFO  bench] BlockUndo [@806000](/bitcoin-bitcoin/contributor/806000/) 1509[us/call]
23[2025-06-01T11:59:26Z INFO  bench] BlockUndo [@807000](/bitcoin-bitcoin/contributor/807000/) 1464[us/call]
24[2025-06-01T11:59:27Z INFO  bench] BlockUndo [@808000](/bitcoin-bitcoin/contributor/808000/) 1476[us/call]
25[2025-06-01T11:59:29Z INFO  bench] BlockUndo [@809000](/bitcoin-bitcoin/contributor/809000/) 1578[us/call]
26[2025-06-01T11:59:30Z INFO  bench] BlockUndo [@810000](/bitcoin-bitcoin/contributor/810000/) 1792[us/call]
278.09user 1.39system 0:16.33elapsed 58%CPU (0avgtext+0avgdata 36448maxresident)k
28113120inputs+0outputs (144major+8837minor)pagefaults 0swaps
29
30$ echo 3 | sudo tee /proc/sys/vm/drop_caches; bash run.sh --type block-undo --start 800000 --count 10000
31[2025-06-01T11:59:35Z INFO  bench] fetching 10000 blocks
32[2025-06-01T11:59:36Z INFO  bench] BlockUndo [@801000](/bitcoin-bitcoin/contributor/801000/) 1599[us/call]
33[2025-06-01T11:59:38Z INFO  bench] BlockUndo [@802000](/bitcoin-bitcoin/contributor/802000/) 1489[us/call]
34[2025-06-01T11:59:39Z INFO  bench] BlockUndo [@803000](/bitcoin-bitcoin/contributor/803000/) 1522[us/call]
35[2025-06-01T11:59:41Z INFO  bench] BlockUndo [@804000](/bitcoin-bitcoin/contributor/804000/) 1496[us/call]
36[2025-06-01T11:59:42Z INFO  bench] BlockUndo [@805000](/bitcoin-bitcoin/contributor/805000/) 1529[us/call]
37[2025-06-01T11:59:44Z INFO  bench] BlockUndo [@806000](/bitcoin-bitcoin/contributor/806000/) 1580[us/call]
38[2025-06-01T11:59:46Z INFO  bench] BlockUndo [@807000](/bitcoin-bitcoin/contributor/807000/) 1546[us/call]
39[2025-06-01T11:59:47Z INFO  bench] BlockUndo [@808000](/bitcoin-bitcoin/contributor/808000/) 1779[us/call]
40[2025-06-01T11:59:49Z INFO  bench] BlockUndo [@809000](/bitcoin-bitcoin/contributor/809000/) 1453[us/call]
41[2025-06-01T11:59:51Z INFO  bench] BlockUndo [@810000](/bitcoin-bitcoin/contributor/810000/) 1747[us/call]
427.91user 1.28system 0:16.00elapsed 57%CPU (0avgtext+0avgdata 36828maxresident)k
43113120inputs+0outputs (144major+8845minor)pagefaults 0swaps

     0$ echo 3 | sudo tee /proc/sys/vm/drop_caches; bash run.sh --type spent-txouts --start 800000 --count 10000
 1[2025-06-01T12:00:12Z INFO  bench] fetching 10000 blocks
 2[2025-06-01T12:00:16Z INFO  bench] SpentTxouts [@801000](/bitcoin-bitcoin/contributor/801000/) 4664[us/call]
 3[2025-06-01T12:00:21Z INFO  bench] SpentTxouts [@802000](/bitcoin-bitcoin/contributor/802000/) 4566[us/call]
 4[2025-06-01T12:00:25Z INFO  bench] SpentTxouts [@803000](/bitcoin-bitcoin/contributor/803000/) 4559[us/call]
 5[2025-06-01T12:00:30Z INFO  bench] SpentTxouts [@804000](/bitcoin-bitcoin/contributor/804000/) 4362[us/call]
 6[2025-06-01T12:00:34Z INFO  bench] SpentTxouts [@805000](/bitcoin-bitcoin/contributor/805000/) 4507[us/call]
 7[2025-06-01T12:00:38Z INFO  bench] SpentTxouts [@806000](/bitcoin-bitcoin/contributor/806000/) 4211[us/call]
 8[2025-06-01T12:00:43Z INFO  bench] SpentTxouts [@807000](/bitcoin-bitcoin/contributor/807000/) 4659[us/call]
 9[2025-06-01T12:00:48Z INFO  bench] SpentTxouts [@808000](/bitcoin-bitcoin/contributor/808000/) 4509[us/call]
10[2025-06-01T12:00:52Z INFO  bench] SpentTxouts [@809000](/bitcoin-bitcoin/contributor/809000/) 4409[us/call]
11[2025-06-01T12:00:57Z INFO  bench] SpentTxouts [@810000](/bitcoin-bitcoin/contributor/810000/) 5108[us/call]
126.14user 2.24system 0:45.82elapsed 18%CPU (0avgtext+0avgdata 36628maxresident)k
13113408inputs+0outputs (144major+8873minor)pagefaults 0swaps
14
15$ echo 3 | sudo tee /proc/sys/vm/drop_caches; bash run.sh --type spent-txouts --start 800000 --count 10000
16[2025-06-01T12:01:13Z INFO  bench] fetching 10000 blocks
17[2025-06-01T12:01:18Z INFO  bench] SpentTxouts [@801000](/bitcoin-bitcoin/contributor/801000/) 4603[us/call]
18[2025-06-01T12:01:23Z INFO  bench] SpentTxouts [@802000](/bitcoin-bitcoin/contributor/802000/) 4644[us/call]
19[2025-06-01T12:01:27Z INFO  bench] SpentTxouts [@803000](/bitcoin-bitcoin/contributor/803000/) 4480[us/call]
20[2025-06-01T12:01:32Z INFO  bench] SpentTxouts [@804000](/bitcoin-bitcoin/contributor/804000/) 4709[us/call]
21[2025-06-01T12:01:36Z INFO  bench] SpentTxouts [@805000](/bitcoin-bitcoin/contributor/805000/) 4472[us/call]
22[2025-06-01T12:01:41Z INFO  bench] SpentTxouts [@806000](/bitcoin-bitcoin/contributor/806000/) 4549[us/call]
23[2025-06-01T12:01:46Z INFO  bench] SpentTxouts [@807000](/bitcoin-bitcoin/contributor/807000/) 4642[us/call]
24[2025-06-01T12:01:50Z INFO  bench] SpentTxouts [@808000](/bitcoin-bitcoin/contributor/808000/) 4535[us/call]
25[2025-06-01T12:01:54Z INFO  bench] SpentTxouts [@809000](/bitcoin-bitcoin/contributor/809000/) 4035[us/call]
26[2025-06-01T12:01:59Z INFO  bench] SpentTxouts [@810000](/bitcoin-bitcoin/contributor/810000/) 4491[us/call]
276.10user 2.18system 0:45.43elapsed 18%CPU (0avgtext+0avgdata 36676maxresident)k
28113096inputs+0outputs (144major+8853minor)pagefaults 0swaps
29
30$ echo 3 | sudo tee /proc/sys/vm/drop_caches; bash run.sh --type spent-txouts --start 800000 --count 10000
31[2025-06-01T12:02:21Z INFO  bench] fetching 10000 blocks
32[2025-06-01T12:02:25Z INFO  bench] SpentTxouts [@801000](/bitcoin-bitcoin/contributor/801000/) 4245[us/call]
33[2025-06-01T12:02:30Z INFO  bench] SpentTxouts [@802000](/bitcoin-bitcoin/contributor/802000/) 4623[us/call]
34[2025-06-01T12:02:34Z INFO  bench] SpentTxouts [@803000](/bitcoin-bitcoin/contributor/803000/) 4373[us/call]
35[2025-06-01T12:02:39Z INFO  bench] SpentTxouts [@804000](/bitcoin-bitcoin/contributor/804000/) 4573[us/call]
36[2025-06-01T12:02:43Z INFO  bench] SpentTxouts [@805000](/bitcoin-bitcoin/contributor/805000/) 4471[us/call]
37[2025-06-01T12:02:47Z INFO  bench] SpentTxouts [@806000](/bitcoin-bitcoin/contributor/806000/) 4377[us/call]
38[2025-06-01T12:02:52Z INFO  bench] SpentTxouts [@807000](/bitcoin-bitcoin/contributor/807000/) 4428[us/call]
39[2025-06-01T12:02:56Z INFO  bench] SpentTxouts [@808000](/bitcoin-bitcoin/contributor/808000/) 4370[us/call]
40[2025-06-01T12:03:01Z INFO  bench] SpentTxouts [@809000](/bitcoin-bitcoin/contributor/809000/) 4924[us/call]
41[2025-06-01T12:03:06Z INFO  bench] SpentTxouts [@810000](/bitcoin-bitcoin/contributor/810000/) 4983[us/call]
426.04user 2.29system 0:45.63elapsed 18%CPU (0avgtext+0avgdata 36672maxresident)k
43113168inputs+0outputs (144major+8872minor)pagefaults 0swaps

     0$ echo 3 | sudo tee /proc/sys/vm/drop_caches; bash run.sh --type block --start 800000 --count 10000
 1[2025-06-01T12:03:48Z INFO  bench] fetching 10000 blocks
 2[2025-06-01T12:03:54Z INFO  bench] Block [@801000](/bitcoin-bitcoin/contributor/801000/) 5623[us/call]
 3[2025-06-01T12:03:59Z INFO  bench] Block [@802000](/bitcoin-bitcoin/contributor/802000/) 5485[us/call]
 4[2025-06-01T12:04:05Z INFO  bench] Block [@803000](/bitcoin-bitcoin/contributor/803000/) 5545[us/call]
 5[2025-06-01T12:04:10Z INFO  bench] Block [@804000](/bitcoin-bitcoin/contributor/804000/) 5667[us/call]
 6[2025-06-01T12:04:16Z INFO  bench] Block [@805000](/bitcoin-bitcoin/contributor/805000/) 5541[us/call]
 7[2025-06-01T12:04:22Z INFO  bench] Block [@806000](/bitcoin-bitcoin/contributor/806000/) 5746[us/call]
 8[2025-06-01T12:04:27Z INFO  bench] Block [@807000](/bitcoin-bitcoin/contributor/807000/) 5445[us/call]
 9[2025-06-01T12:04:33Z INFO  bench] Block [@808000](/bitcoin-bitcoin/contributor/808000/) 5654[us/call]
10[2025-06-01T12:04:38Z INFO  bench] Block [@809000](/bitcoin-bitcoin/contributor/809000/) 5632[us/call]
11[2025-06-01T12:04:44Z INFO  bench] Block [@810000](/bitcoin-bitcoin/contributor/810000/) 5637[us/call]
127.38user 6.91system 0:56.25elapsed 25%CPU (0avgtext+0avgdata 36612maxresident)k
13113656inputs+448outputs (144major+9675minor)pagefaults 0swaps
14
15$ echo 3 | sudo tee /proc/sys/vm/drop_caches; bash run.sh --type block --start 800000 --count 10000
16[2025-06-01T12:04:50Z INFO  bench] fetching 10000 blocks
17[2025-06-01T12:04:57Z INFO  bench] Block [@801000](/bitcoin-bitcoin/contributor/801000/) 6722[us/call]
18[2025-06-01T12:05:03Z INFO  bench] Block [@802000](/bitcoin-bitcoin/contributor/802000/) 5844[us/call]
19[2025-06-01T12:05:08Z INFO  bench] Block [@803000](/bitcoin-bitcoin/contributor/803000/) 5566[us/call]
20[2025-06-01T12:05:14Z INFO  bench] Block [@804000](/bitcoin-bitcoin/contributor/804000/) 5608[us/call]
21[2025-06-01T12:05:19Z INFO  bench] Block [@805000](/bitcoin-bitcoin/contributor/805000/) 5539[us/call]
22[2025-06-01T12:05:25Z INFO  bench] Block [@806000](/bitcoin-bitcoin/contributor/806000/) 5359[us/call]
23[2025-06-01T12:05:30Z INFO  bench] Block [@807000](/bitcoin-bitcoin/contributor/807000/) 5490[us/call]
24[2025-06-01T12:05:36Z INFO  bench] Block [@808000](/bitcoin-bitcoin/contributor/808000/) 5453[us/call]
25[2025-06-01T12:05:41Z INFO  bench] Block [@809000](/bitcoin-bitcoin/contributor/809000/) 5689[us/call]
26[2025-06-01T12:05:47Z INFO  bench] Block [@810000](/bitcoin-bitcoin/contributor/810000/) 5607[us/call]
277.31user 6.80system 0:57.14elapsed 24%CPU (0avgtext+0avgdata 36760maxresident)k
28113160inputs+0outputs (145major+9640minor)pagefaults 0swaps
29
30$ echo 3 | sudo tee /proc/sys/vm/drop_caches; bash run.sh --type block --start 800000 --count 10000
31[2025-06-01T12:05:55Z INFO  bench] fetching 10000 blocks
32[2025-06-01T12:06:01Z INFO  bench] Block [@801000](/bitcoin-bitcoin/contributor/801000/) 5781[us/call]
33[2025-06-01T12:06:07Z INFO  bench] Block [@802000](/bitcoin-bitcoin/contributor/802000/) 5807[us/call]
34[2025-06-01T12:06:13Z INFO  bench] Block [@803000](/bitcoin-bitcoin/contributor/803000/) 5456[us/call]
35[2025-06-01T12:06:18Z INFO  bench] Block [@804000](/bitcoin-bitcoin/contributor/804000/) 5529[us/call]
36[2025-06-01T12:06:24Z INFO  bench] Block [@805000](/bitcoin-bitcoin/contributor/805000/) 5888[us/call]
37[2025-06-01T12:06:30Z INFO  bench] Block [@806000](/bitcoin-bitcoin/contributor/806000/) 5966[us/call]
38[2025-06-01T12:06:36Z INFO  bench] Block [@807000](/bitcoin-bitcoin/contributor/807000/) 5912[us/call]
39[2025-06-01T12:06:42Z INFO  bench] Block [@808000](/bitcoin-bitcoin/contributor/808000/) 5718[us/call]
40[2025-06-01T12:06:48Z INFO  bench] Block [@809000](/bitcoin-bitcoin/contributor/809000/) 5994[us/call]
41[2025-06-01T12:06:54Z INFO  bench] Block [@810000](/bitcoin-bitcoin/contributor/810000/) 6035[us/call]
427.79user 7.71system 0:58.35elapsed 26%CPU (0avgtext+0avgdata 36732maxresident)k
43113416inputs+0outputs (144major+9652minor)pagefaults 0swaps

    As @maflcko and @TheCharlatan suggested above, the overall indexing latency difference is not expected to be significantly improved by switching to using ReadBlockRawUndo instead of ReadBlockUndo - since IIUC the limiting factor in this case is fetching and parsing the block data.

    romanz commented at 1:11 pm on June 1, 2025:
    Would it be OK to switch this PR back to use /spenttxouts/ (i.e. removing 62e68bd467 and keeping 1e2b26e4f8)?
    romanz commented at 4:23 am on June 6, 2025:
    Ping @TheCharlatan @maflcko :)
    maflcko commented at 12:07 pm on June 6, 2025:

    Hmm, is there a reason why blockundo takes only 1.6ms in total end-to-end when spenttxouts takes 4.6ms in total end-to-end, where about 2.6ms happen on the server (https://github.com/bitcoin/bitcoin/pull/32540#discussion_r2109843001). So with spenttxouts, the client spends about 2ms with 18% CPU and with blockundo the client spends about .8ms with 58% CPU.

    Just wondering out of curiosity.

    Personally I’d just go with spenttxouts, but no strong opinion.

    romanz commented at 4:16 pm on June 6, 2025:

    Hmm, is there a reason why blockundo takes only 1.6ms in total end-to-end when spenttxouts takes 4.6ms in total end-to-end, where about 2.6ms happen on the server (https://github.com/bitcoin/bitcoin/pull/32540#discussion_r2109843001). So with spenttxouts, the client spends about 2ms with 18% CPU and with blockundo the client spends about .8ms with 58% CPU.

    I would like to note that #32540 (review) was benchmarked using ApacheBench, but #32540 (review) was benchmarked a custom client tool (based on ureq) - so the latencies’ delta is also affected by the HTTP client performance differences.

    If I understand correctly, the client-side handling of spenttxouts response takes 2ms * 18% = 360us CPU time, which is indeed a bit better compared to client-side handling of blockundo response, which takes 0.8ms * 58% = 464us CPU time.

    Also, most of the scripts during electrs indexing don’t require complex decompression,since most of compressed ones are P2PKH and P2SH types (see the by_type histogram below):

     0$ time cargo run --release --bin blockundo
 1    Finished `release` profile [optimized] target(s) in 0.02s
 2     Running `target/release/blockundo`
 3[700000..701000) 1202[us/call] Stats { total: 5117052, value: 4220850331315943, by_type: [1545802, 2028814, 8, 5, 0, 0, 1542423] }
 4[701000..702000) 1248[us/call] Stats { total: 5016564, value: 3238540334293590, by_type: [1485990, 2015971, 4, 7, 0, 0, 1514592] }
 5[702000..703000) 1364[us/call] Stats { total: 5303401, value: 2664566072512831, by_type: [1586091, 2071826, 2, 4, 0, 0, 1645478] }
 6[703000..704000) 1378[us/call] Stats { total: 5666124, value: 2400056525309041, by_type: [1671073, 2216971, 2, 10, 0, 1, 1778067] }
 7[704000..705000) 1373[us/call] Stats { total: 5835815, value: 2765124924840842, by_type: [1607472, 2347087, 4, 17, 2, 1, 1881232] }
 8[705000..706000) 1283[us/call] Stats { total: 5571989, value: 2720571523603325, by_type: [1438779, 2236071, 4, 8, 2, 0, 1897125] }
 9[706000..707000) 1263[us/call] Stats { total: 5371702, value: 2113610858419429, by_type: [1416760, 2128942, 2, 4, 0, 1, 1825993] }
10[707000..708000) 1318[us/call] Stats { total: 5559416, value: 2439623089965085, by_type: [1492894, 2172092, 4, 10, 0, 2, 1894414] }
11[708000..709000) 1330[us/call] Stats { total: 5611331, value: 2946515191901043, by_type: [1406453, 2246277, 4, 11, 0, 1, 1958585] }
12[709000..710000) 1317[us/call] Stats { total: 5589581, value: 3176856863850345, by_type: [1289761, 2315330, 4, 12, 0, 0, 1984474] }
13
14real	0m14.856s
15user	0m8.244s
16sys	0m1.835s

     0[@10000](/bitcoin-bitcoin/contributor/10000/) { count: 625, count_by_type: [6, 0, 0, 0, 319, 300, 0] }
 1[@20000](/bitcoin-bitcoin/contributor/20000/) { count: 532, count_by_type: [4, 0, 0, 0, 245, 283, 0] }
 2[@30000](/bitcoin-bitcoin/contributor/30000/) { count: 1712, count_by_type: [40, 0, 0, 0, 876, 796, 0] }
 3[@40000](/bitcoin-bitcoin/contributor/40000/) { count: 2208, count_by_type: [55, 0, 0, 0, 1093, 1060, 0] }
 4[@50000](/bitcoin-bitcoin/contributor/50000/) { count: 4372, count_by_type: [191, 0, 0, 0, 2103, 2078, 0] }
 5[@60000](/bitcoin-bitcoin/contributor/60000/) { count: 13608, count_by_type: [6408, 0, 0, 0, 3569, 3631, 0] }
 6[@70000](/bitcoin-bitcoin/contributor/70000/) { count: 22269, count_by_type: [15653, 0, 0, 0, 3386, 3230, 0] }
 7[@80000](/bitcoin-bitcoin/contributor/80000/) { count: 30428, count_by_type: [22242, 0, 0, 0, 4093, 4093, 0] }
 8[@90000](/bitcoin-bitcoin/contributor/90000/) { count: 30269, count_by_type: [20322, 0, 0, 0, 5050, 4897, 0] }
 9[@100000](/bitcoin-bitcoin/contributor/100000/) { count: 86340, count_by_type: [77381, 0, 0, 0, 4443, 4516, 0] }
10[@110000](/bitcoin-bitcoin/contributor/110000/) { count: 94901, count_by_type: [85488, 0, 0, 0, 4778, 4635, 0] }
11[@120000](/bitcoin-bitcoin/contributor/120000/) { count: 216635, count_by_type: [206244, 0, 0, 0, 5256, 5135, 0] }
12[@130000](/bitcoin-bitcoin/contributor/130000/) { count: 467230, count_by_type: [453412, 0, 0, 0, 6869, 6948, 1] }
13[@140000](/bitcoin-bitcoin/contributor/140000/) { count: 976449, count_by_type: [962537, 0, 0, 0, 6911, 7000, 1] }
14[@150000](/bitcoin-bitcoin/contributor/150000/) { count: 920621, count_by_type: [909053, 0, 0, 0, 5875, 5693, 0] }
15[@160000](/bitcoin-bitcoin/contributor/160000/) { count: 770334, count_by_type: [702686, 0, 0, 0, 5200, 62447, 1] }
16[@170000](/bitcoin-bitcoin/contributor/170000/) { count: 950547, count_by_type: [899533, 0, 95, 88, 3929, 46879, 23] }
17[@180000](/bitcoin-bitcoin/contributor/180000/) { count: 1198258, count_by_type: [1095960, 33, 359, 348, 3112, 98441, 5] }
18[@190000](/bitcoin-bitcoin/contributor/190000/) { count: 4185359, count_by_type: [4088500, 31, 316, 321, 2372, 93819, 0] }
19[@200000](/bitcoin-bitcoin/contributor/200000/) { count: 4339502, count_by_type: [4233624, 12, 373, 341, 2053, 103099, 0] }
20[@210000](/bitcoin-bitcoin/contributor/210000/) { count: 4441565, count_by_type: [4331440, 34, 560, 582, 1681, 107267, 1] }
21[@220000](/bitcoin-bitcoin/contributor/220000/) { count: 6371400, count_by_type: [6311580, 58, 620, 588, 1058, 56422, 1074] }
22[@230000](/bitcoin-bitcoin/contributor/230000/) { count: 7385694, count_by_type: [7337631, 26, 241, 254, 868, 45860, 814] }
23[@240000](/bitcoin-bitcoin/contributor/240000/) { count: 7630690, count_by_type: [7570439, 71, 71, 58, 442, 57704, 1905] }
24[@250000](/bitcoin-bitcoin/contributor/250000/) { count: 5437882, count_by_type: [5409041, 31, 146, 150, 413, 27769, 332] }
25[@260000](/bitcoin-bitcoin/contributor/260000/) { count: 6433106, count_by_type: [6421397, 148, 1492, 1745, 60, 8245, 19] }
26[@270000](/bitcoin-bitcoin/contributor/270000/) { count: 6655459, count_by_type: [6648335, 169, 742, 770, 239, 5155, 49] }
27[@280000](/bitcoin-bitcoin/contributor/280000/) { count: 9090816, count_by_type: [9084292, 802, 922, 940, 219, 3609, 32] }
28[@290000](/bitcoin-bitcoin/contributor/290000/) { count: 9979571, count_by_type: [9973322, 2192, 484, 559, 99, 1173, 1742] }
29[@300000](/bitcoin-bitcoin/contributor/300000/) { count: 10654043, count_by_type: [10641469, 5391, 214, 182, 74, 359, 6354] }
30[@310000](/bitcoin-bitcoin/contributor/310000/) { count: 10963860, count_by_type: [10952275, 8245, 153, 75, 59, 193, 2860] }
31[@320000](/bitcoin-bitcoin/contributor/320000/) { count: 11873101, count_by_type: [11854048, 14195, 106, 118, 21, 134, 4479] }
32[@330000](/bitcoin-bitcoin/contributor/330000/) { count: 14230246, count_by_type: [14181599, 42163, 55, 59, 39, 44, 6287] }
33[@340000](/bitcoin-bitcoin/contributor/340000/) { count: 17051600, count_by_type: [16915582, 128618, 49, 65, 12, 19, 7255] }
34[@350000](/bitcoin-bitcoin/contributor/350000/) { count: 18844424, count_by_type: [18555035, 286021, 61, 84, 12, 17, 3194] }
35[@360000](/bitcoin-bitcoin/contributor/360000/) { count: 20168255, count_by_type: [19716328, 448173, 101, 79, 19, 16, 3539] }
36[@370000](/bitcoin-bitcoin/contributor/370000/)  { count: 25484802, count_by_type: [24460043, 799300, 1555, 326, 18, 21, 223539] }
37[@380000](/bitcoin-bitcoin/contributor/380000/)  { count: 26632610, count_by_type: [25420272, 1132582, 990, 78308, 16, 16, 426] }
38[@390000](/bitcoin-bitcoin/contributor/390000/)  { count: 29920038, count_by_type: [28007502, 1911310, 165, 399, 6, 10, 646] }
39[@400000](/bitcoin-bitcoin/contributor/400000/)  { count: 33174951, count_by_type: [29994572, 3179220, 335, 546, 9, 8, 261] }
40[@410000](/bitcoin-bitcoin/contributor/410000/)  { count: 35316148, count_by_type: [30938732, 4376693, 235, 348, 15, 17, 108] }
41[@420000](/bitcoin-bitcoin/contributor/420000/)  { count: 36303753, count_by_type: [28968248, 7334817, 177, 235, 53, 33, 190] }
42[@430000](/bitcoin-bitcoin/contributor/430000/)  { count: 35653677, count_by_type: [27595998, 8055172, 210, 364, 80, 78, 1775] }
43[@440000](/bitcoin-bitcoin/contributor/440000/)  { count: 38766615, count_by_type: [32663128, 6092786, 6114, 4528, 4, 5, 50] }
44[@450000](/bitcoin-bitcoin/contributor/450000/)  { count: 40971286, count_by_type: [33990735, 6940850, 22796, 16756, 59, 67, 23] }
45[@460000](/bitcoin-bitcoin/contributor/460000/)  { count: 43714996, count_by_type: [34797370, 8769454, 78945, 68732, 29, 27, 439] }
46[@470000](/bitcoin-bitcoin/contributor/470000/)  { count: 44793610, count_by_type: [36861662, 7893476, 17469, 13333, 125, 113, 7432] }
47[@480000](/bitcoin-bitcoin/contributor/480000/)  { count: 43067356, count_by_type: [35001816, 8022393, 23940, 19081, 50, 68, 8] }
48[@490000](/bitcoin-bitcoin/contributor/490000/)  { count: 41648952, count_by_type: [34331046, 7243408, 33811, 23866, 126, 103, 16592] }
49[@500000](/bitcoin-bitcoin/contributor/500000/)  { count: 48152093, count_by_type: [39194969, 8699336, 45380, 21680, 126, 638, 189964] }
50[@510000](/bitcoin-bitcoin/contributor/510000/)  { count: 51326853, count_by_type: [41648279, 9514253, 29313, 19764, 56, 56, 115132] }
51[@520000](/bitcoin-bitcoin/contributor/520000/)  { count: 38282217, count_by_type: [26528958, 9846227, 55475, 37867, 8, 830, 1812852] }
52[@530000](/bitcoin-bitcoin/contributor/530000/)  { count: 35641361, count_by_type: [21935229, 11658151, 73825, 63940, 5, 4, 1910207] }
53[@540000](/bitcoin-bitcoin/contributor/540000/)  { count: 37925935, count_by_type: [22191957, 13021405, 59315, 69483, 3, 1, 2583771] }
54[@550000](/bitcoin-bitcoin/contributor/550000/)  { count: 41406281, count_by_type: [22222938, 16756471, 65310, 62101, 0, 1, 2299460] }
55[@560000](/bitcoin-bitcoin/contributor/560000/)  { count: 43870253, count_by_type: [22823642, 18214259, 70668, 63685, 13, 17, 2697969] }
56[@570000](/bitcoin-bitcoin/contributor/570000/)  { count: 45559248, count_by_type: [23905511, 18634825, 65411, 59658, 7, 0, 2893836] }
57[@580000](/bitcoin-bitcoin/contributor/580000/)  { count: 53162820, count_by_type: [28191011, 21458817, 120620, 62791, 2, 3, 3329576] }
58[@590000](/bitcoin-bitcoin/contributor/590000/)  { count: 49851027, count_by_type: [25669849, 20537560, 87097, 68658, 20, 18, 3487825] }
59[@600000](/bitcoin-bitcoin/contributor/600000/)  { count: 45488260, count_by_type: [21660173, 17924275, 70819, 76051, 1, 3, 5756938] }
60[@610000](/bitcoin-bitcoin/contributor/610000/)  { count: 47894909, count_by_type: [22678756, 18364758, 69839, 72874, 13, 20, 6708649] }
61[@620000](/bitcoin-bitcoin/contributor/620000/)  { count: 46658506, count_by_type: [20975983, 18233797, 66047, 69796, 0, 1, 7312882] }
62[@630000](/bitcoin-bitcoin/contributor/630000/)  { count: 51374869, count_by_type: [24060911, 22276161, 69099, 76358, 12, 13, 4892315] }
63[@640000](/bitcoin-bitcoin/contributor/640000/)  { count: 55236113, count_by_type: [25829996, 24292211, 77550, 63710, 4, 5, 4972637] }
64[@650000](/bitcoin-bitcoin/contributor/650000/)  { count: 58526986, count_by_type: [27012329, 25968726, 70162, 42038, 1, 1, 5433729] }
65[@660000](/bitcoin-bitcoin/contributor/660000/)  { count: 58860657, count_by_type: [27378085, 25013577, 44432, 27502, 31, 41, 6396989] }
66[@670000](/bitcoin-bitcoin/contributor/670000/)  { count: 61278503, count_by_type: [28702102, 25080930, 42787, 23058, 50, 52, 7429524] }
67[@680000](/bitcoin-bitcoin/contributor/680000/)  { count: 61777564, count_by_type: [27253460, 26312639, 44094, 23632, 28, 30, 8143681] }
68[@690000](/bitcoin-bitcoin/contributor/690000/)  { count: 62519812, count_by_type: [25375197, 26596005, 11322, 4831, 13, 13, 10532431] }
69[@700000](/bitcoin-bitcoin/contributor/700000/)  { count: 48081507, count_by_type: [15082138, 19654137, 68, 57, 0, 2, 13345105] }
70[@710000](/bitcoin-bitcoin/contributor/710000/)  { count: 54343089, count_by_type: [15047415, 21643297, 49, 79, 27, 25, 17652197] }
71[@720000](/bitcoin-bitcoin/contributor/720000/)  { count: 54058927, count_by_type: [12764783, 22038270, 31, 165, 1, 3, 19255674] }
72[@730000](/bitcoin-bitcoin/contributor/730000/)  { count: 55869021, count_by_type: [13969490, 21107581, 121, 65, 4, 2, 20791758] }
73[@740000](/bitcoin-bitcoin/contributor/740000/)  { count: 56314981, count_by_type: [13982223, 19821462, 13, 423, 3, 4, 22510853] }
74[@750000](/bitcoin-bitcoin/contributor/750000/)  { count: 56873212, count_by_type: [13431311, 19640397, 16, 705, 4, 4, 23800775] }
75[@760000](/bitcoin-bitcoin/contributor/760000/)  { count: 54458039, count_by_type: [12599682, 17740021, 61, 32, 2, 1, 24118240] }
76[@770000](/bitcoin-bitcoin/contributor/770000/)  { count: 60185194, count_by_type: [15003214, 18161157, 109, 40, 17, 11, 27020646] }
77[@780000](/bitcoin-bitcoin/contributor/780000/)  { count: 56773118, count_by_type: [12743378, 15004250, 12, 55, 0, 3, 29025420] }
78[@790000](/bitcoin-bitcoin/contributor/790000/)  { count: 61986604, count_by_type: [11754470, 13425167, 5, 35, 1, 2, 36806924] }
79[@800000](/bitcoin-bitcoin/contributor/800000/)  { count: 66687770, count_by_type: [11366185, 13651998, 817, 1010, 2, 4, 41667754] }
80[@810000](/bitcoin-bitcoin/contributor/810000/)  { count: 68412707, count_by_type: [11444358, 11516461, 7, 18, 0, 1, 45451862] }
81[@820000](/bitcoin-bitcoin/contributor/820000/)  { count: 70786202, count_by_type: [8420325, 10293361, 269, 21, 10, 12, 52072204] }
82[@830000](/bitcoin-bitcoin/contributor/830000/)  { count: 67585727, count_by_type: [8215289, 8912455, 0, 0, 2, 1, 50457980] }
83[@840000](/bitcoin-bitcoin/contributor/840000/)  { count: 67915846, count_by_type: [7825901, 12450054, 15, 63, 51, 57, 47639705] }
84[@850000](/bitcoin-bitcoin/contributor/850000/)  { count: 74258694, count_by_type: [5661373, 9702626, 3, 6, 27, 18, 58894641] }
85[@860000](/bitcoin-bitcoin/contributor/860000/)  { count: 77988536, count_by_type: [7161911, 9020000, 8, 2, 0, 0, 61806615] }
86[@870000](/bitcoin-bitcoin/contributor/870000/)  { count: 78477781, count_by_type: [6867005, 8178130, 16, 13, 5, 5, 63432607] }
87[@880000](/bitcoin-bitcoin/contributor/880000/)  { count: 75299928, count_by_type: [8618571, 9341597, 24, 9, 22, 20, 57339685] }
88[@890000](/bitcoin-bitcoin/contributor/890000/)  { count: 80929642, count_by_type: [9942192, 6562388, 14, 11, 1, 1, 64425035] }
89[@900000](/bitcoin-bitcoin/contributor/900000/)  { count: 71287987, count_by_type: [10087225, 5561436, 16, 16, 1, 1, 55639292] }

    Personally I’d just go with spenttxouts, but no strong opinion.

    Sounds good, thanks! If there are no objections, I’ll drop https://github.com/bitcoin/bitcoin/commit/62e68bd46726dc17b657c4c334b06351fc761868 from this PR.

  29. romanz marked this as a draft on May 26, 2025
  30. romanz force-pushed on May 28, 2025
  31. romanz marked this as ready for review on May 29, 2025
  32. romanz requested review from maflcko on May 29, 2025
  33. romanz requested review from TheCharlatan on May 29, 2025
  34. romanz force-pushed on Jun 1, 2025
  35. romanz force-pushed on Jun 6, 2025
  36. rest: fetch spent transaction outputs by blockhash 
    Today, it is possible to fetch a block's spent prevouts in order to
build an external index by using the `/rest/block/HASH.json` endpoint.
However, its performance is low due to JSON serialization overhead.

We can significantly optimize it by adding a new REST endpoint, using
a binary response format:

```
$ BLOCKHASH=00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054

$ ab -k -c 1 -n 100 http://localhost:8332/rest/block/$BLOCKHASH.json
Document Length:        13278152 bytes
Requests per second:    3.53 [#/sec] (mean)
Time per request:       283.569 [ms] (mean)

$ ab -k -c 1 -n 10000 http://localhost:8332/rest/spentoutputs/$BLOCKHASH.bin
Document Length:        195591 bytes
Requests per second:    254.47 [#/sec] (mean)
Time per request:       3.930 [ms] (mean)
```

Currently, this PR is being used and tested by Bindex:

 * https://github.com/romanz/bindex-rs

This PR would allow to improve the performance of external indexers
such as electrs, ElectrumX, Fulcrum and Blockbook:

 * https://github.com/romanz/electrs (also https://github.com/Blockstream/electrs and https://github.com/mempool/electrs)
 * https://github.com/spesmilo/electrumx
 * https://github.com/cculianu/Fulcrum
 * https://github.com/trezor/blockbook
    d4e212e8a6
  37. romanz force-pushed on Jun 7, 2025
  38. maflcko commented at 2:17 pm on June 9, 2025: member

    Only change since my previous review is removing the cast.

    re-ACK d4e212e8a69ea118acb6caa1a7efe64a77bdfdd2 🚘

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
2trusted comment: re-ACK d4e212e8a69ea118acb6caa1a7efe64a77bdfdd2 🚘
3VxfR7wSBgPaQX1vc+ro41f4+N2bwayfQpgVJ9S5SISkZPNToHtNZY3B+nnVzO87DK6EV+ID3Kdx4kTA7FdM2DA==
  39. TheCharlatan approved
  40. TheCharlatan commented at 1:18 pm on June 15, 2025: contributor

    ACK d4e212e8a69ea118acb6caa1a7efe64a77bdfdd2

    And I guess this should get a release note?

  41. doc: add release notes for #32540 c48846ec41
  42. romanz commented at 6:21 pm on June 15, 2025: contributor
    Many thanks! Added a release note in https://github.com/bitcoin/bitcoin/pull/32540/commits/c48846ec4169f749d28da05de849c43a488c3a70 - please let me know if it’s OK :)
  43. maflcko commented at 7:26 am on June 16, 2025: member

    re-ACK c48846ec4169f749d28da05de849c43a488c3a70 📶

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
2trusted comment: re-ACK c48846ec4169f749d28da05de849c43a488c3a70 📶
3uGe86fOs2dFzU1UXO6ZFgLrq+AbqZ4rMcAlNYGwHYl6XqYjtam3F0q+pLXgYCFaqe9RI9XDXx/yPlH7m7HZXCA==
  44. DrahtBot requested review from TheCharlatan on Jun 16, 2025
  45. TheCharlatan approved
  46. TheCharlatan commented at 7:42 am on June 16, 2025: contributor
    Re-ACK c48846ec4169f749d28da05de849c43a488c3a70
  47. fanquake referenced this in commit 084eee0291 on Jun 16, 2025
  48. maflcko removed review request from tapcrafter on Jun 24, 2025
  49. maflcko requested review from stickies-v on Jun 24, 2025
  50. maflcko requested review from tapcrafter on Jun 24, 2025
  51. in src/rest.cpp:322 in d4e212e8a6 outdated 
    317+        }
318+        result.push_back(std::move(tx_prevouts));
319+    }
320+}
321+
322+static bool rest_spent_txouts(const std::any& context, HTTPRequest* req, const std::string& strURIPart)
    achow101 commented at 9:37 pm on June 27, 2025:

    In d4e212e8a69ea118acb6caa1a7efe64a77bdfdd2 “rest: fetch spent transaction outputs by blockhash”

    nit: New code should use snake_case

    0static bool rest_spent_txouts(const std::any& context, HTTPRequest* req, const std::string& uri_part)
    romanz commented at 6:17 am on June 28, 2025:
    Thanks! Will open a new PR to fix: https://github.com/bitcoin/bitcoin/pull/32825
    romanz commented at 1:12 pm on June 30, 2025:
    Fixed in #32825.
  52. achow101 commented at 9:37 pm on June 27, 2025: member
    ACK c48846ec4169f749d28da05de849c43a488c3a70
  53. achow101 merged this on Jun 27, 2025
  54. achow101 closed this on Jun 27, 2025
  55. romanz deleted the branch on Jun 28, 2025
  56. romanz commented at 6:18 am on June 28, 2025: contributor
    Many thanks!
  57. romanz referenced this in commit d228c4480f on Jun 28, 2025
  58. romanz referenced this in commit cc2ec2c174 on Jun 28, 2025
  59. romanz referenced this in commit 856f4235b1 on Jun 28, 2025
  60. HowHsu referenced this in commit f7690a7c0a on Jun 28, 2025
  61. fanquake referenced this in commit 9501738e1c on Jun 30, 2025


romanz DrahtBot yancyribbens tapcrafter adyshimony luke-jr maflcko TheCharlatan achow101


maflcko stickies-v tapcrafter

Labels
RPC/REST/ZMQ

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-07-31 09:13 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me