rest: fetch spent transaction outputs by blockhash

romanz commented at 7:27 am on May 17, 2025: contributor

Today, it is possible to fetch a block’s spent prevouts in order to build an external index by using the /rest/block/BLOCKHASH.json endpoint. However, its performance is low due to JSON serialization overhead.

We can significantly optimize it by adding a new REST API endpoint, using a binary response format (returning a collection of spent txout lists, one per each block transaction):

 0$ BLOCKHASH=00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054
 1
 2$ ab -k -c 1 -n 100 http://localhost:8332/rest/block/$BLOCKHASH.json
 3Document Length:        13278152 bytes
 4Requests per second:    3.53 [#/sec] (mean)
 5Time per request:       283.569 [ms] (mean)
 6
 7$ ab -k -c 1 -n 10000 http://localhost:8332/rest/spenttxouts/$BLOCKHASH.bin
 8Document Length:        195591 bytes
 9Requests per second:    254.47 [#/sec] (mean)
10Time per request:       3.930 [ms] (mean)

Currently, this PR is being used and tested by Bindex¹.

This PR would allow to improve the performance of external indexers such as electrs², ElectrumX³, Fulcrum⁴ and Blockbook⁵.

DrahtBot commented at 7:27 am on May 17, 2025: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/32540.

Reviews

See the guideline for information on the review process.

Type	Reviewers
Concept ACK	adyshimony
Stale ACK	tapcrafter

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

DrahtBot added the label RPC/REST/ZMQ on May 17, 2025

romanz force-pushed on May 17, 2025

romanz commented at 3:47 pm on May 17, 2025: contributor

Force-pushed to fix a typo in commit description.

yancyribbens commented at 6:42 pm on May 17, 2025: contributor

Nice. I wonder about just adding a content-type like option so that the api endpoint is the same but you simply say if you want binary or json content returned? Maybe there’s other endpoints that it could then be applied to as well..

romanz commented at 9:06 am on May 18, 2025: contributor

Nice.

Thanks :)

I wonder about just adding a content-type like option so that the api endpoint is the same but you simply say if you want binary or json content returned? Maybe there’s other endpoints that it could then be applied to as well.

Good idea - would it be OK to implement it in a separate PR?

in src/rest.cpp:324 in ffe571f461 outdated

319+    WriteCompactSize(ssSpentResponse, block_undo.vtxundo.size() + 1);
320+    WriteCompactSize(ssSpentResponse, 0); // block_undo.vtxundo doesn't contain coinbase tx
321+    for (const CTxUndo& tx_undo : block_undo.vtxundo) {
322+        WriteCompactSize(ssSpentResponse, tx_undo.vprevout.size());
323+        for (const Coin& coin : tx_undo.vprevout) {
324+            coin.out.Serialize(ssSpentResponse);

tapcrafter commented at 12:53 pm on May 18, 2025:

Is there a way to document the output of this REST method? From the name alone I would’ve expected it to return a list of outpoints. But it seems to return a list of transaction outputs (CTxOut or Coin depending on the context). Which absolutely makes sense given the use case. So perhaps a different name would help make that more clear? Perhaps rest/spenttxouts?

romanz commented at 5:09 am on May 19, 2025:

So perhaps a different name would help make that more clear? Perhaps rest/spenttxouts?

Sounds good, thanks! Fixed in 8cb0465def.

tapcrafter commented at 1:00 pm on May 18, 2025: none

tACK ffe571f461930b7a05a3cf9f7128e843ea9f7e2d

Running ffe571f461930b7a05a3cf9f7128e843ea9f7e2d:

 0$ ./build/bin/bitcoind -regtest -rpcallowip=::1 -rpcuser=u -rpcpassword=p -rest -txindex
 1
 2# Invalid extension:
 3
 4$ curl -v -g 'u:p@localhost:18443/rest/spentoutputs/4376e2f945afef224981b665778ec45ebe64745e7d768c2937a8e271b69d708a.json'
 5
 6* Host localhost:18443 was resolved.
 7...
 8< HTTP/1.1 404 Not Found
 9< Content-Type: text/plain
10< Date: Sun, 18 May 2025 12:28:13 GMT
11< Content-Length: 47
12< 
13output format not found (available: bin, hex)
14* Connection [#0](/bitcoin-bitcoin/0/) to host localhost left intact
15
16
17# Correct extension:
18
19$ curl -v -g 'u:p@localhost:18443/rest/spentoutputs/4376e2f945afef224981b665778ec45ebe64745e7d768c2937a8e271b69d708a.hex'
20* Host localhost:18443 was resolved.
21...
22< HTTP/1.1 200 OK
23< Content-Type: text/plain
24< Date: Sun, 18 May 2025 12:28:43 GMT
25< Content-Length: 75
26< 
2702000100f2052a010000001976a914090454b16d8fd499365c53a2226373c6d8734b5188ac
28* Connection [#0](/bitcoin-bitcoin/0/) to host localhost left intact
29
30# Which corresponds to the TxOut of the spent transaction:
31
32$ bitcoin-cli -regtest -rpcuser=u -rpcpassword=p decoderawtransaction 020000000001010000000000000000000000000000000000000000000000000000000000000000ffffffff025500feffffff0200f2052a010000001976a914090454b16d8fd499365c53a2226373c6d8734b5188ac0000000000000000266a24aa21a9ede2f61c3f71d1defd3fa999dfa36953755c690689799962b48bebd836974e8cf90120000000000000000000000000000000000000000000000000000000000000000004000000
33{
34  "txid": "38e6ad741857df84af27a709c1d99397b6e33c5078b284d49da6ae74b78d9bc6",
35  "hash": "b254f236b5141afd961bf80918b1417aa1d7f7ced2cbdd7a3bfbd243748e2c6f",
36  "version": 2,
37  "size": 170,
38  "vsize": 143,
39  "weight": 572,
40  "locktime": 4,
41  "vin": [
42    {
43      "coinbase": "5500",
44      "txinwitness": [
45        "0000000000000000000000000000000000000000000000000000000000000000"
46      ],
47      "sequence": 4294967294
48    }
49  ],
50  "vout": [
51    {
52      "value": 50.00000000,
53      "n": 0,
54      "scriptPubKey": {
55        "asm": "OP_DUP OP_HASH160 090454b16d8fd499365c53a2226373c6d8734b51 OP_EQUALVERIFY OP_CHECKSIG",
56        "desc": "addr(mgLdZh69L6Zzs68mfvSkTcnz12XtgSQVJE)#ggfft2ks",
57        "hex": "76a914090454b16d8fd499365c53a2226373c6d8734b5188ac",
58        "address": "mgLdZh69L6Zzs68mfvSkTcnz12XtgSQVJE",
59        "type": "pubkeyhash"
60      }
61    },
62    {
63      "value": 0.00000000,
64      "n": 1,
65      "scriptPubKey": {
66        "asm": "OP_RETURN aa21a9ede2f61c3f71d1defd3fa999dfa36953755c690689799962b48bebd836974e8cf9",
67        "desc": "raw(6a24aa21a9ede2f61c3f71d1defd3fa999dfa36953755c690689799962b48bebd836974e8cf9)#cav96mf3",
68        "hex": "6a24aa21a9ede2f61c3f71d1defd3fa999dfa36953755c690689799962b48bebd836974e8cf9",
69        "type": "nulldata"
70      }
71    }
72  ]
73}

bitcoin deleted a comment on May 18, 2025

yancyribbens commented at 1:50 pm on May 18, 2025: contributor

Good idea - would it be OK to implement it in a separate PR?

Sure, as you wish. I’m no authority :)

rest: fetch spent transaction outputs by blockhash

Today, it is possible to fetch a block's spent prevouts in order to
build an external index by using the `/rest/block/HASH.json` endpoint.
However, its performance is low due to JSON serialization overhead.

We can significantly optimize it by adding a new REST endpoint, using
a binary response format:

```
$ BLOCKHASH=00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054

$ ab -k -c 1 -n 100 http://localhost:8332/rest/block/$BLOCKHASH.json
Document Length:        13278152 bytes
Requests per second:    3.53 [#/sec] (mean)
Time per request:       283.569 [ms] (mean)

$ ab -k -c 1 -n 10000 http://localhost:8332/rest/spentoutputs/$BLOCKHASH.bin
Document Length:        195591 bytes
Requests per second:    254.47 [#/sec] (mean)
Time per request:       3.930 [ms] (mean)
```

Currently, this PR is being used and tested by Bindex:

 * https://github.com/romanz/bindex-rs

This PR would allow to improve the performance of external indexers
such as electrs, ElectrumX, Fulcrum and Blockbook:

 * https://github.com/romanz/electrs (also https://github.com/Blockstream/electrs and https://github.com/mempool/electrs)
 * https://github.com/spesmilo/electrumx
 * https://github.com/cculianu/Fulcrum
 * https://github.com/trezor/blockbook

8cb0465def

romanz force-pushed on May 19, 2025

adyshimony commented at 2:26 am on May 22, 2025: none

tACK

https://github.com/romanz/bitcoin/commit/8cb0465defdf96a86b7485ff098d2ba70843e942

Test:

 0$ ab -k -c 1 -n 100 http://localhost:8332/rest/spenttxouts/00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054.bin
 1This is ApacheBench, Version 2.3 <$Revision: 1903618 $>
 2Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
 3Licensed to The Apache Software Foundation, http://www.apache.org/
 4
 5Benchmarking localhost (be patient).....done
 6
 7
 8Server Software:        
 9Server Hostname:        localhost
10Server Port:            8332
11
12Document Path:          /rest/spenttxouts/00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054.bin
13Document Length:        195591 bytes
14
15Concurrency Level:      1
16Time taken for tests:   0.444 seconds
17Complete requests:      100
18Failed requests:        0
19Keep-Alive requests:    100
20Total transferred:      19569800 bytes
21HTML transferred:       19559100 bytes
22Requests per second:    225.09 [#/sec] (mean)
23Time per request:       4.443 [ms] (mean)
24Time per request:       4.443 [ms] (mean, across all concurrent requests)
25Transfer rate:          43016.55 [Kbytes/sec] received
26
27Connection Times (ms)
28              min  mean[+/-sd] median   max
29Connect:        0    0   0.0      0       0
30Processing:     3    4   1.3      4       7
31Waiting:        3    4   1.3      4       7
32Total:          3    4   1.3      4       7
33
34Percentage of the requests served within a certain time (ms)
35  50%      4
36  66%      5
37  75%      6
38  80%      6
39  90%      7
40  95%      7
41  98%      7
42  99%      7
43 100%      7 (longest request)

On my machine there is a significant improvement , well done.

 0$ ab -k -c 1 -n 100 http://localhost:8332/rest/block/00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054.json
 1This is ApacheBench, Version 2.3 <$Revision: 1903618 $>
 2Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
 3Licensed to The Apache Software Foundation, http://www.apache.org/
 4
 5Benchmarking localhost (be patient).....done
 6
 7
 8Server Software:        
 9Server Hostname:        localhost
10Server Port:            8332
11
12Document Path:          /rest/block/00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054.json
13Document Length:        13278152 bytes
14
15Concurrency Level:      1
16Time taken for tests:   33.053 seconds
17Complete requests:      100
18Failed requests:        0
19Keep-Alive requests:    100
20Total transferred:      1327825300 bytes
21HTML transferred:       1327815200 bytes
22Requests per second:    3.03 [#/sec] (mean)
23Time per request:       330.530 [ms] (mean)
24Time per request:       330.530 [ms] (mean, across all concurrent requests)
25Transfer rate:          39231.07 [Kbytes/sec] received
26
27Connection Times (ms)
28              min  mean[+/-sd] median   max
29Connect:        0    0   0.0      0       0
30Processing:   308  331  17.5    327     396
31Waiting:      297  315  16.7    310     386
32Total:        308  331  17.5    327     396
33
34Percentage of the requests served within a certain time (ms)
35  50%    327
36  66%    334
37  75%    337
38  80%    339
39  90%    345
40  95%    377
41  98%    396
42  99%    396
43 100%    396 (longest request)

in src/rest.cpp:287 in 8cb0465def outdated

281@@ -281,6 +282,69 @@ static bool rest_headers(const std::any& context,
282     }
283 }
284 
285+static bool rest_spent_txouts(const std::any& context, HTTPRequest* req, const std::string& strURIPart)
286+{
287+    if (!CheckWarmup(req))

luke-jr commented at 9:00 am on May 22, 2025:

nit: braces required

romanz commented at 3:39 pm on May 22, 2025:

Thanks, fixed in https://github.com/bitcoin/bitcoin/pull/32540/commits/9d7e23e2f505ce6cbe830fc607cf203b1a48ba0d.

luke-jr commented at 9:00 am on May 22, 2025: member

Not sure we should export a custom binary format for undo data like this. At least there should be a JSON option?

romanz marked this as a draft on May 22, 2025

fixup! rest: fetch spent transaction outputs by blockhash 9d7e23e2f5

romanz commented at 3:58 pm on May 22, 2025: contributor

Not sure we should export a custom binary format for undo data like this.

In my opinion, using a binary format allows us to achieve significantly better performance when building an external index (compared to using /rest/block/HASH.json).

I have followed the example of the /rest/getutxos/ endpoint, which also uses binary encoding when it returns a std::vector<CCoin> outs: https://github.com/bitcoin/bitcoin/blob/d2c9fc84e17120f186a54ef92bab76ea7e8d31b5/src/rest.cpp#L906

romanz commented at 3:58 pm on May 22, 2025: contributor

At least there should be a JSON option?

Sounds good, added in https://github.com/bitcoin/bitcoin/pull/32540/commits/9d7e23e2f505ce6cbe830fc607cf203b1a48ba0d:

Tested with:

0curl -s http://localhost:8332/rest/spenttxouts/00000000000034a32d25a4df37018194645cd62bf311b04ce322da9d800d4576.json | jq .

 0[
 1  [],
 2  [
 3    {
 4      "value": 19.9,
 5      "scriptPubKey": {
 6        "asm": "OP_DUP OP_HASH160 9f1c396fc47b5e8127d8fca2cbe0da12fb65b18b OP_EQUALVERIFY OP_CHECKSIG",
 7        "desc": "addr(1FWJFZiX8v1y6Mf14bzT2qTjzKgJ948gTN)#n6pdnxnu",
 8        "hex": "76a9149f1c396fc47b5e8127d8fca2cbe0da12fb65b18b88ac",
 9        "address": "1FWJFZiX8v1y6Mf14bzT2qTjzKgJ948gTN",
10        "type": "pubkeyhash"
11      }
12    }
13  ],
14  [
15    {
16      "value": 1.13,
17      "scriptPubKey": {
18        "asm": "OP_DUP OP_HASH160 63ecfce5e3b086c6b4c9a8f68b64c70eaf4f579e OP_EQUALVERIFY OP_CHECKSIG",
19        "desc": "addr(1A7MoCV1q8L6K2k6sJLnPFDJSR3kmiyubF)#k7amcrpk",
20        "hex": "76a91463ecfce5e3b086c6b4c9a8f68b64c70eaf4f579e88ac",
21        "address": "1A7MoCV1q8L6K2k6sJLnPFDJSR3kmiyubF",
22        "type": "pubkeyhash"
23      }
24    }
25  ],
26  [
27    {
28      "value": 0.129,
29      "scriptPubKey": {
30        "asm": "OP_DUP OP_HASH160 7c78cf144b8bf68372403cf5690aa49560275423 OP_EQUALVERIFY OP_CHECKSIG",
31        "desc": "addr(1CM9WjjPScLSfQaYXq9YY5VgqUh6bP87rw)#nrcg7jla",
32        "hex": "76a9147c78cf144b8bf68372403cf5690aa4956027542388ac",
33        "address": "1CM9WjjPScLSfQaYXq9YY5VgqUh6bP87rw",
34        "type": "pubkeyhash"
35      }
36    }
37  ]
38]

romanz marked this as ready for review on May 22, 2025

yancyribbens commented at 1:10 pm on May 24, 2025: contributor

would it be OK to implement it in a separate PR?

I was thinking it might be not ideal to have this endpoint in addition to adding a content-type. Unless the plan is to deprecate this after content type is added? Maybe that’s what is meant by the two conflicting types mentioned #32583 (comment)

yancyribbens commented at 2:12 pm on May 24, 2025: contributor

I was thinking it might be not ideal to have this endpoint in addition to adding a content-type. Unless the plan is to deprecate this after content type is added? Maybe that’s what is meant by the two conflicting types mentioned #32583 (comment)

Please disregard

rest: fetch spent transaction outputs by blockhash #32540

Code Coverage & Benchmarks

Reviews