rest: allow reading partial block data from storage #33657

pull romanz wants to merge 1 commits into bitcoin:master from romanz:romanz/rest-blockdata changing 7 files +200 −17
  1. romanz commented at 11:08 am on October 19, 2025: contributor

    It allows fetching specific transactions using an external index, following #32541 (comment).

    Currently, electrs and other indexers map between an address/scripthash to the list of the relevant transactions.

    However, in order to fetch those transactions from bitcoind, electrs relies on reading the whole block and post-filtering for a specific transaction1. Other indexers use a txindex to fetch a transaction using its txid 234.

    The above approach has significant storage and CPU overhead, since the txid is a pseudo-random 32-byte value. Also, mainnet txindex takes ~60GB today.

    This PR is adding support for using the transaction’s position within its block to be able to fetch it directly using REST API, using the following HTTP request:

    0GET /rest/blockpart/BLOCKHASH.bin?offset=OFFSET&size=SIZE
    
  2. DrahtBot added the label RPC/REST/ZMQ on Oct 19, 2025
  3. DrahtBot commented at 11:08 am on October 19, 2025: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33657.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    Concept ACK optout21
    Stale ACK ubbabeck, sedited, hodlinator

    If your review is incorrectly listed, please copy-paste <!–meta-tag:bot-skip–> into the comment that the bot should ignore.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #31860 (init: Take lock on blocks directory in BlockManager ctor by sedited)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  4. romanz force-pushed on Oct 19, 2025
  5. andrewtoth commented at 9:30 pm on October 19, 2025: contributor

    Some of the PR description is reused from the previous PR, but is now stale.

    This PR is adding support for using the transaction’s position within its block to be able to fetch it directly using REST API, using the following HTTP request (to fetch the N-th transaction from BLOCKHASH):

    This does not support fetching an N-th transaction, only a slice of a block.

    If binary response format is used, the transaction data will be read directly from the storage and sent back to the client, without any deserialization overhead.

    There is no way to use this in non-binary form (except hex I suppose).

  6. in src/node/blockstorage.cpp:1096 in 787d6df48a
    1093+            offset = *part_offset;
    1094+        }
    1095+
    1096+        size_t size = blk_size - offset;
    1097+        if (part_size.has_value()) {
    1098+            if (*part_size > size) {
    


    andrewtoth commented at 11:04 pm on October 19, 2025:
    Do we want to support zero *part_size? Should probably return false too?

    romanz commented at 8:21 pm on October 20, 2025:
    Sounds good - fixed in 787d6df48a..c4bf7a77c4.

    andrewtoth commented at 10:22 pm on October 20, 2025:
    We should probably cover this as well as other cases of ReadRawBlockPart in blockmanager_tests.cpp.

    romanz commented at 8:26 pm on October 21, 2025:
  7. in src/node/blockstorage.h:417 in 787d6df48a
    413@@ -414,6 +414,7 @@ class BlockManager
    414     bool ReadBlock(CBlock& block, const FlatFilePos& pos, const std::optional<uint256>& expected_hash) const;
    415     bool ReadBlock(CBlock& block, const CBlockIndex& index) const;
    416     bool ReadRawBlock(std::vector<std::byte>& block, const FlatFilePos& pos) const;
    417+    bool ReadRawBlockPart(std::vector<std::byte>& block, const FlatFilePos& pos, std::optional<size_t> part_offset, std::optional<size_t> part_size) const;
    


    optout21 commented at 11:26 am on October 20, 2025:
    nit: Declaration differs from definition, block vs data parameter name

    romanz commented at 8:21 pm on October 20, 2025:
    Thanks - fixed in 787d6df48a..c4bf7a77c4.
  8. in src/rest.cpp:492 in 787d6df48a outdated
    495@@ -472,6 +496,11 @@ static bool rest_block_notxdetails(const std::any& context, HTTPRequest* req, co
    496     return rest_block(context, req, uri_part, TxVerbosity::SHOW_TXID);
    497 }
    498 
    499+static bool rest_block_part(const std::any& context, HTTPRequest* req, const std::string& uri_part)
    


    optout21 commented at 11:31 am on October 20, 2025:
    This way both block and blockpart api’s take the same paramaters, including "offset" and "size", right? Is that intended?

    romanz commented at 8:14 pm on October 20, 2025:
    Right - I have used the same implementation for both endpoints for simplicity and to avoid code duplication. I can add an additional boolean argument to enable offset and size parameter handling for the /rest/blockpart endpoint (and disable it for /rest/block endpoint). WDYT?

    andrewtoth commented at 9:28 pm on October 20, 2025:

    If I understand correctly, this would allow passing offset or size to /rest/block along with .json format specification. Normally this would fail to deserialize, but if very unlucky it could deserialize to what looks like a block and return? That would be very bad. Similarly if not using .json, this would return incorrect data that a user might not expect (for instance if they switched rest endpoints but forgot to remove parameters).

    I would recommend disabling these parameters if not calling /rest/blockpart.


    romanz commented at 6:01 am on October 21, 2025:
    Agree - fixed in c4bf7a77c4..c9bfd298be (offset and size are parsed only for /rest/blockpart endpoint).
  9. romanz force-pushed on Oct 20, 2025
  10. romanz commented at 8:23 pm on October 20, 2025: contributor

    Some of the PR description is reused from the previous PR, but is now stale.

    Good catch, thanks! Updated PR description: #33657#issue-3529967048

  11. romanz force-pushed on Oct 21, 2025
  12. optout21 commented at 9:48 am on October 21, 2025: none
    Concept ACK c9bfd298be422de7e989fe244fb4281c507068a3
  13. romanz force-pushed on Oct 21, 2025
  14. romanz requested review from andrewtoth on Oct 22, 2025
  15. ubbabeck commented at 7:33 pm on October 22, 2025: none

    tACK 301116e855

    What was done:

    • functional test.
    • unit test
    • tested the new rest endpoint rest/blockpart/<blockhash>.bin manually with curl with different offsets and sizes.

    If there are any additional test you’d like me to help with let me know. I also did a perf dump, but I need some guidelines on what to measure/ grep for.

  16. DrahtBot requested review from optout21 on Oct 22, 2025
  17. romanz commented at 9:44 pm on October 22, 2025: contributor

    Many thanks @ubbabeck!

    If there are any additional test you’d like me to help with let me know.

    It would be interesting to compare the performance for single transaction fetching, similar to how it was done in #32541 (comment), comparing txindex-based and locationindex-based queries.

  18. ubbabeck commented at 5:12 pm on October 23, 2025: none

    It would be interesting to compare the performance for single transaction fetching, similar to how it was done in #32541 (comment), comparing txindex-based and locationindex-based queries.

    Here are the results from testing. I used hey to benchmark and your rust getrawtransaction.

    Was this somewhat you were looking for? If not let me know and I’ll redo them correctly.

    Maybe not relevant to test, but fetching the whole block using blockpart.

    hey -c 1 -n 100000 http://localhost:8332/rest/blockpart/000000000000000000017bfd05b5fa367a424c4a565a4baf7950d9e8605df8ec.bin

     0Summary:
     1  Total:	114.4020 secs
     2  Slowest:	0.0048 secs
     3  Fastest:	0.0009 secs
     4  Average:	0.0011 secs
     5  Requests/sec:	874.1103
     6
     7  Total data:	146938500000 bytes
     8  Size/request:	1469385 bytes
     9
    10Response time histogram:
    11  0.001 [1]	|
    12  0.001 [93859]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
    13  0.002 [4835]	|■■
    14  0.002 [1052]	|
    15  0.002 [165]	|
    16  0.003 [65]	|
    17  0.003 [20]	|
    18  0.004 [2]	|
    19  0.004 [0]	|
    20  0.004 [0]	|
    21  0.005 [1]	|
    22
    23
    24Latency distribution:
    25  10% in 0.0010 secs
    26  25% in 0.0011 secs
    27  50% in 0.0011 secs
    28  75% in 0.0012 secs
    29  90% in 0.0013 secs
    30  95% in 0.0013 secs
    31  99% in 0.0018 secs
    32
    33Details (average, fastest, slowest):
    34  DNS+dialup:	0.0000 secs, 0.0009 secs, 0.0048 secs
    35  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0006 secs
    36  req write:	0.0000 secs, 0.0000 secs, 0.0003 secs
    37  resp wait:	0.0004 secs, 0.0002 secs, 0.0034 secs
    38  resp read:	0.0007 secs, 0.0002 secs, 0.0030 secs
    39
    40Status code distribution:
    41  [200]	100000 responses
    

    Using a random offset and size of 310 bytes to simulate fetching a transaction.

    hey -c 16 -n 100000 http://localhost:8332/rest/blockpart/000000000000000000017bfd05b5fa367a424c4a565a4baf7950d9e8605df8ec.bin\?offset\=4000\&size\=310

     0Summary:
     1  Total:	1.7338 secs
     2  Slowest:	0.0023 secs
     3  Fastest:	0.0001 secs
     4  Average:	0.0003 secs
     5  Requests/sec:	57675.3814
     6  
     7  Total data:	31000000 bytes
     8  Size/request:	310 bytes
     9
    10Response time histogram:
    11  0.000 [1]	|
    12  0.000 [57797]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
    13  0.000 [41217]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
    14  0.001 [824]	|■
    15  0.001 [76]	|
    16  0.001 [10]	|
    17  0.001 [15]	|
    18  0.002 [20]	|
    19  0.002 [24]	|
    20  0.002 [10]	|
    21  0.002 [6]	|
    22
    23
    24Latency distribution:
    25  10% in 0.0002 secs
    26  25% in 0.0003 secs
    27  50% in 0.0003 secs
    28  75% in 0.0003 secs
    29  90% in 0.0003 secs
    30  95% in 0.0003 secs
    31  99% in 0.0005 secs
    32
    33Details (average, fastest, slowest):
    34  DNS+dialup:	0.0000 secs, 0.0001 secs, 0.0023 secs
    35  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0015 secs
    36  req write:	0.0000 secs, 0.0000 secs, 0.0008 secs
    37  resp wait:	0.0003 secs, 0.0000 secs, 0.0018 secs
    38  resp read:	0.0000 secs, 0.0000 secs, 0.0007 secs
    39
    40Status code distribution:
    41  [200]	100000 responses
    

    One runner

    hey -c 1 -n 100000 http://localhost:8332/rest/blockpart/000000000000000000017bfd05b5fa367a424c4a565a4baf7950d9e8605df8ec.bin\?offset\=4000\&size\=310

     0Summary:
     1  Total:	1.7338 secs
     2  Slowest:	0.0023 secs
     3  Fastest:	0.0001 secs
     4  Average:	0.0003 secs
     5  Requests/sec:	57675.3814
     6  
     7  Total data:	31000000 bytes
     8  Size/request:	310 bytes
     9
    10Response time histogram:
    11  0.000 [1]	|
    12  0.000 [57797]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
    13  0.000 [41217]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
    14  0.001 [824]	|■
    15  0.001 [76]	|
    16  0.001 [10]	|
    17  0.001 [15]	|
    18  0.002 [20]	|
    19  0.002 [24]	|
    20  0.002 [10]	|
    21  0.002 [6]	|
    22
    23
    24Latency distribution:
    25  10% in 0.0002 secs
    26  25% in 0.0003 secs
    27  50% in 0.0003 secs
    28  75% in 0.0003 secs
    29  90% in 0.0003 secs
    30  95% in 0.0003 secs
    31  99% in 0.0005 secs
    32
    33Details (average, fastest, slowest):
    34  DNS+dialup:	0.0000 secs, 0.0001 secs, 0.0023 secs
    35  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0015 secs
    36  req write:	0.0000 secs, 0.0000 secs, 0.0008 secs
    37  resp wait:	0.0003 secs, 0.0000 secs, 0.0018 secs
    38  resp read:	0.0000 secs, 0.0000 secs, 0.0007 secs
    39
    40Status code distribution:
    41  [200]	100000 responses
    

    Rust getrawtransaction benchmark

    0time cargo run --release -- 4137d0dbad434d68a4f52b7bebcba91ddac3f7f5c92b84130432bd6b5e2ea57a 0000000000000000000083a0cff38278aae196d6d923a7e8ee7e5a0e371226fe
    1   Compiling bench-getrawtx v0.1.0 (review/bench-getrawtx)
    2    Finished `release` profile [optimized] target(s) in 0.29s
    3     Running `target/x86_64-unknown-linux-gnu/release/bench-getrawtx 4137d0dbad434d68a4f52b7bebcba91ddac3f7f5c92b84130432bd6b5e2ea57a 0000000000000000000083a0cff38278aae196d6d923a7e8ee7e5a0e371226fe`
    4iterations = 100000
    5average RPC duration = 130.319µs
    6cargo run --release --    1.29s user 1.09s system 17% cpu 13.342 total
    
  19. romanz commented at 9:43 pm on October 25, 2025: contributor

    Tested the performance of the new REST endpoint using ab on mainnet with various SIZE parameters:

    0$ BLOCKHASH=00000000000000000001eae3683a5350b67ddb17d9c7b6c8010ab5b36ccbaa09
    1$ ab -q -k -c 1 -n 100000 'http://localhost:8332/rest/blockpart/BLOCKHASH.bin?offset=0&size=SIZE'
    
    SIZE Time per request [ms]
    1 0.03
    10 0.03
    100 0.03
    1k 0.03
    10k 0.04
    100k 0.07
    1M 0.6

    So fetching a typical transaction is expected to take <40us on my machine. @ubbabeck could you please test it too?

  20. andrewtoth commented at 10:19 pm on October 25, 2025: contributor

    Compared your command fetching SIZE=788 and block 850000 with a 788 byte tx in that block 2d2dcc80195541dd44209dcfeb25393c8c8710f262360368a618b7ff3fa3f08c using getrawtransaction with txindex. I used hex encoding to make the comparison more fair.

    0ab -q -k -c 1 -n 1000000 'http://localhost:8332/rest/blockpart/00000000000000000002a0b5db2a7f8d9087464c2586b546be7bce8eb53b8187.hex?offset=0&size=788'
    

    I got Time per request: 0.037 [ms] (mean).

    Compared with RPC getrawtransaction:

    0ab -q -k -c 1 -n 1000000 -p data.json -A user:password 'http://localhost:8332/'
    

    where user:password are what is set in the config and data.json is

    0{"jsonrpc": "1.0", "id": "curltest", "method": "getrawtransaction", "params": ["2d2dcc80195541dd44209dcfeb25393c8c8710f262360368a618b7ff3fa3f08c"]}
    

    I got Time per request: 0.135 [ms] (mean). So quite a bit faster with the blockpart method.

    Using binary encoding Time per request: 0.032 [ms] (mean). Slightly faster.

  21. ubbabeck commented at 1:39 pm on October 26, 2025: none

    @ubbabeck could you please test it too?

    Sure here her my results for the same test using the same params and ApacheBench, Version 2.3

    SIZE Time per request [ms]
    1 0.046
    10 0.046
    100 0.050
    1k 0.050
    10k 0.054
    100k 0.113
    1m 0.803

    Fetching on average tx on my system is expected to take <0.05 ms

    For fetching 2d2dcc80195541dd44209dcfeb25393c8c8710f262360368a618b7ff3fa3f08c as Andrew did.

    Blockpart ms rpc getrawtransaction ms diff
    0.022 0.104 0.082
  22. in src/rest.cpp:514 in 301116e855 outdated
    497+            if (!part_size.has_value()) {
    498+                return RESTERR(req, HTTP_BAD_REQUEST, strprintf("Block part size is invalid: %s", *part_size_str));
    499+            }
    500+        }
    501+    } catch (const std::runtime_error& e) {
    502+        return RESTERR(req, HTTP_BAD_REQUEST, e.what());
    


    sedited commented at 12:35 pm on October 30, 2025:
    Can you add a test case for the error conditions? Maybe I am missing it, but corecheck’s coverage report also thinks they are not covered.

    romanz commented at 8:59 pm on October 30, 2025:

    Thanks - added in 301116e855…c30647c4d3.

    Also, tested an invalid RFC 3986 query:

     0$ curl -v 'http://localhost:8332/rest/blockpart/000000000000000000016d31a675b96acb4aacca6e4653039703b9fe03997c78.hex?%XY'
     1* Host localhost:8332 was resolved.
     2* IPv6: ::1
     3* IPv4: 127.0.0.1
     4*   Trying [::1]:8332...
     5* Connected to localhost (::1) port 8332
     6> GET /rest/blockpart/000000000000000000016d31a675b96acb4aacca6e4653039703b9fe03997c78.hex?%XY HTTP/1.1
     7> Host: localhost:8332
     8> User-Agent: curl/8.5.0
     9> Accept: */*
    10> 
    11< HTTP/1.1 400 Bad Request
    12< Content-Type: text/plain
    13< Date: Thu, 30 Oct 2025 20:58:12 GMT
    14< Content-Length: 69
    15< 
    16URI parsing failed, it likely contained RFC 3986 invalid characters
    17* Connection [#0](/bitcoin-bitcoin/0/) to host localhost left intact
    

    as well as querying a pruned block:

     0$ curl -v 'http://localhost:8332/rest/blockpart/0000000000002917ed80650c6174aac8dfc46f5fe36480aaef682ff6cd83c3ca.hex?offset=3&size=5'
     1* Host localhost:8332 was resolved.
     2* IPv6: ::1
     3* IPv4: 127.0.0.1
     4*   Trying [::1]:8332...
     5* Connected to localhost (::1) port 8332
     6> GET /rest/blockpart/0000000000002917ed80650c6174aac8dfc46f5fe36480aaef682ff6cd83c3ca.hex?offset=3&size=5 HTTP/1.1
     7> Host: localhost:8332
     8> User-Agent: curl/8.5.0
     9> Accept: */*
    10> 
    11< HTTP/1.1 404 Not Found
    12< Content-Type: text/plain
    13< Date: Thu, 30 Oct 2025 20:59:08 GMT
    14< Content-Length: 94
    15< 
    160000000000002917ed80650c6174aac8dfc46f5fe36480aaef682ff6cd83c3ca not available (pruned data)
    17* Connection [#0](/bitcoin-bitcoin/0/) to host localhost left intact
    

    maflcko commented at 9:33 am on November 26, 2025:
    I also don’t see the code path here. rest_block is called in the scope and line below, so why would pruning have any effect here?

    romanz commented at 10:33 pm on November 26, 2025:

    You’re right - I just wanted to mention that trying to fetch a pruned block still returns 404 Not Found error.

    For testing the catch (const std::runtime_error& e) clause, I have used an invalid URL encoding by passing the %XY parameter:

    0$ curl -v "http://localhost:8332/rest/blockpart/00.hex?%XY"
    

    hodlinator commented at 9:19 am on November 28, 2025:

    Maybe do something like this then?

     0--- a/test/functional/interface_rest.py
     1+++ b/test/functional/interface_rest.py
     2@@ -67,12 +67,15 @@ class RESTTest (BitcoinTestFramework):
     3             status: int = 200,
     4             ret_type: RetType = RetType.JSON,
     5             query_params: Optional[dict[str, typing.Any]] = None,
     6+            raw_query_params: Optional[str] = None,
     7             ) -> typing.Union[http.client.HTTPResponse, bytes, str, None]:
     8         rest_uri = '/rest' + uri
     9         if req_type in ReqType:
    10             rest_uri += f'.{req_type.name.lower()}'
    11         if query_params:
    12             rest_uri += f'?{urllib.parse.urlencode(query_params)}'
    13+        elif raw_query_params:
    14+            rest_uri += f'?{raw_query_params}'
    15 
    16         conn = http.client.HTTPConnection(self.url.hostname, self.url.port)
    17         self.log.debug(f'{http_method} {rest_uri} {body}')
    18@@ -482,6 +485,7 @@ class RESTTest (BitcoinTestFramework):
    19             _get_block_part(status=400, query_params={"offset": "x"})
    20             _get_block_part(status=400, query_params={"size": "y"})
    21             _get_block_part(status=400, query_params={"offset": "x", "size": "y"})
    22+            assert _get_block_part(status=400, raw_query_params="%XY").decode("utf-8").startswith("URI parsing failed")
    23 
    24             _get_block_part(status=404, query_params={"offset": 0, "size": 0})
    25             _get_block_part(status=404, query_params={"offset": len(block_bin), "size": 0})
    

    romanz commented at 9:23 am on November 29, 2025:
    Done in f50c0d9a3e..f33eafff0f (by allowing the caller to pass raw string query_params).
  23. sedited commented at 12:53 pm on October 30, 2025: contributor
    Made sure this also doesn’t slow down normal block retrieval. There is a bit of extra logic in the hot path for ReadRawBlock, but I think this should be fine.
  24. romanz force-pushed on Oct 30, 2025
  25. sedited approved
  26. sedited commented at 8:17 am on October 31, 2025: contributor
    ACK c30647c4d34c2941696729704854467b30657c43
  27. in src/rest.cpp:401 in c30647c4d3 outdated
    391@@ -392,7 +392,9 @@ static bool rest_spent_txouts(const std::any& context, HTTPRequest* req, const s
    392 static bool rest_block(const std::any& context,
    393                        HTTPRequest* req,
    394                        const std::string& uri_part,
    395-                       TxVerbosity tx_verbosity)
    396+                       std::optional<TxVerbosity> tx_verbosity,
    


    oren-z0 commented at 2:02 pm on November 1, 2025:
    Why make verbosity optional, and not keep as-is? Where is it called with a null value?

    romanz commented at 2:39 pm on November 1, 2025:

    Why make verbosity optional, and not keep as-is?

    It allows reusing this function also for the new REST endpoint (/blockpart/) which doesn’t support JSON response format.

    Where is it called with a null value?

    It is called by rest_block_part() here: https://github.com/bitcoin/bitcoin/blob/c30647c4d34c2941696729704854467b30657c43/src/rest.cpp#L480-L503


    oren-z0 commented at 10:25 pm on November 1, 2025:
    Oh, now I get it. So the question is why tx_verbosity was required in the first place even when the data-format was binary or hex (where it is ignored). Consider splitting this into two commits, one to make tx_verbosity optional, and another to add the new blockpart feature.

    romanz commented at 6:02 pm on November 2, 2025:
    Not sure if splitting this PR into two commits will make it clearer… Maybe I should add a comment at rest_block() definition?

    romanz commented at 6:14 pm on November 2, 2025:

    e.g.:

     0/**
     1 * This handler is used for multiple HTTP endpoints:
     2 * - `/block/` via `rest_block_extended()`
     3 * - `/block/notxdetails/` via `rest_block_notxdetails()`
     4 * - `/blockpart/` via `rest_block_part` (doesn't support JSON response so `tx_verbosity` is unset)
     5 */
     6static bool rest_block(const std::any& context,
     7                       HTTPRequest* req,
     8                       const std::string& uri_part,
     9                       std::optional<TxVerbosity> tx_verbosity,
    10                       std::optional<size_t> part_offset,
    11                       std::optional<size_t> part_size)
    

  28. andrewtoth commented at 7:26 pm on November 2, 2025: contributor
    @romanz do you have a proof-of-concept for the tx indexer that will utilize this? It would be good to make sure this will be sufficient and performant enough for your requirements.
  29. romanz commented at 9:41 pm on November 3, 2025: contributor

    do you have a proof-of-concept for the tx indexer that will utilize this?

    Yes - I am working on a PR to adapt bindex to use the new REST API endpoint. Assuming warm OS block cache, https://github.com/romanz/bindex-rs/pull/63#issuecomment-3448841256 achieves retrieval rate of ~10k txs/second when querying a “popular” signet address.

    Will test it on mainnet as well :)

  30. romanz commented at 6:33 am on November 4, 2025: contributor
    Tested mainnet query performance using https://mempool.space/address/1BitcoinEaterAddressDontSendf59kuE. Assuming warm OS block cache, https://github.com/romanz/bindex-rs/pull/63#issuecomment-3484098889 takes <1s to fetch 5190 txs, i.e. <0.2ms per tx :+1:
  31. romanz force-pushed on Nov 8, 2025
  32. romanz commented at 5:07 pm on November 8, 2025: contributor
    Added REST API documentation and release notes & added a comment for rest_block (following @oren-z0 review).
  33. romanz force-pushed on Nov 8, 2025
  34. DrahtBot added the label CI failed on Nov 8, 2025
  35. DrahtBot commented at 5:14 pm on November 8, 2025: contributor

    🚧 At least one of the CI tasks failed. Task lint: https://github.com/bitcoin/bitcoin/actions/runs/19196021604/job/54877130784 LLM reason (✨ experimental): Lint failure: release note snippets are in the wrong folder (doc_release_note_snippets detected; should be /doc/release-notes-.md).

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  36. DrahtBot removed the label CI failed on Nov 8, 2025
  37. romanz requested review from oren-z0 on Nov 8, 2025
  38. romanz commented at 12:15 pm on November 16, 2025: contributor
    Updated PR’s description with query performance and external offset index size for mainnet:
  39. in src/node/blockstorage.cpp:1087 in e14650967d
    1081@@ -1077,8 +1082,28 @@ bool BlockManager::ReadRawBlock(std::vector<std::byte>& block, const FlatFilePos
    1082             return false;
    1083         }
    1084 
    1085-        block.resize(blk_size); // Zeroing of memory is intentional here
    1086-        filein.read(block);
    1087+        size_t offset = 0;
    1088+        if (part_offset.has_value()) {
    1089+            if (*part_offset > blk_size) {
    


    andrewtoth commented at 3:38 pm on November 16, 2025:
    0            if (*part_offset > blk_size) [[unlikely]] {
    
  40. in src/node/blockstorage.cpp:1093 in e14650967d outdated
    1093+            offset = *part_offset;
    1094+        }
    1095+
    1096+        size_t size = blk_size - offset;
    1097+        if (part_size.has_value()) {
    1098+            if (*part_size > size || *part_size == 0) {
    


    andrewtoth commented at 3:39 pm on November 16, 2025:
    0            if (*part_size > size || *part_size == 0) [[unlikely]] {
    

    maflcko commented at 10:37 am on November 25, 2025:

    [[unlikely]]

    Will this actually help in a large function without a loop, where a single if is annotated?

    Generally, I have the impression that [[likely]], and [[unlikely]] should not be used. Possibly in rare cases, where there is a really hot loop, and all release compilers agree that the attribute will help.

    In other cases, if the attributes are used too liberal, it seems an accidental pessimization is plausible: It is unclear what the meaning of the attributes are, if nested scopes use both [[likely]] and [[unlikely]] at different scope levels, but the same path of execution, possibly hidden in macros.

    Ref:

    https://eel.is/c++draft/dcl.attr.likelihood#note-2

    [Note 2: Excessive usage of either of these attributes is liable to result in performance degradation. — end note]

    and https://blog.aaronballman.com/2020/08/dont-use-the-likely-or-unlikely-attributes/

    Otherwise, if there was a benefit, it would be easier to just annotate the LogError macro with unlikely and not every scope that calls it.

  41. in src/rest.cpp:463 in e14650967d outdated
    453@@ -446,10 +454,13 @@ static bool rest_block(const std::any& context,
    454     }
    455 
    456     case RESTResponseFormat::JSON: {
    457+        if (!tx_verbosity.has_value()) {
    


    andrewtoth commented at 3:40 pm on November 16, 2025:
    0        if (!tx_verbosity.has_value()) [[unlikely]] { 
    
  42. in src/rest.cpp:494 in e14650967d
    491+    std::optional<size_t> part_size{};
    492+    try {
    493+        const auto part_offset_str = req->GetQueryParameter("offset");
    494+        if (part_offset_str.has_value()) {
    495+            part_offset = ToIntegral<size_t>(*part_offset_str);
    496+            if (!part_offset.has_value()) {
    


    andrewtoth commented at 3:40 pm on November 16, 2025:
    0            if (!part_offset.has_value()) [[unlikely]] {
    
  43. in src/rest.cpp:509 in e14650967d outdated
    498+            }
    499+        }
    500+        const auto part_size_str = req->GetQueryParameter("size");
    501+        if (part_size_str.has_value()) {
    502+            part_size = ToIntegral<size_t>(*part_size_str);
    503+            if (!part_size.has_value()) {
    


    andrewtoth commented at 3:40 pm on November 16, 2025:
    0            if (!part_size.has_value()) [[unlikely]] {
    
  44. in src/rest.cpp:513 in e14650967d outdated
    502+            part_size = ToIntegral<size_t>(*part_size_str);
    503+            if (!part_size.has_value()) {
    504+                return RESTERR(req, HTTP_BAD_REQUEST, strprintf("Block part size is invalid: %s", *part_size_str));
    505+            }
    506+        }
    507+    } catch (const std::runtime_error& e) {
    


    andrewtoth commented at 3:42 pm on November 16, 2025:
    0    } catch (const std::runtime_error& e) [[unlikely]] {
    

    romanz commented at 5:08 pm on November 16, 2025:

    The above suggestion fails to compile on my machine:

    0src/rest.cpp: In function bool rest_block_part(const std::any&, HTTPRequest*, const std::string&):
    1src/rest.cpp:505:43: error: expected { before [ token
    2  505 |     } catch (const std::runtime_error& e) [[unlikely]] {
    3      |                                           ^
    4
    5$ gcc --version
    6gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
    

    andrewtoth commented at 5:13 pm on November 16, 2025:
    Ahh I didn’t test these suggestions before submitting. I wasn’t sure if we could have that annotation after a catch block. Please ignore.
  45. andrewtoth commented at 3:43 pm on November 16, 2025: contributor
    Since happy path performance is important here, maybe adding [[unlikely]] attributes for all error paths could yield a benefit.
  46. romanz commented at 5:28 pm on November 16, 2025: contributor

    Thanks @andrewtoth! I have applied the patch below on e14650967d, but it seems that it didn’t improve the following benchmark performance (reading the smallest possible amount of data from a block):

     0$ BLOCKHASH=00000000000000000001878540abd91b4dd591287ee58de73a163b698748f23b
     1$ ab -q -k -c 1 -n 100000 "http://localhost:8332/rest/blockpart/$BLOCK.bin?offset=0&size=1"
     2...
     3Server Software:        
     4Server Hostname:        localhost
     5Server Port:            8332
     6
     7Document Path:          /rest/blockpart/00000000000000000001878540abd91b4dd591287ee58de73a163b698748f23b.bin?offset=0&size=1
     8Document Length:        1 bytes
     9
    10Concurrency Level:      1
    11Time taken for tests:   4.395 seconds
    12Complete requests:      100000
    13Failed requests:        0
    14Keep-Alive requests:    100000
    15Total transferred:      10300000 bytes
    16HTML transferred:       100000 bytes
    17Requests per second:    22750.88 [#/sec] (mean)
    18Time per request:       0.044 [ms] (mean)
    19Time per request:       0.044 [ms] (mean, across all concurrent requests)
    20Transfer rate:          2288.42 [Kbytes/sec] received
    21
    22
    23$ ab -V
    24This is ApacheBench, Version 2.3 <$Revision: 1903618 $>
    
     0diff --git a/src/node/blockstorage.cpp b/src/node/blockstorage.cpp
     1index 47beac674c..6a3f39f737 100644
     2--- a/src/node/blockstorage.cpp
     3+++ b/src/node/blockstorage.cpp
     4@@ -1084,7 +1084,7 @@ bool BlockManager::ReadRawBlockPart(std::vector<std::byte>& data, const FlatFile
     5 
     6         size_t offset = 0;
     7         if (part_offset.has_value()) {
     8-            if (*part_offset > blk_size) {
     9+            if (*part_offset > blk_size) [[unlikely]] {
    10                 LogError("Read from block file failed: invalid offset: %u", *part_offset);
    11                 return false;
    12             }
    13@@ -1093,7 +1093,7 @@ bool BlockManager::ReadRawBlockPart(std::vector<std::byte>& data, const FlatFile
    14 
    15         size_t size = blk_size - offset;
    16         if (part_size.has_value()) {
    17-            if (*part_size > size || *part_size == 0) {
    18+            if (*part_size > size || *part_size == 0) [[unlikely]] {
    19                 LogError("Read from block file failed: invalid size: %u", *part_size);
    20                 return false;
    21             }
    22diff --git a/src/rest.cpp b/src/rest.cpp
    23index f3edc69d09..8e1c35b153 100644
    24--- a/src/rest.cpp
    25+++ b/src/rest.cpp
    26@@ -454,7 +454,7 @@ static bool rest_block(const std::any& context,
    27     }
    28 
    29     case RESTResponseFormat::JSON: {
    30-        if (!tx_verbosity.has_value()) {
    31+        if (!tx_verbosity.has_value()) [[unlikely]] {
    32             return RESTERR(req, HTTP_BAD_REQUEST, "JSON output is not supported for this request type");
    33         }
    34         CBlock block{};
    35@@ -491,14 +491,14 @@ static bool rest_block_part(const std::any& context, HTTPRequest* req, const std
    36         const auto part_offset_str = req->GetQueryParameter("offset");
    37         if (part_offset_str.has_value()) {
    38             part_offset = ToIntegral<size_t>(*part_offset_str);
    39-            if (!part_offset.has_value()) {
    40+            if (!part_offset.has_value()) [[unlikely]] {
    41                 return RESTERR(req, HTTP_BAD_REQUEST, strprintf("Block part offset is invalid: %s", *part_offset_str));
    42             }
    43         }
    44         const auto part_size_str = req->GetQueryParameter("size");
    45         if (part_size_str.has_value()) {
    46             part_size = ToIntegral<size_t>(*part_size_str);
    47-            if (!part_size.has_value()) {
    48+            if (!part_size.has_value()) [[unlikely]] {
    49                 return RESTERR(req, HTTP_BAD_REQUEST, strprintf("Block part size is invalid: %s", *part_size_str));
    50             }
    51         }
    

    In both cases (with the patch and without it), the benchmark takes ~4.4s (~44us/req = ~23k req/s).

  47. romanz requested review from andrewtoth on Nov 16, 2025
  48. in src/node/blockstorage.cpp:1105 in e14650967d
    1102+            size = *part_size;
    1103+        }
    1104+
    1105+
    1106+        data.resize(size); // Zeroing of memory is intentional here
    1107+        filein.seek(offset, SEEK_CUR);
    


    hodlinator commented at 1:19 pm on November 21, 2025:

    nit: Could avoid syscall in common case

    0        if (offset > 0) filein.seek(offset, SEEK_CUR);
    

    Other PR avoiding similar syscall which was leading to slowdowns on Windows: #30884 Alternatively one could perform the noop-check inside of AutoFile itself.


    romanz commented at 7:50 pm on November 21, 2025:
  49. in src/node/blockstorage.cpp:1052 in e14650967d
    1044@@ -1046,6 +1045,11 @@ bool BlockManager::ReadBlock(CBlock& block, const CBlockIndex& index) const
    1045 }
    1046 
    1047 bool BlockManager::ReadRawBlock(std::vector<std::byte>& block, const FlatFilePos& pos) const
    1048+{
    1049+    return ReadRawBlockPart(block, pos, std::nullopt, std::nullopt);
    1050+}
    1051+
    1052+bool BlockManager::ReadRawBlockPart(std::vector<std::byte>& data, const FlatFilePos& pos, std::optional<size_t> part_offset, std::optional<size_t> part_size) const
    


    hodlinator commented at 1:23 pm on November 21, 2025:

    No need to have std::optional for the offset, could just be sent in as 0 when reading whole blocks. Same for parameter types in rest.cpp.

    0bool BlockManager::ReadRawBlockPart(std::vector<std::byte>& data, const FlatFilePos& pos, size_t part_offset, std::optional<size_t> part_size) const
    
      0diff --git a/src/node/blockstorage.cpp b/src/node/blockstorage.cpp
      1index 47beac674c..7062735067 100644
      2--- a/src/node/blockstorage.cpp
      3+++ b/src/node/blockstorage.cpp
      4@@ -1046,10 +1046,10 @@ bool BlockManager::ReadBlock(CBlock& block, const CBlockIndex& index) const
      5 
      6 bool BlockManager::ReadRawBlock(std::vector<std::byte>& block, const FlatFilePos& pos) const
      7 {
      8-    return ReadRawBlockPart(block, pos, std::nullopt, std::nullopt);
      9+    return ReadRawBlockPart(block, pos, 0, std::nullopt);
     10 }
     11 
     12-bool BlockManager::ReadRawBlockPart(std::vector<std::byte>& data, const FlatFilePos& pos, std::optional<size_t> part_offset, std::optional<size_t> part_size) const
     13+bool BlockManager::ReadRawBlockPart(std::vector<std::byte>& data, const FlatFilePos& pos, size_t part_offset, std::optional<size_t> part_size) const
     14 {
     15     if (pos.nPos < STORAGE_HEADER_BYTES) {
     16         // If nPos is less than STORAGE_HEADER_BYTES, we can't read the header that precedes the block data
     17@@ -1082,16 +1082,12 @@ bool BlockManager::ReadRawBlockPart(std::vector<std::byte>& data, const FlatFile
     18             return false;
     19         }
     20 
     21-        size_t offset = 0;
     22-        if (part_offset.has_value()) {
     23-            if (*part_offset > blk_size) {
     24-                LogError("Read from block file failed: invalid offset: %u", *part_offset);
     25-                return false;
     26-            }
     27-            offset = *part_offset;
     28+        if (part_offset > blk_size) {
     29+            LogError("Read from block file failed: invalid offset: %u", part_offset);
     30+            return false;
     31         }
     32 
     33-        size_t size = blk_size - offset;
     34+        size_t size = blk_size - part_offset;
     35         if (part_size.has_value()) {
     36             if (*part_size > size || *part_size == 0) {
     37                 LogError("Read from block file failed: invalid size: %u", *part_size);
     38@@ -1102,7 +1098,7 @@ bool BlockManager::ReadRawBlockPart(std::vector<std::byte>& data, const FlatFile
     39 
     40 
     41         data.resize(size); // Zeroing of memory is intentional here
     42-        filein.seek(offset, SEEK_CUR);
     43+        if (part_offset > 0) filein.seek(part_offset, SEEK_CUR);
     44         filein.read(data);
     45     } catch (const std::exception& e) {
     46         LogError("Read from block file failed: %s for %s while reading raw block", e.what(), pos.ToString());
     47diff --git a/src/node/blockstorage.h b/src/node/blockstorage.h
     48index 61128942b3..9f33d826a1 100644
     49--- a/src/node/blockstorage.h
     50+++ b/src/node/blockstorage.h
     51@@ -414,7 +414,7 @@ public:
     52     bool ReadBlock(CBlock& block, const FlatFilePos& pos, const std::optional<uint256>& expected_hash) const;
     53     bool ReadBlock(CBlock& block, const CBlockIndex& index) const;
     54     bool ReadRawBlock(std::vector<std::byte>& block, const FlatFilePos& pos) const;
     55-    bool ReadRawBlockPart(std::vector<std::byte>& data, const FlatFilePos& pos, std::optional<size_t> part_offset, std::optional<size_t> part_size) const;
     56+    bool ReadRawBlockPart(std::vector<std::byte>& data, const FlatFilePos& pos, size_t part_offset, std::optional<size_t> part_size) const;
     57 
     58     bool ReadBlockUndo(CBlockUndo& blockundo, const CBlockIndex& index) const;
     59 
     60diff --git a/src/rest.cpp b/src/rest.cpp
     61index f3edc69d09..294337a24b 100644
     62--- a/src/rest.cpp
     63+++ b/src/rest.cpp
     64@@ -399,7 +399,7 @@ static bool rest_block(const std::any& context,
     65                        HTTPRequest* req,
     66                        const std::string& uri_part,
     67                        std::optional<TxVerbosity> tx_verbosity,
     68-                       std::optional<size_t> part_offset,
     69+                       size_t part_offset,
     70                        std::optional<size_t> part_size)
     71 {
     72     if (!CheckWarmup(req))
     73@@ -475,23 +475,25 @@ static bool rest_block(const std::any& context,
     74 
     75 static bool rest_block_extended(const std::any& context, HTTPRequest* req, const std::string& uri_part)
     76 {
     77-    return rest_block(context, req, uri_part, TxVerbosity::SHOW_DETAILS_AND_PREVOUT, /*part_offset=*/std::nullopt, /*part_size=*/std::nullopt);
     78+    return rest_block(context, req, uri_part, TxVerbosity::SHOW_DETAILS_AND_PREVOUT, /*part_offset=*/0, /*part_size=*/std::nullopt);
     79 }
     80 
     81 static bool rest_block_notxdetails(const std::any& context, HTTPRequest* req, const std::string& uri_part)
     82 {
     83-    return rest_block(context, req, uri_part, TxVerbosity::SHOW_TXID, /*part_offset=*/std::nullopt, /*part_size=*/std::nullopt);
     84+    return rest_block(context, req, uri_part, TxVerbosity::SHOW_TXID, /*part_offset=*/0, /*part_size=*/std::nullopt);
     85 }
     86 
     87 static bool rest_block_part(const std::any& context, HTTPRequest* req, const std::string& uri_part)
     88 {
     89-    std::optional<size_t> part_offset{};
     90+    size_t part_offset{0};
     91     std::optional<size_t> part_size{};
     92     try {
     93         const auto part_offset_str = req->GetQueryParameter("offset");
     94         if (part_offset_str.has_value()) {
     95-            part_offset = ToIntegral<size_t>(*part_offset_str);
     96-            if (!part_offset.has_value()) {
     97+            auto opt = ToIntegral<size_t>(*part_offset_str);
     98+            if (opt.has_value()) {
     99+                part_offset = opt.value();
    100+            } else {
    101                 return RESTERR(req, HTTP_BAD_REQUEST, strprintf("Block part offset is invalid: %s", *part_offset_str));
    102             }
    103         }
    

    romanz commented at 7:50 pm on November 21, 2025:
    Many thanks! e14650967d..8457a71a64
  50. in src/test/blockmanager_tests.cpp:152 in e14650967d outdated
    151+    std::vector<std::byte> block_part{};
    152+
    153+    BOOST_CHECK(blockman.ReadRawBlockPart(block_part, tip.GetBlockPos(), std::nullopt, std::nullopt));
    154+    BOOST_CHECK_EQUAL_COLLECTIONS(block_part.begin(), block_part.end(), block.begin(), block.end());
    155+
    156+    BOOST_CHECK(blockman.ReadRawBlockPart(block_part, tip.GetBlockPos(), 0, std::nullopt));
    


    hodlinator commented at 1:30 pm on November 21, 2025:
    nit: Could call block_part.clear() between each read.

    romanz commented at 7:49 pm on November 21, 2025:
    Sounds good - fixed in e14650967d..8457a71a64.
  51. hodlinator commented at 2:50 pm on November 21, 2025: contributor

    Concept ACK e14650967dc95da5c10e0d6183b6eac3e8243fe5

    #32541 contained some useful tricks that seem to be useful to shrink other index implementations. Thanks for letting it go in favor of this less invasive change though!

  52. romanz force-pushed on Nov 21, 2025
  53. romanz force-pushed on Nov 21, 2025
  54. DrahtBot added the label CI failed on Nov 21, 2025
  55. DrahtBot removed the label CI failed on Nov 21, 2025
  56. romanz requested review from hodlinator on Nov 21, 2025
  57. in src/test/blockmanager_tests.cpp:200 in 8457a71a64
    196+    block_part.clear();
    197+    BOOST_CHECK(blockman.ReadRawBlockPart(block_part, tip.GetBlockPos(), block.size() - 1, 1));
    198+    BOOST_CHECK_EQUAL_COLLECTIONS(block_part.begin(), block_part.end(), block.end() - 1, block.end());
    199+
    200+    BOOST_CHECK(!blockman.ReadRawBlockPart(block_part, tip.GetBlockPos(), 0, 0));
    201+    BOOST_CHECK(!blockman.ReadRawBlockPart(block_part, tip.GetBlockPos(), 0, 0));
    



    romanz commented at 7:26 am on November 22, 2025:
    Oops - thanks! Fixed in 8457a71a64..78d6402458 (also removed 2 duplicate test cases above).
  58. romanz force-pushed on Nov 22, 2025
  59. romanz requested review from hodlinator on Nov 22, 2025
  60. in doc/REST-interface.md:50 in 78d6402458
    46@@ -47,6 +47,11 @@ The HTTP request and response are both handled entirely in-memory.
    47 
    48 With the /notxdetails/ option JSON response will only contain the transaction hash instead of the complete transaction details. The option only affects the JSON response.
    49 
    50+- `GET /rest/blockpart/<BLOCK-HASH>.<bin|hex>?start=X&offset=Y`
    


    hodlinator commented at 8:51 am on November 24, 2025:

    Incorrect names:

    0- `GET /rest/blockpart/<BLOCK-HASH>.<bin|hex>?offset=X&size=Y`
    

    romanz commented at 8:48 pm on November 24, 2025:
    Thanks! Fixed in 78d6402458..c2088a7458.
  61. in doc/release-notes-33657.md:4 in 78d6402458
    0@@ -0,0 +1,5 @@
    1+New RPCs
    2+--------
    3+
    4+- A new REST API endpoint (`/rest/blockpart/BLOCKHASH`) has been introduced
    


    hodlinator commented at 8:53 am on November 24, 2025:

    nit: I think it’s worth including the parameters to make the reason behind adding this more understandable.

    0- A new REST API endpoint (`/rest/blockpart/BLOCKHASH?offset=X&size=Y`) has been introduced
    

    romanz commented at 8:48 pm on November 24, 2025:
    Sounds good -> 78d6402458..c2088a7458
  62. in src/rest.cpp:396 in 78d6402458
    388@@ -379,10 +389,18 @@ static bool rest_spent_txouts(const std::any& context, HTTPRequest* req, const s
    389     }
    390 }
    391 
    392+/**
    393+ * This handler is used by multiple HTTP endpoints:
    394+ * - `/block/` via `rest_block_extended()`
    395+ * - `/block/notxdetails/` via `rest_block_notxdetails()`
    396+ * - `/blockpart/` via `rest_block_part` (doesn't support JSON response, so `tx_verbosity` is unset)
    


    hodlinator commented at 8:54 am on November 24, 2025:

    nit: Consistency

    0 * - `/blockpart/` via `rest_block_part()` (doesn't support JSON response, so `tx_verbosity` is unset)
    

    romanz commented at 8:48 pm on November 24, 2025:
  63. sedited approved
  64. sedited commented at 12:50 pm on November 24, 2025: contributor
    Re-ACK 78d6402458d7d14533444d893c989f0534a3896f
  65. hodlinator commented at 2:43 pm on November 24, 2025: contributor

    Reviewed 78d6402458d7d14533444d893c989f0534a3896f

    Main issue blocking A-C-K is in doc/REST-interface.md (see inline comment).

    Left minor comment/question in related presentation: https://docs.google.com/presentation/d/1Zez-6DApKRu59kke4i_g9jwxQlaFKKRpOPdYFYsFXfA/edit?disco=AAABwl3JY0U (I was re-reading it to remind myself why switching from a tx hash to a block hash would be more efficient, re-realized that mapping block height=>block hash is quite natural).

    Experimented with further optimizations in https://github.com/hodlinator/bitcoin/tree/pr/33657_follow-ups, but as branch name suggests, they need more work and should probably be done as follow-ups. I’ve been unable to measure a stable & really notable performance-improvement yet.

  66. DrahtBot requested review from hodlinator on Nov 24, 2025
  67. romanz force-pushed on Nov 24, 2025
  68. romanz commented at 9:12 pm on November 24, 2025: contributor

    Thanks!

    Left minor comment/question in related presentation: https://docs.google.com/presentation/d/1Zez-6DApKRu59kke4i_g9jwxQlaFKKRpOPdYFYsFXfA/edit?disco=AAABwl3JY0U

    Good point - I’ve changed a bit the slide, and added a few comments.

    This should be similar to the electrs schema, but with txnum instead of block height.

  69. in src/node/blockstorage.cpp:1049 in c2088a7458
    1044@@ -1049,6 +1045,11 @@ bool BlockManager::ReadBlock(CBlock& block, const CBlockIndex& index) const
    1045 }
    1046 
    1047 bool BlockManager::ReadRawBlock(std::vector<std::byte>& block, const FlatFilePos& pos) const
    1048+{
    1049+    return ReadRawBlockPart(block, pos, 0, std::nullopt);
    


    maflcko commented at 10:40 am on November 25, 2025:
    nit: Could use named args consistently, like below? /*part_offset=*/0, /*part_size=*/std::nullopt);

    romanz commented at 5:25 pm on November 25, 2025:
    Thanks! Fixed in c2088a7458...ff5c3fecea.
  70. romanz force-pushed on Nov 25, 2025
  71. romanz requested review from maflcko on Nov 25, 2025
  72. in doc/REST-interface.md:53 in ff5c3fecea
    46@@ -47,6 +47,11 @@ The HTTP request and response are both handled entirely in-memory.
    47 
    48 With the /notxdetails/ option JSON response will only contain the transaction hash instead of the complete transaction details. The option only affects the JSON response.
    49 
    50+- `GET /rest/blockpart/<BLOCK-HASH>.<bin|hex>?offset=X&size=Y`
    51+
    52+Given a block hash: returns a block part, in binary or hex-encoded binary formats.
    53+Responds with 404 if the block doesn't exist.
    


    maflcko commented at 9:02 am on November 26, 2025:
    0Responds with 404 if the block or the byte range doesn't exist.
    

    nit


    romanz commented at 10:23 pm on November 26, 2025:
    Thanks! Fixed in ff5c3fecea..f50c0d9a3e.
  73. in src/node/blockstorage.cpp:1086 in ff5c3fecea outdated
    1081@@ -1081,8 +1082,24 @@ bool BlockManager::ReadRawBlock(std::vector<std::byte>& block, const FlatFilePos
    1082             return false;
    1083         }
    1084 
    1085-        block.resize(blk_size); // Zeroing of memory is intentional here
    1086-        filein.read(block);
    1087+        if (part_offset > blk_size) {
    1088+            LogError("Read from block file failed: invalid offset: %u", part_offset);
    


    maflcko commented at 9:20 am on November 26, 2025:

    the rest interface should be fine to expose to remote untrusted parties, no? So i am not sure about allowing them to unconditionally log confusing errors to the debug log.

    i think it would be better if the caller could decide if there really was a storage corruption and an error should be logged, or if the user just had a typo in the rest url.

    One way to achieve this, would be by returning a failure value.

    Unfortunately, util::Result does not yet allow this: #25665. So maybe for now, the return value could be a std::optional<ReadRawError>?

    0struct ReadRawError{
    1  std::string msg;
    2  bool invalid_range;
    3};
    

    romanz commented at 10:19 pm on November 26, 2025:

    i think it would be better if the caller could decide if there really was a storage corruption and an error should be logged, or if the user just had a typo in the rest url.

    Sounds good - WDYT about https://github.com/romanz/bitcoin/commit/9716ff4dd92b31fa4d97b909892f7fc8e3c82b9c?


    hodlinator commented at 8:50 am on November 28, 2025:

    Did a tweak commit on top of https://github.com/romanz/bitcoin/commit/9716ff4dd92b31fa4d97b909892f7fc8e3c82b9c: https://github.com/hodlinator/bitcoin/commit/0c75c322a404708d82d9bbf31b70f3b036c831ba

    • NotFound -> IO seems more accurate.
    • Add comments to explain why we don’t log for some errors.
    • Marginally nicer errors in rest.cpp (used contrived 0 for unset part_size). Feel less strongly about this.

    maflcko commented at 9:40 am on November 28, 2025:

    Did a tweak commit on top of romanz@9716ff4: hodlinator@0c75c32

    lgtm. Could also merge the two size/offset errors into one about just BadPartRange?

    Seems fine to push as a commit here. It can stay separate, or be squashed, up to you.


    romanz commented at 9:20 am on November 29, 2025:
    Thanks! Fixed in f50c0d9a3e..f33eafff0f.

    hodlinator commented at 4:06 pm on December 2, 2025:

    nit on my own suggestion if you re-touch :grimacing: - consistent ordering of offset/size:

    0-            return RESTERR(req, HTTP_NOT_FOUND, strprintf("Bad block part size/offset %d/%d for %s", part_size.value_or(0), part_offset, hashStr));
    1+            return RESTERR(req, HTTP_NOT_FOUND, strprintf("Bad block part offset/size %d/%d for %s", part_offset, part_size.value_or(0), hashStr));
    

  74. maflcko commented at 9:34 am on November 26, 2025: member

    lgtm, but i think the error handling can be improved?

    lgtm ff5c3feceaba496ff25efd8420cfcc32e0864bcc 🌽

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: lgtm ff5c3feceaba496ff25efd8420cfcc32e0864bcc 🌽
    3rYeG+6HnNmTxgjfUwQVqSoWF82BPiTgTmWMWaiT1YjY7jaNXF081kvStje78wygH2tE+kYnAmG1JxdSD79W+Dg==
    
  75. romanz force-pushed on Nov 26, 2025
  76. in src/test/blockmanager_tests.cpp:152 in f50c0d9a3e
    148+    std::vector<std::byte> block{};
    149+    BOOST_CHECK(blockman.ReadRawBlock(block, tip.GetBlockPos()));
    150+    BOOST_CHECK_GE(block.size(), 200);
    151+
    152+    std::vector<std::byte> block_part{};
    153+    BOOST_CHECK(blockman.ReadRawBlockPart(block_part, tip.GetBlockPos(), 0, std::nullopt));
    


    maflcko commented at 9:55 am on November 28, 2025:
    0    const auto read_tip_part{[&](auto part_offset, auto part_size){return blockman.ReadRawBlockPart(block_part, tip.GetBlockPos(), part_offset, part_size);}};
    1    BOOST_CHECK(read_tip_part(0, std::nullopt));
    

    style nit: Could reduce the verbosity in this line (and below) with a lambda.


    romanz commented at 9:20 am on November 29, 2025:
    Thanks - fixed in f50c0d9a3e..f33eafff0f.
  77. romanz force-pushed on Nov 28, 2025
  78. romanz force-pushed on Nov 28, 2025
  79. DrahtBot added the label CI failed on Nov 28, 2025
  80. DrahtBot commented at 8:54 pm on November 28, 2025: contributor

    🚧 At least one of the CI tasks failed. Task lint: https://github.com/bitcoin/bitcoin/actions/runs/19773241080/job/56661356997 LLM reason (✨ experimental): Lint check failed: unused import typing.Optional detected by ruff, causing non-zero exit.

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  81. romanz force-pushed on Nov 28, 2025
  82. romanz force-pushed on Nov 28, 2025
  83. DrahtBot removed the label CI failed on Nov 28, 2025
  84. romanz commented at 10:14 am on November 29, 2025: contributor
    Many thanks for the review! Pushed f50c0d9a3e..f33eafff0f.
  85. hodlinator approved
  86. hodlinator commented at 4:23 pm on December 2, 2025: contributor

    ACK f33eafff0f3c2698ee69ccef800bbc70a4eeae86

    Provides a REST API useful to external indexers, that shouldn’t be a source of egregious log spam (https://github.com/bitcoin/bitcoin/pull/33657#discussion_r2564184625).

    Measured performance before/after PR change to confirm no slowdown happened when fetching large blocks (URL: http://127.0.0.1:8332/rest/block/0000000000000000000515e202c8ae73c8155fc472422d7593af87aa74f2cf3d.bin).

  87. DrahtBot requested review from sedited on Dec 2, 2025
  88. rest: allow reading partial block data from storage
    It will allow fetching specific transactions using an external index,
    following https://github.com/bitcoin/bitcoin/pull/32541#issuecomment-3267485313.
    
    Co-authored-by: Hodlinator <172445034+hodlinator@users.noreply.github.com>
    6826375e85
  89. romanz force-pushed on Dec 2, 2025

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-12-02 18:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me