rest: allow reading partial block data from storage #33657

pull romanz wants to merge 3 commits into bitcoin:master from romanz:romanz/rest-blockdata changing 11 files +216 −48
  1. romanz commented at 11:08 am on October 19, 2025: contributor

    It allows fetching specific transactions using an external index, following #32541 (comment).

    Currently, electrs and other indexers map between an address/scripthash to the list of the relevant transactions.

    However, in order to fetch those transactions from bitcoind, electrs relies on reading the whole block and post-filtering for a specific transaction1. Other indexers use a txindex to fetch a transaction using its txid 234.

    The above approach has significant storage and CPU overhead, since the txid is a pseudo-random 32-byte value. Also, mainnet txindex takes ~60GB today.

    This PR is adding support for using the transaction’s position within its block to be able to fetch it directly using REST API, using the following HTTP request:

    0GET /rest/blockpart/BLOCKHASH.bin?offset=OFFSET&size=SIZE
    
  2. DrahtBot added the label RPC/REST/ZMQ on Oct 19, 2025
  3. DrahtBot commented at 11:08 am on October 19, 2025: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33657.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK maflcko, hodlinator, l0rinc
    Concept ACK optout21
    Stale ACK ubbabeck, sedited

    If your review is incorrectly listed, please copy-paste <!–meta-tag:bot-skip–> into the comment that the bot should ignore.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #33727 (zmq: Log bind error at Error level, abort startup on init error by isrod)
    • #31860 (init: Take lock on blocks directory in BlockManager ctor by sedited)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  4. romanz force-pushed on Oct 19, 2025
  5. andrewtoth commented at 9:30 pm on October 19, 2025: contributor

    Some of the PR description is reused from the previous PR, but is now stale.

    This PR is adding support for using the transaction’s position within its block to be able to fetch it directly using REST API, using the following HTTP request (to fetch the N-th transaction from BLOCKHASH):

    This does not support fetching an N-th transaction, only a slice of a block.

    If binary response format is used, the transaction data will be read directly from the storage and sent back to the client, without any deserialization overhead.

    There is no way to use this in non-binary form (except hex I suppose).

  6. in src/node/blockstorage.cpp:1096 in 787d6df48a
    1093+            offset = *part_offset;
    1094+        }
    1095+
    1096+        size_t size = blk_size - offset;
    1097+        if (part_size.has_value()) {
    1098+            if (*part_size > size) {
    


    andrewtoth commented at 11:04 pm on October 19, 2025:
    Do we want to support zero *part_size? Should probably return false too?

    romanz commented at 8:21 pm on October 20, 2025:
    Sounds good - fixed in 787d6df48a..c4bf7a77c4.

    andrewtoth commented at 10:22 pm on October 20, 2025:
    We should probably cover this as well as other cases of ReadRawBlockPart in blockmanager_tests.cpp.

    romanz commented at 8:26 pm on October 21, 2025:
  7. in src/node/blockstorage.h:417 in 787d6df48a
    413@@ -414,6 +414,7 @@ class BlockManager
    414     bool ReadBlock(CBlock& block, const FlatFilePos& pos, const std::optional<uint256>& expected_hash) const;
    415     bool ReadBlock(CBlock& block, const CBlockIndex& index) const;
    416     bool ReadRawBlock(std::vector<std::byte>& block, const FlatFilePos& pos) const;
    417+    bool ReadRawBlockPart(std::vector<std::byte>& block, const FlatFilePos& pos, std::optional<size_t> part_offset, std::optional<size_t> part_size) const;
    


    optout21 commented at 11:26 am on October 20, 2025:
    nit: Declaration differs from definition, block vs data parameter name

    romanz commented at 8:21 pm on October 20, 2025:
    Thanks - fixed in 787d6df48a..c4bf7a77c4.
  8. in src/rest.cpp:481 in 787d6df48a outdated
    495@@ -472,6 +496,11 @@ static bool rest_block_notxdetails(const std::any& context, HTTPRequest* req, co
    496     return rest_block(context, req, uri_part, TxVerbosity::SHOW_TXID);
    497 }
    498 
    499+static bool rest_block_part(const std::any& context, HTTPRequest* req, const std::string& uri_part)
    


    optout21 commented at 11:31 am on October 20, 2025:
    This way both block and blockpart api’s take the same paramaters, including "offset" and "size", right? Is that intended?

    romanz commented at 8:14 pm on October 20, 2025:
    Right - I have used the same implementation for both endpoints for simplicity and to avoid code duplication. I can add an additional boolean argument to enable offset and size parameter handling for the /rest/blockpart endpoint (and disable it for /rest/block endpoint). WDYT?

    andrewtoth commented at 9:28 pm on October 20, 2025:

    If I understand correctly, this would allow passing offset or size to /rest/block along with .json format specification. Normally this would fail to deserialize, but if very unlucky it could deserialize to what looks like a block and return? That would be very bad. Similarly if not using .json, this would return incorrect data that a user might not expect (for instance if they switched rest endpoints but forgot to remove parameters).

    I would recommend disabling these parameters if not calling /rest/blockpart.


    romanz commented at 6:01 am on October 21, 2025:
    Agree - fixed in c4bf7a77c4..c9bfd298be (offset and size are parsed only for /rest/blockpart endpoint).
  9. romanz force-pushed on Oct 20, 2025
  10. romanz commented at 8:23 pm on October 20, 2025: contributor

    Some of the PR description is reused from the previous PR, but is now stale.

    Good catch, thanks! Updated PR description: #33657#issue-3529967048

  11. romanz force-pushed on Oct 21, 2025
  12. optout21 commented at 9:48 am on October 21, 2025: contributor
    Concept ACK c9bfd298be422de7e989fe244fb4281c507068a3
  13. romanz force-pushed on Oct 21, 2025
  14. romanz requested review from andrewtoth on Oct 22, 2025
  15. ubbabeck commented at 7:33 pm on October 22, 2025: none

    tACK 301116e855

    What was done:

    • functional test.
    • unit test
    • tested the new rest endpoint rest/blockpart/<blockhash>.bin manually with curl with different offsets and sizes.

    If there are any additional test you’d like me to help with let me know. I also did a perf dump, but I need some guidelines on what to measure/ grep for.

  16. DrahtBot requested review from optout21 on Oct 22, 2025
  17. romanz commented at 9:44 pm on October 22, 2025: contributor

    Many thanks @ubbabeck!

    If there are any additional test you’d like me to help with let me know.

    It would be interesting to compare the performance for single transaction fetching, similar to how it was done in #32541 (comment), comparing txindex-based and locationindex-based queries.

  18. ubbabeck commented at 5:12 pm on October 23, 2025: none

    It would be interesting to compare the performance for single transaction fetching, similar to how it was done in #32541 (comment), comparing txindex-based and locationindex-based queries.

    Here are the results from testing. I used hey to benchmark and your rust getrawtransaction.

    Was this somewhat you were looking for? If not let me know and I’ll redo them correctly.

    Maybe not relevant to test, but fetching the whole block using blockpart.

    hey -c 1 -n 100000 http://localhost:8332/rest/blockpart/000000000000000000017bfd05b5fa367a424c4a565a4baf7950d9e8605df8ec.bin

     0Summary:
     1  Total:	114.4020 secs
     2  Slowest:	0.0048 secs
     3  Fastest:	0.0009 secs
     4  Average:	0.0011 secs
     5  Requests/sec:	874.1103
     6
     7  Total data:	146938500000 bytes
     8  Size/request:	1469385 bytes
     9
    10Response time histogram:
    11  0.001 [1]	|
    12  0.001 [93859]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
    13  0.002 [4835]	|■■
    14  0.002 [1052]	|
    15  0.002 [165]	|
    16  0.003 [65]	|
    17  0.003 [20]	|
    18  0.004 [2]	|
    19  0.004 [0]	|
    20  0.004 [0]	|
    21  0.005 [1]	|
    22
    23
    24Latency distribution:
    25  10% in 0.0010 secs
    26  25% in 0.0011 secs
    27  50% in 0.0011 secs
    28  75% in 0.0012 secs
    29  90% in 0.0013 secs
    30  95% in 0.0013 secs
    31  99% in 0.0018 secs
    32
    33Details (average, fastest, slowest):
    34  DNS+dialup:	0.0000 secs, 0.0009 secs, 0.0048 secs
    35  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0006 secs
    36  req write:	0.0000 secs, 0.0000 secs, 0.0003 secs
    37  resp wait:	0.0004 secs, 0.0002 secs, 0.0034 secs
    38  resp read:	0.0007 secs, 0.0002 secs, 0.0030 secs
    39
    40Status code distribution:
    41  [200]	100000 responses
    

    Using a random offset and size of 310 bytes to simulate fetching a transaction.

    hey -c 16 -n 100000 http://localhost:8332/rest/blockpart/000000000000000000017bfd05b5fa367a424c4a565a4baf7950d9e8605df8ec.bin\?offset\=4000\&size\=310

     0Summary:
     1  Total:	1.7338 secs
     2  Slowest:	0.0023 secs
     3  Fastest:	0.0001 secs
     4  Average:	0.0003 secs
     5  Requests/sec:	57675.3814
     6  
     7  Total data:	31000000 bytes
     8  Size/request:	310 bytes
     9
    10Response time histogram:
    11  0.000 [1]	|
    12  0.000 [57797]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
    13  0.000 [41217]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
    14  0.001 [824]	|■
    15  0.001 [76]	|
    16  0.001 [10]	|
    17  0.001 [15]	|
    18  0.002 [20]	|
    19  0.002 [24]	|
    20  0.002 [10]	|
    21  0.002 [6]	|
    22
    23
    24Latency distribution:
    25  10% in 0.0002 secs
    26  25% in 0.0003 secs
    27  50% in 0.0003 secs
    28  75% in 0.0003 secs
    29  90% in 0.0003 secs
    30  95% in 0.0003 secs
    31  99% in 0.0005 secs
    32
    33Details (average, fastest, slowest):
    34  DNS+dialup:	0.0000 secs, 0.0001 secs, 0.0023 secs
    35  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0015 secs
    36  req write:	0.0000 secs, 0.0000 secs, 0.0008 secs
    37  resp wait:	0.0003 secs, 0.0000 secs, 0.0018 secs
    38  resp read:	0.0000 secs, 0.0000 secs, 0.0007 secs
    39
    40Status code distribution:
    41  [200]	100000 responses
    

    One runner

    hey -c 1 -n 100000 http://localhost:8332/rest/blockpart/000000000000000000017bfd05b5fa367a424c4a565a4baf7950d9e8605df8ec.bin\?offset\=4000\&size\=310

     0Summary:
     1  Total:	1.7338 secs
     2  Slowest:	0.0023 secs
     3  Fastest:	0.0001 secs
     4  Average:	0.0003 secs
     5  Requests/sec:	57675.3814
     6  
     7  Total data:	31000000 bytes
     8  Size/request:	310 bytes
     9
    10Response time histogram:
    11  0.000 [1]	|
    12  0.000 [57797]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
    13  0.000 [41217]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
    14  0.001 [824]	|■
    15  0.001 [76]	|
    16  0.001 [10]	|
    17  0.001 [15]	|
    18  0.002 [20]	|
    19  0.002 [24]	|
    20  0.002 [10]	|
    21  0.002 [6]	|
    22
    23
    24Latency distribution:
    25  10% in 0.0002 secs
    26  25% in 0.0003 secs
    27  50% in 0.0003 secs
    28  75% in 0.0003 secs
    29  90% in 0.0003 secs
    30  95% in 0.0003 secs
    31  99% in 0.0005 secs
    32
    33Details (average, fastest, slowest):
    34  DNS+dialup:	0.0000 secs, 0.0001 secs, 0.0023 secs
    35  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0015 secs
    36  req write:	0.0000 secs, 0.0000 secs, 0.0008 secs
    37  resp wait:	0.0003 secs, 0.0000 secs, 0.0018 secs
    38  resp read:	0.0000 secs, 0.0000 secs, 0.0007 secs
    39
    40Status code distribution:
    41  [200]	100000 responses
    

    Rust getrawtransaction benchmark

    0time cargo run --release -- 4137d0dbad434d68a4f52b7bebcba91ddac3f7f5c92b84130432bd6b5e2ea57a 0000000000000000000083a0cff38278aae196d6d923a7e8ee7e5a0e371226fe
    1   Compiling bench-getrawtx v0.1.0 (review/bench-getrawtx)
    2    Finished `release` profile [optimized] target(s) in 0.29s
    3     Running `target/x86_64-unknown-linux-gnu/release/bench-getrawtx 4137d0dbad434d68a4f52b7bebcba91ddac3f7f5c92b84130432bd6b5e2ea57a 0000000000000000000083a0cff38278aae196d6d923a7e8ee7e5a0e371226fe`
    4iterations = 100000
    5average RPC duration = 130.319µs
    6cargo run --release --    1.29s user 1.09s system 17% cpu 13.342 total
    
  19. romanz commented at 9:43 pm on October 25, 2025: contributor

    Tested the performance of the new REST endpoint using ab on mainnet with various SIZE parameters:

    0$ BLOCKHASH=00000000000000000001eae3683a5350b67ddb17d9c7b6c8010ab5b36ccbaa09
    1$ ab -q -k -c 1 -n 100000 'http://localhost:8332/rest/blockpart/BLOCKHASH.bin?offset=0&size=SIZE'
    
    SIZE Time per request [ms]
    1 0.03
    10 0.03
    100 0.03
    1k 0.03
    10k 0.04
    100k 0.07
    1M 0.6

    So fetching a typical transaction is expected to take <40us on my machine. @ubbabeck could you please test it too?

  20. andrewtoth commented at 10:19 pm on October 25, 2025: contributor

    Compared your command fetching SIZE=788 and block 850000 with a 788 byte tx in that block 2d2dcc80195541dd44209dcfeb25393c8c8710f262360368a618b7ff3fa3f08c using getrawtransaction with txindex. I used hex encoding to make the comparison more fair.

    0ab -q -k -c 1 -n 1000000 'http://localhost:8332/rest/blockpart/00000000000000000002a0b5db2a7f8d9087464c2586b546be7bce8eb53b8187.hex?offset=0&size=788'
    

    I got Time per request: 0.037 [ms] (mean).

    Compared with RPC getrawtransaction:

    0ab -q -k -c 1 -n 1000000 -p data.json -A user:password 'http://localhost:8332/'
    

    where user:password are what is set in the config and data.json is

    0{"jsonrpc": "1.0", "id": "curltest", "method": "getrawtransaction", "params": ["2d2dcc80195541dd44209dcfeb25393c8c8710f262360368a618b7ff3fa3f08c"]}
    

    I got Time per request: 0.135 [ms] (mean). So quite a bit faster with the blockpart method.

    Using binary encoding Time per request: 0.032 [ms] (mean). Slightly faster.

  21. ubbabeck commented at 1:39 pm on October 26, 2025: none

    @ubbabeck could you please test it too?

    Sure here her my results for the same test using the same params and ApacheBench, Version 2.3

    SIZE Time per request [ms]
    1 0.046
    10 0.046
    100 0.050
    1k 0.050
    10k 0.054
    100k 0.113
    1m 0.803

    Fetching on average tx on my system is expected to take <0.05 ms

    For fetching 2d2dcc80195541dd44209dcfeb25393c8c8710f262360368a618b7ff3fa3f08c as Andrew did.

    Blockpart ms rpc getrawtransaction ms diff
    0.022 0.104 0.082
  22. in src/rest.cpp:517 in 301116e855 outdated
    497+            if (!part_size.has_value()) {
    498+                return RESTERR(req, HTTP_BAD_REQUEST, strprintf("Block part size is invalid: %s", *part_size_str));
    499+            }
    500+        }
    501+    } catch (const std::runtime_error& e) {
    502+        return RESTERR(req, HTTP_BAD_REQUEST, e.what());
    


    sedited commented at 12:35 pm on October 30, 2025:
    Can you add a test case for the error conditions? Maybe I am missing it, but corecheck’s coverage report also thinks they are not covered.

    romanz commented at 8:59 pm on October 30, 2025:

    Thanks - added in 301116e855…c30647c4d3.

    Also, tested an invalid RFC 3986 query:

     0$ curl -v 'http://localhost:8332/rest/blockpart/000000000000000000016d31a675b96acb4aacca6e4653039703b9fe03997c78.hex?%XY'
     1* Host localhost:8332 was resolved.
     2* IPv6: ::1
     3* IPv4: 127.0.0.1
     4*   Trying [::1]:8332...
     5* Connected to localhost (::1) port 8332
     6> GET /rest/blockpart/000000000000000000016d31a675b96acb4aacca6e4653039703b9fe03997c78.hex?%XY HTTP/1.1
     7> Host: localhost:8332
     8> User-Agent: curl/8.5.0
     9> Accept: */*
    10> 
    11< HTTP/1.1 400 Bad Request
    12< Content-Type: text/plain
    13< Date: Thu, 30 Oct 2025 20:58:12 GMT
    14< Content-Length: 69
    15< 
    16URI parsing failed, it likely contained RFC 3986 invalid characters
    17* Connection [#0](/bitcoin-bitcoin/0/) to host localhost left intact
    

    as well as querying a pruned block:

     0$ curl -v 'http://localhost:8332/rest/blockpart/0000000000002917ed80650c6174aac8dfc46f5fe36480aaef682ff6cd83c3ca.hex?offset=3&size=5'
     1* Host localhost:8332 was resolved.
     2* IPv6: ::1
     3* IPv4: 127.0.0.1
     4*   Trying [::1]:8332...
     5* Connected to localhost (::1) port 8332
     6> GET /rest/blockpart/0000000000002917ed80650c6174aac8dfc46f5fe36480aaef682ff6cd83c3ca.hex?offset=3&size=5 HTTP/1.1
     7> Host: localhost:8332
     8> User-Agent: curl/8.5.0
     9> Accept: */*
    10> 
    11< HTTP/1.1 404 Not Found
    12< Content-Type: text/plain
    13< Date: Thu, 30 Oct 2025 20:59:08 GMT
    14< Content-Length: 94
    15< 
    160000000000002917ed80650c6174aac8dfc46f5fe36480aaef682ff6cd83c3ca not available (pruned data)
    17* Connection [#0](/bitcoin-bitcoin/0/) to host localhost left intact
    

    maflcko commented at 9:33 am on November 26, 2025:
    I also don’t see the code path here. rest_block is called in the scope and line below, so why would pruning have any effect here?

    romanz commented at 10:33 pm on November 26, 2025:

    You’re right - I just wanted to mention that trying to fetch a pruned block still returns 404 Not Found error.

    For testing the catch (const std::runtime_error& e) clause, I have used an invalid URL encoding by passing the %XY parameter:

    0$ curl -v "http://localhost:8332/rest/blockpart/00.hex?%XY"
    

    hodlinator commented at 9:19 am on November 28, 2025:

    Maybe do something like this then?

     0--- a/test/functional/interface_rest.py
     1+++ b/test/functional/interface_rest.py
     2@@ -67,12 +67,15 @@ class RESTTest (BitcoinTestFramework):
     3             status: int = 200,
     4             ret_type: RetType = RetType.JSON,
     5             query_params: Optional[dict[str, typing.Any]] = None,
     6+            raw_query_params: Optional[str] = None,
     7             ) -> typing.Union[http.client.HTTPResponse, bytes, str, None]:
     8         rest_uri = '/rest' + uri
     9         if req_type in ReqType:
    10             rest_uri += f'.{req_type.name.lower()}'
    11         if query_params:
    12             rest_uri += f'?{urllib.parse.urlencode(query_params)}'
    13+        elif raw_query_params:
    14+            rest_uri += f'?{raw_query_params}'
    15 
    16         conn = http.client.HTTPConnection(self.url.hostname, self.url.port)
    17         self.log.debug(f'{http_method} {rest_uri} {body}')
    18@@ -482,6 +485,7 @@ class RESTTest (BitcoinTestFramework):
    19             _get_block_part(status=400, query_params={"offset": "x"})
    20             _get_block_part(status=400, query_params={"size": "y"})
    21             _get_block_part(status=400, query_params={"offset": "x", "size": "y"})
    22+            assert _get_block_part(status=400, raw_query_params="%XY").decode("utf-8").startswith("URI parsing failed")
    23 
    24             _get_block_part(status=404, query_params={"offset": 0, "size": 0})
    25             _get_block_part(status=404, query_params={"offset": len(block_bin), "size": 0})
    

    romanz commented at 9:23 am on November 29, 2025:
    Done in f50c0d9a3e..f33eafff0f (by allowing the caller to pass raw string query_params).
  23. sedited commented at 12:53 pm on October 30, 2025: contributor
    Made sure this also doesn’t slow down normal block retrieval. There is a bit of extra logic in the hot path for ReadRawBlock, but I think this should be fine.
  24. romanz force-pushed on Oct 30, 2025
  25. sedited approved
  26. sedited commented at 8:17 am on October 31, 2025: contributor
    ACK c30647c4d34c2941696729704854467b30657c43
  27. in src/rest.cpp:391 in c30647c4d3 outdated
    391@@ -392,7 +392,9 @@ static bool rest_spent_txouts(const std::any& context, HTTPRequest* req, const s
    392 static bool rest_block(const std::any& context,
    393                        HTTPRequest* req,
    394                        const std::string& uri_part,
    395-                       TxVerbosity tx_verbosity)
    396+                       std::optional<TxVerbosity> tx_verbosity,
    


    oren-z0 commented at 2:02 pm on November 1, 2025:
    Why make verbosity optional, and not keep as-is? Where is it called with a null value?

    romanz commented at 2:39 pm on November 1, 2025:

    Why make verbosity optional, and not keep as-is?

    It allows reusing this function also for the new REST endpoint (/blockpart/) which doesn’t support JSON response format.

    Where is it called with a null value?

    It is called by rest_block_part() here: https://github.com/bitcoin/bitcoin/blob/c30647c4d34c2941696729704854467b30657c43/src/rest.cpp#L480-L503


    oren-z0 commented at 10:25 pm on November 1, 2025:
    Oh, now I get it. So the question is why tx_verbosity was required in the first place even when the data-format was binary or hex (where it is ignored). Consider splitting this into two commits, one to make tx_verbosity optional, and another to add the new blockpart feature.

    romanz commented at 6:02 pm on November 2, 2025:
    Not sure if splitting this PR into two commits will make it clearer… Maybe I should add a comment at rest_block() definition?

    romanz commented at 6:14 pm on November 2, 2025:

    e.g.:

     0/**
     1 * This handler is used for multiple HTTP endpoints:
     2 * - `/block/` via `rest_block_extended()`
     3 * - `/block/notxdetails/` via `rest_block_notxdetails()`
     4 * - `/blockpart/` via `rest_block_part` (doesn't support JSON response so `tx_verbosity` is unset)
     5 */
     6static bool rest_block(const std::any& context,
     7                       HTTPRequest* req,
     8                       const std::string& uri_part,
     9                       std::optional<TxVerbosity> tx_verbosity,
    10                       std::optional<size_t> part_offset,
    11                       std::optional<size_t> part_size)
    

  28. andrewtoth commented at 7:26 pm on November 2, 2025: contributor
    @romanz do you have a proof-of-concept for the tx indexer that will utilize this? It would be good to make sure this will be sufficient and performant enough for your requirements.
  29. romanz commented at 9:41 pm on November 3, 2025: contributor

    do you have a proof-of-concept for the tx indexer that will utilize this?

    Yes - I am working on a PR to adapt bindex to use the new REST API endpoint. Assuming warm OS block cache, https://github.com/romanz/bindex-rs/pull/63#issuecomment-3448841256 achieves retrieval rate of ~10k txs/second when querying a “popular” signet address.

    Will test it on mainnet as well :)

  30. romanz commented at 6:33 am on November 4, 2025: contributor
    Tested mainnet query performance using https://mempool.space/address/1BitcoinEaterAddressDontSendf59kuE. Assuming warm OS block cache, https://github.com/romanz/bindex-rs/pull/63#issuecomment-3484098889 takes <1s to fetch 5190 txs, i.e. <0.2ms per tx :+1:
  31. romanz force-pushed on Nov 8, 2025
  32. romanz commented at 5:07 pm on November 8, 2025: contributor
    Added REST API documentation and release notes & added a comment for rest_block (following @oren-z0 review).
  33. romanz force-pushed on Nov 8, 2025
  34. DrahtBot added the label CI failed on Nov 8, 2025
  35. DrahtBot commented at 5:14 pm on November 8, 2025: contributor

    🚧 At least one of the CI tasks failed. Task lint: https://github.com/bitcoin/bitcoin/actions/runs/19196021604/job/54877130784 LLM reason (✨ experimental): Lint failure: release note snippets are in the wrong folder (doc_release_note_snippets detected; should be /doc/release-notes-.md).

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  36. DrahtBot removed the label CI failed on Nov 8, 2025
  37. romanz requested review from oren-z0 on Nov 8, 2025
  38. romanz commented at 12:15 pm on November 16, 2025: contributor
    Updated PR’s description with query performance and external offset index size for mainnet:
  39. in src/node/blockstorage.cpp:1087 in e14650967d
    1081@@ -1077,8 +1082,28 @@ bool BlockManager::ReadRawBlock(std::vector<std::byte>& block, const FlatFilePos
    1082             return false;
    1083         }
    1084 
    1085-        block.resize(blk_size); // Zeroing of memory is intentional here
    1086-        filein.read(block);
    1087+        size_t offset = 0;
    1088+        if (part_offset.has_value()) {
    1089+            if (*part_offset > blk_size) {
    


    andrewtoth commented at 3:38 pm on November 16, 2025:
    0            if (*part_offset > blk_size) [[unlikely]] {
    
  40. in src/node/blockstorage.cpp:1098 in e14650967d outdated
    1093+            offset = *part_offset;
    1094+        }
    1095+
    1096+        size_t size = blk_size - offset;
    1097+        if (part_size.has_value()) {
    1098+            if (*part_size > size || *part_size == 0) {
    


    andrewtoth commented at 3:39 pm on November 16, 2025:
    0            if (*part_size > size || *part_size == 0) [[unlikely]] {
    

    maflcko commented at 10:37 am on November 25, 2025:

    [[unlikely]]

    Will this actually help in a large function without a loop, where a single if is annotated?

    Generally, I have the impression that [[likely]], and [[unlikely]] should not be used. Possibly in rare cases, where there is a really hot loop, and all release compilers agree that the attribute will help.

    In other cases, if the attributes are used too liberal, it seems an accidental pessimization is plausible: It is unclear what the meaning of the attributes are, if nested scopes use both [[likely]] and [[unlikely]] at different scope levels, but the same path of execution, possibly hidden in macros.

    Ref:

    https://eel.is/c++draft/dcl.attr.likelihood#note-2

    [Note 2: Excessive usage of either of these attributes is liable to result in performance degradation. — end note]

    and https://blog.aaronballman.com/2020/08/dont-use-the-likely-or-unlikely-attributes/

    Otherwise, if there was a benefit, it would be easier to just annotate the LogError macro with unlikely and not every scope that calls it.

  41. in src/rest.cpp:466 in e14650967d outdated
    453@@ -446,10 +454,13 @@ static bool rest_block(const std::any& context,
    454     }
    455 
    456     case RESTResponseFormat::JSON: {
    457+        if (!tx_verbosity.has_value()) {
    


    andrewtoth commented at 3:40 pm on November 16, 2025:
    0        if (!tx_verbosity.has_value()) [[unlikely]] { 
    
  42. in src/rest.cpp:494 in e14650967d
    491+    std::optional<size_t> part_size{};
    492+    try {
    493+        const auto part_offset_str = req->GetQueryParameter("offset");
    494+        if (part_offset_str.has_value()) {
    495+            part_offset = ToIntegral<size_t>(*part_offset_str);
    496+            if (!part_offset.has_value()) {
    


    andrewtoth commented at 3:40 pm on November 16, 2025:
    0            if (!part_offset.has_value()) [[unlikely]] {
    
  43. in src/rest.cpp:512 in e14650967d outdated
    498+            }
    499+        }
    500+        const auto part_size_str = req->GetQueryParameter("size");
    501+        if (part_size_str.has_value()) {
    502+            part_size = ToIntegral<size_t>(*part_size_str);
    503+            if (!part_size.has_value()) {
    


    andrewtoth commented at 3:40 pm on November 16, 2025:
    0            if (!part_size.has_value()) [[unlikely]] {
    
  44. in src/rest.cpp:495 in e14650967d outdated
    502+            part_size = ToIntegral<size_t>(*part_size_str);
    503+            if (!part_size.has_value()) {
    504+                return RESTERR(req, HTTP_BAD_REQUEST, strprintf("Block part size is invalid: %s", *part_size_str));
    505+            }
    506+        }
    507+    } catch (const std::runtime_error& e) {
    


    andrewtoth commented at 3:42 pm on November 16, 2025:
    0    } catch (const std::runtime_error& e) [[unlikely]] {
    

    romanz commented at 5:08 pm on November 16, 2025:

    The above suggestion fails to compile on my machine:

    0src/rest.cpp: In function bool rest_block_part(const std::any&, HTTPRequest*, const std::string&):
    1src/rest.cpp:505:43: error: expected { before [ token
    2  505 |     } catch (const std::runtime_error& e) [[unlikely]] {
    3      |                                           ^
    4
    5$ gcc --version
    6gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
    

    andrewtoth commented at 5:13 pm on November 16, 2025:
    Ahh I didn’t test these suggestions before submitting. I wasn’t sure if we could have that annotation after a catch block. Please ignore.
  45. andrewtoth commented at 3:43 pm on November 16, 2025: contributor
    Since happy path performance is important here, maybe adding [[unlikely]] attributes for all error paths could yield a benefit.
  46. romanz commented at 5:28 pm on November 16, 2025: contributor

    Thanks @andrewtoth! I have applied the patch below on e14650967d, but it seems that it didn’t improve the following benchmark performance (reading the smallest possible amount of data from a block):

     0$ BLOCKHASH=00000000000000000001878540abd91b4dd591287ee58de73a163b698748f23b
     1$ ab -q -k -c 1 -n 100000 "http://localhost:8332/rest/blockpart/$BLOCK.bin?offset=0&size=1"
     2...
     3Server Software:        
     4Server Hostname:        localhost
     5Server Port:            8332
     6
     7Document Path:          /rest/blockpart/00000000000000000001878540abd91b4dd591287ee58de73a163b698748f23b.bin?offset=0&size=1
     8Document Length:        1 bytes
     9
    10Concurrency Level:      1
    11Time taken for tests:   4.395 seconds
    12Complete requests:      100000
    13Failed requests:        0
    14Keep-Alive requests:    100000
    15Total transferred:      10300000 bytes
    16HTML transferred:       100000 bytes
    17Requests per second:    22750.88 [#/sec] (mean)
    18Time per request:       0.044 [ms] (mean)
    19Time per request:       0.044 [ms] (mean, across all concurrent requests)
    20Transfer rate:          2288.42 [Kbytes/sec] received
    21
    22
    23$ ab -V
    24This is ApacheBench, Version 2.3 <$Revision: 1903618 $>
    
     0diff --git a/src/node/blockstorage.cpp b/src/node/blockstorage.cpp
     1index 47beac674c..6a3f39f737 100644
     2--- a/src/node/blockstorage.cpp
     3+++ b/src/node/blockstorage.cpp
     4@@ -1084,7 +1084,7 @@ bool BlockManager::ReadRawBlockPart(std::vector<std::byte>& data, const FlatFile
     5 
     6         size_t offset = 0;
     7         if (part_offset.has_value()) {
     8-            if (*part_offset > blk_size) {
     9+            if (*part_offset > blk_size) [[unlikely]] {
    10                 LogError("Read from block file failed: invalid offset: %u", *part_offset);
    11                 return false;
    12             }
    13@@ -1093,7 +1093,7 @@ bool BlockManager::ReadRawBlockPart(std::vector<std::byte>& data, const FlatFile
    14 
    15         size_t size = blk_size - offset;
    16         if (part_size.has_value()) {
    17-            if (*part_size > size || *part_size == 0) {
    18+            if (*part_size > size || *part_size == 0) [[unlikely]] {
    19                 LogError("Read from block file failed: invalid size: %u", *part_size);
    20                 return false;
    21             }
    22diff --git a/src/rest.cpp b/src/rest.cpp
    23index f3edc69d09..8e1c35b153 100644
    24--- a/src/rest.cpp
    25+++ b/src/rest.cpp
    26@@ -454,7 +454,7 @@ static bool rest_block(const std::any& context,
    27     }
    28 
    29     case RESTResponseFormat::JSON: {
    30-        if (!tx_verbosity.has_value()) {
    31+        if (!tx_verbosity.has_value()) [[unlikely]] {
    32             return RESTERR(req, HTTP_BAD_REQUEST, "JSON output is not supported for this request type");
    33         }
    34         CBlock block{};
    35@@ -491,14 +491,14 @@ static bool rest_block_part(const std::any& context, HTTPRequest* req, const std
    36         const auto part_offset_str = req->GetQueryParameter("offset");
    37         if (part_offset_str.has_value()) {
    38             part_offset = ToIntegral<size_t>(*part_offset_str);
    39-            if (!part_offset.has_value()) {
    40+            if (!part_offset.has_value()) [[unlikely]] {
    41                 return RESTERR(req, HTTP_BAD_REQUEST, strprintf("Block part offset is invalid: %s", *part_offset_str));
    42             }
    43         }
    44         const auto part_size_str = req->GetQueryParameter("size");
    45         if (part_size_str.has_value()) {
    46             part_size = ToIntegral<size_t>(*part_size_str);
    47-            if (!part_size.has_value()) {
    48+            if (!part_size.has_value()) [[unlikely]] {
    49                 return RESTERR(req, HTTP_BAD_REQUEST, strprintf("Block part size is invalid: %s", *part_size_str));
    50             }
    51         }
    

    In both cases (with the patch and without it), the benchmark takes ~4.4s (~44us/req = ~23k req/s).

  47. romanz requested review from andrewtoth on Nov 16, 2025
  48. in src/node/blockstorage.cpp:1105 in e14650967d
    1102+            size = *part_size;
    1103+        }
    1104+
    1105+
    1106+        data.resize(size); // Zeroing of memory is intentional here
    1107+        filein.seek(offset, SEEK_CUR);
    


    hodlinator commented at 1:19 pm on November 21, 2025:

    nit: Could avoid syscall in common case

    0        if (offset > 0) filein.seek(offset, SEEK_CUR);
    

    Other PR avoiding similar syscall which was leading to slowdowns on Windows: #30884 Alternatively one could perform the noop-check inside of AutoFile itself.


    romanz commented at 7:50 pm on November 21, 2025:
  49. in src/node/blockstorage.cpp:1052 in e14650967d
    1044@@ -1046,6 +1045,11 @@ bool BlockManager::ReadBlock(CBlock& block, const CBlockIndex& index) const
    1045 }
    1046 
    1047 bool BlockManager::ReadRawBlock(std::vector<std::byte>& block, const FlatFilePos& pos) const
    1048+{
    1049+    return ReadRawBlockPart(block, pos, std::nullopt, std::nullopt);
    1050+}
    1051+
    1052+bool BlockManager::ReadRawBlockPart(std::vector<std::byte>& data, const FlatFilePos& pos, std::optional<size_t> part_offset, std::optional<size_t> part_size) const
    


    hodlinator commented at 1:23 pm on November 21, 2025:

    No need to have std::optional for the offset, could just be sent in as 0 when reading whole blocks. Same for parameter types in rest.cpp.

    0bool BlockManager::ReadRawBlockPart(std::vector<std::byte>& data, const FlatFilePos& pos, size_t part_offset, std::optional<size_t> part_size) const
    
      0diff --git a/src/node/blockstorage.cpp b/src/node/blockstorage.cpp
      1index 47beac674c..7062735067 100644
      2--- a/src/node/blockstorage.cpp
      3+++ b/src/node/blockstorage.cpp
      4@@ -1046,10 +1046,10 @@ bool BlockManager::ReadBlock(CBlock& block, const CBlockIndex& index) const
      5 
      6 bool BlockManager::ReadRawBlock(std::vector<std::byte>& block, const FlatFilePos& pos) const
      7 {
      8-    return ReadRawBlockPart(block, pos, std::nullopt, std::nullopt);
      9+    return ReadRawBlockPart(block, pos, 0, std::nullopt);
     10 }
     11 
     12-bool BlockManager::ReadRawBlockPart(std::vector<std::byte>& data, const FlatFilePos& pos, std::optional<size_t> part_offset, std::optional<size_t> part_size) const
     13+bool BlockManager::ReadRawBlockPart(std::vector<std::byte>& data, const FlatFilePos& pos, size_t part_offset, std::optional<size_t> part_size) const
     14 {
     15     if (pos.nPos < STORAGE_HEADER_BYTES) {
     16         // If nPos is less than STORAGE_HEADER_BYTES, we can't read the header that precedes the block data
     17@@ -1082,16 +1082,12 @@ bool BlockManager::ReadRawBlockPart(std::vector<std::byte>& data, const FlatFile
     18             return false;
     19         }
     20 
     21-        size_t offset = 0;
     22-        if (part_offset.has_value()) {
     23-            if (*part_offset > blk_size) {
     24-                LogError("Read from block file failed: invalid offset: %u", *part_offset);
     25-                return false;
     26-            }
     27-            offset = *part_offset;
     28+        if (part_offset > blk_size) {
     29+            LogError("Read from block file failed: invalid offset: %u", part_offset);
     30+            return false;
     31         }
     32 
     33-        size_t size = blk_size - offset;
     34+        size_t size = blk_size - part_offset;
     35         if (part_size.has_value()) {
     36             if (*part_size > size || *part_size == 0) {
     37                 LogError("Read from block file failed: invalid size: %u", *part_size);
     38@@ -1102,7 +1098,7 @@ bool BlockManager::ReadRawBlockPart(std::vector<std::byte>& data, const FlatFile
     39 
     40 
     41         data.resize(size); // Zeroing of memory is intentional here
     42-        filein.seek(offset, SEEK_CUR);
     43+        if (part_offset > 0) filein.seek(part_offset, SEEK_CUR);
     44         filein.read(data);
     45     } catch (const std::exception& e) {
     46         LogError("Read from block file failed: %s for %s while reading raw block", e.what(), pos.ToString());
     47diff --git a/src/node/blockstorage.h b/src/node/blockstorage.h
     48index 61128942b3..9f33d826a1 100644
     49--- a/src/node/blockstorage.h
     50+++ b/src/node/blockstorage.h
     51@@ -414,7 +414,7 @@ public:
     52     bool ReadBlock(CBlock& block, const FlatFilePos& pos, const std::optional<uint256>& expected_hash) const;
     53     bool ReadBlock(CBlock& block, const CBlockIndex& index) const;
     54     bool ReadRawBlock(std::vector<std::byte>& block, const FlatFilePos& pos) const;
     55-    bool ReadRawBlockPart(std::vector<std::byte>& data, const FlatFilePos& pos, std::optional<size_t> part_offset, std::optional<size_t> part_size) const;
     56+    bool ReadRawBlockPart(std::vector<std::byte>& data, const FlatFilePos& pos, size_t part_offset, std::optional<size_t> part_size) const;
     57 
     58     bool ReadBlockUndo(CBlockUndo& blockundo, const CBlockIndex& index) const;
     59 
     60diff --git a/src/rest.cpp b/src/rest.cpp
     61index f3edc69d09..294337a24b 100644
     62--- a/src/rest.cpp
     63+++ b/src/rest.cpp
     64@@ -399,7 +399,7 @@ static bool rest_block(const std::any& context,
     65                        HTTPRequest* req,
     66                        const std::string& uri_part,
     67                        std::optional<TxVerbosity> tx_verbosity,
     68-                       std::optional<size_t> part_offset,
     69+                       size_t part_offset,
     70                        std::optional<size_t> part_size)
     71 {
     72     if (!CheckWarmup(req))
     73@@ -475,23 +475,25 @@ static bool rest_block(const std::any& context,
     74 
     75 static bool rest_block_extended(const std::any& context, HTTPRequest* req, const std::string& uri_part)
     76 {
     77-    return rest_block(context, req, uri_part, TxVerbosity::SHOW_DETAILS_AND_PREVOUT, /*part_offset=*/std::nullopt, /*part_size=*/std::nullopt);
     78+    return rest_block(context, req, uri_part, TxVerbosity::SHOW_DETAILS_AND_PREVOUT, /*part_offset=*/0, /*part_size=*/std::nullopt);
     79 }
     80 
     81 static bool rest_block_notxdetails(const std::any& context, HTTPRequest* req, const std::string& uri_part)
     82 {
     83-    return rest_block(context, req, uri_part, TxVerbosity::SHOW_TXID, /*part_offset=*/std::nullopt, /*part_size=*/std::nullopt);
     84+    return rest_block(context, req, uri_part, TxVerbosity::SHOW_TXID, /*part_offset=*/0, /*part_size=*/std::nullopt);
     85 }
     86 
     87 static bool rest_block_part(const std::any& context, HTTPRequest* req, const std::string& uri_part)
     88 {
     89-    std::optional<size_t> part_offset{};
     90+    size_t part_offset{0};
     91     std::optional<size_t> part_size{};
     92     try {
     93         const auto part_offset_str = req->GetQueryParameter("offset");
     94         if (part_offset_str.has_value()) {
     95-            part_offset = ToIntegral<size_t>(*part_offset_str);
     96-            if (!part_offset.has_value()) {
     97+            auto opt = ToIntegral<size_t>(*part_offset_str);
     98+            if (opt.has_value()) {
     99+                part_offset = opt.value();
    100+            } else {
    101                 return RESTERR(req, HTTP_BAD_REQUEST, strprintf("Block part offset is invalid: %s", *part_offset_str));
    102             }
    103         }
    

    romanz commented at 7:50 pm on November 21, 2025:
    Many thanks! e14650967d..8457a71a64
  50. in src/test/blockmanager_tests.cpp:152 in e14650967d outdated
    151+    std::vector<std::byte> block_part{};
    152+
    153+    BOOST_CHECK(blockman.ReadRawBlockPart(block_part, tip.GetBlockPos(), std::nullopt, std::nullopt));
    154+    BOOST_CHECK_EQUAL_COLLECTIONS(block_part.begin(), block_part.end(), block.begin(), block.end());
    155+
    156+    BOOST_CHECK(blockman.ReadRawBlockPart(block_part, tip.GetBlockPos(), 0, std::nullopt));
    


    hodlinator commented at 1:30 pm on November 21, 2025:
    nit: Could call block_part.clear() between each read.

    romanz commented at 7:49 pm on November 21, 2025:
    Sounds good - fixed in e14650967d..8457a71a64.
  51. hodlinator commented at 2:50 pm on November 21, 2025: contributor

    Concept ACK e14650967dc95da5c10e0d6183b6eac3e8243fe5

    #32541 contained some useful tricks that seem to be useful to shrink other index implementations. Thanks for letting it go in favor of this less invasive change though!

  52. romanz force-pushed on Nov 21, 2025
  53. romanz force-pushed on Nov 21, 2025
  54. DrahtBot added the label CI failed on Nov 21, 2025
  55. DrahtBot removed the label CI failed on Nov 21, 2025
  56. romanz requested review from hodlinator on Nov 21, 2025
  57. in src/test/blockmanager_tests.cpp:200 in 8457a71a64
    196+    block_part.clear();
    197+    BOOST_CHECK(blockman.ReadRawBlockPart(block_part, tip.GetBlockPos(), block.size() - 1, 1));
    198+    BOOST_CHECK_EQUAL_COLLECTIONS(block_part.begin(), block_part.end(), block.end() - 1, block.end());
    199+
    200+    BOOST_CHECK(!blockman.ReadRawBlockPart(block_part, tip.GetBlockPos(), 0, 0));
    201+    BOOST_CHECK(!blockman.ReadRawBlockPart(block_part, tip.GetBlockPos(), 0, 0));
    



    romanz commented at 7:26 am on November 22, 2025:
    Oops - thanks! Fixed in 8457a71a64..78d6402458 (also removed 2 duplicate test cases above).
  58. romanz force-pushed on Nov 22, 2025
  59. romanz requested review from hodlinator on Nov 22, 2025
  60. in doc/REST-interface.md:50 in 78d6402458
    46@@ -47,6 +47,11 @@ The HTTP request and response are both handled entirely in-memory.
    47 
    48 With the /notxdetails/ option JSON response will only contain the transaction hash instead of the complete transaction details. The option only affects the JSON response.
    49 
    50+- `GET /rest/blockpart/<BLOCK-HASH>.<bin|hex>?start=X&offset=Y`
    


    hodlinator commented at 8:51 am on November 24, 2025:

    Incorrect names:

    0- `GET /rest/blockpart/<BLOCK-HASH>.<bin|hex>?offset=X&size=Y`
    

    romanz commented at 8:48 pm on November 24, 2025:
    Thanks! Fixed in 78d6402458..c2088a7458.
  61. in doc/release-notes-33657.md:4 in 78d6402458
    0@@ -0,0 +1,5 @@
    1+New RPCs
    2+--------
    3+
    4+- A new REST API endpoint (`/rest/blockpart/BLOCKHASH`) has been introduced
    


    hodlinator commented at 8:53 am on November 24, 2025:

    nit: I think it’s worth including the parameters to make the reason behind adding this more understandable.

    0- A new REST API endpoint (`/rest/blockpart/BLOCKHASH?offset=X&size=Y`) has been introduced
    

    romanz commented at 8:48 pm on November 24, 2025:
    Sounds good -> 78d6402458..c2088a7458
  62. in src/rest.cpp:396 in 78d6402458
    388@@ -379,10 +389,18 @@ static bool rest_spent_txouts(const std::any& context, HTTPRequest* req, const s
    389     }
    390 }
    391 
    392+/**
    393+ * This handler is used by multiple HTTP endpoints:
    394+ * - `/block/` via `rest_block_extended()`
    395+ * - `/block/notxdetails/` via `rest_block_notxdetails()`
    396+ * - `/blockpart/` via `rest_block_part` (doesn't support JSON response, so `tx_verbosity` is unset)
    


    hodlinator commented at 8:54 am on November 24, 2025:

    nit: Consistency

    0 * - `/blockpart/` via `rest_block_part()` (doesn't support JSON response, so `tx_verbosity` is unset)
    

    romanz commented at 8:48 pm on November 24, 2025:
  63. sedited approved
  64. sedited commented at 12:50 pm on November 24, 2025: contributor
    Re-ACK 78d6402458d7d14533444d893c989f0534a3896f
  65. hodlinator commented at 2:43 pm on November 24, 2025: contributor

    Reviewed 78d6402458d7d14533444d893c989f0534a3896f

    Main issue blocking A-C-K is in doc/REST-interface.md (see inline comment).

    Left minor comment/question in related presentation: https://docs.google.com/presentation/d/1Zez-6DApKRu59kke4i_g9jwxQlaFKKRpOPdYFYsFXfA/edit?disco=AAABwl3JY0U (I was re-reading it to remind myself why switching from a tx hash to a block hash would be more efficient, re-realized that mapping block height=>block hash is quite natural).

    Experimented with further optimizations in https://github.com/hodlinator/bitcoin/tree/pr/33657_follow-ups, but as branch name suggests, they need more work and should probably be done as follow-ups. I’ve been unable to measure a stable & really notable performance-improvement yet.

  66. DrahtBot requested review from hodlinator on Nov 24, 2025
  67. romanz force-pushed on Nov 24, 2025
  68. romanz commented at 9:12 pm on November 24, 2025: contributor

    Thanks!

    Left minor comment/question in related presentation: https://docs.google.com/presentation/d/1Zez-6DApKRu59kke4i_g9jwxQlaFKKRpOPdYFYsFXfA/edit?disco=AAABwl3JY0U

    Good point - I’ve changed a bit the slide, and added a few comments.

    This should be similar to the electrs schema, but with txnum instead of block height.

  69. in src/node/blockstorage.cpp:1049 in c2088a7458
    1044@@ -1049,6 +1045,11 @@ bool BlockManager::ReadBlock(CBlock& block, const CBlockIndex& index) const
    1045 }
    1046 
    1047 bool BlockManager::ReadRawBlock(std::vector<std::byte>& block, const FlatFilePos& pos) const
    1048+{
    1049+    return ReadRawBlockPart(block, pos, 0, std::nullopt);
    


    maflcko commented at 10:40 am on November 25, 2025:
    nit: Could use named args consistently, like below? /*part_offset=*/0, /*part_size=*/std::nullopt);

    romanz commented at 5:25 pm on November 25, 2025:
    Thanks! Fixed in c2088a7458...ff5c3fecea.
  70. romanz force-pushed on Nov 25, 2025
  71. romanz requested review from maflcko on Nov 25, 2025
  72. in doc/REST-interface.md:53 in ff5c3fecea
    46@@ -47,6 +47,11 @@ The HTTP request and response are both handled entirely in-memory.
    47 
    48 With the /notxdetails/ option JSON response will only contain the transaction hash instead of the complete transaction details. The option only affects the JSON response.
    49 
    50+- `GET /rest/blockpart/<BLOCK-HASH>.<bin|hex>?offset=X&size=Y`
    51+
    52+Given a block hash: returns a block part, in binary or hex-encoded binary formats.
    53+Responds with 404 if the block doesn't exist.
    


    maflcko commented at 9:02 am on November 26, 2025:
    0Responds with 404 if the block or the byte range doesn't exist.
    

    nit


    romanz commented at 10:23 pm on November 26, 2025:
    Thanks! Fixed in ff5c3fecea..f50c0d9a3e.
  73. in src/node/blockstorage.cpp:1086 in ff5c3fecea outdated
    1081@@ -1081,8 +1082,24 @@ bool BlockManager::ReadRawBlock(std::vector<std::byte>& block, const FlatFilePos
    1082             return false;
    1083         }
    1084 
    1085-        block.resize(blk_size); // Zeroing of memory is intentional here
    1086-        filein.read(block);
    1087+        if (part_offset > blk_size) {
    1088+            LogError("Read from block file failed: invalid offset: %u", part_offset);
    


    maflcko commented at 9:20 am on November 26, 2025:

    the rest interface should be fine to expose to remote untrusted parties, no? So i am not sure about allowing them to unconditionally log confusing errors to the debug log.

    i think it would be better if the caller could decide if there really was a storage corruption and an error should be logged, or if the user just had a typo in the rest url.

    One way to achieve this, would be by returning a failure value.

    Unfortunately, util::Result does not yet allow this: #25665. So maybe for now, the return value could be a std::optional<ReadRawError>?

    0struct ReadRawError{
    1  std::string msg;
    2  bool invalid_range;
    3};
    

    romanz commented at 10:19 pm on November 26, 2025:

    i think it would be better if the caller could decide if there really was a storage corruption and an error should be logged, or if the user just had a typo in the rest url.

    Sounds good - WDYT about https://github.com/romanz/bitcoin/commit/9716ff4dd92b31fa4d97b909892f7fc8e3c82b9c?


    hodlinator commented at 8:50 am on November 28, 2025:

    Did a tweak commit on top of https://github.com/romanz/bitcoin/commit/9716ff4dd92b31fa4d97b909892f7fc8e3c82b9c: https://github.com/hodlinator/bitcoin/commit/0c75c322a404708d82d9bbf31b70f3b036c831ba

    • NotFound -> IO seems more accurate.
    • Add comments to explain why we don’t log for some errors.
    • Marginally nicer errors in rest.cpp (used contrived 0 for unset part_size). Feel less strongly about this.

    maflcko commented at 9:40 am on November 28, 2025:

    Did a tweak commit on top of romanz@9716ff4: hodlinator@0c75c32

    lgtm. Could also merge the two size/offset errors into one about just BadPartRange?

    Seems fine to push as a commit here. It can stay separate, or be squashed, up to you.


    romanz commented at 9:20 am on November 29, 2025:
    Thanks! Fixed in f50c0d9a3e..f33eafff0f.

    hodlinator commented at 4:06 pm on December 2, 2025:

    nit on my own suggestion if you re-touch :grimacing: - consistent ordering of offset/size:

    0-            return RESTERR(req, HTTP_NOT_FOUND, strprintf("Bad block part size/offset %d/%d for %s", part_size.value_or(0), part_offset, hashStr));
    1+            return RESTERR(req, HTTP_NOT_FOUND, strprintf("Bad block part offset/size %d/%d for %s", part_offset, part_size.value_or(0), hashStr));
    

  74. maflcko commented at 9:34 am on November 26, 2025: member

    lgtm, but i think the error handling can be improved?

    lgtm ff5c3feceaba496ff25efd8420cfcc32e0864bcc 🌽

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: lgtm ff5c3feceaba496ff25efd8420cfcc32e0864bcc 🌽
    3rYeG+6HnNmTxgjfUwQVqSoWF82BPiTgTmWMWaiT1YjY7jaNXF081kvStje78wygH2tE+kYnAmG1JxdSD79W+Dg==
    
  75. romanz force-pushed on Nov 26, 2025
  76. in src/test/blockmanager_tests.cpp:152 in f50c0d9a3e
    148+    std::vector<std::byte> block{};
    149+    BOOST_CHECK(blockman.ReadRawBlock(block, tip.GetBlockPos()));
    150+    BOOST_CHECK_GE(block.size(), 200);
    151+
    152+    std::vector<std::byte> block_part{};
    153+    BOOST_CHECK(blockman.ReadRawBlockPart(block_part, tip.GetBlockPos(), 0, std::nullopt));
    


    maflcko commented at 9:55 am on November 28, 2025:
    0    const auto read_tip_part{[&](auto part_offset, auto part_size){return blockman.ReadRawBlockPart(block_part, tip.GetBlockPos(), part_offset, part_size);}};
    1    BOOST_CHECK(read_tip_part(0, std::nullopt));
    

    style nit: Could reduce the verbosity in this line (and below) with a lambda.


    romanz commented at 9:20 am on November 29, 2025:
    Thanks - fixed in f50c0d9a3e..f33eafff0f.
  77. romanz force-pushed on Nov 28, 2025
  78. romanz force-pushed on Nov 28, 2025
  79. DrahtBot added the label CI failed on Nov 28, 2025
  80. DrahtBot commented at 8:54 pm on November 28, 2025: contributor

    🚧 At least one of the CI tasks failed. Task lint: https://github.com/bitcoin/bitcoin/actions/runs/19773241080/job/56661356997 LLM reason (✨ experimental): Lint check failed: unused import typing.Optional detected by ruff, causing non-zero exit.

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  81. romanz force-pushed on Nov 28, 2025
  82. romanz force-pushed on Nov 28, 2025
  83. DrahtBot removed the label CI failed on Nov 28, 2025
  84. romanz commented at 10:14 am on November 29, 2025: contributor
    Many thanks for the review! Pushed f50c0d9a3e..f33eafff0f.
  85. hodlinator approved
  86. hodlinator commented at 4:23 pm on December 2, 2025: contributor

    ACK f33eafff0f3c2698ee69ccef800bbc70a4eeae86

    Provides a REST API useful to external indexers, that shouldn’t be a source of egregious log spam (https://github.com/bitcoin/bitcoin/pull/33657#discussion_r2564184625).

    Measured performance before/after PR change to confirm no slowdown happened when fetching large blocks (URL: http://127.0.0.1:8332/rest/block/0000000000000000000515e202c8ae73c8155fc472422d7593af87aa74f2cf3d.bin).

  87. DrahtBot requested review from sedited on Dec 2, 2025
  88. romanz force-pushed on Dec 2, 2025
  89. hodlinator approved
  90. hodlinator commented at 7:16 pm on December 2, 2025: contributor
    re-ACK 6826375e85a690d583cdeae3ff68d8703b1802ca
  91. in src/rest.cpp:433 in 6826375e85 outdated
    442+        case node::ReadRawError::IO:
    443+            return RESTERR(req, HTTP_NOT_FOUND, "I/O error reading " + hashStr);
    444+
    445+        case node::ReadRawError::BadPartRange:
    446+            return RESTERR(req, HTTP_NOT_FOUND, strprintf("Bad block part offset/size %d/%d for %s", part_offset, part_size.value_or(0), hashStr));
    447+        }
    


    sedited commented at 7:46 pm on December 2, 2025:
    Nit: It would be good to assert that we don’t fall through here, as per the developer-notes, by adding an assert(false) here.


    hodlinator commented at 9:34 am on December 8, 2025:

    The example in the developer notes linked above doesn’t include the default case, but rather puts the assert(false) after the switch. This means the compiler will start warning if enum values are missing from the switch. (Only applicable when all cases return, which they do in this part of the code).

    Please adjust the commits accordingly.


    l0rinc commented at 10:59 am on December 9, 2025:
    is there a difference? Isn’t the default way more self-contained?

    hodlinator commented at 7:23 pm on December 10, 2025:
    Having a default prevents the compiler warning for missing enum values as it is assumed the developer meant for all missing enum values to go to the default.

    l0rinc commented at 10:12 pm on December 10, 2025:
    If we don’t have a default, the compiler will warn since we have the -Wswitch set - in which case I don’t see why the assert(false) is needed. Alternatively, if we add a default: assert(false), we don’t need to specify all values and the compiler won’t warn, the failure will be on runtime. Anyway, off topic, I’m fine either way.
  92. in src/node/blockstorage.cpp:1094 in 6826375e85
    1093+        }
    1094+
    1095+        size_t size = blk_size - part_offset;
    1096+        if (part_size.has_value()) {
    1097+            if (*part_size > size || *part_size == 0) {
    1098+                // Avoid logging - part_offset & part_size are can come from an untrusted source (REST)
    


    sedited commented at 7:53 pm on December 2, 2025:
    Nit: I think the extra are is a typo.

  93. in src/node/blockstorage.cpp:1065 in 6826375e85
    1056         // If nPos is less than STORAGE_HEADER_BYTES, we can't read the header that precedes the block data
    1057         // This would cause an unsigned integer underflow when trying to position the file cursor
    1058         // This can happen after pruning or default constructed positions
    1059         LogError("Failed for %s while reading raw block storage header", pos.ToString());
    1060-        return false;
    1061+        return ReadRawError::IO;
    


    l0rinc commented at 7:55 pm on December 2, 2025:
    there’s a lot happening in a single commit - please consider splitting the low risk changes such as this line from introducing a new feature or a doc change. It would help guide the reviewers if it were split into trivial-to-review chunks where the commit message explains the thinking.

    romanz commented at 10:22 pm on December 2, 2025:

    If f07c765aa6bdba2511ceec56aa7f9755fa29a81e is OK, I will split it into the following commits:

    1. adding support for returning error details via a new ReadRawBlock() overload.
    2. adding support for range reads to the previously added ReadRawBlock() overload.
    3. adding new REST endpoint using the previously added ReadRawBlock() overload.

    WDYT?


    hodlinator commented at 9:55 am on December 3, 2025:
    I’m fine with both 1 commit or breaking apart as suggested.

    romanz commented at 8:34 pm on December 3, 2025:

    Split d2f4bccbf70031b84b6af1a6c8480c4b3071bfd7 into 3 commits:

    0$ git lg -3
    1* e6eb4a57df Roman Zeyde:  rest: allow reading partial block data from storage
    2* 8f0bfb9343 Roman Zeyde:  blockstorage: allow reading partial block data from storage
    3* 07b8278299 Roman Zeyde:  blockstorage: overload `ReadRawBlock()` to return an error code
    

    https://github.com/bitcoin/bitcoin/compare/d2f4bccbf70031b84b6af1a6c8480c4b3071bfd7..e6eb4a57df9f0379550358e4362729bfe5953007


    l0rinc commented at 1:21 pm on December 4, 2025:
    Thanks!
  94. in src/node/blockstorage.h:421 in 6826375e85
    417@@ -414,6 +418,7 @@ class BlockManager
    418     bool ReadBlock(CBlock& block, const FlatFilePos& pos, const std::optional<uint256>& expected_hash) const;
    419     bool ReadBlock(CBlock& block, const CBlockIndex& index) const;
    420     bool ReadRawBlock(std::vector<std::byte>& block, const FlatFilePos& pos) const;
    421+    std::optional<ReadRawError> ReadRawBlockPart(std::vector<std::byte>& data, const FlatFilePos& pos, size_t part_offset, std::optional<size_t> part_size) const;
    


    sedited commented at 8:18 pm on December 2, 2025:
    Nit: I don’t like that we are returning a nullopt for a success code here. If we’d finally have #25665 we could have a proper result type here. Leaving as a nit, because I don’t have a good alternative either.

    romanz commented at 10:19 pm on December 2, 2025:

    If we’d finally have #25665 we could have a proper result type here.

    Agree - but maybe meanwhile we can use the following? https://github.com/bitcoin/bitcoin/compare/6826375e85a690d583cdeae3ff68d8703b1802ca..f07c765aa6bdba2511ceec56aa7f9755fa29a81e (changed the return type back to be boolean + added an output parameter for the error enum)


    hodlinator commented at 9:51 am on December 3, 2025:
    nit: I prefer using std::variant instead of out-parameters. See ab11b4bd7a5844d6ce654b39fca034d73defe973 / https://github.com/bitcoin/bitcoin/compare/master...hodlinator:bitcoin:pr/33657_suggestions2

  95. sedited approved
  96. sedited commented at 8:39 pm on December 2, 2025: contributor
    Re-ACK 6826375e85a690d583cdeae3ff68d8703b1802ca
  97. romanz force-pushed on Dec 2, 2025
  98. in src/node/blockstorage.cpp:1106 in f07c765aa6
    1103+                return false;
    1104+            }
    1105+            size = *part_size;
    1106+        }
    1107+
    1108+
    


    hodlinator commented at 9:52 am on December 3, 2025:
    meganit: double newline

  99. hodlinator approved
  100. hodlinator commented at 9:55 am on December 3, 2025: contributor
    re-ACK f07c765aa6bdba2511ceec56aa7f9755fa29a81e
  101. DrahtBot requested review from sedited on Dec 3, 2025
  102. sedited approved
  103. sedited commented at 10:07 am on December 3, 2025: contributor
    Re-ACK f07c765aa6bdba2511ceec56aa7f9755fa29a81e
  104. romanz force-pushed on Dec 3, 2025
  105. hodlinator approved
  106. hodlinator commented at 7:22 pm on December 3, 2025: contributor
    re-ACK d2f4bccbf70031b84b6af1a6c8480c4b3071bfd7
  107. DrahtBot requested review from sedited on Dec 3, 2025
  108. sedited approved
  109. sedited commented at 7:46 pm on December 3, 2025: contributor
    Re-ACK d2f4bccbf70031b84b6af1a6c8480c4b3071bfd7
  110. romanz force-pushed on Dec 3, 2025
  111. in src/node/blockstorage.cpp:1099 in 07b8278299 outdated
    1101-        return false;
    1102+        return ReadRawError::IO;
    1103     }
    1104-
    1105-    return true;
    1106+    assert(false);
    


    l0rinc commented at 8:48 pm on December 3, 2025:
    what role does this serve, don’t all paths already return? Seems like dead code to me.

    hodlinator commented at 10:04 am on December 5, 2025:
    It was part of my suggestion (https://github.com/bitcoin/bitcoin/pull/33657#discussion_r2584393520), inspired by #33657 (review). A somewhat common alternative to C++23’s std::unreachable().


    l0rinc commented at 12:18 pm on December 5, 2025:
    This is unreachable code, all code paths before return, both in the try and in the catch. The compiler would already warn if any of the code paths wouldn’t return. We don’t add “assert(false)” after a throw or return, since those are all terminating execution. This isn’t the same as non-exhaustive switches (which can change and leave cases uncovered, though a linter could be configured to warn instead), but this is just confusing dead code.

    romanz commented at 2:05 pm on December 7, 2025:
    Fixed in 7802c1f9ca09b5669b8400e96baeadc238fa9df2
  112. in src/node/blockstorage.cpp:1092 in 07b8278299 outdated
    1091         }
    1092 
    1093-        block.resize(blk_size); // Zeroing of memory is intentional here
    1094-        filein.read(block);
    1095+        std::vector<std::byte> data;
    1096+        data.resize(blk_size); // Zeroing of memory is intentional here
    


    l0rinc commented at 8:49 pm on December 3, 2025:
    0        std::vector<std::byte> data(blk_size); // Zeroing of memory is intentional here
    

    romanz commented at 2:05 pm on December 7, 2025:
    Fixed in 7802c1f9ca09b5669b8400e96baeadc238fa9df2
  113. romanz commented at 8:51 pm on December 3, 2025: contributor

    Split https://github.com/bitcoin/bitcoin/commit/d2f4bccbf70031b84b6af1a6c8480c4b3071bfd7 into 3 commits:

    $ git lg -3

    • e6eb4a57df Roman Zeyde: rest: allow reading partial block data from storage
    • 8f0bfb9343 Roman Zeyde: blockstorage: allow reading partial block data from storage
    • 07b8278299 Roman Zeyde: blockstorage: overload ReadRawBlock() to return an error code

    https://github.com/bitcoin/bitcoin/compare/d2f4bccbf70031b84b6af1a6c8480c4b3071bfd7..e6eb4a57df9f0379550358e4362729bfe5953007

  114. sedited approved
  115. sedited commented at 9:01 pm on December 3, 2025: contributor
    Re-ACK e6eb4a57df9f0379550358e4362729bfe5953007
  116. DrahtBot requested review from hodlinator on Dec 3, 2025
  117. hodlinator commented at 10:27 pm on December 3, 2025: contributor

    Been running some benchmarks:

    0₿ cmake --build build -t bench_bitcoin && ./build/bin/bench_bitcoin -filter=ReadRawBlockBench -min-time=15000
    

    (Usually second run has lower error rate).

    baseline - master - 07b82782~ = d30f149360

    0|               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |         bra/op |   miss% |     total | benchmark
    1|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
    2|           85,950.97 |           11,634.54 |    0.1% |      262,668.04 |      118,748.52 |  2.212 |      18,171.03 |    0.1% |     16.44 | `ReadRawBlockBench`
    

    f07c765aa6bdba2511ceec56aa7f9755fa29a81e - out parameters

    0|               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |         bra/op |   miss% |     total | benchmark
    1|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
    2|           86,022.91 |           11,624.81 |    0.1% |      262,706.03 |      117,878.09 |  2.229 |      18,177.03 |    0.1% |     16.51 | `ReadRawBlockBench`
    

    07b82782991bba8e9a798421be742ba5bf729fc4 - std::variant

    0|               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |         bra/op |   miss% |     total | benchmark
    1|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
    2|          108,644.58 |            9,204.33 |    0.1% |      263,338.06 |      199,792.36 |  1.318 |      18,307.05 |    0.1% |     15.97 | `ReadRawBlockBench`
    

    Oh, no!?

    Tweaked 07b82782991bba8e9a798421be742ba5bf729fc4 to not use std::variant

     0diff --git a/src/node/blockstorage.cpp b/src/node/blockstorage.cpp
     1index 5eb80a6205..85deeff3c0 100644
     2--- a/src/node/blockstorage.cpp
     3+++ b/src/node/blockstorage.cpp
     4@@ -1047,27 +1047,22 @@ bool BlockManager::ReadBlock(CBlock& block, const CBlockIndex& index) const
     5 
     6 bool BlockManager::ReadRawBlock(std::vector<std::byte>& block, const FlatFilePos& pos) const
     7 {
     8-    auto ret{ReadRawBlock(pos)};
     9-    if (auto* vec{std::get_if<std::vector<std::byte>>(&ret)}) {
    10-        block = std::move(*vec);
    11-        return true;
    12-    }
    13-    return false;
    14+    return ReadRawBlockPart(block, pos) == ReadRawResult::Success;
    15 }
    16 
    17-std::variant<std::vector<std::byte>, ReadRawError> BlockManager::ReadRawBlock(const FlatFilePos& pos) const
    18+ReadRawResult BlockManager::ReadRawBlockPart(std::vector<std::byte>& data, const FlatFilePos& pos) const
    19 {
    20     if (pos.nPos < STORAGE_HEADER_BYTES) {
    21         // If nPos is less than STORAGE_HEADER_BYTES, we can't read the header that precedes the block data
    22         // This would cause an unsigned integer underflow when trying to position the file cursor
    23         // This can happen after pruning or default constructed positions
    24         LogError("Failed for %s while reading raw block storage header", pos.ToString());
    25-        return ReadRawError::IO;
    26+        return ReadRawResult::IOError;
    27     }
    28     AutoFile filein{OpenBlockFile({pos.nFile, pos.nPos - STORAGE_HEADER_BYTES}, /*fReadOnly=*/true)};
    29     if (filein.IsNull()) {
    30         LogError("OpenBlockFile failed for %s while reading raw block", pos.ToString());
    31-        return ReadRawError::IO;
    32+        return ReadRawResult::IOError;
    33     }
    34 
    35     try {
    36@@ -1079,24 +1074,23 @@ std::variant<std::vector<std::byte>, ReadRawError> BlockManager::ReadRawBlock(co
    37         if (blk_start != GetParams().MessageStart()) {
    38             LogError("Block magic mismatch for %s: %s versus expected %s while reading raw block",
    39                 pos.ToString(), HexStr(blk_start), HexStr(GetParams().MessageStart()));
    40-            return ReadRawError::IO;
    41+            return ReadRawResult::IOError;
    42         }
    43 
    44         if (blk_size > MAX_SIZE) {
    45             LogError("Block data is larger than maximum deserialization size for %s: %s versus %s while reading raw block",
    46                 pos.ToString(), blk_size, MAX_SIZE);
    47-            return ReadRawError::IO;
    48+            return ReadRawResult::IOError;
    49         }
    50 
    51-        std::vector<std::byte> data;
    52         data.resize(blk_size); // Zeroing of memory is intentional here
    53         filein.read(data);
    54-        return data;
    55     } catch (const std::exception& e) {
    56         LogError("Read from block file failed: %s for %s while reading raw block", e.what(), pos.ToString());
    57-        return ReadRawError::IO;
    58+        return ReadRawResult::IOError;
    59     }
    60-    assert(false);
    61+
    62+    return ReadRawResult::Success;
    63 }
    64 
    65 FlatFilePos BlockManager::WriteBlock(const CBlock& block, int nHeight)
    66diff --git a/src/node/blockstorage.h b/src/node/blockstorage.h
    67index a58a661f7f..e335b9bc73 100644
    68--- a/src/node/blockstorage.h
    69+++ b/src/node/blockstorage.h
    70@@ -127,8 +127,9 @@ struct BlockfileCursor {
    71 
    72 std::ostream& operator<<(std::ostream& os, const BlockfileCursor& cursor);
    73 
    74-enum class ReadRawError {
    75-    IO,
    76+enum class ReadRawResult {
    77+    Success,
    78+    IOError,
    79 };
    80 
    81 /**
    82@@ -417,7 +418,7 @@ public:
    83     bool ReadBlock(CBlock& block, const FlatFilePos& pos, const std::optional<uint256>& expected_hash) const;
    84     bool ReadBlock(CBlock& block, const CBlockIndex& index) const;
    85     bool ReadRawBlock(std::vector<std::byte>& block, const FlatFilePos& pos) const;
    86-    std::variant<std::vector<std::byte>, ReadRawError> ReadRawBlock(const FlatFilePos& pos) const;
    87+    ReadRawResult ReadRawBlockPart(std::vector<std::byte>& block, const FlatFilePos& pos) const;
    88 
    89     bool ReadBlockUndo(CBlockUndo& blockundo, const CBlockIndex& index) const;
    90 
    

    Needed to rename to ReadRawBlockPart to avoid collision with overload.

    0|               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |         bra/op |   miss% |     total | benchmark
    1|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
    2|           85,165.30 |           11,741.87 |    0.1% |      262,683.04 |      118,447.23 |  2.218 |      18,174.03 |    0.1% |     16.50 | `ReadRawBlockBench`
    

    Synthetic benchmark too synthetic?

    However, changing the benchmark to not reuse the same vector gets us back to slower speeds:

     0--- a/src/bench/readwriteblock.cpp
     1+++ b/src/bench/readwriteblock.cpp
     2@@ -60,7 +60,8 @@ static void ReadRawBlockBench(benchmark::Bench& bench)
     3     std::vector<std::byte> block_data;
     4     blockman.ReadRawBlock(block_data, pos); // warmup
     5     bench.run([&] {
     6-        const auto success{blockman.ReadRawBlock(block_data, pos)};
     7+        std::vector<std::byte> new_block_data;
     8+        const auto success{blockman.ReadRawBlock(new_block_data, pos)};
     9         assert(success);
    10     });
    11 }
    
    0|               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |         bra/op |   miss% |     total | benchmark
    1|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
    2|          111,627.84 |            8,958.34 |    0.1% |      263,330.06 |      197,929.86 |  1.330 |      18,299.06 |    0.1% |     16.03 | `ReadRawBlockBench`
    

    std::variant version from 07b82782991bba8e9a798421be742ba5bf729fc4 using the tweaked benchmark diff from just above

    0|               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |         bra/op |   miss% |     total | benchmark
    1|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
    2|          111,701.66 |            8,952.42 |    0.1% |      263,285.06 |      197,639.24 |  1.332 |      18,292.05 |    0.1% |     16.01 | `ReadRawBlockBench`
    

    Conclusion

    It would be good to decide whether we want the benchmark to re-use the vector or not. All current callers of ReadRawBlock outside of the benchmark send in fresh vectors AFAIK.

  118. romanz commented at 6:33 am on December 4, 2025: contributor

    Tested REST throughput using block #900000:

    0ab -k -c 1 -n 10000 'http://localhost:8332/rest/block/000000000000000000010538edbfd2d5b809a33dd83f284aeea41c6d0d96968a.bin'
    

    d30f149360 (baseline - before this PR)

     0Concurrency Level:      1
     1Time taken for tests:   11.033 seconds
     2Complete requests:      10000
     3Failed requests:        0
     4Keep-Alive requests:    10000
     5Total transferred:      19208850000 bytes
     6HTML transferred:       19207770000 bytes
     7Requests per second:    906.33 [#/sec] (mean)
     8Time per request:       1.103 [ms] (mean)
     9Time per request:       1.103 [ms] (mean, across all concurrent requests)
    10Transfer rate:          1700158.95 [Kbytes/sec] received
    

    e6eb4a57df (this PR)

     0Concurrency Level:      1
     1Time taken for tests:   10.519 seconds
     2Complete requests:      10000
     3Failed requests:        0
     4Keep-Alive requests:    10000
     5Total transferred:      19208850000 bytes
     6HTML transferred:       19207770000 bytes
     7Requests per second:    950.70 [#/sec] (mean)
     8Time per request:       1.052 [ms] (mean)
     9Time per request:       1.052 [ms] (mean, across all concurrent requests)
    10Transfer rate:          1783387.76 [Kbytes/sec] received
    

    There is no block fetching performance degradation due to this PR.

  119. in src/node/blockstorage.cpp:1056 in e6eb4a57df
    1051+    if (auto* vec{std::get_if<std::vector<std::byte>>(&ret)}) {
    1052+        block = std::move(*vec);
    1053+        return true;
    1054+    }
    1055+    return false;
    1056+}
    


    l0rinc commented at 10:32 am on December 4, 2025:

    While std::variant is a lot better that output parameters (as long as it doesn’t incur a slowdown, since this is on the critical path for IBD), it is a bit awkwards for this situation.

    C++23 provides a better alternative (https://en.cppreference.com/w/cpp/utility/expected.html), but that’s not available for us yet. But we have our own alternative: https://github.com/bitcoin/bitcoin/blob/9a29b2d331eed5b4cbd6922f63e397b68ff12447/src/util/result.h#L19-L90

    Which would make usage so simple that we wouldn’t even need two ReadRawBlock anymore.


    romanz commented at 2:05 pm on December 7, 2025:
    Used util::Expected from fa114be27b17ed32c1d9a7106f313a0df8755fa2
  120. in src/node/blockstorage.cpp:1091 in e6eb4a57df
    1088                 pos.ToString(), blk_size, MAX_SIZE);
    1089-            return false;
    1090+            return ReadRawError::IO;
    1091+        }
    1092+
    1093+        if (part_offset > blk_size) {
    


    l0rinc commented at 10:36 am on December 4, 2025:

    What would be the point of an offset equaling the total size? if we don’t want to get the value of the last valid offset, at least prohibit 0 sizes:

    0        if (part_offset >= blk_size) {
    

    Which should split out the *part_size == 0 case below as well.

    If we added the, a single optional it would also improve readability by separating the partial case neatly, it doesn’t pollute the whole method. We could even filein.seek only when the param is specified.


    romanz commented at 2:06 pm on December 7, 2025:
    Fixed in 71c7b747091fce045dbf25e4e0cfc0e6a4ad6435
  121. in src/node/blockstorage.cpp:1058 in e6eb4a57df
    1053+        return true;
    1054+    }
    1055+    return false;
    1056+}
    1057+
    1058+std::variant<std::vector<std::byte>, ReadRawError> BlockManager::ReadRawBlock(const FlatFilePos& pos, size_t part_offset, std::optional<size_t> part_size) const
    


    l0rinc commented at 10:44 am on December 4, 2025:

    what’s the point of an optional part_size but a mandatory part_offset? We’ve introduced two related args here that should likely be groupes, either as

    0std::optional<std::pair<size_t, size_t>> block_part
    

    or

    0struct BlockPart {
    1    size_t offset;
    2    size_t size;
    3}
    

    romanz commented at 2:07 pm on December 7, 2025:
    Fixed in 71c7b747091fce045dbf25e4e0cfc0e6a4ad6435
  122. in src/test/blockmanager_tests.cpp:146 in e6eb4a57df
    142+{
    143+    LOCK(::cs_main);
    144+    auto& chainman = m_node.chainman;
    145+    auto& blockman = chainman->m_blockman;
    146+    const CBlockIndex& tip = *chainman->ActiveTip();
    147+    const FlatFilePos tip_block_pos = tip.GetBlockPos();
    


    l0rinc commented at 10:47 am on December 4, 2025:
    nit: could we use brace init consistently in the change (it helps with a few cases, e.g. warns on narrowing conversions)

    romanz commented at 2:08 pm on December 7, 2025:
    Fixed in 71c7b747091fce045dbf25e4e0cfc0e6a4ad6435
  123. in src/test/blockmanager_tests.cpp:164 in e6eb4a57df
    160+        }
    161+        return false;
    162+    }};
    163+
    164+    BOOST_CHECK(read_tip_part(0, std::nullopt));
    165+    BOOST_CHECK_EQUAL_COLLECTIONS(block_part.begin(), block_part.end(), block.begin(), block.end());
    


    l0rinc commented at 10:47 am on December 4, 2025:

    there’s a lot of repetition here, can we separate the data from the algorithm?

    And could you please add a fuzz test which exercises the same - if the randomly generated range returns valid bytes for the complete block, the new rpc should also return the same values. If it errs, so should the new RPC.


    hodlinator commented at 10:09 am on December 5, 2025:
    Unit test seems sufficient to me.

    l0rinc commented at 12:20 pm on December 5, 2025:
    If so, I’d randomize the unit tests to avoid our bias (such as not having max values)
  124. in src/node/blockstorage.h:421 in e6eb4a57df
    417@@ -456,6 +418,7 @@ class BlockManager
    418     bool ReadBlock(CBlock& block, const FlatFilePos& pos, const std::optional<uint256>& expected_hash) const;
    419     bool ReadBlock(CBlock& block, const CBlockIndex& index) const;
    420     bool ReadRawBlock(std::vector<std::byte>& block, const FlatFilePos& pos) const;
    421+    std::variant<std::vector<std::byte>, ReadRawError> ReadRawBlock(const FlatFilePos& pos, size_t part_offset, std::optional<size_t> part_size) const;
    


    l0rinc commented at 11:23 am on December 4, 2025:

    I don’t like that we have 2 separate methods doing the same thing. if we grouped the new parameters (with default nullopt) and used a dedicated return value of

    0using ReadRawBlockResult = util::Result<std::vector<std::byte>, ReadRawError>;
    

    (needs a slight adjustment to util::Result)

    we could have a single:

    0ReadRawBlockResult ReadRawBlock(const FlatFilePos& pos, std::optional<std::pair<size_t, size_t>> block_part = std::nullopt) const;
    

    We would need to adjust a few usages, please see my suggested implementation of that.


    hodlinator commented at 10:17 am on December 5, 2025:

    Agree it probably makes sense to have an optional for the range. It’s interesting to explore what could be done with util::Result, but I feel that could be done as a follow-up to this PR.

    Agree that the ergonomics of std::expected are better than std::variant, but not that they are terrible. If you are strongly against std::variant then maybe we could step back to using out-parameters for now in this PR.


    l0rinc commented at 12:20 pm on December 5, 2025:
    I’m mostly against introducing an extra method that basically does the same thing
  125. in src/node/blockstorage.cpp:1096 in e6eb4a57df
    1095+            return ReadRawError::BadPartRange;
    1096         }
    1097 
    1098-        block.resize(blk_size); // Zeroing of memory is intentional here
    1099-        filein.read(block);
    1100+        size_t size = blk_size - part_offset;
    


    l0rinc commented at 11:46 am on December 4, 2025:
    can we add a test with std::numeric_limits<size_t>::max() for both offset and size to make sure we don’t have overflow problems?

    hodlinator commented at 10:01 am on December 8, 2025:

    Would be nice to include something like this in unit tests.

     0--- a/src/test/blockmanager_tests.cpp
     1+++ b/src/test/blockmanager_tests.cpp
     2@@ -188,6 +188,9 @@ BOOST_FIXTURE_TEST_CASE(blockmanager_block_data_part_error, TestChain100Setup)
     3     }};
     4 
     5     expect_part_error(0, 0);
     6+    expect_part_error(std::numeric_limits<size_t>::max(), 1);
     7+    expect_part_error(0, std::numeric_limits<size_t>::max());
     8+    expect_part_error(std::numeric_limits<size_t>::max(), std::numeric_limits<size_t>::max());
     9     expect_part_error(0, block->size() + 1);
    10     expect_part_error(1, block->size());
    11     expect_part_error(2, block->size() - 1);
    
  126. in src/rest.cpp:517 in e6eb4a57df
    514+            if (!part_size.has_value()) {
    515+                return RESTERR(req, HTTP_BAD_REQUEST, strprintf("Block part size is invalid: %s", *part_size_str));
    516+            }
    517+        }
    518+    } catch (const std::runtime_error& e) {
    519+        return RESTERR(req, HTTP_BAD_REQUEST, e.what());
    


    l0rinc commented at 12:02 pm on December 4, 2025:

    The idiomatic way to use the std::optional is to guard in the if condition for its existence. .value() only really makes sense over * when we need it to throw - which it can’t if we’re already guarding. Also, the first condition has an if-exists-assign-else-return format, while the second checks for non-existence and assigns to external. We could also just return from the try for the happy case, narrowing the scopes of the related variables.

    And what is the expected behavior when part_size_str is nullopt, I don’t think we should try to dereference it in case of an error. We could likely simplify this by only assigning the pair optional when both are present and use value_or("") for simpler validation:

     0static bool rest_block_part(const std::any& context, HTTPRequest* req, const std::string& uri_part)
     1{
     2    try {
     3        std::optional<std::pair<size_t, size_t>> block_part{};
     4        if (const auto opt_offset{ToIntegral<size_t>(req->GetQueryParameter("offset").value_or(""))}) {
     5            if (const auto opt_size{ToIntegral<size_t>(req->GetQueryParameter("size").value_or(""))}) {
     6                block_part = {*opt_offset, *opt_size};
     7            } else {
     8                return RESTERR(req, HTTP_BAD_REQUEST, "Block part size missing or invalid");
     9            }
    10        } else {
    11            return RESTERR(req, HTTP_BAD_REQUEST, "Block part offset missing or invalid");
    12        }
    13        return rest_block(context, req, uri_part, /*tx_verbosity=*/std::nullopt, block_part);
    14    } catch (const std::runtime_error& e) {
    15        return RESTERR(req, HTTP_BAD_REQUEST, e.what());
    16    }
    17}
    

    andrewtoth commented at 7:37 pm on December 6, 2025:

    We can make this a little easier to parse with less nesting if we did:

     0        std::pair<size_t, size_t> block_part;
     1        if (const auto opt_offset{ToIntegral<size_t>(req->GetQueryParameter("offset").value_or(""))}) {
     2            block_part.first = *opt_offset;
     3        } else {
     4            return RESTERR(req, HTTP_BAD_REQUEST, "Block part offset missing or invalid");
     5        }
     6        if (const auto opt_size{ToIntegral<size_t>(req->GetQueryParameter("size").value_or(""))}) {
     7            block_part.second = *opt_size;
     8        } else {
     9            return RESTERR(req, HTTP_BAD_REQUEST, "Block part size missing or invalid");
    10        }
    11        return rest_block(context, req, uri_part, /*tx_verbosity=*/std::nullopt, block_part);
    

    l0rinc commented at 9:27 pm on December 6, 2025:
    In this case I think the nesting’s fine, it signals clearly that opt_size should only be checked if opt_offset was already successful - in the suggested version we have to check the return values. Both are fine, I’m just explaining why we may not want to avoid the nesting here.

    andrewtoth commented at 4:23 pm on December 7, 2025:

    We can at least change the declaration to not be an optional:

    0std::pair<size_t, size_t> block_part;
    

  127. in src/test/blockmanager_tests.cpp:193 in e6eb4a57df
    189+    BOOST_CHECK_EQUAL_COLLECTIONS(block_part.begin(), block_part.end(), block.begin() + 10, block.begin() + 30);
    190+
    191+    BOOST_CHECK(read_tip_part(block.size() - 1, 1));
    192+    BOOST_CHECK_EQUAL_COLLECTIONS(block_part.begin(), block_part.end(), block.end() - 1, block.end());
    193+
    194+    // check errors
    


    l0rinc commented at 1:33 pm on December 4, 2025:
    these should be in a separate test for clarity - similarly deduplicating the data from the algo
  128. in src/rest.cpp:466 in e6eb4a57df
    462@@ -436,10 +463,13 @@ static bool rest_block(const std::any& context,
    463     }
    464 
    465     case RESTResponseFormat::JSON: {
    466+        if (!tx_verbosity.has_value()) {
    


    l0rinc commented at 1:34 pm on December 4, 2025:
    we should invert this to be able to use the optional in the idiomatic way of if (opt) ... *opt
  129. in src/rest.cpp:487 in e6eb4a57df
    483@@ -454,12 +484,39 @@ static bool rest_block(const std::any& context,
    484 
    485 static bool rest_block_extended(const std::any& context, HTTPRequest* req, const std::string& uri_part)
    486 {
    487-    return rest_block(context, req, uri_part, TxVerbosity::SHOW_DETAILS_AND_PREVOUT);
    488+    return rest_block(context, req, uri_part, TxVerbosity::SHOW_DETAILS_AND_PREVOUT, /*part_offset=*/0, /*part_size=*/std::nullopt);
    


    l0rinc commented at 1:35 pm on December 4, 2025:
    we could avoid the default block_part value here
  130. in test/functional/interface_rest.py:475 in e6eb4a57df
    470+                    resp = bytes.fromhex(resp.decode().strip())
    471+                return resp
    472+
    473+            assert_equal(block_bin, _get_block_part())
    474+            assert_equal(block_bin, _get_block_part(query_params={"offset": 0}))
    475+            assert_equal(block_bin, _get_block_part(query_params={"size": len(block_bin)}))
    


    l0rinc commented at 1:49 pm on December 4, 2025:
    I don’t like these partial matches, is there any disadvantage in mandating both parameters?
  131. l0rinc changes_requested
  132. l0rinc commented at 2:18 pm on December 4, 2025: contributor

    Thanks for taking care of this and for splitting the change into smaller chunks. The change is quite straightforward now, it helped with experimenting with it locally.

    Concept ACK

    I left a few suggestions:

    • std::variant is not meant to be used as an error monad, it results in very awkward code (e.g. compared to the usage of std::optional or std::expected). But it seems we already have such a try sum type that we only have to adjust slightly for it to simplify the change a lot. I pushed that separately in #34005, feel free to cherry-pick it here if you agree it helps.
    • I don’t see why we would make both part_size and part_offset optional, it complicates the use case for no obvious benefit. We’re even dereferencing the nullopt when trying to parse them.
    • The above would enable us to have a single BlockManager::ReadRawBlock implementation instead of this weird delegation, and even the usages would be simplified as a result.
    • We need to add some fuzz tests comparing it against the previous full reads, and maybe a std::numeric_limits<size_t>::max() test for the unit tests.
    • Could you provide a patch for me which would go through every single transaction in mainchain history and query it both ways (full block + new RPC) and assert that they’re equal? I expect it to be heavy, I can run it on my benchmark servers to make sure it works for every real scenario we have.
    • Some brace-init inconsistencies and test duplications could also be addressed if we’re touching it again.
    • As @hodlinator also mentioned, we should adjust the benchmark to make sure this doesn’t introduce a regression (and we have to revive https://corecheck.dev/bitcoin/bitcoin/pulls/33657 somehow, seems it’s offline for some time now).

    Please see https://github.com/l0rinc/bitcoin/pull/61 for my detailed suggestions.

    Edit: I ran a differential reindex-chainstate on an rpi5 to see if this introduces any slowdown - it’s within noise.

     0COMMITS="9a29b2d331eed5b4cbd6922f63e397b68ff12447 3e2e155e3ebe3718af1a496b08f48d078b6a5b65"; STOP=923319; DBCACHE=4500; CC=gcc; CXX=g++; BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; (echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done) && (echo "" && echo "reindex-chainstate | ${STOP} blocks | dbcache ${DBCACHE} | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM | $(df -T $BASE_DIR | awk 'NR==2{print $2}') | $(lsblk -no ROTA $(df --output=source $BASE_DIR | tail -1) | grep -q 0 && echo SSD || echo HDD)"; echo "") &&hyperfine   --sort command   --runs 1   --export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json"   --parameter-list COMMIT ${COMMITS// /,}   --prepare "killall -9 bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
     1    cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_IPC=OFF && ninja -C build bitcoind -j2 && \
     2    ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20"   --conclude "killall bitcoind || true; sleep 5; grep -q 'height=0' $DATA_DIR/debug.log && grep -q 'Disabling script verification at block [#1](/bitcoin-bitcoin/1/)' $DATA_DIR/debug.log && grep -q 'height=$STOP' $DATA_DIR/debug.log; \
     3              cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log"   "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0";
     4
     59a29b2d331 Merge bitcoin/bitcoin#33857: doc: Add `x86_64-w64-mingw32ucrt` triplet to `depends/README.md`
     63e2e155e3e rest: allow reading partial block data from storage
     7
     8reindex-chainstate | 923319 blocks | dbcache 4500 | rpi5-16-2 | aarch64 | Cortex-A76 | 4 cores | 15Gi RAM | ext4 | SSD
     9
    10Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=923319 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 9a29b2d331eed5b4cbd6922f63e397b68ff12447)
    11  Time (abs ):        37089.832 s               [User: 45041.206 s, System: 3372.164 s]
    12 
    13Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=923319 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 3e2e155e3ebe3718af1a496b08f48d078b6a5b65)
    14  Time (abs ):        37277.807 s               [User: 45134.733 s, System: 3456.199 s]
    15 
    16Relative speed comparison
    17        1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=923319 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 9a29b2d331eed5b4cbd6922f63e397b68ff12447)
    18        1.01          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=923319 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 3e2e155e3ebe3718af1a496b08f48d078b6a5b65)
    
  133. hodlinator commented at 10:32 am on December 5, 2025: contributor

    Could you provide a patch for me which would go through every single transaction in mainchain history and query it both ways (full block + new RPC) and assert that they’re equal? I expect it to be heavy, I can run it on my benchmark servers to make sure it works for every real scenario we have.

    What value would that provide? The new API allows fetching arbitrary byte-ranges from within blocks. The current unit tests checking the bytes match seem sufficient for validating the logic (testing with std::numeric_limits::max() as you suggested would be a nice touch though).

  134. DrahtBot requested review from hodlinator on Dec 5, 2025
  135. l0rinc commented at 12:23 pm on December 5, 2025: contributor

    Could you provide a patch for me which would go through every single transaction in mainchain history and query it both ways (full block + new RPC) and assert that they’re equal? I expect it to be heavy, I can run it on my benchmark servers to make sure it works for every real scenario we have.

    What value would that provide? The new API allows fetching arbitrary byte-ranges from within blocks. The current unit tests checking the bytes match seem sufficient for validating the logic (testing with std::numeric_limits::max() as you suggested would be a nice touch though).

    We have a very unique opportunity compared to other software: many of our problems are bounded, we can actually try out every meaningful combination that we expect the users to try - that’s what I’m suggesting, validating that every single transaction so far can be fetched with both the old and the new api to be sure we won’t have surprises.

  136. romanz commented at 2:38 pm on December 6, 2025: contributor

    Many thanks @l0rinc!

    I have rebased this PR over #34006 together with https://github.com/l0rinc/bitcoin/pull/61/commits/536f93df79975f50781072301947cafa4640b606 - please take a look :)

  137. romanz force-pushed on Dec 6, 2025
  138. romanz commented at 2:41 pm on December 6, 2025: contributor

    Could you provide a patch for me which would go through every single transaction in mainchain history and query it both ways (full block + new RPC) and assert that they’re equal?

    Sounds good - will it be OK to write it as a Rust CLI tool to interact with bitcoind?

  139. romanz force-pushed on Dec 6, 2025
  140. DrahtBot added the label CI failed on Dec 6, 2025
  141. DrahtBot commented at 3:35 pm on December 6, 2025: contributor

    🚧 At least one of the CI tasks failed. Task test each commit: https://github.com/bitcoin/bitcoin/actions/runs/19989968537/job/57329252547 LLM reason (✨ experimental): CI failed due to git rebase stopping: the index/working tree has uncommitted changes.

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  142. in src/net_processing.cpp:2279 in 194a8ebf8c outdated
    2275@@ -2276,8 +2276,9 @@ void PeerManagerImpl::ProcessGetBlockData(CNode& pfrom, Peer& peer, const CInv&
    2276     } else if (inv.IsMsgWitnessBlk()) {
    2277         // Fast-path: in this case it is possible to serve the block directly from disk,
    2278         // as the network format matches the format on disk
    2279-        std::vector<std::byte> block_data;
    2280-        if (!m_chainman.m_blockman.ReadRawBlock(block_data, block_pos)) {
    2281+        if (auto block_data{m_chainman.m_blockman.ReadRawBlock(block_pos)}) {
    


    andrewtoth commented at 7:38 pm on December 6, 2025:

    nit

    0        if (const auto block_data{m_chainman.m_blockman.ReadRawBlock(block_pos)}) {
    

  143. in src/node/blockstorage.cpp:1009 in 194a8ebf8c outdated
    1005@@ -1006,14 +1006,14 @@ bool BlockManager::ReadBlock(CBlock& block, const FlatFilePos& pos, const std::o
    1006     block.SetNull();
    1007 
    1008     // Open history file to read
    1009-    std::vector<std::byte> block_data;
    1010-    if (!ReadRawBlock(block_data, pos)) {
    1011+    auto block_data{ReadRawBlock(pos)};
    


    andrewtoth commented at 7:38 pm on December 6, 2025:

    nit

    0    const auto block_data{ReadRawBlock(pos)};
    

  144. in src/rest.cpp:426 in 194a8ebf8c outdated
    422@@ -416,34 +423,46 @@ static bool rest_block(const std::any& context,
    423         pos = pblockindex->GetBlockPos();
    424     }
    425 
    426-    std::vector<std::byte> block_data{};
    427-    if (!chainman.m_blockman.ReadRawBlock(block_data, pos)) {
    428-        return RESTERR(req, HTTP_NOT_FOUND, hashStr + " not found");
    429+    auto block_data{chainman.m_blockman.ReadRawBlock(pos, block_part)};
    


    andrewtoth commented at 7:39 pm on December 6, 2025:

    nit

    0    const auto block_data{chainman.m_blockman.ReadRawBlock(pos, block_part)};
    

  145. in src/rpc/blockchain.cpp:690 in 194a8ebf8c outdated
    693-        // pruned right after we released the lock above.
    694-        throw JSONRPCError(RPC_MISC_ERROR, "Block not found on disk");
    695-    }
    696-
    697-    return data;
    698+    if (auto data{blockman.ReadRawBlock(pos)}) return *data;
    


    andrewtoth commented at 7:39 pm on December 6, 2025:

    nit

    0    if (const auto data{blockman.ReadRawBlock(pos)}) return *data;
    


    hodlinator commented at 9:58 am on December 8, 2025:
    (Don’t see the value in const here as we return it directly, but the noise isn’t too bad).

    andrewtoth commented at 3:07 pm on December 9, 2025:
    So, I think making this const prevents RVO here and this should be reverted (sorry). I’m also unclear if the compiler will move the value out of the util::Expected. Do we need this to be return std::move(*data);?

    l0rinc commented at 4:00 pm on December 9, 2025:

    I think making this const prevents RVO here

    Can you explain why you think that’s the case?


    andrewtoth commented at 4:07 pm on December 9, 2025:

    The compiler can’t automatically move const lvalue return values to the callsite.

    If this was just returning the value, like return data;, then it would be straightforward and const would definitely be undesirable and require a copy. Regardless, the compiler here won’t be able to move the expected value out of the Expected struct if it is const.

    Since we have to dereference here and are not returning the lvalue itself, I’m unsure if the compiler will be able to move the expected value out of the Expected or not. If not, we should wrap it in std::move.


    l0rinc commented at 4:49 pm on December 10, 2025:
    @romanz, can you please check if there’s a performance difference in the RPC with and without a const here?

    romanz commented at 10:33 pm on December 10, 2025:

    Not sure whether getblock JSON RPC performance measurement will be precise enough (due to RPC & serialization overhead)…

    Maybe it would be easier to inline GetRawBlockChecked (similar to how it’s done in rest.cpp)?

     0diff --git a/src/rpc/blockchain.cpp b/src/rpc/blockchain.cpp
     1index 40761d5cab..e179284fcf 100644
     2--- a/src/rpc/blockchain.cpp
     3+++ b/src/rpc/blockchain.cpp
     4@@ -678,21 +678,6 @@ static CBlock GetBlockChecked(BlockManager& blockman, const CBlockIndex& blockin
     5     return block;
     6 }
     7 
     8-static std::vector<std::byte> GetRawBlockChecked(BlockManager& blockman, const CBlockIndex& blockindex)
     9-{
    10-    FlatFilePos pos{};
    11-    {
    12-        LOCK(cs_main);
    13-        CheckBlockDataAvailability(blockman, blockindex, /*check_for_undo=*/false);
    14-        pos = blockindex.GetBlockPos();
    15-    }
    16-
    17-    if (const auto data{blockman.ReadRawBlock(pos)}) return *data;
    18-    // Block not found on disk. This shouldn't normally happen unless the block was
    19-    // pruned right after we released the lock above.
    20-    throw JSONRPCError(RPC_MISC_ERROR, "Block not found on disk");
    21-}
    22-
    23 static CBlockUndo GetUndoChecked(BlockManager& blockman, const CBlockIndex& blockindex)
    24 {
    25     CBlockUndo blockUndo;
    26@@ -815,7 +800,8 @@ static RPCHelpMan getblock()
    27 
    28     const CBlockIndex* pblockindex;
    29     const CBlockIndex* tip;
    30-    ChainstateManager& chainman = EnsureAnyChainman(request.context);
    31+    ChainstateManager& chainman{EnsureAnyChainman(request.context)};
    32+    FlatFilePos pos{};
    33     {
    34         LOCK(cs_main);
    35         pblockindex = chainman.m_blockman.LookupBlockIndex(hash);
    36@@ -824,15 +810,22 @@ static RPCHelpMan getblock()
    37         if (!pblockindex) {
    38             throw JSONRPCError(RPC_INVALID_ADDRESS_OR_KEY, "Block not found");
    39         }
    40+
    41+        CheckBlockDataAvailability(chainman.m_blockman, *pblockindex, /*check_for_undo=*/false);
    42+        pos = pblockindex->GetBlockPos();
    43     }
    44 
    45-    const std::vector<std::byte> block_data{GetRawBlockChecked(chainman.m_blockman, *pblockindex)};
    46+    const auto block_data{chainman.m_blockman.ReadRawBlock(pos)};
    47+    // Block not found on disk. This shouldn't normally happen unless the block was
    48+    // pruned right after we released the lock above.
    49+    if (!block_data) throw JSONRPCError(RPC_MISC_ERROR, "Block not found on disk");
    50+
    51 
    52     if (verbosity <= 0) {
    53-        return HexStr(block_data);
    54+        return HexStr(*block_data);
    55     }
    56 
    57-    DataStream block_stream{block_data};
    58+    DataStream block_stream{*block_data};
    59     CBlock block{};
    60     block_stream >> TX_WITH_WITNESS(block);
    
  146. romanz force-pushed on Dec 7, 2025
  147. romanz force-pushed on Dec 7, 2025
  148. romanz commented at 11:29 am on December 7, 2025: contributor

    Could you provide a patch for me which would go through every single transaction in mainchain history and query it both ways (full block + new RPC) and assert that they’re equal?

    Sounds good - will it be OK to write it as a Rust CLI tool to interact with bitcoind?

    https://github.com/romanz/fetch-txs can be used for equivalence testing:

     0$ cargo run --release -- 900000 900010
     1    Finished `release` profile [optimized] target(s) in 0.11s
     2     Running `target/release/fetch-txs 900000 900010`
     3000000000000000000010538edbfd2d5b809a33dd83f284aeea41c6d0d96968a  1920777 bytes, 1562 txs
     400000000000000000001a8ff030609a6248e0f6e77f9f141aeb21e4eac4f83fc  1543737 bytes, 2170 txs
     500000000000000000001045e6be5e294b6c3a12e3ae00ad36287cf9579e0c1c6  1121121 bytes, 2455 txs
     6000000000000000000019ba085f3b1933b9431fd4a57b80e5d0f67996ccbe02c  553149 bytes, 968 txs
     70000000000000000000166ff021ddd7c5736e917a3a7d16f2e2fff684cb6ecdd  1332731 bytes, 1656 txs
     8000000000000000000017bfd05b5fa367a424c4a565a4baf7950d9e8605df8ec  1469385 bytes, 5402 txs
     9000000000000000000022fa88a5322ec04b22aa89c1e4ecc649477a39bf334cb  1710919 bytes, 2258 txs
    1000000000000000000001456223bb003fbf52eef4c7875d1bd57744d899f9a3fc  1160045 bytes, 1930 txs
    110000000000000000000163b288aeee6711ff667818053fe1e712513ca9c19776  1114941 bytes, 2083 txs
    12000000000000000000015ed1e7ea326a848e643264b738e9520f7cbcbc40b0b8  949976 bytes, 1910 txs
    13000000000000000000002c331b475c269de59d6d422a658cb843cb2cc3487098  1442990 bytes, 2829 txs
    
  149. DrahtBot removed the label CI failed on Dec 7, 2025
  150. in doc/release-notes-33657.md:1 in 9efccb0d3e outdated
    0@@ -0,0 +1,5 @@
    1+New RPCs
    


    l0rinc commented at 4:25 pm on December 7, 2025:

    I think this should rather be

    0New REST API
    

    romanz commented at 3:51 pm on December 10, 2025:
    Thanks - updated in 8b417087ae
  151. romanz force-pushed on Dec 7, 2025
  152. in src/bench/readwriteblock.cpp:1 in 262f4dfe69 outdated


    hodlinator commented at 9:20 am on December 8, 2025:
    nit in 7802c1f9ca09b5669b8400e96baeadc238fa9df2 “blockstorage: return an error code from ReadRawBlock()”: The commit message should probably note that the benchmark performance decreases due to no longer reusing a vector with an unchanging capacity (but that it mirrors our production code behavior).

    l0rinc commented at 10:55 am on December 9, 2025:
    Unfortunately https://corecheck.dev/bitcoin/bitcoin/pulls/33657 is still dead, so we don’t actually see this change :/

    l0rinc commented at 4:31 pm on December 10, 2025:

    It came back, as expected, it shows the benchmark getting slower because of the vector recreation (making it more realistic):


    I have checked that the refactor doesn’t introduce any regression now 👍


    romanz commented at 4:01 pm on December 11, 2025:
    Updated the commit message in f195f46a996b9803d26865cdcef185806c22a723.
  153. in src/node/blockstorage.cpp:1058 in 262f4dfe69
    1055         // If nPos is less than STORAGE_HEADER_BYTES, we can't read the header that precedes the block data
    1056         // This would cause an unsigned integer underflow when trying to position the file cursor
    1057         // This can happen after pruning or default constructed positions
    1058         LogError("Failed for %s while reading raw block storage header", pos.ToString());
    1059-        return false;
    1060+        return util::Unexpected(ReadRawError::IO);
    


    hodlinator commented at 9:25 am on December 8, 2025:

    meganit: Prefer curly braces by convention in modern code

    0        return util::Unexpected{ReadRawError::IO};
    

    romanz commented at 3:53 pm on December 10, 2025:
    Fixed in 1be1c165b3 & 52ac60d141
  154. in src/rest.cpp:432 in 262f4dfe69
    430+    if (!block_data) {
    431+        switch (block_data.error()) {
    432+        case node::ReadRawError::IO: return RESTERR(req, HTTP_NOT_FOUND, "I/O error reading " + hashStr);
    433+        case node::ReadRawError::BadPartRange:
    434+            if (block_part) {
    435+                return RESTERR(req, HTTP_NOT_FOUND, strprintf("Bad block part offset/size %d/%d for %s", block_part->first, block_part->second, hashStr));
    


    hodlinator commented at 9:47 am on December 8, 2025:

    nit: Might be a more appropriate error code?

    0                return RESTERR(req, HTTP_BAD_REQUEST, strprintf("Bad block part offset/size %d/%d for %s", block_part->first, block_part->second, hashStr));
    

    l0rinc commented at 11:05 am on December 9, 2025:
    Indeed

    romanz commented at 3:52 pm on December 10, 2025:
    Fixed in 8b417087ae
  155. in src/rest.cpp:434 in 262f4dfe69
    432+        case node::ReadRawError::IO: return RESTERR(req, HTTP_NOT_FOUND, "I/O error reading " + hashStr);
    433+        case node::ReadRawError::BadPartRange:
    434+            if (block_part) {
    435+                return RESTERR(req, HTTP_NOT_FOUND, strprintf("Bad block part offset/size %d/%d for %s", block_part->first, block_part->second, hashStr));
    436+            } else {
    437+                return RESTERR(req, HTTP_NOT_FOUND, strprintf("Empty block part for %s", hashStr));
    


    hodlinator commented at 9:50 am on December 8, 2025:

    BadPartRange should only happen when a block_part is specified, so I would prefer an assert() over this contrived error condition:

    0        case node::ReadRawError::BadPartRange:
    1            assert(block_part);
    2            return RESTERR(req, HTTP_NOT_FOUND, strprintf("Bad block part offset/size %d/%d for %s", block_part->first, block_part->second, hashStr));
    

    romanz commented at 3:52 pm on December 10, 2025:
    Changed in 8b417087ae.
  156. in src/rest.cpp:452 in 262f4dfe69 outdated
    463-        UniValue objBlock = blockToJSON(chainman.m_blockman, block, *tip, *pblockindex, tx_verbosity, chainman.GetConsensus().powLimit);
    464-        std::string strJSON = objBlock.write() + "\n";
    465-        req->WriteHeader("Content-Type", "application/json");
    466-        req->WriteReply(HTTP_OK, strJSON);
    467-        return true;
    468+        if (tx_verbosity) {
    


    hodlinator commented at 9:52 am on December 8, 2025:

    nit: Preference for the following in favor of smaller diff:

    0        if (!tx_verbosity) {
    1            return RESTERR(req, HTTP_BAD_REQUEST, "JSON output is not supported for this request type");
    2        }
    

    l0rinc commented at 11:07 am on December 9, 2025:
    That was the previous version before I explicitly suggested this. Either is fine I guess, this way we have the dereferencing inside the checking condition, with your suggestion the diff would be smaller.
  157. hodlinator commented at 10:09 am on December 8, 2025: contributor

    Reviewed 262f4dfe691b12171a31c9219ba33f1fe55f67a7

    Love how elegant 71c7b747091fce045dbf25e4e0cfc0e6a4ad6435 turned out!

  158. in src/net_processing.cpp:533 in fa114be27b outdated
    529@@ -530,7 +530,7 @@ class PeerManagerImpl final : public PeerManager
    530     /** Implement PeerManager */
    531     void StartScheduledTasks(CScheduler& scheduler) override;
    532     void CheckForStaleTipAndEvictPeers() override;
    533-    std::optional<std::string> FetchBlock(NodeId peer_id, const CBlockIndex& block_index) override
    534+    util::Expected<void, std::string> FetchBlock(NodeId peer_id, const CBlockIndex& block_index) override
    


    l0rinc commented at 10:50 am on December 9, 2025:
    util::Expected was just merged, you can rebase the PR now, thanks for using it here, the results are a lot more readable
  159. in src/bench/readwriteblock.cpp:60 in 7802c1f9ca outdated
    56@@ -57,11 +57,10 @@ static void ReadRawBlockBench(benchmark::Bench& bench)
    57     const auto testing_setup{MakeNoLogFileContext<const TestingSetup>(ChainType::MAIN)};
    58     auto& blockman{testing_setup->m_node.chainman->m_blockman};
    59     const auto pos{blockman.WriteBlock(CreateTestBlock(), 413'567)};
    60-    std::vector<std::byte> block_data;
    61-    blockman.ReadRawBlock(block_data, pos); // warmup
    62+    assert(blockman.ReadRawBlock(pos)); // warmup
    


    l0rinc commented at 10:55 am on December 9, 2025:
    does this still warm up anything? I think we can safely delete this line now…

    romanz commented at 3:51 pm on December 10, 2025:
    Dropped in 1be1c165b3
  160. in doc/REST-interface.md:50 in 262f4dfe69
    46@@ -47,6 +47,11 @@ The HTTP request and response are both handled entirely in-memory.
    47 
    48 With the /notxdetails/ option JSON response will only contain the transaction hash instead of the complete transaction details. The option only affects the JSON response.
    49 
    50+- `GET /rest/blockpart/<BLOCK-HASH>.<bin|hex>?offset=X&size=Y`
    


    l0rinc commented at 11:03 am on December 9, 2025:

    if we’re writing <BLOCK-HASH> and not just BLOCK-HASH, we should so the same for X and Y` (which can also be reworded for consistency, similarly to the PRs description):

    0- `GET /rest/blockpart/<BLOCK-HASH>.<bin|hex>?offset=<OFFSET>&size=<SIZE>`
    1- ```
    

    romanz commented at 3:50 pm on December 10, 2025:
    Fixed in 8b417087ae
  161. in src/rest.cpp:486 in 262f4dfe69 outdated
    480@@ -465,6 +481,25 @@ static bool rest_block_notxdetails(const std::any& context, HTTPRequest* req, co
    481     return rest_block(context, req, uri_part, TxVerbosity::SHOW_TXID);
    482 }
    483 
    484+static bool rest_block_part(const std::any& context, HTTPRequest* req, const std::string& uri_part)
    485+{
    486+    try {
    487+        if (const auto opt_offset{ToIntegral<size_t>(req->GetQueryParameter("offset").value_or(""))}) {
    488+            if (const auto opt_size{ToIntegral<size_t>(req->GetQueryParameter("size").value_or(""))}) {
    489+                return rest_block(context, req, uri_part,
    


    l0rinc commented at 11:07 am on December 9, 2025:
    I like this new version, it’s very compact
  162. l0rinc approved
  163. l0rinc commented at 11:48 am on December 9, 2025: contributor

    Changes since my last review:

    • ReadRawBlock takes a single optional param now, all call sites have been updated, there’s a single method now, the change is a lot simpler and cleaner this way \:D/
    • block_part case in ReadRawBlock is completely isolated from the rest of the method.
    • Now that both params are needed, the partial argument tests were correctly udpated
    • blockmanager_block_data_part was updated to use brace init and stricter BOOST_REQUIRE instead of BOOST_CHECK and was deduplicated and split as requested.
    • previous std::variant was changed to the recently merged std::expected
    • rest_block and ReadRawBlock now take default values which simplifies the diff.
    • rest_block_part is simplified a lot now, I like this version.

    Left a few remaining suggestions (no warming needed in bench, numeric_limits::max test, rest API doc), but I’m fine with this version as well. I know the PR still has to be rebased, but I find this version very easy to review, makes me want to review your other work as well :) Happy to reack on other changes.

    ACK 262f4dfe691b12171a31c9219ba33f1fe55f67a7

      0diff --git a/src/bench/readwriteblock.cpp b/src/bench/readwriteblock.cpp
      1index 4ca5b8ebe2..3a9a0204eb 100644
      2--- a/src/bench/readwriteblock.cpp
      3+++ b/src/bench/readwriteblock.cpp
      4@@ -57,11 +57,10 @@ static void ReadRawBlockBench(benchmark::Bench& bench)
      5     const auto testing_setup{MakeNoLogFileContext<const TestingSetup>(ChainType::MAIN)};
      6     auto& blockman{testing_setup->m_node.chainman->m_blockman};
      7     const auto pos{blockman.WriteBlock(CreateTestBlock(), 413'567)};
      8-    std::vector<std::byte> block_data;
      9-    blockman.ReadRawBlock(block_data, pos); // warmup
     10+    assert(blockman.ReadRawBlock(pos)); // warmup
     11     bench.run([&] {
     12-        const auto success{blockman.ReadRawBlock(block_data, pos)};
     13-        assert(success);
     14+        const auto res{blockman.ReadRawBlock(pos)};
     15+        assert(res);
     16     });
     17 }
     18
     19diff --git a/src/init.cpp b/src/init.cpp
     20index bfb9483ad8..25ceecbb02 100644
     21--- a/src/init.cpp
     22+++ b/src/init.cpp
     23@@ -1759,7 +1759,11 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
     24     g_zmq_notification_interface = CZMQNotificationInterface::Create(
     25         [&chainman = node.chainman](std::vector<std::byte>& block, const CBlockIndex& index) {
     26             assert(chainman);
     27-            return chainman->m_blockman.ReadRawBlock(block, WITH_LOCK(cs_main, return index.GetBlockPos()));
     28+            if (auto ret{chainman->m_blockman.ReadRawBlock(WITH_LOCK(cs_main, return index.GetBlockPos()))}) {
     29+                block = std::move(*ret);
     30+                return true;
     31+            }
     32+            return false;
     33         });
     34
     35     if (g_zmq_notification_interface) {
     36diff --git a/src/net_processing.cpp b/src/net_processing.cpp
     37index 34749c9680..6c28f5d7e3 100644
     38--- a/src/net_processing.cpp
     39+++ b/src/net_processing.cpp
     40@@ -2276,8 +2276,9 @@ void PeerManagerImpl::ProcessGetBlockData(CNode& pfrom, Peer& peer, const CInv&
     41     } else if (inv.IsMsgWitnessBlk()) {
     42         // Fast-path: in this case it is possible to serve the block directly from disk,
     43         // as the network format matches the format on disk
     44-        std::vector<std::byte> block_data;
     45-        if (!m_chainman.m_blockman.ReadRawBlock(block_data, block_pos)) {
     46+        if (const auto block_data{m_chainman.m_blockman.ReadRawBlock(block_pos)}) {
     47+            MakeAndPushMessage(pfrom, NetMsgType::BLOCK, std::span{*block_data});
     48+        } else {
     49             if (WITH_LOCK(m_chainman.GetMutex(), return m_chainman.m_blockman.IsBlockPruned(*pindex))) {
     50                 LogDebug(BCLog::NET, "Block was pruned before it could be read, %s\n", pfrom.DisconnectMsg(fLogIPs));
     51             } else {
     52@@ -2286,7 +2287,6 @@ void PeerManagerImpl::ProcessGetBlockData(CNode& pfrom, Peer& peer, const CInv&
     53             pfrom.fDisconnect = true;
     54             return;
     55         }
     56-        MakeAndPushMessage(pfrom, NetMsgType::BLOCK, std::span{block_data});
     57         // Don't set pblock as we've sent the block
     58     } else {
     59         // Send block from disk
     60diff --git a/src/node/blockstorage.cpp b/src/node/blockstorage.cpp
     61index 85bf729feb..24d203cd51 100644
     62--- a/src/node/blockstorage.cpp
     63+++ b/src/node/blockstorage.cpp
     64@@ -43,7 +43,6 @@
     65 #include <map>
     66 #include <optional>
     67 #include <unordered_map>
     68-#include <variant>
     69
     70 namespace kernel {
     71 static constexpr uint8_t DB_BLOCK_FILES{'f'};
     72@@ -1007,14 +1006,14 @@ bool BlockManager::ReadBlock(CBlock& block, const FlatFilePos& pos, const std::o
     73     block.SetNull();
     74
     75     // Open history file to read
     76-    std::vector<std::byte> block_data;
     77-    if (!ReadRawBlock(block_data, pos)) {
     78+    const auto block_data{ReadRawBlock(pos)};
     79+    if (!block_data) {
     80         return false;
     81     }
     82
     83     try {
     84         // Read block
     85-        SpanReader{block_data} >> TX_WITH_WITNESS(block);
     86+        SpanReader{*block_data} >> TX_WITH_WITNESS(block);
     87     } catch (const std::exception& e) {
     88         LogError("Deserialize or I/O error - %s at %s while reading block", e.what(), pos.ToString());
     89         return false;
     90@@ -1049,29 +1048,19 @@ bool BlockManager::ReadBlock(CBlock& block, const CBlockIndex& index) const
     91     return ReadBlock(block, block_pos, index.GetBlockHash());
     92 }
     93
     94-bool BlockManager::ReadRawBlock(std::vector<std::byte>& block, const FlatFilePos& pos) const
     95-{
     96-    auto ret{ReadRawBlock(pos, /*part_offset=*/0, /*part_size=*/std::nullopt)};
     97-    if (auto* vec{std::get_if<std::vector<std::byte>>(&ret)}) {
     98-        block = std::move(*vec);
     99-        return true;
    100-    }
    101-    return false;
    102-}
    103-
    104-std::variant<std::vector<std::byte>, ReadRawError> BlockManager::ReadRawBlock(const FlatFilePos& pos, size_t part_offset, std::optional<size_t> part_size) const
    105+BlockManager::ReadRawBlockResult BlockManager::ReadRawBlock(const FlatFilePos& pos, std::optional<std::pair<size_t, size_t>> block_part) const
    106 {
    107     if (pos.nPos < STORAGE_HEADER_BYTES) {
    108         // If nPos is less than STORAGE_HEADER_BYTES, we can't read the header that precedes the block data
    109         // This would cause an unsigned integer underflow when trying to position the file cursor
    110         // This can happen after pruning or default constructed positions
    111         LogError("Failed for %s while reading raw block storage header", pos.ToString());
    112-        return ReadRawError::IO;
    113+        return util::Unexpected(ReadRawError::IO);
    114     }
    115     AutoFile filein{OpenBlockFile({pos.nFile, pos.nPos - STORAGE_HEADER_BYTES}, /*fReadOnly=*/true)};
    116     if (filein.IsNull()) {
    117         LogError("OpenBlockFile failed for %s while reading raw block", pos.ToString());
    118-        return ReadRawError::IO;
    119+        return util::Unexpected(ReadRawError::IO);
    120     }
    121
    122     try {
    123@@ -1083,39 +1072,31 @@ std::variant<std::vector<std::byte>, ReadRawError> BlockManager::ReadRawBlock(co
    124         if (blk_start != GetParams().MessageStart()) {
    125             LogError("Block magic mismatch for %s: %s versus expected %s while reading raw block",
    126                 pos.ToString(), HexStr(blk_start), HexStr(GetParams().MessageStart()));
    127-            return ReadRawError::IO;
    128+            return util::Unexpected(ReadRawError::IO);
    129         }
    130
    131         if (blk_size > MAX_SIZE) {
    132             LogError("Block data is larger than maximum deserialization size for %s: %s versus %s while reading raw block",
    133                 pos.ToString(), blk_size, MAX_SIZE);
    134-            return ReadRawError::IO;
    135-        }
    136-
    137-        if (part_offset > blk_size) {
    138-            // Avoid logging - part_offset can come from an untrusted source (REST)
    139-            return ReadRawError::BadPartRange;
    140+            return util::Unexpected(ReadRawError::IO);
    141         }
    142
    143-        size_t size = blk_size - part_offset;
    144-        if (part_size.has_value()) {
    145-            if (*part_size > size || *part_size == 0) {
    146-                // Avoid logging - part_offset & part_size can come from an untrusted source (REST)
    147-                return ReadRawError::BadPartRange;
    148+        if (block_part) {
    149+            const auto [offset, size]{*block_part};
    150+            if (size == 0 || offset >= blk_size || size > blk_size - offset) {
    151+                return util::Unexpected(ReadRawError::BadPartRange); // Avoid logging - offset/size come from untrusted REST input
    152             }
    153-            size = *part_size;
    154+            filein.seek(offset, SEEK_CUR);
    155+            blk_size = size;
    156         }
    157
    158-        std::vector<std::byte> data;
    159-        data.resize(size); // Zeroing of memory is intentional here
    160-        if (part_offset > 0) filein.seek(part_offset, SEEK_CUR);
    161+        std::vector<std::byte> data(blk_size); // Zeroing of memory is intentional here
    162         filein.read(data);
    163         return data;
    164     } catch (const std::exception& e) {
    165         LogError("Read from block file failed: %s for %s while reading raw block", e.what(), pos.ToString());
    166-        return ReadRawError::IO;
    167+        return util::Unexpected(ReadRawError::IO);
    168     }
    169-    assert(false);
    170 }
    171
    172 FlatFilePos BlockManager::WriteBlock(const CBlock& block, int nHeight)
    173diff --git a/src/node/blockstorage.h b/src/node/blockstorage.h
    174index 477983a694..e3f9c445ea 100644
    175--- a/src/node/blockstorage.h
    176+++ b/src/node/blockstorage.h
    177@@ -17,6 +17,7 @@
    178 #include <streams.h>
    179 #include <sync.h>
    180 #include <uint256.h>
    181+#include <util/expected.h>
    182 #include <util/fs.h>
    183 #include <util/hasher.h>
    184
    185@@ -306,6 +307,7 @@ private:
    186
    187 public:
    188     using Options = kernel::BlockManagerOpts;
    189+    using ReadRawBlockResult = util::Expected<std::vector<std::byte>, ReadRawError>;
    190
    191     explicit BlockManager(const util::SignalInterrupt& interrupt, Options opts);
    192
    193@@ -459,8 +461,7 @@ public:
    194     /** Functions for disk access for blocks */
    195     bool ReadBlock(CBlock& block, const FlatFilePos& pos, const std::optional<uint256>& expected_hash) const;
    196     bool ReadBlock(CBlock& block, const CBlockIndex& index) const;
    197-    bool ReadRawBlock(std::vector<std::byte>& block, const FlatFilePos& pos) const;
    198-    std::variant<std::vector<std::byte>, ReadRawError> ReadRawBlock(const FlatFilePos& pos, size_t part_offset, std::optional<size_t> part_size) const;
    199+    ReadRawBlockResult ReadRawBlock(const FlatFilePos& pos, std::optional<std::pair<size_t, size_t>> block_part = std::nullopt) const;
    200
    201     bool ReadBlockUndo(CBlockUndo& blockundo, const CBlockIndex& index) const;
    202
    203diff --git a/src/rest.cpp b/src/rest.cpp
    204index 669da54b98..92fd747f01 100644
    205--- a/src/rest.cpp
    206+++ b/src/rest.cpp
    207@@ -32,7 +32,6 @@
    208 #include <validation.h>
    209
    210 #include <any>
    211-#include <variant>
    212 #include <vector>
    213
    214 #include <univalue.h>
    215@@ -390,8 +389,7 @@ static bool rest_block(const std::any& context,
    216                        HTTPRequest* req,
    217                        const std::string& uri_part,
    218                        std::optional<TxVerbosity> tx_verbosity,
    219-                       size_t part_offset,
    220-                       std::optional<size_t> part_size)
    221+                       std::optional<std::pair<size_t, size_t>> block_part = std::nullopt)
    222 {
    223     if (!CheckWarmup(req))
    224         return false;
    225@@ -425,45 +423,46 @@ static bool rest_block(const std::any& context,
    226         pos = pblockindex->GetBlockPos();
    227     }
    228
    229-    const auto block_result{chainman.m_blockman.ReadRawBlock(pos, part_offset, part_size)};
    230-    if (const auto* error{std::get_if<node::ReadRawError>(&block_result)}) {
    231-        switch (*error) {
    232-        case node::ReadRawError::IO:
    233-            return RESTERR(req, HTTP_NOT_FOUND, "I/O error reading " + hashStr);
    234-
    235+    const auto block_data{chainman.m_blockman.ReadRawBlock(pos, block_part)};
    236+    if (!block_data) {
    237+        switch (block_data.error()) {
    238+        case node::ReadRawError::IO: return RESTERR(req, HTTP_NOT_FOUND, "I/O error reading " + hashStr);
    239         case node::ReadRawError::BadPartRange:
    240-            return RESTERR(req, HTTP_NOT_FOUND, strprintf("Bad block part offset/size %d/%d for %s", part_offset, part_size.value_or(0), hashStr));
    241+            if (block_part) {
    242+                return RESTERR(req, HTTP_NOT_FOUND, strprintf("Bad block part offset/size %d/%d for %s", block_part->first, block_part->second, hashStr));
    243+            } else {
    244+                return RESTERR(req, HTTP_NOT_FOUND, strprintf("Empty block part for %s", hashStr));
    245+            }
    246+        default: assert(false);
    247         }
    248-        assert(false);
    249     }
    250-    const auto& block_data{std::get<std::vector<std::byte>>(block_result)};
    251
    252     switch (rf) {
    253     case RESTResponseFormat::BINARY: {
    254         req->WriteHeader("Content-Type", "application/octet-stream");
    255-        req->WriteReply(HTTP_OK, block_data);
    256+        req->WriteReply(HTTP_OK, *block_data);
    257         return true;
    258     }
    259
    260     case RESTResponseFormat::HEX: {
    261-        const std::string strHex{HexStr(block_data) + "\n"};
    262+        const std::string strHex{HexStr(*block_data) + "\n"};
    263         req->WriteHeader("Content-Type", "text/plain");
    264         req->WriteReply(HTTP_OK, strHex);
    265         return true;
    266     }
    267
    268     case RESTResponseFormat::JSON: {
    269-        if (!tx_verbosity.has_value()) {
    270-            return RESTERR(req, HTTP_BAD_REQUEST, "JSON output is not supported for this request type");
    271+        if (tx_verbosity) {
    272+            CBlock block{};
    273+            DataStream block_stream{*block_data};
    274+            block_stream >> TX_WITH_WITNESS(block);
    275+            UniValue objBlock = blockToJSON(chainman.m_blockman, block, *tip, *pblockindex, *tx_verbosity, chainman.GetConsensus().powLimit);
    276+            std::string strJSON = objBlock.write() + "\n";
    277+            req->WriteHeader("Content-Type", "application/json");
    278+            req->WriteReply(HTTP_OK, strJSON);
    279+            return true;
    280         }
    281-        CBlock block{};
    282-        DataStream block_stream{block_data};
    283-        block_stream >> TX_WITH_WITNESS(block);
    284-        UniValue objBlock = blockToJSON(chainman.m_blockman, block, *tip, *pblockindex, *tx_verbosity, chainman.GetConsensus().powLimit);
    285-        std::string strJSON = objBlock.write() + "\n";
    286-        req->WriteHeader("Content-Type", "application/json");
    287-        req->WriteReply(HTTP_OK, strJSON);
    288-        return true;
    289+        return RESTERR(req, HTTP_BAD_REQUEST, "JSON output is not supported for this request type");
    290     }
    291
    292     default: {
    293@@ -474,39 +473,31 @@ static bool rest_block(const std::any& context,
    294
    295 static bool rest_block_extended(const std::any& context, HTTPRequest* req, const std::string& uri_part)
    296 {
    297-    return rest_block(context, req, uri_part, TxVerbosity::SHOW_DETAILS_AND_PREVOUT, /*part_offset=*/0, /*part_size=*/std::nullopt);
    298+    return rest_block(context, req, uri_part, TxVerbosity::SHOW_DETAILS_AND_PREVOUT);
    299 }
    300
    301 static bool rest_block_notxdetails(const std::any& context, HTTPRequest* req, const std::string& uri_part)
    302 {
    303-    return rest_block(context, req, uri_part, TxVerbosity::SHOW_TXID, /*part_offset=*/0, /*part_size=*/std::nullopt);
    304+    return rest_block(context, req, uri_part, TxVerbosity::SHOW_TXID);
    305 }
    306
    307 static bool rest_block_part(const std::any& context, HTTPRequest* req, const std::string& uri_part)
    308 {
    309-    size_t part_offset{0};
    310-    std::optional<size_t> part_size{};
    311     try {
    312-        const auto part_offset_str = req->GetQueryParameter("offset");
    313-        if (part_offset_str.has_value()) {
    314-            auto opt = ToIntegral<size_t>(*part_offset_str);
    315-            if (opt.has_value()) {
    316-                part_offset = opt.value();
    317+        if (const auto opt_offset{ToIntegral<size_t>(req->GetQueryParameter("offset").value_or(""))}) {
    318+            if (const auto opt_size{ToIntegral<size_t>(req->GetQueryParameter("size").value_or(""))}) {
    319+                return rest_block(context, req, uri_part,
    320+                                  /*tx_verbosity=*/std::nullopt,
    321+                                  /*block_part=*/{{*opt_offset, *opt_size}});
    322             } else {
    323-                return RESTERR(req, HTTP_BAD_REQUEST, strprintf("Block part offset is invalid: %s", *part_offset_str));
    324-            }
    325-        }
    326-        const auto part_size_str = req->GetQueryParameter("size");
    327-        if (part_size_str.has_value()) {
    328-            part_size = ToIntegral<size_t>(*part_size_str);
    329-            if (!part_size.has_value()) {
    330-                return RESTERR(req, HTTP_BAD_REQUEST, strprintf("Block part size is invalid: %s", *part_size_str));
    331+                return RESTERR(req, HTTP_BAD_REQUEST, "Block part size missing or invalid");
    332             }
    333+        } else {
    334+            return RESTERR(req, HTTP_BAD_REQUEST, "Block part offset missing or invalid");
    335         }
    336     } catch (const std::runtime_error& e) {
    337         return RESTERR(req, HTTP_BAD_REQUEST, e.what());
    338     }
    339-    return rest_block(context, req, uri_part, /*tx_verbosity=*/std::nullopt, part_offset, part_size);
    340 }
    341
    342 static bool rest_filter_header(const std::any& context, HTTPRequest* req, const std::string& uri_part)
    343diff --git a/src/rpc/blockchain.cpp b/src/rpc/blockchain.cpp
    344index 8f283ff4f0..40761d5cab 100644
    345--- a/src/rpc/blockchain.cpp
    346+++ b/src/rpc/blockchain.cpp
    347@@ -680,7 +680,6 @@ static CBlock GetBlockChecked(BlockManager& blockman, const CBlockIndex& blockin
    348
    349 static std::vector<std::byte> GetRawBlockChecked(BlockManager& blockman, const CBlockIndex& blockindex)
    350 {
    351-    std::vector<std::byte> data{};
    352     FlatFilePos pos{};
    353     {
    354         LOCK(cs_main);
    355@@ -688,13 +687,10 @@ static std::vector<std::byte> GetRawBlockChecked(BlockManager& blockman, const C
    356         pos = blockindex.GetBlockPos();
    357     }
    358
    359-    if (!blockman.ReadRawBlock(data, pos)) {
    360-        // Block not found on disk. This shouldn't normally happen unless the block was
    361-        // pruned right after we released the lock above.
    362-        throw JSONRPCError(RPC_MISC_ERROR, "Block not found on disk");
    363-    }
    364-
    365-    return data;
    366+    if (const auto data{blockman.ReadRawBlock(pos)}) return *data;
    367+    // Block not found on disk. This shouldn't normally happen unless the block was
    368+    // pruned right after we released the lock above.
    369+    throw JSONRPCError(RPC_MISC_ERROR, "Block not found on disk");
    370 }
    371
    372 static CBlockUndo GetUndoChecked(BlockManager& blockman, const CBlockIndex& blockindex)
    373diff --git a/src/test/blockmanager_tests.cpp b/src/test/blockmanager_tests.cpp
    374index 06fe021dc0..3080ad080d 100644
    375--- a/src/test/blockmanager_tests.cpp
    376+++ b/src/test/blockmanager_tests.cpp
    377@@ -141,68 +141,63 @@ BOOST_FIXTURE_TEST_CASE(blockmanager_block_data_availability, TestChain100Setup)
    378 BOOST_FIXTURE_TEST_CASE(blockmanager_block_data_part, TestChain100Setup)
    379 {
    380     LOCK(::cs_main);
    381-    auto& chainman = m_node.chainman;
    382-    auto& blockman = chainman->m_blockman;
    383-    const CBlockIndex& tip = *chainman->ActiveTip();
    384-    const FlatFilePos tip_block_pos = tip.GetBlockPos();
    385-
    386-    std::vector<std::byte> block{};
    387-    BOOST_CHECK(blockman.ReadRawBlock(block, tip_block_pos));
    388-    BOOST_CHECK_GE(block.size(), 200);
    389-
    390-    std::vector<std::byte> block_part{};
    391-    const auto read_tip_part{[&](auto part_offset, auto part_size) {
    392-        block_part.clear();
    393-        auto block_result{blockman.ReadRawBlock(tip_block_pos, part_offset, part_size)};
    394-        if (auto* vec{std::get_if<std::vector<std::byte>>(&block_result)}) {
    395-            block_part = std::move(*vec);
    396-            return true;
    397-        }
    398-        return false;
    399+    auto& chainman{m_node.chainman};
    400+    auto& blockman{chainman->m_blockman};
    401+    const CBlockIndex& tip{*chainman->ActiveTip()};
    402+    const FlatFilePos tip_block_pos{tip.GetBlockPos()};
    403+
    404+    auto block{blockman.ReadRawBlock(tip_block_pos)};
    405+    BOOST_REQUIRE(block);
    406+    BOOST_REQUIRE_GE(block->size(), 200);
    407+
    408+    const auto expect_part{[&](size_t offset, size_t size) {
    409+        auto res{blockman.ReadRawBlock(tip_block_pos, std::pair{offset, size})};
    410+        BOOST_REQUIRE(res);
    411+        const auto& part{res.value()};
    412+        BOOST_CHECK_EQUAL_COLLECTIONS(part.begin(), part.end(), block->begin() + offset, block->begin() + offset + size);
    413     }};
    414
    415-    BOOST_CHECK(read_tip_part(0, std::nullopt));
    416-    BOOST_CHECK_EQUAL_COLLECTIONS(block_part.begin(), block_part.end(), block.begin(), block.end());
    417-
    418-    BOOST_CHECK(read_tip_part(1, std::nullopt));
    419-    BOOST_CHECK_EQUAL_COLLECTIONS(block_part.begin(), block_part.end(), block.begin() + 1, block.end());
    420-
    421-    BOOST_CHECK(read_tip_part(10, std::nullopt));
    422-    BOOST_CHECK_EQUAL_COLLECTIONS(block_part.begin(), block_part.end(), block.begin() + 10, block.end());
    423-
    424-    BOOST_CHECK(read_tip_part(0, block.size()));
    425-    BOOST_CHECK_EQUAL_COLLECTIONS(block_part.begin(), block_part.end(), block.begin(), block.end());
    426-
    427-    BOOST_CHECK(read_tip_part(0, block.size() - 1));
    428-    BOOST_CHECK_EQUAL_COLLECTIONS(block_part.begin(), block_part.end(), block.begin(), block.end() - 1);
    429-
    430-    BOOST_CHECK(read_tip_part(0, block.size() - 10));
    431-    BOOST_CHECK_EQUAL_COLLECTIONS(block_part.begin(), block_part.end(), block.begin(), block.end() - 10);
    432-
    433-    BOOST_CHECK(read_tip_part(0, 20));
    434-    BOOST_CHECK_EQUAL_COLLECTIONS(block_part.begin(), block_part.end(), block.begin(), block.begin() + 20);
    435-
    436-    BOOST_CHECK(read_tip_part(1, block.size() - 1));
    437-    BOOST_CHECK_EQUAL_COLLECTIONS(block_part.begin(), block_part.end(), block.begin() + 1, block.end());
    438-
    439-    BOOST_CHECK(read_tip_part(10, 20));
    440-    BOOST_CHECK_EQUAL_COLLECTIONS(block_part.begin(), block_part.end(), block.begin() + 10, block.begin() + 30);
    441+    expect_part(0, block->size());
    442+    expect_part(1, block->size() - 1);
    443+    expect_part(10, 20);
    444+    expect_part(0, block->size());
    445+    expect_part(0, block->size() - 1);
    446+    expect_part(0, block->size() - 10);
    447+    expect_part(0, 20);
    448+    expect_part(1, block->size() - 1);
    449+    expect_part(10, 20);
    450+    expect_part(block->size() - 1, 1);
    451+}
    452
    453-    BOOST_CHECK(read_tip_part(block.size() - 1, 1));
    454-    BOOST_CHECK_EQUAL_COLLECTIONS(block_part.begin(), block_part.end(), block.end() - 1, block.end());
    455+BOOST_FIXTURE_TEST_CASE(blockmanager_block_data_part_error, TestChain100Setup)
    456+{
    457+    LOCK(::cs_main);
    458+    auto& chainman{m_node.chainman};
    459+    auto& blockman{chainman->m_blockman};
    460+    const CBlockIndex& tip{*chainman->ActiveTip()};
    461+    const FlatFilePos tip_block_pos{tip.GetBlockPos()};
    462+
    463+    auto block{blockman.ReadRawBlock(tip_block_pos)};
    464+    BOOST_REQUIRE(block);
    465+    BOOST_REQUIRE_GE(block->size(), 200);
    466+
    467+    const auto expect_part_error{[&](size_t offset, size_t size) {
    468+        auto res{blockman.ReadRawBlock(tip_block_pos, std::pair{offset, size})};
    469+        BOOST_CHECK(!res);
    470+        BOOST_CHECK_EQUAL(res.error(), node::ReadRawError::BadPartRange);
    471+    }};
    472
    473-    // check errors
    474-    BOOST_CHECK(!read_tip_part(0, 0));
    475-    BOOST_CHECK(!read_tip_part(0, block.size() + 1));
    476-    BOOST_CHECK(!read_tip_part(1, block.size()));
    477-    BOOST_CHECK(!read_tip_part(2, block.size() - 1));
    478-    BOOST_CHECK(!read_tip_part(block.size() - 2, 3));
    479-    BOOST_CHECK(!read_tip_part(block.size() - 1, 2));
    480-    BOOST_CHECK(!read_tip_part(block.size(), 0));
    481-    BOOST_CHECK(!read_tip_part(block.size(), 1));
    482-    BOOST_CHECK(!read_tip_part(block.size() + 1, 0));
    483-    BOOST_CHECK(!read_tip_part(block.size() + 1, 1));
    484-    BOOST_CHECK(!read_tip_part(block.size() + 2, 2));
    485+    expect_part_error(0, 0);
    486+    expect_part_error(0, block->size() + 1);
    487+    expect_part_error(1, block->size());
    488+    expect_part_error(2, block->size() - 1);
    489+    expect_part_error(block->size() - 2, 3);
    490+    expect_part_error(block->size() - 1, 2);
    491+    expect_part_error(block->size(), 0);
    492+    expect_part_error(block->size(), 1);
    493+    expect_part_error(block->size() + 1, 0);
    494+    expect_part_error(block->size() + 1, 1);
    495+    expect_part_error(block->size() + 2, 2);
    496 }
    497
    498 BOOST_FIXTURE_TEST_CASE(blockmanager_readblock_hash_mismatch, TestingSetup)
    499diff --git a/test/functional/interface_rest.py b/test/functional/interface_rest.py
    500index e7dff20239..32254535ce 100755
    501--- a/test/functional/interface_rest.py
    502+++ b/test/functional/interface_rest.py
    503@@ -470,17 +470,14 @@ class RESTTest (BitcoinTestFramework):
    504                     resp = bytes.fromhex(resp.decode().strip())
    505                 return resp
    506
    507-            assert_equal(block_bin, _get_block_part())
    508-            assert_equal(block_bin, _get_block_part(query_params={"offset": 0}))
    509-            assert_equal(block_bin, _get_block_part(query_params={"size": len(block_bin)}))
    510             assert_equal(block_bin, _get_block_part(query_params={"offset": 0, "size": len(block_bin)}))
    511
    512             assert len(block_bin) >= 500
    513-            assert_equal(block_bin[10:], _get_block_part(query_params={"offset": 10}))
    514-            assert_equal(block_bin[:100], _get_block_part(query_params={"size": 100}))
    515             assert_equal(block_bin[20:320], _get_block_part(query_params={"offset": 20, "size": 300}))
    516             assert_equal(block_bin[-5:], _get_block_part(query_params={"offset": len(block_bin) - 5, "size": 5}))
    517
    518+            _get_block_part(status=400, query_params={"offset": 10})
    519+            _get_block_part(status=400, query_params={"size": 100})
    520             _get_block_part(status=400, query_params={"offset": "x"})
    521             _get_block_part(status=400, query_params={"size": "y"})
    522             _get_block_part(status=400, query_params={"offset": "x", "size": "y"})
    523@@ -490,8 +487,8 @@ class RESTTest (BitcoinTestFramework):
    524             _get_block_part(status=404, query_params={"offset": len(block_bin), "size": 0})
    525             _get_block_part(status=404, query_params={"offset": len(block_bin) + 1, "size": 1})
    526             _get_block_part(status=404, query_params={"offset": len(block_bin), "size": 1})
    527-            _get_block_part(status=404, query_params={"offset": len(block_bin) + 1})
    528-            _get_block_part(status=404, query_params={"size": len(block_bin) + 1})
    529+            _get_block_part(status=404, query_params={"offset": len(block_bin) + 1, "size": 1})
    530+            _get_block_part(status=404, query_params={"offset": 0, "size": len(block_bin) + 1})
    531
    532         self.test_rest_request(f"/blockpart/{blockhash}", status=400, req_type=ReqType.JSON, ret_type=RetType.OBJ)
    

    Sounds good - will it be OK to write it as a Rust CLI tool to interact with bitcoind?

    This was a bit slow, reimplemented it in multithreaded Python, it successfully validated the first ~300k blocks so far:

    0...
    1287090 0000000000000000401489a5114dc753058ef77222823a599b730cd5a179aedd ✅ 149845 bytes, 116 txs
    2287093 00000000000000001f2a8974ae934fe70e1bc8037aa45b687b2d5bb17e23f3cd ✅ 55402 bytes, 126 txs
    3287091 0000000000000000d36060aa424ef7b830db264d267a1b69f31d8dca16076553 ✅ 157275 bytes, 347 txs
    4...
    
     0#!/usr/bin/env python3
     1import sys
     2import json
     3import subprocess
     4import urllib.request
     5from concurrent.futures import ThreadPoolExecutor, as_completed
     6
     7DATA_DIR = "/mnt/my_storage/BitcoinData"
     8
     9def rest(path):
    10    with urllib.request.urlopen(f"http://127.0.0.1:8332/rest/{path}") as r:
    11        return r.read()
    12
    13def cli(*args):
    14    return subprocess.run(["./build/bin/bitcoin-cli", f"-datadir={DATA_DIR}", *args],
    15                          capture_output=True, text=True, check=True).stdout.strip()
    16
    17def varint_size(n):
    18    return 1 if n < 0xfd else 3 if n <= 0xffff else 5 if n <= 0xffffffff else 9
    19
    20def check_block(height):
    21    block_hash = rest(f"blockhashbyheight/{height}.hex").decode().strip()
    22    block_bytes = rest(f"block/{block_hash}.bin")
    23    txids = json.loads(rest(f"block/notxdetails/{block_hash}.json"))["tx"]
    24    offset = 80 + varint_size(len(txids))
    25    for i, txid in enumerate(txids):
    26        tx_bytes = bytes.fromhex(cli("getrawtransaction", txid, "false", block_hash))
    27        part = rest(f"blockpart/{block_hash}.bin?offset={offset}&size={len(tx_bytes)}")
    28        assert part == tx_bytes, f"mismatch at {block_hash} tx#{i}"
    29        offset += len(tx_bytes)
    30    assert offset == len(block_bytes), f"size mismatch: {offset} vs {len(block_bytes)}"
    31    return height, block_hash, offset, len(txids)
    32
    33with ThreadPoolExecutor() as pool:
    34    futures = {pool.submit(check_block, h): h for h in range(int(sys.argv[1]), int(sys.argv[2]) + 1)}
    35    for future in as_completed(futures):
    36        height, block_hash, size, tx_count = future.result()
    37        print(f"{height} {block_hash}{size} bytes, {tx_count} txs")
    
  164. DrahtBot requested review from sedited on Dec 9, 2025
  165. DrahtBot requested review from hodlinator on Dec 9, 2025
  166. romanz force-pushed on Dec 9, 2025
  167. DrahtBot added the label CI failed on Dec 9, 2025
  168. DrahtBot commented at 9:51 pm on December 9, 2025: contributor

    🚧 At least one of the CI tasks failed. Task test max 6 ancestor commits: https://github.com/bitcoin/bitcoin/actions/runs/20077800373/job/57596718560 LLM reason (✨ experimental): Build failed due to an unhandled enumeration value BadPartRange in rest.cpp, flagged as an error by -Werror.

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  169. in src/node/blockstorage.h:175 in 0fa70db617 outdated
    171@@ -172,6 +172,7 @@ std::ostream& operator<<(std::ostream& os, const BlockfileCursor& cursor);
    172 
    173 enum class ReadRawError {
    174     IO,
    175+    BadPartRange,
    


    l0rinc commented at 10:48 pm on December 9, 2025:

    If you add the value in 0fa70db617a0f003422f66a0ef634b0af1bbd520, you have to handle it in every switch: https://github.com/bitcoin/bitcoin/actions/runs/20077800373/job/57596718560?pr=33657#step:8:2692 @sedited, are you sure still we need the assert(false) hack, it seems -Wswitch already warns:

    0/home/runner/work/bitcoin/bitcoin/src/rest.cpp:421:17: error: enumeration value 'BadPartRange' not handled in switch [-Werror,-Wswitch]
    1  421 |         switch (block_data.error()) {
    2      |                 ^~~~~~~~~~~~~~~~~~
    31 error generated.
    

    Edit: we should keep the assert for now, we can adjust them all later, if needed….


    romanz commented at 3:40 pm on December 10, 2025:

    Modified blockstorage: allow reading partial block data from storage to fix the issue above:

     0$ git range-diff origin/master 2251ac1 8b41708
     1
     21:  1be1c165b3 = 1:  1be1c165b3 blockstorage: return an error code from `ReadRawBlock()`
     32:  0fa70db617 ! 2:  52ac60d141 blockstorage: allow reading partial block data from storage
     4    @@ src/node/blockstorage.h: public:
     5          bool ReadBlockUndo(CBlockUndo& blockundo, const CBlockIndex& index) const;
     6      
     7     
     8    + ## src/rest.cpp ##
     9    +@@ src/rest.cpp: static bool rest_block(const std::any& context,
    10    +     if (!block_data) {
    11    +         switch (block_data.error()) {
    12    +         case node::ReadRawError::IO: return RESTERR(req, HTTP_NOT_FOUND, "I/O error reading " + hashStr);
    13    ++        case node::ReadRawError::BadPartRange: break; // can happen only when reading a block part
    14    +         }
    15    +         assert(false);
    16    +     }
    17    +
    18      ## src/test/blockmanager_tests.cpp ##
    19     @@ src/test/blockmanager_tests.cpp: BOOST_FIXTURE_TEST_CASE(blockmanager_block_data_availability, TestChain100Setup)
    20          BOOST_CHECK(!blockman.CheckBlockDataAvailability(tip, *last_pruned_block));
    213:  2251ac101f ! 3:  8b417087ae rest: allow reading partial block data from storage
    22    @@ src/rest.cpp: static bool rest_block(const std::any& context,
    23          if (!block_data) {
    24              switch (block_data.error()) {
    25              case node::ReadRawError::IO: return RESTERR(req, HTTP_NOT_FOUND, "I/O error reading " + hashStr);
    26    +-        case node::ReadRawError::BadPartRange: break; // can happen only when reading a block part
    27     +        case node::ReadRawError::BadPartRange:
    28     +            assert(block_part);
    29     +            return RESTERR(req, HTTP_BAD_REQUEST, strprintf("Bad block part offset/size %d/%d for %s", block_part->first, block_part->second, hashStr));
    

    maflcko commented at 5:00 pm on December 11, 2025:

    @sedited, are you sure still we need the assert(false) hack, it seems -Wswitch already warns:

    Yes, the assert(false) is needed, even if the compiler can warn about missing enum class cases. this is, because even with an enum class, there could be unhandled cases. This is not forbidden by the language. So the switch may fall through and lead to UB or unexpected behavior. Generally, we prefer assert(false) over UB


    l0rinc commented at 5:16 pm on December 11, 2025:
    So the -Wswitch wouldn’t catch them? I have checked multiple and they all failed the build if I left out any enum value

    maflcko commented at 5:31 pm on December 11, 2025:

    So the -Wswitch wouldn’t catch them? I have checked multiple and they all failed the build if I left out any enum value

    No. See https://godbolt.org/z/6zGqrcnax

     0#include <iostream>
     1
     2enum class Color : uint8_t { Red, Green, Blue };
     3
     4int to_int(Color c) {
     5    switch (c) {
     6        case Color::Red:
     7            return 1;
     8        case Color::Green:
     9            return 2;
    10        case Color::Blue:
    11            return 3;
    12    }
    13    return -1; // this should never happen?
    14}
    15
    16int main() {
    17    Color c = static_cast<Color>(7);
    18    std::cout << to_int(c) << "\n";
    19}
    

    l0rinc commented at 5:21 pm on December 18, 2025:
    Thanks, I was under the wrong impression this was only the case with bare enums - thanks for the example.
  170. romanz force-pushed on Dec 10, 2025
  171. in src/rest.cpp:429 in 8b417087ae
    427-    if (!chainman.m_blockman.ReadRawBlock(block_data, pos)) {
    428-        return RESTERR(req, HTTP_NOT_FOUND, hashStr + " not found");
    429+    const auto block_data{chainman.m_blockman.ReadRawBlock(pos, block_part)};
    430+    if (!block_data) {
    431+        switch (block_data.error()) {
    432+        case node::ReadRawError::IO: return RESTERR(req, HTTP_NOT_FOUND, "I/O error reading " + hashStr);
    


    hodlinator commented at 3:59 pm on December 10, 2025:

    nit: Sorry for not suggesting earlier, this could possibly be more appropriate:

    0        case node::ReadRawError::IO: return RESTERR(req, HTTP_INTERNAL_SERVER_ERROR, "I/O error reading " + hashStr);
    

    hodlinator commented at 7:45 pm on December 10, 2025:

    Could add a test for this:

     0--- a/test/functional/interface_rest.py
     1+++ b/test/functional/interface_rest.py
     2@@ -9,6 +9,7 @@ from enum import Enum
     3 from io import BytesIO
     4 import http.client
     5 import json
     6+import os
     7 import typing
     8 import urllib.parse
     9 
    10@@ -84,7 +85,7 @@ class RESTTest (BitcoinTestFramework):
    11             conn.request('POST', rest_uri, body)
    12         resp = conn.getresponse()
    13 
    14-        assert_equal(resp.status, status)
    15+        assert resp.status == status, f"Expected: {status}, Got: {resp.status} - Response: {str(resp.read())}"
    16 
    17         if ret_type == RetType.OBJ:
    18             return resp
    19@@ -492,6 +493,11 @@ class RESTTest (BitcoinTestFramework):
    20 
    21         self.test_rest_request(f"/blockpart/{blockhash}", status=400, req_type=ReqType.JSON, ret_type=RetType.OBJ)
    22 
    23+        self.test_rest_request(f"/blockpart/{blockhash}", query_params={"offset": 0, "size": 1}, status=200, req_type=ReqType.BIN, ret_type=RetType.OBJ)
    24+        os.rename(self.nodes[0].blocks_path, self.nodes[0].blocks_path.with_suffix(".bkp"))
    25+        self.test_rest_request(f"/blockpart/{blockhash}", query_params={"offset": 0, "size": 1}, status=404, req_type=ReqType.BIN, ret_type=RetType.OBJ)
    26+        os.rename(self.nodes[0].blocks_path.with_suffix(".bkp"), self.nodes[0].blocks_path)
    27+
    28         self.log.info("Test the /deploymentinfo URI")
    29 
    30         deployment_info = self.nodes[0].getdeploymentinfo()
    


    romanz commented at 10:07 pm on December 10, 2025:

    No worries, thanks :)

     01:  1be1c165b3 ! 1:  7a2a6b398b blockstorage: return an error code from `ReadRawBlock()`
     1    @@ src/rest.cpp: static bool rest_block(const std::any& context,
     2     +    const auto block_data{chainman.m_blockman.ReadRawBlock(pos)};
     3     +    if (!block_data) {
     4     +        switch (block_data.error()) {
     5    -+        case node::ReadRawError::IO: return RESTERR(req, HTTP_NOT_FOUND, "I/O error reading " + hashStr);
     6    ++        case node::ReadRawError::IO: return RESTERR(req, HTTP_INTERNAL_SERVER_ERROR, "I/O error reading " + hashStr);
     7     +        }
     8     +        assert(false);
     9          }
    102:  52ac60d141 ! 2:  f03f9d7dcd blockstorage: allow reading partial block data from storage
    11    @@ src/rest.cpp
    12     @@ src/rest.cpp: static bool rest_block(const std::any& context,
    13          if (!block_data) {
    14              switch (block_data.error()) {
    15    -         case node::ReadRawError::IO: return RESTERR(req, HTTP_NOT_FOUND, "I/O error reading " + hashStr);
    16    +         case node::ReadRawError::IO: return RESTERR(req, HTTP_INTERNAL_SERVER_ERROR, "I/O error reading " + hashStr);
    17     +        case node::ReadRawError::BadPartRange: break; // can happen only when reading a block part
    18              }
    19              assert(false);
    20    @@ src/test/blockmanager_tests.cpp: BOOST_FIXTURE_TEST_CASE(blockmanager_block_data
    21     +
    22     +    const auto expect_part{[&](size_t offset, size_t size) {
    23     +        auto res{blockman.ReadRawBlock(tip_block_pos, std::pair{offset, size})};
    24    -+        BOOST_REQUIRE(res);
    25    ++        BOOST_CHECK(res);
    26     +        const auto& part{res.value()};
    27     +        BOOST_CHECK_EQUAL_COLLECTIONS(part.begin(), part.end(), block->begin() + offset, block->begin() + offset + size);
    28     +    }};
    293:  8b417087ae ! 3:  44758e8387 rest: allow reading partial block data from storage
    30    @@ src/rest.cpp: static bool rest_block(const std::any& context,
    31     +    const auto block_data{chainman.m_blockman.ReadRawBlock(pos, block_part)};
    32          if (!block_data) {
    33              switch (block_data.error()) {
    34    -         case node::ReadRawError::IO: return RESTERR(req, HTTP_NOT_FOUND, "I/O error reading " + hashStr);
    35    +         case node::ReadRawError::IO: return RESTERR(req, HTTP_INTERNAL_SERVER_ERROR, "I/O error reading " + hashStr);
    36     -        case node::ReadRawError::BadPartRange: break; // can happen only when reading a block part
    37     +        case node::ReadRawError::BadPartRange:
    38     +            assert(block_part);
    39    @@ test/functional/interface_rest.py: class RESTTest (BitcoinTestFramework):
    40      
    41              conn = http.client.HTTPConnection(self.url.hostname, self.url.port)
    42              self.log.debug(f'{http_method} {rest_uri} {body}')
    43    +@@ test/functional/interface_rest.py: class RESTTest (BitcoinTestFramework):
    44    +             conn.request('POST', rest_uri, body)
    45    +         resp = conn.getresponse()
    46    + 
    47    +-        assert_equal(resp.status, status)
    48    ++        assert resp.status == status, f"Expected: {status}, Got: {resp.status} - Response: {str(resp.read())}"
    49    + 
    50    +         if ret_type == RetType.OBJ:
    51    +             return resp
    52     @@ test/functional/interface_rest.py: class RESTTest (BitcoinTestFramework):
    53                      expected = [(p["scriptPubKey"], p["value"]) for p in prevouts]
    54                      assert_equal(expected, actual)
    55    @@ test/functional/interface_rest.py: class RESTTest (BitcoinTestFramework):
    56     +            _get_block_part(status=400, query_params={"offset": 0, "size": len(block_bin) + 1})
    57     +
    58     +        self.test_rest_request(f"/blockpart/{blockhash}", status=400, req_type=ReqType.JSON, ret_type=RetType.OBJ)
    59    ++
    60    ++        self.test_rest_request(f"/block/{blockhash}", status=200, req_type=ReqType.BIN, ret_type=RetType.OBJ)
    61    ++        self.test_rest_request(f"/blockpart/{blockhash}", query_params={"offset": 0, "size": 1}, status=200, req_type=ReqType.BIN, ret_type=RetType.OBJ)
    62    ++        # Missing block data should cause REST API to fail
    63    ++        blocks_path = self.nodes[0].blocks_path
    64    ++        backup_path = blocks_path.with_suffix(".bkp")
    65    ++        blocks_path.rename(backup_path)
    66    ++        try:
    67    ++            self.test_rest_request(f"/block/{blockhash}", status=500, req_type=ReqType.BIN, ret_type=RetType.OBJ)
    68    ++            self.test_rest_request(f"/blockpart/{blockhash}", query_params={"offset": 0, "size": 1}, status=500, req_type=ReqType.BIN, ret_type=RetType.OBJ)
    69    ++        finally:
    70    ++            backup_path.rename(blocks_path)
    71      
    72              self.log.info("Test the /deploymentinfo URI")
    

    hodlinator commented at 11:25 pm on December 10, 2025:

    Argh, saw Windows CI failures: https://github.com/bitcoin/bitcoin/actions/runs/20114716290/job/57721194424?pr=33657#step:14:3972

    I guess the OS doesn’t like moving directories with opened files in them. Maybe best to make these checks conditional on the platform (if platform.system() != "Windows":).



    hodlinator commented at 8:30 am on December 11, 2025:

    Not sure about the try/finally. I think it may be better to leave the filesystem as-is in case of failure, so that devs inspecting the directory of a failed test finds it closer to how it looked when the failure happened. Haven’t seen this kind of pattern in other functional tests. We don’t continue the test regardless, so there’s no benefit to checks later in the test.

     0--- a/test/functional/interface_rest.py
     1+++ b/test/functional/interface_rest.py
     2@@ -500,11 +500,9 @@ class RESTTest (BitcoinTestFramework):
     3             blocks_path = self.nodes[0].blocks_path
     4             backup_path = blocks_path.with_suffix(".bkp")
     5             blocks_path.rename(backup_path)
     6-            try:
     7-                self.test_rest_request(f"/block/{blockhash}", status=500, req_type=ReqType.BIN, ret_type=RetType.OBJ)
     8-                self.test_rest_request(f"/blockpart/{blockhash}", query_params={"offset": 0, "size": 1}, status=500, req_type=ReqType.BIN, ret_type=RetType.OBJ)
     9-            finally:
    10-                backup_path.rename(blocks_path)
    11+            self.test_rest_request(f"/block/{blockhash}", status=500, req_type=ReqType.BIN, ret_type=RetType.OBJ)
    12+            self.test_rest_request(f"/blockpart/{blockhash}", query_params={"offset": 0, "size": 1}, status=500, req_type=ReqType.BIN, ret_type=RetType.OBJ)
    13+            backup_path.rename(blocks_path)
    14 
    15         self.log.info("Test the /deploymentinfo URI")
    16 
    

    l0rinc commented at 10:37 am on December 11, 2025:
    I don’t really like this Windows distinction - can we rather find a way that does work on all platforms? Haven’t checked, is it only triggered when the file is missing, or can a truncated file also trigger the error?

    hodlinator commented at 10:49 am on December 11, 2025:

    Yeah, finding a solution that works on all platforms would be even better. Don’t think it’s a case of the file missing, but rather that Windows forbids renaming parent directories when a process holds a file handle to a file.

    Maybe we could rename files inside of the directory matching blk*.dat to blk*.dat.bkp? Haven’t tested yet.


    hodlinator commented at 1:14 pm on December 11, 2025:

    This might do it - an earlier version of it passed Windows CI (https://github.com/hodlinator/bitcoin/actions/runs/20131062861):

     0--- a/test/functional/interface_rest.py
     1+++ b/test/functional/interface_rest.py
     2@@ -7,9 +7,11 @@
     3 from decimal import Decimal
     4 from enum import Enum
     5 from io import BytesIO
     6+from pathlib import Path
     7 import http.client
     8 import json
     9-import platform
    10+import os
    11+import re
    12 import typing
    13 import urllib.parse
    14 
    15@@ -496,15 +498,14 @@ class RESTTest (BitcoinTestFramework):
    16         self.test_rest_request(f"/block/{blockhash}", status=200, req_type=ReqType.BIN, ret_type=RetType.OBJ)
    17         self.test_rest_request(f"/blockpart/{blockhash}", query_params={"offset": 0, "size": 1}, status=200, req_type=ReqType.BIN, ret_type=RetType.OBJ)
    18         # Missing block data should cause REST API to fail
    19-        if platform.system() != "Windows":
    20-            blocks_path = self.nodes[0].blocks_path
    21-            backup_path = blocks_path.with_suffix(".bkp")
    22-            blocks_path.rename(backup_path)
    23-            try:
    24-                self.test_rest_request(f"/block/{blockhash}", status=500, req_type=ReqType.BIN, ret_type=RetType.OBJ)
    25-                self.test_rest_request(f"/blockpart/{blockhash}", query_params={"offset": 0, "size": 1}, status=500, req_type=ReqType.BIN, ret_type=RetType.OBJ)
    26-            finally:
    27-                backup_path.rename(blocks_path)
    28+        for f in os.scandir(self.nodes[0].blocks_path):
    29+            if re.match(r"blk.*\.dat", f.name):
    30+                Path(f.path).rename(f"{f.path}.bkp")
    31+        self.test_rest_request(f"/block/{blockhash}", status=500, req_type=ReqType.BIN, ret_type=RetType.OBJ)
    32+        self.test_rest_request(f"/blockpart/{blockhash}", query_params={"offset": 0, "size": 1}, status=500, req_type=ReqType.BIN, ret_type=RetType.OBJ)
    33+        for f in os.scandir(self.nodes[0].blocks_path):
    34+            if re.match(r"blk.*\.dat\.bkp", f.name):
    35+                Path(f.path).rename(f"{f.path[:-4]}")
    36 
    37         self.log.info("Test the /deploymentinfo URI")
    

    CI in progress: https://github.com/hodlinator/bitcoin/actions/runs/20134156831



    hodlinator commented at 4:28 pm on December 11, 2025:
    Slick! :pinched_fingers:
  172. DrahtBot removed the label CI failed on Dec 10, 2025
  173. in src/test/blockmanager_tests.cpp:183 in 8b417087ae outdated
    181+    BOOST_REQUIRE(block);
    182+    BOOST_REQUIRE_GE(block->size(), 200);
    183+
    184+    const auto expect_part_error{[&](size_t offset, size_t size) {
    185+        auto res{blockman.ReadRawBlock(tip_block_pos, std::pair{offset, size})};
    186+        BOOST_CHECK(!res);
    


    l0rinc commented at 4:51 pm on December 10, 2025:
    nit: the above test makes res a REQUIRE, if you touch agan, consider unifying. Definitely not a blocker.

    romanz commented at 10:08 pm on December 10, 2025:
    Unified in f03f9d7dcd929f73f7b0c7030d9822e54f1621ec

    hodlinator commented at 8:21 am on December 11, 2025:

    nit: In my opinion it would be better to use REQUIRE which stops the test in case of failure, so that we don’t trigger less clear failures on the following lines (internal assert in Expected::value() - might be an exception after #34032, assert inside Expected::error()).

     0--- a/src/test/blockmanager_tests.cpp
     1+++ b/src/test/blockmanager_tests.cpp
     2@@ -152,7 +152,7 @@ BOOST_FIXTURE_TEST_CASE(blockmanager_block_data_part, TestChain100Setup)
     3 
     4     const auto expect_part{[&](size_t offset, size_t size) {
     5         auto res{blockman.ReadRawBlock(tip_block_pos, std::pair{offset, size})};
     6-        BOOST_CHECK(res);
     7+        BOOST_REQUIRE(res);
     8         const auto& part{res.value()};
     9         BOOST_CHECK_EQUAL_COLLECTIONS(part.begin(), part.end(), block->begin() + offset, block->begin() + offset + size);
    10     }};
    11@@ -183,7 +183,7 @@ BOOST_FIXTURE_TEST_CASE(blockmanager_block_data_part_error, TestChain100Setup)
    12 
    13     const auto expect_part_error{[&](size_t offset, size_t size) {
    14         auto res{blockman.ReadRawBlock(tip_block_pos, std::pair{offset, size})};
    15-        BOOST_CHECK(!res);
    16+        BOOST_REQUIRE(!res);
    17         BOOST_CHECK_EQUAL(res.error(), node::ReadRawError::BadPartRange);
    18     }};
    19 
    

    hodlinator commented at 10:16 am on December 11, 2025:
    (Others seem to disagree with me here though: #34032 (review)).

    l0rinc commented at 10:34 am on December 11, 2025:
    I asked the two to be unified, both are fine here
  174. l0rinc commented at 4:53 pm on December 10, 2025: contributor

    ACK 8b417087aec4671a1ce58f2331d1688e665d9935

    The implementation doesn’t slow down the critical path anymore and is beautifully isolated from the rest of the raw block reading. The tests contain the extremes (added since last ACK), docs were update, braces added.

  175. l0rinc approved
  176. hodlinator approved
  177. hodlinator commented at 7:54 pm on December 10, 2025: contributor
    re-ACK 8b417087aec4671a1ce58f2331d1688e665d9935
  178. romanz force-pushed on Dec 10, 2025
  179. l0rinc commented at 10:16 pm on December 10, 2025: contributor

    ACK 44758e8387e66ec49d190b59f8a8fefab84b1e69

    Since last push a HTTP_NOT_FOUND error code was changed to HTTP_INTERNAL_SERVER_ERROR, a test BOOST_REQUIRE was changed to a more permissive BOOST_CHECK, and in functional tests an assert was adjusted and a missing-block-data case was added for the status of the response.

  180. DrahtBot requested review from hodlinator on Dec 10, 2025
  181. DrahtBot added the label CI failed on Dec 11, 2025
  182. romanz force-pushed on Dec 11, 2025
  183. DrahtBot removed the label CI failed on Dec 11, 2025
  184. in src/node/blockstorage.h:310 in d2ffbb5985 outdated
    305@@ -302,6 +306,7 @@ class BlockManager
    306 
    307 public:
    308     using Options = kernel::BlockManagerOpts;
    309+    using ReadRawBlockResult = util::Expected<std::vector<std::byte>, ReadRawError>;
    


    maflcko commented at 1:31 pm on December 11, 2025:
    nit in d2ffbb5985717f9b6348a7aab60fcad6e271a296: This is only used twice. Could remove it, or move it right above the function declaration that needs it?

    l0rinc commented at 4:36 pm on December 11, 2025:
    it’s a mouthful, I personally would prefer keeping it as it is
  185. in src/rest.cpp:433 in 9f8809f2b1 outdated
    431+        switch (block_data.error()) {
    432+        case node::ReadRawError::IO: return RESTERR(req, HTTP_INTERNAL_SERVER_ERROR, "I/O error reading " + hashStr);
    433+        case node::ReadRawError::BadPartRange:
    434+            assert(block_part);
    435+            return RESTERR(req, HTTP_BAD_REQUEST, strprintf("Bad block part offset/size %d/%d for %s", block_part->first, block_part->second, hashStr));
    436+        }
    


    maflcko commented at 1:44 pm on December 11, 2025:
    0        } // no default case, so the compiler can warn about missing cases
    

    nit in the first commit: This is normally done, to explain the code. See git grep 'no default'.


    romanz commented at 6:01 pm on December 11, 2025:
    Added in e4f460bbe44ba11e3997da80daf39cbb2515fefc

    hodlinator commented at 12:24 pm on December 12, 2025:
    (meganit: Would add the comment in the first commit instead of the last if you re-touch).
  186. in src/rest.cpp:429 in 9f8809f2b1 outdated
    427-    if (!chainman.m_blockman.ReadRawBlock(block_data, pos)) {
    428-        return RESTERR(req, HTTP_NOT_FOUND, hashStr + " not found");
    429+    const auto block_data{chainman.m_blockman.ReadRawBlock(pos, block_part)};
    430+    if (!block_data) {
    431+        switch (block_data.error()) {
    432+        case node::ReadRawError::IO: return RESTERR(req, HTTP_INTERNAL_SERVER_ERROR, "I/O error reading " + hashStr);
    


    maflcko commented at 1:49 pm on December 11, 2025:

    nit in the first commit: This is not always an internal server error. It is rarely possible that this is due to a pruning race.

    Not sure if anything needs to change, though. Just wanted to leave the note.

  187. in src/rpc/blockchain.cpp:690 in 9f8809f2b1 outdated
    693-        // pruned right after we released the lock above.
    694-        throw JSONRPCError(RPC_MISC_ERROR, "Block not found on disk");
    695-    }
    696-
    697-    return data;
    698+    if (const auto data{blockman.ReadRawBlock(pos)}) return *data;
    


    maflcko commented at 2:00 pm on December 11, 2025:

    nit in the first commit: i am pretty sure this is wrong and creates a copy now, when previously it did not.

    Ref: https://godbolt.org/z/GT7sdYWxM (see the const)


    l0rinc commented at 4:35 pm on December 11, 2025:

    hodlinator commented at 4:49 pm on December 11, 2025:
    Good point about the copy. If I keep the const and add return std::move(*data) at the end my LSP complains about it having no effect. (My build setup does not). Not sure if removing the const but adding the explicit move would help.

    andrewtoth commented at 5:10 pm on December 11, 2025:
    How can you move out of data if it is const though? The const needs to be removed. I like the approach suggested by @romanz to inline it in the previous discussion.

    maflcko commented at 5:25 pm on December 11, 2025:

    Good point about the copy. If I keep the const and add return std::move(*data) at the end my LSP complains about it having no effect. (My build setup does not). Not sure if removing the const but adding the explicit move would help.

    If the const is kept, the std::move will hopefully fail clang-tidy in the CI.

    The const needs to be removed and an std::move will need to be added. This is what I was trying to show with the godbolt, but maybe it wasn’t clear.


    romanz commented at 6:00 pm on December 11, 2025:
    Changed in f2fd1aa21c7694cef393b4a13e472ae9d3fc54fc
  188. in src/node/blockstorage.cpp:1086 in 9f8809f2b1 outdated
    1083+            return util::Unexpected{ReadRawError::IO};
    1084+        }
    1085+
    1086+        if (block_part) {
    1087+            const auto [offset, size]{*block_part};
    1088+            if (size == 0 || offset >= blk_size || size > blk_size - offset) {
    


    maflcko commented at 2:35 pm on December 11, 2025:
    0            if (size == 0 || SaturatingAdd(offset, size) > blk_size) {
    

    nit in the second commit: Could use a single saturating add and a single > compare?


    maflcko commented at 2:35 pm on December 11, 2025:
    nit: Also seems fine to allow zero-size?

    l0rinc commented at 4:35 pm on December 11, 2025:
    what would be the advantage of zero size?

    maflcko commented at 5:09 pm on December 11, 2025:

    what would be the advantage of zero size?

    idk. It is generally valid to read zero bytes:

    0with open("example.txt", "rb") as f:
    1    data = f.read(0)
    2    print(data)   # prints: b''
    

    i don’t mind either way. I just wanted to mention it, because some non-ideal hacky script may read blocks in ranges and read the last range, even if it is of zero size. yes, the script should be adjusted, but I don’t see the harm in allowing it. In any case, just a nit and not important at all.


    hodlinator commented at 12:31 pm on December 12, 2025:

    nit: Agree on reducing the number of direct conditions by using SaturatingAdd as sugggested at the beginning of the thread (passes current unit tests).

    No strong opinion for/against disallowing 0-size ranges.


    maflcko commented at 12:54 pm on December 12, 2025:

    nit: Agree on reducing the number of direct conditions by using SaturatingAdd as sugggested at the beginning of the thread (passes current unit tests).

    Let’s leave the nits for a follow-up? The thread has more than 250 comments, and at least on my end, I am having difficulties loading, reading, and writing new comments, due to GitHub slowness or brittleness.

  189. in src/test/blockmanager_tests.cpp:144 in 3a24113dea outdated
    137@@ -138,6 +138,71 @@ BOOST_FIXTURE_TEST_CASE(blockmanager_block_data_availability, TestChain100Setup)
    138     BOOST_CHECK(!blockman.CheckBlockDataAvailability(tip, *last_pruned_block));
    139 }
    140 
    141+BOOST_FIXTURE_TEST_CASE(blockmanager_block_data_part, TestChain100Setup)
    142+{
    143+    LOCK(::cs_main);
    144+    auto& chainman{m_node.chainman};
    


    maflcko commented at 2:45 pm on December 11, 2025:

    nit in 3a24113dea228b1bbe14d02c6cab2579d49231fb:

    Could use LOCK(chainman.GetMutex()) instead for new code? (same below)


    maflcko commented at 2:52 pm on December 11, 2025:
    Could also merge the two tests into one, to avoid the boilerplate in the beginning.
  190. in src/test/blockmanager_tests.cpp:163 in 3a24113dea outdated
    158+    }};
    159+
    160+    expect_part(0, block->size());
    161+    expect_part(1, block->size() - 1);
    162+    expect_part(10, 20);
    163+    expect_part(0, block->size());
    


    maflcko commented at 2:51 pm on December 11, 2025:
    nit: Some of these are duplicates?

    romanz commented at 5:59 pm on December 11, 2025:
    Deduplicated tests in 4e2af1c06547230b9245d94e7bcb1129f2c49714
  191. in test/functional/interface_rest.py:465 in 9f8809f2b1 outdated
    457@@ -455,6 +458,53 @@ def run_test(self):
    458                 expected = [(p["scriptPubKey"], p["value"]) for p in prevouts]
    459                 assert_equal(expected, actual)
    460 
    461+        self.log.info("Test the /blockpart URI")
    462+
    463+        blockhash = self.nodes[0].getbestblockhash()
    464+        block_bin = self.test_rest_request(f"/block/{blockhash}", req_type=ReqType.BIN, ret_type=RetType.BYTES)
    465+        for req_type in (ReqType.BIN, ReqType.HEX):
    466+            def _get_block_part(status: int = 200, **kwargs):
    


    maflcko commented at 3:10 pm on December 11, 2025:
    nit in the last commit: What does the _ mean here? Could drop it?

    romanz commented at 5:59 pm on December 11, 2025:
    Dropped in e4f460bbe44ba11e3997da80daf39cbb2515fefc
  192. romanz force-pushed on Dec 11, 2025
  193. in test/functional/interface_rest.py:12 in 6e71622d85
     8@@ -9,6 +9,7 @@
     9 from io import BytesIO
    10 import http.client
    11 import json
    12+import platform
    


    l0rinc commented at 4:03 pm on December 11, 2025:
    nit: this isn’t needed anymore

    romanz commented at 4:06 pm on December 11, 2025:
    Thanks - fixed in 389eafc631733c1ac2890e1b012a94f66a31ccad.
  194. l0rinc approved
  195. romanz force-pushed on Dec 11, 2025
  196. DrahtBot added the label CI failed on Dec 11, 2025
  197. DrahtBot commented at 4:06 pm on December 11, 2025: contributor

    🚧 At least one of the CI tasks failed. Task lint: https://github.com/bitcoin/bitcoin/actions/runs/20139296974/job/57801727809 LLM reason (✨ experimental): Lint check failed: Ruff reports an unused import (platform) in test/interface_rest.py, causing the CI to fail.

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  198. in test/functional/interface_rest.py:497 in 389eafc631
    492+
    493+        self.test_rest_request(f"/blockpart/{blockhash}", status=400, req_type=ReqType.JSON, ret_type=RetType.OBJ)
    494+
    495+        self.test_rest_request(f"/block/{blockhash}", status=200, req_type=ReqType.BIN, ret_type=RetType.OBJ)
    496+        self.test_rest_request(f"/blockpart/{blockhash}", query_params={"offset": 0, "size": 1}, status=200, req_type=ReqType.BIN, ret_type=RetType.OBJ)
    497+        # Missing block data should cause REST API to fail
    


    maflcko commented at 4:08 pm on December 11, 2025:
    nit in the last commit: I didn’t understand that this is a new test case. Could add a self.log.info("Missing block data should cause REST API to fail")?

    romanz commented at 5:59 pm on December 11, 2025:
    Added in e4f460bbe44ba11e3997da80daf39cbb2515fefc
  199. romanz commented at 4:08 pm on December 11, 2025: contributor
    Windows CI jobs fail, probably due to https://www.githubstatus.com/incidents/xntfc1fz5rfb :confused:
  200. maflcko approved
  201. maflcko commented at 4:09 pm on December 11, 2025: member

    looks good, the style nits can be ignored and are not important.

    review ACK 9f8809f2b1206ad34dfc68831c10db76e618de3f 🛳

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: review ACK 9f8809f2b1206ad34dfc68831c10db76e618de3f 🛳
    3NUYr34EtZ+EeDqlvABg/SqtOUoi+6/Wf8HHOwwzPB636/nDHpQsbkGLTmW2Nz8zu27KwzUXxzqojtB5pV3MRAQ==
    
  202. l0rinc commented at 4:39 pm on December 11, 2025: contributor

    ACK 389eafc631733c1ac2890e1b012a94f66a31ccad

    Only test changes since my last review.

    Thanks for your patience and responsiveness, @romanz. Feel free to apply other recommendations as well, happy to reack.

  203. DrahtBot requested review from maflcko on Dec 11, 2025
  204. blockstorage: return an error code from `ReadRawBlock()`
    It will enable different error handling flows for different error types.
    
    Also, `ReadRawBlockBench` performance has decreased due to no longer reusing a vector
    with an unchanging capacity - mirroring our production code behavior.
    
    Co-authored-by: Hodlinator <172445034+hodlinator@users.noreply.github.com>
    Co-authored-by: Lőrinc <pap.lorinc@gmail.com>
    f2fd1aa21c
  205. blockstorage: allow reading partial block data from storage
    It will allow fetching specific transactions using an external index,
    following https://github.com/bitcoin/bitcoin/pull/32541#issuecomment-3267485313.
    
    No logging takes place in case of an invalid offset/size (to avoid spamming the log),
    by using a new `ReadRawError::BadPartRange` error variant.
    
    Co-authored-by: Hodlinator <172445034+hodlinator@users.noreply.github.com>
    Co-authored-by: Lőrinc <pap.lorinc@gmail.com>
    4e2af1c065
  206. romanz force-pushed on Dec 11, 2025
  207. rest: allow reading partial block data from storage
    It will allow fetching specific transactions using an external index,
    following https://github.com/bitcoin/bitcoin/pull/32541#issuecomment-3267485313.
    
    Co-authored-by: Hodlinator <172445034+hodlinator@users.noreply.github.com>
    Co-authored-by: Lőrinc <pap.lorinc@gmail.com>
    07135290c1
  208. romanz force-pushed on Dec 11, 2025
  209. romanz commented at 6:04 pm on December 11, 2025: contributor
  210. maflcko commented at 9:24 pm on December 11, 2025: member

    Nice. Only non-test change is adding the missing std::move to avoid the copy.

    review ACK 07135290c1720a14c9d2f18a5700bb6565ae7a10 🏪

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: review ACK 07135290c1720a14c9d2f18a5700bb6565ae7a10 🏪
    38tJyFyaSq9eBol4xP04vrx1VzK8edVKSVO+Xe4aZZctHc6j+sDo6BBuFcYwXO9R+a1pCtl/H34i6ppXCjD9bDg==
    
  211. DrahtBot requested review from l0rinc on Dec 11, 2025
  212. DrahtBot removed the label CI failed on Dec 11, 2025
  213. hodlinator approved
  214. hodlinator commented at 12:47 pm on December 12, 2025: contributor
    re-ACK 07135290c1720a14c9d2f18a5700bb6565ae7a10
  215. l0rinc commented at 12:55 pm on December 12, 2025: contributor

    ACK 07135290c1720a14c9d2f18a5700bb6565ae7a10

    Since my last ACK: deduplicated & sorted the unit test cases, moved non-const return value in rpc GetRawBlockChecked, added default switch comment, renamed functional test lambda, and changed functional test category to be delimited by log instead of comment.

    rfm?

  216. fanquake merged this on Dec 12, 2025
  217. fanquake closed this on Dec 12, 2025

  218. romanz deleted the branch on Dec 12, 2025
  219. romanz commented at 3:22 pm on December 12, 2025: contributor
    Many thanks!
  220. romanz referenced this in commit 17d6782bf3 on Dec 12, 2025
  221. stringintech referenced this in commit b7c3ec1af3 on Dec 12, 2025
  222. romanz referenced this in commit 599effdeab on Dec 14, 2025
  223. romanz referenced this in commit a059385438 on Dec 14, 2025
  224. romanz referenced this in commit 41118e17f8 on Dec 14, 2025
  225. in test/functional/interface_rest.py:467 in 07135290c1
    462+        blockhash = self.nodes[0].getbestblockhash()
    463+        block_bin = self.test_rest_request(f"/block/{blockhash}", req_type=ReqType.BIN, ret_type=RetType.BYTES)
    464+        for req_type in (ReqType.BIN, ReqType.HEX):
    465+            def get_block_part(status: int = 200, **kwargs):
    466+                resp = self.test_rest_request(f"/blockpart/{blockhash}", status=status,
    467+                                              req_type=req_type, ret_type=RetType.BYTES, **kwargs)
    


    ryanofsky commented at 7:14 pm on December 15, 2025:

    Just want to note that I saw CI failures here on windows after this was merged where the request for /block/ on line 463 seemed to succeed, but the /blockpart/ request immediately following it returned a 404 on line 466 (this line). Unfortunately the only error information was Expected: 200, Got: 404 - Response: b'' because line 87 above is only printing resp.status not resp.reason, so cause of the 404 is not clear.

    The failures happened in 4 different runs:

    https://github.com/bitcoin/bitcoin/actions/runs/20166531437/job/57892614796?pr=10102#step:13:3933 https://github.com/bitcoin/bitcoin/actions/runs/20166531437/job/57892614796?pr=10102#step:13:3933 https://github.com/bitcoin/bitcoin/actions/runs/20166532530/job/57895147947?pr=19460#step:13:3913 https://github.com/bitcoin/bitcoin/actions/runs/20166532530/job/57895147999?pr=19460#step:13:3917

    And the logs look exactly the same exactly the same except for the blockhashes:

     0test  TestFramework (INFO): Test the /blockpart URI 
     1node0 [http] [httpserver.cpp:307] [http] Received a POST request for / from 127.0.0.1:56381 
     2node0 [httpworker.0] [rpc/request.cpp:243] [void JSONRPCRequest::parse(const UniValue&)] [rpc] ThreadRPCServer method=getbestblockhash user=__cookie__ 
     3test  TestFramework (DEBUG): GET /rest/block/7bb6d8cde00d88f6a4342f0273a5981d2fdf4d397b35954e22c015fc55e19b7b.bin 
     4node0 [http] [httpserver.cpp:307] [http] Received a GET request for /rest/block/7bb6d8cde00d88f6a4342f0273a5981d2fdf4d397b35954e22c015fc55e19b7b.bin from 127.0.0.1:57139 
     5test  TestFramework (DEBUG): GET /rest/blockpart/7bb6d8cde00d88f6a4342f0273a5981d2fdf4d397b35954e22c015fc55e19b7b.bin?offset=0&size=652 
     6node0 [http] [httpserver.cpp:307] [http] Received a GET request for /rest/blockpart/7bb6d8cde00d88f6a4342f0273a5981d2fdf4d397b35954e22c015fc55e19b7b.bin?offset=0&size=6 from 127.0.0.1:57140 
     7test  TestFramework (ERROR): Unexpected exception 
     8Traceback (most recent call last):
     9  File "D:\a\bitcoin\bitcoin\test\functional\test_framework\test_framework.py", line 142, in main
    10    self.run_test()
    11    ~~~~~~~~~~~~~^^
    12  File "D:\a\bitcoin\bitcoin/test/functional/interface_rest.py", line 473, in run_test
    13    assert_equal(block_bin, get_block_part(query_params={"offset": 0, "size": len(block_bin)}))
    14                            ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    15  File "D:\a\bitcoin\bitcoin/test/functional/interface_rest.py", line 466, in get_block_part
    16    resp = self.test_rest_request(f"/blockpart/{blockhash}", status=status,
    17                                  req_type=req_type, ret_type=RetType.BYTES, **kwargs)
    18  File "D:\a\bitcoin\bitcoin/test/functional/interface_rest.py", line 87, in test_rest_request
    19    assert resp.status == status, f"Expected: {status}, Got: {resp.status} - Response: {str(resp.read())}"
    20           ^^^^^^^^^^^^^^^^^^^^^
    21AssertionError: Expected: 200, Got: 404 - Response: b''
    

    One thing that is weird here about the logs is the failing request shows up as ?offset=0&size=652 in the test logs but ?offset=0&size=6 in the node logs, where 652 is somehow truncated to 6. I don’t think the truncation explains the error, because it doesn’t look like there is a way for an incorrect size to trigger a 404, but the problems may be related.

    Also, even though the same failure happened in the two PR’s above, it did NOT happen in a third PR based on the other two (#19461) and pushed at the same time, which suggests the failure might happen randomly. I also don’t think there’s any code changes in any of the PR that would be related to this test or should change its behavior.

    Not sure if this report is actionable, but wanted to say:

    • If anyone has any idea what might cause these type of failures on windows CI, that would be helpful.
    • It would be nice if the assert resp.status == status error included resp.reason, since knowing the message associated with the status would probably be helpful for debugging.

    ryanofsky commented at 7:26 pm on December 15, 2025:

    where 652 is somehow truncated to 6

    Actually this appears to be normal because requests are truncated at 100 characters and 6 is the last character that fits https://github.com/bitcoin/bitcoin/blob/41bf8f2d5eceaac72b98c1791f5fc368b3af90cb/src/httpserver.cpp#L308


    ryanofsky commented at 7:28 pm on December 15, 2025:
    I also wonder if there may be a CI bug causing this, where windows jobs were using old bitcoind binaries that did not have the /blockpart/ endpoint defined so it returned a 404?

    l0rinc commented at 7:53 pm on December 15, 2025:

    Not sure I understand what’s going on, but can we assume that it will happen again if it’s a real problem (and not just some commit order mixup or whatever)?

    doing a

    0curl 'https://productionresultssa2.blob.core.windows.net/actions-results/b958a1af-d4e2-49e6-9411-5feedc7e3f3a/workflow-job-run-1192ebfe-48b0-5433-a77e-092eca6802f2/logs/job/job-logs.txt?rsct=text%2Fplain&se=2025-12-15T19%3A49%3A59Z&sig=LJxVtw7oMg%2Byssj%2FbCpdhXU3T3PfIVYuOxqJwuu2FyE%3D&ske=2025-12-16T06%3A42%3A59Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2025-12-15T18%3A42%3A59Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2025-11-05&sp=r&spr=https&sr=b&st=2025-12-15T19%3A39%3A54Z&sv=2025-11-05' | grep 'Registering HTTP handler for /rest/block'
    

    shows no Registering HTTP handler for /rest/blockpart/ for https://github.com/bitcoin/bitcoin/actions/runs/20166531437/job/57892614796?pr=10102#step:13:3933:

    02025-12-12T14:35:29.4245912Z  node0 2025-12-12T14:35:23.889620Z [init] [httpserver.cpp:761] [void RegisterHTTPHandler(const std::string&, bool, const HTTPRequestHandler&)] [http] Registering HTTP handler for /rest/block/notxdetails/ (exactmatch 0) 
    12025-12-12T14:35:29.4248088Z  node0 2025-12-12T14:35:23.889630Z [init] [httpserver.cpp:761] [void RegisterHTTPHandler(const std::string&, bool, const HTTPRequestHandler&)] [http] Registering HTTP handler for /rest/block/ (exactmatch 0) 
    22025-12-12T14:35:29.4249218Z  node0 2025-12-12T14:35:23.889640Z [init] [httpserver.cpp:761] [void RegisterHTTPHandler(const std::string&, bool, const HTTPRequestHandler&)] [http] Registering HTTP handler for /rest/blockfilter/ (exactmatch 0) 
    32025-12-12T14:35:29.4250381Z  node0 2025-12-12T14:35:23.889650Z [init] [httpserver.cpp:761] [void RegisterHTTPHandler(const std::string&, bool, const HTTPRequestHandler&)] [http] Registering HTTP handler for /rest/blockfilterheaders/ (exactmatch 0) 
    42025-12-12T14:35:29.4259046Z  node0 2025-12-12T14:35:23.889720Z [init] [httpserver.cpp:761] [void RegisterHTTPHandler(const std::string&, bool, const HTTPRequestHandler&)] [http] Registering HTTP handler for /rest/blockhashbyheight/ (exactmatch 0) 
    

    so a 404 is indeed expected.

    Also, the build seems to be for GITHUB_SHA: 369ecaca0e89f90647edc41faa73b84ca31eebfa which doesn’t seem to contain this PR - why are we even running those test, is it an invalid checkout, some leftovers that we forgot to clean up?


    ryanofsky commented at 8:06 pm on December 15, 2025:

    re: #33657 (review)

    shows no Registering HTTP handler for /rest/blockpart/ for https://github.com/bitcoin/bitcoin/actions/runs/20166531437/job/57892614796?pr=10102#step:13:3933:

    Nice, thanks! That seems like a clear indication that the CI job was running the new test with an old bitcoind binary.

    Not sure I understand what’s going on, but can we assume that it will happen again if it’s a real problem (and not just some commit order mixup or whatever)?

    From what I can tell it seems like this is a real problem and it’s probably fair to assume it will happen again at some point. But maybe it will happen rarely enough that it doesn’t matter. It seems like I was just unlucky pushing my PR’s around the same time that this PR was merged.

    I do think it would be good if windows CI jobs that do not build their own binaries could check whether the binaries came from the same commit being tested, and produce an explicit error or warning if they don’t. (Or maybe they could just check out the same commit that the binaries came from.) No idea how difficult these things would be to implement, though


    l0rinc commented at 8:08 pm on December 15, 2025:
    is it possible that this was rather a leftover file in the build folder that wasn’t properly reverted before checkout?

    ryanofsky commented at 8:21 pm on December 15, 2025:

    re: #33657 (review)

    is it possible that this was rather a leftover file in the build folder that wasn’t properly reverted before checkout?

    I probably don’t know enough about the CI system to answer this question, but I believe these CI jobs are not building anything, they are just using binaries built by other CI jobs, and apparently not requiring those binaries to come from the same commit. The jobs are defined here:

    https://github.com/bitcoin/bitcoin/blob/41bf8f2d5eceaac72b98c1791f5fc368b3af90cb/.github/workflows/ci.yml#L402-L403

    EDIT: Updated link above to point to windows-native-test job not windows-cross job


    l0rinc commented at 8:47 pm on December 15, 2025:

    @ryanofsky do you mean that even with

    0GITHUB_SHA: 369ecaca0e89f90647edc41faa73b84ca31eebfa
    

    since we’re fetching by PR number:

    0"C:\Program Files\Git\bin\git.exe" checkout --progress --force refs/remotes/pull/10102/merge
    

    we might switch versions mid-run if they push?

    02025-12-12T14:21:28.8214717Z HEAD is now at e3a830d Merge 3638b89090f539c360c5d4d358c4ed17db59c25e into 938d7aacabd0bb3784bb3e529b1ed06bb2891864
    

    https://github.com/bitcoin/bitcoin/blob/d2a199bca73ba26838ebbaebd6f6e5857fe3d34f/.github/workflows/ci.yml#L153-L154 should github.ref be GITHUB_SHA - or maybe we should fail if the two don’t coincide?



    ryanofsky commented at 9:09 pm on December 15, 2025:

    re: #33657 (review)

    we might switch versions mid-run if they push?

    Yes that seems like what happened. Test appears to be from e3a830d9d810b085eeb78e3e99c697508df52696 while binaries were built from 369ecaca0e89f90647edc41faa73b84ca31eebfa (GITHUB_SHA)

    should github.ref be GITHUB_SHA - or maybe we should fail if the two don’t coincide?

    It does seem like using GITHUB_SHA would fix this, but might have other negative consequences. I don’t know enough to know what the right fix should be, or if this is worth fixing. The problem seems easy enough to work around for now by pushing again (which I need to do anyway because the linter job failed).


    maflcko commented at 6:47 am on December 16, 2025:
    This was intentionally left as-is, but a clean fix would be to record the commit id and check out the exact commit id later in the second task. see #33303 (review).

    maflcko commented at 8:11 am on December 16, 2025:
  226. fanquake referenced this in commit a005fdff6c on Dec 17, 2025

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-12-23 00:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me