rest: allow reading partial block data from storage #33657

1. 
   romanz commented at 11:08 am on October 19, 2025: contributor

   It allows fetching specific transactions using an external index, following #32541 (comment).

   Currently, electrs and other indexers map between an address/scripthash to the list of the relevant transactions.

   However, in order to fetch those transactions from bitcoind, electrs relies on reading the whole block and post-filtering for a specific transaction¹. Other indexers use a txindex to fetch a transaction using its txid ²³⁴.

   The above approach has significant storage and CPU overhead, since the txid is a pseudo-random 32-byte value. Also, mainnet txindex takes ~60GB today.

   This PR is adding support for using the transaction’s position within its block to be able to fetch it directly using REST API, using the following HTTP request:

   0GET /rest/blockpart/BLOCKHASH.bin?offset=OFFSET&size=SIZE
   

   The offsets’ index can be encoded much more efficiently (<3GB today).

   Only binary and hex response formats are supported.

   ---

   1. https://github.com/romanz/electrs/blob/master/doc/schema.md ↩︎

   2. https://github.com/Blockstream/electrs/blob/new-index/doc/schema.md#txstore ↩︎

   3. https://github.com/spesmilo/electrumx/blob/master/docs/HOWTO.rst#prerequisites ↩︎

   4. https://github.com/cculianu/Fulcrum/blob/master/README.md#requirements ↩︎

2. DrahtBot added the label RPC/REST/ZMQ on Oct 19, 2025
3. 
   DrahtBot commented at 11:08 am on October 19, 2025: contributor

   The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

   Code Coverage & Benchmarks

   For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33657.

   Reviews

   See the guideline for information on the review process.

   Type	Reviewers
   ACK	ubbabeck
   Concept ACK	optout21

   If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

4. romanz force-pushed on Oct 19, 2025
5. 
   andrewtoth commented at 9:30 pm on October 19, 2025: contributor

   Some of the PR description is reused from the previous PR, but is now stale.

   This PR is adding support for using the transaction’s position within its block to be able to fetch it directly using REST API, using the following HTTP request (to fetch the N-th transaction from BLOCKHASH):

   This does not support fetching an N-th transaction, only a slice of a block.

   If binary response format is used, the transaction data will be read directly from the storage and sent back to the client, without any deserialization overhead.

   There is no way to use this in non-binary form (except hex I suppose).

6. in src/node/blockstorage.cpp:1096 in 787d6df48a

   1093+            offset = *part_offset;
   1094+        }
   1095+
   1096+        size_t size = blk_size - offset;
   1097+        if (part_size.has_value()) {
   1098+            if (*part_size > size) {
   

   ---

   ---

   

   andrewtoth commented at 11:04 pm on October 19, 2025:

   Do we want to support zero *part_size? Should probably return false too?

   ---

   

   romanz commented at 8:21 pm on October 20, 2025:

   Sounds good - fixed in 787d6df48a..c4bf7a77c4.

   ---

   

   andrewtoth commented at 10:22 pm on October 20, 2025:

   We should probably cover this as well as other cases of ReadRawBlockPart in blockmanager_tests.cpp.

   ---

   

   romanz commented at 8:26 pm on October 21, 2025:

   Added in c9bfd298be..301116e855.
7. in src/node/blockstorage.h:417 in 787d6df48a

   413@@ -414,6 +414,7 @@ class BlockManager
   414     bool ReadBlock(CBlock& block, const FlatFilePos& pos, const std::optional<uint256>& expected_hash) const;
   415     bool ReadBlock(CBlock& block, const CBlockIndex& index) const;
   416     bool ReadRawBlock(std::vector<std::byte>& block, const FlatFilePos& pos) const;
   417+    bool ReadRawBlockPart(std::vector<std::byte>& block, const FlatFilePos& pos, std::optional<size_t> part_offset, std::optional<size_t> part_size) const;
   

   ---

   ---

   

   optout21 commented at 11:26 am on October 20, 2025:

   nit: Declaration differs from definition, block vs data parameter name

   ---

   

   romanz commented at 8:21 pm on October 20, 2025:

   Thanks - fixed in 787d6df48a..c4bf7a77c4.
8. in src/rest.cpp:480 in 787d6df48a outdated

   495@@ -472,6 +496,11 @@ static bool rest_block_notxdetails(const std::any& context, HTTPRequest* req, co
   496     return rest_block(context, req, uri_part, TxVerbosity::SHOW_TXID);
   497 }
   498 
   499+static bool rest_block_part(const std::any& context, HTTPRequest* req, const std::string& uri_part)
   

   ---

   ---

   

   optout21 commented at 11:31 am on October 20, 2025:

   This way both block and blockpart api’s take the same paramaters, including "offset" and "size", right? Is that intended?

   ---

   

   romanz commented at 8:14 pm on October 20, 2025:

   Right - I have used the same implementation for both endpoints for simplicity and to avoid code duplication. I can add an additional boolean argument to enable offset and size parameter handling for the /rest/blockpart endpoint (and disable it for /rest/block endpoint). WDYT?

   ---

   

   andrewtoth commented at 9:28 pm on October 20, 2025:

   If I understand correctly, this would allow passing offset or size to /rest/block along with .json format specification. Normally this would fail to deserialize, but if very unlucky it could deserialize to what looks like a block and return? That would be very bad. Similarly if not using .json, this would return incorrect data that a user might not expect (for instance if they switched rest endpoints but forgot to remove parameters).

   I would recommend disabling these parameters if not calling /rest/blockpart.

   ---

   

   romanz commented at 6:01 am on October 21, 2025:

   Agree - fixed in c4bf7a77c4..c9bfd298be (offset and size are parsed only for /rest/blockpart endpoint).
9. romanz force-pushed on Oct 20, 2025
10. 
    romanz commented at 8:23 pm on October 20, 2025: contributor

    Some of the PR description is reused from the previous PR, but is now stale.

    Good catch, thanks! Updated PR description: #33657#issue-3529967048

11. romanz force-pushed on Oct 21, 2025
12. 
    optout21 commented at 9:48 am on October 21, 2025: none

    Concept ACK c9bfd298be422de7e989fe244fb4281c507068a3
13. rest: allow reading partial block data from storage

    It will allow fetching specific transactions using an external index,
    following https://github.com/bitcoin/bitcoin/pull/32541#issuecomment-3267485313.

    301116e855
14. romanz force-pushed on Oct 21, 2025
15. romanz requested review from andrewtoth on Oct 22, 2025
16. 
    ubbabeck commented at 7:33 pm on October 22, 2025: none

    tACK 301116e855

    What was done:

    • functional test.
    • unit test
    • tested the new rest endpoint rest/blockpart/<blockhash>.bin manually with curl with different offsets and sizes.

    If there are any additional test you’d like me to help with let me know. I also did a perf dump, but I need some guidelines on what to measure/ grep for.

17. DrahtBot requested review from optout21 on Oct 22, 2025

Contributors
romanz DrahtBot andrewtoth optout21 ubbabeck

Review Requested
andrewtoth optout21

Labels
RPC/REST/ZMQ