rest: fix getutxos POST body decoding #36385

l0rinc commented at 1:04 AM on September 30, 2026: contributor

Problem: Binary GET and POST requests to /rest/getutxos for the same outpoints should agree. GET finds the unspent output, but POST incorrectly reports no hit. The POST decoder writes the body into a DataStream with operator<<, serializing it as a string and prepending a CompactSize length. It then reads the first byte of that added length as the checkmempool flag, misaligning the later request fields. Hex POST bodies use the same decoding and are affected too.

Fix: Deserialize the supplied body bytes directly. Separate commits apply the existing 15-outpoint limit during decoding before allocating the vector, and use SpanReader to avoid the extra copy into a DataStream.

DrahtBot added the label RPC/REST/ZMQ on Sep 30, 2026

DrahtBot commented at 1:04 AM on September 30, 2026: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

External sites

Code coverage & benchmarks: https://corecheck.dev/bitcoin/bitcoin/pulls/36385
Ralph review <img src=https://review.fish.foo/assets/bitcoin/bitcoin/36385.png alt="Ralph review status"> https://git.fish.foo/bitcoin/bitcoin/pulls/36385#author-ralph

Reviews

See the guideline and AI policy for information on the review process. A summary of reviews will appear here.

l0rinc force-pushed on Sep 30, 2026

DrahtBot added the label CI failed on Sep 30, 2026

DrahtBot commented at 1:22 AM on September 30, 2026: contributor

🚧 At least one of the CI tasks failed. <sub>Task iwyu: https://github.com/bitcoin/bitcoin/actions/runs/36653350378/job/109692242470</sub> <sub>LLM reason (✨ experimental): CI failed because IWYU reported a missing header (added #include <span.h> in src/rest.cpp) and its fix check aborted the job.</sub>

<details><summary>Hints</summary>

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

</details>

DrahtBot removed the label CI failed on Sep 30, 2026

Zeegaths commented at 9:14 AM on October 1, 2026: none

I reproduced the issue locally by running this script:

#!/usr/bin/env python3
import sys, struct, urllib.request

def post_body(txid_hex, vout):
    txid_le = bytes.fromhex(txid_hex)[::-1]   # RPC displays big-endian, wire format is little-endian
    return bytes([1]) + bytes([1]) + txid_le + struct.pack('<I', vout)  # checkmempool=1, count=1

def fetch(url, data=None):
    req = urllib.request.Request(url, data=data, method='POST' if data else 'GET')
    with urllib.request.urlopen(req) as r:
        return r.read()

if __name__ == '__main__':
    txid, vout = sys.argv[1], int(sys.argv[2])
    host = 'http://127.0.0.1:18443'

    get_resp = fetch(f'{host}/rest/getutxos/checkmempool/{txid}-{vout}.bin')
    post_resp = fetch(f'{host}/rest/getutxos.bin', data=post_body(txid, vout))

    print('GET :', get_resp.hex())
    print('POST:', post_resp.hex())
    print('MATCH' if get_resp == post_resp else 'MISMATCH')

Response mismatch on master : <img width="729" height="151" alt="Screenshot from 2026-10-01 11-44-52" src="https://github.com/user-attachments/assets/5fb06f25-121f-4561-af63-be0a651d67d3" />

Response match on PR branch (pr-36385) : <img width="741" height="172" alt="image" src="https://github.com/user-attachments/assets/fc20f36e-c738-400c-b15e-192c7e854635" />

l0rinc commented at 5:22 PM on October 1, 2026: contributor

@Zeegaths you posted the same image twice, could you please update the first one?

in src/rest.cpp:1015 in 0cc326caae

1011 | @@ -1011,10 +1012,7 @@ static bool rest_getutxos(const std::any& context, HTTPRequest* req, const std::
1012 |                  if (fInputParsed) //don't allow sending input over URI and HTTP RAW DATA
1013 |                      return RESTERR(req, HTTP_BAD_REQUEST, "Combination of URI scheme inputs and raw post data is not allowed");
1014 |  
1015 | -                DataStream oss{};
1016 | -                oss << strRequestMutable;
1017 | -                oss >> fCheckMemPool;
1018 | -                oss >> vOutPoints;
1019 | +                SpanReader{MakeByteSpan(strRequestMutable)} >> fCheckMemPool >> LIMITED_VECTOR(vOutPoints, MAX_GETUTXOS_OUTPOINTS);

b-l-u-e commented at 4:41 PM on October 2, 2026:

seems like trailing data is silently being accepted... the decoder doesnt check that the body was consumed fully..so any bytes after the outpoint vector are ignored

i reproduced the issue by adding this line in interface_rest.py

git diff test/functional/interface_rest.py
diff --git a/test/functional/interface_rest.py b/test/functional/interface_rest.py
index eb18569478..9a32c49e3b 100755
--- a/test/functional/interface_rest.py
+++ b/test/functional/interface_rest.py
@@ -213,6 +213,7 @@ class RESTTest (BitcoinTestFramework):
         self.test_rest_request("/getutxos", http_method='POST', req_type=ReqType.JSON, body='{"checkmempool', status=400, ret_type=RetType.OBJ)
         self.test_rest_request("/getutxos", http_method='POST', req_type=ReqType.BIN, body='{"checkmempool', status=400, ret_type=RetType.OBJ)
         self.test_rest_request("/getutxos/checkmempool", http_method='POST', req_type=ReqType.JSON, status=400, ret_type=RetType.OBJ)
+        self.test_rest_request("/getutxos", http_method='POST', req_type=ReqType.BIN, body=bin_request + b'\x00', status=400, ret_type=RetType.OBJ)
         self.test_rest_request(f"/getutxos/{spending[0]}_+1", ret_type=RetType.OBJ, status=400)

here below shows the node returns success response

<details> <summary>output results</summary>

build/test/functional/interface_rest.py
2026-10-02T16:09:34.675935Z TestFramework (INFO): PRNG seed is: 6815011314856331724
2026-10-02T16:09:34.726606Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test__mdx5r6s
2026-10-02T16:09:35.188130Z TestFramework (INFO): Broadcast test transaction and sync nodes
2026-10-02T16:09:36.216533Z TestFramework (INFO): Test the /tx URI
2026-10-02T16:09:36.222070Z TestFramework (INFO): Query an unspent TXO using the /getutxos URI
2026-10-02T16:09:36.240350Z TestFramework (INFO): Query a spent TXO using the /getutxos URI
2026-10-02T16:09:36.241426Z TestFramework (INFO): Query two TXOs using the /getutxos URI
2026-10-02T16:09:36.242814Z TestFramework (INFO): Compare binary GET and POST /getutxos responses for the same TXOs
2026-10-02T16:09:36.244860Z TestFramework (INFO): Test the /getutxos URI with and without /checkmempool
2026-10-02T16:09:36.277380Z TestFramework (INFO): Check some invalid requests
2026-10-02T16:09:36.280077Z TestFramework (ERROR): Unexpected exception:
Traceback (most recent call last):
  File "/home/user/projects/bitcoin/test/functional/test_framework/test_framework.py", line 145, in main
    self.run_test()
  File "/home/user/projects/bitcoin/build/test/functional/interface_rest.py", line 216, in run_test
    self.test_rest_request("/getutxos", http_method='POST', req_type=ReqType.BIN, body=bin_request + b'\x00', status=400, ret_type=RetType.OBJ)
  File "/home/user/projects/bitcoin/build/test/functional/interface_rest.py", line 90, in test_rest_request
    assert resp.status == status, f"Expected: {status}, Got: {resp.status} ({resp.reason}) - Response: {str(resp.read())}"
           ^^^^^^^^^^^^^^^^^^^^^
AssertionError: Expected: 400, Got: 200 (OK) - Response: b'\xca\x00\x00\x00>\xac\xfa1\xe1J\xe7nm\x8cp\xe8\x8e\x05~\x050Xx\xe1\x14\xb9\x0e\xa8\x86d\xab\xbed\xd3\x920\x01\x01\x01\x00\x00\x00\x00\xc9\x00\x00\x00\x80\x96\x98\x00\x00\x00\x00\x00"Q h\xe0|\xdaU\xb4<\x01q\xa8\x19\xeaT\xb91O=\x17\x05\x821\x04\xba\xc6f\xa6\x8eE\xee\x0cX\xd6'
2026-10-02T16:09:36.333952Z TestFramework (INFO): Not stopping nodes as test failed. The dangling processes will be cleaned up later.
2026-10-02T16:09:36.334372Z TestFramework (WARNING): Not cleaning up dir /tmp/bitcoin_func_test__mdx5r6s
2026-10-02T16:09:36.334541Z TestFramework (ERROR): Test failed. Test logging available at /tmp/bitcoin_func_test__mdx5r6s/test_framework.log
2026-10-02T16:09:36.334838Z TestFramework (ERROR): 
2026-10-02T16:09:36.335156Z TestFramework (ERROR): Hint: Call /home/user/projects/bitcoin/test/functional/combine_logs.py '/tmp/bitcoin_func_test__mdx5r6s' to consolidate all logs
2026-10-02T16:09:36.335312Z TestFramework (ERROR): 
2026-10-02T16:09:36.335445Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2026-10-02T16:09:36.335608Z TestFramework (ERROR): https://github.com/bitcoin/bitcoin/issues
2026-10-02T16:09:36.335720Z TestFramework (ERROR): 
[node 1] Cleaning up leftover process
[node 0] Cleaning up leftover process

</details>

so perhaps we could keep reader named and reject leftover bytes like how transaction decoding being handled here: https://github.com/bitcoin/bitcoin/blob/65896ac58ea63e29ad19d41b492e37f8722ac715/src/core_io.cpp#L174-L182

                SpanReader reader{MakeByteSpan(strRequestMutable)};
                reader >> fCheckMemPool >> LIMITED_VECTOR(vOutPoints, MAX_GETUTXOS_OUTPOINTS);
                if (!reader.empty()) return RESTERR(req, HTTP_BAD_REQUEST, "Parse error");

l0rinc commented at 2:38 PM on October 4, 2026:

Thanks, added trailing-byte rejection and coverage for binary and hex POST bodies

Zeegaths commented at 6:10 PM on October 2, 2026: none

@Zeegaths you posted the same image twice, could you please update the first one?

sorry about that, i've updated it

l0rinc force-pushed on Oct 4, 2026

b-l-u-e commented at 7:00 PM on October 4, 2026: contributor

i tested and found this issue below

git diff test/functional/interface_rest.py
diff --git a/test/functional/interface_rest.py b/test/functional/interface_rest.py
index 68fb491567..7f7ff5692d 100755
--- a/test/functional/interface_rest.py
+++ b/test/functional/interface_rest.py
@@ -180,6 +180,7 @@ class RESTTest (BitcoinTestFramework):
 
         for req_type, body in [(ReqType.BIN, bin_request + b'\x00'), (ReqType.HEX, (bin_request + b'\x00').hex())]:
             self.test_rest_request("/getutxos", http_method='POST', req_type=req_type, body=body, status=400, ret_type=RetType.OBJ)
+        self.test_rest_request("/getutxos", http_method='POST', req_type=ReqType.HEX, body=bin_request.hex() + '0', status=400, ret_type=RetType.OBJ)
 
         self.log.info("Test the /getutxos URI with and without /checkmempool")
         # Create a transaction, check that it's found with /checkmempool, but

<details> <summary>output results</summary>

build/test/functional/interface_rest.py
2026-10-04T18:53:45.192405Z TestFramework (INFO): PRNG seed is: 8612403223243651630
2026-10-04T18:53:45.243022Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test_0yvdhtym
2026-10-04T18:53:45.690260Z TestFramework (INFO): Broadcast test transaction and sync nodes
2026-10-04T18:53:46.718605Z TestFramework (INFO): Test the /tx URI
2026-10-04T18:53:46.722420Z TestFramework (INFO): Query an unspent TXO using the /getutxos URI
2026-10-04T18:53:46.737722Z TestFramework (INFO): Query a spent TXO using the /getutxos URI
2026-10-04T18:53:46.739081Z TestFramework (INFO): Query two TXOs using the /getutxos URI
2026-10-04T18:53:46.740170Z TestFramework (INFO): Compare binary GET and POST /getutxos responses for the same TXOs
2026-10-04T18:53:46.745517Z TestFramework (ERROR): Unexpected exception:
Traceback (most recent call last):
  File "/home/user/projects/bitcoin/test/functional/test_framework/test_framework.py", line 145, in main
    self.run_test()
  File "/home/user/projects/bitcoin/build/test/functional/interface_rest.py", line 183, in run_test
    self.test_rest_request("/getutxos", http_method='POST', req_type=ReqType.HEX, body=bin_request.hex() + '0', status=400, ret_type=RetType.OBJ)
  File "/home/user/projects/bitcoin/build/test/functional/interface_rest.py", line 90, in test_rest_request
    assert resp.status == status, f"Expected: {status}, Got: {resp.status} ({resp.reason}) - Response: {str(resp.read())}"
           ^^^^^^^^^^^^^^^^^^^^^
AssertionError: Expected: 400, Got: 200 (OK) - Response: b'c900000096269ba75ce72cbc9b444672644ecd2b8754d79f8625303504fb9b8b628223440000\n'
2026-10-04T18:53:46.799148Z TestFramework (INFO): Not stopping nodes as test failed. The dangling processes will be cleaned up later.
2026-10-04T18:53:46.799530Z TestFramework (WARNING): Not cleaning up dir /tmp/bitcoin_func_test_0yvdhtym
2026-10-04T18:53:46.799703Z TestFramework (ERROR): Test failed. Test logging available at /tmp/bitcoin_func_test_0yvdhtym/test_framework.log
2026-10-04T18:53:46.799976Z TestFramework (ERROR): 
2026-10-04T18:53:46.800302Z TestFramework (ERROR): Hint: Call /home/user/projects/bitcoin/test/functional/combine_logs.py '/tmp/bitcoin_func_test_0yvdhtym' to consolidate all logs
2026-10-04T18:53:46.800453Z TestFramework (ERROR): 
2026-10-04T18:53:46.800569Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2026-10-04T18:53:46.800735Z TestFramework (ERROR): https://github.com/bitcoin/bitcoin/issues
2026-10-04T18:53:46.800845Z TestFramework (ERROR): 
[node 1] Cleaning up leftover process
[node 0] Cleaning up leftover process

</details>

test: characterize getutxos GET and POST disagreement

GET and POST requests for the same outpoints should return the same binary response. The current POST parser incorrectly returns a different result. Record this disagreement before fixing body decoding.

The existing POST test incorrectly wrote txids in display byte order, so it queried different outpoints. Use COutPoint and ser_vector to encode the same outpoints as the GET request while retaining the chain-tip checks.

Cover hex POST bodies as well and record the current acceptance of trailing bytes and malformed hex before tightening request parsing.

Co-authored-by: b-l-u-e <winnie.gitau282@gmail.com>

dd7998e248

rest: fix getutxos request body decoding

Binary GET and POST requests for the same outpoints incorrectly return different UTXO results. Writing the POST body to DataStream with operator<< serializes it as a string and prepends a CompactSize length, which the parser incorrectly reads as request fields.

Initialize the stream from the supplied body bytes. The existing binary and hex comparisons now require GET and POST responses to match.

Reject bytes left after the outpoint vector so a successful POST consumes the complete decoded body.
Reject malformed hex with TryParseHex instead of treating failed decoding as an empty request.

Co-authored-by: b-l-u-e <winnie.gitau282@gmail.com>

57dc676b59

rest: bound getutxos POST vector decoding

The ordinary vector reader may reserve a batch based on the declared count before enough outpoint bytes are present. Apply the existing 15-outpoint limit during deserialization so an oversized count fails before allocation. The later size check continues to cover URI inputs.

f1512131e9

rest: read getutxos POST body in place

The DataStream span constructor copies the request body, which the HTTP layer permits up to 32 MiB. Use SpanReader to read the already held bytes without another full-body allocation and copy. Deserialization and parse-error handling stay the same.

b7622767c6

l0rinc force-pushed on Oct 4, 2026

l0rinc commented at 11:35 PM on October 4, 2026: contributor

Not sure how important rejecting extra bytes is - given that nobody even noticed it was fundamentally broken -, but we might as well fix the remaining issues. Added @b-l-u-e as co-author.