Add state to HTTPRequest #35735

pull pinheadmz wants to merge 5 commits into bitcoin:master from pinheadmz:http-req-state changing 4 files +614 −175
  1. pinheadmz commented at 10:22 AM on July 16, 2026: member

    This PR reduces the memory consumption of the HTTP Server when reading data from connected clients, and improves performance especially when requests are large (i.e. requiring multiple TCP packets).

    In #35182 the server copies as much data as it can from the socket into application memory, and then tries to parse as many complete HTTP requests as possible from that data. If a request is discovered to be incomplete, the in-progress request is abandoned. The server tries again on the next I/O cycle to read the same data from the buffer, duplicating work as many times as it takes before the client finishes sending the request (or times out).

    This PR implements two improvements to this:

    1. Only parse one request at a time from the receive buffer. The server processes requests from each client in series anyway.
    2. Add state to HTTPRequest so it can be filled with data from the receive buffer over multiple I/O loop iterations without losing progress.

    If a client sends large or multiple requests, that data will sit in the kernel's socket buffer instead of the application memory. Eventually the socket buffer will fill up and TCP backpressure will kick in, dropping the TCP window to 0 and blocking the client from sending any more.

    A state machine for HTTPRemoteClient was discussed previously to control resource consumption. Another nice benefit of this model (for a follow-up PR) will be to insert the RPC authentication check after reading 8kB-limited headers but before the 32MB-limited request body.

  2. DrahtBot commented at 10:22 AM on July 16, 2026: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35735.

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline and AI policy for information on the review process.

    Type Reviewers
    ACK winterrdog, janb84, frankomosh
    Concept ACK brunoerg, w0xlt

    If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #35829 (http: Make class fields private or convert to struct by hodlinator)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

    LLM Linter (✨ experimental)

    Possible places where comparison-specific test macros should replace generic comparisons:

    • [test/functional/interface_http.py] assert res.index(b"HTTP/1.1 200") < res.index(b"HTTP/1.1 400") -> consider assert_greater_than(res.index(b"HTTP/1.1 400"), res.index(b"HTTP/1.1 200")) for a comparison-specific helper.

    <sup>2026-08-10 15:14:57</sup>

  3. brunoerg commented at 11:35 AM on July 16, 2026: contributor

    Concept ACK

  4. janb84 commented at 4:51 PM on July 16, 2026: contributor

    big concept ACK fb0a085cc9152725b8152975c2fb8a0898f6939b

  5. w0xlt commented at 5:50 PM on July 16, 2026: contributor

    Concept ACK

  6. in src/httpserver.cpp:480 in fb0a085cc9
     511 |  
     512 | -            // Pack chunk onto body
     513 | -            m_body += reader.ReadLength(*chunk_size);
     514 | +            // Pack chunk onto body and clear state
     515 | +            m_body += reader.ReadLength(*m_chunk_size);
     516 | +            m_chunk_size.reset();
    


    brunoerg commented at 6:40 PM on July 16, 2026:

    fb0a085cc9152725b8152975c2fb8a0898f6939b: A mix of my head and claude: I think that here might be wrong because It resets m_chunk_size before confirming it actually read the trailing CRLF that terminates a chunk.


    brunoerg commented at 6:47 PM on July 16, 2026:

    It's interesting that the fuzz target would not catch it because it feeds the whole buffer in one shot.


    pinheadmz commented at 2:28 PM on July 21, 2026:

    Good catch here and actually indicates a bit more logic was needed on chunk processing. What I had before wouldn't read an incomplete chunk from the buffer (it would wait until the entire chunk was in the buffer). I updated it to read partial chunks and just track the amount of data read for an in progress chunk as part of the state. Then reading the terminal CRLF is done after a chunk is complete, no matter how many calls to LoadBody() it needed. Chunk state is only cleared after that.

    I also added a state-clearing after the final chunk (with size 0) is read. It shouldn't be needed any more there but I felt it indicates to the code reviewer a bit more where the ends of the logic branches are.

  7. in test/functional/interface_http.py:240 in fb0a085cc9
     244 | -        # detecting the broken socket (which it may still be trying to write to).
     245 | +        # Split off the send into a background thread. When the server detects
     246 | +        # the excessive size it will stop reading from the socket, but the client
     247 | +        # will continue trying to write until the backpressure eventually
     248 | +        # drops the TCP window size to 0. While the send operation is blocking until
     249 | +        # it times out, we can still receive the server's reponse in the foreground.
    


    winterrdog commented at 10:12 PM on July 18, 2026:

    nit: small typo on response

            # it times out, we can still receive the server's response in the foreground.
    

    pinheadmz commented at 7:21 PM on July 21, 2026:

    👍

  8. in test/functional/interface_http.py:346 in fb0a085cc9
     354 | -        # detecting the broken socket (which it may still be trying to write to).
     355 | +        # Split off the send into a background thread. When the server detects
     356 | +        # the excessive size it will stop reading from the socket, but the client
     357 | +        # will continue trying to write until the backpressure eventually
     358 | +        # drops the TCP window size to 0. While the send operation is blocking until
     359 | +        # it times out, we can still receive the server's reponse in the foreground.
    


    winterrdog commented at 10:13 PM on July 18, 2026:

    nit: same as before, just a small typo on response

            # it times out, we can still receive the server's response in the foreground.
    

    pinheadmz commented at 7:21 PM on July 21, 2026:

    👍

  9. in src/httpserver.cpp:1115 in fb0a085cc9
    1114 | @@ -1106,22 +1115,41 @@ void HTTPServer::ClearConnectedClients()
    1115 |      m_connected.clear();
    


    winterrdog commented at 12:01 AM on July 19, 2026:

    this is probably outside the scope of this PR, but it felt close enough to the surrounding changes that it seemed worth mentioning

    just like you did in DisconnectClients, i wonder if it also belongs here. it looks like clients still sitting in m_connected when ClearConnectedClients() runs can keep themselves alive through HTTPRequest::m_client, preventing the HTTPRemoteClient (and its socket) from ever being destroyed

    my reasoning is roughly: HTTPRequest holds a shared_ptr<HTTPRemoteClient> back to its owning client, while client->m_req owns the HTTPRequest. as long as m_req is non-null, the client effectively holds an extra reference to itself. m_connected.clear() only drops the references owned by m_connected. if m_req is still set, the reference count never reaches zero, so the destructor never runs. ClearConnectedClients() only deals with the stragglers that made it past DisconnectClients(), so these seem like the clients most likely to still have a live m_req. by this point the HTTP thread pool has already been stopped, so i do not see another owner that would eventually break the cycle

    sth along these lines is what i had in mind:

    diff --git a/src/httpserver.cpp b/src/httpserver.cpp
    index c3a541f8c1..0b506a3895 100644
    --- a/src/httpserver.cpp
    +++ b/src/httpserver.cpp
    @@ -1111,6 +1111,7 @@ void HTTPServer::ClearConnectedClients()
         Assume(!m_thread_socket_handler.joinable()); // must be called after JoinSocketsThreads()
         if (m_connected.empty()) return;
         LogWarning("Force-disconnecting %d HTTP client(s) that did not disconnect gracefully", m_connected.size());
    +    for (const auto& client : m_connected) client->m_req.reset();
         m_connected_size.fetch_sub(m_connected.size(), std::memory_order_relaxed);
         m_connected.clear();
     }
    

    i could well be missing another cleanup path. does anything else clear m_req for these clients before this runs, or is this a real leak ?


    pinheadmz commented at 7:09 PM on July 21, 2026:

    Thanks I think this is a real leak, although it would only leak during shutdown. Still, we should clean it up. I added a helper method to HTTPRemoteClient and called it from both disconnect sites.

  10. pinheadmz force-pushed on Jul 21, 2026
  11. pinheadmz commented at 7:29 PM on July 21, 2026: member

    push to ce01fddeb1caafaec2bb2e7157cd607d4a350620

    Addressed review feedback and added another commit with unit test coverage over the HTTPRequest state machine.

    Major behavioral changes:

    • Read incomplete chunked-encoding chunks from the buffer and don't reset state until the chunk-terminal CRLF is parsed
    • Catch errors in ReadRequest, set error state and clear memory there before continuing (by throwing)
  12. pinheadmz force-pushed on Jul 21, 2026
  13. DrahtBot added the label CI failed on Jul 21, 2026
  14. DrahtBot commented at 7:38 PM on July 21, 2026: contributor

    <!--85328a0da195eb286784d51f73fa0af9-->

    🚧 At least one of the CI tasks failed. <sub>Task ASan + LSan + UBSan + integer: https://github.com/bitcoin/bitcoin/actions/runs/29861699031/job/88739755985</sub> <sub>LLM reason (✨ experimental): CI failed because ctest/AddressSanitizer reported memory leaks during httpserver_tests (test 251), causing the docker exec ... 03_test_script.sh step to exit non-zero.</sub>

    <details><summary>Hints</summary>

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

    </details>

  15. pinheadmz commented at 7:38 PM on July 21, 2026: member

    push to d359b053e189ce85b5f54929334ef94b4a3f370f

    hot fix to the new unit tests which were leaking for the exact reason mentioned by @winterrdog in #35735 (review)

  16. DrahtBot removed the label CI failed on Jul 21, 2026
  17. Ameen-Alam commented at 3:55 PM on July 22, 2026: contributor

    I spent some time comparing fb0a085cc9152725b8152975c2fb8a0898f6939b with the merge base, c8459b6, and found what looks like a change in how MAX_HEADERS_SIZE is enforced. The cumulative interpretation was also established during the review discussion in #32061.

    I sent 25,350 bytes of headers in roughly 1 KiB writes. On the base commit, the server disconnected mid-stream, with my client failing on the ninth write and the server logging HTTP headers exceed size limit. On fb0a085, all 25 writes were accepted and the request was parsed and processed with a 200 OK. When I sent the same headers in a single write, both versions correctly returned 400 Bad Request.

    I think the difference is that the base keeps m_recv_buffer intact when the request is incomplete, so the next call parses everything again and LineReader::Consumed() sees the cumulative size. This branch erases the parsed bytes and keeps the parsing state in m_req, so each new LineReader only counts the bytes received in that cycle. The earlier headers remain in m_headers, but I couldn’t find anything tracking their cumulative size.

    I also tried the split-header case without an Authorization header. The full 25,350 bytes were accepted and Received a POST request for / was logged before the worker returned 401. This might be relevant to the proposed follow-up that moves the auth check after the 8 KiB-limited headers.

    I could also reproduce the chunked parsing issue reported by @brunoerg and earlier by @b-l-u-e in #35182. Splitting immediately before the trailing CRLF returns 200 OK on the base but Cannot parse chunk length value on this branch. Sending it in one write works on both.

    Maybe the header size needs to be tracked cumulatively on HTTPRequest or HTTPRemoteClient, rather than through the LineReader created for each cycle. I have the small probe scripts and would be happy to turn these cases into functional tests.

  18. pinheadmz force-pushed on Jul 22, 2026
  19. pinheadmz commented at 6:14 PM on July 22, 2026: member

    push to 94cd5de8a178d9c6b9d92d8f8cba6ecec00ebf81

    Fix headers size accounting over multiple reads. Great catch thanks @Ameen-Alam I had this for the maximum body size but forgot to implement for headers as well. Covered both with new unit tests

  20. pinheadmz force-pushed on Jul 23, 2026
  21. pinheadmz commented at 6:14 PM on July 23, 2026: member

    push to d59e63729af83d03ebcabaa76914c649c345093a

    Address a regression caught by @l0rinc driving Kimi K3. I had missed a state, which involves the "trailer" in chunked encoding. This data is exactly like headers but comes at the end of the request. In #35182 we just ignored it but by reusing the HTTPHeaders class not only can we parse the data, we can validate it and most importantly, compare the total headers and trailers against the MAX_HEADERS_SIZE limit. The cumulative size of course persists over I/O iterations.

  22. in src/httpserver.cpp:452 in 18b9a1a7aa
     462 | -                    if (maybe_trailer->empty()) break;
     463 | -                }
     464 | -                // Complete request has been parsed, reader is now pointing
     465 | -                // to beginning of next request or end of the buffer.
     466 | -                return true;
     467 | +                return m_headers.Read(reader);
    


    winterrdog commented at 10:46 PM on July 23, 2026:

    18b9a1a http: reuse HTTPHeaders to parse chunked trailer:

    just curious. a tangential question came to mind while staring at the trailer support: now that trailer fields and regular headers both end up in the same m_headers via HTTPHeaders::Read(), is it worth considering whether they should remain distinguishable ?

    the reason i ask is that RFC 9110 6.5.1 notes that trailer fields cannot be given the same trust as header fields, since they arrive after the body and intermediaries are allowed to remove or ignore them. with the current merge, GetHeader() cannot tell which section a field originally came from. i tried a quick Boost test with a chunked request that omitted Authorization from the real header block but included it as a trailer, and GetHeader("Authorization") found it just like any other header.

    <details><summary>this is the test i used</summary>

     
    BOOST_AUTO_TEST_CASE(http_request_state_tests)
    {
         // ...existing tests in the current commit...
    
         {
             std::shared_ptr<DummyClient> client{std::make_shared<DummyClient>()};
             client->m_req = std::make_unique<HTTPRequest>(client);
             
             // real header block deliberately omits 'Authorization'.
             // the only 'Authorization' field arrives later, as a chunk trailer.
             client->receive("POST / HTTP/1.0\n"
                             "Host: 127.0.0.1\n"
                             "Transfer-Encoding: chunked\n"
                             "\n"
                             "1\n"
                             "x\n"
                             "0\n"
                             "Authorization: Basic ALSKDjldsscj\n"
                             "\n");
             client->ReadRequest(*client->m_req);
             BOOST_CHECK_EQUAL(client->m_req->GetState(), HTTPRequest::State::Complete);
    
             // this finds the field; a trailer-supplied Authorization is
             // indistinguishable from a real one to any caller of GetHeader()
             auto [found, value] = client->m_req->GetHeader("Authorization");
             BOOST_CHECK_MESSAGE(!found,
                 "'Authorization' supplied only as a chunk trailer field was accepted "
                 "by GetHeader() as if it were a real header (value: " + value + ")");
    
             client->ReleaseRequest();
         }
    }
    

    </details>

    i do not think this is an auth bypass by itself because without valid credentials the request still gets a 401 regardless (just a small illustration of what i think). i was mostly wondering about deployments where another component (e.g. a proxy, WAF, or monitoring layer) only considers the initial header block. in that case, bitcoind and the intermediary could end up disagreeing about whether a request carried a particular header at all. the same thought would apply to other headers related to message framing, routing, request modifiers, response controls, or content format

    maybe this is not something worth worrying about, but i figured i would ask while the trailer changes are still fresh :). so, would it make sense to keep trailer fields separate (or otherwise avoid returning them from GetHeader() for certain lookups), or is merging them into m_headers just fine ?


    pinheadmz commented at 12:27 AM on July 24, 2026:

    A few thoughts:

    • We could add another member to HTTPRequest that's another HTTPHeaders called m_trailer, and give it a separate limit to m_headers (or still sum them during chunked encoding parsing)
    • We could add an ignore argument to HTTPHeaders::Read() that it does all the validation and size accounting but still ignore the actual data there.

    Ultimately my big plan, with the state machine, was to handle RPC auth after LoadHeaders() so that we don't even read the up-to-32MB-body without authorization. Then trailer data would be ignored.


    winterrdog commented at 9:39 PM on July 25, 2026:

    while looking into how other web servers handle trailers in chunked requests, i checked nginx and HAProxy since both are widely deployed and well battle-tested in this area. nginx ignores trailers by default, only parsing or forwarding them when that behavior is explicitly enabled. HAproxy stores & forwards them by default since it is meant to be used as a proxy server, but it can be turned off. in standalone or non-proxy mode, they simply drop them.

    that seems to map nicely to our situation. bitcoind is not acting as a proxy, and i do not see a use case where we would consume trailer fields ourselves. given that, i actually think your original approach of just ignoring trailers was the right one, and your last paragraph (plan for the future) reinforces that reasoning

    We could add an ignore argument to HTTPHeaders::Read() that it does all the validation and size accounting but still ignore the actual data there.

    i am truly on board with this. it preserves the part we actually care about (validation and size accounting) while dropping the part that introduces ambiguity. it will also avoid introducing a second HTTPHeaders member (such as m_trailers) just to store data we never read

    as for the argument's name, we could call it sth like: keep_trailers or ignore_trailers; your call, really :)

    handle RPC auth after LoadHeaders() so that we don't even read the up-to-32MB body without authorization. Then trailer data would be ignored.

    furthermore, i think this settles it. once authorisation happens right after headers, trailers arrive after that decision has already been made, so there is no reasonable path where trailer content should ever influence anything hence they can be ignored

  23. pinheadmz force-pushed on Jul 27, 2026
  24. pinheadmz commented at 3:37 PM on July 27, 2026: member

    push to 0ced8c35ae49f71eb42b549d6a7870e2fab93c49

    address feedback from @winterrdog about reusing HTTPHeaders for trailer data. Simple behavior change, adding a bool write to HTTPHeaders::Read() so when reading actual headers we validate and then save the data (used for authorization among other things) but for chunked encoding trailers we validate and then ignore the data (so the amount of data counts towards the MAX_HEADERS_SIZE limit, but without any possible conflicts with actual headers)

  25. in src/httpserver.cpp:305 in 0ced8c35ae outdated
     302 |      // Headers https://httpwg.org/specs/rfc9110.html#rfc.section.6.3
     303 |      // A sequence of Field Lines https://httpwg.org/specs/rfc9110.html#rfc.section.5.2
     304 | +    size_t start{reader.Consumed()};
     305 |      while (auto maybe_line = reader.ReadLine()) {
     306 | -        if (reader.Consumed() > MAX_HEADERS_SIZE) throw std::runtime_error("HTTP headers exceed size limit");
     307 | +        if (reader.Consumed() - start + m_consumed > MAX_HEADERS_SIZE) throw std::runtime_error("HTTP headers exceed size limit");
    


    winterrdog commented at 12:17 AM on July 29, 2026:

    only a small observability note.

    now that trailer parsing goes through the same HTTPHeaders::Read() as regular headers, an oversized trailer section will throw the same error & message as an oversized header block.

    before this reuse, that case had its own message ("HTTP chunked trailer exceeds size limit"), so anyone reading logs could tell which section actually blew the limit. functionally nothing changes, but is losing that distinction intentional, or worth keeping around (e.g. catching it at the trailer call site and rethrowing with the old wording, or passing some context into Read() for the message) ?

    <details><summary>the old excerpt of interest</summary>

    +if (*m_chunk_size == 0) {
    +     // Validate Chunked Trailer section, which is used for
    +     // additional headers sent at the end of the message.
    +     // Data consumed here is counted towards MAX_HEADERS_SIZE
    +     // along with the headers we read in the beginning of the request.
    +     // At this time we ignore and drop these data after validating.
          // See https://httpwg.org/specs/rfc9112.html#rfc.section.7.1.2
    -     const size_t trailer_start{reader.Consumed()};
    -     while (true) {
    -          auto maybe_trailer = reader.ReadLine();
    -          if (reader.Consumed() - trailer_start > MAX_HEADERS_SIZE) {
    -              throw std::runtime_error("HTTP chunked trailer exceeds size limit");
    -          }
    -          if (!maybe_trailer) return false;
    -          if (maybe_trailer->empty()) break;
    -     }
    -     // Complete request has been parsed, reader is now pointing
    -     // to beginning of next request or end of the buffer.
    -     return true;
    +     return m_headers.Read(reader, /*write=*/false);
    }
    

    </details>

    not a big deal either way. just thought it was worth pointing out in case that distinction ends up being useful when debugging misbehaving clients or proxies down the road


    pinheadmz commented at 2:04 PM on July 29, 2026:

    The old code also effectively gave the trailers section its own independent limit, so this PR does make a functional change and I think only having one error message to cover that limit is ok. The error message sent to the client also changes and arguably improves in this PR: an excessive trailer got a generic 400 before but now it'll get a 413 Entity Too Large.

    I doubt any bitcoin clients take advantage of chunked transfer encoding anyway, and if we do ever add support for trailer data in the future we can revisit this.


    winterrdog commented at 2:30 PM on July 29, 2026:

    ah! that makes sense. it was not much of a big deal too. this is resolved

  26. winterrdog commented at 12:28 AM on July 29, 2026: contributor

    approach ACK

  27. in src/test/httpserver_tests.cpp:566 in 0ced8c35ae outdated
     580 | +        BOOST_CHECK_EQUAL(client->m_req->m_target, "/endpoint");
     581 | +        BOOST_CHECK_EQUAL(client->m_req->m_body.size(), 0);
     582 | +        // Buffer is cleared
     583 | +        BOOST_CHECK_EQUAL(client->m_recv_buffer.size(), 0);
     584 | +
     585 | +        client->ReleaseRequest();
    


    janb84 commented at 12:18 PM on August 4, 2026:
            client->ReleaseRequest();
        }
        {
            // A Content-Length body is drained out of the receive buffer as it
            // arrives, instead of accumulating there until the request is complete.
    
            std::shared_ptr<DummyClient> client{std::make_shared<DummyClient>()};
            client->m_req = std::make_unique<HTTPRequest>(client);
            BOOST_CHECK_EQUAL(client->m_req->GetState(), HTTPRequest::State::Init);
    
            client->receive("POST / HTTP/1.0\n"
                            "Content-Length: 30000\n\n");
            client->ReadRequest(*client->m_req);
            BOOST_CHECK_EQUAL(client->m_req->GetState(), HTTPRequest::State::NeedsBody);
    
            // Body arrives in 10kB pieces. Each one is copied onto m_body and
            // erased from the receive buffer, which never holds more than one piece.
            for (int i = 1; i <= 3; ++i) {
                client->receive(std::string(10000, 'x'));
                BOOST_CHECK_EQUAL(client->m_recv_buffer.size(), 10000);
                client->ReadRequest(*client->m_req);
                BOOST_CHECK_EQUAL(client->m_req->m_body.size(), 10000 * i);
                BOOST_CHECK_EQUAL(client->m_recv_buffer.size(), 0);
            }
            BOOST_CHECK_EQUAL(client->m_req->GetState(), HTTPRequest::State::Complete);
    
            client->ReleaseRequest();
        }
        {
            // A body sent in the same push as the next request is split correctly
            std::shared_ptr<DummyClient> client{std::make_shared<DummyClient>()};
            client->m_req = std::make_unique<HTTPRequest>(client);
            BOOST_CHECK_EQUAL(client->m_req->GetState(), HTTPRequest::State::Init);
    
            client->receive("POST / HTTP/1.0\n"
                            "Content-Length: 4\n\n"
                            "body"
                            "GET /next HTTP/1.0\n\n");
            client->ReadRequest(*client->m_req);
            BOOST_CHECK_EQUAL(client->m_req->GetState(), HTTPRequest::State::Complete);
            BOOST_CHECK_EQUAL(client->m_req->m_body, "body");
            // Only the second request is left over
            BOOST_CHECK_EQUAL(client->m_recv_buffer.size(), 20);
    
            client->ReleaseRequest();
    
  28. in src/httpserver.cpp:528 in 0ced8c35ae


    janb84 commented at 12:24 PM on August 4, 2026:
    
            // A large body may arrive over multiple I/O loop iterations. Copy
            // whatever the buffer has now; m_body's size tracks our progress.
            const uint64_t body_need{*content_length - m_body.size()};
            const uint64_t buffer_has{std::min(body_need, static_cast<uint64_t>(reader.Remaining()))};
    
            // Pack [partial] body on and update state
            m_body += reader.ReadLength(buffer_has);
    
            return m_body.size() == *content_length;
    

    frankomosh commented at 11:30 AM on August 5, 2026:

    client->m_disconnect has no guard (same way is_idle is gated on !client→m_req_busy), and since MaybeDispatchRequestsFromClient() now calls ReadRequest() before checking m_req_busy, a second request that fails to parse while a first is still on a worker could erase the client mid-flight. WriteReply() never checks m_disconnect, so the worker's reply would still gets queued and attempted. However, if that send doesn't drain in one call, it wouldn’t be be revisited, since the I/O thread's send-poll only iterates m_connected. The reply would then be silently dropped once the last shared_ptr<HTTPRemoteClient> reference releases and the socket closes.

    I’m not sure if this sequence poses any threat but I think its only a UX concern, especially for a client that pipelines by design, or in a situation where they might be having a framing bug.

    Sequence tested locally: I dispatched request A and held busy, a failing B pipelined behind it, confirmed through GetConnectionsCount() the client is erased while A is still busy, confirmed via a DebugLogHelper on WriteReply()'s log line that A's worker genuinely queues a reply, and those bytes never reach the client.

    <details> <summary>A candidate fix(or more of a starting point):</summary>

    -                                        if (client->m_disconnect || is_idle) {
    +                                        if (client->m_req_busy) {
    +                                            return false;
    +                                        } else if (client->m_disconnect || is_idle) {
    

    </details>

    This proposed ‘fix’ is just a best-effort. This is because recv() still keeps appending to m_recv_buffer for a disconnect-flagged client for the duration of the deferral, since nothing currently guards that path either.


    winterrdog commented at 12:52 PM on August 6, 2026:

    Sequence tested locally: I dispatched request A and held busy, a failing B pipelined behind it, confirmed through GetConnectionsCount() the client is erased while A is still busy, confirmed via a DebugLogHelper on WriteReply()'s log line that A's worker genuinely queues a reply, and those bytes never reach the client.

    sounds like out-of-order processing. eager parsing seems to be the issue here


    pinheadmz commented at 1:45 PM on August 6, 2026:

    Sequence tested locally:

    Do you have code for this I can look at?

    I understand the scenario you're describing: a request is sent to a worker, and a new request is being parsed. The new request has an error, m_disconnected is set true, and we log a message that the client is disconnected.

    My doubt is this: the worker thread executing the request still has a pointer to the client and so even though the main I/O thread is done with the client, the socket should still be connected until that pointer is dropped at the end of the RPC (after WriteReply() returns).


    frankomosh commented at 6:05 PM on August 6, 2026:

    Do you have code for this I can look at?

    Yes

    <details> <summary>a test code I had used</summary>

    @@ -1059,4 +1119,126 @@ BOOST_AUTO_TEST_CASE(http_socket_error_tests)
         server.StopListening();
     }
     
    +BOOST_AUTO_TEST_CASE(http_busy_worker_disconnect_tests)
    +{
    +    // Reproduces: a malformed second request, pipelined on a keep-alive connection
    +    // behind a first valid request still being handled by a worker, can set
    +    // client->m_disconnect. Unlike is_idle, the m_disconnect branch in
    +    // DisconnectClients()'s erase-if is not guarded by m_req_busy, so the client
    +    // can be erased from m_connected while still busy. If the busy worker's
    +    // eventual WriteReply() can't fully flush via the optimistic send, nothing
    +    // ever polls its m_send_buffer again, since the I/O loop's send-poll only
    +    // iterates m_connected – silently, losing the reply.
    +
    +    ThreadPool workers("http");
    +    workers.Start(1);
    +
    +    std::atomic<bool> release_worker{false};
    +    std::atomic<bool> worker_started{false};
    +
    +    HTTPServer server{[&](std::unique_ptr<HTTPRequest>&& req) {
    +        std::shared_ptr<HTTPRequest> shared_req{std::move(req)};
    +        auto item = [shared_req, &release_worker, &worker_started]() {
    +            worker_started = true;
    +            // Simulate a slow worker: block until the test thread releases us,
    +            // matching this file's existing poll-loop idiom rather than a
    +            // condition_variable.
    +            while (!release_worker) {
    +                std::this_thread::sleep_for(10ms);
    +            }
    +            shared_req->WriteReply(HTTP_OK, "reply from busy worker\n");
    +        };
    +        Assert(workers.Submit(std::move(item)));
    +    }};
    +    server.InitHTTPAllowList();
    +
    +    // Mocked socket that forces exactly one Send() containing "200 OK" to fail
    +    // with a recoverable error, simulating a stalled optimistic send. All other
    +    // sends (e.g. B's 400 reply) succeed normally.
    +    class ErrorSock : public DynSock
    +    {
    +    public:
    +        explicit ErrorSock(std::shared_ptr<Pipes> pipes) : DynSock{std::move(pipes)} {}
    +        DynSock& operator=(Sock&&) override { assert(false); return *this; }
    +        ssize_t Send(const void* buf, size_t len, int flags) const override
    +        {
    +            std::string_view data(static_cast<const char*>(buf), len);
    +            if (!m_have_blocked && data.find("200 OK") != std::string_view::npos) {
    +                m_have_blocked = true;
    +                #ifdef WIN32
    +                WSASetLastError(WSAEWOULDBLOCK);
    +                #else
    +                errno = WSAEAGAIN;
    +                #endif
    +                return -1;
    +            }
    +            return DynSock::Send(buf, len, flags);
    +        }
    +        mutable bool m_have_blocked{false};
    +    };
    +
    +    CService addr_bind{Lookup("0.0.0.0", /*portDefault=*/0, /*fAllowLookup=*/false).value()};
    +    BOOST_REQUIRE(server.BindAndStartListening(addr_bind));
    +    server.StartSocketsThreads();
    +
    +    // Request A: valid, keep-alive, will be dispatched to the (blocked) worker.
    +    std::string request_a{full_request};
    +    request_a.replace(request_a.find("Connection: close"), 17, "Connection: keep-alive");
    +
    +    // Request B: malformed (header with no colon), pipelined right behind A.
    +    std::string request_b = "GET /malformed HTTP/1.1\r\n"
    +                            "Invalid header with no colon\r\n"
    +                            "\r\n";
    +
    +    std::string combined = request_a + request_b;
    +    std::shared_ptr<ErrorSock::Pipes> mock_client_socket_pipes{
    +        ConnectClient<ErrorSock>(std::as_bytes(std::span(combined)))
    +    };
    +
    +    // Wait for A to be dispatched (m_req_busy is true from this point on,
    +    // until the worker itself calls WriteReply()).
    +    int attempts = 6000;
    +    while (!worker_started) {
    +        std::this_thread::sleep_for(10ms);
    +        BOOST_REQUIRE(--attempts > 0);
    +    }
    +
    +    // Wait for the connection to disappear from m_connected. Since the worker
    +    // is still blocked (release_worker is false), this can only be B's parse
    +    // error setting m_disconnect -- while the client was still busy.
    +    attempts = 6000;
    +    while (server.GetConnectionsCount() != 0) {
    +        std::this_thread::sleep_for(10ms);
    +        BOOST_REQUIRE(--attempts > 0);
    +    }
    +
    +    // Confirmed: erased from m_connected while m_req_busy was still true.
    +    // Now release A's worker. This DebugLogHelper hard-aborts if the worker
    +    // never actually reaches WriteReply() -- proving the reply had been genuinely
    +    // queued, and not just absent from the wire for some unrelated reason.
    +    DebugLogHelper find_worker_reply{"status code: 200"};
    +    release_worker = true;
    +
    +    // Give the worker's WriteReply()/optimistic-send attempt (which we've
    +    // forced to fail once) time to run, and give the I/O loop several ticks
    +    // to see whether anything ever revisits this client's send buffer.
    +    std::this_thread::sleep_for(500ms);
    +
    +    std::string actual;
    +    char buf[0x10000] = {};
    +    ssize_t bytes_read = mock_client_socket_pipes->send.GetBytes(buf, sizeof(buf), 0);
    +    if (bytes_read > 0) actual.append(buf, bytes_read);
    +
    +    // The point of the test: does A's reply ever reach the client, or is it
    +    // permanently stuck in a send buffer nobody polls anymore?
    +    BOOST_CHECK_MESSAGE(actual.find("200 OK") == std::string::npos,
    +        "Expected A's reply to be lost after the client was erased from "
    +        "m_connected while still busy -- if this fails (reply WAS found), "
    +        "either the hypothesis is wrong or something else is delivering it.");
    +
    +    workers.Stop();
    +    server.InterruptNet();
    +    server.JoinSocketsThreads();
    +    server.StopListening();
    +}
    

    </details>

    the worker thread executing the request still has a pointer to the client and so even though the main I/O thread is done with the client, the socket should still be connected until that pointer is dropped at the end of the RPC (after WriteReply() returns).

    The socket stays connected until that pointer drops, yes. but that's the same instant WriteReply() returns. Before then, I believe only one send() happens: the optimistic one inside WriteReply() itself. If that doesn't fully drain, I don't think anything comes back to retry it, since the I/O thread's poll only covers m_connected.


    pinheadmz commented at 6:37 PM on August 6, 2026:

    Ah, I see, yeah if the worker can't optimistic-send then the I/O loop won't either... Ok thanks for walking through that with me I'll review your test.


    winterrdog commented at 7:29 PM on August 6, 2026:

    modified a copy of the check_pipelining functional test from interface_http.py into check_pipelined_malformed_disconnect and the issue showed up

    <details> <summary>the functional test i used </summary>

    diff --git a/test/functional/interface_http.py b/test/functional/interface_http.py
    index 1350015258..3402a86c46 100755
    --- a/test/functional/interface_http.py
    +++ b/test/functional/interface_http.py
    @@ -117,6 +117,7 @@ class HTTPBasicsTest (BitcoinTestFramework):
             self.check_close_connection()
             self.check_excessive_request_size()
             self.check_pipelining()
    +        self.check_pipelined_malformed_disconnect()
             self.check_chunked_transfer()
             self.check_idle_timeout()
             self.check_server_busy_idle_timeout()
    @@ -131,6 +132,50 @@ class HTTPBasicsTest (BitcoinTestFramework):
             self.check_invalid_http_version()
             self.check_whitespace_in_headers()
    
    +    def check_pipelined_malformed_disconnect(self):
    +        """
    +        Regression check: a malformed pipelined request must not cause the
    +        server to drop the reply to an earlier, still-in-flight valid request
    +        on the same connection.
    +        """
    +
    +        self.log.info("+ check malformed pipelined request does not drop a busy reply")
    +        tip_height = self.node.getblockcount()
    +        conn = BitcoinHTTPConnection(self.node)
    +        conn.set_timeout(5)
    +
    +        # A: slow, valid request. blocks server-side until a new block comes back
    +        conn.post_raw('/', f'{{"method": "waitforblockheight", "params": [{tip_height + 1}]}}')
    +
    +        # B: send valid HTTP request line/headers with an invalid header field pipelined
    +        #    just after A, while A is still busy so as to force an instant 400 parse error
    +        malformed_b = (
    +            b"POST / HTTP/1.1\r\n"
    +            b"Host: 127.0.0.1\r\n"
    +            b"Very-Very-Bad-Header-With-No-Colon\r\n\r\n"
    +        )
    +        conn.send_raw(malformed_b)
    +
    +        # right now, A has not yet been unblocked, so nothing should come back, otherwise that is a BUG
    +        try:
    +            early_data = conn.recv_raw()
    +            assert False, f"+ server sent data prematurely while A was busy (NOT nice): {early_data!r}"
    +        except TimeoutError:
    +            pass
    +
    +        # unblock A
    +        self.generate(self.node, 1, sync_fun=self.no_op)
    +
    +        # A's reply should still arrive. If the bug is present, the server
    +        # disconnected the client back when B failed to parse (while A's worker
    +        # was still onto sth), silently dropping A's queued reply -- recv_raw()
    +        # would then time out or just return nothing instead
    +        try:
    +            res = conn.recv_raw()
    +        except TimeoutError:
    +            assert False, "A's reply was dropped: server disconnected while A was still busy handling B's parse failure"
    +
    +        assert b'"hash"' in res, f"expected waitforblockheight's reply, got: {res!r}"
    +
    
         def check_default_connection(self):
             self.log.info("Checking default HTTP/1.1 connection persistence")
    

    </details>

    <details> <summary>output i got </summary>

    Temporary test directory at /tmp/test_runner_₿_🏃_20260806_221242
    Remaining jobs: [interface_http.py]
    1/1 - interface_http.py failed, Duration: 2 s
    
    stdout:
    2026-08-06T19:12:43.015370Z TestFramework (INFO): PRNG seed is: 3727582087972046593
    2026-08-06T19:12:43.066100Z TestFramework (INFO): Initializing test directory /tmp/test_runner_₿_🏃_20260806_221242/interface_http_0
    2026-08-06T19:12:45.221845Z TestFramework (INFO): + check malformed pipelined request does not drop a busy reply
    2026-08-06T19:12:45.273692Z TestFramework (ERROR): Unexpected exception:
    Traceback (most recent call last):
      File "/home/hacked/Documents/btc/btc-core/my-btc-fork/test/functional/test_framework/test_framework.py", line 145, in main
        self.run_test()
      File "/home/hacked/Documents/btc/btc-core/my-btc-fork/build/test/functional/interface_http.py", line 120, in run_test
        self.check_pipelined_malformed_disconnect()
      File "/home/hacked/Documents/btc/btc-core/my-btc-fork/build/test/functional/interface_http.py", line 161, in check_pipelined_malformed_disconnect
        assert False, f""+ server sent data prematurely while A was busy (NOT nice): {early_data!r}"
               ^^^^^
    AssertionError: "+ server sent data prematurely while A was busy (NOT nice): b'HTTP/1.1 400 Bad Request\r\nDate: Thu, 06 Aug 2026 19:12:45 GMT\r\nContent-Length: 0\r\nContent-Type: text/html; charset=ISO-8859-1\r\n\r\n'
    2026-08-06T19:12:45.327671Z TestFramework (INFO): Not stopping nodes as test failed. The dangling processes will be cleaned up later.
    2026-08-06T19:12:45.328113Z TestFramework (WARNING): Not cleaning up dir /tmp/test_runner_₿_🏃_20260806_221242/interface_http_0
    2026-08-06T19:12:45.328342Z TestFramework (ERROR): Test failed. Test logging available at /tmp/test_runner_₿_🏃_20260806_221242/interface_http_0/test_framework.log
    2026-08-06T19:12:45.328713Z TestFramework (ERROR):
    2026-08-06T19:12:45.329174Z TestFramework (ERROR): Hint: Call /home/hacked/Documents/btc/btc-core/my-btc-fork/test/functional/combine_logs.py '/tmp/test_runner_₿_🏃_20260806_221242/interface_http_0' to consolidate all logs
    ...
    

    </details>

    -- the request processing does not seem strictly serial

    rough idea for a fix:

    how about if we checked if the server was still processing a request from the client at these spots:

    1. before parsing another request (in HTTPServer::MaybeDispatchRequestsFromClient) and,
    2. before disconnecting a client while a request is still being handled (in HTTPServer::DisconnectClients())

    <details> <summary>sth like this </summary>

    diff --git a/src/httpserver.cpp b/src/httpserver.cpp
    index 288a8a477e..b3b4c66a88 100644
    --- a/src/httpserver.cpp
    +++ b/src/httpserver.cpp
    @@ -1010,6 +1010,11 @@ void HTTPServer::ThreadSocketHandler()
    
     void HTTPServer::MaybeDispatchRequestsFromClient(const std::shared_ptr<HTTPRemoteClient>& client) const
     {
    +    // If we are already handling a request from
    +    // this client, do nothing. We'll check again on the next I/O
    +    // loop iteration.
    +    if (client->m_req_busy) return;
    +
         if (!client->m_req) {
             client->m_req = std::make_unique<HTTPRequest>(client);
         }
    @@ -1042,11 +1047,6 @@ void HTTPServer::MaybeDispatchRequestsFromClient(const std::shared_ptr<HTTPRemot
             return;
         }
    
    -    // If we are already handling a request from
    -    // this client, do nothing. We'll check again on the next I/O
    -    // loop iteration.
    -    if (client->m_req_busy) return;
    -
         // Otherwise, if the request is ready, hand it to a worker.
         if (client->m_req->GetState() == HTTPRequest::State::Complete) {
             LogDebug(
    @@ -1068,6 +1068,10 @@ void HTTPServer::DisconnectClients()
         const auto now{Now<SteadySeconds>()};
         size_t erased = std::erase_if(m_connected,
                                       [&](auto& client) {
    +                                        // Don't disconnect a client if the server is busy with its request in order to avoid premature disconnection
    +                                        if (client->m_req_busy) {                                                                              
    +                                            return false;
    +                                        }
                                             // First check for idle timeout. We reset the timer when we send and receive data,
                                             // but if the server is busy handling a request we should ignore the timeout until
                                             // the reply is sent. If we did erase the shared_ptr<HTTPRemoteClient> reference in m_connected
    

    </details>

    thoughts ?


    frankomosh commented at 7:23 AM on August 11, 2026:
                                            // Don't erase a client while a reply is still queued to send,
                                            // erasing here drops m_connected as the only thing that retries
                                            // MaybeSendBytesFromBuffer() on a later I/O tick.
                                            {
                                                LOCK(client->m_send_mutex);
                                                if (!client->m_send_buffer.empty() && !is_idle) return false;
                                            }
                                            if (client->m_disconnect || is_idle) {
                                            
    

    Even with the fix(to the issue I raised earlier), we can still have a small window right after a worker finishes where the same kind of drop can still happen, later on maybe. Perhaps we could add one more check: don't disconnect a client while it still has a reply waiting to go out, unless it's also been idle a while (so a client that never reads anything back still eventually times out normally, instead of being held open forever).


    frankomosh commented at 7:48 AM on August 11, 2026:

    thoughts ?

    Sorry for the slow follow-up here. I tested the fix and I think it works well. On your second suggested change (if (client->m_req_busy) return false; guard inside DisconnectClients(), I am not too sure because by the time the danger window opens, m_req_busy is already false (that's the whole point of the reorder fix), so I doubt if checking it again in DisconnectClients() would catch anything ?

  29. janb84 commented at 12:41 PM on August 4, 2026: contributor

    Transfer-Encoding: chunked is now fixed but I think there is a similar issue with Transfer-Encoding: content-length. It consumes nothing until the entire declared body has accumulated in m_recv_buffer, then copies it to m_body in one shot. Draining the buffer each cycle with m_body.size() tracking progress mirrors what the chunked branch already does, and keeps the receive buffer at one socket read.

  30. in src/test/httpserver_tests.cpp:497 in 0ced8c35ae outdated
     511 | -        BOOST_CHECK(req.LoadHeaders(reader2));
     512 | -        BOOST_CHECK(req.LoadBody(reader2));
     513 | +    public:
     514 | +            DummyClient() : HTTPRemoteClient{/*id=*/0, /*addr=*/CService(), /*socket=*/CreateSock(0, 0, 0)} {}
     515 | +
     516 | +            void receive(std::string_view s)
    


    frankomosh commented at 2:46 PM on August 5, 2026:
                void receive(std::string_view s)
                {
                    m_recv_buffer += s;
                }
    

    I think dff44e4c8f refactored HTTPRemoteClient::m_recv_buffer from std::vector<std::byte> to std::string, and this still inserts std::bytes into it ?


    frankomosh commented at 4:46 PM on August 5, 2026:

    Sorry, I think this is fixed now

  31. pinheadmz force-pushed on Aug 5, 2026
  32. pinheadmz commented at 3:02 PM on August 5, 2026: member

    push to c863cff2547dcfbb13a526aaf0f783482b913504

    Reviewed and then applied code suggestions from @janb84. New behavior reads large request bodies incrementally over multiple I/O loop iterations, similar to chunked transfer-encoding.

  33. pinheadmz force-pushed on Aug 5, 2026
  34. DrahtBot added the label CI failed on Aug 5, 2026
  35. DrahtBot commented at 3:21 PM on August 5, 2026: contributor

    <!--85328a0da195eb286784d51f73fa0af9-->

    🚧 At least one of the CI tasks failed. <sub>Task ASan + LSan + UBSan + integer: https://github.com/bitcoin/bitcoin/actions/runs/31018056099/job/92347251023</sub> <sub>LLM reason (✨ experimental): CI failed due to a C++ compilation error in httpserver_tests.cpp (passing const std::byte* to std::string operations, causing “no matching function for call to assign/insert”).</sub>

    <details><summary>Hints</summary>

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

    </details>

  36. pinheadmz force-pushed on Aug 5, 2026
  37. pinheadmz commented at 3:38 PM on August 5, 2026: member

    push to d354b72b56

    • Rebase on master to fix silent conflict with #35828 which made m_recv_buffer a string, so the reinterpret_cast<const std::byte*> in the DummyClient was broken.
  38. pinheadmz force-pushed on Aug 5, 2026
  39. frankomosh commented at 4:44 PM on August 5, 2026: contributor

    Concept ACK

  40. DrahtBot removed the label CI failed on Aug 5, 2026
  41. janb84 commented at 6:30 PM on August 5, 2026: contributor

    ACK d354b72b56304f4b542086b8d6b14b4233f4ccb4

    LGTM ! I'm happy with the current state of this PR.

    The PR adds a message-parsing state machine for a single HTTP request. The state machine to is a clear improvement (imho) over re-parsing the buffer every I/O cycle, and draining the body as it arrives. Clear memory advantages as stated in the PR description.

    Thanks for incorporating my suggestions

  42. DrahtBot requested review from brunoerg on Aug 5, 2026
  43. DrahtBot requested review from winterrdog on Aug 5, 2026
  44. DrahtBot requested review from frankomosh on Aug 5, 2026
  45. in src/httpserver.cpp:525 in d354b72b56 outdated
     523 | +        const uint64_t body_need{*content_length - m_body.size()};
     524 | +        const uint64_t buffer_has{std::min(body_need, static_cast<uint64_t>(reader.Remaining()))};
     525 |  
     526 | -        m_body = reader.ReadLength(*content_length);
     527 | +        // Pack [partial] body on and update state
     528 | +        m_body += reader.ReadLength(buffer_has);
    


    winterrdog commented at 9:44 PM on August 5, 2026:

    premise: pre-allocation

    due to repeated allocations on every IO cycle, we can allocate the space needed upfront -- since *content_length is already known & validated before this code actually runs thus we would have exactly one allocation vs the arbitrary ones we would have if we relied on std::string's own amortised growth strategy -- this is an efficiency the previous copy-body-in-one-shot code had. we can also get a bit of cache locality from it

    <details> <summary>diff </summary>

    diff --git a/src/httpserver.cpp b/src/httpserver.cpp
    index 288a8a477e..a97f810c55 100644
    --- a/src/httpserver.cpp
    +++ b/src/httpserver.cpp
    @@ -516,6 +516,11 @@ bool HTTPRequest::LoadBody(LineReader& reader)
    
             if (*content_length > MAX_BODY_SIZE) throw ContentTooLargeError("Max body size exceeded");
    
    +        // Reserve the full body size up front so incremental appends below
    +        // do not repeatedly reallocate/copy as more data arrives across I/O
    +        // loop cycles
    +        m_body.reserve(*content_length);
    +
             // A large body may arrive over multiple I/O loop iterations. Copy
             // whatever the buffer has now; m_body's size tracks our progress.
             const uint64_t body_need{*content_length - m_body.size()};
    

    </details>

    [!NOTE] reserve() is a no-op once capacity() already covers content_length, so it is safe to call on every I/O cycle with no extra guard e.g. if (m_body.empty()) ... or if (m_body.capacity() < *content_length) ...

    thoughts ?


    janb84 commented at 8:47 AM on August 6, 2026:

    No we should not do this. The pre-allocation is intentionally left out.

    since *content_length is already known & validated

    Is it ? I do not see a gate in the code checking that client had already transmitted all the bytes that where advertised to been send in content_length. Yes we establish that *content_length <= MAX_BODY_SIZE but that says nothing about whether the client intends to send that much.

    So introducing m_body.reserve(*content_length) is setting a reserve() on a number that the peer picked, before the peer has sent anything. Thus re-enabling the option to reserve 32MiB of memory with 45 bytes eg:

    POST / HTTP/1.1\r\nContent-Length: 33554432\r\n\r\n
    

    winterrdog commented at 11:28 AM on August 6, 2026:

    Yes we establish that *content_length <= MAX_BODY_SIZE but that says nothing about whether the client intends to send that much.

    So introducing m_body.reserve(*content_length) is setting a reserve() on a number that the peer picked, before the peer has sent anything

    yes, indeed!

    that is a fair point. we cannot solely rely on content_length to imply the client actually has that much data ready to send. good catch (it skipped my mind).

    even a bounded version like:

    m_body.reserve(std::min(*content_length, m_body.size() + static_cast<uint64_t>(reader.Remaining())));
    

    only really helps when most of the body arrives in one or a few large reads. in the slower or more adversarial case, it buys us little over the present approach

    also, std::string already grows amortised, so the current += path is already fairly efficient without any pre-allocating

    i do not think the narrow benefit is worth the extra complexity, so i greatly agree that we should leave it as-is

    marking this resolved. thanks for pointing that out, @janb84

  46. DrahtBot requested review from winterrdog on Aug 5, 2026
  47. pinheadmz force-pushed on Aug 7, 2026
  48. pinheadmz commented at 12:22 PM on August 7, 2026: member

    push to d86fe8e6392e06a640654233bde221619e2c6f35

    • rebase on master to check for silent conflicts
    • a handful of code style fixes and added test assertions suggested by claude
    • Biggest change is addressing the pipelining issue brought up by @frankomosh I ended up coming to the same conclusion as @winterrdog which was to simply not parse any new requests from a client while one is busy. The current code would have a parsed request "on deck" but if it raised a parsing error, the pipeline would be broken because the error would be sent before the older request response. @frankomosh the unit test you wrote was helpful but I found it simpler to cover this behavior in a functional test. There's already a pipeline test in interface_http so I just run it a second time with an invalid request in the pipeline. That test will fail on master, fixed in a new commit on this branch.
  49. in test/functional/interface_http.py:308 in d86fe8e639
     306 |          # waitforblockheight was responded to first, and then getblockcount
     307 |          # which includes the block added after the request was made
     308 |          chunks = res.split(b'"result":')
     309 |          assert chunks[1].startswith(b'{"hash":')
     310 | -        assert chunks[2].startswith(bytes(f'{tip_height + 1}', 'utf8'))
     311 | +        if not with_invalid_second_request:
    


    janb84 commented at 7:03 PM on August 7, 2026:

    NIT: remove the NOT and add a assert to check that the requests are responded to in the correct order as stated in the docstring: "requests are responded to in the order in which they were received, see RFC 7230 6.3.2""

            if with_invalid_second_request:
                # The response to the in-flight first request is sent before the
                # error generated by parsing the second one, even though the second
                # request could have been rejected much earlier.
                assert res.index(b"HTTP/1.1 200") < res.index(b"HTTP/1.1 400")
            else:
    

    pinheadmz commented at 7:17 PM on August 7, 2026:

    Great idea, taking

  50. janb84 commented at 7:10 PM on August 7, 2026: contributor

    re ACK d86fe8e6392e06a640654233bde221619e2c6f35

    small nit suggestion to add an assert in a test.

  51. pinheadmz force-pushed on Aug 7, 2026
  52. pinheadmz commented at 7:18 PM on August 7, 2026: member

    push to 61df8b59adce705c82800d55528edb9a472413bb

    functional test improvement suggested by @janb84

  53. janb84 commented at 7:26 PM on August 7, 2026: contributor

    ACK 61df8b59adce705c82800d55528edb9a472413bb

    changes since last ACK:

    • Small functional test change

    tnx for taking my suggestion.

  54. winterrdog commented at 9:22 PM on August 9, 2026: contributor

    ACK 61df8b59adce

    the main thing i found in this PR is that an in-progress HTTP request is now treated as resumable state via a state machine attached to the request. this means we can pause parsing when we run out of data and continue from where we left off when more data arrives, instead of starting the same work over again like before

    i reviewed the approach and the changes, and confirmed that they behave as described. looks correct to me.

  55. fanquake added this to the milestone 32.0 on Aug 10, 2026
  56. DrahtBot added the label Needs rebase on Aug 10, 2026
  57. http: only read one HTTPRequest at a time per client 902d8908c9
  58. http: reuse HTTPHeaders to parse chunked trailer
    Chunked transfer trailers are just headers that are included
    at the end of the request. We can parse and validate them with code
    we already use to read headers. In a future commit we will also
    be able to use one MAX_HEADERS_SIZE limit to cover both sections.
    
    Even though we parse and validate, we ignore these data.
    507e528e84
  59. Add state to HTTPRequest to avoid duplicate work over I/O cycles 90676e24ad
  60. test: cover HTTPRequest state machine c7db3ae1f9
  61. http: don't parse any new requests from a client if m_req_busy = true 9954aa7728
  62. pinheadmz force-pushed on Aug 10, 2026
  63. pinheadmz commented at 3:14 PM on August 10, 2026: member

    push to 9954aa77280ecd67816e784815c6478a973f6635

    rebase on master and fix conflict with #34794

  64. DrahtBot removed the label Needs rebase on Aug 10, 2026
  65. winterrdog commented at 6:36 PM on August 10, 2026: contributor

    re-ACK 9954aa77280ecd67816e784815c6478a973f6635

  66. DrahtBot requested review from janb84 on Aug 10, 2026
  67. janb84 commented at 6:40 PM on August 10, 2026: contributor

    re ACK 9954aa77280ecd67816e784815c6478a973f6635

    changes since last ack:

    • rebase on master to fix conflict
  68. frankomosh commented at 8:06 AM on August 11, 2026: contributor

    ACK 9954aa77280ecd67816e784815c6478a973f6635.


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-08-11 09:51 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me