http: throttle send buffer when client stops draining #36174

pull pinheadmz wants to merge 1 commits into bitcoin:master from pinheadmz:http-throttle-send changing 3 files +83 −9
  1. pinheadmz commented at 11:49 AM on September 5, 2026: member

    This is a follow-up to #36123 and applies a small tweak to the read-side throttle mechanism in addition to applying a second mechanism to the send-side. If a misbehaving client refuses to read data from the socket, the server will now stop processing requests instead of packing more and more responses to m_send_buffer without bound.

    After we parse a complete request from a client, before we dispatch it to a worker, we quickly lock and check the size of m_send_buffer. If there's already 32MiB of data there (reusing MAX_BODY_SIZE here, open for bikeshedding...) we do not dispatch the request to a worker, leaving it in place as m_req. This is where the tweak to #36123 comes in. In that PR we throttle reads unless m_req is present, assuming it is incomplete and needs more data. This commit changes that assumption and so != State::Complete must be checked in addition.

    This was found and disclosed responsibly by the Red Team 🟥.

  2. DrahtBot added the label RPC/REST/ZMQ on Sep 5, 2026
  3. DrahtBot commented at 11:49 AM on September 5, 2026: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/36174.

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline and AI policy for information on the review process.

    Type Reviewers
    Concept ACK winterrdog, l0rinc, jeanpablojp, hodlinator, janb84

    If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

    LLM Linter (✨ experimental)

    Possible places where comparison-specific test macros should replace generic comparisons:

    • [test/functional/interface_http.py] assert tries > 0, f"Progress failed to stall after {count} requests were handled." -> replace with assert_greater_than(tries, 0) for the comparison check.

    <sup>2026-09-05 22:06:29</sup>

  4. fanquake added this to the milestone 32.0 on Sep 5, 2026
  5. winterrdog commented at 4:02 PM on September 5, 2026: contributor

    Concept ACK

  6. l0rinc commented at 6:46 PM on September 5, 2026: contributor

    Concept ACK

  7. http: stop processing requests from a client when send buffer is full
    Prevents a memory exhaustion case where a misbehaving client
    refuses to read responses and drain the socket buffer. Instead of
    packing more data on to the server-side m_send_buffer, stop
    dispatching requests from the client to workers
    4e2caa0bd4
  8. pinheadmz force-pushed on Sep 5, 2026
  9. pinheadmz commented at 10:07 PM on September 5, 2026: member

    push to 4e2caa0bd43c12866ab1179fe949d43d4cd61325

    Rebase on master after merge of #36123

  10. jeanpablojp commented at 1:03 AM on September 7, 2026: contributor

    Concept ACK

    The new test does not separate a correct throttle from a wrong one. Swapping MAX_BODY_SIZE for zero in the condition, the server stops after the first response and the test passes all the same, reporting "stalled after 1 requests were handled". With 1 MiB in its place it passes too. Neither of those made it fail. Deleting the return nullptr did, and then it reports 58 requests.

    The wait loop breaks as soon as two consecutive samples of the log give the same number, whatever that number is.

    Would it make sense to pin how many responses have to be queued before that counts as a stall?

  11. in test/functional/interface_http.py:832 in 4e2caa0bd4
     827 | +            while True:
     828 | +                dl.seek(dl_prev_size)
     829 | +                log = dl.read()
     830 | +                count = log.count(URI)
     831 | +                if count == prev_count:
     832 | +                    self.log.info(f"Response progress stalled after {count} requests were handled.")
    


    jeanpablojp commented at 1:03 AM on September 7, 2026:

    This comparison has no floor, so it matches a count of zero and a server that answered only once.

    The product below is not the buffer size, the log counts dispatches and the kernel drains as it goes, but a stall reached with less output than the threshold cannot be this throttle. MAX_BODY_SIZE is already at the top of the file and response_size was measured just above.

                        self.log.info(f"Response progress stalled after {count} requests were handled.")
                        assert count < num_req, f"server handled the whole batch of {num_req}: nothing was throttled"
                        assert count * response_size > MAX_BODY_SIZE, (
                            f"progress stopped after {count} responses ({count * response_size} bytes), "
                            f"below the {MAX_BODY_SIZE} byte send-buffer throttle")
    

    Green on the head with count 5 and red with the constant swapped for zero. Worth it?

  12. in src/httpserver.cpp:1009 in 4e2caa0bd4
    1005 | @@ -1006,14 +1006,17 @@ HTTPServer::IOReadiness HTTPServer::GenerateWaitSockets() const
    1006 |          Sock::Event event{0};
    1007 |          if (http_client->ReadyToSend()) {
    1008 |              event = Sock::SendEvent;
    1009 | -        } else if (http_client->GetRequest() != nullptr || http_client->ReceiveBufferEmpty()) {
    1010 | +        } else if ((http_client->GetRequest() != nullptr && http_client->GetRequest()->GetState() != HTTPRequest::State::Complete)
    


    jeanpablojp commented at 1:03 AM on September 7, 2026:

    Did you manage to reach this branch? ReadyToSend() is tested before it, and a request is only held with m_send_buffer above 32 MiB, which as far as I read already guarantees m_send_ready on any connection that is not already being dropped for a permanent send error.

    Instrumenting the loop, the held-and-complete state showed up 44 times over two runs of the whole test file, every one of them with ReadyToSend() true. Reverting the condition to the previous form leaves the file passing with the same numbers.

    Is there a live case where m_send_ready goes false with the buffer full?

  13. in src/httpserver.cpp:1103 in 4e2caa0bd4
    1098 | +    // data from this client until they drain their end of the socket,
    1099 | +    // and prevent the server from packing more responses into the send buffer.
    1100 | +    const size_t buffer_used{WITH_LOCK(
    1101 | +        client->m_send_mutex,
    1102 | +        return client->m_send_buffer.size();)};
    1103 | +    if (buffer_used > MAX_BODY_SIZE) return nullptr;
    


    jeanpablojp commented at 1:03 AM on September 7, 2026:

    On the bikeshed, with numbers from both sides. The suspension threshold is per connection and -rpcmaxconnections defaults to 16, so that is 512 MiB before counting the in-flight response each connection can still put on top of it.

    With 16 connections that never read the plateau sits at 1474 MB, against 327 MB with 1 MiB in place of the constant, and with the default -rpcservertimeout they are dropped after 36 seconds.

    On the other side, the allowance is what pays for pipelining. Twelve 7.5 MB responses to a fast-reading client take 1.3 to 1.9 s here and 2.5 to 2.9 s with 1 MiB, over twenty-six runs with no overlap, and at 8 MB/s the two are level. Is 32 MiB deliberate?


  14. in src/httpserver.h:533 in 4e2caa0bd4
     527 | @@ -527,6 +528,9 @@ class HTTPRemoteClient
     528 |       * Used to determine if an incomplete request is in progress.
     529 |       * @returns nullptr after a complete request is moved to a worker thread,
     530 |       *          but before reading any new data from m_recv_buffer.
     531 | +     * @note The returned request may also be in State::Complete when it has
     532 | +     *       been fully parsed but is being held back by TryReadRequest()'s
     533 | +     *       send-buffer throttle; check GetState() to distinguish the two.
    


    jeanpablojp commented at 1:03 AM on September 7, 2026:

    The functional test never lets the client drain, so neither the release of the held request nor this Complete state ends up covered, and nor does the order between the held one and the one queued behind it.

    The DummyClient already in httpserver_tests.cpp closes all three. It passes on the head, and without the new return nullptr it fails on the two checks that assert the request was held. Want it?

    BOOST_AUTO_TEST_CASE(http_send_buffer_throttle_tests)
    {
        // A socket that accepts no outbound bytes while m_blocked is set, standing in
        // for a client that has stopped draining its end of the connection.
        class StalledSock : public ZeroSock
        {
        public:
            ssize_t Send(const void*, size_t len, int) const override
            {
                return m_blocked ? 0 : static_cast<ssize_t>(len);
            }
            mutable bool m_blocked{true};
        };
    
        class DummyClient : public HTTPRemoteClient
        {
        public:
            explicit DummyClient(std::unique_ptr<Sock> sock)
                : HTTPRemoteClient{/*id=*/0, /*addr=*/CService(), /*socket=*/std::move(sock)} {}
    
            void receive(std::string_view s) { MutateRecvBuffer().append(s); }
        };
    
        const std::string wire1{"GET /first HTTP/1.1\r\nHost: 127.0.0.1\r\n\r\n"};
        const std::string wire2{"GET /second HTTP/1.1\r\nHost: 127.0.0.1\r\n\r\n"};
        const std::string wire3{"GET /third HTTP/1.1\r\nHost: 127.0.0.1\r\n\r\n"};
    
        auto sock{std::make_unique<StalledSock>()};
        StalledSock& stalled{*sock};
        std::shared_ptr<DummyClient> client{std::make_shared<DummyClient>(std::move(sock))};
    
        // The first request is dispatched normally.
        client->receive(wire1);
        auto first{HTTPRemoteClient::TryReadRequest(client)};
        BOOST_REQUIRE(first);
        BOOST_CHECK_EQUAL(first->GetURI(), "/first");
        BOOST_CHECK(!client->GetRequest());
    
        // Its reply is larger than the throttle and the client is not reading it,
        // so it stays in m_send_buffer.
        first->WriteReply(HTTP_OK, std::string(MAX_BODY_SIZE + 1, 'x'));
        BOOST_CHECK(client->ReadyToSend());
    
        // The next two arrive while the throttle holds. The second parses to
        // Complete and is held back instead of dispatched, and the third is left
        // untouched in the receive buffer behind it.
        client->receive(wire2);
        client->receive(wire3);
        BOOST_CHECK(!HTTPRemoteClient::TryReadRequest(client));
        BOOST_REQUIRE(client->GetRequest());
        BOOST_CHECK_EQUAL(client->GetRequest()->GetState(), HTTPRequest::State::Complete);
        BOOST_CHECK_EQUAL(client->GetRequest()->GetURI(), "/second");
        BOOST_CHECK_EQUAL(client->GetRecvBuffer(), wire3);
    
        // Draining the send buffer releases the held request, and the one queued
        // behind it follows in the order it arrived.
        stalled.m_blocked = false;
        BOOST_CHECK(client->MaybeSendBytesFromBuffer());
        BOOST_CHECK(!client->ReadyToSend());
        auto second{HTTPRemoteClient::TryReadRequest(client)};
        BOOST_REQUIRE(second);
        BOOST_CHECK_EQUAL(second->GetURI(), "/second");
        second->WriteReply(HTTP_OK, "");
        auto third{HTTPRemoteClient::TryReadRequest(client)};
        BOOST_REQUIRE(third);
        BOOST_CHECK_EQUAL(third->GetURI(), "/third");
        BOOST_CHECK_EQUAL(client->GetRecvBuffer().size(), 0);
    }
    
  15. in src/httpserver.cpp:1010 in 4e2caa0bd4
    1005 | @@ -1006,14 +1006,17 @@ HTTPServer::IOReadiness HTTPServer::GenerateWaitSockets() const
    1006 |          Sock::Event event{0};
    1007 |          if (http_client->ReadyToSend()) {
    1008 |              event = Sock::SendEvent;
    1009 | -        } else if (http_client->GetRequest() != nullptr || http_client->ReceiveBufferEmpty()) {
    1010 | +        } else if ((http_client->GetRequest() != nullptr && http_client->GetRequest()->GetState() != HTTPRequest::State::Complete)
    1011 | +                    || http_client->ReceiveBufferEmpty()) {
    


    hodlinator commented at 9:42 AM on September 7, 2026:

    nit: The current indentation implies the or-operator is part of the inner parenthesis above.

            } else if ((http_client->GetRequest() != nullptr && http_client->GetRequest()->GetState() != HTTPRequest::State::Complete)
                       || http_client->ReceiveBufferEmpty()) {
    
  16. hodlinator commented at 11:17 AM on September 7, 2026: contributor

    Concept ACK 4e2caa0bd43c12866ab1179fe949d43d4cd61325

  17. in test/functional/interface_http.py:794 in 4e2caa0bd4
     785 | @@ -779,5 +786,56 @@ def check_pipelined_data_is_throttled(self):
     786 |          assert generated_block in response
     787 |  
     788 |  
     789 | +    def check_slow_read_throttle(self):
     790 | +        self.log.info("Check that request processing is throttled if the client is not draining the socket")
     791 | +        self.restart_node(0, extra_args=["-rest=1"])
     792 | +        # Generate a big block
     793 | +        self.wallet = MiniWallet(self.node)
     794 | +        self.generate(self.wallet, 130)
    


    hodlinator commented at 11:43 AM on September 7, 2026:

    Why generate more than 1?

            self.generate(self.wallet, 1)
    
  18. in test/functional/interface_http.py:812 in 4e2caa0bd4
     807 | +        # least one server-side read operation (about 65kB, see HTTPRemoteClient::Receive()).
     808 | +        batch = ""
     809 | +        num_req = 0
     810 | +        while len(batch) < 0x10000:
     811 | +            batch += f"GET {URI} HTTP/1.1\r\nHost: somehost\r\n\r\n"
     812 | +            num_req += 1
    


    hodlinator commented at 12:36 PM on September 7, 2026:

    Could make this more declarative and only interpolate the string once:

            single_req = f"GET {URI} HTTP/1.1\r\nHost: somehost\r\n\r\n"
            num_req = 0x10000 // len(single_req)
            batch = single_req * num_req
    
  19. in src/httpserver.cpp:1106 in 4e2caa0bd4
    1101 | +        client->m_send_mutex,
    1102 | +        return client->m_send_buffer.size();)};
    1103 | +    if (buffer_used > MAX_BODY_SIZE) return nullptr;
    1104 | +
    1105 |      // If the request is ready, hand it to a worker.
    1106 |      if (client->m_req->GetState() == HTTPRequest::State::Complete) {
    


    janb84 commented at 2:27 PM on September 7, 2026:

    NIT; could move the if statement up. Currently, the lock is taken on every call, including for clients whose receive buffer yielded nothing to dispatch.

    The move also keeps the same behaviour but with less cognitive load. An incomplete request with a full send buffer returned null here before, and afterwards falls through to the same return nullptr at the end of the function. The check also stays ahead of the LogDebug, so a throttled request still does not log on every loop iteration. In the case of an incomplete request, it's pretty easy to follow what happens, where before you it was not as clear. (imho)

    // If the request is ready, hand it to a worker.
    if (client->m_req->GetState() == HTTPRequest::State::Complete) {
            // Unless this client's send buffer is full: in that case hold the
            // parsed request here instead of moving it to a worker. This prevents
            // the server from reading any more data from this client until they
            // drain their end of the socket, and prevents the server from packing
            // more responses into the send buffer.
        const size_t buffer_used{WITH_LOCK(
            client->m_send_mutex,
            return client->m_send_buffer.size();)};
        if (buffer_used > MAX_BODY_SIZE) return nullptr;
    
  20. janb84 commented at 2:28 PM on September 7, 2026: contributor

    Concept ACK 4e2caa0bd43c12866ab1179fe949d43d4cd61325

    I agree with (most) of the NITS/suggestions above. Have one suggestion myself. The direction of the PR looks good.


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-09-08 12:50 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me