qa: timeout in StopHTTPServer() #31894

hodlinator commented at 10:49 AM on February 18, 2025: contributor

Ran into a CI timeout in feature_assumeutxo.py on Windows: https://github.com/hodlinator/bitcoin/actions/runs/13371466603/job/37340702068#step:12:123

In the combined log we see the following, "Restarting node to stop at height" is referring to node 1):

test  ...T14:10:53.420000Z TestFramework (INFO): Restarting node to stop at height 359
test  ...T14:10:53.420000Z TestFramework.node1 (DEBUG): Stopping node
node1 ...T14:10:53.420155Z (mocktime...) [http] [D:\a\bitcoin\bitcoin\src\httpserver.cpp:306] [http_request_cb] [http] Received a POST request for / from 127.0.0.1:56740
node1 ...T14:10:53.420240Z (mocktime...) [httpworker.8] [D:\a\bitcoin\bitcoin\src\rpc\request.cpp:241] [parse] [rpc] ThreadRPCServer method=gettxout user=__cookie__
node1 ...T14:10:53.421024Z (mocktime...) [http] [D:\a\bitcoin\bitcoin\src\httpserver.cpp:306] [http_request_cb] [http] Received a POST request for / from 127.0.0.1:56740
node1 ...T14:10:53.421114Z (mocktime...) [httpworker.11] [D:\a\bitcoin\bitcoin\src\rpc\request.cpp:241] [parse] [rpc] ThreadRPCServer method=stop user=__cookie__
node1 ...T14:10:53.421203Z (mocktime...) [init] [D:\a\bitcoin\bitcoin\src\httpserver.cpp:508] [InterruptHTTPServer] [http] Interrupting HTTP server
node1 ...T14:10:53.421251Z (mocktime...) [init] [D:\a\bitcoin\bitcoin\src\httprpc.cpp:382] [InterruptHTTPRPC] [rpc] Interrupting HTTP RPC server
node1 ...T14:10:53.421280Z (mocktime...) [init] [D:\a\bitcoin\bitcoin\src\rpc\server.cpp:291] [operator ()] [rpc] Interrupting RPC
node1 ...T14:10:53.421320Z (mocktime...) [init] [D:\a\bitcoin\bitcoin\src\init.cpp:287] [Shutdown] Shutdown: In progress...
node1 ...T14:10:53.421345Z (mocktime...) [shutoff] [D:\a\bitcoin\bitcoin\src\httprpc.cpp:387] [StopHTTPRPC] [rpc] Stopping HTTP RPC server
node1 ...T14:10:53.422054Z (mocktime...) [addcon] [D:\a\bitcoin\bitcoin\src\util\thread.cpp:22] [TraceThread] addcon thread exit 
node1 ...T14:10:53.422540Z (mocktime...) [shutoff] [D:\a\bitcoin\bitcoin\src\httpserver.cpp:770] [UnregisterHTTPHandler] [http] Unregistering HTTP handler for / (exactmatch 1) 
node1 ...T14:10:53.422641Z (mocktime...) [shutoff] [D:\a\bitcoin\bitcoin\src\httpserver.cpp:770] [UnregisterHTTPHandler] [http] Unregistering HTTP handler for /wallet/ (exactmatch 0) 
node1 ...T14:10:53.422674Z (mocktime...) [shutoff] [D:\a\bitcoin\bitcoin\src\rpc\server.cpp:303] [operator ()] [rpc] Stopping RPC
node1 ...T14:10:53.422847Z (mocktime...) [shutoff] [D:\a\bitcoin\bitcoin\src\rpc\server.cpp:306] [operator ()] [rpc] RPC stopped.
node1 ...T14:10:53.422879Z (mocktime...) [shutoff] [D:\a\bitcoin\bitcoin\src\httpserver.cpp:520] [StopHTTPServer] [http] Stopping HTTP server
node1 ...T14:10:53.422945Z (mocktime...) [shutoff] [D:\a\bitcoin\bitcoin\src\httpserver.cpp:522] [StopHTTPServer] [http] Waiting for HTTP worker threads to exit
node1 ...T14:10:53.423118Z (mocktime...) [shutoff] [D:\a\bitcoin\bitcoin\src\httpserver.cpp:536] [StopHTTPServer] [http] Waiting for 1 connections to stop HTTP server
node1 ...T14:10:53.423164Z (mocktime...) [http] [D:\a\bitcoin\bitcoin\src\httpserver.cpp:355] [ThreadHTTP] [http] Exited http event loop
node1 ...T14:10:53.440186Z (mocktime...) [net] [D:\a\bitcoin\bitcoin\src\util\thread.cpp:22] [TraceThread] net thread exit
node1 ...T14:10:53.652881Z (mocktime...) [msghand] [D:\a\bitcoin\bitcoin\src\util\thread.cpp:22] [TraceThread] msghand thread exit
...
node1 ...T14:11:32.925206Z (mocktime...) [scheduler] [D:\a\bitcoin\bitcoin\src\net.cpp:2428] [StartExtraBlockRelayPeers] [net] enabling extra block-relay-only peers
...
node1 ...T14:25:47.935847Z (mocktime...) [scheduler] [D:\a\bitcoin\bitcoin\src\net.cpp:2391] [DumpAddresses] [net] Flushed 0 addresses to peers.dat  1ms
...
node1 ...T14:40:47.947785Z (mocktime...) [scheduler] [D:\a\bitcoin\bitcoin\src\net.cpp:2391] [DumpAddresses] [net] Flushed 0 addresses to peers.dat  1ms
...
test  ...T14:50:53.446000Z TestFramework (ERROR): JSONRPC error

    Traceback (most recent call last):
      File "D:\a\bitcoin\bitcoin\build\test\functional\test_framework\authproxy.py", line 164, in _get_response
        http_response = self.__conn.getresponse()
...
      File "C:\hostedtoolcache\windows\Python\3.13.2\x64\Lib\socket.py", line 719, in readinto
        return self._sock.recv_into(b)
               ~~~~~~~~~~~~~~~~~~~~^^^
    TimeoutError: timed out
    During handling of the above exception, another exception occurred:
    Traceback (most recent call last):
      File "D:\a\bitcoin\bitcoin\build\test\functional\test_framework\test_framework.py", line 135, in main
        self.run_test()
        ~~~~~~~~~~~~~^^
      File "D:\a\bitcoin\bitcoin\build\test\functional\feature_assumeutxo.py", line 587, in run_test
        self.restart_node(1, extra_args=[
        ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
            f"-stopatheight={PAUSE_HEIGHT}", *self.extra_args[1]])
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
      File "D:\a\bitcoin\bitcoin\build\test\functional\test_framework\test_node.py", line 395, in stop_node
        self.stop(wait=wait)
        ~~~~~~~~~^^^^^^^^^^^
...
      File "D:\a\bitcoin\bitcoin\build\test\functional\test_framework\authproxy.py", line 106, in _request
        return self._get_response()
               ~~~~~~~~~~~~~~~~~~^^
      File "D:\a\bitcoin\bitcoin\build\test\functional\test_framework\authproxy.py", line 166, in _get_response
        raise JSONRPCException({
...
    test_framework.authproxy.JSONRPCException: 'stop' RPC took longer than 2400.000000 seconds. Consider using larger timeout for calls that take longer to return. (-344)

Note how it takes:

~40 seconds to go from receiving stop-RPC to "enabling extra block-relay-only peers"
Additional 14 minutes to reach "Flushed 0 addresses to peers.dat"
Additional 15 minutes for the second one
Additional 10 minutes until we reach the JSONRPC error/timeout.

It would be good to have log output describing what the node is doing, it obviously started shutting down, and is maybe still flushing some state to disk?

Also, I wonder if bitcoind might have terminated the TCP/HTTP connection before responding to the stop-RPC (maybe gettxout-RPC being called right before it is a factor)?

Included in the log, but seems unrelated as it's another node:

node0 stderr Error: A fatal internal error occurred, see debug.log for details: Assumeutxo data not found for the given blockhash '7a7a7a7a7a7a7a7a7a7a7a7a7a7a7a7a7a7a7a7a7a7a7a7a7a7a7a7a7a7a7a7a'.

maflcko added the label CI failed on Feb 18, 2025

maflcko added the label Windows on Feb 18, 2025

hodlinator commented at 11:02 AM on February 18, 2025: contributor

@pinheadmz since you are working on the HTTP rewrite... have you noticed anything fishy regarding whether the stop-RPC will actually always get the response sent off back to the client or not?

Current stop-RPC implementation implies it should always work: https://github.com/bitcoin/bitcoin/blob/790c509a4796ec47be2275d86b35370ff25a599a/src/rpc/server.cpp#L170-L172

But it's from 2015 and the log above makes me very suspicious.

maflcko commented at 11:23 AM on February 18, 2025: member

It looks like the scheduler thread never exited? It may be related to the comment:

    // After everything has been shut down, but before things get flushed, stop the
    // the scheduler. After this point, SyncWithValidationInterfaceQueue() should not be called anymore
    // as this would prevent the shutdown from completing.
    if (node.scheduler) node.scheduler->stop();

However, the scheduler thread seems to cycle normally, so there doesn't seem to be any SyncWithValidationInterfaceQueue event stuck in it.

pinheadmz commented at 11:44 AM on February 18, 2025: member

@pinheadmz since you are working on the HTTP rewrite...

I hadn't noticed anything about the current implementation but managing the RPC stop race condition was definitely a tricky part of the rewrite.

One thing to note about the actual total time elapsed in your case is that on CI the timeout factors are multiplied by large factors in some cases to avoid timeout errors on low-resource runners:

https://github.com/bitcoin/bitcoin/blob/78c19ddd7c56a275d2ae835fd13ed1508acd7806/.github/workflows/ci.yml#L153-L163

I suppose enabling libevent logs may help understanding whats happening...

Or maybe investigate why the net thread appears to not get the interrupt signal?

maflcko commented at 12:06 PM on February 18, 2025: member

Or maybe investigate why the net thread appears to not get the interrupt signal?

My impression was that the net thread is stopped, along with the http workers, and the http thread. See:

node1 ...T14:10:53.440186Z (mocktime...) [net] [D:\a\bitcoin\bitcoin\src\util\thread.cpp:22] [TraceThread] net thread exit

There seem to be two issues here:

The stop RPC timing out
The node not stopping

pinheadmz commented at 12:08 PM on February 18, 2025: member

Hm yeah, but then why does enabling extra block-relay-only peers happen after that?

maflcko commented at 1:15 PM on February 18, 2025: member

Hm yeah, but then why does enabling extra block-relay-only peers happen after that?

The StartExtraBlockRelayPeers function is logging to the [net] category (not to be confused with the [net] thread) and is called by CheckForStaleTipAndEvictPeers, which is running in the [scheduler] thread.

hodlinator commented at 3:15 PM on February 18, 2025: contributor

[shutoff] [...\httpserver.cpp:536] [StopHTTPServer] [http] Waiting for 1 connections to stop HTTP server
[http] [...\httpserver.cpp:355] [ThreadHTTP] [http] Exited http event loop
[net] [...\util\thread.cpp:22] [TraceThread] net thread exit

The fact that we don't get "Stopped HTTP server" or even "Waiting for HTTP event thread to exit" after these indicates that we are stuck in the lower part of StopHTTPServer():

https://github.com/bitcoin/bitcoin/blob/790c509a4796ec47be2275d86b35370ff25a599a/src/httpserver.cpp#L518-L557

So we're either stuck on g_requests.WaitUntilEmpty(); or just after, in the evhttp_free logic.

The pending request taking time to complete is likely stop or gettxout.

It is a bit worrying that we get "net thread exit" before completely stopping HTTP, but I guess the "net" thread (CConnman::ThreadSocketHandler) is only used for P2P traffic anyways, not HTTP/RPC, so it shouldn't be related.

mzumsande commented at 4:21 PM on February 18, 2025: contributor

#31019 (macOS) might be the same issue, in which case this would not be windows-specific.

maflcko removed the label Windows on Feb 18, 2025

maflcko commented at 4:32 PM on February 18, 2025: member

StopHTTPServer

Could update the issue title to say "timeout in StopHTTPServer"?

#31019

They are not exactly identical. 31019 likely never received an RPC, and the node is expected to raise an init error after running into a wallet init error. However, the function of the deadlock is the same:

 node0 2024-10-02T00:15:45.783904Z [shutoff] [src/httpserver.cpp:522] [StopHTTPServer] [http] Stopping HTTP server 
 node0 2024-10-02T00:15:45.783911Z [shutoff] [src/httpserver.cpp:524] [StopHTTPServer] [http] Waiting for HTTP worker threads to exit 
...
(no further log messages from the `[shutoff]` thread)

So I agree with you that they seem to be the same underlying cause/issue.

hodlinator renamed this:
~~test: feature_assumeutxo.py Windows timeout~~
qa: timeout in StopHTTPServer()
on Feb 18, 2025

hodlinator commented at 3:32 PM on February 19, 2025: contributor

I'm working on making the code a bit more robust:

Impose a timeout on HTTP connections shutting down - Maybe related to this issue.
Fix an issue where we were not always processing the event to free eventHTTP - Would probably just leak otherwise, shutdown completes regardless of this in my testing.

diff --git a/src/httpserver.cpp b/src/httpserver.cpp
index 88e640c377..ce2534fbd0 100644
--- a/src/httpserver.cpp
+++ b/src/httpserver.cpp
@@ -197,10 +197,15 @@ public:
         return WITH_LOCK(m_mutex, return m_tracker.size());
     }
     //! Wait until there are no more connections with active requests in the tracker
-    void WaitUntilEmpty() const EXCLUSIVE_LOCKS_REQUIRED(!m_mutex)
+    //! or return the number of remaining requests on timeout.
+    size_t WaitUntilEmpty(std::chrono::seconds timeout) const EXCLUSIVE_LOCKS_REQUIRED(!m_mutex)
     {
-        WAIT_LOCK(m_mutex, lock);
-        m_cv.wait(lock, [this]() EXCLUSIVE_LOCKS_REQUIRED(m_mutex) { return m_tracker.empty(); });
+        {
+            WAIT_LOCK(m_mutex, lock);
+            m_cv.wait_for(lock, timeout, [this]() EXCLUSIVE_LOCKS_REQUIRED(m_mutex) { return m_tracker.empty(); });
+        }
+
+        return WITH_LOCK(m_mutex, return m_tracker.size());
     }
 };
 //! Track active requests
@@ -535,13 +540,17 @@ void StopHTTPServer()
         if (const auto n_connections{g_requests.CountActiveConnections()}; n_connections != 0) {
             LogDebug(BCLog::HTTP, "Waiting for %d connections to stop HTTP server\n", n_connections);
         }
-        g_requests.WaitUntilEmpty();
+        if (int connections = g_requests.WaitUntilEmpty(/*timeout=*/5min); connections > 0) {
+            LogError("%d HTTP connections remain after timeout expired. Aborting to avoid freeing resources still under use.", connections);
+            std::abort();
+        }
     }
     if (eventHTTP) {
         // Schedule a callback to call evhttp_free in the event base thread, so
         // that evhttp_free does not need to be called again after the handling
         // of unfinished request connections that follows.
         event_base_once(eventBase, -1, EV_TIMEOUT, [](evutil_socket_t, short, void*) {
+            LogDebug(BCLog::HTTP, "Freeing eventHTTP");
             evhttp_free(eventHTTP);
             eventHTTP = nullptr;
         }, nullptr, nullptr);
@@ -549,6 +558,17 @@ void StopHTTPServer()
     if (eventBase) {
         LogDebug(BCLog::HTTP, "Waiting for HTTP event thread to exit\n");
         if (g_thread_http.joinable()) g_thread_http.join();
+        bool first_run = true;
+        while (event_base_get_num_events(eventBase, EVENT_BASE_COUNT_ACTIVE | EVENT_BASE_COUNT_ADDED) > 0) {
+            if (first_run) LogDebug(BCLog::HTTP, "Processing events left behind by HTTP event thread");
+            event_base_loop(eventBase, EVLOOP_NONBLOCK);
+            if (first_run) {
+                first_run = false;
+            } else {
+                // Only sleep when this happens repetitively.
+                std::this_thread::sleep_for(100ms);
+            }
+        }
         event_base_free(eventBase);
         eventBase = nullptr;
     }

With the change I get:

2025-02-19T15:22:35Z [http] Stopping HTTP server
2025-02-19T15:22:35Z [http] Waiting for HTTP worker threads to exit
2025-02-19T15:22:35Z [http] Exited http event loop
2025-02-19T15:22:35Z [http] Waiting for HTTP event thread to exit
2025-02-19T15:22:35Z [http] Processing events left behind by HTTP event thread
2025-02-19T15:22:35Z [http] Freeing eventHTTP
2025-02-19T15:22:35Z [http] Stopped HTTP server

</details>

maflcko added the label RPC/REST/ZMQ on Feb 20, 2025

hodlinator commented at 11:01 AM on February 24, 2025: contributor

Spent a 2-3 days last week experimenting with solutions, ended up with:

Improving docs.
Fixing an adjacent memory leak.
Improving debug-ability of future occurrences of this issue.

See #31929.

Looking forward to not relying in libevent in the future! Much nicer to be able to debug our code directly.

maflcko commented at 4:37 PM on February 27, 2026: member

The issue keeps happening, see #34624 (macOS)

Though, I guess it may be fine to ignore this for now. Similar to https://github.com/bitcoin/bitcoin/issues/34634

polespinasa commented at 7:45 PM on June 3, 2026: member

I think I ran into this same (or similar) issue in PR #35179

Logs here I am not sure if the CI logs are persistent...

2026-06-03T19:28:28.2006913Z 2026-06-03T19:27:57.312906Z TestFramework (INFO): Test that a node only announcing a maximally sized ancestor package is protected in orphanage
2026-06-03T19:28:28.2007604Z 2026-06-03T19:28:28.120483Z TestFramework (ERROR): Unexpected exception:
2026-06-03T19:28:28.2007983Z Traceback (most recent call last):
2026-06-03T19:28:28.2008493Z   File "/home/runner/work/bitcoin/bitcoin/test/functional/test_framework/authproxy.py", line 165, in _get_response
2026-06-03T19:28:28.2009028Z     http_response = self.__conn.getresponse()
2026-06-03T19:28:28.2009314Z                     ^^^^^^^^^^^^^^^^^^^^^^^^^
2026-06-03T19:28:28.2009655Z   File "/usr/lib/python3.12/http/client.py", line 1448, in getresponse
2026-06-03T19:28:28.2009997Z     response.begin()
2026-06-03T19:28:28.2010271Z   File "/usr/lib/python3.12/http/client.py", line 336, in begin
2026-06-03T19:28:28.2010623Z     version, status, reason = self._read_status()
2026-06-03T19:28:28.2010909Z                               ^^^^^^^^^^^^^^^^^^^
2026-06-03T19:28:28.2011256Z   File "/usr/lib/python3.12/http/client.py", line 297, in _read_status
2026-06-03T19:28:28.2011837Z     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
2026-06-03T19:28:28.2012242Z                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-06-03T19:28:28.2012564Z   File "/usr/lib/python3.12/socket.py", line 707, in readinto
2026-06-03T19:28:28.2012921Z     return self._sock.recv_into(b)
2026-06-03T19:28:28.2013184Z            ^^^^^^^^^^^^^^^^^^^^^^^
2026-06-03T19:28:28.2013435Z TimeoutError: timed out
2026-06-03T19:28:28.2013583Z 
2026-06-03T19:28:28.2013794Z During handling of the above exception, another exception occurred:
2026-06-03T19:28:28.2014056Z 
2026-06-03T19:28:28.2014171Z Traceback (most recent call last):
2026-06-03T19:28:28.2014680Z   File "/home/runner/work/bitcoin/bitcoin/ci_build/test/functional/p2p_orphan_handling.py", line 54, in wrapper
2026-06-03T19:28:28.2015190Z     func(self)
2026-06-03T19:28:28.2015692Z   File "/home/runner/work/bitcoin/bitcoin/ci_build/test/functional/p2p_orphan_handling.py", line 633, in test_maximal_package_protected
2026-06-03T19:28:28.2018664Z     testres = node.testmempoolaccept([large_orphan.serialize().hex()])
2026-06-03T19:28:28.2019899Z               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-06-03T19:28:28.2020621Z   File "/home/runner/work/bitcoin/bitcoin/test/functional/test_framework/coverage.py", line 50, in __call__
2026-06-03T19:28:28.2021253Z     return_val = self.auth_service_proxy_instance.__call__(*args, **kwargs)
2026-06-03T19:28:28.2021675Z                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-06-03T19:28:28.2022223Z   File "/home/runner/work/bitcoin/bitcoin/test/functional/test_framework/authproxy.py", line 128, in __call__
2026-06-03T19:28:28.2022888Z     response, status = self._request('POST', self.__url.path, postdata.encode('utf-8'))
2026-06-03T19:28:28.2023363Z                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-06-03T19:28:28.2023941Z   File "/home/runner/work/bitcoin/bitcoin/test/functional/test_framework/authproxy.py", line 102, in _request
2026-06-03T19:28:28.2024508Z     return self._get_response()
2026-06-03T19:28:28.2024782Z            ^^^^^^^^^^^^^^^^^^^^
2026-06-03T19:28:28.2025278Z   File "/home/runner/work/bitcoin/bitcoin/test/functional/test_framework/authproxy.py", line 167, in _get_response
2026-06-03T19:28:28.2025930Z     raise JSONRPCException({
2026-06-03T19:28:28.2026811Z test_framework.util.JSONRPCException: 'testmempoolaccept' RPC took longer than 30.000000 seconds. Consider using larger timeout for calls that take longer to return. (-344)  [http_status=None]
2026-06-03T19:28:28.2027492Z 
2026-06-03T19:28:28.2027687Z During handling of the above exception, another exception occurred:
2026-06-03T19:28:28.2027962Z 
2026-06-03T19:28:28.2028064Z Traceback (most recent call last):
2026-06-03T19:28:28.2028561Z   File "/home/runner/work/bitcoin/bitcoin/test/functional/test_framework/test_framework.py", line 143, in main
2026-06-03T19:28:28.2029037Z     self.run_test()
2026-06-03T19:28:28.2029490Z   File "/home/runner/work/bitcoin/bitcoin/ci_build/test/functional/p2p_orphan_handling.py", line 837, in run_test
2026-06-03T19:28:28.2030070Z     self.test_maximal_package_protected()
2026-06-03T19:28:28.2030582Z   File "/home/runner/work/bitcoin/bitcoin/ci_build/test/functional/p2p_orphan_handling.py", line 57, in wrapper
2026-06-03T19:28:28.2031117Z     self.generate(self.nodes[0], 1)
2026-06-03T19:28:28.2031626Z   File "/home/runner/work/bitcoin/bitcoin/test/functional/test_framework/test_framework.py", line 666, in generate
2026-06-03T19:28:28.2032219Z     blocks = generator.generate(*args, called_by_framework=True, **kwargs)
2026-06-03T19:28:28.2032602Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-06-03T19:28:28.2033122Z   File "/home/runner/work/bitcoin/bitcoin/test/functional/test_framework/test_node.py", line 445, in generate
2026-06-03T19:28:28.2033908Z     return self.generatetoaddress(nblocks=nblocks, address=self.get_deterministic_priv_key().address, maxtries=maxtries, **kwargs)
2026-06-03T19:28:28.2034742Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-06-03T19:28:28.2035402Z   File "/home/runner/work/bitcoin/bitcoin/test/functional/test_framework/test_node.py", line 453, in generatetoaddress
2026-06-03T19:28:28.2036065Z     return self.__getattr__('generatetoaddress')(*args, **kwargs)
2026-06-03T19:28:28.2036536Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-06-03T19:28:28.2037049Z   File "/home/runner/work/bitcoin/bitcoin/test/functional/test_framework/coverage.py", line 50, in __call__
2026-06-03T19:28:28.2037627Z     return_val = self.auth_service_proxy_instance.__call__(*args, **kwargs)
2026-06-03T19:28:28.2038001Z                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-06-03T19:28:28.2038525Z   File "/home/runner/work/bitcoin/bitcoin/test/functional/test_framework/authproxy.py", line 128, in __call__
2026-06-03T19:28:28.2039167Z     response, status = self._request('POST', self.__url.path, postdata.encode('utf-8'))
2026-06-03T19:28:28.2039639Z                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-06-03T19:28:28.2040308Z   File "/home/runner/work/bitcoin/bitcoin/test/functional/test_framework/authproxy.py", line 101, in _request
2026-06-03T19:28:28.2040901Z     self.__conn.request(method, path, postdata, headers)
2026-06-03T19:28:28.2041306Z   File "/usr/lib/python3.12/http/client.py", line 1356, in request
2026-06-03T19:28:28.2041731Z     self._send_request(method, url, body, headers, encode_chunked)
2026-06-03T19:28:28.2042192Z   File "/usr/lib/python3.12/http/client.py", line 1367, in _send_request
2026-06-03T19:28:28.2042605Z     self.putrequest(method, url, **skips)
2026-06-03T19:28:28.2042985Z   File "/usr/lib/python3.12/http/client.py", line 1193, in putrequest
2026-06-03T19:28:28.2043404Z     raise CannotSendRequest(self.__state)
2026-06-03T19:28:28.2043763Z http.client.CannotSendRequest: Request-sent
2026-06-03T19:28:28.2044421Z 2026-06-03T19:28:28.175374Z TestFramework (INFO): Not stopping nodes as test failed. The dangling processes will be cleaned up later.
2026-06-03T19:28:28.2045533Z 2026-06-03T19:28:28.175519Z TestFramework (WARNING): Not cleaning up dir /tmp/test_runner_₿_🏃_20260603_192705/p2p_orphan_handling_259
2026-06-03T19:28:28.2046854Z 2026-06-03T19:28:28.175572Z TestFramework (ERROR): Test failed. Test logging available at /tmp/test_runner_₿_🏃_20260603_192705/p2p_orphan_handling_259/test_framework.log
2026-06-03T19:28:28.2047890Z 2026-06-03T19:28:28.175724Z TestFramework (ERROR): 
2026-06-03T19:28:28.2055421Z 2026-06-03T19:28:28.175841Z TestFramework (ERROR): Hint: Call /home/runner/work/bitcoin/bitcoin/test/functional/combine_logs.py '/tmp/test_runner_₿_🏃_20260603_192705/p2p_orphan_handling_259' to consolidate all logs
2026-06-03T19:28:28.2056569Z 2026-06-03T19:28:28.175884Z TestFramework (ERROR): 
2026-06-03T19:28:28.2057298Z 2026-06-03T19:28:28.175918Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2026-06-03T19:28:28.2058309Z 2026-06-03T19:28:28.175970Z TestFramework (ERROR): https://github.com/bitcoin/bitcoin/issues
2026-06-03T19:28:28.2058773Z 2026-06-03T19:28:28.176006Z TestFramework (ERROR): 
2026-06-03T19:28:28.2058983Z 
2026-06-03T19:28:28.3076683Z 
2026-06-03T19:28:28.3106867Z stderr:
2026-06-03T19:28:28.3136847Z [node 0] Cleaning up leftover process

</details>

hodlinator commented at 9:03 AM on June 4, 2026: contributor

I think I ran into this same (or similar) issue in PR #35179 @polespinasa I don't see how they are related. In your case, something is stalling the node, making the RPC request timeout, but the node doesn't appear to have initiated shutdown.

On the other hand, here is a PR to tidy up the exception tracebacks you got: #35458