ci: failure in p2p_private_broadcast: ConnectionResetError: [Errno 54] Connection reset by peer

rkrux commented at 10:15 am on March 18, 2026: contributor

https://github.com/bitcoin/bitcoin/actions/runs/23192191180/job/67390951612?pr=34841

 02026-03-17T11:57:04.087893Z TestFramework (INFO): Trying to send a transaction when none of Tor or I2P is reachable
 12026-03-17T11:57:04.289600Z TestFramework.socks5 (ERROR): socks5 request handling failed.
 2Traceback (most recent call last):
 3  File "/Users/runner/work/bitcoin/bitcoin/repo_archive/test/functional/test_framework/socks5.py", line 185, in handle
 4    forward_sockets(self.conn, conn_to)
 5    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
 6  File "/Users/runner/work/bitcoin/bitcoin/repo_archive/test/functional/test_framework/socks5.py", line 68, in forward_sockets
 7    data = s.recv(4096)
 8ConnectionResetError: [Errno 54] Connection reset by peer
 92026-03-17T11:57:04.780915Z TestFramework (INFO): Stopping nodes
102026-03-17T11:57:04.972885Z TestFramework (INFO): Cleaning up /Users/runner/work/bitcoin/bitcoin/repo_archive/ci/scratch/test_runner/test_runner_₿_🏃_20260317_114135/p2p_private_broadcast_74 on exit
112026-03-17T11:57:04.972946Z TestFramework (INFO): Tests successful
12
13
14stderr:
15/opt/homebrew/Cellar/python@3.14/3.14.3_1/Frameworks/Python.framework/Versions/3.14/lib/python3.14/asyncio/base_events.py:744: RuntimeWarning: coroutine 'NetworkThread.create_listen_server' was never awaited
16  self._ready.clear()
17RuntimeWarning: Enable tracemalloc to get the object allocation traceback

maflcko added the label CI failed on Mar 18, 2026

maflcko added the label Tests on Mar 18, 2026

fanquake renamed this:
~~(p2p_private_broadcast): ConnectionResetError: [Errno 54] Connection reset by peer~~
ci: failure in p2p_private_broadcast: ConnectionResetError: [Errno 54] Connection reset by peer
on Mar 19, 2026

fanquake commented at 5:39 am on March 19, 2026: member

cc @vasild

fanquake commented at 5:40 am on March 19, 2026: member

Also seen here: https://github.com/bitcoin/bitcoin/actions/runs/23202474958/job/67428457683#step:9:3361

0
1stderr:
2/opt/homebrew/Cellar/python@3.14/3.14.3_1/Frameworks/Python.framework/Versions/3.14/lib/python3.14/asyncio/base_events.py:744: RuntimeWarning: coroutine 'NetworkThread.create_listen_server' was never awaited
3  self._ready.clear()
4RuntimeWarning: Enable tracemalloc to get the object allocation traceback

optout21 commented at 7:23 am on March 19, 2026: contributor

I had a look, and it seems to be caused by unclean shutdown in Socks5Server. Incoming connections are handled in separate threads, but the thread handles are not stored and they are never joined. As a result, it can happen that stop() returns (it unblocks the accepting thread and joins that), but a connection handler thread is still active. If after stopping the socks server – like in the test case – the node is closed, connection errors can happen in the still-running background thread. Working on a fix.

maflcko commented at 7:32 am on March 19, 2026: member

This happens frequently now: … https://github.com/bitcoin/bitcoin/actions/runs/23283723893/job/67702452594?pr=34857#step:9:3363

maflcko added this to the milestone 31.0 on Mar 19, 2026

optout21 commented at 9:40 am on March 19, 2026: contributor

Attempted fix: https://github.com/bitcoin/bitcoin/pull/34863

vasild commented at 12:17 pm on March 19, 2026: contributor

Just an observation, I am not sure what to make of it - in some cases there is “connection reset by peer” error like in the OP and in some cases not, for example:

02026-03-17T15:57:46.2720000Z 2026-03-17T15:57:45.279622Z TestFramework (INFO): Trying to send a transaction when none of Tor or I2P is reachable
12026-03-17T15:57:46.2720640Z 2026-03-17T15:57:45.877166Z TestFramework (INFO): Stopping nodes
22026-03-17T15:57:46.2721670Z 2026-03-17T15:57:46.201743Z TestFramework (INFO): Cleaning up /Users/runner/work/bitcoin/bitcoin/repo_archive/ci/scratch/test_runner/test_runner_₿_🏃_20260317_154215/p2p_private_broadcast_74 on exit
32026-03-17T15:57:46.2722580Z 2026-03-17T15:57:46.201925Z TestFramework (INFO): Tests successful
42026-03-17T15:57:46.2722810Z 
52026-03-17T15:57:46.2722810Z 
62026-03-17T15:57:46.2722940Z [1mstderr:
72026-03-17T15:57:46.2723790Z [0m/opt/homebrew/Cellar/python@3.14/3.14.3_1/Frameworks/Python.framework/Versions/3.14/lib/python3.14/asyncio/base_events.py:744: RuntimeWarning: coroutine 'NetworkThread.create_listen_server' was never awaited
82026-03-17T15:57:46.2724650Z   self._ready.clear()
92026-03-17T15:57:46.2724990Z RuntimeWarning: Enable tracemalloc to get the object allocation traceback

fanquake commented at 6:25 am on March 20, 2026: member

https://github.com/bitcoin/bitcoin/actions/runs/23291749357/job/67728845083#step:9:3384

optout21 commented at 12:46 pm on March 25, 2026: contributor

I did some more analysis, and managed to force-reproduce the failure locally.

There are 2 errors in the failing CI runs reported:

Exception in forward_sockets() socks5.py:68 . This is due to an exception due to closed socked while the proxy tries to read for forwarding. This exception is handled in Socks5Connection.handle(), logged & stored & masked, so this is fine.
Exception in NetworkThread:

02026-03-17T15:57:46.2722940Z �[1mstderr:
12026-03-17T15:57:46.2723790Z �[0m/opt/homebrew/Cellar/python@3.14/3.14.3_1/Frameworks/Python.framework/Versions/3.14/lib/python3.14/asyncio/base_events.py:744: RuntimeWarning: coroutine 'NetworkThread.create_listen_server' was never awaited

Symbol create_listen_server is in test/functional/test_framework/p2p.py, in NetworkThread.listen(). This is called in one place in p2p_private_broadcast.py: in destinations_factory().

Based on this, here is the hypothesis:

Nodes and the socks5 proxy are set up
Some connections are made (first part of the test)
A new connection is made, but not yet handled by the proxy
The proxy is stopped, but this does not wait or affect the handling threads
Node 0 is restarted
The proxy connection handling happens, it decided to do a listen
The test exits, the runtime is deconstructed, but the listen coroutine has not been called. This is reported as an error by the asyncio Python subsystem.

Note that the comment on stopping the proxy states:

0        # Stop the SOCKS5 proxy server to avoid it being upset by the bitcoin
1        # node disconnecting in the middle of the SOCKS5 handshake when we
2        # restart below.

This is incorrect, as the stopping does not affect the processing thread (only the accpepting thread of the proxy).

Based on this, I managed to force-reproduce the failure, by adding a 0.25s sleep in NetworkThread.listen(), between creating the listening coroutine and scheduling calling it (diff and branch in the details).

 0diff --git a/test/functional/test_framework/p2p.py b/test/functional/test_framework/p2p.py
 1index fae4eb94c2..478ab98351 100755
 2--- a/test/functional/test_framework/p2p.py
 3+++ b/test/functional/test_framework/p2p.py
 4@@ -773,6 +773,8 @@ class NetworkThread(threading.Thread):
 5 
 6         cls.network_event_loop.set_exception_handler(exception_handler)
 7         coroutine = cls.create_listen_server(addr, port, callback, p2p)
 8+        print("create_listen_server")
 9+        import time; time.sleep(0.25)
10         cls.network_event_loop.call_soon_threadsafe(cls.network_event_loop.create_task, coroutine) [@classmethod](/bitcoin-bitcoin/contributor/classmethod/)

With this sleep I’ve managed to trigger the same failure, with about 50% chance on my system.

The good news is that with the fix of #34863, I was not able to trigger the failure, so apparently #34863 is a proper fix.

maflcko commented at 6:28 pm on March 26, 2026: member

With this sleep I’ve managed to trigger the same failure, with about 50% chance on my system.

I ran ./bld-cmake/test/functional/test_runner.py -j 40 $( printf 'p2p_private_broadcast.py %.0s' {1..40} on Linux and 2 out of the 40 runs failed with a slightly different error than the one on macOS:

 02026-03-26T18:23:23.814500Z TestFramework (INFO): Checking abortprivatebroadcast removes a pending private-broadcast transaction
 12026-03-26T18:23:23.818433Z TestFramework (INFO): Checking abortprivatebroadcast fails for non-existent transaction
 22026-03-26T18:23:23.818892Z TestFramework (INFO): Trying to send a transaction when none of Tor or I2P is reachable
 3create_listen_server
 42026-03-26T18:23:23.982948Z TestFramework.p2p (WARNING): Connection lost to 127.0.0.1:35620 due to [Errno 32] Broken pipe
 52026-03-26T18:23:24.173174Z TestFramework (INFO): Stopping nodes
 62026-03-26T18:23:24.231976Z TestFramework.socks5 (ERROR): socks5 request handling failed.
 7Traceback (most recent call last):
 8test/functional/test_framework/socks5.py", line 180, in handle
 9    dest = self.serv.conf.destinations_factory(requested_to_addr, port)
10test/functional/p2p_private_broadcast.py", line 247, in destinations_factory
11    self.network_thread.listen(
12    ~~~~~~~~~~~~~~~~~~~~~~~~~~^
13        addr="127.0.0.1",
14        ^^^^^^^^^^^^^^^^^
15        port=0,
16        ^^^^^^^
17        p2p=listener,
18        ^^^^^^^^^^^^^
19        callback=on_listen_done)
20        ^^^^^^^^^^^^^^^^^^^^^^^^
21test/functional/test_framework/p2p.py", line 778, in listen
22    cls.network_event_loop.call_soon_threadsafe(cls.network_event_loop.create_task, coroutine)
23    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
24AttributeError: 'NoneType' object has no attribute 'call_soon_threadsafe'
252026-03-26T18:23:24.275270Z TestFramework (INFO): Cleaning up /tmp/test_runner_₿_🏃_20260326_192310/p2p_private_broadcast_0 on exit
262026-03-26T18:23:24.275341Z TestFramework (INFO): Tests successful
27
28
29stderr:
30<sys>:0: RuntimeWarning: coroutine 'NetworkThread.create_listen_server' was never awaited
31RuntimeWarning: Enable tracemalloc to get the object allocation traceback

optout21 commented at 11:12 pm on March 26, 2026: contributor

2 out of the 40 runs failed with a slightly different error

In this log the error(s) before the “Tests successful” line are indeed different, but the error printed at the end (“coroutine ‘NetworkThread.create_listen_server’ was never awaited”) is the same as in other failures, and I think this is what is relevant, as in fact this causes the “red CI” (but there’s aways some uncertainty with such sporadic errors…).

ci: failure in p2p_private_broadcast: ConnectionResetError: [Errno 54] Connection reset by peer #34849