ci: failure in p2p_private_broadcast: ConnectionResetError: [Errno 54] Connection reset by peer #34849

rkrux commented at 10:15 AM on March 18, 2026: contributor

https://github.com/bitcoin/bitcoin/actions/runs/23192191180/job/67390951612?pr=34841

2026-03-17T11:57:04.087893Z TestFramework (INFO): Trying to send a transaction when none of Tor or I2P is reachable
2026-03-17T11:57:04.289600Z TestFramework.socks5 (ERROR): socks5 request handling failed.
Traceback (most recent call last):
  File "/Users/runner/work/bitcoin/bitcoin/repo_archive/test/functional/test_framework/socks5.py", line 185, in handle
    forward_sockets(self.conn, conn_to)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/work/bitcoin/bitcoin/repo_archive/test/functional/test_framework/socks5.py", line 68, in forward_sockets
    data = s.recv(4096)
ConnectionResetError: [Errno 54] Connection reset by peer
2026-03-17T11:57:04.780915Z TestFramework (INFO): Stopping nodes
2026-03-17T11:57:04.972885Z TestFramework (INFO): Cleaning up /Users/runner/work/bitcoin/bitcoin/repo_archive/ci/scratch/test_runner/test_runner_₿_🏃_20260317_114135/p2p_private_broadcast_74 on exit
2026-03-17T11:57:04.972946Z TestFramework (INFO): Tests successful


stderr:
/opt/homebrew/Cellar/python@3.14/3.14.3_1/Frameworks/Python.framework/Versions/3.14/lib/python3.14/asyncio/base_events.py:744: RuntimeWarning: coroutine 'NetworkThread.create_listen_server' was never awaited
  self._ready.clear()
RuntimeWarning: Enable tracemalloc to get the object allocation traceback

maflcko added the label CI failed on Mar 18, 2026

maflcko added the label Tests on Mar 18, 2026

fanquake renamed this:
~~(p2p_private_broadcast): ConnectionResetError: [Errno 54] Connection reset by peer~~
ci: failure in p2p_private_broadcast: ConnectionResetError: [Errno 54] Connection reset by peer
on Mar 19, 2026

fanquake commented at 5:39 AM on March 19, 2026: member

cc @vasild

fanquake commented at 5:40 AM on March 19, 2026: member

Also seen here: https://github.com/bitcoin/bitcoin/actions/runs/23202474958/job/67428457683#step:9:3361


stderr:
/opt/homebrew/Cellar/python@3.14/3.14.3_1/Frameworks/Python.framework/Versions/3.14/lib/python3.14/asyncio/base_events.py:744: RuntimeWarning: coroutine 'NetworkThread.create_listen_server' was never awaited
  self._ready.clear()
RuntimeWarning: Enable tracemalloc to get the object allocation traceback

optout21 commented at 7:23 AM on March 19, 2026: contributor

I had a look, and it seems to be caused by unclean shutdown in Socks5Server. Incoming connections are handled in separate threads, but the thread handles are not stored and they are never joined. As a result, it can happen that stop() returns (it unblocks the accepting thread and joins that), but a connection handler thread is still active. If after stopping the socks server -- like in the test case -- the node is closed, connection errors can happen in the still-running background thread. Working on a fix.

maflcko commented at 7:32 AM on March 19, 2026: member

This happens frequently now: ... https://github.com/bitcoin/bitcoin/actions/runs/23283723893/job/67702452594?pr=34857#step:9:3363

maflcko added this to the milestone 31.0 on Mar 19, 2026

optout21 commented at 9:40 AM on March 19, 2026: contributor

Attempted fix: https://github.com/bitcoin/bitcoin/pull/34863

vasild commented at 12:17 PM on March 19, 2026: contributor

Just an observation, I am not sure what to make of it - in some cases there is "connection reset by peer" error like in the OP and in some cases not, for example:

2026-03-17T15:57:46.2720000Z 2026-03-17T15:57:45.279622Z TestFramework (INFO): Trying to send a transaction when none of Tor or I2P is reachable
2026-03-17T15:57:46.2720640Z 2026-03-17T15:57:45.877166Z TestFramework (INFO): Stopping nodes
2026-03-17T15:57:46.2721670Z 2026-03-17T15:57:46.201743Z TestFramework (INFO): Cleaning up /Users/runner/work/bitcoin/bitcoin/repo_archive/ci/scratch/test_runner/test_runner_₿_🏃_20260317_154215/p2p_private_broadcast_74 on exit
2026-03-17T15:57:46.2722580Z 2026-03-17T15:57:46.201925Z TestFramework (INFO): Tests successful
2026-03-17T15:57:46.2722810Z 
2026-03-17T15:57:46.2722810Z 
2026-03-17T15:57:46.2722940Z [1mstderr:
2026-03-17T15:57:46.2723790Z [0m/opt/homebrew/Cellar/python@3.14/3.14.3_1/Frameworks/Python.framework/Versions/3.14/lib/python3.14/asyncio/base_events.py:744: RuntimeWarning: coroutine 'NetworkThread.create_listen_server' was never awaited
2026-03-17T15:57:46.2724650Z   self._ready.clear()
2026-03-17T15:57:46.2724990Z RuntimeWarning: Enable tracemalloc to get the object allocation traceback

fanquake commented at 6:25 AM on March 20, 2026: member

https://github.com/bitcoin/bitcoin/actions/runs/23291749357/job/67728845083#step:9:3384

optout21 commented at 12:46 PM on March 25, 2026: contributor

I did some more analysis, and managed to force-reproduce the failure locally.

There are 2 errors in the failing CI runs reported:

Exception in forward_sockets() socks5.py:68 . This is due to an exception due to closed socked while the proxy tries to read for forwarding. This exception is handled in Socks5Connection.handle(), logged & stored & masked, so this is fine.
Exception in NetworkThread:

2026-03-17T15:57:46.2722940Z �[1mstderr:
2026-03-17T15:57:46.2723790Z �[0m/opt/homebrew/Cellar/python@3.14/3.14.3_1/Frameworks/Python.framework/Versions/3.14/lib/python3.14/asyncio/base_events.py:744: RuntimeWarning: coroutine 'NetworkThread.create_listen_server' was never awaited

Symbol create_listen_server is in test/functional/test_framework/p2p.py, in NetworkThread.listen(). This is called in one place in p2p_private_broadcast.py: in destinations_factory().

Based on this, here is the hypothesis:

Nodes and the socks5 proxy are set up
Some connections are made (first part of the test)
A new connection is made, but not yet handled by the proxy
The proxy is stopped, but this does not wait or affect the handling threads
Node 0 is restarted
The proxy connection handling happens, it decided to do a listen
The test exits, the runtime is deconstructed, but the listen coroutine has not been called. This is reported as an error by the asyncio Python subsystem.

Note that the comment on stopping the proxy states:

        # Stop the SOCKS5 proxy server to avoid it being upset by the bitcoin
        # node disconnecting in the middle of the SOCKS5 handshake when we
        # restart below.

This is incorrect, as the stopping does not affect the processing thread (only the accpepting thread of the proxy).

Based on this, I managed to force-reproduce the failure, by adding a 0.25s sleep in NetworkThread.listen(), between creating the listening coroutine and scheduling calling it (diff and branch in the details).

<details> <summary>Branch: https://github.com/optout21/bitcoin/tree/is34849-force, diff:</summary>

diff --git a/test/functional/test_framework/p2p.py b/test/functional/test_framework/p2p.py
index fae4eb94c2..478ab98351 100755
--- a/test/functional/test_framework/p2p.py
+++ b/test/functional/test_framework/p2p.py
@@ -773,6 +773,8 @@ class NetworkThread(threading.Thread):
 
         cls.network_event_loop.set_exception_handler(exception_handler)
         coroutine = cls.create_listen_server(addr, port, callback, p2p)
+        print("create_listen_server")
+        import time; time.sleep(0.25)
         cls.network_event_loop.call_soon_threadsafe(cls.network_event_loop.create_task, coroutine) [@classmethod](/bitcoin-bitcoin/contributor/classmethod/)

</details>

With this sleep I've managed to trigger the same failure, with about 50% chance on my system.

The good news is that with the fix of #34863, I was not able to trigger the failure, so apparently #34863 is a proper fix.

maflcko commented at 6:28 PM on March 26, 2026: member

With this sleep I've managed to trigger the same failure, with about 50% chance on my system.

I ran ./bld-cmake/test/functional/test_runner.py -j 40 $( printf 'p2p_private_broadcast.py %.0s' {1..40} on Linux and 2 out of the 40 runs failed with a slightly different error than the one on macOS:

2026-03-26T18:23:23.814500Z TestFramework (INFO): Checking abortprivatebroadcast removes a pending private-broadcast transaction
2026-03-26T18:23:23.818433Z TestFramework (INFO): Checking abortprivatebroadcast fails for non-existent transaction
2026-03-26T18:23:23.818892Z TestFramework (INFO): Trying to send a transaction when none of Tor or I2P is reachable
create_listen_server
2026-03-26T18:23:23.982948Z TestFramework.p2p (WARNING): Connection lost to 127.0.0.1:35620 due to [Errno 32] Broken pipe
2026-03-26T18:23:24.173174Z TestFramework (INFO): Stopping nodes
2026-03-26T18:23:24.231976Z TestFramework.socks5 (ERROR): socks5 request handling failed.
Traceback (most recent call last):
test/functional/test_framework/socks5.py", line 180, in handle
    dest = self.serv.conf.destinations_factory(requested_to_addr, port)
test/functional/p2p_private_broadcast.py", line 247, in destinations_factory
    self.network_thread.listen(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^
        addr="127.0.0.1",
        ^^^^^^^^^^^^^^^^^
        port=0,
        ^^^^^^^
        p2p=listener,
        ^^^^^^^^^^^^^
        callback=on_listen_done)
        ^^^^^^^^^^^^^^^^^^^^^^^^
test/functional/test_framework/p2p.py", line 778, in listen
    cls.network_event_loop.call_soon_threadsafe(cls.network_event_loop.create_task, coroutine)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'call_soon_threadsafe'
2026-03-26T18:23:24.275270Z TestFramework (INFO): Cleaning up /tmp/test_runner_₿_🏃_20260326_192310/p2p_private_broadcast_0 on exit
2026-03-26T18:23:24.275341Z TestFramework (INFO): Tests successful


stderr:
<sys>:0: RuntimeWarning: coroutine 'NetworkThread.create_listen_server' was never awaited
RuntimeWarning: Enable tracemalloc to get the object allocation traceback

optout21 commented at 11:12 PM on March 26, 2026: contributor

2 out of the 40 runs failed with a slightly different error

In this log the error(s) before the "Tests successful" line are indeed different, but the error printed at the end ("coroutine 'NetworkThread.create_listen_server' was never awaited") is the same as in other failures, and I think this is what is relevant, as in fact this causes the "red CI" (but there's aways some uncertainty with such sporadic errors...).

maflcko commented at 10:04 AM on April 8, 2026: member

Looks like this missed 31.0rc3, so maybe it could make sense to move it to 31.1?

achow101 removed this from the milestone 31.0 on Apr 8, 2026

achow101 added this to the milestone 31.1 on Apr 8, 2026

fanquake commented at 1:48 PM on April 20, 2026: member

https://github.com/fanquake/bitcoin/actions/runs/24664438913/job/72118355917#step:9:3782

achow101 closed this on Apr 20, 2026

achow101 referenced this in commit 6b3dd6314f on Apr 20, 2026