Assertion hit on shutdown of bitcoin-node with connected mining interface client

TheCharlatan commented at 10:36 am on September 14, 2025: contributor

Is there an existing issue for this?

I have searched the existing issues

Current behaviour

Running c0894a0a2be032cd9a5d5945643689230ab10255 bitcoin-node with Sjor’s [sv2-tp] connected resulted in an unclean shutdown:

 02025-09-14T10:24:34Z CreateNewBlock(): block weight: 460455 txs: 371 fees: 230840 sigops 1336 
 12025-09-14T10:25:05Z CreateNewBlock(): block weight: 513153 txs: 441 fees: 290027 sigops 1489 
 22025-09-14T10:25:36Z CreateNewBlock(): block weight: 635048 txs: 541 fees: 370980 sigops 1872 
 3^C2025-09-14T10:25:53Z tor: Thread interrupt                                                 
 42025-09-14T10:25:53Z torcontrol thread exit
 52025-09-14T10:25:53Z addcon thread exit                                                                                                                                                    
 62025-09-14T10:25:53Z opencon thread exit
 72025-09-14T10:25:53Z Shutdown: In progress...                                                
 82025-09-14T10:25:53Z net thread exit 
 92025-09-14T10:25:54Z msghand thread exit
102025-09-14T10:25:54Z DumpAnchors: Flush 2 outbound block-relay-only peer addresses to anchors.dat started
112025-09-14T10:25:54Z DumpAnchors: Flush 2 outbound block-relay-only peer addresses to anchors.dat completed (0.00s)
122025-09-14T10:25:54Z scheduler thread exit
132025-09-14T10:25:54Z Writing 591 mempool transactions to file...
142025-09-14T10:25:54Z Writing 0 unbroadcast transactions to file.                   
152025-09-14T10:25:54Z Dumped mempool: 0.005s to copy, 0.004s to dump, 348937 bytes dumped to file
162025-09-14T10:25:54Z Flushed fee estimates to fee_estimates.dat.
17./node/interfaces.cpp:947 chainman: Assertion `m_node.chainman' failed.
18Aborted (core dumped)

Expected behaviour

No assertions triggered.

Steps to reproduce

Seems to happen reliably.

Relevant log output

No response

How did you obtain Bitcoin Core

Compiled from source

What version of Bitcoin Core are you using?

master@c0894a0a2be032cd9a5d5945643689230ab10255

Operating system and version

Ubuntu 24.04

Machine specifications

No response

TheCharlatan commented at 11:29 am on September 14, 2025: contributor

Tried reproducing it again from scratch and could not. I think this was just a version mismatch. There is a crash on the sv2 side, but that is for a different issue tracker. Closing.

TheCharlatan closed this on Sep 14, 2025

ryanofsky commented at 7:13 pm on September 14, 2025: contributor

Nice find, I think this probably is a real bug. I wouldn’t expect it to happen normally because there is an Ipc::disconnectIncoming() call in Shutdown() to close all IPC connections before the chainman object is freed. However, the disconnectIncoming() implementation is only closing the IPC connections without waiting for the objects and threads associated with them to be freed. So if an IPC client makes a request and the node receives it, and the node starts to shut down before the request is executed, asserts like this could be triggered.

A change like the following could fix the issue:

 0--- a/src/ipc/capnp/protocol.cpp
 1+++ b/src/ipc/capnp/protocol.cpp
 2@@ -75,9 +75,23 @@ public:
 3         // Delete incoming connections, except the connection to a parent
 4         // process (if there is one), since a parent process should be able to
 5         // monitor and control this process, even during shutdown.
 6+        int num_disconnected = 0;
 7+        std::promise<void> promise;
 8         m_loop->sync([&] {
 9-            m_loop->m_incoming_connections.remove_if([this](mp::Connection& c) { return &c != m_parent_connection; });
10+            m_loop->m_incoming_connections.remove_if([&](mp::Connection& c) {
11+                if (&c != m_parent_connection) {
12+                    ++num_disconnected;
13+                    return true;
14+                }
15+                return false;
16+            });
17+            if (num_disconnected) m_loop->addAsyncCleanup([&] { promise.set_value(); } );
18         });
19+        if (num_disconnected) {
20+            LogDebug(BCLog::IPC, "Disconnected %i IPC connections, starting cleanup...", num_disconnected);
21+            promise.get_future().wait();
22+            LogDebug(BCLog::IPC, "IPC connection cleanup complete.");
23+        }
24     }
25     void addCleanup(std::type_index type, void* iface, std::function<void()> cleanup) override
26     {

fanquake added the label interfaces on Sep 15, 2025

fanquake commented at 9:19 am on September 15, 2025: member

@Sjors

ryanofsky commented at 12:28 pm on September 15, 2025: contributor

Will reopen since I think this is probably worth following up on, even if it may not be an urgent priority.

If I’m understanding the problem correctly, it might not be to hard to reproduce with a functional test calling IPC methods during shutdown, though the test would be inherently racy. It might be easier to reproduce reliably with a unit test.

ryanofsky reopened this on Sep 15, 2025

Assertion hit on shutdown of bitcoin-node with connected mining interface client #33387