IPC crash with client disconnecting during IPC call

ryanofsky commented at 6:44 pm on June 10, 2025: collaborator

https://github.com/bitcoin/bitcoin/pull/32345 fixes different bitcoin-node crashes during shutdown when IPC connections are not terminated cleanly (either disconnected or unresponsive), letting the node shut down cleanly without blocking in all cases.

But testing https://github.com/bitcoin/bitcoin/pull/32345 in combination with https://github.com/bitcoin/bitcoin/pull/29409 and a custom client that connects and disconnects in a loop @darosior found another crash that seems to happen when a client disconnects in the middle of an IPC call apparently because an assumption in the ~Connection destructor is violated:

https://github.com/bitcoin-core/libmultiprocess/blob/27c7e8e5a581b3c41330e758951251ef11807b11/src/mp/proxy.cpp#L53-L57

that cap’n proto will garbage collect server objects right away when the canp::RpcSystem is destroyed.

Apparently this does not happen if an asynchronous call is currently in progress.

I was able to reproduce the problem with https://github.com/ryanofsky/bitcoin/commit/87b2ae6314e49e637e509c1c1fae852ac83c45ab (tag) and I think the fix will probably require the Connection refcounting described #176 (comment) that I started implementing but abandoned later #176 (comment) because it didn’t seem necessary.

ryanofsky commented at 6:48 pm on June 10, 2025: collaborator

Other information from @darosior:

 02025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server recv request  [#28647](/bitcoin-core-multiprocess/28647/) Init.construct$Params ()
 12025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server send response [#28647](/bitcoin-core-multiprocess/28647/) Init.construct$Results (threadMap = <external capability>)
 22025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server recv request  [#28648](/bitcoin-core-multiprocess/28648/) Init.makeChain$Params (context = (thread = <external capability>))
 32025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server post request  [#28648](/bitcoin-core-multiprocess/28648/) {bitcoin-node-2129701/b-capnp-loop-2167998 (from )}
 42025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server send response [#28648](/bitcoin-core-multiprocess/28648/) Init.makeChain$Results (result = <external capability>)
 52025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server recv request  [#28649](/bitcoin-core-multiprocess/28649/) Chain.initMessage$Params (context = (thread = <external capability>), message = "Oxydation of the Bitcoin Core wallet in progress..")
 62025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server post request  [#28649](/bitcoin-core-multiprocess/28649/) {bitcoin-node-2129701/b-capnp-loop-2167998 (from )}
 72025-06-10T13:46:38Z init message: Oxydation of the Bitcoin Core wallet in progress..
 82025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server send response [#28649](/bitcoin-core-multiprocess/28649/) Chain.initMessage$Results ()
 92025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages5ChainEEE
102025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server: socket disconnected.
112025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages4InitEEE
122025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server recv request  [#28650](/bitcoin-core-multiprocess/28650/) Init.construct$Params ()
132025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server send response [#28650](/bitcoin-core-multiprocess/28650/) Init.construct$Results (threadMap = <external capability>)
142025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server recv request  [#28651](/bitcoin-core-multiprocess/28651/) Init.makeChain$Params (context = (thread = <external capability>))
152025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server post request  [#28651](/bitcoin-core-multiprocess/28651/) {bitcoin-node-2129701/b-capnp-loop-2168002 (from )}
162025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server send response [#28651](/bitcoin-core-multiprocess/28651/) Init.makeChain$Results (result = <external capability>)
172025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server recv request  [#28652](/bitcoin-core-multiprocess/28652/) Chain.initMessage$Params (context = (thread = <external capability>), message = "Oxydation of the Bitcoin Core wallet in progress..")
182025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server post request  [#28652](/bitcoin-core-multiprocess/28652/) {bitcoin-node-2129701/b-capnp-loop-2168002 (from )}
192025-06-10T13:46:38Z init message: Oxydation of the Bitcoin Core wallet in progress..
202025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server send response [#28652](/bitcoin-core-multiprocess/28652/) Chain.initMessage$Results ()
212025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages5ChainEEE
222025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server: socket disconnected.
232025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages4InitEEE
242025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server recv request  [#28653](/bitcoin-core-multiprocess/28653/) Init.construct$Params ()
252025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server send response [#28653](/bitcoin-core-multiprocess/28653/) Init.construct$Results (threadMap = <external capability>)
262025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server recv request  [#28654](/bitcoin-core-multiprocess/28654/) Init.makeChain$Params (context = (thread = <external capability>))
272025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server post request  [#28654](/bitcoin-core-multiprocess/28654/) {bitcoin-node-2129701/b-capnp-loop-2168006 (from )}
282025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server send response [#28654](/bitcoin-core-multiprocess/28654/) Init.makeChain$Results (result = <external capability>)
292025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server recv request  [#28655](/bitcoin-core-multiprocess/28655/) Chain.initMessage$Params (context = (thread = <external capability>), message = "Oxydation of the Bitcoin Core wallet in progress..")
302025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server post request  [#28655](/bitcoin-core-multiprocess/28655/) {bitcoin-node-2129701/b-capnp-loop-2168006 (from )}
312025-06-10T13:46:38Z init message: Oxydation of the Bitcoin Core wallet in progress..
322025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server send response [#28655](/bitcoin-core-multiprocess/28655/) Chain.initMessage$Results ()
332025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages5ChainEEE
342025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server: socket disconnected.
352025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages4InitEEE
362025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server recv request  [#28656](/bitcoin-core-multiprocess/28656/) Init.construct$Params ()
372025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server send response [#28656](/bitcoin-core-multiprocess/28656/) Init.construct$Results (threadMap = <external capability>)
382025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server recv request  [#28657](/bitcoin-core-multiprocess/28657/) Init.makeChain$Params (context = (thread = <external capability>))
392025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server post request  [#28657](/bitcoin-core-multiprocess/28657/) {bitcoin-node-2129701/b-capnp-loop-2168010 (from )}
402025-06-10T13:46:38Z [ipc] {bitcoin-node-2129701/b-capnp-loop-2129703} IPC server: socket disconnected.
41bitcoin-node: ./ipc/libmultiprocess/include/mp/proxy.h:63: mp::EventLoop& mp::EventLoopRef::operator*() const: Assertion `m_loop' failed.
42Aborted

 02025-06-10T14:16:01Z [ipc] {bitcoin-node-2177791/b-capnp-loop-2177793} IPC server recv request  [#1885](/bitcoin-core-multiprocess/1885/) Init.construct$Params ()
 12025-06-10T14:16:01Z [ipc] {bitcoin-node-2177791/b-capnp-loop-2177793} IPC server send response [#1885](/bitcoin-core-multiprocess/1885/) Init.construct$Results (threadMap = <external capability>)
 22025-06-10T14:16:01Z [ipc] {bitcoin-node-2177791/b-capnp-loop-2177793} IPC server recv request  [#1886](/bitcoin-core-multiprocess/1886/) Init.makeChain$Params (context = (thread = <external capability>))
 32025-06-10T14:16:01Z [ipc] {bitcoin-node-2177791/b-capnp-loop-2177793} IPC server post request  [#1886](/bitcoin-core-multiprocess/1886/) {bitcoin-node-2177791/b-capnp-loop-2180334 (from )}
 42025-06-10T14:16:01Z [ipc] {bitcoin-node-2177791/b-capnp-loop-2177793} IPC server send response [#1886](/bitcoin-core-multiprocess/1886/) Init.makeChain$Results (result = <external capability>)
 52025-06-10T14:16:01Z [ipc] {bitcoin-node-2177791/b-capnp-loop-2177793} IPC server recv request  [#1887](/bitcoin-core-multiprocess/1887/) Chain.initMessage$Params (context = (thread = <external capability>), message = "Oxydation of the Bitcoin Core wallet in progress..")
 62025-06-10T14:16:01Z [ipc] {bitcoin-node-2177791/b-capnp-loop-2177793} IPC server post request  [#1887](/bitcoin-core-multiprocess/1887/) {bitcoin-node-2177791/b-capnp-loop-2180334 (from )}
 72025-06-10T14:16:01Z init message: Oxydation of the Bitcoin Core wallet in progress..
 82025-06-10T14:16:01Z [ipc] {bitcoin-node-2177791/b-capnp-loop-2177793} IPC server: socket disconnected.
 92025-06-10T14:16:01Z [ipc] {bitcoin-node-2177791/b-capnp-loop-2177793} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages4InitEEE
102025-06-10T14:16:01Z [ipc] {bitcoin-node-2177791/b-capnp-loop-2177793} IPC server send response [#1887](/bitcoin-core-multiprocess/1887/) Chain.initMessage$Results ()
112025-06-10T14:16:01Z [ipc] {bitcoin-node-2177791/b-capnp-loop-2177793} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages5ChainEEE
12=================================================================
13==2177791==ERROR: AddressSanitizer: heap-use-after-free on address 0x619000145090 at pc 0x55e3f7305d91 bp 0x7ffa213914f0 sp 0x7ffa213914e8
14READ of size 8 at 0x619000145090 thread T2 (b-capnp-loop)
15    [#0](/bitcoin-core-multiprocess/0/) 0x55e3f7305d90 in mp::EventLoopRef::operator->() const ipc/libmultiprocess/include/mp/proxy.h:64
16    [#1](/bitcoin-core-multiprocess/1/) 0x55e3f7305d90 in mp::Connection::addAsyncCleanup(std::function<void ()>) ipc/libmultiprocess/src/mp/proxy.cpp:168
17    [#2](/bitcoin-core-multiprocess/2/) 0x55e3f4c96b89 in mp::ProxyServerBase<ipc::capnp::messages::Chain, interfaces::Chain>::~ProxyServerBase() ipc/libmultiprocess/include/mp/proxy-io.h:480
18    [#3](/bitcoin-core-multiprocess/3/) 0x55e3f4c6fdf0 in mp::ProxyServer<ipc::capnp::messages::Chain>::~ProxyServer() /home/darosior/code/core/bitcoin/sanmultiprocbuild/src/ipc/capnp/chain.capnp.proxy-types.c++:8
19    [#4](/bitcoin-core-multiprocess/4/) 0x7ffa266b6985 in non-virtual thunk to capnp::LocalClient::~LocalClient() (/lib/x86_64-linux-gnu/libcapnp-rpc-0.9.2.so+0x7b985)
20    [#5](/bitcoin-core-multiprocess/5/) 0x7ffa266b6ca4 in kj::_::HeapDisposer<kj::_::AttachmentPromiseNode<kj::Own<capnp::LocalClient> > >::disposeImpl(void*) const (/lib/x86_64-linux-gnu/libcapnp-rpc-0.9.2.so+0x7bca4)
21    [#6](/bitcoin-core-multiprocess/6/) 0x7ffa2644e3e8 in kj::_::runCatchingExceptions(kj::_::Runnable&) (/lib/x86_64-linux-gnu/libkj-0.9.2.so+0x343e8)
22    [#7](/bitcoin-core-multiprocess/7/) 0x7ffa264edc81 in kj::_::ForkHubBase::fire() (/lib/x86_64-linux-gnu/libkj-async-0.9.2.so+0x46c81)
23    [#8](/bitcoin-core-multiprocess/8/) 0x7ffa264ea797 in kj::EventLoop::turn() (/lib/x86_64-linux-gnu/libkj-async-0.9.2.so+0x43797)
24    [#9](/bitcoin-core-multiprocess/9/) 0x7ffa264f1fb8 in kj::_::waitImpl(kj::Own<kj::_::PromiseNode>&&, kj::_::ExceptionOrValue&, kj::WaitScope&) (/lib/x86_64-linux-gnu/libkj-async-0.9.2.so+0x4afb8)
25    [#10](/bitcoin-core-multiprocess/10/) 0x55e3f733ecf9 in kj::Promise<unsigned long>::wait(kj::WaitScope&) /usr/include/kj/async-inl.h:1120
26    [#11](/bitcoin-core-multiprocess/11/) 0x55e3f7331c3e in mp::EventLoop::loop() ipc/libmultiprocess/src/mp/proxy.cpp:227
27    [#12](/bitcoin-core-multiprocess/12/) 0x55e3f4272383 in operator() ipc/capnp/protocol.cpp:96
28    [#13](/bitcoin-core-multiprocess/13/) 0x55e3f4272383 in __invoke_impl<void, ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::<lambda()> > /usr/include/c++/12/bits/invoke.h:61
29    [#14](/bitcoin-core-multiprocess/14/) 0x55e3f4272383 in __invoke<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::<lambda()> > /usr/include/c++/12/bits/invoke.h:96
30    [#15](/bitcoin-core-multiprocess/15/) 0x55e3f4272383 in _M_invoke<0> /usr/include/c++/12/bits/std_thread.h:252
31    [#16](/bitcoin-core-multiprocess/16/) 0x55e3f4272383 in operator() /usr/include/c++/12/bits/std_thread.h:259
32    [#17](/bitcoin-core-multiprocess/17/) 0x55e3f4272383 in _M_run /usr/include/c++/12/bits/std_thread.h:210
33    [#18](/bitcoin-core-multiprocess/18/) 0x7ffa260d44a2  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xd44a2)
34    [#19](/bitcoin-core-multiprocess/19/) 0x7ffa256a81f4 in start_thread nptl/pthread_create.c:442
35    [#20](/bitcoin-core-multiprocess/20/) 0x7ffa2572889b in clone3 ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
36
370x619000145090 is located 16 bytes inside of 1032-byte region [0x619000145080,0x619000145488)
38freed by thread T2 (b-capnp-loop) here:
39    [#0](/bitcoin-core-multiprocess/0/) 0x7ffa268ba3c8 in operator delete(void*, unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:164
40    [#1](/bitcoin-core-multiprocess/1/) 0x55e3f429b035 in std::__new_allocator<std::_List_node<mp::Connection> >::deallocate(std::_List_node<mp::Connection>*, unsigned long) /usr/include/c++/12/bits/new_allocator.h:158
41    [#2](/bitcoin-core-multiprocess/2/) 0x55e3f429b035 in std::allocator<std::_List_node<mp::Connection> >::deallocate(std::_List_node<mp::Connection>*, unsigned long) /usr/include/c++/12/bits/allocator.h:200
42    [#3](/bitcoin-core-multiprocess/3/) 0x55e3f429b035 in std::allocator_traits<std::allocator<std::_List_node<mp::Connection> > >::deallocate(std::allocator<std::_List_node<mp::Connection> >&, std::_List_node<mp::Connection>*, unsigned long) /usr/include/c++/12/bits/alloc_traits.h:496
43    [#4](/bitcoin-core-multiprocess/4/) 0x55e3f429b035 in std::__cxx11::_List_base<mp::Connection, std::allocator<mp::Connection> >::_M_put_node(std::_List_node<mp::Connection>*) /usr/include/c++/12/bits/stl_list.h:522
44    [#5](/bitcoin-core-multiprocess/5/) 0x55e3f429b035 in std::__cxx11::list<mp::Connection, std::allocator<mp::Connection> >::_M_erase(std::_List_iterator<mp::Connection>) /usr/include/c++/12/bits/stl_list.h:2024
45    [#6](/bitcoin-core-multiprocess/6/) 0x55e3f429b035 in std::__cxx11::list<mp::Connection, std::allocator<mp::Connection> >::erase(std::_List_const_iterator<mp::Connection>) /usr/include/c++/12/bits/list.tcc:158
46    [#7](/bitcoin-core-multiprocess/7/) 0x55e3f429b035 in mp::_Serve<ipc::capnp::messages::Init, interfaces::Init>(mp::EventLoop&, kj::Own<kj::AsyncIoStream>&&, interfaces::Init&)::{lambda()#2}::operator()() const ipc/libmultiprocess/include/mp/proxy-io.h:590
47
48previously allocated by thread T2 (b-capnp-loop) here:
49    [#0](/bitcoin-core-multiprocess/0/) 0x7ffa268b94c8 in operator new(unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:95
50    [#1](/bitcoin-core-multiprocess/1/) 0x55e3f42be0a8 in std::__new_allocator<std::_List_node<mp::Connection> >::allocate(unsigned long, void const*) /usr/include/c++/12/bits/new_allocator.h:137
51    [#2](/bitcoin-core-multiprocess/2/) 0x55e3f42be0a8 in std::allocator<std::_List_node<mp::Connection> >::allocate(unsigned long) /usr/include/c++/12/bits/allocator.h:188
52    [#3](/bitcoin-core-multiprocess/3/) 0x55e3f42be0a8 in std::allocator_traits<std::allocator<std::_List_node<mp::Connection> > >::allocate(std::allocator<std::_List_node<mp::Connection> >&, unsigned long) /usr/include/c++/12/bits/alloc_traits.h:464
53    [#4](/bitcoin-core-multiprocess/4/) 0x55e3f42be0a8 in std::__cxx11::_List_base<mp::Connection, std::allocator<mp::Connection> >::_M_get_node() /usr/include/c++/12/bits/stl_list.h:518
54    [#5](/bitcoin-core-multiprocess/5/) 0x55e3f42be0a8 in std::_List_node<mp::Connection>* std::__cxx11::list<mp::Connection, std::allocator<mp::Connection> >::_M_create_node<mp::EventLoop&, kj::Own<kj::AsyncIoStream>, mp::_Serve<ipc::capnp::messages::Init, interfaces::Init>(mp::EventLoop&, kj::Own<kj::AsyncIoStream>&&, interfaces::Init&)::{lambda(mp::Connection&)#1}>(mp::EventLoop&, kj::Own<kj::AsyncIoStream>&&, mp::_Serve<ipc::capnp::messages::Init, interfaces::Init>(mp::EventLoop&, kj::Own<kj::AsyncIoStream>&&, interfaces::Init&)::{lambda(mp::Connection&)#1}&&) /usr/include/c++/12/bits/stl_list.h:710
55    [#6](/bitcoin-core-multiprocess/6/) 0x55e3f42be0a8 in void std::__cxx11::list<mp::Connection, std::allocator<mp::Connection> >::_M_insert<mp::EventLoop&, kj::Own<kj::AsyncIoStream>, mp::_Serve<ipc::capnp::messages::Init, interfaces::Init>(mp::EventLoop&, kj::Own<kj::AsyncIoStream>&&, interfaces::Init&)::{lambda(mp::Connection&)#1}>(std::_List_iterator<mp::Connection>, mp::EventLoop&, kj::Own<kj::AsyncIoStream>&&, mp::_Serve<ipc::capnp::messages::Init, interfaces::Init>(mp::EventLoop&, kj::Own<kj::AsyncIoStream>&&, interfaces::Init&)::{lambda(mp::Connection&)#1}&&) /usr/include/c++/12/bits/stl_list.h:2005
56    [#7](/bitcoin-core-multiprocess/7/) 0x55e3f42be0a8 in mp::Connection& std::__cxx11::list<mp::Connection, std::allocator<mp::Connection> >::emplace_front<mp::EventLoop&, kj::Own<kj::AsyncIoStream>, mp::_Serve<ipc::capnp::messages::Init, interfaces::Init>(mp::EventLoop&, kj::Own<kj::AsyncIoStream>&&, interfaces::Init&)::{lambda(mp::Connection&)#1}>(mp::EventLoop&, kj::Own<kj::AsyncIoStream>&&, mp::_Serve<ipc::capnp::messages::Init, interfaces::Init>(mp::EventLoop&, kj::Own<kj::AsyncIoStream>&&, interfaces::Init&)::{lambda(mp::Connection&)#1}&&) /usr/include/c++/12/bits/stl_list.h:1271
57    [#8](/bitcoin-core-multiprocess/8/) 0x55e3f42be0a8 in void mp::_Serve<ipc::capnp::messages::Init, interfaces::Init>(mp::EventLoop&, kj::Own<kj::AsyncIoStream>&&, interfaces::Init&) ipc/libmultiprocess/include/mp/proxy-io.h:581
58
59Thread T2 (b-capnp-loop) created by T0 here:
60    [#0](/bitcoin-core-multiprocess/0/) 0x7ffa26849726 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:207
61    [#1](/bitcoin-core-multiprocess/1/) 0x7ffa260d4578 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/lib/x86_64-linux-gnu/libstdc++.so.6+0xd4578)
62
63SUMMARY: AddressSanitizer: heap-use-after-free ipc/libmultiprocess/include/mp/proxy.h:64 in mp::EventLoopRef::operator->() const
64Shadow bytes around the buggy address:
65  0x0c32800209c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
66  0x0c32800209d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
67  0x0c32800209e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
68  0x0c32800209f0: fd fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
69  0x0c3280020a00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
70=>0x0c3280020a10: fd fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd
71  0x0c3280020a20: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
72  0x0c3280020a30: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
73  0x0c3280020a40: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
74  0x0c3280020a50: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
75  0x0c3280020a60: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
76Shadow byte legend (one shadow byte represents 8 application bytes):
77  Addressable:           00
78  Partially addressable: 01 02 03 04 05 06 07 
79  Heap left redzone:       fa
80  Freed heap region:       fd
81  Stack left redzone:      f1
82  Stack mid redzone:       f2
83  Stack right redzone:     f3
84  Stack after return:      f5
85  Stack use after scope:   f8
86  Global redzone:          f9
87  Global init order:       f6
88  Poisoned by user:        f7
89  Container overflow:      fc
90  Array cookie:            ac
91  Intra object redzone:    bb
92  ASan internal:           fe
93  Left alloca redzone:     ca
94  Right alloca redzone:    cb
95==2177791==ABORTING

ryanofsky referenced this in commit 6d7c88b7bb on Jun 11, 2025

ryanofsky commented at 7:34 pm on June 11, 2025: collaborator

I was able to write a unit test that reproduces the crash and confirms cap’n proto really is not destroying server objects before m_rpc_system.reset(); returns if a connection is destroyed is in the middle of an IPC call. Code was written with the assumption that the reset call would free all Server objects, so to fix the bug I need to switch to another approach like the connection refcounting one I started implementing #176 (comment).

New unit test is in commit 6d7c88b7bb7b42557c2cf5ef5bb24bd08b2efc6e (tag)

ryanofsky referenced this in commit 42caa69d57 on Jun 13, 2025

ryanofsky referenced this in commit 5c40b1786a on Jun 13, 2025

ryanofsky commented at 8:17 pm on June 13, 2025: collaborator

Latest version of https://github.com/bitcoin/bitcoin/pull/32345 (tag pr/ipc-stop.8) should fix these issues. The actual fixes are implemented in base PR #160 in these commits:

42caa69d571134e80b7da7662b45e6d188520ebc Prevent EventLoop async cleanup thread early exit during shutdown
5c40b1786a72dc78fec931cebc45c78e7e814a4b Prevent IPC server crash if disconnected during IPC call
4c3373459d3a2abfa2c5e143f271ec8a4e25b346 test: Test disconnects during IPC calls

Contrary to previous comment I did not wind up needing to implement the connection refcounting approach since I found a simpler more direct fix. But I did do more work on it before finding the simpler fix and plan to make it a followup since it is a bigger change but simplifies things overall

ryanofsky referenced this in commit 241f9ce1df on Jun 13, 2025

ryanofsky referenced this in commit 6910fe8fee on Jun 13, 2025

ryanofsky referenced this in commit 27304b547c on Jun 13, 2025

ryanofsky referenced this in commit fc89c23766 on Jun 16, 2025

ryanofsky referenced this in commit 7dc8befd86 on Jun 16, 2025

ryanofsky referenced this in commit ea38392960 on Jun 16, 2025

ryanofsky referenced this in commit 949573da84 on Jun 16, 2025

pinheadmz commented at 7:45 pm on June 16, 2025: none

Not sure if my observation belongs here or in a new issue. Testing https://github.com/bitcoin/bitcoin/pull/32297 I tried to use socat to inspect the IPC request:

socat -v UNIX-LISTEN:/Users/matthewzipkin/.bitcoin/regtest/node.sock STDOUT

and then make a bitcoin-cli request even though bitcoin-node was not running:

bccli -regtest getblockcount

error:

0libc++abi: terminating due to uncaught exception of type kj::ExceptionImpl: /opt/homebrew/include/kj/memory.h:258: failed: expected ptr != nullptr; null Own<> dereference
1stack: 1070007cb 10489843b 10494a81f 10494a74b 104951f4f 104951def 104951d93 104951d33 104951c73 104951c17 10494f59b 1048acccb 1048ac9d7 104ddc727 104ddc077 10488ab97 10488aa0f 10488a8e7 104889977 1a01b6f93 1a01b1d33
2Abort trap: 6

Stepping through with lldb I found the error in ConnectStream():

https://github.com/bitcoin-core/libmultiprocess/blob/019839758085cb094f107e9ac354cbda79b388e2/include/mp/proxy-io.h#L559-L580

What I believe happens is, we get a file descriptor way earlier in the stack (in bitcoin: src/ipc/interfaces.cpp::connectAddress() but by the time we get to ConnectStream() the listener has disconnected (or, just isn’t bitcoin) and then things start going wrong in ProxyClientBase()

ryanofsky commented at 8:02 pm on June 16, 2025: collaborator

re: #182 (comment)

Testing https://github.com/bitcoin/bitcoin/pull/32297 I tried to use socat to inspect the IPC request

Thanks I created #183 to track this issue and replied there. I think the changes in https://github.com/bitcoin/bitcoin/pull/32345 to detect disconnects might help this but would probably also need to be accompanied by changes in https://github.com/bitcoin/bitcoin/pull/32297 too

ryanofsky referenced this in commit 258a617c1e on Jun 19, 2025

fanquake referenced this in commit f58de8749e on Aug 18, 2025

ryanofsky commented at 11:29 am on August 23, 2025: collaborator

bitcoin/bitcoin#32345 is merged now so will resolving this

ryanofsky closed this on Aug 23, 2025

IPC crash with client disconnecting during IPC call #182