SIGABRT in ~ProxyClientBase with #29409 and rust client #219

Originally posted by @TheCharlatan in https://github.com/bitcoin/bitcoin/issues/29409#issuecomment-3345899017

Thanks for the rebase :)

I updated darosior's core_bdk_wallet to use this newest version and I ran into a few crashes again. This is the one I managed to get a reproducible backtrace for:

<details> <summary>Crash backtrace + ipc debug logging</summary>

2025-09-29T11:18:20Z [ipc] {bitcoin-node-165737/b-capnp-loop-165742} IPC server post request  [#32](/bitcoin-core-multiprocess/32/) {bitcoin-node-165737/b-capnp-loop-166141 (from )}
2025-09-29T11:18:20Z [ipc] {bitcoin-node-165737/b-capnp-loop-165742} IPC server send response [#32](/bitcoin-core-multiprocess/32/) Chain.handleNotifications$Results (result = <external capability>)
2025-09-29T11:18:20Z [ipc] {bitcoin-node-165737/b-capnp-loop-165742} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages7HandlerEEE
2025-09-29T11:18:20Z [ipc] {bitcoin-node-165737/b-capnp-loop-165873} IPC client destroy N2mp11ProxyClientIN3ipc5capnp8messages18ChainNotificationsEEE
2025-09-29T11:18:20Z [ipc] {bitcoin-node-165737/b-capnp-loop-165873} IPC client first request from current thread, constructing waiter
2025-09-29T11:18:20Z [ipc] {bitcoin-node-165737/b-capnp-loop-165873} IPC client send ChainNotifications.destroy$Params (context = (thread = <external capability>, callbackThread = <external capability>))
2025-09-29T11:18:20Z [ipc] {bitcoin-node-165737/b-capnp-loop-165742} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages5ChainEEE
2025-09-29T11:18:20Z [ipc] IPC client method call interrupted by disconnect.
terminate called after throwing an instance of 'ipc::Exception'
  what():  IPC client method call interrupted by disconnect.
[Thread 0x7ffed37fe6c0 (LWP 166141) exited]
2025-09-29T11:18:20Z [ipc] {bitcoin-node-165737/b-capnp-loop-165742} IPC server: socket disconnected.
2025-09-29T11:18:20Z [ipc] {bitcoin-node-165737/b-capnp-loop-165742} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages4InitEEE

Thread 45 "b-capnp-loop" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffed2ffd6c0 (LWP 165873)]
__pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
warning: 44	./nptl/pthread_kill.c: No such file or directory
(gdb) bt
[#0](/bitcoin-core-multiprocess/0/)  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
[#1](/bitcoin-core-multiprocess/1/)  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
[#2](/bitcoin-core-multiprocess/2/)  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
[#3](/bitcoin-core-multiprocess/3/)  0x00007ffff724527e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
[#4](/bitcoin-core-multiprocess/4/)  0x00007ffff72288ff in __GI_abort () at ./stdlib/abort.c:79
[#5](/bitcoin-core-multiprocess/5/)  0x00007ffff76a5ff5 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
[#6](/bitcoin-core-multiprocess/6/)  0x00007ffff76bb0da in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
[#7](/bitcoin-core-multiprocess/7/)  0x00007ffff76a5a55 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6
[#8](/bitcoin-core-multiprocess/8/)  0x000055555563c47f in __clang_call_terminate ()
[#9](/bitcoin-core-multiprocess/9/)  0x0000555555b39999 in mp::ProxyClientBase<ipc::capnp::messages::ChainNotifications, interfaces::Chain::Notifications>::~ProxyClientBase (this=0x7ffed40cfe60)
    at ./ipc/libmultiprocess/include/mp/proxy-io.h:468
[#10](/bitcoin-core-multiprocess/10/) 0x0000555555b36e9f in mp::ProxyClient<ipc::capnp::messages::ChainNotifications>::~ProxyClient (this=0x28769)
    at /home/drgrid/bitcoin/build_dev_mode_clang/src/ipc/capnp/chain.capnp.proxy-types.c++:12
[#11](/bitcoin-core-multiprocess/11/) 0x000055555575637e in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x7ffed40049a0)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/shared_ptr_base.h:346
[#12](/bitcoin-core-multiprocess/12/) std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=<optimized out>)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/shared_ptr_base.h:1071
[#13](/bitcoin-core-multiprocess/13/) std::__shared_ptr<interfaces::Chain::Notifications, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=<optimized out>)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/shared_ptr_base.h:1524
[#14](/bitcoin-core-multiprocess/14/) node::(anonymous namespace)::NotificationsProxy::~NotificationsProxy (this=<optimized out>) at ./node/interfaces.cpp:456
[#15](/bitcoin-core-multiprocess/15/) 0x0000555555755fbc in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x7ffed4000d80)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/shared_ptr_base.h:346
[#16](/bitcoin-core-multiprocess/16/) std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=<optimized out>)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/shared_ptr_base.h:1071
[#17](/bitcoin-core-multiprocess/17/) std::__shared_ptr<node::(anonymous namespace)::NotificationsProxy, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=<optimized out>)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/shared_ptr_base.h:1524
[#18](/bitcoin-core-multiprocess/18/) std::__shared_ptr<node::(anonymous namespace)::NotificationsProxy, (__gnu_cxx::_Lock_policy)2>::reset (this=0x7ffed4005a50)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/shared_ptr_base.h:1642
[#19](/bitcoin-core-multiprocess/19/) node::(anonymous namespace)::NotificationsHandlerImpl::disconnect (this=0x7ffed4005a40) at ./node/interfaces.cpp:496
[#20](/bitcoin-core-multiprocess/20/) 0x0000555555755e47 in node::(anonymous namespace)::NotificationsHandlerImpl::~NotificationsHandlerImpl (this=0x7ffed4005a40) at ./node/interfaces.cpp:491
[#21](/bitcoin-core-multiprocess/21/) node::(anonymous namespace)::NotificationsHandlerImpl::~NotificationsHandlerImpl (this=0x28769) at ./node/interfaces.cpp:491
[#22](/bitcoin-core-multiprocess/22/) 0x0000555555b49cb7 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x7ffed4004d00)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/shared_ptr_base.h:346
[#23](/bitcoin-core-multiprocess/23/) std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=<optimized out>)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/shared_ptr_base.h:1071
[#24](/bitcoin-core-multiprocess/24/) std::__shared_ptr<interfaces::Handler, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=<optimized out>)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/shared_ptr_base.h:1524
[#25](/bitcoin-core-multiprocess/25/) std::__shared_ptr<interfaces::Handler, (__gnu_cxx::_Lock_policy)2>::reset (this=0x7fffe801c3e0)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/shared_ptr_base.h:1642
[#26](/bitcoin-core-multiprocess/26/) mp::ProxyServerBase<ipc::capnp::messages::Handler, interfaces::Handler>::~ProxyServerBase()::{lambda()#1}::operator()() (this=0x7fffe801c3e0)
    at ./ipc/libmultiprocess/include/mp/proxy-io.h:514
[#27](/bitcoin-core-multiprocess/27/) std::__invoke_impl<void, mp::ProxyServerBase<ipc::capnp::messages::Handler, interfaces::Handler>::~ProxyServerBase()::{lambda()#1}&>(std::__invoke_other, mp::ProxyServerBase<ipc::capnp::messages::Handler, interfaces::Handler>::~ProxyServerBase()::{lambda()#1}&) (__f=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61
[#28](/bitcoin-core-multiprocess/28/) std::__invoke_r<void, mp::ProxyServerBase<ipc::capnp::messages::Handler, interfaces::Handler>::~ProxyServerBase()::{lambda()#1}&>(mp::ProxyServerBase<ipc::capnp::messages::Handler, interfaces::Handler>::~ProxyServerBase()::{lambda()#1}&) (__fn=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111
[#29](/bitcoin-core-multiprocess/29/) std::_Function_handler<void (), mp::ProxyServerBase<ipc::capnp::messages::Handler, interfaces::Handler>::~ProxyServerBase()::{lambda()#1}>::_M_invoke(std::_Any_data const&) (
    __functor=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290
[#30](/bitcoin-core-multiprocess/30/) 0x0000555555f41979 in std::function<void ()>::operator()() const (this=0x7ffed2ffca70) at /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591
[#31](/bitcoin-core-multiprocess/31/) mp::Unlock<mp::Lock, std::function<void ()> const&>(mp::Lock&, std::function<void ()> const&) (lock=..., callback=...) at ./ipc/libmultiprocess/include/mp/util.h:209
[#32](/bitcoin-core-multiprocess/32/) 0x0000555555f3f56f in mp::EventLoop::startAsyncThread()::$_0::operator()() const (this=<optimized out>) at ./ipc/libmultiprocess/src/mp/proxy.cpp:298
[#33](/bitcoin-core-multiprocess/33/) std::__invoke_impl<void, mp::EventLoop::startAsyncThread()::$_0>(std::__invoke_other, mp::EventLoop::startAsyncThread()::$_0&&) (__f=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61
[#34](/bitcoin-core-multiprocess/34/) std::__invoke<mp::EventLoop::startAsyncThread()::$_0>(mp::EventLoop::startAsyncThread()::$_0&&) (__fn=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96
[#35](/bitcoin-core-multiprocess/35/) std::thread::_Invoker<std::tuple<mp::EventLoop::startAsyncThread()::$_0> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=<optimized out>)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:292
[#36](/bitcoin-core-multiprocess/36/) std::thread::_Invoker<std::tuple<mp::EventLoop::startAsyncThread()::$_0> >::operator()() (this=<optimized out>)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:299
[#37](/bitcoin-core-multiprocess/37/) std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::EventLoop::startAsyncThread()::$_0> > >::_M_run() (this=<optimized out>)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:244
[#38](/bitcoin-core-multiprocess/38/) 0x00007ffff76ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
[#39](/bitcoin-core-multiprocess/39/) 0x00007ffff729caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
[#40](/bitcoin-core-multiprocess/40/) 0x00007ffff7329c6c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

</details>

Seems that an exception is being thrown from the ~ProxyServerBase() destructor, causing std::terminate to be called. I'll need to look into stack trace more to try to figure out the how this is happening. Hopefully it should not be too hard to write a unit test to reproduce it.

From the log messages above it looks like the rust client is not shutting down cleanly like a libmultiprocess c++ client would.

A libmultiprocess client that registered for notifications would call Handler.destroy to stop receiving them, and wait for it to return before disconnecting. By contrast, it looks like the rust client is calling Handler.destroy and immediately disconnecting, or just disconnecting and letting the Handler be garbage collected.

When this happens and the Handler destructor is called in bitcoin-node, it tries to make an ChainNotifications.destroy IPC call to the client, to free the ChainNotifications callback, but this call fails and throws ipc::Exception because the client has already disconnected by that point. Since the call happens in a destructor which does not specify noexcept(false), this terminates bitcoin-node.

Probably adding noexcept(false) to ~Handler and ~ProxyServerBase would be sufficient to fix this. Alternately the ipc::Exception could be caught in the Handler and ignored or log a message. Alternately, it could make sense to just drop the ChainNotifications.destroy method entirely because having the method adds an unnecessary round-trip when unregistering for notifications, and most of the time it is not useful for the client to be notified that the server will stop sending notifications, because it can determine this by waiting for Handler.destroy to return or by disconnecting. Keeping the ChainNotifications.destroy method would be more useful if the server ever stopped sending notifications on its own without disconnecting from the client.

There's a new update in bitcoin/bitcoin#29409 (comment) about this issue. I was able to reproduce it and matched your analysis.

I put together a fix + regression test. I went with a variant of your option 2 but placed at the libmultiprocess layer (in ~ProxyClientBase rather than Core's Handler) so any interface with a destroy method is covered rather than just ChainNotifications.

Happy to turn this into a PR if you'd like. Patch (compare):

diff --git a/include/mp/proxy-io.h b/include/mp/proxy-io.h
index d7b9f0e..09465c0 100644
--- a/include/mp/proxy-io.h
+++ b/include/mp/proxy-io.h
@@ -538,7 +538,12 @@ ProxyClientBase<Interface, Impl>::ProxyClientBase(typename Interface::Client cli
         // the remote object, waiting for it to be deleted server side. If the
         // capnp interface does not define a destroy method, this will just call
         // an empty stub defined in the ProxyClientBase class and do nothing.
-        Sub::destroy(*this);
+        // Exceptions are caught and logged rather than propagated because
+        // ~ProxyClientBase is noexcept and the peer may be gone by the time
+        // this runs.
+        if (kj::runCatchingExceptions([&]{ Sub::destroy(*this); }) != nullptr) {
+            MP_LOG(*m_context.loop, Log::Warning) << "Remote destroy call failed during cleanup. Continuing.";
+        }
 
         // FIXME: Could just invoke removed addCleanup fn here instead of duplicating code
         m_context.loop->sync([&]() {
diff --git a/test/mp/test/test.cpp b/test/mp/test/test.cpp
index d91edb4..b259790 100644
--- a/test/mp/test/test.cpp
+++ b/test/mp/test/test.cpp
@@ -427,6 +427,32 @@ KJ_TEST("Calling async IPC method, with server disconnect after cleanup")
     }
 }
 
+KJ_TEST("Destroying ProxyClient<> with destroy method after peer disconnect")
+{
+    // Regression test for bitcoin-core/libmultiprocess#219 where
+    // ~ProxyClientBase would call std::terminate if the remote destroy RPC
+    // failed during teardown.
+    //
+    // Save a callback on the server so it holds a ProxyClient<FooCallback>
+    // pointing back to this side, then disconnect. When the server is torn
+    // down, the ProxyClient<FooCallback> destructor issues a destroy RPC over
+    // the now dead connection; without the bugfix the exception escapes the
+    // noexcept destructor and aborts the process.
+
+    TestSetup setup{/*client_owns_connection=*/false};
+    ProxyClient<messages::FooInterface>* foo = setup.client.get();
+    foo->initThreadMap();
+
+    class Callback : public FooCallback
+    {
+    public:
+        int call(int arg) override { return arg; }
+    };
+
+    foo->saveCallback(std::make_shared<Callback>());
+    setup.client_disconnect();
+}
+
 KJ_TEST("Make simultaneous IPC calls on single remote thread")
 {
     TestSetup setup;

I went with a variant of your option 2 but placed at the libmultiprocess layer (in ~ProxyClientBase rather than Core's Handler) so any interface with a destroy method is covered rather than just ChainNotifications.

Thanks for the fix! But I am concerned it could lead to resource leaks and deadlocks potentially if these exceptions are only logged and suppressed. I am curious if we just add noexcept(false) to ~ProxyClientBase and ~Handler if that would fix this bug?

(Rereading https://github.com/capnproto/capnproto/blob/v2/kjdoc/tour.md#exceptions-in-destructors I can also see that there are situations where adding noexcept(false) would not be enough, and using kj::UnwindDetector to suppress and log could make sense, but for this bug it doesn't seem necessary.)

I am curious if we just add noexcept(false) to ~ProxyClientBase and ~Handler if that would fix this bug?

I tried the noexcept(false) approach against the regression test I added. Two findings:

1. The test still fails.

[ TEST ] test.cpp:430: Destroying ProxyClient<> with destroy method after peer disconnect
*** Fatal uncaught std::exception: IPC client method called after disconnect.
stack: 10449bae7 1045d8173 10447fd1f 10447fcc3 10449c7a3 1044745d3 10447f8f3 10447f8af 10447f887 10447f863 10447f833 10447f52f 10447fd1f 10447fcc3 10447fc93 10447e453 1046cce27 1046ccdd7 1046ccdaf 1046ccd8b 1046ccd67 1046ccd43 1046cbebf 104499b4b 1044769ab 1046dde5b 1046ddd67 1046ddc77

2. On Core it would require changes not only to ~Handler but every interface that will input into mpgen, otherwise won't compile. And even applying those changes the crash just moved, running darosior's PoC against that build still SIGABRTs:

2026-04-22T13:56:14Z [ipc] {bitcoin-node-82862/163544799} IPC client send ChainNotifications.destroy$Params
2026-04-22T13:56:14Z ipc: {bitcoin-node-82862/163544799} IPC client exception kj::Exception: (remote):0: unimplemented: remote exception: Destroy notification
stack: 1025dd450 10046d058 10046d854
2026-04-22T13:56:14Z [error] ipc: kj::Exception: (remote):0: unimplemented: remote exception: Destroy notification
stack: 1025dd450 10046d058 10046d854
libc++abi: terminating due to uncaught exception of type kj::ExceptionImpl: kj/async.c++:170: failed: expected loop != nullptr [0 != nullptr]; No event loop is running on this thread.
stack: 102743d6b 102745c4b 10274921b 1025a5ecf 1025ef40b 1025eefaf 1025de49f 1025d5577 1025d5617 1005dc5bf 100687d03 1005db6d7 100156a3f 10015660b 1001564db 1003ac37b 1006bb60b 100979ea3 100979ccf 1984d5c07 1984d0ba7
Process 82862 stopped

Relevant frames:

    frame [#8](/bitcoin-core-multiprocess/8/): 0x00000001025a233c libcapnp-rpc.1.3.0.dylib`__clang_call_terminate + 16
    frame [#9](/bitcoin-core-multiprocess/9/): 0x00000001025a5f4c libcapnp-rpc.1.3.0.dylib`kj::Own<kj::_::ChainPromiseNode, kj::_::PromiseDisposer> 
    frame [#11](/bitcoin-core-multiprocess/11/): 0x00000001025eefb0 libcapnp-rpc.1.3.0.dylib`capnp::TwoPartyVatNetwork::OutgoingMessageImpl::send() + 352
    frame [#13](/bitcoin-core-multiprocess/13/): 0x00000001025d5578 libcapnp-rpc.1.3.0.dylib`capnp::_::(anonymous namespace)::RpcConnectionState::ImportClient::~ImportClient() + 44
    frame [#22](/bitcoin-core-multiprocess/22/): 0x00000001005dc598 bitcoin-node`mp::ProxyClientBase<ipc::capnp::messages::ChainNotifications, interfaces::Chain::Notifications>::~ProxyClientBase(this=0x0000000a5ac540a0) at proxy-io.h:573:1 [opt]
    frame [#33](/bitcoin-core-multiprocess/33/): 0x0000000100156a14 bitcoin-node`node::(anonymous namespace)::NotificationsProxy::~NotificationsProxy(this=<unavailable>) at interfaces.cpp:461:43 [opt] [inlined]
    frame [#40](/bitcoin-core-multiprocess/40/): 0x00000001001565e0 bitcoin-node`node::(anonymous namespace)::NotificationsHandlerImpl::disconnect(this=0x0000000a5ac20c80) at interfaces.cpp:502:21 [opt]
    frame [#51](/bitcoin-core-multiprocess/51/): 0x00000001006bb5dc bitcoin-node`mp::ProxyServerBase<ipc::capnp::messages::Handler, interfaces::Handler>::~ProxyServerBase()::'lambda'()::operator()(this=<unavailable>) at proxy-io.h:618:18 [opt] [inlined]
    frame [#60](/bitcoin-core-multiprocess/60/): 0x0000000100979cd0 bitcoin-node`mp::EventLoop::startAsyncThread()::$_0::operator()(this=0x0000000a5afc0008) const at proxy.cpp:303:21 [opt]

I'm still investigating and see if I can find a proper fix on libmultiprocess side

Tested dropping ChainNotifications.destroy too, it worked. Actually, this is the direction I like the most based on your own rationale:

Alternately, it could make sense to just drop the ChainNotifications.destroy method entirely because having the method adds an unnecessary round-trip when unregistering for notifications, and most of the time it is not useful for the client to be notified that the server will stop sending notifications, because it can determine this by waiting for Handler.destroy to return or by disconnecting. Keeping the ChainNotifications.destroy method would be more useful if the server ever stopped sending notifications on its own without disconnecting from the client.

I'm still investigating and see if I can find a proper fix on libmultiprocess side

I think I see what is happening here. If trying to call the remote destroy() method throws, then the ProxyClientBase::m_client reference which is supposed to be freed a few lines below will not be freed, leading an exception ("No event loop is running on this thread"`) when it is freed from the wrong thread later.

To fix this you would need to rearrange control a little so cleanup code below Sub::destroy always runs even if destroy throws:

--- a/include/mp/proxy-io.h
+++ b/include/mp/proxy-io.h
@@ -534,14 +534,8 @@ ProxyClientBase<Interface, Impl>::ProxyClientBase(typename Interface::Client cli
     // m_context.connection to null so nothing happens here.
     m_context.cleanup_fns.emplace_front([this, destroy_connection, disconnect_cb]{
     {
-        // If the capnp interface defines a destroy method, call it to destroy
-        // the remote object, waiting for it to be deleted server side. If the
-        // capnp interface does not define a destroy method, this will just call
-        // an empty stub defined in the ProxyClientBase class and do nothing.
-        Sub::destroy(*this);
-
         // FIXME: Could just invoke removed addCleanup fn here instead of duplicating code
-        m_context.loop->sync([&]() {
+        KJ_DEFER(m_context.loop->sync([&]() {
             // Remove disconnect callback on cleanup so it doesn't run and try
             // to access this object after it's destroyed. This call needs to
             // run inside loop->sync() on the event loop thread because
@@ -558,7 +552,13 @@ ProxyClientBase<Interface, Impl>::ProxyClientBase(typename Interface::Client cli
                 delete m_context.connection;
                 m_context.connection = nullptr;
             }
-        });
+        }));
+
+        // If the capnp interface defines a destroy method, call it to destroy
+        // the remote object, waiting for it to be deleted server side. If the
+        // capnp interface does not define a destroy method, this will just call
+        // an empty stub defined in the ProxyClientBase class and do nothing.
+        Sub::destroy(*this);
     }
     });
     Sub::construct(*this);

2. On Core it would require changes not only to ~Handler but every interface that will input into mpgen, otherwise won't compile.

I didn't realize this but it makes sense. I feel like it could be a good change though, since interface classes are meant to be opaque and clients should not make assumptions about how they are implemented. So if destructor can throw, it makes sense to document that they can throw.

I guess I am having trouble trying to figure out if destructors should throw or not. If the node tries to destroy a remote object but can't because the remote side has disconnected, maybe it actually fine to suppress the exception and assume the node doesn't need to do any error handling. And if error handling is needed in the future, we already provide an Ipc::addCleanup method that lets IPC clients run a callback when any particular client object is destroyed. That callback could have a parameter with exception information.

Another observation that the startAsyncThread() and other code running cleanup functions doesn't seem currently set up to handle exceptions at all so it seems likely there will be more problems even with above problems fixed.

I guess I'm a little unsure what to recommend right now. i would be curious what a full working noexcept(false) solution would look like. But maybe there are not real downsides to suppressing and logging exceptions from client destructors and your original approach is ok, especially if clients can have another way to handle these errors. Dropping ChainNotifications.destroy could also be a good fix for the problem specifically reported here though the more general problem would still exist

I think I see what is happening here. If trying to call the remote destroy() method throws, then the ProxyClientBase::m_client reference which is supposed to be freed a few lines below will not be freed, leading an exception ("No event loop is running on this thread"`) when it is freed from the wrong thread later.

To fix this you would need to rearrange control a little so cleanup code below Sub::destroy always runs even if destroy throws:

--- a/include/mp/proxy-io.h
+++ b/include/mp/proxy-io.h
@@ -534,14 +534,8 @@ ProxyClientBase<Interface, Impl>::ProxyClientBase(typename Interface::Client cli
     // m_context.connection to null so nothing happens here.
     m_context.cleanup_fns.emplace_front([this, destroy_connection, disconnect_cb]{
     {
-        // If the capnp interface defines a destroy method, call it to destroy
-        // the remote object, waiting for it to be deleted server side. If the
-        // capnp interface does not define a destroy method, this will just call
-        // an empty stub defined in the ProxyClientBase class and do nothing.
-        Sub::destroy(*this);
-
         // FIXME: Could just invoke removed addCleanup fn here instead of duplicating code
-        m_context.loop->sync([&]() {
+        KJ_DEFER(m_context.loop->sync([&]() {
             // Remove disconnect callback on cleanup so it doesn't run and try
             // to access this object after it's destroyed. This call needs to
             // run inside loop->sync() on the event loop thread because
@@ -558,7 +552,13 @@ ProxyClientBase<Interface, Impl>::ProxyClientBase(typename Interface::Client cli
                 delete m_context.connection;
                 m_context.connection = nullptr;
             }
-        });
+        }));
+
+        // If the capnp interface defines a destroy method, call it to destroy
+        // the remote object, waiting for it to be deleted server side. If the
+        // capnp interface does not define a destroy method, this will just call
+        // an empty stub defined in the ProxyClientBase class and do nothing.
+        Sub::destroy(*this);
     }
     });
     Sub::construct(*this);

Yeah that matches what I found. I actually tried that KJ_DEFER arrangement earlier, it gets rid of the "No event loop" crash, but the original exception still escapes out to the async thread root and terminates the process there, mainly because of this:

Another observation that the startAsyncThread() and other code running cleanup functions doesn't seem currently set up to handle exceptions at all so it seems likely there will be more problems even with above problems fixed.

Opened a PR (#273) with the original catch-and-log at Sub::destroy since it's the simplest point that covers every destructor path in one spot.