[`v30.0`] `createNewBlock` never returns #33647

while connected to testnet4 and letting the current implementation of SRI client connected for a while, bitcoin-node stops responding to IPC calls

steps to reproduce:

• build bitcoin-node from v30.0 tag
• clone https://github.com/plebhash/sv2-bitcoin-core
• check out 2025-10-17-hanging-ipc branch
• launch bitcoin-node with -testnet4 -ipc-bind=unix -debug=ipc
• launch sv2-bitcoin-core with RUST_LOG=debug cargo run --example logger "/path/to/node.sock"
• let it run for a while (for me it took ~25 minutes)

eventually, the logs on the rust client start to look like this:

2025-10-17T15:29:29.614817Z  INFO logger: Sent new CoinbaseOutputConstraints
2025-10-17T15:29:29.614853Z DEBUG bitcoin_core_sv2: Received: CoinbaseOutputConstraints: CoinbaseOutputConstraints(coinbase_output_max_additional_size: 2, coinbase_output_max_additional_sigops: 2)
2025-10-17T15:29:29.614934Z DEBUG bitcoin_core_sv2: monitor_incoming_messages() processing message
2025-10-17T15:29:29.614973Z DEBUG bitcoin_core_sv2: Received CoinbaseOutputConstraints - max_additional_size: 2, max_additional_sigops: 2
2025-10-17T15:29:29.615022Z DEBUG bitcoin_core_sv2: handle_coinbase_output_constraints() called
2025-10-17T15:29:29.615058Z DEBUG bitcoin_core_sv2: Cancelling template_ipc_client_cancellation_token
2025-10-17T15:29:29.615099Z DEBUG bitcoin_core_sv2: Creating new template IPC client with new constraints
2025-10-17T15:29:29.615139Z DEBUG bitcoin_core_sv2: new_template_ipc_client() called - max_size: 2, max_sigops: 2
2025-10-17T15:29:29.615219Z DEBUG bitcoin_core_sv2: Setting block_reserved_weight: 2000
2025-10-17T15:29:29.615255Z DEBUG bitcoin_core_sv2: Sending createNewBlock request to Bitcoin Core
2025-10-17T15:29:39.616976Z  INFO logger: Sent new CoinbaseOutputConstraints
2025-10-17T15:29:49.619034Z  INFO logger: Sent new CoinbaseOutputConstraints
2025-10-17T15:29:59.621747Z  INFO logger: Sent new CoinbaseOutputConstraints
2025-10-17T15:30:09.624253Z  INFO logger: Sent new CoinbaseOutputConstraints
2025-10-17T15:30:19.625848Z  INFO logger: Sent new CoinbaseOutputConstraints
2025-10-17T15:30:29.627359Z  INFO logger: Sent new CoinbaseOutputConstraints

this is a clear indication that createNewBlock IPC call got stuck and never returned

the rust client becomes unkillable via ctrl+c, since there's hanging futures inside... so the process must be killed with other methods

if we restart the rust client, the normal bootstrapping process doesn't go through, which indicates that bitcoin-node is no longer able to respond to any IPC calls whatsoever

additionally, bitcoin-node also becomes unkillable via ctrl+c

here's logs of bitcoin-node around the timestamp where createNewBlock IPC call got stuck:

2025-10-17T15:29:24Z [ipc] {bitcoin-node-58573/b-capnp-loop-26537354} IPC server send response [#347](/bitcoin-bitcoin/347/) BlockTemplate.waitNext$Results (result = <external capability>)
2025-10-17T15:29:24Z [ipc] {bitcoin-node-58573/b-capnp-loop-26537354} IPC server send response [#349](/bitcoin-bitcoin/349/) BlockTemplate.waitNext$Results (result = <external capability>)
2025-10-17T15:29:24Z [ipc] {bitcoin-node-58573/b-capnp-loop-26537354} IPC server recv request  [#350](/bitcoin-bitcoin/350/) BlockTemplate.getBlock$Params (context = (thread = <external capability>))
2025-10-17T15:29:24Z [ipc] {bitcoin-node-58573/b-capnp-loop-26537354} IPC server post request  [#350](/bitcoin-bitcoin/350/) {bitcoin-node-58573/26538479 (from )}
2025-10-17T15:29:24Z [ipc] {bitcoin-node-58573/b-capnp-loop-26537354} IPC server recv request  [#351](/bitcoin-bitcoin/351/) BlockTemplate.getBlock$Params (context = (thread = <external capability>))
2025-10-17T15:29:24Z [ipc] {bitcoin-node-58573/b-capnp-loop-26537354} IPC server post request  [#351](/bitcoin-bitcoin/351/) {bitcoin-node-58573/26538479 (from )}
2025-10-17T15:42:11Z Saw new header hash=000000001fe51f6e5b13b64e87e1c66f475adc2b4fa97b0b39c7b9a20fe0ea0e height=106954 peer=5
2025-10-17T15:42:11Z UpdateTip: new best=000000001fe51f6e5b13b64e87e1c66f475adc2b4fa97b0b39c7b9a20fe0ea0e height=106954 version=0x20000000 log2_work=74.802497 tx=12113955 date='2025-10-17T17:42:11Z' progress=1.000000 cache=0.3MiB(337txo)
2025-10-17T15:42:11Z Saw new header hash=00000000e206db80ad182ffdabc05bf9f6a1e13bd62b5ec4a51750c42e456648 height=106954 peer=8
2025-10-17T15:45:11Z New block-relay-only v2 peer connected: version: 70016, blocks=106954, peer=15
2025-10-17T15:58:34Z New block-relay-only v2 peer connected: version: 70016, blocks=106954, peer=16
2025-10-17T15:59:44Z New block-relay-only v2 peer connected: version: 70016, blocks=106954, peer=17
2025-10-17T16:02:12Z Saw new cmpctblock header hash=000000009a3bbd6b55b974200199e2cb7272deb8c892d2ee7b4215ec9b1e0876 height=106955 peer=4
2025-10-17T16:02:12Z UpdateTip: new best=000000009a3bbd6b55b974200199e2cb7272deb8c892d2ee7b4215ec9b1e0876 height=106955 version=0x20000000 log2_work=74.802497 tx=12113988 date='2025-10-17T18:02:12Z' progress=1.000000 cache=0.3MiB(342txo)
2025-10-17T16:04:58Z Flushed fee estimates to fee_estimates.dat.
2025-10-17T16:22:13Z Saw new header hash=0000000073d29bfc7123c09a0dda5e64bd3e58383d85621c64625fad817215fa height=106956 peer=5
2025-10-17T16:22:13Z Saw new header hash=0000000001f79e1b92f6f254cc5a5ff72bb8ce76731c53d50c28176663949972 height=106956 peer=1
2025-10-17T16:22:13Z UpdateTip: new best=0000000001f79e1b92f6f254cc5a5ff72bb8ce76731c53d50c28176663949972 height=106956 version=0x20000000 log2_work=74.802497 tx=12114006 date='2025-10-17T18:22:13Z' progress=1.000000 cache=0.3MiB(391txo)
2025-10-17T16:22:13Z Saw new header hash=00000000a820139f0791d0a2994bec73a29cb8563c34f4a68b96d7fe31cac7ef height=106956 peer=10
2025-10-17T16:35:29Z New block-relay-only v2 peer connected: version: 70016, blocks=106956, peer=19
2025-10-17T16:36:29Z New block-relay-only v2 peer connected: version: 70016, blocks=106956, peer=20
2025-10-17T16:39:45Z Saw new cmpctblock header hash=00000000000000015ef791c938c1399989dcc68ead8057c508e03e585d2110c0 height=106957 peer=4
2025-10-17T16:39:45Z UpdateTip: new best=00000000000000015ef791c938c1399989dcc68ead8057c508e03e585d2110c0 height=106957 version=0x200f2000 log2_work=74.803065 tx=12114029 date='2025-10-17T16:42:09Z' progress=1.000000 cache=0.3MiB(429txo)

cc @ryanofsky

cc @Sjors

And it was not syncing new blocks during createNewBlock()? That's the only thing it's expect to wait for, but only once at startup:

   /**
     * Construct a new block template.
     *
     * During node initialization, this will wait until the tip is connected.
     *
     * [@param](/bitcoin-bitcoin/contributor/param/)[in] options options for creating the block
     * [@retval](/bitcoin-bitcoin/contributor/retval/) BlockTemplate a block template.
     * [@retval](/bitcoin-bitcoin/contributor/retval/) std::nullptr if the node is shut down.
     */
    virtual std::unique_ptr<BlockTemplate> createNewBlock(const node::BlockCreateOptions& options = {}) = 0;

cc @ryanofsky

Thanks for the clear steps to reproduce. I was able to see the rust client make the bitcoin node IPC hang very quickly (after around a minute) following them.

It seems like rust client is able to trigger a deadlock bug that was fixed in https://github.com/bitcoin-core/libmultiprocess/pull/201, and was backported in #33519 in the 30.x branch that was made after the v30.0 tag.

If you build with the 30.x branch instead of the v30.0 tag, I think it should fix this issue.

I debugged the hang attaching with gdb (sudo gdb build/bin/bitcoin-node <bitcoin-node pid>) and running thread apply all bt.

The stack trace is below and shows the event loop thread (Thread 13) stuck trying to post an incoming request to worker thread (Thread 5), waiting to acquire the worker thread's Waiter::m_mutex mutex. It also shows the worker thread holding on to Waiter::m_mutex trying to run code on the event loop to destroy a ProxyClient<Thread> object.

So it's just a deadlock with two threads stuck waiting for each other, and it's the same deadlock that was fixed in https://github.com/bitcoin-core/libmultiprocess/pull/201.

<details><summary>stack trace</summary> <p>

Thread 13 (Thread 0x7f67fe9fe6c0 (LWP 3342614) "b-capnp-loop"):
[#0](/bitcoin-bitcoin/0/)  0x00007f67ff69762f in __lll_lock_wait () from /nix/store/776irwlgfb65a782cxmyk61pck460fs9-glibc-2.40-66/lib/libc.so.6
[#1](/bitcoin-bitcoin/1/)  0x00007f67ff69e1a1 in pthread_mutex_lock@@GLIBC_2.2.5 () from /nix/store/776irwlgfb65a782cxmyk61pck460fs9-glibc-2.40-66/lib/libc.so.6
[#2](/bitcoin-bitcoin/2/)  0x000055977887f71c in __gthread_mutex_lock (__mutex=0x7f6774000e60) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/x86_64-unknown-linux-gnu/bits/gthr-default.h:762
[#3](/bitcoin-bitcoin/3/)  0x00005597788a7306 in std::mutex::lock (this=0x7f6774000e60) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/std_mutex.h:113
[#4](/bitcoin-bitcoin/4/)  0x00005597788c6b99 in std::unique_lock<std::mutex>::lock (this=0x7f67fe9faeb8) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/unique_lock.h:147
[#5](/bitcoin-bitcoin/5/)  0x000055977907d519 in std::unique_lock<std::mutex>::unique_lock (this=0x7f67fe9faeb8, __m=...) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/unique_lock.h:73
[#6](/bitcoin-bitcoin/6/)  0x0000559779207982 in post<(lambda at ./ipc/libmultiprocess/include/mp/type-context.h:67:19)> (this=0x7f6774000e60, fn=...) at ./ipc/libmultiprocess/include/mp/proxy-io.h:288

   285      template <typename Fn>
   286      void post(Fn&& fn)
   287      {
>  288          const std::unique_lock<std::mutex> lock(m_mutex);
   289          assert(!m_fn);
   290          m_fn = std::forward<Fn>(fn);
   291          m_cv.notify_all();
   292      }

[#7](/bitcoin-bitcoin/7/)  0x0000559779207790 in operator() (this=0x7f67f0017950, perhaps=...) at ./ipc/libmultiprocess/include/mp/type-context.h:145

   136      return server.m_context.connection->m_threads.getLocalServer(thread_client)
   137          .then([&server, invoke = kj::mv(invoke), req](const kj::Maybe<Thread::Server&>& perhaps) mutable {
   138              // Assuming the thread object is found, pass it a pointer to the
   139              // `invoke` lambda above which will invoke the function on that
   140              // thread.
   141              KJ_IF_MAYBE (thread_server, perhaps) {
   142                  const auto& thread = static_cast<ProxyServer<Thread>&>(*thread_server);
   143                  server.m_context.loop->log()
   144                      << "IPC server post request  #" << req << " {" << thread.m_thread_context.thread_name << "}";
>  145                  thread.m_thread_context.waiter->post(std::move(invoke));
   146              } else {
   147                  server.m_context.loop->log()
   148                      << "IPC server error request #" << req << ", missing thread to execute request";
   149                  throw std::runtime_error("invalid thread handle");
   150              }
   151          })
   152          // Wait for the invocation to finish before returning to the caller.
   153          .then([invoke_wait = kj::mv(future.promise)]() mutable { return kj::mv(invoke_wait); });


[#8](/bitcoin-bitcoin/8/)  0x000055977920c29e in apply<(lambda at ./ipc/libmultiprocess/include/mp/type-context.h:137:15)> (func=..., in=...) at /nix/store/6inpywqa7f4njlq1i0bmkznxn5y5x69g-capnproto-1.2.0/include/kj/async-prelude.h:179

>  137          .then([&server, invoke = kj::mv(invoke), req](const kj::Maybe<Thread::Server&>& perhaps) mutable {

[#9](/bitcoin-bitcoin/9/)  0x000055977920be8e in getImpl (this=0x7f67f0017930, output=...) at /nix/store/6inpywqa7f4njlq1i0bmkznxn5y5x69g-capnproto-1.2.0/include/kj/async-inl.h:739
[#10](/bitcoin-bitcoin/10/) 0x00007f67fff18b4d in kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue&) () from /nix/store/6inpywqa7f4njlq1i0bmkznxn5y5x69g-capnproto-1.2.0/lib/libkj-async.so.1.2.0
[#11](/bitcoin-bitcoin/11/) 0x00007f67fff18f81 in kj::_::TransformPromiseNodeBase::getDepResult(kj::_::ExceptionOrValue&) () from /nix/store/6inpywqa7f4njlq1i0bmkznxn5y5x69g-capnproto-1.2.0/lib/libkj-async.so.1.2.0
[#12](/bitcoin-bitcoin/12/) 0x000055977920ce0e in getImpl (this=0x7f67f0017900, output=...) at /nix/store/6inpywqa7f4njlq1i0bmkznxn5y5x69g-capnproto-1.2.0/include/kj/async-inl.h:733
[#13](/bitcoin-bitcoin/13/) 0x00007f67fff18b4d in kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue&) () from /nix/store/6inpywqa7f4njlq1i0bmkznxn5y5x69g-capnproto-1.2.0/lib/libkj-async.so.1.2.0
[#14](/bitcoin-bitcoin/14/) 0x00007f67fff1a249 in kj::_::ChainPromiseNode::fire() () from /nix/store/6inpywqa7f4njlq1i0bmkznxn5y5x69g-capnproto-1.2.0/lib/libkj-async.so.1.2.0
[#15](/bitcoin-bitcoin/15/) 0x00007f67fff1a9d2 in non-virtual thunk to kj::_::ChainPromiseNode::fire() () from /nix/store/6inpywqa7f4njlq1i0bmkznxn5y5x69g-capnproto-1.2.0/lib/libkj-async.so.1.2.0
[#16](/bitcoin-bitcoin/16/) 0x00007f67fff1de8c in kj::_::waitImpl(kj::Own<kj::_::PromiseNode, kj::_::PromiseDisposer>&&, kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation)::$_2::operator()() const () from /nix/store/6inpywqa7f4njlq1i0bmkznxn5y5x69g-capnproto-1.2.0/lib/libkj-async.so.1.2.0
[#17](/bitcoin-bitcoin/17/) 0x00007f67fff16dd8 in kj::_::waitImpl(kj::Own<kj::_::PromiseNode, kj::_::PromiseDisposer>&&, kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation) () from /nix/store/6inpywqa7f4njlq1i0bmkznxn5y5x69g-capnproto-1.2.0/lib/libkj-async.so.1.2.0
[#18](/bitcoin-bitcoin/18/) 0x000055977963860c in kj::Promise<unsigned long>::wait (this=0x7f67fe9fd620, waitScope=..., location=...) at /nix/store/6inpywqa7f4njlq1i0bmkznxn5y5x69g-capnproto-1.2.0/include/kj/async-inl.h:1357
[#19](/bitcoin-bitcoin/19/) 0x0000559779632555 in mp::EventLoop::loop (this=0x55979fb711b8) at ./ipc/libmultiprocess/src/mp/proxy.cpp:231

   230      for (;;) {
>  231          const size_t read_bytes = wait_stream->read(&buffer, 0, 1).wait(m_io_context.waitScope);
   232          if (read_bytes != 1) throw std::logic_error("EventLoop wait_stream closed unexpectedly");

[#20](/bitcoin-bitcoin/20/) 0x000055977906e799 in ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::{lambda()#1}::operator()() const (this=0x55979fb7cf98) at ./ipc/capnp/protocol.cpp:96

    91          m_loop_thread = std::thread([&] {
    92              util::ThreadRename("capnp-loop");
    93              m_loop.emplace(exe_name, &IpcLogFn, &m_context);
    94              m_loop_ref.emplace(*m_loop);
    95              promise.set_value();
>   96              m_loop->loop();
    97              m_loop.reset();
    98          });

[#21](/bitcoin-bitcoin/21/) 0x000055977906e672 in std::__invoke_impl<void, ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::{lambda()#1}>(std::__invoke_other, ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::{lambda()#1}&&) (__f=...) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:61
[#22](/bitcoin-bitcoin/22/) 0x000055977906e5d2 in std::__invoke<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::{lambda()#1}>(ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::{lambda()#1}&&) (__fn=...) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:96
[#23](/bitcoin-bitcoin/23/) 0x000055977906e58a in std::thread::_Invoker<std::tuple<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x55979fb7cf98) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/std_thread.h:301
[#24](/bitcoin-bitcoin/24/) 0x000055977906e532 in std::thread::_Invoker<std::tuple<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::{lambda()#1}> >::operator()() (this=0x55979fb7cf98) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/std_thread.h:308
[#25](/bitcoin-bitcoin/25/) 0x000055977906e3da in std::thread::_State_impl<std::thread::_Invoker<std::tuple<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::{lambda()#1}> > >::_M_run() (this=0x55979fb7cf90) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/std_thread.h:253
[#26](/bitcoin-bitcoin/26/) 0x00007f67ffaed064 in execute_native_thread_routine () from /nix/store/41ym1jm1b7j3rhglk82gwg9jml26z1km-gcc-14.3.0-lib/lib/libstdc++.so.6
[#27](/bitcoin-bitcoin/27/) 0x00007f67ff69a97a in start_thread () from /nix/store/776irwlgfb65a782cxmyk61pck460fs9-glibc-2.40-66/lib/libc.so.6
[#28](/bitcoin-bitcoin/28/) 0x00007f67ff722d2c in __clone3 () from /nix/store/776irwlgfb65a782cxmyk61pck460fs9-glibc-2.40-66/lib/libc.so.6

Thread 5 (Thread 0x7f677a7fc6c0 (LWP 3343266) "b-capnp-loop"):
[#0](/bitcoin-bitcoin/0/)  0x00007f67ff697389 in __futex_abstimed_wait_common () from /nix/store/776irwlgfb65a782cxmyk61pck460fs9-glibc-2.40-66/lib/libc.so.6
[#1](/bitcoin-bitcoin/1/)  0x00007f67ff699e1e in pthread_cond_wait@@GLIBC_2.3.2 () from /nix/store/776irwlgfb65a782cxmyk61pck460fs9-glibc-2.40-66/lib/libc.so.6
[#2](/bitcoin-bitcoin/2/)  0x0000559779632ecb in std::condition_variable::wait<mp::EventLoop::post(kj::Function<void ()>)::$_2>(std::unique_lock<std::mutex>&, mp::EventLoop::post(kj::Function<void ()>)::$_2) (this=0x55979fb71230, __lock=..., __p=...) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/condition_variable:105
[#3](/bitcoin-bitcoin/3/)  0x0000559779632cb7 in mp::EventLoop::post (this=0x55979fb711b8, fn=...) at ./ipc/libmultiprocess/src/mp/proxy.cpp:273

   258  void EventLoop::post(kj::Function<void()> fn)
   259  {
   260      if (std::this_thread::get_id() == m_thread_id) {
   261          fn();
   262          return;
   263      }
   264      Lock lock(m_mutex);
   265      EventLoopRef ref(*this, &lock);
   266      m_cv.wait(lock.m_lock, [this]() MP_REQUIRES(m_mutex) { return m_post_fn == nullptr; });
   267      m_post_fn = &fn;
   268      int post_fd{m_post_fd};
   269      Unlock(lock, [&] {
   270          char buffer = 0;
   271          KJ_SYSCALL(write(post_fd, &buffer, 1));
   272      });
>  273      m_cv.wait(lock.m_lock, [this, &fn]() MP_REQUIRES(m_mutex) { return m_post_fn != &fn; });
   274  }

[#4](/bitcoin-bitcoin/4/)  0x0000559779641ea7 in mp::EventLoop::sync<mp::ProxyClientBase<mp::Thread, capnp::Void>::ProxyClientBase(mp::Thread::Client, mp::Connection*, bool)::{lambda()#2}::operator()() const::{lambda()#1}>(mp::ProxyClientBase<mp::Thread, capnp::Void>::ProxyClientBase(mp::Thread::Client, mp::Connection*, bool)::{lambda()#2}::operator()() const::{lambda()#1}&&) (this=0x55979fb711b8, callable=...) at ./ipc/libmultiprocess/include/mp/proxy-io.h:192

   186      //! Wrapper around EventLoop::post that takes advantage of the
   187      //! fact that callable will not go out of scope to avoid requirement that it
   188      //! be copyable.
   189      template <typename Callable>
   190      void sync(Callable&& callable)
   191      {
>  192          post(std::forward<Callable>(callable));
   193      }

[#5](/bitcoin-bitcoin/5/)  0x0000559779641df9 in mp::ProxyClientBase<mp::Thread, capnp::Void>::ProxyClientBase(mp::Thread::Client, mp::Connection*, bool)::{lambda()#2}::operator()() const (this=0x7f675c004110) at ./ipc/libmultiprocess/include/mp/proxy-io.h:444

   435      m_context.cleanup_fns.emplace_front([this, destroy_connection, disconnect_cb]{
   436      {
   437          // If the capnp interface defines a destroy method, call it to destroy
   438          // the remote object, waiting for it to be deleted server side. If the
   439          // capnp interface does not define a destroy method, this will just call
   440          // an empty stub defined in the ProxyClientBase class and do nothing.
   441          Sub::destroy(*this);
   442
   443          // FIXME: Could just invoke removed addCleanup fn here instead of duplicating code
>  444          m_context.loop->sync([&]() {

[#6](/bitcoin-bitcoin/6/)  0x0000559779641d66 in std::__invoke_impl<void, mp::ProxyClientBase<mp::Thread, capnp::Void>::ProxyClientBase(mp::Thread::Client, mp::Connection*, bool)::{lambda()#2}&>(std::__invoke_other, mp::ProxyClientBase<mp::Thread, capnp::Void>::ProxyClientBase(mp::Thread::Client, mp::Connection*, bool)::{lambda()#2}&) (__f=...) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:61
[#7](/bitcoin-bitcoin/7/)  0x0000559779641cb6 in std::__invoke_r<void, mp::ProxyClientBase<mp::Thread, capnp::Void>::ProxyClientBase(mp::Thread::Client, mp::Connection*, bool)::{lambda()#2}&>(mp::ProxyClientBase<mp::Thread, capnp::Void>::ProxyClientBase(mp::Thread::Client, mp::Connection*, bool)::{lambda()#2}&) (__fn=...) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:111
[#8](/bitcoin-bitcoin/8/)  0x0000559779641ade in std::_Function_handler<void (), mp::ProxyClientBase<mp::Thread, capnp::Void>::ProxyClientBase(mp::Thread::Client, mp::Connection*, bool)::{lambda()#2}>::_M_invoke(std::_Any_data const&) (__functor=...) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/std_function.h:290
[#9](/bitcoin-bitcoin/9/)  0x0000559778b04a31 in std::function<void()>::operator() (this=0x7f677a7faec8) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/std_function.h:591
[#10](/bitcoin-bitcoin/10/) 0x000055977907f2e9 in mp::CleanupRun (fns=empty std::__cxx11::list) at ./ipc/libmultiprocess/include/mp/proxy.h:43

    39  inline void CleanupRun(CleanupList& fns) {
    40      while (!fns.empty()) {
    41          auto fn = std::move(fns.front());
    42          fns.pop_front();
>   43          fn();
    44      }
    45  }

[#11](/bitcoin-bitcoin/11/) 0x0000559779638e3e in mp::ProxyClientBase<mp::Thread, capnp::Void>::~ProxyClientBase (this=0x7f6774000fb8) at ./ipc/libmultiprocess/include/mp/proxy-io.h:470

   467  template <typename Interface, typename Impl>
   468  ProxyClientBase<Interface, Impl>::~ProxyClientBase() noexcept
   469  {
>  470      CleanupRun(m_context.cleanup_fns);
   471  }

[#12](/bitcoin-bitcoin/12/) 0x00005597796335ba in mp::ProxyClient<mp::Thread>::~ProxyClient (this=0x7f6774000fb8) at ./ipc/libmultiprocess/src/mp/proxy.cpp:341

   333  ProxyClient<Thread>::~ProxyClient()
   334  {
   335      // If thread is being destroyed before connection is destroyed, remove the
   336      // cleanup callback that was registered to handle the connection being
   337      // destroyed before the thread being destroyed.
   338      if (m_disconnect_cb) {
   339          m_context.connection->removeSyncCleanup(*m_disconnect_cb);
   340      }
>  341  }

[#13](/bitcoin-bitcoin/13/) 0x00005597790dfd9a in std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >::~pair (this=0x7f6774000fb0) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/stl_iterator.h:3013
[#14](/bitcoin-bitcoin/14/) 0x00005597790dfd46 in std::destroy_at<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > (__location=0x7f6774000fb0) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/stl_construct.h:88
[#15](/bitcoin-bitcoin/15/) 0x00005597790dfbe6 in std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > > >::destroy<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > (__a=..., __p=0x7f6774000fb0) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/alloc_traits.h:599
[#16](/bitcoin-bitcoin/16/) std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::_M_destroy_node (this=0x7f677a7fc5f8, __p=0x7f6774000f90) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/stl_tree.h:621
[#17](/bitcoin-bitcoin/17/) 0x00005597790dfb52 in std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::_M_drop_node (this=0x7f677a7fc5f8, __p=0x7f6774000f90) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/stl_tree.h:629
[#18](/bitcoin-bitcoin/18/) 0x00005597790dfaef in std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::_M_erase (this=0x7f677a7fc5f8, __x=0x7f6774000f90) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/stl_tree.h:1934
[#19](/bitcoin-bitcoin/19/) 0x00005597790df956 in std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::clear (this=0x7f677a7fc5f8) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/stl_tree.h:1251
[#20](/bitcoin-bitcoin/20/) 0x00005597790df13a in std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::_M_erase_aux (this=0x7f677a7fc5f8, __first={...}, __last={...}) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/stl_tree.h:2505
[#21](/bitcoin-bitcoin/21/) 0x00005597790dee4a in std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::erase (this=0x7f677a7fc5f8, __x=@0x7f675c006e98: 0x7f67f0001dc0) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/stl_tree.h:2519
[#22](/bitcoin-bitcoin/22/) 0x00005597790ded9e in std::map<mp::Connection*, mp::ProxyClient<mp::Thread>, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::erase (this=0x7f677a7fc5f8, __x=@0x7f675c006e98: 0x7f67f0001dc0) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/stl_map.h:1118
[#23](/bitcoin-bitcoin/23/) 0x0000559779209e1f in operator() (this=0x7f677a7fb248) at ./ipc/libmultiprocess/include/mp/type-context.h:103

>  103                      KJ_DEFER(if (erase_thread) {
   104                          std::unique_lock<std::mutex> lock(thread_context.waiter->m_mutex);
   105                          // Call erase here with a Connection* argument instead
   106                          // of an iterator argument, because the `request_thread`
   107                          // iterator may be invalid if the connection is closed
   108                          // during this function call. More specifically, the
   109                          // iterator may be invalid because SetThread adds a
   110                          // cleanup callback to the Connection destructor that
   111                          // erases the thread from the map, and also because the
   112                          // ProxyServer<Thread> destructor calls
   113                          // request_threads.clear().
   114                          request_threads.erase(server.m_context.connection);
   115                      });
   116                      fn.invoke(server_context, args...);
   117                  }

[#24](/bitcoin-bitcoin/24/) 0x0000559779209bad in run (this=0x7f677a7fb3b0) at /nix/store/6inpywqa7f4njlq1i0bmkznxn5y5x69g-capnproto-1.2.0/include/kj/common.h:2010
[#25](/bitcoin-bitcoin/25/) 0x000055977920865a in ~Deferred (this=0x7f677a7fb3b0) at /nix/store/6inpywqa7f4njlq1i0bmkznxn5y5x69g-capnproto-1.2.0/include/kj/common.h:1999
[#26](/bitcoin-bitcoin/26/) 0x0000559779208182 in operator() (this=0x7f6760001088) at ./ipc/libmultiprocess/include/mp/type-context.h:117

   116                      fn.invoke(server_context, args...);
>  117                  }

[#27](/bitcoin-bitcoin/27/) 0x0000559779207f5a in operator() (this=0x7f6760001080) at /nix/store/6inpywqa7f4njlq1i0bmkznxn5y5x69g-capnproto-1.2.0/include/kj/function.h:142
[#28](/bitcoin-bitcoin/28/) 0x0000559779097881 in kj::Function<void()>::operator() (this=0x7f677a7fb878) at /nix/store/6inpywqa7f4njlq1i0bmkznxn5y5x69g-capnproto-1.2.0/include/kj/function.h:119
[#29](/bitcoin-bitcoin/29/) 0x0000559779097567 in mp::Unlock<std::unique_lock<std::mutex>, kj::Function<void()>&> (lock=..., callback=...) at ./ipc/libmultiprocess/include/mp/util.h:198
[#30](/bitcoin-bitcoin/30/) 0x00005597796367c6 in mp::Waiter::wait<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::{lambda()#1}>(std::unique_lock<std::mutex>&, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::{lambda()#1})::{lambda()#1}::operator()() const (this=0x7f677a7fb8e0) at ./ipc/libmultiprocess/include/mp/proxy-io.h:306

   294      template <class Predicate>
   295      void wait(std::unique_lock<std::mutex>& lock, Predicate pred)
   296      {
   297          m_cv.wait(lock, [&] {
   298              // Important for this to be "while (m_fn)", not "if (m_fn)" to avoid
   299              // a lost-wakeup bug. A new m_fn and m_cv notification might be sent
   300              // after the fn() call and before the lock.lock() call in this loop
   301              // in the case where a capnp response is sent and a brand new
   302              // request is immediately received.
   303              while (m_fn) {
   304                  auto fn = std::move(*m_fn);
   305                  m_fn.reset();
>  306                  Unlock(lock, fn);
   307              }
   308              const bool done = pred();
   309              return done;
   310          });
   311      }

[#31](/bitcoin-bitcoin/31/) 0x0000559779636706 in std::condition_variable::wait<mp::Waiter::wait<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::{lambda()#1}>(std::unique_lock<std::mutex>&, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::{lambda()#1})::{lambda()#1}>(std::unique_lock<std::mutex>&, mp::Waiter::wait<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::{lambda()#1}>(std::unique_lock<std::mutex>&, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::{lambda()#1})::{lambda()#1}) (this=0x7f6774000e88, __lock=..., __p=...) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/condition_variable:104
[#32](/bitcoin-bitcoin/32/) 0x0000559779636637 in mp::Waiter::wait<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::{lambda()#1}>(std::unique_lock<std::mutex>&, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::{lambda()#1}) (this=0x7f6774000e60, lock=..., pred=...) at ./ipc/libmultiprocess/include/mp/proxy-io.h:297
[#33](/bitcoin-bitcoin/33/) 0x00005597796364e9 in mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const (this=0x7f67f00188c8) at ./ipc/libmultiprocess/src/mp/proxy.cpp:404
[#34](/bitcoin-bitcoin/34/) 0x0000559779636332 in std::__invoke_impl<void, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>(std::__invoke_other, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0&&) (__f=...) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:61
[#35](/bitcoin-bitcoin/35/) 0x0000559779636292 in std::__invoke<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>(mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0&&) (__fn=...) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:96
[#36](/bitcoin-bitcoin/36/) 0x000055977963624a in std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0> >::_M_invoke<0ul> (this=0x7f67f00188c8) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/std_thread.h:301
[#37](/bitcoin-bitcoin/37/) 0x00005597796361f2 in std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0> >::operator() (this=0x7f67f00188c8) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/std_thread.h:308
[#38](/bitcoin-bitcoin/38/) 0x0000559779635efa in std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0> > >::_M_run (this=0x7f67f00188c0) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/std_thread.h:253
[#39](/bitcoin-bitcoin/39/) 0x00007f67ffaed064 in execute_native_thread_routine () from /nix/store/41ym1jm1b7j3rhglk82gwg9jml26z1km-gcc-14.3.0-lib/lib/libstdc++.so.6
[#40](/bitcoin-bitcoin/40/) 0x00007f67ff69a97a in start_thread () from /nix/store/776irwlgfb65a782cxmyk61pck460fs9-glibc-2.40-66/lib/libc.so.6
[#41](/bitcoin-bitcoin/41/) 0x00007f67ff722d2c in __clone3 () from /nix/store/776irwlgfb65a782cxmyk61pck460fs9-glibc-2.40-66/lib/libc.so.6

</p> </details>

running against 30.x branch made it go away, thanks!