Fix mptest failures in bitcoin CI #186

pull ryanofsky wants to merge 11 commits into bitcoin-core:master from ryanofsky:pr/tsan changing 9 files +89 −68
  1. ryanofsky commented at 2:30 am on June 27, 2025: collaborator

    Recently merged PR #160 expanded unit tests to cover various unclean disconnection scenarios, but the new unit tests cause failures in bitcoin CI, despite passing in local CI (which doesn’t test as many sanitizers and platforms). Some of the errors are just test bugs, but others are real library bugs and race conditions.

    The bugs were reported in two threads starting https://github.com/Sjors/bitcoin/pull/90#issuecomment-3003870162 and https://github.com/bitcoin/bitcoin/pull/32345#issuecomment-3003863654, and they are described in detail in individual commit messages in this PR. The changes here fix all the known bugs and add new CI jobs and tests to detect them and catch regressions.

  2. DrahtBot commented at 2:30 am on June 27, 2025: none

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK Sjors

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

  3. ryanofsky force-pushed on Jun 27, 2025
  4. Sjors commented at 6:47 am on June 27, 2025: member
    You’ll need this to please the Bitcoin Core linter: https://github.com/bitcoin-core/libmultiprocess/commit/8ac8b4c85f80c6510a0569ad8467f9dfef9ccfd9
  5. in test/mp/test/test.cpp:59 in 5975fef52c outdated 
    53     std::function<void()> server_disconnect;
54     std::function<void()> client_disconnect;
55     std::promise<std::unique_ptr<ProxyClient<messages::FooInterface>>> client_promise;
56     std::unique_ptr<ProxyClient<messages::FooInterface>> client;
57     ProxyServer<messages::FooInterface>* server{nullptr};
58+    std::thread thread;
    Sjors commented at 6:56 am on June 27, 2025:
    5975fef52c8f6092979acfaeac0c665d45436e65: maybe add a comment that this has to go after any member variable touched.
    ryanofsky commented at 8:26 pm on June 27, 2025:

    re: #186 (review)

    5975fef: maybe add a comment that this has to go after any member variable touched.

    Thanks, added comment

  6. ci: add thread sanitizer job d2f6aa2e84
  7. mptest: fix race condition in TestSetup constructor 
    Fix race condition caught by TSAN between TestSetup constructor and std::thread
launched by the constructor, which could lead to the constructor setting member
variables to default values after the std::thread set them, which would not be
the intended order and could lead to bugs.

This change fixes all TSAN errors from the test except one, which will be fixed
in a subsequent commit in this PR.

The TSAN errors looked like the following (this is just the first error, there
were different ones as well):

[ TEST ] test.cpp:108: Call FooInterface methods
==================
WARNING: ThreadSanitizer: data race (pid=1970628)
  Write of size 8 at 0x7ffcf7ba4c30 by thread T1:
    #0 mp::test::TestSetup::TestSetup(bool)::{lambda()#1}::operator()() const::{lambda(mp::Connection&)#1}::operator()(mp::Connection&) const test/mp/test/test.cpp:71 (mptest+0x131c3d)
    #1 capnp::Capability::Client std::__invoke_impl<capnp::Capability::Client, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}::operator()() const::{lambda(mp::Connection&)#1}&, mp::Connection&>(std::__invoke_other, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}::operator()() const::{lambda(mp::Connection&)#1}&, mp::Connection&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:61 (mptest+0x131a25)
    #2 std::enable_if<is_invocable_r_v<capnp::Capability::Client, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}::operator()() const::{lambda(mp::Connection&)#1}&, mp::Connection&>, capnp::Capability::Client>::type std::__invoke_r<capnp::Capability::Client, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}::operator()() const::{lambda(mp::Connection&)#1}&, mp::Connection&>(mp::test::TestSetup::TestSetup(bool)::{lambda()#1}::operator()() const::{lambda(mp::Connection&)#1}&, mp::Connection&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:114 (mptest+0x131a25)
    #3 std::_Function_handler<capnp::Capability::Client (mp::Connection&), mp::test::TestSetup::TestSetup(bool)::{lambda()#1}::operator()() const::{lambda(mp::Connection&)#1}>::_M_invoke(std::_Any_data const&, mp::Connection&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_function.h:290 (mptest+0x131a25)
    #4 mp::Connection::Connection(mp::EventLoop&, kj::Own<kj::AsyncIoStream, decltype(nullptr)>&&, std::function<capnp::Capability::Client (mp::Connection&)> const&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_function.h:591 (mptest+0x13167f)
    #5 mp::test::TestSetup::TestSetup(bool)::{lambda()#1}::operator()() const /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/unique_ptr.h:1077 (mptest+0x13054c)
    #6 void std::__invoke_impl<void, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>(std::__invoke_other, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:61 (mptest+0x1303f9)
    #7 std::__invoke_result<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>::type std::__invoke<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>(mp::test::TestSetup::TestSetup(bool)::{lambda()#1}&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:96 (mptest+0x1303f9)
    #8 void std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:301 (mptest+0x1303f9)
    #9 std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> >::operator()() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:308 (mptest+0x1303f9)
    #10 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> > >::_M_run() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:253 (mptest+0x1303f9)
    #11 execute_native_thread_routine ??:? (libstdc++.so.6+0xed063)

  Previous write of size 8 at 0x7ffcf7ba4c30 by main thread:
    #0 __tsan_memset ??:? (mptest+0x8811f)
    #1 mp::test::TestSetup::TestSetup(bool) test/mp/test/test.cpp:57 (mptest+0x12c3b4)
    #2 mp::test::TestCase108::run() test/mp/test/test.cpp:110 (mptest+0x128160)
    #3 kj::Maybe<kj::Exception> kj::runCatchingExceptions<kj::TestRunner::run()::{lambda()#1}>(kj::TestRunner::run()::{lambda()#1}&&) ??:? (libkj-test.so.1.1.0+0x7290) (BuildId: 50ddf81234cd06daf5f2d3f11713ed193ade4eb7)

  Location is stack of main thread.

  Thread T1 (tid=1970630, running) created by main thread at:
    #0 pthread_create ??:? (mptest+0x9a2a5)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) ??:? (libstdc++.so.6+0xed138)
    #2 mp::test::TestCase108::run() test/mp/test/test.cpp:110 (mptest+0x128160)
    #3 kj::Maybe<kj::Exception> kj::runCatchingExceptions<kj::TestRunner::run()::{lambda()#1}>(kj::TestRunner::run()::{lambda()#1}&&) ??:? (libkj-test.so.1.1.0+0x7290) (BuildId: 50ddf81234cd06daf5f2d3f11713ed193ade4eb7)

SUMMARY: ThreadSanitizer: data race test/mp/test/test.cpp:71 in mp::test::TestSetup::TestSetup(bool)::{lambda()#1}::operator()() const::{lambda(mp::Connection&)#1}::operator()(mp::Connection&) const
    0d986ff144
  8. refactor: rename ProxyClient<Thread>::m_cleanup_it 
    Rename `ProxyClient<Thread>` `m_cleanup_it` member to `m_disconnect_cb`.

There is no change in behavor in this commit. The change is just being made so
a bugfix in an upcoming commit will be more obvious.

The `m_cleanup_it` name was a bad name because it didn't make it clear that
this cleanup function only runs on sudden disconnects, not clean shutdowns. It
was also confusing because this is a member of ProxyClient object, which has
its own cleanup functions, but this doesn't actually refer to one of those
functions, it refers to a cleanup function in an entirely different object (the
Connection object).
    07230f259f
  9. DrahtBot added the label Needs rebase on Jun 27, 2025
  10. refactor: Rename ProxyClient cleanup_it variable 
    Rename `ProxyClient` constructor `cleanup_it` local variable to `disconnect_cb`
to make a bugfix in an upcoming commit more obvious. Reasons for renaming this
variable are the same as the ones stated for renaming a similar variable in the
previous commit.
    81d58f5580
  11. ryanofsky force-pushed on Jun 27, 2025
  12. ryanofsky commented at 8:29 pm on June 27, 2025: collaborator
    Rebased 1ff5ad1ef9cedd45834ee2bb522c56b3afe89c60 -> 258b83cdbae0cd13a98a95bc12117caf216d4993 (pr/tsan.2 -> pr/tsan.3, compare) splitting up the second commit, simplifying it and making it broader so I think it will cover the other data race failure reported https://github.com/Sjors/bitcoin/pull/90#issuecomment-3012101603 https://cirrus-ci.com/task/4902764083412992?logs=ci#L3539, but I haven’t looked into the other failures reported there yet.
  13. DrahtBot removed the label Needs rebase on Jun 27, 2025
  14. doc: Add note about Waiter::m_mutex and interaction with the EventLoop::m_mutex 97e82ce19c
  15. proxy-io: fix race conditions in ProxyClientBase cleanup handler 
    This fixes a TSAN data race in the "disconnecting and blocking during the call"
unit test. The error output is below and problem it shows is that the
ProxyClientBase `m_context.connection->removeSyncCleanup(disconnect_cb);` call
is dereferencing `Connection::m_loop.m_loop` pointer variable shortly before
the ~Connection destructor sets the pointer to null in another thread, without
any synchronization in between.

This race could potentially cause segfaults if the timing was different and the
onDisconnect call happened at the same time as the removeSyncCleanup call.

Fix the error by moving the removeSyncCleanup call into a loop->sync() block so
it runs on the event loop thread instead of running asynchronously.

Moving this call fixes the TSAN error shown below but it causes 3 new TSAN
errors in the same unit test, which will be fixed in a seperate commit. The
TSAN error below error shows the ProxyClientBase object not using proper
synchronization when accessing a Connection object field before the Connection
object is destroyed. But this fix here causes Connection destruction to
delayed, so the new errors show the opposite problem of code called from the
Connection destructor not using proper synchronization before accessing
ProxyClientBase fields.

To explain the error output below in more detail: The test case that triggers
this data race has 3 threads. There is the main client thread calling
foo->callFnAsync() which is idle during this time. There is the corresponding
server thread T11 in the stack trace below which is executing the callFnAsync()
function, and calling setup.client_disconnect() to check for correct behavior
when there is a disconnect during the IPC call. Finally there is the event
loop thread T10 in the stack trace below which is actually deleting the server
Connection object in response to the client_disconnect() call (which deletes
the client Connection object).

During the callFnAsync() call, the server thread T11 responsible for executing
that call creates a ProxyClient<mp::Thread> object and inserts it into the
g_thread_context.request_threads std::map in the type-context.h CustomPassField
function, in case the server needs to make an IPC call to the client (which it
does not in this test). Before the CustomPassField funcion returns it deletes
the entry in the request_threads map, deleting the ProxyClient<Thread> object,
and leading to the ProxyClientBase<Thread>
`m_context.connection->removeSyncCleanup(disconnect_cb);` call referenced in the
beginning which accesses the m_loop pointer variable without a lock.

Error looks like:

WARNING: ThreadSanitizer: data race (pid=2025645)
  Write of size 8 at 0x726000012c00 by thread T10:
    #0 mp::EventLoopRef::reset(bool) src/mp/proxy.cpp:61 (mptest+0x2213bf)
    #1 mp::Connection::~Connection() include/mp/proxy.h:58 (mptest+0x221a9b)
    #2 kj::_::TransformPromiseNode<kj::_::Void, kj::_::Void, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}::operator()() const::{lambda()#2}, kj::_::PropagateException>::getImpl(kj::_::ExceptionOrValue&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/unique_ptr.h:93 (mptest+0x133817)
    #3 kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue&) ??:? (libkj-async.so.1.1.0+0x3da3d) (BuildId: 21ff3d5ab929b5e72be4dbbab1a9223f705af6e4)
    #4 mp::EventLoop::loop() src/mp/proxy.cpp:229 (mptest+0x222f60)
    #5 mp::test::TestSetup::TestSetup(bool)::{lambda()#1}::operator()() const test/mp/test/test.cpp:92 (mptest+0x130d23)
    #6 void std::__invoke_impl<void, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>(std::__invoke_other, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:61 (mptest+0x130789)
    #7 std::__invoke_result<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>::type std::__invoke<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>(mp::test::TestSetup::TestSetup(bool)::{lambda()#1}&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:96 (mptest+0x130789)
    #8 void std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:301 (mptest+0x130789)
    #9 std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> >::operator()() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:308 (mptest+0x130789)
    #10 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> > >::_M_run() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:253 (mptest+0x130789)
    #11 execute_native_thread_routine ??:? (libstdc++.so.6+0xed063)

  Previous read of size 8 at 0x726000012c00 by thread T11 (mutexes: write M0):
    #0 mp::Connection::removeSyncCleanup(std::_List_iterator<std::function<void ()> >) include/mp/proxy.h:60 (mptest+0x221dbf)
    #1 std::_Function_handler<void (), mp::ProxyClientBase<mp::Thread, capnp::Void>::ProxyClientBase(mp::Thread::Client, mp::Connection*, bool)::{lambda()#2}>::_M_invoke(std::_Any_data const&) include/mp/proxy-io.h:419 (mptest+0x228814)
    #2 mp::CleanupRun(std::__cxx11::list<std::function<void ()>, std::allocator<std::function<void ()> > >&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_function.h:591 (mptest+0x1ca65b)
    #3 mp::ProxyClientBase<mp::Thread, capnp::Void>::~ProxyClientBase() include/mp/proxy-io.h:446 (mptest+0x227790)
    #4 mp::ProxyClient<mp::Thread>::~ProxyClient() src/mp/proxy.cpp:339 (mptest+0x223fbd)
    #5 std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >::~pair() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_iterator.h:3013 (mptest+0x1cad8a)
    #6 void std::destroy_at<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >(std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >*) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_construct.h:88 (mptest+0x1cad8a)
    #7 void std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > > >::destroy<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >(std::allocator<std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >&, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >*) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/alloc_traits.h:599 (mptest+0x1cad8a)
    #8 std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::_M_destroy_node(std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >*) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_tree.h:621 (mptest+0x1cad8a)
    #9 std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::_M_drop_node(std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >*) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_tree.h:629 (mptest+0x1cad8a)
    #10 std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::_M_erase(std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >*) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_tree.h:1934 (mptest+0x1cad8a)
    #11 std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::_Rb_tree_const_iterator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_tree.h:1251 (mptest+0x1cace1)
    #12 _ZZZN2mp9PassFieldINS_8AccessorINS_10foo_fields7ContextELi17EEENS_19ServerInvokeContextINS_11ProxyServerINS_4test8messages12FooInterfaceEEEN5capnp11CallContextINS9_17CallFnAsyncParamsENS9_18CallFnAsyncResultsEEEEENS_10ServerCallEJNS_8TypeListIJEEEEEENSt9enable_ifIXsr3std7is_sameIDTclsrT_3getcldtdtfL1p1_12call_context9getParamsEEENS_7Context6ReaderEEE5valueEN2kj7PromiseINT0_11CallContextEEEE4typeENS_8PriorityILi1EEESJ_RSR_RKT1_DpOT2_ENUlvE_clEvENKUlvE0_clEv /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_tree.h:2519 (mptest+0x1f9b7e)
    #13 _ZN2kj1_8DeferredIZZN2mp9PassFieldINS2_8AccessorINS2_10foo_fields7ContextELi17EEENS2_19ServerInvokeContextINS2_11ProxyServerINS2_4test8messages12FooInterfaceEEEN5capnp11CallContextINSC_17CallFnAsyncParamsENSC_18CallFnAsyncResultsEEEEENS2_10ServerCallEJNS2_8TypeListIJEEEEEENSt9enable_ifIXsr3std7is_sameIDTclsrT_3getcldtdtfL1p1_12call_context9getParamsEEENS2_7Context6ReaderEEE5valueENS_7PromiseINT0_11CallContextEEEE4typeENS2_8PriorityILi1EEESM_RST_RKT1_DpOT2_ENUlvE_clEvEUlvE0_E3runEv /nix/store/46kiq9naswgbqfc14kc9nxcbgd0rv0m2-capnproto-1.1.0/include/kj/common.h:2007 (mptest+0x1f915d)
    #14 _ZN2kj1_8DeferredIZZN2mp9PassFieldINS2_8AccessorINS2_10foo_fields7ContextELi17EEENS2_19ServerInvokeContextINS2_11ProxyServerINS2_4test8messages12FooInterfaceEEEN5capnp11CallContextINSC_17CallFnAsyncParamsENSC_18CallFnAsyncResultsEEEEENS2_10ServerCallEJNS2_8TypeListIJEEEEEENSt9enable_ifIXsr3std7is_sameIDTclsrT_3getcldtdtfL1p1_12call_context9getParamsEEENS2_7Context6ReaderEEE5valueENS_7PromiseINT0_11CallContextEEEE4typeENS2_8PriorityILi1EEESM_RST_RKT1_DpOT2_ENUlvE_clEvEUlvE0_ED2Ev /nix/store/46kiq9naswgbqfc14kc9nxcbgd0rv0m2-capnproto-1.1.0/include/kj/common.h:1996 (mptest+0x1f915d)
    #15 _ZZN2mp9PassFieldINS_8AccessorINS_10foo_fields7ContextELi17EEENS_19ServerInvokeContextINS_11ProxyServerINS_4test8messages12FooInterfaceEEEN5capnp11CallContextINS9_17CallFnAsyncParamsENS9_18CallFnAsyncResultsEEEEENS_10ServerCallEJNS_8TypeListIJEEEEEENSt9enable_ifIXsr3std7is_sameIDTclsrT_3getcldtdtfL0p1_12call_context9getParamsEEENS_7Context6ReaderEEE5valueEN2kj7PromiseINT0_11CallContextEEEE4typeENS_8PriorityILi1EEESJ_RSR_RKT1_DpOT2_ENUlvE_clEv include/mp/type-context.h:122 (mptest+0x1f915d)
    #16 _ZN2kj8FunctionIFvvEE4ImplIZN2mp9PassFieldINS4_8AccessorINS4_10foo_fields7ContextELi17EEENS4_19ServerInvokeContextINS4_11ProxyServerINS4_4test8messages12FooInterfaceEEEN5capnp11CallContextINSE_17CallFnAsyncParamsENSE_18CallFnAsyncResultsEEEEENS4_10ServerCallEJNS4_8TypeListIJEEEEEENSt9enable_ifIXsr3std7is_sameIDTclsrT_3getcldtdtfL0p1_12call_context9getParamsEEENS4_7Context6ReaderEEE5valueENS_7PromiseINT0_11CallContextEEEE4typeENS4_8PriorityILi1EEESO_RSV_RKT1_DpOT2_EUlvE_EclEv /nix/store/46kiq9naswgbqfc14kc9nxcbgd0rv0m2-capnproto-1.1.0/include/kj/function.h:142 (mptest+0x1f8e89)
    #17 kj::Function<void ()>::operator()() /nix/store/46kiq9naswgbqfc14kc9nxcbgd0rv0m2-capnproto-1.1.0/include/kj/function.h:119 (mptest+0x1547f6)
    #18 void mp::Unlock<std::unique_lock<std::mutex>, kj::Function<void ()>&>(std::unique_lock<std::mutex>&, kj::Function<void ()>&) include/mp/util.h:198 (mptest+0x1547f6)
    #19 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0> > >::_M_run() include/mp/proxy-io.h:294 (mptest+0x2266ca)
    #20 execute_native_thread_routine ??:? (libstdc++.so.6+0xed063)

  Location is heap block of size 1024 at 0x726000012c00 allocated by thread T10:
    #0 operator new(unsigned long) ??:? (mptest+0x127c3c)
    #1 mp::test::TestSetup::TestSetup(bool)::{lambda()#1}::operator()() const /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/unique_ptr.h:1077 (mptest+0x130879)
    #2 void std::__invoke_impl<void, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>(std::__invoke_other, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:61 (mptest+0x130789)
    #3 std::__invoke_result<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>::type std::__invoke<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>(mp::test::TestSetup::TestSetup(bool)::{lambda()#1}&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:96 (mptest+0x130789)
    #4 void std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:301 (mptest+0x130789)
    #5 std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> >::operator()() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:308 (mptest+0x130789)
    #6 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> > >::_M_run() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:253 (mptest+0x130789)
    #7 execute_native_thread_routine ??:? (libstdc++.so.6+0xed063)

  Mutex M0 (0x721c00003790) created at:
    #0 pthread_mutex_lock ??:? (mptest+0x9aa5c)
    #1 __gthread_mutex_lock(pthread_mutex_t*) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/x86_64-unknown-linux-gnu/bits/gthr-default.h:762 (mptest+0x2265e2)
    #2 std::mutex::lock() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_mutex.h:113 (mptest+0x2265e2)
    #3 std::unique_lock<std::mutex>::lock() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/unique_lock.h:147 (mptest+0x2265e2)
    #4 std::unique_lock<std::mutex>::unique_lock(std::mutex&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/unique_lock.h:73 (mptest+0x2265e2)
    #5 mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const src/mp/proxy.cpp:399 (mptest+0x2265e2)
    #6 void std::__invoke_impl<void, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>(std::__invoke_other, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:61 (mptest+0x2265e2)
    #7 std::__invoke_result<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>::type std::__invoke<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>(mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:96 (mptest+0x2265e2)
    #8 void std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:301 (mptest+0x2265e2)
    #9 std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0> >::operator()() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:308 (mptest+0x2265e2)
    #10 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0> > >::_M_run() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:253 (mptest+0x2265e2)
    #11 execute_native_thread_routine ??:? (libstdc++.so.6+0xed063)

  Thread T10 (tid=2025656, running) created by main thread at:
    #0 pthread_create ??:? (mptest+0x9a2a5)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) ??:? (libstdc++.so.6+0xed138)
    #2 mp::test::TestCase251::run() test/mp/test/test.cpp:271 (mptest+0x12b5bd)
    #3 kj::Maybe<kj::Exception> kj::runCatchingExceptions<kj::TestRunner::run()::{lambda()#1}>(kj::TestRunner::run()::{lambda()#1}&&) ??:? (libkj-test.so.1.1.0+0x7290) (BuildId: 50ddf81234cd06daf5f2d3f11713ed193ade4eb7)

  Thread T11 (tid=2025657, running) created by thread T10 at:
    #0 pthread_create ??:? (mptest+0x9a2a5)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) ??:? (libstdc++.so.6+0xed138)
    #2 mp::ThreadMap::Server::dispatchCall(unsigned long, unsigned short, capnp::CallContext<capnp::AnyPointer, capnp::AnyPointer>) build-tsan/include/mp/proxy.capnp.c++:602 (mptest+0x220bb2)
    #3 virtual thunk to mp::ThreadMap::Server::dispatchCall(unsigned long, unsigned short, capnp::CallContext<capnp::AnyPointer, capnp::AnyPointer>) build-tsan/include/mp/proxy.capnp.c++:? (mptest+0x220bb2)
    #4 capnp::LocalClient::callInternal(unsigned long, unsigned short, capnp::CallContextHook&) ??:? (libcapnp-rpc.so.1.1.0+0x7551d) (BuildId: ec28fb7ea510d2e28e99522abd0ce44adde7cf44)
    #5 mp::EventLoop::loop() src/mp/proxy.cpp:229 (mptest+0x222f60)
    #6 mp::test::TestSetup::TestSetup(bool)::{lambda()#1}::operator()() const test/mp/test/test.cpp:92 (mptest+0x130d23)
    #7 void std::__invoke_impl<void, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>(std::__invoke_other, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:61 (mptest+0x130789)
    #8 std::__invoke_result<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>::type std::__invoke<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>(mp::test::TestSetup::TestSetup(bool)::{lambda()#1}&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:96 (mptest+0x130789)
    #9 void std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:301 (mptest+0x130789)
    #10 std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> >::operator()() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:308 (mptest+0x130789)
    #11 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> > >::_M_run() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:253 (mptest+0x130789)
    #12 execute_native_thread_routine ??:? (libstdc++.so.6+0xed063)

SUMMARY: ThreadSanitizer: data race src/mp/proxy.cpp:61 in mp::EventLoopRef::reset(bool)
==================
[ PASS ] test.cpp:251: Calling IPC method, disconnecting and blocking during the call (175646 μs)
5 test(s) passed
ThreadSanitizer: reported 1 warnings
nt method call interrupted by disconnect.
LOG0: {mptest-2025645/mptest-2025654} IPC server destroy N2mp11ProxyServerINS_4test8messages12FooInterfaceEEE
LOG0: {mptest-2025645/mptest-2025654} EventLoop::loop done, cancelling event listeners.
LOG0: {mptest-2025645/mptest-2025654} EventLoop::loop bye.
LOG0: {mptest-2025645/mptest-2025645} IPC client send FooInterface.add$Params (a = 1, b = 2)
LOG0: {mptest-2025645/mptest-2025656} IPC server recv request  #35 FooInterface.add$Params (a = 1, b = 2)
LOG0: {mptest-2025645/mptest-2025656} IPC server send response #35 FooInterface.add$Results (result = 3)
LOG0: {mptest-2025645/mptest-2025645} IPC client recv FooInterface.add$Results (result = 3)
LOG0: {mptest-2025645/mptest-2025645} IPC client send FooInterface.initThreadMap$Params (threadMap = <external capability>)
LOG0: {mptest-2025645/mptest-2025656} IPC server recv request  #36 FooInterface.initThreadMap$Params (threadMap = <external capability>)
LOG0: {mptest-2025645/mptest-2025656} IPC server send response #36 FooInterface.initThreadMap$Results (threadMap = <external capability>)
LOG0: {mptest-2025645/mptest-2025645} IPC client recv FooInterface.initThreadMap$Results (threadMap = <external capability>)
LOG0: {mptest-2025645/mptest-2025645} IPC client send FooInterface.callFnAsync$Params (context = (thread = <external capability>, callbackThread = <external capability>))
LOG0: {mptest-2025645/mptest-2025656} IPC server recv request  #37 FooInterface.callFnAsync$Params (context = (thread = <external capability>, callbackThread = <external capability>))
LOG0: {mptest-2025645/mptest-2025656} IPC server post request  #37 {mptest-2025645/mptest-2025657 (from mptest-2025645/mptest-2025645)}
LOG1: IPC client method call interrupted by disconnect.
LOG0: {mptest-2025645/mptest-2025656} IPC server send response #37 FooInterface.callFnAsync$Results ()
LOG0: {mptest-2025645/mptest-2025656} IPC server destroy N2mp11ProxyServerINS_4test8messages12FooInterfaceEEE
LOG0: {mptest-2025645/mptest-2025656} EventLoop::loop done, cancelling event listeners.
LOG0: {mptest-2025645/mptest-2025656} EventLoop::loop bye.

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.22 sec

The following tests FAILED:
	  1 - mptest (Failed)
    d8011c8360
  16. proxy-io: fix race conditions in disconnect callback code 
    Fix 3 TSAN data races exposed by previous commit. (Context is described in the
previous commit message.)

The first race condition happens setting `m_context.connection = nullptr` in
ProxyClientBase disconnect_cb callback without a lock, after a prior
`m_context.connection` pointer dereference in the
ProxyClient<Thread>::~ProxyClient() destructor in another thread. This is fixed
by adding using the `m_context.loop->m_mutex` mutex to guard acceses to the
`m_connection` variable.

The second race condition happens in the ~Connection destructor accessing the
`m_sync_cleanup_fns` without synchronization after a prior call to
removeSyncCleanup from ProxyClient<Thread>::~ProxyClient() in the other thread.
This is fixed by using a lock to guard access to `m_sync_cleanup_fns` here.
(Was already covered by this lock in the other functions).

The third race condition happens in the Connection destructor setting its
m_loop.m_loop pointer to null without synchronization after the
ProxyClient<Thread>::~ProxyClient() destructor dereferenced it earlier inside a
connection->removeSyncCleanup() call. The destructor will never actually make
that call if the Connection is destroyed because it checks to see if
m_disconnect_cb is null beforehand. However m_disconnect_cb is sometimes
accesed without synchronization so TSAN correctly reports a data race. This
problem is fixed by consistently guarding m_disconnect_cb with Waiter::m_mutex
locks. This fix also depends on the second fix above to make the ~Connection
destructor actually wait for the removeSyncCleanup() call to finish.

>>> Data race #1 accessing ProxyClientBase::m_context.connection <<<

[ TEST ] test.cpp:251: Calling IPC method, disconnecting and blocking during the call
LOG0: {mptest-2029510/mptest-2029510} IPC client send FooInterface.add$Params (a = 1, b = 2)
LOG0: {mptest-2029510/mptest-2029521} IPC server recv request  #35 FooInterface.add$Params (a = 1, b = 2)
LOG0: {mptest-2029510/mptest-2029521} IPC server send response #35 FooInterface.add$Results (result = 3)
LOG0: {mptest-2029510/mptest-2029510} IPC client recv FooInterface.add$Results (result = 3)
LOG0: {mptest-2029510/mptest-2029510} IPC client send FooInterface.initThreadMap$Params (threadMap = <external capability>)
LOG0: {mptest-2029510/mptest-2029521} IPC server recv request  #36 FooInterface.initThreadMap$Params (threadMap = <external capability>)
LOG0: {mptest-2029510/mptest-2029521} IPC server send response #36 FooInterface.initThreadMap$Results (threadMap = <external capability>)
LOG0: {mptest-2029510/mptest-2029510} IPC client recv FooInterface.initThreadMap$Results (threadMap = <external capability>)
LOG0: {mptest-2029510/mptest-2029510} IPC client send FooInterface.callFnAsync$Params (context = (thread = <external capability>, callbackThread = <external capability>))
LOG0: {mptest-2029510/mptest-2029521} IPC server recv request  #37 FooInterface.callFnAsync$Params (context = (thread = <external capability>, callbackThread = <external capability>))
LOG0: {mptest-2029510/mptest-2029521} IPC server post request  #37 {mptest-2029510/mptest-2029522 (from mptest-2029510/mptest-2029510)}
LOG1: IPC client method call interrupted by disconnect.
==================
WARNING: ThreadSanitizer: data race (pid=2029510)
  Write of size 8 at 0x722000004240 by thread T10:
    #0 mp::ProxyClientBase<mp::Thread, capnp::Void>::ProxyClientBase(mp::Thread::Client, mp::Connection*, bool)::{lambda()#1}::operator()() const include/mp/proxy-io.h:401 (mptest+0x2287c6)

   396      auto disconnect_cb = m_context.connection->addSyncCleanup([this]() {
   397          // Release client capability by move-assigning to temporary.
   398          {
   399              typename Interface::Client(std::move(m_client));
   400          }
   401          m_context.connection = nullptr;
   402      });

    #1 void std::__invoke_impl<void, mp::ProxyClientBase<mp::Thread, capnp::Void>::ProxyClientBase(mp::Thread::Client, mp::Connection*, bool)::{lambda()#1}&>(std::__invoke_other, mp::ProxyClientBase<mp::Thread, capnp::Void>::ProxyClientBase(mp::Thread::Client, mp::Connection*, bool)::{lambda()#1}&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:61 (mptest+0x2287c6)
    #2 std::enable_if<is_invocable_r_v<void, mp::ProxyClientBase<mp::Thread, capnp::Void>::ProxyClientBase(mp::Thread::Client, mp::Connection*, bool)::{lambda()#1}&>, void>::type std::__invoke_r<void, mp::ProxyClientBase<mp::Thread, capnp::Void>::ProxyClientBase(mp::Thread::Client, mp::Connection*, bool)::{lambda()#1}&>(mp::ProxyClientBase<mp::Thread, capnp::Void>::ProxyClientBase(mp::Thread::Client, mp::Connection*, bool)::{lambda()#1}&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:111 (mptest+0x2287c6)
    #3 std::_Function_handler<void (), mp::ProxyClientBase<mp::Thread, capnp::Void>::ProxyClientBase(mp::Thread::Client, mp::Connection*, bool)::{lambda()#1}>::_M_invoke(std::_Any_data const&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_function.h:290 (mptest+0x2287c6)
    #4 mp::Connection::~Connection() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_function.h:591 (mptest+0x221a05)
    #5 kj::_::TransformPromiseNode<kj::_::Void, kj::_::Void, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}::operator()() const::{lambda()#2}, kj::_::PropagateException>::getImpl(kj::_::ExceptionOrValue&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/unique_ptr.h:93 (mptest+0x133817)
    #6 kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue&) ??:? (libkj-async.so.1.1.0+0x3da3d) (BuildId: 21ff3d5ab929b5e72be4dbbab1a9223f705af6e4)
    #7 mp::EventLoop::loop() src/mp/proxy.cpp:229 (mptest+0x223010)
    #8 mp::test::TestSetup::TestSetup(bool)::{lambda()#1}::operator()() const test/mp/test/test.cpp:92 (mptest+0x130d23)
    #9 void std::__invoke_impl<void, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>(std::__invoke_other, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:61 (mptest+0x130789)
    #10 std::__invoke_result<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>::type std::__invoke<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>(mp::test::TestSetup::TestSetup(bool)::{lambda()#1}&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:96 (mptest+0x130789)
    #11 void std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:301 (mptest+0x130789)
    #12 std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> >::operator()() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:308 (mptest+0x130789)
    #13 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> > >::_M_run() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:253 (mptest+0x130789)
    #14 execute_native_thread_routine ??:? (libstdc++.so.6+0xed063)

  Previous read of size 8 at 0x722000004240 by thread T11 (mutexes: write M0):
    #0 mp::ProxyClient<mp::Thread>::~ProxyClient() src/mp/proxy.cpp:337 (mptest+0x22404d)

   336      if (m_disconnect_cb) {
   337          m_context.connection->removeSyncCleanup(*m_disconnect_cb);
   338      }

    #1 std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >::~pair() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_iterator.h:3013 (mptest+0x1cadda)
    #2 void std::destroy_at<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >(std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >*) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_construct.h:88 (mptest+0x1cadda)
    #3 void std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > > >::destroy<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >(std::allocator<std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >&, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >*) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/alloc_traits.h:599 (mptest+0x1cadda)
    #4 std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::_M_destroy_node(std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >*) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_tree.h:621 (mptest+0x1cadda)
    #5 std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::_M_drop_node(std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >*) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_tree.h:629 (mptest+0x1cadda)
    #6 std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::_M_erase(std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >*) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_tree.h:1934 (mptest+0x1cadda)
    #7 std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::_Rb_tree_const_iterator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_tree.h:1251 (mptest+0x1cad31)
    #8 _ZZZN2mp9PassFieldINS_8AccessorINS_10foo_fields7ContextELi17EEENS_19ServerInvokeContextINS_11ProxyServerINS_4test8messages12FooInterfaceEEEN5capnp11CallContextINS9_17CallFnAsyncParamsENS9_18CallFnAsyncResultsEEEEENS_10ServerCallEJNS_8TypeListIJEEEEEENSt9enable_ifIXsr3std7is_sameIDTclsrT_3getcldtdtfL1p1_12call_context9getParamsEEENS_7Context6ReaderEEE5valueEN2kj7PromiseINT0_11CallContextEEEE4typeENS_8PriorityILi1EEESJ_RSR_RKT1_DpOT2_ENUlvE_clEvENKUlvE0_clEv /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_tree.h:2519 (mptest+0x1f9c2e)
    #9 _ZN2kj1_8DeferredIZZN2mp9PassFieldINS2_8AccessorINS2_10foo_fields7ContextELi17EEENS2_19ServerInvokeContextINS2_11ProxyServerINS2_4test8messages12FooInterfaceEEEN5capnp11CallContextINSC_17CallFnAsyncParamsENSC_18CallFnAsyncResultsEEEEENS2_10ServerCallEJNS2_8TypeListIJEEEEEENSt9enable_ifIXsr3std7is_sameIDTclsrT_3getcldtdtfL1p1_12call_context9getParamsEEENS2_7Context6ReaderEEE5valueENS_7PromiseINT0_11CallContextEEEE4typeENS2_8PriorityILi1EEESM_RST_RKT1_DpOT2_ENUlvE_clEvEUlvE0_E3runEv /nix/store/46kiq9naswgbqfc14kc9nxcbgd0rv0m2-capnproto-1.1.0/include/kj/common.h:2007 (mptest+0x1f920d)
    #10 _ZN2kj1_8DeferredIZZN2mp9PassFieldINS2_8AccessorINS2_10foo_fields7ContextELi17EEENS2_19ServerInvokeContextINS2_11ProxyServerINS2_4test8messages12FooInterfaceEEEN5capnp11CallContextINSC_17CallFnAsyncParamsENSC_18CallFnAsyncResultsEEEEENS2_10ServerCallEJNS2_8TypeListIJEEEEEENSt9enable_ifIXsr3std7is_sameIDTclsrT_3getcldtdtfL1p1_12call_context9getParamsEEENS2_7Context6ReaderEEE5valueENS_7PromiseINT0_11CallContextEEEE4typeENS2_8PriorityILi1EEESM_RST_RKT1_DpOT2_ENUlvE_clEvEUlvE0_ED2Ev /nix/store/46kiq9naswgbqfc14kc9nxcbgd0rv0m2-capnproto-1.1.0/include/kj/common.h:1996 (mptest+0x1f920d)
    #11 _ZZN2mp9PassFieldINS_8AccessorINS_10foo_fields7ContextELi17EEENS_19ServerInvokeContextINS_11ProxyServerINS_4test8messages12FooInterfaceEEEN5capnp11CallContextINS9_17CallFnAsyncParamsENS9_18CallFnAsyncResultsEEEEENS_10ServerCallEJNS_8TypeListIJEEEEEENSt9enable_ifIXsr3std7is_sameIDTclsrT_3getcldtdtfL0p1_12call_context9getParamsEEENS_7Context6ReaderEEE5valueEN2kj7PromiseINT0_11CallContextEEEE4typeENS_8PriorityILi1EEESJ_RSR_RKT1_DpOT2_ENUlvE_clEv include/mp/type-context.h:122 (mptest+0x1f920d)
    #12 _ZN2kj8FunctionIFvvEE4ImplIZN2mp9PassFieldINS4_8AccessorINS4_10foo_fields7ContextELi17EEENS4_19ServerInvokeContextINS4_11ProxyServerINS4_4test8messages12FooInterfaceEEEN5capnp11CallContextINSE_17CallFnAsyncParamsENSE_18CallFnAsyncResultsEEEEENS4_10ServerCallEJNS4_8TypeListIJEEEEEENSt9enable_ifIXsr3std7is_sameIDTclsrT_3getcldtdtfL0p1_12call_context9getParamsEEENS4_7Context6ReaderEEE5valueENS_7PromiseINT0_11CallContextEEEE4typeENS4_8PriorityILi1EEESO_RSV_RKT1_DpOT2_EUlvE_EclEv /nix/store/46kiq9naswgbqfc14kc9nxcbgd0rv0m2-capnproto-1.1.0/include/kj/function.h:142 (mptest+0x1f8f39)
    #13 kj::Function<void ()>::operator()() /nix/store/46kiq9naswgbqfc14kc9nxcbgd0rv0m2-capnproto-1.1.0/include/kj/function.h:119 (mptest+0x154816)
    #14 void mp::Unlock<std::unique_lock<std::mutex>, kj::Function<void ()>&>(std::unique_lock<std::mutex>&, kj::Function<void ()>&) include/mp/util.h:198 (mptest+0x154816)
    #15 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0> > >::_M_run() include/mp/proxy-io.h:294 (mptest+0x22677a)
    #16 execute_native_thread_routine ??:? (libstdc++.so.6+0xed063)

  Location is heap block of size 128 at 0x722000004200 allocated by thread T11:
    #0 operator new(unsigned long) ??:? (mptest+0x127c3c)
    #1 std::__new_allocator<std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::allocate(unsigned long, void const*) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/new_allocator.h:151 (mptest+0x2281e4)
    #2 std::allocator<std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::allocate(unsigned long) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/allocator.h:196 (mptest+0x2281e4)
    #3 std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > > >::allocate(std::allocator<std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >&, unsigned long) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/alloc_traits.h:515 (mptest+0x2281e4)
    #4 std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::_M_get_node() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_tree.h:559 (mptest+0x2281e4)
    #5 std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >* std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::_M_create_node<std::piecewise_construct_t const&, std::tuple<mp::Connection*&>, std::tuple<mp::Thread::Client&&, mp::Connection*&, bool&&> >(std::piecewise_construct_t const&, std::tuple<mp::Connection*&>&&, std::tuple<mp::Thread::Client&&, mp::Connection*&, bool&&>&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_tree.h:609 (mptest+0x2281e4)
    #6 std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::_Auto_node::_Auto_node<std::piecewise_construct_t const&, std::tuple<mp::Connection*&>, std::tuple<mp::Thread::Client&&, mp::Connection*&, bool&&> >(std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >&, std::piecewise_construct_t const&, std::tuple<mp::Connection*&>&&, std::tuple<mp::Thread::Client&&, mp::Connection*&, bool&&>&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_tree.h:1633 (mptest+0x228064)
    #7 std::pair<std::_Rb_tree_iterator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, bool> std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::_M_emplace_unique<std::piecewise_construct_t const&, std::tuple<mp::Connection*&>, std::tuple<mp::Thread::Client&&, mp::Connection*&, bool&&> >(std::piecewise_construct_t const&, std::tuple<mp::Connection*&>&&, std::tuple<mp::Thread::Client&&, mp::Connection*&, bool&&>&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_tree.h:2432 (mptest+0x228064)
    #8 std::pair<std::_Rb_tree_iterator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, bool> std::map<mp::Connection*, mp::ProxyClient<mp::Thread>, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::emplace<std::piecewise_construct_t const&, std::tuple<mp::Connection*&>, std::tuple<mp::Thread::Client&&, mp::Connection*&, bool&&> >(std::piecewise_construct_t const&, std::tuple<mp::Connection*&>&&, std::tuple<mp::Thread::Client&&, mp::Connection*&, bool&&>&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_map.h:608 (mptest+0x223c07)
    #9 mp::SetThread(std::map<mp::Connection*, mp::ProxyClient<mp::Thread>, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >&, std::mutex&, mp::Connection*, std::function<mp::Thread::Client ()> const&) src/mp/proxy.cpp:311 (mptest+0x223c07)
    #10 _ZZN2mp9PassFieldINS_8AccessorINS_10foo_fields7ContextELi17EEENS_19ServerInvokeContextINS_11ProxyServerINS_4test8messages12FooInterfaceEEEN5capnp11CallContextINS9_17CallFnAsyncParamsENS9_18CallFnAsyncResultsEEEEENS_10ServerCallEJNS_8TypeListIJEEEEEENSt9enable_ifIXsr3std7is_sameIDTclsrT_3getcldtdtfL0p1_12call_context9getParamsEEENS_7Context6ReaderEEE5valueEN2kj7PromiseINT0_11CallContextEEEE4typeENS_8PriorityILi1EEESJ_RSR_RKT1_DpOT2_ENUlvE_clEv include/mp/type-context.h:93 (mptest+0x1f912b)
    #11 _ZN2kj8FunctionIFvvEE4ImplIZN2mp9PassFieldINS4_8AccessorINS4_10foo_fields7ContextELi17EEENS4_19ServerInvokeContextINS4_11ProxyServerINS4_4test8messages12FooInterfaceEEEN5capnp11CallContextINSE_17CallFnAsyncParamsENSE_18CallFnAsyncResultsEEEEENS4_10ServerCallEJNS4_8TypeListIJEEEEEENSt9enable_ifIXsr3std7is_sameIDTclsrT_3getcldtdtfL0p1_12call_context9getParamsEEENS4_7Context6ReaderEEE5valueENS_7PromiseINT0_11CallContextEEEE4typeENS4_8PriorityILi1EEESO_RSV_RKT1_DpOT2_EUlvE_EclEv /nix/store/46kiq9naswgbqfc14kc9nxcbgd0rv0m2-capnproto-1.1.0/include/kj/function.h:142 (mptest+0x1f8f39)
    #12 kj::Function<void ()>::operator()() /nix/store/46kiq9naswgbqfc14kc9nxcbgd0rv0m2-capnproto-1.1.0/include/kj/function.h:119 (mptest+0x154816)
    #13 void mp::Unlock<std::unique_lock<std::mutex>, kj::Function<void ()>&>(std::unique_lock<std::mutex>&, kj::Function<void ()>&) include/mp/util.h:198 (mptest+0x154816)
    #14 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0> > >::_M_run() include/mp/proxy-io.h:294 (mptest+0x22677a)
    #15 execute_native_thread_routine ??:? (libstdc++.so.6+0xed063)

  Mutex M0 (0x721c00003790) created at:
    #0 pthread_mutex_lock ??:? (mptest+0x9aa5c)
    #1 __gthread_mutex_lock(pthread_mutex_t*) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/x86_64-unknown-linux-gnu/bits/gthr-default.h:762 (mptest+0x226692)
    #2 std::mutex::lock() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_mutex.h:113 (mptest+0x226692)
    #3 std::unique_lock<std::mutex>::lock() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/unique_lock.h:147 (mptest+0x226692)
    #4 std::unique_lock<std::mutex>::unique_lock(std::mutex&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/unique_lock.h:73 (mptest+0x226692)
    #5 mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const src/mp/proxy.cpp:399 (mptest+0x226692)
    #6 void std::__invoke_impl<void, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>(std::__invoke_other, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:61 (mptest+0x226692)
    #7 std::__invoke_result<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>::type std::__invoke<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>(mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:96 (mptest+0x226692)
    #8 void std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:301 (mptest+0x226692)
    #9 std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0> >::operator()() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:308 (mptest+0x226692)
    #10 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0> > >::_M_run() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:253 (mptest+0x226692)
    #11 execute_native_thread_routine ??:? (libstdc++.so.6+0xed063)

  Thread T10 (tid=2029521, running) created by main thread at:
    #0 pthread_create ??:? (mptest+0x9a2a5)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) ??:? (libstdc++.so.6+0xed138)
    #2 mp::test::TestCase251::run() test/mp/test/test.cpp:271 (mptest+0x12b5bd)
    #3 kj::Maybe<kj::Exception> kj::runCatchingExceptions<kj::TestRunner::run()::{lambda()#1}>(kj::TestRunner::run()::{lambda()#1}&&) ??:? (libkj-test.so.1.1.0+0x7290) (BuildId: 50ddf81234cd06daf5f2d3f11713ed193ade4eb7)

  Thread T11 (tid=2029522, running) created by thread T10 at:
    #0 pthread_create ??:? (mptest+0x9a2a5)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) ??:? (libstdc++.so.6+0xed138)
    #2 mp::ThreadMap::Server::dispatchCall(unsigned long, unsigned short, capnp::CallContext<capnp::AnyPointer, capnp::AnyPointer>) build-tsan/include/mp/proxy.capnp.c++:602 (mptest+0x220c62)
    #3 virtual thunk to mp::ThreadMap::Server::dispatchCall(unsigned long, unsigned short, capnp::CallContext<capnp::AnyPointer, capnp::AnyPointer>) build-tsan/include/mp/proxy.capnp.c++:? (mptest+0x220c62)
    #4 capnp::LocalClient::callInternal(unsigned long, unsigned short, capnp::CallContextHook&) ??:? (libcapnp-rpc.so.1.1.0+0x7551d) (BuildId: ec28fb7ea510d2e28e99522abd0ce44adde7cf44)
    #5 mp::EventLoop::loop() src/mp/proxy.cpp:229 (mptest+0x223010)
    #6 mp::test::TestSetup::TestSetup(bool)::{lambda()#1}::operator()() const test/mp/test/test.cpp:92 (mptest+0x130d23)
    #7 void std::__invoke_impl<void, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>(std::__invoke_other, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:61 (mptest+0x130789)
    #8 std::__invoke_result<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>::type std::__invoke<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>(mp::test::TestSetup::TestSetup(bool)::{lambda()#1}&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:96 (mptest+0x130789)
    #9 void std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:301 (mptest+0x130789)
    #10 std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> >::operator()() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:308 (mptest+0x130789)
    #11 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> > >::_M_run() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:253 (mptest+0x130789)
    #12 execute_native_thread_routine ??:? (libstdc++.so.6+0xed063)

SUMMARY: ThreadSanitizer: data race include/mp/proxy-io.h:401 in mp::ProxyClientBase<mp::Thread, capnp::Void>::ProxyClientBase(mp::Thread::Client, mp::Connection*, bool)::{lambda()#1}::operator()() const
==================

>>> Data race #2 accessing Connection::m_sync_cleanup_fns <<<

==================
WARNING: ThreadSanitizer: data race (pid=2029510)
  Write of size 8 at 0x726000012ff8 by thread T10:
    #0 std::__cxx11::_List_base<std::function<void ()>, std::allocator<std::function<void ()> > >::_M_dec_size(unsigned long) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_list.h:487 (mptest+0x221a20)
    #1 std::__cxx11::list<std::function<void ()>, std::allocator<std::function<void ()> > >::_M_erase(std::_List_iterator<std::function<void ()> >) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_list.h:2023 (mptest+0x221a20)
    #2 std::__cxx11::list<std::function<void ()>, std::allocator<std::function<void ()> > >::pop_front() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_list.h:1301 (mptest+0x221a20)
    #3 mp::Connection::~Connection() src/mp/proxy.cpp:137 (mptest+0x221a20)

   135      while (!m_sync_cleanup_fns.empty()) {
   136          m_sync_cleanup_fns.front()();
   137          m_sync_cleanup_fns.pop_front();
   138      }

    #4 kj::_::TransformPromiseNode<kj::_::Void, kj::_::Void, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}::operator()() const::{lambda()#2}, kj::_::PropagateException>::getImpl(kj::_::ExceptionOrValue&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/unique_ptr.h:93 (mptest+0x133817)
    #5 kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue&) ??:? (libkj-async.so.1.1.0+0x3da3d) (BuildId: 21ff3d5ab929b5e72be4dbbab1a9223f705af6e4)
    #6 mp::EventLoop::loop() src/mp/proxy.cpp:229 (mptest+0x223010)
    #7 mp::test::TestSetup::TestSetup(bool)::{lambda()#1}::operator()() const test/mp/test/test.cpp:92 (mptest+0x130d23)
    #8 void std::__invoke_impl<void, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>(std::__invoke_other, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:61 (mptest+0x130789)
    #9 std::__invoke_result<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>::type std::__invoke<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>(mp::test::TestSetup::TestSetup(bool)::{lambda()#1}&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:96 (mptest+0x130789)
    #10 void std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:301 (mptest+0x130789)
    #11 std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> >::operator()() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:308 (mptest+0x130789)
    #12 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> > >::_M_run() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:253 (mptest+0x130789)
    #13 execute_native_thread_routine ??:? (libstdc++.so.6+0xed063)

  Previous write of size 8 at 0x726000012ff8 by thread T11 (mutexes: write M0, write M1):
    #0 std::__cxx11::_List_base<std::function<void ()>, std::allocator<std::function<void ()> > >::_M_dec_size(unsigned long) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_list.h:487 (mptest+0x221ea5)
    #1 std::__cxx11::list<std::function<void ()>, std::allocator<std::function<void ()> > >::_M_erase(std::_List_iterator<std::function<void ()> >) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_list.h:2023 (mptest+0x221ea5)
    #2 std::__cxx11::list<std::function<void ()>, std::allocator<std::function<void ()> > >::erase(std::_List_const_iterator<std::function<void ()> >) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/list.tcc:158 (mptest+0x221ea5)
    #3 mp::Connection::removeSyncCleanup(std::_List_iterator<std::function<void ()> >) src/mp/proxy.cpp:158 (mptest+0x221ea5)

   155  void Connection::removeSyncCleanup(CleanupIt it)
   156  {
   157      const Lock lock(m_loop->m_mutex);
   158      m_sync_cleanup_fns.erase(it);
   159  }

    #4 mp::ProxyClient<mp::Thread>::~ProxyClient() src/mp/proxy.cpp:337 (mptest+0x224065)
    #5 std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >::~pair() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_iterator.h:3013 (mptest+0x1cadda)
    #6 void std::destroy_at<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >(std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >*) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_construct.h:88 (mptest+0x1cadda)
    #7 void std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > > >::destroy<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >(std::allocator<std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >&, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >*) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/alloc_traits.h:599 (mptest+0x1cadda)
    #8 std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::_M_destroy_node(std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >*) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_tree.h:621 (mptest+0x1cadda)
    #9 std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::_M_drop_node(std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >*) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_tree.h:629 (mptest+0x1cadda)
    #10 std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::_M_erase(std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >*) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_tree.h:1934 (mptest+0x1cadda)
    #11 std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::_Rb_tree_const_iterator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_tree.h:1251 (mptest+0x1cad31)
    #12 _ZZZN2mp9PassFieldINS_8AccessorINS_10foo_fields7ContextELi17EEENS_19ServerInvokeContextINS_11ProxyServerINS_4test8messages12FooInterfaceEEEN5capnp11CallContextINS9_17CallFnAsyncParamsENS9_18CallFnAsyncResultsEEEEENS_10ServerCallEJNS_8TypeListIJEEEEEENSt9enable_ifIXsr3std7is_sameIDTclsrT_3getcldtdtfL1p1_12call_context9getParamsEEENS_7Context6ReaderEEE5valueEN2kj7PromiseINT0_11CallContextEEEE4typeENS_8PriorityILi1EEESJ_RSR_RKT1_DpOT2_ENUlvE_clEvENKUlvE0_clEv /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_tree.h:2519 (mptest+0x1f9c2e)
    #13 _ZN2kj1_8DeferredIZZN2mp9PassFieldINS2_8AccessorINS2_10foo_fields7ContextELi17EEENS2_19ServerInvokeContextINS2_11ProxyServerINS2_4test8messages12FooInterfaceEEEN5capnp11CallContextINSC_17CallFnAsyncParamsENSC_18CallFnAsyncResultsEEEEENS2_10ServerCallEJNS2_8TypeListIJEEEEEENSt9enable_ifIXsr3std7is_sameIDTclsrT_3getcldtdtfL1p1_12call_context9getParamsEEENS2_7Context6ReaderEEE5valueENS_7PromiseINT0_11CallContextEEEE4typeENS2_8PriorityILi1EEESM_RST_RKT1_DpOT2_ENUlvE_clEvEUlvE0_E3runEv /nix/store/46kiq9naswgbqfc14kc9nxcbgd0rv0m2-capnproto-1.1.0/include/kj/common.h:2007 (mptest+0x1f920d)
    #14 _ZN2kj1_8DeferredIZZN2mp9PassFieldINS2_8AccessorINS2_10foo_fields7ContextELi17EEENS2_19ServerInvokeContextINS2_11ProxyServerINS2_4test8messages12FooInterfaceEEEN5capnp11CallContextINSC_17CallFnAsyncParamsENSC_18CallFnAsyncResultsEEEEENS2_10ServerCallEJNS2_8TypeListIJEEEEEENSt9enable_ifIXsr3std7is_sameIDTclsrT_3getcldtdtfL1p1_12call_context9getParamsEEENS2_7Context6ReaderEEE5valueENS_7PromiseINT0_11CallContextEEEE4typeENS2_8PriorityILi1EEESM_RST_RKT1_DpOT2_ENUlvE_clEvEUlvE0_ED2Ev /nix/store/46kiq9naswgbqfc14kc9nxcbgd0rv0m2-capnproto-1.1.0/include/kj/common.h:1996 (mptest+0x1f920d)
    #15 _ZZN2mp9PassFieldINS_8AccessorINS_10foo_fields7ContextELi17EEENS_19ServerInvokeContextINS_11ProxyServerINS_4test8messages12FooInterfaceEEEN5capnp11CallContextINS9_17CallFnAsyncParamsENS9_18CallFnAsyncResultsEEEEENS_10ServerCallEJNS_8TypeListIJEEEEEENSt9enable_ifIXsr3std7is_sameIDTclsrT_3getcldtdtfL0p1_12call_context9getParamsEEENS_7Context6ReaderEEE5valueEN2kj7PromiseINT0_11CallContextEEEE4typeENS_8PriorityILi1EEESJ_RSR_RKT1_DpOT2_ENUlvE_clEv include/mp/type-context.h:122 (mptest+0x1f920d)
    #16 _ZN2kj8FunctionIFvvEE4ImplIZN2mp9PassFieldINS4_8AccessorINS4_10foo_fields7ContextELi17EEENS4_19ServerInvokeContextINS4_11ProxyServerINS4_4test8messages12FooInterfaceEEEN5capnp11CallContextINSE_17CallFnAsyncParamsENSE_18CallFnAsyncResultsEEEEENS4_10ServerCallEJNS4_8TypeListIJEEEEEENSt9enable_ifIXsr3std7is_sameIDTclsrT_3getcldtdtfL0p1_12call_context9getParamsEEENS4_7Context6ReaderEEE5valueENS_7PromiseINT0_11CallContextEEEE4typeENS4_8PriorityILi1EEESO_RSV_RKT1_DpOT2_EUlvE_EclEv /nix/store/46kiq9naswgbqfc14kc9nxcbgd0rv0m2-capnproto-1.1.0/include/kj/function.h:142 (mptest+0x1f8f39)
    #17 kj::Function<void ()>::operator()() /nix/store/46kiq9naswgbqfc14kc9nxcbgd0rv0m2-capnproto-1.1.0/include/kj/function.h:119 (mptest+0x154816)
    #18 void mp::Unlock<std::unique_lock<std::mutex>, kj::Function<void ()>&>(std::unique_lock<std::mutex>&, kj::Function<void ()>&) include/mp/util.h:198 (mptest+0x154816)
    #19 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0> > >::_M_run() include/mp/proxy-io.h:294 (mptest+0x22677a)
    #20 execute_native_thread_routine ??:? (libstdc++.so.6+0xed063)

  Location is heap block of size 1024 at 0x726000012c00 allocated by thread T10:
    #0 operator new(unsigned long) ??:? (mptest+0x127c3c)
    #1 mp::test::TestSetup::TestSetup(bool)::{lambda()#1}::operator()() const /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/unique_ptr.h:1077 (mptest+0x130879)
    #2 void std::__invoke_impl<void, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>(std::__invoke_other, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:61 (mptest+0x130789)
    #3 std::__invoke_result<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>::type std::__invoke<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>(mp::test::TestSetup::TestSetup(bool)::{lambda()#1}&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:96 (mptest+0x130789)
    #4 void std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:301 (mptest+0x130789)
    #5 std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> >::operator()() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:308 (mptest+0x130789)
    #6 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> > >::_M_run() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:253 (mptest+0x130789)
    #7 execute_native_thread_routine ??:? (libstdc++.so.6+0xed063)

  Mutex M0 (0x721c00003790) created at:
    #0 pthread_mutex_lock ??:? (mptest+0x9aa5c)
    #1 __gthread_mutex_lock(pthread_mutex_t*) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/x86_64-unknown-linux-gnu/bits/gthr-default.h:762 (mptest+0x226692)
    #2 std::mutex::lock() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_mutex.h:113 (mptest+0x226692)
    #3 std::unique_lock<std::mutex>::lock() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/unique_lock.h:147 (mptest+0x226692)
    #4 std::unique_lock<std::mutex>::unique_lock(std::mutex&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/unique_lock.h:73 (mptest+0x226692)
    #5 mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const src/mp/proxy.cpp:399 (mptest+0x226692)
    #6 void std::__invoke_impl<void, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>(std::__invoke_other, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:61 (mptest+0x226692)
    #7 std::__invoke_result<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>::type std::__invoke<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>(mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:96 (mptest+0x226692)
    #8 void std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:301 (mptest+0x226692)
    #9 std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0> >::operator()() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:308 (mptest+0x226692)
    #10 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0> > >::_M_run() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:253 (mptest+0x226692)
    #11 execute_native_thread_routine ??:? (libstdc++.so.6+0xed063)

  Mutex M1 (0x7f5d5d7fd510) created at:
    #0 pthread_mutex_lock ??:? (mptest+0x9aa5c)
    #1 __gthread_mutex_lock(pthread_mutex_t*) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/x86_64-unknown-linux-gnu/bits/gthr-default.h:762 (mptest+0x22131c)
    #2 std::mutex::lock() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_mutex.h:113 (mptest+0x22131c)
    #3 std::unique_lock<std::mutex>::lock() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/unique_lock.h:147 (mptest+0x22131c)
    #4 std::unique_lock<std::mutex>::unique_lock(std::mutex&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/unique_lock.h:73 (mptest+0x22131c)
    #5 mp::Lock::Lock(mp::Mutex&) include/mp/util.h:172 (mptest+0x22131c)
    #6 std::__detail::__variant::_Uninitialized<mp::Lock, false>::_Uninitialized<mp::Mutex&>(std::in_place_index_t<0ul>, mp::Mutex&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/variant:250 (mptest+0x22131c)
    #7 std::__detail::__variant::_Variadic_union<false, mp::Lock>::_Variadic_union<mp::Mutex&>(std::in_place_index_t<0ul>, mp::Mutex&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/variant:412 (mptest+0x22131c)
    #8 std::__detail::__variant::_Variadic_union<false, mp::Lock*, mp::Lock>::_Variadic_union<1ul, mp::Mutex&>(std::in_place_index_t<1ul>, mp::Mutex&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/variant:418 (mptest+0x22131c)
    #9 std::__detail::__variant::_Variant_storage<false, mp::Lock*, mp::Lock>::_Variant_storage<1ul, mp::Mutex&>(std::in_place_index_t<1ul>, mp::Mutex&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/variant:486 (mptest+0x22131c)
    #10 std::__detail::__variant::_Copy_ctor_base<false, mp::Lock*, mp::Lock>::_Variant_storage<1ul, mp::Mutex&>(std::in_place_index_t<1ul>, mp::Mutex&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/variant:584 (mptest+0x22131c)
    #11 std::__detail::__variant::_Move_ctor_base<false, mp::Lock*, mp::Lock>::_Variant_storage<1ul, mp::Mutex&>(std::in_place_index_t<1ul>, mp::Mutex&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/variant:621 (mptest+0x22131c)
    #12 std::__detail::__variant::_Copy_assign_base<false, mp::Lock*, mp::Lock>::_Variant_storage<1ul, mp::Mutex&>(std::in_place_index_t<1ul>, mp::Mutex&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/variant:659 (mptest+0x22131c)
    #13 std::__detail::__variant::_Move_assign_base<false, mp::Lock*, mp::Lock>::_Variant_storage<1ul, mp::Mutex&>(std::in_place_index_t<1ul>, mp::Mutex&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/variant:711 (mptest+0x22131c)
    #14 std::__detail::__variant::_Variant_base<mp::Lock*, mp::Lock>::_Variant_base<1ul, mp::Mutex&>(std::in_place_index_t<1ul>, mp::Mutex&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/variant:773 (mptest+0x22131c)
    #15 std::variant<mp::Lock*, mp::Lock>::variant<1ul, mp::Mutex&, mp::Lock, void>(std::in_place_index_t<1ul>, mp::Mutex&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/variant:1478 (mptest+0x22131c)
    #16 std::variant<mp::Lock*, mp::Lock>::variant<mp::Lock, mp::Mutex&, void>(std::in_place_type_t<mp::Lock>, mp::Mutex&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/variant:1458 (mptest+0x22131c)
    #17 mp::PtrOrValue<mp::Lock>::PtrOrValue<mp::Mutex&>(mp::Lock*, mp::Mutex&) include/mp/util.h:138 (mptest+0x22131c)
    #18 mp::EventLoopRef::EventLoopRef(mp::EventLoop&, mp::Lock*) src/mp/proxy.cpp:50 (mptest+0x22131c)
    #19 mp::Connection::Connection(mp::EventLoop&, kj::Own<kj::AsyncIoStream, decltype(nullptr)>&&, std::function<capnp::Capability::Client (mp::Connection&)> const&) include/mp/proxy-io.h:321 (mptest+0x1318e7)
    #20 mp::test::TestSetup::TestSetup(bool)::{lambda()#1}::operator()() const /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/unique_ptr.h:1077 (mptest+0x1308dc)
    #21 void std::__invoke_impl<void, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>(std::__invoke_other, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:61 (mptest+0x130789)
    #22 std::__invoke_result<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>::type std::__invoke<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>(mp::test::TestSetup::TestSetup(bool)::{lambda()#1}&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:96 (mptest+0x130789)
    #23 void std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:301 (mptest+0x130789)
    #24 std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> >::operator()() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:308 (mptest+0x130789)
    #25 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> > >::_M_run() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:253 (mptest+0x130789)
    #26 execute_native_thread_routine ??:? (libstdc++.so.6+0xed063)

  Thread T10 (tid=2029521, running) created by main thread at:
    #0 pthread_create ??:? (mptest+0x9a2a5)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) ??:? (libstdc++.so.6+0xed138)
    #2 mp::test::TestCase251::run() test/mp/test/test.cpp:271 (mptest+0x12b5bd)
    #3 kj::Maybe<kj::Exception> kj::runCatchingExceptions<kj::TestRunner::run()::{lambda()#1}>(kj::TestRunner::run()::{lambda()#1}&&) ??:? (libkj-test.so.1.1.0+0x7290) (BuildId: 50ddf81234cd06daf5f2d3f11713ed193ade4eb7)

  Thread T11 (tid=2029522, running) created by thread T10 at:
    #0 pthread_create ??:? (mptest+0x9a2a5)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) ??:? (libstdc++.so.6+0xed138)
    #2 mp::ThreadMap::Server::dispatchCall(unsigned long, unsigned short, capnp::CallContext<capnp::AnyPointer, capnp::AnyPointer>) build-tsan/include/mp/proxy.capnp.c++:602 (mptest+0x220c62)
    #3 virtual thunk to mp::ThreadMap::Server::dispatchCall(unsigned long, unsigned short, capnp::CallContext<capnp::AnyPointer, capnp::AnyPointer>) build-tsan/include/mp/proxy.capnp.c++:? (mptest+0x220c62)
    #4 capnp::LocalClient::callInternal(unsigned long, unsigned short, capnp::CallContextHook&) ??:? (libcapnp-rpc.so.1.1.0+0x7551d) (BuildId: ec28fb7ea510d2e28e99522abd0ce44adde7cf44)
    #5 mp::EventLoop::loop() src/mp/proxy.cpp:229 (mptest+0x223010)
    #6 mp::test::TestSetup::TestSetup(bool)::{lambda()#1}::operator()() const test/mp/test/test.cpp:92 (mptest+0x130d23)
    #7 void std::__invoke_impl<void, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>(std::__invoke_other, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:61 (mptest+0x130789)
    #8 std::__invoke_result<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>::type std::__invoke<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>(mp::test::TestSetup::TestSetup(bool)::{lambda()#1}&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:96 (mptest+0x130789)
    #9 void std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:301 (mptest+0x130789)
    #10 std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> >::operator()() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:308 (mptest+0x130789)
    #11 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> > >::_M_run() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:253 (mptest+0x130789)
    #12 execute_native_thread_routine ??:? (libstdc++.so.6+0xed063)

SUMMARY: ThreadSanitizer: data race src/mp/proxy.cpp:137 in mp::Connection::~Connection()
==================

>>> Data race #3 accesing Connection::m_loop::m_loop <<<

==================
WARNING: ThreadSanitizer: data race (pid=2029510)
  Write of size 8 at 0x726000012c00 by thread T10:
    #0 mp::EventLoopRef::reset(bool) src/mp/proxy.cpp:61 (mptest+0x22146f)

    58  void EventLoopRef::reset(bool relock) MP_NO_TSA
    59  {
    60      if (auto* loop{m_loop}) {
    61          m_loop = nullptr;
    62          auto loop_lock{PtrOrValue{m_lock, loop->m_mutex}};
    63          loop_lock->assert_locked(loop->m_mutex);
    64          assert(loop->m_num_clients > 0);
    65          loop->m_num_clients -= 1;

    #1 mp::Connection::~Connection() include/mp/proxy.h:58 (mptest+0x221b4b)

    58      ~EventLoopRef() { reset(); }

    #2 kj::_::TransformPromiseNode<kj::_::Void, kj::_::Void, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}::operator()() const::{lambda()#2}, kj::_::PropagateException>::getImpl(kj::_::ExceptionOrValue&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/unique_ptr.h:93 (mptest+0x133817)
    #3 kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue&) ??:? (libkj-async.so.1.1.0+0x3da3d) (BuildId: 21ff3d5ab929b5e72be4dbbab1a9223f705af6e4)
    #4 mp::EventLoop::loop() src/mp/proxy.cpp:229 (mptest+0x223010)

   228      for (;;) {
   229          const size_t read_bytes = wait_stream->read(&buffer, 0, 1).wait(m_io_context.waitScope);
   230          if (read_bytes != 1) throw std::logic_error("EventLoop wait_stream closed unexpectedly");

    #5 mp::test::TestSetup::TestSetup(bool)::{lambda()#1}::operator()() const test/mp/test/test.cpp:92 (mptest+0x130d23)
    #6 void std::__invoke_impl<void, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>(std::__invoke_other, mp::test::TestSetup::TestSetup(bool)::{lambda()#1}&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:61 (mptest+0x130789)
    #7 std::__invoke_result<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>::type std::__invoke<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}>(mp::test::TestSetup::TestSetup(bool)::{lambda()#1}&&) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/invoke.h:96 (mptest+0x130789)
    #8 void std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:301 (mptest+0x130789)
    #9 std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> >::operator()() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:308 (mptest+0x130789)
    #10 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::test::TestSetup::TestSetup(bool)::{lambda()#1}> > >::_M_run() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/std_thread.h:253 (mptest+0x130789)
    #11 execute_native_thread_routine ??:? (libstdc++.so.6+0xed063)

  Previous read of size 8 at 0x726000012c00 by thread T11 (mutexes: write M0):
    #0 mp::Connection::removeSyncCleanup(std::_List_iterator<std::function<void ()> >) include/mp/proxy.h:60 (mptest+0x221e6f)

    60      EventLoop* operator->() const { assert(m_loop); return m_loop; }

    #1 mp::ProxyClient<mp::Thread>::~ProxyClient() src/mp/proxy.cpp:337 (mptest+0x224065)

   155  void Connection::removeSyncCleanup(CleanupIt it)
   156  {
   157      const Lock lock(m_loop->m_mutex);
   158      m_sync_cleanup_fns.erase(it);
   159  }
   ...
   336      if (m_disconnect_cb) {
   337          m_context.connection->removeSyncCleanup(*m_disconnect_cb);
   338      }

    #2 std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >::~pair() /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_iterator.h:3013 (mptest+0x1cadda)
    #3 void std::destroy_at<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >(std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >*) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_construct.h:88 (mptest+0x1cadda)
    #4 void std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > > >::destroy<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >(std::allocator<std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >&, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >*) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/alloc_traits.h:599 (mptest+0x1cadda)
    #5 std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::_M_destroy_node(std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >*) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_tree.h:621 (mptest+0x1cadda)
    #6 std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::_M_drop_node(std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >*) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_tree.h:629 (mptest+0x1cadda)
    #7 std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::_M_erase(std::_Rb_tree_node<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >*) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_tree.h:1934 (mptest+0x1cadda)
    #8 std::_Rb_tree<mp::Connection*, std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> >, std::_Select1st<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::less<mp::Connection*>, std::allocator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >, std::_Rb_tree_const_iterator<std::pair<mp::Connection* const, mp::ProxyClient<mp::Thread> > >) /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_tree.h:1251 (mptest+0x1cad31)
    #9 _ZZZN2mp9PassFieldINS_8AccessorINS_10foo_fields7ContextELi17EEENS_19ServerInvokeContextINS_11ProxyServerINS_4test8messages12FooInterfaceEEEN5capnp11CallContextINS9_17CallFnAsyncParamsENS9_18CallFnAsyncResultsEEEEENS_10ServerCallEJNS_8TypeListIJEEEEEENSt9enable_ifIXsr3std7is_sameIDTclsrT_3getcldtdtfL1p1_12call_context9getParamsEEENS_7Context6ReaderEEE5valueEN2kj7PromiseINT0_11CallContextEEEE4typeENS_8PriorityILi1EEESJ_RSR_RKT1_DpOT2_ENUlvE_clEvENKUlvE0_clEv /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/stl_tree.h:2519 (mptest+0x1f9c2e)
    #10 _ZN2kj1_8DeferredIZZN2mp9PassFieldINS2_8AccessorINS2_10foo_fields7ContextELi17EEENS2_19ServerInvokeContextINS2_11ProxyServerINS2_4test8messages12FooInterfaceEEEN5capnp11CallContextINSC_17CallFnAsyncParamsENSC_18CallFnAsyncResultsEEEEENS2_10ServerCallEJNS2_8TypeListIJEEEEEENSt9enable_ifIXsr3std7is_sameIDTclsrT_3getcldtdtfL1p1_12call_context9getParamsEEENS2_7Context6ReaderEEE5valueENS_7PromiseINT0_11CallContextEEEE4typeENS2_8PriorityILi1EEESM_RST_RKT1_DpOT2_ENUlvE_clEvEUlvE0_E3runEv /nix/store/46kiq9naswgbqfc14kc9nxcbgd0rv0m2-capnproto-1.1.0/include/kj/common.h:2007 (mptest+0x1f920d)
    #11 _ZN2kj1_8DeferredIZZN2mp9PassFieldINS2_8AccessorINS2_10foo_fields7ContextELi17EEENS2_19ServerInvokeContextINS2_11ProxyServerINS2_4test8messages12FooInterfaceEEEN5capnp11CallContextINSC_17CallFnAsyncParamsENSC_18CallFnAsyncResultsEEEEENS2_10ServerCallEJNS2_8TypeListIJEEEEEENSt9enable_ifIXsr3std7is_sameIDTclsrT_3getcldtdtfL1p1_12call_context9getParamsEEENS2_7Context6ReaderEEE5valueENS_7PromiseINT0_11CallContextEEEE4typeENS2_8PriorityILi1EEESM_RST_RKT1_DpOT2_ENUlvE_clEvEUlvE0_ED2Ev /nix/store/46kiq9naswgbqfc14kc9nxcbgd0rv0m2-capnproto-1.1.0/include/kj/common.h:1996 (mptest+0x1f920d)
    #12 _ZZN2mp9PassFieldINS_8AccessorINS_10foo_fields7ContextELi17EEENS_19ServerInvokeContextINS_11ProxyServerINS_4test8messages12FooInterfaceEEEN5capnp11CallContextINS9_17CallFnAsyncParamsENS9_18CallFnAsyncResultsEEEEENS_10ServerCallEJNS_8TypeListIJEEEEEENSt9enable_ifIXsr3std7is_sameIDTclsrT_3getcldtdtfL0p1_12call_context9getParamsEEENS_7Context6ReaderEEE5valueEN2kj7PromiseINT0_11CallContextEEEE4typeENS_8PriorityILi1EEESJ_RSR_RKT1_DpOT2_ENUlvE_clEv include/mp/type-context.h:122 (mptest+0x1f920d)

   104                      KJ_DEFER({
   105                          std::unique_lock<std::mutex> lock(thread_context.waiter->m_mutex);
   106                          // Call erase here with a Connection* argument instead
   107                          // of an iterator argument, because the `request_thread`
   108                          // iterator may be invalid if the connection is closed
   109                          // during this function call. More specifically, the
   110                          // iterator may be invalid because SetThread adds a
   111                          // cleanup callback to the Connection destructor that
   112                          // erases the thread from the map, and also because the
   113                          // ProxyServer<Thread> destructor calls
   114                          // request_threads.clear().
   115                          if (erase_thread) {
   116                              disconnected = !request_threads.erase(server.m_context.connection);
   117                          } else {
   118                              disconnected = !request_threads.count(server.m_context.connection);
   119                          }
   120                      });
   121                      fn.invoke(server_context, args...);
   122                  }

    #13 _ZN2kj8FunctionIFvvEE4ImplIZN2mp9PassFieldINS4_8AccessorINS4_10foo_fields7ContextELi17EEENS4_19ServerInvokeContextINS4_11ProxyServerINS4_4test8messages12FooInterfaceEEEN5capnp11CallContextINSE_17CallFnAsyncParamsENSE_18CallFnAsyncResultsEEEEENS4_10ServerCallEJNS4_8TypeListIJEEEEEENSt9enable_ifIXsr3std7is_sameIDTclsrT_3getcldtdtfL0p1_12call_context9getParamsEEENS4_7Context6ReaderEEE5valueENS_7PromiseINT0_11CallContextEEEE4typeENS4_8PriorityILi1EEESO_RSV_RKT1_DpOT2_EUlvE_EclEv /nix/store/46kiq9naswgbqfc14kc9nxcbgd0rv0m2-capnproto-1.1.0/include/kj/function.h:142 (mptest+0x1f8f39)
    #14 kj::Function<void ()>::operator()() /nix/store/46kiq9naswgbqfc14kc9nxcbgd0rv0m2-capnproto-1.1.0/include/kj/function.h:119 (mptest+0x154816)
    #15 void mp::Unlock<std::unique_lock<std::mutex>, kj::Function<void ()>&>(std::unique_lock<std::mutex>&, kj::Function<void ()>&) include/mp/util.h:198 (mptest+0x154816)
    #16 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0> > >::_M_run() include/mp/proxy-io.h:294 (mptest+0x22677a)
    #17 execute_native_thread_routine ??:? (libstdc++.so.6+0xed063)

  Location is heap block of size 1024 at 0x726000012c00 allocated by thread T10:
    #0 operator new(unsigned long) ??:? (mptest+0x127c3c)
    #1 mp::test::TestSetup::TestSetup(bool)::{lambda()#1}::operator()() const /nix/store/9ds850ifd4jwcccpp3v14818kk74ldf2-gcc-14.2.1.202503…
    fe4a188803
  17. mptest: fix MemorySanitizer: use-of-uninitialized-value 
    Fix MemorySanitizer: use-of-uninitialized-value error in "disconnecting and
blocking" promise::get_future call reported:

- https://github.com/Sjors/bitcoin/pull/90#issuecomment-3015153067
- https://cirrus-ci.com/task/4658925234028544and

and fixed:

- https://github.com/Sjors/bitcoin/pull/90#issuecomment-3015232660

An issue exists to add an MSAN CI job to catch errors like this more quickly in
the future https://github.com/bitcoin-core/libmultiprocess/issues/188

Error looks like:

[ TEST ] test.cpp:251: Calling IPC method, disconnecting and blocking during the call
...
MemorySanitizer: use-of-uninitialized-value
    #0 0x7f83ecb19853 in std::__1::promise<void>::get_future() /msan/llvm-project/libcxx/src/future.cpp:154:16
    #1 0x55bf563af03c in mp::test::TestCase251::run()::$_0::operator()() const /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/test/mp/test/test.cpp:280:16
    #2 0x55bf563af03c in decltype(std::declval<mp::test::TestCase251::run()::$_0&>()()) std::__1::__invoke[abi:de200100]<mp::test::TestCase251::run()::$_0&>(mp::test::TestCase251::run()::$_0&) /msan/cxx_build/include/c++/v1/__type_traits/invoke.h:179:25
    #3 0x55bf563af03c in void std::__1::__invoke_void_return_wrapper<void, true>::__call[abi:de200100]<mp::test::TestCase251::run()::$_0&>(mp::test::TestCase251::run()::$_0&) /msan/cxx_build/include/c++/v1/__type_traits/invoke.h:251:5
    #4 0x55bf563af03c in void std::__1::__invoke_r[abi:de200100]<void, mp::test::TestCase251::run()::$_0&>(mp::test::TestCase251::run()::$_0&) /msan/cxx_build/include/c++/v1/__type_traits/invoke.h:273:10
    #5 0x55bf563af03c in std::__1::__function::__alloc_func<mp::test::TestCase251::run()::$_0, std::__1::allocator<mp::test::TestCase251::run()::$_0>, void ()>::operator()[abi:de200100]() /msan/cxx_build/include/c++/v1/__functional/function.h:167:12
    #6 0x55bf563af03c in std::__1::__function::__func<mp::test::TestCase251::run()::$_0, std::__1::allocator<mp::test::TestCase251::run()::$_0>, void ()>::operator()() /msan/cxx_build/include/c++/v1/__functional/function.h:319:10
    #7 0x55bf565f1b25 in std::__1::__function::__value_func<void ()>::operator()[abi:de200100]() const /msan/cxx_build/include/c++/v1/__functional/function.h:436:12
    #8 0x55bf565f1b25 in std::__1::function<void ()>::operator()() const /msan/cxx_build/include/c++/v1/__functional/function.h:995:10
    #9 0x55bf565f1b25 in mp::test::FooImplementation::callFnAsync() /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/test/mp/test/foo.h:83:40
    #10 0x55bf565f1b25 in decltype(auto) mp::ProxyMethodTraits<mp::test::messages::FooInterface::CallFnAsyncParams, void>::invoke<mp::ServerInvokeContext<mp::ProxyServer<mp::test::messages::FooInterface>, capnp::CallContext<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>>(mp::ServerInvokeContext<mp::ProxyServer<mp::test::messages::FooInterface>, capnp::CallContext<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>&) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/include/mp/proxy.h:288:16
    #11 0x55bf565f1b25 in decltype(auto) mp::ServerCall::invoke<mp::ServerInvokeContext<mp::ProxyServer<mp::test::messages::FooInterface>, capnp::CallContext<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>>(mp::ServerInvokeContext<mp::ProxyServer<mp::test::messages::FooInterface>, capnp::CallContext<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>&, mp::TypeList<>) const /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/include/mp/proxy-types.h:448:16
    #12 0x55bf565f1b25 in std::__1::enable_if<std::is_same<decltype(mp::Accessor<mp::foo_fields::Context, 17>::get(fp1.call_context.getParams())), mp::Context::Reader>::value, kj::Promise<mp::ServerInvokeContext<mp::ProxyServer<mp::test::messages::FooInterface>, capnp::CallContext<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>::CallContext>>::type mp::PassField<mp::Accessor<mp::foo_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<mp::test::messages::FooInterface>, capnp::CallContext<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>, mp::ServerCall, mp::TypeList<>>(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<mp::test::messages::FooInterface>, capnp::CallContext<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>&, mp::ServerCall const&, mp::TypeList<>&&)::'lambda'()::operator()() /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/include/mp/type-context.h:121:24
    #13 0x55bf565f12d1 in kj::Function<void ()>::Impl<std::__1::enable_if<std::is_same<decltype(mp::Accessor<mp::foo_fields::Context, 17>::get(fp1.call_context.getParams())), mp::Context::Reader>::value, kj::Promise<mp::ServerInvokeContext<mp::ProxyServer<mp::test::messages::FooInterface>, capnp::CallContext<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>::CallContext>>::type mp::PassField<mp::Accessor<mp::foo_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<mp::test::messages::FooInterface>, capnp::CallContext<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>, mp::ServerCall, mp::TypeList<>>(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<mp::test::messages::FooInterface>, capnp::CallContext<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>&, mp::ServerCall const&, mp::TypeList<>&&)::'lambda'()>::operator()() /ci_container_base/depends/x86_64-pc-linux-gnu/include/kj/function.h:142:14
    #14 0x55bf5641125e in kj::Function<void ()>::operator()() /ci_container_base/depends/x86_64-pc-linux-gnu/include/kj/function.h:119:12
    #15 0x55bf5641125e in void mp::Unlock<std::__1::unique_lock<std::__1::mutex>, kj::Function<void ()>&>(std::__1::unique_lock<std::__1::mutex>&, kj::Function<void ()>&) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/include/mp/util.h:198:5
    #16 0x55bf5667e45b in void mp::Waiter::wait<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::'lambda'()>(std::__1::unique_lock<std::__1::mutex>&, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::'lambda'())::'lambda'()::operator()() const /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/ipc/libmultiprocess/./ipc/libmultiprocess/include/mp/proxy-io.h:294:17
    #17 0x55bf5667e45b in void std::__1::condition_variable::wait<void mp::Waiter::wait<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::'lambda'()>(std::__1::unique_lock<std::__1::mutex>&, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::'lambda'())::'lambda'()>(std::__1::unique_lock<std::__1::mutex>&, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::'lambda'()) /msan/cxx_build/include/c++/v1/__condition_variable/condition_variable.h:146:11
    #18 0x55bf5667e45b in void mp::Waiter::wait<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::'lambda'()>(std::__1::unique_lock<std::__1::mutex>&, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::'lambda'()) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/ipc/libmultiprocess/./ipc/libmultiprocess/include/mp/proxy-io.h:285:14
    #19 0x55bf5667e45b in mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/ipc/libmultiprocess/./ipc/libmultiprocess/src/mp/proxy.cpp:404:34
    #20 0x55bf5667e45b in decltype(std::declval<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>()()) std::__1::__invoke[abi:de200100]<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>(mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0&&) /msan/cxx_build/include/c++/v1/__type_traits/invoke.h:179:25
    #21 0x55bf5667e45b in void std::__1::__thread_execute[abi:de200100]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>&, std::__1::__tuple_indices<...>) /msan/cxx_build/include/c++/v1/__thread/thread.h:199:3
    #22 0x55bf5667e45b in void* std::__1::__thread_proxy[abi:de200100]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>>(void*) /msan/cxx_build/include/c++/v1/__thread/thread.h:208:3
    #23 0x7f83ec69caa3  (/lib/x86_64-linux-gnu/libc.so.6+0x9caa3) (BuildId: 42c84c92e6f98126b3e2230ebfdead22c235b667)
    #24 0x7f83ec729c3b  (/lib/x86_64-linux-gnu/libc.so.6+0x129c3b) (BuildId: 42c84c92e6f98126b3e2230ebfdead22c235b667)
[11:46:33.109]
  Member fields were destroyed
    #0 0x55bf56348abd in __sanitizer_dtor_callback_fields /msan/llvm-project/compiler-rt/lib/msan/msan_interceptors.cpp:1044:5
    #1 0x7f83ecb196de in ~promise /msan/cxx_build/include/c++/v1/future:1341:22
    #2 0x7f83ecb196de in std::__1::promise<void>::~promise() /msan/llvm-project/libcxx/src/future.cpp:151:1
    #3 0x55bf563adb36 in mp::test::TestCase251::run() /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/test/mp/test/test.cpp:293:1
    #4 0x55bf5669252e in kj::TestRunner::run()::'lambda'()::operator()() const /usr/src/kj/test.c++:318:11
    #5 0x55bf5669252e in kj::Maybe<kj::Exception> kj::runCatchingExceptions<kj::TestRunner::run()::'lambda'()>(kj::TestRunner::run()::'lambda'()&&) /usr/src/kj/exception.h:371:5
    #6 0x55bf5669071e in kj::TestRunner::run() /usr/src/kj/test.c++:318:11
    #7 0x55bf5668f977 in auto kj::TestRunner::getMain()::'lambda5'(auto&, auto&&...)::operator()<kj::TestRunner>(auto&, auto&&...) /usr/src/kj/test.c++:217:27
    #8 0x55bf5668f977 in auto kj::_::BoundMethod<kj::TestRunner&, kj::TestRunner::getMain()::'lambda5'(auto&, auto&&...), kj::TestRunner::getMain()::'lambda6'(auto&, auto&&...)>::operator()<>() /usr/src/kj/function.h:263:12
    #9 0x55bf5668f977 in kj::Function<kj::MainBuilder::Validity ()>::Impl<kj::_::BoundMethod<kj::TestRunner&, kj::TestRunner::getMain()::'lambda5'(auto&, auto&&...), kj::TestRunner::getMain()::'lambda6'(auto&, auto&&...)>>::operator()() /usr/src/kj/function.h:142:14
    #10 0x55bf56b7362c in kj::Function<kj::MainBuilder::Validity ()>::operator()() /usr/src/kj/function.h:119:12
    #11 0x55bf56b7362c in kj::MainBuilder::MainImpl::operator()(kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>) /usr/src/kj/main.c++:623:5
    #12 0x55bf56b8865c in kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>::Impl<kj::MainBuilder::MainImpl>::operator()(kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>) /usr/src/kj/function.h:142:14
    #13 0x55bf56b6a592 in kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>::operator()(kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>) /usr/src/kj/function.h:119:12
    #14 0x55bf56b6a592 in kj::runMainAndExit(kj::ProcessContext&, kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>&&, int, char**)::$_0::operator()() const /usr/src/kj/main.c++:228:5
    #15 0x55bf56b6a592 in kj::Maybe<kj::Exception> kj::runCatchingExceptions<kj::runMainAndExit(kj::ProcessContext&, kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>&&, int, char**)::$_0>(kj::runMainAndExit(kj::ProcessContext&, kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>&&, int, char**)::$_0&&) /usr/src/kj/exception.h:371:5
    #16 0x55bf56b69b5d in kj::runMainAndExit(kj::ProcessContext&, kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>&&, int, char**) /usr/src/kj/main.c++:228:5
    #17 0x55bf5668be8f in main /usr/src/kj/test.c++:381:1
    #18 0x7f83ec62a1c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 42c84c92e6f98126b3e2230ebfdead22c235b667)
    #19 0x7f83ec62a28a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 42c84c92e6f98126b3e2230ebfdead22c235b667)
    #20 0x55bf5630b374 in _start (/ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/ipc/libmultiprocess/test/mptest+0x77374)
[11:46:33.109]
SUMMARY: MemorySanitizer: use-of-uninitialized-value /msan/llvm-project/libcxx/src/future.cpp:154:16 in std::__1::promise<void>::get_future()
Exiting
    84b292fcc4
  18. ryanofsky force-pushed on Jun 28, 2025
  19. ryanofsky commented at 12:45 pm on June 28, 2025: collaborator
    Updated 258b83cdbae0cd13a98a95bc12117caf216d4993 -> 71d3107ad2b7d1a275302e239caac97e9a6d1228 (pr/tsan.3 -> pr/tsan.4, compare) tweaking fix to lock m_disconnect_cb consistently, also writing more commit documentation and adding more code comments to explain the fixes
  20. proxy-types: fix UndefinedBehaviorSanitizer: null-pointer-use 
    Fix UndefinedBehaviorSanitizer: null-pointer-use error binding a null pointer
to an unused reference variable. This error should not cause problems in
practice because the reference is not used, but is technically undefined
behavior. Issue was reported:

- https://github.com/Sjors/bitcoin/pull/90#issuecomment-3012101603
- https://github.com/Sjors/bitcoin/actions/runs/15921104847/job/44907875067?pr=90
- https://github.com/Sjors/bitcoin/pull/90#issuecomment-3015153067
- https://github.com/Sjors/bitcoin/actions/runs/15942739184/job/44973004692?pr=90

and fixed:

- https://github.com/Sjors/bitcoin/pull/90#issuecomment-3015255113

Error looks like:

[ TEST ] test.cpp:197: Call IPC method after client connection is closed
/home/runner/work/_temp/src/ipc/libmultiprocess/include/mp/proxy-types.h:612:40: runtime error: reference binding to null pointer of type 'Connection'
    #0 0x5647ad32fc50 in void mp::clientInvoke<mp::ProxyClient<mp::test::messages::FooInterface>, capnp::Request<mp::test::messages::FooInterface::AddParams, mp::test::messages::FooInterface::AddResults> (mp::test::messages::FooInterface::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::foo_fields::A, 1>, int>, mp::ClientParam<mp::Accessor<mp::foo_fields::B, 1>, int>, mp::ClientParam<mp::Accessor<mp::foo_fields::Result, 2>, int&>>(mp::ProxyClient<mp::test::messages::FooInterface>&, capnp::Request<mp::test::messages::FooInterface::AddParams, mp::test::messages::FooInterface::AddResults> (mp::test::messages::FooInterface::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::foo_fields::A, 1>, int>&&, mp::ClientParam<mp::Accessor<mp::foo_fields::B, 1>, int>&&, mp::ClientParam<mp::Accessor<mp::foo_fields::Result, 2>, int&>&&) /home/runner/work/_temp/build-asan/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/include/mp/proxy-types.h:612:25
    #1 0x5647ad32b62a in mp::ProxyClient<mp::test::messages::FooInterface>::add(int, int) /home/runner/work/_temp/build-asan/src/ipc/libmultiprocess/test/mp/test/foo.capnp.proxy-client.c++:20:5
    #2 0x5647ad2eb9ef in mp::test::TestCase197::run() /home/runner/work/_temp/build-asan/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/test/mp/test/test.cpp:206:14
    #3 0x7f6aaf7100e1  (/lib/x86_64-linux-gnu/libkj-test-1.0.1.so+0x50e1) (BuildId: 2ff7f524274168e50347a2d6dd423f273ae8d268)
    #4 0x7f6aaf710657  (/lib/x86_64-linux-gnu/libkj-test-1.0.1.so+0x5657) (BuildId: 2ff7f524274168e50347a2d6dd423f273ae8d268)
    #5 0x7f6aaf49f37f in kj::MainBuilder::MainImpl::operator()(kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>) (/lib/x86_64-linux-gnu/libkj-1.0.1.so+0x5537f) (BuildId: 4b52c0e2756bcb53e58e705fcb10bab8f63f24fd)
    #6 0x7f6aaf49a499 in kj::runMainAndExit(kj::ProcessContext&, kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>&&, int, char**) (/lib/x86_64-linux-gnu/libkj-1.0.1.so+0x50499) (BuildId: 4b52c0e2756bcb53e58e705fcb10bab8f63f24fd)
    #7 0x7f6aaf70efcb in main (/lib/x86_64-linux-gnu/libkj-test-1.0.1.so+0x3fcb) (BuildId: 2ff7f524274168e50347a2d6dd423f273ae8d268)
    #8 0x7f6aaeeb81c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 42c84c92e6f98126b3e2230ebfdead22c235b667)
    #9 0x7f6aaeeb828a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 42c84c92e6f98126b3e2230ebfdead22c235b667)
    #10 0x5647ad1fd614 in _start (/home/runner/work/_temp/build-asan/src/ipc/libmultiprocess/test/mptest+0x11d614) (BuildId: a172f78701ced3f93ad52c3562f181811a1c98e8)

SUMMARY: UndefinedBehaviorSanitizer: null-pointer-use /home/runner/work/_temp/src/ipc/libmultiprocess/include/mp/proxy-types.h:612:40
    e09143d2ea
  21. type-context: revert client disconnect workaround 
    This reverts commit 196e6fcdbc1f4d7d03f4fd05b9221917c44f5807 from
https://github.com/bitcoin-core/libmultiprocess/pull/118 which added a
workaround which is no longer needed after
315ff537fb6550468c3148e795dffb78313bd11f from
https://github.com/bitcoin-core/libmultiprocess/pull/160

When the workaround was introduced it prevented segfaults that happened when an
IPC client disconnected during a long-running asynchronous call. But it
prevented them by leaking server objects, and these leaks now sometimes cause
the new "disconnecting and blocking" unit test introduced in #160 to hang.
Since the workaround is no longer necessary, revert it now to fix the test
hangs.

The problem with test hangs was reported:

- https://github.com/Sjors/bitcoin/pull/90#issuecomment-3018191015
- https://github.com/Sjors/bitcoin/actions/runs/15966265407/job/45027248310?pr=90
- https://github.com/Sjors/bitcoin/pull/90#issuecomment-3019582045
- https://cirrus-ci.com/task/4999408900636672

And the fix was posted:

- https://github.com/Sjors/bitcoin/pull/90#issuecomment-3019738107
    c6f7fdf173
  22. ryanofsky renamed this:
    Fix threadsanitizer errors and add tsan CI job
    Fix mptest failures in bitcoin CI
    on Jun 30, 2025
  23. doc: fix DrahtBot LLM Linter error 
    Possible typos and grammar issues:

- “try unregister this callback” -> “try to unregister this callback” [missing “to” for infinitive]

https://github.com/bitcoin-core/libmultiprocess/pull/186#issuecomment-3011157858
    6f340a583f
  24. ryanofsky force-pushed on Jul 1, 2025
  25. ryanofsky referenced this in commit 725beeadc9 on Jul 1, 2025
  26. ryanofsky commented at 2:50 am on July 1, 2025: collaborator

    Updated 71d3107ad2b7d1a275302e239caac97e9a6d1228 -> 725beeadc9ae30664aa3ed12a9ac2b77a9d8d4e0 (pr/tsan.4 -> pr/tsan.5, compare) adding fixes for more errors reported https://github.com/Sjors/bitcoin/pull/90

    Will mark this ready for review. There are still followups I would like to make but they can be added separately:

    • Adding an MemorySanitizer job (#188)
    • Enabling UndefinedBehaviorSanitizer in the sanitizers job. This is a trivial change but it seems to have an interaction with ThreadSanitizer causing new tsan errors I don’t understand yet.
    • Adding a dedicated unit test to detect the 7455fc0e3781403b3e87c7b7e0f093211f63a233 hang. The existing “disconnecting and blocking” unit test sometimes catches it but usually doesn’t.
  27. ryanofsky marked this as ready for review on Jul 1, 2025
  28. in include/mp/proxy-io.h:79 in 8218a1d876 outdated 
    74@@ -75,6 +75,8 @@ struct ProxyClient<Thread> : public ProxyClientBase<Thread, ::capnp::Void>
75     //! map. It will also reset m_disconnect_cb so the destructor does not
76     //! access it. In the normal case where there is no sudden disconnect, the
77     //! destructor will unregister m_disconnect_cb so the callback is never run.
78+    //! Since this variable is accessed from multiple threads accesses should be
79+    //! guarded with the associated Waiter::m_mutex.
    Sjors commented at 9:17 am on July 1, 2025:
    8218a1d8760d94057022c5bf5a52001db205835b: is there some way to annotate this requirement so the compiler complains? (for a followup)
    ryanofsky commented at 12:03 pm on July 1, 2025:

    re: #186 (review)

    8218a1d: is there some way to annotate this requirement so the compiler complains? (for a followup)

    I’ll look into it for a followup, hopefully it’s possible. The problem I’ve had doing this is that many times the objects protected by mutexes don’t have any way of accessing the mutex variables and referring to them in GUARDED_BY(x) annotations.

    But after these fixes, I would like to go through the code and add thread safety annotations and asserts more systematically to enforce that synchronization is applied where needed.

  29. in test/mp/test/test.cpp:296 in f6bd4385c0 outdated 
    288@@ -289,6 +289,11 @@ KJ_TEST("Calling IPC method, disconnecting and blocking during the call")
289     }
290     KJ_EXPECT(disconnected);
291 
292+    // Now that the disconnect has been detected, set signal allowing the
293+    // callFnAsync() IPC call to return. Since signalling may not wake up the
294+    // thread right away, it is important for the signal variable to be declared
295+    // *before* the TestSetup variable so is available during the entire
296+    // TestSetup shutdown process.
    Sjors commented at 9:36 am on July 1, 2025:
    f6bd4385c057ce4fd0f935094940d943dbb0a8e3: what do you mean by “is available”? Is signal being captured by any of the lines below it?
    ryanofsky commented at 12:03 pm on July 1, 2025:

    re: https://github.com/bitcoin-core/libmultiprocess/pull/186/files#r2177017698

    f6bd438: what do you mean by “is available”? Is signal being captured by any of the lines below it?

    Thanks, will clarify. This is just trying to say that the signal variable should not be destroyed until after the signal.get_future().get() call in the callFnAsync execution thread is finished. Destroying the signal variable after the setup variable guarantees this because the setup destructor waits for IPC to shut down and the callFnAsync thread to finish as part of that.

    Sjors commented at 1:01 pm on July 1, 2025:
    Ah, because local variables are destructed in the reverse order they’re created.
    ryanofsky commented at 4:12 pm on July 1, 2025:

    re: #186 (review)

    Comment should be clearer now

  30. in include/mp/proxy-types.h:630 in 47f052c3ed outdated 
    629 
630         auto request = (proxy_client.m_client.*get_request)(nullptr);
631         using Request = CapRequestTraits<decltype(request)>;
632         using FieldList = typename ProxyClientMethodTraits<typename Request::Params>::Fields;
633-        IterateFields().handleChain(invoke_context, request, FieldList(), typename FieldObjs::BuildParams{&fields}...);
634+        invoke_context.emplace(*proxy_client.m_context.connection, thread_context);
    Sjors commented at 9:45 am on July 1, 2025:

    In 47f052c3ed947a7a75f17881ca632d6f0fdb7a90 proxy-types: fix UndefinedBehaviorSanitizer: null-pointer-use: since you no longer need invoke_context outside the sync([&]() { scope, why not delay defining it until you get here? It avoids the need for std::optional.

     0diff --git a/include/mp/proxy-types.h b/include/mp/proxy-types.h
 1index a2f3cd7..0f93087 100644
 2--- a/include/mp/proxy-types.h
 3+++ b/include/mp/proxy-types.h
 4@@ -610,7 +610,6 @@ void clientInvoke(ProxyClient& proxy_client, const GetRequest& get_request, Fiel
 5             << "} IPC client first request from current thread, constructing waiter";
 6     }
 7     ThreadContext& thread_context{g_thread_context};
 8-    std::optional<ClientInvokeContext> invoke_context;
 9     std::exception_ptr exception;
10     std::string kj_exception;
11     bool done = false;
12@@ -627,8 +626,8 @@ void clientInvoke(ProxyClient& proxy_client, const GetRequest& get_request, Fiel
13         auto request = (proxy_client.m_client.*get_request)(nullptr);
14         using Request = CapRequestTraits<decltype(request)>;
15         using FieldList = typename ProxyClientMethodTraits<typename Request::Params>::Fields;
16-        invoke_context.emplace(*proxy_client.m_context.connection, thread_context);
17-        IterateFields().handleChain(*invoke_context, request, FieldList(), typename FieldObjs::BuildParams{&fields}...);
18+        ClientInvokeContext invoke_context{*proxy_client.m_context.connection, thread_context};
19+        IterateFields().handleChain(invoke_context, request, FieldList(), typename FieldObjs::BuildParams{&fields}...);
20         proxy_client.m_context.loop->logPlain()
21             << "{" << thread_context.thread_name << "} IPC client send "
22             << TypeName<typename Request::Params>() << " " << LogEscape(request.toString());
23@@ -640,7 +639,7 @@ void clientInvoke(ProxyClient& proxy_client, const GetRequest& get_request, Fiel
24                     << TypeName<typename Request::Results>() << " " << LogEscape(response.toString());
25                 try {
26                     IterateFields().handleChain(
27-                        *invoke_context, response, FieldList(), typename FieldObjs::ReadResults{&fields}...);
28+                        invoke_context, response, FieldList(), typename FieldObjs::ReadResults{&fields}...);
29                 } catch (...) {
30                     exception = std::current_exception();
31                 }
    ryanofsky commented at 12:03 pm on July 1, 2025:

    re: #186 (review)

    In 47f052c proxy-types: fix UndefinedBehaviorSanitizer: null-pointer-use: since you no longer need invoke_context outside the sync([&]() { scope, why not delay defining it until you get here? It avoids the need for std::optional.

    Will add a comment. The invoke_context variable needs to be declared at the outer scope not the inner one because it can’t be destroyed before the thread_context.waiter->wait() call below which waits for the IPC response to come back from the server. This invoke_context variable needs to stay alive that long because it is referenced inside the request.send().then(...) lambda which processes the IPC response.

    ryanofsky commented at 4:10 pm on July 1, 2025:

    re: #186 (review)

    Added comment about this

  31. Sjors approved
  32. Sjors commented at 10:11 am on July 1, 2025: member

    ACK 725beeadc9ae30664aa3ed12a9ac2b77a9d8d4e0

    I understand the race condition concerns that this PR addresses, though not always the exact solution - but it’s well documented.

  33. ryanofsky commented at 12:33 pm on July 1, 2025: collaborator
    Thanks for the review! Will make a few updates here to address comments
  34. ryanofsky force-pushed on Jul 1, 2025
  35. ryanofsky commented at 4:14 pm on July 1, 2025: collaborator
    Updated 725beeadc9ae30664aa3ed12a9ac2b77a9d8d4e0 -> 6f340a583f2b29b32866c289d5f4d7662b4fe90e (pr/tsan.5 -> pr/tsan.6, compare) just improving a few comments in response to review questions
  36. Sjors commented at 5:26 pm on July 1, 2025: member
    re-ACK 6f340a583f2b29b32866c289d5f4d7662b4fe90e
  37. ryanofsky commented at 7:05 pm on July 1, 2025: collaborator

    Thanks for the reack! I will go ahead and merge this to unblock https://github.com/bitcoin/bitcoin/pull/32345 and get it out of draft.

    I’m working on the followups mentioned #186 (comment) (writing a unit test to cover the c6f7fdf1735015d069b173dd5d511ab148b678e7 fix and adding msan and ubsan jobs). Adding thread safety annotations suggested #186 (review) is another possible followup.

    Trying to write a new test for c6f7fdf1735015d069b173dd5d511ab148b678e7 made me realize the existing “disconnecting and blocking” unit test is inherently racy, and it will usually test the ProxyServer object being destroyed before the server Connection object, but sometimes tests the opposite, and it’s not easily possible to modify the test to guarantee it tests the first one. The test also isn’t covering the case charlatan suggested where the server tries to shut itself down during an IPC call #160 (review). So I’m trying to rewrite that test using promises instead of threads to cover all 3 cases deterministically instead of 1 case randomly.

  38. ryanofsky merged this on Jul 1, 2025
  39. ryanofsky closed this on Jul 1, 2025
  40. ryanofsky referenced this in commit ca509dd2b9 on Jul 1, 2025
  41. ryanofsky referenced this in commit e886c65b6b on Aug 8, 2025
  42. Sjors referenced this in commit 8ce8288ee6 on Aug 12, 2025
  43. fanquake referenced this in commit f58de8749e on Aug 18, 2025
  44. janus referenced this in commit d7ab135840 on Sep 15, 2025


ryanofsky DrahtBot Sjors

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin-core/libmultiprocess. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-12-04 19:30 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me