test, ci: Fix threadsanitizer errors in mptest #222

pull ryanofsky wants to merge 5 commits into bitcoin-core:master from ryanofsky:pr/san changing 5 files +50 −16
  1. ryanofsky commented at 7:10 pm on October 2, 2025: collaborator

    This PR fixes threadsanitizer errors that were found in bitcoin CI and reported by maflcko in https://github.com/bitcoin/bitcoin/pull/33518#issuecomment-3360524889. It also extends the libmultiprocess sanitize CI job to report the same errors.

    Before this PR, libmultiprocess was built with threadsanitzer in CI, but cap’n proto and libc++ dependencies were built without it, so races in these libraries could be undetected.

    Building libc++ with threadsanitzer triggered threadsanitzer errors about accessing std::cout from multiple threads. Those errors are suppressed in bitcoin CI, but there is not really a reason to suppress there here, so they are just fixed by replacing std::cout with KJ_LOG in the test. This change also cleans up mptest output, which is nice. Full output can be seen by running mptest --verbose.

    Building cap’nproto with threadsanitzer triggered threadsanitzer errors in the new “thread busy” test added in #214. These were just caused by accessing some test variables from two different threads and are fixed in the last commit here.

  2. ci: Use tsan-instrumented libcxx in sanitizers job 
    mptest thread sanitizer checks that were failing in bitcoin core CI were not
failing in libmultiprocess CI, and this change instruments libcxx so the same
checks should happen both places.

Credit to MarcoFalke <*~=`'#}+{/-|&$^_@721217.xyz> for pointing this out
https://github.com/bitcoin/bitcoin/pull/33518#issuecomment-3360857978
    7eb1da120a
  3. test: Use KJ_LOG instead of std::cout for logging 
    This avoids thread sanitizer errors like:

  Read of size 8 at 0x7ffff7c81cb8 by thread T8:
    #0 std::__1::ios_base::width[abi:ne210101]() const /nix/store/71bq4557fksv43c9lf9j5ywrj6zk17hl-libcxx-21.1.1-dev/include/c++/v1/ios:497 (mptest+0x13e737)
    #1 std::__1::ostreambuf_iterator<char, std::__1::char_traits<char> > std::__1::__pad_and_output[abi:ne210101]<char, std::__1::char_traits<char> >(std::__1::ostreambuf_iterator<char, std::__1::char_traits<char> >, char const*, char const*, char const*, std::__1::ios_base&, char) /nix/store/71bq4557fksv43c9lf9j5ywrj6zk17hl-libcxx-21.1.1-dev/include/c++/v1/__locale_dir/pad_and_output.h:54 (mptest+0x13e737)
    #2 std::__1::basic_ostream<char, std::__1::char_traits<char> >& std::__1::__put_character_sequence[abi:ne210101]<char, std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, char const*, unsigned long) /nix/store/71bq4557fksv43c9lf9j5ywrj6zk17hl-libcxx-21.1.1-dev/include/c++/v1/__ostream/put_character_sequence.h:37 (mptest+0x13e5e0)
    #3 std::__1::basic_ostream<char, std::__1::char_traits<char> >& std::__1::operator<<[abi:ne210101]<std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, char const*) /nix/store/71bq4557fksv43c9lf9j5ywrj6zk17hl-libcxx-21.1.1-dev/include/c++/v1/__ostream/basic_ostream.h:438 (mptest+0x13e0ee)

  Previous write of size 8 at 0x7ffff7c81cb8 by main thread (mutexes: write M0):
    #0 std::__1::ios_base::width[abi:ne210101](long) /nix/store/71bq4557fksv43c9lf9j5ywrj6zk17hl-libcxx-21.1.1-dev/include/c++/v1/ios:501 (mptest+0x13e8bc)
    #1 std::__1::ostreambuf_iterator<char, std::__1::char_traits<char> > std::__1::__pad_and_output[abi:ne210101]<char, std::__1::char_traits<char> >(std::__1::ostreambuf_iterator<char, std::__1::char_traits<char> >, char const*, char const*, char const*, std::__1::ios_base&, char) /nix/store/71bq4557fksv43c9lf9j5ywrj6zk17hl-libcxx-21.1.1-dev/include/c++/v1/__locale_dir/pad_and_output.h:80 (mptest+0x13e8bc)
    #2 std::__1::basic_ostream<char, std::__1::char_traits<char> >& std::__1::__put_character_sequence[abi:ne210101]<char, std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, char const*, unsigned long) /nix/store/71bq4557fksv43c9lf9j5ywrj6zk17hl-libcxx-21.1.1-dev/include/c++/v1/__ostream/put_character_sequence.h:37 (mptest+0x13e5e0)
    #3 std::__1::basic_ostream<char, std::__1::char_traits<char> >& std::__1::operator<<[abi:ne210101]<std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, char const*) /nix/store/71bq4557fksv43c9lf9j5ywrj6zk17hl-libcxx-21.1.1-dev/include/c++/v1/__ostream/basic_ostream.h:438 (mptest+0x13e0ee)
    #4 mp::test::TestSetup::TestSetup(bool)::{lambda()#1}::operator()() const::{lambda(mp::LogMessage)#1}::operator()(mp::LogMessage) const test/mp/test/test.cpp:71 (mptest+0x13e0ee)
    ca3c05d567
  4. test: Fix failing exception check in new thread busy test 
    New thread busy test from 1238170f68e8fa7ae41c79465df5cdae34d568e9
(https://github.com/bitcoin-core/libmultiprocess/pull/214) is checking for
"Future already retrieved" error with string match which is fragile and does
not work on all platforms, resulting in error:

test/mp/test/test.cpp:339: error: failed: expected e.what() == std::string("Future already retrieved") [std::future_error: Future already retrieved == Future already retrieved]

This change fixes the test to check for an error code instead.

Separately there seems to be a problem with this KJ_EXPECT call because it
prints error output without causing the test to fail. This may be happening
because it is not called on the main test thread. That issue is not addressed
by this change and requires more followup.
    c332774409
  5. ci: Use tsan-instrumented cap'n proto in sanitizers job 
    This makes libmultiprocess CI output match bitcoin core CI output and catches a
race in the new thread busy test from 1238170f68e8fa7ae41c79465df5cdae34d568e9
(https://github.com/bitcoin-core/libmultiprocess/pull/214).
    b74e1bba01
  6. test: Fix tsan race in thread busy test 
    New thread busy test from 1238170f68e8fa7ae41c79465df5cdae34d568e9
(https://github.com/bitcoin-core/libmultiprocess/pull/214) was triggering tsan
race errors that happen in bitcoin ci and libmultiprocess CI (after previous
commit) reported by maflcko in
https://github.com/bitcoin/bitcoin/pull/33518#issuecomment-3360524889

The errors just happen because callback_thread and request_thread objects in
the test are accessed by two different threads: the main test thread, and the
EventLoop thread, and it is only actually safe to reference them from the
EventLoop thread.

The errors look like:

WARNING: ThreadSanitizer: data race (pid=13288)
  Write of size 4 at 0x725000000c10 by thread T13:
    #0 kj::Refcounted::disposeImpl(void*) const <null> (mptest+0x6ee095) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #1 kj::Disposer::Dispose_<capnp::ClientHook, true>::dispose(capnp::ClientHook*, kj::Disposer const&) <null> (mptest+0x2da681) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #2 void kj::Disposer::dispose<capnp::ClientHook>(capnp::ClientHook*) const <null> (mptest+0x2da5d5) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #3 kj::Own<capnp::ClientHook, std::nullptr_t>::dispose() <null> (mptest+0x2da57e) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #4 kj::Own<capnp::ClientHook, std::nullptr_t>::~Own() <null> (mptest+0x2cf135) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #5 capnp::_::(anonymous namespace)::RpcConnectionState::Export::~Export() rpc.c++ (mptest+0x32a4cc) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #6 capnp::_::(anonymous namespace)::RpcConnectionState::releaseExport(unsigned int, unsigned int) rpc.c++ (mptest+0x332ba9) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #7 capnp::_::(anonymous namespace)::RpcConnectionState::handleRelease(capnp::rpc::Release::Reader const&) rpc.c++ (mptest+0x3321ab) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #8 capnp::_::(anonymous namespace)::RpcConnectionState::handleMessage(kj::Own<capnp::IncomingRpcMessage, std::nullptr_t>) rpc.c++ (mptest+0x32eb75) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #9 capnp::_::(anonymous namespace)::RpcConnectionState::messageLoop()::'lambda'(kj::Maybe<kj::Own<capnp::IncomingRpcMessage, std::nullptr_t>>&&)::operator()(kj::Maybe<kj::Own<capnp::IncomingRpcMessage, std::nullptr_t>>&&) const rpc.c++ (mptest+0x32e75b) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #10 bool kj::_::MaybeVoidCaller<kj::Maybe<kj::Own<capnp::IncomingRpcMessage, std::nullptr_t>>, bool>::apply<capnp::_::(anonymous namespace)::RpcConnectionState::messageLoop()::'lambda'(kj::Maybe<kj::Own<capnp::IncomingRpcMessage, std::nullptr_t>>&&)>(capnp::_::(anonymous namespace)::RpcConnectionState::messageLoop()::'lambda'(kj::Maybe<kj::Own<capnp::IncomingRpcMessage, std::nullptr_t>>&&)&, kj::Maybe<kj::Own<capnp::IncomingRpcMessage, std::nullptr_t>>&&) rpc.c++ (mptest+0x393ae5) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #11 kj::_::TransformPromiseNode<bool, kj::Maybe<kj::Own<capnp::IncomingRpcMessage, std::nullptr_t>>, capnp::_::(anonymous namespace)::RpcConnectionState::messageLoop()::'lambda'(kj::Maybe<kj::Own<capnp::IncomingRpcMessage, std::nullptr_t>>&&), capnp::_::(anonymous namespace)::RpcConnectionState::messageLoop()::'lambda'(kj::Exception&&)>::getImpl(kj::_::ExceptionOrValue&) rpc.c++ (mptest+0x3936f1) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #12 kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue&)::$_0::operator()() const async.c++ (mptest+0x515123) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #13 kj::Maybe<kj::Exception> kj::runCatchingExceptions<kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue&)::$_0>(kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue&)::$_0&&) async.c++ (mptest+0x5055e7) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #14 kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue&) <null> (mptest+0x5054db) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #15 kj::_::TransformPromiseNodeBase::getDepResult(kj::_::ExceptionOrValue&) <null> (mptest+0x505892) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #16 kj::_::TransformPromiseNode<kj::_::Void, bool, capnp::_::(anonymous namespace)::RpcConnectionState::messageLoop()::'lambda'(bool), kj::_::PropagateException>::getImpl(kj::_::ExceptionOrValue&) rpc.c++ (mptest+0x395a83) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #17 kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue&)::$_0::operator()() const async.c++ (mptest+0x515123) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #18 kj::Maybe<kj::Exception> kj::runCatchingExceptions<kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue&)::$_0>(kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue&)::$_0&&) async.c++ (mptest+0x5055e7) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #19 kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue&) <null> (mptest+0x5054db) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #20 kj::TaskSet::Task::fire() <null> (mptest+0x52f703) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #21 non-virtual thunk to kj::TaskSet::Task::fire() <null> (mptest+0x52fc39) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #22 kj::EventLoop::turn() <null> (mptest+0x500065) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #23 kj::_::waitImpl(kj::Own<kj::_::PromiseNode, kj::_::PromiseDisposer>&&, kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation)::$_2::operator()() const async.c++ (mptest+0x513eb4) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #24 void kj::WaitScope::runOnStackPool<kj::_::waitImpl(kj::Own<kj::_::PromiseNode, kj::_::PromiseDisposer>&&, kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation)::$_2>(kj::_::waitImpl(kj::Own<kj::_::PromiseNode, kj::_::PromiseDisposer>&&, kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation)::$_2&&) async.c++ (mptest+0x501fd7) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #25 kj::_::waitImpl(kj::Own<kj::_::PromiseNode, kj::_::PromiseDisposer>&&, kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation) <null> (mptest+0x501b51) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #26 kj::Promise<unsigned long>::wait(kj::WaitScope&, kj::SourceLocation) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/kj/async-inl.h:1357:3 (mptest+0x2ad447) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #27 mp::EventLoop::loop() /home/admin/actions-runner/_work/_temp/build/src/ipc/libmultiprocess/./ipc/libmultiprocess/src/mp/proxy.cpp:239:68 (mptest+0x2a836e) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #28 mp::test::TestSetup::TestSetup(bool)::'lambda'()::operator()() const /home/admin/actions-runner/_work/_temp/build/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/test/mp/test/test.cpp:99:20 (mptest+0x13f6d7) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #29 std::__1::__invoke_result_impl<void, mp::test::TestSetup::TestSetup(bool)::'lambda'()>::type std::__1::__invoke[abi:de210101]<mp::test::TestSetup::TestSetup(bool)::'lambda'()>(mp::test::TestSetup::TestSetup(bool)::'lambda'()&&) /cxx_build/include/c++/v1/__type_traits/invoke.h:87:27 (mptest+0x13edee) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #30 void std::__1::__thread_execute[abi:de210101]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, mp::test::TestSetup::TestSetup(bool)::'lambda'()>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, mp::test::TestSetup::TestSetup(bool)::'lambda'()>&, std::__1::__tuple_indices<...>) /cxx_build/include/c++/v1/__thread/thread.h:159:3 (mptest+0x13edee)
    #31 void* std::__1::__thread_proxy[abi:de210101]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, mp::test::TestSetup::TestSetup(bool)::'lambda'()>>(void*) /cxx_build/include/c++/v1/__thread/thread.h:168:3 (mptest+0x13edee)

  Previous write of size 4 at 0x725000000c10 by main thread (mutexes: write M0):
    #0 kj::Own<capnp::LocalClient, std::nullptr_t> kj::Refcounted::addRefInternal<capnp::LocalClient>(capnp::LocalClient*) <null> (mptest+0x303e41) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #1 kj::Own<capnp::LocalClient, std::nullptr_t> kj::addRef<capnp::LocalClient>(capnp::LocalClient&) <null> (mptest+0x30d7e2) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #2 capnp::LocalClient::addRef() <null> (mptest+0x3059f9) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #3 capnp::Capability::Client::Client(capnp::Capability::Client&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/capnp/capability.h:1096:68 (mptest+0x13e526) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #4 mp::Thread::Client::Client(mp::Thread::Client&) /home/admin/actions-runner/_work/_temp/build/src/ipc/libmultiprocess/include/mp/proxy.capnp.h:393:3 (mptest+0x13e526)
    #5 mp::Thread::Client* std::__1::construct_at[abi:de210101]<mp::Thread::Client, mp::Thread::Client&, mp::Thread::Client*>(mp::Thread::Client*, mp::Thread::Client&) /cxx_build/include/c++/v1/__memory/construct_at.h:38:49 (mptest+0x13e526)
    #6 mp::Thread::Client* std::__1::__construct_at[abi:de210101]<mp::Thread::Client, mp::Thread::Client&, mp::Thread::Client*>(mp::Thread::Client*, mp::Thread::Client&) /cxx_build/include/c++/v1/__memory/construct_at.h:46:10 (mptest+0x13e526)
    #7 void std::__1::__optional_storage_base<mp::Thread::Client, false>::__construct[abi:de210101]<mp::Thread::Client&>(mp::Thread::Client&) /cxx_build/include/c++/v1/optional:371:5 (mptest+0x13e526)
    #8 _ZNSt3__18optionalIN2mp6Thread6ClientEEaSB8de210101IRS3_TnNS_9enable_ifIXsr4_AndINS_17integral_constantIbXntu9__is_sameu14__remove_cvrefIT_ES4_EEEENS_7_OrImplIXaantcvbsr10_IsNotSameISA_S3_EE5valuenesZT1_Li0EEE7_ResultINS8_IbXntu9__is_sameSA_S3_EEEENS_4_NotINS_9is_scalarIS3_EEEEEENS_16is_constructibleIS3_JS9_EEENS_13is_assignableIS6_S9_EEEE5valueEiE4typeELi0EEERS4_OS9_ /cxx_build/include/c++/v1/optional:744:13 (mptest+0x13e526)
    #9 mp::test::TestCase312::run() /home/admin/actions-runner/_work/_temp/build/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/test/mp/test/test.cpp:327:25 (mptest+0x136eb6) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #10 kj::TestRunner::run()::'lambda'()::operator()() const <null> (mptest+0x2bc628) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #11 kj::Maybe<kj::Exception> kj::runCatchingExceptions<kj::TestRunner::run()::'lambda'()>(kj::TestRunner::run()::'lambda'()&&) <null> (mptest+0x2b9ab7) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #12 kj::TestRunner::run() <null> (mptest+0x2b88c3) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #13 auto kj::TestRunner::getMain()::'lambda5'(auto&, auto&&...)::operator()<kj::TestRunner>(auto&, auto&&...) <null> (mptest+0x2b8161) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #14 auto kj::_::BoundMethod<kj::TestRunner&, kj::TestRunner::getMain()::'lambda5'(auto&, auto&&...), kj::TestRunner::getMain()::'lambda6'(auto&, auto&&...)>::operator()<>() <null> (mptest+0x2b80f8) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #15 kj::Function<kj::MainBuilder::Validity ()>::Impl<kj::_::BoundMethod<kj::TestRunner&, kj::TestRunner::getMain()::'lambda5'(auto&, auto&&...), kj::TestRunner::getMain()::'lambda6'(auto&, auto&&...)>>::operator()() <null> (mptest+0x2b8079) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #16 kj::Function<kj::MainBuilder::Validity ()>::operator()() <null> (mptest+0x6c9204) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #17 kj::MainBuilder::MainImpl::operator()(kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>) <null> (mptest+0x6c368d) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #18 kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>::Impl<kj::MainBuilder::MainImpl>::operator()(kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>) <null> (mptest+0x6d5e63) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #19 kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>::operator()(kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>) <null> (mptest+0x6ca206) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #20 kj::runMainAndExit(kj::ProcessContext&, kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>&&, int, char**)::$_0::operator()() const main.c++ (mptest+0x6c52d7) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #21 kj::Maybe<kj::Exception> kj::runCatchingExceptions<kj::runMainAndExit(kj::ProcessContext&, kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>&&, int, char**)::$_0>(kj::runMainAndExit(kj::ProcessContext&, kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>&&, int, char**)::$_0&&) main.c++ (mptest+0x6be577) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #22 kj::runMainAndExit(kj::ProcessContext&, kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>&&, int, char**) <null> (mptest+0x6be2db) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #23 main <null> (mptest+0x2b20d1) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)

  Location is heap block of size 512 at 0x725000000c00 allocated by thread T13:
    #0 operator new(unsigned long) <null> (mptest+0x132166) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #1 kj::Own<capnp::LocalClient, std::nullptr_t> kj::refcounted<capnp::LocalClient, kj::Own<capnp::Capability::Server, std::nullptr_t>, capnp::_::CapabilityServerSetBase&, void*&>(kj::Own<capnp::Capability::Server, std::nullptr_t>&&, capnp::_::CapabilityServerSetBase&, void*&) <null> (mptest+0x2d1a11) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #2 capnp::_::CapabilityServerSetBase::addInternal(kj::Own<capnp::Capability::Server, std::nullptr_t>&&, void*) <null> (mptest+0x2c382c) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #3 capnp::CapabilityServerSet<mp::Thread>::add(kj::Own<mp::Thread::Server, std::nullptr_t>&&) /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/capnp/capability.h:1268:10 (mptest+0x18142f) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #4 void mp::CustomBuildField<mp::StructField<mp::Accessor<mp::foo_fields::Context, 17>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>>(mp::TypeList<>, mp::Priority<1>, mp::ClientInvokeContext&, mp::StructField<mp::Accessor<mp::foo_fields::Context, 17>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>&&, std::__1::enable_if<std::is_same<decltype(fp2.get()), mp::Context::Builder>::value, void>::type*)::'lambda'()::operator()() const /home/admin/actions-runner/_work/_temp/build/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/include/mp/type-context.h:29:43 (mptest+0x1c1e51) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #5 std::__1::__invoke_result_impl<void, void mp::CustomBuildField<mp::StructField<mp::Accessor<mp::foo_fields::Context, 17>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>>(mp::TypeList<>, mp::Priority<1>, mp::ClientInvokeContext&, mp::StructField<mp::Accessor<mp::foo_fields::Context, 17>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>&&, std::__1::enable_if<std::is_same<decltype(fp2.get()), mp::Context::Builder>::value, void>::type*)::'lambda'()&>::type std::__1::__invoke[abi:de210101]<void mp::CustomBuildField<mp::StructField<mp::Accessor<mp::foo_fields::Context, 17>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>>(mp::TypeList<>, mp::Priority<1>, mp::ClientInvokeContext&, mp::StructField<mp::Accessor<mp::foo_fields::Context, 17>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>&&, std::__1::enable_if<std::is_same<decltype(fp2.get()), mp::Context::Builder>::value, void>::type*)::'lambda'()&>(void mp::CustomBuildField<mp::StructField<mp::Accessor<mp::foo_fields::Context, 17>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>>(mp::TypeList<>, mp::Priority<1>, mp::ClientInvokeContext&, mp::StructField<mp::Accessor<mp::foo_fields::Context, 17>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>&&, std::__1::enable_if<std::is_same<decltype(fp2.get()), mp::Context::Builder>::value, void>::type*)::'lambda'()&) /cxx_build/include/c++/v1/__type_traits/invoke.h:87:27 (mptest+0x1c1cb3) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #6 mp::Thread::Client std::__1::__invoke_void_return_wrapper<mp::Thread::Client, false>::__call[abi:de210101]<void mp::CustomBuildField<mp::StructField<mp::Accessor<mp::foo_fields::Context, 17>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>>(mp::TypeList<>, mp::Priority<1>, mp::ClientInvokeContext&, mp::StructField<mp::Accessor<mp::foo_fields::Context, 17>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>&&, std::__1::enable_if<std::is_same<decltype(fp2.get()), mp::Context::Builder>::value, void>::type*)::'lambda'()&>(void mp::CustomBuildField<mp::StructField<mp::Accessor<mp::foo_fields::Context, 17>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>>(mp::TypeList<>, mp::Priority<1>, mp::ClientInvokeContext&, mp::StructField<mp::Accessor<mp::foo_fields::Context, 17>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>&&, std::__1::enable_if<std::is_same<decltype(fp2.get()), mp::Context::Builder>::value, void>::type*)::'lambda'()&) /cxx_build/include/c++/v1/__type_traits/invoke.h:334:12 (mptest+0x1c1cb3)
    #7 mp::StructField<mp::Accessor<mp::foo_fields::Context, 17>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>> std::__1::__invoke_r[abi:de210101]<mp::Thread::Client, void mp::CustomBuildField<mp::StructField<mp::Accessor<mp::foo_fields::Context, 17>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>>(mp::TypeList<>, mp::Priority<1>, mp::ClientInvokeContext&, mp::StructField<mp::Accessor<mp::foo_fields::Context, 17>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>&&, std::__1::enable_if<std::is_same<decltype(fp2.get()), mp::Context::Builder>::value, void>::type*)::'lambda'()&>(void mp::CustomBuildField<mp::StructField<mp::Accessor<mp::foo_fields::Context, 17>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>>(mp::TypeList<>, mp::Priority<1>, mp::ClientInvokeContext&, mp::StructField<mp::Accessor<mp::foo_fields::Context, 17>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>&&, std::__1::enable_if<std::is_same<decltype(fp2.get()), mp::Context::Builder>::value, void>::type*)::'lambda'()&) /cxx_build/include/c++/v1/__type_traits/invoke.h:348:10 (mptest+0x1c1cb3)
    #8 std::__1::__function::__func<void mp::CustomBuildField<mp::StructField<mp::Accessor<mp::foo_fields::Context, 17>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>>(mp::TypeList<>, mp::Priority<1>, mp::ClientInvokeContext&, mp::StructField<mp::Accessor<mp::foo_fields::Context, 17>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>&&, std::__1::enable_if<std::is_same<decltype(fp2.get()), mp::Context::Builder>::value, void>::type*)::'lambda'(), mp::Thread::Client ()>::operator()() /cxx_build/include/c++/v1/__functional/function.h:174:12 (mptest+0x1c1cb3)
    #9 std::__1::__function::__value_func<mp::Thread::Client ()>::operator()[abi:de210101]() const /cxx_build/include/c++/v1/__functional/function.h:274:12 (mptest+0x2a961f) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #10 std::__1::function<mp::Thread::Client ()>::operator()() const /cxx_build/include/c++/v1/__functional/function.h:772:10 (mptest+0x2a961f)
    #11 mp::SetThread(mp::GuardedRef<std::__1::map<mp::Connection*, std::__1::optional<mp::ProxyClient<mp::Thread>>, std::__1::less<mp::Connection*>, std::__1::allocator<std::__1::pair<mp::Connection* const, std::__1::optional<mp::ProxyClient<mp::Thread>>>>>>, mp::Connection*, std::__1::function<mp::Thread::Client ()> const&) /home/admin/actions-runner/_work/_temp/build/src/ipc/libmultiprocess/./ipc/libmultiprocess/src/mp/proxy.cpp:326:32 (mptest+0x2a961f)
    #12 void mp::CustomBuildField<mp::StructField<mp::Accessor<mp::foo_fields::Context, 17>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>>(mp::TypeList<>, mp::Priority<1>, mp::ClientInvokeContext&, mp::StructField<mp::Accessor<mp::foo_fields::Context, 17>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>&&, std::__1::enable_if<std::is_same<decltype(fp2.get()), mp::Context::Builder>::value, void>::type*) /home/admin/actions-runner/_work/_temp/build/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/include/mp/type-context.h:27:31 (mptest+0x1c1554) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #13 void mp::BuildField<mp::ClientInvokeContext, mp::StructField<mp::Accessor<mp::foo_fields::Context, 17>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>>(mp::TypeList<>, mp::ClientInvokeContext&, mp::StructField<mp::Accessor<mp::foo_fields::Context, 17>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>&&) /home/admin/actions-runner/_work/_temp/build/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/include/mp/proxy-types.h:203:9 (mptest+0x1c0768) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #14 void mp::MaybeBuildField<mp::TypeList<>, mp::ClientInvokeContext&, mp::StructField<mp::Accessor<mp::foo_fields::Context, 17>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>>(std::__1::integral_constant<bool, true>, mp::TypeList<>&&, mp::ClientInvokeContext&, mp::StructField<mp::Accessor<mp::foo_fields::Context, 17>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>>&&) /home/admin/actions-runner/_work/_temp/build/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/include/mp/proxy-types.h:260:5 (mptest+0x1c0768)
    #15 auto void mp::ClientParam<mp::Accessor<mp::foo_fields::Context, 17>>::BuildParams::handleField<capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>, mp::TypeList<>>(mp::ClientInvokeContext&, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>&, mp::TypeList<>)::'lambda'<typename ...$T>($T&&...)::operator()<>($T&&...) const /home/admin/actions-runner/_work/_temp/build/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/include/mp/proxy-types.h:398:17 (mptest+0x1c0768)
    #16 std::__1::__invoke_result_impl<void, $T...>::type std::__1::__invoke[abi:de210101]<void mp::ClientParam<mp::Accessor<mp::foo_fields::Context, 17>>::BuildParams::handleField<capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>, mp::TypeList<>>(mp::ClientInvokeContext&, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>&, mp::TypeList<>)::'lambda'<typename ...$T>($T&&...) const&>($T&&...) /cxx_build/include/c++/v1/__type_traits/invoke.h:87:27 (mptest+0x1c0768)
    #17 decltype(auto) std::__1::__apply_tuple_impl[abi:de210101]<void mp::ClientParam<mp::Accessor<mp::foo_fields::Context, 17>>::BuildParams::handleField<capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>, mp::TypeList<>>(mp::ClientInvokeContext&, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>&, mp::TypeList<>)::'lambda'<typename ...$T>($T&&...) const&, std::__1::tuple<>>(capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>&&, mp::TypeList<>&&, std::__1::__tuple_indices<...>) /cxx_build/include/c++/v1/tuple:1380:5 (mptest+0x1c0768)
    #18 decltype(auto) std::__1::apply[abi:de210101]<void mp::ClientParam<mp::Accessor<mp::foo_fields::Context, 17>>::BuildParams::handleField<capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>, mp::TypeList<>>(mp::ClientInvokeContext&, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>&, mp::TypeList<>)::'lambda'<typename ...$T>($T&&...) const&, std::__1::tuple<>>(capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>&&, mp::TypeList<>&&) /cxx_build/include/c++/v1/tuple:1384:5 (mptest+0x1c0768)
    #19 void mp::ClientParam<mp::Accessor<mp::foo_fields::Context, 17>>::BuildParams::handleField<capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>, mp::TypeList<>>(mp::ClientInvokeContext&, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>&, mp::TypeList<>) /home/admin/actions-runner/_work/_temp/build/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/include/mp/proxy-types.h:409:13 (mptest+0x1c0768)
    #20 void mp::IterateFieldsHelper<mp::ClientParam<mp::Accessor<mp::foo_fields::Context, 17>>::BuildParams, 0ul>::handleChain<mp::ClientInvokeContext, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>, mp::TypeList<>>(mp::ClientInvokeContext&, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>&, mp::TypeList<>) /home/admin/actions-runner/_work/_temp/build/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/include/mp/proxy-types.h:340:38 (mptest+0x1c0768)
    #21 void mp::IterateFieldsHelper<mp::IterateFields, 0ul>::handleChain<mp::ClientInvokeContext, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>, mp::TypeList<>, mp::ClientParam<mp::Accessor<mp::foo_fields::Context, 17>>::BuildParams>(mp::ClientInvokeContext&, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults>&, mp::TypeList<>, mp::ClientParam<mp::Accessor<mp::foo_fields::Context, 17>>::BuildParams&&) /home/admin/actions-runner/_work/_temp/build/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/include/mp/proxy-types.h:333:17 (mptest+0x1c0768)
    #22 void mp::clientInvoke<mp::ProxyClient<mp::test::messages::FooInterface>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults> (mp::test::messages::FooInterface::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::foo_fields::Context, 17>>>(mp::ProxyClient<mp::test::messages::FooInterface>&, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults> (mp::test::messages::FooInterface::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::foo_fields::Context, 17>>&&)::'lambda'()::operator()() const /home/admin/actions-runner/_work/_temp/build/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/include/mp/proxy-types.h:631:25 (mptest+0x1c0768)
    #23 kj::Function<void ()>::Impl<void mp::clientInvoke<mp::ProxyClient<mp::test::messages::FooInterface>, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults> (mp::test::messages::FooInterface::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::foo_fields::Context, 17>>>(mp::ProxyClient<mp::test::messages::FooInterface>&, capnp::Request<mp::test::messages::FooInterface::CallFnAsyncParams, mp::test::messages::FooInterface::CallFnAsyncResults> (mp::test::messages::FooInterface::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::foo_fields::Context, 17>>&&)::'lambda'()>::operator()() /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/kj/function.h:142:14 (mptest+0x1c05cf) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #24 kj::Function<void ()>::operator()() /home/admin/actions-runner/_work/_temp/depends/x86_64-pc-linux-gnu/include/kj/function.h:119:12 (mptest+0x14d537) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #25 void mp::Unlock<mp::Lock, kj::Function<void ()>&>(mp::Lock&, kj::Function<void ()>&) /home/admin/actions-runner/_work/_temp/build/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/include/mp/util.h:209:5 (mptest+0x14d537)
    #26 mp::EventLoop::loop() /home/admin/actions-runner/_work/_temp/build/src/ipc/libmultiprocess/./ipc/libmultiprocess/src/mp/proxy.cpp:243:13 (mptest+0x2a8425) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #27 mp::test::TestSetup::TestSetup(bool)::'lambda'()::operator()() const /home/admin/actions-runner/_work/_temp/build/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/test/mp/test/test.cpp:99:20 (mptest+0x13f6d7) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #28 std::__1::__invoke_result_impl<void, mp::test::TestSetup::TestSetup(bool)::'lambda'()>::type std::__1::__invoke[abi:de210101]<mp::test::TestSetup::TestSetup(bool)::'lambda'()>(mp::test::TestSetup::TestSetup(bool)::'lambda'()&&) /cxx_build/include/c++/v1/__type_traits/invoke.h:87:27 (mptest+0x13edee) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #29 void std::__1::__thread_execute[abi:de210101]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, mp::test::TestSetup::TestSetup(bool)::'lambda'()>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, mp::test::TestSetup::TestSetup(bool)::'lambda'()>&, std::__1::__tuple_indices<...>) /cxx_build/include/c++/v1/__thread/thread.h:159:3 (mptest+0x13edee)
    #30 void* std::__1::__thread_proxy[abi:de210101]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, mp::test::TestSetup::TestSetup(bool)::'lambda'()>>(void*) /cxx_build/include/c++/v1/__thread/thread.h:168:3 (mptest+0x13edee)

  Mutex M0 (0x721c00003800) created at:
    #0 pthread_mutex_lock <null> (mptest+0xaee6b) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #1 __libcpp_mutex_lock /llvm-project/libcxx/include/__thread/support/pthread.h:95:10 (libc++.so.1+0x8d059) (BuildId: 41d4816981cf4a619bef1925c75c1c35d01dfd39)
    #2 std::__1::mutex::lock() /llvm-project/libcxx/src/mutex.cpp:30:12 (libc++.so.1+0x8d059)
    #3 std::__1::unique_lock<std::__1::mutex>::unique_lock[abi:de210101](std::__1::mutex&) /cxx_build/include/c++/v1/__mutex/unique_lock.h:40:11 (mptest+0x154acc) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #4 mp::Lock::Lock(mp::Mutex&) /home/admin/actions-runner/_work/_temp/build/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/include/mp/util.h:172:45 (mptest+0x154acc)
    #5 void mp::clientInvoke<mp::ProxyClient<mp::test::messages::FooInterface>, capnp::Request<mp::test::messages::FooInterface::AddParams, mp::test::messages::FooInterface::AddResults> (mp::test::messages::FooInterface::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::foo_fields::A, 1>, int>, mp::ClientParam<mp::Accessor<mp::foo_fields::B, 1>, int>, mp::ClientParam<mp::Accessor<mp::foo_fields::Result, 2>, int&>>(mp::ProxyClient<mp::test::messages::FooInterface>&, capnp::Request<mp::test::messages::FooInterface::AddParams, mp::test::messages::FooInterface::AddResults> (mp::test::messages::FooInterface::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::foo_fields::A, 1>, int>&&, mp::ClientParam<mp::Accessor<mp::foo_fields::B, 1>, int>&&, mp::ClientParam<mp::Accessor<mp::foo_fields::Result, 2>, int&>&&) /home/admin/actions-runner/_work/_temp/build/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/include/mp/proxy-types.h:669:10 (mptest+0x154acc)
    #6 mp::ProxyClient<mp::test::messages::FooInterface>::add(int, int) /home/admin/actions-runner/_work/_temp/build/src/ipc/libmultiprocess/test/mp/test/foo.capnp.proxy-client.c++:20:5 (mptest+0x152b1f) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #7 mp::test::TestCase117::run() /home/admin/actions-runner/_work/_temp/build/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/test/mp/test/test.cpp:122:5 (mptest+0x13335b) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #8 kj::TestRunner::run()::'lambda'()::operator()() const <null> (mptest+0x2bc628) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #9 kj::Maybe<kj::Exception> kj::runCatchingExceptions<kj::TestRunner::run()::'lambda'()>(kj::TestRunner::run()::'lambda'()&&) <null> (mptest+0x2b9ab7) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #10 kj::TestRunner::run() <null> (mptest+0x2b88c3) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #11 auto kj::TestRunner::getMain()::'lambda5'(auto&, auto&&...)::operator()<kj::TestRunner>(auto&, auto&&...) <null> (mptest+0x2b8161) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #12 auto kj::_::BoundMethod<kj::TestRunner&, kj::TestRunner::getMain()::'lambda5'(auto&, auto&&...), kj::TestRunner::getMain()::'lambda6'(auto&, auto&&...)>::operator()<>() <null> (mptest+0x2b80f8) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #13 kj::Function<kj::MainBuilder::Validity ()>::Impl<kj::_::BoundMethod<kj::TestRunner&, kj::TestRunner::getMain()::'lambda5'(auto&, auto&&...), kj::TestRunner::getMain()::'lambda6'(auto&, auto&&...)>>::operator()() <null> (mptest+0x2b8079) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #14 kj::Function<kj::MainBuilder::Validity ()>::operator()() <null> (mptest+0x6c9204) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #15 kj::MainBuilder::MainImpl::operator()(kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>) <null> (mptest+0x6c368d) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #16 kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>::Impl<kj::MainBuilder::MainImpl>::operator()(kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>) <null> (mptest+0x6d5e63) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #17 kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>::operator()(kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>) <null> (mptest+0x6ca206) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #18 kj::runMainAndExit(kj::ProcessContext&, kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>&&, int, char**)::$_0::operator()() const main.c++ (mptest+0x6c52d7) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #19 kj::Maybe<kj::Exception> kj::runCatchingExceptions<kj::runMainAndExit(kj::ProcessContext&, kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>&&, int, char**)::$_0>(kj::runMainAndExit(kj::ProcessContext&, kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>&&, int, char**)::$_0&&) main.c++ (mptest+0x6be577) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #20 kj::runMainAndExit(kj::ProcessContext&, kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>&&, int, char**) <null> (mptest+0x6be2db) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #21 main <null> (mptest+0x2b20d1) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)

  Thread T13 (tid=13312, running) created by main thread at:
    #0 pthread_create <null> (mptest+0xad15e) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #1 std::__1::__libcpp_thread_create[abi:de210101](unsigned long*, void* (*)(void*), void*) /cxx_build/include/c++/v1/__thread/support/pthread.h:182:10 (mptest+0x13e8f0) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #2 std::__1::thread::thread[abi:de210101]<mp::test::TestSetup::TestSetup(bool)::'lambda'(), 0>(mp::test::TestSetup::TestSetup(bool)::'lambda'()&&) /cxx_build/include/c++/v1/__thread/thread.h:213:16 (mptest+0x13e8f0)
    #3 mp::test::TestSetup::TestSetup(bool) /home/admin/actions-runner/_work/_temp/build/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/test/mp/test/test.cpp:69:11 (mptest+0x139f37) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #4 mp::test::TestCase312::run() /home/admin/actions-runner/_work/_temp/build/src/ipc/libmultiprocess/test/./ipc/libmultiprocess/test/mp/test/test.cpp:314:15 (mptest+0x136ca4) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #5 kj::TestRunner::run()::'lambda'()::operator()() const <null> (mptest+0x2bc628) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #6 kj::Maybe<kj::Exception> kj::runCatchingExceptions<kj::TestRunner::run()::'lambda'()>(kj::TestRunner::run()::'lambda'()&&) <null> (mptest+0x2b9ab7) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #7 kj::TestRunner::run() <null> (mptest+0x2b88c3) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #8 auto kj::TestRunner::getMain()::'lambda5'(auto&, auto&&...)::operator()<kj::TestRunner>(auto&, auto&&...) <null> (mptest+0x2b8161) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #9 auto kj::_::BoundMethod<kj::TestRunner&, kj::TestRunner::getMain()::'lambda5'(auto&, auto&&...), kj::TestRunner::getMain()::'lambda6'(auto&, auto&&...)>::operator()<>() <null> (mptest+0x2b80f8) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #10 kj::Function<kj::MainBuilder::Validity ()>::Impl<kj::_::BoundMethod<kj::TestRunner&, kj::TestRunner::getMain()::'lambda5'(auto&, auto&&...), kj::TestRunner::getMain()::'lambda6'(auto&, auto&&...)>>::operator()() <null> (mptest+0x2b8079) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #11 kj::Function<kj::MainBuilder::Validity ()>::operator()() <null> (mptest+0x6c9204) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #12 kj::MainBuilder::MainImpl::operator()(kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>) <null> (mptest+0x6c368d) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #13 kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>::Impl<kj::MainBuilder::MainImpl>::operator()(kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>) <null> (mptest+0x6d5e63) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #14 kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>::operator()(kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>) <null> (mptest+0x6ca206) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #15 kj::runMainAndExit(kj::ProcessContext&, kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>&&, int, char**)::$_0::operator()() const main.c++ (mptest+0x6c52d7) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #16 kj::Maybe<kj::Exception> kj::runCatchingExceptions<kj::runMainAndExit(kj::ProcessContext&, kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>&&, int, char**)::$_0>(kj::runMainAndExit(kj::ProcessContext&, kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>&&, int, char**)::$_0&&) main.c++ (mptest+0x6be577) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #17 kj::runMainAndExit(kj::ProcessContext&, kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>&&, int, char**) <null> (mptest+0x6be2db) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)
    #18 main <null> (mptest+0x2b20d1) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d)

SUMMARY: ThreadSanitizer: data race (/home/admin/actions-runner/_work/_temp/build/src/ipc/libmultiprocess/test/mptest+0x6ee095) (BuildId: 0a10180b43648715b83845ce52ff1702609db59d) in kj::Refcounted::disposeImpl(void*) const
    73d22ba2e9
  7. DrahtBot commented at 7:10 pm on October 2, 2025: none

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK Eunovo, Sjors

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #218 (Better error and log messages by ryanofsky)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  8. ryanofsky force-pushed on Oct 2, 2025
  9. ryanofsky commented at 7:32 pm on October 2, 2025: collaborator
    Updated 1cfe54149d9dbe2b42b1cd98f31004ed0a3ae047 -> 73d22ba2e9301fd77727a2193fc289e5299563bb (pr/san.1 -> pr/san.2, compare) fixing IWYU errors
  10. theuni commented at 8:08 pm on October 2, 2025: contributor

    The errors just happen because callback_thread and request_thread objects in the test are accessed by two different threads: the main test thread, and the EventLoop thread, and it is only actually safe to reference them from the EventLoop thread.

    Would it be possible to add threading annotations to enforce this contract? For ex, adding a private dummy mutex, guarding the vulnerable variables with it, and holding that mutex for the duration of the EventLoop thread? We use that trick (g_msgproc_mutex) in Core to enforce a similar contract.

    (Technically clang doesn’t require a real mutex, only a capability, so the dummy could just be an int. But as there’s no actual contention, using a real mutex seems fine imo).

  11. in test/mp/test/test.cpp:71 in ca3c05d567 outdated 
    65@@ -67,9 +66,10 @@ class TestSetup
66 
67     TestSetup(bool client_owns_connection = true)
68         : thread{[&] {
69-              EventLoop loop("mptest", [](mp::LogMessage log_data) {
70-                  std::cout << "LOG" << (int)log_data.level << ": " << log_data.message << "\n";
    theuni commented at 8:12 pm on October 2, 2025:
    Thanks for fixing this. This was a placeholder that I forgot to revisit.
  12. ryanofsky commented at 8:31 pm on October 2, 2025: collaborator

    Would it be possible to add threading annotations to enforce this contract? For ex, adding a private dummy mutex, guarding the vulnerable variables with it, and holding that mutex for the duration of the EventLoop thread? We use that trick (g_msgproc_mutex) in Core to enforce a similar contract.

    I think that trick might not apply exactly because the state here is internal to cap’nproto, and cap’nproto is single threaded and doesn’t use any thread synchronization or mutexes itself. The requirement that needs to be enforced is that any capnproto function which might trigger IO (including some capnproto object destructors in this case) need to be called from the EventLoop thread with the EventLoop::sync method.

    I imagine there could be compile time checks to enforce this (maybe using a pointer-like object overloading * and -> to wrap capnproto objects) but I don’t have a clear idea of what it would look like. Does seem worth thinking about.

  13. ryanofsky referenced this in commit 0fc31ec036 on Oct 3, 2025
  14. ryanofsky commented at 9:24 pm on October 3, 2025: collaborator

    re: #222 (comment)

    Would it be possible to add threading annotations to enforce this contract?

    Was thinking about this a little more, and I think an approach that could work could be to have a LoopObj<T> struct with T and EventLoop& members and * and -> operators returning T& and requiring requiring a TSA capability to dereference. The EventLoop object could be the capability object, or it could have a dummy member that’s is marked as a capability. Then there could be a dummy locking object to acquire the capability, just asserting the current thread is the event loop thread when constructed.

    Concretely then if the ProxyClientBase::m_client variable were declared with LoopObj<Interface::Client> type instead of Interface::Client type, the race in the #214 test wouldn’t have happened because the callback_thread and request_thread variables would be protected.

  15. ryanofsky commented at 1:34 am on October 7, 2025: collaborator

    @Sjors @theuni @Eunovo, or others it would be helpful to have at least one ACK to be able to merge this PR.

    This PR is directly blocking https://github.com/bitcoin/bitcoin/pull/33518, https://github.com/bitcoin/bitcoin/pull/33519, and https://github.com/bitcoin/bitcoin/pull/33517, and indirectly blocking backport of https://github.com/bitcoin/bitcoin/pull/33229 according to https://github.com/bitcoin/bitcoin/pull/33473#issuecomment-3370701596.

    Except for a new stringify function, this PR is just test and CI changes and I think even a partial review (for example excluding CI changes) would be helpful.

  16. Sjors commented at 8:11 am on October 7, 2025: member
    Will try to review this today or tomorrow.
  17. in shell.nix:12 in 7eb1da120a outdated 
     8 }:
 9 
10 let
11   lib  = pkgs.lib;
12-  llvm = crossPkgs.llvmPackages_20;
13+  llvmBase = crossPkgs.llvmPackages_21;
    Sjors commented at 8:21 am on October 7, 2025:
    In 7eb1da120ab6dad5fd63807eb6aeda373d767cdf ci: Use tsan-instrumented libcxx in sanitizers job: not sure if the llvm version bump is worth mentioning, but 21 is also the version we use in the Bitcoin Core TSan job.
  18. in shell.nix:51 in b74e1bba01 outdated 
    47+    env = (old.env or { }) // {
48+      CXXFLAGS =
49+        lib.concatStringsSep " " [
50+          (old.env.CXXFLAGS or "")
51+          "-fsanitize=${capnprotoSanitizers}"
52+          "-fno-omit-frame-pointer"
    Sjors commented at 8:45 am on October 7, 2025:
    In b74e1bba014d9cae34fe4b4dfe24d1530284735e ci: Use tsan-instrumented cap’n proto in sanitizers job: in Bitcoin Core -fno-omit-frame-pointer is only used with MSan, but I guess it’s fine to use with TSan too?
    Eunovo commented at 11:20 am on October 7, 2025:
    AddressSanitizer also uses it, https://clang.llvm.org/docs/AddressSanitizer.html#usage. It should be fine to leave it in, since it’s good for sanitizers
    ryanofsky commented at 12:11 pm on October 7, 2025:

    In b74e1bb ci: Use tsan-instrumented cap’n proto in sanitizers job: in Bitcoin Core -fno-omit-frame-pointer is only used with MSan, but I guess it’s fine to use with TSan too?

    Yes my understanding is it just helps provide better debug information in general, so maybe also be useful with GDB. This suggestion just came from chatgpt though, so it is good to know different sanitizer docs also recommend it. Thanks for the links!

    theuni commented at 2:43 pm on October 7, 2025:
    Yeah, this is one of the first things I do when debugging. Also necessary for anything that needs traces, like generating flamegraphs. It forces the compiler to use the frame-pointer register as-intended, rather than as a general purpose register. Abuse of that extra register is necessary in certain places (like our sha2 impl, IIRC), and useful in others to avoid stack spilling, but is generally just an optimization that’s in conflict with debugging.
  19. in shell.nix:13 in 7eb1da120a outdated 
     9 
10 let
11   lib  = pkgs.lib;
12-  llvm = crossPkgs.llvmPackages_20;
13+  llvmBase = crossPkgs.llvmPackages_21;
14+  llvm = llvmBase // lib.optionalAttrs (libcxxSanitizers != null) {
    Eunovo commented at 8:48 am on October 7, 2025:

    https://github.com/bitcoin-core/libmultiprocess/pull/222/commits/7eb1da120ab6dad5fd63807eb6aeda373d767cdf:

    Applies LLVM_USE_SANITIZER if specified. This will apply the “Thread” sanitizer specified in NIX_ARGS above.

  20. in src/mp/proxy.cpp:434 in ca3c05d567 outdated 
    430@@ -430,4 +431,16 @@ std::string LongThreadName(const char* exe_name)
431     return g_thread_context.thread_name.empty() ? ThreadName(exe_name) : g_thread_context.thread_name;
432 }
433 
434+kj::StringPtr KJ_STRINGIFY(Log v)
  21. in test/mp/test/test.cpp:338 in c332774409 outdated 
    332@@ -333,9 +333,9 @@ KJ_TEST("Make simultaneous IPC callbacks with same request_thread and callback_t
333         {
334             signal.get_future().get();
335         }
336-        catch(const std::exception& e)
337+        catch (const std::future_error& e)
338         {
339-            KJ_EXPECT(e.what() == std::string("Future already retrieved"));
340+            KJ_EXPECT(e.code() == std::make_error_code(std::future_errc::future_already_retrieved));
    Eunovo commented at 9:23 am on October 7, 2025:

    https://github.com/bitcoin-core/libmultiprocess/pull/222/commits/c332774409adfe0168636376800fbefb27b98ded:

    Not important, but I think the following is worth mentioning: Maybe we don’t need the KJ_EXPECT here at all? The signal itself is a prop for testing, and the catch block already catches the error we expect. I don’t think it will be detrimental to the test to remove the KJ_EXPECT call.

    ryanofsky commented at 12:07 pm on October 7, 2025:
    Yes I KJ_EXPECT checking for the right error code is pretty inessential to the test and would be ok to drop. I don’t think I see any downsides of keeping it though, assuming it still passes and correctly describes what’s expected to happen during the test.
    Sjors commented at 12:50 pm on October 7, 2025:
    Let’s keep the error code check unless it causes problems later. It makes the test easier to understand too.
    Eunovo commented at 1:27 pm on October 7, 2025:
    SGTM
  22. in shell.nix:45 in b74e1bba01 outdated 
    41@@ -41,7 +42,17 @@ let
42   } // (lib.optionalAttrs (lib.versionOlder capnprotoVersion "0.10") {
43     env = { }; # Drop -std=c++20 flag forced by nixpkgs
44   }));
45-  capnproto = capnprotoBase.override (lib.optionalAttrs enableLibcxx { clangStdenv = llvm.libcxxStdenv; });
46+  capnproto = (capnprotoBase.overrideAttrs (old: lib.optionalAttrs (capnprotoSanitizers != null) {
    Eunovo commented at 9:29 am on October 7, 2025:

    https://github.com/bitcoin-core/libmultiprocess/pull/222/commits/b74e1bba014d9cae34fe4b4dfe24d1530284735e:

    Adds capnprotoSanitizers if specified. This apply the “Thread” sanitizer specified in sanitize.bash.

    The flags -fno-omit-frame-pointer and -g help with producing better stack traces and showing file names and line numbers in warning messages. For example see https://clang.llvm.org/docs/ThreadSanitizer.html#usage

  23. Eunovo approved
  24. Eunovo commented at 10:01 am on October 7, 2025: contributor

    Tested ACK https://github.com/bitcoin-core/libmultiprocess/pull/222/commits/73d22ba2e9301fd77727a2193fc289e5299563bb

    I first reproduced the tsan errors by running ci_native_tsan locally, then I pulled this branch into my local Bitcoin repo with the git subtree pull command, and ran ci_native_tsan locally to see that the errors have been resolved.

    I left comments as I was reviewing, mostly explaining the code as I understood it.

  25. Sjors approved
  26. Sjors commented at 10:58 am on October 7, 2025: member

    ACK 73d22ba2e9301fd77727a2193fc289e5299563bb

    I’m not very familiar with Nix so only glossed over those changes.

    I reproduced locally that b74e1bba014d9cae34fe4b4dfe24d1530284735e fails TSan (didn’t check the error in detail) and the next commit fixes it.

  27. ryanofsky commented at 12:14 pm on October 7, 2025: collaborator
    Thanks for the reviews!
  28. ryanofsky merged this on Oct 7, 2025
  29. ryanofsky closed this on Oct 7, 2025
  30. ryanofsky referenced this in commit 5191781886 on Oct 7, 2025
  31. ryanofsky referenced this in commit 0f01e1577f on Oct 7, 2025
  32. ryanofsky referenced this in commit abcd4c4ff9 on Oct 7, 2025
  33. theuni commented at 2:36 pm on October 7, 2025: contributor
    Post-merge ACK 73d22ba2e9301fd77727a2193fc289e5299563bb. Reviewed the test changes, and the c-i changes make sense to me, but I didn’t repro before or after.
  34. Sjors referenced this in commit 3e34afe882 on Oct 7, 2025
  35. fanquake referenced this in commit becf150013 on Oct 10, 2025
  36. Sjors referenced this in commit 7e61dcfa61 on Oct 10, 2025
  37. fanquake referenced this in commit a14e7b9dee on Oct 16, 2025


ryanofsky DrahtBot theuni Sjors Eunovo

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin-core/libmultiprocess. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-12-04 19:30 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me