bitcoin-cli uncaught kj::ExceptionImpl with #32297

ryanofsky commented at 7:53 pm on June 16, 2025: collaborator

Originally posted by @pinheadmz in #182

Not sure if my observation belongs here or in a new issue. Testing https://github.com/bitcoin/bitcoin/pull/32297 I tried to use socat to inspect the IPC request:

socat -v UNIX-LISTEN:/Users/matthewzipkin/.bitcoin/regtest/node.sock STDOUT

and then make a bitcoin-cli request even though bitcoin-node was not running:

bccli -regtest getblockcount

error:
0libc++abi: terminating due to uncaught exception of type kj::ExceptionImpl: /opt/homebrew/include/kj/memory.h:258: failed: expected ptr != nullptr; null Own<> dereference
1stack: 1070007cb 10489843b 10494a81f 10494a74b 104951f4f 104951def 104951d93 104951d33 104951c73 104951c17 10494f59b 1048acccb 1048ac9d7 104ddc727 104ddc077 10488ab97 10488aa0f 10488a8e7 104889977 1a01b6f93 1a01b1d33
2Abort trap: 6
Stepping through with lldb I found the error in ConnectStream():

https://github.com/bitcoin-core/libmultiprocess/blob/019839758085cb094f107e9ac354cbda79b388e2/include/mp/proxy-io.h#L559-L580

What I believe happens is, we get a file descriptor way earlier in the stack (in bitcoin: src/ipc/interfaces.cpp::connectAddress() but by the time we get to ConnectStream() the listener has disconnected (or, just isn’t bitcoin) and then things start going wrong in ProxyClientBase()

ryanofsky commented at 7:58 pm on June 16, 2025: collaborator

It could be useful to see the backtrace from lldb, but depending on where the exception is being thrown, I think #32345 could help with this by turning the kj::Exception into an ipc::Exception that could be more easily caught in bitcoin-cli. It would make sense for the CallIPC method in https://github.com/bitcoin/bitcoin/pull/32297 to catch this exception and treat it the same as the connection failing.

pinheadmz commented at 8:01 pm on June 16, 2025: none

bt:

 0(lldb) bt
 1* thread [#2](/bitcoin-core-multiprocess/2/), name = 'b-capnp-loop', stop reason = signal SIGABRT
 2  * frame [#0](/bitcoin-core-multiprocess/0/): 0x00000001a017ea60 libsystem_kernel.dylib`__pthread_kill + 8
 3    frame [#1](/bitcoin-core-multiprocess/1/): 0x00000001a01b6c20 libsystem_pthread.dylib`pthread_kill + 288
 4    frame [#2](/bitcoin-core-multiprocess/2/): 0x00000001a00c3a30 libsystem_c.dylib`abort + 180
 5    frame [#3](/bitcoin-core-multiprocess/3/): 0x00000001a016dd08 libc++abi.dylib`abort_message + 132
 6    frame [#4](/bitcoin-core-multiprocess/4/): 0x00000001a015dfa4 libc++abi.dylib`demangling_terminate_handler() + 320
 7    frame [#5](/bitcoin-core-multiprocess/5/): 0x000000019fdfc1e0 libobjc.A.dylib`_objc_terminate() + 160
 8    frame [#6](/bitcoin-core-multiprocess/6/): 0x00000001a016d0cc libc++abi.dylib`std::__terminate(void (*)()) + 16
 9    frame [#7](/bitcoin-core-multiprocess/7/): 0x00000001a0170348 libc++abi.dylib`__cxxabiv1::failed_throw(__cxxabiv1::__cxa_exception*) + 88
10    frame [#8](/bitcoin-core-multiprocess/8/): 0x00000001a017028c libc++abi.dylib`__cxa_throw + 308
11    frame [#9](/bitcoin-core-multiprocess/9/): 0x00000001028262a0 libkj.1.1.0.dylib`kj::ExceptionCallback::RootExceptionCallback::onFatalException(kj::Exception&&) + 60
12    frame [#10](/bitcoin-core-multiprocess/10/): 0x0000000102825184 libkj.1.1.0.dylib`kj::throwFatalException(kj::Exception&&, unsigned int) + 60
13    frame [#11](/bitcoin-core-multiprocess/11/): 0x00000001028214cc libkj.1.1.0.dylib`kj::_::Debug::Fault::fatal() + 148
14    frame [#12](/bitcoin-core-multiprocess/12/): 0x00000001028207cc libkj.1.1.0.dylib`kj::_::inlineRequireFailure(char const*, int, char const*, char const*, char const*) + 108
15    frame [#13](/bitcoin-core-multiprocess/13/): 0x00000001000b843c bitcoin-cli`kj::Own<capnp::ClientHook, std::nullptr_t>::operator->(this=0x0000600002120050) at memory.h:258:28
16    frame [#14](/bitcoin-core-multiprocess/14/): 0x000000010016a820 bitcoin-cli`capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> capnp::Capability::Client::newCall<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults>(this=0x0000600002120050, interfaceId=9815814193794562661, methodId=0, sizeHint=Maybe<capnp::MessageSize> @ 0x000000016fe866b0, hints=(noPromisePipelining = false, onlyPromisePipeline = false)) at capability.h:1113:19
17    frame [#15](/bitcoin-core-multiprocess/15/): 0x000000010016a74c bitcoin-cli`ipc::capnp::messages::Init::Client::constructRequest(this=0x0000600002120048, sizeHint=Maybe<capnp::MessageSize> @ 0x000000016fe86880) at init.capnp.c++:500:10
18    frame [#16](/bitcoin-core-multiprocess/16/): 0x0000000100171f50 bitcoin-cli`void mp::clientInvoke<mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>>(this=0x000000016fdfd358)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>&&)::'lambda'()::operator()() const at proxy-types.h:620:24
19    frame [#17](/bitcoin-core-multiprocess/17/): 0x0000000100171df0 bitcoin-cli`decltype(std::declval<mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>>()()) std::__1::__invoke[abi:ne180100]<void mp::clientInvoke<mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>>(mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>&, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>&&)::'lambda'()&>(__f=0x000000016fdfd358) at invoke.h:344:25
20    frame [#18](/bitcoin-core-multiprocess/18/): 0x0000000100171d94 bitcoin-cli`std::__1::__invoke_of<void mp::clientInvoke<mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>>(mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>&, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>&&)::'lambda'()&>::type std::__1::reference_wrapper<void mp::clientInvoke<mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>>(mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>&, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>&&)::'lambda'()>::operator()[abi:ne180100]<>(this=0x000000016fdfd2a0) const at reference_wrapper.h:64:12
21    frame [#19](/bitcoin-core-multiprocess/19/): 0x0000000100171d34 bitcoin-cli`decltype(std::declval<mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>>()()) std::__1::__invoke[abi:ne180100]<std::__1::reference_wrapper<void mp::clientInvoke<mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>>(mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>&, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>&&)::'lambda'()>&>(__f=0x000000016fdfd2a0) at invoke.h:344:25
22    frame [#20](/bitcoin-core-multiprocess/20/): 0x0000000100171c74 bitcoin-cli`void std::__1::__invoke_void_return_wrapper<void, true>::__call[abi:ne180100]<std::__1::reference_wrapper<void mp::clientInvoke<mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>>(mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>&, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>&&)::'lambda'()>&>(__args=0x000000016fdfd2a0) at invoke.h:419:5
23    frame [#21](/bitcoin-core-multiprocess/21/): 0x0000000100171c18 bitcoin-cli`std::__1::__function::__alloc_func<std::__1::reference_wrapper<void mp::clientInvoke<mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>>(mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>&, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>&&)::'lambda'()>, std::__1::allocator<std::__1::reference_wrapper<void mp::clientInvoke<mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>>(mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>&, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>&&)::'lambda'()>>, void ()>::operator()[abi:ne180100](this=0x000000016fdfd2a0) at function.h:169:12
24    frame [#22](/bitcoin-core-multiprocess/22/): 0x000000010016f59c bitcoin-cli`std::__1::__function::__func<std::__1::reference_wrapper<void mp::clientInvoke<mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>>(mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>&, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>&&)::'lambda'()>, std::__1::allocator<std::__1::reference_wrapper<void mp::clientInvoke<mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>>(mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>&, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>&&)::'lambda'()>>, void ()>::operator()(this=0x000000016fdfd298) at function.h:311:10
25    frame [#23](/bitcoin-core-multiprocess/23/): 0x00000001000ccccc bitcoin-cli`std::__1::__function::__value_func<void ()>::operator()[abi:ne180100](this=0x000000016fdfd298) const at function.h:428:12
26    frame [#24](/bitcoin-core-multiprocess/24/): 0x00000001000cc9d8 bitcoin-cli`std::__1::function<void ()>::operator()(this=0x000000016fdfd298) const at function.h:981:10
27    frame [#25](/bitcoin-core-multiprocess/25/): 0x00000001005fc728 bitcoin-cli`void mp::Unlock<std::__1::unique_lock<std::__1::mutex>, std::__1::function<void ()> const&>(lock=0x000000016fe86e88, callback=0x000000016fdfd298) at util.h:146:5
28    frame [#26](/bitcoin-core-multiprocess/26/): 0x00000001005fc078 bitcoin-cli`mp::EventLoop::loop(this=0x000000012be04c88) at proxy.cpp:200:13
29    frame [#27](/bitcoin-core-multiprocess/27/): 0x00000001000aab98 bitcoin-cli`ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(this=0x0000600003425568)::'lambda'()::operator()() const at protocol.cpp:91:21
30    frame [#28](/bitcoin-core-multiprocess/28/): 0x00000001000aaa10 bitcoin-cli`decltype(std::declval<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>()()) std::__1::__invoke[abi:ne180100]<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>(__f=0x0000600003425568) at invoke.h:344:25
31    frame [#29](/bitcoin-core-multiprocess/29/): 0x00000001000aa8e8 bitcoin-cli`void std::__1::__thread_execute[abi:ne180100]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>(__t=size=2, (null)=__tuple_indices<> @ 0x000000016fe86f77) at thread.h:199:3
32    frame [#30](/bitcoin-core-multiprocess/30/): 0x00000001000a9978 bitcoin-cli`void* std::__1::__thread_proxy[abi:ne180100]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>>(__vp=0x0000600003425560) at thread.h:208:3
33    frame [#31](/bitcoin-core-multiprocess/31/): 0x00000001a01b6f94 libsystem_pthread.dylib`_pthread_start + 136

ryanofsky commented at 8:18 pm on June 16, 2025: collaborator

Thanks, so from the stack trace it doesn’t seem like any changes to https://github.com/bitcoin/bitcoin/pull/32297 need to be made to fix this because the failing IPC call is happening when the Init object is being constructed in ConnectStream calling down to the ProxyClientBase<Init> constructor.

So https://github.com/bitcoin/bitcoin/pull/32345 might be sufficient by itself to fix this, or if not the clientInvoke function needs to be improved more to handle exceptions coming from this place.

ryanofsky commented at 8:39 pm on June 16, 2025: collaborator

Looking at stack trace more, it does seem like a new fix will be required to prevent this crash. The line that is throwing is:

https://github.com/bitcoin-core/libmultiprocess/blob/019839758085cb094f107e9ac354cbda79b388e2/include/mp/proxy-types.h#L620

So probably need to add a try / catch around this line and turn it into an exception similar to 56fff76f940b5be7dc41a2532195becf8b15230e from https://github.com/bitcoin-core/libmultiprocess/pull/160

pinheadmz commented at 3:24 pm on June 17, 2025: none

For what it’s worth, I rebased https://github.com/bitcoin/bitcoin/pull/32297/ on top of https://github.com/bitcoin/bitcoin/pull/32345 and the edge case of ipcconnecting bitcoin-cli to a non-bitcoin socket failed gracefully:

0$ bccli -regtest getblockcount
1error: timeout on transient error: IPC client method called after disconnect.
2
3Probably bitcoin-node is not running or not listening on a unix socket. Can be started with:
4
5    bitcoin-node -chain=regtest -ipcbind=unix

ryanofsky commented at 3:33 pm on June 17, 2025: collaborator

Thanks for testing and that’s interesting. I’m a little surprised https://github.com/bitcoin/bitcoin/pull/32345 would fix this issue based on where the exception seemed to be coming from in the stack trace.

And either way, I would like to add unit test to cover the case of client connecting to a socket that just disconnects.

ryanofsky commented at 11:31 am on August 23, 2025: collaborator

bitcoin/bitcoin#32345 is merged but it still seems like a good idea to add a test to covers connecting to a socket that just disconnects without replying, so will keep this open

bitcoin-cli uncaught kj::ExceptionImpl with #32297 #183