IPC segfault with 32345 + 29409 + custom client #176

issue ryanofsky openend this issue on May 21, 2025
  1. ryanofsky commented at 2:11 am on May 21, 2025: collaborator

    Originally posted by @darosior in https://github.com/bitcoin/bitcoin/issues/32345#issuecomment-2895479357

    I tested this PR by rebasing #29409 on top of it and running my (updated) core_bdk_wallet against it.

    I found that bitcoin-node will (still) segfault if i try to stop it while the client is running. The steps i followed were to 1) start bitcoin-node 2) start my PoC which will connect to it and stall with the open connection for 6 seconds 3) before the 6 seconds have elapsed, stop the bitcoin-node process (in this specific case by sending it a SIGINT). Here are the logs of the bitcoin-node process:

     0[...]
 12025-05-20T18:44:44Z init message: Starting network threads…
 22025-05-20T18:44:44Z DNS seeding disabled
 32025-05-20T18:44:44Z init message: Done loading
 42025-05-20T18:44:44Z initload thread start
 52025-05-20T18:44:44Z msghand thread start
 62025-05-20T18:44:44Z Loading 0 mempool transactions from file...
 72025-05-20T18:44:44Z Imported mempool transactions from file: 0 succeeded, 0 failed, 0 expired, 0 already there, 0 waiting for initial broadcast
 82025-05-20T18:44:44Z opencon thread start
 92025-05-20T18:44:44Z addcon thread start
102025-05-20T18:44:44Z net thread start
112025-05-20T18:44:44Z initload thread exit
122025-05-20T18:44:44Z New manual v2 peer connected: version: 70016, blocks=113, peer=0
132025-05-20T18:44:44Z Synchronizing blockheaders, height: 113 (~0.47%)
142025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server recv request  [#1](/bitcoin-core-multiprocess/1/) Init.construct$Params ()
152025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server send response [#1](/bitcoin-core-multiprocess/1/) Init.construct$Results (threadMap = <external capability>)
162025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server recv request  [#2](/bitcoin-core-multiprocess/2/) Init.makeChain$Params (context = (thread = <external capability>))
172025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server post request  [#2](/bitcoin-core-multiprocess/2/) {bitcoin-node-948798/b-capnp-loop-948823 (from )}
182025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server send response [#2](/bitcoin-core-multiprocess/2/) Init.makeChain$Results (result = <external capability>)
192025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server recv request  [#3](/bitcoin-core-multiprocess/3/) Chain.initMessage$Params (context = (thread = <external capability>), message = "Oxydation of the Bitcoin Core wallet in progress..")
202025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server post request  [#3](/bitcoin-core-multiprocess/3/) {bitcoin-node-948798/b-capnp-loop-948823 (from )}
212025-05-20T18:44:45Z init message: Oxydation of the Bitcoin Core wallet in progress..
222025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server send response [#3](/bitcoin-core-multiprocess/3/) Chain.initMessage$Results ()
232025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server recv request  [#4](/bitcoin-core-multiprocess/4/) Chain.showProgress$Params (context = (thread = <external capability>), title = "BDK Core startup", progress = 1, resumePossible = false)
242025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server post request  [#4](/bitcoin-core-multiprocess/4/) {bitcoin-node-948798/b-capnp-loop-948823 (from )}
252025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server send response [#4](/bitcoin-core-multiprocess/4/) Chain.showProgress$Results ()
262025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server recv request  [#5](/bitcoin-core-multiprocess/5/) Chain.getHeight$Params (context = (thread = <external capability>))
272025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server post request  [#5](/bitcoin-core-multiprocess/5/) {bitcoin-node-948798/b-capnp-loop-948823 (from )}
282025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server send response [#5](/bitcoin-core-multiprocess/5/) Chain.getHeight$Results (result = 113, hasResult = true)
292025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server recv request  [#6](/bitcoin-core-multiprocess/6/) Chain.getBlockHash$Params (context = (thread = <external capability>), height = 113)
302025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server post request  [#6](/bitcoin-core-multiprocess/6/) {bitcoin-node-948798/b-capnp-loop-948823 (from )}
312025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server send response [#6](/bitcoin-core-multiprocess/6/) Chain.getBlockHash$Results (result = "H\\3678\\307#\\020|\\020\\017\\340\\2469\\330\\356\\335\\277\\341dOB{\\220\\327\\354[\\361\\036\\233\\301\\035DD")
322025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server recv request  [#7](/bitcoin-core-multiprocess/7/) Chain.showProgress$Params (context = (thread = <external capability>), title = "BDK Core startup", progress = 100, resumePossible = true)
332025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server post request  [#7](/bitcoin-core-multiprocess/7/) {bitcoin-node-948798/b-capnp-loop-948823 (from )}
342025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server send response [#7](/bitcoin-core-multiprocess/7/) Chain.showProgress$Results ()
352025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server recv request  [#8](/bitcoin-core-multiprocess/8/) Chain.handleNotifications$Params (context = (thread = <external capability>), notifications = <external capability>)
362025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server post request  [#8](/bitcoin-core-multiprocess/8/) {bitcoin-node-948798/b-capnp-loop-948823 (from )}
372025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server send response [#8](/bitcoin-core-multiprocess/8/) Chain.handleNotifications$Results (result = <external capability>)
382025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages7HandlerEEE
39^C2025-05-20T18:44:47Z Shutdown: In progress...
402025-05-20T18:44:47Z addcon thread exit
412025-05-20T18:44:47Z opencon thread exit
422025-05-20T18:44:47Z net thread exit
432025-05-20T18:44:47Z msghand thread exit
442025-05-20T18:44:47Z scheduler thread exit
452025-05-20T18:44:47Z Writing 0 mempool transactions to file...
462025-05-20T18:44:47Z Writing 0 unbroadcast transactions to file.
472025-05-20T18:44:47Z Dumped mempool: 0.000s to copy, 0.003s to dump, 27 bytes dumped to file
482025-05-20T18:44:47Z Flushed fee estimates to fee_estimates.dat.
492025-05-20T18:44:47Z [ipc] {bitcoin-node-948798/b-shutoff-948798} IPC client first request from current thread, constructing waiter
502025-05-20T18:44:47Z [ipc] {bitcoin-node-948798/b-shutoff-948798} IPC client send ChainNotifications.chainStateFlushed$Params (context = (thread = <external capability>, callbackThread = <external capability>), role = 0, locator = "\\200\\021\\001\\000\\022H\\3678\\307#\\020|\\020\\017\\340\\2469\\330\\356\\335\\277\\341dOB{\\220\\327\\354[\\361\\036\\233\\301\\035DD\\tGm\\237]YbZDV9\\254\\252f\\205\\001\\323K\\375!\\340\\t\\367\\375\\346\\264W\\207\\000\\2101\\035:\\314\\233\\246\\275\\207\\v\\\\\\\\\\343\\202\\026\\274\\037\\213\\314\\303V\\2116<*+\\352\\255\\026\\204M\\332\\340\\nT\\306\\360Pa\\304\\002\\024\\356\\247\\220\\263\\257l\\230\\350\\362\\027Yl\\300\\316lk\\232\\f\\333\\235,\\234\\336\\204+JE\\263>\\237\\177\\305z\\017\\252\\311@\\341e\\026\\300dk\\305\\212V\\334\\377\\004cb\\217p\\n\\307w\\0046\\2261|\\024\\\\N\\336\\223G\\263~n\\271\\256\\264\\346\\227\\301\\032\\301\\030\\bw\\241Z.\\302\\261\\027 \\005\\374B\\357R\\231\\237Ct\\324\\232I\\322\\370\\303T\\253\\205\\354\\233\\337\\031C\\3215O\\3746\\016\\360b &\\300\\201\\244U\\221\\306 Pql+c\\212\\2524\\314a\\t\\216\\275=\\230b\\207ouv[\\365(\\3...
512025-05-20T18:44:47Z [ipc] {bitcoin-node-948798/b-shutoff-948798} IPC client recv ChainNotifications.chainStateFlushed$Results ()
522025-05-20T18:44:47Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages5ChainEEE
532025-05-20T18:44:47Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages4InitEEE
54Segmentation fault
  2. ryanofsky referenced this in commit 42bf3ec007 on May 21, 2025
  3. ryanofsky commented at 2:29 am on May 21, 2025: collaborator

    So far could not reproduce this combining https://github.com/bitcoin/bitcoin/pull/32345 + https://github.com/bitcoin/bitcoin/pull/29409 + https://github.com/bitcoin/bitcoin/pull/30437 and shutting down node with control-c while client is still connected.

    Code tested was: https://github.com/ryanofsky/bitcoin/commit/42bf3ec007db8ed9de7b09b30f0c0a7bc17b5bd7 (branch) and testing steps are in the commit description

    One difference between the client I tested with and the custom client reported in this bug is that apparently from the logs the custom client is calling Chain.handleNotifcations and registering for notifications. So maybe the node is running into problems if the notifications ChainNotifications object is being destroyed too late, or something like that.

    It would help to have a stack trace from the segfault. It should be easy to get a stack trace by running the node with:

    0gdb -ex run --args build/bin/bitcoin-node <...node arguments here...>

    and typing bt (backtrace) after the segfault happens

  4. darosior commented at 3:25 pm on May 21, 2025: member

    Here is the backtrace

      02025-05-21T15:22:20Z init message: Done loading
  12025-05-21T15:22:20Z msghand thread start
  22025-05-21T15:22:20Z New manual v2 peer connected: version: 70016, blocks=113, peer=0
  32025-05-21T15:22:20Z Synchronizing blockheaders, height: 113 (~0.47%)
  42025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server recv request  [#1](/bitcoin-core-multiprocess/1/) Init.construct$Params ()
  52025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server send response [#1](/bitcoin-core-multiprocess/1/) Init.construct$Results (threadMap = <external capability>)
  6[New Thread 0x7fffa8ffa6c0 (LWP 1160824)]
  72025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server recv request  [#2](/bitcoin-core-multiprocess/2/) Init.makeChain$Params (context = (thread = <external capability>))
  82025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server post request  [#2](/bitcoin-core-multiprocess/2/) {bitcoin-node-1160793/b-capnp-loop-1160824 (from )}
  92025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server send response [#2](/bitcoin-core-multiprocess/2/) Init.makeChain$Results (result = <external capability>)
 102025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server recv request  [#3](/bitcoin-core-multiprocess/3/) Chain.initMessage$Params (context = (thread = <external capability>), message = "Oxydation of the Bitcoin Core wallet in progress..")
 112025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server post request  [#3](/bitcoin-core-multiprocess/3/) {bitcoin-node-1160793/b-capnp-loop-1160824 (from )}
 122025-05-21T15:22:21Z init message: Oxydation of the Bitcoin Core wallet in progress..
 132025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server send response [#3](/bitcoin-core-multiprocess/3/) Chain.initMessage$Results ()
 142025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server recv request  [#4](/bitcoin-core-multiprocess/4/) Chain.showProgress$Params (context = (thread = <external capability>), title = "BDK Core startup", progress = 1, resumePossible = false)
 152025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server post request  [#4](/bitcoin-core-multiprocess/4/) {bitcoin-node-1160793/b-capnp-loop-1160824 (from )}
 162025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server send response [#4](/bitcoin-core-multiprocess/4/) Chain.showProgress$Results ()
 172025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server recv request  [#5](/bitcoin-core-multiprocess/5/) Chain.getHeight$Params (context = (thread = <external capability>))
 182025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server post request  [#5](/bitcoin-core-multiprocess/5/) {bitcoin-node-1160793/b-capnp-loop-1160824 (from )}
 192025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server send response [#5](/bitcoin-core-multiprocess/5/) Chain.getHeight$Results (result = 113, hasResult = true)
 202025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server recv request  [#6](/bitcoin-core-multiprocess/6/) Chain.getBlockHash$Params (context = (thread = <external capability>), height = 113)
 212025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server post request  [#6](/bitcoin-core-multiprocess/6/) {bitcoin-node-1160793/b-capnp-loop-1160824 (from )}
 222025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server send response [#6](/bitcoin-core-multiprocess/6/) Chain.getBlockHash$Results (result = "H\\3678\\307#\\020|\\020\\017\\340\\2469\\330\\356\\335\\277\\341dOB{\\220\\327\\354[\\361\\036\\233\\301\\035DD")
 232025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server recv request  [#7](/bitcoin-core-multiprocess/7/) Chain.showProgress$Params (context = (thread = <external capability>), title = "BDK Core startup", progress = 100, resumePossible = true)
 242025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server post request  [#7](/bitcoin-core-multiprocess/7/) {bitcoin-node-1160793/b-capnp-loop-1160824 (from )}
 252025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server send response [#7](/bitcoin-core-multiprocess/7/) Chain.showProgress$Results ()
 262025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server recv request  [#8](/bitcoin-core-multiprocess/8/) Chain.handleNotifications$Params (context = (thread = <external capability>), notifications = <external capability>)
 272025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server post request  [#8](/bitcoin-core-multiprocess/8/) {bitcoin-node-1160793/b-capnp-loop-1160824 (from )}
 282025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server send response [#8](/bitcoin-core-multiprocess/8/) Chain.handleNotifications$Results (result = <external capability>)
 292025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages7HandlerEEE
 30^C
 31Thread 1 "bitcoin-node" received signal SIGINT, Interrupt.
 32__GI___libc_read (nbytes=1, buf=0x7fffffffd747, fd=3) at ../sysdeps/unix/sysv/linux/read.c:26
 3326      ../sysdeps/unix/sysv/linux/read.c: No such file or directory.
 34(gdb) signal 2
 35Continuing with signal SIGINT.
 362025-05-21T15:22:24Z Shutdown: In progress...
 372025-05-21T15:22:24Z addcon thread exit
 382025-05-21T15:22:24Z opencon thread exit
 392025-05-21T15:22:24Z net thread exit
 40[Thread 0x7fffa9ffc6c0 (LWP 1160819) exited]
 412025-05-21T15:22:24Z msghand thread exit
 42[Thread 0x7fffaa7fd6c0 (LWP 1160818) exited]
 43[Thread 0x7fffaaffe6c0 (LWP 1160817) exited]
 44[Thread 0x7fffa97fb6c0 (LWP 1160820) exited]
 452025-05-21T15:22:24Z scheduler thread exit
 46[Thread 0x7ffff71ff6c0 (LWP 1160797) exited]
 472025-05-21T15:22:24Z Writing 0 mempool transactions to file...
 482025-05-21T15:22:24Z Writing 0 unbroadcast transactions to file.
 492025-05-21T15:22:24Z Dumped mempool: 0.000s to copy, 0.001s to dump, 27 bytes dumped to file
 502025-05-21T15:22:24Z Flushed fee estimates to fee_estimates.dat.
 512025-05-21T15:22:24Z [ipc] {bitcoin-node-1160793/b-shutoff-1160793} IPC client first request from current thread, constructing waiter
 522025-05-21T15:22:24Z [ipc] {bitcoin-node-1160793/b-shutoff-1160793} IPC client send ChainNotifications.chainStateFlushed$Params (context = (thread = <external capability>, callbackThread = <external capability>), role = 0, locator = "\\200\\021\\001\\000\\022H\\3678\\307#\\020|\\020\\017\\340\\2469\\330\\356\\335\\277\\341dOB{\\220\\327\\354[\\361\\036\\233\\301\\035DD\\tGm\\237]YbZDV9\\254\\252f\\205\\001\\323K\\375!\\340\\t\\367\\375\\346\\264W\\207\\000\\2101\\035:\\314\\233\\246\\275\\207\\v\\\\\\\\\\343\\202\\026\\274\\037\\213\\314\\303V\\2116<*+\\352\\255\\026\\204M\\332\\340\\nT\\306\\360Pa\\304\\002\\024\\356\\247\\220\\263\\257l\\230\\350\\362\\027Yl\\300\\316lk\\232\\f\\333\\235,\\234\\336\\204+JE\\263>\\237\\177\\305z\\017\\252\\311@\\341e\\026\\300dk\\305\\212V\\334\\377\\004cb\\217p\\n\\307w\\0046\\2261|\\024\\\\N\\336\\223G\\263~n\\271\\256\\264\\346\\227\\301\\032\\301\\030\\bw\\241Z.\\302\\261\\027 \\005\\374B\\357R\\231\\237Ct\\324\\232I\\322\\370\\303T\\253\\205\\354\\233\\337\\031C\\3215O\\3746\\016\\360b &\\300\\201\\244U\\221\\306 Pql+c\\212\\2524\\314a\\t\\216\\275=\\230b\\207ouv[\\365(\\3...
 532025-05-21T15:22:24Z [ipc] {bitcoin-node-1160793/b-shutoff-1160793} IPC client recv ChainNotifications.chainStateFlushed$Results ()
 54[Thread 0x7fffbeffd6c0 (LWP 1160814) exited]
 55[Thread 0x7fffbe7fc6c0 (LWP 1160815) exited]
 56[Thread 0x7fffbf7fe6c0 (LWP 1160813) exited]
 57[Thread 0x7fffbffff6c0 (LWP 1160812) exited]
 58[Thread 0x7fffecff96c0 (LWP 1160811) exited]
 59[Thread 0x7fffed7fa6c0 (LWP 1160810) exited]
 60[Thread 0x7fffedffb6c0 (LWP 1160809) exited]
 61[Thread 0x7fffee7fc6c0 (LWP 1160808) exited]
 62[Thread 0x7fffeeffd6c0 (LWP 1160807) exited]
 63[Thread 0x7fffef7fe6c0 (LWP 1160806) exited]
 64[Thread 0x7ffff48d56c0 (LWP 1160805) exited]
 65[Thread 0x7ffff50d66c0 (LWP 1160804) exited]
 66[Thread 0x7ffff58d76c0 (LWP 1160803) exited]
 67[Thread 0x7ffff60d86c0 (LWP 1160802) exited]
 682025-05-21T15:22:24Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages5ChainEEE
 69[Thread 0x7ffff68d96c0 (LWP 1160801) exited]
 70[Thread 0x7fffa8ffa6c0 (LWP 1160824) exited]
 712025-05-21T15:22:24Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages4InitEEE
 72[New Thread 0x7fffa8ffa6c0 (LWP 1160831)]
 732025-05-21T15:22:24Z Shutdown: done
 74
 75Thread 25 "b-capnp-loop" received signal SIGSEGV, Segmentation fault.
 76[Switching to Thread 0x7fffa8ffa6c0 (LWP 1160831)]
 77___pthread_mutex_lock (mutex=0x5550033b99fb) at ./nptl/pthread_mutex_lock.c:80
 7880      ./nptl/pthread_mutex_lock.c: No such file or directory.
 79(gdb) bt
 80[#0](/bitcoin-core-multiprocess/0/)  ___pthread_mutex_lock (mutex=0x5550033b99fb) at ./nptl/pthread_mutex_lock.c:80
 81[#1](/bitcoin-core-multiprocess/1/)  0x00005555559c35e3 in __gthread_mutex_lock (__mutex=0x5550033b99fb) at /usr/include/x86_64-linux-gnu/c++/12/bits/gthr-default.h:749
 82[#2](/bitcoin-core-multiprocess/2/)  std::mutex::lock (this=0x5550033b99fb) at /usr/include/c++/12/bits/std_mutex.h:100
 83[#3](/bitcoin-core-multiprocess/3/)  std::unique_lock<std::mutex>::lock (this=<synthetic pointer>) at /usr/include/c++/12/bits/unique_lock.h:139
 84[#4](/bitcoin-core-multiprocess/4/)  UniqueLock<AnnotatedMixin<std::mutex> >::Enter (nLine=<optimized out>, pszFile=<optimized out>, pszName=<optimized out>, this=<synthetic pointer>) at ./sync.h:163
 85[#5](/bitcoin-core-multiprocess/5/)  UniqueLock<AnnotatedMixin<std::mutex> >::UniqueLock (pszName=<optimized out>, pszFile=<optimized out>, nLine=<optimized out>, fTry=false, mutexIn=..., this=<synthetic pointer>) at ./sync.h:182
 86[#6](/bitcoin-core-multiprocess/6/)  ValidationSignalsImpl::Unregister (callbacks=0x7fff94001ee0, this=0x5550033b99fb) at ./validationinterface.cpp:59
 87[#7](/bitcoin-core-multiprocess/7/)  ValidationSignals::UnregisterValidationInterface (this=<optimized out>, callbacks=0x7fff94001ee0) at ./validationinterface.cpp:130
 88[#8](/bitcoin-core-multiprocess/8/)  0x00005555559c37e4 in ValidationSignals::UnregisterSharedValidationInterface (this=<optimized out>, callbacks=...) at ./validationinterface.cpp:125
 89[#9](/bitcoin-core-multiprocess/9/)  0x00005555557e4d10 in node::(anonymous namespace)::NotificationsHandlerImpl::disconnect (this=0x7fff94003a70) at ./node/interfaces.cpp:497
 90[#10](/bitcoin-core-multiprocess/10/) node::(anonymous namespace)::NotificationsHandlerImpl::~NotificationsHandlerImpl (this=<optimized out>, __in_chrg=<optimized out>) at ./node/interfaces.cpp:493
 91[#11](/bitcoin-core-multiprocess/11/) node::(anonymous namespace)::NotificationsHandlerImpl::~NotificationsHandlerImpl (this=0x7fff94003a70, __in_chrg=<optimized out>) at ./node/interfaces.cpp:493
 92[#12](/bitcoin-core-multiprocess/12/) 0x0000555555b53ed9 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x7fff940035e0) at /usr/include/c++/12/bits/shared_ptr_base.h:346
 93[#13](/bitcoin-core-multiprocess/13/) std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x7fff940035e0) at /usr/include/c++/12/bits/shared_ptr_base.h:317
 94[#14](/bitcoin-core-multiprocess/14/) std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/12/bits/shared_ptr_base.h:1071
 95[#15](/bitcoin-core-multiprocess/15/) std::__shared_ptr<interfaces::Handler, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/12/bits/shared_ptr_base.h:1524
 96[#16](/bitcoin-core-multiprocess/16/) std::__shared_ptr<interfaces::Handler, (__gnu_cxx::_Lock_policy)2>::reset (this=0x7fffe800c6d0) at /usr/include/c++/12/bits/shared_ptr_base.h:1642
 97[#17](/bitcoin-core-multiprocess/17/) mp::ProxyServerBase<ipc::capnp::messages::Handler, interfaces::Handler>::~ProxyServerBase()::{lambda()#1}::operator()() (__closure=0x7fffe800c6d0) at ./ipc/libmultiprocess/include/mp/proxy-io.h:478
 98[#18](/bitcoin-core-multiprocess/18/) std::__invoke_impl<void, mp::ProxyServerBase<ipc::capnp::messages::Handler, interfaces::Handler>::~ProxyServerBase()::{lambda()#1}&>(std::__invoke_other, mp::ProxyServerBase<ipc::capnp::messages::Handler, interfaces::Handler>::~ProxyServerBase()::{lambda()#1}&) (__f=...) at /usr/include/c++/12/bits/invoke.h:61
 99[#19](/bitcoin-core-multiprocess/19/) std::__invoke_r<void, mp::ProxyServerBase<ipc::capnp::messages::Handler, interfaces::Handler>::~ProxyServerBase()::{lambda()#1}&>(mp::ProxyServerBase<ipc::capnp::messages::Handler, interfaces::Handler>::~ProxyServerBase()::{lambda()#1}&) (__fn=...) at /usr/include/c++/12/bits/invoke.h:111
100[#20](/bitcoin-core-multiprocess/20/) std::_Function_handler<void (), mp::ProxyServerBase<ipc::capnp::messages::Handler, interfaces::Handler>::~ProxyServerBase()::{lambda()#1}>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/include/c++/12/bits/std_function.h:290
101[#21](/bitcoin-core-multiprocess/21/) 0x0000555555f403c2 in std::function<void ()>::operator()() const (this=0x7fffa8ff9d20) at /usr/include/c++/12/bits/std_function.h:591
102[#22](/bitcoin-core-multiprocess/22/) mp::Unlock<mp::Lock, std::function<void ()> const&>(mp::Lock&, std::function<void ()> const&) (callback=..., lock=...) at ./ipc/libmultiprocess/include/mp/util.h:201
103[#23](/bitcoin-core-multiprocess/23/) operator() (__closure=<optimized out>) at ./ipc/libmultiprocess/src/mp/proxy.cpp:282
104[#24](/bitcoin-core-multiprocess/24/) std::__invoke_impl<void, mp::EventLoop::startAsyncThread()::<lambda()> > (__f=...) at /usr/include/c++/12/bits/invoke.h:61
105[#25](/bitcoin-core-multiprocess/25/) std::__invoke<mp::EventLoop::startAsyncThread()::<lambda()> > (__fn=...) at /usr/include/c++/12/bits/invoke.h:96
106[#26](/bitcoin-core-multiprocess/26/) std::thread::_Invoker<std::tuple<mp::EventLoop::startAsyncThread()::<lambda()> > >::_M_invoke<0> (this=<optimized out>) at /usr/include/c++/12/bits/std_thread.h:252
107[#27](/bitcoin-core-multiprocess/27/) std::thread::_Invoker<std::tuple<mp::EventLoop::startAsyncThread()::<lambda()> > >::operator() (this=<optimized out>) at /usr/include/c++/12/bits/std_thread.h:259
108[#28](/bitcoin-core-multiprocess/28/) std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::EventLoop::startAsyncThread()::<lambda()> > > >::_M_run(void) (this=0x7fffe800cc00) at /usr/include/c++/12/bits/std_thread.h:210
109[#29](/bitcoin-core-multiprocess/29/) 0x00007ffff78d44a3 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
110[#30](/bitcoin-core-multiprocess/30/) 0x00007ffff75c81f5 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
111[#31](/bitcoin-core-multiprocess/31/) 0x00007ffff764889c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
  5. ryanofsky commented at 3:29 pm on May 21, 2025: collaborator
    Thanks! I also just reproduced this and got a similar backtrace with: https://github.com/ryanofsky/bitcoin/commit/7d6a1087900a98bd5b16a48d7ea9f0d352851023 (branch)
  6. ryanofsky commented at 4:15 pm on May 21, 2025: collaborator

    So commit https://github.com/ryanofsky/bitcoin/commit/c93c9852d6408a3022e346f457c730c08c175b02 (branch) fixes the problem for me.

    It should be possible to cherry-pick it or just apply the diff https://github.com/ryanofsky/bitcoin/commit/c93c9852d6408a3022e346f457c730c08c175b02.diff to test.

    I’m not very satisfied with this fix because it is not really compatible with the way bitcoin-gui works in https://github.com/bitcoin/bitcoin/pull/10102 (details in commit description), but assuming it solves current problems, it is a clean fix.

  7. darosior commented at 5:15 pm on May 21, 2025: member

    Can confirm it’s fixed. Are we sure it only happens in this specific instance and there should be no other way the order of destruction can lead to a crash? Can i continue testing with this commit applied or should i wait till you’ve updated your PR?

     02025-05-21T17:13:00Z [ipc] {bitcoin-node-1168991/b-capnp-loop-1168993} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages7HandlerEEE
 1^C2025-05-21T17:13:02Z Shutdown: In progress...
 22025-05-21T17:13:02Z addcon thread exit
 32025-05-21T17:13:02Z opencon thread exit
 42025-05-21T17:13:02Z net thread exit
 52025-05-21T17:13:02Z msghand thread exit
 62025-05-21T17:13:02Z scheduler thread exit
 72025-05-21T17:13:02Z Writing 0 mempool transactions to file...
 82025-05-21T17:13:02Z Writing 0 unbroadcast transactions to file.
 92025-05-21T17:13:02Z Dumped mempool: 0.000s to copy, 0.003s to dump, 27 bytes dumped to file
102025-05-21T17:13:02Z Flushed fee estimates to fee_estimates.dat.
112025-05-21T17:13:02Z [ipc] {bitcoin-node-1168991/b-shutoff-1168991} IPC client first request from current thread, constructing waiter
122025-05-21T17:13:02Z [ipc] {bitcoin-node-1168991/b-shutoff-1168991} IPC client send ChainNotifications.chainStateFlushed$Params (context = (thread = <external capability>, callbackThread = <external capability>), role = 0, locator = "\\200\\021\\001\\000\\022H\\3678\\307#\\020|\\020\\017\\340\\2469\\330\\356\\335\\277\\341dOB{\\220\\327\\354[\\361\\036\\233\\301\\035DD\\tGm\\237]YbZDV9\\254\\252f\\205\\001\\323K\\375!\\340\\t\\367\\375\\346\\264W\\207\\000\\2101\\035:\\314\\233\\246\\275\\207\\v\\\\\\\\\\343\\202\\026\\274\\037\\213\\314\\303V\\2116<*+\\352\\255\\026\\204M\\332\\340\\nT\\306\\360Pa\\304\\002\\024\\356\\247\\220\\263\\257l\\230\\350\\362\\027Yl\\300\\316lk\\232\\f\\333\\235,\\234\\336\\204+JE\\263>\\237\\177\\305z\\017\\252\\311@\\341e\\026\\300dk\\305\\212V\\334\\377\\004cb\\217p\\n\\307w\\0046\\2261|\\024\\\\N\\336\\223G\\263~n\\271\\256\\264\\346\\227\\301\\032\\301\\030\\bw\\241Z.\\302\\261\\027 \\005\\374B\\357R\\231\\237Ct\\324\\232I\\322\\370\\303T\\253\\205\\354\\233\\337\\031C\\3215O\\3746\\016\\360b &\\300\\201\\244U\\221\\306 Pql+c\\212\\2524\\314a\\t\\216\\275=\\230b\\207ouv[\\365(\\3...
132025-05-21T17:13:02Z [ipc] {bitcoin-node-1168991/b-shutoff-1168991} IPC client recv ChainNotifications.chainStateFlushed$Results ()
142025-05-21T17:13:02Z [ipc] {bitcoin-node-1168991/b-capnp-loop-1168993} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages5ChainEEE
152025-05-21T17:13:02Z [ipc] {bitcoin-node-1168991/b-capnp-loop-1168993} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages4InitEEE
162025-05-21T17:13:02Z [ipc] {bitcoin-node-1168991/b-capnp-loop-1168993} EventLoop::loop done, cancelling event listeners.
172025-05-21T17:13:02Z [ipc] {bitcoin-node-1168991/b-capnp-loop-1168993} EventLoop::loop bye.
182025-05-21T17:13:02Z Shutdown: done
  8. ryanofsky commented at 5:43 pm on May 21, 2025: collaborator

    Are we sure it only happens in this specific instance and there should be no other way the order of destruction can lead to a crash? Can i continue testing with this commit applied or should i wait till you’ve updated your PR?

    This fix should be a pretty general fix because it just waits for all objects associated with any IPC connection to be destroyed before freeing resources they could be referencing. So I think if you continued to test it you would not find similar problems. To be clear, IPC connections using the Chain interface are pretty trusted, so there are probably other things IPC clients could do using it to crash the node. But the fix should address any shutdown issues.

    I’m not sure yet if I will update https://github.com/bitcoin/bitcoin/pull/32345 with this specific fix or a more general one that could work better with later changes in https://github.com/bitcoin/bitcoin/pull/10102.

    The problem in https://github.com/bitcoin/bitcoin/pull/10102 is that bitcoin-node can be spawned by a parent bitcoin-gui process there. To work correctly, it should be able to keep its connection to the bitcoin-gui process alive while destroying other connections. I think to allow this, Connection objects in libmultiprocess could have a new bool m_controlling field indicating whether they are “controlling” connections that should not be disconnected. Then in addition to the EventLoop::m_num_cients reference count incremented by normal connections, there could be an EventLoop::m_num_cients_controlling reference count incremented by controlling connections. Then, when bitcoin-node is shutting down, it could disconnect all non-controlling connections, wait for m_num_cients == m_num_cients_controlling to be true, free all resources the non-controlling connections could have been referencing, and the finally disconnect the controlling connection. Implementing this should not be too complicated and it should be behave same way as the current fix, while being compatible with https://github.com/bitcoin/bitcoin/pull/10102

  9. ryanofsky commented at 8:08 pm on May 22, 2025: collaborator

    Connection objects in libmultiprocess could have a new bool m_controlling field indicating whether they are “controlling” connections that should not be disconnected.

    Thinking about it more, a different and more general approach would probably be simpler.

    Instead of creating a distinction between controlling and non-controlling connections, and adding way to wait for objects associated with non-controlling connections to be freed after they are disconnected, it would make more sense to just add a general way to wait for objects associated with any connection to be freed after disconnection.

    This should be pretty straightforward to implement. Instead of having EventLoop::m_num_clients and EventLoop::m_num_clients_controlling reference counts there could be EventLoop::m_num_clients and Connection::m_num_clients reference counts so objects associated with each connection are tracked separately. This will require another change to make Connection objects outlive all the client and server objects associated with them which is not currently the case. But that should not be too hard to implement.

  10. ryanofsky referenced this in commit 8d62ec743e on Jun 5, 2025
  11. ryanofsky commented at 5:42 pm on June 5, 2025: collaborator

    Thanks for your patience. I went on a real journey with this implementing the connection refcounting idea described above before figuring out that a much simpler fix is possible, because cap’n proto provides better guarantees about when objects will be destroyed than I was assuming.

    I updated https://github.com/bitcoin/bitcoin/pull/32345 with the simpler fix, and fix itself is commit https://github.com/ryanofsky/bitcoin/commit/94db35300d3ca2cc54344e059396646ce6c1ebbe (branch).

    That branch also contains combined code from https://github.com/bitcoin/bitcoin/pull/32345, https://github.com/bitcoin/bitcoin/pull/29409, and https://github.com/bitcoin/bitcoin/pull/30437 which I was using to test this issue.

    The unused code I had written to the implement the connection refcounting idea is https://github.com/ryanofsky/libmultiprocess/commit/a35138acc6003e86a84e61cbc81b7665e02e90b4 and I will probably try to port over some of the documentation and organizational improvements it makes in a separate PR. I might keep the refcounting implementation too since in some ways it makes code simpler, and having separate connection counts instead of a global count could allow adding more asserts and make problems easier to debug.

    Update: I wound up doing work on the connection refcounting idea to fix another problem in #182, before abandoning it again after finding a simpler solution to that problem as well. Current implementation is c3a85ae83bd279b27799decd003367c53adb524b (tag) and I will probably clean it up and turn it into a followup PR

  12. ryanofsky referenced this in commit 87b2ae6314 on Jun 10, 2025
  13. ryanofsky referenced this in commit 258a617c1e on Jun 19, 2025
  14. fanquake referenced this in commit f58de8749e on Aug 18, 2025
  15. ryanofsky commented at 11:27 am on August 23, 2025: collaborator
    The fix above https://github.com/ryanofsky/bitcoin/commit/94db35300d3ca2cc54344e059396646ce6c1ebbe was merged in bitcoin/bitcoin#32345
  16. ryanofsky closed this on Aug 23, 2025


ryanofsky darosior

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin-core/libmultiprocess. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-12-04 19:30 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me