should concurrent IPC requests directed to the same thread cause a crash?

plebhash commented at 10:05 pm on November 21, 2025: none

recently I reported #33554, which was fixed by the introduction of interruptWait via #33575

then I noticed crashes for similar reasons (although unrelated to waitNext), which I explain in detail here: https://github.com/stratum-mining/sv2-apps/issues/88#issuecomment-3558003166

TLDR: I was trying to do a getBlock while a submitSolution was happening at the same time, both against the same Bitcoin Core thread

this made me realize that regardless of whether the call is going to block for a long time (like waitNext), I simply cannot leave room for concurrent calls against the same Bitcoin Core thread

which feels a bit contradictory to what @ryanofsky suggested here: #33554 (comment)

More broadly, I think I’d try just having a c++ “waiter” thread dedicated to running waitNext calls, and a c++ “worker” thread to run all other IPC operations besides waitNext. The “waiter” thread should be mostly blocked monitoring mempool and network, and the “worker” thread should be available to do anything else you might need like fetching and submitting block data.

and it’s making me re-evaluate some assumptions I had when I started https://github.com/stratum-mining/sv2-apps/pull/59

overall, I’m getting to a place where I’m having to go great lengths in terms of defensive programming just to avoid crashing Bitcoin Core

I’m still evaluating the trade-offs of potential defensive strategies, namely:

having synchronization primitives that avoid overlapping requests against the same Bitcoin Core thread
instantiating multiple Bitcoin Core threads, one dedicated for each potentially concurrent request

but I digress… the main point I wanted to articulate here was: shouldn’t Bitcoin Core simply not crash if two requests arrive for the same thread?

ideally, a naive client should simply be forced to wait a bit longer if they send concurrent requests against the same Bitcoin Core thread, instead of being forced to avoid doing this at all cost because it could cause havoc on the other side of the IPC connection

ryanofsky commented at 11:29 pm on November 21, 2025: contributor

shouldn’t Bitcoin Core simply not crash if two requests arrive for the same thread?

Yes this shouldn’t crash. Hopefully, if you are using a version of bitcoin after https://github.com/bitcoin-core/libmultiprocess/pull/214 (part of #33518 in master and part of #33519 in 30.x branch) you should just see “thread busy” errors instead of crashes, and it is a bug if there are still crashes.

ideally, a naive client should simply be forced to wait a bit longer if they send concurrent requests against the same Bitcoin Core thread

This is reasonable and would be pretty easy to implement, but it raises the question of how many requests should be allowed? Should it allow 5, 10, or 100 requests before returning “thread busy”? Or an arbitrary amount? Or a configurable amount?

My thinking has been that if a client needs a request queue, it is probably better for the client implement the queue, than to have the server provide one because the client should know its own needs better. Also, not every client needs a queue. For example, Sjor’s SV2 client is multithreaded and only makes blocking requests. It just creates one server thread for each client thread, so every server thread is guaranteed to not be busy when a request comes in.

Since the rust client is single threaded and nonblocking, it needs to create different server threads for those requests to execute in parallel like the c++ client does, or it needs to have a queue so requests created in parallel can be executed serially on a single server thread.

In practice, I think it make sense for the rust client to just create 2 server threads: a waiting thread to execute waitNext() calls, and a worker thread using a queue to execute all other calls.

It if would help, I could add a enableRequestQueue() or similar method to the Thread interface so the rust client can ask for the requests sent to a thread to be queued.

However, I think it would probably be straightforward to implement a queue in rust on the client side instead. And I wouldn’t want queuing to be enabled on the server by default because it could mask other problems. Like if a request was accidentally sent to the waiting thread rather than the worker thread, it could just sit there forever if queuing was enabled on the waiting thread, instead of returning an error.

Thanks for reporting the problem here and I hope this can lead to a good solution.

plebhash commented at 4:18 pm on November 22, 2025: none

I guess the main thing I wanted to report here is a duplicate of https://github.com/bitcoin-core/libmultiprocess/pull/214, so it’s good to know these crashes won’t happen on v30.x!

how many requests should be allowed? Should it allow 5, 10, or 100 requests before returning “thread busy”? Or an arbitrary amount? Or a configurable amount?

it would be up to the client to know what are the reasonable bounds for backpressure so that it can avoid the need to handle errors, so I’d say that a configurable amount with a reasonably large default

but since you also mentioned:

And I wouldn’t want queuing to be enabled on the server by default because it could mask other problems.

I wouldn’t necessarily frame this as an actual feature request, mainly because we’re working around this without having to resort to server-side queueing.

Since the rust client is single threaded and nonblocking, it needs to create different server threads for those requests to execute in parallel like the c++ client does, or it needs to have a queue so requests created in parallel can be executed serially on a single server thread.

currently we’re going with the first approach, so now we create 5 server threads, one dedicated for each potentially concurrent call:

submitSolution
getHeader
getCoinbaseTx
getCoinbaseMerklePath
waitNext

plus some ad-hoc threads that are created ephemerally whenever we need to do a getBlock (either to reply to a RequestTransactionData or because we want to dump a valid solution to disk)

to be honest, we could have gotten away with only 2 by forcing getHeader, getCoinbaseTx, getCoinbaseMerklePath and waitNext to be done sequentially under the same server thread

but we’re trying to squeeze out every possible micro/nano/pico second out of the way with concurrent execution of getHeader, getCoinbaseTx, getCoinbaseMerklePath so we can craft a NewTemplate message ASAP (the ones for mempool updates are actually rate-limited, so we’re not really in a race… but for chain tip updates we do want this to happen as fast as possible)

with this approach, queuing requests on the server side is not needed at all.

but it made me wonder about something (which I was eventually going to be forced to consider anyways):

we eventually want to adapt this code so that it can become a sidecar that’s attached to Bitcoin Core, which then acts as a Sv2 Template Provider, listening on a TCP port and catering for as many clients as possible… so now, for every client that connects to it, we create 5 new threads on Bitcoin Core

whether we create 1, 2 or 5 threads for each client is not too relevant, but my main question is whether there are there protection mechanisms in place in case we try to create too many threads at once? if so, what happens once Bitcoin Core is no longer willing to create new threads?

Sjors commented at 10:11 am on November 24, 2025: member

to be honest, we could have gotten away with only 2 by forcing getHeader, getCoinbaseTx, getCoinbaseMerklePath and waitNext to be done sequentially under the same server thread

I think you should bench this before deciding if it needs optimising and also check whether it’s actually faster and not e.g. slowing the node responses down with overhead.

If getHeader, getCoinbase(Tx) and getCoinbaseMerklePath really need to be called in parallel, then I think it would be better if the BlockTemplate interface is somehow pre-populated with those values - avoiding these round trips entirely.

should concurrent IPC requests directed to the same thread cause a crash? #33923