Add `makePool` method on `ThreadMap` #283

pull rustaceanrob wants to merge 1 commits into bitcoin-core:master from rustaceanrob:26-5-29-pooled-threads changing 6 files +133 −8

rustaceanrob commented at 2:37 PM on May 29, 2026: none

This patch introduces a pool of threads to the Connection class, and allows this pool to be populated with the thread map via makePool. When a client thread is not set in a request context, it is delegated to the pool. This is unable to handle the guarentees with server-invoked callbacks that the current API offers, but these callbacks are not yet present in the interface.

The pool is implemented as round-robin as it is simplest, but perhaps the pool could be a queue of requests with work-stealing for threads that are available.

This was raised to me by Rust users, as they did not particularly care where work is executed on the server-side, but they have to set the thread regardless.

Tests in Rust can be seen here: https://github.com/2140-dev/bitcoin-capnp-types/pull/24

ref: https://github.com/2140-dev/bitcoin-capnp-types/blob/master/tests/util/bitcoin_core.rs#L149 ref: #281
DrahtBot commented at 2:37 PM on May 29, 2026: none



The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.



Reviews

See the guideline for information on the review process. A summary of reviews will appear here.
rustaceanrob renamed this:
~~Add `makePool` method on `ThreadMap`~~
WIP: Add `makePool` method on `ThreadMap`
on May 29, 2026
rustaceanrob force-pushed on May 29, 2026
rustaceanrob renamed this:
~~WIP: Add `makePool` method on `ThreadMap`~~
Add `makePool` method on `ThreadMap`
on May 29, 2026
rustaceanrob marked this as ready for review on May 29, 2026

in include/mp/proxy.capnp:51 in 158f30ddb2

  44 | @@ -45,6 +45,10 @@ interface ThreadMap $count(0) {
  45 |      # execute on. Clients create and name threads and pass the thread handle as
  46 |      # a call parameter.
  47 |      makeThread @0 (name :Text) -> (result :Thread);
  48 | +    # Pre-allocate a pool of server threads for implicit dispatch. When a
  49 | +    # request arrives with no context.thread set, the server round-robins
  50 | +    # across the pool instead of returning an error.
  51 | +    makePool @1 (count :UInt32) -> ();

ViniciusCestarii commented at 11:34 PM on May 29, 2026:

    makePool [@1](/bitcoin-core-multiprocess/contributor/1/) (name: Text, count :UInt32) -> ();

It would be usefull to have a name parameter like makeThread. Since each client may have its own thread pool, naming them would make log lines like "IPC server post request #{name} {pool/0}" attributable to a specific client

rustaceanrob commented at 9:06 AM on May 30, 2026:

Updated.How does that look?

in src/mp/proxy.cpp:438 in 158f30ddb2 outdated
```
 433 | +        });
 434 | +        auto thread_server = kj::heap<ProxyServer<Thread>>(m_connection, *thread_context.get_future().get(), std::move(thread));
 435 | +        m_connection.m_thread_pool.push_back(m_connection.m_threads.add(kj::mv(thread_server)));
 436 | +    }
 437 | +    return kj::READY_NOW;
 438 | +}
```
ViniciusCestarii commented at 12:07 AM on May 30, 2026:

Repeated calls silently grow the pool beyond the intended size, and since naming is based on the loop index starting from 0 each time, you'd end up with duplicate names like two threads both called pool/0 making logs ambiguous.

I'm not sure what would be the correct approach but I thought about:

Return an error if m_connection.m_thread_pool is not empty

Replace the old pool by cleaning the m_connection.m_thread_pool and create a fresh one with the new count. This makes sense if the client needs to resize its dedicated pool.
rustaceanrob force-pushed on May 30, 2026

Add `makePool` method on `ThreadMap`

This patch introduces a pool of threads to the `Connection` class, and
allows this pool to be populated with the thread map via `makePool`.
When a client thread is not set in a request context, it is delegated to
the pool. This is unable to handle the guarentees with server-invoked
callbacks that the current API offers, but these callbacks are not yet
present in the interface.

The pool is implemented as round-robin as it is simplest, but perhaps
the pool could be a queue of requests with work-stealing for threads
that are available.

This was raised to me by Rust users, as they did not particularly care
where work is executed on the server-side, but they have to set the
thread regardless.

ref: https://github.com/2140-dev/bitcoin-capnp-types/blob/master/tests/util/bitcoin_core.rs#L149

d5bc804b16

rustaceanrob force-pushed on May 30, 2026

Contributors

rustaceanrob

DrahtBot

ViniciusCestarii

Linked (view graph)

#281 RFC: Server should execute requests on a thread pool when client does not specify an execution thread

Add `makePool` method on `ThreadMap` #283

Reviews