bug: uncaught exception from RPC invocation during shutdown #34573

1. 
   dergoegge commented at 11:31 am on February 12, 2026: member

   The new thread pool introduced in #33689 will throw an exception (which will not be caught and lead to an unclean exit) if an RPC is called during shutdown.

   02026-02-11T18:56:34Z [http:debug] Received a POST request for / from 10.89.0.6:58998
   12026-02-11T18:56:34Z [http:debug] Interrupting HTTP server
   22026-02-11T18:56:34Z [rpc:debug] Interrupting HTTP RPC server
   32026-02-11T18:56:34Z [all:info] http_pool_1 thread exit
   42026-02-11T18:56:34Z [all:info] http_pool_0 thread exit
   52026-02-11T18:56:34Z [rpc:debug] Interrupting RPC
   6terminate called after throwing an instance of 'std::runtime_error'
   7  what():  No active workers; cannot accept new tasks
   

   This was found with a test running on Antithesis.

2. fanquake added this to the milestone 31.0 on Feb 12, 2026
3. 
   fanquake commented at 11:34 am on February 12, 2026: member

   cc @furszy
4. 
   pinheadmz commented at 11:42 am on February 12, 2026: member

   I’m not familiar with Antihesis, are you able to reliably reproduce the crash? And if so, can you test this branch which is #32061 rebased on threadpool, just not cleaned up yet.
5. 
   dergoegge commented at 11:51 am on February 12, 2026: member

   There is no reproducer testcase that I can quickly run to verify, it’s more like the test running on antithesis itself is the reproducer (as it will very likely be able to re-find the issue). You can read more about their methodology here https://antithesis.com/resources/deterministic_simulation_testing/.

   It would probably be fairly easy to write a functional test to reproduce this (perhaps combined with a strategically places sleep on the c++ side).

   I will run the Antithesis test on your PR changes, but I’m not sure if it would be the right fix unless that PR is ready to be merged soon for v31.

6. 
   pinheadmz commented at 12:01 pm on February 12, 2026: member

   I just wonder if its the same as #31929 which is addressed in #32061
7. 
   dergoegge commented at 12:04 pm on February 12, 2026: member

   This only popped up now with the new thread pool, so I don’t think so but I could be wrong of course.
8. 
   furszy commented at 3:25 pm on February 12, 2026: member

   Cool tool @dergoegge. This just uncovers an existing issue.

   Before, this would have returned an incorrect error: “Request rejected because http work queue depth exceeded” during shutdown, because the previous Enqueue() method was returning false both when the queue exceeded the max size and when it was interrupted. Now we have split these two cases to handle errors better, but miss to handle the latter one properly.

   We can either catch the exception or change submission to retrieve std::expected/util::Result with the error.

   Reproducing it should also be straightforward:

   1. Receive an http request.
   2. sleep() during request processing, before pool submission.
   3. Trigger shutdown, sleep() after unregistering the libevent callback and interrupting the pool.
   4. Let step (2) resume. This attempts to submit a task to an interrupted pool.

   Will fix it. Thanks.

9. 
   furszy commented at 8:21 pm on February 12, 2026: member

   #34577

Contributors
dergoegge fanquake pinheadmz furszy

Milestone
31.0