rpc shutdown race: [error] libevent: http.c:2842: Assertion TAILQ_FIRST(&evcon->requests) == req failed in evhttp

maflcko commented at 12:34 pm on February 20, 2026: member

Diff to reproduce:

 0diff --git a/src/httpserver.cpp b/src/httpserver.cpp
 1index b84f0da08f..98dc24eb36 100644
 2--- a/src/httpserver.cpp
 3+++ b/src/httpserver.cpp
 4@@ -190,2 +190,3 @@ static void http_request_cb(struct evhttp_request* req, void* arg)
 5 {
 6+        UninterruptibleSleep(std::chrono::milliseconds{100});
 7     evhttp_connection* conn{evhttp_request_get_connection(req)};
 8diff --git a/src/init.cpp b/src/init.cpp
 9index e2cdb9c647..f1a820147b 100644
10--- a/src/init.cpp
11+++ b/src/init.cpp
12@@ -1552,2 +1552,4 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
13 
14+    UninterruptibleSleep(std::chrono::milliseconds{60});
15+
16     // ********************************************************* Step 5: verify wallet database integrity

Steps to reproduce (takes about 3-4 tries):

0valgrind --tool=none ./bld-cmake/bin/bitcoind -datadir=/tmp/ -walletdir=wallets_missing -server=1 -printtoconsole=1 -debug=http -regtest -rpcuser=u -rpcpassword=pw & while sleep 0.05 ; do ./bld-cmake/bin/bitcoin-cli -datadir=/tmp/ -regtest -rpcuser=u -rpcpassword=pw getblockcount &> /dev/null ; done

Output:

 0...
 12026-02-20T12:30:16Z [http] Waiting for HTTP event thread to exit
 22026-02-20T12:30:16Z [http] Received a POST request for / from 127.0.0.1:51160
 32026-02-20T12:30:16Z [error] libevent: http.c:2842: Assertion TAILQ_FIRST(&evcon->requests) == req failed in evhttp_send
 4==2885428== 
 5==2885428== Process terminating with default action of signal 6 (SIGABRT): dumping core
 6==2885428==    at 0x5D3AE5C: __pthread_kill_implementation (in /usr/lib64/libc.so.6)
 7==2885428==    by 0x5CE0F0D: raise (in /usr/lib64/libc.so.6)
 8==2885428==    by 0x5CC86CF: abort (in /usr/lib64/libc.so.6)
 9==2885428==    by 0x57016FC: ??? (in /usr/lib64/libevent_core-2.1.so.7.0.1)
10==2885428==    by 0x571D8B2: event_errx (in /usr/lib64/libevent_core-2.1.so.7.0.1)
11==2885428==    by 0x574AD2C: evhttp_send_reply (in /usr/lib64/libevent_extra-2.1.so.7.0.1)
12==2885428==    by 0x44EDDE8: std::_Function_handler<void (), HTTPRequest::WriteReply(int, std::span<std::byte const, 18446744073709551615ul>)::$_0>::_M_invoke(std::_Any_data const&) (src/httpserver.cpp:596)
13==2885428==    by 0x44EBB87: httpevent_callback_fn(int, short, void*) (std_function.h:593)
14==2885428==    by 0x57183C7: ??? (in /usr/lib64/libevent_core-2.1.so.7.0.1)
15==2885428==    by 0x571A22E: event_base_loop (in /usr/lib64/libevent_core-2.1.so.7.0.1)
16==2885428==    by 0x44EB4CC: ThreadHTTP(event_base*) (src/httpserver.cpp:304)
17==2885428==    by 0x59833E3: ??? (in /usr/lib64/libstdc++.so.6.0.34)
18==2885428== 
19[1]+  Aborted                 (core dumped) valgrind --tool=none ./bld-cmake/bin/bitcoind -datadir=/tmp/ -walletdir=wallets_missing -server=1 -printtoconsole=1 -debug=http -regtest -rpcuser=u -rpcpassword=pw
20^C

maflcko commented at 12:38 pm on February 20, 2026: member

I initially tried to reproduce #34624, but I guess I ran into a separate issue instead?

hodlinator commented at 8:32 pm on February 20, 2026: contributor

This seems to fix the issue - waiting for the eventBase dispatch thread to finish processing events before we free eventHTTP:

 0--- a/src/httpserver.cpp
 1+++ b/src/httpserver.cpp
 2@@ -473,18 +473,15 @@ void StopHTTPServer()
 3         }
 4         g_requests.WaitUntilEmpty();
 5     }
 6-    if (eventHTTP) {
 7-        // Schedule a callback to call evhttp_free in the event base thread, so
 8-        // that evhttp_free does not need to be called again after the handling
 9-        // of unfinished request connections that follows.
10-        event_base_once(eventBase, -1, EV_TIMEOUT, [](evutil_socket_t, short, void*) {
11-            evhttp_free(eventHTTP);
12-            eventHTTP = nullptr;
13-        }, nullptr, nullptr);
14-    }
15     if (eventBase) {
16         LogDebug(BCLog::HTTP, "Waiting for HTTP event thread to exit\n");
17         if (g_thread_http.joinable()) g_thread_http.join();
18+    }
19+    if (eventHTTP) {
20+        evhttp_free(eventHTTP);
21+        eventHTTP = nullptr;
22+    }
23+    if (eventBase) {
24         event_base_free(eventBase);
25         eventBase = nullptr;
26     }

hodlinator commented at 8:34 pm on February 20, 2026: contributor

(I was curious whether it had anything to do with ThreadPool in 4a05825a3f3993b8e402a7c18e2bc04f6d287e96 but it reproduces for the merge-commit before that one).

maflcko commented at 7:31 am on February 23, 2026: member

thx for the check and fix. Not sure if anything needs to be done here at this point, given that libevent will be removed anyway?

hodlinator commented at 8:37 am on February 23, 2026: contributor

Yeah, I guess it’s not a new issue for v31, so no rush.

I’m also not sure whether the fix above is 100% complete, maybe eventHTTP in some cases will prevent event dispatch from completing and the thread from joining.

maflcko commented at 8:53 am on February 23, 2026: member

Your diff looks like a revert of 660bdbf785a32024f0694915fa043968a0afb573, so it may fix one bug, but add another one 😅

hodlinator commented at 8:57 am on February 23, 2026: contributor

Indeed, I think an additional event_base_loopexit() on top of my diff/revert might counteract eventHTTP keeping the dispatch loop/thread alive.

maflcko added the label RPC/REST/ZMQ on Feb 23, 2026

rpc shutdown race: [error] libevent: http.c:2842: Assertion TAILQ_FIRST(&evcon->requests) == req failed in evhttp_send #34634