rpc shutdown race: [error] libevent: http.c:2842: Assertion TAILQ_FIRST(&evcon->requests) == req failed in evhttp_send #34634

Diff to reproduce:

diff --git a/src/httpserver.cpp b/src/httpserver.cpp
index b84f0da08f..98dc24eb36 100644
--- a/src/httpserver.cpp
+++ b/src/httpserver.cpp
@@ -190,2 +190,3 @@ static void http_request_cb(struct evhttp_request* req, void* arg)
 {
+        UninterruptibleSleep(std::chrono::milliseconds{100});
     evhttp_connection* conn{evhttp_request_get_connection(req)};
diff --git a/src/init.cpp b/src/init.cpp
index e2cdb9c647..f1a820147b 100644
--- a/src/init.cpp
+++ b/src/init.cpp
@@ -1552,2 +1552,4 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
 
+    UninterruptibleSleep(std::chrono::milliseconds{60});
+
     // ********************************************************* Step 5: verify wallet database integrity

Steps to reproduce (takes about 3-4 tries):

valgrind --tool=none ./bld-cmake/bin/bitcoind -datadir=/tmp/ -walletdir=wallets_missing -server=1 -printtoconsole=1 -debug=http -regtest -rpcuser=u -rpcpassword=pw & while sleep 0.05 ; do ./bld-cmake/bin/bitcoin-cli -datadir=/tmp/ -regtest -rpcuser=u -rpcpassword=pw getblockcount &> /dev/null ; done

Output:

...
2026-02-20T12:30:16Z [http] Waiting for HTTP event thread to exit
2026-02-20T12:30:16Z [http] Received a POST request for / from 127.0.0.1:51160
2026-02-20T12:30:16Z [error] libevent: http.c:2842: Assertion TAILQ_FIRST(&evcon->requests) == req failed in evhttp_send
==2885428== 
==2885428== Process terminating with default action of signal 6 (SIGABRT): dumping core
==2885428==    at 0x5D3AE5C: __pthread_kill_implementation (in /usr/lib64/libc.so.6)
==2885428==    by 0x5CE0F0D: raise (in /usr/lib64/libc.so.6)
==2885428==    by 0x5CC86CF: abort (in /usr/lib64/libc.so.6)
==2885428==    by 0x57016FC: ??? (in /usr/lib64/libevent_core-2.1.so.7.0.1)
==2885428==    by 0x571D8B2: event_errx (in /usr/lib64/libevent_core-2.1.so.7.0.1)
==2885428==    by 0x574AD2C: evhttp_send_reply (in /usr/lib64/libevent_extra-2.1.so.7.0.1)
==2885428==    by 0x44EDDE8: std::_Function_handler<void (), HTTPRequest::WriteReply(int, std::span<std::byte const, 18446744073709551615ul>)::$_0>::_M_invoke(std::_Any_data const&) (src/httpserver.cpp:596)
==2885428==    by 0x44EBB87: httpevent_callback_fn(int, short, void*) (std_function.h:593)
==2885428==    by 0x57183C7: ??? (in /usr/lib64/libevent_core-2.1.so.7.0.1)
==2885428==    by 0x571A22E: event_base_loop (in /usr/lib64/libevent_core-2.1.so.7.0.1)
==2885428==    by 0x44EB4CC: ThreadHTTP(event_base*) (src/httpserver.cpp:304)
==2885428==    by 0x59833E3: ??? (in /usr/lib64/libstdc++.so.6.0.34)
==2885428== 
[1]+  Aborted                 (core dumped) valgrind --tool=none ./bld-cmake/bin/bitcoind -datadir=/tmp/ -walletdir=wallets_missing -server=1 -printtoconsole=1 -debug=http -regtest -rpcuser=u -rpcpassword=pw
^C

I initially tried to reproduce #34624, but I guess I ran into a separate issue instead?

This seems to fix the issue - waiting for the eventBase dispatch thread to finish processing events before we free eventHTTP:

--- a/src/httpserver.cpp
+++ b/src/httpserver.cpp
@@ -473,18 +473,15 @@ void StopHTTPServer()
         }
         g_requests.WaitUntilEmpty();
     }
-    if (eventHTTP) {
-        // Schedule a callback to call evhttp_free in the event base thread, so
-        // that evhttp_free does not need to be called again after the handling
-        // of unfinished request connections that follows.
-        event_base_once(eventBase, -1, EV_TIMEOUT, [](evutil_socket_t, short, void*) {
-            evhttp_free(eventHTTP);
-            eventHTTP = nullptr;
-        }, nullptr, nullptr);
-    }
     if (eventBase) {
         LogDebug(BCLog::HTTP, "Waiting for HTTP event thread to exit\n");
         if (g_thread_http.joinable()) g_thread_http.join();
+    }
+    if (eventHTTP) {
+        evhttp_free(eventHTTP);
+        eventHTTP = nullptr;
+    }
+    if (eventBase) {
         event_base_free(eventBase);
         eventBase = nullptr;
     }

(I was curious whether it had anything to do with ThreadPool in 4a05825a3f3993b8e402a7c18e2bc04f6d287e96 but it reproduces for the merge-commit before that one).

thx for the check and fix. Not sure if anything needs to be done here at this point, given that libevent will be removed anyway?

Yeah, I guess it's not a new issue for v31, so no rush.

I'm also not sure whether the fix above is 100% complete, maybe eventHTTP in some cases will prevent event dispatch from completing and the thread from joining.

Your diff looks like a revert of 660bdbf785a32024f0694915fa043968a0afb573, so it may fix one bug, but add another one 😅

Indeed, I think an additional event_base_loopexit() on top of my diff/revert might counteract eventHTTP keeping the dispatch loop/thread alive.