Intermittent shutdown deadlock (in wallet_multiwallet.py; AssertionError: [node 0] bitcoind should have exited within 4800s with expected error Error: Specified -walletdir "wallets" is a relative path) #34624

Not sure how this could happen https://github.com/bitcoin/bitcoin/actions/runs/22177363139/job/64129376799?pr=34533#step:9:6127

It looks like the error is properly printed, and the node starts to shut down, but it deadlocks after [http] [httpserver.cpp:299] [ThreadHTTP] [http] Exited http event loop:

node0 2026-02-19T10:43:28.250836Z [init] [httpserver.cpp:435] [StartHTTPServer] Starting HTTP server with 2 worker threads 
Error:  node0 2026-02-19T10:43:28.250951Z [init] [noui.cpp:31] [noui_ThreadSafeMessageBox] [error] Specified -walletdir "wallets" is a relative path 
 node0 2026-02-19T10:43:28.250982Z [init] [httpserver.cpp:442] [InterruptHTTPServer] [http] Interrupting HTTP server 
 node0 2026-02-19T10:43:28.250988Z [init] [httprpc.cpp:349] [InterruptHTTPRPC] [rpc] Interrupting HTTP RPC server 
 node0 2026-02-19T10:43:28.250993Z [init] [rpc/server.cpp:284] [operator()] [rpc] Interrupting RPC 
 node0 2026-02-19T10:43:28.251002Z [init] [init.cpp:292] [Shutdown] Shutdown in progress... 
 node0 2026-02-19T10:43:28.251009Z [shutoff] [httprpc.cpp:354] [StopHTTPRPC] [rpc] Stopping HTTP RPC server 
 node0 2026-02-19T10:43:28.251016Z [shutoff] [httpserver.cpp:699] [UnregisterHTTPHandler] [http] Unregistering HTTP handler for / (exactmatch 1) 
 node0 2026-02-19T10:43:28.251021Z [shutoff] [httpserver.cpp:699] [UnregisterHTTPHandler] [http] Unregistering HTTP handler for /wallet/ (exactmatch 0) 
 node0 2026-02-19T10:43:28.251027Z [shutoff] [rpc/server.cpp:296] [operator()] [rpc] Stopping RPC 
 node0 2026-02-19T10:43:28.251076Z [http_pool_0] [util/thread.cpp:20] [TraceThread] http_pool_0 thread start 
 node0 2026-02-19T10:43:28.251083Z [http_pool_0] [util/thread.cpp:22] [TraceThread] http_pool_0 thread exit 
 node0 2026-02-19T10:43:28.251106Z [http] [httpserver.cpp:296] [ThreadHTTP] [http] Entering http event loop 
 node0 2026-02-19T10:43:28.251158Z [http_pool_1] [util/thread.cpp:20] [TraceThread] http_pool_1 thread start 
 node0 2026-02-19T10:43:28.251167Z [http_pool_1] [util/thread.cpp:22] [TraceThread] http_pool_1 thread exit 
 node0 2026-02-19T10:43:28.251344Z [shutoff] [rpc/server.cpp:298] [operator()] [rpc] RPC stopped. 
 node0 2026-02-19T10:43:28.251350Z [shutoff] [httpserver.cpp:453] [StopHTTPServer] [http] Stopping HTTP server 
 node0 2026-02-19T10:43:28.251355Z [shutoff] [httpserver.cpp:455] [StopHTTPServer] [http] Waiting for HTTP worker threads to exit 
 node0 2026-02-19T10:43:28.251502Z [http] [httpserver.cpp:299] [ThreadHTTP] [http] Exited http event loop 
 test  2026-02-19T12:03:28.189795Z TestFramework (ERROR): Unexpected exception: 
                                   Traceback (most recent call last):
                                     File "/Users/runner/work/bitcoin/bitcoin/repo_archive/test/functional/test_framework/test_framework.py", line 142, in main
                                       self.run_test()
                                       ~~~~~~~~~~~~~^^
                                     File "/Users/runner/work/bitcoin/bitcoin/repo_archive/ci/scratch/build-aarch64-apple-darwin24.6.0/test/functional/wallet_multiwallet.py", line 168, in run_test
                                       self.nodes[0].assert_start_raises_init_error(['-walletdir=wallets'], 'Error: Specified -walletdir "wallets" is a relative path', cwd=data_dir())
                                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                     File "/Users/runner/work/bitcoin/bitcoin/repo_archive/test/functional/test_framework/test_node.py", line 762, in assert_start_raises_init_error
                                       self._raise_assertion_error(assert_msg)
                                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
                                     File "/Users/runner/work/bitcoin/bitcoin/repo_archive/test/functional/test_framework/test_node.py", line 229, in _raise_assertion_error
                                       raise AssertionError(self._node_msg(msg))
                                   AssertionError: [node 0] bitcoind should have exited within 4800s with expected error Error: Specified -walletdir "wallets" is a relative path (cmd: ['bitcoin', '-m', 'node', '-datadir=/Users/runner/work/bitcoin/bitcoin/repo_archive/ci/scratch/test_runner/test_runner_₿_🏃_20260219_103433/wallet_multiwallet_253/node0', '-logtimemicros', '-debug', '-debugexclude=libevent', '-debugexclude=leveldb', '-debugexclude=rand', '-uacomment=testnode0', '-logthreadnames', '-logsourcelocations', '-loglevel=trace', '-nologratelimit', '-v2transport=0', '-walletdir=wallets'])

I guess it may be fine to ignore this for now. Similar to #34634

<!-- So I want to locally reproduce this intermittent issue. It happens 1/1000 runs. It looks like the error is properly printed, and the node starts to shut down, but it deadlocks after [http] [httpserver.cpp:299] [ThreadHTTP] [http] Exited http event loop: ``` node0 2026-02-19T10:43:28.250836Z [init] [httpserver.cpp:435] [StartHTTPServer] Starting HTTP server with 2 worker threads Error: node0 2026-02-19T10:43:28.250951Z [init] [noui.cpp:31] [noui_ThreadSafeMessageBox] [error] Specified -walletdir "wallets" is a relative path node0 2026-02-19T10:43:28.250982Z [init] [httpserver.cpp:442] [InterruptHTTPServer] [http] Interrupting HTTP server node0 2026-02-19T10:43:28.250988Z [init] [httprpc.cpp:349] [InterruptHTTPRPC] [rpc] Interrupting HTTP RPC server node0 2026-02-19T10:43:28.250993Z [init] [rpc/server.cpp:284] [operator()] [rpc] Interrupting RPC node0 2026-02-19T10:43:28.251002Z [init] [init.cpp:292] [Shutdown] Shutdown in progress... node0 2026-02-19T10:43:28.251009Z [shutoff] [httprpc.cpp:354] [StopHTTPRPC] [rpc] Stopping HTTP RPC server node0 2026-02-19T10:43:28.251016Z [shutoff] [httpserver.cpp:699] [UnregisterHTTPHandler] [http] Unregistering HTTP handler for / (exactmatch 1) node0 2026-02-19T10:43:28.251021Z [shutoff] [httpserver.cpp:699] [UnregisterHTTPHandler] [http] Unregistering HTTP handler for /wallet/ (exactmatch 0) node0 2026-02-19T10:43:28.251027Z [shutoff] [rpc/server.cpp:296] [operator()] [rpc] Stopping RPC node0 2026-02-19T10:43:28.251076Z [http_pool_0] [util/thread.cpp:20] [TraceThread] http_pool_0 thread start node0 2026-02-19T10:43:28.251083Z [http_pool_0] [util/thread.cpp:22] [TraceThread] http_pool_0 thread exit node0 2026-02-19T10:43:28.251106Z [http] [httpserver.cpp:296] [ThreadHTTP] [http] Entering http event loop node0 2026-02-19T10:43:28.251158Z [http_pool_1] [util/thread.cpp:20] [TraceThread] http_pool_1 thread start node0 2026-02-19T10:43:28.251167Z [http_pool_1] [util/thread.cpp:22] [TraceThread] http_pool_1 thread exit node0 2026-02-19T10:43:28.251344Z [shutoff] [rpc/server.cpp:298] [operator()] [rpc] RPC stopped. node0 2026-02-19T10:43:28.251350Z [shutoff] [httpserver.cpp:453] [StopHTTPServer] [http] Stopping HTTP server node0 2026-02-19T10:43:28.251355Z [shutoff] [httpserver.cpp:455] [StopHTTPServer] [http] Waiting for HTTP worker threads to exit node0 2026-02-19T10:43:28.251502Z [http] [httpserver.cpp:299] [ThreadHTTP] [http] Exited http event loop test 2026-02-19T12:03:28.189795Z TestFramework (ERROR): Unexpected exception: Traceback (most recent call last): File "/Users/runner/work/bitcoin/bitcoin/repo_archive/test/functional/test_framework/test_framework.py", line 142, in main self.run_test() ~~~~~~~~~~~~~^^ File "/Users/runner/work/bitcoin/bitcoin/repo_archive/ci/scratch/build-aarch64-apple-darwin24.6.0/test/functional/wallet_multiwallet.py", line 168, in run_test self.nodes[0].assert_start_raises_init_error(['-walletdir=wallets'], 'Error: Specified -walletdir "wallets" is a relative path', cwd=data_dir()) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/runner/work/bitcoin/bitcoin/repo_archive/test/functional/test_framework/test_node.py", line 762, in assert_start_raises_init_error self._raise_assertion_error(assert_msg) ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^ File "/Users/runner/work/bitcoin/bitcoin/repo_archive/test/functional/test_framework/test_node.py", line 229, in _raise_assertion_error raise AssertionError(self._node_msg(msg)) AssertionError: [node 0] bitcoind should have exited within 4800s with expected error Error: Specified -walletdir "wallets" is a relative path (cmd: ['bitcoin', '-m', 'node', '-datadir=/Users/runner/work/bitcoin/bitcoin/repo_archive/ci/scratch/test_runner/test_runner_₿_🏃_20260219_103433/wallet_multiwallet_253/node0', '-logtimemicros', '-debug', '-debugexclude=libevent', '-debugexclude=leveldb', '-debugexclude=rand', '-uacomment=testnode0', '-logthreadnames', '-logsourcelocations', '-loglevel=trace', '-nologratelimit', '-v2transport=0', '-walletdir=wallets']) ``` So I guess it could be reproduced outside the test via: ./bld-cmake/bin/bitcoind -datadir=/tmp/ -walletdir=wallets_missing -server=1 -printtoconsole=1 -debug=http -regtest -rpcuser=u -rpcpassword=pw However, this does not reproduce. Maybe it needs an rpc call in the background? ```sh ./bld-cmake/bin/bitcoind -datadir=/tmp/ -walletdir=wallets_missing -server=1 -printtoconsole=1 -debug=http -regtest -rpcuser=u -rpcpassword=pw & while true; do ./bld-cmake/bin/bitcoin-cli -datadir=/tmp/ -regtest -rpcuser=u -rpcpassword=pw getblockcount &> /dev/null ; done # Exit via CTRL+C after done, or limit the while loop ``` However, this still does not reproduce. So I wonder if the source code can be modified to reproduce this reliably. Can you see any possible cause, and a way to reproduce? DO NOT FIX THE CAUSE, just make the bug reproducible. You can compile via `cmake --build ./bld-cmake --parallel 8` I am thinking about some internal automated fake sync (using sleeps, atomics or condition variables) between two threads, assuming that more than one thread is involved in the race/timeout bug. You may add sleeps, or waits, or whatever, or logging. However, adding different logic will only reproduce a different (synthetic issue, not the original issue). So Do not fix any bugs. only reproduce. ok? wdyt?

Dupe of #31019 and https://github.com/bitcoin/bitcoin/issues/31894