Suggested in #32524 (comment)
Needs upstream PR before merging here: https://github.com/arun11299/cpp-subprocess
Suggested in #32524 (comment)
Needs upstream PR before merging here: https://github.com/arun11299/cpp-subprocess
<!--e57a25ab6845829454e8d69fc972939a-->
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
<!--006a51241073e994b41acfe9ec718e94-->
For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33061.
<!--021abf342d371248e50ceaed478a90ca-->
See the guideline for information on the review process. A summary of reviews will appear here.
<!--5faf32d7da4f0f540f40219e4f7537a3-->
I suppose we should catch OSError and not crash if this ever happens...
290 | @@ -278,7 +291,7 @@ namespace util 291 | 292 | FILE *fp = _fdopen(os_fhandle, mode); 293 | if (fp == 0) { 294 | - subprocess_close(os_fhandle); 295 | + util::close(os_fhandle);
This will just confusingly change the error message, claiming that the close failed, because the fd is not available? How would the fd be available if the open failed?
Should we even call close() in that case? Alternatively I could add a ignore_failure argument to util::close.
854 | @@ -842,28 +855,28 @@ class Streams 855 | void cleanup_fds() 856 | { 857 | if (write_to_child_ != -1 && read_from_parent_ != -1) { 858 | - subprocess_close(write_to_child_); 859 | + util::close(write_to_child_);
returning early here will leak fds if the exception is caught and everything is tried again?
(same below)
Good point. So IIUC cleanup_fds makes up to three close calls, and we want to do them all even if one fails. This could be handled by catching OSError each time and throwing at the end. Though that does add complexity.
This "fix" is just introducing other bugs down the line, no?
<!--85328a0da195eb286784d51f73fa0af9-->
🚧 At least one of the CI tasks failed.
<sub>Task CentOS, depends, gui: https://github.com/bitcoin/bitcoin/runs/46711064321</sub>
<sub>LLM reason (✨ experimental): Linker error due to multiple definitions of subprocess functions in util/subprocess.h, causing build failure.</sub>
<details><summary>Hints</summary>
Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:
Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.
Leave a comment here, if you need help tracking down a confusing failure.
</details>
Allows the next commit to use the return value.
This may help debug intermittent failures such as #32524.
Have you also sent this change upstream? https://github.com/arun11299/cpp-subprocess
This "fix" is just introducing other bugs down the line, no?
That's possible, though in general it seems not ignoring errors is a good thing.
Have you also sent this change upstream? https://github.com/arun11299/cpp-subprocess
Not yet, will do depending on feedback here. Marking as draft.
I suppose we should catch
OSErrorand not crash if this ever happens...
The rpc doesn't crash on exceptions, no?
That's possible, though in general it seems not ignoring errors is a good thing.
Yes, but it depends on the context. As mentioned above, most of the places intentionally and correctly ignore the errors, as I understand it? Blindly throwing exceptions without looking at the context doesn't seem the right approach to me.
Having gone through the code a bit more, it seems that throwing OSError is problematic for at least a few reasons:
close is often called as part of error handlingOSError in more scenariosA better approach is probably to log (a narrow subset of) close() failures. There's no logging functionality at all upstream, so that would be just for us. I don't know if we want to have Bitcoin Core specific code at all in this subprocess.h?
If we go that route, I could at least upstream the first commit since it gives us access to the return value.
Also, it would be good to have evidence that the underlying bug is related to the close at all.
Even with code added to throw on a failed close, the error keeps happening for me:
diff:
diff --git a/ci/test/00_setup_env_i686_multiprocess.sh b/ci/test/00_setup_env_i686_multiprocess.sh
index e19a45d..4a7c3ea 100755
--- a/ci/test/00_setup_env_i686_multiprocess.sh
+++ b/ci/test/00_setup_env_i686_multiprocess.sh
@@ -16,6 +16,8 @@ export GOAL="install"
export TEST_RUNNER_EXTRA="--v2transport --usecli"
export BITCOIN_CONFIG="\
-DCMAKE_BUILD_TYPE=Debug \
+ -DCMAKE_C_FLAGS_DEBUG='-g2 -O0' \
+ -DCMAKE_CXX_FLAGS_DEBUG='-g2 -O0' \
-DCMAKE_C_COMPILER='clang;-m32' \
-DCMAKE_CXX_COMPILER='clang++;-m32' \
-DAPPEND_CPPFLAGS='-DBOOST_MULTI_INDEX_ENABLE_SAFE_MODE' \
diff --git a/ci/test/03_test_script.sh b/ci/test/03_test_script.sh
index b136762..a601e77 100755
--- a/ci/test/03_test_script.sh
+++ b/ci/test/03_test_script.sh
@@ -122,7 +122,7 @@ if [[ "${RUN_TIDY}" == "true" ]]; then
BITCOIN_CONFIG_ALL="$BITCOIN_CONFIG_ALL -DCMAKE_EXPORT_COMPILE_COMMANDS=ON"
fi
-bash -c "cmake -S $BASE_ROOT_DIR -B ${BASE_BUILD_DIR} $BITCOIN_CONFIG_ALL $BITCOIN_CONFIG || ( (cat $(cmake -P "${BASE_ROOT_DIR}/ci/test/GetCMakeLogFiles.cmake")) && false)"
+bash -c "cmake --fresh -S $BASE_ROOT_DIR -B ${BASE_BUILD_DIR} $BITCOIN_CONFIG_ALL $BITCOIN_CONFIG || ( (cat $(cmake -P "${BASE_ROOT_DIR}/ci/test/GetCMakeLogFiles.cmake")) && false)"
# shellcheck disable=SC2086
cmake --build "${BASE_BUILD_DIR}" "$MAKEJOBS" --target all $GOAL || (
diff --git a/src/util/subprocess.h b/src/util/subprocess.h
index ff812d7..d0ab14e 100644
--- a/src/util/subprocess.h
+++ b/src/util/subprocess.h
@@ -1232,7 +1232,17 @@ inline void Popen::execute_process() noexcept(false)
}
else
{
- subprocess_close(err_wr_pipe);// close child side of pipe, else get stuck in read below
+ //subprocess_close(err_wr_pipe);// close child side of pipe, else get stuck in read below
+ if (subprocess_close(err_wr_pipe) == -1) {
+ // Based on the close(2) man page, retrying is not safe, especially
+ // on Linux, as the file descriptor is released even when an error is
+ // returned. The error indicates a problem (e.g., I/O error), but
+ // the descriptor is gone. The primary risk of a hanging read() is averted.
+ // We should treat this as a fatal error for the operation.
+ //subprocess_close(err_rd_pipe); // Best-effort cleanup of the read end.
+ //stream_.close_child_fds();
+ throw OSError("failed to close child error pipe writer", errno);
+ }
stream_.close_child_fds();
(inside the ci):
while ( date && /ci_container_base/ci/scratch/build-i686-pc-linux-gnu/test/functional/test_runner.py --ci -j12 --tmpdirprefix /ci_container_base/ci/scratch/test_runner/ --ansi --combinedlogslen=99999999 --timeout-factor=40 --v2transport --usecli --quiet --failfast rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer ) ; do true ; done
(then wait a few hours)
Bt parent:
Thread 33 (Thread 0xf58f4b40 (LWP 773609) "b-httpworker.2"):
[#0](/bitcoin-bitcoin/0/) 0xf7f4f589 in __kernel_vsyscall ()
[#1](/bitcoin-bitcoin/1/) 0xf7a6fac7 in read () from /lib32/libc.so.6
[#2](/bitcoin-bitcoin/2/) 0x57d6c736 in subprocess::util::read_atmost_n (fp=0x592995e0, buf=0xf58f15fc "", read_upto=1024) at ./util/subprocess.h:437
[#3](/bitcoin-bitcoin/3/) 0x57d80dde in subprocess::Popen::execute_process (this=0xf58f1cd8) at ./util/subprocess.h:1257
[#4](/bitcoin-bitcoin/4/) 0x57d6d2b4 in subprocess::Popen::Popen<subprocess::input, subprocess::output, subprocess::error, subprocess::close_fds> (this=0xf58f1cd8, cmd_args="fake.py enumerate", args=..., args=..., args=..., args=...) at ./util/subprocess.h:964
[#5](/bitcoin-bitcoin/5/) 0x57d6b597 in RunCommandParseJSON (str_command="fake.py enumerate", str_std_in="") at ./common/run_command.cpp:27
[#6](/bitcoin-bitcoin/6/) 0x57a90547 in ExternalSigner::Enumerate (command="fake.py", signers=std::__debug::vector of length 0, capacity 0, chain="regtest") at ./external_signer.cpp:28
[#7](/bitcoin-bitcoin/7/) 0x56defdab in enumeratesigners()::$_0::operator()(RPCHelpMan const&, JSONRPCRequest const&) const (this=0xf58f2ba0, self=..., request=...) at ./rpc/external_signer.cpp:51
Bt child:
(gdb) thread apply all bt
Thread 1 (Thread 0xf58f4b40 (LWP 774911) "b-httpworker.2"):
[#0](/bitcoin-bitcoin/0/) 0xf7f4f589 in __kernel_vsyscall ()
[#1](/bitcoin-bitcoin/1/) 0xf79e467e in ?? () from /lib32/libc.so.6
[#2](/bitcoin-bitcoin/2/) 0xf79eb582 in pthread_mutex_lock () from /lib32/libc.so.6
[#3](/bitcoin-bitcoin/3/) 0xf7d93bf2 in ?? () from /lib32/libstdc++.so.6
[#4](/bitcoin-bitcoin/4/) 0xf7d93f36 in __gnu_debug::_Safe_iterator_base::_M_attach(__gnu_debug::_Safe_sequence_base*, bool) () from /lib32/libstdc++.so.6
[#5](/bitcoin-bitcoin/5/) 0x5668810a in __gnu_debug::_Safe_iterator_base::_Safe_iterator_base (this=0xf58f13ac, __seq=0xf58f13f8, __constant=false) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/debug/safe_base.h:91
[#6](/bitcoin-bitcoin/6/) 0x56ddfb50 in __gnu_debug::_Safe_iterator<__gnu_cxx::__normal_iterator<int*, std::__cxx1998::vector<int, std::allocator<int> > >, std::__debug::vector<int, std::allocator<int> >, std::forward_iterator_tag>::_Safe_iterator (this=0xf58f13a8, __i=3, __seq=0xf58f13f8) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/debug/safe_iterator.h:162
[#7](/bitcoin-bitcoin/7/) 0x56ddfacb in __gnu_debug::_Safe_iterator<__gnu_cxx::__normal_iterator<int*, std::__cxx1998::vector<int, std::allocator<int> > >, std::__debug::vector<int, std::allocator<int> >, std::bidirectional_iterator_tag>::_Safe_iterator (this=0xf58f13a8, __i=3, __seq=0xf58f13f8) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/debug/safe_iterator.h:539
[#8](/bitcoin-bitcoin/8/) 0x56ddfa5b in __gnu_debug::_Safe_iterator<__gnu_cxx::__normal_iterator<int*, std::__cxx1998::vector<int, std::allocator<int> > >, std::__debug::vector<int, std::allocator<int> >, std::random_access_iterator_tag>::_Safe_iterator (this=0xf58f13a8, __i=3, __seq=0xf58f13f8) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/debug/safe_iterator.h:687
[#9](/bitcoin-bitcoin/9/) 0x56ddd3f6 in std::__debug::vector<int, std::allocator<int> >::begin (this=0xf58f13f8) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/debug/vector:300
[#10](/bitcoin-bitcoin/10/) 0x57d83701 in subprocess::detail::Child::execute_child (this=0xf58f156c) at ./util/subprocess.h:1372
[#11](/bitcoin-bitcoin/11/) 0x57d80a7c in subprocess::Popen::execute_process (this=0xf58f1cd8) at ./util/subprocess.h:1231
[#12](/bitcoin-bitcoin/12/) 0x57d6d2b4 in subprocess::Popen::Popen<subprocess::input, subprocess::output, subprocess::error, subprocess::close_fds> (this=0xf58f1cd8, cmd_args="fake.py enumerate", args=..., args=..., args=..., args=...) at ./util/subprocess.h:964
[#13](/bitcoin-bitcoin/13/) 0x57d6b597 in RunCommandParseJSON (str_command="fake.py enumerate", str_std_in="") at ./common/run_command.cpp:27
[#14](/bitcoin-bitcoin/14/) 0x57a90547 in ExternalSigner::Enumerate (command="fake.py", signers=std::__debug::vector of length 0, capacity 0, chain="regtest") at ./external_signer.cpp:28
[#15](/bitcoin-bitcoin/15/) 0x56defdab in enumeratesigners()::$_0::operator()(RPCHelpMan const&, JSONRPCRequest const&) const (this=0xf58f2ba0, self=..., request=...) at ./rpc/external_signer.cpp:51
[#16](/bitcoin-bitcoin/16/) 0x56def98d in std::__invoke_impl<UniValue, enumeratesigners()::$_0&, RPCHelpMan const&, JSONRPCRequest const&>(std::__invoke_other, enumeratesigners()::$_0&, RPCHelpMan const&, JSONRPCRequest const&) (__f=..., __args=..., __args=...) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61
[#17](/bitcoin-bitcoin/17/) 0x56def8ad in std::__invoke_r<UniValue, enumeratesigners()::$_0&, RPCHelpMan const&, JSONRPCRequest const&>(enumeratesigners()::$_0&, RPCHelpMan const&, JSONRPCRequest const&) (__fn=..., __args=..., __args=...) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:114
[#18](/bitcoin-bitcoin/18/) 0x56def703 in std::_Function_handler<UniValue (RPCHelpMan const&, JSONRPCRequest const&), enumeratesigners()::$_0>::_M_invoke(std::_Any_data const&, RPCHelpMan const&, JSONRPCRequest const&) (__functor=..., __args=..., __args=...) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290
[#19](/bitcoin-bitcoin/19/) 0x57b8665c in std::function<UniValue (RPCHelpMan const&, JSONRPCRequest const&)>::operator()(RPCHelpMan const&, JSONRPCRequest const&) const (this=0xf58f2ba0, __args=..., __args=...) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591
[#20](/bitcoin-bitcoin/20/) 0x57b787f3 in RPCHelpMan::HandleRequest (this=0xf58f2b88, request=...) at ./rpc/util.cpp:663
[#21](/bitcoin-bitcoin/21/) 0x56da2c29 in CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}::operator()(JSONRPCRequest const&, UniValue&, bool) const (this=0x58b10ce8 <RegisterSignerRPCCommands(CRPCTable&)::commands+48>, request=..., result=...) at ./rpc/server.h:60
[#22](/bitcoin-bitcoin/22/) 0x56da2b75 in std::__invoke_impl<bool, CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}&, JSONRPCRequest const&, UniValue&, bool>(std::__invoke_other, CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}&, JSONRPCRequest const&, UniValue&, bool&&) (__f=..., __args=@0xf58f2d37: true, __args=@0xf58f2d37: true, __args=@0xf58f2d37: true) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61
[#23](/bitcoin-bitcoin/23/) 0x56da2a8f in std::__invoke_r<bool, CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}&, JSONRPCRequest const&, UniValue&, bool>(CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}&, JSONRPCRequest const&, UniValue&, bool&&) (__fn=..., __args=@0xf58f2d37: true, __args=@0xf58f2d37: true, __args=@0xf58f2d37: true) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:114
[#24](/bitcoin-bitcoin/24/) 0x56da28cd in std::_Function_handler<bool (JSONRPCRequest const&, UniValue&, bool), CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}>::_M_invoke(std::_Any_data const&, JSONRPCRequest const&, UniValue&, bool&&) (__functor=..., __args=@0xf58f2d37: true, __args=@0xf58f2d37: true, __args=@0xf58f2d37: true) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290
[#25](/bitcoin-bitcoin/25/) 0x56b3e86f in std::function<bool (JSONRPCRequest const&, UniValue&, bool)>::operator()(JSONRPCRequest const&, UniValue&, bool) const (this=0x58b10ce8 <RegisterSignerRPCCommands(CRPCTable&)::commands+48>, __args=true, __args=true, __args=true) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591
[#26](/bitcoin-bitcoin/26/) 0x56fdd861 in ExecuteCommand (command=..., request=..., result=..., last_handler=true) at ./rpc/server.cpp:506
[#27](/bitcoin-bitcoin/27/) 0x56fd9e05 in ExecuteCommands (commands=std::__debug::vector of length 1, capacity 1 = {...}, request=..., result=...) at ./rpc/server.cpp:471
[#28](/bitcoin-bitcoin/28/) 0x56fd9afd in CRPCTable::execute (this=0x58b12bd4 <tableRPC>, request=...) at ./rpc/server.cpp:491
[#29](/bitcoin-bitcoin/29/) 0x56fd91c3 in JSONRPCExec (jreq=..., catch_errors=true) at ./rpc/server.cpp:347
[#30](/bitcoin-bitcoin/30/) 0x572b255a in HTTPReq_JSONRPC (context=std::any containing node::NodeContext * = {...}, req=0x59277460) at ./httprpc.cpp:165
[#31](/bitcoin-bitcoin/31/) 0x572b17ea in StartHTTPRPC(std::any const&)::$_0::operator()(HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const (this=0x5927c870, req=0x59277460) at ./httprpc.cpp:337
[#32](/bitcoin-bitcoin/32/) 0x572b1783 in std::__invoke_impl<bool, StartHTTPRPC(std::any const&)::$_0&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>(std::__invoke_other, StartHTTPRPC(std::any const&)::$_0&, HTTPRequest*&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (__f=..., __args="", __args="") at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61
[#33](/bitcoin-bitcoin/33/) 0x572b16b1 in std::__invoke_r<bool, StartHTTPRPC(std::any const&)::$_0&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>(StartHTTPRPC(std::any const&)::$_0&, HTTPRequest*&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (__fn=..., __args="", __args="") at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:114
--Type <RET> for more, q to quit, c to continue without paging--c
[#34](/bitcoin-bitcoin/34/) 0x572b13d3 in std::_Function_handler<bool (HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), StartHTTPRPC(std::any const&)::$_0>::_M_invoke(std::_Any_data const&, HTTPRequest*&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (__functor=..., __args="", __args="") at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290
[#35](/bitcoin-bitcoin/35/) 0x572e016c in std::function<bool (HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::operator()(HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const (this=0x592916d0, __args="", __args="") at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591
[#36](/bitcoin-bitcoin/36/) 0x572dfeea in HTTPWorkItem::operator() (this=0x592916b0) at ./httpserver.cpp:62
[#37](/bitcoin-bitcoin/37/) 0x572ea7f1 in WorkQueue<HTTPClosure>::Run (this=0x5902b720) at ./httpserver.cpp:117
[#38](/bitcoin-bitcoin/38/) 0x572cac42 in HTTPWorkQueueRun (queue=0x5902b720, worker_num=2) at ./httpserver.cpp:419
[#39](/bitcoin-bitcoin/39/) 0x572efc55 in std::__invoke_impl<void, void (*)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int> (__f=@0x5904efdc: 0x572cabc0 <HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int)>, __args=@0x5904efd4: 2, __args=@0x5904efd4: 2) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61
[#40](/bitcoin-bitcoin/40/) 0x572efac5 in std::__invoke<void (*)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int> (__fn=@0x5904efdc: 0x572cabc0 <HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int)>, __args=@0x5904efd4: 2, __args=@0x5904efd4: 2) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96
[#41](/bitcoin-bitcoin/41/) 0x572efa54 in std::thread::_Invoker<std::tuple<void (*)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int> >::_M_invoke<0u, 1u, 2u> (this=0x5904efd4) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:292
[#42](/bitcoin-bitcoin/42/) 0x572ef9b2 in std::thread::_Invoker<std::tuple<void (*)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int> >::operator() (this=0x5904efd4) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:299
[#43](/bitcoin-bitcoin/43/) 0x572ef635 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int> > >::_M_run (this=0x5904efd0) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:244
[#44](/bitcoin-bitcoin/44/) 0xf7d9ad21 in ?? () from /lib32/libstdc++.so.6
[#45](/bitcoin-bitcoin/45/) 0xf79e8137 in ?? () from /lib32/libc.so.6
[#46](/bitcoin-bitcoin/46/) 0xf7a7cde8 in ?? () from /lib32/libc.so.6
So it seems like this is a bug in our code (taking a lock before execv)