Suggested in #32524 (comment)
Needs upstream PR before merging here: https://github.com/arun11299/cpp-subprocess
Suggested in #32524 (comment)
Needs upstream PR before merging here: https://github.com/arun11299/cpp-subprocess
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33061.
See the guideline for information on the review process. A summary of reviews will appear here.
OSError
and not crash if this ever happens…
290@@ -278,7 +291,7 @@ namespace util
291
292 FILE *fp = _fdopen(os_fhandle, mode);
293 if (fp == 0) {
294- subprocess_close(os_fhandle);
295+ util::close(os_fhandle);
close()
in that case? Alternatively I could add a ignore_failure
argument to util::close
.
854@@ -842,28 +855,28 @@ class Streams
855 void cleanup_fds()
856 {
857 if (write_to_child_ != -1 && read_from_parent_ != -1) {
858- subprocess_close(write_to_child_);
859+ util::close(write_to_child_);
cleanup_fds
makes up to three close
calls, and we want to do them all even if one fails. This could be handled by catching OSError each time and throwing at the end. Though that does add complexity.
🚧 At least one of the CI tasks failed.
Task CentOS, depends, gui
: https://github.com/bitcoin/bitcoin/runs/46711064321
LLM reason (✨ experimental): Linker error due to multiple definitions of subprocess functions in util/subprocess.h, causing build failure.
Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:
Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.
Leave a comment here, if you need help tracking down a confusing failure.
Allows the next commit to use the return value.
This may help debug intermittent failures such as #32524.
This “fix” is just introducing other bugs down the line, no?
That’s possible, though in general it seems not ignoring errors is a good thing.
Have you also sent this change upstream? https://github.com/arun11299/cpp-subprocess
Not yet, will do depending on feedback here. Marking as draft.
I suppose we should catch
OSError
and not crash if this ever happens…
The rpc doesn’t crash on exceptions, no?
That’s possible, though in general it seems not ignoring errors is a good thing.
Yes, but it depends on the context. As mentioned above, most of the places intentionally and correctly ignore the errors, as I understand it? Blindly throwing exceptions without looking at the context doesn’t seem the right approach to me.
Having gone through the code a bit more, it seems that throwing OSError
is problematic for at least a few reasons:
close
is often called as part of error handlingOSError
in more scenariosA better approach is probably to log (a narrow subset of) close()
failures. There’s no logging functionality at all upstream, so that would be just for us. I don’t know if we want to have Bitcoin Core specific code at all in this subprocess.h
?
If we go that route, I could at least upstream the first commit since it gives us access to the return value.
Also, it would be good to have evidence that the underlying bug is related to the close at all.
Even with code added to throw on a failed close, the error keeps happening for me:
diff:
0diff --git a/ci/test/00_setup_env_i686_multiprocess.sh b/ci/test/00_setup_env_i686_multiprocess.sh
1index e19a45d..4a7c3ea 100755
2--- a/ci/test/00_setup_env_i686_multiprocess.sh
3+++ b/ci/test/00_setup_env_i686_multiprocess.sh
4@@ -16,6 +16,8 @@ export GOAL="install"
5 export TEST_RUNNER_EXTRA="--v2transport --usecli"
6 export BITCOIN_CONFIG="\
7 -DCMAKE_BUILD_TYPE=Debug \
8+ -DCMAKE_C_FLAGS_DEBUG='-g2 -O0' \
9+ -DCMAKE_CXX_FLAGS_DEBUG='-g2 -O0' \
10 -DCMAKE_C_COMPILER='clang;-m32' \
11 -DCMAKE_CXX_COMPILER='clang++;-m32' \
12 -DAPPEND_CPPFLAGS='-DBOOST_MULTI_INDEX_ENABLE_SAFE_MODE' \
13diff --git a/ci/test/03_test_script.sh b/ci/test/03_test_script.sh
14index b136762..a601e77 100755
15--- a/ci/test/03_test_script.sh
16+++ b/ci/test/03_test_script.sh
17@@ -122,7 +122,7 @@ if [[ "${RUN_TIDY}" == "true" ]]; then
18 BITCOIN_CONFIG_ALL="$BITCOIN_CONFIG_ALL -DCMAKE_EXPORT_COMPILE_COMMANDS=ON"
19 fi
20
21-bash -c "cmake -S $BASE_ROOT_DIR -B ${BASE_BUILD_DIR} $BITCOIN_CONFIG_ALL $BITCOIN_CONFIG || ( (cat $(cmake -P "${BASE_ROOT_DIR}/ci/test/GetCMakeLogFiles.cmake")) && false)"
22+bash -c "cmake --fresh -S $BASE_ROOT_DIR -B ${BASE_BUILD_DIR} $BITCOIN_CONFIG_ALL $BITCOIN_CONFIG || ( (cat $(cmake -P "${BASE_ROOT_DIR}/ci/test/GetCMakeLogFiles.cmake")) && false)"
23
24 # shellcheck disable=SC2086
25 cmake --build "${BASE_BUILD_DIR}" "$MAKEJOBS" --target all $GOAL || (
26diff --git a/src/util/subprocess.h b/src/util/subprocess.h
27index ff812d7..d0ab14e 100644
28--- a/src/util/subprocess.h
29+++ b/src/util/subprocess.h
30@@ -1232,7 +1232,17 @@ inline void Popen::execute_process() noexcept(false)
31 }
32 else
33 {
34- subprocess_close(err_wr_pipe);// close child side of pipe, else get stuck in read below
35+ //subprocess_close(err_wr_pipe);// close child side of pipe, else get stuck in read below
36+ if (subprocess_close(err_wr_pipe) == -1) {
37+ // Based on the close(2) man page, retrying is not safe, especially
38+ // on Linux, as the file descriptor is released even when an error is
39+ // returned. The error indicates a problem (e.g., I/O error), but
40+ // the descriptor is gone. The primary risk of a hanging read() is averted.
41+ // We should treat this as a fatal error for the operation.
42+ //subprocess_close(err_rd_pipe); // Best-effort cleanup of the read end.
43+ //stream_.close_child_fds();
44+ throw OSError("failed to close child error pipe writer", errno);
45+ }
46
47 stream_.close_child_fds();
48
(inside the ci):
while ( date && /ci_container_base/ci/scratch/build-i686-pc-linux-gnu/test/functional/test_runner.py --ci -j12 --tmpdirprefix /ci_container_base/ci/scratch/test_runner/ --ansi --combinedlogslen=99999999 --timeout-factor=40 --v2transport --usecli --quiet --failfast rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer rpc_signer ) ; do true ; done
(then wait a few hours)
Bt parent:
0Thread 33 (Thread 0xf58f4b40 (LWP 773609) "b-httpworker.2"):
1[#0](/bitcoin-bitcoin/0/) 0xf7f4f589 in __kernel_vsyscall ()
2[#1](/bitcoin-bitcoin/1/) 0xf7a6fac7 in read () from /lib32/libc.so.6
3[#2](/bitcoin-bitcoin/2/) 0x57d6c736 in subprocess::util::read_atmost_n (fp=0x592995e0, buf=0xf58f15fc "", read_upto=1024) at ./util/subprocess.h:437
4[#3](/bitcoin-bitcoin/3/) 0x57d80dde in subprocess::Popen::execute_process (this=0xf58f1cd8) at ./util/subprocess.h:1257
5[#4](/bitcoin-bitcoin/4/) 0x57d6d2b4 in subprocess::Popen::Popen<subprocess::input, subprocess::output, subprocess::error, subprocess::close_fds> (this=0xf58f1cd8, cmd_args="fake.py enumerate", args=..., args=..., args=..., args=...) at ./util/subprocess.h:964
6[#5](/bitcoin-bitcoin/5/) 0x57d6b597 in RunCommandParseJSON (str_command="fake.py enumerate", str_std_in="") at ./common/run_command.cpp:27
7[#6](/bitcoin-bitcoin/6/) 0x57a90547 in ExternalSigner::Enumerate (command="fake.py", signers=std::__debug::vector of length 0, capacity 0, chain="regtest") at ./external_signer.cpp:28
8[#7](/bitcoin-bitcoin/7/) 0x56defdab in enumeratesigners()::$_0::operator()(RPCHelpMan const&, JSONRPCRequest const&) const (this=0xf58f2ba0, self=..., request=...) at ./rpc/external_signer.cpp:51
Bt child:
0(gdb) thread apply all bt
1
2Thread 1 (Thread 0xf58f4b40 (LWP 774911) "b-httpworker.2"):
3[#0](/bitcoin-bitcoin/0/) 0xf7f4f589 in __kernel_vsyscall ()
4[#1](/bitcoin-bitcoin/1/) 0xf79e467e in ?? () from /lib32/libc.so.6
5[#2](/bitcoin-bitcoin/2/) 0xf79eb582 in pthread_mutex_lock () from /lib32/libc.so.6
6[#3](/bitcoin-bitcoin/3/) 0xf7d93bf2 in ?? () from /lib32/libstdc++.so.6
7[#4](/bitcoin-bitcoin/4/) 0xf7d93f36 in __gnu_debug::_Safe_iterator_base::_M_attach(__gnu_debug::_Safe_sequence_base*, bool) () from /lib32/libstdc++.so.6
8[#5](/bitcoin-bitcoin/5/) 0x5668810a in __gnu_debug::_Safe_iterator_base::_Safe_iterator_base (this=0xf58f13ac, __seq=0xf58f13f8, __constant=false) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/debug/safe_base.h:91
9[#6](/bitcoin-bitcoin/6/) 0x56ddfb50 in __gnu_debug::_Safe_iterator<__gnu_cxx::__normal_iterator<int*, std::__cxx1998::vector<int, std::allocator<int> > >, std::__debug::vector<int, std::allocator<int> >, std::forward_iterator_tag>::_Safe_iterator (this=0xf58f13a8, __i=3, __seq=0xf58f13f8) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/debug/safe_iterator.h:162
10[#7](/bitcoin-bitcoin/7/) 0x56ddfacb in __gnu_debug::_Safe_iterator<__gnu_cxx::__normal_iterator<int*, std::__cxx1998::vector<int, std::allocator<int> > >, std::__debug::vector<int, std::allocator<int> >, std::bidirectional_iterator_tag>::_Safe_iterator (this=0xf58f13a8, __i=3, __seq=0xf58f13f8) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/debug/safe_iterator.h:539
11[#8](/bitcoin-bitcoin/8/) 0x56ddfa5b in __gnu_debug::_Safe_iterator<__gnu_cxx::__normal_iterator<int*, std::__cxx1998::vector<int, std::allocator<int> > >, std::__debug::vector<int, std::allocator<int> >, std::random_access_iterator_tag>::_Safe_iterator (this=0xf58f13a8, __i=3, __seq=0xf58f13f8) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/debug/safe_iterator.h:687
12[#9](/bitcoin-bitcoin/9/) 0x56ddd3f6 in std::__debug::vector<int, std::allocator<int> >::begin (this=0xf58f13f8) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/debug/vector:300
13[#10](/bitcoin-bitcoin/10/) 0x57d83701 in subprocess::detail::Child::execute_child (this=0xf58f156c) at ./util/subprocess.h:1372
14[#11](/bitcoin-bitcoin/11/) 0x57d80a7c in subprocess::Popen::execute_process (this=0xf58f1cd8) at ./util/subprocess.h:1231
15[#12](/bitcoin-bitcoin/12/) 0x57d6d2b4 in subprocess::Popen::Popen<subprocess::input, subprocess::output, subprocess::error, subprocess::close_fds> (this=0xf58f1cd8, cmd_args="fake.py enumerate", args=..., args=..., args=..., args=...) at ./util/subprocess.h:964
16[#13](/bitcoin-bitcoin/13/) 0x57d6b597 in RunCommandParseJSON (str_command="fake.py enumerate", str_std_in="") at ./common/run_command.cpp:27
17[#14](/bitcoin-bitcoin/14/) 0x57a90547 in ExternalSigner::Enumerate (command="fake.py", signers=std::__debug::vector of length 0, capacity 0, chain="regtest") at ./external_signer.cpp:28
18[#15](/bitcoin-bitcoin/15/) 0x56defdab in enumeratesigners()::$_0::operator()(RPCHelpMan const&, JSONRPCRequest const&) const (this=0xf58f2ba0, self=..., request=...) at ./rpc/external_signer.cpp:51
19[#16](/bitcoin-bitcoin/16/) 0x56def98d in std::__invoke_impl<UniValue, enumeratesigners()::$_0&, RPCHelpMan const&, JSONRPCRequest const&>(std::__invoke_other, enumeratesigners()::$_0&, RPCHelpMan const&, JSONRPCRequest const&) (__f=..., __args=..., __args=...) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61
20[#17](/bitcoin-bitcoin/17/) 0x56def8ad in std::__invoke_r<UniValue, enumeratesigners()::$_0&, RPCHelpMan const&, JSONRPCRequest const&>(enumeratesigners()::$_0&, RPCHelpMan const&, JSONRPCRequest const&) (__fn=..., __args=..., __args=...) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:114
21[#18](/bitcoin-bitcoin/18/) 0x56def703 in std::_Function_handler<UniValue (RPCHelpMan const&, JSONRPCRequest const&), enumeratesigners()::$_0>::_M_invoke(std::_Any_data const&, RPCHelpMan const&, JSONRPCRequest const&) (__functor=..., __args=..., __args=...) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290
22[#19](/bitcoin-bitcoin/19/) 0x57b8665c in std::function<UniValue (RPCHelpMan const&, JSONRPCRequest const&)>::operator()(RPCHelpMan const&, JSONRPCRequest const&) const (this=0xf58f2ba0, __args=..., __args=...) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591
23[#20](/bitcoin-bitcoin/20/) 0x57b787f3 in RPCHelpMan::HandleRequest (this=0xf58f2b88, request=...) at ./rpc/util.cpp:663
24[#21](/bitcoin-bitcoin/21/) 0x56da2c29 in CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}::operator()(JSONRPCRequest const&, UniValue&, bool) const (this=0x58b10ce8 <RegisterSignerRPCCommands(CRPCTable&)::commands+48>, request=..., result=...) at ./rpc/server.h:60
25[#22](/bitcoin-bitcoin/22/) 0x56da2b75 in std::__invoke_impl<bool, CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}&, JSONRPCRequest const&, UniValue&, bool>(std::__invoke_other, CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}&, JSONRPCRequest const&, UniValue&, bool&&) (__f=..., __args=@0xf58f2d37: true, __args=@0xf58f2d37: true, __args=@0xf58f2d37: true) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61
26[#23](/bitcoin-bitcoin/23/) 0x56da2a8f in std::__invoke_r<bool, CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}&, JSONRPCRequest const&, UniValue&, bool>(CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}&, JSONRPCRequest const&, UniValue&, bool&&) (__fn=..., __args=@0xf58f2d37: true, __args=@0xf58f2d37: true, __args=@0xf58f2d37: true) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:114
27[#24](/bitcoin-bitcoin/24/) 0x56da28cd in std::_Function_handler<bool (JSONRPCRequest const&, UniValue&, bool), CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}>::_M_invoke(std::_Any_data const&, JSONRPCRequest const&, UniValue&, bool&&) (__functor=..., __args=@0xf58f2d37: true, __args=@0xf58f2d37: true, __args=@0xf58f2d37: true) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290
28[#25](/bitcoin-bitcoin/25/) 0x56b3e86f in std::function<bool (JSONRPCRequest const&, UniValue&, bool)>::operator()(JSONRPCRequest const&, UniValue&, bool) const (this=0x58b10ce8 <RegisterSignerRPCCommands(CRPCTable&)::commands+48>, __args=true, __args=true, __args=true) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591
29[#26](/bitcoin-bitcoin/26/) 0x56fdd861 in ExecuteCommand (command=..., request=..., result=..., last_handler=true) at ./rpc/server.cpp:506
30[#27](/bitcoin-bitcoin/27/) 0x56fd9e05 in ExecuteCommands (commands=std::__debug::vector of length 1, capacity 1 = {...}, request=..., result=...) at ./rpc/server.cpp:471
31[#28](/bitcoin-bitcoin/28/) 0x56fd9afd in CRPCTable::execute (this=0x58b12bd4 <tableRPC>, request=...) at ./rpc/server.cpp:491
32[#29](/bitcoin-bitcoin/29/) 0x56fd91c3 in JSONRPCExec (jreq=..., catch_errors=true) at ./rpc/server.cpp:347
33[#30](/bitcoin-bitcoin/30/) 0x572b255a in HTTPReq_JSONRPC (context=std::any containing node::NodeContext * = {...}, req=0x59277460) at ./httprpc.cpp:165
34[#31](/bitcoin-bitcoin/31/) 0x572b17ea in StartHTTPRPC(std::any const&)::$_0::operator()(HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const (this=0x5927c870, req=0x59277460) at ./httprpc.cpp:337
35[#32](/bitcoin-bitcoin/32/) 0x572b1783 in std::__invoke_impl<bool, StartHTTPRPC(std::any const&)::$_0&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>(std::__invoke_other, StartHTTPRPC(std::any const&)::$_0&, HTTPRequest*&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (__f=..., __args="", __args="") at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61
36[#33](/bitcoin-bitcoin/33/) 0x572b16b1 in std::__invoke_r<bool, StartHTTPRPC(std::any const&)::$_0&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>(StartHTTPRPC(std::any const&)::$_0&, HTTPRequest*&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (__fn=..., __args="", __args="") at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:114
37--Type <RET> for more, q to quit, c to continue without paging--c
38[#34](/bitcoin-bitcoin/34/) 0x572b13d3 in std::_Function_handler<bool (HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), StartHTTPRPC(std::any const&)::$_0>::_M_invoke(std::_Any_data const&, HTTPRequest*&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (__functor=..., __args="", __args="") at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290
39[#35](/bitcoin-bitcoin/35/) 0x572e016c in std::function<bool (HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::operator()(HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const (this=0x592916d0, __args="", __args="") at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591
40[#36](/bitcoin-bitcoin/36/) 0x572dfeea in HTTPWorkItem::operator() (this=0x592916b0) at ./httpserver.cpp:62
41[#37](/bitcoin-bitcoin/37/) 0x572ea7f1 in WorkQueue<HTTPClosure>::Run (this=0x5902b720) at ./httpserver.cpp:117
42[#38](/bitcoin-bitcoin/38/) 0x572cac42 in HTTPWorkQueueRun (queue=0x5902b720, worker_num=2) at ./httpserver.cpp:419
43[#39](/bitcoin-bitcoin/39/) 0x572efc55 in std::__invoke_impl<void, void (*)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int> (__f=@0x5904efdc: 0x572cabc0 <HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int)>, __args=@0x5904efd4: 2, __args=@0x5904efd4: 2) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61
44[#40](/bitcoin-bitcoin/40/) 0x572efac5 in std::__invoke<void (*)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int> (__fn=@0x5904efdc: 0x572cabc0 <HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int)>, __args=@0x5904efd4: 2, __args=@0x5904efd4: 2) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96
45[#41](/bitcoin-bitcoin/41/) 0x572efa54 in std::thread::_Invoker<std::tuple<void (*)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int> >::_M_invoke<0u, 1u, 2u> (this=0x5904efd4) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:292
46[#42](/bitcoin-bitcoin/42/) 0x572ef9b2 in std::thread::_Invoker<std::tuple<void (*)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int> >::operator() (this=0x5904efd4) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:299
47[#43](/bitcoin-bitcoin/43/) 0x572ef635 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int> > >::_M_run (this=0x5904efd0) at /bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:244
48[#44](/bitcoin-bitcoin/44/) 0xf7d9ad21 in ?? () from /lib32/libstdc++.so.6
49[#45](/bitcoin-bitcoin/45/) 0xf79e8137 in ?? () from /lib32/libc.so.6
50[#46](/bitcoin-bitcoin/46/) 0xf7a7cde8 in ?? () from /lib32/libc.so.6
So it seems like this is a bug in our code (taking a lock before execv)