index: ThreadSanitizer: data race on vptr #27355

Seen with master @ 220008604f15d5078092dea28be3e3f7f11b6c8f on aarch64 (with NO_BDB=1). Follow up from #27298. See also #26188.

2023-03-28T13:52:47.882295Z (mocktime: 2020-08-31T15:34:12Z) [test] [dbwrapper.cpp:158] [CDBWrapper] Opened LevelDB successfully
2023-03-28T13:52:47.882390Z (mocktime: 2020-08-31T15:34:12Z) [test] [dbwrapper.cpp:183] [CDBWrapper] Using obfuscation key for /tmp/test_common_Bitcoin Core/0959fba80d599fe246b0b5ced04b4d1ecd504db9812edbff4ada2d88c6b218ae/regtest/indexes/txindex: 0000000000000000
2023-03-28T13:52:47.884002Z (mocktime: 2020-08-31T15:34:12Z) [txindex] [util/thread.cpp:20] [TraceThread] txindex thread start
test/txindex_tests.cpp(38): fatal error: in "txindex_tests/txindex_initial_sync": critical check time_start + timeout_ms > GetTimeMillis() has failed
LLVMSymbolizer: error reading file: No such file or directory
make[3]: *** [Makefile:21823: test/txindex_tests.cpp.test] Error 1
make[3]: Leaving directory '/home/fedora/bitcoin/ci/scratch/build/bitcoin-aarch64-unknown-linux-gnu/src'
make[2]: *** [Makefile:19828: check-am] Error 2
make[2]: Leaving directory '/home/fedora/bitcoin/ci/scratch/build/bitcoin-aarch64-unknown-linux-gnu/src'
make[1]: *** [Makefile:19493: check-recursive] Error 1
make[1]: Leaving directory '/home/fedora/bitcoin/ci/scratch/build/bitcoin-aarch64-unknown-linux-gnu/src'
make: *** [Makefile:816: check-recursive] Error 1
==================
WARNING: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) (pid=24142)
  Write of size 8 at 0xfffffee67968 by main thread:
    [#0](/bitcoin-bitcoin/0/) BaseIndex::~BaseIndex() src/index/base.cpp:80:1 (test_bitcoin+0xc54cc8) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#1](/bitcoin-bitcoin/1/) TxIndex::~TxIndex() src/index/txindex.cpp:56:19 (test_bitcoin+0xc6daec) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#2](/bitcoin-bitcoin/2/) txindex_tests::txindex_initial_sync::test_method() src/test/txindex_tests.cpp:82:1 (test_bitcoin+0x7c281c) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#3](/bitcoin-bitcoin/3/) txindex_tests::txindex_initial_sync_invoker() src/test/txindex_tests.cpp:17:1 (test_bitcoin+0x7c0f78) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#4](/bitcoin-bitcoin/4/) boost::detail::function::void_function_invoker0<void (*)(), void>::invoke(boost::detail::function::function_buffer&) /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/function/function_template.hpp:117:11 (test_bitcoin+0x2b78e8) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#5](/bitcoin-bitcoin/5/) boost::function0<void>::operator()() const /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/function/function_template.hpp:763:14 (test_bitcoin+0x24f35c) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#6](/bitcoin-bitcoin/6/) boost::detail::forward::operator()() /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1388:32 (test_bitcoin+0x24f35c)
    [#7](/bitcoin-bitcoin/7/) boost::detail::function::function_obj_invoker0<boost::detail::forward, int>::invoke(boost::detail::function::function_buffer&) /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/function/function_template.hpp:137:18 (test_bitcoin+0x24f35c)
    [#8](/bitcoin-bitcoin/8/) boost::function0<int>::operator()() const /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/function/function_template.hpp:763:14 (test_bitcoin+0x1e3384) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#9](/bitcoin-bitcoin/9/) int boost::detail::do_invoke<boost::shared_ptr<boost::detail::translator_holder_base>, boost::function<int ()>>(boost::shared_ptr<boost::detail::translator_holder_base> const&, boost::function<int ()> const&) /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/execution_monitor.ipp:301:30 (test_bitcoin+0x1e3384)
    [#10](/bitcoin-bitcoin/10/) boost::execution_monitor::catch_signals(boost::function<int ()> const&) /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/execution_monitor.ipp:903:16 (test_bitcoin+0x1e3384)
    [#11](/bitcoin-bitcoin/11/) boost::execution_monitor::execute(boost::function<int ()> const&) /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1301:16 (test_bitcoin+0x1e367c) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#12](/bitcoin-bitcoin/12/) boost::execution_monitor::vexecute(boost::function<void ()> const&) /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1397:5 (test_bitcoin+0x1dcec0) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#13](/bitcoin-bitcoin/13/) boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned long) /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/unit_test_monitor.ipp:49:9 (test_bitcoin+0x1dcec0)
    [#14](/bitcoin-bitcoin/14/) boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/framework.ipp:815:44 (test_bitcoin+0x20cdfc) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#15](/bitcoin-bitcoin/15/) boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/framework.ipp:784:58 (test_bitcoin+0x20d128) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#16](/bitcoin-bitcoin/16/) boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/framework.ipp:784:58 (test_bitcoin+0x20d128) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#17](/bitcoin-bitcoin/17/) boost::unit_test::framework::run(unsigned long, bool) /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/framework.ipp:1722:29 (test_bitcoin+0x1dbe64) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#18](/bitcoin-bitcoin/18/) boost::unit_test::unit_test_main(boost::unit_test::test_suite* (*)(int, char**), int, char**) /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/unit_test_main.ipp:250:9 (test_bitcoin+0x1f57c8) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#19](/bitcoin-bitcoin/19/) main /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/unit_test_main.ipp:306:12 (test_bitcoin+0x1f5d74) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)

  Previous read of size 8 at 0xfffffee67968 by thread T4:
    [#0](/bitcoin-bitcoin/0/) BaseIndex::ThreadSync() src/index/base.cpp:217:18 (test_bitcoin+0xc55ba0) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#1](/bitcoin-bitcoin/1/) BaseIndex::Start()::$_0::operator()() const src/index/base.cpp:404:73 (test_bitcoin+0xc59324) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#2](/bitcoin-bitcoin/2/) decltype(std::declval<BaseIndex::Start()::$_0&>()()) std::__1::__invoke[abi:v160000]<BaseIndex::Start()::$_0&>(BaseIndex::Start()::$_0&) /usr/lib/llvm-16/bin/../include/c++/v1/__functional/invoke.h:394:23 (test_bitcoin+0xc59324)
    [#3](/bitcoin-bitcoin/3/) void std::__1::__invoke_void_return_wrapper<void, true>::__call<BaseIndex::Start()::$_0&>(BaseIndex::Start()::$_0&) /usr/lib/llvm-16/bin/../include/c++/v1/__functional/invoke.h:487:9 (test_bitcoin+0xc59324)
    [#4](/bitcoin-bitcoin/4/) std::__1::__function::__alloc_func<BaseIndex::Start()::$_0, std::__1::allocator<BaseIndex::Start()::$_0>, void ()>::operator()[abi:v160000]() /usr/lib/llvm-16/bin/../include/c++/v1/__functional/function.h:185:16 (test_bitcoin+0xc59324)
    [#5](/bitcoin-bitcoin/5/) std::__1::__function::__func<BaseIndex::Start()::$_0, std::__1::allocator<BaseIndex::Start()::$_0>, void ()>::operator()() /usr/lib/llvm-16/bin/../include/c++/v1/__functional/function.h:356:12 (test_bitcoin+0xc59324)
    [#6](/bitcoin-bitcoin/6/) std::__1::__function::__value_func<void ()>::operator()[abi:v160000]() const /usr/lib/llvm-16/bin/../include/c++/v1/__functional/function.h:510:16 (test_bitcoin+0x10cbd30) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#7](/bitcoin-bitcoin/7/) std::__1::function<void ()>::operator()() const /usr/lib/llvm-16/bin/../include/c++/v1/__functional/function.h:1156:12 (test_bitcoin+0x10cbd30)
    [#8](/bitcoin-bitcoin/8/) util::TraceThread(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>) src/util/thread.cpp:21:9 (test_bitcoin+0x10cbd30)
    [#9](/bitcoin-bitcoin/9/) decltype(std::declval<void (*)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>)>()(std::declval<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>(), std::declval<BaseIndex::Start()::$_0>())) std::__1::__invoke[abi:v160000]<void (*)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, BaseIndex::Start()::$_0>(void (*&&)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&&, BaseIndex::Start()::$_0&&) /usr/lib/llvm-16/bin/../include/c++/v1/__functional/invoke.h:394:23 (test_bitcoin+0xc58e70) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#10](/bitcoin-bitcoin/10/) void std::__1::__thread_execute[abi:v160000]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (*)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, BaseIndex::Start()::$_0, 2ul, 3ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (*)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, BaseIndex::Start()::$_0>&, std::__1::__tuple_indices<2ul, 3ul>) /usr/lib/llvm-16/bin/../include/c++/v1/thread:282:5 (test_bitcoin+0xc58e70)
    [#11](/bitcoin-bitcoin/11/) void* std::__1::__thread_proxy[abi:v160000]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (*)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, BaseIndex::Start()::$_0>>(void*) /usr/lib/llvm-16/bin/../include/c++/v1/thread:293:5 (test_bitcoin+0xc58e70)

  Location is stack of main thread.

  Location is global '??' at 0xfffffee4a000 ([stack]+0x1d968)

  Thread T4 'b-txindex' (tid=24147, running) created by main thread at:
    [#0](/bitcoin-bitcoin/0/) pthread_create <null> (test_bitcoin+0x13776c) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#1](/bitcoin-bitcoin/1/) std::__1::__libcpp_thread_create[abi:v160000](unsigned long*, void* (*)(void*), void*) /usr/lib/llvm-16/bin/../include/c++/v1/__threading_support:378:10 (test_bitcoin+0xc58b00) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#2](/bitcoin-bitcoin/2/) std::__1::thread::thread<void (*)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, BaseIndex::Start()::$_0, void>(void (*&&)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, BaseIndex::Start()::$_0&&) /usr/lib/llvm-16/bin/../include/c++/v1/thread:309:16 (test_bitcoin+0xc58b00)
    [#3](/bitcoin-bitcoin/3/) BaseIndex::Start() src/index/base.cpp:404:21 (test_bitcoin+0xc58b00)
    [#4](/bitcoin-bitcoin/4/) txindex_tests::txindex_initial_sync::test_method() src/test/txindex_tests.cpp:32:5 (test_bitcoin+0x7c18d8) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#5](/bitcoin-bitcoin/5/) txindex_tests::txindex_initial_sync_invoker() src/test/txindex_tests.cpp:17:1 (test_bitcoin+0x7c0f78) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#6](/bitcoin-bitcoin/6/) boost::detail::function::void_function_invoker0<void (*)(), void>::invoke(boost::detail::function::function_buffer&) /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/function/function_template.hpp:117:11 (test_bitcoin+0x2b78e8) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#7](/bitcoin-bitcoin/7/) boost::function0<void>::operator()() const /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/function/function_template.hpp:763:14 (test_bitcoin+0x24f35c) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#8](/bitcoin-bitcoin/8/) boost::detail::forward::operator()() /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1388:32 (test_bitcoin+0x24f35c)
    [#9](/bitcoin-bitcoin/9/) boost::detail::function::function_obj_invoker0<boost::detail::forward, int>::invoke(boost::detail::function::function_buffer&) /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/function/function_template.hpp:137:18 (test_bitcoin+0x24f35c)
    [#10](/bitcoin-bitcoin/10/) boost::function0<int>::operator()() const /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/function/function_template.hpp:763:14 (test_bitcoin+0x1e3384) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#11](/bitcoin-bitcoin/11/) int boost::detail::do_invoke<boost::shared_ptr<boost::detail::translator_holder_base>, boost::function<int ()>>(boost::shared_ptr<boost::detail::translator_holder_base> const&, boost::function<int ()> const&) /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/execution_monitor.ipp:301:30 (test_bitcoin+0x1e3384)
    [#12](/bitcoin-bitcoin/12/) boost::execution_monitor::catch_signals(boost::function<int ()> const&) /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/execution_monitor.ipp:903:16 (test_bitcoin+0x1e3384)
    [#13](/bitcoin-bitcoin/13/) boost::execution_monitor::execute(boost::function<int ()> const&) /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1301:16 (test_bitcoin+0x1e367c) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#14](/bitcoin-bitcoin/14/) boost::execution_monitor::vexecute(boost::function<void ()> const&) /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1397:5 (test_bitcoin+0x1dcec0) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#15](/bitcoin-bitcoin/15/) boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned long) /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/unit_test_monitor.ipp:49:9 (test_bitcoin+0x1dcec0)
    [#16](/bitcoin-bitcoin/16/) boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/framework.ipp:815:44 (test_bitcoin+0x20cdfc) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#17](/bitcoin-bitcoin/17/) boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/framework.ipp:784:58 (test_bitcoin+0x20d128) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#18](/bitcoin-bitcoin/18/) boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/framework.ipp:784:58 (test_bitcoin+0x20d128) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#19](/bitcoin-bitcoin/19/) boost::unit_test::framework::run(unsigned long, bool) /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/framework.ipp:1722:29 (test_bitcoin+0x1dbe64) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#20](/bitcoin-bitcoin/20/) boost::unit_test::unit_test_main(boost::unit_test::test_suite* (*)(int, char**), int, char**) /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/unit_test_main.ipp:250:9 (test_bitcoin+0x1f57c8) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)
    [#21](/bitcoin-bitcoin/21/) main /home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/unit_test_main.ipp:306:12 (test_bitcoin+0x1f5d74) (BuildId: 4e139378821749f690b2477a97607af03b8272e9)

SUMMARY: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) src/index/base.cpp:80:1 in BaseIndex::~BaseIndex()
==================

real	30m28.818s
user	0m1.731s
sys	0m1.874s

Reproducible? If yes, what are the steps to reproduce?

Reproducible

100%. Running time FILE_ENV="./ci/test/00_setup_env_native_tsan.sh" ./ci/test_run_all.sh on a aarch64 machine running Fedora 37.

Retesting this on master (04595484d97100a50c146e5fc080319d9e0f5ca4). 1 line missing from the above output, which is dumped out in the test failure output before the test is re-run is libc++abi: Pure virtual function called!:

2023-04-05T15:47:47.686929Z (mocktime: 2020-08-31T15:34:12Z) [basic block filter index] [util/thread.cpp:20] [TraceThread] basic block filter index thread start
test/blockfilter_index_tests.cpp(147): fatal error: in "blockfilter_index_tests/blockfilter_index_initial_sync": critical check time_start + timeout_ms > GetTimeMillis() has failed
libc++abi: Pure virtual function called!
make[3]: *** [Makefile:21866: test/blockfilter_index_tests.cpp.test] Error 1

I think that the data race is not the root cause, but happens as a result of the absolute timeout https://github.com/bitcoin/bitcoin/blob/04595484d97100a50c146e5fc080319d9e0f5ca4/src/test/txindex_tests.cpp#L35 not being sufficient, because the test environment is too slow for whatever reason. In this case, the BOOST_REQUIRE in https://github.com/bitcoin/bitcoin/blob/04595484d97100a50c146e5fc080319d9e0f5ca4/src/test/txindex_tests.cpp#L38 fails, and the index isn't terminated correctly (by waiting for ThreadSync to stop, as it would be done in txindex.Stop() below if the boost assert wasn't hit)

I could reproduce the error and the data race on my machine (x86_64 Ubuntu) by setting timeout_ms = 0 in the line above.

As a fix, I think that we should wait longer for the sync to finish, by either getting rid of the constant timeout (though that would mean it could run forever if something was wrong) or at least increasing it significantly. I'll look into that.

Hmm, the existing timeout of 10 seconds seems plenty for the sync process, which takes just a few milliseconds on my computer. @fanquake would a higher timeout fix this in your environment? Are you doing some extreme parallelization or someting, such that the test is simply that slow? Or is there possibly some other reason that prevents the Index sync from finishing on aarch64, so that bumping the timeout wouldn't help?

would a higher timeout fix this in your environment?

I bumped the timeout, and it didn't seem to make any difference.

Are you doing some extreme parallelization or someting, such that the test is simply that slow?

Nothing like that, and the machine isn't particularly slow. i.e it has no issue running various valgrind and other CI jobs. I'll take another look.

Hmm, then my guess that it could be the timeout was probably wrong. Could you run the test in isolation with -- -printtoconsole and post the log?

Reproducible

100%. Running time FILE_ENV="./ci/test/00_setup_env_native_tsan.sh" ./ci/test_run_all.sh on a aarch64 machine running Fedora 37.

Idk. Locally this fails for me when building bdb:

Extracting bdb...
/root/bitcoin-core/depends/sources/db-4.8.30.NC.tar.gz: OK
Preprocessing bdb...
Configuring bdb...
patching file dbinc/atomic.h
patching file mp/mp_fget.c
patching file mp/mp_mvcc.c
patching file mp/mp_region.c
patching file mutex/mut_method.c
patching file mutex/mut_tas.c
checking build system type... aarch64-unknown-linux-gnu
checking host system type... aarch64-unknown-linux-gnu
checking if building in the top-level or dist directories... no
checking if --disable-cryptography option specified... no
checking if --disable-hash option specified... no
checking if --disable-partition option specified... no
checking if --disable-compression option specified... no
checking if --disable-mutexsupport option specified... no
checking if --disable-atomicsupport option specified... no
checking if --disable-queue option specified... no
checking if --disable-replication option specified... yes
checking if --disable-statistics option specified... no
checking if --disable-verify option specified... no
checking if --enable-compat185 option specified... no
checking if --enable-cxx option specified... yes
checking if --enable-debug option specified... no
checking if --enable-debug_rop option specified... no
checking if --enable-debug_wop option specified... no
checking if --enable-diagnostic option specified... no
checking if --enable-dump185 option specified... no
checking if --enable-java option specified... no
checking if --enable-mingw option specified... no
checking if --enable-o_direct option specified... no
checking if --enable-posixmutexes option specified... no
checking if --enable-rpc option specified... no
checking if --enable-smallbuild option specified... no
checking if --enable-stl option specified... no
checking if --enable-tcl option specified... no
checking if --enable-test option specified... no
checking if --enable-uimutexes option specified... no
checking if --enable-umrw option specified... no
checking if --with-mutex=MUTEX option specified... no
checking if --with-tcl=DIR option specified... no
checking if --with-uniquename=NAME option specified... no
checking for aarch64-unknown-linux-gnu-chmod... no
checking for chmod... chmod
checking for aarch64-unknown-linux-gnu-cp... no
checking for cp... cp
checking for aarch64-unknown-linux-gnu-ln... no
checking for ln... ln
checking for aarch64-unknown-linux-gnu-mkdir... no
checking for mkdir... mkdir
checking for aarch64-unknown-linux-gnu-rm... no
checking for rm... rm
checking for aarch64-unknown-linux-gnu-sh... no
checking for sh... /usr/bin/sh
checking for a BSD-compatible install... /usr/bin/install -c
checking for aarch64-unknown-linux-gnu-cc... clang-16
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether clang-16 accepts -g... yes
checking for clang-16 option to accept ISO C89... none needed
checking for an ANSI C-conforming const... yes
checking for inline... inline
checking whether we are using the GNU C++ compiler... yes
checking whether clang++-16 -stdlib=libc++ accepts -g... yes
checking how to run the C++ preprocessor... clang++-16 -stdlib=libc++ -E
checking whether C++ supports the ISO C++ standard includes... yes
checking whether we are using gcc version 2.96... no
checking for a sed that does not truncate output... /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by clang-16... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... nm
checking the name lister (nm) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking for /usr/bin/ld option to reload object files... -r
checking for aarch64-unknown-linux-gnu-objdump... no
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for aarch64-unknown-linux-gnu-ar... ar
checking for aarch64-unknown-linux-gnu-strip... no
checking for strip... strip
checking for aarch64-unknown-linux-gnu-ranlib... ranlib
checking command to parse nm output from clang-16 object... ok
checking how to run the C preprocessor... clang-16 -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking whether we are using the GNU C++ compiler... (cached) yes
checking whether clang++-16 -stdlib=libc++ accepts -g... (cached) yes
checking how to run the C++ preprocessor... clang++-16 -stdlib=libc++ -E
checking for objdir... .libs
checking if clang-16 supports -fno-rtti -fno-exceptions... yes
checking for clang-16 option to produce PIC... -fPIC -DPIC
checking if clang-16 PIC flag -fPIC -DPIC works... yes
checking if clang-16 static flag -static works... yes
checking if clang-16 supports -c -o file.o... yes
checking if clang-16 supports -c -o file.o... (cached) yes
checking whether the clang-16 linker (/usr/bin/ld) supports shared libraries... yes
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... no
checking whether to build static libraries... yes
checking for ld used by clang++-16 -stdlib=libc++... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking whether the clang++-16 -stdlib=libc++ linker (/usr/bin/ld) supports shared libraries... yes
checking for clang++-16 -stdlib=libc++ option to produce PIC... -fPIC -DPIC
checking if clang++-16 -stdlib=libc++ PIC flag -fPIC -DPIC works... yes
checking if clang++-16 -stdlib=libc++ static flag -static works... yes
checking if clang++-16 -stdlib=libc++ supports -c -o file.o... yes
checking if clang++-16 -stdlib=libc++ supports -c -o file.o... (cached) yes
checking whether the clang++-16 -stdlib=libc++ linker (/usr/bin/ld) supports shared libraries... yes
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking SOSUFFIX from libtool... .so
checking MODSUFFIX from libtool... .so
checking JMODSUFFIX from libtool... .so
checking whether stat file-mode macros are broken... no
checking whether time.h and sys/time.h may both be included... yes
checking for dirent.h that defines DIR... yes
checking for library containing opendir... none required
checking execinfo.h usability... yes
checking execinfo.h presence... yes
checking for execinfo.h... yes
checking sys/select.h usability... yes
checking sys/select.h presence... yes
checking for sys/select.h... yes
checking sys/socket.h usability... yes
checking sys/socket.h presence... yes
checking for sys/socket.h... yes
checking sys/time.h usability... yes
checking sys/time.h presence... yes
checking for sys/time.h... yes
checking for struct stat.st_blksize... yes
checking for inttypes.h... (cached) yes
checking for stdint.h... yes
checking if stdint.h can be used by C++... yes
checking stddef.h usability... yes
checking stddef.h presence... yes
checking for stddef.h... yes
checking for unistd.h... (cached) yes
checking size of char... 1
checking size of unsigned char... 1
checking size of short... 2
checking size of unsigned short... 2
checking size of int... 4
checking size of unsigned int... 4
checking size of long... 8
checking size of unsigned long... 8
checking size of long long... 8
checking size of unsigned long long... 8
checking size of char *... 8
checking for u_char... yes
checking for u_short... yes
checking for u_int... yes
checking for u_long... yes
checking for u_int8_t... yes
checking for u_int16_t... yes
checking for int16_t... yes
checking for u_int32_t... yes
checking for int32_t... yes
checking for u_int64_t... yes
checking for int64_t... yes
checking for FILE *... yes
checking for off_t... yes
checking for pid_t... yes
checking for size_t... yes
checking for time_t... yes
checking size of size_t... 8
checking for ssize_t... yes
checking for uintmax_t... yes
checking for uintptr_t... yes
checking for socklen_t... yes
checking for ANSI C exit success/failure values... yes
checking for getopt optreset variable... no
checking for mutexes... UNIX/fcntl
configure: WARNING: NO SHARED LATCH IMPLEMENTATION FOUND FOR THIS PLATFORM.
configure: error: Unable to find a mutex implementation
make: *** [funcs.mk:292: /root/bitcoin-core/depends/aarch64-unknown-linux-gnu/.bdb_stamp_configured] Error 1
make: Leaving directory '/root/bitcoin-core/depends'

Idk. Locally this fails for me when building bdb:

Yea. You'll have to use NO_BDB=1, or fix BDB.

Ok, it is passing for me with NO_BDB. Could be a hardware issue on your side.

# lscpu
Architecture:            aarch64
  CPU op-mode(s):        32-bit, 64-bit
  Byte Order:            Little Endian
CPU(s):                  4
  On-line CPU(s) list:   0-3
Vendor ID:               ARM
  BIOS Vendor ID:        QEMU
  Model name:            Neoverse-N1
    BIOS Model name:     NotSpecified  CPU @ 2.0GHz
    BIOS CPU family:     1
    Model:               1
    Thread(s) per core:  1
    Core(s) per cluster: 4
    Socket(s):           1
    Cluster(s):          1
    Stepping:            r3p1
    BogoMIPS:            50.00
    Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
NUMA:                    
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-3
Vulnerabilities:         
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Not affected
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:            Mitigation; __user pointer sanitization
  Spectre v2:            Mitigation; CSV2, BHB
  Srbds:                 Not affected
  Tsx async abort:       Not affected

I am using podman, but that shouldn't matter

Maybe you can share your CPU model/hardware, so that someone can try on that, but other than that I am not sure what to do here

Maybe you can share your CPU model/hardware,

It's the same as yours above:

Architecture:           aarch64
  CPU op-mode(s):       32-bit, 64-bit
  Byte Order:           Little Endian
CPU(s):                 4
  On-line CPU(s) list:  0-3
Vendor ID:              ARM
  Model name:           Neoverse-N1
    Model:              1
    Thread(s) per core: 1
    Core(s) per socket: 4
    Socket(s):          1
    Stepping:           r3p1
    BogoMIPS:           243.75
    Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
Caches (sum of all):    
  L1d:                  256 KiB (4 instances)
  L1i:                  256 KiB (4 instances)
  L2:                   4 MiB (4 instances)
  L3:                   32 MiB (1 instance)
NUMA:                   
  NUMA node(s):         1
  NUMA node0 CPU(s):    0-3
Vulnerabilities:        
  Itlb multihit:        Not affected
  L1tf:                 Not affected
  Mds:                  Not affected
  Meltdown:             Not affected
  Mmio stale data:      Not affected
  Retbleed:             Not affected
  Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:           Mitigation; __user pointer sanitization
  Spectre v2:           Mitigation; CSV2, BHB
  Srbds:                Not affected
  Tsx async abort:      Not affected

Hmm, then my guess that it could be the timeout was probably wrong. Could you run the test in isolation with -- -printtoconsole and post the log?

With the branch in #27462 (which just fixes BDB compilation), if i run the TSAN CI, it fails as it has been. However if I exec into the container after it fails, and run src/test/test_bitcoin --catch_system_errors=no -l test_suite -t blockfilter_index_tests, this issue does not reproduce.

I also can not reproduce with:

time bash -c 'RUN_UNIT_TESTS_SEQUENTIAL="true" RUN_UNIT_TESTS="false" RUN_FUNCTIONAL_TESTS=false CCACHE_SIZE=500M MAKEJOBS="-j9" FILE_ENV="./ci/test/00_setup_env_native_tsan.sh" ./ci/test_run_all.sh'

and diff:

diff --git a/ci/test/06_script_b.sh b/ci/test/06_script_b.sh
index f7dcbcee5..899c99119 100755
--- a/ci/test/06_script_b.sh
+++ b/ci/test/06_script_b.sh
@@ -29,7 +29,7 @@ if [ "$RUN_UNIT_TESTS" = "true" ]; then
 fi
 
 if [ "$RUN_UNIT_TESTS_SEQUENTIAL" = "true" ]; then
-  CI_EXEC "${TEST_RUNNER_ENV}" DIR_UNIT_TEST_DATA="${DIR_UNIT_TEST_DATA}" LD_LIBRARY_PATH="${DEPENDS_DIR}/${HOST}/lib" "${BASE_OUTDIR}/bin/test_bitcoin" --catch_system_errors=no -l test_suite
+  CI_EXEC "${TEST_RUNNER_ENV}" DIR_UNIT_TEST_DATA="${DIR_UNIT_TEST_DATA}" LD_LIBRARY_PATH="${DEPENDS_DIR}/${HOST}/lib" "${BASE_OUTDIR}/bin/test_bitcoin" --catch_system_errors=no -l test_suite -t blockfilter_index_tests
 fi
 
 if [ "$RUN_FUNCTIONAL_TESTS" = "true" ]; then
diff --git a/depends/packages/bdb.mk b/depends/packages/bdb.mk
index d60733605..9f5a92501 100644
--- a/depends/packages/bdb.mk
+++ b/depends/packages/bdb.mk
@@ -14,7 +14,7 @@ $(package)_config_opts_freebsd=--with-pic
 $(package)_config_opts_netbsd=--with-pic
 $(package)_config_opts_openbsd=--with-pic
 $(package)_config_opts_android=--with-pic
-$(package)_cflags+=-Wno-error=implicit-function-declaration -Wno-error=format-security
+$(package)_cflags+=-Wno-error=implicit-function-declaration -Wno-error=format-security -Wno-error=implicit-int
 $(package)_cppflags_freebsd=-D_XOPEN_SOURCE=600 -D__BSD_VISIBLE=1
 $(package)_cppflags_netbsd=-D_XOPEN_SOURCE=600
 $(package)_cppflags_openbsd=-D_XOPEN_SOURCE=600

All I run into here is a bdb issue, which can be worked around with:

diff --git a/test/sanitizer_suppressions/tsan b/test/sanitizer_suppressions/tsan
index d33199127..35c6a5f95 100644
--- a/test/sanitizer_suppressions/tsan
+++ b/test/sanitizer_suppressions/tsan
@@ -23,7 +23,7 @@ race:src/qt/test/*
 deadlock:src/qt/test/*
 
 # External libraries
-deadlock:libdb
+deadlock:__db_pthread_mutex_lock
 race:libzmq
 
 # Intermittent issues

Not really a fix if there is something else going on but.. could use https://github.com/furszy/bitcoin-core/commit/8495c852ebe199a96a4e205937861b76f54227eb (quick and dirty) to remove the active-wait timeout error.

I ran into this as well:

2023-05-12T12:24:59.051246Z (mocktime: 2020-08-31T15:34:11Z) [test] [node/miner.cpp:162] [CreateNewBlock] CreateNewBlock(): block weight: 940 txs: 0 fees: 0 sigops 400
2023-05-12T12:24:59.051432Z (mocktime: 2020-08-31T15:34:11Z) [test] [validation.cpp:2159] [ConnectBlock] [bench]     - Sanity checks: 0.01ms [0.00s (0.00ms/blk)]
2023-05-12T12:24:59.051514Z (mocktime: 2020-08-31T15:34:11Z) [test] [validation.cpp:2261] [ConnectBlock] [bench]     - Fork checks: 0.09ms [0.02s (0.09ms/blk)]
2023-05-12T12:24:59.051591Z (mocktime: 2020-08-31T15:34:11Z) [test] [validation.cpp:2351] [ConnectBlock] [bench]       - Connect 1 transactions: 0.07ms (0.071ms/tx, 0.000ms/txin) [0.02s (0.08ms/blk)]
2023-05-12T12:24:59.051658Z (mocktime: 2020-08-31T15:34:11Z) [test] [validation.cpp:2369] [ConnectBlock] [bench]     - Verify 0 txins: 0.14ms (0.000ms/txin) [0.03s (0.15ms/blk)]
2023-05-12T12:24:59.051782Z (mocktime: 2020-08-31T15:34:11Z) [test] [node/miner.cpp:181] [CreateNewBlock] [bench] CreateNewBlock() packages: 0.03ms (0 packages, 0 updated descendants), validity: 0.60ms (total 0.63ms)
2023-05-12T12:24:59.052016Z (mocktime: 2020-08-31T15:34:11Z) [test] [validation.cpp:3870] [AcceptBlockHeader] Saw new header hash=571d80a9967ae599cec0448b0b0ba1cfb606f584d8069bd7166b86854ba7a191 height=100
2023-05-12T12:24:59.052580Z (mocktime: 2020-08-31T15:34:11Z) [test] [validationinterface.cpp:260] [NewPoWValidBlock] [validation] NewPoWValidBlock: block hash=571d80a9967ae599cec0448b0b0ba1cfb606f584d8069bd7166b86854ba7a191
2023-05-12T12:24:59.053339Z (mocktime: 2020-08-31T15:34:11Z) [test] [validation.cpp:2831] [ConnectTip] [bench]   - Using cached block
2023-05-12T12:24:59.053400Z (mocktime: 2020-08-31T15:34:11Z) [test] [validation.cpp:2842] [ConnectTip] [bench]   - Load block from disk: 0.05ms [0.01s (0.03ms/blk)]
2023-05-12T12:24:59.053525Z (mocktime: 2020-08-31T15:34:11Z) [test] [validation.cpp:2159] [ConnectBlock] [bench]     - Sanity checks: 0.00ms [0.00s (0.00ms/blk)]
2023-05-12T12:24:59.053604Z (mocktime: 2020-08-31T15:34:11Z) [test] [validation.cpp:2261] [ConnectBlock] [bench]     - Fork checks: 0.08ms [0.02s (0.09ms/blk)]
2023-05-12T12:24:59.053686Z (mocktime: 2020-08-31T15:34:11Z) [test] [validation.cpp:2351] [ConnectBlock] [bench]       - Connect 1 transactions: 0.08ms (0.076ms/tx, 0.000ms/txin) [0.02s (0.08ms/blk)]
2023-05-12T12:24:59.053785Z (mocktime: 2020-08-31T15:34:11Z) [test] [validation.cpp:2369] [ConnectBlock] [bench]     - Verify 0 txins: 0.18ms (0.000ms/txin) [0.03s (0.15ms/blk)]
2023-05-12T12:24:59.053947Z (mocktime: 2020-08-31T15:34:11Z) [test] [validation.cpp:2383] [ConnectBlock] [bench]     - Write undo data: 0.16ms [0.02s (0.11ms/blk)]
2023-05-12T12:24:59.054000Z (mocktime: 2020-08-31T15:34:11Z) [test] [validation.cpp:2398] [ConnectBlock] [bench]     - Index writing: 0.06ms [0.01s (0.03ms/blk)]
2023-05-12T12:24:59.054061Z (mocktime: 2020-08-31T15:34:11Z) [test] [validationinterface.cpp:255] [BlockChecked] [validation] BlockChecked: block hash=571d80a9967ae599cec0448b0b0ba1cfb606f584d8069bd7166b86854ba7a191 state=Valid
2023-05-12T12:24:59.054118Z (mocktime: 2020-08-31T15:34:11Z) [test] [validation.cpp:2858] [ConnectTip] [bench]   - Connect total: 0.73ms [0.08s (0.38ms/blk)]
2023-05-12T12:24:59.054336Z (mocktime: 2020-08-31T15:34:11Z) [test] [validation.cpp:2867] [ConnectTip] [bench]   - Flush: 0.21ms [0.02s (0.11ms/blk)]
2023-05-12T12:24:59.054409Z (mocktime: 2020-08-31T15:34:11Z) [test] [validation.cpp:2877] [ConnectTip] [bench]   - Writing chainstate: 0.08ms [0.01s (0.04ms/blk)]
2023-05-12T12:24:59.055286Z (mocktime: 2020-08-31T15:34:11Z) [test] [policy/fees.cpp:667] [processBlock] [estimatefee] Blockpolicy estimates updated by 0 of 0 block txs, since last block 0 of 0 tracked, mempool map size 0, max target 0 from current
2023-05-12T12:24:59.055419Z (mocktime: 2020-08-31T15:34:11Z) [test] [validation.cpp:2637] [UpdateTipLog] UpdateTip: new best=571d80a9967ae599cec0448b0b0ba1cfb606f584d8069bd7166b86854ba7a191 height=100 version=0x20000000 log2_work=7.658211 tx=101 date='2020-08-31T15:34:11Z' progress=1.000000 cache=0.3MiB(100txo)
2023-05-12T12:24:59.055471Z (mocktime: 2020-08-31T15:34:11Z) [test] [validation.cpp:2893] [ConnectTip] [bench]   - Connect postprocess: 1.06ms [0.11s (0.54ms/blk)]
2023-05-12T12:24:59.055516Z (mocktime: 2020-08-31T15:34:11Z) [test] [validation.cpp:2897] [ConnectTip] [bench] - Connect block: 2.13ms [0.22s (1.09ms/blk)]
2023-05-12T12:24:59.055585Z (mocktime: 2020-08-31T15:34:11Z) [test] [txmempool.cpp:659] [check] [mempool] Checking mempool with 0 transactions and 0 inputs
2023-05-12T12:24:59.055797Z (mocktime: 2020-08-31T15:34:11Z) [test] [validationinterface.cpp:232] [BlockConnected] [validation] Enqueuing BlockConnected: block hash=571d80a9967ae599cec0448b0b0ba1cfb606f584d8069bd7166b86854ba7a191 block height=100
2023-05-12T12:24:59.055935Z (mocktime: 2020-08-31T15:34:11Z) [scheduler] [validationinterface.cpp:232] [operator()] [validation] BlockConnected: block hash=571d80a9967ae599cec0448b0b0ba1cfb606f584d8069bd7166b86854ba7a191 block height=100
2023-05-12T12:24:59.056065Z (mocktime: 2020-08-31T15:34:11Z) [test] [validationinterface.cpp:204] [UpdatedBlockTip] [validation] Enqueuing UpdatedBlockTip: new block hash=571d80a9967ae599cec0448b0b0ba1cfb606f584d8069bd7166b86854ba7a191 fork block hash=68aef0c7c1c2cc15ca20a558ea1d6e66aecc1d6398bddea75e4c031cb79cc07e (in IBD=false)
2023-05-12T12:24:59.056203Z (mocktime: 2020-08-31T15:34:11Z) [scheduler] [validationinterface.cpp:204] [operator()] [validation] UpdatedBlockTip: new block hash=571d80a9967ae599cec0448b0b0ba1cfb606f584d8069bd7166b86854ba7a191 fork block hash=68aef0c7c1c2cc15ca20a558ea1d6e66aecc1d6398bddea75e4c031cb79cc07e (in IBD=false)
2023-05-12T12:24:59.068884Z (mocktime: 2020-08-31T15:34:12Z) [test] [dbwrapper.cpp:158] [CDBWrapper] Opened LevelDB successfully
2023-05-12T12:24:59.068996Z (mocktime: 2020-08-31T15:34:12Z) [test] [dbwrapper.cpp:183] [CDBWrapper] Using obfuscation key for /tmp/test_common_Bitcoin Core/2ce9328e486919722790b84cffa4be425cf708aac6a4db24624ed0ff4e487f1d/regtest/indexes/blockfilter/basic/db: 0000000000000000
test/blockfilter_index_tests.cpp(148): fatal error: in "blockfilter_index_tests/blockfilter_index_initial_sync": critical check time_start + timeout > SteadyClock::now() has failed
libc++abi: Pure virtual function called!
2023-05-12T12:24:59.113190Z (mocktime: 2020-08-31T15:34:12Z) [basic block filter index] [util/thread.cpp:20] [TraceThread] basic block filter index thread start
make[3]: Leaving directory '/root/b-c-ci/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src'
make[2]: *** [Makefile:20096: check-am] Error 2
make[2]: Leaving directory '/root/b-c-ci/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src'
make[1]: *** [Makefile:19761: check-recursive] Error 1
make[1]: Leaving directory '/root/b-c-ci/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src'
make: *** [Makefile:816: check-recursive] Error 1
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.

Is it me or DrahtBot is typing as a human?

Not really a fix if there is something else going on but.. could use furszy@8495c85 (quick and dirty) to remove the active-wait timeout error.

Mind opening a pull so that it can be reviewed and tested?

Review note: I don't think you need to embed the exception into the promise. A thread (re-)throwing into void will lead to immediate abort anyway. It should be sufficient to replace [this] { ThreadSync(); } with [this] { ThreadSync(); p.set_value(); }

To reproduce locally, one can try to build with tsan, then run the functional tests in the background to eat all available CPU, then run the failing test case in the foreground in a loop until it fails.

Not really a fix if there is something else going on but.. could use furszy@8495c85 (quick and dirty) to remove the active-wait timeout error.

Mind opening a pull so that it can be reviewed and tested?

Sure. Need a bit more time and will do it. I'm already tackling few issues in the area (#27607 and #27720) and would be good to receive a bit more feedback there prior creating this new PR, so I don't have to rework it some days after pushing it.

A quick hack in the meantime could be:

sed -i 's/constexpr auto timeout{10s};/constexpr auto timeout{300s};/g' ./src/test/blockfilter_index_tests.cpp ./src/test/txindex_tests.cpp

Looks like there is a variation with ERROR: Commit: Failed to commit latest basic block filter index state?

https://cirrus-ci.com/task/6737719183802368?logs=ci#L4142

2023-06-22T20:05:34.892973Z (mocktime: 2020-08-31T15:34:12Z) [basic block filter index] [util/thread.cpp:20] [TraceThread] basic block filter index thread start
2023-06-22T20:05:55.312479Z (mocktime: 2020-08-31T15:34:12Z) [basic block filter index] [index/base.cpp:212] [ThreadSync] Syncing basic block filter index with block chain from height 0
2023-06-22T20:05:55.312843Z (mocktime: 2020-08-31T15:34:12Z) [basic block filter index] [logging.h:263] [error] ERROR: Commit: Failed to commit latest basic block filter index state
2023-06-22T20:05:55.313942Z (mocktime: 2020-08-31T15:34:12Z) [basic block filter index] [flatfile.cpp:69] [Allocate] [validation] Pre-allocating up to position 0x100000 in fltr00000.dat
test/blockfilter_index_tests.cpp(148): fatal error: in "blockfilter_index_tests/blockfilter_index_initial_sync": critical check time_start + timeout > SteadyClock::now() has failed
LLVMSymbolizer: error reading file: No such file or directory
make[3]: *** [Makefile:22216: test/blockfilter_index_tests.cpp.test] Error 1

And the same without the error:

https://cirrus-ci.com/task/5505476792680448?logs=ci#L4221

2023-06-23T10:38:01.749709Z (mocktime: 2020-08-31T15:34:12Z) [txindex] [util/thread.cpp:20] [TraceThread] txindex thread start
test/txindex_tests.cpp(38): fatal error: in "txindex_tests/txindex_initial_sync": critical check time_start + timeout > SteadyClock::now() has failed
libc++abi: Pure virtual function called!
make[3]: *** [Makefile:22216: test/txindex_tests.cpp.test] Error 1

This is getting too frequent. I recently encountered this 8 times in a row.

Maybe #27988 makes it less frequent for now, by only modifying test code?

Hmm interesting. #27607 is getting quite close now. It's making some substantial changes to the index Init and Start code. Will rebase furszy@8495c85 on top and push it so we can try that one too.

Looks like there is a variation with ERROR: Commit: Failed to commit latest basic block filter index state?

And the same without the error:

It's different tests for different indexes: txindex_tests/txindex_initial_sync fails without the Commit Error, blockfilter_index_tests/blockfilter_index_initial_sync fails with it (but same root cause obviously).

Looks like #27988 is a test-only fix for the initial issue reported here, and it increases consistency for now. I think it should be good to go, trivial to remove or rework in the future with a std::future-based approach.