Intermittent CI failure: ThreadSanitizer failure (data race) when running wallet

practicalswift commented at 6:36 am on June 30, 2020: contributor

Intermittent CI failure: ThreadSanitizer failure (data race) when running wallet_multiwallet.py --usecli.

See Travis log for details.

 0…
 1wallet_multiwallet.py --usecli        | ✖ Failed  | 241 s
 2ALL                                   | ✖ Failed  | 3896 s (accumulated) 
 3Runtime: 994 s
 4==================
 5WARNING: ThreadSanitizer: data race (pid=30325)
 6  Write of size 8 at 0x7ba0000001e0 by thread T4 (mutexes: write M214):
 7    [#0](/bitcoin-bitcoin/0/) closedir <null> (bitcoind+0x75bf6)
 8    [#1](/bitcoin-bitcoin/1/) __os_dirlist <null> (bitcoind+0x80992c)
 9    [#2](/bitcoin-bitcoin/2/) BerkeleyBatch::VerifyEnvironment(boost::filesystem::path const&, bilingual_str&) <null> (bitcoind+0x45b342)
10    [#3](/bitcoin-bitcoin/3/) WalletBatch::VerifyEnvironment(boost::filesystem::path const&, bilingual_str&) <null> (bitcoind+0x58e2bb)
11    [#4](/bitcoin-bitcoin/4/) CWallet::Verify(interfaces::Chain&, WalletLocation const&, bilingual_str&, std::__1::vector<bilingual_str, std::__1::allocator<bilingual_str> >&) <null> (bitcoind+0x52a6b9)
12    [#5](/bitcoin-bitcoin/5/) LoadWallet(interfaces::Chain&, WalletLocation const&, bilingual_str&, std::__1::vector<bilingual_str, std::__1::allocator<bilingual_str> >&) <null> (bitcoind+0x527901)
13    [#6](/bitcoin-bitcoin/6/) loadwallet(JSONRPCRequest const&) <null> (bitcoind+0x4d2f0f)
14    [#7](/bitcoin-bitcoin/7/) std::__1::__function::__func<CRPCCommand::CRPCCommand(char const*, char const*, UniValue (*)(JSONRPCRequest const&), std::initializer_list<char const*>)::'lambda'(JSONRPCRequest const&, UniValue&, bool), std::__1::allocator<CRPCCommand::CRPCCommand(char const*, char const*, UniValue (*)(JSONRPCRequest const&), std::initializer_list<char const*>)::'lambda'(JSONRPCRequest const&, UniValue&, bool)>, bool (JSONRPCRequest const&, UniValue&, bool)>::operator()(JSONRPCRequest const&, UniValue&, bool&&) <null> (bitcoind+0x243b6d)
15    [#8](/bitcoin-bitcoin/8/) std::__1::__function::__func<interfaces::(anonymous namespace)::WalletClientImpl::registerRpcs()::'lambda'(JSONRPCRequest const&, UniValue&, bool), std::__1::allocator<interfaces::(anonymous namespace)::WalletClientImpl::registerRpcs()::'lambda'(JSONRPCRequest const&, UniValue&, bool)>, bool (JSONRPCRequest const&, UniValue&, bool)>::operator()(JSONRPCRequest const&, UniValue&, bool&&) <null> (bitcoind+0x43ec6e)
16    [#9](/bitcoin-bitcoin/9/) std::__1::__function::__func<interfaces::(anonymous namespace)::RpcHandlerImpl::RpcHandlerImpl(CRPCCommand const&)::'lambda'(JSONRPCRequest const&, UniValue&, bool), std::__1::allocator<interfaces::(anonymous namespace)::RpcHandlerImpl::RpcHandlerImpl(CRPCCommand const&)::'lambda'(JSONRPCRequest const&, UniValue&, bool)>, bool (JSONRPCRequest const&, UniValue&, bool)>::operator()(JSONRPCRequest const&, UniValue&, bool&&) <null> (bitcoind+0x108e42)
17    [#10](/bitcoin-bitcoin/10/) CRPCTable::execute(JSONRPCRequest const&) const <null> (bitcoind+0x2ec756)
18    [#11](/bitcoin-bitcoin/11/) std::__1::__function::__func<StartHTTPRPC(util::Ref const&)::$_0, std::__1::allocator<StartHTTPRPC(util::Ref const&)::$_0>, bool (HTTPRequest*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>::operator()(HTTPRequest*&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) <null> (bitcoind+0x403cea)
19    [#12](/bitcoin-bitcoin/12/) HTTPWorkItem::operator()() <null> (bitcoind+0x40e3f2)
20    [#13](/bitcoin-bitcoin/13/) WorkQueue<HTTPClosure>::Run() <null> (bitcoind+0x4104f5)
21    [#14](/bitcoin-bitcoin/14/) HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int) <null> (bitcoind+0x40a5fc)
22    [#15](/bitcoin-bitcoin/15/) void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int> >(void*) <null> (bitcoind+0x410fb8)
23  Previous read of size 8 at 0x7ba0000001e0 by thread T3 (mutexes: write M131465):
24    [#0](/bitcoin-bitcoin/0/) epoll_ctl <null> (bitcoind+0x75fa2)
25    [#1](/bitcoin-bitcoin/1/) epoll_apply_one_change <null> (bitcoind+0x8af70e)
26    [#2](/bitcoin-bitcoin/2/) void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, bool (*)(event_base*), event_base*> >(void*) <null> (bitcoind+0x410648)
27  Location is file descriptor 30 created by main thread at:
28    [failed to restore the stack]
29  Mutex M214 (0x55ae4f8baba8) created at:
30    [#0](/bitcoin-bitcoin/0/) pthread_mutex_init <null> (bitcoind+0x7172d)
31    [#1](/bitcoin-bitcoin/1/) std::__1::recursive_mutex::recursive_mutex() <null> (libc++.so.1+0x83583)
32    [#2](/bitcoin-bitcoin/2/) __libc_csu_init <null> (bitcoind+0x9501dc)
33  Mutex M131465 (0x7b0c00004740) created at:
34    [#0](/bitcoin-bitcoin/0/) pthread_mutex_init <null> (bitcoind+0x7172d)
35    [#1](/bitcoin-bitcoin/1/) evthread_posix_lock_alloc <null> (bitcoind+0x897aaf)
36    [#2](/bitcoin-bitcoin/2/) AppInitMain(util::Ref const&, NodeContext&) <null> (bitcoind+0x11cd57)
37    [#3](/bitcoin-bitcoin/3/) main <null> (bitcoind+0x10000c)
38  Thread T4 'b-httpworker.0' (tid=30350, running) created by main thread at:
39    [#0](/bitcoin-bitcoin/0/) pthread_create <null> (bitcoind+0x701ab)
40    [#1](/bitcoin-bitcoin/1/) std::__1::thread::thread<void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&, int&, void>(void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&, int&) <null> (bitcoind+0x410e90)
41    [#2](/bitcoin-bitcoin/2/) void std::__1::vector<std::__1::thread, std::__1::allocator<std::__1::thread> >::__emplace_back_slow_path<void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&, int&>(void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&, int&) <null> (bitcoind+0x4107f2)
42    [#3](/bitcoin-bitcoin/3/) StartHTTPServer() <null> (bitcoind+0x40a3b8)
43    [#4](/bitcoin-bitcoin/4/) AppInitMain(util::Ref const&, NodeContext&) <null> (bitcoind+0x11cde1)
44    [#5](/bitcoin-bitcoin/5/) main <null> (bitcoind+0x10000c)
45  Thread T3 'b-http' (tid=30349, running) created by main thread at:
46    [#0](/bitcoin-bitcoin/0/) pthread_create <null> (bitcoind+0x701ab)
47    [#1](/bitcoin-bitcoin/1/) std::__1::thread::thread<bool (&)(event_base*), event_base*&, void>(bool (&)(event_base*), event_base*&) <null> (bitcoind+0x40dc87)
48    [#2](/bitcoin-bitcoin/2/) StartHTTPServer() <null> (bitcoind+0x40a31c)
49    [#3](/bitcoin-bitcoin/3/) AppInitMain(util::Ref const&, NodeContext&) <null> (bitcoind+0x11cde1)
50    [#4](/bitcoin-bitcoin/4/) main <null> (bitcoind+0x10000c)
51SUMMARY: ThreadSanitizer: data race (/home/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/bitcoind+0x75bf6) in closedir

practicalswift added the label Bug on Jun 30, 2020

fanquake added the label Wallet on Jun 30, 2020

promag commented at 9:25 am on July 1, 2020: member

Not sure if I’m understanding the above output, there’s a race with BDB?

MarcoFalke closed this on Jul 2, 2020

ryanofsky commented at 2:58 pm on July 2, 2020: member

Another instance of this https://travis-ci.org/github/bitcoin/bitcoin/jobs/703992618 in #18608, will try rebasing after #19422

practicalswift commented at 3:05 pm on July 2, 2020: contributor

Note that the addition of the TSan suppression auto-closed this issue, but the data race remains :)

Do we want to have an open issue on this until the root cause is addressed?

DrahtBot locked this on Feb 15, 2022

Intermittent CI failure: ThreadSanitizer failure (data race) when running wallet_multiwallet.py –usecli #19417