Intermittent CI failure: ThreadSanitizer failure (data race) when running wallet_multiwallet.py --usecli #19417

issue practicalswift opened this issue on June 30, 2020

practicalswift commented at 6:36 AM on June 30, 2020: contributor

Intermittent CI failure: ThreadSanitizer failure (data race) when running wallet_multiwallet.py --usecli.

See Travis log for details.

…
wallet_multiwallet.py --usecli        | ✖ Failed  | 241 s
ALL                                   | ✖ Failed  | 3896 s (accumulated) 
Runtime: 994 s
==================
WARNING: ThreadSanitizer: data race (pid=30325)
  Write of size 8 at 0x7ba0000001e0 by thread T4 (mutexes: write M214):
    [#0](/bitcoin-bitcoin/0/) closedir <null> (bitcoind+0x75bf6)
    [#1](/bitcoin-bitcoin/1/) __os_dirlist <null> (bitcoind+0x80992c)
    [#2](/bitcoin-bitcoin/2/) BerkeleyBatch::VerifyEnvironment(boost::filesystem::path const&, bilingual_str&) <null> (bitcoind+0x45b342)
    [#3](/bitcoin-bitcoin/3/) WalletBatch::VerifyEnvironment(boost::filesystem::path const&, bilingual_str&) <null> (bitcoind+0x58e2bb)
    [#4](/bitcoin-bitcoin/4/) CWallet::Verify(interfaces::Chain&, WalletLocation const&, bilingual_str&, std::__1::vector<bilingual_str, std::__1::allocator<bilingual_str> >&) <null> (bitcoind+0x52a6b9)
    [#5](/bitcoin-bitcoin/5/) LoadWallet(interfaces::Chain&, WalletLocation const&, bilingual_str&, std::__1::vector<bilingual_str, std::__1::allocator<bilingual_str> >&) <null> (bitcoind+0x527901)
    [#6](/bitcoin-bitcoin/6/) loadwallet(JSONRPCRequest const&) <null> (bitcoind+0x4d2f0f)
    [#7](/bitcoin-bitcoin/7/) std::__1::__function::__func<CRPCCommand::CRPCCommand(char const*, char const*, UniValue (*)(JSONRPCRequest const&), std::initializer_list<char const*>)::'lambda'(JSONRPCRequest const&, UniValue&, bool), std::__1::allocator<CRPCCommand::CRPCCommand(char const*, char const*, UniValue (*)(JSONRPCRequest const&), std::initializer_list<char const*>)::'lambda'(JSONRPCRequest const&, UniValue&, bool)>, bool (JSONRPCRequest const&, UniValue&, bool)>::operator()(JSONRPCRequest const&, UniValue&, bool&&) <null> (bitcoind+0x243b6d)
    [#8](/bitcoin-bitcoin/8/) std::__1::__function::__func<interfaces::(anonymous namespace)::WalletClientImpl::registerRpcs()::'lambda'(JSONRPCRequest const&, UniValue&, bool), std::__1::allocator<interfaces::(anonymous namespace)::WalletClientImpl::registerRpcs()::'lambda'(JSONRPCRequest const&, UniValue&, bool)>, bool (JSONRPCRequest const&, UniValue&, bool)>::operator()(JSONRPCRequest const&, UniValue&, bool&&) <null> (bitcoind+0x43ec6e)
    [#9](/bitcoin-bitcoin/9/) std::__1::__function::__func<interfaces::(anonymous namespace)::RpcHandlerImpl::RpcHandlerImpl(CRPCCommand const&)::'lambda'(JSONRPCRequest const&, UniValue&, bool), std::__1::allocator<interfaces::(anonymous namespace)::RpcHandlerImpl::RpcHandlerImpl(CRPCCommand const&)::'lambda'(JSONRPCRequest const&, UniValue&, bool)>, bool (JSONRPCRequest const&, UniValue&, bool)>::operator()(JSONRPCRequest const&, UniValue&, bool&&) <null> (bitcoind+0x108e42)
    [#10](/bitcoin-bitcoin/10/) CRPCTable::execute(JSONRPCRequest const&) const <null> (bitcoind+0x2ec756)
    [#11](/bitcoin-bitcoin/11/) std::__1::__function::__func<StartHTTPRPC(util::Ref const&)::$_0, std::__1::allocator<StartHTTPRPC(util::Ref const&)::$_0>, bool (HTTPRequest*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>::operator()(HTTPRequest*&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) <null> (bitcoind+0x403cea)
    [#12](/bitcoin-bitcoin/12/) HTTPWorkItem::operator()() <null> (bitcoind+0x40e3f2)
    [#13](/bitcoin-bitcoin/13/) WorkQueue<HTTPClosure>::Run() <null> (bitcoind+0x4104f5)
    [#14](/bitcoin-bitcoin/14/) HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int) <null> (bitcoind+0x40a5fc)
    [#15](/bitcoin-bitcoin/15/) void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int> >(void*) <null> (bitcoind+0x410fb8)
  Previous read of size 8 at 0x7ba0000001e0 by thread T3 (mutexes: write M131465):
    [#0](/bitcoin-bitcoin/0/) epoll_ctl <null> (bitcoind+0x75fa2)
    [#1](/bitcoin-bitcoin/1/) epoll_apply_one_change <null> (bitcoind+0x8af70e)
    [#2](/bitcoin-bitcoin/2/) void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, bool (*)(event_base*), event_base*> >(void*) <null> (bitcoind+0x410648)
  Location is file descriptor 30 created by main thread at:
    [failed to restore the stack]
  Mutex M214 (0x55ae4f8baba8) created at:
    [#0](/bitcoin-bitcoin/0/) pthread_mutex_init <null> (bitcoind+0x7172d)
    [#1](/bitcoin-bitcoin/1/) std::__1::recursive_mutex::recursive_mutex() <null> (libc++.so.1+0x83583)
    [#2](/bitcoin-bitcoin/2/) __libc_csu_init <null> (bitcoind+0x9501dc)
  Mutex M131465 (0x7b0c00004740) created at:
    [#0](/bitcoin-bitcoin/0/) pthread_mutex_init <null> (bitcoind+0x7172d)
    [#1](/bitcoin-bitcoin/1/) evthread_posix_lock_alloc <null> (bitcoind+0x897aaf)
    [#2](/bitcoin-bitcoin/2/) AppInitMain(util::Ref const&, NodeContext&) <null> (bitcoind+0x11cd57)
    [#3](/bitcoin-bitcoin/3/) main <null> (bitcoind+0x10000c)
  Thread T4 'b-httpworker.0' (tid=30350, running) created by main thread at:
    [#0](/bitcoin-bitcoin/0/) pthread_create <null> (bitcoind+0x701ab)
    [#1](/bitcoin-bitcoin/1/) std::__1::thread::thread<void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&, int&, void>(void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&, int&) <null> (bitcoind+0x410e90)
    [#2](/bitcoin-bitcoin/2/) void std::__1::vector<std::__1::thread, std::__1::allocator<std::__1::thread> >::__emplace_back_slow_path<void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&, int&>(void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&, int&) <null> (bitcoind+0x4107f2)
    [#3](/bitcoin-bitcoin/3/) StartHTTPServer() <null> (bitcoind+0x40a3b8)
    [#4](/bitcoin-bitcoin/4/) AppInitMain(util::Ref const&, NodeContext&) <null> (bitcoind+0x11cde1)
    [#5](/bitcoin-bitcoin/5/) main <null> (bitcoind+0x10000c)
  Thread T3 'b-http' (tid=30349, running) created by main thread at:
    [#0](/bitcoin-bitcoin/0/) pthread_create <null> (bitcoind+0x701ab)
    [#1](/bitcoin-bitcoin/1/) std::__1::thread::thread<bool (&)(event_base*), event_base*&, void>(bool (&)(event_base*), event_base*&) <null> (bitcoind+0x40dc87)
    [#2](/bitcoin-bitcoin/2/) StartHTTPServer() <null> (bitcoind+0x40a31c)
    [#3](/bitcoin-bitcoin/3/) AppInitMain(util::Ref const&, NodeContext&) <null> (bitcoind+0x11cde1)
    [#4](/bitcoin-bitcoin/4/) main <null> (bitcoind+0x10000c)
SUMMARY: ThreadSanitizer: data race (/home/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/bitcoind+0x75bf6) in closedir

practicalswift added the label Bug on Jun 30, 2020
fanquake added the label Wallet on Jun 30, 2020
promag commented at 9:25 AM on July 1, 2020: member

Not sure if I'm understanding the above output, there's a race with BDB?
MarcoFalke closed this on Jul 2, 2020
ryanofsky commented at 2:58 PM on July 2, 2020: member

Another instance of this https://travis-ci.org/github/bitcoin/bitcoin/jobs/703992618 in #18608, will try rebasing after #19422
practicalswift commented at 3:05 PM on July 2, 2020: contributor

Note that the addition of the TSan suppression auto-closed this issue, but the data race remains :)

Do we want to have an open issue on this until the root cause is addressed?
DrahtBot locked this on Feb 15, 2022

Contributors

Labels

Linked (view graph)

#1 JSON-RPC support for mobile devices ("ultra-lightweight" clients)#2 Long-term, safe, store-of-value #3 Encrypt wallet #4 Export/Import wallet in a human readable, future-proof format #5 Make the version number the protocol version and not the client version #6 Treat wallet as a generic keystore #7 Block-header-only, faster startup client #8 RPC command to sign text with wallet private key #9 Fix for GUI on Macs and latest wxWidgets #10 Add address to listtransactions output #11 Nolisten patch #12 Monitor transactions and/or blocks #13 Messages with or about transactions #14 bitcoin: URI and/or bitcoin-request MIME type for click-to-pay #15 Option to specify external IP address #19422 ci: Add tsan suppression for race in wallet #19448 qa: Data race detected when running `mempool_expiry.py`