test: Explain how to reproduce zmq:: upstream race

maflcko commented at 2:41 pm on June 8, 2025: member

An explanation makes it easier to reproduce, if needed.

test: Explain how to reproduce zmq:: upstream race fa4b659dcd

DrahtBot commented at 2:41 pm on June 8, 2025: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/32703.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	fanquake

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

DrahtBot added the label Tests on Jun 8, 2025

maflcko commented at 2:42 pm on June 8, 2025: member

For example, the following diff gives the following output.

 0diff --git a/ci/test/00_setup_env_native_tsan.sh b/ci/test/00_setup_env_native_tsan.sh
 1index b341adfec5..f2b72b8256 100755
 2--- a/ci/test/00_setup_env_native_tsan.sh
 3+++ b/ci/test/00_setup_env_native_tsan.sh
 4@@ -10,7 +10,11 @@ export CONTAINER_NAME=ci_native_tsan
 5 export CI_IMAGE_NAME_TAG="mirror.gcr.io/ubuntu:24.04"
 6 export APT_LLVM_V="20"
 7 export PACKAGES="clang-${APT_LLVM_V} llvm-${APT_LLVM_V} libclang-rt-${APT_LLVM_V}-dev libc++abi-${APT_LLVM_V}-dev libc++-${APT_LLVM_V}-dev python3-zmq"
 8-export DEP_OPTS="CC=clang-${APT_LLVM_V} CXX='clang++-${APT_LLVM_V} -stdlib=libc++'"
 9+export DEP_OPTS="NO_QT=1 NO_WALLET=1 CFLAGS='-O0 -g' CXXFLAGS='-O0 -g' CC='clang-${APT_LLVM_V} -O0 -g' CXX='clang++-${APT_LLVM_V} -O0 -g -stdlib=libc++'"
10 export GOAL="install"
11+export RUN_UNIT_TESTS="false"
12+export TEST_RUNNER_EXTRA=interface_zmq
13 export BITCOIN_CONFIG="-DWITH_ZMQ=ON -DSANITIZERS=thread \
14+-DAPPEND_CXXFLAGS='-O0 -g' \
15+-DAPPEND_CFLAGS='-O0 -g' \
16 -DAPPEND_CPPFLAGS='-DARENA_DEBUG -DDEBUG_LOCKORDER -DDEBUG_LOCKCONTENTION -D_LIBCPP_REMOVE_TRANSITIVE_INCLUDES'"
17diff --git a/test/sanitizer_suppressions/tsan b/test/sanitizer_suppressions/tsan
18index 298ad175da..884b47c4b2 100644
19--- a/test/sanitizer_suppressions/tsan
20+++ b/test/sanitizer_suppressions/tsan
21@@ -6,7 +6,6 @@
22 # race (TODO fix)
23 race:LoadWallet
24 race:DatabaseBatch
25-race:zmq::*
26 race:bitcoin-qt
27 
28 # deadlock (TODO fix)

 0 node0 stderr ==================
 1WARNING: ThreadSanitizer: data race (pid=9622)
 2  Read of size 8 at 0x72640012fc40 by thread T19:
 3    [#0](/bitcoin-bitcoin/0/) memcpy <null> (bitcoind+0xcdc72) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
 4    [#1](/bitcoin-bitcoin/1/) zmq::ypipe_t<zmq::command_t, 16>::read(zmq::command_t*) /usr/src/ypipe.hpp:134:17 (bitcoind+0x9c9ef4) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
 5
 6  Previous write of size 8 at 0x72640012fc40 by thread T20 (mutexes: write M0):
 7    [#0](/bitcoin-bitcoin/0/) memcpy <null> (bitcoind+0xcdc72) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
 8    [#1](/bitcoin-bitcoin/1/) zmq::ypipe_t<zmq::command_t, 16>::write(zmq::command_t const&, bool) /usr/src/ypipe.hpp:50:24 (bitcoind+0x9c9de0) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
 9
10  Location is heap block of size 1088 at 0x72640012fc00 allocated by thread T20:
11    [#0](/bitcoin-bitcoin/0/) posix_memalign <null> (bitcoind+0xd21d7) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
12    [#1](/bitcoin-bitcoin/1/) zmq::yqueue_t<zmq::command_t, 16, 64ul>::allocate_chunk() /usr/src/yqueue.hpp:160:13 (bitcoind+0x9ca4da) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
13
14  Mutex M0 (0x722c00000128) created at:
15    [#0](/bitcoin-bitcoin/0/) pthread_mutex_init <null> (bitcoind+0xd4260) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
16    [#1](/bitcoin-bitcoin/1/) zmq::mutex_t::mutex_t() /usr/src/mutex.hpp:93:14 (bitcoind+0x9b8886) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
17    [#2](/bitcoin-bitcoin/2/) CZMQNotificationInterface::Initialize() /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/zmq/./zmq/zmqnotificationinterface.cpp:105:23 (bitcoind+0x9a3275) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
18    [#3](/bitcoin-bitcoin/3/) CZMQNotificationInterface::Create(std::__1::function<bool (std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>&, CBlockIndex const&)>) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/zmq/./zmq/zmqnotificationinterface.cpp:78:36 (bitcoind+0x9a2cc6) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
19    [#4](/bitcoin-bitcoin/4/) AppInitMain(node::NodeContext&, interfaces::BlockAndHeaderTipInfo*) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./init.cpp:1624:36 (bitcoind+0x178157) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
20    [#5](/bitcoin-bitcoin/5/) AppInit(node::NodeContext&) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./bitcoind.cpp:237:43 (bitcoind+0x158882) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
21    [#6](/bitcoin-bitcoin/6/) main /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./bitcoind.cpp:283:10 (bitcoind+0x158882)
22
23  Thread T19 'ZMQbg/Reaper' (tid=9642, running) created by main thread at:
24    [#0](/bitcoin-bitcoin/0/) pthread_create <null> (bitcoind+0xd287e) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
25    [#1](/bitcoin-bitcoin/1/) zmq::thread_t::start(void (*)(void*), void*, char const*) /usr/src/thread.cpp:240:14 (bitcoind+0x9edc56) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
26    [#2](/bitcoin-bitcoin/2/) CZMQNotificationInterface::Initialize() /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/zmq/./zmq/zmqnotificationinterface.cpp:105:23 (bitcoind+0x9a3275) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
27    [#3](/bitcoin-bitcoin/3/) CZMQNotificationInterface::Create(std::__1::function<bool (std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>&, CBlockIndex const&)>) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/zmq/./zmq/zmqnotificationinterface.cpp:78:36 (bitcoind+0x9a2cc6) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
28    [#4](/bitcoin-bitcoin/4/) AppInitMain(node::NodeContext&, interfaces::BlockAndHeaderTipInfo*) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./init.cpp:1624:36 (bitcoind+0x178157) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
29    [#5](/bitcoin-bitcoin/5/) AppInit(node::NodeContext&) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./bitcoind.cpp:237:43 (bitcoind+0x158882) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
30    [#6](/bitcoin-bitcoin/6/) main /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./bitcoind.cpp:283:10 (bitcoind+0x158882)
31
32  Thread T20 'ZMQbg/IO/0' (tid=9643, running) created by main thread at:
33    [#0](/bitcoin-bitcoin/0/) pthread_create <null> (bitcoind+0xd287e) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
34    [#1](/bitcoin-bitcoin/1/) zmq::thread_t::start(void (*)(void*), void*, char const*) /usr/src/thread.cpp:240:14 (bitcoind+0x9edc56) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
35    [#2](/bitcoin-bitcoin/2/) CZMQNotificationInterface::Initialize() /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/zmq/./zmq/zmqnotificationinterface.cpp:105:23 (bitcoind+0x9a3275) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
36    [#3](/bitcoin-bitcoin/3/) CZMQNotificationInterface::Create(std::__1::function<bool (std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>&, CBlockIndex const&)>) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/zmq/./zmq/zmqnotificationinterface.cpp:78:36 (bitcoind+0x9a2cc6) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
37    [#4](/bitcoin-bitcoin/4/) AppInitMain(node::NodeContext&, interfaces::BlockAndHeaderTipInfo*) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./init.cpp:1624:36 (bitcoind+0x178157) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
38    [#5](/bitcoin-bitcoin/5/) AppInit(node::NodeContext&) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./bitcoind.cpp:237:43 (bitcoind+0x158882) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
39    [#6](/bitcoin-bitcoin/6/) main /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./bitcoind.cpp:283:10 (bitcoind+0x158882)
40
41SUMMARY: ThreadSanitizer: data race (/ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/bin/bitcoind+0xcdc72) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221) in memcpy
42================== 
43
44TEST             | STATUS    | DURATION
45
46interface_zmq.py | ✖ Failed  | 5 s

test: Remove intermittent and presumed fixed tsan race suppressions

LoadWallet was added in commit d77170d52687c3d708daffb7976c3e0d345ff7dc, which
points to a traceback with BerkeleyBatch in it. Now that BDB is removed,
this can be removed as well.

The race in DatabaseBatch was added in commit
fd59670642f511523570607b752ae409b5e04e7d, which does not point to a
traceback. This was likely also fixed with the BDB removal.

If not, the suppressions should be added back, mentioning that they are
intermittent and including a traceback and possibly steps to reproduce.

fa0b766f43

maflcko commented at 2:31 pm on June 9, 2025: member

Added a commit that removes intermittent, and presumed fixed BDB suppressions. (explanation in the commit message)

fanquake approved

fanquake commented at 3:55 pm on June 11, 2025: member

ACK fa0b766f43df2ead7413363941d74244a55f090c

 0 test  2025-06-11T15:52:02.512422Z TestFramework (ERROR): 
 1
 2 node0 stderr ==================
 3WARNING: ThreadSanitizer: data race (pid=175810)
 4  Read of size 8 at 0xffffb1e01e40 by thread T1:
 5    [#0](/bitcoin-bitcoin/0/) memcpy <null> (bitcoind+0xd18ac) (BuildId: 143d05b2727482274889d2193abfe2904ebd4be7)
 6    [#1](/bitcoin-bitcoin/1/) zmq::ypipe_t<zmq::command_t, 16>::read(zmq::command_t*) /usr/src/ypipe.hpp:134:17 (bitcoind+0x82ef98) (BuildId: 143d05b2727482274889d2193abfe2904ebd4be7)
 7    [#2](/bitcoin-bitcoin/2/) CZMQAbstractPublishNotifier::SendZmqMessage(char const*, void const*, unsigned long) /root/bitcoin/build/src/zmq/./zmq/zmqpublishnotifier.cpp:210:14 (bitcoind+0x80b498) (BuildId: 143d05b2727482274889d2193abfe2904ebd4be7)
 8    [#3](/bitcoin-bitcoin/3/) CZMQPublishHashTransactionNotifier::NotifyTransaction(CTransaction const&) /root/bitcoin/build/src/zmq/./zmq/zmqpublishnotifier.cpp:239:12 (bitcoind+0x80b498)
 9    [#4](/bitcoin-bitcoin/4/) CZMQNotificationInterface::BlockConnected(ChainstateRole, std::__1::shared_ptr<CBlock const> const&, CBlockIndex const*)::$_0::operator()(CZMQAbstractNotifier*) const /root/bitcoin/build/src/zmq/./zmq/zmqnotificationinterface.cpp:187:30 (bitcoind+0x8081d0) (BuildId: 143d05b2727482274889d2193abfe2904ebd4be7)
10    [#5](/bitcoin-bitcoin/5/) void (anonymous namespace)::TryForEachAndRemoveFailed<CZMQNotificationInterface::BlockConnected(ChainstateRole, std::__1::shared_ptr<CBlock const> const&, CBlockIndex const*)::$_0>(std::__1::list<std::__1::unique_ptr<CZMQAbstractNotifier, std::__1::default_delete<CZMQAbstractNotifier>>, std::__1::allocator<std::__1::unique_ptr<CZMQAbstractNotifier, std::__1::default_delete<CZMQAbstractNotifier>>>>&, CZMQNotificationInterface::BlockConnected(ChainstateRole, std::__1::shared_ptr<CBlock const> const&, CBlockIndex const*)::$_0 const&) /root/bitcoin/build/src/zmq/./zmq/zmqnotificationinterface.cpp:139:13 (bitcoind+0x8081d0)
11    [#6](/bitcoin-bitcoin/6/) CZMQNotificationInterface::BlockConnected(ChainstateRole, std::__1::shared_ptr<CBlock const> const&, CBlockIndex const*) /root/bitcoin/build/src/zmq/./zmq/zmqnotificationinterface.cpp:186:9 (bitcoind+0x8081d0)
12    [#7](/bitcoin-bitcoin/7/) ValidationSignals::BlockConnected(ChainstateRole, std::__1::shared_ptr<CBlock const> const&, CBlockIndex const*)::$_1::operator()() const::'lambda'(CValidationInterface&)::operator()(CValidationInterface&) const /root/bitcoin/build/src/./validationinterface.cpp:213:79 (bitcoind+0x57479c) (BuildId: 143d05b2727482274889d2193abfe2904ebd4be7)

fanquake merged this on Jun 11, 2025

fanquake closed this on Jun 11, 2025

maflcko deleted the branch on Jun 11, 2025

test: Explain how to reproduce zmq:: upstream race #32703

Code Coverage & Benchmarks

Reviews