test: Explain how to reproduce zmq:: upstream race #32703

pull maflcko wants to merge 2 commits into bitcoin:master from maflcko:2506-tsan-doc changing 1 files +6 −7

maflcko commented at 2:41 PM on June 8, 2025: member

An explanation makes it easier to reproduce, if needed.
test: Explain how to reproduce zmq:: upstream race fa4b659dcd
DrahtBot commented at 2:41 PM on June 8, 2025: contributor



The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.



Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/32703.



Reviews

See the guideline for information on the review process.

Type Reviewers

ACK fanquake

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.
DrahtBot added the label Tests on Jun 8, 2025

Type	Reviewers
ACK	fanquake

maflcko commented at 2:42 PM on June 8, 2025: member

For example, the following diff gives the following output.

diff --git a/ci/test/00_setup_env_native_tsan.sh b/ci/test/00_setup_env_native_tsan.sh
index b341adfec5..f2b72b8256 100755
--- a/ci/test/00_setup_env_native_tsan.sh
+++ b/ci/test/00_setup_env_native_tsan.sh
@@ -10,7 +10,11 @@ export CONTAINER_NAME=ci_native_tsan
 export CI_IMAGE_NAME_TAG="mirror.gcr.io/ubuntu:24.04"
 export APT_LLVM_V="20"
 export PACKAGES="clang-${APT_LLVM_V} llvm-${APT_LLVM_V} libclang-rt-${APT_LLVM_V}-dev libc++abi-${APT_LLVM_V}-dev libc++-${APT_LLVM_V}-dev python3-zmq"
-export DEP_OPTS="CC=clang-${APT_LLVM_V} CXX='clang++-${APT_LLVM_V} -stdlib=libc++'"
+export DEP_OPTS="NO_QT=1 NO_WALLET=1 CFLAGS='-O0 -g' CXXFLAGS='-O0 -g' CC='clang-${APT_LLVM_V} -O0 -g' CXX='clang++-${APT_LLVM_V} -O0 -g -stdlib=libc++'"
 export GOAL="install"
+export RUN_UNIT_TESTS="false"
+export TEST_RUNNER_EXTRA=interface_zmq
 export BITCOIN_CONFIG="-DWITH_ZMQ=ON -DSANITIZERS=thread \
+-DAPPEND_CXXFLAGS='-O0 -g' \
+-DAPPEND_CFLAGS='-O0 -g' \
 -DAPPEND_CPPFLAGS='-DARENA_DEBUG -DDEBUG_LOCKORDER -DDEBUG_LOCKCONTENTION -D_LIBCPP_REMOVE_TRANSITIVE_INCLUDES'"
diff --git a/test/sanitizer_suppressions/tsan b/test/sanitizer_suppressions/tsan
index 298ad175da..884b47c4b2 100644
--- a/test/sanitizer_suppressions/tsan
+++ b/test/sanitizer_suppressions/tsan
@@ -6,7 +6,6 @@
 # race (TODO fix)
 race:LoadWallet
 race:DatabaseBatch
-race:zmq::*
 race:bitcoin-qt
 
 # deadlock (TODO fix)

 node0 stderr ==================
WARNING: ThreadSanitizer: data race (pid=9622)
  Read of size 8 at 0x72640012fc40 by thread T19:
    [#0](/bitcoin-bitcoin/0/) memcpy <null> (bitcoind+0xcdc72) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#1](/bitcoin-bitcoin/1/) zmq::ypipe_t<zmq::command_t, 16>::read(zmq::command_t*) /usr/src/ypipe.hpp:134:17 (bitcoind+0x9c9ef4) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)

  Previous write of size 8 at 0x72640012fc40 by thread T20 (mutexes: write M0):
    [#0](/bitcoin-bitcoin/0/) memcpy <null> (bitcoind+0xcdc72) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#1](/bitcoin-bitcoin/1/) zmq::ypipe_t<zmq::command_t, 16>::write(zmq::command_t const&, bool) /usr/src/ypipe.hpp:50:24 (bitcoind+0x9c9de0) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)

  Location is heap block of size 1088 at 0x72640012fc00 allocated by thread T20:
    [#0](/bitcoin-bitcoin/0/) posix_memalign <null> (bitcoind+0xd21d7) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#1](/bitcoin-bitcoin/1/) zmq::yqueue_t<zmq::command_t, 16, 64ul>::allocate_chunk() /usr/src/yqueue.hpp:160:13 (bitcoind+0x9ca4da) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)

  Mutex M0 (0x722c00000128) created at:
    [#0](/bitcoin-bitcoin/0/) pthread_mutex_init <null> (bitcoind+0xd4260) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#1](/bitcoin-bitcoin/1/) zmq::mutex_t::mutex_t() /usr/src/mutex.hpp:93:14 (bitcoind+0x9b8886) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#2](/bitcoin-bitcoin/2/) CZMQNotificationInterface::Initialize() /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/zmq/./zmq/zmqnotificationinterface.cpp:105:23 (bitcoind+0x9a3275) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#3](/bitcoin-bitcoin/3/) CZMQNotificationInterface::Create(std::__1::function<bool (std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>&, CBlockIndex const&)>) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/zmq/./zmq/zmqnotificationinterface.cpp:78:36 (bitcoind+0x9a2cc6) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#4](/bitcoin-bitcoin/4/) AppInitMain(node::NodeContext&, interfaces::BlockAndHeaderTipInfo*) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./init.cpp:1624:36 (bitcoind+0x178157) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#5](/bitcoin-bitcoin/5/) AppInit(node::NodeContext&) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./bitcoind.cpp:237:43 (bitcoind+0x158882) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#6](/bitcoin-bitcoin/6/) main /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./bitcoind.cpp:283:10 (bitcoind+0x158882)

  Thread T19 'ZMQbg/Reaper' (tid=9642, running) created by main thread at:
    [#0](/bitcoin-bitcoin/0/) pthread_create <null> (bitcoind+0xd287e) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#1](/bitcoin-bitcoin/1/) zmq::thread_t::start(void (*)(void*), void*, char const*) /usr/src/thread.cpp:240:14 (bitcoind+0x9edc56) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#2](/bitcoin-bitcoin/2/) CZMQNotificationInterface::Initialize() /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/zmq/./zmq/zmqnotificationinterface.cpp:105:23 (bitcoind+0x9a3275) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#3](/bitcoin-bitcoin/3/) CZMQNotificationInterface::Create(std::__1::function<bool (std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>&, CBlockIndex const&)>) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/zmq/./zmq/zmqnotificationinterface.cpp:78:36 (bitcoind+0x9a2cc6) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#4](/bitcoin-bitcoin/4/) AppInitMain(node::NodeContext&, interfaces::BlockAndHeaderTipInfo*) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./init.cpp:1624:36 (bitcoind+0x178157) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#5](/bitcoin-bitcoin/5/) AppInit(node::NodeContext&) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./bitcoind.cpp:237:43 (bitcoind+0x158882) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#6](/bitcoin-bitcoin/6/) main /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./bitcoind.cpp:283:10 (bitcoind+0x158882)

  Thread T20 'ZMQbg/IO/0' (tid=9643, running) created by main thread at:
    [#0](/bitcoin-bitcoin/0/) pthread_create <null> (bitcoind+0xd287e) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#1](/bitcoin-bitcoin/1/) zmq::thread_t::start(void (*)(void*), void*, char const*) /usr/src/thread.cpp:240:14 (bitcoind+0x9edc56) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#2](/bitcoin-bitcoin/2/) CZMQNotificationInterface::Initialize() /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/zmq/./zmq/zmqnotificationinterface.cpp:105:23 (bitcoind+0x9a3275) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#3](/bitcoin-bitcoin/3/) CZMQNotificationInterface::Create(std::__1::function<bool (std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>&, CBlockIndex const&)>) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/zmq/./zmq/zmqnotificationinterface.cpp:78:36 (bitcoind+0x9a2cc6) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#4](/bitcoin-bitcoin/4/) AppInitMain(node::NodeContext&, interfaces::BlockAndHeaderTipInfo*) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./init.cpp:1624:36 (bitcoind+0x178157) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#5](/bitcoin-bitcoin/5/) AppInit(node::NodeContext&) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./bitcoind.cpp:237:43 (bitcoind+0x158882) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#6](/bitcoin-bitcoin/6/) main /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./bitcoind.cpp:283:10 (bitcoind+0x158882)

SUMMARY: ThreadSanitizer: data race (/ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/bin/bitcoind+0xcdc72) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221) in memcpy
================== 

TEST             | STATUS    | DURATION

interface_zmq.py | ✖ Failed  | 5 s

test: Remove intermittent and presumed fixed tsan race suppressions

LoadWallet was added in commit d77170d52687c3d708daffb7976c3e0d345ff7dc, which
points to a traceback with BerkeleyBatch in it. Now that BDB is removed,
this can be removed as well.

The race in DatabaseBatch was added in commit
fd59670642f511523570607b752ae409b5e04e7d, which does not point to a
traceback. This was likely also fixed with the BDB removal.

If not, the suppressions should be added back, mentioning that they are
intermittent and including a traceback and possibly steps to reproduce.

fa0b766f43

maflcko commented at 2:31 PM on June 9, 2025: member

Added a commit that removes intermittent, and presumed fixed BDB suppressions. (explanation in the commit message)
fanquake approved

fanquake commented at 3:55 PM on June 11, 2025: member

ACK fa0b766f43df2ead7413363941d74244a55f090c

 test  2025-06-11T15:52:02.512422Z TestFramework (ERROR): 

 node0 stderr ==================
WARNING: ThreadSanitizer: data race (pid=175810)
  Read of size 8 at 0xffffb1e01e40 by thread T1:
    [#0](/bitcoin-bitcoin/0/) memcpy <null> (bitcoind+0xd18ac) (BuildId: 143d05b2727482274889d2193abfe2904ebd4be7)
    [#1](/bitcoin-bitcoin/1/) zmq::ypipe_t<zmq::command_t, 16>::read(zmq::command_t*) /usr/src/ypipe.hpp:134:17 (bitcoind+0x82ef98) (BuildId: 143d05b2727482274889d2193abfe2904ebd4be7)
    [#2](/bitcoin-bitcoin/2/) CZMQAbstractPublishNotifier::SendZmqMessage(char const*, void const*, unsigned long) /root/bitcoin/build/src/zmq/./zmq/zmqpublishnotifier.cpp:210:14 (bitcoind+0x80b498) (BuildId: 143d05b2727482274889d2193abfe2904ebd4be7)
    [#3](/bitcoin-bitcoin/3/) CZMQPublishHashTransactionNotifier::NotifyTransaction(CTransaction const&) /root/bitcoin/build/src/zmq/./zmq/zmqpublishnotifier.cpp:239:12 (bitcoind+0x80b498)
    [#4](/bitcoin-bitcoin/4/) CZMQNotificationInterface::BlockConnected(ChainstateRole, std::__1::shared_ptr<CBlock const> const&, CBlockIndex const*)::$_0::operator()(CZMQAbstractNotifier*) const /root/bitcoin/build/src/zmq/./zmq/zmqnotificationinterface.cpp:187:30 (bitcoind+0x8081d0) (BuildId: 143d05b2727482274889d2193abfe2904ebd4be7)
    [#5](/bitcoin-bitcoin/5/) void (anonymous namespace)::TryForEachAndRemoveFailed<CZMQNotificationInterface::BlockConnected(ChainstateRole, std::__1::shared_ptr<CBlock const> const&, CBlockIndex const*)::$_0>(std::__1::list<std::__1::unique_ptr<CZMQAbstractNotifier, std::__1::default_delete<CZMQAbstractNotifier>>, std::__1::allocator<std::__1::unique_ptr<CZMQAbstractNotifier, std::__1::default_delete<CZMQAbstractNotifier>>>>&, CZMQNotificationInterface::BlockConnected(ChainstateRole, std::__1::shared_ptr<CBlock const> const&, CBlockIndex const*)::$_0 const&) /root/bitcoin/build/src/zmq/./zmq/zmqnotificationinterface.cpp:139:13 (bitcoind+0x8081d0)
    [#6](/bitcoin-bitcoin/6/) CZMQNotificationInterface::BlockConnected(ChainstateRole, std::__1::shared_ptr<CBlock const> const&, CBlockIndex const*) /root/bitcoin/build/src/zmq/./zmq/zmqnotificationinterface.cpp:186:9 (bitcoind+0x8081d0)
    [#7](/bitcoin-bitcoin/7/) ValidationSignals::BlockConnected(ChainstateRole, std::__1::shared_ptr<CBlock const> const&, CBlockIndex const*)::$_1::operator()() const::'lambda'(CValidationInterface&)::operator()(CValidationInterface&) const /root/bitcoin/build/src/./validationinterface.cpp:213:79 (bitcoind+0x57479c) (BuildId: 143d05b2727482274889d2193abfe2904ebd4be7)

fanquake merged this on Jun 11, 2025
fanquake closed this on Jun 11, 2025
maflcko deleted the branch on Jun 11, 2025

Contributors

Labels

test: Explain how to reproduce zmq:: upstream race #32703

Code Coverage & Benchmarks

Reviews