test: Explain how to reproduce zmq:: upstream race #32703

pull maflcko wants to merge 2 commits into bitcoin:master from maflcko:2506-tsan-doc changing 1 files +6 −7

maflcko commented at 2:41 PM on June 8, 2025: member

An explanation makes it easier to reproduce, if needed.
test: Explain how to reproduce zmq:: upstream race fa4b659dcd
DrahtBot commented at 2:41 PM on June 8, 2025: contributor



The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.



Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/32703.



Reviews

See the guideline for information on the review process.

Type Reviewers

ACK fanquake

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.
DrahtBot added the label Tests on Jun 8, 2025

Type	Reviewers
ACK	fanquake

maflcko commented at 2:42 PM on June 8, 2025: member

For example, the following diff gives the following output.

diff --git a/ci/test/00_setup_env_native_tsan.sh b/ci/test/00_setup_env_native_tsan.sh
index b341adfec5..f2b72b8256 100755
--- a/ci/test/00_setup_env_native_tsan.sh
+++ b/ci/test/00_setup_env_native_tsan.sh
@@ -10,7 +10,11 @@ export CONTAINER_NAME=ci_native_tsan
 export CI_IMAGE_NAME_TAG="mirror.gcr.io/ubuntu:24.04"
 export APT_LLVM_V="20"
 export PACKAGES="clang-${APT_LLVM_V} llvm-${APT_LLVM_V} libclang-rt-${APT_LLVM_V}-dev libc++abi-${APT_LLVM_V}-dev libc++-${APT_LLVM_V}-dev python3-zmq"
-export DEP_OPTS="CC=clang-${APT_LLVM_V} CXX='clang++-${APT_LLVM_V} -stdlib=libc++'"
+export DEP_OPTS="NO_QT=1 NO_WALLET=1 CFLAGS='-O0 -g' CXXFLAGS='-O0 -g' CC='clang-${APT_LLVM_V} -O0 -g' CXX='clang++-${APT_LLVM_V} -O0 -g -stdlib=libc++'"
 export GOAL="install"
+export RUN_UNIT_TESTS="false"
+export TEST_RUNNER_EXTRA=interface_zmq
 export BITCOIN_CONFIG="-DWITH_ZMQ=ON -DSANITIZERS=thread \
+-DAPPEND_CXXFLAGS='-O0 -g' \
+-DAPPEND_CFLAGS='-O0 -g' \
 -DAPPEND_CPPFLAGS='-DARENA_DEBUG -DDEBUG_LOCKORDER -DDEBUG_LOCKCONTENTION -D_LIBCPP_REMOVE_TRANSITIVE_INCLUDES'"
diff --git a/test/sanitizer_suppressions/tsan b/test/sanitizer_suppressions/tsan
index 298ad175da..884b47c4b2 100644
--- a/test/sanitizer_suppressions/tsan
+++ b/test/sanitizer_suppressions/tsan
@@ -6,7 +6,6 @@
 # race (TODO fix)
 race:LoadWallet
 race:DatabaseBatch
-race:zmq::*
 race:bitcoin-qt
 
 # deadlock (TODO fix)

 node0 stderr ==================
WARNING: ThreadSanitizer: data race (pid=9622)
  Read of size 8 at 0x72640012fc40 by thread T19:
    [#0](/bitcoin-bitcoin/0/) memcpy <null> (bitcoind+0xcdc72) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#1](/bitcoin-bitcoin/1/) zmq::ypipe_t<zmq::command_t, 16>::read(zmq::command_t*) /usr/src/ypipe.hpp:134:17 (bitcoind+0x9c9ef4) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)

  Previous write of size 8 at 0x72640012fc40 by thread T20 (mutexes: write M0):
    [#0](/bitcoin-bitcoin/0/) memcpy <null> (bitcoind+0xcdc72) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#1](/bitcoin-bitcoin/1/) zmq::ypipe_t<zmq::command_t, 16>::write(zmq::command_t const&, bool) /usr/src/ypipe.hpp:50:24 (bitcoind+0x9c9de0) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)

  Location is heap block of size 1088 at 0x72640012fc00 allocated by thread T20:
    [#0](/bitcoin-bitcoin/0/) posix_memalign <null> (bitcoind+0xd21d7) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#1](/bitcoin-bitcoin/1/) zmq::yqueue_t<zmq::command_t, 16, 64ul>::allocate_chunk() /usr/src/yqueue.hpp:160:13 (bitcoind+0x9ca4da) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)

  Mutex M0 (0x722c00000128) created at:
    [#0](/bitcoin-bitcoin/0/) pthread_mutex_init <null> (bitcoind+0xd4260) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#1](/bitcoin-bitcoin/1/) zmq::mutex_t::mutex_t() /usr/src/mutex.hpp:93:14 (bitcoind+0x9b8886) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#2](/bitcoin-bitcoin/2/) CZMQNotificationInterface::Initialize() /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/zmq/./zmq/zmqnotificationinterface.cpp:105:23 (bitcoind+0x9a3275) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#3](/bitcoin-bitcoin/3/) CZMQNotificationInterface::Create(std::__1::function<bool (std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>&, CBlockIndex const&)>) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/zmq/./zmq/zmqnotificationinterface.cpp:78:36 (bitcoind+0x9a2cc6) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#4](/bitcoin-bitcoin/4/) AppInitMain(node::NodeContext&, interfaces::BlockAndHeaderTipInfo*) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./init.cpp:1624:36 (bitcoind+0x178157) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#5](/bitcoin-bitcoin/5/) AppInit(node::NodeContext&) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./bitcoind.cpp:237:43 (bitcoind+0x158882) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#6](/bitcoin-bitcoin/6/) main /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./bitcoind.cpp:283:10 (bitcoind+0x158882)

  Thread T19 'ZMQbg/Reaper' (tid=9642, running) created by main thread at:
    [#0](/bitcoin-bitcoin/0/) pthread_create <null> (bitcoind+0xd287e) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#1](/bitcoin-bitcoin/1/) zmq::thread_t::start(void (*)(void*), void*, char const*) /usr/src/thread.cpp:240:14 (bitcoind+0x9edc56) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#2](/bitcoin-bitcoin/2/) CZMQNotificationInterface::Initialize() /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/zmq/./zmq/zmqnotificationinterface.cpp:105:23 (bitcoind+0x9a3275) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#3](/bitcoin-bitcoin/3/) CZMQNotificationInterface::Create(std::__1::function<bool (std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>&, CBlockIndex const&)>) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/zmq/./zmq/zmqnotificationinterface.cpp:78:36 (bitcoind+0x9a2cc6) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#4](/bitcoin-bitcoin/4/) AppInitMain(node::NodeContext&, interfaces::BlockAndHeaderTipInfo*) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./init.cpp:1624:36 (bitcoind+0x178157) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#5](/bitcoin-bitcoin/5/) AppInit(node::NodeContext&) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./bitcoind.cpp:237:43 (bitcoind+0x158882) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#6](/bitcoin-bitcoin/6/) main /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./bitcoind.cpp:283:10 (bitcoind+0x158882)

  Thread T20 'ZMQbg/IO/0' (tid=9643, running) created by main thread at:
    [#0](/bitcoin-bitcoin/0/) pthread_create <null> (bitcoind+0xd287e) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#1](/bitcoin-bitcoin/1/) zmq::thread_t::start(void (*)(void*), void*, char const*) /usr/src/thread.cpp:240:14 (bitcoind+0x9edc56) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#2](/bitcoin-bitcoin/2/) CZMQNotificationInterface::Initialize() /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/zmq/./zmq/zmqnotificationinterface.cpp:105:23 (bitcoind+0x9a3275) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#3](/bitcoin-bitcoin/3/) CZMQNotificationInterface::Create(std::__1::function<bool (std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>&, CBlockIndex const&)>) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/zmq/./zmq/zmqnotificationinterface.cpp:78:36 (bitcoind+0x9a2cc6) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#4](/bitcoin-bitcoin/4/) AppInitMain(node::NodeContext&, interfaces::BlockAndHeaderTipInfo*) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./init.cpp:1624:36 (bitcoind+0x178157) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#5](/bitcoin-bitcoin/5/) AppInit(node::NodeContext&) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./bitcoind.cpp:237:43 (bitcoind+0x158882) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221)
    [#6](/bitcoin-bitcoin/6/) main /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./bitcoind.cpp:283:10 (bitcoind+0x158882)

SUMMARY: ThreadSanitizer: data race (/ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/bin/bitcoind+0xcdc72) (BuildId: dfd87f09345b71955cfcecb9e703b6c783948221) in memcpy
================== 

TEST             | STATUS    | DURATION

interface_zmq.py | ✖ Failed  | 5 s

test: Remove intermittent and presumed fixed tsan race suppressions

LoadWallet was added in commit d77170d52687c3d708daffb7976c3e0d345ff7dc, which
points to a traceback with BerkeleyBatch in it. Now that BDB is removed,
this can be removed as well.

The race in DatabaseBatch was added in commit
fd59670642f511523570607b752ae409b5e04e7d, which does not point to a
traceback. This was likely also fixed with the BDB removal.

If not, the suppressions should be added back, mentioning that they are
intermittent and including a traceback and possibly steps to reproduce.

fa0b766f43

maflcko commented at 2:31 PM on June 9, 2025: member

Added a commit that removes intermittent, and presumed fixed BDB suppressions. (explanation in the commit message)
fanquake approved

fanquake commented at 3:55 PM on June 11, 2025: member

ACK fa0b766f43df2ead7413363941d74244a55f090c

 test  2025-06-11T15:52:02.512422Z TestFramework (ERROR): 

 node0 stderr ==================
WARNING: ThreadSanitizer: data race (pid=175810)
  Read of size 8 at 0xffffb1e01e40 by thread T1:
    [#0](/bitcoin-bitcoin/0/) memcpy <null> (bitcoind+0xd18ac) (BuildId: 143d05b2727482274889d2193abfe2904ebd4be7)
    [#1](/bitcoin-bitcoin/1/) zmq::ypipe_t<zmq::command_t, 16>::read(zmq::command_t*) /usr/src/ypipe.hpp:134:17 (bitcoind+0x82ef98) (BuildId: 143d05b2727482274889d2193abfe2904ebd4be7)
    [#2](/bitcoin-bitcoin/2/) CZMQAbstractPublishNotifier::SendZmqMessage(char const*, void const*, unsigned long) /root/bitcoin/build/src/zmq/./zmq/zmqpublishnotifier.cpp:210:14 (bitcoind+0x80b498) (BuildId: 143d05b2727482274889d2193abfe2904ebd4be7)
    [#3](/bitcoin-bitcoin/3/) CZMQPublishHashTransactionNotifier::NotifyTransaction(CTransaction const&) /root/bitcoin/build/src/zmq/./zmq/zmqpublishnotifier.cpp:239:12 (bitcoind+0x80b498)
    [#4](/bitcoin-bitcoin/4/) CZMQNotificationInterface::BlockConnected(ChainstateRole, std::__1::shared_ptr<CBlock const> const&, CBlockIndex const*)::$_0::operator()(CZMQAbstractNotifier*) const /root/bitcoin/build/src/zmq/./zmq/zmqnotificationinterface.cpp:187:30 (bitcoind+0x8081d0) (BuildId: 143d05b2727482274889d2193abfe2904ebd4be7)
    [#5](/bitcoin-bitcoin/5/) void (anonymous namespace)::TryForEachAndRemoveFailed<CZMQNotificationInterface::BlockConnected(ChainstateRole, std::__1::shared_ptr<CBlock const> const&, CBlockIndex const*)::$_0>(std::__1::list<std::__1::unique_ptr<CZMQAbstractNotifier, std::__1::default_delete<CZMQAbstractNotifier>>, std::__1::allocator<std::__1::unique_ptr<CZMQAbstractNotifier, std::__1::default_delete<CZMQAbstractNotifier>>>>&, CZMQNotificationInterface::BlockConnected(ChainstateRole, std::__1::shared_ptr<CBlock const> const&, CBlockIndex const*)::$_0 const&) /root/bitcoin/build/src/zmq/./zmq/zmqnotificationinterface.cpp:139:13 (bitcoind+0x8081d0)
    [#6](/bitcoin-bitcoin/6/) CZMQNotificationInterface::BlockConnected(ChainstateRole, std::__1::shared_ptr<CBlock const> const&, CBlockIndex const*) /root/bitcoin/build/src/zmq/./zmq/zmqnotificationinterface.cpp:186:9 (bitcoind+0x8081d0)
    [#7](/bitcoin-bitcoin/7/) ValidationSignals::BlockConnected(ChainstateRole, std::__1::shared_ptr<CBlock const> const&, CBlockIndex const*)::$_1::operator()() const::'lambda'(CValidationInterface&)::operator()(CValidationInterface&) const /root/bitcoin/build/src/./validationinterface.cpp:213:79 (bitcoind+0x57479c) (BuildId: 143d05b2727482274889d2193abfe2904ebd4be7)

fanquake merged this on Jun 11, 2025
fanquake closed this on Jun 11, 2025
maflcko deleted the branch on Jun 11, 2025
stringintech referenced this in commit f19e7819e9 on Jul 19, 2025
alexanderwiederin referenced this in commit 9997c3d993 on Jul 25, 2025
alexanderwiederin referenced this in commit 8d6ebb9c4e on Jul 28, 2025
alexanderwiederin referenced this in commit 533a4585b5 on Jul 28, 2025
yuvicc referenced this in commit 22f55cf11d on Aug 26, 2025
bug-castercv502 referenced this in commit d4bd8a1259 on Sep 28, 2025
stickies-v referenced this in commit a19c56cd7c on Nov 4, 2025
Kino1994 referenced this in commit 183a4fcaf2 on Jun 28, 2026
BigcoinBGC referenced this in commit 7efdeea3d6 on Jun 30, 2026
bitcoin locked this on Jul 30, 2026

Contributors

Labels

test: Explain how to reproduce zmq:: upstream race #32703

Code Coverage & Benchmarks

Reviews