test: Use BasicTestingSetup when sufficient #34922

pull hodlinator wants to merge 2 commits into bitcoin:master from hodlinator:2026/03/unittest_log_opt changing 9 files +19 −19
  1. hodlinator commented at 8:39 PM on March 25, 2026: contributor

    Improves run-times (4 warmups + 60 runs each):

    test improvement
    checkqueue_tests 1.12
    merkle_tests 1.14
    orphanage_tests 1.48
    prevector_tests 1.07
    rbf_tests 1.01
    validation_tests 1.60

    Removed overhead is barely measurable on full test suite runtime sequential test_bitcoinor parallel ctest --test-dir build.

    Initial version of PR also extracted a DataDirTestingSetup from BasicTestingSetup, making the latter not use the disk, but the added complexity didn't deliver sufficient speedups to clearly justify itself (at least not while running on NVMe).

    Follow-up to #34562 (review) Arguably in a similar vein as #22086.

  2. DrahtBot added the label Tests on Mar 25, 2026
  3. DrahtBot commented at 8:39 PM on March 25, 2026: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK ryanofsky, w0xlt, l0rinc, achow101
    Stale ACK maflcko

    If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

  4. maflcko commented at 8:46 AM on March 26, 2026: member

    Interesting approach, but I wonder if all the code is worth it for a few ms of savings, that aren't usually visible at all?

  5. hodlinator commented at 10:31 AM on March 26, 2026: contributor

    Interesting approach, but I wonder if all the code is worth it for a few ms of savings, that aren't usually visible at all?

    Thanks for checking it out!

    Yeah, I'm a bit underwhelmed by the small difference as well. I've only tested on NVMe, but nobody should be running tests on bare HDD anyway.

    I've re-run benchmarks to check the difference from just the first 2 commits (replacing TestingSetup with BasicTestingSetup).

    test improvement
    checkqueue_tests 1.12
    merkle_tests 1.14
    orphanage_tests 1.48
    prevector_tests 1.07
    rbf_tests 1.01
    validation_tests 1.60

    (Not sure why orphanage_tests was measured to run even quicker, did 60 runs).

    Maybe it makes sense to shave off the latter commits and just go for the bare minimum. Although it might be interesting to also measure the effect of lighter weight BasicTestingSetup on fuzz-tests. Any ideas on how to measure that are very much welcome.

  6. maflcko commented at 10:49 AM on March 26, 2026: member

    I've re-run benchmarks to check the difference from just the first 2 commits (replacing TestingSetup with BasicTestingSetup).

    I wonder if there is a trivial heuristic to use here, so that ideally tests are self-validating and fail at runtime if their setup is too expensive. But I guess this again will be extra code, which may not be worth it.

    So maybe just pick the two commits, which should be correct, if the test passes, right?

    fuzz-tests. Any ideas on how to measure that are very much welcome.

    I think the fuzz tests use an expensive init function and run that only once, after which they can run for minutes/hours/days. Saving a few milliseconds there won't even show up when running for a minute.

  7. hodlinator force-pushed on Mar 26, 2026
  8. hodlinator renamed this:
    test: Make *TestingSetup more lightweight
    test: Use BasicTestingSetup when sufficient
    on Mar 26, 2026
  9. hodlinator commented at 11:39 AM on March 26, 2026: contributor

    I wonder if there is a trivial heuristic to use here, so that ideally tests are self-validating and fail at runtime if their setup is too expensive. But I guess this again will be extra code, which may not be worth it.

    Yeah, would be cool if one could use code coverage and some lightweight code annotation to figure out unused test dependencies.

    So maybe just pick the two commits, which should be correct, if the test passes, right?

    Done!

    I think the fuzz tests use an expensive init function and run that only once, after which they can run for minutes/hours/days. Saving a few milliseconds there won't even show up when running for a minute.

    True, it would only be somewhat measurable when doing something like running ./build_fuzz/test/fuzz/test_runner.py without --empty_min_time which uses -runs=1, and that's not very interesting.

    I'm a bit mystified by why I felt such an inclination towards optimizing this. It's not like it's helping end-users running nodes on HDD. At least it made me get a feel for how cheap the NodeContext used by BasicTestingSetup actually is.

  10. maflcko commented at 6:41 PM on March 26, 2026: member

    lgtm, should be fine when the tests pass. If there was an issue, I'd expect a nullptr deref.

    review ACK 794ba77adafcb2d0ba138cc17da5f6bd1ff38c0e 😼

    <details><summary>Show signature</summary>

    Signature:

    untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
trusted comment: review ACK 794ba77adafcb2d0ba138cc17da5f6bd1ff38c0e 😼
yHFa8OqrKx1adaEIY/R4OOCeVcFBsS+1rvX/qktzt7Vs0zL0GQFo0nWNUSG0Q96byNIglhcpxs3U5V7UvpbnAg==

    </details>

  11. stickies-v commented at 7:13 PM on March 26, 2026: contributor

    ~0. Looks like a nice little clean up but I'm not sure how I can make it trivial to assure myself that the TestingSetup and ChainTestingSetup constructors didn't do any initialization that might affect the tests, and I'm not sure it's worth digging through the code for such small gains? So I don't think I'll revisit this until the benefits obviously outweight the costs.

    edit: to be clear, I'm not against this PR. If people more familiar with the affected code find this trivial to review, it absolutely should make progress. For me, the trade-off just isn't there.

  12. w0xlt commented at 8:32 PM on March 26, 2026: contributor

    lgtm ACK 794ba77adafcb2d0ba138cc17da5f6bd1ff38c0e

  13. hodlinator commented at 8:59 PM on March 26, 2026: contributor

    @stickies-v, thanks for having a glance and taking the time to communicate your impression.

    I'm not sure how I can make it trivial to assure myself that the TestingSetup and ChainTestingSetup constructors didn't do any initialization that might affect the tests, and I'm not sure it's worth digging through the code for such small gains? So I don't think I'll revisit this until the benefits obviously outweight the costs.

    Here is an overview:

    • check_queue_tests.cpp - Tests properties of a fairly self-contained data structure.
    • merkle_tests.cpp - Tests the pure functions from consensus/merkle.h, the only #include outside of test headers.
    • orphanage_tests.cpp - Focuses on individual transactions, two test cases call orphanage->EraseForBlock(block) but the block's are not part of a chain.
    • prevector_tests.cpp - No comment
    • rbf_tests.cpp - All test cases override the fixture to use TestChain100Setup except the last one ("feerate_chunks_utilities"), which is trivial to review.
    • txdownload_tests.cpp - Pure refactor, uses the same *TestingSetup as on master.
    • validation_tests.cpp - These are quite clear, verifying different properties of consensus / chain params. Maybe this one would be the most consequential to break. I could potentially drop it if you insist?

    It's unfortunate that the code base often uses globals rather than dependency injection. But it is generally written in a style where tests would trigger nullptr deref as pointed out in #34922 (comment). In early prototypes of what later became this PR I was even trying to make a LogTestingSetup that didn't create an ECC_Context nor select chain params, both lead to clear test failures when they were required by a test.

  14. in src/test/txdownload_tests.cpp:20 in 742309b9e2 
      16 | @@ -17,7 +17,7 @@
  17 |  
  18 |  #include <boost/test/unit_test.hpp>
  19 |  
  20 | -BOOST_FIXTURE_TEST_SUITE(txdownload_tests, TestingSetup)
  21 | +BOOST_FIXTURE_TEST_SUITE(txdownload_tests, TestChain100Setup)
    furszy commented at 12:45 AM on March 27, 2026:

    I'm not sure about this initial commit. It looks like it could be overlooked, causing any new test in this file to unnecessarily generate the chain when they might not need it.

    hodlinator commented at 9:26 AM on March 27, 2026:

    True, there is a window between this PR being created and merged in which someone could start adding a test case to this file and we end up with a kind of silent merge-conflict. As you say, in the unlikely event that it does happen, the net result would be that the added test case(s) unnecessarily generate chains. I guess there's also a very small risk that it would lead to false positives in tests.

    Have you seen anyone working on tests in this area lately?

    furszy commented at 1:44 PM on March 27, 2026:

    I wasn't talking about the merge window. It was more like: "When writing a test, I rarely check the base class used in BOOST_FIXTURE_TEST_SUITE, which in this case means the new test would create an extra chain for no reason".

    I'm not strong on this, it just seems slightly better to be specific when using TestChain100Setup. It is also more descriptive for the particular test case.

    hodlinator commented at 8:58 PM on March 27, 2026:

    Hm... sounds like it might be worth adding a linter to prevent test suites from directly specifying the costly TestChain100Setup at that level, instead requiring it to be specified on the test case level? Only found 2 violations on relatively recent commit.

    â‚¿ git grep "BOOST_FIXTURE_TEST_SUITE.*TestChain100Setup"
src/test/disconnected_transactions.cpp:BOOST_FIXTURE_TEST_SUITE(disconnected_transactions, TestChain100Setup)
src/test/interfaces_tests.cpp:BOOST_FIXTURE_TEST_SUITE(interfaces_tests, TestChain100Setup)
    hodlinator commented at 8:02 PM on April 7, 2026:

    Opted to move TestChain100Setup-specification into test cases for those two test files as well, but skipped adding a linter for now.

  15. ryanofsky approved
  16. ryanofsky commented at 4:49 PM on April 7, 2026: contributor

    Code review ACK 794ba77adafcb2d0ba138cc17da5f6bd1ff38c0e. Seems good to skip unnecessary setup.

    It does look like the first commit and second commit are working at cross purposes, because second commit avoids slow test setup while first commit increases likelihood of slow test setup. My suggestion for resolving this would be to use BOOST_AUTO_TEST_SUITE(txdownload_tests) in the first commit and leave two test cases there alone.

    Would be good to know from @hodlinator if planning more changes here like following up on #34922 (review)

    re: #34922#pullrequestreview-4016770383

    I'm not sure how I can make it trivial to assure myself that the TestingSetup and ChainTestingSetup constructors didn't do any initialization that might affect the tests

    I was also puzzled by this comment because it would seem like in order for this to be true the tests would either have to be flaky or not clearly documenting their prerequisites, and in both cases would seem better to fix.

  17. refactor(test): Only specify TestChain100Setup in test cases
    Reduces risk for accidental use of costly TestChain100Setup in later added test cases.
    9ee77701dd
  18. test: Use BasicTestingSetup when TestingSetup is not necessary 2af003ae37
  19. hodlinator force-pushed on Apr 7, 2026
  20. hodlinator commented at 8:16 PM on April 7, 2026: contributor

    Agree with the feedback regarding the first commit, it has now been changed to make the repo only specify TestChain100Setup on a test case level.

    re #34922#pullrequestreview-4069848503:

    I'm not sure how I can make it trivial to assure myself that the TestingSetup and ChainTestingSetup constructors didn't do any initialization that might affect the tests

    I was also puzzled by this comment because it would seem like in order for this to be true the tests would either have to be flaky or not clearly documenting their prerequisites, and in both cases would seem better to fix.

    I must confess that one thing that did "silently" break while I was experimenting with a smaller version of BasicTestingSetup was that SHA256 was not configured to use SIMD since I skipped over kernel::Context creation which calls SHA256AutoDetect() (and RandomInit()). The effect of that was tests mysteriously running slower with a more minimal fixture.

  21. ryanofsky approved
  22. ryanofsky commented at 4:39 PM on April 8, 2026: contributor

    Code review ACK 2af003ae37206f6d20b6458042a954a12be815e5. Changes seem good: switching to minimum required test fixtures and avoiding expensive TestChain100Setup usage for whole fixtures.

    If we are worried about expense of TestChain100Setup more generally, maybe we could make it a normal class instead of a fixture, and use it as a local variable like BOOST_AUTO_TEST_CASE(name) { TestChain100Setup setup{m_node}; ... }.

    re: #34922#pullrequestreview-4070951917

    I must confess that one thing that did "silently" break while I was experimenting with a smaller version of BasicTestingSetup was that SHA256 was not configured to use SIMD since I skipped over kernel::Context creation which calls SHA256AutoDetect() (and RandomInit()). The effect of that was tests mysteriously running slower with a more minimal fixture.

    This is concerning. Maybe there should be an Assume-like check in debug builds checking that hash/random functions are not called before state is initialized.

  23. DrahtBot requested review from maflcko on Apr 8, 2026
  24. DrahtBot requested review from w0xlt on Apr 8, 2026
  25. w0xlt commented at 5:44 PM on April 8, 2026: contributor

    ACK 2af003ae37206f6d20b6458042a954a12be815e5

  26. l0rinc commented at 3:45 PM on April 9, 2026: contributor

    ACK 2af003ae37206f6d20b6458042a954a12be815e5

    I haven't seen the earlier revisions, but the current changes look safe to me for replacing the affected suite fixtures.

    ~We could probably extend this to a few more suites (e.g. denialofservice_tests, miniminer_tests, validation_chainstatemanager_tests, validation_chainstate_tests), but it's also fine to leave those for follow-up work.~

    <details><summary>Remaining tests</summary>

    diff --git a/src/test/denialofservice_tests.cpp b/src/test/denialofservice_tests.cpp
index 3d002637b7..776a18788e 100644
--- a/src/test/denialofservice_tests.cpp
+++ b/src/test/denialofservice_tests.cpp
@@ -32,7 +32,7 @@ static CService ip(uint32_t i)
     return CService(CNetAddr(s), Params().GetDefaultPort());
 }

-BOOST_AUTO_TEST_SUITE(denialofservice_tests)
+BOOST_FIXTURE_TEST_SUITE(denialofservice_tests, TestingSetup)

 // Test eviction of an outbound peer whose chain never advances
 // Mock a node connection, and use mocktime to simulate a peer
@@ -42,7 +42,7 @@ BOOST_AUTO_TEST_SUITE(denialofservice_tests)
 // this logic; this test takes advantage of that protection only
 // being applied to nodes which send headers with sufficient
 // work.
-BOOST_FIXTURE_TEST_CASE(outbound_slow_chain_eviction, TestingSetup)
+BOOST_AUTO_TEST_CASE(outbound_slow_chain_eviction)
 {
     LOCK(NetEventsInterface::g_msgproc_mutex);

@@ -298,7 +298,7 @@ BOOST_FIXTURE_TEST_CASE(block_relay_only_eviction, OutboundTest)
     connman->ClearTestNodes();
 }

-BOOST_FIXTURE_TEST_CASE(peer_discouragement, TestingSetup)
+BOOST_AUTO_TEST_CASE(peer_discouragement)
 {
     LOCK(NetEventsInterface::g_msgproc_mutex);

@@ -402,7 +402,7 @@ BOOST_FIXTURE_TEST_CASE(peer_discouragement, TestingSetup)
     connman->ClearTestNodes();
 }

-BOOST_FIXTURE_TEST_CASE(DoS_bantime, TestingSetup)
+BOOST_AUTO_TEST_CASE(DoS_bantime)
 {
     LOCK(NetEventsInterface::g_msgproc_mutex);

diff --git a/src/test/miniminer_tests.cpp b/src/test/miniminer_tests.cpp
index 482fea2e15..de4a310a04 100644
--- a/src/test/miniminer_tests.cpp
+++ b/src/test/miniminer_tests.cpp
@@ -13,7 +13,7 @@
 #include <optional>
 #include <vector>

-BOOST_AUTO_TEST_SUITE(miniminer_tests)
+BOOST_FIXTURE_TEST_SUITE(miniminer_tests, TestingSetup)

 const CAmount low_fee{CENT/2000}; // 500 ṩ
 const CAmount med_fee{CENT/200}; // 5000 ṩ
diff --git a/src/test/validation_chainstate_tests.cpp b/src/test/validation_chainstate_tests.cpp
index cb3a04ef9a..2461cdc740 100644
--- a/src/test/validation_chainstate_tests.cpp
+++ b/src/test/validation_chainstate_tests.cpp
@@ -23,11 +23,11 @@

 #include <boost/test/unit_test.hpp>

-BOOST_FIXTURE_TEST_SUITE(validation_chainstate_tests, ChainTestingSetup)
+BOOST_AUTO_TEST_SUITE(validation_chainstate_tests)

 //! Test resizing coins-related Chainstate caches during runtime.
 //!
-BOOST_AUTO_TEST_CASE(validation_chainstate_resize_caches)
+BOOST_FIXTURE_TEST_CASE(validation_chainstate_resize_caches, ChainTestingSetup)
 {
     ChainstateManager& manager = *Assert(m_node.chainman);
     CTxMemPool& mempool = *Assert(m_node.mempool);
diff --git a/src/test/validation_chainstatemanager_tests.cpp b/src/test/validation_chainstatemanager_tests.cpp
index 1a0d5f84dc..67c90a0073 100644
--- a/src/test/validation_chainstatemanager_tests.cpp
+++ b/src/test/validation_chainstatemanager_tests.cpp
@@ -33,7 +33,7 @@ using node::BlockManager;
 using node::KernelNotifications;
 using node::SnapshotMetadata;

-BOOST_AUTO_TEST_SUITE(validation_chainstatemanager_tests)
+BOOST_FIXTURE_TEST_SUITE(validation_chainstatemanager_tests, TestingSetup)

 //! Basic tests for ChainstateManager.
 //!

    </details>

    I ran the affected tests, including the additional suites mentioned above, before and after the change. These were my measurements and results:

    <details><summary>Unit test benchmark - separate</summary>

    COMMITS="141fbe4d530b51345e62dee1348e82d8a0406ffc cf6de306d7801a0e17b7231b840e6bae831b4deb" && \
git reset --hard && git clean -fxd && \
(echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done) && \
for c in $COMMITS; do \
  git checkout "$c" && \
  cmake -B "build-$c" -DCMAKE_BUILD_TYPE=Release && \
  cmake --build "build-$c" --target test_bitcoin -j"$(nproc)" || exit 1; \
done && \
(echo "" && echo "$(date -I) | unit tests | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM"; echo "") && \
hyperfine \
  --sort command \
  --runs 50 \
  --parameter-list COMMIT ${COMMITS// /,} \
  --parameter-list SUITE checkqueue_tests,denialofservice_tests,disconnected_transactions,interfaces_tests,merkle_tests,miniminer_tests,orphanage_tests,prevector_tests,rbf_tests,txdownload_tests,validation_chainstatemanager_tests,validation_tests,validation_chainstate_tests \
  "./build-{COMMIT}/bin/test_bitcoin --run_test='{SUITE}/*' --log_level=nothing"

    </details>

    <details><summary>2026-04-09 | unit tests | i9-ssd | x86_64 | Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz | 16 cores | 62Gi RAM</summary>

    Benchmark 1: ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='checkqueue_tests/*' --log_level=nothing
  Time (mean ± σ):      1.086 s ±  0.134 s    [User: 1.599 s, System: 1.107 s]
  Range (min … max):    0.866 s …  1.304 s    50 runs

Benchmark 2: ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='checkqueue_tests/*' --log_level=nothing
  Time (mean ± σ):     908.9 ms ± 124.8 ms    [User: 1559.5 ms, System: 1004.3 ms]
  Range (min … max):   694.2 ms … 1137.1 ms    50 runs

Benchmark 3: ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='denialofservice_tests/*' --log_level=nothing
  Time (mean ± σ):     239.2 ms ±   0.6 ms    [User: 143.5 ms, System: 96.2 ms]
  Range (min … max):   238.2 ms … 240.4 ms    50 runs

Benchmark 4: ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='denialofservice_tests/*' --log_level=nothing
  Time (mean ± σ):     239.4 ms ±   0.6 ms    [User: 143.1 ms, System: 96.8 ms]
  Range (min … max):   238.0 ms … 240.2 ms    50 runs

Benchmark 5: ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='disconnected_transactions/*' --log_level=nothing
  Time (mean ± σ):     167.7 ms ±   0.4 ms    [User: 141.9 ms, System: 30.6 ms]
  Range (min … max):   166.9 ms … 168.8 ms    50 runs

Benchmark 6: ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='disconnected_transactions/*' --log_level=nothing
  Time (mean ± σ):     167.8 ms ±   0.5 ms    [User: 143.3 ms, System: 29.2 ms]
  Range (min … max):   166.8 ms … 169.2 ms    50 runs

Benchmark 7: ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='interfaces_tests/*' --log_level=nothing
  Time (mean ± σ):     443.3 ms ±   2.6 ms    [User: 316.2 ms, System: 156.6 ms]
  Range (min … max):   438.8 ms … 449.6 ms    50 runs

Benchmark 8: ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='interfaces_tests/*' --log_level=nothing
  Time (mean ± σ):     444.5 ms ±   2.7 ms    [User: 315.8 ms, System: 158.1 ms]
  Range (min … max):   439.6 ms … 452.9 ms    50 runs

Benchmark 9: ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='merkle_tests/*' --log_level=nothing
  Time (mean ± σ):     723.4 ms ±  58.1 ms    [User: 611.8 ms, System: 112.1 ms]
  Range (min … max):   601.4 ms … 840.5 ms    50 runs

Benchmark 10: ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='merkle_tests/*' --log_level=nothing
  Time (mean ± σ):     595.6 ms ±  73.5 ms    [User: 589.8 ms, System: 5.6 ms]
  Range (min … max):   448.7 ms … 828.8 ms    50 runs

Benchmark 11: ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='miniminer_tests/*' --log_level=nothing
  Time (mean ± σ):     384.4 ms ±   2.5 ms    [User: 290.5 ms, System: 117.5 ms]
  Range (min … max):   380.1 ms … 393.8 ms    50 runs

Benchmark 12: ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='miniminer_tests/*' --log_level=nothing
  Time (mean ± σ):     384.3 ms ±   3.9 ms    [User: 291.7 ms, System: 116.2 ms]
  Range (min … max):   379.9 ms … 407.2 ms    50 runs

Benchmark 13: ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='orphanage_tests/*' --log_level=nothing
  Time (mean ± σ):     294.5 ms ±  12.0 ms    [User: 199.2 ms, System: 96.2 ms]
  Range (min … max):   263.8 ms … 330.2 ms    50 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 14: ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='orphanage_tests/*' --log_level=nothing
  Time (mean ± σ):     168.4 ms ±   0.3 ms    [User: 162.3 ms, System: 6.1 ms]
  Range (min … max):   167.9 ms … 170.2 ms    50 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 15: ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='prevector_tests/*' --log_level=nothing
  Time (mean ± σ):     263.0 ms ±   1.2 ms    [User: 239.9 ms, System: 23.1 ms]
  Range (min … max):   260.4 ms … 265.3 ms    50 runs

Benchmark 16: ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='prevector_tests/*' --log_level=nothing
  Time (mean ± σ):     234.8 ms ±   1.3 ms    [User: 230.2 ms, System: 4.5 ms]
  Range (min … max):   232.9 ms … 239.2 ms    50 runs

Benchmark 17: ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='rbf_tests/*' --log_level=nothing
  Time (mean ± σ):     329.1 ms ±   6.0 ms    [User: 263.3 ms, System: 84.8 ms]
  Range (min … max):   325.2 ms … 369.0 ms    50 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 18: ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='rbf_tests/*' --log_level=nothing
  Time (mean ± σ):     320.3 ms ±   4.8 ms    [User: 258.8 ms, System: 80.4 ms]
  Range (min … max):   317.2 ms … 351.2 ms    50 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 19: ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='txdownload_tests/*' --log_level=nothing
  Time (mean ± σ):     318.4 ms ±   1.5 ms    [User: 222.9 ms, System: 106.1 ms]
  Range (min … max):   315.7 ms … 322.3 ms    50 runs

Benchmark 20: ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='txdownload_tests/*' --log_level=nothing
  Time (mean ± σ):     319.0 ms ±   4.0 ms    [User: 223.3 ms, System: 106.2 ms]
  Range (min … max):   315.4 ms … 344.4 ms    50 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 21: ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='validation_chainstatemanager_tests/*' --log_level=nothing
  Time (mean ± σ):     777.7 ms ±  13.3 ms    [User: 611.0 ms, System: 240.6 ms]
  Range (min … max):   766.0 ms … 828.6 ms    50 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 22: ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='validation_chainstatemanager_tests/*' --log_level=nothing
  Time (mean ± σ):     777.6 ms ±   8.1 ms    [User: 607.2 ms, System: 244.3 ms]
  Range (min … max):   766.8 ms … 802.5 ms    50 runs

Benchmark 23: ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='validation_tests/*' --log_level=nothing
  Time (mean ± σ):     242.4 ms ±   5.2 ms    [User: 147.9 ms, System: 95.0 ms]
  Range (min … max):   240.1 ms … 277.9 ms    50 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 24: ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='validation_tests/*' --log_level=nothing
  Time (mean ± σ):     117.8 ms ±   0.1 ms    [User: 113.0 ms, System: 4.7 ms]
  Range (min … max):   117.4 ms … 118.2 ms    50 runs

Benchmark 25: ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='validation_chainstate_tests/*' --log_level=nothing
  Time (mean ± σ):     239.0 ms ±   0.9 ms    [User: 183.4 ms, System: 65.7 ms]
  Range (min … max):   237.6 ms … 240.4 ms    50 runs

Benchmark 26: ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='validation_chainstate_tests/*' --log_level=nothing
  Time (mean ± σ):     240.0 ms ±   5.5 ms    [User: 182.8 ms, System: 67.3 ms]
  Range (min … max):   237.6 ms … 277.3 ms    50 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Relative speed comparison
        9.22 ±  1.14  ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='checkqueue_tests/*' --log_level=nothing
        7.72 ±  1.06  ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='checkqueue_tests/*' --log_level=nothing
        2.03 ±  0.01  ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='denialofservice_tests/*' --log_level=nothing
        2.03 ±  0.01  ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='denialofservice_tests/*' --log_level=nothing
        1.42 ±  0.00  ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='disconnected_transactions/*' --log_level=nothing
        1.42 ±  0.00  ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='disconnected_transactions/*' --log_level=nothing
        3.76 ±  0.02  ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='interfaces_tests/*' --log_level=nothing
        3.77 ±  0.02  ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='interfaces_tests/*' --log_level=nothing
        6.14 ±  0.49  ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='merkle_tests/*' --log_level=nothing
        5.06 ±  0.62  ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='merkle_tests/*' --log_level=nothing
        3.26 ±  0.02  ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='miniminer_tests/*' --log_level=nothing
        3.26 ±  0.03  ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='miniminer_tests/*' --log_level=nothing
        2.50 ±  0.10  ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='orphanage_tests/*' --log_level=nothing
        1.43 ±  0.00  ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='orphanage_tests/*' --log_level=nothing
        2.23 ±  0.01  ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='prevector_tests/*' --log_level=nothing
        1.99 ±  0.01  ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='prevector_tests/*' --log_level=nothing
        2.79 ±  0.05  ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='rbf_tests/*' --log_level=nothing
        2.72 ±  0.04  ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='rbf_tests/*' --log_level=nothing
        2.70 ±  0.01  ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='txdownload_tests/*' --log_level=nothing
        2.71 ±  0.03  ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='txdownload_tests/*' --log_level=nothing
        6.60 ±  0.11  ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='validation_chainstatemanager_tests/*' --log_level=nothing
        6.60 ±  0.07  ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='validation_chainstatemanager_tests/*' --log_level=nothing
        2.06 ±  0.04  ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='validation_tests/*' --log_level=nothing
        1.00          ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='validation_tests/*' --log_level=nothing
        2.03 ±  0.01  ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --run_test='validation_chainstate_tests/*' --log_level=nothing
        2.04 ±  0.05  ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --run_test='validation_chainstate_tests/*' --log_level=nothing

    </details>

    <img width="2084" height="882" alt="image" src="https://github.com/user-attachments/assets/55738a29-be36-4db1-87e7-bdd05dd6982b" />

    Test Suite Before (ms) After (ms) Change
    checkqueue_tests 1086.0 ± 134.0 908.9 ± 124.8 -16.3%
    denialofservice_tests 239.2 ± 0.6 239.4 ± 0.6 +0.1%
    disconnected_transactions 167.7 ± 0.4 167.8 ± 0.5 +0.1%
    interfaces_tests 443.3 ± 2.6 444.5 ± 2.7 +0.3%
    merkle_tests 723.4 ± 58.1 595.6 ± 73.5 -17.7%
    miniminer_tests 384.4 ± 2.5 384.3 ± 3.9 -0.0%
    orphanage_tests 294.5 ± 12.0 168.4 ± 0.3 -42.8%
    prevector_tests 263.0 ± 1.2 234.8 ± 1.3 -10.7%
    rbf_tests 329.1 ± 6.0 320.3 ± 4.8 -2.7%
    txdownload_tests 318.4 ± 1.5 319.0 ± 4.0 +0.2%
    validation_chainstate_tests 239.0 ± 0.9 240.0 ± 5.5 +0.4%
    validation_chainstatemanager_tests 777.7 ± 13.3 777.6 ± 8.1 -0.0%
    validation_tests 242.4 ± 5.2 117.8 ± 0.1 -51.4%

    <details><summary>Unit test benchmark - overall</summary>

    COMMITS="141fbe4d530b51345e62dee1348e82d8a0406ffc cf6de306d7801a0e17b7231b840e6bae831b4deb" && \
git reset --hard && git clean -fxd && \
(echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done) && \
for c in $COMMITS; do \
  git checkout "$c" && \
  cmake -B "build-$c" -DCMAKE_BUILD_TYPE=Release && \
  cmake --build "build-$c" --target test_bitcoin -j"$(nproc)" || exit 1; \
done && \
(echo "" && echo "$(date -I) | unit tests | sequential full test_bitcoin run | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) cores | $(free -h | awk '/^Mem:/{print $2}') RAM"; echo "") && \
hyperfine \
  --sort command \
  --runs 5 \
  --parameter-list COMMIT ${COMMITS// /,} \
  "./build-{COMMIT}/bin/test_bitcoin --log_level=nothing"

    </details>

    <details><summary>2026-04-09 | unit tests | sequential full test_bitcoin run | i9-ssd | x86_64 | Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz | 16 cores | 62Gi RAM</summary>

    Benchmark 1: ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --log_level=nothing
  Time (mean ± σ):     65.603 s ±  2.203 s    [User: 62.613 s, System: 3.110 s]
  Range (min … max):   62.742 s … 68.054 s    5 runs
 
Benchmark 2: ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --log_level=nothing
  Time (mean ± σ):     63.355 s ±  0.496 s    [User: 60.482 s, System: 2.843 s]
  Range (min … max):   62.495 s … 63.706 s    5 runs
 
Relative speed comparison
        1.04 ±  0.04  ./build-141fbe4d530b51345e62dee1348e82d8a0406ffc/bin/test_bitcoin --log_level=nothing
        1.00          ./build-cf6de306d7801a0e17b7231b840e6bae831b4deb/bin/test_bitcoin --log_level=nothing

    </details>

    Slightly more stable/predictable runtime and a bit faster overall execution: <img width="884" height="731" alt="image" src="https://github.com/user-attachments/assets/1f8215d0-7976-4dd1-954f-c72c4318900f" />

  27. achow101 commented at 7:38 PM on April 9, 2026: member

    ACK 2af003ae37206f6d20b6458042a954a12be815e5

  28. achow101 merged this on Apr 9, 2026
  29. achow101 closed this on Apr 9, 2026
  30. hodlinator deleted the branch on Apr 9, 2026
  31. l0rinc commented at 8:11 PM on April 9, 2026: contributor

    For the record, the diff I posted above used the wrong base so the PR already contained most of my suggestions - no follow-up needed :)

  32. hodlinator commented at 8:14 PM on April 9, 2026: contributor

    re #34922#pullrequestreview-4076712895:

    If we are worried about expense of TestChain100Setup more generally, maybe we could make it a normal class instead of a fixture, and use it as a local variable like BOOST_AUTO_TEST_CASE(name) { TestChain100Setup setup{m_node}; ... }.

    That seems like an elegant approach!

    This is concerning. Maybe there should be an Assume-like check in debug builds checking that hash/random functions are not called before state is initialized.

    Strongly agree.

    re #34922 (comment):

    Thanks for the thorough benchmarks and exploration!

    I'm satisfied the measurements confirm that the tests I've chosen to downgrade the *TestingSetup type for are the ones which benefit the most. Since there have been concerns about introducing false positives I'd rather avoid altering the other tests in this way.

    Happy you managed to clearly measure an overall speedup when running synchronously with your environment!!

Contributors
hodlinatorDrahtBotmaflckostickies-vw0xltfurszyryanofskyl0rincachow101


maflcko

Labels
Tests
Linked (view graph)
#22086 test: remove BasicTestingSetup from unit tests that don't need it#34562 ThreadPool follow-ups, proactive shutdown and HasReason dependency cleanup
github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-04-21 15:12 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me