test: Run bench sanity checks in parallel with functional tests #33142

The ctest target bench_sanity_check has many issues:

• With sanitizers enabled, it is one of the slowest targets, often taking several minutes. See #32770 (comment).
• There is no insight from ctest into how long each individual sanity check takes.
• On a timeout, or OOM issue, there is no insight into which sub-bench failed. The failure will generally just look like 75/153 Test [#9](/bitcoin-bitcoin/9/): bench_sanity_check ...................***Failed 770.84 sec out of memory
• Places that can't use ctest (like the Windows-cross CI task) have to explicitly run it, or risk forgetting to run it.
• All benchmarks are run sequentially, when they could run in parallel instead.

Both issues can lead to CI timeouts and leave CPU unused during testing.

Fix all issues by running it as part of the functional tests instead. This is similar to the rpcauth tests (https://github.com/bitcoin/bitcoin/pull/32881) and util tests [bitcoin-tx, and bitcoin-util] (https://github.com/bitcoin/bitcoin/pull/32697).

<!--e57a25ab6845829454e8d69fc972939a-->

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

<!--006a51241073e994b41acfe9ec718e94-->

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33142.

<!--021abf342d371248e50ceaed478a90ca-->

Reviews

See the guideline for information on the review process.

If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

<!--174a7506f384e20aa4161008e828411d-->

Conflicts

Reviewers, this pull request conflicts with the following ones:

• #33483 (CMake: Add dynamic test discovery by purpleKarrot)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

<!--5faf32d7da4f0f540f40219e4f7537a3-->

<!--85328a0da195eb286784d51f73fa0af9-->

🚧 At least one of the CI tasks failed. _{Task lint: https://github.com/bitcoin/bitcoin/runs/47488255049} _{LLM reason (✨ experimental): The CI failure is due to errors from the Python linting check.}

<details><summary>Hints</summary>

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

• Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

• A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

• An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

</details>

<!--85328a0da195eb286784d51f73fa0af9-->

🚧 At least one of the CI tasks failed. _{Task lint: https://github.com/bitcoin/bitcoin/runs/47498178413} _{LLM reason (✨ experimental): The CI failure is caused by a lint error due to an unused import detected by ruff.}

<details><summary>Hints</summary>

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

• Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

• A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

• An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

</details>

I get an error compiling this without bench_bitcoin:

❯ cmake -B build; and cmake --build build; and ./build/test/functional/test_runner.py -j14
-- The CXX compiler identification is Clang 19.1.7
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /etc/profiles/per-user/will/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Setting build type to "RelWithDebInfo" as none was specified
-- Performing Test CXX_SUPPORTS__WERROR
-- Performing Test CXX_SUPPORTS__WERROR - Success
-- Performing Test CXX_SUPPORTS__G3
-- Performing Test CXX_SUPPORTS__G3 - Success
-- Performing Test LINKER_SUPPORTS__G3
-- Performing Test LINKER_SUPPORTS__G3 - Success
-- Performing Test CXX_SUPPORTS__FTRAPV
-- Performing Test CXX_SUPPORTS__FTRAPV - Success
-- Performing Test LINKER_SUPPORTS__FTRAPV
-- Performing Test LINKER_SUPPORTS__FTRAPV - Success
-- Found SQLite3: /nix/store/92gwa4j45skp8d096csmnj2a8jcn0q9w-sqlite-3.48.0-dev/include (found suitable version "3.48.0", minimum required is "3.7.17")
-- Performing Test LINKER_SUPPORTS__WL___FATAL_WARNINGS
-- Performing Test LINKER_SUPPORTS__WL___FATAL_WARNINGS - Success
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Performing Test NO_DIAGNOSTICS_BOOST_NO_CXX98_FUNCTION_BASE
-- Performing Test NO_DIAGNOSTICS_BOOST_NO_CXX98_FUNCTION_BASE - Failed
-- Found PkgConfig: /nix/store/2crk9xnq5x9v7yf0r2nwkgj8qsmxr4ly-pkg-config-wrapper-0.29.2/bin/pkg-config (found version "0.29.2")
-- Found Libevent: /nix/store/yai7mpy5d4rw0jvflyxdf0vzjkiqxhv6-libevent-2.1.12/lib (found suitable version "2.1.12-stable", minimum required is "2.1.8")
-- Performing Test HAVE_EVHTTP_CONNECTION_GET_PEER_CONST_CHAR
-- Performing Test HAVE_EVHTTP_CONNECTION_GET_PEER_CONST_CHAR - Failed
-- Looking for O_CLOEXEC
-- Looking for O_CLOEXEC - found
-- Looking for fdatasync
-- Looking for fdatasync - found
-- Looking for fork
-- Looking for fork - found
-- Looking for pipe2
-- Looking for pipe2 - found
-- Looking for setsid
-- Looking for setsid - found
-- Performing Test IFADDR_LINKS_WITHOUT_LIBSOCKET
-- Performing Test IFADDR_LINKS_WITHOUT_LIBSOCKET - Success
-- Performing Test STD_ATOMIC_LINKS_WITHOUT_LIBATOMIC
-- Performing Test STD_ATOMIC_LINKS_WITHOUT_LIBATOMIC - Success
-- Looking for std::system
-- Looking for std::system - found
-- Looking for ::_wsystem
-- Looking for ::_wsystem - not found
-- Performing Test STRERROR_R_CHAR_P
-- Performing Test STRERROR_R_CHAR_P - Success
-- Looking for malloc_info
-- Looking for malloc_info - found
-- Performing Test HAVE_MALLOPT_ARENA_MAX
-- Performing Test HAVE_MALLOPT_ARENA_MAX - Success
-- Performing Test HAVE_POSIX_FALLOCATE
-- Performing Test HAVE_POSIX_FALLOCATE - Success
-- Performing Test HAVE_STRONG_GETAUXVAL
-- Performing Test HAVE_STRONG_GETAUXVAL - Success
-- Performing Test HAVE_SOCKADDR_UN
-- Performing Test HAVE_SOCKADDR_UN - Success
-- Performing Test HAVE_GETRANDOM
-- Performing Test HAVE_GETRANDOM - Success
-- Performing Test HAVE_GETENTROPY_RAND
-- Performing Test HAVE_GETENTROPY_RAND - Success
-- Performing Test HAVE_SYSCTL
-- Performing Test HAVE_SYSCTL - Failed
-- Performing Test HAVE_SYSCTL_ARND
-- Performing Test HAVE_SYSCTL_ARND - Failed
-- Performing Test HAVE_SSE41
-- Performing Test HAVE_SSE41 - Success
-- Performing Test HAVE_AVX2
-- Performing Test HAVE_AVX2 - Success
-- Performing Test HAVE_X86_SHANI
-- Performing Test HAVE_X86_SHANI - Success
-- Performing Test HAVE_ARM_SHANI
-- Performing Test HAVE_ARM_SHANI - Failed
-- Performing Test CXX_SUPPORTS__WALL
-- Performing Test CXX_SUPPORTS__WALL - Success
-- Performing Test CXX_SUPPORTS__WEXTRA
-- Performing Test CXX_SUPPORTS__WEXTRA - Success
-- Performing Test CXX_SUPPORTS__WGNU
-- Performing Test CXX_SUPPORTS__WGNU - Success
-- Performing Test CXX_SUPPORTS__WFORMAT__WFORMAT_SECURITY
-- Performing Test CXX_SUPPORTS__WFORMAT__WFORMAT_SECURITY - Success
-- Performing Test CXX_SUPPORTS__WVLA
-- Performing Test CXX_SUPPORTS__WVLA - Success
-- Performing Test CXX_SUPPORTS__WSHADOW_FIELD
-- Performing Test CXX_SUPPORTS__WSHADOW_FIELD - Success
-- Performing Test CXX_SUPPORTS__WTHREAD_SAFETY
-- Performing Test CXX_SUPPORTS__WTHREAD_SAFETY - Success
-- Performing Test CXX_SUPPORTS__WTHREAD_SAFETY_POINTER
-- Performing Test CXX_SUPPORTS__WTHREAD_SAFETY_POINTER - Failed
-- Performing Test CXX_SUPPORTS__WLOOP_ANALYSIS
-- Performing Test CXX_SUPPORTS__WLOOP_ANALYSIS - Success
-- Performing Test CXX_SUPPORTS__WREDUNDANT_DECLS
-- Performing Test CXX_SUPPORTS__WREDUNDANT_DECLS - Success
-- Performing Test CXX_SUPPORTS__WUNUSED_MEMBER_FUNCTION
-- Performing Test CXX_SUPPORTS__WUNUSED_MEMBER_FUNCTION - Success
-- Performing Test CXX_SUPPORTS__WDATE_TIME
-- Performing Test CXX_SUPPORTS__WDATE_TIME - Success
-- Performing Test CXX_SUPPORTS__WCONDITIONAL_UNINITIALIZED
-- Performing Test CXX_SUPPORTS__WCONDITIONAL_UNINITIALIZED - Success
-- Performing Test CXX_SUPPORTS__WDUPLICATED_BRANCHES
-- Performing Test CXX_SUPPORTS__WDUPLICATED_BRANCHES - Failed
-- Performing Test CXX_SUPPORTS__WDUPLICATED_COND
-- Performing Test CXX_SUPPORTS__WDUPLICATED_COND - Failed
-- Performing Test CXX_SUPPORTS__WLOGICAL_OP
-- Performing Test CXX_SUPPORTS__WLOGICAL_OP - Failed
-- Performing Test CXX_SUPPORTS__WOVERLOADED_VIRTUAL
-- Performing Test CXX_SUPPORTS__WOVERLOADED_VIRTUAL - Success
-- Performing Test CXX_SUPPORTS__WSUGGEST_OVERRIDE
-- Performing Test CXX_SUPPORTS__WSUGGEST_OVERRIDE - Success
-- Performing Test CXX_SUPPORTS__WIMPLICIT_FALLTHROUGH
-- Performing Test CXX_SUPPORTS__WIMPLICIT_FALLTHROUGH - Success
-- Performing Test CXX_SUPPORTS__WUNREACHABLE_CODE
-- Performing Test CXX_SUPPORTS__WUNREACHABLE_CODE - Success
-- Performing Test CXX_SUPPORTS__WDOCUMENTATION
-- Performing Test CXX_SUPPORTS__WDOCUMENTATION - Success
-- Performing Test CXX_SUPPORTS__WSELF_ASSIGN
-- Performing Test CXX_SUPPORTS__WSELF_ASSIGN - Success
-- Performing Test CXX_SUPPORTS__WBIDI_CHARS_ANY
-- Performing Test CXX_SUPPORTS__WBIDI_CHARS_ANY - Failed
-- Performing Test CXX_SUPPORTS__WUNDEF
-- Performing Test CXX_SUPPORTS__WUNDEF - Success
-- Performing Test CXX_SUPPORTS__WUNUSED_PARAMETER
-- Performing Test CXX_SUPPORTS__WUNUSED_PARAMETER - Success
-- Performing Test CXX_SUPPORTS__FNO_EXTENDED_IDENTIFIERS
-- Performing Test CXX_SUPPORTS__FNO_EXTENDED_IDENTIFIERS - Failed
-- Performing Test CXX_SUPPORTS__FDEBUG_PREFIX_MAP_A_B
-- Performing Test CXX_SUPPORTS__FDEBUG_PREFIX_MAP_A_B - Success
-- Performing Test CXX_SUPPORTS__FMACRO_PREFIX_MAP_A_B
-- Performing Test CXX_SUPPORTS__FMACRO_PREFIX_MAP_A_B - Success
-- Performing Test CXX_SUPPORTS__FSTACK_REUSE_NONE
-- Performing Test CXX_SUPPORTS__FSTACK_REUSE_NONE - Failed
-- Performing Test CXX_SUPPORTS__U_FORTIFY_SOURCE__D_FORTIFY_SOURCE_3_2d08
-- Performing Test CXX_SUPPORTS__U_FORTIFY_SOURCE__D_FORTIFY_SOURCE_3_2d08 - Success
-- Performing Test LINKER_SUPPORTS__U_FORTIFY_SOURCE__D_FORTIFY_SOURCE_3_2d08
-- Performing Test LINKER_SUPPORTS__U_FORTIFY_SOURCE__D_FORTIFY_SOURCE_3_2d08 - Success
-- Performing Test CXX_SUPPORTS__WSTACK_PROTECTOR
-- Performing Test CXX_SUPPORTS__WSTACK_PROTECTOR - Success
-- Performing Test CXX_SUPPORTS__FSTACK_PROTECTOR_ALL
-- Performing Test CXX_SUPPORTS__FSTACK_PROTECTOR_ALL - Success
-- Performing Test LINKER_SUPPORTS__FSTACK_PROTECTOR_ALL
-- Performing Test LINKER_SUPPORTS__FSTACK_PROTECTOR_ALL - Success
-- Performing Test CXX_SUPPORTS__FCF_PROTECTION_FULL
-- Performing Test CXX_SUPPORTS__FCF_PROTECTION_FULL - Success
-- Performing Test LINKER_SUPPORTS__FCF_PROTECTION_FULL
-- Performing Test LINKER_SUPPORTS__FCF_PROTECTION_FULL - Success
-- Performing Test CXX_SUPPORTS__FSTACK_CLASH_PROTECTION
-- Performing Test CXX_SUPPORTS__FSTACK_CLASH_PROTECTION - Success
-- Performing Test LINKER_SUPPORTS__FSTACK_CLASH_PROTECTION
-- Performing Test LINKER_SUPPORTS__FSTACK_CLASH_PROTECTION - Success
-- Performing Test LINKER_SUPPORTS__WL___ENABLE_RELOC_SECTION
-- Performing Test LINKER_SUPPORTS__WL___ENABLE_RELOC_SECTION - Failed
-- Performing Test LINKER_SUPPORTS__WL___DYNAMICBASE
-- Performing Test LINKER_SUPPORTS__WL___DYNAMICBASE - Failed
-- Performing Test LINKER_SUPPORTS__WL___NXCOMPAT
-- Performing Test LINKER_SUPPORTS__WL___NXCOMPAT - Failed
-- Performing Test LINKER_SUPPORTS__WL___HIGH_ENTROPY_VA
-- Performing Test LINKER_SUPPORTS__WL___HIGH_ENTROPY_VA - Failed
-- Performing Test LINKER_SUPPORTS__WL__Z_RELRO
-- Performing Test LINKER_SUPPORTS__WL__Z_RELRO - Success
-- Performing Test LINKER_SUPPORTS__WL__Z_NOW
-- Performing Test LINKER_SUPPORTS__WL__Z_NOW - Success
-- Performing Test LINKER_SUPPORTS__WL__Z_SEPARATE_CODE
-- Performing Test LINKER_SUPPORTS__WL__Z_SEPARATE_CODE - Success
-- Found Python3: /nix/store/yqy95kjk7mz7y62www64krlkrrs4w7fh-python3-3.13.3-env/bin/python3 (found suitable version "3.13.3", minimum required is "3.10") found components: Interpreter
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE dot)
-- Performing Test HAVE_BUILTIN_PREFETCH
-- Performing Test HAVE_BUILTIN_PREFETCH - Success
-- Performing Test HAVE_MM_PREFETCH
-- Performing Test HAVE_MM_PREFETCH - Success
-- Performing Test HAVE_SSE42
-- Performing Test HAVE_SSE42 - Success
-- Performing Test HAVE_ARM64_CRC32C
-- Performing Test HAVE_ARM64_CRC32C - Failed
-- Looking for F_FULLFSYNC
-- Looking for F_FULLFSYNC - not found
-- Performing Test HAVE_CLMUL
-- Performing Test HAVE_CLMUL - Success

Configuring secp256k1 subtree...
-- The C compiler identification is Clang 19.1.7
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /etc/profiles/per-user/will/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Performing Test HAVE_X86_64_ASM
-- Performing Test HAVE_X86_64_ASM - Success
-- Could NOT find Valgrind (missing: Valgrind_INCLUDE_DIR Valgrind_WORKS)
-- Performing Test C_SUPPORTS__WALL
-- Performing Test C_SUPPORTS__WALL - Success
-- Performing Test C_SUPPORTS__PEDANTIC
-- Performing Test C_SUPPORTS__PEDANTIC - Success
-- Performing Test C_SUPPORTS__WCAST_ALIGN
-- Performing Test C_SUPPORTS__WCAST_ALIGN - Success
-- Performing Test C_SUPPORTS__WCAST_ALIGN_STRICT
-- Performing Test C_SUPPORTS__WCAST_ALIGN_STRICT - Failed
-- Performing Test C_SUPPORTS__WCONDITIONAL_UNINITIALIZED
-- Performing Test C_SUPPORTS__WCONDITIONAL_UNINITIALIZED - Success
-- Performing Test C_SUPPORTS__WEXTRA
-- Performing Test C_SUPPORTS__WEXTRA - Success
-- Performing Test C_SUPPORTS__WNESTED_EXTERNS
-- Performing Test C_SUPPORTS__WNESTED_EXTERNS - Success
-- Performing Test C_SUPPORTS__WNO_LONG_LONG
-- Performing Test C_SUPPORTS__WNO_LONG_LONG - Success
-- Performing Test C_SUPPORTS__WNO_OVERLENGTH_STRINGS
-- Performing Test C_SUPPORTS__WNO_OVERLENGTH_STRINGS - Success
-- Performing Test C_SUPPORTS__WNO_UNUSED_FUNCTION
-- Performing Test C_SUPPORTS__WNO_UNUSED_FUNCTION - Success
-- Performing Test C_SUPPORTS__WRESERVED_IDENTIFIER
-- Performing Test C_SUPPORTS__WRESERVED_IDENTIFIER - Success
-- Performing Test C_SUPPORTS__WSHADOW
-- Performing Test C_SUPPORTS__WSHADOW - Success
-- Performing Test C_SUPPORTS__WSTRICT_PROTOTYPES
-- Performing Test C_SUPPORTS__WSTRICT_PROTOTYPES - Success
-- Performing Test C_SUPPORTS__WUNDEF
-- Performing Test C_SUPPORTS__WUNDEF - Success


secp256k1 configure summary
===========================
Build artifacts:
  library type ........................ Static
Optional modules:
  ECDH ................................ OFF
  ECDSA pubkey recovery ............... ON
  extrakeys ........................... ON
  schnorrsig .......................... ON
  musig ............................... ON
  ElligatorSwift ...................... ON
Parameters:
  ecmult window size .................. 15
  ecmult gen table size ............... 86 KiB
Optional features:
  assembly ............................ x86_64
  external callbacks .................. OFF
Optional binaries:
  benchmark ........................... OFF
  noverify_tests ...................... ON
  tests ............................... ON
  exhaustive tests .................... ON
  ctime_tests ......................... OFF
  examples ............................ OFF

Cross compiling ....................... FALSE
API visibility attributes ............. ON
Valgrind .............................. OFF
Preprocessor defined macros ........... ECMULT_WINDOW_SIZE=15 COMB_BLOCKS=43 COMB_TEETH=6 USE_ASM_X86_64=1
C compiler ............................ Clang 19.1.7, /etc/profiles/per-user/will/bin/cc
CFLAGS ................................
Compile options ....................... -Wall -pedantic -Wcast-align -Wconditional-uninitialized -Wextra -Wnested-externs -Wno-long-long -Wno-overlength-strings -Wno-unused-function -Wreserved-identifier -Wshadow -Wstrict-prototypes -Wundef
Build type:
 - CMAKE_BUILD_TYPE ................... RelWithDebInfo
 - CFLAGS ............................. -O2 -g
 - LDFLAGS for executables ............
 - LDFLAGS for shared libraries .......



Configure summary
=================
Executables:
  bitcoin ............................. ON
  bitcoind ............................ ON
  bitcoin-node (multiprocess) ......... OFF
  bitcoin-qt (GUI) .................... OFF
  bitcoin-gui (GUI, multiprocess) ..... OFF
  bitcoin-cli ......................... ON
  bitcoin-tx .......................... ON
  bitcoin-util ........................ ON
  bitcoin-wallet ...................... ON
  bitcoin-chainstate (experimental) ... OFF
  libbitcoinkernel (experimental) ..... OFF
Optional features:
  wallet support ...................... ON
  external signer ..................... ON
  ZeroMQ .............................. OFF
  IPC ................................. OFF
  USDT tracing ........................ OFF
  QR code (GUI) ....................... OFF
  DBus (GUI) .......................... OFF
Tests:
  test_bitcoin ........................ ON
  test_bitcoin-qt ..................... OFF
  bench_bitcoin ....................... OFF
  fuzz binary ......................... OFF

Cross compiling ....................... FALSE
C++ compiler .......................... Clang 19.1.7, /etc/profiles/per-user/will/bin/c++
CMAKE_BUILD_TYPE ...................... RelWithDebInfo
Preprocessor defined macros ...........
C++ compiler flags .................... -O2 -g -std=c++20 -fPIC -fdebug-prefix-map=/home/will/src/core/bitcoin/src=. -fmacro-prefix-map=/home/will/src/core/bitcoin/src=. -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -Wstack-protector -fstack-protector-all -fcf-protection=full -fstack-clash-protection -Wall -Wextra -Wgnu -Wformat -Wformat-security -Wvla -Wshadow-field -Wthread-safety -Wloop-analysis -Wredundant-decls -Wunused-member-function -Wdate-time -Wconditional-uninitialized -Woverloaded-virtual -Wsuggest-override -Wimplicit-fallthrough -Wunreachable-code -Wdocumentation -Wself-assign -Wundef -Wno-unused-parameter
Linker flags .......................... -O2 -g -fstack-protector-all -fcf-protection=full -fstack-clash-protection -Wl,-z,relro -Wl,-z,now -Wl,-z,separate-code -fPIE -pie

NOTE: The summary above may not exactly match the final applied build flags
      if any additional CMAKE_* or environment variables have been modified.
      To see the exact flags applied, build with the --verbose option.

Treat compiler warnings as errors ..... OFF
Use ccache for compiling .............. ON


-- Configuring done (15.7s)
-- Generating done (0.0s)
-- Build files have been written to: /home/will/src/core/bitcoin/build
[566/566] Linking CXX executable bin/test_bitcoin
Temporary test directory at /tmp/test_runner_₿_🏃_20250806_151617
Traceback (most recent call last):
  File "/home/will/src/core/bitcoin/./build/test/functional/test_runner.py", line 931, in <module>
    main()
    ~~~~^^
  File "/home/will/src/core/bitcoin/./build/test/functional/test_runner.py", line 521, in main
    bench_list = subprocess.check_output(bench_cmd).decode("ascii").splitlines()
                 ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/nix/store/2mab9iiwhcqwk75qwvp3zv0bvbiaq6cs-python3-3.13.3/lib/python3.13/subprocess.py", line 472, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
               **kwargs).stdout
               ^^^^^^^^^
  File "/nix/store/2mab9iiwhcqwk75qwvp3zv0bvbiaq6cs-python3-3.13.3/lib/python3.13/subprocess.py", line 554, in run
    with Popen(*popenargs, **kwargs) as process:
         ~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/2mab9iiwhcqwk75qwvp3zv0bvbiaq6cs-python3-3.13.3/lib/python3.13/subprocess.py", line 1039, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                        pass_fds, cwd, env,
                        ^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
                        gid, gids, uid, umask,
                        ^^^^^^^^^^^^^^^^^^^^^^
                        start_new_session, process_group)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/2mab9iiwhcqwk75qwvp3zv0bvbiaq6cs-python3-3.13.3/lib/python3.13/subprocess.py", line 1854, in _execute_child
    self._posix_spawn(args, executable, env, restore_signals, close_fds,
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      p2cread, p2cwrite,
                      ^^^^^^^^^^^^^^^^^^
                      c2pread, c2pwrite,
                      ^^^^^^^^^^^^^^^^^^
                      errread, errwrite)
                      ^^^^^^^^^^^^^^^^^^
  File "/nix/store/2mab9iiwhcqwk75qwvp3zv0bvbiaq6cs-python3-3.13.3/lib/python3.13/subprocess.py", line 1798, in _posix_spawn
    self.pid = os.posix_spawn(executable, args, env, **kwargs)
               ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/home/will/src/core/bitcoin/build/bin/bench_bitcoin'

bench_bitcoin ....................... OFF

thx, fixed

Concept ACK, thanks for fixing this!

Concept NACK. I think this development goes into the wrong direction.

• With sanitizers enabled, it is one of the slowest targets, often taking several minutes.

The test could be sharded to allow better parallelization. (Nit: "target" is the wrong terminology here)

• There is no insight from ctest into how long each individual sanity check takes.

CTest can provide much more detailed statistics, with duration, memory usage and custom measurements. It just needs to be configured properly.

• Places that can't use ctest (like the Windows-cross CI task) have to explicitly run it, or risk forgetting to run it.

Why can't CTest be used in the Windows-cross CI task? That should be fixed.

• All benchmarks are run sequentially, when they could run in parallel instead.

Agree. But this is possible with CTest.

Both issues can lead to CI timeouts and leave CPU unused during testing.

CTest can schedule tests depending on their resource usage.

Fix all issues by running it as part of the functional tests instead.

Instead of migrating away from CTest, we should fully embrace i.

Why can't CTest be used in the Windows-cross CI task? That should be fixed.

It is documented right in the file: # Can't use ctest here like other jobs as we don't have a CMake build tree.

No one is anyone holding back from fixing that, but personally I don't think it is possible, or only possible with massive code and maintenance overhead. Edit: In any case, the fix for that is ideally done in a separate pull independently, not here, as the issue exists on master.

Instead of migrating away from CTest, we should fully embrace i.

If you want to implement all of this in CTest, please go for it. However, I don't think the functional test framework is going to be removed any time soon. It is well understood and tested. Also, the diff here is mostly move-only and the remaining python logic is trivial.

I won't mind if someone is re-writing the Python code to Cmake code, but I think blocking an improvement based on the language used seems a bit subjective.

I have had a play with porting this to cmake-native (ctest) code, and IMO the main difference is that (in my implementation) it just that it feels more obscure to collect the benchmarks, cost them (not implemented in this PR), and run them in parallel using cmake files vs a few lines of python.

Perhaps I'm doing it wrong, but I found I needed a whole new target generate_bench_tests to dynamically parse benchmark names after bitcoin_bench was built, which can then create the necessary ctest file, which can then be run in "true parallel" by ctest.

We already have a parallel test runner which as @maflcko says is unlikely to go away soon, so it seems easier to me to just run from there.

Are there any tangible benefits we are missing from doing it this way, other than "not being ctest-native"?

Perhaps I'm doing it wrong, but I found I needed a whole new target generate_bench_tests to dynamically parse benchmark names after bitcoin_bench was built, which can then create the necessary ctest file, which can then be run in "true parallel" by ctest.

Yes, it is possible (and thanks for trying), see also https://habla.news/u/purplekarrot.net/cmake-and-test-suites. However,

• it requires more code, and the review for it.
• the win-cross issue is pre-existing and even if the cmake tree was copied, the paths would have to adjusted. A suggested fix for a pre-existing issue is best put in a separate pull, not mixed with other changes.
• i am not going to re-write the functional test framework to cmake, as it is pre-existing. A suggested rewrite is best put in a separate pull or issue, not mixed with other changes.

Ah interesting, that is quite similar in the end to my "workaround" (of building first, then a generate_bench_tests target).

I am Concept ACK using the python framework then, in the absence of any other benefits.

Please see #33483 for dynamic test discovery.

rebased after #33313 (queue refactor)

@maflcko, @purpleKarrot, can you help reviewers understand the relation between this and #33483? Are the two mutually exclusive or is this a temporary optimization until the other one lands (which would make this obsolete)?

@maflcko, @purpleKarrot, can you help reviewers understand the relation between this and #33483?

It is basically the same thing, just written in different languages. I don't really care about the language it is written in. Though, I don't think anyone is going to review the other pull, which looks unmaintained? #33483 (comment)

Given that this pull request has been maintained for almost half a year, with the goal of fixing all the annoying QA issues listed in the pull description, my recommendation would be to go ahead and review this one, merge it, and possibly revert it in #33483, if and when that is ready. The review, testing, and revert should be not too hard.

For testing notes (since someone asked privately): I think compiling once master, and once this pull request (possibly with heavy sanitizers and/or debugging enabled) should be enough. Then, check that the new approach is faster (albeit speed is just one of the QA issues to fix, and probably the least important):

• on master: ctest --test-dir ./bld-cmake --tests-regex bench_sanity_check (obviously -j won't make a difference here)
• on this pull (should take half the time): ./bld-cmake/test/functional/test_runner.py -j 128 tool_bench_sanity_check.py

ACK fafb0b495235a32b7622f2d9be07ebfbb5344a05

Given that this pull request has been maintained for almost half a year, with the goal of fixing all the annoying QA issues listed in the pull description, my recommendation would be to go ahead and review this one, merge it, and possibly revert it in #33483, if and when that is ready. The review, testing, and revert should be not too hard.

I agree with this approach.

Ran test with sanitizers enabled, as per comment: Master: 42 seconds. This PR: 13 seconds.

Note: The direct consequence of this PR is that the functional test are a bit slower due to the addition of the bench sanity checks 280 vs 452. For CI this has no impact because both ran anyway. I do not see this as a downsite, it's just a logical consequence of this PR.

Also the output is more verbose, with this change it is visible which sub test has ran and the progress on these subtests. This is something positive imho, (This feature is also outlined in the pr description)

The sanity checks are skipped if not compiled (as per code):

1/280 - tool_bench_sanity_check.py skipped (bitcoin_bench has not been compiled)

Before (9s):

git checkout master && \
cmake -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_BENCH=ON && \
cmake --build build -j$(nproc) && \
time ctest --test-dir build --tests-regex bench_sanity_check

results in:

Test project /Users/lorinc/IdeaProjects/bitcoin/build
    Start 7: bench_sanity_check
1/1 Test [#7](/bitcoin-bitcoin/7/): bench_sanity_check ...............   Passed    9.11 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   9.11 sec

After (3s):

git checkout fafb0b495235a32b7622f2d9be07ebfbb5344a05 && \
cmake -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_BENCH=ON && \
cmake --build build -j$(nproc) && \
time ./build/test/functional/test_runner.py tool_bench_sanity_check.py -j$(nproc)

results in:

TEST                                                                             | STATUS    | DURATION

tool_bench_sanity_check.py --bench=AddAndRemoveDisconnectedBlockTransactions10   | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=AddAndRemoveDisconnectedBlockTransactions90   | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=AddAndRemoveDisconnectedBlockTransactionsAll  | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=AddrManAdd                                    | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=AddrManAddThenGood                            | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=AddrManGetAddr                                | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=AddrManSelect                                 | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=AddrManSelectByNetwork                        | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=AddrManSelectFromAlmostEmpty                  | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=AssembleBlock                                 | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=Base58CheckEncode                             | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=Base58Decode                                  | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=Base58Encode                                  | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=Bech32Decode                                  | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=Bech32Encode                                  | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=BenchLockedPool                               | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=BenchRIPEMD160                                | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=BenchTimeDeprecated                           | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=BenchTimeMillis                               | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=BenchTimeMillisSys                            | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=BenchTimeMock                                 | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=BIP324_ECDH                                   | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=BlockAssemblerAddPackageTxns                  | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=BlockEncodingLargeExtra                       | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=BlockEncodingNoExtra                          | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=BlockEncodingStdExtra                         | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=BlockFilterIndexSync                          | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=BlockToJsonVerboseWrite                       | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=BlockToJsonVerbosity1                         | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=BlockToJsonVerbosity2                         | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=BlockToJsonVerbosity3                         | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=BnBExhaustion                                 | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=CCheckQueueSpeedPrevectorJob                  | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=CCoinsCaching                                 | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=CHACHA20_1MB                                  | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=CHACHA20_256BYTES                             | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=CHACHA20_64BYTES                              | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=CheckBlockIndex                               | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=CoinSelection                                 | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=ComplexMemPool                                | ✓ Passed  | 1 s
tool_bench_sanity_check.py --bench=ConnectBlockAllEcdsa                          | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=ConnectBlockAllSchnorr                        | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=ConnectBlockMixedEcdsaSchnorr                 | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=DeserializeAndCheckBlockTest                  | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=DeserializeBlockTest                          | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=DuplicateInputs                               | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=EllSwiftCreate                                | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=EvictionProtection0Networks250Candidates      | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=EvictionProtection1Networks250Candidates      | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=EvictionProtection2Networks250Candidates      | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=EvictionProtection3Networks050Candidates      | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=EvictionProtection3Networks100Candidates      | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=EvictionProtection3Networks250Candidates      | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=ExpandDescriptor                              | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=FastRandom_rand32                             | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=FastRandom_rand64                             | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=FastRandom_randbits                           | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=FastRandom_randbool                           | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=FastRandom_randrange100                       | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=FastRandom_randrange1000                      | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=FastRandom_randrange1000000                   | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=FastRandom_stdshuffle100                      | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=FindByte                                      | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=FSCHACHA20POLY1305_1MB                        | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=FSCHACHA20POLY1305_256BYTES                   | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=FSCHACHA20POLY1305_64BYTES                    | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=GCSBlockFilterGetHash                         | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=GCSFilterConstruct                            | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=GCSFilterDecode                               | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=GCSFilterDecodeSkipCheck                      | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=GCSFilterMatch                                | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=HexParse                                      | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=HexStrBench                                   | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=InsecureRandom_rand32                         | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=InsecureRandom_rand64                         | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=InsecureRandom_randbits                       | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=InsecureRandom_randbool                       | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=InsecureRandom_randrange100                   | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=InsecureRandom_randrange1000                  | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=InsecureRandom_randrange1000000               | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=InsecureRandom_stdshuffle100                  | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=LinearizeOptimallyPerCost                     | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=LinearizeOptimallyTotal                       | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=LoadExternalBlockFile                         | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=LogWithDebug                                  | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=LogWithoutDebug                               | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=LogWithoutThreadNames                         | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=LogWithoutWriteToFile                         | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=LogWithThreadNames                            | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=MemPoolAddTransactions                        | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=MemPoolAncestorsDescendants                   | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=MempoolCheck                                  | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=MempoolCheckEphemeralSpends                   | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=MempoolEviction                               | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=MerkleRoot                                    | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=MuHash                                        | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=MuHashDiv                                     | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=MuHashFinalize                                | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=MuHashMul                                     | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=MuHashPrecompute                              | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=ObfuscationBench                              | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=OrphanageEraseForBlock                        | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=OrphanageEraseForPeer                         | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=OrphanageMultiPeerEviction                    | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=OrphanageSinglePeerEviction                   | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=POLY1305_1MB                                  | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=POLY1305_256BYTES                             | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=POLY1305_64BYTES                              | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=PoolAllocator_StdUnorderedMap                 | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=PoolAllocator_StdUnorderedMapWithPoolResource | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=PostLinearize16TxWorstCase                    | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=PostLinearize32TxWorstCase                    | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=PostLinearize48TxWorstCase                    | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=PostLinearize64TxWorstCase                    | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=PostLinearize75TxWorstCase                    | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=PostLinearize99TxWorstCase                    | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=PrePadded                                     | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=PrevectorClearNontrivial                      | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=PrevectorClearTrivial                         | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=PrevectorDeserializeNontrivial                | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=PrevectorDeserializeTrivial                   | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=PrevectorDestructorNontrivial                 | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=PrevectorDestructorTrivial                    | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=PrevectorFillVectorDirectNontrivial           | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=PrevectorFillVectorDirectTrivial              | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=PrevectorFillVectorIndirectNontrivial         | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=PrevectorFillVectorIndirectTrivial            | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=PrevectorResizeNontrivial                     | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=PrevectorResizeTrivial                        | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=ReadBlockBench                                | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=ReadRawBlockBench                             | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=RegularPadded                                 | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=RollingBloom                                  | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=RollingBloomReset                             | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=RpcMempool                                    | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=SHA1                                          | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=SHA256_32b_AVX2                               | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=SHA256_32b_SHANI                              | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=SHA256_32b_SSE4                               | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=SHA256_32b_STANDARD                           | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=SHA256_AVX2                                   | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=SHA256_SHANI                                  | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=SHA256_SSE4                                   | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=SHA256_STANDARD                               | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=SHA256D64_1024_AVX2                           | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=SHA256D64_1024_SHANI                          | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=SHA256D64_1024_SSE4                           | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=SHA256D64_1024_STANDARD                       | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=SHA3_256_1M                                   | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=SHA512                                        | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=SignSchnorrWithMerkleRoot                     | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=SignSchnorrWithNullMerkleRoot                 | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=SignTransactionECDSA                          | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=SignTransactionSchnorr                        | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=SipHash_32b                                   | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=Trig                                          | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=TxGraphTrim                                   | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=VerifyNestedIfScript                          | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=VerifyScriptBench                             | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=WalletAvailableCoins                          | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=WalletBalanceClean                            | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=WalletBalanceDirty                            | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=WalletBalanceMine                             | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=WalletBalanceWatch                            | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=WalletCreateEncrypted                         | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=WalletCreatePlain                             | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=WalletCreateTxUseOnlyPresetInputs             | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=WalletCreateTxUsePresetInputsAndCoinSelection | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=WalletIsMineDescriptors                       | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=WalletIsMineMigratedDescriptors               | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=WalletLoadingDescriptors                      | ✓ Passed  | 1 s
tool_bench_sanity_check.py --bench=WalletMigration                               | ✓ Passed  | 2 s
tool_bench_sanity_check.py --bench=WriteBlockBench                               | ✓ Passed  | 0 s

ALL                                                                              | ✓ Passed  | 4 s (accumulated) 
Runtime: 3 s

Concept ACK - I will review the code a bit later.

The change makes sense, I think we should fix a few references to the benchmarking binary (and keep the alphabetical order of the benchmarks), otherwise LGTM.

diff --git a/test/functional/test_framework/test_framework.py b/test/functional/test_framework/test_framework.py
index 31660cad9c..ecc9ffa231 100755
--- a/test/functional/test_framework/test_framework.py
+++ b/test/functional/test_framework/test_framework.py
@@ -944,9 +944,9 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass):
             raise SkipTest("bitcoin-chainstate has not been compiled")
 
     def skip_if_no_bitcoin_bench(self):
-        """Skip the running test if bitcoin_bench has not been compiled."""
+        """Skip the running test if bench_bitcoin has not been compiled."""
         if not self.is_bench_compiled():
-            raise SkipTest("bitcoin_bench has not been compiled")
+            raise SkipTest("bench_bitcoin has not been compiled")
 
     def skip_if_no_cli(self):
         """Skip the running test if bitcoin-cli has not been compiled."""
@@ -982,7 +982,7 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass):
             raise SkipTest("This test is not compatible with Valgrind.")
 
     def is_bench_compiled(self):
-        """Checks whether bitcoin_bench was compiled."""
+        """Checks whether bench_bitcoin was compiled."""
         return self.config["components"].getboolean("BUILD_BENCH")
 
     def is_cli_compiled(self):
diff --git a/test/functional/test_framework/util.py b/test/functional/test_framework/util.py
index d53c59297f..0c190ee936 100644
--- a/test/functional/test_framework/util.py
+++ b/test/functional/test_framework/util.py
@@ -262,7 +262,7 @@ class Binaries:
         return self._argv("rpc", self.paths.bitcoincli) + ["-nonamed"]
 
     def bench_argv(self):
-        "Return argv array that should be used to invoke bitcoin_bench"
+        "Return argv array that should be used to invoke bench_bitcoin"
         return self._argv("bench", self.paths.bitcoin_bench)
 
     def tx_argv(self):
diff --git a/test/functional/test_runner.py b/test/functional/test_runner.py
index 28ef0d010f..a4af557846 100755
--- a/test/functional/test_runner.py
+++ b/test/functional/test_runner.py
@@ -528,10 +528,10 @@ def main():
         # Remove it, and expand it for each bench in the list
         test_list.remove(TOOL_BENCH_SANITY_CHECK)
         bench_cmd = Binaries(get_binary_paths(config), bin_dir=None).bench_argv() + ["-list"]
-        bench_list = subprocess.check_output(bench_cmd).decode("ascii").splitlines()
+        bench_list = subprocess.check_output(bench_cmd, text=True).splitlines()
         bench_list = [f"{TOOL_BENCH_SANITY_CHECK} --bench={b}" for b in bench_list]
         # Start with special scripts (variable, unknown runtime)
-        test_list.extendleft(bench_list)
+        test_list.extendleft(reversed(bench_list))
 
     if args.filter:
         test_list = deque(filter(re.compile(args.filter).search, test_list))
diff --git a/test/functional/tool_bench_sanity_check.py b/test/functional/tool_bench_sanity_check.py
index ad70fcb9be..d29587861c 100755
--- a/test/functional/tool_bench_sanity_check.py
+++ b/test/functional/tool_bench_sanity_check.py
@@ -4,6 +4,7 @@
 # file COPYING or https://opensource.org/license/mit/.
 """Special script to run each bench sanity check
 """
+import shlex
 import subprocess
 
 from test_framework.test_framework import BitcoinTestFramework
@@ -31,9 +32,9 @@ class BenchSanityCheck(BitcoinTestFramework):
             f"-filter={self.options.bench}",
             "-sanity-check",
         ]
-        self.log.info(f"Starting to run: {cmd}")
+        self.log.info(f"Starting: {shlex.join(cmd)}")
         subprocess.run(cmd, check=True)
-        self.log.info(f"Command passed: {cmd}")
+        self.log.info("Success!")
 
 
 if __name__ == "__main__":

Tested ACK fa65bc0e79dab8a97e0e3f9f31540f1c029a2f6e

tACK fa65bc0e79dab8a97e0e3f9f31540f1c029a2f6e

Changes since last ACK: mainly related to using bench_bitcoin instead of bitcoin_bench keeping the ordering the same by reversing an (unwanted) reversed list.

Also verified the speedup:

master:

src/core/bitcoin on  master [$?⇕] via △ v4.1.2 via 🐍 v3.13.9 via ❄️  impure (nix-shell-env)
❯ ctest --test-dir ./build --tests-regex bench_sanity_check
Test project /home/will/src/core/bitcoin/build
    Start 7: bench_sanity_check
1/1 Test [#7](/bitcoin-bitcoin/7/): bench_sanity_check ...............   Passed   18.20 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =  18.21 sec

PR #33142:

src/core/bitcoin on  pr-33142 [$?] via △ v4.1.2 via 🐍 v3.13.9 via ❄️  impure (nix-shell-env) took 25s
❯ build/test/functional/test_runner.py -j 128 tool_bench_sanity_check.py
Temporary test directory at /tmp/test_runner_₿_🏃_20260105_135859
<snip>
ALL                                                                              | ✓ Passed  | 272 s (accumulated)
Runtime: 8 s

ACK fa65bc0e79dab8a97e0e3f9f31540f1c029a2f6e

This test alone can take 2 minutes or longer in CI, which is reduced here to ~20 seconds when parallelised as here in this changeset. Agree that we can revert if/when we move to a more cmake-native solution in the future.

ACK fa65bc0e79dab8a97e0e3f9f31540f1c029a2f6e

Merging this, but this is not saying that this direction is set in stone. This PR gives us some immediate wins that are useful to all developers with a fairly small amount of code changed. However, rewriting everything to utilize ctest is still on the table, but that does seem like a much larger task that will take significantly longer to implement and review.