test: Run bench sanity checks in parallel with functional tests

maflcko commented at 8:30 am on August 6, 2025: member

The ctest target bench_sanity_check has many issues:

With sanitizers enabled, it is one of the slowest targets, often taking several minutes. See #32770 (comment).
There is no insight from ctest into how long each individual sanity check takes.
On a timeout, or OOM issue, there is no insight into which sub-bench failed. The failure will generally just look like 75/153 Test [#9](/bitcoin-bitcoin/9/): bench_sanity_check ...................***Failed 770.84 sec out of memory
Places that can’t use ctest (like the Windows-cross CI task) have to explicitly run it, or risk forgetting to run it.
All benchmarks are run sequentially, when they could run in parallel instead.

Both issues can lead to CI timeouts and leave CPU unused during testing.

Fix all issues by running it as part of the functional tests instead. This is similar to the rpcauth tests (https://github.com/bitcoin/bitcoin/pull/32881) and util tests [bitcoin-tx, and bitcoin-util] (https://github.com/bitcoin/bitcoin/pull/32697).

DrahtBot added the label Tests on Aug 6, 2025

DrahtBot commented at 8:30 am on August 6, 2025: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33142.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	l0rinc, janb84, willcl-ark, achow101
Concept NACK	purpleKarrot

If your review is incorrectly listed, please copy-paste <!–meta-tag:bot-skip–> into the comment that the bot should ignore.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#33483 (CMake: Add dynamic test discovery by purpleKarrot)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

maflcko force-pushed on Aug 6, 2025

DrahtBot added the label CI failed on Aug 6, 2025

DrahtBot commented at 8:56 am on August 6, 2025: contributor

🚧 At least one of the CI tasks failed. Task lint: https://github.com/bitcoin/bitcoin/runs/47488255049 LLM reason (✨ experimental): The CI failure is due to errors from the Python linting check.

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

DrahtBot removed the label CI failed on Aug 6, 2025

maflcko force-pushed on Aug 6, 2025

DrahtBot added the label CI failed on Aug 6, 2025

DrahtBot commented at 10:52 am on August 6, 2025: contributor

🚧 At least one of the CI tasks failed. Task lint: https://github.com/bitcoin/bitcoin/runs/47498178413 LLM reason (✨ experimental): The CI failure is caused by a lint error due to an unused import detected by ruff.

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

maflcko force-pushed on Aug 6, 2025

willcl-ark commented at 2:17 pm on August 6, 2025: member

I get an error compiling this without bench_bitcoin:

  0❯ cmake -B build; and cmake --build build; and ./build/test/functional/test_runner.py -j14
  1-- The CXX compiler identification is Clang 19.1.7
  2-- Detecting CXX compiler ABI info
  3-- Detecting CXX compiler ABI info - done
  4-- Check for working CXX compiler: /etc/profiles/per-user/will/bin/c++ - skipped
  5-- Detecting CXX compile features
  6-- Detecting CXX compile features - done
  7-- Setting build type to "RelWithDebInfo" as none was specified
  8-- Performing Test CXX_SUPPORTS__WERROR
  9-- Performing Test CXX_SUPPORTS__WERROR - Success
 10-- Performing Test CXX_SUPPORTS__G3
 11-- Performing Test CXX_SUPPORTS__G3 - Success
 12-- Performing Test LINKER_SUPPORTS__G3
 13-- Performing Test LINKER_SUPPORTS__G3 - Success
 14-- Performing Test CXX_SUPPORTS__FTRAPV
 15-- Performing Test CXX_SUPPORTS__FTRAPV - Success
 16-- Performing Test LINKER_SUPPORTS__FTRAPV
 17-- Performing Test LINKER_SUPPORTS__FTRAPV - Success
 18-- Found SQLite3: /nix/store/92gwa4j45skp8d096csmnj2a8jcn0q9w-sqlite-3.48.0-dev/include (found suitable version "3.48.0", minimum required is "3.7.17")
 19-- Performing Test LINKER_SUPPORTS__WL___FATAL_WARNINGS
 20-- Performing Test LINKER_SUPPORTS__WL___FATAL_WARNINGS - Success
 21-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
 22-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
 23-- Found Threads: TRUE
 24-- Performing Test NO_DIAGNOSTICS_BOOST_NO_CXX98_FUNCTION_BASE
 25-- Performing Test NO_DIAGNOSTICS_BOOST_NO_CXX98_FUNCTION_BASE - Failed
 26-- Found PkgConfig: /nix/store/2crk9xnq5x9v7yf0r2nwkgj8qsmxr4ly-pkg-config-wrapper-0.29.2/bin/pkg-config (found version "0.29.2")
 27-- Found Libevent: /nix/store/yai7mpy5d4rw0jvflyxdf0vzjkiqxhv6-libevent-2.1.12/lib (found suitable version "2.1.12-stable", minimum required is "2.1.8")
 28-- Performing Test HAVE_EVHTTP_CONNECTION_GET_PEER_CONST_CHAR
 29-- Performing Test HAVE_EVHTTP_CONNECTION_GET_PEER_CONST_CHAR - Failed
 30-- Looking for O_CLOEXEC
 31-- Looking for O_CLOEXEC - found
 32-- Looking for fdatasync
 33-- Looking for fdatasync - found
 34-- Looking for fork
 35-- Looking for fork - found
 36-- Looking for pipe2
 37-- Looking for pipe2 - found
 38-- Looking for setsid
 39-- Looking for setsid - found
 40-- Performing Test IFADDR_LINKS_WITHOUT_LIBSOCKET
 41-- Performing Test IFADDR_LINKS_WITHOUT_LIBSOCKET - Success
 42-- Performing Test STD_ATOMIC_LINKS_WITHOUT_LIBATOMIC
 43-- Performing Test STD_ATOMIC_LINKS_WITHOUT_LIBATOMIC - Success
 44-- Looking for std::system
 45-- Looking for std::system - found
 46-- Looking for ::_wsystem
 47-- Looking for ::_wsystem - not found
 48-- Performing Test STRERROR_R_CHAR_P
 49-- Performing Test STRERROR_R_CHAR_P - Success
 50-- Looking for malloc_info
 51-- Looking for malloc_info - found
 52-- Performing Test HAVE_MALLOPT_ARENA_MAX
 53-- Performing Test HAVE_MALLOPT_ARENA_MAX - Success
 54-- Performing Test HAVE_POSIX_FALLOCATE
 55-- Performing Test HAVE_POSIX_FALLOCATE - Success
 56-- Performing Test HAVE_STRONG_GETAUXVAL
 57-- Performing Test HAVE_STRONG_GETAUXVAL - Success
 58-- Performing Test HAVE_SOCKADDR_UN
 59-- Performing Test HAVE_SOCKADDR_UN - Success
 60-- Performing Test HAVE_GETRANDOM
 61-- Performing Test HAVE_GETRANDOM - Success
 62-- Performing Test HAVE_GETENTROPY_RAND
 63-- Performing Test HAVE_GETENTROPY_RAND - Success
 64-- Performing Test HAVE_SYSCTL
 65-- Performing Test HAVE_SYSCTL - Failed
 66-- Performing Test HAVE_SYSCTL_ARND
 67-- Performing Test HAVE_SYSCTL_ARND - Failed
 68-- Performing Test HAVE_SSE41
 69-- Performing Test HAVE_SSE41 - Success
 70-- Performing Test HAVE_AVX2
 71-- Performing Test HAVE_AVX2 - Success
 72-- Performing Test HAVE_X86_SHANI
 73-- Performing Test HAVE_X86_SHANI - Success
 74-- Performing Test HAVE_ARM_SHANI
 75-- Performing Test HAVE_ARM_SHANI - Failed
 76-- Performing Test CXX_SUPPORTS__WALL
 77-- Performing Test CXX_SUPPORTS__WALL - Success
 78-- Performing Test CXX_SUPPORTS__WEXTRA
 79-- Performing Test CXX_SUPPORTS__WEXTRA - Success
 80-- Performing Test CXX_SUPPORTS__WGNU
 81-- Performing Test CXX_SUPPORTS__WGNU - Success
 82-- Performing Test CXX_SUPPORTS__WFORMAT__WFORMAT_SECURITY
 83-- Performing Test CXX_SUPPORTS__WFORMAT__WFORMAT_SECURITY - Success
 84-- Performing Test CXX_SUPPORTS__WVLA
 85-- Performing Test CXX_SUPPORTS__WVLA - Success
 86-- Performing Test CXX_SUPPORTS__WSHADOW_FIELD
 87-- Performing Test CXX_SUPPORTS__WSHADOW_FIELD - Success
 88-- Performing Test CXX_SUPPORTS__WTHREAD_SAFETY
 89-- Performing Test CXX_SUPPORTS__WTHREAD_SAFETY - Success
 90-- Performing Test CXX_SUPPORTS__WTHREAD_SAFETY_POINTER
 91-- Performing Test CXX_SUPPORTS__WTHREAD_SAFETY_POINTER - Failed
 92-- Performing Test CXX_SUPPORTS__WLOOP_ANALYSIS
 93-- Performing Test CXX_SUPPORTS__WLOOP_ANALYSIS - Success
 94-- Performing Test CXX_SUPPORTS__WREDUNDANT_DECLS
 95-- Performing Test CXX_SUPPORTS__WREDUNDANT_DECLS - Success
 96-- Performing Test CXX_SUPPORTS__WUNUSED_MEMBER_FUNCTION
 97-- Performing Test CXX_SUPPORTS__WUNUSED_MEMBER_FUNCTION - Success
 98-- Performing Test CXX_SUPPORTS__WDATE_TIME
 99-- Performing Test CXX_SUPPORTS__WDATE_TIME - Success
100-- Performing Test CXX_SUPPORTS__WCONDITIONAL_UNINITIALIZED
101-- Performing Test CXX_SUPPORTS__WCONDITIONAL_UNINITIALIZED - Success
102-- Performing Test CXX_SUPPORTS__WDUPLICATED_BRANCHES
103-- Performing Test CXX_SUPPORTS__WDUPLICATED_BRANCHES - Failed
104-- Performing Test CXX_SUPPORTS__WDUPLICATED_COND
105-- Performing Test CXX_SUPPORTS__WDUPLICATED_COND - Failed
106-- Performing Test CXX_SUPPORTS__WLOGICAL_OP
107-- Performing Test CXX_SUPPORTS__WLOGICAL_OP - Failed
108-- Performing Test CXX_SUPPORTS__WOVERLOADED_VIRTUAL
109-- Performing Test CXX_SUPPORTS__WOVERLOADED_VIRTUAL - Success
110-- Performing Test CXX_SUPPORTS__WSUGGEST_OVERRIDE
111-- Performing Test CXX_SUPPORTS__WSUGGEST_OVERRIDE - Success
112-- Performing Test CXX_SUPPORTS__WIMPLICIT_FALLTHROUGH
113-- Performing Test CXX_SUPPORTS__WIMPLICIT_FALLTHROUGH - Success
114-- Performing Test CXX_SUPPORTS__WUNREACHABLE_CODE
115-- Performing Test CXX_SUPPORTS__WUNREACHABLE_CODE - Success
116-- Performing Test CXX_SUPPORTS__WDOCUMENTATION
117-- Performing Test CXX_SUPPORTS__WDOCUMENTATION - Success
118-- Performing Test CXX_SUPPORTS__WSELF_ASSIGN
119-- Performing Test CXX_SUPPORTS__WSELF_ASSIGN - Success
120-- Performing Test CXX_SUPPORTS__WBIDI_CHARS_ANY
121-- Performing Test CXX_SUPPORTS__WBIDI_CHARS_ANY - Failed
122-- Performing Test CXX_SUPPORTS__WUNDEF
123-- Performing Test CXX_SUPPORTS__WUNDEF - Success
124-- Performing Test CXX_SUPPORTS__WUNUSED_PARAMETER
125-- Performing Test CXX_SUPPORTS__WUNUSED_PARAMETER - Success
126-- Performing Test CXX_SUPPORTS__FNO_EXTENDED_IDENTIFIERS
127-- Performing Test CXX_SUPPORTS__FNO_EXTENDED_IDENTIFIERS - Failed
128-- Performing Test CXX_SUPPORTS__FDEBUG_PREFIX_MAP_A_B
129-- Performing Test CXX_SUPPORTS__FDEBUG_PREFIX_MAP_A_B - Success
130-- Performing Test CXX_SUPPORTS__FMACRO_PREFIX_MAP_A_B
131-- Performing Test CXX_SUPPORTS__FMACRO_PREFIX_MAP_A_B - Success
132-- Performing Test CXX_SUPPORTS__FSTACK_REUSE_NONE
133-- Performing Test CXX_SUPPORTS__FSTACK_REUSE_NONE - Failed
134-- Performing Test CXX_SUPPORTS__U_FORTIFY_SOURCE__D_FORTIFY_SOURCE_3_2d08
135-- Performing Test CXX_SUPPORTS__U_FORTIFY_SOURCE__D_FORTIFY_SOURCE_3_2d08 - Success
136-- Performing Test LINKER_SUPPORTS__U_FORTIFY_SOURCE__D_FORTIFY_SOURCE_3_2d08
137-- Performing Test LINKER_SUPPORTS__U_FORTIFY_SOURCE__D_FORTIFY_SOURCE_3_2d08 - Success
138-- Performing Test CXX_SUPPORTS__WSTACK_PROTECTOR
139-- Performing Test CXX_SUPPORTS__WSTACK_PROTECTOR - Success
140-- Performing Test CXX_SUPPORTS__FSTACK_PROTECTOR_ALL
141-- Performing Test CXX_SUPPORTS__FSTACK_PROTECTOR_ALL - Success
142-- Performing Test LINKER_SUPPORTS__FSTACK_PROTECTOR_ALL
143-- Performing Test LINKER_SUPPORTS__FSTACK_PROTECTOR_ALL - Success
144-- Performing Test CXX_SUPPORTS__FCF_PROTECTION_FULL
145-- Performing Test CXX_SUPPORTS__FCF_PROTECTION_FULL - Success
146-- Performing Test LINKER_SUPPORTS__FCF_PROTECTION_FULL
147-- Performing Test LINKER_SUPPORTS__FCF_PROTECTION_FULL - Success
148-- Performing Test CXX_SUPPORTS__FSTACK_CLASH_PROTECTION
149-- Performing Test CXX_SUPPORTS__FSTACK_CLASH_PROTECTION - Success
150-- Performing Test LINKER_SUPPORTS__FSTACK_CLASH_PROTECTION
151-- Performing Test LINKER_SUPPORTS__FSTACK_CLASH_PROTECTION - Success
152-- Performing Test LINKER_SUPPORTS__WL___ENABLE_RELOC_SECTION
153-- Performing Test LINKER_SUPPORTS__WL___ENABLE_RELOC_SECTION - Failed
154-- Performing Test LINKER_SUPPORTS__WL___DYNAMICBASE
155-- Performing Test LINKER_SUPPORTS__WL___DYNAMICBASE - Failed
156-- Performing Test LINKER_SUPPORTS__WL___NXCOMPAT
157-- Performing Test LINKER_SUPPORTS__WL___NXCOMPAT - Failed
158-- Performing Test LINKER_SUPPORTS__WL___HIGH_ENTROPY_VA
159-- Performing Test LINKER_SUPPORTS__WL___HIGH_ENTROPY_VA - Failed
160-- Performing Test LINKER_SUPPORTS__WL__Z_RELRO
161-- Performing Test LINKER_SUPPORTS__WL__Z_RELRO - Success
162-- Performing Test LINKER_SUPPORTS__WL__Z_NOW
163-- Performing Test LINKER_SUPPORTS__WL__Z_NOW - Success
164-- Performing Test LINKER_SUPPORTS__WL__Z_SEPARATE_CODE
165-- Performing Test LINKER_SUPPORTS__WL__Z_SEPARATE_CODE - Success
166-- Found Python3: /nix/store/yqy95kjk7mz7y62www64krlkrrs4w7fh-python3-3.13.3-env/bin/python3 (found suitable version "3.13.3", minimum required is "3.10") found components: Interpreter
167-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE dot)
168-- Performing Test HAVE_BUILTIN_PREFETCH
169-- Performing Test HAVE_BUILTIN_PREFETCH - Success
170-- Performing Test HAVE_MM_PREFETCH
171-- Performing Test HAVE_MM_PREFETCH - Success
172-- Performing Test HAVE_SSE42
173-- Performing Test HAVE_SSE42 - Success
174-- Performing Test HAVE_ARM64_CRC32C
175-- Performing Test HAVE_ARM64_CRC32C - Failed
176-- Looking for F_FULLFSYNC
177-- Looking for F_FULLFSYNC - not found
178-- Performing Test HAVE_CLMUL
179-- Performing Test HAVE_CLMUL - Success
180
181Configuring secp256k1 subtree...
182-- The C compiler identification is Clang 19.1.7
183-- Detecting C compiler ABI info
184-- Detecting C compiler ABI info - done
185-- Check for working C compiler: /etc/profiles/per-user/will/bin/cc - skipped
186-- Detecting C compile features
187-- Detecting C compile features - done
188-- Performing Test HAVE_X86_64_ASM
189-- Performing Test HAVE_X86_64_ASM - Success
190-- Could NOT find Valgrind (missing: Valgrind_INCLUDE_DIR Valgrind_WORKS)
191-- Performing Test C_SUPPORTS__WALL
192-- Performing Test C_SUPPORTS__WALL - Success
193-- Performing Test C_SUPPORTS__PEDANTIC
194-- Performing Test C_SUPPORTS__PEDANTIC - Success
195-- Performing Test C_SUPPORTS__WCAST_ALIGN
196-- Performing Test C_SUPPORTS__WCAST_ALIGN - Success
197-- Performing Test C_SUPPORTS__WCAST_ALIGN_STRICT
198-- Performing Test C_SUPPORTS__WCAST_ALIGN_STRICT - Failed
199-- Performing Test C_SUPPORTS__WCONDITIONAL_UNINITIALIZED
200-- Performing Test C_SUPPORTS__WCONDITIONAL_UNINITIALIZED - Success
201-- Performing Test C_SUPPORTS__WEXTRA
202-- Performing Test C_SUPPORTS__WEXTRA - Success
203-- Performing Test C_SUPPORTS__WNESTED_EXTERNS
204-- Performing Test C_SUPPORTS__WNESTED_EXTERNS - Success
205-- Performing Test C_SUPPORTS__WNO_LONG_LONG
206-- Performing Test C_SUPPORTS__WNO_LONG_LONG - Success
207-- Performing Test C_SUPPORTS__WNO_OVERLENGTH_STRINGS
208-- Performing Test C_SUPPORTS__WNO_OVERLENGTH_STRINGS - Success
209-- Performing Test C_SUPPORTS__WNO_UNUSED_FUNCTION
210-- Performing Test C_SUPPORTS__WNO_UNUSED_FUNCTION - Success
211-- Performing Test C_SUPPORTS__WRESERVED_IDENTIFIER
212-- Performing Test C_SUPPORTS__WRESERVED_IDENTIFIER - Success
213-- Performing Test C_SUPPORTS__WSHADOW
214-- Performing Test C_SUPPORTS__WSHADOW - Success
215-- Performing Test C_SUPPORTS__WSTRICT_PROTOTYPES
216-- Performing Test C_SUPPORTS__WSTRICT_PROTOTYPES - Success
217-- Performing Test C_SUPPORTS__WUNDEF
218-- Performing Test C_SUPPORTS__WUNDEF - Success
219
220
221secp256k1 configure summary
222===========================
223Build artifacts:
224  library type ........................ Static
225Optional modules:
226  ECDH ................................ OFF
227  ECDSA pubkey recovery ............... ON
228  extrakeys ........................... ON
229  schnorrsig .......................... ON
230  musig ............................... ON
231  ElligatorSwift ...................... ON
232Parameters:
233  ecmult window size .................. 15
234  ecmult gen table size ............... 86 KiB
235Optional features:
236  assembly ............................ x86_64
237  external callbacks .................. OFF
238Optional binaries:
239  benchmark ........................... OFF
240  noverify_tests ...................... ON
241  tests ............................... ON
242  exhaustive tests .................... ON
243  ctime_tests ......................... OFF
244  examples ............................ OFF
245
246Cross compiling ....................... FALSE
247API visibility attributes ............. ON
248Valgrind .............................. OFF
249Preprocessor defined macros ........... ECMULT_WINDOW_SIZE=15 COMB_BLOCKS=43 COMB_TEETH=6 USE_ASM_X86_64=1
250C compiler ............................ Clang 19.1.7, /etc/profiles/per-user/will/bin/cc
251CFLAGS ................................
252Compile options ....................... -Wall -pedantic -Wcast-align -Wconditional-uninitialized -Wextra -Wnested-externs -Wno-long-long -Wno-overlength-strings -Wno-unused-function -Wreserved-identifier -Wshadow -Wstrict-prototypes -Wundef
253Build type:
254 - CMAKE_BUILD_TYPE ................... RelWithDebInfo
255 - CFLAGS ............................. -O2 -g
256 - LDFLAGS for executables ............
257 - LDFLAGS for shared libraries .......
258
259
260
261Configure summary
262=================
263Executables:
264  bitcoin ............................. ON
265  bitcoind ............................ ON
266  bitcoin-node (multiprocess) ......... OFF
267  bitcoin-qt (GUI) .................... OFF
268  bitcoin-gui (GUI, multiprocess) ..... OFF
269  bitcoin-cli ......................... ON
270  bitcoin-tx .......................... ON
271  bitcoin-util ........................ ON
272  bitcoin-wallet ...................... ON
273  bitcoin-chainstate (experimental) ... OFF
274  libbitcoinkernel (experimental) ..... OFF
275Optional features:
276  wallet support ...................... ON
277  external signer ..................... ON
278  ZeroMQ .............................. OFF
279  IPC ................................. OFF
280  USDT tracing ........................ OFF
281  QR code (GUI) ....................... OFF
282  DBus (GUI) .......................... OFF
283Tests:
284  test_bitcoin ........................ ON
285  test_bitcoin-qt ..................... OFF
286  bench_bitcoin ....................... OFF
287  fuzz binary ......................... OFF
288
289Cross compiling ....................... FALSE
290C++ compiler .......................... Clang 19.1.7, /etc/profiles/per-user/will/bin/c++
291CMAKE_BUILD_TYPE ...................... RelWithDebInfo
292Preprocessor defined macros ...........
293C++ compiler flags .................... -O2 -g -std=c++20 -fPIC -fdebug-prefix-map=/home/will/src/core/bitcoin/src=. -fmacro-prefix-map=/home/will/src/core/bitcoin/src=. -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -Wstack-protector -fstack-protector-all -fcf-protection=full -fstack-clash-protection -Wall -Wextra -Wgnu -Wformat -Wformat-security -Wvla -Wshadow-field -Wthread-safety -Wloop-analysis -Wredundant-decls -Wunused-member-function -Wdate-time -Wconditional-uninitialized -Woverloaded-virtual -Wsuggest-override -Wimplicit-fallthrough -Wunreachable-code -Wdocumentation -Wself-assign -Wundef -Wno-unused-parameter
294Linker flags .......................... -O2 -g -fstack-protector-all -fcf-protection=full -fstack-clash-protection -Wl,-z,relro -Wl,-z,now -Wl,-z,separate-code -fPIE -pie
295
296NOTE: The summary above may not exactly match the final applied build flags
297      if any additional CMAKE_* or environment variables have been modified.
298      To see the exact flags applied, build with the --verbose option.
299
300Treat compiler warnings as errors ..... OFF
301Use ccache for compiling .............. ON
302
303
304-- Configuring done (15.7s)
305-- Generating done (0.0s)
306-- Build files have been written to: /home/will/src/core/bitcoin/build
307[566/566] Linking CXX executable bin/test_bitcoin
308Temporary test directory at /tmp/test_runner_₿_🏃_20250806_151617
309Traceback (most recent call last):
310  File "/home/will/src/core/bitcoin/./build/test/functional/test_runner.py", line 931, in <module>
311    main()
312    ~~~~^^
313  File "/home/will/src/core/bitcoin/./build/test/functional/test_runner.py", line 521, in main
314    bench_list = subprocess.check_output(bench_cmd).decode("ascii").splitlines()
315                 ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
316  File "/nix/store/2mab9iiwhcqwk75qwvp3zv0bvbiaq6cs-python3-3.13.3/lib/python3.13/subprocess.py", line 472, in check_output
317    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
318           ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
319               **kwargs).stdout
320               ^^^^^^^^^
321  File "/nix/store/2mab9iiwhcqwk75qwvp3zv0bvbiaq6cs-python3-3.13.3/lib/python3.13/subprocess.py", line 554, in run
322    with Popen(*popenargs, **kwargs) as process:
323         ~~~~~^^^^^^^^^^^^^^^^^^^^^^
324  File "/nix/store/2mab9iiwhcqwk75qwvp3zv0bvbiaq6cs-python3-3.13.3/lib/python3.13/subprocess.py", line 1039, in __init__
325    self._execute_child(args, executable, preexec_fn, close_fds,
326    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
327                        pass_fds, cwd, env,
328                        ^^^^^^^^^^^^^^^^^^^
329    ...<5 lines>...
330                        gid, gids, uid, umask,
331                        ^^^^^^^^^^^^^^^^^^^^^^
332                        start_new_session, process_group)
333                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
334  File "/nix/store/2mab9iiwhcqwk75qwvp3zv0bvbiaq6cs-python3-3.13.3/lib/python3.13/subprocess.py", line 1854, in _execute_child
335    self._posix_spawn(args, executable, env, restore_signals, close_fds,
336    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
337                      p2cread, p2cwrite,
338                      ^^^^^^^^^^^^^^^^^^
339                      c2pread, c2pwrite,
340                      ^^^^^^^^^^^^^^^^^^
341                      errread, errwrite)
342                      ^^^^^^^^^^^^^^^^^^
343  File "/nix/store/2mab9iiwhcqwk75qwvp3zv0bvbiaq6cs-python3-3.13.3/lib/python3.13/subprocess.py", line 1798, in _posix_spawn
344    self.pid = os.posix_spawn(executable, args, env, **kwargs)
345               ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
346FileNotFoundError: [Errno 2] No such file or directory: '/home/will/src/core/bitcoin/build/bin/bench_bitcoin'

maflcko force-pushed on Aug 6, 2025

maflcko commented at 3:03 pm on August 6, 2025: member

bench_bitcoin ………………….. OFF

thx, fixed

DrahtBot removed the label CI failed on Aug 7, 2025

DrahtBot added the label Needs rebase on Aug 7, 2025

maflcko force-pushed on Aug 7, 2025

DrahtBot removed the label Needs rebase on Aug 7, 2025

DrahtBot added the label Needs rebase on Sep 5, 2025

l0rinc commented at 5:40 pm on September 5, 2025: contributor

Concept ACK, thanks for fixing this!

maflcko force-pushed on Sep 25, 2025

purpleKarrot commented at 3:44 pm on September 25, 2025: contributor

Concept NACK. I think this development goes into the wrong direction.

With sanitizers enabled, it is one of the slowest targets, often taking several minutes.

The test could be sharded to allow better parallelization. (Nit: “target” is the wrong terminology here)

There is no insight from ctest into how long each individual sanity check takes.

CTest can provide much more detailed statistics, with duration, memory usage and custom measurements. It just needs to be configured properly.

Places that can’t use ctest (like the Windows-cross CI task) have to explicitly run it, or risk forgetting to run it.

Why can’t CTest be used in the Windows-cross CI task? That should be fixed.

All benchmarks are run sequentially, when they could run in parallel instead.

Agree. But this is possible with CTest.

Both issues can lead to CI timeouts and leave CPU unused during testing.

CTest can schedule tests depending on their resource usage.

Fix all issues by running it as part of the functional tests instead.

Instead of migrating away from CTest, we should fully embrace i.

maflcko commented at 3:59 pm on September 25, 2025: member

Why can’t CTest be used in the Windows-cross CI task? That should be fixed.

It is documented right in the file: # Can't use ctest here like other jobs as we don't have a CMake build tree.

No one is anyone holding back from fixing that, but personally I don’t think it is possible, or only possible with massive code and maintenance overhead. Edit: In any case, the fix for that is ideally done in a separate pull independently, not here, as the issue exists on master.

Instead of migrating away from CTest, we should fully embrace i.

If you want to implement all of this in CTest, please go for it. However, I don’t think the functional test framework is going to be removed any time soon. It is well understood and tested. Also, the diff here is mostly move-only and the remaining python logic is trivial.

I won’t mind if someone is re-writing the Python code to Cmake code, but I think blocking an improvement based on the language used seems a bit subjective.

DrahtBot removed the label Needs rebase on Sep 25, 2025

willcl-ark commented at 9:27 am on September 26, 2025: member

I have had a play with porting this to cmake-native (ctest) code, and IMO the main difference is that (in my implementation) it just that it feels more obscure to collect the benchmarks, cost them (not implemented in this PR), and run them in parallel using cmake files vs a few lines of python.

Perhaps I’m doing it wrong, but I found I needed a whole new target generate_bench_tests to dynamically parse benchmark names after bitcoin_bench was built, which can then create the necessary ctest file, which can then be run in “true parallel” by ctest.

We already have a parallel test runner which as @maflcko says is unlikely to go away soon, so it seems easier to me to just run from there.

Are there any tangible benefits we are missing from doing it this way, other than “not being ctest-native”?

maflcko commented at 9:46 am on September 26, 2025: member

Perhaps I’m doing it wrong, but I found I needed a whole new target generate_bench_tests to dynamically parse benchmark names after bitcoin_bench was built, which can then create the necessary ctest file, which can then be run in “true parallel” by ctest.

Yes, it is possible (and thanks for trying), see also https://habla.news/u/purplekarrot.net/cmake-and-test-suites. However,

it requires more code, and the review for it.
the win-cross issue is pre-existing and even if the cmake tree was copied, the paths would have to adjusted. A suggested fix for a pre-existing issue is best put in a separate pull, not mixed with other changes.
i am not going to re-write the functional test framework to cmake, as it is pre-existing. A suggested rewrite is best put in a separate pull or issue, not mixed with other changes.

willcl-ark commented at 1:35 pm on September 26, 2025: member

Ah interesting, that is quite similar in the end to my “workaround” (of building first, then a generate_bench_tests target).

I am Concept ACK using the python framework then, in the absence of any other benefits.

DrahtBot added the label Needs rebase on Sep 26, 2025

purpleKarrot commented at 5:30 pm on September 26, 2025: contributor

Please see #33483 for dynamic test discovery.

maflcko force-pushed on Sep 29, 2025

maflcko commented at 12:12 pm on September 29, 2025: member

rebased after #33313 (queue refactor)

DrahtBot removed the label Needs rebase on Sep 29, 2025

DrahtBot added the label Needs rebase on Oct 21, 2025

maflcko force-pushed on Oct 21, 2025

DrahtBot removed the label Needs rebase on Oct 21, 2025

maflcko force-pushed on Oct 29, 2025

maflcko force-pushed on Nov 12, 2025

maflcko force-pushed on Dec 19, 2025

l0rinc commented at 1:00 pm on December 19, 2025: contributor

@maflcko, @purpleKarrot, can you help reviewers understand the relation between this and #33483? Are the two mutually exclusive or is this a temporary optimization until the other one lands (which would make this obsolete)?

maflcko commented at 1:04 pm on December 19, 2025: member

@maflcko, @purpleKarrot, can you help reviewers understand the relation between this and #33483?

It is basically the same thing, just written in different languages. I don’t really care about the language it is written in. Though, I don’t think anyone is going to review the other pull, which looks unmaintained? #33483 (comment)

maflcko force-pushed on Dec 26, 2025

maflcko commented at 9:48 am on December 26, 2025: member

Given that this pull request has been maintained for almost half a year, with the goal of fixing all the annoying QA issues listed in the pull description, my recommendation would be to go ahead and review this one, merge it, and possibly revert it in #33483, if and when that is ready. The review, testing, and revert should be not too hard.

For testing notes (since someone asked privately): I think compiling once master, and once this pull request (possibly with heavy sanitizers and/or debugging enabled) should be enough. Then, check that the new approach is faster (albeit speed is just one of the QA issues to fix, and probably the least important):

on master: ctest --test-dir ./bld-cmake --tests-regex bench_sanity_check (obviously -j won’t make a difference here)
on this pull (should take half the time): ./bld-cmake/test/functional/test_runner.py -j 128 tool_bench_sanity_check.py

janb84 commented at 5:32 pm on December 30, 2025: contributor

ACK fafb0b495235a32b7622f2d9be07ebfbb5344a05

Given that this pull request has been maintained for almost half a year, with the goal of fixing all the annoying QA issues listed in the pull description, my recommendation would be to go ahead and review this one, merge it, and possibly revert it in #33483, if and when that is ready. The review, testing, and revert should be not too hard.

I agree with this approach.

Ran test with sanitizers enabled, as per comment: Master: 42 seconds. This PR: 13 seconds.

Note: The direct consequence of this PR is that the functional test are a bit slower due to the addition of the bench sanity checks 280 vs 452. For CI this has no impact because both ran anyway. I do not see this as a downsite, it’s just a logical consequence of this PR.

Also the output is more verbose, with this change it is visible which sub test has ran and the progress on these subtests. This is something positive imho, (This feature is also outlined in the pr description)

The sanity checks are skipped if not compiled (as per code):

01/280 - tool_bench_sanity_check.py skipped (bitcoin_bench has not been compiled)

DrahtBot requested review from l0rinc on Dec 30, 2025

DrahtBot requested review from willcl-ark on Dec 30, 2025

l0rinc commented at 9:13 pm on December 30, 2025: contributor

Before (9s):

0git checkout master && \
1cmake -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_BENCH=ON && \
2cmake --build build -j$(nproc) && \
3time ctest --test-dir build --tests-regex bench_sanity_check

results in:

0Test project /Users/lorinc/IdeaProjects/bitcoin/build
1    Start 7: bench_sanity_check
21/1 Test [#7](/bitcoin-bitcoin/7/): bench_sanity_check ...............   Passed    9.11 sec
3
4100% tests passed, 0 tests failed out of 1
5
6Total Test time (real) =   9.11 sec

After (3s):

0git checkout fafb0b495235a32b7622f2d9be07ebfbb5344a05 && \
1cmake -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_BENCH=ON && \
2cmake --build build -j$(nproc) && \
3time ./build/test/functional/test_runner.py tool_bench_sanity_check.py -j$(nproc)

results in:

  0TEST                                                                             | STATUS    | DURATION
  1
  2tool_bench_sanity_check.py --bench=AddAndRemoveDisconnectedBlockTransactions10   | ✓ Passed  | 0 s
  3tool_bench_sanity_check.py --bench=AddAndRemoveDisconnectedBlockTransactions90   | ✓ Passed  | 0 s
  4tool_bench_sanity_check.py --bench=AddAndRemoveDisconnectedBlockTransactionsAll  | ✓ Passed  | 0 s
  5tool_bench_sanity_check.py --bench=AddrManAdd                                    | ✓ Passed  | 0 s
  6tool_bench_sanity_check.py --bench=AddrManAddThenGood                            | ✓ Passed  | 0 s
  7tool_bench_sanity_check.py --bench=AddrManGetAddr                                | ✓ Passed  | 0 s
  8tool_bench_sanity_check.py --bench=AddrManSelect                                 | ✓ Passed  | 0 s
  9tool_bench_sanity_check.py --bench=AddrManSelectByNetwork                        | ✓ Passed  | 0 s
 10tool_bench_sanity_check.py --bench=AddrManSelectFromAlmostEmpty                  | ✓ Passed  | 0 s
 11tool_bench_sanity_check.py --bench=AssembleBlock                                 | ✓ Passed  | 0 s
 12tool_bench_sanity_check.py --bench=Base58CheckEncode                             | ✓ Passed  | 0 s
 13tool_bench_sanity_check.py --bench=Base58Decode                                  | ✓ Passed  | 0 s
 14tool_bench_sanity_check.py --bench=Base58Encode                                  | ✓ Passed  | 0 s
 15tool_bench_sanity_check.py --bench=Bech32Decode                                  | ✓ Passed  | 0 s
 16tool_bench_sanity_check.py --bench=Bech32Encode                                  | ✓ Passed  | 0 s
 17tool_bench_sanity_check.py --bench=BenchLockedPool                               | ✓ Passed  | 0 s
 18tool_bench_sanity_check.py --bench=BenchRIPEMD160                                | ✓ Passed  | 0 s
 19tool_bench_sanity_check.py --bench=BenchTimeDeprecated                           | ✓ Passed  | 0 s
 20tool_bench_sanity_check.py --bench=BenchTimeMillis                               | ✓ Passed  | 0 s
 21tool_bench_sanity_check.py --bench=BenchTimeMillisSys                            | ✓ Passed  | 0 s
 22tool_bench_sanity_check.py --bench=BenchTimeMock                                 | ✓ Passed  | 0 s
 23tool_bench_sanity_check.py --bench=BIP324_ECDH                                   | ✓ Passed  | 0 s
 24tool_bench_sanity_check.py --bench=BlockAssemblerAddPackageTxns                  | ✓ Passed  | 0 s
 25tool_bench_sanity_check.py --bench=BlockEncodingLargeExtra                       | ✓ Passed  | 0 s
 26tool_bench_sanity_check.py --bench=BlockEncodingNoExtra                          | ✓ Passed  | 0 s
 27tool_bench_sanity_check.py --bench=BlockEncodingStdExtra                         | ✓ Passed  | 0 s
 28tool_bench_sanity_check.py --bench=BlockFilterIndexSync                          | ✓ Passed  | 0 s
 29tool_bench_sanity_check.py --bench=BlockToJsonVerboseWrite                       | ✓ Passed  | 0 s
 30tool_bench_sanity_check.py --bench=BlockToJsonVerbosity1                         | ✓ Passed  | 0 s
 31tool_bench_sanity_check.py --bench=BlockToJsonVerbosity2                         | ✓ Passed  | 0 s
 32tool_bench_sanity_check.py --bench=BlockToJsonVerbosity3                         | ✓ Passed  | 0 s
 33tool_bench_sanity_check.py --bench=BnBExhaustion                                 | ✓ Passed  | 0 s
 34tool_bench_sanity_check.py --bench=CCheckQueueSpeedPrevectorJob                  | ✓ Passed  | 0 s
 35tool_bench_sanity_check.py --bench=CCoinsCaching                                 | ✓ Passed  | 0 s
 36tool_bench_sanity_check.py --bench=CHACHA20_1MB                                  | ✓ Passed  | 0 s
 37tool_bench_sanity_check.py --bench=CHACHA20_256BYTES                             | ✓ Passed  | 0 s
 38tool_bench_sanity_check.py --bench=CHACHA20_64BYTES                              | ✓ Passed  | 0 s
 39tool_bench_sanity_check.py --bench=CheckBlockIndex                               | ✓ Passed  | 0 s
 40tool_bench_sanity_check.py --bench=CoinSelection                                 | ✓ Passed  | 0 s
 41tool_bench_sanity_check.py --bench=ComplexMemPool                                | ✓ Passed  | 1 s
 42tool_bench_sanity_check.py --bench=ConnectBlockAllEcdsa                          | ✓ Passed  | 0 s
 43tool_bench_sanity_check.py --bench=ConnectBlockAllSchnorr                        | ✓ Passed  | 0 s
 44tool_bench_sanity_check.py --bench=ConnectBlockMixedEcdsaSchnorr                 | ✓ Passed  | 0 s
 45tool_bench_sanity_check.py --bench=DeserializeAndCheckBlockTest                  | ✓ Passed  | 0 s
 46tool_bench_sanity_check.py --bench=DeserializeBlockTest                          | ✓ Passed  | 0 s
 47tool_bench_sanity_check.py --bench=DuplicateInputs                               | ✓ Passed  | 0 s
 48tool_bench_sanity_check.py --bench=EllSwiftCreate                                | ✓ Passed  | 0 s
 49tool_bench_sanity_check.py --bench=EvictionProtection0Networks250Candidates      | ✓ Passed  | 0 s
 50tool_bench_sanity_check.py --bench=EvictionProtection1Networks250Candidates      | ✓ Passed  | 0 s
 51tool_bench_sanity_check.py --bench=EvictionProtection2Networks250Candidates      | ✓ Passed  | 0 s
 52tool_bench_sanity_check.py --bench=EvictionProtection3Networks050Candidates      | ✓ Passed  | 0 s
 53tool_bench_sanity_check.py --bench=EvictionProtection3Networks100Candidates      | ✓ Passed  | 0 s
 54tool_bench_sanity_check.py --bench=EvictionProtection3Networks250Candidates      | ✓ Passed  | 0 s
 55tool_bench_sanity_check.py --bench=ExpandDescriptor                              | ✓ Passed  | 0 s
 56tool_bench_sanity_check.py --bench=FastRandom_rand32                             | ✓ Passed  | 0 s
 57tool_bench_sanity_check.py --bench=FastRandom_rand64                             | ✓ Passed  | 0 s
 58tool_bench_sanity_check.py --bench=FastRandom_randbits                           | ✓ Passed  | 0 s
 59tool_bench_sanity_check.py --bench=FastRandom_randbool                           | ✓ Passed  | 0 s
 60tool_bench_sanity_check.py --bench=FastRandom_randrange100                       | ✓ Passed  | 0 s
 61tool_bench_sanity_check.py --bench=FastRandom_randrange1000                      | ✓ Passed  | 0 s
 62tool_bench_sanity_check.py --bench=FastRandom_randrange1000000                   | ✓ Passed  | 0 s
 63tool_bench_sanity_check.py --bench=FastRandom_stdshuffle100                      | ✓ Passed  | 0 s
 64tool_bench_sanity_check.py --bench=FindByte                                      | ✓ Passed  | 0 s
 65tool_bench_sanity_check.py --bench=FSCHACHA20POLY1305_1MB                        | ✓ Passed  | 0 s
 66tool_bench_sanity_check.py --bench=FSCHACHA20POLY1305_256BYTES                   | ✓ Passed  | 0 s
 67tool_bench_sanity_check.py --bench=FSCHACHA20POLY1305_64BYTES                    | ✓ Passed  | 0 s
 68tool_bench_sanity_check.py --bench=GCSBlockFilterGetHash                         | ✓ Passed  | 0 s
 69tool_bench_sanity_check.py --bench=GCSFilterConstruct                            | ✓ Passed  | 0 s
 70tool_bench_sanity_check.py --bench=GCSFilterDecode                               | ✓ Passed  | 0 s
 71tool_bench_sanity_check.py --bench=GCSFilterDecodeSkipCheck                      | ✓ Passed  | 0 s
 72tool_bench_sanity_check.py --bench=GCSFilterMatch                                | ✓ Passed  | 0 s
 73tool_bench_sanity_check.py --bench=HexParse                                      | ✓ Passed  | 0 s
 74tool_bench_sanity_check.py --bench=HexStrBench                                   | ✓ Passed  | 0 s
 75tool_bench_sanity_check.py --bench=InsecureRandom_rand32                         | ✓ Passed  | 0 s
 76tool_bench_sanity_check.py --bench=InsecureRandom_rand64                         | ✓ Passed  | 0 s
 77tool_bench_sanity_check.py --bench=InsecureRandom_randbits                       | ✓ Passed  | 0 s
 78tool_bench_sanity_check.py --bench=InsecureRandom_randbool                       | ✓ Passed  | 0 s
 79tool_bench_sanity_check.py --bench=InsecureRandom_randrange100                   | ✓ Passed  | 0 s
 80tool_bench_sanity_check.py --bench=InsecureRandom_randrange1000                  | ✓ Passed  | 0 s
 81tool_bench_sanity_check.py --bench=InsecureRandom_randrange1000000               | ✓ Passed  | 0 s
 82tool_bench_sanity_check.py --bench=InsecureRandom_stdshuffle100                  | ✓ Passed  | 0 s
 83tool_bench_sanity_check.py --bench=LinearizeOptimallyPerCost                     | ✓ Passed  | 0 s
 84tool_bench_sanity_check.py --bench=LinearizeOptimallyTotal                       | ✓ Passed  | 0 s
 85tool_bench_sanity_check.py --bench=LoadExternalBlockFile                         | ✓ Passed  | 0 s
 86tool_bench_sanity_check.py --bench=LogWithDebug                                  | ✓ Passed  | 0 s
 87tool_bench_sanity_check.py --bench=LogWithoutDebug                               | ✓ Passed  | 0 s
 88tool_bench_sanity_check.py --bench=LogWithoutThreadNames                         | ✓ Passed  | 0 s
 89tool_bench_sanity_check.py --bench=LogWithoutWriteToFile                         | ✓ Passed  | 0 s
 90tool_bench_sanity_check.py --bench=LogWithThreadNames                            | ✓ Passed  | 0 s
 91tool_bench_sanity_check.py --bench=MemPoolAddTransactions                        | ✓ Passed  | 0 s
 92tool_bench_sanity_check.py --bench=MemPoolAncestorsDescendants                   | ✓ Passed  | 0 s
 93tool_bench_sanity_check.py --bench=MempoolCheck                                  | ✓ Passed  | 0 s
 94tool_bench_sanity_check.py --bench=MempoolCheckEphemeralSpends                   | ✓ Passed  | 0 s
 95tool_bench_sanity_check.py --bench=MempoolEviction                               | ✓ Passed  | 0 s
 96tool_bench_sanity_check.py --bench=MerkleRoot                                    | ✓ Passed  | 0 s
 97tool_bench_sanity_check.py --bench=MuHash                                        | ✓ Passed  | 0 s
 98tool_bench_sanity_check.py --bench=MuHashDiv                                     | ✓ Passed  | 0 s
 99tool_bench_sanity_check.py --bench=MuHashFinalize                                | ✓ Passed  | 0 s
100tool_bench_sanity_check.py --bench=MuHashMul                                     | ✓ Passed  | 0 s
101tool_bench_sanity_check.py --bench=MuHashPrecompute                              | ✓ Passed  | 0 s
102tool_bench_sanity_check.py --bench=ObfuscationBench                              | ✓ Passed  | 0 s
103tool_bench_sanity_check.py --bench=OrphanageEraseForBlock                        | ✓ Passed  | 0 s
104tool_bench_sanity_check.py --bench=OrphanageEraseForPeer                         | ✓ Passed  | 0 s
105tool_bench_sanity_check.py --bench=OrphanageMultiPeerEviction                    | ✓ Passed  | 0 s
106tool_bench_sanity_check.py --bench=OrphanageSinglePeerEviction                   | ✓ Passed  | 0 s
107tool_bench_sanity_check.py --bench=POLY1305_1MB                                  | ✓ Passed  | 0 s
108tool_bench_sanity_check.py --bench=POLY1305_256BYTES                             | ✓ Passed  | 0 s
109tool_bench_sanity_check.py --bench=POLY1305_64BYTES                              | ✓ Passed  | 0 s
110tool_bench_sanity_check.py --bench=PoolAllocator_StdUnorderedMap                 | ✓ Passed  | 0 s
111tool_bench_sanity_check.py --bench=PoolAllocator_StdUnorderedMapWithPoolResource | ✓ Passed  | 0 s
112tool_bench_sanity_check.py --bench=PostLinearize16TxWorstCase                    | ✓ Passed  | 0 s
113tool_bench_sanity_check.py --bench=PostLinearize32TxWorstCase                    | ✓ Passed  | 0 s
114tool_bench_sanity_check.py --bench=PostLinearize48TxWorstCase                    | ✓ Passed  | 0 s
115tool_bench_sanity_check.py --bench=PostLinearize64TxWorstCase                    | ✓ Passed  | 0 s
116tool_bench_sanity_check.py --bench=PostLinearize75TxWorstCase                    | ✓ Passed  | 0 s
117tool_bench_sanity_check.py --bench=PostLinearize99TxWorstCase                    | ✓ Passed  | 0 s
118tool_bench_sanity_check.py --bench=PrePadded                                     | ✓ Passed  | 0 s
119tool_bench_sanity_check.py --bench=PrevectorClearNontrivial                      | ✓ Passed  | 0 s
120tool_bench_sanity_check.py --bench=PrevectorClearTrivial                         | ✓ Passed  | 0 s
121tool_bench_sanity_check.py --bench=PrevectorDeserializeNontrivial                | ✓ Passed  | 0 s
122tool_bench_sanity_check.py --bench=PrevectorDeserializeTrivial                   | ✓ Passed  | 0 s
123tool_bench_sanity_check.py --bench=PrevectorDestructorNontrivial                 | ✓ Passed  | 0 s
124tool_bench_sanity_check.py --bench=PrevectorDestructorTrivial                    | ✓ Passed  | 0 s
125tool_bench_sanity_check.py --bench=PrevectorFillVectorDirectNontrivial           | ✓ Passed  | 0 s
126tool_bench_sanity_check.py --bench=PrevectorFillVectorDirectTrivial              | ✓ Passed  | 0 s
127tool_bench_sanity_check.py --bench=PrevectorFillVectorIndirectNontrivial         | ✓ Passed  | 0 s
128tool_bench_sanity_check.py --bench=PrevectorFillVectorIndirectTrivial            | ✓ Passed  | 0 s
129tool_bench_sanity_check.py --bench=PrevectorResizeNontrivial                     | ✓ Passed  | 0 s
130tool_bench_sanity_check.py --bench=PrevectorResizeTrivial                        | ✓ Passed  | 0 s
131tool_bench_sanity_check.py --bench=ReadBlockBench                                | ✓ Passed  | 0 s
132tool_bench_sanity_check.py --bench=ReadRawBlockBench                             | ✓ Passed  | 0 s
133tool_bench_sanity_check.py --bench=RegularPadded                                 | ✓ Passed  | 0 s
134tool_bench_sanity_check.py --bench=RollingBloom                                  | ✓ Passed  | 0 s
135tool_bench_sanity_check.py --bench=RollingBloomReset                             | ✓ Passed  | 0 s
136tool_bench_sanity_check.py --bench=RpcMempool                                    | ✓ Passed  | 0 s
137tool_bench_sanity_check.py --bench=SHA1                                          | ✓ Passed  | 0 s
138tool_bench_sanity_check.py --bench=SHA256_32b_AVX2                               | ✓ Passed  | 0 s
139tool_bench_sanity_check.py --bench=SHA256_32b_SHANI                              | ✓ Passed  | 0 s
140tool_bench_sanity_check.py --bench=SHA256_32b_SSE4                               | ✓ Passed  | 0 s
141tool_bench_sanity_check.py --bench=SHA256_32b_STANDARD                           | ✓ Passed  | 0 s
142tool_bench_sanity_check.py --bench=SHA256_AVX2                                   | ✓ Passed  | 0 s
143tool_bench_sanity_check.py --bench=SHA256_SHANI                                  | ✓ Passed  | 0 s
144tool_bench_sanity_check.py --bench=SHA256_SSE4                                   | ✓ Passed  | 0 s
145tool_bench_sanity_check.py --bench=SHA256_STANDARD                               | ✓ Passed  | 0 s
146tool_bench_sanity_check.py --bench=SHA256D64_1024_AVX2                           | ✓ Passed  | 0 s
147tool_bench_sanity_check.py --bench=SHA256D64_1024_SHANI                          | ✓ Passed  | 0 s
148tool_bench_sanity_check.py --bench=SHA256D64_1024_SSE4                           | ✓ Passed  | 0 s
149tool_bench_sanity_check.py --bench=SHA256D64_1024_STANDARD                       | ✓ Passed  | 0 s
150tool_bench_sanity_check.py --bench=SHA3_256_1M                                   | ✓ Passed  | 0 s
151tool_bench_sanity_check.py --bench=SHA512                                        | ✓ Passed  | 0 s
152tool_bench_sanity_check.py --bench=SignSchnorrWithMerkleRoot                     | ✓ Passed  | 0 s
153tool_bench_sanity_check.py --bench=SignSchnorrWithNullMerkleRoot                 | ✓ Passed  | 0 s
154tool_bench_sanity_check.py --bench=SignTransactionECDSA                          | ✓ Passed  | 0 s
155tool_bench_sanity_check.py --bench=SignTransactionSchnorr                        | ✓ Passed  | 0 s
156tool_bench_sanity_check.py --bench=SipHash_32b                                   | ✓ Passed  | 0 s
157tool_bench_sanity_check.py --bench=Trig                                          | ✓ Passed  | 0 s
158tool_bench_sanity_check.py --bench=TxGraphTrim                                   | ✓ Passed  | 0 s
159tool_bench_sanity_check.py --bench=VerifyNestedIfScript                          | ✓ Passed  | 0 s
160tool_bench_sanity_check.py --bench=VerifyScriptBench                             | ✓ Passed  | 0 s
161tool_bench_sanity_check.py --bench=WalletAvailableCoins                          | ✓ Passed  | 0 s
162tool_bench_sanity_check.py --bench=WalletBalanceClean                            | ✓ Passed  | 0 s
163tool_bench_sanity_check.py --bench=WalletBalanceDirty                            | ✓ Passed  | 0 s
164tool_bench_sanity_check.py --bench=WalletBalanceMine                             | ✓ Passed  | 0 s
165tool_bench_sanity_check.py --bench=WalletBalanceWatch                            | ✓ Passed  | 0 s
166tool_bench_sanity_check.py --bench=WalletCreateEncrypted                         | ✓ Passed  | 0 s
167tool_bench_sanity_check.py --bench=WalletCreatePlain                             | ✓ Passed  | 0 s
168tool_bench_sanity_check.py --bench=WalletCreateTxUseOnlyPresetInputs             | ✓ Passed  | 0 s
169tool_bench_sanity_check.py --bench=WalletCreateTxUsePresetInputsAndCoinSelection | ✓ Passed  | 0 s
170tool_bench_sanity_check.py --bench=WalletIsMineDescriptors                       | ✓ Passed  | 0 s
171tool_bench_sanity_check.py --bench=WalletIsMineMigratedDescriptors               | ✓ Passed  | 0 s
172tool_bench_sanity_check.py --bench=WalletLoadingDescriptors                      | ✓ Passed  | 1 s
173tool_bench_sanity_check.py --bench=WalletMigration                               | ✓ Passed  | 2 s
174tool_bench_sanity_check.py --bench=WriteBlockBench                               | ✓ Passed  | 0 s
175
176ALL                                                                              | ✓ Passed  | 4 s (accumulated) 
177Runtime: 3 s

Concept ACK - I will review the code a bit later.

in test/functional/test_framework/test_framework.py:949 in fafb0b4952

942@@ -943,6 +943,11 @@ def skip_if_no_bitcoin_chainstate(self):
943         if not self.is_bitcoin_chainstate_compiled():
944             raise SkipTest("bitcoin-chainstate has not been compiled")
945 
946+    def skip_if_no_bitcoin_bench(self):
947+        """Skip the running test if bitcoin_bench has not been compiled."""
948+        if not self.is_bench_compiled():
949+            raise SkipTest("bitcoin_bench has not been compiled")

l0rinc commented at 6:48 pm on January 1, 2026:

Debatable if it should be bitcoin_bench (https://github.com/bitcoin/bitcoin/pull/33142/files#diff-e20e1f68486e5c096fdc11bca1cda063aacef524411f011d9c3497eb650c5eafR265) or bench_bitcoin (https://github.com/bitcoin/bitcoin/pull/33142/files#diff-e20e1f68486e5c096fdc11bca1cda063aacef524411f011d9c3497eb650c5eafR308), but this way the error (and a few other places) are confusing.

0            raise SkipTest("bench_bitcoin has not been compiled")

in test/functional/test_runner.py:533 in fafb0b4952

528+        test_list.remove(TOOL_BENCH_SANITY_CHECK)
529+        bench_cmd = Binaries(get_binary_paths(config), bin_dir=None).bench_argv() + ["-list"]
530+        bench_list = subprocess.check_output(bench_cmd).decode("ascii").splitlines()
531+        bench_list = [f"{TOOL_BENCH_SANITY_CHECK} --bench={b}" for b in bench_list]
532+        # Start with special scripts (variable, unknown runtime)
533+        test_list.extendleft(bench_list)

l0rinc commented at 7:19 pm on January 1, 2026:

./build/test/functional/test_runner.py tool_bench_sanity_check.py | grep 'Duration:' shows the list in reverse order:

 0Temporary test directory at /var/folders/5t/04gq0pqj5yv4t8cxw51q0s2m0000gn/T/test_runner_₿_🏃_20260101_201856
 11/173 - tool_bench_sanity_check.py --bench=WriteBlockBench passed, Duration: 0 s
 22/173 - tool_bench_sanity_check.py --bench=WalletIsMineMigratedDescriptors passed, Duration: 0 s
 33/173 - tool_bench_sanity_check.py --bench=WalletIsMineDescriptors passed, Duration: 0 s
 44/173 - tool_bench_sanity_check.py --bench=WalletCreateTxUsePresetInputsAndCoinSelection passed, Duration: 0 s
 55/173 - tool_bench_sanity_check.py --bench=WalletCreateTxUseOnlyPresetInputs passed, Duration: 0 s
 66/173 - tool_bench_sanity_check.py --bench=WalletLoadingDescriptors passed, Duration: 1 s
 77/173 - tool_bench_sanity_check.py --bench=WalletCreatePlain passed, Duration: 0 s
 88/173 - tool_bench_sanity_check.py --bench=WalletBalanceWatch passed, Duration: 0 s
 99/173 - tool_bench_sanity_check.py --bench=WalletCreateEncrypted passed, Duration: 0 s
10...

0        test_list.extendleft(reversed(bench_list))

in test/functional/test_runner.py:530 in fafb0b4952

522@@ -511,6 +523,15 @@ def remove_tests(exclude_list):
523                 # Exclude all variants of a test
524                 remove_tests([test for test in test_list if test.split('.py')[0] == exclude_test.split('.py')[0]])
525 
526+    if config["components"].getboolean("BUILD_BENCH") and TOOL_BENCH_SANITY_CHECK in test_list:
527+        # Remove it, and expand it for each bench in the list
528+        test_list.remove(TOOL_BENCH_SANITY_CHECK)
529+        bench_cmd = Binaries(get_binary_paths(config), bin_dir=None).bench_argv() + ["-list"]
530+        bench_list = subprocess.check_output(bench_cmd).decode("ascii").splitlines()

l0rinc commented at 7:21 pm on January 1, 2026:

In other cases we just specify that the output is text: https://github.com/bitcoin/bitcoin/blob/fab085c15f7221986f73af7e05e799edf3eadaf0/contrib/verify-commits/verify-commits.py#L155

0        bench_list = subprocess.check_output(bench_cmd, text=True).splitlines()

Note: the failure is really weird if bitcoin_bench isn’t in the bin folder, we might want to give a better error message in that case.

maflcko commented at 8:04 pm on January 1, 2026:

thanks for the diff! I pushed it as-is.

Note: the failure is really weird if bitcoin_bench isn’t in the bin folder, we might want to give a better error message in that case.

It fails correctly with FileNotFoundError: [Errno 2] No such file or directory in that case. This seems desirable and intentional. Also, it shouldn’t happen in any case, unless the dev manually removes the file, which is not a supported mode of operation anyway. So i left this as-is for now.

l0rinc commented at 8:07 pm on January 1, 2026:

unless the dev manually removes the file

I had a build failure when I ran it - but I’m fine either way…

in test/functional/tool_bench_sanity_check.py:36 in fafb0b4952

31+            f"-filter={self.options.bench}",
32+            "-sanity-check",
33+        ]
34+        self.log.info(f"Starting to run: {cmd}")
35+        subprocess.run(cmd, check=True)
36+        self.log.info(f"Command passed: {cmd}")

l0rinc commented at 7:24 pm on January 1, 2026:

In other cases we’re printing the cmd via shlex.join(cmd): https://github.com/bitcoin/bitcoin/blob/c80fd910f96cc58ebd9cb2141b3a8cdc2382ccc1/ci/test/02_run_container.py#L15

Q: what’s the point of printing the cmd twice?

0        self.log.info(f"Starting: {shlex.join(cmd)}")
1        subprocess.run(cmd, check=True)
2        self.log.info("Success!")

l0rinc changes_requested

l0rinc commented at 7:31 pm on January 1, 2026: contributor

The change makes sense, I think we should fix a few references to the benchmarking binary (and keep the alphabetical order of the benchmarks), otherwise LGTM.

 0diff --git a/test/functional/test_framework/test_framework.py b/test/functional/test_framework/test_framework.py
 1index 31660cad9c..ecc9ffa231 100755
 2--- a/test/functional/test_framework/test_framework.py
 3+++ b/test/functional/test_framework/test_framework.py
 4@@ -944,9 +944,9 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass):
 5             raise SkipTest("bitcoin-chainstate has not been compiled")
 6 
 7     def skip_if_no_bitcoin_bench(self):
 8-        """Skip the running test if bitcoin_bench has not been compiled."""
 9+        """Skip the running test if bench_bitcoin has not been compiled."""
10         if not self.is_bench_compiled():
11-            raise SkipTest("bitcoin_bench has not been compiled")
12+            raise SkipTest("bench_bitcoin has not been compiled")
13 
14     def skip_if_no_cli(self):
15         """Skip the running test if bitcoin-cli has not been compiled."""
16@@ -982,7 +982,7 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass):
17             raise SkipTest("This test is not compatible with Valgrind.")
18 
19     def is_bench_compiled(self):
20-        """Checks whether bitcoin_bench was compiled."""
21+        """Checks whether bench_bitcoin was compiled."""
22         return self.config["components"].getboolean("BUILD_BENCH")
23 
24     def is_cli_compiled(self):
25diff --git a/test/functional/test_framework/util.py b/test/functional/test_framework/util.py
26index d53c59297f..0c190ee936 100644
27--- a/test/functional/test_framework/util.py
28+++ b/test/functional/test_framework/util.py
29@@ -262,7 +262,7 @@ class Binaries:
30         return self._argv("rpc", self.paths.bitcoincli) + ["-nonamed"]
31 
32     def bench_argv(self):
33-        "Return argv array that should be used to invoke bitcoin_bench"
34+        "Return argv array that should be used to invoke bench_bitcoin"
35         return self._argv("bench", self.paths.bitcoin_bench)
36 
37     def tx_argv(self):
38diff --git a/test/functional/test_runner.py b/test/functional/test_runner.py
39index 28ef0d010f..a4af557846 100755
40--- a/test/functional/test_runner.py
41+++ b/test/functional/test_runner.py
42@@ -528,10 +528,10 @@ def main():
43         # Remove it, and expand it for each bench in the list
44         test_list.remove(TOOL_BENCH_SANITY_CHECK)
45         bench_cmd = Binaries(get_binary_paths(config), bin_dir=None).bench_argv() + ["-list"]
46-        bench_list = subprocess.check_output(bench_cmd).decode("ascii").splitlines()
47+        bench_list = subprocess.check_output(bench_cmd, text=True).splitlines()
48         bench_list = [f"{TOOL_BENCH_SANITY_CHECK} --bench={b}" for b in bench_list]
49         # Start with special scripts (variable, unknown runtime)
50-        test_list.extendleft(bench_list)
51+        test_list.extendleft(reversed(bench_list))
52 
53     if args.filter:
54         test_list = deque(filter(re.compile(args.filter).search, test_list))
55diff --git a/test/functional/tool_bench_sanity_check.py b/test/functional/tool_bench_sanity_check.py
56index ad70fcb9be..d29587861c 100755
57--- a/test/functional/tool_bench_sanity_check.py
58+++ b/test/functional/tool_bench_sanity_check.py
59@@ -4,6 +4,7 @@
60 # file COPYING or https://opensource.org/license/mit/.
61 """Special script to run each bench sanity check
62 """
63+import shlex
64 import subprocess
65 
66 from test_framework.test_framework import BitcoinTestFramework
67@@ -31,9 +32,9 @@ class BenchSanityCheck(BitcoinTestFramework):
68             f"-filter={self.options.bench}",
69             "-sanity-check",
70         ]
71-        self.log.info(f"Starting to run: {cmd}")
72+        self.log.info(f"Starting: {shlex.join(cmd)}")
73         subprocess.run(cmd, check=True)
74-        self.log.info(f"Command passed: {cmd}")
75+        self.log.info("Success!")
76 
77 
78 if __name__ == "__main__":

test: Pass bench exe into test framework utils

This teaches the test framework about the bench executable, which is
required for the next commit.

fa9fdbce79

test: Run bench sanity checks in parallel with functional tests fa65bc0e79

maflcko force-pushed on Jan 1, 2026

l0rinc approved

l0rinc commented at 8:10 pm on January 1, 2026: contributor

Tested ACK fa65bc0e79dab8a97e0e3f9f31540f1c029a2f6e

maflcko requested review from janb84 on Jan 1, 2026

janb84 commented at 8:33 am on January 5, 2026: contributor

tACK fa65bc0e79dab8a97e0e3f9f31540f1c029a2f6e

Changes since last ACK: mainly related to using bench_bitcoin instead of bitcoin_bench keeping the ordering the same by reversing an (unwanted) reversed list.

willcl-ark commented at 2:04 pm on January 5, 2026: member

Also verified the speedup:

master:

0src/core/bitcoin on  master [$?⇕] via △ v4.1.2 via 🐍 v3.13.9 via ❄️  impure (nix-shell-env)
1❯ ctest --test-dir ./build --tests-regex bench_sanity_check
2Test project /home/will/src/core/bitcoin/build
3    Start 7: bench_sanity_check
41/1 Test [#7](/bitcoin-bitcoin/7/): bench_sanity_check ...............   Passed   18.20 sec
5
6100% tests passed, 0 tests failed out of 1
7
8Total Test time (real) =  18.21 sec

PR #33142:

0src/core/bitcoin on  pr-33142 [$?] via △ v4.1.2 via 🐍 v3.13.9 via ❄️  impure (nix-shell-env) took 25s
1❯ build/test/functional/test_runner.py -j 128 tool_bench_sanity_check.py
2Temporary test directory at /tmp/test_runner_₿_🏃_20260105_135859
3<snip>
4ALL                                                                              | ✓ Passed  | 272 s (accumulated)
5Runtime: 8 s

willcl-ark approved

willcl-ark commented at 2:08 pm on January 5, 2026: member

ACK fa65bc0e79dab8a97e0e3f9f31540f1c029a2f6e

This test alone can take 2 minutes or longer in CI, which is reduced here to ~20 seconds when parallelised as here in this changeset. Agree that we can revert if/when we move to a more cmake-native solution in the future.

achow101 commented at 11:27 pm on January 5, 2026: member

ACK fa65bc0e79dab8a97e0e3f9f31540f1c029a2f6e

Merging this, but this is not saying that this direction is set in stone. This PR gives us some immediate wins that are useful to all developers with a fairly small amount of code changed. However, rewriting everything to utilize ctest is still on the table, but that does seem like a much larger task that will take significantly longer to implement and review.

achow101 merged this on Jan 5, 2026

achow101 closed this on Jan 5, 2026

maflcko deleted the branch on Jan 6, 2026

test: Run bench sanity checks in parallel with functional tests #33142

Code Coverage & Benchmarks

Reviews

Conflicts