cmake: Split test cases to improve parallelism

hebasto commented at 2:24 pm on October 16, 2025: member

This PR implements the approach suggested in #1734#pullrequestreview-3284918572.

Here is an example of the ctest output:

 0$ ctest --test-dir build -j $(nproc)
 1Test project /home/hebasto/dev/secp256k1/secp256k1/build
 2      Start  1: secp256k1_noverify::ellswift
 3      Start  2: secp256k1_noverify::ecmult_constants
 4      Start  3: secp256k1_noverify::ecmult
 5      Start  4: secp256k1_noverify::integer
 6      Start  5: secp256k1_noverify::scalar
 7      Start  6: secp256k1_noverify::field
 8      Start  7: secp256k1_noverify::group
 9      Start  8: secp256k1_noverify::ec
10      Start  9: secp256k1_noverify::ecdh
11      Start 10: secp256k1_noverify::ecdsa
12      Start 11: secp256k1_noverify::recovery
13      Start 12: secp256k1_noverify::extrakeys
14      Start 13: secp256k1_noverify::schnorrsig
15      Start 14: secp256k1_noverify::musig
16      Start 15: secp256k1_noverify::general,hash,utils
17      Start 16: secp256k1_verify::ellswift
18 1/36 Test [#11](/bitcoin-core-secp256k1/11/): secp256k1_noverify::recovery .............***Skipped   0.00 sec
19      Start 17: secp256k1_verify::ecmult_constants
20 2/36 Test [#12](/bitcoin-core-secp256k1/12/): secp256k1_noverify::extrakeys ............   Passed    0.01 sec
21      Start 18: secp256k1_verify::ecmult
22 3/36 Test  [#8](/bitcoin-core-secp256k1/8/): secp256k1_noverify::ec ...................   Passed    0.01 sec
23      Start 19: secp256k1_verify::integer
24 4/36 Test  [#5](/bitcoin-core-secp256k1/5/): secp256k1_noverify::scalar ...............   Passed    0.02 sec
25      Start 20: secp256k1_verify::scalar
26 5/36 Test [#15](/bitcoin-core-secp256k1/15/): secp256k1_noverify::general,hash,utils ...   Passed    0.02 sec
27      Start 21: secp256k1_verify::field
28 6/36 Test [#13](/bitcoin-core-secp256k1/13/): secp256k1_noverify::schnorrsig ...........   Passed    0.03 sec
29      Start 22: secp256k1_verify::group
30 7/36 Test  [#9](/bitcoin-core-secp256k1/9/): secp256k1_noverify::ecdh .................   Passed    0.04 sec
31      Start 23: secp256k1_verify::ec
32 8/36 Test [#23](/bitcoin-core-secp256k1/23/): secp256k1_verify::ec .....................   Passed    0.03 sec
33      Start 24: secp256k1_verify::ecdh
34 9/36 Test [#20](/bitcoin-core-secp256k1/20/): secp256k1_verify::scalar .................   Passed    0.05 sec
35      Start 25: secp256k1_verify::ecdsa
3610/36 Test [#14](/bitcoin-core-secp256k1/14/): secp256k1_noverify::musig ................   Passed    0.10 sec
37      Start 26: secp256k1_verify::recovery
3811/36 Test [#26](/bitcoin-core-secp256k1/26/): secp256k1_verify::recovery ...............***Skipped   0.00 sec
39      Start 27: secp256k1_verify::extrakeys
4012/36 Test [#27](/bitcoin-core-secp256k1/27/): secp256k1_verify::extrakeys ..............   Passed    0.02 sec
41      Start 28: secp256k1_verify::schnorrsig
4213/36 Test [#24](/bitcoin-core-secp256k1/24/): secp256k1_verify::ecdh ...................   Passed    0.12 sec
43      Start 29: secp256k1_verify::musig
4414/36 Test [#28](/bitcoin-core-secp256k1/28/): secp256k1_verify::schnorrsig .............   Passed    0.07 sec
45      Start 30: secp256k1_verify::general,hash,utils
4615/36 Test [#30](/bitcoin-core-secp256k1/30/): secp256k1_verify::general,hash,utils .....   Passed    0.02 sec
47      Start 31: secp256k1_exhaustive
4816/36 Test [#29](/bitcoin-core-secp256k1/29/): secp256k1_verify::musig ..................   Passed    0.24 sec
49      Start 32: secp256k1_example::ecdsa
5017/36 Test [#32](/bitcoin-core-secp256k1/32/): secp256k1_example::ecdsa .................   Passed    0.00 sec
51      Start 33: secp256k1_example::ecdh
5218/36 Test [#33](/bitcoin-core-secp256k1/33/): secp256k1_example::ecdh ..................   Passed    0.00 sec
53      Start 34: secp256k1_example::schnorr
5419/36 Test [#34](/bitcoin-core-secp256k1/34/): secp256k1_example::schnorr ...............   Passed    0.00 sec
55      Start 35: secp256k1_example::ellswift
5620/36 Test [#35](/bitcoin-core-secp256k1/35/): secp256k1_example::ellswift ..............   Passed    0.00 sec
57      Start 36: secp256k1_example::musig
5821/36 Test [#36](/bitcoin-core-secp256k1/36/): secp256k1_example::musig .................   Passed    0.00 sec
5922/36 Test [#10](/bitcoin-core-secp256k1/10/): secp256k1_noverify::ecdsa ................   Passed    0.89 sec
6023/36 Test  [#6](/bitcoin-core-secp256k1/6/): secp256k1_noverify::field ................   Passed    1.38 sec
6124/36 Test [#21](/bitcoin-core-secp256k1/21/): secp256k1_verify::field ..................   Passed    1.47 sec
6225/36 Test  [#7](/bitcoin-core-secp256k1/7/): secp256k1_noverify::group ................   Passed    1.51 sec
6326/36 Test [#25](/bitcoin-core-secp256k1/25/): secp256k1_verify::ecdsa ..................   Passed    1.68 sec
6427/36 Test [#19](/bitcoin-core-secp256k1/19/): secp256k1_verify::integer ................   Passed    2.76 sec
6528/36 Test  [#4](/bitcoin-core-secp256k1/4/): secp256k1_noverify::integer ..............   Passed    3.16 sec
6629/36 Test  [#3](/bitcoin-core-secp256k1/3/): secp256k1_noverify::ecmult ...............   Passed    3.39 sec
6730/36 Test  [#1](/bitcoin-core-secp256k1/1/): secp256k1_noverify::ellswift .............   Passed    3.74 sec
6831/36 Test [#22](/bitcoin-core-secp256k1/22/): secp256k1_verify::group ..................   Passed    3.97 sec
6932/36 Test  [#2](/bitcoin-core-secp256k1/2/): secp256k1_noverify::ecmult_constants .....   Passed    5.15 sec
7033/36 Test [#18](/bitcoin-core-secp256k1/18/): secp256k1_verify::ecmult .................   Passed    6.18 sec
7134/36 Test [#31](/bitcoin-core-secp256k1/31/): secp256k1_exhaustive .....................   Passed    7.53 sec
7235/36 Test [#17](/bitcoin-core-secp256k1/17/): secp256k1_verify::ecmult_constants .......   Passed    9.50 sec
7336/36 Test [#16](/bitcoin-core-secp256k1/16/): secp256k1_verify::ellswift ...............   Passed   12.04 sec
74
75100% tests passed, 0 tests failed out of 36
76
77Label Time Summary:
78secp256k1_example       =   0.01 sec*proc (5 tests)
79secp256k1_exhaustive    =   7.53 sec*proc (1 test)
80secp256k1_noverify      =  19.46 sec*proc (15 tests)
81secp256k1_verify        =  38.16 sec*proc (15 tests)
82
83Total Test time (real) =  12.05 sec
84
85The following tests did not run:
86	 11 - secp256k1_noverify::recovery (Skipped)
87	 26 - secp256k1_verify::recovery (Skipped)

real-or-random commented at 7:02 am on October 17, 2025: contributor

The output is nice for sure, but wouldn’t it be much simpler to just invoke the test binaries with -j?

My concern is about the list of tests here. It will require maintenance work, and it’s easy to forget a test.

real-or-random added the label ci on Oct 17, 2025

real-or-random added the label tweak/refactor on Oct 17, 2025

sipa commented at 1:06 pm on October 17, 2025: contributor

The output is nice for sure, but wouldn’t it be much simpler to just invoke the test binaries with -j?

I don’t think that’s desirable, because it lacks the ability to share the granularity with other test targets.

Specifically, I’m thinking in the context of the Bitcoin Core build, which has tons of test cases of its own. The ideal scenario is that you have a single process pool (controlled by the -j argument to cmake), and it is used to run both the Bitcoin Core and libsecp256k1 (and other) tests. This will generally result in all processes making progress throughout the entire run, until cmake runs out of test tasks to schedule.

In contrast, if you push the task scheduling down to libsecp256k1’s own test binary, you at best end up with a situation where you first run all Bitcoin Core tests, wait for those to complete (temporarily running out of tasks in the process), and then run libsecp256k1’s tests with -j. I don’t know if that’s actually possible to do in cmake’s supported generators, and more likely you’ll end up in a situation where the entire libsecp256k1 test run (-j and all) is seen as a single process from the perspective of the build system - meaning it’d simultaneously run N-1 Bitcoin Core tests, plus 1 libsecp256k1 test (which itself runs in N processes), temporarily raising the parallellism to 2N-1.

in src/CMakeLists.txt:163 in 225103b4e1 outdated

166   if(NOT CMAKE_BUILD_TYPE STREQUAL "Coverage")
167-    add_executable(tests tests.c)
168-    target_compile_definitions(tests PRIVATE VERIFY ${TEST_DEFINITIONS})
169-    target_link_libraries(tests secp256k1_precomputed secp256k1_asm)
170-    add_test(NAME secp256k1_tests COMMAND tests)
171+    add_executable_and_tests(YES)

furszy commented at 2:04 pm on October 17, 2025:

Could write this shorter as:

 0function(add_executable_and_tests bin_name verify_definition)
 1  add_executable(${bin_name} tests.c)
 2  target_link_libraries(${bin_name} secp256k1_precomputed secp256k1_asm)
 3  target_compile_definitions(${bin_name} PRIVATE ${verify_definition} ${TEST_DEFINITIONS})
 4  add_test(NAME secp256k1_${bin_name} COMMAND ${bin_name})
 5endfunction()
 6
 7add_executable_and_tests(noverify_tests "")
 8if(NOT CMAKE_BUILD_TYPE STREQUAL "Coverage")
 9  add_executable_and_tests(tests VERIFY)
10endif()

Also, probably the target_link_libraries should be PRIVATE too.

See 00f20f694704079f1f9c05851e7407883763268e (I applied it on top of your first commit - could directly squash it there)

hebasto commented at 7:43 am on October 21, 2025:

Also, probably the target_link_libraries should be PRIVATE too.

All target_link_libraries commands should use the same signature. I think this change should be made in a separate PR.

hebasto commented at 7:58 am on October 21, 2025:

Could write this shorter as: …

Thanks! Reworked.

furszy commented at 3:28 pm on October 17, 2025: member

Regarding the maintenance topic, wouldn’t be better to call --list_tests to fetch the test names instead of manually adding them one by one? We could make --list_tests=2 output the tests names separated by commas to make your life easier at the parsing side.

I experimented a bit with this idea and was able to dynamically generate a file containing all the tests names. See https://github.com/furszy/secp256k1/tree/2025_ci_concurrent_tests (just need to run the tests_discover target). The next step would be to connect this to run separate ctest targets in some way, which doesn’t seem to be so simple because the binary does not exist during configure time but ctest targets are added there.. (chicken-egg problem). Maybe we could execute a custom script that reads the file and launches per target ctest runs dynamically too.

hebasto commented at 7:36 am on October 21, 2025: member

The output is nice for sure, but wouldn’t it be much simpler to just invoke the test binaries with -j?

I don’t think that’s desirable, because it lacks the ability to share the granularity with other test targets.

Specifically, I’m thinking in the context of the Bitcoin Core build, which has tons of test cases of its own. The ideal scenario is that you have a single process pool (controlled by the -j argument to cmake), and it is used to run both the Bitcoin Core and libsecp256k1 (and other) tests. This will generally result in all processes making progress throughout the entire run, until cmake runs out of test tasks to schedule.

I share the same perspective.

hebasto commented at 7:39 am on October 21, 2025: member

Regarding the maintenance topic, wouldn’t be better to call --list_tests to fetch the test names instead of manually adding them one by one? We could make --list_tests=2 output the tests names separated by commas to make your life easier at the parsing side.

I experimented a bit with this idea and was able to dynamically generate a file containing all the tests names. See https://github.com/furszy/secp256k1/tree/2025_ci_concurrent_tests (just need to run the tests_discover target). The next step would be to connect this to run separate ctest targets in some way, which doesn’t seem to be so simple because the binary does not exist during configure time but ctest targets are added there.. (chicken-egg problem). Maybe we could execute a custom script that reads the file and launches per target ctest runs dynamically too. @purpleKarrot’s approach could be applied as a follow-up.

cmake, refactor: De-duplicate test-related code

Co-authored-by: furszy <matiasfurszyfer@protonmail.com>

7a9dfedda2

cmake: Split test cases to improve parallelism 7600f5f5fb

cmake: Set `LABELS` property for tests 2d8da55816

hebasto force-pushed on Oct 21, 2025

hebasto commented at 7:58 am on October 21, 2025: member

Feedback from @furszy has been addressed.

in src/CMakeLists.txt:157 in 7600f5f5fb outdated

153+  list(APPEND test_targets ecmult_constants)
154+  list(APPEND test_names   ecmult_constants)
155+  list(APPEND test_targets "ecmult_pre_g wnaf point_times_order ecmult_near_split_bound ecmult_chain ecmult_gen_blind ecmult_const_tests ecmult_multi_tests ec_combine")
156+  list(APPEND test_names   ecmult)
157+  # Continue with the remaining tests.
158+  list(APPEND test_targets integer scalar field group ec ecdh ecdsa recovery extrakeys schnorrsig musig)

real-or-random commented at 9:59 am on October 22, 2025:

Is there a reason why you have quotation marks in the first line here but not in the last?

hebasto commented at 8:27 am on October 23, 2025:

The first line has quotes to combine the quoted test cases into a single ctest’s test “ecmult”.

In the last line, all mentioned test cases and modules are wrapped as its own ctest’s tests.

purpleKarrot commented at 1:58 pm on October 30, 2025:

I take @real-or-random’s question as a proof that procedural logic in CMakeLists.txt is hard to reason about and should be avoided.

purpleKarrot commented at 2:21 pm on October 30, 2025:

If we switch to automated test discovery in a follow-up, we will no longer have the ability to combine test cases. Maybe we should keep them separated so that the follow-up will not change the number of tests again?

cmake: Split test cases to improve parallelism #1760