tests: allow user to select tests via command line args

jonasnick commented at 4:45 pm on February 9, 2023: contributor

This PR introduces the ability for users to select specific tests to run via command line arguments. The aim is to eliminate the need to comment out parts of the tests.c file in order to speed up testing. The implementation uses similar command line flags as the benchmarks, for example:

0./tests ecmult schnorrsig

The approach taken in this PR may not be the most straightforward, but it ensures backwards compatibility. The shell script located at https://gist.github.com/jonasnick/5e37248e3fa5911cd41e1da2f5f4e395 can be used to test the changes introduced in this PR. I’d be happy to add more flags if needed.

real-or-random commented at 7:28 pm on February 9, 2023: contributor

Concept ACK

I still think it’s an interesting idea for the future to look into existing test frameworks, which typically support running a subset of tests. (But that’s no reason not to merge this, of course.)

tests: add "help" command line flag and output 81f791ee77

tests: error when both env var and cmd line arg try to set iteration

`./tests a` results in the COUNT being set to 64 instead of erroring out as
before. This is fixed in the next commit.

38dee2ed66

tests: allow user to select tests via command line args 3e404f2b86

in src/tests.c:33 in c4ffc97285 outdated

28@@ -28,7 +29,8 @@
29 
30 #define CONDITIONAL_TEST(cnt, nam) if (COUNT < (cnt)) { printf("Skipping %s (iteration count too low)\n", nam); } else
31 
32-static int COUNT = 64;
33+static const int DEFAULT_COUNT = 64;
34+static int COUNT = DEFAULT_COUNT;

real-or-random commented at 7:37 pm on February 9, 2023:

MSVC doesn’t like that line apparently because DEFAULT_COUNT is not a constant, which may be true by a strict reading of the standard…

Maybe try #defining DEFAULT_COUNT instead, or set COUNT = DEFAULT_COUNT; only at the beginning of main().

(By the way, this shows that CI on MSVC will timeout after 60 min in case of failure instead of just aborting. This will be fixed by https://github.com/bitcoin-core/secp256k1/pull/1209/commits/e4330341bd648e93b60fe70c631e311a98bce549)

jonasnick force-pushed on Feb 10, 2023

sipa commented at 7:53 pm on February 10, 2023: contributor

Concept ACK

hebasto commented at 8:56 am on May 2, 2023: member

In the commit 81f791ee771a9b3ba54045efe7247e39869ce1e9 “tests: add “help” command line flag and output”, shouldn’t #include "cli_util.h" be added to all src/modules/*/bench_impl.h where the have_flag function is actually used?

real-or-random added the label assurance on Jul 9, 2024

furszy commented at 7:18 pm on September 5, 2025: member

A bit late to comment here, but I ended up implementing this feature differently in #1734 (built on top of the new unit test framework introduced there).

Originally, I just wanted to parallelize test execution because the suite was taking too long on my end. Since I couldn’t find a good C89-compatible unit test framework without dynamic memory allocations, I built one from scratch; supporting parallelization from the start and adding an easy to expand named-args command-line interface. Then, once the framework was in place, adding new options was easy, so I also added support for running specific targets or groups of targets (along with a few other useful options). That’s how I ended up approaching this problem from another angle. I hope I didn’t step on your toes with that. The main idea behind the new framework is to make the testing infrastructure scalable and development smoother.

jonasnick commented at 3:32 pm on September 7, 2025: contributor

I hope I didn’t step on your toes with that.

Definitely not. The approaches in both PRs seem to be very different. The goal of this PR is to allow running specific tests with minimal changes to the testing code.

hebasto commented at 1:25 pm on September 10, 2025: member

While reviewing #1734, I decided to re-check this PR from a different angle.

This branch is essentially the PR branch rebased on the latest master branch @ 03fb60ad2e38ae9415e64bf5ba523a10100d4c3c, with an additional commit introducing greater parallelism when running ctest.

Below are typical runs on my laptop:

master branch:

 0$ ctest --test-dir build_master/ -j 16
 1Test project /home/hebasto/dev/secp256k1/secp256k1/build_master
 2    Start 2: secp256k1_tests
 3    Start 1: secp256k1_noverify_tests
 4    Start 3: secp256k1_exhaustive_tests
 51/3 Test [#3](/bitcoin-core-secp256k1/3/): secp256k1_exhaustive_tests .......   Passed    5.91 sec
 62/3 Test [#1](/bitcoin-core-secp256k1/1/): secp256k1_noverify_tests .........   Passed   14.57 sec
 73/3 Test [#2](/bitcoin-core-secp256k1/2/): secp256k1_tests ..................   Passed   32.50 sec
 8
 9100% tests passed, 0 tests failed out of 3
10
11Total Test time (real) =  32.51 sec

amended branch:

 0$ ctest --test-dir build_pr/ -j 16
 1Test project /home/hebasto/dev/secp256k1/secp256k1/build_pr
 2      Start 17: secp256k1_tests::ecmult
 3      Start 22: secp256k1_tests::ellswift
 4      Start  6: secp256k1_noverify_tests::ecmult
 5      Start 23: secp256k1_exhaustive_tests
 6      Start 11: secp256k1_noverify_tests::ellswift
 7      Start 16: secp256k1_tests::group
 8      Start  1: secp256k1_noverify_tests::integer
 9      Start 12: secp256k1_tests::integer
10      Start  4: secp256k1_noverify_tests::field
11      Start 15: secp256k1_tests::field
12      Start  5: secp256k1_noverify_tests::group
13      Start 21: secp256k1_tests::musig
14      Start 18: secp256k1_tests::ecdh
15      Start 10: secp256k1_noverify_tests::musig
16      Start 14: secp256k1_tests::scalar
17      Start 20: secp256k1_tests::schnorrsig
18 1/23 Test [#14](/bitcoin-core-secp256k1/14/): secp256k1_tests::scalar ................   Passed    0.08 sec
19      Start  7: secp256k1_noverify_tests::ecdh
20 2/23 Test [#20](/bitcoin-core-secp256k1/20/): secp256k1_tests::schnorrsig ............   Passed    0.11 sec
21      Start 19: secp256k1_tests::extrakeys
22 3/23 Test [#10](/bitcoin-core-secp256k1/10/): secp256k1_noverify_tests::musig ........   Passed    0.12 sec
23      Start 13: secp256k1_tests::hash
24 4/23 Test  [#7](/bitcoin-core-secp256k1/7/): secp256k1_noverify_tests::ecdh .........   Passed    0.07 sec
25      Start  9: secp256k1_noverify_tests::schnorrsig
26 5/23 Test [#19](/bitcoin-core-secp256k1/19/): secp256k1_tests::extrakeys .............   Passed    0.07 sec
27      Start  3: secp256k1_noverify_tests::scalar
28 6/23 Test [#18](/bitcoin-core-secp256k1/18/): secp256k1_tests::ecdh ..................   Passed    0.18 sec
29      Start  2: secp256k1_noverify_tests::hash
30 7/23 Test [#13](/bitcoin-core-secp256k1/13/): secp256k1_tests::hash ..................   Passed    0.06 sec
31      Start  8: secp256k1_noverify_tests::extrakeys
32 8/23 Test  [#9](/bitcoin-core-secp256k1/9/): secp256k1_noverify_tests::schnorrsig ...   Passed    0.05 sec
33 9/23 Test  [#8](/bitcoin-core-secp256k1/8/): secp256k1_noverify_tests::extrakeys ....   Passed    0.03 sec
3410/23 Test  [#2](/bitcoin-core-secp256k1/2/): secp256k1_noverify_tests::hash .........   Passed    0.04 sec
3511/23 Test  [#3](/bitcoin-core-secp256k1/3/): secp256k1_noverify_tests::scalar .......   Passed    0.04 sec
3612/23 Test [#21](/bitcoin-core-secp256k1/21/): secp256k1_tests::musig .................   Passed    0.33 sec
3713/23 Test  [#5](/bitcoin-core-secp256k1/5/): secp256k1_noverify_tests::group ........   Passed    1.76 sec
3814/23 Test [#15](/bitcoin-core-secp256k1/15/): secp256k1_tests::field .................   Passed    2.14 sec
3915/23 Test  [#1](/bitcoin-core-secp256k1/1/): secp256k1_noverify_tests::integer ......   Passed    2.16 sec
4016/23 Test  [#4](/bitcoin-core-secp256k1/4/): secp256k1_noverify_tests::field ........   Passed    2.18 sec
4117/23 Test [#12](/bitcoin-core-secp256k1/12/): secp256k1_tests::integer ...............   Passed    2.41 sec
4218/23 Test [#16](/bitcoin-core-secp256k1/16/): secp256k1_tests::group .................   Passed    3.41 sec
4319/23 Test [#11](/bitcoin-core-secp256k1/11/): secp256k1_noverify_tests::ellswift .....   Passed    4.29 sec
4420/23 Test [#23](/bitcoin-core-secp256k1/23/): secp256k1_exhaustive_tests .............   Passed    6.86 sec
4521/23 Test  [#6](/bitcoin-core-secp256k1/6/): secp256k1_noverify_tests::ecmult .......   Passed    8.78 sec
4622/23 Test [#22](/bitcoin-core-secp256k1/22/): secp256k1_tests::ellswift ..............   Passed   13.06 sec
4723/23 Test [#17](/bitcoin-core-secp256k1/17/): secp256k1_tests::ecmult ................   Passed   17.17 sec
48
49100% tests passed, 0 tests failed out of 23
50
51Total Test time (real) =  17.17 sec

hebasto commented at 1:51 pm on September 10, 2025: member

Concept ACK, with the hope that build system changes will be picked up (perhaps in a follow-up).

in src/tests.c:7479 in 3e404f2b86

7474+
7475+static int module_unavailable(int argc, char **argv, char *module) {
7476+    if (have_flag(argc, argv, module)) {
7477+        fprintf(stderr, "./tests: %s module not enabled.\n", module);
7478+        fprintf(stderr, "Use ./configure --enable-module-%s.\n\n", module);
7479+        return 1;

hebasto commented at 2:31 pm on September 10, 2025:

0        fprintf(stderr, "Use ./configure --enable-module-%s.\n\n", module);
1        fprintf(stderr, "%s tests skipped.\n\n", module);
2        return 1;

This change allows CTest to skip tests for disabled modules gracefully, rather than merely ignoring them:

0...
127/27 Test [#19](/bitcoin-core-secp256k1/19/): secp256k1_tests::ecmult ................   Passed   19.21 sec
2
3100% tests passed, 0 tests failed out of 27
4
5Total Test time (real) =  19.24 sec
6
7The following tests did not run:
8	  9 - secp256k1_noverify_tests::recovery (Skipped)
9	 22 - secp256k1_tests::recovery (Skipped)

jonasnick commented at 12:20 pm on September 23, 2025: contributor

Closing because I prefer #1734:

parallelism and the resulting speedup
much more extensible, while only adding a reasonable number of locs
easier to add and change granularity of tests
this PR has a bug where the second command line argument is always interpreted as a test target and as the seed.

jonasnick closed this on Sep 23, 2025

tests: allow user to select tests via command line args #1211