tests: allow user to select tests via command line args #1211

1. 
   jonasnick commented at 4:45 pm on February 9, 2023: contributor

   This PR introduces the ability for users to select specific tests to run via command line arguments. The aim is to eliminate the need to comment out parts of the tests.c file in order to speed up testing. The implementation uses similar command line flags as the benchmarks, for example:

   0./tests ecmult schnorrsig
   

   The approach taken in this PR may not be the most straightforward, but it ensures backwards compatibility. The shell script located at https://gist.github.com/jonasnick/5e37248e3fa5911cd41e1da2f5f4e395 can be used to test the changes introduced in this PR. I’d be happy to add more flags if needed.

2. 
   real-or-random commented at 7:28 pm on February 9, 2023: contributor

   Concept ACK

   I still think it’s an interesting idea for the future to look into existing test frameworks, which typically support running a subset of tests. (But that’s no reason not to merge this, of course.)

3. tests: add "help" command line flag and output 81f791ee77
4. tests: error when both env var and cmd line arg try to set iteration

   `./tests a` results in the COUNT being set to 64 instead of erroring out as
   before. This is fixed in the next commit.

   38dee2ed66
5. tests: allow user to select tests via command line args 3e404f2b86
6. in src/tests.c:33 in c4ffc97285 outdated

   28@@ -28,7 +29,8 @@
   29 
   30 #define CONDITIONAL_TEST(cnt, nam) if (COUNT < (cnt)) { printf("Skipping %s (iteration count too low)\n", nam); } else
   31 
   32-static int COUNT = 64;
   33+static const int DEFAULT_COUNT = 64;
   34+static int COUNT = DEFAULT_COUNT;
   

   ---

   ---

   

   real-or-random commented at 7:37 pm on February 9, 2023:

   MSVC doesn’t like that line apparently because DEFAULT_COUNT is not a constant, which may be true by a strict reading of the standard…

   Maybe try #defining DEFAULT_COUNT instead, or set COUNT = DEFAULT_COUNT; only at the beginning of main().

   (By the way, this shows that CI on MSVC will timeout after 60 min in case of failure instead of just aborting. This will be fixed by https://github.com/bitcoin-core/secp256k1/pull/1209/commits/e4330341bd648e93b60fe70c631e311a98bce549)

7. jonasnick force-pushed on Feb 10, 2023
8. 
   sipa commented at 7:53 pm on February 10, 2023: contributor

   Concept ACK
9. 
   hebasto commented at 8:56 am on May 2, 2023: member

   In the commit 81f791ee771a9b3ba54045efe7247e39869ce1e9 “tests: add “help” command line flag and output”, shouldn’t #include "cli_util.h" be added to all src/modules/*/bench_impl.h where the have_flag function is actually used?
10. real-or-random added the label assurance on Jul 9, 2024

Contributors
jonasnick real-or-random sipa hebasto

Labels
assurance