Initial gcov work #219

1. 
   cbatson commented at 4:39 pm on February 17, 2015: none

   Separation of deterministic tests from randomized tests; initial work for gcov build & run. Next step is filling out deterministic tests to trigger branches. All feedback welcome.
2. introduce state object for random number generator, to make randomized fuzz tests explicit and prevent accidental randomization of deterministic tests bded2cb82c
3. initial gcov build f69fa3b691
4. ignore some test & gcov-related files 4c68eb0fb3
5. don't use pkgconfig on osx so that openssl can be located 990d61a7c3
6. segregate deterministic tests from randomized ones; random number generator state for bignum & openssl tests 9076e500d7
7. parameterize count to further differentiate fuzz testing 549211abce
8. only deterministic tests for gcov run 22e588aecb
9. for now, compile in only deterministic tests for gcov build 2b13cd0daa
10. 
    gmaxwell commented at 9:36 pm on February 17, 2015: contributor

    Just a quick question I had: Whats the motivation of not running the randomized tests in coverage testing? They’re just as deterministic as the rest when run with a static seed.
11. 
    cbatson commented at 1:49 am on February 18, 2015: none

    I think that could get challenging to maintain in practice. In order to guarantee identical test behavior across changes, it’d be necessary to not only never change the RNG algorithm, but also not change the sequence of calls to the RNG (as could happen with adding, removing, or changing the order of tests, for example). Or, statically seed before every test and make sure the intra-test RNG call pattern doesn’t change.

    Also, for me personally, I find a fixed sequence of instructions used to evoke a specific condition in the code under test is easier to reason confidently about.

    There’s philosophical motivations as well, upon which I’m happy to expound if you like. :-)

12. 
    cbatson commented at 1:39 am on February 19, 2015: none

    P.S. I’m happy to take a different approach if you prefer.
13. 
    sipa commented at 11:44 am on February 23, 2015: contributor

    Hi,

    sorry for not commenting on this earlier, but there are several changes in the pipeline which will likely conflict with this, in particular #217, #215, and #208. I’d like to see those go in first.

    I’m definitely interested in better automatic coverage tests afterwards, though.

14. 
    cbatson commented at 4:21 pm on February 23, 2015: none

    No worries, let me know when you’re ready.
15. 
    real-or-random commented at 9:16 am on March 24, 2022: contributor

    Closing as obsolete, we have coverage support for a long time now (https://github.com/bitcoin-core/secp256k1/pull/426). There’s still #954 open though.
16. real-or-random closed this on Mar 24, 2022

    ---

Contributors
cbatson gmaxwell sipa real-or-random