API tests #187

1. 
   gmaxwell commented at 5:20 am on January 11, 2015: contributor

   We’re under-covered currently for tests that simulate crazy callers. The practise of asserting on statically wrong use limits the space of things that can / need to be tested here, but there is still some freedom.

   This should be less “test” and more “mechanical verification that the interface we present to callers isn’t unintentionally changed”.

   (As an example, libopus’ test_opus_api, calls into the library about 116 million times. I’d say that libopus has an API surface area of about 50x larger that libsecp256k1)

2. gmaxwell added the label assurance on Jan 11, 2015
3. gmaxwell assigned gmaxwell on Jan 11, 2015
4. gmaxwell added this to the milestone initial release on Aug 31, 2015
5. 
   gmaxwell commented at 3:02 pm on July 26, 2020: contributor

   API tests should explicitly perform tests with variously unaligned input/output buffers.

   (not likely to currently be a source of bugs except through compiler screwup, but if anyone starts going around adding SIMD code it would rapidly become a potential source of extremely hard to spot bugs).

6. 
   real-or-random commented at 7:21 pm on December 2, 2020: contributor

   From all the existing work, what comes probably closest is #744 .

   This should be less “test” and more “mechanical verification that the interface we present to callers isn’t unintentionally changed”. @gmaxwell Does fuzzing match this idea, or did you have something else in mind?

7. 
   gmaxwell commented at 1:47 am on December 5, 2020: contributor

   I would describe that test as the opposite of an api test: All it tests is the payloads, the usage of the API is hardcoded into the harness.

   An API test does stuff like calls with a null context or a number of pubkeys = 0, it acts like callers that fail to check errors or which call functions out of order… the goal is to validate the contract implied by the header documentation and maybe a little beyond it.

8. gmaxwell unassigned gmaxwell on Dec 5, 2020
9. 
   rickmark commented at 9:23 am on March 24, 2021: none

   It seems to me one of the first steps here is to separate the library code from the test code better. (Compile the library, then compile tests that link to that library from code in a separate directory)

   Broadly I see:

   • Conformance tests - Like OpenSSL to ensure that the cyrpto is correct, these are very good today
   • Performance tests - ensure that the operations are performant (handed by bench*.c and friends), these exist
   • Client tests - Build and sanity check that a consumer can accomplish it’s task, needs work
   • (NEW) API tests - Do all the things that a client shouldn’t do and make sure its gracefully handled. Because of the nature of this it seems like the wise idea might be to use a language like python/ruby and FFI the API surface because it’s easier to create a large matrix of tests this way. It also tests that FFI works correctly (as much as I hate FFI 😆)

Contributors
gmaxwell real-or-random rickmark

Labels
assurance

Milestone
stable release (1.0.0-rc.1)