CI job for verifying coverage increase #166

issue dergoegge openend this issue on January 23, 2024
  1. dergoegge commented at 5:52 pm on January 23, 2024: member

    We could consider adding a CI job that checks that the coverage for newly added inputs actually goes up (this should be possible with afl-showmap). This would help with review and avoiding bloat in the corpora.

    (Fuzz harnesses with low stability could be annoying here)

  2. maflcko commented at 6:00 pm on January 23, 2024: contributor

    DrahtBot can (manually triggered) create html coverage reports. See https://github.com/maflcko/DrahtBot/blob/main/coverage_fuzz/src/main.rs#L151 and https://drahtbot.space/host_reports/DrahtBot/reports/coverage_fuzz/monotree/004367dba8a3e852/428a2e7b0def5102/fuzz.coverage/index.html

    libFuzzer could also be used by using the ft metric, printed from the summary. Example:

    0[#1206](/bitcoin-core-qa-assets/1206/)	DONE   cov: 6161 ft: 36449 corp: 941/15703Kb lim: 446621 exec/s: 0 rss: 658Mb
    
  3. maflcko commented at 4:17 pm on January 26, 2024: contributor
  4. dergoegge commented at 4:32 pm on January 26, 2024: member
    I presume the plan is to use that for a new CI job here? i.e. run once for both qa-assets from main and the pr, then compare coverage summary for relevant harnesses and if the coverage does not go up fail the job
  5. maflcko commented at 4:57 pm on January 26, 2024: contributor
    I think for now the comparison can be done manually, but a CI task to run diff on the previous CI tasks’ output can be added.
  6. maflcko commented at 8:26 am on April 16, 2024: contributor
    Another idea to limit the added files would be to re-create the merge step in the CI and abort if the difference in added files is larger than 30% or some magic value?
  7. murchandamus commented at 4:00 pm on April 16, 2024: contributor

    Maybe I’m just going about it wrong, but it’s kinda annoying to do it manually. So far I’ve been doing:

    1. Reset to upstream/main
    2. Run test_runner, check back a couple times at least thirty minutes later to grab the results
    3. (Add and) fetch submitters remote repository, check out the PR branch
    4. Run test_runner, check back a couple times at least thirty minutes later to grab the results
    5. Run a few nifty search and replaces in vim to combine the two outputs such that each line is shown separately rather than all changes in block (I tried figuring out how to output the diff line-by-line with diff and git diff but didn’t find a satisfactory solution)

    Am I overlooking an automated tool or smth?

  8. maflcko commented at 4:33 pm on April 16, 2024: contributor

    If you don’t want to spend CPU locally, and you trust the CI, you can fetch the results from:

    The last step will still have to be done manually as well.

  9. murchandamus commented at 5:14 pm on April 16, 2024: contributor

    I’ve been able to streamline the last step:

    0paste -d '\n' <(sed 's/^/-/' main-results-file) <(sed 's/^/+/' pr-results-file)
    

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin-core/qa-assets. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-12-06 22:25 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me