CI job for verifying coverage increase #166

1. 
   dergoegge commented at 5:52 pm on January 23, 2024: member

   We could consider adding a CI job that checks that the coverage for newly added inputs actually goes up (this should be possible with afl-showmap). This would help with review and avoiding bloat in the corpora.

   (Fuzz harnesses with low stability could be annoying here)

2. 
   maflcko commented at 6:00 pm on January 23, 2024: contributor

   DrahtBot can (manually triggered) create html coverage reports. See https://github.com/maflcko/DrahtBot/blob/main/coverage_fuzz/src/main.rs#L151 and https://drahtbot.space/host_reports/DrahtBot/reports/coverage_fuzz/monotree/004367dba8a3e852/428a2e7b0def5102/fuzz.coverage/index.html

   libFuzzer could also be used by using the ft metric, printed from the summary. Example:

   0[#1206](/bitcoin-core-qa-assets/1206/)	DONE   cov: 6161 ft: 36449 corp: 941/15703Kb lim: 446621 exec/s: 0 rss: 658Mb
   

3. 
   maflcko commented at 4:17 pm on January 26, 2024: contributor

   Fixed in https://github.com/bitcoin/bitcoin/pull/29329
4. 
   dergoegge commented at 4:32 pm on January 26, 2024: member

   I presume the plan is to use that for a new CI job here? i.e. run once for both qa-assets from main and the pr, then compare coverage summary for relevant harnesses and if the coverage does not go up fail the job
5. 
   maflcko commented at 4:57 pm on January 26, 2024: contributor

   I think for now the comparison can be done manually, but a CI task to run diff on the previous CI tasks’ output can be added.
6. 
   maflcko commented at 8:26 am on April 16, 2024: contributor

   Another idea to limit the added files would be to re-create the merge step in the CI and abort if the difference in added files is larger than 30% or some magic value?
7. 
   murchandamus commented at 4:00 pm on April 16, 2024: contributor

   Maybe I’m just going about it wrong, but it’s kinda annoying to do it manually. So far I’ve been doing:

   1. Reset to upstream/main
   2. Run test_runner, check back a couple times at least thirty minutes later to grab the results
   3. (Add and) fetch submitters remote repository, check out the PR branch
   4. Run test_runner, check back a couple times at least thirty minutes later to grab the results
   5. Run a few nifty search and replaces in vim to combine the two outputs such that each line is shown separately rather than all changes in block (I tried figuring out how to output the diff line-by-line with diff and git diff but didn’t find a satisfactory solution)

   Am I overlooking an automated tool or smth?

8. 
   maflcko commented at 4:33 pm on April 16, 2024: contributor

   If you don’t want to spend CPU locally, and you trust the CI, you can fetch the results from:

   • https://cirrus-ci.com/github/bitcoin/bitcoin/master , for example https://cirrus-ci.com/task/5312073368862720?logs=ci#L4129
   • The pull request here, for example https://cirrus-ci.com/task/5447597169573888?logs=ci#L4030

   The last step will still have to be done manually as well.

9. 
   murchandamus commented at 5:14 pm on April 16, 2024: contributor

   I’ve been able to streamline the last step:

   0paste -d '\n' <(sed 's/^/-/' main-results-file) <(sed 's/^/+/' pr-results-file)
   

Contributors
dergoegge maflcko murchandamus