fuzz: Fix stability, determinism issues #29018

1. 
   maflcko commented at 8:42 am on December 7, 2023: member

   It would be good to track fuzz “stability” and determinism, and fix any issues.

   Is there an easy way to generate a table for this metric for each fuzz target, maybe as a side effect of CI, or in another way?

   edit: oss-fuzz link https://oss-fuzz.com/fuzzer-stats?group_by=by-fuzzer&fuzzer=afl&job=afl_asan_bitcoin-core&project=bitcoin-core

2. maflcko added the label Brainstorming on Dec 7, 2023
3. maflcko added the label Tests on Dec 7, 2023
4. 
   dergoegge commented at 10:40 am on December 7, 2023: member

   oss-fuzz has a table for the afl++ jobs: https://oss-fuzz.com/fuzzer-stats?group_by=by-fuzzer&date_start=2023-08-31&date_end=2023-12-06&fuzzer=afl&job=afl_asan_bitcoin-core&project=bitcoin-core
5. 
   mzumsande commented at 4:20 pm on December 7, 2023: contributor

   How is the “stability” metric calculated?
6. 
   dergoegge commented at 4:22 pm on December 7, 2023: member

   How is the “stability” metric calculated?

   “Stability is measured by how many percent of the edges in the target are “stable”. Sending the same input again and again should take the exact same path through the target every time. If that is the case, the stability is 100%.” - https://github.com/AFLplusplus/AFLplusplus/blob/stable/docs/FAQ.md

7. 
   maflcko commented at 4:45 pm on December 7, 2023: member

   oss-fuzz has a table for the afl++ jobs

   Nice. Though, I wonder if there is something public available. Similar to the coverage report (https://github.com/bitcoin/bitcoin/blame/fcdb39d3ee17015776c0759e4742334a962219db/doc/fuzzing.md#L350) or the inputs zip (https://github.com/bitcoin-core/qa-assets/blob/38d7a06e9544bada01d558e6f85129334c228076/download_oss_fuzz_inputs.py#L35)

8. 
   dergoegge commented at 4:54 pm on December 7, 2023: member

   Nice. Though, I wonder if there is something public available.

   I’m not aware of something like that hosted by oss-fuzz, all the per fuzzer stats seem to require auth.

   I’ve been primarily fuzzing with afl++ lately, I can look at hosting some stats from that.

9. 
   maflcko commented at 5:01 pm on December 7, 2023: member

   Yeah, or alternatively steps to reproduce the stability output with afl locally, so that non-afl-gifted people can have some fun, too.
10. fanquake referenced this in commit 40bc501bf4 on Dec 11, 2023
11. fanquake referenced this in commit eef19c4ce2 on Dec 18, 2023
12. fanquake referenced this in commit 4ae5171d42 on Jan 11, 2024
13. 
    maflcko commented at 9:55 am on June 19, 2024: member

    Looks like there is currently no data for the last week. https://oss-fuzz.com/fuzzer-stats?group_by=by-fuzzer&date_start=2024-06-12&date_end=2024-06-18&fuzzer=afl&job=afl_asan_bitcoin-core&project=bitcoin-core returns a 404

Contributors
maflcko dergoegge mzumsande

Labels
Brainstorming Tests