fuzz: Fix stability, determinism issues #29018

1. 
   maflcko commented at 8:42 am on December 7, 2023: member

   It would be good to track fuzz “stability” and determinism, and fix any issues.

   Is there an easy way to generate a table for this metric for each fuzz target, maybe as a side effect of CI, or in another way?

   edit: oss-fuzz link https://oss-fuzz.com/fuzzer-stats?group_by=by-fuzzer&fuzzer=afl&job=afl_asan_bitcoin-core&project=bitcoin-core

2. maflcko added the label Brainstorming on Dec 7, 2023
3. maflcko added the label Tests on Dec 7, 2023
4. 
   dergoegge commented at 10:40 am on December 7, 2023: member

   oss-fuzz has a table for the afl++ jobs: https://oss-fuzz.com/fuzzer-stats?group_by=by-fuzzer&date_start=2023-08-31&date_end=2023-12-06&fuzzer=afl&job=afl_asan_bitcoin-core&project=bitcoin-core
5. 
   mzumsande commented at 4:20 pm on December 7, 2023: contributor

   How is the “stability” metric calculated?
6. 
   dergoegge commented at 4:22 pm on December 7, 2023: member

   How is the “stability” metric calculated?

   “Stability is measured by how many percent of the edges in the target are “stable”. Sending the same input again and again should take the exact same path through the target every time. If that is the case, the stability is 100%.” - https://github.com/AFLplusplus/AFLplusplus/blob/stable/docs/FAQ.md

7. 
   maflcko commented at 4:45 pm on December 7, 2023: member

   oss-fuzz has a table for the afl++ jobs

   Nice. Though, I wonder if there is something public available. Similar to the coverage report (https://github.com/bitcoin/bitcoin/blame/fcdb39d3ee17015776c0759e4742334a962219db/doc/fuzzing.md#L350) or the inputs zip (https://github.com/bitcoin-core/qa-assets/blob/38d7a06e9544bada01d558e6f85129334c228076/download_oss_fuzz_inputs.py#L35)

8. 
   dergoegge commented at 4:54 pm on December 7, 2023: member

   Nice. Though, I wonder if there is something public available.

   I’m not aware of something like that hosted by oss-fuzz, all the per fuzzer stats seem to require auth.

   I’ve been primarily fuzzing with afl++ lately, I can look at hosting some stats from that.

9. 
   maflcko commented at 5:01 pm on December 7, 2023: member

   Yeah, or alternatively steps to reproduce the stability output with afl locally, so that non-afl-gifted people can have some fun, too.
10. fanquake referenced this in commit 40bc501bf4 on Dec 11, 2023
11. fanquake referenced this in commit eef19c4ce2 on Dec 18, 2023
12. fanquake referenced this in commit 4ae5171d42 on Jan 11, 2024
13. 
    maflcko commented at 9:55 am on June 19, 2024: member

    Looks like there is currently no data for the last week. https://oss-fuzz.com/fuzzer-stats?group_by=by-fuzzer&date_start=2024-06-12&date_end=2024-06-18&fuzzer=afl&job=afl_asan_bitcoin-core&project=bitcoin-core returns a 404
14. fanquake referenced this in commit 4c573e5718 on Jul 1, 2024
15. 
    maflcko commented at 11:36 am on July 1, 2024: member

    Reported the oss-fuzz bug on https://www.github.com/google/oss-fuzz/issues/12142
16. 
    maflcko commented at 12:43 pm on July 1, 2024: member

    The latest build log from https://oss-fuzz-build-logs.storage.googleapis.com/index.html#bitcoin-core says:

    0Step [#4](/bitcoin-bitcoin/4/) - "build-check-afl-address-x86_64": Retrying failed fuzz targets sequentially 4
    1Step [#4](/bitcoin-bitcoin/4/) - "build-check-afl-address-x86_64": INFO: performing bad build checks for /tmp/not-out/tmpjtp3xr64/process_message
    2Step [#4](/bitcoin-bitcoin/4/) - "build-check-afl-address-x86_64": INFO: performing bad build checks for /tmp/not-out/tmpjtp3xr64/process_messages
    3Step [#4](/bitcoin-bitcoin/4/) - "build-check-afl-address-x86_64": INFO: performing bad build checks for /tmp/not-out/tmpjtp3xr64/tx_pool
    4Step [#4](/bitcoin-bitcoin/4/) - "build-check-afl-address-x86_64": INFO: performing bad build checks for /tmp/not-out/tmpjtp3xr64/tx_pool_standard
    5Step [#4](/bitcoin-bitcoin/4/) - "build-check-afl-address-x86_64": Broken fuzz targets 4
    

    However, I don’t think this is the cause. Pretty sure this has happened for years, but I haven’t checked.

17. PastaPastaPasta referenced this in commit 25466709a5 on Oct 24, 2024
18. PastaPastaPasta referenced this in commit c887b54074 on Oct 24, 2024
19. PastaPastaPasta referenced this in commit 2e41562d81 on Oct 24, 2024
20. 
    maflcko commented at 10:51 am on December 12, 2024: member

    Current screenshot, for reference

    

Contributors
maflcko dergoegge mzumsande

Labels
Brainstorming Tests