fuzz: Fix stability, determinism issues

maflcko commented at 8:42 am on December 7, 2023: member

It would be good to track fuzz “stability” and determinism, and fix any issues.

Is there an easy way to generate a table for this metric for each fuzz target, maybe as a side effect of CI, or in another way?

edit: oss-fuzz link https://oss-fuzz.com/fuzzer-stats?group_by=by-fuzzer&fuzzer=afl&job=afl_asan_bitcoin-core&project=bitcoin-core

maflcko added the label Brainstorming on Dec 7, 2023

maflcko added the label Tests on Dec 7, 2023

dergoegge commented at 10:40 am on December 7, 2023: member

oss-fuzz has a table for the afl++ jobs: https://oss-fuzz.com/fuzzer-stats?group_by=by-fuzzer&date_start=2023-08-31&date_end=2023-12-06&fuzzer=afl&job=afl_asan_bitcoin-core&project=bitcoin-core Screenshot from 2023-12-07 10-39-17

mzumsande commented at 4:20 pm on December 7, 2023: contributor

How is the “stability” metric calculated?

dergoegge commented at 4:22 pm on December 7, 2023: member

How is the “stability” metric calculated?

“Stability is measured by how many percent of the edges in the target are “stable”. Sending the same input again and again should take the exact same path through the target every time. If that is the case, the stability is 100%.” - https://github.com/AFLplusplus/AFLplusplus/blob/stable/docs/FAQ.md

maflcko commented at 4:45 pm on December 7, 2023: member

oss-fuzz has a table for the afl++ jobs

Nice. Though, I wonder if there is something public available. Similar to the coverage report (https://github.com/bitcoin/bitcoin/blame/fcdb39d3ee17015776c0759e4742334a962219db/doc/fuzzing.md#L350) or the inputs zip (https://github.com/bitcoin-core/qa-assets/blob/38d7a06e9544bada01d558e6f85129334c228076/download_oss_fuzz_inputs.py#L35)

dergoegge commented at 4:54 pm on December 7, 2023: member

Nice. Though, I wonder if there is something public available.

I’m not aware of something like that hosted by oss-fuzz, all the per fuzzer stats seem to require auth.

I’ve been primarily fuzzing with afl++ lately, I can look at hosting some stats from that.

maflcko commented at 5:01 pm on December 7, 2023: member

Yeah, or alternatively steps to reproduce the stability output with afl locally, so that non-afl-gifted people can have some fun, too.

fanquake referenced this in commit 40bc501bf4 on Dec 11, 2023

fanquake referenced this in commit eef19c4ce2 on Dec 18, 2023

fanquake referenced this in commit 4ae5171d42 on Jan 11, 2024

maflcko commented at 9:55 am on June 19, 2024: member

Looks like there is currently no data for the last week. https://oss-fuzz.com/fuzzer-stats?group_by=by-fuzzer&date_start=2024-06-12&date_end=2024-06-18&fuzzer=afl&job=afl_asan_bitcoin-core&project=bitcoin-core returns a 404

fanquake referenced this in commit 4c573e5718 on Jul 1, 2024

maflcko commented at 11:36 am on July 1, 2024: member

Reported the oss-fuzz bug on https://www.github.com/google/oss-fuzz/issues/12142

maflcko commented at 12:43 pm on July 1, 2024: member

The latest build log from https://oss-fuzz-build-logs.storage.googleapis.com/index.html#bitcoin-core says:

0Step [#4](/bitcoin-bitcoin/4/) - "build-check-afl-address-x86_64": Retrying failed fuzz targets sequentially 4
1Step [#4](/bitcoin-bitcoin/4/) - "build-check-afl-address-x86_64": INFO: performing bad build checks for /tmp/not-out/tmpjtp3xr64/process_message
2Step [#4](/bitcoin-bitcoin/4/) - "build-check-afl-address-x86_64": INFO: performing bad build checks for /tmp/not-out/tmpjtp3xr64/process_messages
3Step [#4](/bitcoin-bitcoin/4/) - "build-check-afl-address-x86_64": INFO: performing bad build checks for /tmp/not-out/tmpjtp3xr64/tx_pool
4Step [#4](/bitcoin-bitcoin/4/) - "build-check-afl-address-x86_64": INFO: performing bad build checks for /tmp/not-out/tmpjtp3xr64/tx_pool_standard
5Step [#4](/bitcoin-bitcoin/4/) - "build-check-afl-address-x86_64": Broken fuzz targets 4

However, I don’t think this is the cause. Pretty sure this has happened for years, but I haven’t checked.

fuzz: Fix stability, determinism issues #29018