Also keep fuzz inputs that increase coverage on older branches #266

This is going to create a commit per sanitizer and ref. I think it should create one commit per sanitizer for all refs (without wasting time rebuilding the same ref multiple times), or at least mention the ref in the commit message.

not sure how resource-constrained the vms are on which this will run

I think it is fine to just ask the user to pass in --extra-ref=29.x --extra-ref=...x and then fetch those with --depth=1, but no strong opinion, auto-detection may work as well.

Oh, thanks for the idea! I'll consider it.

Wouldn’t that mean we’d cover the four most recent versions? The current plus three previous?

Oh, yes, current plus three previous. Classic off-by-one. I didn't verify the output with the mentioned versions in #265 (master, 30.x, 29.x).

For reference, this is the output on current master (2fe76ed8324):

$ CURRENT_MAJOR_VERSION=$(git tag --list 'v*' --sort=-v:refname | sed 's/^v//' | awk -F. '{ print $1 }' | head -1)
$ echo $CURRENT_MAJOR_VERSION
31
$ PREV_MAJOR_VERSIONS=$(seq $((CURRENT_MAJOR_VERSION - 3)) $((CURRENT_MAJOR_VERSION - 1)))
$ echo $PREV_MAJOR_VERSIONS
28 29 30

So it should be this:

diff --git a/delete_nonreduced_fuzz_inputs.sh b/delete_nonreduced_fuzz_inputs.sh
index 49968fe7da..61426864f9 100644
--- a/delete_nonreduced_fuzz_inputs.sh
+++ b/delete_nonreduced_fuzz_inputs.sh
@@ -50,7 +50,7 @@ git clone --depth=1 --no-single-branch https://github.com/bitcoin/bitcoin.git
   # last three major versions.
   REFS=("master")
   CURRENT_MAJOR_VERSION=$(git tag --list 'v*' --sort=-v:refname | sed 's/^v//' | awk -F. '{ print $1 }' | head -1)
-  PREV_MAJOR_VERSIONS=$(seq $((CURRENT_MAJOR_VERSION - 3)) $((CURRENT_MAJOR_VERSION - 1)))
+  PREV_MAJOR_VERSIONS=$(seq $((CURRENT_MAJOR_VERSION - 2)) $((CURRENT_MAJOR_VERSION - 1)))
   for version in $PREV_MAJOR_VERSIONS; do
     # versions before 29.x didn't use cmake
     if [ "$version" -lt 29 ]; then

Then I can also remove the check for >= 29.x because of the new cmake build system below.

I have the impression that this makes separate corpora in different directories for the fuzzing with each version. Am I understanding that right? If so, I’m not sure I understand why you chose to do it this way. When you fuzz, each input that exercises new code paths will be written to the target directory. So if you instead fuzz with all versions into the same directory, it should still retain any inputs that lead to new coverage, but it might produce a smaller input set, because the inputs already present in the target repository will be processed before inputs of the same length from other sources.

Hey, thanks for taking a look at this!

I have the impression that this makes separate corpora in different directories for the fuzzing with each version

The intention is to have a single corpora for all versions. Maybe you got confused by -o $FUZZ_TARGET_DIR/$ref_sha1? I'm using a version-specific output dir because afl-cmin requires an empty output directory. FUZZ_TARGET_DIR itself is version-independent. I then run cp afterwards to move everything up into FUZZ_TARGET_DIR, which is also where the previous code put it.

(I just realized that I don't need to fuzz into a version-specific output directory. It just needs to be empty. Maybe reusing the output dir but clearing it before every run would make its purpose more clear.)

the inputs already present in the target repository will be processed before inputs of the same length from other sources.

Oh, that's good to know! But since afl-cmin needs an empty output directory, I think you mean libFuzzer here, which is what fuzz/test_runner.py uses.

Ah okay, I was unaware that afl-cmin behaves differently there. I’m indeed more familiar with libfuzzer which I use for my fuzzing cronjobs. Carry on then!