fuzz_seed_corpus: sub_net_deserialize and address_deserialize don't have any fuzz tests #76

While rebasing and testing the PR #21496, I found out that the seed directories sub_net_deserialize and address_deserialize in fuzz_seed_corpus don't have any fuzz tests in src/test/fuzz and therefore, an assertion failed in fuzz.cpp.

Error message:

sub_net_deserialize
fuzz: test/fuzz/fuzz.cpp:70: auto initialize()::(anonymous class)::operator()() const: Assertion "it != FuzzTargets().end()" && check' failed.
Aborted (core dumped)
address_deserialize
fuzz: test/fuzz/fuzz.cpp:70: auto initialize()::(anonymous class)::operator()() const: Assertion "it != FuzzTargets().end()" && check' failed.
Aborted (core dumped)

As a solution to this problem, I was thinking to remove these directories from fuzz_seed_corpus. @MarcoFalke, is it possible?

I think you shouldn't be calling the fuzz target if it doesn't exist

After merging this PR, we would be using the following script to call all the fuzz targets at once.

for D in qa-assets/fuzz_seed_corpus/*; do [ -d "${D}" ] && echo "${D##*/}" && FUZZ="${D##*/}" src/test/fuzz/fuzz qa-assets/fuzz_seed_corpus/"${D##*/}"; done

So to avoid calling invalid fuzz targets, the only way I can think of is running the above for loop in the src/tests/fuzz directory, which would create even more issues (as it contains many header files) to handle manually.

Besides, in my understanding, we should be calling fuzz targets using seed directory names and not from the fuzz tests directory because after modifying the existing fuzz tests, we would like to verify that all the existing seeds are still not throwing any error.

I am still figuring out the codebase, so please let me know if I am missing something here. I would love to hear more suggestions on it.

Why does it not work to do it in the same way that the test/fuzz/test_runner does it?

Why does it not work to do it in the same way that the test/fuzz/test_runner does it?

Err, the Assert(!should_abort) in fuzz.cpp seems to mean that PRINT_ALL_FUZZ_TARGETS_AND_ABORT=1 test/fuzz/fuzz will output "Aborted" along with the test names on stdout. Is there any reason that isn't just an exit(0); instead of an assertion failure?

That aside,

$ FUZZCORPUS=../../qa-assets/fuzz_seed_corpus
$ PRINT_ALL_FUZZ_TARGETS_AND_ABORT=1 src/test/fuzz/fuzz 2>/dev/null | grep -v '^Aborted$' | (export FUZZ; while read FUZZ; do c=$FUZZCORPUS/$FUZZ; [ -d "$c" ] && echo -n "$FUZZ: " && src/test/fuzz/fuzz "$c"; done)

seems to do the right thing. Creating a Makefile.fuzzdeps based on PRINT_ALL_FUZZ_TARGETS would let you automatically parallelise going through the different fuzz tests as well.

(ugh, sorry about the "i can't write github comments" spam)

Doing a Makefile.fuzzdeps containing:

RUN_FUZZERS: FUZZER/addition_overflow FUZZER/addr_info_deserialize ...
.PHONY: RUN_FUZZERS FUZZER/%

FUZZER/%: [@if](/bitcoin-core-qa-assets/contributor/if/) [ ! -d "$(FUZZCORPUS)" ]; then echo must specify FUZZCORPUS make variable; exit 1; fi [@FUZZ](/bitcoin-core-qa-assets/contributor/fuzz/)=$(@F) src/test/fuzz/fuzz $(FUZZCORPUS)/$(@F)

and running make -j12 -f Makefile.fuzzdeps RUN_FUZZERS FUZZCORPUS=../../qa-assets/fuzz_seed_corpus/ seems to work great. The RUN_FUZZERS: line can be generated as:

$ PRINT_ALL_FUZZ_TARGETS_AND_ABORT=1 src/test/fuzz/fuzz 2>/dev/null | grep -v '^Aborted$' |
     sed 's/^/FUZZER_/' | tr '\n' ' ' | sed 's/^/RUN_FUZZERS: /;s/ *$//' > Makefile.fuzzdeps

The process_messages and some of the addrman fuzzers seem to take forever to get through all the corpus data though, and the rpc fuzzer is failing unconditionally due to enumeratesigners for me.

Is there any reason that isn't just an exit(0); instead of an assertion failure?

No

[...] seems to work great

ok, closing then.

Let us know if there are any other issues.