tests: Add test_fuzzing_harnesses.sh for quick testing/verification of fuzzing harness coverage #17000

Add test_fuzzing_harnesses.sh for easy verification that our fuzz harnesses work the way we expect them to work. One thing tested is that we are able to reach different unique code paths when varying input.

Each fuzz target is given one second of running time after which coverage is evaluated.

Intentionally running without a starting corpus to make sure the fuzzers can evolve from thin air :)

The total running time of test_fuzzing_harnesses.sh is less than a minute.

Example output:

$ CC=clang CXX=clang++ ./configure --enable-fuzz --with-sanitizers=address,fuzzer,undefined
$ make
$ contrib/devtools/test_fuzzing_harnesses.sh
Found 21 fuzz harnesses to test.

Testing fuzzer address_deserialize during 1 second(s)
A subset of reached functions:
	NEW_FUNC[1/6]: 0x5592a0fa3430 in CDataStream& CDataStream::operator>><CAddress&>(CAddress&) src/./streams.h:460
	NEW_FUNC[2/6]: 0x5592a0fa4950 in void CAddress::SerializationOp<CDataStream, CSerActionUnserialize>(CDataStream&, CSerActionUnserialize) src/./protocol.h:337
	NEW_FUNC[3/6]: 0x5592a1ed2360 in CService::CService() src/netaddress.cpp:570
	NEW_FUNC[4/6]: 0x5592a1f03670 in CAddress::CAddress() src/protocol.cpp:145
	NEW_FUNC[5/6]: 0x5592a1f03780 in CAddress::Init() src/protocol.cpp:156
	NEW_FUNC[0/3]: 0x5592a0fa5440 in void SerReadWriteMany<CDataStream, CService&>(CDataStream&, CSerActionUnserialize, CService&) src/./serialize.h:989
	NEW_FUNC[1/3]: 0x5592a0fa5670 in void CService::SerializationOp<CDataStream, CSerActionUnserialize>(CDataStream&, CSerActionUnserialize) src/./netaddress.h:167
	NEW_FUNC[2/3]: 0x5592a0fa5ac0 in void BigEndian<unsigned short>::Unserialize<CDataStream>(CDataStream&) src/./serialize.h:474
stat::number_of_executed_units: 10343
stat::average_exec_per_sec:     5171
stat::new_units_added:          41
stat::slowest_unit_time_sec:    0
stat::peak_rss_mb:              147
Number of unique code paths reached during fuzzing round: 13

Testing fuzzer addrman_deserialize during 1 second(s)
A subset of reached functions:
	NEW_FUNC[0/61]: 0x55c567eabaa0 in UniqueLock<AnnotatedMixin<std::recursive_mutex>, std::unique_lock<std::recursive_mutex> >::UniqueLock(AnnotatedMixin<std::recursive_mutex>&, char const*, char const*, int, bool) src/./sync.h:146
	NEW_FUNC[1/61]: 0x55c567eabf00 in UniqueLock<AnnotatedMixin<std::recursive_mutex>, std::unique_lock<std::recursive_mutex> >::~UniqueLock() src/./sync.h:165
	NEW_FUNC[2/61]: 0x55c567eae5b0 in FastRandomContext::FillByteBuffer() src/./random.h:114
	NEW_FUNC[3/61]: 0x55c567eca850 in uint256::uint256() src/./uint256.h:123
	NEW_FUNC[4/61]: 0x55c567ed4f50 in UniqueLock<AnnotatedMixin<std::recursive_mutex>, std::unique_lock<std::recursive_mutex> >::Enter(char const*, char const*, int) src/./sync.h:123
	NEW_FUNC[12/61]: 0x55c567f07320 in CDataStream& CDataStream::operator>><unsigned char&>(unsigned char&) src/./streams.h:460
	NEW_FUNC[13/61]: 0x55c567f1d750 in CAddrMan::CAddrMan() src/./addrman.h:481
	NEW_FUNC[14/61]: 0x55c567f1dec0 in CDataStream& CDataStream::operator>><CAddrMan&>(CAddrMan&) src/./streams.h:460
	NEW_FUNC[15/61]: 0x55c567f1e050 in CAddrMan::~CAddrMan() src/./addrman.h:486
	NEW_FUNC[16/61]: 0x55c567f1e250 in CAddrMan::Clear() src/./addrman.h:457
stat::number_of_executed_units: 394
stat::average_exec_per_sec:     197
stat::new_units_added:          14
stat::slowest_unit_time_sec:    0
stat::peak_rss_mb:              111
Number of unique code paths reached during fuzzing round: 12

…

All fuzz harnesses seem to work as expected.

Commits:

• Add contrib/devtools/test_fuzzing_harnesses.sh
• Run test_fuzzing_harnesses.sh as part of RUN_FUZZ_TESTS in Travis
• Enable UBSan for Travis fuzzer job

Add test_fuzzing_harnesses.sh for easy verification that our fuzz harnesses work the way we expect them to. More specifically that we are able to reach different unique code paths when varying input.

Has this ever been an issue? Instead of adding a script that seems to give little reward, I'd prefer a script that exports the coverage to html.

This can be done with clang out of the box:

@MarcoFalke Oh, I use this script all the time and thought others might find it useful :) Did you read the output? I find this script gives me much more valuable information than our current fuzz runner, but I could be biased :)

Personally I prefer getting coverage information in the terminal so I'll let someone else run with the export HTML idea :)

llvm-cov can also export to plain text instead of html

@MarcoFalke Have you tried this script? if not I suggest taking it for a spin before dismissing it :) The runtime is really quick compared to llvm-cov :)

No, I have not tried the script. I am using ./test/fuzz/test_runner.py

ACK 19e3175

Output of test_fuzzing_harnesses.sh with this PR

Output of test_fuzzing_harnesses.sh with PR #17051

Personally I prefer getting coverage information in the terminal

Same. Terminal output seems complementary to HTML for online dashboards.