tests: Add test_fuzzing_harnesses.sh for quick testing/verification of fuzzing harness coverage #17000

pull practicalswift wants to merge 3 commits into bitcoin:master from practicalswift:test_fuzzing_harnesses changing 4 files +61 −2
  1. practicalswift commented at 6:22 pm on September 30, 2019: contributor

    Add test_fuzzing_harnesses.sh for easy verification that our fuzz harnesses work the way we expect them to work. One thing tested is that we are able to reach different unique code paths when varying input.

    Each fuzz target is given one second of running time after which coverage is evaluated.

    Intentionally running without a starting corpus to make sure the fuzzers can evolve from thin air :)

    The total running time of test_fuzzing_harnesses.sh is less than a minute.

    Example output:

     0$ CC=clang CXX=clang++ ./configure --enable-fuzz --with-sanitizers=address,fuzzer,undefined
 1$ make
 2$ contrib/devtools/test_fuzzing_harnesses.sh
 3Found 21 fuzz harnesses to test.
 4
 5Testing fuzzer address_deserialize during 1 second(s)
 6A subset of reached functions:
 7	NEW_FUNC[1/6]: 0x5592a0fa3430 in CDataStream& CDataStream::operator>><CAddress&>(CAddress&) src/./streams.h:460
 8	NEW_FUNC[2/6]: 0x5592a0fa4950 in void CAddress::SerializationOp<CDataStream, CSerActionUnserialize>(CDataStream&, CSerActionUnserialize) src/./protocol.h:337
 9	NEW_FUNC[3/6]: 0x5592a1ed2360 in CService::CService() src/netaddress.cpp:570
10	NEW_FUNC[4/6]: 0x5592a1f03670 in CAddress::CAddress() src/protocol.cpp:145
11	NEW_FUNC[5/6]: 0x5592a1f03780 in CAddress::Init() src/protocol.cpp:156
12	NEW_FUNC[0/3]: 0x5592a0fa5440 in void SerReadWriteMany<CDataStream, CService&>(CDataStream&, CSerActionUnserialize, CService&) src/./serialize.h:989
13	NEW_FUNC[1/3]: 0x5592a0fa5670 in void CService::SerializationOp<CDataStream, CSerActionUnserialize>(CDataStream&, CSerActionUnserialize) src/./netaddress.h:167
14	NEW_FUNC[2/3]: 0x5592a0fa5ac0 in void BigEndian<unsigned short>::Unserialize<CDataStream>(CDataStream&) src/./serialize.h:474
15stat::number_of_executed_units: 10343
16stat::average_exec_per_sec:     5171
17stat::new_units_added:          41
18stat::slowest_unit_time_sec:    0
19stat::peak_rss_mb:              147
20Number of unique code paths reached during fuzzing round: 13
21
22Testing fuzzer addrman_deserialize during 1 second(s)
23A subset of reached functions:
24	NEW_FUNC[0/61]: 0x55c567eabaa0 in UniqueLock<AnnotatedMixin<std::recursive_mutex>, std::unique_lock<std::recursive_mutex> >::UniqueLock(AnnotatedMixin<std::recursive_mutex>&, char const*, char const*, int, bool) src/./sync.h:146
25	NEW_FUNC[1/61]: 0x55c567eabf00 in UniqueLock<AnnotatedMixin<std::recursive_mutex>, std::unique_lock<std::recursive_mutex> >::~UniqueLock() src/./sync.h:165
26	NEW_FUNC[2/61]: 0x55c567eae5b0 in FastRandomContext::FillByteBuffer() src/./random.h:114
27	NEW_FUNC[3/61]: 0x55c567eca850 in uint256::uint256() src/./uint256.h:123
28	NEW_FUNC[4/61]: 0x55c567ed4f50 in UniqueLock<AnnotatedMixin<std::recursive_mutex>, std::unique_lock<std::recursive_mutex> >::Enter(char const*, char const*, int) src/./sync.h:123
29	NEW_FUNC[12/61]: 0x55c567f07320 in CDataStream& CDataStream::operator>><unsigned char&>(unsigned char&) src/./streams.h:460
30	NEW_FUNC[13/61]: 0x55c567f1d750 in CAddrMan::CAddrMan() src/./addrman.h:481
31	NEW_FUNC[14/61]: 0x55c567f1dec0 in CDataStream& CDataStream::operator>><CAddrMan&>(CAddrMan&) src/./streams.h:460
32	NEW_FUNC[15/61]: 0x55c567f1e050 in CAddrMan::~CAddrMan() src/./addrman.h:486
33	NEW_FUNC[16/61]: 0x55c567f1e250 in CAddrMan::Clear() src/./addrman.h:457
34stat::number_of_executed_units: 394
35stat::average_exec_per_sec:     197
36stat::new_units_added:          14
37stat::slowest_unit_time_sec:    0
38stat::peak_rss_mb:              111
39Number of unique code paths reached during fuzzing round: 12
40
41…
42
43All fuzz harnesses seem to work as expected.

    Commits:

    • Add contrib/devtools/test_fuzzing_harnesses.sh
    • Run test_fuzzing_harnesses.sh as part of RUN_FUZZ_TESTS in Travis
    • Enable UBSan for Travis fuzzer job
  2. practicalswift force-pushed on Sep 30, 2019
  3. practicalswift force-pushed on Sep 30, 2019
  4. MarcoFalke commented at 6:41 pm on September 30, 2019: member

    Add test_fuzzing_harnesses.sh for easy verification that our fuzz harnesses work the way we expect them to. More specifically that we are able to reach different unique code paths when varying input.

    Has this ever been an issue? Instead of adding a script that seems to give little reward, I’d prefer a script that exports the coverage to html.

    This can be done with clang out of the box:

    Screenshot_2019-09-30 Screenshot

  5. DrahtBot added the label Scripts and tools on Sep 30, 2019
  6. DrahtBot added the label Tests on Sep 30, 2019
  7. practicalswift commented at 9:48 pm on September 30, 2019: contributor

    @MarcoFalke Oh, I use this script all the time and thought others might find it useful :) Did you read the output? I find this script gives me much more valuable information than our current fuzz runner, but I could be biased :)

    Personally I prefer getting coverage information in the terminal so I’ll let someone else run with the export HTML idea :)

  8. practicalswift renamed this:
    tests: Add coverage test to verify that our fuzz harnesses reach different unique code paths when varying input
    tests: Add test_fuzzing_harnesses.sh for quick testing/verification of fuzzing harness coverage
    on Oct 1, 2019
  9. practicalswift force-pushed on Oct 1, 2019
  10. Add contrib/devtools/test_fuzzing_harnesses.sh for quick testing of all fuzzing harnesses 0d69a741e6
  11. Run test_fuzzing_harnesses.sh as part of RUN_FUZZ_TESTS in Travis 2a92b0f0f7
  12. Enable UBSan for Travis fuzzer job 19e317557a
  13. practicalswift force-pushed on Oct 1, 2019
  14. MarcoFalke commented at 4:13 pm on October 1, 2019: member
    llvm-cov can also export to plain text instead of html
  15. practicalswift commented at 7:23 am on October 2, 2019: contributor
    @MarcoFalke Have you tried this script? if not I suggest taking it for a spin before dismissing it :) The runtime is really quick compared to llvm-cov :)
  16. MarcoFalke referenced this in commit 2352aec9fc on Oct 10, 2019
  17. MarcoFalke commented at 9:13 pm on October 10, 2019: member
    No, I have not tried the script. I am using ./test/fuzz/test_runner.py
  18. MarcoFalke referenced this in commit 556820ee57 on Oct 14, 2019
  19. practicalswift closed this on Oct 14, 2019
  20. MarcoFalke referenced this in commit deb2327b43 on Oct 23, 2019
  21. MarcoFalke referenced this in commit 693e40090a on Oct 25, 2019
  22. laanwj referenced this in commit cb11324a63 on Dec 6, 2019
  23. jonatack commented at 11:23 am on December 6, 2019: member

    ACK 19e3175

    Output of test_fuzzing_harnesses.sh with this PR

    Output of test_fuzzing_harnesses.sh with PR #17051

  24. jonatack commented at 11:25 am on December 6, 2019: member

    Personally I prefer getting coverage information in the terminal

    Same. Terminal output seems complementary to HTML for online dashboards.

  25. sidhujag referenced this in commit 3e39c95207 on Dec 6, 2019
  26. MarcoFalke referenced this in commit 347dd76ec8 on Dec 9, 2019
  27. sidhujag referenced this in commit 317c64a54c on Dec 9, 2019
  28. MarcoFalke referenced this in commit 7da9e3a817 on Dec 11, 2019
  29. sidhujag referenced this in commit 03b450894e on Dec 12, 2019
  30. MarcoFalke referenced this in commit 806a2c602c on Dec 16, 2019
  31. sidhujag referenced this in commit 2a85553e25 on Nov 10, 2020
  32. sidhujag referenced this in commit ab87595d47 on Nov 10, 2020
  33. sidhujag referenced this in commit 81659f7a7c on Nov 10, 2020
  34. practicalswift deleted the branch on Apr 10, 2021
  35. PastaPastaPasta referenced this in commit 853592e0f5 on Sep 17, 2021
  36. thelazier referenced this in commit c81e1ecfeb on Sep 25, 2021
  37. PastaPastaPasta referenced this in commit 0e25465e15 on Feb 26, 2022
  38. PastaPastaPasta referenced this in commit 1a628a596a on Feb 26, 2022
  39. PastaPastaPasta referenced this in commit d254330372 on Feb 27, 2022
  40. PastaPastaPasta referenced this in commit d9ce5df395 on Mar 1, 2022
  41. PastaPastaPasta referenced this in commit b26851e7e2 on Mar 3, 2022
  42. PastaPastaPasta referenced this in commit 54684986a9 on Mar 3, 2022
  43. PastaPastaPasta referenced this in commit 36a5344f58 on Mar 5, 2022
  44. PastaPastaPasta referenced this in commit 0485b3a187 on Mar 7, 2022
  45. vijaydasmp referenced this in commit 23efc68863 on Mar 26, 2022
  46. DrahtBot locked this on Aug 18, 2022


practicalswift MarcoFalke jonatack

Labels
Tests Scripts and tools

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-07-03 13:13 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me