Delete nonreduced fuzz inputs #204

1. 
   maflcko commented at 6:31 pm on August 29, 2024: contributor

   As per the usual process to avoid wasted CI resources and timeouts when it runs on large and presumed irrelevant inputs.

   Previous: #177

   To “reproduce”

   Install a fresh VM, as explained in the bash script’s doc, and run it:

   0apt update && apt install curl -y
   1curl -L -O https://raw.githubusercontent.com/bitcoin-core/bitcoin-maintainer-tools/main/delete_nonreduced_fuzz_inputs.sh
   2bash delete_nonreduced_fuzz_inputs.sh
   

   To “test”

   • Keep an eye on coverage stats, to ensure it doesn’t drop
   • Re-run the script, to ensure it is “reproducible” to some extent
   • Anything else you think is important to test or review

   CI

   CI should pass, except for a lint failure, which should light up on any changes like this pull request, which delete fuzz inputs.

2. Delete fuzz inputs 8da149a1dc
3. Reduced inputs for afl-cmin 22e7dfad82
4. Reduced inputs for fuzzer 306d20e289
5. Reduced inputs for fuzzer,address,undefined,integer b306615230
6. maflcko renamed this:
   Delete nonreduced inputs
   Delete nonreduced fuzz inputs
   on Aug 29, 2024
7. 
   maflcko commented at 6:32 pm on August 29, 2024: contributor

   :warning: This was run with the AFL-patched script from https://github.com/bitcoin-core/bitcoin-maintainer-tools/pull/169#issuecomment-2318553524, not from the main branch.
8. 
   maflcko commented at 6:49 pm on August 29, 2024: contributor

   Storage device usage (du -sh ./fuzz_seed_corpus/)

   4.0G -> 1.7G

   Determinism

   • ~100k files deleted

   0git diff --stat origin/main..b306615230c6e4a4ffc82cac1f8882d259e097de | tail -1
   1 107411 files changed, 632551 deletions(-)
   

   • Cross diff with the non-afl script result of ~40k files

   0git -c diff.renameLimit=25218 diff --stat HEAD..b306615230c6e4a4ffc82cac1f8882d259e097de | tail -1
   1 39980 files changed, 15224 insertions(+), 71963 deletions(-)
   

   • Cross-diff with a second run of the afl script of ~7k files

   0git -c diff.renameLimit=4159 diff --stat HEAD..b306615230c6e4a4ffc82cac1f8882d259e097de | tail -1 
   1 7678 files changed, 2408 insertions(+), 581 deletions(-)
   

   Coverage

   • main: https://drahtbot.space/host_reports/DrahtBot/reports/coverage_fuzz/monotree/ffdc3d6060f6e65e/65ac5a6b0b3f11ad/fuzz.coverage/index.html
   • pull: https://drahtbot.space/host_reports/DrahtBot/reports/coverage_fuzz/monotree/ffdc3d6060f6e65e/b306615230c6e4a4/fuzz.coverage/index.html
9. 
   maflcko commented at 10:18 am on August 30, 2024: contributor

   msan CI fail is unrelated (https://github.com/bitcoin/bitcoin/issues/30760)
10. 
    fanquake commented at 2:12 pm on August 30, 2024: member

    Are our logs going missing more quickly recently? Tried to look at the msan fuzz run here, but it seems to be gone: https://api.cirrus-ci.com/v1/task/5379875672948736/logs/ci.log ? The snippet still availalble is:

     0  Misses:           91 / 447 (20.36%)
     1+ du -sh /ci_container_base/depends/SDKs/ /ci_container_base/depends/builders/ /ci_container_base/depends/built/ /ci_container_base/depends/hosts/ /ci_container_base/depends/packages/ /ci_container_base/depends/patches/ /ci_container_base/depends/sdk-sources/ /ci_container_base/depends/sources/ /ci_container_base/depends/x86_64-pc-linux-gnu/
     24.0K	/ci_container_base/depends/SDKs/
     328K	/ci_container_base/depends/builders/
     4393M	/ci_container_base/depends/built/
     532K	/ci_container_base/depends/hosts/
     6128K	/ci_container_base/depends/packages/
     7204K	/ci_container_base/depends/patches/
     84.0K	/ci_container_base/depends/sdk-sources/
     9262M	/ci_container_base/depends/sources/
    10216M	/ci_container_base/depends/x86_64-pc-linux-gnu/
    11+ du -sh /ci_container_base/prev_releases
    124.0K	/ci_container_base/prev_releases
    13+ [[ x86_64-pc-linux-gnu = *-mingw32 ]]
    14+ '[' -n '' ']'
    15+ '[' false = true ']'
    16+ '[' '' = true ']'
    17+ '[' false = true ']'
    18+ '[' false = true ']'
    19+ '[' true = true ']'
    20+ LD_LIBRARY_PATH=/ci_container_base/depends/x86_64-pc-linux-gnu/lib
    21+ test/fuzz/test_runner.py -j6 -l DEBUG /ci_container_base/ci/scratch/qa-assets/fuzz_seed_corpus/ --empty_min_time=60
    22==4331==WARNING: MemorySanitizer: use-of-uninitialized-value
    23    [#0](/bitcoin-core-qa-assets/0/) 0x562684dc42db in SetArgs(int, char**) ci/scratch/build-x86_64-pc-linux-gnu/src/test/fuzz/util/./src/test/fuzz/fuzz.cpp:50:5
    24    [#1](/bitcoin-core-qa-assets/1/) 0x562684dc42db in LLVMFuzzerInitialize ci/scratch/build-x86_64-pc-linux-gnu/src/test/fuzz/util/./src/test/fuzz/fuzz.cpp:216:5
    25    [#2](/bitcoin-core-qa-assets/2/) 0x562684113cb8 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /msan/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:650:5
    26    [#3](/bitcoin-core-qa-assets/3/) 0x562684141062 in main /msan/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
    27    [#4](/bitcoin-core-qa-assets/4/) 0x7f36a60831c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 6d64b17fbac799e68da7ebd9985ddf9b5cb375e6)
    28    [#5](/bitcoin-core-qa-assets/5/) 0x7f36a608328a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 6d64b17fbac799e68da7ebd9985ddf9b5cb375e6)
    29    [#6](/bitcoin-core-qa-assets/6/) 0x562684108364 in _start (/ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/test/fuzz/fuzz+0x892364)
    30
    31  Member fields were destroyed
    32    [#0](/bitcoin-core-qa-assets/0/) 0x5626841d3c4d in __sanitizer_dtor_callback_fields /msan/llvm-project/compiler-rt/lib/msan/msan_interceptors.cpp:1048:5
    33    [#1](/bitcoin-core-qa-assets/1/) 0x562684107222 in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::~basic_string() /msan/cxx_build/include/c++/v1/string:840:44
    34    [#2](/bitcoin-core-qa-assets/2/) 0x562684107222 in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::~basic_string() /msan/cxx_build/include/c++/v1/string:1106:3
    35    [#3](/bitcoin-core-qa-assets/3/) 0x562684107222 in std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, BCLog::LogFlags>::~pair() /msan/cxx_build/include/c++/v1/__utility/pair.h:80:29
    36    [#4](/bitcoin-core-qa-assets/4/) 0x562684107222 in __cxx_global_var_init ci/scratch/build-x86_64-pc-linux-gnu/src/util/./src/logging.cpp:170:66
    37    [#5](/bitcoin-core-qa-assets/5/) 0x562684107222 in _GLOBAL__sub_I_logging.cpp ci/scratch/build-x86_64-pc-linux-gnu/src/util/./src/logging.cpp
    38    [#6](/bitcoin-core-qa-assets/6/) 0x7f36a6083303 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a303) (BuildId: 6d64b17fbac799e68da7ebd9985ddf9b5cb375e6)
    39    [#7](/bitcoin-core-qa-assets/7/) 0x562684108364 in _start (/ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/test/fuzz/fuzz+0x892364)
    40
    41SUMMARY: MemorySanitizer: use-of-uninitialized-value ci/scratch/build-x86_64-pc-linux-gnu/src/test/fuzz/util/./src/test/fuzz/fuzz.cpp:50:5 in SetArgs(int, char**)
    42Exiting
    43Traceback (most recent call last):
    44  File "/ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/test/fuzz/test_runner.py", line 411, in <module>
    45    main()
    46  File "/ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/test/fuzz/test_runner.py", line 115, in main
    47    test_list_all = parse_test_list(
    48                    ^^^^^^^^^^^^^^^^
    49  File "/ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/test/fuzz/test_runner.py", line 397, in parse_test_list
    50    test_list_all = subprocess.run(
    51                    ^^^^^^^^^^^^^^^
    52  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    53    raise CalledProcessError(retcode, process.args,
    54subprocess.CalledProcessError: Command '/ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/test/fuzz/fuzz' returned non-zero exit status 1.
    55��������
    

11. 
    murchandamus commented at 6:18 pm on August 30, 2024: contributor

    Determinism

    • ~100k deletions

    0git diff --stat origin/main..b306615230c6e4a4ffc82cac1f8882d259e097de | tail -1
    1 107411 files changed, 632551 deletions(-)
    

    Did you mean 632k deletions instead of 100k deletions?

    

    I noticed that the Branch Coverage went down minusculely in src/policy, src/rpc, src/script, and src (circled in image). That seems like an acceptable tradeoff for reducing the corpora by over 630,000 fuzz inputs.

12. 
    maflcko commented at 7:44 am on September 2, 2024: contributor

    Did you mean 632k deletions instead of 100k deletions?

    No, with deletions in this context I mean the number of fuzz input files that were deleted. I think the git “human readable” estimate of how many lines of “code” were deleted isn’t useful in this context. I guess it is counting the number of deleted newline characters (or so), which may be skewed. I’ve renamed “deletions” to “files deleted”.

13. 
    maflcko commented at 7:46 am on September 2, 2024: contributor

    I noticed that the Branch Coverage went down minusculely in src/policy, src/rpc, src/script, and src (circled in image). That seems like an acceptable tradeoff for reducing the corpora by over 630,000 fuzz inputs.

    Correct. I think this is due to some leftover non-stability or non-determinism.

    Other than that the cross-diff with a second run is the lowest ever recorded (just ~7k files). I presume it is either due to the afl addition, or due to the randomness changes in the master branch.

14. 
    fanquake commented at 11:16 am on September 3, 2024: member

    Kicked the fuzz with msan build, now that https://github.com/bitcoin/bitcoin/pull/30778 has been merged.
15. 
    maflcko commented at 2:58 pm on September 3, 2024: contributor

    Is this acceptable to merge, or is more review needed?
16. fanquake merged this on Sep 3, 2024
17. fanquake closed this on Sep 3, 2024

    ---

Contributors
maflcko fanquake murchandamus