fuzz: execute each file in dir without fuzz engine

ajtowns commented at 4:29 am on March 4, 2022: member

Phony fuzzing (phuzzing)! Run the fuzz testing code against known inputs to detect errors. Advantage is you can easily test using the existing qa-assets datasets without having to compile with fuzzing enabled; disadvantage is that it doesn’t do any actual fuzzing.

Example usage:

0$ for a in ${QA_ASSETS}/fuzz_seed_corpus/*; do echo ${a##*/}; done | xargs -P8 -I {} /bin/sh -c "FUZZ={} test/fuzz/fuzz ${QA_ASSETS}/fuzz_seed_corpus/{}"
1No fuzzer for address_deserialize.
2No fuzzer for addrdb.
3No fuzzer for banentry_deserialize.
4addition_overflow: succeeded against 848 files in 0s.
5asmap: succeeded against 981 files in 0s.
6checkqueue: succeeded against 211 files in 0s.
7...

(-P8 says run 8 of the tasks in parallel)

If there are failures, the first one will be reported and the program will abort with output like:

0fuzz: test/fuzz/versionbits.cpp:336: void (anonymous namespace)::versionbits_fuzz_target(FuzzBufferType): Assertion `exp_state != ThresholdState::FAILED' failed.
1Error processing seed "corpus/versionbits/35345ae8e722234095810b1117a29b63af7621af"

Rebase of #22763, which was a rebase of #21496, but also reports the name of the fuzzer and the time taken.

Fixes #21461

ajtowns commented at 4:33 am on March 4, 2022: member

cc @AnthonyRonning @prakash1512 @sipa @practicalswift

DrahtBot added the label Tests on Mar 4, 2022

laanwj commented at 10:48 am on March 7, 2022: member

Concept ACK.

ghost commented at 5:06 am on March 9, 2022: none

ACK.

Looks good to me, sorry for dropping that originally. Ran through some of the same tests I was doing before, all working as expected.

Example of that:

 0FUZZ=process_messages src/test/fuzz/fuzz ../qa-assets/fuzz_seed_corpus/process_messages/
 1process_messages: succeeded against 14024 files in 153s.
 2
 3
 4FUZZ=process_messages src/test/fuzz/fuzz ../qa-assets/fuzz_seed_corpus/process_messages/11265df2b1a0c93629515a880a2a851d6ceb133c
 5process_messages: succeeded against 1 files in 0s.
 6
 7for a in ../qa-assets/fuzz_seed_corpus/*; do echo ${a##*/}; done | xargs -P8 -I {} /bin/sh -c "FUZZ={} src/test/fuzz/fuzz ../qa-assets/fuzz_seed_corpus/{}"
 8addition_overflow: succeeded against 1047 files in 0s.
 9address_deserialize_v1_withtime: succeeded against 420 files in 0s.
10address_deserialize_v1_notime: succeeded against 400 files in 0s.
11addr_info_deserialize: succeeded against 1232 files in 1s.
12address_deserialize_v2: succeeded against 949 files in 1s.
13...

in src/test/fuzz/fuzz.cpp:64 in 80dd3013e0 outdated

60@@ -59,6 +61,7 @@ void FuzzFrameworkRegisterTarget(std::string_view name, TypeTestOneInput target,
61     Assert(it_ins.second);
62 }
63 
64+std::string_view g_fuzz_target;

MarcoFalke commented at 5:32 pm on March 16, 2022:

nit:

0static std::string_view g_fuzz_target;

in src/test/fuzz/fuzz.cpp:102 in 80dd3013e0 outdated

100-    Assert(it != FuzzTargets().end());
101+    g_fuzz_target = Assert(std::getenv("FUZZ"));
102+    const auto it = FuzzTargets().find(g_fuzz_target);
103+    if (it == FuzzTargets().end()) {
104+        std::cerr << "No fuzzer for " << g_fuzz_target << "." << std::endl;
105+        exit(1);

MarcoFalke commented at 5:35 pm on March 16, 2022:

nit:

0        std::exit(EXIT_FAILURE);

https://en.cppreference.com/w/cpp/utility/program/EXIT_status https://en.cppreference.com/w/cpp/utility/program/exit

in src/test/fuzz/fuzz.cpp:196 in 80dd3013e0 outdated

193+        }
194+        test_one_input(buffer);
195         return 0;
196     }
197-    test_one_input(buffer);
198+    signal(SIGABRT, signal_handler);

MarcoFalke commented at 5:48 pm on March 16, 2022:

nit: Will have to call std::signal, if the signal calls std::_Exit, otherwise it might be UB?

in src/test/fuzz/fuzz.cpp:144 in 80dd3013e0 outdated

139+void signal_handler(int signal)
140+{
141+    if (signal == SIGABRT) {
142+        std::cerr << "Error processing seed " << g_seed_path << std::endl;
143+    } else {
144+        std::cerr << "Unexpected signal " << signal << " received\n";

MarcoFalke commented at 5:48 pm on March 16, 2022:

Unclear if this stl call is allowed? Might be UB, but I guess it doesn’t matter either way?

ajtowns commented at 9:12 pm on March 16, 2022:

It matches the code in https://en.cppreference.com/w/cpp/utility/program/abort so should be mostly okay hopefully?

I suppose in theory you could setup worker threads, and have a monitor thread watch for them aborting – then you could do multiple inputs in parallel, and report on multiple failures rather than exiting after the first one.

in src/test/fuzz/fuzz.cpp:142 in 80dd3013e0 outdated

137+#if defined(PROVIDE_FUZZ_MAIN_FUNCTION) && !defined(__AFL_LOOP)
138+fs::path g_seed_path;
139+void signal_handler(int signal)
140+{
141+    if (signal == SIGABRT) {
142+        std::cerr << "Error processing seed " << g_seed_path << std::endl;

MarcoFalke commented at 5:49 pm on March 16, 2022:

nit: I think when simply iterating over inputs, they are not called seeds?

0        std::cerr << "Error processing input " << g_input_path << std::endl;

MarcoFalke commented at 5:56 pm on March 16, 2022:

( https://github.com/google/fuzzing/blob/master/docs/glossary.md#test-input )

MarcoFalke approved

MarcoFalke commented at 5:51 pm on March 16, 2022: member

review ACK

Left some nits

fuzz: execute each file in dir without fuzz engine

Co-Authored-By: Anthony Ronning <anthonyronning@gmail.com>

f59bee3fb2

ajtowns force-pushed on Mar 16, 2022

ajtowns commented at 9:30 pm on March 16, 2022: member

Renamed seed to input, added static/std, switched from signal.h to csignal header.

MarcoFalke approved

MarcoFalke merged this on Mar 17, 2022

MarcoFalke closed this on Mar 17, 2022

sidhujag referenced this in commit 16b7f5f572 on Mar 18, 2022

DrahtBot locked this on Mar 17, 2023

fuzz: execute each file in dir without fuzz engine #24472