process_messages seeds #40

These are seeds produced with: --with-sanitizers=address,undefined,integer,fuzzer on Arch Linux. Merging with python test_runner.py --m_dir ~/merge-assets/fuzz_seed_corpus/ ~/qa-assets/fuzz_seed_corpus process_messages gives 13605 files from 13128.

As an extra data point: I also tried a build with --with-sanitizers=fuzzer and then merging with the python script. The first merge run gave 13407 files and the second run gave 13419 files so there is some variance.

The first merge run gave 13407 files and the second run gave 13419 files so there is some variance.

Not a single of our fuzz targets is deterministic (:sob:), so this is expected

Thanks

Not a single of our fuzz targets is deterministic (😭), so this is expected

Huh, that sounds... incorrect? :)

What makes you think that we don't have any deterministic harnesses? :)

The randomness module consumes environment data from the OS, which is not deterministic. This will upset at the very least use_value_profile because hashing the data into the rng will yield completely different non-deterministic states (as expected).

If there is a target that is fully deterministic with use_value_profile (what we use for merging), I am eager to know.

Not a single of our fuzz targets is deterministic (😭), so this is expected

Huh, that sounds... incorrect? :)

What makes you think that we don't have any deterministic harnesses? :)

The randomness module consumes environment data from the OS, which is not deterministic.

If you're referring to src/random.{cpp,h}: we have a lot of fuzzing harnesses that don't depend on it at all, no? :)

If there is a target that is fully deterministic with use_value_profile (what we use for merging), I am eager to know.

Determinism is a property of the harness, no?

A nondeterministic fuzzing harness is a fuzzing harness that, even for the same input, can exhibit different code paths on different runs.

Let's look at a fuzzing harness that is guaranteed to exhibit identical code paths across different runs given the same input.

I choose the trivial addition_overflow as an example: by reading the code it is clear that the harness is fully deterministic. There is simply no source of randomness or non-input driven variation across runs :)

See src/test/fuzz/addition_overflow.cpp:

FUZZ_TARGET(addition_overflow)
{
    FuzzedDataProvider fuzzed_data_provider(buffer.data(), buffer.size());
    TestAdditionOverflow<int64_t>(fuzzed_data_provider);
    TestAdditionOverflow<uint64_t>(fuzzed_data_provider);
    TestAdditionOverflow<int32_t>(fuzzed_data_provider);
    TestAdditionOverflow<uint32_t>(fuzzed_data_provider);
    TestAdditionOverflow<int16_t>(fuzzed_data_provider);
    TestAdditionOverflow<uint16_t>(fuzzed_data_provider);
    TestAdditionOverflow<char>(fuzzed_data_provider);
    TestAdditionOverflow<unsigned char>(fuzzed_data_provider);
    TestAdditionOverflow<signed char>(fuzzed_data_provider);
}

Do we agree that the addition_overflow fuzzing harness is fully deterministic? :)

If not, what is the source of non-determinism?

I am not sure if that affects the fuzz engine in any way, but the fuzz binary will at the very least call into random before main:

$ FUZZ=addition_overflow ./src/test/fuzz/fuzz 
fuzz: random.cpp:560: void ProcRand(unsigned char *, int, RNGLevel): Assertion `false' failed.
Aborted (core dumped)

With the assert(false) added in random.cpp.

==443853==    by 0x4CBB025: __assert_fail (in /usr/lib64/libc-2.32.so)
==443853==    by 0xC14DF7: ProcRand(unsigned char*, int, RNGLevel) (random.cpp:560)
==443853==    by 0xC14E36: GetRandBytes (random.cpp:585)
==443853==    by 0xC14E36: GetRandHash() (random.cpp:605)
==443853==    by 0x6DFE1F: CSignatureCache (sigcache.cpp:35)
==443853==    by 0x6DFE1F: __cxx_global_var_init.1 (sigcache.cpp:86)
==443853==    by 0x6DFE1F: _GLOBAL__sub_I_sigcache.cpp (sigcache.cpp:0)

@MarcoFalke Good catch! That's super ugly!

I read your earlier comment "not a single of our fuzz targets is deterministic" as a comment on the contents on the fuzzing harnesses (in other words the content of the individual LLVMFuzzerTestOneInput:s), but I now understand it is a comment on the pre-main behaviour of src/test/fuzz/fuzz. Thanks for clarifying!

I think this needs to be fixed.

What is the best way forward? Should we try to opt-out of randomness via a build macro when building src/test/fuzz/fuzz?

Perhaps we need a fuzzer friendly build mode FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION as suggested in the libFuzzer documentation.

I think the way forward is to:

• Remove globals (since they are polluting the fuzz binary)
• Make classes deterministic (either by passing a bool to the constructor or mocking them in ./src/test/util)
• (Last resort) Change the Core code to build or mock away the randomness module, but that might be controversial