tests: Add fuzzing harnesses for functions in script/ #18994

pull practicalswift wants to merge 7 commits into bitcoin:master from practicalswift:fuzzers-script-slash changing 9 files +442 −2
  1. practicalswift commented at 5:54 am on May 17, 2020: contributor

    Add fuzzing harnesses for functions in script/:

    • Add fuzzing helper functions ConsumeDataStream and ConsumeUInt160
    • Fill fuzzing coverage gaps for functions in script/script.h, script/script_error.h and script/standard.h
    • Add fuzzing harness for functions in script/bitcoinconsensus.h
    • Add fuzzing harness for functions in script/descriptor.h
    • Add fuzzing harness for functions in script/interpreter.h
    • Add fuzzing harness for functions in script/sigcache.h
    • Add fuzzing harness for functions in script/sign.h

    See doc/fuzzing.md for information on how to fuzz Bitcoin Core. Don’t forget to contribute any coverage increasing inputs you find to the Bitcoin Core fuzzing corpus repo.

    Happy fuzzing :)

  2. fanquake added the label Tests on May 17, 2020
  3. DrahtBot commented at 11:05 am on May 17, 2020: member

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Conflicts

    No conflicts as of last run.

  4. in src/test/fuzz/script.cpp:135 in 55778f1472 outdated
    124@@ -119,4 +125,41 @@ void test_one_input(const std::vector<uint8_t>& buffer)
    125             wit.SetNull();
    126         }
    127     }
    128+
    129+    const char* op_name = GetOpName(ConsumeOpcodeType(fuzzed_data_provider));
    130+    assert(op_name != nullptr);
    131+
    132+    const char* error_string = ScriptErrorString(static_cast<ScriptError>(fuzzed_data_provider.ConsumeIntegralInRange<int>(0, SCRIPT_ERR_ERROR_COUNT)));
    133+    assert(error_string != nullptr);
    


    MarcoFalke commented at 11:37 am on May 17, 2020:
    unrelated nit: At compile time this can never be null. I’d prefer if the return value for strings was always std::string. Every caller of ScriptErrorString converts to std::string anyway.

    practicalswift commented at 4:24 pm on May 17, 2020:
    I agree, we should always return std::string. Will do that in a follow-up PR since that will touch validation code and thus need more scrutiny than adding tests.
  5. in src/test/fuzz/script.cpp:165 in 55778f1472 outdated
    158+        (void)(witness_unknown_1 < witness_unknown_2);
    159+
    160+        const CNoDestination no_destination_1;
    161+        const CNoDestination no_destination_2;
    162+        (void)(no_destination_1 == no_destination_2);
    163+        (void)(no_destination_1 < no_destination_2);
    


    MarcoFalke commented at 11:42 am on May 17, 2020:
    this looks like a unit test that does not depend on any fuzz input

    MarcoFalke commented at 11:59 am on May 17, 2020:
    My suggestion would be to construct a random CTxDestination.

    practicalswift commented at 1:28 pm on May 18, 2020:
    Done! :)
  6. in src/test/fuzz/script_sign.cpp:134 in 55778f1472 outdated
    129+                if (!outpoint) {
    130+                    break;
    131+                }
    132+                const Optional<Coin> coin = ConsumeDeserializable<Coin>(fuzzed_data_provider);
    133+                if (!coin) {
    134+                    break;
    


    MarcoFalke commented at 11:58 am on May 17, 2020:
    unrelated, but I was wondering if ConsumeDeserializable<T> could always return type T (if the stream was empty, it would return T{}). Not returning optional would bring this helper in line with all other helpers. For example ConsumeRandomLengthString always return std::string.

    practicalswift commented at 1:48 pm on May 18, 2020:

    You mean something along the lines of:

     0template <typename T>
     1NODISCARD inline T ConsumeDeserializable(FuzzedDataProvider& fuzzed_data_provider, const size_t max_length = 4096) noexcept
     2{
     3    const std::vector<uint8_t> buffer = ConsumeRandomLengthByteVector(fuzzed_data_provider, max_length);
     4    CDataStream ds{buffer, SER_NETWORK, INIT_PROTO_VERSION};
     5    T obj;
     6    try {
     7        ds >> obj;
     8    } catch (const std::ios_base::failure&) {
     9    }
    10    return obj;
    11}
    

    I’ve thought about that and while it would be better from a developer ergonomics perspective I think it might come with some negative impact on overall fuzzing. One risk I see is that the fuzzer might try to work also on “meaningless” inputs (inputs which cause std::ios_base::failure) which would now become “somewhat meaningful” due to default construction.

    Take the extreme example of a huge input containing a few million bytes worth of 0x41 scream (AAAAAAAAAAA…) being deserialized to an object of type CFoo. The current fuzzer would quickly reach std::ios_base::failure, return a nullopt and then give up without further processing (fail early thanks to if (!opt_foo) { return; } or similar). If we were to return a CFoo{} instead then the fuzzer would proceed with processing (fail late) which may slow down the fuzzing.

    Does it make sense? :)

    tl;dr – would be worth doing if no negative fuzzing speed impact can be measured :)

    Edit: Proof of concept here if someone wants to experiment :)


    MarcoFalke commented at 2:34 pm on May 18, 2020:

    I think when it comes to trading off giving the fuzz test more freedom to explore more code branches vs micro-optimizing for performance, we should always pick the option that gives the fuzz engine more freedom. Fuzzing is great to complement traditional tests (e.g. our unit and functional tests). Those traditional tests offer a reasonable line coverage, but fuzz testing can offer great branch coverage. Giving the fuzz engine to explore as much branches as possible is vital. With modern coverage guided fuzz engines, the strength of fuzzing is that even with a large search space, the fuzz engine will always find useful inputs that increase coverage. Additional search space may (or may not!) come with a performance penalty, but I think we shouldn’t use performance as a way to measure usefulness of a fuzz tests. The fastest fuzz test is the one that tests nothing. However, it is also the most useless one. Anything that can be solved by letting the fuzz engine run for another hour or by simply adding one more core should not be a reason to cripple the test into one that yields less coverage.

    In fact, this has shown to be effective in the past. Take for example the process_message harness vs the process_messages harness. The latter is allowed to send an arbitrary number of messages and start an arbitrary number of peers. The latter (more flexible and broader) test that gives the engine more freedom is also the only test that has found bugs in pull requests that none of the other fuzz test or unit test or even functional tests have found. Specifically:

    • It found a nullptr dereference: #18808 (review)
    • It found an unidentified problem: #15197#pullrequestreview-408731734
    • Some more that I am missing…?

    practicalswift commented at 3:06 pm on May 18, 2020:

    I agree regarding the coverage vs performance trade-off: I’d choose “explore more code branches” over performance any day in the week :)

    Do you see a scenario where returning a default constructed object in case of serialisation failure is likely to give more freedom to explore more code branches (in a way that will have a positive impact of coverage) compared to not doing so?

    If so we’d have two reasons to do this: developer ergonomics (no more std::optional) and increased coverage :)

    My thinking is more along the lines of “how do we guide the coverage-guided fuzzer in a meaningful way” rather than “micro-optimizing for performance”. I guess the question boils down to if and to what extent the guidance provided by failing early is meaningful for the fuzzer :)

    Code if anyone wants to experiment: https://github.com/bitcoin/bitcoin/compare/master...practicalswift:consume


    MarcoFalke commented at 3:11 pm on May 18, 2020:

    Do you see a scenario where returning a default constructed object in case of serialisation failure is likely to give more freedom to explore more code branches compared to not doing so?

    Unclear. This question boils down to whether default initialization leaves members uninitialized or initializes them with values that can not be the result of deserialization.


    practicalswift commented at 7:43 pm on May 18, 2020:
    Agreed. Sounds like we agree on the need for experimentation and measuring before proceeding with this :)

    MarcoFalke commented at 10:59 pm on May 31, 2020:
    @practicalswift Are you going to create a pull with the branch or should I do it?

    practicalswift commented at 9:21 am on June 1, 2020:
    @MarcoFalke I plan to do a FuzzBench style test between the two variations (T ConsumeDeserializable(…) and std::optional<T> ConsumeDeserializable(…)) to see which one that helps us reach the highest coverage after 24 hours of fuzzing using libFuzzer (with each harness given 1/N slice of the runtime) with an empty input corpus. Does that sound like a fair experimental setup that could guide us regarding the trade-offs we’re facing here? If so, I’ll do it :)

    MarcoFalke commented at 10:27 am on June 1, 2020:
    Sure, can’t hurt to do a benchmark
  7. MarcoFalke approved
  8. MarcoFalke commented at 11:58 am on May 17, 2020: member
    ACK
  9. DrahtBot added the label Needs rebase on May 17, 2020
  10. practicalswift force-pushed on May 17, 2020
  11. DrahtBot removed the label Needs rebase on May 17, 2020
  12. practicalswift force-pushed on May 18, 2020
  13. MarcoFalke commented at 0:01 am on May 29, 2020: member
    Needs rebase
  14. tests: Add fuzzing helper functions ConsumeDataStream, ConsumeTxDestination and ConsumeUInt160 c571ecb071
  15. tests: Fill fuzzing coverage gaps for functions in script/script.h, script/script_error.h and script/standard.h 8de72711c6
  16. tests: Add fuzzing harness for functions in script/bitcoinconsensus.h 43fb8f0ca3
  17. tests: Add fuzzing harness for functions in script/descriptor.h fa80117cfd
  18. tests: Add fuzzing harness for functions in script/interpreter.h d3d8adb79f
  19. tests: Add fuzzing harness for functions in script/sigcache.h c91d2f0615
  20. tests: Add fuzzing harness for functions in script/sign.h f898ef65c9
  21. practicalswift force-pushed on May 30, 2020
  22. MarcoFalke commented at 10:58 pm on May 31, 2020: member

    ACK f898ef65c947776750e49d050633f830546bbdc6 🔉

    Signature:

     0-----BEGIN PGP SIGNED MESSAGE-----
     1Hash: SHA512
     2
     3ACK f898ef65c947776750e49d050633f830546bbdc6 🔉
     4-----BEGIN PGP SIGNATURE-----
     5
     6iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
     7pUgtcwwAk2++KR5/UnaZ6/Gls4rZq7sjVolrOPUw/ADuxRyFu/qM7nmAOuzhX645
     8/tuQPOszycpK7MuS8INKbw2gV4TTP35W7N9z8GhtbyaSlRXzBGF5vLGk0P1diJrp
     9GwSiwUle6VU4sRLX44DctSgP9NvNMBkl+zqFEJpjspXkGZuYvtDJrP4Ipm+oSNOw
    10d5K0zg6VKwz/+23qt9OJErovRltC7QUxIecI+Y0KmJzfkpScf+jGoMTFgo9q0zC4
    11JDLJ0ZAU8BeiN2J/IK0yyxG1LhZPOK1LikFWqebeD4JEOSA+FnWopXHCUAj+sL7H
    12hRZmjH1CJ8my2d2nOdqCmOn1R3K+kEuVB0L2cMpPsISRcPjgI8GlF4RcY23PpTBJ
    13FtjGQdg30fzpNeLW0aPvti4RV7e8eDsi87BOWOVpNKY6AqeKWWcgJwNejrQREfNY
    14/xHzGR2zvhFYcO7x5V2lafPHaB9CT6lxaRBsNFYfQf7ajaNGeHKXhjHsnQnlW46a
    15ELWIGbLJ
    16=55wg
    17-----END PGP SIGNATURE-----
    

    Timestamp of file with hash 6d0233da06adba4dcb766f9a62ac736cf11c827239891f0b3af49ea1152fcac1 -

  23. MarcoFalke merged this on May 31, 2020
  24. MarcoFalke closed this on May 31, 2020

  25. sidhujag referenced this in commit a24e2ef932 on Jun 1, 2020
  26. Fabcien referenced this in commit e9af331a77 on Jan 29, 2021
  27. practicalswift deleted the branch on Apr 10, 2021
  28. DrahtBot locked this on Aug 16, 2022

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-09-29 01:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me