fuzz: support for Eclipser fuzzing #22572

issue agroce openend this issue on July 28, 2021
  1. agroce commented at 7:53 pm on July 28, 2021: contributor

    For improving coverage in process_message fuzzing, Eclipser (https://github.com/SoftSec-KAIST/Eclipser/tree/v1.x) has proven extremely effective (see https://github.com/bitcoin-core/qa-assets/pull/70 https://github.com/bitcoin-core/qa-assets/pull/69 https://github.com/bitcoin-core/qa-assets/pull/68 https://github.com/bitcoin-core/qa-assets/pull/67).

    However, it is not supported by OSS-Fuzz, and currently running standalone with requires using a patch such as in #22472.

    I would like to add instructions on fuzzing with Eclipser to https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md but it isn’t really useful until some solution to the /tmp cleanup issues is available.

    Long term, there needs to be a procedure for running Eclipser at regular intervals, for long runs to cover code that OSS-Fuzz libFuzzer and afl are bad at.

  2. agroce added the label Feature on Jul 28, 2021
  3. maflcko commented at 6:48 am on July 29, 2021: member

    has proven extremely effective

    I looked at the new coverage added and it didn’t include any new line coverage (just edge coverage). When pulling new inputs from other fuzz engines I also see many files added (even with new line coverage), so it is not yet clear whether the files added by Eclipser are actually ones that greybox engines can’t find.

  4. maflcko added the label Docs on Jul 29, 2021
  5. agroce commented at 6:55 am on July 29, 2021: contributor
    Aha. Ok, no actual new lines – not as exciting. Does merge have a message for new lines being found? I assume new edges are worth adding?
  6. practicalswift commented at 8:56 am on July 29, 2021: contributor

    I assume new edges are worth adding?

    Absolutely! :)

    BTW, extremely happy to have you onboard the Bitcoin Core fuzzing train @agroce: I’m a huge huge fan of your fuzzing and testing research (context for those unaware).

    Super excited to see fuzzing researchers taking interest in Bitcoin Core as a fuzzing target. That’s something I’ve wished for and tried to facilitate/make happen for years :)

  7. maflcko commented at 10:54 am on July 29, 2021: member

    Indeed they are worth adding.

    I run the coverage builds myself. I haven’t yet run them on PR70…

  8. maflcko commented at 10:55 am on July 29, 2021: member

    but it isn’t really useful until some solution to the /tmp cleanup issues is available.

    The issue should only affect fuzz targets that boot up a “full node” (process_message*, …), but not the others. So I think the instructions could still be useful.

  9. maflcko added the label Tests on Jul 29, 2021
  10. maflcko commented at 2:15 pm on July 29, 2021: member

    Just added the latest OSS-Fuzz inputs. (process_message: MERGE-OUTER: 113 new files with 214 new features added; 41 new coverage edges)

    Also the inputs from my server (libFuzzer).

    Result: https://drahtbot.space/host_reports/DrahtBot/reports/coverage_fuzz/monotree/be92dc5876969da2/fuzz.coverage/index.html

    Seems to have added ~50 new coverage lines across all fuzz tests.

    Some examples:

  11. agroce commented at 5:57 pm on July 29, 2021: contributor
    @practicalswift I’d looked at bitcoin fuzzing before, but saw that there was nice multi-fuzzer infrastructure in place. I actually got involved when @adamjonas contacted me in late March to see if I knew anyone who might help with saturation issues you all were facing in the fuzzing. I did – me!
  12. agroce commented at 5:59 pm on July 29, 2021: contributor
    @MarcoFalke what’s the process for these on the server? Just continuous libFuzzer, and occasionally checking for new stuff to merge? Any idea if more came from your libFuzzer runs or from OSS-Fuzz (or an equal mix)?
  13. agroce commented at 1:31 am on July 30, 2021: contributor

    @MarcoFalke @practicalswift this is a tangent, but there’s not a better place to discuss it (would either of you like to join an email thread for some “dumb questions” from me?)

    Is rpc stateful in that one call can have impact on the next, in the fuzzing setup? It looks like the fuzz harness only does one command. If rpc is not somewhere this would be useful, are there targets where performing a sequence would be useful? If so, are they doing so now?

  14. maflcko commented at 8:03 am on July 30, 2021: member

    @MarcoFalke what’s the process for these on the server? Just continuous libFuzzer, and occasionally checking for new stuff to merge?

    Jup, libFuzzer for each target (with/without sanitizers). I manually rsync the inputs and add them to qa-assets if they are useful.

    Any idea if more came from your libFuzzer runs or from OSS-Fuzz (or an equal mix)?

    It is hard to say. OSS-Fuzz inputs are 30 days old (the public archives) https://github.com/bitcoin-core/qa-assets/blob/2a7d883b08c8b132e61371c1bc9f336c71c56bc6/download_oss_fuzz_inputs.py#L9. The inputs on my server are fresh (up-to-date for the latest code). OSS-Fuzz added 8k files (https://github.com/bitcoin-core/qa-assets/commit/9c6d701938ee8b7be5e1b66895fcba4ac728b7dd). My run afterward added 27k files (https://github.com/bitcoin-core/qa-assets/commit/2a7d883b08c8b132e61371c1bc9f336c71c56bc6)

    @MarcoFalke @practicalswift this is a tangent, but there’s not a better place to discuss it (would either of you like to join an email thread for some “dumb questions” from me?)

    I like all my Bitcoin Core communication to be public for transparency and documentation purposes. And often someone’s dumb questions are someone else’s dumb questions, too. If you prefer a place with less watchers you can also ask over at https://github.com/bitcoin-core/qa-assets/issues. Though, I am also happy to start an email thread.

    Is rpc stateful in that one call can have impact on the next, in the fuzzing setup? It looks like the fuzz harness only does one command. If rpc is not somewhere this would be useful, are there targets where performing a sequence would be useful? If so, are they doing so now?

    Yes, some RPCs are stateful. However in practice this doesn’t lead to issues, because the only state IIRC is blockchain, mempool and wallet state. For blockchain it will be hard for the fuzzer to find new state (valid blocks) on it’s own. Same goes for mempool. The wallet is currently not compiled into the fuzz binary, so shouldn’t cause issues either.

    If in the future any state turns out to make issues non-reproducible, we should look into fixing that. And conversely if we believe there are issues to be found only with state preserved between calls, we should look into extending the fuzz target to cover that.

    I’ve written a bunch of targets to be keep state around for a longer time. For example there is process_messages (trailing s) that allows for multiple messages to be sent, because net processing is stateful.

  15. agroce commented at 3:07 pm on July 30, 2021: contributor

    Thanks! That’s very helpful, and here is fine by me!

    I should take a look at process_messages…

  16. maflcko commented at 6:31 pm on February 8, 2024: member
    Closing for now, but pull requests with documentation on the build with Eclipser are weclome
  17. maflcko closed this on Feb 8, 2024


agroce maflcko practicalswift

Labels
Feature Docs Tests

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-09-29 04:12 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me