Resurrect pstratem's "Simple fuzzing framework" #9172

pull laanwj wants to merge 2 commits into bitcoin:master from laanwj:2016_11_fuzzing_framework changing 6 files +346 −0
  1. laanwj commented at 10:33 AM on November 16, 2016: member

    Because I feel strongly that we should have it, this brings back #7940, with some minor changes to make it ready for merge:

    • Rebase for removal of CDataStream::GetVersion
    • Move to src/tests, rename to test_bitcoin_fuzzy to make it clear it is part of the tests, not a user entry point
    • Make sure that it doesn't get installed on make install nor shipped as part of the distribution (it is kind of useless without instrumentation). I opted to make it build by default with --enable-tests because if it is a specific option, no one will ever bother to enable it (which means it doesn't get built and runs out of date), and we don't want too many --enable-X anyway.
  2. laanwj added the label Tests on Nov 16, 2016
  3. jonasschnelli commented at 10:35 AM on November 16, 2016: contributor

    Concept ACK. Agree that a simple documentation would be useful. I guess right now you have to pipe in some data that will be used for the fuzz tests?

  4. laanwj commented at 10:38 AM on November 16, 2016: member

    I guess right now you have to pipe in some data that will be used for the fuzz tests?

    Piping in data is the part that the fuzzer should do :)

  5. laanwj commented at 11:27 AM on November 16, 2016: member

    I've added some bare-bones instructions on how to use AFL with Bitcoin Core. What I don't have are example inputs, so this doesn't do anything yet at the moment. @pstratem Would you mind sharing some of your example inputs?

  6. MarcoFalke commented at 12:11 PM on November 16, 2016: member

    Concept ACK. Would prefer to see at least one MWE.

  7. laanwj added this to the milestone 0.14.0 on Nov 16, 2016
  8. laanwj commented at 12:44 PM on November 16, 2016: member

    What's an MWE?

  9. MarcoFalke commented at 12:59 PM on November 16, 2016: member

    Minimal working example: Just a code bit that is of the smallest size, but still demonstrates how the framework is used.

  10. laanwj commented at 1:04 PM on November 16, 2016: member

    Minimal working example: Just a code bit that is of the smallest size, but still demonstrates how the framework is used.

    Agreed. I demonstrate how to use the framework in the documentation I added, but there need to be some example inputs to put in inputs/ or it croaks. I could make a few but I'd rather wait for Patrick a bit whether he already has some as it's quite time-consuming.

  11. pstratem commented at 5:05 AM on November 19, 2016: contributor

    @laanwj suggestion for where examples should go?

    (which directory)

  12. laanwj commented at 8:45 AM on November 19, 2016: member

    Don't know. Probably a special repo with bitcoin core "manual testing" data eventually. At this point though I'd be happy with a dropbox link as I'd just like to start fuzzing.

  13. gmaxwell commented at 10:48 AM on November 19, 2016: contributor

    FWIW, I have a 24 core host grinding on this in AFL for most of a day on this... I guess I'll leave it running all weekend and spin up some more cores. if someone wants the resulting vectors after I cmin them, I'd be glad to hand them out.

    (I just started from some particularly dumb /dev/urandom starting points... a future improvement I would suggest (not this PR!) might be a command-line argument that makes it serialize a dummy object or few for each of the modes and write out a set of files.)

  14. laanwj commented at 7:54 AM on November 21, 2016: member

    a future improvement I would suggest (not this PR!) might be a command-line argument that makes it serialize a dummy object or few for each of the modes and write out a set of files.

    Yes that would be useful. Should ideally have realistic examples that come up in bitcoind itself (just grabbed from the wire randomly and saved) along with synthetic ones.

    if someone wants the resulting vectors after I cmin them, I'd be glad to hand them out.

    Yes please!

  15. in doc/fuzzing.md:None in 18b4c6cf21 outdated 
      18 | +
  19 | +To build Bitcoin Core using AFL instrumentation (this assumes that the
  20 | +`AFLPATH` was set as above):
  21 | +```
  22 | +./configure --disable-ccache --disable-shared --enable-tests CC=${AFLPATH}/afl-gcc CXX=${AFLPATH}/afl-g++
  23 | +export AFL_HARDEN=1
    fanquake commented at 7:49 AM on November 23, 2016:

    For anyone wondering what AFL_HARDEN does. From the afl README.

    Setting AFL_HARDEN=1 when calling 'make' will cause the CC wrapper to
automatically enable code hardening options that make it easier to detect
simple memory bugs. Libdislocator, a helper library included with AFL (see
libdislocator/README.dislocator) can help uncover heap corruption issues, too.
    laanwj commented at 7:54 AM on November 23, 2016:

    It may actually be unnecessary with bitcoin core which already enables a lot of hardening options. Then again, it can't hurt either I think.

  16. fanquake commented at 1:09 PM on November 24, 2016: member

    Started testing this on OS X 10.12. afl compiles fine, and bitcoin configure completes (using afl-clang and afl-clang++ instead of afl-gcc and afl-gcc++):

    Options used to compile and link:
  with wallet   = yes
  with gui / qt = yes
    qt version  = 5
    with qr     = yes
  with zmq      = yes
  with test     = yes
  with bench    = yes
  with upnp     = yes
  debug enabled = no

  target os     = darwin
  build os      = darwin

  CC            = /Users/xxx/github/bitcoin/afl-2.35b/afl-clang
  CFLAGS        = -g -O2
  CPPFLAGS      =  -DHAVE_BUILD_INFO -D__STDC_FORMAT_MACROS -I/usr/local/opt/berkeley-db4/include -DMAC_OSX
  CXX           = /Users/xxx/github/bitcoin/afl-2.35b/afl-clang++ -std=c++11
  CXXFLAGS      = -g -O2 -Wall -Wextra -Wformat -Wformat-security -Wno-unused-parameter -Wno-self-assign -Wno-unused-local-typedef -Wno-deprecated-register
  LDFLAGS       =  -Wl,-headerpad_max_install_names -Wl,-dead_strip

    test_bitcoin_fuzzy compiles with one warning

    bash-3.2$ make test/test_bitcoin_fuzzy
  CXX      test/test_test_bitcoin_fuzzy-test_bitcoin_fuzzy.o
afl-cc 2.35b by <lcamtuf@google.com>
afl-as 2.35b by <lcamtuf@google.com>
[+] Instrumented 2777 locations (64-bit, hardened mode, ratio 100%).
  CXX      lib/libunivalue_la-univalue.lo
... SNIP ...
afl-cc 2.35b by <lcamtuf@google.com>
afl-as 2.35b by <lcamtuf@google.com>
[!] WARNING: No instrumentation targets found.
  CXX      libbitcoin_util_a-random.o
... SNIP ...
afl-cc 2.35b by <lcamtuf@google.com>
afl-as 2.35b by <lcamtuf@google.com>
[+] Instrumented 48 locations (64-bit, hardened mode, ratio 100%).
  AR       crypto/libbitcoin_crypto.a
  CC       src/libsecp256k1_la-secp256k1.lo
afl-cc 2.35b by <lcamtuf@google.com>
afl-as 2.35b by <lcamtuf@google.com>
[+] Instrumented 903 locations (64-bit, hardened mode, ratio 100%).
  CCLD     libsecp256k1.la
  CXXLD    test/test_bitcoin_fuzzy

    However, running gave:

    bash-3.2$ $AFLPATH/afl-fuzz -i ${AFLIN} -o ${AFLOUT} -- test/test_bitcoin_fuzzy
afl-fuzz 2.35b by <lcamtuf@google.com>
[+] You have 8 CPU cores and 3 runnable tasks (utilization: 38%).
[+] Try parallel jobs - see /usr/local/share/doc/afl/parallel_fuzzing.txt.
[*] Setting up output directories...
[+] Output directory exists but deemed OK to reuse.
[*] Deleting old session data...
[+] Output dir cleanup successful.
[*] Scanning '/Users/xxx/github/bitcoin/src/inputs'...
[+] No auto-generated dictionary tokens to reuse.
[*] Creating hard links for all input files...
[*] Validating target binary...
[*] Attempting dry run with 'id:000000,orig:auto_000000'...
[*] Spinning up the fork server...

[-] Hmm, looks like the target binary terminated before we could complete a
    handshake with the injected code. There are two probable explanations:

    - The current memory limit (50.0 MB) is too restrictive, causing an OOM
      fault in the dynamic linker. This can be fixed with the -m option. A
      simple way to confirm the diagnosis may be:

[-] PROGRAM ABORT : Fork server handshake failed
         Location : init_forkserver(), afl-fuzz.c:2247

    Same result even after upping the memory limit to 4GB (-m4096) which should be more than enough.

    Setting AFL_NO_FORKSRV=1 seems to have made it run. From the readme:

    Setting AFL_NO_FORKSRV disables the forkserver optimization, reverting to
    fork + execve() call for every tested input. This is useful mostly when
    working with unruly libraries that create threads or do other crazy
    things when initializing (before the instrumentation has a chance to run).

    So for now it's running: screen shot 1

    Also testing on Ubuntu 16.04 running in VirtualBox.

    Options used to compile and link:
  with wallet   = yes
  with gui / qt = yes
    qt version  = 5
    with qr     = auto
  with zmq      = yes
  with test     = yes
  with bench    = yes
  with upnp     = yes
  debug enabled = no

  target os     = linux
  build os      = 

  CC            = /home/ubuntu/bitcoin/afl-2.35b/afl-gcc
  CFLAGS        = -g -O2
  CPPFLAGS      =  -DHAVE_BUILD_INFO -D__STDC_FORMAT_MACROS
  CXX           = /home/ubuntu/bitcoin/afl-2.35b/afl-g++ -std=c++11
  CXXFLAGS      = -g -O2 -Wall -Wextra -Wformat -Wformat-security -Wno-unused-parameter
  LDFLAGS       =

    screen shot

  17. fanquake commented at 1:37 PM on November 24, 2016: member

    Given that one was working alright, I've moved to running five in parallel. Will leave them running for a while. You can use the afl-whatsup tool to watch the progress, sample output below:

    $AFLPATH/afl-whatsup sync_dir
status check tool for afl-fuzz by <lcamtuf@google.com>

Individual fuzzers
==================

>>> fuzzer01 (1 days, 0 hrs) <<<

  cycle 1, lifetime speed 159 execs/sec, path 744/1201 (61%)
  pending 52/1043, coverage 6.22%, no crashes yet

>>> fuzzer02 (1 days, 0 hrs) <<<

  cycle 80, lifetime speed 179 execs/sec, path 346/1199 (28%)
  pending 0/0, coverage 6.22%, no crashes yet

>>> fuzzer03 (1 days, 0 hrs) <<<

  cycle 80, lifetime speed 181 execs/sec, path 504/1190 (42%)
  pending 0/0, coverage 6.22%, no crashes yet

>>> fuzzer04 (1 days, 0 hrs) <<<

  cycle 83, lifetime speed 181 execs/sec, path 1180/1190 (99%)
  pending 0/0, coverage 6.22%, no crashes yet

>>> fuzzer05 (0 days, 1 hrs) <<<

  cycle 3, lifetime speed 146 execs/sec, path 621/1206 (51%)
  pending 0/687, coverage 6.22%, no crashes yet

Summary stats
=============

       Fuzzers alive : 5
      Total run time : 4 days, 4 hours
         Total execs : 63 million
    Cumulative speed : 846 execs/sec
       Pending paths : 52 faves, 1730 total
  Pending per fuzzer : 10 faves, 346 total (on average)
       Crashes found : 0 locally unique
  18. in doc/fuzzing.md:None in 18b4c6cf21 outdated 
      33 | +
  34 | +AFL needs an input directory with examples, and an output directory where it will place
  35 | +examples that it found.
  36 | +
  37 | +```
  38 | +mkdir inputs
    MarcoFalke commented at 12:37 PM on November 25, 2016:

    µnit:

    mkdir test/afl-inputs
AFLIN=$PWD/test/afl-inputs
...
    fanquake commented at 12:52 PM on November 29, 2016:

    nit: we could also add .gitignore entries for test/inputs, test/outputs and test/sync_dir

    laanwj commented at 7:38 AM on December 5, 2016:

    Well, this is just an example. There's no strong reason to make these directories inside the bitcoin repository.

  19. MarcoFalke commented at 12:38 PM on November 25, 2016: member

    Concept ACK 18b4c6c

  20. in doc/fuzzing.md:None in 18b4c6cf21 outdated 
      44 | +Fuzzing
  45 | +--------
  46 | +
  47 | +To start the actual fuzzing use:
  48 | +```
  49 | +$AFLPATH/afl-fuzz -i ${AFLIN} -o ${AFLOUT} -- test/test_bitcoin_fuzzy
    MarcoFalke commented at 2:01 PM on November 25, 2016:

    I think the default memory needs to be increased to prevent OOM?

    fanquake commented at 2:04 PM on November 25, 2016:

    Which OS did you see out of memory errors on? What did you need to increase it to to make it run?

    MarcoFalke commented at 2:22 PM on November 25, 2016:

    I used the inputs by @laanwj (bitcoin_fuzzy_in.tar.xz) and the default of -m50 caused a failure in the dry run.

    MarcoFalke commented at 2:26 PM on November 25, 2016:

    Setting -m52 seems to be enough in this case. (OS: fedora 24)

  21. laanwj commented at 2:14 PM on November 25, 2016: member

    I didn't have to change anything memory related on ubuntu at least

  22. fanquake commented at 2:15 PM on November 25, 2016: member

    Some results. The "Master" instance is yet to complete a cycle (~62%), and the exec speed seems to have dropped off in the last few hours. Current stats: screen shot 4 The other instances all have similar output, and have run through ~80cycles: screen shot 5

  23. theuni commented at 8:39 PM on November 25, 2016: member

    tested ACK. Works well on OSX with AFL_NO_FORKSRV set (probably worth documenting that it's necessary there).

    Also, feel free to take this as a simplification: https://github.com/theuni/bitcoin/commit/88b8f922c4b4a3db5d299cd64e0c9669269873a1

  24. laanwj commented at 12:32 PM on November 26, 2016: member

    @cfields thanks - though I'm not sure moving addrman to common is the right thing to do. It is decidedly a server thing and not used by any of the other (nontest) executables. And later on we may want to fuzz more things in _server.

  25. pstratem commented at 5:03 AM on November 27, 2016: contributor

    @fanquake if you completely remove the memory limit there are a few tests which can use nearly unbounded amounts of memory

    I suggest running with:

    export AFL_SKIP_CRASHES=1; export AFL_PARAMETERS="-t 1000+ -m 1024";

    afl-fuzz $AFL_PARAMETERS -i fuzzing/input -o fuzzing/output ./bitcoin/src/test/test_bitcoin_fuzzy

  26. fanquake commented at 2:26 PM on November 27, 2016: member

    @pstratem Thanks for the suggestions. I've restarted some of the fuzzers with the new vars.

  27. fanquake commented at 12:49 PM on November 29, 2016: member

    Wrapping up my testing of this. Results from individual fuzzers are below, and the sync_dir (outputs) as well as the inputs used are available for download. inputs.zip sync_dir.zip Fuzzer 01 (deterministic tests): fuzzer01 Fuzzer 02: fuzzer02 Fuzzer 03: fuzzer03 Fuzzer 04: fuzzer04 Fuzzer 05 (restarted to up allocated memory and pass -t 1000): fuzzer05 Total Output:

    $AFLPATH/afl-whatsup sync_dir
status check tool for afl-fuzz by <lcamtuf@google.com>

Individual fuzzers
==================

>>> fuzzer01 (4 days, 23 hrs) <<<

  cycle 2, lifetime speed 114 execs/sec, path 297/1222 (24%)
  pending 0/935, coverage 6.22%, no crashes yet

>>> fuzzer02 (4 days, 23 hrs) <<<

  cycle 376, lifetime speed 165 execs/sec, path 87/1220 (7%)
  pending 0/0, coverage 6.22%, no crashes yet

>>> fuzzer03 (4 days, 23 hrs) <<<

  cycle 379, lifetime speed 166 execs/sec, path 114/1211 (9%)
  pending 0/0, coverage 6.22%, no crashes yet

>>> fuzzer04 (4 days, 23 hrs) <<<

  cycle 388, lifetime speed 166 execs/sec, path 1030/1211 (85%)
  pending 0/0, coverage 6.22%, no crashes yet

>>> fuzzer05 (2 days, 4 hrs) <<<

  cycle 161, lifetime speed 162 execs/sec, path 62/1227 (5%)
  pending 0/0, coverage 6.22%, no crashes yet

Summary stats
=============

       Fuzzers alive : 5
      Total run time : 22 days, 1 hours
         Total execs : 294 million
    Cumulative speed : 773 execs/sec
       Pending paths : 0 faves, 935 total
  Pending per fuzzer : 0 faves, 187 total (on average)
       Crashes found : 0 locally unique
  28. sipa commented at 1:12 AM on December 1, 2016: member

    Concept ACK

  29. MarcoFalke commented at 7:31 PM on December 2, 2016: member

    I think this ready for merge and we should select and add appropriate afl-inputs in another pull.

    (Maybe fix the doc nits before merge?)

  30. laanwj commented at 6:42 AM on December 3, 2016: member

    I think this ready for merge and we should select and add appropriate afl-inputs in another pull.

    I don't think we should put the inputs in the repository. But I'll just add a link where they can be downloaded. Will fix the doc nits.

  31. pstratem commented at 8:44 PM on December 4, 2016: contributor

    My current test cases are here http://strateman.ninja/fuzzing.tar.xz

  32. laanwj force-pushed on Dec 5, 2016
  33. laanwj force-pushed on Dec 5, 2016
  34. Simple fuzzing framework a4153e20ec
  35. doc: Add bare-bones documentation for fuzzing 8b15434b59
  36. in src/test/test_bitcoin_fuzzy.cpp:None in 8e0550fd38 outdated 
      93 | +        case CTRANSACTION_DESERIALIZE:
  94 | +        {
  95 | +            try
  96 | +            {
  97 | +                CTransaction tx;
  98 | +                ds >> tx;
    MarcoFalke commented at 10:41 AM on December 5, 2016: 
    error: ‘class CTransaction’ has no member named ‘Unserialize’
    laanwj commented at 11:41 AM on December 5, 2016:

    right, this should probably be converted to CMutableTransaction

  37. laanwj force-pushed on Dec 15, 2016
  38. laanwj commented at 3:54 PM on December 15, 2016: member

    Switched to the new way of transaction deserialization. This should be ready for merge now.

  39. laanwj merged this on Dec 15, 2016
  40. laanwj closed this on Dec 15, 2016
  41. laanwj referenced this in commit 5bc209c73f on Dec 15, 2016
  42. sipa commented at 5:21 PM on December 15, 2016: member

    utACK 8b15434b59c6cd7368b4db680544cd77ed337bd3 (apart from #9354).

  43. PastaPastaPasta referenced this in commit f55cf17040 on Aug 24, 2019
  44. barrystyle referenced this in commit 9c4dc3a720 on Jan 22, 2020
  45. random-zebra referenced this in commit 44b5327e61 on May 28, 2021
  46. DrahtBot locked this on Sep 8, 2021
Contributors
laanwjjonasschnelliMarcoFalkepstratemgmaxwellfanquaketheunisipa
Labels
Tests

Milestone
0.14.0

Linked (view graph)
#7940 [WIP] Fuzzing framework#9354 Make fuzzer actually test CTxOutCompressor
github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-04-19 12:15 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me