Resurrect pstratem's "Simple fuzzing framework"

laanwj commented at 10:33 am on November 16, 2016: member

Because I feel strongly that we should have it, this brings back #7940, with some minor changes to make it ready for merge:

Rebase for removal of CDataStream::GetVersion
Move to src/tests, rename to test_bitcoin_fuzzy to make it clear it is part of the tests, not a user entry point
Make sure that it doesn’t get installed on make install nor shipped as part of the distribution (it is kind of useless without instrumentation). I opted to make it build by default with --enable-tests because if it is a specific option, no one will ever bother to enable it (which means it doesn’t get built and runs out of date), and we don’t want too many --enable-X anyway.

laanwj added the label Tests on Nov 16, 2016

jonasschnelli commented at 10:35 am on November 16, 2016: contributor

Concept ACK. Agree that a simple documentation would be useful. I guess right now you have to pipe in some data that will be used for the fuzz tests?

laanwj commented at 10:38 am on November 16, 2016: member

I guess right now you have to pipe in some data that will be used for the fuzz tests?

Piping in data is the part that the fuzzer should do :)

laanwj commented at 11:27 am on November 16, 2016: member

I’ve added some bare-bones instructions on how to use AFL with Bitcoin Core. What I don’t have are example inputs, so this doesn’t do anything yet at the moment. @pstratem Would you mind sharing some of your example inputs?

MarcoFalke commented at 12:11 pm on November 16, 2016: member

Concept ACK. Would prefer to see at least one MWE.

laanwj added this to the milestone 0.14.0 on Nov 16, 2016

laanwj commented at 12:44 pm on November 16, 2016: member

What’s an MWE?

MarcoFalke commented at 12:59 pm on November 16, 2016: member

Minimal working example: Just a code bit that is of the smallest size, but still demonstrates how the framework is used.

laanwj commented at 1:04 pm on November 16, 2016: member

Minimal working example: Just a code bit that is of the smallest size, but still demonstrates how the framework is used.

Agreed. I demonstrate how to use the framework in the documentation I added, but there need to be some example inputs to put in inputs/ or it croaks. I could make a few but I’d rather wait for Patrick a bit whether he already has some as it’s quite time-consuming.

pstratem commented at 5:05 am on November 19, 2016: contributor

@laanwj suggestion for where examples should go?

(which directory)

laanwj commented at 8:45 am on November 19, 2016: member

Don’t know. Probably a special repo with bitcoin core “manual testing” data eventually. At this point though I’d be happy with a dropbox link as I’d just like to start fuzzing.

gmaxwell commented at 10:48 am on November 19, 2016: contributor

FWIW, I have a 24 core host grinding on this in AFL for most of a day on this… I guess I’ll leave it running all weekend and spin up some more cores. if someone wants the resulting vectors after I cmin them, I’d be glad to hand them out.

(I just started from some particularly dumb /dev/urandom starting points… a future improvement I would suggest (not this PR!) might be a command-line argument that makes it serialize a dummy object or few for each of the modes and write out a set of files.)

laanwj commented at 7:54 am on November 21, 2016: member

a future improvement I would suggest (not this PR!) might be a command-line argument that makes it serialize a dummy object or few for each of the modes and write out a set of files.

Yes that would be useful. Should ideally have realistic examples that come up in bitcoind itself (just grabbed from the wire randomly and saved) along with synthetic ones.

if someone wants the resulting vectors after I cmin them, I’d be glad to hand them out.

Yes please!

in doc/fuzzing.md: in 18b4c6cf21 outdated

18+
19+To build Bitcoin Core using AFL instrumentation (this assumes that the
20+`AFLPATH` was set as above):
21+```
22+./configure --disable-ccache --disable-shared --enable-tests CC=${AFLPATH}/afl-gcc CXX=${AFLPATH}/afl-g++
23+export AFL_HARDEN=1

fanquake commented at 7:49 am on November 23, 2016:

For anyone wondering what AFL_HARDEN does. From the afl README.

0Setting AFL_HARDEN=1 when calling 'make' will cause the CC wrapper to
1automatically enable code hardening options that make it easier to detect
2simple memory bugs. Libdislocator, a helper library included with AFL (see
3libdislocator/README.dislocator) can help uncover heap corruption issues, too.

laanwj commented at 7:54 am on November 23, 2016:

It may actually be unnecessary with bitcoin core which already enables a lot of hardening options. Then again, it can’t hurt either I think.

fanquake commented at 1:09 pm on November 24, 2016: member

Started testing this on OS X 10.12. afl compiles fine, and bitcoin configure completes (using afl-clang and afl-clang++ instead of afl-gcc and afl-gcc++):

 0Options used to compile and link:
 1  with wallet   = yes
 2  with gui / qt = yes
 3    qt version  = 5
 4    with qr     = yes
 5  with zmq      = yes
 6  with test     = yes
 7  with bench    = yes
 8  with upnp     = yes
 9  debug enabled = no
10
11  target os     = darwin
12  build os      = darwin
13
14  CC            = /Users/xxx/github/bitcoin/afl-2.35b/afl-clang
15  CFLAGS        = -g -O2
16  CPPFLAGS      =  -DHAVE_BUILD_INFO -D__STDC_FORMAT_MACROS -I/usr/local/opt/berkeley-db4/include -DMAC_OSX
17  CXX           = /Users/xxx/github/bitcoin/afl-2.35b/afl-clang++ -std=c++11
18  CXXFLAGS      = -g -O2 -Wall -Wextra -Wformat -Wformat-security -Wno-unused-parameter -Wno-self-assign -Wno-unused-local-typedef -Wno-deprecated-register
19  LDFLAGS       =  -Wl,-headerpad_max_install_names -Wl,-dead_strip

test_bitcoin_fuzzy compiles with one warning

 0bash-3.2$ make test/test_bitcoin_fuzzy
 1  CXX      test/test_test_bitcoin_fuzzy-test_bitcoin_fuzzy.o
 2afl-cc 2.35b by <lcamtuf@google.com>
 3afl-as 2.35b by <lcamtuf@google.com>
 4[+] Instrumented 2777 locations (64-bit, hardened mode, ratio 100%).
 5  CXX      lib/libunivalue_la-univalue.lo
 6... SNIP ...
 7afl-cc 2.35b by <lcamtuf@google.com>
 8afl-as 2.35b by <lcamtuf@google.com>
 9[!] WARNING: No instrumentation targets found.
10  CXX      libbitcoin_util_a-random.o
11... SNIP ...
12afl-cc 2.35b by <lcamtuf@google.com>
13afl-as 2.35b by <lcamtuf@google.com>
14[+] Instrumented 48 locations (64-bit, hardened mode, ratio 100%).
15  AR       crypto/libbitcoin_crypto.a
16  CC       src/libsecp256k1_la-secp256k1.lo
17afl-cc 2.35b by <lcamtuf@google.com>
18afl-as 2.35b by <lcamtuf@google.com>
19[+] Instrumented 903 locations (64-bit, hardened mode, ratio 100%).
20  CCLD     libsecp256k1.la
21  CXXLD    test/test_bitcoin_fuzzy

However, running gave:

 0bash-3.2$ $AFLPATH/afl-fuzz -i ${AFLIN} -o ${AFLOUT} -- test/test_bitcoin_fuzzy
 1afl-fuzz 2.35b by <lcamtuf@google.com>
 2[+] You have 8 CPU cores and 3 runnable tasks (utilization: 38%).
 3[+] Try parallel jobs - see /usr/local/share/doc/afl/parallel_fuzzing.txt.
 4[*] Setting up output directories...
 5[+] Output directory exists but deemed OK to reuse.
 6[*] Deleting old session data...
 7[+] Output dir cleanup successful.
 8[*] Scanning '/Users/xxx/github/bitcoin/src/inputs'...
 9[+] No auto-generated dictionary tokens to reuse.
10[*] Creating hard links for all input files...
11[*] Validating target binary...
12[*] Attempting dry run with 'id:000000,orig:auto_000000'...
13[*] Spinning up the fork server...
14
15[-] Hmm, looks like the target binary terminated before we could complete a
16    handshake with the injected code. There are two probable explanations:
17
18    - The current memory limit (50.0 MB) is too restrictive, causing an OOM
19      fault in the dynamic linker. This can be fixed with the -m option. A
20      simple way to confirm the diagnosis may be:
21
22[-] PROGRAM ABORT : Fork server handshake failed
23         Location : init_forkserver(), afl-fuzz.c:2247

Same result even after upping the memory limit to 4GB (-m4096) which should be more than enough.

Setting AFL_NO_FORKSRV=1 seems to have made it run. From the readme:

0Setting AFL_NO_FORKSRV disables the forkserver optimization, reverting to
1    fork + execve() call for every tested input. This is useful mostly when
2    working with unruly libraries that create threads or do other crazy
3    things when initializing (before the instrumentation has a chance to run).

So for now it’s running: screen shot 1

Also testing on Ubuntu 16.04 running in VirtualBox.

 0Options used to compile and link:
 1  with wallet   = yes
 2  with gui / qt = yes
 3    qt version  = 5
 4    with qr     = auto
 5  with zmq      = yes
 6  with test     = yes
 7  with bench    = yes
 8  with upnp     = yes
 9  debug enabled = no
10
11  target os     = linux
12  build os      = 
13
14  CC            = /home/ubuntu/bitcoin/afl-2.35b/afl-gcc
15  CFLAGS        = -g -O2
16  CPPFLAGS      =  -DHAVE_BUILD_INFO -D__STDC_FORMAT_MACROS
17  CXX           = /home/ubuntu/bitcoin/afl-2.35b/afl-g++ -std=c++11
18  CXXFLAGS      = -g -O2 -Wall -Wextra -Wformat -Wformat-security -Wno-unused-parameter
19  LDFLAGS       =

screen shot

fanquake commented at 1:37 pm on November 24, 2016: member

Given that one was working alright, I’ve moved to running five in parallel. Will leave them running for a while. You can use the afl-whatsup tool to watch the progress, sample output below:

 0$AFLPATH/afl-whatsup sync_dir
 1status check tool for afl-fuzz by <lcamtuf@google.com>
 2
 3Individual fuzzers
 4==================
 5
 6>>> fuzzer01 (1 days, 0 hrs) <<<
 7
 8  cycle 1, lifetime speed 159 execs/sec, path 744/1201 (61%)
 9  pending 52/1043, coverage 6.22%, no crashes yet
10
11>>> fuzzer02 (1 days, 0 hrs) <<<
12
13  cycle 80, lifetime speed 179 execs/sec, path 346/1199 (28%)
14  pending 0/0, coverage 6.22%, no crashes yet
15
16>>> fuzzer03 (1 days, 0 hrs) <<<
17
18  cycle 80, lifetime speed 181 execs/sec, path 504/1190 (42%)
19  pending 0/0, coverage 6.22%, no crashes yet
20
21>>> fuzzer04 (1 days, 0 hrs) <<<
22
23  cycle 83, lifetime speed 181 execs/sec, path 1180/1190 (99%)
24  pending 0/0, coverage 6.22%, no crashes yet
25
26>>> fuzzer05 (0 days, 1 hrs) <<<
27
28  cycle 3, lifetime speed 146 execs/sec, path 621/1206 (51%)
29  pending 0/687, coverage 6.22%, no crashes yet
30
31Summary stats
32=============
33
34       Fuzzers alive : 5
35      Total run time : 4 days, 4 hours
36         Total execs : 63 million
37    Cumulative speed : 846 execs/sec
38       Pending paths : 52 faves, 1730 total
39  Pending per fuzzer : 10 faves, 346 total (on average)
40       Crashes found : 0 locally unique

in doc/fuzzing.md: in 18b4c6cf21 outdated

33+
34+AFL needs an input directory with examples, and an output directory where it will place
35+examples that it found.
36+
37+```
38+mkdir inputs

MarcoFalke commented at 12:37 pm on November 25, 2016:

µnit:

0mkdir test/afl-inputs
1AFLIN=$PWD/test/afl-inputs
2...

fanquake commented at 12:52 pm on November 29, 2016:

nit: we could also add .gitignore entries for test/inputs, test/outputs and test/sync_dir

laanwj commented at 7:38 am on December 5, 2016:

Well, this is just an example. There’s no strong reason to make these directories inside the bitcoin repository.

MarcoFalke commented at 12:38 pm on November 25, 2016: member

Concept ACK 18b4c6c

in doc/fuzzing.md: in 18b4c6cf21 outdated

44+Fuzzing
45+--------
46+
47+To start the actual fuzzing use:
48+```
49+$AFLPATH/afl-fuzz -i ${AFLIN} -o ${AFLOUT} -- test/test_bitcoin_fuzzy

MarcoFalke commented at 2:01 pm on November 25, 2016:

I think the default memory needs to be increased to prevent OOM?

fanquake commented at 2:04 pm on November 25, 2016:

Which OS did you see out of memory errors on? What did you need to increase it to to make it run?

MarcoFalke commented at 2:22 pm on November 25, 2016:

I used the inputs by @laanwj (bitcoin_fuzzy_in.tar.xz) and the default of -m50 caused a failure in the dry run.

MarcoFalke commented at 2:26 pm on November 25, 2016:

Setting -m52 seems to be enough in this case. (OS: fedora 24)

laanwj commented at 2:14 pm on November 25, 2016: member

I didn’t have to change anything memory related on ubuntu at least

fanquake commented at 2:15 pm on November 25, 2016: member

Some results. The “Master” instance is yet to complete a cycle (~62%), and the exec speed seems to have dropped off in the last few hours. Current stats: screen shot 4

The other instances all have similar output, and have run through ~80cycles: screen shot 5

theuni commented at 8:39 pm on November 25, 2016: member

tested ACK. Works well on OSX with AFL_NO_FORKSRV set (probably worth documenting that it’s necessary there).

Also, feel free to take this as a simplification: https://github.com/theuni/bitcoin/commit/88b8f922c4b4a3db5d299cd64e0c9669269873a1

laanwj commented at 12:32 pm on November 26, 2016: member

@cfields thanks - though I’m not sure moving addrman to common is the right thing to do. It is decidedly a server thing and not used by any of the other (nontest) executables. And later on we may want to fuzz more things in _server.

pstratem commented at 5:03 am on November 27, 2016: contributor

@fanquake if you completely remove the memory limit there are a few tests which can use nearly unbounded amounts of memory

I suggest running with:

export AFL_SKIP_CRASHES=1; export AFL_PARAMETERS="-t 1000+ -m 1024";

afl-fuzz $AFL_PARAMETERS -i fuzzing/input -o fuzzing/output ./bitcoin/src/test/test_bitcoin_fuzzy

fanquake commented at 2:26 pm on November 27, 2016: member

@pstratem Thanks for the suggestions. I’ve restarted some of the fuzzers with the new vars.

fanquake commented at 12:49 pm on November 29, 2016: member

Wrapping up my testing of this. Results from individual fuzzers are below, and the sync_dir (outputs) as well as the inputs used are available for download. inputs.zip sync_dir.zip Fuzzer 01 (deterministic tests): fuzzer01 Fuzzer 02: fuzzer02 Fuzzer 03: fuzzer03 Fuzzer 04: fuzzer04 Fuzzer 05 (restarted to up allocated memory and pass -t 1000): fuzzer05 Total Output:

 0$AFLPATH/afl-whatsup sync_dir
 1status check tool for afl-fuzz by <lcamtuf@google.com>
 2
 3Individual fuzzers
 4==================
 5
 6>>> fuzzer01 (4 days, 23 hrs) <<<
 7
 8  cycle 2, lifetime speed 114 execs/sec, path 297/1222 (24%)
 9  pending 0/935, coverage 6.22%, no crashes yet
10
11>>> fuzzer02 (4 days, 23 hrs) <<<
12
13  cycle 376, lifetime speed 165 execs/sec, path 87/1220 (7%)
14  pending 0/0, coverage 6.22%, no crashes yet
15
16>>> fuzzer03 (4 days, 23 hrs) <<<
17
18  cycle 379, lifetime speed 166 execs/sec, path 114/1211 (9%)
19  pending 0/0, coverage 6.22%, no crashes yet
20
21>>> fuzzer04 (4 days, 23 hrs) <<<
22
23  cycle 388, lifetime speed 166 execs/sec, path 1030/1211 (85%)
24  pending 0/0, coverage 6.22%, no crashes yet
25
26>>> fuzzer05 (2 days, 4 hrs) <<<
27
28  cycle 161, lifetime speed 162 execs/sec, path 62/1227 (5%)
29  pending 0/0, coverage 6.22%, no crashes yet
30
31Summary stats
32=============
33
34       Fuzzers alive : 5
35      Total run time : 22 days, 1 hours
36         Total execs : 294 million
37    Cumulative speed : 773 execs/sec
38       Pending paths : 0 faves, 935 total
39  Pending per fuzzer : 0 faves, 187 total (on average)
40       Crashes found : 0 locally unique

sipa commented at 1:12 am on December 1, 2016: member

Concept ACK

MarcoFalke commented at 7:31 pm on December 2, 2016: member

I think this ready for merge and we should select and add appropriate afl-inputs in another pull.

(Maybe fix the doc nits before merge?)

laanwj commented at 6:42 am on December 3, 2016: member

I think this ready for merge and we should select and add appropriate afl-inputs in another pull.

I don’t think we should put the inputs in the repository. But I’ll just add a link where they can be downloaded. Will fix the doc nits.

pstratem commented at 8:44 pm on December 4, 2016: contributor

My current test cases are here http://strateman.ninja/fuzzing.tar.xz

laanwj force-pushed on Dec 5, 2016

Simple fuzzing framework a4153e20ec

doc: Add bare-bones documentation for fuzzing 8b15434b59

in src/test/test_bitcoin_fuzzy.cpp: in 8e0550fd38 outdated

93+        case CTRANSACTION_DESERIALIZE:
94+        {
95+            try
96+            {
97+                CTransaction tx;
98+                ds >> tx;

MarcoFalke commented at 10:41 am on December 5, 2016:

0error: ‘class CTransaction’ has no member named ‘Unserialize’

laanwj commented at 11:41 am on December 5, 2016:

right, this should probably be converted to CMutableTransaction

laanwj force-pushed on Dec 15, 2016

laanwj commented at 3:54 pm on December 15, 2016: member

Switched to the new way of transaction deserialization. This should be ready for merge now.

laanwj merged this on Dec 15, 2016

laanwj closed this on Dec 15, 2016

laanwj referenced this in commit 5bc209c73f on Dec 15, 2016

sipa commented at 5:21 pm on December 15, 2016: member

utACK 8b15434b59c6cd7368b4db680544cd77ed337bd3 (apart from #9354).

PastaPastaPasta referenced this in commit f55cf17040 on Aug 24, 2019

barrystyle referenced this in commit 9c4dc3a720 on Jan 22, 2020

random-zebra referenced this in commit 44b5327e61 on May 28, 2021

DrahtBot locked this on Sep 8, 2021

Resurrect pstratem’s “Simple fuzzing framework” #9172