[tests] Speed up fuzzing by ~200x when using afl-fuzz

practicalswift commented at 10:54 am on May 17, 2017: contributor

Enable the afl-clang-fast++ features deferred forkserver (__AFL_INIT) and persistent mode (__AFL_LOOP(1000)).

Before this patch:

 0$ afl-fuzz -i input -o output -m512 -- src/test/test_bitcoin_fuzzy
 1[*] Validating target binary...
 2[!] WARNING: The target binary is pretty slow! See /usr/local/share/doc/afl/perf_tips.txt.
 3[+] Here are some useful stats:
 4
 5    Test case count : 1 favored, 0 variable, 1 total
 6       Bitmap range : 1072 to 1072 bits (average: 1072.00 bits)
 7        Exec timing : 20.4k to 20.4k us (average: 20.4k us)
 8…
 9exec speed : 57.58/sec (slow!)
10exec speed : 48.35/sec (slow!)
11exec speed : 53.78/sec (slow!)

After this patch:

 0$ afl-fuzz -i input -o output -m512 -- src/test/test_bitcoin_fuzzy
 1[*] Validating target binary...
 2[+] Persistent mode binary detected.
 3[+] Deferred forkserver binary detected.
 4[+] Here are some useful stats:
 5
 6    Test case count : 1 favored, 0 variable, 1 total
 7       Bitmap range : 24 to 24 bits (average: 24.00 bits)
 8        Exec timing : 114 to 114 us (average: 114 us)
 9…
10exec speed : 15.9k/sec
11exec speed : 13.1k/sec
12exec speed : 15.1k/sec

For more fuzzing discussions, see #10364.

fanquake added the label Tests on May 17, 2017

practicalswift force-pushed on May 17, 2017

practicalswift renamed this:
~~[fuzz] Speed up fuzzing by ~200x when using afl-fuzz~~
[test] Speed up fuzzing by ~200x when using afl-fuzz
on May 17, 2017

practicalswift force-pushed on May 17, 2017

laanwj commented at 11:06 am on May 17, 2017: member

Interesting. Concept ACK.

practicalswift renamed this:
~~[test] Speed up fuzzing by ~200x when using afl-fuzz~~
[tests] Speed up fuzzing by ~200x when using afl-fuzz
on May 18, 2017

fanquake commented at 8:44 am on May 18, 2017: member

I’m currently seeing no speedup when using this new code. I’m wondering if it’s because I have to run with AFL_NO_FORKSRV on OS X (I haven’t been able to make afl-fuzz run without it being set) ?

Or does this require using afl-clang-fast and afl-clang-fast++ in place of afl-clang and afl-clang++ to see the speed improvements?

I’m seeing:

 0[*] Attempting dry run with 'id:000566,orig:id:000958,sync:39,src:000955'...
 1    len = 7368, map size = 1430, exec speed = 15345 us
 2[+] All test cases processed.
 3
 4[!] WARNING: The target binary is pretty slow! See /usr/local/Cellar/afl-fuzz/2.41b/share/doc/afl/perf_tips.txt.
 5[+] Here are some useful stats:
 6
 7    Test case count : 141 favored, 4 variable, 567 total
 8       Bitmap range : 995 to 1684 bits (average: 1240.06 bits)
 9        Exec timing : 13.9k to 90.9k us (average: 15.6k us)
10
11[*] No -t option specified, so I'll use exec timeout of 100 ms.
12[+] All set and ready to roll!

practicalswift commented at 9:08 am on May 18, 2017: contributor

@fanquake The binary you created is not instrumented correctly (the messages Persistent mode binary detected and Deferred forkserver binary detected are missing). You’re correct that the instrumentation needed for this speedup requires using afl-clang-fast and afl-clang-fast++ in place of afl-clang and afl-clang++. See https://github.com/mirrorer/afl/tree/master/llvm_mode for details :-)

practicalswift force-pushed on May 18, 2017

fanquake commented at 0:06 am on May 19, 2017: member

In that case, the code changes should be accompanied by a new? section in the fuzzing documentation, outlining how to setup and run the much faster fuzzing. I’ll retest this shortly.

[test] Speed up fuzzing by ~200x when using afl-fuzz

Enable the `afl-clang-fast++` features deferred forkserver (`__AFL_INIT`) and persistent mode (`__AFL_LOOP(1000)`).

Before this patch:

```
$ afl-fuzz -i input -o output -m512 -- src/test/test_bitcoin_fuzzy
[*] Validating target binary...
[!] WARNING: The target binary is pretty slow! See /usr/local/share/doc/afl/perf_tips.txt.
[+] Here are some useful stats:

    Test case count : 1 favored, 0 variable, 1 total
       Bitmap range : 1072 to 1072 bits (average: 1072.00 bits)
        Exec timing : 20.4k to 20.4k us (average: 20.4k us)
…
exec speed : 57.58/sec (slow!)
exec speed : 48.35/sec (slow!)
exec speed : 53.78/sec (slow!)
```

After this patch:

```
$ afl-fuzz -i input -o output -m512 -- src/test/test_bitcoin_fuzzy
[*] Validating target binary...
[+] Persistent mode binary detected.
[+] Deferred forkserver binary detected.
[+] Here are some useful stats:

    Test case count : 1 favored, 0 variable, 1 total
       Bitmap range : 24 to 24 bits (average: 24.00 bits)
        Exec timing : 114 to 114 us (average: 114 us)
…
exec speed : 15.9k/sec
exec speed : 13.1k/sec
exec speed : 15.1k/sec
```

693247b82b

practicalswift force-pushed on May 19, 2017

practicalswift commented at 5:29 am on May 19, 2017: contributor

@fanquake Good point! Documentation and comments added. Looks good? :-)

practicalswift commented at 12:33 pm on May 22, 2017: contributor

Related pull request: #10440 (“Add libFuzzer support”).

laanwj commented at 2:52 pm on May 22, 2017: member

@fanquake thanks for testing @practicalswift thanks for adding documentation utACK https://github.com/bitcoin/bitcoin/pull/10415/commits/693247b82bbccada37f7bc61380ddac799100eb8

laanwj merged this on May 22, 2017

laanwj closed this on May 22, 2017

laanwj referenced this in commit 217b416c72 on May 22, 2017

PastaPastaPasta referenced this in commit 122da986c1 on Aug 24, 2019

barrystyle referenced this in commit bd007fe64c on Jan 22, 2020

practicalswift deleted the branch on Apr 10, 2021

random-zebra referenced this in commit 44b5327e61 on May 28, 2021

DrahtBot locked this on Aug 18, 2022

[tests] Speed up fuzzing by ~200x when using afl-fuzz #10415