doc: Improve fuzzing docs for macOS users #17942

pull fjahr wants to merge 1 commits into bitcoin:master from fjahr:fuzzing_docs changing 1 files +58 −17
  1. fjahr commented at 4:54 pm on January 16, 2020: member

    Adds several helpful hints for macOS users trying to get fuzzers to run locally using AFL or libFuzzer. These are partly based on this comment #17657 (comment) and discussions in the review club for #17860. See: https://bitcoincore.reviews/17860.html

    Based on the doc in the current state I could not compile fuzzers for AFL or libFuzzer. Using these hints, I can

    • compile and run fuzzers with AFL
    • compile but not run fuzzers with libFuzzer

    Fuzzers compiled with libFuzzers may be running but don’t produce any output. Looking for others to test this to see if it is an issue with my local system. Especially interesting if you have been running libFuzzer fuzzers successfully on macOS before.

    Edit: Closes #17914

  2. DrahtBot added the label Docs on Jan 16, 2020
  3. laanwj added the label macOS on Jan 16, 2020
  4. in doc/fuzzing.md:65 in 76f589d6b8 outdated 
    56@@ -51,6 +57,10 @@ export AFL_HARDEN=1
57 cd src/
58 make
59 ```
60+
61+If you are using clang you will need to substitute `afl-gcc` with `afl-clang`
62+and `afl-g++` with `afl-clang++`.
63+
    eriknylund commented at 7:57 pm on January 16, 2020:

    If you are using clang you will need to substitute afl-gcc with afl-clang and afl-g++ with afl-clang++ so the first line above becomes:

    0./configure --disable-ccache --disable-shared --enable-tests --enable-fuzz CC=${AFLPATH}/afl-clang CXX=${AFLPATH}/afl-clang++
    fjahr commented at 11:11 pm on January 23, 2020:
    done
  5. in doc/fuzzing.md:95 in 76f589d6b8 outdated 
    84@@ -75,10 +85,15 @@ $AFLPATH/afl-fuzz -i ${DIR_FUZZ_IN}/${FUZZ_TARGET} -o ${AFLOUT}/${FUZZ_TARGET} -
85 You may have to change a few kernel parameters to test optimally - `afl-fuzz`
86 will print an error and suggestion if so.
87 
88+On macOS you may need to set `AFL_NO_FORKSRV=1` to get the target to run.
89+```
90+AFL_NO_FORKSRV=1 $AFLPATH/afl-fuzz -i ${DIR_FUZZ_IN}/${FUZZ_TARGET} -o ${AFLOUT}/${FUZZ_TARGET} -m52 -- test/fuzz/${FUZZ_TARGET}
    eriknylund commented at 8:06 pm on January 16, 2020:
    I seem to be missing the ${FUZZ_TARGET} declaration. The comment below contains the two rows from the similar section at https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md
    eriknylund commented at 8:36 pm on January 20, 2020: 
    0export FUZZ_TARGET=fuzz_target_foo  # Pick a fuzz_target from ${DIR_FUZZ_IN}
1mkdir ${AFLOUT}/${FUZZ_TARGET}
2AFL_NO_FORKSRV=1 $AFLPATH/afl-fuzz -i ${DIR_FUZZ_IN}/${FUZZ_TARGET} -o ${AFLOUT}/${FUZZ_TARGET} -m52 -- test/fuzz/${FUZZ_TARGET}
    fjahr commented at 11:09 pm on January 23, 2020:
    done
    fjahr commented at 11:11 pm on January 23, 2020:
    done
  6. MarcoFalke commented at 8:20 pm on January 16, 2020: member
    Thanks. Much needed!
  7. eriknylund changes_requested
  8. in doc/fuzzing.md:122 in 76f589d6b8 outdated 
    115+
116+The default clang/llvm version supplied by Apple on MacOS does not include
117+fuzzing libraries, so macOS users will need to install a full version, for
118+example using `brew install llvm`.
119+
120+When executing ./configure you should put --disable-asm to avoid errors
    fanquake commented at 7:04 am on January 17, 2020:
    What errors are you seeing with the asm? Can you post logs?
    fjahr commented at 3:46 pm on January 17, 2020:
    Hm, I am sure I had an issue without it before but I can not reproduce it right now and I did not keep the logs since I tried libFuzzer one week ago. So it is possible that it is not an issue anymore or I simply mistook this as fixing an unrelated issue. However, the issue is also noted in the docs on sanitizers: https://github.com/bitcoin/bitcoin/blob/master/doc/developer-notes.md#sanitizers
  9. in doc/fuzzing.md:129 in 76f589d6b8 outdated 
    122+
123+You may also need to take care of giving the correct path for clang and
124+clang++, like `CC=/path/to/clang CXX=/path/to/clang++` if the non-systems
125+clang does not come first in your path.
126+
127+If you run into problems with "boost sleep" or some of boost's libraries
    fanquake commented at 7:04 am on January 17, 2020:
    What errors are you seeing with Boost? Can you post logs?
    fjahr commented at 3:42 pm on January 17, 2020:
    I have not seen this particular error myself but included it for others since it seems to have been an issue here: #17657 (comment). So I can remove it or improve it as you describe in the comment on the flags. Or maybe @brakmic can help (sorry for including you unsolicited).
    brakmic commented at 9:50 am on January 18, 2020:
    I have seen such errors when experimenting with libFuzzer and after some search through GitHub I found a solution to a similar problem: macOS should be provided an isysroot variable that points to the proper SDK. However, this could maybe only be a problem when working with macOs Catalina. I’ve never tested it under older variants. Catalina, in many ways, is a “breaking change” regarding handling of SDK paths, include directories etc.
    fjahr commented at 11:11 pm on January 23, 2020:
    Updated to use xcrun –show-sdk-path.
  10. in doc/fuzzing.md:128 in 76f589d6b8 outdated 
    126+
127+If you run into problems with "boost sleep" or some of boost's libraries
128+can't be found, like boost.thread or boost.filesystem, add this to your
129+configure:
130+```
131+CXXFLAGS="-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk"
    fanquake commented at 7:17 am on January 17, 2020:

    If this is needed, we should be using something like $(xcrun --show-sdk-path) (same as we do in depends), rather than hard-coding a path to a specific SDK version.

    Although I’m curious to know why this is would be required, as it’s already added to the compiler flags by clang. i.e:

    0/usr/local/Cellar/llvm/9.0.0_1/bin/clang++ test.cpp -o test -v
1...
2"/usr/local/Cellar/llvm/9.0.0_1/bin/clang-9" 
3-cc1 
4-triple x86_64-apple-macosx10.14.0 
5< snip a bunch of args> 
6-target-linker-version 512.4 
7-v -resource-dir /usr/local/Cellar/llvm/9.0.0_1/lib/clang/9.0.0 
8-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
9``
    brakmic commented at 9:54 am on January 18, 2020:
    The sysroot there is pointing to /Library/Developer which is not correct when using under Catalina (but could be correct under previous macOS versions). Here a link to a question from SO regarding this particular problem: https://stackoverflow.com/questions/58278260/cant-compile-a-c-program-on-a-mac-after-upgrading-to-catalina-10-15
    fanquake commented at 1:02 am on January 20, 2020:

    The sysroot there is pointing to /Library/Developer which is not correct when using under Catalina (but could be correct under previous macOS versions). @brakmic Sure, however that’s why we’d want to use xcrun --show-sdk-path, because it should point to the right location for the SDK regardless of the OS it’s being run on. What does it output when you run it on Catalina?

  11. fanquake commented at 7:17 am on January 17, 2020: member
    Concept ACK
  12. jonasschnelli commented at 4:11 am on January 20, 2020: contributor
    Thanks! utACK 76f589d6b8a4d74735f3987fdf054eaad662a2a0
  13. fjahr force-pushed on Jan 23, 2020
  14. fjahr commented at 11:14 pm on January 23, 2020: member
    Updated to include feedback by @eriknylund and @fanquake .
  15. fjahr force-pushed on Jan 28, 2020
  16. fjahr commented at 11:16 am on January 28, 2020: member
    With the fix in #18008 the fuzzers compiled with libFuzzer run for me now. I did not need to specify the system path or --disable-asm. After looking at some of the build code I agree with @fanquake that the system path should not be necessary in any case, so I removed it. But I left --disable-asm and linked to the mention of the problem in the developer notes. I also made some minor editing changes.
  17. in doc/fuzzing.md:133 in 465ea82787 outdated 
    131+clang++, like `CC=/path/to/clang CXX=/path/to/clang++` if the non-systems
132+clang does not come first in your path.
133+
134+Full configure that was tested on macOS Catalina with `brew` installed `llvm`:
135+```
136+./configure --disable-ccache --enable-fuzz --with-sanitizers=fuzzer,address,undefined --with-boost CC=/usr/local/opt/llvm/bin/clang CXX=/usr/local/opt/llvm/bin/clang++ --disable-asm
    fanquake commented at 12:55 pm on January 28, 2020:
    You should be able to drop --with-boost here.
    fjahr commented at 4:10 pm on January 28, 2020:
    done (also tested it)
  18. in doc/fuzzing.md:48 in 465ea82787 outdated 
    43@@ -41,6 +44,9 @@ make
44 export AFLPATH=$PWD
45 ```
46 
47+For macOS you may need to ignore x86 compilation checks when running `make`:
48+`AFL_NO_X86=1 make`.
    fanquake commented at 12:56 pm on January 28, 2020:
    Was this something you ran into, or also a suggestion from elsewhere in the docs?
    fjahr commented at 4:07 pm on January 28, 2020:

    I had a problem with this. I got this output without the flag:

     0$ make
 1[*] Checking for the ability to compile x86 code...
 2[+] Everything seems to be working, ready to compile.
 3cc -O3 -funroll-loops -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/local/lib/afl\" -DDOC_PATH=\"/usr/local/share/doc/afl\" -DBIN_PATH=\"/usr/local/bin\" afl-gcc.c -o afl-gcc
 4set -e; for i in afl-g++ afl-clang afl-clang++; do ln -sf afl-gcc $i; done
 5cc -O3 -funroll-loops -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/local/lib/afl\" -DDOC_PATH=\"/usr/local/share/doc/afl\" -DBIN_PATH=\"/usr/local/bin\" afl-fuzz.c -o afl-fuzz
 6cc -O3 -funroll-loops -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/local/lib/afl\" -DDOC_PATH=\"/usr/local/share/doc/afl\" -DBIN_PATH=\"/usr/local/bin\" afl-showmap.c -o afl-showmap
 7cc -O3 -funroll-loops -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/local/lib/afl\" -DDOC_PATH=\"/usr/local/share/doc/afl\" -DBIN_PATH=\"/usr/local/bin\" afl-tmin.c -o afl-tmin
 8cc -O3 -funroll-loops -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/local/lib/afl\" -DDOC_PATH=\"/usr/local/share/doc/afl\" -DBIN_PATH=\"/usr/local/bin\" afl-gotcpu.c -o afl-gotcpu
 9cc -O3 -funroll-loops -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/local/lib/afl\" -DDOC_PATH=\"/usr/local/share/doc/afl\" -DBIN_PATH=\"/usr/local/bin\" afl-analyze.c -o afl-analyze
10cc -O3 -funroll-loops -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/local/lib/afl\" -DDOC_PATH=\"/usr/local/share/doc/afl\" -DBIN_PATH=\"/usr/local/bin\" afl-as.c -o afl-as
11ln -sf afl-as as
12[*] Testing the CC wrapper and instrumentation output...
13unset AFL_USE_ASAN AFL_USE_MSAN; AFL_QUIET=1 AFL_INST_RATIO=100 AFL_PATH=. ./afl-clang -O3 -funroll-loops -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/local/lib/afl\" -DDOC_PATH=\"/usr/local/share/doc/afl\" -DBIN_PATH=\"/usr/local/bin\" test-instr.c -o test-instr
14echo 0 | ./afl-showmap -m none -q -o .test-instr0 ./test-instr
15echo 1 | ./afl-showmap -m none -q -o .test-instr1 ./test-instr
16
17Oops, the instrumentation does not seem to be behaving correctly!
18
19Please ping <lcamtuf@google.com> to troubleshoot the issue.
20
21make: *** [test_build] Error 1
  19. fanquake commented at 12:56 pm on January 28, 2020: member
    Thanks for following up again. This is looking pretty good now 👍.
  20. fjahr force-pushed on Jan 28, 2020
  21. in doc/fuzzing.md:63 in 8126f5e6f1 outdated 
    56@@ -51,6 +57,13 @@ export AFL_HARDEN=1
57 cd src/
58 make
59 ```
60+
61+If you are using clang you will need to substitute `afl-gcc` with `afl-clang`
62+and `afl-g++` with `afl-clang++`, so the first line above becomes:
63+```
64+./configure --disable-ccache --disable-shared --enable-tests --enable-fuzz CC=${AFLPATH}/afl-clang CXX=${AFLPATH}/afl-clang++
    Sjors commented at 4:11 pm on January 28, 2020:
    Maybe use afl-clang-fast here? It’s a bit weird that this is recommended below. Unless there’s some downside.
    fjahr commented at 11:40 pm on January 28, 2020:
    My takeaway from the docs here is that afl-clang-fast is not as stable as afl-clang and that it will become afl-clang once it has become stable enough, at which point afl-clang-fast would probably be removed. So I thought keeping afl-clang would be the better option to keep as the default.
    Sjors commented at 8:09 am on January 29, 2020:
    I see. Might be useful to mention that potential downside below.
  22. in doc/fuzzing.md:93 in 8126f5e6f1 outdated 
    87@@ -75,10 +88,17 @@ $AFLPATH/afl-fuzz -i ${DIR_FUZZ_IN}/${FUZZ_TARGET} -o ${AFLOUT}/${FUZZ_TARGET} -
88 You may have to change a few kernel parameters to test optimally - `afl-fuzz`
89 will print an error and suggestion if so.
90 
91+On macOS you may need to set `AFL_NO_FORKSRV=1` to get the target to run.
92+```
93+export FUZZ_TARGET=fuzz_target_foo  # Pick a fuzz_target
    Sjors commented at 4:20 pm on January 28, 2020:
    Please add an example here, e.g. FUZZ_TARGET=bech32
  23. in doc/fuzzing.md:95 in 8126f5e6f1 outdated 
    87@@ -75,10 +88,17 @@ $AFLPATH/afl-fuzz -i ${DIR_FUZZ_IN}/${FUZZ_TARGET} -o ${AFLOUT}/${FUZZ_TARGET} -
88 You may have to change a few kernel parameters to test optimally - `afl-fuzz`
89 will print an error and suggestion if so.
90 
91+On macOS you may need to set `AFL_NO_FORKSRV=1` to get the target to run.
92+```
93+export FUZZ_TARGET=fuzz_target_foo  # Pick a fuzz_target
94+mkdir ${AFLOUT}/${FUZZ_TARGET}
95+AFL_NO_FORKSRV=1 $AFLPATH/afl-fuzz -i ${DIR_FUZZ_IN}/${FUZZ_TARGET} -o ${AFLOUT}/${FUZZ_TARGET} -m52 -- test/fuzz/${FUZZ_TARGET}
    Sjors commented at 4:32 pm on January 28, 2020:
    I usually run command from the project root, not from src, so maybe use src/test/fuzz?
    fjahr commented at 11:40 pm on January 28, 2020:
    done, for some reason the guide had a cd src/ somewhere at the top and the rest of the paths were kept consistent with it. I removed the cd and made the rest of the paths consistent.
  24. Sjors commented at 4:33 pm on January 28, 2020: member
    I was able to follow along with the AFL instructions.
  25. in doc/fuzzing.md:21 in 8126f5e6f1 outdated 
    17@@ -21,13 +18,19 @@ git clone https://github.com/bitcoin-core/qa-assets
18 export DIR_FUZZ_IN=$PWD/qa-assets/fuzz_seed_corpus
19 ```
20 
21-Only for AFL:
22+AFL needs an input directory with examples, and an output directory where it
    Sjors commented at 4:36 pm on January 28, 2020:
    While you’re at it, maybe clarify that the git clone and export above need to happen in src/test/fuzz
    fjahr commented at 11:40 pm on January 28, 2020:
    I don’t think that’s the case. I have my qa-assets folder in the same parent directory as the bitcoin source and it seems to run fine.
    Sjors commented at 8:10 am on January 29, 2020:
    You’re right
  26. in doc/fuzzing.md:124 in 8126f5e6f1 outdated 
    118+
119+### macOS hints for libFuzzer
120+
121+The default clang/llvm version supplied by Apple on macOS does not include
122+fuzzing libraries, so macOS users will need to install a full version, for
123+example using `brew install llvm`.
    Sjors commented at 4:43 pm on January 28, 2020:

    Which version did you test this with? I started with an earlier install of 9.0.0 which failed with:

    0checking whether we are cross compiling... configure: error: in `/Users/sjors/dev/bitcoin-libfuzzer':
1configure: error: cannot run C++ compiled programs.
2If you meant to cross compile, use `--host'.

    But once I did brew upgrade llvm and bumped to 9.0.1 it worked.

    fanquake commented at 11:36 pm on January 28, 2020:
    This looks like a local issue. libFuzzer based fuzzing works fine with LLVM 8.x, 9.0.0 and 9.0.1. Looking in config.log would tell you the cause of the issue.
    fjahr commented at 11:40 pm on January 28, 2020:
    I don’t think I tested with a different version than 9.0.1, I had upgraded shortly before. Do you still have 9.0.0 installed and can reproduce it?
    Sjors commented at 8:06 am on January 29, 2020:
    It’s probably a local issue.
  27. in doc/fuzzing.md:119 in 8126f5e6f1 outdated 
    113 See https://llvm.org/docs/LibFuzzer.html#running on how to run the libFuzzer
114 instrumented executable.
115 
116 Alternatively run the script in `./test/fuzz/test_runner.py` and provide it
117 with the `${DIR_FUZZ_IN}` created earlier.
118+
    Sjors commented at 4:58 pm on January 28, 2020:
    ^ can you clarify what “provide it with” means? Something like ./test/fuzz/test_runner.py $DIR_FUZZ_IN is more clear. Also point out that you then need to run one of the executable files in src/test/fuzz
    fjahr commented at 11:39 pm on January 28, 2020:
    yeah, that was not very clear, I reworded the whole paragraph.
    fanquake commented at 11:57 pm on January 28, 2020:

    Also point out that you then need to run one of the executable files in src/test/fuzz

    This isn’t actually required. If you don’t pick a specific fuzz target all of them will be run. However we can leave as is.

    Sjors commented at 8:22 am on January 29, 2020:
    Maybe add -l DEBUG so you can see it’s running.
    fjahr commented at 12:05 pm on January 29, 2020:
    Hm, I will save the ACKs now but if it’s confusing to run the tests without -l DEBUG then maybe the default log level should be changed or there should be an additional INFO level output like The test is running....
  28. Sjors commented at 5:11 pm on January 28, 2020: member
    I think I’m also able to run with libFuzzer (with #18008).
  29. fjahr force-pushed on Jan 28, 2020
  30. in doc/fuzzing.md:75 in 24da7c1619 outdated 
    71@@ -60,25 +72,32 @@ The fuzzing can be sped up significantly (~200x) by using `afl-clang-fast` and
72 compiling using `afl-clang-fast`/`afl-clang-fast++` the resulting
73 binary will be instrumented in such a way that the AFL
74 features "persistent mode" and "deferred forkserver" can be used. See
75-https://github.com/mcarpenter/afl/tree/master/llvm_mode for details.
76+https://github.com/mirrorer/afl/tree/master/llvm_mode for details.
    fanquake commented at 11:44 pm on January 28, 2020:
    I think this should be https://github.com/google/AFL
    fjahr commented at 11:46 pm on January 28, 2020:
    done
  31. doc: Improve fuzzing docs for macOS users b6c3e84e87
  32. fjahr force-pushed on Jan 28, 2020
  33. Sjors approved
  34. Sjors commented at 8:28 am on January 29, 2020: member

    ACK b6c3e84

    Nit: maybe add qa-assets/ to .gitignore

    From the other thread, @fanquake wrote:

    With this branch it just exits after Fuzz targets selected: [‘bech32’]

    @Sjors It doesn’t just exit, the bech32 fuzzer is run. However the log level defaults to INFO, which doesn’t generate any output. Pass -l DEBUG and you’ll see:

    I’m a bit surprised by how fast it completes.

  35. in doc/fuzzing.md:60 in b6c3e84e87 
    56 export AFL_HARDEN=1
57-cd src/
58 make
59 ```
60+
61+If you are using clang you will need to substitute `afl-gcc` with `afl-clang`
    Sjors commented at 8:40 am on January 29, 2020:
    nit: using clang, e.g. on macOS, you will
  36. fanquake approved
  37. fanquake commented at 11:54 am on January 29, 2020: member
    ACK b6c3e84e87055be311347d7b636d68a6a828f563 - I think this has been nitpicked enough, and importantly the commands look better now.
  38. MarcoFalke commented at 1:44 pm on January 29, 2020: member
    ACK b6c3e84e87055be311347d7b636d68a6a828f563
  39. MarcoFalke referenced this in commit 3b5b276734 on Jan 29, 2020
  40. MarcoFalke merged this on Jan 29, 2020
  41. MarcoFalke closed this on Jan 29, 2020
  42. Fabcien referenced this in commit 4ffa90d363 on Jan 19, 2021
  43. random-zebra referenced this in commit 44b5327e61 on May 28, 2021
  44. kittywhiskers referenced this in commit 2058b68f8a on Aug 2, 2021
  45. kittywhiskers referenced this in commit 5d760f8dfe on Aug 5, 2021
  46. kittywhiskers referenced this in commit cf78627d3f on Aug 5, 2021
  47. kittywhiskers referenced this in commit f2563c7164 on Aug 5, 2021
  48. PastaPastaPasta referenced this in commit a9ccd9c6d2 on Aug 6, 2021
  49. kittywhiskers referenced this in commit 61b3680920 on Aug 8, 2021
  50. kittywhiskers referenced this in commit 45d0b240b8 on Aug 11, 2021
  51. 5tefan referenced this in commit 783ca78a19 on Aug 12, 2021
  52. DrahtBot locked this on Feb 15, 2022


fjahr eriknylund MarcoFalke fanquake brakmic jonasschnelli Sjors

Labels
Docs macOS

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-10-04 22:12 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me