build: macOS fuzz instructions broken using latest macOS linker #31049

Testing master at 62e4516722115c2d5aeb6c197abc73ca7c078b23 and the fuzzing.md instructions:

cmake --preset=libfuzzer \   
   -DCMAKE_C_COMPILER="$(brew --prefix llvm)/bin/clang" \
   -DCMAKE_CXX_COMPILER="$(brew --prefix llvm)/bin/clang++" \
   -DAPPEND_LDFLAGS=-Wl,-no_warn_duplicate_libraries
cmake --build build_fuzz
<snip>
[100%] Linking CXX executable fuzz
ld: multiple errors: invalid r_symbolnum=1 in '/Users/michael/fanquake-bitcoin/build_fuzz/src/test/fuzz/CMakeFiles/fuzz.dir/addition_overflow.cpp.o'; invalid r_symbolnum=1 in '/Users/michael/fanquake-bitcoin/build_fuzz/src/test/fuzz/CMakeFiles/fuzz.dir/fees.cpp.o'; invalid r_symbolnum=1 in '/Users/michael/fanquake-bitcoin/build_fuzz/src/test/fuzz/CMakeFiles/fuzz.dir/float.cpp.o'; invalid r_symbolnum=1 in '/Users/michael/fanquake-bitcoin/build_fuzz/src/test/fuzz/CMakeFiles/fuzz.dir/multiplication_overflow.cpp.o'; invalid r_symbolnum=1 in '../../libbitcoin_cli.a[2](stdin.cpp.o)'; invalid r_symbolnum=1 in '../../../libcrc32c.a[3](crc32c_portable.cc.o)'; invalid r_symbolnum=1 in '../../../libcrc32c_arm64.a[2](crc32c_arm64.cc.o)'; invalid r_symbolnum=1 in '../../libbitcoin_consensus.a[11](script_error.cpp.o)'; invalid r_symbolnum=1 in '../../crypto/libbitcoin_crypto.a[15](sha3.cpp.o)'; invalid r_symbolnum=5 in '../../util/libbitcoin_util.a[29](randomenv.cpp.o)'; invalid r_symbolnum=1 in '../../crypto/libbitcoin_crypto.a[10](poly1305.cpp.o)'; invalid r_symbolnum=18 in '../../crypto/libbitcoin_crypto_arm_shani.a[2](sha256_arm_shani.cpp.o)'; invalid r_symbolnum=1 in '../../crypto/libbitcoin_crypto.a[5](hex_base.cpp.o)'; invalid r_symbolnum=1 in '../../util/libbitcoin_util.a[27](logging.cpp.o)'; invalid r_symbolnum=1 in '../../util/libbitcoin_util.a[24](threadnames.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_consensus.a[5](hash.cpp.o)'; invalid r_symbolnum=1 in '../../../libleveldb.a[37](logging.cc.o)'; invalid r_symbolnum=1 in '../../../libleveldb.a[35](hash.cc.o)'; invalid r_symbolnum=1 in '../../../libleveldb.a[31](crc32c.cc.o)'; invalid r_symbolnum=1 in '../../../libleveldb.a[27](bloom.cc.o)'; invalid r_symbolnum=1 in '../../util/libbitcoin_util.a[16](serfloat.cpp.o)'; invalid r_symbolnum=1 in '../../util/libbitcoin_util.a[15](readwritefile.cpp.o)'; invalid r_symbolnum=1 in '../../util/libbitcoin_util.a[14](rbf.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_common.a[47](parsing.cpp.o)'; invalid r_symbolnum=1 in '../../util/libbitcoin_util.a[9](feefrac.cpp.o)'; invalid r_symbolnum=1 in '../../util/libbitcoin_util.a[6](chaintype.cpp.o)'; invalid r_symbolnum=1 in '../../../libcrc32c.a[2](crc32c.cc.o)'; invalid r_symbolnum=1 in '../../../libleveldb.a[8](filename.cc.o)'; invalid r_symbolnum=1 in '../../../libleveldb.a[2](builder.cc.o)'; invalid r_symbolnum=1 in '../util/libtest_util.a[12](str.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_common.a[42](request.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_common.a[41](rawtransaction_util.cpp.o)'; invalid r_symbolnum=1 in '../util/libtest_util.a[4](index.cpp.o)'; invalid r_symbolnum=1 in '../util/libtest_util.a[3](coins.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[84](torcontrol.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[79](server_util.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_common.a[30](merkleblock.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_common.a[29](key_io.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_common.a[24](deploymentinfo.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_common.a[21](compressor.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_common.a[20](url.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_common.a[16](run_command.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[68](pow.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[63](fees_args.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[55](psbt.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[54](peerman_args.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[51](miner.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[50](mempool_persist_args.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[43](database_args.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[41](connection_types.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[40](coins_view_args.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[39](coin.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[36](caches.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[34](blockmanager_args.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[31](net_processing.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[28](mempool_removal_reason.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[23](checks.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[22](chain.cpp.o)'
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[2]: *** [src/test/fuzz/CMakeFiles/fuzz.dir/build.make:2186: src/test/fuzz/fuzz] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:1722: src/test/fuzz/CMakeFiles/fuzz.dir/all] Error 2

I'm not sure when this broke, but my assumption is that it's an issue with a newer version of ld. Used here is:

ld -v
@(#)PROGRAM:ld PROJECT:ld-1115.7.3
BUILD 13:29:00 Aug  9 2024
configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em
will use ld-classic for: armv6 armv7 armv7s i386 armv6m armv7k armv7m armv7em
LTO support using: LLVM version 16.0.0 (static support for 29, runtime is 29)
TAPI support using: Apple TAPI version 16.0.0 (tapi-1600.0.11.8)

I tried compiling with -Wl,-ld_classic, which should use the older verison of the linker, but it hit an assertion:

[100%] Linking CXX executable fuzz
ld: warning: -ld_classic is deprecated and will be removed in a future release
0  0x100483ee4  __assert_rtn + 160
1  0x100485804  ld::tool::LinkEditAtom::size() const (.cold.1) + 0
2  0x10035c200  ld::tool::OutputFile::addressOf(ld::Internal const&, ld::Fixup const*, ld::Atom const**) + 244
3  0x10036b58c  ld::tool::OutputFile::buildChainedFixupInfo(ld::Internal&) + 1196
4  0x1003702f4  ___ZN2ld4tool10OutputFile20buildLINKEDITContentERNS_8InternalE_block_invoke.408 + 28
5  0x191fc28f8  _dispatch_call_block_and_release + 32
6  0x191fc4658  _dispatch_client_callout + 20
7  0x191fd6570  _dispatch_root_queue_drain + 996
8  0x191fd6b20  _dispatch_worker_thread2 + 156
9  0x19217339c  _pthread_wqthread + 228
A linker snapshot was created at:
	/tmp/fuzz-2024-10-07-142750.ld-snapshot
ld: Assertion failed: (_mode == modeFinalAddress), function finalAddress, file ld.hpp, line 1462.
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

Does it also happen with llvm@16, instead of the current version?

It works with llvm@18, so it looks like this could be an incompatibility with LLVM 19 (19.1.1) and Apples ld.

Replicated on current master (0a931a9787b196d7a620863cc143d9319ffd356d) with Clang 19.1.7

[100%] Linking CXX executable fuzz
ld: multiple errors: invalid r_symbolnum=1 in '/Users/max/source/bitcoin/build_fuzz/src/test/fuzz/CMakeFiles/fuzz.dir/addition_overflow.cpp.o'; invalid r_symbolnum=1 in '/Users/max/source/bitcoin/build_fuzz/src/test/fuzz/CMakeFiles/fuzz.dir/float.cpp.o'; invalid r_symbolnum=1 in '/Users/max/source/bitcoin/build_fuzz/src/test/fuzz/CMakeFiles/fuzz.dir/fees.cpp.o'; invalid r_symbolnum=1 in '/Users/max/source/bitcoin/build_fuzz/src/test/fuzz/CMakeFiles/fuzz.dir/multiplication_overflow.cpp.o'; invalid r_symbolnum=1 in '/Users/max/source/bitcoin/build_fuzz/src/test/fuzz/CMakeFiles/fuzz.dir/overflow.cpp.o'; invalid r_symbolnum=1 in '/Users/max/source/bitcoin/build_fuzz/src/test/fuzz/CMakeFiles/fuzz.dir/connman.cpp.o'; invalid r_symbolnum=1 in '../../libbitcoin_cli.a[2](stdin.cpp.o)'; invalid r_symbolnum=1 in '../../../libcrc32c.a[3](crc32c_portable.cc.o)'; invalid r_symbolnum=1 in '../../../libcrc32c_arm64.a[2](crc32c_arm64.cc.o)'; invalid r_symbolnum=1 in '../../libbitcoin_consensus.a[11](script_error.cpp.o)'; invalid r_symbolnum=5 in '../../util/libbitcoin_util.a[29](randomenv.cpp.o)'; invalid r_symbolnum=1 in '../../crypto/libbitcoin_crypto.a[15](sha3.cpp.o)'; invalid r_symbolnum=18 in '../../crypto/libbitcoin_crypto_arm_shani.a[2](sha256_arm_shani.cpp.o)'; invalid r_symbolnum=1 in '../../crypto/libbitcoin_crypto.a[10](poly1305.cpp.o)'; invalid r_symbolnum=1 in '../../crypto/libbitcoin_crypto.a[5](hex_base.cpp.o)'; invalid r_symbolnum=1 in '../util/libtest_util.a[12](str.cpp.o)'; invalid r_symbolnum=1 in '../../util/libbitcoin_util.a[27](logging.cpp.o)'; invalid r_symbolnum=1 in '../../util/libbitcoin_util.a[24](threadnames.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_consensus.a[5](hash.cpp.o)'; invalid r_symbolnum=1 in '../../util/libbitcoin_util.a[16](serfloat.cpp.o)'; invalid r_symbolnum=1 in '../../util/libbitcoin_util.a[15](readwritefile.cpp.o)'; invalid r_symbolnum=1 in '../../util/libbitcoin_util.a[14](rbf.cpp.o)'; invalid r_symbolnum=1 in '../util/libtest_util.a[4](index.cpp.o)'; invalid r_symbolnum=1 in '../util/libtest_util.a[3](coins.cpp.o)'; invalid r_symbolnum=1 in '../../util/libbitcoin_util.a[9](feefrac.cpp.o)'; invalid r_symbolnum=1 in '../../util/libbitcoin_util.a[6](chaintype.cpp.o)'; invalid r_symbolnum=1 in '../../../libcrc32c.a[2](crc32c.cc.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[84](torcontrol.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[64](fees_args.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[54](psbt.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[53](peerman_args.cpp.o)'; invalid r_symbolnum=1 in '../../../libleveldb.a[37](logging.cc.o)'; invalid r_symbolnum=1 in '../../../libleveldb.a[35](hash.cc.o)'; invalid r_symbolnum=1 in '../../../libleveldb.a[31](crc32c.cc.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[50](miner.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[49](mempool_persist_args.cpp.o)'; invalid r_symbolnum=1 in '../../../libleveldb.a[27](bloom.cc.o)'; invalid r_symbolnum=1 in '../../../libleveldb.a[8](filename.cc.o)'; invalid r_symbolnum=1 in '../../../libleveldb.a[2](builder.cc.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[42](database_args.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[40](connection_types.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[39](coins_view_args.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[38](coin.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[37](chainstatemanager_args.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[35](caches.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_common.a[49](parsing.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[33](blockmanager_args.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_common.a[44](request.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_common.a[43](rawtransaction_util.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[27](mempool_removal_reason.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[22](checks.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_node.a[21](chain.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_common.a[39](pow.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_common.a[31](merkleblock.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_common.a[30](key_io.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_common.a[25](deploymentinfo.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_common.a[22](compressor.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_common.a[21](url.cpp.o)'; invalid r_symbolnum=1 in '../../libbitcoin_common.a[17](run_command.cpp.o)'
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[2]: *** [src/test/fuzz/CMakeFiles/fuzz.dir/build.make:2239: src/test/fuzz/fuzz] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:1170: src/test/fuzz/CMakeFiles/fuzz.dir/all] Error 2
gmake: *** [Makefile:136: all] Error 2

I was able to compile with Clang 19.1.7 by using LLVM's linker lld.

brew install lld
cmake --preset=libfuzzer -DCMAKE_C_COMPILER="$(brew --prefix llvm)/bin/clang" -DCMAKE_CXX_COMPILER="$(brew --prefix llvm)/bin/clang++" -DAPPEND_LDFLAGS=-Wl,-no_warn_duplicate_libraries -DCMAKE_CXX_FLAGS="-fuse-ld=lld"
cmake --build build_fuzz

[100%] Linking CXX executable fuzz
[100%] Built target fuzz

A note that clang 19.1.7's sanitizers required a minimum OSX target of 14.7.

As commented another thread #31591 (comment), I'd like to paste my answer here so other people will install the correct llvm version:

I believe the right llvm version should be the one matches with the output indicated in "ld -v". For example, on my mac (Apple M1 chipset, OS: Sequoia 15.1.1):

% ld -v @(#)PROGRAM:ld PROJECT:ld-1115.7.3 BUILD 23:52:02 Dec 5 2024 configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em will use ld-classic for: armv6 armv7 armv7s i386 armv6m armv7k armv7m armv7em LTO support using: LLVM version 16.0.0 (static support for 29, runtime is 29) TAPI support using: Apple TAPI version 16.0.0 (tapi-1600.0.11.9)

So, we need to install llvm 16.0.0.

% brew install llvm@16

Would it make sense to temporarily pin the clang version to 18, in https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md#macos-hints-for-libfuzzer, for now? Otherwise, people will keep running into this?

@maflcko Instead of hard-code to a specific version, I believe the right instruction for Mac is to run "ld -v" from console. Then, depend on the output of "Apple TAPI version x.x.x", install the right llvm version accordingly.

Actually, re-reading the thread, it looks like the simplest solution would be to install lld and pass -DCMAKE_CXX_FLAGS="-fuse-ld=lld", according to #31049 (comment).

This avoids having to hard-code a specific version of clang in the docs (or ask the users to pick one), but no strong opinion.

@maflcko Did you test it? Actually, re-reading the thread, it looks like the simplest solution would be to install lld and pass -DCMAKE_CXX_FLAGS="-fuse-ld=lld"

I tested on Mac (Apple M1 chipset, OS: Sequoia 15.1.1) with following commands and it failed on building

% brew install llvm # this will install llvm version 19 % brew install lld % cmake --preset=libfuzzer -DCMAKE_C_COMPILER="$(brew --prefix llvm)/bin/clang" -DCMAKE_CXX_COMPILER="$(brew --prefix llvm)/bin/clang++" -DAPPEND_LDFLAGS=-Wl,-no_warn_duplicate_libraries -DCMAKE_CXX_FLAGS="-fuse-ld=lld" % cmake --build build_fuzz ... ... ... [ 15%] Building CXX object CMakeFiles/leveldb.dir/src/leveldb/helpers/memenv/memenv.cc.o [ 15%] Building CXX object CMakeFiles/leveldb.dir/src/leveldb/util/status.cc.o 4 warnings generated. clang++: warning: argument unused during compilation: '-fuse-ld=lld' [-Wunused-command-line-argument] clang++: warning: argument unused during compilation: '-fstack-clash-protection' [-Wunused-command-line-argument] warning: unknown warning option '-Wduplicated-branches' [-Wunknown-warning-option] warning: unknown warning option '-Wduplicated-cond' [-Wunknown-warning-option] warning: unknown warning option '-Wlogical-op'; did you mean '-Wlong-long'? [-Wunknown-warning-option] warning: unknown warning option '-Wbidi-chars=any' [-Wunknown-warning-option] clang++: warning: argument unused during compilation: '-fuse-ld=lld' [-Wunused-command-line-argument] clang++: warning: argument unused during compilation: '-fstack-clash-protection' [-Wunused-command-line-argument] warning: unknown warning option '-Wduplicated-branches' [-Wunknown-warning-option] warning: unknown warning option '-Wduplicated-cond' [-Wunknown-warning-option] warning: unknown warning option '-Wlogical-op'; did you mean '-Wlong-long'? [-Wunknown-warning-option] warning: unknown warning option '-Wbidi-chars=any' [-Wunknown-warning-option] 4 warnings generated. 4 warnings generated. 4 warnings generated. 4 warnings generated. 4 warnings generated. 4 warnings generated. 4 warnings generated. [ 15%] Linking CXX static library libleveldb.a [ 15%] Built target leveldb make: *** [all] Error 2

@maflcko Did you test it? Actually, re-reading the thread, it looks like the simplest solution would be to install lld and pass -DCMAKE_CXX_FLAGS="-fuse-ld=lld"

I don't have a mac, so no. Theoretically, I also would have expected -DAPPEND_LDFLAGS='-fuse-ld=lld' instead, but I haven't tested this either.

I also would have expected -DAPPEND_LDFLAGS='-fuse-ld=lld' instead

Good spot, this would probably work. I updated the doc change in #31954 to use the flag CMAKE_EXE_LINKER_FLAG which also works.