Fuzz: Runtime errors when running fuzz tests on MacOs

Prabhat1308 commented at 11:33 am on January 2, 2025: none

Is there an existing issue for this?

I have searched the existing issues

Current behaviour

Whenever running any fuzz test on any fuzz target on MacOs according to the steps provided here. There is a big error log being thrown before starting the fuzz tests. To be noted that the fuzz tests run fine afterwards.

Expected behaviour

No error is thrown and fuzz tests run as expected.

Steps to reproduce

Set up environment variables for LLVM 18

0export LDFLAGS="-L$(brew --prefix llvm@18)/lib -L$(brew --prefix llvm@18)/lib/c++ -L$(brew --prefix llvm@18)/lib/unwind -lunwind"
1export CPPFLAGS="-I$(brew --prefix llvm@18)/include"
2export PATH="$(brew --prefix llvm@18)/bin:$PATH"
3export CC="$(brew --prefix llvm@18)/bin/clang"
4export CXX="$(brew --prefix llvm@18)/bin/clang++"

Run CMake with the preset

0cmake --preset=libfuzzer \
1    -DCMAKE_C_COMPILER="$(brew --prefix llvm@18)/bin/clang" \
2    -DCMAKE_CXX_COMPILER="$(brew --prefix llvm@18)/bin/clang++" \
3    -DAPPEND_LDFLAGS="-Wl,-no_warn_duplicate_libraries" \
4    -DCMAKE_EXE_LINKER_FLAGS="$LDFLAGS"

build and run with any fuzz target

0cmake --build build_fuzz -j$(sysctl -n hw.ncpu)
1FUZZ=process_message build_fuzz/src/test/fuzz/fuzz

Relevant log output

 0/opt/homebrew/opt/llvm@18/bin/../include/c++/v1/variant:495:12: runtime error: call to function decltype(auto) std::__1::__variant_detail::__visitation::__base::__dispatcher<0ul, 0ul>::__dispatch[abi:ne180100]<void std::__1::__variant_detail::__ctor<std::__1::__variant_detail::__traits<RPCArg::Optional, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, UniValue>>::__generic_construct[abi:ne180100]<std::__1::__variant_detail::__move_constructor<std::__1::__variant_detail::__traits<RPCArg::Optional, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, UniValue>, (std::__1::__variant_detail::_Trait)1>>(std::__1::__variant_detail::__ctor<std::__1::__variant_detail::__traits<RPCArg::Optional, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, UniValue>>&, std::__1::__variant_detail::__move_constructor<std::__1::__variant_detail::__traits<RPCArg::Optional, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, UniValue>, (std::__1::__variant_detail::_Trait)1>&&)::'lambda'(std::__1::__variant_detail::__move_constructor<std::__1::__variant_detail::__traits<RPCArg::Optional, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, UniValue>, (std::__1::__variant_detail::_Trait)1>&, auto&&)&&, std::__1::__variant_detail::__base<(std::__1::__variant_detail::_Trait)1, RPCArg::Optional, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, UniValue>&, std::__1::__variant_detail::__base<(std::__1::__variant_detail::_Trait)1, RPCArg::Optional, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, UniValue>&&>(std::__1::__variant_detail::__move_constructor<std::__1::__variant_detail::__traits<RPCArg::Optional, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, UniValue>, (std::__1::__variant_detail::_Trait)1>, std::__1::__variant_detail::__base<(std::__1::__variant_detail::_Trait)1, RPCArg::Optional, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, UniValue>&, std::__1::__variant_detail::__base<(std::__1::__variant_detail::_Trait)1, RPCArg::Optional, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, UniValue>&&) through pointer to incorrect function type 'void (*)((lambda at /opt/homebrew/opt/llvm@18/bin/../include/c++/v1/variant:814:11) &&, std::__variant_detail::__base<std::__variant_detail::_Trait::_Available, RPCArg::Optional, std::string, UniValue> &, std::__variant_detail::__base<std::__variant_detail::_Trait::_Available, RPCArg::Optional, std::string, UniValue> &&)'
 1variant:532: note: decltype(auto) std::__1::__variant_detail::__visitation::__base::__dispatcher<0ul, 0ul>::__dispatch[abi:ne180100]<void std::__1::__variant_detail::__ctor<std::__1::__variant_detail::__traits<RPCArg::Optional, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, UniValue>>::__generic_construct[abi:ne180100]<std::__1::__variant_detail::__move_constructor<std::__1::__variant_detail::__traits<RPCArg::Optional, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, UniValue>, (std::__1::__variant_detail::_Trait)1>>(std::__1::__variant_detail::__ctor<std::__1::__variant_detail::__traits<RPCArg::Optional, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, UniValue>>&, std::__1::__variant_detail::__move_constructor<std::__1::__variant_detail::__traits<RPCArg::Optional, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, UniValue>, (std::__1::__variant_detail::_Trait)1>&&)::'lambda'(std::__1::__variant_detail::__move_constructor<std::__1::__variant_detail::__traits<RPCArg::Optional, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, UniValue>, (std::__1::__variant_detail::_Trait)1>&, auto&&)&&, std::__1::__variant_detail::__base<(std::__1::__variant_detail::_Trait)1, RPCArg::Optional, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, UniValue>&, std::__1::__variant_detail::__base<(std::__1::__variant_detail::_Trait)1, RPCArg::Optional, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, UniValue>&&>(std::__1::__variant_detail::__move_constructor<std::__1::__variant_detail::__traits<RPCArg::Optional, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, UniValue>, (std::__1::__variant_detail::_Trait)1>, std::__1::__variant_detail::__base<(std::__1::__variant_detail::_Trait)1, RPCArg::Optional, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, UniValue>&, std::__1::__variant_detail::__base<(std::__1::__variant_detail::_Trait)1, RPCArg::Optional, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, UniValue>&&) defined here
 2SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /opt/homebrew/opt/llvm@18/bin/../include/c++/v1/variant:495:12 
 3/Users/prabhatverma/projects/bitcoin/src/rpc/server.h:100:15: runtime error: call to function getblockchaininfo() through pointer to incorrect function type 'RPCHelpMan (*)()'
 4blockchain.cpp:1288: note: getblockchaininfo() defined here
 5SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /Users/prabhatverma/projects/bitcoin/src/rpc/server.h:100:15 
 6/Users/prabhatverma/projects/bitcoin/src/rpc/server.h:102:15: runtime error: call to function getblockchaininfo() through pointer to incorrect function type 'RPCHelpMan (*)()'
 7blockchain.cpp:1288: note: getblockchaininfo() defined here
 8SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /Users/prabhatverma/projects/bitcoin/src/rpc/server.h:102:15 
 9/Users/prabhatverma/projects/bitcoin/src/tinyformat.h:537:13: runtime error: call to function void tinyformat::detail::FormatArg::formatImpl<char [13]>(std::__1::basic_ostream<char, std::__1::char_traits<char>>&, char const*, char const*, int, void const*) through pointer to incorrect function type 'void (*)(std::ostream &, const char *, const char *, int, const void *)'
10tinyformat.h:551: note: void tinyformat::detail::FormatArg::formatImpl<char [13]>(std::__1::basic_ostream<char, std::__1::char_traits<char>>&, char const*, char const*, int, void const*) defined here
11SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /Users/prabhatverma/projects/bitcoin/src/tinyformat.h:537:13 
12INFO: Running with entropic power schedule (0xFF, 100).
13INFO: Seed: 1726115676
14INFO: Loaded 1 modules   (1253576 inline 8-bit counters): 1253576 [0x1059e58c8, 0x105b17990), 
15INFO: Loaded 1 PC tables (1253576 PCs): 1253576 [0x105b17990,0x106e38610), 
16INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
17/opt/homebrew/opt/llvm@18/bin/../include/c++/v1/__type_traits/invoke.h:344:25: runtime error: call to function process_message_fuzz_target(std::__1::span<unsigned char const, 18446744073709551615ul>) through pointer to incorrect function type 'void (*)(std::span<const unsigned char>)'
18process_message.cpp:54: note: process_message_fuzz_target(std::__1::span<unsigned char const, 18446744073709551615ul>) defined here
19SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /opt/homebrew/opt/llvm@18/bin/../include/c++/v1/__type_traits/invoke.h:344:25 
20INFO: A corpus is not provided, starting from an empty corpus
21[#2](/bitcoin-bitcoin/2/)      INITED cov: 2751 ft: 2749 corp: 1/1b exec/s: 0 rss: 187Mb
22[#3](/bitcoin-bitcoin/3/)      NEW    cov: 2757 ft: 2851 corp: 2/2b lim: 4 exec/s: 0 rss: 187Mb L: 1/1 MS: 1 ChangeByte-
23[#5](/bitcoin-bitcoin/5/)      NEW    cov: 2757 ft: 2855 corp: 3/4b lim: 4 exec/s: 0 rss: 187Mb L: 2/2 MS: 2 CrossOver-InsertByte-

How did you obtain Bitcoin Core

Compiled from source

What version of Bitcoin Core are you using?

master@bb57017b2945d5e0bbd95c7f1a9369a8ab7c6fcd

Operating system and version

MacOs Sequioa 15.1.1

Machine specifications

Silicon Macbook Pro Chip - M4 Pro Memory - 24 gb (14 core cpu , 20 core gpu)

maflcko added the label macOS on Jan 2, 2025

maflcko added the label Tests on Jan 2, 2025

maflcko commented at 11:52 am on January 2, 2025: member

Does it work with the libfuzzer-nosan preset?

Prabhat1308 commented at 3:30 pm on January 2, 2025: none

Does it work with the libfuzzer-nosan preset?

works with the following warnings and without any errors.

0WARNING: Failed to find function "__sanitizer_acquire_crash_state". Reason dlsym(RTLD_DEFAULT, __sanitizer_acquire_crash_state): symbol not found.
1WARNING: Failed to find function "__sanitizer_print_stack_trace". Reason dlsym(RTLD_DEFAULT, __sanitizer_print_stack_trace): symbol not found.
2WARNING: Failed to find function "__sanitizer_set_death_callback". Reason dlsym(RTLD_DEFAULT, __sanitizer_set_death_callback): symbol not found.

Just tried on the latest master branch (master@228aba2c4d9ac0b2ca3edd3c2cdf0a92e55f669b) and I am able to reproduce the error in that too.

brunoerg commented at 8:03 pm on January 2, 2025: contributor

I have the same issue and I think other people also have reported it before, I simply ignore it.

bitcoin deleted a comment on Jan 7, 2025

tnndbtc commented at 1:39 am on February 15, 2025: none

I believe the right llvm version should be the one matches with the output indicated in “ld -v”. For example, on my mac (Apple M1 chipset, OS: Sequoia 15.1.1):

% ld -v @(#)PROGRAM:ld PROJECT:ld-1115.7.3 BUILD 23:52:02 Dec 5 2024 configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em will use ld-classic for: armv6 armv7 armv7s i386 armv6m armv7k armv7m armv7em LTO support using: LLVM version 16.0.0 (static support for 29, runtime is 29) TAPI support using: Apple TAPI version 16.0.0 (tapi-1600.0.11.9)

So, we need to install llvm 16.0.0.

% brew install llvm@16

Then I followed your instruction, replacing 18 to 16, and I cannot reproduce the issue.

I tried to force install llvm@18 but it failed to compile fuzzer with cmake –build build_fuzz.

Please check the output of “ld -v”.

Prabhat1308 commented at 2:26 am on February 15, 2025: none

@tnndbtc Thank you for this. I see that the linker in my machine also is using llvm 16 for support although llvm 19 came natively installed in machine. Right now the workaround is to build using llvm 16 but do you have any other suggestions to fix this ? Although right now it looks like an issue from Apple itself .

tnndbtc commented at 5:42 pm on February 16, 2025: none

@Prabhat1308 I don’t have an answer to fix the incompatibility issue between the native llvm@16 and the customized llvm@18. However, from the error message itself, it looks like the cause is the discrepancy between llvm@16 and llvm@18.

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /opt/homebrew/opt/llvm@18/bin/../include/c++/v1/variant:495:12 /Users/prabhatverma/projects/bitcoin/src/rpc/server.h:100:15: runtime error: call to function getblockchaininfo() through pointer to incorrect function type ‘RPCHelpMan (*)()’ blockchain.cpp:1288: note: getblockchaininfo() defined here

So, instead of trying to “fix” the bitcoin code to work with llvm@18 on a llvm@16 native environment, the right solution is to use the expected llvm@16 binaries.

I have struggled a lot when first follow the instruction by just downloading default llvm, i.e., brew install llvm , as instructed in the official page. Later, I realize that the instruction is not well written and thus conclude that we should download the right llvm@<version> based on the output of “ld -v”.

If there is a reason why you have to use llvm@18 in the native llvm@16 environment, please elaborate.

tnndbtc commented at 5:52 pm on February 16, 2025: none

Also, from llvm@18, /opt/homebrew/opt/llvm@18/bin/../include/c++/v1/variant:495

struct __base { template <class _Visitor, class… _Vs> _LIBCPP_HIDE_FROM_ABI static constexpr decltype(auto) __visit_alt_at(size_t __index, _Visitor&& __visitor, _Vs&&… __vs) { constexpr auto __fdiagonal = __make_fdiagonal<_Visitor&&, decltype(std::forward<_Vs>(__vs).__as_base())…>(); return __fdiagonal[__index](std::forward<_Visitor>(__visitor), std::forward<_Vs>(__vs).__as_base()…); }

This declaration is different from llvm@16, /opt/homebrew/opt/llvm@16/bin/../include/c++/v1/variant struct __base { template <class _Visitor, class… _Vs> _LIBCPP_HIDE_FROM_ABI static constexpr decltype(auto) __visit_alt_at(size_t __index, _Visitor&& __visitor, _Vs&&… __vs) { constexpr auto __fdiagonal = __make_fdiagonal<_Visitor&&, decltype(_VSTD::forward<_Vs>(__vs).__as_base())…>(); return __fdiagonal[__index](_VSTD::forward<_Visitor>(__visitor), _VSTD::forward<_Vs>(__vs).__as_base()…); }

So, I’d not spend more time on a mismatched llvm environment, unless there is pressing reason to do so.

Fuzz: Runtime errors when running fuzz tests on MacOs #31591