fuzz: ASAN complaint on macOS with -fsanitize=fuzzer,address,undefined #19789

Compiler: clang installed recently via brew install llvm Machine: macOS v10.15.4 Catalina configure script:

./configure --enable-fuzz --with-sanitizers=fuzzer,address,undefined --disable-asm

Error when running src/test/fuzz/process_messages harness:

AddressSanitizer:DEADLYSIGNAL
=================================================================
==62428==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000096 (pc 0x7fff7396e98d bp 0x7ffee1ff7a70 sp 0x7ffee1ff7a70 T0)
==62428==The signal is caused by a WRITE memory access.
==62428==Hint: address points to the zero page.
    [#0](/bitcoin-bitcoin/0/) 0x7fff7396e98d in _platform_memmove$VARIANT$Haswell+0x8d (libsystem_platform.dylib:x86_64+0x98d)
    [#1](/bitcoin-bitcoin/1/) 0x112255f5c in __asan_memcpy+0x29c (libclang_rt.asan_osx_dynamic.dylib:x86_64+0x41f5c)
    [#2](/bitcoin-bitcoin/2/) 0x10ea42578 in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > std::__1::operator+<char, std::__1::char_traits<char>, std::__1::allocator<char> >(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&, char const*) string:4157
    [#3](/bitcoin-bitcoin/3/) 0x10ed7cf09 in BCLog::Logger::LogPrintStr(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) logging.cpp:245
    [#4](/bitcoin-bitcoin/4/) 0x10ddc6b3a in void LogPrintf<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(char const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) logging.h:176
    [#5](/bitcoin-bitcoin/5/) 0x10ddc64d0 in InitLogging() init.cpp:885
    [#6](/bitcoin-bitcoin/6/) 0x10eebb369 in BasicTestingSetup::BasicTestingSetup(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<char const*, std::__1::allocator<char const*> > const&) setup_common.cpp:100
    [#7](/bitcoin-bitcoin/7/) 0x10eebd320 in TestingSetup::TestingSetup(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<char const*, std::__1::allocator<char const*> > const&) setup_common.cpp:128
    [#8](/bitcoin-bitcoin/8/) 0x10dc09bd8 in initialize() process_messages.cpp:23
    [#9](/bitcoin-bitcoin/9/) 0x10ef6c7c8 in LLVMFuzzerInitialize fuzz.cpp:52
    [#10](/bitcoin-bitcoin/10/) 0x10f1adf77 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) FuzzerDriver.cpp:616
    [#11](/bitcoin-bitcoin/11/) 0x10f1db8d2 in main FuzzerMain.cpp:19
    [#12](/bitcoin-bitcoin/12/) 0x7fff73778cc8 in start+0x0 (libdyld.dylib:x86_64+0x1acc8)

==62428==Register values:
rax = 0x0000000000000096  rbx = 0x0000000000000018  rcx = 0x0000000000000001  rdx = 0x0000000000000008  
rdi = 0x0000000000000096  rsi = 0x0000000110fae2c0  rbp = 0x00007ffee1ff7a70  rsp = 0x00007ffee1ff7a70  
 r8 = 0x0000000110fae200   r9 = 0x0000008752129480  r10 = 0x0000000000000004  r11 = 0xfffffffeef051dd6  
r12 = 0x0000000000000096  r13 = 0x0000000000000000  r14 = 0x0000000110fae2c0  r15 = 0x0000000000000096  
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (libsystem_platform.dylib:x86_64+0x98d) in _platform_memmove$VARIANT$Haswell+0x8d
==62428==ABORTING
Abort trap: 6

It only occurs on my macOS with ASAN+UBSAN but not with ASAN alone. Does not occur on Ubuntu box. Offending line: https://github.com/bitcoin/bitcoin/blob/7f609f68d835bece8b01da1b09b127c67769ae7d/src/logging.cpp#L245

Ping @adaminsky

I can reproduce this, and I made a simple proof of concept test which seems to show this might be a problem with thread_local variables on macOS. I get the following error before the AddressSanitizer:DEADLYSIGNAL output:

/usr/local/opt/llvm/bin/../include/c++/v1/string:2728:44: runtime error: member call on misaligned address 0x000000000001 for type 'std::__1::basic_string<char> *', which requires 8 byte alignment

I think the issue is that the address for util::ThreadGetInternalName(), which comes from a thread_local variable, is not 8 byte aligned because macOS does not enforce any alignment for thread_local variables. clang seems to require that thread_local variables be treated similarly to global variables, so they must be properly aligned which may throw off the address sanitizer.

This small example creates a very similar error when compiled with clang++ -g -O3 -fsanitize=address,undefined test.cpp -o test.

#include <iostream>

static thread_local std::string tstr = "test1";
std::string global = "test2";

int main() {
    std::cout << "<" + global + ">" << std::endl; // Works fine
    std::cout << "<" + tstr + ">" << std::endl;   // Error
}

The -O3 flag is important because when I try -O1, no error is found. The alignment problem can be seen by using a debugger to print the addresses of global and tstr to see how the first is correctly aligned and the second isn't. I think the above program should work correctly if I am using a thread_local variable correctly, so this could be an llvm issue.

It happens in the process_messages harness with varying regularity (depending on optimization flags) on my mac only with -fsanitize=address,undefined set (rather than just -fsanitize=address). I think all we can do here is add an asan suppression. What do you think @practicalswift ?

+1

I reproduced this unintentionally while testing #19065.

@Crypt-iQ Sorry, I don't have any Mac to test on and I haven't seen this issue under Linux. I'm afraid I'll have to let someone else tackle this issue :)

I can reproduce this, and I made a simple proof of concept test which seems to show this might be a problem with thread_local variables on macOS. I get the following error before the AddressSanitizer:DEADLYSIGNAL output:

/usr/local/opt/llvm/bin/../include/c++/v1/string:2728:44: runtime error: member call on misaligned address 0x000000000001 for type 'std::__1::basic_string<char> *', which requires 8 byte alignment

That's an interesting find, I'm not sure it's the same one though, because the OP got a Segmentation Violation while you got a misaligned read. Nevertheless you should report this to: https://bugs.llvm.org with the full details(clang version, Xcode version, full sanitizer output).

@elichai So I believe the two reports are related:

This is the full output I get when running the process_messages harness and using --with-sanitizers=fuzzer,address,undefined,integer in the configure script:

<details> <summary>output</summary>

UBSAN_OPTIONS="suppressions=test/sanitizer_suppressions/ubsan:print_stacktrace=1:report_error_type=1" src/test/fuzz/process_messages ~/qa-assets/fuzz_seed_corpus/process_messages
/usr/local/opt/llvm/bin/../include/c++/v1/string:2728:44: runtime error: member call on misaligned address 0x000000000001 for type 'std::__1::basic_string<char> *', which requires 8 byte alignment
0x000000000001: note: pointer points here
<memory cannot be printed>
    [#0](/bitcoin-bitcoin/0/) 0x10d0ac9d0 in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::insert(unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) string
    [#1](/bitcoin-bitcoin/1/) 0x10d02414a in BCLog::Logger::LogPrintStr(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) logging.cpp:245
    [#2](/bitcoin-bitcoin/2/) 0x10bd99d6e in void LogPrintf<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(char const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) logging.h:176
    [#3](/bitcoin-bitcoin/3/) 0x10bd99540 in InitLogging(ArgsManager const&) init.cpp:890
    [#4](/bitcoin-bitcoin/4/) 0x10d18290e in BasicTestingSetup::BasicTestingSetup(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<char const*, std::__1::allocator<char const*> > const&) setup_common.cpp:100
    [#5](/bitcoin-bitcoin/5/) 0x10d184d3b in TestingSetup::TestingSetup(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<char const*, std::__1::allocator<char const*> > const&) setup_common.cpp:128
    [#6](/bitcoin-bitcoin/6/) 0x10bb86ae4 in initialize() process_messages.cpp:23
    [#7](/bitcoin-bitcoin/7/) 0x10d2605dc in LLVMFuzzerInitialize fuzz.cpp:43
    [#8](/bitcoin-bitcoin/8/) 0x10d4e9f77 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) FuzzerDriver.cpp:616
    [#9](/bitcoin-bitcoin/9/) 0x10d5178d2 in main FuzzerMain.cpp:19
    [#10](/bitcoin-bitcoin/10/) 0x7fff6dbe3cc8 in start+0x0 (libdyld.dylib:x86_64+0x1acc8)

SUMMARY: UndefinedBehaviorSanitizer: misaligned-pointer-use /usr/local/opt/llvm/bin/../include/c++/v1/string:2728:44 in 
AddressSanitizer:DEADLYSIGNAL
=================================================================
==37247==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000001 (pc 0x7fff6aee7712 bp 0x7ffee407ca70 sp 0x7ffee407ca40 T0)
==37247==The signal is caused by a READ memory access.
==37247==Hint: address points to the zero page.
    [#0](/bitcoin-bitcoin/0/) 0x7fff6aee7712 in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::insert(unsigned long, char const*, unsigned long)+0x1a (libc++.1.dylib:x86_64+0x38712)
    [#1](/bitcoin-bitcoin/1/) 0x10d0ac969 in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::insert(unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) string:2730
    [#2](/bitcoin-bitcoin/2/) 0x10d02414a in BCLog::Logger::LogPrintStr(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) logging.cpp:245
    [#3](/bitcoin-bitcoin/3/) 0x10bd99d6e in void LogPrintf<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(char const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) logging.h:176
    [#4](/bitcoin-bitcoin/4/) 0x10bd99540 in InitLogging(ArgsManager const&) init.cpp:890
    [#5](/bitcoin-bitcoin/5/) 0x10d18290e in BasicTestingSetup::BasicTestingSetup(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<char const*, std::__1::allocator<char const*> > const&) setup_common.cpp:100
    [#6](/bitcoin-bitcoin/6/) 0x10d184d3b in TestingSetup::TestingSetup(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<char const*, std::__1::allocator<char const*> > const&) setup_common.cpp:128
    [#7](/bitcoin-bitcoin/7/) 0x10bb86ae4 in initialize() process_messages.cpp:23
    [#8](/bitcoin-bitcoin/8/) 0x10d2605dc in LLVMFuzzerInitialize fuzz.cpp:43
    [#9](/bitcoin-bitcoin/9/) 0x10d4e9f77 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) FuzzerDriver.cpp:616
    [#10](/bitcoin-bitcoin/10/) 0x10d5178d2 in main FuzzerMain.cpp:19
    [#11](/bitcoin-bitcoin/11/) 0x7fff6dbe3cc8 in start+0x0 (libdyld.dylib:x86_64+0x1acc8)

==37247==Register values:
rax = 0x0000000000000007  rbx = 0x0000000000000000  rcx = 0x0000000000000007  rdx = 0x000000010f552a81  
rdi = 0x0000000000000001  rsi = 0x0000000000000000  rbp = 0x00007ffee407ca70  rsp = 0x00007ffee407ca40  
 r8 = 0x000000010f552a00   r9 = 0x00000085dc92e700  r10 = 0x00007fff6dd27a46  r11 = 0x0000000000000206  
r12 = 0x000000010f552a81  r13 = 0x0000000000000000  r14 = 0x0000000000000007  r15 = 0x0000000000000001  
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (libc++.1.dylib:x86_64+0x38712) in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::insert(unsigned long, char const*, unsigned long)+0x1a
==37247==ABORTING
Abort trap: 6

</details>

Closing because I was able to fix this by installing a more recent llvm: brew install llvm --HEAD