fuzz: ASAN complaint on macOS with -fsanitize=fuzzer,address,undefined

Crypt-iQ commented at 3:52 pm on August 24, 2020: contributor

Compiler: clang installed recently via brew install llvm Machine: macOS v10.15.4 Catalina configure script:

0./configure --enable-fuzz --with-sanitizers=fuzzer,address,undefined --disable-asm

Error when running src/test/fuzz/process_messages harness:

 0AddressSanitizer:DEADLYSIGNAL
 1=================================================================
 2==62428==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000096 (pc 0x7fff7396e98d bp 0x7ffee1ff7a70 sp 0x7ffee1ff7a70 T0)
 3==62428==The signal is caused by a WRITE memory access.
 4==62428==Hint: address points to the zero page.
 5    [#0](/bitcoin-bitcoin/0/) 0x7fff7396e98d in _platform_memmove$VARIANT$Haswell+0x8d (libsystem_platform.dylib:x86_64+0x98d)
 6    [#1](/bitcoin-bitcoin/1/) 0x112255f5c in __asan_memcpy+0x29c (libclang_rt.asan_osx_dynamic.dylib:x86_64+0x41f5c)
 7    [#2](/bitcoin-bitcoin/2/) 0x10ea42578 in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > std::__1::operator+<char, std::__1::char_traits<char>, std::__1::allocator<char> >(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&, char const*) string:4157
 8    [#3](/bitcoin-bitcoin/3/) 0x10ed7cf09 in BCLog::Logger::LogPrintStr(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) logging.cpp:245
 9    [#4](/bitcoin-bitcoin/4/) 0x10ddc6b3a in void LogPrintf<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(char const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) logging.h:176
10    [#5](/bitcoin-bitcoin/5/) 0x10ddc64d0 in InitLogging() init.cpp:885
11    [#6](/bitcoin-bitcoin/6/) 0x10eebb369 in BasicTestingSetup::BasicTestingSetup(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<char const*, std::__1::allocator<char const*> > const&) setup_common.cpp:100
12    [#7](/bitcoin-bitcoin/7/) 0x10eebd320 in TestingSetup::TestingSetup(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<char const*, std::__1::allocator<char const*> > const&) setup_common.cpp:128
13    [#8](/bitcoin-bitcoin/8/) 0x10dc09bd8 in initialize() process_messages.cpp:23
14    [#9](/bitcoin-bitcoin/9/) 0x10ef6c7c8 in LLVMFuzzerInitialize fuzz.cpp:52
15    [#10](/bitcoin-bitcoin/10/) 0x10f1adf77 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) FuzzerDriver.cpp:616
16    [#11](/bitcoin-bitcoin/11/) 0x10f1db8d2 in main FuzzerMain.cpp:19
17    [#12](/bitcoin-bitcoin/12/) 0x7fff73778cc8 in start+0x0 (libdyld.dylib:x86_64+0x1acc8)
18
19==62428==Register values:
20rax = 0x0000000000000096  rbx = 0x0000000000000018  rcx = 0x0000000000000001  rdx = 0x0000000000000008  
21rdi = 0x0000000000000096  rsi = 0x0000000110fae2c0  rbp = 0x00007ffee1ff7a70  rsp = 0x00007ffee1ff7a70  
22 r8 = 0x0000000110fae200   r9 = 0x0000008752129480  r10 = 0x0000000000000004  r11 = 0xfffffffeef051dd6  
23r12 = 0x0000000000000096  r13 = 0x0000000000000000  r14 = 0x0000000110fae2c0  r15 = 0x0000000000000096  
24AddressSanitizer can not provide additional info.
25SUMMARY: AddressSanitizer: SEGV (libsystem_platform.dylib:x86_64+0x98d) in _platform_memmove$VARIANT$Haswell+0x8d
26==62428==ABORTING
27Abort trap: 6

It only occurs on my macOS with ASAN+UBSAN but not with ASAN alone. Does not occur on Ubuntu box. Offending line: https://github.com/bitcoin/bitcoin/blob/7f609f68d835bece8b01da1b09b127c67769ae7d/src/logging.cpp#L245

Crypt-iQ commented at 3:54 pm on August 24, 2020: contributor

Ping @adaminsky

MarcoFalke added the label macOS on Aug 24, 2020

MarcoFalke added the label Tests on Aug 24, 2020

adaminsky commented at 9:24 pm on August 25, 2020: contributor

I can reproduce this, and I made a simple proof of concept test which seems to show this might be a problem with thread_local variables on macOS. I get the following error before the AddressSanitizer:DEADLYSIGNAL output:

0/usr/local/opt/llvm/bin/../include/c++/v1/string:2728:44: runtime error: member call on misaligned address 0x000000000001 for type 'std::__1::basic_string<char> *', which requires 8 byte alignment

I think the issue is that the address for util::ThreadGetInternalName(), which comes from a thread_local variable, is not 8 byte aligned because macOS does not enforce any alignment for thread_local variables. clang seems to require that thread_local variables be treated similarly to global variables, so they must be properly aligned which may throw off the address sanitizer.

This small example creates a very similar error when compiled with clang++ -g -O3 -fsanitize=address,undefined test.cpp -o test.

0#include <iostream>
1
2static thread_local std::string tstr = "test1";
3std::string global = "test2";
4
5int main() {
6    std::cout << "<" + global + ">" << std::endl; // Works fine
7    std::cout << "<" + tstr + ">" << std::endl;   // Error
8}

The -O3 flag is important because when I try -O1, no error is found. The alignment problem can be seen by using a debugger to print the addresses of global and tstr to see how the first is correctly aligned and the second isn’t. I think the above program should work correctly if I am using a thread_local variable correctly, so this could be an llvm issue.

Crypt-iQ commented at 3:28 am on September 2, 2020: contributor

It happens in the process_messages harness with varying regularity (depending on optimization flags) on my mac only with -fsanitize=address,undefined set (rather than just -fsanitize=address). I think all we can do here is add an asan suppression. What do you think @practicalswift ?

fjahr commented at 8:38 pm on October 18, 2020: member

+1

I reproduced this unintentionally while testing #19065.

practicalswift commented at 8:44 pm on October 18, 2020: contributor

@Crypt-iQ Sorry, I don’t have any Mac to test on and I haven’t seen this issue under Linux. I’m afraid I’ll have to let someone else tackle this issue :)

elichai commented at 1:31 pm on October 22, 2020: contributor

I can reproduce this, and I made a simple proof of concept test which seems to show this might be a problem with thread_local variables on macOS. I get the following error before the AddressSanitizer:DEADLYSIGNAL output:
0/usr/local/opt/llvm/bin/../include/c++/v1/string:2728:44: runtime error: member call on misaligned address 0x000000000001 for type 'std::__1::basic_string<char> *', which requires 8 byte alignment

That’s an interesting find, I’m not sure it’s the same one though, because the OP got a Segmentation Violation while you got a misaligned read. Nevertheless you should report this to: https://bugs.llvm.org with the full details(clang version, Xcode version, full sanitizer output).

Crypt-iQ commented at 6:14 pm on October 24, 2020: contributor

@elichai So I believe the two reports are related:

This is the full output I get when running the process_messages harness and using --with-sanitizers=fuzzer,address,undefined,integer in the configure script:

 0UBSAN_OPTIONS="suppressions=test/sanitizer_suppressions/ubsan:print_stacktrace=1:report_error_type=1" src/test/fuzz/process_messages ~/qa-assets/fuzz_seed_corpus/process_messages
 1/usr/local/opt/llvm/bin/../include/c++/v1/string:2728:44: runtime error: member call on misaligned address 0x000000000001 for type 'std::__1::basic_string<char> *', which requires 8 byte alignment
 20x000000000001: note: pointer points here
 3<memory cannot be printed>
 4    [#0](/bitcoin-bitcoin/0/) 0x10d0ac9d0 in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::insert(unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) string
 5    [#1](/bitcoin-bitcoin/1/) 0x10d02414a in BCLog::Logger::LogPrintStr(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) logging.cpp:245
 6    [#2](/bitcoin-bitcoin/2/) 0x10bd99d6e in void LogPrintf<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(char const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) logging.h:176
 7    [#3](/bitcoin-bitcoin/3/) 0x10bd99540 in InitLogging(ArgsManager const&) init.cpp:890
 8    [#4](/bitcoin-bitcoin/4/) 0x10d18290e in BasicTestingSetup::BasicTestingSetup(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<char const*, std::__1::allocator<char const*> > const&) setup_common.cpp:100
 9    [#5](/bitcoin-bitcoin/5/) 0x10d184d3b in TestingSetup::TestingSetup(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<char const*, std::__1::allocator<char const*> > const&) setup_common.cpp:128
10    [#6](/bitcoin-bitcoin/6/) 0x10bb86ae4 in initialize() process_messages.cpp:23
11    [#7](/bitcoin-bitcoin/7/) 0x10d2605dc in LLVMFuzzerInitialize fuzz.cpp:43
12    [#8](/bitcoin-bitcoin/8/) 0x10d4e9f77 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) FuzzerDriver.cpp:616
13    [#9](/bitcoin-bitcoin/9/) 0x10d5178d2 in main FuzzerMain.cpp:19
14    [#10](/bitcoin-bitcoin/10/) 0x7fff6dbe3cc8 in start+0x0 (libdyld.dylib:x86_64+0x1acc8)
15
16SUMMARY: UndefinedBehaviorSanitizer: misaligned-pointer-use /usr/local/opt/llvm/bin/../include/c++/v1/string:2728:44 in 
17AddressSanitizer:DEADLYSIGNAL
18=================================================================
19==37247==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000001 (pc 0x7fff6aee7712 bp 0x7ffee407ca70 sp 0x7ffee407ca40 T0)
20==37247==The signal is caused by a READ memory access.
21==37247==Hint: address points to the zero page.
22    [#0](/bitcoin-bitcoin/0/) 0x7fff6aee7712 in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::insert(unsigned long, char const*, unsigned long)+0x1a (libc++.1.dylib:x86_64+0x38712)
23    [#1](/bitcoin-bitcoin/1/) 0x10d0ac969 in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::insert(unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) string:2730
24    [#2](/bitcoin-bitcoin/2/) 0x10d02414a in BCLog::Logger::LogPrintStr(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) logging.cpp:245
25    [#3](/bitcoin-bitcoin/3/) 0x10bd99d6e in void LogPrintf<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(char const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) logging.h:176
26    [#4](/bitcoin-bitcoin/4/) 0x10bd99540 in InitLogging(ArgsManager const&) init.cpp:890
27    [#5](/bitcoin-bitcoin/5/) 0x10d18290e in BasicTestingSetup::BasicTestingSetup(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<char const*, std::__1::allocator<char const*> > const&) setup_common.cpp:100
28    [#6](/bitcoin-bitcoin/6/) 0x10d184d3b in TestingSetup::TestingSetup(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<char const*, std::__1::allocator<char const*> > const&) setup_common.cpp:128
29    [#7](/bitcoin-bitcoin/7/) 0x10bb86ae4 in initialize() process_messages.cpp:23
30    [#8](/bitcoin-bitcoin/8/) 0x10d2605dc in LLVMFuzzerInitialize fuzz.cpp:43
31    [#9](/bitcoin-bitcoin/9/) 0x10d4e9f77 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) FuzzerDriver.cpp:616
32    [#10](/bitcoin-bitcoin/10/) 0x10d5178d2 in main FuzzerMain.cpp:19
33    [#11](/bitcoin-bitcoin/11/) 0x7fff6dbe3cc8 in start+0x0 (libdyld.dylib:x86_64+0x1acc8)
34
35==37247==Register values:
36rax = 0x0000000000000007  rbx = 0x0000000000000000  rcx = 0x0000000000000007  rdx = 0x000000010f552a81  
37rdi = 0x0000000000000001  rsi = 0x0000000000000000  rbp = 0x00007ffee407ca70  rsp = 0x00007ffee407ca40  
38 r8 = 0x000000010f552a00   r9 = 0x00000085dc92e700  r10 = 0x00007fff6dd27a46  r11 = 0x0000000000000206  
39r12 = 0x000000010f552a81  r13 = 0x0000000000000000  r14 = 0x0000000000000007  r15 = 0x0000000000000001  
40AddressSanitizer can not provide additional info.
41SUMMARY: AddressSanitizer: SEGV (libc++.1.dylib:x86_64+0x38712) in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::insert(unsigned long, char const*, unsigned long)+0x1a
42==37247==ABORTING
43Abort trap: 6

Crypt-iQ commented at 11:19 pm on March 1, 2021: contributor

Closing because I was able to fix this by installing a more recent llvm: brew install llvm --HEAD

Crypt-iQ closed this on Mar 1, 2021

DrahtBot locked this on Aug 18, 2022

fuzz: ASAN complaint on macOS with -fsanitize=fuzzer,address,undefined #19789