Segfault running make check with macOS 10.15 SDK (default for Catalina) #674

Sjors commented at 11:52 AM on October 11, 2019: member

This also impacts running the test suite on Bitcoin Core:

% ./autogen.sh
...
% ./configure
...
Build Options:
  with endomorphism       = no
  with ecmult precomp     = yes
  with external callbacks = no
  with jni                = no
  with benchmarks         = yes
  with coverage           = no
  module ecdh             = no
  module recovery         = no

  asm                     = x86_64
  bignum                  = gmp
  field                   = 64bit
  scalar                  = 64bit
  ecmult window size      = 15
  ecmult gen prec. bits   = 4

  CC                      = gcc
  CFLAGS                  = -g -O2 -W -std=c89 -pedantic -Wall -Wextra -Wcast-align -Wnested-externs -Wshadow -Wstrict-prototypes -Wno-unused-function -Wno-long-long -Wno-overlength-strings -fvisibility=hidden -O3
  CPPFLAGS                = 
  LDFLAGS                 = 

% make
...
% make check
/Applications/Xcode.app/Contents/Developer/usr/bin/make  check-TESTS
./build-aux/test-driver: line 107: 70234 Segmentation fault: 11  "$@" > $log_file 2>&1
FAIL: tests
PASS: exhaustive_tests
============================================================================
Testsuite summary for libsecp256k1 0.1
============================================================================
# TOTAL: 2
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
============================================================================
See ./test-suite.log
============================================================================
make[2]: *** [test-suite.log] Error 1
make[1]: *** [check-TESTS] Error 2
make: *** [check-am] Error 2

Sjors cross-referenced this on Oct 11, 2019 from issue macOS Catalina by Sjors

elichai commented at 11:55 AM on October 11, 2019: contributor

Care to run it through valgrind or something else to get the stack trace?

fjahr commented at 3:23 PM on October 11, 2019: none

+1

Ran through valgrind but it did not give me much yet. Any specific tools/options I should use?

$ valgrind make check
==17696== Memcheck, a memory error detector
==17696== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==17696== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==17696== Command: make check
==17696==
--17696-- UNKNOWN fcntl 101!
--17696-- UNKNOWN fcntl 101! (repeated 2 times)
--17696-- UNKNOWN fcntl 101! (repeated 4 times)
--17696-- UNKNOWN fcntl 101! (repeated 8 times)
--17696-- UNKNOWN fcntl 101! (repeated 16 times)
--17696-- UNKNOWN fcntl 101! (repeated 32 times)
--17696-- UNKNOWN task message [id 8000, to mach_task_self(), reply 0x707]
--17696-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option
--17696-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 2 times)
--17696-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 4 times)
--17696-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 8 times)
/Applications/Xcode.app/Contents/Developer/usr/bin/make  check-TESTS
./build-aux/test-driver: line 107: 17719 Segmentation fault: 11  "$@" > $log_file 2>&1
FAIL: tests
PASS: exhaustive_tests
============================================================================
Testsuite summary for libsecp256k1 0.1
============================================================================
# TOTAL: 2
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
============================================================================
See ./test-suite.log
============================================================================
make[2]: *** [test-suite.log] Error 1
make[1]: *** [check-TESTS] Error 2
make: *** [check-am] Error 2

elichai commented at 3:39 PM on October 11, 2019: contributor

@fjahr I hoped valgrind will be enough. try running the tests executable via gdb. that would hopefully give some useful info

fjahr commented at 4:04 PM on October 11, 2019: none

This may be connected to clang optimization because if I set -O0 manually or use --coverage then I don't get a failure.

Here is the lldb output with optimization:

$ lldb tests
(lldb) target create "tests"
Current executable set to 'tests' (x86_64).
(lldb) run
Process 18139 launched: '/Users/FJ/projects/clones/secp256k1/tests' (x86_64)
test count = 64
random seed = 7669762ec80237764c15c37733ac9962
Process 18139 stopped
* thread [#1](/bitcoin-core-secp256k1/1/), queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame [#0](/bitcoin-core-secp256k1/0/): 0x00007fff66a77476 libdyld.dylib`stack_not_16_byte_aligned_error
libdyld.dylib`stack_not_16_byte_aligned_error:
->  0x7fff66a77476 <+0>: movdqa %xmm0, (%rsp)
    0x7fff66a7747b <+5>: int3
    0x7fff66a7747c <+6>: nop
    0x7fff66a7747d <+7>: nop
Target 0: (tests) stopped.

sipa commented at 4:42 PM on October 11, 2019: contributor

movdqa %xmm0, (%rsp)

That's an SSE instruction that requires a 16-byte aligned argument. @real-or-random found that it may be the case that the libdyld library being invoked assumes 16-byte aligned stacks, but somehow our code is being compiled to something that doesn't obey that requirement.

real-or-random commented at 4:55 PM on October 11, 2019: contributor

What's the gcc version you use? Can we get a backtrace? The command is bt in lldb.

fjahr commented at 5:03 PM on October 11, 2019: none

MacOS uses clang under the hood so that is what I am using:

$ gcc -v
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 11.0.0 (clang-1100.0.33.8)
Target: x86_64-apple-darwin19.0.0

With backtrace:

$ lldb tests
(lldb) target create "tests"
Current executable set to 'tests' (x86_64).
(lldb) run
Process 37406 launched: '/Users/FJ/projects/clones/secp256k1/tests' (x86_64)
test count = 64
random seed = 74932005d06c1e2031169dd02eb41146
Process 37406 stopped
* thread [#1](/bitcoin-core-secp256k1/1/), queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame [#0](/bitcoin-core-secp256k1/0/): 0x00007fff66a77476 libdyld.dylib`stack_not_16_byte_aligned_error
libdyld.dylib`stack_not_16_byte_aligned_error:
->  0x7fff66a77476 <+0>: movdqa %xmm0, (%rsp)
    0x7fff66a7747b <+5>: int3
    0x7fff66a7747c <+6>: nop
    0x7fff66a7747d <+7>: nop
Target 0: (tests) stopped.
(lldb) bt
* thread [#1](/bitcoin-core-secp256k1/1/), queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
  * frame [#0](/bitcoin-core-secp256k1/0/): 0x00007fff66a77476 libdyld.dylib`stack_not_16_byte_aligned_error
    frame [#1](/bitcoin-core-secp256k1/1/): 0x00007ffeefbfeef0
    frame [#2](/bitcoin-core-secp256k1/2/): 0x000000010006c5d6 tests`main(argc=<unavailable>, argv=<unavailable>) at tests.c:5237:5 [opt]
    frame [#3](/bitcoin-core-secp256k1/3/): 0x00007fff66a86405 libdyld.dylib`start + 1
    frame [#4](/bitcoin-core-secp256k1/4/): 0x00007fff66a86405 libdyld.dylib`start + 1

I did a quick search on "stack_not_16_byte_aligned_error clang 11" and that seems to confirm that it is an compiler issue since several other projects have the same issue.

MarcoFalke commented at 5:20 PM on October 11, 2019: none

valgrind make check

I think you want to use valgrind ./tests instead to get a bt

fjahr commented at 5:28 PM on October 11, 2019: none

valgrind make check

I think you want to use valgrind ./tests instead to get a bt

True, thanks @MarcoFalke

$ valgrind ./tests
==37614== Memcheck, a memory error detector
==37614== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==37614== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==37614== Command: ./tests
==37614==
--37614-- run: /usr/bin/dsymutil "./tests"
--37614-- UNKNOWN fcntl 101!
--37614-- UNKNOWN fcntl 101! (repeated 2 times)
--37614-- UNKNOWN fcntl 101! (repeated 4 times)
--37614-- UNKNOWN fcntl 101! (repeated 8 times)
--37614-- UNKNOWN fcntl 101! (repeated 16 times)
--37614-- UNKNOWN fcntl 101! (repeated 32 times)
--37614-- UNKNOWN task message [id 8000, to mach_task_self(), reply 0x707]
test count = 64
random seed = a8e65eed034f14a39adaf577e70d7c64
==37614==
==37614== Process terminating with default action of signal 11 (SIGSEGV)
==37614==  General Protection Fault
==37614==    at 0x1003F2476: stack_not_16_byte_aligned_error (in /usr/lib/system/libdyld.dylib)
==37614==    by 0x10006C5D5: main (tests.c:5237)
==37614==
==37614== HEAP SUMMARY:
==37614==     in use at exit: 546,371 bytes in 164 blocks
==37614==   total heap usage: 269,004 allocs, 268,840 frees, 10,293,899 bytes allocated
==37614==
==37614== LEAK SUMMARY:
==37614==    definitely lost: 0 bytes in 0 blocks
==37614==    indirectly lost: 0 bytes in 0 blocks
==37614==      possibly lost: 4,224 bytes in 6 blocks
==37614==    still reachable: 527,264 bytes in 12 blocks
==37614==         suppressed: 14,883 bytes in 146 blocks
==37614== Rerun with --leak-check=full to see details of leaked memory
==37614==
==37614== For lists of detected and suppressed errors, rerun with: -s
==37614== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 4 from 4)
Segmentation fault: 11

MarcoFalke commented at 5:33 PM on October 11, 2019: none

Oh, looks like that function is inlined (optimized) by the compiler. https://github.com/bitcoin-core/secp256k1/blob/0d9540b13ffcd7cd44cc361b8744b93d88aa76ba/src/tests.c#L5237

Maybe compile without optimiztions or try using the god method of debugging (printf)

fjahr commented at 6:01 PM on October 11, 2019: none

Hm, I still can not reproduce the error without optimizations. I tried to get more info with printf but I think the optimization makes it hard: I can get something printed from line 5236 just before the function is called but not from inside the function (1083).

MarcoFalke commented at 6:17 PM on October 11, 2019: none

clang-1100.0.33.8 is upstream 8.0.0 according to https://en.wikipedia.org/wiki/Xcode#Xcode_7.0_-_11.x_%28since_Free_On-Device_Development%29

However, I couldn't reproduce on debian with either clang-8, nor clang-9

MarcoFalke commented at 6:33 PM on October 11, 2019: none

Might be a compiler bug. If you have a couple of hours to debug, you can start by bisecting the tests.c file, and then the compiler flags

Sjors commented at 6:55 PM on October 11, 2019: member

I'm installing llvm@9 via homebrew, so I can test with that and 8 later.

elichai commented at 7:49 PM on October 11, 2019: contributor

Can any of you retry with ./configure --with-bignum=no? according to quick google libdyld.dylib is the dynamic linker. and the only dynamic lib here is libgmp, so I'm curious if it's related. @sjors @fjahr

and if that doesn't help then maybe ./configure --with-asm=no to make sure this is not the hand written assembly somehow

elichai commented at 8:02 PM on October 11, 2019: contributor

It seems like a lot of people are experiencing that. 2 examples: https://trac.ffmpeg.org/ticket/8073 https://forums.developer.apple.com/thread/121887

Trying with -fno-stack-check should verify if it's the same problem

Sjors commented at 8:48 AM on October 12, 2019: member

./configure CFLAGS="-fno-stack-check" does the trick! I tried a few other things below:

Upgrading to LLVM 8 or 9 didn't make the error go away.

Configuring --with-asm=no or --with-bignum=no (LLVM 8) doesn't help, neither do both:

Build Options:
  with endomorphism       = no
  with ecmult precomp     = yes
  with external callbacks = no
  with jni                = no
  with benchmarks         = yes
  with coverage           = no
  module ecdh             = no
  module recovery         = no

  asm                     = no
  bignum                  = no
  field                   = 64bit
  scalar                  = 64bit
  ecmult window size      = 15
  ecmult gen prec. bits   = 4

  CC                      = gcc
  CFLAGS                  = -g -O2 -W -std=c89 -pedantic -Wall -Wextra -Wcast-align -Wnested-externs -Wshadow -Wstrict-prototypes -Wno-unused-function -Wno-long-long -Wno-overlength-strings -fvisibility=hidden -O3
  CPPFLAGS                = -I/usr/local/opt/llvm@8/include
  LDFLAGS                 = -L/usr/local/opt/llvm@8/lib

If I do ./configure CFLAGS=-O0, remove the CFLAGS="$CFLAGS -O3" line in configure.ac and do a fresh ./autogen, now the error goes away. I can take it to -O1, it fails at -O2.

Maybe also worth noting that ./exhaustive_tests runs fine.

lldb backtrace on the default configuration shows the same stack_not_16_byte_aligned_error issue @fjahr found:

% lldb ./tests
(lldb) target create "./tests"
Current executable set to './tests' (x86_64).
(lldb) run
Process 22550 launched: '/Users/sjors/dev/secp256k1/tests' (x86_64)
warning: unable to find and load segment named '__OBJC_RO' at 0x7fff714e6000 in '/usr/lib/libobjc.A.dylib' in macosx dynamic loader plug-in.
warning: unable to find and load segment named '__OBJC_RW' at 0x7fff94e8f080 in '/usr/lib/libobjc.A.dylib' in macosx dynamic loader plug-in.
test count = 64
random seed = 7f1cc6f517f5427d7453b1c1dd0aada5
Process 22550 stopped
* thread [#1](/bitcoin-core-secp256k1/1/), queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame [#0](/bitcoin-core-secp256k1/0/): 0x00007fff67aca476 libdyld.dylib`stack_not_16_byte_aligned_error
libdyld.dylib`stack_not_16_byte_aligned_error:
->  0x7fff67aca476 <+0>: movdqa %xmm0, (%rsp)
    0x7fff67aca47b <+5>: int3   
    0x7fff67aca47c <+6>: nop    
    0x7fff67aca47d <+7>: nop    
(lldb) bt
* thread [#1](/bitcoin-core-secp256k1/1/), queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
  * frame [#0](/bitcoin-core-secp256k1/0/): 0x00007fff67aca476 libdyld.dylib`stack_not_16_byte_aligned_error
    frame [#1](/bitcoin-core-secp256k1/1/): 0x00007ffeefbff8e0
    frame [#2](/bitcoin-core-secp256k1/2/): 0x000000010006cbe6 tests`main(argc=<unavailable>, argv=<unavailable>) at tests.c:5237:5 [opt]
    frame [#3](/bitcoin-core-secp256k1/3/): 0x00007fff67ad9405 libdyld.dylib`start + 1
    frame [#4](/bitcoin-core-secp256k1/4/): 0x00007fff67ad9405 libdyld.dylib`start + 1

fjahr commented at 10:25 AM on October 12, 2019: none

./configure CFLAGS="-fno-stack-check" does the trick!

I can confirm!

fanquake commented at 9:30 PM on October 25, 2019: member

This problem isn't Catalina specific, but instead seems to be the combination of Xcode 11.1 and the macOS 10.15 SDK. I can reproduce the same issue seen here on macOS 10.14 (using Xcode 11.1 provided Clang), by targeting the macOS 10.15 SDK at compile time. i.e MACOSX_DEPLOYMENT_TARGET=10.15.

So instead of -fno-stack-check, an alternative work around for users running Catalina could instead be targeting the macOS 10.14 SDK. i.e MACOSX_DEPLOYMENT_TARGET=10.14 when compiling.

Sjors renamed this:
~~Segfault running make check on macOS Catalina~~
Segfault running make check on macOS 10.15 SDK (default for Catalina)
on Oct 26, 2019

Sjors renamed this:
~~Segfault running make check on macOS 10.15 SDK (default for Catalina)~~
Segfault running make check with macOS 10.15 SDK (default for Catalina)
on Oct 26, 2019

theuni commented at 2:13 PM on October 31, 2019: contributor

Agree with @fanquake on setting the minimum deployment as the easy fix.

I have a hunch about what may be going on here, but I don't have a 10.15 machine to test with (and upgrading right now seems unwise ;).

Here's a backtrace before the stack is smashed. Note that ___chkstk_darwin has been misaligned by the dynamic loader.

* thread [#1](/bitcoin-core-secp256k1/1/), queue = 'com.apple.main-thread', stop reason = instruction step into
    frame [#0](/bitcoin-core-secp256k1/0/): 0x000000010002716d tests`run_scalar_tests + 13
tests`run_scalar_tests:
->  0x10002716d <+13>: andq   $-0x20, %rsp
    0x100027171 <+17>: pushq  %rax
    0x100027172 <+18>: movl   $0x11e0, %eax             ; imm = 0x11E0
    0x100027177 <+23>: callq  0x100081232               ; symbol stub for: ___chkstk_darwin

Could someone who's able to reproduce try 2 potential workarounds?

# Force immediate symbol resolving, don't find chkstk_darwin lazily.
./configure CC=clang CFLAGS="-mmacosx-version-min=10.15" LDFLAGS="-Wl,-bind_at_load"

and

# Force chkstk_darwin symbols to be pulled in at link-time.
./configure CC=clang CFLAGS="-mmacosx-version-min=10.15" LDFLAGS="-Wl,-all_load"

Edit: Removed bignum options from the workarounds above, they're unnecessary for reproducing.

theuni commented at 3:11 PM on October 31, 2019: contributor

Here's a backtrace with register dump just before the ___chkstk_darwin. Note that rsp is 8-byte aligned rather than 16.

* thread [#1](/bitcoin-core-secp256k1/1/), queue = 'com.apple.main-thread', stop reason = instruction step over
    frame [#0](/bitcoin-core-secp256k1/0/): 0x000000010002d3b2 tests`run_scalar_tests + 18
tests`run_scalar_tests:
->  0x10002d3b2 <+18>: movl   $0x1200, %eax             ; imm = 0x1200
    0x10002d3b7 <+23>: callq  0x10008b472               ; symbol stub for: ___chkstk_darwin
    0x10002d3bc <+28>: subq   %rax, %rsp
    0x10002d3bf <+31>: popq   %rax
Target 0: (tests) stopped.
(lldb) bt all
* thread [#1](/bitcoin-core-secp256k1/1/), queue = 'com.apple.main-thread', stop reason = instruction step over
  * frame [#0](/bitcoin-core-secp256k1/0/): 0x000000010002d3b2 tests`run_scalar_tests + 18
    frame [#1](/bitcoin-core-secp256k1/1/): 0x000000010006c576 tests`main + 2134
    frame [#2](/bitcoin-core-secp256k1/2/): 0x00007fff6f77b3d5 libdyld.dylib`start + 1
    frame [#3](/bitcoin-core-secp256k1/3/): 0x00007fff6f77b3d5 libdyld.dylib`start + 1
(lldb) register read
General Purpose Registers:
       rax = 0x0000000000001900
       rbx = 0x0000000000001900
       rcx = 0x0000000000000018
       rdx = 0x00000000000dc1f0
       rdi = 0x0000000000000000
       rsi = 0x0000000100500000
       rbp = 0x00007ffeefbff6b0
       rsp = 0x00007ffeefbff678
        r8 = 0x0000000000000009
        r9 = 0x0000000000000000
       r10 = 0x000000000000000a
       r11 = 0x000000000000000a
       r12 = 0x000000000000006d
       r13 = 0x000000000000003f
       r14 = 0x00007ffeefbff720
       r15 = 0x0000000000000040
       rip = 0x000000010002d3b2  tests`run_scalar_tests + 18
    rflags = 0x0000000000000202
        cs = 0x000000000000002b
        fs = 0x0000000000000000
        gs = 0x0000000000000000

theuni commented at 4:15 PM on October 31, 2019: contributor

For the sake of completeness, and because Apple hasn't posted the sources yet, here's lldb's dasm of ___chkstk_darwin as pulled-in by all_load when linked against the 10.15 sdk.

I don't actually believe the contents of the function are all that interesting, as I currently suspect the problem to be a combination of dynamic stack allocation and lazy runtime symbol resolving.

Process 71731 stopped
* thread [#1](/bitcoin-core-secp256k1/1/), queue = 'com.apple.main-thread', stop reason = instruction step over
    frame [#0](/bitcoin-core-secp256k1/0/): 0x00000001000278d7 tests`run_scalar_tests + 23
tests`run_scalar_tests:
->  0x1000278d7 <+23>: callq  0x10006cdf0               ; ___chkstk_darwin
    0x1000278dc <+28>: subq   %rax, %rsp
    0x1000278df <+31>: popq   %rax
    0x1000278e0 <+32>: movq   0x76719(%rip), %rax       ; (void *)0x00007fffa60e2070: __stack_chk_guard
Target 0: (tests) stopped.
(lldb) d -a 0x10006cdf0
tests`___chkstk_darwin:
    0x10006cdf0 <+0>:  jmp    0x10006cdc0               ; __chkstk_darwin_probe
    0x10006cdf5 <+5>:  nop
    0x10006cdf6 <+6>:  nop
    0x10006cdf7 <+7>:  nop
    0x10006cdf8 <+8>:  nop
    0x10006cdf9 <+9>:  nop
    0x10006cdfa <+10>: nop
    0x10006cdfb <+11>: nop
    0x10006cdfc <+12>: nop
    0x10006cdfd <+13>: nop
    0x10006cdfe <+14>: nop
    0x10006cdff <+15>: nop
(lldb) d -a 0x10006cdc0
tests`__chkstk_darwin_probe:
    0x10006cdc0 <+0>:  pushq  %rcx
    0x10006cdc1 <+1>:  pushq  %rax
    0x10006cdc2 <+2>:  cmpq   $0x1000, %rax             ; imm = 0x1000
    0x10006cdc8 <+8>:  leaq   0x18(%rsp), %rcx
    0x10006cdcd <+13>: jb     0x10006cde7               ; <+39>
    0x10006cdcf <+15>: subq   $0x1000, %rcx             ; imm = 0x1000
    0x10006cdd6 <+22>: testq  %rcx, (%rcx)
    0x10006cdd9 <+25>: subq   $0x1000, %rax             ; imm = 0x1000
    0x10006cddf <+31>: cmpq   $0x1000, %rax             ; imm = 0x1000
    0x10006cde5 <+37>: ja     0x10006cdcf               ; <+15>
    0x10006cde7 <+39>: subq   %rax, %rcx
    0x10006cdea <+42>: testq  %rcx, (%rcx)
    0x10006cded <+45>: popq   %rax
    0x10006cdee <+46>: popq   %rcx
    0x10006cdef <+47>: retq

Sjors commented at 5:51 PM on October 31, 2019: member

Tested @theuni's workarounds on macOS 10.15.1 directly on the secp256k repo (doing make clean first, as it doesn't pick up changes in ./configure):

# Force immediate symbol resolving, don't find chkstk_darwin lazily. ./configure CC=clang CFLAGS="-mmacosx-version-min=10.15" LDFLAGS="-Wl,-bind_at_load"

Works

# Force chkstk_darwin symbols to be pulled in at link-time. ./configure CC=clang CFLAGS="-mmacosx-version-min=10.15" LDFLAGS="-Wl,-all_load"

Works

Tested on Bitcoin repo master @ 1c5e0ccabae10eaab21960aece3c6f70cb204e58 . This first one works, the second causes trouble:

make
...
  CXXLD    bench/bench_bitcoin
duplicate symbol 'ParseFixedPoint(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, long long*)' in:
    libbitcoin_util.a(libbitcoin_util_a-strencodings.o)
    libbitcoin_consensus.a(libbitcoin_consensus_a-strencodings.o)
duplicate symbol 'ParseInt64(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, long long*)' in:
    libbitcoin_util.a(libbitcoin_util_a-strencodings.o)
    libbitcoin_consensus.a(libbitcoin_consensus_a-strencodings.o)
duplicate symbol 'FormatParagraph(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned long, unsigned long)' in:
    libbitcoin_util.a(libbitcoin_util_a-strencodings.o)
    libbitcoin_consensus.a(libbitcoin_consensus_a-strencodings.o)
duplicate symbol 'itostr(int)' in:
    libbitcoin_util.a(libbitcoin_util_a-strencodings.o)
    libbitcoin_consensus.a(libbitcoin_consensus_a-strencodings.o)
duplicate symbol 'HexDigit(char)' in:
    libbitcoin_util.a(libbitcoin_util_a-strencodings.o)
    libbitcoin_consensus.a(libbitcoin_consensus_a-strencodings.o)
duplicate symbol 'ParseHex(char const*)' in:
    libbitcoin_util.a(libbitcoin_util_a-strencodings.o)
    libbitcoin_consensus.a(libbitcoin_consensus_a-strencodings.o)
duplicate symbol 'IsHex(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)' in:
    libbitcoin_util.a(libbitcoin_util_a-strencodings.o)
    libbitcoin_consensus.a(libbitcoin_consensus_a-strencodings.o)
duplicate symbol 'ParseHex(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)' in:
    libbitcoin_util.a(libbitcoin_util_a-strencodings.o)
    libbitcoin_consensus.a(libbitcoin_consensus_a-strencodings.o)
duplicate symbol 'atoi(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)' in:
    libbitcoin_util.a(libbitcoin_util_a-strencodings.o)
    libbitcoin_consensus.a(libbitcoin_consensus_a-strencodings.o)
duplicate symbol 'atoi64(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)' in:
    libbitcoin_util.a(libbitcoin_util_a-strencodings.o)
    libbitcoin_consensus.a(libbitcoin_consensus_a-strencodings.o)
ld: 30 duplicate symbols for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
...  (etc, etc)

Sjors commented at 6:28 PM on October 31, 2019: member

The first workaround fails however when configured with --enable-debug:

bench/bench_bitcoin -evals=1 -scaling=0 > /dev/null
WARNING: This is a debug build - may result in slower benchmarks.
/Applications/Xcode.app/Contents/Developer/usr/bin/make  check-TESTS
./build-aux/test-driver: line 107: 63129 Segmentation fault: 11  "$@" > $log_file 2>&1
FAIL: tests

Or may it doesn't really "fail" to fix this issue, it just exposed another problem in the Bitcoin Core benchmarks?

theuni commented at 7:14 PM on October 31, 2019: contributor

Let's just focus on this repo. I don't trust that all flags get passed down through the buildsystem correctly, so let's try to avoid chasing ghosts in Core for now.

-Wl,-all_load was just for a hammer-test, as it's likely to blow up on sufficiently complicated projects, as @Sjors pointed out. If we were to go this route, we'd want to use the more targeted -Wl,-force_load instead.

-Wl,-bind_at_load should be a reasonable thing to turn on. I haven't looked into the specifics, but I assume it works pretty much like gnu ld's -Wl,-z,now, which we use wherever possible.

So, to clarify, does building with -Wl,-bind_at_load fix the existing crashes in libsecp256k1 alone?

Sjors commented at 8:20 PM on October 31, 2019: member

Both -Wl,-bind_at_load and -Wl,-all_load let me run make check for this repo.

I tried Wl,-force_load, but that doesn't build:

make
...
  CCLD     libsecp256k1.la
ld: file not found: -L/usr/local/lib
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [libsecp256k1.la] Error 1

anton-trunov cross-referenced this on Nov 12, 2019 from issue secp256k1 opam and brew installation fails by simonelnahas

anton-trunov referenced this in commit 55e48f2db6 on Nov 12, 2019

anton-trunov cross-referenced this on Nov 12, 2019 from issue secp256k1: --with-enable-recovery-module option and fix build on macOS 10.15 by anton-trunov

MarcoFalke cross-referenced this on Nov 12, 2019 from issue build: macOS toolchain update by fanquake

fanquake commented at 6:48 PM on November 13, 2019: member

I can no longer produce this crash on macOS 10.14 when targeting the 10.15 SDK, using the latest version of XCode (11.2.1). There have been some Apple Clang changes since Xcode 11.1: 11.0.0 (clang-1100.0.33.8) -> 11.0.0 (clang-1100.0.33.12), so the ___chkstk_darwin issue may have been fixed. Can anyone else confirm?

Sjors commented at 9:57 AM on November 14, 2019: member

As of Xcode 11.2 I no longer have this problem. Haven't even upgraded to 11.2.2 yet.

Sjors closed this on Nov 14, 2019

instagibbs cross-referenced this on Nov 22, 2019 from issue Client Crashes with sendtoaddress by nharmon9

theuni cross-referenced this on Apr 3, 2020 from issue scripts: add MACHO lazy bindings check to security-check.py by fanquake

AdamISZ cross-referenced this on Feb 28, 2021 from issue libsecp256k1 was not built. Exiting. by ChzenChzen