build: LLD based macOS toolchain #21778

This switches us to using a LLD based toolchain for macOS builds.

Benefits

• Less complicated macOS toolchain.
• No longer beholden to Apple releasing it’s source for cctools, ld64 & libtapi.
• No more reliance on third parties to modify those sources for us. i.e apple-libtapi, cctools-port (cctools + ld64).

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

• #30022 (releases: use LLVM 18 for macOS by fanquake)
• #29881 (guix: use GCC 13 to builds releases by fanquake)
• #29072 (build: use -no_exported_symbols on macOS by fanquake)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

This definitely seems like a good thing to add support for, but it seems sad to have to drop support for the open source compiler at the same time as adding support for the closed source one. I don’t know enough about this to have an informed opinion, though.

EDIT: Just a misunderstanding

but it seems sad to have to drop support for the open source compiler at the same time as adding support for the closed source one.

I might be misunderstanding your comment, but the opposite is happening here. We’re migrating away from using Apples, irregularly open sourced tools (cctools, ld64), to just vanilla LLVM (lld), which is entirely open source.

I might be misunderstanding your comment, but the opposite is happening here. We’re migrating away from using Apples, irregularly open sourced tools (cctools, ld64), to just vanilla LLVM (lld), which is entirely open source.

My bad! I thought this was switching from an unsupported apple open-source toolchain to an supported apple closed source toolchain, not to a vanilla toolchain. My blinding hate of apple has kept me from seeing reality! Thanks for the clarification. This all seems very good.

Very nice! Strong concept ACK, but obviously we can’t use this until all features we need are supported. But it’s great to see that linking finally works!

For a little extra context beyond what @fanquake said above, the macOS clang+cctools build has always been a wonky outlier because we have to build our own (very unsupported) toolchain first. LD64 is apple’s libre/open-source linker, and there is a community project that maintains a build environment for cross builds.

For a few years, there has been slow development on a darwin functionality for llvm’s lld, but it’s only really picked up speed and become usable in the last year or two. This PR would bring our Apple builds much more inline with others in that it would finally be using a supported libre/open-source compiler/linker combo. @fanquake Is cctools still needed at all? LLVM ships bitcode equivalents of most binutils tools (ar, nm, ranlib, etc), I wonder if we can use those instead. Maybe they require lto?

Is cctools still needed at all? LLVM ships bitcode equivalents of most binutils tools (ar, nm, ranlib, etc), I wonder if we can use those instead. Maybe they require lto? @theuni no cctools is no longer required. I drop the native_cctools as part of this PR and we just use the LLVM tools.

@theuni no cctools is no longer required. I drop the native_cctools as part of this PR and we just use the LLVM tools.

Apologies for reading too quickly, I see now that I only looked at the first commit. Turns out the 3rd one is the one I expected :)

Rebased for #21793, and rebased on top of #19817. Made some additional changes, such as renaming the native_cctools package to native_llvm, and removed some now-redundant info from the macdeploy documentation.

Note: I’m pretty sure the use of ZERO_AR_DATE can now be dropped entirely, but have left that out for now.

Sorry for the noise. This is now pointing at LLVM 14.0.0 rc2, and no longer using binaries I compiled, given the release on GitHub has the binaries we need.

The build is currently failing for Qt, something to follow up with:

01 warning generated.
1x86_64-apple-darwin-ranlib: error: Invalid option: '-no_warning_for_no_symbols'
2make[4]: *** [Makefile:436: ../../lib/libQt5Network.a] Error 1

The build is currently failing for Qt, something to follow up with:

01 warning generated.
1x86_64-apple-darwin-ranlib: error: Invalid option: '-no_warning_for_no_symbols'
2make[4]: *** [Makefile:436: ../../lib/libQt5Network.a] Error 1

I think the https://github.com/qt/qtbase/commit/9a1eb86f3d4cd4b0d99f053f9bc9d304289b99f7 commit is responsible for that error. Reverting of its changes in mkspecs/features/qt_build_config.prf should fix the error.

I’m still wondering why does the Qt build system recognize llvm’s clang as Apple’s clang:

0$ make -C depends qt_configured HOST=x86_64-apple-darwin
1...
2Configure summary:
3
4Building on: linux-g++ (x86_64, CPU features: mmx sse sse2)
5Building for: macx-clang-linux (x86_64, CPU features: cx16 mmx sse sse2 sse3 ssse3 sse4.1)
6Target compiler: clang (Apple) 14.0.0
7Configuration: cross_compile sse2 aesni sse3 ssse3 sse4_1 sse4_2 avx avx2 avx512f avx512bw avx512cd avx512dq avx512er avx512ifma avx512pf avx512vbmi avx512vl f16c largefile precompile_header rdrnd rdseed shani silent x86SimdAlways release c++11 c++14 c++17 c++1z reduce_exports static stl

?

Here is the relevant Qt’s code:

0#  ifdef __apple_build_version__ // Apple clang has other version numbers
1#    define COMPILER_STRING "Clang " __clang_version__ " (Apple)"
2#  else
3#    define COMPILER_STRING "Clang " __clang_version__
4#  endif

Rebased to include #24613.

Reverting of its changes in mkspecs/features/qt_build_config.prf should fix the error.

I’ve added a qt patch that does this. However note that bitcoin-qt still isn’t being built because of duplicate symbol issues when running our configure checks.

Note that this week the review club is looking at this PR. There are improvements / changes that could be made here, some which I noticed while writing the questions, but I’ll leave any followup changes until after Wednesday.

However note that bitcoin-qt still isn’t being built because of duplicate symbol issues when running our configure checks.

0x86_64-apple-darwin-ld: error: duplicate symbol: __Z10lcQpaFontsv
1>>> defined in /home/hebasto/GitHub/bitcoin/depends/x86_64-apple-darwin/lib/libQt5ThemeSupport.a(qgenericunixthemes.o)
2>>> defined in /home/hebasto/GitHub/bitcoin/depends/x86_64-apple-darwin/lib/libQt5FontDatabaseSupport.a(qfontengine_coretext.o)

Concept ACK

After the PR review club I agree moving from a non-standard toolchain to using clang and lld is preferable now that lld can be used for this purpose.

Bump to https://github.com/llvm/llvm-project/tree/llvmorg-14.0.0?

I will do once binaries are available, currently there is only aarch64.

Rebased and dropped the redundant preprocessing step.

Bump to https://github.com/llvm/llvm-project/tree/llvmorg-14.0.0?

Rebased and done now.

Update ZERO_AR_DATE docs / comments.

lld (and ld64) also respects ZERO_AR_DATE, so I’ve resolved this TODO by removing the documentation block about cctools and ar.

Guix builds

Tested 46d8f32086d89de919036613b8cb437bc71a9286 on Ubuntu 22.04.

The f41493b9023acb6f3c237f6609f34f99d7e52dc1 commit introduced an error on my system:

0$ make -C depends HOST=x86_64-apple-darwin boost FORCE_USE_SYSTEM_CLANG=1
1...
2Preprocessing boost...
3Configuring boost...
4tar: option requires an argument -- 'f'
5Try 'tar --help' or 'tar --usage' for more information.
6...

I’ve rebased this, as well as updated our depends build to use LLVM 16.0.0. Guix doesn’t yet have LLVM 16 available, either in master or core-updates, so for now, I’ve bumped the time-machine further, so that it gives us LLVM 15.0.7 there.

With LLVM 16.0.0 in depends, this can be tested / played around with further. However I’ll move this back to draft for now.

• -bind_at_load support.

I explored this today, and it seems this feature is deprecated. It has been replaced by -fixup_chains as of ld64 v512.4. Here’s a write-up on the new functionality. For lld (in conflict with what the site above says), the new functionality kicks in when OSX_MIN_VERSION >= 11.0.

• Fix building libbitcoinconsensus.

By this, I believe you mean unsupported lld -r? If so, that should be unnecessary with CMake.

Edit: Core is a beast to test with, but using libsecp256k1 as a test-case: Playing around with the link options: -Wl,-no_fixup_chains -Wl,-bind-at-load (and their opposites) do as expected as checked with: objdump -t --lazy-bind libsecp256k1.2.dylib. When using -fixup_chains (the new default for >=11.0) the lazy bind table is empty as we’d hope.

I’ve experimented and implemented the above here: https://github.com/theuni/bitcoin/commits/21778 @fanquake I think we want https://github.com/theuni/bitcoin/commit/a0427fc986294ef81605b4c44b54c3ec9825bf48 regardless, which fixes the libbitcoinconsensus linking issue. tl;dr: It makes lld linking work like ld64 by skipping a busted test. The -r was a red herring as a result of that busted test.

Shall I go ahead and PR that commit separately?

Edit: So, now the only thing stopping us from using vanilla llvm is a minimum version bump, right? So.. I guess it’s time for that discussion :)

10.15 has been unsupported for 8 months now…

Shall I go ahead and PR that commit separately? @theuni sgtm. I’ve got some related compile flag changes in #27493, but I’m going to just combine most of the relevant changes here. If we get the LLD switchover done for 26.x, then we can skip another bump in the interim.

For now, I’ve rebased this PR on top of all of your other commits above, and reverted back to LLVM 15.0.6 in depends. Also updated the PR description.

did 13 or 14 have the required options?

IIRC I think so. I’l go back and check, so we can establish a minimum required version.

Hmm, Looks like fixup_chains didn’t go into lld until v16. So I guess that would be our minimum version.

Which I suppose means that fixup_chains isn’t actually being used in this PR. I’ll check around in the logs and if I’m right about that I’ll work up a test that this PR should fail as-is.

Edit: yep:

checking whether the linker accepts -Wl,-bind_at_load… no checking whether the linker accepts -Wl,-fixup_chains… no

I’ll poke around with Lief to see if we can enforce fixup_chains there.

powerpc64-linux-gnu build is failing with:

 0Backtrace:
 1In srfi/srfi-1.scm:
 2   586:29 19 (map1 (#<<manifest-entry> name: "python-lief" versio?> ?))
 3   586:17 18 (map1 (#<<manifest-entry> name: "powerpc64-linux-gnu-t?>))
 4In guix/profiles.scm:
 5  1930:19 17 (_ _)
 6In guix/packages.scm:
 7  1348:17 16 (supported-package? #<package powerpc64-linux-gnu-tool?> ?)
 8In guix/memoization.scm:
 9    101:0 15 (_ #<hash-table 7f27a126c4c0 282/443> #<package powerp?> ?)
10In guix/packages.scm:
11  1318:12 14 (_)
12In srfi/srfi-1.scm:
13   460:18 13 (fold #<procedure 7f27a03a2560 at guix/packages.scm:13?> ?)
14In guix/packages.scm:
15  1322:42 12 (_ _ ("x86_64-linux" "i686-linux"))
16In guix/memoization.scm:
17    101:0 11 (_ #<hash-table 7f27a126c4c0 282/443> #<package glibc-?> ?)
18In guix/packages.scm:
19  1318:12 10 (_)
20In srfi/srfi-1.scm:
21   460:18  9 (fold #<procedure 7f27a03a75c0 at guix/packages.scm:13?> ?)
22In guix/packages.scm:
23  1322:42  8 (_ _ ("x86_64-linux" "i686-linux"))
24In guix/memoization.scm:
25    101:0  7 (_ #<hash-table 7f27a126c4c0 282/443> #<package linux-?> ?)
26In guix/packages.scm:
27  1326:37  6 (_)
28  1588:16  5 (package->bag _ _ _ #:graft? _)
29  1693:43  4 (thunk)
30In guix/build-system/gnu.scm:
31   187:19  3 (arguments _)
32In gnu/packages/cross-base.scm:
33   389:34  2 (loop (#:phases (modify-phases %standard-phases # ?) ?) #)
34In ice-9/boot-9.scm:
35  1685:16  1 (raise-exception _ #:continuable? _)
36  1685:16  0 (raise-exception _ #:continuable? _)
37
38ice-9/boot-9.scm:1685:16: In procedure raise-exception:
39ERROR:
40  1. &platform-not-found-error: "powerpc64-linux-gnu"

It’s not defined in https://github.com/guix-mirror/guix/blob/7c7853d269fe53271dd35d5bd941d18a2cb55120/guix/platforms/powerpc.scm

powerpc64-linux-gnu build is failing with:

Thanks, thats’s already mentioned as a known issue here: #27897, which also does a time-machine bump. You don’t need to build for powerpc to test the two macOS build, if you want to test the changes here.

Guix builds

Rebased past #27999. Now that we require fixup_chains, we also require at least LLVM 16+ (which doesn’t yet exist in Guix). Hopefully we’ll have the infra to construct it ourselves shortly: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=64100 (which will mean a more recent time-machine bump). https://reviews.llvm.org/D151926 / https://reviews.llvm.org/D151944 has also landed in LLVM (17.x), so we can backport if needed.

In the meantime, I’ve rebased this, and updated depends to use LLVM 16.0.4. There aren’t currently x86_64 binaries available for 16.0.5 or 16.0.6.

Guix builds (on x86_64)

Rebased on #28580 (LLVM 17 in Guix) and #28783, which should be the last blockers for this.

Why #28783 is considered as a blocker for this PR?

Why #28783 is considered as a blocker for this PR?

LLD will output spurious warnings about not implementing bind_at_load, which we can avoid by merging #28783.

Guix building df71f448cdaf429f54367ab251e21e628c1903d3 is having issues for me:

 0Extracting libevent...
 1/home/guix/depends-sources-cache/libevent-2.1.12-stable.tar.gz: OK
 2Preprocessing libevent...
 3Configuring libevent...
 4checking for a BSD-compatible install... /home/guix/.guix-profile/bin/install -c
 5checking whether build environment is sane... yes
 6checking for x86_64-apple-darwin-strip... no
 7checking for strip... strip
 8configure: WARNING: using cross tools not prefixed with host triplet
 9checking for a thread-safe mkdir -p... /home/guix/.guix-profile/bin/mkdir -p
10checking for gawk... gawk
11checking whether make sets $(MAKE)... yes
12checking whether make supports nested variables... yes
13checking whether make supports nested variables... (cached) yes
14checking whether make supports the include directive... yes (GNU style)
15checking for x86_64-apple-darwin-gcc... env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH -u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH -u LIBRARY_PATH /home/guix/.guix-profile/bin/clang --target=x86_64-apple-darwin -mmacosx-version-min=11.0 -B/bitcoin/depends/x86_64-apple-darwin/native/bin -mlinker-version=711 -isysroot/home/guix/SDKs/Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers -iwithsysroot/usr/include -iframeworkwithsysroot/System/Library/Frameworks
16checking whether the C compiler works... no
17configure: error: in `/bitcoin/depends/work/build/x86_64-apple-darwin/libevent/2.1.12-stable-ab81f19d3b8':
18configure: error: C compiler cannot create executables
19See `config.log' for more details
20make: *** [funcs.mk:292: /bitcoin/depends/x86_64-apple-darwin/.libevent_stamp_configured] Error 77
21make: Leaving directory '/bitcoin/depends'

Can add more details…

Can add more details…

I’d suggest reviewing the base PRs instead, for now (pretty sure this is just -fuse-ld usage that needs tidying up).

Rebased after #28880. Cleaned up the commits a bit. Doesn’t work yet, because Qt doesn’t build..

Also rethinking the best approach here a bit, as we’ve now essentially got to accomodate 3 different builds for macOS:

• Building for mac, on mac, using Apple Clang & ld64 (which also doesn’t like -fuse-ld=lld)
• Building for mac, on Linux, using our downloaded LLVM (Clang + lld + tools)
• Building for mac, in Guix, using Guix’s Clang + our downloaded lld + tools??
  • Or shift further into using all Guix linker + tools, or, try drop Guix Clang?

cc @theuni @TheCharlatan.

• Building for mac, in Guix, using Guix’s Clang + our downloaded lld + tools??

  • Or shift further into using all Guix linker + tools, or, try drop Guix Clang?

Would you say that (temporarily) using Apple’s Clang instead of Guix would still make this PR an overall improvement for macOS reproducibility? If so, and if makes things easier, that could make sense.

Guix builds (on x86_64)

• Building for mac, on Linux, using our downloaded LLVM (Clang + lld + tools)

• Building for mac, in Guix, using Guix’s Clang + our downloaded lld + tools??

  • Or shift further into using all Guix linker + tools, or, try drop Guix Clang?

I think the benefit of using guix (over downloading a random binary blob from someone else’s computer) is that cross compilation to macOS works on more architectures than the uploads of the llvm binaries. (For example riscv64)

So simply requiring guix to do a macOS depends build (and nuking clang from depends, or keeping it as a fallback, if needed) seems preferable?

and nuking clang from depends,

After discussion today with @theuni @hebasto @TheCharlatan, I’m actually about to open a PR to do exactly this.

Pushed up a few more improvements.

Somewhat annoyingly, some distros (Ubuntu) only ship llvm-libtool-darwin with a version suffix, i.e llvm-libtool-darwin-17, however many others (Fedora, Alpine etc) do not do this. I think we should just set darwin_LIBTOOL=llvm-libtool-darwin as a best effort, and I’ve added a note to the depends readme, that if anyone sees issues, they should set LIBTOOL=llvm-libtool-darwin-version when calling make.

The CI failure here is a bit odd, because it only happens for ZeroMQ, and only when using what looks like clang-14? This doesn’t happen when using newer Clang, or when doing a Guix build. The actual error is:

 0configure:20428: checking whether the C++ compiler works
 1configure:20445: env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH -u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH -u LIBRARY_PATH /usr/bin/clang++ --target=arm64-apple-darwin -mmacosx-version-min=11.0 -mlinker-version=711 -isysroot/bitcoin/depends/SDKs/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers -nostdlibinc -iwithsysroot/usr/include/c++/v1 -iwithsysroot/usr/include -iframeworkwithsysroot/System/Library/Frameworks -std=gnu++11 -o conftest -pipe -std=c++20 -O2   -I/bitcoin/depends/arm64-apple-darwin/include -Wl,-platform_version,macos,11.0,14.0 -fuse-ld=lld  -L/bitcoin/depends/arm64-apple-darwin/lib conftest.cpp  >&5
 2ld64.lld: error: library not found for -lstdc++
 3clang: error: linker command failed with exit code 1 (use -v to see invocation)
 4configure:20445: $? = 1
 5configure: failed program was:
 6| /* confdefs.h */
 7| #define PACKAGE_NAME "zeromq"
 8| #define PACKAGE_TARNAME "zeromq"
 9| #define PACKAGE_VERSION "4.3.5"
10| #define PACKAGE_STRING "zeromq 4.3.5"
11| #define PACKAGE_BUGREPORT "zeromq-dev@lists.zeromq.org"
12| #define PACKAGE_URL ""
13| #define PACKAGE "zeromq"
14| #define VERSION "4.3.5"
15| #define HAVE_STDIO_H 1
16| #define HAVE_STDLIB_H 1
17| #define HAVE_STRING_H 1
18| #define HAVE_INTTYPES_H 1
19| #define HAVE_STDINT_H 1
20| #define HAVE_STRINGS_H 1
21| #define HAVE_SYS_STAT_H 1
22| #define HAVE_SYS_TYPES_H 1
23| #define HAVE_UNISTD_H 1
24| #define STDC_HEADERS 1
25| #define HAVE_DLFCN_H 1
26| #define LT_OBJDIR ".libs/"
27| #define HAVE_CXX11 1
28| /* end confdefs.h.  */
29| 
30| int
31| main (void)
32| {
33| 
34|   ;
35|   return 0;
36| }
37configure:20455: result: no
38configure:20459: error: Unable to find a working C++ compiler

It’s failing to find -lstdc++ while running the AX_CXX_COMPILE_STDCXX_11 macro (zeromq also ships an ancient version of AX_CXX_COMPILE_STDCXX), but it shouldn’t be trying to use that in any case. We want libc++.

The CI failure here is a bit odd, because it only happens for ZeroMQ, and only when using what looks like clang-14?

I think the solution to this is just to keep our current Clang version requirement for macOS cross-compilation. Currently, we vendor our own Clang (17) and use 17 in Guix. Switching to using system Clang doesn’t mean we now need to support clang-13 + when building for macOS, because we never did previously, and no-one building Core could have been using a version other than the only one we supported (17).

So, when switching to system Clang, we retain the Clang 17 requirement (for macOS). The CI should also be using 17, because it should match what we are doing in Guix. So I’ve now switched it to a newer version of Ubuntu (24.04), which ships clang-17 by default.

In regards to the ZeroMQ failure, my suspicion is that the failure is realted to -nostdlibinc and earlier versions of Clang.

Re libtool: As far as I can see, it’s only used for miniupnpc/libnatpnp?

If that’s the case, I don’t think it’s worth the trouble of having users work around it.

I’d suggest we either:

• patch out libtool in the Makefiles and use ar, then drop our libtool machinery entirely.
• Switch to their cmake builds where I assume libtool is unused.

I have a branch with commits to do the latter as part of a larger CMake switch, but maybe now is a good time to take those two? (If so, ignore the native_cmake part, that was just to get myself bootstrapped.)

Edit: If there’s interest in the above I can PR it separately.

Edit: If there’s interest in the above I can PR it separately.

Sure, I can rebase on top of that PR, if you want to open it.

The remaining issue here is with make -C depends CC= CXX= usage. If we override, we lose the c(xx) flags which are currently embedded into darwin_C(XX), which means things don’t compile. This is why this PR works fine, if you just invoke make -C depends HOST=arm64-apple-darwin, with clang installed, but if you make -C depends HOST=arm64-apple-darwin CC=clang-17 CXX=clang++-17, like we want to do in the CI, things don’t quite work because we loose flags, like --target, and clang things it’s compiling for Linux etc. So we’ll need to move (some) flags into darwin_C(XX)FLAGS.

If you Guix build this branch now, everything will compile fine, but it will fail during the test-security check stage, that issue should also be easily resolved.

Would still like to see them PR’d separately, but I cherry-picked the two relevant commits, dropped native_cmake, and have dropped anything libtool related here.

It’s been quite a while since I worked on those, I don’t remember if they were completely merge-ready (I suspect windows cross might be broken for ex.). Playing with them locally now.

Would still like to see them PR’d separately, but I cherry-picked the two relevant commits, dropped native_cmake, and have dropped anything libtool related here.

It’s been quite a while since I worked on those, I don’t remember if they were completely merge-ready (I suspect windows cross might be broken for ex.). Playing with them locally now.

Here’s a cleaned up version: https://github.com/theuni/bitcoin/commits/depends-no-libtool/ They needed an update after 63c0c4ff10b2f6ed8510ef372a5b1f6a6494b179.

Also adds a commit which actually removes the libtool machinery.

Did quick tests for mac and mingw locally to make sure that proper flags were applied, looks ok to me.

Blah, it looks like the win32 filename is borked with cmake: liblibminiupnpc.a

Want to punt the libtool removal to a follow-up PR to avoid slowing this one down?

Edit: I grabbed the upstream fix: https://github.com/miniupnp/miniupnp/pull/604

Updated my branch. Up to you if you want to do this here or not.

This is why this PR works fine, if you just invoke make -C depends HOST=arm64-apple-darwin, with clang installed, but if you make -C depends HOST=arm64-apple-darwin CC=clang-17 CXX=clang++-17, like we want to do in the CI

If picking the clang version via depends is not a use case to support outside the CI, what about aliasing clang=clang-17 in the CI container? Obviously fixing depends would be cleaner, but if it is only used by CI, then it may not be needed.

The only requirement for the CI is that the version is pinned (or that this pull request is put on hold until Ubuntu bumped to clang-18, in which case it will remain pinned by itself)

Want to punt the libtool removal to a follow-up PR to avoid slowing this one down?

Looks like the cmake changes will require some additional CI changes at least. I think we can still open an additional PR, with upnpc and natpmp, and get that merged. Seems cleanest to just nuke libtool here, and avoid interim workarounds for Ubuntus binarys etc. I’ll pull your new commits in here in any case.

If picking the clang version via depends is not a use case to support outside the CI,

We do want to support that, but only with clang-17+. The issue is the flags being embedded into c/cxx, which is an artifact of previous problems. We can move the flags out, and this will be solved.

Pushed up one WIP commit. This almost works how we want it too:

• Depends builds using system clang, i.e make -C depends/ HOST=arm64-apple-darwin -j17. Tested on x86_64 Ubuntu with Ubuntu clang version 17.0.6 (3).
• Depends builds with overriden CC & CXX, i.e make -C depends/ HOST=arm64-apple-darwin CC=clang-17 CXX=clang++-17. See the CI (which fails during make deploy due to no otool), also tested on aarch64 Fedora with clang version 17.0.6 (Fedora 17.0.6-4.fc40).
• Guix builds are ~95% there, but fail during test-security-check because they invoke ld:

 0  CXXLD    bitcoin-cli
 1ld64.lld: warning: Option `-bind_at_load' is not yet implemented. Stay tuned...
 2  AR       minisketch/libminisketch.a
 3ld64.lld: warning: Option `-bind_at_load' is not yet implemented. Stay tuned...
 4ld64.lld: warning: Option `-bind_at_load' is not yet implemented. Stay tuned...
 5  CXXLD    univalue/test/unitester
 6ld64.lld: warning: Option `-bind_at_load' is not yet implemented. Stay tuned...
 7ld64.lld: warning: Option `-bind_at_load' is not yet implemented. Stay tuned...
 8ld64.lld: warning: Option `-bind_at_load' is not yet implemented. Stay tuned...
 9  CXXLD    univalue/test/object
10ld64.lld: warning: Option `-bind_at_load' is not yet implemented. Stay tuned...
11  AR       libtest_util.a
12  CXXLD    test/test_bitcoin
13ld64.lld: warning: Option `-bind_at_load' is not yet implemented. Stay tuned...
14make[2]: Leaving directory '/distsrc-base/distsrc-cb758c13eb6c-arm64-apple-darwin/src'
15make[1]: Leaving directory '/distsrc-base/distsrc-cb758c13eb6c-arm64-apple-darwin/src'
16Making all in doc/man
17make[1]: Entering directory '/distsrc-base/distsrc-cb758c13eb6c-arm64-apple-darwin/doc/man'
18make[1]: Nothing to be done for 'all'.
19make[1]: Leaving directory '/distsrc-base/distsrc-cb758c13eb6c-arm64-apple-darwin/doc/man'
20make[1]: Entering directory '/distsrc-base/distsrc-cb758c13eb6c-arm64-apple-darwin'
21make[1]: Nothing to be done for 'all-am'.
22make[1]: Leaving directory '/distsrc-base/distsrc-cb758c13eb6c-arm64-apple-darwin'
23ld: unrecognised emulation mode: acosx_version_min
24Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu elf_l1om elf_k1om
25clang: error: linker command failed with exit code 1 (use -v to see invocation)
26E
27======================================================================
28ERROR: test_MACHO (__main__.TestSecurityChecks)
29----------------------------------------------------------------------
30Traceback (most recent call last):
31  File "/distsrc-base/distsrc-cb758c13eb6c-arm64-apple-darwin/./contrib/devtools/test-security-check.py", line 118, in test_MACHO
32    arch = get_arch(cc, source, executable)
33  File "/distsrc-base/distsrc-cb758c13eb6c-arm64-apple-darwin/./contrib/devtools/test-security-check.py", line 45, in get_arch
34    subprocess.run([*cc, source, '-o', executable], check=True)
35  File "/gnu/store/r4ylp5184rybw9s3lpg49hb27fwgk03i-python-minimal-3.10.7/lib/python3.10/subprocess.py", line 524, in run
36    raise CalledProcessError(retcode, process.args,
37subprocess.CalledProcessError: Command '['/root/.guix-profile/bin/clang', '--target=arm64-apple-darwin', '-isysroot/root/SDKs/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers', '-nostdlibinc', '-iwithsysroot/usr/include/c++/v1', '-iwithsysroot/usr/include', '-iframeworkwithsysroot/System/Library/Frameworks', 'test1.c', '-o', 'test1']' returned non-zero exit status 1.
38
39----------------------------------------------------------------------
40Ran 1 test in 0.117s
41
42FAILED (errors=1)
43make: *** [Makefile:1417: test-security-check] Error 1

Linking of all our bins happens with lld, and this shouldn’t happen given lld-as-ld-wrapper, following up.

Squashed down the changes, and did some Guix cleanup. We can entirely remove the GCC toolchain from the macOS build now. Apart from removing unused deps, my other reason to do this was to make sure that ld wasn’t even in the build environment, so couldn’t be getting used over lld.

However this has come with a new issue. Qt now fails to build, because it’s trying to build qmake with g++, which no-longer exists. i.e

0Configuring qt...
1Creating qmake...
2make[1]: Entering directory '/bitcoin/depends/work/build/arm64-apple-darwin/qt/5.15.11-f4932583bde/qtbase/qmake'
3make[1]: g++: No such file or directory
4make[1]: *** [Makefile:250: main.o] Error 127

This would seem to be a bug in any case, because if we are using a clang toolchain, there shouldn’t be an assumption that GCC is installed. So we need to make it use clang++ instead.

Guix builds (on x86_64)

🐙 This pull request conflicts with the target branch and needs rebase.

Guix builds (on x86_64) [untrusted test-only build, possibly unsafe, not for production use]

Darwin Guix build (aarch64):

01de4299604e1711c90134a94a7ba6bc2685cc43e4acb283a7b3689afba883205  guix-build-c5f2df00a26c/output/arm64-apple-darwin/SHA256SUMS.part
17dcb77afb8dfa36c39be686827bec06bd39d197faed179b6e17d62492567ce9a  guix-build-c5f2df00a26c/output/arm64-apple-darwin/bitcoin-c5f2df00a26c-arm64-apple-darwin-unsigned.tar.gz
22f2bcf41e7b1f7ac28258e6db4600b0230db515f9549dae1c714948c9986c433  guix-build-c5f2df00a26c/output/arm64-apple-darwin/bitcoin-c5f2df00a26c-arm64-apple-darwin-unsigned.zip
393d42c4cc8c01f882bba24c72823974347b1baed77f4f89a48869dbbcd07f236  guix-build-c5f2df00a26c/output/arm64-apple-darwin/bitcoin-c5f2df00a26c-arm64-apple-darwin.tar.gz
4fc3c07f5d43ea8527c9a708bbd58075bb390434c52f7a628208ca0f98dd23308  guix-build-c5f2df00a26c/output/dist-archive/bitcoin-c5f2df00a26c.tar.gz
54e038157c4941ea9d880e034b31ae235ab324fe68de200a62e74098b6e91a096  guix-build-c5f2df00a26c/output/x86_64-apple-darwin/SHA256SUMS.part
66c1e094668b084be9af4069328750c42a330890c90632af485be152426e6094f  guix-build-c5f2df00a26c/output/x86_64-apple-darwin/bitcoin-c5f2df00a26c-x86_64-apple-darwin-unsigned.tar.gz
7ce59288aee5ee1bb5adac2c3bca5ed72289a9ac15ea2f722a58c18475336b9fc  guix-build-c5f2df00a26c/output/x86_64-apple-darwin/bitcoin-c5f2df00a26c-x86_64-apple-darwin-unsigned.zip
876aec809181c6097e6a07de9f0508df14432c7982cf8a553c3751a2ae34d7b14  guix-build-c5f2df00a26c/output/x86_64-apple-darwin/bitcoin-c5f2df00a26c-x86_64-apple-darwin.tar.gz

Guix builds (on x86_64) [untrusted test-only build, possibly unsafe, not for production use]

Guix build (aarch64):

 0cf99b6597c64c4d0cd75176d261a53f24779e8c3dbe9e891d9d166ae9d00d182  guix-build-f58a8678957e/output/aarch64-linux-gnu/SHA256SUMS.part
 1922e38e2b1a091f20caa014600936689fb9f14bf56fbd80e4c836c518897ff1f  guix-build-f58a8678957e/output/aarch64-linux-gnu/bitcoin-f58a8678957e-aarch64-linux-gnu-debug.tar.gz
 2256c68e14a905aa6933a2dc83cad636a14ee99ac4661a287e7817efe421965ab  guix-build-f58a8678957e/output/aarch64-linux-gnu/bitcoin-f58a8678957e-aarch64-linux-gnu.tar.gz
 350eafd4c679cab81c40ed02b1ad4aafb53f8ccf40c497e084596a078977221ed  guix-build-f58a8678957e/output/arm-linux-gnueabihf/SHA256SUMS.part
 4bb89cc308567954302710c809b09992befbbf589c2fa3c33be4b575ad5c619d5  guix-build-f58a8678957e/output/arm-linux-gnueabihf/bitcoin-f58a8678957e-arm-linux-gnueabihf-debug.tar.gz
 5a83ebe295c39f7581b0d12f5ef58059a54ef0950673a3d0444a73983f83f2468  guix-build-f58a8678957e/output/arm-linux-gnueabihf/bitcoin-f58a8678957e-arm-linux-gnueabihf.tar.gz
 67db661a4ecabd812a1630b25062f5cfda15564e8ddbe81c8f5aee7e1d0c670c6  guix-build-f58a8678957e/output/arm64-apple-darwin/SHA256SUMS.part
 7dc4e3673d1aedc94e6e59aa838508dbfcb3a0b1e50af48f94ac3ed691871acf8  guix-build-f58a8678957e/output/arm64-apple-darwin/bitcoin-f58a8678957e-arm64-apple-darwin-unsigned.tar.gz
 80899c00ebeac5709fbd38424c9eb93a75da658b9af55be7688dbaeefb005bdc9  guix-build-f58a8678957e/output/arm64-apple-darwin/bitcoin-f58a8678957e-arm64-apple-darwin-unsigned.zip
 9aa51ca52f7d6289c38417ac83919825ebef540b3681bfa5e95b954d0bbc1ece5  guix-build-f58a8678957e/output/arm64-apple-darwin/bitcoin-f58a8678957e-arm64-apple-darwin.tar.gz
1041f187eeba20db1087328fd9bf21cd60ea869aa6b0288864a6c42c585022d7ed  guix-build-f58a8678957e/output/dist-archive/bitcoin-f58a8678957e.tar.gz
11cd1be902c2d0af6967bed3fbc3c2b5064580333403be6b395b00f294fb558e7d  guix-build-f58a8678957e/output/powerpc64-linux-gnu/SHA256SUMS.part
120a3a40bce2e52e1c60859504042f90e7a7fc63fa98258f19edc08de358933e94  guix-build-f58a8678957e/output/powerpc64-linux-gnu/bitcoin-f58a8678957e-powerpc64-linux-gnu-debug.tar.gz
13011cb745d14aae9f96fd6610204ea6670f2b4dde723d8ab30241996761253fd1  guix-build-f58a8678957e/output/powerpc64-linux-gnu/bitcoin-f58a8678957e-powerpc64-linux-gnu.tar.gz
14b50afc980b4f07b7452577b2a84c746e6c282c6a1239b122155d31b7656346e4  guix-build-f58a8678957e/output/riscv64-linux-gnu/SHA256SUMS.part
152fd021da35b43c748cfa560da6d6755dbed370767fceb11eb8427767ab3c6fcc  guix-build-f58a8678957e/output/riscv64-linux-gnu/bitcoin-f58a8678957e-riscv64-linux-gnu-debug.tar.gz
16c4d7af36e66eabc9069db5774629aa06882c2d6be464100057d58a16a13fc119  guix-build-f58a8678957e/output/riscv64-linux-gnu/bitcoin-f58a8678957e-riscv64-linux-gnu.tar.gz
17038fca155f03bc2ad159ac4546e767b0fab83e61db951df208d33b575be2fb31  guix-build-f58a8678957e/output/x86_64-apple-darwin/SHA256SUMS.part
189e50d7f1e574e8c9ab55f734be6d56ac196cd014f70d88a675044422f13e4cb0  guix-build-f58a8678957e/output/x86_64-apple-darwin/bitcoin-f58a8678957e-x86_64-apple-darwin-unsigned.tar.gz
191edf854f8a0cbe52a36bd2ee4990b38220567038d3585231d20d6e91d076400d  guix-build-f58a8678957e/output/x86_64-apple-darwin/bitcoin-f58a8678957e-x86_64-apple-darwin-unsigned.zip
20697e8b6a825f1577c275195ea6f77bd75f475cac900a1726e688651d6debafba  guix-build-f58a8678957e/output/x86_64-apple-darwin/bitcoin-f58a8678957e-x86_64-apple-darwin.tar.gz
21fc6aaa438adab766507f56cd997cd6071a3cbfe0ec6a1a16a16c60f0d7a3e9b4  guix-build-f58a8678957e/output/x86_64-linux-gnu/SHA256SUMS.part
224cb2ea625df852bed28dc569427439a57788f11a652e7be75c7f9e825ac37882  guix-build-f58a8678957e/output/x86_64-linux-gnu/bitcoin-f58a8678957e-x86_64-linux-gnu-debug.tar.gz
23b361dddbf2af228ce466e34d0e87a6a2fd51b695e89c6cb03b77e3beaa9c7cf1  guix-build-f58a8678957e/output/x86_64-linux-gnu/bitcoin-f58a8678957e-x86_64-linux-gnu.tar.gz
24f73b62162f6e55530c0d785793de85e67a40490174658d70f1004bee78b7a60a  guix-build-f58a8678957e/output/x86_64-w64-mingw32/SHA256SUMS.part
250f82a06fac33f8f42cd226f1b61af7157cf24cba41b5955807c71e50fa2d1afa  guix-build-f58a8678957e/output/x86_64-w64-mingw32/bitcoin-f58a8678957e-win64-debug.zip
26edbc360fc6a4cb3e7b720dfdc3612f8dcb3eb7d827b6de2a964e5a567ab89914  guix-build-f58a8678957e/output/x86_64-w64-mingw32/bitcoin-f58a8678957e-win64-setup-unsigned.exe
2725d517098b25269ab895a443852abc103d856d9db165dce7ede595533ba077e2  guix-build-f58a8678957e/output/x86_64-w64-mingw32/bitcoin-f58a8678957e-win64-unsigned.tar.gz
282bcf90f2c17016f026250d1220a8524a59d165f005f8c1fc153c851e385ff125  guix-build-f58a8678957e/output/x86_64-w64-mingw32/bitcoin-f58a8678957e-win64.zip

Guix builds (on x86_64) [untrusted test-only build, possibly unsafe, not for production use]