depends: build LTO support into Apple's ld64 #19530

This didn't work for a few reasons (various toolchain compatibility issues) the last time I tested it, but after the last round of bumps it works with no apparent issues.

Note that this does not enable LTO by default in any way, only hooks up the machinery for -flto to work correctly when specified.

Lines were split for an easier rebase after #17919 is merged.

Concept ACK 🚀

Rebased after #17919 .

Also added another commit after https://github.com/tpoechtrager/cctools-port/pull/85 was merged upstream to fix lto detection with external clang.

Checked out this branch at https://github.com/bitcoin/bitcoin/pull/19530/commits/5962522fbcb7b72c269fc271ac4463791a103918 and did a make -C depends/ -j6 HOST=x86_64-apple-darwin16. I'm seeing:

[ 97%] Building C object genisoimage/CMakeFiles/genisoimage.dir/endian.o
[ 97%] Building C object genisoimage/CMakeFiles/genisoimage.dir/sha1.o
[ 97%] Building C object genisoimage/CMakeFiles/genisoimage.dir/sha256.o
[ 98%] Building C object genisoimage/CMakeFiles/genisoimage.dir/sha512.o
[ 98%] Building C object genisoimage/CMakeFiles/genisoimage.dir/checksum.o
[100%] Linking C executable genisoimage
/usr/bin/ld: CMakeFiles/genisoimage.dir/apple.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/boot.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/desktop.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/dvd_file.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/dvd_reader.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/eltorito.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/files.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/hash.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/ifo_read.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/joliet.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/mac_label.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/multi.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/name.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/rock.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/scsi.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/stream.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/tree.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/udf.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/volume.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/write.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/boot-alpha.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/boot-hppa.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/boot-mips.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/jte.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/rsync.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/boot-mipsel.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/checksum.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
collect2: error: ld returned 1 exit status

Have not investigated further.

@fanquake Strange... this is fine locally, and our c-i didn't have any trouble with it either.

Is the problem locally reproducible?

For me, builds of bitcoind and bitcoin-qt even succeed with lto enabled for dependencies as well with the following hack:

diff --git a/depends/hosts/darwin.mk b/depends/hosts/darwin.mk
index 6099fd4c71..c4180ac2f8 100644
--- a/depends/hosts/darwin.mk
+++ b/depends/hosts/darwin.mk
@@ -28,8 +28,8 @@ OSX_SDK=$(SDK_PATH)/Xcode-$(XCODE_VERSION)-$(XCODE_BUILD_ID)-extracted-SDK-with-
 #         https://reviews.llvm.org/D64089, we should use that instead. Read the
 #         differential summary there for more details.
 #
-darwin_CC=clang -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK) -mlinker-version=$(LD64_VERSION) -B$(build_prefix)/bin
-darwin_CXX=clang++ -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK) -stdlib=libc++ -mlinker-version=$(LD64_VERSION) -B$(build_prefix)/bin -nostdinc++ -isystem $(OSX_SDK)/usr/include/c++/v1
+darwin_CC=clang -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK) -mlinker-version=$(LD64_VERSION) -B$(build_prefix)/bin -flto=thin
+darwin_CXX=clang++ -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK) -stdlib=libc++ -mlinker-version=$(LD64_VERSION) -B$(build_prefix)/bin -nostdinc++ -isystem $(OSX_SDK)/usr/include/c++/v1 -flto=thin

 darwin_CFLAGS=-pipe
 darwin_CXXFLAGS=$(darwin_CFLAGS)

It's almost suspicious how well/easily that worked. I'll dig further to make sure it's actually doing full lto under the hood.

<!--a722867cd34abeea1fadc8d60700f111-->

Gitian builds

@fanquake Strange... this is fine locally, and our c-i didn't have any trouble with it either. Is the problem locally reproducible?

Yes. Turns out the issue is actually a change in defaults in GCC 10.1. See #19553. In the mean time, I'll retest here with an earlier GCC.

Concept ACK, even though we don't use LTO for builds currently, if it isn't unnecessary to disable it explicitly in the toolchain, we shouldn't.

Continuing the conversation from #19565:

The solution seemed to be to pass -object_path_lto through to ld64. Doing that I end up with a properly generated .dSYM:

I think I understand what's happening here. From the clang docs

  The default for :option:`-flto` is "full", in which the
  LLVM bitcode is suitable for monolithic Link Time Optimization (LTO), where
  the linker merges all such modules into a single combined module for
  optimization. With "thin", :doc:`ThinLTO <../ThinLTO>`
  compilation is invoked instead.

  .. note::

     On Darwin, when using :option:`-flto` along with :option:`-g` and
     compiling and linking in separate steps, you also need to pass
     ``-Wl,-object_path_lto,<lto-filename>.o`` at the linking step to instruct the
     ld64 linker not to delete the temporary object file generated during Link
     Time Optimization (this flag is automatically passed to the linker by Clang
     if compilation and linking are done in a single step). This allows debugging
     the executable as well as generating the ``.dSYM`` bundle using :manpage:`dsymutil(1)`.

That's describing us.

The machinery to get clang to do this for you only applies when compiling and linking at the same time. Here's a comment from elsewhere explaining why:

      // Add a 'dsymutil' step if necessary, when debug info is enabled and we
      // have a compile input. We need to run 'dsymutil' ourselves in such cases
      // because the debug info will refer to a temporary object file which
      // will be removed at the end of the compilation process.

I can confirm that is working as intended with a test-case:

$ cat lto1.cpp lto2.cpp

//lto1.cpp
void MyFunc();
int main(){
MyFunc();
}
//lto2.cpp
#include <stdio.h>
void MyFunc()
{
    printf("hello world\n");
}

$ clang++ -target x86_64-apple-darwin16 -mmacosx-version-min=10.12 --sysroot /home/cory/dev/bitcoin2/depends/SDKs/Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers -stdlib=libc++ -mlinker-version=530 -B/home/cory/dev/bitcoin2/depends/x86_64-apple-darwin16/native/bin -nostdinc++ -isystem /home/cory/dev/bitcoin2/depends/SDKs/Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers/usr/include/c++/v1 -flto=thin -std=c++11 lto1.cpp lto2.cpp -v

Which produces the following link line and calls dsymutil:

"/home/cory/dev/bitcoin2/depends/x86_64-apple-darwin16/native/bin/x86_64-apple-darwin16-ld" -demangle -object_path_lto /tmp/thinlto-d130c2 -lto_library /home/cory/dev/llvm-project/build/lib/libLTO.so -no_deduplicate -dynamic -arch x86_64 -platform_version macos 10.12.0 0.0.0 -syslibroot /home/cory/dev/bitcoin2/depends/SDKs/Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers -o a.out /tmp/lto1-bd26d0.o /tmp/lto2-439b81.o -lc++ -lSystem "/home/cory/dev/llvm-project/build/bin/dsymutil" -o a.out.dSYM a.out

So, it sounds as though everything is actually working as intended. We just need to add that link flag if lto and debugging are on. But we're getting ahead of ourselves, as we don't have an option for lto yet :)

ACK 5962522fbcb7b72c269fc271ac4463791a103918. The relevant option upstream is here.