This didn’t work for a few reasons (various toolchain compatibility issues) the last time I tested it, but after the last round of bumps it works with no apparent issues.
Note that this does not enable LTO by default in any way, only hooks up the machinery for -flto to work correctly when specified.
Lines were split for an easier rebase after #17919 is merged.
Note that this does not _enable_ lto by default in any way, only hooks up the
machinery for -flto to work correctly.
enable-lto-support is explicitly used for pinned-clang because we know it
works. It is neither enabled nor disabled in the external clang case so that
it can be auto-detected.
https://github.com/tpoechtrager/cctools-port/pull/85 was merged upstream, which
fixes lto detection for external clang with some Linux Distro's including
Ubuntu.
Rebased after #17919 .
Also added another commit after https://github.com/tpoechtrager/cctools-port/pull/85 was merged upstream to fix lto detection with external clang.
Checked out this branch at https://github.com/bitcoin/bitcoin/pull/19530/commits/5962522fbcb7b72c269fc271ac4463791a103918 and did a make -C depends/ -j6 HOST=x86_64-apple-darwin16. I’m seeing:
0[ 97%] Building C object genisoimage/CMakeFiles/genisoimage.dir/endian.o
1[ 97%] Building C object genisoimage/CMakeFiles/genisoimage.dir/sha1.o
2[ 97%] Building C object genisoimage/CMakeFiles/genisoimage.dir/sha256.o
3[ 98%] Building C object genisoimage/CMakeFiles/genisoimage.dir/sha512.o
4[ 98%] Building C object genisoimage/CMakeFiles/genisoimage.dir/checksum.o
5[100%] Linking C executable genisoimage
6/usr/bin/ld: CMakeFiles/genisoimage.dir/apple.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
7/usr/bin/ld: CMakeFiles/genisoimage.dir/boot.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
8/usr/bin/ld: CMakeFiles/genisoimage.dir/desktop.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
9/usr/bin/ld: CMakeFiles/genisoimage.dir/dvd_file.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
10/usr/bin/ld: CMakeFiles/genisoimage.dir/dvd_reader.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
11/usr/bin/ld: CMakeFiles/genisoimage.dir/eltorito.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
12/usr/bin/ld: CMakeFiles/genisoimage.dir/files.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
13/usr/bin/ld: CMakeFiles/genisoimage.dir/hash.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
14/usr/bin/ld: CMakeFiles/genisoimage.dir/ifo_read.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
15/usr/bin/ld: CMakeFiles/genisoimage.dir/joliet.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
16/usr/bin/ld: CMakeFiles/genisoimage.dir/mac_label.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
17/usr/bin/ld: CMakeFiles/genisoimage.dir/multi.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
18/usr/bin/ld: CMakeFiles/genisoimage.dir/name.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
19/usr/bin/ld: CMakeFiles/genisoimage.dir/rock.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
20/usr/bin/ld: CMakeFiles/genisoimage.dir/scsi.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
21/usr/bin/ld: CMakeFiles/genisoimage.dir/stream.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
22/usr/bin/ld: CMakeFiles/genisoimage.dir/tree.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
23/usr/bin/ld: CMakeFiles/genisoimage.dir/udf.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
24/usr/bin/ld: CMakeFiles/genisoimage.dir/volume.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
25/usr/bin/ld: CMakeFiles/genisoimage.dir/write.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
26/usr/bin/ld: CMakeFiles/genisoimage.dir/boot-alpha.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
27/usr/bin/ld: CMakeFiles/genisoimage.dir/boot-hppa.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
28/usr/bin/ld: CMakeFiles/genisoimage.dir/boot-mips.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
29/usr/bin/ld: CMakeFiles/genisoimage.dir/jte.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
30/usr/bin/ld: CMakeFiles/genisoimage.dir/rsync.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
31/usr/bin/ld: CMakeFiles/genisoimage.dir/boot-mipsel.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
32/usr/bin/ld: CMakeFiles/genisoimage.dir/checksum.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
33collect2: error: ld returned 1 exit status
Have not investigated further.
For me, builds of bitcoind and bitcoin-qt even succeed with lto enabled for dependencies as well with the following hack:
0diff --git a/depends/hosts/darwin.mk b/depends/hosts/darwin.mk
1index 6099fd4c71..c4180ac2f8 100644
2--- a/depends/hosts/darwin.mk
3+++ b/depends/hosts/darwin.mk
4@@ -28,8 +28,8 @@ OSX_SDK=$(SDK_PATH)/Xcode-$(XCODE_VERSION)-$(XCODE_BUILD_ID)-extracted-SDK-with-
5 # https://reviews.llvm.org/D64089, we should use that instead. Read the
6 # differential summary there for more details.
7 #
8-darwin_CC=clang -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK) -mlinker-version=$(LD64_VERSION) -B$(build_prefix)/bin
9-darwin_CXX=clang++ -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK) -stdlib=libc++ -mlinker-version=$(LD64_VERSION) -B$(build_prefix)/bin -nostdinc++ -isystem $(OSX_SDK)/usr/include/c++/v1
10+darwin_CC=clang -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK) -mlinker-version=$(LD64_VERSION) -B$(build_prefix)/bin -flto=thin
11+darwin_CXX=clang++ -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK) -stdlib=libc++ -mlinker-version=$(LD64_VERSION) -B$(build_prefix)/bin -nostdinc++ -isystem $(OSX_SDK)/usr/include/c++/v1 -flto=thin
12
13 darwin_CFLAGS=-pipe
14 darwin_CXXFLAGS=$(darwin_CFLAGS)
It’s almost suspicious how well/easily that worked. I’ll dig further to make sure it’s actually doing full lto under the hood.
@fanquake Strange… this is fine locally, and our c-i didn’t have any trouble with it either. Is the problem locally reproducible?
Yes. Turns out the issue is actually a change in defaults in GCC 10.1. See #19553. In the mean time, I’ll retest here with an earlier GCC.
Continuing the conversation from #19565:
The solution seemed to be to pass -object_path_lto through to ld64. Doing that I end up with a properly generated .dSYM:
I think I understand what’s happening here. From the clang docs
0 The default for :option:`-flto` is "full", in which the
1 LLVM bitcode is suitable for monolithic Link Time Optimization (LTO), where
2 the linker merges all such modules into a single combined module for
3 optimization. With "thin", :doc:`ThinLTO <../ThinLTO>`
4 compilation is invoked instead.
5
6 .. note::
7
8 On Darwin, when using :option:`-flto` along with :option:`-g` and
9 compiling and linking in separate steps, you also need to pass
10 ``-Wl,-object_path_lto,<lto-filename>.o`` at the linking step to instruct the
11 ld64 linker not to delete the temporary object file generated during Link
12 Time Optimization (this flag is automatically passed to the linker by Clang
13 if compilation and linking are done in a single step). This allows debugging
14 the executable as well as generating the ``.dSYM`` bundle using :manpage:`dsymutil(1)`.
That’s describing us.
The machinery to get clang to do this for you only applies when compiling and linking at the same time. Here’s a comment from elsewhere explaining why:
0 // Add a 'dsymutil' step if necessary, when debug info is enabled and we
1 // have a compile input. We need to run 'dsymutil' ourselves in such cases
2 // because the debug info will refer to a temporary object file which
3 // will be removed at the end of the compilation process.
I can confirm that is working as intended with a test-case:
$ cat lto1.cpp lto2.cpp
0//lto1.cpp
1void MyFunc();
2int main(){
3MyFunc();
4}
5//lto2.cpp
6#include <stdio.h>
7void MyFunc()
8{
9 printf("hello world\n");
10}
$ clang++ -target x86_64-apple-darwin16 -mmacosx-version-min=10.12 –sysroot /home/cory/dev/bitcoin2/depends/SDKs/Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers -stdlib=libc++ -mlinker-version=530 -B/home/cory/dev/bitcoin2/depends/x86_64-apple-darwin16/native/bin -nostdinc++ -isystem /home/cory/dev/bitcoin2/depends/SDKs/Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers/usr/include/c++/v1 -flto=thin -std=c++11 lto1.cpp lto2.cpp -v
Which produces the following link line and calls dsymutil:
“/home/cory/dev/bitcoin2/depends/x86_64-apple-darwin16/native/bin/x86_64-apple-darwin16-ld” -demangle -object_path_lto /tmp/thinlto-d130c2 -lto_library /home/cory/dev/llvm-project/build/lib/libLTO.so -no_deduplicate -dynamic -arch x86_64 -platform_version macos 10.12.0 0.0.0 -syslibroot /home/cory/dev/bitcoin2/depends/SDKs/Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers -o a.out /tmp/lto1-bd26d0.o /tmp/lto2-439b81.o -lc++ -lSystem "/home/cory/dev/llvm-project/build/bin/dsymutil" -o a.out.dSYM a.out
So, it sounds as though everything is actually working as intended. We just need to add that link flag if lto and debugging are on. But we’re getting ahead of ourselves, as we don’t have an option for lto yet :)
Labels
Build system