build: depends cross-compile using Clang fails

fanquake commented at 4:57 pm on January 28, 2025: member

Seems like CMake is getting confused, and trying to build and link a x86_64 binary, when it’s meant to be (and thinks it is, according to the configure output) building for aarch64:

 0make -C depends/ CC=clang CXX=clang++ AR=llvm-ar NM=llvm-nm RANLIB=llvm-ranlib STRIP=llvm-strip NO_QT=1 NO_WALLET=1 NO_USDT=1 NO_ZMQ=1 -j18 HOST=aarch64-linux-gnu
 1<snip>
 2copying packages: boost libevent
 3to: /root/ci_scratch/depends/aarch64-linux-gnu
 4To build Bitcoin Core with these packages, pass '--toolchain /root/ci_scratch/depends/aarch64-linux-gnu/toolchain.cmake' to the first CMake invocation.
 5
 6cmake -B build --toolchain /root/ci_scratch/depends/aarch64-linux-gnu/toolchain.cmake
 7<snip>
 8Cross compiling ....................... TRUE, for Linux, aarch64
 9C++ compiler .......................... Clang 18.1.3, /usr/bin/clang++
10<snip>
11cmake --build build -j18 --target bitcoind --verbose
12[ 98%] Linking CXX executable bitcoind
13cd /root/ci_scratch/build/src && /usr/bin/cmake -E cmake_link_script CMakeFiles/bitcoind.dir/link.txt --verbose=1
14/usr/bin/clang++ -pipe -std=c++20 -O2 -O2 -g -fstack-protector-all -fcf-protection=full -fstack-clash-protection -Wl,-z,relro -Wl,-z,now -Wl,-z,separate-code -fPIE -pie CMakeFiles/bitcoind.dir/bitcoind.cpp.o CMakeFiles/bitcoind.dir/init/bitcoind.cpp.o -o bitcoind  libbitcoin_node.a libbitcoin_common.a libbitcoin_consensus.a secp256k1/lib/libsecp256k1.a util/libbitcoin_util.a crypto/libbitcoin_crypto.a crypto/libbitcoin_crypto_sse41.a crypto/libbitcoin_crypto_avx2.a crypto/libbitcoin_crypto_x86_shani.a ../libleveldb.a ../libcrc32c.a ../libcrc32c_sse42.a ../libminisketch.a univalue/libunivalue.a /root/ci_scratch/depends/aarch64-linux-gnu/lib/libevent_extra.a /root/ci_scratch/depends/aarch64-linux-gnu/lib/libevent_pthreads.a /root/ci_scratch/depends/aarch64-linux-gnu/lib/libevent_core.a  
15/usr/bin/ld: /root/ci_scratch/depends/aarch64-linux-gnu/lib/libevent_extra.a(http.c.o): Relocations in generic ELF (EM: 183)
16/usr/bin/ld: /root/ci_scratch/depends/aarch64-linux-gnu/lib/libevent_extra.a(http.c.o): Relocations in generic ELF (EM: 183)
17/usr/bin/ld: /root/ci_scratch/depends/aarch64-linux-gnu/lib/libevent_extra.a(http.c.o): Relocations in generic ELF (EM: 183)
18/usr/bin/ld: /root/ci_scratch/depends/aarch64-linux-gnu/lib/libevent_extra.a(http.c.o): Relocations in generic ELF (EM: 183)
19/usr/bin/ld: /root/ci_scratch/depends/aarch64-linux-gnu/lib/libevent_extra.a: error adding symbols: file in wrong format
20clang++: error: linker command failed with exit code 1 (use -v to see invocation)
21gmake[3]: *** [src/CMakeFiles/bitcoind.dir/build.make:130: src/bitcoind] Error 1

Maybe there’s something missing from the toolchain that we should be passing through. The libs in depends have been compiled for aarch64, i.e:

0objdump -x /root/ci_scratch/depends/aarch64-linux-gnu/lib/libevent_extra.a | grep aarch64
1In archive /root/ci_scratch/depends/aarch64-linux-gnu/lib/libevent_extra.a:
2event_tagging.c.o:     file format elf64-littleaarch64
3architecture: aarch64, flags 0x00000011:
4http.c.o:     file format elf64-littleaarch64

However CMake is building x86_64 object files:

0file build/CMakeFiles/leveldb.dir/src/leveldb/db/filename.cc.o
1build/CMakeFiles/leveldb.dir/src/leveldb/db/filename.cc.o: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), with debug_info, not stripped

and binaries:

0file build/src/bitcoin-util 
1build/src/bitcoin-util: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=4ccab1d492154df5dd9a115e6bbc52371d7439b1, for GNU/Linux 3.2.0, with debug_info, not stripped

fanquake added the label Build system on Jan 28, 2025

ryanofsky commented at 6:08 pm on January 28, 2025: contributor

Maybe dumb question but are you building on a x86_64 system where clang and clang++ executables produce x86_64 binaries? If so I would guess this is caused by your CC=clang CXX=clang++ variables.

fanquake commented at 2:39 pm on January 29, 2025: member

are you building on a x86_64 system where clang and clang++ executables produce x86_64 binaries?

Yes.

If so I would guess this is caused by your CC=clang CXX=clang++ variables.

Not sure I follow. Depends was configured to cross-compile for aarch64-linux using Clang as the compiler. This has worked fine and produced aarch64 code and libraries. We then pass the toolchain file from depends to CMake (which should contain everything CMake needs to know produce aarch64 linux binaries using the just compiled aarch64 libraries). CMake has configured using Clang as the compiler, and think it’s cross-compiling for aarch64, according to it’s own output:

Cross compiling ………………….. TRUE, for Linux, aarch64

Then it proceeds to produce x86_64 code.

fanquake renamed this:
~~build: depends cros-compile using Clang fails~~
build: depends cross-compile using Clang fails
on Jan 29, 2025

ryanofsky commented at 2:57 pm on January 29, 2025: contributor

CC=clang CXX=clang++ specify paths to specific compiler binaries, and compiler binaries usually have a default platform that they target. On an x86_64 system they will usually produce x86_64 binaries unless passed additional flags which I don’t think the depends system is using.

The host string aarch64-linux-gnu you are specifying will be used to find the right compiler binaries for that host by default, but if you are overriding the compiler, the compiler you specified will take precedence.

I could be wrong about this or misunderstanding something fundamental but I’d try to do this without the CC=clang CXX=clang++ AR=… stuff if that makes any sense for your use case. If it doesn’t make sense please ignore.

You could also try CC=aarch64-linux-gnu-clang etc or whatever the aarch64 cross-compiler path is

fanquake commented at 3:07 pm on January 29, 2025: member

You could also try CC=aarch64-linux-gnu-clang etc or whatever the aarch64 cross-compiler path is

You don’t really need to do this with, clang, it is natively a cross-compiler. You can (generally) just pass --target=whatever, and generate code for that target, without needing a specially compiled clang binary. In this case, by specifying HOST=aarch64-linux-gnu, I end up with compiler invocations like clang --target=aarch64-linux-gnu, and everything works as expected in depends. The issue seems to be that these flags are not being properly propogated into the CMake toolchain file, and then missing for the Core build.

ryanofsky commented at 3:25 pm on January 29, 2025: contributor

You could also try CC=aarch64-linux-gnu-clang etc or whatever the aarch64 cross-compiler path is

You don’t really need to do this with, clang, it is natively a cross-compiler.

Yes that is what I meant by “compiler binaries usually have a default platform that they target”.

Maybe I misunderstood context for your post and thought you were reporting a bug and looking for a workaround, not just reporting a bug. I think I agree that dropping CC=clang or specifying CC=aarch64-linux-gnu-clang would just be workarounds.

build: depends cross-compile using Clang fails #31748