travis: Run unit and functional tests on native arm #17233

MarcoFalke commented at 7:08 PM on October 23, 2019: member

This keeps the cross-compilation to make it easy to run the ci on non-arm hardware. To run this locally in qemu-user as it used to be, just export QEMU_USER_CMD="qemu-arm -L /usr/arm-linux-gnueabihf/".

MarcoFalke added the label Tests on Oct 23, 2019

practicalswift commented at 9:14 PM on October 23, 2019: contributor

Concept ACK

Diversity in testing is good

MarcoFalke commented at 9:15 PM on October 23, 2019: member

Even though it shouldn't fail, this fails with

/bin/bash: /home/travis/build/bitcoin/bitcoin/depends/arm-linux-gnueabihf/share/../native/bin/uic: cannot execute binary file: Exec format error

I might try to debug this locally some day...

laanwj commented at 11:31 AM on October 24, 2019: member

Concept ACK

I think the error has to do with using the output of cross-compilation for normal compilation; the "native" uic will be x86-64, the expected build platform.

MarcoFalke commented at 12:22 PM on October 24, 2019: member

Oh, so we can't cross compile on arm64 to other arches?

laanwj commented at 5:15 PM on October 24, 2019: member

Oh, so we can't cross compile on arm64 to other arches?

There is no conceptual reason why not, but maybe not with how depends are set up right now. Not sure.

But I think I misunderstand what is happening. I thought the cross compile was from x86 to armhf. Is it cross-compiling from arm64 to armhf?

(I didn't think so because I assumed you don't get the Exec format error between different ARM bitnesses)

What is the "native" platform here, and what is the target platform?

MarcoFalke commented at 5:37 PM on October 24, 2019: member

travis offers arm64 (== aarch64 ) machines, so the cross compilation is on a arm64 host to an arm32 guest

laanwj commented at 10:10 PM on October 24, 2019: member

So we're sure it's not using cached arm-linux-gnueabihf depends that have been built on x86-64? I can't think of any explanation why the native tools like uic (which are by definition not cross-compiled, but built with the native C++ compiler, so that they can be part of the build processs) would be the wrong architecture.

MarcoFalke commented at 10:28 PM on October 24, 2019: member

Good point. The depends cache is not invalidated when the compiler changes

ci: Remove ccache requirement on the host

ccache is only needed to create the cache dir on the host, if it didn't
already exist. The same can be achieved with mkdir, so just use that
instead.

fafa064d2a

MarcoFalke force-pushed on Oct 24, 2019

laanwj commented at 9:52 AM on October 25, 2019: member

Looks like it finishes the build now, but the testing stage cannot find the binaries… strange

laanwj commented at 10:29 AM on October 27, 2019: member

I think I get the reason for this: you are executing ARM32 executables on ARM64 without dynamic linker and glibc installed. Missing the ELF interpreter will give the same "not found" error as missing the executable itself.

I think this is package libc6:armhf.

odroid@odroid64:~$ dpkg -S /lib/ld-linux-armhf.so.3
libc6:armhf: /lib/ld-linux-armhf.so.3

MarcoFalke force-pushed on Oct 28, 2019

MarcoFalke commented at 4:50 PM on October 28, 2019: member

Added libc6-armhf-armel-cross to the install packages

laanwj commented at 7:53 PM on October 28, 2019: member

I think that's the old package naming. It should work though.

bionic (18.04LTS) (devel): Dummy package to get libc6:armhf installed [universe] 2.27-3ubuntu1cross1: all

MarcoFalke commented at 8:19 PM on October 28, 2019: member

It fails locally and on travis:

$ sudo apt install libc6-dev-armhf-armel-cross && make check -j 2
Reading package lists... Done
Building dependency tree
Reading state information... Done
libc6-dev-armhf-armel-cross is already the newest version (2.27-3ubuntu1cross1.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Making check in src
make[1]: Entering directory '/home/ubuntu/bitcoin-core/src'
make[2]: Entering directory '/home/ubuntu/bitcoin-core/src'
make[3]: Entering directory '/home/ubuntu/bitcoin-core'
make[3]: Leaving directory '/home/ubuntu/bitcoin-core'
make  check-TESTS check-local
make[3]: Entering directory '/home/ubuntu/bitcoin-core/src'
Running tests: arith_uint256_tests from test/arith_uint256_tests.cpp
/bin/bash: test/test_bitcoin: No such file or directory

laanwj commented at 8:32 PM on October 28, 2019: member

What is the output of readelf -a test/test_bitcoin | grep interpreter? does that file exist?

MarcoFalke commented at 8:46 PM on October 28, 2019: member

$ readelf -a src/test/test_bitcoin | grep interpreter
      [Requesting program interpreter: /lib/ld-linux-armhf.so.3]
0x59004c <_ZNK5boost7runtime17value_interpreterIbLb0EE9interpretENS_9unit_test13basic_cstringIKcEES6_>: [@0x9c7a78](/bitcoin-bitcoin/contributor/0x9c7a78/)
0x5957bc <_ZNK5boost7runtime17value_interpreterIjLb0EE9interpretENS_9unit_test13basic_cstringIKcEES6_>: [@0x9c807c](/bitcoin-bitcoin/contributor/0x9c807c/)
0x595dc0 <_ZNK5boost7runtime17value_interpreterImLb0EE9interpretENS_9unit_test13basic_cstringIKcEES6_>: [@0x9c8188](/bitcoin-bitcoin/contributor/0x9c8188/)
 29187: 00000000     0 FILE    LOCAL  DEFAULT  ABS interpreter.cpp


$ ll /lib/ld-linux-armhf.so.3
ls: cannot access '/lib/ld-linux-armhf.so.3': No such file or directory

$ ll /usr/arm-linux-gnueabihf/lib/ld-linux-armhf.so.3 
lrwxrwxrwx 1 root root 10 Sep 17  2018 /usr/arm-linux-gnueabihf/lib/ld-linux-armhf.so.3 -> ld-2.27.so*

laanwj commented at 11:18 AM on October 30, 2019: member

It really needs to be in /lib for multi-arch to work.

Can you try:

dpkg --add-architecture armhf
apt-get update
apt-get install libc6:armhf

MarcoFalke force-pushed on Oct 30, 2019

MarcoFalke commented at 6:27 PM on October 30, 2019: member

I feel dumb for not trying that, but then I remember that it didn't work locally:

# dpkg --add-architecture armhf
root@729a87425702:/# apt update
Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease
Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease
Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
Ign:4 http://security.ubuntu.com/ubuntu bionic-security/main armhf Packages    
Hit:5 http://archive.ubuntu.com/ubuntu bionic-backports InRelease              
Ign:6 http://security.ubuntu.com/ubuntu bionic-security/restricted armhf Packages
Ign:7 http://archive.ubuntu.com/ubuntu bionic/multiverse armhf Packages                                                            
Ign:8 http://security.ubuntu.com/ubuntu bionic-security/multiverse armhf Packages                                                  
Ign:9 http://archive.ubuntu.com/ubuntu bionic/main armhf Packages                                                                  
Ign:10 http://security.ubuntu.com/ubuntu bionic-security/universe armhf Packages                                                   
Ign:11 http://archive.ubuntu.com/ubuntu bionic/universe armhf Packages                                   
Ign:4 http://security.ubuntu.com/ubuntu bionic-security/main armhf Packages                              
Ign:12 http://archive.ubuntu.com/ubuntu bionic/restricted armhf Packages    
Ign:6 http://security.ubuntu.com/ubuntu bionic-security/restricted armhf Packages
Ign:7 http://archive.ubuntu.com/ubuntu bionic/multiverse armhf Packages
Ign:8 http://security.ubuntu.com/ubuntu bionic-security/multiverse armhf Packages
Ign:13 http://archive.ubuntu.com/ubuntu bionic-updates/universe armhf Packages
Ign:10 http://security.ubuntu.com/ubuntu bionic-security/universe armhf Packages
Ign:14 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse armhf Packages
Ign:4 http://security.ubuntu.com/ubuntu bionic-security/main armhf Packages
Ign:15 http://archive.ubuntu.com/ubuntu bionic-updates/main armhf Packages
Ign:6 http://security.ubuntu.com/ubuntu bionic-security/restricted armhf Packages
Ign:16 http://archive.ubuntu.com/ubuntu bionic-updates/restricted armhf Packages
Ign:8 http://security.ubuntu.com/ubuntu bionic-security/multiverse armhf Packages
Ign:9 http://archive.ubuntu.com/ubuntu bionic/main armhf Packages
Ign:10 http://security.ubuntu.com/ubuntu bionic-security/universe armhf Packages
Ign:17 http://archive.ubuntu.com/ubuntu bionic-backports/main armhf Packages
Err:4 http://security.ubuntu.com/ubuntu bionic-security/main armhf Packages
  404  Not Found [IP: 91.189.88.24 80]
Ign:18 http://archive.ubuntu.com/ubuntu bionic-backports/universe armhf Packages
Ign:6 http://security.ubuntu.com/ubuntu bionic-security/restricted armhf Packages
Ign:11 http://archive.ubuntu.com/ubuntu bionic/universe armhf Packages      
Ign:8 http://security.ubuntu.com/ubuntu bionic-security/multiverse armhf Packages
Ign:12 http://archive.ubuntu.com/ubuntu bionic/restricted armhf Packages    
Ign:10 http://security.ubuntu.com/ubuntu bionic-security/universe armhf Packages
Ign:7 http://archive.ubuntu.com/ubuntu bionic/multiverse armhf Packages     
Ign:13 http://archive.ubuntu.com/ubuntu bionic-updates/universe armhf Packages
Ign:14 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse armhf Packages
Ign:15 http://archive.ubuntu.com/ubuntu bionic-updates/main armhf Packages
Ign:16 http://archive.ubuntu.com/ubuntu bionic-updates/restricted armhf Packages
Ign:9 http://archive.ubuntu.com/ubuntu bionic/main armhf Packages
Ign:17 http://archive.ubuntu.com/ubuntu bionic-backports/main armhf Packages
Ign:18 http://archive.ubuntu.com/ubuntu bionic-backports/universe armhf Packages
Ign:11 http://archive.ubuntu.com/ubuntu bionic/universe armhf Packages
Ign:12 http://archive.ubuntu.com/ubuntu bionic/restricted armhf Packages
Err:7 http://archive.ubuntu.com/ubuntu bionic/multiverse armhf Packages
  404  Not Found [IP: 91.189.88.149 80]
Ign:13 http://archive.ubuntu.com/ubuntu bionic-updates/universe armhf Packages
Ign:14 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse armhf Packages
Ign:15 http://archive.ubuntu.com/ubuntu bionic-updates/main armhf Packages
Ign:16 http://archive.ubuntu.com/ubuntu bionic-updates/restricted armhf Packages
Ign:9 http://archive.ubuntu.com/ubuntu bionic/main armhf Packages
Ign:17 http://archive.ubuntu.com/ubuntu bionic-backports/main armhf Packages
Ign:18 http://archive.ubuntu.com/ubuntu bionic-backports/universe armhf Packages
Ign:11 http://archive.ubuntu.com/ubuntu bionic/universe armhf Packages
Ign:12 http://archive.ubuntu.com/ubuntu bionic/restricted armhf Packages
Err:13 http://archive.ubuntu.com/ubuntu bionic-updates/universe armhf Packages
  404  Not Found [IP: 91.189.88.149 80]
Ign:14 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse armhf Packages
Ign:15 http://archive.ubuntu.com/ubuntu bionic-updates/main armhf Packages
Ign:16 http://archive.ubuntu.com/ubuntu bionic-updates/restricted armhf Packages
Err:17 http://archive.ubuntu.com/ubuntu bionic-backports/main armhf Packages
  404  Not Found [IP: 91.189.88.149 80]
Ign:18 http://archive.ubuntu.com/ubuntu bionic-backports/universe armhf Packages
Reading package lists... Done
E: Failed to fetch http://security.ubuntu.com/ubuntu/dists/bionic-security/main/binary-armhf/Packages  404  Not Found [IP: 91.189.88.24 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic/multiverse/binary-armhf/Packages  404  Not Found [IP: 91.189.88.149 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-updates/universe/binary-armhf/Packages  404  Not Found [IP: 91.189.88.149 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-backports/main/binary-armhf/Packages  404  Not Found [IP: 91.189.88.149 80]
E: Some index files failed to download. They have been ignored, or old ones used instead.
root@729a87425702:/# apt-get install libc6:armhf
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package libc6:armhf

laanwj commented at 6:29 PM on October 30, 2019: member

I'm confused;

root@odroid64:~# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"

root@odroid64:~# apt update
Hit:1 http://ports.ubuntu.com/ubuntu-ports bionic InRelease
Hit:2 http://deb.odroid.in/c2 bionic InRelease                          
Get:3 http://ports.ubuntu.com/ubuntu-ports bionic-updates InRelease [88.7 kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports bionic-backports InRelease [74.6 kB]
Get:5 http://ports.ubuntu.com/ubuntu-ports bionic-security InRelease [88.7 kB]
Get:6 http://ports.ubuntu.com/ubuntu-ports bionic-proposed InRelease [242 kB]
Get:7 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main Sources [295 kB]
Get:8 http://ports.ubuntu.com/ubuntu-ports bionic-updates/universe Sources [265 kB]
Get:9 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 Packages [579 kB]
Get:10 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main armhf Packages [548 kB]
Get:11 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main Translation-en [275 kB]
Get:12 http://ports.ubuntu.com/ubuntu-ports bionic-updates/universe armhf Packages [854 kB]
Get:13 http://ports.ubuntu.com/ubuntu-ports bionic-updates/universe arm64 Packages [911 kB]
Get:14 http://ports.ubuntu.com/ubuntu-ports bionic-updates/universe Translation-en [313 kB]
Get:15 http://ports.ubuntu.com/ubuntu-ports bionic-security/main Sources [127 kB]
Get:16 http://ports.ubuntu.com/ubuntu-ports bionic-security/main armhf Packages [343 kB]
Get:17 http://ports.ubuntu.com/ubuntu-ports bionic-security/main arm64 Packages [372 kB]
Get:18 http://ports.ubuntu.com/ubuntu-ports bionic-security/main Translation-en [182 kB]
Get:19 http://ports.ubuntu.com/ubuntu-ports bionic-security/universe arm64 Packages [551 kB]
Get:20 http://ports.ubuntu.com/ubuntu-ports bionic-security/universe armhf Packages [498 kB]
Get:21 http://ports.ubuntu.com/ubuntu-ports bionic-security/universe Translation-en [206 kB]
Get:22 http://ports.ubuntu.com/ubuntu-ports bionic-proposed/universe Sources [15.4 kB]
Get:23 http://ports.ubuntu.com/ubuntu-ports bionic-proposed/main Sources [27.6 kB]
Get:24 http://ports.ubuntu.com/ubuntu-ports bionic-proposed/main armhf Packages [35.9 kB]
Get:25 http://ports.ubuntu.com/ubuntu-ports bionic-proposed/main arm64 Packages [40.6 kB]
Get:26 http://ports.ubuntu.com/ubuntu-ports bionic-proposed/main Translation-en [27.2 kB]
Get:27 http://ports.ubuntu.com/ubuntu-ports bionic-proposed/universe arm64 Packages [24.6 kB]
Get:28 http://ports.ubuntu.com/ubuntu-ports bionic-proposed/universe armhf Packages [22.9 kB]
Get:29 http://ports.ubuntu.com/ubuntu-ports bionic-proposed/universe Translation-en [16.2 kB]
Fetched 7,022 kB in 5s (1,534 kB/s)                           
Reading package lists... Done
Building dependency tree       
Reading state information... Done

root@odroid64:~# apt-get install libc6:armhf
Reading package lists... Done
Building dependency tree       
Reading state information... Done
libc6:armhf is already the newest version (2.27-3ubuntu1).
libc6:armhf set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 5 not upgraded.

No idea what could be the difference (though we're clearly using a different mirror).

MarcoFalke commented at 6:34 PM on October 30, 2019: member

Oh, I was running the dpkg --add-architecture armhf on x86_64 locally. Does that make it not work?

Either way. Looks like on travis and the odroid c2 it worked. Let's hope it runs the tests now :pray:

laanwj commented at 6:38 PM on October 30, 2019: member

Oh, I was running the dpkg --add-architecture armhf on x86_64 locally. Does that make it not work?

I'm not sure. I think it in principle it should be possible to install armhf as additional architecture on x86 (using qemu-user to execute things), but it's a much less-traveled path than armhf on aarch64. You might have to mess around with apt configuration.

travis: Run unit and functional tests on native arm facc0da63a

MarcoFalke force-pushed on Oct 30, 2019

MarcoFalke commented at 12:41 AM on November 1, 2019: member

@laanwj Thanks for the help. Passes locally and on travis. This is ready for review now!

laanwj commented at 4:04 PM on November 2, 2019: member

Thanks for seeing this through!

LGTM ACK facc0da63a8fa4bd6fc2782cbe92eb9f920f2256

practicalswift commented at 5:33 PM on November 2, 2019: contributor

ACK facc0da63a8fa4bd6fc2782cbe92eb9f920f2256 -- diff looks correct and Travis seems happy

Thanks for working on this!

MarcoFalke referenced this in commit def69e5a86 on Nov 4, 2019

MarcoFalke merged this on Nov 4, 2019

MarcoFalke closed this on Nov 4, 2019

MarcoFalke deleted the branch on Nov 4, 2019

sidhujag referenced this in commit 1c6ec8552a on Nov 7, 2019

sidhujag referenced this in commit 15a2c66988 on Nov 10, 2020

MarcoFalke locked this on Dec 16, 2021