arm64 CI failure #27879

arm64 CI is failing with:

FAIL: minisketch/test
=====================

../build-aux/test-driver: line 109: ./minisketch/test: cannot execute binary file: Exec format error
FAIL minisketch/test (exit status: 126)

FAIL: univalue/test/object
==========================

../build-aux/test-driver: line 109: ./univalue/test/object: cannot execute binary file: Exec format error
FAIL univalue/test/object (exit status: 126)

FAIL: univalue/test/unitester
=============================

../build-aux/test-driver: line 109: ./univalue/test/unitester: cannot execute binary file: Exec format error
FAIL univalue/test/unitester (exit status: 126)

FAIL: qt/test/test_bitcoin-qt
=============================

../build-aux/test-driver: line 109: ./qt/test/test_bitcoin-qt: cannot execute binary file: Exec format error
FAIL qt/test/test_bitcoin-qt (exit status: 126)

See eg https://github.com/bitcoin/bitcoin/pull/27647/checks?check_run_id=14232303009 or https://github.com/bitcoin/bitcoin/pull/27746/checks?check_run_id=14231876127

I tried re-running the arm64 task for #27675 which had succeeded yesterday, and it now fails too (before vs after), which suggests either that the bug was introduced after commit fbe48f97dfec3138b06b5f00b75655da0c985008 up to commit 427853ab49f610e971b73ea4cc1d5366747e52b1 (nothing seemed likely to break minisketch/test there) or is an infrastructure issue (bad cache or bad hardware?).

Maybe worth adding git show FETCH_HEAD at the end of base_template: in .cirrus.yml so it's easier to see which version of master a CI run was merged against?

Saw this issue today on two pushes as well, #27425 and #27632. Re-running the task didn't help.

Maybe worth adding git show FETCH_HEAD at the end of base_template: in .cirrus.yml so it's easier to see which version of master a CI run was merged against?

I think you can check the output of merge_base to see what was used? For example, https://cirrus-ci.com/task/6628961686388736?logs=merge_base#L18

or is an infrastructure issue (bad cache or bad hardware?).

Seems likely. Let's poke a bit...

I think you can check the output of merge_base to see what was used? For example, https://cirrus-ci.com/task/6628961686388736?logs=merge_base#L18

I looked at that and somehow didn't see it. Weird.

There is certainly something sketchy going on with the upstream hardware/software. I've had to reset the pre-build of the image thrice: https://cirrus-ci.com/task/4729934984249344. Also, once that passes, the actual build fails half the time with curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to raw.githubusercontent.com:443 https://cirrus-ci.com/task/5920692756021248?logs=ci#L208

There is certainly something sketchy going on with the upstream hardware/software. I've had to reset the pre-build of the image thrice: https://cirrus-ci.com/task/4729934984249344. Also, once that passes, the actual build fails half the time with curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to raw.githubusercontent.com:443 https://cirrus-ci.com/task/5920692756021248?logs=ci#L208

@fkorotkov What do you think?

Aparently AWS was down yesterday? Maybe someone needs to turn a server off and on again to fix it?

My suggestion would be to wait 24 hours to see if the issue fixes itself. If not, replace the arm64 hardware with amd64, and run everything in qemu-arm again.

There is certainly something sketchy going on with the upstream hardware...

It looks like the underlying hardware stopped to support running ARM-32bit binaries on arm64 hardware. Actually, there was not such a promise from Cirrus Labs about that.

My suggestion would be to wait 24 hours to see if the issue fixes itself. If not, replace the arm64 hardware with amd64, and run everything in qemu-arm again.

Another option is to build aarch64 binaries using an arm_container in this task.

Thanks for taking a look, do you mind updating the CI task config for aarch64? https://github.com/bitcoin/bitcoin/blob/427853ab49f610e971b73ea4cc1d5366747e52b1/ci/test/00_setup_env_arm.sh#L9-L14

do you mind updating the CI task config for aarch64?

I'll do it. No need to wait a happy self fix? :)

Longer term it may also be worth to consider removing the arm32 release builds, unless there are still people using them? Maybe someone can ask on the mailing list?

arm32 release builds

meaning (for example) bitcoin-25.0-arm-linux-gnueabihf.tar.gz 😬 that's the Raspberry Pi binary

meaning (for example) bitcoin-25.0-arm-linux-gnueabihf.tar.gz grimacing that's the Raspberry Pi binary

Ah ok, that'd be the Pi 1 and Zero? All other (Zero 2, 3, 4, ...) are 64 bit, IIRC.

the hardware has been 64 bit since Pi3 I think but the OS itself has only been 64 bit for a year or so:

https://www.raspberrypi.com/news/raspberry-pi-os-64-bit/

The only Arm related change yesterday was that we switched from AWS using Graviton2 CPUs to GCP using Ampere CPUs. Both Graviton2 and Ampere Alta support EL0 AArch32.

Both Graviton2 and Ampere Alta support EL0 AArch32.

Yes, however GCP explicitly doesn't support 32-bit mode?

The Tau T2A machine series does not support: 32-bit mode ELO (guest userspace support)

Thanks everyone for the comments. I think a third alternative for us, if we want minimal changes, would be to unset QEMU_USER_CMD and run qemu on aarch64 for now?

Also on Google Cloud t2a-standard-4:

lscpu | grep op-mode
CPU op-mode(s):                  64-bit

<!--- In the meantime I checked `MAKEJOBS="-j32" QEMU_USER_CMD="" FILE_ENV="./ci/test/00_setup_env_arm.sh" ./ci/test_run_all.sh` on a few CPUs: <details><summary>Qemu</summary> ``` # echo "Passed: $?" && lscpu && git log -1 --oneline Passed: 0 Architecture: aarch64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: ARM BIOS Vendor ID: QEMU Model name: Neoverse-N1 BIOS Model name: NotSpecified CPU @ 2.0GHz BIOS CPU family: 1 Model: 1 Thread(s) per core: 1 Core(s) per cluster: 16 Socket(s): 1 Cluster(s): 1 Stepping: r3p1 BogoMIPS: 50.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs NUMA: NUMA node(s): 1 NUMA node0 CPU(s): 0-15 Vulnerabilities: Itlb multihit: Not affected L1tf: Not affected Mds: Not affected Meltdown: Not affected Mmio stale data: Not affected Retbleed: Not affected Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Spectre v1: Mitigation; __user pointer sanitization Spectre v2: Mitigation; CSV2, BHB Srbds: Not affected Tsx async abort: Not affected 427853a (grafted, HEAD -> master, origin/master, origin/HEAD) Merge bitcoin/bitcoin#27876: test: (refactor) Use datadir from options in chainstatemanager test ``` </details> <details><summary>Google Cloud `t2a-standard-4`</summary> ``` Passed: 2 Architecture: aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Vendor ID: ARM Model name: Neoverse-N1 Model: 1 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 1 Stepping: r3p1 BogoMIPS: 50.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs NUMA: NUMA node(s): 1 NUMA node0 CPU(s): 0-3 Vulnerabilities: Itlb multihit: Not affected L1tf: Not affected Mds: Not affected Meltdown: Not affected Mmio stale data: Not affected Retbleed: Not affected Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Spectre v1: Mitigation; __user pointer sanitization Spectre v2: Mitigation; CSV2, BHB Srbds: Not affected Tsx async abort: Not affected 427853a (grafted, HEAD -> master, origin/master, origin/HEAD) Merge bitcoin/bitcoin#27876: test: (refactor) Use datadir from options in chainstatemanager test ``` </details>

I think a third alternative for us, if we want minimal changes, would be to unset QEMU_USER_CMD and run qemu on aarch64 for now?

I've been testing this change since this morning. The CI constantly timed out for some reasons.