ci: how to run native arm job on Apple silicon?

Sjors commented at 11:00 am on November 22, 2024: member

I’m trying to run the arm64 worker job (ARM, unit tests, no functional tests) on my M4 mac, because that should be lot faster than emulating it in Qemu on my x86_64 machine.

Unfortunately so far it fails:

 0[10:34:36.739] Test project /ci_container_base/ci/scratch/build-arm-linux-gnueabihf
 1[10:34:36.742]         Start   2: util_rpcauth_test
 2[10:34:36.744]         Start   4: univalue_object_test
 3[10:34:36.747]         Start   6: secp256k1_tests
 4[10:34:36.751]         Start   8: test_bitcoin-qt
 5[10:34:36.754]         Start  10: addrman_tests
 6[10:34:36.756]   2/141 Test   [#4](/bitcoin-bitcoin/4/): univalue_object_test .................***Failed    0.01 sec
 7[10:34:36.756] /ci_container_base/ci/scratch/build-arm-linux-gnueabihf/src/univalue/object: 1: Syntax error: word unexpected (expecting ")")
 8[10:34:36.756] 
 9[10:34:36.769]   9/141 Test   [#2](/bitcoin-bitcoin/2/): util_rpcauth_test ....................   Passed    0.03 sec
10[10:34:36.888]  10/141 Test   [#1](/bitcoin-bitcoin/1/): util_test_runner .....................***Failed    0.15 sec
11[10:34:36.888] 2024-11-22 10:34:36,767 - ERROR - OSError, Failed to execute /ci_container_base/ci/scratch/build-arm-linux-gnueabihf/src/bitcoin-util

https://cirrus-ci.com/task/4863394807808000

This is running https://github.com/Sjors/bitcoin/pull/67 which is based on master @ 2638fdb4f934be96b7c798dbac38ea5ab8a6374a.

My current setup:

MacBook Pro with M4 Pro chip
UTM to run a VM: https://mac.getutm.app
Ubuntu Server 24.0.1 for ARM: https://ubuntu.com/download/server/arm
UTM machine settings:
- ARM64 (aarch64)
- System: QEMU 7.2 ARM virt-8.2
- 14 GB RAM
- 10 CPU cores
- haven’t touched other default settings

For .cirrus.yaml I use:

0name: MBP24-I
1token: ...
2
3labels:
4  type: arm64

I start the worker with:

0RESTART_CI_DOCKER_BEFORE_RUN=1 cirrus worker run -f ~/.cirrus.yaml

I also tried running the cirrus worker outside a VM, with cirrus installed via Homebrew. For setup that I used Podman Desktop with the Docker extension.

It fails with the same error. Here’s the log for that:

https://cirrus-ci.com/task/4838331626815488

For comparison here’s the same job (still) running on x86_64 with qemu:

https://cirrus-ci.com/task/4837454681735168

maflcko commented at 11:24 am on November 22, 2024: member

My recommendation would be to try without the cirrus overhead for debugging. Just copy-pasting the one-line bash command may be easier and faster for troubleshooting. See https://github.com/bitcoin/bitcoin/tree/master/ci#running-a-stage-locally

As for the error itself: It is not possible to run 32-bit arm binaries on a CPU that only supports 64-bit mode.

I don’t know how to fix this, but if nothing helps, you’ll have to select qemu-arm in UTM. (Is there a reason why you haven’t done this in the first place?)

Relevant CI output:

0[11:51:42.955] + lscpu
1[11:51:42.957] Architecture:                         aarch64
2[11:51:42.957] CPU op-mode(s):                       64-bit

It should be 32-bit, or at least include it:

0[20:50:34.606] + lscpu
1[20:50:34.612] Architecture:                         aarch64
2[20:50:34.612] CPU op-mode(s):                       32-bit, 64-bit

maflcko added the label Tests on Nov 22, 2024

maflcko added the label Questions and Help on Nov 22, 2024

l0rinc commented at 12:09 pm on November 22, 2024: contributor

Not sure it helps, but I have tried reproducing it on my M4 Max, emulating an s390x platform (as I’ve done in #31144 (comment)) - while it’s still very (very) slow:

0brew install podman pigz
1softwareupdate --install-rosetta
2podman machine init
3podman machine start
4docker run --platform linux/s390x -it ubuntu:24.10 /bin/bash

0export DEBIAN_FRONTEND=noninteractive && apt update && apt install -y git build-essential cmake ccache pkg-config libevent-dev libboost-dev libzmq3-dev systemtap-sdt-dev qtbase5-dev qttools5-dev qttools5-dev-tools libqrencode-dev libsqlite3-dev python3 \
1&& cd /mnt && git clone https://github.com/bitcoin/bitcoin.git && cd bitcoin && git remote add Sjors https://github.com/Sjors/bitcoin.git && git fetch --all && git checkout sv2-transport \
2&& cmake --preset dev-mode -DWITH_MULTIPROCESS=OFF && cmake --build build_dev_mode -j$(nproc) \
3&& ctest --test-dir build_dev_mode -j$(nproc) \
4&& build_dev_mode/test/functional/test_runner.py -j$(nproc)

I think I’ve reproduced the error you got:

 0...
 1 util_test_runner .....................***Failed
 2...
 396% tests passed, 5 tests failed out of 136
 4
 5Total Test time (real) = 278.88 sec
 6
 7The following tests FAILED:
 8          1 - univalue_test (Not Run)
 9          2 - univalue_object_test (Not Run)
10          3 - secp256k1_noverify_tests (Not Run)
11          4 - secp256k1_tests (Not Run)
12          5 - secp256k1_exhaustive_tests (Not Run)

Edit: I have fixed the instructions above to work correctly.

Maybe it’s easier to debug this way.

Sjors commented at 12:25 pm on November 22, 2024: member

It is not possible to run 32-bit arm binaries on a CPU that only supports 64-bit mode.

Ah, in that case it’s probably pointless for a performance improvement.

select qemu-arm in UTM. (Is there a reason why you haven’t done this in the first place?)

My initial (and main) goal for that Ubuntu VM was to make Guix builds.

I could spin up a separate machine in 32 bit mode, but that seems like a world of pain on its own. Closing for now.

Sjors closed this on Nov 22, 2024

Sjors commented at 12:27 pm on November 22, 2024: member

I guess my confusion came from the use of arm64 as the label for cirrus, as well as its description “For an aarch64 machine”.

maflcko commented at 12:49 pm on November 22, 2024: member

Yes, the cirrus machine is intentionally an aarch64 (also referred to as arm64) machine, because arm64 is generally available as performant hardware, compared to 32-bit arm. It was also intentionally selected to support 32-bit mode. (If you are looking for a cloud solution, IIRC only Hetzner cloud ships compatible VMs). Otherwise, you’ll have to shop for compatible bare-metal yourself, or use qemu (or similar).

Sjors commented at 2:04 pm on November 22, 2024: member

It was also intentionally selected to support 32-bit mode.

That’s useful to know.

ci: how to run native arm job on Apple silicon? #31344