arm64 CI failure #27879

issue ajtowns openend this issue on June 14, 2023
  1. ajtowns commented at 2:23 am on June 14, 2023: contributor

    arm64 CI is failing with:

     0FAIL: minisketch/test
     1=====================
     2
     3../build-aux/test-driver: line 109: ./minisketch/test: cannot execute binary file: Exec format error
     4FAIL minisketch/test (exit status: 126)
     5
     6FAIL: univalue/test/object
     7==========================
     8
     9../build-aux/test-driver: line 109: ./univalue/test/object: cannot execute binary file: Exec format error
    10FAIL univalue/test/object (exit status: 126)
    11
    12FAIL: univalue/test/unitester
    13=============================
    14
    15../build-aux/test-driver: line 109: ./univalue/test/unitester: cannot execute binary file: Exec format error
    16FAIL univalue/test/unitester (exit status: 126)
    17
    18FAIL: qt/test/test_bitcoin-qt
    19=============================
    20
    21../build-aux/test-driver: line 109: ./qt/test/test_bitcoin-qt: cannot execute binary file: Exec format error
    22FAIL qt/test/test_bitcoin-qt (exit status: 126)
    

    See eg https://github.com/bitcoin/bitcoin/pull/27647/checks?check_run_id=14232303009 or https://github.com/bitcoin/bitcoin/pull/27746/checks?check_run_id=14231876127

    I tried re-running the arm64 task for #27675 which had succeeded yesterday, and it now fails too (before vs after), which suggests either that the bug was introduced after commit fbe48f97dfec3138b06b5f00b75655da0c985008 up to commit 427853ab49f610e971b73ea4cc1d5366747e52b1 (nothing seemed likely to break minisketch/test there) or is an infrastructure issue (bad cache or bad hardware?).

  2. ajtowns commented at 2:29 am on June 14, 2023: contributor
    Maybe worth adding git show FETCH_HEAD at the end of base_template: in .cirrus.yml so it’s easier to see which version of master a CI run was merged against?
  3. jonatack commented at 3:07 am on June 14, 2023: contributor
    Saw this issue today on two pushes as well, #27425 and #27632. Re-running the task didn’t help.
  4. maflcko commented at 7:04 am on June 14, 2023: member

    Maybe worth adding git show FETCH_HEAD at the end of base_template: in .cirrus.yml so it’s easier to see which version of master a CI run was merged against?

    I think you can check the output of merge_base to see what was used? For example, https://cirrus-ci.com/task/6628961686388736?logs=merge_base#L18

    or is an infrastructure issue (bad cache or bad hardware?).

    Seems likely. Let’s poke a bit…

  5. ajtowns commented at 7:12 am on June 14, 2023: contributor

    I think you can check the output of merge_base to see what was used? For example, https://cirrus-ci.com/task/6628961686388736?logs=merge_base#L18

    I looked at that and somehow didn’t see it. Weird.

  6. maflcko commented at 7:15 am on June 14, 2023: member
    There is certainly something sketchy going on with the upstream hardware/software. I’ve had to reset the pre-build of the image thrice: https://cirrus-ci.com/task/4729934984249344. Also, once that passes, the actual build fails half the time with curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to raw.githubusercontent.com:443 https://cirrus-ci.com/task/5920692756021248?logs=ci#L208
  7. maflcko added the label Tests on Jun 14, 2023
  8. maflcko added the label Upstream on Jun 14, 2023
  9. hebasto commented at 7:20 am on June 14, 2023: member

    There is certainly something sketchy going on with the upstream hardware/software. I’ve had to reset the pre-build of the image thrice: https://cirrus-ci.com/task/4729934984249344. Also, once that passes, the actual build fails half the time with curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to raw.githubusercontent.com:443 https://cirrus-ci.com/task/5920692756021248?logs=ci#L208

    @fkorotkov What do you think?

  10. maflcko commented at 8:56 am on June 14, 2023: member
    Aparently AWS was down yesterday? Maybe someone needs to turn a server off and on again to fix it?
  11. maflcko commented at 10:54 am on June 14, 2023: member
    My suggestion would be to wait 24 hours to see if the issue fixes itself. If not, replace the arm64 hardware with amd64, and run everything in qemu-arm again.
  12. hebasto commented at 11:05 am on June 14, 2023: member

    There is certainly something sketchy going on with the upstream hardware…

    It looks like the underlying hardware stopped to support running ARM-32bit binaries on arm64 hardware. Actually, there was not such a promise from Cirrus Labs about that.

    My suggestion would be to wait 24 hours to see if the issue fixes itself. If not, replace the arm64 hardware with amd64, and run everything in qemu-arm again.

    Another option is to build aarch64 binaries using an arm_container in this task.

  13. maflcko commented at 11:16 am on June 14, 2023: member
    Thanks for taking a look, do you mind updating the CI task config for aarch64? https://github.com/bitcoin/bitcoin/blob/427853ab49f610e971b73ea4cc1d5366747e52b1/ci/test/00_setup_env_arm.sh#L9-L14
  14. hebasto commented at 11:20 am on June 14, 2023: member

    do you mind updating the CI task config for aarch64?

    I’ll do it. No need to wait a happy self fix? :)

  15. maflcko commented at 11:22 am on June 14, 2023: member
    Longer term it may also be worth to consider removing the arm32 release builds, unless there are still people using them? Maybe someone can ask on the mailing list?
  16. pinheadmz commented at 11:25 am on June 14, 2023: member

    arm32 release builds

    meaning (for example) bitcoin-25.0-arm-linux-gnueabihf.tar.gz 😬 that’s the Raspberry Pi binary

  17. maflcko commented at 11:29 am on June 14, 2023: member

    meaning (for example) bitcoin-25.0-arm-linux-gnueabihf.tar.gz grimacing that’s the Raspberry Pi binary

    Ah ok, that’d be the Pi 1 and Zero? All other (Zero 2, 3, 4, …) are 64 bit, IIRC.

  18. pinheadmz commented at 11:36 am on June 14, 2023: member

    the hardware has been 64 bit since Pi3 I think but the OS itself has only been 64 bit for a year or so:

    https://www.raspberrypi.com/news/raspberry-pi-os-64-bit/

  19. fkorotkov commented at 11:39 am on June 14, 2023: none
    The only Arm related change yesterday was that we switched from AWS using Graviton2 CPUs to GCP using Ampere CPUs. Both Graviton2 and Ampere Alta support EL0 AArch32.
  20. fanquake commented at 12:14 pm on June 14, 2023: member

    Both Graviton2 and Ampere Alta support EL0 AArch32.

    Yes, however GCP explicitly doesn’t support 32-bit mode?

    The Tau T2A machine series does not support: 32-bit mode ELO (guest userspace support)

  21. hebasto referenced this in commit d36f664349 on Jun 14, 2023
  22. hebasto referenced this in commit 6cfd5d3081 on Jun 14, 2023
  23. maflcko commented at 12:32 pm on June 14, 2023: member

    Thanks everyone for the comments. I think a third alternative for us, if we want minimal changes, would be to unset QEMU_USER_CMD and run qemu on aarch64 for now?

    Also on Google Cloud t2a-standard-4:

    0lscpu | grep op-mode
    1CPU op-mode(s):                  64-bit
    
  24. hebasto referenced this in commit 016fe6d828 on Jun 14, 2023
  25. hebasto commented at 12:34 pm on June 14, 2023: member

    I think a third alternative for us, if we want minimal changes, would be to unset QEMU_USER_CMD and run qemu on aarch64 for now?

    I’ve been testing this change since this morning. The CI constantly timed out for some reasons.

  26. fanquake closed this on Jun 14, 2023

  27. fanquake referenced this in commit e6141298e3 on Jun 14, 2023
  28. fanquake referenced this in commit 6233049709 on Jun 15, 2023
  29. sidhujag referenced this in commit fa9c9c7b9e on Jun 15, 2023
  30. janus referenced this in commit fd93765328 on Sep 11, 2023
  31. delta1 referenced this in commit c694cc66eb on Sep 27, 2023
  32. psgreco referenced this in commit aa1969d96b on Sep 28, 2023
  33. psgreco referenced this in commit 5b36368269 on Sep 28, 2023
  34. fanquake referenced this in commit 43596499b2 on Oct 2, 2023
  35. delta1 referenced this in commit 475b52c533 on Oct 9, 2023
  36. jamesdorfman referenced this in commit de8b4c87bc on Oct 9, 2023
  37. timemarkovqtum referenced this in commit a8d0f6c863 on Jan 30, 2024
  38. bitcoin locked this on Jun 13, 2024

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-07-08 19:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me