Compiler ICEs in CI? #36125

Looks like there are a bunch of GCC 12 ICEs in the Warp CI:

• Task previous releases: https://github.com/bitcoin/bitcoin/actions/runs/32101345101/job/95602400388 LLM reason (✨ experimental): CI failed due to a GCC 12 internal compiler error (segmentation fault) while compiling src/rpc/blockchain.cpp.

• Task previous releases: https://github.com/bitcoin/bitcoin/actions/runs/33372693809/job/99427161856 LLM reason (✨ experimental): CI failed due to a GCC 12 internal compiler error (segmentation fault) while compiling txmempool/tinyformat.h during the build.

Not sure why they would pop up now, when they never happened before?

cc @willcl-ark Maybe someone can ask warpbuild if the two builds ran on the same hardware/VM. If yes, maybe that hardware should be checked for defects?

Done 👍🏼

Thx.

On the other hand, both happening for the exact same CI config (and on no other config), seems to indicate that maybe something may be wrong in that config (or any of it's dependencies)?

An OOM seems unlikely, given the 20GiB of free RAM.

Maybe this will fix itself, when minimum GCC is switched to 13.

Ok, nvm. Clang 17 also ICEs:

• Task No wallet: https://github.com/bitcoin/bitcoin/actions/runs/33087954636/job/99830997698 LLM reason (✨ experimental): CI failed due to a Clang 17 compiler segmentation fault (exit code 139) while compiling capnp/mining.capnp.proxy-server.c++.

Warp confirmed the two first jobs were on different hardware too.

Not sure what the issue is, but this is getting out of hand:

• Task Alpine (musl): https://github.com/bitcoin/bitcoin/actions/runs/33526907826/job/99919961765 LLM reason (✨ experimental): CI failed due to a compiler crash (internal compiler error: Segmentation fault) while building the fuzz target (cmpctblock.cpp).

This seems to have stopped. Closing for now, but lets re-open if we see another ICE.

This keeps happening:

/home/runner/work/_temp/depends/work/build/x86_64-pc-linux-musl/qt/6.8.4-07e5dcbf2ce/qtbase/src/corelib/global/qnamespace.h:1081:5: internal compiler error: Segmentation fault

https://github.com/bitcoin/bitcoin/actions/runs/36682038540/job/109779396458?pr=36033#step:11:2551

Probably also #36217?

bitcoin/bitcoin/actions/runs/36682038540/job/109779396458?pr=36033#step:11:2551

Hmmm, this looks like it's segfaulted in a few places. And qt is still using -std=gnu++20 which pretty definitively rules out the c++23 update...

Noting for future self this was run on runner warp-8x-x64-w9y4tfitn8d6bn36

Have sent a ticket into Warp to ask if there's anything they can help investigate on their side.

Fed the system logs from the run to chatgpt:

<details> <summary>Details</summary>

[INFO]2026-09-30T07:16:46.718308+00:00 warp-8x-x64-w9y4tfitn8d6bn36 kernel: moc[48422]: segfault at 7fda98ba8d61 ip 0000560b1d34788b sp 00007ffd2633d820 error 6 in moc[1e88b,560b1d32f000+c5000] likely on CPU 2 (core 2, socket 0)
[INFO]2026-09-30T07:16:46.718331+00:00 warp-8x-x64-w9y4tfitn8d6bn36 kernel: Code: 00 85 c0 75 45 48 8b 7b 08 ba 10 00 00 00 be 01 00 00 00 e8 87 dd 08 00 48 83 c3 30 49 39 dc 74 8d 48 8b 43 08 48 85 c0 74 ee <f0> 83 28 01 74 c6 eb e6 ba 62 00 00 00 48 8d 35 71 e7 0a 00 48 8d
[INFO]2026-09-30T07:16:46.727227+00:00 warp-8x-x64-w9y4tfitn8d6bn36 systemd[1]: Created slice system-systemd\x2dcoredump.slice - Slice /system/systemd-coredump.
[INFO]2026-09-30T07:16:46.733835+00:00 warp-8x-x64-w9y4tfitn8d6bn36 systemd[1]: Started systemd-coredump@0-48429-0.service - Process Core Dump (PID 48429/UID 0).
[INFO]2026-09-30T07:16:47.143008+00:00 warp-8x-x64-w9y4tfitn8d6bn36 systemd-coredump[48430]: Process 48422 (moc) of user 0 dumped core.
[INFO]2026-09-30T07:16:47.145306+00:00 warp-8x-x64-w9y4tfitn8d6bn36 kernel: audit: type=1400 audit(1790752607.142:5): apparmor="DENIED" operation="userns_create" class="namespace" info="Userns create restricted - failed to find unprivileged_userns profile" error=-13 profile="unconfined" pid=48430 comm="systemd-coredum" requested="userns_create" denied="userns_create" target="unprivileged_userns" execpath="/usr/lib/systemd/systemd-coredump"
[INFO]2026-09-30T07:16:47.166241+00:00 warp-8x-x64-w9y4tfitn8d6bn36 systemd[1]: systemd-coredump@0-48429-0.service: Deactivated successfully.
[INFO]2026-09-30T07:16:47.166452+00:00 warp-8x-x64-w9y4tfitn8d6bn36 systemd[1]: systemd-coredump@0-48429-0.service: Triggering OnSuccess= dependencies.
[INFO]2026-09-30T07:16:47.172113+00:00 warp-8x-x64-w9y4tfitn8d6bn36 systemd[1]: Created slice system-apport\x2dcoredump\x2dhook.slice - Slice /system/apport-coredump-hook.
[INFO]2026-09-30T07:16:47.181631+00:00 warp-8x-x64-w9y4tfitn8d6bn36 systemd[1]: Starting apport-coredump-hook@0-48429-0.service...
[INFO]2026-09-30T07:16:47.547322+00:00 warp-8x-x64-w9y4tfitn8d6bn36 kernel: moc[48426]: segfault at 7fe35bc63acd ip 00007fe25cbd4aa0 sp 00007ffcc8e20278 error 4 in ld-musl-x86_64.so.1[23aa0,7fe25cbc5000+59000] likely on CPU 6 (core 6, socket 0)
[INFO]2026-09-30T07:16:47.547343+00:00 warp-8x-x64-w9y4tfitn8d6bn36 kernel: Code: c0 c3 0f 1f 84 00 00 00 00 00 49 89 f8 49 89 d1 eb 1d 66 90 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 <a4> 49 83 e9 01 48 89 f0 40 f6 c6 03 74 4e 4d 85 c9 75 ed 48 89 f9
[INFO]2026-09-30T07:16:47.562647+00:00 warp-8x-x64-w9y4tfitn8d6bn36 systemd[1]: Started systemd-coredump@1-48446-0.service - Process Core Dump (PID 48446/UID 0).
[INFO]2026-09-30T07:16:48.020303+00:00 warp-8x-x64-w9y4tfitn8d6bn36 kernel: audit: type=1400 audit(1790752608.019:6): apparmor="DENIED" operation="userns_create" class="namespace" info="Userns create restricted - failed to find unprivileged_userns profile" error=-13 profile="unconfined" pid=48447 comm="systemd-coredum" requested="userns_create" denied="userns_create" target="unprivileged_userns" execpath="/usr/lib/systemd/systemd-coredump"
[INFO]2026-09-30T07:16:48.021833+00:00 warp-8x-x64-w9y4tfitn8d6bn36 systemd-coredump[48447]: Process 48426 (moc) of user 0 dumped core.
[INFO]2026-09-30T07:16:48.040372+00:00 warp-8x-x64-w9y4tfitn8d6bn36 systemd[1]: systemd-coredump@1-48446-0.service: Deactivated successfully.
[INFO]2026-09-30T07:16:48.042945+00:00 warp-8x-x64-w9y4tfitn8d6bn36 systemd[1]: systemd-coredump@1-48446-0.service: Triggering OnSuccess= dependencies.
[INFO]2026-09-30T07:16:48.052602+00:00 warp-8x-x64-w9y4tfitn8d6bn36 systemd[1]: Starting apport-coredump-hook@1-48446-0.service...
[INFO]2026-09-30T07:16:49.101999+00:00 warp-8x-x64-w9y4tfitn8d6bn36 systemd[1]: apport-coredump-hook@0-48429-0.service: Main process exited, code=exited, status=1/FAILURE
[INFO]2026-09-30T07:16:49.102132+00:00 warp-8x-x64-w9y4tfitn8d6bn36 systemd[1]: apport-coredump-hook@0-48429-0.service: Failed with result 'exit-code'.
[INFO]2026-09-30T07:16:49.102190+00:00 warp-8x-x64-w9y4tfitn8d6bn36 systemd[1]: Failed to start apport-coredump-hook@0-48429-0.service.
[INFO]2026-09-30T07:16:49.102239+00:00 warp-8x-x64-w9y4tfitn8d6bn36 systemd[1]: apport-coredump-hook@0-48429-0.service: Consumed 1.692s CPU time.
[INFO]2026-09-30T07:16:49.384821+00:00 warp-8x-x64-w9y4tfitn8d6bn36 systemd[1]: apport-coredump-hook@1-48446-0.service: Main process exited, code=exited, status=1/FAILURE
[INFO]2026-09-30T07:16:49.384926+00:00 warp-8x-x64-w9y4tfitn8d6bn36 systemd[1]: apport-coredump-hook@1-48446-0.service: Failed with result 'exit-code'.
[INFO]2026-09-30T07:16:49.385125+00:00 warp-8x-x64-w9y4tfitn8d6bn36 systemd[1]: Failed to start apport-coredump-hook@1-48446-0.service.
[INFO]2026-09-30T07:16:49.385262+00:00 warp-8x-x64-w9y4tfitn8d6bn36 systemd[1]: apport-coredump-hook@1-48446-0.service: Consumed 1.329s CPU time.

</details>

These confirm real memory-access faults, but still don’t establish faulty runner hardware.

The relevant events are:

| Time UTC | Event | Meaning | |---|---|---| | 07:16:46.718 | moc[48422], error 6, instruction in moc | User-mode write to a non-present page | | 07:16:47.547 | moc[48426], error 4, instruction in musl | User-mode read from a non-present page |

Those interpretations follow the Linux x86 fault-code definitions. A fault inside musl can result from the caller supplying an invalid pointer; it doesn’t establish a musl bug. Likewise, “likely on CPU 2” identifies where the process ran, not a defective CPU.

The AppArmor denials are subsequent crash-handling events. They name systemd-coredump and occur after the segfaults. They may interfere with collecting or processing the dumps, but didn’t cause these initial crashes.

The earlier Docker, cgroup, and DNS errors have no demonstrated connection to the invalid memory accesses. There’s also no OOM-kill, machine-check, or ECC error report in this excerpt.

My assessment remains: three GCC crashes plus two moc crashes make a shared environment problem worth investigating, but hardware corruption is only one possibility. Software bugs, a miscompiled executable, or corrupted build artifacts remain possible.

The useful new lead is potentially available core dumps. Ask WarpBuild to preserve the dumps for PIDs 48422 and 48426, along with the exact moc executable and Alpine libraries. Those could reveal the bad pointers and calling paths. Comparing an identical clean build on another runner remains the best next discriminator.

Ah, wasn't aware there are system logs. Maybe we can add a temporary CI patch to upload all core dumps somewhere on failure? However, I presume the coredump will just show the invalid pointer/memory and not explain if this was a cosmic ray or a software bug. But it can't hurt to try?

Update: Warp have moved the bitcoin org onto a different hardware pool, which should hopefully put a stop to these intermittent issues.

Going to close this (again) for now.

I suspect the time to add a CI coredump debug/upload is if this happens again, even on the now-changed hardware.