ci: GHA fallback centos task runs out of space

maflcko commented at 2:26 pm on September 3, 2025: member

https://github.com/bitcoin-core/gui/actions/runs/17433074590/job/49495964258?pr=884#step:8:3785

 0...
 1+ eval 'TEST_RUNNER_EXTRA=()'
 2++ TEST_RUNNER_EXTRA=()
 3+ LD_LIBRARY_PATH=/home/runner/work/_temp/depends/x86_64-pc-linux-gnu/lib
 4+ /home/runner/work/_temp/build/test/functional/test_runner.py --ci -j4 --tmpdirprefix /home/runner/work/_temp/ci/scratch/test_runner/ --ansi --combinedlogslen=99999999 --timeout-factor=40 --quiet --failfast
 5WARNING! There may be insufficient free space in /home/runner/work/_temp/ci/scratch/test_runner//test_runner_₿_🏃_20250903_092842 to run the Bitcoin functional test suite. Running the test suite with fewer than 1426.0 MB of free space might cause tests to fail.
 6...................................................................................................
 7                                                                                                   
 81/272 - feature_block.py failed, Duration: 50 s
 9...
10
11
12 node0 stderr Error: A fatal internal error occurred, see debug.log for details: Disk space is too low!
13Error: A fatal internal error occurred, see debug.log for details: Disk space is too low!
14Error: A fatal internal error occurred, see debug.log for details: Disk space is too low!

maflcko added the label CI failed on Sep 3, 2025

maflcko commented at 2:27 pm on September 3, 2025: member

It may be possible to free up some space with something like https://github.com/google/oss-fuzz/blob/14cab84abbc736e24fede3f0267a8e220fb4974e/.github/workflows/project_tests.yml#L65-L74

willcl-ark commented at 2:29 pm on September 3, 2025: member

 0Free disk space:
 1+ [[ ci_native_centos == \c\i\_\n\a\t\i\v\e\_\a\s\a\n ]]
 2++ /home/runner/work/_temp/depends/config.guess
 3Filesystem      Size  Used Avail Use% Mounted on
 4overlay          72G   48G   25G  67% /
 5tmpfs            64M     0   64M   0% /dev
 6shm              64M     0   64M   0% /dev/shm
 7/dev/root        72G   48G   25G  67% /etc/hosts
 8tmpfs           7.9G     0  7.9G   0% /proc/acpi
 9tmpfs           7.9G     0  7.9G   0% /proc/scsi
10tmpfs           7.9G     0  7.9G   0% /sys/firmware

This job is using more than 25GB?

It may be possible to free up some space with something like google/oss-fuzz@14cab84/.github/workflows/project_tests.yml#L65-L74

Yes we can try to free up space. I’ll take a look where the space is being used and see what we can free.

willcl-ark commented at 7:53 pm on September 3, 2025: member

Seems we are probably already swapping before we start the functional tests:

https://github.com/willcl-ark/bitcoin/actions/runs/17442003339/job/49527090325#step:8:3768

I tested cleaning scratch dir and object files in this run which got it to pass. They seem the obvious choices to me to remove (probably right after building), any other ideas here?

maflcko commented at 6:24 am on September 4, 2025: member

Yeah, I guess we could nuke the build dir object files, but it seems tedious and could make debugging later harder?

I’d say the underlying issue is with GitHub Actions. Conceptually they looked at the npm hell and implemented it for CI. The GHA runners are optimized so that you can test your “sort” module out of the box without installing any dependencies. The runners ship with npm, android, chromium, powershell, graalvm, dotnet, all popular docker images … (at least the last time I looked in https://github.com/google/oss-fuzz/commit/b7f04d782277638a67bc44865de445977eed4708).

So I think the correct fix is to remove stuff on the GHA vm, not inside our CI docker container.

fanquake commented at 12:52 pm on September 4, 2025: member

Yes we can try to free up space. I’ll take a look where the space is being used and see what we can free.

#33304 should free up ~1.5GB of disk usage, an take some pressure off caching generally.

fanquake referenced this in commit 32e2484b67 on Sep 5, 2025

willcl-ark referenced this in commit f411b33a0f on Sep 10, 2025

willcl-ark referenced this in commit f5885d0516 on Sep 10, 2025

willcl-ark referenced this in commit 1c96479056 on Sep 11, 2025

maflcko closed this on Sep 23, 2025

maflcko commented at 6:07 am on September 29, 2025: member

Still happening: https://github.com/bitcoin-core/gui/actions/runs/18064088639

maflcko reopened this on Sep 29, 2025

willcl-ark commented at 10:03 am on September 29, 2025: member

This will be “fixed” by #33480 (as a side-effect). If we decide not to merge that, I can open a PR with something like https://github.com/willcl-ark/bitcoin/commit/f5885d05160fe0ac167976f2b36b6d5bec373e08 in to fix the space issue on CentOS.

maflcko commented at 10:23 am on September 29, 2025: member

This will be “fixed” by #33480 (as a side-effect).

Are you sure? The task build config should be unchanged in this pull request (only the runtime distro is changed) . So if the pull works around it, it doesn’t seem like something that can be relied upon.

I can open a PR with something like willcl-ark@f5885d0 in to fix the space issue on CentOS.

That’d be fine. Maybe just apply it to all GHA runners in the matrix? I guess a vanilla GHA runner may take some time (https://github.com/bitcoin/bitcoin/pull/33365#issuecomment-3324154229)?

ci: GHA fallback centos task runs out of space #33293