ci: GHA fallback centos task runs out of space

maflcko commented at 2:26 pm on September 3, 2025: member

https://github.com/bitcoin-core/gui/actions/runs/17433074590/job/49495964258?pr=884#step:8:3785

 0...
 1+ eval 'TEST_RUNNER_EXTRA=()'
 2++ TEST_RUNNER_EXTRA=()
 3+ LD_LIBRARY_PATH=/home/runner/work/_temp/depends/x86_64-pc-linux-gnu/lib
 4+ /home/runner/work/_temp/build/test/functional/test_runner.py --ci -j4 --tmpdirprefix /home/runner/work/_temp/ci/scratch/test_runner/ --ansi --combinedlogslen=99999999 --timeout-factor=40 --quiet --failfast
 5WARNING! There may be insufficient free space in /home/runner/work/_temp/ci/scratch/test_runner//test_runner_₿_🏃_20250903_092842 to run the Bitcoin functional test suite. Running the test suite with fewer than 1426.0 MB of free space might cause tests to fail.
 6...................................................................................................
 7                                                                                                   
 81/272 - feature_block.py failed, Duration: 50 s
 9...
10
11
12 node0 stderr Error: A fatal internal error occurred, see debug.log for details: Disk space is too low!
13Error: A fatal internal error occurred, see debug.log for details: Disk space is too low!
14Error: A fatal internal error occurred, see debug.log for details: Disk space is too low!

maflcko added the label CI failed on Sep 3, 2025

maflcko commented at 2:27 pm on September 3, 2025: member

It may be possible to free up some space with something like https://github.com/google/oss-fuzz/blob/14cab84abbc736e24fede3f0267a8e220fb4974e/.github/workflows/project_tests.yml#L65-L74

willcl-ark commented at 2:29 pm on September 3, 2025: member

 0Free disk space:
 1+ [[ ci_native_centos == \c\i\_\n\a\t\i\v\e\_\a\s\a\n ]]
 2++ /home/runner/work/_temp/depends/config.guess
 3Filesystem      Size  Used Avail Use% Mounted on
 4overlay          72G   48G   25G  67% /
 5tmpfs            64M     0   64M   0% /dev
 6shm              64M     0   64M   0% /dev/shm
 7/dev/root        72G   48G   25G  67% /etc/hosts
 8tmpfs           7.9G     0  7.9G   0% /proc/acpi
 9tmpfs           7.9G     0  7.9G   0% /proc/scsi
10tmpfs           7.9G     0  7.9G   0% /sys/firmware

This job is using more than 25GB?

It may be possible to free up some space with something like google/oss-fuzz@14cab84/.github/workflows/project_tests.yml#L65-L74

Yes we can try to free up space. I’ll take a look where the space is being used and see what we can free.

willcl-ark commented at 7:53 pm on September 3, 2025: member

Seems we are probably already swapping before we start the functional tests:

https://github.com/willcl-ark/bitcoin/actions/runs/17442003339/job/49527090325#step:8:3768

I tested cleaning scratch dir and object files in this run which got it to pass. They seem the obvious choices to me to remove (probably right after building), any other ideas here?

maflcko commented at 6:24 am on September 4, 2025: member

Yeah, I guess we could nuke the build dir object files, but it seems tedious and could make debugging later harder?

I’d say the underlying issue is with GitHub Actions. Conceptually they looked at the npm hell and implemented it for CI. The GHA runners are optimized so that you can test your “sort” module out of the box without installing any dependencies. The runners ship with npm, android, chromium, powershell, graalvm, dotnet, all popular docker images … (at least the last time I looked in https://github.com/google/oss-fuzz/commit/b7f04d782277638a67bc44865de445977eed4708).

So I think the correct fix is to remove stuff on the GHA vm, not inside our CI docker container.

fanquake commented at 12:52 pm on September 4, 2025: member

Yes we can try to free up space. I’ll take a look where the space is being used and see what we can free.

#33304 should free up ~1.5GB of disk usage, an take some pressure off caching generally.

ci: GHA fallback centos task runs out of space #33293