Clear out space on GHA jobs

willcl-ark commented at 2:04 pm on October 1, 2025: member

Clear out space on jobs running on GHA by deleteing unnecessary files.

Raised in #33293 which pointed to a solution like https://github.com/google/oss-fuzz/commit/b7f04d782277638a67bc44865de445977eed4708 which is adapted slightly here.

Only runs when cache provider (runner) is gha.

~~A run on my fork can be seen here: https://github.com/willcl-ark/bitcoin/actions/runs/18130629028/job/51596196163#step:6:2~~

A run applying to all jobs when using GHA can be seen here: https://github.com/willcl-ark/bitcoin/actions/runs/18274913987

ci: clear out space on centos job

Clear out space on the centos job be deleteing unnecessary files.

Raised by #33293 which pointed to a solution like https://github.com/google/oss-fuzz/commit/b7f04d782277638a67bc44865de445977eed4708

Only runs when cache provider (runner) is `gha`, and on the CentOS job.

014b9f6268

DrahtBot commented at 2:04 pm on October 1, 2025: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33514.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	janb84, pablomartin4btc, hodlinator

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#bitcoin-core/gui/896 (rpcconsole: display signet challenge by RandyMcMillan)
#33562 (DRAFT: add a freebsd job using systemlibs by willcl-ark)
#33480 (ci: Turn CentOS config into Alpine musl config by maflcko)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

willcl-ark commented at 2:06 pm on October 1, 2025: member

Maintaining an action in this repo which only caters to forks is not ideal, but unless GH increases the space on runners, or provides a “slim” image variant, then it might be the best there is for now.

As it doesn’t run in this repo it should at least never break our CI.

katesalazar commented at 2:21 pm on October 1, 2025: contributor

Which one is the chain needing this change?

janb84 commented at 2:22 pm on October 1, 2025: contributor

So how does this relate to #33480 (remove centos for alpine) . Is this a fix until that PR gets merged ?

willcl-ark commented at 2:23 pm on October 1, 2025: member

Which one is the chain needing this change?

Ah I was slightly unclear; this actually applies to “personal forks” (e.g. my own, your own) as well as actual coin-forks. Anywhere where Cirrus runners, which have more space, are not being used.

So how does this relate to #33480 (remove centos for alpine) . Is this a fix until that PR gets merged ?

I had thought the alpine job would use less space, but I think they are ~ equivalent, so this fix is still needed on that job for runs on forks (see above) to not run out of space.

janb84 commented at 2:55 pm on October 1, 2025: contributor

ACK 014b9f6268390b78e66e1be24813d3f069588442

PR introduces new CI action. This action will remove unnecessary packages to reduce space. This action will only run on GHA runners and will save about 22gb.

~~Kinda surprised about the space savings or why are the packages are there in the first place~~. Anyhow this will help to resolve the GHA space problems for Centos job. (resolves #33293)

EDIT: It’s now more clear that the files are removed of the GHA runner image/host.

fanquake commented at 9:26 pm on October 3, 2025: member

cc @m3dwards

in .github/workflows/ci.yml:521 in 014b9f6268

516@@ -516,6 +517,10 @@ jobs:
517         with:
518           cache-provider: ${{ matrix.provider || needs.runners.outputs.provider }}
519 
520+      - name: Clear unnecessary files
521+        if: ${{ matrix.freeup-space == true }}

maflcko commented at 2:50 pm on October 4, 2025:

0        if: ${{ needs.runners.outputs.provider == 'gha' }}

nit: Could run it for all tasks, to avoid having to select them individually?

hodlinator commented at 7:50 am on October 6, 2025:

Don’t see a problem with the current approach. Would just prefer:

0        if: ${{ matrix.freeup-space }}

maflcko commented at 8:19 am on October 6, 2025:

Don’t see a problem with the current approach.

The problem is that GHA does not run in this repo, so there likely won’t be any indication when this setting is needed. It can practically most likely only be removed/added in a follow-up, which increases the churn a bit.

If the overhead is too large, the step could be run in the background, albeit that would be dropping the reporting feature.

No strong opinion, though, just wanted to mention it.

willcl-ark commented at 8:25 am on October 6, 2025:

Yeah it’s a fair shout.

The step in my run in OP took 4 mins, but I have seen it also take 2 minutes. Either way that’s “wasted time” on the run, although this doesn’t particularly matter on a fork run IMO, as they’re way less time-sensitive, less likely to be queued up with a backlog (and using free compute).

When each job is takining in the order of 1-2 hours, adding 4 minutes on doesnt’ feel that problematic to me.

willcl-ark commented at 8:33 am on October 6, 2025:

nit: Could run it for all tasks, to avoid having to select them individually?

OK I took this suggestion in c01214e72b9876c59924c98e96f43906f5df3353 as the future-proofing/churn-reduction seems worth it.

maflcko approved

maflcko commented at 2:53 pm on October 4, 2025: member

lgtm. left a comment to avoid churn in the future and one of the merge conflicts.

pablomartin4btc commented at 0:28 am on October 6, 2025: member

0Space freed: 22GB (23654664KB)

https://github.com/pablomartin4btc/bitcoin/actions/runs/18266112018/job/52000630402

It happened to me a couple of days ago: https://github.com/pablomartin4btc/bitcoin/actions/runs/18234273673/job/51924700571

in .github/workflows/ci.yml:485 in 014b9f6268 outdated

481@@ -482,6 +482,7 @@ jobs:
482             fallback-runner: 'ubuntu-24.04'
483             timeout-minutes: 120
484             file-env: './ci/test/00_setup_env_native_centos.sh'
485+            freeup-space: ${{ needs.runners.outputs.provider == 'gha' && true || false }} # Only needed on hosted GH runners (forks)

hodlinator commented at 7:46 am on October 6, 2025:

nit: Remove leftovers, unless they serve a purpose?

0            freeup-space: ${{ needs.runners.outputs.provider == 'gha' }} # Only needed on hosted GH runners (forks)

willcl-ark commented at 8:36 am on October 6, 2025:

Removed line altogether as we now run on all jobs on GHA

hodlinator commented at 11:04 am on October 8, 2025:

(Would be nice to clear this up since you switched back to this commit).

hodlinator approved

hodlinator commented at 8:14 am on October 6, 2025: contributor

ACK 014b9f6268390b78e66e1be24813d3f069588442

Was curious if it would take less time to switch to the stream10-minimal container, but it seems to come with the same amount of /usr/local/lib/android files, taking over 3min to remove regardless (https://github.com/hodlinator/bitcoin/actions/runs/18273895919/job/52021497688#step:6:249).

Edit: Just realized the /usr/local/lib/android being removed exists on the host Ubuntu OS, not inside the CentOS container.

maflcko commented at 8:21 am on October 6, 2025: member

Was curious if it would take less time to switch to the stream10-minimal container,

Just for reference, the added step here runs before any of the CI logic is run, so changing the CI config should not have any effect on this step here.

hodlinator commented at 8:23 am on October 6, 2025: contributor

Was curious if it would take less time to switch to the stream10-minimal container,

Just for reference, the added step here runs before any of the CI logic is run, so changing the CI config should not have any effect on this step here.

Yeah, just realized /usr/local/lib/android was being removed on the host OS, not inside the container.

willcl-ark force-pushed on Oct 6, 2025

willcl-ark renamed this:
~~Clear out space on centos job~~
Clear out space on GHA jobs
on Oct 6, 2025

willcl-ark force-pushed on Oct 6, 2025

janb84 commented at 12:19 pm on October 6, 2025: contributor

re ACK c01214e72b9876c59924c98e96f43906f5df3353

Changes sinds last ACK:

changed precondition test for action step.
removed superfluous line that sets a (now unused) variable

I like the solution because the alternative would be to create an custom GHA Image (as suggested by gh) which would be a more maintenance burden than this script.

DrahtBot requested review from pablomartin4btc on Oct 6, 2025

DrahtBot requested review from hodlinator on Oct 6, 2025

fanquake commented at 12:26 pm on October 6, 2025: member

A run applying to all jobs when using GHA can be seen here: https://github.com/willcl-ark/bitcoin/actions/runs/18274913987

Looking there, it seems like the cleanup step can take up to 10 minutes to complete i.e. https://github.com/willcl-ark/bitcoin/actions/runs/18274913987/job/52024710440. Although I guess that is less of an issue if it’s not running in this repo.

willcl-ark commented at 12:35 pm on October 6, 2025: member

A run applying to all jobs when using GHA can be seen here: willcl-ark/bitcoin/actions/runs/18274913987

Looking there, it seems like the cleanup step can take up to 10 minutes to complete i.e. willcl-ark/bitcoin/actions/runs/18274913987/job/52024710440. Although I guess that is less of an issue if it’s not running in this repo.

This is even longer than I’ve seen it take previously. Happy to revert this to only apply to the CentOS job if preferred. Seems pointless to add 10 mins runtime onto like 11 jobs where it’s not needed…

hodlinator approved

hodlinator commented at 9:05 pm on October 6, 2025: contributor

re-ACK c01214e72b9876c59924c98e96f43906f5df3353

I’m okay with applying broadly to ease maintenance, with a very slight preference for changing it back to more surgically apply to CentOS.

janb84 commented at 7:59 am on October 7, 2025: contributor

A run applying to all jobs when using GHA can be seen here: https://github.com/willcl-ark/bitcoin/actions/runs/18274913987

Looking there, it seems like the cleanup step can take up to 10 minutes to complete i.e. https://github.com/willcl-ark/bitcoin/actions/runs/18274913987/job/52024710440. Although I guess that is less of an issue if it’s not running in this repo.

Ok I missed this in my review, I also have a slight preference to revert this to only apply to the CentOS job. I would say lets revert and if the cleanup action needs to be applied on more actions in the future, then go for the generic solution (as applied now) to reduce churn.

m3dwards commented at 2:49 pm on October 7, 2025: contributor

I wondered if it could be sped up and it looks like deleting /usr/local/lib/android is what takes the time. It’s 9.8gb so probably not worth skipping the deletion. I tried a few tricks like rsyncing an empty directory over it but it looks like rm -rf is king in terms of speed.

So I concur, perhaps best to just apply to the job that needs it.

Sjors referenced this in commit 42a69f6e05 on Oct 8, 2025

in .github/workflows/ci.yml:520 in c01214e72b

515@@ -516,6 +516,10 @@ jobs:
516         with:
517           cache-provider: ${{ matrix.provider || needs.runners.outputs.provider }}
518 
519+      - name: Clear unnecessary files
520+        if: ${{ needs.runners.outputs.provider == 'gha' }}

Sjors commented at 10:09 am on October 8, 2025:

Is this automatically set?

willcl-ark commented at 10:18 am on October 8, 2025:

Yes, this is set by the determine-runners job: https://github.com/bitcoin/bitcoin/blob/b510893d00760083ac36948747aa6ebd84656192/.github/workflows/ci.yml#L33-L46

If you have cirrus runners in your own repo you can set a repo variable REPO_USE_CIRRUS_RUNNERS to your repo name to enable cirrus runners in your fork.

willcl-ark commented at 10:19 am on October 8, 2025:

You will see the setting in the Annotations section of the Actions tab on a run:

willcl-ark force-pushed on Oct 8, 2025

Sjors referenced this in commit 3a4a0884d1 on Oct 8, 2025

Sjors commented at 11:16 am on October 8, 2025: member

Happy to revert this to only apply to the CentOS job if preferred. Seems pointless to add 10 mins runtime onto like 11 jobs where it’s not needed…

While testing on with sv2-tp https://github.com/Sjors/sv2-tp/pull/37 I also noticed the cleanup task is quite slow. Because my codebase is much smallers, it basically doubles the runtime. Of course I could simply not cherry-pick this commit - I haven’t run out of space yet - but just FYI.

Sjors referenced this in commit 0a4f1d3728 on Oct 8, 2025

Sjors referenced this in commit 6c2dba678d on Oct 8, 2025

janb84 commented at 3:09 pm on October 13, 2025: contributor

re ACK 014b9f6268390b78e66e1be24813d3f069588442

changes since last ack:

The PR is back to only running the Clear unnecessary files action on the 'CentOS, depends, gui'

DrahtBot commented at 11:54 am on October 15, 2025: contributor

🐙 This pull request conflicts with the target branch and needs rebase.

DrahtBot added the label Needs rebase on Oct 15, 2025

Clear out space on GHA jobs #33514

Code Coverage & Benchmarks

Reviews

Conflicts