ci: Run "macOS 11.0 [gui, no tests] [jammy]" job on GitHub Actions

hebasto commented at 8:27 pm on August 13, 2023: member

In response to upcoming limiting free usage of Cirrus CI, suggesting to move the “macOS 11.0 [gui, no tests] [jammy]” task from Cirrus CI to GitHub Actions (GHA).

(See slightly related discussion in #28098)

Builds in my personal repo:

unpopulated caches: https://github.com/hebasto/bitcoin/actions/runs/5849385308/attempts/1
populated caches: https://github.com/hebasto/bitcoin/actions/runs/5849385308/attempts/2

A list of the used actions for the “Actions permissions” repository settings:

0actions/cache@*,
1actions/checkout@*,
2docker/build-push-action@*,
3docker/setup-buildx-action@*,

ci: Run "macOS 11.0 [gui, no tests] [jammy]" job on GitHub Actions

Also, the "macOS 11.0 [gui, no tests] [jammy]" task has been removed
from the Cirrus CI.

2f1d452740

hebasto added the label macOS on Aug 13, 2023

hebasto added the label Tests on Aug 13, 2023

DrahtBot commented at 8:27 pm on August 13, 2023: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Reviews

See the guideline for information on the review process. A summary of reviews will appear here.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#28269 (ci: Drop no longer needed macos_sdk_cache by hebasto)
#28187 (ci: Run “macOS native x86_64” job on GitHub Actions by hebasto)
#28173 (ci: Run Windows native task on GitHub Actions by hebasto)
#21652 (ci: Switch more tasks to self-hosted by MarcoFalke)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

hebasto marked this as ready for review on Aug 13, 2023

maflcko commented at 9:20 am on August 14, 2023: member

So this is needed for the gui-qml repo to upload assets? Are there any other tasks that need this feature?

hebasto commented at 9:50 am on August 14, 2023: member

So this is needed for the gui-qml repo to upload assets?

Yes, but this is not the only reason for moving to GitHub Actions (GHA).

Other reasons are:

GHA are quicker and cheaper (free) than Cirrus CI. Consider:
GHA are quicker and cheaper (free) than self-hosted permanent Cirrus CI workers. Consider:

Are there any other tasks that need this feature?

https://github.com/bitcoin-core/gui-qml/blob/main/.cirrus.yml uploads artifacts for the tasks as follow:

“‘Win64 [unit tests, no gui tests, no boost::process, no functional tests] [jammy]’”
“macOS 10.15 [gui, no tests] [focal]”
and other 3 tasks not available in the main repo

in .github/workflows/ci.yml:57 in 2f1d452740

52+            CI_IMAGE_NAME_TAG
53+            FILE_ENV
54+          tags: test_image
55+          load: true
56+          cache-from: type=gha
57+          cache-to: type=gha,mode=min

maflcko commented at 10:08 am on August 14, 2023:

What is the cache limit and the inner working?

hebasto commented at 11:05 am on August 14, 2023:

What is the cache limit …?

https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#usage-limits-and-eviction-policy:

… the total size of all caches in a repository is limited to 10 GB.

What is … the inner working?

https://docs.docker.com/build/ci/github-actions/cache/#github-cache

maflcko commented at 11:22 am on August 14, 2023:

So every image will be re-built every week regardless of whether it changed?

Also, is the 10 GB enough to store all ccache + depends + image + … stuff for all tasks?

hebasto commented at 11:33 am on August 14, 2023:

So every image will be re-built every week regardless of whether it changed?

Sorry, I don’t follow. Why will it be re-built every week?

maflcko commented at 11:43 am on August 14, 2023:

Ah, sorry, I assumed cache entries will be unconditionally deleted after 7 days. New question: So every image will be re-built regardless of whether it changed, given 10GB of non-image cache items (ccache+depends+…) have been created?

hebasto commented at 11:47 am on August 14, 2023:

New question: So every image will be re-built regardless of whether it changed…

If no changes were introduced, the image is built from the cached layers.

maflcko commented at 11:49 am on August 14, 2023:

But the cached layers will be evicted after 10GB of (let’s say) ccache items.

the cache eviction policy will create space by deleting the oldest caches in the repository.

hebasto commented at 11:52 am on August 14, 2023:

Yes, it will happen from time to time. But not every run :)

There is an alternative to that approach, which is using Container Registry for images (with write package permission).

maflcko commented at 2:15 pm on August 14, 2023:

Yes, it will happen from time to time. But not every run :)

Assuming a total of 10 GitHub tasks with 500 MB of caches, it will happen every second run. Either way, I am still wondering:

Also, is the 10 GB enough to store all ccache + depends + image + … stuff for all tasks?

There is an alternative to that approach, which is using Container Registry for images (with write package permission).

What about artefacts? (Ref. discussion in the https://github.com/bitcoin-core/secp256k1/pull/1398#issuecomment-1677074743)

hebasto commented at 2:22 pm on August 14, 2023:

What about artefacts? (Ref. discussion in the bitcoin-core/secp256k1#1398 (comment))

Artifacts are subjects of 90 days retention period. Therefore, I still lean to ghcr.io.

hebasto commented at 2:26 pm on August 14, 2023:

Also, is the 10 GB enough to store all ccache + depends + image + … stuff for all tasks?

Well, it depends on what “all tasks” encompass.

Anyway, ccache caches are to be updated on every run (that is not the case in libsecp repo, btw), which means that the current approach to handle Docker images is suboptimal (

maflcko commented at 3:20 pm on August 14, 2023:

Artifacts are subjects of 90 days retention period. Therefore, I still lean to ghcr.io.

A re-build every 3 months seems perfectly fine. ghcr is currently free and unlimited, but I can’t imagine this being the case forever. (The whole reason this pull was opened is because we relied on a service assuming it will be free and unlimited forever). Also, ghcr requires write access.

In any case, CI images are temporary, and it seems cleaner if they are unconditionally deleted after a timeout.

real-or-random commented at 6:53 pm on August 14, 2023:

Yes, artifacts will be deleted after 90 days, but the problem with artifacts is that they can only be downloaded by jobs in the same run. So artifacts are a good way to pass data around within the same run, but they cannot be used for caching across runs.

But I agree that using GHCR feels wrong…. Its purpose is to publish images, so it has a bit of an “official” character. The packages will listed on the repo page for example: https://github.com/hebasto/secp256k1/ (@hebasto Can this be disabled?) On the other hand, it’s possible to delete images, so it wouldn’t be too crazy to “abuse” GHCR for passing CI images and still avoid without eating storage without a limit.

maflcko commented at 6:50 am on August 15, 2023:

On the other hand, it’s possible to delete images, so it wouldn’t be too crazy to “abuse” GHCR for passing CI images and still avoid without eating storage without a limit.

Ah, looks like this is something that should probably be done either way, because untagged images are also kept around. So even if only images for the master branch are used, it would be good to delete “stale tags” (that is, untagged images).

maflcko commented at 7:33 am on August 16, 2023:

Also, is the 10 GB enough to store all ccache + depends + image + … stuff for all tasks?

Well, it depends on what “all tasks” encompass.

A single task currently will cycle through 10GB in less than a day, see https://github.com/bitcoin/bitcoin/actions/caches

maflcko commented at 9:44 am on August 16, 2023:

I guess for this repo, it should be sufficient to only have a master cache (written only by master) and all pull requests and other branches simply read-only from master. This should make the storage needs constant (limited) and easier to think about, without much downside.

real-or-random commented at 10:56 am on August 16, 2023:

You, mean also for the Docker build cache? Yeah, that’s a damn simple approach. Then 10 GB are enough for sure, and okay, PRs that modify the Dockerfile will rebuild the Docker image on every push time, but seems acceptable. Anyway, if working on the images, it makes sense to test/build them locally first.

But wouldn’t you need artifacts again to transfer the image from the Docker build task to the actual CI task, namely in PRs that modify the Dockerfile, to avoid building the image in every single CI task? Or is this acceptable for this repo, because there won’t be many tasks?

The only theoretical problem is this:

GitHub will remove any cache entries that have not been accessed in over 7 days.

It’s rather unlikely in this repo that there will be no PR for 7 days. ;) It may happen in the secp256k1 repo, then we could simply have a cron job that runs once every week to rebuild the Docker image. (This is anyway not a bad idea if we want to rebuild our compiler snapshots.)

maflcko commented at 11:11 am on August 16, 2023:

Or is this acceptable for this repo, because there won’t be many tasks?

There shouldn’t be any difference for this repo, because every task has its own image.

DrahtBot added the label Needs rebase on Aug 15, 2023

DrahtBot commented at 12:03 pm on August 15, 2023: contributor

🐙 This pull request conflicts with the target branch and needs rebase.

dergoegge commented at 12:07 pm on August 17, 2023: member

Any reason to not run this on a self-hosted worker through cirrus? I thought we only wanted to convert to GHA where nobody is willing to self host e.g. the native macOS jobs.

maflcko commented at 12:16 pm on August 17, 2023: member

Any reason to not run this on a self-hosted worker through cirrus?

Self-hosted workers have downsides. For example, they need to be set-up for every fork in a different org, before the CI can be run. But GHA also has downsides. For example, tasks can not be re-run by authors, unless they (force-)push.

And the qml repo needs GHA artefacts?

My suggestion for now would be to use Cirrus self-hosted workers. The qml repo can cherry-pick this pull request in the meantime, until the cache issue is fixed one way or another.

dergoegge commented at 12:18 pm on August 17, 2023: member

My suggestion for now would be to use Cirrus self-hosted workers. The qml repo can cherry-pick this pull request in the meantime, until the cache issue is fixed one way or another.

Sounds good to me. I don’t think we should have a dependency on the experimental gui rewrite in this repo.

fanquake commented at 12:23 pm on August 17, 2023: member

I agree. The requirements of a separate (experimental) repository are not really our concern when making CI infrastructure decisions.

hebasto commented at 12:44 pm on August 17, 2023: member

I agree that UX of GitHub Actions is not perfect, unfortunately.

Closing this PR.

hebasto closed this on Aug 17, 2023

maflcko commented at 7:19 am on August 18, 2023: member

I guess neither solution is perfect, but longer term it would still be fun to explore if something can be achieved via ghcr (or some other registry), see https://github.com/bitcoin/bitcoin/pull/28265/files#r1295647557

bitcoin locked this on Aug 17, 2024

ci: Run “macOS 11.0 [gui, no tests] [jammy]” job on GitHub Actions #28265

Reviews

Conflicts