ci: Future of macOS and Windows MSVC CI tasks #28098

issue maflcko openend this issue on July 18, 2023
  1. maflcko commented at 10:53 am on July 18, 2023: member

    Cirrus CI will cap the community cluster, see https://cirrus-ci.org/blog/2023/07/17/limiting-free-usage-of-cirrus-ci/ .

    Thus, someone would have to sponsor an amount of roughly 5kUSD/mo for those two tasks. Not sure if this is worth it, given that appveyor CI used to cost 50 USD/mo. Also, I am not sure if a sponsor can be found. Personally, I don’t use macOS nor Windows, and I can’t recommend it to anyone, so I don’t mind if the tasks are removed. Though, I am opening this issue to gather feedback and alternatives.

    cc @jonasnick @real-or-random (You’ll probably have to do something about this as well)

    The alternatives I can see (someone would have to implement them):

    • Someone sets up Windows+macOS VMs and connects them to the bitcoin and bitcoin-core Cirrus CI account as a persistent worker. (I’ve only tested this with Linux, so no idea if it works)
    • Go back to appveyor, which was 50 USD/mo. (See removal commit e7c6ed605ca4991856603971f1306dd484dc2f75)
    • Use GitHub Actions CI?
    • Something else?
  2. real-or-random commented at 12:08 pm on July 18, 2023: contributor
    Why are Linux tasks not affected? As far as I understand the blog post, there will be a monthly limit of 40 compute credits (= 40 USD) for all tasks, including Linux tasks. (Or are we already using a persistent worker for Linux?)
  3. maflcko commented at 6:08 pm on July 18, 2023: member

    We already use credits for some Linux tasks, so nothing will change for them. The others (or all) can be moved to persistent workers easily, I suspect.

    Edited OP to mention GitHub Actions CI as alternative. Last time we looked it required write access, but GitHub already has full write access, so I am not sure if this is still a concern or if it even applies?

  4. real-or-random commented at 11:11 pm on July 18, 2023: contributor

    Someone sets up Windows+macOS VMs and connects them to the bitcoin and bitcoin-core Cirrus CI account as a persistent worker. (I’ve only tested this with Linux, so no idea if it works)

    If we can make this happen, this will probably be ideal in the short term. (Heck, even if we convince a sponsor to hire someone, this is much cheaper than paying for compute credits…)

    Last time we looked it required write access,

    See the discussion here: #17803 You had brought up strong arguments against it, but it may indeed be worth reconsidering. As you point out, we trust GitHub anyway. (Or we don’t trust them anyway, in the sense that it’s not the end of the world if they become hostile.) And GitHub Actions is hopefully more mature now, so I don’t think the risk of vulnerabilities should be a showstopper.

    In secp256k1, we currently rely on Cirrus CI’s “Dockerfile as a CI environment” feature for the Linux tasks. Moving away from it could require a bit of work. GitHub Actions apparently has a similar feature, but I haven’t tried that one.

  5. maflcko commented at 6:53 am on July 19, 2023: member

    Maybe Azure Pipelines can be used as an alternative to GitHub CI? Likely they use the same virtual machine pools, but the permissions and write access might be better handled? Also, apparently all of Circle CI, Appveyor CI, and Travis CI now support macOS+Windows, so maybe they can be considered as alternative as well?

    Cirrus CI’s “Dockerfile as a CI env

    Same here. But Linux builds should be trivial to keep on Cirrus CI with persistent workers, which can run podman/docker themselves to cache the result. Ideally we figure out what to do with macOS/Windows within the next 30 days, then dealing with Linux should be trivial.

  6. maflcko added the label Tests on Jul 19, 2023
  7. hebasto commented at 8:42 pm on July 23, 2023: member

    Among free options, CircleCI looks better than others, IMO:

    Personally, I don’t use macOS nor Windows, and I can’t recommend it to anyone, so I don’t mind if the tasks are removed.

    We are responsible for quality of every release we are shipping. And the user is free to make their own choice.

  8. maflcko commented at 6:48 am on July 24, 2023: member

    CircleCI looks better than others

    Can you elaborate on this? The free plan would give less than 10 hours of macOS and Windows per month. And the paid plan doesn’t look cheaper than Cirrus CI, does it?

  9. hebasto commented at 6:51 am on July 24, 2023: member

    CircleCI looks better than others

    Can you elaborate on this? The free plan would give less than 10 hours of macOS and Windows per month. And the paid plan doesn’t look cheaper than Cirrus CI, does it?

    I’ve made comparison among available free plans. I did not consider paid plans in my comment.

  10. maflcko commented at 7:27 am on July 24, 2023: member
    If the goal is to stay on a free plan, I think the only option is GitHub Actions CI. If the goal is to pay a similar amount as with the previous appveyor task, I think the only options are appveyor or travis.
  11. hebasto commented at 4:30 pm on July 24, 2023: member

    If the goal is to stay on a free plan, I think the only option is GitHub Actions CI.

    GitHub Actions: 2000 minutes/month

    CircleCI: 6000 minutes/month

  12. hebasto commented at 8:20 pm on July 24, 2023: member

    Another consideration. AppVeyor still has:

    60 minutes quota per build job.

    That makes running functional tests questionable.

  13. real-or-random commented at 6:07 am on July 25, 2023: contributor

    GitHub Actions: 2000 minutes/month

    AFAIU public repos are free (no limit) for public repos: https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions

  14. hebasto commented at 10:30 pm on July 25, 2023: member

    If the goal is to stay on a free plan, I think the only option is GitHub Actions CI.

    I’ve tried to port our MSVC build from Cirrus CI to GitHub Actions CI and faced an issue: https://github.com/actions/runner-images/issues/7832. A suggested workaround did not help. I’m going to investigate it shortly.

    UPD. A workaround has been found.

  15. maflcko commented at 9:44 am on July 26, 2023: member
    I think before working on GitHub Actions CI, it should be clear how GITHUB_TOKEN behaves inside GitHub CI and whether it allows code pushes from the CI or from pull requests to the main branch.
  16. hebasto commented at 10:01 am on July 26, 2023: member

    I think before working on GitHub Actions CI, it should be clear how GITHUB_TOKEN behaves inside GitHub CI and whether it allows code pushes from the CI or from pull requests to the main branch.

    GitHub Action docs have a plenty of hints about that:

  17. maflcko commented at 10:11 am on July 26, 2023: member
    Ok, so I presume “Maximum access for pull requests from public forked repositories” applies and no further action is needed? Or maybe recommend that in the org (or repo) settings the permission is lowered to restricted just to be safe?
  18. hebasto commented at 10:14 am on July 26, 2023: member

    Or maybe recommend that in the org (or repo) settings the permission is lowered to restricted just to be safe?

    Good idea.

  19. hebasto commented at 12:13 pm on July 26, 2023: member

    Or maybe recommend that in the org (or repo) settings the permission is lowered to restricted just to be safe?

    Here: image

    • Check “Read repository contents and packages permissions”
    • Uncheck “Allow GitHub Actions to create and approve pull requests”
  20. maflcko commented at 1:39 pm on July 29, 2023: member
    Unrelated to this thread, but I am moving the Linux stuff to self-hosted runners (like the “previous releases” task already is). Starting in #28161 with the ASan task, which needs a dedicated Ubuntu Lunar VM.
  21. hebasto commented at 7:17 pm on July 30, 2023: member

    Two PRs are aiming to address this issue:

  22. hebasto commented at 10:02 am on August 2, 2023: member

    Thus, someone would have to sponsor an amount of roughly 5kUSD/mo for those two tasks.

    According to Cirrus CI stats for June / July 2023, the usage of resources was equivalent to compute credits/USD as follows:

    • Windows – 2275 / 1941
    • Linux – 2546 / 3160
    • macOS – 2053 / 3568

    Therefore, future of Linux tasks should be considered as well.

  23. maflcko commented at 10:16 am on August 2, 2023: member

    Therefore, future of Linux tasks should be considered as well.

    Have you seen #28161 ?

  24. fanquake referenced this in commit a4ca497588 on Aug 3, 2023
  25. real-or-random commented at 2:02 pm on August 4, 2023: contributor
    If we now have self-hosted runners for Cirrus, would it make sense (perhaps in the long-run) to make them also self-hosted runners for GitHub Actions? This will remove Cirrus from the loop entirely, which, I think, is a good thing. Cirrus anyway acts only as a coordination layer between GitHub and the runner.
  26. maflcko commented at 2:30 pm on August 4, 2023: member

    This will remove Cirrus from the loop entirely

    Yes, that should work. Assigning labels is also possible, according to https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/using-labels-with-self-hosted-runners#programmatically-assign-labels

    But I guess in the short run it seems easier to stick to Cirrus for now, because the diff is a lot smaller (just replace container: in the yml with persistent_worker:, etc)

  27. real-or-random commented at 10:11 am on August 5, 2023: contributor

    Thus, someone would have to sponsor an amount of roughly 5kUSD/mo for those two tasks

    How did you arrive at these numbers? Is there a better source than the cirrus “metrics” page? I looked at this in the past for libsecp256k1 and arrived at very high USD numbers. Now I redid the numbers for Linux tasks and that’s pretty cheap (for secp256k1):

    Based on the graph in https://cirrus-ci.com/metrics/repository/github/bitcoin-core/secp256k1, we used ~12h/d of Linux tasks in the last 31 days on average, which equals means 66 USD with the new prices. Now I’m really unsure what I did last time.

    edit: Ok, the usage in this repo is way higher, so 5k was probably realistic, assuming the old prices. This will be more like 3-4k with the new prices then, but of course, that’s still not a reasonable alternative to a persistent worker.

  28. hebasto commented at 10:26 am on August 5, 2023: member

    Thus, someone would have to sponsor an amount of roughly 5kUSD/mo for those two tasks

    How did you arrive at these numbers? Is there a better source than the cirrus “metrics” page?

    From https://cirrus-ci.com/settings/github/bitcoin:

    image

    But “Monthly Usage” numbers in https://cirrus-ci.com/settings/github/bitcoin-core are not separated by repositories.

  29. real-or-random commented at 10:39 am on August 5, 2023: contributor
    Oh, thanks, this was the page I couldn’t find anymore. But I don’t understand the numbers. For example for the “bitcoin” org, the page shows 3495 CPU hours and 3160 credits on Linux in July. How does this work out? 3495 h = 209700 min, which translates to 629,1 credits assuming 3 credits/1000 min.
  30. hebasto commented at 10:45 am on August 5, 2023: member

    Oh, thanks, this was the page I couldn’t find anymore. But I don’t understand the numbers. For example for the “bitcoin” org, the page shows 3495 CPU hours and 3160 credits on Linux in July. How does this work out? 3495 h = 209700 min, which translates to 629,1 credits assuming 3 credits/1000 min.

    cc @fkorotkov

  31. hebasto commented at 11:29 am on August 5, 2023: member

    But I don’t understand the numbers.

    Me neither. See https://github.com/cirruslabs/cirrus-ci-web/pull/572#issuecomment-1666478538.

  32. maflcko commented at 10:03 am on August 6, 2023: member

    This will remove Cirrus from the loop entirely

    On a second thought, I don’t think this is possible. We use Cirrus CI compute credits to run the linters immediately, which take a few seconds. I don’t think there is another CI provider that offers this feature, and I think this feature is well worth it to pay for (usually USD$ ~0.01 per run)

  33. hebasto commented at 9:53 am on August 7, 2023: member

    @real-or-random

    But I don’t understand the numbers.

    There was a bug, which has been fixed now.

  34. real-or-random referenced this in commit 96294c00fb on Aug 9, 2023
  35. achow101 referenced this in commit b97b05048d on Aug 15, 2023
  36. fanquake referenced this in commit d78ff380a2 on Aug 17, 2023
  37. maflcko commented at 2:16 pm on August 17, 2023: member

    Closing for now. macOS has been merged and Windows can be discussed in the already open pull: #28173

    For other issues or new issues, a new issue can be opened.

  38. maflcko closed this on Aug 17, 2023
  39. fanquake referenced this in commit 33da5d0eb1 on Aug 23, 2023
  40. fanquake referenced this in commit db57574aa8 on Aug 28, 2023
  41. bitcoin locked this on Aug 16, 2024


maflcko real-or-random hebasto

Labels
Tests

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-10-04 19:12 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me