ci: Switch remaining Linux tasks to self-hosted #21652

pull MarcoFalke wants to merge 2 commits into bitcoin:master from MarcoFalke:2104-ciMoreSelf changing 2 files +48 −88
  1. MarcoFalke commented at 11:16 am on April 11, 2021: member

    Cirrus CI will be capping the free compute soon. For now, switch more tasks to persistent worker, as recommended by Cirrus CI.

    (See slightly related discussion in #28098)

  2. fanquake added the label Tests on Apr 11, 2021
  3. MarcoFalke force-pushed on Apr 11, 2021
  4. DrahtBot commented at 3:24 pm on April 11, 2021: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK dergoegge, pinheadmz, hebasto

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #28210 (build: Bump minimum supported Clang to clang-13 by MarcoFalke)
    • #28173 (ci: Run Windows native task on GitHub Actions by hebasto)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  5. MarcoFalke force-pushed on Apr 14, 2021
  6. MarcoFalke force-pushed on Apr 17, 2021
  7. MarcoFalke force-pushed on Jun 4, 2021
  8. DrahtBot added the label Needs rebase on Dec 13, 2021
  9. MarcoFalke closed this on Dec 27, 2021
  10. MarcoFalke deleted the branch on Dec 27, 2021
  11. bitcoin locked this on Dec 27, 2022
  12. bitcoin unlocked this on Jul 24, 2023
  13. MarcoFalke restored the branch on Jul 24, 2023
  14. MarcoFalke renamed this:
    [WIP NOMERGE DRAFT] ci: Switch more tasks to self-hosted
    ci: Switch more tasks to self-hosted
    on Jul 24, 2023
  15. MarcoFalke reopened this on Jul 24, 2023
  16. MarcoFalke force-pushed on Jul 24, 2023
  17. MarcoFalke force-pushed on Jul 24, 2023
  18. DrahtBot added the label CI failed on Jul 24, 2023
  19. DrahtBot removed the label Needs rebase on Jul 24, 2023
  20. MarcoFalke force-pushed on Jul 28, 2023
  21. MarcoFalke force-pushed on Jul 29, 2023
  22. MarcoFalke force-pushed on Jul 30, 2023
  23. MarcoFalke force-pushed on Jul 30, 2023
  24. MarcoFalke commented at 10:07 am on July 30, 2023: member

    Looks like there is an intermittent issue, which is fixed in podman 4.1:

    0[#135258](/bitcoin-bitcoin/135258/)	REDUCE cov: 2818 ft: 7924 corp: 483/10261b lim: 254 exec/s: 3468 rss: 251Mb L: 15/177 MS: 1 EraseBytes-
1[#135494](/bitcoin-bitcoin/135494/)	NEWError: timed out waiting for file /var/lib/containers/storage/overlay-containers/2b5173104c7716f28471c2aed46932cd57b0904326ef3b5e3c9c0462dad553a6/userdata/75976ef6638693018268a2b4187292027a833405bd50e9ab4a3ddccae789be0b/exit/2b5173104c7716f28471c2aed46932cd57b0904326ef3b5e3c9c0462dad553a6: internal libpod error
2
3Exit status: 255
  25. MarcoFalke force-pushed on Jul 31, 2023
  26. MarcoFalke force-pushed on Aug 4, 2023
  27. MarcoFalke force-pushed on Aug 9, 2023
  28. MarcoFalke force-pushed on Aug 15, 2023
  29. MarcoFalke force-pushed on Aug 15, 2023
  30. MarcoFalke force-pushed on Aug 15, 2023
  31. MarcoFalke force-pushed on Aug 15, 2023
  32. DrahtBot added the label Needs rebase on Aug 15, 2023
  33. MarcoFalke force-pushed on Aug 16, 2023
  34. DrahtBot removed the label Needs rebase on Aug 16, 2023
  35. DrahtBot removed the label CI failed on Aug 16, 2023
  36. DrahtBot added the label Needs rebase on Aug 16, 2023
  37. MarcoFalke force-pushed on Aug 16, 2023
  38. MarcoFalke force-pushed on Aug 16, 2023
  39. DrahtBot added the label CI failed on Aug 16, 2023
  40. DrahtBot removed the label Needs rebase on Aug 16, 2023
  41. MarcoFalke marked this as ready for review on Aug 17, 2023
  42. MarcoFalke force-pushed on Aug 17, 2023
  43. MarcoFalke force-pushed on Aug 18, 2023
  44. ci: Switch remaining tasks to self-hosted 
    This allows to drop unused templates, such as
cirrus_ephemeral_worker_template_env, or container_depends_template.

Also, ccache_cache, previous_releases_cache, and
base_depends_built_cache can be dropped, because the caching is done in
container volumes on the self-hosted runners.
    fad006fa0a
  45. ci: Remove distro-name from task name 
    The exact distro name should not be important. Also, it is easy to find
out, if needed. Thus, remove it to avoid bloat and maintenance overhead
having to keep it in sync.
    fa8e89d5e4
  46. MarcoFalke force-pushed on Aug 18, 2023
  47. MarcoFalke commented at 4:18 pm on August 18, 2023: member
    Added a description to the commits, to make it easier to review.
  48. MarcoFalke added this to the milestone 26.0 on Aug 18, 2023
  49. DrahtBot removed the label CI failed on Aug 18, 2023
  50. MarcoFalke requested review from hebasto on Aug 21, 2023
  51. MarcoFalke requested review from pinheadmz on Aug 21, 2023
  52. MarcoFalke renamed this:
    ci: Switch more tasks to self-hosted
    ci: Switch remaining tasks to self-hosted
    on Aug 22, 2023
  53. dergoegge approved
  54. dergoegge commented at 11:55 am on August 22, 2023: member
    ACK fa8e89d5e48c4554eddef611eb002b61f3305272
  55. pinheadmz commented at 5:09 pm on August 22, 2023: member

    concept ACK fa8e89d5e48c4554eddef611eb002b61f3305272

    Questions:

    Is Win64 native still running on cirrus? The code changes look like it shouldnt be but it has a warning:

    Monthly free compute limit exceeded and will be limited next month!

    The MSan task took 2.5 hours! yow. Is the bottleneck there just hardware?

  56. DrahtBot removed review request from pinheadmz on Aug 22, 2023
  57. MarcoFalke renamed this:
    ci: Switch remaining tasks to self-hosted
    ci: Switch remaining Linux tasks to self-hosted
    on Aug 23, 2023
  58. MarcoFalke commented at 7:44 am on August 23, 2023: member

    Is Win64 native still running on cirrus? The code changes look like it shouldnt be but it has a warning:

    Yes, sorry for the confusion. This is only about Linux. I’ve adjusted the title. There is another pull about the msvc on Windows.

  59. MarcoFalke commented at 7:45 am on August 23, 2023: member

    The MSan task took 2.5 hours! yow. Is the bottleneck there just hardware?

    On the first run it will build llvm/clang/msan + depends + Bitcoin Core on a fresh cache. See also the comment in the cirrus yaml.

  60. MarcoFalke requested review from fanquake on Aug 23, 2023
  61. hebasto approved
  62. hebasto commented at 10:30 am on August 23, 2023: member

    ACK fa8e89d5e48c4554eddef611eb002b61f3305272.

    Observing very low Ccache hit rate. For example, in https://cirrus-ci.com/task/6728143201894400:

     0ccache version 3.7.7
 1cache directory                     /tmp/ccache_dir
 2primary config                      /tmp/ccache_dir/ccache.conf
 3secondary config      (readonly)    /etc/ccache.conf
 4stats updated                       Fri Aug 18 17:06:37 2023
 5cache hit (direct)                   110
 6cache hit (preprocessed)              15
 7cache miss                           737
 8cache hit rate                     14.50 %
 9called for link                       12
10cleanups performed                     0
11files in cache                      1489
12cache size                          33.6 MB
13max cache size                     200.0 MB
  63. MarcoFalke commented at 10:35 am on August 23, 2023: member
    I wonder why the hit rate is non-zero. On the first run, the cache is empty, obviously. Maybe a leftover ccache from a previous push to this pull request?
  64. hebasto commented at 10:38 am on August 23, 2023: member

    @MarcoFalke

    I wonder why the hit rate is non-zero. On the first run, the cache is empty, obviously. Maybe a leftover ccache from a previous push to this pull request?

    Can you please re-run all Cirrus jobs to test caching facilities?

  65. MarcoFalke commented at 10:51 am on August 23, 2023: member

    Can you please re-run all Cirrus jobs to test caching facilities?

    There are multiple workers per label type and multiple tasks using the same label type. Thus, it will take many re-runs to populate the initial cache on all workers.

  66. MarcoFalke commented at 10:52 am on August 23, 2023: member
    I could limit the label type to each task, but that would be micro-managment. And free resources on one label type can not be used on another label type.
  67. hebasto commented at 10:53 am on August 23, 2023: member

    Can you please re-run all Cirrus jobs to test caching facilities?

    There are multiple workers per label type and multiple tasks using the same label type. Thus, it will take many re-runs to populate the initial cache on all workers.

    Fair enough. We can observe caching quality in-progress later.

  68. hebasto commented at 10:56 am on August 23, 2023: member

    There are multiple workers per label type and multiple tasks using the same label type.

    Sounds like we need to increase CCACHE_MAXSIZE to cope such a usage, no?

  69. MarcoFalke commented at 11:02 am on August 23, 2023: member

    Sounds like we need to increase CCACHE_MAXSIZE to cope such a usage, no?

    Why? Each task has its own name (space). The CI is now doing exactly what happens when you run the CI locally.

  70. hebasto commented at 11:12 am on August 23, 2023: member

    Sounds like we need to increase CCACHE_MAXSIZE to cope such a usage, no?

    Why? Each task has its own name (space). The CI is now doing exactly what happens when you run the CI locally.

    IIUC, multiple tasks share the same CCACHE_DIR. If they use different compilers or different compiler flags, they need more space to do not purge cached items from another task, no?

  71. MarcoFalke commented at 11:31 am on August 23, 2023: member

    Sounds like we need to increase CCACHE_MAXSIZE to cope such a usage, no?

    Why? Each task has its own name (space). The CI is now doing exactly what happens when you run the CI locally.

    IIUC, multiple tasks share the same CCACHE_DIR. If they use different compilers or different compiler flags, they need more space to do not purge cached items from another task, no?

    No, and as I said this is unrelated to the changes here. If there is a bug in the CI system when running locally, it should be fixed separately, not as part of this pull.

    See also:

    0$ git grep _ccache ci
1ci/test/04_install.sh:  docker volume create "${CONTAINER_NAME}_ccache" || true
2ci/test/04_install.sh:                  --mount "type=volume,src=${CONTAINER_NAME}_ccache,dst=$CCACHE_DIR" \
  72. fanquake merged this on Aug 23, 2023
  73. fanquake closed this on Aug 23, 2023
  74. MarcoFalke deleted the branch on Aug 23, 2023
  75. hebasto commented at 1:30 pm on August 23, 2023: member
    https://github.com/bitcoin/bitcoin/runs/16142913468 – timeouts. Probably unrelated…
  76. MarcoFalke commented at 1:35 pm on August 23, 2023: member

    yeah, the blob is missing.

    0Fetching xcb-proto-1.15.2.tar.xz from https://bitcoincore.org/depends-sources
1  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2                                 Dload  Upload   Total   Spent    Left  Speed
3  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
4  0  222k    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
5curl: (22) The requested URL returned error: 404

    It would be good to switch the mirror URL to something that supports ipv6 and is maintained. Hint: https://github.com/bitcoin/bitcoin/pull/17704

  77. MarcoFalke commented at 12:12 pm on August 25, 2023: member

    Apart from the blob missing, this seems to be an odd issue, because it works outside of podman and inside podman, any other site works as well:

    0# podman run --rm -it ubuntu:mantic  bash -c  'apt update && apt install curl -y && curl --location --fail  --connect-timeout 3 --retry 0  -o /tmp/ab http://xorg.freedesktop.org/archive/individual/proto/xcb-proto-1.15.2.tar.xz ' 
1
2curl: (28) Failed to connect to xorg.freedesktop.org port 80 after 3001 ms: Timeout was reached

    0# podman run --rm -it ubuntu:mantic  bash -c  'apt update && apt install curl -y && curl --location --fail  --connect-timeout 3 --retry 0  -o /tmp/ab https://drahtbot.space/depends_download_fallback/xcb-proto-1.15.2.tar.xz '
1
2100  144k  100  144k    0     0   614k      0 --:--:-- --:--:-- --:--:--  612k

    0# curl --location --fail  --connect-timeout 3 --retry 0  -o /tmp/ab http://xorg.freedesktop.org/archive/individual/proto/xcb-proto-1.15.2.tar.xz
1
2100  144k  100  144k    0     0   118k      0  0:00:01  0:00:01 --:--:--  208k

    See https://serverfault.com/questions/1108095/docker-container-unable-to-connect-to-one-specific-external-ip-yet-host-can


MarcoFalke DrahtBot dergoegge pinheadmz hebasto


fanquake

Labels
Tests

Milestone
26.0

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-07-05 22:12 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me