ci: Run self-hosted ci #21619

1. 
   MarcoFalke commented at 6:33 pm on April 6, 2021: member

   Due to our heavy use of the Cirrus CI community cluster, some tasks may take a long time to get scheduled. While it is possible to use “Compute Credits” to get immediately scheduled on the cluster, I couldn’t find a sponsor that’d be willing to cover the total cost, if all tasks were paid for with credits.

   However, it is also possible to bring our own runners to Cirrus CI.

   For testing purposes, a single task will be transformed to run on the DrahtBot infrastructure. If all goes well, the other tasks can be moved, too.

2. DrahtBot added the label Tests on Apr 6, 2021
3. MarcoFalke force-pushed on Apr 6, 2021
4. 
   jarolrod commented at 11:28 pm on April 6, 2021: member

   Concept ACK

   DrahtBot strong 🦾

5. 
   DrahtBot commented at 11:53 pm on April 6, 2021: member

   The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

   Conflicts

   Reviewers, this pull request conflicts with the following ones:

   • #21551 ([PoC] ci: Add Windows task to Cirrus CI by hebasto)

   If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

6. MarcoFalke force-pushed on Apr 9, 2021
7. in .cirrus.yml:15 in fa4d30c464 outdated

   11+
   12+cirrus_ephemeral_worker_template_env: &CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
   13+  DANGER_RUN_CI_ON_HOST: "1"  # Containers willl be discarded after the run, so there is no risk that the ci scripts modify the system
   14+
   15+persistent_worker_template_env: &PERSISTENT_WORKER_TEMPLATE_ENV
   16+  CI_RESTART_DOCKER_BEFORE_RUN: "1"
   

   ---

   ---

   

   hebasto commented at 12:01 pm on April 9, 2021:

   I’d leave CI_-prefixed names for CI :) Maybe just RESTART_DOCKER_BEFORE_RUN?

   ---

   

   MarcoFalke commented at 2:07 pm on April 9, 2021:

   Renamed to RESTART_CI_DOCKER_BEFORE_RUN. I think ci is fine in the name as long as it is not a prefix
8. in .cirrus.yml:12 in fa4d30c464 outdated

    8   CCACHE_SIZE: "200M"
    9   CCACHE_DIR: "/tmp/ccache_dir"
   10+  CCACHE_NOHASHDIR: "1"  # Debug info might contain a stale path if the build dir changes, but this is fine
   11+
   12+cirrus_ephemeral_worker_template_env: &CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
   13+  DANGER_RUN_CI_ON_HOST: "1"  # Containers willl be discarded after the run, so there is no risk that the ci scripts modify the system
   

   ---

   ---

   

   hebasto commented at 12:02 pm on April 9, 2021:

   typo:

   0  DANGER_RUN_CI_ON_HOST: "1"  # Containers will be discarded after the run, so there is no risk that the ci scripts modify the system
   

   ---

   

   MarcoFalke commented at 2:07 pm on April 9, 2021:

   thanks, fixed
9. 
   hebasto commented at 12:03 pm on April 9, 2021: member

   Concept ACK.

   Commit “ci: Restart docker before run”. Why we need to restart docker? Isn’t it a CI’s responsibility to create a new container for each task?

10. ci: [refactor] Create setting for ephemeral config in .cirrus.yml

    This allows easier switching between self-hosted runners and the
    community cluster. Also, named variables can be documented better.

    fad4f48e07
11. ci: Restart docker before run

    Also, add setting for persistent worker in .cirrus.yml

    fa278412a0
12. ci: Make cirrus cache folders relative to cirrus base dir fa52a40f0e
13. ci: Run self-hosted ci fa41a91735
14. MarcoFalke force-pushed on Apr 9, 2021
15. 
    MarcoFalke commented at 2:09 pm on April 9, 2021: member

    Why we need to restart docker? Isn’t it a CI’s responsibility to create a new container for each task?

    This can simplify the local workflow when ci tasks are executed repeatedly. Also, it is needed for persistent runners, which are not ephemeral. See also https://cirrus-ci.org/guide/persistent-workers/#what-is-a-persistent-worker

16. hebasto approved
17. 
    hebasto commented at 1:04 pm on April 10, 2021: member

    ACK fa41a917356a7f5e0f3286b227ced7f2e6797e67, I have reviewed the code and it looks OK.

    CI log looks OK.

18. 
    hebasto commented at 1:06 pm on April 10, 2021: member

    Unrelated to this PR, but why is depends_release cache populated after CI run?

    

19. MarcoFalke merged this on Apr 11, 2021
20. MarcoFalke closed this on Apr 11, 2021

    ---

21. MarcoFalke deleted the branch on Apr 11, 2021
22. 
    MarcoFalke commented at 11:57 am on April 11, 2021: member

    Unrelated …

    Fixed in #21653

23. sidhujag referenced this in commit cec8ee96c6 on Apr 11, 2021
24. 
    RonSherfey commented at 3:27 pm on April 11, 2021: none

    reviewed changes
25. 
    78051301012 commented at 9:09 pm on April 11, 2021: none

    .
26. DrahtBot locked this on Aug 16, 2022

Contributors
MarcoFalke jarolrod DrahtBot hebasto RonSherfey 78051301012

Labels
Tests