doc: clarify Cirrus self-hosted workers setup

Sjors commented at 2:05 pm on June 20, 2024: member

Taken from #29274 (except for two paragraphs that require the other commits in that PR).

DrahtBot commented at 2:05 pm on June 20, 2024: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	maflcko, tdb3

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#29432 (Stratum v2 Template Provider (take 3) by Sjors)
#29274 (Support self-hosted Cirrus workers on forks by Sjors)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

DrahtBot added the label Tests on Jun 20, 2024

Sjors renamed this:
~~ci: clarify Cirrus self-hosted workers setup~~
doc: clarify Cirrus self-hosted workers setup
on Jun 20, 2024

in .cirrus.yml:27 in 9f4255ac57 outdated

23+#
24+# CI jobs for the latter configuration can be run on x86_64 hardware
25+# by installing qemu-user-static, which may work out of the box with
26+# podman or docker.
27+#
28+# The above machine types are matched to each job by their label. Refer to the

maflcko commented at 2:26 pm on June 20, 2024:

0# The above machine types are matched to each task by their label. Refer to the

Cirrus calls it task. In yaml it is task:

in .cirrus.yml:47 in 9f4255ac57 outdated

56@@ -40,11 +57,6 @@ env:  # Global defaults
57 #   RESTART_CI_DOCKER_BEFORE_RUN=1 screen cirrus worker run --labels type=todo_fill_in_type --token todo_fill_in_token
58 #   ```
59 #
60-# The following specific types should exist, with the following requirements:
61-# - small: For an x86_64 machine, recommended to have 2 CPUs and 8 GB of memory.
62-# - medium: For an x86_64 machine, recommended to have 4 CPUs and 16 GB of memory.
63-# - arm64: For an aarch64 machine, recommended to have 2 CPUs and 8 GB of memory.
64-

maflcko commented at 2:30 pm on June 20, 2024:

nit: empty newline should stay here, because the section ends. you’ll have to remove the # above instead.

maflcko commented at 2:31 pm on June 20, 2024: member

ACK 9f4255ac57929de985425f26ad904a12176a0e85 🌂

Signature:

0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
2trusted comment: ACK 9f4255ac57929de985425f26ad904a12176a0e85 🌂
3UUSQcgPjO07LFUTZUP8JVqqHdsEpkqLnB5MQjPPzt6EDDztzLcIRrJIR/pQPe6hCTTzAK6Yz/mVfi1zCB2FsAA==

in .cirrus.yml:25 in 9f4255ac57 outdated

21+# - medium: For an x86_64 machine, recommended to have 4 CPUs and 16 GB of memory.
22+# - arm64: For an aarch64 machine, recommended to have 2 CPUs and 8 GB of memory.
23+#
24+# CI jobs for the latter configuration can be run on x86_64 hardware
25+# by installing qemu-user-static, which may work out of the box with
26+# podman or docker.

fanquake commented at 2:33 pm on June 20, 2024:

which may work out of the box with podman or docker.

Please don’t add vaugeries like this to docs. If you have a specific issue that you’re running into, please open an issue, otherwise things should be assumed to be working (otherwise they should be fixed, rather than adding docs claiming they might not work).

Sjors commented at 4:31 pm on June 20, 2024:

I don’t know if it works out of the box for every distro out there. It does for Ubuntu 24.04 when used with podman, at least after a reboot. I haven’t tested with docker nor on any other distro. The documentation for https://github.com/multiarch/qemu-user-static says you need to run sudo podman run --rm --privileged multiarch/qemu-user-static --reset -p yes first, but this is not (no longer?) true.

What I’m trying to convey with “may work” is: it might work with just sudo apt install qemu-user-static, even though the projects own README says it doesn’t. I can change it to “works”, but that might be wrong - hard to know without trying every permutation out there.

Sjors commented at 4:34 pm on June 20, 2024:

This uncertainty is also part of the reason #29274 adds NO_ARM as an option.

fanquake commented at 5:55 pm on June 20, 2024:

I don’t know if it works out of the box for every distro out there

That’s fine. Just assume it does until we know otherwise.

Sjors commented at 6:59 am on June 21, 2024:

Ok, changed to “works” and added a link that explains what this does.

Sjors force-pushed on Jun 20, 2024

Sjors commented at 4:32 pm on June 20, 2024: member

Fixed @maflcko’s nits.

DrahtBot added the label CI failed on Jun 20, 2024

in .cirrus.yml:16 in 66b9740011 outdated

12+# multiple users to run tasks in parallel. No sudo permission is required.
13+#
14 # https://cirrus-ci.org/guide/persistent-workers/
15 #
16-# It is possible to select a specific persistent worker by label. Refer to the
17+# Generally, a persistent worker must run Ubuntu 23.04+ or Debian 12+.

tdb3 commented at 2:55 am on June 21, 2024:

These specs will become outdated over time (e.g. Ubuntu 23.04 which is already obsolete). Maybe instead it would be better to point the reader to doc/dependencies.md (or doc/build-unix.md, etc.)? dependencies.md doesn’t currently specify distro versions, but would it be the case that dependency versions are really what matters, and distro version is a downstream consequence of that?

Perhaps this is a little less easy for the reader but helps prevent this file from becoming stale. Something like:

0- # Generally, a persistent worker must run Ubuntu 23.04+ or Debian 12+.
1+ # Generally, a persistent worker must run a distribution that supports the minimum dependencies in `doc/dependencies.md`

maflcko commented at 6:00 am on June 21, 2024:

This is just a moved sentence and still correct. 23.04+ is required, and should work, if someone manages to install it.

maflcko commented at 6:01 am on June 21, 2024:

Also, your suggestion is incorrect. doc/dependencies.md has nothing to do with a CI system runner. In fact, you can probably use any distro, as long as you can install podman4.1+ or docker on it.

tdb3 commented at 11:01 am on June 21, 2024:

Good point about the +. The thought was “Is there a way to describe what the needs are while minimizing the future changes needed to this file?” If we think pointing elsewhere (e.g. to a place that describes the needs, but is being updated for additional reasons beyond this file) is overkill, then we don’t need to change this line.

maflcko commented at 11:26 am on June 21, 2024:

If we think pointing elsewhere (e.g. to a place that describes the needs, but is being updated for additional reasons beyond this file) is overkill, then we don’t need to change this line.

The file is self-contained regarding this line and there is no need to point elsewhere. “podman4.1+” is equivalent to “vanilla Ubuntu 23.04+” (forever), so there is no need to update it in the future.

in .cirrus.yml:21 in 66b9740011 outdated

17+# Generally, a persistent worker must run Ubuntu 23.04+ or Debian 12+.
18+#
19+# The following specific types should exist, with the following requirements:
20+# - small: For an x86_64 machine, recommended to have 2 CPUs and 8 GB of memory.
21+# - medium: For an x86_64 machine, recommended to have 4 CPUs and 16 GB of memory.
22+# - arm64: For an aarch64 machine, recommended to have 2 CPUs and 8 GB of memory.

tdb3 commented at 2:58 am on June 21, 2024:

This one I’m a little torn about. Ideally, these machine specs would change less frequently than Linux distro versions (a side effect of keeping node requirements modest/reasonable), so maybe it makes sense to leave these specifics in this file.

maflcko commented at 6:02 am on June 21, 2024:

Not sure what you mean. This is just a moved section to explain the three possible and required labels.

tdb3 commented at 11:07 am on June 21, 2024:

Initially, the thought was similar to above, if specs change, do we want to come back to this file and adjust them here, or would it be better to have a place that we can point to for minimum/recommended specs more generally or globally (since building and running tests happen for development in general rather than solely in CI)? If these specs won’t change very frequently, then it’s not a big risk to keep them here and update this file over time.

maflcko commented at 11:29 am on June 21, 2024:

This file .cirrus.yml is self-contained and only concerns Cirrus CI. It has no meaning for the outside CI system. The labels are used in this config file only, they are only required here, and they are explained here.

tdb3 commented at 3:01 am on June 21, 2024: contributor

Approach ACK. This adds more clarity to CI config, much appreciated. Left a comment and thought.

maflcko commented at 6:02 am on June 21, 2024: member

ACK 66b9740011222002ae9026e32e03a117628daae1

DrahtBot requested review from tdb3 on Jun 21, 2024

ci: clarify Cirrus self-hosted workers setup c67f215ea5

Sjors force-pushed on Jun 21, 2024

maflcko commented at 7:04 am on June 21, 2024: member

ACK c67f215ea5fd57cd05e5346b8cd292dd879303ff

DrahtBot removed the label CI failed on Jun 21, 2024

tdb3 approved

tdb3 commented at 12:09 pm on June 21, 2024: contributor

ACK c67f215ea5fd57cd05e5346b8cd292dd879303ff

fanquake merged this on Jun 21, 2024

fanquake closed this on Jun 21, 2024

Sjors deleted the branch on Jun 21, 2024

Mtiisf125 commented at 7:33 pm on June 24, 2024: none

Mri>..ta**>_isf

doc: clarify Cirrus self-hosted workers setup #30314

Code Coverage

Reviews

Conflicts