To run the USDT functional tests, the ASan task currently requires the container host to run the Ubuntu Lunar Linux kernel (or later). Cirrus CI is the only provider that allows to spin up full VMs with Ubuntu Lunar, however they will start to charge for all tasks (See slightly related discussion in #28098).
Since it is cheaper and recommended by Cirrus CI to just run a persistent worker, do that.
Also, using a persistent worker allows to make use of the docker image cache.
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
See the guideline for information on the review process.
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.
Reviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.
ACK, looks like the tests are run and pass.
0interface_usdt_coinselection.py | ✓ Passed | 17 s
1interface_usdt_mempool.py | ✓ Passed | 37 s
2interface_usdt_net.py | ✓ Passed | 14 s
3interface_usdt_utxocache.py | ✓ Passed | 35 s
4interface_usdt_validation.py | ✓ Passed | 8 s
sed command, and add documentation for why it is needed)
concept ACK
I read the linked cirrus guide about persistent workers, everything makes sense with these changes. Two questions:
previous releases task need persistent worker?
Otherwise the task will throw in skip_if_no_python_bcc.
Also, adjust CI_CONTAINER_CAP for all needed permissions.
Why does the previous releases task need persistent worker?
I did that to detect silent merge conflicts without using Cirrus CI resources. (A machine is running this task in a loop on all pull requests, constantly rebased on master). Unrelated: See https://github.com/MarcoFalke/DrahtBot/blob/main/rerun_ci/src/main.rs#L30 for the details on how to re-run.
Is the persistent worker hosted by you? Are there any documented details about it?
Currently it is running on someone’s leftover machine, but someone will spin up a proper cluster soon (either before merge or shortly after). I’ve pushed docs on the requirements of the workers. Let me know if I should document anything else. The other steps are taken from the Cirrus CI docs and are something like:
curl -L -o cirrus https://github.com/cirruslabs/cirrus-cli/releases/latest/download/cirrus-$(uname | tr '[:upper:]' '[:lower:]')-amd64 && mv cirrus /usr/local/bin/cirrus && chmod +x /usr/local/bin/cirrusscreencirrus worker run --labels type=lunar --token todo_get_the_correct_token
Concept ACK on moving CI tasks outside from Cirrus CI as a response to their new limits.
Historically, the Bitcoin Core project started to use self-hosted CI in April 2021:
For testing purposes, a single task will be transformed to run on the DrahtBot infrastructure. If all goes well, the other tasks can be moved, too.
At the same time, there were some usage of the bitcoinbuilds.org infrastructure, which was completely independent of any CI provider. During that time, Cirrus CI seemed to be a suitable choice, and we stopped using bitcoinbuilds.org sometime around the end of 2021 or the beginning of 2022.
Also, using a persistent worker allows to make use of the docker image cache.
This feature is really great.
Since it is cheaper…
Tbh, I’d feel much more confident and comfortable knowing that some sponsors publicly pledged to cover all related costs.
272@@ -257,22 +273,17 @@ task:
273
274 task:
275 name: '[ASan + LSan + UBSan + integer, no depends, USDT] [lunar]'
276+ enable_bpfcc_script:
277+ # In the image build step, no external environment variables are available,
278+ # so any settings will need to be written to the settings env file:
279+ - sed -i "s|\${CIRRUS_CI}|true|g" ./ci/test/00_setup_env_native_asan.sh
Might it be more straightforward to define BPFCC_PACKAGE in ci\test_imagefile?
Running FILE_ENV="./ci/test/00_setup_env_native_asan.sh" ./ci/test_run_all.sh locally skips interface_usdt_*.py tests.
--cap-add SYS_PTRACE is in the docker command and USDT tracing is set by configure, what’s missing?
what’s missing?
These are the checks to determine if we want to skip the USDT test: https://github.com/bitcoin/bitcoin/blob/2fa60f0b683cefd7956273986dafe3bde00c98fd/test/functional/interface_usdt_net.py#L87-L91
I haven’t looked into this in detail. With which user are the test run in docker locally? Is --cap-add SYS_PTRACE enough?
Is
--cap-add SYS_PTRACEenough?
No it is not. You can see the second commit of this pull request to see what is needed: fa474397b5d4235efdfc5a5ddce2d637234548a7
sed
🥲
When I set CIRRUS_CI to true locally I end up failing installing the kernel headers. I see this error running this branch locally on both Ubuntu and macOS hosts. But the “generic” version there is coming from inside the container right (uname --kernel-release)? So I’m confused why this works on actual CI but not locally.
0[#8](/bitcoin-bitcoin/8/) 7.217 E: Unable to locate package linux-headers-5.15.0-78-generic
1[#8](/bitcoin-bitcoin/8/) 7.217 E: Couldn't find any package by glob 'linux-headers-5.15.0-78-generic'
2[#8](/bitcoin-bitcoin/8/) 7.217 E: Couldn't find any package by regex 'linux-headers-5.15.0-78-generic'
When you want to run the exact same env like the Cirrus worker does, you will also need the exact same machine:
A “machine running the Linux kernel shipped with Ubuntu Lunar 23.04.”
Thus, other Linuxes and macOS are not supported, unless it happens to work for some reason.
You can check the version locally with cat /etc/os-release.
However, be warned that when you set CIRRUS_CI to true locally, you’ll pass --privileged -v /sys/kernel:/sys/kernel:rw to docker, which allows the CI system more privileges.
Tbh, I’d feel much more confident and comfortable knowing that some sponsors publicly pledged to cover all related costs.
Ok, I’ll try to encourage the sponsor to publicly identify themselves, but I wonder what you worry about? If you are worried about the sponsor walking away, I can assure you that there are currently three (3) willing sponsors in line. Also, while it isn’t free, running Linux is extremely cheap compared to Windows/macOS.
If you have any alternative ideas, it would be good to share them. The only thing I could come up with would be to use GitHub Actions CI ubuntu-jammy and do a upgrade to Ubuntu Lunar on every CI run. You’d have to increase the task runtime to 3 hours, but that seems unlikely to work and a worse outcome either way, so I am not going to pursue that.
Tbh, I’d feel much more confident and comfortable knowing that some sponsors publicly pledged to cover all related costs.
Ok, I’ll try to encourage the sponsor to publicly identify themselves
Sorry for being misunderstood. I don’t want to force current sponsors to any additional actions.
… but I wonder what you worry about?
Sustainability.
Also, while it isn’t free, running Linux is extremely cheap compared to Windows/macOS.
I don’t think so. In June, total Linux tasks cost was the biggest one followed by Windows.
The only thing I could come up with would be to use GitHub Actions CI ubuntu-jammy and do a upgrade to Ubuntu Lunar on every CI run. You’d have to increase the task runtime to 3 hours, but that seems unlikely to work and a worse outcome either way, so I am not going to pursue that.
This should definitely be avoided.
Also, while it isn’t free, running Linux is extremely cheap compared to Windows/macOS.
I don’t think so. In June, total Linux tasks cost was the biggest one followed by Windows.
You are quoting Linux costs that Cirrus CI “sells”. The goal of this pull request is to use Linux “sold” by someone else. (There are many professional companies that offer Linux boxes. If you are unsure, you can use your favourite search engine to get typical competitive pricing on Linux boxes.)
Also, while it isn’t free, running Linux is extremely cheap compared to Windows/macOS.
I don’t think so. In June, total Linux tasks cost was the biggest one followed by Windows.
You are quoting Linux costs that Cirrus CI “sells”.
Yes. We have to cope with new Cirrus’s offers.
The goal of this pull request is to use Linux “sold” by someone else.
Using someone’s else resource equals to not using Cirrus provided resources.
(There are many professional companies that offer Linux boxes. If you are unsure, you can use your favourite search engine to get typical competitive pricing on Linux boxes.)
Thank you. I am aware of it :)