As the title says, this PR aims to address #26548 by switching the lint CI job from ubuntu's bionic base image which has python 3.6.5 to ubuntu jammy and use pyenv to install the correct python version as dictated in the project's .python-version file.
I have tested this in a VS Code Dev Container to as closely emulate the Cirrus CI Service
Bumps the base image used in the linting task for the cirrus CI service to use jammy instead of bionic
<!--e57a25ab6845829454e8d69fc972939a-->
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
<!--021abf342d371248e50ceaed478a90ca-->
See the guideline for information on the review process. A summary of reviews will appear here.
<!--174a7506f384e20aa4161008e828411d-->
Reviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.
16 | - ${CI_RETRY_EXE} apt-get install -y --reinstall git 17 | -) 18 | +${CI_RETRY_EXE} apt-get install -y curl git gawk jq 19 | +${CI_RETRY_EXE} apt-get install -y make build-essential libssl-dev zlib1g-dev \ 20 | +libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm \ 21 | +libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev
Why is all of this needed? Is pyenv compiling itself and then the python version?
Yes just noticed, gonna inspect this rq to see why this happens. As for the deps, pyenv requires this according to the docs
Pyenv seems to exit prematurely and somehow I stupidly didn't pick this up.
Ah interesting. I wonder if this even works then. Initially, I tried using https://launchpad.net/%7Edeadsnakes/+archive/ubuntu/ppa , however it notes " for jammy, older python versions requre libssl<3 so they are not currently built".
In my dev container, it does indeed build the python version without issue. I of course used the ubuntu:jammy image
Why is all of this needed? Is pyenv compiling itself and then the python version?
If this was Github Actions I would have cached the build of Python 3.6.15, but I don't think Cirrus supports this
looks like the CI is aborting green with empty output?
@MarcoFalke I have addressed the issues in the new commit, and now it follows through with the rest of the testing. Except that the test scripts in the repo seem to error out with git (See attached) <img width="1074" alt="image" src="https://user-images.githubusercontent.com/35210888/203976009-19ec849e-209e-40de-9783-4f264d8968e4.png"> I am not going to try to address this issue as it's got to do with the linting scripts themselves which are interacting with Git, and Git throwing an issue. And given that I haven't touched any of that I'll leave it be unless I am missing something which in that case I'll try to remedy the issue that this change causes
Looks like it does OOM
Just took a look at Deadsnake's PPA. They claim they don't build Python 3.6.15 because it's already built by ubuntu's repos. Yet the files list contains a tar ball for python 3.6.15 for focal fossa release. I'd say it's possible to untar the tar ball into pyenv's version if it all works smoothly. As for ssl, I am not sure, how come does the test ci job use jammy despite the repo's python version being set to python 3.6.15?
I'd guess that maybe python 3.6 supports building on Jammy by now? You may be able to fix the OOM by bumping the memory or by using clang. Though, I think we'll also want to cache the python binary, otherwise too much time is spent on compiling it for every push.
I'd guess that maybe python 3.6 supports building on Jammy by now? You may be able to fix the OOM by bumping the memory or by using
clang. Though, I think we'll also want to cache the python binary, otherwise too much time is spent on compiling it for every push.
Python 3.6.15 with pyenv definitely compiles (at least for me which was in a dev container which is pretty close to a real ubuntu jammy based machine). The issue is caching the builds. I'll try bump the memory and see if that works, I don't think using Clang is possible as it doesn't seem that pyenv supports it, and using a custom script to download python and compile it requires maintenance that I can't assure. Using pyenv makes maintaining way easier and simple.
I have discovered that it is indeed possible to cache builds. I have added more commits that try to implement this as automagically as possible
@MarcoFalke The fixes have been implemented. I just don't understand why lint fails despite saying that no issues were found
Edit: I didn't realize it was shell check erroring out. Gonna try to fix this
It actually works. Finally. @MarcoFalke Sorry for the excessive amount of pings, but feel free to merge this. Lmk if any squashing is necessary
Lmk if any squashing is necessary
Yes, it is necessary.
- Install build environment dependencies for pyenv (https://github.com/pyenv/pyenv#install-python-build-dependencies)
- Curl https://pyenv.run installer and pipe it to bash
- Add pyenv's bin directory to path
- Run pyenv's init script (https://github.com/pyenv/pyenv#set-up-your-shell-environment-for-pyenv)s
- Use pyenv's CLI to install python 3.6.15
- Use python cache in Cirrus
- Use intermediary directory for interacting with Cirrus's caching functions
- Assign more resources to the cirrus task to compile python 3.6.15
add a tr() descriptor example to the help deriveaddresses examples
Tracks time-machine bump in 298389e3b51edf561a2b636df0948a232167693a.
This was already suggested in the troubleshooting section, but recommending it upfront would prevent the issue in the first place and speed up builds.
They are mentioned in the figure above, but having them in the table makes it easier to (apt) install everything required.
Co-Authored-By: Stephan Oeste <emzy@emzy.de>
Also drop unused links.
It was introduced in c1ae726a13ecfa5e7e9fdc3030a8110b8bb263f8, and it
has no longer been used since 1dd8cbfbc6318308b5e75023568f818ef6a9c7e4.
This change allows to use the `test-{security,symbol}-check.py` scripts
when building out of source tree with no need to link scripts into the
build directory.
Th build system targets run those scripts from the top source directory.
This:
- Removes previously created env variables
- Adds echo for the copy to cache process
- Uses cp instead of any mv just to be safe
- Removes any comments that are not relevant
Lmk if any squashing is necessary
Yes, it is necessary.
Now that I have done a destructive action, I start to believe that I can't have squashed those commits in the first place...
Now that I've slept on it, I think it might convene me to just generate a patch file for the files I have modified, and open up a new PR. As I forgot merging instead of rebasing does this kind of stuff. I'll wait for further comment from you guys
Yea no I'll do that