ci: Fix "macOS native" job

hebasto commented at 1:30 pm on March 10, 2024: member

Homebrew promoted python@3.12 to the default python3. Now, our “macOS native” CI job is facing the following issues:

Installing qt@5 requires re-installing python@3.12:

0==> Fetching dependencies for qt@5: readline, python@3.12 and gettext

Re-installing python@3.12 fails due to symbolic link conflicts on macOS x86_64:

0==> Pouring python@3.12--3.12.2_1.ventura.bottle.tar.gz
1Error: The `brew link` step did not complete successfully

Homebrew’s python@3.12 is marked as externally managed (according to PEP 668), necessitating different approaches for installing Python packages.

This pull request resolves all the issues mentioned above.

ci: Add workaround for Homebrew's python link error

Promoting Homebrew's python@3.12 to the default python3 breaks symbolic
links on macOS x86_64.

This change adds a workaround for that issue.

Also see: https://github.com/actions/runner-images/issues/9471 etc.

ae5f72027f

DrahtBot commented at 1:30 pm on March 10, 2024: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	m3dwards

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

DrahtBot added the label Tests on Mar 10, 2024

m3dwards commented at 2:43 pm on March 11, 2024: contributor

ACK https://github.com/bitcoin/bitcoin/pull/29610/commits/eca024fa97de1500bf2a176cfcfe3177685fab0c

Don’t have a x86 Mac to hand right now so ran mac cross CI job to test.

This works but two alternative approaches to the pip install problem could have been to either use a python virtual environment (venv) or to pass --break-system-packages when pip installing. As this is a throwaway build runner venv is probably overkill.

maflcko commented at 2:58 pm on March 11, 2024: member

venv

I thought about this, too. However, I presume it would require activating the env twice. Once, when creating the container image (pip), and another time, when running the tests in the container.

--break-system-packages

If this causes a smaller diff, I’d say this is preferred.

fanquake commented at 3:09 pm on March 11, 2024: member

It does seem not ideal to have to forcefully install one verison of Python, only to then not be able to use it for anything Python related, and then also refactor the CI to use a different (unclear which?) version of Python.

ci, macos: Use `--break-system-packages` with Homebrew's python

Homebrew's python@3.12 is marked as externally managed (PEP 668),
necessitating different approaches for installing Python packages.

For more details, please refer to https://github.com/orgs/Homebrew/discussions/3404.

acc06bc91f

hebasto force-pushed on Mar 11, 2024

hebasto commented at 8:04 pm on March 11, 2024: member

@m3dwards @maflcko @fanquake

Thank you for your reviews!

--break-system-packages

If this causes a smaller diff, I’d say this is preferred.

Implemented.

It does seem not ideal to have to forcefully install one verison of Python, only to then not be able to use it for anything Python related, and then also refactor the CI to use a different (unclear which?) version of Python.

Reworked, so Homebrew’s python is still using.

in ci/test/00_setup_env_mac_native.sh:12 in acc06bc91f

 6@@ -7,7 +7,9 @@
 7 export LC_ALL=C.UTF-8
 8 
 9 export HOST=x86_64-apple-darwin
10-export PIP_PACKAGES="zmq"
11+# Homebrew's python@3.12 is marked as externally managed (PEP 668).
12+# Therefore, `--break-system-packages` is needed.
13+export PIP_PACKAGES="--break-system-packages zmq"

maflcko commented at 7:41 am on March 12, 2024:

nit: Seems fine to enable it globally in the CI, but this can be done in a follow-up.

m3dwards commented at 10:39 am on March 12, 2024:

Agreed. It’s probably going to start coming up in other jobs as python and base containers get upgraded.

maflcko approved

maflcko commented at 7:41 am on March 12, 2024: member

lgtm

m3dwards commented at 10:46 am on March 12, 2024: contributor

reACK https://github.com/bitcoin/bitcoin/pull/29610/commits/acc06bc91f80ddf4e015dcdf0b984bbdbfcb5ca3 to get the CI passing again.

Contrary to my previous comment, when making a more global change to CI perhaps the more permanent and proper thing would be to use a venv.

maflcko requested review from fanquake on Mar 12, 2024

fanquake commented at 11:43 am on March 12, 2024: member

proper thing would be to use a venv.

Yea, it’s pretty clear this is the direction we should be going, given we are going to start running into this with Python on other distros.

fanquake merged this on Mar 12, 2024

fanquake closed this on Mar 12, 2024

hebasto deleted the branch on Mar 12, 2024

hebasto referenced this in commit bd5672c030 on Mar 12, 2024

hebasto referenced this in commit 2de625d90c on Mar 12, 2024

hebasto referenced this in commit 5e58e05eb3 on Mar 13, 2024

luke-jr referenced this in commit 87c365cce0 on Mar 14, 2024

luke-jr referenced this in commit 1bf4940142 on Mar 14, 2024

luke-jr referenced this in commit 1f594e6b0e on Mar 15, 2024

luke-jr referenced this in commit 71d397f657 on Mar 15, 2024

luke-jr referenced this in commit 545ac7a13f on Mar 18, 2024

luke-jr referenced this in commit 770a504c3e on Mar 18, 2024

fanquake referenced this in commit 603f0368a5 on Mar 21, 2024

fanquake referenced this in commit 05f69b36d1 on Mar 21, 2024

glozow referenced this in commit 324e562399 on Mar 25, 2024

glozow referenced this in commit b53bf22c72 on Mar 25, 2024

fanquake commented at 10:59 am on March 25, 2024: member

Backported to 27.x in #29693 and 26.x in #29719.

ci: Fix “macOS native” job #29610

Code Coverage

Reviews