cmake: Improve compatibility with Python version managers #31421

hebasto commented at 5:11 PM on December 4, 2024: member

This PR resolves the issue highlighted in #31411:

Here's another case where CMake just picks some other Python...

The fix leverages two hints for the CMake FindPython3 module:

Python3_FIND_FRAMEWORK is set to LAST. This ensures that Unix-style package components are preferred over frameworks on macOS. As a side effect, the FindPython3 module reports a shim or symlink (e.g., from pyenv) rather than the underlying framework's binary. The module's output aligns with the result of the which command.
Python3_FIND_UNVERSIONED_NAMES is set to FIRST. This supports scenarios where tools like pyenv—which use shims—have multiple Python versions installed.

Here are examples of output on my macOS 15.1.1 (Intel) with installed Homebrew's Python 3.13.0:

without any Python version manager:

% which -a python3                
/usr/local/bin/python3
/usr/bin/python3
% cmake -B build
<snip>
-- Found Python3: /usr/local/bin/python3 (found suitable version "3.13.0", minimum required is "3.10") found components: Interpreter
<snip>

using pyenv:

% pyenv versions       
  system
* 3.10.14 (set by /Users/hebasto/dev/bitcoin/.python-version)
  3.12.8
  3.13.1
% which -a python3
/Users/hebasto/.pyenv/shims/python3
/usr/local/bin/python3
/usr/bin/python3
% cmake -B build
<snip>
-- Found Python3: /Users/hebasto/.pyenv/shims/python3 (found suitable version "3.10.14", minimum required is "3.10") found components: Interpreter
<snip>

Both variables, Python3_FIND_FRAMEWORK and Python3_FIND_UNVERSIONED_NAMES, can still be overridden by the user via the command line if needed.

hebasto added the label Build system on Dec 4, 2024

hebasto force-pushed on Dec 4, 2024

willcl-ark commented at 1:34 PM on December 5, 2024: member

Concept ACK

I have tested this using a uv-style venv to ensure nothing breaks, but not pyenv, where this is expected to make a difference:

will@ubuntu in …/src/core/bitcoin on  241204-python [$] via △ v3.30.5 : 🐍 (core)
₿ command -v python3
/home/will/src/core/.venv/bin/python3

will@ubuntu in …/src/core/bitcoin on  241204-python [$] via △ v3.30.5 : 🐍 (core)
₿ cmake -B build --fresh | rg python
-- Found Python3: /home/will/src/core/.venv/bin/python3 (found suitable version "3.10.14", minimum required is "3.10") found components: Interpreter

Trying to target symbol checks, which launches with the configured python also shows the correct binary in use:

will@ubuntu in …/src/core/bitcoin on  241204-python [$] via △ v3.30.5 : 🐍 (core) took 10s
₿ cmake --build build --target test-security-check
[1/1] cd /home/will/src/core/bitcoin/build && /home/linuxbrew/.linuxbrew/Cellar/cmake/3.30.5/bin/cmake -E env CXX=/usr/lib/ccache/clang...LAGS= /home/will/src/core/.venv/bin/python3 /home/will/src/core/bitcoin/contrib/devtools/test-symbol-check.py TestSymbolChecks.test_ELF

As a side effect, the FindPython3 module reports a shim or symlink (e.g., from pyenv) rather than the underlying framework's binary. The module's output aligns with the result of the which command

Just curious, how did you find this out? It's not clear to me from the docs this would happen.

DrahtBot commented at 1:34 PM on December 5, 2024: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31421.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	theuni, willcl-ark

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#31669 (cmake: Introduce WITH_PYTHON build option by hebasto)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

hebasto commented at 1:43 PM on December 5, 2024: member

As a side effect, the FindPython3 module reports a shim or symlink (e.g., from pyenv) rather than the underlying framework's binary. The module's output aligns with the result of the which command

Just curious, how did you find this out? It's not clear to me from the docs this would happen.

From Python3_FIND_FRAMEWORK docs:

On macOS the Python3_FIND_FRAMEWORK variable determine the order of preference between Apple-style and unix-style package components.

You could observe the difference by configuring with -DPython3_FIND_FRAMEWORK=FIRST vs -DPython3_FIND_FRAMEWORK=LAST. The real Python binary can be verified with sys.executable.

fanquake added this to the milestone 29.0 on Jan 6, 2025

in CMakeLists.txt:538 in 490dbfa2e8 outdated

 533 | @@ -534,6 +534,12 @@ if(WERROR)
 534 |    unset(werror_flag)
 535 |  endif()
 536 |  
 537 | +# Prefer Unix-style package components over frameworks on macOS.
 538 | +# This improves compatibility with Python version managers.

theuni commented at 7:07 PM on February 7, 2025:

I think these should be tested for existence first (in case set by the user), and set to INTERNAL ?

hebasto commented at 8:36 PM on February 7, 2025:

From CMake docs:

Since cache entries are meant to provide user-settable values this does not overwrite existing cache entries by default.

It overrides only when FORCE is provided or the type is INTERNAL, which implies FORCE.

theuni commented at 9:31 PM on February 7, 2025:

Right, my comment was about trying to keep them out of the gui menus.

hebasto commented at 9:45 PM on February 7, 2025:

Ah, I see. In that case, wouldn't me be easier and consistent with the other code to mark these both variables as advanced?

theuni commented at 10:01 PM on February 7, 2025:

Whoops, of course! That's what I meant to suggest, not Internal. Advanced please :)

hebasto commented at 6:50 AM on February 8, 2025:

Thanks! Amended.

cmake: Improve compatibility with Python version managers dead908654

hebasto force-pushed on Feb 8, 2025

theuni approved

theuni commented at 5:50 PM on February 12, 2025: member

No opinion on the python selection changes themselves, but code-review ACK dead9086543671b07e6ce041821e4d2a6627075b

DrahtBot requested review from willcl-ark on Feb 12, 2025

willcl-ark approved

willcl-ark commented at 8:26 PM on February 13, 2025: member

ACK dead9086543671b07e6ce041821e4d2a6627075b

Tested with uv and pyenv on linux and macOS.

fanquake merged this on Feb 14, 2025

fanquake closed this on Feb 14, 2025

hebasto deleted the branch on Feb 14, 2025

Sjors commented at 11:19 AM on February 14, 2025: member

Tested dead9086543671b07e6ce041821e4d2a6627075b on arm macOS 15.3 that before this PR cmake would find the Python version installed by Homebrew (3.13.1) rather than the one I set with PyEnv (3.10.14).

The system version at /usr/bin/python3 is 3.9.6.

With this change it still does that and ignores PyEnv version:

-- Found Python3: /opt/homebrew/bin/python3 (found suitable version "3.13.1", minimum required is "3.10") found components: Interpreter

 which -a python3
/opt/homebrew/bin//python3
/Users/sjors/.pyenv/shims/python3
/opt/homebrew/bin/python3
/usr/bin/python3

Sjors commented at 11:23 AM on February 14, 2025: member

Same when testing on master @ 7bbd761e816b713ac88c87f8b0045ea2f958ecbf after this was merged. I can't uninstall python3 from Homebrew because it's a dependency for a bunch of things.

Full cmake -B build log: https://gist.github.com/Sjors/becdc73b4dadcfa26121f37a11665e59

It's not causing any practical issues for me just yet though.

Sjors commented at 11:40 AM on February 14, 2025: member

brew unlink python3 "fixes" it for me. We could mention it as a build hint if someone runs into a problem. Not sure how likely that is to be problem though.

fanquake commented at 11:45 AM on February 14, 2025: member

It's not clear what the problem is here. Looking at #31421 (comment), it seems like the correct Python is being picked, because the brew installed python is python3 on your system. If you want the pyenv Python to be picked, you should setup pyenv such that it's python is python3 on your system.

hebasto commented at 11:46 AM on February 14, 2025: member

The initial issue was that CMake finds the Python version other than the one reported by which python3 command.

 which -a python3
/opt/homebrew/bin//python3
/Users/sjors/.pyenv/shims/python3
/opt/homebrew/bin/python3
/usr/bin/python3

Your PATH looks a bit messy :) (/opt/homebrew/bin//python3 and /opt/homebrew/bin/python3)

willcl-ark commented at 12:36 PM on February 14, 2025: member

If you want the pyenv Python to be picked, you should setup pyenv such that it's python is python3 on your system.

Or said another way, rather than fixing by unlinking brew's python, if you want pyenv python to be picked up first, the usual move is to ensure the $PYENV_ROOT/bin component appears earlier than /opt/homebrew/bin/ on your $PATH.

This is explained further in the shell setup and version selection sections of the pyenv docs.

Sjors commented at 2:07 PM on February 14, 2025: member

I had export PATH=/opt/homebrew/bin/:$PATH in my ~/.zshrc probably to use gnu versions of various tools. But that's not the right way.

Once I removed that, and restore brew link python3, the path is as follows:

which -a python3
/Users/sjors/.pyenv/shims/python3
/opt/homebrew/bin/python3
/usr/bin/python3

And cmake finds 3.10.14.