depends: fix cmake package discovery for depends-built packages #35068

When building cmake-based depends packages (e.g. native_libmultiprocess), cmake needs to find dependencies that were previously built and installed into the depends prefix. Since 603fd6a2e70 ("depends: add MULTIPROCESS depends option"), funcs.mk has set CMAKE_MODULE_PATH in the environment to point cmake at the depends prefix. This never worked for two reasons:

1. CMake does not read CMAKE_MODULE_PATH from the environment; it only honors it when set via -D flags or set() in CMakeLists.

2. CMAKE_MODULE_PATH only affects module-mode find_package() (Find*.cmake files), but CapnProto and most modern cmake packages use config mode (NO_MODULE / *Config.cmake files), which ignores CMAKE_MODULE_PATH entirely and instead uses CMAKE_PREFIX_PATH.

CMake does read CMAKE_PREFIX_PATH from the environment, and it is the correct variable for config-mode package discovery. Replace the broken CMAKE_MODULE_PATH assignment with CMAKE_PREFIX_PATH prepended to any existing value (mirroring how PATH is handled), pointing at the type-appropriate depends prefix.

This fixes a build failure in native_libmultiprocess where cmake would find the system (nix) CapnProto 1.2.0 instead of the depends-built CapnProto 1.4.0, resulting in a compile-time version mismatch error: #error "Version mismatch between generated code and library headers."

<!--e57a25ab6845829454e8d69fc972939a-->

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

<!--021abf342d371248e50ceaed478a90ca-->

Reviews

See the guideline for information on the review process. A summary of reviews will appear here.

<!--5faf32d7da4f0f540f40219e4f7537a3-->

Why not explicitly set CMAKE_PREFIX_PATH only where it's needed?

For example:https://github.com/bitcoin/bitcoin/blob/976985eccd546a95e38973b854ccc6589e8afc74/depends/packages/qt.mk#L172

This fixes a build failure in native_libmultiprocess where cmake would find the system (nix) CapnProto 1.2.0 instead of the depends-built CapnProto 1.4.0, resulting in a compile-time version mismatch error: #error "Version mismatch between generated code and library headers."

Why doesn't setting CMAKE_INSTALL_PREFIX help? https://github.com/bitcoin/bitcoin/blob/976985eccd546a95e38973b854ccc6589e8afc74/depends/funcs.mk#L221

According to the documentation, it gets included in the search paths by default.

Thanks for the review! Re: the suggestions to use -DCMAKE_PREFIX_PATH or -DCMAKE_INSTALL_PREFIX, I do not think CMAKE_INSTALL_PREFIX is the right way to control the search paths CMake uses to find dependencies, because its primary purpose is to set the destination path where the package will be installed. It could make sense to set that variable for other reasons, but it would not address the underlying issue here. (For example, the bug would reappear if the depends system ever gained stronger separation between packages and no longer placed them all under the same prefix.)

Setting -DCMAKE_PREFIX_PATH would be more reasonable than -DCMAKE_INSTALL_PREFIX, since it does affect dependency search, but I still think the better fix here is to set environment variables correctly (PATH, CMAKE_PREFIX_PATH, etc) in one place, instead of relying hardcoded on CMakeCache.txt settings to indirectly influence search behavior. Setting environment variables is preferable to hardcoding cache entries, because it makes fewer assumptions about how cmake projects manage the cache, and it ensures the configuration is inherited by subprocesses. Cache variables are a good way of controlling builds, but environment variables are a better way of providing information about build environments IMO.

This fixes a build failure in native_libmultiprocess where cmake would find the system (nix) CapnProto 1.2.0 instead of the depends-built CapnProto 1.4.0, resulting in a compile-time version mismatch error: #error "Version mismatch between generated code and library headers."

Why doesn't setting CMAKE_INSTALL_PREFIX help?

https://github.com/bitcoin/bitcoin/blob/976985eccd546a95e38973b854ccc6589e8afc74/depends/funcs.mk#L221

According to the documentation, it gets included in the search paths by default.

Setting -DCMAKE_PREFIX_PATH would be more reasonable than -DCMAKE_INSTALL_PREFIX...

In that comment, I'm not debating which approach is more reasonable. My point is that if the current code setting CMAKE_INSTALL_PREFIX doesn't work for you, but this branch, which additionally sets CMAKE_PREFIX_PATH, does, it might point to a deeper issue.

By the way, is the original issue Nix-specific? If not, could you provide the exact steps to reproduce it?

In that comment, I'm not debating which approach is more reasonable. My point is that if the current code setting CMAKE_INSTALL_PREFIX doesn't work for you, but this branch, which additionally sets CMAKE_PREFIX_PATH, does, it might point to a deeper issue.

Oh, thanks. I don't see how it could point to deeper issues other than cmake's documentation being a little confusing, or nix overriding some of cmake's defaults, and we already know both things happen.

Setting CMAKE_INSTALL_PREFIX isn't a reliable way to tell cmake where to find dependencies or how to prioritize different versions of the same dependency. If setting CMAKE_INSTALL_PREFIX happens to have positive side effects on some systems, that's great, but the point of CMAKE_INSTALL_PREFIX is to control where packages are installed not where dependencies are found.

If you think there could be a deeper issue here, could explain what type of issue you think it might be?

By the way, is the original issue Nix-specific? If not, could you provide the exact steps to reproduce it?

I think the main bug this PR is fixing is that the CMAKE_MODULE_PATH value depends currently specifies is ignored, and this bug is not nix-specific.

I happened to encounter the problem by running make -C depends using a shell.nix with capnproto in its buildInputs. This caused nix to install capnproto 1.2 and libmultiprocess to use the capnproto 1.2 package and code generator. But at compile time the -I/home/bitcoin/depends/x86_64-pc-linux-gnu/native/include include path added by depends would cause capnproto 1.4 library headers to be preferred over capnproto 1.2 headers, leading to the error "Version mismatch between generated code and library headers." (EDIT: fixed original description of the version mismatch which had some details wrong)

cc @purpleKarrot

There are three ways to approach this:

1. Install all projects to a common prefix

In the simplest approach, you configure, build, (optionally test), and install all packages to a single install prefix. And you want to find dependencies in that same prefix. Hence, you would set both CMAKE_INSTALL_PREFIX and CMAKE_PREFIX_PATH to the same value for all projects. The former controls where to put things, the latter controls where to search things.

2. Install each project to a dedicated prefix

This approach gives you more control which package may access which other package. You would set CMAKE_INSTALL_PREFIX differently for each project and you would explicitly set <project>_DIR for each dependency.

3. Consume projects from their build tree

In this approach, you don't perform the install steps, so setting CMAKE_INSTALL_PREFIX is superfluous. You would still set <project>_DIR for each dependency.

Remarks

Approaches 1 and 2 require that all projects properly install their targets using install(EXPORT). Approach 3 requires that projects export their targets using export(EXPORT).

It is good practice to use both install(EXPORT) and export(EXPORT) consistently in all CMake projects, so that all three approaches described above (plus FetchContent) work transparently and the downstream users may consume the project however they prefer. In practice, it is really rare to see a project that does it properly, because, well ...

cmake's documentation being a little confusing

All of this is independent to CMake's modules. Hence, CMAKE_MODULE_PATH should not even be mentioned in the description.

CMAKE_MODULE_PATH should not even be mentioned in the description

Thanks, CMAKE_MODULE_PATH is mentioned in the description because 9a7f122d07005021252ca8f3c272cae740fefb26 is dropping a broken CMAKE_MODULE_PATH assignment in the code.

Note also that support for CMAKE_INSTALL_PREFIX as an environment variable requires CMake version 3.29. Our minimum supported version is still 3.22, so you should set the cmake variable via -D.