build: add a depends dependency provider

willcl-ark commented at 9:25 pm on May 22, 2025: member

This PR adds a dependency provider to depends builds.

Currently the depends toolchain appears to rely on certain assumptions [citation needed] to find packages correctly. It does largely do this correctly, however NixOS sets some environment variables which interfere with this and cause builds from depends to fail, due to attempting to link system packages.

We can reproduce this interferrence by running a NixOS container and loading system packages using a flake:

 0docker run --pull=always -it nixos/nix
 1
 2# In the container:
 3git clone --depth=1 https://github.com/bitcoin/bitcoin && cd bitcoin
 4
 5# Install system packages for a non-depends build from https://github.com/bitcoin-dev-tools/bix
 6nix develop github:bitcoin-dev-tools/bix --extra-experimental-features flakes --extra-experimental-features nix-command --no-write-lock-file
 7
 8# Attempt depends build
 9make -C depends -j$(nproc)
10cmake -B build --toolchain /bitcoin/depends/<host-platform-triplet>/toolchain.cmake
11
12# Find possibly-interferring env vars:
13env | grep -E 'CMAKE|NIX'

A dependency provider allows the overriding of cmake’s find_package() therefore giving total control over where dependencies come from in a build.

This achieves two things:

Provides stronger guarantees about where dependencies come from during a (depends) build; not permitting environment flags to override these locations [citation needed].
Fixes issues like a non-standard CMAKE_PREFIX_PATH breaking builds (i.e. 32428)

This provider must be specified along with the toolchain, with an invocation of the type:

0make -C depends
1cmake -B build --toolchain depends/x86_64-pc-linux-gnu/toolchain.cmake -DCMAKE_PROJECT_TOP_LEVEL_INCLUDES=depends/x86_64-pc-linux-gnu/dependency_provider.cmake
2cmake --build build

To-do

Figure out why the provider does not like to find boost (link) fixed by rebasing on #32665 and #32667
Fix the failing CI job. This appears to somehow use an old cmake, even after we install cmake 4.0.2. Fixed by using bookworm, gcc-12 and -Wno-error=array-bounds in the previous releases job.

DrahtBot commented at 9:25 pm on May 22, 2025: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/32595.

Reviews

See the guideline for information on the review process.

Type	Reviewers
Concept NACK	fanquake, purpleKarrot

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

No conflicts as of last run.

DrahtBot added the label Build system on May 22, 2025

willcl-ark commented at 9:28 pm on May 22, 2025: member

Opened in draft for feedback on approach.

There is also a bypass for the boost package (in cmake/module/AddBoostIfNeeded.cmake) which I’d like to get working with the dependency provider ideally.

Note that this also bumps the minimum cmake version to 3.24.

willcl-ark requested review from josibake on May 22, 2025

hebasto commented at 9:29 pm on May 22, 2025: member

cc @purpleKarrot

fanquake commented at 9:42 pm on May 22, 2025: member

Provides stronger guarantees about where dependencies come from during a (depends) build; i.e. only from depends.

Can you elaborate on this? Currently, CMake configured with a depends toolchain, should never look outside depends (doing so would be a bug). What are the stronger guarantees?

willcl-ark commented at 10:37 pm on May 22, 2025: member

Provides stronger guarantees about where dependencies come from during a (depends) build; i.e. only from depends.

Can you elaborate on this? Currently, CMake configured with a depends toolchain, should never look outside depends (doing so would be a bug). What are the stronger guarantees?

Right, perhaps I didn’t use the best wording here.

When using the toolchain I have not found us pulling in extraneous deps from outside of depends, but have encountered us missing dependencies from inside of depends, e.g. in my original issue.

By using a dependency provider and intercepting calls to find_package(), as I understand it, we are getting even more control over which packages can be found than when using the toolchain.

This is what I termed “stronger guarantees”, perhaps misleadingly!

DrahtBot added the label CI failed on May 22, 2025

willcl-ark commented at 9:39 am on May 23, 2025: member

The windows job fails as currently we activate the dependency provider automatically based on whether a(ny) toolchain is in use.

This works fine for depends, but doesn’t make sense on windows where the vcpkg toolchain is used, without depends.

The idea for doing it this way was to try and avoid users having to pass two flags in order to build from depends – a toolchain and a dependency-provider – but perhaps this is necessary after all…

I do recall reading somewhere in cmake docs that choice of dependency provider should always be left to the user too, so I suppose doing it this way would fit with that.

willcl-ark force-pushed on May 23, 2025

willcl-ark commented at 12:24 pm on May 23, 2025: member

Pushed an update which now only activates the dependency provider when using the depends toolchain.

This will hopefully appease the windows CI job, as well as making more sense in general and being more expected behaviour.

in CMakeLists.txt:10 in 7bdf1af8f3 outdated

 6@@ -7,7 +7,7 @@
 7 #
 8 # Centos Stream 9, https://www.centos.org/cl-vs-cs/#end-of-life, EOL in May 2027:
 9 #  - CMake 3.26.5, https://mirror.stream.centos.org/9-stream/AppStream/x86_64/os/Packages/
10-cmake_minimum_required(VERSION 3.22)
11+cmake_minimum_required(VERSION 3.24)

maflcko commented at 8:43 am on May 26, 2025:

nit: If we do this, may as well hop to https://packages.debian.org/bookworm/cmake 3.25 and g++-12?

hebasto commented at 5:16 pm on May 27, 2025:

Guix’s cmake-minimal remains at version 3.24.2.

willcl-ark commented at 9:44 am on June 17, 2025:

Going to leave this at 3.24 for now

purpleKarrot commented at 8:44 am on May 26, 2025: contributor

Thank you for working on a dependency provider!

I think core should provide two dependency providers, which will give users three different ways to build:

Not using a dependency provider: CMake will produce an error when dependencies are not found on the system. This will be the preferred way for distro packagers.
A fallback provider: This will use system packages when they are found and fall back to compiling vendored depends. Using this provider, find_package will always succeed, but it will not compile any “unnecessary” code. This approach may be preferred by developers or for quick experiments.
A depends provider: This will ignore system packages and always use vendored depends. This approach will be used for reproducible builds.

At the moment, the dependency provider points to prebuilt depends. Alternatively, the dependency provider could take the responsibility of building depends at the moment they are requested. This would make the current, Makefile based depends system an implementation detail that can be ported piece by piece to CMake.

willcl-ark commented at 8:17 am on May 27, 2025: member

Thanks for the feedback @purpleKarrot.

I like the sound of your approach here myself and will be happy to adapt to it. I think i’ll start off by targetting 3.:

A depends provider: This will ignore system packages and always use vendored depends. This approach will be used for reproducible builds.

…without the optionl “auto-build depends” functionality. This is close to what I have here already (and solves my immediate personal itch regarding building using depends on NixOS).

I would appreciate your input though on one more aspect of this; namely the (dev) UX involved with the 3 options. Option 1 would not see any UX changes.

Option 2 would require dev to specify the “fallback provider” file with a -DCMAKE_PROJECT_TOP_LEVEL_INCLUDES flag.

Option 3 would require the dev to specify both a --toolchain and a -DCMAKE_PROJECT_TOP_LEVEL_INCLUDES.

Option 3 in particular feels a little clunky to me (as a cmake novice), as I’d expect the toolchain to “contain everything needed to build”. But I am guessing this can be abstracted away with a CMakePreset handling passing of the toolchain and dep. provider. This would become even more seamless in a version which auto-built depends for you too.

Does my understanding above sound roughly logical to you? If so I can make the changes later this week.

fanquake commented at 10:43 am on May 29, 2025: member

A fallback provider: This will use system packages when they are found and fall back to compiling vendored depends. Using this provider, find_package will always succeed, but it will not compile any “unnecessary” code. This approach may be preferred by developers or for quick experiments.

I can’t tell from this if you are saying to use depends to “fill in” any missing system packages, or, if all the required system packages aren’t available, we would switch to using depends (unclear if that is actually desirable). Certainly a NACK on mixing and matching the two.

purpleKarrot commented at 2:52 pm on May 30, 2025: contributor

I can’t tell from this if you are saying to use depends to “fill in” any missing system packages,

Yes. The idea is to provide an approach to build core with minimal effort. If you NACK mixing system packages with vendored dependencies, are you saying that nobody will ever want that, or that nobody should ever want that?

fanquake commented at 3:45 pm on May 30, 2025: member

Yes. The idea is to provide an approach to build core with minimal effort.

I think make -C depends && cmake -B --toolchain and apt install your_dependencies && cmake -B are both already pretty minimal effort. It’s not really clear why a developer would apt install x, but not apt install x y, and then would need a behaviour to fallback to depends for y?

I’m also not currently aware of a platform that we support, where the dependencies needed to compile Core, aren’t available via the system package manager; one used to be our specific version of BDB 4.x, but that dependency has now been removed. So this fallback usecase mostly seems like additional build complexity, which creates a larger matrix of “supported” builds to test / debug, for not much gain. We used to have a similar kind of option in our depends system (ALLOW_HOST_PACKAGES), but allowing the use of system packages, with depends, defeats the point of a self-contained dependency builder, so it was removed.

There’s also some nuance around how this would function, are you imaging it all integrated into into CMake, so depends is no-longer a stand alone system? Otherwise, we need logic in CMake, to detect if the depends requirements (curl, compilers, patch, binutils etc) are actually installed / usable (i.e otherwise the fallback would fail because curl isn’t installed, probably leaving the user wondering why it’s trying to download stuff), as well as making sure that build flags / compiler selection are all passed through to depends (i.e if the user is trying to build with Clang & libc++, then any tools / flags need to be forwarded to the depends compilation).

I’m also just not sure that if a dependency is missing, downloading and compiling something is even the best behaviour. Maybe the user just forgot to apt install it, and will do so when CMake fails to configure?

willcl-ark force-pushed on Jun 2, 2025

willcl-ark commented at 1:20 pm on June 2, 2025: member

OK I reworked this a little so that it works as described in top comment.

The overview is that the provider must be manually specified with -DCMAKE_PROJECT_TOP_LEVEL_INCLUDES, thus keeping full control for the user.

The toolchain remains usable (as today) without the provider being specified.

fanquake commented at 4:58 pm on June 2, 2025: member

https://cirrus-ci.com/task/5352790488252416?logs=ci#L1675:

 0[09:31:48.288] Extracting freetype...
 1[09:31:48.316] /ci_container_base/depends/sources/freetype-2.11.0.tar.xz: OK
 2[09:31:48.562] Preprocessing freetype...
 3[09:31:48.575] Configuring freetype...
 4[09:31:48.656] CMake Error at CMakeLists.txt:100 (cmake_minimum_required):
 5[09:31:48.656]   Compatibility with CMake < 3.5 has been removed from CMake.
 6[09:31:48.656] 
 7[09:31:48.656]   Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
 8[09:31:48.656]   to tell CMake that the project requires at least <min> but has been updated
 9[09:31:48.656]   to work with policies introduced by <max> or earlier.
10[09:31:48.656] 
11[09:31:48.656]   Or, add -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to try configuring anyway.
12[09:31:48.656] 
13[09:31:48.656] 
14[09:31:48.656] -- Configuring incomplete, errors occurred!

in ci/test/01_base_install.sh:35 in 4d568f95e4 outdated

31@@ -32,6 +32,18 @@ if [ -n "${APT_LLVM_V}" ]; then
32   )
33 fi
34 
35+# Fetch newer cmake on Ubuntu 22.04

maflcko commented at 7:41 am on June 3, 2025:

not sure. The CI in this repo is probably the wrong place for fragile nightly tests. At a minimum, it would be good to pin the version. My preference would be to just use bookworm, see https://github.com/bitcoin/bitcoin/pull/32595/files#r2106854007

willcl-ark commented at 9:18 am on June 3, 2025:

Just to be clear, you mean switching the native_previous_releases job from ubuntu:22.04 to debian:bookworm?

willcl-ark commented at 7:37 am on June 13, 2025:

Hmmm, on bookworm (+ gcc12) we seem to fail on a new error: https://github.com/bitcoin/bitcoin/pull/32595/checks?check_run_id=44008178986 which I haven’t looked into much yet

fanquake commented at 8:06 am on June 13, 2025:

fail on a new error:

This looks similar to what we are already working around in the native fuzz valgrind task, so it seems fine to use the same workaround here (-Wno-error=array-bounds).

maflcko commented at 8:20 am on June 13, 2025:

My preference would be to keep at least one CI with the min required gcc version. Albeit I am not aware of any changes that would affect us, compared to gcc-12, it seems inconsistent to bump it early before bumping the minimum required.

willcl-ark force-pushed on Jun 12, 2025

in ci/test/00_setup_env_native_previous_releases.sh:12 in 05b9e17b3a outdated

 6@@ -7,10 +7,10 @@
 7 export LC_ALL=C.UTF-8
 8 
 9 export CONTAINER_NAME=ci_native_previous_releases
10-export CI_IMAGE_NAME_TAG="mirror.gcr.io/ubuntu:22.04"
11+export CI_IMAGE_NAME_TAG="mirror.gcr.io/debian:bookworm"
12 # Use minimum supported python3.10 and gcc-11, see doc/dependencies.md
13-export PACKAGES="gcc-11 g++-11 python3-zmq"
14-export DEP_OPTS="CC=gcc-11 CXX=g++-11"
15+export PACKAGES="gcc-12 g++-12 python3-zmq"

maflcko commented at 7:44 am on June 13, 2025:

bookworm has gcc-11, so I guess you can use that for now https://packages.debian.org/en/bookworm/g++-11

willcl-ark commented at 7:48 am on June 13, 2025:

This throws an error ~ that FORTIFY_SOURCE=3 is a warning on gcc-11, and we are treating warnings as errors. I could use FORTIFY_SOURCE=2 in this job as a workaround that (which is why I moved to gcc-12 initially here)?

maflcko commented at 7:55 am on June 13, 2025:

FORTIFY_SOURCE=2

Seems fine. An alternative would be to disable the warning specifically, if possible. I wonder why it doesn’t happen on the Ubuntu gcc-11 today.

willcl-ark commented at 10:09 am on June 13, 2025:

I see the same warning (error) locally on bookworm with gcc-11 too:

 0C++ compiler .......................... GNU 11.3.0, /usr/bin/g++-11
 1CMAKE_BUILD_TYPE ...................... Debug
 2Preprocessor defined macros ........... DEBUG DEBUG_LOCKORDER DEBUG_LOCKCONTENTION RPC_DOC_CHECK ABORT_ON_FAILED_ASSUME
 3C++ compiler flags .................... -funsigned-char -g2 -O2 -std=c++20 -fPIC -fno-extended-identifiers -fdebug-prefix-map=/ci_container_base/src=. -fmacro-prefix-map=/ci_container_base/src=. -fstack-reuse=none -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -Wstack-protector -fstack-protector-all -fcf-protection=full -fstack-clash-protection -Werror -Wall -Wextra -Wformat -Wformat-security -Wvla -Wredundant-decls -Wdate-time -Wduplicated-branches -Wduplicated-cond -Wlogical-op -Woverloaded-virtual -Wsuggest-override -Wimplicit-fallthrough -Wunreachable-code -Wundef -Wno-unused-parameter -DBOOST_MULTI_INDEX_ENABLE_SAFE_MODE
 4Linker flags .......................... -funsigned-char -g2 -O2 -fstack-reuse=none -fstack-protector-all -fcf-protection=full -fstack-clash-protection -Wl,-z,relro -Wl,-z,now -Wl,-z,separate-code -Wl,--exclude-libs,ALL -fPIE -pie
 5
 6NOTE: The summary above may not exactly match the final applied build flags
 7      if any additional CMAKE_* or environment variables have been modified.
 8      To see the exact flags applied, build with the --verbose option.
 9
10Treat compiler warnings as errors ..... ON
11Use ccache for compiling .............. ON
12
13
14-- Configuring done
15-- Generating done
16-- Build files have been written to: /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu
17+ bash -c 'cmake --build . -j15 --target all install'
18[  0%] Building CXX object src/CMakeFiles/crc32c.dir/crc32c/src/crc32c.cc.o
19[  0%] Building CXX object src/univalue/CMakeFiles/univalue.dir/lib/univalue.cpp.o
20[  0%] Building C object src/secp256k1/src/CMakeFiles/secp256k1_precomputed.dir/precomputed_ecmult.c.o
21[  0%] Building CXX object src/CMakeFiles/bitcoin_cli.dir/compat/stdin.cpp.o
22[  0%] Building CXX object src/crypto/CMakeFiles/bitcoin_crypto.dir/aes.cpp.o
23[  0%] Building CXX object src/univalue/CMakeFiles/univalue.dir/lib/univalue_get.cpp.o
24[  0%] Building C object src/secp256k1/src/CMakeFiles/exhaustive_tests.dir/tests_exhaustive.c.o
25[  2%] Building CXX object src/test/util/CMakeFiles/test_util.dir/blockfilter.cpp.o
26[  2%] Building CXX object src/CMakeFiles/minisketch.dir/minisketch/src/minisketch.cpp.o
27[  2%] Generating bitcoin-build-info.h
28[  2%] Building CXX object src/CMakeFiles/bitcoin_consensus.dir/arith_uint256.cpp.o
29[  2%] Building CXX object src/univalue/CMakeFiles/univalue.dir/lib/univalue_read.cpp.o
30[  2%] Building CXX object src/zmq/CMakeFiles/bitcoin_zmq.dir/zmqabstractnotifier.cpp.o
31[  2%] Building CXX object src/CMakeFiles/leveldb.dir/leveldb/db/builder.cc.o
32[  2%] Generating locale/bitcoin_am.qm
33[  2%] Generating locale/bitcoin_ar.qm
34[  2%] Generating locale/bitcoin_ast_ES.qm
35In file included from /usr/include/x86_64-linux-gnu/c++/11/bits/os_defines.h:39,
36                 from /usr/include/x86_64-linux-gnu/c++/11/bits/c++config.h:586,
37                 from /usr/include/c++/11/cstdio:41,
38                 from /ci_container_base/src/compat/stdin.cpp:7:
39/usr/include/features.h:423:5: error: #warning _FORTIFY_SOURCE > 2 is treated like 2 on this platform [-Werror=cpp]
40  423 | #   warning _FORTIFY_SOURCE > 2 is treated like 2 on this platform
41      |     ^~~~~~~
42cc1plus: all warnings being treated as errors
43gmake[2]: *** [src/CMakeFiles/bitcoin_cli.dir/build.make:76: src/CMakeFiles/bitcoin_cli.dir/compat/stdin.cpp.o] Error 1
44gmake[1]: *** [CMakeFiles/Makefile2:880: src/CMakeFiles/bitcoin_cli.dir/all] Error 2
45gmake[1]: *** Waiting for unfinished jobs....

I added -Wno-error=array-bounds to CMAKE_CXX_FLAGS as in fuzz job, but this just still complained about “the fortify source level on this platform”.

Going to reduce the FORTIFY_SOURCE level for now in absence of another fix.

fanquake commented at 10:14 am on June 13, 2025:

I added #32595 (review) to CMAKE_CXX_FLAGS

That workaround is only for GCC 12.

willcl-ark commented at 10:16 am on June 13, 2025:

Ah ok

willcl-ark commented at 9:44 am on June 17, 2025:

taken -Wno-error=array-bounds and gcc-12 plus bookworm

willcl-ark force-pushed on Jun 13, 2025

DrahtBot removed the label CI failed on Jun 13, 2025

willcl-ark force-pushed on Jun 13, 2025

depends: Bump boost to 1.88.0 and use new CMake buildsystem

This has a few advantages over the old method of simply copying headers:
- Installs proper cmake files which can be picked up by our buildsystem
- Only installs necessary headers, not all of boost

Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>

948d00c4c2

build: Find Boost in config mode

The `FindBoost` module has been removed by policy CMP0167.

a7c62255bc

willcl-ark force-pushed on Jun 16, 2025

ci: use bookworm in native_previous_releases

Ubuntu 22.04 supplies cmake 3.22 which is below our minimum version.

- Switch to bookworm for cmake 3.25.
- Set FORTIFY_SOURCE=2, as level 3 creates a wraning on this platform.

4cbb32e55c

willcl-ark force-pushed on Jun 17, 2025

build: add a depends dependency provider 9d6529e641

build: declare dependency provider from depends

After depends have been built, print out the required flags needed to
use the dependency provider along with the existing --toolchain message.

Include configure output showing active toolchain and dependency
provider information.

9347fdb126

willcl-ark force-pushed on Jun 17, 2025

build: add a depends dependency provider #32595

To-do

Code Coverage & Benchmarks

Reviews

Conflicts