kernel: create monolithic kernel static library

hebasto commented at 9:50 am on July 28, 2025: member

Currently, consuming libbitcoinkernel.a requires all its dependency static libraries to be available. A switch to a monolithic variant, which contains object files from its dependencies, was discussed in the Kernel WG. The necessary preparations in the libsecp256k1 build scripts were completed in https://github.com/bitcoin-core/secp256k1/pull/1678, which are now available in this repository since #33036.

The changes in this PR were picked from https://github.com/theuni/bitcoin/commits/static_kernel/, with an additional adjustment in libbitcoinkernel.pc.in.

This PR can be tested as described in #30814#issue-2505698234.

kernel: create monolithic kernel static library fdbade6f8d

hebasto added the label Build system on Jul 28, 2025

DrahtBot commented at 9:50 am on July 28, 2025: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33077.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	TheCharlatan, stickies-v
Concept NACK	purpleKarrot

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#32543 (kernel: Remove dependency on clientversion by TheCharlatan)
#31807 (kernel: Avoid duplicating symbols in the kernel and downstream users by theuni)
#30595 (kernel: Introduce initial C header API by TheCharlatan)
#28690 (build: Introduce internal kernel library by TheCharlatan)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

TheCharlatan commented at 10:43 am on July 28, 2025: contributor

Tested it out over here: https://github.com/TheCharlatan/rust-bitcoinkernel/tree/kernelApi_symbols_minimal_boost

purpleKarrot commented at 11:02 am on July 28, 2025: contributor

I think we should reduce the amount of “clever hacks” in the CMake code, not increase them. If .pc files are not expressive enough for exporting all our target properies (and they will never be, because .pc files are not relocatable), we should stop providing them and export target information as CMake packages only. If CMake packages are not expressive enough either, because we don’t want to export dependencies in case of a static library, we should simply not export a static library to a package at all. This would still allow clients to to use the static library if they build it as a subproject. They will want to do that anyway, because they want to make sure that their static libraries are built with the same toolchain and compile flags.

Examples for exporting, importing, as well as using subprojects can be found in https://habla.news/u/purplekarrot.net/cmake-import-export

Currently, when clients want to use bitcoin as a suproject, they get their build-type and compile flags silently changed, and they get a summary spewed into their configure log. This is what should be fixed instead.

hebasto commented at 11:54 am on July 28, 2025: member

Currently, when clients want to use bitcoin as a suproject, they get their build-type and compile flags silently changed, and they get a summary spewed into their configure log. This is what should be fixed instead.

These ideas are orthogonal to this PR, aren’t they?

purpleKarrot commented at 1:58 pm on July 28, 2025: contributor

These ideas are orthogonal to this PR, aren’t they?

From the perspective of the implementation, yes. From the perspective of the client, no.

A client may want to build an application that links the kernel. There are two options: One is to import the kernel package, the other is to build the kernel as a subproject. If the client prefers to statically link the kernel, option two should be preferred. This PR optimizes for option one.

TheCharlatan approved

TheCharlatan commented at 10:14 am on July 29, 2025: contributor

ACK fdbade6f8ded63519b637c8fa6f39914a400ab5e

The alternative of requiring upstream consumers to use cmake to import the library topology into their projects seems a bit heavy handed to me.

purpleKarrot commented at 12:03 pm on July 29, 2025: contributor

For https://habla.news/u/purplekarrot.net/cmake-import-export, I have set up several example projects for exporting cmake packages and I deliberately mismatched consumers by using gcc, clang, and msvc.

I am making the bold claim that there is no working alternative to shipping a shared library with cmake package information. Consuming a static library requires a consistent toolchain. .pc files are not reloatable.

I will happily remove my NACK once I am proven wrong with a counterexample. I will hold anyone’s beer in the meantime.

ryanofsky commented at 2:05 pm on July 29, 2025: contributor

Consuming a static library requires a consistent toolchain. .pc files are not reloatable. @purpleKarrot I read your post the other day and enjoyed it and learned a number of things. And I understand why cmake files are more flexible than .pc files and why they can’t support mismatched compilers / libraries and relocation.

But is there something wrong with providing and using .pc files in the case where there is a consistent toolchain and nothing is relocated? This seems like a common & useful case to support, and could have the advantage of supporting downstream projects that don’t use cmake like rust projects, meson projects, etc.

Even though it is not the general case, I think the case where only one toolchain is used and nothing is relocated is pretty common when you are building packages locally or building an fsroot for another system.

purpleKarrot commented at 2:29 pm on July 29, 2025: contributor

is there something wrong with providing and using .pc files in the case where there is a consistent toolchain and nothing is relocated?

Of course not. But how do you guarantee that case? You mentioned it: By building from source, which is exactly my point.

We cannot provide packages that contain static libraries and/or .pc files. @theuni said:

We’ve always tried to discourage distros and 3rd parties from packaging, in favor of our deterministic build process. In Core’s case, we go out of our way to make safe decisions so that downstreams don’t make bad ones.

But if we want to make sure that 3rd parties use a deterministically built kernel, we either need to force them to use a dynamic library or to build their application inside the same deterministic build environment.

TheCharlatan commented at 3:08 pm on July 29, 2025: contributor

But if we want to make sure that 3rd parties use a deterministically built kernel, we either need to force them to use a dynamic library or to build their application inside the same deterministic build environment.

I don’t think we should force anything. If we try to, people will find worse ways to get around this limitation, e.g. by manually picking source files out of the repository and creating the library that way (which they did for libbitcoinconsensus). That said, releasing a shared library through our process would still be nice to have. As you say there is a relocation issue with pkgconfig, and in experimenting with it over the past twelve months I also found it a surprisingly inconsistent tool between platforms. Would be nice if we could get rid of it for something sane. I originally added the pkgconfig file when we were still on autotools.

However it is more convenient in the case of compiling from source to just link a single static object instead of a potentially changing list of them. That’s all this change does. FWIW I don’t think we should ship a static library as part of a release. Having an install target for it nonetheless could still make it a little more convenient for projects compiling from source.

ryanofsky commented at 3:10 pm on July 29, 2025: contributor

We cannot provide packages that contain static libraries and/or .pc files.

Oh, that makes sense to me. I thought you might have been objecting to having code in the build system that generates .pc files. But I agree distributing .pc files with precompiled static libraries does not seem like it would be very useful, and I assume most developers would just compile from source. Would be interested to know more if this is incorrect and anyone is asking for static binaries or would find them useful.

I am unsure about the relationship between that issue and this PR. This PR seems to want to install a single kernel library instead of a public kernel library and a few private ones it depends on. I can see why you might not like that change because it would not be compatible with the dependency injection described in your blog post. But the PR is only dropping code that was specific to .pc files, so I think as a practical matter this PR would not actually make it harder to support dependency injection if we exported cmake config files in the future?

(On the other side, I’m also not sure about the benefits of this PR. PR description links to a lot of issues and might be clearer if it just described the problem it solves, or explained if the main movitation is simplifying cmake code or reducing the number of installed files or something else.)

theuni commented at 6:50 pm on July 30, 2025: member

(On the other side, I’m also not sure about the benefits of this PR. PR description links to a lot of issues and might be clearer if it just described the problem it solves, or explained if the main movitation is simplifying cmake code or reducing the number of installed files or something else.)

At a high level, this simply encapsulates the kernel library. The shared lib already works this way, this PR would just align the static lib’s behavior. The other dependencies (crypto, leveldb, secp, etc) are implementation details and are of no interest to the builder.

If the internal library details are of interest to the builder, they’re likely misusing the kernel. For example, in the past distros (gentoo for ex) have wanted to link against a system leveldb. This change makes it clear that that is unsupported, and prevents users from doing so without hacking the buildsystem.

stickies-v approved

stickies-v commented at 10:58 am on August 4, 2025: contributor

tACK fdbade6f8ded63519b637c8fa6f39914a400ab5e

I rebased https://github.com/TheCharlatan/bitcoin/commits/kernelApi_Cpp_Internal_Headers/ onto fdbade6f8ded63519b637c8fa6f39914a400ab5e (resulting in https://github.com/stickies-v/bitcoin/tree/kernel-cpp-monolith), and I was able to statically link my pbk_ext py-bitcoinkernel nanobind extension to the monolithic libbitcoinkernel.

I reviewed the code changes and could not see any issues, but I am also a cmake novice. The monolithic approach makes sense to me, even if we’ll need to improve on symbol visibility in the future.

fanquake merged this on Aug 6, 2025

fanquake closed this on Aug 6, 2025

hebasto deleted the branch on Aug 6, 2025

kernel: create monolithic kernel static library #33077

Code Coverage & Benchmarks

Reviews

Conflicts