Project Board: https://github.com/orgs/bitcoin/projects/3
This is the tracking issue for the libbitcoinkernel project. The original tracking issue is found in #24303.
The libbitcoinkernel project is a new attempt at extracting Bitcoin Core’s consensus engine. The kernel part of the name highlights one of the key functional differences from the deprecated libbitcoinconsensus and in fact, most libraries: it is a stateful library that can spawn threads, do caching, do I/O, and many other things that one may not normally expect from a library.
This statefulness is necessary for libbitcoinkernel’s decidedly incremental approach to extracting our consensus engine. This approach favors:
-
Reusing existing code …which allows us to be continually integrated with Bitcoin Core and benefit from our extensive test suite
-
Incremental decoupling instead of building from scratch …which allows us to avoid having to prematurely optimize for a “perfect” boundary or API (tends to be highly subjective, non-obvious, may lead to unproductive bike-shedding before we’ve even done anything meaningful)
The work of extracting the validation engine into a library and making the API ergonomic is likely to be a multi-release project involving multiple contributors. The incremental approach takes this into account and respects the sheer size of work (both in writing code and getting it through review) that needs to be undertaken.
PRs
Please see the project board: https://github.com/orgs/bitcoin/projects/3
The Game Plan
- Stage 1
- Step 1: Create
bitcoin-chainstatebinary - Step 2: Prune unnecessary modules
- Step 3: Prune unnecessary headers
- Step 1: Create
- Stage 2
Stage 1: Extracting out a usable libbitcoinkernel.{so,dylib,dll}
The introducedbitcoin-chainstate executable uses the validation engine to validate blocks passed to it on the command line. Its build system code revealed the minimal set of files needed to link in to use the consensus engine as-is. Over time, these files were further pruned to only include functionality that is strictly required for validation. This pruning is enforced by the build system such that future re-coupling of validation code with non-validation modules will result in linker errors.
The mempool is not decoupled, because some users of libbitcoinkernel may want to have an embedded mempool with Bitcoin Core’s policies. Instead these kind of “Bitcoin Core specific” functionality (also includes assumevalid, assumeutxo, and checkpoints) should be completely optional and configurable by the user.
The current design of the mempool within the validation code requires the boost headers to be exported. This is not ideal, since it forces the users of the kernel library to include the boost headers too if they wish to use this functionality. Removing them has been attempted in #28335, but the approach taken has received mixed review. In future, this might be solved by only exporting a C header to users, which would internalize any exported boost symbols.
libbitcoinkernel could also be used as an internal library for libbitcoin_node. The desired library organization is shown in doc/design/libraries.md. This is attempted in #28690.
Stage 2: Ship an external kernel API
Now that an internal kernel library with a limited feature set exists, expose it to external users with a C header. Start with a reduced header, exposing just enough functionality to build a utility tool. Ideas for such a utility tool could be a rewrite of bitcoin-chainstate using the C header’s API, or a reindexer tool. These tools could also be written in other languages, such as Rust or Python. Care should be taken that the API is as consistent as possible in all respects, manages its own memory to the extent possible, minimizes type conversions between C and C++, and keeps versioning in mind.
- Introduce initial C API #30595
- Export symbols required for interface
- Settle on library topology
- Add fatal error return codes
Ideally, users looking to integrate with libbitcoinkernel will provide inputs on which library features might be desirable to have exposed. The API will initially have a very idiosyncratic, Bitcoin Core-specific interface. Continual polishing over multiple versions will incrementally make the libbitcoinkernel API more ergonomic for users outside of Bitcoin Core. Note though, that there is a possible dichotomy here between optimizing the library interface for external users and for the internal interface within Bitcoin Core.
Another goal of this final stage of the project could be to ship an “IO-less” version of the library. This would mean that the block store and coins database would have to be abstracted such that the user can provide their own implementations. It should optionally also not use any threads, atomics, or otherwise platform-dependent features. This would allow the library to target bare metal environment such as targeted by riscv-unknown-elf-g++.
Action Items
- If you have any questions, please post them below!
- If you have any ideas for the future direction of “Stage 2: Polishing the API / Continual De-coupling”, please leave a comment below, I’d love to talk!
Project-wide TODOs
These are suggestions for further cleanup and improvements that came up during review:
- Russell’s post-merge review comment: [[kernel 2e/n] miner: Make
mempooloptional, stop constructing temporary empty mempools #25223 (comment)](https://github.com/bitcoin/bitcoin/pull/25223#discussion_r898164322) - Various followups for [refactor, kernel: Decouple ArgsManager from blockstorage #27125](https://github.com/bitcoin/bitcoin/pull/27125)
- Martin’s request for a follow-up fixing the docstring: #27125 (review)
Other various items that arose during review and should be tracked
- Cory’s request for cleaning up the kernel interface functions of pointer and reference types: #27636 (review)
- Marcos’ request for unrolling the reindex loop: #27125 (review)
- Marco’s request for getting rid of exceptions in the
ArgsManager: #27491 (review) - Russell’s request for eliminating the BlockNotify signal: #27636 (review)
