Bitcoin Kernel Library Project Tracking #27587

issue TheCharlatan openend this issue on May 6, 2023
  1. TheCharlatan commented at 8:27 am on May 6, 2023: contributor

    Project Board for pull requests past and present: https://github.com/orgs/bitcoin/projects/3

    This is the tracking issue for the bitcoin kernel library (libbitcoinkernel) project. The original tracking issue is found in #24303.

    The bitcoin kernel project is a new attempt at extracting Bitcoin Core’s consensus engine. The kernel part of the name highlights one of the key functional differences from the deprecated libbitcoinconsensus and in fact, most libraries: it is a stateful library that can spawn threads, do caching, do I/O, and many other things that one may not normally expect from a library.

    libbitcoinkernel could also be used as an internal library for libbitcoin_node. The desired library organization is shown in doc/design/libraries.md. This is attempted in #28690.

    The statefulness is necessary for the bitcoin kernel’s decidedly incremental approach to extracting our consensus engine. This approach favors:

    1. Reusing existing code …which allows us to be continually integrated with Bitcoin Core and benefit from our extensive test suite

    2. Incremental decoupling instead of building from scratch …which allows us to avoid having to prematurely optimize for a “perfect” boundary or API (tends to be highly subjective, non-obvious, may lead to unproductive bike-shedding before we’ve even done anything meaningful)

    The work of extracting the validation engine into a library and making the API ergonomic is likely to be a multi-release project involving multiple contributors. The incremental approach takes this into account and respects the sheer size of work (both in writing code and getting it through review) that needs to be undertaken.

    Areas with open work

    Defining an API for the library

    Headers installed alongside the library should define a safe and stable interface for its users. The main thrust of work is currently towards providing a first version for a C header providing rudimentary support for validating scripts and blocks, as well as reading from the block store: #30595. See the pull request description for a complete list of binaries and projects using it already. Re-using the API for some of our internal binaries would guarantee dogfooding as well as provide incentives for improving and maintaining it.

    The headers have bindings in:

    More bindings would be desirable for the following categories:

    • Application developers and systems engineers (Swift, Kotlin, Dart, Zig, Haskell, Clojure, Javascript etc.)
    • Data analysis and research (R, Julia etc.)
    • Data queries (e.g. PostgreSQL external data etc)

    Some work has also been done for expanding this initial API, including:

    An alternative to installing C headers would be installing C++ headers directly. This can either be done by installing all the existing Bitcoin Core headers wholesale, or by exposing a mix between existing headers and headers providing a cleanly wrapped API. The latter approach was also implemented in the following branch: https://github.com/TheCharlatan/bitcoin/tree/kernelApi_Cpp_Internal_Headers.

    The current API defined through the C header only has very rudimentary error handling. While it attempts to follow the philosophy that errors arising from unsanitized user input are programming errors, it does surface more concrete information on validation failures, as well as errors that arise from problems with the underlying system. These are referred to as fatal errors. The current Bitcoin Core code does not always surface these fatal errors to the caller. #29642 attempts to surface this error information directly.

    It is also currently not possible to instantiate multiple kernel validation objects and distinguish between their log messages. The library currently only has a global logger. See #30342 for a potential solution to this.

    Some small build system tweaks to ensure the correct symbols get exported would also be required, see the discussion in https://github.com/bitcoin-core/secp256k1/pull/1677.

    Read-only kernel clients

    The kernel library provides the functionality to read Bitcoin Core’s data, like blocks, headers, and undo data, directly from disk without additional overhead. External applications like Electrs and Esplora parse block.dat files directly for index building. This is problematic because Bitcoin Core’s block storage is internal and provides no guarantees for external consumers. Reading this data directly can give these guarantees, as well as improve performance, since the block store might contain invalid blocks, or blocks from dead forks.

    The current drawback to this approach is that leveldb, which stores the block tree and the chainstate, does not allow for readers in parallel processes. Bitcoin Core therefore has to shutdown before an external process using the kernel library can read its data. A pull request moving the block tree database from leveldb to a simple hand-rolled format, which would allow reading the data in parallel, was opened in #32427.

    A summary of how the kernel library might be used for reading data is provided here. Applications building on #32427 could validate this architecture as well as provide benchmarks for this approach over using Bitcoin Core’s existing interfaces to read block data.

    Context-minimal validation

    The current design of the library includes all the Bitcoin Core specific data storage methods. Alternative clients wishing to validate blocks therefore have to build the chain block-by-block and have to re-use the UTXO model.

    However there exist projects and proposed alternative data models like utreexo, swift sync, UHS, and libbitcoin, that do not require a UTXO set for validation, but could still profit from re-using Bitcoin Core’s Block validation code. Floresta, a utreexo implementation, and a project working on swiftsync have voiced strong desire for these features.

    There are a few possible directions to take here, including:

    Context-minimal validation might also be interesting for SPV clients, which wish to re-use only a subset of validation.

    cs_main split

    Currently the cs_main recursive mutex protects not only the bulk of validation-specific mutable state, but also state in net processing. Ideally validation would have its own mutexes, that do not have to be shared with other functionality. Function calls requiring a lock to be held as a pre-condition to ensure consistent state transitions should be consolidated to allow internal locking. This could eventually allow users to process validation tasks and query for data in parallel with less locking contention. It also could allow parallel processing and better prioritization of network messages.

    Some draft work towards this goal has been started

    Chainstate / ChainstateManager split

    The Chainstate and ChainstateManager are currently tightly coupled, but need not be. In practice this means working with a single chainstate is often needlessly complicated. A draft split of the two classes has been attempted in https://github.com/TheCharlatan/bitcoin/tree/chainmansplit.

    Compile targets for embedded, web assembly, and proof systems

    One step further than context-free validation would be a kernel API that can compile to architectures with reduced capabilities, e.g. no support for threading, file systems, or atomics. Adding these capabilities could allow the library to be used in embedded environments, e.g. as part of a validating lightning signer or SPV wallet. Targetting riscv bare metal, or llvm IR, could allow re-use in ZK proof systems as well as formal specification languages.

    Eventually the goal here should be to provide a fully “sans IO” version of the API.

    A PR adding a riscv bare metal CI job as opened in #31425.

    Evicting the mempool / plug-able policy

    The mempool and its policy rules are currently inside the kernel library, but won’t be exposed in the initial API version. Ideally the library should not force its users to re-use the existing Bitcoin Core opinionated implementation, but allow them to either define their own mempool or at least define their own policy rules. A rough proof of concept for removing the mempool from the library is implemented in https://github.com/TheCharlatan/bitcoin/tree/mempoolout.

    The mempool is currently also the only dependent on boost within the library. Having boost as a dependency makes it a bit harder to build, and makes using the C++ headers in the library directly more annoying.

    Evicting AssumeUTXO / AssumeValid

    AssumeUTXO/Assumevalid are Bitcoin Core specific approaches to improving IBD speed. It is likely external users of Kernel will not want to be tightly coupled to Bitcoin Core specific processes, e.g., choosing the AssumeUTXO snapshots and assumevalid points. Philosophically, kernel should be as agnostic as possible and not tied to a particular implementation’s decisions.

    Isolated kernel repository

    Eventually Bitcoin Core could itself depend on a standalone kernel repository. Bitcoin Core would likely end up pulling in this project as a subtree. Some open questions around this subject are:

    • How would the CI for this repository be set up?
    • How would its own fuzzing and unit test setup look like?
    • What does the release process look like?
    • What is the governance of the project? E.g., is it under Bitcoin Core?

    In many ways, this could be similar to how the libsecp256k1 is maintained in its own repository. Some steps towards this goal have been taken:


    Action Items

    1. If you have any questions, please post them below!
    2. If you have any ideas for the future directions of the project, or want to work on one of the open areas, I’d love to talk about them and support you!

    Project-wide TODOs

    These are suggestions for further cleanup and improvements that came up during review:

    Other various items that arose during review and should be tracked

    • Cory’s request for cleaning up the kernel interface functions of pointer and reference types: #27636 (review)
    • Marco’s request for getting rid of exceptions in the ArgsManager: #27491 (review)
    • Russell’s request for eliminating the BlockNotify signal: #27636 (review)
  2. fanquake pinned this on May 6, 2023
  3. willcl-ark commented at 1:16 pm on May 30, 2023: member

    Would it make sense to update the project page to https://github.com/orgs/bitcoin/projects/3/views/1 (which if I’m not mistaken seems to be the current one)?

    Could keep a link to the old project https://github.com/bitcoin/bitcoin/projects/18 for reference?

  4. TheCharlatan commented at 1:25 pm on May 30, 2023: contributor

    Re #27587 (comment)

    Would it make sense to update the project page to https://github.com/orgs/bitcoin/projects/3/views/1 (which if I’m not mistaken seems to be the current one)?

    Thanks for the notice, updated to the new one.

    Could keep a link to the old project https://github.com/bitcoin/bitcoin/projects/18 for reference?

    There is no useful extra information on the old board that is not in the new one, so I don’t think this is necessary.

  5. fanquake referenced this in commit 9564f98fee on May 30, 2023
  6. ryanofsky referenced this in commit 153a6882f4 on Jun 9, 2023
  7. ryanofsky referenced this in commit 75135c673e on Jul 6, 2023
  8. fanquake referenced this in commit b565485c24 on Aug 7, 2023
  9. darosior commented at 2:01 pm on September 8, 2023: member

    @TheCharlatan sometimes mentions sourcetrail as a tool to visually inspect the currently required kernel headers. I’ve set it up and figured it could be worth sharing with anyone who’s interested in having a quick look to the trail of headers without having to set up the tool themselves.

    The list of headers included in libbitcoinkernel as of current master d2ccca253f4294b8f480e1d192913f4985a1af08: bitcoin-chainstate_sourcetrail

  10. fanquake commented at 3:42 pm on September 8, 2023: member

    Nice. Played around with this. If you merge a couple things, and turn on c++20, so we can use <bit> (see corys branch), bitcoin-config.h disappears:

    less_headers

  11. fanquake referenced this in commit 1e9d367d0d on Sep 14, 2023
  12. Frank-GER referenced this in commit a202eda9b4 on Sep 19, 2023
  13. achow101 unpinned this on Oct 5, 2023
  14. bitcoin deleted a comment on Dec 3, 2023
  15. bitcoin deleted a comment on Jan 19, 2024
  16. willcl-ark added the label Brainstorming on Feb 4, 2024
  17. achow101 referenced this in commit c07935bcf5 on Mar 9, 2024
  18. TheCharlatan commented at 1:19 pm on April 11, 2024: contributor
    Updated with initial description of stage 2.
  19. achow101 referenced this in commit 2cedb42a92 on May 10, 2024
  20. ryanofsky referenced this in commit dbb3113082 on May 14, 2024
  21. achow101 referenced this in commit 058af75874 on May 17, 2024
  22. ryanofsky referenced this in commit 94d56b9def on Jul 8, 2024
  23. ryanofsky referenced this in commit 4687832680 on Jul 16, 2024
  24. fanquake referenced this in commit b27ef8ec7f on Jul 31, 2024
  25. denavila referenced this in commit 881e304866 on Sep 4, 2024
  26. achow101 referenced this in commit 6f24662eb9 on Dec 3, 2024
  27. achow101 referenced this in commit ff873a20a7 on Dec 3, 2024
  28. fanquake referenced this in commit df8bf65745 on Jan 16, 2025
  29. TheCharlatan renamed this:
    Libbitcoinkernel Project Tracking
    Bitcoin Kernel Library Project Tracking
    on Mar 6, 2025
  30. medobrens83 commented at 8:48 am on April 18, 2025: none
    👍👍🏻💯
  31. TheCharlatan commented at 10:19 am on July 12, 2025: contributor
    I re-wrote the tracking issue. It now better reflects all the different areas being worked on and also surfaces a lot of the work I have not opened a pull request for.

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-07-24 12:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me