Add libbitcoinkernel example files #33669

pull w0xlt wants to merge 25 commits into bitcoin:master from w0xlt:kernel_examples changing 21 files +5913 −271
  1. w0xlt commented at 2:44 pm on October 21, 2025: contributor

    Many Bitcoin-related libraries (such as libsecp256k1 and other Bitcoin Rust crates) include an examples/ folder that demonstrates proper library usage.

    These examples are especially valuable because they help users learn by example and reduce the need to consult external documentation.

    This PR proposes adopting the same approach for libbitcoinkernel.

    For now, I’ve added two examples for the C library (not the C++ wrapper) to gather feedback. If the approach is well-received, I can add more examples within this PR.

    Built on top of https://github.com/bitcoin/bitcoin/pull/30595

  2. kernel: Introduce initial kernel C header API
    As a first step, implement the equivalent of what was implemented in the
    now deprecated libbitcoinconsensus header. Also add a test binary to
    exercise the header and library.
    
    Unlike the deprecated libbitcoinconsensus the kernel library can now use
    the hardware-accelerated sha256 implementations thanks for its
    statically-initialzed context. The functions kept around for
    backwards-compatibility in the libbitcoinconsensus header are not ported
    over. As a new header, it should not be burdened by previous
    implementations. Also add a new error code for handling invalid flag
    combinations, which would otherwise cause a crash.
    
    The macros used in the new C header were adapted from the libsecp256k1
    header.
    
    To make use of the C header from C++ code, a C++ header is also
    introduced for wrapping the C header. This makes it safer and easier to
    use from C++ code.
    
    Co-authored-by: stickies-v <stickies-v@protonmail.com>
    dc504f57b3
  3. kernel: Add logging to kernel library C header
    Exposing logging in the kernel library allows users to follow
    operations. Users of the C header can use
    `kernel_logging_connection_create(...)` to pass a callback function to
    Bitcoin Core's internal logger. Additionally the level and category can
    be globally configured.
    
    By default, the logger buffers messages until
    `kernel_loggin_connection_create(...)` is called. If the user does not
    want any logging messages, it is recommended that
    `kernel_disable_logging()` is called, which permanently disables the
    logging and any buffering of messages.
    7744997596
  4. kernel: Add kernel library context object
    The context introduced here holds the objects that will be required for
    running validation tasks, such as the chosen chain parameters, callbacks
    for validation events, and interrupt handling. These will be used by the
    chainstate manager introduced in subsequent commits.
    
    This commit also introduces conventions for defining option objects. A
    common pattern throughout the C header will be:
    ```
    options = object_option_create();
    object = object_create(options);
    ```
    This allows for more consistent usage of a "builder pattern" for
    objects where options can be configured independently from
    instantiation.
    dc58ae9fc0
  5. kernel: Add chain params context option to C header
    As a first option, add the chainparams. For now these can only be
    instantiated with default values. In future they may be expanded to take
    their own options for regtest and signet configurations.
    
    This commit also introduces a unique pattern for setting the option
    values when calling the `*_set(...)` function.
    d838a934be
  6. kernel: Add notifications context option to C header
    The notifications are used for notifying on connected blocks and on
    warning and fatal error conditions.
    
    The user of the C header may define callbacks that gets passed to the
    internal notification object in the
    `kernel_NotificationInterfaceCallbacks` struct.
    
    Each of the callbacks take a `user_data` argument that gets populated
    from the `user_data` value in the struct. It can be used to recreate the
    structure containing the callbacks on the user's side, or to give the
    callbacks additional contextual information.
    37a3395d27
  7. kernel: Add chainstate manager object to C header
    This is the main driver class for anything validation related, so expose
    it here.
    
    Creating the chainstate manager options will currently also trigger the
    creation of their respectively configured directories.
    
    The chainstate manager and block manager options are consolidated into a
    single object. The kernel might eventually introduce a separate block
    manager object for the purposes of being a light-weight block store
    reader.
    
    The chainstate manager will associate with the context with which it was
    created for the duration of its lifetime. It is only valid if that
    context remains in memory too.
    
    The tests now also create dedicated temporary directories. This is
    similar to the behaviour in the existing unit test framework.
    
    Co-authored-by: stickies-v <stickies-v@protonmail.com>
    add5904e0a
  8. kernel: Add chainstate manager option for setting worker threads
    Re-use the same pattern used for the context options. This allows users
    to set the number of threads used in the validation thread pool.
    ea01a8caf3
  9. kernel: Add chainstate loading when instantiating a ChainstateManager
    The library will now internally load the chainstate when a new
    ChainstateManager is instantiated.
    
    Options for controlling details of loading the chainstate will be added
    over the next few commits.
    51a24c4004
  10. kernel: Add block validation to C header
    The added function allows the user process and validate a given block
    with the chainstate manager. The *_process_block(...) function does some
    preliminary checks on the block before passing it to
    `ProcessNewBlock(...)`. These are similar to the checks in the
    `submitblock()` rpc.
    
    Richer processing of the block validation result will be made available
    in the following commits through the validation interface.
    
    The commits also adds a utility for deserializing a `CBlock`
    (`kernel_block_create()`) that may then be passed to the library for
    processing.
    
    The tests exercise the function for both mainnet and regtest. The
    commit also adds the data of 206 regtest blocks (some blocks also
    contain transactions).
    2707fc515c
  11. kernel: Add options for reindexing in C header
    Adds options for wiping the chainstate and block tree indexes to the
    chainstate load options. In combination and once the
    `*_import_blocks(...)` function is added in a later commit, this
    triggers a reindex. For now, it just wipes the existing data.
    e788b3ba06
  12. kernel: Add chainstate load options for in-memory dbs in C header
    This allows a user to run the kernel without creating on-disk files for
    the block tree and chainstate indexes. This is potentially useful in
    scenarios where the user needs to do some ephemeral validation
    operations.
    
    One specific use case is when linearizing the blocks on disk. The block
    files store blocks out of order, so a program may utilize the library
    and its header to read the blocks with one chainstate manager, and then
    write them back in order, and without orphans, with another chainstate
    maanger. To save disk resources and if the indexes are not required once
    done, it may be beneficial to keep the indexes in memory for the
    chainstate manager that writes the blocks back again.
    c29a6b87cc
  13. kernel: Add import blocks function to C header
    Add `btck_import_blocks` to import block data and rebuild indexes. The
    function can either reindex all existing block files if the indexes were
    previously wiped through the chainstate load options, or import blocks
    from a single specified file path.
    fee8f6ff38
  14. kernel: Add interrupt function to C header
    Calling interrupt can halt long-running functions associated with
    objects that were created through the passed-in context.
    38a990dd48
  15. kernel: Add validation interface to C header
    This adds the infrastructure required to process validation events. For
    now the external validation interface only has support for the
    `BlockChecked` , `NewPoWValidBlock`, `BlockConnected`, and
    `BlockDisconnected` callback. Support for the other internal
    validation interface methods can be added in the future.
    
    The validation interface follows an architecture for defining its
    callbacks and ownership that is similar to the notifications.
    
    The task runner is created internally with a context, which itself
    internally creates a unique ValidationSignals object. When the user
    creates a new chainstate manager the validation signals are internally
    passed to the chainstate manager through the context.
    
    A validation interface can register for validation events with a
    context. Internally the passed in validation interface is registerd with
    the validation signals of a context.
    
    The callbacks block any further validation execution when they are
    called. It is up to the user to either multiplex them, or use them
    otherwise in a multithreaded mechanism to make processing the validation
    events non-blocking.
    
    I.e. for a synchronous mechanism, the user executes instructions
    directly at the end of the callback function:
    
    ```mermaid
    sequenceDiagram
        participant V as Validation
        participant C as Callback
        V->>C: Call callback
        Note over C: Process event (blocks)
        C-->>V: Return
        Note over V: Validation resumes
    
    ```
    
    To avoid blocking, the user can submit the data to e.g. a worker thread
    or event manager, so processing happens asynchronously:
    
    ```mermaid
    sequenceDiagram
        participant V as Validation
        participant C as Callback
        participant W as Worker Thread
        V->>C: Call callback
        C->>W: Submit to worker thread
        C-->>V: Return immediately
        Note over V: Validation continues
        Note over W: Process event async
    ```
    38e5c526a8
  16. kernel: Add functions for the block validation state to C header
    These allow for the interpretation of the data in a `BlockChecked`
    validation interface callback. The validation state passed through
    `BlockChecked` is the source of truth for the validity of a block (the
    mode). It is
    also useful to get richer information in case a block failed to
    validate (the result).
    5ed5a9724f
  17. kernel: Add function for copying block data to C header
    This adds a function for streaming bytes into a user-owned data
    structure.
    
    Use it in the tests for verifying the implementation of the validation
    interface's `BlockChecked` method.
    51634ca37a
  18. kernel: Add functions to read block from disk to C header
    This adds functions for reading a block from disk with a retrieved block
    tree entry. External services that wish to build their own index, or
    analyze blocks can use this to retrieve block data.
    
    The block tree can now be traversed from the tip backwards. This is
    guaranteed to work, since the chainstate maintains an internal block
    tree index in memory and every block (besides the genesis) has an
    ancestor.
    
    The user can use this function to iterate through all blocks in the
    chain (starting from the tip). The tip is retrieved from a separate
    `Chain` object, which allows distinguishing whether entries are
    currently in the best chain. Once the block tree entry for the genesis
    block is reached a nullptr is returned if the user attempts to get the
    previous entry.
    e080e80da7
  19. kernel: Add function to read block undo data from disk to C header
    This adds functions for reading the undo data from disk with a retrieved
    block tree entry. The undo data of a block contains all the spent
    script pubkeys of all the transactions in a block. For ease of
    understanding the undo data is renamed to spent outputs with seperate
    data structures exposed for a block's and a transaction's spent outputs.
    
    In normal operations undo data is used during re-orgs. This data might
    also be useful for building external indexes, or to scan for silent
    payment transactions.
    
    Internally the block undo data contains a vector of transaction undo
    data which contains a vector of the coins consumed. The coins are all
    int the order of the transaction inputs of the consuming transactions.
    Each coin can be used to retrieve a transaction output and in turn a
    script pubkey and amount.
    
    This translates to the three-level hierarchy the api provides: Block
    spent outputs contain transaction spent outputs, which contain
    individual coins. Each coin includes the associated output, the height
    of the block is contained in, and whether it is from a coinbase
    transaction.
    43855d656e
  20. kernel: Add block hash type and block tree utility functions to C header
    Introduce btck_BlockHash as a type-safe identifier for a block. Adds
    functions to retrieve block tree entries by hash or height, get block
    hashes and heights from entries. access the genesis block, and check if
    blocks are in the active chain.
    64a4564ccb
  21. Kernel: Add functions for working with outpoints
    This introduces the transaction outpoint, input and id types. This now
    allows a user to retrieve a transaction output from a prior transaction
    that a transaction outpoint is pointing to.
    
    This is exercised in the tests by verifying the script of every
    transaction in the test chain.
    7cf58f8c4c
  22. kernel: Add pure kernel bitcoin-chainstate
    Re-write the bitcoin-chainstate utility by only using the kernel C++ API
    header instead of internal Bitcoin Core code.
    57b20b5a4d
  23. kernel: Allowing reducing exports
    Now that an API has been defined, remove the override for symbol
    visibility of the library. It is now possible to build the library with
    reduced exports.
    06ee1c7336
  24. kernel: Add Purpose section to header documentation 670eb49289
  25. kernel: Fix bitcoin-chainstate for windows
    And turn it on in the CI.
    a755e00a13
  26. kernel: Add example files d0a0d676fc
  27. DrahtBot commented at 2:44 pm on October 21, 2025: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33669.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    Concept ACK TheCharlatan, stickies-v

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #33680 (validation: do not wipe utxo cache for stats/scans/snapshots by l0rinc)
    • #33455 (CPack by purpleKarrot)
    • #33247 (build: Remove CMAKE_SKIP_BUILD_RPATH and SKIP_BUILD_RPATH settings by 151henry151)
    • #33191 (net: Provide block templates to peers on request by ajtowns)
    • #32953 ([POC] ci: Skip compilation when running static code analysis by hebasto)
    • #32427 ((RFC) kernel: Replace leveldb-based BlockTreeDB with flat-file based store by TheCharlatan)
    • #32380 (Modernize use of UTF-8 in Windows code by hebasto)
    • #31507 (build: Use clang-cl to build on Windows natively by hebasto)
    • #31382 (kernel: Flush in ChainstateManager destructor by TheCharlatan)
    • #30595 (kernel: Introduce C header API by TheCharlatan)
    • #30437 (ipc: add bitcoin-mine test program by ryanofsky)
    • #30342 (kernel, logging: Pass Logger instances to kernel objects by ryanofsky)
    • #30214 (refactor: Improve assumeutxo state representation by ryanofsky)
    • #29700 (kernel, refactor: return error status on all fatal errors by ryanofsky)
    • #29415 (Broadcast own transactions only via short-lived Tor or I2P connections by vasild)
    • #28792 (Embed default ASMap as binary dump header file by fjahr)
    • #26022 (Add util::ResultPtr class by ryanofsky)
    • #25722 (refactor: Use util::Result class for wallet loading by ryanofsky)
    • #25665 (refactor: Add util::Result failure values, multiple error and warning messages by ryanofsky)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  28. w0xlt marked this as a draft on Oct 21, 2025
  29. TheCharlatan commented at 3:27 pm on October 21, 2025: contributor

    Concept ACK

    I have something similar over at http://thecharlatan.ch/kernel-docs/examples.html . Purplelkarrot also suggested to have code examples in the style of https://purplekarrot.github.io/btck/design/naming_conventions.html (at the bottom). The tabbed views between the languages look particularly nice imo. I am not sure if we should host this in this repo though. We discussed having a separate repo (under the bitcoin-core org) to host language bindings, examples, and docs in a coherent fashion.

  30. stickies-v commented at 8:26 am on October 22, 2025: contributor
    Concept ACK, but I agree with @TheCharlatan that this repo is not the most productive place to put them.
  31. w0xlt commented at 11:48 am on October 22, 2025: contributor

    I wasn’t aware of the theCharlatan’s website, but that’s literally the idea behind this PR — the user shouldn’t need to look for any external resources. Also, many other Bitcoin libraries use the examples folder pattern.

    That said, I’m not opposed to having these examples in another place.

  32. DrahtBot added the label Needs rebase on Oct 28, 2025
  33. DrahtBot commented at 11:46 pm on October 28, 2025: contributor
    🐙 This pull request conflicts with the target branch and needs rebase.
  34. w0xlt commented at 0:00 am on November 6, 2025: contributor
    Closing this PR for now. If there’s further interest in adding these examples to this project, we can reopen it.
  35. w0xlt closed this on Nov 6, 2025


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-12-02 21:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me