TheCharlatan
commented at 9:18 am on August 6, 2024:
contributor
This is a first attempt at introducing a C header for the libbitcoinkernel library that may be used by external applications for interfacing with Bitcoin Core’s validation logic. It currently is limited to operations on blocks. This is a conscious choice, since it already offers a lot of powerful functionality, but sits just on the cusp of still being reviewable scope-wise while giving some pointers on how the rest of the API could look like.
The current design was informed by the development of some small tools using the C header:
Next to the C++ header also made available in this pull request, rust bindings are available here: https://github.com/TheCharlatan/rust-bitcoinkernel. The rust bindings include unit and fuzz tests for the API.
The header currently exposes logic for enabling the following functionality:
Feature-parity with the now deprecated libbitcoin-consensus
Optimized sha256 implementations that were not available to previous users of libbitcoin-consensus thanks to a static kernel context
Full support for logging as well as control over categories and severity
Feature parity with the existing experimental bitcoin-chainstate
Traversing the block index as well and using block index entries for reading block and undo data.
Running the chainstate in memory
Reindexing (both full and chainstate-only)
Interrupting long-running functions
The pull request introduces a new kernel-only test binary that purely relies on the kernel C header and the C++ standard library. This is intentionally done to show its capabilities without relying on other code inside the project. This may be relaxed to include some of the existing utilities, or even be merged into the existing test suite.
How can I review this PR?
Scrutinize the commit messages, run the tests, write your own little applications using the library, let your favorite code sanitizer loose on it, hook it up to your fuzzing infrastructure, profile the difference between the existing bitcoin-chainstate and the bitcoin-chainstate introduced here, be nitty on the documentation, police the C interface, opine on your own API design philosophy.
To get a feeling for the API, read through the tests, or one of the examples.
Please try to avoid nits for the tests, these can wait for later and easily be improved over time. Docs exhaustively explaining all the intricacies of the internal goings-on of the library can also be added later.
To configure this PR for making the shared library and the bitcoin-chainstate and test_kernel utilities available:
For external applications to read the block index of an existing Bitcoin Core node, Bitcoin Core needs to shut down first, since leveldb does not support reading across multiple processes. Other than migrating away from leveldb, there does not seem to be a solution for this problem.
The fatal error handling through the notifications is awkward.
Handling shared pointers in the interfaces is unfortunate. They make ownership and freeing of the resources fuzzy and poison the interfaces with additional types and complexity. However, they seem to be an artifact of the current code that interfaces with the validation engine. The validation engine itself does not seem to make extensive use of these shared pointers.
If multiple instances of the same type of objects are used, there is no mechanism for distinguishing the log messages produced by each of them.
The background leveldb compaction thread may not finish in time leading to a non-clean exit. There seems to be nothing we can do about this, outside of patching leveldb.
DrahtBot
commented at 9:18 am on August 6, 2024:
contributor
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.
DrahtBot added the label
Validation
on Aug 6, 2024
TheCharlatan force-pushed
on Aug 6, 2024
DrahtBot added the label
CI failed
on Aug 6, 2024
DrahtBot
commented at 10:42 am on August 6, 2024:
contributor
Make sure to run all tests locally, according to the documentation.
The failure may happen due to a number of reasons, for example:
Possibly due to a silent merge conflict (the changes in this pull request being
incompatible with the current code in the target branch). If so, make sure to rebase on the latest
commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the
affected test.
An intermittent issue.
Leave a comment here, if you need help tracking down a confusing failure.
TheCharlatan force-pushed
on Aug 6, 2024
DrahtBot removed the label
CI failed
on Aug 6, 2024
theuni
commented at 10:14 pm on August 7, 2024:
member
Very cool. Can’t wait to dig in when I have some free time.
ryanofsky
commented at 6:05 pm on August 12, 2024:
contributor
This seems to offer a lot of nice features, but can you explain the tradeoffs of wrapping the C++ interface in C instead of using C++ from rust directly? It seems like having a C middle layer introduces a lot of boilerplate, and I’m wondering if it is really necessary. For example it seems like there is a rust cxx crate (https://docs.rs/cxx/latest/cxx/, https://chatgpt.com/share/dd4dde59-66d6-4486-88a6-2f42144be056) that lets you call C++ directly from Rust and avoid the need for C boilerplate. It looks like https://cppyy.readthedocs.io/en/latest/index.html is an even more full-featured way of calling c++ from python.
Another drawback of going through a C API seems like not just increased boilerplate, but reduced safety. For example, the implementation is using reinterpret_cast everywhere and it seems like the exposed C functions use a kernel_ErrorCode enum type with the union of every possible error type, so callers don’t have a way to know which functions can return which errors.
TheCharlatan
commented at 8:54 am on August 13, 2024:
contributor
Thank you for the questions and kicking this discussion off @ryanofsky! I’ll update the PR description with a better motiviation re. C vs C++ header, but will also try to answer your questions here.
This seems to offer a lot of nice features, but can you explain the tradeoffs of wrapping the C++ interface in C instead of using C++ from rust directly? It seems like having a C middle layer introduces a lot of boilerplate, and I’m wondering if it is really necessary. For example it seems like there is a rust cxx crate (https://docs.rs/cxx/latest/cxx/, https://chatgpt.com/share/dd4dde59-66d6-4486-88a6-2f42144be056) that lets you call C++ directly from Rust and avoid the need for C boilerplate. It looks like https://cppyy.readthedocs.io/en/latest/index.html is an even more full-featured way of calling c++ from python.
It is true that the interoperability between C++ and Rust has become very good. In fact there is someone working on wrapping the entirety of Bitcoin Core in Rust: https://github.com/klebs6/bitcoin-rs.
During the last Core Dev meeting in Berlin I also asked if a C API were desirable in the first place (notes here) during the libbitcoinkernel session. I moved forward with this implementation, because the consensus at the time with many contributors in the room was that it was desirable. The reasons for this as discussed during the session at the meeting can be briefly summarised:
Shipping a shared library with a C++ header is hard
Mature and well-supported tooling for integrating C exists for nearly every popular language.
C offers a reasonably stable ABI
So if we want the broadest possible support, across as many languages as possible with both dynamic and statically compiled libraries, a C header is the go-to option. I’m speculating here, but a C++ header might also make future standard version bumps and adoption of new standard library features harder. If having some trade-offs with compatibility, library portability, and language support is acceptable, a C++ header might be acceptaple though. It would be nice to hear more reviewers give their opinions here.
I’d also like to add that two libraries that we use and depend on in this project, minisketch and zeromq, use the same pattern. They are C++ codebases, that only expose a C API that in both instances can be used with a C++ RAII wrapper. So there is precedent in the free software ecosystem for doing things this way.
The quality of C++ language interop seems to vary a lot between languages. Python and Rust seem to have decent support, ziglang on the other hand has no support for C++ bindings. JVM family languages are a bit hit and miss, and many of the common academic and industrial data analysis languages, like Julia, R, and Matlab have no support for direct C++ bindings. The latter category should not be disregarded as potential future users, since this library might be useful to access Bitcoin Core data for data analysis projects.
Another drawback of going through a C API seems like not just increased boilerplate, but reduced safety. For example, the implementation is using reinterpret_cast everywhere
I feel like the reduced type safety due to casting is bit of a red herring. The type casting can be harder to abuse if you always use a dedicated helper function for interpreting passed in data types (as I believe is implemented here). Casting is also a pattern used in many other projects; both minisketch and libzmq use similar type casts extensively. It should definitely be possible to scrutinize the API in this PR to a point where it offers decent safety to its users as well as contributors to and maintainers of this code base.
The concerns around boilerplate are more serious in my view, but at least with the current internal code and headers I feel like exposing a safe C++ API is not trivial either. The current headers do not lend themselves to it well, for example through tricky locking mechanics, exposing boost types, or confusing lifetimes. There also comes a point where we should probably stop extensively refactoring internal code for the kernel. I’ve heard some voices during the last two Core Dev meetings with concerns that the kernel project might turn the validation code into an extensive forever building site. Having some boilerplate and glue to abstract some the ugliness and make it safe seems like an acceptable solution for this dilemma. If this means boilerplate is required anyway, I would personally prefer a C API.
Some of the boilerplate-y duplicate definitions in the header could be dropped again eventually if some of the enums are moved to C-style enums instead of class enum. As long as they are properly namespaced, I don’t see a big drawback for this. Similarly, some of the structs could be defined in a way where they can be used on both sides using pimpl or similar idioms. All in all, most of these translations seem very straightforward.
It might be interesting to see how some of the RPC methods could be re-implemented using the kernel header. There have been some RPC implementation bugs over the years that were due to unsafe usage of our internal code within the method implementations. Using the kernel header instead might make this safer and reduce boilerplate. To be clear, I am not suggesting replacing the implementations, but separately re-implementing some of them to show where the kernel header might shine.
it seems like the exposed C functions use a kernel_ErrorCode enum type with the union of every possible error type, so callers don’t have a way to know which functions can return which errors.
We have disagreed on the design of this before. If I understood you correctly, consolidating all error codes into a single enumeration was one of the reasons you opened your version for handling fatal errors in the kernel: #29700 as an alternative to my original: #29642. I am still a bit torn by the two approaches. I get that it may be useful to exactly see which errors may be encountered by invoking a certain routine, but at the same time I get the feeling this often ends up splintering the error handling to the point where you end up with a catch all approach after all. I also think that it is nice to have a single, central list for looking up all error codes and defining some routines for handling them in close proximity to their definition. It would be nice to finally hear some more voices besides the two of us discussing this. real-or-random has recently provided some good points on error handling in the libsecp silent payments pr (that I mostly did not adopt in this PR) and argues that most error codes are not useful to the user. As mentioned in the description, error handling is a weak spot of this pull request and I would like to improve it.
ryanofsky
commented at 4:01 pm on August 13, 2024:
contributor
I guess another thing I’d like to know is if this is the initial C API, and the implementation is around 3000 lines, and it doesn’t handle “transactions, block headers, coins cache, utxo set, meta data, and the mempool”, how much bigger do you think it will get if it does cover most of the things you would like it to cover? Like is this 20%, 30%, or 50% of the expected size?
I like the idea of reviewing and merging this PR, and establishing a way to interoperate with rust libraries and external projects. I just think going forward we should not lock ourselves into an approach that requires everything to go through a C interface. As we build on this and add features, we should experiment with other approaches that use C++ directly, especially when it can reduce boilerplate and avoid bugs.
Thanks for pointing to me to the other error handling discussion. I very much agree with the post that says having a single error handling path is highly desirable. I especially agree with this in cases where detailed error messages are still provided (keeping in mind that error handling != error reporting, you can return simple error states with detailed messages or logging). Of course there are places where callers do need to handle separate error cases, especially when there are temporary failures, timeouts, and interruptions, and in these cases functions should return 2 or 3 error states instead of 1. But I don’t think there is a reason in modern application code for functions to be able to return 5, 10, 20, or 50 error states generally. In low-level or very general OS, networking or DBMS code it might make sense, but for application code it seems like a cargo cult programming practice that made IBM service manuals very impressive in the 1980s but does not have a present day rationale. There are special cases, but I don’t think it should be a normal thing for functions to be returning 15 error codes if we are trying to provide a safe and easy to use API.
Again though, if this approach is the easiest way to get cross-language interoperability working right now, I think we should try it. I just think we should be looking for ways to make things simpler and safer going forward.
TheCharlatan
commented at 7:38 pm on August 13, 2024:
contributor
I guess another thing I’d like to know is if this is the initial C API, and the implementation is around 3000 lines, and it doesn’t handle “transactions, block headers, coins cache, utxo set, meta data, and the mempool”, how much bigger do you think it will get if it does cover most of the things you would like it to cover? Like is this 20%, 30%, or 50% of the expected size?
I think a fair comparison would be comparing the amount of code “glue” required, e.g. the size of the bitcoinkernel.cpp file in this pull request. The size of the header is very dependent on the detail of documentation and I think judging it by the amount of test code is also hard. On my branch including iterators for the UTXO set, handling headers, and simple mempool processing, basically all the stuff required to drop-in replace the calls to validation code in net_processing with the C API, is about similar in size: https://github.com/bitcoin/bitcoin/pull/30595/files#diff-cc28221ef8d0c7294dda4e3df9f70bb6c062006b387468380c2c2cc02b6762c3 . The code on that branch is more hacky than the code here, so I would expect a bit less than a doubling in size to get all the features required to run a full node with transaction relay.
In low-level or very general OS, networking or DBMS code it might make sense, but for application code it seems like a cargo cult programming practice that made IBM service manuals very impressive in the 1980s but does not have a present day rationale.
Heh, well put. I think for most functions here it could be feasible to have more concise error codes without too much effort, but I feel like I have to detach from this a bit before being able to come up with an alternative.
ryanofsky
commented at 10:09 pm on August 13, 2024:
contributor
I think for most functions here it could be feasible to have more concise error codes without too much effort, but I feel like I have to detach from this a bit before being able to come up with an alternative.
Thanks, I think I’d need to look at this more to give concrete suggestions, but I’d hope most functions would just return a simple success or failure status, with a descriptive error message in the case of failure. When functions need to return more complicated information or can fail in different ways that callers will want to distinguish, it should be easy to return the relevant information in custom struct or enum types. I think it’s usually better for functions to return simpler custom types than more complicated shared types, because it lets callers know what values functions can return just by looking at their declarations.
hebasto added the label
Needs CMake port
on Aug 16, 2024
TheCharlatan force-pushed
on Aug 28, 2024
TheCharlatan force-pushed
on Aug 29, 2024
hebasto removed the label
Needs CMake port
on Aug 29, 2024
TheCharlatan force-pushed
on Aug 29, 2024
DrahtBot added the label
CI failed
on Aug 30, 2024
DrahtBot removed the label
CI failed
on Aug 30, 2024
TheCharlatan force-pushed
on Sep 1, 2024
TheCharlatan force-pushed
on Sep 1, 2024
TheCharlatan
commented at 8:29 pm on September 1, 2024:
contributor
I think for most functions here it could be feasible to have more concise error codes without too much effort, but I feel like I have to detach from this a bit before being able to come up with an alternative.
Completely got rid of the kernel_Error with the last push. Thanks for laying out your logic ryanofsky, I feel like this is cleaner now. When looking at the Rust wrapper, the code seems much clearer too. Errors are now communicated through nullptr or false values. Where required, so far only for the verification functions, a richer status code is communicated to the developer.
ryanofsky
commented at 8:13 pm on September 2, 2024:
contributor
Thanks for the update. It’s good to drop the error codes so the C API can correspond 1:1 with the C++ API and not be tied to a more old fashioned and cumbersome error handling paradigm (for callers that want to know which errors are possible and not have to code defensively or fall back to failing generically).
I am still -0 on the approach of introducing a C API to begin with, but happy to help review this and get merged and maintain it if other developers think this is the right approach to take (short term or long term). It would be great to have more concept and approach ACKs for this PR particularly from the @theuni who commented earlier and @josibake who seems to have some projects built on this and linked in the PR description.
I think personally, if I wanted to use bitcoin core code from python or rust I would use tools like:
And interoperate with C++ directly, instead of wrapping the C++ interface in a C interface first. Tools like these do not support all C++ types and features, and can make it necessary to selectively wrap more complicated C++ interfaces with simpler C++ interfaces, or even C interfaces, but I don’t think this would be a justification for preemptively requiring every C++ type and function to be wrapped in C before it can be exposed. I just think the resulting boilerplate code:
0kernel_Warning cast_kernel_warning(kernel::Warning warning)
1{
2switch (warning) {
3case kernel::Warning::UNKNOWN_NEW_RULES_ACTIVATED:
4return kernel_Warning::kernel_LARGE_WORK_INVALID_CHAIN;
5case kernel::Warning::LARGE_WORK_INVALID_CHAIN:
6return kernel_Warning::kernel_LARGE_WORK_INVALID_CHAIN;
7 } // no default case, so the compiler can warn about missing cases
8 assert(false);
9}
and duplicative type definitions and documentation:
0/**
1 * A struct for holding the kernel notification callbacks. The user data pointer
2 * may be used to point to user-defined structures to make processing the
3 * notifications easier.
4 */ 5typedefstruct {
6void* user_data; //!< Holds a user-defined opaque structure that is passed to the notification callbacks.
7 kernel_NotifyBlockTip block_tip; //!< The chain's tip was updated to the provided block index.
8 kernel_NotifyHeaderTip header_tip; //!< A new best block header was added.
9 kernel_NotifyProgress progress; //!< Reports on current block synchronization progress.
10 kernel_NotifyWarningSet warning_set; //!< A warning issued by the kernel library during validation.
11 kernel_NotifyWarningUnset warning_unset; //!< A previous condition leading to the issuance of a warning is no longer given.
12 kernel_NotifyFlushError flush_error; //!< An error encountered when flushing data to disk.
13 kernel_NotifyFatalError fatal_error; //!< A un-recoverable system error encountered by the library.
14} kernel_NotificationInterfaceCallbacks;
are fundamentally unnecessary and not worth effort of writing and maintaining when C++ is not a new or unusual language and not meaningfully less accessible or interoperable than C is.
There are legitimate reasons to wrap C++ in C. One reason would be to provide ABI compatibility. Another would be to make code accessible with dlopen/dlsym. But I think even in these cases you would want to wrap C++ in C selectively, or just define an intermediate C interface to pass pointers but use C++ on either side of the interface. I don’t think you would want to drop down to C when not otherwise needed.
This is just to explain my point of view though. Overall I think this is very nice work, and I want to help with it, not hold it up.
ryanofsky
commented at 8:24 pm on September 2, 2024:
contributor
Another idea worth mentioning is that a bitcoin kernel C API could be implemented as a separate C library depending on the C++ library. The new code here does not necessarily need to be part of the main bitcoin core git repository, and it could be in a separate project. A benefit of this approach is it could relieve bitcoin core developers from the responsibility of updating the C API and API documention when they change the C++ code. But a drawback is that C API might not always be up to date with latest version of bitcoin core code and could be broken between releases. Also it might not be as well reviewed or understood and might have more bugs.
josibake
commented at 7:59 am on September 3, 2024:
member
Concept ACK
Also an implicit approach ACK despite not heavily reviewing the code (yet). I have been focusing on using the kernel library in proof of concept applications to get a better sense of how well the library works for downstream users and to hopefully uncover any pain points preemptively. A few of these projects are linked in the PR description.
Regarding a C header vs C++ header, thanks @ryanofsky for taking the time to explain your thought process. I think you raise some excellent points. I’ll try to respond as best I can, despite being slightly out of my depth on this topic 😅
For me, the value of libbitcoinkernel is only fully realised with the broadest possible language support and ease of use for downstream projects. This is why I strongly prefer the C header approach for the following reasons:
Mature tooling for C language bindings
Stable ABI
Well established pattern in other open source projects
If we agree that broad language support is a goal of libbitcoinkernel, highlighting languages that do not support C++ bindings is a much more compelling argument for a C header than highlighting languages that do support C++ bindings as an argument for a C++ header.
Regarding some of the mentioned languages/tools which do have C++ language binding support:
Tools like these do not support all C++ types and features, and can make it necessary to selectively wrap more complicated C++ interfaces with simpler C++ interfaces, or even C interfaces
In this example, who is doing the wrapping to be able to use these tools? If it’s us, this seems much more complicated to ship and maintain a mixed wrapper and also feels over engineered to a specific set of tools and languages. It also does nothing for languages that do not support C++ bindings at all. As @TheCharlatan mentioned, languages favoured by academia lack C++ binding support and making libbitcoinkernel useful for academic research is a particularly important use case of libbitcoinkernel for me.
If we are exposing just a C++ header and expecting the downstream user to wrap selective parts in C interfaces to use libbitcoinkernel, we’ve eroded a fundamental value proposition of libbitcoinkernel, in my opinion. Namely, we want to provide a safe to use consensus library for users and minimise the risk of downstream projects introducing consensus bugs. Requiring downstream projects to write their own C++/C interfaces to be able to use kernel means that a) they just won’t use libbitcoinkernel or b) will introduce bugs when writing these wrappers. Said differently, if boilerplate will be needed for broad language support, I would prefer we focus our energy on writing and reviewing boilerplate code that ensures the usefulness of the library for the broadest possible user base, instead of requiring a subset of users to each write their own boilerplate without any review from us.
DrahtBot added the label
Needs rebase
on Sep 3, 2024
TheCharlatan force-pushed
on Sep 4, 2024
DrahtBot removed the label
Needs rebase
on Sep 4, 2024
DrahtBot added the label
Needs rebase
on Sep 12, 2024
TheCharlatan force-pushed
on Sep 12, 2024
TheCharlatan
commented at 9:19 pm on September 12, 2024:
contributor
Rebased.
DrahtBot removed the label
Needs rebase
on Sep 12, 2024
TheCharlatan force-pushed
on Sep 13, 2024
TheCharlatan force-pushed
on Sep 13, 2024
kernel: Introduce initial kernel C header API
As a first step, implement the equivalent of what was implemented in the
now deprecated libbitcoinconsensus header. Also add a test binary to
exercise the header and library.
Unlike the deprecated libbitcoinconsensus the kernel library can now use
the hardware-accelerated sha256 implementations thanks for its
statically-initialzed context. The functions kept around for
backwards-compatibility in the libbitcoinconsensus header are not ported
over. As a new header, it should not be burdened by previous
implementations. Also add a new error code for handling invalid flag
combinations, which would otherwise cause a crash.
The macros used in the new C header were adapted from the libsecp256k1
header.
To make use of the C header from C++ code, a C++ header is also
introduced for wrapping the C header. This makes it safer and easier to
use from C++ code.
17b98b95eb
kernel: Add logging to kernel library C header
Exposing logging in the kernel library allows users to follow what is
going on when using it. Users of the C header can use
`kernel_logging_connection_create(...)` to pass a callback function to
Bitcoin Core's internal logger. Additionally the level and severity can
be globally configured.
By default, the logger buffers messages until
`kernel_loggin_connection_create(...)` is called. If the user does not
want any logging messages, it is recommended that
`kernel_disable_logging()` is called, which permanently disables the
logging and any buffering of messages.
ba01b6bd0e
kernel: Add kernel library context object
The context introduced here holds the objects that will be required for
running validation tasks, such as the chosen chain parameters, callbacks
for validation events, and an interrupt utility. These will be used in a
few commits, once the chainstate manager is introduced.
This commit also introduces conventions for defining option objects. A
common pattern throughout the C header will be:
```
options = object_option_create();
object = object_create(options);
```
This allows for more consistent usage of a "builder pattern" for
objects where options can be configured independently from
instantiation.
7cfc892152
kerenl: Add chain params context option to C header
As a first option, add the chainparams. For now these can only be
instantiated with default values. In future they may be expanded to take
their own options for regtest and signet configurations.
This commit also introduces a unique pattern for setting the option
values. Since the number of potential future options that may be
configured is not really bounded, it may hurt discoverability in the
header to just add a function for each of them. Instead add a single
function for setting the options of an option object, in this case
`kernel_context_options_set(...)`, that takes a unique identifier and a
void* value as arguments. The user passes the option value to be
configured as a void pointer alongside its identifier. Valid option
values are tagged internally in their struct and checked if their value
matches with the selected identifier when calling the `*_set(...)`
function.
d71bbbab94
kernel: Add notifications context option to C header
The notifications are used for notifying on connected blocks and on
warning and fatal error conditions.
The user of the C header may define callbacks that gets passed to the
internal notification object in the
`kernel_NotificationInterfaceCallbacks` struct. Each of the callbacks
take a `user_data` argument that gets populated from the `user_data`
value in the struct. It can be used to recreate the structure containing
the callbacks on the user's side, or to give the callbacks additional
contextual information.
3fb9cec5d4
kernel: Add chainstate manager object to C header
This is the main driver class for anything validation related, so expose
it here.
Creating the chainstate manager and block manager options will currently
also trigger the creation of their respectively configured directories.
The chainstate manager and block manager options were not consolidated
into a single object, since the kernel might eventually introduce a
block manager object for the purposes of being a light-weight block
store reader.
The chainstate manager will associate with the context with which it was
created for the duration of its lifetime. It is only valid if that
context remains in memory too.
The tests now also create dedicated temporary directories. This is
similar to the behaviour in the existing unit test framework.
7ed0a89a3e
Kernel: Add chainstate loading to kernel C header
The `kernel_chainstate_manager_load_chainstate(...)` function is the
final step required to prepare the chainstate manager for future tasks.
Its main responsibility is loading the coins and block tree indexes.
Though its `context` argument is not strictly required this was added to
ensure that the context remains in memory for this operation. This
pattern of a "dummy" context will be re-used for functions introduced in
later commits.
The chainstate load options will be populated over the next few commits.
f511ce1489
kernel: Add block validation to C header
The added function allows the user process and validate a given block
with the chainstate manager. The *_process_block(...) function does some
preliminary checks on the block before passing it to
`ProcessNewBlock(...)`. These are similar to the checks in the
`submitblock()` rpc.
Richer processing of the block validation result will be made available
in the following commits through the validation interface.
The commits also adds a utility for serializing a `CBlock`
(`kernel_block_create()`) that may then be passed to the library for
processing.
The tests exercise the function for both mainnet and regtest. The
commit also adds the data of 206 regtest blocks (some blocks also
contain transactions).
2e28c5516d
kernel: Add options for reindexing in C header
Adds options for wiping the chainstate and block tree indexes to the
chainstate load options. In combination and once the
`*_import_blocks(...)` function is added in a later commit, this
triggers a reindex. For now, it just wipes the existing data.
7de1202a01
kernel: Add chainstate load options for in-memory dbs in C header
This allows a user to run the kernel without creating on-disk files for
the block tree and chainstate indexes. This is potentially useful in
scenarios where the user needs to do some ephemeral validation
operations.
One specific use case is when linearizing the blocks on disk. The block
files store blocks out of order, so a program may utilize the library
and its header to read the blocks with one chainstate manager, and then
write them back in order, and without orphans, with another chainstate
maanger. To save disk resources and if the indexes are not required once
done, it may be beneficial to keep the indexes in memory for the
chainstate manager that writes the blocks back again.
28dc294a13
kernel: Add import blocks function to C header
The `kernel_import_blocks` function is used to both trigger a reindex,
if the indexes were previously wiped through the chainstate load
options, or import the block data of a single block file.
The behaviour of the import can be verified through the test logs.
d96ffb9165
kernel: Add interrupt function to C header
Calling interrupt can halt long-running functions associated with
objects that were created through the passed-in context.
b96eb0c49c
kernel: Add validation interface and task runner to C header
This adds the infrastructure required to process validation events. For
now the external validation interface only has support for the
`BlockChecked` callback, but support for the other internal validation
interface methods can be added in the future.
Both the validation interface and the task runner follow a architecture
for defining their callbacks and ownership that is similar to the
notifications.
The task runner is registered with a context, which internally creates a
unique ValidationSignals object. When the user creates a new chainstate
manager the validation signals are internally passed to the chainstate
manager through the context.
The user has great flexibility when defining the callbacks of the task
runner, allowing for immediate, asynchronous, or threaded processing of
the inserted validation events. A validation event has to be processed
through the `kernel_execute_event_and_destroy` function.
A validation interface can register for validation events with a
context. Internally the passed in validation interface is registerd with
the validation signals of a context.
The BlockChecked callback introduces a seperate type for a non-owned
block. Since a library-internal object owns this data, the user needs to
be explicitly prevented from deleting it. In a later commit a utility
will be added to copy its data.
1ad10251b4
kernel: Add functions for the block validation state to C header
These allow for the interpretation of the data in a `BlockChecked`
validation interface callback. This is useful to get richer information
in case a block failed to validate.
8827ebd5ea
kernel: Add function for copying block data to C header
This adds functions for copying serialized block data into a user-owned
variable-sized byte array.
Use it in the tests for verifying the implementation of the validation
interface's `BlockChecked` method.
440d7a7a86
kernel: Add functions to read block from disk to C header
This adds functions for reading a block from disk with a retrieved block
index entry. External services that wish to build their own index, or
analyze blocks can use this to retrieve block data.
The block index can now be traversed from the tip backwards. This is
guaranteed to work, since the chainstate maintains an internal block
tree index in memory and every block (besides the genesis) has an
ancestor.
The user can use this function to iterate through all blocks in the
chain (starting from the tip). Once the block index entry for the
genesis block is reached a nullptr is returned if the user attempts to
get the previous entry.
62b2926c0a
kernel: Add function to read block undo data from disk to C header
This adds functions for reading the undo data from disk with a retrieved
block index entry. The undo data of a block contains all the spent
script pubkeys of all the transactions in a block.
In normal operations undo data is used during re-orgs. This data might
also be useful for building external indexes, or to scan for silent
payment transactions.
Internally the block undo data contains a vector of transaction undo
data which contains a vector of the spent outputs. For this reason, the
`kernel_get_block_undo_size(...)` function is added to the header for
retrieving the size of the transaction undo data vector, as well as the
`kernel_get_transaction_undo_size(...) function for retrieving the size
of each spent outputs vector contained within each transaction undo data
entry. With these two sizes the user can iterate through the undo data
by accessing the transaction outputs by their indeces with
`kernel_get_undo_output_by_index`. If an invalid index is passed in, the
`kernel_ERROR_OUT_OF_BOUNDS` error is returned again.
The returned `kernel_TransactionOutput` is entirely owned by the user
and may be destroyed with the `kernel_transaction_output_destroy(...)`
convenience function.
266d21c9c6
kernel: Add block index utility functions to C header
Adds further functions useful for traversing the block index and
retrieving block information.
The added `kernel_BlockIndexInfo` struct can be expanded in future to
hold richer information about a certain block index.
fcc69abdb9
kernel: Add pure kernel bitcoin-chainstate
This showcases a re-implementation of bitcoin-chainstate only using the
kernel C++ API header.
This is a metadata mirror of the GitHub repository
bitcoin/bitcoin.
This site is not affiliated with GitHub.
Content is generated from a GitHub metadata backup.
generated: 2024-09-16 19:12 UTC
This site is hosted by @0xB10C More mirrored repositories can be found on mirror.b10c.me
