RFC: Formal description of the RPC API

maflcko commented at 11:45 am on April 19, 2024: member

Currently (all?) clients manually implement the RPC API, which is problematic, because:

It leads to accidental implementation bugs, such as unit mistakes (s/vB vs BTC/kvB)
It is hard to maintain, when the API changes
It requires effort to implement in a new programming language

So it could make sense to have a formal specification.

Please leave a comment if there are additional aspects to consider here, or if you have ideas for a solution.

One solution for a formal description could be OpenAPI. Generators exist, such as (random example) https://github.com/microsoft/kiota?tab=readme-ov-file#supported-languages

maflcko added the label Brainstorming on Apr 19, 2024

maflcko added the label RPC/REST/ZMQ on Apr 19, 2024

nflatrea commented at 12:08 pm on April 19, 2024: none

Don’t we have plethora of well documented guides and specifications on the JSON-RPC api ?

https://developer.bitcoin.org/reference/rpc/ https://wbnns.github.io/bitcoin-dev-docs/reference/rpc/index.html https://github.com/bitcoin/bitcoin/blob/master/doc/JSON-RPC-interface.md

https://en.bitcoin.it/wiki/API_reference_%28JSON-RPC%29

Considering your need for a state-of-the-art framework / wrapper, you can always find a suitable project on Github, You have, for example, the btcsuite project allowing Go clients building on bitcoin easily. For JSON-RPC wrappers you have https://github.com/btcsuite/btcd/tree/master/rpcclient and the same on C/C++, Rust, Python, Ruby etc..

But it might be interesting to maintain a cross-language suite under the bitcoin core account authority

maflcko commented at 12:34 pm on April 19, 2024: member

Don’t we have plethora of well documented guides and specifications on the JSON-RPC api ?

No. I’d highly doubt that there is a better source of the documentation and specification than Bitcoin Core itself. (If there is, then that is a bug in Bitcoin Core, which should be fixed)

https://developer.bitcoin.org/reference/rpc/

This example has many issues:

It is not machine readable
It does not indicate the version of Bitcoin Core that is documented
It is wrong and outdated. For example https://developer.bitcoin.org/reference/rpc/getrawtransaction.html#argument-2-verbose is wrong. The type should be int, not a bool. Also it is missing verbose=3.
It has all the problems that I mentioned in the original post.

https://wbnns.github.io/bitcoin-dev-docs/reference/rpc/index.html

This is the same as above, just rendered differently.

https://github.com/bitcoin/bitcoin/blob/master/doc/JSON-RPC-interface.md

I am aware of the file, because I initially wrote it. It describes the concept of the interface, but it is not a specification.

https://en.bitcoin.it/wiki/API_reference_%28JSON-RPC%29

This is a wiki page that lists some manually maintained software projects, so all of them have the issues I mentioned in the original post. I don’t have the time right now to list all the bugs they have, but if you spend enough time you will find that they are either not a specification, or that they are unmaintained for years, or that they have implementation errors.

Edit: If there is a single software project out there, which currently correctly and fully implements the RPC API in a type-safe manner, it would be interesting to see how they approached it. But I doubt there is one?

achow101 commented at 2:13 pm on April 19, 2024: member

Note that automatically generated RPC docs are also available for each major version at https://bitcoincore.org/en/doc/

maflcko commented at 2:36 pm on April 19, 2024: member

https://bitcoincore.org/en/doc/

Yes, I am aware. They are based on the help RPC reply. However, they are not machine readable, at least not in a standard format, so I don’t think it helps to solve the issues highlighted in the original post.

sipa commented at 2:38 pm on April 19, 2024: member

I vaguely recall an issue or discussion around machine-readable API specs in the past, but I cannot find it. I believe @laanwj commented on it.

maflcko commented at 2:43 pm on April 19, 2024: member

I can only recall the gRPC (https://github.com/bitcoin/bitcoin/issues/16719) and CBOR (https://github.com/bitcoin/bitcoin/issues/22866) discussions, but those are serialization related, not about a standardized machine-readable interface documentation and specification.

stickies-v commented at 4:06 pm on April 19, 2024: contributor

I think this is an interesting approach to consider. I’m quite fond of OpenAPI but I’m not sure how well it would lend itself to a JSON-RPC, but perhaps it works well enough? There is a sibling project specifically focused on JSON-RPC (https://open-rpc.org/ ) but it has less tooling and support.

Would be cool if this means we could be spec-driven, build a tool that generates (server) header files from the spec so the RPC method implementations have compile time checks on parameters and responses etc.

nflatrea commented at 7:54 pm on April 20, 2024: none

Don’t we have plethora of well documented guides and specifications on the JSON-RPC api ?

No. I’d highly doubt that there is a better source of the documentation and specification than Bitcoin Core itself. (If there is, then that is a bug in Bitcoin Core, which should be fixed)

https://developer.bitcoin.org/reference/rpc/

This example has many issues:
* It is not machine readable

* It does not indicate the version of Bitcoin Core that is documented

* It is wrong and outdated. For example https://developer.bitcoin.org/reference/rpc/getrawtransaction.html#argument-2-verbose is wrong. The type should be `int`, not a `bool`. Also it is missing verbose=3.

* It has all the problems that I mentioned in the original post.
https://wbnns.github.io/bitcoin-dev-docs/reference/rpc/index.html

This is the same as above, just rendered differently.

https://github.com/bitcoin/bitcoin/blob/master/doc/JSON-RPC-interface.md

I am aware of the file, because I initially wrote it. It describes the concept of the interface, but it is not a specification.

https://en.bitcoin.it/wiki/API_reference_%28JSON-RPC%29

This is a wiki page that lists some manually maintained software projects, so all of them have the issues I mentioned in the original post. I don’t have the time right now to list all the bugs they have, but if you spend enough time you will find that they are either not a specification, or that they are unmaintained for years, or that they have implementation errors.

Edit: If there is a single software project out there, which currently correctly and fully implements the RPC API in a type-safe manner, it would be interesting to see how they approached it. But I doubt there is one?

I must have failed to understand your question, my apology 🧐 You just raised the fact that we need clear API specifications for core ?

If that so, I could be interesting to build something with OpenRPC close to OpenAPI https://open-rpc.org/

With a little bit of tweaking, we might also be able to use the Swagger UI originaly for REST API

https://petstore.swagger.io/ https://dev.to/vearutop/json-rpc-2-0-with-swagger-ui-2h3g

laanwj commented at 4:52 pm on April 22, 2024: member

I vaguely recall an issue or discussion around machine-readable API specs in the past, but I cannot find it. I believe @laanwj commented on it.

Much of it is already there, just not exposed. The RPCHelpMan/RPCResult structure for help handling was intended to be a start of formalizing the API, and to have some form of introspection. E.g. types are already checked against the spec while testing. It could be extended to include other data that’s needed.

The same data that’s used for rendering help could be returned on the RPC in a raw, machine-readable structured format. Easiest would be to define our own format, but if there is a suitable standard format that will work for JSON-RPC that’s of course preferable. I don’t think there was at the time I looked into it, but who knows.

tdb3 commented at 10:21 pm on April 28, 2024: contributor

Concept ACK. IMHO, a formal specification (especially machine ingestible) would add value for downstream clients.

There is some great info/guidance in https://github.com/bitcoin/bitcoin/blob/master/doc/JSON-RPC-interface.md#versioning https://github.com/bitcoin/bitcoin/blob/master/doc/developer-notes.md#rpc-interface-guidelines

Maybe it’s just a matter of adding content to the RPC interface guidelines section of the developer notes, but I think there is value in defining the approach that should be taken by PR authors proposing RPC changes.

For example, covering:

Simple changes where additional fields are appended (e.g. #29954)
When to employ deprecatedrpc (e.g. #29845 (review))
Naming style for deprecatedrpc=<name>
Approaches to take for significant protocol changes (e.g. #27101), e.g. when to favor backward compatibility over new protocol compliance, etc.)
Additional suggestions?

kilianmh commented at 10:17 pm on August 4, 2024: none

Some time ago, I created an openrpc spec file for bitcoin core which should be mostly complete up to version 0.27. You can find it here: https://codeberg.org/kilianmh/bitcoin-core-openrpc/src/branch/master/openrpc.json If there is interest, I can contribute it via a PR. See #29477

hodlinator commented at 8:57 pm on October 5, 2024: contributor

Having a formal spec, either generated similarly to the current RPC docs (or possibly used to generate stub headers which are then implemented in Bitcoin Core) lowers friction for application developers building on top of Bitcoin Core and reduces temptation to build on top of other projects (including non-Bitcoin ones).

Been digging into OpenRPC a bit. It seems to be used primarily by Ethereum and Ethereum Classic devs, see the usage section: https://open-rpc.org/use

Regardless, I think the idea behind OpenRPC seems fairly sound, since OpenAPI (formerly Swagger) is fairly path-centric, while our JSON-RPC interface only uses one main path + wallet path.

@kilianmh Thanks for bringing awareness of OpenRPC!

While Common Lisp is cool, I’m not sure how easy it would be to get it merged into the Bitcoin Core contrib/ directory, nor the repo for bitcoin.org. Furthermore, it seems like your implementation consists of manually copy-pasting the text from the C++ code into main.lisp unless some wizardry is escaping me. A more automated way would be to do something like the Golang RPC doc generator in my first link above.

How well does OpenRPC support units like “BTC/kvB”? It seems you’ve defined maxfeerate in sendrawtransaction quite loosely in https://codeberg.org/kilianmh/bitcoin-core-openrpc/src/branch/master/src/main.lisp#L1307-L1312 (and you dropped the “v” from “kvB”, which changes meaning).
Conversely, have you noticed anything bitcoind does that goes against the OpenRPC spec?

kilianmh commented at 0:32 am on October 6, 2024: none

Thank you for your comments @hodlinator

1.

A more automated way would be to do something like the Golang RPC doc generator in my first link above.

Indeed, autogeneration is the best way forward. I just did copy paste to showcase what can be done.

2.

Conversely, have you noticed anything bitcoind does that goes against the OpenRPC spec?

You mentioned the main thing that i am not sure of if it is possible:

JSON-RPC interface only uses one main path + wallet path.

This might work with a server array where one server object has the url "/" and the other server object has an url "/wallet/{walletname}/" with runtime expression and a server variables walletname.

3.

How well does OpenRPC support units like "BTC/kvB"?

The JSON types are quite limited (e.g. only integer or floats) We can only document it in the text, or set specific minimum and maximum values with JSON Schema.

hodlinator commented at 10:09 am on October 10, 2024: contributor

One of the OpenRPC examples has:

 0  ...
 1  "components": {
 2    "contentDescriptors": {
 3      "PetId": {
 4        "name": "petId",
 5        "required": true,
 6        "description": "The id of the pet to retrieve",
 7        "schema": {
 8          "$ref": "#/components/schemas/PetId"
 9        }
10      }
11    },
12    "schemas": {
13      "PetId": {
14        "type": "integer",
15        "minimum": 0
16      },
17      ...

So maybe the sendrawtransaction argument… https://github.com/bitcoin/bitcoin/blob/0c2c3bb3f5c6f52c8db625c3edb51409c72c14b0/src/rpc/mempool.cpp#L53-L55 …could be expressed as…

 0  ...
 1  "components": {
 2    "contentDescriptors": {
 3      "MaxFeeRate": {
 4        "name": "maxfeerate",
 5        "required": true,
 6        "description": "Reject transactions whose fee rate is higher than the specified value, expressed in BTC/kvB.\nFee rates larger than 1BTC/kvB are rejected.\nSet to 0 to accept any fee rate.",
 7        "schema": {
 8          "$ref": "#/components/schemas/BTCPerKVB"
 9        }
10      }
11    },
12    "schemas": {
13      "BTCPerKVB": {
14        // JSON does not have a distinct floating point type.
15        // Furthermore RPCArg::Type::AMOUNT can be either float or string, the
16        // latter removing risk of rounding errors, but would probably not allow
17        // the example to have min/max values.
18        "type": "integer",
19        "minimum": 0
20        "maximum": 21000000
21      },
22      ...

Most cases of RPCArg::Type::AMOUNT currently take BTC, but some newer ones take satoshis: https://github.com/bitcoin/bitcoin/blob/0c2c3bb3f5c6f52c8db625c3edb51409c72c14b0/src/wallet/rpc/spend.cpp#L1704-L1705

A) One approach would be to make several distinct types in the C++ code, which could more easily be exported into an OpenRPC spec.

B) Another alternative would be to move towards requiring callers to always pass strings and include the unit in the parameter’s value: “1BTC” “2sat”, or even “1BTC/kvB”. It is very important to get these units right after all. Moving in that direction may decrease the need somewhat for something like OpenRPC.

laanwj commented at 8:59 am on October 17, 2024: member

i’m going to look into creating a JSON schema based specification for the RPC JSON interface. This is the same standard that c-lightning uses, so we could even share some of their code generation tooling (say, for generating Rust RPC bindings automatically, or a JSON-to-protobuf proxy :sweat_smile: ) and documentation generation (manpages, website).

This can, for large part, be exported mechanically from the descriptions in RPCHelpMan, though it will have to be augmented to be complete and precise (different units and such), and to have better documentation.

0xB10C commented at 7:12 am on November 14, 2024: contributor

fwiw: https://x.com/rodarmor/status/1856879301578428826

I did a little hacking on Bitcoin Core and wrote a new RPC command to export JSON describing the entirety of the RPC interface. If you’re writing a RPC client for Core, using this with a code generator is definitely the way. https://github.com/casey/bitcoin/blob/rpc-json/api.json

cc @casey

laanwj commented at 1:23 pm on November 14, 2024: member

oh neat ! i’ve been working on a similar export functionality to get the RPCHelpMan data into json, but he beat me to it. This isn’t the format i was targeting, and i still think it’s better to avoid introducing a bitcoin-core specific description format, if we can, but exporting it in the first place an important step.

casey commented at 7:22 pm on November 14, 2024: contributor

I had no idea this issue existed! Thanks @0xB10C for sharing and for the ping.

My motivation was the current situation with Rust Bitcoin Core JSON RPC client crates, which is not great. There’s one which is mostly complete, rust-bitcoincore-rpc, but which is now unmaintained, and a successor project rust-bitcoind-json-rpc, which is incomplete, but looks like it’s a little more principled, and is being actively developed.

The RPC API has a huge surface area, so I was thinking that trying to extract some description of the API from Core as JSON and using code generation to automate generation of as much of the API as possible might be a good approach.

I was pleased to find that RPCHelpMan has all the structured information you might want, so I wrote the above branch, which adds a new api RPC method.

The API RPC method exports an API description as JSON based on the entries in mapCommand, the dictionary which holds all RPC commands. It introduces a new UniValue ToDescriptionValue() method (so named because it’s very close to ToDescriptionString()) which is implemented for RPCHelpMan, RPCArg, etc, and is called recursively to build up a UniValue which describes the entire API surface area.

I went with an ad-hoc format, mostly out of concerns that the RPC API is quirky, with things like return types differing depending on argument values (like verbosity levels), so I was concerned that the RPC API might not be possible to express in, e.g., JSON Schema.

The structure of the JSON is not ideal, and carries with it many quirks of how the API is internally structured in Core. I wasn’t sure which of these quirks were important, and should be exposed, and which should be papered over, so I erred on the side of caution, and exposed all of those quirks in the generated JSON.

I could definitely use help understanding those quirks.

Those quirks are:

mapCommands is a map<string, vector<CRPCCommand*>>, so a map of strings to lists of commands, so the api.json that I generate also maps names to lists of commands, instead of mapping each name to a single command.

Why is this? Are there actually multiple implementations of the same command? I tested this, and I could be wrong, but I think that when you are not building in test configuration, the map is never populated with more than one command of the same name, but in testing it is (my assert which fired whenever a duplicate command was added fired in tests.).

If multiple implementations aren’t present in a release configuration, I change the generated JSON to be a map of names to single commands, which would be simpler.
RPCArg::m_names has this comment: can contain multiple aliases separated by | for named request arguments. As a result, the generated JSON for an RPCArg has names key, whose value is a list of names. However, I could not find an instance of this being used in the code.

Is there appetite for landing this in core? Some random thoughts:

I think that using JSON Schema would be ideal, assuming that the API can be expressed in JSON Schema. OpenAPI is also appealing, which would allow describing both the types and the list of endpoints, but it does not look to be geared towards RPC-style APIs where all calls are issued against the same path. It expects a REST-style API where each endpoint has a different path.
To keep PRs small and easy to review, it might be a good idea to implement it in layers. I.e., first implement an API RPC call which simply returns a map of RPC methods to JSON Schema definitions for the request and response types, but only define those request and response types as returning object (or some any type). Then, follow-up PRs could add a new layer, for example, defining the names of the request properties, but keeping their types as any, and so forth and so on, until the API is fully defined.

maflcko commented at 7:32 pm on November 14, 2024: member

RPCArg::m_names has this comment: can contain multiple aliases separated by | for named request arguments. As a result, the generated JSON for an RPCArg has names key, whose value is a list of names. However, I could not find an instance of this being used in the code.

Are you sure? I checked for this in your generated json and found it https://github.com/casey/bitcoin/blob/f2f32b6cc44ef88e4c57e5b0a75935aa912e1862/api.json#L6669-L6674

I think that using JSON Schema would be ideal

I agree, for the reasons given in #29912 (comment). If the current specs don’t fit within JSON Schema, it would be good to enumerate the violations, to see if anything can be done about them.

casey commented at 7:47 pm on November 14, 2024: contributor

Are you sure? I checked for this in your generated json and found it https://github.com/casey/bitcoin/blob/f2f32b6cc44ef88e4c57e5b0a75935aa912e1862/api.json#L6669-L6674

Oh nice! I overlooked that, I was looking at the codebase, not the generated schema.

I think that using JSON Schema would be ideal

I agree, for the reasons given in #29912 (comment). If the current specs don’t fit within JSON Schema, it would be good to enumerate the violations, to see if anything can be done about them.

Definitely agreed.

casey commented at 1:11 am on November 17, 2024: contributor

I started working on a branch which uses JSON schema, instead of an ad-hoc format. So far it seems fine! I started with command arguments first, since they’re much simpler than command results.

I tried to keep everything self-contained in src/rpc/schema.{h,cpp} files. If someone winds up using this to codegen support for current versions of core, they may wish to back-port it to older versions of core, and having it mostly be in one place helps with that.

To easily see the output, the generated schema is saved in schema.json.

What would be needed, in addition to cleaning it up and supporting results, to consider merging this into Core?

In particular, I’m curious what kind of tests would be needed. At a minimum, I’d like to have a way to validate the schemas themselves, i.e., that they are at least properly formed JSON Schemas. However, that would not guarantee that the schemas actually correctly describe the JSON RPC API, which is the goal.

The fastest way to add tests for the schemas would be to leverage existing tests which call the JSON RPC API. During testing, requests and responses would be validated against the corresponding schema, and the test would fail if the schemas didn’t match the actual values.

Unfortunately, I can’t find a good-looking C++ JSON Schema library to use for this, but it may be possible to call an external command which does the validation, in which case there are a number of good options available.

Another option is to consider them experimental, and hide the schema command which produces them until there’s more confidence.

cdecker commented at 2:02 pm on December 18, 2024: contributor

i’m going to look into creating a JSON schema based specification for the RPC JSON interface. This is the same standard that c-lightning uses, so we could even share some of their code generation tooling (say, for generating Rust RPC bindings automatically, or a JSON-to-protobuf proxy 😅 ) and documentation generation (manpages, website).

This can, for large part, be exported mechanically from the descriptions in RPCHelpMan, though it will have to be augmented to be complete and precise (different units and such), and to have better documentation.

I just stumbled over this reading the bitcoinoptech mail. I’d be absolutely thrilled if the RPC generation code were to be adopted by Bitcoin Core. As the original author let me add a bit of color: we currently generate two kinds of things in CLN (the wire protocol using a generate-wire.py script, and the RPC starting from the JSON Schema). Since we want to eventually unify the two we went for a source agnostic representation, so we start by reading the schema source (weird CSV files for the wire, and JSON Schema for the RPC), we then lift it into an in-memory model, that we can then enrich with data that isn’t part of the schema (e.g., protobuf field numbering, versioning information such as the version a field was added / removed, …). From that rich in-memory representation we can then generate all other representations, and conversion codes between them.

Here’s a brief summary of generated interfaces:

Rust client library
Python client library
Protobuf and grpc schema generation
Conversion from Rust to grpc
Reverse proxies with and without translation between representations

In addition, since we have funneled the schemas through to the client libraries we can actually verify and enforce schema-adherence while running tests, by just setting a validation flag in the client connection.

Now for what I can offer to do here:

We likely want to split the msggen package out of CLN, put it in a separately controlled repo
CLN and Bitcoin Core would then make use of the library to implement the conversions and representations they need, upstreaming if/when we notice that there is a common pattern.

That would also play to our own needs, since we have msggen-derivations in a number of other projects that currently are dependent on the CLN repo.

Of course, it’s just an offer, and I’m happy if some other solution is better suited for Core :-)

RFC: Formal description of the RPC API #29912