RFC: Formal description of the RPC API

maflcko commented at 11:45 am on April 19, 2024: member

Currently (all?) clients manually implement the RPC API, which is problematic, because:

It leads to accidental implementation bugs, such as unit mistakes (s/vB vs BTC/kvB)
It is hard to maintain, when the API changes
It requires effort to implement in a new programming language

So it could make sense to have a formal specification.

Please leave a comment if there are additional aspects to consider here, or if you have ideas for a solution.

One solution for a formal description could be OpenAPI. Generators exist, such as (random example) https://github.com/microsoft/kiota?tab=readme-ov-file#supported-languages

maflcko added the label Brainstorming on Apr 19, 2024

maflcko added the label RPC/REST/ZMQ on Apr 19, 2024

nflatrea commented at 12:08 pm on April 19, 2024: none

Don’t we have plethora of well documented guides and specifications on the JSON-RPC api ?

https://developer.bitcoin.org/reference/rpc/ https://wbnns.github.io/bitcoin-dev-docs/reference/rpc/index.html https://github.com/bitcoin/bitcoin/blob/master/doc/JSON-RPC-interface.md

https://en.bitcoin.it/wiki/API_reference_%28JSON-RPC%29

Considering your need for a state-of-the-art framework / wrapper, you can always find a suitable project on Github, You have, for example, the btcsuite project allowing Go clients building on bitcoin easily. For JSON-RPC wrappers you have https://github.com/btcsuite/btcd/tree/master/rpcclient and the same on C/C++, Rust, Python, Ruby etc..

But it might be interesting to maintain a cross-language suite under the bitcoin core account authority

maflcko commented at 12:34 pm on April 19, 2024: member

Don’t we have plethora of well documented guides and specifications on the JSON-RPC api ?

No. I’d highly doubt that there is a better source of the documentation and specification than Bitcoin Core itself. (If there is, then that is a bug in Bitcoin Core, which should be fixed)

https://developer.bitcoin.org/reference/rpc/

This example has many issues:

It is not machine readable
It does not indicate the version of Bitcoin Core that is documented
It is wrong and outdated. For example https://developer.bitcoin.org/reference/rpc/getrawtransaction.html#argument-2-verbose is wrong. The type should be int, not a bool. Also it is missing verbose=3.
It has all the problems that I mentioned in the original post.

https://wbnns.github.io/bitcoin-dev-docs/reference/rpc/index.html

This is the same as above, just rendered differently.

https://github.com/bitcoin/bitcoin/blob/master/doc/JSON-RPC-interface.md

I am aware of the file, because I initially wrote it. It describes the concept of the interface, but it is not a specification.

https://en.bitcoin.it/wiki/API_reference_%28JSON-RPC%29

This is a wiki page that lists some manually maintained software projects, so all of them have the issues I mentioned in the original post. I don’t have the time right now to list all the bugs they have, but if you spend enough time you will find that they are either not a specification, or that they are unmaintained for years, or that they have implementation errors.

Edit: If there is a single software project out there, which currently correctly and fully implements the RPC API in a type-safe manner, it would be interesting to see how they approached it. But I doubt there is one?

achow101 commented at 2:13 pm on April 19, 2024: member

Note that automatically generated RPC docs are also available for each major version at https://bitcoincore.org/en/doc/

maflcko commented at 2:36 pm on April 19, 2024: member

https://bitcoincore.org/en/doc/

Yes, I am aware. They are based on the help RPC reply. However, they are not machine readable, at least not in a standard format, so I don’t think it helps to solve the issues highlighted in the original post.

sipa commented at 2:38 pm on April 19, 2024: member

I vaguely recall an issue or discussion around machine-readable API specs in the past, but I cannot find it. I believe @laanwj commented on it.

maflcko commented at 2:43 pm on April 19, 2024: member

I can only recall the gRPC (https://github.com/bitcoin/bitcoin/issues/16719) and CBOR (https://github.com/bitcoin/bitcoin/issues/22866) discussions, but those are serialization related, not about a standardized machine-readable interface documentation and specification.

stickies-v commented at 4:06 pm on April 19, 2024: contributor

I think this is an interesting approach to consider. I’m quite fond of OpenAPI but I’m not sure how well it would lend itself to a JSON-RPC, but perhaps it works well enough? There is a sibling project specifically focused on JSON-RPC (https://open-rpc.org/ ) but it has less tooling and support.

Would be cool if this means we could be spec-driven, build a tool that generates (server) header files from the spec so the RPC method implementations have compile time checks on parameters and responses etc.

nflatrea commented at 7:54 pm on April 20, 2024: none

Don’t we have plethora of well documented guides and specifications on the JSON-RPC api ?

No. I’d highly doubt that there is a better source of the documentation and specification than Bitcoin Core itself. (If there is, then that is a bug in Bitcoin Core, which should be fixed)

https://developer.bitcoin.org/reference/rpc/

This example has many issues:
* It is not machine readable

* It does not indicate the version of Bitcoin Core that is documented

* It is wrong and outdated. For example https://developer.bitcoin.org/reference/rpc/getrawtransaction.html#argument-2-verbose is wrong. The type should be `int`, not a `bool`. Also it is missing verbose=3.

* It has all the problems that I mentioned in the original post.
https://wbnns.github.io/bitcoin-dev-docs/reference/rpc/index.html

This is the same as above, just rendered differently.

https://github.com/bitcoin/bitcoin/blob/master/doc/JSON-RPC-interface.md

I am aware of the file, because I initially wrote it. It describes the concept of the interface, but it is not a specification.

https://en.bitcoin.it/wiki/API_reference_%28JSON-RPC%29

This is a wiki page that lists some manually maintained software projects, so all of them have the issues I mentioned in the original post. I don’t have the time right now to list all the bugs they have, but if you spend enough time you will find that they are either not a specification, or that they are unmaintained for years, or that they have implementation errors.

Edit: If there is a single software project out there, which currently correctly and fully implements the RPC API in a type-safe manner, it would be interesting to see how they approached it. But I doubt there is one?

I must have failed to understand your question, my apology 🧐 You just raised the fact that we need clear API specifications for core ?

If that so, I could be interesting to build something with OpenRPC close to OpenAPI https://open-rpc.org/

With a little bit of tweaking, we might also be able to use the Swagger UI originaly for REST API

https://petstore.swagger.io/ https://dev.to/vearutop/json-rpc-2-0-with-swagger-ui-2h3g

laanwj commented at 4:52 pm on April 22, 2024: member

I vaguely recall an issue or discussion around machine-readable API specs in the past, but I cannot find it. I believe @laanwj commented on it.

Much of it is already there, just not exposed. The RPCHelpMan/RPCResult structure for help handling was intended to be a start of formalizing the API, and to have some form of introspection. E.g. types are already checked against the spec while testing. It could be extended to include other data that’s needed.

The same data that’s used for rendering help could be returned on the RPC in a raw, machine-readable structured format. Easiest would be to define our own format, but if there is a suitable standard format that will work for JSON-RPC that’s of course preferable. I don’t think there was at the time I looked into it, but who knows.

tdb3 commented at 10:21 pm on April 28, 2024: contributor

Concept ACK. IMHO, a formal specification (especially machine ingestible) would add value for downstream clients.

There is some great info/guidance in https://github.com/bitcoin/bitcoin/blob/master/doc/JSON-RPC-interface.md#versioning https://github.com/bitcoin/bitcoin/blob/master/doc/developer-notes.md#rpc-interface-guidelines

Maybe it’s just a matter of adding content to the RPC interface guidelines section of the developer notes, but I think there is value in defining the approach that should be taken by PR authors proposing RPC changes.

For example, covering:

Simple changes where additional fields are appended (e.g. #29954)
When to employ deprecatedrpc (e.g. #29845 (review))
Naming style for deprecatedrpc=<name>
Approaches to take for significant protocol changes (e.g. #27101), e.g. when to favor backward compatibility over new protocol compliance, etc.)
Additional suggestions?

kilianmh commented at 10:17 pm on August 4, 2024: none

Some time ago, I created an openrpc spec file for bitcoin core which should be mostly complete up to version 0.27. You can find it here: https://codeberg.org/kilianmh/bitcoin-core-openrpc/src/branch/master/openrpc.json If there is interest, I can contribute it via a PR. See #29477

RFC: Formal description of the RPC API #29912