[REST] API versioning #6855

.. API client developers can choose to either:

• develop against the latest one (committing themselves to maintain the application protecting it from eventual API changes that might break their badly designed API client).
• bind to a specific version of the API (which becomes apparent) but only for a limited time

From http://stackoverflow.com/a/398564

Unfortunately there is no standard when it comes to REST API versioning, but the way it's recommended in this Stackoverflow post makes sense to me. @jonasschnelli brought up that since Bitcoin Core itself isn't version 1.0 people don't expect a stable REST API. Hard to say if that is the case, but since it's definitely widely used it don't think we should break backwards compatibility too easily. And those who don't want / need it can use the first option.

Yes, I agree on the versioning. It would be good to mark the old API as deprecated and we can eventually remove it several releases later.

Concept ACK

utACK/Concept ACK [if the REST interface is to continue being part of core] It seems more and more out of scope though.

Concept ACK.

utACK/Concept ACK [if the REST interface is to continue being part of core] It seems more and more out of scope though.

Yes, I definitely think REST should move out of core, and be a wrapper that accesses core through RPC. This is increasingly out of scope.

The discussion regarding REST being in core or not was already back then in the initial PR: #2844

Quoting @jgarzik (https://github.com/bitcoin/bitcoin/pull/2844#issuecomment-22459961):

• Dramatically easier interface for developers and general queries
• Out of the box SSL support, already built into bitcoind
• This pull adds functionality not available via RPC (there is no getrawblock) -- thus the btchttp.py example does not provide what this pull request provides.
• Wider selection of command line tools work out of the box with HTTP REST, versus JSON-RPC
• Nobody will write their own proxy to accomplish this
• A contributed proxy, shipped with bitcoind, might be used -- but it is obviously a fragile, two-process unsupported solution

I don't think anything changed regarding this.

But anyway, if this is what people want there should be a separate Github issue to discuss. (EDIT: or this one #3440)

I think the versioning is needed, independent of REST being part of Core or not. And it's probably less controversial.

[...] I definitely think REST should move out of core [...]

Agree with @laanwj.

[...] and be a wrapper that accesses core through RPC. This is increasingly out of scope.

Disagree to use RPC. Current RPC implementation is inefficiency. If we tend going towards the "wrapper-approach" and supporting one core interface, we should use something more efficient like ZMQ/TCP (with package transmission guarantee). We currently follow the JSON RPC 2.0 standard which is inefficient for some bitcoin datastructures (blocks, large transactions).

Supporting a core access (bidirectional) over ZMQ would also allow to split off the RPC server (which would also be a nice step regarding process separation).

@jonasschnelli I think the end result is that there should only be 1 communication channel (other than the network protocol itself) to communicate with bitcoind. A REST interface would at least have clear semantics and easy plug and play, but its existence along with the RPC server as well is IMHO out of scope.

@dcousens Agree. Currently we support three protocol: ZMQ, JSON RPC, REST. I personally tend to support the most core-ish one, which would be ZMQ. But I think the JSON RPC server is fundamental for the current bitcoin-core user-base. Splitting off the server would therefore be relatively hard to achieve.

IMHO, REST > RPC > ZMQ. Simply because the amount of dependencies (and therefore, underlying code) required grows with each interface:

• REST - HTTP
• RPC - HTTP, RPC protocol
• ZMQ - native ZMQ bindings

As mentioned, the RPC interface is also inefficient due to its reliance on using JSON solely, whereas a REST interface could support whatever is needed. IMHO it should be binary only (discourages public usage).

Disagree to use RPC. Current RPC implementation is inefficiency. If we tend going towards the "wrapper-approach" and supporting one core interface, we should use something more efficient like ZMQ/TCP (with package transmission guarantee).

This discussion is tangential, but you're confounding a few things:

• IP network protocol (TCP, UDP)
• Transport protocol (HTTP, ZMQ)
• Request protocol (JSON)

IMO, JSON over HTTP would be what to use here, it's entrenched and works pretty well. I don't like using ZMQ for anything but notifications.

Current RPC overhead is virtually all caused by application-level issues (such as all-encompassing locking), not the protocol. Switching to a different protocol (at any level) would not change that. Performance also should have improved now that we no longer use boost::spirit_json. libevent http server should be very fast too.

For blocks, because of the amount of data, I agree that the JSON encoding overhead matters. We could think about some alternative path for requesting those from core (e.g. keeping a REST-like stub). Apart from that there is, AFAIK, nothing that returns a lot of data.

Weak NAK - this was always considered and discussed - proposed direction from IRC at the time was to add "/v2/" should any v2 API version appear.

@laanwj I'm not sure I agree. Assuming the wallet moves to a separate process, I think I'd rather keep REST and drop authenticated RPC. The only thing a pure router should need as authenticated interface is things like restart and reload config, for which there are fine other solutions (signals, ...).

@sipa I agree that given that the whole API could be redesigned from scratch, that would have been a better choice. But realistically I don't see JSON RPC going away any time soon, though. The RPC interface is all bitcoind has as a stable interface. Even if the wallet is to move to a separate process (which has been said many times before, but there are no concrete plans by anyone to work on this, so acting based on it makes little sense). Practically this means having to maintain two interfaces for a long time, handle bugs that come up, as well as review changes to them. As REST is growing all the time this feels like terrible scope creep.

@jgarzik I saw it in the PR. The difference is that if we add later later it would be: /rest/tx is v1 /rest/v2/tx is v2 ..

With this PR it would be: /rest/tx being always the latest (v1 atm) /rest/v1/tx is v1 /rest/v2/tx is v2 ..

So there is no way to always use the latest API and have consistent REST URIs with the current implementation. More details in the linked Stackoverflow answer: https://stackoverflow.com/questions/389169/best-practices-for-api-versioning/398564#398564

/rest/tx being always the latest (v1 atm)

Absolute NACK on that, you cant have APIs upgrading themselves to new versions. Consumers explicitly address the version they need. /rest/tx can only become a legacy path to be deprecated, or remain forever an alias for /rest/v1/tx.

Right @btcdrak

/rest/tx is always by definition v1 and that should not auto-upgrade for users - that just means future breakage for users.

Thus "weak NAK" - if people really want the duplicative /v1/ that's fine - but within API versioning needs the current works as long as future versions change the URL.

Do tend to agree with @jgarzik and @btcdrak here . I don't think adding an alias for v1 is desirable. It's kind of too late to change the URL for v1: having people migrate now is a waste of time. And when there is a v2 it's straightforward to add a /v2/ for that.

Agree about that. The current URL should always refer to v1. If we need a v2, a new URL style can be included.

Ok, closing this then

I saw it as giving people a choice, like I wrote in the PR. To have both, consistent URIs and versioning (as described in the link).

Since we talk about developers using this API I'd expect that they know what they decide for.

I'm generally opposed to have multiple interfaces to get the same information, as it creates unclear expectations.

If both / and /v1/ exist, I think it's much less clear what the future for / is.

For now, only one API exists, and there is no reason to assume we'll need a second. If we do need one, we can introduce it as v2, and those who don't know about it can keep using all existing URLs.

If the wallet were ever to be separated from bitcoind, I think switching to the REST only interface would make sense. Until then, I guess we're left supporting both?

The wallet can be separated from bitcoind (--disable-wallet). I call this "router mode", and is the long term direction of bitcoind.

@jgarzik I meant in terms of the underlying implementation, not the command line flag.