Add CBOR RPC interface #22866

This issue is a proposal to add a CBOR RPC interface to bitcoin-core. The interface would be in addition to the current JSON-RPC, not a replacement.

The main goal of the proposal is to offer an RPC interface that is more efficient than the current JSON-RPC. Remote clients with constraints on data usage will see the biggest advantage.

CBOR seems like the best choice for a minimal RPC implementation, since it has wide support in industry, and is the de-facto standard for IoT communication protocols like CoAP. Meaning, there is likely to be a large community outside Bitcoin to get support/developers.

Advantages

• RFC standard specification: RFC 8949
• CBOR uses binary representation
  • no need for Base64 or other encoding
  • well-defined data types with minimal encoding overhead
• Small amount of types
• Several existing free open-source libraries to fork
• Many existing implementations are small (~400-750 LoC)
• Decreased data transmission for remote clients

Disadvantages

• Added code to the core implementation
  • larger attack surface
  • maintenance costs
• Added complexity
• Unclear data-size savings versus compressed JSON/Base64

Alternative approaches

External CBOR proxy

One possible alternative, suggested by @cfields, is to write a proxy translating CBOR to JSON.

The proxy could be stand-alone from bitcoin-core, thus removing the disadvantage of added code.

Also, the proxy could be written in a memory-safe language like Rust, decreasing the attack surface.

There would also be disadvantages to a proxy implementation:

• small overhead of translating CBOR to JSON, and passing to the original JSON-RPC
• installation of an additional piece of software

Internal CBOR proxy

Another alternative is to add CBOR support to Univalue, and implement the proxy in bitcoin-core.

This would have the advantage of direct access to RPC internals, and potentially reduce overall code size.

The proxy would listen on a separate port, translate the incoming CBOR to JSON, and pass the JSON to existing RPC interfaces.

Free open-source implementations

Here is a list of some of the better candidates for a bitcoin-core CBOR fork:

• cb0r: https://github.com/quartzjer/cb0r
  • zero-allocation C implementation
• cppbor: https://github.com/rantydave/cppbor
  • C++17 implementation based on std::variant
• cbor11: https://github.com/jakobvarmose/cbor11
  • C++11 implementation
• tinycbor: https://github.com/intel/tinycbor
  • Intel implementation, includes CBOR-JSON translation
  • ironically, largest LoC count of the candidates
• cbor-lite: https://bitbucket.org/isode/cbor-lite
  • C++14 implementation, header-only
  • used in bc-ur

Comparing to JSON-RPC

@laanwj raised the point that compressed JSON may provide similar savings to CBOR. To test whether the savings from CBOR provides significant size reduction, I will implement a small number of RPC calls in CBOR, and compare the uncompressed and compressed sizes against the current JSON encoding.

See also #16719 (gRPC)

Example encoding of createrawtransaction body:

JSON:

"{'txid':'b4cc287e58f87cdae59417329f710f3ecd75a4ee1d2872b7248f50977c8493f3','vout':1,'scriptPubKey':'a914b10c9df5f7edf436c697f02f1efdba4cf399615187','redeemScript':'512103debedc17b3df2badbcdd86d5feb4562b86fe182e5998abd8bcd4f122c6155b1b21027e940bb73ab8732bfdf7f9216ecefca5b94d6df834e77e108f68e66f126044c052ae'}"

CBOR:

b'\xa4dtxid\x98 \x0b\x0c\x02\x07\x05\x0f\x07\r\x0e\t\x01\x03\t\x07\x00\x03\x0c\x07\n\x0e\x01\x02\x07\x0b\x02\x08\x05\t\x07\x08\t\x0fdvout\x01lscriptPubKey\x97\n\x01\x0b\x00\t\x0f\x0f\x0e\x0f\x03\x0c\t\x0f\x02\x01\x0f\x0b\x04\x0f\t\x06\x05\x08lredeemScript\x98G\x05\x02\x00\r\x0b\r\x01\x0b\r\x02\n\x0b\r\x08\r\x0f\x0b\x05\x02\x08\x0f\x01\x02\x05\t\n\r\x0b\r\x0f\x02\x0c\x01\x05\x01\x02\x00\x07\t\x00\x0b\x03\x0b\x07\x02\x0f\x0f\x0f\x02\x06\x0c\x0f\n\x0b\x04\x06\x0f\x03\x0e\x07\x01\x08\x06\x0e\x06\x01\x06\x04\x0c\x05\n'

• JSON length: 308
• CBOR length: 169

CBOR serialization with cbor2 python library

test script:

import cbor2

tx={'txid': 'b4cc287e58f87cdae59417329f710f3ecd75a4ee1d2872b7248f50977c8493f3', 'vout': 1, 'scriptPubKey': 'a914b10c9df5f7edf436c697f02f1efdba4cf399615187', 'redeemScript': '512103debedc17b3df2badbcdd86d5feb4562b86fe182e5998abd8bcd4f122c6155b1b21027e940bb73ab8732bfdf7f9216ecefca5b94d6df834e77e108f68e66f126044c052ae'}
# naive JSON, no compression
txjson="{'txid':'b4cc287e58f87cdae59417329f710f3ecd75a4ee1d2872b7248f50977c8493f3','vout':1,'scriptPubKey':'a914b10c9df5f7edf436c697f02f1efdba4cf399615187','redeemScript':'512103debedc17b3df2badbcdd86d5feb4562b86fe182e5998abd8bcd4f122c6155b1b21027e940bb73ab8732bfdf7f9216ecefca5b94d6df834e77e108f68e66f126044c052ae'}"
# decode hex values to bytes
txid=b"".join([int(x, 16).to_bytes(1, 'little') for x in 'b4cc287e58f87cdae59417329f710f3ecd75a4ee1d2872b7248f50977c8493f3'[::2]])
scriptPubKey=b"".join([int(x, 16).to_bytes(1, 'little') for x in 'a914b10c9df5f7edf436c697f02f1efdba4cf399615187'[::2]])
redeemScript=b"".join([int(x, 16).to_bytes(1, 'little') for x in '512103debedc17b3df2badbcdd86d5feb4562b86fe182e5998abd8bcd4f122c6155b1b21027e940bb73ab8732bfdf7f9216ecefca5b94d6df834e77e108f68e66f126044c052ae'[::2]])
# encode to CBOR binary string
txcbor=cbor2.dumps({"txid": txid, "vout": 1, "scriptPubKey": scriptPubKey, "redeemScript": redeemScript})
json_len = len(txjson)
cbor_len = len(txcbor)

We make extensive use of CBOR in a number of emerging airgapped wallet interoperability specs at Blockchain Commons, and some of our c & c++ code might be helpful.

Why CBOR? As RFC 7049 states:

1. unambiguous encoding of most common data formats from Internet standards
2. code compactness for encoder and decoder
3. no schema description needed
4. reasonably compact serialization
5. applicability to constrained and unconstrained applications
6. good JSON conversion extensibility

We have an overview on how we use CBOR for cryptographic objects (seeds, hdkeys, descriptors, etc.) starting at https://github.com/BlockchainCommons/crypto-commons/blob/master/Docs/ur-overview.md

I’m interested in seeing what emerges from this discussion.

Concept NACK to another format in core, this should be handled as a rpc proxy service.

This could likely be done as a weekend project on top of https://crates.io/crates/bitcoincore-rpc-json-async or https://crates.io/crates/bitcoincore-rpc-json.

Concept NACK to another format in core, this should be handled as a rpc proxy service.

That is listed as one of the alternatives, definitely open to building as an external service on top of other projects. Thanks for the links.

I think a solid external service could also work as a PoC for something that may or may not be integrated into bitcoin-core. A thin proxy that translates in real-time will likely have installing separate software as the biggest overhead.

We make extensive use of CBOR in a number of emerging airgapped wallet interoperability specs at Blockchain Commons, and some of our c & c++ code might be helpful

Thank you for letting me know about your project, I'll definitely take a look at your code!

There doesn't seem to be any forward progress on this feature request. Closing based on staleness.