bitcoin-cli -yaml support #15726

This adds a -yaml option to bitcoin-cli to output RPC responses formatted in YAML instead of JSON, mostly intended to be more human-readable. In particular, it adds _ as a thousands separator both before and after the decimal point, and aligns adjacent numeric fields via the decimal point. Since it's yaml, it uses a bit less syntax than json (ie, fewer quotes, no trailing commas, no brackets).

(Looking for conceptACKs; it treads a cpp file as a header rather than making it a .o and doesn't allow yaml output via the gui debug console which would presumably make sense)

Examples:

$ ./bitcoin-cli -yaml getmempoolinfo
size:            54_616
bytes:       38_839_175
usage:      129_049_712
maxmempool: 500_000_000
mempoolminfee:        0.000_000_50
minrelaytxfee:        0.000_000_50

$ ./bitcoin-cli -yaml getblockchaininfo
chain: "main"
blocks:  570_003
headers: 570_003
bestblockhash: "0000000000000000000b315638f7b8c7373f6e3c5fbfa9414ad3b0772452bde6"
difficulty: 6_379_265_451_411.053
mediantime:     1_554_282_152
verificationprogress:       0.999_998_574_278_052_3
initialblockdownload: false
chainwork: "000000000000000000000000000000000000000005ae43ffb81728a85e61da1c"
size_on_disk: 239_561_677_455
pruned: false
softforks:
  - id: "bip34"
    version: 2
    reject:
      status: true
  - id: "bip66"
    version: 3
    reject:
      status: true
  - id: "bip65"
    version: 4
    reject:
      status: true
bip9_softforks:
  csv:
    status: "active"
    startTime: 1_462_060_800
    timeout:   1_493_596_800
    since:           419_328
  segwit:
    status: "active"
    startTime: 1_479_168_000
    timeout:   1_510_704_000
    since:           481_824
warnings: "This is a pre-release test build - use at your own risk - do not use for mining or merchant applications"

$ ./bitcoin-cli -yaml getpeerinfo | sed 's/^/    /' | head -n80
- id: 1
  addr: "XXX"
  addrlocal: "XXX"
  addrbind: "XXX"
  services: "000000000000040d"
  relaytxes: true
  lastsend: 1_554_284_532
  lastrecv: 1_554_284_549
  bytessent:  103_738_305
  bytesrecv:  191_637_234
  conntime: 1_553_227_536
  timeoffset:          -2
  pingtime:             0.194_575
  minping:              0.190_53
  version:         70_015
  subver: "/Satoshi:0.17.99/"
  inbound: false
  addnode: false
  startingheight: 568_231
  banscore:             0
  synced_headers: 570_000
  synced_blocks:  570_000
  inflight: []
  whitelisted: false
  minfeefilter: 0.000_010_00
  bytessent_per_msg:
    addr:       324_110
    blocktxn:    27_148
    cmpctblock: 668_185
    feefilter:       32
    getaddr:         24
    getdata:  9_814_913
    getheaders:   1_053
    headers:    177_974
    inv:     84_986_022
    notfound:   194_280
    ping:       281_728
    pong:       281_568
    sendcmpct:       66
    sendheaders:     24
    tx:       6_981_027
    verack:          24
    version:        127

  bytesrecv_per_msg:
    addr:     345_107
    feefilter:     32
    getblocktxn:  198
    getdata:  878_829
    getheaders: 1_053
    headers:   26_924
    inv:   83_608_254
    notfound:   5_900
    ping:     281_568
    pong:     281_728
    sendcmpct:    198
    sendheaders:   24
    tx:   106_207_268
    verack:        24
    version:      127

$ ./bitcoin-cli -yaml getrawmempool true
"881da193e4c102c6adbfe7b186799f967ed4ac78945e5944e9e86f0ca92e1adc":
  fees:
    base:       0.000_136_47
    modified:   0.000_136_47
    ancestor:   0.000_136_47
    descendant: 0.000_136_47
  size:              200
  fee:                 0.000_136_47
  modifiedfee:         0.000_136_47
  time:    1_554_275_223
  height:        569_988
  descendantcount:     1
  descendantsize:    200
  descendantfees: 13_647
  ancestorcount:       1
  ancestorsize:      200
  ancestorfees:   13_647
  wtxid: "150e1a6f6345f053a6b4e9997588c1c706b7a8b72c84320c53af88341f59730c"
  depends: []
  spentby: []
  "bip125-replaceable": false

cc61f6cd1d160081e6ea30ed431f1dced2f7acf891356015ef531e5dad3bcf29:
  fees:
    base:       0.000_016_80
    modified:   0.000_016_80
    ancestor:   0.000_446_40
    descendant: 0.000_016_80
  size:             168
  fee:                0.000_016_80
  modifiedfee:        0.000_016_80
  time:   1_554_202_301
  height:       569_869
  descendantcount:    1
  descendantsize:   168
  descendantfees: 1_680
  ancestorcount:     19
  ancestorsize:   4_461
  ancestorfees:  44_640
  wtxid: "15151f93c288aa35e13b1b5fb3f94994f054c9cc8c9f16db2519b8c258dbce0e"
  depends:
    - "d956c67976dad97365fb02ccb9032d6c8ca76093c1230b2ce908d4bd4503b063"
  spentby: []
  "bip125-replaceable": false

Concept ACK

Very nice usability improvement! Thanks!

<!--e57a25ab6845829454e8d69fc972939a-->

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

<!--174a7506f384e20aa4161008e828411d-->

Conflicts

Reviewers, this pull request conflicts with the following ones:

• #13716 (bitcoin-cli: -stdinwalletpassphrase and non-echo stdin passwords by kallewoof)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

JSON output is already non minified, which helps readability. Compared to JSON, YAML doesn't reduce the "syntax" that much IMO. But if you really prefer yaml then you can pipe the json to a converter:

./src/bitcoin-cli -regtest getblockchaininfo | json-yaml
chain: regtest
blocks: 0
headers: 0
bestblockhash: 0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206
difficulty: 4.656542373906925e-10
mediantime: 1296688602
verificationprogress: 1
initialblockdownload: true
chainwork: 0000000000000000000000000000000000000000000000000000000000000002
size_on_disk: 293
pruned: false
softforks:
- id: bip34
  version: 2
  reject:
    status: false
- id: bip66
  version: 3
  reject:
    status: false
- id: bip65
  version: 4
  reject:
    status: false
bip9_softforks:
  csv:
    status: defined
    startTime: 0
    timeout: 9223372036854775807
    since: 0
  segwit:
    status: active
    startTime: -1
    timeout: 9223372036854775807
    since: 0
warnings: This is a pre-release test build - use at your own risk - do not use for
  mining or merchant applications

I'm -0 on the feature.

I think that the underscore notation for integers is not part of the last YAML specification. It is part of the YAML version 1.1 specification, but there is no mention of it in the YAML version 1.2 specification.

utACK -- main comment is that this is not bitcoin-specific, and makes the most sense as part of https://github.com/jgarzik/univalue

There, the integration would be as a compiled module, moving code out of the header file and into an optionally-linked separate module.

I don't see a reason to add another output format to the codebase. Isn't that something that can be handled on shell level (JSON to YAML conversion via pipe) or in the consuming app directly?

I think that the underscore notation for integers is not part of the last YAML specification. It is part of the YAML version 1.1 specification, but there is no mention of it in the YAML version 1.2 specification.

I read it that way too, but looking at other implementations and validators, it seems to be at least treated as ambiguous. yaml 1.3, meanwhile, is apparently going to drop all the fancy semantics and just have the same scalar formats that json allows, per https://github.com/yaml/yaml-spec/wiki/RFC-005 anyway. So probably not workable.

An alternative wold be TOML, which also supports separators in numbers, and also supports native timestamps, which could give something like:

chain = "main"
blocks = 571_031
headers = 571_031
bestblockhash = "000000000000000000265087c7e78e246f80c2cc4c4cb0b5fd1d5ab7649c9549"
difficulty = 6_393_023_717_201.863_281_25
mediantime = 2019-04-10T12:34:10Z
verificationprogress = 0.99998781
initialblockdownload = false
chainwork = "000000000000000000000000000000000000000005c596c2165719130b34d068"
size_on_disk = 240_976_533_652
pruned = false
warnings = "This is a pre-release test build - use at your own risk - do not use for mining or merchant applications"

[[softforks]]
id = "bip34"
version = 2
reject.status = true

[[softforks]]
id = "bip66"
version = 3
reject.status = true

[[softforks]]
id = "bip65"
version = 4
reject.status = true

[bip9_softforks.csv]
status = "active"
startTime = 2016-05-01T00:00:00Z
timeout = 2017-05-01T00:00:00Z
since = 419_328

[bip9_softforks.segwit]
status = "active"
startTime = 2016-11-15T00:00:00Z
timeout = 2017-11-15T00:00:00Z
since = 481_824

Disadvantages compared to json/yaml include that you can't have scalars after a dict (dicts have [dictname] followed by key=value entries like ini files, so there's no "end" to them, just the start of some other dict), and arrays are weird and not very intuitive (see the [[softforks]] array of dicts). As a result, serialisation/deserialisation seems a bit hairy. OTOH, it's pretty compatible with ini files, so we could maybe use it for bitcoin.conf and/or bitcoin-rw.conf?

Closing this -- doesn't seem like YAML will keep supporting this syntax, and TOML doesn't seem enough of a win.