Add reference documentation for descriptors language #14096

sipa commented at 9:12 PM on August 28, 2018: member

No description provided.

sipa force-pushed on Aug 28, 2018

sipa commented at 9:15 PM on August 28, 2018: member

In case there is another 0.17 RC, this should probably be included (the RPC help still says "documentation at TODO").

sipa added this to the milestone 0.17.0 on Aug 28, 2018

in doc/descriptors.md:43 in f1665448a3 outdated

  38 | +
  39 | +Scripts:
  40 | +- `sh(SCRIPT)` (toplevel only): P2SH embed the argument.
  41 | +- `wsh(SCRIPT)` (not inside another 'wsh'): P2WSH embed the argument.
  42 | +- `pk(PUBKEY)` (anywhere): P2PK output for the given public key.
  43 | +- `pkh(PUBKEY)` (anywhere): P2PKH output for the given public key (use `addr` if you only thos the pubkey hash).

practicalswift commented at 9:18 PM on August 28, 2018:

Typo: "thos" :-)

sipa commented at 3:18 AM on August 29, 2018:

Fixed.

MarcoFalke added the label Needs backport on Aug 28, 2018

MarcoFalke added the label Docs on Aug 28, 2018

in doc/descriptors.md:53 in f1665448a3 outdated

  48 | +- `raw(HEX)` (toplevel only): the script whose hex encoding is HEX.
  49 | +
  50 | +Public keys:
  51 | +- Hex encoded public keys (66 characters starting with `02` or `03`, or 130 characters starting with `04`).
  52 | +  - Inside `wpkh` and `wsh`, only compressed public keys are permitted.
  53 | +- WIF encoded private keys, whose denote their corresponding public keys.

practicalswift commented at 9:20 PM on August 28, 2018:

Should "whose" be "which" here?

sipa commented at 3:18 AM on August 29, 2018:

Fixed.

practicalswift commented at 9:27 PM on August 28, 2018: contributor

FWIW, codespell catched the "thos" typo :-)

See #13954 for codespell integration: Warn (don't fail!) on spelling errors. Fix typos reported by codespell

in doc/descriptors.md:7 in f1665448a3 outdated

   0 | @@ -0,0 +1,95 @@
   1 | +# Support for Output Descriptors in Bitcoin Core
   2 | +
   3 | +Since Bitcoin Core v0.17, there is support for Output Descriptors in the
   4 | +`scantxoutset` RPC call. This is a simple language which can be used to
   5 | +describe collections of output scripts.
   6 | +
   7 | +This document describes the language. For the specfics on usage for scanning

practicalswift commented at 9:30 PM on August 28, 2018:

Typo: "specfics"

sipa commented at 3:18 AM on August 29, 2018:

Fixed.

in doc/descriptors.md:40 in f1665448a3 outdated

  35 | +- `pkh(xpub68Gmy5EdvgibQVfPdqkBBCHxA5htiqg55crXYuXoQRKfDBFA1WEjWgP6LHhwBZeNK1VTsfTFUHCdrfp1bgwQ9xv5ski8PX9rL2dZXvgGDnw/1/0/*)`
  36 | +
  37 | +## Quick reference
  38 | +
  39 | +Scripts:
  40 | +- `sh(SCRIPT)` (toplevel only): P2SH embed the argument.

practicalswift commented at 9:31 PM on August 28, 2018:

Nit: "toplevel" should probably be "top level" (applies throughout), no?

sipa commented at 3:18 AM on August 29, 2018:

Done.

in doc/descriptors.md:74 in f1665448a3 outdated

  69 | +* `sh(wpkh(03fff97bd5755eeea420453a14355235d382f6472f8568a18b2f057a1460297556))` represents a P2SH-P2WPKH output.
  70 | +* `sh(wsh(pkh(02e493dbf1c10d80f3581e4904930b1404cc6c13900ee0758474fa94abe8c4cd13)))` represents a P2SH-P2WSH-P2PKH output.
  71 | +
  72 | +### Multisig
  73 | +
  74 | +Several pieces of software use multisignature scripts based on Bitcoin's OP_CHECKMULTISIG opcode. To support these, we introduce the `multi(k,key_1,key_2,...,key_n)` function. It represents a *k-of-n* multisig policy, where any *k* out of the *n* provided public keys must sign. Examples:

practicalswift commented at 9:33 PM on August 28, 2018:

Nit: "multisignature" should be "multi signature" or "multisig"?

sipa commented at 3:40 AM on August 29, 2018:

Done.

practicalswift commented at 9:35 PM on August 28, 2018: contributor

Excellent documentation! Great work as always!

ACK f1665448a3d6ae244e36a74a311b27db4e4144a5 modulo nits

sipa force-pushed on Aug 29, 2018

practicalswift commented at 7:23 AM on August 29, 2018: contributor

ACK db7ebf0f45a62f685375ce80a0a8ea57420b0ab0

in doc/descriptors.md:44 in db7ebf0f45 outdated

  35 | +- `pkh(xpub68Gmy5EdvgibQVfPdqkBBCHxA5htiqg55crXYuXoQRKfDBFA1WEjWgP6LHhwBZeNK1VTsfTFUHCdrfp1bgwQ9xv5ski8PX9rL2dZXvgGDnw/1/0/*)`
  36 | +
  37 | +## Quick reference
  38 | +
  39 | +Scripts:
  40 | +- `sh(SCRIPT)` (top level only): P2SH embed the argument.

instagibbs commented at 2:10 PM on August 29, 2018:

One explicit example of each of the arguments in caps would help.

sipa commented at 7:31 PM on September 4, 2018:

I've reworded things a bit. I don't think it's necessary to give full examples of everything again (I'm a bit worried that the "features", "examples", "reference", and "explanation" sections are becoming repetitions of the same information over and over again).

sipa commented at 6:20 PM on September 5, 2018:

It's now rewritten by moving all the examples from the explanation section there.

MarcoFalke commented at 2:49 PM on August 29, 2018: member

utACK db7ebf0f45a62f685375ce80a0a8ea57420b0ab0

in doc/descriptors.md:14 in db7ebf0f45 outdated

   9 | +
  10 | +## Features
  11 | +
  12 | +Output descriptors currently support:
  13 | +- Pay-to-pubkey scripts (P2PK), through the `pk` function.
  14 | +- Pay-to-pubkey-hash scripts (P2PK), through the `pkh` function.

flack commented at 8:03 PM on August 29, 2018:

shouldn't this be (P2PKH) ?

sipa commented at 7:29 PM on September 4, 2018:

Fixed.

in doc/descriptors.md:15 in db7ebf0f45 outdated

  10 | +## Features
  11 | +
  12 | +Output descriptors currently support:
  13 | +- Pay-to-pubkey scripts (P2PK), through the `pk` function.
  14 | +- Pay-to-pubkey-hash scripts (P2PK), through the `pkh` function.
  15 | +- Pay-to-witness-pubkey-hash scripts (P2WPK), through the `wpkh` function.

flack commented at 8:04 PM on August 29, 2018:

shouldn't this be (P2WPKH)?

sipa commented at 7:29 PM on September 4, 2018:

Fixed.

in doc/descriptors.md:19 in db7ebf0f45 outdated

  14 | +- Pay-to-pubkey-hash scripts (P2PK), through the `pkh` function.
  15 | +- Pay-to-witness-pubkey-hash scripts (P2WPK), through the `wpkh` function.
  16 | +- Pay-to-script-hash scripts (P2SH), through the `sh` function.
  17 | +- Pay-to-witness-script-hash scripts (P2WSH), through the `wsh` function.
  18 | +- Multisig scripts, through the `multi` function.
  19 | +- Any type of supported address (P2PKH, P2SH, Bech32), through the `addr` function.

instagibbs commented at 8:06 PM on August 29, 2018:

add P2PK as well(if actually supported?)

sipa commented at 8:10 PM on August 29, 2018:

No, it isn't. There is no address format for P2PK.

instagibbs commented at 8:15 PM on August 29, 2018:

huh, curious why Core API always returns P2PKH address for them, at least for coinbase outputs generated via generate.

TIL.

sipa commented at 8:29 PM on August 29, 2018:

@instagibbs That's due to a historical confusion where "address" and "pubkey identifier" were conflated. You can compute the hash160 of pubkeys, and those used to be represented in P2PK address format. For a long time for example validateaddress used to do that (it would list the P2PKH address for the pubkeys in a multisig too, for example, which makes no sense at all).

As an address, the P2PKH address format refers specifically to P2PKH scripts.

in doc/descriptors.md:53 in db7ebf0f45 outdated

  48 | +- `raw(HEX)` (top level only): the script whose hex encoding is HEX.
  49 | +
  50 | +Public keys:
  51 | +- Hex encoded public keys (66 characters starting with `02` or `03`, or 130 characters starting with `04`).
  52 | +  - Inside `wpkh` and `wsh`, only compressed public keys are permitted.
  53 | +  - WIF encoded private keys may be specified instead of the correspondong public key, with the same meaning.

flack commented at 8:06 PM on August 29, 2018:

Typo: correspondong

practicalswift commented at 8:12 PM on August 29, 2018:

Good catch! I can't believe I missed this typo yesterday. Perhaps it was not present then? :-)

laanwj commented at 10:32 AM on September 2, 2018:

we definitely want to fix this one before merge 🤣

sipa commented at 7:29 PM on September 4, 2018:

Fixed.

sipa force-pushed on Sep 4, 2018

in doc/descriptors.md:41 in 843bc92035 outdated

  34 | +A chain of P2PKH outputs:
  35 | +- `pkh(xpub68Gmy5EdvgibQVfPdqkBBCHxA5htiqg55crXYuXoQRKfDBFA1WEjWgP6LHhwBZeNK1VTsfTFUHCdrfp1bgwQ9xv5ski8PX9rL2dZXvgGDnw/1/0/*)`
  36 | +
  37 | +## Reference
  38 | +
  39 | +Descriptors consist of several types of expressions. The top level expression is always a `SCRIPT`.

ryanofsky commented at 4:51 PM on September 5, 2018:

A lot of this duplicates information in descriptor.h. Should that file link to this document to avoid this?

sipa commented at 6:21 PM on September 5, 2018:

Fixed.

in doc/descriptors.md:67 in 843bc92035 outdated

  62 | +
  63 | +### Single-key scripts
  64 | +
  65 | +Many single-key constructions are used in practice, generally including P2PK, P2PKH, P2WPKH, and P2SH-P2WPKH. Many more combinations are imaginable, though they may not be optimal: P2SH-P2PK, P2SH-P2PKH, P2WSH-P2PK, P2WSH-P2PKH, P2SH-P2WSH-P2PK, P2SH-P2WSH-P2PKH.
  66 | +
  67 | +To describe these, we model these as functions. The functions `pk` (P2PK), `pkh` (P2PKH) and `wpkh` (P2WPKH) take as input a public key in hexadecimal notation, and return the corresponding *scriptPubKey*. The functions `sh` (P2SH) and `wsh` (P2WSH) take as input a script, and return the script describing P2SH and P2WSH outputs with the input as embedded script. The names of the functions do not contain "p2" for brevity. Examples:

ryanofsky commented at 5:14 PM on September 5, 2018:

"take as input a public key in hexadecimal notation"

Maybe clarify these don't only take hexadecimal notation (since they should accept any KEY expressions).

sipa commented at 6:21 PM on September 5, 2018:

The explanation section builds things up step-by-step; I think it's more helpful to not give all information at once. I've instead added "which will be extended later", is that ok?

in doc/descriptors.md:68 in 843bc92035 outdated

  63 | +### Single-key scripts
  64 | +
  65 | +Many single-key constructions are used in practice, generally including P2PK, P2PKH, P2WPKH, and P2SH-P2WPKH. Many more combinations are imaginable, though they may not be optimal: P2SH-P2PK, P2SH-P2PKH, P2WSH-P2PK, P2WSH-P2PKH, P2SH-P2WSH-P2PK, P2SH-P2WSH-P2PKH.
  66 | +
  67 | +To describe these, we model these as functions. The functions `pk` (P2PK), `pkh` (P2PKH) and `wpkh` (P2WPKH) take as input a public key in hexadecimal notation, and return the corresponding *scriptPubKey*. The functions `sh` (P2SH) and `wsh` (P2WSH) take as input a script, and return the script describing P2SH and P2WSH outputs with the input as embedded script. The names of the functions do not contain "p2" for brevity. Examples:
  68 | +* `pk(0279be667ef9dcbbac55a06295ce870b07029bfcdb2dce28d959f2815b16f81798)` represents a P2PK output.

ryanofsky commented at 5:21 PM on September 5, 2018:

I think it would be good to drop the lists of examples in this ## Explanation section and move them up into the ## Examples section above, since that section is currently pretty meager and doesn't have simple, practical examples, and the explanations here are independent and don't really reference the examples that are mixed in.

sipa commented at 6:22 PM on September 5, 2018:

Good point, done.

in doc/descriptors.md:93 in 843bc92035 outdated

  88 | +
  89 | +Whenever a public key is described using a hardened derivation step, the script cannot be computed without access to the corresponding private key.
  90 | +
  91 | +### Including private keys
  92 | +
  93 | +Often it is useful to communicate a description of scripts along with the necessary private keys. For this reason, anywhere a public key or xpub is supported, a private key (in WIF format) or xprv may be provided instead. This may be useful for dumping wallet descriptors including private key material.

ryanofsky commented at 5:25 PM on September 5, 2018:

Seems inconsistent to parenthesize mention of WIF format but not xprv format. It might also be good to say in this section that xprv format is specified in BIP32.

sipa commented at 6:22 PM on September 5, 2018:

Done (I've added the link higher up).

in doc/descriptors.md:97 in 843bc92035 outdated

  92 | +
  93 | +Often it is useful to communicate a description of scripts along with the necessary private keys. For this reason, anywhere a public key or xpub is supported, a private key (in WIF format) or xprv may be provided instead. This may be useful for dumping wallet descriptors including private key material.
  94 | +
  95 | +### Compatibility with old wallets
  96 | +
  97 | +In order to easily represent the sets of scripts currently supported by existing Bitcoin Core wallets, a special function `combo` is provided, which takes as input a public key, and constructs the P2PK, P2PKH, P2WPKH, and P2SH-P2WPH scripts for that key. In case a key is uncompressed, it only constructs P2PK and P2PKH.

ryanofsky commented at 5:29 PM on September 5, 2018:

Maybe call it a convenience function instead of a special function. Special sounds like there's something magic about it, when actually it's generating a set of scripts just like all the other functions.

sipa commented at 6:22 PM on September 5, 2018:

Done.

ryanofsky approved

ryanofsky commented at 5:43 PM on September 5, 2018: member

ACK 843bc92035bfcfa4327b5400b3a722ad6df393ec

sipa force-pushed on Sep 5, 2018

sipa commented at 6:23 PM on September 5, 2018: member

Restructured the document, reformatted the markdown, and added a number of links.

Add descriptor reference documentation 9254ffcf2d

Adjust TODO link 89709db7a2

in src/script/descriptor.h:26 in df9cc0d36b outdated

  70 | -// * E/I': hardened child
  71 | -// * E/Ih: hardened child (alternative notation)
  72 | -//
  73 | -// The top level is S.
  74 | +// Reference documentation about the descriptor language can be found in
  75 | +// doc/descriptrs.md.

flack commented at 6:25 PM on September 5, 2018:

Typo: descriptrs

sipa commented at 6:28 PM on September 5, 2018:

Fixed.

sipa force-pushed on Sep 5, 2018

Replace duplcate reference with reference to reference doc 2b5d6f8df2

sipa force-pushed on Sep 5, 2018

ryanofsky commented at 6:31 PM on September 5, 2018: member

reACK 2b5d6f8df24b381d35b75187c97ae0cc9f7c3ed0, various changes since last review including some suggested changes and also adding links to more documentation

ken2812221 commented at 9:25 AM on September 6, 2018: contributor

utACK 2b5d6f8

practicalswift commented at 9:27 AM on September 6, 2018: contributor

utACK 2b5d6f8df24b381d35b75187c97ae0cc9f7c3ed0

laanwj referenced this in commit 0f681cecad on Sep 6, 2018

laanwj referenced this in commit eece974d0a on Sep 6, 2018

laanwj referenced this in commit 465a583f9d on Sep 6, 2018

laanwj merged this on Sep 6, 2018

laanwj closed this on Sep 6, 2018

laanwj referenced this in commit 35d7d9471f on Sep 6, 2018

laanwj removed the label Needs backport on Sep 6, 2018

laanwj commented at 12:53 PM on September 6, 2018: member

backported to 0.17 in 465a583f9dcc2383eb20bbb98bffa66fe82f2cde eece974d0a3b8049e73cc29a4d14efddc00101a3 0f681cecadb9472b3d3ae2c1dd23d8173dc2e316

HashUnlimited referenced this in commit 16bd7227ac on Sep 14, 2018

HashUnlimited referenced this in commit 902a3b4249 on Sep 14, 2018

HashUnlimited referenced this in commit d00d14a835 on Sep 14, 2018

uhliksk referenced this in commit bdc26c5a26 on Oct 18, 2018

uhliksk referenced this in commit a3da46bf72 on Oct 18, 2018

uhliksk referenced this in commit dbd5578714 on Oct 18, 2018

Munkybooty referenced this in commit e42d401204 on Sep 10, 2021

Munkybooty referenced this in commit dbb5abeeb0 on Sep 12, 2021

UdjinM6 referenced this in commit e5cdf55a47 on Sep 15, 2021

Munkybooty referenced this in commit b1d47a68a8 on Sep 15, 2021

Munkybooty referenced this in commit c2428c4c42 on Sep 15, 2021

Munkybooty referenced this in commit b8a87799f0 on Sep 17, 2021

vijaydasmp referenced this in commit b6963f6b18 on Sep 21, 2021

vijaydasmp referenced this in commit 07fc6f89c8 on Sep 21, 2021

vijaydasmp referenced this in commit e907d60537 on Sep 22, 2021

kittywhiskers referenced this in commit a044e1a703 on Oct 12, 2021

DrahtBot locked this on Feb 15, 2022