[BIPs 380-386] Output Script Descriptors

achow101 commented at 10:08 pm on July 14, 2021: member

Here are several BIPs specifying output script descriptors as implemented in Bitcoin Core.

In order to make it easier for implementors to indicate which descriptors they support, I’ve split up the specifications into multiple BIPs. The first is a general BIP describing the philosophy, general structure, shared expressions, and the checksum. The rest of the BIPs specify each descriptor function itself and are grouped into categories of non-segwit (pk, pkh, p2sh), segwit (wpkh, wsh), multi (multi, sortedmulti), combo (by itself), opaque wrappers (raw, addr), and tr (by itself).

achow101 cross-referenced this on Jul 14, 2021 from issue Descriptors by achow101

Rspigler commented at 1:12 am on July 15, 2021: contributor

As stated previously, strong concept ACK.

I think after this gets merged, it will be important to go through existing BIPs that make use of descriptors, and put in needed references.

michaelfolkson commented at 10:51 am on July 20, 2021: contributor

Concept ACK

In order to make it easier for implementors to indicate which descriptors they support, I’ve split up the specifications into multiple BIPs.

Will you be using the BIP numbers of these 7 BIPs in the actual descriptors? My initial response was that having as many as 7 BIPs seemed like overkill.

The motivation as stated in the Bitcoin dev mailing list post:

Since descriptors are modular and have optional components, I’ve decided to split it into 7 BIPs, rather than a single one.

This separation is so that implementors can choose to not implement some descriptors and still say which descriptors they support without being too difficult to understand.

I think (?) it is general practice that implementers aren’t expected to implement the BIP in full regardless so unless the BIP number is needed for the descriptor I wonder if it would be more accessible to implementers/reviewers to shrink the number of BIPs down from its current 7. Jumping between different BIPs for context (and creating a need for multiple BIPs to be updated in future) isn’t optimal imo if it can be avoided.

edit: Presumably also descriptor support for CHECKSIGADD (Taproot multisig) and eventually MuSig(2) will also have their own BIPs too? So the number of descriptor related BIPs will be increasing well past 7 too.

achow101 commented at 4:19 pm on July 20, 2021: member

Will you be using the BIP numbers of these 7 BIPs in the actual descriptors

No.

I think (?) it is general practice that implementers aren’t expected to implement the BIP in full regardless so unless the BIP number is needed for the descriptor I wonder if it would be more accessible to implementers/reviewers to shrink the number of BIPs down from its current 7

The expectation is that each implementations will implement in full the BIPs they want to support.

IMO it’s easier to say “software implements BIPs X, Y, and Z” than “software implements BIP XX but only the wpkh, wsh, and tr descriptors”. Every time you would say “software supports BIP XX descriptors”, you would also have to caveat it with which descriptors are supported. By using separate BIPs, you can just give the BIP numbers corresponding to the groups of descriptors supported.

edit: Presumably also descriptor support for CHECKSIGADD (Taproot multisig) and eventually MuSig(2) will also have their own BIPs too? So the number of descriptor related BIPs will be increasing well past 7 too.

Yes.

michaelfolkson commented at 4:40 pm on July 20, 2021: contributor

IMO it’s easier to say “software implements BIPs X, Y, and Z” than “software implements BIP XX but only the wpkh, wsh, and tr descriptors”. Every time you would say “software supports BIP XX descriptors”, you would also have to caveat it with which descriptors are supported. By using separate BIPs, you can just give the BIP numbers corresponding to the groups of descriptors supported.

If this approach (lots of modular BIPs) is followed for other projects it is going to mean a lot more BIPs (and BIP numbers to keep track of). I wondered if there was anything specific to descriptors that made this modular approach make sense.

But enough on the number of BIPs (!) I’ll read through them in more detail now I know you’re fixed on this.

achow101 commented at 4:46 pm on July 20, 2021: member

If this approach (lots of modular BIPs) is followed for other projects it is going to mean a lot more BIPs (and BIP numbers to keep track of). I wondered if there was anything specific to descriptors that made this modular approach make sense.

This is something already being done for other aspects of Bitcoin. Off the top of my head, both BIP 43 and 174 are now using this approach of using multiple BIPs to extend existing functionality. New BIP 43 purpose numbers are defined in new BIPs (see 44, 49, 85, and 86). BIP 174 is much like descriptors in that it is modular and extensible. New fields are being specified in new BIPs rather than being just added to BIP 174.

luke-jr added the label New BIP on Jul 20, 2021

luke-jr commented at 7:01 pm on July 20, 2021: member

Going to leave this unassigned for now, just to collect ACKs/comments on the specific splitting of the BIPs (which would affect the number range and meaning of those numbers) for a few weeks.

While the acceptance of BIPs does not depend on third-party ACKs, number assignments can’t be adjusted/changed quite so easily once done, so I think it would be better to flesh out any review on the specific splitting beforehand.

(My personal opinion is that perhaps the pretty-much-just-for-compatibility stuff should be combined, but I don’t care strongly.)

KimJi55 commented at 10:13 pm on July 23, 2021: none

How do I put it in my pocket without a computer? I only have a mobile phone…please give me some advice,thanks

Rspigler commented at 8:25 pm on July 27, 2021: contributor

As I stated here, I think it makes sense to split the BIPs up, especially as that seems to be the direction we are heading in with extensions/compatibility.

Not all implementations will comply fully with the complete spec

in bip-descriptors-non-segwit.mediawiki:49 in 38157a5ac1 outdated

44+<KEY> OP_CHECKSIG
45+</pre>
46+
47+===<tt>pkh()</tt>===
48+
49+The <tt>pkh(KEY)</tt> expression can be used as a top level expression, or inside of either a <tt>sh()</tt> or <tt>wsh()</tt> descriptor.

sanket1729 commented at 7:28 pm on July 28, 2021:

Is there any special reason for the description of pk() script expression and pkh() script expression different? Also, we are using wsh without actually defining/(referring to) what it is.

achow101 commented at 10:14 pm on July 28, 2021:

pk is allowed inside of tr but pkh is not.

Also, we are using wsh without actually defining/(referring to) what it

These definitions all have mentions of descriptors from other documents. I haven’t really figured out a good way to do.

sanket1729 commented at 10:32 pm on July 28, 2021:

pk is allowed inside of tr but pkh is not.

Makes sense.

These definitions all have mentions of descriptors from other documents. I haven’t really figured out a good way to do it.

I think the main descriptor BIP could be kept updated with all the top-level contexts and a very high-level summary. How about something like?

0Legacy output descriptors: sh(), pk(), pkh(), multi(), sorted_multi()
1Segwitv0 output Desctipors: wsh(), wpkh(), 
2Taproot output Descriptor: tr(),

achow101 commented at 11:53 pm on July 28, 2021:

I’ve added a listing of expressions as appendices.

in bip-descriptors-segwit.mediawiki:47 in 148b0ba417 outdated

42+OP_0 <KEY_hash160>
43+</pre>
44+
45+===<tt>wsh()</tt>===
46+
47+The <tt>wsh(SCRIPT)</tt> expression can be used as a top level expression, or inside of a <tt>sh()</tt> descriptor.

sanket1729 commented at 7:34 pm on July 28, 2021:

wsh() Should also have a note about only compressed keys being allowed

achow101 commented at 10:15 pm on July 28, 2021:

There is a few lines down “Any key expression found in any script expression contained by a wsh() expression must only produce compresed public keys.”

in bip-descriptors-tr.mediawiki:79 in a4c0b268c7 outdated

74+</pre>
75+
76+===Modified Key Expression===
77+
78+Key Expressions within a <tt>tr()</tt> expression must only create x-only public keys.
79+Uncompressed public keys are not allowed, but compressed public keys may be implicitly converted to x-only public keys.

sanket1729 commented at 7:44 pm on July 28, 2021:

0- may be implicitly
1+ would be implicitly

achow101 commented at 10:15 pm on July 28, 2021:

Done

sanket1729 commented at 7:54 pm on July 28, 2021: contributor

ACK for different BIPs. I think the numbers would numerically close from each other that this should not matter.

achow101 force-pushed on Jul 28, 2021

in bip-descriptors-tr.mediawiki:46 in 1e7ab4d536 outdated

41+* A pair of Tree Expressions consisting of:
42+** An open brace <tt>{</tt>
43+** A Tree Expression
44+** A comma <tt>,</tt>
45+** A Tree Expression
46+** A closing brance <tt>}</tt>

michaelfolkson commented at 12:25 pm on July 29, 2021:

s/brance/brace

achow101 commented at 5:08 pm on July 29, 2021:

Done

in bip-descriptors-general.mediawiki:118 in 1e7ab4d536 outdated

113+The first group are the most common "unprotected" characters (i.e. things such as hex and keypaths that do not already have their own checksums).
114+Case errors cause an offset that is a multiple of 32 while as many alphabetic characters are in the same group while following the previous restrictions.
115+
116+===Checksum===
117+
118+Follwing the top level script expression is a single octothorpe (<tt>#</tt>) followed by the 8 character checksum.

michaelfolkson commented at 12:33 pm on July 29, 2021:

s/Follwing/Following

achow101 commented at 5:08 pm on July 29, 2021:

Done

in bip-descriptors-general.mediawiki:124 in 1e7ab4d536 outdated

119+The checksum is an error correcting checksum similar to bech32.
120+
121+The checksum has the following properties:
122+* Mistakes in a descriptor string are measured in "symbol errors". The higher the number of symbol errors, the harder it is to detect:
123+** An error substituting a character from <tt>0123456789()[],'/*abcdefgh@:$%{}</tt> for another in that set always counts as 1 symbol error.
124+*** Note that hex encoded keys are covered by these characters. Extended keys (<tt>xpub</tt> and <tt>xprv</tt>) use other characters too, but also have their own checksum mechansim.

michaelfolkson commented at 12:34 pm on July 29, 2021:

s/mechansim/mechanism

achow101 commented at 5:08 pm on July 29, 2021:

Done

in bip-descriptors-general.mediawiki:119 in 1e7ab4d536 outdated

114+Case errors cause an offset that is a multiple of 32 while as many alphabetic characters are in the same group while following the previous restrictions.
115+
116+===Checksum===
117+
118+Follwing the top level script expression is a single octothorpe (<tt>#</tt>) followed by the 8 character checksum.
119+The checksum is an error correcting checksum similar to bech32.

michaelfolkson commented at 12:36 pm on July 29, 2021:

May want to say bech32m here rather than bech32 for future proofing.

achow101 commented at 5:02 pm on July 29, 2021:

Bech32m is bech32 with a different constant. The method of the encoding itself is bech32, and specified by bech32, not bech32m.

in bip-descriptors-tr.mediawiki:92 in 1e7ab4d536 outdated

87+TBD
88+
89+==Backwards Compatibility==
90+
91+<tt>tr()</tt> descriptors use the format and general operation specified in [[bip-descriptor-general.mediawiki|bip-descriptor-general]].
92+As these are a wholly new descriptors, they are not compatible with any implementation.

michaelfolkson commented at 12:40 pm on July 29, 2021:

s/these are a wholly new descriptors/these are a set of wholly new descriptors

achow101 commented at 5:08 pm on July 29, 2021:

Done

in bip-descriptors-tr.mediawiki:54 in 1e7ab4d536 outdated

49+
50+The <tt>tr(KEY)</tt> or <tt>tr(KEY, TREE)</tt> expression can only be used as a top level expression.
51+All key expressions under any <tt>tr()</tt> expression must create x-only public keys.
52+
53+<tt>tr(KEY)</tt> takes a single key expression as an argument and produces a P2TR output script which does not have a script path.
54+The keys produced by the key expression are used as the internal key as specified by [[bip-0341.mediawiki#cite_ref-22-0|BIP 341]].

michaelfolkson commented at 12:43 pm on July 29, 2021:

s/keys/key s/are used as/is used as.

There is only one (internal) key produced by the key expression right?

achow101 commented at 5:08 pm on July 29, 2021:

A single key expression produces one key per index, but multiple keys in general. Each key is used as the internal key to a separate output. This should already be obvious from knowing BIP 341, but I’ve tried to make it clearer.

in bip-descriptors-tr.mediawiki:31 in 1e7ab4d536 outdated

26+Taproot added one additional standard output script format: P2TR.
27+These expressions allow specifying those formats as a descriptor.
28+
29+==Specification==
30+
31+A new script expressions are defined: <tt>tr()</tt>.

michaelfolkson commented at 12:45 pm on July 29, 2021:

s/A new script expressions are defined/A new script expression is defined

achow101 commented at 5:08 pm on July 29, 2021:

Done

in bip-descriptors-general.mediawiki:135 in 1e7ab4d536 outdated

130+* Any 4 symbol error in a descriptor of up to 507 characters is always detected.
131+* Any 5 symbol error in a descriptor of up to 77 characters is always detected.
132+* Is optimized to minimize the chance of a 5 symbol error in a descriptor up to 387 characters is undetected
133+* Random errors have a chance of 1 in 2<super>40</super> of being undetected.
134+
135+The checksum itself uses the same character set as bech32: <tt>qpzry9x8gf2tvdw0s3jn54khce6mua7l</tt>

michaelfolkson commented at 12:50 pm on July 29, 2021:

Also bech32m here (if changed above)

in bip-descriptors-general.mediawiki:114 in 1e7ab4d536 outdated

109+</pre>
110+Note that <tt><space></tt> on the last line is a space character.
111+
112+This character set is written as 3 groups of 32 characters in this specific order so that the checksum below can identify more errors.
113+The first group are the most common "unprotected" characters (i.e. things such as hex and keypaths that do not already have their own checksums).
114+Case errors cause an offset that is a multiple of 32 while as many alphabetic characters are in the same group while following the previous restrictions.

michaelfolkson commented at 12:51 pm on July 29, 2021:

s/alphabetic/alphanumeric

achow101 commented at 5:03 pm on July 29, 2021:

Alphabetic is correct.

in bip-descriptors-general.mediawiki:90 in 1e7ab4d536 outdated

85+If the <tt>KEY</tt> is a BIP 32 extended key, before output scripts can be created, child keys must be derived using the derivation information that follows the extended key.
86+When the final step is <tt>/*</tt> or <tt>/*'</tt>, an output script will be produced for every child key index.
87+The derived key must be not be serialized as an uncompressed public key.
88+Script Expressions may have further requirements on how derived public keys are serialized for script creation.
89+
90+In the above specification, the hardened indicator <tt>h</tt> may be replaced with alternative hardnened indicators of <tt>H</tt> or <tt>'</tt>.

michaelfolkson commented at 12:52 pm on July 29, 2021:

s/hardnened/hardened

achow101 commented at 5:09 pm on July 29, 2021:

Done

michaelfolkson commented at 12:54 pm on July 29, 2021: contributor

Just some nits reading through.

achow101 force-pushed on Jul 29, 2021

michaelfolkson commented at 2:57 pm on July 30, 2021: contributor

I wonder if another consequence of having so many granular BIPs is you need to be clearer on what a BIP doesn’t contain and provide cross references to the BIPs that an implementer may be interested in (having initially gone to wrong BIP or unsure which BIP to go to first). Otherwise an implementer is going to be bouncing around a number of different BIPs (especially once we have 10+ descriptor BIPs).

e.g. “This multisig descriptor BIP only contains information on descriptors using legacy multisig and does not contain information on Taproot multisig (CHECKSIGADD) or MuSig, that information can be found in BIP X and BIP Y.”

(Or more accurately today specifically for Taproot multisig and MuSig, these are not currently specified.)

Rspigler commented at 8:00 pm on July 30, 2021: contributor

Maybe an index of sorts could be placed in BIP-descriptors-general, which is where I imagine everyone will start?

luke-jr commented at 9:26 pm on August 29, 2021: member

Assigning these BIP 380-386, in order listed in the current PR’s README.mediawiki

Specify BIP 380: Descriptors general operation 71f435e76b

achow101 force-pushed on Aug 30, 2021

achow101 renamed this:
~~[New BIPs] Output Script Descriptors~~
[New BIPs] BIPs 380-386 Output Script Descriptors
on Aug 30, 2021

achow101 renamed this:
~~[New BIPs] BIPs 380-386 Output Script Descriptors~~
[BIPs 380-386] Output Script Descriptors
on Aug 30, 2021

achow101 commented at 0:09 am on August 30, 2021: member

Updated with BIP numbers

Specify BIP 381: Non-segwit descriptors bd12ea7e9d

Specify BIP 382: Segwit descriptors fb56a18c89

Specify BIP 383: Multisig descriptors 608f40b498

Specify BIP 384: Combo descriptors f52f1e82e5

Specify BIP 385: Raw and addr descriptors 5403ff90d6

Specify BIP 386: Taproot descriptors 761ef12782

achow101 force-pushed on Aug 30, 2021

luke-jr merged this on Aug 30, 2021

luke-jr closed this on Aug 30, 2021

ChristopherA commented at 9:55 pm on September 4, 2021: none

I know that I’d appreciate if @luke-jr could hold as placeholder a few consecutive BIP numbers after this series for further descriptor related BIPs that also will be coming soon. Like 387-389.

[BIPs 380-386] Output Script Descriptors #1143