descriptors: represent multiple derivation paths within one descriptor

achow101 commented at 4:25 pm on October 18, 2019: member

It would be useful to express a full wallet of external and change scripts derived from the same xpub as one descriptor. Currently this requires two different descriptors where most of the descriptor is the same but only the derivation path at the end is different. So instead, one descriptor could contain multiple derivation paths following the xpub.

I propose that we could use {} to contain the multiple derivation paths, separated by ,. So it would look something like:

0xpub...{0,1}/*

Thoughts?

cc @sipa

achow101 added the label Feature on Oct 18, 2019

fanquake added the label Descriptors on Oct 18, 2019

ryanofsky commented at 4:54 pm on October 18, 2019: contributor

Maybe say more about the concrete use case? This seems similar to bash expanding abc{d,e} to abcd abce, but bash commands are written by people, while I’d expect descriptor strings to be mostly produced and consumed by software, so on the surface it’s not clear who would benefit from this feature.

Also it’s not clear why the syntax wouldn’t be xpub...{0,1}/* instead of xpub...{0/*,1/*} or if expansions into more than 2 descriptors should be allowed, or if first and second descriptors should always be interpreted as nonchange and change descriptors, respectively.

sipa commented at 5:20 pm on October 18, 2019: member

My first question is the same: how is this useful? I can imagine that someone might want to include both an xpub…/0/* and an xpub…/1/* into the same wallet, but I expect you’d want one of them to be public and the other to be change. That makes it hard to merge them, as the changeness information would be in another layer?

achow101 commented at 5:59 pm on October 18, 2019: member

The specific use cases that was mentioned was for the construction of multisig wallets. The idea was that it would be easier in terms of UI/UX for users to be able to just import one descriptor that specified both the receive and change addresses to be used.

Perhaps @ghost43 can provide some more insight here.

That makes it hard to merge them, as the changeness information would be in another layer?

I agree that changeness should be another layer. I think they were going to just assume that the 0 path is receive, and 1 is change.

Also it’s not clear why the syntax wouldn’t be xpub...{0,1}/* instead of xpub...{0/*,1/*}

I agree that that would be a better syntax. Didn’t think of it earlier.

or if expansions into more than 2 descriptors should be allowed,

My initial thinking was to allow expansion into more than 2. So this could be used for things like multiple BIP 44 accounts in scantxoutset.

or if first and second descriptors should always be interpreted as nonchange and change descriptors, respectively.

I’m not sure about how to represent changeness, and I’m not sure that it should be represented in the descriptor layer.

sipa commented at 6:17 pm on October 18, 2019: member

I agree that changeness should be another layer. I think they were going to just assume that the 0 path is receive, and 1 is change.

Hmm, what do you expect the semantics to be? Do you literally treat the “xpub…{0,1}/*” descriptor as a list of two descriptors, which the layer above needs to deal with? Or do you treat it as a single descriptor which expands to a merging of those two descriptors (e.g. by expanding even indices i as index i/2 of the first one, and odd ones i as index (i-1)/2 of the second one)?

If you treat them as two descriptors, arguably you’re not extending the descriptor syntax, but instead building a language on top that describes lists of descriptors - something that perhaps won’t (or even shouldn’t) work in all places descriptors are accepted.

If you treat them as a single merged descriptor, the layer above can’t see which ones are from 0/* and which ones are from 1/*, and shouldn’t treat them non-homogeneously.

achow101 commented at 6:23 pm on October 18, 2019: member

I would expect it to be treated as two descriptors.

ryanofsky commented at 6:49 pm on October 18, 2019: contributor

The specific use cases that was mentioned was for the construction of multisig wallets.

That isn’t really a use case description. Is the pair of descriptors handcrafted by a user, copied and pasted somewhere, saved in a text file, transmitted over a wire protocol? I know I have a problem not hating json as much as I’m supposed to, but just for example, what would be the drawbacks of a verbose format like:

0{
1  "payment_descriptor": "xpub...0/*",
2  "change_descriptor": "xpub...1/*"
3}

SomberNight commented at 10:16 pm on October 18, 2019: contributor

See https://github.com/spesmilo/electrum/issues/5694#issuecomment-543544564 and also https://github.com/spesmilo/electrum/issues/5715 (specifically “point 1”)

Basically the actual thing that we would need is a way to represent a cosigner; so at the very least the following:

master fingerprint
derivation prefix
xpub

All of which you can already do using script descriptor KEY expression. It would be really nice however if the KEY expression could also contain the derivation suffixes to be used as well. Specifically for Electrum’s current HD wallets it would be a m/0/i (receive), m/1/j (change) hierarchy. To do this you however currently need two KEY expressions. I think basically all light wallets use this hierarchy atm (m/0/i, m/1/j), this is also what BIP44 uses.

As detailed in first link above, currently in Electrum when constructing a multisig wallet, cosigners share xpubs with each other; other contextual information such as number of required signers are selected in the GUI. To be able to construct an output script descriptor that describes a multisig where cosigners have key origin information for example, instead of xpubs, we could show KEY expressions to cosigners that they would share each other. The issue is HOW the cosigners are supposed to create the script descriptor.

For now, we would want to support at least the above described receive/change hierarchy, but with the current KEY expression syntax cosigners would need to be shown and then expected to share two of these.

I know there are some undesirable properties of sharing these instead of the xpubs as now not the whole thing is checksummed, but the script descriptor needs to be constructed somehow. When looking at the output descriptors doc, I feel like this point has been considered completely out of scope so far, but it is a real UX issue.

dhruvbansal commented at 3:34 am on December 19, 2019: none

+1 on @SomberNight’s comments. Collaborative multisig requires exchanging xpubs and then agreeing on how to derive & use addresses (script template, derivation paths, &c.). If these decisions aren’t tracked well, this “contextual information” becomes hidden state which frustrates users and developers alike.

Descriptors are almost the perfect way to encode these decisions into a concise, robust, portable format. The ability to specify multiple “families” of addresses for (say) receiving vs. change addresses would be extremely useful.

One can argue that multiple descriptors are just as good but practically they lack a shared checksum and aesthetically using to almost identical long strings to represent two related families seems wrong. Related descriptors are extremely similar and differ merely in one derivation path segment. It would be more natural for the descriptor definition to allow multiple wildcards .../*/* or constructs such as .../{0,1}/* to enable this use case.

Rspigler commented at 0:28 am on January 14, 2021: contributor

It would be useful to express a full wallet of external and change scripts derived from the same xpub as one descriptor. Currently this requires two different descriptors where most of the descriptor is the same but only the derivation path at the end is different. So instead, one descriptor could contain multiple derivation paths following the xpub.

This is something we are running into at Yeti, which was seen here #20831

The use case is that it is much easier to create and backup a descriptor-based multisig wallet if you can do it with a single descriptor rather than two.

benma commented at 10:59 am on January 15, 2021: none

In the BitBox02, we have arrived at using one descriptor at the account level. Change and receive addresses (scripts) are simply assumed to be at ./1/* and ./0/*, as derived from the xpubs given in the multisig descriptor.

We also register the multisig setup in the device. Having one descriptor correspond to one multisig account makes things easier than to register multiple descriptors (one for receive, one for change), as the user has to verify less data. Backup is also simplified - one descriptor is easier to backup and restore than two.

Since pretty much every wallet deals with changes and receive addresses in this way, I don’t see it as very useful track two separate descriptors, or even to put two separate derivation paths into one descriptor. Sounds easiest to stick to the convention of ./1/* and ./0/* for change and receive addresses.

Rspigler commented at 0:43 am on January 16, 2021: contributor

Yes, that sounds great! That would be very helpful for us. I would definitely support a standard being built around this.

craigraw commented at 6:47 am on January 16, 2021: none

Sparrow Wallet does the same, stripping off the /0/* and /1/* when an output descriptor is pasted in to configure a wallet.

I appreciate that the Core wallet may need to handle arbitrary output descriptors, but from a user experience POV greater adoption of output descriptors is much easier when a single output descriptor can describe a wallet (consistent with most users expectations).

Rspigler commented at 10:44 pm on January 16, 2021: contributor

from a user experience POV greater adoption of output descriptors is much easier when a single output descriptor can describe a wallet (consistent with most users expectations).

That could be the default, and allowing arbitrary output descriptors could be under expert options?

Rspigler commented at 7:14 am on May 13, 2021: contributor

This seems to have been NACKd by @sipa here

But using a descriptor template allows for the same result, using a different layer, as suggested by Sipa.

Sjors commented at 11:52 am on June 25, 2021: member

See also BIP 88: https://github.com/bitcoin/bips/blob/master/bip-0088.mediawiki

Rspigler commented at 10:10 pm on June 25, 2021: contributor

This can probably be closed now?

Sjors commented at 1:52 pm on June 26, 2021: member

I suggest we leave this open until BIP 88 or something like it is implemented.

NicolasDorier commented at 2:53 am on August 4, 2021: contributor

stumbled upon this problem while trying to expose output descriptor in BTCPay.

From UX perspective, the current situation sucks for a user wanting to import descriptor from wallet A to B, where B only follow deposit/change paths.

The user goes into his wallet A, and see two, almost identical output descriptor.
The user then get confused into which one he should copy/paste to import the other wallet B
Our solution is to just ignore the deposit/change path and making an error if it is anything else that 0/* or 1/*.

When we expose the descriptors of the wallet, this is also confusing: The user wonder why there is two descriptors, when they imported only one previously.

I don’t think this can be solved by now, we need to live with this issue.

achow101 commented at 3:18 am on August 4, 2021: member

Another question that was posed to me recently was why do we even have split derivation paths in the first place? It seems to me that there isn’t really an advantage to having receiving and change addresses be separated. If anything, it is creating more of a problem because it makes backups harder. Unless there is a strong reason to continue to use separate receiving and change descriptors, I think the entire ecosystem should move away from this separation. That would make this entire issue moot.

From reading BIP 32, it seems like the primary reason for recommending split key chains is to allow for an xpub to be given out to people (for some reason) but still maintain some privacy about funds that were sent to change. I don’t think xpubs are shared publicly, and there are known issues with unhardened derivation, so I don’t think this is a valid use case anymore.

NicolasDorier commented at 3:40 am on August 4, 2021: contributor

yes, though this is one of those thing integrated already so deeply in software dealing with bitcoin that reversing this recommendation will not be possible anytime soon.

shesek commented at 3:43 am on August 4, 2021: contributor

Some benefits of split key chains that I can think of:

It reduces the gap limit of the change chain, which guarantees that you’ll at least be able to recover all of your change.
It helps avoid mixing them up (e.g. using addresses that were previously handed out to people and remained unused as change).
It makes it easier tell change and receive outputs apart, which can be useful to determine if they’re ‘unsafe’ to spend or not (although they could also be told apart by inspecting the transaction that created the output and checking if any of its inputs are wallet-owned).

I’m not sure if this makes it worth the hassle, though. OTOH it seems like much of the (user-facing) hassle would be gone if something like BIP 88 allowed specifying multiple descriptors as a single copy-paste-able string.

NicolasDorier commented at 3:54 am on August 4, 2021: contributor

BIP88 is yet one more standard to show almost the same thing. Damn I hate HD :(

sipa commented at 4:05 am on August 4, 2021: member

Damn I hate HD :(

Bitcoin developer makes emotional plea to revert to 640x480 video resolution.

achow101 commented at 4:44 am on August 4, 2021: member

It reduces the gap limit of the change chain, which guarantees that you’ll at least be able to recover all of your change.

I don’t think it has a significant effect on that. Gap limits should already be fairly high anyways.

It helps avoid mixing them up (e.g. using addresses that were previously handed out to people and remained unused as change).

That shouldn’t ever be a problem because wallets shouldn’t ever give out already given out addresses.

The only issue is with restoring a backup, but I think a consolidated key chain actually makes that better. It is possible that a user has given out several addresses but were never used. If a backup without the metadata indicating that were restored, then the user could end up giving out those same addresses again. This could occur in both split and consolidated key chain cases. However, in the consolidated case, if the user had made a transaction that had a change output in the time between giving out unused addresses and restoring, then the change address would be marked as used, and that would mark all of the addresses up to that change address as being used, and if those were ones given out but unused, then there is no possibility for the user to accidentally reuse those addresses. With a split key chain, the restored wallet would never know that those addresses had been used already.

It makes it easier tell change and receive outputs apart, which can be useful to determine if they’re ‘unsafe’ to spend or not (although they could also be told apart by inspecting the transaction that created the output and checking if any of its inputs are wallet-owned).

I think all wallets already have some way to track which addresses are change independent of the derivation path used (or at least I hope they do). So this shouldn’t be a problem. Even when restoring a wallet, it is obvious which outputs are change through inspection, and restored wallets already have to be doing that to figure out what index to start new addresses at.

I’m not sure if this makes it worth the hassle, though. OTOH it seems like much of the (user-facing) hassle would be gone if something like BIP 88 allowed specifying multiple descriptors as a single copy-paste-able string.

Looking briefly at BIP 88, it seems like it does basically what we want, so perhaps we should just include that into the descriptor spec?

Finding a way to represent multiple derivation paths and discouraging use of separate receive and change descriptors aren’t mutually exclusive. We could do both.

ghost commented at 4:57 am on August 4, 2021: none

From UX perspective, the current situation sucks for a user wanting to import descriptor from wallet A to B, where B only follow deposit/change paths. The user goes into his wallet A, and see two, almost identical output descriptor. The user then get confused into which one he should copy/paste to import the other wallet B

Two descriptors from core wallet returned in listdescriptors:

 0    {
 1      "desc": "wpkh([5a448b3e/84'/1'/0']tpubDDmPGDzxy5QDAKVqnZCuXYqAGxLbgKUD5Dms14Xw7kVaMQ4qvyHTc5Dp59b5MJXXBHGB9UgSdAddKDry7rP9zvBMkHjKRTKxmiy7Q4Hpo2f/0/*)#cdpwktpj",
 2      "timestamp": 1628031603,
 3      "active": true,
 4      "internal": false,
 5      "range": [
 6        0,
 7        999
 8      ],
 9      "next": 0
10    },
11    {
12      "desc": "wpkh([5a448b3e/84'/1'/0']tpubDDmPGDzxy5QDAKVqnZCuXYqAGxLbgKUD5Dms14Xw7kVaMQ4qvyHTc5Dp59b5MJXXBHGB9UgSdAddKDry7rP9zvBMkHjKRTKxmiy7Q4Hpo2f/1/*)#fey0t732",
13      "timestamp": 1628031604,
14      "active": true,
15      "internal": true,
16      "range": [
17        0,
18        999
19      ],
20      "next": 0
21    }

@NicolasDorier Maybe below thing can be used in BTCPay and other wallets GUI for better UI/UX using above descriptors which look same:

Rspigler commented at 6:02 am on August 4, 2021: contributor

I too am a big fan of BIP 88 @dgpv

I have no opinion currently on discouraging use of separate receive and change descriptors - I’ll have to think about that more.

benma commented at 11:41 am on August 4, 2021: none

@achow101

I think all wallets already have some way to track which addresses are change independent of the derivation path used (or at least I hope they do). So this shouldn’t be a problem. Even when restoring a wallet, it is obvious which outputs are change through inspection, and restored wallets already have to be doing that to figure out what index to start new addresses at.

The BitBoxApp, and most other wallets I looked at (e.g. Electrum), use the derivation path to determine which outputs are change and which aren’t. How exactly would it be obvious when restoring a wallet which outputs are change if they are not derived on separate derivation paths?

Knowing which outputs are change is important for the wallet to function properly, e.g. to display the correct balance.

Rspigler commented at 5:01 pm on August 4, 2021: contributor

@benma - I believe it is possible through “inspecting the transaction that created the output and checking if any of its inputs are wallet-owned” @achow101:

However, in the consolidated case, if the user had made a transaction that had a change output in the time between giving out unused addresses and restoring, then the change address would be marked as used

If the backup doesn’t contain metadata, and the address is unused, how would the wallet know to mark the change address used?

If that is the case, I agree consolidating is the way to move forward.

Does Bitcoin Core currently track which addresses are change independent of the derivation path?

achow101 commented at 5:07 pm on August 4, 2021: member

How exactly would it be obvious when restoring a wallet which outputs are change if they are not derived on separate derivation paths?

By observing the transactions it is involved in. If a transaction contains inputs from the wallet, and has an output that belongs to the wallet. then that outputs is change.

Knowing which outputs are change is important for the wallet to function properly, e.g. to display the correct balance.

Knowing whether an output belonging to the wallet is receiving or change is not necessary to calculate the correct balance. It is needed to make transaction history and transaction display understandable to users.

If the backup doesn’t contain metadata, and the address is unused, how would the wallet know to mark the change address used?

Since the wallet typically does not give change addresses to the user, the only place they appear is in a transaction. When restoring a backup without metadata, the restore process will observe an address used in an output of a transaction which has inputs that belonged to the wallet. This address is then clearly a change address, and because it appears in a transaction, marked as used.

Does Bitcoin Core currently track which addresses are change independent of the derivation path?

Yes. It has always done so, especially because prior to HD, there was no other way.

Rspigler commented at 6:58 pm on August 4, 2021: contributor

Oh yes, of course, that makes sense.

Strong Concept ACK for BIP 88 and consolidating use of change and receive paths, although the later should probably be discussed on the ML.

achow101 commented at 7:53 pm on August 9, 2021: member

#19856 is one case where having split derivation paths is useful. In that issue, the user is making send-to-self transactions with the wallet loaded simultaneously in another node. Those send-to-self txs are detected as change so the txs are not listed. But with split derivation paths (or rather more generally, different descriptors for receiving and change), it becomes possible to distinguish send-to-self from change. This scenario only applies to restored wallets, and wallets loaded in multiple places (which is inadvisable).

achow101 commented at 8:22 pm on August 9, 2021: member

Thinking on BIP 88 (and this idea in general), it seems like having multiple possible derivation paths starts to break down once you get past the basic change and not change aspects, and especially when there are multiple keys and different derivation path patterns.

In a single key case, this is pretty obvious, it just becomes a lot of descriptors. So something like

wpkh(xpub.../{0-10}/{0-1}/*

becomes

wpkh(xpub.../0/0/*)
wpkh(xpub.../0/1/*)
wpkh(xpub.../1/0/*)
wpkh(xpub.../1/1/*)
wpkh(xpub.../2/0/*)
wpkh(xpub.../2/1/*)
... etc.

But if there are multiple keys, such as in multisigs, it gets a bit more complicated. For example

multi(2, xpub.../{0-1}/*, xpub.../{0-1}/*)

To fit with the expectation of change and not change, this would become

multi(2, xpub.../0/*, xpub.../0/*)
multi(2, xpub.../1/*, xpub.../1/*)

rather than

multi(2, xpub.../0/*, xpub.../0/*)
multi(2, xpub.../0/*, xpub.../1/*)
multi(2, xpub.../1/*, xpub.../0/*)
multi(2, xpub.../1/*, xpub.../1/*)

But maintaining this pattern for change and not change doesn’t quite work if there are multiple of these multipath expressions as BIP 88 allows. For example, how would

multi(2,xpub.../48h/0h/{0,1}/*,xpub.../48h/{0-10}/{0,1})`

be expanded?

So I think perhaps this is not a tenable solution, or it may result in expansions that contain unexpected descriptors. I suppose one method is to just limit this syntax to single key descriptors, but then it isn’t quite as useful, and I would prefer to have Key expressions which don’t have special syntaxes that only work for certain descriptors.

Instead of a generic solution like BIP 88, we could implement a specific solution for just the receive and change issue. Perhaps a special path index {r,c} that is allowed to appear once per path. r is the path to use for receiving addresses, and c is the path to use for change addresses. This works for descriptors with multiple keys as this would become two descriptors, one with only the r indexes, and one with only the c indexes. Since only one is allowed per path specifier, there is no issue with multiple path specifiers as BIP 88 would have. In the common case, it would still be written as {0,1}, but other derivation paths could be used for r and c if desired.

For example:

multi(2, xpub.../{0,1}/*, xpub.../{2,3}/*)

would become

multi(2, xpub.../0/*, xpub.../2/*)
multi(2, xpub.../1/*, xpub.../3/*)

Rspigler commented at 9:56 pm on August 9, 2021: contributor

#19856 is one case where having split derivation paths is useful.

Interesting. So there is no way to examine the transaction during restore and differentiate b/w send to self and change? (without separate descriptors). Could this be marked by metadata?

Re: BIP88:

I don’t believe multisigs would have multi-account path expressions (at least when BIP88 is used with BIP87 and BIP129) - or atleast that was my understanding.

How would wsh(sortedmulti(2,[xfpForA/87'/0'/0']XpubA/{0,1}/*,[xfpForB/87'/0'/0']XpubB/{0,1}/*)) work if one of the participants had a BIP88 account range of {0,10}? I don’t believe BIP129 would currently support multiple accounts per signer, as it would require multiple signatures per key record.

I like your {r,c} proposal, because then it is explicit about how something like your example of :

multi(2, xpub.../{0-1}/*, xpub.../{0-1}/*) is to be expanded

achow101 commented at 11:00 pm on August 9, 2021: member

Interesting. So there is no way to examine the transaction during restore and differentiate b/w send to self and change? (without separate descriptors).

The only case where I think it would be obvious is when the transaction does not have any change. So a tx that spends inputs from the wallet and creates a single output also owned by the wallet can be assumed to be a send to self. But if there is a change (so 2 outputs to self), then we cannot distinguish which is for change and which is the send to self without checking which descriptor was used to generate each address.

Could this be marked by metadata?

Yes, and it is. But that doesn’t apply when restoring a wallet.

Re: BIP88:

I don’t believe multisigs would have multi-account path expressions (at least when BIP88 is used with BIP87 and BIP129) - or atleast that was my understanding.

How would wsh(sortedmulti(2,[xfpForA/87'/0'/0']XpubA/{0,1}/*,[xfpForB/87'/0'/0']XpubB/{0,1}/*)) work if one of the participants had a BIP88 account range of {0,10}? I don’t believe BIP129 would currently support multiple accounts per signer, as it would require multiple signatures per key record.

Within the framework of existing BIPs, this scenario is probably not possible. But we can’t just think within what already exists. We have to think about all of the other weird stuff that could happen. So that means considering what could happen if a descriptor had multiple multi-path specifiers with multiple keys in the descriptor. That would still be BIP 88 valid, and I don’t think the way to expand that would work well with what people actually want.

Rspigler commented at 0:57 am on August 10, 2021: contributor

The only case where I think it would be obvious is when the transaction does not have any change

Great point.

Yes, and it is. But that doesn’t apply when restoring a wallet.

If the wallet is restored via descriptor, but if it restored via wallet.dat, it should (and is how documentation recommends) #22523

we can’t just think within what already exists. We have to think about all of the other weird stuff that could happen

Makes sense, thank you for the explanation.

dgpv commented at 6:45 am on August 10, 2021: contributor

Perhaps a special path index {r,c} that is allowed to appear once per path.

Note that if such path is also a valid BIP88 path, the receive index would need to always be the lower one, as indexes should come in increasing order in the index range. You would not be able to express ‘receive index 1, change index 0’, for example, because receive index would be larger than change index

mjdietzx commented at 4:15 pm on August 31, 2021: contributor

Maybe I’m showing my inexperience, but here it goes…

I’m reviewing #22838 and don’t really like the idea of making descriptors more complex than they already are (and to me this seems like a narrow use-case that is already pretty straightforward to do with descriptors as-is). So it’s not like we’re gaining extra functionality with this change.

From my view, in its current form, multipath descriptors would add more complexity to make the UX feel nice/“correct”, but is really putting makeup on some (weird?) underlying design decisions: why do we need this convention of derivation paths at {xpub}/0/* and {xpub}/1/* to differentiate between receiving and change (internal) addresses? Since pretty much every wallet deals with changes and receive addresses in this way, couldn’t we just always consider even paths range_index % 2 == 0 (ie {xpub}/0/0, {xpub}/0/2, …) receiving, and odd indexes ({xpub}/0/1, {xpub}/0/3, …) change/internal? I don’t see how this imposes any limitations or additional complexity, and seems to fit better with the boolean nature of internal == true/false. At that point we have one simple descriptor, and I think this aligns more with what users would actually expect to see (why would they need to do anything extra to handle change? Even a multipath <0;1> in the descriptor is extra, and seems like unnecessary details for pretty much everyone but those working on the implementation of descriptor wallets).

Reading through the descriptor/wallet code, I don’t understand why we need both:

0std::map<OutputType, ScriptPubKeyMan*> m_external_spk_managers;
1std::map<OutputType, ScriptPubKeyMan*> m_internal_spk_managers;

The differentiation between the two doesn’t make much sense to me, as it seems to just come down to “when we receive to this address do we show it as an incoming transaction, or is this just change from a send we did?” So I’m also wondering if there’s an opportunity to simplify/refactor code instead of make it more complicated.

I have no idea how entrenched these conventions are, or maybe I’m missing some implementation details that make my idea impractical. Please lmk!

Reading through the discussion here, @benma seemed to have somewhat similar thoughts as me, and took an approach that wouldn’t require any changes to core (although my idea would involve a deprecation process and refactor):

Since pretty much every wallet deals with changes and receive addresses in this way, I don’t see it as very useful track two separate descriptors, or even to put two separate derivation paths into one descriptor. Sounds easiest to stick to the convention of ./1/* and ./0/* for change and receive addresses.

achow101 commented at 5:53 pm on August 31, 2021: member

@mjdietzx

The main reason for multipath descriptors is that people may want to have their change addresses derived at different derivation paths. For example, it may be desirable to have change addresses use a hardened derivation path while receiving addresses are unhardened.

It is important to keep in mind that this is not really to deal with the default/usual case and pattern (i.e. the descriptors that the wallet will generate) but rather with unusual and custom descriptors that advanced users may want to use. The name of the game is flexibility.

From my view, in its current form, multipath descriptors would add more complexity to make the UX feel nice/“correct”, but is really putting makeup on some (weird?) underlying design decisions: why do we need this convention of derivation paths at {xpub}/0/* and {xpub}/1/* to differentiate between receiving and change (internal) addresses? Since pretty much every wallet deals with changes and receive addresses in this way, couldn’t we just always consider even paths range_index % 2 == 0 (ie {xpub}/0/0, {xpub}/0/2, …) receiving, and odd indexes ({xpub}/0/1, {xpub}/0/3, …) change/internal? I don’t see how this imposes any limitations or additional complexity, and seems to fit better with the boolean nature of internal == true/false.

That is a possible solution, but it really limits flexibility. It would make it more difficult to have a completely different descriptor for change as well as a different derivation path. Again, this is not a problem for the automatically generated descriptor but rather if a user were to import something non-standard.

Reading through the descriptor/wallet code, I don’t understand why we need both:
0std::map<OutputType, ScriptPubKeyMan*> m_external_spk_managers;
1std::map<OutputType, ScriptPubKeyMan*> m_internal_spk_managers;
The differentiation between the two doesn’t make much sense to me, as it seems to just come down to “when we receive to this address do we show it as an incoming transaction, or is this just change from a send we did?” So I’m also wondering if there’s an opportunity to simplify/refactor code instead of make it more complicated.

The purpose of this particular split is to be able to retrieve receiving and change addresses for the correct type. It facilitates having completely unrelated and wildly different descriptors for receiving and change. The user is not locked into a particular pattern if they don’t want to be.

Reading through the discussion here, @benma seemed to have somewhat similar thoughts as me, and took an approach that wouldn’t require any changes to core (although my idea would involve a deprecation process and refactor):

Since pretty much every wallet deals with changes and receive addresses in this way, I don’t see it as very useful track two separate descriptors, or even to put two separate derivation paths into one descriptor. Sounds easiest to stick to the convention of ./1/* and ./0/* for change and receive addresses.

I don’t like this particular idea because it is an implicit conversion which goes against the entire ethos of descriptors - everything is explicitly stated. Furthermore, such an implicit conversion assumes that derivation paths follow a particular pattern and it will tend to break down when someone does something different.

mjdietzx commented at 8:45 pm on August 31, 2021: contributor

Thanks for the explanation, makes sense and gets me close to a Concept/Approach ack. One last thought, hopefully this either crosses the idea out or lets me know if it’s worth looking into more:

If we were to implement the “rule”:

even derivation paths range_index % 2 == 0 (ie {xpub}/*/0, {xpub}/*/2, …) are always receiving
odd paths (ie {xpub}/*/1, {xpub}/*/3, …) are always change/internal

I would think there’s a way to do this that doesn’t limit flexibility or have significant downsides. Let’s say we add an RPC option: include_change

Then, going with the multisig example from #22067:

 0multisig = node.get_wallet_rpc(f"{self.name}_{i}")
 1external = multisig.getdescriptorinfo(f"wsh(sortedmulti({self.M},{f'/0/*,'.join(xpubs)}/0/*))")
 2internal = multisig.getdescriptorinfo(f"wsh(sortedmulti({self.M},{f'/1/*,'.join(xpubs)}/1m/*))").  # hardened
 3result = multisig.importdescriptors([
 4  {  # receiving addresses (internal: False)
 5      "desc": external["descriptor"],
 6      "internal": False,
 7      "include_change": False,
 8  }
 9  { # change addresses (internal: True)
10      "desc": internal["descriptor"],
11      "internal": True,
12      "include_change": False,
13   },
14])

So this shows that since include_change is False we can still accomplish this with two descriptors, same as before. This highlights that it’s still simple to have a completely different descriptor for change as well as a different derivation path.

But, we could also now do:

0multisig = node.get_wallet_rpc(f"{self.name}_{i}")
1descriptor = multisig.getdescriptorinfo(f"wsh(sortedmulti({self.M},{f'/0/*,'.join(xpubs)}/0/*))")
2result = multisig.importdescriptors([
3  {  # both change and receiving addresses
4      "desc": descriptor["descriptor"],
5      "include_change": True,  # maybe this defaults to `True` to make it even more intuitive for the average user?
6  },
7])

To me, it seems this is a win in terms of UX (even when compared to multipath descriptors in many cases). And may allow us to simplify some of the wallet / signer / descriptor code. It also may be a solid convention in general to build on for the wallet ecosystem?

Important to note, that even when include_change is False, receiving addresses will only ever use even paths and change/internal will only use odd paths. So the derivation paths actually used in the range is only ever half utilized as (even/odd paths are skipped over depending on receiving/change). But I don’t see a practical reason that would be a problem.

Either way I’ll definitely be finishing my review of #22838 and have some ideas how I want to test it. But it’s a pretty heavy change so I want to push back and make it’s the best way forward

Rspigler commented at 5:36 am on September 1, 2021: contributor

I agree with @achow101 as discussed earlier:

Finding a way to represent multiple derivation paths and discouraging use of separate receive and change descriptors aren’t mutually exclusive. We could do both.

However, I don’t think even/odd is the way to do it.

instagibbs commented at 0:53 am on September 22, 2021: member

This scenario only applies to restored wallets, and wallets loaded in multiple places (which is inadvisable).

Having multiple (watch-only) wallets is industry standard and advisable.

We also register the multisig setup in the device. Having one descriptor correspond to one multisig account makes things easier than to register multiple descriptors (one for receive, one for change), as the user has to verify less data.

There’s no reason that the user has to verify both if the firmware checks that the only difference is in the expected spots(e.g., the “changeness” derivaiton part)

I have likely given this less thought than many of you here, but I’m -0 on the idea. I feel like most of the application issues can be solved at another layer, or should at least be attempted first before complicating things further for specific(albeit popular!) use-cases.

craigraw commented at 1:44 pm on November 13, 2021: none

Sparrow Wallet will in future releases display wallet output descriptors using the <0;1> multipath syntax to specify receive and change indexes in one descriptor as per #22838.

laanwj commented at 7:48 pm on December 17, 2021: member

I have likely given this less thought than many of you here, but I’m -0 on the idea. I feel like most of the application issues can be solved at another layer, or should at least be attempted first before complicating things further for specific(albeit popular!) use-cases.

I had the same concern while reviewing #22838. At first glance, defining a special syntax to specify two descriptors in one string, even if it’s a very common use case, seems to be to be a bit in conflict with the original goal of descriptors to be a general language.

A syntax to specify multiple (labeled?) descriptors (could be one for receiving, one for change, in this case) in one copy/paste string would be more general.

So also ~0.

craigraw commented at 6:55 am on December 18, 2021: none

Given that:

It is recommended to make a backup of a wallet’s description (particularly in the case of multisig wallet);
An output descriptor is the most widely accepted and standards compliant way to do so;
Probably 99% of use cases use receive and change chains;
It is generally advised to write out such privacy sensitive information rather than making a unencrypted digital copy

I think there is a case to be made for this “special” syntax. I find it is already difficult to convince users to write out a long multisig output descriptor. Asking them to write what is almost the same descriptor twice will be even more difficult, leading to more likelihood of loss of funds. This situation is even worse when more durable media are considered.

Rspigler commented at 7:59 pm on December 21, 2021: contributor

I feel like most of the application issues can be solved at another layer

I’ve heard this a few times, but I guess I don’t fully understand it.

A syntax to specify multiple (labeled?) descriptors (could be one for receiving, one for change, in this case) in one copy/paste string would be more general.

Can the current PR not be viewed this way?

Sjors commented at 4:01 pm on January 28, 2022: member

I tend to agree with @craigraw that receive/change is a common and old enough pattern to warrant a special case. And having paper backups of the full string is useful, and a big reason why we bothered with a checksum in the first place. This could be done implicitly though, by only communicating the receive descriptor. But it seems much safer to slightly complicate the descriptor standard to add support for n derivation paths.

The <NUM;NUM> format used in #22838, or the {NUM,NUM} format above, can trivially be expanded to arbitrary length later. If constraining it to 2 items for now makes implementations easier, that’s fine by me.

One thing to consider is the question of whether we should leave unused characters alone for future use. Any descriptor parser already needs to be significantly smarter than just looking for the presence of a specific character, so it probably doesn’t matter what we pick here. That said, I don’t find the argument for the current choice in #22838 (comment) particularly strong. And it seems several wallets have already jumped the gun on <NUM;NUM>.

As for multiple accounts: I think it’s OK for users and wallet software to treat those as separate wallets. You probably wouldn’t want to import multiple accounts into a single Bitcoin Core wallet, since it won’t segregate the funds. So I can live with not having a nested version of this. But again it seems possible to add that later, without making the syntax too atrocious. That may actually be an argument in favor of < ; over { ,, since it won’t be ambiguous with multi().

benma commented at 6:45 am on March 20, 2023: none

Here is another use case to consider:

The BitBoxApp merges wpkh(xpub1/<0;1>/*) and tr(xpub2/<0;1>/*) into a single user-facing account, so users don’t have to worry about low level things like choosing which script type to use.

While The <0;1> syntax is great to merge the receive and change chains into a single output descriptor, it would be great if one could also represent keys derived from different xpubs and script types in a single output descriptor. BIP88 doesn’t seem to cover that.

A descriptor like this could be imported into watch-only wallets and other tools, solving the same problems listed throughout this issue that exist for the receive/change chains.

Maybe a new top-level descriptor function could work for this, like combined(wpkh(...),tr(...)). The existing combo() function doesn’t fit the bill as it covers too much (e.g. legacy P2PK) and too little (no P2TR?), they are at one pubkey (not BIP44-compatible) and its specification can change over time, while descriptors should remain static.

achow101 referenced this in commit cbf385058b on Jul 27, 2023

sidhujag referenced this in commit b11d694c10 on Aug 9, 2023

seedhammer commented at 8:38 pm on September 1, 2023: none

FWIW, at SeedHammer we engrave descriptors on steel, and would very much appreciate a standard and compact way of representing the special, but common, case of a pair of receive, change descriptors.

descriptors: represent multiple derivation paths within one descriptor #17190