BIP 178: Version Extended WIF

kallewoof commented at 4:00 am on April 12, 2018: member

laanwj commented at 4:04 am on April 12, 2018: member

ACK (we should probably summarize the disucssion on IRC here)

Changing this to a Concept ACK for now, as there may also be a good argument to switch to a deliberately incompatible WIF format for the new key types: because other implementations handle non-1 suffix bytes in different ways. They don’t all explicitly reject them, or even treat them as compressed.

For example bitcoin-python https://github.com/petertodd/python-bitcoinlib/blob/master/bitcoin/wallet.py#L253 treats only keys with suffix byte AND suffix byte=1 as compressed and the rest as uncompressed. @petertodd

Not sure how much of a problem this is, but it seems like an edge case that might result in fund loss (or temporary panic at least…).

luke-jr renamed this:
~~[BIP Proposal] Typed WIF (private key format)~~
BIP 178: Typed Private Keys
on Apr 12, 2018

luke-jr added the label New BIP on Apr 12, 2018

luke-jr commented at 4:08 am on April 12, 2018: member

(It may be desirable to change Title?)

dcousens approved

dcousens commented at 4:09 am on April 12, 2018: contributor

ACK

dcousens cross-referenced this on Apr 12, 2018 from issue Support ypub/zprv, plus custom addresses by afk11

kallewoof force-pushed on Apr 12, 2018

dcousens cross-referenced this on Apr 12, 2018 from issue Support for BIP49 YPUB/UPUB by plasticlobster

kallewoof force-pushed on Apr 12, 2018

kallewoof commented at 4:19 am on April 12, 2018: member

@luke-jr You mean “Typed Private Keys”? It seemed good enough to me, but maybe “Extended Wallet Import Format”?

kallewoof force-pushed on Apr 12, 2018

kallewoof cross-referenced this on Apr 12, 2018 from issue wallet: Treat any key with a suffix as compressed by kallewoof

kallewoof commented at 4:40 am on April 12, 2018: member

An alternative to this approach is to use bech32 for the private keys as well, i.e. switch to a whole new format and skip trying to be backwards-compatibility.

Pro: clean new format with benefits.
Con: may take time, as I believe @sipa is still researching how to do this in optimal ways.

dcousens commented at 4:43 am on April 12, 2018: contributor

An alternative to this approach is to use bech32 for the private keys as well, i.e. switch to a whole new format and skip trying to be backwards-compatibility.

Pro: potential HD key support

I know this is the source of my contention, and as a library implementor, users would love this BIP, but complain about no standard solution for BIP32 wallets.

NicolasDorier commented at 5:17 am on April 12, 2018: contributor

I think there is no reason users would need to deal with individual private key anymore.

Though a standard for this for HD private and public keys would be useful. The way I am dealing with it now is by requesting users to specify the HD key + the first address of his wallet, and I try to know what type of derivation to use.

kallewoof commented at 5:24 am on April 12, 2018: member

I was importing private keys from one node into another, which is what made me interested in this in the first place, so I think this is still a useful feature. Electrum added their own suffixes for keys, and other wallet software may follow suite, so a BIP for this seems reasonable.

Current wallet implementations (incomplete list):

python-bitcoinlib: incompatible, but trivially fixed (compressed == len>32 && value[32] == 0x01)
btcd: incompatible, but trivially fixed (same assumption)
NBitcoin: incompatible, but trivially fixed (same assumption)

dcousens commented at 5:27 am on April 12, 2018: contributor

Npm package wif would be trivial too.

If the alternative to this BIP is 100 years of bike-shedding over BIP32 structures, I’d rather not. This BIP is still useful in isolation, and it might at least set a standard for the compression/script template encoding which could be used for BIP32 structures, if desired.

kallewoof force-pushed on Apr 12, 2018

NicolasDorier commented at 10:37 am on April 12, 2018: contributor

I would support it on NBitcoin side.

clarkmoody commented at 2:22 pm on April 12, 2018: contributor

Nit: Use MediaWiki formatting for the *is*

Concept ACK from me, and keep the BIP focused on single-address keys. SLIP-0032 already proposes a solution for an upgrade to HD seeds.

in bip-0178.mediawiki:47 in 0060780d77 outdated

42+|-
43+|<code>0x11</code>||P2WPKH||Yes||Bech32 format (native Segwit)
44+|-
45+|<code>0x12</code>||P2WPKH_P2SH||Yes||Segwit nested in BIP16 P2SH (<code>3...</code>)
46+|-
47+|<code>0x13</code>||P2SH|| &mdash; ||Non-Segwit BIP16 P2SH (<code>3...</code>)

afk11 commented at 2:52 pm on April 12, 2018:

What script type does this refer to? The slip32 prefixes don’t allow arbitrary redeem scripts (it would defeat the point IMO). If this is meant to be P2PKH_P2SH, please refer to it as that!

dcousens commented at 9:25 pm on April 12, 2018:

@afk11 IMHO it should be P2SH_P2PKH, as better represents P2SH(P2PKH)

kallewoof commented at 5:35 am on April 13, 2018:

I believe this would be used for arbitrary P2SH scripts with a related private key (e.g. multisig). Is there a reason to even have a P2PKH in P2SH variant?

dcousens commented at 5:45 am on April 13, 2018:

@kallewoof given the lack of redeem script context, wouldn’t this be implemented as an ‘unsupported’ flag by most implementations? Or is half-way-there still better than nothing?

Otherwise we are assuming the context and it is no better than providing a standard WIF?

kallewoof commented at 5:53 am on April 13, 2018:

Well, if I make a custom multisig with someone else, and I export the private key, even if I don’t know the redeem script itself, it may be useful to know what the key was used for. An implementation that doesn’t support custom redeem scripts would probably keep it unimplemented, but that’s a good thing, right? They can’t spend the UTXOs anyway.

dcousens commented at 6:09 am on April 13, 2018:

@kallewoof but they have no assurance that the key is actually using their redeem scripts. Its a back-to-square one issue, an implementation could always check every type to see if there are transactions on those addresses, but this BIP was about marking the WIF with an encoded ‘guarantee’ to reduce that, no?

kallewoof commented at 6:16 am on April 13, 2018:

Yes, that is true. For completeness sake, there is something to be gained from having every type listed, even if you don’t have all the details. Contrarily, having a custom script with an associated private key, which type would it be, if we removed the P2SH type?

sipa commented at 8:41 am on April 13, 2018:

@kallewoof I think there is no point in having a flag to say “this is some form of P2SH but you don’t know what”. If you’re going to need to give additional information anyway, you might as well just give the private key in normal WIF format plus whatever other information is needed.

My view is that this typed WIF format is really just a shorthand notation for “here is a private key, and I’ve conveniently packed the address alongside it”. If using the shorthand doesn’t let you get rid of additional information, don’t use it.

kallewoof commented at 9:19 am on April 13, 2018:

So basically, use 0x01 for keys that are in custom stuff? That means wallets will import all types rather than import nothing, knowing they will never see the UTXO.

Presuming the user wants to store the private key separately. I guess you’re saying they shouldn’t.

sipa commented at 9:24 am on April 13, 2018:

Yes, my point is that if you’re using a custom script/address, you must have more information. There is simply no way you could hope to have everything in one string. And there is no need for that, I think.

In the general case, you will provide your wallet with a bunch of information to describe what outputs you mean, and which private keys are relevant for it. In a few exceptionally simple (but remarkably common) cases, you get to put everything in one string.

kallewoof commented at 9:29 am on April 13, 2018:

So, conclusively, do not include types that we won’t be able to use without additional information anyway. I.e. remove P2SH, P2WSH, and P2WSH_P2SH, and keep the rest.

kallewoof commented at 9:29 am on April 13, 2018:

And then use 0x01=P2PKH_COMPRESSED for ‘custom stuff’ exactly like is done today.

kallewoof commented at 9:52 am on April 13, 2018:

Updated BIP, removing the 3 script-dependent types.

afk11 commented at 12:10 pm on April 13, 2018:

@kallewoof I think having script-hash types is fine once the flag tells you what the redeemscript is. I see no harm in having P2SH_P2WPKH, P2WSH_P2PKH, and if we support producing those scriptPubKeys. Script hash scripts have no degrees of freedom besides the script they commit to, so IMO supporting script-hash extensions is pretty natural.

We could leave it the way it was before, assuming they refer to P2PKH.. Or we could use bits 2 and 3 to indicate P2SH and P2WSH, and maybe use others to indicate the scriptPubKey type?

Another, is just ignore script-hash usage entirely, and assume the software will deal with it.

in bip-0178.mediawiki:51 in 0060780d77 outdated

46+|-
47+|<code>0x13</code>||P2SH|| &mdash; ||Non-Segwit BIP16 P2SH (<code>3...</code>)
48+|-
49+|<code>0x14</code>||P2WSH|| &mdash; ||Native Segwit P2SH
50+|-
51+|<code>0x15</code>||P2WSH_P2SH|| &mdash; ||Native Segwit P2SH nested in BIP16 P2SH

afk11 commented at 2:53 pm on April 12, 2018:

See above - both of these are script-hash, not script types to be run.

in bip-0178.mediawiki:49 in 0060780d77 outdated

44+|-
45+|<code>0x12</code>||P2WPKH_P2SH||Yes||Segwit nested in BIP16 P2SH (<code>3...</code>)
46+|-
47+|<code>0x13</code>||P2SH|| &mdash; ||Non-Segwit BIP16 P2SH (<code>3...</code>)
48+|-
49+|<code>0x14</code>||P2WSH|| &mdash; ||Native Segwit P2SH

afk11 commented at 2:54 pm on April 12, 2018:

see above - is this P2WSH P2PKH?

afk11 commented at 3:25 pm on April 12, 2018: contributor

Concept ACK - in a similar vein to key compression, think things are a lot simpler if we know the script type from it’s encoding.

I do think we should specify the redeemScript types in places where we just use ‘P2SH’ or ‘P2WSH’, if it’s P2PKH we should be clear about it.

afk11 commented at 3:27 pm on April 12, 2018: contributor

It does seem there’s a dependency between the prefix spec for BIP32, and this BIP. I guess implementations should fail if someone attempts to use BIP32 a SLIP132 prefix for which there is no WIF prefix?

If someone were to export a WIF from a SLIP132 key, they would need to be in sync.

kallewoof commented at 9:59 am on April 13, 2018: member

@clarkmoody

Nit: Use MediaWiki formatting for the *is*

I’m not sure what you mean.

in bip-0178.mediawiki:41 in 86dd0da70f outdated

36+|-
37+|No suffix||P2PKH_UNCOMPRESSED||No||Uncompressed legacy public key. Unknown public key format
38+|-
39+|<code>0x01</code>||P2PKH_COMPRESSED||Yes||Compressed legacy public key. Unknown public key format
40+|-
41+|<code>0x10</code>||P2PKH||Yes||Compressed legacy public key. Legacy public key format (<code>1...</code>)

dcousens commented at 10:09 am on April 13, 2018:

What is the semantic difference between 0x01 and 0x10 here?

Unknown public key format

?

kallewoof commented at 2:06 am on April 16, 2018:

@dcousens 0x10 means it is a 1.. key, and nothing else. 0x01 means it can be any kind of key (i.e. the key was made before the BIP-178 was implemented so you don’t know what kind).

dcousens commented at 2:09 am on April 16, 2018:

@kallewoof a 1... key… so P2PKH?

Are you suggesting that this is for users who generated the uncompressed form from the compressed key given, and therefore the targeted compression [for the address] is ambiguous? If so, ACK, but, that wasn’t clear to me from the text.

kallewoof commented at 2:23 am on April 16, 2018:

Yup. The 0x01 is “old format, we don’t know what kind it is”. The 0x10 is “this is a legacy non-segwit p2pkh”. I’ll update to clarify.

dcousens commented at 10:13 am on April 13, 2018: contributor

With the limited scope limited only to N=1 public-key scripts, I do wonder if a different format (bech32 encoded maybe) wouldn’t be simpler and safer for implementations compared to incompatibly changing the WIF format. The UX for old software might be immensely confusing.

kallewoof commented at 2:07 am on April 16, 2018: member

@dcousens The idea is to make an upgrade to WIF that is fairly simple to do (one line change in all cases I’ve seen so far). I was hoping 0-line but most people seem to do “len > 32 & v[32] == 0x01”. If they dropped the latter part for “compressed” it would be compatible.

kallewoof force-pushed on Apr 18, 2018

clarkmoody commented at 3:00 pm on April 18, 2018: contributor

@kallewoof Surrounding text with asterisks does not produce any different formatting under MediaWiki rules. I suggest changing depending on what formatting you wish to achieve.

kallewoof commented at 11:19 pm on April 18, 2018: member

@clarkmoody D’oh. Thanks, fixed!

afk11 commented at 1:39 pm on April 19, 2018: contributor

Hmm, any reason against including the script types exposed in slip132? It’s possible people would be dumping WIF’s for those keys, and lacking a defined script type for them (P2SH_P2WSH_P2PKH and P2WSH_P2PKH) people would just use \x01

I did see discussion about script-hash types, but it the discussion seemed to be against arbitrary redeem & witness script types, these would explicitly be P2PKH.

kallewoof commented at 2:40 am on April 20, 2018: member

@afk11 I would like the list to be complete for types which represent a single key, so yes, these should be added I think. Are there any other combinations?

Edit: I can’t find these in slip-132 anywhere. What are they used for, anyway? Seems redundant, but maybe I’m missing something.

kallewoof cross-referenced this on May 9, 2018 from issue BIP-178: Extend WIF format with key type by kallewoof

afk11 commented at 6:23 pm on May 16, 2018: contributor

@kallewoof sorry for the delay! thought I’d got back to this already. First I was sure P2SH and P2SH_P2WSH meant P2PKH, but it seems electrum uses those for multi-signature scripts https://github.com/spesmilo/electrum-docs/blob/master/xpub_version_bytes.rst#specification so I can’t say for sure that’s the case. @clarkmoody is it intended that P2SH and P2WSH don’t have known type in slip132? P2WPKH is mentioned specifically by name in all forms, but P2PKH is only mentioned in the first entry.

It really would be nice, as then P2WPKH and P2PKH (in all script-hash cases) would have a prefix.

clarkmoody commented at 10:15 pm on May 16, 2018: contributor

@afk11 SLIP-0132 is mainly following the lead of Electrum in the x- y- z-pub version bytes. P2WSH is listed as 0x02aa7ed3 - Zpub, but you are correct, there is no specific HD prefix for P2SH addresses.

kallewoof commented at 3:19 am on May 17, 2018: member

@afk11 So the conclusion here is that no new entries need to be added, right?

~~@luke-jr can we merge this so people can make PR against it if they feel any more changes are needed?~~

afk11 commented at 2:58 pm on May 17, 2018: contributor

@clarkmoody OK, thanks for answering! (off topic, but it might be worth mentioning on slip132 that P2SH, P2WSH, P2SH|P2WSH refer to BIP11 scripts) @kallewoof - yep, all good.

petertodd cross-referenced this on May 17, 2018 from issue Potential WIF format incompatibility by petertodd

Janaka-Steph commented at 9:07 am on May 18, 2018: none

I am very confused by the usage of the term “public key” in this BIP. An ECDSA public key will always remain a public key, it can be uncompressed or compressed, but there is not different kinds. It seems to me that the term “bitcoin address” should be use instead.

There are several types of public keys which can all be associated with a given private key: P2PKH (legacy 1… format), P2SH-P2WPKH (SegWit public key inside P2SH), P2WPKH (bech32), etc.

We should say “There are several types of bitcoin addresses…”

It’s even more confusing when we are talking about pay to script hash.

kallewoof commented at 9:15 am on May 18, 2018: member

We should say “There are several types of bitcoin addresses…”

While a compressed public key and an uncompressed public key are different types of public keys, the same doesn’t really go for the other kinds, so yeah, I overall agree with your point. I think “bitcoin addresses” makes more sense. Will fix.

Janaka-Steph commented at 9:33 am on May 18, 2018: none

Thank you for the quick fix. I would like to express the same criticism regarding the term “Typed Private Keys”. We use to say WIF-compressed to express the fact that a private key must be derived to a compressed public key. The attribute “compressed” is attached to the Wallet Import Format, not directly the private key. So imo we should say something like “Address-typed WIF” instead.

kallewoof commented at 9:36 am on May 18, 2018: member

I’m not sure I agree. The format describes a private key, with a type, so “typed private key” isn’t too far off, I think. Will think more about it.

Janaka-Steph commented at 10:32 am on May 18, 2018: none

What about “Version bytes extended WIF”?
It seems to me that this idea of describing the address type (according to the output type) into the WIF is similar to the Electrum Seed Version System or the versioning of Aezeed by LND. “Version bytes” is the keyword I would use to search about this concept on the net. But in the meantime I think it is too vague, I would prefer something more descriptive like “Output type version bytes extended WIF”.

kallewoof commented at 11:46 am on May 18, 2018: member

It’s not really a version, I think. “Type extended WIF”? “Address type extended WIF”?

Janaka-Steph commented at 12:49 pm on May 18, 2018: none

BIP32 defines the extended key serialization format with 4 bytes “version bytes” to encode the network (https://github.com/bitcoin/bips/blob/master/bip-0032.mediawiki#serialization-format). Then we have SLIP132 which suggest to use these version bytes to also describe the address format (https://github.com/satoshilabs/slips/blob/master/slip-0132.md). If my understanding is correct, SLIP132 is trying to solve the exact same problem. This is why I have suggested the term “version bytes” for BIP178. Because the term is loaded with this historical BIP32 background, and also to set a common ground with SLIP132.

kallewoof commented at 1:59 am on May 19, 2018: member

That’s a good point, yeah. “Version extended WIF”? ‘bytes’ seems unnecessary (and it’s only one byte).

“VEW” is a horrible abbreviation though. “XWIF”? Hum.

luke-jr commented at 8:30 am on May 20, 2018: member

Is this ready for merging?

kallewoof renamed this:
~~BIP 178: Typed Private Keys~~
BIP 178: Version Extended WIF
on May 20, 2018

kallewoof force-pushed on May 20, 2018

[BIP-178] Version Extended WIF. f667e7a3c7

kallewoof force-pushed on May 20, 2018

kallewoof commented at 11:33 pm on May 20, 2018: member

@luke-jr Yes, I believe so.

luke-jr merged this on May 20, 2018

luke-jr closed this on May 20, 2018

kallewoof deleted the branch on May 20, 2018

Janaka-Steph commented at 8:25 am on May 21, 2018: none

“Version extended WIF” sounds good to me, but I don’t know, it would be good to have more inputs.

Janaka-Steph commented at 8:30 am on May 21, 2018: none

@kallewoof The abstract still says “…to specify what kind of public key the private key corresponds to.”, instead of “bitcoin addresses”. Sorry to bother you with that but it is an important detail.

afk11 commented at 12:29 pm on May 21, 2018: contributor

@Janaka-Steph maybe PR the changes you feel are necessary so they can be reviewed / ACKed if ok? the PR has already been merged :)

kallewoof commented at 11:19 pm on May 21, 2018: member

I’m PR’ing the change now. Thanks for the pointer, @Janaka-Steph.

kallewoof commented at 11:22 pm on May 21, 2018: member

#683

BIP 178: Version Extended WIF #673