Revisions to BIP 174 #694

achow101 commented at 2:55 AM on June 28, 2018: member

Describes the changes to BIP 174 as discussed on the bitcoin-dev mailing list. There are new tests, updated formatting, and clarifications.

achow101 cross-referenced this on Jun 28, 2018 from issue BIP 174 PSBT Serializations and RPCs by achow101

in bip-0174.mediawiki:303 in bbfe04cbe8 outdated

 305 | +
 306 |  ===Combiner===
 307 |  
 308 |  The Combiner can accept 1 or many PSBTs.
 309 |  The Combiner must merge them into one PSBT (if possible), or fail.
 310 |  The resulting PSBT must contains all of the key-value pairs from each of the PSBTs.

sipa commented at 4:39 AM on June 28, 2018:

Typo: contains

achow101 commented at 6:31 PM on June 28, 2018:

Fixed

in bip-0174.mediawiki:304 in bbfe04cbe8 outdated

 307 |  
 308 |  The Combiner can accept 1 or many PSBTs.
 309 |  The Combiner must merge them into one PSBT (if possible), or fail.
 310 |  The resulting PSBT must contains all of the key-value pairs from each of the PSBTs.
 311 | -The Combined must remove any duplicate key-value pairs, in accordance with the specification.
 312 | +The Combiner must remove any duplicate key-value pairs, in accordance with the specification.

sipa commented at 4:39 AM on June 28, 2018:

Specify that it can pick arbitrarily in case of conflicts.

achow101 commented at 6:31 PM on June 28, 2018:

Fixed

in bip-0174.mediawiki:313 in bbfe04cbe8 outdated

 321 |  
 322 | -For any inputs which are not complete, the Finalizer will emplace an empty scriptSig in the network serialized transaction.
 323 | -For any input which has a complete set of signatures, the Finalizer must attempt to build the complete scriptSig and encode it into the network serialized transaction.
 324 | +The Input Finalizer must only accept a PSBT.
 325 | +For each input, the Input Finalizer determines if the input has enough data to pass validation. If it does, it must construct the scriptSig and scriptWitness and place them into the input key-value map.
 326 | +All other data except the UTXO in the input key-value map should be cleared from the PSBT. The UTXO should be kept to allow Transaction Extractors to verify the final network serialized transaction.

sipa commented at 4:40 AM on June 28, 2018:

Specify that it should leave unknown fields untouched (these may serve a similar purpose to UTXOs in extensions).

achow101 commented at 6:34 PM on June 28, 2018:

Done

in bip-0174.mediawiki:343 in bbfe04cbe8 outdated

 337 |  
 338 |  ==Extensibility==
 339 |  
 340 |  The Partially Signed Transaction format can be extended in the future by adding
 341 |  new types for key-value pairs. Backwards compatibilty will still be maintained as those new
 342 |  types will be ignored and passed-through by signers which do not know about them.

sipa commented at 4:42 AM on June 28, 2018:

Perhaps mention that if ever the number of field types may run out, new extensions could start defining multi-byte type identifiers.

achow101 commented at 6:31 PM on June 28, 2018:

Done

in bip-0174.mediawiki:330 in 7276fd15a3 outdated

 321 | @@ -322,6 +322,13 @@ The Transaction Extractor does not need to know how to interpret scripts in orde
 322 |  
 323 |  A single entity is likely to be both a Transaction Extractor and an Input Finalizer.
 324 |  
 325 | +==Encoding==
 326 | +
 327 | +A PSBT can be represented in two ways: in binary (as a file) or as a Base64 string using the encoding described in [https://tools.ietf.org/html/rfc4648#section-4 RFC4648].
 328 | +
 329 | +Binary PSBT files should use the <tt>.psbt</tt> file extension.
 330 | +The MIME type <tt>application/psbt</tt> can be used.

sipa commented at 4:48 AM on June 28, 2018:

I don't think you can just start using a mime type without registration.

Practices around mime types are defined in https://tools.ietf.org/html/rfc6838, but I think it may be too early to go through the registration processes. I'm not sure what to do here, but perhaps just write "A mime type name will be added to this document after blabla"?

achow101 commented at 6:34 PM on June 28, 2018:

Done

in bip-0174.mediawiki:117 in f086dffa3a outdated

 205 | +The currently defined global types are as follows:
 206 | +
 207 | +* Type: Unsigned Transaction <tt>PSBT_GLOBAL_UNSIGNED_TX = 0x00</tt>
 208 | +** Key: None. The key must only contain the 1 byte type.
 209 | +*** <tt>{0x00}</tt>
 210 | +** Value: The transaction in network serialization. The scriptSigs and witnesses for each input must be empty. The transaction must be in the witness serialization format as defined in BIP 144. A PSBT must have a transaction, otherwise it is invalid.

sipa commented at 4:50 AM on June 28, 2018:

If all witnesses are empty, BIP144 does not permit using extended serialization.

achow101 commented at 6:31 PM on June 28, 2018:

Fixed

in bip-0174.mediawiki:126 in f086dffa3a outdated

 291 | -</pre>
 292 | -|}
 293 | +* Type: Non-Witness UTXO <tt>PSBT_IN_NON_WITNESS_UTXO = 0x00</tt>
 294 | +** Key: None. The key must only contain the 1 byte type.
 295 | +***<tt>{0x00}</tt>
 296 | +** Value: The transaction in network serialization format the current input spends from.

sipa commented at 4:51 AM on June 28, 2018:

Add a note that it may only be present for inputs which spend non-segwit outputs.

achow101 commented at 6:31 PM on June 28, 2018:

Fixed

in bip-0174.mediawiki:132 in f086dffa3a outdated

 297 | +*** <tt>{transaction}</tt>
 298 | +
 299 | +* Type: Witness UTXO <tt>PSBT_IN_WITNESS_UTXO = 0x01</tt>
 300 | +** Key: None. The key must only contain the 1 byte type.
 301 | +*** <tt>{0x01}</tt>
 302 | +** Value: The entire transaction output in network serialization which the current input spends from.

sipa commented at 4:52 AM on June 28, 2018:

Add a note that it may only be present for inputs which spend segwit outputs (including P2SH embedded ones).

achow101 commented at 6:31 PM on June 28, 2018:

Done

in bip-0174.mediawiki:144 in f086dffa3a outdated

 309 | +*** <tt>{signature}</tt>
 310 | +
 311 | +* Type: Sighash Type <tt>PSBT_IN_SIGHASH_TYPE = 0x03</tt>
 312 | +** Key: None. The key must only contain the 1 byte type.
 313 | +*** <tt>{0x03}</tt>
 314 | +** Value: The 32-bit unsigned integer specifying the sighash type to be used for this input. Signatures for this input must use the sighash type, finalizers must fail to finalize inputs which have signatures that do not use the sighash type. Signers who cannot produce signatures with the sighash type must not provide a signature.

sipa commented at 4:54 AM on June 28, 2018:

"that do not use" is confusing here. Perhaps "If present, finalizers must fail to finalize inputs which have signatures that don't match the specified sighash type" ?

achow101 commented at 6:31 PM on June 28, 2018:

Fixed

in bip-0174.mediawiki:162 in f086dffa3a outdated

 327 | +*** <tt>{witnessScript}</tt>
 328 | +
 329 | +* Type:  BIP 32 Derivation Path <tt>PSBT_IN_BIP32_DERIVATION = 0x06</tt>
 330 | +** Key: The public key
 331 | +*** <tt>{0x06}|{public key}</tt>
 332 | +** Value: The master key fingerprint concatenated with the derivation path of the public key. The derivation path is represented as 32 bit unsigned integer indexes concatenated with each other. This must omit the index of the master key. Public keys are those that will be needed to sign this input.

sipa commented at 4:55 AM on June 28, 2018:

Maybe explain that the term 'fingerprint' is used here as defined by BIP32.

I don't understand "This must omit the index of the master key".

achow101 commented at 6:31 PM on June 28, 2018:

Done

achow101 force-pushed on Jun 28, 2018

in bip-0174.mediawiki:126 in 1874991d8b outdated

 291 | -</pre>
 292 | -|}
 293 | +* Type: Non-Witness UTXO <tt>PSBT_IN_NON_WITNESS_UTXO = 0x00</tt>
 294 | +** Key: None. The key must only contain the 1 byte type.
 295 | +***<tt>{0x00}</tt>
 296 | +** Value: The transaction in network serialization format the current input spends from. This should only be present for inputs which spend non-segwit outputs.

sipa commented at 1:41 AM on June 29, 2018:

After discussing this more, perhaps it should be legal to include a non-witness UTXO even for witness inputs. The entity that adds the UTXO may not know it's a witness one (especially when it's P2SH embedded). But the entity that does know can always convert a PSBT_IN_NON_WITNESS_UTXO into a PSBT_IN_WITNESS_UTXO by extracting the right output.

achow101 commented at 1:50 AM on June 29, 2018:

Done

achow101 force-pushed on Jun 29, 2018

in bip-0174.mediawiki:311 in 414193c900 outdated

 313 |  The Combiner must merge them into one PSBT (if possible), or fail.
 314 | -The resulting PSBT must contains all of the key-value pairs from each of the PSBTs.
 315 | -The Combined must remove any duplicate key-value pairs, in accordance with the specification.
 316 | +The resulting PSBT must contain all of the key-value pairs from each of the PSBTs.
 317 | +The Combiner must remove any duplicate key-value pairs, in accordance with the specification. It can pick arbitrarily when conflicts occur.
 318 | +A Combiner must not combine two different PSBTs. PSBTs can be uniquely identified by <tt>0x00</tt> global transaction typed key-value pair.

sipa commented at 5:41 PM on June 29, 2018:

Perhaps add that a Combiner is allowed to detect inconsistencies in known fields when combining, but cannot do this for unknown fields.

achow101 commented at 6:12 PM on June 29, 2018:

Done

in bip-0174.mediawiki:330 in 414193c900 outdated

 343 | -For any inputs which are not complete, the Finalizer will emplace an empty scriptSig in the network serialized transaction.
 344 | -For any input which has a complete set of signatures, the Finalizer must attempt to build the complete scriptSig and encode it into the network serialized transaction.
 345 | +A PSBT can be represented in two ways: in binary (as a file) or as a Base64 string using the encoding described in [https://tools.ietf.org/html/rfc4648#section-4 RFC4648].
 346 | +
 347 | +Binary PSBT files should use the <tt>.psbt</tt> file extension.
 348 | +A MIME type name will be added to this document once one hs been registered.

sipa commented at 5:46 PM on June 29, 2018:

Typo: hs

achow101 commented at 6:12 PM on June 29, 2018:

Fixed

in bip-0174.mediawiki:517 in 414193c900 outdated

 505 | @@ -522,10 +506,14 @@ Any data types, their associated scope and BIP number must be defined here
 506 |  !BIP Number
 507 |  |-
 508 |  | Global
 509 | -| 0,1,2,3,4
 510 | +| 0

sipa commented at 5:55 PM on June 29, 2018:

Could this be written as a table that includes the name or one-line description of the field? I expect that needing looking things up across BIPs may end up being a pain.

achow101 commented at 6:12 PM on June 29, 2018:

Done

nvk commented at 6:08 PM on June 29, 2018: none

We will start implementing as soon as this gets merged. Thanks for sorting it out so fast 👍

achow101 force-pushed on Jun 29, 2018

in bip-0174.mediawiki:454 in 63da954bf8 outdated

 506 | +An updater which adds SIGHASH_ALL to the above PSBT must create this PSBT:
 507 | +
 508 | +* Bytes in Hex: <pre>70736274ff01009a020000000258e87a21b56daf0c23be8e7070456c336f7cbaa5c8757924f545887bb2abdd750000000000ffffffff838d0427d0ec650a68aa46bb0b098aea4422c071b2ca78352a077959d07cea1d0100000000ffffffff0270aaf00800000000160014d85c2b71d0060b09c9886aeb815e50991dda124d00e1f5050000000016001400aea9a2e5f0f876a588df5546e8742d1d87008f00000000000100bb0200000001aad73931018bd25f84ae400b68848be09db706eac2ac18298babee71ab656f8b0000000048473044022058f6fc7c6a33e1b31548d481c826c015bd30135aad42cd67790dab66d2ad243b02204a1ced2604c6735b6393e5b41691dd78b00f0c5942fb9f751856faa938157dba01feffffff0280f0fa020000000017a9140fb9463421696b82c833af241c78c17ddbde493487d0f20a270100000017a91429ca74f8a08f81999428185c97b5d852e4063f6187650000000104475221029583bf39ae0a609747ad199addd634fa6108559d6c5cd39b4c2183f1ab96e07f2102dab61ff49a14db6a7d02b0cd1fbb78fc4b18312b5b4e54dae4dba2fbfef536d752ae2206029583bf39ae0a609747ad199addd634fa6108559d6c5cd39b4c2183f1ab96e07f10d90c6a4f000000800000008000000080220602dab61ff49a14db6a7d02b0cd1fbb78fc4b18312b5b4e54dae4dba2fbfef536d710d90c6a4f000000800000008001000080010304010000000001012000c2eb0b0000000017a914b7f5faf40e3d40a5a459b1db3535f2b72fa921e88701042200208c2353173743b595dfb4a07b72ba8e42e3797da74e87fe7d9d7497e3b2028903010547522103089dc10c7ac6db54f91329af617333db388cead0c231f723379d1b99030b02dc21023add904f3d6dcf59ddb906b0dee23529b7ffb9ed50e5e86151926860221f0e7352ae2206023add904f3d6dcf59ddb906b0dee23529b7ffb9ed50e5e86151926860221f0e7310d90c6a4f000000800000008003000080220603089dc10c7ac6db54f91329af617333db388cead0c231f723379d1b99030b02dc10d90c6a4f0000008000000080020000800103040100000000220203a9a4c37f5996d3aa25dbac6b570af0650394492942460b354753ed9eeca5877110d90c6a4f000000800000008004000080002202027f6399757d2eff55a136ad02c684b1838b6556e5f1b6b34282a94b6b5005109610d90c6a4f00000080000000800500008000</pre>
 509 | +* Base64 String: <pre>cHNidP8BAJoCAAAAAljoeiG1ba8MI76OcHBFbDNvfLqlyHV5JPVFiHuyq911AAAAAAD/////g40EJ9DsZQpoqka7CwmK6kQiwHGyyng1Kgd5WdB86h0BAAAAAP////8CcKrwCAAAAAAWABTYXCtx0AYLCcmIauuBXlCZHdoSTQDh9QUAAAAAFgAUAK6pouXw+HaliN9VRuh0LR2HAI8AAAAAAAEAuwIAAAABqtc5MQGL0l+ErkALaISL4J23BurCrBgpi6vucatlb4sAAAAASEcwRAIgWPb8fGoz4bMVSNSByCbAFb0wE1qtQs1neQ2rZtKtJDsCIEoc7SYExnNbY5PltBaR3XiwDwxZQvufdRhW+qk4FX26Af7///8CgPD6AgAAAAAXqRQPuUY0IWlrgsgzryQceMF9295JNIfQ8gonAQAAABepFCnKdPigj4GZlCgYXJe12FLkBj9hh2UAAAABBEdSIQKVg785rgpgl0etGZrd1jT6YQhVnWxc05tMIYPxq5bgfyEC2rYf9JoU22p9ArDNH7t4/EsYMStbTlTa5Nui+/71NtdSriIGApWDvzmuCmCXR60Zmt3WNPphCFWdbFzTm0whg/GrluB/ENkMak8AAACAAAAAgAAAAIAiBgLath/0mhTban0CsM0fu3j8SxgxK1tOVNrk26L7/vU21xDZDGpPAAAAgAAAAIABAACAAQMEAQAAAAABASAAwusLAAAAABepFLf1+vQOPUClpFmx2zU18rcvqSHohwEEIgAgjCNTFzdDtZXftKB7crqOQuN5fadOh/59nXSX47ICiQMBBUdSIQMIncEMesbbVPkTKa9hczPbOIzq0MIx9yM3nRuZAwsC3CECOt2QTz1tz1nduQaw3uI1Kbf/ue1Q5ehhUZJoYCIfDnNSriIGAjrdkE89bc9Z3bkGsN7iNSm3/7ntUOXoYVGSaGAiHw5zENkMak8AAACAAAAAgAMAAIAiBgMIncEMesbbVPkTKa9hczPbOIzq0MIx9yM3nRuZAwsC3BDZDGpPAAAAgAAAAIACAACAAQMEAQAAAAAiAgOppMN/WZbTqiXbrGtXCvBlA5RJKUJGCzVHU+2e7KWHcRDZDGpPAAAAgAAAAIAEAACAACICAn9jmXV9Lv9VoTatAsaEsYOLZVbl8bazQoKpS2tQBRCWENkMak8AAACAAAAAgAUAAIAA</pre>
 510 | +
 511 | +Given the above updated PSBT, a signer with the following keys:

sipa commented at 12:20 AM on June 30, 2018:

Specify that this is only the case for signers which support SIGHASH_ALL signing for both P2PKH and P2WPKH spends, and use RFC6979 nonce generation.

achow101 commented at 12:29 AM on June 30, 2018:

Done

achow101 force-pushed on Jun 30, 2018

achow101 force-pushed on Jul 4, 2018

achow101 commented at 8:24 PM on July 4, 2018: member

Clarified that keys must match the format. Clarified that combiners may check for inconsistencies in known types.

luke-jr commented at 5:01 AM on July 5, 2018: member

@achow101 Ready to merge then?

jb55 commented at 6:26 PM on July 5, 2018: none

@achow101 I'm having trouble with my deserializer, as far as I can tell, there is no way to tell when the array of input kv maps stops and the array of kv output maps starts?

with this psbt I got from the revised bip:

70736274ff01009a020000000258e87a21b56daf0c23be8e7070456c336f7cbaa5c8757924f545887bb2abdd750000000000ffffffff838d0427d0ec650a68aa46bb0b098aea4422c071b2ca78352a077959d07cea1d0100000000ffffffff0270aaf00800000000160014d85c2b71d0060b09c9886aeb815e50991dda124d00e1f5050000000016001400aea9a2e5f0f876a588df5546e8742d1d87008f00000000000100bb0200000001aad73931018bd25f84ae400b68848be09db706eac2ac18298babee71ab656f8b0000000048473044022058f6fc7c6a33e1b31548d481c826c015bd30135aad42cd67790dab66d2ad243b02204a1ced2604c6735b6393e5b41691dd78b00f0c5942fb9f751856faa938157dba01feffffff0280f0fa020000000017a9140fb9463421696b82c833af241c78c17ddbde493487d0f20a270100000017a91429ca74f8a08f81999428185c97b5d852e4063f6187650000000104475221029583bf39ae0a609747ad199addd634fa6108559d6c5cd39b4c2183f1ab96e07f2102dab61ff49a14db6a7d02b0cd1fbb78fc4b18312b5b4e54dae4dba2fbfef536d752ae2206029583bf39ae0a609747ad199addd634fa6108559d6c5cd39b4c2183f1ab96e07f10d90c6a4f000000800000008000000080220602dab61ff49a14db6a7d02b0cd1fbb78fc4b18312b5b4e54dae4dba2fbfef536d710d90c6a4f0000008000000080010000800001012000c2eb0b0000000017a914b7f5faf40e3d40a5a459b1db3535f2b72fa921e88701042200208c2353173743b595dfb4a07b72ba8e42e3797da74e87fe7d9d7497e3b2028903010547522103089dc10c7ac6db54f91329af617333db388cead0c231f723379d1b99030b02dc21023add904f3d6dcf59ddb906b0dee23529b7ffb9ed50e5e86151926860221f0e7352ae2206023add904f3d6dcf59ddb906b0dee23529b7ffb9ed50e5e86151926860221f0e7310d90c6a4f000000800000008003000080220603089dc10c7ac6db54f91329af617333db388cead0c231f723379d1b99030b02dc10d90c6a4f00000080000000800200008000220203a9a4c37f5996d3aa25dbac6b570af0650394492942460b354753ed9eeca5877110d90c6a4f000000800000008004000080002202027f6399757d2eff55a136ad02c684b1838b6556e5f1b6b34282a94b6b5005109610d90c6a4f00000080000000800500008000

my parser spits out:

GLOBAL_UNSIGNED_TX (0)                                                                                 
	020000000258e87a21b56daf0c23be8e7070456c336f7cbaa5c8757924f545887bb2abdd750000000000ffffffff838d0427d0ec650a68aa46bb0b098aea4422c071b2ca78352a077959d07cea1d0100000000ffffffff0270aaf00800000000160014d85c2b71d0060b09c9886aeb815e50991dda124d00e1f5050000000016001400aea9a2e5f0f876a588df5546e8742d1d87008f00000000
IN_NON_WITNESS_UTXO (0)
	0200000001aad73931018bd25f84ae400b68848be09db706eac2ac18298babee71ab656f8b0000000048473044022058f6fc7c6a33e1b31548d481c826c015bd30135aad42cd67790dab66d2ad243b02204a1ced2604c6735b6393e5b41691dd78b00f0c5942fb9f751856faa938157dba01feffffff0280f0fa020000000017a9140fb9463421696b82c833af241c78c17ddbde493487d0f20a270100000017a91429ca74f8a08f81999428185c97b5d852e4063f618765000000
IN_REDEEM_SCRIPT (4)
	5221029583bf39ae0a609747ad199addd634fa6108559d6c5cd39b4c2183f1ab96e07f2102dab61ff49a14db6a7d02b0cd1fbb78fc4b18312b5b4e54dae4dba2fbfef536d752ae
IN_BIP32_DERIVATION (6)
	029583bf39ae0a609747ad199addd634fa6108559d6c5cd39b4c2183f1ab96e07f: d90c6a4f000000800000008000000080
IN_BIP32_DERIVATION (6)
	02dab61ff49a14db6a7d02b0cd1fbb78fc4b18312b5b4e54dae4dba2fbfef536d7: d90c6a4f000000800000008001000080
OUT_WITNESS_SCRIPT (1)
	00c2eb0b0000000017a914b7f5faf40e3d40a5a459b1db3535f2b72fa921e887
UNKNOWN_OUTPUT_TYPE (4)
	00208c2353173743b595dfb4a07b72ba8e42e3797da74e87fe7d9d7497e3b2028903
UNKNOWN_OUTPUT_TYPE (5)
	522103089dc10c7ac6db54f91329af617333db388cead0c231f723379d1b99030b02dc21023add904f3d6dcf59ddb906b0dee23529b7ffb9ed50e5e86151926860221f0e7352ae
UNKNOWN_OUTPUT_TYPE (6)
	023add904f3d6dcf59ddb906b0dee23529b7ffb9ed50e5e86151926860221f0e73: d90c6a4f000000800000008003000080
UNKNOWN_OUTPUT_TYPE (6)
	03089dc10c7ac6db54f91329af617333db388cead0c231f723379d1b99030b02dc: d90c6a4f000000800000008002000080

it looks like it see's a zero byte after

IN_BIP32_DERIVATION (6)
	02dab61ff49a14db6a7d02b0cd1fbb78fc4b18312b5b4e54dae4dba2fbfef536d7: d90c6a4f000000800000008001000080

at this point my parser assumes the input kv-map has ended, and starts interpreting the new kv-map as the output map. but perhaps this is wrong, since we could be in the second input kv-map in the input kv-map array. given this, as far as I can tell, there's no way to tell when we have switched to the output key value map array? am I missing something?

achow101 commented at 6:40 PM on July 5, 2018: member

@jb55 The number of inputs and outputs are specified by the unsigned tx itself. You will need to partially deserialize it to know how many inputs and outputs. @luke-jr I'm not sure. There's still discussion on the mailing list about this.

jb55 commented at 6:41 PM on July 5, 2018: none

@achow101 ah right. that makes sense, that was next on my todo. thanks!

Move global data to per-input and per-output data

Change from a global map with input data to a global k/v pair with input and output data.

Add new types for finalized scriptSigs and scriptWitnesses.

Redefined types to support new model

Updated the formatting of the listing

bfae1799d3

Clarify handling of duplicated keys 45df424759

Add new responsibilities, update responsibility, update extensibility descriptions

Add updater, transaction finalizer, and transaction extractor roles.

Update the description of other roles to clarify

Update extensibility section

f515da0552

Add sections for encoding, file extension, and mime-type 81c20a635e

Update tests for new serialization 4fcd8bf008

achow101 force-pushed on Jul 5, 2018

achow101 commented at 10:32 PM on July 5, 2018: member

I have added a few changes that clarify that signers must check input txids for non-witness utxos and that a map must be provided for every input and output. @luke-jr I think this is ready to be merged.

luke-jr merged this on Jul 6, 2018

luke-jr closed this on Jul 6, 2018

laanwj referenced this in commit b654723461 on Jul 18, 2018

HashUnlimited cross-referenced this on Sep 11, 2018 from issue BIP 174 PSCT Serializations and RPCs by HashUnlimited