This proposed BIP was posted to the mailing list just under two months ago. As there has been no serious objections, (unfortunately no mailing list feedback at all), I wish to formally ask for a BIP number to be assigned to our proposal.
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-May/014396.html
@jonasschnelli I think that you would like to subscribe to this pull request.
@jonasschnelli @ChristopherA @shannona @kimdhamilton I've updated this draft specification to include a Litecoin Display Format.
54 | +** This warning SHOULD state that in the case of a large reorganization, the TxRefs Displayed may point to a different transaction, or to no transaction at all. 55 | + 56 | +=== Human Display Formats === 57 | +This specification provides standardisation of encoding and namespacing for '''application display formats'''. 58 | + 59 | +If a display format doesn't comply with this standard, it MUST be called a '''non-standard application display format'''.
If it doesn't comply with the standard, the standard can't tell it what to do... :p
The 'specified' testnet display format is added for convenience only. It should not be considered part of the formal standard. This is why it was decided to put it in the appendix as a non-standard extension.
58 | + 59 | +If a display format doesn't comply with this standard, it MUST be called a '''non-standard application display format'''. 60 | + 61 | +==== Encoding ==== 62 | +* All display formats MUST be encoded with standard Bech32<ref>'''Why use Bech32 Encoding for Confirmed Transaction References?''' The error detection and correction properties of this encoding format make it very attractive. We expect that it will be reasonable for software to correct a maximum of two characters; however, we haven’t specified this yet.</ref> encoding as defined within the [https://github.com/bitcoin/bips/blob/master/bip-0173.mediawiki BIP 0173] specification. 63 | +* The Human Readable Part of the Bech32 encoding MUST be "tx"<ref>'''Why use the same Human Readable Part for different systems or blockchains?''' The ‘tx1’ prefix is to namespace the Bech32 encoding to this specification. We use Magic Values internally to namespace between projects.</ref>.
Seems to defeat the purpose of the human readable part.
Why? Isn't the human readable ideal for detecting (on the human level) if this is a testnet, mainnet or – lets say – lightcoin txref?
Exactly, it should be "tx" + "bc" in some order or another, not just "tx". :)
The human readable part is to inform users that the string should conform to this particular BIP. We have out own magic and versioning system so that we can upgrade and extend the format transparently for our users.
59 | +If a display format doesn't comply with this standard, it MUST be called a '''non-standard application display format'''. 60 | + 61 | +==== Encoding ==== 62 | +* All display formats MUST be encoded with standard Bech32<ref>'''Why use Bech32 Encoding for Confirmed Transaction References?''' The error detection and correction properties of this encoding format make it very attractive. We expect that it will be reasonable for software to correct a maximum of two characters; however, we haven’t specified this yet.</ref> encoding as defined within the [https://github.com/bitcoin/bips/blob/master/bip-0173.mediawiki BIP 0173] specification. 63 | +* The Human Readable Part of the Bech32 encoding MUST be "tx"<ref>'''Why use the same Human Readable Part for different systems or blockchains?''' The ‘tx1’ prefix is to namespace the Bech32 encoding to this specification. We use Magic Values internally to namespace between projects.</ref>. 64 | +* All Non Bech32 alphabet characters after the Bech32 code separator MUST be ignored when phasing, (except for terminating characters).<ref>'''Why Strip all Non-Bech32 Alphabet Characters?''' We do not wish to expect the users to keep their TxRef's in good unicode form. We expect them to copy, paste, write by-hand, write in a mix of character sets, etc. Phasers should automaticity correct for all of these sorts of errors. However, we only specify the removal of non-bech32-alphabet characters.
By specifying the removal, you actually prohibit correction...
I do not see how this is the case, can you please provide an example?
138 | +|'''Bitcoin''' 139 | +|'''0x3''' 140 | +|'''‘r’''' 141 | +|'''Bitcoin Main Chain''' 142 | +|- 143 | +|'''Litecoin'''
Litecoin stuff is inappropriate for BIPs.
As definition, yeah maybe. But we already have Litecoin references in BIP44 and BIP122.
As examples of altcoins, not as specification.
If including a Litecoin display format as an appendix is a strong objection I will be happy to remove it. It was included, by request, as Litecoin is used by Bitcoin Developers as a legitimate and functional testing ground.
108 | + 109 | +=== Allocated Magic Values === 110 | + 111 | +This appendix contains a table of the allocated or reserved magic codes. 112 | +* If you wish to claim a magic code, please open a pull request adding the code to this list. 113 | +* If this list becomes too popular, we will move it to another location.
Suggest just reusing SLIP 44 (or better yet 173), possibly with a bitshift or sub-namespace.
@luke-jr wrote: "Suggest just reusing SLIP 44 (or better yet 173), possibly with a bitshift or sub-namespace." & "Litecoin stuff is inappropriate for BIPs."
I agree that this may be more appropriate for SLIP (though I have no experience with their process) but unfortunately both SLIP 44 and 173 don't work well as bech32 prefixes because of the limitations of that encoding.
I personally would like to see a SLIP that includes a number of prefixes, not only for blockchain types, but also reserve placeholders for private key, public public key, xprv, xpub, ecdsa signature, schnorr signature.
I'm also hoping to see this BIP have tx1: rather than tx1- as that allows it to conform better with W3C URN/URL standards, as TX1: is a method, and the data after is the the encoding of that method. If someone really preferred to use base58 for transaction references, it would be TX58: and then its encodling, etc.
I agree that this may be more appropriate for SLIP (though I have no experience with their process) but unfortunately both SLIP 44 and 173 don't work well as bech32 prefixes because of the limitations of that encoding.
SLIP 173 is specifically designed for Bech32 prefixes...
I personally would like to see a SLIP that includes a number of prefixes, not only for blockchain types, but also reserve placeholders for private key, public public key, xprv, xpub, ecdsa signature, schnorr signature.
That's more of a BIP thing, and could be part of this if appropriate.
@ChristopherA Thank you for the feedback of using the colon instead of the dash for the separating character after the Bech32 separator. I am going to update this proposal adopting this change. @jonasschnelli any comments / objections?
@ChristopherA I have updated this bit to use the colon instead of the dash for the first visual breaker, as recommended. @jonasschnelli and @shannona please take note that your code will need to be updated.
@luke-jr I have moved the non-bitcoin related appendixes to https://github.com/satoshilabs/slips/pull/89
However I still haven't decided what to do with the magic values, In this case I don't think that https://github.com/satoshilabs/slips/blob/master/slip-0044.md suits our purposes; as there is a large priority having the codes very concise.
I think the codes should be human-readable more than concise, so SLIP 173 is a good match. But that's up to you.
@luke-jr I think that it is the job of the GUI to provide this information in a clear way to the user. Users should not be expected to understand the difference between x-coin and y-scam-coin. Implementers should take care to display this information in a clear way to the users.
However I think that the non-bitcoin related magic code listing should be moved to a SLIP. I will leave Litecoin as an exception; as I've granted it a first-class 5-bit code, (maybe this should be reviewed and a 2nd class 10-byte magic code should be used for Litecoin). @jonasschnelli what do you think?
59 | +If a display format doesn't comply with this standard, it MUST be called a '''non-standard application display format'''. 60 | + 61 | +==== Encoding ==== 62 | +* All display formats MUST be encoded with standard Bech32<ref>'''Why use Bech32 Encoding for Confirmed Transaction References?''' The error detection and correction properties of this encoding format make it very attractive. We expect that it will be reasonable for software to correct a maximum of two characters; however, we haven’t specified this yet.</ref> encoding as defined within the [https://github.com/bitcoin/bips/blob/master/bip-0173.mediawiki BIP 0173] specification. 63 | +* The Human Readable Part of the Bech32 encoding MUST be "tx"<ref>'''Why use the same Human Readable Part for different systems or blockchains?''' The ‘tx1’ prefix is to namespace the Bech32 encoding to this specification. We use Magic Values internally to namespace between projects.</ref>. 64 | +* All Non Bech32 alphabet characters after the Bech32 code separator MUST be ignored when phasing, (except for terminating characters).<ref>'''Why Strip all Non-Bech32 Alphabet Characters?''' We do not wish to expect the users to keep their TxRef's in good unicode form. We expect them to copy, paste, write by-hand, write in a mix of character sets, etc. Phasers should automaticity correct for all of these sorts of errors. However, we only specify the removal of non-bech32-alphabet characters.
Shouldn't "phasers/phasing" be "parsers/parsing" here and elsewhere?
I agree with @luke-jr that the human-readable part should inform the user that this is more than just "some transaction reference." For instance, as the user, I have to understand that tx1:r... is a Bitcoin transaction because of the r in the data part.
Since the human-readable part is specifically for humans, why not use bc-tx or similar string for the HRP? The spec could read
use the SLIP-0173 coin prefix, followed by `-tx` as the human-readable part
For instance, I would read bc-tx1:rqqq-qqqq-qmhu-qk as "bitcoin-transaction blah."
151 | +|Reserved 152 | +|- 153 | +|'''Testnet''' 154 | +|'''0x6''' 155 | +|'''‘x’''' 156 | +|'''Any Test Network'''
Why "any test network"? This sort of ambiguity means that multiple transactions have identical reference strings.
You could use the 10-bit codes for testnets (or stick with the SLIP-0173 prefixes in the human-readable part).
Different test-nets can use different magic values. The idea behind "Any Test Network" is probably to have a longer and clear visually different txref form for testnets (longer bech32 part, longer HRP).
I do not want to assign different Magic values to different test networks; Test Networks are for testing purposes only. Developers should be aware about what network they are testing. (without needing a separate testing magic value).
For example, should testnet v4 and (future) testnet v5 be assigned different testing magic values? Such ideas quickly get ridiculous.
Some wallet interacting with two testnet chains, say for two altcoins, should be enough motivation to assign testnets separate Magic values. If not, the wallet will need to carry around metadata with each transaction reference, since all the references will be for "Any Test Network." There would also be extra code to check if it's a testnet or mainnet and switch around handling based on that.
209 | +|- 210 | +|Tx Position 211 | +|27 – 39 212 | +|13 213 | +|8191 transactions. (A full Bitcoin Block without Hard Fork). 214 | +|}
I can understand the desire to keep all reference strings to the nice 14-character version by keeping the data payload to these 40 bits, but it seems to place artificial limitations on the format (year 2048 & 8191 transactions). I also understand that this might be addressed with Version 1 encoding. But current blocks are not that far from having 8191 transactions.
You could go with a variable-length encoding similar to Bitcoin's variable ints and gain the benefit of having a format that will work for very large blocks and the very far future.
Also, the Bech32 reference libraries allow encoding from byte arrays into the base-5 arrays native to Bech32. It seems like bit-packing to these 40 bits might be overkill. As an alternative you could have one bit-packed byte to start:
# First two bits are the protocol version, supporting values 0-3
V = ((protocol version) & 0x03) << 6
# Next two bits are magic for the blockchain
# 0x00 = Bitcoin
# 0x01 = Testnet3
# 0x02 = Byte1 is another coin's magic code (gives 256 options)
# 0x03 = Byte1-2 is treated as the coin magic code (gives 65280 more options)
M = (magic & 0x03) << 4
# Next two bits are the byte length of the block reference
B = ((byte length of block reference) & 0x03) << 2
# Final two bits are the byte length of the transaction index
T = ((byte length of transaction index) & 0x03)
# Assemble into the first byte
Byte0 = V | M | B | T
This gives you up to 3 bytes for each block and transaction reference, which is 16.7 M blocks, or year 2336, and 16.7 M transaction slots.
Data part: [Byte0][optional magic bytes 1-2][block reference bytes][tx reference bytes]
So the shortest data part would have 3 bytes in it, with the reference version 0 genesis coinbase transaction having data part 0x050000.
I know this is a departure from your vision, but it would be much more flexible for the long term (in my opinion).
We should move the technical discussion to the mailing list where it belongs: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-May/014396.html.
Posted to mailing list. My comments will have no context, since the original BIP content was not sent to the list, only a GitHub link.
@clarkmoody I have replied to your review comment on the mailing list: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-July/014800.html
I'm sorry about the formatting when viewing it online, it seems like protonmail has a incapability to send plain-text emails. 👎
On Specifying the The Network within the Human-Readable-Part: I particularly do not support this idea. The core reason is that I don't expect users to memorise and use many different Human Readable Parts in real-life. - I expect the Developers of the GUI to clearly display what network (or unknown network) the TxRef relates to.
Users should not be expected to understand this specification; hence, they will only see "tx1:" and then know they can copy-paste or click on it and the application will do everything for them. (the application should say: "hey this is a Bitcoin / Litecoin / Sidechain / Extension Block / etc Transaction Reference". Then direct the user to do an appropriate action.
In the future users will not install 10 different *coin wallets, they will instead install a block-explorer that support 10 different alt-coins and sidechains and all colours of different variants, and have that link to wallet apps they use.
Consensus systems work by most users running a full node. Full nodes are inherently single-blockchain. It may be possible in the future there might be some abstraction where the nodes are all invisibly managed in the background, but that's not very near term.
Handling tx1-bc:, tx1-tb:, etc is no more work than translating an arbitrary number to a name...
Besides, these aren't URIs as-is. If you want URIs, then they should use the URI format, not Bech32...
tx://bitcoin/<height>/<txnum> or something.
@luke-jr for example, I hope that the application that implements this format is something like a block-explorer type application that would make sense to support many different blockchains. Such block-explorers can then call the appropriate wallet if the user desires such an action. - The resource requirements for users to run their own Full-Node-Block-Explorers is already quite prohibitive. (I've updated my previous post to call the application a block-explorer instead of a wallet).
This proposal isn't a URI. This is a Transaction Reference; that can reference transactions both in the Bitcoin blockchains, and other supported blockchains.
Actually the #RebootingWebOfTrust community plans to use a bech32 transaction reference for part of a W3C standard for Decentralized Identifiers, which will be a a true URN and under the W3C international standards. In our case the urn will be did:method:transactionref. There are already ones being defined for ethereum and sovrin/hyperledger indy (a public but permissioned blockchain) and I want bitcoin to be included as well. As @veleslavs and @jonasschnelli are planning to use this for an identity enrollment proof system for satoshi-style blockchains, the in the W3C Credentials CG (of which I'm co-chair) are quite willing to help tx1 be an international standard properly registered URN as well. The only thing needing a change in their proposal is tx1- to tx1:
@luke-jr URI don't have to have :// — that is an artifact of web based URLs having an "authority", this this case the domain owner. See https://tools.ietf.org/html/rfc3986 The authority component is preceded by a double slash ("//")— in this case, as transaction references are decentralized and there is no authority, there is no ://
In our hackathon this week we already have one implementation in C that works with bitcoind to find the transaction data, the other one uses javascript with a bitcoin explorer API to get the transaction data, and a third (still in progress) uses python which can do both. So we are beginning to see implementation of this proposal now.
@ChristopherA suggested change from tx1- to tx1: has been adopted, please see the latest revision of this document.
59 | +If a display format doesn't comply with this standard, it MUST be called a '''non-standard application display format'''. 60 | + 61 | +==== Encoding ==== 62 | +* All display formats MUST be encoded with standard Bech32<ref>'''Why use Bech32 Encoding for Confirmed Transaction References?''' The error detection and correction properties of this encoding format make it very attractive. We expect that it will be reasonable for software to correct a maximum of two characters; however, we haven’t specified this yet.</ref> encoding as defined within the [https://github.com/bitcoin/bips/blob/master/bip-0173.mediawiki BIP 0173] specification. 63 | +* The Human Readable Part of the Bech32 encoding MUST be "tx"<ref>'''Why use the same Human Readable Part for different systems or blockchains?''' The ‘tx1’ prefix is to namespace the Bech32 encoding to this specification. We use Magic Values internally to namespace between projects.</ref>. 64 | +* All Non Bech32 alphabet characters after the Bech32 code separator MUST be ignored when phasing, (except for terminating characters).<ref>'''Why Strip all Non-Bech32 Alphabet Characters?''' We do not wish to expect the users to keep their TxRef's in good unicode form. We expect them to copy, paste, write by-hand, write in a mix of character sets, etc. Parsers should automaticity correct for all of these sorts of errors. However, we only specify the removal of non-bech32-alphabet characters.
"automaticity" => "automatically"
"phasing," => "parsing" (change word and remove comma)
23 | + 24 | +This BIP is licensed under the 2-clause BSD license. 25 | + 26 | +=== Motivation === 27 | + 28 | +Bitcoin uses regularly use TxID (Transaction Identifiers) for referencing individual transactions. TxID's are a core part of the consensus protocol and were naturally exported to the user-space since the first release of Bitcoin.
"Bitcoin uses" => "Bitcoin users"
72 | + 73 | +=== Magic Codes === 74 | +* Magic codes are either 5, 10, 15, or 20-bits long. 75 | + 76 | +* The length of the magic code is set by the first 5 bits: 77 | +** 0x0 to 0x7: 5-bit long. (All Reserved<ref>'''Why reserve all single character Magic Codes?''' The authors of this specification see many potential Bitcoin Related uses for additional magic codes, such as with Sidechains and other Special User Facing Projects.</ref> for Bitcoin-Related Use, we have the special exception for Litecoin, that is also assigned a 5-bit code.)
"Special User Facing Projects" => "special user facing projects". This whole document has strange capitalization like that above, was there some intention behind that?
171 | +For non-bitcoin related display formats please reference the external SLIP-XXXX "TxRef Non-Bitcoin Display Formats". 172 | + 173 | +=== Bitcoin Display Formats === 174 | +This appendix contains the specifications for Bitcoin Related Application Display Formats. 175 | +* If you wish to contribute a new display-format, please open a pull request adding to this appendix. 176 | +* We have considered various encoding schemes, and have found that variable-length schemes have a too-high overhead.
"have a too-high overhead" => "minimize overhead"
172 | + 173 | +=== Bitcoin Display Formats === 174 | +This appendix contains the specifications for Bitcoin Related Application Display Formats. 175 | +* If you wish to contribute a new display-format, please open a pull request adding to this appendix. 176 | +* We have considered various encoding schemes, and have found that variable-length schemes have a too-high overhead. 177 | +** In the case there is a Hard-Fork that extends Bitcoin's Block Transaction Capacity over the 8191 of the provided display format, we will simply define a `Bitcoin Display Format 1`. Considering the considerable disruption a Hard Fork causes; the creation of a secondary Display Format should not be of significant concern.
I suggest:
If there is a hard-fork which expands the number of transactions per block beyond the current 8191 limit, we will simply define a new
Bitcoin Display Format 1.
Considering the considerable
Would remove "considerable" as this is a bit awkward and unnecessary.
Some small typos and copy editing suggestions.
@jonathancross Thank you for the proof-reading. I have adopted all of your suggestions: many thanks.
138 | +|'''Bitcoin''' 139 | +|'''0x3''' 140 | +|'''‘r’''' 141 | +|'''Bitcoin Main Chain''' 142 | +|- 143 | +|'''Litecoin'''
So we need a reserved value for every altcoin and altchain?
I believe the plan is to move altcoin values to a SLIP, similar to SLIP-0044 and SLIP-0173.
Then the plan is to have centralized registration of a limited set of arbitrary numbers again to...coin name? a url? Nack that. We should use genesis block hash for identifying chains.
@jtimon each other chain will need to provide their own Display Format to describe their own particular unique properties, some may not even have the concept of a "genesis block hash", even then how many bits should we allocate for all these different chain "genesis block hashes"?
For any such standard, unless we allocate 128bit of space for the hashes, we must have some centralised lookup table that is DoS proof. (I can prettily quickly generate 64bit of ChainID's.).
Edit: This concern is only when the genesis block-hash is checked and a Network has deliberately chooses to 'mine' a ChainID the same as another network. In reflection real-life this is not a concern; I think. However it should-not be included as a salt.
I propose a non-standard display format for this purpose. This is for the 'catch-all' case where a Network Magic Code hasn't been specified. It uses 30bit for the ChainID that should be more-than-enough for any real-world use-case.
This is not what I'm proposing, please, don't call it chain ID if it's 30 bits. That's surely not a secure ID. There's no need for a "catch all" case if you identify them on the human readable part as luke suggests. But even if you don't, you could just try the checksum for the same txref with all the 256 bit chain ids you support to find out which one is the appropriate one.
FWIW, I think trying to identify coins by genesis block is misguided at best. There is no reason to assume altcoins each have a unique genesis block.
But the whole problem goes away by just using BIP173 identifiers in the prefix...
It is not about identifying a coin, but a chain: some chains could have several "coins". For altcoins that didn't create their own unique genesis block, a later block when it starts to diverge can be used. The point is to have less prune to collisions than a memorable string, which we would need to register somewhere. It is sane to assume that altchains will respect BIP173's "bc" and "bt" but not so much to assume they will know about each other or will wait for some centralized registry to assign them a memorable and "unique" string. Of course if someone wants to "collide" one can always still use bitcoin's genesis block for their altchain and "bc" to "identify it". My point is that altchains that don't want to collide with each other or with btc or any testnet still have it very easy if we rely only on the memorable string (unless we assume the centralized registry is actually maintained or we also use some chain id like the genesis block hash or the first unique block or whatever).
Litecoin doesn't need a special place in this BIP, nor does any other altcoin, testchain or altchain (but 0x4 fits more Freicoin than Litecoin IMO if we have to discuss that); nor in any "SLIP".
When you introduce a BIPXXX txid on a litecoin program that supports it the only concern is making sure the transaction isn't referring to another chain. But that program doesn't need to know about a centralized registry with all chains in the world: it only cares about the "TxRef" being created for litecoin's chain to not waste time with something that doesn't exist in litecoin's chain. The right place IMO us thus bech32's checksum.
On one side litecoin TxRefs can be created hashing the genesis block hash (aka "chain id") when calculating the checksum, without it having to be part of the rest of the bech32 string.
Since the litecoin program expects a litecoin TxRef, it can checksum without receiving the genesis block hash every time (which can have as a constant in the program instead). If we're talking about a program expecting TxRefs from various chains, we need more.
We could iterate for all supported chains and try if the txref is valid for all chains in an arbitrary order and pick the first valid one, like PaymentServer::ipcParseCommandLine does for addresses and registered chain names to support bip70 (which doesn't really scale to many supported chains in the same program).
Or perhaps the human readable part could be extended beyond "tx..." to "txltc..." or something.
I don't see how this BIP can be compatible with https://github.com/bitcoin/bitcoin/pull/8994 (not that it has to be) at this point.
@jtimon Thank you for your review of the magic encodings in our proposal.
I would like to attempt to address your concerns.
On Litecoin or any other, altcoin, testchain or altchain being described in this BIP:
If this is a genuine concern, I will move the non-bitcoin magic code registration to a SLIP or other place that it doesn't disturb this BIP process. However, I think that the religious exclusion of the acknowledgement that crypto-finance is broader than just Bitcoin hurts the BIP process overall. (Personally, I think that the focus of a BIP should be Bitcoin, however not to the strict exclusion of any detail that isn't related to Bitcoin.)
I also seek clarification that you suggest that we should also remove all references to the test-network, this would very likely be used by the Bitcoin Testnet.
On using as salt for the Bech32 encoding: I recommend that this should not be done for two reasons.
On using the human-readable-part for designating the network: My most recent reasoning why I recommend against this approach is in my comment above: #555 (comment)
I will take the time to revise this BIP so that this comment can be included in the rationale.
On your testchains proposal: I do not understand how this proposal is conceptually incompatible with the testchains proposal. Can you please provide more details?
Thank you again, Your full careful review of this proposal would be most appreciated.
@veleslavs I think you are completely missing my point: I want this to more easily support more chains, not less. A centralized registry is a barrier to that, whether it is here or in a SLIP. Moving it to the SLIP doesn't help in any way. I'm not suggesting this BIP doesn't support testnet3 or any other testchain. Pretty much the contrary: I want support of them without having to list them all one by one here (or in a SLIP). But I agree with @clarkmoody that testnets shouldn't be grouped together.
This is a security risk, as this opens up the possibility that a single TxRef may reference more than one transaction. (A cryptographic hash of 160bit is sufficient for such salting. However Bech32 is a non-cryptographic 30bit checksum).
Unless I'm missing something, identifying litecoin with the sha256d hash of its genesis block (256 bits) instead of with "0x4" doesn't make the possibility that a single txref references more than one tx more likely.
This will break the Bech32 standard and cannot be considered Bech32 encoding.
That's strange, because it is my understanding that using the chain id (genesis block hash) for the checksums was considered for bech32 addresses too. In fact, I don't remember why it was decided against it. Perhaps @sipa can help adding some clarity here?
On using the human-readable-part for designating the network: My most recent reasoning why I recommend against this approach is in my comment above: #555 (comment)
I think @luke-jr 's response is very good: #555 (comment)
I do not understand how this proposal is conceptually incompatible with the testchains proposal. Can you please provide more details?
My proposal for testchains easily allows you to create "infinite" independent testchains. The existence of the centralized registry of "known altchains" limits the number of chains this BIP can support to 2^5, 2^10, 2^15, or 2^20 as the maximum number of chains that can be supported.
To reiterate: I'm not against supporting multiple chains but against the way it is done here via a centralized registry enumerating known chains.
@jtimon One of the key design goals of the TxRef was to keep it very short. If we include 256bit checksum then, of course that could support infinite chain-types.
Great effort has been taken to keep the number of bits to encode low (in the case for Bitcoin, 40bits are needed), while still extendable. (for a total 70bits with checksum).
I am unaware of any way to have the cake and eat it also: Keep the number of bits low, and not have a central place to lookup Networks Magic Values.
I'm not against making a additional by-hash-of-genisis display-format for the catch-all case. However this would not be the case of a short TxRef anymore, It would be about 250bit long in total. (salted_hash(160) + block height (30) + tx_position (30) + checksum (30)).
I'm also unconvinced that Test Network TxRef need to be globally unique. For test-networks you can just prefix the full 256bit genesis hash to the HRP, however, such TxRef's shouldn't be use outside of strict testing purposes. As Test Networks Should only be used within their Testing Scope.
One of the key design goals of the TxRef was to keep it very short. If we include 256bit checksum then, of course that could support infinite chain-types.
I don't mean including the genesis block in either the txref nor the checksum, only taking it into account when calculating the checksum. The chain id must be known to calculate the checksum. My understanding is that this is possible, but perhaps @sipa or someone else can correct me.
However this would not be the case of a short TxRef anymore, It would be about 250bit long in total. (salted_hash(160) + block height (30) + tx_position (30) + checksum (30)).
I'm not sure where this numbers come from, but perhaps the specification could contain something like this? Are these the dimensions you have in mind without my suggestion?
magic numbers(5) + block height (30) + tx_position (30) + checksum (30) = 95 bit ? If so, with my suggestion the data part would not become bigger but smaller, since the magic numbers are removed and everything else remains the same.
@jtimon The checksum should not contain any salted data. Otherwise with 30bits of non-cryptographic checksum one can easily calculate a TxID that is valid for more than one Network.
The network data must be included in the data-part (or human readable) for security. Salting a 30bit checksum with a ChainID is not secure.
This is why I proposed a 160bit crypto-salted-checksum, because that would have the security properties where it would be infeasible to find a intersection. Maybe 256bit would be needed in reality to protect against birthday attacks.
@jtimon I propose the following non-standard display format:
HRP chaintx1:
Data | Length | Value | Comment
------------ | ------|-----|--
Magic | 5bit | 0x0 | Letter: q
Version | 1bit | 0x0 |
Block Height | 32bit
Transaction Position | 32bit
Short-ChainID | 30bit | | There will be lots of intersections, however this isn't a concern for real-life use-cases. Use the first 30bits of the full 256bit hash.
Checksum | 30bit
Total Bech32 Length: 130bit or 26 characters.
Bitcoin can use it's own special optimised format, Other Networks that want an optimised format can use the SLIP process. However this can be the default for Non-Bitcoin alt-chains.
The checksum should not contain any salted data.
I'm not proposing that either. I feel like you are trying to "fix my proposal on the fly" without actually understanding it.
The network data must be included in the data-part (or human readable) for security
I propose on the human readable part or nowhere.
Salting a 30bit checksum with a ChainID is not secure.
Then don't do that, I'm not proposing that. Please, please don't call anything that isn't a 256 bit hash of the genesis block a chainID or we will get more confused. If it's not 256 bit long, it is not a chain ID by my definition. And you had no notion of chainID before. This is something new, please find another name to avoid confussion.
This is why I proposed a 160bit crypto-salted-checksum, because that would have the security properties where it would be infeasible to find a intersection. Maybe 256bit would be needed in reality to protect against birthday attacks.
Please stop trying to "fix" my suggestion and focus on understanding it first. Yes, I'm proposing to use 256 bits for the chain ID, just not include it as part of the txref.
@jtimon I propose the following non-standard display format:
My proposal is to remove the 5 bit magic part and the newly introduced 30 bit "something that is not a chain ID". Thus with my proposal the txref is smaller, not bigger.
You may ask, if it's not inthe txref, where is the 256 bit chain ID?
Nowhere. Either the programs assume it, or they know which one to use (exclusively for the checksum) from the human readable part.
How is it involved in the checksum if it's not part of the data part of txref ?
Before calculating the checksum, you append the extra 256 bits.
Forget for a moment it is a chain ID. We could append the same extra constant 256 bits before calculating the checksum, right?
Bitcoin can use it's own special optimised format, Other Networks that want an optimised format can use the SLIP process
To reiterate: my point is that no special case is needed for any chain, not even the main chain. And the SLIP process is not needed here at all.
This sounds like a bad idea-- unless I'm misunderstanding something: This proposal is not reorg safe or chain split safe. It claims that txids don't have a checksum but txid's are a checksum.
The document is thin on supporting motivations, but it seems to me that using a block hash and index (plus a check value) or a block hash and transaction hash would be strictly superior; and avoid those problems.
Though even with identifying the block it is likely an error to expect people to have them in the future. Bitcoin has been pretty carefully designed to require no random access to the blockchain and it is my expectation that in the future almost no nodes will. At that point anything built using this will be pushed onto centralized services, which is against the stated purpose of this document-- it would actually be creating the centralization it hopes to avoid.
There is already a facility built into Bitcoin software to identify transactions in blocks without requiring the verifier actually have the blocks, see gettxoutproof and verifytxoutproof in Bitcoin Core. I believe these achieve what you're looking to accomplish in a much better manner; though it isn't "short" because it's not depending on the network to store and retrieve things on your behalf.
The checksum should not contain any salted data. Otherwise with 30bits of non-cryptographic checksum one can easily calculate a TxID that is valid for more than one Network
Sorry, it seems I misunderstood this comment. Since it's only 30 bits of checksum, with 2^15 tries you can create a collision that is valid for more than 1 network. Thus you would still need something on the human readable part (or the 'magic' bits I strongly dislike) to discriminate chain even if you follow my suggestion of using the 256 bits of the chain when calculating the checksum.
Anyway, it seems there's more important concerns with the proposal than mine.
@gmaxwell Thank you for your time taken looking-over this document.
Firstly, thank you for the comment on the motivations section of this proposal; I will expand on them here and then update the document for future reviews.
I would now like to attempt to address some of your concerns: Please note that this proposal makes no attempt to point to an actual transaction, this proposal is to reference transaction positions within a blockchain, as clearly stated in the abstract, with emphasis:
Please note: that a TxRef does not reference an actual transaction itself, rather it references a possible location within a blockchain for a transaction, that may or may not, point to an actual transaction and may change with reorganisations.''
You are quite right that this proposal is not reorg safe for referencing transactions. However, this proposal doesn't reference transactions, rather transaction positions that are completely re-org safe.
The use of transaction positions to reference transactions should only be done in special situations: in particular when a transaction has been buried for a significant amount of time.
Therefore, implementers must be careful not to display TxRef's to users prematurely:
- Applications MUST NOT display TxRef's for transactions with less than 6 confirmations.
- Application MUST show a warning for TxRef's for transactions with less than 100 confirmations.
- This warning SHOULD state that in the case of a large reorganization, the TxRefs Displayed may point to a different transaction, or to no transaction at all.
Your comment about new the transaction positions becoming inconsistent in the case of a persistent chain-split is apt. Transaction position references may represent more than one chain in the case of schism fork. (until the ChainID or Network Magic values have been updated to compensate for this fact). If such a schism fork is planned for a future time, we would recommend the users of the affected chains to negotiate new magic numbers, (or use new ChainID’s).
I recommend that a new BIP, or an extension to this proposal that uses the full-block-hash + transaction position within the block, as you suggested, to be considered.
Again, thank you for your time. Your careful review of our entire proposal would be most welcome.
On centralising effect of depending upon blockchain lookup: Unless the block size is quickly increased, I expect it to be always quite reasonable to run archival nodes for the network. Disk space is the network resource of leas-concern. It is plausible that many will run Archival Nodes to serve this very sort of data.
If you are suggesting that archival nodes become so few that transaction lookup becomes unfeasible, then it would indeed be of significant concern.
My personal view is that I hope, and expect, that a Bitcoin Archival node will always reasonably be able to be run on common hardware and shouldn't become a significantly centralised service.
Hence, I find your concerns on random-access to the Bitcoin Network more of an academic concern than a concern that is representative of the real-life reality.
The obvious exception is huge-blocks, and that leads to a more fundamental centralising that concerns more than just this proposed BIP.
I think it is very likely that far historic data will not be generally queryable once nodes no longer need it for initialization; we can see that e.g. in the case of ripple they've even lost the historic data, and in the case of ethereum it appears that nodes which attempt to do a full sync will never finish now (though at the moment the data is still available). I'm sure that the data will exist as an academic exercise somewhere; but not in a way it is today.
Even at today's rate of blockchain growth it has pretty much become infeasible to store the whole blockchain on the most common SSD size... and that rate of growth is about to double. But even if people can store it multiple days of initialization aren't acceptable, and so I believe nodes will need to transition to a different approach, and once they have there will be little reason for anyone to store it.. except for supporting things like this proposal.
In particular I don't want to see node authors being nagged to take on the technical debt of continuing to support carrying around data which isn't actually used by the network on account of an effort to act as a bit of data storage for this proposal.
It is also not clear to me that the BIP/SLIP altcoin lists respect the bech32 character restrictions. It is important to me that the chain in some form is encoded into the bech32 (I totally ignore the tx1 part for my use), and that the bech32 be as short as reasonably possible. The current 1 char Magic codes or switch to two letter is fine.
@veleslavs @jonasschnelli If you wish to go forward with this, please use BIP number 136. To be merged, the following changes are needed:
That being said, I would suggest you consider the comments here as well.
If for any reason you have decided to abandon this BIP, please let me know ASAP.
@gmaxwell Sorry for the late response, I have been somewhat preoccupied for longer than I have expected.
On the archival storage of the blockchain. For our purposes I cannot see any reason why shading isn't a reasonable option for our purposes, providing there is a sufficient overlap between the shards, there shouldn't be any reason for a node in particular to have a completely copy of the entire historical blockchain.
If TxRef's become popular, then this use case alone will suggest that users have a desire to look-up historical transaction within the blockchain, and will either be supported by completely centralised services such as block-explorers, or otherwise.
0 | @@ -0,0 +1,278 @@ 1 | +<pre> 2 | + BIP: 0136
Drop the leading zero here.
0 | @@ -0,0 +1,278 @@ 1 | +<pre> 2 | + BIP: 0136 3 | + Layer: Applications 4 | + Title: Bech32 Encoded Transaction Position References 5 | + Author: Велеслав <veleslav.bips@protonmail.com>, Jonas Schnelli <dev@jonasschnelli.ch>
One author per line.
0 | @@ -0,0 +1,278 @@ 1 | +<pre> 2 | + BIP: 0136 3 | + Layer: Applications 4 | + Title: Bech32 Encoded Transaction Position References 5 | + Author: Велеслав <veleslav.bips@protonmail.com>, Jonas Schnelli <dev@jonasschnelli.ch> 6 | + Comments-Summary: No comments yet. 7 | + Comments-URI: https://github.com/bitcoin/bips/wiki/Comments:BIP-XXXX
Fix the BIP number here.
5 | + Author: Велеслав <veleslav.bips@protonmail.com>, Jonas Schnelli <dev@jonasschnelli.ch> 6 | + Comments-Summary: No comments yet. 7 | + Comments-URI: https://github.com/bitcoin/bips/wiki/Comments:BIP-XXXX 8 | + Status: Draft 9 | + Type: Informational 10 | + Created:
Needs a date.
@veleslavs Is this still active?
I note that the Lightning Network uses a similar chain/block/transactionindex methodology for labeling persistent nodes. However, in their case they have different ways of representing which chain, and they also have an index to specific output, i.e. chain/block/transactionindex/outputindex
I would like to see that a common bech32 value that can be used for both of these purposes as well as for other purposes (in my case, pointing to specific transaction output).
cc: @rustyrussell
Hello @jtimon , @ChristopherA , @clarkmoody I have significantly rewrote this BIP. Hopefully it is much more clear and simple to understand. Your review will be much appreciated. I'm sorry for my long-stretch of absence. @jonasschnelli and @kulpreet , In particular I hope to have your careful review. Please also update your reference implementations, and please send me an updated set of test-vectors that I'll Include in the BIP.
Overall, the technical changes to this BIP has been the modification from a 70-bit Data Payload to a 75-bit Payload (for Bitcoin). Additionally, now MainNet and TestNet use an almost identical format.
This comes at the cost of one-extra character.
I remain unconvinced that this is useful as written. @gmaxwell 's comments about archival data notwithstanding, it seems overly complex for its purpose.
Let's consider two cases:
Machine-to-machine
If two machines want to communicate the position of a transaction, they would simply use the block height and txid. There is no real need for brevity or additional checksums between machines, and a txid lookup within a block is a small linear search. Checksum information would be present in the machine connection itself, so it is not needed in the message format.
Person-to-person
The spec says As TxRef's are short, we expect that they will be quoted via voice or written by hand. It may take a shorter amount of time to communicate two numbers for block height and tx position than the fewer-but-random letters. So @luke-jr 's proposal above of something like tx:<chain>/<block>/<position> would yield tx:bc/0/0 for the genesis transaction. It remains human-readable but also becomes human-understandable. All coinbase transactions would have /0, making them recognizable.
You could add a variable number of bytes of the txid as a "checksum" of sorts, similar to Git using 4 bytes of the commit hash (tx:bc/541091/4/a51f appends two bytes). You could also change the / to be something else (-,:,_).
This alternate proposal suffers from no limitations on ultimate block height or transaction count.
We have been using a slight variant of txref in the #RebootingWebOfTrust for the BTC Decentralized Identifier method called BTCR (https://github.com/WebOfTrustInfo/rebooting-the-web-of-trust-spring2018/blob/master/draft-documents/btcr_resolver.md implementations at https://github.com/dcdpr/btcr-DID-method and https://github.com/WebOfTrustInfo/txref-conversion-java). The biggest difference is that we added an optional output index. If the output index is absent, it is considered 0 (and thus matches the older TXREF exactly), and if it is encoded it is the index.
An addition to our use, a particular advantage of the encoding with the output index is that this is basically what the Lightning Network uses for nodeids. In we add this to TXREF means that a bech32 txref can also be used as a shorter more readable alternative to the current hex notation for Lightning Nodes. (cc @rustyrussell)
@veleslavs The changes @ChristopherA is referring to are detailed in a branch I made from an old version of your BIP. We proposed adding extra bits to encode the output index in the txref. I modified your old BIP with two new sections for display formats "0-ext" and "A-ext" You can see those additions here: https://github.com/danpape/bips/commit/c8ab8c9aa0de1667045d030208143930d0d8c85a I'm sorry to have dragged my feet asking to you review them--now that you have made significant changes to your BIP I can revise my changes and get them to you in a few days if you are interested in incorporating them into this BIP.
In addition to the C and Go implementations you note in your new BIP, @kimdhamilton has one for javascript: https://github.com/WebOfTrustInfo/txref-conversion-java , @rxgrant and I have one for C++ within https://github.com/dcdpr/btcr-DID-method. Both implementations support the "txref-ext" proposal. Finally, I have changes I can submit to jonasschnelli for adding "txref-ext" support in the original C code: https://github.com/danpape/bitcoin_txref_code/commit/1d04b9db9d6b006fdffd90cd7e60b1750a27a6d9
@clarkmoody By not using Bech32 there is only guess-and-check to repair transmission error. @gmaxwell This BIP is not an implicit requirement that bitcoind offer generally queryable access to any txrefs.
@rxgrant I'm not sure you read my critique.
@clarkmoody , Thank-you for your response. - I quite agree, some sort of human understandable transaction URI format would be very useful. However that is out-of-scope of this design.
Such a format could be naturally extended to something like:
(for Bitcoin):
block: block:// bc/<block-height>
transaction: tx::// bc/<block-height>/<transaction-index>
input: input::// bc/<block-height>/<transaction-index>/<input-index>
output: output:://bc/<block-height>/<transaction-index>/<output-index>
I think that such a standard format would be very useful for URL's and other web-applications. However the format of this particular BIP is more focused, and has different use case in mind. I would suggest such a URI design should be made with a new BIP.
@ChristopherA and @danpape I think that this proposal could easily be extended to support referencing either inputs or outputs.
We have a single version bit that is currently unspecified. By setting this bit we can easily support an extra field.
About the other display formats, I'm sorry that I have removed the concept from the latest edition of this BIP. Of course there is nothing stopping you from defining your own; just pease take a new magic code.
@clarkmoody When the checksum does not match Bech32 offers a better user experience than guess-and-check repairs. This is what I mean by "repair transmission error".
@veleslavs Since BTCR references transactions on the Bitcoin blockchain, it does not look like a new magic code is useful.
The beginning of our URLs, which look like did:btcr:<bech32-data> offer the same contextual cues as the HRP, so we do not waste characters on it. However, we do keep the txref magic code in the Bech32 data part, to identify testnet references. How do you feel about loosening the encoding section to not require the HRP, in circumstances where it is obvious from context?
@veleslavs A question regarding the version bit--perhaps I don't understand, but why just one bit? Wouldn't it be better to have a few more bits to support more than two versions in the future?
@danpape Version 1 would specify additional space for further versions.
71 | +To increase portability and readability additional separators SHOULD be added: 72 | + 73 | +* A Colon<ref>'''Why add a colon here?''' This allows it to conform better with W3C URN/URL standards.</ref> '''":"''' added after '1'. 74 | +* Hyphens<ref>'''Why hyphens to the TxRef?''' As TxRef's are short, we expect that they will be quoted via voice or written by hand. The inclusion of hyphens every 4 characters breaks the string and means people don't loose their place so easily.</ref> '''"-"''' added after every 4 characters beyond the colon. 75 | + 76 | +All non-bech32-alphabet characters after the bech32 code separator MUST be ignored/removed when parsing (except for terminating characters).<ref>'''Why strip all non-bech32-alphabet characters?''' We do not wish to expect the users to keep their TxRef's in good unicode form (hyphens, colons, invisible spaces, random unicode characters, etc). We expect them to copy, paste, write by-hand, write in a mix of character sets, etc. Parsers should automatically correct for all sorts these sorts of these errors.
for all sorts these sorts
typo
@DanielWeigl thank you, corrected.
@veleslavs Let me know when this is ready to merge
@danpape @clarkmoody @rxgrant @ChristopherA @jonasschnelli I think that we are ready for a quick final review (particularly focusing on the test vectors). Then we can get this document merged.
@veleslavs FYI you don't need to wait for reviews if you want me to just merge this...
I made the last edits and had double checked all the test vectors at that time. Perhaps someone else should review as well, but I'm happy with it. Thanks.
230 | + 231 | +=== Test Vectors === 232 | +There are two sets of Test Vectors included here: 233 | + 234 | +* Bech32 Encoding Test Vectors. These are to test if a implementation accepts the encoding, with the correct human readable part, and separator. 235 | +* Bitcoin TxRef Test Vectors. These test the full specification, in particular, correct values for block hight and the transaction index.
Typo: "hight"
260 | +* <tt>txtest1:xqqq-qqqq-qkla-64l</tt>: <tt>(0x0, 0x0)</tt> 261 | +* <tt>txtest1:xqqq-qqll-l2wk-g5k</tt>: <tt>(0x0, 0x7FFF)</tt> 262 | +* <tt>txtest1:x7ll-llqq-q9lp-6pe</tt>: <tt>(0xFFFFFF, 0x0)</tt> 263 | +* <tt>txtest1:x7ll-llll-lew2-gqs</tt>: <tt>(0xFFFFFF, 0x7FFF)</tt> 264 | + 265 | +The following list gives valid (though strangley formatted) Bitcoin TxRef's and the values in hex. (block height, transaction index)
Typo: "strangley"
With thanks to Daniel Pape for the work behind this idea.
Please not that the test-vectors still need to be updated (again).