I thought it might be valuable to keep track of actual semantic changes being made since the initial out-of-draft version.
68+ 2. locktime
69+ 3. current input index
70+ 4. current input control block (only in case of tapscript spend)
71+ 5. current script last OP_CODESEPARATOR position
72+ 6. inputs
73+ 7. outputs
125+above rules:
126+
127+* If there are 0 bytes, it becomes the default of 0xff|0xf6|0xff|0xff; succeed.
128+* If the first byte is exactly 0x00, the Script execution succeeds immediately.
129+ //TODO(stevenroose) is this valuable? it's the only "exception case" that
130+ could potentially be hooked for some future upgrade
@roconnor-blockstream has previously warned about non-trivial OP_SUCCESS semantics. Though the current SUCCESS semantics are “any OP_SUCCESS opcode occurring in the script means SUCCESS”, but we could have different semantics that allow any opcode internally to trigger “instant success”, but (1) that are very different semantics and will require entirely different code and (2) it becomes way harder to reason about.
IIRC, @sanket1729 also noted that such SUCCESS semantics make reasoning about scripts for things like miniscript way harder.
Actually this BIP seems outdated, I have to push a small update. I decided to propose to make the 0x00 special case mean “ALL” to make this more ergonomic to use as a sighash together with CSFS. (“ALL” isn’t valuable as a template check because it contains the prevout scriptPubkey which should contain the hash) Other suggestions welcome.
OP_CLTV), we could introduce four special-cased bytes to mimick other popular SIGHASH modes: 0x00, 0x01, 0x02 and 0x03. Though locktime might be useful with OP_TX at some point. So I would argue against that. Mimicking “regular” sighashes isn’t super useful in the first place because any system that expects to use regular sighashes can use current regular schnorr keys.
I just pushed an updated version of this BIP. It has a reference implementation that produces test vectors that are tested against an implementation for Bitcoin Core and for rust-bitcoin.
I think it should be ready for review. I have one small last TODO in the specification related to txfs malleability.
26+
27+
28+# Summary
29+
30+## OP_CHECKTXHASHVERIFY
31+
66+ * 1. `TXFS_ALL`
67+ * 2. `TXFS_INPUTS_TEMPLATE | TXFS_OUTPUTS_ALL`
68+ * 3. `TXFS_INOUT_NUMBER | TXFS_INOUT_SELECTION_ALL`
69+ * 4. `TXFS_INOUT_NUMBER | TXFS_INOUT_SELECTION_ALL`
70+ * the `0x00` byte: it is set equal to `TXFS_SPECIAL_ALL`, which means "ALL" and is primarily
71+ useful to emulate `SIGHASH_ALL` when `OP_TXHASH` is used in combination
Hmm, “would be useful if that were proposed which it isn’t”. I am skeptical of this magic value.
While I understand Russell O’Connor’s dislike of runtime OP_SUCCESS, it is a lesser evil here than this kind of guessing of future utility which will no doubt prove suboptimal when we get there.
And for miniscript: sure, it will only generate and decode a push followed by TXHASH. But there are other things it can’t decode too, and that’s OK.
58+
59+The TxFieldSelector has the following semantics. We will give a brief conceptual
60+summary, followed by a reference implementation of the CalculateTxHash function.
61+
62+* There are two special cases for the TxFieldSelector:
63+ * the empty value, zero bytes long: it is set equal to `TXFS_SPECIAL_TEMPLATE`,
TXFS_SPECIAL_ALL. Fixing.
TXFS_SPECIAL_TEMPLATE? Maybe define what the bytes of the field selector means before the special cases.
82+ * 3. current input index (`TXFS_CURRENT_INPUT_IDX`)
83+ * 4. current input control block (or empty) (`TXFS_CURRENT_INPUT_CONTROL_BLOCK`)
84+ * 5. current script last `OP_CODESEPARATOR` position (or 0xffffffff)
85+ (`TXFS_CURRENT_INPUT_LAST_CODESEPARATOR_POS`)
86+ * 6. inputs (`TXFS_INPUTS`)
87+ * 7. outputs (`TXFS_OUTPUTS`)
HTML makes reading the markdown awkward, BTW.
89+* The last (highest) bit of the first byte (`TXFS_CONTROL`), we will call the
90+ "control bit", and it can be used to control the behavior of the opcode. For
91+ `OP_TXHASH` and `OP_CHECKTXHASHVERIFY`, the control bit is used to determine
92+ whether the TxFieldSelector itself has to be included in the resulting hash.
93+ (For potential other uses of the TxFieldSelector (like a hypothetical
94+ `OP_TX`), this bit can be repurposed.)
That’s a footnote, at best, mentioning how this could be expanded for a new OP_TX. But there’s no reason to design for it now that I can see, except to leave a clear carve-out for future expansion.
So TXFS_CONTROL is a terrible name. TXFS_FIELD_SELECTOR perhaps?
113+ * `TXFS_INPUTS_TEMPLATE = TXFS_INPUTS_SEQUENCES | TXFS_INPUTS_SCRIPTSIGS | TXFS_INPUTS_PREV_VALUES | TXFS_INPUTS_TAPROOT_ANNEXES`
114+ * `TXFS_OUTPUTS_ALL = TXFS_OUTPUTS_SCRIPTPUBKEYS | TXFS_OUTPUTS_VALUES`
115+
116+For both inputs and then outputs, do the following:
117+
118+* If the "in/outputs" field is set to 1, another additional byte is expected:
If I’m following this correctly, the (non-special) TxFieldSelector format is, in bytes:
CORE_SELECTOR [INOUT_SELECTOR] [IN_SELECTOR] [OUT_SELECTOR]
If TXFS_INPUTS is set in the CORE_SELECTOR, then INOUT_SELECTOR and IN_SELECTOR are present. If TXFS_OUTPUTS is set in CORE_SELECTOR, then INOUT_SELECTOR and OUT_SELECTOR are present?
Exactly. I’m thinking of changing this as follows:
TXFS_INPUTS and TXFS_OUTPUTS bitsINOUT_SELECTOR byte to know whether to expect IN_SELECTOR and/or OUT_SELECTOR.CORE_SELECTOR, one of which I’m thinking to repurpose for SPEND_SCRIPT (i.e. scriptCode for segwit v0 inputs and tapscript for v1 inputs, scriptPubkey for non-segwit)116+For both inputs and then outputs, do the following:
117+
118+* If the "in/outputs" field is set to 1, another additional byte is expected:
119+ * The highest bit (`TXFS_INOUT_NUMBER`) indicates whether the "number of in-/outputs"
120+ should be committed to.
121+ * For the remaining bits, there are three exceptional values:
TXFS_INOUT_NUMBER (might rename COUNT) bit. 0x40 is is 0b01000000.
156+* all in/outputs
157+* the current input index
158+* the leading in/outputs up to 8192
159+* up to 64 individually selected in/outputs
160+** using absolute indices up to 16384
161+** using indices relative to the current input index from -64 to +64.
This is incredibly complex, and seems to mismatch what I can see covenants being used for in practice. I anticipate fees being high in future, such that people will do a reasonable amount of engineering to minimize their total footprint. In particular, they will want to add fees after commitment, and want to batch transactions using stacking.
The first case implies you want to exclude a specific input and output, to allow for fees, or at least allow binding not to cover the final input/output. The second case implies you want to mul/divide an input number to get the corresponding range of outputs.
The simplest case is a single input and output pair: a-la SIGHASH_SINGLE. This both allows almost arbitrary fee inputs/outputs, and stacking.
But what if you want to bind a pair of inputs to one output? Or a pair of outputs to one input? Both seem reasonably common things to want to do (e.g. opening a dual-funded lightning channel, and closing a channel).
That means you need to be able to select outputs as “current input index / 2” or “current input index * 2 and current input index * 2 + 1”. Numbers other than 2 are possible but this is the most likely case (since, in order to stack, all txs must be of same input-output number form, and I consider 1 and 2 by far the most likely numbers here).
Yeah this is true. Initially I didn’t have relative indices. I’m still not entirely convinced they are useful. Precisely for the 1-in 2-out case which seems super common to me. I heard “you’d be surprised how easy it is to add an extra input”.
My initial thought was that private aggregation (i.e. not through public broadcast media like mempools) would be easily possible as a user can just create/sign a thousand variants of their txs, for each possible input index. This works with absolute indices and doesn’t need relative indices. It might even work with public broadcast.
The problem is that doing this with absolute indices only works if everyone in the protocol has the same in-out ratio. (Everyone needs 1-to-2 so you can sign 1in1,2out, 2in3,4out, 3in5,6out, etc). Otherwise you get a quadratic amount of data. With relative indices, you can sign XinX,1out, XinX,2out, XinX,3out,.. and this way the coordinator can put you in any place and put your second output in some arbitraty place and pick your signatures based on where your second output is placed.
Ok this doesn’t really require relative indices, but it requires the ability to mix “current” with absolute.
This leaves some protocols vulnerable to the partial signature attacks. Say the covenant requires your outputs go half to pubkeyA and half to pubkeyB. Now I have two identical 1BTC covenant UTXO inputs, but re-use the same outputs to satisfy both, and steal the other 1BTC.
The same problem applies to “tell me the outputs in the witness data”.
189+* For tapscript, primarily motivated by the cheaper opcode `OP_TXHASH` (it
190+ doesn't require an additional 32 witness bytes be provided) and the potential
191+ future addition of byte manipulation opcodes like `OP_CAT`, an additional
192+ cost is specified per TransactionHash execution. Using the same validation
193+ budget ("sigops budget") introduced in BIP-0342, each TransactionHash
194+ decreases the validation budget by 10. If this brings the budget below zero,
Yeah it’s tricky. In practice, it has a similar amortized per-tx hash cost that sighashes have. It’s hard to count those to the budget because they are amortized, it’s basically hashing all the large tx fields once so that if they are repeatedly requested their hash can be used.
After the amortized hash cost, it’s just a finite series of ~32-byte chunks with maximally 64 in/out which in total can have 8 fields that are each ~32 bytes. This is ~16,384 bytes max.
Then, another consideration is that it would be nice and reasonable if TXHASH+CSFS would not have a higher cost than what naturally would be placed in the witness, the 64-byte signature.
I see it like this: we have a 64-byte budget to divide over TXHASH+CSFS as I think it’s reasonable that this combination doesn’t cost more than 28% more than a CHECKSIG (which is 50).
So maybe it’s right that TXHASH can actually cost more, something like 25 if CSFS would be priced at 35 or 40.
This is not a signature budget, it’s a hashing budget. Perhaps we should make this a first-class citizen then?
See https://rusty.ozlabs.org/2023/12/22/script-limits-opcat.html#my-proposal-a-dynamic-limit-for-hashing
240+ `<txfs> OP_TXHASH <pubkey> OP_CHECKSIGFROMSTACK` effectively emulates `SIGHASH_ANYPREVOUT`.
241+
242+
243+# Detailed Specification
244+
245+A reference implementation in Rust is provided attached as part of this BIP
OK, I would really appreciate a table of all the bits and exactly what and how they encode them. It’s particularly nasty because some values are little-endian 32 bit encoded, not CScriptNum encoded, and others are varint encoded?
But it’s nice to be explicit in each case, for people like me who are not deep in the weeds of bitcoin’s onchain representation, since it helps when considering how to use this alongside things like OP_CAT and extended arithmetic opcodes.
35+
36+* There is at least one element on the stack, fail otherwise.
37+* The element on the stack is at least 32 bytes long, fail otherwise.
38+* The first 32 bytes are interpreted as the TxHash and the remaining suffix bytes specify the TxFieldSelector.
39+* If the TxFieldSelector is invalid, fail.
40+* The actual TxHash of the transaction at the current input index, calculated
-VERIFY opcode to not touch the stack.
67+ * 2. `TXFS_INPUTS_TEMPLATE | TXFS_OUTPUTS_ALL`
68+ * 3. `TXFS_INOUT_NUMBER | TXFS_INOUT_SELECTION_ALL`
69+ * 4. `TXFS_INOUT_NUMBER | TXFS_INOUT_SELECTION_ALL`
70+ * the `0x00` byte: it is set equal to `TXFS_SPECIAL_ALL`, which means "ALL" and is primarily
71+ useful to emulate `SIGHASH_ALL` when `OP_TXHASH` is used in combination
72+ with `OP_CHECKSIGFROMSTACK`.<br>Special case `TXFS_SPECIAL_TEMPLATE` is 4
TXFS_SPECIAL_ALL? Maybe same as Rusty is saying.
74+ * 1. `TXFS_ALL`
75+ * 2. `TXFS_INPUTS_ALL | TXFS_OUTPUTS_ALL`
76+ * 3. `TXFS_INOUT_NUMBER | TXFS_INOUT_SELECTION_ALL`
77+ * 4. `TXFS_INOUT_NUMBER | TXFS_INOUT_SELECTION_ALL`
78+
79+* The first byte of the TxFieldSelector has its 8 bits assigned as follows, from lowest to highest:
154+
155+Effectively, this allows a user to select
156+* all in/outputs
157+* the current input index
158+* the leading in/outputs up to 8192
159+* up to 64 individually selected in/outputs
PUSH3[0x00 0x18 0x40] TXHASH PUSH3[0x00 0x60 0x40] TXHASH EQUAL (don’t commit the field selector, grab the sPK/value, for just this input/output, check the txhashes are equal) ? You mightn’t be able to do anything very interesting though.
Instead, add a notice about malleability.
5+ Author: Steven Roose <steven@roose.io>
6+ Comments-URI: https://github.com/bitcoin/bips/wiki/Comments:BIP-tbd
7+ Status: Draft
8+ Type: Standards Track
9+ Created: 2023-09-03
10+ License: BSD-3-Clause
113+ * Specifying which fields of the outputs will be selected:
114+ * 7: scriptPubkeys (`TXFS_OUTPUTS_SCRIPTPUBKEYS`)
115+ * 8: values (`TXFS_OUTPUTS_VALUES`)
116+
117+* We define as follows:
118+ * `TXFS_ALL = TXFS_VERSION | TXFS_LOCKTIME | TXFS_CURRENT_INPUT_IDX | TXFS_CURRENT_INPUT_CONTROL_BLOCK | TXFS_CURRENT_INPUT_LAST_CODESEPARATOR_POS | TXFS_INPUTS | TXFS_OUTPUTS | TXFS_CONTROL`
I think this is a typo?
0 * `TXFS_ALL = TXFS_VERSION | TXFS_LOCKTIME | TXFS_CURRENT_INPUT_IDX | TXFS_CURRENT_INPUT_CONTROL_BLOCK | TXFS_CURRENT_INPUT_LAST_CODESEPARATOR_POS | TXFS_INPUTS_ALL | TXFS_OUTPUTS_ALL | TXFS_CONTROL`
57+## TxFieldSelector
58+
59+The TxFieldSelector has the following semantics. We will give a brief conceptual
60+summary, followed by a reference implementation of the CalculateTxHash function.
61+
62+* There are two special cases for the TxFieldSelector:
Can I suggest that we move the section on the special cases to be after the fundamental atomic selector flags? It’s much easier to read this document by building up from the building blocks rather than starting with the template and having to squint to find where all the sub-properties are defined later.
I think the discussion for the optimized case follows very naturally once we understand the components here.
208+| | | | | ^ scriptSigs
209+| | | | ^ prevout scriptPubkeys
210+| | | ^ prevout values
211+| | ^ taproot annexes
212+| ^ scriptPubkeys
213+^ values
The “v” threw me off so I’m offering a suggestion that may help readability.
0<-> outputs
1| | <---------> inputs
21 1 1 1 1 1 1 1
3| | | | | | | ^ prevouts
4| | | | | | ^ sequences
5| | | | | ^ scriptSigs
6| | | | ^ prevout scriptPubkeys
7| | | ^ prevout values
8| | ^ taproot annexes
9| ^ scriptPubkeys
10^ values
240+| | | | <--------------> first idx: 1
241+| | | | <-----> selection count: 0b10 == 2
242+| | | ^ index size 0: single byte per index
243+| | ^ absolute index
244+| ^ individual mode
245+^ don't commit the number of in/outputs
00 1 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 1
1| | | | <-------------> second idx: 3
2| | | | <-------------> first idx: 1
3| | | | <-----> selection count: 0b10 == 2
4| | | ^ index size 0: single byte per index
5| | ^ absolute index
6| ^ individual mode
7^ don't commit the number of in/outputs
Hey @stevenroose, this pull request has had unaddressed review for over six months. Are you still working on this? If not, is there someone else that wants to pick this project up?
pinging @stevenroose
