bip-0375: assign k in output index order #2256

BIP375 states one rule for assigning k to the silent payment outputs of a transaction, and its test vectors and reference validator implement another. This amends the prose to the rule the shipped artefacts follow.

The discrepancy

bip-0375.mediawiki L240-241 asks for the codes sharing a scan key to be sorted lexicographically, with a tie-break by output index for codes sharing both keys. bip-0375/validator/validate_psbt.py does not sort: it tracks k per scan key while walking enumerate(psbt.o). The vectors were generated the same way.

Measured with bip-0375/test_runner.py on Python 3.12, at 60f5b33b:

That vector's two spend keys are in descending order, so a lexicographic sort gives k = 0 to output 1, and the scripts the file publishes are the ones index order derives. No invalid vector changes verdict under either rule; the two invalid vectors named after ordering do not discriminate, since in one the spend keys are already ascending and in the other all three codes are identical.

Why index order rather than the reverse fix

#2207 closes the same gap the other way, by correcting the vectors and the validator to the prose. Both work. This one is proposed because the rule is cheaper to state and to implement:

• lexicographic order needs a canonical encoding of a "code" and a total order on it. "The codes" admits at least three readings — the 66-byte PSBT_OUT_SP_V0_INFO, the bech32m address string, or the (scan, spend) pair — and the BIP names none of them. Output indices are already in the psbt and already agreed by both parties;
• the property a code-based order would buy, invariance under output reordering, is unavailable here: a Signer that sets any missing PSBT_OUT_SCRIPT must clear the Inputs Modifiable and Outputs Modifiable flags (L185), and a psbt carrying a script with PSBT_GLOBAL_TX_MODIFIABLE non-zero is invalid. The outputs cannot be reordered once the scripts exist;
• what the rule has to buy appears to be inter-party determinism only. A BIP352 receiver scans by deriving P_k for k = 0, 1, 2 … and matching against the transaction's outputs, so it never learns which output was assigned which k; and permuting k within a scan-key group still pays each recipient the same amount, since an output's script is derived from its own spend key. If that is right, any deterministic rule is correct and the choice is one of cost.

If the owners prefer #2207's direction, this should be closed in its favour — the two are alternatives on the ordering question, and #2207's other half, the labeled spend key in PSBT_OUT_SP_V0_INFO, is independent of it and looks right either way.

The second sentence

Dropping it is not a loss: one counter per scan key covers codes sharing both keys with no tie-break. What it left unsaid is added in its place, because it is the part an implementer gets wrong — the counter counts the outputs of that scan key, not the output index, which the vector "three sp outputs (same scan key) / two regular outputs - k values assigned independently of output index" exercises.

Checks

scripts/link-format-chk.sh, scripts/buildtable.pl, scripts/diffcheck.sh and typos all pass locally. The version header and the changelog entry are the first thing to rebase if #2207 lands first.

Found while implementing BIP375 in btclib, which follows the vectors: https://github.com/btclib-org/btclib/issues/768

@fametrano Thank you for your proposal. Can you summarize the PR description more concisely in your own words, please.

@jonatack fair request, in my own words:

BIP375's text says to sort the silent payment codes lexicographically to assign k; instead, its validator and its vectors assign k by output index. This PR changes the text to match the vectors, #2207 changes the vectors to match the text; either removes the contradiction. I prefer index order because it needs no definition of how a "code" is encoded and compared, and the outputs cannot be reordered once the scripts are set anyway (L185).

To reproduce in a minute, apply this to bip-0375/validator/validate_psbt.py and run python3 test_runner.py in bip-0375/:

@@ -317,11 +317,13 @@
     scan_key_k_values = {}

     # Validate each SP output
-    for output_idx, output_map in enumerate(psbt.o):
-        if PSBT_OUT_SP_V0_INFO not in output_map:
-            continue  # Skip non-SP outputs
-
-        sp_info = output_map[PSBT_OUT_SP_V0_INFO]
+    sp_outputs = [
+        (output_map[PSBT_OUT_SP_V0_INFO], output_idx, output_map)
+        for output_idx, output_map in enumerate(psbt.o)
+        if PSBT_OUT_SP_V0_INFO in output_map
+    ]
+    sp_outputs.sort(key=lambda entry: (entry[0], entry[1]))
+    for sp_info, output_idx, output_map in sp_outputs:
         scan_pubkey_bytes = sp_info[:33]
         spend_pubkey_bytes = sp_info[33:]

Today, at master 09e2103: unchanged, 42 passed, 0 failed; with the sort, 41 passed, 1 failed, the failure being "two sp outputs - output 0 uses label=3 / output 1 uses label=1", whose spend keys are in descending order. (My description above says valid[8] for that vector; it is index 9 in the file.)

The PR also rewords two vector descriptions that no longer said what they test once the text is index order ("not sorted lexicographically by spend key" names a rule the text no longer states): both are reworded, in the file and in the BIP's table, and the validator's counter comment now says which order it follows. No vector bytes change.

On #2207. If the owners prefer this direction, #2207's sort and its ordering-driven vector regeneration become unnecessary, but its other half, the labeled spend key in PSBT_OUT_SP_V0_INFO, is independent and still needed: I would suggest reducing #2207 to that half rather than closing it, and I am glad to help rebase it on this. If the owners prefer #2207's direction, this one should be closed instead.

@fametrano #2207 was opened to match the vectors and validation with the BIP text.

@macgyver13 right, #2207 and this PR fix the same contradiction in opposite directions: #2207 edits the vectors and validator to match the prose (lexicographic), this PR edits the prose to match the vectors and validator (output-index order).

Measured against master (09e2103): the validator as published assigns k by output index and passes all 42 vectors. Applying #2207's lexicographic sort instead gives 41/42 — the "two sp outputs, output 0 label=3 / output 1 label=1" vector, published as valid, no longer produces its own PSBT_OUT_SCRIPTs, because its two spend keys are in descending order.

So the vectors and the reference validator already agree on index order; only the prose dissents. That's why I proposed changing the sentence rather than the vectors. If the editors prefer #2207's direction, that vector's expected scripts have to change too. Happy to go whichever way they decide.

nACK

Prefer #2207. Correcting implementations to match BIP text should be the default.

Also e4ba7f8 includes pycache files

Thanks for your input @macgyver13. @fametrano: I think it’s up to the owners of the BIP rather than the Editors whether they want to adjust the text of the BIP or the implementation. cc: @andrewtoth, @achow101, @josibake

Thanks @macgyver13 — good catch. The stray __pycache__ files are removed from the branch; nothing under bip-0375/deps or the validator carries bytecode any more. The direction itself is for the BIP owners to weigh, as @murchandamus notes.