bip-0375: fix P2WPKH scriptPubKeys and signatures in test vectors #2316

What is wrong

In bip375_test_vectors.json every P2WPKH input spends the program SHA256(pubkey)[:20] instead of HASH160(pubkey), so the key the input carries cannot spend it. For example, key 02c817bb…08bf has program 229a72d3…4f94 where it should be 1e2ad787…163b.

None of the 42 ECDSA signatures in PSBT_IN_PARTIAL_SIG verifies against the transaction its PSBT describes, whatever the input type. Where I could reconstruct the signed message, it is a BIP 143 hash with the SHA256-based program as script code, byte-reversed txids, and the silent payment output scripts empty or dropped. The reference validator does not check signatures, so no test fails.

The generator has since changed the program hash (5ef0d13) and the txid byte order in signing (1e574cf). Regenerating would also change unrelated values, such as the prevout txids, so the file is repaired in place.

What changes

• Each P2WPKH program becomes HASH160(pubkey): in PSBT_IN_WITNESS_UTXO, and in the supplementary witness_utxo and prevout_scriptpubkey.

• The signatures are re-made with RFC 6979 and low S:

  • BIP 143 for P2WPKH and P2SH-P2WPKH;
  • the legacy sighash for P2PKH and for the bare OP_2 input.

  The keys come from the file's own supplementary data.

• In "non-SIGHASH_ALL signature on input with sp output", PSBT_IN_SIGHASH_TYPE is SIGHASH_NONE but the signature was SIGHASH_ALL. It is now a SIGHASH_NONE signature, as BIP 174 requires.

• The two signatures with no defined sighash are removed: on the segwit v2 input (now signed: false in the supplementary data) and in "missing PSBT_OUT_SCRIPT field when sending to non-sp output". Both vectors still fail for the same reason.

Nothing else changes. In the PSBTs, only PSBT_IN_WITNESS_UTXO and PSBT_IN_PARTIAL_SIG differ. The runner's -vv output is byte-identical before and after: 43 passed.

How it was verified

Signatures were checked with the script below. It uses the Bitcoin Core test_framework vendored in the generator at 4811197.

Flipping one byte of a verifying signature makes it fail.

git clone https://github.com/macgyver13/bip375-test-generator gen   # 4811197
python3 sigcheck2.py bip-0375 gen bip-0375/bip375_test_vectors.json --controls

<details> <summary>sigcheck2.py</summary>

#!/usr/bin/env python3
"""Verify every signature in a BIP-375 test vector file.

Usage: sigcheck2.py <bip-0375 dir> <generator checkout> <vectors.json> [--controls] [--quiet] [--empty-missing]

The generator checkout supplies the vendored Bitcoin Core test_framework
(macgyver13/bip375-test-generator). Each signature is checked against the
unsigned transaction its PSBT describes:
- P2WPKH and P2SH-P2WPKH under BIP 143, P2PKH and other non-witness scripts
  under the legacy signature hash with the scriptPubKey (or redeem script)
  as scriptCode, P2TR key path under BIP 341;
- a scriptPubKey that should commit to the signing key (P2WPKH,
  P2SH-P2WPKH, P2PKH) and does not is reported as UNSPENDABLE;
- a witness program of version 2..16 has no signature hash: UNDEFINED;
- an output without PSBT_OUT_SCRIPT leaves the transaction undefined for a
  signature committing to it: UNDEFINED, or, with --empty-missing, signed
  as an empty scriptPubKey;
- a signature whose type differs from PSBT_IN_SIGHASH_TYPE is flagged.
"""
import json
import struct
import sys
from io import BytesIO

BIP_DIR, GEN_DIR, VEC = sys.argv[1:4]
CONTROLS = "--controls" in sys.argv
QUIET = "--quiet" in sys.argv
EMPTY_MISSING = "--empty-missing" in sys.argv  # convention: a missing PSBT_OUT_SCRIPT signs as empty
sys.path[:0] = [BIP_DIR, BIP_DIR + "/deps", BIP_DIR + "/deps/secp256k1lab/src"]
from validator.psbt_bip375 import BIP375PSBT  # noqa: E402

sys.path.insert(0, GEN_DIR)
from test_framework.key import ECPubKey, verify_schnorr  # noqa: E402
from test_framework.messages import CTransaction, CTxIn, CTxOut, COutPoint  # noqa: E402
from test_framework.script import (  # noqa: E402
    CScript,
    LegacySignatureHash,
    SegwitV0SignatureHash,
    TaprootSignatureHash,
    hash160,
)
from test_framework.script_util import keyhash_to_p2pkh_script  # noqa: E402

G_TX_VERSION, G_FALLBACK_LOCKTIME = 0x02, 0x03
I_NON_WITNESS_UTXO, I_WITNESS_UTXO, I_PARTIAL_SIG, I_SIGHASH_TYPE = 0x00, 0x01, 0x02, 0x03
I_REDEEM, I_WITNESS_SCRIPT = 0x04, 0x05
I_PREV_TXID, I_OUT_INDEX, I_SEQUENCE = 0x0E, 0x0F, 0x10
I_REQ_TIME_LOCKTIME, I_REQ_HEIGHT_LOCKTIME = 0x11, 0x12
I_TAP_KEY_SIG, I_TAP_SCRIPT_SIG = 0x13, 0x14
O_AMOUNT, O_SCRIPT = 0x03, 0x04


class Undefined(Exception):
    pass


class Unspendable(Exception):
    pass


def u32(b):
    return struct.unpack("<I", b)[0]


def get(m, key, default=None):
    return m.map.get(key, default)


def prevout(im):
    """(amount, scriptPubKey) of the spent output, or None."""
    wu = get(im, I_WITNESS_UTXO)
    nw = get(im, I_NON_WITNESS_UTXO)
    if nw is not None:
        tx = CTransaction()
        tx.deserialize(BytesIO(nw))
        out = tx.vout[u32(get(im, I_OUT_INDEX))]
        if wu is not None:
            assert wu == out.serialize(), "witness_utxo disagrees with non_witness_utxo"
        return out.nValue, bytes(out.scriptPubKey)
    if wu is not None:
        return struct.unpack("<q", wu[:8])[0], wu[9:]
    return None


def classify(im):
    po = prevout(im)
    if po is None:
        return "no-utxo"
    spk = po[1]
    if len(spk) == 22 and spk[:2] == b"\x00\x14":
        return "p2wpkh"
    if len(spk) == 34 and spk[:2] == b"\x00\x20":
        return "p2wsh"
    if len(spk) == 34 and spk[:2] == b"\x51\x20":
        return "p2tr"
    if 4 <= len(spk) <= 42 and 0x52 <= spk[0] <= 0x60 and spk[1] == len(spk) - 2:
        return "witness-v%d" % (spk[0] - 0x50)
    if len(spk) == 25 and spk[:3] == b"\x76\xa9\x14" and spk[23:] == b"\x88\xac":
        return "p2pkh"
    if len(spk) == 23 and spk[:2] == b"\xa9\x14" and spk[22] == 0x87:
        rs = get(im, I_REDEEM)
        if rs and len(rs) == 22 and rs[:2] == b"\x00\x14":
            return "p2sh-p2wpkh"
        if rs and len(rs) == 34 and rs[:2] == b"\x00\x20":
            return "p2sh-p2wsh"
        return "p2sh"
    return "bare(%s)" % spk.hex()


def locktime(psbt):
    t = [u32(get(im, I_REQ_TIME_LOCKTIME)) for im in psbt.i if get(im, I_REQ_TIME_LOCKTIME) is not None]
    h = [u32(get(im, I_REQ_HEIGHT_LOCKTIME)) for im in psbt.i if get(im, I_REQ_HEIGHT_LOCKTIME) is not None]
    if t or h:
        raise Undefined("required locktimes present; not implemented")
    return u32(get(psbt.g, G_FALLBACK_LOCKTIME, b"\0" * 4))


def build_tx(psbt, ht, idx):
    """The unsigned transaction a signature of type ht on input idx commits to."""
    tx = CTransaction()
    tx.version = u32(get(psbt.g, G_TX_VERSION))
    tx.nLockTime = locktime(psbt)
    for im in psbt.i:
        seq = u32(get(im, I_SEQUENCE, b"\xff" * 4))
        op = COutPoint(int.from_bytes(get(im, I_PREV_TXID), "little"), u32(get(im, I_OUT_INDEX)))
        tx.vin.append(CTxIn(op, b"", seq))
    base = ht & 0x1F
    for j, om in enumerate(psbt.o):
        spk = get(om, O_SCRIPT)
        committed = base == 1 or (base == 3 and j == idx) or base not in (1, 2, 3)
        if spk is None:
            if committed and not EMPTY_MISSING:
                raise Undefined("output %d has no PSBT_OUT_SCRIPT" % j)
            spk = b""
        tx.vout.append(CTxOut(struct.unpack("<q", get(om, O_AMOUNT))[0], spk))
    return tx


def ecdsa_msg(psbt, idx, kind, pub, ht):
    im = psbt.i[idx]
    amount, spk = prevout(im)
    tx = build_tx(psbt, ht, idx)
    if kind == "p2wpkh":
        if hash160(pub) != spk[2:]:
            raise Unspendable("HASH160(pubkey) is not the witness program")
        return SegwitV0SignatureHash(keyhash_to_p2pkh_script(spk[2:]), tx, idx, ht, amount)
    if kind == "p2sh-p2wpkh":
        rs = get(im, I_REDEEM)
        if hash160(rs) != spk[2:22] or hash160(pub) != rs[2:]:
            raise Unspendable("redeem script or pubkey does not match")
        return SegwitV0SignatureHash(keyhash_to_p2pkh_script(rs[2:]), tx, idx, ht, amount)
    if kind in ("p2wsh", "p2sh-p2wsh"):
        return SegwitV0SignatureHash(CScript(get(im, I_WITNESS_SCRIPT)), tx, idx, ht, amount)
    if kind == "p2pkh":
        if hash160(pub) != spk[3:23]:
            raise Unspendable("HASH160(pubkey) is not the P2PKH hash")
        return LegacySignatureHash(CScript(spk), tx, idx, ht)[0]
    if kind == "p2sh":
        return LegacySignatureHash(CScript(get(im, I_REDEEM)), tx, idx, ht)[0]
    if kind.startswith("bare("):
        return LegacySignatureHash(CScript(spk), tx, idx, ht)[0]
    if kind.startswith("witness-v"):
        raise Undefined("%s has no signature hash" % kind)
    raise Undefined("no sighash rule for " + kind)


def check_ecdsa(psbt, idx, kind, pub, sig):
    ht = sig[-1]
    msg = ecdsa_msg(psbt, idx, kind, pub, ht)
    key = ECPubKey()
    key.set(pub)
    return key.verify_ecdsa(sig[:-1], msg), ht


def check_taproot(psbt, idx, sig):
    spent = []
    for im in psbt.i:
        po = prevout(im)
        if po is None:
            raise Undefined("BIP341 needs every input's utxo")
        spent.append(CTxOut(po[0], po[1]))
    ht = sig[64] if len(sig) == 65 else 0
    tx = build_tx(psbt, ht if ht else 1, idx)
    msg = TaprootSignatureHash(tx, spent, ht, idx)
    return verify_schnorr(spent[idx].scriptPubKey[2:], sig[:64], msg), ht


def sigs_of(im):
    for k, val in im.map.items():
        if isinstance(k, int):
            if k == I_TAP_KEY_SIG:
                yield "TAP_KEY_SIG", k, val
            continue
        if k[0] == I_PARTIAL_SIG:
            yield "PARTIAL_SIG", k, val
        elif k[0] == I_TAP_KEY_SIG and len(k) == 1:
            yield "TAP_KEY_SIG", k, val
        elif k[0] == I_TAP_SCRIPT_SIG:
            yield "TAP_SCRIPT_SIG", k, val


def verify(psbt, idx, kind, field, k, val):
    try:
        if field == "PARTIAL_SIG":
            ok, ht = check_ecdsa(psbt, idx, kind, k[1:], val)
        elif field == "TAP_KEY_SIG":
            ok, ht = check_taproot(psbt, idx, val)
        else:
            return "NOT-CHECKED (script path)", None
    except Undefined as e:
        return "UNDEFINED: %s" % e, None
    except Unspendable as e:
        return "UNSPENDABLE: %s" % e, None
    return ("VERIFIES" if ok else "FAILS"), ht


def main():
    global CONTROLS
    with open(VEC) as f:
        data = json.load(f)
    tally = {}
    for kind in ("valid", "invalid"):
        for n, v in enumerate(data[kind]):
            psbt = BIP375PSBT.from_base64(v["psbt"])
            assert psbt.to_base64() == v["psbt"], "round trip not byte-identical"
            for idx, im in enumerate(psbt.i):
                itype = classify(im)
                for field, k, val in sigs_of(im):
                    res, ht = verify(psbt, idx, itype, field, k, val)
                    st = get(im, I_SIGHASH_TYPE)
                    note = ""
                    if st is not None and ht is not None and u32(st) != (ht if ht else 0):
                        note = " [PSBT_IN_SIGHASH_TYPE=%d, signature byte=%d]" % (u32(st), ht)
                    key = (kind, itype, res.split(":")[0])
                    tally[key] = tally.get(key, 0) + 1
                    if not QUIET:
                        print("%-7s #%-2d in%d %-12s %-11s sighash=%-4s %s%s | %s" % (
                            kind, n, idx, itype, field, "0x%02x" % ht if ht is not None else "-",
                            res, note, v["description"][:80]))
                    if CONTROLS and res == "VERIFIES" and field == "PARTIAL_SIG":
                        bad = bytearray(val)
                        bad[10] ^= 0x01
                        r2, _ = verify(psbt, idx, itype, field, k, bytes(bad))
                        print("    control: byte 10 of the signature flipped ->", r2)
                        CONTROLS = False
    print("summary:")
    for key in sorted(tally):
        print("  %-7s %-26s %-12s %d" % (*key, tally[key]))


if __name__ == "__main__":
    main()

</details>

Interaction with #2256 and #2207

#2256 (also mine) and #2207 conflict with this PR in bip375_test_vectors.json. Each edits some of the same vectors and has P2WPKH inputs with the same defect. Whichever of this PR and #2256 lands second needs a rebase and the same P2WPKH repair. I have prepared and checked the resolved file for both orders. The same repair applies to #2207's vectors.