clarify message serialization and add test vectors to BIP-322 #1279

kallewoof commented at 10:19 AM on January 26, 2022: member

This

adds test vectors to the BIP-322 (signmessage) proposal (more needed, but better than nothing)
clarifies how messages are serialized as this was slightly ambiguous
clarifies that the <code>to_sign</code> transaction must have version, locktime, and sequence all set to 0 for the SIMPLE format (signature only)

kallewoof cross-referenced this on Jan 26, 2022 from issue BIP 322 utility functions by benthecarman

kallewoof cross-referenced this on Jan 26, 2022 from issue BIP-322 basic support by kallewoof

kallewoof force-pushed on Jan 26, 2022

in bip-0322.mediawiki:168 in aeeda3084e outdated

 164 | +=== Message hashing ===
 165 | +
 166 | +Message hashes are BIP340-tagged hashes of a message, i.e. sha256_tag(m), where tag = <code>BIP0322-signed-message</code>:
 167 | +
 168 | +* Message = "" (empty string): <code>6e1acf4f6786c3e939571f3e186f45a941208625b78ef293da8fd18327260fff</code>
 169 | +* Message = "Hello World": <code>6c5f83704b791f004f0b462ff0b4dd74b4a066facedfe87c12ea3153de5c03d0</code>

benthecarman commented at 1:50 AM on January 28, 2022:

this is concerning, I am getting c90c269c4f8fcbe6880f72a721ddfbf1914268a794cbb21cfafee13770ae19f1 for empty string and f0eb03b1a75ac6d9847f55c624a99169b5dccba2a31f5b23bea77ba270de0a7a for Hello World

kallewoof commented at 10:07 AM on January 31, 2022:

Same after length prefix removal.

Edit: to clarify, after removing the length prefix part of the serialization, I am now getting the same hashes as @benthecarman.

in bip-0322.mediawiki:180 in aeeda3084e outdated

 176 | +* corresponding address <code>bc1q9vza2e8x573nczrlzms0wvx3gsqjx7vavgkx0l</code>
 177 | +
 178 | +Produce signatures:
 179 | +
 180 | +* Message = "" (empty string): <code>AkcwRAIgClVQ8S9yX1h8YThlGElD9lOrQbOwbFDjkYb0ebfiq+oCIDHgb/X9WNalNNtqTXb465ufbv9JuLxcJf8qi7DP6yOXASECx/EgAxlkQpQ9hYjgGu6EBCPMVPwVIVJqO4XCsMvViHI=</code>
 181 | +* Message = "Hello World": <code>AkcwRAIgOi+9FWeR72LaQwZheoopqzQ8apqy2V4J9xtC+WAuMWUCIBxRYlIeSB8B00lzCEwhYIG4oSegnVPWnAmjrBJlNaCQASECx/EgAxlkQpQ9hYjgGu6EBCPMVPwVIVJqO4XCsMvViHI=</code>

benthecarman commented at 2:36 AM on January 28, 2022:

I am getting AkcwRAIgM2gBAQqvZX15ZiysmKmQpDrG83avLIT492QBzLnQIxYCIBaTpOaD20qRlEylyxFSeEA2ba9YOixpX8z46TSDtS40ASECx/EgAxlkQpQ9hYjgGu6EBCPMVPwVIVJqO4XCsMvViHI=

and

AkcwRAIgZRfIY3p7/DoVTty6YZbWS71bc5Vct9p9Fia83eRmw2QCICK/ENGfwLtptFluMGs2KsqoNSk89pO7F29zJLUx9a/sASECx/EgAxlkQpQ9hYjgGu6EBCPMVPwVIVJqO4XCsMvViHI=

probably because the hashes are different tho, looking into this

kallewoof commented at 9:59 AM on January 31, 2022:

~Removing length prefix and only serializing the message string itself I got the same result.~

kallewoof commented at 11:04 AM on January 31, 2022:

Actually, I'm still getting different signatures, even though the hash is the same:

Message = "" (empty string): <code>AkcwRAIgFuS8y5m0ym9Gj2odoVB5NIL+cPYkeEj8LL1N/6P58X8CIA6jJ9QH2iYKRXVfmhsDzHq1bMS4Adj0nb8DDSdN/SpBASECx/EgAxlkQpQ9hYjgGu6EBCPMVPwVIVJqO4XCsMvViHI=</code>
Message = "Hello World": <code>AkcwRAIgG3PASL/vRTgAqogWT6S8rUOQXNnfRzX6JncmbFlHc1ACIGQdsW+rnVmsQzyAYRQisHKFMigDmKiL7LUw4x17Fw5tASECx/EgAxlkQpQ9hYjgGu6EBCPMVPwVIVJqO4XCsMvViHI=</code>

kallewoof commented at 11:16 AM on January 31, 2022:

What does your to_spend.vin[0].scriptSig end up as? I'm wondering if the message hash serialization is again doing something unexpected.

kallewoof commented at 11:55 AM on January 31, 2022:

My end:

"" = 0020c90c269c4f8fcbe6880f72a721ddfbf1914268a794cbb21cfafee13770ae19f1
"Hello World" = 0020f0eb03b1a75ac6d9847f55c624a99169b5dccba2a31f5b23bea77ba270de0a7a

The spec says

vin[0].scriptSig = OP_0 PUSH32[ message_hash ]

which seems to be what I'm doing:

00 (OP_0)
20 (PUSH32)
c90c269c4f8fcbe6880f72a721ddfbf1914268a794cbb21cfafee13770ae19f1 (hash)

benthecarman commented at 9:16 PM on February 1, 2022:

"" = 0020c90c269c4f8fcbe6880f72a721ddfbf1914268a794cbb21cfafee13770ae19f1
"Hello World" =  0020f0eb03b1a75ac6d9847f55c624a99169b5dccba2a31f5b23bea77ba270de0a7a

maybe something is divering in our to_spend tx creation somehow

"" = 00000000010000000000000000000000000000000000000000000000000000000000000000ffffffff220020c90c269c4f8fcbe6880f72a721ddfbf1914268a794cbb21cfafee13770ae19f1000000000100000000000000001600142b05d564e6a7a33c087f16e0f730d1440123799d00000000
"Hello World" = 00000000010000000000000000000000000000000000000000000000000000000000000000ffffffff220020f0eb03b1a75ac6d9847f55c624a99169b5dccba2a31f5b23bea77ba270de0a7a000000000100000000000000001600142b05d564e6a7a33c087f16e0f730d1440123799d00000000

kallewoof commented at 3:35 AM on February 2, 2022:

For "":

to_spend = <code>00000000010000000000000000000000000000000000000000000000000000000000000000ffffffff220020c90c269c4f8fcbe6880f72a721ddfbf1914268a794cbb21cfafee13770ae19f1000000000100000000000000001600142b05d564e6a7a33c087f16e0f730d1440123799d00000000</code>
to_sign = <code>0200000001a7995a543c2866b51ba965e78d01de5db50435cde9d482bf60d8b89ba60a68c5000000000000000000010000000000000000016a00000000</code>

The to_spend looks identical to yours.

kallewoof commented at 3:37 AM on February 2, 2022:

I realize that to_sign is version 2 on my side. It can be either 0 or 2, but this isn't clearly defined anywhere, so unless it's FULL format, you have to guess.

Gut tells me you're correct on this one, i.e. it should use version 0 (disallow locktimes) for SIMPLE format. Changing that, I get

<code>AkcwRAIgM2gBAQqvZX15ZiysmKmQpDrG83avLIT492QBzLnQIxYCIBaTpOaD20qRlEylyxFSeEA2ba9YOixpX8z46TSDtS40ASECx/EgAxlkQpQ9hYjgGu6EBCPMVPwVIVJqO4XCsMvViHI=</code>

which is the same as yours.

benthecarman commented at 7:31 AM on January 31, 2022: contributor

Wasn't able to find any problems with my tagged hashes, @kallewoof are you sure yours are correct?

kallewoof commented at 7:53 AM on January 31, 2022: member

@benthecarman Will investigate. I assume your tagged hashes / BIP 340 stuff passes the test vectors in https://github.com/bitcoin/bips/blob/master/bip-0340/test-vectors.csv ?

benthecarman commented at 7:54 AM on January 31, 2022: contributor

@benthecarman Will investigate. I assume your tagged hashes / BIP 340 stuff passes the test vectors in https://github.com/bitcoin/bips/blob/master/bip-0340/test-vectors.csv ?

Yes

kallewoof commented at 9:41 AM on January 31, 2022: member

I realize the message serialization into the hasher is a little ambiguous. That's probably where the difference is happening.

Edit: this is how it is done in Core:

/**
 * string
 */
template<typename Stream, typename C>
void Serialize(Stream& os, const std::basic_string<C>& str)
{
    WriteCompactSize(os, str.size());
    if (!str.empty())
        os.write((char*)str.data(), str.size() * sizeof(C));
}

So for the empty string case, it would result in 0x00 (the length 0), and for the string "Hello World" it would be 0x0B + "Hello World", i.e. it prefixes the compact size of the string.

bip-322: clarify how the message is serialized 1b6c85b0b4

add test vectors to BIP-322 aa92d9cd6e

kallewoof force-pushed on Jan 31, 2022

kallewoof commented at 10:07 AM on January 31, 2022: member

Updated BIP to clarify that the message serialization is done without length prefix and without null terminator.

kallewoof renamed this:
~~add test vectors to BIP-322~~
clarify message serialization and add test vectors to BIP-322
on Jan 31, 2022

kallewoof commented at 11:08 AM on January 31, 2022: member

I mistakenly thought the signatures were the same but looking closer they aren't. So, hashes are now identical but the signatures are different (and only mine pass the verifymessage call).

kallewoof commented at 3:40 AM on February 2, 2022: member

I added a clarification to the to_sign transaction that version must be 0 for SIMPLE (signature only) format.

If the spent destination turns out to have locktimes and/or such, they need to use FULL format. (Half-hearted ping @apoelstra who's probably too busy to look at this.)

clarify that SIMPLE format requires version/locktime/sequence=0 for to_sign transaction f52e047d09

kallewoof force-pushed on Feb 3, 2022

kallewoof commented at 3:12 AM on February 3, 2022: member

Since it's all guess-work for SIMPLE format, I've changed the spec to require version, locktime, and sequence to all be 0 for the to_sign transaction. Will convert this to a non-draft if no one objects.

kallewoof marked this as ready for review on Feb 4, 2022

kallewoof commented at 4:51 AM on February 4, 2022: member

Self-ACK f52e047d09d6ddc7c2dbf6549dfdc91c09e9a026

Ping @luke-jr -- I will merge this soon unless you have reservations; I don't think there's anything controversial in here, but better safe than sorry.

benthecarman commented at 4:53 AM on February 4, 2022: contributor

ack f52e047d09d6ddc7c2dbf6549dfdc91c09e9a026

kallewoof merged this on Feb 7, 2022

kallewoof closed this on Feb 7, 2022

kallewoof deleted the branch on Feb 7, 2022