invalid transaction decoding

widdoh commented at 7:06 pm on December 5, 2020: none

using: bitcoin-tx -json 020000000001010000000000000000000000000000000000000000000000000000000000000000ffffffff4b03c68708046ff8415c622f4254432e434f4d2ffabe6d6de1965d02c68f928e5b244ab1965115a36f56eb997633c7f690124bbf43644e23080000000ca3d3af6d005a65ff0200fd00000000ffffffff03f4c1fb4b0000000016001497cfc76442fe717f2a3f0cc9c175f7561b6619970000000000000000266a24aa21a9ed957d1036a80343e0d1b659497e1b48a38ebe876a056d45965fac4a85cda84e1900000000000000002952534b424c4f434b3a8e092581ab01986cbadc84f4b43f4fa4bb9e7a2e2a0caf9b7cf64d939028e22c0120000000000000000000000000000000000000000000000000000000000000000000000000 gives an error: invalid transaction encoding

Decode the transaction into a JSON block

error: invalid transaction encoding

All the time under a debug build x86 or release build x86 or debug Release Build x64

Downloaded the entire source code on 30/11/2020 19:21:00 UTC from Github

Windows 10 Pro, Intel Cor I7-5500 CPU@2.40 GHz, 512 GB SSD PS same error on windows server 2019 48 cores, and various other windows systems

No, not GUI related

Compiled the source code using Visual Studio 2019 ver. 16.8.2 Compiles Successfully P.S. Downloaded pre compiled bitcoin-tx.exe File Version 0.20.1.0 31/07/2020 and tested it. It decodes the transaction successfully

widdoh renamed this:
~~invalid transaction encoding~~
invalid transaction decoding
on Dec 5, 2020

jonasschnelli commented at 8:46 am on December 7, 2020: contributor

Where have you encoded this transaction? Could it be that the encoding is invalid 😎?

widdoh commented at 9:20 am on December 7, 2020: none

Lol possible except this and many more transactions where extracted from the blockchain. I decoded the entire chain and there were a few thousand similar transactions. Also note in my post I said that the precompiled Bitcoin-tx decided it properly

iFA88 commented at 2:41 pm on December 7, 2020: none

Hey, i just want to open the same issue. I use Bitcoin Core version v0.20.1 and can not decode my raw transaction:

00200000000010176da57a8c5e8ea391d0454991aff153f2250e8b27040f20f122d07370bcac876000000006a47304402204ade4923b31e6d42c1b703a4c0fb44cb6de058abcda03e580864da3bfd14449c02205e038e3125091547f59e158067837df6e48bd958be8ce5b463ec9daca90f91440121037da65957101c17ca267052f9b7c8ddda088d37ee5b70a3ee3d2ea70b3554ce35ffffffff02e80300000000000017a914dfe89617d47489acf2fd3b024b232122f3fbc72a87281c0000000000001976a914118c67c7bf1596fa67c2571ea7595ed8021308cc88ac0000000000

The bug is that when the segwit array length indicator is zero then the decoder throws error.

When i remove the segwit mark and the data then works:

0020000000176da57a8c5e8ea391d0454991aff153f2250e8b27040f20f122d07370bcac876000000006a47304402204ade4923b31e6d42c1b703a4c0fb44cb6de058abcda03e580864da3bfd14449c02205e038e3125091547f59e158067837df6e48bd958be8ce5b463ec9daca90f91440121037da65957101c17ca267052f9b7c8ddda088d37ee5b70a3ee3d2ea70b3554ce35ffffffff02e80300000000000017a914dfe89617d47489acf2fd3b024b232122f3fbc72a87281c0000000000001976a914118c67c7bf1596fa67c2571ea7595ed8021308cc88ac00000000

MarcoFalke commented at 4:08 pm on December 7, 2020: member

Probably another regression of #17775

MarcoFalke added this to the milestone 0.21.0 on Dec 7, 2020

MarcoFalke added the label Utils/log/libs on Dec 7, 2020

MarcoFalke added the label Bug on Dec 7, 2020

jonatack commented at 9:30 pm on December 7, 2020: member

On master: I can decode @widdoh’s transaction but see the same thing as what @iFA88 reports.

widdoh commented at 9:50 pm on December 7, 2020: none

After decoding the blockchain data files I came across 8656 transactions similar to the one I posted!

iFA88 commented at 10:36 pm on December 7, 2020: none

It’s possible that there is a regression here that we should look into, but inevitably the result will always be wrong for some (possibly exceptional) cases.

That’s wrong, it is very easy to decode any transaction even signed or empty.

widdoh commented at 10:40 pm on December 7, 2020: none

I am using Bitcoin-tx to do the decoding. In the source code the error is generated by : if (!DecodeHexTx(tx, line, true)) { throw std::runtime_error(“invalid transaction encoding”);

sipa commented at 10:42 pm on December 7, 2020: member

@widdoh You’ve indeed stumbled upon (yet another) inadequacy in the heuristic hex decoder. It incorrectly decides to not return the extended serialization if it doesn’t pass a heurstic, even if the legacy decoder fails. See #20595 for a fix. @iFA88 Your hex encoding is actually invalid. It has the extended serialization marker (“0001”) for a transaction with no witnesses. That is invalid according to BIP144.

sipa commented at 10:54 pm on December 7, 2020: member

@widdoh Your issue seems like it may be a regression caused by #17775.

iFA88 commented at 11:11 pm on December 7, 2020: none

iFA88 Your hex encoding is actually invalid. It has the extended serialization marker (“0001”) for a transaction with no witnesses. That is invalid according to BIP144.

I dont see that in https://en.bitcoin.it/wiki/BIP_0141 can you please highlight for me?

sipa commented at 11:12 pm on December 7, 2020: member

BIP144 specifies the P2P serialization, not BIP141.

If the witness is empty, the old serialization format must be used.

iFA88 commented at 11:26 pm on December 7, 2020: none

You have right, sorry i linked bad page.

widdoh commented at 0:11 am on December 8, 2020: none

@sipa thanks i will try the fix and decode the entire blockchain again and report any issues

sipa commented at 0:13 am on December 8, 2020: member

@widdoh Cool. I believe it will succeed on all blockchain transaction with that patch, though in theory it’s still possible that some decode incorrectly as 0-input transactions instead.

widdoh commented at 0:25 am on December 8, 2020: none

@sipa sorry no cigar but an improvement. File blk01499.dat (just a file i tested before) had 56 such transactions. After your fix they are down to 28! Lucky i did not decode the whole chain to check :) Below is a sample tx that failed! 020000000001010000000000000000000000000000000000000000000000000000000000000000FFFFFFFF4B03C68708046FF8415C622F4254432E434F4D2FFABE6D6DE1965D02C68F928E5B244AB1965115A36F56EB997633C7F690124BBF43644E23080000000CA3D3AF6D005A65FF0200FD00000000FFFFFFFF03F4C1FB4B0000000016001497CFC76442FE717F2A3F0CC9C175F7561B6619970000000000000000266A24AA21A9ED957D1036A80343E0D1B659497E1B48A38EBE876A056D45965FAC4A85CDA84E1900000000000000002952534B424C4F434B3A8E092581AB01986CBADC84F4B43F4FA4BB9E7A2E2A0CAF9B7CF64D939028E22C0120000000000000000000000000000000000000000000000000000000000000000000000000

sipa commented at 0:29 am on December 8, 2020: member

@widdoh That one works for me?

0$ ./src/bitcoin-tx -json 020000000001010000000000000000000000000000000000000000000000000000000000000000FFFFFFFF4B03C68708046FF8415C622F4254432E434F4D2FFABE6D6DE1965D02C68F928E5B244AB1965115A36F56EB997633C7F690124BBF43644E23080000000CA3D3AF6D005A65FF0200FD00000000FFFFFFFF03F4C1FB4B0000000016001497CFC76442FE717F2A3F0CC9C175F7561B6619970000000000000000266A24AA21A9ED957D1036A80343E0D1B659497E1B48A38EBE876A056D45965FAC4A85CDA84E1900000000000000002952534B424C4F434B3A8E092581AB01986CBADC84F4B43F4FA4BB9E7A2E2A0CAF9B7CF64D939028E22C0120000000000000000000000000000000000000000000000000000000000000000000000000
1{
2    "txid": "c61adde6f1c09ef4c9864e5c5aa7d92e0c6e76e4413710cf989bae9145ef141e",
3    "hash": "ed87639c4c575798908a307e4f4938b52ab1335208fdbd461d67fdab64134fb3",
4    "version": 2,
5...

widdoh commented at 0:36 am on December 8, 2020: none

@sipa could you please post the source code for the decodr function just in case I copied the wrong one. Although I copied it from your post

sipa commented at 0:38 am on December 8, 2020: member

 0static bool DecodeTx(CMutableTransaction& tx, const std::vector<unsigned char>& tx_data, bool try_no_witness, bool try_witness)
 1{
 2    CMutableTransaction tx_extended, tx_legacy;
 3    bool ok_extended = false, ok_legacy = false;
 4
 5    // Try decoding with extended serialization support, and remember if the result successfully consumes the entire input.
 6    // Note that this will generally also work correctly on transactions in legacy format (without witness). This version
 7    // however will interpret the 0001 marker as a marker rather than as a no-input transaction.
 8    if (try_witness) {
 9        CDataStream ssData(tx_data, SER_NETWORK, PROTOCOL_VERSION);
10        try {
11            ssData >> tx_extended;
12            if (ssData.empty()) ok_extended = true;
13        } catch (const std::exception&) {
14            // Fall through.
15        }
16    }
17
18    // Try decoding with legacy serialization, and remember if the result successfully consumes the entire input.
19    if (try_no_witness) {
20        CDataStream ssData(tx_data, SER_NETWORK, PROTOCOL_VERSION | SERIALIZE_TRANSACTION_NO_WITNESS);
21        try {
22            ssData >> tx_legacy;
23            if (ssData.empty()) ok_legacy = true;
24        } catch (const std::exception&) {
25            // Fall through.
26        }
27    }
28
29    // If both deserializations succeeded, prefer the one for which CheckTxScriptsSanity returns true.
30    // If that's the case for both, prefer the extended one.
31    if (ok_extended && CheckTxScriptsSanity(tx_extended)) {
32        tx = std::move(tx_extended);
33        return true;
34    }
35    if (ok_legacy && CheckTxScriptsSanity(tx_legacy)) {
36        tx = std::move(tx_legacy);
37        return true;
38    }
39
40    // If only one deserialization succeeded, use that one.
41    if (ok_extended) {
42        tx = std::move(tx_extended);
43        return true;
44    }
45    if (ok_legacy) {
46        tx = std::move(tx_legacy);
47        return true;
48    }
49
50    // If none succeeded, we failed.
51    return false;
52}

widdoh commented at 10:10 am on December 8, 2020: none

@sipa my bad. The code i copied is the same you posted but apparently forgot to delete the error file from a previous cancelled run and though it was generated by your code :) I will run it now on the rest of the chain. Thanks

widdoh commented at 12:23 pm on December 9, 2020: none

@sipa well done. Entire blockchain decoded and no errors! Thank you

laanwj closed this on Dec 10, 2020

sidhujag referenced this in commit 0fc0cf7160 on Dec 10, 2020

DrahtBot locked this on Feb 15, 2022

invalid transaction decoding #20579