Problem with decoding transaction #12989

<!-- Describe the issue -->

I have a problem with decoding transaction on testnet. I call method getrawtransaction for txhash f0a1d41cb2d1cea36b334c77f90f1a88b0f01332fffefaac3be9bb77f5b5f5b6

But then I call the decoderawtransaction it returns another txid in decoded transaction data: 22519827b1dc985ed4d0115e58c3f327a60e44015c8499ff3c5bc7fe5a8ce3f0

<!--- What behavior did you expect? -->

I expect that txid in both cases should be the same

<!--- How reliably can you reproduce the issue, what are the steps to do so? -->

It occurs everytime, on two different machines

<!-- What version of Bitcoin Core are you using, where did you get it (website, self-compiled, etc)? -->

Bitcoin Core RPC client version v0.16.0.0-g4b4d7eb255

<!-- What type of machine are you observing the error on (OS/CPU and disk type)? -->

Ubuntu linux 16.04, i7-2600 CPU

<!-- Any extra information that might be useful in the debugging process. -->

<!--- This is normally the contents of a `debug.log` or `config.log` file. Raw text or a link to a pastebin type site are preferred. -->

genosse@local:~$ bitcoin-cli -testnet getrawtransaction f0a1d41cb2d1cea36b334c77f90f1a88b0f01332fffefaac3be9bb77f5b5f5b6
02000000000101c79161617adc1cad419254fd0194193dd6786a7faf1f65df9b3a369e8e600a3c0000000000ffffffff017b134c000000000016001443aac20a116e09ea4f7914be1c55e4c17aa600b70248304502210087116858148c7f4e33c05881103444ae5d67b1b5fd922ca18c00127c9683c4a502206ede2a47953942365a01fb772d2644dcfa59968fcece1caa305d8235557b0456012103335134d7414e1d1a154600b124a96f5ef2c6ca21434d2622469a96bd5262fd5600000000
genosse@local:~$ bitcoin-cli -testnet decoderawtransaction 02000000000101c79161617adc1cad419254fd0194193dd6786a7faf1f65df9b3a369e8e600a3c0000000000ffffffff017b134c000000000016001443aac20a116e09ea4f7914be1c55e4c17aa600b70248304502210087116858148c7f4e33c05881103444ae5d67b1b5fd922ca18c00127c9683c4a502206ede2a47953942365a01fb772d2644dcfa59968fcece1caa305d8235557b0456012103335134d7414e1d1a154600b124a96f5ef2c6ca21434d2622469a96bd5262fd5600000000
{
  "txid": "22519827b1dc985ed4d0115e58c3f327a60e44015c8499ff3c5bc7fe5a8ce3f0",
  "hash": "22519827b1dc985ed4d0115e58c3f327a60e44015c8499ff3c5bc7fe5a8ce3f0",
  "version": 2,
  "size": 192,
  "vsize": 192,
  "locktime": 0,
  "vin": [
  ],
  "vout": [
    {
      "value": 20796716866.05686529,
      "n": 0,
      "scriptPubKey": {
        "asm": "9254fd0194193dd6786a7faf1f65df9b3a369e8e600a3c0000000000ffffffff017b134c000000000016001443aac20a116e09ea4f7914be1c55e4c17aa600b702 304502210087116858148c7f4e33c05881103444ae5d67b1b5fd922ca18c00127c9683c4a502206ede2a47953942365a01fb772d2644dcfa59968fcece1caa305d8235557b045601 03335134d7414e1d1a154600b124a96f5ef2c6ca21434d2622469a96bd5262fd56",
        "hex": "419254fd0194193dd6786a7faf1f65df9b3a369e8e600a3c0000000000ffffffff017b134c000000000016001443aac20a116e09ea4f7914be1c55e4c17aa600b70248304502210087116858148c7f4e33c05881103444ae5d67b1b5fd922ca18c00127c9683c4a502206ede2a47953942365a01fb772d2644dcfa59968fcece1caa305d8235557b0456012103335134d7414e1d1a154600b124a96f5ef2c6ca21434d2622469a96bd5262fd56",
        "type": "nonstandard"
      }
    }
  ]
}

genosse@local:~$ bitcoin-cli -testnet getrawtransaction d259c4573cffb992df4c864c8fb172fb934e83ce3fa58e5fabb16bccfc59e273
02000000000101d3171d300fefb4ada940a7e6b4411391909908838db0532db8cd90b36367407a0000000000ffffffff0106ea4b000000000016001435600b44f82d414e451a4d5ed2e23e7f38a0de3502483045022100da76f404dfe7d29e8283f715509a2bbec14a4fbb13603d374f760c1da6b67705022017c15cad3e1420369e6734bbdc158f58cf07f4e4afe4f854ce9a11dfaefe6d00012102192a09100e62eefcf1b6b6ed4d6c81e3b95d29f6c2f8cb39841a8158e41d0d6600000000
genosse@local:~$ raw=02000000000101d3171d300fefb4ada940a7e6b4411391909908838db0532db8cd90b36367407a0000000000ffffffff0106ea4b000000000016001435600b44f82d414e451a4d5ed2e23e7f38a0de3502483045022100da76f404dfe7d29e8283f715509a2bbec14a4fbb13603d374f760c1da6b67705022017c15cad3e1420369e6734bbdc158f58cf07f4e4afe4f854ce9a11dfaefe6d00012102192a09100e62eefcf1b6b6ed4d6c81e3b95d29f6c2f8cb39841a8158e41d0d6600000000
genosse@local:~$ bitcoin-cli -testnet decoderawtransaction $raw
{
  "txid": "a4d7ce0d0df329bc697a85c9f1108a4466d765a123b8a35e4ba835fd57346649",
  "hash": "a4d7ce0d0df329bc697a85c9f1108a4466d765a123b8a35e4ba835fd57346649",
  "version": 2,
  "size": 192,
  "vsize": 192,
  "locktime": 0,
  "vin": [
  ],
  "vout": [
    {
      "value": -54090879281.27728895,
      "n": 0,
      "scriptPubKey": {
        "asm": "OP_HASH160 a7e6b4411391909908838db0532db8cd90b36367407a0000000000ffffffff0106ea4b000000000016001435600b44f82d414e451a4d5ed2e23e7f38a0de3502 3045022100da76f404dfe7d29e8283f715509a2bbec14a4fbb13603d374f760c1da6b67705022017c15cad3e1420369e6734bbdc158f58cf07f4e4afe4f854ce9a11dfaefe6d0001 02192a09100e62eefcf1b6b6ed4d6c81e3b95d29f6c2f8cb39841a8158e41d0d66",
        "hex": "a940a7e6b4411391909908838db0532db8cd90b36367407a0000000000ffffffff0106ea4b000000000016001435600b44f82d414e451a4d5ed2e23e7f38a0de3502483045022100da76f404dfe7d29e8283f715509a2bbec14a4fbb13603d374f760c1da6b67705022017c15cad3e1420369e6734bbdc158f58cf07f4e4afe4f854ce9a11dfaefe6d00012102192a09100e62eefcf1b6b6ed4d6c81e3b95d29f6c2f8cb39841a8158e41d0d66",
        "type": "nonstandard"
      }
    }
  ]
}

Last version of bitcoin core does not work neither, there are steps to reproduce with current release (Bitcoin Core RPC client version v0.16.99.0-e76acf3):

git clone ...
rebuild ...
rm -rf ~/.bitcoin/testnet3 # remove database 
bitcoind -txindex -testnet # reload blockchain 

genosse@local:~$ /usr/local/bin/bitcoin-cli getrawtransaction f0a1d41cb2d1cea36b334c77f90f1a88b0f01332fffefaac3be9bb77f5b5f5b6
02000000000101c79161617adc1cad419254fd0194193dd6786a7faf1f65df9b3a369e8e600a3c0000000000ffffffff017b134c000000000016001443aac20a116e09ea4f7914be1c55e4c17aa600b70248304502210087116858148c7f4e33c05881103444ae5d67b1b5fd922ca18c00127c9683c4a502206ede2a47953942365a01fb772d2644dcfa59968fcece1caa305d8235557b0456012103335134d7414e1d1a154600b124a96f5ef2c6ca21434d2622469a96bd5262fd5600000000
genosse@local:~$ /usr/local/bin/bitcoin-cli decoderawtransaction 02000000000101c79161617adc1cad419254fd0194193dd6786a7faf1f65df9b3a369e8e600a3c0000000000ffffffff017b134c000000000016001443aac20a116e09ea4f7914be1c55e4c17aa600b70248304502210087116858148c7f4e33c05881103444ae5d67b1b5fd922ca18c00127c9683c4a502206ede2a47953942365a01fb772d2644dcfa59968fcece1caa305d8235557b0456012103335134d7414e1d1a154600b124a96f5ef2c6ca21434d2622469a96bd5262fd5600000000
{
  "txid": "22519827b1dc985ed4d0115e58c3f327a60e44015c8499ff3c5bc7fe5a8ce3f0",
  "hash": "22519827b1dc985ed4d0115e58c3f327a60e44015c8499ff3c5bc7fe5a8ce3f0",
  "version": 2,
  "size": 192,
  "vsize": 192,
  "locktime": 0,
  "vin": [
  ],
  "vout": [
    {
      "value": 20796716866.05686529,
      "n": 0,
      "scriptPubKey": {
        "asm": "9254fd0194193dd6786a7faf1f65df9b3a369e8e600a3c0000000000ffffffff017b134c000000000016001443aac20a116e09ea4f7914be1c55e4c17aa600b702 304502210087116858148c7f4e33c05881103444ae5d67b1b5fd922ca18c00127c9683c4a502206ede2a47953942365a01fb772d2644dcfa59968fcece1caa305d8235557b045601 03335134d7414e1d1a154600b124a96f5ef2c6ca21434d2622469a96bd5262fd56",
        "hex": "419254fd0194193dd6786a7faf1f65df9b3a369e8e600a3c0000000000ffffffff017b134c000000000016001443aac20a116e09ea4f7914be1c55e4c17aa600b70248304502210087116858148c7f4e33c05881103444ae5d67b1b5fd922ca18c00127c9683c4a502206ede2a47953942365a01fb772d2644dcfa59968fcece1caa305d8235557b0456012103335134d7414e1d1a154600b124a96f5ef2c6ca21434d2622469a96bd5262fd56",
        "type": "nonstandard"
      }
    }
  ]
}

This is a transaction that includes witness data. Does it help if you explicitly specify iswitness=true?

decoderawtransaction 02000000000101c79161617adc1cad419254fd0194193dd6786a7faf1f65df9b3a369e8e600a3c0000000000ffffffff017b134c000000000016001443aac20a116e09ea4f7914be1c55e4c17aa600b70248304502210087116858148c7f4e33c05881103444ae5d67b1b5fd922ca18c00127c9683c4a502206ede2a47953942365a01fb772d2644dcfa59968fcece1caa305d8235557b0456012103335134d7414e1d1a154600b124a96f5ef2c6ca21434d2622469a96bd5262fd5600000000 true

There are a few strange things about this:

• The txid and hash both list the transaction's (correct) wtxid. For hash this is correct behaviour; for txid it isn't.
• The decoded output does not include the inputs (which sounds like it fails to detect it as a witness tx).

The transaction has correct encoding in both witness and non-witness forms because the prev-txid makes it all align

              size of
              first "item"
              pushed
                VV
C79161617ADC1CAD419254FD0194193DD6786A7FAF1F65DF9B3A369E8E600A3C
              ^^
            size of
            perceived
            scriptpubkey

The remainder is exactly the nlocktime field

Does it help if you explicitly specify iswitness=true?

Yes, It helps. So should I specify iswitness=true every time I call decoderawtransaction?

https://github.com/btcsuite/btcd using the golang version of bitcoin, it's ok！ BTW, how can we know whether if the tx has witness data?

I think decoderawtransaction should be aware if tx contains witness data. I think it is better to remove iswitness option altogether. It will make the RPC more consistent as segwit is already active and large percentage of tx contain witness data.

'iswitness' option looks useless to me, I have to specify iswitness=true every time I call decoderawtranscation, agreed with @vizeet

I think decoderawtransaction should be aware if tx contains witness data.

decoderawtransaction attempts to do that. However it is difficult to be aware of whether a transaction includes witness data because decoderawtransaction also needs to handle the case of an un-funded transaction so 0-input 1-output transactions are acceptable. Unfortunately witness transactions are signaled by having 0 inputs and 1 output to a traditional parser, and when we see these from other peers, we know that it must be witness, but the same assumption cannot be made for decoderawtransaction.

decoderawtransaction does in fact do some detection. There are some heuristics that it checks for, and it can properly identify many witness transactions when in the past were decoded as non-witness transactions. However some transactions still do slip through the heuristics which is the problem that you are experiencing.

Is it ok to unconditionally set iswitness=true? This appears to work for both segwit and non-segwit txns, so it appears to function better than the default (without specifying iswitness flag).

Can someone please confirm or deny?

@kushal256 No. If you specify iswitness=true, and there are two possible interpretations, it will prefer the one which has a witness. If you specify iswitness=false, only a witness-free version will be tried. If you don't specify anything, heuristics will be used to determine which is the most reasonable interpretation.

You can specify iswitness=true if you know the transaction has at least one input.

Thanks @sipa for your feedback, much appreciated!

So what is the recommended logic for 0 input coinbase txns, if it's not safe to unconditionally set isWitness=true?

For any real, fully valid, or on-chain transaction you can you use iswitness=true, even coinbase transaction.

iswitness=false or not set is purely for the case of decoding partial not-fully-signed transactions.

Whew, big relief! Thanks, that makes it clear.