signrawtransactionwithwallet fails with signed non-wallet inputs with many witness stack elements #21151

I'm seeing a very strange behavior with signrawtransactionwithwallet when I provide signed non-wallet inputs. The following sample transaction (regtest) is a lightning HTLC transaction with added bitcoind wallet inputs/outputs to bump the fee:

020000000001027697c418d3c56138f1ea0469026d081705290bd0317ad9a133319fa7b4ac59b10200000000010000001cd5376d42e12bad24911ae73b13c40b530a18c3da34de8d16fcf356bf2e32180000000000fdffffff02bb080000000000002200209357506c5a4a2df7c685481aad0302a3ccfe44c4e41122e33818861a37e2e6cccdfbea020000000016001487c7624537496328484f590d04a805f0c2a213e30500483045022100aef712b8d806077872e894343f917473497841b6d859bb7cf20473597353e7d8022030c39a0dc43b742ad0b746e8968b763e3053ec1001bbae4f5368d15d20d1152683483045022100a71dba3a0ed7d005c5e99b3a3e603026e60cd2708e7590162892bae9391965ba0220179d15465950863401956d0d4bbad0f413ce465b111dc71fe511d9bbf4200d0f0120b27cf03ab42eb16a475e289df7878d8c36d9465ba6b4f22ce1d21ef217e4f3d88d76a914c333ee02e6633f9bce0281ae0dc09e29b1b2761f8763ac672103bcdf028d4e53f7fbddb817091a60118b1d2fb7ab8cbab4b14ebeb981450d6ff77c8201208763a91487520d9fc7939063f7244ce842e4a5d2507d93d188527c21036df69837ccd9b8b353bdd37a8c616e3ea094ffee4c9512fec902ae8250ffb9f952ae6775022701b175ac6851b275680000000000
{
  "txid": "d922b3e02d77e7c522fcf46a308996df764dc61ef708a5702dc3dce6c2f58882",
  "hash": "1f558b7fd77d1339c48772f6ef12a0bf6cdbc05e371c6871d885afc89e6a5e3c",
  "version": 2,
  "size": 492,
  "vsize": 248,
  "weight": 990,
  "locktime": 0,
  "vin": [
    {
      "txid": "b159acb4a79f3133a1d97a31d00b290517086d026904eaf13861c5d318c49776",
      "vout": 2,
      "scriptSig": {
        "asm": "",
        "hex": ""
      },
      "txinwitness": [
        "",
        "3045022100aef712b8d806077872e894343f917473497841b6d859bb7cf20473597353e7d8022030c39a0dc43b742ad0b746e8968b763e3053ec1001bbae4f5368d15d20d1152683",
        "3045022100a71dba3a0ed7d005c5e99b3a3e603026e60cd2708e7590162892bae9391965ba0220179d15465950863401956d0d4bbad0f413ce465b111dc71fe511d9bbf4200d0f01",
        "b27cf03ab42eb16a475e289df7878d8c36d9465ba6b4f22ce1d21ef217e4f3d8",
        "76a914c333ee02e6633f9bce0281ae0dc09e29b1b2761f8763ac672103bcdf028d4e53f7fbddb817091a60118b1d2fb7ab8cbab4b14ebeb981450d6ff77c8201208763a91487520d9fc7939063f7244ce842e4a5d2507d93d188527c21036df69837ccd9b8b353bdd37a8c616e3ea094ffee4c9512fec902ae8250ffb9f952ae6775022701b175ac6851b27568"
      ],
      "sequence": 1
    },
    {
      "txid": "18322ebf56f3fc168dde34dac3180a530bc4133be71a9124ad2be1426d37d51c",
      "vout": 0,
      "scriptSig": {
        "asm": "",
        "hex": ""
      },
      "sequence": 4294967293
    }
  ],
  "vout": [
    {
      "value": 0.00002235,
      "n": 0,
      "scriptPubKey": {
        "asm": "0 9357506c5a4a2df7c685481aad0302a3ccfe44c4e41122e33818861a37e2e6cc",
        "hex": "00209357506c5a4a2df7c685481aad0302a3ccfe44c4e41122e33818861a37e2e6cc",
        "reqSigs": 1,
        "type": "witness_v0_scripthash",
        "addresses": [
          "bcrt1qjdt4qmz6fgkl0359fqd26qcz50x0u3xyusgj9cecrzrp5dlzumxq6sxuyz"
        ]
      }
    },
    {
      "value": 0.48954317,
      "n": 1,
      "scriptPubKey": {
        "asm": "0 87c7624537496328484f590d04a805f0c2a213e3",
        "hex": "001487c7624537496328484f590d04a805f0c2a213e3",
        "reqSigs": 1,
        "type": "witness_v0_keyhash",
        "addresses": [
          "bcrt1qslrky3fhf93jsjz0tyxsf2q97rp2yylr3fd72n"
        ]
      }
    }
  ]
}

Notice that the first input (the non-wallet one) has a stack with 5 elements (the first of which is empty). It is correctly signed and passes script validation.

When I send this transaction to signrawtransactionwithwallet, bitcoind returns the following error: Unable to sign input, invalid stack size (possibly missing key) (code: -1) referencing the non-wallet input.

But it does correctly sign the wallet input and returns the following transaction:

020000000001027697c418d3c56138f1ea0469026d081705290bd0317ad9a133319fa7b4ac59b10200000000010000001cd5376d42e12bad24911ae73b13c40b530a18c3da34de8d16fcf356bf2e32180000000000fdffffff02bb080000000000002200209357506c5a4a2df7c685481aad0302a3ccfe44c4e41122e33818861a37e2e6cccdfbea020000000016001487c7624537496328484f590d04a805f0c2a213e3018d76a914c333ee02e6633f9bce0281ae0dc09e29b1b2761f8763ac672103bcdf028d4e53f7fbddb817091a60118b1d2fb7ab8cbab4b14ebeb981450d6ff77c8201208763a91487520d9fc7939063f7244ce842e4a5d2507d93d188527c21036df69837ccd9b8b353bdd37a8c616e3ea094ffee4c9512fec902ae8250ffb9f952ae6775022701b175ac6851b275680247304402201566b009b4e392ca16d0e5658e7fd2257e511e8974f198fb8ac265473d492949022001c9b41d261f8eacc19c41f179aa36a449200fe66abb8f3aab45c5d49e280f2901210271f8ed45eddce4cb9f990998f1d28f91fe4f747a7d67900b5382484276ae4bfd00000000
{
  "txid": "d922b3e02d77e7c522fcf46a308996df764dc61ef708a5702dc3dce6c2f58882",
  "hash": "06722d22306472337f1229ad9642c3530f161b7ef3d54e061acf7ca09fb8ed76",
  "version": 2,
  "size": 418,
  "vsize": 229,
  "weight": 916,
  "locktime": 0,
  "vin": [
    {
      "txid": "b159acb4a79f3133a1d97a31d00b290517086d026904eaf13861c5d318c49776",
      "vout": 2,
      "scriptSig": {
        "asm": "",
        "hex": ""
      },
      "txinwitness": [
        "76a914c333ee02e6633f9bce0281ae0dc09e29b1b2761f8763ac672103bcdf028d4e53f7fbddb817091a60118b1d2fb7ab8cbab4b14ebeb981450d6ff77c8201208763a91487520d9fc7939063f7244ce842e4a5d2507d93d188527c21036df69837ccd9b8b353bdd37a8c616e3ea094ffee4c9512fec902ae8250ffb9f952ae6775022701b175ac6851b27568"
      ],
      "sequence": 1
    },
    {
      "txid": "18322ebf56f3fc168dde34dac3180a530bc4133be71a9124ad2be1426d37d51c",
      "vout": 0,
      "scriptSig": {
        "asm": "",
        "hex": ""
      },
      "txinwitness": [
        "304402201566b009b4e392ca16d0e5658e7fd2257e511e8974f198fb8ac265473d492949022001c9b41d261f8eacc19c41f179aa36a449200fe66abb8f3aab45c5d49e280f2901",
        "0271f8ed45eddce4cb9f990998f1d28f91fe4f747a7d67900b5382484276ae4bfd"
      ],
      "sequence": 4294967293
    }
  ],
  "vout": [
    {
      "value": 0.00002235,
      "n": 0,
      "scriptPubKey": {
        "asm": "0 9357506c5a4a2df7c685481aad0302a3ccfe44c4e41122e33818861a37e2e6cc",
        "hex": "00209357506c5a4a2df7c685481aad0302a3ccfe44c4e41122e33818861a37e2e6cc",
        "reqSigs": 1,
        "type": "witness_v0_scripthash",
        "addresses": [
          "bcrt1qjdt4qmz6fgkl0359fqd26qcz50x0u3xyusgj9cecrzrp5dlzumxq6sxuyz"
        ]
      }
    },
    {
      "value": 0.48954317,
      "n": 1,
      "scriptPubKey": {
        "asm": "0 87c7624537496328484f590d04a805f0c2a213e3",
        "hex": "001487c7624537496328484f590d04a805f0c2a213e3",
        "reqSigs": 1,
        "type": "witness_v0_keyhash",
        "addresses": [
          "bcrt1qslrky3fhf93jsjz0tyxsf2q97rp2yylr3fd72n"
        ]
      }
    }
  ]
}

Notice how it drops all witness stack elements from the non-wallet input except the last one: this seems to be the culprit. If I ignore bitcoind's error and restore the witness stack of the non-wallet input, the transaction becomes valid and can be correctly published.

I tested this with Bitcoin Core 0.20.1 and 0.21.0, let me know if you want me to test against other versions.

The signing should be ignoring fully signed inputs. Can you provide a sample that I can check with testnet? I would like to run this with gdb to see what it is doing.

Sure, here are the same kind of transactions made on testnet:

• this one is what I send to signrawtransaction (the non-wallet input is signed): 02000000000102519769cd1236d254a359be0b7e6346249964e7c924652c53d88900ce7f10e3ba02000000000100000032567cc9550e68c2e781231af49f3b72671e72f31101d0c212233d845525e07f0100000000fdffffff027e74000000000000220020744423a7acd7005ef45f8c96931d1b0477d2892cb5efaa12a8a6a3458fdf171d8069110000000000160014b4189c012a436718ca20aa2133d4d1cd9be65a2a050047304402200aa02bbe36e751a8edd67e0980a79c93bdb94a1a9032e7f2317e1570912402780220180be3d8b2312e2ad77069fc44994ad3ef143fdeefad377699e4c4bee6e58cea83483045022100d2dc1e98630f42dc9ac1908b5db9031a2da4af46a6bc46ca44ea4c1c4e947afa02206bdb9b04ae671305e6708fbef81f0120c41c274bb71508f11001418c3f8ca4870120aab09210ace4b6134b01aa03e755cdb8610cf8c60a513124e535984d66c3f6668e76a914d0953f591363f638bec2acddd6325aff8c98956c8763ac67210219d34adfe9d4d41c4a1234ed2fda35f9e7d51c002b4c3e99a03a3ee4fa467b437c8201208763a9143b6649e2e706814918c700b54a9b34df495662b588527c2102a01bf50ab395f125b3a0c4d26e3dc4d4b97c0f1044f91a9362e21b86651feb2652ae67750303891db175ac6851b275680000000000
• this is what signrawtransaction returns (along with the error and complete=false, note that it modified the witness stack of my non-wallet input): 02000000000102519769cd1236d254a359be0b7e6346249964e7c924652c53d88900ce7f10e3ba02000000000100000032567cc9550e68c2e781231af49f3b72671e72f31101d0c212233d845525e07f0100000000fdffffff027e74000000000000220020744423a7acd7005ef45f8c96931d1b0477d2892cb5efaa12a8a6a3458fdf171d8069110000000000160014b4189c012a436718ca20aa2133d4d1cd9be65a2a018e76a914d0953f591363f638bec2acddd6325aff8c98956c8763ac67210219d34adfe9d4d41c4a1234ed2fda35f9e7d51c002b4c3e99a03a3ee4fa467b437c8201208763a9143b6649e2e706814918c700b54a9b34df495662b588527c2102a01bf50ab395f125b3a0c4d26e3dc4d4b97c0f1044f91a9362e21b86651feb2652ae67750303891db175ac6851b275680247304402206e637a522dd98b58952caa0b6d41a949daf6787d8d23250a4ee9caf3498bf14002205c16a5c26032b85eb47de8fb64d6a2ae176b681e4da1c431fb116c3563cb9a0b0121028f99fc31a955bdc85fdb90410f4e930e2c2a4dea400da6f9993b717b5db20e6c00000000
• when I then restore the correct witness stack, the transaction passes testmempoolaccept: 02000000000102519769cd1236d254a359be0b7e6346249964e7c924652c53d88900ce7f10e3ba02000000000100000032567cc9550e68c2e781231af49f3b72671e72f31101d0c212233d845525e07f0100000000fdffffff027e74000000000000220020744423a7acd7005ef45f8c96931d1b0477d2892cb5efaa12a8a6a3458fdf171d8069110000000000160014b4189c012a436718ca20aa2133d4d1cd9be65a2a050047304402200aa02bbe36e751a8edd67e0980a79c93bdb94a1a9032e7f2317e1570912402780220180be3d8b2312e2ad77069fc44994ad3ef143fdeefad377699e4c4bee6e58cea83483045022100d2dc1e98630f42dc9ac1908b5db9031a2da4af46a6bc46ca44ea4c1c4e947afa02206bdb9b04ae671305e6708fbef81f0120c41c274bb71508f11001418c3f8ca4870120aab09210ace4b6134b01aa03e755cdb8610cf8c60a513124e535984d66c3f6668e76a914d0953f591363f638bec2acddd6325aff8c98956c8763ac67210219d34adfe9d4d41c4a1234ed2fda35f9e7d51c002b4c3e99a03a3ee4fa467b437c8201208763a9143b6649e2e706814918c700b54a9b34df495662b588527c2102a01bf50ab395f125b3a0c4d26e3dc4d4b97c0f1044f91a9362e21b86651feb2652ae67750303891db175ac6851b275680247304402206e637a522dd98b58952caa0b6d41a949daf6787d8d23250a4ee9caf3498bf14002205c16a5c26032b85eb47de8fb64d6a2ae176b681e4da1c431fb116c3563cb9a0b0121028f99fc31a955bdc85fdb90410f4e930e2c2a4dea400da6f9993b717b5db20e6c00000000

I made sure the non-wallet input stays unspent, so you should be able to test it on vanilla testnet without any issues. Let me know if I can help with anything.

@t-bast Can you try #21166

Thanks @achow101! I tested #21166 for both HTLC-success and HTLC-timeout txs, and it works as we'd expect: complete is now correctly set to true, wallet inputs are signed and the non-wallet input is untouched.