Empty Witness Data, although transaction has witness populated in other full-nodes

ziggie1984 commented at 2:16 pm on October 25, 2023: none

Is there an existing issue for this?

I have searched the existing issues

Current behaviour

Writing this issue on behalf of several reported cases (see below), so the exact bitcoind specifications are different in each case.

All lightning implementation scan for the preimage in case an HTLC as to be resolved onchain. Doing this we analyse the witness data of related outputs because we know what to expect. Example for this is an offered HTLC we have to resolve onchain. Let’s imagine that we locally force-closed a channel, this would lead to the following Witnessscript for the offered HTLC:

 0# To remote node with revocation key
 1OP_DUP OP_HASH160 <RIPEMD160(SHA256(revocationpubkey))> OP_EQUAL
 2OP_IF
 3    OP_CHECKSIG
 4OP_ELSE
 5    <remote_htlcpubkey> OP_SWAP OP_SIZE 32 OP_EQUAL
 6    OP_NOTIF
 7        # To local node via HTLC-timeout transaction (timelocked).
 8        OP_DROP 2 OP_SWAP <local_htlcpubkey> 2 OP_CHECKMULTISIG
 9    OP_ELSE
10        # To remote node with preimage.
11        OP_HASH160 <RIPEMD160(payment_hash)> OP_EQUALVERIFY
12        OP_CHECKSIG
13    OP_ENDIF
14OP_ENDIF

This Script can only be unlocked by the following 3 types of witness data:

0//  SENDR: <0> <sendr sig>  <recvr sig> <0> (2nd level timeout)
1//  RECVR: <recvr sig>  <preimage>
2//  REVOK: <revoke sig> <revoke key>

So we can conclude that the witness data will be at least of size 2. In lnd we therefore do a check for the second witness element to be a 32 byte preimage.

Now comes the problem. People reported now several times, that their lightning-node was crashing because of this behaviour. And when queried their backend with bitcoin-cli getrawtransaction their output did also not have the witness data populated. A reindexing of the blockchain helped I think in all cases which reported this behaviour. I am reporting this issue here because I have the feeling that somehow bitcoind might have an edge case for this behaviour (in case the hardware is not the problem).

Reported cases: 1.https://github.com/lightningnetwork/lnd/issues/8028#issuecomment-1778777711 2.https://github.com/lightningnetwork/lnd/issues/7803 3.https://github.com/lightningnetwork/lnd/issues/5977 4.https://github.com/lightningnetwork/lnd/issues/6381

Wondering whether some of you have an idea where to look at. Can I assume that a bitcoin node could only report this transaction (hence accepting it via a block) if the witness data was not provided? Otherwise we would have failed the block verification I guess?

Happy to get your thoughts on this.

Thank you in advance!

Expected behaviour

Not sure whether this is just a hardware failure which manifests itself via the above described behaviour. So no expected behaviour yet.

Steps to reproduce

TBD.

Relevant log output

No response

How did you obtain Bitcoin Core

Pre-built binaries

What version of Bitcoin Core are you using?

-

Operating system and version

-

Machine specifications

No response

maflcko commented at 2:43 pm on October 25, 2023: member

Can you check the config file, and the full process command for any appearance of -rpcserialversion=0 (or similar)?

Due to its problems, the upcoming release has this flag deprecated and -deprecatedrpc=serialversion is required to use it.

maflcko added the label RPC/REST/ZMQ on Oct 25, 2023

maflcko commented at 2:53 pm on October 25, 2023: member

Looking at the conf in https://github.com/lightningnetwork/lnd/issues/7803#issuecomment-1629619642 and given that a reindex fixed it here https://github.com/lightningnetwork/lnd/issues/5977#issuecomment-971090110, it seems unlikely to be caused by -rpcserialversion=0, but still possible?

ziggie1984 commented at 3:02 pm on October 25, 2023: none

Ahh ok I see what this flag does, you can tell bitcoind to only return the non-segwit serialization, meaning that getrawtransaction will never return any witness data, right ? The case I was investigating today had already several force-closes onchain (with htlcs-involved) so then it would have crashed earlier in the past but hasn’t happened before to him at least.

maflcko commented at 3:21 pm on October 25, 2023: member

Are there logs available when the transaction first entered their Bitcoin Core node (either via RPC or via P2P), either as a mempool transaction or a transaction inside a block?

ziggie1984 commented at 3:29 pm on October 25, 2023: none

Are there logs available when the transaction first entered their Bitcoin Core node (either via RPC or via P2P), either as a mempool transaction or a transaction inside a block?

can you provide us with some logs for your case (bitcoind logs) @sanjay-shah, @Megasley

maflcko commented at 3:37 pm on October 25, 2023: member

Also the results of:

0bitcoin-cli getrawtransaction 0ce6fe4890977aa4e253e0c3eb2ef3288b5b7ca028eef906c98e5223348a3535 2
1bitcoin-cli getrawtransaction 0ce6fe4890977aa4e253e0c3eb2ef3288b5b7ca028eef906c98e5223348a3535 2 00000000000000000000f155e5c79b337bf37b12fcc88c8d686653eda50e75fb
2bitcoin-cli verifychain 4 500
3bitcoin-cli gettxoutsetinfo muhash
4bitcoin-cli getblock 00000000000000000000f155e5c79b337bf37b12fcc88c8d686653eda50e75fb 2

sanjay-shah commented at 8:05 am on October 26, 2023: none

bitcoin-cli getrawtransaction 0ce6fe4890977aa4e253e0c3eb2ef3288b5b7ca028eef906c98e5223348a3535 2 { “txid”: “0ce6fe4890977aa4e253e0c3eb2ef3288b5b7ca028eef906c98e5223348a3535”, “hash”: “c620bec3925915b557887d03ade5ff02f11399d67b4e3638dabb01e75089f763”, “version”: 2, “size”: 531, “vsize”: 267, “weight”: 1065, “locktime”: 759519, “vin”: [ { “txid”: “beb2a6201db505b9f827ea37484d0467c4bdd26c3cf60c8cb76bd4bcf9352ba7”, “vout”: 2, “scriptSig”: { “asm”: “”, “hex”: "" }, “txinwitness”: [ “”, “3045022100a1fe9f370ac1fde977ec6a147d8e32f07c8b6f6548667bcc0f6cf5b6a10e4dde02202484ab1a9ec46a075c209c3f1127948e0baa26137b0501a6a51df64ba1cfebbd83”, “304402204a46f33d8fbe7e51ce462d0099caf66fa2c4a94128d7f0e5d4828de25d13b7f002205ab9a9f8d997033e500ce81bdf6ca21a07262311517266f61e6e452992e91ff401”, “”, “76a9145a9526206556cafdda02fc05c1da0aa49d2f17bb8763ac67210332bdfbea33e7be2d3d8873b144b401c02aedb13ee2211138406c4d574dc4685c7c820120876475527c210334888d0231807688c2cdb942fb61605b460789605c734539e7d749b652527bbf52ae67a91453d453c8befad357c7de27a98858165e2b7a152388ac6851b27568” ], “prevout”: { “generated”: false, “height”: 759520, “value”: 0.00031250, “scriptPubKey”: { “asm”: “0 601a5363d45524065b94ffcc9eb0ce8cec77f7a13780ece8b9b0791dafc3b504”, “desc”: “addr(bc1qvqd9xc7525jqvku5llxfavxw3nk80aapx7qwe69ekpu3mt7rk5zqlfr9u8)#ju8h9vc7”, “hex”: “0020601a5363d45524065b94ffcc9eb0ce8cec77f7a13780ece8b9b0791dafc3b504”, “address”: “bc1qvqd9xc7525jqvku5llxfavxw3nk80aapx7qwe69ekpu3mt7rk5zqlfr9u8”, “type”: “witness_v0_scripthash” } }, “sequence”: 1 }, { “txid”: “9f8826cfc5b3dce4386025fd46081f6fba95e26d1eb6e33310b1cb50771cf5f3”, “vout”: 0, “scriptSig”: { “asm”: “”, “hex”: "" }, “txinwitness”: [ “11de1eab981fc10d5efcb0543890b9b80851eeba7627226601d2a2ad9675542abad51eb8b1f22b8140eb9db9eccae9a0190e17fb9656b14b8cb7c0c86e98be85” ], “prevout”: { “generated”: false, “height”: 813735, “value”: 0.03158620, “scriptPubKey”: { “asm”: “1 2d61b7ed3448cfb9422fe62786931fdb2661c3c7f5a51de9998ffcfa68ec7dfa”, “desc”: “rawtr(2d61b7ed3448cfb9422fe62786931fdb2661c3c7f5a51de9998ffcfa68ec7dfa)#ww77p9et”, “hex”: “51202d61b7ed3448cfb9422fe62786931fdb2661c3c7f5a51de9998ffcfa68ec7dfa”, “address”: “bc1p94sm0mf5fr8mjs30ucncdyclmvnxrs787kj3m6ve3l70568v0haqpe04dr”, “type”: “witness_v1_taproot” } }, “sequence”: 0 } ], “vout”: [ { “value”: 0.00031250, “n”: 0, “scriptPubKey”: { “asm”: “0 ead078680d77a590ac3f065ce64ce8d9dbe4462b857857fe39d2f6df5ddf7c61”, “desc”: “addr(bc1qatg8s6qdw7jeptplqewwvn8gm8d7g33ts4u90l3e6tmd7hwl03ss4q7q6e)#j8ece954”, “hex”: “0020ead078680d77a590ac3f065ce64ce8d9dbe4462b857857fe39d2f6df5ddf7c61”, “address”: “bc1qatg8s6qdw7jeptplqewwvn8gm8d7g33ts4u90l3e6tmd7hwl03ss4q7q6e”, “type”: “witness_v0_scripthash” } }, { “value”: 0.03149377, “n”: 1, “scriptPubKey”: { “asm”: “1 89a3a0047f3733310a43fa77c55acd855bcec63040d090a055a28ef3377e8ace”, “desc”: “rawtr(89a3a0047f3733310a43fa77c55acd855bcec63040d090a055a28ef3377e8ace)#jrzvn05l”, “hex”: “512089a3a0047f3733310a43fa77c55acd855bcec63040d090a055a28ef3377e8ace”, “address”: “bc1p3x36qprlxuenzzjrlfmu2kkds4dua33sgrgfpgz45280xdm73t8q7qwc0f”, “type”: “witness_v1_taproot” } } ], “fee”: 0.00009243, “hex”: “0200000002a72b35f9bcd46bb78c0cf63c6cd2bdc467044d4837ea27f8b905b51d20a6b2be020000000001000000f3f51c7750cbb11033e3b61e6de295ba6f1f0846fd256038e4dcb3c5cf26889f00000000000000000002127a000000000000220020ead078680d77a590ac3f065ce64ce8d9dbe4462b857857fe39d2f6df5ddf7c61410e30000000000022512089a3a0047f3733310a43fa77c55acd855bcec63040d090a055a28ef3377e8acedf960b00”, “blockhash”: “00000000000000000000f155e5c79b337bf37b12fcc88c8d686653eda50e75fb”, “confirmations”: 147, “time”: 1698216321, “blocktime”: 1698216321 }

bitcoin-cli getrawtransaction 0ce6fe4890977aa4e253e0c3eb2ef3288b5b7ca028eef906c98e5223348a3535 2 00000000000000000000f155e5c79b337bf37b12fcc88c8d686653eda50e75fb { “in_active_chain”: true, “txid”: “0ce6fe4890977aa4e253e0c3eb2ef3288b5b7ca028eef906c98e5223348a3535”, “hash”: “c620bec3925915b557887d03ade5ff02f11399d67b4e3638dabb01e75089f763”, “version”: 2, “size”: 531, “vsize”: 267, “weight”: 1065, “locktime”: 759519, “vin”: [ { “txid”: “beb2a6201db505b9f827ea37484d0467c4bdd26c3cf60c8cb76bd4bcf9352ba7”, “vout”: 2, “scriptSig”: { “asm”: “”, “hex”: "" }, “txinwitness”: [ “”, “3045022100a1fe9f370ac1fde977ec6a147d8e32f07c8b6f6548667bcc0f6cf5b6a10e4dde02202484ab1a9ec46a075c209c3f1127948e0baa26137b0501a6a51df64ba1cfebbd83”, “304402204a46f33d8fbe7e51ce462d0099caf66fa2c4a94128d7f0e5d4828de25d13b7f002205ab9a9f8d997033e500ce81bdf6ca21a07262311517266f61e6e452992e91ff401”, “”, “76a9145a9526206556cafdda02fc05c1da0aa49d2f17bb8763ac67210332bdfbea33e7be2d3d8873b144b401c02aedb13ee2211138406c4d574dc4685c7c820120876475527c210334888d0231807688c2cdb942fb61605b460789605c734539e7d749b652527bbf52ae67a91453d453c8befad357c7de27a98858165e2b7a152388ac6851b27568” ], “prevout”: { “generated”: false, “height”: 759520, “value”: 0.00031250, “scriptPubKey”: { “asm”: “0 601a5363d45524065b94ffcc9eb0ce8cec77f7a13780ece8b9b0791dafc3b504”, “desc”: “addr(bc1qvqd9xc7525jqvku5llxfavxw3nk80aapx7qwe69ekpu3mt7rk5zqlfr9u8)#ju8h9vc7”, “hex”: “0020601a5363d45524065b94ffcc9eb0ce8cec77f7a13780ece8b9b0791dafc3b504”, “address”: “bc1qvqd9xc7525jqvku5llxfavxw3nk80aapx7qwe69ekpu3mt7rk5zqlfr9u8”, “type”: “witness_v0_scripthash” } }, “sequence”: 1 }, { “txid”: “9f8826cfc5b3dce4386025fd46081f6fba95e26d1eb6e33310b1cb50771cf5f3”, “vout”: 0, “scriptSig”: { “asm”: “”, “hex”: "" }, “txinwitness”: [ “11de1eab981fc10d5efcb0543890b9b80851eeba7627226601d2a2ad9675542abad51eb8b1f22b8140eb9db9eccae9a0190e17fb9656b14b8cb7c0c86e98be85” ], “prevout”: { “generated”: false, “height”: 813735, “value”: 0.03158620, “scriptPubKey”: { “asm”: “1 2d61b7ed3448cfb9422fe62786931fdb2661c3c7f5a51de9998ffcfa68ec7dfa”, “desc”: “rawtr(2d61b7ed3448cfb9422fe62786931fdb2661c3c7f5a51de9998ffcfa68ec7dfa)#ww77p9et”, “hex”: “51202d61b7ed3448cfb9422fe62786931fdb2661c3c7f5a51de9998ffcfa68ec7dfa”, “address”: “bc1p94sm0mf5fr8mjs30ucncdyclmvnxrs787kj3m6ve3l70568v0haqpe04dr”, “type”: “witness_v1_taproot” } }, “sequence”: 0 } ], “vout”: [ { “value”: 0.00031250, “n”: 0, “scriptPubKey”: { “asm”: “0 ead078680d77a590ac3f065ce64ce8d9dbe4462b857857fe39d2f6df5ddf7c61”, “desc”: “addr(bc1qatg8s6qdw7jeptplqewwvn8gm8d7g33ts4u90l3e6tmd7hwl03ss4q7q6e)#j8ece954”, “hex”: “0020ead078680d77a590ac3f065ce64ce8d9dbe4462b857857fe39d2f6df5ddf7c61”, “address”: “bc1qatg8s6qdw7jeptplqewwvn8gm8d7g33ts4u90l3e6tmd7hwl03ss4q7q6e”, “type”: “witness_v0_scripthash” } }, { “value”: 0.03149377, “n”: 1, “scriptPubKey”: { “asm”: “1 89a3a0047f3733310a43fa77c55acd855bcec63040d090a055a28ef3377e8ace”, “desc”: “rawtr(89a3a0047f3733310a43fa77c55acd855bcec63040d090a055a28ef3377e8ace)#jrzvn05l”, “hex”: “512089a3a0047f3733310a43fa77c55acd855bcec63040d090a055a28ef3377e8ace”, “address”: “bc1p3x36qprlxuenzzjrlfmu2kkds4dua33sgrgfpgz45280xdm73t8q7qwc0f”, “type”: “witness_v1_taproot” } } ], “fee”: 0.00009243, “hex”: “0200000002a72b35f9bcd46bb78c0cf63c6cd2bdc467044d4837ea27f8b905b51d20a6b2be020000000001000000f3f51c7750cbb11033e3b61e6de295ba6f1f0846fd256038e4dcb3c5cf26889f00000000000000000002127a000000000000220020ead078680d77a590ac3f065ce64ce8d9dbe4462b857857fe39d2f6df5ddf7c61410e30000000000022512089a3a0047f3733310a43fa77c55acd855bcec63040d090a055a28ef3377e8acedf960b00”, “blockhash”: “00000000000000000000f155e5c79b337bf37b12fcc88c8d686653eda50e75fb”, “confirmations”: 147, “time”: 1698216321, “blocktime”: 1698216321 }

bitcoin-cli verifychain 4 500 false

on the debug.log I see the following error:

2023-10-26T07:26:16Z Verifying last 500 blocks at level 4 2023-10-26T07:26:16Z Verification progress: 0% 2023-10-26T07:26:55Z Verification progress: 10% 2023-10-26T07:28:03Z Verification progress: 20% 2023-10-26T07:28:09Z Verification progress: 30% 2023-10-26T07:28:16Z Verification progress: 40% 2023-10-26T07:28:23Z Verification progress: 50% 2023-10-26T07:28:23Z Skipped verification of level >=3 (insufficient database cache size). Consider increasing -dbcache. 2023-10-26T07:28:23Z Verification: No coin database inconsistencies in last 500 blocks (379717 transactions)

bitcoin-cli gettxoutsetinfo muhash error: timeout on transient error: Could not connect to the server 127.0.0.1:8332 (error code 0 - “timeout reached”)

Make sure the bitcoind server is running and that you are connecting to the correct RPC port.

bitcoin-cli getblock 00000000000000000000f155e5c79b337bf37b12fcc88c8d686653eda50e75fb 2 getblock.json

sanjay-shah commented at 8:09 am on October 26, 2023: none

Looking at the conf in lightningnetwork/lnd#7803 (comment) and given that a reindex fixed it here lightningnetwork/lnd#5977 (comment), it seems unlikely to be caused by -rpcserialversion=0, but still possible? @maflcko I do have this set: -rpcserialversion=0 in my conf.

cat .bitcoin/bitcoin.conf server=1 rpcuser=xxxxxxxxx rpcpassword=xxxxxxxxxx txindex=1 listen=1 rpcserialversion=0 maxorphantx=1 banscore=1 bind=0.0.0.0:8333 rpcallowip=143.198.140.93 rpcallowip=10.124.0.2 rpcallowip=10.48.0.5 rpcallowip=127.0.0.1 rpcbind=0.0.0.0 rpcport=8332

maflcko commented at 8:13 am on October 26, 2023: member

I do have this set: -rpcserialversion=0

I wonder if there is an easy way for lnd to protect against this. For example, by asking for a transaction with witness. I guess not, given that the user may have pruning enabled, no txindex enabled, and the mempool may be empty of transactions with witness.

ziggie1984 commented at 8:24 am on October 26, 2023: none

I am wondering whether my bitcoin-core node will accept tx relays from non-segwit enabled nodes into my mempool (because they cannot supply the relevant witness data?)

maflcko commented at 8:27 am on October 26, 2023: member

Those nodes can only provide you with transactions whose witness and non-witness serialization is identical, that is, transactions that do not have any witness data.

ziggie1984 commented at 8:31 am on October 26, 2023: none

Those nodes can only provide you with transactions whose witness and non-witness serialization is identical, that is, transactions that do not have any witness data.

But how do non-witness enabled nodes know this, because they have only the non-witness serialized tx in their store? So I think they will try to relay the tx but we(as witness enabled node) would reject it ?

EDIT: I am asking because when it is possible that a full-node can have transaction data in its mempool received from non-segwit enabled notes (transactions without witness data), then we definitely need to make sure on the lnd side we do not shutdown the daemon when we encounter this case.

maflcko commented at 9:29 am on October 26, 2023: member

@sanjay-shah Was there a reason why you set -rpcserialversion=0? Was this generated automatically by a config generator?

maflcko commented at 9:41 am on October 26, 2023: member

EDIT: I am asking because when it is possible that a full-node can have transaction data in its mempool received from non-segwit enabled notes (transactions without witness data), then we definitely need to make sure on the lnd side we do not shutdown the daemon when we encounter this case.

I can’t help you with questions about lnd, but if your question is whether it is possible to have an HTLC (via P2WSH) transaction in the mempool without witness, then the answer is no, because that’d be rejected by validation by all supported versions of Bitcoin Core.

If your question is whether it is possible to have a valid transaction without witness data in the mempool, the the answer is yes.

ziggie1984 commented at 9:46 am on October 26, 2023: none

I can’t help you with questions about lnd, but if your question is whether it is possible to have an HTLC (via P2WSH) transaction in the mempool without witness, then the answer is no, because that’d be rejected by validation by all supported versions of Bitcoin Core.

Perfect, that was exactly my question (having a transaction which in reality is a segwit transaction but received from a non-segwit enabled fullnode in the mempool) . Wondering because we are talking about the mempool here, whether this validation can be considered as Policy or Consensus ?

sanjay-shah commented at 9:58 am on October 26, 2023: none

@sanjay-shah Was there a reason why you set -rpcserialversion=0? Was this generated automatically by a config generator? @maflcko I really don’t remember, this was very old node that I kept upgrading, maybe I generated the config using lopp’s generator, or the config got inherited from very old version of bitcoind where I might have needed it in the past.

maflcko commented at 10:48 am on October 26, 2023: member

Perfect, that was exactly my question (having a transaction which in reality is a segwit transaction but received from a non-segwit enabled fullnode in the mempool) . Wondering because we are talking about the mempool here, whether this validation can be considered as Policy or Consensus ?

It doesn’t matter where you get the transaction from. Either the transaction is added to the mempool, in which case it is valid (consensus valid and policy valid). Or the transaction is rejected from the mempool, in which case you can’t ask for the transaction over RPC anyway, so it shouldn’t matter for the purposes of this issue.

ziggie1984 commented at 11:28 am on October 26, 2023: none

Ok I also found the problem to https://github.com/lightningnetwork/lnd/issues/7803

This node was running in pruned node, what lnd does when not finding the block locally it asks its peers via the iventory messages to provide the data. The old version of the library used: [InvTypeBlock] (https://github.com/btcsuite/btcwallet/blob/f33c42289bafd785ff9516fc651ff84a55374ba9/chain/pruned_block_dispatcher.go#L546) which according to the description only responds with the non-witness block data (https://github.com/btcsuite/btcd/blob/master/wire/invvect.go#L32-L38 - maybe you can verify whether bitcoin-core really strips the data when set)

In the newer version of the library we switched to InvTypeWitnessBlock but this will only be available in the next lnd release (18).

I will close this issue, because bitcoin-core is not the source of the problem ok. Thanks a lot for your help :) @maflcko

ziggie1984 closed this on Oct 26, 2023

maflcko commented at 11:35 am on October 26, 2023: member

Good to hear this issue is resolved. It is also good to have another data point to justify the removal of the confusing and brittle -rpcserialversion setting.

Empty Witness Data, although transaction has witness populated in other full-nodes #28730