Wallet Missing Balances/Unspent

TheBlueMatt commented at 5:40 pm on November 5, 2023: contributor

Is there an existing issue for this?

I have searched the existing issues

Current behaviour

Wallet is missing the balance for txid d49589ccd8488c56fbf12aee2652f760e1e84e6da1edd3dcf98bf6d556151396. The output address (and input address) are mine:

 0 getaddressinfo bc1qag6gtevhvuhafph4d73ajlj5vyhh7k4xff0p4f
 1{
 2  "address": "bc1qag6gtevhvuhafph4d73ajlj5vyhh7k4xff0p4f",
 3  "scriptPubKey": "0014ea3485e597672fd486f56fa3d97e54612f7f5aa6",
 4  "ismine": true,
 5  "solvable": true,
 6  "desc": "wpkh([e330d50b/0'/0'/1882']02beedde844990949a471bef29330f6a777ccb88aaecaf3491d9119f9454ffbd1f)#5tasvy0h",
 7  "iswatchonly": false,
 8  "isscript": false,
 9  "iswitness": true,
10  "witness_version": 0,
11  "witness_program": "ea3485e597672fd486f56fa3d97e54612f7f5aa6",
12  "pubkey": "02beedde844990949a471bef29330f6a777ccb88aaecaf3491d9119f9454ffbd1f",
13  "ischange": false,
14  "timestamp": 1634944153,
15  "hdkeypath": "m/0'/0'/1882'",
16  "hdseedid": "bf8e10d1998ca0f7a73b372d0ab4d9ddfa6aa4c6",
17  "hdmasterfingerprint": "e330d50b",
18  "labels": [
19    "LDK output address"
20  ]
21}
22getaddressinfo bc1qqdaevqjz84r0m8wxejugvcnjreuuu0xz0wwxym
23{
24  "address": "bc1qqdaevqjz84r0m8wxejugvcnjreuuu0xz0wwxym",
25  "scriptPubKey": "0014037b9602423d46fd9dc6ccb88662721e79ce3cc2",
26  "ismine": true,
27  "solvable": true,
28  "desc": "wpkh([e330d50b/0'/0'/1822']03e243567081536017cb11508b7fb53ccdde657bcc5df10b9405f4cb9aa3fa6858)#48670jcm",
29  "iswatchonly": false,
30  "isscript": false,
31  "iswitness": true,
32  "witness_version": 0,
33  "witness_program": "037b9602423d46fd9dc6ccb88662721e79ce3cc2",
34  "pubkey": "03e243567081536017cb11508b7fb53ccdde657bcc5df10b9405f4cb9aa3fa6858",
35  "ischange": false,
36  "timestamp": 1634856115,
37  "hdkeypath": "m/0'/0'/1822'",
38  "hdseedid": "bf8e10d1998ca0f7a73b372d0ab4d9ddfa6aa4c6",
39  "hdmasterfingerprint": "e330d50b",
40  "labels": [
41    "LDK output address"
42  ]
43}

listtransactions lists it with 200+ confirmations

 0  {
 1    "address": "bc1qag6gtevhvuhafph4d73ajlj5vyhh7k4xff0p4f",
 2    "parent_descs": [
 3    ],
 4    "category": "receive",
 5    "amount": 0.07856602,
 6    "label": "LDK output address",
 7    "vout": 0,
 8    "confirmations": 257,
 9    "blockhash": "00000000000000000000a1c320c11959e670e2fb26ca0d5df64a2c453b4105d2",
10    "blockheight": 815214,
11    "blockindex": 41,
12    "blocktime": 1699074707,
13    "txid": "d49589ccd8488c56fbf12aee2652f760e1e84e6da1edd3dcf98bf6d556151396",
14    "wtxid": "137bf6d9142d8a3a5a554f86298c43b635c40d8e4486a0d92a320d03c7160708",
15    "walletconflicts": [
16      "1bd8d162be5a4b91c083866386a9e6ca0201f2c837241350855a61a7d4a9f78c"
17    ],
18    "time": 1699074653,
19    "timereceived": 1699074653,
20    "bip125-replaceable": "no"
21  },
22  {
23    "address": "bc1qag6gtevhvuhafph4d73ajlj5vyhh7k4xff0p4f",
24    "category": "send",
25    "amount": -0.07856602,
26    "label": "LDK output address",
27    "vout": 0,
28    "fee": -0.00019383,
29    "confirmations": 257,
30    "blockhash": "00000000000000000000a1c320c11959e670e2fb26ca0d5df64a2c453b4105d2",
31    "blockheight": 815214,
32    "blockindex": 41,
33    "blocktime": 1699074707,
34    "txid": "d49589ccd8488c56fbf12aee2652f760e1e84e6da1edd3dcf98bf6d556151396",
35    "wtxid": "137bf6d9142d8a3a5a554f86298c43b635c40d8e4486a0d92a320d03c7160708",
36    "walletconflicts": [
37      "1bd8d162be5a4b91c083866386a9e6ca0201f2c837241350855a61a7d4a9f78c"
38    ],
39    "time": 1699074653,
40    "timereceived": 1699074653,
41    "bip125-replaceable": "no",
42    "abandoned": false
43  },

but its not in listunspent and its missing from balances

0$ listunspent 0 | grep d49589ccd8488c56fbf12aee2652f760e1e84e6da1edd3dcf98bf6d556151396
1$  getbalance "*" 0
20.00000000

Expected behaviour

My money is listed as spendable

Steps to reproduce

Probably import those keys as watchonly and see what happens

Relevant log output

No response

How did you obtain Bitcoin Core

Pre-built binaries

What version of Bitcoin Core are you using?

25.1, also happened on 23.0 which i was using until i upgraded to try to fix this

Operating system and version

Linux

Machine specifications

No response

TheBlueMatt commented at 4:26 pm on November 6, 2023: contributor

 0getwalletinfo
 1{
 2  "walletname": "",
 3  "walletversion": 169900,
 4  "format": "bdb",
 5...
 6  "keypoolsize": 1000,
 7  "keypoolsize_hd_internal": 1000,
 8  "paytxfee": 0.00000000,
 9  "private_keys_enabled": true,
10  "avoid_reuse": false,
11  "scanning": false,
12  "descriptors": false,
13  "external_signer": false
14}

1440000bytes commented at 4:31 pm on November 6, 2023: none

"walletconflicts": [
 "1bd8d162be5a4b91c083866386a9e6ca0201f2c837241350855a61a7d4a9f78c"

Maybe this is the reason for incorrect balance?

TheBlueMatt commented at 6:30 pm on November 6, 2023: contributor

Probably, but just because a transaction was RBF-bumped there shouldn’t be an incorrect balance/missing unspent entry :)

fanquake commented at 7:03 pm on November 6, 2023: member

cc @achow101

achow101 commented at 7:40 pm on November 6, 2023: member

I think the only conditions for this to happen are either the output has been spent by something in the wallet that has not been broadcast yet, or if the wallet thinks that those outputs are not ismine. The latter shouldn’t be possible given that getaddressinfo reports both addresses as ismine. Can you check if there is some transaction in your wallet that is spending that output but wasn’t broadcast?

It could be that the output is locked with lockunspent, but that should only affect listunspent and not getbalance.

Also, I don’t think it’s possible to replicate this without having the private keys or even the wallet itself. Watchonly is a different set of conditions.

Can you provide more details on how this wallet was constructed, e.g. were things imported?

If you import the privkeys into a new wallet, does it exhibit the same problem?

TheBlueMatt commented at 10:37 pm on November 6, 2023: contributor

Nothing has ever been imported into this wallet, its pretty much bog-standard, except that transactions are built with the raw transaction interface (eg the linked transaction, which spends my money back to myself and adds another input, in this case a lightning anchor). Nothing touching this wallet uses lockunspent.

TheBlueMatt commented at 10:42 pm on November 6, 2023: contributor

There doesn’t appear to be anything in-wallet that is spending that output:

bitcoin-cli listtransactions "*" 10000000 | jq '.[] | select(.confirmations <= 0) | .txid' | xargs -L1 bitcoin-cli gettransaction | grep bc1qag6gtevhvuhafph4d73ajlj5vyhh7k4xff0p4f shows nothing and the non-confirmations-filtered version only shows d49589ccd8488c56fbf12aee2652f760e1e84e6da1edd3dcf98bf6d556151396 twice.

TheBlueMatt commented at 11:32 pm on November 6, 2023: contributor

Ah, okay, so what happened is two bugs and one need-package-relay issue:

The above snippet didn’t work because of the bug you can see in the above listtransactions output: if a transaction spends from an address of ours to an address of ours its (correctly) listed twice, but both times with the output address, neither time with the input address, even though one listing is “send”. That’s a bug.

This did turn up a spending tx, but that tx was in the broader mempool because its parent didn’t have sufficient feerate to enter common mempools (the spending tx is a CPFP fee-bump of an unrelated transaction): listtransactions "*" 10000000 | jq '.[] | select(.confirmations <= 0) | .txid' | xargs -L1 ~/search-tx.sh | grep -C50 d49589ccd8488c56fbf12aee2652f760e1e84e6da1edd3dcf98bf6d556151396

0cat ~/search-tx.sh
1bitcoin-cli "$1" | jq '.hex' | tr -d '"' | bitcoin-cli decoderawtransaction

So, were there package relay it would have broadcasted.

Further, the spending tx here is a send-from-self-to-self, which should be listed in balance/listunspent when I ask for 0-conf as the limit, AFAIU, or at least should have an option to, which is a second bug.

achow101 commented at 11:44 pm on November 6, 2023: member

The above snippet didn’t work because of the bug you can see in the above listtransactions output: if a transaction spends from an address of ours to an address of ours its (correctly) listed twice, but both times with the output address, neither time with the input address, even though one listing is “send”. That’s a bug.

It doesn’t show input addresses since they don’t exist (or something like that). However we do have the spent-by info so each “transaction” in listtransactions could show that.

Further, the spending tx here is a send-from-self-to-self, which should be listed in balance/listunspent when I ask for 0-conf as the limit, AFAIU, or at least should have an option to, which is a second bug.

It only includes the utxo in the balance and in listunspent if it is available for spending. A transaction that is not in the mempool should not have any of its outputs available for spending since any attempt to spend them (currently) would result in a transaction that probably cannot be broadcast. I’m pretty sure there are several ancient issues describing this and I believe the conclusion is generally to err on the safe side of telling the user that they don’t have the money.

I think it will show up if you use getbalances though?

TheBlueMatt commented at 11:55 pm on November 6, 2023: contributor

It doesn’t show input addresses since they don’t exist (or something like that). However we do have the spent-by info so each “transaction” in listtransactions could show that.

Right, then the “address” field should be skipped. A “send” entry is referring to the fact that we spent some of our money, and generally we don’t list an address, the address inclusion in a “send” entry that’s unrelated to the “send” part of the transaction is…very confusing :)

That said, given such an entry is specifically related a an input which spends our own address, I think its safe to say there’s a “from” address when we’re talking about a specific input in a greater transaction :).

It only includes the utxo in the balance and in listunspent if it is available for spending. A transaction that is not in the mempool should not have any of its outputs available for spending since any attempt to spend them (currently) would result in a transaction that probably cannot be broadcast. I’m pretty sure there are several ancient issues describing this and I believe the conclusion is generally to err on the safe side of telling the user that they don’t have the money.

~To be clear it was in the local node’s mempool, it however had not propagated. Even now with it pushed to a neighboring node the balance is still listed zero.~ Oh, that’s a lie, I was looking in the wrong place. Indeed, not a bug then, though with modern technologies like RBF and CPFP we may want to reconsider the state of things here.

willcl-ark added the label Wallet on Jan 24, 2024

Wallet Missing Balances/Unspent #28797