listreceivedbyaddress is empty for descriptor (but not legacy) wallets

vibs29 commented at 5:13 pm on January 4, 2023: none

https://bitcoincore.org/bin/bitcoin-core-24.0.1/bitcoin-24.0.1-x86_64-linux-gnu.tar.gz

Problem: listreceivedbyaddress always returns an empty array for a watch-only wallet when it’s of type descriptor (not legacy). This problem doesn’t occur with a legacy watch-only wallet, only a descriptor watch-only wallet.

Apparent reason: bitcoind thinks the addresses are for change.

Work-around: When importing descriptors, set active=true.

Thanks to achow101 on IRC Libera Chat #bitcoin-core-dev, 2023-01-04.

 0$ cat $HOME/data1/bitcoin.conf 
 1printtoconsole=1
 2regtest=1
 3txindex=1
 4[regtest]
 5port=18444
 6rpcport=18443
 7
 8$ $HOME/bitcoin-24.0.1/bin/bitcoin-qt -datadir=$HOME/data1 -server -fallbackfee=0.00001 &
 9
10$ cat bcli.sh 
11#!/bin/bash
12$HOME/bitcoin-24.0.1/bin/bitcoin-cli -datadir=$HOME/data1 "$@"
13
14
15$ bcli.sh createwallet w1
16$ bcli.sh createwallet w2 true true
17$ bcli.sh -rpcwallet=w1 listdescriptors
18...
19    {
20      "desc": "wpkh([577ce3ca/84'/1'/0']tpubDCNwCevBBSLGZm5d2urnpLDD2B4s5dWf7TRigFngte9M7sijNQg6QVjq1K7Nb7uLAP7xLn2h4X1MoiT3JmMP3sQYuKsY1qnPj1HR63hQDuT/0/*)#cw6xz6mx",
21      "timestamp": 1672848018,
22      "active": true,
23      "internal": false,
24      "range": [
25        0,
26        999
27      ],
28      "next": 0
29    },
30...
31
32$ bcli.sh -rpcwallet=w2 importdescriptors "[{\"desc\": \"wpkh([577ce3ca/84'/1'/0']tpubDCNwCevBBSLGZm5d2urnpLDD2B4s5dWf7TRigFngte9M7sijNQg6QVjq1K7Nb7uLAP7xLn2h4X1MoiT3JmMP3sQYuKsY1qnPj1HR63hQDuT/0/*)#cw6xz6mx\", \"timestamp\": 0, \"range\": 999}]"
33
34$ bcli.sh -rpcwallet=w1 getnewaddress
35bcrt1qxu3fl6lyqydvtgxsdpeu69gyywy2dya5a4kray
36
37$ bcli.sh -rpcwallet=w1 generatetoaddress 101 bcrt1qxu3fl6lyqydvtgxsdpeu69gyywy2dya5a4kray
38
39$ bcli.sh -rpcwallet=w2 listreceivedbyaddress 1 true true "" true
40[
41]

vibs29 added the label Bug on Jan 4, 2023

w0xlt commented at 2:29 pm on January 5, 2023: contributor

I think the reason for this problem is: listreceivedbyaddress iterates through CWallet::m_address_book to get the items.

But an address is only added to m_address_book when it is generated via getnewaddress. Since this command has been run in the hot wallet, the watch-only has no way of knowing that this has been added to the CWallet::m_address_book.

But you reported it is working in a legacy watch-only wallet. How are you creating it ?

w0xlt commented at 2:32 pm on January 5, 2023: contributor

When loading the wallets, you can see what I described above.

02023-01-05T14:20:56Z [w1] Wallet completed loading in             528ms
12023-01-05T14:20:56Z [w1] setKeyPool.size() = 8000
22023-01-05T14:20:56Z [w1] mapWallet.size() = 1
32023-01-05T14:20:56Z [w1] m_address_book.size() = 1
4...
52023-01-05T14:20:58Z [w2] Wallet completed loading in              84ms
62023-01-05T14:20:58Z [w2] setKeyPool.size() = 0
72023-01-05T14:20:58Z [w2] mapWallet.size() = 1
82023-01-05T14:20:58Z [w2] m_address_book.size() = 0

Note the difference between m_address_book.size() in [w1] and [w2].

furszy commented at 3:10 pm on January 5, 2023: member

hmm yeah, it’s a deeper problem. This make me remember to #25979, where I started heading towards a general solution for the change detection but.. it’s still not ready (atm detects change from active descriptors only; IIRC, I should be able to expand it to non-active ones based on their derivation paths pretty easily, which will solve this issue. But still need to re-review the detection when a descriptor get removed – which is basically achow101’s comment there –).

will start reviving it, and probably decouple most of the base functionality tests into another PR, so 25979 is mostly focused on the new change detection functionality.

achow101 commented at 4:40 pm on January 5, 2023: member

What’s happening here is that with descriptor wallets, we do not add any address book entries for descriptors imported without a label. Ranged descriptors cannot have any labels applied because labels are per-address, and a ranged descriptor can produce (basically) infinite addresses. So any addresses for ranged descriptors do not get any address book entries, and so we would detect any coins sent to such addresses to be change.

However, if the descriptor is an active descriptor, when we add a transaction to the wallet, we will additionally lookup whether the active descriptor is also an internal descriptor. If it is not, we will create an address book entry so that anything that needs to know whether the address is change can use the address book lookup mechanism to figure this out. This is how the active=true workaround works.

This is not an issue for legacy wallets because legacy wallets do not truly support ranged descriptors. It will expand the ranged descriptor and get all of the scriptPubKeys it produces, then import each one individually. When it does this, it can apply a label and add an address book entry for every single thing imported. Thus it avoids this problem.

I think the only real solution to this problem is to change how we detect change. Ideally we would have some system that can infer it intelligently, and also have change be explicitly marked rather, so the default is to assume everything is receiving, rather than everything is change.

vibs29 commented at 9:01 pm on January 5, 2023: none

you reported it is working in a legacy watch-only wallet. How are you creating it ?

I would have imported specific public items using one of importaddress, importmulti or importpubkey, rather than ranged descriptors (xpub/*). I would import the address or public key but not the private key. And listreceivedbyaddress would then always show what I expected. It was only when I switched to descriptor wallets and importdescriptor that I found that listreceivedbyaddress would produce an empty result.

Thinking back, it’s likely that my legacy wallet experiments (and importing specific items) were only with an older bitcoind, e.g. whatever is currently in the docker hub image named: “bitcoindevelopernetwork/bitcoind-regtest”[0]. Otherwise, it would have created a descriptor wallet by default.

My guess is that the listreceivedbyaddress anomaly isn’t to do with one version of bitcoind versus another, or even legacy wallets versus descriptor wallets, but in fact between importing specific public items and importing ranged public items.

With a ranged public descriptor, the wallet notices incoming payments, updates its balance, has listsinceblock answer what I’d expect, etc. So I’d never have predicted that a correct answer from listreceivedbyaddress would be the empty list. That led to my enquiry in IRC, which led to creating this GitHub issue. Updating listreceivedbyaddress’s help text to say “does not yet work with ranged public descriptors” is also a great outcome if sorting out the underlying semantics is too hard or will only happen indefinitely far in the future.

[0]: docker image ID: 0a110218f3cb Bitcoin Core Daemon version v0.17.1.0-gef70f9b52b851c7997a9f1a0834714e3eebc1fd8

listreceivedbyaddress is empty for descriptor (but not legacy) wallets #26813