`listtransactions` fails to list self-send transactions (for imported descriptor wallet) #26096

Expected behavior:

listtransactions "*" 100 in a wallet with 35 transactions should list all 35 transactions in the wallet.

(Update: and given the way listtransactions works there will likely be more than one array entry for each transaction, right?)

Actual behavior:

listtransactions "*" 100 lists 14 transactions, but does not list 21 transactions that are self-sends.

Note that getwalletinfo returns:

  "txcount": 35,

So the wallet does seem to know about the self-send transactions. (The "missing" transactions also show up in the GUI)

To reproduce

I have only seen the problem with one particular wallet. The only other wallet I have tested with is running on an older Bitcoin Core v0.20.1 (actually Omni Core) version and does seem to correctly list self-sends.

The wallet showing the problem is an imported HD tpub descriptor wallet that was created with bitcoinj and then the descriptors were imported via importdescriptors. I noticed that there is a Python functional test for listtransactions that checks self-sends, so perhaps the problem is related to it being a watch-only descriptor wallet with imported descriptors. It may also be relevant that the self-sends had two-outputs, both a "destination" and a change address.

I can reliably reproduce the problem with the wallet that is showing the problem.

System information

MacBook Pro M1 macOS Monterey 12.5.1

Bitcoin Core v23.0.0 pre-built "official" binary.

I am willing to provide additional information and/or run additional tests to help track this down.

I've tried to reproduce the issue with a python test but it runs fine. Is this setup close to your situation?

diff --git a/test/functional/wallet_listtransactions.py b/test/functional/wallet_listtransactions.py
index 7c16b6328..266a8c486 100755
--- a/test/functional/wallet_listtransactions.py
+++ b/test/functional/wallet_listtransactions.py
@@ -31,6 +31,8 @@ class ListTransactionsTest(BitcoinTestFramework):
         self.skip_if_no_wallet()
 
     def run_test(self):
+        self.run_watch_only_externally_imported_descriptors_test()
+
         self.log.info("Test simple send from node0 to node1")
         txid = self.nodes[0].sendtoaddress(self.nodes[1].getnewaddress(), 0.1)
         self.sync_all()
@@ -277,6 +279,46 @@ class ListTransactionsTest(BitcoinTestFramework):
         assert_equal(['pizza2'], self.nodes[0].getaddressinfo(addr2)['labels'])
         assert_equal(['pizza3'], self.nodes[0].getaddressinfo(addr3)['labels'])
 
+    def run_watch_only_externally_imported_descriptors_test(self):
+        # create watch-only wallet
+        w1 = self.nodes[1].get_wallet_rpc(self.default_wallet_name)
+        self.log.info("Import descriptors on node 2")
+        self.nodes[2].createwallet(wallet_name='watchonly', disable_private_keys=True, descriptors=True)
+        watchonly_wallet = self.nodes[2].get_wallet_rpc('watchonly')
+        addr = self.nodes[1].getnewaddress()
+        desc = [{
+            "desc": w1.getaddressinfo(addr)['desc'],
+            "timestamp": "now",
+            "active": False,
+            "internal": False,
+            "watchonly": True,
+        }]
+        ret = watchonly_wallet.importdescriptors(desc)
+        assert ret[0]['success']
+
+        txid = self.nodes[1].sendtoaddress(addr, "0.001")
+        self.generate(self.nodes[0], 1)
+
+        self.log.info("Verify both wallets see the same transaction")
+        transactions1 = self.nodes[1].listtransactions()
+        transactions2 = watchonly_wallet.listtransactions()
+
+        found = False
+        for tx in transactions1:
+            if tx['txid'] == txid:
+                found = True
+
+        assert found
+
+        found = False
+        for tx in transactions2:
+            if tx['txid'] == txid:
+                found = True
+
+        assert found
+
+        exit(0) # temporary
+
     def run_invalid_parameters_test(self):
         self.log.info("Test listtransactions RPC parameter validity")
         assert_raises_rpc_error(-8, 'Label argument must be a valid label name or "*".', self.nodes[0].listtransactions, label="")

My situation is different in at least the following ways (I'm not sure which of the differences are salient):

1. TestNet not RegTest
2. The descriptors were imported with existing transactions going back to 2019-05-21 and the chain was rescanned
3. HD descriptors
4. Multiple descriptors: 2 HD and 1 regular address
5. The "missing" transactions were likely sent from keys/address in one HD subpath to the other (I recall this but haven't verified)
6. The transactions were created with bitcoinj-based code that I was testing/debugging not with Bitcoin Core (but are valid, of course)

Here is the output of listdescriptors for the wallet:

{
  "wallet_name": "desc-import",
  "descriptors": [
    {
      "desc": "pkh(tpubDDpSwdfCsfnYP8SH7YZvu1LK3BUMr3RQruCKTkKdtnHy2iBNJWn1CYvLwgskZxVNBV4KhicZ4FfgFCGjTwo4ATqdwoQcb5UjJ6ejaey5Ff8/1/*)#l9ydstel",
      "timestamp": 1,
      "active": false,
      "range": [
        0,
        1022
      ],
      "next": 23
    },
    {
      "desc": "addr(mqJVcXdXCuaueDRCtg92gyZXjpv3DF7jTN)#9lnmu846",
      "timestamp": 1455191478,
      "active": false
    },
    {
      "desc": "pkh(tpubDDpSwdfCsfnYP8SH7YZvu1LK3BUMr3RQruCKTkKdtnHy2iBNJWn1CYvLwgskZxVNBV4KhicZ4FfgFCGjTwo4ATqdwoQcb5UjJ6ejaey5Ff8/0/*)#w3pvd7f8",
      "timestamp": 1,
      "active": false,
      "range": [
        0,
        1022
      ],
      "next": 23
    }
  ]
}

And here is the output of getwalletinfo:

{
  "walletname": "desc-import",
  "walletversion": 169900,
  "format": "sqlite",
  "balance": 0.23178842,
  "unconfirmed_balance": 0.00000000,
  "immature_balance": 0.00000000,
  "txcount": 35,
  "keypoolsize": 0,
  "keypoolsize_hd_internal": 0,
  "paytxfee": 0.00000000,
  "private_keys_enabled": false,
  "avoid_reuse": false,
  "scanning": false,
  "descriptors": true,
  "external_signer": false
}

Note the txcount of 35 and balance of 0.23178842. The incorrect balance I calculated from listtransactions is 0.23789642.

An example of a "missing" transaction is txid 1105265b74dd9d3386d1983971548fffd8f4e3e2a11215695721978e1b108272: https://www.blockchain.com/btc-testnet/tx/1105265b74dd9d3386d1983971548fffd8f4e3e2a11215695721978e1b108272

The "missing" transactions were likely sent from keys/address in one HD subpath to the other (I recall this but haven't verified)

If you are willing to use listsinceblock instead, i suspect the new include_change parameter added in #25504 would be helpful for you. See #25504 (comment) in particular.

Thanks for the response @darosior!

I've been working on a functional test (in Groovy/Spock using the ConsensusJ Bitcoin JSON-RPC client) to reproduce the problem, but haven't succeeded yet.

I assume the fix in #25504 is available in the current v24.0 RC? If so, I will give it a try.

When I get the functional test working, I also may try changing some of the parameters to importdescriptor to see if those are a factor.

For self-send detection to work we need to distinguish between receiving and change addresses. For descriptor wallets we do that by checking internal flag on the descriptor. But internal flag could only be set on an active descriptor. Both your imported descriptors are not active, therefore we can't determine which outputs are change and which are receiving.

@msgilligan any luck with your testing?

For self-send detection to work we need to distinguish between receiving and change addresses. For descriptor wallets we do that by checking internal flag on the descriptor. But internal flag could only be set on an active descriptor. Both your imported descriptors are not active, therefore we can't determine which outputs are change and which are receiving. @S3RK, #25979 aims to fix that. note: It's currently up-to-date with master but I should revisit it after that many months.