wallet, rpc: add an option to list private descriptors #21500

Rationale: make it possible to backup your wallet with listdescriptors command

• The default behaviour is still to show public version
• For private version only the root xprv is returned

Example use-case:

> bitcoin-cli -regtest -named createwallet wallet_name=old descriptors=true
> bitcoin-cli -regtest -rpcwallet=old listdescriptors true | jq '.descriptors' > descriptors.txt

> bitcoin-cli -regtest -named createwallet wallet_name=new descriptors=true blank=true
> bitcoin-cli -regtest -rpcwallet=new importdescriptors "$(cat descriptors.txt)"

In case of watch-only wallet without private keys there will be following output:

error code: -4
error message:
Can't get descriptor string.

<!--e57a25ab6845829454e8d69fc972939a-->

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

<!--174a7506f384e20aa4161008e828411d-->

Conflicts

No conflicts as of last run.

<!--4a62be1de6b64f3ed646cdc7932c8cf5-->

🕵️ @achow101 @sipa have been requested to review this pull request as specified in the REVIEWERS file.

Concept ACK. The second commit seems unrelated.

Fixed silent merge conflict and split the 2nd commit into #22334

Tried listdescriptors RPC although I have no clue what are these 6 descriptors.

Create Wallet:

POST / HTTP/1.1
Host: 127.0.0.1:18777
Authorization: Basic hObGNqTTZjR0Z6YzNkd
Content-Type: text/plain
Content-Length: 106

{"jsonrpc": "1.0", "id": "curltest", "method": "createwallet", "params": ["DT1",false,false,"",true,true]}

List descriptors:

POST /wallet/DT1 HTTP/1.1
Host: 127.0.0.1:18777
Authorization: Basic hObGNqTTZjR0Z6YzNkd
Content-Type: text/plain
Content-Length: 83

{"jsonrpc": "1.0", "id": "curltest", "method": "listdescriptors", "params": [true]}

Response (I get the same response for true and false in params:

{
    "result": {
        "wallet_name": "DT1",
        "descriptors": [
            {
                "desc": "pkh([07f2e700/44'/1'/0']tpubDC89pDwHmvJV6xtEEWniQYEZ9Nga9cshwTHAT3dHfyWftFUtUWoQ5qAr4scvE853a2rFLqH4ZhoJtNDhSB8HS9oG8KDN5rWCaFotTFmKSAV/0/*)#ek3c29xk",
                "timestamp": 1625657174,
                "active": true,
                "internal": false,
                "range": [
                    0,
                    999
                ],
                "next": 0
            },
            {
                "desc": "wpkh([07f2e700/84'/1'/0']tpubDD49UXKp4mBeHhZKURhTFqhnnuMu5EA88v5GWFm3wnQJfZmyMwSsQqM6X3eFUwkAcRD4j9wiwoGzmphJMwEB1PwdwQuTbPdKUjhmpEsfqmh/0/*)#07l98jpn",
                "timestamp": 1625657175,
                "active": true,
                "internal": false,
                "range": [
                    0,
                    999
                ],
                "next": 0
            },
            {
                "desc": "pkh([07f2e700/44'/1'/0']tpubDC89pDwHmvJV6xtEEWniQYEZ9Nga9cshwTHAT3dHfyWftFUtUWoQ5qAr4scvE853a2rFLqH4ZhoJtNDhSB8HS9oG8KDN5rWCaFotTFmKSAV/1/*)#gz5ehskw",
                "timestamp": 1625657175,
                "active": true,
                "internal": true,
                "range": [
                    0,
                    999
                ],
                "next": 0
            },
            {
                "desc": "sh(wpkh([07f2e700/49'/1'/0']tpubDDMzDzQabs6VaDCvGa8jxNPZuDp28No6UBzathahL63xj3zWndRnNzTGqrCRSNpJDSZNnhkEKq2GwPfyGMkGip9Kqq4UkqE5Jb2x9EBsMzM/0/*))#kx6a8xhn",
                "timestamp": 1625657174,
                "active": true,
                "internal": false,
                "range": [
                    0,
                    999
                ],
                "next": 0
            },
            {
                "desc": "sh(wpkh([07f2e700/49'/1'/0']tpubDDMzDzQabs6VaDCvGa8jxNPZuDp28No6UBzathahL63xj3zWndRnNzTGqrCRSNpJDSZNnhkEKq2GwPfyGMkGip9Kqq4UkqE5Jb2x9EBsMzM/1/*))#r85tlezv",
                "timestamp": 1625657175,
                "active": true,
                "internal": true,
                "range": [
                    0,
                    999
                ],
                "next": 0
            },
            {
                "desc": "wpkh([07f2e700/84'/1'/0']tpubDD49UXKp4mBeHhZKURhTFqhnnuMu5EA88v5GWFm3wnQJfZmyMwSsQqM6X3eFUwkAcRD4j9wiwoGzmphJMwEB1PwdwQuTbPdKUjhmpEsfqmh/1/*)#726y683t",
                "timestamp": 1625657175,
                "active": true,
                "internal": true,
                "range": [
                    0,
                    999
                ],
                "next": 0
            }
        ]
    },
    "error": null,
    "id": "curltest"
}

Response (I get the same response for true and false in params:

Are you sure you are using this branch? I get the private key with true, and the normalized public descriptor with false.

ACK 2854ddca9bf9662170fecc2657ec3c19d364d2e2

Reviewed code and tested manually.

Are you sure you are using this branch? I get the private key with true, and the normalized public descriptor with false.

Yes just noticed the difference. false returns tpub and true returns tprv

Sorry they looked same and both had 6 results. So, I was confused.

@prayank23 Thanks for testing! By default descriptor wallet has two descriptors for each type:

1. Legacy — pkh(...)
2. Wrapped SegWit — sh(wpkh(...))
3. Native SegWit, bech32 — wpkh(...)

One of the two descriptors is for receiving, marked with internal: false and one for change internal: true.

So in the end you've got 6 of them. Each descriptor is derived from the same extended master private key (you can see it's fingerprint repeating in the descriptors; 07f2e700 in your example) using corresponding derivation paths. See BIPs: 44, 49 and 84.

tACK https://github.com/bitcoin/bitcoin/pull/21500/commits/2854ddca9bf9662170fecc2657ec3c19d364d2e2

I'm getting a build error trying to test:

(edited out).

I'm getting these on master as well, so investigating/removing from here as it is not this PR specific.

ACK 2854ddca9bf9662170fecc2657ec3c19d364d2e2

Some minor suggestions, happy to re-ACK if you update.

Added a test case for the error when trying to get private descriptors for watch-only wallet. Plus minor readability and grammar fixes

Thanks for adding the test!

ACK bb822a7af86897a9b6a5d616f193c258e8e76729 per git diff 2854ddc bb822a7

<details><summary>Verified the test assertions added in this pull fail on master as expected, e.g. they have to be commented-out for the test file to pass</summary><p>

diff --git a/test/functional/wallet_listdescriptors.py b/test/functional/wallet_listdescriptors.py
index 6e7be929cc..61de4621bf 100755
--- a/test/functional/wallet_listdescriptors.py
+++ b/test/functional/wallet_listdescriptors.py
@@ -71,7 +71,7 @@ class ListDescriptorsTest(BitcoinTestFramework):
             ],
         }
         assert_equal(expected, wallet.listdescriptors())
-        assert_equal(expected, wallet.listdescriptors(False))
+        # assert_equal(expected, wallet.listdescriptors(False))
 
         self.log.info('Test list private descriptors')
         expected_private = {
@@ -84,16 +84,16 @@ class ListDescriptorsTest(BitcoinTestFramework):
                  'next': 0},
             ],
         }
-        assert_equal(expected_private, wallet.listdescriptors(True))
+        # assert_equal(expected_private, wallet.listdescriptors(True))
 
         self.log.info("Test listdescriptors with encrypted wallet")
         wallet.encryptwallet("pass")
         assert_equal(expected, wallet.listdescriptors())
 
         self.log.info('Test list private descriptors with encrypted wallet')
-        assert_raises_rpc_error(-13, 'Please enter the wallet passphrase with walletpassphrase first.', wallet.listdescriptors, True)
+        # assert_raises_rpc_error(-13, 'Please enter the wallet passphrase with walletpassphrase first.', wallet.listdescriptors, True)
         wallet.walletpassphrase(passphrase="pass", timeout=1000000)
-        assert_equal(expected_private, wallet.listdescriptors(True))
+        # assert_equal(expected_private, wallet.listdescriptors(True))
 
         self.log.info('Test list private descriptors with watch-only wallet')
         node.createwallet(wallet_name='watch-only', descriptors=True, disable_private_keys=True)
@@ -102,7 +102,7 @@ class ListDescriptorsTest(BitcoinTestFramework):
             'desc': descsum_create('wpkh(' + xpub_acc + ')'),
             'timestamp': 1296688602,
         }])
-        assert_raises_rpc_error(-4, 'Can\'t get descriptor string', watch_only_wallet.listdescriptors, True)
+        # assert_raises_rpc_error(-4, 'Can\'t get descriptor string', watch_only_wallet.listdescriptors, True)
 
         self.log.info('Test non-active non-range combo descriptor')
         node.createwallet(wallet_name='w4', blank=True, descriptors=True)

</p></details>

re-ACK bb822a7af86897a9b6a5d616f193c258e8e76729

@prayank23 @Rspigler do you want to check the latest version?

Sure. Will try today. I had two questions, maybe @practicalswift can help me:

1. Does it make sense to try SQL queries in fuzzing descriptor wallet related RPCs? I tried but could not find anything interesting: https://link.medium.com/iQBbB5Tbqib
2. Are emojis a part of strings that are used in fuzzing Bitcoin Core? https://github.com/danielmiessler/SecLists/blob/master/Fuzzing/big-list-of-naughty-strings.txt (few emojis in this list)

tACK https://github.com/bitcoin/bitcoin/pull/21500/commits/bb822a7af86897a9b6a5d616f193c258e8e76729

Tested everything mentioned in PR description and below things:

W1: Descriptor wallet with private keys W2: Blank Descriptor wallet W3: Descriptor wallet with private keys disabled W4: Another Descriptor wallet with private keys

1. Export descriptors from wallet W1 and import in same wallet
2. Export descriptors from wallet W1 and import in Wallet W2 and W4
3. Export descriptors from wallet W1 and import in Wallet W3

Everything looks good with listdescriptors, maybe we can improve one thing in importdescriptors:

When exporting-importing in same wallet it does not import duplicates however response is same as 2.

{
  "success": true
}

In case of watch-only wallet without private keys there will be following output: error code: -4 error message: Can't get descriptor string.

I don't see this warning for watch only wallet. It returns wallet name and blank array for descriptors.

tACK bb822a7af86897a9b6a5d616f193c258e8e76729

This would be very helpful for metal or paper backups

listdescriptors:

{ "wallet_name": "Test_ListDescriptors", "descriptors": [ { "desc": "pkh([db98369e/44'/0'/0']xpub6DFFBCcFzoG7cfKDPRX39zd3ZBydFnM26VmDEVDLryPU6u6coYZKkBpe7DKPa8V1RjAvvdx4UebiXqSirkevEpsdByAbS1GjF1WGofMWcNc/0/)#6j60t7ur", "timestamp": 1628016073, "active": true, "internal": false, "range": [ 0, 999 ], "next": 0 }, { "desc": "sh(wpkh([db98369e/49'/0'/0']xpub6DG6d9WNZi2EwnkKm8AZTRTijvff5kiMUPFyQN7aDiNJ5Hu7Nea4zeMVi7g8st1PxNKVq6gidk5vPiV2uMyu8QgmCa92A9XaXfZ7ZfCMrZP/0/))#luvn8xr5", "timestamp": 1628016073, "active": true, "internal": false, "range": [ 0, 999 ], "next": 0 }, { "desc": "wpkh([db98369e/84'/0'/0']xpub6CPCUJH2M2VkskpSijywKte99eQ1WXZnVtTBEbWUCWKzHB36ddWSwmmg4jL4M2Nk8PRK3XS4YbTrKk5sCx1iCEgjKsZU2Vp5xQvgThhNQFH/0/)#g22fa57g", "timestamp": 1628016074, "active": true, "internal": false, "range": [ 0, 999 ], "next": 0 }, { "desc": "pkh([db98369e/44'/0'/0']xpub6DFFBCcFzoG7cfKDPRX39zd3ZBydFnM26VmDEVDLryPU6u6coYZKkBpe7DKPa8V1RjAvvdx4UebiXqSirkevEpsdByAbS1GjF1WGofMWcNc/1/)#txlwktvm", "timestamp": 1628016074, "active": true, "internal": true, "range": [ 0, 999 ], "next": 0 }, { "desc": "sh(wpkh([db98369e/49'/0'/0']xpub6DG6d9WNZi2EwnkKm8AZTRTijvff5kiMUPFyQN7aDiNJ5Hu7Nea4zeMVi7g8st1PxNKVq6gidk5vPiV2uMyu8QgmCa92A9XaXfZ7ZfCMrZP/1/))#2az9lekt", "timestamp": 1628016074, "active": true, "internal": true, "range": [ 0, 999 ], "next": 0 }, { "desc": "wpkh([db98369e/84'/0'/0']xpub6CPCUJH2M2VkskpSijywKte99eQ1WXZnVtTBEbWUCWKzHB36ddWSwmmg4jL4M2Nk8PRK3XS4YbTrKk5sCx1iCEgjKsZU2Vp5xQvgThhNQFH/1/)#e70gqpws", "timestamp": 1628016075, "active": true, "internal": true, "range": [ 0, 999 ], "next": 0 } ] }

listdescriptors true:

{ "wallet_name": "Test_ListDescriptors", "descriptors": [ { "desc": "pkh(xprv9s21ZrQH143K2CBvvodeKZrxF474uqw846zfy9Q4hWEtkuNZoSRdUrtPjntix8JXUh6M5VXPFEw7z8w7jvMkqZEP2D5qVoyfvmMC2SVfHc5/44'/0'/0'/0/)#3ln4pscr", "timestamp": 1628016073, "active": true, "internal": false, "range": [ 0, 999 ], "next": 0 }, { "desc": "sh(wpkh(xprv9s21ZrQH143K2CBvvodeKZrxF474uqw846zfy9Q4hWEtkuNZoSRdUrtPjntix8JXUh6M5VXPFEw7z8w7jvMkqZEP2D5qVoyfvmMC2SVfHc5/49'/0'/0'/0/))#ulkh8xrl", "timestamp": 1628016073, "active": true, "internal": false, "range": [ 0, 999 ], "next": 0 }, { "desc": "wpkh(xprv9s21ZrQH143K2CBvvodeKZrxF474uqw846zfy9Q4hWEtkuNZoSRdUrtPjntix8JXUh6M5VXPFEw7z8w7jvMkqZEP2D5qVoyfvmMC2SVfHc5/84'/0'/0'/0/)#8mevkeep", "timestamp": 1628016074, "active": true, "internal": false, "range": [ 0, 999 ], "next": 0 }, { "desc": "pkh(xprv9s21ZrQH143K2CBvvodeKZrxF474uqw846zfy9Q4hWEtkuNZoSRdUrtPjntix8JXUh6M5VXPFEw7z8w7jvMkqZEP2D5qVoyfvmMC2SVfHc5/44'/0'/0'/1/)#qtk5u9gm", "timestamp": 1628016074, "active": true, "internal": true, "range": [ 0, 999 ], "next": 0 }, { "desc": "sh(wpkh(xprv9s21ZrQH143K2CBvvodeKZrxF474uqw846zfy9Q4hWEtkuNZoSRdUrtPjntix8JXUh6M5VXPFEw7z8w7jvMkqZEP2D5qVoyfvmMC2SVfHc5/49'/0'/0'/1/))#6u7jutgt", "timestamp": 1628016074, "active": true, "internal": true, "range": [ 0, 999 ], "next": 0 }, { "desc": "wpkh(xprv9s21ZrQH143K2CBvvodeKZrxF474uqw846zfy9Q4hWEtkuNZoSRdUrtPjntix8JXUh6M5VXPFEw7z8w7jvMkqZEP2D5qVoyfvmMC2SVfHc5/84'/0'/0'/1/)#k0udtvfe", "timestamp": 1628016075, "active": true, "internal": true, "range": [ 0, 999 ], "next": 0 } ] }

Code review ACK bb822a7af86897a9b6a5d616f193c258e8e76729