importdescriptors hanging on importing/updating descriptor with large range #25895

issue gituser opened this issue on August 21, 2022
  1. gituser commented at 5:05 PM on August 21, 2022: none

    <!-- Describe the issue -->

    When running importdescriptors with range parameter like [0, 900000] if there is a transaction to any of the addresses in high range, e.g. address derived from 84'/1'/0'/0/777777, bitcoind will hang and no longer respond to some of the RPC requests and syncing stops too.

    Furthermore in top I can observe for hours bitcoind process consuming 100% of CPU and doing nothing, also wallet isn't being touched.

    Expected behavior

    Bitcoin descriptor wallet should be working correctly and import such descriptor without any issues even if the range specified is huge.

    To reproduce

    1. Install latest bitcoin v23.0 (built using depends system), sync to testnet, by adding testnet=1 in your bitcoin.conf
    2. Create test descriptor wallet:
    ./bitcoin-cli createwallet test_1
    {
      "name": "test_1",
      "warning": ""
    }
    
    1. Get descriptors for the wallet:
    $ ./bitcoin-cli -rpcwallet=test_1 listdescriptors true
    {
      "wallet_name": "test_1",
      "descriptors": [
        {
          "desc": "tr(tprvXXX/86'/1'/0'/0/*)#fuh4d6pe",
          "timestamp": 1661100252,
          "active": true,
          "internal": false,
          "range": [
            0,
            999
          ],
          "next": 0
        },
        {
          "desc": "tr(tprvXXX/86'/1'/0'/1/*)#cgj5s03p",
          "timestamp": 1661100254,
          "active": true,
          "internal": true,
          "range": [
            0,
            999
          ],
          "next": 0
        },
        {
          "desc": "sh(wpkh(tprvXXX/49'/1'/0'/0/*))#5uegq3cr",
          "timestamp": 1661100252,
          "active": true,
          "internal": false,
          "range": [
            0,
            999
          ],
          "next": 0
        },
        {
          "desc": "sh(wpkh(tprvXXX/49'/1'/0'/1/*))#jl3dmunh",
          "timestamp": 1661100253,
          "active": true,
          "internal": true,
          "range": [
            0,
            999
          ],
          "next": 0
        },
        {
          "desc": "pkh(tprvXXX/44'/1'/0'/1/*)#t2man3ks",
          "timestamp": 1661100253,
          "active": true,
          "internal": true,
          "range": [
            0,
            999
          ],
          "next": 0
        },
        {
          "desc": "wpkh(tprvXXX/84'/1'/0'/1/*)#dxzkmnkj",
          "timestamp": 1661100253,
          "active": true,
          "internal": true,
          "range": [
            0,
            999
          ],
          "next": 0
        },
        {
          "desc": "pkh(tprvXXX/44'/1'/0'/0/*)#677uwyxg",
          "timestamp": 1661100251,
          "active": true,
          "internal": false,
          "range": [
            0,
            999
          ],
          "next": 0
        },
        {
          "desc": "wpkh(tprvXXX/84'/1'/0'/0/*)#uj8hxxx2",
          "timestamp": 1661100252,
          "active": true,
          "internal": false,
          "range": [
            0,
            999
          ],
          "next": 0
        }
      ]
    }
    
    1. Derive address with huge derivation path 84'/1'/0'/0/777777 for test:
    ./bitcoin-cli deriveaddresses "wpkh(tprvXXX/84'/1'/0'/0/777777)#60yed6sv"
    [
      "tb1q5ge5pu77784lzyukur27wz2y2mtc8zzpyfw8nl"
    ]
    
    1. Send some funds to that address:
    ./bitcoin-cli -rpcwallet="" sendtoaddress tb1q5ge5pu77784lzyukur27wz2y2mtc8zzpyfw8nl 0.00001
    c7ca6d0a3c94085d393f973af4cf0bfc4e09e2885a5d0ef42af356ffcbeee41d
    
    1. Update descriptor on the wallet to reflect the range (up to 900000):
    ./bitcoin-cli -rpcwallet=test_1 importdescriptors '[{ "desc": "wpkh(tprvXXX/84h/1h/0h/0/*)#slgn8chc", "timestamp":"now", "range": [0,900000]}]'
    [
      {
        "success": true
      }
    ]
    
    1. Now wait for the transaction you've sent earlier to confirm, check logs:
    2022-08-21T16:55:15Z [default wallet] AddToWallet c7ca6d0a3c94085d393f973af4cf0bfc4e09e2885a5d0ef42af356ffcbeee41d  update
    2022-08-21T16:55:15Z [test_1] MarkUnusedAddresses: Detected a used keypool item at index 777777, mark all keypool items up to this item as used
    
    1. After this bitcoind and wallet test_1 will hang and no longer sync for hours doing something..
    2. E.g. doing any request to wallet like:
    ./bitcoin-cli -rpcwallet=test_1 getwalletinfo
    

    will be hanging forever until the timeout occurs.

    System information Ubuntu 18.04.6 LTS

    <!-- What version of Bitcoin Core are you using, where did you get it (website, self-compiled, etc)? -->

    self-compiled v23.0 using internal depends

    <!-- What type of machine are you observing the error on (OS/CPU and disk type)? -->

    Wallet is on the NVME, while bitcoind blockchain data is on HDDs, plenty of memory about 16GB and 24 CPUs (AMD Ryzen 9 3900 12-Core Processor)

    <!-- GUI-related issue? What is your operating system and its version? If Linux, what is your desktop environment and graphical shell? -->

    <!-- Any extra information that might be useful in the debugging process. -->

    <!--- This is normally the contents of a `debug.log` or `config.log` file. Raw text or a link to a pastebin type site are preferred. -->

    bitcoin.conf:

    rpcuser=bitcoin_testnet_rpc
    rpcpassword=password
    rpcbind=127.0.0.1
    rpcallowip=127.0.0.1
    
    walletrbf=1
    txindex=1
    
    settxfee=0.0006
    paytxfee=0.0006
    
    dbcache=512
    walletnotify=/home/bitcoin/.bitcoin/cryptonotify.sh %s %w
    testnet=1
    
    [test]
    rpcworkqueue=128
    txindex=1
    rpcbind=127.0.0.1
    rpcallowip=127.0.0.1
    
    # default wallet
    wallet=
    

    debug.log:

    2022-08-21T16:45:01Z Using SQLite Version 3.32.1
    2022-08-21T16:45:01Z Using wallet /home/bitcoin/.bitcoin/testnet3/test_1
    2022-08-21T16:45:01Z init message: Loading wallet…
    2022-08-21T16:45:01Z [test_1] Setting spkMan to active: id = xxx, type = 0, internal = 0
    2022-08-21T16:45:01Z [test_1] Setting spkMan to active: id = xxx, type = 1, internal = 0
    2022-08-21T16:45:01Z [test_1] Setting spkMan to active: id = xxx, type = 2, internal = 0
    2022-08-21T16:45:01Z [test_1] Setting spkMan to active: id = xxx, type = 3, internal = 0
    2022-08-21T16:45:01Z [test_1] Setting spkMan to active: id = xxx, type = 0, internal = 1
    2022-08-21T16:45:01Z [test_1] Setting spkMan to active: id = xxx, type = 1, internal = 1
    2022-08-21T16:45:01Z [test_1] Setting spkMan to active: id = xxx, type = 2, internal = 1
    2022-08-21T16:45:01Z [test_1] Setting spkMan to active: id = xxx, type = 3, internal = 1
    2022-08-21T16:45:01Z [test_1] Wallet File Version = 169900
    2022-08-21T16:45:01Z [test_1] Keys: 8 plaintext, 0 encrypted, 0 w/ metadata, 8 total. Unknown wallet records: 0
    2022-08-21T16:45:01Z [test_1] Wallet completed loading in             397ms
    2022-08-21T16:45:01Z [test_1] setKeyPool.size() = 8000
    2022-08-21T16:45:01Z [test_1] mapWallet.size() = 0
    2022-08-21T16:45:01Z [test_1] m_address_book.size() = 0
    2022-08-21T16:49:52Z [test_1] Update existing descriptor: wpkh(tpubXXX/84'/1'/0'/0/*)#qfxctt9d
    2022-08-21T16:50:29Z [test_1] Deactivate spkMan: id = xxx, type = 2, internal = 0
    2022-08-21T16:50:29Z [test_1] RescanFromTime: Rescanning last 23 blocks
    2022-08-21T16:50:29Z [test_1] Rescan started from block 000000000000000cda1c49b7202b4714498675d4017d4b49e00f6d13926f3e7a...
    2022-08-21T16:50:29Z [test_1] Rescan completed in               2ms
    2022-08-21T16:55:15Z UpdateTip: new best=00000000a24fd66fed58a7f821f74af68541b303bae8e26642499b4f5ff9cc35 height=2343214 version=0x20000000 log2_work=74.879888 tx=63453651 date='2022-08-21T16:55:11Z' progress=1.000000 cache=0.1MiB(612txo)
    2022-08-21T16:55:15Z [default wallet] AddToWallet c7ca6d0a3c94085d393f973af4cf0bfc4e09e2885a5d0ef42af356ffcbeee41d  update
    2022-08-21T16:55:15Z [test_1] MarkUnusedAddresses: Detected a used keypool item at index 777777, mark all keypool items up to this item as used
    
  2. gituser added the label Bug on Aug 21, 2022
  3. gituser commented at 5:06 PM on August 21, 2022: none

    Ping @achow101

  4. furszy commented at 2:05 PM on August 22, 2022: member

    As you manually derived the address number 777,777 in the ranged descriptor path and then imported the descriptor with the default next_index param value (which is 0), you forced the wallet to derive the 777,776 previous addresses during a single block connection to mark them as used (default wallet behavior).

    So, probably, your wallet isn't stalled, it's just deriving and marking all those addresses as used at once during block connection/rescan (storing a new addressbook entry on disk for each of them among other data)

    So, some points:

    1. You could import the descriptor with the next_index field at 777,777 so the wallet can omit the derivation of all the previous addresses. Not marking them as used as they were never created (or just import the address as a standalone address and not a ranged descriptor)

    2. If you are actually using the previous addresses, and you don't care about address re-using, maybe could add a customizable option to mark / not mark addresses as used on ranged descriptors (open for discussion)

    3. There is a PR, that is still a WIP, that adds db txn batch writes to the block connection and tx sync processes, so the processes don't go to disk for each independent address that requires to be marked as used (and other stuff) #25297. You might want to give that PR branch a try.

  5. theStack commented at 2:33 PM on August 22, 2022: contributor

    FWIW, in a first quick test I couldn't reproduce this issue on signet with the latest version of the master branch. Sure, deriving hundreds of thousands of addresses and marking them as unused takes some time (about 5-10 minutes on my system, which is way slower compared to the HW specs given in the PR description), but there was no permanent hanging. Will test with v23 and testnet later.

  6. furszy commented at 2:40 PM on August 22, 2022: member

    FWIW, in a first quick test I couldn't reproduce this issue on signet with the latest version of the master branch. Sure, deriving hundreds of thousands of addresses and marking them as unused takes some time (about 5-10 minutes on my system, which is way slower compared to the HW specs given in the PR description), but there was no permanent hanging. Will test with v23 and testnet later.

    same here, tested it on master and on 25297. Had no problems there, took less than a minute on my machine. crafted a quick test for it https://gist.github.com/furszy/2fab16877d787d4d9001b0be93090cf3

  7. gituser commented at 3:00 PM on August 22, 2022: none

    @furszy thanks for the quick reply!

    This bug is 100% reproduceable for me both on testnet and production wallets, in both cases bitcoind goes stuck forever (even left it overnight).

    I'll try your PR or master branch, thanks, but I'd like to use a stable version though with a fix.

    Is there a version of this PR for v23.0 by any chance?

  8. gituser commented at 3:01 PM on August 22, 2022: none

    @theStack

    FWIW, in a first quick test I couldn't reproduce this issue on signet with the latest version of the master branch. Sure, deriving hundreds of thousands of addresses and marking them as unused takes some time (about 5-10 minutes on my system, which is way slower compared to the HW specs given in the PR description), but there was no permanent hanging. Will test with v23 and testnet later.

    In order to reproduce you need to create a transaction to that 777777 derived address and make sure it was confirmed on the blockchain and added to your newly created wallet.

  9. theStack commented at 3:46 PM on August 22, 2022: contributor

    In order to reproduce you need to create a transaction to that 777777 derived address and make sure it was confirmed on the blockchain and added to your newly created wallet.

    I tried that with two scenarios so far that both worked: (1) send tx, wait for the tx to confirm, import the descriptor (2) import the descriptor, send tx, wait for the tx to confirm

    Reading your opening post more carefully, it seems you did yet another variant: (3) send tx, import the descriptor, wait for the tx to confirm

    Will also try (3) in a bit. Could you try also variants (1) and (2) and see if it leads to the same hanging behaviour?

  10. gituser commented at 8:06 PM on January 26, 2023: none

    @furszy I've been running bitcoin with your pull request applied (though it's quite old from August 22nd 2022 I think).

    The only issue I'm encountering is:

    • when I try to withdraw some funds from descriptor wallet with fundrawtransaction function
    • and simultaneously try to get new address from the wallet with getnewaddress
    • getnewaddress gets a timeout of 60 seconds (possible lock issue?)

    In ~/.bitcoin/debug.log when hanging happens I get these lines:

    2023-01-26T19:55:43Z [wallet] Fee Calculation: Fee:2144 Bytes:177 Tgt:0 (requested 0) Reason:"PayTxFee set" Decay 0.00000: Estimation: (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out) Fail: (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out)
    2023-01-26T19:56:20Z [wallet] Fee Calculation: Fee:2144 Bytes:177 Tgt:0 (requested 0) Reason:"PayTxFee set" Decay 0.00000: Estimation: (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out) Fail: (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out)
    2023-01-26T19:56:20Z [wallet] Fee non-grouped = 2144, grouped = 2144, using grouped
    

    Any idea if this issue has been fixed in the latest master or in your latest changes in the #25297 ?

    Thank you.

  11. willcl-ark commented at 3:19 PM on April 10, 2024: member

    @furszy is this fixed in master?

  12. Krokochik commented at 7:21 PM on April 16, 2024: none

    @willcl-ark no it doesn't

  13. furszy commented at 2:24 PM on April 19, 2024: member

    @furszy is this fixed in master?

    Locally, the test doesn't take much. Around 100 secs. But I can see how it could take a while on a spinning disk. #28894 should have improved the import situation but we are still far from getting #25297. Once #28574 gets merged, will continue decoupling #25297 in smaller PRs to improve this scenario.


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-04-15 18:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me