Running RPC call importmulti with a xpub key with very large range results in a timeout error #17797

<!-- Describe the issue -->

Running the RPC call importmulti with a xpub key with very large range results in a timeout error and seems to ultimately fail to import all addresses. Re-running the same RPC line may succeed in importing the missing addresses but not always.

Expected behavior

I run this RPC $ bitcoin-cli -rpcwallet=wallet-my-test-wallet.dat importmulti '[{"desc": "sh(wpkh(xpub6EcgD73U5aQe8F9k1VQmz3bDdnMmgNAouVixdsKVC9YTJb1BNPPq9jSCuDArzxzj2JAWcGyL9sJofgLQHAYovQqCibh3ovuWVtwiagp1x7d/*))#9fdjk2ls", "range": [0, 999999], "timestamp": "now", "watchonly": true}]' '{"rescan": false}'

I'd expect it to return nothing and import all those addresses.

<!--- What behavior did you expect? -->

Actual behavior

The RPC call runs for a couple of minutes and fails with the follow error:

error: Could not connect to the server 127.0.0.1:8332 (error code 0 - "timeout reached")

Make sure the bitcoind server is running and that you are connecting to the correct RPC port.

If I re-run the same RPC call then the call may return without the error message.

Other RPC calls (e.g. getbestblockhash) work as normal.

<!--- What was the actual behavior (provide screenshots if the issue is GUI-related)? -->

To reproduce

The issue is reproducible every time. With a new empty wallet file (my-test-wallet.dat), run this:

$ bitcoin-cli -rpcwallet=wallet-my-test-wallet.dat importmulti '[{"desc": "sh(wpkh(xpub6EcgD73U5aQe8F9k1VQmz3bDdnMmgNAouVixdsKVC9YTJb1BNPPq9jSCuDArzxzj2JAWcGyL9sJofgLQHAYovQqCibh3ovuWVtwiagp1x7d/*))#9fdjk2ls", "range": [0, 999999], "timestamp": "now", "watchonly": true}]' '{"rescan": false}'

Warning: this will make the wallet file become about 1GB in size.

Importing 999999 addresses seems to never succeed no matter how many times its retried. Importing 500000 addresses seems to succeed after re-running the command once.

<!--- How reliably can you reproduce the issue, what are the steps to do so? -->

System information

<!-- What version of Bitcoin Core are you using, where did you get it (website, self-compiled, etc)? -->

Bitcoin Core 0.19.0.1, binary downloaded from bitcoincore.org

<!-- What type of machine are you observing the error on (OS/CPU and disk type)? -->

$ uname -a
Linux hostname 4.4.0-142-generic [#168](/bitcoin-bitcoin/168/)-Ubuntu SMP Wed Jan 16 21:00:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Using internal hard disk

<!-- GUI-related issue? What is your operating system and its version? If Linux, what is your desktop environment and graphical shell? -->

<!-- Any extra information that might be useful in the debugging process. -->

<!--- This is normally the contents of a `debug.log` or `config.log` file. Raw text or a link to a pastebin type site are preferred. -->

After some more experimenting I've found that actually all the addresses are imported correctly, even though RPC call returns with an error. So this isn't really a bug and the function works properly, except that the error message is confusing.

@chris-belcher I believe the error is because of your shell. Executing

importmulti '[{"desc": "sh(wpkh(xpub6EcgD73U5aQe8F9k1VQmz3bDdnMmgNAouVixdsKVC9YTJb1BNPPq9jSCuDArzxzj2JAWcGyL9sJofgLQHAYovQqCibh3ovuWVtwiagp1x7d/*))#9fdjk2ls", "range": [0, 999999], "timestamp": "now", "watchonly": true}]' '{"rescan": false}'

in the bitcoin-qt console works just fine

Isn't this just a HTTP timeout? Haven't looked at code, but I assume it could be that bitcoin-qt console calls method directly, without JSON-RPC over HTTP.

Isn't this just a HTTP timeout? Haven't looked at code, but I assume it could be that bitcoin-qt console calls method directly, without JSON-RPC over HTTP.

Yes it calls directly ut I don't think that HTTP timeouts are that common especially not in localhost

I think that for this use case using bitcoin-cli -rpcclienttimeout=0 is a good enough workaround? Once we have native descriptor wallet support, importing should be instant.

I don't think it's worth it but it would be possible to use a start/stop/status approach like scantxoutset does.

@jonasschnelli that doesn't avoid the client timeout in the start request.

FWIW getrpcinfo lets you know which RPC calls are active. Only problem is that it doesn't let you know the import result.

@promag: Probably to complicated,... but IMO the start-call could start a thread that does all the timely blocking stuff. While the status call would report back if its complete and the completion-results (if complete).

@chris-belcher have you tried this with the new native descriptor wallet from April 2020? I have just tried to reproduce* and importing the range of 999,999 took 1m25s (and therefore didn't trigger the http timeout you experienced):

will@ubuntu in ~/.bitcoin
₿ /home/will/src/bitcoin/src/bitcoin-cli createwallet "test-wallet-disable-privkeys" true
{
  "name": "test-wallet-disable-privkeys",
  "warning": ""
}

will@ubuntu in ~/.bitcoin
₿ /home/will/src/bitcoin/src/bitcoin-cli -rpcwallet=test-wallet-disable-privkeys importdescriptors '[{"desc": "sh(wpkh(xpub6EcgD73U5aQe8F9k1VQmz3bDdnMmgNAouVixdsKVC9YTJb1BNPPq9jSCuDArzxzj2JAWcGyL9sJofgLQHAYovQqCibh3ovuWVtwiagp1x7d/*))#9fdjk2ls", "active": true, "range": [0, 999999], "timestamp": "now", "internal": false}]'
[
  {
    "success": true
  }
]

will@ubuntu in ~/.bitcoin took 1m25s
₿

* Note the call to importdescriptors along with slightly modified arguments

Closing for now, because bdb will be phased out and this doesn't seem to be happening with descriptor wallets? Let us know if this is still an issue and should be reopened.