The dumpwallet RPC command stops/kills bitcoind

lsilva01 commented at 10:29 pm on July 18, 2021: contributor

The dumpwallet RPC command throws an error at the signet node and stops it. ~~This does not happen in mainnet or testnet, where the command worked successfully in my tests..~~ Originally this error was only happening on signet, but it also happened on testnet after it worked successfully for a while.

Expected behavior

The RPC dumpwallet command is expected to return a success message and create the dump file that contains the private keys.

Actual behavior

The command returns an error on client side (bitcoin-cli): error: timeout on transient error: Could not connect to the server 127.0.0.1:38332 (error code 1 - "EOF reached") Make sure the bitcoind server is running and that you are connecting to the correct RPC port.

And it throws an error at the signet node and stops it.

 0POTENTIAL DEADLOCK DETECTED
 1Previous lock order was:
 2  'cs_wallet' in wallet/wallet.cpp:2860 (in thread 'httpworker.1')
 3  (2) '::cs_main' in node/interfaces.cpp:687 (in thread 'httpworker.1')
 4  'm_node.mempool->cs' in node/interfaces.cpp:687 (in thread 'httpworker.1')
 5  'cs_wallet' in wallet/wallet.cpp:1204 (in thread 'httpworker.1')
 6  (1) 'cs_KeyStore' in script/signingprovider.cpp:165 (in thread 'httpworker.1')
 7Current lock order is:
 8  'wallet.cs_wallet' in wallet/rpcdump.cpp:743 (in thread 'httpworker.3')
 9  (1) 'spk_man.cs_KeyStore' in wallet/rpcdump.cpp:743 (in thread 'httpworker.3')
10  (1) 'spk_man->cs_KeyStore' in wallet/wallet.cpp:2303 (in thread 'httpworker.3')
11  (2) 'cs_main' in node/interfaces.cpp:493 (in thread 'httpworker.3')
12Assertion failed: detected inconsistent lock order for 'cs_main' in node/interfaces.cpp:493 (in thread 'httpworker.3'), details in debug log.
13Aborted (core dumped)

To reproduce

Start the node on signet mode: bitcoind -signet

And dump an existing wallet: bitcoin-cli -signet -rpcwallet="legacy_wallet_01" dumpwallet "filename"

System information

The bitcoin node was compiled from master (commit: df2b5dadfb2dc21b05e5a60b71987fb5b96201f5) Ubuntu 20.04, 8 CPUs, 12 GiB RAM and SSD.

lsilva01 added the label Bug on Jul 18, 2021

achow101 commented at 10:41 pm on July 18, 2021: member

Does this only occur on signet?

lsilva01 commented at 11:09 pm on July 18, 2021: contributor

I just got this same error on testnet. So it is not related to the network.

lsilva01 renamed this:
~~The dumpwallet RPC command does not work on signet~~
The dumpwallet RPC command does not work on signet / testnet
on Jul 18, 2021

ghost commented at 11:37 pm on July 18, 2021: none

I just got this same error on testnet. So it is not related to the network.

I get this error when I use -datadir while running bitcoind and forget to add it in bitcoin-cli command.

Maybe you can try adding -datadir if you have used it while running bitcoind or check other things which maybe the reason why bitcoin-cli is unable to connect with bitcoind if it is running.

lsilva01 commented at 11:48 pm on July 18, 2021: contributor

The reason bitcoin-cli cannot connect to bitcoind is because this exception causes the node to shut down.

I’m running the node with bitcoind -testnet, so it’s getting the default datadir, which is ~/.bitcoin/testnet3. Or ~/.bitcoin/signet for signet.

After the error occurred on testnet, I am unable to dump the wallets again on this network. I delete the bitcoin node directory and cloned it again from Github., I also unloaded and loaded the wallet, but the error persists.

achow101 commented at 0:26 am on July 19, 2021: member

This appears to be a race condition between two RPC handler threads. While one thread is running, another one has begun. Are you using loadwallet to load the wallet? While the wallet is loading, are you running dumpwallet immediately (before loadwallet has returned)? Does the wallet being loaded have unconfirmed transactions in the mempool when you loaded it?

lsilva01 commented at 0:58 am on July 19, 2021: contributor

Are you using loadwallet to load the wallet? Normally, when the node is started (bitcoind -testnet), the wallet is already loaded (it appears in listwallets). I run the dumpwallet command and the node gets this error. I tried loading and unloading the wallet to fix the problem, but it had no effect.
While the wallet is loading, are you running dumpwallet immediately ? No, I wait for the loading / unloading response.
Does the wallet being loaded have unconfirmed transactions in the mempool when you loaded it? No, all transactions have 6 or more confirmations.

lsilva01 commented at 1:03 am on July 19, 2021: contributor

On signet, I have several wallets (legacy type) and this happens to all of them (since the first time I ran dumpwallet).

On testnet I only have one legacy type wallet and this command worked for a while. But it stopped working. There was no activity in the wallet between the period it worked and the period it stopped working.

ghost commented at 1:17 am on July 19, 2021: none

The reason bitcoin-cli cannot connect to bitcoind is because this exception causes the node to shut down.

Interesting. Just noticed you mentioned node stops after running this command. Maybe title of this issue can be changed to

0-The dumpwallet RPC command does not work on signet / testnet
1+The dumpwallet RPC command stops/kills bitcoind

Although I compiled master branch just now and it works fine for me on testnet, regtest and mainnet.

lsilva01 renamed this:
~~The dumpwallet RPC command does not work on signet / testnet~~
The dumpwallet RPC command stops/kills bitcoind
on Jul 19, 2021

lsilva01 commented at 1:42 am on July 19, 2021: contributor

@prayank23 Thanks for suggestion.

This is weird. What is causing this race condition? A few minutes ago I ran the command successfully on testnet, but soon after it stopped working.

achow101 commented at 1:52 am on July 19, 2021: member

Note that this issue is due to the behavior of DEBUG_LOCKORDER so to test, you must have configured with --enable-debug. This is not something that would occur normally (releases are not compiled with debug), but it could cause a deadlock (hence the error). I am able to replicate this on regtest by first creating an unconfirmed transaction (sendtoaddress but don’t do generatetoaddress), then reloading the wallet (either by restarting bitcoind or unloadwallet/loadwallet), and finally dumpwallet.

I’m working on a fix.

lsilva01 commented at 2:25 am on July 19, 2021: contributor

I can confirm that I was able to run dumpwallet without errors after recompiling the node without --enable-debug.

MarcoFalke added the label Tests on Jul 19, 2021

MarcoFalke closed this on Jul 20, 2021

DrahtBot locked this on Aug 18, 2022

The dumpwallet RPC command stops/kills bitcoind #22489