getnewaddress very slow to respond: python-bitcoinrpc often times-out #7486

Bitcoin 0.11 on a Raspberry Pi 2

From the command line, bitcoin-cli getnewaddress can take up to 1 minute to respond.

When using jgarzik's python bitcoin RPC module to make this RPC call, it often times out.

Sometimes after it times-out on getnewaddress, even stranger behavior follows: getmempoolinfo reports NO INCREASE in mempool size until the original script that timed-out is interrupted and killed. Even when requesting bitcoin-cli getmempoolinfo from a separate terminal session, without the python RPC module.

bitcoin-cli dumpwallet still responds immediately. Wallet is unencrypted, only 100 keys, only 327kB

Can you try with 0.12?

Also is this while catching up with the chain? If so, due to slower validation on those hardware, it happens that the main lock is sometimes contended for a while, making it respond to RPC commands very slowly.

Node is fully caught up but we are talking about a Raspberry Pi here, the blockchain and wallet.dat are stored on the same USB stick, so I wonder if its an i/o bottleneck?

I will upgrade to 0.12 and try tonight. Any reason that version should be any faster?

An USB stick is terribly slow for this, that is probably the bottleneck. You'll also wear it out quite quickly.

0.12 has an improved HTTP/RPC server, as well as faster block validation. It may make a difference though in your case I think it makes more sense to blame the hardware.

OK I'll try with a small USB SSD drive instead of the flash drive. Do you have any idea what is actually taking so long though? dumpwallet seems to open the wallet.dat file very quickly

Dumpwallet actually doesn't touch the wallet file at all, as all keys (potentially in encrypted form) are kept in memory.

Getnewaddress does touch it though, as it needs to generate a new key to replenish the key pool, and make sure it's secure on disk.

Ahhhh ok I was wondering if the lag was due to generating a key, however many EC-adds are required, but I thought getnewaddress just grabbed a key from the existing wallet. So yes, it does, but then also generates a new key every time to keep the pool at 100 or whatever?

Computing a new key should be in the order of milliseconds, though, and a wallet of a few 100kB should take a minute to write...

@pinheadmz getnewaddress triggers a flush of the wallet which causes bdb to issue a number of fsync calls which trigger a flush of the entire write cache

the problem is almost certainly just very very slow io from using a flash drive.

If all keys are already in memory, is there an RPC call that can return a receiving address without having to read or write from disk?

@pinheadmz no, the getnewaddress command has to mark the key as being not in the keypool anymore no matter what

@laanwj I upgraded to v0.12.0rc3 on the Raspberry Pi.

It does seem like block validation is going much faster! Especially during catch-up. getnewaddress via RPC still takes a long time and will frequently time-out bitcoinrpc. Sometimes I can get it on a streak and get a few successful requests in a row, then more time-outs.

I am using your revision to jgarzik's bitcoinrpc, I modified these lines: https://github.com/bitcoin/bitcoin/blob/v0.12.0rc3/qa/rpc-tests/test_framework/authproxy.py#L110-L127 ...to also catch socket.timeout and httplib.CannotSendRequest, and print to the console that a timeout has occurred: https://github.com/pinheadmz/ClockBlocker/blob/master/bitcoinrpc.py#L111-L126

Sometimes, the timeout-reload process works and I get a response on the "first" retry. However, sometimes the second retry also times out and fails. In this case, the function does NOT retry a second time and execution fails. Any thoughts on how to make this try/except infinite retry-until-satisfied?

...to also catch socket.timeout and httplib.CannotSendRequest, and print to the console that a timeout has occurred: https://github.com/pinheadmz/ClockBlocker/blob/master/bitcoinrpc.py#L111-L126

This logic is not there to catch timeouts, but to handle the case where the http connection has been disconnected (the server was waiting too long for a new command).

The solution here would be to disable the client-side timeout, or set it to something like 3 hours. The server won't timeout while it is busy processing a request so that shouldn't be a problem.

Ok I fixed this problem by lowering the priority on other processes running on the computer:

http://raspberrypi.stackexchange.com/q/42474/39301

I'm still not sure why getmempoolinfo wasn't impacted as badly as getnewaddress but this issue can be closed. Thanks guys!

Is that the right link?

@cloudnthings yes, turns out my answer was somewhat unrelated to bitcoin core. I am running a python script that plays with a RGB LED grid, and that process was apparently high priority, leaving fewer cycles for bitcoind

add keypoolsize and try to keypoolrefill , it work from my server and you can use iotop to show the i/o bottleneck