large wallet: Bitcoind freezes #15148

<!-- This issue tracker is only for technical issues related to Bitcoin Core. General bitcoin questions and/or support requests are best directed to the Bitcoin StackExchange at https://bitcoin.stackexchange.com. For reporting security issues, please read instructions at https://bitcoincore.org/en/contact/. If the node is "stuck" during sync or giving "block checksum mismatch" errors, please ensure your hardware is stable by running memtest and observe CPU temperature with a load-test tool such as linpack before creating an issue! -->

<!-- Describe the issue -->

<!--- What behavior did you expect? -->

We are experiencing the following node failure on one of our big wallets (about 300k addresses) from time to time: The node stops to do any I/O operations, but the bitcoind process still lives on. The node does not recover after some time. We are not sure that the constant reboots are a way to resolve this issue and are looking for help.

<!--- What was the actual behavior (provide screenshots if the issue is GUI-related)? -->

<!--- How reliably can you reproduce the issue, what are the steps to do so? -->

The issue occurs from time to time with no real steps to reproduce. We are currently unaware on how to reproduce the issue on our own.

<!-- What version of Bitcoin Core are you using, where did you get it (website, self-compiled, etc)? -->

Bitcoin Core 0.16.3 (bitcoin-0.16.3-x86_64-linux-gnu), grabbed from https://bitcoincore.org/bin/bitcoin-core-0.16.3/

<!-- What type of machine are you observing the error on (OS/CPU and disk type)? -->

Ubuntu 16.04 instance with 8 vCPU, 16Gb RAM and SSD. The machine also runs some other crypto wallets, however, disk is not overloaded with activity and handles (or at least it seems so) just fine.

<!-- For the GUI-related issue on Linux provide names and versions of a distro, a desktop environment and a graphical shell (if relevant). -->

<!-- Any extra information that might be useful in the debugging process. -->

<!--- This is normally the contents of a `debug.log` or `config.log` file. Raw text or a link to a pastebin type site are preferred. -->

We are doing quite a bunch of RPC requests on this node. Something about 25 requests per second on average, mostly getRawTransaction and validateAddress. Is this an OK load for bitcoind? Are there any suggestions on how to tweak RPC or related node params? I haven't seen any specific suggestions on the Internet except for "max them out and pray that your machine handles it"

Possibly related to #15015

@RomanBelkov some questions:

• does the RPC interface stops responding?
• I assume you have -txindex?
• Have you tweaked -rpcthreads and rpcworkqueue?
• How big is the wallet? (transaction count, address count)

If it happens again please dump the process threads.

@promag

• We receive timeouts (no answer for 60 seconds) for all RPC calls at some point of time. After 4-5 minutes from first timed out request bitcoind stops any disk I/O.
• No, actually we do not use txindex.
• I have tweaked the -rpcthreads in the range from 8 to 64, but was unable to find any effect on freezes. I have never tweaked -rpcworkqueue, because I failed to find any good pieces of advice on how to tune this parameter.
• Can you please give some advice on how to track these values? I have only the rough estimate of 300.000 addresses.

Sorry if this is a newbie question, but how do you dump the process threads for bitcoind? If there is a special guide for fine tuning/debugging the node, can you please share it with me so I won't ask any basic questions?

bitcoind stops any disk I/O.

How do you check that? And what happens with new calls? Are they rejected or they also timeout?

we do not use txindex

It doesn't matter for validateaddress but calls to getrawtransaction are serialized, in other words, you don't have concurrency there. And your wallet might be big enough to cause slow getrawtransactions.

how to track these values

You can call getwalletinfo to see transaction count. You can run dumpwallet to see how many keys you have.

dump the process threads for bitcoind

Try ps -T -p <pid> to view process threads. You could also enable -debug and check debug.log.

I'd be surprise if your node is deadlock, but it's hard to tell from your feedback. Have you tried running a newer version?

How do you check that? And what happens with new calls? Are they rejected or they also timeout?

I believe that we use some collectd monitoring for this node right now and it shows clearly that no I/O is performed by the process. If we check the debug log (we started to run it with -debug=1), the last messages would be something like

2019-01-25 13:55:33 ping timeout: 1200.021239s
2019-01-25 13:55:33 disconnecting peer=3
2019-01-25 14:40:03
-----a considerable amount of newlines-----
2019-01-25 14:40:03 Bitcoin Core version v0.16.3 (release build)

All the RPC requests are timed out as soon as we hit the freeze.

Regarding the validateaddress/getrawtransaction -- we've introduced the caches on our side to reduce the load on the node and it did not help.

Transaction count is 230160. I am yet unable to perform dumpwallet due to various reasons for this wallet. However, I've tried running the listaddressgroupings RPC method and received a node freeze. The node thread dump after I/O stoppage:

PID  SPID TTY          TIME CMD
 9603  9603 ?        00:00:33 bitcoind
 9603  9604 ?        00:00:28 bitcoin-scriptc
 9603  9605 ?        00:00:28 bitcoin-scriptc
 9603  9606 ?        00:00:28 bitcoin-scriptc
 9603  9607 ?        00:04:50 bitcoin-schedul
 9603  9608 ?        00:00:19 bitcoin-http
 9603  9609 ?        00:17:44 bitcoin-httpwor
 9603  9610 ?        00:14:01 bitcoin-httpwor
 9603  9611 ?        00:12:45 bitcoin-httpwor
 9603  9612 ?        00:14:01 bitcoin-httpwor
 9603  9613 ?        00:24:09 bitcoin-httpwor
 9603  9614 ?        00:15:27 bitcoin-httpwor
 9603  9615 ?        00:14:23 bitcoin-httpwor
 9603  9616 ?        00:14:58 bitcoin-httpwor
 9603  9617 ?        00:13:25 bitcoin-httpwor
 9603  9618 ?        00:13:34 bitcoin-httpwor
 9603  9619 ?        00:15:15 bitcoin-httpwor
 9603  9620 ?        00:13:24 bitcoin-httpwor
 9603  9621 ?        00:15:10 bitcoin-httpwor
 9603  9622 ?        00:14:34 bitcoin-httpwor
 9603  9623 ?        00:15:52 bitcoin-httpwor
 9603  9624 ?        00:13:47 bitcoin-httpwor
 9603  9836 ?        00:00:39 bitcoind
 9603 10578 ?        00:00:00 bitcoin-torcont
 9603 10585 ?        00:00:18 bitcoin-net
 9603 10587 ?        00:00:00 bitcoin-addcon
 9603 10588 ?        00:00:01 bitcoin-opencon
 9603 10589 ?        00:01:38 bitcoin-msghand

@promag Here also is the thread dump when the node freezes 'naturally'.

PID  SPID TTY          TIME CMD
1858  1858 ?        00:00:38 bitcoind
1858  1859 ?        00:00:44 bitcoin-scriptc
1858  1860 ?        00:00:44 bitcoin-scriptc
1858  1861 ?        00:00:44 bitcoin-scriptc
1858  1862 ?        00:07:16 bitcoin-schedul
1858  1863 ?        00:00:40 bitcoin-http
1858  1864 ?        00:30:45 bitcoin-httpwor
1858  1865 ?        00:28:41 bitcoin-httpwor
1858  1866 ?        00:31:43 bitcoin-httpwor
1858  1867 ?        00:30:00 bitcoin-httpwor
1858  1868 ?        00:35:15 bitcoin-httpwor
1858  1869 ?        00:31:24 bitcoin-httpwor
1858  1870 ?        00:33:17 bitcoin-httpwor
1858  1871 ?        00:38:23 bitcoin-httpwor
1858  1872 ?        00:35:16 bitcoin-httpwor
1858  1873 ?        00:34:52 bitcoin-httpwor
1858  1874 ?        00:30:29 bitcoin-httpwor
1858  1875 ?        00:32:15 bitcoin-httpwor
1858  1876 ?        00:36:47 bitcoin-httpwor
1858  1877 ?        00:33:09 bitcoin-httpwor
1858  1878 ?        00:28:32 bitcoin-httpwor
1858  1879 ?        00:30:34 bitcoin-httpwor
1858  2159 ?        00:00:00 bitcoind
1858  3907 ?        00:00:00 bitcoin-torcont
1858  3911 ?        00:00:46 bitcoin-net
1858  3913 ?        00:00:00 bitcoin-addcon
1858  3914 ?        00:00:02 bitcoin-opencon
1858  3915 ?        00:03:30 bitcoin-msghand

I have also found out that calling listaddressgroupings during our usual node load always causes node to freeze.

Have you tried running a newer version?

No, we have not, unfortunately.

@promag what would be the next logical step to overcome the issue? Trying to upgrade to 0.17.x?

same issue on testnet mining pool. it has large txs and qt stops if its updating balances...

Hello, Just in case if someone wondered: At first, I have upgraded the node to 0.17 and it did not help. I ended up cleaning the wallet on this node (removed the change addresses) and was able to reduce the amount of freezes. They still occur from time to time, however.

If someone has any performance tuning suggestions/links/comments, I would be very glad to receive these pieces of advice.

same issue on testnet mining pool. it has large txs and qt stops if its updating balances...

There are some open pulls to improve bitcoin-qt in that regard. But @RomanBelkov is not referring to bitcoin-qt, I think? @RomanBelkov to be clear, the freeze happens during some RPC? Are you able to reproduce while running in the debugger? The above debug.log doesn't help at all.

@promag I refer to bitcoind only. Yes, the freezes did happen after some of the RPCs. We did not run the bitcoind in the debugger, unfortunately. After cleaning the wallet I am getting a decent run from June of 2019 with freezes occuring approximately once/twice a month.

@RomanBelkov is this issue still present on the current release of Bitcoin Core (v24.0.1), as there have been many wallet improvements in the years since this issue was opened?

@willcl-ark unfortunately I'm unable to comment on issue as I'm not working with Bitcoin Core anymore for quite a while. I guess the issue can be closed as there was no development/reports from other people for 3 years :)