sendrawtransaction takes too long

toolsopen commented at 3:17 pm on October 27, 2023: none

Is there an existing issue for this?

I have searched the existing issues

Current behaviour

tx:https://mempool.space/zh/tx/93cce12faca9fbbc51916852ae7d690769cd4812a0834b9b323defa7a5265f46 sendrawtransaction calling time: Oct 26, 2023 @ 11:37:31.268 bitcoin-conf: listen=1 port=8433 rpcport=8432 addressindex=1 rpcthreads=1000 rpcworkqueue=4000 server=1 txindex=1 coinstatsindex=1 keypool=0 stake=0 staking=0 gen=0 bitcoind logs: 2023-10-26T13:18:36Z UpdateTip: new best=000000000000000000032cf9d0ad42da7dcdeb87ea80a36c8b2406d3b9ebbb95 height=813929 version=0x26f26000 log2_work=94.499899 tx=909456202 date=‘2023-1 0-26T13:17:42Z’ progress=1.000000 cache=121.5MiB(713411txo) 2023-10-26T13:18:36Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications 2023-10-26T13:18:36Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications 2023-10-26T13:18:36Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications 2023-10-26T13:18:36Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications 2023-10-26T13:18:36Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications 2023-10-26T13:18:36Z [main] AddToWallet c4d3b5ddfcd40c3751123a18f48952749acab526f2d0b2754b1dc64331b82c2a update 2023-10-26T13:18:36Z [main] AddToWallet 51e8c45598bcde01c639e8f2a345172e7c8bd33c0c5dc1688c3d1547f4191642 update 2023-10-26T13:21:48Z [main] AddToWallet 93cce12faca9fbbc51916852ae7d690769cd4812a0834b9b323defa7a5265f46 new

Expected behaviour

Expected to return in seconds, but actually took more than 3 minutes

Steps to reproduce

This situation occurs from time to time. Randomly checking the last two log records, it seems that it happened near the block generation log.

Relevant log output

2023-10-26T13:18:36Z UpdateTip: new best=000000000000000000032cf9d0ad42da7dcdeb87ea80a36c8b2406d3b9ebbb95 height=813929 version=0x26f26000 log2_work=94.499899 tx=909456202 date=‘2023-1 0-26T13:17:42Z’ progress=1.000000 cache=121.5MiB(713411txo) 2023-10-26T13:18:36Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications 2023-10-26T13:18:36Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications 2023-10-26T13:18:36Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications 2023-10-26T13:18:36Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications 2023-10-26T13:18:36Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications 2023-10-26T13:18:36Z [main] AddToWallet c4d3b5ddfcd40c3751123a18f48952749acab526f2d0b2754b1dc64331b82c2a update 2023-10-26T13:18:36Z [main] AddToWallet 51e8c45598bcde01c639e8f2a345172e7c8bd33c0c5dc1688c3d1547f4191642 update 2023-10-26T13:21:48Z [main] AddToWallet 93cce12faca9fbbc51916852ae7d690769cd4812a0834b9b323defa7a5265f46 new

How did you obtain Bitcoin Core

Pre-built binaries

What version of Bitcoin Core are you using?

Bitcoin Core 22.0

Operating system and version

Ubuntu 18.04.5 LTS

Machine specifications

OS:Ubuntu 18.04.5 LTS machine:aws m5 2xlarge cpu:8 cores Memory:32GB disk:SSD 1 TB

maflcko commented at 3:28 pm on October 27, 2023: member

rpcthreads=1000 rpcworkqueue=4000

Yes, it is possible that all CPUs are busy if many threads are running, and possibly working.

maflcko commented at 3:28 pm on October 27, 2023: member

Bitcoin Core 22.0

Is this still an issue with a recent version of Bitcoin Core? If yes, what are the steps to reproduce?

toolsopen commented at 5:25 pm on October 28, 2023: none

rpcthreads=1000 rpcworkqueue=4000

Yes, it is possible that all CPUs are busy if many threads are running, and possibly working.

Observe that the machine performance indicators are not full, but the current memory usage is 20GB.

toolsopen commented at 5:31 pm on October 28, 2023: none

Bitcoin Core 22.0

Is this still an issue with a recent version of Bitcoin Core? If yes, what are the steps to reproduce?

Currently I don’t know how to reproduce it, it only happens from time to time in the prod environment.

maflcko commented at 5:08 pm on October 29, 2023: member

Observe that the machine performance indicators are not full, but the current memory usage is 20GB.

How large is your wallet, roughly?

toolsopen commented at 2:30 am on October 30, 2023: none

How large is your wallet, roughly? There are currently about 3,000 utxos available. A large number of transactions have been generated in history, and transaction reception and sending are also very active.

maflcko commented at 8:06 am on October 30, 2023: member

So another reason for the delay may be that the wallet is currently persisted to storage in the background?

toolsopen commented at 9:20 am on October 30, 2023: none

background

Is it related to the fact that the historical transaction data of the wallet is too large? The total number of transactions in our wallet is 1,170,364

maflcko commented at 10:37 am on October 30, 2023: member

What is the approximate size of the BDB wallet? What is the SSD read/write usage around the slowdown?

If this is the cause, I think you can try an sqlite descriptor wallet, whose flush is less expensive on the storage read/write usage.

toolsopen commented at 4:03 am on October 31, 2023: none

What is the approximate size of the BDB wallet? What is the SSD read/write usage around the slowdown?

If this is the cause, I think you can try an sqlite descriptor wallet, whose flush is less expensive on the storage read/write usage.

9.3G Oct 31 03:55 wallet.dat

maflcko commented at 7:52 am on October 31, 2023: member

You’ll have to look for read/write usage in the unit [size in bytes]/[time] when the slow-down occurs.

toolsopen commented at 10:25 am on October 31, 2023: none

One thing I need to state is that all transactions in my wallet are sent and received from the same address, and the read-only wallet mode used is only for broadcast transactions and utxo queries.

toolsopen commented at 10:26 am on October 31, 2023: none

sqlite

I tried to use sqlite, but it was prompted that read-only import addresses are not supported.

toolsopen commented at 3:13 pm on October 31, 2023: none

@maflcko Will wallet.dat 9.3G affect performance? Is there any way to clear the used transactions in it?

willcl-ark commented at 3:21 pm on October 31, 2023: contributor

Hi @toolsopen

Bitcoin Core v22 is outside of maintenance window now, you should probably consider upgrading (to at least 24.2, or better still 25.1)

A few questions:

what do these config values do?

0stake=0
1staking=0
2gen=0

How are you creating the sqlite wallet?

I can create a (new) sqlite descriptor wallet and import a wpkh pubkey on v25.1, after compiling with sqlite like this:

  0will@ubuntu in ~/src/bitcoin on master [$?⇡] : C v16.0.6-clang : 🐍
  1$ /home/will/src/bitcoin/src/bitcoin-cli -regtest -named createwallet wallet_name="toolsopen" disable_private_keys=true blank=true descriptors=true
  2{
  3  "name": "toolsopen"
  4}
  5
  6will@ubuntu in ~/src/bitcoin on master [$?⇡] : C v16.0.6-clang : 🐍
  7$ /home/will/src/bitcoin/src/bitcoin-cli -regtest getdescriptorinfo "wpkh(02f9308a019258c31049344f85f89d5229b531c845836f99b08601f113bce036f9)"
  8{
  9  "descriptor": "wpkh(02f9308a019258c31049344f85f89d5229b531c845836f99b08601f113bce036f9)#8zl0zxma",
 10  "checksum": "8zl0zxma",
 11  "isrange": false,
 12  "issolvable": true,
 13  "hasprivatekeys": false
 14}
 15
 16# import descriptor
 17# NOTE: this will import as inactive so `getnewaddress` on this wallet won't work natively
 18will@ubuntu in ~/src/bitcoin on master [$?⇡] : C v16.0.6-clang : 🐍 
 19$ /home/will/src/bitcoin/src/bitcoin-cli -regtest importdescriptors '[{ "desc": "wpkh(02f9308a019258c31049344f85f89d5229b531c845836f99b08601f113bce036f9)#8zl0zxma", "timestamp": 0, "internal": true  }]'
 20[
 21  {
 22    "success": true
 23  }
 24]
 25
 26# derive the address
 27will@ubuntu in ~/src/bitcoin on master [$?⇡] : C v16.0.6-clang : 🐍 on ☁️  (europe-west1)
 28$ /home/will/src/bitcoin/src/bitcoin-cli -regtest deriveaddresses "wpkh(02f9308a019258c31049344f85f89d5229b531c845836f99b08601f113bce036f9)#8zl0zxma"
 29[
 30  "bcrt1q0ht9tyks4vh7p5p904t340cr9nvahy7uevmqwj"
 31]
 32
 33# load wallet with funds to send from
 34
 35# send coins and check balance
 36will@ubuntu in ~/src/bitcoin on master [$?⇡] : C v16.0.6-clang : 🐍 on ☁️  (europe-west1)
 37$ /home/will/src/bitcoin/src/bitcoin-cli -regtest -rpcwallet=descriptor sendtoaddress bcrt1q0ht9tyks4vh7p5p904t340cr9nvahy7uevmqwj 1
 38dac907862d74fb45afc9dd174794a0132b3a8a59c35e67229d670fb241e4e1e5
 39
 40will@ubuntu in ~/src/bitcoin on master [$?⇡] : C v16.0.6-clang : 🐍 on ☁️  (europe-west1)
 41$ /home/will/src/bitcoin/src/bitcoin-cli -regtest -rpcwallet=descriptor -generate 1
 42{
 43  "address": "bcrt1qf4ps9gxeperfnszz7c25ls9yupmt45d6x9pvea",
 44  "blocks": [
 45    "3a4ae953cac8b899b36293e0f45892bec804588f721e47e5b30529c276fc74b8"
 46  ]
 47}
 48
 49will@ubuntu in ~/src/bitcoin on master [$?⇡] : C v16.0.6-clang : 🐍 on ☁️  (europe-west1)
 50$ /home/will/src/bitcoin/src/bitcoin-cli -regtest -rpcwallet=toolsopen getbalance
 511.00000000
 52
 53will@ubuntu in ~/src/bitcoin on master [$?⇡] : C v16.0.6-clang : 🐍 on ☁️  (europe-west1)
 54$ /home/will/src/bitcoin/src/bitcoin-cli -regtest -rpcwallet=toolsopen listunspent
 55[
 56  {
 57    "txid": "dac907862d74fb45afc9dd174794a0132b3a8a59c35e67229d670fb241e4e1e5",
 58    "vout": 0,
 59    "address": "bcrt1q0ht9tyks4vh7p5p904t340cr9nvahy7uevmqwj",
 60    "scriptPubKey": "00147dd65592d0ab2fe0d0257d571abf032cd9db93dc",
 61    "amount": 1.00000000,
 62    "confirmations": 1,
 63    "spendable": true,
 64    "solvable": true,
 65    "desc": "wpkh([7dd65592]02f9308a019258c31049344f85f89d5229b531c845836f99b08601f113bce036f9)#lps3yegk",
 66    "parent_descs": [
 67      "wpkh(02f9308a019258c31049344f85f89d5229b531c845836f99b08601f113bce036f9)#8zl0zxma"
 68    ],
 69    "safe": true
 70  }
 71]
 72
 73will@ubuntu in ~/src/bitcoin on master [$?⇡] : C v16.0.6-clang : 🐍 on ☁️  (europe-west1)
 74$ /home/will/src/bitcoin/src/bitcoin-cli -regtest -rpcwallet=descriptor getnewaddress
 75bcrt1qqwegjptkfuwqddznkqf8q5968xw83y60zvyqyc
 76
 77will@ubuntu in ~/src/bitcoin on master [$?⇡] : C v16.0.6-clang : 🐍 on ☁️  (europe-west1)
 78$ /home/will/src/bitcoin/src/bitcoin-cli -regtest -rpcwallet=toolsopen createrawtransaction "[{\"txid\":\"dac907862d74fb45afc9dd174794a0132b3a8a59c35e67229d670fb241e4e1e5\",\"vout\":0}]" "[{\"bcrt1qqwegjptkfuwqddznkqf8q5968xw83y60zvyqyc\":0.01}]"
 790200000001e5e1e441b20f679d22675ec3598a3a2b13a0944717ddc9af45fb742d8607c9da0000000000fdffffff0140420f000000000016001403b28905764f1c06b453b0127050ba399c78934f00000000
 80
 81will@ubuntu in ~/src/bitcoin on master [$?⇡] : C v16.0.6-clang : 🐍 on ☁️  (europe-west1)
 82$ /home/will/src/bitcoin/src/bitcoin-cli -regtest -rpcwallet=toolsopen decoderawtransaction 0200000001e5e1e441b20f679d22675ec3598a3a2b13a0944717ddc9af45fb742d8607c9da0000000000fdffffff0140420f000000000016001403b28905764f1c06b453b0127050ba399c78934f00000000
 83{
 84  "txid": "e39f32d1b266b0c7504cd22b4fec445a7aae7e99a80ef4b87f8339f128169152",
 85  "hash": "e39f32d1b266b0c7504cd22b4fec445a7aae7e99a80ef4b87f8339f128169152",
 86  "version": 2,
 87  "size": 82,
 88  "vsize": 82,
 89  "weight": 328,
 90  "locktime": 0,
 91  "vin": [
 92    {
 93      "txid": "dac907862d74fb45afc9dd174794a0132b3a8a59c35e67229d670fb241e4e1e5",
 94      "vout": 0,
 95      "scriptSig": {
 96        "asm": "",
 97        "hex": ""
 98      },
 99      "sequence": 4294967293
100    }
101  ],
102  "vout": [
103    {
104      "value": 0.01000000,
105      "n": 0,
106      "scriptPubKey": {
107        "asm": "0 03b28905764f1c06b453b0127050ba399c78934f",
108        "desc": "addr(bcrt1qqwegjptkfuwqddznkqf8q5968xw83y60zvyqyc)#quh7r7my",
109        "hex": "001403b28905764f1c06b453b0127050ba399c78934f",
110        "address": "bcrt1qqwegjptkfuwqddznkqf8q5968xw83y60zvyqyc",
111        "type": "witness_v0_keyhash"
112      }
113    }
114  ]
115}

This will rescan the blockchain for historical transactions but you would lose transaction metadata. There does exist a migration tool which should be released in v26.0 which may also handle metadata too, if this is important to you (unsure about this).

@maflcko Will wallet.dat 9.3G affect performance? Is there any way to clear the used transactions in it?

If you import the descriptor as above and use "timestamp": "now" then it won’t rescan for historical transactions. Although if any time in the future you happened to rescan your chain, these would be picked back up by the wallet. But AFAIK you can’t “clear” them from the wallet.

You could create a new wallet with a new address and use that? Each transaction should use a unique address. Address re-use is terrible for sender and receiver privacy.

achow101 commented at 3:34 pm on October 31, 2023: member

Will wallet.dat 9.3G affect performance?

Yes, significantly.

Is there any way to clear the used transactions in it?

Not automatically. However you can use removeprunedfunds and delete old transactions manually.

toolsopen commented at 7:22 pm on October 31, 2023: none

If you import the descriptor as above and use "timestamp": "now" then it won’t rescan for historical transactions. Although if any time in the future you happened to rescan your chain, these would be picked back up by the wallet. But AFAIK you can’t “clear” them from the wallet.

1.is to turn off some mining settings 2.The following versions and commands were used for testing

https://bitcoincore.org/bin/bitcoin-core-25.1/bitcoin-25.1-x86_64-linux-gnu.tar.gz curl -X POST ‘http://test:test@127.0.0.1:8432’ -H ‘Content-Type: application/json’ -d ‘{“jsonrpc”:“2.0”,“method”:“createwallet”,“params”:[“main”,true,true,null,false,true,true],“id”:“1”}’ curl -X POST ‘http://test:test@127.0.0.1:8432’ -H ‘Content-Type: application/json’ -d ‘{“jsonrpc”:“2.0”,“method”:“importaddress”,“params”:[ “bc1qmxxxxxxxxxxxxxxxxx”, “main”, true],“id”:“1”}’

One address is easier to maintain

toolsopen commented at 7:23 pm on October 31, 2023: none

Will wallet.dat 9.3G affect performance?

Yes, significantly.

Is there any way to clear the used transactions in it?

Not automatically. However you can use removeprunedfunds and delete old transactions manually.

This helps, let me write a script and try it out

toolsopen commented at 3:15 am on November 1, 2023: none

#12115 @achow101 @maflcko @willcl-ark @gregwebs I saw that this issue was raised in 2018, but no attention was paid to it

maflcko commented at 7:00 am on November 1, 2023: member

Again, it is hard or impossible to fix your problem if it is wasn’t properly diagnosed. The delay could be entirely unrelated to the wallet.

However, if the delay is caused by storage speed limitations in combination with the BDB flush behavior, a solution could be to migrate your wallet to sqlite. The (experimental) migratewallet RPC is only available in 25.x or the upcoming 26.x.

toolsopen commented at 12:45 pm on November 21, 2023: none

@maflcko @achow101 @willcl-ark @gregwebs After the read-only wallet was reset and utxo was selectively imported, the performance was greatly improved and the problem was solved.

toolsopen commented at 12:46 pm on November 21, 2023: none

This proves that it would be a very good way if bitcoind can self-isolate expired transactions.

willcl-ark commented at 12:53 pm on November 21, 2023: contributor

Thanks for the update. Wallet performance is something that’s being continually worked on.

Did you end up upgrading versions (too), or still running on V22?

maflcko added the label Wallet on Nov 21, 2023

maflcko added the label Resource usage on Nov 21, 2023

maflcko commented at 1:12 pm on November 21, 2023: member

Assuming this is a bdb wallet (can be checked via getwalletinfo()[format]), I don’t think it is worth to spend time on optimizing the code, given that it will be removed in the future.

toolsopen commented at 2:15 pm on November 21, 2023: none

Thanks for the update. Wallet performance is something that’s being continually worked on.

Did you end up upgrading versions (too), or still running on V22?

I also updated the final version to 25.1. It must have nothing to do with the version update at that time. I tried the complete rescan wallet to compare it.

toolsopen commented at 2:18 pm on November 21, 2023: none

Assuming this is a bdb wallet (can be checked via getwalletinfo()[format]), I don’t think it is worth to spend time on optimizing the code, given that it will be removed in the future.

I use it for utxo maintenance and transaction broadcasting, but an overly large wallet will definitely have a great impact on performance.

maflcko commented at 2:22 pm on November 21, 2023: member

What does the getwalletinfo RPC return in the format field, for you?

maflcko commented at 6:35 pm on November 30, 2023: member

Not sure what the status here is, and what is left to be done. Can this be closed?

toolsopen closed this on Dec 8, 2023

sendrawtransaction takes too long #28745