Bitcoind crashed, how to debug

alex-lt-kong commented at 3:35 pm on February 24, 2023: none

OS: Debian GNU/Linux 11 (bullseye) Available RAM: ~3GB

I run bitcoind on the machine by ~/bin/bitcoin-22.0/bin/bitcoind -daemon as root (as this is a VM dedicated to bitcoind, no other ordinary users are added). I make local RPC calls relatively frequently (up to 10k calls per sec) but only sequentially (i.e., the 2nd RPC call is only made if the first RPC call returned).

It usually works fine for a few days/weeks, then the process just disappears. The log is like the following (the 2nd part of the log is written by the new bitcoind process):

 02023-02-23T07:21:49Z UpdateTip: new best=00000000000000000005260d63755c13802609649b30886a021edc108f724294 height=777918 version=0x2c8d2000 log2_work=94.020512 tx=807945114 date='2023-02-23T07:21:10Z' progress=1.000000 cache=125.0MiB(896432txo)
 12023-02-23T07:24:36Z New outbound peer connected: version: 70016, blocks=777918, peer=386 (block-relay-only)
 22023-02-23T07:26:04Z UpdateTip: new best=0000000000000000000194297149fa2e2a5a8003335ec1f05bd5826e49286203 height=777919 version=0x2ba92000 log2_work=94.020524 tx=807946053 date='2023-02-23T07:25:35Z' progress=1.000000 cache=125.2MiB(898284txo)
 32023-02-23T07:26:26Z UpdateTip: new best=00000000000000000002bd51f8197b30b21799f42ac7bd305a5114c486bc6699 height=777920 version=0x2aaa0000 log2_work=94.020536 tx=807946317 date='2023-02-23T07:26:04Z' progress=1.000000 cache=125.3MiB(898554txo)
 42023-02-23T07:27:32Z UpdateTip: new best=00000000000000000004d9729560ddab94371c2d357d9b6e9eb9bdcccd7ef6ba height=777921 version=0x20400000 log2_work=94.020548 tx=807946689 date='2023-02-23T07:27:06Z' progress=1.000000 cache=125.3MiB(898977txo)
 52023-02-23T07:28:22Z New outbound peer connected: version: 70016, blocks=777921, peer=387 (block-relay-only)
 62023-02-23T07:28:53Z UpdateTip: new best=000000000000000000043a08e446a1493a97472fe2f110d8479b7107584b8ec5 height=777922 version=0x2456e000 log2_work=94.020560 tx=807947060 date='2023-02-23T07:28:33Z' progress=1.000000 cache=125.4MiB(899783txo)
 72023-02-23T07:33:13Z UpdateTip: new best=0000000000000000000559b6d6e4f6e0b023f989b9049762f40a3efe622832e1 height=777923 version=0x20084000 log2_work=94.020572 tx=807947934 date='2023-02-23T07:32:56Z' progress=1.000000 cache=126.0MiB(904538txo)
 82023-02-23T07:45:29Z UpdateTip: new best=000000000000000000033d0f15e488c5205370c9ded96f3fdb40ac65238db6ad height=777924 version=0x2001c000 log2_work=94.020585 tx=807949972 date='2023-02-23T07:45:25Z' progress=1.000000 cache=126.8MiB(911094txo)
 92023-02-23T07:46:02Z UpdateTip: new best=00000000000000000005cbd091c3a153f2bcd918f9d08306a76c0ca93c105f8b height=777925 version=0x20000000 log2_work=94.020597 tx=807950138 date='2023-02-23T07:45:46Z' progress=1.000000 cache=126.8MiB(910923txo)
102023-02-23T07:48:33Z UpdateTip: new best=0000000000000000000436a430222ee553d9898c8f7f113f337372f6df2d0bb2 height=777926 version=0x20e00000 log2_work=94.020609 tx=807950852 date='2023-02-23T07:48:18Z' progress=1.000000 cache=127.0MiB(912707txo)
112023-02-23T07:53:04Z New outbound peer connected: version: 70016, blocks=777926, peer=388 (block-relay-only)
12
13
14
15
16
17
18
19
20
21
222023-02-24T15:29:56Z Bitcoin Core version v22.0.0 (release build)
232023-02-24T15:29:56Z Assuming ancestors of block 00000000000000000008a89e854d57e5667df88f1cdef6fde2fbca1de5b639ad have valid signatures.
242023-02-24T15:29:56Z Setting nMinimumChainWork=00000000000000000000000000000000000000001fa4663bbbe19f82de910280
252023-02-24T15:29:56Z Using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation
262023-02-24T15:29:56Z Using RdSeed as additional entropy source
272023-02-24T15:29:56Z Using RdRand as an additional entropy source
282023-02-24T15:29:56Z Default data directory /root/.bitcoin
292023-02-24T15:29:56Z Using data directory /mnt/bitcoin
302023-02-24T15:29:56Z Config file: /mnt/bitcoin/bitcoin.conf (not found, skipping)
312023-02-24T15:29:56Z Config file arg: datadir="/mnt/bitcoin/"
322023-02-24T15:29:56Z Config file arg: rpcpassword=****
332023-02-24T15:29:56Z Config file arg: rpcuser=****
342023-02-24T15:29:56Z Config file arg: txindex="1"
352023-02-24T15:29:56Z Config file arg: walletdir="/mnt/bitcoin/wallets/ak_wallet/"
362023-02-24T15:29:56Z Command-line arg: daemon=""
372023-02-24T15:29:56Z Using at most 125 automatic connections (1024 file descriptors available)
382023-02-24T15:29:56Z Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
392023-02-24T15:29:56Z Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
402023-02-24T15:29:56Z Script verification uses 1 additional threads

Is there anyway for me to get debug info etc which could hopefully give me some hints on what exactly happened?

alex-lt-kong added the label Bug on Feb 24, 2023

maflcko commented at 3:43 pm on February 24, 2023: member

Did you check for OOM?

fanquake commented at 3:43 pm on February 24, 2023: member

Going to assume this is related to lack-of-ram/a memory leak. You could run with -debug to produce more logs. You could also upgrade to 24.0.1, which may have fixed whichever bug you might be seeing.

alex-lt-kong commented at 4:21 pm on February 24, 2023: none

Going to assume this is related to lack-of-ram/a memory leak. You could run with -debug to produce more logs. You could also upgrade to 24.0.1, which may have fixed whichever bug you might be seeing.

I always download Bitcoin Core from https://bitcoin.org/en/download, is this considered the “official” site? On this site the latest version is 22.0 so I am still using 22.0.

alex-lt-kong commented at 4:23 pm on February 24, 2023: none

Did you check for OOM?

I thought about this, but considering that the requirements page shows that 1GB is enough, I did not drill down this possibility. Let me increase the available memory from 3GB to 5GB and see if the issue persists. Btw, is bitcoin.org considered the “official” site? Is this 1GB requirement still valid?

fanquake commented at 4:24 pm on February 24, 2023: member

I always download Bitcoin Core from https://bitcoin.org/en/download, is this considered the “official” site?

The website for the Bitcoin Core project is https://bitcoincore.org/en/download/. bitcoin.org currently lists an outdated version of Bitcoin Core for download.

alex-lt-kong commented at 4:28 pm on February 24, 2023: none

I always download Bitcoin Core from https://bitcoin.org/en/download, is this considered the “official” site?

The website for the Bitcoin Core project is https://bitcoincore.org/en/download/. bitcoin.org currently lists an outdated version of Bitcoin Core for download.

Okay then. Let me close the issue, upgrade to 24.0 and increase available RAM. See if the problem could be solved.

alex-lt-kong closed this on Feb 24, 2023

fanquake removed the label Bug on Feb 24, 2023

fanquake added the label Questions and Help on Feb 24, 2023

bitcoin locked this on Feb 24, 2024

Bitcoind crashed, how to debug #27156