Bitcoin Core 0.15.0 crashes repeatedly while rebuilding local blockchain copy, no error reported #11356

What follows is a report of unexplained crashes of the new Bitcoin Core 0.15. The available information may not be enough to identify the cause but perhaps other users can recognize identical behavior and use this ticket to acknowledge.

In case the kind dev team wants to investigate the issue further, I will cooperate fully. Thanks for your time to whoever reads this.

Describe the issue

I run a Bitcoin Core node 0.14.2 node under Ubuntu 17.04. I decided to upgrade to the new 0.15 version as soon as it became possible to do so via apt upgrade.

It is important to note that the node was fully synced and running fine before the update. It is an old piece of hardware with a brand new HDD added, and a fresh copy of the Ubuntu OS installed & configured just for the purpose of running Bitcoin Core.

The first time I launched bitcoin-qt 0.15, it said a DB upgrade is necessary. Shortly afterwards it reported that the database was corrupt and the local copy of the blockchain needed to be rebuilt (I quote from memory, sadly I did not write down the exact message).

The program then proceeded to process the blocks on disk. Since I am running the node on underpowered hardware (full specs below) this is a slow process: 60+ hours and counting; approx. 75% done.

During this update/resync process, the bitcoin software has crashed at least 4 times so far without any apparent error message.

debug.log contents (attached below) look normal: long sets of UpdateTip: new best= messages for all processed blocks interspersed with service messages like Pre-allocating up to position xxxx in xxxx.dat and handshakes with other nodes.

Can you reliably reproduce the issue?

Sadly, no. Crashes may happen after more than 24 hours of work or as little as 12 hours after restarting the process.

The NUC is dedicated to this task only; it operates on its own and I only keep a remote shell open that runs a tail -f debug.log that I look at a couple of times a day.

bitcoin.conf contents:

onlynet=ipv4
server=1
rpcbind=127.0.0.1
rcpbind=10.9.9.46 // This is my inhouse VPN
rpcport=8332
rpcuser=[redacted]
rpcpassword=[redacted]

Besides the above configuration, I have increased the size of the database cache from the default value to 1024MB, using the bitcoin-qt settings panel. Memory usage is stable at 2GB out of 3.7GB usable, and looking for out of memory crashes using grep "Killed process" /var/log/syslog yields no results.

What version of bitcoin-core are you using?

I am using the package available from the ppa:bitcoin/bitcoin repository Version string is v0.15.0.0-g3751912e8e

Machine specs:

• OS: Ubuntu 17.04 (4.10.0-33-generic Ubuntu SMP Fri Aug 11 10:55:28 UTC 2017 x86_64 GNU/Linux)
• CPU: Intel Celeron N2820
• RAM: 4GB (3.7GB available)
• SWAP: 8GB
• Disk size: 1.0TB (850 GB available)
• Disk Type: HDD

Any extra information that might be useful in the debugging process.

The contents of debug.log are provided as follows:

• The startup part following a crash is attached as plaintext file. debug.log.txt
• The full log (10.8MB uncompressed) is attached as zip file. debug.log.zip

db.log is empty. I know of no other logs related to this software, but will post any info upon request.

Is it crashing immediately after the splashscreen goes away or later after the main window has appeared?

Can you please provide the contents of this file: ~/.config/Bitcoin/Bitcoin-Qt.conf?

This may be related to #11171

@achow101 I think they are not using the gui

@achow101 I think they are not using the gui

I think they are. I see lines in their debug.log for the GUI, e.g. GUI: Platform customization: "other"

Is it crashing immediately after the splashscreen goes away or later after the main window has appeared? @achow101 The software crashes much, much later than startup (many hours after restart). Behavior is not matching what is described in #11171

Contents of ~/.config/Bitcoin/Bitcoin-Qt.conf as requested:

[General]
strDataDir=/home/bitcoin/.bitcoin
fReset=false
nSettingsVersion=150000
fRestartRequired=false
fHideTrayIcon=false
fMinimizeToTray=true
fMinimizeOnClose=true
nDisplayUnit=0
strThirdPartyTxUrls=
fCoinControlFeatures=true
nDatabaseCache=1024
nThreadsScriptVerif=0
bSpendZeroConfChange=true
fUseUPnP=false
fListen=true
fUseProxy=false
addrProxy=127.0.0.1:9050
fUseSeparateProxyTor=false
addrSeparateProxyTor=127.0.0.1:9050
language=en
fFeeSectionMinimized=true
nFeeRadio=0
nCustomFeeRadio=0
nTransactionFee=0
fPayOnlyMinFee=false
nWindowPos=@Point(42 26)
nWindowSize=@Size(850 550)
nRPCConsoleWindowPos=@Point(128 376)
nRPCConsoleWindowSize=@Size(740 430)
nConfTarget=6
nSmartFeeSliderPosition=0

@drkskwlkr If you run Bitcoin Core from the terminal, what is the message that it gets when it crashes? Can you also have it create a core dump file and debug that?

If you run Bitcoin Core from the terminal, what is the message that it gets when it crashes?

I have never run Bitcoin Core as daemon so far, I just did that for the first time. It did not crash at startup (and I wouldn't expect it to). It just went on its business verifying blocks. I will leave it running like that and see if it would crash again at some point like the GUI version did.

As for creating core dump files, I haven't done that before (I know Linux just enough to run my webserver). But I checked /var/crash just now and there is a 100k crash file associated with bitcoin-qt from the last crash (see attached). Does it provide any meaningful information?

_usr_bin_bitcoin-qt.1000.crash.txt

Sadly the crash file provided isn't very useful (just indicates SEGFAULT). If possible, it would be of great help for you to install gdb (via apt), as well as the debug symbols from the launchpad build at https://launchpad.net/~bitcoin/+archive/ubuntu/bitcoin/+files/bitcoin-qt-dbgsym_0.15.0-zesty9_amd64.ddeb (installed as you would any other .deb package).

Then in termainal run gdb /usr/bin/bitcoin-qt, when it finishes loading, type "run" and hit enter. Eventually, when it crashes, type "thread apply all bt", hit enter, and paste the output of what it displays there. This should tell us exactly where the issue was.

@TheBlueMatt: Hi, thank you for the detailed directions and sorry for not responding earlier.

Before I was able to do as you suggest, my node managed to fully resync the blockchain and is churning along nicely. However it did manage to crash one more time after reaching 100%.

I am now running bitcoin-qt via gdb as instructed and will update the thread if/when another crash occurs.

If nothing happens for a couple of days, and if I am able to figure out how to force my node to rebuild the blocks from the last 6-12 months, I will do that in order to provoke the same conditions under which the initial crashes were observed.

@drkskwlkr Any followup?

@gmaxwell Hi Gregory - sorry, no. The node has been working reliably 24/7 since my report. As nobody else seems to have this issue, I think it'd be best to close the ticket.