Node continues running after LevelDB throws an error

gmaxwell commented at 6:56 pm on December 11, 2014: contributor

I observed a node w/ git master at continuing to run after encountering a leveldb error:

2014-12-11 03:48:57 LevelDB read failure: Corruption: block checksum mismatch 2014-12-11 03:48:57 Corruption: block checksum mismatch

EXCEPTION: 13leveldb_error
Database corrupted
bitcoin in ProcessMessages() … 2014-12-11 03:48:57 ProcessMessage(block, 162248 bytes) FAILED peer=1 2014-12-11 03:48:58 UpdateTip: new best=0000000000000000053cc9e138c09d9923bab3c1274bbbab762a2f5d5512c767 height=326925 log2_work=81.217819 tx=49707296 date=2014-10-25 12:21:19 progress=0.879732 cache=66628

We should shut down immediately after database errors to avoid a repeat of the pre-0.8 forking behavior on BDB-error. If an “impossible” database error happens, better to assume we’re busted and shut down than potentially start creating a chain fork.

(Looking into it, just opening the issue so this isn’t forgotten. Should be fixed before release if at all possible.)

gmaxwell assigned gmaxwell on Dec 11, 2014

gmaxwell added this to the milestone 0.10.0 on Dec 11, 2014

eN0Rm commented at 9:56 pm on January 4, 2015: none

I get the same:

015-01-04 20:57:51 UpdateTip: new best=0000000000000000070feeb3c5396aba0a71478823f2dd296029b8c2ad758ad8 height=314449 log2_work=80.082734 tx=44076572 date=2014-08-07 21:28:57 progress=0.622530 cache=42687 2015-01-04 20:57:52 UpdateTip: new best=000000000000000001955d4aac3a5c52695fb08fd937b6ea66b0cbcbe9b4f92c height=314450 log2_work=80.082825 tx=44077246 date=2014-08-07 21:36:00 progress=0.622549 cache=43372 2015-01-04 20:57:52 UpdateTip: new best=000000000000000039db3a041eeed1b27f7bff57653c1e1fa951f631b713ddd1 height=314451 log2_work=80.082915 tx=44077460 date=2014-08-07 21:39:59 progress=0.622556 cache=43591 2015-01-04 20:57:53 UpdateTip: new best=0000000000000000393962f5a96c3e732a1f46c3668188bd44e5f81479244ad1 height=314452 log2_work=80.083006 tx=44077729 date=2014-08-07 21:42:36 progress=0.622563 cache=43970 2015-01-04 20:57:54 LevelDB read failure: Corruption: block checksum mismatch 2015-01-04 20:57:54 Corruption: block checksum mismatch 2015-01-04 20:57:54

EXCEPTION: 13leveldb_error
Database corrupted
bitcoin in ProcessMessages()

2015-01-04 20:57:54 ProcessMessage(block, 460587 bytes) FAILED peer=6 2015-01-04 20:57:54 LevelDB read failure: Corruption: block checksum mismatch 2015-01-04 20:57:54 Corruption: block checksum mismatch

./bitcoind -par=1 -reindex

gmaxwell commented at 11:20 pm on January 4, 2015: contributor

@eN0Rm This issue is related to the fact that it kept running, not that there was an error. The error I observed was almost certainly state corruption due to faulty hardware (overheating cpu). What hardware / OS are you on?

eN0Rm commented at 9:22 am on January 6, 2015: none

Hi,

This was on a kvm guest running on an Intel NUC with the virtual disk stored on a external USB disk. I’ve read that this is most likely due to hw problems. I need to do some more research on that part. Anyway, the client kept running and I compiled the latest master from 2. or 3. of Jan.

Host is Ubuntu 14.10. Guest is 14.04.

-f-

On Mon, Jan 5, 2015 at 12:20 AM, Gregory Maxwell notifications@github.com wrote:

@eN0Rm https://github.com/eN0Rm This issue is related to the fact that it kept running, not that there was an error. The error I observed was almost certainly state corruption due to faulty hardware (overheating cpu). What hardware / OS are you on?

— Reply to this email directly or view it on GitHub #5462 (comment).

Mvh -fredrik-normann-

Sent from my Gmail Account

laanwj commented at 10:38 am on January 12, 2015: member

Closed by #5619

laanwj closed this on Jan 12, 2015

MarcoFalke locked this on Sep 8, 2021

Node continues running after LevelDB throws an error #5462