Corruption: block checksum mismatch

kollokollo commented at 7:17 am on November 7, 2017: none

Describe the issue

Bitcoin core runs normal, but when closing it, The WIndow appears (do not shut down the computer…) and during this an Error message pops up.

Can you reliably reproduce the issue?

If so, please list the steps to reproduce below:

close bitcoin core
start bitcoin core and wait until everything is synchronized
klick on the close button of the bitcoin core window
error occurs

Expected behaviour

Bitcoin core should close without an error.

Actual behaviour

It seems that besides this error on cloning the application there is no other misbehaviour observed,

OS: Ubuntu 17.04

RAM: 2GB
Disk size: 1TB
Disk Type (HD/SDD): HD, chainstate linked to SDD

Any extra information that might be useful in the debugging process.

[code] 2017-11-07 07:05:00 net thread exit 2017-11-07 07:05:00 scheduler thread interrupt 2017-11-07 07:05:00 Shutdown: In progress… 2017-11-07 07:05:00 msghand thread exit 2017-11-07 07:05:03 Dumped mempool: 0.009742s to copy, 1.33576s to dump 2017-11-07 07:05:04 Corruption: block checksum mismatch 2017-11-07 07:05:04 *** System error while flushing: Database corrupted 2017-11-07 07:05:13 Corruption: block checksum mismatch 2017-11-07 07:05:13 *** System error while flushing: Database corrupted 2017-11-07 07:05:15 Shutdown: done [/code]

The misbehaviour started, after I have run bitcoun core on different (maybe old) wallets, causing a reindexing. I am now back in the original wallet which has not produced such error before.

fanquake added the label Linux/Unix on Nov 7, 2017

kollokollo commented at 7:20 am on November 7, 2017: none

BTW: Bitcoin core version v.0.15.0.0-g3751912e8e and dmesg shows nothing special related to the Harddisk or the SSD. (no file system errors)

fanquake commented at 3:20 am on November 8, 2017: member

How old were the “old” wallets? Does the crash only happen on an exit during syncing, or after it has completed? Does the corruption disappear if you revert to using a newer wallet?

TheBlueMatt commented at 4:20 am on November 8, 2017: member

This isn’t a wallet error, its a data-read error which indicates either your disk is silently corrupting data (which can happen without filesystem errors) or your memory is bad/unstable or your CPU is too host/unstable/overclocked. Most common is memory errors or unstable CPU - run memtest and check the temperature of your CPUs under load (eg via linpack or similar load/performance tests)

kollokollo commented at 6:47 am on November 8, 2017: none

The old wallets were 2011 and 2014. The problem started when I reverted back to the 2017 wallet and is now persistant. The Error only appears on an exit. Until then, Bitcoin core runs normally. It appears everytime now on exit. The corruption has not disappeared (using the new wallet now again).

The Computer (its a laptop) has never been shut down without having closed bitcoin core. The wallets were exchanged always when bitcoin core was not running. The Harddisk nor the SSD (which I use for the chainstate folder) have reported errors. The Laptop is a recent model, I never saw RAM problems. To verify if it was a read error: Which file should I read to test and what against should I compare the content?

kollokollo commented at 6:48 am on November 8, 2017: none

BTW: Syncing was complete when the error appeared. The Laptop is an ideapad 110S, max 1 year old, not modified/overclocked or so. Temperature under load was not noticable, however the (external) SSD for the chainstate got quite hot.

kollokollo commented at 6:57 am on November 8, 2017: none

Here is another symptom: Everytime now bitcoin core is restarted, it starts syncing from Mo, Nov6 8:45, al though the syncing had completed before. If this persists, this would make bitcoin core unusable with time (with the corrupted data).

kollokollo commented at 7:55 am on November 9, 2017: none

Today I got the following error, during synchronization: [code] o) warning=‘1 of last 100 blocks have unexpected version’ 2017-11-09 07:52:48 UpdateTip: new best=00000000000000000093ed958fcdee0e7025d7b8d8f15fed221e4c4da32d0369 height=493686 version=0x20000000 log2_work=87.438028 tx=269732432 date=‘2017-11-08 23:51:00’ progress=0.999668 cache=224.6MiB(1705500txo) warning=‘1 of last 100 blocks have unexpected version’ 2017-11-09 07:52:50 LevelDB read failure: Corruption: block checksum mismatch 2017-11-09 07:52:50 Corruption: block checksum mismatch 2017-11-09 07:53:17 Error reading from database: Database corrupted [/code]

Bitcoin-qt then stopped working (windo not responding anymore). After a while it core-dumped (At least I got the message, that the app has crashed and there is not enough memory to save the core and inform the developpers). With the next start, the problem persisted (again not responding) and I had to kill the process with -9. .

No IO-Errors in dmesg.

So If this problem is not caused by a bug in bitcoin-qt, how can one fix the database?

laanwj added the label Data corruption on Nov 9, 2017

laanwj commented at 11:56 am on November 9, 2017: member

This fortunately has nothing to do with the wallets. Keep a good backup of them. Try syncing them on a different computer.

The old wallets were 2011 and 2014. The problem started when I reverted back to the 2017 wallet and is now persistant.

The reason for this is likely that part of your blocks or other on-disk state was corrupted, and replacing your wallet triggers a rescan over the blocks from the point it was last used to now, so it was only detected then.

kollokollo commented at 12:49 pm on November 9, 2017: none

Well, that might be. All though the rescan was sucessful at least 3 times, until the problems occured after the original wallet was restored, which should not have triggered a rescan (and as far as I remember none has happened). All rescans were done on the same day with more or less the same block and database data. (With wallet I mean only the file wallet.dat). I now also have the suspicion, that the error has nothing to do with the changed wallets. And this because it looks like, that the corruption time was Mo, Nov6 8:45, after any resan, but the same bitcoin-qt session, were the original walled.dat was put back.

So: what can I do to repair the corrupted database?

kollokollo commented at 10:02 am on November 14, 2017: none

I ended up starting from the beginning and reloaded the whole blockchain. This time from a different computer. So far so good, it worked again without problems. But then, after importing a single private key (using the console) new strange problems came up: The rescanning took a while and finally the bitcoins belonging to the newly imported address were shown in the wallet. BUT: The dates of the transactions belonging to the imported adress where totally messed up. They all show the latest date of the latest transacrion in the wallet, all the same. This is strange. Imagine having a transaction date from yesterday, but 160000 confirmations. So I suggest there is indeed a bug in bitcoin core, which shows up after importing private keys or whole (old) wallets.

TheBlueMatt commented at 5:54 pm on November 14, 2017: member

@kollokollo the timestamps displayed are the timestamps from the block when the transaction was included, as there is no other way to figure out when a transaction was. It appears the original issue here was a hardware issue, so this should likely be closed. If the transaction timestamps are displaying incorrectly, please open a new issue for that.

MarcoFalke commented at 6:00 pm on November 14, 2017: member

Closing per @TheBlueMatt

MarcoFalke closed this on Nov 14, 2017

kollokollo commented at 6:54 am on November 15, 2017: none

I always thought, that each block has its individual timestamp, so if you know how many confirmations a tranaction has, you should also know when the transaction happened(+/- 10 min). In my case the timestamps arenot taken from the blocks, but an arbitrary timestamp (from the latest transaction) was taken/shown. Since bitcoin core so far and besides this semms to work normally, I do not care about these old transactions. But the display of them looks really confusing. (In one case, the recipient of a transaction is shown as (n/a) instead of an adress.) Bitcoin core could do better, since all information should have been retreived correctly during the rescan process.

TheBlueMatt commented at 10:31 pm on November 15, 2017: member

@kollokollo Could you please open a new ticket with more details about these “strange” transaction timestamps?

kollokollo commented at 7:38 am on November 16, 2017: none

Issue is opened. Unfortuantely I could not upload the screenshots (using firefox).

coralnode commented at 10:26 pm on December 16, 2017: none

I am getting a similar warning since a couple of hours ago. ‘1 of last 100 bocks have unexpected version’. Any resolution?

sipa commented at 11:39 pm on December 16, 2017: member

@coralnode That is unrelated, and completely expected. It’s due to a miner using a weird version number in a block, not due to local corruption.

usab commented at 11:53 am on January 18, 2018: none

Same problem here Corruption: block checksum mismatch

TheBlueMatt commented at 7:56 pm on January 18, 2018: member

Please provide hardware information as well as run hardware tests (memtest, linpack/prime95/similar, and a disk SMART check of some kind) when responding to this issue. There are no (known) issues with Bitcoin Core itself which would generate the errors, but there are many examples of unstable hardware which causes them (including hardware which is stable under other workloads).

On January 18, 2018 11:53:10 AM UTC, usab notifications@github.com wrote:

Same problem here Corruption: block checksum mismatch

– You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/bitcoin/bitcoin/issues/11627#issuecomment-358624670