erraneous InvalidChainFound messages #2726

2013-05-31 09:28:32 Bitcoin version v0.8.2-beta (2013-05-25 08:48:25 -0700) 2013-05-31 09:28:41 LoadBlockIndexDB(): hashBestChain=000000000000036b0687c15229f7c337265f2154c6ece7a1fde625f97c260ebf height=197884 date=2012-09-08 19:08:11 2013-05-31 11:05:45 SetBestChain: new best=000000000000002f6afc75f359c785e48c294a881ab5ace093ce84146f8c1f0b height=228337 log2_work=69.636427 tx=15075580 date =2013-03-28 03:34:39 progress=0.487031 2013-05-31 11:05:46 InvalidChainFound: invalid block=000000000000002a0bd37c0f44c2f35a75ede3b01f6710c0369a79dfdd4837da height=228338 log2_work=69.636473 date=20 13-03-28 03:38:55

The hash of block 228388 is 00000000000000821df4084902335c1736e715c1d2fa62e5655bbdb9276e6604

ok, so what does this mean? Why do I still continue to get invalid blocks, even today? e.g.:-

2013-06-02 03:40:03 received block 000000000000014400eae885c5d28444caff6910d2d863728d527f530e47e563 2013-06-02 03:40:03 InvalidChainFound: invalid block=000000000000014400eae885c5d28444caff6910d2d863728d527f530e47e563 height=239195 log2_work=70.200777 date=2013-06-02 03:28:25

Surely there is not a complete invalid chain from 228388 to 239195 is there?

Ah, I just noticed the typo in the issue title. It is block 228338, not block 228388. Now corrected the title.

@rebroad I assume you chainstate database got corrupted. You can rebuild it by starting with the -reindex command-line option.

I will try this, but I literally just did a reindex before this issue occurred as part of an upgrade from 0.6.3 to 0.8.2, so I#m skeptical, plus I think perhaps it gets too easily corrupted given this.

@sipa ok, I have done the reindex now, but the problem still exists, although the block it got stuck at is different (it managed a further 3 days and is stuck at 1st April 2013 now). Does this mean i need to keep reindexing each time? if so, this will take forever!

@rebroad You wouldn't happen to be near a train yard? (http://jakepoz.com/soviet_debugging.html)

Another potential issue with this is that no warning is displayed in the GUI - only in the debug.log.

I have tried reindexing again, and using the latest downloadable bitcoin client for windows, but it's still stuck.

2013-06-08 08:52:18 SetBestChain: new best=00000000000001abe4ec1d257b30edb23b3e5d05397cc01089ff4a5750266347 height=229659 log2_work=69.694998 tx=15516508 date=2013-04-04 13:47:29 progress=0.513570 2013-06-08 08:52:18 ProcessBlock: ACCEPTED 2013-06-08 08:52:20 ERROR: CScriptCheck() : 51de91a4fd671baae0a33fbe36cc34898c3bd5aa4e45169ab8e3e9ba07a5a8bf VerifySignature failed 2013-06-08 08:52:21 InvalidChainFound: invalid block=00000000000001ca78f39a60d424b5d39716cd8b874d1d67f8e5946c44fe74ba height=229660 log2_work=69.695041 date=2013-04-04 14:04:18 2013-06-08 08:52:21 InvalidChainFound: current best=00000000000001abe4ec1d257b30edb23b3e5d05397cc01089ff4a5750266347 height=229659 log2_work=69.694998 date=2013-04-04 13:47:29 2013-06-08 08:52:21 InvalidChainFound: invalid block=00000000000001ca78f39a60d424b5d39716cd8b874d1d67f8e5946c44fe74ba height=229660 log2_work=69.695041 date=2013-04-04 14:04:18 2013-06-08 08:52:21 InvalidChainFound: current best=00000000000001abe4ec1d257b30edb23b3e5d05397cc01089ff4a5750266347 height=229659 log2_work=69.694998 date=2013-04-04 13:47:29 2013-06-08 08:52:21 ERROR: SetBestBlock() : ConnectBlock 00000000000001ca78f39a60d424b5d39716cd8b874d1d67f8e5946c44fe74ba failed 2013-06-08 08:52:21 ERROR: AcceptBlock() : AddToBlockIndex failed 2013-06-08 08:52:21 ERROR: ProcessBlock() : AcceptBlock FAILED

@rebroad Can you try reindexing again, with par=1 in your bitcoin.conf file, or with -par=1 on the command line? I just want to rule out this is an issue caused by multithreaded verification, but my best guess is still that this is a hardware problem.

@sipa i will try that next. Just to also clarify. I've also tried removing the database entirely and redownloading the blockchain previously. i've now just tried another reindex, and this time it's stuck again, but again on a more recent block:-

2013-06-09 07:57:18 SetBestChain: new best=0000000000000007e967587557d2c26fbd6259a6bdad4f2943d1f23f5fec2517 height=232631 work=1057302097488701699688 tx=16563589 date=2013-04-22 20:27:57 2013-06-09 07:57:18 ProcessBlock: ACCEPTED 2013-06-09 07:57:19 ERROR: CScriptCheck() : e50c94cf77 VerifySignature failed 2013-06-09 07:57:19 InvalidChainFound: invalid block=0000000000000141686ceba58449a93c304b07aac8188583fbbf35aa66fff586 height=232632 work=1057340642384738704203 date=2013-04-22 20:36:24 2013-06-09 07:57:19 InvalidChainFound: current best=0000000000000007e967587557d2c26fbd6259a6bdad4f2943d1f23f5fec2517 height=232631 work=1057302097488701699688 date=2013-04-22 20:27:57 2013-06-09 07:57:19 InvalidChainFound: invalid block=0000000000000141686ceba58449a93c304b07aac8188583fbbf35aa66fff586 height=232632 work=1057340642384738704203 date=2013-04-22 20:36:24 2013-06-09 07:57:19 InvalidChainFound: current best=0000000000000007e967587557d2c26fbd6259a6bdad4f2943d1f23f5fec2517 height=232631 work=1057302097488701699688 date=2013-04-22 20:27:57 2013-06-09 07:57:19 ERROR: SetBestBlock() : ConnectBlock 0000000000000141686ceba58449a93c304b07aac8188583fbbf35aa66fff586 failed 2013-06-09 07:57:19 ERROR: AcceptBlock() : AddToBlockIndex failed 2013-06-09 07:57:19 ERROR: ProcessBlock() : AcceptBlock FAILED

@sipa what sort of hardware problem would cause this? and surely if I had such a hardware problem wouldn't other applications crash and it'd be unlikely I'd have a system uptime of several weeks.

I've re-run bitcoin-qt 0.8.2 with the -par=1 option and after completely removing the database so that it downloaded the blockchain from the genesis block. This time it got stuck on 23rd March 2013, marking that as in invalid block. I've had enough of this, so I'm reverting back to 0.6.3 and will see how I go with that.

0.6.3 is also having the same problem now. I've deleted the database and re-run the latest version:

SetBestChain: new best=000000000000022f951a82f1e2d8fe06378edd5cef926d39c09d44ac25c5e5d5 height=231305 log2_work=69.773668 tx=16119387 date=2013-04-14 12:15:08 progress=0.533195 init message: Loading addresses... Loaded 14683 addresses from peers.dat 219ms mapBlockIndex.size() = 231307 nBestHeight = 231305 setKeyPool.size() = 100 mapWallet.size() = 0 mapAddressBook.size() = 1 dnsseed thread start upnp thread start Loading addresses from DNS seeds (could take a while) init message: Done loading msghand thread start addcon thread start dumpaddr thread start net thread start opencon thread start Added 19 addresses from x.x.x.x: 68 tried, 14618 new refreshWallet GetMyExternalIP() received [x.x.x.x] x.x.x.x:0 GetMyExternalIP() returned x.x.x.x AddLocal(x.x.x.x:8333,4) Flushed 14686 addresses to peers.dat 328ms trying connection x.x.x.x:8333 lastseen=6.4hrs UPnP: ExternalIPAddress = x.x.x.x AddLocal(x.x.x.x:8333,3) UPnP Port Mapping successful. Added 24 addresses from 0:0:0:0:0:0:0:0: 68 tried, 14621 new Added 12 addresses from 173.242.112.53: 68 tried, 14622 new Added 10 addresses from 0:0:0:0:0:0:0:0: 68 tried, 14624 new 93 addresses found from DNS seeds dnsseed thread exit connection timeout received block 000000000000015d0c5bc75517a586be19f3b4bd6f91fde4659127f96464f3af InvalidChainFound: invalid block=000000000000015d0c5bc75517a586be19f3b4bd6f91fde4659127f96464f3af height=231307 log2_work=69.773762 date=2013-04-14 12:55:06 InvalidChainFound: current best=000000000000022f951a82f1e2d8fe06378edd5cef926d39c09d44ac25c5e5d5 height=231305 log2_work=69.773668 date=2013-04-14 12:15:08 ProcessBlock: ACCEPTED

It always seems to get stuck in March or April 2013, so there seems to be some significance here perhaps.

Can perhaps some facility be added so that it can re-check a block that fails just to see if it fails the same way a second time?

I am now running bitcoin-qt 0.8.3 on a different machine, and a different OS (ubuntu 64bit rather than Windows XP), and I am still seeing this problem. Am only I getting this?!

You did see my earlier post— right? :p

No, people really aren't getting this. I would be fascinated in knowing what is unusual in your configuration beyond you!

I had exactly this issue. I tried running bitcoind -daemon -reindex more times than I can count. I even tried deleting my whole ~/.bitcoin directory twice. Every time I got to about April 2013 and then the "InvalidChainFound" errors appear in debug.log. From this point onwards, bitcoind getinfo shows the error:

Warning: Displayed transactions may not be correct! You may need to upgrade, or other nodes may need to upgrade

I just wanted to confirm that adding par=1 to bitcoin.conf has fixed the problem for me.

(I should add, though, that I'm running bitcoind 8.3 from Debian sid, which has had some patches applied to it.)

@edam Interesting. This is a case of database error that goes undetected likely, so incorrect data is read back at some point, which mismatches the chain spends.

I'd very much like to know whether it is consistently the same block that triggers it (and if so, which). Can you try to reproduce the problem again, and report the exact InvalidChainFound errors reported (in particular "InvalidChainFound: current best=%s height=%d log2_work=%.8g date=%s" lines)?

That would be very helpful.

@sipa: I don't think it was the same place, but I'll have to re-run it to find out. I'll report back with some logfile text after a couple of failed attempts.

Here are the results of two separate runs, each taking approx. 6+ hrs before InvalidChainFound error messages started appearing in the logs. The following is the first occurrence of those errors in the logs for each run...

First run:

SetBestChain: new best=000000000000012a2ebbc76eb1bcdeffceafbac248faaebd10c45a1400fabad6  height=225967  log2_work=69.549673  tx=14452116  date=2013-03-15 09:18:16 progress=0.280715
ERROR: CScriptCheck() : 406168d4647a3fc8c9b85d256efd88a7ce4506f8145e0cd2cc0951827e62dff8 VerifySignature failed
InvalidChainFound: invalid block=0000000000000299ca28abcc46ecc4bc3b8ea307d2a03a53def1ead1ba5af36e  height=225968  log2_work=69.549708  date=2013-03-15 09:29:04
InvalidChainFound:  current best=000000000000012a2ebbc76eb1bcdeffceafbac248faaebd10c45a1400fabad6  height=225967  log2_work=69.549673  date=2013-03-15 09:18:16

Second run:

received block 0000000000000223d53ab405d956434482daa9055e08ea7e425551a88969025d
SetBestChain: new best=0000000000000223d53ab405d956434482daa9055e08ea7e425551a88969025d  height=225902  log2_work=69.547411  tx=14428878  date=2013-03-14 23:49:37 progress=0.278031
ProcessBlock: ACCEPTED
received block 000000000000008b4bde87880e58fb9c4a633f571a327d5e69786db6a524eff0
ERROR: CScriptCheck() : 3fd9b0a92a72dfa7d733e6c78db957871eb8e31c3867e266800afcbe1ba6e5a0 VerifySignature failed
InvalidChainFound: invalid block=000000000000008b4bde87880e58fb9c4a633f571a327d5e69786db6a524eff0  height=225903  log2_work=69.547446  date=2013-03-15 00:04:37
InvalidChainFound:  current best=0000000000000223d53ab405d956434482daa9055e08ea7e425551a88969025d  height=225902  log2_work=69.547411  date=2013-03-14 23:49:37

So it looks like a different place, which tallies with what other people have said as well.

I don't know if it's relevant, but I thought I'd mention that there had been InvalidChainFound messages in the logs for a couple of hours when I came back to check on the first run this morning, and bitcoind getinfo wasn't reporting an error. It was only after I restarted it (via bitcoind stop and bitcoind -daemon), and then for a connection to register, that the "Warning: Displayed transactions [...]" error started showing up. I seem to remember that this is what was happening before, as well. But the transaction verification process was clearly stuck at progress=0.280715 regardless. For the second run, bitcoind getinfo was reporting the error when I went back to check it.

I still have all the logs from both failed runs. Is there any more info I can provide, or further tests I can run?

@edam To exclude the possibility that the Debian patches were the cause, could you try the same thing with the released bitcoind binaries on sourceforge?

@sipa: OK, built from bitcoin-0.8.3-linux.tar.gz, obtained from sourceforge (against DB5.1, as there is no libdb4.8-dev package in Debain unstable). I started getting InvalidChainFound in debug.log after about 5hrs. First occurrence is shown below:

SetBestChain: new best=00000000000002e795c6ad77bd81d9599147eda74b54c96bd841716020b2d3df  height=226197  log2_work=69.557647  tx=14514594  date=2013-03-16 17:00:55 progress=0.284534
ProcessBlock: ACCEPTED
received block 0000000000000295b44d976591991b889faf5407c621c961a98132b6bd529f1b
ERROR: CScriptCheck() : 2328ffd25c0344ed8ed2b455a4bdfbf466326bbabcbbbe12b81502b76161a750 VerifySignature failed
InvalidChainFound: invalid block=0000000000000295b44d976591991b889faf5407c621c961a98132b6bd529f1b  height=226198  log2_work=69.557681  date=2013-03-16 17:11:15
InvalidChainFound:  current best=00000000000002e795c6ad77bd81d9599147eda74b54c96bd841716020b2d3df  height=226197  log2_work=69.557647  date=2013-03-16 17:00:55

So the three runs have failed at 14th, 15th and 16th April, 2013! Strange that the points of failure are so close together.

@edam Can you try running the binaries in that tar? They are statically linked and will isolate out weirdness with your compiler and system libraries.

If you're concerned about the provenance of random binaries, they're built using a determinist build process which you can run yourself (needs an ubuntu VM, however). They've also been built by several independent parties who have uploaded signatures here: https://github.com/bitcoin/gitian.sigs

@gmaxwell: thanks, I was concerned about that, yes (gitian's a clever idea, though!). Anyway, I've run the pre-built binary now and I still get the same results:

SetBestChain: new best=000000000000019a6cd4b53febbe8c0521496c9cd1015b089fdcc1c78d920a77  height=228172  log2_work=69.628947  tx=15013716  date=2013-03-26 23:14:26 progress=0.332393
ProcessBlock: ACCEPTED
stored orphan tx 9b2ba43727008fa8af0c46c4c5389409beceb3fca52b8fd7485c0b6ad8f09443 (mapsz 6735)
received block 0000000000000121cb62f9ba2039908031c9ca324b9caf01f5efb6ff41bdc01d
ERROR: CScriptCheck() : 0e3d910bbc2c801ac59e350b681c85a83ad7aa44f328238d8b04d2eb7f17ec47 VerifySignature failed
InvalidChainFound: invalid block=0000000000000121cb62f9ba2039908031c9ca324b9caf01f5efb6ff41bdc01d  height=228173  log2_work=69.628993  date=2013-03-26 23:17:53
InvalidChainFound:  current best=000000000000019a6cd4b53febbe8c0521496c9cd1015b089fdcc1c78d920a77  height=228172  log2_work=69.628947  date=2013-03-26 23:14:26

Just to be clear; you restarted from scratch with the pre-built binary?

May I ask you to run memtest86 for a few hours on your system?

@sipa: that is correct. I deleted (renamed, actually) ~/.bitcoin, re-ran bitcoind -daemon using the pre-built binary in the tarball (and again, after setting the rpcuser/rcppassword), left it running for several hours and then checked bitcoind getinfo and the logs for errors.

The machine this is running on is well used and has not shown any memory- or cpu-like errors. All the same, I ran memtest86+ for over an hour without error, just to be sure. If it's something odd about my system that's causing this, it may be that I am using Debian sid/unstable (with a few packages from experimental). So I wouldn't rule out the kernel or the few shared libraries that the pre-built binary uses from the system. Should I try the pre-built binary on a live-CD of, say, Ubuntu 13.04 to see if it exhibits the same problems?

@edam I'm running out of guesses, so yes— starting "something else" would be an interesting thing to test.

Thank you for being so helpful on this.

Memtest86+ can run for many hours before showing faulty RAM.

Exactly the same problem here. Debian Squeeze amd64 using precompiled bitcoind 8.3 binary.

Yesterday, it randomly started to show (in getinfo):

Warning: Displayed transactions may not be correct! You may need to upgrade, or other nodes may need to upgrade

Removed the whole .bitcoin folder, but during rebuilding, the above mentioned errors appeared in debug.log.

Currently trying with the par=1 option.

Edit: Worked with the par=1 option without a problem. Thanks!

@sipa Does this mean we may have a problem with the parallel verification code?

Sorry for my lack of response -- I'm on holiday at the moment. @gmaxwell: I can test that, just in case. But it will be at least a week before I can. In the meantime I'm also re-trying verification with par=1, just to be sure that it wasn't a fluke. @Diapolo: If it helps, I am a developer, so I'm quite happy to apply "debugging" patches against the 0.8.3 code, or to fiddle about in gdb. But it takes about 6hrs to go wrong on my machine, and I'm not familiar with the bitcoind code, so you'd have to direct me as to what information would be useful to you.

We'd really like to get this fixed! Has anyone experiencing this had a chance to try the above test?

@gmaxwell: I've re-run bitcoind another 3 times, using a fresh ~/.bitcoin directory each time, with par=1 in the config file and I haven't received any errors. So, it looks like I'm getting a fairly reliable 100% failure rate when not using par=1 and 100% success rate when using it.

I'll try to run the above test (live CD of another distro, running binary from your tarball, no par=1) tomorrow, while I'm at work, and report back in the evening. Since Ubuntu is based on Debian, though, would it be wiser to try something more different, like Fedora? Or would you rather I use Ubuntu?

Hi!

I have the same error on block 254354. Tried it with 0.8.2, 0.8.3 and 0.8.4RC2 win32 (cleaned ./bitcoin-dir befor everytime) - no way! Tried -par=1 (on all versions obove) - no way. :(

<pre> received block 0000000000000025d41d0269b661cc504a4e15927b94006e2978fdd54a5e12ac SetBestChain: new best=0000000000000025d41d0269b661cc504a4e15927b94006e2978fdd54a5e12ac height=254354 log2_work=71,455504 tx=22765161 date=2013-08-26 17:42:00 progress=0,940830 ProcessBlock: ACCEPTED received block 0000000000000005c8172a4db7f2785121e6bf02d60376c2c159eabb8ead1b90 ERROR: CScriptCheck() : ac40e0204588ff77f8d536f042930b1e85ef826fdc9ddaca301b5d144064b7e0 VerifySignature failed InvalidChainFound: invalid block=0000000000000005c8172a4db7f2785121e6bf02d60376c2c159eabb8ead1b90 height=254355 log2_work=71,45563 date=2013-08-26 17:49:24 InvalidChainFound: current best=0000000000000025d41d0269b661cc504a4e15927b94006e2978fdd54a5e12ac height=254354 log2_work=71,455504 date=2013-08-26 17:42:00 InvalidChainFound: invalid block=0000000000000005c8172a4db7f2785121e6bf02d60376c2c159eabb8ead1b90 height=254355 log2_work=71,45563 date=2013-08-26 17:49:24 InvalidChainFound: current best=0000000000000025d41d0269b661cc504a4e15927b94006e2978fdd54a5e12ac height=254354 log2_work=71,455504 date=2013-08-26 17:42:00 ERROR: SetBestBlock() : ConnectBlock 0000000000000005c8172a4db7f2785121e6bf02d60376c2c159eabb8ead1b90 failed ERROR: AcceptBlock() : AddToBlockIndex failed ERROR: ProcessBlock() : AcceptBlock FAILED Misbehaving: 37.200.67.186:8333 (0 -> 100) DISCONNECTING disconnecting node 37.200.67.186:8333 socket send error 10038 getblocks 254337 to 0000000000000000000000000000000000000000000000000000000000000000 limit 500 getblocks 254337 to 0000000000000000000000000000000000000000000000000000000000000000 limit 500 stored orphan tx 0a5df865c74d56e8219f363ed56599d7bcd7524c5976b2cb73f3276dc68d67e9 (mapsz 191) stored orphan tx 1401c073c8f7cedd993fd23e5d2c4717761e2bf98c62b58c90e033da64bc7320 (mapsz 192) stored orphan tx 28c2495288c2f1d1e5650eb392e939e6bc13a81efd775f605b1cd16505f00d03 (mapsz 193) trying connection 24.34.199.73:8333 lastseen=4,9hrs Added 1 addresses from 173.51.246.36: 164 tried, 12881 new getblocks 254337 to 0000000000000000000000000000000000000000000000000000000000000000 limit 500 Added 1 addresses from 5.10.73.2: 164 tried, 12881 new getblocks 254337 to 0000000000000000000000000000000000000000000000000000000000000000 limit 500 Added 1 addresses from 46.172.164.168: 164 tried, 12881 new getblocks 254337 to 0000000000000000000000000000000000000000000000000000000000000000 limit 500 getblocks 254337 to 0000000000000000000000000000000000000000000000000000000000000000 limit 500 Added 1 addresses from 76.185.122.238: 164 tried, 12882 new received block 00000000000000232714857099392dc1e99859d139869bd235663aa5f1694c1d ERROR: ProcessBlock() : already have block 254348 00000000000000232714857099392dc1e99859d139869bd235663aa5f1694c1d Misbehaving: 177.71.192.164:8333 (0 -> 0) received block 0000000000000031ab46309379fceb468ee334ca53eabb221aa17a8bce1c13ab ERROR: ProcessBlock() : already have block 254349 0000000000000031ab46309379fceb468ee334ca53eabb221aa17a8bce1c13ab Misbehaving: 177.71.192.164:8333 (0 -> 0) received block 00000000000000348fb7b94cad608621fa6aec2a51cf9f5aec9c93cee717884a ERROR: ProcessBlock() : already have block 254350 00000000000000348fb7b94cad608621fa6aec2a51cf9f5aec9c93cee717884a Misbehaving: 177.71.192.164:8333 (0 -> 0) Added 1 addresses from 177.71.192.164: 164 tried, 12882 new received block 000000000000003a898241d039b3f5878a6caac994527a3827adb2b5b3600527 ERROR: ProcessBlock() : already have block 254351 000000000000003a898241d039b3f5878a6caac994527a3827adb2b5b3600527 Misbehaving: 177.71.192.164:8333 (0 -> 0) received block 00000000000000239b34bf8e60217edfaa81eec6f834f2b335e6bb6e77c8c80b ERROR: ProcessBlock() : already have block 254352 00000000000000239b34bf8e60217edfaa81eec6f834f2b335e6bb6e77c8c80b Misbehaving: 177.71.192.164:8333 (0 -> 0) received block 000000000000000b98e925838eeb367bd7cf6b955436e3989e6755dbf8b6a3cb ERROR: ProcessBlock() : already have block 254353 000000000000000b98e925838eeb367bd7cf6b955436e3989e6755dbf8b6a3cb Misbehaving: 177.71.192.164:8333 (0 -> 0) connected 24.34.199.73:8333 send version message: version 70001, blocks=254354, us=93.130.103.179:8333, them=24.34.199.73:8333, peer=24.34.199.73:8333 received block 0000000000000025d41d0269b661cc504a4e15927b94006e2978fdd54a5e12ac ERROR: ProcessBlock() : already have block 254354 0000000000000025d41d0269b661cc504a4e15927b94006e2978fdd54a5e12ac Misbehaving: 177.71.192.164:8333 (0 -> 0) received block 0000000000000005c8172a4db7f2785121e6bf02d60376c2c159eabb8ead1b90 ERROR: ProcessBlock() : already have block 254355 0000000000000005c8172a4db7f2785121e6bf02d60376c2c159eabb8ead1b90 Misbehaving: 177.71.192.164:8333 (0 -> 0) received block 000000000000001849dc7b81f12297fe5448e928fc02566d5da58fc90817dd66 Pre-allocating up to position 0x6000000 in blk00077.dat InvalidChainFound: invalid block=000000000000001849dc7b81f12297fe5448e928fc02566d5da58fc90817dd66 height=254356 log2_work=71,455756 date=2013-08-26 17:57:29 InvalidChainFound: current best=0000000000000025d41d0269b661cc504a4e15927b94006e2978fdd54a5e12ac height=254354 log2_work=71,455504 date=2013-08-26 17:42:00 ProcessBlock: ACCEPTED received block 0000000000000031baa260f1c6b7527c35ffd2ea6868beb3f82deeb62ab018df InvalidChainFound: invalid block=0000000000000031baa260f1c6b7527c35ffd2ea6868beb3f82deeb62ab018df height=254357 log2_work=71,455882 date=2013-08-26 18:07:50 InvalidChainFound: current best=0000000000000025d41d0269b661cc504a4e15927b94006e2978fdd54a5e12ac height=254354 log2_work=71,455504 date=2013-08-26 17:42:00 ProcessBlock: ACCEPTED </pre>

@Schmollen Are you using some heave overclocking with that system or do you encounter other system instability?

@edam Great. Can you check to see if your system has a separate multithreaded copy of boost and if bitcoin is being linked to that one? @Schmollen can you try booting your system into memtest 86x as a sanity check?

Are you running bitcoind -daemon or ./bitcoind -daemon

The first will still be running the debian bitcoind even if you're in the ./bin/64 directory of the tarball.

Hi! @Diapolo No, I don't. It is an old AMD Athlon (no overclock, no undervolt) Win XP (32bit) PC, Only for surfing (a litlle bit) and Bitcoin QT. Streestest (Prime95 & Memtest86) are okay. @gmaxwell: Memtest: sanity check is okay. @pstratem If you mean me: No, it doesn't run with -daemon

Just put it on a WinXP-PentiumM-Laptop same error, same block 254354. Using 0.8.3 with -par=1. :( Tried 0.8.1 now: the same. :(

Same issuse with block 263561. Will try reindex and -par=1 later. Edit: bitcoind -reindex -par=1 fix this.

@sipa I find it really odd, that quite a few people reported that -par=1 works but more threads seem to cause issues.

Sorry about my period of silence on this. For some reason the problem seems to have gone away (at least on the 64bit ubuntu computer). It is actually a different computer, but same make and model. Also I'm now in Guatemala, whereas previously I was in Thailand (I doubt this makes a difference). I didn't use the par=1 option in the bitcoin config file. I'm using the Bitcoin version v0.8.0-199-g88f70f5-beta (2013-04-21 10:33:18 +0700).

Confirm the problem for version 0.8.5. I have Debian stable release (wheezy) on 32-bit i386 platform. I tried different sources: get binary from sourceforge.net, get binary from Debian unstable release (sid) repository, compile sources. All results the same:

received block 0000000000038703a7ec5ef25ac27471d1db2b32e7b8d1414e078fcdc3f0abeb SetBestChain: new best=0000000000038703a7ec5ef25ac27471d1db2b32e7b8d1414e078fcdc3f0abeb height=101997 log2_work=54.761447 tx=3540 date=2011-01-10 20:34:35 progress=0.000063 ProcessBlock: ACCEPTED received block 0000000000030c522a40397eaf099b1572d7368a31d0891f71c7222d7980dfa8 ERROR: ConnectBlock() : inputs missing/spent InvalidChainFound: invalid block=0000000000030c522a40397eaf099b1572d7368a31d0891f71c7222d7980dfa8 height=101998 log2_work=54.764752 date=2011-01-10 20:37:04 InvalidChainFound: current best=0000000000038703a7ec5ef25ac27471d1db2b32e7b8d1414e078fcdc3f0abeb height=101997 log2_work=54.761447 date=2011-01-10 20:34:35 InvalidChainFound: invalid block=0000000000030c522a40397eaf099b1572d7368a31d0891f71c7222d7980dfa8 height=101998 log2_work=54.764752 date=2011-01-10 20:37:04 InvalidChainFound: current best=0000000000038703a7ec5ef25ac27471d1db2b32e7b8d1414e078fcdc3f0abeb height=101997 log2_work=54.761447 date=2011-01-10 20:34:35

But 0000000000030c522a40397eaf099b1572d7368a31d0891f71c7222d7980dfa8 is VALID block with height=101998!!! After that daemon accepts no one valid block in the next chain:

received block 0000000000038b80cf5db1173e96f2290cfda12c505b0fe1bd37d6975e164a8a InvalidChainFound: invalid block=0000000000038b80cf5db1173e96f2290cfda12c505b0fe1bd37d6975e164a8a height=101999 log2_work=54.76805 date=2011-01-10 20:39:08 InvalidChainFound: current best=0000000000038703a7ec5ef25ac27471d1db2b32e7b8d1414e078fcdc3f0abeb height=101997 log2_work=54.761447 date=2011-01-10 20:34:35 ProcessBlock: ACCEPTED received block 00000000000335c47dd6ae953912d172a4d9839355f2083165043bb6f43c2f58 InvalidChainFound: invalid block=00000000000335c47dd6ae953912d172a4d9839355f2083165043bb6f43c2f58 height=102000 log2_work=54.77134 date=2011-01-10 20:39:40 InvalidChainFound: current best=0000000000038703a7ec5ef25ac27471d1db2b32e7b8d1414e078fcdc3f0abeb height=101997 log2_work=54.761447 date=2011-01-10 20:34:35 ProcessBlock: ACCEPTED received block 0000000000035107dce8eb675c6fa9a08c7617c109b3553ad8f208dda24065a6 InvalidChainFound: invalid block=0000000000035107dce8eb675c6fa9a08c7617c109b3553ad8f208dda24065a6 height=102001 log2_work=54.774623 date=2011-01-10 20:41:59 InvalidChainFound: current best=0000000000038703a7ec5ef25ac27471d1db2b32e7b8d1414e078fcdc3f0abeb height=101997 log2_work=54.761447 date=2011-01-10 20:34:35

And so on. A bit later JSON connection attempt failed, and command line request returns: "error: couldn't connect". After restart of the daemon getblockcounter is resetting to 0...

-par=1 didn't help. It only reduce risk of spontaneous error when accepting received block, but reaction on JSON requests remains unpredictable. I had getcblockcount risen up to 250554, but then at some moment daemon didn't respond JSON request and command-line request hung (I should press ^C). Entering "ps -ef" revealed, that every new command-line request (including "bitcoind stop") generates new copy of bitcoind process (!!!). I should use "kill -9 $pid" to clear system from them. After restart blockchain database occurs broken :-( So, now I should load the blockchain again: -reindex doesn't work. It's a pity, but on my platform the product is COMPLETELY NON OPERABLE. I've already wasted a lot of time for this buggy thing...

P.S. Wow, now the daemon spontaniously stopped and I got this message on stdout: "bitcoind: main.cpp:1138: unsigned int GetNextWorkRequired(const CBlockIndex_, const CBlockHeader_): Assertion `pindexFirst' failed." I looked in debug.log and found out, that problems began immediately after restart of the daemon (I stopped it to backup blockchain DB):

trying connection 62.77.93.31:8333 lastseen=8.0hrs received block 0000000000008d05a7e6cfeb4e6767abd3b45aec54bda7fd06bccb8affa60ec9 Pre-allocating up to position 0x100000 in rev00000.dat SetBestChain: new best=0000000000008d05a7e6cfeb4e6767abd3b45aec54bda7fd06bccb8affa60ec9 height=121548 log2_work=56.708617 tx=3755 date=2011-05-03 05:18:37 progress=0.000066 ProcessBlock: ACCEPTED received block 0000000000001c06fe3805f664cdb1b877bf6eebcc48c0f95419bb60e2468dd1 SetBestChain: new best=0000000000001c06fe3805f664cdb1b877bf6eebcc48c0f95419bb60e2468dd1 height=121549 log2_work=56.714377 tx=3760 date=2011-05-03 05:37:58 progress=0.000066 ProcessBlock: ACCEPTED received block 000000000000541a0744067bdb1458b6b0581da0d1989272ebb55acb67e73428 ERROR: ConnectBlock() : inputs missing/spent InvalidChainFound: invalid block=000000000000541a0744067bdb1458b6b0581da0d1989272ebb55acb67e73428 height=121550 log2_work=56.720113 date=2011-05-03 05:50:24 InvalidChainFound: current best=0000000000001c06fe3805f664cdb1b877bf6eebcc48c0f95419bb60e2468dd1 height=121549 log2_work=56.714377 date=2011-05-03 05:37:58 InvalidChainFound: invalid block=000000000000541a0744067bdb1458b6b0581da0d1989272ebb55acb67e73428 height=121550 log2_work=56.720113 date=2011-05-03 05:50:24 InvalidChainFound: current best=0000000000001c06fe3805f664cdb1b877bf6eebcc48c0f95419bb60e2468dd1 height=121549 log2_work=56.714377 date=2011-05-03 05:37:58

Hello, dear developers! Do you have some interest?

Hmm, restoring blockchain from backup, then restarting the daemon, and exactly the same things repeat: Starting from height=121550 I a see errors. What's wrong? But system restart helped. Now I'm again about height=135500.

Is anyone still experiencing this problem?

ah, yes, I am! Even running the latest 0.8.6 client - it's now 51 weeks behind the main chain as a result of this bug!

This is still a problem in V0.9.1.0. Now 55 weeks behind the main chain.

@rebroad Your database is corrupted. It is not going to magically become uncorrupted ever. Unfortunately we don't have any leads on how your database became corrupted but this is not a behavior which is generally happening to other people. Unless you have some ideas on how to further bisect down why you've had this problem there isn't anything else I can do except recommend you reindex.

@gmaxwell I've already during the duration of this issue reindexed at least ten times.

So it seems there are different issues being discussed here. @rebroad, @edam and @Schmollen see signature checks failing (where par=1 fixes it for @edam, but not for the others), causing bitcoind to mark the block that contains it (and all descendants) as invalid. This is not technically database corruption, but it does end in an incorrect state. This seems like a hardware problem, but there could be a bug too of course - though I would expect it to be reported much more often in that case. @epros 's is actual corruption, with LevelDB detecting a corruption, and after continuing noticing a missing transaction output in the chainstate. That is certainly outside of bitcoind's control (either hardware or a LevelDB problem).

I was getting a lot of failures due to either ERROR: ConnectBlock() : inputs missing/spent or LevelDB corruption, and -par=1 -reindex did not fix it for me. Until I also told my backup software (CrashPlan) to ignore ~/.bitcoin/blocks. After doing that I'm finally back in sync with the network again. Not sure what the connection might be (no idea what low-level FS operations CrashPlan might've been doing), but wanted to post it here in case it helps anyone.

Hello rebroad,

Are you still having that problem getting past block 228338, and Schmollen, on block 254354?

The reason I ask, is that I had a similar issue with block 310,271 of July 11, 2014 18:36 Z I could not build the block chain past that point, using bitcoind 0.8.6, or 0.9.1. Interestingly, a bitcoin 0.8.1 could build the block chain just fine. And then that block chain was "acceptable to the newer versions mentioned. The error seen was an infinite loop (recursion) in 0.8.6 starting in SetBestChain() but getting trapped forever in InvalidBlockFound calling InvalidChainFound() apparently forever. I didn't wait to see if it was actually doing some long crawl back up a tree? In bitcoin 0.9.1 there was a similar apparent infinite loop, though the code is different in this area, so I can't say what exactly it was repeating. Needles to say, I don't think it is hardware related, since the different versions behave differently with apparently the same data coming in. Just a straw in the wind.

Ron

I just ran 0.9.1 on the problematic machine, and now it's producing a constant flow of "ProcessBlock: ACCEPTED" messages to debug.log, so it would appear that for some reason it's now catching up.