VerifySignature failed on publicly accepted transaction

glondu commented at 10:20 am on December 3, 2015: contributor

Hello,

After a reindexing (done after a database inconsistency error), bitcoind (Debian package 0.11.1-1) is stuck at block 337586. I get the following error:

02015-12-03 10:11:33 ERROR: CScriptCheck(): 79f82443a1d1c3ecd440db7880d26abd70f8c522a8b57ff182340542d4e6a8e6:194 VerifySignature failed: Opcode missing or not understood
12015-12-03 10:11:33 InvalidChainFound: invalid block=000000000000000016b31eefe6fa6eb521a972407d0385584acbeebd016b7126  height=337587  log2_work=81.931573  date=2015-01-05 10:11:44
22015-12-03 10:11:33 InvalidChainFound:  current best=000000000000000016ff08b25ad3d20c3ae1be16fd551a1627539ad53e8b4996  height=337586  log2_work=81.931518  date=2015-01-05 10:04:46
32015-12-03 10:11:33 ERROR: ConnectTip(): ConnectBlock 000000000000000016b31eefe6fa6eb521a972407d0385584acbeebd016b7126 failed
42015-12-03 10:11:33 InvalidChainFound: invalid block=000000000000000016b31eefe6fa6eb521a972407d0385584acbeebd016b7126  height=337587  log2_work=81.931573  date=2015-01-05 10:11:44
52015-12-03 10:11:33 InvalidChainFound:  current best=000000000000000016ff08b25ad3d20c3ae1be16fd551a1627539ad53e8b4996  height=337586  log2_work=81.931518  date=2015-01-05 10:04:46

I get it each time I issue “bitcoin-cli reconsiderblock 000000000000000016b31eefe6fa6eb521a972407d0385584acbeebd016b7126”.

According to blockchain.info, 79f82443a1d1c3ecd440db7880d26abd70f8c522a8b57ff182340542d4e6a8e6 is a valid transaction and contained in block 000000000000000016b31eefe6fa6eb521a972407d0385584acbeebd016b7126 which is part of the main chain.

jonasschnelli added the label Block storage on Dec 3, 2015

laanwj commented at 12:04 pm on December 3, 2015: member

I’m not aware of any previously reported issues with that specific block. I think the block data on your disk was corrupted, usually this is an hardware issue. To check, can you upload the output of ‘bitcoin-cli getblock 000000000000000016b31eefe6fa6eb521a972407d0385584acbeebd016b7126 false’ somewhere?

sipa commented at 12:09 pm on December 3, 2015: member

Sounds more like UTXO corruption. If the block was corrupted, its merkle root or PoW would mismatch.

laanwj commented at 12:12 pm on December 3, 2015: member

@sipa right

sipa commented at 12:13 pm on December 3, 2015: member

@glondu Feel like reindexing again with -par=1 turned on?

glondu commented at 1:15 pm on December 3, 2015: contributor

@sipa I launched “bitcoind -reindex -par=1”… Let’s see…

glondu commented at 1:49 pm on December 4, 2015: contributor

It failed with:

 0[...]
 12015-12-04 07:56:56 UpdateTip: new best=00000000000000005371e4fd5084e16e03cecaaf7ce565c4c970102c1fac776a  height=287878  log2_work=76.880053  tx=33690994  date=2014-02-26 08:24:37 progress=0.154850  cache=50.2M$
 22015-12-04 07:56:57 UpdateTip: new best=00000000000000002ffeebd7efd5795f53ea9430de7023b7e8a4a94529fc95e3  height=287879  log2_work=76.880192  tx=33691328  date=2014-02-26 08:33:37 progress=0.154852  cache=50.7M$
 32015-12-04 07:56:58 UpdateTip: new best=00000000000000013a0e370b55e40478f7886b91f22ba436a85b94fb9b180a63  height=287880  log2_work=76.880332  tx=33691630  date=2014-02-26 08:39:52 progress=0.154853  cache=53.4M$
 42015-12-04 07:56:58 Corruption: block checksum mismatch
 52015-12-04 07:56:58 *** System error while flushing: Database corrupted
 62015-12-04 07:56:58 Error: Error: A fatal internal error occurred, see debug.log for details
 72015-12-04 07:56:58 ERROR: ProcessNewBlock: ActivateBestChain failed
 82015-12-04 07:56:58 Loaded 22 blocks from external file in 47433ms
 92015-12-04 07:56:58 Reindexing block file blk00072.dat...
102015-12-04 07:56:59 addcon thread interrupt
112015-12-04 07:56:59 net thread interrupt
122015-12-04 07:56:59 opencon thread interrupt
132015-12-04 07:56:59 msghand thread interrupt
142015-12-04 07:56:59 scheduler thread interrupt
152015-12-04 07:56:59 Shutdown: In progress...
162015-12-04 07:56:59 RPCAcceptHandler: Error: Operation canceled
172015-12-04 07:56:59 StopNode()
182015-12-04 07:56:59 Corruption: block checksum mismatch
192015-12-04 07:56:59 *** System error while flushing: Database corrupted
202015-12-04 07:56:59 Error: Error: A fatal internal error occurred, see debug.log for details
212015-12-04 07:57:00 Shutdown: done

Then, I launched “bitcoind -par=1” and now, it is stuck at block 290381. I get numerous “ERROR: AcceptToMemoryPool: non-final”. “bitcoin-cli reconsiderblock 000000000000000031665933b2dc474d396f1a91d0813ea789e8347e4574cdec” returns “Block not found”.

Then, I stopped with “bitcoin-cli stop”, and I got the following messages:

 0[...]
 12015-12-04 13:08:10 ERROR: AcceptToMemoryPool: non-final
 22015-12-04 13:08:16 scheduler thread interrupt
 32015-12-04 13:08:16 addcon thread interrupt
 42015-12-04 13:08:16 msghand thread interrupt
 52015-12-04 13:08:16 opencon thread interrupt
 62015-12-04 13:08:16 net thread interrupt
 72015-12-04 13:08:16 Shutdown: In progress...
 82015-12-04 13:08:16 StopNode()
 92015-12-04 13:08:17 Corruption: block checksum mismatch
102015-12-04 13:08:17 *** System error while flushing: Database corrupted
112015-12-04 13:08:17 Error: Error: A fatal internal error occurred, see debug.log for details
122015-12-04 13:08:17 Shutdown: done

Then, I launched “bitcoind -par=1” again and it is still running…

FYI, that instance has a non-empty wallet and last transaction dates back to block 346184. Memtest86+ reports no errors, and I see no disk errors in logs.

jgarzik commented at 6:34 pm on December 4, 2015: contributor

Definitely still looks like corruption below bitcoind level. I would delete all databases and blocks and download everything from scratch…

glondu commented at 10:01 am on December 7, 2015: contributor

Now, bitcoind is stuck at block 302040 and I get:

02015-12-07 09:55:55 ERROR: CScriptCheck(): 630751d2572851628f09abfc52d12bb203e87ca47dc747384fc8e3a7fb72ef8e:6 VerifySignature failed: Opcode missing or not understood
12015-12-07 09:55:55 InvalidChainFound: invalid block=00000000000000005da87f6d202065faeb243b7fe8292a8322266eef715f0414  height=302041  log2_work=78.736322  date=2014-05-22 08:23:20
22015-12-07 09:55:55 InvalidChainFound:  current best=0000000000000000357b09e5271313fc39517f35a63146e92922c6e47f7e76b7  height=302040  log2_work=78.736213  date=2014-05-22 08:20:15
32015-12-07 09:55:55 ERROR: ConnectTip(): ConnectBlock 00000000000000005da87f6d202065faeb243b7fe8292a8322266eef715f0414 failed
42015-12-07 09:55:55 InvalidChainFound: invalid block=00000000000000005da87f6d202065faeb243b7fe8292a8322266eef715f0414  height=302041  log2_work=78.736322  date=2014-05-22 08:23:20
52015-12-07 09:55:55 InvalidChainFound:  current best=0000000000000000357b09e5271313fc39517f35a63146e92922c6e47f7e76b7  height=302040  log2_work=78.736213  date=2014-05-22 08:20:15

each time I do “bitcoin-cli reconsiderblock 0000000000000000357b09e5271313fc39517f35a63146e92922c6e47f7e76b7”.

luke-jr commented at 10:11 am on December 7, 2015: member

(Note you may wish not to use defective hardware for a wallet)

MarcoFalke commented at 10:26 am on December 7, 2015: member

@glondu What is bitcoin-cli getblock 0000000000000000357b09e5271313fc39517f35a63146e92922c6e47f7e76b7 false?

sipa commented at 10:41 am on December 7, 2015: member

You’d need to go look at the UTXO entry being spent that fails.

laanwj commented at 12:27 pm on December 7, 2015: member

Definitely still looks like corruption below bitcoind level. I would delete all databases and blocks and download everything from scratch…

Agree. Every time I’ve seen this happening at random blocks it was corruption at some level. Memory, overheating CPU (or motherboard/bus) are most likely culprit - disk-level corruptions are possible but usually caught as leveldb checksum errors.

Closing this issue as there’s nothing we can do about this.

laanwj closed this on Dec 7, 2015

laanwj added the label Data corruption on Feb 9, 2016

MarcoFalke locked this on Sep 8, 2021

VerifySignature failed on publicly accepted transaction #7161