bitcoind: chain.cpp:102: const CBlockIndex* CBlockIndex::GetAncestor(int) const: Assertion `pindexWalk->pprev' failed.

duckartes commented at 10:08 pm on November 17, 2018: none

Started to sync a fresh node, I have started and stopped it a few times during initial sync with: sudo service bitcoind stop

Using default .service file supplied by contrib/init

Start bitcoind: sudo service bitcoind start Fails and loops at: Using obfuscation key

Running from: bitcoind: bitcoind: chain.cpp:102: const CBlockIndex* CBlockIndex::GetAncestor(int) const: Assertion `pindexWalk->pprev’ failed.

Starting node to sync

Loops without continuing.

Happens every time.

Git: 2018-11-17T21:55:04Z Bitcoin Core version v0.17.99.0-35739976c (release build)

Ubuntu 19.10

4.19.1-041901-generic #201811041431 SMP Sun Nov 4 14:33:06 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

1TB spinning disk.

I don’t think Bitcoin likes being shut down while syncing through systemd?

bitcoin.conf:

0txindex=1
1server=1
2onion=127.0.0.1:9050
3externalip=$ADDRESS.onion
4discover=1
5minrelaytxfee=0
6datadir=/mnt/data/.n0de/bitcoin

debug.log

 02018-11-17T21:55:04Z Bitcoin Core version v0.17.99.0-35739976c (release build)
 12018-11-17T21:55:04Z InitParameterInteraction: parameter interaction: -whitelistforcerelay=1 -> setting -whitelistrelay=1
 22018-11-17T21:55:04Z Assuming ancestors of block 0000000000000000002e63058c023a9a1de233554f28c7b21380b6c9003f36a8 have valid signatures.
 32018-11-17T21:55:04Z Setting nMinimumChainWork=0000000000000000000000000000000000000000028822fef1c230963535a90d
 42018-11-17T21:55:04Z Using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation
 52018-11-17T21:55:04Z Using RdRand as an additional entropy source
 62018-11-17T21:55:04Z Default data directory /home/duck/.bitcoin
 72018-11-17T21:55:04Z Using data directory /home/duck/.bitcoin
 82018-11-17T21:55:04Z Config file: /home/duck/.bitcoin/bitcoin.conf (not found, skipping)
 92018-11-17T21:55:04Z Using at most 125 automatic connections (1024 file descriptors available)
102018-11-17T21:55:04Z Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
112018-11-17T21:55:04Z Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
122018-11-17T21:55:04Z Using 8 threads for script verification
132018-11-17T21:55:04Z scheduler thread start
142018-11-17T21:55:04Z HTTP: creating work queue of depth 16
152018-11-17T21:55:04Z No rpcpassword set - using random cookie authentication.
162018-11-17T21:55:04Z Generated RPC authentication cookie /home/duck/.bitcoin/.cookie
172018-11-17T21:55:04Z HTTP: starting 4 worker threads
182018-11-17T21:55:04Z Using wallet directory /home/duck/.bitcoin
192018-11-17T21:55:04Z init message: Verifying wallet(s)...
202018-11-17T21:55:04Z Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
212018-11-17T21:55:04Z Using wallet wallet.dat
222018-11-17T21:55:04Z BerkeleyEnvironment::Open: LogDir=/home/duck/.bitcoin/database ErrorFile=/home/duck/.bitcoin/db.log
232018-11-17T21:55:04Z Cache configuration:
242018-11-17T21:55:04Z * Using 2.0MiB for block index database
252018-11-17T21:55:04Z * Using 8.0MiB for chain state database
262018-11-17T21:55:04Z * Using 440.0MiB for in-memory UTXO set (plus up to 286.1MiB of unused mempool space)
272018-11-17T21:55:04Z init message: Loading block index...
282018-11-17T21:55:04Z Opening LevelDB in /home/duck/.bitcoin/blocks/index
292018-11-17T21:55:04Z Opened LevelDB successfully
302018-11-17T21:55:04Z Using obfuscation key for /home/duck/.bitcoin/blocks/index: 0000000000000000

Shows up after debug.log output when running from bitcoind command from cli: bitcoind: chain.cpp:102: const CBlockIndex* CBlockIndex::GetAncestor(int) const: Assertion `pindexWalk->pprev’ failed. Aborted (core dumped)

fanquake added the label Linux/Unix on Nov 18, 2018

duckartes commented at 0:18 am on November 18, 2018: none

Currently running reindex.

duckartes commented at 0:46 am on November 18, 2018: none

Related? #9001

I’m not pruning though.

duckartes commented at 1:00 am on November 18, 2018: none

Looks like datadir can’t be passed in .conf, or I’m doing it wrong?

I have a symlink of ~/.bitcoin to /mnt/data/.n0de/bitcoin though for clarity.

(Not sure of comment etiquette, just adding as I realize things and to limit chance of crossing streams.)

duckartes commented at 1:16 pm on November 18, 2018: none

Woke up to a:

 0
 12018-11-18T13:13:10Z net thread start
 22018-11-18T13:13:10Z dnsseed thread start
 32018-11-18T13:13:10Z addcon thread start
 42018-11-18T13:13:10Z opencon thread start
 52018-11-18T13:13:10Z init message: Done loading
 62018-11-18T13:13:10Z msghand thread start
 72018-11-18T13:13:21Z Loading addresses from DNS seeds (could take a while)
 82018-11-18T13:13:23Z 192 addresses found from DNS seeds
 92018-11-18T13:13:23Z dnsseed thread exit
102018-11-18T13:13:30Z New outbound peer connected: version: 70015, blocks=550592, peer=1
112018-11-18T13:13:35Z UpdateTip: new best=0000000000000000037ae862498d27c4a2a49eb4ab776fb5c3f3e504b328b5f1 height=404686 version=0x00000004 log2_work=84.382722 tx=118849502 date='2016-03-28T16:56:39Z' progress=0.335286 cache=1.1MiB(8001txo)
122018-11-18T13:13:35Z Fatal LevelDB error: Corruption: block checksum mismatch: /mnt/data/.n0de/bitcoin/indexes/txindex/023889.ldb
132018-11-18T13:13:35Z You can use -debug=leveldb to get more complete diagnostic messages
142018-11-18T13:13:35Z 
15
16************************
17EXCEPTION: 15dbwrapper_error       
18Fatal LevelDB error: Corruption: block checksum mismatch: /mnt/data/.n0de/bitcoin/indexes/txindex/023889.ldb       
19bitcoin in scheduler       
20
212018-11-18T13:13:35Z

duckartes commented at 8:54 pm on November 19, 2018: none

Doesn’t seem like you can stop it without doing a full sync before shutting down, or you’re gonna have to reindex it everytime…

MarcoFalke commented at 9:00 pm on November 19, 2018: member

Reindex should pick up where it stopped before you shut down

duckartes commented at 11:49 pm on November 20, 2018: none

reindex=1 starts from 0.

sipa commented at 11:56 pm on November 20, 2018: member

If pass reindex after restart, it will start over from 0.

If you don’t pass reindex after restart, it will continue where it left off.

MarcoFalke commented at 0:00 am on November 21, 2018: member

Indeed, it is a flag that you only need to pass on the first start. To make sure you don’t forget to untoggle it, I suggest restarting with reindex=0 right after it reindexed the block at height 1.

duckartes commented at 2:30 am on November 21, 2018: none

Maybe misunderstanding?

When I shut down bitcoin, I get the error, and the only way to fix it is to reindex.

If I shut it down… get the error… must reindex to start again…

(on initial sync, not fully validated chain)

duckartes commented at 5:34 am on November 21, 2018: none

I set dbcache=8192 and began a fresh reindex.

Had to shut it down before it was done.

Started it back up and it rolled forward from genesis to where I shut it off and began txindex from 1.

MarcoFalke added the label Data corruption on Nov 22, 2018

MarcoFalke commented at 6:58 pm on November 22, 2018: member

You might want to check your hardware for defects.

memtest86 to check your RAM
to check the CPU behaviour under load, use linpack or Prime95
to test your storage device use smartctl or CrystalDiskInfo

Source: https://bitcoin.stackexchange.com/a/12206

duckartes commented at 0:19 am on November 23, 2018: none

Will do.

After setting the dbcache=8192 I had no issues starting and stopping it multiple times and achieved a full sync while I was able to obtain high speed access today.

Usually I’m tethering from a phone sub 250-500KB speeds (rural.)

duckartes commented at 4:15 am on November 23, 2018: none

Also forgot to add I changed to: linux-headers-4.18.0-11-generic/disco,now 4.18.0-11.12 amd64 [installed,automatic]

The previous kernel came from: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.19.1/

So this may have had something to do with it?

Haven’t had free time to run checks, but will do and report back.

duckartes commented at 9:28 pm on November 28, 2018: none

“Some Users Have Been Hitting EXT4 File-System Corruption On Linux 4.19”

https://www.phoronix.com/scan.php?page=news_item&px=EXT4-Linux-4.19-Corruption

duckartes commented at 6:10 am on December 10, 2018: none

“The “EXT4 Corruption Issue” Has Been Fixed In Linux 4.20, Backport Pending To 4.19”

https://www.phoronix.com/scan.php?page=news_item&px=Linux-4.19-4.20-BLK-MQ-Fix

I’m assuming this was the problem as I’ve had no further issues while on 4.18 (although I have not tried syncing from genesis again.)

duckartes closed this on Dec 10, 2018

rebroad commented at 10:33 am on April 24, 2020: contributor

I saw this today when I tried to shutdown bitcoin-qt during the LoadBlockIndexDB phase, while parsing vSortedByHeight in the BuildSkip() function.

DrahtBot locked this on Feb 15, 2022

bitcoind: chain.cpp:102: const CBlockIndex* CBlockIndex::GetAncestor(int) const: Assertion `pindexWalk->pprev’ failed. #14750