bitcoind: chain.cpp:102: const CBlockIndex* CBlockIndex::GetAncestor(int) const: Assertion `pindexWalk->pprev' failed. #14750

<!-- Describe the issue -->

Started to sync a fresh node, I have started and stopped it a few times during initial sync with: sudo service bitcoind stop

Using default .service file supplied by contrib/init

Start bitcoind: sudo service bitcoind start Fails and loops at: Using obfuscation key

Running from: bitcoind: bitcoind: chain.cpp:102: const CBlockIndex* CBlockIndex::GetAncestor(int) const: Assertion `pindexWalk->pprev' failed.

<!--- What behavior did you expect? -->

Starting node to sync

<!--- What was the actual behavior (provide screenshots if the issue is GUI-related)? -->

Loops without continuing.

<!--- How reliably can you reproduce the issue, what are the steps to do so? -->

Happens every time.

<!-- What version of Bitcoin Core are you using, where did you get it (website, self-compiled, etc)? -->

Git: 2018-11-17T21:55:04Z Bitcoin Core version v0.17.99.0-35739976c (release build)

<!-- What type of machine are you observing the error on (OS/CPU and disk type)? -->

Ubuntu 19.10

4.19.1-041901-generic #201811041431 SMP Sun Nov 4 14:33:06 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

1TB spinning disk.

<!-- Any extra information that might be useful in the debugging process. -->

I don't think Bitcoin likes being shut down while syncing through systemd?

<!--- This is normally the contents of a `debug.log` or `config.log` file. Raw text or a link to a pastebin type site are preferred. -->

bitcoin.conf:

txindex=1
server=1
onion=127.0.0.1:9050
externalip=$ADDRESS.onion
discover=1
minrelaytxfee=0
datadir=/mnt/data/.n0de/bitcoin

debug.log

2018-11-17T21:55:04Z Bitcoin Core version v0.17.99.0-35739976c (release build)
2018-11-17T21:55:04Z InitParameterInteraction: parameter interaction: -whitelistforcerelay=1 -> setting -whitelistrelay=1
2018-11-17T21:55:04Z Assuming ancestors of block 0000000000000000002e63058c023a9a1de233554f28c7b21380b6c9003f36a8 have valid signatures.
2018-11-17T21:55:04Z Setting nMinimumChainWork=0000000000000000000000000000000000000000028822fef1c230963535a90d
2018-11-17T21:55:04Z Using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation
2018-11-17T21:55:04Z Using RdRand as an additional entropy source
2018-11-17T21:55:04Z Default data directory /home/duck/.bitcoin
2018-11-17T21:55:04Z Using data directory /home/duck/.bitcoin
2018-11-17T21:55:04Z Config file: /home/duck/.bitcoin/bitcoin.conf (not found, skipping)
2018-11-17T21:55:04Z Using at most 125 automatic connections (1024 file descriptors available)
2018-11-17T21:55:04Z Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
2018-11-17T21:55:04Z Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
2018-11-17T21:55:04Z Using 8 threads for script verification
2018-11-17T21:55:04Z scheduler thread start
2018-11-17T21:55:04Z HTTP: creating work queue of depth 16
2018-11-17T21:55:04Z No rpcpassword set - using random cookie authentication.
2018-11-17T21:55:04Z Generated RPC authentication cookie /home/duck/.bitcoin/.cookie
2018-11-17T21:55:04Z HTTP: starting 4 worker threads
2018-11-17T21:55:04Z Using wallet directory /home/duck/.bitcoin
2018-11-17T21:55:04Z init message: Verifying wallet(s)...
2018-11-17T21:55:04Z Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2018-11-17T21:55:04Z Using wallet wallet.dat
2018-11-17T21:55:04Z BerkeleyEnvironment::Open: LogDir=/home/duck/.bitcoin/database ErrorFile=/home/duck/.bitcoin/db.log
2018-11-17T21:55:04Z Cache configuration:
2018-11-17T21:55:04Z * Using 2.0MiB for block index database
2018-11-17T21:55:04Z * Using 8.0MiB for chain state database
2018-11-17T21:55:04Z * Using 440.0MiB for in-memory UTXO set (plus up to 286.1MiB of unused mempool space)
2018-11-17T21:55:04Z init message: Loading block index...
2018-11-17T21:55:04Z Opening LevelDB in /home/duck/.bitcoin/blocks/index
2018-11-17T21:55:04Z Opened LevelDB successfully
2018-11-17T21:55:04Z Using obfuscation key for /home/duck/.bitcoin/blocks/index: 0000000000000000

Shows up after debug.log output when running from bitcoind command from cli: bitcoind: chain.cpp:102: const CBlockIndex* CBlockIndex::GetAncestor(int) const: Assertion `pindexWalk->pprev' failed. Aborted (core dumped)

Currently running reindex.

Related? #9001

I'm not pruning though.

Looks like datadir can't be passed in .conf, or I'm doing it wrong?

I have a symlink of ~/.bitcoin to /mnt/data/.n0de/bitcoin though for clarity.

(Not sure of comment etiquette, just adding as I realize things and to limit chance of crossing streams.)

Woke up to a:

2018-11-18T13:13:10Z net thread start
2018-11-18T13:13:10Z dnsseed thread start
2018-11-18T13:13:10Z addcon thread start
2018-11-18T13:13:10Z opencon thread start
2018-11-18T13:13:10Z init message: Done loading
2018-11-18T13:13:10Z msghand thread start
2018-11-18T13:13:21Z Loading addresses from DNS seeds (could take a while)
2018-11-18T13:13:23Z 192 addresses found from DNS seeds
2018-11-18T13:13:23Z dnsseed thread exit
2018-11-18T13:13:30Z New outbound peer connected: version: 70015, blocks=550592, peer=1
2018-11-18T13:13:35Z UpdateTip: new best=0000000000000000037ae862498d27c4a2a49eb4ab776fb5c3f3e504b328b5f1 height=404686 version=0x00000004 log2_work=84.382722 tx=118849502 date='2016-03-28T16:56:39Z' progress=0.335286 cache=1.1MiB(8001txo)
2018-11-18T13:13:35Z Fatal LevelDB error: Corruption: block checksum mismatch: /mnt/data/.n0de/bitcoin/indexes/txindex/023889.ldb
2018-11-18T13:13:35Z You can use -debug=leveldb to get more complete diagnostic messages
2018-11-18T13:13:35Z 

************************
EXCEPTION: 15dbwrapper_error       
Fatal LevelDB error: Corruption: block checksum mismatch: /mnt/data/.n0de/bitcoin/indexes/txindex/023889.ldb       
bitcoin in scheduler       

2018-11-18T13:13:35Z 

Doesn't seem like you can stop it without doing a full sync before shutting down, or you're gonna have to reindex it everytime...

Reindex should pick up where it stopped before you shut down

reindex=1 starts from 0.

If pass reindex after restart, it will start over from 0.

If you don't pass reindex after restart, it will continue where it left off.

Indeed, it is a flag that you only need to pass on the first start. To make sure you don't forget to untoggle it, I suggest restarting with reindex=0 right after it reindexed the block at height 1.

Maybe misunderstanding?

When I shut down bitcoin, I get the error, and the only way to fix it is to reindex.

If I shut it down... get the error... must reindex to start again...

(on initial sync, not fully validated chain)

I set dbcache=8192 and began a fresh reindex.

Had to shut it down before it was done.

Started it back up and it rolled forward from genesis to where I shut it off and began txindex from 1.

You might want to check your hardware for defects.

• memtest86 to check your RAM
• to check the CPU behaviour under load, use linpack or Prime95
• to test your storage device use smartctl or CrystalDiskInfo

Source: https://bitcoin.stackexchange.com/a/12206

Will do.

After setting the dbcache=8192 I had no issues starting and stopping it multiple times and achieved a full sync while I was able to obtain high speed access today.

Usually I'm tethering from a phone sub 250-500KB speeds (rural.)

Also forgot to add I changed to: linux-headers-4.18.0-11-generic/disco,now 4.18.0-11.12 amd64 [installed,automatic]

The previous kernel came from: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.19.1/

So this may have had something to do with it?

Haven't had free time to run checks, but will do and report back.

"Some Users Have Been Hitting EXT4 File-System Corruption On Linux 4.19"

https://www.phoronix.com/scan.php?page=news_item&px=EXT4-Linux-4.19-Corruption

"The "EXT4 Corruption Issue" Has Been Fixed In Linux 4.20, Backport Pending To 4.19"

https://www.phoronix.com/scan.php?page=news_item&px=Linux-4.19-4.20-BLK-MQ-Fix

I'm assuming this was the problem as I've had no further issues while on 4.18 (although I have not tried syncing from genesis again.)

I saw this today when I tried to shutdown bitcoin-qt during the LoadBlockIndexDB phase, while parsing vSortedByHeight in the BuildSkip() function.