After upgrading to 0.10.2 bitcoind hangs at Opened LevelDB successfully #6386

On ubuntu Trusty, I get the following after upgrading to 0.10.2 and running bitcoind -daemon

2015-07-07 07:27:49 init message: Loading block index... 2015-07-07 07:27:49 Opening LevelDB in /home/ubuntu/.bitcoin/blocks/index 2015-07-07 07:27:49 Opened LevelDB successfully 2015-07-07 07:27:49 Opening LevelDB in /home/ubuntu/.bitcoin/chainstate 2015-07-07 07:27:49 Opened LevelDB successfully

And it just hangs there, I upgraded another node and did not have this issue. Is there any way to debug what is wrong. Should I do a reindex?

Strange. The line after that should be:

2015-06-26 02:18:36 LoadBlockIndexDB: last block file = 287

So possibly it gets stuck while accessing the block index?

Can you try to start bitcoind in gdb, then Ctrl-C when it is stuck, to see if you can get a backtrace?

Thanks @laanwj , I am not sure if the below helps, but seems like some sort of Segmentation fault. (gdb) run Starting program: /usr/bin/bitcoind [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x7ffff554b700 (LWP 1937)] [New Thread 0x7ffff4d4a700 (LWP 1938)] [New Thread 0x7ffff4549700 (LWP 1939)] [New Thread 0x7ffff3d48700 (LWP 1940)] [New Thread 0x7ffff3547700 (LWP 1941)]

Program received signal SIGSEGV, Segmentation fault. 0x00005555555c91e2 in ?? () (gdb) bt #0 0x00005555555c91e2 in ?? () #1 0x00005555555c92fa in ?? () #2 0x00005555555d3cb0 in ?? () #3 0x00005555555d4845 in ?? () #4 0x00005555555a1681 in ?? () #5 0x0000555555587cb0 in ?? () #6 0x000055555557d5af in ?? () #7 0x00007ffff5979ec5 in __libc_start_main (main=0x55555557d580, argc=1, argv=0x7fffffffe698, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe688) at libc-start.c:287 #8 0x0000555555586eb6 in ?? () Starting program: /usr/bin/bitcoind [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x7ffff554b700 (LWP 1915)] [New Thread 0x7ffff4d4a700 (LWP 1916)] [New Thread 0x7ffff4549700 (LWP 1917)] [New Thread 0x7ffff3d48700 (LWP 1918)] [New Thread 0x7ffff3547700 (LWP 1919)]

Program received signal SIGSEGV, Segmentation fault. 0x00005555555c91e2 in ?? ()

Anything that I can do to analyze that further? Is it anything to do with the number of threads or cpu specs?

Whoa. Is that the bitcoind from bitcoin.org's 0.10.2 64-bit linux release? If so I can try to convert those addresses to symbol names.

Bitcoin version v0.10.2.0-g16f4560 , its from the ppa. The machine is 64 bit.

I can't do the same for the ppa version. Can you please try with the release from bitcoin.org, or, alternatively, compile from tag v0.10.2 yourself from git.

Hey, Thanks @laanwj for helping out

Here is the op from bitcoin org one: Starting program: /home/ubuntu/bitcoin_org/bitcoin-0.10.2/bin/bitcoind [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x7ffff6bca700 (LWP 2250)] [New Thread 0x7ffff63c9700 (LWP 2251)] [New Thread 0x7ffff5bc8700 (LWP 2252)] [New Thread 0x7ffff53c7700 (LWP 2253)] [New Thread 0x7ffff4bc6700 (LWP 2254)]

Program received signal SIGSEGV, Segmentation fault. 0x00005555555f722a in ?? () (gdb) bt #0 0x00005555555f722a in ?? () #1 0x00005555555f72db in ?? () #2 0x000055555560341f in ?? () #3 0x00005555555c9fa7 in ?? () #4 0x00005555555b1804 in ?? () #5 0x00005555555bda1f in ?? () #6 0x00007ffff6becec5 in __libc_start_main (main=0x5555555bd9f0, argc=1, argv=0x7fffffffe638, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe628) at libc-start.c:287 #7 0x00005555555b0769 in ?? () #8 0x00007fffffffe628 in ?? () #9 0x000000000000001c in ?? () #10 0x0000000000000001 in ?? () #11 0x00007fffffffe855 in ?? () #12 0x0000000000000000 in ?? ()

Building from source: gives the following op: Starting program: /usr/local/bin/bitcoind [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x7ffff5754700 (LWP 2621)] [New Thread 0x7ffff4f53700 (LWP 2622)] [New Thread 0x7ffff4752700 (LWP 2623)] [New Thread 0x7ffff3f51700 (LWP 2624)] [New Thread 0x7ffff3750700 (LWP 2625)]

Program received signal SIGSEGV, Segmentation fault. 0x00005555555c8cd2 in CBlockIndex::GetAncestor (this=<optimized out>, height=213168) at main.cpp:2718 2718 pindexWalk = pindexWalk->pprev;

(gdb) bt #0 0x00005555555c8cd2 in CBlockIndex::GetAncestor (this=<optimized out>, height=213168) at main.cpp:2718 #1 0x00005555555c8dea in CBlockIndex::BuildSkip (this=this@entry=0x5555572c11e0) at main.cpp:2733 #2 0x00005555555d3780 in LoadBlockIndexDB () at main.cpp:2910 #3 0x00005555555d4395 in LoadBlockIndex () at main.cpp:3074 #4 0x00005555555a12c1 in AppInit2 (threadGroup=...) at init.cpp:1025 #5 0x0000555555587990 in AppInit (argc=<optimized out>, argv=<optimized out>) at bitcoind.cpp:148 #6 0x000055555557d28f in main (argc=1, argv=0x7fffffffe678) at bitcoind.cpp:185

Started a reindex.

@seigneur Looks like a weird database corruption issue. It is a bug that this raises a segmentation fault, but a reindex is what I'd have recommended as well to get back to a reliable state.

@laanwj Thanks, will report back after the reIndex. Should set everything back.