Bus error (core dumped) after "Using obfuscation key for" log line

alevchuk commented at 12:47 pm on September 18, 2019: none

bitcoind is crashing on startup

Expected behavior

It worked fine on this same hardware for years

Actual behavior

02019-09-18T12:43:58Z LevelDB using max_open_files=1000 (default=1000)
12019-09-18T12:43:58Z Opening LevelDB in /home/bitcoin/.bitcoin/blocks/index
22019-09-18T12:43:58Z leveldb: Recovering log [#1159](/bitcoin-bitcoin/1159/)
32019-09-18T12:44:00Z leveldb: Delete type=0 [#1159](/bitcoin-bitcoin/1159/)
42019-09-18T12:44:00Z leveldb: Delete type=3 [#1158](/bitcoin-bitcoin/1158/)
52019-09-18T12:44:00Z Opened LevelDB successfully
62019-09-18T12:44:00Z Using obfuscation key for /home/bitcoin/.bitcoin/blocks/index: 0000000000000000
7Bus error (core dumped)

System information

self-compiled from this github repo from b3067f4338d901394fb0237567cb5e19ccee5f8a

2019-09-18T02:44:13Z Bitcoin Core version v0.18.99.0-b3067f4 (release build)

Linux 4.15.0-42-generic #45~16.04.1+8.0trisquel3 SMP Tue Dec 4 11:28:55 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

I have bitcoind and the data on BTRFS mirrored across two drives. BTFS Scrub found errors yet all were correctable. When re-running BTRFS scrub 0 errors are reported.

 0gdb bitcoind
 1...
 2Reading symbols from bitcoind...done.
 3(gdb) r
 4Starting program: /mnt/btrfs/bitcoin/bin/bin/bitcoind
 5[Thread debugging using libthread_db enabled]
 6...
 72019-09-18T03:28:55Z Opened LevelDB successfully
 82019-09-18T12:43:58Z Opening LevelDB in /home/bitcoin/.bitcoin/blocks/index
 92019-09-18T12:43:58Z leveldb: Recovering log [#1159](/bitcoin-bitcoin/1159/)
102019-09-18T12:44:00Z leveldb: Delete type=0 [#1159](/bitcoin-bitcoin/1159/)
112019-09-18T12:44:00Z leveldb: Delete type=3 [#1158](/bitcoin-bitcoin/1158/)
12
132019-09-18T03:28:55Z Using obfuscation key for /home/bitcoin/.bitcoin/blocks/index: 0000000000000000
14
15Thread 1 "bitcoin-init" received signal SIGBUS, Bus error.
16leveldb::ReadBlock (file=0x55555dadb740, options=..., handle=..., result=result@entry=0x7fffffffa100) at leveldb/table/format.cc:91
1791          const uint32_t crc = crc32c::Unmask(DecodeFixed32(data + n + 1));
18
19(gdb) bt
20[#0](/bitcoin-bitcoin/0/)  leveldb::crc32c::Unmask (masked_crc=<optimized out>) at ./leveldb/util/crc32c.h:39
21[#1](/bitcoin-bitcoin/1/)  leveldb::ReadBlock (file=0x55555dadb740, options=..., handle=..., result=result@entry=0x7fffffffa100) at leveldb/table/format.cc:91
22[#2](/bitcoin-bitcoin/2/)  0x000055555590854c in leveldb::Table::BlockReader (arg=0x55555a0ef0d0, options=..., index_value=...) at leveldb/table/table.cc:189
23[#3](/bitcoin-bitcoin/3/)  0x000055555590b2c6 in leveldb::(anonymous namespace)::TwoLevelIterator::InitDataBlock (this=this@entry=0x55555dadb930)
24    at leveldb/table/two_level_iterator.cc:165
25[#4](/bitcoin-bitcoin/4/)  0x000055555590b370 in leveldb::(anonymous namespace)::TwoLevelIterator::SkipEmptyDataBlocksForward (this=0x55555dadb930)
26    at leveldb/table/two_level_iterator.cc:133
27[#5](/bitcoin-bitcoin/5/)  0x000055555590b545 in leveldb::IteratorWrapper::Next (this=0x555555e15b30) at ./leveldb/table/iterator_wrapper.h:45
28[#6](/bitcoin-bitcoin/6/)  leveldb::(anonymous namespace)::TwoLevelIterator::Next (this=0x555555e15ac0) at leveldb/table/two_level_iterator.cc:114
29[#7](/bitcoin-bitcoin/7/)  0x0000555555905d47 in leveldb::IteratorWrapper::Next (this=<optimized out>) at ./leveldb/table/iterator_wrapper.h:45
30[#8](/bitcoin-bitcoin/8/)  leveldb::(anonymous namespace)::MergingIterator::Next (this=0x555555e15b80) at leveldb/table/merger.cc:81
31[#9](/bitcoin-bitcoin/9/)  0x00005555558f2540 in leveldb::(anonymous namespace)::DBIter::FindNextUserEntry (this=0x555555e15c70, skipping=true, skip=0x555555e15cc0)
32    at leveldb/db/db_iter.cc:199
33[#10](/bitcoin-bitcoin/10/) 0x00005555556dc32b in CBlockTreeDB::LoadBlockIndexGuts(Consensus::Params const&, std::function<CBlockIndex* (uint256 const&)>) (this=<optimized out>,
34    consensusParams=..., insertBlockIndex=...) at txdb.cpp:280
35[#11](/bitcoin-bitcoin/11/) 0x000055555572f3ee in BlockManager::LoadBlockIndex (this=this@entry=0x555555c77ee0 <(anonymous namespace)::g_blockman>, consensus_params=...,
36    blocktree=..., block_index_candidates=std::set with 0 elements) at validation.cpp:3820
37[#12](/bitcoin-bitcoin/12/) 0x000055555572fa71 in LoadBlockIndexDB (chainparams=...) at validation.cpp:3883
38[#13](/bitcoin-bitcoin/13/) 0x0000555555730445 in LoadBlockIndex (chainparams=...) at validation.cpp:4344
39[#14](/bitcoin-bitcoin/14/) 0x00005555555a6ca9 in AppInitMain (interfaces=...) at init.cpp:1497
40[#15](/bitcoin-bitcoin/15/) 0x000055555558c1e9 in AppInit (argc=<optimized out>, argv=<optimized out>) at bitcoind.cpp:164
41[#16](/bitcoin-bitcoin/16/) 0x000055555558436f in main (argc=1, argv=0x7fffffffe648) at bitcoind.cpp:194

alevchuk added the label Bug on Sep 18, 2019

fanquake added the label Linux/Unix on Sep 18, 2019

MarcoFalke commented at 2:10 pm on September 18, 2019: member

I could imagine that the disk errors corrupted (other parts of) the database earlier. So while some of them could be fixed, the db might stay corrupted.

alevchuk commented at 1:02 am on September 19, 2019: none

Would a friendlier message be useful, e.g. “leveldb database is corrupted, go to wiki page X”?

MarcoFalke commented at 1:29 pm on September 19, 2019: member

Yeah, but I am not sure if SIGBUS from leveldb can be caught and thrown as an error message instead.

laanwj added the label Upstream on Sep 20, 2019

laanwj commented at 10:04 am on September 20, 2019: member

I’m sure it’s possible for leveldb to recognize this kind of corruption, instead of blindly trusting what is on disk (and as here, doing an unaligned read/write to memory). However this is an upstream problem.

I’d suggest creating a minimum reproducible example and filing an issue with leveldb/leveldb.

MarcoFalke closed this on Apr 26, 2020

DrahtBot locked this on Feb 15, 2022

Bus error (core dumped) after “Using obfuscation key for” log line #16904