-reindex always end with fail (0.9.2) #4362

issue analogic openend this issue on June 18, 2014
  1. analogic commented at 8:19 am on June 18, 2014: none

    It always end with this messages around block 200k

    2014-06-18 08:00:42 LevelDB read failure: Invalid argument: not an sstable (bad magic number) 2014-06-18 08:00:42 Invalid argument: not an sstable (bad magic number) 2014-06-18 08:00:42 *** System error: Unknown database error 2014-06-18 08:00:42 Error: System error: Unknown database error 2014-06-18 08:00:42 ERROR: ProcessBlock() : AcceptBlock FAILED

    I am unable to run my services for a week because this hell and its not first bug i found (https://github.com/bitcoin/bitcoin/issues/4356) WTF

  2. laanwj added the label P2P on Jun 18, 2014
  3. laanwj removed the label P2P on Jun 18, 2014
  4. laanwj added the label UTXO Db and Indexes on Jun 18, 2014
  5. laanwj commented at 8:25 am on June 18, 2014: member

    That’s weird. A reindex should never cause leveldb error, as it should remove and rebuild leveldb databases. It’s there to fix corrupted databases.

    What platform? Do you store the database or block data on a non-standard filesystem? (network, non-native, etc)

  6. analogic commented at 8:35 am on June 18, 2014: none
    Hmm. Freebsd 10 with local mirrored disc. Maybe i am hitting some system limmits (but its standard server 12gb ram, 1TB free space) or I have theory that some rcp commands to reindexing bitcoind is doing that. I will try reindex from backup with all services blocked - blockchain was fataly damaged after 5 reindexes - it kept spaming log with “ProcessBlock: ACCEPTED” (on block ~180k) only, no other message.
  7. laanwj commented at 9:20 am on June 18, 2014: member

    Getting Processblock: ACCEPTED spam is a symptom of a corrupted leveldb UTXO database or block index (*.sst). I had the same issue after copying leveldb databases from an ARM system to a AMD64 system and vice versa. Then again – -reindex solved it in all cases.

    If it doesn’t, a corruption is happening on the fly.

    The blocks files (blk*.dat) themselves getting corrupted will never result in that issue, although it may be possible that that could result in orphans while reindexing. But this specific problem clearly has to do with the leveldb database.

    If this was windows I’d have suggested it could be the virus scanner deleting files under the instance (#4069).

  8. analogic commented at 9:41 am on June 18, 2014: none

    Same result with disabled wallet. Now on block 204k

    It was freshly loaded from bootstrap - and loading bootstrap was with this errors too, but i ignored it, reindexed and loaded rest from network (happend in 0.8.6 too, so i tried 0.9.2). After loading it seemed all ok, bitcoind was working, then i stoped deamon normaly and after start it was corrupted again. What should i do to identify bug?

  9. analogic commented at 9:52 am on June 18, 2014: none

    Again :-(

    2014-06-18 09:47:33 UpdateTip: new best=000000000000017469ae4c96c606d588042a1a82c0d0b7086a29058233676754 height=209885 log2_work=69.087637 tx=932 2014-06-18 09:47:33 ProcessBlock: ACCEPTED 2014-06-18 09:47:33 got inventory: tx b3a069e0e402a262a534a94d756ef8f31ff6527094f03878f914ade427895900 new 2014-06-18 09:47:33 got inventory: tx 578833feb2bed326d552a57e2058497993afcc777512036bcedc6171f5c6b93f new 2014-06-18 09:47:33 got inventory: tx 77404f853f5c2a0319451efe6d9808263d08affe363d9b69538ff5b38f65e426 new 2014-06-18 09:47:33 got inventory: tx 2f6c2e54df0e7e8404db3bad4983fc71622dce4e1c2186ed982996ffc7b63750 new 2014-06-18 09:47:33 got inventory: tx 0e25ae67e1fb062769092378a7e21d63f92c56e806aa7f65f3bddc44bc408fec new 2014-06-18 09:47:33 got inventory: tx e9948e9a1359c4055cb9366debf6d223f8c879d8aa44408436114b18414ac990 new 2014-06-18 09:47:33 sending: addr (31 bytes) 2014-06-18 09:47:33 sending: inv (1333 bytes) 2014-06-18 09:47:33 sending: inv (1333 bytes) 2014-06-18 09:47:33 sending: inv (1333 bytes) 2014-06-18 09:47:33 sending: inv (325 bytes) 2014-06-18 09:47:33 Committing 171675 changed transactions to coin database… 2014-06-18 09:47:33 Invalid argument: not an sstable (bad magic number) 2014-06-18 09:47:33 *** System error: Unknown database error 2014-06-18 09:47:33 Error: System error: Unknown database error 2014-06-18 09:47:33 ERROR: ProcessBlock() : AcceptBlock FAILED

  10. laanwj commented at 11:16 am on June 18, 2014: member
    Freebsd - hmm. Could be the macosx leveldb corruption issue in disguise? May be that it needs the same special-cases for fbsd as for mac.
  11. analogic commented at 11:59 am on June 19, 2014: none

    Yes it could be. Its seems problem with LevelDB only.

    I am not experienced in C++ especialy in these memory problems but LevelDB 1.17 included in last release have completely different env_posix.c - no unmap in write (if i understand http://hackingdistributed.com/2013/11/27/bitcoin-leveldb/ correctly), is last release working in osx ok?

  12. laanwj commented at 12:22 pm on June 19, 2014: member
    Yes it works fine on OSX. But as said, there are conditions specifically for OSX/Darwin that may have to be active for FreeBSD as well. I don’t think there is anyone regularly testing LevelDB on FreeBSD.
  13. analogic commented at 8:21 pm on June 19, 2014: none

    I found it!!!ยง

    There is something wrong with mmap on FreeBSD also. Simpliest solution is to force LevelDB use PosixRandomAccessFile instead of PosixMmapReadableFile.

    Commenting lines 322 to 336 in file src/leveldb/util/env_posix.cc will do the work

    (ticket in leveldb http://code.google.com/p/leveldb/issues/detail?id=241&thanks=241&ts=1403208937)

  14. laanwj commented at 5:56 am on June 20, 2014: member
    Good catch!
  15. ghost commented at 8:31 am on June 22, 2014: none
    Compiled with commented-out lines as per @analogic ’s comment. I just started running bitcoind 0.9.2 with txindex=1. If I spot this problem again I’ll update this issue.
  16. rsdcastro commented at 9:08 pm on June 30, 2014: none

    @analogic @rippler Would you have some code that repros the LevelDb issue that I could look at to?

    I also added some questions to the LevelDb bug you filed: https://code.google.com/p/leveldb/issues/detail?id=241

    Thanks

  17. rsdcastro commented at 9:10 pm on June 30, 2014: none
    Also, if you have a LevelDb database that triggers the error and you’re comfortable sharing, that could be very helpful.
  18. ghost commented at 5:56 pm on July 1, 2014: none

    @rdcastro, sorry, I have only one Raspberry Pi and since I wiped the SD card and did everything anew with a newer OS and the latest bitcoin source (and I’m still downloading the blockchain :-) so I can’t wipe this system) FWIW, here are my exact specs from the time. #4364 Anyone with a Raspberry Pi could get the Raspbian from January 2014, install bitcoin mentioned above and he should be able to see all these problems with LevelDB that I’ve reported.

    Update to this bug: after following @analogic ’s recipe above (to commment the lines) and I haven’t had any problems for 10+ days. My current config:

    0hostname = raspberrypi
1uname -m = armv6l
2uname -r = 3.12.22+
3uname -s = Linux
4uname -v = [#691](/bitcoin-bitcoin/691/) PREEMPT Wed Jun 18 18:29:58 BST 2014
5Bitcoin Core Daemon version v0.9.2.0-b64b1c6-dirty-beta

    Although I’ve upgraded the OS and bitcoind, IIRC I had problems with LevelDB with that setup as well until I marked out those lines in the code.

  19. laanwj closed this on May 18, 2015
  20. DrahtBot locked this on Sep 8, 2021


analogic laanwj ghost rsdcastro

Labels
UTXO Db and Indexes

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-01-02 12:12 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me