New block database files platform-dependent? #2293

I compiled the content of the git tag 0.8-rc1 on my raspberry pi (ARM) and downloaded the blockchain (well most of it) into the new format. When I try to copy the contents of blocks and chainstate an x86 machine and start the precompiled 0.8-rc1 bitcoind it crashes. Same happens with pi-created block chain files if you try to start the 0.8-rc1 windows. (Copying the datafiles between x86 and ARM was no problem with 0.7)

Bitcoin version v0.8.0rc1-beta (2013-02-06 16:06:43 -0500) Using OpenSSL version OpenSSL 0.9.8k 25 Mar 2009 Startup time: 2013-02-10 14:07:00 Default data directory /root/.bitcoin Used data directory /mnt/store_0/bitcoin/bitcoincodes init message: Verifying wallet integrity... dbenv.open LogDir=/mnt/store_0/bitcoin/bitcoincodes/database ErrorFile=/mnt/store_0/bitcoin/bitcoincodes/db.log Bound to [::]:8333 Bound to 0.0.0.0:8333 init message: Loading block index... Opening LevelDB in /mnt/store_0/bitcoin/bitcoincodes/blocks/index Opened LevelDB successfully Opening LevelDB in /mnt/store_0/bitcoin/bitcoincodes/chainstate Opened LevelDB successfully LoadBlockIndex(): last block file = 37 LoadBlockIndex(): last block file: CBlockFileInfo(blocks=133, size=23791072, heights=216794..216926, time=2013-01-16..2013-01-17) LoadBlockIndex(): transaction index disabled LoadBlockIndex(): hashBestChain=00000000000004e3fa816b4c834a13e581e446c2b6881c83b28f9864d615a663 height=216926 date=2013-01-17 19:45:03 init message: Verifying block database integrity... Verifying last 288 blocks at level 3 ERROR: DisconnectBlock() : outputs still spent? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : undo data adding output to missing transaction ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : undo data adding output to missing transaction ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : undo data adding output to missing transaction ERROR: DisconnectBlock() : undo data adding output to missing transaction ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : undo data adding output to missing transaction ERROR: DisconnectBlock() : undo data adding output to missing transaction ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : outputs still spent? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : undo data adding output to missing transaction ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : undo data adding output to missing transaction ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted ERROR: DisconnectBlock() : added transaction mismatch? database corrupted Unable to open file /mnt/store_0/bitcoin/bitcoincodes/blocks/rev00037.dat ERROR: CBlockUndo::ReadFromDisk() : OpenBlockFile failed ERROR: VerifyDB() : *** found bad undo data at 216819, hash=0000000000000333189f2bc4a671ed10abedb5c9fb3f6d69a2498c8c2317eeaa

Error: Corrupted block database detected. Please restart the client with -reindex. Flush(false) DBFlush(false) ended 0ms StopNode() Flushed 0 addresses to peers.dat 467ms Committing 27465 changed transactions to coin database... Flush(true) DBFlush(true) ended 0ms Bitcoin exited

Did you copy the files while the application was running?

No, I shut it down and it was running with --detach-db=1 (if that matters) Also: I tried this twice.

Ok, interesting. Can you try:

• If you're able to compile on/for the RPi, can you try running the unit tests on it? (cd src; make -f makefile.unix test_bitcoin && ./test_bitcoin).
• Are you able to sync (even if it's very slow) on the RPi directly?
• Can you copy the files from an RPi to a desktop system, and continue the sync?

1. Will do
2. It syncs and runs fine on the rpi and on the x86 machine. You just can't copy the db from one to the other (both ways will give errors)
3. See 2

The test build said "Running 89 test cases, no errors"

AFAIK the database files on linux and MS windows are also incompatible with each other. Are there any plans to make it so that the files can be used for the client irrespective of platform/architecture?

This is low priority, given that the database may be regenerated from publicly available data at any time.

I just checked with master on my cubox-i (4 core, armv7l-linux-gnueabihf, debian jessie). Tried two scenarios:

1. On ARM box

• bootstrap the first ? blocks using blk00000.dat file from other install
• shut down bitcoind
• copy resulting bitcoin directory to AMD64 box
• run bitcoind on AMD64 box

2. On AMD64 box

• bootstrap the first 142647 blocks using bootstrap.dat file
• shut down bitcoind
• copy resulting bitcoin directory to ARM box
• run bitcoind on ARM box

Curiously, (1) seemed to work, whereas (2) manages to get past the leveldb validation but then starts to behave very erratic. It accepts blocks but never does anything with them, ie

2014-03-08 07:51:31 ProcessBlock: ACCEPTED
2014-03-08 07:51:31 ProcessBlock: ACCEPTED
2014-03-08 07:51:31 ProcessBlock: ACCEPTED
2014-03-08 07:51:31 ProcessBlock: ACCEPTED
2014-03-08 07:51:31 ProcessBlock: ACCEPTED
2014-03-08 07:51:31 ProcessBlock: ACCEPTED
2014-03-08 07:51:31 ProcessBlock: ACCEPTED
....

Starting with -checkblocks=0 (which should check the entire database) gives

2014-03-08 07:47:38 Verifying last 142647 blocks at level 3
2014-03-08 07:51:22 No coin database inconsistencies in last 4083 blocks (216726 transactions)
2014-03-08 07:51:23  block index          233301ms

After a -reindex it works fine.

Edit: re-trying with the copied block index and -checkblocks=0 -checklevel=3 gives the same kind of errors OP has:

2014-03-08 08:21:54 Verifying last 142647 blocks at level 3
2014-03-08 08:22:57 ERROR: DisconnectBlock() : undo data adding output to missing transaction
2014-03-08 08:22:57 ERROR: DisconnectBlock() : undo data adding output to missing transaction
2014-03-08 08:22:57 ERROR: DisconnectBlock() : undo data adding output to missing transaction
2014-03-08 08:22:57 ERROR: DisconnectBlock() : undo data adding output to missing transaction
2014-03-08 08:22:57 ERROR: DisconnectBlock() : undo data adding output to missing transaction
....

The daemon hangs after this.

OK this is curious. It it non-deterministic. Sometimes the verify passes, as pindexState gets 'stuck'

2014-03-08 08:45:20 Verifying pindexState=138564 pIndex=1
2014-03-08 08:45:20 No coin database inconsistencies in last 4083 blocks (216726 transactions)
2014-03-08 08:45:21  block index          244825ms

(this doesn't imply that the resulting index actually works, it doesn't)

I too tried to initialize a block index and chainstate database on my x86 system and then copy the files over to a Raspberry Pi. Since both systems are Linux, 32-bit, little-endian, I figured there would be no differences in the on-disk format. However, this is apparently not so. When Bitcoind starts verifying blocks, it immediately starts complaining about missing transactions. However, using the exact same files on the x86 machine (i.e., everything inside blocks/ and chainstate/ identical to what it was when starting on the RPi), Bitcoind starts up and verifies the blocks just fine.

Very disappointing that the on-disk format of these LevelDB databases differs between two architectures that are both 32-bit, little-endian.

I think this is unintentional, as the LevelDB on-disk format is very well described (up to the byte level). My guess is a LevelDB bug that only occurs on ARM.

We should probably report this upstream.

I guess we would need a minimal test case that produces different data files on ARM than on x86.

It may just be a bug in Bitcoin's LevelDB glue code or serialization code too...

This is really strange. I've indexed the first four block files both on ARM and AMD64. To compare the resulting leveldb files I've built a py-leveldb against bitcoin's leveldb and started poking at the database files, comparing them in various ways.

# Compare two bitcoind's leveldbs
import leveldb

dbdira = 'armtest'
dbdirb = 'amd64test'

databases = ['/chainstate', '/blocks/index']

def compare_databases(cs1, cs2):
    ai = cs1.RangeIter()
    bi = cs2.RangeIter()

    n = 0
    err = 0
    aend = bend = False
    while True:
        try:
            a = ai.next()
        except StopIteration:
            aend = True
        try:
            b = bi.next()
        except StopIteration:
            bend = True
        if aend and bend:
            break # both at end of iteration
        if aend or bend or a != b:
            err += 1
        n += 1

    print "Compared %i records" % n
    print "Found %i differences" % err

def compare_databases2(cs1, cs2):
    '''
    Check that all keys of cs1 are in cs2
    and that the values match.
    '''
    ai = cs1.RangeIter()

    n = 0
    err = 0
    for (key,value) in ai:
        try:
            if cs2.Get(key) != value:
                err += 1
        except KeyError:
            err += 1
        n += 1

    print "Compared %i records" % n
    print "Found %i differences" % err

for db in databases:
    print "* Comparing %s" % db
    cs1 = leveldb.LevelDB(dbdira + db)
    cs2 = leveldb.LevelDB(dbdirb + db)
    print "** Check 1"
    compare_databases(cs1, cs2)
    print "** Check 2"
    compare_databases2(cs1, cs2)
    print "** Check 3"
    compare_databases2(cs2, cs1)
    print

Output:

* Comparing /chainstate
** Check 1
Compared 390331 records
Found 0 differences
** Check 2
Compared 390331 records
Found 0 differences
** Check 3
Compared 390331 records
Found 0 differences

* Comparing /blocks/index
** Check 1
Compared 142654 records
Found 0 differences
** Check 2
Compared 142654 records
Found 0 differences
** Check 3
Compared 142654 records
Found 0 differences

Seemingly, there are no differences at all in the data. I've also checked the blk* and rev*.dat files. They match to the byte. However, bitcoin on AMD64 refuses to work with the ARM files. I'm baffled.

Of course, bitcoin may have some other access pattern that brings to light a bug/difference in a lower level, but this is turning out to be elusive.

Also: Re-silvering the ARM-created databases by creating a new database and simply copying all keys/values, makes a database that can be used on AMD64.

@laanwj Feel like trying again with #4161 and #4177 merged?

OK, going to try

I tried using the leveldb files from ARM on AMD64 and vice-versa. Same problem as before - alas, the paranoid checks do not catch this.

Thanks for testing!

I wonder: do we get any kind of warning/error in chainstate/LOG ?

I'll check next time.

I'm working on recording a trace of bitcoind's access pattern, then replaying that against the two database instance and seeing where returned values start to diverge.

I found the bug! Well at least I've narrowed it down a lot. The problem is in the filter_policy, which is set to leveldb::NewBloomFilterPolicy(10). Seemingly this filter policy data is not portable. Commenting out that line makes the same block index files work on ARM and AMD64.

This explains why the database works fine after resilvering in my above post (and in general in py_leveldb, which doesn't use a filter policy...).

The Hash function used for the bloom filter was returning different values based on character signedness in some edge cases. See https://github.com/bitcoin/leveldb/pull/5 for fix. Upstream issue: https://code.google.com/p/leveldb/issues/detail?id=237

It doesn't seem that the upstream issue is being picked up. Platform compatibility of the databases is likely very low-priority for them.

We could work around this completely on the bitcoind side by providing our own FilterPolicy() implementation and passing it in with the options. That's just a small class that hashes keys and tests against them: https://github.com/bitcoin/bitcoin/blob/master/src/leveldb/util/bloom.cc#L17 .

See #5093