Unable to load the databases when mounted over a samba file share #7981

When the data directory is mounted on a samba file share using cifs, I am unable to load the databases. It always starts with an error message asking to reindex the databases.

This is the error from the debug.log

2016-04-27 20:36:22 init message: Loading block index... 2016-04-27 20:36:22 Opening LevelDB in /home/achow101/.bitcoin/blocks/index 2016-04-27 20:36:24 IO error: /home/achow101/.bitcoin/blocks/index: Invalid argument 2016-04-27 20:36:28 Aborted block database rebuild. Exiting.

I think it has to do with how cifs does file sharing, but I'm not sure.

Other people have also seen this problem: https://bitcointalk.org/index.php?topic=1454844.msg14707662#msg14707662

This looks like an open bug in leveldb: google/leveldb#281

Most network based filesystems are currently not supported by leveldb/bitcoind. Related to #3641 (comment).

Please consider the block database is huge and growing, so the need of storing it on a NAS could become more popular in future.

I partial agree with @gtozzi. I think the UTXO database should not be on a NAS (or lets say high latency & slow speed persistence store) but there could be reasons to keep the block files on a NAS (especially age >144 blocks).

You could try my dummy_db branch https://github.com/laanwj/bitcoin/tree/2016_04_dummy_db. It builds the entire database in memory and only saves it at shutdown, and re-reads it at startup. Warning: needs quite some memory :) (I also have a LMDB branch but I wouldn't recommend it to anyone at this moment)

I agree with @jonasschnelli though - what I'd suggest doing is to store the databases ($DATADIR/blocks/index) and ($DATADIR/chainstate) locally on a fast disk, and the block files on a (slower) remote device. There is no (direct) option to do this at the moment, but it can be accomplished by symbolic linking.

I use this script to move old *.dat files into a cifs mount point:

#!/usr/bin/env python

import os
import shutil
import time

# Tuples of source directory where *.dat files are written to by bitcoind and
# destination directory where old *.dat files will be moved to.
DIRS = [
    (
        os.path.expanduser('~/.testnet3/testnet3/blocks'),  # Source
        '/mnt/testnet3-blocks-dat',  # Destination
    ),
    (
        os.path.expanduser('~/.bitcoin/blocks'),
        '/mnt/bitcoin-blocks-dat',
    ),
]

# Archive *.dat files that are modified over `MAX_AGE` ago.
MAX_AGE = 3600

# Change to `False` to actually archive the *.dat files.
DRY_RUN = True


if __name__ == '__main__':
    for (src_dir, dst_dir) in DIRS:
        if not os.path.isdir(src_dir):
            raise Exception('%s does not exist!' % src_dir)

        if not os.path.isdir(dst_dir):
            raise Exception('%s does not exist!' % dst_dir)

        os.chdir(src_dir)

        files = os.listdir(src_dir)
        for filename in files:
            if not filename.endswith('.dat'):
                continue

            src_path = os.path.join(src_dir, filename)
            last_modified = int(os.path.getmtime(src_path))

            dst_path = os.path.join(dst_dir, filename)
            if os.path.exists(dst_path):
                continue

            now = int(time.time())
            if now - last_modified > MAX_AGE:
                print('mv {} {}'.format(src_path, dst_path))
                if not DRY_RUN:
                    shutil.move(src_path, dst_path)

                print('ln -s {} {}'.format(dst_path, filename))
                if not DRY_RUN:
                    os.symlink(dst_path, filename)