Unable to load the databases when mounted over a samba file share #7981

1. 
   achow101 commented at 10:23 pm on April 30, 2016: member

   When the data directory is mounted on a samba file share using cifs, I am unable to load the databases. It always starts with an error message asking to reindex the databases.

   This is the error from the debug.log

   2016-04-27 20:36:22 init message: Loading block index... 2016-04-27 20:36:22 Opening LevelDB in /home/achow101/.bitcoin/blocks/index 2016-04-27 20:36:24 IO error: /home/achow101/.bitcoin/blocks/index: Invalid argument 2016-04-27 20:36:28 Aborted block database rebuild. Exiting.

   I think it has to do with how cifs does file sharing, but I’m not sure.

   Other people have also seen this problem: https://bitcointalk.org/index.php?topic=1454844.msg14707662#msg14707662

2. 
   kazcw commented at 4:03 pm on May 1, 2016: contributor

   This looks like an open bug in leveldb: google/leveldb#281
3. 
   jonasschnelli commented at 6:48 am on May 2, 2016: contributor

   Most network based filesystems are currently not supported by leveldb/bitcoind. Related to #3641 (comment).
4. 
   gtozzi commented at 7:28 am on May 2, 2016: none

   Please consider the block database is huge and growing, so the need of storing it on a NAS could become more popular in future.
5. 
   jonasschnelli commented at 7:33 am on May 2, 2016: contributor

   I partial agree with @gtozzi. I think the UTXO database should not be on a NAS (or lets say high latency & slow speed persistence store) but there could be reasons to keep the block files on a NAS (especially age >144 blocks).
6. jonasschnelli added the label Block storage on May 2, 2016
7. 
   laanwj commented at 10:52 am on May 2, 2016: member

   You could try my dummy_db branch https://github.com/laanwj/bitcoin/tree/2016_04_dummy_db. It builds the entire database in memory and only saves it at shutdown, and re-reads it at startup. Warning: needs quite some memory :) (I also have a LMDB branch but I wouldn’t recommend it to anyone at this moment)

   I agree with @jonasschnelli though - what I’d suggest doing is to store the databases ($DATADIR/blocks/index) and ($DATADIR/chainstate) locally on a fast disk, and the block files on a (slower) remote device. There is no (direct) option to do this at the moment, but it can be accomplished by symbolic linking.

8. 
   ayeowch commented at 9:05 am on September 30, 2016: contributor

   I use this script to move old *.dat files into a cifs mount point:

    0#!/usr/bin/env python
    1
    2import os
    3import shutil
    4import time
    5
    6# Tuples of source directory where *.dat files are written to by bitcoind and
    7# destination directory where old *.dat files will be moved to.
    8DIRS = [
    9    (
   10        os.path.expanduser('~/.testnet3/testnet3/blocks'),  # Source
   11        '/mnt/testnet3-blocks-dat',  # Destination
   12    ),
   13    (
   14        os.path.expanduser('~/.bitcoin/blocks'),
   15        '/mnt/bitcoin-blocks-dat',
   16    ),
   17]
   18
   19# Archive *.dat files that are modified over `MAX_AGE` ago.
   20MAX_AGE = 3600
   21
   22# Change to `False` to actually archive the *.dat files.
   23DRY_RUN = True
   24
   25
   26if __name__ == '__main__':
   27    for (src_dir, dst_dir) in DIRS:
   28        if not os.path.isdir(src_dir):
   29            raise Exception('%s does not exist!' % src_dir)
   30
   31        if not os.path.isdir(dst_dir):
   32            raise Exception('%s does not exist!' % dst_dir)
   33
   34        os.chdir(src_dir)
   35
   36        files = os.listdir(src_dir)
   37        for filename in files:
   38            if not filename.endswith('.dat'):
   39                continue
   40
   41            src_path = os.path.join(src_dir, filename)
   42            last_modified = int(os.path.getmtime(src_path))
   43
   44            dst_path = os.path.join(dst_dir, filename)
   45            if os.path.exists(dst_path):
   46                continue
   47
   48            now = int(time.time())
   49            if now - last_modified > MAX_AGE:
   50                print('mv {} {}'.format(src_path, dst_path))
   51                if not DRY_RUN:
   52                    shutil.move(src_path, dst_path)
   53
   54                print('ln -s {} {}'.format(dst_path, filename))
   55                if not DRY_RUN:
   56                    os.symlink(dst_path, filename)
   

9. achow101 closed this on Oct 18, 2016

   ---

10. DrahtBot locked this on Sep 8, 2021

Contributors
achow101 kazcw jonasschnelli gtozzi laanwj ayeowch

Labels
Block storage