Feature Request: The client should have a separate option for the storage location of the database data

fresheneesz commented at 6:44 pm on July 3, 2017: none

After wondering why my bitcoin client was taking literally weeks to download the blockchain, I found out that it was going so ultra slow because I was storing the data directory on an external hard drive. After using a symlink to put the database data on my SSD and keep the blockchain stored on my external HD (via the symlink), my blockchain validation sped up by at least 6 fold. Details here: #10647 .

Anyways, because this can be so important for a non-frustrating experience for users who don’t have enough SSD space, I’m requesting that bitcoin core store the database data on the same drive as the program executables, and give the user the option of only where to put the blockchain data (what is currently the blocks directory within the data directory). Alternatively, we could give the user separate options for where to store the blockchain and where to store everything else, with some info about why the storage location is important.

If I’d wanted to buy something with bitcoins in the last month, I simply wouldn’t have been able to because my client was taking forfuckingever to sync up. Its really important to make these kinds of performance improvements idiot-proof so users with less technical savvy can use bitcoin core without rage quitting.

bytting commented at 9:52 pm on July 10, 2017: contributor

store the database data on the same drive as the program executables

This shouldn’t be necessary. I have everything on a secondary drive and have no issues with speed. It’s probably the USB that causes the problem

sipa commented at 9:57 pm on July 10, 2017: member

The client should store the database data on the same drive as the program

That’s not generally possible. On UNIX systems, the filesystem with binaries may not be writable by the user.

I do however think it’s useful to allow the chainstate to be stored in another directory from the blockchain. The former mostly needs high IOPS, the latter only needs serial read/writes.

fresheneesz renamed this:
~~Feature Request: The client should store the database data on the same drive as the program~~
Feature Request: The client should have a separate option for the storage location of the database data
on Jul 10, 2017

fresheneesz commented at 10:42 pm on July 10, 2017: none

Alright, I changed the name of the request to be a separate option for storage location. @corebob Sipa also mentioned that USB is probably the issue. Do you have any information you could point me to that substantiates this? I don’t see any logical reason why USB would cause a bottleneck here nor does searching the googles turn up anything obvious.

bytting commented at 2:14 pm on July 11, 2017: contributor

@fresheneesz Not at the moment, it was just an assumption I made based on your description.

jimhashhq commented at 11:59 pm on July 24, 2017: none

Hi, I just opened: pull request #10922 which adds a file-partition.md doc file which describes what I believe to be a simple interim solution to putting the large blockchain files on an external drive (like my inexpensive USB 3.0 in the example). Longer term, it might be easier if there was a configuration setting or flag for storing all of the leveldb stuff on an internal drive and/or the blockchain blocks folder on an external drive. As I mentioned in the pull request, this may not be the best way for these instructions to reach their intended audience so please feel free to redirect me. Thanks!

opacey commented at 2:46 pm on January 17, 2018: none

+1

This is an excellent suggestion. The blockchain itself, by nature is massive and doesn’t need to be on local or high speed and throughput storage medium.

The data directory contains reasonable sized files and folders which could go anywhere, and an elephant of a directory called blocks (172GB today), we really need flexibility to optionally relocate this one.

I have in the past tried multiple way to store either the whole datadir or just blocks on…

Network attached drives via SMB or AFP
Network attached drives via link
Locally attached USB 3.0 drive

I haven’t tried iSCSI NAS which I gather works but my two NAS attempts failed with corrupted data and the USB 3.0 drive had the same issues @fresheneesz encountered (thanks also to @sipa for the analysis!). Would be delighted if this could be worked into the pipeline.

jimhashhq commented at 0:18 am on January 19, 2018: none

@opacey It’s not clear from your comment if you read file-partition.md which describes setting up the necessary symlinks, allowing you to run on low-end commodity hardware without the large internal disk space requirements.

fresheneesz commented at 2:23 am on January 19, 2018: none

@jimhashhq These things are far too complicated for the average user (even granting that the average bitcoin user is more tech savvy than the general population). What we need is a gui wizard that asks “what’s your main harddrive? (SSD works best) [10GB required]” and “Which drive do you want to store the bulk of the blockchain? (150GB+ required)”. These should be simple dropdowns for the user. And this would be a perfect place to ask the user “Do you want to prune the blockchain? To what number of GBs?”

Its great that its possible to split this up, but its a huge, error prone PITA doing this through command line right now. And many people might not even be aware its possible to do this even if they are already savvy enough to find these guides and correctly follow them.

opacey commented at 9:29 am on January 22, 2018: none

@jimhashhq Indeed I neglected to address your formalised sym link solution. I like it a lot as an interim solution, as you suggest, and my vote is to have in included. I then also second @fresheneesz urging for a GUI level version. Thanks again.

Sjors commented at 2:13 pm on March 16, 2018: member

See #12653

Sjors commented at 11:25 am on March 29, 2018: member

This seems fixed by #12653.

MarcoFalke closed this on Mar 29, 2018

DrahtBot locked this on Sep 8, 2021

Feature Request: The client should have a separate option for the storage location of the database data #10736