Migrate chainstate to SSD, or load it to memory #14904

1. 
   andronoob commented at 1:54 am on December 10, 2018: none

   The UTXO database is stored in chainstate directory, which is intensively accessed. A lot of PCs don’t have a large volume SSD, instead, they have a combination of a moderate volume, turbo SSD and a large volume, slow HDD. Syncing a full node on slow HDD seems to be very painfull… A viable solution is migrating chainstate to SSD, while keeping blocks stored on HDD. For PCs which are equipped with more than 8GB of RAM but without SSDs, there’s another solution: loading all files placed in chainstate into memory before launching the full node.
2. fanquake added the label Block storage on Dec 10, 2018
3. 
   achow101 commented at 2:09 am on December 10, 2018: member

   You can already do both of these things.

   A viable solution is migrating chainstate to SSD, while keeping blocks stored on HDD.

   You can start Bitcoin Core with -blocksdir to specify the location to store the blocks. The indexes and chainstate databases will remain in the datadir. So you can specify your datadir to be your SSD, and the blocksdir to be your hard drive.

   For PCs which are equipped with more than 8GB of RAM but without SSDs, there’s another solution: loading all files placed in chainstate into memory before launching the full node.

   Use the -dbcache option to increase the size of the database cache. This will cache the chainstate database in memory. If it is larger than 8000 (8 GB), then the entire chainstate database will be cached in memory.

4. 
   andronoob commented at 2:35 am on December 10, 2018: none

   @achow101 Thank you for instruction!

   You can start Bitcoin Core with -blocksdir to specify the location to store the blocks.

   Is it possible to make this config available through GUI?

   If it is larger than 8000 (8 GB), then the entire chainstate database will be cached in memory.

   For scenarios like #12058, tricks like using cat chainstate/* > /dev/null to manually load all files in chainstate seem to work. Additionally, what if the computer has exactly 8GB of total RAM? Will storing compressed UTXO database in memory help? Though dbcache is available through GUI, I wish there could be some more hints for the user.

5. 
   promag commented at 1:14 am on December 11, 2018: member

   Is it possible to make this config available through GUI? @andronoob this option is new, available from 0.17.0, which was added in #12653. In #14374 the information panel was also extended. I guess you could submit a feature request with the use case.

6. fanquake closed this on Feb 3, 2020

   ---

7. 
   andronoob commented at 3:02 am on August 13, 2020: none

   I don’t think this issue should be closed yet.

   Firstly I wrote:

   A viable solution is migrating chainstate to SSD, while keeping blocks stored on HDD.

   I recently posted similar idea in another issue thread, so indeedly it may be duplicate to discuss it here.

   For PCs which are equipped with more than 8GB of RAM but without SSDs, there’s another solution: loading all files placed in chainstate into memory before launching the full node.

   I think this can be done just in a way similar to the simple trick like cat chainstate/* > /dev/null, in fact this cat method earned many upvotes on serverfault.

   Why I decide to disturb you despite I already know this trick? Because I still remember the old days waiting for the node to sync when I was still a outright noob (then I’m proud to be an experienced noob now). At that time I also worried about whether my precious non-SMR (CMR) HDD would break after putting such a such a crazy intensive I/O burden on it - as we all know, HDD vendors are shameless enough to silently push unlabeled DM-SMR drives to the consumer market.

8. DrahtBot locked this on Feb 15, 2022

Contributors
andronoob achow101 promag

Labels
Block storage