Idea suggestion: allow using external DB (for instance RethinkDB) engines for block storage #20647

Considering the huge size of block database, maybe it is practical to allow users to use external DB engines for this.

Imho, RethinkDB, with it's GUI, is very easy to setup, including it's mirroring and clustering.

Because of size of blocks DB and of amount of time it is required to download and/or reindex existing backup, mirroring features of real DB engines may be more practical to use.

use case:

1. user downloads Bitcoin Core
2. user downloads and sets up RethinkDB (possibly with mirroring and/or clustering)
3. user selects usage of RethinkDB in bitcoin-qt options window
4. user restarts bitcoin-qt
5. bitcoin-qt checks DB structure and tables and indexes and creates/changes them as needed
6. bitcoin-qt does it's work on blocks DB using RethinkDB

• such db could be placed on external computer
• connection to db could be encrypted and secured
• less chances to destroy hdd/ssd on local PC and data on it (because of intensive use of drive by bitcoin software)
• less chances to lose blocks DB
• less load on network from those who lost their blocks DB and now re-downloading it
• easier for user to shutdown bitcoin-qt - less time to wait while bitcoin-qt syncs and closes DB
• easier to restart in case of power failures

It is already possible to backup the blocksdir to another drive (or even specify a blocksdir on a different drive). And surely the other drive could be a RAID to add redundancy.

I don't think this adds any new features and only complicates the code.

not every user can manually setup RAID. also RAID is much harder to repair in case of problems with it. especially if it is a notebook or smartphone. for average user it is much easier to buy a Net-Drive or setup dedicated computer with raid, but it is impractical to have server exclusively for one single application.

is already possible to backup the blocksdir to another drive

this is only offline method. and external DB engine, like said RethinkDB, does mirroring and clustering on the fly, online. so bitcoin-qt don't need to be closed for time of those operations on DB

by the way, RAID - does not provides security from other applications running under same user on same computer. and RethinkDB have user management, password protections and access rights.

@AnimusPEXUS

Aside from this issue, is there any currently open issue in this repo that documents a non-theoretical and commonly occurring user pain point that would be resolved by introducing an external database? (The "user pain point" question.)

If there is: what are the best technical arguments for choosing a distributed document-oriented database that stores schemaless JSON documents (RethinkDB) to resolve the user pain point? :) (The "developer solution" question.)

This sounds like a solution (RethinkDB) in search of a problem TBH :)

@practicalswift In fact, it is third or fourth day in the row I'm trying to reuse and update my blocks backup which I left 2 years ago. and I can't leave PC work through night and have to turn it off. If You, guys, want as many as possible average users to download and host complete blocks database, then all this boredom with this > 250GiB size DB inconvenience have to be somewhat more flexible. be realists: bitcoin core is not the database maintenance software and this leads to bunch of problems for users.

JSON documents

it's stores data in binary format. JSON is only for query protocol

Aside from this issue

Inconvenience for users to treasure and shake upon >250GiB db of data isn't enough for you?

for my 10 years of attempts to use bitcoin core, I'm got 3 hdds killed with all my data on them. And you know how people treat unusable things today. and simply using wallet software which utilizes online block bases isn't in bitcoin's decentralization motto.

block DB must be much more flexible for users to maintain and backup, includingly restore from failures, and bitcoin core can't provide this in it's current state.

I'm got 3 hdds killed

While software can help to protect against data loss, it can't magically turn broken hardware into working hardware. I suggest you purchase high quality storage.

@MarcoFalke

You don't know how bitcoin can kill hdd? probably it can degrade ssds fast too. I was always used black WDs on my PCs. you suggest some other hdds?, or I didn't understood your advice.

You know what.. ok.. maybe I'll use bitcoin core some other time.. I just don't have time to play in this "dawnload 300GiB db game" god knows which time again..