[enhancement] Separate Node and Wallet functions #7525

We have a usability problem.

Problem When new layman users approach bitcoin, the first thing ANYONE ever does is seek out the original wallet software. This means that every single new user that comes along is inadvertently running a FULL NODE. While having more nodes can be useful, you wind up with users that just "leave it alone" and not keep up to date.

Reaction When a user is nervous and learning about a new system it can scare, intimidate, and ward them off to be confronted with a full node download (currently 60GB), especially when all they expected was a "wallet"

Solution I suggest separating the products, because they ARE distinct.

• bitcoin-node
• bitcoin-wallet (thin by default, ALLOW to connect to your own defined node)

While 3rd party wallets / thin clients DO exist, I feel we need a rock solid place for users to start. Yes, people should do their research, but we need to make it as easy as possible to approach and use.

We want to make this as stupidly easy as possible for any member of the public to approach.

Thoughts?

It isn't secure to have a wallet without a node. Might as well be using PayPal.

That being said, I do think it makes sense to:

1. Keep the node/wallet code separate.
2. Improve the node's performance.
3. Implement SPV support for the node so it works before/while the sync goes in the background.

The first two have been improving with every release, but it takes time. Additional development help is welcome.

@luke-jr

I would recommend having a thin-client wallet by default, but allowing them to configure it to use their own full node if they choose.

@cluelessperson That's a good recommendation, but Bitcoin Core isn't that thin client. There are other projects that implement wallets.

Splitting the Core wallet and node to separate executables, communicating over RPC/P2P, is a very old idea. No one has ever implemented it yet, though, it's pretty hard to do in practice. Or at least a lot of work. I don't know of anyone that has concrete plans to do so.

If you need anything soon I'd also suggest looking at other thin client software for that.

Is it really a usability problem? I might be okay with a special option to configure full node or not for people with lower end hardware, but by default a full node is probably better.

@Jonnyb42

The problem is that we need to make this so that any person from your average public sampling can use this. We need bitcoin to easy to use. This will help public adoption.

I can imagine an online service where you import your public keys, the service keeps track of your transactions and balance, and every time you need to send value you just sign the transaction in your desktop client and upload.

Agree with your problem statement as that is what I went through. However I think that as soon as the average public finds out they need 60Gb of HDD, the next logical thing to do is Google for other options.

@cloudnthings Perhaps it would be interesting to know where you read what about Bitcoin, and how you ended up deciding to use Bitcoin Core as a wallet. For example https://bitcoin.org/en/choose-your-wallet warns you about its resource requirements.

@sipa When I started, I went to what I considered the "home of bitcoin" at bitcoin.org. For me, I was happy to run the full node and use the Bitcoin Core wallet but have since tried others as well over time. (via the choose your wallet link).

Just for reference, we're at about $25/TB storage costs now...

60 GB is (as @Jonnyb42 points out) practically nothing these days, certainly not a barrier to adoption. There are many far more important things needed before Bitcoin is ready for mainstream use.

Third-party services are, as already mentioned, not much better than PayPal (even if you hold the private keys).

Since 0.12 it is possible to start QT/daemon with both wallet and pruned mode enabled. In that case the minimal HDD footprint is only 1.6 GB for /chainstate and 550 MB for /blocks as of February 2016. The cli argument -prune=550 or setting in bitcoin.conf is available but the UI does not offer the less technical user a convenient way to start bitcoin-qt in that mode yet.

An easy activation of this feature on the first start combined with a new service flag for these non-archive, pruned nodes that always serve at least the last 7-14 days of blocks would still be very helpful to the network and most other nodes / SPV clients that catch up at least once every few days.

@FelixWeis There's no reason to encourage people to make their node less useful in order to save a mere $2 of storage (65 GB).

@luke-jr @FelixWeis @Jonnyb42

Every single GB is longer that a person has to WAIT to see the product work. Every single millisecond longer that it takes costs us adoption, money, time, effort. Seriously. Reality is that people work on LIMITED resources. Yes, downloading 65GB for you or me is easy, but is almost IMPOSSIBLE for someone in Africa, Rural America, etc.

Whatever you feel about the current cost of storage is irrelevant. Again, this is a usability problem that makes it harder for layman to approach. @sipa @cloudnthings You two are suggesting that customers/investors/layman/public take FURTHER steps, do MORE work to find other options. Every single step further that a person has to take, is more unlikelihood that they will adopt it. The more difficult it is to use, the more research/time/effort it takes to use safely, the worse adoption will be.

Why on earth would you make it harder for people to use your product?

• Most people do NOT need a full node.
• Random people running nodes without the deep knowledge of when to upgrade them can be damaging.
• Layman approaching bitcoin, when they find the "source", should NOT be confronted with "the only original option is to run a full node."

I don't think we need to argue here at all. Separating the wallet and node executable would be good for other reasons as well, for example if one needs to run multiple isolated wallets against one local node.

However the sad truth is that this (good) idea has existed for a long time, it still isn't executed, we're not expecting anyone to do it in the near future. No amount of energetic arguing will bring it closer.

If you're a developer you're very welcome to pick it up.

Final note: There are projects that run an external wallet against bitcoind RPC. For example https://github.com/JoinMarket-Org/joinmarket does this. It is a bit inefficient at the moment, relying on watch-only, so there is scope for improvement, but it is a step on the way.

That sounds like the sort of masochistic and soul-destroying project I might enjoy. Anybody else interested (or who has tried and failed before), let me know if you want to compare notes.

Looks like @jonasschnelli has a significant interest in wallet improvements. Are you still working on this stuff Jonas? Don't want to step on your toes or duplicate work...

Yes. I'm still working on this. Though, there are some challenges (more conceptual) and hight risks of creating work that will end up in the bin.

IMO:

1. Core's wallet should run on it's own process and communication only over p2p (at the beginning, maybe also over RPC).
2. The wallet should use SPV against the fullnode on localhost and/or SPV against a trusted fullnode on a remote system
3. The GUI should be split into "node GUI" and "wallet GUI" (maybe to separated applications, or at least two separated code bases [one could be a submodule/subtree of the other]).

Point 1 can be realized by adding a runtime- (or even compile) option that allows running the bitcoind in SPV mode (only headers will be loaded, no mempool), most node code is unused in this mode. But: the question the might be raised, if having a wallet-only mode (SPV) in bitcoin-core is something we really want to have (or if it should be writing in a new independent repository).

Point 2 is challenging. I guess using p2p and RPC at the same time will be hard to configure for a "non-experienced" users if the node is running on a remote server. Also privacy is a problem. IMO the best way would be, to add optional encryption to the p2p layer by ECDH and preshared pub keys (auth) and allow to server some special commands to allowed nodes (your local SPV wallet). Fee estimation and things like that would be required to run a detached, process separated wallet. Also, if we fully would like to decouple the GUI form the node, we could think about commands to control your node over p2p (although, this includes some serious conceptual security risks). Configuring a SSH tunnel for p2p and RPC seems after a really bad user experience (setup hassle) and would probably not allow to use smartphone SPV wallets against your trusted fill node. If we would allow p2p auth & encryption, this would result in a very easy setup (pre-share a ec pubkey between fullnode and spv-node, done).

One of the main problem is, touching the bitcoin-cores wallet can introduce new bugs in the wallet code which could result in loosing funds (worst case). Also main bottleneck for wallet changes and reviews.

@cluelessperson Totally with you on this suggestion. What you are saying is to a adapt Bitcoin-Qt so that it becomes essentially a thin-client wallet and runs against a (not necessarily local) bitcoind which it trusts. I think full nodes ought to have signatures so that thin-clients can recognize the nodes they trust with something other than IP addresses to enable this. It may be that there are already other wallets which can be set to trust full-nodes - can anyone here comment on which wallets already do this? (And do they use anything other than IP address to ensure they are talking to a trusted full-node?)

@rebroad: not sure if connecting a SPV wallet against your trusted full node is something we should encourage for now. A ISP, WiFi-Provider or other network authority could analyze the p2p traffic and create a profile, etc. IMO SPV together with bloomfilters should only be used in tunneled/encrypted connections.

If I do some work on this, it seems to me the logical place to start is init.cpp. My plan would be to separate out wallet code by using something along the lines of:

MultiWalletServiceModule walletServiceModules;
#ifdef ENABLE_WALLET
LegacyWalletServiceModule legacyWalletModule;
walletServiceModules.install(legacyWalletModule);
#endif

and hopefully getting rid of most other ENABLE_WALLET checks by, for example, changing this section

    // if using block pruning, then disable txindex
    if (GetArg("-prune", 0)) {
        if (GetBoolArg("-txindex", DEFAULT_TXINDEX))
            return InitError(_("Prune mode is incompatible with -txindex."));
#ifdef ENABLE_WALLET
        if (GetBoolArg("-rescan", false)) {
            return InitError(_("Rescans are not possible in pruned mode....."));
        }
#endif
    }

to

    // if using block pruning, then disable txindex and ...
    if (GetArg("-prune", 0)) {
        if (GetBoolArg("-txindex", DEFAULT_TXINDEX))
            return InitError(_("Rescans are not possible in pruned mode....."));
        if (!walletServiceModules.BlockPruningOptionChecks())
            return false;
    }

Looking for concept ACKs or other comments. Is this just shifting code around without really achieving much towards actual separation? If that is the conclusion, perhaps I can redeem the idea by starting to identify node-specific code and shifting it into a similar nodeServiceModules object? (Suggestions on better naming welcome if we go ahead ...).

Or perhaps stick with the ENABLE_WALLET checks and start adding in new ENABLE_NODE checks where appropriate?

@kirklax: Check: #5990 and #6008

The wallet modularization is (partial) done in those PRs. Feel free to pick it up, rebase it and complete it.

Cheers Jonas, will take a look. Hope you had a great holiday!

I had seen some of this the other day while browsing your github repo but hadn't noticed the PRs and hadn't realized how much you had achieved - very impressive...

@jonasschnelli it would of course be an encrypted connection only for this reason. ssh or zerotier perhaps.

What I would like is to be able to have a single block chain but multiple user wallets on the same system that communicate with bitcoind

That seems more logical to me. Why have 3 copies of the blockchain on a system with three different users each of which has a different bitcoin wallet? In fact you can't because they can't run at the same time unless from different VMs.

Also, the reason I don't run a bitcoin client on my laptop is because 60 GB is too much, maybe for some people buying a bigger drive is not a problem but for some of us, we don't have the same financial resources available so we don't replace hardware until it fails.

It would be great if I could use the blockchain on my desktop from my laptop - with the Qt client - specifying the IP address.

Thin clients may be beyond the scope of bitcoin-core now but it doesn't have to be.

Especially now that many ISPs are implementing bandwidth caps, it makes sense to have bitcoind running on one computer on a LAN that all clients on the LAN can communicate with. Sure you can configure all the other clients to only use the one on the LAN as a source for blocks, and some people do that, but is that any more secure than just having thin clients connect to the one on the LAN?

There seem to have been updates that might relate to this issue. Will comment again if significant observations are made.

https://github.com/bitcoin/bitcoin/commit/2607d960a02e21098092e95ba6bc2a64c93e6127

Fixed in #10973