Adoption of BIP39/44/49/84, and classification of (extended) pub/priv keys, addresses, mnemonics, etc #17748

Is your feature request related to a problem? Please describe.

Currently, the only stupidly easily accessible backup option is just making a plain copy of the entire wallet.dat database. Comparing to mnemonic words, a piggy wallet.dat file is very much less portable: mnemonic words could be easily written down on paper/carved onto steel plate/assembled by alphanumeric tiles...etc. Furthermore, it could even be memorized into the user's own brain with little effort.

Even WIF privkeys are still much more portable than the piggy wallet.dat file - I think the phenomenon that, up till now, a number of users still have "fetish for one single precious WIF privkey" (and address/key reusing), should really have something to do with this - because WIF is still considered to be the "battlefield-tested, most straightforward/KISS-compliant, most widely compatible, most reliable, and most secure" option of backup by those users, I guess.

To be clear, I don't think such opinion is completely irrational - so far, there is no unanimously accepted/adopted mnemonic words specification. As a widely adopted proposal, BIP39, seemed to be strongly opposed by some devs in the past, so far it was even marked as "Unanimously Discourage for implementation". Therefore, it's reasonable for a user to feel anxious that his/her carefully preserved mnemonic words may suffer from the annoying "bit-rot" issue in the future.

Besides, the classical "screw-up-by-change-address" tragedy might still occassionally repeat in the future. This really seemed to have something to do with this issue - not only with the "fetish for one single precious WIF privkey" mentioned above, but also with the current primitive management of wallet.dat contents. (sorry, but I don't know better words) Bitcoin Core's GUI also lacks persistent backup warning, as far as I know, so that the user might carelessly ignore the fact that the HD seed may be already replaced with a newly generated, not-yet-backuped one.

Currently Bitcoin Core supports importing of various types of things, including privkeys, pubkeys, addresses. Multi-wallet(.dat) is also supported in newer versions. However, a wallet.dat could still gradually turn to a mess, as more and more messy things got imported.

Additionally, hardened address derivation makes the watch-only side of cold storage unable to derive addresses for receiving/change/watching.

Describe the solution you'd like

I really appreciate the BIP32/39/44/49/84 scheme adopted by Trezor. Although I don't think such scheme is ideal either, I think it should be the priority to deliver an usable solution to the users, rather than to wait for a nealy-ideal proposal to appear, let alone its implementation & delivery.

So far, IMHO, adopting BIP32/39/44/49/84 scheme seems to be the best choice for Bitcoin Core.

To address the "messy contents of wallet.dat" problem, I think something like #3314 is missing. The wallet should have flags/timestamps to distinguish among various types of its contents, including, but not limited to:

1. Observe-only/owned: the later may contain watch-only addresses for cold storage. Only the later would be counted as available balance, or be used to provide receiving addresses.

2. Compromised (or not): generally, paper wallets from other people should be considered compromised, since the former owner may still keep a copy of the privkey, therefore, imported (extended) privkeys/mnemonics should be marked as compromised by default, then this flag could be removed by cautious consent by the user, or sweeped into owned address(es) completely.

3. Backup required/Last backup at [time]: since it's a self-custody situation, it seems that no "silver bullet" is possible - it's always possible for the crucial data to be destroyed/leaked unexpectedly. Also, if two backup mechanisms (wallet.dat or text-based WIF/xprv/mnemonic) co-exist, it might confuse the user, too. In my opinion, the task of backup/recovery should mainly rely on text-based ways, rather than simply making a plain copy of wallet.dat, since text is human-readable. There is also an issue like #1666.

4. [READ-ONLY] This wallet.dat database was last saved at [time]: it might be better that an external wallet.dat (typically a backup) could be loaded in read-only mode by default. In this mode, the "backup required" flag(s) mentioned above might be greyed, so that the confusion between two backup mechanisms could be relieved. If the user wants to sync the wallet, I think Bitcoin Core should create a writable copy for it, in the data directory.

5. Derived/independently generated/imported, at [time]: the wallet should remember the origins of its contents.

6. Last exported at [time] for [purpose]: the reason why hardened address derivation is used by Bitcoin Core is that revealing extended pubkey together with a non extended child privkey would result in revealing the whole chain of keys. Besides the risk mentioned above, it's obvious that "one compromised, all compromised", while (extended) privkeys/mnemonics are imported into multiple wallets. Therefore, I think Bitcoin Core should educate/warn its users not to carelessly export private keys.

7. Default address derivation seed: currently, sethdseed (by default) simply replaces the HD seed with a newly generated, not-yet-backuped one, as far as I know. With this flag, the old HD seed could still be retained, thus the origins of privkeys derived by it could still be clear. This should also be useful to the watch-only side of cold storage.

Once there's a classification mechanism, the wallet would then gain the ability to filter its contents, by these flags/properties.

Why BIP39/44/49/84?

First of all, as far as I know, Mastering Bitcoin has been introducing BIP32/39/44 scheme to its readers for many years.

There are also so many users who trust Bitcoin Core so deeply that they don't even bother to try any other wallet. Shouldn't these users be taken care of?

Bitcoin Core is also recognized as "the reference implementation", that's saying, what Bitcoin Core supported is highly hopeful to be supported by other projects as well, which really seems to be hopeful to relieve the current fragmented situation.

BTW, I actually googled "bikeshedding" for what it means - I learned that it means "technical disputes over minor, marginal issues conducted while more serious ones are being overlooked." Frankly speaking, I think the current fragmented situation of mnemonic seed specification really fits with this definition - sorry for my rudeness, but I would rather to be a rude egoistic d*ck, than just keep silent to watch more and more new recruits joining the army of "single WIF privkey fetishism".

1. It's said that BIP39 has very weak checksum, thus it might deviate into an insecure human-picked brainwallet scheme - then why couldn't Bitcoin Core itself always securely&randomly generate mnemonic seeds for its user? It could also warn the user about "don't pick your mnemonic words with your brain" while importing a BIP39 mnemonic.

2. It's said that BIP39 doesn't really have plausible deniability, and, it has a weak KDF - then at least Bitcoin Core could simply refuse to create passphrase while generating a new seed, and, it could also warn the user about the "inherent linkage" and "prone-to-crack" issues while importing a passphrase-enabled BIP39 mnemonic. (I'm still confused/unconvinced on this topic, to my understanding, it's impossible for BIP39 to generate addresses with any publicly observable linkage - was @gmaxwell talking about situations like DreadPirateRoberts being arrested? If so, I don't think this is very convincing either, because as long as ther user's computer got impounded, and the encrypted wallet got unlocked, the difference mentioned by Greg seems to be really litte, otherwise, almost anything found on a computer could be "plausibly deniable".)

3. It's said that BIP39 lacks a version number - to be honest, I don't think this should be considered as a "flaw", since a seed should be treated essentially as an universal "root" backup of everything, for example, a user can upgrade to SegWit anytime (s)he wants, without making a new backup.

Describe alternatives you've considered

What if a better proposal come into being? In my opinion, a standardized proposal is always the best proposal, far better than a mess of "improved proposals" which are incompatible with each other.

I don't think there's completely no hope that a better solution could be come up with, like, an invertible encoding that is able to express the BIP32 seed directly, so that any fancy mnemonic could be converted into it - however, in my view, BIP39 has already penetrated deeply into the public mind, that it's very unlikely to replace it.

Additionally, BIP39 has already made use of so many common words, that any potential successor of BIP39 is very likely to face the ambiguity problem.

In one word: I think it's far more important to make the users feel confident and safe, that their carefully preserved mnemonic words would still be usable in the future (without being "bit-rotten").

Additional context

https://bitcoin.stackexchange.com/questions/88237/is-there-a-reason-to-why-bitcoin-core-does-not-implement-bip39

To address the "messy wallet.dat" problem, there is an ongoing project called descriptor wallets (https://github.com/bitcoin/bitcoin/projects/12) that will revamp how thing are structured internally. While descriptors (see https://github.com/bitcoin/bitcoin/blob/master/doc/descriptors.md) are already supported in import, they're still converted at import time to the old wallet.dat structures. With descriptor wallets, the actual description of the addresses/keys in the wallet will be one or a few descriptors, along with some metadata. That's not quite what you're talking about with BIP39-like constructions, but I believe it's a large improvement in minimizing and structuring the amount of state that needs to be safeguarded.

As far as mnemonics goes, I think it's very reasonable to have support for converting various kinds to descriptors and importing those once we have descriptor wallets, as that should be very easy and safe at that point.

If you're talking about just mnemonics there is the complication that there isn't really a watch-only version of them. Wallets should know what to watch (what type of scripts, keys for other multisig participants, possibly policies that degrade over time using locktime, ...) independently of what their own private key is. I expect that that will end up just being descriptors. So I don't know what mnemonics support would look like, and how useful it is, if you still need a separate (but non-secret) description of your wallet.

I just suggested that, at least, Bitcoin Core should support the standardized scheme of BIP39/44/49/84.

The user could just simply import the mnemonic, then the first accounts (in use) of three different address types (legacy/p2sh-segwit/bech32) would show up. If the user opened something like "wallet explorer" (could be the successor of " sending/ receiving address" and "coin selection" window), there would be a hierarchical view showing that the mnemonic derived everything as "root".

@sipa Sorry, I'm not sure whether I have completely understood what you said...

To address the "messy wallet.dat" problem, there is an ongoing project called descriptor wallets

What you were talking about seemed to be low-level technical things that I'm not able to evaluate.

I actually suggested that a single wallet.dat could already provide the functions of "multi-wallet". I was not quite sure on this, since Bitcoin Core had already supported multi-wallet (loading multiple wallet.dat) feature.

Besides, what I suggested is far more than "cleaning up the wallet.dat database" - I think the current situation that, text labels being the only way to mark/classify addresses/keys/etc, is too primitive, so that we need a more advanced/powerful way to manage them.

If you're talking about just mnemonics there is the complication that there isn't really a watch-only version of them. Wallets should know what to watch (what type of scripts, keys for other multisig participants, possibly policies that degrade over time using locktime, ...) independently of what their own private key is.

To my understanding, since a BIP39 mnemonic phrase is supposed to be universal, there will never be a "ultimate" comprehensive "description" of "what to watch".

However, the wallet only needs to watch what its user needs to watch.

For BIP44/49/84, three xpubs would be enough to cover the simplest situation (only one account was in use).

Here comes another question: since we already have PSBT that was intended to solve the cross-implementation interoperability problem during transaction signing process, why couldn't we have a similar standard for "watch-only wallet descrption"?

@sipa

With descriptor wallets, the actual description of the addresses/keys in the wallet will be one or a few descriptors, along with some metadata.

Addresses/keys, and xpubs (and its various descriptors) are distinct things.

The text labels provided by Bitcoin Core are not quite a powerful tool to manage the wallet. However, text labels will obviously still be needed in the future, on per address/key basis, even if we will be able to automatically derive addresses on-the-fly from descriptors.

I don't know how these data should be stored at lower level. I just think it will be still necessary to expose the addresses/keys to the user, thus the user will be able to manage them efficiently. Besides, I think some advices should be told to those "single WIF privkey fetishists", especially to their potential new recruits.

A lot of what you are asking for is already done, or in the works via descriptor wallets. Just some things aren't exposed to users yet.

Please also stop mixing together BIP 39 with BIP 44/49/84, they describe different things. BIP 39 is the mnemonic scheme which is wholly independent of the derivation paths described by BIPs 44/49/84.

I actually suggested that a single wallet.dat could already provide the functions of "multi-wallet". I was not quite sure on this, since Bitcoin Core had already supported multi-wallet (loading multiple wallet.dat) feature.

This is what we did in the past. A single wallet.dat did a ton of things as if it were multiple wallets. This is why the wallet.dat is so messy, because it was one file that tried to act like multiple. We are moving away from this model, and I don't think it is a good idea to try to keep it up. Everything that you want that are multiple wallets should be multiple wallet files. Otherwise the code gets extremely messy, wallet.dat files get messy, and we get a combinatorial blow up of possibilities. It's far simpler to just have multiple wallets with each wallet having some flag indicating what it is.

For example, with descriptor wallets, we are getting rid of mixed watch-only and non-watch-only. A wallet will have the PRIVATE_KEYS_DISABLED flag to indicate that it is a watch-only wallet.

[READ-ONLY] This wallet.dat database was last saved at [time]

We basically already have this. The wallet records the height and hash of the most recent block. It's close enough to a timestamp.

Derived/independently generated/imported, at [time]

The birth/import time of keys is already recorded.

Default address derivation seed: currently, sethdseed (by default) simply replaces the HD seed with a newly generated, not-yet-backuped one, as far as I know. With this flag, the old HD seed could still be retained, thus the origins of privkeys derived by it could still be clear. This should also be useful to the watch-only side of cold storage.

The seed is not deleted. It is still in the wallet file, just not active. Furthermore, any keys that were generated from it still have their metadata say that it was generated by the old seed.

Regarding BIPs 44/49/84, descriptor wallets will default to creating descriptors that follow the derivation scheme. We will not be using the ypub/zpub export format as descriptors are far more descriptive than those are. The current wallet architecture does not allow for us to use the different derivation paths for the different address types, but descriptor wallets will.

@achow101

A lot of what you are asking for is already done, or in the works via descriptor wallets. Just some things aren't exposed to users yet.

Thank you for clarification. It may or may not be proper to expose them to the users in the future, however, it's not my core idea. So far, in my opinion, it seems to be better to expose them.

This is what we did in the past. A single wallet.dat did a ton of things as if it were multiple wallets. This is why the wallet.dat is so messy, because it was one file that tried to act like multiple. We are moving away from this model, and I don't think it is a good idea to try to keep it up.

It's fait accompli. Creating new wallet.dat types won't eliminate all those existing old messy wallet.dats.

Please also stop mixing together BIP 39 with BIP 44/49/84, they describe different things. BIP 39 is the mnemonic scheme which is wholly independent of the derivation paths described by BIPs 44/49/84.

I didn't mix them up. I just said that it's exactly the point of BIP39 to be "wholly independent of the derivation paths described by BIPs 44/49/84", which was on the contrary considered a flaw of it.

Just like what Mastering Bitcoin had said:

As bitcoin wallet technology has matured, certain common industry standards have emerged that make bitcoin wallets broadly interoperable, easy to use, secure, and flexible. These common standards are:

• Mnemonic code words, based on BIP-39

• HD wallets, based on BIP-32

• Multipurpose HD wallet structure, based on BIP-43

• Multicurrency and multiaccount wallets, based on BIP-44

These standards may change or may become obsolete by future developments, but for now they form a set of interlocking technologies that have become the de facto wallet standard for bitcoin.

The standards have been adopted by a broad range of software and hardware bitcoin wallets, making all these wallets interoperable. A user can export a mnemonic generated on one of these wallets and import it in another wallet, recovering all transactions, keys, and addresses.

The core idea I had suggested was that:

1. Bitcoin Core should support the standardized scheme of BIP39/44/49/84.

2. It should be the priority to deliver an usable solution to the users, rather than to wait for a nealy-ideal proposal to appear.

3. A standardized proposal is always the best proposal, far better than a mess of "improved proposals" which are incompatible with each other.

Otherwise, it's already scary enough to push the users to join the army of "single precious WIF privkey fetishism".

We will not be using the ypub/zpub export format as descriptors are far more descriptive than those are.

At least Bitcoin Core should support importing them. I think the interoperability provided by supporting ypub/zpub really worth it, although it might seem to be "a non-normative way" or "a layer violation".

I have built Bitcoin Core from source with the "native descriptor wallet" commit.

I can see the "Disable private key" option in the wallet creation dialog. I agree that it might be an useful feature (or a good forcing function).

However, I still disagree with the design or idea behind this option, that abilities of wallet.dat should be artificially limited according to its "predetermined type".

As we all know, wallet.dat is a database, which contains detailed information, including both recoverable things like raw transaction data (and corresponding amount/date/recipient address etc), and non-recoverable things like labels and generation dates.

Therefore, a piggy backup of the whole wallet.dat should always be a distinct/independent function other than portable last-resort backup/recovery measures like WIF/mnemonic.

Since wallet.dat is a database, why couldn't it provide comprehensive management of various types of contents? IMHO, it should be able to remember (or even better, to analyse/locate) origins of each addr/pubkey/privkey stored inside it.

To address the privacy issue around coin selection, I think Bitcoin Core should bring back the grouping feature, which seems to be similar to the abandoned "account" mechanism.

The grouping mechanism should be exposed to the user in some form like "sub-wallet". A simple "Venn diagram" should be able to explain this to the user.

Edit: the user could be switch among different sub-wallets (or in other words, to activate one of them, while deactivating others at same time), just like the current multi-wallet.dat situation.

If this feature is considered to face too many complicated situations, I think Bitcoin Core could simply automatically complete the "grouping" job by itself (disallowing the user to change the grouping), just as if they were in different wallet.dat's.

I think there could be a compromise still, that only standalone privkeys could be manually grouped by the user.

Oh, the classic screw-up-by-change-address issue should also be addressed here.

I think Bitcoin Core should simply never automatically put the change coins onto newly generated (not-yet-backuped) addresses, since it seems really obvious to me that fund safety is much more important than privacy. It's better to use warnings/click-throughs to shape the user behavoir/habit, instead of silently introducing the risk of fund loss.

Ah. I found myself to be in fact bikeshedding... It's really embarrassing. If I've generated too much noise, I would say sorry for this.

However, I didn't change my idea that the problems mentioned in my initial post should be addressed.

Anyway, I found that the "native descriptor wallet" commit had adopted the standard BIP44/49/84 derivation path, which seems to be a good start.

I still hope that mnemonic-based HD wallet backup could also been brought into Bitcoin Core soon, BIP39 to be best.

As far as I know, "multi-wallet management" could be a really complicated problem, that even some security vulnerabilities are probably related to this - just like the case of Trezor, which was considered to be vague borderline of "what addresses belong to the current wallet" by myself.

I feel that this problem might be really out of my scope of knowledge. Maybe I have just overthought/misunderstood this...

Since wallet.dat is a database, why couldn't it provide comprehensive management of various types of contents?

Limitations of previous wallet architecture prevent us from doing that. We could just outright change it to something else but such changes are drastic, large, and hard to review, so they basically don't get merged. Moving slowly to another architecture is hard because incrementally changing to something very different is hard to do. It might be easier now with the ScritpPubKeyMan separation and I've been exploring more of BDB's features that could be useful to us.

I think Bitcoin Core should bring back the grouping feature, which seems to be similar to the abandoned "account" mechanism.

This feature never existed. The fact that people think that's what accounts did is part of why accounts was removed.

I think Bitcoin Core should simply never automatically put the change coins onto newly generated (not-yet-backuped) addresses,

How do you know if someone has made a backup?

How do you know if someone has made a backup?

This could be done by a backup warning/guidance mechanism (which is very common in other wallet apps nowadays!), in which only the "portable last-resort backup/recovery measures like WIF/mnemonic" should really count.

However, it's not my central point. In my opinion, even address reusing is still probably much better than silently introducing fund loss risks.

such changes are drastic, large, and hard to review

Agreed. I would actually like to put this topic aside for the moment, and then bring up the mnemonic-based backup to priority.

Edit: boldly speaking, maybe some drastic, large changes are inevitably needed to make Bitcoin Core really both user-friendly and powerful, although I don't think I have the knowledge to do the job of wallet architecture design, however.

Ah, during an experiment of creating a watch-only wallet of Satoshi coins, I imported Satoshi's pubkey into my own wallet.dat for daily use by mistake :-/

I think it would be great if I can revert this stupid human error (like, to disable it, or mark-as-hidden), otherwise I would have to recreate my daily-use wallet.dat, or just live with the fact that my Bitcoin Core always counts Satoshi's coins as if they were mine.

See also #19151: “Support BIP39 mnemonic in descriptors”

I think this issue has been solved by descriptor wallets. We can keep open #19151 for more discussion about using BIP39 in Bitcoin Core

cc @achow101

This issue is unlikely to be fixed in Bitcoin Core. We'll close for now, but feel free to open another issue or pull request with a fix.