wallet: Migrate legacy wallets to descriptor wallets without requiring BDB #26596

pull achow101 wants to merge 6 commits into bitcoin:master from achow101:indep-desc-migrate2 changing 6 files +221 −125
  1. achow101 commented at 11:46 pm on November 28, 2022: member
    #26606 introduced BerkeleyRODatabase which is an independent parser for BDB files. This PR uses this in legacy wallet migration so that migration will continue to work once the legacy wallet and BDB are removed. LegacyDataSPKM is introduced to have the minimum data and functions necessary for a legacy wallet to be loaded for migration.
  2. DrahtBot commented at 11:46 pm on November 28, 2022: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage

    For detailed information about the code coverage, see the test coverage report.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK cbergqvist, theStack, furszy

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #29124 (wallet: Automatically repair corrupted metadata with doubled derivation path by achow101)
    • #28574 (wallet: optimize migration process, batch db transactions by furszy)
    • #28333 (wallet: Construct ScriptPubKeyMans with all data rather than loaded progressively by achow101)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  3. DrahtBot added the label Wallet on Nov 28, 2022
  4. achow101 force-pushed on Nov 28, 2022
  5. achow101 force-pushed on Nov 29, 2022
  6. Sjors commented at 2:41 pm on November 29, 2022: member

    I think it would be easier to review if you made a separate PR for BerkeleyRODatabase and have it work as a regular, albeit read-only, wallet database backend.

    I’m pleasantly surprised at how small the implementation is.

  7. achow101 force-pushed on Nov 29, 2022
  8. luke-jr commented at 9:20 pm on November 29, 2022: member
    I don’t see how reimplementing BDB is better than just continuing to use BDB.
  9. achow101 force-pushed on Nov 29, 2022
  10. achow101 commented at 9:27 pm on November 29, 2022: member

    I don’t see how reimplementing BDB is better than just continuing to use BDB.

    The reimplementation is quite compact because it is for only reading BDB files, only the features that we use, and none of the actual DB system stuff. BDB itself contains a ton of code, a lot of which we don’t use. Maintaining it as a dependency will eventually be more burdensome - we already need to patch it in order to compile in the depends system. This reimplementation does not carry those issues - it implements a format that isn’t going to change, and it only depends on things that are already being used elsewhere in the codebase.

  11. achow101 commented at 9:51 pm on November 29, 2022: member

    I think it would be easier to review if you made a separate PR for BerkeleyRODatabase and have it work as a regular, albeit read-only, wallet database backend.

    I’ve opeend #26606 for just BerkeleyRODatabase and its use in wallettool’s dump.

  12. achow101 force-pushed on Nov 30, 2022
  13. achow101 force-pushed on Nov 30, 2022
  14. achow101 force-pushed on Nov 30, 2022
  15. DrahtBot added the label Needs rebase on Dec 1, 2022
  16. achow101 force-pushed on Dec 1, 2022
  17. DrahtBot removed the label Needs rebase on Dec 1, 2022
  18. achow101 force-pushed on Dec 2, 2022
  19. achow101 force-pushed on Dec 2, 2022
  20. DrahtBot added the label Needs rebase on Feb 1, 2023
  21. achow101 force-pushed on Feb 1, 2023
  22. achow101 force-pushed on Feb 2, 2023
  23. DrahtBot removed the label Needs rebase on Feb 2, 2023
  24. DrahtBot added the label Needs rebase on Feb 17, 2023
  25. achow101 force-pushed on Feb 17, 2023
  26. DrahtBot removed the label Needs rebase on Feb 17, 2023
  27. DrahtBot added the label Needs rebase on Feb 22, 2023
  28. achow101 force-pushed on Mar 1, 2023
  29. DrahtBot removed the label Needs rebase on Mar 1, 2023
  30. DrahtBot added the label CI failed on May 2, 2023
  31. achow101 force-pushed on May 2, 2023
  32. achow101 force-pushed on May 2, 2023
  33. DrahtBot removed the label CI failed on May 2, 2023
  34. achow101 force-pushed on May 27, 2023
  35. DrahtBot added the label CI failed on Jun 7, 2023
  36. achow101 force-pushed on Jun 8, 2023
  37. DrahtBot removed the label CI failed on Jun 8, 2023
  38. DrahtBot added the label Needs rebase on Jun 28, 2023
  39. achow101 force-pushed on Jun 28, 2023
  40. DrahtBot removed the label Needs rebase on Jun 28, 2023
  41. DrahtBot added the label CI failed on Jun 29, 2023
  42. achow101 force-pushed on Jun 29, 2023
  43. DrahtBot removed the label CI failed on Jul 6, 2023
  44. DrahtBot added the label CI failed on Jul 23, 2023
  45. DrahtBot added the label Needs rebase on Aug 1, 2023
  46. achow101 force-pushed on Aug 1, 2023
  47. DrahtBot removed the label Needs rebase on Aug 1, 2023
  48. achow101 force-pushed on Sep 15, 2023
  49. DrahtBot added the label Needs rebase on Sep 19, 2023
  50. achow101 force-pushed on Sep 27, 2023
  51. DrahtBot removed the label Needs rebase on Sep 27, 2023
  52. achow101 force-pushed on Sep 27, 2023
  53. DrahtBot removed the label CI failed on Sep 28, 2023
  54. DrahtBot added the label Needs rebase on Oct 23, 2023
  55. achow101 force-pushed on Oct 23, 2023
  56. DrahtBot removed the label Needs rebase on Oct 24, 2023
  57. DrahtBot added the label CI failed on Oct 24, 2023
  58. achow101 force-pushed on Oct 24, 2023
  59. achow101 force-pushed on Nov 28, 2023
  60. DrahtBot removed the label CI failed on Nov 28, 2023
  61. DrahtBot added the label Needs rebase on Dec 14, 2023
  62. achow101 force-pushed on Dec 19, 2023
  63. DrahtBot removed the label Needs rebase on Dec 19, 2023
  64. DrahtBot added the label Needs rebase on Jan 5, 2024
  65. achow101 force-pushed on Jan 5, 2024
  66. DrahtBot removed the label Needs rebase on Jan 6, 2024
  67. DrahtBot added the label CI failed on Jan 13, 2024
  68. DrahtBot added the label Needs rebase on Jan 31, 2024
  69. achow101 force-pushed on Feb 1, 2024
  70. achow101 force-pushed on Feb 1, 2024
  71. DrahtBot removed the label Needs rebase on Feb 1, 2024
  72. achow101 force-pushed on Feb 1, 2024
  73. achow101 force-pushed on Feb 1, 2024
  74. DrahtBot removed the label CI failed on Feb 2, 2024
  75. DrahtBot added the label Needs rebase on Feb 3, 2024
  76. achow101 force-pushed on Feb 3, 2024
  77. DrahtBot removed the label Needs rebase on Feb 3, 2024
  78. DrahtBot added the label Needs rebase on Feb 6, 2024
  79. achow101 force-pushed on Feb 8, 2024
  80. DrahtBot removed the label Needs rebase on Feb 8, 2024
  81. DrahtBot added the label CI failed on Feb 18, 2024
  82. achow101 force-pushed on Feb 20, 2024
  83. DrahtBot removed the label CI failed on Feb 20, 2024
  84. DrahtBot added the label CI failed on Feb 28, 2024
  85. DrahtBot commented at 6:51 am on February 28, 2024: contributor

    🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the documentation.

    Possibly this is due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    Leave a comment here, if you need help tracking down a confusing failure.

    Debug: https://github.com/bitcoin/bitcoin/runs/21780750809

  86. achow101 force-pushed on Feb 29, 2024
  87. achow101 force-pushed on Mar 11, 2024
  88. achow101 force-pushed on Apr 1, 2024
  89. DrahtBot removed the label CI failed on Apr 5, 2024
  90. achow101 force-pushed on Apr 25, 2024
  91. achow101 force-pushed on Apr 25, 2024
  92. achow101 force-pushed on Apr 26, 2024
  93. fanquake referenced this in commit 5acdc2b97d on May 21, 2024
  94. DrahtBot added the label Needs rebase on May 21, 2024
  95. achow101 force-pushed on May 21, 2024
  96. achow101 commented at 4:47 pm on May 21, 2024: member
    Rebased following #26606. Now ready for review.
  97. achow101 marked this as ready for review on May 21, 2024
  98. DrahtBot removed the label Needs rebase on May 21, 2024
  99. in src/wallet/wallet.cpp:4484 in 37df50fffa outdated
    4464@@ -4438,6 +4465,7 @@ util::Result<MigrationResult> MigrateLegacyToDescriptor(const std::string& walle
    4465     // both before and after reloading. This ensures the set is complete even if one of the wallets
    4466     // fails to reload.
    4467     std::set<fs::path> wallet_dirs;
    4468+    options.require_format = std::nullopt;
    


    cbergqvist commented at 7:22 pm on May 30, 2024:

    Why reset require_format here? Seems options is not going to be read by anything after this line and none of the database types is holding a reference to it.

    Avoiding that would mean that you could introduce more immutability around line 4386:

    0    const DatabaseOptions options{
    1        .require_existing = true,
    2        .require_format = DatabaseFormat::BERKELEY_RO,
    3    };
    

    achow101 commented at 0:51 am on June 6, 2024:
    It’s used by the reload_wallet calls which capture it.

    cbergqvist commented at 9:32 am on June 6, 2024:
    Thanks, sorry for missing that. Worth an explaining comment like "Reset require_format since reload_wallet will only be called on non-Berkeley DB sub-wallets from here on." or something more correct?

    achow101 commented at 4:12 pm on June 6, 2024:
    Added a comment.
  100. in src/wallet/wallet.cpp:3659 in 20c8fd5dfc outdated
    3646+    // Only create base LegacyDataSPKM if using BERKELEY_RO
    3647+    if (m_database->Format() == "bdb_ro") {
    3648+        spk_manager = std::unique_ptr<ScriptPubKeyMan>(new LegacyDataSPKM(*this));
    3649+    } else {
    3650+        spk_manager = std::unique_ptr<ScriptPubKeyMan>(new LegacyScriptPubKeyMan(*this, m_keypool_size));
    3651+    }
    


    cbergqvist commented at 8:42 pm on May 30, 2024:
    0    // Only create base LegacyDataSPKM if using BERKELEY_RO
    1    std::unique_ptr<ScriptPubKeyMan> spk_manager = m_database->Format() == "bdb_ro" ?
    2        std::make_unique<LegacyDataSPKM>(*this) :
    3        std::make_unique<LegacyScriptPubKeyMan>(*this, m_keypool_size);
    

    achow101 commented at 1:09 am on June 6, 2024:
    Done as suggested
  101. in src/wallet/wallet.cpp:4379 in 37df50fffa outdated
    4374+
    4375+        if (wallet->IsWalletFlagSet(WALLET_FLAG_DESCRIPTORS)) {
    4376+            return util::Error{_("Error: This wallet is already a descriptor wallet")};
    4377+        }
    4378+        // Unload
    4379+        wallet.reset();
    


    furszy commented at 2:59 pm on June 3, 2024:
    Loading the entire wallet just to obtain the descriptors flag shouldn’t be needed. Could just check the file type instead. We want a bdb file here. So, something like https://github.com/furszy/bitcoin-core/commit/7a03a039ae99e69504fe9df95b04f5e326c45a70 would work well. Yet, the code isn’t the best but it includes a test that verifies the behavior.

    achow101 commented at 11:34 pm on June 4, 2024:
    Good point, I think the original intention was to be able to handle the legacy-sqlite case, but that’s moot since we require BDB_RO later. I’ve changed this to check for just look for BDB files, and also added your test.
  102. achow101 force-pushed on Jun 4, 2024
  103. in src/wallet/scriptpubkeyman.h:281 in e11fa45247 outdated
    275@@ -276,31 +276,106 @@ static const std::unordered_set<OutputType> LEGACY_OUTPUT_TYPES {
    276 
    277 class DescriptorScriptPubKeyMan;
    278 
    279-class LegacyScriptPubKeyMan : public ScriptPubKeyMan, public FillableSigningProvider
    280+class LegacyDataSPKM : public ScriptPubKeyMan, public FillableSigningProvider
    


    cbergqvist commented at 7:04 pm on June 5, 2024:
    This class and possibly LegacyScriptPubKeyMan below could do with comments summarizing their responsibilities since breaking them apart and naming one LegacyDataSPKM makes things less clear.

    achow101 commented at 1:09 am on June 6, 2024:
    Add a comment to each
  104. in src/wallet/scriptpubkeyman.h:386 in e11fa45247 outdated
    386-    bool AddKeyPubKeyInner(const CKey& key, const CPubKey &pubkey);
    387-    bool AddCryptedKeyInner(const CPubKey &vchPubKey, const std::vector<unsigned char> &vchCryptedSecret);
    388+    bool AddKeyPubKeyInner(const CKey& key, const CPubKey &pubkey) override;
    389 
    390     /**
    391      * Private version of AddWatchOnly method which does not accept a
    


    cbergqvist commented at 8:04 pm on June 5, 2024:
    This comment goes on to say AddWatchOnly(const CScript&) is an inherited virtual method which appears untrue. Maybe add an extra commit ahead of the others that tidies up that comment?

    achow101 commented at 0:52 am on June 6, 2024:
    I don’t think it’s necessary to add changes that correct this documentation when it’s all going to be deleted soon anyways.
  105. in src/wallet/walletdb.cpp:359 in e11fa45247 outdated
    353@@ -354,7 +354,7 @@ bool LoadKey(CWallet* pwallet, DataStream& ssKey, DataStream& ssValue, std::stri
    354             strErr = "Error reading wallet database: CPrivKey corrupt";
    355             return false;
    356         }
    357-        if (!pwallet->GetOrCreateLegacyScriptPubKeyMan()->LoadKey(key, vchPubKey))
    358+        if (!pwallet->GetOrCreateLegacyDataSPKM()->LoadKey(key, vchPubKey))
    359         {
    360             strErr = "Error reading wallet database: LegacyScriptPubKeyMan::LoadKey failed";
    


    cbergqvist commented at 8:10 pm on June 5, 2024:
    *"Error reading wallet database: LegacyDataSPKM::LoadKey failed", same for LoadCryptedKey and LoadCScript below.

    achow101 commented at 1:09 am on June 6, 2024:
    Done as suggested
  106. in src/wallet/wallet.h:967 in e11fa45247 outdated
    960@@ -961,6 +961,8 @@ class CWallet final : public WalletStorage, public interfaces::Chain::Notificati
    961     //! Get the LegacyScriptPubKeyMan which is used for all types, internal, and external.
    962     LegacyScriptPubKeyMan* GetLegacyScriptPubKeyMan() const;
    963     LegacyScriptPubKeyMan* GetOrCreateLegacyScriptPubKeyMan();
    964+    LegacyDataSPKM* GetLegacyDataSPKM() const;
    965+    LegacyDataSPKM* GetOrCreateLegacyDataSPKM();
    966 
    967     //! Make a LegacyScriptPubKeyMan and set it for all types, internal, and external.
    


    cbergqvist commented at 8:25 pm on June 5, 2024:
    ("Make a *LegacyDataSPKM and set it...". Similar need for update in wallet.cpp around line 3019 before call to SetupLegacyScriptPubKeyMan().)

    achow101 commented at 0:53 am on June 6, 2024:
    The function still creates a LegacyScriptPubKeyMan in normal operation.

    cbergqvist commented at 9:40 am on June 6, 2024:
    LegacyScriptPubKeyMan is a LegacyDataSPKM, so the comment would be more correct, but it could be corrected later when removing LegacyScriptPubKeyMan - unless you plan to rename LegacyDataSPKM back into LegacyScriptPubKeyMan once the deletion of the latter is complete.

    achow101 commented at 3:57 pm on June 6, 2024:

    I think the meaning is well understood, both before and after LegacyScriptPubKeyMan is deleted. SPKM is an abbreviation for ScriptPubKeyMan anyways.

    Changed it to Legacy(Data)SPKM.

  107. in src/wallet/wallet.cpp:2926 in e11fa45247 outdated
    2922@@ -2923,7 +2923,7 @@ bool CWallet::EraseAddressReceiveRequest(WalletBatch& batch, const CTxDestinatio
    2923     return true;
    2924 }
    2925 
    2926-std::unique_ptr<WalletDatabase> MakeWalletDatabase(const std::string& name, const DatabaseOptions& options, DatabaseStatus& status, bilingual_str& error_string)
    2927+util::Result<fs::path> GetWalletPath(const std::string& name)
    


    cbergqvist commented at 8:42 pm on June 5, 2024:
    Could be marked static?

    achow101 commented at 1:09 am on June 6, 2024:
    Done
  108. in src/wallet/scriptpubkeyman.cpp:1869 in 46520aacc4 outdated
    1885@@ -1887,7 +1886,6 @@ std::optional<MigrationData> LegacyScriptPubKeyMan::MigrateToDescriptor()
    1886             for (const CScript& spk : desc_spks) {
    1887                 size_t erased = spks.erase(spk);
    1888                 assert(erased == 1);
    1889-                assert(IsMine(spk) == ISMINE_SPENDABLE);
    


    cbergqvist commented at 9:07 pm on June 5, 2024:
    In commit 46520aacc43dd42a7968587ab9ece488bbe9503d: Should this line have been removed in the following commit (31e918afc3920a2f8aa367ad323f590b4856aa26)?

    achow101 commented at 0:55 am on June 6, 2024:
    No. LegacyDataSPKM does not have an IsMine() function, so this call needs to be removed otherwise it will not compile.

    furszy commented at 9:21 pm on June 11, 2024:

    No. LegacyDataSPKM does not have an IsMine() function, so this call needs to be removed otherwise it will not compile.

    I think the outcome would be worst. It would compile using the base ScriptPubKeyMan::IsMine function. Returning ISMINE_NO for all IsMine calls.

    Still, I think we should state why this assertion removal is ok in the commit description too.

  109. in src/wallet/scriptpubkeyman.h:455 in e11fa45247 outdated
    499@@ -500,7 +500,6 @@ class LegacyScriptPubKeyMan : public LegacyDataSPKM
    500 
    501     /* Set the HD chain model (chain child index counters) and writes it to the database */
    502     void AddHDChain(const CHDChain& chain);
    503-    const CHDChain& GetHDChain() const { return m_hd_chain; }
    


    cbergqvist commented at 9:28 pm on June 5, 2024:
    This was removed in e11fa4524723aebc0d879ee083c2d6d46849d89a but probably shouldn’t have been included in 84a5fb96382960300ebbd2cf2fe8756142bea572 (verified through compiling on that commit + with it removed).

    achow101 commented at 1:09 am on June 6, 2024:
    Done
  110. in src/wallet/scriptpubkeyman.cpp:1956 in e11fa45247 outdated
    1952@@ -1812,7 +1953,7 @@ std::optional<MigrationData> LegacyScriptPubKeyMan::MigrateToDescriptor()
    1953         WalletDescriptor w_desc(std::move(desc), creation_time, 0, 0, 0);
    1954 
    1955         // Make the DescriptorScriptPubKeyMan and get the scriptPubKeys
    1956-        auto desc_spk_man = std::unique_ptr<DescriptorScriptPubKeyMan>(new DescriptorScriptPubKeyMan(m_storage, w_desc, m_keypool_size));
    1957+        auto desc_spk_man = std::unique_ptr<DescriptorScriptPubKeyMan>(new DescriptorScriptPubKeyMan(m_storage, w_desc, /*keypool_size=*/0));
    


    cbergqvist commented at 10:04 pm on June 5, 2024:
    Worth adding a comment about the consequences of setting keypool_size=0? I’m guessing there are still cases where we generate some pubkeys despite having a zero pool size, otherwise you would have removed some of the code and the for-loop below?

    cbergqvist commented at 10:13 pm on June 5, 2024:

    nit: Might as well use make_unique on the lines you modify:

    0        auto desc_spk_man = std::make_unique<DescriptorScriptPubKeyMan>(m_storage, w_desc, /*keypool_size=*/0);
    

    achow101 commented at 0:58 am on June 6, 2024:
    I don’t think this needs to have any additional explanation. keypool_size only matters for ranged descriptors. Non-HD keys do not make ranged descriptors, so the value here is ignored. The loop below is for an entirely different set of descriptors and is unrelated to this.

    achow101 commented at 1:09 am on June 6, 2024:
    Done as suggested

    cbergqvist commented at 10:59 am on June 6, 2024:

    Haven’t fully grokked the code yet, so I understand if you’d rather not answer my questions, but if you want:

    I don’t think this needs to have any additional explanation. keypool_size only matters for ranged descriptors. Non-HD keys do not make ranged descriptors, so the value here is ignored.

    We’re in an HD chain for-loop, how is this not involving HD keys? Surely legacy wallets could be HD wallets.

    The loop below is for an entirely different set of descriptors and is unrelated to this.

    I’m referring to the loop over desc_spks which is using the pubkeys from the DescriptorScriptPubKeyMan that’s just been created and comes from DescriptorScriptPubKeyMan::m_map_script_pub_keys which is filled out in DescriptorScriptPubKeyMan::TopUpWithDB() which is affected by m_keypool_size.

    Would m_keypool_size already always be 0 before this change?


    achow101 commented at 4:07 pm on June 6, 2024:

    The line this comment is on is in the one that goes over the non-HD keys. However, I see that you’re confused about the similar line in the loop below that iterates the hd chains.

    The keypool_size here is a runtime value that depends on the startup options. It’s is set when the DescriptorScriptPubKeyman is instantiated on loading, and the value is never directly persisted to disk. It ends up written as part of range_end when range_end is updated during TopUp(), but range_end can also be set directly.

    The value that does matter is the range_end stored in WalletDescriptor. When we are going over the hd chains, we set range_end to match the chain counter stored in the CHDChain that is being migrated. This results in a descriptor that will produce exactly the same scriptPubKeys as the original hd chain, including it’s keypool. We are enforcing that the migrated descriptors produce exactly the same scriptPubKeys (no more, no less). Setting keypool_size could mean that TopUp() would generate more scriptPubKeys than we need/expect, which would cause migration to assert and crash.


    cbergqvist commented at 6:57 pm on June 6, 2024:
    Thanks for taking the time to clarify this matter!
  111. in src/wallet/walletdb.cpp:766 in e11fa45247 outdated
    762@@ -763,7 +763,7 @@ static DBErrors LoadLegacyWalletRecords(CWallet* pwallet, DatabaseBatch& batch,
    763 
    764         // nTimeFirstKey is only reliable if all keys have metadata
    765         if (pwallet->IsLegacy() && (key_res.m_records + ckey_res.m_records + watch_script_res.m_records) != (keymeta_res.m_records + watch_meta_res.m_records)) {
    766-            auto spk_man = pwallet->GetOrCreateLegacyScriptPubKeyMan();
    767+            auto spk_man = pwallet->GetLegacyScriptPubKeyMan();
    


    cbergqvist commented at 10:25 pm on June 5, 2024:
    Why not OrCreate here any longer?

    achow101 commented at 1:07 am on June 6, 2024:
    It’s redundant since IsLegacy() ensures the existence of a LegacyScriptPubKeyMan, so there’s no need to try to create one.
  112. cbergqvist commented at 10:36 pm on June 5, 2024: contributor

    Code reviewed e11fa4524723aebc0d879ee083c2d6d46849d89a.

    Overall concept: Not fully convinced about the benefit of breaking out LegacyDataSPKM, seems to add complexity. Is the goal to delete LegacyScriptPubKeyMan in the future when ripping out BDB and only keep LegacyDataSPKM?

    Why perform the inlining of IsMine() in 31e918afc3920a2f8aa367ad323f590b4856aa26 before the function is removed? Seems risky in case of other PRs modifying it prior to removal etc.

    LegacyScriptPubKeyMan::NewKeyPool() only touches data from LegacyDataSPKM, should it be moved to that class as public (or protected and then exposed in the subclass?). (If LegacyScriptPubKeyMan is slated for removal I can understand you want to signal that method will no longer be needed). (Have not checked if there are other methods along the same lines).

    Assuming LegacyDataSPKM name is kept, comment in src/script/signingprovider.h could be updated to refer to LegacyDataSPKM::setWatchOnly instead.

  113. achow101 commented at 0:36 am on June 6, 2024: member

    Is the goal to delete LegacyScriptPubKeyMan in the future when ripping out BDB and only keep LegacyDataSPKM?

    Yes.

    We are trying to remove the legacy wallet in its entirety, leaving only behind the absolute minimum necessary to perform migration.

    Why perform the inlining of IsMine() in 31e918a before the function is removed? Seems risky in case of other PRs modifying it prior to removal etc.

    IsMine() is unlikely to change. Besides the fact that any changes to the legacy wallet are likely to be closed instead of seriously considered, IsMine() has not had any material changes in the last several years. It could be included in #28710 but that PR is big enough as it is.

    LegacyScriptPubKeyMan::NewKeyPool() only touches data from LegacyDataSPKM, should it be moved to that class as public (or protected and then exposed in the subclass?)

    No. The separation exists now to indicate which things can be removed at the end. LegacyDataSPKM is the bare minimum that must stick around for migration, everything else remains in LegacyScriptPubKeyMan and will be removed at the same time.

  114. achow101 force-pushed on Jun 6, 2024
  115. in src/wallet/scriptpubkeyman.h:372 in 31632d18ea outdated
    373+    /** Delete all the records ofthis LegacyScriptPubKeyMan from disk*/
    374+    bool DeleteRecords();
    375+};
    376+
    377+// Implements the full legacy wallet behavior
    378+class LegacyScriptPubKeyMan : public LegacyDataSPKM
    


    cbergqvist commented at 9:48 am on June 6, 2024:
    0// Implements the full legacy wallet behavior
    1// Slated for deletion once Berkeley DB library dependency is removed.
    2class LegacyScriptPubKeyMan : public LegacyDataSPKM
    

    achow101 commented at 4:13 pm on June 6, 2024:
    I don’t think it’s necessary to have a comment saying such.

    cbergqvist commented at 6:48 pm on June 6, 2024:

    If the code is unable to communicate why it is like it is, it’s good to explain the otherwise surprising state of it. It would hopefully limit the amount of questions like my prior one from #26596#pullrequestreview-2089208116:

    LegacyScriptPubKeyMan::NewKeyPool() only touches data from LegacyDataSPKM, should it be moved to that class as public (or protected and then exposed in the subclass?).

    But I guess you’re fairly confident it will be deleted in time for the next release anyway.

  116. achow101 force-pushed on Jun 6, 2024
  117. cbergqvist approved
  118. cbergqvist commented at 7:24 pm on June 6, 2024: contributor

    ACK a8f8a2d5d0e2328cbe2e569c7e12f653d18f456d

    (Ran tests on 31632d18ea122def82820770f14d9aa009726114 but only comments changed since then). Functional tests including --extended passed (a few automatically skipped). make check passed.

  119. in src/wallet/wallet.cpp:4394 in f732495191 outdated
    4370+        if (!wallet_path) {
    4371+            return util::Error{util::ErrorString(wallet_path)};
    4372+        }
    4373+        if (!IsBDBFile(BDBDataFile(*wallet_path))) {
    4374+            return util::Error{_("Error: This wallet is already a descriptor wallet")};
    4375+        }
    


    theStack commented at 12:31 pm on June 7, 2024:

    in commit f732495191d346b877ffe8402ec68761713eaaf7: now that the legacy-wallet-check is done without loading the wallet, is the check for the descriptor flag below still needed? (can a BDB wallet ever have this flag set?)

    0    // Before anything else, check if there is something to migrate.
    1    if (local_wallet->IsWalletFlagSet(WALLET_FLAG_DESCRIPTORS)) {
    2        if (was_loaded) {
    3            reload_wallet(local_wallet);
    4        }
    5        return util::Error{_("Error: This wallet is already a descriptor wallet")};
    6    }
    

    https://github.com/achow101/bitcoin/blob/a8f8a2d5d0e2328cbe2e569c7e12f653d18f456d/src/wallet/wallet.cpp#L4425-L4431


    achow101 commented at 4:24 pm on June 7, 2024:

    It is theoretically possible to have a descriptor wallet using BDB under 2 scenarios, however this is still an unsupported configuration:

    1. The descriptor wallet was created on an unreleased version after #16528 but before #19077 (there was a period of about 6 months where this could have happened)
    2. The user manipulated the database using external tooling where they created a BDB file and manually inserted descriptor wallet records
  120. theStack commented at 12:35 pm on June 7, 2024: contributor
    Concept ACK
  121. in src/wallet/scriptpubkeyman.cpp:1833 in c653f4fdbf outdated
    1832@@ -1833,7 +1833,7 @@ std::optional<MigrationData> LegacyScriptPubKeyMan::MigrateToDescriptor()
    1833         WalletDescriptor w_desc(std::move(desc), creation_time, 0, 0, 0);
    1834 
    1835         // Make the DescriptorScriptPubKeyMan and get the scriptPubKeys
    1836-        auto desc_spk_man = std::unique_ptr<DescriptorScriptPubKeyMan>(new DescriptorScriptPubKeyMan(m_storage, w_desc, m_keypool_size));
    1837+        auto desc_spk_man = std::make_unique<DescriptorScriptPubKeyMan>(m_storage, w_desc, /*keypool_size=*/0);
    


    furszy commented at 1:46 pm on June 18, 2024:

    Not really an issue but this works because TopUp sets the descriptor range_end to 1 on the first run when the descriptor is not ranged. As this would fail if we ever change that, what if we set the range properly in WalletDescriptor constructor? E.g.

    0WalletDescriptor w_desc(std::move(desc), creation_time, /*range_start=*/0, /*range_end=*/1, /*next_index=*/0)
    

    Also, this change doesn’t seem to be related to c653f4fdbfe06 description? –> nvm about this last point, theStack just made me notice that this is because m_keypool_size is not accesible from LegacyDataSPKM.

  122. furszy commented at 10:26 pm on June 18, 2024: member

    Just a small comment; you don’t have to take it.

    I believe this PR could get merged quite fast if you leave 89503710386f52d9162f67fcd707eceffa954faa for a follow-up and make IsMine compatible with LegacyDataSPKM in this PR.

    I’m suggesting this because this PR starts testing the newly introduced BDB reader class and allows people to use it, which is great since we want it polished for the next release. The IsMine removal is also nice but I wouldn’t miss a deadline for it.

  123. theStack commented at 3:38 pm on June 21, 2024: contributor

    Still trying to fully grasp the IsMine inlining commit 89503710386f52d9162f67fcd707eceffa954faa, meanwhile a few comments:

    • The PR description is outdated, as it still mentions adding BerkeleyRODatabase which was already merged in #26606
    • in c653f4fdbfe0607ad9be27e80ad22c9aee314bb7: typo in commit subject (s/MigrateToLegacy/MigrateToDescriptor/)
    • To my understanding, the class LegacyDataSPKM is supposed to never write to the database (as BerkeleyRODatabase obviously doesn’t support that), but LegacyDataSPKM::CheckDecryptionKey contains a code-path for that (for a necessary rewrite): https://github.com/achow101/bitcoin/blob/a8f8a2d5d0e2328cbe2e569c7e12f653d18f456d/src/wallet/scriptpubkeyman.cpp#L254-L255 Should CheckDecryptionKey stay in LegacyScriptPubKeyMan?
  124. furszy commented at 9:27 pm on June 21, 2024: member

    It isn’t really an issue because CheckDecryptionKey does not verify the outcome of the write call (which would be “successful” because the read-only bdb class return true for all write operations). But.. for the sake of correctness I see two possible paths to follow:

    1. As this write is done exclusively on wallets with no checksum on crypted keys, we could move the upgrade code to the loading code in walletdb.cpp; removing fDecryptionThoroughlyChecked entirely and requiring such ancient wallets to provide the passphrase during load to perform the upgrade (adding a “passphrase” arg to the loadwallet RPC command).

    2. Introduce a new WalletBatch::IsReadOnly() method to avoid calling the write inside CheckDecryptionKey.

    The first approach will make the legacy wallet code simpler while the second one is a minimal change.

  125. achow101 force-pushed on Jun 24, 2024
  126. achow101 force-pushed on Jun 24, 2024
  127. achow101 commented at 4:47 pm on June 24, 2024: member

    I believe this PR could get merged quite fast if you leave 8950371 for a follow-up and make IsMine compatible with LegacyDataSPKM in this PR.

    Moved to #30328

    The IsMine removal is also nice but I wouldn’t miss a deadline for it.

    I think it needs to be in the next release as it is a critical part of migration. I would prefer to have it in a release that people can use before deleting the legacy wallet.

    Updated

    • in c653f4f: typo in commit subject (s/MigrateToLegacy/MigrateToDescriptor/)

    Fixed

    No, we should not be migrating a wallet if we cannot decrypt the keys. BerkeleyRODatabase always returns true for all write operations so it doesn’t matter. Also, I don’t think it is useful to invert that and have to have specific bypasses for BerkeleyRODatabase in all functions that attempts to write to it.

  128. furszy commented at 1:35 pm on June 26, 2024: member
    Strange CI https://github.com/bitcoin/bitcoin/actions/runs/9649050504/job/26611651920?pr=26596 error.. it doesn’t seem to be related. The “salvage” command opens the wallet in DatabaseFormat::BERKELEY format.
  129. in src/wallet/wallet.cpp:4465 in f732495191 outdated
    4460@@ -4438,6 +4461,8 @@ util::Result<MigrationResult> MigrateLegacyToDescriptor(const std::string& walle
    4461     // both before and after reloading. This ensures the set is complete even if one of the wallets
    4462     // fails to reload.
    4463     std::set<fs::path> wallet_dirs;
    4464+    // Reset options.require_format as options are used by reload_wallet which may reload wallets of any format.
    4465+    options.require_format = std::nullopt;
    


    furszy commented at 7:49 pm on June 26, 2024:

    In f732495191d346b8:

    The reset call needs to be moved above the reload_wallet function declaration. Otherwise, the wallet will be reloaded in read-only mode for any failure occurring in-between the reload_wallet declaration and this line, such as failures during backup creation or the decryption procedure.

    I have written a test to trigger this misbehavior: furszy@c24b673 (feel free to pull it here). The test passes when the fix is cherry-picked on top: furszy@2d5d9a4.


    achow101 commented at 8:39 pm on June 26, 2024:
    Fixed and pulled the test. I decided to just move this into reload_wallet directly, it doesn’t need to live outside of it.
  130. Change MigrateLegacyToDescriptor to reopen wallet as BERKELEY_RO
    When we reopen the wallet to do the migration, instead of opening using
    BDB, open it using the BerkeleyRO implementation.
    517e204bac
  131. achow101 force-pushed on Jun 26, 2024
  132. in src/wallet/scriptpubkeyman.cpp:813 in 5fee29f541 outdated
    810+    LegacyDataSPKM::LoadScriptMetadata(script_id, meta);
    811     UpdateTimeFirstKey(meta.nCreateTime);
    812-    m_script_metadata[script_id] = meta;
    813+}
    814+
    815+bool LegacyDataSPKM::AddKeyPubKeyInner(const CKey& key, const CPubKey &pubkey)
    


    furszy commented at 10:09 pm on June 26, 2024:

    nano nit:

    0bool LegacyDataSPKM::AddKeyPubKeyInner(const CKey& key, const CPubKey& pubkey)
    

    cbergqvist commented at 10:16 pm on June 27, 2024:

    (In response to #26596 (review))

    This made me cringe when first looking at the code too, but I guess retaining the inconsistency makes it easier to search for AddKeyPubKeyInner(const CKey& key, const CPubKey &pubkey) and find all 4 places it exists. Changing it in this PR would arguably add noise (on the other hand - if not now, then when?).


    achow101 commented at 6:26 pm on July 1, 2024:
    Done

    cbergqvist commented at 9:38 pm on July 1, 2024:

    (Now grepping “AddKeyPubKeyInner(const CKey& key, const CPubKey& pubkey)” gives 1 result. grepping “AddKeyPubKeyInner(const CKey& key, const CPubKey &pubkey)” gives 3 results.

    It would be nice to align style within the same class between .CPP/.H).


    achow101 commented at 9:56 pm on July 1, 2024:
    Not going to change existing code, especially code that will be deleted, to match style

    cbergqvist commented at 10:01 pm on July 1, 2024:
    (Agreed, but then I would skip doing the suggested change in the .CPP file).
  133. in src/wallet/scriptpubkeyman.cpp:818 in 5fee29f541 outdated
    815+bool LegacyDataSPKM::AddKeyPubKeyInner(const CKey& key, const CPubKey &pubkey)
    816+{
    817+    // This function should only be called during loading of a legacy to be migrated.
    818+    // As such, the wallet should not be encrypted if this is called.
    819+    LOCK(cs_KeyStore);
    820+    assert(!m_storage.HasEncryptionKeys());
    


    furszy commented at 10:12 pm on June 26, 2024:
    I’m not sure this is asserting what you expect. In the loading process, we load the encryption keys after the key records. So the encryption keys map should always be empty and HasEncryptionKeys should always return false but not for the reasons stated here.

    achow101 commented at 11:30 pm on June 26, 2024:
    Yes, that’s what the comment is saying. It is because this function is only called during loading, and because legacy wallet records are loaded first, that this assertion makes sense. After all, to be migrated legacy wallets can still be encrypted.

    furszy commented at 8:11 pm on June 27, 2024:

    I think calling IsLocked() would be more precise and make more sense then. It seems confusing to say that the wallet can still be encrypted but expect no encryption keys (we understand it because we know the loading procedure but others might not understand it).

    Other than that, what if instead of creating this new LegacyDataSPKM::AddKeyPubKeyInner method you place this three lines inside the LegacyDataSPKM::LoadKey which is only called during load?


    achow101 commented at 6:23 pm on July 1, 2024:

    Other than that, what if instead of creating this new LegacyDataSPKM::AddKeyPubKeyInner method you place this three lines inside the LegacyDataSPKM::LoadKey which is only called during load?

    We can’t do that because LoadKey needs to call LegacyScriptPubKeyMan::AddKeyPubKeyInner for legacy wallets that are being loaded normally. Since migration cannot do all of the things in LegacyScriptPubKeyMan::AddKeyPubKeyInner, we need to have LegacyDataSPKM::AddKeyPubKeyInner.

    I think calling IsLocked() would be more precise and make more sense then. It seems confusing to say that the wallet can still be encrypted but expect no encryption keys (we understand it because we know the loading procedure but others might not understand it).

    Actually, looking at this more closely, your original comment was correct. This function is only reached during legacy migration wallet migration loading for unencrypted keys only. Since the encryption keys are loaded later, this check doesn’t actually do anything useful.


    achow101 commented at 6:26 pm on July 1, 2024:
    Removed the assert and the comment.
  134. DrahtBot added the label CI failed on Jun 27, 2024
  135. furszy commented at 8:33 pm on June 27, 2024: member

    Now that the IsMine changes were moved to another PR, could reorder the commits just so 0cce574 is introduced before c3373a39a1acc94dbf644103fab2e477d49da671. This will let us keep the IsMine assertions inside MigrateToDescriptor for now.

    Other than that, code review ACK 9700c21.

  136. DrahtBot requested review from theStack on Jun 27, 2024
  137. DrahtBot requested review from cbergqvist on Jun 27, 2024
  138. cbergqvist commented at 10:27 pm on June 27, 2024: contributor

    git range-diff master a8f8a2d 9700c217

    Agree with @furszy on reordering the commits to make it possible to retain the 3 asserts in MigrateToDescriptor().

  139. DrahtBot requested review from cbergqvist on Jun 27, 2024
  140. wallet: Move LegacySPKM data storage and handling to LegacyDataSPKM
    In order to load the necessary data for migrating a legacy wallet
    without the full LegacyScriptPubKeyMan, move the data storage and
    loading components to LegacyDataSPKM. LegacyScriptPubKeyMan now
    subclasses that.
    7461d0c006
  141. wallet: Move LegacyScriptPubKeyMan::IsMine to LegacyDataSPKM
    IsMine is necessary for migration. It should be inlined with migration
    when the legacy wallet is removed.
    b231f4d556
  142. wallet: Move MigrateToDescriptor and DeleteRecords to LegacyDataSPKM 61d872f1b3
  143. wallet: Use LegacyDataSPKM when loading
    In SetupLegacyScriptPubKeyMan, a base LegacyDataSPKM will be created if
    the database has the format "bdb_ro" (i.e. the wallet was opened only
    for migration purposes).
    
    All of the loading functions are now called with a LegacyDataSPKM object
    instead of LegacyScriptPubKeyMan.
    771bc60f13
  144. test: verify wallet is still active post-migration failure
    The migration process reloads the wallet after all failures.
    This commit tests the behavior by trying to obtain a new address
    after a decryption failure during migration.
    8ce3739edb
  145. achow101 force-pushed on Jul 1, 2024
  146. achow101 commented at 6:26 pm on July 1, 2024: member
    Reordered the commits and preserved the IsMine asserts.
  147. cbergqvist approved
  148. cbergqvist commented at 10:03 pm on July 1, 2024: contributor

    ACK 8ce3739edbcf6437bf2695087e0ebe8c633df19b

    git range-diff 9700c217~5..9700c217 8ce3739~5..8ce3739 - not the easiest read, also used GH compare.

    Passed make check & test/functional/test_runner.py including p2p_handshake.py which failed on CI.

    Only one non-critical reservation: #26596 (review)

  149. DrahtBot requested review from furszy on Jul 1, 2024
  150. DrahtBot removed the label CI failed on Jul 1, 2024
  151. in src/wallet/wallet.cpp:3622 in 771bc60f13 outdated
    3617+        return nullptr;
    3618+    }
    3619+    auto it = m_internal_spk_managers.find(OutputType::LEGACY);
    3620+    if (it == m_internal_spk_managers.end()) return nullptr;
    3621+    return dynamic_cast<LegacyDataSPKM*>(it->second);
    3622+}
    


    theStack commented at 3:13 pm on July 3, 2024:
    nit: was about to suggest to deduplicate shared code between this method and ::GetLegacyScriptPubKeyMan above, but since the latter is removed soon anyway (in PR #28710, commit https://github.com/bitcoin/bitcoin/pull/28710/commits/5652a9c59f9ec793886eff00126c382003e4ce02#diff-1f2db0e4d5c12d109c7f0962333c245b49b696cb39ff432da048e9d6c08944d8), it’s probably not worth it.

    furszy commented at 7:51 pm on July 3, 2024:

    If you re-touch it again. Could:

    0LegacyDataSPKM* CWallet::GetLegacyDataSPKM() const
    1{
    2    if (IsWalletFlagSet(WALLET_FLAG_DESCRIPTORS)) { return nullptr; }
    3    return dynamic_cast<LegacyDataSPKM*>(GetScriptPubKeyMan(OutputType::LEGACY, /*internal=*/true));
    4}
    
  152. theStack approved
  153. theStack commented at 3:14 pm on July 3, 2024: contributor
    Code-review ACK 8ce3739edbcf6437bf2695087e0ebe8c633df19b
  154. in src/wallet/scriptpubkeyman.cpp:798 in 7461d0c006 outdated
    794 {
    795     LOCK(cs_KeyStore);
    796+    LegacyDataSPKM::LoadKeyMetadata(keyID, meta);
    797     UpdateTimeFirstKey(meta.nCreateTime);
    798-    mapKeyMetadata[keyID] = meta;
    799+}
    


    furszy commented at 8:00 pm on July 3, 2024:

    In 7461d0c006c92:

    At this point, just a non-blocking nit but you are locking cs_KeyStore twice. One here, and another one inside LegacyDataSPKM::LoadKeyMetadata.

    Same for LoadScriptMetadata.

  155. in src/wallet/scriptpubkeyman.cpp:816 in 7461d0c006 outdated
    813+}
    814+
    815+bool LegacyDataSPKM::AddKeyPubKeyInner(const CKey& key, const CPubKey& pubkey)
    816+{
    817+    LOCK(cs_KeyStore);
    818+    return FillableSigningProvider::AddKeyPubKey(key, pubkey);
    


    furszy commented at 8:02 pm on July 3, 2024:

    In https://github.com/bitcoin/bitcoin/commit/7461d0c006c92ede2f2595b79a5509eaf3509fb7:

    FillableSigningProvider::AddKeyPubKey already locks cs_Keystore internally.

  156. furszy commented at 8:10 pm on July 3, 2024: member
    Code review ACK 8ce3739edbcf6437bf2695087e0ebe8c633df19b
  157. glozow merged this on Jul 11, 2024
  158. glozow closed this on Jul 11, 2024


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-11-17 15:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me