wallet: Load database records in a particular order #24914

pull achow101 wants to merge 16 commits into bitcoin:master from achow101:wallet-load-order changing 5 files +809 −568
  1. achow101 commented at 9:18 pm on April 18, 2022: member

    Currently when we load a wallet, we just iterate through all of the records in the database and add them completely statelessly. However we have some records which do rely on other records being loaded before they are. To deal with this, we use CWalletScanState to hold things temporarily until all of the records have been read and then we load the stateful things.

    However this can be slow, and with some future improvements, can cause some pretty drastic slowdowns to retain this pattern. So this PR changes the way we load records by choosing to load the records in a particular order. This lets us do things such as loading a descriptor record, then finding and loading that descriptor’s cache and key records. In the future, this will also let us use IsMine when loading transactions as then IsMine will actually be working as we now always load keys and descriptors before transactions.

    In order to get records of a specific type, this PR includes some refactors to how we do database cursors. Functionality is also added to retrieve a cursor that will give us records beginning with a specified prefix.

    Lastly, one thing that iterating the entire database let us do was to find unknown records. However even if unknown records were found, we would not do anything with this information except output a number in a log line. With this PR, we would no longer be aware of any unknown records. This does not change functionality as we don’t do anything with unknown records, and having unknown records is not an error. Now we would just not be aware that unknown records even exist.

  2. achow101 added the label Wallet on Apr 18, 2022
  3. DrahtBot commented at 1:59 am on April 19, 2022: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK MarcoFalke, furszy, ryanofsky
    Concept ACK theStack, pk-b2, laanwj
    Approach ACK w0xlt

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #27920 (wallet: bugfix, always use apostrophe for spkm descriptor ID by furszy)
    • #27865 (wallet: Track no-longer-spendable TXOs separately by achow101)
    • #27286 (wallet: Keep track of the wallet’s own transaction outputs in memory by achow101)
    • #26836 (wallet: simplify addressbook migration and encapsulate access by furszy)
    • #26728 (wallet: Have the wallet store the key for automatically generated descriptors by achow101)
    • #26596 (wallet: Migrate legacy wallets to descriptor wallets without requiring BDB by achow101)
    • #25907 (wallet: rpc to add automatically generated descriptors by achow101)
    • #22341 (rpc: add getxpub by Sjors)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  4. achow101 force-pushed on Apr 20, 2022
  5. DrahtBot added the label Needs rebase on Apr 26, 2022
  6. achow101 force-pushed on Apr 26, 2022
  7. DrahtBot removed the label Needs rebase on Apr 26, 2022
  8. theStack commented at 8:13 am on May 2, 2022: contributor
    Concept ACK
  9. w0xlt commented at 11:15 am on May 2, 2022: contributor
    Approach ACK. Getting rid of CWalletScanState and ReadKeyValue and moving the read logic to WalletBatch::LoadWallet makes the code clearer and easier to understand.
  10. pk-b2 commented at 8:52 pm on May 6, 2022: none
    Concept ACK
  11. in src/wallet/sqlite.cpp:552 in 3a27791244 outdated
    547+    }
    548+
    549+    std::unique_ptr<SQLiteCursor> cursor = std::make_unique<SQLiteCursor>(start_range, end_range);
    550+    if (!cursor) return nullptr;
    551+
    552+    const char* stmt_text = "SELECT key, value FROM main WHERE key >= ? AND key < ?";
    


    pk-b2 commented at 9:00 pm on May 6, 2022:
    0> explain query plan SELECT key, value FROM main WHERE key >= ? AND key < ?
    

    Verified that this uses an index avoiding a table scan

    0SEARCH TABLE main USING INDEX sqlite_autoindex_main_1 (key>? AND key<?)`
    
  12. in src/wallet/sqlite.cpp:586 in 3a27791244 outdated
    542+        *it = std::byte(std::to_integer<unsigned char>(*it) + 1);
    543+        break;
    544+    }
    545+    if (it == end_range.rend()) {
    546+        end_range.insert(end_range.begin(), std::byte(0x01));
    547+    }
    


    pk-b2 commented at 9:02 pm on May 6, 2022:
    👍 Like that approach
  13. DrahtBot added the label Needs rebase on May 12, 2022
  14. achow101 force-pushed on May 16, 2022
  15. achow101 force-pushed on May 16, 2022
  16. DrahtBot removed the label Needs rebase on May 16, 2022
  17. achow101 force-pushed on May 16, 2022
  18. achow101 force-pushed on May 17, 2022
  19. DrahtBot added the label Needs rebase on May 18, 2022
  20. achow101 force-pushed on May 18, 2022
  21. DrahtBot removed the label Needs rebase on May 18, 2022
  22. laanwj commented at 4:51 pm on May 31, 2022: member
    Concept ACK on querying specifically what is needed from the database when loading. Maybe some day we don’t even have to read all records into memory.
  23. in src/wallet/bdb.cpp:729 in 950c94f5cc outdated
    704 
    705+std::unique_ptr<DatabaseCursor> BerkeleyBatch::GetCursor()
    706+{
    707+    if (!pdb) return nullptr;
    708+    return std::make_unique<BerkeleyCursor>(m_database);
    709+}
    


    furszy commented at 4:34 pm on June 9, 2022:
    tiny nit: maybe GetNewCursor()?

    achow101 commented at 7:47 pm on June 21, 2022:
    Done. Also renamed GetPrefixCursor to GetNewPrefixCursor.
  24. in src/wallet/bdb.cpp:663 in 950c94f5cc outdated
    669-        return false;
    670-    int ret = pdb->cursor(nullptr, &m_cursor, 0);
    671-    return ret == 0;
    672+    assert(database.m_db.get());
    673+    int ret = database.m_db->cursor(nullptr, &m_cursor, 0);
    674+    assert(ret == 0);
    


    furszy commented at 4:43 pm on June 9, 2022:

    in 950c94f5:

    Probably better an exception instead of an assertion (similar to what is being done in the SQLite method).


    achow101 commented at 7:47 pm on June 21, 2022:
    Done

    theStack commented at 7:36 pm on December 11, 2022:
    Seems like the assertion is still there? Considering the documentation saying “The DB->cursor function may fail and return a non-zero error for errors specified for other Berkeley DB and C library or system functions.” (http://web.mit.edu/ghudson/dev/nokrb/third/db/docs/api_c/db_cursor.html), I’d also agree that throwing an exception would be preferred.
  25. in src/wallet/walletdb.cpp:780 in 040fa33fcc outdated
    775+    AssertLockHeld(pwallet->cs_wallet);
    776+    uint64_t flags;
    777+    if (batch.Read(DBKeys::FLAGS, flags)) {
    778+        if (!pwallet->LoadWalletFlags(flags)) {
    779+            pwallet->WalletLogPrintf("Error reading wallet database: Unknown non-tolerable wallet flags found\n");
    780+            return DBErrors::TOO_NEW;
    


    furszy commented at 5:33 pm on June 9, 2022:

    In 040fa33f:

    Bad copy here, error is DBErrors::CORRUPT.


    achow101 commented at 7:22 pm on June 17, 2022:
    The original error is incorrect, it’s supposed to be TOO_NEW.

    ryanofsky commented at 1:10 pm on May 19, 2023:

    In commit “walletdb: Refactor wallet flags loading” (c825d7c85b0353bdfdf9968272bf5b6cc78c9f12)

    The original error is incorrect, it’s supposed to be TOO_NEW.

    Would be good to note that this change is intentional in the commit message


    achow101 commented at 4:56 pm on May 22, 2023:
    Added a line in the commit message.
  26. furszy commented at 5:52 pm on June 9, 2022: member
    Concept ACK, code reviewed till e2ab0797 (inclusive).
  27. in src/wallet/walletdb.cpp:516 in a1d8d563d0 outdated
    776+        std::string type;
    777+        ssKey >> type;
    778+        assert(type == DBKeys::KEY);
    779+        if (!LoadKey(pwallet, ssKey, ssValue, err)) {
    780+            result = DBErrors::CORRUPT;
    781+            pwallet->WalletLogPrintf("%s\n", err);
    


    furszy commented at 3:31 pm on June 21, 2022:

    In a1d8d563:

    Shouldn’t this be a direct return DBErrors::CORRUPT as it was before? (same for the other one at line 807)


    achow101 commented at 7:38 pm on June 21, 2022:
    No, If you trace the code, it doesn’t actually immediately return DBErrors::CORRPUT. Instead it does result = DBErrors::CORRUPT, continues with reading the rest of the records, and returns result towards the end.
  28. in src/wallet/walletdb.cpp:501 in a1d8d563d0 outdated
    761+    if (!cursor) {
    762+        pwallet->WalletLogPrintf("Error getting database cursor for unencrypted keys\n");
    763+        return DBErrors::CORRUPT;
    764+    }
    765+
    766+    int num_keys = 0;
    


    furszy commented at 3:38 pm on June 21, 2022:
    num_keys isn’t being increased.

    achow101 commented at 7:47 pm on June 21, 2022:
    Done
  29. in src/wallet/walletdb.cpp:529 in a1d8d563d0 outdated
    788+    if (!cursor) {
    789+        pwallet->WalletLogPrintf("Error getting database cursor for crypted keys\n");
    790+        return DBErrors::CORRUPT;
    791+    }
    792+
    793+    int num_ckeys = 0;
    


    furszy commented at 3:38 pm on June 21, 2022:
    num_ckeys isn’t being increased.

    achow101 commented at 7:47 pm on June 21, 2022:
    Done
  30. furszy commented at 4:01 pm on June 21, 2022: member

    Code reviewed till a1d8d563 (non-inclusive).

    What about abstracting the cursor creation, looped read and error handling into a generic function so we can get rid-off most of the boilerplate code?

    Quick example that could be extended to most of the flows: https://github.com/furszy/bitcoin-core/commit/8526a87cae04a3704d86a5157894d42ecaca2ab9

  31. achow101 force-pushed on Jun 21, 2022
  32. achow101 commented at 7:47 pm on June 21, 2022: member

    What about abstracting the cursor creation, looped read and error handling into a generic function so we can get rid-off most of the boilerplate code?

    Quick example that could be extended to most of the flows: furszy@8526a87

    Looking into it.

  33. achow101 force-pushed on Jun 22, 2022
  34. achow101 commented at 0:49 am on June 22, 2022: member
    I’ve refactored the loading loop boilerplate into its own function.
  35. maflcko commented at 5:30 am on June 22, 2022: member
    0wallet/walletdb.cpp:912:18: error: reading variable 'm_address_book' requires holding mutex 'pwallet->cs_wallet' [-Werror,-Wthread-safety-analysis]
    1        pwallet->m_address_book[DecodeDestination(strAddress)].SetLabel(label);
    
  36. DrahtBot added the label Needs rebase on Jun 30, 2022
  37. achow101 force-pushed on Jun 30, 2022
  38. DrahtBot removed the label Needs rebase on Jun 30, 2022
  39. furszy commented at 3:06 am on July 6, 2022: member
    Patch for the ci error “requires holding mutex ‘pwallet->cs_wallet’”: https://github.com/furszy/bitcoin/commit/b3c51af13a04eaf3a374f778dd0858ad128aa512
  40. achow101 force-pushed on Jul 6, 2022
  41. achow101 force-pushed on Jul 6, 2022
  42. in src/wallet/sqlite.cpp:483 in d1316204a7 outdated
    490         complete = true;
    491         return true;
    492     }
    493     if (res != SQLITE_ROW) {
    494-        LogPrintf("SQLiteBatch::ReadAtCursor: Unable to execute cursor step: %s\n", sqlite3_errstr(res));
    495+        LogPrintf("SQLiteCursor::Next: Unable to execute cursor step: %s\n", sqlite3_errstr(res));
    


    furszy commented at 7:51 pm on July 6, 2022:

    tiny nit:

    0        LogPrintf("%s: Unable to execute cursor step: %s\n", __func__, sqlite3_errstr(res));
    

    achow101 commented at 5:52 pm on July 7, 2022:
    Done
  43. in src/wallet/sqlite.cpp:503 in d1316204a7 outdated
    500     sqlite3_reset(m_cursor_stmt);
    501-    m_cursor_init = false;
    502+    int res = sqlite3_finalize(m_cursor_stmt);
    503+    if (res != SQLITE_OK) {
    504+        LogPrintf("SQLiteBatch: cursor closed but could not finalize cursor statement: %s\n",
    505+                  sqlite3_errstr(res));
    


    furszy commented at 7:52 pm on July 6, 2022:

    Should log SQLiteCursor now, not SQLiteBatch.

    0        LogPrintf("%s: cursor closed but could not finalize cursor statement: %s\n", __func__, sqlite3_errstr(res));
    

    achow101 commented at 5:52 pm on July 7, 2022:
    Done
  44. in src/wallet/bdb.cpp:720 in d1316204a7 outdated
    687@@ -690,13 +688,19 @@ bool BerkeleyBatch::ReadAtCursor(CDataStream& ssKey, CDataStream& ssValue, bool&
    688     return true;
    689 }
    690 
    691-void BerkeleyBatch::CloseCursor()
    692+BerkeleyCursor::~BerkeleyCursor()
    693 {
    694     if (!m_cursor) return;
    


    furszy commented at 7:55 pm on July 6, 2022:
    could delete this line, m_cursor will never be null.

    achow101 commented at 5:34 pm on July 7, 2022:
    Leaving it as belt-and-suspenders in case something happens to it in the future.
  45. in src/wallet/sqlite.h:22 in d1316204a7 outdated
    13@@ -14,19 +14,27 @@ struct bilingual_str;
    14 namespace wallet {
    15 class SQLiteDatabase;
    16 
    17+class SQLiteCursor : public DatabaseCursor
    18+{
    19+public:
    20+    sqlite3_stmt* m_cursor_stmt{nullptr};
    


    furszy commented at 8:00 pm on July 6, 2022:
    nit: what about making the cursor private and passing a sqlite3_stmt pointer to the constructor?

    achow101 commented at 5:30 pm on July 7, 2022:
    I tried doing that but kept running into issues with allocating the memory for it as the underlying object would have to survive the scope in which it was originally created. It was just easier to do it in the struct directly.
  46. in src/wallet/sqlite.cpp:511 in d1316204a7 outdated
    508+
    509+std::unique_ptr<DatabaseCursor> SQLiteBatch::GetNewCursor()
    510+{
    511+    if (!m_database.m_db) return nullptr;
    512+    std::unique_ptr<SQLiteCursor> cursor = std::make_unique<SQLiteCursor>();
    513+    if (!cursor) return nullptr;
    


    furszy commented at 8:04 pm on July 6, 2022:
    nit: this line can be removed.

    achow101 commented at 5:52 pm on July 7, 2022:
    Done
  47. in src/wallet/sqlite.cpp:596 in d1316204a7 outdated
    514+
    515+    const char* stmt_text = "SELECT key, value FROM main";
    516+    int res = sqlite3_prepare_v2(m_database.m_db, stmt_text, -1, &cursor->m_cursor_stmt, nullptr);
    517+    if (res != SQLITE_OK) {
    518+        throw std::runtime_error(strprintf(
    519+            "SQLiteDatabase: Failed to setup cursor SQL statement: %s\n", sqlite3_errstr(res)));
    


    furszy commented at 8:09 pm on July 6, 2022:

    nit:

    0             "%s: Failed to setup cursor SQL statement: %s\n", __func__ sqlite3_errstr(res)));
    

    achow101 commented at 5:52 pm on July 7, 2022:
    Done

    aureleoules commented at 4:17 pm on September 26, 2022:
    It’s still there :grimacing:
  48. in src/wallet/bdb.cpp:659 in 7208976504 outdated
    654@@ -655,7 +655,8 @@ void BerkeleyDatabase::ReloadDbEnv()
    655     env->ReloadDbEnv();
    656 }
    657 
    658-BerkeleyCursor::BerkeleyCursor(BerkeleyDatabase& database)
    659+BerkeleyCursor::BerkeleyCursor(BerkeleyDatabase& database, Span<std::byte> prefix)
    660+    : m_key_prefix(prefix.begin(), prefix.end()), m_first(true)
    


    furszy commented at 12:54 pm on July 7, 2022:
    nit: could default initialize m_first in the field declaration.

    achow101 commented at 5:52 pm on July 7, 2022:
    Done
  49. in src/wallet/bdb.cpp:698 in 7208976504 outdated
    691@@ -685,6 +692,11 @@ bool BerkeleyCursor::Next(CDataStream& ssKey, CDataStream& ssValue, bool& comple
    692     ssValue.SetType(SER_DISK);
    693     ssValue.clear();
    694     ssValue.write({AsBytePtr(datValue.get_data()), datValue.get_size()});
    695+
    696+    if (!m_key_prefix.empty() && std::mismatch(ssKey.begin(), ssKey.end(), m_key_prefix.begin(), m_key_prefix.end()).second != m_key_prefix.end()) {
    697+        complete = true;
    698+    }
    


    furszy commented at 3:17 pm on July 7, 2022:

    In 72089765:

    With this code, at the end of the range, Next returns one extra element pair that is not part of the range set. So, if the complete flag is not checked by the caller first and the ssValue tries to be unserialized, a runtime error will be thrown (different value type).

    Would suggest to move this block above the ssKey and ssValue return values sets; so the function doesn’t return elements that aren’t in the filtered range, and directly return false:

     0Span<const std::byte> raw_key = {AsBytePtr(datKey.get_data()), datKey.get_size()};
     1if (!m_key_prefix.empty() && std::mismatch(raw_key.begin(), raw_key.end(), m_key_prefix.begin(), m_key_prefix.end()).second != m_key_prefix.end()) {
     2    complete = true;
     3    return false;
     4}
     5
     6ssKey.SetType(SER_DISK);
     7ssKey.clear();
     8ssKey.write(raw_key);
     9// bla bla, all good..
    10return true;
    

    Commit here: https://github.com/furszy/bitcoin/commit/e4b37e82ed19ff103f295ac5897e33e80e2f0679


    achow101 commented at 5:52 pm on July 7, 2022:
    Done
  50. furszy commented at 4:47 pm on July 7, 2022: member

    Did another review round till 72089765 this time.

    Found that the new prefix cursor is returning an extra element that is not part of the filtered set (fix https://github.com/furszy/bitcoin/commit/e4b37e82ed19ff103f295ac5897e33e80e2f0679), which.. encouraged me to add test coverage for the new functionality: https://github.com/furszy/bitcoin/commit/0f76c4c5e989fc9eef82c46386311a7069c25bdc

    (note: without the fix, at the end of the test, ssValue contains a type that is not the expected one, which if it’s unserialized will crash or UB depending to which type the code tries to unserialize it).

    Tested on each supported db engine:

    1. Write two different key->value elements to db.
    2. Create a new prefix cursor and walk-through every returned element, verifying that gets parsed properly.
    3. Try to move the cursor outside the filtered range: expect failure and flag complete=true.

    Plus, the new test made me found one extra point: sqlite does not clear the return key and value references before write the new data (which causes the new data to be appended to the previous one if it’s not cleared by the caller). Solved here: https://github.com/furszy/bitcoin/commit/558fcb0c56cb0b50a8dd2c76ed216bb0ed2af58a.

  51. achow101 force-pushed on Jul 7, 2022
  52. achow101 commented at 5:54 pm on July 7, 2022: member

    @furszy Thanks for the review. I’ve pulled in your test commit.

    On the topic of the return values for Next, I’m not a fan of the current interface and I think it could be improved/rewritten with optionals. Mainly the problem is distinguishing between an error and being done.

  53. furszy commented at 12:38 pm on July 8, 2022: member

    On the topic of the return values for Next, I’m not a fan of the current interface and I think it could be improved/rewritten with optionals. Mainly the problem is distinguishing between an error and being done.

    Probably we could return a three states enum instead. So we can get rid of that ugly complete ref arg. (but yeah, no need to add it here)

  54. in src/wallet/bdb.cpp:697 in c704f83488 outdated
    692+    }
    693+
    694     // Convert to streams
    695     ssKey.SetType(SER_DISK);
    696     ssKey.clear();
    697     ssKey.write({AsBytePtr(datKey.get_data()), datKey.get_size()});
    


    furszy commented at 1:33 pm on July 15, 2022:

    nit:

    0    ssKey.write(raw_key);
    

    achow101 commented at 11:12 pm on July 15, 2022:
    Done
  55. in src/wallet/bdb.h:190 in c704f83488 outdated
    186@@ -187,11 +187,16 @@ class SafeDbt final
    187 
    188 class BerkeleyCursor : public DatabaseCursor
    189 {
    190-private:
    191+protected:
    


    furszy commented at 1:35 pm on July 15, 2022:
    this change shouldn’t be needed.

    achow101 commented at 11:12 pm on July 15, 2022:
    Done
  56. in src/wallet/walletdb.cpp:759 in ba1d48a293 outdated
    754+        pwallet->WalletLogPrintf("Error getting database cursor for '%s' records\n", key);
    755+        result.m_result = DBErrors::CORRUPT;
    756+        return result;
    757+    }
    758+
    759+    result.m_result = DBErrors::LOAD_OK;
    


    furszy commented at 5:49 pm on July 15, 2022:
    nit: unnecessary set. m_result member is default initialized to LOAD_OK.

    achow101 commented at 11:12 pm on July 15, 2022:
    Done
  57. in src/wallet/walletdb.cpp:821 in ba1d48a293 outdated
    816+        key >> hash;
    817+        CScript script;
    818+        value >> script;
    819+        if (!pwallet->GetOrCreateLegacyScriptPubKeyMan()->LoadCScript(script))
    820+        {
    821+            pwallet->WalletLogPrintf("Error reading wallet database: LegacyScriptPubKeyMan::LoadCScript failed\n");
    


    furszy commented at 5:59 pm on July 15, 2022:

    LoadRecords now is logging this:

    0            err = "Error reading wallet database: LegacyScriptPubKeyMan::LoadCScript failed\n";
    

    achow101 commented at 11:13 pm on July 15, 2022:
    Done here and elsewhere.
  58. in src/wallet/walletdb.cpp:567 in ba1d48a293 outdated
    795+        if (!LoadKey(pwallet, key, value, err)) {
    796+            return DBErrors::CORRUPT;
    797+        }
    798+        return DBErrors::LOAD_OK;
    799+    });
    800+    result = std::max(result, key_res.m_result);
    


    furszy commented at 6:01 pm on July 15, 2022:

    Why not return early here?:

    0if (key_res.m_result != DBErrors::LOAD_OK) {
    1    return key_res.m_result;
    2}
    

    (Same for the crypted keys)


    achow101 commented at 8:22 pm on July 15, 2022:
    It is the retain the current behavior where we generally go through all the records in the wallet regardless of whether they are corrupt, and just report the worst result at the end. This allows us to check all of the records for corruption.
  59. in src/wallet/walletdb.cpp:982 in 369a042c73 outdated
    977+            key >> desc_id;
    978+            assert(desc_id == id);
    979+            key >> pubkey;
    980+            if (!pubkey.IsValid())
    981+            {
    982+                pwallet->WalletLogPrintf("Error reading wallet database: descriptor unencrypted key CPubKey corrupt\n");
    


    furszy commented at 6:46 pm on July 15, 2022:

    LoadRecords logs the corrupt errors.

    0                err = "Error reading wallet database: descriptor unencrypted key CPubKey corrupt\n";
    

    (Same for the other ones below)

  60. in src/wallet/walletdb.cpp:1034 in 369a042c73 outdated
    1029+            }
    1030+            std::vector<unsigned char> privkey;
    1031+            value >> privkey;
    1032+            num_ckeys++;
    1033+
    1034+            descriptor_crypt_keys.insert(std::make_pair(pubkey.GetID(), std::make_pair(pubkey, privkey)));
    


    furszy commented at 6:54 pm on July 15, 2022:

    Same as above here:

    0spk_man->AddCryptedKey(pubkey.GetID(), pubkey, privkey);
    

    achow101 commented at 11:13 pm on July 15, 2022:
    Done
  61. in src/wallet/walletdb.cpp:997 in 369a042c73 outdated
    1040+        for (auto desc_key_pair : descriptor_keys) {
    1041+            spk_man->AddKey(desc_key_pair.first, desc_key_pair.second);
    1042+        }
    1043+        for (auto desc_key_pair : descriptor_crypt_keys) {
    1044+            spk_man->AddCryptedKey(desc_key_pair.first, desc_key_pair.second.first, desc_key_pair.second.second);
    1045+        }
    


    furszy commented at 6:56 pm on July 15, 2022:
    could remove this two loops, the map and vector need If the keys are added to the spkm inside each LoadRecords (check above, left a comment there).

    achow101 commented at 11:13 pm on July 15, 2022:
    Done
  62. furszy commented at 7:19 pm on July 15, 2022: member

    Made another review round from scratch, code reviewed till 369a042c. Getting closer 🚜

    Found few bugs (same cause for all of them) inside 369a042c:

    The LoadRecords of the WALLETDESCRIPTOR prefix is always returning LOAD_OK result even if the load internally failed (corrupted db).

    The reason is the not contemplation of the result on each inner LoadRecords calls like WALLETDESCRIPTORCACHE, WALLETDESCRIPTORLHCACHE, WALLETDESCRIPTORKEY and WALLETDESCRIPTORCKEY in the final result (or better, after each of the LoadRecords calls).

    Example: after setting each lh_cache_res, key_cache_res, key_res, ckey_res should do:

    0if (key_res.m_result != DBErrors::LOAD_OK) {
    1    return key_res;
    2}
    3
    4or 
    5
    6result = std::max(result, key_res.m_result); // "key_res" is just the example
    
  63. in src/wallet/walletdb.cpp:1024 in 7ed39f6d34 outdated
    1049+                // There's some corruption here since the tx we just tried to load was already in the wallet.
    1050+                // We don't consider this type of corruption critical, and can fix it by removing tx data and
    1051+                // rescanning.
    1052+                pwallet->WalletLogPrintf("Error: Corrupt transaction found. This can be fixed by removing transactions from wallet and rescanning.\n");
    1053+                result = DBErrors::CORRUPT;
    1054+                return false;
    


    furszy commented at 7:51 pm on July 15, 2022:
    Missing corrupted_tx = true set here.

    achow101 commented at 11:13 pm on July 15, 2022:
    Done
  64. furszy commented at 7:52 pm on July 15, 2022: member

    Another one, in 7ed39f6d:

    The corrupted_tx variable is always false.

  65. achow101 force-pushed on Jul 15, 2022
  66. achow101 commented at 11:14 pm on July 15, 2022: member

    The LoadRecords of the WALLETDESCRIPTOR prefix is always returning LOAD_OK result even if the load internally failed (corrupted db).

    Good catch. I’ve added result = std::max(result, res.m_result); for each of these.

  67. in src/wallet/db.h:109 in 94584601f1 outdated
    107-    virtual bool ReadAtCursor(CDataStream& ssKey, CDataStream& ssValue, bool& complete) = 0;
    108-    virtual void CloseCursor() = 0;
    109+    virtual std::unique_ptr<DatabaseCursor> GetNewCursor() = 0;
    110+    bool StartCursor()
    111+    {
    112+        m_cursor = std::move(GetNewCursor());
    


    theStack commented at 12:42 pm on August 16, 2022:
    0        m_cursor = GetNewCursor();
    

    I think this std::move on a temporary isn’t needed? The compiler says (commit 94584601f12c51b3a781c84536f6b56be1ea8077)

    0In file included from ./wallet/walletdb.h:10:
    1./wallet/db.h:109:20: warning: moving a temporary object prevents copy elision [-Wpessimizing-move]
    2        m_cursor = std::move(GetNewCursor());
    3                   ^
    4./wallet/db.h:109:20: note: remove std::move call here
    5        m_cursor = std::move(GetNewCursor());
    6                   ^~~~~~~~~~              ~
    
  68. achow101 force-pushed on Aug 16, 2022
  69. achow101 force-pushed on Aug 17, 2022
  70. DrahtBot added the label Needs rebase on Aug 19, 2022
  71. achow101 force-pushed on Aug 19, 2022
  72. DrahtBot removed the label Needs rebase on Aug 19, 2022
  73. DrahtBot added the label Needs rebase on Sep 13, 2022
  74. achow101 force-pushed on Sep 19, 2022
  75. DrahtBot removed the label Needs rebase on Sep 19, 2022
  76. achow101 force-pushed on Sep 21, 2022
  77. in src/wallet/sqlite.cpp:514 in b7167d0d65 outdated
    511+}
    512+
    513+std::unique_ptr<DatabaseCursor> SQLiteBatch::GetNewCursor()
    514+{
    515+    if (!m_database.m_db) return nullptr;
    516+    std::unique_ptr<SQLiteCursor> cursor = std::make_unique<SQLiteCursor>();
    


    aureleoules commented at 1:16 pm on September 26, 2022:

    nit in 10cf25c112ff4ac7bcece87a821996004cec2b24, less verbose

    0    auto cursor = std::make_unique<SQLiteCursor>();
    

    achow101 commented at 9:50 pm on October 6, 2022:
    Done
  78. in src/wallet/sqlite.cpp:549 in b7167d0d65 outdated
    546+    }
    547+    if (it == end_range.rend()) {
    548+        end_range.insert(end_range.begin(), std::byte(0x01));
    549+    }
    550+
    551+    std::unique_ptr<SQLiteCursor> cursor = std::make_unique<SQLiteCursor>(start_range, end_range);
    


    aureleoules commented at 1:17 pm on September 26, 2022:

    nit in e9b3bcb411b5742d6579d95b5e6aa2ad3f7592be, less verbose

    0    auto cursor = std::make_unique<SQLiteCursor>(start_range, end_range);
    

    achow101 commented at 9:50 pm on October 6, 2022:
    Done
  79. in src/wallet/sqlite.h:19 in b7167d0d65 outdated
    13@@ -14,19 +14,36 @@ struct bilingual_str;
    14 namespace wallet {
    15 class SQLiteDatabase;
    16 
    17+/** RAII class that provides a database cursor */
    18+class SQLiteCursor : public DatabaseCursor
    


    aureleoules commented at 1:20 pm on September 26, 2022:
    in 10cf25c112ff4ac7bcece87a821996004cec2b24: I know this hasn’t been done in other RAII classes but wouldn’t it be better to delete copy and copy assignment operators to avoid potential double frees?

    achow101 commented at 9:51 pm on October 6, 2022:
    Done in DatabaseCursor. AFAIK that should also delete in the child classes.
  80. in src/wallet/sqlite.h:29 in b7167d0d65 outdated
    23+    // Prevents SQLite from accessing temp variables for the prefix things.
    24+    std::vector<std::byte> m_prefix_range_start;
    25+    std::vector<std::byte> m_prefix_range_end;
    26+
    27+    explicit SQLiteCursor() {}
    28+    explicit SQLiteCursor(std::vector<std::byte> start_range, std::vector<std::byte> end_range)
    


    aureleoules commented at 1:22 pm on September 26, 2022:

    in e9b3bcb411b5742d6579d95b5e6aa2ad3f7592be

    0    explicit SQLiteCursor(const std::vector<std::byte>& start_range, const std::vector<std::byte>& end_range)
    

    achow101 commented at 9:51 pm on October 6, 2022:
    Done
  81. in src/wallet/walletdb.cpp:432 in b7167d0d65 outdated
    555+    return true;
    556+}
    557 
    558-                fSkipCheck = true;
    559-            }
    560+bool LoadHDChain(CWallet* pwallet, CDataStream& ssKey, CDataStream& ssValue, std::string& strErr)
    


    aureleoules commented at 1:23 pm on September 26, 2022:
    in c3181a344c7e80ca233be47e7b44a1074c382498 ssKey and strErr are now unused

    achow101 commented at 9:51 pm on October 6, 2022:
    Removed
  82. in src/wallet/walletdb.cpp:634 in b7167d0d65 outdated
    927-                strErr = "Multiple ScriptPubKeyMans specified for a single type";
    928-                return false;
    929+            // Insert a new CHDChain, or get the one that already exists
    930+            auto ins = hd_chains.emplace(keyMeta.hd_seed_id, CHDChain());
    931+            CHDChain& chain = ins.first->second;
    932+            if (ins.second) {
    


    aureleoules commented at 1:26 pm on September 26, 2022:

    in 5444f78824c8b44776acb3523c8a591d2abb570d

    0            auto [ins, inserted] = hd_chains.emplace(keyMeta.hd_seed_id, CHDChain());
    1            CHDChain& chain = ins->second;
    2            if (inserted) {
    

    achow101 commented at 9:51 pm on October 6, 2022:
    Done
  83. in src/wallet/walletdb.cpp:651 in b7167d0d65 outdated
    957+        return DBErrors::LOAD_OK;
    958+    });
    959+    result = std::max(result, script_res.m_result);
    960+
    961+    // Set inactive chains
    962+    if (hd_chains.size() > 0) {
    


    aureleoules commented at 1:27 pm on September 26, 2022:

    in 5444f78824c8b44776acb3523c8a591d2abb570d

    0    if (!hd_chains.empty()) {
    

    achow101 commented at 9:51 pm on October 6, 2022:
    Done
  84. in src/wallet/walletdb.cpp:659 in b7167d0d65 outdated
    965+            pwallet->WalletLogPrintf("Inactive HD Chains found but no Legacy ScriptPubKeyMan\n");
    966+            return DBErrors::CORRUPT;
    967+        }
    968+        for (const auto& chain_pair : hd_chains) {
    969+            if (chain_pair.first != pwallet->GetLegacyScriptPubKeyMan()->GetHDChain().seed_id) {
    970+                pwallet->GetLegacyScriptPubKeyMan()->AddInactiveHDChain(chain_pair.second);
    


    aureleoules commented at 1:29 pm on September 26, 2022:

    in 5444f78824c8b44776acb3523c8a591d2abb570d

    0        for (const auto& [hd_seed_id, chain] : hd_chains) {
    1            if (hd_seed_id != pwallet->GetLegacyScriptPubKeyMan()->GetHDChain().seed_id) {
    2                pwallet->GetLegacyScriptPubKeyMan()->AddInactiveHDChain(chain);
    

    achow101 commented at 9:52 pm on October 6, 2022:
    Done
  85. in src/wallet/walletdb.cpp:1073 in b7167d0d65 outdated
    1113+        if (!LoadEncryptionKey(pwallet, key, value, err)) {
    1114+            return DBErrors::CORRUPT;
    1115+        }
    1116+        return DBErrors::LOAD_OK;
    1117+    });
    1118+    return mkey_res.m_result;;
    


    aureleoules commented at 1:34 pm on September 26, 2022:

    in 69f987e8ecbf8e8d3202c3ad118501aa1d6399e3: i had to do it

    0    return mkey_res.m_result;
    

    achow101 commented at 9:52 pm on October 6, 2022:
    Done
  86. in src/wallet/sqlite.cpp:586 in b7167d0d65 outdated
    544+        *it = std::byte(std::to_integer<unsigned char>(*it) + 1);
    545+        break;
    546+    }
    547+    if (it == end_range.rend()) {
    548+        end_range.insert(end_range.begin(), std::byte(0x01));
    549+    }
    


    aureleoules commented at 3:14 pm on September 26, 2022:

    Considering prefix is a UTF-8 encoded string: https://github.com/bitcoin/bitcoin/blob/f227e153e80c8c50c30d76e1ac638d7206c7ff61/src/wallet/walletdb.cpp#L30-L61

    is it useful to check for trailing 0xFF bytes? Simply incrementing the last byte could reduce code complexity.

    0    end_range.back() = std::byte(static_cast<unsigned char>(end_range.back()) + 1);
    

    ryanofsky commented at 7:38 pm on September 30, 2022:

    re: #24914 (review)

    Considering prefix is a UTF-8 encoded string:

    Just IMO, but I think it is better to have a little code complexity here and do prefix lookups correctly for all binary strings, than to make it subtly or unexpectedly fail for non-utf8 strings. There’s a tradeoff but in this case it seems better to make the API implementation a little more complex if it can avoid making API usage more complex or error prone.


    aureleoules commented at 10:04 pm on October 1, 2022:

    Alright, in this case I think a unit test with a prefix with a trailing 0xFF should be added.

    Edit: this doesn’t seem necessary considering #24914 (review).

  87. aureleoules commented at 3:31 pm on September 26, 2022: member
    I left some code review comments.
  88. in src/wallet/sqlite.cpp:546 in e9b3bcb411 outdated
    541+        }
    542+        *it = std::byte(std::to_integer<unsigned char>(*it) + 1);
    543+        break;
    544+    }
    545+    if (it == end_range.rend()) {
    546+        end_range.insert(end_range.begin(), std::byte(0x01));
    


    ryanofsky commented at 7:59 pm on September 30, 2022:

    In commit “wallet: Add GetPrefixCursor to DatabaseBatch” (e9b3bcb411b5742d6579d95b5e6aa2ad3f7592be)

    The handling of this last condition doesn’t seem right. For example if you are looking for keys with prefix X'FFFF', the right condition to check for is simply key >= X'FFFF'. It is not key >= X'FFFF' AND key < X'01FFFF'.

    You could fix this by replacing this line with end_range.clear(). Then in the end_range.empty() case below, prepare the SQL statement WHERE key >= {begin_range} instead of WHERE key >= {begin_range} AND key < {end_range}.

    This would fix the prefix scan for prefixes that consist of 0xff bytes. It would also fix the prefix scan for empty prefixes (by returning all rows), so you could drop the if (prefix.empty()) return nullptr; error above too.

    Related to this, I previously implemented an sqlite prefix scan more simply using the sqlite instr function in https://github.com/bitcoin/bitcoin/commit/7a05b1dee2fa68b32bfb19e273fb55a5b3836a3e#diff-d43e46c8f7010dde905ec31057a1c5c2674df9be11683f98a21546377f2cd706 from #18608. But probably the approach in this PR is more efficient, unless the sql interpreter is doing some fancy rewriting.


    achow101 commented at 11:55 pm on October 6, 2022:
    Done as suggested.
  89. in src/wallet/db.h:110 in 10cf25c112 outdated
    108-    virtual void CloseCursor() = 0;
    109+    virtual std::unique_ptr<DatabaseCursor> GetNewCursor() = 0;
    110+    bool StartCursor()
    111+    {
    112+        m_cursor = GetNewCursor();
    113+        return m_cursor != nullptr;  
    


    ryanofsky commented at 1:27 am on October 4, 2022:

    In commit “wallet: Introduce DatabaseCursor RAII class for managing cursor” (10cf25c112ff4ac7bcece87a821996004cec2b24)

    trailing whitespace


    achow101 commented at 9:52 pm on October 6, 2022:
    Fixed
  90. in src/wallet/db.h:116 in e9b3bcb411 outdated
    101@@ -102,6 +102,7 @@ class DatabaseBatch
    102     }
    103 
    104     virtual std::unique_ptr<DatabaseCursor> GetNewCursor() = 0;
    105+    virtual std::unique_ptr<DatabaseCursor> GetNewPrefixCursor(CDataStream& prefix) = 0;
    


    ryanofsky commented at 5:45 pm on October 4, 2022:

    In commit “wallet: Add GetPrefixCursor to DatabaseBatch” (e9b3bcb411b5742d6579d95b5e6aa2ad3f7592be)

    Can prefix just be Span<const std::byte>? It seems like a prefix scan should only require an array of bytes to function, not a full CDataStream object, much less a nonconst one.


    achow101 commented at 11:57 pm on October 6, 2022:
    A CDataStream is needed to produce the correct prefix as all of the DBKeys are strings serialized with length prefixes. So CDataStreams need to be made for the prefixes anyways, and it’s easier to just pass that straight in.
  91. in src/wallet/wallet.cpp:3691 in 5a8b0f1668 outdated
    3688     while (true) {
    3689         CDataStream ss_key(SER_DISK, CLIENT_VERSION);
    3690         CDataStream ss_value(SER_DISK, CLIENT_VERSION);
    3691-        bool ret = batch->ReadAtCursor(ss_key, ss_value, complete);
    3692+        bool ret = cursor->Next(ss_key, ss_value, complete);
    3693         if (!ret) {
    


    ryanofsky commented at 5:59 pm on October 4, 2022:

    In commit “wallet: Have cursor users use DatabaseCursor directly” (5a8b0f16681bf8f755e84567e7ecee2feb28336c)

    Code pre-exists this PR, and I guess it works, but this line should probably say if (complete || !ret). It’s not clear that the Next function would or should failure if the end of the stream is hit.

    Changing this also would make it clearer that this break is intended to handle two different conditions, and it would also make this code more similar to other code calling ReadAtCursor/Next


    achow101 commented at 9:52 pm on October 6, 2022:
    Done
  92. in src/wallet/sqlite.cpp:480 in e9b3bcb411 outdated
    476@@ -477,13 +477,16 @@ bool SQLiteCursor::Next(CDataStream& key, CDataStream& value, bool& complete)
    477     int res = sqlite3_step(m_cursor_stmt);
    478     if (res == SQLITE_DONE) {
    479         complete = true;
    480-        return true;
    481+        return false;
    


    ryanofsky commented at 6:12 pm on October 4, 2022:

    In commit “wallet: Add GetPrefixCursor to DatabaseBatch” (e9b3bcb411b5742d6579d95b5e6aa2ad3f7592be)

    Note to reviewers: this change seems safe because all of the cursor next callers except for one (in MigrateToSQLite) check the complete output variable before they use this return value.

    I do think more ideally this function would return something like a util::Result<bool> or an enum { SUCCESS, FAILURE, DONE } so the failure and done cases don’t overlap.


    achow101 commented at 0:23 am on October 7, 2022:
    I think that’s something that could be done in a followup.
  93. in src/wallet/bdb.cpp:689 in e9b3bcb411 outdated
    668@@ -668,9 +669,15 @@ bool BerkeleyCursor::Next(CDataStream& ssKey, CDataStream& ssValue, bool& comple
    669     complete = false;
    670     if (m_cursor == nullptr) return false;
    671     // Read at cursor
    672-    SafeDbt datKey;
    673+    SafeDbt datKey(m_key_prefix.data(), m_key_prefix.size());
    


    ryanofsky commented at 6:51 pm on October 4, 2022:

    In commit “wallet: Add GetPrefixCursor to DatabaseBatch” (e9b3bcb411b5742d6579d95b5e6aa2ad3f7592be)

    I don’t understand how datKey can be used to return the key data if it is now initialized to be a pointer to the m_key_prefix buffer instead of being a freestanding Dbt object that has the DB_DBT_MALLOC flag set and can allocate its own memory. If the keys are longer than the prefix how would they fit in datKey now?

    Probably I am missing something if the code works, but I would expect it look more like:

    0if (!m_first && !m_key_prefix.empty()) {
    1    SafeDbt prefix{m_key_prefix.data(), m_key_prefix.size()};
    2    m_cursor->get(prefix, value, DB_SET_RANGE);
    3}
    4ret = m_cursor->get(datKey, datValue, first ? DB_CURRENT : DB_NEXT);
    5m_first = false;
    

    In case I’m not missing something and there is a problem with this code, there was a BerkeleyBatch::ErasePrefix function in #18608 that did something very similar and could be an example.


    achow101 commented at 9:50 pm on October 6, 2022:
    AFAICT it just overwrites the datKey.data with a pointer to newly allocated memory. There’s nothing that indicates that this doesn’t work (i.e. no tests fail and wallets load succesfully).

    ryanofsky commented at 10:14 pm on October 6, 2022:

    AFAICT it just overwrites the datKey.data with a pointer to newly allocated memory. There’s nothing that indicates that this doesn’t work (i.e. no tests fail and wallets load succesfully).

    I don’t see how it could allocate memory if the DB_DBT_MALLOC flag is not set

    I didn’t get far enough in the PR to see if this has test coverage, but I guess you are pretty confident tests would fail if this was not working. It just seems like a mystery how it could work.


    achow101 commented at 10:50 pm on October 6, 2022:
    Considering that many prefixes are shorter than entire keys, if there was a problem here, I would expect deserialization of keys to be throwing exceptions, and that would cause any test that calls loadwallet to fail, which would result in a ton of test failures. Given that that doesn’t happen, I don’t think there’s any issue here.

    ryanofsky commented at 4:04 am on April 6, 2023:

    re: #24914 (review)

    I don’t see how it could allocate memory if the DB_DBT_MALLOC flag is not set

    I went down a rabbithole trying to answer a question in another PR (https://github.com/bitcoin/bitcoin/pull/27224/files#r1150232826), and finally figured out how this could work…

    It turns the DB_DBT_MALLOC flag is not a switch that controls whether BDB mallocs a new pointer for you or if it uses the pointer that you provide. BDB will malloc a new pointer whether or not you not pass DB_DBT_MALLOC. But you pass it, you are responsible for freeing the pointer, and if you don’t pass it, BDB will manage the memory as part of the cursor object so you don’t have to free it yourself. If you don’t want BDB to malloc memory to return the key, you can pass it a different DB_DBT_USERMEM flag instead.

    This does raises the question of whether it is ok to call memory_cleanse on memory that is owned by the cursor and should probably be read-only. But maybe it doesn’t need to be read-only, or doesn’t need to be read-only as long as you are done with the key and about to advance the cursor.

  94. achow101 force-pushed on Oct 6, 2022
  95. achow101 force-pushed on Oct 6, 2022
  96. in src/wallet/walletdb.cpp:987 in 6106b02d7c outdated
    982+        spk_man->SetCache(cache);
    983+
    984+        // Get unencrypted keys
    985+        prefix = PrefixStream(DBKeys::WALLETDESCRIPTORKEY, id);
    986+        LoadResult key_res = LoadRecords(pwallet, batch, DBKeys::WALLETDESCRIPTORKEY, prefix,
    987+            [&id, &num_keys, &spk_man] (CWallet* pwallet, CDataStream& key, CDataStream& value, std::string& err) {
    


    furszy commented at 3:14 pm on October 31, 2022:

    In 6106b02d:

    No need to pass num_keys to this function. The variable is overwritten below with the key_res.m_records field (LoadRecords returns the read records number).


    achow101 commented at 8:04 pm on November 4, 2022:
    Done
  97. in src/wallet/walletdb.cpp:1032 in 6106b02d7c outdated
    1027+        num_keys = key_res.m_records;
    1028+
    1029+        // Get encrypted keys
    1030+        prefix = PrefixStream(DBKeys::WALLETDESCRIPTORCKEY, id);
    1031+        LoadResult ckey_res = LoadRecords(pwallet, batch, DBKeys::WALLETDESCRIPTORCKEY, prefix,
    1032+            [&id, &num_ckeys, &spk_man] (CWallet* pwallet, CDataStream& key, CDataStream& value, std::string& err) {
    


    furszy commented at 3:16 pm on October 31, 2022:

    In 6106b02d:

    Same as above, no need to pass num_ckeys into this lambda.


    achow101 commented at 8:04 pm on November 4, 2022:
    Done
  98. furszy commented at 3:23 pm on October 31, 2022: member
    Two small findings, nothing biggie. Will do another full review round in the coming days.
  99. in src/wallet/walletdb.cpp:926 in 541cc8a772 outdated
    921+            pwallet->WalletLogPrintf("Inactive HD Chains found but no Legacy ScriptPubKeyMan\n");
    922+            return DBErrors::CORRUPT;
    923+        }
    924+        for (const auto& [hd_seed_id, chain] : hd_chains) {
    925+            if (hd_seed_id != pwallet->GetLegacyScriptPubKeyMan()->GetHDChain().seed_id) {
    926+                pwallet->GetLegacyScriptPubKeyMan()->AddInactiveHDChain(chain);
    


    furszy commented at 4:34 pm on October 31, 2022:

    In 541cc8a7:

    tiny nit:

    0            if (hd_seed_id != legacy_spkm->GetHDChain().seed_id) {
    1                legacy_spkm->AddInactiveHDChain(chain);
    

    (could also save the active chain seed id into a variable so we don’t call the getter on every iteration)


    achow101 commented at 8:04 pm on November 4, 2022:
    Done
  100. in src/wallet/walletdb.cpp:863 in 6106b02d7c outdated
    977+        result = std::max(result, lh_cache_res.m_result);
    978+
    979+        // Set the cache for this descriptor
    980+        auto spk_man = (DescriptorScriptPubKeyMan*)pwallet->GetScriptPubKeyMan(id);
    981+        assert(spk_man);
    982+        spk_man->SetCache(cache);
    


    furszy commented at 7:37 pm on October 31, 2022:

    In 6106b02:

    This is a non-issue, just a conceptual talk: Shouldn’t the cache be set after load the unencrypted/crypted keys?. Because, with it, we could validate that all the pubkeys inside the cache actually exist on the wallet at load time.


    achow101 commented at 8:06 pm on November 4, 2022:
    Perhaps, but it doesn’t particularly matter. The cache is only used for lookup if available, it it’s incorrect, it just means the things derivation needs are not available and so it can’t derive.
  101. furszy commented at 8:48 pm on October 31, 2022: member
    Aside from the previous comments, code review ACK a1a0984a
  102. achow101 force-pushed on Nov 4, 2022
  103. in src/wallet/walletdb.cpp:659 in a0bc6bda27 outdated
    655@@ -647,9 +656,7 @@ ReadKeyValue(CWallet* pwallet, CDataStream& ssKey, CDataStream& ssValue,
    656             ssValue >> strValue;
    657             pwallet->LoadDestData(DecodeDestination(strAddress), strKey, strValue);
    658         } else if (strType == DBKeys::HDCHAIN) {
    659-            CHDChain chain;
    660-            ssValue >> chain;
    661-            pwallet->GetOrCreateLegacyScriptPubKeyMan()->LoadHDChain(chain);
    662+            if (!LoadHDChain(pwallet, ssKey, ssValue, strErr)) return false;
    


    theStack commented at 1:50 am on November 13, 2022:

    Commit a0bc6bda272e7336deb55bdbf535534ed3a40306 currently doesn’t compile, since the call to LoadHDChain(...) has too many arguments:

    0            if (!LoadHDChain(pwallet, ssValue)) return false;
    

    achow101 commented at 3:57 pm on November 18, 2022:
    Fixed.
  104. achow101 force-pushed on Nov 18, 2022
  105. in src/wallet/test/wallet_tests.cpp:877 in f91be0848d outdated
    872+private:
    873+    bool m_pass{true};
    874+
    875+public:
    876+    explicit FailCursor(bool pass) : m_pass(pass) {}
    877+    bool Next(CDataStream& key, CDataStream& value, bool& complete) override { return m_pass; }
    


    furszy commented at 1:13 pm on November 20, 2022:

    nit in f91be084:

    I’m not sure that this line make much sense. Return true without setting any key, value, nor the complete flag sounds like something that could cause a bug in the future. Might be better to just return false for now. (same for the DummyCursor class)


    achow101 commented at 3:06 am on November 30, 2022:
    Done
  106. in src/wallet/walletdb.cpp:915 in b51b3fb458 outdated
    910+                chain.nExternalChainCounter = std::max(chain.nExternalChainCounter, index + 1);
    911+            }
    912+        }
    913+        return DBErrors::LOAD_OK;
    914+    });
    915+    result = std::max(result, script_res.m_result);
    


    furszy commented at 1:32 pm on November 20, 2022:

    small bug in b51b3fb4:

    We just read the keymeta records, the result update has to be against keymeta_res.m_result, not script_res.


    achow101 commented at 3:06 am on November 30, 2022:
    Fixed
  107. in src/wallet/walletdb.cpp:1113 in 327dfbfd97 outdated
    1108+        key >> hash;
    1109+        key >> n;
    1110+        pwallet->LockCoin(COutPoint(hash, n));
    1111+        return DBErrors::LOAD_OK;
    1112+    });
    1113+    result = std::max(result, tx_res.m_result);
    


    furszy commented at 1:39 pm on November 20, 2022:

    In 327dfbfd:

    Just loaded the locked utxo, this should be locked_utxo_res.m_result, not tx_res.m_result.


    achow101 commented at 3:06 am on November 30, 2022:
    Fixed
  108. DrahtBot added the label Needs rebase on Nov 30, 2022
  109. achow101 force-pushed on Nov 30, 2022
  110. DrahtBot removed the label Needs rebase on Nov 30, 2022
  111. achow101 force-pushed on Nov 30, 2022
  112. in src/wallet/test/wallet_tests.cpp:873 in 8dca1bd832 outdated
    866@@ -867,6 +867,16 @@ BOOST_FIXTURE_TEST_CASE(ZapSelectTx, TestChain100Setup)
    867     TestUnloadWallet(std::move(wallet));
    868 }
    869 
    870+class FailCursor : public DatabaseCursor
    871+{
    872+private:
    873+    bool m_pass{true};
    


    furszy commented at 1:01 pm on November 30, 2022:
    CI is complaining because of this unused private field.
  113. achow101 force-pushed on Nov 30, 2022
  114. DrahtBot added the label Needs rebase on Dec 5, 2022
  115. achow101 force-pushed on Dec 6, 2022
  116. in src/wallet/dump.cpp:77 in c8c9b74fd4 outdated
    75+            DatabaseCursor::Status status = cursor->Next(ss_key, ss_value);
    76+            if (status == DatabaseCursor::Status::DONE) {
    77                 ret = true;
    78                 break;
    79-            } else if (!ret) {
    80+            } else if (status == DatabaseCursor::Status::FAIL) {
    


    furszy commented at 10:03 pm on December 6, 2022:

    In c8c9b74f:

    bug here, as the ret variable is initialized to true, need to set ret=false when the returned status is FAIL.


    achow101 commented at 10:18 pm on December 6, 2022:
    Done
  117. DrahtBot removed the label Needs rebase on Dec 6, 2022
  118. achow101 force-pushed on Dec 6, 2022
  119. in src/wallet/db.h:187 in 2c7b214cbc outdated
    181@@ -156,6 +182,11 @@ class WalletDatabase
    182     virtual std::unique_ptr<DatabaseBatch> MakeBatch(bool flush_on_close = true) = 0;
    183 };
    184 
    185+class DummyCursor : public DatabaseCursor
    186+{
    187+    bool Next(CDataStream& key, CDataStream& value, bool& complete) override { return false; }
    


    theStack commented at 7:29 pm on December 11, 2022:
    Obviously it doesn’t matter, since all tests still pass, but just for the sake of understanding the code better: is this change from returning true (compared to the earlier DummyBatch::ReadAtCursor method that is removed below) to false intended here? true would IMHO be a more logical choice for a cursor belonging to a database that never fails.

    furszy commented at 8:34 pm on December 11, 2022:
    Check this #24914 (review). As we usually run this code inside a loop until it fails or return complete=true, if it never fails, then it can become a bug in the future.
  120. in src/wallet/test/wallet_tests.cpp:891 in 2c7b214cbc outdated
    887@@ -882,9 +888,7 @@ class FailBatch : public DatabaseBatch
    888     void Flush() override {}
    889     void Close() override {}
    890 
    891-    bool StartCursor() override { return true; }
    892-    bool ReadAtCursor(CDataStream& ssKey, CDataStream& ssValue, bool& complete) override { return false; }
    893-    void CloseCursor() override {}
    894+    std::unique_ptr<DatabaseCursor> GetNewCursor() override { return std::make_unique<FailCursor>(m_pass); }
    


    theStack commented at 8:07 pm on December 11, 2022:
    0    std::unique_ptr<DatabaseCursor> GetNewCursor() override { return std::make_unique<FailCursor>(); }
    

    Otherwise, compiling the second commit (2c7b214cbc06ac07311add940925683c779e49b4) fails:

     0  CXX      wallet/test/test_test_bitcoin-scriptpubkeyman_tests.o
     1In file included from wallet/test/wallet_tests.cpp:5:
     2In file included from ./wallet/wallet.h:10:
     3In file included from ./fs.h:8:
     4In file included from ./tinyformat.h:144:
     5In file included from /usr/include/c++/v1/algorithm:653:
     6In file included from /usr/include/c++/v1/functional:500:
     7In file included from /usr/include/c++/v1/__functional/function.h:20:
     8In file included from /usr/include/c++/v1/__memory/shared_ptr.h:25:
     9/usr/include/c++/v1/__memory/unique_ptr.h:728:32: error: no matching constructor for initialization of 'wallet::wallet_tests::FailCursor'
    10    return unique_ptr<_Tp>(new _Tp(_VSTD::forward<_Args>(__args)...));
    11                               ^   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    12wallet/test/wallet_tests.cpp:891:75: note: in instantiation of function template specialization 'std::make_unique<wallet::wallet_tests::FailCursor, bool &>' requested here
    13    std::unique_ptr<DatabaseCursor> GetNewCursor() override { return std::make_unique<FailCursor>(m_pass); }
    14                                                                          ^
    15wallet/test/wallet_tests.cpp:870:7: note: candidate constructor (the implicit copy constructor) not viable: no known conversion from 'bool' to 'const wallet::wallet_tests::FailCursor' for 1st argument
    16class FailCursor : public DatabaseCursor
    17      ^
    18wallet/test/wallet_tests.cpp:870:7: note: candidate constructor (the implicit default constructor) not viable: requires 0 arguments, but 1 was provided
    191 error generated.
    
  121. achow101 commented at 8:08 pm on December 12, 2022: member
    I’ve pulled the first 4 commits for the DatabaseCursor stuff into #26690 since I’m finding these changes to be useful elsewhere and would like them to be merged sooner.
  122. achow101 marked this as a draft on Dec 12, 2022
  123. fanquake referenced this in commit a62231bca6 on Jan 23, 2023
  124. achow101 force-pushed on Jan 23, 2023
  125. achow101 marked this as ready for review on Jan 23, 2023
  126. sidhujag referenced this in commit 3c878bbb45 on Jan 24, 2023
  127. DrahtBot added the label Needs rebase on Jan 26, 2023
  128. achow101 force-pushed on Feb 1, 2023
  129. DrahtBot removed the label Needs rebase on Feb 1, 2023
  130. in src/wallet/sqlite.cpp:525 in 0e45584e4e outdated
    519@@ -516,6 +520,53 @@ std::unique_ptr<DatabaseCursor> SQLiteBatch::GetNewCursor()
    520     return cursor;
    521 }
    522 
    523+std::unique_ptr<DatabaseCursor> SQLiteBatch::GetNewPrefixCursor(CDataStream& prefix)
    524+{
    525+    if (prefix.empty()) return nullptr;
    


    furszy commented at 1:27 pm on February 9, 2023:
    In 0e45584e: This emptiness check is done for sqlite but not for bdb. As this is a logical error, what if we throw an exception instead of returning a nullptr?

    achow101 commented at 9:10 pm on February 9, 2023:
    I’ve added the check to bdb. I’ve also changed the check to use Assume. This is basically equivalent with throwing an exception here since the only way to reach it is through programming errors.
  131. in src/wallet/sqlite.cpp:556 in 0e45584e4e outdated
    551+        const char* stmt_text = "SELECT key, value FROM main WHERE key >= ?";
    552+        res = sqlite3_prepare_v2(m_database.m_db, stmt_text, -1, &cursor->m_cursor_stmt, nullptr);
    553+    } else {
    554+        const char* stmt_text = "SELECT key, value FROM main WHERE key >= ? AND key < ?";
    555+        res = sqlite3_prepare_v2(m_database.m_db, stmt_text, -1, &cursor->m_cursor_stmt, nullptr);
    556+    }
    


    furszy commented at 1:36 pm on February 9, 2023:

    In 0e45584e:

    styling tiny nit (no need to do if you don’t have to re-touch the area):

    0const char* stmt_text = end_range.empty() ? "SELECT key, value FROM main WHERE key >= ?" :
    1                            "SELECT key, value FROM main WHERE key >= ? AND key < ?";
    2res = sqlite3_prepare_v2(m_database.m_db, stmt_text, -1, &cursor->m_cursor_stmt, nullptr);
    

    achow101 commented at 9:10 pm on February 9, 2023:
    Done as suggested.
  132. in src/wallet/salvage.cpp:140 in 414457710d outdated
    136@@ -136,21 +137,34 @@ bool RecoverDatabaseFile(const ArgsManager& args, const fs::path& file_path, bil
    137 
    138     DbTxn* ptxn = env->TxnBegin();
    139     CWallet dummyWallet(nullptr, "", gArgs, CreateDummyWalletDatabase());
    140+    LOCK(dummyWallet.cs_wallet);
    


    furszy commented at 1:52 pm on February 9, 2023:

    In 41445771:

    This lock isn’t needed. LoadKey, LoadCryptedKey, LoadEncryptionKey and LoadHDChain lock the wallet mutex internally.


    achow101 commented at 9:10 pm on February 9, 2023:
    Removed
  133. in src/wallet/walletdb.cpp:511 in 9bd810a394 outdated
    782+        std::string type;
    783+        ssKey >> type;
    784+        assert(type == key);
    785+        if ((result.m_result = std::max(result.m_result, load_func(pwallet, ssKey, ssValue, result.m_error))) == DBErrors::CORRUPT) {
    786+            pwallet->WalletLogPrintf("%s\n", result.m_error);
    787+        }
    


    furszy commented at 2:01 pm on February 9, 2023:

    In 9bd810a3:

    Could return here, shouldn’t continue iterating the db if a CORRUPT record was found.


    achow101 commented at 8:40 pm on February 9, 2023:
    No, this was intentional in order to avoid changing current behavior as little as possible. Right now, we will continue to read all records even if a corrupt one is found, so that all corrupt records can be found and reported to the user. So this doesn’t return early in order to preserve that behavior.

    furszy commented at 9:31 pm on February 9, 2023:

    Right now, we will continue to read all records even if a corrupt one is found, so that all corrupt records can be found and reported to the user

    We print all corrupt records of one type, not all the corrupted records. Right now, some flows return due corruption right after finish reading all the records of one type, while others continue for a bit longer.

    I’m just not so convinced that worth to log all corrupted records. I don’t see much of a benefit for the user, might worth for debugging but in that case, might be better to have an RPC command or an external tool that scans the .dat file and dumps overall info about the wallet (including the corrupted info etc).

    But no problem, all good if it’s to preserve the current behavior. This PR is already doing a lot. Let’s keep moving forward.


    Side not, I just had a dejavú. Most likely, we had this convo earlier in the PR and I forgot it.. (my bad if that is the case).


    ryanofsky commented at 11:21 pm on May 19, 2023:

    re: #24914 (review)

    In commit “walletdb: Refactor legacy wallet record loading into its own function” (8648511b567dfcdea7ffa5ac4595a43a768f5525)

    We print all corrupt records of one type, not all the corrupted records

    I guess I disagree a little, but I think it is good to log errors about everything we know is corrupt, even if we don’t log errors about things we can’t know are corrupt.

    Regardless, if there is a change in behavior here I think it would be good mention what’s changing in the commit message.

  134. furszy commented at 2:11 pm on February 9, 2023: member
    Made another half PR review round (the thirtieth one?), left few nits, nothing biggie. Let’s see if we can push further to merge this one soon.
  135. achow101 force-pushed on Feb 9, 2023
  136. in src/wallet/walletdb.cpp:834 in 4f83673a5b outdated
    829+    LoadResult key_res = LoadRecords(pwallet, batch, DBKeys::KEY,
    830+        [] (CWallet* pwallet, CDataStream& key, CDataStream& value, std::string& err) {
    831+        if (!LoadKey(pwallet, key, value, err)) {
    832+            return DBErrors::CORRUPT;
    833+        }
    834+        return DBErrors::LOAD_OK;
    


    furszy commented at 2:00 pm on February 10, 2023:

    In 4f83673a:

    tiny styling nit (no need to do it if you don’t re-touch):

    0return LoadKey(pwallet, key, value, err) ? DBErrors::LOAD_OK : DBErrors::CORRUPT;
    

    (same for the encrypted keys load)


    achow101 commented at 6:15 pm on February 10, 2023:
    Done
  137. in src/wallet/walletdb.cpp:817 in cf4a765eb0 outdated
    960+            bool parent = true;
    961+            uint256 desc_id;
    962+            uint32_t key_exp_index;
    963+            uint32_t der_index;
    964+            key >> desc_id;
    965+            assert(desc_id == id);
    


    furszy commented at 2:31 pm on February 10, 2023:

    In cf4a765e:

    This assertion is new to the code, could return a corruption error instead of abort the soft. (same applies to the similar assertions added to all the other descriptor records)


    achow101 commented at 6:09 pm on February 10, 2023:
    It wouldn’t be corruption error, it would be a (major) bug in SQLite. We’re requesting records by a prefix which includes the id, and then checking that that part of the record contains the same id. The only way this could go wrong is that the prefix retrieval is incorrect, so I think assert is appropriate here and for the other descriptor records.

    furszy commented at 6:39 pm on February 10, 2023:
    yeah true, nvm then.
  138. in src/wallet/walletdb.cpp:1066 in 890143f2ec outdated
    1062@@ -1114,12 +1063,99 @@ static DBErrors LoadAddressBookRecords(CWallet* pwallet, DatabaseBatch& batch) E
    1063     return DBErrors::LOAD_OK;
    1064 }
    1065 
    1066+static DBErrors LoadTxRecords(CWallet* pwallet, DatabaseBatch& batch, std::vector<uint256> upgraded_txs, bool& any_unordered) EXCLUSIVE_LOCKS_REQUIRED(pwallet->cs_wallet)
    


    furszy commented at 2:49 pm on February 10, 2023:

    In 890143f2:

    upgraded_txs is a return arg, needs to be passed by reference.


    achow101 commented at 6:15 pm on February 10, 2023:
    Fixed
  139. achow101 force-pushed on Feb 10, 2023
  140. furszy approved
  141. furszy commented at 6:42 pm on February 10, 2023: member
    Code review ACK 70bf403
  142. DrahtBot added the label Needs rebase on Apr 3, 2023
  143. achow101 force-pushed on Apr 10, 2023
  144. DrahtBot removed the label Needs rebase on Apr 10, 2023
  145. DrahtBot added the label Needs rebase on Apr 12, 2023
  146. achow101 force-pushed on Apr 12, 2023
  147. DrahtBot removed the label Needs rebase on Apr 12, 2023
  148. DrahtBot added the label Needs rebase on May 1, 2023
  149. achow101 force-pushed on May 1, 2023
  150. achow101 force-pushed on May 1, 2023
  151. DrahtBot removed the label Needs rebase on May 1, 2023
  152. in src/wallet/bdb.h:197 in f76830e546 outdated
    193@@ -194,9 +194,8 @@ class BerkeleyCursor : public DatabaseCursor
    194     bool m_first{true};
    195 
    196 public:
    197-    // Constructor for cursor for all records
    198-    explicit BerkeleyCursor(BerkeleyDatabase& database, BerkeleyBatch* batch=nullptr) : BerkeleyCursor(database, batch, {}) {}
    199-    // Constructor for cursor for records matching the prefix
    


    furszy commented at 2:32 pm on May 3, 2023:

    In f76830e5:

    This db cursor changes need to be in the first commit as it’s currently not compiling.


    achow101 commented at 3:02 pm on May 3, 2023:
    Oops, fixed.
  153. achow101 force-pushed on May 3, 2023
  154. furszy approved
  155. furszy commented at 3:21 pm on May 3, 2023: member
    ACK 00b6e0bb
  156. in src/wallet/bdb.cpp:735 in 543d23fa5d outdated
    727@@ -716,6 +728,13 @@ std::unique_ptr<DatabaseCursor> BerkeleyBatch::GetNewCursor()
    728     return std::make_unique<BerkeleyCursor>(m_database);
    729 }
    730 
    731+std::unique_ptr<DatabaseCursor> BerkeleyBatch::GetNewPrefixCursor(CDataStream& prefix)
    732+{
    733+    if (!pdb) return nullptr;
    734+    Assume(!prefix.empty());
    735+    return std::make_unique<BerkeleyCursor>(m_database, /*batch=*/nullptr, prefix);
    


    furszy commented at 3:27 pm on May 3, 2023:
    Note: If this is used in a read-write process, I’m quite sure that it will suffer from the same problematic tackled in #27556.

    achow101 commented at 4:14 pm on May 3, 2023:
    Done
  157. achow101 force-pushed on May 3, 2023
  158. furszy approved
  159. furszy commented at 0:47 am on May 9, 2023: member
    diff ACK 265b9e3
  160. fanquake requested review from ryanofsky on May 11, 2023
  161. fanquake requested review from Sjors on May 11, 2023
  162. DrahtBot added the label Needs rebase on May 15, 2023
  163. achow101 force-pushed on May 15, 2023
  164. DrahtBot removed the label Needs rebase on May 15, 2023
  165. in src/wallet/sqlite.cpp:541 in 42cce1d43b outdated
    510@@ -507,6 +511,7 @@ DatabaseCursor::Status SQLiteCursor::Next(DataStream& key, DataStream& value)
    511 
    512 SQLiteCursor::~SQLiteCursor()
    513 {
    514+    sqlite3_clear_bindings(m_cursor_stmt);
    


    ryanofsky commented at 0:24 am on May 16, 2023:

    In commit “wallet: Add GetPrefixCursor to DatabaseBatch” (42cce1d43be0e662783abf4af60283c352297eac)

    Would be good to log an error if this fails


    achow101 commented at 4:04 pm on May 16, 2023:
    The docs don’t say what the return values may be. I think SQLite itself will also log an error using the ErrorLogCallback that we during initialization.

    ryanofsky commented at 8:00 pm on May 17, 2023:

    re: #24914 (review)

    The docs don’t say what the return values may be. I think SQLite itself will also log an error using the ErrorLogCallback that we during initialization.

    Is it safe to assume from https://www.sqlite.org/rescode.html that this is supposed to return SQLITE_OK? I just think it’s useful to at least log behavior we are not expecting, even if we don’t know it is an error to help debugging. If you disagree though thats fine.


    achow101 commented at 4:55 pm on May 22, 2023:
    I think it’s fine to leave it as is.
  166. in src/wallet/sqlite.cpp:550 in 42cce1d43b outdated
    545+    // the prefix incremented by one (when interpreted as an integer)
    546+    std::vector<std::byte> start_range(prefix.begin(), prefix.end());
    547+    std::vector<std::byte> end_range(prefix.begin(), prefix.end());
    548+    auto it = end_range.rbegin();
    549+    for (; it != end_range.rend(); ++it) {
    550+        if (*it == std::byte(0xff)) {
    


    ryanofsky commented at 0:49 am on May 16, 2023:

    In commit “wallet: Add GetPrefixCursor to DatabaseBatch” (42cce1d43be0e662783abf4af60283c352297eac)

    C++ doesn’t actually require that a byte is 8 bits, so it would be more correct to write this as if (*it == std::byte(std::numeric_limits<unsigned char>::max())) `


    achow101 commented at 4:18 pm on May 16, 2023:
    Done
  167. in src/wallet/sqlite.h:32 in 42cce1d43b outdated
    27 
    28     explicit SQLiteCursor() {}
    29+    explicit SQLiteCursor(const std::vector<std::byte>& start_range, const std::vector<std::byte>& end_range)
    30+        : m_prefix_range_start(start_range),
    31+        m_prefix_range_end(end_range)
    32+    {}
    


    ryanofsky commented at 1:09 am on May 16, 2023:

    In commit “wallet: Add GetPrefixCursor to DatabaseBatch” (42cce1d43be0e662783abf4af60283c352297eac)

    Using const references here forces vectors to be copied. Passing the argument by value would give the caller option to avoid copies:

    0    explicit SQLiteCursor(std::vector<std::byte> start_range, std::vector<std::byte> end_range)
    1        : m_prefix_range_start(std::move(start_range)),
    2        m_prefix_range_end(std::move(end_range))
    3    {}
    

    Passing arguments by value instead of const reference is usually a good pattern for functions inserting the arguments into a data structure.


    achow101 commented at 4:18 pm on May 16, 2023:
    Done
  168. in src/wallet/test/util.cpp:82 in 42cce1d43b outdated
    77+            continue;
    78+        }
    79+        *it = std::byte(std::to_integer<unsigned char>(*it) + 1);
    80+        break;
    81+    }
    82+    m_cursor_end = records.lower_bound(end_range);
    


    ryanofsky commented at 1:44 am on May 16, 2023:

    In commit “wallet: Add GetPrefixCursor to DatabaseBatch” (42cce1d43be0e662783abf4af60283c352297eac)

    I think you can just do:

    0m_cursor_end = records.upper_bound(SerializeData(prefix.begin(), prefix.end()));
    

    or

    0std::tie(m_cursor, m_cursor_end) = records.equal_range(SerializeData(prefix.begin(), prefix.end()));
    

    And don’t need the for loop.


    achow101 commented at 4:11 pm on May 16, 2023:
    I don’t think upper_bound or equal_range will work. They will find the upper bound to be the first item that is greater than our prefix, but any record with the right prefix and is longer than the prefix will be greater than it, so we end up actually only getting the record(s) that exactly match the prefix.

    ryanofsky commented at 12:25 pm on May 19, 2023:

    re: #24914 (review)

    I don’t think upper_bound or equal_range will work. They will find the upper bound to be the first item that is greater than our prefix, but any record with the right prefix and is longer than the prefix will be greater than it, so we end up actually only getting the record(s) that exactly match the prefix.

    Oops, you’re right. But I think the following is an elegant way to make equal_range work (implementation based on https://stackoverflow.com/questions/44717939/elegant-way-to-find-keys-with-given-prefix-in-stdmap-or-elements-in-stdset):

      0diff --git a/src/wallet/test/util.cpp b/src/wallet/test/util.cpp
      1index f14f451b52b8..0256d3fafbf6 100644
      2--- a/src/wallet/test/util.cpp
      3+++ b/src/wallet/test/util.cpp
      4@@ -61,25 +61,15 @@ CTxDestination getNewDestination(CWallet& w, OutputType output_type)
      5     return *Assert(w.GetNewDestination(output_type, ""));
      6 }
      7 
      8-MockableCursor::MockableCursor(const std::map<SerializeData, SerializeData>& records, bool pass, Span<std::byte> prefix)
      9+// BytePrefix object compares equal with other byte spans beginning with the same prefix.
     10+struct BytePrefix { Span<const std::byte> prefix; };
     11+bool operator<(BytePrefix a, Span<const std::byte> b) { return a.prefix < b.subspan(0, std::min(a.prefix.size(), b.size())); }
     12+bool operator<(Span<const std::byte> a, BytePrefix b) { return a.subspan(0, std::min(a.size(), b.prefix.size())) < b.prefix; }
     13+
     14+MockableCursor::MockableCursor(const MockableData& records, bool pass, Span<std::byte> prefix)
     15 {
     16     m_pass = pass;
     17-
     18-    // Start the cursor at the first value that is greater than or equal to the prefix
     19-    m_cursor = records.lower_bound(SerializeData(prefix.begin(), prefix.end()));
     20-
     21-    // The end cursor is the first item that is greater than or equal to the prefix + 1 (when interpreted as an integer)
     22-    SerializeData end_range(prefix.begin(), prefix.end());
     23-    auto it = end_range.rbegin();
     24-    for (; it != end_range.rend(); ++it) {
     25-        if (*it == std::byte(std::numeric_limits<unsigned char>::max())) {
     26-            *it = std::byte(0);
     27-            continue;
     28-        }
     29-        *it = std::byte(std::to_integer<unsigned char>(*it) + 1);
     30-        break;
     31-    }
     32-    m_cursor_end = records.lower_bound(end_range);
     33+    std::tie(m_cursor, m_cursor_end) = records.equal_range(BytePrefix{prefix});
     34 }
     35 
     36 DatabaseCursor::Status MockableCursor::Next(DataStream& key, DataStream& value)
     37@@ -165,7 +155,7 @@ bool MockableBatch::ErasePrefix(Span<const std::byte> prefix)
     38     return true;
     39 }
     40 
     41-std::unique_ptr<WalletDatabase> CreateMockableWalletDatabase(std::map<SerializeData, SerializeData> records)
     42+std::unique_ptr<WalletDatabase> CreateMockableWalletDatabase(MockableData records)
     43 {
     44     return std::make_unique<MockableDatabase>(records);
     45 }
     46diff --git a/src/wallet/test/util.h b/src/wallet/test/util.h
     47index 75f9b46c2a1d..3fdf40d13ccb 100644
     48--- a/src/wallet/test/util.h
     49+++ b/src/wallet/test/util.h
     50@@ -32,15 +32,17 @@ std::string getnewaddress(CWallet& w);
     51 /** Returns a new destination, of an specific type, from the wallet */
     52 CTxDestination getNewDestination(CWallet& w, OutputType output_type);
     53 
     54+using MockableData = std::map<SerializeData, SerializeData, std::less<>>;
     55+
     56 class MockableCursor: public DatabaseCursor
     57 {
     58 public:
     59-    std::map<SerializeData, SerializeData>::const_iterator m_cursor;
     60-    std::map<SerializeData, SerializeData>::const_iterator m_cursor_end;
     61+    MockableData::const_iterator m_cursor;
     62+    MockableData::const_iterator m_cursor_end;
     63     bool m_pass;
     64 
     65-    explicit MockableCursor(const std::map<SerializeData, SerializeData>& records, bool pass) : m_cursor(records.begin()), m_cursor_end(records.end()), m_pass(pass) {}
     66-    MockableCursor(const std::map<SerializeData, SerializeData>& records, bool pass, Span<std::byte> prefix);
     67+    explicit MockableCursor(const MockableData& records, bool pass) : m_cursor(records.begin()), m_cursor_end(records.end()), m_pass(pass) {}
     68+    MockableCursor(const MockableData& records, bool pass, Span<std::byte> prefix);
     69     ~MockableCursor() {}
     70 
     71     Status Next(DataStream& key, DataStream& value) override;
     72@@ -49,7 +51,7 @@ public:
     73 class MockableBatch : public DatabaseBatch
     74 {
     75 private:
     76-    std::map<SerializeData, SerializeData>& m_records;
     77+    MockableData& m_records;
     78     bool m_pass;
     79 
     80     bool ReadKey(DataStream&& key, DataStream& value) override;
     81@@ -59,7 +61,7 @@ private:
     82     bool ErasePrefix(Span<const std::byte> prefix) override;
     83 
     84 public:
     85-    explicit MockableBatch(std::map<SerializeData, SerializeData>& records, bool pass) : m_records(records), m_pass(pass) {}
     86+    explicit MockableBatch(MockableData& records, bool pass) : m_records(records), m_pass(pass) {}
     87     ~MockableBatch() {}
     88 
     89     void Flush() override {}
     90@@ -82,10 +84,10 @@ public:
     91 class MockableDatabase : public WalletDatabase
     92 {
     93 public:
     94-    std::map<SerializeData, SerializeData> m_records;
     95+    MockableData m_records;
     96     bool m_pass{true};
     97 
     98-    MockableDatabase(std::map<SerializeData, SerializeData> records = {}) : WalletDatabase(), m_records(records) {}
     99+    MockableDatabase(MockableData records = {}) : WalletDatabase(), m_records(records) {}
    100     ~MockableDatabase() {};
    101 
    102     void Open() override {}
    103@@ -105,7 +107,7 @@ public:
    104     std::unique_ptr<DatabaseBatch> MakeBatch(bool flush_on_close = true) override { return std::make_unique<MockableBatch>(m_records, m_pass); }
    105 };
    106 
    107-std::unique_ptr<WalletDatabase> CreateMockableWalletDatabase(std::map<SerializeData, SerializeData> records = {});
    108+std::unique_ptr<WalletDatabase> CreateMockableWalletDatabase(MockableData records = {});
    109 
    110 MockableDatabase& GetMockableDatabase(CWallet& wallet);
    111 } // namespace wallet
    112diff --git a/src/wallet/test/walletload_tests.cpp b/src/wallet/test/walletload_tests.cpp
    113index 6823eafdfa7f..c1ff7baae117 100644
    114--- a/src/wallet/test/walletload_tests.cpp
    115+++ b/src/wallet/test/walletload_tests.cpp
    116@@ -83,7 +83,7 @@ BOOST_FIXTURE_TEST_CASE(wallet_load_ckey, TestingSetup)
    117 {
    118     SerializeData ckey_record_key;
    119     SerializeData ckey_record_value;
    120-    std::map<SerializeData, SerializeData> records;
    121+    MockableData records;
    122 
    123     {
    124         // Context setup.
    

    achow101 commented at 4:55 pm on May 22, 2023:
    Adopted these suggestions.
  169. DrahtBot added the label Needs rebase on May 16, 2023
  170. achow101 force-pushed on May 16, 2023
  171. achow101 force-pushed on May 16, 2023
  172. DrahtBot removed the label Needs rebase on May 16, 2023
  173. achow101 force-pushed on May 18, 2023
  174. DrahtBot added the label CI failed on May 18, 2023
  175. in src/wallet/test/db_tests.cpp:111 in 83b4e50291 outdated
    106+
    107+    // Test each supported db
    108+    for (const auto& database : dbs) {
    109+        BOOST_ASSERT(database);
    110+
    111+        std::vector<std::string> prefixes = {"FIRST", "SECOND", "P\xfe\xff", "P\xff\x01"};
    


    ryanofsky commented at 12:59 pm on May 19, 2023:

    In commit “test: add coverage for db cursor prefix range iteration” (83b4e50291a0bd2c45f370cb18bb479d8f73bc71)

    I think you need to add a case like "\xff\xff" to test sqlite code without a key < end_range condition

    Would also consider squashing this commit into earlier commit “wallet: Add GetPrefixCursor to DatabaseBatch” (57f35c90d2ebbc657a781ba3918398720b4319eb) because it’s helpful to see test code calling a new function in the same commit that introduces the function.


    achow101 commented at 4:55 pm on May 22, 2023:
    Added the case and squashed.
  176. in src/wallet/test/db_tests.cpp:95 in 83b4e50291 outdated
    90+{
    91+    std::vector<std::unique_ptr<WalletDatabase>> dbs;
    92+
    93+    // Create dbs
    94+    DatabaseOptions options;
    95+    options.create_flags = WALLET_FLAG_DESCRIPTORS;
    


    ryanofsky commented at 1:03 pm on May 19, 2023:

    In commit “test: add coverage for db cursor prefix range iteration” (83b4e50291a0bd2c45f370cb18bb479d8f73bc71)

    Setting this flag seems unnecessary and unrelated to the test


    achow101 commented at 4:55 pm on May 22, 2023:
    Removed
  177. in src/wallet/walletdb.cpp:316 in 1a91609e50 outdated
    333+        }
    334+        CKey key;
    335+        CPrivKey pkey;
    336+        uint256 hash;
    337+
    338+        ssValue >> pkey;
    


    ryanofsky commented at 1:22 pm on May 19, 2023:

    In commit “walletdb: Refactor key reading and loading to its own function” (e048fe71038caddb49a5875727a14e05f6edb0d3)

    Note: there is a slight change in behavior this commit which could be noted in the commit message.

    If this deserialization fails, or if any error conditions are hit below, wss.nKeys++ will not be incremented anymore. This seems ok, but previously wss.nKeys++ included the total number of keys found, even ones that could not be loaded.


    achow101 commented at 4:58 pm on May 22, 2023:

    I’ve changed it to increment unconditionally as mentioned below. Ultimately, wss gets removed so I don’t think the correctness of this counting particularly matters.

    Additionally, the current behavior doesn’t use the counts when there are failures, so having it be inaccurate in such cases shouldn’t matter.

  178. in src/wallet/walletdb.cpp:392 in 59ac3bccff outdated
    409+                strErr = "Error reading wallet database: Encrypted key corrupt";
    410+                return false;
    411+            }
    412+        }
    413+
    414+        if (!pwallet->GetOrCreateLegacyScriptPubKeyMan()->LoadCryptedKey(vchPubKey, vchPrivKey, checksum_valid))
    


    ryanofsky commented at 1:26 pm on May 19, 2023:

    In commit “walletdb: Refactor crypted key loading to its own function” (59ac3bccffe3fda352dc068fa3a0b507ed9a73d6)

    Similar to previous commit, now wss.nCKeys will not be incremented if this fails. Change in behavior could be noted in commit message.


    achow101 commented at 4:58 pm on May 22, 2023:
    Changed to increment unconditionally.
  179. in src/wallet/walletdb.cpp:465 in c48aa9d0df outdated
    460@@ -461,6 +461,9 @@ bool LoadHDChain(CWallet* pwallet, CDataStream& ssValue)
    461     return true;
    462 }
    463 
    464+//! Callback for filtering key types to deserialize in ReadKeyValue
    465+using KeyFilterFn = std::function<bool(const std::string&)>;
    


    ryanofsky commented at 1:34 pm on May 19, 2023:

    In commit “salvage: Remove use of ReadKeyValue in salvage” (c48aa9d0df7e34493a666c55e060304ef262d4b9)

    No need to move KeyFilterFn here, can just drop it entirely:

     0diff --git a/src/wallet/walletdb.cpp b/src/wallet/walletdb.cpp
     1index 5d3ced0ad225..6ab29379fb26 100644
     2--- a/src/wallet/walletdb.cpp
     3+++ b/src/wallet/walletdb.cpp
     4@@ -461,22 +461,15 @@ bool LoadHDChain(CWallet* pwallet, CDataStream& ssValue)
     5     return true;
     6 }
     7 
     8-//! Callback for filtering key types to deserialize in ReadKeyValue
     9-using KeyFilterFn = std::function<bool(const std::string&)>;
    10-
    11 static bool
    12 ReadKeyValue(CWallet* pwallet, DataStream& ssKey, CDataStream& ssValue,
    13-             CWalletScanState &wss, std::string& strType, std::string& strErr, const KeyFilterFn& filter_fn = nullptr) EXCLUSIVE_LOCKS_REQUIRED(pwallet->cs_wallet)
    14+             CWalletScanState &wss, std::string& strType, std::string& strErr) EXCLUSIVE_LOCKS_REQUIRED(pwallet->cs_wallet)
    15 {
    16     try {
    17         // Unserialize
    18         // Taking advantage of the fact that pair serialization
    19         // is just the two items serialized one after the other
    20         ssKey >> strType;
    21-        // If we have a filter, check if this matches the filter
    22-        if (filter_fn && !filter_fn(strType)) {
    23-            return true;
    24-        }
    25         // Legacy entries in descriptor wallets are not allowed, abort immediately
    26         if (pwallet->IsWalletFlagSet(WALLET_FLAG_DESCRIPTORS) && DBKeys::LEGACY_TYPES.count(strType) > 0) {
    27             wss.unexpected_legacy_entry = true;
    

    EDIT: I guess this is deleted later anyway, but it would be a little cleaner to remove it this commit.


    achow101 commented at 4:58 pm on May 22, 2023:
    Done
  180. achow101 force-pushed on May 19, 2023
  181. achow101 force-pushed on May 19, 2023
  182. in src/wallet/walletdb.cpp:783 in 8648511b56 outdated
    778+{
    779+    LoadResult result;
    780+    CDataStream ssKey(SER_DISK, CLIENT_VERSION);
    781+    CDataStream ssValue(SER_DISK, CLIENT_VERSION);
    782+
    783+    CDataStream prefix(0, 0);
    


    ryanofsky commented at 11:01 pm on May 19, 2023:

    In commit “walletdb: Refactor legacy wallet record loading into its own function” (8648511b567dfcdea7ffa5ac4595a43a768f5525)

    What’s the reason for using (0, 0) flags here and other places instead of (SER_DISK, CLIENT_VERSION) flags used elsewhere? Having a code comment about this somewhere would be very helpful for understanding. It seems a little dodgy to get rid of SER_DISK flag if present or future code might rely on it.


    achow101 commented at 4:44 pm on May 22, 2023:
    For serializing the strings for the prefix, the stream type and client version don’t matter so I just didn’t use them.

    maflcko commented at 6:48 am on May 23, 2023:

    You can just use DataStream for that, which comes with compile-time checks to enforce that both values are in fact never read, and thus can be omitted completely.

    Also, in the same commit, is there a reason why you are changing DataStream ssKey to CDataStream ssKey? That seems like a step backward, no?


    achow101 commented at 3:59 pm on May 23, 2023:

    Good point.

    I’ve changed the prefix to be made with a DataStream and for GetPrefixCursor to take Span<const std::byte> rather than a stream. Also updated to use DataStream key everywhere, I think that got lost in a rebase.

  183. in src/wallet/walletdb.h:48 in 8648511b56 outdated
    42@@ -43,18 +43,18 @@ struct WalletContext;
    43 static const bool DEFAULT_FLUSHWALLET = true;
    44 
    45 /** Error statuses for the wallet database */
    46-enum class DBErrors
    47+enum class DBErrors : int
    


    ryanofsky commented at 11:05 pm on May 19, 2023:

    In commit “walletdb: Refactor legacy wallet record loading into its own function” (8648511b567dfcdea7ffa5ac4595a43a768f5525)

    Would be helpful to have a code comment here saying enum number values are significant because if there are different error codes from reading different rows of the database, the error code with the highest number is the one that will be returned.


    achow101 commented at 4:59 pm on May 22, 2023:
    Added a comment.
  184. in src/wallet/walletdb.cpp:562 in 8648511b56 outdated
    557-            if (!LoadKey(pwallet, ssKey, ssValue, strErr)) return false;
    558             wss.nKeys++;
    559         } else if (strType == DBKeys::MASTER_KEY) {
    560             if (!LoadEncryptionKey(pwallet, ssKey, ssValue, strErr)) return false;
    561         } else if (strType == DBKeys::CRYPTED_KEY) {
    562-            if (!LoadCryptedKey(pwallet, ssKey, ssValue, strErr)) return false;
    


    ryanofsky commented at 11:11 pm on May 19, 2023:

    In commit “walletdb: Refactor legacy wallet record loading into its own function” (8648511b567dfcdea7ffa5ac4595a43a768f5525)

    It looks like in this commit, the side-effect in previous commit “walletdb: Refactor key reading and loading to its own function” (e048fe71038caddb49a5875727a14e05f6edb0d3) of changing the wss.nKeys value in case of key errors is partially reverted, because now the number of keys is incremented unconditionally, while the previous commit added more conditions.

    Would suggest changing previous commit to just increment wss.nKeys++ unconditionally, so that behavior doesn’t change here and this commit is simpler.

    Same comment applies to wss.nCKeys and commit “walletdb: Refactor crypted key loading to its own function” (4940c1cb3dca97faa2b6a4ddfab889cd4b6f00f4)

    The wss.nKeyMeta count also changes here if there is a deserialization exception. It would be good to note this in the commit message for this commit.


    achow101 commented at 4:59 pm on May 22, 2023:
    Changed to increment unconditionally, but these counters all get dropped in a later commit anyways.
  185. ryanofsky commented at 11:28 pm on May 19, 2023: contributor

    This looks great so far! Here is where I am in the review:

    • 79075f970802031d42c3808351baa91d5edf9783 wallet: Add GetPrefixCursor to DatabaseBatch (1/18)
    • 37a3e64ec84899468aab8c55fd556c93d195e322 walletdb: Consistently clear key and value streams before writing (2/18)
    • 39bfce498a7576cd41b2b362ee7688b665472f33 test: add coverage for db cursor prefix range iteration (3/18)
    • 3fbbdb2925a2e65b75af9116773e059dd5453afa walletdb: Refactor minversion loading (4/18)
    • bc2d88888f8c483823c1b718deedaf08bcd97b41 walletdb: Refactor wallet flags loading (5/18)
    • e048fe71038caddb49a5875727a14e05f6edb0d3 walletdb: Refactor key reading and loading to its own function (6/18)
    • 4940c1cb3dca97faa2b6a4ddfab889cd4b6f00f4 walletdb: Refactor crypted key loading to its own function (7/18)
    • 4c7d35da1d479714df6feb457ee0f723c59f71ee walletdb: Refactor encryption key loading to its own function (8/18)
    • a43f7d436b86a3f008edfe796f0915f848bc0406 walletdb: Refactor hd chain loading to its own function (9/18)
    • af8b890ae7c01c5f2b806948fbdf92767a9bd5c6 salvage: Remove use of ReadKeyValue in salvage (10/18)
    • 8648511b567dfcdea7ffa5ac4595a43a768f5525 walletdb: Refactor legacy wallet record loading into its own function (11/18)
    • f227c81e780dd5d8bf301250b6bec118373ba076 walletdb: Refactor descriptor wallet records loading (12/18)
    • 6d72a673e24ad92a062d311bb6deaa497c30997c walletdb: refactor address book loading (13/18)
    • f51bfb43723e2a43171d7f3a8a6abfb56c2ce675 walletdb: refactor tx loading (14/18)
    • 59d393c5a179249879addb6f94966fdac698b249 walletdb: refactor active spkm loading (15/18)
    • 8bb356f9594c61fc3043ed632be7130b97238d9b walletdb: refactor defaultkey and wkey loading (16/18)
    • a4fc170ef692cd2e1b3f16543c96080846673d82 walletdb: refactor decryption key loading (17/18)
    • 0a2db031d2532caa2a1b6ff5f6f2cb4bf1cdc2bf walletdb: Remove loading code where the database is iterated (18/18)
  186. achow101 force-pushed on May 22, 2023
  187. DrahtBot removed the label CI failed on May 22, 2023
  188. achow101 force-pushed on May 23, 2023
  189. in src/wallet/bdb.cpp:734 in 422a843608 outdated
    727@@ -716,6 +728,13 @@ std::unique_ptr<DatabaseCursor> BerkeleyBatch::GetNewCursor()
    728     return std::make_unique<BerkeleyCursor>(m_database, *this);
    729 }
    730 
    731+std::unique_ptr<DatabaseCursor> BerkeleyBatch::GetNewPrefixCursor(Span<const std::byte> prefix)
    732+{
    733+    if (!pdb) return nullptr;
    734+    Assume(!prefix.empty());
    


    ryanofsky commented at 2:46 pm on May 24, 2023:

    In commit “wallet: Add GetPrefixCursor to DatabaseBatch” (422a8436089c844934903a61fcec6b7b93995c07)

    If prefix being non-empty is a requirement, it would be good to add this Assume to the MockableBatch cursor as well, so all of the implementations of this method consistently reject empty prefixes, and test cases don’t appear to work with the mock database and then fail with real databases.

    Alternately you could consider just moving the Assume checks out of the GetNewPrefixCursor methods into the into LoadRecords / LoadLegacyWalletRecords functions calling those methods. This would make the database code less fragile and basically catch same programming errors at a higher level in the wallet.


    achow101 commented at 2:50 am on May 31, 2023:
    Moved the Assume to LoadRecords.
  190. in src/wallet/walletdb.cpp:479 in 82e025297b outdated
    485-            }
    486-            CKey key;
    487-            CPrivKey pkey;
    488-            uint256 hash;
    489-
    490             wss.nKeys++;
    


    ryanofsky commented at 2:58 pm on May 24, 2023:

    In commit “walletdb: Refactor key reading and loading to its own function” (82e025297ba8939660b15fad738be723d783165e)

    Note for other reviewers: There appears to be a slight change of behavior because now wss.nKeys will be incremented if the public key is not valid. This doesn’t actually change behavior because counts are not used if there are any failures:

    https://github.com/bitcoin/bitcoin/blob/214f8f18b310e3af88eba6a005439ae423ccd76a/src/wallet/walletdb.cpp#L932-L933

    Also wss variable is removed later anyway (https://github.com/bitcoin/bitcoin/pull/24914#discussion_r1200783492)

  191. in src/wallet/walletdb.cpp:455 in f079b7580d outdated
    451@@ -452,6 +452,15 @@ bool LoadEncryptionKey(CWallet* pwallet, DataStream& ssKey, CDataStream& ssValue
    452     return true;
    453 }
    454 
    455+bool LoadHDChain(CWallet* pwallet, CDataStream& ssValue)
    


    ryanofsky commented at 3:36 am on May 25, 2023:

    In commit “walletdb: Refactor hd chain loading to its own function” (f079b7580d4e7fa0d373decd1ffd7144abb83bab)

    Seems ok, but is there a rationale for previous commits catching exceptions in the refactored functions and this commit not catching exceptions?


    achow101 commented at 2:51 am on May 31, 2023:
    No, added the exception handling back.
  192. in src/wallet/salvage.cpp:206 in e9d974b7f4 outdated
    207+        } else if (strType == DBKeys::CRYPTED_KEY) {
    208+            fReadOK = LoadCryptedKey(&dummyWallet, ssKey, ssValue, strErr);
    209+        } else if (strType == DBKeys::MASTER_KEY) {
    210+            fReadOK = LoadEncryptionKey(&dummyWallet, ssKey, ssValue, strErr);
    211+        } else if (strType == DBKeys::HDCHAIN) {
    212+            fReadOK = LoadHDChain(&dummyWallet, ssValue);
    


    ryanofsky commented at 1:01 pm on May 25, 2023:

    In commit “salvage: Remove use of ReadKeyValue in salvage” (e9d974b7f4136c04f35091438f5e78e17d77299b)

    I guess this will no longer catch std::exception, but that is ok?


    achow101 commented at 2:51 am on May 31, 2023:
    Added it back.
  193. in src/wallet/walletdb.cpp:1171 in b3425cc31f outdated
    1167@@ -1014,7 +1168,7 @@ DBErrors WalletBatch::LoadWallet(CWallet* pwallet)
    1168     try {
    1169         pwallet->UpgradeKeyMetadata();
    1170     } catch (...) {
    1171-        result = DBErrors::CORRUPT;
    1172+        return DBErrors::CORRUPT;
    


    ryanofsky commented at 6:41 pm on May 25, 2023:

    In commit “walletdb: Refactor legacy wallet record loading into its own function” (b3425cc31feba14c5a0b0c4532437f12b5a901ff)

    What its this change doing? It seems unrelated to the commit and maybe would be better as a separate commit with an explanation.


    achow101 commented at 2:51 am on May 31, 2023:
    Don’t remember, undone. Probably a bad rebase?
  194. in src/wallet/walletdb.cpp:1051 in 3c51e44380 outdated
    1070+        };
    1071+        if (!pwallet->LoadToWallet(hash, fill_wtx)) {
    1072+            if (corrupted_tx) {
    1073+                result = DBErrors::CORRUPT;
    1074+            } else {
    1075+                result = DBErrors::NEED_RESCAN;
    


    ryanofsky commented at 11:44 pm on May 25, 2023:

    In commit “walletdb: refactor tx loading” (ad3ae8b6f4b2f3430b9d6c1b5bcf2627be15d961)

    It’s confusing that this commit leaves behind dead code implementing the same logic as the new code. Would suggest getting rid of it:

     0--- a/src/wallet/walletdb.cpp
     1+++ b/src/wallet/walletdb.cpp
     2@@ -300,11 +300,8 @@ bool WalletBatch::EraseLockedUTXO(const COutPoint& output)
     3 
     4 class CWalletScanState {
     5 public:
     6-    bool fAnyUnordered{false};
     7-    std::vector<uint256> vWalletUpgrade;
     8     std::map<OutputType, uint256> m_active_external_spks;
     9     std::map<OutputType, uint256> m_active_internal_spks;
    10-    bool tx_corrupt{false};
    11 
    12     CWalletScanState() = default;
    13 };
    14@@ -1134,7 +1131,6 @@ DBErrors WalletBatch::LoadWallet(CWallet* pwallet)
    15 {
    16     CWalletScanState wss;
    17     bool fNoncriticalErrors = false;
    18-    bool rescan_required = false;
    19     DBErrors result = DBErrors::LOAD_OK;
    20     bool any_unordered = false;
    21     std::vector<uint256> upgraded_txs;
    22@@ -1205,17 +1201,9 @@ DBErrors WalletBatch::LoadWallet(CWallet* pwallet)
    23                 if (strType == DBKeys::MASTER_KEY ||
    24                     strType == DBKeys::DEFAULTKEY) {
    25                     result = DBErrors::CORRUPT;
    26-                } else if (wss.tx_corrupt) {
    27-                    pwallet->WalletLogPrintf("Error: Corrupt transaction found. This can be fixed by removing transactions from wallet and rescanning.\n");
    28-                    // Set tx_corrupt back to false so that the error is only printed once (per corrupt tx)
    29-                    wss.tx_corrupt = false;
    30-                    result = DBErrors::CORRUPT;
    31                 } else {
    32                     // Leave other errors alone, if we try to fix them we might make things worse.
    33                     fNoncriticalErrors = true; // ... but do warn the user there is something wrong.
    34-                    if (strType == DBKeys::TX)
    35-                        // Rescan if there is a bad transaction record:
    36-                        rescan_required = true;
    37                 }
    38             }
    39             if (!strErr.empty())
    40@@ -1233,9 +1221,7 @@ DBErrors WalletBatch::LoadWallet(CWallet* pwallet)
    41         pwallet->LoadActiveScriptPubKeyMan(spk_man_pair.second, spk_man_pair.first, /*internal=*/true);
    42     }
    43 
    44-    if (rescan_required && result == DBErrors::LOAD_OK) {
    45-        result = DBErrors::NEED_RESCAN;
    46-    } else if (fNoncriticalErrors && result == DBErrors::LOAD_OK) {
    47+    if (fNoncriticalErrors && result == DBErrors::LOAD_OK) {
    48         result = DBErrors::NONCRITICAL_ERROR;
    49     }
    50 
    

    achow101 commented at 2:51 am on May 31, 2023:
    Done as suggested.
  195. in src/wallet/walletdb.cpp:781 in 51518daa60 outdated
    947+        key >> id;
    948+        WalletDescriptor desc;
    949+        try {
    950+            value >> desc;
    951+        } catch (const std::ios_base::failure&) {
    952+            err = strprintf("Error: Unrecognized descriptor found in wallet %s. ", pwallet->GetName());
    


    ryanofsky commented at 11:49 pm on May 25, 2023:

    In commit “walletdb: Refactor descriptor wallet records loading” (51518daa60952f31c34013c643feb8fe11c9777e)

    This commit is leaving behind dead code, including another copy of this print statement. Would be better to get rid of it here:

     0--- a/src/wallet/walletdb.cpp
     1+++ b/src/wallet/walletdb.cpp
     2@@ -310,11 +310,7 @@ public:
     3     std::vector<uint256> vWalletUpgrade;
     4     std::map<OutputType, uint256> m_active_external_spks;
     5     std::map<OutputType, uint256> m_active_internal_spks;
     6-    std::map<uint256, DescriptorCache> m_descriptor_caches;
     7-    std::map<std::pair<uint256, CKeyID>, CKey> m_descriptor_keys;
     8-    std::map<std::pair<uint256, CKeyID>, std::pair<CPubKey, std::vector<unsigned char>>> m_descriptor_crypt_keys;
     9     bool tx_corrupt{false};
    10-    bool descriptor_unknown{false};
    11 
    12     CWalletScanState() = default;
    13 };
    14@@ -1157,13 +1153,6 @@ DBErrors WalletBatch::LoadWallet(CWallet* pwallet)
    15                     // Set tx_corrupt back to false so that the error is only printed once (per corrupt tx)
    16                     wss.tx_corrupt = false;
    17                     result = DBErrors::CORRUPT;
    18-                } else if (wss.descriptor_unknown) {
    19-                    strErr = strprintf("Error: Unrecognized descriptor found in wallet %s. ", pwallet->GetName());
    20-                    strErr += (last_client > CLIENT_VERSION) ? "The wallet might had been created on a newer version. " :
    21-                            "The database might be corrupted or the software version is not compatible with one of your wallet descriptors. ";
    22-                    strErr += "Please try running the latest software version";
    23-                    pwallet->WalletLogPrintf("%s\n", strErr);
    24-                    return DBErrors::UNKNOWN_DESCRIPTOR;
    25                 } else {
    26                     // Leave other errors alone, if we try to fix them we might make things worse.
    27                     fNoncriticalErrors = true; // ... but do warn the user there is something wrong.
    

    achow101 commented at 2:51 am on May 31, 2023:
    Done as suggested.
  196. in src/wallet/walletdb.cpp:813 in ede0c95237 outdated
    808+    AssertLockHeld(pwallet->cs_wallet);
    809+    uint64_t flags;
    810+    if (batch.Read(DBKeys::FLAGS, flags)) {
    811+        if (!pwallet->LoadWalletFlags(flags)) {
    812+            pwallet->WalletLogPrintf("Error reading wallet database: Unknown non-tolerable wallet flags found\n");
    813+            return DBErrors::TOO_NEW;
    


    ryanofsky commented at 11:53 pm on May 25, 2023:

    In commit “walletdb: Refactor wallet flags loading” (ede0c95237e47de8a93af229f33f0cbedf5780fd)

    It seems good that this commit is changing the error code from CORRUPT to TOO_NEW when there are recognized flags, but it is confusing that this still leaves behind more code below that attempts to do this same thing. It would be good to remove that code as part of this commit:

     0--- a/src/wallet/walletdb.cpp
     1+++ b/src/wallet/walletdb.cpp
     2@@ -880,9 +880,6 @@ DBErrors WalletBatch::LoadWallet(CWallet* pwallet)
     3                 // we assume the user can live with:
     4                 if (IsKeyType(strType) || strType == DBKeys::DEFAULTKEY) {
     5                     result = DBErrors::CORRUPT;
     6-                } else if (strType == DBKeys::FLAGS) {
     7-                    // reading the wallet flags can only fail if unknown flags are present
     8-                    result = DBErrors::TOO_NEW;
     9                 } else if (wss.tx_corrupt) {
    10                     pwallet->WalletLogPrintf("Error: Corrupt transaction found. This can be fixed by removing transactions from wallet and rescanning.\n");
    11                     // Set tx_corrupt back to false so that the error is only printed once (per corrupt tx)
    

    achow101 commented at 2:52 am on May 31, 2023:
    Done as suggested.
  197. in src/wallet/walletdb.cpp:565 in b3425cc31f outdated
    839+    }
    840+
    841+    // Load unencrypted keys
    842+    LoadResult key_res = LoadRecords(pwallet, batch, DBKeys::KEY,
    843+        [] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) {
    844+        return LoadKey(pwallet, key, value, err) ? DBErrors::LOAD_OK : DBErrors::CORRUPT;
    


    ryanofsky commented at 5:04 pm on May 30, 2023:

    In commit “walletdb: Refactor legacy wallet record loading into its own function” (b3425cc31feba14c5a0b0c4532437f12b5a901ff)

    There is still another LoadKey call above on line 544, so is this loading the keys twice?

    https://github.com/bitcoin/bitcoin/blob/b3425cc31feba14c5a0b0c4532437f12b5a901ff/src/wallet/walletdb.cpp#L542-L544

    It seems like the call on line 544 should be dropped at the same time this is added.

    In general it would make this commit a lot clearer if it removed code that wasn’t needed and didn’t leave dead code behind. Would suggest:

     0--- a/src/wallet/walletdb.cpp
     1+++ b/src/wallet/walletdb.cpp
     2@@ -312,10 +312,8 @@ public:
     3     std::map<uint256, DescriptorCache> m_descriptor_caches;
     4     std::map<std::pair<uint256, CKeyID>, CKey> m_descriptor_keys;
     5     std::map<std::pair<uint256, CKeyID>, std::pair<CPubKey, std::vector<unsigned char>>> m_descriptor_crypt_keys;
     6-    std::map<uint160, CHDChain> m_hd_chains;
     7     bool tx_corrupt{false};
     8     bool descriptor_unknown{false};
     9-    bool unexpected_legacy_entry{false};
    10 
    11     CWalletScanState() = default;
    12 };
    13@@ -470,10 +468,8 @@ ReadKeyValue(CWallet* pwallet, DataStream& ssKey, CDataStream& ssValue,
    14         // Taking advantage of the fact that pair serialization
    15         // is just the two items serialized one after the other
    16         ssKey >> strType;
    17-        // Legacy entries in descriptor wallets are not allowed, abort immediately
    18         if (pwallet->IsWalletFlagSet(WALLET_FLAG_DESCRIPTORS) && DBKeys::LEGACY_TYPES.count(strType) > 0) {
    19-            wss.unexpected_legacy_entry = true;
    20-            return false;
    21+            return true;
    22         }
    23         if (strType == DBKeys::NAME) {
    24             std::string strAddress;
    25@@ -541,7 +537,6 @@ ReadKeyValue(CWallet* pwallet, DataStream& ssKey, CDataStream& ssValue,
    26             wss.nWatchKeys++;
    27         } else if (strType == DBKeys::KEY) {
    28             wss.nKeys++;
    29-            if (!LoadKey(pwallet, ssKey, ssValue, strErr)) return false;
    30         } else if (strType == DBKeys::MASTER_KEY) {
    31             if (!LoadEncryptionKey(pwallet, ssKey, ssValue, strErr)) return false;
    32         } else if (strType == DBKeys::CRYPTED_KEY) {
    33@@ -1069,17 +1064,9 @@ DBErrors WalletBatch::LoadWallet(CWallet* pwallet)
    34             std::string strType, strErr;
    35             if (!ReadKeyValue(pwallet, ssKey, ssValue, wss, strType, strErr))
    36             {
    37-                if (wss.unexpected_legacy_entry) {
    38-                    strErr = strprintf("Error: Unexpected legacy entry found in descriptor wallet %s. ", pwallet->GetName());
    39-                    strErr += "The wallet might have been tampered with or created with malicious intent.";
    40-                    pwallet->WalletLogPrintf("%s\n", strErr);
    41-                    return DBErrors::UNEXPECTED_LEGACY_ENTRY;
    42-                }
    43                 // losing keys is considered a catastrophic error, anything else
    44                 // we assume the user can live with:
    45-                if (strType == DBKeys::KEY ||
    46-                    strType == DBKeys::MASTER_KEY ||
    47-                    strType == DBKeys::CRYPTED_KEY||
    48+                if (strType == DBKeys::MASTER_KEY ||
    49                     strType == DBKeys::DEFAULTKEY) {
    50                     result = DBErrors::CORRUPT;
    51                 } else if (wss.tx_corrupt) {
    

    achow101 commented at 2:52 am on May 31, 2023:
    Done as suggested.
  198. in src/wallet/salvage.cpp:209 in e9d974b7f4 outdated
    210+            fReadOK = LoadEncryptionKey(&dummyWallet, ssKey, ssValue, strErr);
    211+        } else if (strType == DBKeys::HDCHAIN) {
    212+            fReadOK = LoadHDChain(&dummyWallet, ssValue);
    213+        } else {
    214+            // This is a bug
    215+            CHECK_NONFATAL(false);
    


    ryanofsky commented at 7:58 pm on May 30, 2023:

    In commit “salvage: Remove use of ReadKeyValue in salvage” (e9d974b7f4136c04f35091438f5e78e17d77299b)

    It’s unnecessarily complicated to call KeyFilter, then repeat the same filtering and raise an error if the filtering is different. Would suggest just getting rid of the KeyFilter function here:

     0--- a/src/wallet/salvage.cpp
     1+++ b/src/wallet/salvage.cpp
     2@@ -19,11 +19,6 @@ static const char *HEADER_END = "HEADER=END";
     3 static const char *DATA_END = "DATA=END";
     4 typedef std::pair<std::vector<unsigned char>, std::vector<unsigned char> > KeyValPair;
     5 
     6-static bool KeyFilter(const std::string& type)
     7-{
     8-    return WalletBatch::IsKeyType(type) || type == DBKeys::HDCHAIN;
     9-}
    10-
    11 class DummyCursor : public DatabaseCursor
    12 {
    13     Status Next(DataStream& key, DataStream& value) override { return Status::FAIL; }
    14@@ -192,9 +187,6 @@ bool RecoverDatabaseFile(const ArgsManager& args, const fs::path& file_path, bil
    15 
    16         // We only care about KEY, MASTER_KEY, CRYPTED_KEY, and HDCHAIN types
    17         ssKey >> strType;
    18-        if (!KeyFilter(strType)) {
    19-            continue;
    20-        }
    21         bool fReadOK = false;
    22         if (strType == DBKeys::KEY) {
    23             fReadOK = LoadKey(&dummyWallet, ssKey, ssValue, strErr);
    24@@ -205,8 +197,6 @@ bool RecoverDatabaseFile(const ArgsManager& args, const fs::path& file_path, bil
    25         } else if (strType == DBKeys::HDCHAIN) {
    26             fReadOK = LoadHDChain(&dummyWallet, ssValue);
    27         } else {
    28-            // This is a bug
    29-            CHECK_NONFATAL(false);
    30             continue;
    31         }
    32 
    33--- a/src/wallet/walletdb.cpp
    34+++ b/src/wallet/walletdb.cpp
    35@@ -823,12 +823,6 @@ ReadKeyValue(CWallet* pwallet, DataStream& ssKey, CDataStream& ssValue,
    36     return true;
    37 }
    38 
    39-bool WalletBatch::IsKeyType(const std::string& strType)
    40-{
    41-    return (strType == DBKeys::KEY ||
    42-            strType == DBKeys::MASTER_KEY || strType == DBKeys::CRYPTED_KEY);
    43-}
    44-
    45 static DBErrors LoadMinVersion(CWallet* pwallet, DatabaseBatch& batch) EXCLUSIVE_LOCKS_REQUIRED(pwallet->cs_wallet)
    46 {
    47     AssertLockHeld(pwallet->cs_wallet);
    48@@ -916,7 +910,10 @@ DBErrors WalletBatch::LoadWallet(CWallet* pwallet)
    49                 }
    50                 // losing keys is considered a catastrophic error, anything else
    51                 // we assume the user can live with:
    52-                if (IsKeyType(strType) || strType == DBKeys::DEFAULTKEY) {
    53+                if (strType == DBKeys::KEY ||
    54+                    strType == DBKeys::MASTER_KEY ||
    55+                    strType == DBKeys::CRYPTED_KEY||
    56+                    strType == DBKeys::DEFAULTKEY) {
    57                     result = DBErrors::CORRUPT;
    58                 } else if (wss.tx_corrupt) {
    59                     pwallet->WalletLogPrintf("Error: Corrupt transaction found. This can be fixed by removing transactions from wallet and rescanning.\n");
    60--- a/src/wallet/walletdb.h
    61+++ b/src/wallet/walletdb.h
    62@@ -276,8 +276,6 @@ public:
    63     DBErrors LoadWallet(CWallet* pwallet);
    64     DBErrors FindWalletTxHashes(std::vector<uint256>& tx_hashes);
    65     DBErrors ZapSelectTx(std::vector<uint256>& vHashIn, std::vector<uint256>& vHashOut);
    66-    /* Function to determine if a certain KV/key-type is a key (cryptographical key) type */
    67-    static bool IsKeyType(const std::string& strType);
    68 
    69     //! write the hdchain model (external chain child index counter)
    70     bool WriteHDChain(const CHDChain& chain);
    

    achow101 commented at 2:52 am on May 31, 2023:
    Done as suggested.
  199. in src/wallet/walletdb.cpp:1099 in 8a4362f9bc outdated
    1142@@ -1155,6 +1143,34 @@ static DBErrors LoadTxRecords(CWallet* pwallet, DatabaseBatch& batch, std::vecto
    1143     return DBErrors::LOAD_OK;
    1144 }
    1145 
    1146+static DBErrors LoadActiveSPKMs(CWallet* pwallet, DatabaseBatch& batch) EXCLUSIVE_LOCKS_REQUIRED(pwallet->cs_wallet)
    1147+{
    1148+    AssertLockHeld(pwallet->cs_wallet);
    1149+
    1150+    // Load spk records
    1151+    std::set<std::pair<OutputType, bool>> seen_spks;
    


    ryanofsky commented at 8:13 pm on May 30, 2023:

    In commit “walletdb: refactor active spkm loading” (8a4362f9bcf213164f43395faa164e50aab9363b)

    There’s still some dead code left behind this commit:

     0--- a/src/wallet/walletdb.cpp
     1+++ b/src/wallet/walletdb.cpp
     2@@ -300,8 +300,6 @@ bool WalletBatch::EraseLockedUTXO(const COutPoint& output)
     3 
     4 class CWalletScanState {
     5 public:
     6-    std::map<OutputType, uint256> m_active_external_spks;
     7-    std::map<OutputType, uint256> m_active_internal_spks;
     8 
     9     CWalletScanState() = default;
    10 };
    

    achow101 commented at 2:52 am on May 31, 2023:
    Done as suggested.
  200. ryanofsky approved
  201. ryanofsky commented at 8:58 pm on May 30, 2023: contributor

    Code review ACK 3c51e44380ba8041e5d6c4cb29b9b2c54fad0b4b

    Overall changes look good, but I think two things could be done to make this easier to review:

    1. The main thing would be making the “walletdb: refactor X loading” commits self contained and not leave behind dead code that needs to be cleaned up later. It’s hard to identify what code is being replaced in current commits when not all the code being replaced is actually removed.
    2. I think it might be good to move first 2 commits up to “wallet: Add GetPrefixCursor to DatabaseBatch” (422a8436089c844934903a61fcec6b7b93995c07) to a different PR. Those commits require a different kind of review than the rest of the PR, and I suspect there are probably wallet developers who’d be perfectly comfortable reviewing the rest of the PR but less comfortable acking the lower level bdb & sqlite code there.

    I left a bunch of review comments, but also posted a branch https://github.com/ryanofsky/bitcoin/commits/review.24914.9-edit with suggested changes to fix up the “refactor X loading commits”

  202. DrahtBot requested review from furszy on May 30, 2023
  203. DrahtBot added the label CI failed on May 31, 2023
  204. achow101 force-pushed on May 31, 2023
  205. achow101 force-pushed on May 31, 2023
  206. achow101 commented at 2:56 am on May 31, 2023: member
    1. The main thing would be making the “walletdb: refactor X loading” commits self contained and not leave behind dead code that needs to be cleaned up later. It’s hard to identify what code is being replaced in current commits when not all the code being replaced is actually removed.

    I’ve implemented your suggestions and some additional dead code cleanups.

    1. I think it might be good to move first 2 commits up to “wallet: Add GetPrefixCursor to DatabaseBatch” (422a843) to a different PR. Those commits require a different kind of review than the rest of the PR, and I suspect there are probably wallet developers who’d be perfectly comfortable reviewing the rest of the PR but less comfortable acking the lower level bdb & sqlite code there.

    #27790

  207. achow101 marked this as a draft on May 31, 2023
  208. achow101 force-pushed on May 31, 2023
  209. DrahtBot removed the label CI failed on Jun 1, 2023
  210. achow101 force-pushed on Jun 1, 2023
  211. fanquake referenced this in commit 7f2019755d on Jun 2, 2023
  212. DrahtBot added the label Needs rebase on Jun 2, 2023
  213. achow101 force-pushed on Jun 2, 2023
  214. DrahtBot removed the label Needs rebase on Jun 2, 2023
  215. DrahtBot added the label CI failed on Jun 2, 2023
  216. sidhujag referenced this in commit 1aa882af51 on Jun 2, 2023
  217. achow101 marked this as ready for review on Jun 7, 2023
  218. achow101 force-pushed on Jun 7, 2023
  219. walletdb: Refactor minversion loading
    Move minversion loading to its own function in WalletBatch
    01b35b55a1
  220. walletdb: Refactor wallet flags loading
    Move wallet flags loading to its own function in WalletBatch
    
    The return value is changed to be TOO_NEW rather than CORRUPT when
    unknown flags are found.
    52932c5adb
  221. achow101 force-pushed on Jun 8, 2023
  222. DrahtBot removed the label CI failed on Jun 8, 2023
  223. ryanofsky approved
  224. ryanofsky commented at 4:21 pm on June 13, 2023: contributor

    Code review ACK ce88124bf499ae9608a2607ccddc00df3d7439bc. This looks great and I would encourage others to review. It should be much simpler to review now. Basically none of the commits are adding new code anymore, just moving code from one place to another.

    Since my last review #24914#pullrequestreview-1442091917 there were a lot of internal changes to make commits self-contained, but the only changes to the final code were making LoadHDChain catch exceptions, dropping an unneeded new return DBErrors::CORRUPT; line, getting rid of unneeded KeyType and KeyFilter functions, and rebasing.

  225. fanquake requested review from josibake on Jun 14, 2023
  226. in src/wallet/walletdb.cpp:816 in 0506cc2314 outdated
    811+            if (!cursor) {
    812+                pwallet->WalletLogPrintf("Error getting database cursor for '%s' records\n", type);
    813+                return DBErrors::CORRUPT;
    814+            }
    815+
    816+            DatabaseCursor::Status status = cursor->Next(value, value);
    


    furszy commented at 6:37 pm on June 17, 2023:
    In 0506cc23: The key nor the value are used here but this should be (key, value) instead of (value, value).

    achow101 commented at 7:29 pm on June 17, 2023:
    oops, fixed
  227. furszy commented at 6:42 pm on June 17, 2023: member
    left one last comment, looking great otherwise.
  228. DrahtBot requested review from furszy on Jun 17, 2023
  229. achow101 force-pushed on Jun 17, 2023
  230. DrahtBot added the label CI failed on Jun 17, 2023
  231. DrahtBot removed the label CI failed on Jun 18, 2023
  232. in src/wallet/salvage.cpp:8 in c729f1143f outdated
    4@@ -5,6 +5,7 @@
    5 
    6 #include <streams.h>
    7 #include <util/fs.h>
    8+#include <util/check.h>
    


    furszy commented at 1:32 pm on June 18, 2023:
    In c729f114: Since you removed the CHECK_NONFATAL call, this include is no longer needed.

    achow101 commented at 8:52 pm on June 19, 2023:
    Done
  233. in src/wallet/walletdb.cpp:778 in c0545c6094 outdated
    906@@ -1008,6 +907,175 @@ static DBErrors LoadLegacyWalletRecords(CWallet* pwallet, DatabaseBatch& batch,
    907     return result;
    908 }
    909 
    910+template<typename... Args>
    911+static CDataStream PrefixStream(const Args&... args)
    912+{
    913+    CDataStream prefix(0, 0);
    914+    SerializeMany(prefix, args...);
    915+    return prefix;
    


    furszy commented at 1:54 pm on June 18, 2023:

    In c0545c6:

    As the prefix is part of the key, could use a DataStream here too.


    achow101 commented at 8:52 pm on June 19, 2023:
    Done
  234. furszy approved
  235. furszy commented at 2:03 pm on June 18, 2023: member

    ACK 27ed8898

    Left two non-blocking findings.

  236. DrahtBot requested review from ryanofsky on Jun 18, 2023
  237. in src/wallet/walletdb.cpp:323 in 3814b9a07a outdated
    319@@ -320,6 +320,72 @@ class CWalletScanState {
    320     CWalletScanState() = default;
    321 };
    322 
    323+bool LoadKey(CWallet* pwallet, DataStream& ssKey, CDataStream& ssValue, std::string& strErr)
    


    maflcko commented at 9:25 am on June 19, 2023:

    style nit in 3814b9a07aba1d1a1a0ef2c4bd470bcb25669db7:

    Only CTransaction uses the legacy serialization code in the wallet, so everything else can just use the slimmed-down version, aka DataStream:

     0diff --git a/src/wallet/walletdb.cpp b/src/wallet/walletdb.cpp
     1index a8728b46ce..3958211da8 100644
     2--- a/src/wallet/walletdb.cpp
     3+++ b/src/wallet/walletdb.cpp
     4@@ -320,7 +320,7 @@ public:
     5     CWalletScanState() = default;
     6 };
     7 
     8-bool LoadKey(CWallet* pwallet, DataStream& ssKey, CDataStream& ssValue, std::string& strErr)
     9+bool LoadKey(CWallet* pwallet, DataStream& ssKey, DataStream& ssValue, std::string& strErr)
    10 {
    11     LOCK(pwallet->cs_wallet);
    12     try {
    13diff --git a/src/wallet/walletdb.h b/src/wallet/walletdb.h
    14index 55a36c4722..4228d428fa 100644
    15--- a/src/wallet/walletdb.h
    16+++ b/src/wallet/walletdb.h
    17@@ -306,7 +306,7 @@ using KeyFilterFn = std::function<bool(const std::string&)>;
    18 //! Unserialize a given Key-Value pair and load it into the wallet
    19 bool ReadKeyValue(CWallet* pwallet, DataStream& ssKey, CDataStream& ssValue, std::string& strType, std::string& strErr, const KeyFilterFn& filter_fn = nullptr);
    20 
    21-bool LoadKey(CWallet* pwallet, DataStream& ssKey, CDataStream& ssValue, std::string& strErr);
    22+bool LoadKey(CWallet* pwallet, DataStream& ssKey, DataStream& ssValue, std::string& strErr);
    23 } // namespace wallet
    24 
    25 #endif // BITCOIN_WALLET_WALLETDB_H
    

    achow101 commented at 5:03 pm on June 19, 2023:
    Done
  238. in src/wallet/walletdb.cpp:389 in 39a7f25269 outdated
    385@@ -386,6 +386,45 @@ bool LoadKey(CWallet* pwallet, DataStream& ssKey, CDataStream& ssValue, std::str
    386     return true;
    387 }
    388 
    389+bool LoadCryptedKey(CWallet* pwallet, DataStream& ssKey, CDataStream& ssValue, std::string& strErr)
    


    maflcko commented at 9:40 am on June 19, 2023:

    Style nit in 39a7f252699680cd1588be944ef15bff9e5a8157:

    Same?


    achow101 commented at 5:04 pm on June 19, 2023:
    Done
  239. in src/wallet/walletdb.cpp:428 in 3f0ab45a08 outdated
    424@@ -425,6 +425,33 @@ bool LoadCryptedKey(CWallet* pwallet, DataStream& ssKey, CDataStream& ssValue, s
    425     return true;
    426 }
    427 
    428+bool LoadEncryptionKey(CWallet* pwallet, DataStream& ssKey, CDataStream& ssValue, std::string& strErr)
    


    maflcko commented at 9:41 am on June 19, 2023:

    style nit in 3f0ab45a08ce33518e5bc275fce5c9f0cb713543:

    Same?


    achow101 commented at 5:04 pm on June 19, 2023:
    Done
  240. in src/wallet/walletdb.cpp:455 in b634ac2b8a outdated
    451@@ -452,6 +452,22 @@ bool LoadEncryptionKey(CWallet* pwallet, DataStream& ssKey, CDataStream& ssValue
    452     return true;
    453 }
    454 
    455+bool LoadHDChain(CWallet* pwallet, CDataStream& ssValue, std::string& strErr)
    


    maflcko commented at 9:42 am on June 19, 2023:

    style nit in b634ac2b8a0a9d33afd32535630a693ccf17f49a:

    Same?


    achow101 commented at 5:04 pm on June 19, 2023:
    Done
  241. in src/wallet/salvage.cpp:185 in c729f1143f outdated
    183@@ -188,15 +184,22 @@ bool RecoverDatabaseFile(const ArgsManager& args, const fs::path& file_path, bil
    184         DataStream ssKey{row.first};
    185         CDataStream ssValue(row.second, SER_DISK, CLIENT_VERSION);
    


    maflcko commented at 9:45 am on June 19, 2023:

    unrelated in c729f1143f80c875f110ed5b5b8969ab94f9df24:

    Can probably switch to DataStream here, because CTransaction are not read by salvage and the slimmed-down DataStream is enough?


    achow101 commented at 5:04 pm on June 19, 2023:
    Done
  242. in src/wallet/walletdb.cpp:477 in 1a3e49106e outdated
    472@@ -477,10 +473,8 @@ ReadKeyValue(CWallet* pwallet, DataStream& ssKey, CDataStream& ssValue,
    473         // Taking advantage of the fact that pair serialization
    474         // is just the two items serialized one after the other
    475         ssKey >> strType;
    476-        // Legacy entries in descriptor wallets are not allowed, abort immediately
    477         if (pwallet->IsWalletFlagSet(WALLET_FLAG_DESCRIPTORS) && DBKeys::LEGACY_TYPES.count(strType) > 0) {
    478-            wss.unexpected_legacy_entry = true;
    479-            return false;
    480+            return true;
    


    maflcko commented at 10:41 am on June 19, 2023:
    nit in 1a3e49106ec42506f435d604addf56978ac93765: Maybe add a comment that this was already checked in LoadLegacyWalletRecords and is dead code?

    maflcko commented at 12:14 pm on June 19, 2023:
    Nvm, this can be closed, because the code is removed anyway in a later commit
  243. in src/wallet/walletdb.cpp:856 in 1a3e49106e outdated
    851+        CScript script;
    852+        value >> script;
    853+        if (!pwallet->GetOrCreateLegacyScriptPubKeyMan()->LoadCScript(script))
    854+        {
    855+            err = "Error reading wallet database: LegacyScriptPubKeyMan::LoadCScript failed\n";
    856+            return DBErrors::CORRUPT;
    


    maflcko commented at 11:07 am on June 19, 2023:
    in 1a3e49106ec42506f435d604addf56978ac93765: Any reason to change this from NONCRITICAL_ERROR to CORRUPT?

    achow101 commented at 5:04 pm on June 19, 2023:
    No, changed back
  244. in src/wallet/walletdb.cpp:855 in 1a3e49106e outdated
    850+        key >> hash;
    851+        CScript script;
    852+        value >> script;
    853+        if (!pwallet->GetOrCreateLegacyScriptPubKeyMan()->LoadCScript(script))
    854+        {
    855+            err = "Error reading wallet database: LegacyScriptPubKeyMan::LoadCScript failed\n";
    


    maflcko commented at 11:15 am on June 19, 2023:
    in https://github.com/bitcoin/bitcoin/commit/1a3e49106ec42506f435d604addf56978ac93765: Any reason to change this from strErr to err when the code is only supposed to be moved? Also, any reason to add two \n\n, when one is sufficient and was used previously?

    achow101 commented at 5:04 pm on June 19, 2023:
    No, changed back
  245. in src/wallet/walletdb.cpp:896 in 1a3e49106e outdated
    891+                    // We have a key origin, so pull it from its path vector
    892+                    path = keyMeta.key_origin.path;
    893+                } else {
    894+                    // No key origin, have to parse the string
    895+                    if (!ParseHDKeypath(keyMeta.hdKeypath, path)) {
    896+                        err = "Error reading wallet database: keymeta with invalid HD keypath\n";
    


    maflcko commented at 11:15 am on June 19, 2023:
    in https://github.com/bitcoin/bitcoin/commit/1a3e49106ec42506f435d604addf56978ac93765: Any reason to change this from strErr to err when the code is only supposed to be moved? Also, any reason to add two \n\n, when one is sufficient and was used previously?

    achow101 commented at 5:04 pm on June 19, 2023:
    No, changed back
  246. in src/wallet/walletdb.cpp:906 in 1a3e49106e outdated
    901+                // Extract the index and internal from the path
    902+                // Path string is m/0'/k'/i'
    903+                // Path vector is [0', k', i'] (but as ints OR'd with the hardened bit
    904+                // k == 0 for external, 1 for internal. i is the index
    905+                if (path.size() != 3) {
    906+                    err = "Error reading wallet database: keymeta found with unexpected path\n";
    


    maflcko commented at 11:15 am on June 19, 2023:
    in https://github.com/bitcoin/bitcoin/commit/1a3e49106ec42506f435d604addf56978ac93765: Any reason to change this from strErr to err when the code is only supposed to be moved? Also, any reason to add two \n\n, when one is sufficient and was used previously?

    achow101 commented at 5:04 pm on June 19, 2023:
    No, changed back
  247. in src/wallet/walletdb.cpp:910 in 1a3e49106e outdated
    905+                if (path.size() != 3) {
    906+                    err = "Error reading wallet database: keymeta found with unexpected path\n";
    907+                    return DBErrors::NONCRITICAL_ERROR;
    908+                }
    909+                if (path[0] != 0x80000000) {
    910+                    err = strprintf("Unexpected path index of 0x%08x (expected 0x80000000) for the element at index 0\n", path[0]);
    


    maflcko commented at 11:15 am on June 19, 2023:
    in https://github.com/bitcoin/bitcoin/commit/1a3e49106ec42506f435d604addf56978ac93765: Any reason to change this from strErr to err when the code is only supposed to be moved? Also, any reason to add two \n\n, when one is sufficient and was used previously?

    achow101 commented at 5:04 pm on June 19, 2023:
    No, changed back
  248. in src/wallet/walletdb.cpp:914 in 1a3e49106e outdated
    909+                if (path[0] != 0x80000000) {
    910+                    err = strprintf("Unexpected path index of 0x%08x (expected 0x80000000) for the element at index 0\n", path[0]);
    911+                    return DBErrors::NONCRITICAL_ERROR;
    912+                }
    913+                if (path[1] != 0x80000000 && path[1] != (1 | 0x80000000)) {
    914+                    err = strprintf("Unexpected path index of 0x%08x (expected 0x80000000 or 0x80000001) for the element at index 1\n", path[1]);
    


    maflcko commented at 11:15 am on June 19, 2023:
    in https://github.com/bitcoin/bitcoin/commit/1a3e49106ec42506f435d604addf56978ac93765: Any reason to change this from strErr to err when the code is only supposed to be moved? Also, any reason to add two \n\n, when one is sufficient and was used previously?

    achow101 commented at 5:04 pm on June 19, 2023:
    No, changed back
  249. in src/wallet/walletdb.cpp:918 in 1a3e49106e outdated
    913+                if (path[1] != 0x80000000 && path[1] != (1 | 0x80000000)) {
    914+                    err = strprintf("Unexpected path index of 0x%08x (expected 0x80000000 or 0x80000001) for the element at index 1\n", path[1]);
    915+                    return DBErrors::NONCRITICAL_ERROR;
    916+                }
    917+                if ((path[2] & 0x80000000) == 0) {
    918+                    err = strprintf("Unexpected path index of 0x%08x (expected to be greater than or equal to 0x80000000)\n", path[2]);
    


    maflcko commented at 11:16 am on June 19, 2023:
    in https://github.com/bitcoin/bitcoin/commit/1a3e49106ec42506f435d604addf56978ac93765: Any reason to change this from strErr to err when the code is only supposed to be moved? Also, any reason to add two \n\n, when one is sufficient and was used previously?

    achow101 commented at 5:04 pm on June 19, 2023:
    No, changed back
  250. in src/wallet/walletdb.cpp:1014 in c0545c6094 outdated
    1009+            key >> desc_id;
    1010+            assert(desc_id == id);
    1011+            key >> pubkey;
    1012+            if (!pubkey.IsValid())
    1013+            {
    1014+                err = "Error reading wallet database: descriptor unencrypted key CPubKey corrupt\n";
    


    maflcko commented at 11:41 am on June 19, 2023:
    c0545c60944a32b7842e28d8d4ab35db4a5c40a3: Double newline?

    achow101 commented at 5:05 pm on June 19, 2023:
    Done here and elsewhere.
  251. in src/wallet/walletdb.cpp:1032 in c0545c6094 outdated
    1027+            to_hash.insert(to_hash.end(), pubkey.begin(), pubkey.end());
    1028+            to_hash.insert(to_hash.end(), pkey.begin(), pkey.end());
    1029+
    1030+            if (Hash(to_hash) != hash)
    1031+            {
    1032+                err = "Error reading wallet database: descriptor unencrypted key CPubKey/CPrivKey corrupt\n";
    


    maflcko commented at 11:41 am on June 19, 2023:
    c0545c60944a32b7842e28d8d4ab35db4a5c40a3: Double newline?
  252. in src/wallet/walletdb.cpp:1038 in c0545c6094 outdated
    1033+                return DBErrors::CORRUPT;
    1034+            }
    1035+
    1036+            if (!privkey.Load(pkey, pubkey, true))
    1037+            {
    1038+                err = "Error reading wallet database: descriptor unencrypted key CPrivKey corrupt\n";
    


    maflcko commented at 11:41 am on June 19, 2023:
    c0545c60944a32b7842e28d8d4ab35db4a5c40a3: Double newline?
  253. in src/wallet/walletdb.cpp:565 in 1a3e49106e outdated
    830+    }
    831+
    832+    // Load unencrypted keys
    833+    LoadResult key_res = LoadRecords(pwallet, batch, DBKeys::KEY,
    834+        [] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) {
    835+        return LoadKey(pwallet, key, value, err) ? DBErrors::LOAD_OK : DBErrors::CORRUPT;
    


    maflcko commented at 11:51 am on June 19, 2023:
    1a3e49106ec42506f435d604addf56978ac93765: Should probably mention in the commit description that a std::exception in this function will now lead to early-return, where previously it was caught?

    maflcko commented at 11:52 am on June 19, 2023:
    (Same for all other functions with the same issue)

    achow101 commented at 5:05 pm on June 19, 2023:
    Done

    maflcko commented at 10:39 am on June 20, 2023:
    Sorry, I think my suggestion was wrong. Behaviour shouldn’t be changed because previously exceptions were caught outside the try { while(1) ReadKeyValue(); } catch (...), so an exception would result in early return previously. The same is true after, because the while (1) ReadyKeyValue() is simply replaced by LoadRecords, which remains in the same try-catch.
  254. in src/wallet/walletdb.cpp:1073 in 3c616521b4 outdated
    1068+        auto fill_wtx = [&](CWalletTx& wtx, bool new_tx) {
    1069+            if(!new_tx) {
    1070+                // There's some corruption here since the tx we just tried to load was already in the wallet.
    1071+                // We don't consider this type of corruption critical, and can fix it by removing tx data and
    1072+                // rescanning.
    1073+                err = "Error: Corrupt transaction found. This can be fixed by removing transactions from wallet and rescanning.\n";
    


    maflcko commented at 11:55 am on June 19, 2023:
    3c616521b4f554cd391dfdf4be1be7a76f128a92: double \n?
  255. in src/wallet/walletdb.cpp:510 in 1a3e49106e outdated
    785+        }
    786+        std::string type;
    787+        ssKey >> type;
    788+        assert(type == key);
    789+        if ((result.m_result = std::max(result.m_result, load_func(pwallet, ssKey, ssValue, result.m_error))) == DBErrors::CORRUPT) {
    790+            pwallet->WalletLogPrintf("%s\n", result.m_error);
    


    maflcko commented at 12:04 pm on June 19, 2023:
    1a3e49106ec42506f435d604addf56978ac93765: Given that m_error is ignored if the return value is not CORRUPT, maybe rename the error string in all load functions to err_corrupt?

    achow101 commented at 5:06 pm on June 19, 2023:
    Renamed

    furszy commented at 0:14 am on June 20, 2023:

    1a3e491: Given that m_error is ignored if the return value is not CORRUPT, maybe rename the error string in all load functions to err_corrupt?

    That is actually a bug. We should continue logging non-critical errors too. Which, as far as can see, are only for legacy records: KEYMETA and CSCRIPT.

    Made a quick test to verify it. It passes on master, and fails on this branch.

     0+#include <test/util/logging.h>
     1 #include <wallet/test/util.h>
     2 #include <wallet/wallet.h>
     3 #include <test/util/setup_common.h>
     4@@ -178,5 +179,38 @@
     5     }
     6 }
     7 
     8+BOOST_FIXTURE_TEST_CASE(wallet_load_keymeta_noncritical_error, TestingSetup)
     9+{
    10+    // Verify that non-critical errors al logged too.
    11+    bool found = false;
    12+    DebugLogHelper logHelper("Error reading wallet database: keymeta with invalid HD keypath", [&](const std::string* s) {
    13+        found = true;
    14+        return false;
    15+    });
    16+
    17+    std::unique_ptr<WalletDatabase> database = CreateMockableWalletDatabase();
    18+    {
    19+        // Write key meta with invalid HD keypath
    20+        WalletBatch batch(*database, false);
    21+
    22+        CKey key;
    23+        key.MakeNewKey(true);
    24+        CPubKey pubkey = key.GetPubKey();
    25+
    26+        CKeyMetadata meta;
    27+        meta.nVersion = CKeyMetadata::VERSION_WITH_HDDATA;
    28+        meta.hd_seed_id = pubkey.GetID();
    29+        meta.hdKeypath = "a";
    30+        BOOST_CHECK(batch.WriteKeyMetadata(meta, pubkey, /*overwrite=*/true));
    31+    }
    32+
    33+    {
    34+        // Now try to load the wallet and verify the non-critical error and the logging
    35+        const std::shared_ptr<CWallet> wallet(new CWallet(m_node.chain.get(), "", std::move(database)));
    36+        BOOST_CHECK_EQUAL(wallet->LoadWallet(), DBErrors::NONCRITICAL_ERROR);
    37+        BOOST_CHECK(found);
    38+    }
    39+}
    40+
    41 BOOST_AUTO_TEST_SUITE_END()
    42 } // namespace wallet
    

    maflcko commented at 8:18 am on June 20, 2023:

    Yeah, thanks, just realized that my suggestion was wrong, too.

    There was a

    0            if (!strErr.empty())
    1                pwallet->WalletLogPrintf("%s\n", strErr);
    

    in the old code


    achow101 commented at 5:22 pm on June 20, 2023:
    Changed this to check for != LOAD_OK rather than just CORRUPT.
  256. in src/wallet/walletdb.cpp:1141 in 72741608e0 outdated
    1136+            value >> id;
    1137+
    1138+            bool internal = spk_key == DBKeys::ACTIVEINTERNALSPK;
    1139+            auto [it, insert] = seen_spks.emplace(static_cast<OutputType>(output_type), internal);
    1140+            if (!insert) {
    1141+                err = "Multiple ScriptpubKeyMans specified for a single type\n";
    


    maflcko commented at 12:07 pm on June 19, 2023:
    72741608e0f5fa155374d8e948712b0ac6462b1d: double \n?
  257. maflcko approved
  258. maflcko commented at 12:13 pm on June 19, 2023: member

    ACK 27ed889890ceb29f62dc1a4082647922968012d4 👥

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: ACK 27ed889890ceb29f62dc1a4082647922968012d4 👥
    3QwjXzK3AhLotpx6ik+g9hYFHBUWVMOMR0l2YFXJcCdLFjNY6gmwvdk9YeL2ZfkI/3yG8E0WQl1qNz2gw+5S9Bw==
    
  259. walletdb: Refactor key reading and loading to its own function 7be10adff3
  260. walletdb: Refactor crypted key loading to its own function 3ccde4599b
  261. walletdb: Refactor encryption key loading to its own function 72c2a54ebb
  262. walletdb: Refactor hd chain loading to its own function ad779e9ece
  263. achow101 force-pushed on Jun 19, 2023
  264. achow101 force-pushed on Jun 19, 2023
  265. salvage: Remove use of ReadKeyValue in salvage
    To prepare to remove ReadKeyValue, change salvage to not use it
    9e077d9b42
  266. achow101 force-pushed on Jun 19, 2023
  267. furszy commented at 0:19 am on June 20, 2023: member
    Left another finding in #24914 (review). Should be the last thing.
  268. in src/wallet/walletdb.cpp:918 in 4336764a32 outdated
    913+                if (path[1] != 0x80000000 && path[1] != (1 | 0x80000000)) {
    914+                    strErr = strprintf("Unexpected path index of 0x%08x (expected 0x80000000 or 0x80000001) for the element at index 1\n", path[1]);
    915+                    return DBErrors::NONCRITICAL_ERROR;
    916+                }
    917+                if ((path[2] & 0x80000000) == 0) {
    918+                    strErr = strprintf("Unexpected path index of 0x%08x (expected to be greater than or equal to 0x80000000)\n", path[2]);
    


    maflcko commented at 8:24 am on June 20, 2023:
    nit in 4336764a320352a4b06698edca3f5841c0851b2c: double \n for the 4 strErr here?

    achow101 commented at 5:22 pm on June 20, 2023:
    Done
  269. in src/wallet/walletdb.cpp:883 in 643859ecc2 outdated
    1033+                return DBErrors::CORRUPT;
    1034+            }
    1035+
    1036+            if (!privkey.Load(pkey, pubkey, true))
    1037+            {
    1038+                strErr = "Error reading wallet database: descriptor unencrypted key CPrivKey corrupt\n";
    


    maflcko commented at 8:31 am on June 20, 2023:
    643859ecc289885ed5f0ba67608aa69691724f46: double \n?
  270. maflcko approved
  271. maflcko commented at 8:34 am on June 20, 2023: member

    Some nits below and here: #24914 (review)

    re-ACK 54916e3fda3172ce3bb81d4213266a58a67c59f2 🌞

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: re-ACK 54916e3fda3172ce3bb81d4213266a58a67c59f2 🌞
    3AW3a1aYFYStKNpUhe+blkBm9rJGhGF7DarXiFtWkcErw+w5nBB72npBzZtxPZ7qNy62ONR0McfKlJgiJ0YhdAg==
    
  272. DrahtBot requested review from furszy on Jun 20, 2023
  273. in src/wallet/walletdb.cpp:939 in 643859ecc2 outdated
    934+        } catch (const std::ios_base::failure&) {
    935+            err = strprintf("Error: Unrecognized descriptor found in wallet %s. ", pwallet->GetName());
    936+            err += (last_client > CLIENT_VERSION) ? "The wallet might had been created on a newer version. " :
    937+                    "The database might be corrupted or the software version is not compatible with one of your wallet descriptors. ";
    938+            err += "Please try running the latest software version";
    939+            pwallet->WalletLogPrintf("%s\n", err);
    


    maflcko commented at 8:45 am on June 20, 2023:
    643859ecc289885ed5f0ba67608aa69691724f46: Depending on how you fix the logging feedback in the other comment, this will either log twice, or you’ll have to avoid setting err. (Which should have been called strErr, if you want it to be set and be move-only.)

    achow101 commented at 5:23 pm on June 20, 2023:
    Removed this log to let LoadRecords log it.
  274. in src/wallet/walletdb.cpp:326 in 7be10adff3 outdated
    319@@ -320,6 +320,72 @@ class CWalletScanState {
    320     CWalletScanState() = default;
    321 };
    322 
    323+bool LoadKey(CWallet* pwallet, DataStream& ssKey, DataStream& ssValue, std::string& strErr)
    324+{
    325+    LOCK(pwallet->cs_wallet);
    326+    try {
    


    maflcko commented at 10:49 am on June 20, 2023:
    in 7be10adff36c0dc49ae56ac571bb033cba7a565b: Can you explain why it is ok to put the try-catch here, where it previously wasn’t? It looks like this will cause ReadKeyValue to not break out of the while-loop and instead continue?

    ryanofsky commented at 2:18 pm on June 20, 2023:

    re: #24914 (review)

    in 7be10ad: Can you explain why it is ok to put the try-catch here, where it previously wasn’t? It looks like this will cause ReadKeyValue to not break out of the while-loop and instead continue?

    IIUC, it is ok to catch this exception here because the same exception was also caught previously at the bottom of the ReadKeyValue function here:

    https://github.com/bitcoin/bitcoin/blob/7be10adff36c0dc49ae56ac571bb033cba7a565b/src/wallet/walletdb.cpp#L781

    and in both cases the exception handler just cause the ReadKeyValue function to just return false. There should be no case when an exception reading one key would cause ReadKeyValue function to throw and break out of the while loop in WalletBatch::LoadWallet. The catch (...) line around the while loop can catch exceptions, but not ones from ReadKeyValue


    maflcko commented at 5:53 pm on June 21, 2023:
    Ok, I see. Though, that probably means that the lambdas which were created from move-only code from ReadKeyValue are now no longer covered by the catch(const std::exception&) and catch(...) in ReadKeyValue?

    maflcko commented at 6:12 pm on June 21, 2023:
    This has a bunch of consequences, for example a corrupt address book entry will now result in Wallet corrupted and the error string All keys read correctly, but transaction data or address book entries might be missing or incorrect. is now incorrect, or at least can no longer be hit with a corrupt address book entry.

    achow101 commented at 6:23 pm on June 21, 2023:

    This has a bunch of consequences, for example a corrupt address book entry will now result in Wallet corrupted and the error string All keys read correctly, but transaction data or address book entries might be missing or incorrect. is now incorrect, or at least can no longer be hit with a corrupt address book entry.

    I think it’s preferable to treat all corruption errors as critical and fail to load the wallet. We can then loosen these over time. I think that would be better and easier than trying to reverse engineer when and what errors we would previously return.


    ryanofsky commented at 6:36 pm on June 21, 2023:

    I think it’s preferable to treat all corruption errors as critical and fail to load the wallet. We can then loosen these over time.

    This sounds good, but it would be good to say this explicitly in a code comment so it is clear what code is trying to do. It would also be good to update the error string if it is no longer accurate, and to describe in commit messages what behavior is changing, so we can know what changes are intended.


    ryanofsky commented at 6:49 pm on June 21, 2023:

    re: #24914 (review)

    This has a bunch of consequences, for example a corrupt address book entry will now result in Wallet corrupted […]

    Great catch! I didn’t realize that when I was reviewing.

    I think it actually would be preferable to just keep old behavior if that’s easily possible, so it doesn’t prevent loading wallets which previously just triggered warnings. But I can also see the case for being more strict. I mainly think if behavior is going to change, intent should clearly be documented in the code, and commit message should mention the change. Current commit eddfbcab498a8af200e006aef23fa43ad7c712cb “walletdb: refactor address book loading” description makes it seem like there is no change


    achow101 commented at 7:00 pm on June 21, 2023:
    Changed a few commit messages to mention the change of behavior in exception handling. Also added a comment at the catch (...) in LoadWallet, and a commit that updates the non-critical errors string to better represent the kinds of things that can hit it.

    ryanofsky commented at 7:12 pm on June 21, 2023:
    Would it also make sense to add release notes to say that some wallets which previously had corrupt records won’t be loaded by the new release? I could imagine panicking if a wallet that previously loaded with warnings no longer loaded, so release notes could clarify or say what to do. This is assuming affected wallets were usable before though..

    achow101 commented at 7:32 pm on June 21, 2023:

    Would it also make sense to add release notes to say that some wallets which previously had corrupt records won’t be loaded by the new release? I could imagine panicking if a wallet that previously loaded with warnings no longer loaded, so release notes could clarify or say what to do. This is assuming affected wallets were usable before though..

    Perhaps, but enumerating what failures result in which error messages is looking to be quite a non-trivial task for me. Looking at it more closely, I think that actually there’s a lot of things that we would have said are non-critical errors but really should be critical, such as corrupted watchonly scripts and keys. These are things that would most definitely result in incorrect balances, but I think we would have allowed loading the wallet anyways. I think the wallet corruption error is a good enough message that people will open issues if they run into it, but I don’t think we’ll really see any of those since I think several of these should be causing actual other problems.


    ryanofsky commented at 8:19 pm on June 21, 2023:

    Perhaps, but enumerating what failures result in which error messages is looking to be quite a non-trivial task for me

    That’s understandable. I just meant a general release note like “Wallet database loading has changed in this new release, and wallets that could previously be loaded but had some corrupt records no may no longer load. If this happens it is recommended to load the wallet in the previous version of the software and import data into a new wallet, or report an issue.” Idea would be to make it clear this is behavior we’re aware of and not something to panic over.


    achow101 commented at 8:33 pm on June 21, 2023:
    Added a release note.
  275. in src/wallet/walletdb.cpp:392 in 3ccde4599b outdated
    385@@ -386,6 +386,45 @@ bool LoadKey(CWallet* pwallet, DataStream& ssKey, DataStream& ssValue, std::stri
    386     return true;
    387 }
    388 
    389+bool LoadCryptedKey(CWallet* pwallet, DataStream& ssKey, DataStream& ssValue, std::string& strErr)
    390+{
    391+    LOCK(pwallet->cs_wallet);
    392+    try {
    


    maflcko commented at 10:50 am on June 20, 2023:
    3ccde4599b5150577400c4fa9029f4146617f751:Same
  276. in src/wallet/walletdb.cpp:431 in 72c2a54ebb outdated
    424@@ -425,6 +425,33 @@ bool LoadCryptedKey(CWallet* pwallet, DataStream& ssKey, DataStream& ssValue, st
    425     return true;
    426 }
    427 
    428+bool LoadEncryptionKey(CWallet* pwallet, DataStream& ssKey, DataStream& ssValue, std::string& strErr)
    429+{
    430+    LOCK(pwallet->cs_wallet);
    431+    try {
    


    maflcko commented at 10:50 am on June 20, 2023:
    72c2a54ebb99fa3d91d7d15bd8a38a8d16e0ea6c:Same?
  277. in src/wallet/walletdb.cpp:458 in ad779e9ece outdated
    451@@ -452,6 +452,22 @@ bool LoadEncryptionKey(CWallet* pwallet, DataStream& ssKey, DataStream& ssValue,
    452     return true;
    453 }
    454 
    455+bool LoadHDChain(CWallet* pwallet, DataStream& ssValue, std::string& strErr)
    456+{
    457+    LOCK(pwallet->cs_wallet);
    458+    try {
    


  278. achow101 force-pushed on Jun 20, 2023
  279. achow101 force-pushed on Jun 20, 2023
  280. in src/wallet/walletdb.cpp:751 in 80ccced2b6 outdated
    795+
    796+    // "wkey" records are unsupported, if we see any, throw an error
    797+    LoadResult wkey_res = LoadRecords(pwallet, batch, DBKeys::OLD_KEY,
    798+        [] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) {
    799+        pwallet->WalletLogPrintf("Found unsupported 'wkey' record, try loading with version 0.18\n");
    800+        return DBErrors::LOAD_FAIL;
    


    furszy commented at 12:19 pm on June 21, 2023:

    nit: could remove the logging and let LoadRecords log it:

    0err = "Found unsupported 'wkey' record, try loading with version 0.18";
    1return DBErrors::LOAD_FAIL;
    

    achow101 commented at 7:40 pm on June 21, 2023:
    Done
  281. furszy approved
  282. furszy commented at 12:27 pm on June 21, 2023: member

    ACK 0c3c7f7a

    Since my last review: -> Log of noncritical errors ✓ -> Removal of extra \n

  283. DrahtBot requested review from maflcko on Jun 21, 2023
  284. maflcko commented at 5:54 pm on June 21, 2023: member

    re-ACK 0c3c7f7a7b03eb954d25ce0ad003792079385557 🍼

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: re-ACK 0c3c7f7a7b03eb954d25ce0ad003792079385557 🍼
    3yKiEOgf+6FtUTFcknjlAZmq1YmMbu+RWfrT7PfsLhX02t8YyQdcye8ALbnHvRC2Vmo8rNpuXqlTmMb40vSVhBw==
    
  285. DrahtBot removed review request from maflcko on Jun 21, 2023
  286. maflcko commented at 6:13 pm on June 21, 2023: member

    Not sure if anything needs to be done about the changed exception handling: #24914 (review)

    Maybe adjusting the error string(s) is enough?

  287. achow101 force-pushed on Jun 21, 2023
  288. achow101 force-pushed on Jun 21, 2023
  289. DrahtBot added the label CI failed on Jun 21, 2023
  290. achow101 force-pushed on Jun 21, 2023
  291. achow101 force-pushed on Jun 21, 2023
  292. DrahtBot removed the label CI failed on Jun 21, 2023
  293. in src/wallet/walletdb.cpp:510 in f0eea2749b outdated
    674+        }
    675+        std::string type;
    676+        ssKey >> type;
    677+        assert(type == key);
    678+        if ((result.m_result = std::max(result.m_result, load_func(pwallet, ssKey, ssValue, result.m_error))) != DBErrors::LOAD_OK) {
    679+            pwallet->WalletLogPrintf("%s\n", result.m_error);
    


    maflcko commented at 11:28 am on June 22, 2023:
    Pretty sure this will re-log the same message in a loop (for the number of records), if the first record logged?

    achow101 commented at 2:31 pm on June 22, 2023:
    Changed this to log only if the load_func is not LOAD_OK.
  294. in src/wallet/walletdb.cpp:596 in f0eea2749b outdated
    824+    result = std::max(result, script_res.m_result);
    825+
    826+    // Check whether rewrite is needed
    827+    if (ckey_res.m_records > 0) {
    828+        // Rewrite encrypted wallets of versions 0.4.0 and 0.5.0rc:
    829+        if (last_client == 40000 || last_client == 50000) return DBErrors::NEED_REWRITE;
    


    maflcko commented at 11:48 am on June 22, 2023:
    Looks like this just ignores if result is corrupt, no?

    achow101 commented at 2:31 pm on June 22, 2023:
    Added a check for != CORRUPT.

    maflcko commented at 2:48 pm on June 22, 2023:

    Hmm, more generally I wonder what the point is to continue further if corruption was detected? IIUC the wallet will fail to load anyway, so might as well return early at the earliest convenience? I am not really a fan of how some types of corruption are ignored/caught and the code continues and others lead to an early abort.

    Wild guess: What about throwing instead of using the CORRUPT return code?


    achow101 commented at 2:51 pm on June 22, 2023:
    I think the general principle was to try to detect as much corruption as possible before returning the error. But it would also make this a lot simpler to just abort on the first error.

    ryanofsky commented at 3:02 pm on June 22, 2023:

    In commit “walletdb: Refactor legacy wallet record loading into its own function” (9d58900d8a754721964e628f0c44344652f4bb80)

    I think the general principle was to try to detect as much corruption as possible before returning the error. But it would also make this a lot simpler to just abort on the first error.

    This is still giving NEED_REWRITE error higher priority over other errors besides CORRUPT. And it is adding an early return in the middle of a 211 line function that mostly accumulates errors instead of returning early. If this error needs to be handled differently than other errors and can’t be merged with std::max, I think there should a comment calling out the special case and early return. Otherwise better to make this more consistent and write this this as:

     0--- a/src/wallet/walletdb.cpp
     1+++ b/src/wallet/walletdb.cpp
     2@@ -865,9 +865,9 @@ static DBErrors LoadLegacyWalletRecords(CWallet* pwallet, DatabaseBatch& batch,
     3     result = std::max(result, script_res.m_result);
     4 
     5     // Check whether rewrite is needed
     6-    if (ckey_res.m_records > 0 && result != DBErrors::CORRUPT) {
     7+    if (ckey_res.m_records > 0) {
     8         // Rewrite encrypted wallets of versions 0.4.0 and 0.5.0rc:
     9-        if (last_client == 40000 || last_client == 50000) return DBErrors::NEED_REWRITE;
    10+        if (last_client == 40000 || last_client == 50000) result = std::max(result, DBErrors::NEED_REWRITE);
    11     }
    12 
    13     // Load keymeta
    

    achow101 commented at 4:43 pm on June 22, 2023:
    Hmm, I suppose the early return is unnecessary. Changed as suggested.
  295. in src/wallet/wallet.cpp:2932 in 5ada171675 outdated
    2928@@ -2929,7 +2929,7 @@ std::shared_ptr<CWallet> CWallet::Create(WalletContext& context, const std::stri
    2929         else if (nLoadWalletRet == DBErrors::NONCRITICAL_ERROR)
    2930         {
    2931             warnings.push_back(strprintf(_("Error reading %s! All keys read correctly, but transaction data"
    2932-                                           " or address book entries might be missing or incorrect."),
    2933+                                           " or address metadata may be missing or incorrect."),
    


    maflcko commented at 11:58 am on June 22, 2023:
    5ada171675: Should this be squashed into 6b46d41d7b

    achow101 commented at 2:31 pm on June 22, 2023:
    Done

    ryanofsky commented at 2:22 pm on June 23, 2023:

    In commit “walletdb: refactor address book loading” (53116fb2f982e88b713b5855dd9e1fa74d6635bb)

    re: #24914 (review)

    Could note change in behavior in commit message now that 5ada171675b5d424ec57df9f9681ac31d3ac0fae is squashed


    achow101 commented at 5:06 pm on June 23, 2023:
    Added to that commit message.
  296. maflcko commented at 12:00 pm on June 22, 2023: member

    f0eea2749bcc73cde0cabc6fee0b7981a7539cf4 📀

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: f0eea2749bcc73cde0cabc6fee0b7981a7539cf4 📀
    3g6grEJToJOQFdY5asSP/tSfzsPyxsg1LuOGhk77aeNGB1LRWvfDOeDcCZc4SMRjtpHnOlHutU/D9a3ENDnlCBw==
    
  297. achow101 force-pushed on Jun 22, 2023
  298. in src/wallet/walletdb.cpp:590 in 9d58900d8a outdated
    860+        return DBErrors::LOAD_OK;
    861+    });
    862+    if (script_res.m_result != DBErrors::LOAD_OK) {
    863+        return script_res.m_result;
    864+    }
    865+    result = std::max(result, script_res.m_result);
    


    ryanofsky commented at 3:08 pm on June 22, 2023:

    In commit “walletdb: Refactor legacy wallet record loading into its own function” (9d58900d8a754721964e628f0c44344652f4bb80)

    Early return seems unintended here? It is inconsistent with the way most other errors are handled in this function, and is makes the std::max call pointless, since the early return guarantees script_res result is LOAD_OK at that point.


    achow101 commented at 4:43 pm on June 22, 2023:
    Removed
  299. in src/wallet/walletdb.cpp:951 in 9d58900d8a outdated
    946+    // Set inactive chains
    947+    if (!hd_chains.empty()) {
    948+        LegacyScriptPubKeyMan* legacy_spkm = pwallet->GetLegacyScriptPubKeyMan();
    949+        if (!legacy_spkm) {
    950+            pwallet->WalletLogPrintf("Inactive HD Chains found but no Legacy ScriptPubKeyMan\n");
    951+            return DBErrors::CORRUPT;
    


    ryanofsky commented at 3:11 pm on June 22, 2023:

    In commit “walletdb: Refactor legacy wallet record loading into its own function” (9d58900d8a754721964e628f0c44344652f4bb80)

    Another early return that should maybe result = std::max(result, DBErrors::CORRUPT);. If not it would be good to have a code comment that explains why this error is different.


    achow101 commented at 4:44 pm on June 22, 2023:
    Changed
  300. ryanofsky commented at 3:27 pm on June 22, 2023: contributor

    This looks like is getting close to being read to be merged, and I’m going through the changes from the beginning again to start looking more closely at the error handling in light of the new comments.

    I do generally think it is better to let errors accumulate and return information about all of them, not just the first one. And I think it would be better to handle more exceptions closer to where they are thrown rather than at the end in a final catch(...). It seems like the PR takes a few steps backwards in these respects. But not in a major way, and not in any way that can’t be improved later.

    I’ll keep reviewing but for now just left some feedback below on LoadLegacyWalletRecords

  301. achow101 force-pushed on Jun 22, 2023
  302. achow101 commented at 4:45 pm on June 22, 2023: member

    I’ve gone through and changed all of the early returns to instead update result.

    However I’ve had to add an early return after LoadDescriptorRecords for UNKNOWN_DESCRIPTOR as otherwise LoadActiveSPKMs will assert when the wallet has unknown descriptors. There is a comment in the code explaining this.

  303. DrahtBot added the label CI failed on Jun 22, 2023
  304. in src/wallet/walletdb.cpp:1141 in 8405e9a0d1 outdated
    1136+    result = std::max(result, locked_utxo_res.m_result);
    1137+
    1138+    // Load orderposnext record
    1139+    batch.Read(DBKeys::ORDERPOSNEXT, pwallet->nOrderPosNext);
    1140+
    1141+    return DBErrors::LOAD_OK;
    


    furszy commented at 8:46 pm on June 22, 2023:

    In 8405e9a0:

    Need to return result, not LOAD_OK.


    achow101 commented at 0:30 am on June 23, 2023:
    Fixed.
  305. achow101 force-pushed on Jun 23, 2023
  306. in src/wallet/walletdb.cpp:1118 in c434b28ad8 outdated
    1112@@ -1043,6 +1113,12 @@ DBErrors WalletBatch::LoadWallet(CWallet* pwallet)
    1113         // Load legacy wallet keys
    1114         result = std::max(LoadLegacyWalletRecords(pwallet, *m_batch, last_client), result);
    1115 
    1116+        // Load descriptors
    1117+        result = std::max(LoadDescriptorWalletRecords(pwallet, *m_batch, last_client), result);
    1118+        // Early return if there are unknown descriptors. Later loading that may reference the unknown descriptor's ID
    


    ryanofsky commented at 12:04 pm on June 23, 2023:

    In commit “walletdb: Refactor descriptor wallet records loading” (c434b28ad859bc0d8240ebedd3d3bee83e394987)

    Just for my curiousity, what’s something that could be loaded after the LoadDescriptorWalletRecords function that could reference a descriptor id? Is this for future wallet functionality that builds on descriptors, or is this referring to something in current code?


    EDIT: There’s a review comment explaining this #24914 (comment) that says this is done so a LoadActiveSPKMs call that is added in a later commit doesn’t throw.

    In light of this, I’d think it’d be helpful to replace “Later loading that may reference the unknown descriptor” with “ACTIVEINTERNALSPK or ACTIVEEXTERNALSPK records that may reference the unknown descriptor”


    achow101 commented at 5:00 pm on June 23, 2023:
    Updated the comment.
  307. in doc/release-notes-24914.md:5 in 290bc96209 outdated
    0@@ -0,0 +1,7 @@
    1+Wallet
    2+------
    3+
    4+- Wallet loading has changed in this release. Wallets with some corrupted records that could be
    5+  previously loaded (with warnings) may no longer load. If this happens, it is recommended
    


    ryanofsky commented at 12:49 pm on June 23, 2023:

    In commit “doc: Add release note for wallet loading changes” (290bc962096ca109edd038fbe2f32322ffdabb7b)

    To make this concrete and put in the error in context, maybe add “For example, wallets with corrupt address book entries may no longer load. If this happens, it should be possible to recover by loading the wallet in a previous version of Bitcoin Core and importing the data into a new wallet. Also, you can report an issue to help improve the software and make wallet loading more robust in these cases.”


    achow101 commented at 5:00 pm on June 23, 2023:
    Updated.
  308. in src/wallet/walletdb.cpp:1119 in b26573ef95 outdated
    1114+        };
    1115+        if (!pwallet->LoadToWallet(hash, fill_wtx)) {
    1116+            if (corrupted_tx) {
    1117+                result = DBErrors::CORRUPT;
    1118+            } else {
    1119+                result = DBErrors::NEED_RESCAN;
    


    ryanofsky commented at 12:59 pm on June 23, 2023:

    In commit “walletdb: refactor tx loading” (b26573ef95891bb69f00c3aaa67fb4678a82c818)

    Since NEED_RESCAN is a pretty low priority error it seems like it would be safer and more consistent with other code to write result = std::max(result, DBErrors::NEED_RESCAN)


    achow101 commented at 5:00 pm on June 23, 2023:
    Done, although this shouldn’t have any effect on the error returned since result is not an accumulator here.
  309. in src/wallet/walletdb.cpp:1078 in b26573ef95 outdated
    1073+        // LoadToWallet call below creates a new CWalletTx that fill_wtx
    1074+        // callback fills with transaction metadata.
    1075+        auto fill_wtx = [&](CWalletTx& wtx, bool new_tx) {
    1076+            if(!new_tx) {
    1077+                // There's some corruption here since the tx we just tried to load was already in the wallet.
    1078+                // We don't consider this type of corruption critical, and can fix it by removing tx data and
    


    ryanofsky commented at 1:03 pm on June 23, 2023:

    In commit “walletdb: refactor tx loading” (b26573ef95891bb69f00c3aaa67fb4678a82c818)

    There’s no change in behavior, but this line seems to contradict what the code is doing: “We don’t consider this type of corruption critical, and can fix it by removing tx data and rescanning,” because it is setting CORRUPT not NEED_RESCAN. Maybe replace it with just “This can be fixed by removing tx data and rescanning.”


    achow101 commented at 5:01 pm on June 23, 2023:
    Done
  310. in src/wallet/walletdb.cpp:1137 in 6b5a530b7a outdated
    1132+    AssertLockHeld(pwallet->cs_wallet);
    1133+    DBErrors result = DBErrors::LOAD_OK;
    1134+
    1135+    // Load spk records
    1136+    std::set<std::pair<OutputType, bool>> seen_spks;
    1137+    for (auto& spk_key : {DBKeys::ACTIVEEXTERNALSPK, DBKeys::ACTIVEINTERNALSPK}) {
    


    ryanofsky commented at 1:08 pm on June 23, 2023:

    In commit “walletdb: refactor active spkm loading” (6b5a530b7ad1c7b9a1ff9c946458b797a6ef785a)

    Would be good this make this const auto& to be clear loading code below isn’t going to change it.


    achow101 commented at 5:01 pm on June 23, 2023:
    Done
  311. in src/wallet/walletdb.cpp:830 in 9eae22cf6e outdated
    825+        return DBErrors::LOAD_OK;
    826+    }
    827+
    828+    // Load HD Chain
    829+    CHDChain chain;
    830+    if (batch.Read(DBKeys::HDCHAIN, chain)) {
    


    ryanofsky commented at 1:38 pm on June 23, 2023:

    In commit “walletdb: Refactor hd chain loading to its own function” (ad779e9ece9829677c1735d8865f14b23459da80)

    There are two changes in behavior here. One is that if the CHDChain value can’t be deserialized, the error is now ignored instead of reported. The other change is that the code is now doing an exact match for the DBKeys::HDCHAIN key rather than a prefix match. Arguably the second change is good and unlikely to cause problems. But it’s good to be aware in this case, and any in other case that switches from ReadKeyValue to batch.Read that batch.Read is going to match a subset of keys. I’ve been reviewing this PR for a pretty long time and just realized this now.

    I’m thinking it would be better to avoid changing behavior here and make the code in this function more consistent with:

     0diff --git a/src/wallet/walletdb.cpp b/src/wallet/walletdb.cpp
     1index dad5c3b531ef..f9c43c04ef31 100644
     2--- a/src/wallet/walletdb.cpp
     3+++ b/src/wallet/walletdb.cpp
     4@@ -826,10 +826,11 @@ static DBErrors LoadLegacyWalletRecords(CWallet* pwallet, DatabaseBatch& batch,
     5     }
     6 
     7     // Load HD Chain
     8-    CHDChain chain;
     9-    if (batch.Read(DBKeys::HDCHAIN, chain)) {
    10-        pwallet->GetOrCreateLegacyScriptPubKeyMan()->LoadHDChain(chain);
    11-    }
    12+    LoadResult hd_chain_res = LoadRecords(pwallet, batch, DBKeys::HDCHAIN,
    13+        [] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) {
    14+        return LoadHDChain(pwallet, value, err) ? DBErrors::LOAD_OK : DBErrors::CORRUPT;
    15+    });
    16+    result = std::max(result, hd_chain_res.m_result);
    17 
    18     // Load unencrypted keys
    19     LoadResult key_res = LoadRecords(pwallet, batch, DBKeys::KEY,
    

    Separately, this also makes me think the DatabaseBatch::Read method is potentially dangerous if it going to swallow deserialization errors, and it would be better if it were marked [[nodiscard]]. Or maybe better if it just went away, and wallet code could call batch.ReadKey to access values directly.


    achow101 commented at 5:02 pm on June 23, 2023:
    Done, but I think we should more strongly enforce that there should only be one of these kinds of records. That can be done in a followup.
  312. in src/wallet/walletdb.cpp:1139 in b26573ef95 outdated
    1134+        return DBErrors::LOAD_OK;
    1135+    });
    1136+    result = std::max(result, locked_utxo_res.m_result);
    1137+
    1138+    // Load orderposnext record
    1139+    batch.Read(DBKeys::ORDERPOSNEXT, pwallet->nOrderPosNext);
    


    ryanofsky commented at 2:01 pm on June 23, 2023:

    In commit “walletdb: refactor tx loading” (b26573ef95891bb69f00c3aaa67fb4678a82c818)

    Again, there’s two changes in behavior since this switching from ReadKeyValue to Read. One is that deserialization errors are ignored with no error or warning. The other is that this is switching from a prefix match to an exact match so keys that have other fields after the ORDERPOSNEXT string will be ignored.

    I don’t think either current or previous behavior is ideal, but I think I might opt to preserve previous behavior with:

     0diff --git a/src/wallet/walletdb.cpp b/src/wallet/walletdb.cpp
     1index 7292c6fda414..e6d86bf2ad30 100644
     2--- a/src/wallet/walletdb.cpp
     3+++ b/src/wallet/walletdb.cpp
     4@@ -1136,7 +1136,18 @@ static DBErrors LoadTxRecords(CWallet* pwallet, DatabaseBatch& batch, std::vecto
     5     result = std::max(result, locked_utxo_res.m_result);
     6 
     7     // Load orderposnext record
     8-    batch.Read(DBKeys::ORDERPOSNEXT, pwallet->nOrderPosNext);
     9+    LoadResult order_pos_res = LoadRecords(pwallet, batch, DBKeys::ORDERPOSNEXT,
    10+        [] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) EXCLUSIVE_LOCKS_REQUIRED(pwallet->cs_wallet) {
    11+        try {
    12+           value >> pwallet->nOrderPosNext;
    13+        } catch (const std::exception& e) {
    14+           if (err.empty()) {
    15+              err = e.what();
    16+           }
    17+           return DBErrors::NONCRITICAL_ERROR;
    18+        }
    19+        return DBErrors::LOAD_OK;
    20+    });
    21 
    22     return result;
    23 }
    

    achow101 commented at 5:02 pm on June 23, 2023:
    Done
  313. in src/wallet/walletdb.cpp:792 in 89c4119cb8 outdated
    787+    // These are not actually loaded, but we need to check for them
    788+
    789+    // We don't want or need the default key, but if there is one set,
    790+    // we want to make sure that it is valid so that we can detect corruption
    791+    CPubKey default_pubkey;
    792+    if (batch.Read(DBKeys::DEFAULTKEY, default_pubkey) && !default_pubkey.IsValid()) {
    


    ryanofsky commented at 2:10 pm on June 23, 2023:

    In commit “walletdb: refactor defaultkey and wkey loading” (89c4119cb8be6a62b24c693b329aec565aea07dc)

    Again switching from ReadKeyValue to Read now ignores deserialization errors and keys with extra data fields after the DEFAULTKEY sting. Could preserve previous behavior with:

     0--- a/src/wallet/walletdb.cpp
     1+++ b/src/wallet/walletdb.cpp
     2@@ -788,11 +788,24 @@ static DBErrors LoadLegacyWalletRecords(CWallet* pwallet, DatabaseBatch& batch,
     3 
     4     // We don't want or need the default key, but if there is one set,
     5     // we want to make sure that it is valid so that we can detect corruption
     6-    CPubKey default_pubkey;
     7-    if (batch.Read(DBKeys::DEFAULTKEY, default_pubkey) && !default_pubkey.IsValid()) {
     8-        pwallet->WalletLogPrintf("Error reading wallet database: Default Key corrupt\n");
     9-        return DBErrors::CORRUPT;
    10-    }
    11+    LoadResult default_key_res = LoadRecords(pwallet, batch, DBKeys::DEFAULTKEY,
    12+        [] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) {
    13+        CPubKey default_pubkey;
    14+        try {
    15+            value >> default_pubkey;
    16+        } catch (const std::exception& e) {
    17+            if (err.empty()) {
    18+                err = e.what();
    19+            }
    20+            return DBErrors::CORRUPT;
    21+        }
    22+        if (!default_pubkey.IsValid()) {
    23+            err = "Error reading wallet database: Default Key corrupt";
    24+            return DBErrors::CORRUPT;
    25+        }
    26+        return DBErrors::LOAD_OK;
    27+    });
    28+    result = std::max(result, default_key_res.m_result);
    29 
    30     // "wkey" records are unsupported, if we see any, throw an error
    31     LoadResult wkey_res = LoadRecords(pwallet, batch, DBKeys::OLD_KEY,
    

    achow101 commented at 5:03 pm on June 23, 2023:
    Done
  314. in src/wallet/walletdb.cpp:751 in 89c4119cb8 outdated
    796+
    797+    // "wkey" records are unsupported, if we see any, throw an error
    798+    LoadResult wkey_res = LoadRecords(pwallet, batch, DBKeys::OLD_KEY,
    799+        [] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) {
    800+        err = "Found unsupported 'wkey' record, try loading with version 0.18";
    801+        return DBErrors::LOAD_FAIL;
    


    ryanofsky commented at 2:16 pm on June 23, 2023:

    In commit “walletdb: refactor defaultkey and wkey loading” (89c4119cb8be6a62b24c693b329aec565aea07dc)

    IIUC, previously this was a noncritical error, now it is a LOAD_FAIL error? Would be good to document if this intended in the commit message.


    achow101 commented at 5:05 pm on June 23, 2023:

    Mentioned it in the commit message.

    The change was intentional since those records are keys, and we shouldn’t just ignore keys. These records are unsupported since there was never any code that added them, but if some user has a wallet that somehow does, we should be more explicit about the fact that there are keys in a record that we don’t support.

  315. ryanofsky approved
  316. ryanofsky commented at 2:31 pm on June 23, 2023: contributor

    Code review ACK 290bc962096ca109edd038fbe2f32322ffdabb7b. I reviewed pretty much from the beginning, and there is a lot here, but as far as I can tell everything looks good.

    I left some suggestions below, but I don’t think any of them are critical.

  317. DrahtBot requested review from furszy on Jun 23, 2023
  318. DrahtBot requested review from maflcko on Jun 23, 2023
  319. achow101 force-pushed on Jun 23, 2023
  320. achow101 force-pushed on Jun 23, 2023
  321. in src/wallet/walletdb.cpp:800 in f9db8fce3c outdated
    795+        [] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) {
    796+        CPubKey default_pubkey;
    797+        try {
    798+            value >> default_pubkey;
    799+        } catch (const std::exception& e) {
    800+            if (err.empty()) e.what();
    


    furszy commented at 1:01 pm on June 26, 2023:

    In f9db8fce:

    Missed to set the error here.

    0            if (err.empty()) err = e.what();
    

    achow101 commented at 3:26 pm on June 26, 2023:
    Fixed
  322. in src/wallet/walletdb.cpp:1077 in dc1ef26e43 outdated
    1135+        return DBErrors::LOAD_OK;
    1136+    });
    1137+    result = std::max(result, locked_utxo_res.m_result);
    1138+
    1139+    // Load orderposnext record
    1140+    // Note: There should only be one ORDERPOSNEXT record with nothing trailng the type
    


    maflcko commented at 2:40 pm on June 26, 2023:
    dc1ef26e43: trailing (typo)
  323. in src/wallet/walletdb.cpp:955 in 7c13da1379 outdated
    950+                if (hd_seed_id != legacy_spkm->GetHDChain().seed_id) {
    951+                    legacy_spkm->AddInactiveHDChain(chain);
    952+                }
    953+            }
    954+        }
    955+        else {
    


    furszy commented at 3:24 pm on June 26, 2023:
    In 7c13da13: Only if you have to re-touch: extra jump line

    achow101 commented at 10:06 pm on June 26, 2023:
    Done
  324. achow101 force-pushed on Jun 26, 2023
  325. in src/wallet/walletdb.cpp:741 in aea1a43b37 outdated
    800+            if (err.empty()) err = e.what();
    801+            return DBErrors::CORRUPT;
    802+        }
    803+        if (!default_pubkey.IsValid()) {
    804+            pwallet->WalletLogPrintf("Error reading wallet database: Default Key corrupt\n");
    805+            return DBErrors::CORRUPT;
    


    furszy commented at 3:35 pm on June 26, 2023:
    0            err = "Error reading wallet database: Default Key corrupt";
    1            return DBErrors::CORRUPT;
    

    achow101 commented at 10:06 pm on June 26, 2023:
    Done
  326. DrahtBot removed the label CI failed on Jun 26, 2023
  327. in src/wallet/walletdb.cpp:1023 in dc1ef26e43 outdated
    1077+        auto fill_wtx = [&](CWalletTx& wtx, bool new_tx) {
    1078+            if(!new_tx) {
    1079+                // There's some corruption here since the tx we just tried to load was already in the wallet.
    1080+                // it can be fixed by removing tx data and rescanning.
    1081+                err = "Error: Corrupt transaction found. This can be fixed by removing transactions from wallet and rescanning.";
    1082+                result = DBErrors::CORRUPT;
    


    furszy commented at 7:06 pm on June 26, 2023:

    In dc1ef26e:

    nit: this result is also being set to CORRUPT when LoadToWallet returns false with corrupted_tx=true (~30 lines of code below this line).


    ryanofsky commented at 8:22 pm on June 26, 2023:

    re: #24914 (review)

    In dc1ef26:

    nit: this result is also being set to CORRUPT when LoadToWallet returns false with corrupted_tx=true (~30 lines of code below this line).

    Nice catch. Would be nice to just eliminate the CORRUPTED_TX variable:

     0--- a/src/wallet/walletdb.cpp
     1+++ b/src/wallet/walletdb.cpp
     2@@ -1065,10 +1065,9 @@ static DBErrors LoadTxRecords(CWallet* pwallet, DatabaseBatch& batch, std::vecto
     3     DBErrors result = DBErrors::LOAD_OK;
     4 
     5     // Load tx record
     6-    bool corrupted_tx = false;
     7     any_unordered = false;
     8     LoadResult tx_res = LoadRecords(pwallet, batch, DBKeys::TX,
     9-        [&corrupted_tx, &any_unordered, &upgraded_txs] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) EXCLUSIVE_LOCKS_REQUIRED(pwallet->cs_wallet) {
    10+        [&any_unordered, &upgraded_txs] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) EXCLUSIVE_LOCKS_REQUIRED(pwallet->cs_wallet) {
    11         DBErrors result = DBErrors::LOAD_OK;
    12         uint256 hash;
    13         key >> hash;
    14@@ -1077,10 +1076,8 @@ static DBErrors LoadTxRecords(CWallet* pwallet, DatabaseBatch& batch, std::vecto
    15         auto fill_wtx = [&](CWalletTx& wtx, bool new_tx) {
    16             if(!new_tx) {
    17                 // There's some corruption here since the tx we just tried to load was already in the wallet.
    18-                // it can be fixed by removing tx data and rescanning.
    19                 err = "Error: Corrupt transaction found. This can be fixed by removing transactions from wallet and rescanning.";
    20                 result = DBErrors::CORRUPT;
    21-                corrupted_tx = true;
    22                 return false;
    23             }
    24             value >> wtx;
    25@@ -1114,11 +1111,8 @@ static DBErrors LoadTxRecords(CWallet* pwallet, DatabaseBatch& batch, std::vecto
    26             return true;
    27         };
    28         if (!pwallet->LoadToWallet(hash, fill_wtx)) {
    29-            if (corrupted_tx) {
    30-                result = DBErrors::CORRUPT;
    31-            } else {
    32-                result = std::max(result, DBErrors::NEED_RESCAN);
    33-            }
    34+            // Use max because fill_wtx above might have already set result CORRUPT.
    35+            result = std::max(result, DBErrors::NEED_RESCAN);
    36         }
    37         return result;
    38     });
    

    achow101 commented at 10:06 pm on June 26, 2023:
    Removed corrupted_tx as suggested.
  328. furszy commented at 7:16 pm on June 26, 2023: member

    ACK 973d910c

    Only left few more nits. Nothing blocking.

  329. DrahtBot requested review from ryanofsky on Jun 26, 2023
  330. ryanofsky approved
  331. ryanofsky commented at 8:25 pm on June 26, 2023: contributor

    Code review ACK 973d910c287867d8bb1c8041bca906c14596aeda

    Looks good, probably ready for merge with a third ACK.

  332. achow101 force-pushed on Jun 26, 2023
  333. ryanofsky approved
  334. ryanofsky commented at 10:35 pm on June 26, 2023: contributor
    Code review ACK 7cb774048c5f04b3e20d95e794c5a350ea4eff97. Just more suggested error handling tweaks since last review
  335. DrahtBot requested review from furszy on Jun 26, 2023
  336. in src/wallet/walletdb.cpp:1113 in f6b07b20e1 outdated
    1108+                any_unordered = true;
    1109+
    1110+            return true;
    1111+        };
    1112+        if (!pwallet->LoadToWallet(hash, fill_wtx)) {
    1113+            // Use std::max as fill_wtx may have already set result to CORRUP;
    


    furszy commented at 10:40 pm on June 26, 2023:
    0            // Use std::max as fill_wtx may have already set result to CORRUPT;
    

    achow101 commented at 3:08 pm on June 27, 2023:
    Fixed
  337. furszy approved
  338. furszy commented at 10:40 pm on June 26, 2023: member
    reACK 7cb77404
  339. in src/wallet/walletdb.cpp:757 in 20d1fc5061 outdated
    750@@ -855,6 +751,268 @@ static DBErrors LoadWalletFlags(CWallet* pwallet, DatabaseBatch& batch) EXCLUSIV
    751     return DBErrors::LOAD_OK;
    752 }
    753 
    754+struct LoadResult
    755+{
    756+    DBErrors m_result{DBErrors::LOAD_OK};
    757+    std::string m_error{};
    


    maflcko commented at 10:28 am on June 27, 2023:

    nit in 20d1fc50618911decb74e5b977afbb4ae8bb36ce: I think this is still wrong? Currently this only collects the first error (in cases where assigning the error is guarded by an empty check). Also, it is unused by the caller. Suggested fix:

     0diff --git a/src/wallet/walletdb.cpp b/src/wallet/walletdb.cpp
     1index 1fab6a7f9c..e466c1c477 100644
     2--- a/src/wallet/walletdb.cpp
     3+++ b/src/wallet/walletdb.cpp
     4@@ -754,7 +754,6 @@ static DBErrors LoadWalletFlags(CWallet* pwallet, DatabaseBatch& batch) EXCLUSIV
     5 struct LoadResult
     6 {
     7     DBErrors m_result{DBErrors::LOAD_OK};
     8-    std::string m_error{};
     9     int m_records{0};
    10 };
    11 
    12@@ -786,9 +785,10 @@ static LoadResult LoadRecords(CWallet* pwallet, DatabaseBatch& batch, const std:
    13         std::string type;
    14         ssKey >> type;
    15         assert(type == key);
    16-        DBErrors record_res = load_func(pwallet, ssKey, ssValue, result.m_error);
    17+        std::string log_error{};
    18+        DBErrors record_res = load_func(pwallet, ssKey, ssValue, log_error);
    19         if (record_res != DBErrors::LOAD_OK) {
    20-            pwallet->WalletLogPrintf("%s\n", result.m_error);
    21+            pwallet->WalletLogPrintf("%s\n", log_error);
    22         }
    23         result.m_result = std::max(result.m_result, record_res);
    24         ++result.m_records;
    

    Also, the commit description seems to imply behavior changed where it didn’t, no?

    Suggested range diff:

     0 8:  20d1fc5061 !  8:  0bf8188513 walletdb: Refactor legacy wallet record loading into its own function
     1    @@ Commit message
     2     
     3         Exceptions are handled on a per-record type basis, rather than globally.
     4         For private keys in a legacy wallet and the encryption keys,
     5    -    a deserialization error will result in an early return rather than
     6    +    a deserialization error will result in a
     7         contined loading of the remaining keys, with the resulting wallet
     8    -    failing to load. For other record types, errors that were previously
     9    +    failing to load, unchanged from previous behavior. For other record types, errors that were previously
    10         handled by the global exception handler will result in loading failure
    11    +    , also skipping the continued loading of remaining keys,
    12         instead of a non-critical error.
    

    For “other record types”, could also unify the section with the section in the next commit, which should be the same?

    0    Exception handling for these records changes to a per-record type basis,
    1    rather than globally. This results in some records now failing with a
    2    critical error rather than a non-critical one.
    

    maflcko commented at 10:36 am on June 27, 2023:
    Also, could mention that for other record types, previous early returns from LoadWallet like UNEXPECTED_LEGACY_ENTRY would now be scheduled for later, and it continues to load more prefix-keys?

    achow101 commented at 3:10 pm on June 27, 2023:

    Implemented the suggested changes.

    Also, could mention that for other record types, previous early returns from LoadWallet like UNEXPECTED_LEGACY_ENTRY would now be scheduled for later, and it continues to load more prefix-keys?

    I don’t think that’s necessarily true? I’m planning on revisiting the error handling in a followup.


    maflcko commented at 3:24 pm on June 27, 2023:

    achow101 commented at 3:30 pm on June 27, 2023:
    What I meant was that there are records that now early return but didn’t previously, but I suppose that’s the opposite of what you were talking about.
  340. in src/wallet/walletdb.cpp:799 in 877382511c outdated
    794+        [] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) {
    795+        CPubKey default_pubkey;
    796+        try {
    797+            value >> default_pubkey;
    798+        } catch (const std::exception& e) {
    799+            if (err.empty()) err = e.what();
    


    maflcko commented at 11:18 am on June 27, 2023:
    877382511cc63b64e22bb9f06fdbcb1df0c06f55: Any reason to check for err.empty, when the error shouldn’t be empty, and the later assignment also doesn’t check for empty?

    achow101 commented at 3:08 pm on June 27, 2023:
    Changed
  341. maflcko approved
  342. maflcko commented at 11:21 am on June 27, 2023: member

    lgtm ACK 7cb774048c5f04b3e20d95e794c5a350ea4eff97 🍖

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: lgtm ACK 7cb774048c5f04b3e20d95e794c5a350ea4eff97 🍖
    3wuc4npVuTb3Yb6tNza4MK7G2Z6qzHiajO9EsxCW7ib4DxRzWDxs552TGpGVPJxZ5qafuYSW65Z4/WLbHadprCA==
    
  343. maflcko commented at 12:26 pm on June 27, 2023: member
    Also, it may be good to add tests for currently untested code: https://marcofalke.github.io/b-c-cov/total.coverage/src/wallet/wallet.cpp.gcov.html#2926
  344. ryanofsky commented at 1:56 pm on June 27, 2023: contributor

    With 3 up-to-date acks, and all the review that’s happened so far, I think this looks ready to merge now. But will wait to see if @achow101 wants to respond to newest comments.

    I like Marco’s error handling suggestions since they simplify code and just print all the errors. If it turns out there are this cases where this repeats too many errors, it seems like it would be easy to add a bit of code to LoadRecords that just compares new error messages to previous and adds [Repeated ## times] to the log.

    It would also be great to to look at coverage link #24914 (comment) and list places where we should try to add missing test coverage.

  345. maflcko commented at 2:00 pm on June 27, 2023: member

    If it turns out there are this cases where this repeats too many errors

    I think that’d be the case on current master as well. My comment was that this pull may repeat the first error message, even though there may be a second one.

  346. ryanofsky commented at 2:12 pm on June 27, 2023: contributor

    I think that’d be the case on current master as well. My comment was that this pull may repeat the first error message, even though there may be a second one.

    Makes sense, I missed that. I thought the code was only trying to print one error message, and didn’t notice it would print it multiple times. I like your fix of just printing all the messages instead of printing one, but either way to fix this seems fine.

  347. furszy commented at 2:48 pm on June 27, 2023: member

    Also, it may be good to add tests for currently untested code: https://marcofalke.github.io/b-c-cov/total.coverage/src/wallet/wallet.cpp.gcov.html#2926

    I actually have a PR planned to remove most of those lines. The string errors there are duplicated from the internal walletdb errors. We can just bubble up the error instead of swallowing it.

    Also, in the unit tests, we usually call CWallet::LoadWallet instead of CWallet::Create, thus why some of those paths are untested (like the UNKNOWN_DESCRIPTOR one, which is tested here).

    That being said, totally agree on adding much more coverage to the area.

  348. walletdb: Refactor legacy wallet record loading into its own function
    Instead of loading legacy wallet records as we come across them when
    iterating the database, load them explicitly.
    
    Exception handling for these records changes to a per-record type basis,
    rather than globally. This results in some records now failing with a
    critical error rather than a non-critical one.
    30ab11c497
  349. walletdb: Refactor descriptor wallet records loading
    Instead of loading descriptor wallet records as we come across them when
    iterating the database, loading them explicitly.
    
    Exception handling for these records changes to a per-record type basis,
    rather than globally. This results in some records now failing with a
    critical error rather than a non-critical one.
    405b4d9147
  350. walletdb: refactor address book loading
    Instead of loading address book records as we come across them when
    iterating the database, load them explicitly
    
    Due to exception handling changes, deserialization errors are now
    treated as critical.
    
    The error message for noncritical errors has also been updated to
    reflect that there's more data that could be missing than just address
    book entries and tx data.
    abcc13dd24
  351. walletdb: refactor tx loading
    Instead of loading tx records as we come across them when iterating the
    database, load them explicitly.
    6fabb7fc99
  352. walletdb: refactor active spkm loading
    Instead of loading active spkm records as we come across them when
    iterating the database, load them explicitly.
    
    Due to exception handling changes, deserialization errors are now
    treated as critical.
    c978c6d39c
  353. walletdb: refactor defaultkey and wkey loading
    Instead of dealing with these records when iterating the entire
    database, find and handle them explicitly.
    
    Loading of OLD_KEY records is bumped up to a LOAD_FAIL error as we will
    not be able to use these types of keys which can lead to users missing
    funds.
    31c033e5ca
  354. walletdb: refactor decryption key loading
    Instead of loading decryption keys as we iterate the database, load them
    explicitly.
    cd211b3b99
  355. walletdb: Remove loading code where the database is iterated
    Instead of iterating the database to load the wallet, we now load
    particular kinds of records in an order that we want them to be loaded.
    So it is no longer necessary to iterate the entire database to load the
    wallet.
    2636844f53
  356. doc: Add release note for wallet loading changes
    Co-Authored-By: Ryan Ofsky <ryan@ofsky.org>
    3c83b1d884
  357. achow101 force-pushed on Jun 27, 2023
  358. maflcko commented at 3:19 pm on June 27, 2023: member

    re-ACK 3c83b1d884b419adece95b335b6e956e7459a7ef 🍤

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: re-ACK 3c83b1d884b419adece95b335b6e956e7459a7ef 🍤
    3LjPNOXHR9a4Og46x/6TZNAoki0QrUodvQbWJAyOV1qwc7WRRNkI99Gn13HSmcRTEqR3pSz6zwG+lAATKwQF8CA==
    
  359. DrahtBot requested review from furszy on Jun 27, 2023
  360. DrahtBot requested review from ryanofsky on Jun 27, 2023
  361. furszy approved
  362. furszy commented at 9:21 pm on June 27, 2023: member
    reACK 3c83b1d8
  363. ryanofsky approved
  364. ryanofsky commented at 10:39 pm on June 27, 2023: contributor
    Code review ACK 3c83b1d884b419adece95b335b6e956e7459a7ef. Just Marco’s suggested error handling fixes since last review
  365. ryanofsky merged this on Jun 27, 2023
  366. ryanofsky closed this on Jun 27, 2023

  367. sidhujag referenced this in commit 23bcef68f2 on Jun 30, 2023
  368. Thompson1985 commented at 11:56 am on October 29, 2023: none
    Veiwed

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-07-03 07:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me