wallet, test: Replace MockableDatabase with in-memory SQLiteDatabase #33032

pull achow101 wants to merge 4 commits into bitcoin:master from achow101:sqlite-in-memory-db changing 6 files +89 −193
  1. achow101 commented at 10:36 pm on July 21, 2025: member

    MockableDatabase was introduced for the tests to avoid tying non-database tests to a particular database type. However, since the only database type now is sqlite, and because the mockable behavior is no longer used by the tests, we can replace usage of the MockabeDatabase with a SQLite database that lives only in memory.

    This is particularly useful for future work that has the wallet make use of SQLite’s capabilities more, which are less conducive to having a separate mock database implementation.

  2. DrahtBot commented at 10:36 pm on July 21, 2025: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33032.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK rkrux
    Concept ACK brunoerg
    Stale ACK PeterWrighten

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #33034 (wallet: Store transactions in a separate sqlite table by achow101)
    • #32685 (wallet: Allow read-only database access for info and dump commands by PeterWrighten)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  3. PeterWrighten commented at 8:16 pm on July 22, 2025: none
    utACK 78897500676f7ab1b197da56c0a676efcb49ec41
  4. bench, wallet: Make WalletMigration's setup WalletBatch scoped
    WalletBatch needs to be in a scope so that it is destroyed before the
    database is closed during migration.
    b08a77549e
  5. test: Make duplicating MockableDatabases use cursor and batch
    Instead of directly copying the stored records map when duplicating a
    MockableDatabase, use a Cursor to read the records, and a Batch to write
    them into the new database. This prepares for using SQLite as the
    database backend for MockableDatabase.
    5302b44ca5
  6. achow101 force-pushed on Jul 23, 2025
  7. in src/wallet/sqlite.h:94 in f69852441e outdated
    88@@ -95,6 +89,14 @@ class SQLiteBatch : public DatabaseBatch
    89     bool TxnCommit() override;
    90     bool TxnAbort() override;
    91     bool HasActiveTxn() override { return m_txn; }
    92+
    93+    // DO NOT CALL DIRECTLY, use DatabaseBatch::Read, Write, Erase, Exists, and ErasePrefix.
    94+    // These functions are public for testing only.
    


    rkrux commented at 1:36 pm on July 24, 2025:

    In f69852441e56c70edaebeaa7e2db4c6249c7e17c “test, wallet: Replace MockableDatabase with in-memory SQLiteDatabase”

    Okay with removal of classes that extend the generic classes such as WalletDatabase, DatabaseCursor, DatabaseBatch, but I don’t prefer seeing such comments in the non-test code. It’s usually a sign of low cohesion in the design imho, which is not desirable.

    An alternative could be to create a new class MockSQLiteBatch that extends SQLiteBatch. The test specific implementation can go inside the mock class.


    maflcko commented at 2:17 pm on July 24, 2025:

    nit in 78897500676f7ab1b197da56c0a676efcb49ec41 : I guess there is no real risk in having them public, due to the types involved, but if you wanted to make them protected, it could be done with this diff:

     0diff --git a/src/wallet/sqlite.h b/src/wallet/sqlite.h
     1index 57876bd83c..ce703f8f13 100644
     2--- a/src/wallet/sqlite.h
     3+++ b/src/wallet/sqlite.h
     4@@ -75,6 +75,14 @@ private:
     5     void SetupSQLStatements();
     6     bool ExecStatement(sqlite3_stmt* stmt, std::span<const std::byte> blob);
     7 
     8+protected:
     9+
    10+    bool ReadKey(DataStream&& key, DataStream& value) override;
    11+    bool WriteKey(DataStream&& key, DataStream&& value, bool overwrite = true) override;
    12+    bool EraseKey(DataStream&& key) override;
    13+    bool HasKey(DataStream&& key) override;
    14+    bool ErasePrefix(std::span<const std::byte> prefix) override;
    15+
    16 public:
    17     explicit SQLiteBatch(SQLiteDatabase& database);
    18     ~SQLiteBatch() override { Close(); }
    19@@ -89,14 +97,6 @@ public:
    20     bool TxnCommit() override;
    21     bool TxnAbort() override;
    22     bool HasActiveTxn() override { return m_txn; }
    23-
    24-    // DO NOT CALL DIRECTLY, use DatabaseBatch::Read, Write, Erase, Exists, and ErasePrefix.
    25-    // These functions are public for testing only.
    26-    bool ReadKey(DataStream&& key, DataStream& value) override;
    27-    bool WriteKey(DataStream&& key, DataStream&& value, bool overwrite = true) override;
    28-    bool EraseKey(DataStream&& key) override;
    29-    bool HasKey(DataStream&& key) override;
    30-    bool ErasePrefix(std::span<const std::byte> prefix) override;
    31 };
    32 
    33 /** An instance of this class represents one SQLite3 database.
    34diff --git a/src/wallet/test/util.cpp b/src/wallet/test/util.cpp
    35index 5162baed9f..95255b7062 100644
    36--- a/src/wallet/test/util.cpp
    37+++ b/src/wallet/test/util.cpp
    38@@ -79,6 +79,10 @@ void TestUnloadWallet(std::shared_ptr<CWallet>&& wallet)
    39     WaitForDeleteWallet(std::move(wallet));
    40 }
    41 
    42+struct SQLiteBatchTest:public SQLiteBatch{
    43+using SQLiteBatch::WriteKey;
    44+};
    45+
    46 std::unique_ptr<WalletDatabase> DuplicateMockDatabase(WalletDatabase& database)
    47 {
    48     std::unique_ptr<DatabaseBatch> batch_orig = database.MakeBatch();
    49@@ -86,7 +90,7 @@ std::unique_ptr<WalletDatabase> DuplicateMockDatabase(WalletDatabase& database)
    50 
    51     std::unique_ptr<WalletDatabase> new_db = CreateMockableWalletDatabase();
    52     std::unique_ptr<DatabaseBatch> new_db_batch = new_db->MakeBatch();
    53-    SQLiteBatch* batch_new = dynamic_cast<SQLiteBatch*>(new_db_batch.get());
    54+    auto* batch_new = dynamic_cast<SQLiteBatchTest*>(new_db_batch.get());
    55     Assert(batch_new);
    56 
    57     while (true) {
    

    achow101 commented at 7:25 pm on July 24, 2025:
    Made a MockableSQLiteBatch and MockableSQLiteDatabase which does this.
  8. in src/wallet/sqlite.h:164 in f69852441e outdated
    160     }
    161-    std::string Format() override { return "sqlite"; }
    162+    std::string Format() override
    163+    {
    164+        if (m_mock) return "sqlite-mock";
    165+        return "sqlite";
    


    rkrux commented at 1:38 pm on July 24, 2025:

    In https://github.com/bitcoin/bitcoin/commit/f69852441e56c70edaebeaa7e2db4c6249c7e17c “test, wallet: Replace MockableDatabase with in-memory SQLiteDatabase”

    I don’t think there should be mock related checks in the non-test code. A MockSQLiteDatabase class extending SQLiteDatabase class can override this function.


    achow101 commented at 7:25 pm on July 24, 2025:
    Added MockableSQLiteDatabase
  9. in src/wallet/sqlite.h:155 in f69852441e outdated
    151@@ -150,12 +152,17 @@ class SQLiteDatabase : public WalletDatabase
    152     /** Return paths to all database created files */
    153     std::vector<fs::path> Files() override
    154     {
    155+        if (m_mock) return {};
    


    rkrux commented at 1:40 pm on July 24, 2025:

    In https://github.com/bitcoin/bitcoin/commit/f69852441e56c70edaebeaa7e2db4c6249c7e17c “test, wallet: Replace MockableDatabase with in-memory SQLiteDatabase”

    I see the m_mock was already present in SQLiteDatabase class but perhaps this PR can be an opportunity to take it out from it by extending this class by a MockSQLiteDatabase. git grep -n m_mock shows limited if checks in the non-test code that can be removed now.


    achow101 commented at 7:25 pm on July 24, 2025:
    Removed now that there’s MockableSQLiteDatabase
  10. in src/wallet/test/util.cpp:117 in f69852441e outdated
    207-
    208-MockableDatabase& GetMockableDatabase(CWallet& wallet)
    209-{
    210-    return dynamic_cast<MockableDatabase&>(wallet.GetDatabase());
    211+    DatabaseOptions dummy_options;
    212+    return std::make_unique<SQLiteDatabase>(fs::PathFromString("mock/"), fs::PathFromString("mock/wallet.dat"), dummy_options, /*mock=*/true);
    


    rkrux commented at 1:42 pm on July 24, 2025:

    In https://github.com/bitcoin/bitcoin/commit/f69852441e56c70edaebeaa7e2db4c6249c7e17c “test, wallet: Replace MockableDatabase with in-memory SQLiteDatabase”

    0- return std::make_unique<SQLiteDatabase>(fs::PathFromString("mock/"), fs::PathFromString("mock/wallet.dat"), dummy_options, /*mock=*/true);
    1+ return std::make_unique<MockSQLiteDatabase>();
    

    The MockSQLiteDatabase constructor can hardcode most of these mock values.


    achow101 commented at 7:25 pm on July 24, 2025:
    done
  11. in src/wallet/wallet.cpp:3451 in f69852441e outdated
    3447@@ -3448,7 +3448,7 @@ void CWallet::SetupLegacyScriptPubKeyMan()
    3448         return;
    3449     }
    3450 
    3451-    Assert(m_database->Format() == "bdb_ro" || m_database->Format() == "mock");
    3452+    Assert(m_database->Format() == "bdb_ro" || m_database->Format() == "sqlite-mock");
    


    rkrux commented at 1:58 pm on July 24, 2025:

    In https://github.com/bitcoin/bitcoin/commit/f69852441e56c70edaebeaa7e2db4c6249c7e17c “test, wallet: Replace MockableDatabase with in-memory SQLiteDatabase”

    Nit and need not be done in this PR: To avoid hardcoding mocking constants in non-test code, perhaps the following? If the return value of Format had been an enum, the following would come across as a tighter check.

    0- Assert(m_database->Format() == "bdb_ro" || m_database->Format() == "sqlite-mock");
    1+ Assert(m_database->Format() != "sqlite"); 
    

    achow101 commented at 7:26 pm on July 24, 2025:
    I prefer to leave this as-is as it is a more specific check. It checks for the db types that are allowed to be legacy wallets, rather than allowing legacy wallets for all non-sqlite databases.
  12. in src/wallet/sqlite.cpp:348 in f69852441e outdated
    344@@ -345,6 +345,7 @@ bool SQLiteDatabase::Rewrite()
    345 
    346 bool SQLiteDatabase::Backup(const std::string& dest) const
    347 {
    348+    if (m_mock) return true;
    


    rkrux commented at 2:04 pm on July 24, 2025:

    In https://github.com/bitcoin/bitcoin/commit/f69852441e56c70edaebeaa7e2db4c6249c7e17c “test, wallet: Replace MockableDatabase with in-memory SQLiteDatabase”

    Checks like these can be distracting to read in non-test code and I believe can be avoided with the MockSQLiteDatabase class.


    achow101 commented at 7:26 pm on July 24, 2025:
    Removed now that there’s MockableSQLiteDatabase
  13. rkrux commented at 2:12 pm on July 24, 2025: contributor

    Concept ACK f69852441e56c70edaebeaa7e2db4c6249c7e17c

    I have looked in depth only the last commit. Almost all of my suggestions merge towards having child mock class(es) extending the base SQLite specific classes, wherever required. They can be deleted in the future if the need arises without touching the base SQLite classes. Also, I believe they would not make it difficult to achieve the following goal stated in the PR desc.

    This is particularly useful for future work that has the wallet make use of SQLite’s capabilities more, which are less conducive to having a separate mock database implementation.

  14. wallet: Make Mockable{Database,Batch} subclasses of SQLite classes
    The mocking functionality of MockableDatabase, MockableBatch, and
    MockableCursor was not really being used. These are changed to be
    subclasses of their respective SQLite* classes and will use in-memory
    SQLite databases so that the tests are more representative of actual
    database behavior.
    
    MockableCursor is removed as there are no overrides needed in
    SQLiteCursor for the tests.
    998238934f
  15. achow101 force-pushed on Jul 24, 2025
  16. brunoerg commented at 7:43 pm on July 24, 2025: contributor
    Concept ACK
  17. walletdb: Remove m_mock from SQLiteDatabase 1cdb916177
  18. achow101 force-pushed on Jul 24, 2025
  19. DrahtBot added the label CI failed on Jul 24, 2025
  20. DrahtBot commented at 8:16 pm on July 24, 2025: contributor

    🚧 At least one of the CI tasks failed. Task MSan, depends: https://github.com/bitcoin/bitcoin/runs/46676643655 LLM reason (✨ experimental): The CI failure is caused by a compilation error due to the use of the -Werror flag, which treats a warning about an unused private field as an error, preventing successful build.

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  21. in src/wallet/test/util.cpp:91 in 998238934f outdated
    88     std::unique_ptr<DatabaseBatch> new_db_batch = new_db->MakeBatch();
    89-    MockableBatch* batch_new = dynamic_cast<MockableBatch*>(new_db_batch.get());
    90+    MockableSQLiteBatch* batch_new = dynamic_cast<MockableSQLiteBatch*>(new_db_batch.get());
    91     Assert(batch_new);
    92 
    93     while (true) {
    


    rkrux commented at 5:48 am on July 25, 2025:

    Nit to use tighter function signature that goes along with the function name too. Builds & tests fine on my system.

     0diff --git a/src/wallet/test/util.cpp b/src/wallet/test/util.cpp
     1index 6017582395..58fafeecf7 100644
     2--- a/src/wallet/test/util.cpp
     3+++ b/src/wallet/test/util.cpp
     4@@ -80,12 +80,12 @@ void TestUnloadWallet(std::shared_ptr<CWallet>&& wallet)
     5     WaitForDeleteWallet(std::move(wallet));
     6 }
     7 
     8-std::unique_ptr<WalletDatabase> DuplicateMockDatabase(WalletDatabase& database)
     9+std::unique_ptr<MockableSQLiteDatabase> DuplicateMockDatabase(MockableSQLiteDatabase& database)
    10 {
    11     std::unique_ptr<DatabaseBatch> batch_orig = database.MakeBatch();
    12     std::unique_ptr<DatabaseCursor> cursor_orig = batch_orig->GetNewCursor();
    13 
    14-    std::unique_ptr<WalletDatabase> new_db = CreateMockableWalletDatabase();
    15+    std::unique_ptr<MockableSQLiteDatabase> new_db = CreateMockableWalletDatabase();
    16     std::unique_ptr<DatabaseBatch> new_db_batch = new_db->MakeBatch();
    17     MockableSQLiteBatch* batch_new = dynamic_cast<MockableSQLiteBatch*>(new_db_batch.get());
    18     Assert(batch_new);
    19@@ -116,7 +116,7 @@ MockableSQLiteDatabase::MockableSQLiteDatabase()
    20     : SQLiteDatabase(fs::PathFromString("mock/"), fs::PathFromString("mock/wallet.dat"), DatabaseOptions(), SQLITE_OPEN_MEMORY)
    21 {}
    22 
    23-std::unique_ptr<WalletDatabase> CreateMockableWalletDatabase()
    24+std::unique_ptr<MockableSQLiteDatabase> CreateMockableWalletDatabase()
    25 {
    26     return std::make_unique<MockableSQLiteDatabase>();
    27 }
    28diff --git a/src/wallet/test/util.h b/src/wallet/test/util.h
    29index a240a8ae4f..cedd37ee7e 100644
    30--- a/src/wallet/test/util.h
    31+++ b/src/wallet/test/util.h
    32@@ -70,7 +70,7 @@ public:
    33     std::unique_ptr<DatabaseBatch> MakeBatch() override { return std::make_unique<MockableSQLiteBatch>(*this); }
    34 };
    35 
    36-std::unique_ptr<WalletDatabase> CreateMockableWalletDatabase();
    37+std::unique_ptr<MockableSQLiteDatabase> CreateMockableWalletDatabase();
    38 MockableSQLiteDatabase& GetMockableDatabase(CWallet& wallet);
    39 
    40 DescriptorScriptPubKeyMan* CreateDescriptor(CWallet& keystore, const std::string& desc_str, const bool success);
    
  22. rkrux approved
  23. rkrux commented at 5:49 am on July 25, 2025: contributor

    tACK 1cdb9161774ccf3edcdcb1e482b20be430ed5430

    Thanks for incorporating suggestions.

  24. DrahtBot requested review from brunoerg on Jul 25, 2025
  25. DrahtBot removed the label CI failed on Jul 25, 2025

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-07-26 09:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me