wallet, test: Replace MockableDatabase with in-memory SQLiteDatabase #33032

pull achow101 wants to merge 5 commits into bitcoin:master from achow101:sqlite-in-memory-db changing 8 files +96 −209
  1. achow101 commented at 10:36 PM on July 21, 2025: member

    MockableDatabase was introduced for the tests to avoid tying non-database tests to a particular database type. However, since the only database type now is sqlite, and because the mockable behavior is no longer used by the tests, we can replace usage of the MockabeDatabase with a SQLite database that lives only in memory.

    This is particularly useful for future work that has the wallet make use of SQLite's capabilities more, which are less conducive to having a separate mock database implementation.

  2. DrahtBot commented at 10:36 PM on July 21, 2025: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33032.

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK brunoerg
    Stale ACK PeterWrighten, rkrux, sedited

    If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #35018 (wallet, bench: Use Nanobench setup() for wallet benchmarks, and remove DuplicateMockDatabase by achow101)
    • #34909 (wallet, refactor: modularise wallet by extracting out legacy wallet migration by rkrux)
    • #33160 (bench: Add more realistic Coin Selection Bench by murchandamus)
    • #33034 (wallet: Store transactions in a separate sqlite table by achow101)
    • #31260 (scripted-diff: Type-safe settings retrieval by ryanofsky)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

    LLM Linter (✨ experimental)

    Possible places where named args for integral literals may be used (e.g. func(x, /*named_arg=*/0) in C++, and func(x, named_arg=0) in Python):

    • SQLiteDatabase(fs::PathFromString("mock/"), fs::PathFromString("mock/wallet.dat"), DatabaseOptions(), SQLITE_OPEN_MEMORY) in src/wallet/test/util.cpp

    <sup>2026-04-02 18:40:17</sup>

  3. PeterWrighten commented at 8:16 PM on July 22, 2025: none

    utACK 78897500676f7ab1b197da56c0a676efcb49ec41

  4. achow101 force-pushed on Jul 23, 2025
  5. in src/wallet/sqlite.h:94 in f69852441e outdated
      88 | @@ -95,6 +89,14 @@ class SQLiteBatch : public DatabaseBatch
      89 |      bool TxnCommit() override;
      90 |      bool TxnAbort() override;
      91 |      bool HasActiveTxn() override { return m_txn; }
      92 | +
      93 | +    // DO NOT CALL DIRECTLY, use DatabaseBatch::Read, Write, Erase, Exists, and ErasePrefix.
      94 | +    // These functions are public for testing only.
    


    rkrux commented at 1:36 PM on July 24, 2025:

    In f69852441e56c70edaebeaa7e2db4c6249c7e17c "test, wallet: Replace MockableDatabase with in-memory SQLiteDatabase"

    Okay with removal of classes that extend the generic classes such as WalletDatabase, DatabaseCursor, DatabaseBatch, but I don't prefer seeing such comments in the non-test code. It's usually a sign of low cohesion in the design imho, which is not desirable.

    An alternative could be to create a new class MockSQLiteBatch that extends SQLiteBatch. The test specific implementation can go inside the mock class.


    maflcko commented at 2:17 PM on July 24, 2025:

    nit in 78897500676f7ab1b197da56c0a676efcb49ec41 : I guess there is no real risk in having them public, due to the types involved, but if you wanted to make them protected, it could be done with this diff:

    diff --git a/src/wallet/sqlite.h b/src/wallet/sqlite.h
    index 57876bd83c..ce703f8f13 100644
    --- a/src/wallet/sqlite.h
    +++ b/src/wallet/sqlite.h
    @@ -75,6 +75,14 @@ private:
         void SetupSQLStatements();
         bool ExecStatement(sqlite3_stmt* stmt, std::span<const std::byte> blob);
     
    +protected:
    +
    +    bool ReadKey(DataStream&& key, DataStream& value) override;
    +    bool WriteKey(DataStream&& key, DataStream&& value, bool overwrite = true) override;
    +    bool EraseKey(DataStream&& key) override;
    +    bool HasKey(DataStream&& key) override;
    +    bool ErasePrefix(std::span<const std::byte> prefix) override;
    +
     public:
         explicit SQLiteBatch(SQLiteDatabase& database);
         ~SQLiteBatch() override { Close(); }
    @@ -89,14 +97,6 @@ public:
         bool TxnCommit() override;
         bool TxnAbort() override;
         bool HasActiveTxn() override { return m_txn; }
    -
    -    // DO NOT CALL DIRECTLY, use DatabaseBatch::Read, Write, Erase, Exists, and ErasePrefix.
    -    // These functions are public for testing only.
    -    bool ReadKey(DataStream&& key, DataStream& value) override;
    -    bool WriteKey(DataStream&& key, DataStream&& value, bool overwrite = true) override;
    -    bool EraseKey(DataStream&& key) override;
    -    bool HasKey(DataStream&& key) override;
    -    bool ErasePrefix(std::span<const std::byte> prefix) override;
     };
     
     /** An instance of this class represents one SQLite3 database.
    diff --git a/src/wallet/test/util.cpp b/src/wallet/test/util.cpp
    index 5162baed9f..95255b7062 100644
    --- a/src/wallet/test/util.cpp
    +++ b/src/wallet/test/util.cpp
    @@ -79,6 +79,10 @@ void TestUnloadWallet(std::shared_ptr<CWallet>&& wallet)
         WaitForDeleteWallet(std::move(wallet));
     }
     
    +struct SQLiteBatchTest:public SQLiteBatch{
    +using SQLiteBatch::WriteKey;
    +};
    +
     std::unique_ptr<WalletDatabase> DuplicateMockDatabase(WalletDatabase& database)
     {
         std::unique_ptr<DatabaseBatch> batch_orig = database.MakeBatch();
    @@ -86,7 +90,7 @@ std::unique_ptr<WalletDatabase> DuplicateMockDatabase(WalletDatabase& database)
     
         std::unique_ptr<WalletDatabase> new_db = CreateMockableWalletDatabase();
         std::unique_ptr<DatabaseBatch> new_db_batch = new_db->MakeBatch();
    -    SQLiteBatch* batch_new = dynamic_cast<SQLiteBatch*>(new_db_batch.get());
    +    auto* batch_new = dynamic_cast<SQLiteBatchTest*>(new_db_batch.get());
         Assert(batch_new);
     
         while (true) {
    

    achow101 commented at 7:25 PM on July 24, 2025:

    Made a MockableSQLiteBatch and MockableSQLiteDatabase which does this.

  6. in src/wallet/sqlite.h:164 in f69852441e outdated
     160 |      }
     161 | -    std::string Format() override { return "sqlite"; }
     162 | +    std::string Format() override
     163 | +    {
     164 | +        if (m_mock) return "sqlite-mock";
     165 | +        return "sqlite";
    


    rkrux commented at 1:38 PM on July 24, 2025:

    In https://github.com/bitcoin/bitcoin/commit/f69852441e56c70edaebeaa7e2db4c6249c7e17c "test, wallet: Replace MockableDatabase with in-memory SQLiteDatabase"

    I don't think there should be mock related checks in the non-test code. A MockSQLiteDatabase class extending SQLiteDatabase class can override this function.


    achow101 commented at 7:25 PM on July 24, 2025:

    Added MockableSQLiteDatabase

  7. in src/wallet/sqlite.h:155 in f69852441e outdated
     151 | @@ -150,12 +152,17 @@ class SQLiteDatabase : public WalletDatabase
     152 |      /** Return paths to all database created files */
     153 |      std::vector<fs::path> Files() override
     154 |      {
     155 | +        if (m_mock) return {};
    


    rkrux commented at 1:40 PM on July 24, 2025:

    In https://github.com/bitcoin/bitcoin/commit/f69852441e56c70edaebeaa7e2db4c6249c7e17c "test, wallet: Replace MockableDatabase with in-memory SQLiteDatabase"

    I see the m_mock was already present in SQLiteDatabase class but perhaps this PR can be an opportunity to take it out from it by extending this class by a MockSQLiteDatabase. git grep -n m_mock shows limited if checks in the non-test code that can be removed now.


    achow101 commented at 7:25 PM on July 24, 2025:

    Removed now that there's MockableSQLiteDatabase

  8. in src/wallet/test/util.cpp:117 in f69852441e outdated
     207 | -
     208 | -MockableDatabase& GetMockableDatabase(CWallet& wallet)
     209 | -{
     210 | -    return dynamic_cast<MockableDatabase&>(wallet.GetDatabase());
     211 | +    DatabaseOptions dummy_options;
     212 | +    return std::make_unique<SQLiteDatabase>(fs::PathFromString("mock/"), fs::PathFromString("mock/wallet.dat"), dummy_options, /*mock=*/true);
    


    rkrux commented at 1:42 PM on July 24, 2025:

    In https://github.com/bitcoin/bitcoin/commit/f69852441e56c70edaebeaa7e2db4c6249c7e17c "test, wallet: Replace MockableDatabase with in-memory SQLiteDatabase"

    - return std::make_unique<SQLiteDatabase>(fs::PathFromString("mock/"), fs::PathFromString("mock/wallet.dat"), dummy_options, /*mock=*/true);
    + return std::make_unique<MockSQLiteDatabase>();
    

    The MockSQLiteDatabase constructor can hardcode most of these mock values.


    achow101 commented at 7:25 PM on July 24, 2025:

    done

  9. in src/wallet/wallet.cpp:3451 in f69852441e outdated
    3447 | @@ -3448,7 +3448,7 @@ void CWallet::SetupLegacyScriptPubKeyMan()
    3448 |          return;
    3449 |      }
    3450 |  
    3451 | -    Assert(m_database->Format() == "bdb_ro" || m_database->Format() == "mock");
    3452 | +    Assert(m_database->Format() == "bdb_ro" || m_database->Format() == "sqlite-mock");
    


    rkrux commented at 1:58 PM on July 24, 2025:

    In https://github.com/bitcoin/bitcoin/commit/f69852441e56c70edaebeaa7e2db4c6249c7e17c "test, wallet: Replace MockableDatabase with in-memory SQLiteDatabase"

    Nit and need not be done in this PR: To avoid hardcoding mocking constants in non-test code, perhaps the following? If the return value of Format had been an enum, the following would come across as a tighter check.

    - Assert(m_database->Format() == "bdb_ro" || m_database->Format() == "sqlite-mock");
    + Assert(m_database->Format() != "sqlite"); 
    

    achow101 commented at 7:26 PM on July 24, 2025:

    I prefer to leave this as-is as it is a more specific check. It checks for the db types that are allowed to be legacy wallets, rather than allowing legacy wallets for all non-sqlite databases.

  10. in src/wallet/sqlite.cpp:348 in f69852441e outdated
     344 | @@ -345,6 +345,7 @@ bool SQLiteDatabase::Rewrite()
     345 |  
     346 |  bool SQLiteDatabase::Backup(const std::string& dest) const
     347 |  {
     348 | +    if (m_mock) return true;
    


    rkrux commented at 2:04 PM on July 24, 2025:

    In https://github.com/bitcoin/bitcoin/commit/f69852441e56c70edaebeaa7e2db4c6249c7e17c "test, wallet: Replace MockableDatabase with in-memory SQLiteDatabase"

    Checks like these can be distracting to read in non-test code and I believe can be avoided with the MockSQLiteDatabase class.


    achow101 commented at 7:26 PM on July 24, 2025:

    Removed now that there's MockableSQLiteDatabase

  11. rkrux commented at 2:12 PM on July 24, 2025: contributor

    Concept ACK f69852441e56c70edaebeaa7e2db4c6249c7e17c

    I have looked in depth only the last commit. Almost all of my suggestions merge towards having child mock class(es) extending the base SQLite specific classes, wherever required. They can be deleted in the future if the need arises without touching the base SQLite classes. Also, I believe they would not make it difficult to achieve the following goal stated in the PR desc.

    This is particularly useful for future work that has the wallet make use of SQLite's capabilities more, which are less conducive to having a separate mock database implementation.

  12. achow101 force-pushed on Jul 24, 2025
  13. brunoerg commented at 7:43 PM on July 24, 2025: contributor

    Concept ACK

  14. achow101 force-pushed on Jul 24, 2025
  15. DrahtBot added the label CI failed on Jul 24, 2025
  16. DrahtBot commented at 8:16 PM on July 24, 2025: contributor

    <!--85328a0da195eb286784d51f73fa0af9-->

    🚧 At least one of the CI tasks failed. <sub>Task MSan, depends: https://github.com/bitcoin/bitcoin/runs/46676643655</sub> <sub>LLM reason (✨ experimental): The CI failure is caused by a compilation error due to the use of the -Werror flag, which treats a warning about an unused private field as an error, preventing successful build.</sub>

    <details><summary>Hints</summary>

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

    </details>

  17. in src/wallet/test/util.cpp:91 in 998238934f outdated
      88 |      std::unique_ptr<DatabaseBatch> new_db_batch = new_db->MakeBatch();
      89 | -    MockableBatch* batch_new = dynamic_cast<MockableBatch*>(new_db_batch.get());
      90 | +    MockableSQLiteBatch* batch_new = dynamic_cast<MockableSQLiteBatch*>(new_db_batch.get());
      91 |      Assert(batch_new);
      92 |  
      93 |      while (true) {
    


    rkrux commented at 5:48 AM on July 25, 2025:

    Nit to use tighter function signature that goes along with the function name too. Builds & tests fine on my system.

    <details> <summary>Use MockableSQLiteDatabase in return type that passes for WalletDatabase</summary>

    diff --git a/src/wallet/test/util.cpp b/src/wallet/test/util.cpp
    index 6017582395..58fafeecf7 100644
    --- a/src/wallet/test/util.cpp
    +++ b/src/wallet/test/util.cpp
    @@ -80,12 +80,12 @@ void TestUnloadWallet(std::shared_ptr<CWallet>&& wallet)
         WaitForDeleteWallet(std::move(wallet));
     }
     
    -std::unique_ptr<WalletDatabase> DuplicateMockDatabase(WalletDatabase& database)
    +std::unique_ptr<MockableSQLiteDatabase> DuplicateMockDatabase(MockableSQLiteDatabase& database)
     {
         std::unique_ptr<DatabaseBatch> batch_orig = database.MakeBatch();
         std::unique_ptr<DatabaseCursor> cursor_orig = batch_orig->GetNewCursor();
     
    -    std::unique_ptr<WalletDatabase> new_db = CreateMockableWalletDatabase();
    +    std::unique_ptr<MockableSQLiteDatabase> new_db = CreateMockableWalletDatabase();
         std::unique_ptr<DatabaseBatch> new_db_batch = new_db->MakeBatch();
         MockableSQLiteBatch* batch_new = dynamic_cast<MockableSQLiteBatch*>(new_db_batch.get());
         Assert(batch_new);
    @@ -116,7 +116,7 @@ MockableSQLiteDatabase::MockableSQLiteDatabase()
         : SQLiteDatabase(fs::PathFromString("mock/"), fs::PathFromString("mock/wallet.dat"), DatabaseOptions(), SQLITE_OPEN_MEMORY)
     {}
     
    -std::unique_ptr<WalletDatabase> CreateMockableWalletDatabase()
    +std::unique_ptr<MockableSQLiteDatabase> CreateMockableWalletDatabase()
     {
         return std::make_unique<MockableSQLiteDatabase>();
     }
    diff --git a/src/wallet/test/util.h b/src/wallet/test/util.h
    index a240a8ae4f..cedd37ee7e 100644
    --- a/src/wallet/test/util.h
    +++ b/src/wallet/test/util.h
    @@ -70,7 +70,7 @@ public:
         std::unique_ptr<DatabaseBatch> MakeBatch() override { return std::make_unique<MockableSQLiteBatch>(*this); }
     };
     
    -std::unique_ptr<WalletDatabase> CreateMockableWalletDatabase();
    +std::unique_ptr<MockableSQLiteDatabase> CreateMockableWalletDatabase();
     MockableSQLiteDatabase& GetMockableDatabase(CWallet& wallet);
     
     DescriptorScriptPubKeyMan* CreateDescriptor(CWallet& keystore, const std::string& desc_str, const bool success);
    
    

    </details>

  18. rkrux approved
  19. rkrux commented at 5:49 AM on July 25, 2025: contributor

    tACK 1cdb9161774ccf3edcdcb1e482b20be430ed5430

    Thanks for incorporating suggestions.

  20. DrahtBot requested review from brunoerg on Jul 25, 2025
  21. DrahtBot removed the label CI failed on Jul 25, 2025
  22. achow101 requested review from Sjors on Oct 22, 2025
  23. DrahtBot added the label Needs rebase on Dec 17, 2025
  24. achow101 force-pushed on Dec 22, 2025
  25. DrahtBot removed the label Needs rebase on Dec 22, 2025
  26. DrahtBot added the label CI failed on Dec 23, 2025
  27. DrahtBot removed the label CI failed on Dec 23, 2025
  28. achow101 force-pushed on Jan 19, 2026
  29. sedited approved
  30. sedited commented at 9:14 AM on March 16, 2026: contributor

    ACK b544aabdb373b6e0f50ef7301744e862ada7f525

  31. DrahtBot requested review from rkrux on Mar 16, 2026
  32. bench, wallet: Make WalletMigration's setup WalletBatch scoped
    WalletBatch needs to be in a scope so that it is destroyed before the
    database is closed during migration.
    964eafb71c
  33. test: Make duplicating MockableDatabases use cursor and batch
    Instead of directly copying the stored records map when duplicating a
    MockableDatabase, use a Cursor to read the records, and a Batch to write
    them into the new database. This prepares for using SQLite as the
    database backend for MockableDatabase.
    e7d67c9fd9
  34. wallet, bench: Use TestingSetup in CoinSelection benchmark b69f989dc5
  35. wallet: Make Mockable{Database,Batch} subclasses of SQLite classes
    The mocking functionality of MockableDatabase, MockableBatch, and
    MockableCursor was not really being used. These are changed to be
    subclasses of their respective SQLite* classes and will use in-memory
    SQLite databases so that the tests are more representative of actual
    database behavior.
    
    MockableCursor is removed as there are no overrides needed in
    SQLiteCursor for the tests.
    59484e2fdb
  36. walletdb: Remove m_mock from SQLiteDatabase 037ea2c714
  37. achow101 force-pushed on Apr 2, 2026
  38. achow101 commented at 6:39 PM on April 2, 2026: member

    rebased for a silent merge conflict, depends builds were failing.

  39. in src/wallet/test/util.h:76 in 037ea2c714
     126 |  };
     127 |  
     128 | -std::unique_ptr<WalletDatabase> CreateMockableWalletDatabase(MockableData records = {});
     129 | -MockableDatabase& GetMockableDatabase(CWallet& wallet);
     130 | +std::unique_ptr<WalletDatabase> CreateMockableWalletDatabase();
     131 | +MockableSQLiteDatabase& GetMockableDatabase(CWallet& wallet);
    


    brunoerg commented at 7:11 PM on April 9, 2026:

    59484e2fdbd45364b119090e57e701142e501499: This is declared here but never defined. Can be removed whether it will not be used?

  40. in src/bench/coin_selection.cpp:131 in 037ea2c714
     127 | @@ -139,3 +128,4 @@ static void BnBExhaustion(benchmark::Bench& bench)
     128 |  
     129 |  BENCHMARK(CoinSelection);
     130 |  BENCHMARK(BnBExhaustion);
     131 | +}; // namespace wallet
    


    brunoerg commented at 7:12 PM on April 9, 2026:

    Style nit:

    } // namespace wallet
    
  41. brunoerg approved
  42. brunoerg commented at 7:15 PM on April 9, 2026: contributor

    code review ACK 037ea2c714cb1a9a8287a480b87c0ada3aba3ddf

    left two comments.

  43. DrahtBot requested review from sedited on Apr 9, 2026

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-04-17 09:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me