kernel: Prune leveldb headers #28186

pull TheCharlatan wants to merge 16 commits into bitcoin:master from TheCharlatan:cleaveLeveldbHeaders changing 15 files +254 −159
  1. TheCharlatan commented at 3:10 pm on July 30, 2023: contributor

    Leveldb headers are currently included in the dbwrapper.h file and thus available to many of Bitcoin Core’s source files. However, leveldb-specific functionality should be abstracted by the dbwrapper and does not need to be available to the rest of the code. Having leveldb included in a widely-used header such as dbwrapper.h bloats the entire project’s header tree.

    The dbwrapper is a key component of the libbitcoinkernel library. Future users of this library would not want to contend with having the leveldb headers exposed and potentially polluting their project’s namespace.

    For these reasons, the leveldb headers are removed from the dbwrapper by moving leveldb-specific code to the implementation file and creating a pimpl where leveldb member variables are indispensable. As a final step, the leveldb include flags are removed from the BITCOIN_INCLUDES and moved to places where the dbwrapper is compiled.


    This pull request is part of the libbitcoinkernel project, and more specifically its stage 1 step 3 “Decouple most non-consensus headers from libbitcoinkernel”.

  2. DrahtBot commented at 3:10 pm on July 30, 2023: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK stickies-v, MarcoFalke
    Concept ACK hebasto, dergoegge
    Stale ACK Sjors

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  3. DrahtBot added the label Validation on Jul 30, 2023
  4. hebasto commented at 8:52 pm on July 30, 2023: member
    Concept ACK.
  5. dergoegge commented at 7:33 am on July 31, 2023: member
    Concept ACK
  6. maflcko commented at 8:11 am on July 31, 2023: member

    The dbwrapper is a key component of the libbitcoinkernel library. Future users of this library would not want to contend with having the leveldb headers exposed and potentially polluting their project’s namespace.

    Not sure about the motivation. I don’t think any user should include the dbwrapper either? No user should be directly writing or reading from the db. The only reason why they’d have to include the header is to get access to DBOptions, which would alternatively and trivially be achieved by moving DBOptions into a separate header, no?

    And if you are worried about the txdb.h include, maybe do a pimpl there? IIRC #22242 did that, or something like that.

  7. TheCharlatan commented at 9:23 am on July 31, 2023: contributor

    Re #28186 (comment)

    I don’t think any user should include the dbwrapper either? No user should be directly writing or reading from the db.

    It’s hard to speculate on how this will be used in the future. Something calling the db directly for e.g. data science use cases would not be unheard of.

    And if you are worried about the txdb.h include, maybe do a pimpl there? IIRC #22242 did that, or something like that.

    I am also thinking about the base.h include. Though not in the kernel, it pollutes a very significant portion of source files. Implementing a pimpl in base.h could work, but I don’t think that would really be preferable to this PR here. Getting rid of a big, unused besides a single source file include with a bunch of mostly move-only changes seems like a win to me. If this is not sufficient motivation for you though, I’d be happy to work with a reopened #22242, and separate PR for splitting the DBOptions and DBParams.

  8. in src/dbwrapper.h:82 in 4735590b9f outdated
    80@@ -84,6 +81,8 @@ const std::vector<unsigned char>& GetObfuscateKey(const CDBWrapper &w);
    81 
    82 };
    


    maflcko commented at 6:50 am on August 1, 2023:
    unrelated nit in 4735590b9f3bdfc248086e0e20c9d56fa6156e1c: Add missing }; // namespace dbwrapper_private (via clang-format) to clarify the new function is not in the namespace?
  9. in src/dbwrapper.h:104 in 74b5c3dac5 outdated
    102@@ -101,16 +103,9 @@ class CDBBatch
    103     void EraseImpl(DataStream& ssKey);
    104 
    105 public:
    106-    /**
    107-     * @param[in] _parent   CDBWrapper that this batch is to be submitted to
    108-     */
    


    maflcko commented at 7:01 am on August 1, 2023:
    74b5c3dac581b7597668a6a0cc55678d2af296da: Any reason to remove the docs from the header?
  10. in src/dbwrapper.h:316 in 029ac57bf6 outdated
    312@@ -312,10 +313,7 @@ class CDBWrapper
    313     // Get an estimate of LevelDB memory usage (in bytes).
    314     size_t DynamicMemoryUsage() const;
    315 
    316-    CDBIterator *NewIterator()
    317-    {
    318-        return new CDBIterator(*this, pdb->NewIterator(iteroptions));
    319-    }
    320+    CDBIterator *NewIterator();
    


    maflcko commented at 7:15 am on August 1, 2023:
    029ac57bf6a987f50c31bb8c80e77989fc53d85a: nit: missing clang-format?
  11. in src/dbwrapper.cpp:321 in 029ac57bf6 outdated
    316+
    317+CDBIterator::CDBIterator(const CDBWrapper& _parent, std::unique_ptr<IteratorImpl> _piter) : parent(_parent), m_pimpl_iter(std::move(_piter)) {}
    318+
    319+CDBIterator* CDBWrapper::NewIterator()
    320+{
    321+    return new CDBIterator(*this, std::make_unique<CDBIterator::IteratorImpl>(pdb->NewIterator(iteroptions)));
    


    maflcko commented at 7:16 am on August 1, 2023:

    029ac57bf6a987f50c31bb8c80e77989fc53d85a: nit: I don’t think you need to specify CDBIterator:: to make_unique when you are already in the CDBIterator:: scope?

    Also could use {} for the constructor, instead of ()?


    TheCharlatan commented at 3:41 pm on August 1, 2023:
    I think we are in the CDBWrapper:: scope here.
  12. in src/dbwrapper.h:261 in 6005165242 outdated
    256+
    257+        void WriteKeyToStream(DataStream& ssKey) override
    258+        {
    259+            ssKey << key;
    260+        }
    261+    };
    


    maflcko commented at 7:54 am on August 1, 2023:

    6005165242cf090589934133f01f9dbd496612f1: I think you can remove all of those classes/inheritance/constructors by using two lambdas?

     0diff --git a/src/dbwrapper.cpp b/src/dbwrapper.cpp
     1index 94883ec669..71ec3b5f91 100644
     2--- a/src/dbwrapper.cpp
     3+++ b/src/dbwrapper.cpp
     4@@ -300,11 +300,11 @@ std::vector<unsigned char> CDBWrapper::CreateObfuscateKey() const
     5     return ret;
     6 }
     7 
     8-bool CDBWrapper::ReadImpl(CDBWrapper::ReaderBase& reader) const
     9+bool CDBWrapper::ReadImpl(std::function<void(DataStream&)> write_key_to_stream, std::function<void(CDataStream&)> read_value_from_stream) const
    10 {
    11     DataStream ssKey{};
    12     ssKey.reserve(DBWRAPPER_PREALLOC_KEY_SIZE);
    13-    reader.WriteKeyToStream(ssKey);
    14+    write_key_to_stream(ssKey);
    15     leveldb::Slice slKey(CharCast(ssKey.data()), ssKey.size());
    16 
    17     std::string strValue;
    18@@ -318,7 +318,7 @@ bool CDBWrapper::ReadImpl(CDBWrapper::ReaderBase& reader) const
    19     try {
    20         CDataStream ssValue{MakeByteSpan(strValue), SER_DISK, CLIENT_VERSION};
    21         ssValue.Xor(obfuscate_key);
    22-        reader.ReadValueFromStream(ssValue);
    23+        read_value_from_stream(ssValue);
    24     } catch (const std::exception&) {
    25         return false;
    26     }
    27diff --git a/src/dbwrapper.h b/src/dbwrapper.h
    28index 2a2a23c72d..6d3524f063 100644
    29--- a/src/dbwrapper.h
    30+++ b/src/dbwrapper.h
    31@@ -15,6 +15,7 @@
    32 #include <cstddef>
    33 #include <cstdint>
    34 #include <exception>
    35+#include <functional>
    36 #include <leveldb/db.h>
    37 #include <leveldb/options.h>
    38 #include <leveldb/slice.h>
    39@@ -232,35 +233,7 @@ private:
    40     //! whether or not the database resides in memory
    41     bool m_is_memory;
    42 
    43-    class ReaderBase
    44-    {
    45-    public:
    46-        virtual void ReadValueFromStream(CDataStream& ssValue) = 0;
    47-        virtual void WriteKeyToStream(DataStream& ssKey) = 0;
    48-    };
    49-
    50-    template <typename K, typename V>
    51-    class Reader : public ReaderBase
    52-    {
    53-    private:
    54-        const K& key;
    55-        V& value;
    56-
    57-    public:
    58-        Reader(const K& key, V& value) : key{key}, value{value} {}
    59-
    60-        void ReadValueFromStream(CDataStream& ssValue) override
    61-        {
    62-            ssValue >> value;
    63-        }
    64-
    65-        void WriteKeyToStream(DataStream& ssKey) override
    66-        {
    67-            ssKey << key;
    68-        }
    69-    };
    70-
    71-    bool ReadImpl(ReaderBase& reader) const;
    72+    bool ReadImpl(std::function<void(DataStream&)> write_key_to_stream, std::function<void(CDataStream&)> read_value_from_stream) const;
    73 
    74 public:
    75     CDBWrapper(const DBParams& params);
    76@@ -272,8 +245,9 @@ public:
    77     template <typename K, typename V>
    78     bool Read(const K& key, V& value) const
    79     {
    80-        Reader reader{key, value};
    81-        return ReadImpl(reader);
    82+        auto write_key_to_stream{[&key](DataStream& ssKey) { ssKey << key; }};
    83+        auto read_value_from_stream{[&value](CDataStream& ssValue) { ssValue >> value; }};
    84+        return ReadImpl(write_key_to_stream, read_value_from_stream);
    85     }
    86 
    87     template <typename K, typename V>
    

    TheCharlatan commented at 3:41 pm on August 1, 2023:
    Thank you, this is much simpler indeed.
  13. in src/dbwrapper.h:187 in 9fd6debc9f outdated
    198-    //! options used when sync writing to the database
    199-    leveldb::WriteOptions syncoptions;
    200-
    201-    //! the database itself
    202-    leveldb::DB* pdb;
    203+    std::unique_ptr<LevelDBContext> m_db_context;
    


    maflcko commented at 8:06 am on August 1, 2023:

    nit in 9fd6debc9f50955507d1540310ae4928b561bea0: This is assumed to never be null and heavily used. What do you think about an inline private helper:

    0auto& DBContext() { return *Assert(m_db_context); }
    
  14. maflcko commented at 8:07 am on August 1, 2023: member
    Thanks, makes sense. I initially wondered about the 15 commits and +100 LOC, but having more commits for this template-heavy code makes it easier to review, and I found a way to reduce the number of LOC. Just a style nit though, feel free to ignore.
  15. maflcko approved
  16. maflcko commented at 8:07 am on August 1, 2023: member

    review ACK 0280dc44d227ae34e8fcbb708833bfe9d66c7b5f 🔨

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: review ACK 0280dc44d227ae34e8fcbb708833bfe9d66c7b5f 🔨
    3EwQOJEQzugff502JSeduHe1I8VbH7Du0bAvETIvbAxyKfRuGdPQJ9HgpXRv4wHXJz97qTnvIhTIdbkKbnucEAg==
    
  17. in src/dbwrapper.h:95 in 74b5c3dac5 outdated
    89@@ -90,7 +90,9 @@ class CDBBatch
    90 
    91 private:
    92     const CDBWrapper &parent;
    93-    leveldb::WriteBatch batch;
    94+
    95+    struct WriteBatchImpl;
    96+    const std::unique_ptr<WriteBatchImpl> m_pimpl_batch;
    


    Sjors commented at 8:54 am on August 1, 2023:
    74b5c3dac581b7597668a6a0cc55678d2af296da: m_impl_batch (since we don’t use p for pointers) or m_impl (the pattern used in e.g. AddrManImpl)?
  18. in src/dbwrapper.h:75 in ca6f83d88c outdated
    65@@ -67,10 +66,6 @@ class CDBWrapper;
    66  */
    67 namespace dbwrapper_private {
    68 
    69-/** Handle database error by throwing dbwrapper_error exception.
    


    Sjors commented at 9:50 am on August 1, 2023:
    ca6f83d88cb241786676bfb341fc27f6467d39ee: this description, although not amazing, went missing
  19. Sjors approved
  20. Sjors commented at 9:57 am on August 1, 2023: member

    utACK 0280dc44d227ae34e8fcbb708833bfe9d66c7b5f modulo 6005165242cf090589934133f01f9dbd496612f1.

    I also checked that all the intermediate commits compile and pass the tests (on Ubuntu 23.04).

    I’m not familiar with the LevelBD code, but the refactor makes sense to me.

    I’ll look at 6005165242cf090589934133f01f9dbd496612f1 later since it might get simplified based on @MarcoFalke’s suggestion.

  21. maflcko commented at 10:31 am on August 1, 2023: member

    I’d be happy to work with a reopened #22242

    I guess that pull is mostly orthogonal (shouldn’t (silently) conflict with this one). Rebased and reopened in https://github.com/bitcoin/bitcoin/pull/28195

  22. in src/dbwrapper.cpp:36 in 4735590b9f outdated
    26@@ -27,6 +27,11 @@
    27 #include <memory>
    28 #include <optional>
    29 
    30+bool DestroyDB(std::string& path_str)
    


    stickies-v commented at 11:35 am on August 1, 2023:
    nit: const std::string&?
  23. in src/dbwrapper.h:90 in 93099c4ae6 outdated
     96@@ -97,6 +97,8 @@ class CDBBatch
     97 
     98     size_t size_estimate{0};
     99 
    100+    void WriteImpl(DataStream& ssKey, CDataStream& ssValue);
    


    stickies-v commented at 2:22 pm on August 1, 2023:

    WriteImpl, ssKey and ssValue are all CDBBatch class members, I’m not sure passing these as parameters makes sense? And if so, I think ssKey should be const.

    At first sight, I’m not entirely sure why ssKey and ssValue are CDBBatch members when they get resized and cleared in every operation, so perhaps cleaning that up is the better way to go, but I don’t like the ambiguity the current implementation introduces.

    And this comment applies pretty much identically to 5c9e87fcde410e6dc1a60f0f138ff842fdbcf328


    maflcko commented at 5:57 pm on August 1, 2023:

    At first sight, I’m not entirely sure why ssKey and ssValue are CDBBatch members when they get resized and cleared in every operation

    Haven’t checked it, but if there is more than one operation, they may save a dealloc+alloc?


    TheCharlatan commented at 9:14 pm on August 1, 2023:

    WriteImpl, ssKey and ssValue are all CDBBatch class members, I’m not sure passing these as parameters makes sense? And if so, I think ssKey should be const.

    This gave me pause too, but it felt clearer to me this way. Let me know if you think passing them arguments here is more confusing.

    At first sight, I’m not entirely sure why ssKey and ssValue are CDBBatch members when they get resized and cleared in every operation, so perhaps cleaning that up is the better way to go, but I don’t like the ambiguity the current implementation introduces.

    AFAIU they don’t get resized in every operation. Clear should not be de-allocating its memory and if it is already at the correct capacity, calling reserve should not be doing anything.


    stickies-v commented at 7:13 pm on August 2, 2023:

    Alright yeah I was a bit too quick with my initial suggestion, the current implementation makes sense from a (de)alloc perspective, thanks for pointing that out both.

    Let me know if you think passing them arguments here is more confusing.

    I still don’t really like it, but I agree that taking these 2 arguments is the most natural interface for WriteImpl. The only reason we have this weirdness is just because Write wants to do some allocation optimization. So I’m okay keeping it like this.

    Another approach that could work is to make ssKey and ssValue static vars? It means we would allocate one additional DataStream in Erase, but I think on the CDBWrapper level that’s no big deal?


    stickies-v commented at 10:49 am on August 3, 2023:

    Yeah, unless any objections this would have my preference, less side effects and easier to understand imo:

     0diff --git a/src/dbwrapper.h b/src/dbwrapper.h
     1index 54c4685c48..48a6c9af00 100644
     2--- a/src/dbwrapper.h
     3+++ b/src/dbwrapper.h
     4@@ -92,9 +92,6 @@ private:
     5     const CDBWrapper &parent;
     6     leveldb::WriteBatch batch;
     7 
     8-    DataStream ssKey{};
     9-    CDataStream ssValue;
    10-
    11     size_t size_estimate{0};
    12 
    13     void WriteImpl(const DataStream& ssKey, CDataStream& ssValue);
    14@@ -103,7 +100,7 @@ public:
    15     /**
    16      * [@param](/bitcoin-bitcoin/contributor/param/)[in] _parent   CDBWrapper that this batch is to be submitted to
    17      */
    18-    explicit CDBBatch(const CDBWrapper& _parent) : parent(_parent), ssValue(SER_DISK, CLIENT_VERSION){};
    19+    explicit CDBBatch(const CDBWrapper& _parent) : parent(_parent){};
    20 
    21     void Clear()
    22     {
    23@@ -114,6 +111,9 @@ public:
    24     template <typename K, typename V>
    25     void Write(const K& key, const V& value)
    26     {
    27+        static DataStream ssKey{};
    28+        static CDataStream ssValue{SER_DISK, CLIENT_VERSION};
    29+
    30         ssKey.reserve(DBWRAPPER_PREALLOC_KEY_SIZE);
    31         ssValue.reserve(DBWRAPPER_PREALLOC_VALUE_SIZE);
    32         ssKey << key;
    33@@ -126,6 +126,8 @@ public:
    34     template <typename K>
    35     void Erase(const K& key)
    36     {
    37+        static DataStream ssKey{};
    38+
    39         ssKey.reserve(DBWRAPPER_PREALLOC_KEY_SIZE);
    40         ssKey << key;
    41         leveldb::Slice slKey(CharCast(ssKey.data()), ssKey.size());
    

    TheCharlatan commented at 11:20 am on August 3, 2023:
    Not sure about making this a static var. Now the allocations survive beyond the lifetime of the CDBBatch objects and I am not sure if this is entirely safe.

    stickies-v commented at 11:45 am on August 3, 2023:
    Good point. Okay, let’s mark as resolved.
  24. TheCharlatan force-pushed on Aug 1, 2023
  25. TheCharlatan commented at 3:41 pm on August 1, 2023: contributor

    Thank you for the reviews @Sjors and @MarcoFalke,

    Updated 0280dc44d227ae34e8fcbb708833bfe9d66c7b5f -> e8a3a0a0d761c8544e20ffc3ee3d4c1029c44518 (cleaveLeveldbHeaders_0 -> cleaveLeveldbHeaders_1, compare)

  26. maflcko commented at 3:52 pm on August 1, 2023: member

    re-ACK e8a3a0a0d761c8544e20ffc3ee3d4c1029c44518 🗞

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: re-ACK e8a3a0a0d761c8544e20ffc3ee3d4c1029c44518  🗞
    3H6bBOIgZy20U7oO/DsSwT8YNb6GfT92ONblBYOrZTkfGEfWVP/VaQ7KBD5xXg9+vmO9WNcCQBEkDecHYcykwDg==
    
  27. DrahtBot requested review from Sjors on Aug 1, 2023
  28. in src/dbwrapper.h:143 in b669a40741 outdated
    139@@ -140,6 +140,8 @@ class CDBIterator
    140     const CDBWrapper &parent;
    141     leveldb::Iterator *piter;
    142 
    143+    void SeekImpl(DataStream& ssKey);
    


    stickies-v commented at 4:08 pm on August 1, 2023:
    nit: const DataStream& ssKey?
  29. Sjors approved
  30. Sjors commented at 5:15 pm on August 1, 2023: member
    re-ACK e8a3a0a0d761c8544e20ffc3ee3d4c1029c44518
  31. in src/dbwrapper.h:239 in f29222d8e1 outdated
    235@@ -237,6 +236,7 @@ class CDBWrapper
    236     bool m_is_memory;
    237 
    238     bool ReadImpl(std::function<void(DataStream&)> write_key_to_stream, std::function<void(CDataStream&)> read_value_from_stream) const;
    239+    bool ExistsImpl(DataStream& ssKey) const;
    


    stickies-v commented at 5:27 pm on August 1, 2023:
    nit: const DataStream& ssKey?

    maflcko commented at 5:44 am on August 2, 2023:
    style nit: const DataStream& seems fine here, but if you want to pass an immutable view of raw bytes, Span<const std::byte> may be better. (Span{ssKey} should do the conversion, but it will likely happen implicitly by the compiler already).

    maflcko commented at 5:44 am on August 2, 2023:
    Also, CharCast can be moved to the cpp file in the last commit, if you want
  32. stickies-v commented at 5:28 pm on August 1, 2023: contributor
    Concept ACK, did a first run-through of the code but will dig into it deeper.
  33. refactor: Wrap DestroyDB in dbwrapper helper
    Wrap leveldb::DestroyDB in a helper function without exposing
    leveldb-specifics.
    
    Also, add missing optional include.
    
    The context of this commit is an effort to decouple the dbwrapper header
    file from leveldb includes. To this end, the includes are moved to the
    dbwrapper implementation file. This is done as part of the kernel
    project to reduce the number of required includes for users of the
    kernel.
    afc534df9a
  34. TheCharlatan force-pushed on Aug 1, 2023
  35. TheCharlatan commented at 9:24 pm on August 1, 2023: contributor

    Thank you for the review @stickies-v,

    Updated e8a3a0a0d761c8544e20ffc3ee3d4c1029c44518 -> 8c4481ed3713931247e4cedcb5733d3598050eb7 (cleaveLeveldbHeaders_1 -> cleaveLeveldbHeaders_2, compare)

  36. in src/dbwrapper.cpp:159 in 8c4481ed37 outdated
    149@@ -127,24 +150,90 @@ static leveldb::Options GetOptions(size_t nCacheSize)
    150     return options;
    151 }
    152 
    153+struct CDBBatch::WriteBatchImpl {
    154+    leveldb::WriteBatch batch;
    155+};
    156+
    157+CDBBatch::CDBBatch(const CDBWrapper& _parent) : parent(_parent), m_impl_batch{std::make_unique<CDBBatch::WriteBatchImpl>()}, ssValue(SER_DISK, CLIENT_VERSION){};
    


    stickies-v commented at 11:57 am on August 3, 2023:
    nit: that’s a pretty long line
  37. TheCharlatan force-pushed on Aug 3, 2023
  38. TheCharlatan commented at 12:59 pm on August 3, 2023: contributor

    Thank you for the re-review @MarcoFalke,

    Updated 8c4481ed3713931247e4cedcb5733d3598050eb7 -> f26396732b940095380d76ed77685e0fb11294b8 (cleaveLeveldbHeaders_2 -> cleaveLeveldbHeaders_3, compare)

    • Addressed @MarcoFalke’s comment, moved CharCast to implementation file
    • Addressed @MarcoFalke’s comment, using Span now instead of DataStream types for passing immutable raw bytes.
  39. in src/dbwrapper.cpp:169 in f26396732b outdated
    164+{
    165+    m_impl_batch->batch.Clear();
    166+    size_estimate = 0;
    167+}
    168+
    169+void CDBBatch::WriteImpl(const Span<const std::byte>& key, CDataStream& ssValue)
    


    maflcko commented at 1:40 pm on August 3, 2023:
    0void CDBBatch::WriteImpl(Span<const std::byte> ssKey, CDataStream& ssValue)
    

    Span is cheap to copy, so there is no need to add const&. Also, it would be better to do the renames in a separate commit. (Maybe at the end for all touched lines in this pull, or not at all). Otherwise, this breaks review options such as --color-moved=dimmed-zebra --color-moved-ws=ignore-all-space

  40. in src/dbwrapper.cpp:1 in 39c5b07ab5 outdated


    stickies-v commented at 2:07 pm on August 3, 2023:

    It’s not immediately obvious to me why we can’t stick to our usual pattern of using a const DataStream& ssKey, CDataStream& ssValue function signature, and instead resort to lambdas as is done here. I think the below code is much more straightforward, but perhaps I’m missing some nuance or drawbacks?

     0diff --git a/src/dbwrapper.cpp b/src/dbwrapper.cpp
     1index 1b2548f262..cb27b739fe 100644
     2--- a/src/dbwrapper.cpp
     3+++ b/src/dbwrapper.cpp
     4@@ -335,11 +335,8 @@ std::vector<unsigned char> CDBWrapper::CreateObfuscateKey() const
     5     return ret;
     6 }
     7 
     8-bool CDBWrapper::ReadImpl(std::function<void(DataStream&)> write_key_to_stream, std::function<void(CDataStream&)> read_value_from_stream) const
     9+bool CDBWrapper::ReadImpl(const DataStream& ssKey, CDataStream& ssValue) const
    10 {
    11-    DataStream ssKey{};
    12-    ssKey.reserve(DBWRAPPER_PREALLOC_KEY_SIZE);
    13-    write_key_to_stream(ssKey);
    14     leveldb::Slice slKey(CharCast(ssKey.data()), ssKey.size());
    15 
    16     std::string strValue;
    17@@ -351,9 +348,8 @@ bool CDBWrapper::ReadImpl(std::function<void(DataStream&)> write_key_to_stream,
    18         HandleError(status);
    19     }
    20     try {
    21-        CDataStream ssValue{MakeByteSpan(strValue), SER_DISK, CLIENT_VERSION};
    22+        ssValue << MakeByteSpan(strValue);
    23         ssValue.Xor(obfuscate_key);
    24-        read_value_from_stream(ssValue);
    25     } catch (const std::exception&) {
    26         return false;
    27     }
    28diff --git a/src/dbwrapper.h b/src/dbwrapper.h
    29index e48d077a80..e9536b8e7a 100644
    30--- a/src/dbwrapper.h
    31+++ b/src/dbwrapper.h
    32@@ -205,7 +205,7 @@ private:
    33     //! whether or not the database resides in memory
    34     bool m_is_memory;
    35 
    36-    bool ReadImpl(std::function<void(DataStream&)> write_key_to_stream, std::function<void(CDataStream&)> read_value_from_stream) const;
    37+    bool ReadImpl(const DataStream& ssKey, CDataStream& ssValue) const;
    38     bool ExistsImpl(const Span<const std::byte>& key) const;
    39     size_t EstimateSizeImpl(const Span<const std::byte>& key1, const Span<const std::byte>& key2) const;
    40     auto& DBContext() const { return *Assert(m_db_context); }
    41@@ -220,9 +220,16 @@ public:
    42     template <typename K, typename V>
    43     bool Read(const K& key, V& value) const
    44     {
    45-        auto write_key_to_stream{[&key](DataStream& ssKey) { ssKey << key; }};
    46-        auto read_value_from_stream{[&value](CDataStream& ssValue) { ssValue >> value; }};
    47-        return ReadImpl(write_key_to_stream, read_value_from_stream);
    48+        DataStream ssKey{};
    49+        ssKey.reserve(DBWRAPPER_PREALLOC_KEY_SIZE);
    50+        ssKey << key;
    51+
    52+        CDataStream ssValue{SER_DISK, CLIENT_VERSION};
    53+        if(ReadImpl(ssKey, ssValue)) {
    54+            ssValue >> value;
    55+            return true;
    56+        }
    57+        return false;
    58     }
    59 
    60     template <typename K, typename V>
    

    TheCharlatan commented at 4:18 pm on August 3, 2023:
    I originally did this to keep the stream reading within the try catch block, while changing as few lines in the implementation as possible. We could just return the string value though in the ReadImpl and then complete the data reading in the Read function.

    stickies-v commented at 4:36 pm on August 3, 2023:
    I generally like your approach of keeping it as clean a move as possible, and in the other commits am okay with foregoing some slight improvements because of that. Here, though, I feel like we’re making both the interface and implementation way too complex because of it, so personally I’d much rather make some more changes.
  41. DrahtBot added the label CI failed on Aug 3, 2023
  42. in src/dbwrapper.cpp:223 in f26396732b outdated
    217+    leveldb::DB* pdb;
    218+};
    219+
    220 CDBWrapper::CDBWrapper(const DBParams& params)
    221-    : m_name{fs::PathToString(params.path.stem())}, m_path{params.path}, m_is_memory{params.memory_only}
    222+    : m_db_context{std::make_unique<LevelDBContext>()}, m_name{fs::PathToString(params.path.stem())}, m_path{params.path}, m_is_memory{params.memory_only}
    


    stickies-v commented at 2:21 pm on August 3, 2023:
    nit: pretty long line
  43. in src/dbwrapper.cpp:223 in f26396732b outdated
    226-    iteroptions.verify_checksums = true;
    227-    iteroptions.fill_cache = false;
    228-    syncoptions.sync = true;
    229-    options = GetOptions(params.cache_bytes);
    230-    options.create_if_missing = true;
    231+    m_db_context = std::make_unique<LevelDBContext>();
    


    stickies-v commented at 2:22 pm on August 3, 2023:
    I think this needs to be removed?
  44. in src/dbwrapper.h:211 in f26396732b outdated
    204@@ -254,6 +205,11 @@ class CDBWrapper
    205     //! whether or not the database resides in memory
    206     bool m_is_memory;
    207 
    208+    bool ReadImpl(std::function<void(DataStream&)> write_key_to_stream, std::function<void(CDataStream&)> read_value_from_stream) const;
    209+    bool ExistsImpl(const Span<const std::byte>& key) const;
    210+    size_t EstimateSizeImpl(const Span<const std::byte>& key1, const Span<const std::byte>& key2) const;
    211+    auto& DBContext() const { return *Assert(m_db_context); }
    


    stickies-v commented at 2:25 pm on August 3, 2023:

    nit: I think this needs a LIFETIMEBOUND.

    Also: suggested alternative name DBContextRef() to highlight that the return value can mutate our CDBWrapper state? No strong view on it though, happy with the current one too, just a suggestion.

  45. in src/dbwrapper.cpp:225 in f26396732b outdated
    227-    iteroptions.fill_cache = false;
    228-    syncoptions.sync = true;
    229-    options = GetOptions(params.cache_bytes);
    230-    options.create_if_missing = true;
    231+    m_db_context = std::make_unique<LevelDBContext>();
    232+    DBContext().penv = nullptr;
    


    stickies-v commented at 2:33 pm on August 3, 2023:

    nit: Would this be an improvement, to highlight the reference nature of db_context, as well as I think generally it’s slightly cleaner to not call DBContext() every time (I know it’s a very cheap call)? Not sure though, would be a lot of change, happy to keep as is.

    0    auto& db_context{DBContext()};
    1    db_context.penv = nullptr;
    
  46. TheCharlatan force-pushed on Aug 3, 2023
  47. TheCharlatan commented at 2:42 pm on August 3, 2023: contributor

    Updated f26396732b940095380d76ed77685e0fb11294b8 -> 3fb2dac2ce78092c1006ee082c536bea1b69a152 (cleaveLeveldbHeaders_3 -> cleaveLeveldbHeaders_4, compare)

    • Addressed @MarcoFalke’s comment, changed const Span<const std::byte>& to Span<const std::byte>.
    • Split out renaming of some of the key variables into a separate commit at the end.
  48. in src/dbwrapper.cpp:289 in 3fb2dac2ce outdated
    293+    delete DBContext().options.info_log;
    294+    DBContext().options.info_log = nullptr;
    295+    delete DBContext().options.block_cache;
    296+    DBContext().options.block_cache = nullptr;
    297+    delete DBContext().penv;
    298+    DBContext().options.env = nullptr;
    


    stickies-v commented at 2:46 pm on August 3, 2023:
    nit: would it make sense to move this to the LevelDBContext destructor?

    TheCharlatan commented at 8:19 am on August 5, 2023:
    It would, but then we’d also have to forward-declare a destructor class for it and I don’t think it’s really worth doing that.

    stickies-v commented at 1:38 pm on August 7, 2023:
    Fair enough. Can easily improve later if/when necessary.
  49. stickies-v commented at 2:48 pm on August 3, 2023: contributor
    light crACK 3fb2dac2ce78092c1006ee082c536bea1b69a152
  50. DrahtBot requested review from maflcko on Aug 3, 2023
  51. DrahtBot requested review from Sjors on Aug 3, 2023
  52. DrahtBot removed the label CI failed on Aug 3, 2023
  53. in src/dbwrapper.h:187 in 3fb2dac2ce outdated
    201-    //! options used when sync writing to the database
    202-    leveldb::WriteOptions syncoptions;
    203-
    204-    //! the database itself
    205-    leveldb::DB* pdb;
    206+    std::unique_ptr<LevelDBContext> m_db_context;
    


    stickies-v commented at 1:23 pm on August 4, 2023:
    nit: missing docstring
  54. stickies-v approved
  55. stickies-v commented at 1:54 pm on August 4, 2023: contributor

    ACK 3fb2dac2ce78092c1006ee082c536bea1b69a152

    I would however strongly prefer to simplify the lambda logic, and probably this duplication should also be removed.

  56. refactor: Split dbwrapper CDBBatch::Write implementation
    Keep the generic serialization in the header, while moving
    leveldb-specifics to the implementation file.
    
    The context of this commit is an effort to decouple the dbwrapper header
    file from leveldb includes. To this end, the includes are moved to the
    dbwrapper implementation file. This is done as part of the kernel
    project to reduce the number of required includes for users of the
    kernel.
    532ee812a4
  57. refactor: Split dbwrapper CDBatch::Erase implementation
    Keep the generic serialization in the header, while moving
    leveldb-specifics to the implementation file.
    
    The context of this commit is an effort to decouple the dbwrapper header
    file from leveldb includes. To this end, the includes are moved to the
    dbwrapper implementation file. This is done as part of the kernel
    project to reduce the number of required includes for users of the
    kernel.
    b9870c920d
  58. refactor: Pimpl leveldb::batch for CDBBatch
    Hide the leveldb::WriteBatch member variable with a pimpl in order not
    to expose it directly in the header.
    
    Also move CDBBatch::Clear to the dbwrapper implementation to use the new
    impl_batch.
    
    The context of this commit is an effort to decouple the dbwrapper header
    file from leveldb includes. To this end, the includes are moved to the
    dbwrapper implementation file. This is done as part of the kernel
    project to reduce the number of required includes for users of the
    kernel.
    ea8135de7e
  59. refactor: Split dbwrapper CDBIterator::Seek implementation
    Keep the generic serialization in the header, while moving
    leveldb-specifics to the implementation file.
    
    The context of this commit is an effort to decouple the dbwrapper header
    file from leveldb includes. To this end, the includes are moved to the
    dbwrapper implementation file. This is done as part of the kernel
    project to reduce the number of required includes for users of the
    kernel.
    d7437908cd
  60. refactor: Split dbwrapper CDBIterator::GetKey implementation
    Keep the generic serialization in the header, while moving
    leveldb-specifics to the implementation file.
    
    The context of this commit is an effort to decouple the dbwrapper header
    file from leveldb includes. To this end, the includes are moved to the
    dbwrapper implementation file. This is done as part of the kernel
    project to reduce the number of required includes for users of the
    kernel.
    b7a1ab5cb4
  61. refactor: Split dbwrapper CDBIterator::GetValue implementation
    Keep the generic serialization in the header, while moving
    leveldb-specifics to the implementation file.
    
    The context of this commit is an effort to decouple the dbwrapper header
    file from leveldb includes. To this end, the includes are moved to the
    dbwrapper implementation file. This is done as part of the kernel
    project to reduce the number of required includes for users of the
    kernel.
    ef941ff128
  62. refactor: Pimpl leveldb::Iterator for CDBIterator
    Hide the leveldb::Iterator member variable with a pimpl in order not to
    expose it directly in the header.
    
    Also, move CDBWrapper::NewIterator to the dbwrapper implementation to
    use the pimpl for CDBIterator initialziation.
    
    The context of this commit is an effort to decouple the dbwrapper header
    file from leveldb includes. To this end, the includes are moved to the
    dbwrapper implementation file. This is done as part of the kernel
    project to reduce the number of required includes for users of the
    kernel.
    e4af2408f2
  63. refactor: Split dbwrapper CDBWrapper::Read implementation
    Keep the generic serialization in the header, while moving
    leveldb-specifics to the implementation file.
    
    The context of this commit is an effort to decouple the dbwrapper header
    file from leveldb includes. To this end, the includes are moved to the
    dbwrapper implementation file. This is done as part of the kernel
    project to reduce the number of required includes for users of the
    kernel.
    84058e0eed
  64. refactor: Fix logging.h includes
    These were uncovered as missing by the next commit.
    a5c2eb5748
  65. refactor: Split dbwrapper CDBWrapper::Exists implementation
    Keep the generic serialization in the header, while moving
    leveldb-specifics to the implementation file.
    
    The context of this commit is an effort to decouple the dbwrapper header
    file from leveldb includes. To this end, the includes are moved to the
    dbwrapper implementation file. This is done as part of the kernel
    project to reduce the number of required includes for users of the
    kernel.
    dede0eef7a
  66. refactor: Move HandleError to dbwrapper implementation
    Make it a static function in dbwrapper.cpp, since it is not used
    elsewhere and when left in the header, would expose a leveldb type.
    
    The context of this commit is an effort to decouple the dbwrapper header
    file from leveldb includes. To this end, the includes are moved to the
    dbwrapper implementation file. This is done as part of the kernel
    project to reduce the number of required includes for users of the
    kernel.
    586448888b
  67. refactor: Split dbwrapper CDBWrapper::EstimateSize implementation
    Keep the generic serialization in the header, while moving
    leveldb-specifics to the implementation file.
    
    Since CharCast is no longer needed in the header, move it to the
    implementation file.
    
    The context of this commit is an effort to decouple the dbwrapper header
    file from leveldb includes. To this end, the includes are moved to the
    dbwrapper implementation file. This is done as part of the kernel
    project to reduce the number of required includes for users of the
    kernel.
    c534a615e9
  68. refactor: Move CDBWrapper leveldb members to their own context struct
    The context of this commit is an effort to decouple the dbwrapper header
    file from leveldb includes. To this end, the includes are moved to the
    dbwrapper implementation file. This is done as part of the kernel
    project to reduce the number of required includes for users of the
    kernel.
    c95b37d641
  69. build: Remove leveldb from BITCOIN_INCLUDES
    Since leveldb is no longer in our header tree, move its include flags to
    whereever dbwrapper.cpp is built.
    be8f159ac5
  70. refactor: Correct dbwrapper key naming
    The ss- prefix should connotate a DataStream variable. Now that these
    variables are byte spans, drop the prefix.
    d8f1222ac5
  71. TheCharlatan force-pushed on Aug 5, 2023
  72. TheCharlatan commented at 10:02 am on August 5, 2023: contributor

    Thank you for the review @stickies-v,

    Updated 3fb2dac2ce78092c1006ee082c536bea1b69a152 -> 1be04724a3ac061859118c09b90e2e15ea8d04b0 (cleaveLeveldbHeaders_4 -> cleaveLeveldbHeaders_5, compare)

    • Addressed @stickies-v’s comment_1 and comment_2, added some line breaks in constructor lists.
    • Addressed @stickies-v’s comment, further split up the CDBWrapper::Read implementation. The ReadImpl now returns an option of the read database value string.
    • Addressed @stickies-v’s comment, removed left over m_db_context initializer.
    • Addressed @stickies-v’s comment, added LIFETIMEBOUND to DBContext reference getter.
    • Addressed @stickies-v’s comment, added docstring to m_db_contex CDBWrapper member variable.
  73. in src/dbwrapper.h:18 in 1be04724a3 outdated
    21-#include <leveldb/iterator.h>
    22-#include <leveldb/options.h>
    23-#include <leveldb/slice.h>
    24-#include <leveldb/status.h>
    25-#include <leveldb/write_batch.h>
    26+#include <functional>
    


    stickies-v commented at 11:00 am on August 7, 2023:
    nit: No longer needed, I think? (in 9a63566be5445e7026517f38f10ed760a90dbe2f)
  74. stickies-v approved
  75. stickies-v commented at 11:05 am on August 7, 2023: contributor

    re-ACK 1be04724a3ac061859118c09b90e2e15ea8d04b0

    nit: these don’t seem to actually have been addressed

    Addressed @stickies-v’s #28186 (review) and #28186 (review), added some line breaks in constructor lists.

  76. TheCharlatan force-pushed on Aug 7, 2023
  77. TheCharlatan commented at 1:34 pm on August 7, 2023: contributor

    Updated 1be04724a3ac061859118c09b90e2e15ea8d04b0 -> d8f1222ac50f089a0af29eaf8ce0555bad8366ef (cleaveLeveldbHeaders_5 -> cleaveLeveldbHeaders_6, compare)

    Sorry for the re-push @stickies-v, did not include two changes in my previous push by mistake that should have addressed your latest comments. Pushed now.

  78. in src/dbwrapper.cpp:147 in ea8135de7e outdated
    142+    leveldb::WriteBatch batch;
    143+};
    144+
    145+CDBBatch::CDBBatch(const CDBWrapper& _parent) : parent(_parent),
    146+                                                m_impl_batch{std::make_unique<CDBBatch::WriteBatchImpl>()},
    147+                                                ssValue(SER_DISK, CLIENT_VERSION){};
    


    maflcko commented at 3:13 pm on August 7, 2023:
    style nit in ea8135de7e617259cda3fc7b1c8e7569d454fd57: May be better to add a newline before : to reduce the leading whitespace bloating block.
  79. in src/dbwrapper.cpp:392 in c95b37d641 outdated
    387@@ -365,11 +388,12 @@ struct CDBIterator::IteratorImpl {
    388     explicit IteratorImpl(leveldb::Iterator* _iter) : iter{_iter} {}
    389 };
    390 
    391-CDBIterator::CDBIterator(const CDBWrapper& _parent, std::unique_ptr<IteratorImpl> _piter) : parent(_parent), m_impl_iter(std::move(_piter)) {}
    392+CDBIterator::CDBIterator(const CDBWrapper& _parent, std::unique_ptr<IteratorImpl> _piter) : parent(_parent),
    393+                                                                                            m_impl_iter(std::move(_piter)) {}
    


    maflcko commented at 3:24 pm on August 7, 2023:
    c95b37d641b1eed4a62d55ca5342a6ed8c7a1ce7: same
  80. maflcko approved
  81. maflcko commented at 3:24 pm on August 7, 2023: member

    ACK d8f1222ac50f089a0af29eaf8ce0555bad8366ef 🔠

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: ACK d8f1222ac50f089a0af29eaf8ce0555bad8366ef  🔠
    39kmrmmglu0pLTAnlkFVLGRaV07BiskXJWhuBAXoJ8FtylkMHfRojj+S77g6nhGaMuZlTOzX7YETCJ4ZtX4yEAw==
    
  82. fanquake merged this on Aug 7, 2023
  83. fanquake closed this on Aug 7, 2023

  84. sidhujag referenced this in commit 1a49aed60e on Aug 9, 2023
  85. bitcoin locked this on Aug 6, 2024

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-12-21 15:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me