move-only: move coins statistics utils out of RPC #16728

pull jamesob wants to merge 1 commits into bitcoin:master from jamesob:2019-08-au-coinstats changing 4 files +113 −71
  1. jamesob commented at 3:32 PM on August 26, 2019: member

    This is part of the assumeutxo project:

    Parent PR: #15606 Issue: #15605 Specification: https://github.com/jamesob/assumeutxo-docs/tree/master/proposal


    In the short-term, this move-only commit will help with fuzzing (https://github.com/bitcoin/bitcoin/pull/15606#issuecomment-524482297). Later, these procedures will be used to compute statistics (particularly a content hash) for UTXO sets coming in from snapshots.

    Most easily reviewed with git ... --color-moved=dimmed_zebra. A nice follow-up would be adding unittests, which I'll do if nobody else gets around to it.

  2. in src/Makefile.am:266 in 0afc435af3 outdated
     262 | @@ -262,6 +263,7 @@ libbitcoin_server_a_SOURCES = \
     263 |    banman.cpp \
     264 |    blockencodings.cpp \
     265 |    blockfilter.cpp \
     266 | +  coinstats.cpp \
    


    MarcoFalke commented at 3:47 PM on August 26, 2019:

    please keep this list sorted, as before

  3. in src/coinstats.h:10 in 0afc435af3 outdated
       5 | +
       6 | +#ifndef BITCOIN_COINSTATS_H
       7 | +#define BITCOIN_COINSTATS_H
       8 | +
       9 | +#include <coins.h>
      10 | +#include <chain.h>
    


    MarcoFalke commented at 3:52 PM on August 26, 2019:

    This includes serialize.h, which takes quite a long time to parse. Why not replace those two includes with a simple forward decl?

    +#include <amount.h>
    +class CBlockIndex;
    +class CCoinsView;
    
  4. in src/coinstats.h:15 in 0afc435af3 outdated
      10 | +#include <chain.h>
      11 | +#include <uint256.h>
      12 | +#include <sync.h>
      13 | +
      14 | +extern CCriticalSection cs_main;
      15 | +CBlockIndex* LookupBlockIndex(const uint256& blockhash);
    


    MarcoFalke commented at 3:53 PM on August 26, 2019:

    what is this?


    MarcoFalke commented at 5:21 PM on August 26, 2019:
    In file included from rpc/blockchain.cpp:33:
    ./validation.h:406:14: warning: redundant redeclaration of ‘CBlockIndex* LookupBlockIndex(const uint256&)’ in same scope [-Wredundant-decls]
      406 | CBlockIndex* LookupBlockIndex(const uint256& hash) EXCLUSIVE_LOCKS_REQUIRED(cs_main);
          |              ^~~~~~~~~~~~~~~~
    In file included from rpc/blockchain.cpp:13:
    ./coinstats.h:18:14: note: previous declaration of ‘CBlockIndex* LookupBlockIndex(const uint256&)’
       18 | CBlockIndex* LookupBlockIndex(const uint256& blockhash);
          |              ^~~~~~~~~~~~~~~~
    
  5. MarcoFalke commented at 3:53 PM on August 26, 2019: member

    Concept ACK

  6. DrahtBot added the label Build system on Aug 26, 2019
  7. DrahtBot added the label Refactoring on Aug 26, 2019
  8. DrahtBot added the label RPC/REST/ZMQ on Aug 26, 2019
  9. DrahtBot added the label UTXO Db and Indexes on Aug 26, 2019
  10. MarcoFalke removed the label Build system on Aug 26, 2019
  11. MarcoFalke removed the label RPC/REST/ZMQ on Aug 26, 2019
  12. MarcoFalke removed the label UTXO Db and Indexes on Aug 26, 2019
  13. jamesob commented at 5:43 PM on August 26, 2019: member

    Thanks for the quick feedback @MarcoFalke. Fixed and pushed.

  14. jamesob force-pushed on Aug 26, 2019
  15. in src/coinstats.cpp:50 in 07da7c422d outdated
      45 | +    CHashWriter ss(SER_GETHASH, PROTOCOL_VERSION);
      46 | +    stats.hashBlock = pcursor->GetBestBlock();
      47 | +    {
      48 | +        LOCK(cs_main);
      49 | +        CBlockIndex* block = LookupBlockIndex(stats.hashBlock);
      50 | +        stats.nHeight = block ? block->nHeight : -1;
    


    ryanofsky commented at 5:56 PM on August 26, 2019:

    In commit "move-only: move coins statistics utils out of RPC" (07da7c422d875e54176a81b37de011a4761966cc)

    Note: previous line was

    stats.nHeight = LookupBlockIndex(stats.hashBlock)->nHeight;
    

    which crashes if LookupBlockIndex returns null. Otherwise there's no change.


    MarcoFalke commented at 6:11 PM on August 26, 2019:

    This should never crash, unless there is a data corruption. I'd prefer to keep the previous code or throw an exception instead of withholding the fact that data is corrupted from the user.


    MarcoFalke commented at 8:29 PM on August 26, 2019:

    Alternatively, you could use error(), as done later on in this function. I don't think the current code makes sense, unless you can give an explanation or code comment to motivate it


    jamesob commented at 3:39 PM on August 27, 2019:

    Ah my mistake, I don't actually remember changing this code (it's been a while since I wrote the original patch). I'll revert to current behavior to keep move-onlyness.

  16. in src/Makefile.am:267 in 07da7c422d outdated
     263 | @@ -263,6 +264,7 @@ libbitcoin_server_a_SOURCES = \
     264 |    blockencodings.cpp \
     265 |    blockfilter.cpp \
     266 |    chain.cpp \
     267 | +  coinstats.cpp \
    


    ryanofsky commented at 5:58 PM on August 26, 2019:

    In commit "move-only: move coins statistics utils out of RPC" (07da7c422d875e54176a81b37de011a4761966cc)

    Could consider moving moving these files to src/node/. IMO better to make use of node/, wallet/, qt/, util/, etc subdirectories instead of adding things at the top level. Some notes about this in #15732.


    MarcoFalke commented at 6:08 PM on August 26, 2019:

    Just to be clear on the rationale here: LookupBlockIndex accesses a global that is only properly initialized in the node process. Thus, the coinstats module should be in the src/node folder?


    ryanofsky commented at 7:05 PM on August 26, 2019:

    Just to be clear on the rationale here

    Calling LookupBlockIndex and accessing globals is a more of a rationale for including this code in the libbitcoin_server.a library instead of libbitcoin_util.a or another library.

    There is libbitcoin_server.a code in the top-level directory, but there is also a code from other libraries there. I just think it's nice to use node/, wallet/, qt/, util/, consensus/ subdirectories to organize code so you can know what context the code functions in and what library it will be a part of.


    MarcoFalke commented at 7:13 PM on August 26, 2019:

    more of a rationale for including this code in the libbitcoin_server.a library

    I think we agree then. src/node should eventually be the folder for libbitcoin_server.


    jamesob commented at 7:58 PM on August 26, 2019:

    @ryanofsky thanks for the pointer, forgot about these. Will move.

  17. ryanofsky approved
  18. ryanofsky commented at 6:02 PM on August 26, 2019: member

    utACK 07da7c422d875e54176a81b37de011a4761966cc

  19. in src/coinstats.cpp:19 in 07da7c422d outdated
      14 | +#include <serialize.h>
      15 | +
      16 | +#include <boost/thread.hpp>
      17 | +
      18 | +extern CCriticalSection cs_main;
      19 | +CBlockIndex* LookupBlockIndex(const uint256& blockhash);
    


    MarcoFalke commented at 6:06 PM on August 26, 2019:

    Why are you removing the lock annotation? Can't you just include the header?


    jamesob commented at 7:26 PM on August 26, 2019:

    validation.cpp will eventually include this file (for use validationg snapshots) and I didn't want to create a circular dep. My mistake, forgot this would remove the lock annotation for uses in this file, will add back.


    MarcoFalke commented at 7:41 PM on August 26, 2019:

    This would be the first time that we solved a circular dependency by copy-pasting parts of a header file.

    validation.cpp will eventually include this file

    If, and when, this happens, you can either document the circular dependency or remove the include via this hack. No need to do it now.

  20. DrahtBot commented at 7:12 PM on August 26, 2019: member

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #16345 (rpc: Add getblockbyheight method by emilengler)
    • #15606 ([experimental] UTXO snapshots by jamesob)
    • #13462 (Make SER_GETHASH implicit for CHashWriter and SerializeHash by Empact)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  21. jamesob force-pushed on Aug 26, 2019
  22. jamesob commented at 8:08 PM on August 26, 2019: member

    au.coinstast.1 -> au.coinstast.2 (change)

    Pushed fixes for feedback from Marco and Russ.

  23. jamesob force-pushed on Aug 26, 2019
  24. fanquake added this to the "In progress" column in a project

  25. in src/node/coinstats.h:18 in 8f394b5809 outdated
      13 | +
      14 | +class CCoinsView;
      15 | +
      16 | +struct CCoinsStats
      17 | +{
      18 | +    int nHeight;
    


    ariard commented at 3:56 AM on August 27, 2019:

    Could be an opportunity to comment fields structures like bytes unit of nDiskSize and bitcoin unit of nTotalAmount, why hashSerialized is hash_serilalized_2, what's the purpose of nBogoSize


    promag commented at 2:29 PM on August 27, 2019:

    If you are going to push a separate commit then you could also drop the constructor and initialize members in place, like int nHeight{0};.


    jamesob commented at 3:39 PM on August 27, 2019:

    Good suggestions. Will add if I have to invalidate the move-onlyness for some reason.

  26. ariard approved
  27. ariard commented at 4:07 AM on August 27, 2019: member

    Reviewed and tested ACK 8f394b5

  28. in src/node/coinstats.cpp:49 in 8f394b5809 outdated
      44 | +    CHashWriter ss(SER_GETHASH, PROTOCOL_VERSION);
      45 | +    stats.hashBlock = pcursor->GetBestBlock();
      46 | +    {
      47 | +        LOCK(cs_main);
      48 | +        CBlockIndex* block = LookupBlockIndex(stats.hashBlock);
      49 | +        stats.nHeight = block ? block->nHeight : -1;
    


    Sjors commented at 1:12 PM on August 27, 2019:

    This change deserves its own commit? Assertion works for me too as @MarcoFalke suggested: #16728 (review)


    promag commented at 2:27 PM on August 27, 2019:

    Agree with the separate commit.

  29. Sjors approved
  30. Sjors commented at 1:55 PM on August 27, 2019: member

    ACK 8f394b5809a5ccf042e7ddf4d3921acb3a7a67a2. Happy to see more RPC code move to a seperate and reusable spot. gettxoutsetinfo still works.

  31. in src/node/coinstats.h:31 in 8f394b5809 outdated
      26 | +
      27 | +    CCoinsStats() : nHeight(0), nTransactions(0), nTransactionOutputs(0), nBogoSize(0), nDiskSize(0), nTotalAmount(0) {}
      28 | +};
      29 | +
      30 | +//! Calculate statistics about the unspent transaction output set
      31 | +bool GetUTXOStats(CCoinsView *view, CCoinsStats &stats);
    


    promag commented at 2:28 PM on August 27, 2019:

    Could fix format in this new line (space before arg name).

  32. in src/node/coinstats.h:10 in 8f394b5809 outdated
       5 | +
       6 | +#ifndef BITCOIN_NODE_COINSTATS_H
       7 | +#define BITCOIN_NODE_COINSTATS_H
       8 | +
       9 | +#include <amount.h>
      10 | +#include <sync.h>
    


    promag commented at 2:33 PM on August 27, 2019:

    What is this for?


    jamesob commented at 3:47 PM on August 27, 2019:

    Good point - stale.

  33. in src/node/coinstats.h:20 in 8f394b5809 outdated
      15 | +
      16 | +struct CCoinsStats
      17 | +{
      18 | +    int nHeight;
      19 | +    uint256 hashBlock;
      20 | +    uint64_t nTransactions;
    


    promag commented at 2:33 PM on August 27, 2019:

    Should #include <cstdint>.

  34. promag commented at 2:34 PM on August 27, 2019: member

    Concept ACK, verified it's almost moved-only.

  35. move-only: move coins statistics utils out of RPC
    These procedures will later be used in the ChainstateManager to compute
    statistics (particularly a content hash) for UTXO sets coming in from
    snapshots.
    8a3b2eb175
  36. jamesob force-pushed on Aug 27, 2019
  37. jamesob commented at 3:54 PM on August 27, 2019: member

    au.coinstast.3 -> au.coinstast.4 (change)

    Thanks for the reviews and suggestions, everyone. I've cleaned up some of the imports and reverted the LookupBlockIndex change to restore move-onlyness.

  38. MarcoFalke commented at 4:01 PM on August 27, 2019: member

    ACK 8a3b2eb17572ca2131778d52cc25ec359470a90f, checked --color-moved=dimmed-zebra

    <details><summary>Show signature and timestamp</summary>

    Signature:

    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA512
    
    ACK 8a3b2eb17572ca2131778d52cc25ec359470a90f, checked --color-moved=dimmed-zebra
    -----BEGIN PGP SIGNATURE-----
    
    iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
    pUizvwv9H7WvV2EuXemNemt9vtvZryiXhZSmOxCwxmTEj8j8lxiWvia0sqrck5dR
    sb/b/Yr92kWunZgt2B2xzPnG6twcmR5gfxWGDVSYj0orCUQfqtEJXIDXgfcEHGZi
    gDaPL0Kj8oCYH6Rm5RVkZ7dLcShmRj8kTJWk6oFQFfCOD0LhPCutshZdC3rYcxZQ
    0ZH5AusJUM8YIiqQLOSZkc+658TxwHUV39RHFad5kbozra5kRiQwtUGBpR17jGim
    SPjRnbcv3nNty3tjnlLstC+/WhKh8G+fCro+MSZnuZXqLJ33hy5gA/hy625+PGxR
    /q7ItGPBO9GZpGHfB8unwFp+qgMFckKSNGiMRRt8NgzMDVVQSfE2FRvJDBMFGHN9
    zMNOspTROW7uA1FmW3VCRZJwLIVYfyMBoBNT4TSN7riGuC5ejqBcPnBftPNne/pz
    ME6qyyfjJ6M7ybsZIwX6QrQq/2kYwHpkR9BylefctpnbBgcfvYGM60QsUkdDm8Fq
    c9EoO/St
    =3F0+
    -----END PGP SIGNATURE-----
    

    Timestamp of file with hash 7a6fa2edb4e64bd8de86c3674ec9f4d48567af682d2e26d6152320c838c51897 -

    </details>

  39. in src/node/coinstats.cpp:10 in 8a3b2eb175
       5 | +
       6 | +#include <node/coinstats.h>
       7 | +
       8 | +#include <amount.h>
       9 | +#include <coins.h>
      10 | +#include <chain.h>
    


    promag commented at 4:12 PM on August 27, 2019:

    nit sort :trollface:


    MarcoFalke commented at 4:58 PM on August 27, 2019:

    Can be done the next time this file is touched.

  40. MarcoFalke approved
  41. MarcoFalke commented at 5:03 PM on August 27, 2019: member

    This will be merged after one more ACK

  42. promag commented at 5:03 PM on August 27, 2019: member

    ACK

  43. MarcoFalke referenced this in commit a7be1cc92b on Aug 27, 2019
  44. MarcoFalke merged this on Aug 27, 2019
  45. MarcoFalke closed this on Aug 27, 2019

  46. sidhujag referenced this in commit 717747348e on Aug 27, 2019
  47. fanquake moved this from the "In progress" to the "Done" column in a project

  48. domob1812 referenced this in commit f9d73b92fc on Sep 2, 2019
  49. deadalnix referenced this in commit 310fcb6184 on May 20, 2020
  50. UdjinM6 referenced this in commit 9ab9422d7b on Nov 17, 2020
  51. UdjinM6 referenced this in commit 36d275396f on Dec 1, 2020
  52. PastaPastaPasta referenced this in commit b559a8f904 on Dec 15, 2020
  53. DrahtBot locked this on Dec 16, 2021

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-04-13 15:14 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me