Structure Packing Optimizations in C{,Mutable}Transaction #8330

pull JeremyRubin wants to merge 1 commits into bitcoin:master from JeremyRubin:structurepacking changing 2 files +6 −6
  1. JeremyRubin commented at 7:38 PM on July 11, 2016: contributor

    These commits revise the layout of a few key classes to eliminate padding, eliminating useless memory overhead.

    -This reduces CTransaction from 96 bytes to 88 bytes

  2. in src/primitives/transaction.h:None in 7118b17d55 outdated
     432 | @@ -433,15 +433,14 @@ class CTransaction
     433 |  
     434 |      void UpdateHash() const;
     435 |  };
     436 | -
    


    jonasschnelli commented at 12:01 PM on July 12, 2016:

    nit: we mostly have newlines between classes.

  3. in src/txmempool.h:None in 7118b17d55 outdated
     109 | +    // Space that is typically used as padding. Commented out so that it doesn't cause extra memory use on different architectures.
     110 | +    // char PADDING_FREE_SPACE[2];
     111 | +
     112 | +    // Bool flags
     113 | +    struct CTxMemPoolEntryFlags {
     114 | +        int UNUSED_BITS : 6;
    


    jonasschnelli commented at 12:06 PM on July 12, 2016:

    What's the purpose of UNUSED_BITS?

    Maybe a uint32_t bitmap would be the better choice here?


    JeremyRubin commented at 1:45 PM on July 12, 2016:

    There's no purpose, just to demarcate that these bits are available for future programmers. A uint32_t would be 4 bytes whereas this struct should be 1 byte.


    jonasschnelli commented at 2:01 PM on July 12, 2016:

    Sorry.. overlooked that you are using a c++ bitfield. But I guess the struct will still consume 4 bytes.


    sipa commented at 2:04 PM on July 12, 2016:

    Bitfields are standard C.

    The size of the struct will be 1 byte.

    However, the alignment of the entire CTxMempool object will still be 4, so there will be 3 padding bytes at the end of CTxMempool, I think?


    JeremyRubin commented at 2:40 PM on July 12, 2016:

    @sipa are you sure of the alignment? On my system it is 8 bytes, which leaves no padding for the CTxMemPoolEntry. Perhaps you're on OSX where size_t is 4 bytes not 8 (I didn't look at OSX).


    sipa commented at 12:40 PM on August 25, 2016:

    @jeremyrubin size_t is the size of a pointer, which is 32 bits on 32 bits architectures and 64 bits on 64 bits architectures. You're right that the overall alignment is 8 bytes (on my architecture at least).

    It seems the flags field becomes 4 bytes anyway, so there is no padding afterwards.

  4. jonasschnelli added the label Refactoring on Jul 12, 2016
  5. jonasschnelli commented at 12:07 PM on July 12, 2016: contributor

    Concept ACK

  6. sipa commented at 2:43 PM on July 12, 2016: member

    No, I'm not. I haven't actually tried. But I assume the int before the bitfield is 4-aligned, which would make the end of the bitfield byte be at a multiple of 4 plus 1?

  7. gmaxwell commented at 3:26 AM on December 5, 2016: contributor

    Needs rebase. Concept ACK, on the placeholder bitfield, perhaps better to make that a comment?

  8. JeremyRubin force-pushed on Jan 10, 2017
  9. JeremyRubin force-pushed on Jan 10, 2017
  10. JeremyRubin commented at 10:54 PM on January 10, 2017: contributor

    rebased and addressed feedback

  11. JeremyRubin force-pushed on Mar 27, 2017
  12. JeremyRubin renamed this:
    WIP: Structure Packing Optimizations in CTransaction and CTxMemPoolEntry
    Structure Packing Optimizations in CTransaction and CTxMemPoolEntry
    on Mar 27, 2017
  13. JeremyRubin commented at 4:11 PM on March 27, 2017: contributor

    Rebased! (removing priority conflicted with this)

  14. in src/txmempool.h:93 in 0b5ed42803 outdated
      88 | @@ -94,7 +89,17 @@ class CTxMemPoolEntry
      89 |      uint64_t nSizeWithAncestors;
      90 |      CAmount nModFeesWithAncestors;
      91 |      int64_t nSigOpCostWithAncestors;
      92 | +    size_t nTxWeight;          //!< ... and avoid recomputing tx weight (also used for GetTxSize())
      93 | +    size_t nModSize;           //!< ... and modified size for priority
    


    ryanofsky commented at 10:00 PM on March 27, 2017:

    In commit "Re-pack the CTxMempoolEntry to be closer to optimally packed":

    This isn't used, guessing bad rebase.


    JeremyRubin commented at 5:35 AM on March 28, 2017:

    yeah... hmm :/

    Will fix.

  15. in src/txmempool.h:101 in 0b5ed42803 outdated
      96 | +    mutable size_t vTxHashesIdx; //!< Index in mempool's vTxHashes
      97 | +private:
      98 | +    unsigned int entryHeight;  //!< Chain height when entering the mempool
      99 |  
     100 | +    // Bool flags
     101 | +    bool hadNoDependencies;    //!< Not dependent on any other txs when it entered the mempool
    


    ryanofsky commented at 10:01 PM on March 27, 2017:

    In commit "Re-pack the CTxMempoolEntry to be closer to optimally packed":

    Also unused.

  16. in src/primitives/transaction.cpp:80 in 16f62fe7c9 outdated
      75 | @@ -76,9 +76,12 @@ uint256 CTransaction::GetWitnessHash() const
      76 |  }
      77 |  
      78 |  /* For backward compatibility, the hash is initialized to 0. TODO: remove the need for this default constructor entirely. */
      79 | -CTransaction::CTransaction() : nVersion(CTransaction::CURRENT_VERSION), vin(), vout(), nLockTime(0), hash() {}
      80 | -CTransaction::CTransaction(const CMutableTransaction &tx) : nVersion(tx.nVersion), vin(tx.vin), vout(tx.vout), nLockTime(tx.nLockTime), hash(ComputeHash()) {}
    


    ryanofsky commented at 10:09 PM on March 27, 2017:

    In commit "Reorder C{,Mutable}Transaction for better packing":

    Seems like you could remove ComputeHash now, it doesn't appear to be called anywhere else.

  17. in src/txmempool.h:108 in 117dde16ec outdated
     105 |      // Bool flags
     106 | -    bool hadNoDependencies;    //!< Not dependent on any other txs when it entered the mempool
     107 | -    bool spendsCoinbase;       //!< keep track of transactions that spend a coinbase
     108 | +    struct CTxMemPoolEntryFlags {
     109 | +        //int UNUSED_BITS : 7;
     110 | +        bool spendsCoinbase : 1;       //!< keep track of transactions that spend a coinbase
    


    ryanofsky commented at 10:13 PM on March 27, 2017:

    Kind of a lonely bitfield now with just one bool. But maybe it will have some more flags later.


    JeremyRubin commented at 5:35 AM on March 28, 2017:

    yeah, hopefully inspires whoever adds a flag in the future...

  18. in src/txmempool.cpp:24 in 0b5ed42803 outdated
      19 | @@ -20,9 +20,10 @@
      20 |  
      21 |  CTxMemPoolEntry::CTxMemPoolEntry(const CTransactionRef& _tx, const CAmount& _nFee,
      22 |                                   int64_t _nTime, unsigned int _entryHeight,
      23 | -                                 bool _spendsCoinbase, int64_t _sigOpsCost, LockPoints lp):
      24 | -    tx(_tx), nFee(_nFee), nTime(_nTime), entryHeight(_entryHeight),
      25 | -    spendsCoinbase(_spendsCoinbase), sigOpCost(_sigOpsCost), lockPoints(lp)
      26 | +                                 bool _spendsCoinbase, int64_t _sigOpsCost, LockPoints lp): 
      27 | +lockPoints(lp), tx(_tx), nFee(_nFee)
    


    ryanofsky commented at 12:38 PM on March 28, 2017:

    In commit "Re-pack the CTxMempoolEntry to be closer to optimally packed"

    Unusual indentation here, maybe reformat.

  19. in src/txmempool.h:110 in 16f62fe7c9 outdated
     105 | +    // Bool flags
     106 | +    struct CTxMemPoolEntryFlags {
     107 | +        //int UNUSED_BITS : 7;
     108 | +        bool spendsCoinbase : 1;       //!< keep track of transactions that spend a coinbase
     109 | +    };
     110 | +    CTxMemPoolEntryFlags flags;
    


    TheBlueMatt commented at 6:04 PM on July 11, 2017:

    Ehh, why not just have the single bool spendsCoinbase : 1 here instead of making a struct for it?


    JeremyRubin commented at 6:17 PM on July 12, 2017:

    It's somewhat vestigial; iirc when I originally wrote this patch last year there was another bool that got removed in a later PR.

    The only real remaining reason is to make the code "self documenting" so that when/if someone later adds another flag to mempool (or even adds the flag back), they are encouraged to dtrt rather than adding more memory bloat.


    TheBlueMatt commented at 6:52 PM on July 12, 2017:

    Is there harm in having bool spendsCoinbase : 1; bool newFlags : 1; instead of encapsulating the booleans in a flags struct?


    JeremyRubin commented at 11:40 PM on July 12, 2017:

    Is that valid c++? If so, o_0


    sipa commented at 11:42 PM on July 12, 2017:

    In what way is that different from what you're doing now?

  20. sipa commented at 7:25 PM on July 12, 2017: member

    This reduces CTransaction from 96 bytes to 88 bytes for me (great), but increases CTxMemPoolEntry from 160 bytes to 168 bytes (not so great). NACK as long as that is the case.

  21. JeremyRubin force-pushed on Jul 13, 2017
  22. JeremyRubin force-pushed on Jul 13, 2017
  23. Reorder C{,Mutable}Transaction for better packing 37495e0d8d
  24. JeremyRubin force-pushed on Jul 13, 2017
  25. JeremyRubin commented at 12:24 AM on July 13, 2017: contributor

    @sipa I rewrote this PR to repack the CTransaction versions via a minimal patch.

    I'm not sure what was going on with the CTxMemPoolEntry version, I just think it became stale. At the time I wrote the patch initially they were 192 bytes, so 160 bytes/entry is a good improvement anyways (perhaps from removing priority).

  26. sipa commented at 1:07 AM on July 13, 2017: member

    utACK 37495e0d8d4bd98ae04364eae2f4ecb7084a9234. Verified with printf("%i\n", sizeof(CTransaction)) before and after that this indeed reduces the size of the type.

  27. TheBlueMatt commented at 3:30 PM on July 14, 2017: member

    utACK 37495e0d8d4bd98ae04364eae2f4ecb7084a9234

  28. ryanofsky commented at 4:15 PM on July 25, 2017: member

    utACK 37495e0d8d4bd98ae04364eae2f4ecb7084a9234. Should update outdated PR description though to avoid misleading text in git history.

  29. JeremyRubin renamed this:
    Structure Packing Optimizations in CTransaction and CTxMemPoolEntry
    Structure Packing Optimizations in C{,Mutable}Transaction
    on Jul 31, 2017
  30. JeremyRubin commented at 12:46 AM on July 31, 2017: contributor

    PR text updates done. Original text preserved below.

    These commits revise the layout of a few key classes to eliminate padding, eliminating useless memory overhead.

    CTxMemPoolEntry is decreased to 184 bytes from 192. CTransaction is decreased to 112 bytes from 120.

    Furthermore, it is clarified in CTxMempoolEntry where the extra space is (2 bytes) for future overhead-free use.

    The introduction of CTxMemPoolEntryFlags is used to bit-pack the bool flags, making room for 6 more future-flags. I have not performance tested this, but it should not be much. It does not save any space at the moment due to padding, but if more bools are added it would compared to naive.

    To finish this work, the following should be done: profiling of access patterns to class fields to guarantee cache-line co-residency, cross architecture padding optimizations.

  31. gmaxwell approved
  32. gmaxwell commented at 5:14 PM on July 31, 2017: contributor

    utACK

  33. JeremyRubin commented at 5:51 PM on September 6, 2017: contributor

    @laanwj merge?

  34. laanwj merged this on Sep 6, 2017
  35. laanwj closed this on Sep 6, 2017

  36. laanwj referenced this in commit 59e17899a7 on Sep 6, 2017
  37. PastaPastaPasta referenced this in commit 73cbb0ba95 on Dec 22, 2019
  38. PastaPastaPasta referenced this in commit ab8fd7f477 on Jan 2, 2020
  39. PastaPastaPasta referenced this in commit 6352a0c76b on Jan 4, 2020
  40. PastaPastaPasta referenced this in commit ddfc72c2bd on Jan 4, 2020
  41. PastaPastaPasta referenced this in commit e5ebbd1074 on Jan 10, 2020
  42. PastaPastaPasta referenced this in commit 0980cc9e7d on Jan 10, 2020
  43. PastaPastaPasta referenced this in commit 6962568009 on Jan 10, 2020
  44. PastaPastaPasta referenced this in commit 17a402d93d on Jan 12, 2020
  45. ckti referenced this in commit 431f9c5c15 on Mar 28, 2021
  46. furszy referenced this in commit a306efd8e6 on Jul 19, 2021
  47. DrahtBot locked this on Sep 8, 2021

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-04-15 21:16 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me