6% faster, 6% less memory: more tightly pack CCoinMap::value_type #16970

pull martinus wants to merge 1 commits into bitcoin:master from martinus:2019-09-more-compact-Coin changing 3 files +49 −27
  1. martinus commented at 5:48 pm on September 26, 2019: contributor

    This change reduces CCoinMap::value_type from 96 bytes to 80 bytes by more tightly packing it’s data. This allows to cache a lot more transactions in the cache. I’ve achieved this with these changes:

    • Refactored prevector so it uses a single byte to determine its size when in direct mode
    • Reduce CScriptBase from 28 to 27 indirect bytes
    • align CTxOut to 4 bytes to prevent padding when used as a member in Coin

    This tighter packing means more data can be stored in the coinsCache before it is full and has to be flushed to disk. In my benchmark, -reindex-chainstate was 6% faster and used 6% less memory. The cache could fit 14% more txo’s before it had to resize.

    Some numbers:

    runtime h:mm:ss max RSS kbyte
    master -dbcache=5000 4:13:59 7696728
    2019-09-more-compact-Coin -dbcache=5000 3:57:59 7192772
    change -6.30% -6.55%

    The graph shows nicely that the cache is able to be used a lot longer before it is full and flushed to the disk: out

    I’ve tried this change in combination with #16957 to not cache hash), and then additionally in combination with #16801 :

    runtime h:mm:ss max RSS kbyte
    master -dbcache=5000 4:13:59 7696728
    2019-09-more-compact-Coin -dbcache=5000 3:57:59 7192772
    2019-09-more-compact-Coin & #16957 -dbcache=5500 3:51:04 6600548
    2019-09-more-compact-Coin & #16957 & #16801 -dbcache=5500 3:38:44 6164380

    out

    With all 3 PR’s applied, reindex was 14% faster and used 20% less memory

  2. more tightly pack Coin
    This change reduces CCoinMap's value_type from 96 bytes to 80 bytes by
    more tightly packing it's data. This is achieved by these changes:
    
    * Refactored prevector so it uses a single byte to determine its size
      when in direct mode
    * Reduce CScriptBase from 28 to 27 indirect bytes
    * align CTxOut to 4 bytes to prevent padding when used as a member in
      Coin
    
    This tighter packing means more data can be stored in the coinsCache
    before it is full and has to be flushed to disk. In my benchmark,
    -reindex-chainstate was 6% faster and used 6% less memory. The cache
    could fit 14% more txo's before it had to resize.
    4e4f8f73bd
  3. laanwj added the label Resource usage on Sep 26, 2019
  4. laanwj added the label Validation on Sep 26, 2019
  5. in src/primitives/transaction.h:137 in 4e4f8f73bd
    128@@ -129,7 +129,12 @@ class CTxIn
    129 
    130 /** An output of a transaction.  It contains the public key that the next input
    131  * must be able to sign with to claim it.
    132+ *
    133+ * We align the struct to 4 bytes: that way, CTxOut will have 36 bytes: 8 for
    134+ * CAmount, and 28 for CScript. As a member of Coin, this leaves 4 bytes for
    135+ * Coin's nHeight without the need for any padding.
    136  */
    137+#pragma pack(push, 4)
    


    TheBlueMatt commented at 6:52 pm on September 26, 2019:
    Does this break on oldschool 32-bit ARM where int reads are required to be aligned (and I presume this includes 64-bit ints?)

    martinus commented at 7:08 pm on September 26, 2019:
    I don’t think it’s a problem, we are using uint32_t everywhere and align to 4 byte, so it should be fine

    TheBlueMatt commented at 10:55 pm on September 26, 2019:
    Hmm? Isn’t nValue an 8-bit integer? Or am I misunerstanding what this does?

    martinus commented at 6:25 am on September 27, 2019:
    yes its 8byte, but I thought it would still be ok because of how it’s used in Coin. I guess I’ve missed something, I really need to set up travis builds… I’ll close this for now until I figure out if/how I can do this correctly
  6. MarcoFalke commented at 7:16 pm on September 26, 2019: member
      0Running tests: blockfilter_index_tests from test/blockfilter_index_tests.cpp
      1
      2Running 5 test cases...
      3
      4Test cases order is shuffled using seed: 906380976
      5
      6Entering test module "Bitcoin Core Test Suite"
      7
      8test/blockfilter_tests.cpp(17): Entering test suite "blockfilter_tests"
      9
     10test/blockfilter_tests.cpp(42): Entering test case "gcsfilter_default_constructor"
     11
     12test/blockfilter_tests.cpp(42): Leaving test case "gcsfilter_default_constructor"; testing time: 299us
     13
     14test/blockfilter_tests.cpp(19): Entering test case "gcsfilter_test"
     15
     16test/blockfilter_tests.cpp(19): Leaving test case "gcsfilter_test"; testing time: 18878us
     17
     18test/blockfilter_tests.cpp(55): Entering test case "blockfilter_basic_test"
     19
     20primitives/transaction.h:155:9: runtime error: reference binding to misaligned address 0x60b0000075f4 for type 'const CAmount' (aka 'const long'), which requires 8 byte alignment
     21
     220x60b0000075f4: note: pointer points here
     23
     24  00 00 00 02 c8 00 00 00  00 00 00 00 76 a9 01 14  88 ac 00 00 00 00 00 00  00 00 00 00 00 00 00 00
     25
     26              ^ 
     27
     28    [#0](/bitcoin-bitcoin/0/) 0x560f2fad3375 in void CTxOut::SerializationOp<CHashWriter, CSerActionSerialize>(CHashWriter&, CSerActionSerialize) /home/travis/build/bitcoin/bitcoin/build/bitcoin-x86_64-unknown-linux-gnu/src/./primitives/transaction.h:155:9
     29
     30    [#1](/bitcoin-bitcoin/1/) 0x560f2fad3072 in void CTxOut::Serialize<CHashWriter>(CHashWriter&) const /home/travis/build/bitcoin/bitcoin/build/bitcoin-x86_64-unknown-linux-gnu/src/./primitives/transaction.h:151:5
     31
     32    [#2](/bitcoin-bitcoin/2/) 0x560f2fad3072 in void Serialize<CHashWriter, CTxOut>(CHashWriter&, CTxOut const&) /home/travis/build/bitcoin/bitcoin/build/bitcoin-x86_64-unknown-linux-gnu/src/./serialize.h:603
     33
     34    [#3](/bitcoin-bitcoin/3/) 0x560f2fad3072 in void Serialize_impl<CHashWriter, CTxOut, std::allocator<CTxOut>, CTxOut>(CHashWriter&, std::vector<CTxOut, std::allocator<CTxOut> > const&, CTxOut const&) /home/travis/build/bitcoin/bitcoin/build/bitcoin-x86_64-unknown-linux-gnu/src/./serialize.h:734
     35
     36    [#4](/bitcoin-bitcoin/4/) 0x560f2fad0f41 in void Serialize<CHashWriter, CTxOut, std::allocator<CTxOut> >(CHashWriter&, std::vector<CTxOut, std::allocator<CTxOut> > const&) /home/travis/build/bitcoin/bitcoin/build/bitcoin-x86_64-unknown-linux-gnu/src/./serialize.h:740:5
     37
     38    [#5](/bitcoin-bitcoin/5/) 0x560f2fad0f41 in CHashWriter& CHashWriter::operator<< <std::vector<CTxOut, std::allocator<CTxOut> > >(std::vector<CTxOut, std::allocator<CTxOut> > const&) /home/travis/build/bitcoin/bitcoin/build/bitcoin-x86_64-unknown-linux-gnu/src/./hash.h:154
     39
     40    [#6](/bitcoin-bitcoin/6/) 0x560f310f09ee in void SerializeTransaction<CHashWriter, CTransaction>(CTransaction const&, CHashWriter&) /home/travis/build/bitcoin/bitcoin/build/bitcoin-x86_64-unknown-linux-gnu/src/./primitives/transaction.h:263:7
     41
     42    [#7](/bitcoin-bitcoin/7/) 0x560f310f06b6 in void CTransaction::Serialize<CHashWriter>(CHashWriter&) const /home/travis/build/bitcoin/bitcoin/build/bitcoin-x86_64-unknown-linux-gnu/src/./primitives/transaction.h:316:9
     43
     44    [#8](/bitcoin-bitcoin/8/) 0x560f310f06b6 in void Serialize<CHashWriter, CTransaction>(CHashWriter&, CTransaction const&) /home/travis/build/bitcoin/bitcoin/build/bitcoin-x86_64-unknown-linux-gnu/src/./serialize.h:603
     45
     46    [#9](/bitcoin-bitcoin/9/) 0x560f310f06b6 in CHashWriter& CHashWriter::operator<< <CTransaction>(CTransaction const&) /home/travis/build/bitcoin/bitcoin/build/bitcoin-x86_64-unknown-linux-gnu/src/./hash.h:154
     47
     48    [#10](/bitcoin-bitcoin/10/) 0x560f310edbb6 in uint256 SerializeHash<CTransaction>(CTransaction const&, int, int) /home/travis/build/bitcoin/bitcoin/build/bitcoin-x86_64-unknown-linux-gnu/src/./hash.h:199:8
     49
     50    [#11](/bitcoin-bitcoin/11/) 0x560f310edbb6 in CTransaction::ComputeHash() const /home/travis/build/bitcoin/bitcoin/build/bitcoin-x86_64-unknown-linux-gnu/src/primitives/transaction.cpp:67
     51
     52    [#12](/bitcoin-bitcoin/12/) 0x560f310ee2b6 in CTransaction::CTransaction(CMutableTransaction const&) /home/travis/build/bitcoin/bitcoin/build/bitcoin-x86_64-unknown-linux-gnu/src/primitives/transaction.cpp:80:142
     53
     54    [#13](/bitcoin-bitcoin/13/) 0x560f2f880d86 in void __gnu_cxx::new_allocator<CTransaction>::construct<CTransaction const, CMutableTransaction&>(CTransaction const*, CMutableTransaction&) /usr/bin/../lib/gcc/x86_64-linux-gnu/7.4.0/../../../../include/c++/7.4.0/ext/new_allocator.h:136:23
     55
     56    [#14](/bitcoin-bitcoin/14/) 0x560f2f880cfa in void std::allocator_traits<std::allocator<CTransaction> >::construct<CTransaction const, CMutableTransaction&>(std::allocator<CTransaction>&, CTransaction const*, CMutableTransaction&) /usr/bin/../lib/gcc/x86_64-linux-gnu/7.4.0/../../../../include/c++/7.4.0/bits/alloc_traits.h:475:8
     57
     58    [#15](/bitcoin-bitcoin/15/) 0x560f2f880afd in std::_Sp_counted_ptr_inplace<CTransaction const, std::allocator<CTransaction>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<CMutableTransaction&>(std::allocator<CTransaction>, CMutableTransaction&) /usr/bin/../lib/gcc/x86_64-linux-gnu/7.4.0/../../../../include/c++/7.4.0/bits/shared_ptr_base.h:526:4
     59
     60    [#16](/bitcoin-bitcoin/16/) 0x560f2f880739 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<CTransaction const, std::allocator<CTransaction>, CMutableTransaction&>(std::_Sp_make_shared_tag, CTransaction const*, std::allocator<CTransaction> const&, CMutableTransaction&) /usr/bin/../lib/gcc/x86_64-linux-gnu/7.4.0/../../../../include/c++/7.4.0/bits/shared_ptr_base.h:637:18
     61
     62    [#17](/bitcoin-bitcoin/17/) 0x560f2f8804eb in std::__shared_ptr<CTransaction const, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<CTransaction>, CMutableTransaction&>(std::_Sp_make_shared_tag, std::allocator<CTransaction> const&, CMutableTransaction&) /usr/bin/../lib/gcc/x86_64-linux-gnu/7.4.0/../../../../include/c++/7.4.0/bits/shared_ptr_base.h:1294:14
     63
     64    [#18](/bitcoin-bitcoin/18/) 0x560f2f8803ec in std::shared_ptr<CTransaction const>::shared_ptr<std::allocator<CTransaction>, CMutableTransaction&>(std::_Sp_make_shared_tag, std::allocator<CTransaction> const&, CMutableTransaction&) /usr/bin/../lib/gcc/x86_64-linux-gnu/7.4.0/../../../../include/c++/7.4.0/bits/shared_ptr.h:344:4
     65
     66    [#19](/bitcoin-bitcoin/19/) 0x560f2f880262 in std::shared_ptr<CTransaction const> std::allocate_shared<CTransaction const, std::allocator<CTransaction>, CMutableTransaction&>(std::allocator<CTransaction> const&, CMutableTransaction&) /usr/bin/../lib/gcc/x86_64-linux-gnu/7.4.0/../../../../include/c++/7.4.0/bits/shared_ptr.h:690:14
     67
     68    [#20](/bitcoin-bitcoin/20/) 0x560f2f880262 in std::shared_ptr<CTransaction const> std::make_shared<CTransaction const, CMutableTransaction&>(CMutableTransaction&) /usr/bin/../lib/gcc/x86_64-linux-gnu/7.4.0/../../../../include/c++/7.4.0/bits/shared_ptr.h:706
     69
     70    [#21](/bitcoin-bitcoin/21/) 0x560f2f8a1083 in std::shared_ptr<CTransaction const> MakeTransactionRef<CMutableTransaction&>(CMutableTransaction&) /home/travis/build/bitcoin/bitcoin/build/bitcoin-x86_64-unknown-linux-gnu/src/./primitives/transaction.h:416:93
     71
     72    [#22](/bitcoin-bitcoin/22/) 0x560f2f8a1083 in blockfilter_tests::blockfilter_basic_test::test_method() /home/travis/build/bitcoin/bitcoin/build/bitcoin-x86_64-unknown-linux-gnu/src/test/blockfilter_tests.cpp:91
     73
     74    [#23](/bitcoin-bitcoin/23/) 0x560f2f89f862 in blockfilter_tests::blockfilter_basic_test_invoker() /home/travis/build/bitcoin/bitcoin/build/bitcoin-x86_64-unknown-linux-gnu/src/test/blockfilter_tests.cpp:55:1
     75
     76    [#24](/bitcoin-bitcoin/24/) 0x560f2f79cc25 in boost::detail::function::void_function_invoker0<void (*)(), void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:118:11
     77
     78    [#25](/bitcoin-bitcoin/25/) 0x7f4c029032cd in boost::detail::function::function_obj_invoker0<boost::detail::forward, int>::invoke(boost::detail::function::function_buffer&) (/usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.65.1+0x4b2cd)
     79
     80    [#26](/bitcoin-bitcoin/26/) 0x7f4c0290277c in boost::execution_monitor::catch_signals(boost::function<int ()> const&) (/usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.65.1+0x4a77c)
     81
     82    [#27](/bitcoin-bitcoin/27/) 0x7f4c02902860 in boost::execution_monitor::execute(boost::function<int ()> const&) (/usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.65.1+0x4a860)
     83
     84    [#28](/bitcoin-bitcoin/28/) 0x7f4c02902fdc in boost::execution_monitor::vexecute(boost::function<void ()> const&) (/usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.65.1+0x4afdc)
     85
     86    [#29](/bitcoin-bitcoin/29/) 0x7f4c029318d0 in boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned int) (/usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.65.1+0x798d0)
     87
     88    [#30](/bitcoin-bitcoin/30/) 0x7f4c0290dc6a in boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned int, boost::unit_test::framework::state::random_generator_helper const*) (/usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.65.1+0x55c6a)
     89
     90    [#31](/bitcoin-bitcoin/31/) 0x7f4c0290e12d in boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned int, boost::unit_test::framework::state::random_generator_helper const*) (/usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.65.1+0x5612d)
     91
     92    [#32](/bitcoin-bitcoin/32/) 0x7f4c0290e12d in boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned int, boost::unit_test::framework::state::random_generator_helper const*) (/usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.65.1+0x5612d)
     93
     94    [#33](/bitcoin-bitcoin/33/) 0x7f4c02906cc7 in boost::unit_test::framework::run(unsigned long, bool) (/usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.65.1+0x4ecc7)
     95
     96    [#34](/bitcoin-bitcoin/34/) 0x7f4c0292f13e in boost::unit_test::unit_test_main(bool (*)(), int, char**) (/usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.65.1+0x7713e)
     97
     98    [#35](/bitcoin-bitcoin/35/) 0x560f2f67d56b in main /usr/include/boost/test/unit_test.hpp:63:12
     99
    100    [#36](/bitcoin-bitcoin/36/) 0x7f4c00691b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
    101
    102    [#37](/bitcoin-bitcoin/37/) 0x560f2f580979 in _start (/home/travis/build/bitcoin/bitcoin/build/bitcoin-x86_64-unknown-linux-gnu/src/test/test_bitcoin+0x28f2979)
    
  7. martinus closed this on Sep 27, 2019

  8. DrahtBot locked this on Dec 16, 2021

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-07-03 10:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me