Add bloom filter usage statistics #7685

pull jonasschnelli wants to merge 3 commits into bitcoin:master from jonasschnelli:2016/03/bf_stats changing 4 files +225 −1

jonasschnelli commented at 4:52 PM on March 14, 2016: contributor
It can be useful to know, how many blocks and how many times the mempool was filtered.

Bloom filtering is CPU intense and thus informations about its usage can help to identify sources of resource consumption. Collecting statistics would be required to dynamic limit the amount of accepted bloom filtering.

This PR does only collect global statistics (total value and value per timeframe of currently 1h), though, I kept the design flexible. Extending this to per-node bloom-filter statistics is possible.

Also the time consumption in milliseconds for all filter processes will be collected, although, this value depends on your system workload and might not be very representative.

Statistics can be printed with the new RPC command getfilterstats. Example:
```
/src/bitcoin-cli getfilterstats
{
  "current_timeframe": {
    "time_in_timeframe": 304,
    "blocks": {
      "filteredblocks_count": 1,
      "filteredblocks_amountofbytes": 989168,
      "filteredblocks_timeconsumption": 0
    },
    "mempool": {
      "mempool_filter_count": 4,
      "mempool_filter_amountoftxes": 1788,
      "mempool_filter_timeconsumption": 113
    }
  },
  "total": {
    "blocks": {
      "filteredblocks_count": 1,
      "filteredblocks_amountofbytes": 989168,
      "filteredblocks_timeconsumption": 0
    },
    "mempool": {
      "mempool_filter_count": 4,
      "mempool_filter_amountoftxes": 1788,
      "mempool_filter_timeconsumption": 1788
    }
  }
}
```
jonasschnelli added the label P2P on Mar 14, 2016
jonasschnelli force-pushed on Mar 14, 2016
laanwj commented at 2:08 PM on March 15, 2016: member

Yes, would be interesting to keep track of some statistics regarding resource usage. Concept ACK
paveljanik commented at 8:03 PM on March 15, 2016: contributor
Concept ACK.

Is this pure coincidence?
```
      "mempool_filter_amountoftxes": 1788,
      "mempool_filter_timeconsumption": 1788
```
in src/net.h:None in 6dfbf4daa7 outdated
```
   9 | @@ -10,6 +10,7 @@
  10 |  #include "compat.h"
  11 |  #include "limitedmap.h"
  12 |  #include "netbase.h"
  13 | +#include "primitives/block.h"
```
paveljanik commented at 8:12 PM on March 15, 2016:

Please remove primitives/block.h from the network layer. Compute the size of the block in main.cpp and pass the size instead of the block itself into FilterStatsProcessBlock().

jonasschnelli commented at 4:39 PM on March 16, 2016:

I have also thought about this during the time of writing the code. I think its more flexible the way it is now. FilterStatsProcessBlock could also count transactions or the total amount of vins/vouts per block (which is basically where the filter needs to "go through").

Including block.h should be not a problem.
jonasschnelli renamed this:
~~[WIP] Add bloom filter usage statistics~~
Add bloom filter usage statistics
on Mar 16, 2016

jonasschnelli commented at 4:43 PM on March 16, 2016: contributor

@paveljanik: there was a bug in a previous pushed commit that mixed-up timeconsumption with amountoftxes.

I have tested this PR now on a mainnet node and it seems to work good. Example output:

{
  "current_timeframe": {
    "time_in_timeframe": 1262,
    "timeframe": 3600,
    "blocks": {
      "filteredblocks_count": 2,
      "filteredblocks_amountofbytes": 1923228,
      "filteredblocks_timeconsumption": 24
    },
    "mempool": {
      "mempool_filter_count": 2,
      "mempool_filter_amountoftxes": 55093,
      "mempool_filter_timeconsumption": 4475
    }
  },
  "total": {
    "blocks": {
      "filteredblocks_count": 24443,
      "filteredblocks_amountofbytes": 16788519596,
      "filteredblocks_timeconsumption": 99152
    },
    "mempool": {
      "mempool_filter_count": 5929,
      "mempool_filter_amountoftxes": 60586895,
      "mempool_filter_timeconsumption": 4801408
    }
  }
}

paveljanik commented at 8:20 PM on April 22, 2016: contributor

Rebase needed.
jonasschnelli force-pushed on Apr 25, 2016
jonasschnelli force-pushed on Apr 25, 2016
jonasschnelli commented at 11:23 AM on April 25, 2016: contributor

Rebased. Added information about how many node requested filtering (count of filterload command). Extended default time cycle to 24h.
paveljanik commented at 7:33 PM on April 27, 2016: contributor

Commit message 971e3e4bdf39ff2d13f3aee73e4b8086bc235120: cytle -> cycle Commit message 73e8d32221669f3237a2d3dc71d5930effd73088: ilterload -> filterload

Needs tests and maybe some text to be included in the release notes (low prio IMO, because this is a low level stuff - or -help-debug?).
sipa commented at 3:01 PM on June 2, 2016: member

Needs rebase and nits addressed.
jonasschnelli force-pushed on Jun 2, 2016
fanquake commented at 6:24 AM on June 24, 2016: member

Still needs a rebase.
laanwj added the label Feature on Jun 27, 2016
jonasschnelli force-pushed on Jul 20, 2016
jonasschnelli force-pushed on Jul 20, 2016
jonasschnelli force-pushed on Jul 20, 2016
jonasschnelli force-pushed on Jul 20, 2016
jonasschnelli force-pushed on Jul 20, 2016
jonasschnelli commented at 1:21 PM on July 20, 2016: contributor

Rebased. Needs tests (@MacroFalke?)?
sipa commented at 12:54 PM on August 25, 2016: member

Needs rebase.
[net] add basic stats for filter usage ffc9785f63
Extend filter stats time cycle to 1 day e0a7e66797
Add connect-/filterload-counter to the filterstats 5005595647
jonasschnelli force-pushed on Aug 26, 2016
jonasschnelli commented at 7:40 AM on August 26, 2016: contributor

Rebased. This is something that could be integrated into the general stats collector approach (https://github.com/bitcoin/bitcoin/pull/8501).
jonasschnelli commented at 10:15 AM on December 6, 2016: contributor

Closing because of inactivity.
jonasschnelli closed this on Dec 6, 2016
MarcoFalke locked this on Sep 8, 2021

Contributors

Labels

Linked (view graph)

#8599 Possible Deadlock