This PR is a slightly modified version of #15836 (jonasschnelli):
This follows the approach of adding statistical information to Bitcoin Core that would otherwise be inefficient to calculate outside of the codebase.
Adds an optional feerate histogram to
getmempoolinfo.The concept and code is heavily inspired by the stats jhoenicke runs (https://github.com/jhoenicke/mempool, http://bitcoin-mempool.info).
If someone has a good idea how to make the feerate-groups dynamic but also semi-constant for similar fee environments, please comment.
If this is feature we’d like to have in master (concept ACKs), I’d continue this with writing tests.
A simple plot of the data is here. RPC output sample is here.
My attempts to contact jonasschnelli were, unfortunately, unsuccessful so I decided to create this PR in an attempt to move this forward. If this is somehow problematic, please let me know to work it out. edit: Jonas is happy the work continues.
This PR
Note that REST support which is in #15836 is not included in this PR. It can be improved in this PR or in a follow-up one if it is deemed useful/required/etc.
Applied review comments from the old PR
- #15836 (review) - “Remove these files and maybe update .gitignore?”
- #15836 (review) - “This must be before the above line (order is important) otherwise
rest_mempool_info_with_fee_histogramis never called.” - #15836 (comment) - “This gives me a new warning on build: […]” This is already addressed, I believe.
- #15836 (review) - Simplify
if (feeperbyte >= feelimits[i] && (i == feelimits.size() - 1 || feeperbyte < feelimits[i + 1])) { - https://github.com/bitcoin/bitcoin/commit/0b6ba66238c377116bc6c21e19cffbf1b6dfc788 - Added test proposed by promag
- #15836 (review) -
std::to_string(...)replaced with `ToString(…) as done in #18134
Test commands
0$ ./bitcoin-cli -testnet getmempoolinfo "[0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 17, 20, 25, 30, 40, 50, 60, 70, 80, 100, 120, 140, 170, 200]" # To test the new behavior
0$ test/functional/test_runner.py mempool_fee_histogram.py # To run the new test
0$ ./bitcoin-cli -testnet -rpcuser=test -rpcpassword=test help getmempoolinfo
1getmempoolinfo ( [fee_rate,...] )
2
3Returns details on the active state of the TX memory pool.
4
5Arguments:
61. fee_histogram (json array, optional) Fee statistics grouped by fee rate ranges
7 [
8 fee_rate, (numeric, required) Fee rate (in sat/vB) to group the fees by
9 ...
10 ]
11
12Result:
13{ (json object)
14 "loaded" : true|false, (boolean) True if the mempool is fully loaded
15 "size" : n, (numeric) Current tx count
16 "bytes" : n, (numeric) Sum of all virtual transaction sizes as defined in BIP 141. Differs from actual serialized size because witness data is discounted
17 "usage" : n, (numeric) Total memory usage for the mempool
18 "total_fee" : n, (numeric) Total fees for the mempool in BTC, ignoring modified fees through prioritizetransaction
19 "maxmempool" : n, (numeric) Maximum memory usage for the mempool
20 "mempoolminfee" : n, (numeric) Minimum fee rate in BTC/kvB for tx to be accepted. Is the maximum of minrelaytxfee and minimum mempool fee
21 "minrelaytxfee" : n, (numeric) Current minimum relay fee for transactions
22 "unbroadcastcount" : n, (numeric) Current number of transactions that haven't passed initial broadcast yet
23 "fee_histogram" : { (json object)
24 "fee_rate_groups" : { (json object)
25 "<fee_rate_group>" : { (json object) Fee rate group named by its lower bound (in sat/vB), identical to the "from" field below
26 "size" : n, (numeric) Cumulative size of all transactions in the fee rate group (in vBytes)
27 "count" : n, (numeric) Number of transactions in the fee rate group
28 "fees" : n, (numeric) Cumulative fees of all transactions in the fee rate group (in sat)
29 "from" : n (numeric) Group contains transactions with fee rates equal or greater than this value (in sat/vB)
30 },
31 ...
32 },
33 "total_fees" : n (numeric) Total available fees in mempool (in sat)
34 }
35}
36
37Examples:
38> bitcoin-cli getmempoolinfo
39> bitcoin-cli getmempoolinfo "[0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 17, 20, 25, 30, 40, 50, 60, 70, 80, 100, 120, 140, 170, 200]"
40> curl --user myusername --data-binary '{"jsonrpc": "1.0", "id": "curltest", "method": "getmempoolinfo", "params": []}' -H 'content-type: text/plain;' http://127.0.0.1:8332/
41> curl --user myusername --data-binary '{"jsonrpc": "1.0", "id": "curltest", "method": "getmempoolinfo", "params": [[0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 17, 20, 25, 30, 40, 50, 60, 70, 80, 100, 120, 140, 170, 200]]}' -H 'content-type: text/plain;' http://127.0.0.1:8332/
Output on testnet (2022-07-09)
0./bitcoin-cli -testnet -rpcuser=test -rpcpassword=test getmempoolinfo "[0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 17, 20, 25, 30, 40, 50, 60, 70, 80, 100, 120, 140, 170, 200]"
0{
1"loaded": true,
2"size": 10,
3"bytes": 2652,
4"usage": 13504,
5"total_fee": 0.00363010,
6"maxmempool": 300000000,
7"mempoolminfee": 0.00001000,
8"minrelaytxfee": 0.00001000,
9"unbroadcastcount": 0,
10"fee_histogram": {
11 "fee_rate_groups": {
12 "0": {
13 "size": 0,
14 "count": 0,
15 "fees": 0,
16 "from": 0
17 },
18 "1": {
19 "size": 403,
20 "count": 2,
21 "fees": 403,
22 "from": 1
23 },
24 "2": {
25 "size": 554,
26 "count": 2,
27 "fees": 561,
28 "from": 2
29 },
30 "3": {
31 "size": 0,
32 "count": 0,
33 "fees": 0,
34 "from": 3
35 },
36 "4": {
37 "size": 0,
38 "count": 0,
39 "fees": 0,
40 "from": 4
41 },
42 "5": {
43 "size": 0,
44 "count": 0,
45 "fees": 0,
46 "from": 5
47 },
48 "6": {
49 "size": 255,
50 "count": 1,
51 "fees": 1345,
52 "from": 6
53 },
54 "7": {
55 "size": 0,
56 "count": 0,
57 "fees": 0,
58 "from": 7
59 },
60 "8": {
61 "size": 0,
62 "count": 0,
63 "fees": 0,
64 "from": 8
65 },
66 "10": {
67 "size": 0,
68 "count": 0,
69 "fees": 0,
70 "from": 10
71 },
72 "12": {
73 "size": 0,
74 "count": 0,
75 "fees": 0,
76 "from": 12
77 },
78 "14": {
79 "size": 0,
80 "count": 0,
81 "fees": 0,
82 "from": 14
83 },
84 "17": {
85 "size": 0,
86 "count": 0,
87 "fees": 0,
88 "from": 17
89 },
90 "20": {
91 "size": 0,
92 "count": 0,
93 "fees": 0,
94 "from": 20
95 },
96 "25": {
97 "size": 352,
98 "count": 1,
99 "fees": 9505,
100 "from": 25
101 },
102 "30": {
103 "size": 144,
104 "count": 1,
105 "fees": 4520,
106 "from": 30
107 },
108 "40": {
109 "size": 0,
110 "count": 0,
111 "fees": 0,
112 "from": 40
113 },
114 "50": {
115 "size": 374,
116 "count": 1,
117 "fees": 20000,
118 "from": 50
119 },
120 "60": {
121 "size": 351,
122 "count": 1,
123 "fees": 23937,
124 "from": 60
125 },
126 "70": {
127 "size": 0,
128 "count": 0,
129 "fees": 0,
130 "from": 70
131 },
132 "80": {
133 "size": 0,
134 "count": 0,
135 "fees": 0,
136 "from": 80
137 },
138 "100": {
139 "size": 0,
140 "count": 0,
141 "fees": 0,
142 "from": 100
143 },
144 "120": {
145 "size": 0,
146 "count": 0,
147 "fees": 0,
148 "from": 120
149 },
150 "140": {
151 "size": 0,
152 "count": 0,
153 "fees": 0,
154 "from": 140
155 },
156 "170": {
157 "size": 0,
158 "count": 0,
159 "fees": 0,
160 "from": 170
161 },
162 "200": {
163 "size": 219,
164 "count": 1,
165 "fees": 302739,
166 "from": 200
167 }
168 },
169 "total_fees": 363010
170}
171}
Various inputs
0./bitcoin-cli -testnet -rpcuser=test -rpcpassword=test getmempoolinfo "[0]" # OK, one fee rate group covering all possible transaction fee rates (i.e. `>=0`); total_fee = fee_histogram.total_fees
1./bitcoin-cli -testnet -rpcuser=test -rpcpassword=test getmempoolinfo "[-1]" # reports: Non-negative values are expected
2./bitcoin-cli -testnet -rpcuser=test -rpcpassword=test getmempoolinfo "[0,0]" # reports: Strictly increasing values are expected
3./bitcoin-cli -testnet -rpcuser=test -rpcpassword=test getmempoolinfo "[0,1,0]" # reports: Strictly increasing values are expected
4./bitcoin-cli -testnet -rpcuser=test -rpcpassword=test getmempoolinfo "[0, 1000000000000000000000000000000000]" # reports: JSON integer out of range
5./bitcoin-cli -testnet -rpcuser=test -rpcpassword=test getmempoolinfo '[1.2,2,3]' # reports: JSON integer out of range
6./bitcoin-cli -testnet -rpcuser=test -rpcpassword=test getmempoolinfo '["2"]' # reports: Error parsing JSON: ['2']
