rpc: provide per message stats for global traffic via new RPC 'getnetmsgstats'

vasild commented at 9:32 am on February 11, 2024: contributor

Introduce a new RPC, getnetmsgstats to retrieve traffic bytes and count of messages possibly broken down by:

direction (sent or received)
network (ipv4, tor, etc)
connection type (outbound-full-relay, block-relay-only, etc)
message type (verack, ping, etc)

Implements: #26337 Add per message stats to getnettotals rpc

Examples:

0{
1  "bytes": 5392756,
2  "count": 32683
3}

 0{
 1  "sendcmpct": {
 2    "bytes": 975,
 3    "count": 31
 4  },
 5  "sendaddrv2": {
 6    "bytes": 816,
 7    "count": 28
 8  },
 9  "headers": {
10    "bytes": 1786,
11    "count": 28
12  },
13  "feefilter": {
14    "bytes": 816,
15    "count": 27
16  },
17  "version": {
18    "bytes": 4843,
19    "count": 37
20  },
21  "ping": {
22    "bytes": 1104,
23    "count": 36
24  },
25  "getaddr": {
26    "bytes": 384,
27    "count": 13
28  },
29  "block": {
30    "bytes": 5128467,
31    "count": 20365
32  },
33  "getheaders": {
34    "bytes": 29436,
35    "count": 28
36  },
37  "pong": {
38    "bytes": 1040,
39    "count": 34
40  },
41  "getdata": {
42    "bytes": 1218925,
43    "count": 20008
44  },
45  "sendheaders": {
46    "bytes": 447,
47    "count": 16
48  },
49  "wtxidrelay": {
50    "bytes": 816,
51    "count": 28
52  },
53  "addrv2": {
54    "bytes": 301161,
55    "count": 25
56  },
57  "verack": {
58    "bytes": 816,
59    "count": 28
60  }
61}

 0{
 1  "i2p": {
 2    "bytes": 1311861,
 3    "count": 8226
 4  },
 5  "onion": {
 6    "bytes": 3932760,
 7    "count": 24272
 8  },
 9  "ipv6": {
10    "bytes": 631265,
11    "count": 3362
12  },
13  "ipv4": {
14    "bytes": 1300106,
15    "count": 7773
16  }
17}

  0{
  1  "recv": {
  2    "i2p": {
  3      "manual": {
  4        "sendcmpct": {
  5          "bytes": 90,
  6          "count": 3
  7        },
  8        "sendaddrv2": {
  9          "bytes": 99,
 10          "count": 3
 11        },
 12        "headers": {
 13          "bytes": 309,
 14          "count": 3
 15        },
 16        "feefilter": {
 17          "bytes": 87,
 18          "count": 3
 19        },
 20        "version": {
 21          "bytes": 408,
 22          "count": 3
 23        },
 24        "ping": {
 25          "bytes": 87,
 26          "count": 3
 27        },
 28        "block": {
 29          "bytes": 432950,
 30          "count": 1794
 31        },
 32        "getheaders": {
 33          "bytes": 3150,
 34          "count": 3
 35        },
 36        "pong": {
 37          "bytes": 87,
 38          "count": 3
 39        },
 40        "wtxidrelay": {
 41          "bytes": 99,
 42          "count": 3
 43        },
 44        "addrv2": {
 45          "bytes": 68508,
 46          "count": 3
 47        },
 48        "verack": {
 49          "bytes": 99,
 50          "count": 3
 51        }
 52      }
 53    },
 54    "onion": {
 55      "block-relay-only": {
 56        "sendcmpct": {
 57          "bytes": 66,
 58          "count": 2
 59        },
 60        "sendaddrv2": {
 61          "bytes": 24,
 62          "count": 1
 63        },
 64        "headers": {
 65          "bytes": 106,
 66          "count": 1
 67        },
 68        "feefilter": {
 69          "bytes": 32,
 70          "count": 1
 71        },
 72        "version": {
 73          "bytes": 126,
 74          "count": 1
 75        },
 76        "ping": {
 77          "bytes": 64,
 78          "count": 2
 79        },
 80        "block": {
 81          "bytes": 478479,
 82          "count": 1896
 83        },
 84        "getheaders": {
 85          "bytes": 1053,
 86          "count": 1
 87        },
 88        "pong": {
 89          "bytes": 64,
 90          "count": 2
 91        },
 92        "sendheaders": {
 93          "bytes": 24,
 94          "count": 1
 95        },
 96        "wtxidrelay": {
 97          "bytes": 24,
 98          "count": 1
 99        },
100        "verack": {
101          "bytes": 24,
102          "count": 1
103        }
104      },
105      "outbound-full-relay": {
106        "sendcmpct": {
107          "bytes": 195,
108          "count": 6
109        },
110        "sendaddrv2": {
111          "bytes": 129,
112          "count": 5
113        },
114        "headers": {
115          "bytes": 527,
116          "count": 5
117        },
118        "feefilter": {
119          "bytes": 157,
120          "count": 5
121        },
122        "version": {
123          "bytes": 639,
124          "count": 5
125        },
126        "ping": {
127          "bytes": 189,
128          "count": 6
129        },
130        "block": {
131          "bytes": 1465688,
132          "count": 5957
133        },
134        "getheaders": {
135          "bytes": 5262,
136          "count": 5
137        },
138        "pong": {
139          "bytes": 189,
140          "count": 6
141        },
142        "sendheaders": {
143          "bytes": 24,
144          "count": 1
145        },
146        "wtxidrelay": {
147          "bytes": 129,
148          "count": 5
149        },
150        "addrv2": {
151          "bytes": 120693,
152          "count": 9
153        },
154        "verack": {
155          "bytes": 129,
156          "count": 5
157        }
158      }
159    },
160    "ipv6": {
161      "manual": {
162        "sendcmpct": {
163          "bytes": 120,
164          "count": 4
165        },
166        "sendaddrv2": {
167          "bytes": 132,
168          "count": 4
169        },
170        "headers": {
171          "bytes": 412,
172          "count": 4
173        },
174        "feefilter": {
175          "bytes": 116,
176          "count": 4
177        },
178        "version": {
179          "bytes": 544,
180          "count": 4
181        },
182        "ping": {
183          "bytes": 116,
184          "count": 4
185        },
186        "block": {
187          "bytes": 138074,
188          "count": 583
189        },
190        "getheaders": {
191          "bytes": 4200,
192          "count": 4
193        },
194        "pong": {
195          "bytes": 116,
196          "count": 4
197        },
198        "wtxidrelay": {
199          "bytes": 132,
200          "count": 4
201        },
202        "addrv2": {
203          "bytes": 93436,
204          "count": 4
205        },
206        "verack": {
207          "bytes": 132,
208          "count": 4
209        }
210      },
211      "outbound-full-relay": {
212        "version": {
213          "bytes": 126,
214          "count": 1
215        }
216      }
217    },
218    "ipv4": {
219      "outbound-full-relay": {
220        "sendcmpct": {
221          "bytes": 66,
222          "count": 2
223        },
224        "sendaddrv2": {
225          "bytes": 24,
226          "count": 1
227        },
228        "headers": {
229          "bytes": 106,
230          "count": 1
231        },
232        "feefilter": {
233          "bytes": 32,
234          "count": 1
235        },
236        "version": {
237          "bytes": 252,
238          "count": 2
239        },
240        "ping": {
241          "bytes": 32,
242          "count": 1
243        },
244        "block": {
245          "bytes": 313988,
246          "count": 1255
247        },
248        "getheaders": {
249          "bytes": 1053,
250          "count": 1
251        },
252        "pong": {
253          "bytes": 32,
254          "count": 1
255        },
256        "sendheaders": {
257          "bytes": 24,
258          "count": 1
259        },
260        "wtxidrelay": {
261          "bytes": 24,
262          "count": 1
263        },
264        "addrv2": {
265          "bytes": 40,
266          "count": 1
267        },
268        "verack": {
269          "bytes": 24,
270          "count": 1
271        }
272      }
273    }
274  },
275  "sent": {
276    "i2p": {
277      "manual": {
278        "sendcmpct": {
279          "bytes": 90,
280          "count": 3
281        },
282        "sendaddrv2": {
283          "bytes": 99,
284          "count": 3
285        },
286        "headers": {
287          "bytes": 66,
288          "count": 3
289        },
290        "feefilter": {
291          "bytes": 87,
292          "count": 3
293        },
294        "version": {
295          "bytes": 544,
296          "count": 4
297        },
298        "ping": {
299          "bytes": 87,
300          "count": 3
301        },
302        "getaddr": {
303          "bytes": 99,
304          "count": 3
305        },
306        "getheaders": {
307          "bytes": 3150,
308          "count": 3
309        },
310        "pong": {
311          "bytes": 87,
312          "count": 3
313        },
314        "getdata": {
315          "bytes": 105382,
316          "count": 1789
317        },
318        "sendheaders": {
319          "bytes": 99,
320          "count": 3
321        },
322        "wtxidrelay": {
323          "bytes": 99,
324          "count": 3
325        },
326        "verack": {
327          "bytes": 99,
328          "count": 3
329        }
330      }
331    },
332    "onion": {
333      "block-relay-only": {
334        "sendcmpct": {
335          "bytes": 33,
336          "count": 1
337        },
338        "sendaddrv2": {
339          "bytes": 24,
340          "count": 1
341        },
342        "headers": {
343          "bytes": 25,
344          "count": 1
345        },
346        "version": {
347          "bytes": 127,
348          "count": 1
349        },
350        "ping": {
351          "bytes": 64,
352          "count": 2
353        },
354        "getheaders": {
355          "bytes": 1053,
356          "count": 1
357        },
358        "pong": {
359          "bytes": 64,
360          "count": 2
361        },
362        "getdata": {
363          "bytes": 115932,
364          "count": 1884
365        },
366        "sendheaders": {
367          "bytes": 24,
368          "count": 1
369        },
370        "wtxidrelay": {
371          "bytes": 24,
372          "count": 1
373        },
374        "verack": {
375          "bytes": 24,
376          "count": 1
377        }
378      },
379      "outbound-full-relay": {
380        "sendcmpct": {
381          "bytes": 162,
382          "count": 5
383        },
384        "sendaddrv2": {
385          "bytes": 129,
386          "count": 5
387        },
388        "headers": {
389          "bytes": 122,
390          "count": 5
391        },
392        "feefilter": {
393          "bytes": 157,
394          "count": 5
395        },
396        "version": {
397          "bytes": 771,
398          "count": 6
399        },
400        "ping": {
401          "bytes": 221,
402          "count": 7
403        },
404        "getaddr": {
405          "bytes": 129,
406          "count": 5
407        },
408        "getheaders": {
409          "bytes": 5262,
410          "count": 5
411        },
412        "pong": {
413          "bytes": 189,
414          "count": 6
415        },
416        "getdata": {
417          "bytes": 363478,
418          "count": 5932
419        },
420        "sendheaders": {
421          "bytes": 129,
422          "count": 5
423        },
424        "wtxidrelay": {
425          "bytes": 129,
426          "count": 5
427        },
428        "addrv2": {
429          "bytes": 231,
430          "count": 4
431        },
432        "verack": {
433          "bytes": 129,
434          "count": 5
435        }
436      }
437    },
438    "ipv6": {
439      "manual": {
440        "sendcmpct": {
441          "bytes": 120,
442          "count": 4
443        },
444        "sendaddrv2": {
445          "bytes": 132,
446          "count": 4
447        },
448        "headers": {
449          "bytes": 88,
450          "count": 4
451        },
452        "feefilter": {
453          "bytes": 116,
454          "count": 4
455        },
456        "version": {
457          "bytes": 544,
458          "count": 4
459        },
460        "ping": {
461          "bytes": 116,
462          "count": 4
463        },
464        "getaddr": {
465          "bytes": 132,
466          "count": 4
467        },
468        "getheaders": {
469          "bytes": 4200,
470          "count": 4
471        },
472        "pong": {
473          "bytes": 116,
474          "count": 4
475        },
476        "getdata": {
477          "bytes": 35492,
478          "count": 584
479        },
480        "sendheaders": {
481          "bytes": 99,
482          "count": 3
483        },
484        "wtxidrelay": {
485          "bytes": 132,
486          "count": 4
487        },
488        "verack": {
489          "bytes": 132,
490          "count": 4
491        }
492      },
493      "outbound-full-relay": {
494        "version": {
495          "bytes": 254,
496          "count": 2
497        }
498      }
499    },
500    "ipv4": {
501      "outbound-full-relay": {
502        "sendcmpct": {
503          "bytes": 33,
504          "count": 1
505        },
506        "sendaddrv2": {
507          "bytes": 24,
508          "count": 1
509        },
510        "headers": {
511          "bytes": 25,
512          "count": 1
513        },
514        "feefilter": {
515          "bytes": 32,
516          "count": 1
517        },
518        "version": {
519          "bytes": 508,
520          "count": 4
521        },
522        "ping": {
523          "bytes": 32,
524          "count": 1
525        },
526        "getaddr": {
527          "bytes": 24,
528          "count": 1
529        },
530        "getheaders": {
531          "bytes": 1053,
532          "count": 1
533        },
534        "pong": {
535          "bytes": 32,
536          "count": 1
537        },
538        "getdata": {
539          "bytes": 76531,
540          "count": 1231
541        },
542        "sendheaders": {
543          "bytes": 24,
544          "count": 1
545        },
546        "wtxidrelay": {
547          "bytes": 24,
548          "count": 1
549        },
550        "verack": {
551          "bytes": 24,
552          "count": 1
553        }
554      }
555    }
556  }
557}

Previous incarnations of this: #27534 rpc: add ‘getnetmsgstats’, new rpc to view network message statistics (Thank you, @satsie!) #28926 rpc: add ‘getnetmsgstats’ RPC (Thank you, @willcl-ark!)

DrahtBot commented at 9:32 am on February 11, 2024: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type	Reviewers
Concept ACK	epiccurious

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#30343 (wallet, logging: Replace WalletLogPrintf() with LogInfo() by ryanofsky)
#29641 (scripted-diff: Use LogInfo/LogDebug over LogPrintf/LogPrint by maflcko)
#28521 (net: additional disconnect logging by Sjors)
#19463 (Prune locks by luke-jr)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

DrahtBot added the label RPC/REST/ZMQ on Feb 11, 2024

DrahtBot added the label CI failed on Feb 11, 2024

DrahtBot commented at 10:51 am on February 11, 2024: contributor

🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the documentation.

Possibly this is due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

Leave a comment here, if you need help tracking down a confusing failure.

Debug: https://github.com/bitcoin/bitcoin/runs/21448984768

vasild force-pushed on Feb 11, 2024

DrahtBot removed the label CI failed on Feb 11, 2024

epiccurious commented at 10:26 pm on February 11, 2024: contributor

Concept ACK 91c90d759503aa3fa15b330bba2148698cdc271d.

DrahtBot added the label CI failed on Mar 12, 2024

DrahtBot commented at 2:24 pm on April 4, 2024: contributor

Needs rebase if still relevant

achow101 requested review from mzumsande on Apr 9, 2024

0xB10C commented at 4:20 pm on April 9, 2024: contributor

just fyi as this came up: You could count up message stats with the inbound and outbound message tracepoints. However, this does only gives you stats from the points on where you start hooking into the tracepoints, might not be as user friendly (RPC is easier to use than the tracepoints), and the tracepoints currently only work on Linux.

While I personally don’t need it, I see why some node operators need traffic statistics. E.g. How much of my (limited monthly) traffic is used up by inbound connections?

DrahtBot marked this as a draft on Apr 23, 2024

DrahtBot commented at 6:28 am on April 23, 2024: contributor

Converted to draft due to failing CI and inactivity

vasild force-pushed on May 9, 2024

vasild commented at 1:40 pm on May 9, 2024: contributor

91c90d7595...6bfda84b92: rebase, pick changes from #29421 and adjust the newly added test.

DrahtBot removed the label CI failed on May 9, 2024

achow101 referenced this in commit 6c13b1375f on May 21, 2024

DrahtBot added the label Needs rebase on May 21, 2024

net: count traffic bytes and number of messages globally

Before this change only per-peer stats were gathered. They
are lost when the peer disconnects.

So, collect the traffic stats globally in `CConnman`, broken down by
* direction (sent or received, (2))
* network of the peer (IPv4, IPv6, Tor, I2P, CJDNS (5))
* connection type (inbound, full outbound, feeler, etc, (6))
* message type (verack, ping, etc, (36))

175e2bba33

rpc: visualize global CConnman stats in a new RPC getnetmsgstats

Introduce a new RPC `getnetmsgstats` that renders the global traffic
stats from `CConnman`.

Co-authored-by: Vasil Dimov <vd@FreeBSD.org>

0a89507129

rpc: make it possible to aggregate the result in getnetmsgstats

Aggregation can be by either one of direction, network, connection type
or message type. For example if the following

```
{
    "ipv4": { "ping": 3 },
    "ipv6": { "ping": 4 }
}
```

is aggregated by network, then the result will be

```
{
    "ping": 7
}
```

136e1fe3e4

vasild force-pushed on May 23, 2024

vasild commented at 9:51 am on May 23, 2024: contributor

6bfda84b92...b5189f543c: rebase due to conflicts (part of this PR was merged via #29421)

vasild marked this as ready for review on May 23, 2024

DrahtBot removed the label Needs rebase on May 23, 2024

test: add functional tests for the new getnetmsgstats RPC

Co-authored-by: Vasil Dimov <vd@FreeBSD.org>

68dfd4270c

vasild force-pushed on May 23, 2024

DrahtBot added the label CI failed on May 23, 2024

DrahtBot removed the label CI failed on May 23, 2024

rpc: provide per message stats for global traffic via new RPC ‘getnetmsgstats’ #29418

Code Coverage

Reviews

Conflicts