rpc: provide per message stats for global traffic via new field in 'getnettotals' #29418

vasild commented at 9:32 AM on February 11, 2024: contributor

Introduce a new field named detailed in RPC getnettotals to retrieve traffic bytes and count of messages broken down by:

direction (sent or received)
network (ipv4, tor, etc)
connection type (outbound-full-relay, block-relay-only, etc)
message type (verack, ping, etc)

Implements: #26337 Add per message stats to getnettotals rpc

Examples:

<details> <summary>Group by direction (sum network, connection type, message type)</summary>

$ bitcoin-cli getnettotals | jq '.detailed | map_values([.[][][]] | { bytes: (map(.bytes) | add), count: (map(.count) | add) })'
{
  "recv": {
    "bytes": 288580402,
    "count": 562224
  },
  "sent": {
    "bytes": 11514028329,
    "count": 715094
  }
}

</details>

<details> <summary>Group by network (sum direction, connection type, message type)</summary>

$ bitcoin-cli getnettotals | jq '
    [.detailed[] | to_entries[] | {key, value: [.value[][]]}]
    | group_by(.key)
    | map({
        key: .[0].key,
        value: ([.[].value[]] | { bytes: (map(.bytes) | add), count: (map(.count) | add) })
      })
    | sort_by(-.value.bytes)
    | from_entries
'

{
  "ipv4": {
    "bytes": 13195416760,
    "count": 689590
  },
  "ipv6": {
    "bytes": 850168762,
    "count": 187587
  },
  "onion": {
    "bytes": 516511904,
    "count": 378875
  },
  "i2p": {
    "bytes": 128190870,
    "count": 217514
  }
}

</details>

<details> <summary>Group by connection type (sum direction, network, message type)</summary>

$ bitcoin-cli getnettotals | jq '
    [.detailed[][] | to_entries[] | {key, value: [.value[]]}]
    | group_by(.key)
    | map({
        key: .[0].key,
        value: ([.[].value[]] | { bytes: (map(.bytes) | add), count: (map(.count) | add) })
      })
    | sort_by(-.value.bytes)
    | from_entries
'

{
  "inbound": {
    "bytes": 14739206052,
    "count": 1093958
  },
  "outbound-full-relay": {
    "bytes": 181674080,
    "count": 403496
  },
  "block-relay-only": {
    "bytes": 2111575,
    "count": 2254
  },
  "feeler": {
    "bytes": 19052,
    "count": 321
  }
}

</details>

<details> <summary>Group by message type (sum direction, network, connection type)</summary>

$ bitcoin-cli getnettotals | jq '
    [.detailed[][][] | to_entries[]]
    | group_by(.key)
    | map({key: .[0].key, value: {bytes: (map(.value.bytes) | add), count: (map(.value.count) | add)}})
    | sort_by(-.value.bytes)
    | from_entries
'

{
  "block": {
    "bytes": 14389470456,
    "count": 16292
  },
  "inv": {
    "bytes": 468520895,
    "count": 493445
  },
  "tx": {
    "bytes": 241896347,
    "count": 739362
  },
  "headers": {
    "bytes": 160736053,
    "count": 5113
  },
  "blocktxn": {
    "bytes": 65711296,
    "count": 149
  },
  "addrv2": {
    "bytes": 39362409,
    "count": 32218
  },
  "getdata": {
    "bytes": 31000172,
    "count": 140298
  },
  "cfilter": {
    "bytes": 13663517,
    "count": 675
  },
  "cmpctblock": {
    "bytes": 11462012,
    "count": 351
  },
  "cfheaders": {
    "bytes": 9458330,
    "count": 213
  },
  "getheaders": {
    "bytes": 2988902,
    "count": 2997
  },
  "addr": {
    "bytes": 1384310,
    "count": 1064
  },
  "version": {
    "bytes": 944836,
    "count": 7199
  },
  "ping": {
    "bytes": 934821,
    "count": 30717
  },
  "pong": {
    "bytes": 897571,
    "count": 29519
  },
  "notfound": {
    "bytes": 652364,
    "count": 4081
  },
  "getblocktxn": {
    "bytes": 272299,
    "count": 149
  },
  "verack": {
    "bytes": 183810,
    "count": 6911
  },
  "sendaddrv2": {
    "bytes": 177381,
    "count": 6573
  },
  "wtxidrelay": {
    "bytes": 133062,
    "count": 4726
  },
  "sendcmpct": {
    "bytes": 119817,
    "count": 3764
  },
  "feefilter": {
    "bytes": 105502,
    "count": 3404
  },
  "getaddr": {
    "bytes": 56313,
    "count": 2130
  },
  "sendheaders": {
    "bytes": 33009,
    "count": 1042
  },
  "getcfheaders": {
    "bytes": 13054,
    "count": 214
  },
  "*other*": {
    "bytes": 12222,
    "count": 86
  },
  "getcfilters": {
    "bytes": 2257,
    "count": 37
  }
}

</details>

<details> <summary>Full output of getnettotals</summary>

$ bitcoin-cli getnettotals
{
  "totalbytesrecv": 290796106,
  "totalbytessent": 11516332695,
  "timemillis": 1792494749925,
  "uploadtarget": {
    "timeframe": 86400,
    "target": 0,
    "target_reached": false,
    "serve_historical_blocks": true,
    "bytes_left_in_cycle": 0,
    "time_left_in_cycle": 0
  },
  "detailed": {
    "recv": {
      "i2p": {
        "feeler": {
          "sendaddrv2": {
            "bytes": 66,
            "count": 2
          },
          "version": {
            "bytes": 540,
            "count": 4
          },
          "wtxidrelay": {
            "bytes": 99,
            "count": 3
          },
          "verack": {
            "bytes": 66,
            "count": 2
          }
        },
        "inbound": {
          "getblocktxn": {
            "bytes": 11760,
            "count": 8
          },
          "sendcmpct": {
            "bytes": 1458,
            "count": 48
          },
          "sendaddrv2": {
            "bytes": 3462,
            "count": 115
          },
          "headers": {
            "bytes": 4200647,
            "count": 188
          },
          "feefilter": {
            "bytes": 1085,
            "count": 37
          },
          "notfound": {
            "bytes": 8512,
            "count": 124
          },
          "version": {
            "bytes": 20365,
            "count": 147
          },
          "ping": {
            "bytes": 28462,
            "count": 971
          },
          "getaddr": {
            "bytes": 1932,
            "count": 61
          },
          "tx": {
            "bytes": 11121147,
            "count": 44073
          },
          "getheaders": {
            "bytes": 51468,
            "count": 49
          },
          "pong": {
            "bytes": 28407,
            "count": 969
          },
          "getdata": {
            "bytes": 1193534,
            "count": 4492
          },
          "inv": {
            "bytes": 27330829,
            "count": 39577
          },
          "sendheaders": {
            "bytes": 1167,
            "count": 37
          },
          "wtxidrelay": {
            "bytes": 3120,
            "count": 97
          },
          "addrv2": {
            "bytes": 171634,
            "count": 1859
          },
          "verack": {
            "bytes": 2901,
            "count": 98
          }
        },
        "block-relay-only": {
          "sendcmpct": {
            "bytes": 60,
            "count": 2
          },
          "sendaddrv2": {
            "bytes": 66,
            "count": 2
          },
          "headers": {
            "bytes": 508855,
            "count": 4
          },
          "feefilter": {
            "bytes": 58,
            "count": 2
          },
          "version": {
            "bytes": 285,
            "count": 2
          },
          "ping": {
            "bytes": 58,
            "count": 2
          },
          "getheaders": {
            "bytes": 10500,
            "count": 10
          },
          "pong": {
            "bytes": 58,
            "count": 2
          },
          "sendheaders": {
            "bytes": 33,
            "count": 1
          },
          "wtxidrelay": {
            "bytes": 66,
            "count": 2
          },
          "verack": {
            "bytes": 66,
            "count": 2
          }
        },
        "outbound-full-relay": {
          "sendcmpct": {
            "bytes": 90,
            "count": 3
          },
          "sendaddrv2": {
            "bytes": 99,
            "count": 3
          },
          "headers": {
            "bytes": 4841,
            "count": 47
          },
          "feefilter": {
            "bytes": 87,
            "count": 3
          },
          "notfound": {
            "bytes": 1606,
            "count": 19
          },
          "version": {
            "bytes": 420,
            "count": 3
          },
          "ping": {
            "bytes": 7975,
            "count": 275
          },
          "tx": {
            "bytes": 2693399,
            "count": 7512
          },
          "getheaders": {
            "bytes": 3150,
            "count": 3
          },
          "pong": {
            "bytes": 7946,
            "count": 274
          },
          "getdata": {
            "bytes": 608696,
            "count": 1154
          },
          "inv": {
            "bytes": 3007482,
            "count": 5055
          },
          "sendheaders": {
            "bytes": 99,
            "count": 3
          },
          "wtxidrelay": {
            "bytes": 99,
            "count": 3
          },
          "addrv2": {
            "bytes": 150314,
            "count": 651
          },
          "verack": {
            "bytes": 99,
            "count": 3
          }
        }
      },
      "onion": {
        "feeler": {
          "sendaddrv2": {
            "bytes": 507,
            "count": 17
          },
          "version": {
            "bytes": 2976,
            "count": 22
          },
          "wtxidrelay": {
            "bytes": 606,
            "count": 20
          },
          "verack": {
            "bytes": 450,
            "count": 15
          }
        },
        "inbound": {
          "getcfheaders": {
            "bytes": 9272,
            "count": 152
          },
          "getcfilters": {
            "bytes": 61,
            "count": 1
          },
          "getblocktxn": {
            "bytes": 10196,
            "count": 10
          },
          "sendcmpct": {
            "bytes": 2553,
            "count": 84
          },
          "sendaddrv2": {
            "bytes": 22272,
            "count": 862
          },
          "headers": {
            "bytes": 7083003,
            "count": 138
          },
          "feefilter": {
            "bytes": 1265,
            "count": 43
          },
          "notfound": {
            "bytes": 2681,
            "count": 35
          },
          "version": {
            "bytes": 114464,
            "count": 869
          },
          "ping": {
            "bytes": 45951,
            "count": 1494
          },
          "getaddr": {
            "bytes": 14742,
            "count": 570
          },
          "tx": {
            "bytes": 2974453,
            "count": 6258
          },
          "getheaders": {
            "bytes": 247192,
            "count": 235
          },
          "pong": {
            "bytes": 56799,
            "count": 1833
          },
          "getdata": {
            "bytes": 3098685,
            "count": 25256
          },
          "inv": {
            "bytes": 12651778,
            "count": 16442
          },
          "sendheaders": {
            "bytes": 1974,
            "count": 62
          },
          "wtxidrelay": {
            "bytes": 6285,
            "count": 197
          },
          "addrv2": {
            "bytes": 118132,
            "count": 1375
          },
          "verack": {
            "bytes": 21492,
            "count": 837
          }
        },
        "block-relay-only": {
          "sendcmpct": {
            "bytes": 150,
            "count": 5
          },
          "sendaddrv2": {
            "bytes": 165,
            "count": 5
          },
          "headers": {
            "bytes": 515,
            "count": 5
          },
          "feefilter": {
            "bytes": 145,
            "count": 5
          },
          "version": {
            "bytes": 690,
            "count": 5
          },
          "ping": {
            "bytes": 145,
            "count": 5
          },
          "getheaders": {
            "bytes": 5250,
            "count": 5
          },
          "pong": {
            "bytes": 145,
            "count": 5
          },
          "sendheaders": {
            "bytes": 165,
            "count": 5
          },
          "wtxidrelay": {
            "bytes": 165,
            "count": 5
          },
          "verack": {
            "bytes": 165,
            "count": 5
          }
        },
        "outbound-full-relay": {
          "getblocktxn": {
            "bytes": 35565,
            "count": 23
          },
          "sendcmpct": {
            "bytes": 621,
            "count": 20
          },
          "sendaddrv2": {
            "bytes": 474,
            "count": 16
          },
          "headers": {
            "bytes": 8618,
            "count": 77
          },
          "feefilter": {
            "bytes": 453,
            "count": 15
          },
          "notfound": {
            "bytes": 1416,
            "count": 24
          },
          "version": {
            "bytes": 2151,
            "count": 16
          },
          "ping": {
            "bytes": 16665,
            "count": 552
          },
          "tx": {
            "bytes": 9800969,
            "count": 30650
          },
          "getheaders": {
            "bytes": 16818,
            "count": 16
          },
          "pong": {
            "bytes": 16633,
            "count": 551
          },
          "getdata": {
            "bytes": 4855391,
            "count": 4636
          },
          "inv": {
            "bytes": 6193503,
            "count": 14658
          },
          "sendheaders": {
            "bytes": 474,
            "count": 16
          },
          "wtxidrelay": {
            "bytes": 474,
            "count": 16
          },
          "addrv2": {
            "bytes": 626368,
            "count": 1268
          },
          "verack": {
            "bytes": 474,
            "count": 16
          }
        }
      },
      "ipv6": {
        "feeler": {
          "sendcmpct": {
            "bytes": 30,
            "count": 1
          },
          "sendaddrv2": {
            "bytes": 33,
            "count": 1
          },
          "version": {
            "bytes": 261,
            "count": 2
          },
          "ping": {
            "bytes": 29,
            "count": 1
          },
          "wtxidrelay": {
            "bytes": 33,
            "count": 1
          },
          "verack": {
            "bytes": 57,
            "count": 2
          }
        },
        "inbound": {
          "getcfheaders": {
            "bytes": 976,
            "count": 16
          },
          "getcfilters": {
            "bytes": 1037,
            "count": 17
          },
          "getblocktxn": {
            "bytes": 66702,
            "count": 17
          },
          "sendcmpct": {
            "bytes": 2574,
            "count": 84
          },
          "sendaddrv2": {
            "bytes": 13368,
            "count": 521
          },
          "headers": {
            "bytes": 7701971,
            "count": 126
          },
          "feefilter": {
            "bytes": 1401,
            "count": 48
          },
          "notfound": {
            "bytes": 2835,
            "count": 42
          },
          "*other*": {
            "bytes": 324,
            "count": 12
          },
          "version": {
            "bytes": 88370,
            "count": 683
          },
          "ping": {
            "bytes": 76415,
            "count": 2455
          },
          "getaddr": {
            "bytes": 9654,
            "count": 382
          },
          "tx": {
            "bytes": 706464,
            "count": 2116
          },
          "getheaders": {
            "bytes": 305467,
            "count": 306
          },
          "pong": {
            "bytes": 91365,
            "count": 2922
          },
          "getdata": {
            "bytes": 1717706,
            "count": 7377
          },
          "inv": {
            "bytes": 11736308,
            "count": 19811
          },
          "sendheaders": {
            "bytes": 1869,
            "count": 61
          },
          "wtxidrelay": {
            "bytes": 4254,
            "count": 142
          },
          "addrv2": {
            "bytes": 102634,
            "count": 1105
          },
          "addr": {
            "bytes": 165,
            "count": 3
          },
          "verack": {
            "bytes": 16548,
            "count": 661
          }
        },
        "block-relay-only": {
          "sendcmpct": {
            "bytes": 30,
            "count": 1
          },
          "sendaddrv2": {
            "bytes": 33,
            "count": 1
          },
          "headers": {
            "bytes": 103,
            "count": 1
          },
          "feefilter": {
            "bytes": 29,
            "count": 1
          },
          "version": {
            "bytes": 135,
            "count": 1
          },
          "ping": {
            "bytes": 29,
            "count": 1
          },
          "getheaders": {
            "bytes": 1050,
            "count": 1
          },
          "pong": {
            "bytes": 29,
            "count": 1
          },
          "sendheaders": {
            "bytes": 33,
            "count": 1
          },
          "wtxidrelay": {
            "bytes": 33,
            "count": 1
          },
          "verack": {
            "bytes": 33,
            "count": 1
          }
        },
        "outbound-full-relay": {
          "sendcmpct": {
            "bytes": 63,
            "count": 2
          },
          "sendaddrv2": {
            "bytes": 81,
            "count": 3
          },
          "headers": {
            "bytes": 164735,
            "count": 9
          },
          "feefilter": {
            "bytes": 61,
            "count": 2
          },
          "notfound": {
            "bytes": 305,
            "count": 5
          },
          "version": {
            "bytes": 1144,
            "count": 9
          },
          "ping": {
            "bytes": 4445,
            "count": 139
          },
          "tx": {
            "bytes": 2133640,
            "count": 3896
          },
          "getheaders": {
            "bytes": 2007,
            "count": 2
          },
          "pong": {
            "bytes": 4445,
            "count": 139
          },
          "getdata": {
            "bytes": 2869,
            "count": 37
          },
          "inv": {
            "bytes": 691154,
            "count": 2786
          },
          "sendheaders": {
            "bytes": 24,
            "count": 1
          },
          "wtxidrelay": {
            "bytes": 81,
            "count": 3
          },
          "addrv2": {
            "bytes": 49482,
            "count": 205
          },
          "verack": {
            "bytes": 225,
            "count": 9
          }
        }
      },
      "ipv4": {
        "feeler": {
          "sendcmpct": {
            "bytes": 30,
            "count": 1
          },
          "sendaddrv2": {
            "bytes": 165,
            "count": 5
          },
          "version": {
            "bytes": 950,
            "count": 7
          },
          "wtxidrelay": {
            "bytes": 165,
            "count": 5
          },
          "verack": {
            "bytes": 213,
            "count": 7
          }
        },
        "inbound": {
          "getcfheaders": {
            "bytes": 244,
            "count": 4
          },
          "getcfilters": {
            "bytes": 183,
            "count": 3
          },
          "getblocktxn": {
            "bytes": 55161,
            "count": 15
          },
          "sendcmpct": {
            "bytes": 12099,
            "count": 399
          },
          "sendaddrv2": {
            "bytes": 33105,
            "count": 1178
          },
          "headers": {
            "bytes": 36128046,
            "count": 814
          },
          "feefilter": {
            "bytes": 6158,
            "count": 211
          },
          "notfound": {
            "bytes": 15714,
            "count": 207
          },
          "*other*": {
            "bytes": 324,
            "count": 12
          },
          "version": {
            "bytes": 188525,
            "count": 1405
          },
          "ping": {
            "bytes": 136003,
            "count": 4538
          },
          "getaddr": {
            "bytes": 22929,
            "count": 853
          },
          "cmpctblock": {
            "bytes": 190114,
            "count": 6
          },
          "block": {
            "bytes": 4920635,
            "count": 4
          },
          "tx": {
            "bytes": 28321664,
            "count": 84510
          },
          "getheaders": {
            "bytes": 540716,
            "count": 531
          },
          "pong": {
            "bytes": 192764,
            "count": 6313
          },
          "getdata": {
            "bytes": 5137343,
            "count": 32117
          },
          "inv": {
            "bytes": 48773424,
            "count": 86883
          },
          "sendheaders": {
            "bytes": 9759,
            "count": 308
          },
          "wtxidrelay": {
            "bytes": 19008,
            "count": 594
          },
          "addrv2": {
            "bytes": 246799,
            "count": 3460
          },
          "addr": {
            "bytes": 165,
            "count": 3
          },
          "verack": {
            "bytes": 34350,
            "count": 1270
          }
        },
        "block-relay-only": {
          "sendcmpct": {
            "bytes": 186,
            "count": 6
          },
          "sendaddrv2": {
            "bytes": 156,
            "count": 5
          },
          "headers": {
            "bytes": 8356,
            "count": 61
          },
          "feefilter": {
            "bytes": 148,
            "count": 5
          },
          "version": {
            "bytes": 1184,
            "count": 9
          },
          "ping": {
            "bytes": 11311,
            "count": 371
          },
          "getheaders": {
            "bytes": 5253,
            "count": 5
          },
          "pong": {
            "bytes": 11311,
            "count": 371
          },
          "sendheaders": {
            "bytes": 156,
            "count": 5
          },
          "wtxidrelay": {
            "bytes": 156,
            "count": 5
          },
          "addrv2": {
            "bytes": 5933,
            "count": 114
          },
          "verack": {
            "bytes": 252,
            "count": 9
          }
        },
        "outbound-full-relay": {
          "sendcmpct": {
            "bytes": 336,
            "count": 11
          },
          "sendaddrv2": {
            "bytes": 378,
            "count": 12
          },
          "headers": {
            "bytes": 8638418,
            "count": 98
          },
          "feefilter": {
            "bytes": 325,
            "count": 11
          },
          "notfound": {
            "bytes": 1490,
            "count": 17
          },
          "version": {
            "bytes": 1640,
            "count": 12
          },
          "ping": {
            "bytes": 14738,
            "count": 502
          },
          "cmpctblock": {
            "bytes": 1418449,
            "count": 46
          },
          "tx": {
            "bytes": 21044915,
            "count": 59532
          },
          "getheaders": {
            "bytes": 21342,
            "count": 92
          },
          "pong": {
            "bytes": 14738,
            "count": 502
          },
          "getdata": {
            "bytes": 13310,
            "count": 131
          },
          "inv": {
            "bytes": 6750879,
            "count": 10224
          },
          "sendheaders": {
            "bytes": 312,
            "count": 10
          },
          "wtxidrelay": {
            "bytes": 378,
            "count": 12
          },
          "addrv2": {
            "bytes": 257639,
            "count": 728
          },
          "blocktxn": {
            "bytes": 402708,
            "count": 15
          },
          "verack": {
            "bytes": 378,
            "count": 12
          }
        }
      }
    },
    "sent": {
      "i2p": {
        "feeler": {
          "sendaddrv2": {
            "bytes": 132,
            "count": 4
          },
          "version": {
            "bytes": 671,
            "count": 5
          },
          "wtxidrelay": {
            "bytes": 132,
            "count": 4
          },
          "verack": {
            "bytes": 132,
            "count": 4
          }
        },
        "inbound": {
          "sendcmpct": {
            "bytes": 2961,
            "count": 96
          },
          "sendaddrv2": {
            "bytes": 4491,
            "count": 147
          },
          "headers": {
            "bytes": 1968395,
            "count": 245
          },
          "feefilter": {
            "bytes": 2750,
            "count": 91
          },
          "notfound": {
            "bytes": 26978,
            "count": 281
          },
          "version": {
            "bytes": 19632,
            "count": 147
          },
          "ping": {
            "bytes": 30385,
            "count": 1034
          },
          "cmpctblock": {
            "bytes": 1028207,
            "count": 31
          },
          "block": {
            "bytes": 13581140,
            "count": 9
          },
          "tx": {
            "bytes": 11354505,
            "count": 29797
          },
          "getheaders": {
            "bytes": 82974,
            "count": 83
          },
          "pong": {
            "bytes": 28433,
            "count": 970
          },
          "getdata": {
            "bytes": 1669965,
            "count": 2910
          },
          "inv": {
            "bytes": 8536101,
            "count": 14425
          },
          "sendheaders": {
            "bytes": 1134,
            "count": 36
          },
          "wtxidrelay": {
            "bytes": 4491,
            "count": 147
          },
          "addrv2": {
            "bytes": 1505470,
            "count": 1994
          },
          "blocktxn": {
            "bytes": 2885474,
            "count": 8
          },
          "verack": {
            "bytes": 4491,
            "count": 147
          }
        },
        "block-relay-only": {
          "sendcmpct": {
            "bytes": 60,
            "count": 2
          },
          "sendaddrv2": {
            "bytes": 66,
            "count": 2
          },
          "headers": {
            "bytes": 1458319,
            "count": 10
          },
          "version": {
            "bytes": 272,
            "count": 2
          },
          "ping": {
            "bytes": 58,
            "count": 2
          },
          "getheaders": {
            "bytes": 4200,
            "count": 4
          },
          "pong": {
            "bytes": 58,
            "count": 2
          },
          "sendheaders": {
            "bytes": 33,
            "count": 1
          },
          "wtxidrelay": {
            "bytes": 66,
            "count": 2
          },
          "verack": {
            "bytes": 66,
            "count": 2
          }
        },
        "outbound-full-relay": {
          "sendcmpct": {
            "bytes": 90,
            "count": 3
          },
          "sendaddrv2": {
            "bytes": 99,
            "count": 3
          },
          "headers": {
            "bytes": 7210,
            "count": 70
          },
          "feefilter": {
            "bytes": 87,
            "count": 3
          },
          "notfound": {
            "bytes": 5726,
            "count": 95
          },
          "version": {
            "bytes": 408,
            "count": 3
          },
          "ping": {
            "bytes": 8004,
            "count": 276
          },
          "getaddr": {
            "bytes": 99,
            "count": 3
          },
          "cmpctblock": {
            "bytes": 43119,
            "count": 1
          },
          "tx": {
            "bytes": 4116198,
            "count": 16101
          },
          "getheaders": {
            "bytes": 3150,
            "count": 3
          },
          "pong": {
            "bytes": 7975,
            "count": 275
          },
          "getdata": {
            "bytes": 309526,
            "count": 1705
          },
          "inv": {
            "bytes": 6791282,
            "count": 11680
          },
          "sendheaders": {
            "bytes": 99,
            "count": 3
          },
          "wtxidrelay": {
            "bytes": 99,
            "count": 3
          },
          "addrv2": {
            "bytes": 39694,
            "count": 547
          },
          "verack": {
            "bytes": 99,
            "count": 3
          }
        }
      },
      "onion": {
        "feeler": {
          "sendaddrv2": {
            "bytes": 672,
            "count": 22
          },
          "version": {
            "bytes": 2938,
            "count": 22
          },
          "wtxidrelay": {
            "bytes": 672,
            "count": 22
          },
          "verack": {
            "bytes": 672,
            "count": 22
          }
        },
        "inbound": {
          "cfheaders": {
            "bytes": 8235234,
            "count": 152
          },
          "sendcmpct": {
            "bytes": 24222,
            "count": 744
          },
          "cfilter": {
            "bytes": 22844,
            "count": 1
          },
          "sendaddrv2": {
            "bytes": 22425,
            "count": 868
          },
          "headers": {
            "bytes": 3749134,
            "count": 345
          },
          "feefilter": {
            "bytes": 23449,
            "count": 743
          },
          "notfound": {
            "bytes": 47859,
            "count": 276
          },
          "version": {
            "bytes": 111829,
            "count": 868
          },
          "ping": {
            "bytes": 59370,
            "count": 1917
          },
          "cmpctblock": {
            "bytes": 728569,
            "count": 23
          },
          "block": {
            "bytes": 333742085,
            "count": 209
          },
          "tx": {
            "bytes": 21443260,
            "count": 69085
          },
          "getheaders": {
            "bytes": 158592,
            "count": 151
          },
          "pong": {
            "bytes": 45855,
            "count": 1491
          },
          "getdata": {
            "bytes": 278528,
            "count": 2333
          },
          "inv": {
            "bytes": 11575046,
            "count": 10096
          },
          "sendheaders": {
            "bytes": 1926,
            "count": 60
          },
          "wtxidrelay": {
            "bytes": 22425,
            "count": 868
          },
          "addrv2": {
            "bytes": 12480168,
            "count": 1941
          },
          "blocktxn": {
            "bytes": 2824483,
            "count": 10
          },
          "verack": {
            "bytes": 22425,
            "count": 868
          }
        },
        "block-relay-only": {
          "sendcmpct": {
            "bytes": 150,
            "count": 5
          },
          "sendaddrv2": {
            "bytes": 165,
            "count": 5
          },
          "headers": {
            "bytes": 515,
            "count": 5
          },
          "version": {
            "bytes": 680,
            "count": 5
          },
          "ping": {
            "bytes": 145,
            "count": 5
          },
          "getheaders": {
            "bytes": 5250,
            "count": 5
          },
          "pong": {
            "bytes": 145,
            "count": 5
          },
          "sendheaders": {
            "bytes": 165,
            "count": 5
          },
          "wtxidrelay": {
            "bytes": 165,
            "count": 5
          },
          "verack": {
            "bytes": 165,
            "count": 5
          }
        },
        "outbound-full-relay": {
          "sendcmpct": {
            "bytes": 498,
            "count": 16
          },
          "sendaddrv2": {
            "bytes": 474,
            "count": 16
          },
          "headers": {
            "bytes": 14288,
            "count": 137
          },
          "feefilter": {
            "bytes": 482,
            "count": 16
          },
          "notfound": {
            "bytes": 7469,
            "count": 98
          },
          "version": {
            "bytes": 2122,
            "count": 16
          },
          "ping": {
            "bytes": 16665,
            "count": 552
          },
          "getaddr": {
            "bytes": 474,
            "count": 16
          },
          "cmpctblock": {
            "bytes": 1344325,
            "count": 40
          },
          "tx": {
            "bytes": 31488390,
            "count": 131785
          },
          "getheaders": {
            "bytes": 18918,
            "count": 18
          },
          "pong": {
            "bytes": 16665,
            "count": 552
          },
          "getdata": {
            "bytes": 1236881,
            "count": 5732
          },
          "inv": {
            "bytes": 19452601,
            "count": 22645
          },
          "sendheaders": {
            "bytes": 474,
            "count": 16
          },
          "wtxidrelay": {
            "bytes": 474,
            "count": 16
          },
          "addrv2": {
            "bytes": 76865,
            "count": 1026
          },
          "blocktxn": {
            "bytes": 7513424,
            "count": 23
          },
          "verack": {
            "bytes": 474,
            "count": 16
          }
        }
      },
      "ipv6": {
        "feeler": {
          "sendaddrv2": {
            "bytes": 57,
            "count": 2
          },
          "version": {
            "bytes": 390,
            "count": 3
          },
          "wtxidrelay": {
            "bytes": 57,
            "count": 2
          },
          "verack": {
            "bytes": 57,
            "count": 2
          }
        },
        "inbound": {
          "cfheaders": {
            "bytes": 1862,
            "count": 15
          },
          "sendcmpct": {
            "bytes": 19713,
            "count": 605
          },
          "cfilter": {
            "bytes": 13309812,
            "count": 649
          },
          "sendaddrv2": {
            "bytes": 16176,
            "count": 638
          },
          "headers": {
            "bytes": 9001618,
            "count": 447
          },
          "feefilter": {
            "bytes": 19045,
            "count": 602
          },
          "notfound": {
            "bytes": 77755,
            "count": 529
          },
          "*other*": {
            "bytes": 4992,
            "count": 26
          },
          "version": {
            "bytes": 87605,
            "count": 683
          },
          "ping": {
            "bytes": 100734,
            "count": 3222
          },
          "cmpctblock": {
            "bytes": 1679473,
            "count": 54
          },
          "block": {
            "bytes": 631983309,
            "count": 431
          },
          "tx": {
            "bytes": 18700462,
            "count": 40532
          },
          "getheaders": {
            "bytes": 176586,
            "count": 168
          },
          "pong": {
            "bytes": 76142,
            "count": 2446
          },
          "getdata": {
            "bytes": 90122,
            "count": 542
          },
          "inv": {
            "bytes": 82845445,
            "count": 44248
          },
          "sendheaders": {
            "bytes": 1293,
            "count": 40
          },
          "wtxidrelay": {
            "bytes": 16176,
            "count": 638
          },
          "addrv2": {
            "bytes": 7163154,
            "count": 3283
          },
          "blocktxn": {
            "bytes": 16959500,
            "count": 17
          },
          "addr": {
            "bytes": 371337,
            "count": 313
          },
          "verack": {
            "bytes": 17256,
            "count": 683
          }
        },
        "block-relay-only": {
          "sendcmpct": {
            "bytes": 30,
            "count": 1
          },
          "sendaddrv2": {
            "bytes": 33,
            "count": 1
          },
          "headers": {
            "bytes": 103,
            "count": 1
          },
          "version": {
            "bytes": 136,
            "count": 1
          },
          "ping": {
            "bytes": 29,
            "count": 1
          },
          "getheaders": {
            "bytes": 1050,
            "count": 1
          },
          "pong": {
            "bytes": 29,
            "count": 1
          },
          "sendheaders": {
            "bytes": 33,
            "count": 1
          },
          "wtxidrelay": {
            "bytes": 33,
            "count": 1
          },
          "verack": {
            "bytes": 33,
            "count": 1
          }
        },
        "outbound-full-relay": {
          "sendcmpct": {
            "bytes": 294,
            "count": 9
          },
          "sendaddrv2": {
            "bytes": 225,
            "count": 9
          },
          "headers": {
            "bytes": 131470,
            "count": 33
          },
          "feefilter": {
            "bytes": 285,
            "count": 9
          },
          "notfound": {
            "bytes": 183,
            "count": 3
          },
          "version": {
            "bytes": 1424,
            "count": 11
          },
          "ping": {
            "bytes": 4669,
            "count": 146
          },
          "getaddr": {
            "bytes": 225,
            "count": 9
          },
          "tx": {
            "bytes": 23507,
            "count": 51
          },
          "getheaders": {
            "bytes": 16845,
            "count": 16
          },
          "pong": {
            "bytes": 4445,
            "count": 139
          },
          "getdata": {
            "bytes": 182411,
            "count": 1679
          },
          "inv": {
            "bytes": 1209111,
            "count": 5343
          },
          "sendheaders": {
            "bytes": 24,
            "count": 1
          },
          "wtxidrelay": {
            "bytes": 225,
            "count": 9
          },
          "addrv2": {
            "bytes": 20633,
            "count": 259
          },
          "addr": {
            "bytes": 665,
            "count": 11
          },
          "verack": {
            "bytes": 225,
            "count": 9
          }
        }
      },
      "ipv4": {
        "feeler": {
          "sendaddrv2": {
            "bytes": 213,
            "count": 7
          },
          "version": {
            "bytes": 1061,
            "count": 8
          },
          "wtxidrelay": {
            "bytes": 213,
            "count": 7
          },
          "verack": {
            "bytes": 213,
            "count": 7
          }
        },
        "inbound": {
          "cfheaders": {
            "bytes": 64458,
            "count": 4
          },
          "sendcmpct": {
            "bytes": 37701,
            "count": 1182
          },
          "cfilter": {
            "bytes": 20405,
            "count": 3
          },
          "sendaddrv2": {
            "bytes": 37227,
            "count": 1343
          },
          "headers": {
            "bytes": 19568359,
            "count": 1044
          },
          "feefilter": {
            "bytes": 36465,
            "count": 1179
          },
          "notfound": {
            "bytes": 434752,
            "count": 2196
          },
          "*other*": {
            "bytes": 5184,
            "count": 27
          },
          "version": {
            "bytes": 183430,
            "count": 1405
          },
          "ping": {
            "bytes": 210979,
            "count": 6897
          },
          "cmpctblock": {
            "bytes": 948391,
            "count": 31
          },
          "block": {
            "bytes": 9977590378,
            "count": 12054
          },
          "tx": {
            "bytes": 35121307,
            "count": 99150
          },
          "getheaders": {
            "bytes": 706974,
            "count": 673
          },
          "pong": {
            "bytes": 135742,
            "count": 4529
          },
          "getdata": {
            "bytes": 3430615,
            "count": 17317
          },
          "inv": {
            "bytes": 126231463,
            "count": 77745
          },
          "sendheaders": {
            "bytes": 8205,
            "count": 256
          },
          "wtxidrelay": {
            "bytes": 37227,
            "count": 1343
          },
          "addrv2": {
            "bytes": 11934125,
            "count": 6866
          },
          "blocktxn": {
            "bytes": 13856854,
            "count": 15
          },
          "addr": {
            "bytes": 852179,
            "count": 627
          },
          "verack": {
            "bytes": 38715,
            "count": 1405
          }
        },
        "block-relay-only": {
          "sendcmpct": {
            "bytes": 252,
            "count": 8
          },
          "sendaddrv2": {
            "bytes": 228,
            "count": 8
          },
          "headers": {
            "bytes": 9293,
            "count": 89
          },
          "version": {
            "bytes": 1569,
            "count": 12
          },
          "ping": {
            "bytes": 11407,
            "count": 374
          },
          "getheaders": {
            "bytes": 17889,
            "count": 17
          },
          "pong": {
            "bytes": 11311,
            "count": 371
          },
          "sendheaders": {
            "bytes": 156,
            "count": 5
          },
          "wtxidrelay": {
            "bytes": 228,
            "count": 8
          },
          "verack": {
            "bytes": 228,
            "count": 8
          }
        },
        "outbound-full-relay": {
          "getblocktxn": {
            "bytes": 2604,
            "count": 15
          },
          "sendcmpct": {
            "bytes": 426,
            "count": 14
          },
          "sendaddrv2": {
            "bytes": 378,
            "count": 12
          },
          "headers": {
            "bytes": 13290524,
            "count": 126
          },
          "feefilter": {
            "bytes": 354,
            "count": 12
          },
          "notfound": {
            "bytes": 1594,
            "count": 25
          },
          "version": {
            "bytes": 1614,
            "count": 12
          },
          "ping": {
            "bytes": 14767,
            "count": 503
          },
          "getaddr": {
            "bytes": 378,
            "count": 12
          },
          "tx": {
            "bytes": 121642,
            "count": 260
          },
          "getheaders": {
            "bytes": 22222,
            "count": 68
          },
          "pong": {
            "bytes": 14738,
            "count": 502
          },
          "getdata": {
            "bytes": 2315007,
            "count": 7653
          },
          "inv": {
            "bytes": 9941029,
            "count": 18706
          },
          "sendheaders": {
            "bytes": 312,
            "count": 10
          },
          "wtxidrelay": {
            "bytes": 378,
            "count": 12
          },
          "addrv2": {
            "bytes": 76689,
            "count": 1000
          },
          "verack": {
            "bytes": 378,
            "count": 12
          }
        }
      }
    }
  }
}

</details>

<details> <summary>Omit the new field</summary>

$ bitcoin-cli getnettotals | jq 'del(.detailed)'
{
  "totalbytesrecv": 290796106,
  "totalbytessent": 11516332695,
  "timemillis": 1792494749925,
  "uploadtarget": {
    "timeframe": 86400,
    "target": 0,
    "target_reached": false,
    "serve_historical_blocks": true,
    "bytes_left_in_cycle": 0,
    "time_left_in_cycle": 0
  }
}

</details>

Previous incarnations of this: #27534 rpc: add 'getnetmsgstats', new rpc to view network message statistics (Thank you, @satsie!) #28926 rpc: add 'getnetmsgstats' RPC (Thank you, @willcl-ark!)

DrahtBot commented at 9:32 AM on February 11, 2024: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

External sites

Code coverage & benchmarks: https://corecheck.dev/bitcoin/bitcoin/pulls/29418
Ralph review <img src=https://review.fish.foo/assets/bitcoin/bitcoin/29418.png alt="Ralph review status"> https://git.fish.foo/bitcoin/bitcoin/pulls/29418#author-ralph

Reviews

See the guideline and AI policy for information on the review process.

Type	Reviewers
Concept ACK	epiccurious, danielabrozzoni, stickies-v
Stale ACK	waketraindev

If your review is incorrectly listed, please copy-paste <code></code> into the comment that the bot should ignore.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#36366 <img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/36366.svg"> (net: limit relative ban times to uint32 by Bruce039)
#35558 <img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/35558.svg"> (p2p: Prefill compact blocks by davidgumberg)
#30951 <img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/30951.svg"> (net: option to disallow v1 connection on ipv4 and ipv6 peers by stratospher)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

DrahtBot added the label RPC/REST/ZMQ on Feb 11, 2024

DrahtBot added the label CI failed on Feb 11, 2024

DrahtBot commented at 10:51 AM on February 11, 2024: contributor

🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the documentation.

Possibly this is due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

Leave a comment here, if you need help tracking down a confusing failure.

Debug: https://github.com/bitcoin/bitcoin/runs/21448984768

vasild force-pushed on Feb 11, 2024

DrahtBot removed the label CI failed on Feb 11, 2024

epiccurious commented at 10:26 PM on February 11, 2024: contributor

Concept ACK 91c90d759503aa3fa15b330bba2148698cdc271d.

DrahtBot added the label CI failed on Mar 12, 2024

DrahtBot commented at 2:24 PM on April 4, 2024: contributor

Needs rebase if still relevant

achow101 requested review from mzumsande on Apr 9, 2024

0xB10C commented at 4:20 PM on April 9, 2024: contributor

just fyi as this came up: You could count up message stats with the inbound and outbound message tracepoints. However, this does only gives you stats from the points on where you start hooking into the tracepoints, might not be as user friendly (RPC is easier to use than the tracepoints), and the tracepoints currently only work on Linux.

While I personally don't need it, I see why some node operators need traffic statistics. E.g. How much of my (limited monthly) traffic is used up by inbound connections?

DrahtBot marked this as a draft on Apr 23, 2024

DrahtBot commented at 6:28 AM on April 23, 2024: contributor

Converted to draft due to failing CI and inactivity

vasild force-pushed on May 9, 2024

vasild commented at 1:40 PM on May 9, 2024: contributor

91c90d7595...6bfda84b92: rebase, pick changes from #29421 and adjust the newly added test.

DrahtBot removed the label CI failed on May 9, 2024

achow101 referenced this in commit 6c13b1375f on May 21, 2024

DrahtBot added the label Needs rebase on May 21, 2024

vasild force-pushed on May 23, 2024

vasild commented at 9:51 AM on May 23, 2024: contributor

6bfda84b92...b5189f543c: rebase due to conflicts (part of this PR was merged via #29421)

vasild marked this as ready for review on May 23, 2024

DrahtBot removed the label Needs rebase on May 23, 2024

vasild force-pushed on May 23, 2024

DrahtBot added the label CI failed on May 23, 2024

DrahtBot removed the label CI failed on May 23, 2024

DrahtBot added the label Needs rebase on Aug 13, 2024

vasild force-pushed on Sep 3, 2024

vasild commented at 10:30 AM on September 3, 2024: contributor

68dfd4270c...f85814052d: rebase due to conflicts and to pick CMake

DrahtBot removed the label Needs rebase on Sep 3, 2024

DrahtBot added the label Needs rebase on Sep 19, 2024

vasild force-pushed on Sep 30, 2024

vasild commented at 10:59 AM on September 30, 2024: contributor

f85814052d...a8e531f92f: rebase due to conflicts

DrahtBot removed the label Needs rebase on Sep 30, 2024

in src/rpc/net.cpp:None in a8e531f92f outdated

 609 | +
 610 | +    auto Add = [](UniValue& target, const std::string& key, const UniValue& val) {
 611 | +        if (val.empty()) {
 612 | +            return;
 613 | +        }
 614 | +        if (key == "") {

jonatack commented at 10:27 PM on October 10, 2024:

        if (key.empty()) {

vasild commented at 11:21 AM on October 28, 2024:

Done

in test/functional/rpc_net.py:None in a8e531f92f outdated

 214 | +        assert_equal(len(node0.getpeerinfo()), 0)
 215 | +
 216 | +        # In v2 getnettotals counts also bytes that are not accounted at any message (the v2 handshake).
 217 | +        # Also the v2 handshake's size could vary.
 218 | +        if not self.options.v2transport:
 219 | +            self.log.info("Compare byte count getnetmsgstats vs getnettotals")

jonatack commented at 10:38 PM on October 10, 2024:

Suggest making all the added test logging after "Test getnetmsgstats" debug level rather than info.

            self.log.debug("Compare byte count getnetmsgstats vs getnettotals")

vasild commented at 11:21 AM on October 28, 2024:

Done

in src/rpc/net.cpp:None in a8e531f92f outdated

 657 | +        "Results may optionally be aggregated.\n",
 658 | +        /*args=*/
 659 | +        {RPCArg{
 660 | +            /*name=*/"aggregate_by",
 661 | +            /*type=*/RPCArg::Type::ARR,
 662 | +            /*fallback=*/RPCArg::DefaultHint{"empty (no aggregation)"},

jonatack commented at 10:40 PM on October 10, 2024:

nit, this prints as:

(json array, optional, default=empty (no aggregation)) An array of keywords for aggregating the results.

Perhaps avoid the double braces with

(json array, optional, default=empty, no aggregation) An array of keywords for aggregating the results.

            /*fallback=*/RPCArg::DefaultHint{"empty, no aggregation"},

vasild commented at 11:21 AM on October 28, 2024:

Done

in test/functional/rpc_net.py:None in a8e531f92f outdated

 257 | +                            "sendaddrv2": {"bytes": 33 if self.options.v2transport else 24, "count": 1},
 258 | +                            "sendcmpct": {"bytes": 30 if self.options.v2transport else 33, "count": 1},
 259 | +                            "sendheaders": {"bytes": 33 if self.options.v2transport else 24, "count": 1},
 260 | +                            "verack": {"bytes": 33 if self.options.v2transport else 24, "count": 1},
 261 | +                            "version": {"bytes": 147 if self.options.v2transport else 138, "count": 1},
 262 | +                            "wtxidrelay": {"bytes": 33 if self.options.v2transport else 24, "count": 1},

jonatack commented at 1:45 PM on October 11, 2024:

for lines 233-262, it might be more efficient to replace all of these if self.options.v2transport conditionals with just one (and if v1 is ever removed, doing so would be a cleaner diff)

vasild commented at 11:20 AM on October 28, 2024:

Done

in src/net.h:None in a8e531f92f outdated

 723 | +    // Arrays are used so that this can be accessed from multiple threads without a mutex protection.
 724 | +    std::array<std::array<std::array<std::array<BytesAndCount, ALL_NET_MESSAGE_TYPES.size() + 1>,
 725 | +                                     NUM_CONNECTION_TYPES>,
 726 | +                          NET_MAX>,
 727 | +               NUM_DIRECTIONS>
 728 | +        m_data;

jonatack commented at 1:45 PM on October 11, 2024:

I'm confused by the formatting here.

satsie commented at 6:42 PM on October 11, 2024:

The thinking behind this is because it is a 4D array, all the nesting required to create it can be a headache to look at. Lines 725 - 727 are the array sizes, the second arguments needed for std::array creation. Each one of these lines up with the std::array they are part of.

For example, NUM_DIRECTIONS finishes the declaration off the parent level std::array that all the others are nested in. For that reason, NUM_DIRECTIONS lines up with the space after that array's opening <.

Hope that made some sense! Open to hearing any suggestions on how to make this easier to read.

in src/net.h:None in a8e531f92f outdated

 655 | @@ -656,6 +656,84 @@ class V2Transport final : public Transport
 656 |      Info GetInfo() const noexcept override EXCLUSIVE_LOCKS_REQUIRED(!m_recv_mutex);
 657 |  };
 658 |  
 659 | +/**
 660 | + * Network traffic (bytes and number of messages). Split by direction, network, connection type and message type.
 661 | + */
 662 | +class NetStats

jonatack commented at 1:49 PM on October 11, 2024:

Would there be any benefit to moving this class outside of net.{h,cpp}? It's quite a lot of code.

(Though, much of its verbosity is the formatting style used that often reads top-to-bottom rather than left-to-right. I prefer the latter.)

vasild commented at 11:19 AM on October 28, 2024:

No opinion.

in src/net.cpp:None in a8e531f92f outdated

3809 | +    case 3: return ConnectionType::FEELER;
3810 | +    case 4: return ConnectionType::BLOCK_RELAY;
3811 | +    case 5: return ConnectionType::ADDR_FETCH;
3812 | +    }
3813 | +    assert(false);
3814 | +}

jonatack commented at 1:52 PM on October 11, 2024:

Would it make sense to put these 2 ConnectionType utility methods in node/connection_types.{h,cpp} where they could be more widely used?

vasild commented at 11:17 AM on October 28, 2024:

These are to/from "index". By "index" here I mean the index in some array that is private in NetStats. They are private methods in NetStats.

Not sure exposing those to the outside would be useful. Unless to imply a broader meaning in "index", something like: "a zero based number without gaps". E.g. if there are 3 enums, then each one of them uniquely maps to one of the numbers 0, 1, 2. Would that be useful to some other code, outside of NetStats?

in src/net.cpp:None in a8e531f92f outdated

3783 | +    case 4: return NET_I2P;
3784 | +    case 5: return NET_CJDNS;
3785 | +    case 6: return NET_INTERNAL;
3786 | +    }
3787 | +    assert(false);
3788 | +}

jonatack commented at 1:54 PM on October 11, 2024:

Note to self, these 2 Network methods might be candidates for extraction in #27385.

vasild commented at 11:16 AM on October 28, 2024:

See #29418 (review)

jonatack commented at 1:54 PM on October 11, 2024: member

Lightly tested, a few initial questions/comments. (Edit: I wonder if the code can be a bit less; it seems quite verbose.)

vasild force-pushed on Oct 28, 2024

vasild commented at 11:06 AM on October 28, 2024: contributor

a8e531f92f...f42561e667: rebase and address suggestions

DrahtBot added the label Needs rebase on Dec 11, 2024

vasild force-pushed on Jan 23, 2025

vasild commented at 10:46 AM on January 23, 2025: contributor

f42561e667...58538c0847: rebase due to conflicts

DrahtBot removed the label Needs rebase on Jan 23, 2025

DrahtBot added the label CI failed on Jan 23, 2025

DrahtBot commented at 12:20 PM on January 23, 2025: contributor

🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/36052867071

<details><summary>Hints</summary>

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

</details>

vasild force-pushed on Jan 23, 2025

vasild commented at 12:43 PM on January 23, 2025: contributor

58538c0847...a46cbdd478: pet the py linter

in src/rpc/net.cpp:None in a46cbdd478 outdated

 674 | +                    "Aggregate by network and don't show network in the result."},
 675 | +             RPCArg{/*name=*/"connection_type",
 676 | +                    /*type=*/RPCArg::Type::STR,
 677 | +                    /*fallback=*/RPCArg::Optional::OMITTED,
 678 | +                    /*description=*/
 679 | +                    "Aggregate by connection type and don't show connection type in the result."},

maflcko commented at 1:24 PM on January 23, 2025:

no opinion on the changes, but this is a bit verbose and unclear why. clang-tidy named args can be helpful when there is a risk of mixing args, but for example putting RPCArg::Type anywhere else will already fail to compile before clang-tidy gets a chance to look at it. Similar reasoning for the string, etc...

vasild commented at 8:35 AM on January 24, 2025:

I am fine with anything. Do you suggest to drop /*name=*/, /*type=*/, /*fallback=*/ and /*description=*/? If you make a suggestion in a form of a diff or new code, how you think it would be better, I will apply it right away.

maflcko commented at 8:49 AM on January 24, 2025:

Do you suggest to drop /*name=*/, /*type=*/, /*fallback=*/ and /*description=*/?

Yes, I think it would be better to drop them in this case, because:

This would be the first and only RPC to do this, so it is inconsistent.
There isn't any benefit to do it here in this context (as explained above).
It is overly verbose.
It will break scripted-diffs in the future that assume a halfway uniform format.
If there was a benefit or reason to do it, it may be better to do it for all RPCs in a scripted-diff.

But no strong opinion. It is just a drive-by style-nit, so feel free to ignore.

vasild commented at 4:16 PM on January 24, 2025:

Agree. Removed.

DrahtBot removed the label CI failed on Jan 23, 2025

vasild force-pushed on Jan 24, 2025

vasild commented at 4:16 PM on January 24, 2025: contributor

a46cbdd478...db442f3d6e: do #29418 (review)

DrahtBot added the label Needs rebase on Mar 20, 2025

vasild force-pushed on Mar 20, 2025

vasild commented at 5:38 PM on March 20, 2025: contributor

db442f3d6e...626cc06c69: rebase due to conflicts

DrahtBot removed the label Needs rebase on Mar 20, 2025

DrahtBot added the label CI failed on May 18, 2025

in src/rpc/net.cpp:None in 626cc06c69 outdated

 650 | +
 651 | +static RPCHelpMan getnetmsgstats()
 652 | +{
 653 | +    return RPCHelpMan{
 654 | +        "getnetmsgstats",
 655 | +        "\nReturns the messages count and total number of bytes for network traffic.\n"

maflcko commented at 8:36 PM on May 20, 2025:

        "Returns the messages count and total number of bytes for network traffic.\n"

the ci failure is a bit vague, but I guess this would fix it

vasild commented at 4:18 AM on May 22, 2025:

"All checks have passed", thank you :heart:

vasild force-pushed on May 21, 2025

vasild commented at 9:00 AM on May 21, 2025: contributor

626cc06c69...528b4d6b9c: rebase and do #29418 (review)

DrahtBot removed the label CI failed on May 21, 2025

in test/functional/rpc_net.py:None in 528b4d6b9c outdated

 353 | +
 354 | +        self.log.debug("Test that message count and total number of bytes increment when a ping message is sent")
 355 | +        stats_before_connect = get_stats(node0)
 356 | +        node2 = node0.add_p2p_connection(P2PInterface())
 357 | +        assert_equal(len(node0.getpeerinfo()), 1)
 358 | +        # Wait for the initial PING (that is sent immediately after the connection is estabilshed) to go through.

DrahtBot commented at 8:32 AM on June 25, 2025:

typo -> established [Typo]

DrahtBot added the label Needs rebase on Jul 25, 2025

vasild force-pushed on Sep 9, 2025

vasild commented at 8:03 AM on September 9, 2025: contributor

528b4d6b9c...87d3d640e6: rebase due to conflicts and fix a typo in comment

DrahtBot removed the label Needs rebase on Sep 9, 2025

vasild force-pushed on Sep 10, 2025

vasild commented at 7:57 AM on September 10, 2025: contributor

87d3d640e6...7d4e82c0bb: remove leading \n from RPC help

waketraindev commented at 1:19 AM on December 29, 2025: contributor

Concept ACK

Useful netstats for tracking. Normally, I'd sum and aggregate peer stats. This approach simplifies that.

waketraindev commented at 4:30 PM on December 29, 2025: contributor

ACK 7d4e82c0bbe763acac9ba8770a40c162b79054a7

lgtm. Tested on master and with v30.1 in my node monitoring dashboard.

sedited requested review from danielabrozzoni on Feb 9, 2026

in src/net.cpp:1629 in c26c568ac3 outdated

1621 | @@ -1610,9 +1622,16 @@ std::pair<size_t, bool> CConnman::SocketSendData(CNode& node) const
1622 |              // there is an existing message still being sent, or (for v2 transports) when the
1623 |              // handshake has not yet completed.
1624 |              size_t memusage = it->GetMemoryUsage();
1625 | +            const auto msg_type = it->m_type;
1626 |              if (node.m_transport->SetMessageToSend(*it)) {
1627 |                  // Update memory usage of send buffer (as *it will be deleted).
1628 |                  node.m_send_memusage -= memusage;
1629 | +                m_net_stats.Record(NetStats::SENT,

danielabrozzoni commented at 10:26 AM on February 10, 2026:

nit: I would add a short comment here and below to clarify why there are two separate Record calls. For example: "we record a message when it is accepted by the transport, and we record bytes later, incrementally, when they are actually written to the socket."

vasild commented at 8:45 AM on May 18, 2026:

Added, thanks!

in test/functional/rpc_net.py:324 in 7d4e82c0bb

 319 | +                return dict(bytes=json["bytes"], count=json["count"])
 320 | +
 321 | +            s = dict(bytes=0, count=0)
 322 | +            #print(f'S json={json}')
 323 | +            for k, v in json.items():
 324 | +                #print(f'S k={k}, v={v}')

danielabrozzoni commented at 11:32 AM on February 10, 2026:

Debug leftovers?

vasild commented at 8:45 AM on May 18, 2026:

Yes, removed.

danielabrozzoni commented at 12:13 PM on February 10, 2026: member

Concept ACK, reviewed up to 5ad1f5d83e1f8dddde90c81dab264df0d5e0dc5c, I still haven't finished the review of the last commit

I wonder if in the functional test, instead of hard-coding the message stats values, we could check how many messages/bytes are sent using P2PInterface... I'm not sure how this would look like or if it's possible at all, I'll try it out and report back :)

DrahtBot added the label Needs rebase on Feb 13, 2026

sedited requested review from danielabrozzoni on Mar 9, 2026

sedited removed review request from danielabrozzoni on Mar 9, 2026

sedited commented at 1:04 PM on March 9, 2026: contributor

Ping @vasild for rebase

sedited commented at 10:33 AM on May 14, 2026: contributor

@vasild are you still working on this?

vasild force-pushed on May 18, 2026

vasild commented at 8:45 AM on May 18, 2026: contributor

7d4e82c0bbe763acac9ba8770a40c162b79054a7...9fc5965c554244ae2db7cec2cf696eb7fd3a83d5: rebase and address suggestions

are you still working on this?

Yes. This and a few others got pushed way back on my TODO list, slowly processing it, thanks for the patience.

DrahtBot removed the label Needs rebase on May 18, 2026

DrahtBot added the label CI failed on May 18, 2026

vasild force-pushed on May 18, 2026

DrahtBot removed the label CI failed on May 18, 2026

sedited requested review from danielabrozzoni on May 22, 2026

sedited requested review from stickies-v on May 22, 2026

stickies-v commented at 10:57 AM on May 22, 2026: contributor

Concept ACK. Seems like useful info to expose.

Approach-wise, I like that it adds very little logic in the RPC layer. I quite strongly opppose the aggregation option, just a simple bitcoin-cli getnetmsgstats provides a user all the info they need. Visualization/aggregation can easily be done by the user, e.g. using jq or any other tool / aggregation they like, so adding that to our interface unnecessarily locks us into backwards compatibility promises and code/interface complexity (even if it's not a crazy amount).

I'm not sure I'll review this in-depth anytime soon, but will try to keep engaging on high-level.

in src/rpc/net.cpp:723 in 6293e25135

 718 | +                            RPCResult::Type::OBJ,
 719 | +                            "verack",
 720 | +                            true,
 721 | +                            "Type of the messages transferred.",
 722 | +                            {
 723 | +                                RPCResult{RPCResult::Type::NUM, "bytes", false, "Total number of bytes.", {}, {.skip_type_check = true}},

maflcko commented at 11:20 AM on May 22, 2026:

nit: You'll have to use named args for all those bools, see the LLM linter DrahtBot comment. Otherwise, it is unclear what they mean. Also, all other RPCs use named args for them.

vasild commented at 2:30 PM on June 12, 2026:

Done

vasild force-pushed on Jun 12, 2026

DrahtBot added the label CI failed on Jun 12, 2026

vasild commented at 2:29 PM on June 12, 2026: contributor

6293e251350dd2b6da6cd6018606e8f780b606f9...b2d092efcf94e721dab4a36ce3aeefa7f9d9af5f: rebase and address suggestions

I quite strongly oppose the aggregation option, just a simple bitcoin-cli getnetmsgstats provides a user all the info they need. Visualization/aggregation can easily be done by the user

Alright, that's it. I was thinking the same for some time now - "Should we do aggregation on the server side, given that it is more like a visualization/presentation thing?". Thanks for pushing here. I have now dropped the aggregation, which means removing commit d5a1ad981b rpc: make it possible to aggregate the result in getnetmsgstats and omitting the parts of the test that tested the aggregation and tweaking the parts of the test that used the aggregation.

Thanks!

DrahtBot removed the label CI failed on Jun 12, 2026

Kino1994 referenced this in commit a8cf92f3c5 on Jun 28, 2026

BigcoinBGC referenced this in commit 7d90eeef5a on Jun 30, 2026

DrahtBot added the label Needs rebase on Jul 14, 2026

sedited commented at 8:27 PM on August 10, 2026: contributor

@vasild ping for rebase.

Kino1994 referenced this in commit 81be3eb14a on Aug 19, 2026

sedited commented at 7:58 AM on September 22, 2026: contributor

@vasild are you still interested in this change? This has been in need of a rebase for a while now.

sedited requested review from sr-gi on Oct 16, 2026

sedited requested review from 0xB10C on Oct 17, 2026

vasild force-pushed on Oct 19, 2026

vasild commented at 4:58 PM on October 19, 2026: contributor

b2d092efcf94e721dab4a36ce3aeefa7f9d9af5f...64f0bec95c7bdbd569bcd67638cdfce5b4cd2e89: rebase to resolve conflicts and put the stats in a new field of the existent getnettotals RPC instead of in a new RPC.

vasild renamed this:
~~rpc: provide per message stats for global traffic via new RPC 'getnetmsgstats'~~
rpc: provide per message stats for global traffic via new field in 'getnettotals'
on Oct 19, 2026

DrahtBot removed the label Needs rebase on Oct 19, 2026

DrahtBot added the label CI failed on Oct 19, 2026

DrahtBot commented at 5:34 PM on October 19, 2026: contributor

🚧 At least one of the CI tasks failed. Task i686, no IPC: https://github.com/bitcoin/bitcoin/actions/runs/39289994591/job/118187836905 LLM reason (✨ experimental): The build failed because NUM_CONNECTION_TYPES is unused in node/connection_types.h, and -Werror treats the warning as an error.

<details><summary>Hints</summary>

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

</details>

net: count traffic bytes and number of messages globally

Before this change only per-peer stats were gathered. They
are lost when the peer disconnects.

So, collect the traffic stats globally in `CConnman`, broken down by
* direction (sent or received, (2))
* network of the peer (IPv4, IPv6, Tor, I2P, CJDNS (5))
* connection type (inbound, full outbound, feeler, etc, (6))
* message type (verack, ping, etc, (36))

57e6744aae

vasild force-pushed on Oct 20, 2026

rpc: visualize global CConnman stats in getnettotals.detailed

Introduce a new field named `detailed` in the output of RPC
`getnettotals` that renders the global traffic stats from `CConnman`.

Co-authored-by: Vasil Dimov <vd@FreeBSD.org>

fe609969d7

test: add functional tests for the new getnettotals.detailed field

Co-authored-by: Vasil Dimov <vd@FreeBSD.org>

f33bcc5fd6

vasild force-pushed on Oct 20, 2026

DrahtBot removed the label CI failed on Oct 20, 2026

vasild commented at 12:01 PM on October 20, 2026: contributor

Ready for review.

ping @sr-gi, @0xB10C, @mzumsande