[WIP] RPC: Add Method To Retrieve HTTP Server Metrics #14202

This is a work in progress, all comments and review are welcome and appreciated.

TL;DR: We add a new network RPC call, getnetworkrpcinfo, that returns metrics about JSON RPC HTTP server.

INTRODUCTION

As part of constructing Service Level Agreements (SLAs) around the uptime of Bitcoin nodes, it is useful to measure such things as requests processed per second or average request processing time. While the getnetworkinfo RPC call provides a lot information and many useful metrics about the P2P layer of Bitcoin, it does not provide any metrics about the JSON RPC layer. Since many services interact with Bitcoin almost entirely through its JSON RPC interface, it would be useful to collect and provide metrics about this interface.

OVERVIEW OF CHANGES

This PR contains two major changes: Collecting HTTP server metrics and Adding a new RPC method.

Collect HTTP Server Metrics (httpserver.h, httpserver.cpp)

This change is the first part of adding a new RPC call to report stats from the HTTP server. It modifies the HTTP server functionality to collect and report the gathered statistics.

httpserver.h:

• Add private member variable receivedAtTimeMicros to HTTPRequest to record the time at which the request was started.
• Add data-type, RPCServerStats, that holds all gathered HTTP server stats.
• Add function, GetRPCServerStats, that returns the current RPC server stats via the provider pointer.

httpserver.cpp:

• Add static global g_rpcserverstats to hold HTTP server stats across all requests.
• Add static global g_rpcserverstats_mutex to serialize writes to g_rpcserverstats.
• Modify several functions to collect the RPC server stats.
• Implement GetRPCServerStats.

Add getnetworkrpcinfo RPC Method (rpc/net.cpp)

Add a new RPC command, getnetworkrpcinfo, that returns an object containing various state information about the HTTP server providing a JSON RPC interface. This information can be used to derive information such as requests per second and average request processing time.

The information is provided in a raw form leaving the clients to derive such useful metrics, this is intentional and allows this RPC call to avoid issues around floating point accuracy and floating point error accumulation. Instead, these issues are left to the consumers of the response object to handle in whatever fashion they like.

Concept ACK. Though, wouldn't it be easier to record these metrics outside of Bitcoin Core? I.e. in the "middle" between the rpc endpoint and rpc consumer. The outside approach seems more flexible to me, since it could collect the metrics per-call or overall or per day ...

@MarcoFalke Thank you for the quick feedback!

So, in the configuration I have in mind here we have a set of nodes sitting behind a load-balancer. The desire is to gather info about request volume and latency from the nodes themselves as part of a health check, and to potentially use this information to scale the cluster up or down. Nodes themselves are ephemeral and the entire cluster may be redeployed multiple times per day.

AWS (as far as I know) does not provide a way to query comparable information in realtime from an ELB/NLB/ALB, but instead allows you to export logs for later analysis. So getting this information in real-time from the load balancer isn't possible.

The remaining option is to introduce an additional HTTP proxy between the HTTP server provided by Bitcoin and the incoming requests for the sole purpose of collecting this information, but this seems wasteful since Bitcoin itself is a server and could provide the required information with lower latency and higher quality. Since comparable information is already provided about the P2P services provided by Bitcoin (getnettotals), it seems to make sense to provide similar information about the HTTP services.

As an aside, I think a better name for this is probably getrpctotals or something similar. Could possibly be merged with getnettotals as well.

<!--e57a25ab6845829454e8d69fc972939a-->Note to reviewers: This pull request conflicts with the following ones:

• #13152 ([rpc] Add getnodeaddresses RPC command by chris-belcher)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

I think this is not very flexible. We could just send the events begin/end to somewhere so these can be used as a source to some analytics and monitoring tool.

@promag This works and could be a much lower maintenance burden. I assume you mean using boost::signals2 to create request start/end signals or possibly pushing zeromq notifications? Any recommendations on approach would be appreciated

I assume you mean using boost::signals2 to create request start/end signals or possibly pushing zeromq notifications? @etscrivner actually I was thinking in something simpler. If we log req_id rpc_method start_time ... and req_id rpc_method stop_time ... or just rpc_method duration ... then those metrics could be extracted externally, even in realtime. Or am I missing something?

Should there be a way to reset the counters?

Concept ACK

Though, wouldn't it be easier to record these metrics outside of Bitcoin Core? I.e. in the "middle" between the rpc endpoint and rpc consumer. The outside approach seems more flexible to me, since it could collect the metrics per-call or overall or per day ...

Agree that these current statistics are easy enough to collect from the outside. Though I suppose there are some things that are not easy to collect from the outside using a proxy; say, if you want to record CPU/IO usage per call.

OTOH these kind of changes run a significant risk of creating an "inner platform effect", where the scope of a simple daemon extends to be a fully-fledged OS stack including instrumentation, monitoring. All things that need to be documented, maintained, supported.

I think this is not very flexible. We could just send the events begin/end to somewhere so these can be used as a source to some analytics and monitoring tool.

Right—the preferable way would be to accommodate OS tooling for instrumentation, so that sysadmins, developers, and such can use the existing frameworks.

I remember @eklitzke was talking about something like that, adding hooks for "user-defined static trace probes", which could be monitored using bcc systemtap dtrace etc.

@laanwj Thank you for the review.

I've begun looking into the usage of static trace probes that would allow for these measurements to happen. You make a very good point about the potential for this to create an "inner platform effect". Based on what you and @promag have said, it seems like that is a more flexible approach here.