cli: added -rpcmaxresponsesize to cap HTTP response size #35420

Summary

This is a follow-up to #34342 (comment). This change adds -rpcmaxresponsesize to cap the total response size in bitcoin-cli.

In this change, the MAX_SIZE is set to uint64_t MAX_SIZE = 0x02000000 or 32 MB and can be set to -rpcmaxresponsesize=0 to be unbounded

Users might want to use this for several reasons, such as,

• Avoiding unbounded RAM usage when operating on low-resource machines (Raspberry Pi 2-4GB).
• Wanting to fail fast when having bounded resource usage in automation, or CI.
• Avoiding processing the response from large legitimate responses, such as getrawmempool true, listsinceblock on a busy wallet.

<!--e57a25ab6845829454e8d69fc972939a-->

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

<!--006a51241073e994b41acfe9ec718e94-->

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35420.

<!--021abf342d371248e50ceaed478a90ca-->

Reviews

See the guideline for information on the review process.

If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

<!--174a7506f384e20aa4161008e828411d-->

Conflicts

No conflicts as of last run.

<!--5faf32d7da4f0f540f40219e4f7537a3-->

There is a usecase for the user to lower this amount. The user can also set -rpcmaxresponsesize to set the size to be unbounded.

Instead of claiming that there is a usecase, why not simply state the usecase?

Also, the second sentence is wrong?

Concept NACK. If RPC responses are able to produce unwieldly large responses, we should document and improve the RPC to let users consume it without running into memory issues.

If there's a good use case, I'd be open to hear it and change my view.

Thank you both for the review!

Instead of claiming that there is a usecase, why not simply state the usecase?

Makes sense not to be vague, I added some use cases. e.g. “automation/CI fail-fast, constrained hardware like a Pi”

Also, the second sentence is wrong?

Sorry about that, I've updated it now!

If RPC responses are able to produce unwieldly large responses, we should document and improve the RPC to let users consume it without running into memory issues.

I agree RPC responses that can grow very large should be documented and improved so users can consume them without memory issues. This isn't meant to replace that, and I do think that users would want to be able to lower the max accepted response size, such as in automation/CI where a user would prefer to fail fast on misconfiguration instead of buffering an unexpectedly large HTTP response. This can be especially useful on machines with a small amount of RAM, such as a Raspberry Pi 2-4GB where an unexpectedly large response can slow or stop the machine.

Also, on taking a look at this I think it makes more sense to keep the default response size as unbounded as it currently is and allow the user to set -rpcmaxresponsesize if they do want to restrict it.

we should document and improve the RPC to let users consume it without running into memory issues.

Yeah, this seems the wrong fix. If it was possible for a user to run into this, there should be docs, so that the user knows how to work around this. If the user wanted to set a memory limit on a process, they can already do so without this option.

Given the lack of conceptual buy-in, I don't think this will get review in favour of this change soon, so I'm closing this again.

Given the lack of conceptual buy-in, I don't think this will get review in favour of this change soon, so I'm closing this again.

No problem, if someone else thinks this is worth pursuing, feel free to pick it up!