Per-peer memory usage

amiti · #1 · 2023-11-16T03:25:56.724Z

This is a thread to discuss the memory usage of peers.

Interesting things:

average case / expected memory usage per peer
difference in average case memory usage for block-relay-only peer vs full-relay peer
worst case memory usage for a single peer
worst case memory usage for a block-relay-only peer vs full-relay peer
how different network conditions impact the expected memory usage per peer

Memory monitoring:

I have written a patch to monitor the memory usage of different types of connections. It currently reports (1) current CNode memory usage (2) current Peer memory usage & (3) max Peer memory usage.
Status: the broad strokes are in place, but there are some fields that are not yet accounted for (details in PR description). Also, I would love review to know if the code actually does what I think it does!
Desired improvement: update the max peer memory usage to incorporate CNode memory, so it will be a better representation of the entire connection.
This spreadsheet are the results from running the patch on my node with inbounds enabled for approximately 5 days (nov 10-15). it shows a dramatic difference between the max memory usage of block-relay-only peers vs full-relay peers.

this graph shows memory usage broken down by max & current, for connections with relay enabled or not. y axis: number of bytes, x axis: different nodes.

Warnet:

Warnet can help us observe different network conditions. Here are some questions I’m curious about, where I think warnet can create/isolate behaviors we sometimes see:

When LinkingLion is on the prowl, there is high churn of inbound connections. How much memory does each short-lived connections use?
Especially with ordinals/transcriptions, we sometimes see really high transaction volume. How does that impact the expected/average memory usage of our full-relay connections?

Do you have more ideas?

0xB10C · #2 · 2023-11-16T07:05:19.389Z

Thanks for sharing this! I think the labels for max_no_relay and cur_no_relay are mixed up (max is lower than current memory usage). If I’m reading this correctly it’s just below 1 MB per full-relay peer?

Curious about this too! I’m seeing more than a 100 connections per minute to a few of my nodes. I’m wondering when the memory allocations are made. When the peer connects or during connection lifetime. I think observing memory usage over time would be interesting here too!

Anthony Towns · #3 · 2023-11-16T07:08:52.876Z · in reply to #2

The maximums in that graph counts just the Peer memory usage; the current counts both Peer and CNode

mzumsande · #4 · 2023-11-30T20:19:30.029Z

Looks great! Even though the analysis isn’t complete yet (some field not yet accounted for as you mention), I’m just trying to come up with a few high-level takeaways so far.

Dynamic memory usage per tx-relaying peers is ~600kb - 1.4MB per tx-relaying peer, a large part of it is probably the rolling bloom filter m_tx_inventory_known_filter.
Block-Relay-Only peers only take up a very small fraction of that memory (<10kb)
CNode average memory is small in comparison with Peer on average - however, I’d expect hat there would be large spikes when we send / receive a large message that we haven’t processed yet - these likely wouldn’t be seen in this data.
CNode memory considerations are mostly independent on whether the connection is block-relay-only or not.

Does this make sense? Any important takeaways so far that I am missing?