qa: p2p_invalid_messages needs improvement #14814

After a4eaaa6ac53606a1533b56050af77961d8c27dc7 p2p_invalid_messages isn't really checking anymore what it was supposed to check, so that should be fixed in some way.

The memory usage shouldn't increase, but it is (at least temporarily) even after the messages are flushed. So the check was effectively removed in a4eaaa6 by raising the threshold.

Maybe some of valgrinds tools could help to debug this issue?

Acknowledging #15400 is still unsolved, I've done some testing on comparing the memory footprint for p2p_invalid_messages on Ubuntu 18.04 vs macOS.

Checking the size of the transport buffer and rss memory after each msg_at_size shows that after the second message is sent, on macOS, the rss memory increases dramatically. Additional there are leftover bytes in the transport output buffer. tracemalloc shows that a large memory block is allocated in the call self._buffer.extend(data) in /asyncio/selector_events.py:882 where the output buffer is extended because not everything was able to be sent.

============= Before Sending Message =============
Message [#1](/bitcoin-bitcoin/1/), memory footprint: 77088
Current output buffer size 1735312

============= After Sending Message =============
Message [#1](/bitcoin-bitcoin/1/), memory footprint: 83544
Current output buffer size 2609568

[ TraceMalloc Output ]
/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py:882: size=2548 KiB, count=1, average=2548 KiB
/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py:1420: size=736 B, count=4, average=184 B
/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py:883: size=496 B, count=1, average=496 B
/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py:305: size=496 B, count=1, average=496 B
/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/events.py:88: size=496 B, count=1, average=496 B
/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py:903: size=488 B, count=1, average=488 B

The increase in memory from this step (~13%) accounts for most (~99%) of the overall memory increase in this test. I.e. this step is the reason for the need for https://github.com/bitcoin/bitcoin/commit/a4eaaa6ac53606a1533b56050af77961d8c27dc7

In Ubuntu, the buffer is always emptied after each send and there is a minimal spike in memory footprint.

============= Before Sending Message =============
Message [#1](/bitcoin-bitcoin/1/), memory footprint: 51532
Current output buffer size 0

============= After Sending Message =============
Message [#1](/bitcoin-bitcoin/1/), memory footprint: 52884
Current output buffer size 0

Any thoughts on approaches to solving this would be appreciated. It seems to mainly be an initial overhead penalty in macOS as the memory footprint plateaus for the remaining 78 messages. So perhaps it is a macOS specific quirk?

Fixed in fac942ca57dce6cfa5655a3ac8664d6a051bc01f