I have been noticing for a while that sometimes the mempool memory usage sometimes dramatically starts increasing where previously it was staying relatively minimal. This just happened on my node, and I happened to be logging some additional debug information. Just at the point that the memory usage started climbing rapidly, it was just after a block was received. The additional logging set up was to report partial messages being received (i.e. messages that take several polls of ProcessMessages() to be received). Just after the block was received, this happened from most of the connected nodes as is typical when they are helping to propagate a block through-out the network. However, one node, claiming to be a "Satoshi:0.11.2" client, started sending a series of inv messages, all 36003 bytes long each, followed 1 minute later by a constant stream of txs, all >14K each. It was during this time that the mempool memory usage jumped up, until the minfeefilter kicked in. There were also some messages being received claiming to be 439MB in size (according to the header) with no strCommand. 1 minut later, the 36003 byte long invs messages resumed, followed shortly after by the large txs. Then after a couple of minutes, another node starts exhibiting unusual behaviour, namely the 4294967295 byte messages with no strCommand, and nDataPos being zero (i.e. no bytes sent).
Will continue to try to debug what is going on, but it would appear that some additional DoS detection could be developed.
