I suggest to use unordered_map instead of map because it reduces the complexity. The standard map is ordered (often realized as binary search tree) and hence has a logarithmic complexity. In contrast the unordered_map has a constant complexity on average.
In most cases the ordered property is not necessary. For example in case of mapRelay I believe the map can be substituted by the unordered_map without loss of functionality. I am not sure about the magnitude of inserted inv messages but I can imagine if the network and/or the number of transactions grows this might have an impact on performance.