Opening as draft to facilitate discussions, this has some issues.
This special-case was introduced in #18044 to avoid hindering relay of witness transactions in adversarial situations where most connections on the network were still using txid-based relay. This is not a concern anymore. Since it is expensive to detect, get rid of the special case.
This may however cause issues for orphan resolution in adversarial situations, since fetching the parent is done using its txid. Greg Sanders pointed out to me on IRC this could be an issue for package relay. I’m opening here as draft to facilitate discussion.