The maxtxfee check is applied at the very final stage of creating a transaction, which with a large amount of inputs and/or a low maxtxfee setting will cause coinselection to frequently propose a solution that will just be rejected, even though there are many other possible selections which would not exceed the maxtxfee setting.
The simplest fix I believe would be make each "stage" of the coin selection aware of the maxtxfee setting, and go to the next when it can't satisfy it.
e.g. if the BnB algorithm can find a selection that requires 0.01 BTC of fees, but the maxtxfee is set to 0.005, it shouldn't bother suggesting it.
And then there could be a final fallback algorithm that is an extremely simple greedy one that picks the largest inputs, where if that fails then there is no hope and an error given.