The documentation for -maxtxfee seems to imply that it will abort a transaction if the transaction’s fee is larger than the specified amount. The actual behavior is to just use a fee equal to maxtxfee when the calculated fee is larger.
-maxtxfee= Maximum total fees (in BTC) to use in a single wallet transaction or raw transaction; setting this too low may abort large transactions (default: 0.10)
In a specific case I observed, a large transaction created with sendtransaction and paytxfee 0.0012 ended up being broadcast with a fee of 15 sat/byte rather than aborted, which seems unintuitive.
However, looking at https://github.com/bitcoin/bitcoin/blob/master/src/wallet/wallet.cpp#L2747 in CWallet::GetMinimumFee this seems intended, and as far as I can tell, the only situation where a transaction will be aborted is if if the fee is below minrelaytxfee, rather than paytxfee.
Is this really the intended behavior? I feel like it would make far more sense that, if paytxfee is set explicitly, the transaction should be aborted if maxtxfee causes the calculated fee to drop below that value.
If not, I would argue that at the very least, the documentation for this setting should refer to the minrelaytxfee threshold specifically.