Estimatesmartfee is easily gameable by miners #12706

<!-- This issue tracker is only for technical issues related to Bitcoin Core. General bitcoin questions and/or support requests are best directed to the Bitcoin StackExchange at https://bitcoin.stackexchange.com. For reporting security issues, please read instructions at https://bitcoincore.org/en/contact/. If the node is "stuck" during sync or giving "block checksum mismatch" errors, please ensure your hardware is stable by running memtest and observe CPU temperature with a load-test tool such as linpack before creating an issue! -->

<!-- Describe the issue -->

Currently on testnet, estimatesmartfee 3 displays feerate: 0.01502015. This is a ridiculous value because the blocks are not full and the reason is that some of the miners for some reasons currently do not mine segwit transactions at all. Since some of these segwit transactions carry a huge fee and are nevertheless not mined, the estimatesmartfee increases the estimate to take it into account.

<!--- What behavior did you expect? -->

The estimate algorithm believes the reason the transactions are not mined are too low fees but there are apparently non-monetary reasons. I think the algorithm should also look at the transaction with the 5-10% lowest fees to check if there are lower-paying transactions that are confirmed and in this case, lower the estimate to these values since it means that such a fee is enough. Without such a check even a small cartel of the miners can significantly increase the estimates (by not confirming several percent of the high-paying transactions) and since the estimator is fairly widely used can book a profit. Even now, there are a few miners on the mainnet (e.g. bw.com) that do not mine segwit transactions at all and it's possible that there is some impact on the estimatesmartfee for 1 or 2 blocks

Of course, if estimatesmartfee looks at the bottom 5-10% of the transactions, the miners can game the estimate downwards (by confirming below market fee transactions) but at least in this case, the gaming cannot be profitable is is, in my opinion, preferable to the current system.

<!--- What was the actual behavior (provide screenshots if the issue is GUI-related)? -->

<!--- How reliably can you reproduce the issue, what are the steps to do so? -->

<!-- What version of Bitcoin Core are you using, where did you get it (website, self-compiled, etc)? -->

<!-- What type of machine are you observing the error on (OS/CPU and disk type)? -->

<!-- Any extra information that might be useful in the debugging process. -->

<!--- This is normally the contents of a `debug.log` or `config.log` file. Raw text or a link to a pastebin type site are preferred. -->

Please note that test coins don't have value, so the economic incentives by transaction fees that can be observed on mainnet are void for miners on testnet.

Of course on testnet there is no fee market and nobody (including miners) cares about the fees. I only reported an observation that appeared on testnet (likely by chance) that can be repeated on mainnet if some miners don't mine immediately some fraction of the transactions paying enough to be mined on the open market. I'm pretty sure it can be made profitable even with a not very large fraction of the hashrate involved in this scheme as long as non-neglibible fraction of the transactions blindly follows the Core fee estimate. The cost of this scheme is the opportunity cost of skipping well-paying transactions and the profit is the increase of the average fee rate after the increase in the estimates. It can be even done stealthily by not mining e.g. segwit transactions once in a while at all. The observers would rather think it is a result of misconfiguration or using a different Bitcoin implementation than a malicious move.

The reason it may work is that estmatesmartfee has a very strict 95% criterion so by delaying confirmations of only just a bit above 5% of the transactions, miners can influence the estimate.

It is hard (if not impossible) to estimate fees but the algorithm should at least be not gameable in miners favor.

In a somewhat related observation, 0.16.0 estimatesmartfee currently shows higher fees for 3 confirmations than for 2 on testnet:

bitcoin-cli -testnet estimatesmartfee 2 { "feerate": 0.00728060, "blocks": 2 } bitcoin-cli -testnet estimatesmartfee 3 { "feerate": 0.01010419, "blocks": 3 }

In a somewhat related observation, 0.16.0 estimatesmartfee currently shows higher fees for 3 confirmations than for 2 on testnet

I think this may be related to Issue [#11800](/bitcoin-bitcoin/11800/)

Given that there hasn't been activity on this issue, closing to consolidate with issue #11800.

It should be noted that this (and related issues) appear to only apply to testnet. As Marco mentioned above:

test coins don't have value, so the economic incentives by transaction fees that can be observed on mainnet are void for miners on testnet.