'estimatefee 1' is currently failing #9106

When I run estimatefee 1 right now, I get -1 as the reply, meaning that it isn't possible to get a 95% chance of inclusion in the next block no matter what fee I include, but based on the collected mempool stats that isn't true.

I traced the TxConfirmStats::EstimateMedianVal() call to see what was happening. It went like this:

In order to get "enough transaction data points in this range of buckets", it first grouped buckets 97 through 75 inclusive, found that their average confirm probability was 94.45% (below the 95% threshold), and so hit the break and stopped looking further.

It turns out that if it had looked at bucket 74 it would have had enough data points on its own, and would have had a probability of 95.51%, and so we could have returned a value of around 0.01 BTC.

So I'm seeing a 94.45% chance of confirmation in the next block if the fee is 0.01156269 or more per kB, and a 95.51% chance if the fee is in the range 0.01051153 - 0.01156269 per kB.

It appears that the code makes the assumption that higher fees always result in a higher probability, but that isn't actually the case.

Wouldn't it be better to start at the cheapest bucket and work upwards, stopping at the first one that has a high enough probability, rather than stopping as soon as we find an expensive bucket with too low a probability? Otherwise it is possible for a small miner to sabotage fee estimation by targeting just one bucket. If he has 5% of the hash rate and refuses to confirm transactions in a particular expensive (0.01 BTC per kB) bucket he will ensure that the search stops at that bucket for all estimatefee 1 calls. He can create enough such transactions to make the bucket statistically significant for less than 1 BTC per day which causing almost 'unlimited' damage to the usefulness of estimatefee 1.

Also, isn't 95% too high of a threshold when we see almost that many empty or nearly empty blocks? If I lower the threshold to 90% I see a much more reasonable result, of around 0.0006 BTC per kB for next-block confirmation. In his original email Alex wrote:

Since even the highest fee transactions are confirmed within the first block only 90-93% of the time, I decided to use 80% as my cutoff.

That seems like a reasonable argument. Why was the number later changed from 85% to 95%?

@dooglus I originally played around with slightly "smarter" fee algorithms that started at the most expensive bucket but looked for the point of no return so to speak where the confirmation percentage dropped below the threshold never to return. Every permutation I tried seemed susceptible to not being ideal in some particular circumstance or gameable, so it seemed better to err on the side of simpler.

As for starting from the cheapest bucket, we can't do that because then the gaming you posited could potentially happen in reverse causing the fee estimates of all targets above some minimum to become useless if a miner was indiscriminately including things in that low fee range. For instance perhaps these are paid for out of band or were being confirmed by priority or some other mechanism..

The change from 85% to 95% was motivated by a desire to respond more quickly to changes in market conditions. It's quite typical for there to be popular buckets that get tons of data points. If for instance 97% of transactions in the 40 sat/byte bucket are being confirmed within 2 blocks and all of a sudden the mempool fills up and none get confirmed. It could take many hours for the confirmation rate to drop below 85% especially because people will adjust quicker and stop giving new failing data points at that fee rate.

In general the fee estimation algorithm in Core is meant to be conservative. It's the fee you should put on your transaction if you really want to be very sure your transaction will confirm in that target. You can always see how much money you would save by selecting a longer target and realizing that you are likely to be confirmed sometime within that range, not necessarily at the end of it. I would recommend never ever using 1 as a target.

All that said, it is an art more than a science, and I welcome interest in people trying to improve it.

The change from 85% to 95% was motivated by a desire to respond more quickly to changes in market conditions

How about if there were a 2nd, optional parameter to the estimate*fee calls giving the percentage. It could default to 95, to give the same behavior as the current code by default, but I could estimatefee 1 90 to see how much fee I need to include to have a 90% chance of getting into the next block.

Would just a pull request likely be merged?

The purpose of estimatesmartfee was to be able to add additional parameters like that.

I think the only question is how to add that and not lock it in as some sort of positional argument that makes it harder to add other parameters to estimatesmartfee.

I still wouldn't use it with a target of 1 though.

Just saw #8811, I think once that is merged, then I'd be in favor of such a PR

I think we should probably disable estimate*fee 1 entirely...targeting the next block is bound to fail/be unreliable.

Appears closeable given Fee-Rate improvements in .15 -- @dooglus?

estimatefee
DEPRECATED. Please use estimatesmartfee for more intelligent estimates.