estimatefee returning impossible fee #9811

Not sure if this is worthy of an issue, but I noticed something in the course of building some weekend-ware: https://estimatefee.com

It's currently 49m since the last block (453914) was found, and predictably there's an increase in the mempool. Right now bitcoin-cli estimatefee 2 (on 0.13.2) is returning 0.00123063

However, if we plot the mempool by fees (blue line is 0.00123063 fee rate). We can see there's actually 2MiB of transactions that are already paying more:

<img width="1197" alt="screen shot 2017-02-20 at 10 19 03 am" src="https://cloud.githubusercontent.com/assets/9326759/23133232/0ba49c82-f756-11e6-859f-227811762504.png">

I also verified by applying a topological sort to the transactions (what older versions of core, and bitcoin unlimited do) and there is also >2MiB of transactions exceeding that fee rate.

So even if there wasn't a single new transaction created, our transaction at the estimatefee feerate will not make it in the next 2 blocks.

This is probably a known issue/limitation, so feel free to close if it's not useful.

At what point do "we" give up on automatic fee calculation? It seems like something a computer will never be able to do properly. I think we need to improve the tools that provide information to the user, to help him make as good a decision as possible when it comes to fees. I dont know. Maybe the way in which blocks are produced have to change. For example braiding/juteing to reduce variance. I really dont know. But i think trying to make automatic fee estimation work is starting to become a headache.

Maybe a simple fix in this case could be "Warning, an unusal number of minutes have passed since last block so the fee estimation may be unrealiable at this point" Or something like that.

On the flip-side, when transaction fees are going down -- core's fee estimation similarly seems to behave rather poorly. Here's another screenshot:

<img width="1149" alt="screen shot 2017-02-25 at 7 05 10 pm" src="https://cloud.githubusercontent.com/assets/9326759/23343066/e630afa2-fc2a-11e6-8ce6-dbd1eac9057f.png">

This weekend (when fee competition is low), the fee estimation is typically x2 or x3 of what is reasonable (especially with like 25). I've been comparing core's estimate fee against external ones (e.g. blockcypher ) and the fee estimations seem universally more accurate.

I believe cases where this is possible are nearly universally resolved by the Fee-Rate upgrades in .15, is this closeable @RHavar?

It still can and does happen in 0.15, but I'm not particularly sure the issue is very actionable.

One hack you could do, is that if calling estimatefee N get the fee rate at the place of (N*4e6 + someSortOfLienency) weight of the mempool (sorting by by fee/weight) and pick which ever is lower.

Basically that way if there's a sudden surge in required fees (because of no blocks, or lots of high feee paying transactions), it never suggests something that is outright impossible. It's a total hack, so I understand if it's not wanted (and I'll close the issue). @morcos thoughts?

You are talking about estimatefee and not estimatesmartfee ? There is no reason to use estimatefee any more and it is deprecated. I don't think this would be an issue with estimatesmartfee, but if it is, please let me know.