New estimatesmartfee seems wrong #10490

I'm running master, and estimatesmartfee 1008 is reporting 16 sat/byte.

$ bitcoin-cli estimatesmartfee 1008
{
  "feerate": 0.00016025,
  "blocks": 1008
}

However, yesterday it was returning 50 sat/byte ... and in the elapsed time there's not been any real low fee transactions being confirmed (I know, since I have have several higher fees ones stuck in the mempool for days) .

My guess is that a bunch of transactions got evicted due to the mempool overfilling (I've set my mempool to 100MB) and then the fee estimation thinks it's confirmed.

 bitcoin-cli getmempoolinfo
{
  "size": 29142,
  "bytes": 34126503,
  "usage": 78304032,
  "maxmempool": 100000000,
  "mempoolminfee": 0.00015944
}

Note how the mempoolminfee is 15.9 satoshis/byte, just a tiny bit less than the estimated fee required

Tagging on to this, the estimatesmartfee seems rather unresponsive to a drop in fee market prices. I think it's been like 5+ hours, since there was a spike in the required fees, and estimatesmartfee is still suggesting insane fee rates (while for hours ~120 sat/byte transactions have been getting confirmed).

Here's a bit of a visualization:

<img width="1237" alt="screen shot 2017-06-02 at 11 14 37 pm" src="https://cloud.githubusercontent.com/assets/9326759/26750487/945eb272-47e9-11e7-88e2-2908ca0b1489.png">

While 0.14.1 is returning saner values (350 sat/byte).

--

Also the "conservative" argument for estimatesmartfee doesn't appear to do anything. I have not been able to observe a singe case of it actually changing the returned feerate, which seems odd. @morcos ^ 😄

Just confirming this seems to be a persisting problem:

$ bitcoin-cli estimatesmartfee 1008
{
  "feerate": 0.00021508,
  "blocks": 1008
}

with

$ bitcoin-cli getmempoolinfo
{
  "size": 47677,
  "bytes": 37042308,
  "usage": 99999696,
  "maxmempool": 100000000,
  "mempoolminfee": 0.00021454
}

Meanwhile I've had dozens of transactions at 25 sat/byte that have never confirmed. For reference:

$ bitcoin-cli estimaterawfee 1008 0.95 2
{
  "feerate": 0.00010721,
  "decay": 0.99931,
  "scale": 24,
  "pass": {
    "startrange": 9906,
    "endrange": 11467,
    "withintarget": 196.64,
    "totalconfirmed": 199.65,
    "inmempool": 1,
    "leftmempool": 0
  },
  "fail": {
    "startrange": 0,
    "endrange": 9906,
    "withintarget": 110.93,
    "totalconfirmed": 115.64,
    "inmempool": 0,
    "leftmempool": 0
  }
}

(not really sure why the feerate in estimaterawfee differs though..)

@RHavar You seem to be reporting a few different issues here, so let me know if I don't cover your concerns.

Definitely transactions being evicted from the mempool will not cause the estimator to think they were confirmed. In fact it's an improvement in the new code that they will now count as a failure for any confirmation targets smaller than the number of blocks for which they were outstanding.

What I think you are seeing is that estimatesmartfee 1008 would actually return something less than the mempool min fee so in that event it always returns the mempool min fee. (this is why estimaterawfee is retuning a lower number)

The long term estimates are definitely a bit more experimental, and if the mempool isn't big enough for many transactions to stick around for 1008 blocks, this will make it difficult to return reasonable estimates for that time frame. I'm not really sure of an easy solution to make it much better.

As for the concern about short term estimates being more responsive. It's expected that it'll take several hours to respond to a drop in market prices. This is much more responsive than the prior code. However what I think happened on June 3rd is that just before the drop there was a spike, so the old code didn't respond to the spike and hence temporarily returned slightly lower estimates than the new code, but several hours after the time you reported, I imagine you would have found the new estimates much lower than the old (I can see the new estimates continued to drop but I don't have the old estimates handy).

Thanks @morcos that makes sense. Although:

What I think you are seeing is that estimatesmartfee 1008 would actually return something less than the mempool min fee so in that event it always returns the mempool min fee.

Is the part I'm struggling with. Anecdotally I have dozens of transactions that are paying 25 sat/byte (that I've been continually rebroadcasting for weeks) that aren't getting confirmed. So it seems a bit strange to me that the fee estimate thinks that 10.7 sat/byte is sufficient.

I guess one possibility is my transactions are not getting confirmed because they're a bit big (e.g. cdc70781c95fefc5599aa723d630fdb151c56576b64f6f94dac485d94e28fbbd ) when miners might not have enough space in the end of their block for low fee paying transaction. I can experiment with creating some smaller sized transactions

estimatesmartfee 1008 returns 65 sat/B for me and thats not because of mempool min fee.

It could be that your smaller mempool is causing your estimator to just not get good data that far out.

It's possible that it would be an improvement to remember the highest mempool min fee seen over the last week (or maybe 2 weeks) and never return anything less than that.

It could be that your smaller mempool is causing your estimator to just not get good data that far out.

I guess that's it then. I'm just using a smaller mempool as it gets a bit too slow for me the dump it for analysis when it's larger. If any thing though, I would've assumed that having a smaller mempool would cause the fee estimate to be more aggressive (since transactions get rejected that otherwise might confirm). But if yours is returning 65 sat/byte, that sounds pretty reasonable