Bitcoin is returning higher fees for 36 block window than 2 block window (on testnet)

karelbilek commented at 4:15 pm on November 30, 2017: contributor

In testnet, right now, estimatesmartfee is returning higher estimates for 36 block window than 2 block window. The results:

  0estimatesmartfee 2 ->
  1{
  2  "feerate": 0.01254483,
  3  "blocks": 2
  4}
  5
  6estimatesmartfee 36 ->
  7{
  8  "feerate": 0.03863968,
  9  "blocks": 36
 10}
 11
 12
 13estimaterawfee 2 ->
 14{
 15  "short": {
 16    "decay": 0.962,
 17    "scale": 1,
 18    "fail": {
 19      "startrange": 490954,
 20      "endrange": 1e+99,
 21      "withintarget": 35,
 22      "totalconfirmed": 35.41,
 23      "inmempool": 165,
 24      "leftmempool": 0
 25    },
 26    "errors": [
 27      "Insufficient data or no feerate found which meets threshold"
 28    ]
 29  },
 30  "medium": {
 31    "feerate": 0.01254483,
 32    "decay": 0.9952,
 33    "scale": 2,
 34    "pass": {
 35      "startrange": 626596,
 36      "endrange": 1e+99,
 37      "withintarget": 38,
 38      "totalconfirmed": 38.96,
 39      "inmempool": 1,
 40      "leftmempool": 0
 41    },
 42    "fail": {
 43      "startrange": 596758,
 44      "endrange": 626596,
 45      "withintarget": 15.9,
 46      "totalconfirmed": 50.78,
 47      "inmempool": 14,
 48      "leftmempool": 0
 49    }
 50  },
 51  "long": {
 52    "decay": 0.99931,
 53    "scale": 24,
 54    "fail": {
 55      "startrange": 799713,
 56      "endrange": 1e+99,
 57      "withintarget": 138.78,
 58      "totalconfirmed": 147.56,
 59      "inmempool": 1,
 60      "leftmempool": 0
 61    },
 62    "errors": [
 63      "Insufficient data or no feerate found which meets threshold"
 64    ]
 65  }
 66}
 67
 68estimaterawfee 36 ->
 69{
 70  "medium": {
 71    "feerate": 0.01254483,
 72    "decay": 0.9952,
 73    "scale": 2,
 74    "pass": {
 75      "startrange": 690822,
 76      "endrange": 1e+99,
 77      "withintarget": 37.33,
 78      "totalconfirmed": 38.19,
 79      "inmempool": 1,
 80      "leftmempool": 0
 81    },
 82    "fail": {
 83      "startrange": 596758,
 84      "endrange": 690822,
 85      "withintarget": 47.22,
 86      "totalconfirmed": 51.99,
 87      "inmempool": 12,
 88      "leftmempool": 0
 89    }
 90  },
 91  "long": {
 92    "decay": 0.99931,
 93    "scale": 24,
 94    "fail": {
 95      "startrange": 799713,
 96      "endrange": 1e+99,
 97      "withintarget": 139.47,
 98      "totalconfirmed": 147.66,
 99      "inmempool": 1,
100      "leftmempool": 0
101    },
102    "errors": [
103      "Insufficient data or no feerate found which meets threshold"
104    ]
105  }
106}

I am also attaching fee_estimates.dat

fee_estimates.dat.zip

Of course testnet is always “weird”, but 36 block estimate being higher than 2 block estimate doesn’t make sense, even on testnet.

The version of bitcoind is v0.15.0.0

karelbilek renamed this:
~~Bitcoind is returning higher fees for 36 block window than 2 block window~~
Bitcoin is returning higher fees for 36 block window than 2 block window (on testnet)
on Nov 30, 2017

laanwj added the label TX fees and policy on Nov 30, 2017

laanwj commented at 6:38 pm on November 30, 2017: member

Of course testnet is always “weird”, but 36 block estimate being higher than 2 block estimate doesn’t make sense, even on testnet.

Yes, that seems wrong.

morcos commented at 3:42 pm on December 5, 2017: member

Question about the fee_estimates file you attached. Did you quit bitcoind before grabbing it, or is that the file that was in place while it was running?

karelbilek commented at 4:35 pm on December 5, 2017: contributor

Oh. I don’t remember at this point, but I probably forgot that it’s written only on turn-off and I copied from a live node, so it’s useless. Sorry

On Dec 5, 2017 4:43 PM, “Alex Morcos” notifications@github.com wrote:

Question about the fee_estimates file you attached. Did you quit bitcoind before grabbing it, or is that the file that was in place while it was running?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bitcoin/bitcoin/issues/11800#issuecomment-349343598, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGZ8WloqN4hL17wYMpG400k6jN49bx_ks5s9WSggaJpZM4Qwztn .

morcos commented at 6:29 pm on December 5, 2017: member

That’s ok, I think I have an idea for what the issue is, and perhaps a way to fix it.

morcos commented at 4:38 pm on December 6, 2017: member

OK, so I believe I understand the bug. estimatesmartfee takes the max of three different sub-estimates:

60% threshold at half the target
85% threshold at the target
95% threshold at double the target

An estimate for a higher target should be monotonically decreasing in each of those 3 sub-estimates. So therefore the final estimates should also be monotonically decreasing.

However, there is the possibility that one of the sub-estimates is unable to return an estimate because there is no fee rate that is high enough to meet the threshold. The algorithm was designed to just return the max of the other sub-estimates in this case with the reasoning being that it is better to give some estimate rather than give up and revert to the fallback fee or have a potentially computationally expensive search for the lowest target which is able to return all sub-estimates. The downside of this is that if a higher target can return that sub-estimate, it might actually be higher than either of the other sub-estimates at the lower target and cause a final estimate which is actually greater for the higher target.

Although it’s not ideal, I think this falls in the category of “won’t fix”. What is really happening in this edge case is that the estimate for the lower target is perhaps not as high as it should be. But in analyzing a proper fix, I think that time would be better spent either doing a wholesale improvement to fee estimation, or working on better RBF or CPFP tools.

I believe that given current fee markets this should be very unlikely on mainnet, and that if conditions were to arise that made this more likely, we probably need a better fee estimation algorithm anyway. Also the raw data is accessible through estimaterawfee which could allow anyone to make their own more complicated combination of sub-estimates if they really care.

ryanofsky commented at 5:07 pm on December 6, 2017: member

It might be good to update the existing code comment to mention the edge case, since it’s easy to read it and think that monoticity will always be preserved:

https://github.com/bitcoin/bitcoin/blob/497d0e014cc79d46531d570e74e4aeae72db602d/src/policy/fees.cpp#L860-L868

Bitcoin is returning higher fees for 36 block window than 2 block window (on testnet) #11800