EstimateMedianVal returns higher fee for higher confTarget

sdaftuar commented at 8:16 pm on December 19, 2020: member

Running on recent master, commit f1dbf92ff0475a01d20170ea422c1d086acbbc57

If I run:

0test/functional/feature_fee_estimation.py --randomseed 108574360997204915

I get:

 02020-12-19T20:14:06.175000Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test_bcwexp78
 12020-12-19T20:14:07.738000Z TestFramework (INFO): This test is time consuming, please be patient
 22020-12-19T20:14:07.738000Z TestFramework (INFO): Splitting inputs so we can generate tx's
 32020-12-19T20:14:10.870000Z TestFramework (INFO): Finished splitting
 42020-12-19T20:14:12.104000Z TestFramework (INFO): Will output estimates for 1/2/3/6/15/25 blocks
 52020-12-19T20:14:12.104000Z TestFramework (INFO): Creating transactions and mining them with a block size that can't keep up
 62020-12-19T20:14:15.879000Z TestFramework (INFO): Creating transactions and mining them at a block size that is just big enough
 72020-12-19T20:14:21.104000Z TestFramework (ERROR): Assertion failed
 8Traceback (most recent call last):
 9  File "/home/sdaftuar/ccl-bitcoin/2020-12-latest-master/test/functional/test_framework/test_framework.py", line 126, in main
10    self.run_test()
11  File "test/functional/feature_fee_estimation.py", line 256, in run_test
12    check_estimates(self.nodes[1], self.fees_per_kb)
13  File "test/functional/feature_fee_estimation.py", line 140, in check_estimates
14    check_smart_estimates(node, fees_seen)
15  File "test/functional/feature_fee_estimation.py", line 130, in check_smart_estimates
16    % (feerate, last_feerate))
17AssertionError: Estimated fee (0.000565) larger than last fee (0.000479) for lower number of confirms

sdaftuar added the label Bug on Dec 19, 2020

maflcko commented at 6:09 am on December 20, 2020: member

Thanks for finding the random seed that triggers this reliably

maflcko added the label Tests on Dec 20, 2020

bitcoin deleted a comment on Dec 20, 2020

maflcko removed the label Tests on Dec 21, 2020

maflcko added the label TX fees and policy on Dec 21, 2020

maflcko commented at 8:33 am on December 21, 2020: member

With --tracerpc shows that this is really an issue with estimatesmartfee, not the test:

0...
1-4469-> estimatesmartfee [1]
2<-4469- [0.001963] {"feerate": "0.00067579", "blocks": 2}
3-4470-> estimatesmartfee [2]
4<-4470- [0.002432] {"feerate": "0.00067579", "blocks": 2}
5-4471-> estimatesmartfee [3]
6<-4471- [0.002226] {"feerate": "0.00047895", "blocks": 3}
7-4472-> estimatesmartfee [4]
8<-4472- [0.002212] {"feerate": "0.00056521", "blocks": 4}
9...

maflcko commented at 9:31 am on December 21, 2020: member

The incorrect return values (for target 3 and 4) are calculated in this line (short horizon): https://github.com/bitcoin/bitcoin/blob/e9efb64a0772777f2ce88c491101550b931a21d6/src/policy/fees.cpp#L736

the values are:

confTarget=3 successThreshold=0.85 estimate=47894.7 nBestSeenHeight=230
confTarget=4 successThreshold=0.85 estimate=56521 nBestSeenHeight=230

With corresponding debug-log line:

FeeEst: 3 > 85% decay 0.96200: feerate: 47894.7 from (40774.3 - 66417.1) 85.17% 84.9/(99.7 0 mem 0.0 out) Fail: (38832.7 - 40774.3) 74.91% 26.8/(35.7 0 mem 0.0 out)
FeeEst: 4 > 85% decay 0.96200: feerate: 56521 from (47201.4 - 66417.1) 85.47% 75.9/(88.8 0 mem 0.0 out) Fail: (38832.7 - 47201.4) 82.16% 38.3/(46.6 0 mem 0.0 out)

maflcko renamed this:
~~QA: Occasional failure in feature_fee_estimation.py~~
EstimateMedianVal returns higher fee for higher confTarget
on Dec 21, 2020

maflcko commented at 11:14 am on December 21, 2020: member

Changed title

maflcko commented at 3:08 pm on January 27, 2021: member

Related #11800?

practicalswift commented at 9:06 am on February 12, 2021: contributor

Unfortunately this larger than life^H^H^Hast fee issue in feature_fee_estimation.py keeps popping up.

From https://cirrus-ci.com/task/4756253495787520:

0AssertionError: Estimated fee (0.000400) larger than last fee (0.000338) for lower number of confirms

ajtowns commented at 1:16 pm on February 12, 2021: contributor

Fix in #21161 I think. I think what’s going on is that the bucket ranges weren’t consistent between confirmation targets, because newBucketRange = true is only set when the threshold is met, and buckets are extended otherwise. Not quite sure what the arrangement needs to be for that to result in a higher target giving a higher fee though.

morcos commented at 3:55 pm on February 12, 2021: contributor

Related #11800?

I think its an unrelated different bug

ismaelsadeeq commented at 1:24 pm on July 6, 2023: member

I try recreating this issue on master HEAD bc4f6b13feb29146b7e10e86f93dc7f6fb6937f2
with the same command test/functional/feature_fee_estimation.py --randomseed 108574360997204915

The tests passed.

maflcko commented at 1:37 pm on July 6, 2023: member

@ismaelsadeeq Can you try with the commit specified in OP as well? If it fails, can you try bisecting?

mzumsande commented at 8:37 pm on July 6, 2023: contributor

@ismaelsadeeq Can you try with the commit specified in OP as well? If it fails, can you try bisecting?

Maybe not necessary: There are newer crashes reported in #23165 (which seems to be the same problem). The seed 7288975409214300634 from https://api.cirrus-ci.com/v1/task/6110951150190592/logs/ci.log leads to a crash on current master c325f0fbae2d6472a78be733d499783f8b69d6b8 (the test passes with #21161 rebased on master).

glozow closed this on Nov 2, 2023

Julio-Rats referenced this in commit 2e9454a633 on Jan 26, 2024

PastaPastaPasta referenced this in commit bdddcf3710 on Oct 24, 2024

PastaPastaPasta referenced this in commit 4cb58b5df2 on Oct 24, 2024

PastaPastaPasta referenced this in commit e58d0dde9e on Oct 24, 2024

PastaPastaPasta referenced this in commit a50efe91f8 on Oct 24, 2024

PastaPastaPasta referenced this in commit 95a8d8cfdc on Oct 24, 2024

bitcoin locked this on Nov 1, 2024

EstimateMedianVal returns higher fee for higher confTarget #20725