Smart (floating) fees for the client #3335

gavinandresen commented at 3:10 AM on December 1, 2013: contributor

This is a much more conservative pull request than my previous smart-fees request.

It only tries to solve the problem: what fee (or priority) is needed to get into a block right now?

It adds two new RPC calls: estimatefee and estimatepriority.

Test plan:

Run qa/rpc-tests/estimatefee.sh : EXPECT: runs successfully (it is a -regtest integration test).

Restart bitcoind with this patch. Then run:

bitcoin-cli -rpcwait estimatefee : EXPECT: -1 (not enough data to estimate) bitcoin-cli -rpcwait estimatepriority : EXPECT: -1 (not enough data)

Leave it running an hour or two, and repeat. EXPECT: reasonable fee/priority estimate.

Test different medians; EXPECT: estimatefee 0.1 is less than or equal to estimatefee 0.5. Same for priority.

Test saving/restoring mempool/estimation state: shutdown bitcoind, and restart. EXPECT: same estimates as before.

Test Bitcoin-Qt fee/priority estimation:

Run with a wallet that has a combination of big/old and small/new unspent transaction inputs. Turn on coin control. Test send: EXPECT: coin control values Do the Right Thing (fee turns red if not enough fee is included, preiority turns red if not enough priority to qualify, etc).

ttys2 commented at 6:19 AM on December 1, 2013: none

I have tested this patch.

A watch script polling every 30 seconds shows the values to be stable and sane.
Values persist through a clean node shutdown.
Values persist through an unexpected node shutdown.
Passing invalid queries to either command results in a proper error.

On startup: $ ./bitcoin-cli estimatepriority -1.00000000

$ ./bitcoin-cli estimatefee -1.00000000

After 20 minutes: $ ./bitcoin-cli estimatepriority -1.00000000

$ ./bitcoin-cli estimatefee 0.00044247

After 60 minutes: $ ./bitcoin-cli estimatepriority 320748687.31726366

$ ./bitcoin-cli estimatefee 0.00044247

$ ./bitcoin-cli estimatefee 0.5 0.00044247

$ ./bitcoin-cli estimatefee 0.1 0.00028655

$ ./bitcoin-cli estimatefee 0.01 0.00014947

in src/qt/coincontroldialog.cpp:None in 524c448173 outdated

 525 | -
 526 | -        // Min Fee
 527 | -        int64_t nMinFee = GetMinFee(txDummy, nBytes, AllowFree(dPriority), GMF_SEND);
 528 | -
 529 | -        nPayFee = max(nFee, nMinFee);
 530 | +        if (!AllowFree(txDummy, dPriority, nBytes, GMF_SEND))

cozz commented at 10:42 AM on December 1, 2013:

There is an else-statement missing:

else
    nPayFee = nTransactionFee;

Currently, if AllowFree is true and nTransactionFee > 0, coin control shows zero fee. But this is wrong. Must show nTransactionFee.

gavinandresen commented at 12:04 AM on December 2, 2013:

@cozz: nice catch! I'll fix.

gavinandresen commented at 1:42 AM on December 2, 2013: contributor

Fixed the bug @cozz found, and re-ordered includes to make the windows cross-compiler happy.

in src/qt/coincontroldialog.cpp:None in f60553d066 outdated

 529 | -        nPayFee = max(nFee, nMinFee);
 530 | +        if (nTransactionFee > 0)
 531 | +        {
 532 | +            // If user set nTransactionFee, pay at least that;
 533 | +            // more if transaction is > 1,000 bytes:
 534 | +            nPayFee = max(nTransactionFee, nTransactionFee*nBytes/1000);

cozz commented at 3:08 AM on December 2, 2013:

Sorry to bother, but now you missed the case (nTransactionFee > 0 && nTransactionFee < 10000 && !AllowFree)

Example: Set nTransactionFee to 1 satoshi. Select output < medium. Send Transaction with no CENT-outputs. You have to pay 10000 satoshi for this transaction, but coin control says 1 satoshi.

gavinandresen commented at 3:18 AM on December 2, 2013:

Bah, brain fart on my part. The coin control logic and wallet's send logic REALLY need to be refactored to share the same code....

mikehearn commented at 9:59 AM on December 3, 2013: contributor

I've been running this for a while and recording the fees. It seems that they oscillate a bit. Columns are:

height fee fee0.1 fee0.9

272718 0.00051813 0.00038759 0.00387596 272719 0.00051813 0.00038759 0.00386100 272720 0.00053097 0.00038759 0.00386100 272721 0.00053097 0.00038759 0.00386100 272722 0.00070671 0.00038910 0.00386100 272723 0.00070671 0.00038910 0.00386100 272724 0.00053619 0.00038759 0.00386100 272725 0.00052356 0.00038759 0.00386100 272726 0.00052356 0.00038759 0.00386100 272727 0.00052356 0.00038759 0.00386100 272728 0.00052356 0.00038759 0.00386100 272729 0.00052356 0.00038759 0.00307692 272730 0.00052356 0.00038759 0.00307692 272731 0.00052356 0.00038759 0.00386100 272732 0.00052356 0.00038759 0.00386100 272733 0.00052356 0.00038759 0.00386100 272734 0.00052356 0.00038759 0.00386100 272735 0.00052356 0.00038759 0.00386100 272736 0.00077519 0.00038759 0.00386100 272737 0.00077220 0.00038759 0.00386100 272738 0.00077220 0.00038759 0.00341296 272739 0.00076923 0.00038759 0.00307692

So the current calculated median fee is either ~55 cents or ~80 cents but nothing in between, which is a bit odd. The lowest fees on the other hand are stable at about 40 cents.

The lowest fee seen (estimatefee 0) is 0.00009569 or about 10 cents.

I'm not really sure how to interpret the output of this command though. There's no time dimension. If I paid the lowest fee possible, how long would it take for my tx to confirm? The documentation for estimatefee doesn't help me figure that out.

gavinandresen commented at 10:54 AM on December 3, 2013: contributor

RE: median jumping around: fees are extremely quantized right now; 0.0005 and 0.0001 are hard-coded in lots of wallets. Anybody using Bitcoin-Qt or bitcoind will pay one of those two values per-number-of-bytes-in-your-transaction (assuming transaction is < 1,000 bytes); since transaction size depends on number of inputs and outputs, and since transactions < 1,000 bytes are a handful of inputs, one or two outputs, fees end up in very discrete bands.

RE: time:

If you want to take a crack at a better estimatefee, please be my guest. txmempool.cpp keeps track of time/blocknumber/fee/priority into the memory pool, and is told when transactions exit the memory pool because they were included in a block. Given that information, somebody who knows a lot more statistics than I do might be able to create a CMinerPolicyEstimator that gives "should be confirmed in the next N blocks with X% likelihood."

mikehearn commented at 10:59 AM on December 3, 2013: contributor

Hmm, I suspect my statistics isn't any better than yours :) Will have a think about what is required to calculate that.

If all wallets adopt a policy of trying to out compete a given percentage of transactions, would that risk an upwards spiral?

gavinandresen commented at 11:11 AM on December 3, 2013: contributor

RE: upwards spiral: you'll note that I picked the median for the default, exactly to avoid an upward spiral.

Once estimation is in place, then we need to give users some control over the fee versus confirmation time tradeoff, which should give the right dynamic-- if users compete and force out low-fee transactions, so be it.

gavinandresen commented at 11:14 AM on December 3, 2013: contributor

... I should have written: some MORE control over the confirmation time versus fee tradeoff. Users can set -paytxfee (or set in the GUI) if they want to pay more or less.

mikehearn commented at 11:20 AM on December 3, 2013: contributor

Yeah. I'm wondering what happens if any large user decides to pick something higher than the median. Like let's say a major exchange decides they want faster confirmations so they start using "estimatefee 0.8" for all their transactions. So that puts upwards pressure on the median that all the other nodes follow.

Diapolo commented at 7:50 AM on December 5, 2013: none

Do we need to update some strings in the client about fees (init.cpp / GUI) etc.?

laanwj commented at 8:43 AM on December 5, 2013: member

@Diapolo if you find any, let us know. Most important here for now is to get the fee estimation logic right, so let's not get distracted.

Save/restore memory pool 2a98d704e5

Floating fees for the client

Estimate what fee or priority is needed to get a transaction
accepted into a block (unless user has set a fee using
-paytxfee or the GUI).

Includes a safety net: always pay at least what a 0.8.5 client
would pay. The plan is for future releases to eliminate that
check, and let fees float up and down based on competition for
block space.

Adds two new RPC methods to expose the estimates:
estimatefee and estimatepriority.

I expect future releases might improve the fee estimation code;
it is self-contained in the CMinerPolicyEstimator class, so
"patches welcome."

This is a much more conservative of the pull request I
submitted a couple months ago.

656f8a7f2c

Track fees being paid to get confirmed within N blocks. 0fb46bf4e6

bardiharborow commented at 4:28 AM on January 18, 2014: contributor

Any update on this?

ABISprotocol commented at 6:05 PM on January 22, 2014: none

How does this affect fees in multisignature transactions?

sirk390 commented at 6:49 PM on January 30, 2014: none

Using this method, it seems to me it will be difficult for fees to go down. Users can only guess the minimum fee and risk having their transaction unconfirmed. Most users will probably not risk that and have bitcoins stuck for days and will put at least the median. Also, couldn't miners include a large number of high fee transactions in their own blocks and broadcast the transactions just before publishing the block? This would increase the median fee at almost no cost for the miner.

Did you consider creating a block version 3 and add minfee+minpriority to the block as for block_height? Every transaction in the block would then need at least the minfee or minpriority specified or the block would be rejected(using a similar stragegy as BIP34). This is to make sure that miner don't publish a higher minfee than that they really accept. It would make it easy to compute the minimum fee depending on the how fast a confirmation is needed.

carnesen commented at 7:03 PM on February 25, 2014: contributor

It doesn't look like smart fees aren't going to make it into 0.9 : http://newsbtc.com/2014/02/24/bitcoin-transaction-fees-likely-drop-0-9-update/ #3305

Merge pull request #30 from mikehearn/estimator2

Track fees being paid to get confirmed within N blocks.

07eeec5c12

BitcoinPullTester commented at 10:31 PM on March 18, 2014: none

Automatic sanity-testing: FAILED MERGE, see http://jenkins.bluematt.me/pull-tester/07eeec5c12fbb22b4e53a63fdf0186fa2339f06b for test log.

This pull does not merge cleanly onto current master This test script verifies pulls every time they are updated. It, however, dies sometimes and fails to test properly. If you are waiting on a test, please check timestamps to verify that the test.log is moving at http://jenkins.bluematt.me/pull-tester/current/ Contact BlueMatt on freenode if something looks broken.

gavinandresen commented at 4:56 PM on March 31, 2014: contributor

Closing; this will be rolled out as a couple of separate pull requests.

gavinandresen closed this on Mar 31, 2014

DrahtBot locked this on Sep 8, 2021