paytxfee behavior changed without warning #7633

(updated with new information) Calculation of the transaction fee when "paytxfee" is set has changed, without announcement or documentation.

Per @sipa : In earlier versions however the size of a transaction was rounded up to a multiple of 1000, as that is how the mining code used to sort transaction years ago. In 0.12, it was finally changed to the accurate formula that works at an accuracy per byte. If the paytxfee value was always exactly what you saw as resulting fee, that must mean all your transactions were 1 kB in size.

In bitcoin.conf, I had

paytxfee=0.0001

In 0.11, that meant that BTC 0.0001 would be added to every outbound tx as a fee.

in 0.12, it means every tx will have added to it a fee of BTC 0.0001 per kilobyte

This is not documented in the release notes. As a result of this, I've got 3 days worth of spend transactions caught in unconfirmed limbo, having gone out with fees like 0.00002260 instead of 0.0001.

It has had that meaning for as long as I can remember.

Well that's pretty darned surprising:

$ bitcoin-cli listtransactions '*' 100|egrep '(confirmat|fee)'
    "fee": -0.00010000,
    "confirmations": 857,
    "fee": -0.00010000,
    "confirmations": 850,
    "fee": -0.00010000,
    "confirmations": 844,
    "fee": -0.00010000,
    "confirmations": 843,
    "fee": -0.00010000,
    "confirmations": 826,
    "fee": -0.00010000,
    "confirmations": 825,
    "fee": -0.00010000,
    "confirmations": 797,
    "fee": -0.00010000,
    "confirmations": 794,
    "fee": -0.00010000,
    "confirmations": 778,
    "fee": -0.00010000,
    "confirmations": 772,
    "fee": -0.00010000,
    "confirmations": 772,
    "fee": -0.00010000,
    "confirmations": 772,
    "fee": -0.00010000,
    "confirmations": 761,
    "fee": -0.00010000,
    "confirmations": 765,
    "fee": -0.00010000,
    "confirmations": 759,
    "fee": -0.00010000,
    "confirmations": 735,
    "fee": -0.00010000,
    "confirmations": 730,
    "confirmations": 699,
    "confirmations": 652,
    "fee": -0.00003740,      ### **Upgrade to 0.12**
    "confirmations": 639,
    "fee": -0.00003740,
    "confirmations": 616,
    "confirmations": 505,
    "fee": -0.00002260,
    "confirmations": 0,
    "confirmations": 440,
    "confirmations": 453,
    "fee": -0.00003740,
    "confirmations": 0,
    "confirmations": 436,
    "fee": -0.00002260,
    "confirmations": 0,
    "confirmations": 433,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002250,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 129,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002250,
    "confirmations": 0,
    "fee": -0.00002250,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "confirmations": 350,
    "fee": -0.00003740,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002250,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "confirmations": 322,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00005210,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00003730,
    "confirmations": 0,
    "fee": -0.00003730,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002250,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "confirmations": 9,
$ 

Ah, the transaction size may have been rounded up to a multiple of kilobytes in the past.

I don't think so. It always sent fees exactly equal to the paytxfee config variable

I can assure you that the paytxfee configuration parameter has been a value in BTC/kilobyte at least since 0.3.15, and has always been advertized as such.

In earlier versions however the size of a transaction was rounded up to a multiple of 1000, as that is how the mining code used to sort transaction years ago. In 0.12, it was finally changed to the accurate formula that works at an accuracy per byte. If the paytxfee value was always exactly what you saw as resulting fee, that must mean all your transactions were 1 kB in size.

I do agree this should have been mentioned in the release notes.

It's a breaking change. It should have been announced and documented. Very frustrating.

Put another way, it's not a breaking change. It's evidence of a missing set of tests.

Also, people need to be notified of this. I doubt I'm the only person affected, with transactions stuck in limbo hoping that the average block size drops a bit so that these get included. There's a 16,000 unconfirmed tx backlog right now.

tweeted: https://twitter.com/mikegogulski/status/705169770518286336

The description in issue is incorrect. Could you revise it based on the discussion here, since you've now promoted it?

Done!

same issue with me #7634

reddit thread: https://www.reddit.com/r/Bitcoin/comments/48oz2k/watch_out_upgrading_to_bitcoin_core_012_an/

@sipa is correct; paytxfee was always in BTC per kilobyte but bitcoin core 0.12 will no longer round up to the next kilobyte. And unfortunately, the release notes only implicitly mention that this changed: "pay a predefined fee rate by setting -paytxfee=<n>". (https://github.com/bitcoin/bitcoin/blob/v0.12.0/doc/release-notes.md#wallet-transaction-fees)

OK. Thank you @sipa and @MarcoFalke for clarifying the nature of this issue.

If either of you or @gmaxwell is unsatisfied with the current issue description, would you please suggest how to improve it?

Is there any way to communicate this even though the release notes are impossible to change now? (E.g. add a section to the 0.12.1 release notes?)

I agree that this change of behavior should have been documented in the release notes, as it will result in - on average - in paying less fee if -paytxfee is used. This is good as rounding is wasteful and unnecessary, but I can see it result in confusion if it happens unexpectedly.

• This is the commit in which Satoshi made paytxfee per kB, it has indeed been the case for a long time: 461764cbbea5965ebbd248f221876743d7a9ccd4 (2010!)

I cannot find the commit/pull request that removed the rounding. Anyone?

Put another way, it's not a breaking change. It's evidence of a missing set of tests.

I agree that the wallet fee logic doesn't have very good testing. This is a community effort, you're welcome to contribute more and better tests.

Is there any way to communicate this even though the release notes are impossible to change now? (E.g. add a section to the 0.12.1 release notes?)

What would help a lot, I think, would be to have documentation for the fee system - e.g. a user manual that describes how it is supposed to work, what the expectations should be for the fee computation. There are a lot of knobs, and it is far from clear from the --help output only how these are supposed to interact.

This causes both developers and users to make mistakes in that regard.

@laanwj > I cannot find the commit/pull request that removed the rounding. Anyone?

Is this it?

commit c6cb21d17ab8097b6a425d37e48c955fbb0e9f0c Author: Gavin Andresen gavinandresen@gmail.com Date: Thu Apr 10 14:14:18 2014 -0400

Edit: incidentally I only yesterday noticed that I had settxfee=0.00025 in my bitcoin.conf instead of paytxfee=0.00025. I was wondering why every time I restarted bitcoind I had to manually use the RPC settxfee command again. That's just user error on my part, but the similarity of the config setting and the RPC command names probably contributed.

Or ecc7c82?

I cannot find the commit/pull request that removed the rounding. Anyone?

In case you mean "removed the rounding in 0.12", it is a couple of pull request: #6649 (not merged), #6708 (not merged), #7103 (merged), #7096 (merged but not related to this issue, as it was rebased onto 7103)

What would help a lot, I think, would be to have documentation for the fee system

I tried to do exactly this in the release notes. Unfortunately I did not receive any feedback on the content, but I am happy to extend this. Maybe the website is a better place to curate such documentation?

In case you mean "removed the rounding in 0.12", it is a couple of pull request: #6649 (not merged), #6708 (not merged), #7103 (merged), #7096 (merged but not related to this issue, as it was rebased onto 7103)

Timeline:

• 461764cbbea5965ebbd248f221876743d7a9ccd4: Added the rounding (back in 2010)
• c6cb21d17ab8097b6a425d37e48c955fbb0e9f0c #3959: Removed the rounding (back in 2014). This was already in 0.10.
• ed3e5e468c5b7973d71efe51c90c3a301d570e27 #5200: Added an option fPayAtLeastCustomFee which has, for <1kB transactions, the effect of rounding up. This was also in 0.10.
• a7751824ce8a7e1cc5511794516fe674bc3eaa3c #7103: Default fPayAtLeastCustomFee to false. This made it into 0.12.

So all in all, rounding was already disabled in 0.10. It makes sense for this not to be mentioned in the release notes for 0.12. However what seems to have changed multiple times is the behavior for small transactions, due to fPayAtLeastCustomFee. This should have been mentioned.

I tried to do exactly this in the release notes. Unfortunately I did not receive any feedback on the content, but I am happy to extend this. Maybe the website is a better place to curate such documentation?

I don't really mind where, could just be some file in doc/ but having a document about the fee computation, anywhere, would be great.

Reverted these changes on the 0.11.x branch, so that they're not inadvertently imported there:

• d49f724 - Revert "[wallet] Add rpc tests to verify fee calculations"
• b6f5904 - Revert "Default fPayAtLeastCustomFee to false"

I agree that the wallet fee logic doesn't have very good testing.

I take this back, actually we did have to update the tests for this, see #7103 (a7751824ce8a7e1cc5511794516fe674bc3eaa3c). So it isn't a testing problem, just a documentation one.

Maybe a more permanent solution to the whole fee problem would be to simplify the transaction creation into preparesendfrom (because accounts are the only "off-chain" solution working today)/preparesendto that returns the transaction so that the user can see how much the total cost is going to be, then submittransaction that actually sends the transaction. All in easy to read JSON instead of raw hex. Hiding the transaction fees is confusing users and developers!

@tinspin Well, you can already do that... with createrawtransaction + fundrawtransaction for preparesendto, and signrawtransaction + sendrawtransaction for submittransaction...

Yes but the raw stuff is a bit, how can I put this, offputting. Specially when you are dealing with real money.

Edit: ok fundrawtransaction really helps alot... and this did to:

http://bitcoin.stackexchange.com/questions/44152/is-there-a-way-to-know-the-bitcoin-network-transaction-fee-before-actually-submi/44157

Even better:

http://bitcoin.stackexchange.com/questions/43532/transaction-fees-and-transactions-with-bitcoin-cli-wallet-address-balance-negat

But I'm surprised you can't know how large a transaction will be in bytes in advance?!

You can't control the network fee directly (as it is per byte, and you don't know the size beforehand)

You need to be able to put a fixed fee on a transaction in advance if the system is supposed to be usable?!

The exact size of a transaction can only be determined after it has been signed

Ok, I get it!

So it has to be createrawtransaction + fundrawtransaction + signrawtransaction for preparesendto, and sendrawtransaction for submittransaction... but how can I use raw stuff but associated with an account like sendfrom?

http://bitcoin.stackexchange.com/questions/24516/bitcoind-multisig-and-the-default-account

Ok, nm, I have to build my own account stuff or use sendfrom and suffer from the fee problem.

Or I hack my bitcoind to either take a fixed fee on sendfrom or I add a sendrawtransactionfrom method...

Edit; I tried looking at the code and it seems sendfrom uses a completely different implementation than sendrawtransaction?

Edit 2: #7729 but labels do not keep track of balances which is what we need accounts for.

Dam. Is there a way to with out reverting back to v11 to set a fixed fee amount ?

There never ever was an option for an absolute fee amount. What 0.11 and before had, was fee per kilobyte with the size rounded up to at least one kilobyte.

Relying on that being an absolute fee is just as wrong as assuming the currently behaviour results in an absolute fee, just less frequently visible.

If you need to predict the total fee, use the raw transaction API to inspect it before signing and sending.

Hmm that's just not an option right now, I have an upstream system written to work with what ever version of core we had 4 years ago. It deducts a set fee before sending the transaction and expects that to be the fee. I need a way to set a default fee, with out re-writing the upstream system to play whit transactions and its breaking a live system now by setting the fees too low... If I remember correctly back then there was a version of core when the fee was actually a fixed value.

I'm telling that is not the case. It was pay per kilobyte, rounded up to kilobytes. Your upstream system makes incorrect assumptions, fix them.

Yeah well, however it was doing. re writing the upstream system is just not a feasible option right now. Will have to push it back to v11 for the moment.

@abelgo If you just want to pay a fixed fee, regardless of the size of the transaction, probably the only way to do this is to use the raw transaction interface. (Though, for obvious reasons, not taking the size of the transaction into account when calculating the fees, is not advised)

When I get the chance I will look as some other apis. Ideally you want to give your end user the option of deciding what fee they want to pay. But you have the difficulty of not being able to provide any certainty to your end user as you cant calculate that with out first building the transaction first. The other issue is dealing with the average end user. Some are not going to accept that there 1c transaction is going to cost more than there $1000 transaction because its ... "bigger" What you really want in terms of a API is to provide a simple, Send XXXX amount with fee of YYYY. An end users wants to know how much something will cost them before they do it.

An API to send amount X with fee Y makes no sense at all, as you can't predict the size of the transaction. What matters to the network is the fee/byte of the transaction. If the transaction to send X ends up being unexpectedly large, a fixed fee Y may make it very slow to confirm or unacceptable to the network entirely.

What does make sense is ask the user whether to want to send quickly or slowly, then build a candidate transaction with the estimated feerate for that speed, and then let the user accept the resulting fee or not before sending. And that's exactly what the raw transaction API offers you.

Yeah I get that this is part of the nature of the design of the system. This problem is while it makes no sense to "the network" and is understandable to a developer. Its still something a novice end users wants and requests. The average user on my site wouldn't know the difference between a KB and a kb, and they don't know how many "blocks" they want there transaction confirmed in. They just want to know what it costs. This is one of the many disconnects that stops bitcoin becoming more mainstream. An API that thinks like the end user can help that. But yeah I get it.

This looks like a really major problem, I have transactions from my local exchange have remained unconfirmed for nearly a week. The number of unconfirmed transactions is climbing at an alarming rate: https://blockchain.info/unconfirmed-transactions (I just heard it peaked at 75K, then down to 28K yesterday, now 7K though, so perhaps things will pan out ok)

Some are not going to accept that there 1c transaction is going to cost more than there $1000 transaction because its ... "bigger" [...]

That's just how the system works: fees do not depend on amounts sent but on tx size. It may be counter-intuitive for people used to legacy systems, but it is actually an advantage in Bitcoin and we should simply educate users about it.

The average user on my site wouldn't know the difference between a KB and a kb, and they don't know how many "blocks" they want there transaction confirmed in. They just want to know what it costs.

You can present them only with the total fee before signing as suggested by sipa. Regarding "how many blocks they want to wait" maybe you can explain them that a block takes approximately 10 minutes.

This looks like a really major problem, I have transactions from my local exchange have remained unconfirmed for nearly a week.

You should try higher fees, maybe consider using opt-in RBF. But this is not for technical support. I suggest IRC, maybe the bitcoin-discuss mailing list.

Regarding "how many blocks they want to wait" maybe you can explain them that a block takes approximately 10 minutes.

-txconfirmtarget=<n> allows them to do that, and on the API level per individual tx. It's really just a doco issue, not a "major problem" as albego alleges. Once the UX creator is aware, he can let (or not) the user decide as they see fit.

Just got caught with my pants down. Created a transaction in a pinch; Could the behavior be checked against the estimate fee and stop the TX if the fee smaller than it should ? (Maybe adding a --force flag.)

Closing this now. I think improved our process in the last two years to do a better job at documenting such changes in the release notes. Hopefully this will not happen again.