release notes: Explicitly mention the removal of free transactions, and do not commit to removal of priority in any given release

luke-jr commented at 7:02 am on December 2, 2016: member

Priority sorting when creating block templates for mining is still in use and recommended.

fanquake added the label Docs and Output on Dec 2, 2016

TheBlueMatt commented at 7:05 am on December 2, 2016: member

NACK, it is neither in the best interest of miners to have any concept of free/priority, nor do any miners do this at scale. Supporting it is incredibly painful, and absolutely will be removed.

luke-jr commented at 7:29 am on December 2, 2016: member

On the contrary, it is well within the interests of miners to avoid spam, which priority has helped do for a long time. It is used by at least Eligius, and hopefully other altruistic/pro-Bitcoin miners. Supporting it out-of-bitcoind would be incredibly painful (likely impossible), but not so much keeping it in.

NACK removal from mining.

gmaxwell commented at 12:58 pm on December 2, 2016: contributor

It’s quite straightforward to support it outside of bitcoind (assuming you’re no talking about using it to bypass even minimum relay fees, which I think makes little sense)– and also opens up the possibility of doing things like captcha-for-priority.

morcos commented at 1:04 pm on December 2, 2016: member

NACK

This discussion has gone round and round enough. Continuing to support priority makes the codebase more complicated and is a burden on developers. It will be removed for 0.15 and we should clearly be communicating that.

Prior arguments about whether priority was still in use analyzed the data in different ways. I believe @luke-jr looked just to see if some transactions were sorted by priority and included earlier in the block as a result, and found this to be happening with what he considered some frequency, I think he said 6% in 2014. I, on the other hand, looked at what percentage of transactions were included in the block at all that would not have been on a strict feerate sort and found this to be less than 1%. This analysis is now a year old and happened before we removed the default 50k priority space. I can’t imagine it’s more than a negligible number of transactions now.

More to the point. There is no reason for the reference code base that everyone uses to support complicated logic that we don’t believe to be of general benefit just because some very small minority wants the functionality. It should be implemented with patches or helper programs that apply prioritisetransaction and the burden on creating and maintaining such functionality should be on those who want it.

luke-jr commented at 1:07 pm on December 2, 2016: member

Perhaps I am missing something, but I do not see any way at present to define regions of generated blocks to use different sorting. Part of the value in priority is that you get the first kB sorted by priority, and the rest sorted by feerate. Using two policies within the same block makes it more difficult to game.

gmaxwell commented at 7:19 pm on December 4, 2016: contributor

@luke-jr So to make that complete, you’d basically want a ‘weight-to-always-prioritize first’ setting where it would first fill X weight of the block with transactions that have had their priority altered, then continue with normal selection?

An alternative to that would be to just set the deltas so high that they stand above all of TXN and then only have your limit worth of txned deltaed in the mempool at any time. (getrawmempool will tell you what you have so far)

sipa commented at 5:11 pm on January 10, 2017: member

I think the original text is more accurate, so NACK.

luke-jr commented at 5:19 pm on January 10, 2017: member

The original text implies a future change that has no consensus, so cannot be included as-is.

sipa commented at 5:40 pm on January 10, 2017: member

@luke-jr I believe you’re alone with that opinion.

luke-jr commented at 6:07 pm on January 10, 2017: member

If it had consensus, by definition, there would be no objection, not even from me. Suggesting it has consensus is nothing short of claiming I don’t exist or am irrelevant.

It’s a harmless, isolated ~100 LOC that harms nobody and is used and desired in the real world.

dcousens commented at 0:14 am on January 11, 2017: contributor

@luke-jr I’m not arguing against the merits of priority, but to claim it only impacts 100 LOC is an understatement. As prolific as the policy is, it greatly complicates the ability to determine an appropriate, economical fee algorithmically, more-so abroad and in light wallets, than in any full node.

luke-jr commented at 3:49 am on January 11, 2017: member

@dcousens My claim was only that it is currently comprised of in Core as 100 LOC.

Anyhow, perhaps I am missing something: I am currently under the impression that light clients can’t determine an economical fee at all, and simply trust a third-party to suggest one for them. How are you proposing they figure it out, with no visibility into transactions other than their own in the first place? Why can they not just ignore priority like full nodes?

In any case, any argument along those lines is not very pertinent to the question of providing code for it. The reference codebase should absolutely not be treated as a vehicle to try to force specific agendas and policies on the network.

sipa commented at 5:21 am on January 11, 2017: member

My view is that priority today is a useless concept to the network, and in the future will only become more useless. It cannot long term function as a DoS protection mechanism without significantly being against miner’s rational interests, and we’re seeing that happen already. On that basis, I, and many other contributors, believe it is an obsolete artefact in our codebase that has no reason for being maintained.

rubensayshi commented at 6:31 am on January 11, 2017: contributor

ACK

as a mining pool being able to set priority on certain transactions is something that can be useful and is being actively used (by us).
if it would simplify things it could however be a boolean value instead of a score, since the use case as miner is as far as I can tell (and at least as far as we use it) only to bump a transaction to the top.

we’ve used it / are using it as an easy / quick way of mining a “stuck” transaction when CPFP is either not possible or just too much hassle.
apart from side channels (our own customer support, chat, forum, etc) where we use this, we also have it as an experimental feature on the chinese version of the BTC.com explorer where a user can pay by wechat to bump the priority of a TX!
we’ve seen ViaBTC experiment with a similar free feature.

CPFP would theoretically be able to cover most of these use cases, but most (if not all) wallets lacking it as a feature (it’s not an easy feature to provide / explain UI wise).

Another case where we’ve used this is due to the max ancestor / descendant limit being reduced from 100 to 25 (https://github.com/bitcoin/bitcoin/pull/6771).
Unfortunately I never knew about this limit until 0.12 was released with the 25 limit default, but we’ve seen a few occasions where a user spending his own 0conf change outputs wasn’t able to because someone else was chaining one of the other 0conf outputs. I don’t wanna bring up something off-topic, so let’s ignore the fact that this limit has given me much grief as a wallet provider, but I wanted to bring it up to highlight a case where a miner might want to be able to prioritize a TX to help out a user who is unable to use CPFP (if his wallet would even have the feature).

Tbh I realize my argument probably isn’t very strong from your PoV because in a perfect world it would be unnecesary,
but I can imagine that if this would be completely removed it will be most likely patched back in by us (and others) because it does have it’s uses, particularly because it’s very simple compared to the proper solutions to the various (fee related) problems.

when I say “we” I mean BTC.com pool*

dcousens commented at 6:37 am on January 11, 2017: contributor

@rubensayshi you can use prioritisetransaction with a high virtual fee for that use case, no priority concept needed.

rubensayshi commented at 6:38 am on January 11, 2017: contributor

@dcousens isn’t this discussion about completely removing all means to do so (in the future)?

dcousens commented at 6:40 am on January 11, 2017: contributor

@rubensayshi my understanding is that this is a discussion about the removal of priority in terms of unspent age, that is, if an unspent is “old”, it is rated more favourably as an anti-spam measure, compared to an unspent that is “new”.

See https://en.bitcoin.it/wiki/Transaction_fees#Priority_transactions

rubensayshi commented at 6:42 am on January 11, 2017: contributor

Doesn’t setting prioritisetransaction with a high value (ab)use the concept of "priority"?

And we’re discussing the wording on:

0The concept of "priority" transactions is planned to be removed in the next

for me those 2 things add up to to possibility of a miner easily forcing a TX into a block being removed.

dcousens commented at 6:44 am on January 11, 2017: contributor

@rubensayshi no, because instead you would just use “Parameter 3”:

If positive, the virtual fee to add to the actual fee paid by the transaction; if negative, the virtual fee to subtract from the actual fee paid by the transaction. No change is made to the actual fee paid by the transaction

If you wanted to “prioritise” a transaction now, in a future proof way, you would just set the priority parameter to 0 and set a high virtual fee. @rubensayshi to quote @gmaxwell above (s/delta/virtual fee):

An alternative to that would be to just set the deltas so high that they stand above all of TXN and then only have your limit worth of txned deltaed in the mempool at any time. (getrawmempool will tell you what you have so far)

rubensayshi commented at 6:51 am on January 11, 2017: contributor

okay if that an RPC command remains to set the virtual fee for a TX than I see no reason to keep the concept of priority ;)

bitkevin commented at 3:37 am on January 12, 2017: contributor

NACK. There is no difference between UTXOs, the age is useless but make code more complicated.

Priority transactions should remove in the future. For pools, there are still have rpc command that pool operator could adjust tx’s priority(like add virtual fee) in the mempool.

wizkid057 commented at 5:17 am on January 12, 2017: none

ACK.

I think the PR here appropriately describes the changes and doesn’t hide the nature of the situation behind the term “priority”. At the very least, noting what will actually be lost, which is essentially the mining of legitimate non-spam fee-less transactions, in addition to the existing lines, is important.

I do think some people fail to note that the concept of priority in general helps discourage spam transactions since the fast reuse of coins results in lower priorities. Reducing spam transactions is certainly a positive thing, and using blockprioritysize to let transactions with priority in above potential spam transactions, regardless of fee, is important. Obviously not the most ideal solution, but it is something that is certainly helpful.

As a pool operator I find it incredibly valuable to have control over which transactions are mined by the pool. Miners (and thus by extension, pools) are the ones inherently charged with protecting the blockchain from spam. It is unfortunate that many miners tend to ignore this particular issue.

Aside from just the release notes changes, I don’t see any valid reasoning for removing blockprioritysize at all, especially considering it is in use and does serve a purpose. Having more options for miners to control what they’re including in the chain is a good thing. It definitely looks like it’s pretty unobtrusive code to keep as well.

luke-jr force-pushed on Jan 15, 2017

luke-jr commented at 5:24 am on January 15, 2017: member

Rebased.

sipa commented at 5:31 am on January 15, 2017: member

Thanks for your comments, @wizkid057. I do disagree on a number of points, though:

Priority, as it exists now, is useless merely by virtue of there not being any wallets that optimize for it. That does not mean it can’t possibly return, or that policies can’t change over time, but I do believe that long term, multiple distinct actually used policies on the network will converge to. Anything else essentially requires users to solve a complex optimization problem with hard to estimate parameters. If btc-days-destroyed-per-size is to be incorprated in selection strategy (and I agree that it is possibly a useful metric can protect against spam), I believe it should be integrated into a single metric, rather than being a separately filled area.
I agree that miners are allowed to choose their own selection criteria, but I do not believe they are on their own responsible for fighting abuse. They are participants in an ecosystem, but their existence is a necessity rather than a given (the only alternative has a fixed group of people with censorship rights). By introducing them into the system, we are replacing some traditional security assumptions with economic properties. Thus, long term, we should make sure that their interests are economically aligned with the rest of the network. If abuse protection includes a btc-days-destroyed-based metric, then we should work towards having that integrated in the consensus rules.
I believe the cost of this code is way higher than you think. My impression throughout the mining refactorings, segwit, and ancestor-based mining implementation, the presence of the priority code was a constant burden that significantly increased testing and review.

I believe we will remove priority mining, so there is no point is not saying that. NACK.

release notes: Explicitly mention the removal of free transactions, and do not commit to removal of priority in any given release.

Priority sorting when creating block templates for mining is still in use and recommended.

d6bd36d71a

luke-jr force-pushed on Jan 15, 2017

luke-jr commented at 5:46 am on January 15, 2017: member

@sipa Priority is not meant to be optimised for in the first place, and its usefulness has never depended on such optimisation.

Spam filtering cannot be effectively done in consensus rules, since spam and filtering thereof is inherently an arms race. Perhaps BDD could be added as a factor in the rules, but that is a complicated process, and there is no reason for miners to act against the network’s health (and therefore against their own interests long-term) in the meantime.

My impression of the cost of priority is quite different (IMO most of those costs were from the political attempts to remove it at each step, not from any actual maintenance). In any case, the alleged costs you describe are historical, and there is no reason to expect them to apply to the current code, which is observably cleanly isolated and easily maintained.

I have refactored this PR to explicitly mention the removal of free transactions as distinct from a separate deprecation of priority and advisement that it may be removed in the future, but without a specific target. My proposal is to leave the code in place as-is so long as it isn’t a burden, and if it seems like it might be, we can consider its removal (or more likely, movement to Knots) at that time. I hope this can be accepted as a reasonable compromise.

sipa commented at 5:53 am on January 15, 2017: member

@sipa Priority is not meant to be optimised for in the first place, and its usefulness has never depended on such optimisation.

Its historical useful was in a very different setting, where neither fee estimation or priority estimation was really a requirement. Today, with actual block chain space pressure, priority mining without estimation on the wallet side would just result in a much less usable system if widely used as a selection criterion.

I’m not going to repeat the rest of the arguments.

TheBlueMatt commented at 5:53 am on January 15, 2017: member

While I think there is no question priority as a concept in Bitcoin Core’s mining and other code is going away, this doesn’t mean you can’t do it yourself. If you want to use it as an anti-spam metric, we have an API which allows you to manually prioritize or de-prioritize transactions, which could be easily automated in relatively simple Python scripts to re-implement priority (or any other metric you prefer). If you find this too burdensome, I’d encourage you to add RPC commands which give you easier access to the data you need.

On January 14, 2017 9:46:41 PM PST, Luke Dashjr notifications@github.com wrote:

@sipa Priority is not meant to be optimised for in the first place, and its usefulness has never depended on such optimisation.

Spam filtering cannot be effectively done in consensus rules, since spam and filtering thereof is inherently an arms race. Perhaps BDD could be added as a factor in the rules, but that is a complicated process, and there is no reason for miners to act against the network’s health (and therefore against their own interests long-term) in the meantime.

My impression of the cost of priority is quite different (IMO most of those costs were from the political attempts to remove it at each step, not from any actual maintenance). In any case, the alleged costs you describe are historical, and there is no reason to expect them to apply to the current code, which is observably cleanly isolated and easily maintained.

I have refactored this PR to explicitly mention the removal of free transactions as distinct from a separate deprecation of priority and advisement that it may be removed in the future, but without a specific target. My proposal is to leave the code in place as-is so long as it isn’t a burden, and if it seems like it might be, we can consider its removal (or more likely, movement to Knots) at that time. I hope this can be accepted as a reasonable compromise.

– You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/bitcoin/bitcoin/pull/9263#issuecomment-272675444

luke-jr renamed this:
~~release notes: Only free transactions are being removed, not priority.~~
release notes: Explicitly mention the removal of free transactions, and do not commit to removal of priority in any given release
on Jan 15, 2017

morcos commented at 4:59 pm on January 15, 2017: member

This rewrite after the rebase is much less objectionable. However it’s still NACK from me.

I actually think we should reword it more strongly the other way and make it clear that priority (coin age) calculations and the ability to sort by it for mining are definitely being removed completely for 0.15. @luke-jr I’ve heard you repeat the argument repeatedly over the last couple of years that the small burden of maintaining this falls on you and you don’t understand why anyone else complains. This is false. The vast majority of code that touches priority over the last couple of years has been written or maintained by me with @sdaftuar coming in next.

It’s ridiculous to repeat this same set of arguments about whether we should remove priority every release cycle and its a big waste of developer time. I think we all agreed to let priority stick around a little longer than we thought necessary in order to appease you. But the clock on that has ticked out now.

I plan on creating a PR removing all priority code in the code base as soon as 0.14 has split and I will advocate for its merging so we can all finally move on. We would be doing a disservice to any users which actually use or know about priority (if there are any) if we don’t clearly announce this to them. So please don’t make it less clear that priority is going away. It is.

btcdrak commented at 6:55 am on January 17, 2017: contributor

NACK for the reasons above. Agree with @morcos and @TheBlueMatt.

fanquake commented at 12:16 pm on January 17, 2017: member

Closing. Release notes are now being handled on the wiki: https://github.com/bitcoin-core/bitcoin-devwiki/wiki/0.14.0-Release-notes.

fanquake closed this on Jan 17, 2017

morcos commented at 12:52 pm on January 17, 2017: member

For the record, @luke-jr just made this changes unilaterally to the wiki where we are now constructing the release notes.

achow101 commented at 2:46 pm on January 17, 2017: member

This is the current text in the release notes related to the proposed changes:

Free transactions has long since ceased to function, and as such, the default for the rate limit for them (-limitfreerelay) has been set to 0 kB/minute.

Support for “priority” (coin age) transaction sorting for mining is considered deprecated in Core and may be removed in a future major version. This is not to be confused with the prioritisetransaction RPC which will remain supported for adding fee deltas to transactions.

I propose to change that text to the following (in an attempt to appease both @luke-jr and everyone else):

Free transactions have long since ceased to function, and as such, the default for the rate limit for them (-limitfreerelay) has been set to 0 kB/minute.

Support for “priority” (coin age) transaction sorting for mining is deprecated in Core and will be removed in the next major version. This is not to be confused with the prioritisetransaction RPC which will remain supported for adding fee deltas to transactions.

MarcoFalke locked this on Sep 8, 2021

release notes: Explicitly mention the removal of free transactions, and do not commit to removal of priority in any given release #9263