Mempool trimming regression: Rejects my own high prority 0-fee transactions #6972

Bitcoin was started like this a few days ago, after compiling latest version from git (4ee149a): bitcoind -daemon -txconfirmtarget=3 -sendfreetransactions -spendzeroconfchange=0

After running a few days, mempool trimming has begun: $ bitcoin-cli getmempoolinfo { "size": 12167, "bytes": 137785057, "usage": 297081872, "maxmempool": 300000000, "mempoolminfee": 0.00001034 }

And now it rejects my own transactions. I have a lot of old unspent coins in my wallet, which often result in high enough priority for a free transaction:

$ bitcoin-cli sendfrom "" 1[redacted] 1.05 error code: -4 error message: Error: The transaction was rejected! This might happen if some of the coins in your wallet were already spent, such as if you used a copy of wallet.dat and coins were spent in the copy but not marked as spent here.

It was clearly rejected: $ bitcoin-cli listtransactions "" 1 [ { "account": "", "address": "1[redacted]", "category": "send", "amount": -1.05000000, "vout": 0, "fee": 0.00000000, "confirmations": -1, "txid": "...", "walletconflicts": [ ], "time": 1447066461, "timereceived": 1447066461 } ]

But this can not be trusted! After a restart of bitcoind, the transaction shows up again, now with 0 confirmations. This can lead to double transmissions if the user think the first attempt actually failed: [ { "account": "", "address": "1[redacted]", "category": "send", "amount": -1.05000000, "vout": 0, "fee": 0.00000000, "confirmations": 0, "txid": "...", "walletconflicts": [ ], "time": 1447066461, "timereceived": 1447066461 } ]

Private details about the transaction available on request.

Suggestion: 0. Revert the current mempool trimming. It is dangerous.

1. Take priority into account when trimming. Keep transactions with sufficient priority to get mined before the transactions which are removed due to low fee, using estimatefee and estimatepriority as guidance. This will help prioritize non-spammy transactions as well.
2. Give own transactions infinite priority. If you are mining, this will make sure your own transactions will always be included if you leave room for free transactions.

In the BitcoinXT code, we check to see if a transaction is in mapDeltas before evicting it. We add the transaction to mapDeltas for anything that comes from the wallet, or anything that's named in the RPC prioritisetransaction. If Core doesn't do something similar, perhaps it should.

Something was messed up about the estimatepriority calculations as well. The priority of this transaction was only about 90 million, which is far below what estimatepriority now reports as required for a confirmation target of 24: $ bitcoin-cli estimatepriority 24 2551901276.470781

Interestingly higher than what estimatepriority reports for a confirmation target of 3 blocks, but I assume it doesn't actually try to send anything as a free transaction when the estimated required priority is -1? $ bitcoin-cli estimatepriority 3 -1

@sturles see #6134 for solving the problem of not defaulting to the default AllowFree threshold if the estimate is -1.

As for the original problem. It remains to be determined what the correct behavior should be. See #6959 and comments here and here, but it sounds like perhaps just improving the UI is the best answer rather than actually changing the behavior.

Perhaps what we could do though is put a few checks in the wallet code to recognize a mempool limited situation and not generate free txs or txs of too little fee in that situation. @sdaftuar any thoughts on that?

@morcos Using the priroity and suggested fee targets for the next confirmation target instead of default values when estimatepriority/estimatefee returns -1 is a good idea.

I haven't used the GUI for years, and I am not sure how those suggestions will help in this case. I have sent a transaction which got refused with -1 confirmations (assumed not sent), and then resurrected when I restarted bitcoind.

Since the current trimming code favors relaying of paid spam instead of lower paying high priority transactions (much less likely to be spam), I think only taking the fee into account when trimming is an entirely wrong approach. Keep the transactions which are most likely to get mined instead. Transactions which will be mined in the next block using the default settings should not be evicted.

@sturles Now see the latest commit in #6134. It affects both the gui and the RPC when you are using bitcoinds transaction creation logic. Now you will no longer create a tx that would be rejected by your mempool. @sdaftuar is working on a fix for trying to keep priority in mind for mempool limiting.

#6722 very deliberately disabled high-priority as a thing when mempool gets large. The "correct" behavior here is for wallet and RPC to follow along...high-priority goes away (maybe unless we have a replace-by-fee mechanism).

@TheBlueMatt That would be very unfortunate. Following the last malicious spam attacks against bitcoin, I was hoping for spam mitigation to be emphasized in future versions, not removed to promote more spam. High priority transactions are typically less spammy. I have some ideas on how to change the priority calculation slightly to promote utxo set reduction as well, but I understand this is not a goal. @morcos Does this solve #6959 as well? (Locally generated high priority transaction getting evicted after it was accepted.)

I think there is some general agreement that priority is not a metric we should be optimizing for - it significantly complicates a ton of code and sets a bad expectation from a user's point of view. In the medium term, miners will start using fees as a serious source of income and priority areas will go away (this is already happening for a few miners). We don't want users who are still using free transactions (while most have started paying fees) to be surprised when they suddenly have to pay fees again.

Worse, priority areas only help power users and users who rarely use the system. It is unclear to me why we should be prioritizing these users over all others. Though these users are, indeed, less spammy, the only incentive-compatible way to reduce spam in the Bitcoin system as it exists today is to allow fees to rise to a point where spammers are no longer willing to pay it.

On November 11, 2015 12:23:38 AM PST, sturles notifications@github.com wrote:

@TheBlueMatt That would be very unfortunate. Following the last malicious spam attacks against bitcoin, I was hoping for spam mitigation to be emphasized in future versions, not removed to promote more spam. High priority transactions are typically less spammy. I have some ideas on how to change the priority calculation slightly to promote utxo set reduction as well, but I understand this is not a goal.

@morcos Does this solve #6959 as well? (Locally generated high priority transaction getting evicted after it was accepted.)

---

Reply to this email directly or view it on GitHub: #6972 (comment)

@TheBlueMatt What is your definition of "medium term"? I have failed to find a difference in income per block for pools with a high -blockprioritysize (e.g. BitMinter which use 500k) and some Chineese pools which will include anything based on fee alone. There are a few high paying transactions, and a very long tail which is practically worthless. Since most high priority transactions pay a fee as well, just smaller fees than the spam, the difference in total block income is just a few satoshi. There is no doubt which blocks include the most spam and blocks the most normal transactions, however.

I think this complexity is something we have to live with, if we want it to be hard for a malicious user with a few bitcoins to not only block normal bitcoin transactions for a long time, but even remove them from the memory pools forever, by sending spam. And also if we want to introduce incentives to reduce the utxo set, I can't see how that will be possible without setting a priority.

In the past when someone started to DoS Bitcoin by sending a lot of spam with high fees, previous transactions using lower fees (estimated before the attack started) would still trickle through due to high priority. This won't happen any more if you remove the priority system. Instead the transactions will be erased from all memory pools. This isn't what people have learned to expect. When you send a transaction using the recommended fee, you expect it to get confirmed eventually, if not in the first block. Bitcoin is not always fast, but it used to be reliable.

I don't understand how the priority system only help power users and people who rarely use the system. Which category would e.g. someone who only use bitcoin to pay for a beer and a burger every Friday go in? Will the bar still accept bitcoins when transactions from trusted customers stop confirming eventually, like they used to? I think this change is very bad for usability and adoption.

@TheBlueMatt, my take on this is that the long term status of priority is tenuous at best because I agree unless there is a change in consensus rules its hard to consider attributes that miners aren't incentivized to prioritize. However in the meantime, as long as we're seeing significant hash power still reserve a priority space, it is useful to preserve the ability to send free high priority transactions. This has to be balanced with the complexity of achieving this goal. It was (rightfully) judged more important to safely limit the growth of the mempool than preserve this ability. However if we can, without undue additional complexity, add back in the ability to have priority earn you space in the mempool and still be comfortable in the security properties of the limiting code then we should do so.

@sturles It remains an open problem what to do if your mempool would evict your transaction due to size limiting or time based eviction. I agree the current state of affairs is not ideal, however it seems your mempools behavior is your best guess as to what other mempools are doing, and if you are being evicted from your mempool, you may very well want to spend those outputs again and broadcast a replacement transaction.

@morcos It's most certainly not an open question.

The mempool must not even know that a transaction was generated by the local wallet.

If it does then there will be privacy implications.

@pstratem I tend to agree. But on the other hand, the wallet doesn't have to regard inputs of a transaction that is no longer in the mempool as spendable.

This also messes with manual broadcast with -walletbroadcast=0 in which case the transaction may not end up in the local mempool for a while. If the wallet creates a transaction it has to regard those inputs as spent.

Priority was removed a long time ago. Closing