Increase mempool expiry time to 2 weeks

morcos commented at 7:48 pm on December 9, 2016: member

As discussed in the weekly dev meeting, I think it makes sense to increase the mempool expiry time. https://botbot.me/freenode/bitcoin-core-dev/2016-12-08/?msg=77683289&page=4

I propose 2 weeks.

3 days (the old time) was already not sufficiently short to protect against an attack that could fill the mempool with high fee rate txs that were somehow not attractive or possible to mine. A longer expiry time will reduce network traffic by less rerelay of low fee txs and will allow transactions to take advantage of weekly cycles in tx volume. By keeping the txs in the mempool, future revisions of fee estimation will be able to provide lower estimates for transactions which are low priority and can wait days or a week to be included in a block.

I’ve been running nodes with this code for many months.

Increase mempool expiry time to 2 weeks 5f0e27f1a8

gmaxwell commented at 7:59 pm on December 9, 2016: contributor

Nodes which listen frequently have old transactions blasted back at them by “helpful” parties, so anyone counting on that expiration for something already can’t really count on it.

Concept ACK.

MarcoFalke added the label TX fees and policy on Dec 9, 2016

MarcoFalke added the label Mempool on Dec 9, 2016

MarcoFalke removed the label TX fees and policy on Dec 9, 2016

dcousens approved

paveljanik commented at 9:23 am on December 10, 2016: contributor

Concept ACK

rebroad commented at 11:41 am on December 10, 2016: contributor

seems to me like a fix in the wrong place - surely the algorithm for keeping txs should be the same (or at least) similar to the algorithm used by miners for which txs to include in blocks.

gmaxwell commented at 7:40 pm on December 10, 2016: contributor

@rebroad They are the same.

rebroad commented at 2:00 am on December 12, 2016: contributor

Why expire them at all?

morcos commented at 1:36 pm on December 12, 2016: member

@rebroad I can think of 2 reasons to leave an expiry time.

If we do end up in a situation where mining policy doesn’t match our mempool policy, then at least we will eventually clear transactions that aren’t getting mined. Neither 3 days nor 2 weeks are sufficiently aggressive to resist an attack, but probably both still help in the case of accidental network misbehavior.
It might make sense in the future to separately treat our own wallet transactions that are expired after 2 weeks. They become good candidates for fee bumping or some other type of reuse. It seems to me just abandoning them is probably unsafe advice, but perhaps we’d have a feature which respends the outputs back to ourselves if the original tx is no longer desired. For now we will rebroadcast them, but hopefully this is rare.

sandakersmann commented at 10:17 pm on December 14, 2016: contributor

NACK. This new policy will make it more cumbersome for the average user to transact, and will push the network towards the toxic RBF policy.

morcos commented at 0:55 am on December 15, 2016: member

@sandakersmann Can you explain how it will make it more cumbersome for the average user to transact? Keep in mind that transactions that are expired from your mempool are automatically reaccepted when you restart your node in versions until 0.13.1 and every 20 mins in 0.13.2+.

dcousens commented at 0:56 am on December 15, 2016: contributor

Keep in mind that transactions that are expired from your mempool are automatically reaccepted when you restart

What mechanism are you referring to here out of interest?

sandakersmann commented at 1:02 am on December 15, 2016: contributor

@morcos When your transaction is stuck in limbo it is better that it is only stuck for 3 days and not 14.

morcos commented at 4:24 pm on December 15, 2016: member

@dcousens I should have said your wallet transactions. ReacceptWalletTransactions is called on startup and after #9290 RelayWalletTransactions also tries to reaccept the transactions to the mempool and runs every 20 mins.

sipa commented at 9:10 pm on December 16, 2016: member

@sandakersmann

@morcos When your transaction is stuck in limbo it is better that it is only stuck for 3 days and not 14.

When a transaction is stuck in limbo, it is usually due to too low fees. In that case, it will have been evicted from the mempool already, and the timeout doesn’t affect it at all. The reason the timeout exists is for robustness against unexpected events such as unknown new network rules, that result in apparently valid but high fee transaction not getting confirmed.

sandakersmann commented at 11:39 pm on December 16, 2016: contributor

@sipa Valid and medium/low fee transactions are some times not getting confirmed within 3 days, due to new transactions constantly paying a higher fee. The congestion is real and makes transactions unreliable.

sipa commented at 11:50 pm on December 16, 2016: member

@sandakersmann I am aware. But this PR will not affect those; they’re already evicted from the mempool for other reasons.

sandakersmann commented at 0:25 am on December 17, 2016: contributor

@sipa What reasons do you refer to? I’m talking about transactions with sufficient min relay fee and sufficient fee to stay inside the default max mempool size.

sipa commented at 0:43 am on December 17, 2016: member

@sandakersmann In that case they’re already subject to rebroadcasts.

sandakersmann commented at 0:49 am on December 17, 2016: contributor

@sipa True if you use Core as your wallet, but most users don’t.

morcos commented at 1:01 am on December 17, 2016: member

@sandakersmann This conversation has gone a bit off the rails for a PR. This is a configurable command line option. Always has been. It makes sense for us to ship it with a default that we think is best. Anybody that disagrees is welcome to change the setting on their node, it will still interoperate perfectly well with the rest of the network. As already mentioned:

There is no practical change for Core wallet users.
For wallet users in general, it is my belief that they’d rather have placed transactions make it into a block after some delay rather than not at all, and in many cases there is also no practical difference for them either because of other people rebroadcasting transactions.
Finally and most importantly, having the transactions stay consistently in the mempool will make it easier to properly track how long they are taking to be included in a block and provide more reliable fee estimates at lower fees. This provides an important feedback effect if even only a few people take care of it because it allows fee estimation to detect when those lower feerates are being included in a block relatively quickly and can cause everyone to start paying lower fees.

sandakersmann commented at 1:14 am on December 17, 2016: contributor

@morcos Fine. You guys can and will do whatever you want. Soon there will be no users of Bitcoin the way you’re going. Maybe Luke-Jr gets his wish and we have < 500 kB blocks for next Christmas.

gmaxwell commented at 2:42 am on December 18, 2016: contributor

@sandakersmann The accusatory tone you’re using is not constructive, please try a different approach in the future.

I think if you take the time to read the comments here with an open mind you’ll see that your concerns had already been answered: Third parties already frequently re-transmit other people’s transactions which degrades the effectiveness of the expiration, this cannot be prevented. Moreover, due to full replacing miners, nodes restarting, and the transactions being evicted for having fees too low in practice I’ve found that replacements of non-RBF transactions work long before the expiration. So they’ll work after this but can’t be counted on even already due to the frequent third party rebroadcasts.

sandakersmann commented at 4:54 pm on December 19, 2016: contributor

@gmaxwell So since you already have merged toxic stuff that frequently re-transmit transactions, it should not be an issue to merge this also? This crippled RBF coin will have no future.

sipa commented at 5:10 pm on December 19, 2016: member

Rebroadcast of unconfirmed wallet transactions has been in every release of Bitcoin Core (and Bitcoin, before it was renamed) ever. Please stay on topic here.

btcdrak commented at 5:27 pm on December 19, 2016: contributor

utACK

sandakersmann commented at 5:51 pm on December 19, 2016: contributor

@sipa Was mainly referring to #9290

MarcoFalke commented at 6:10 pm on December 19, 2016: member

@sandakersmann It is not helpful if you complain about code changes which happened in a completely unrelated pull. Also note that #9290 mainly solves an issue where a user created a long chain of transactions in a short time period and had some of them rejected by the mempool (due to too long chains). #9290 will make the wallet “self-heal” from that without the need to restart the node or manually push transactions to the mempool…

When reviewing, please focus on the code changes of the given pull only.

btcdrak approved

gmaxwell commented at 1:20 am on December 20, 2016: contributor

@sandakersmann Bitcoin has always rebroadcast transactions.

#9290 is about retrying mempool insertion more frequently for mempool rejected transactions, which themselves have little expiration interaction because they weren’t going mempools in the first place! … it’s mostly related to long chains of unconfirmed transactions, not low fees or such… and it’s necessary for to achieve broadcast not rebroadcast.

gmaxwell commented at 3:47 pm on January 4, 2017: contributor

ACK.

TheBlueMatt commented at 6:37 pm on January 5, 2017: member

utACK

laanwj merged this on Jan 5, 2017

laanwj closed this on Jan 5, 2017

laanwj referenced this in commit a72f76ca3d on Jan 5, 2017

bimmerdriver commented at 5:34 pm on May 24, 2017: none

I’m not a bitcoin developer or expert, but rather a regular periodic bitcoin user impacted by the recent glut of unconfirmed transactions caused by load and/or insufficient fees. My perspective as such a user is that keeping unconfirmed transactions in the mempool for two weeks is not helpful. I’ve learned the errors of my ways (and not to trust blockchain.info “dynamic” fee recommendations), but I have transactions (including double-spends) with no prospect of being confirmed stuck in the mempool. The sooner they get discarded the better. Alternatively, make two weeks the maximum and allow users to specify a shorter lifetime.

jnewbery commented at 7:14 pm on May 24, 2017: member

@bimmerdriver - users can specify the -mempoolexpiry option if they want a different expiry time for the local mempool.

I don’t think this is actually what you need though. You’re asking about your own transactions being stranded because of insufficient fee. There are a couple of ways to resolve that - Child Pays For Parent (send a transaction with a large fee which relies on your transaction) or Replace Be Fee. You should be able to find information on both of those if you search. I suggest you use https://bitcoin.stackexchange.com/ if you still need help.

bimmerdriver commented at 5:09 am on May 25, 2017: none

@jnewbery - Thanks for your reply. I guess I don’t see what the benefit to anyone is for transactions that have no hope of being confirmed cluttering up the mempool (and the respective wallet) for two weeks. If they aren’t going to get confirmed within a reasonable amount of time, they should go away. I wish I could use RBF or CPFP, but unfortunately, blockchain.info doesn’t support it. As soon as the unconfirmed transactions go away, I will move to another wallet.

sipa commented at 5:45 am on May 25, 2017: member

The reason for increasing the timeout is because there are transactions that confirm over such timeframes. There is typically less load on weekends, allowing a number of transactions from the week(s) before to still confirm. Also, the timeout generally has very little effect, as wallets can always rebroadcast to get another timeout period.

pstratem commented at 6:39 am on May 25, 2017: contributor

@bimmerdriver Transactions do not expire. There are nodes which rebroadcast transactions and have no expiration limit on mempool transactions.

You must invalidate a transaction before assuming it will not be included in a block.

codablock referenced this in commit 5d3401c7e3 on Jan 18, 2018

andvgal referenced this in commit f852f09792 on Jan 6, 2019

CryptoCentric referenced this in commit abb34f5b92 on Feb 26, 2019

MarcoFalke referenced this in commit e9fc8f6e7f on Feb 21, 2020

PastaPastaPasta referenced this in commit 63d922a0d3 on Jun 27, 2021

PastaPastaPasta referenced this in commit 71391cd4c6 on Jun 28, 2021

PastaPastaPasta referenced this in commit 21c3927cc0 on Jun 29, 2021

PastaPastaPasta referenced this in commit abb609ac06 on Jul 1, 2021

PastaPastaPasta referenced this in commit f86ee82d9b on Jul 1, 2021

PastaPastaPasta referenced this in commit 6a0627d073 on Jul 14, 2021

DrahtBot locked this on Sep 8, 2021

Increase mempool expiry time to 2 weeks #9312