Rethinking Transaction Rebroadcasting #36403

Current Behavior

Rebroadcasting is handled entirely by loaded wallets, there is no rebroadcast behavior within the node. With the default of -walletbroadcast=1, a wallet will resubmit transactions into the node's mempool upon loading, and forcibly rebroadcast transactions at a uniformly random time between 12 and 24 hours since the last rebroadcast.

Transactions eligible to be rebroadcast are those known to the wallet that the wallet believes are unconfirmed (transactions confirmed in a pending blockConnected will still be considered eligible). On loading, all such unconfirmed transactions will be rebroadcast. On the timer, the transaction must also have a received time that is older than 5 minutes since the last updatedBlockTip notification.

On wallet loading, transactions that are successfully added to the node's mempool are not added to the unbroadcast set and no broadcast is initiated. On the timer, all transactions that are either already in the mempool or were just added to the mempool are rebroadcast unconditionally; they are added to the unbroadcast set and InitiateTxBroadcastToAll is called to begin broadcast to all tx relay peers.

Why Rebroadcast

Rebroadcasting serves 2 purposes for a wallet:

1. Reminding the network of transactions that may have fallen out of node mempools so that these transactions can be mined
2. Placing transactions into the node's mempool allows the wallet to utilize unconfirmed transactions so that users are not surprised by things that suggest they have less money than they have.
   • Unconfirmed change can be spent, but the transaction must be in the node's mempool
   • Change outputs of inactive (unconfirmed, not abandoned, and not in mempool) transactions are not included in the usual balances, while the spends of the inputs are, resulting in balances that are lower than expected. #33671 helps to address this, but the nonmempool balance does not indicate the amount returned by change as there are double counting issues, so it ultimately understates the balance. Once in the node's mempool, change is counted again.

Issues

Privacy Leak

Since the wallet unconditionally broadcasts transactions, a connected peer will learn about the initial broadcast, and then 12 to 24 hours later, will receive the transaction again from that node. Every 12 to 24 hours, it receives the rebroadcast. Since rebroadcasting is not part of typical node behavior. This indicates that the node doing these rebroadcasts is likely to be a party involved in that transaction, either sending or receiving (#3828).

Private Broadcast Circumvention

The wallet does not use the private broadcast feature at all. All transactions (re)broadcast by the wallet itself are to all tx relay peers and -privatebroadcast is ignored. As such, transactions initially sent via private broadcast are rebroadcast without private broadcast, which reveals the originating IP, as well as receiving IPs.

Doesn't Submit Packages

The wallet does not currently attempt to do any form of package submission to make use of opportunistic package relay that the node may do. While rebroadcasting does submit transactions in order from oldest first to ensure that parents are accepted before children, if a parent is unable to make it into the node's mempool on its own, that parent will not be broadcast, and subsequently, neither will the child. This applies to both new transactions and to rebroadcasts.

Potential Improvements

• Wallets should submit packages to node's mempool, probably limited to 1p1c so that they can be relayed
• Rebroadcast could be performed only on transactions that are newly added to the mempool, rather than unconditionally rebroadcasting every unconfirmed transaction that meets the rebroadcast criteria.
• The node could have a rebroadcast pool which can rebroadcast transactions that are not relevant to loaded wallets, thereby increasing the anonymity set of rebroadcasts, and enabling other nodes to potentially rebroadcast transactions for us. See #21061 for the last attempt at this.

Current PRs

These are currently open PRs that are touching something in rebroadcasting. This is not a list of things to review, and this is not a tracking issue for PRs to be reviewed. Rather, this list is to serve as a collection point of all PRs attempting to do something to rebroadcasting because I am having a hard time keeping track of everything.

PRs Related to Private Broadcast

• #34533
• #36330

Settings For Controlling Rebroadcast

• #36380

Changing What Gets Rebroadcast

• #36398

Thanks for the summary!

This indicates that the node doing these rebroadcasts is likely to be a party involved in that transaction, either sending or receiving

While much of the summary focuses on the case where the wallet is the sender, I think there is an asymmetry: In both the "Privacy Leak" and "Private Broadcast Circumvention" situations, the case where the wallet is the receiver is much worse because

1. The sender (attacker) chooses the fee and can make confirmation within 24h unlikely / rebroadcast likely
2. The sender chooses the timing: a wallet is affected indefinitely, not just in the timeframe until a tx is confirmed.

As such, transactions initially sent via private broadcast are rebroadcast without private broadcast, which reveals the originating IP, as well as receiving IPs.

Same here: The issue is not so much the private broadcast tx itself (which will most of the time confirm before rebroadcast), but potential new transactions sending to the involved addresses, maybe at a later time when the user previously running on tor is now running that same wallet on clearnet.

Thanks for this writeup.

One potential improvement that I would like to advocate for is something similar to #30471.

The rebroadcasting logic should be separated from the wallet into its own module, and the wallet would be a consumer of the rebroadcast module alongside other consumers like sendrawtransaction callers.

Rebroadcasting until your transaction gets mined is an important feature, and makes sending from the wallet more reliable. Other wallets/users should also be able to benefit from this, instead of having to create a daemon or other retry mechanism on their own.

This entire module could be optional, so that users who don't want any potential privacy leaks could just turn it off.

Improving private broadcast dovetails nicely with this. Others have suggested it should be separated into its own module as well. If we tied private broadcast to rebroadcasting it substantially mitigates the privacy leak concerns.

I had an offline discussion about this with @davidgumberg and others at the localhost office. The conclusion was that the mempool expiry timer should be significantly extended or dropped (see current discussion in #33510), the wallet should still resubmit to the mempool but not force relay, nodes should periodically choose random transactions from their mempool to rebroadcast, and sendtemplate and erlay would be immensely helpful. Lastly, we should have an RPC and GUI option for users to do a single rebroadcast of a specified transaction.

The idea is that this addresses the 2 ways that transactions get dropped from mempools (they reach the mempool expiry time limit, or the mempool minimum feerate is increased to the point that the transaction is evicted) while still preserving sender and receiver privacy.

On the expiry timer, the discussion in #33510 is going in the direction of dropping the expiry timer in favor of eviction of things that should have been but weren't mined after some time. With the expiry time extended, low feerate transactions that exceed the timer no longer need to be rebroadcast. High feerate transactions that are not mined could still get evicted, but if they were not mined, then probably something else is going on that would render rebroadcast useless. Additionally, sendtemplate and erlay would help to ensure that that transaction had propagated well.

For low feerate transactions, the goal is to get such transactions into mempools as soon as they are ready to accept them. If mempools are heterogeneous, there should be some node somewhere that has the the low feerate transactions it its mempools. With a random rebroadcast, those transactions should be rebroadcast eventually. Additionally, if wallets still resubmit to their own mempools periodically, our node will also rebroadcast it eventually. No preference is given to transactions that come from loaded wallets or RPC.

And of course, if a user wants to rebroadcast something manually, then they should be able to do so with a RPC or GUI button.

the case where the wallet is the receiver is much worse because

But also when the wallet is the receiver, it is more specifically interested in rebroadcasting. As an intentional receiver, you want to make sure that incoming transactions get mined so that you can get paid. Hence rebroadcasting such transactions is also in the user's interest. Keeping these transactions in our mempool, and in other mempools, allows us to CPFP such transactions. With CPFP, the receiver can increase the package feerate such that a transaction does get confirmed in a timely manner.

Fundamentally, both senders and receivers should want to rebroadcast - senders so that their payment can go through and they can get whatever service, and receivers so that they can get paid. At the same time, there are times where the sender may not be interested in rebroadcast (e.g. if the transaction is not for delivery of a good or service), and that the receiver may not be interested in rebroadcast (e.g. getting dust spam). But I don't think that the wallet should cater to one or the other specifically, as in the average case, both senders and receivers are interested in rebroadcast.

One potential improvement that I would like to advocate for is something similar to #30471.

One issue that came up in the offline discussion was how to decide what to evict from the (re)broadcast pool. It seems like it would be DoS-able if an attacker makes a bunch of low feerate dust to addresses in your wallet, then they could fill up the rebroadcast pool and prevent rebroadcast of desired transactions.

There is an argument for dropping automated rebroadcast entirely, other than mempool submission on load. With the current expiry timer of 2 weeks, it could be that rebroadcast isn't meaningfully helping transactions get confirmed since that is a fairly long timeframe, certainly much longer than when rebroadcast was first introduced. In a dynamic fee environment, rebroadcast is also possibly less helpful as the things that would get rebroadcast might not even be accepted into mempools. So maybe automated rebroadcast is not all that useful? But I would be interested in some data on this before jumping to this conclusion.

In a world without automated rebroadcast, we definitely should make manual rebroadcast easier by adding a RPC and GUI button for it.