Feature Request: Broadcast Pool

craigraw commented at 11:49 am on July 17, 2024: none

Please describe the feature you’d like to see added.

This is a feature request for a broadcast pool. A broadcast pool is a cache local to the node which contains transactions which have been initially broadcast from that node, either from a local wallet or via an RPC like sendrawtransaction. Broadcast transactions are retained in this cache until they are included in a block, or expire after the configured mempoolexpiry period. The transactions in the broadcast pool are included in views on the mempool, ensuring that transactions broadcast from the node are not lost to the user due to fee rate environment changes. Transaction invalidation and replacement is handled in the same way as in the mempool. Transactions in broadcast pool that are not present in the mempool will occasionally be rebroadcasted, making this existing functionality available to all transactions broadcasted through the node and not just those created by a local wallet.

As an additional feature, a future-dated timestamp can be added in metadata associated with a broadcast pool transaction. Should such a timestamp exist, the transaction is only added to the local mempool and broadcast to the network after that date, allowing the user to mitigate timing analysis and obtain greater privacy. Future-dated transactions are also included in views on the mempool, but are marked as unbroadcast: true in mempool RPC responses, and are similarly subject to removal via normal transaction invalidation and replacement rules.

The necessary resource limitations on the mempool lead to an unfortunate consequence for wallet developers and users. Because the mempool does not distinguish between transactions broadcast by the user, and all other transactions, broadcasted transactions can be ’lost’ to the user due to external fee rate environment changes. These ’lost’ transactions are then not available in a wallet interface for view or replacement by fee - they simply (and often unexpectedly) disappear. Further, they may mysteriously reappear, having not been dropped from a larger mempool as typically configured by a miner. This is a poor user experience, and leads users to consider out-of-band solutions, or to increase the size of their local mempool which is resource inefficient and counterproductive to the network as a whole. It is reasonable to assume that a user’s broadcasted transactions carry much more economic value to them than most other transactions, and justify protection through some degree of additional resource allocation. This feature request seeks to realise that rationale, providing a better user experience consistent with the health of the network.

With respect to privacy, timing analysis is unfortunately a simple and convincing approach to link otherwise unconnected transactions. For example, a common use case is to transfer the funds in one wallet to another, perhaps for security reasons. In order to avoid linking the funds in disparate UTXOs, the user must not only send them one by one from the old wallet to the new, but do so with random (and lengthy) time intervals between each send. Failure to do so will result in a much higher chance of funds being linked onchain simply by the transactions being included in the same or adjacent blocks. Most bitcoin nodes are run as continuously online servers, and therefore are ideal candidates to handle this otherwise onerous task.

Describe the solution you’d like

The broadcast pool is a local cache of usually small but configurable maximum size. A default maximum value of 300 kB is proposed. Similarly to the mempool, the cache is persisted on shutdown and restored on startup. Transactions in the broadcast pool are subject to almost all of the same rules as those in the mempool, except that they are not removed due to mempool size limitations.

Transactions in the broadcast pool not present in the mempool are occasionally rebroadcast. Optionally, a future-dated timestamp can be specified when broadcasting a transaction. In this case, the transaction is added to the broadcast pool but not to the local mempool or broadcast to the network until after that date has passed. If the node is restarted after the timestamp on a future-dated transaction, the transaction will be immediately broadcast. All transactions in the broadcast pool are included in views on the mempool, and in that sense the broadcast pool can be viewed as an extension of the mempool where different eviction rules apply. Future-dated broadcast transactions are not included in block templates.

If the broadcast pool is full and the broadcast is not future-dated, the broadcast will proceed as it does currently but with a warning returned to the user. A future-dated broadcast to a full broadcast pool will fail.

Describe any alternatives you’ve considered

It is possible to try to solve the evicted transaction retention problem on the application layer within a wallet application. Practically however, this is far from ideal. Client wallets are generally opened and closed at indeterminate times, leading to a state conflict with the connected node that must be resolved. Light client protocols are generally not optimized for this use case, leading to significant complexity and performance degradation. Further, consensus logic such as that used for transaction replacement by fee must be replicated in each wallet application.

With respect to the timing analysis problem, setting a future-dated broadcast will encourage the user to leave their wallet application open, which is in conflict to the ideal of cold storage. Further, the environment for mobile wallets is not well suited to this long running task.

In summary, addressing these problems in a wallet application leads to poor separation of concerns.

Please leave any additional context

A short list of related issues from some popular Bitcoin wallets:

craigraw added the label Feature on Jul 17, 2024

maflcko commented at 12:18 pm on July 17, 2024: member

making this existing functionality available to all transactions broadcasted through the node and not just those created by a local wallet.

Would it not be possible to implement this by creating a wallet called broadcast_pool and then write simple wrapper functions to implement the broadcast pool features?

Future-dated transactions are also included in views on the mempool, but are marked as unbroadcast: true in mempool RPC responses

I don’t think it is possible to include something in the mempool and hide the fact that it is included from remote peers. This is a recurring topic. See e.g. #27509 (comment)

broadcasted transactions can be ’lost’ to the user due to external fee rate environment changes. These ’lost’ transactions are then not available in a wallet interface for view or replacement by fee - they simply (and often unexpectedly) disappear.

If you really care about putting the transaction in the mempool, you can use prioritisetransaction. However, that comes with the corresponding downsides as well. (Other, higher-fee transactions may be discarded; privacy leaks; …).

Generally, my recommendation would be to implement this feature in the wallet: See #11887.

After it has been implemented in the wallet, it can likely be ported to another slimmer module, if users think the wallet is too heavy to use for this task.

craigraw commented at 8:07 am on July 18, 2024: none

Would it not be possible to implement this by creating a wallet called broadcast_pool and then write simple wrapper functions to implement the broadcast pool features?

All of the wallets linked in the issues above (and of course many others) use the Electrum server protocol, and in general rely on Electrum server implementations that do not require or use the wallet functionality. Even if they were to do so, I cannot see how it is practical to broadcast arbitrary transactions via the wallet RPCs.

I don’t think it is possible to include something in the mempool and hide the fact that it is included from remote peers. This is a recurring topic.

An alternative would be to have a broadcast pool RPC. The minimum requirement is that future-dated transactions be retrievable via RPC, so that they can be included in transactions associated with a wallet.

If you really care about putting the transaction in the mempool, you can use prioritisetransaction.

My understanding from reading the original PR is that this affects inclusion of transactions into a proposed block, and not retention in the mempool. As you suggest though, changing the contents of the mempool for this purpose is not ideal.

Generally, my recommendation would be to implement this feature in the wallet

Unfortunately, I don’t think that would address this use case in a practical manner.

maflcko commented at 9:04 am on July 18, 2024: member

If you really care about putting the transaction in the mempool, you can use prioritisetransaction.

My understanding from reading the original PR is that this affects inclusion of transactions into a proposed block, and not retention in the mempool. As you suggest though, changing the contents of the mempool for this purpose is not ideal.

While the RPC is placed in the “mining” section, it also affects the mempool size trimming calculation. But yeah, using the RPC here may not be ideal.

I don’t think it is possible to include something in the mempool and hide the fact that it is included from remote peers. This is a recurring topic. An alternative would be to have a broadcast pool RPC. The minimum requirement is that future-dated transactions be retrievable via RPC, so that they can be included in transactions associated with a wallet.

If all you need is just a way to get back a byte blob by asking for its hash, and the byte blobs are sent to sendrawtranscation after some time, it would be trivial to implement. However, if you also want to build transaction graphs and do replacements on them, it will be non-trivial. Maybe if you limit yourself to only TRUC v3 transactions, it could be easier to implement, but I haven’t checked this. I guess it could help if you specified why you need the transactions to be “included in views on the mempool”. Do you need the views to determine which transactions are “active”, for calculating the wallet balance and seeing which coins are spendable in the view?

Generally, my recommendation would be to implement this feature in the wallet

Unfortunately, I don’t think that would address this use case in a practical manner.

I meant that the general nature of this problem (and its general solution) is sufficiently similar (if not identical), so that having a solution for one is close to having a solution for the other.

craigraw commented at 9:48 am on July 18, 2024: none

Do you need the views to determine which transactions are “active”, for calculating the wallet balance and seeing which coins are spendable in the view?

Yes - the wallet would consider any UTXOs included in these transactions as spent, and any UTXOs they create would be spendable. Of course there would need to be consideration in a wallet UI to avoid creating a situation where non-existent UTXOs are spent in a broadcasted transaction (which would simply fail on broadcast as it does now).

However, if you also want to build transaction graphs and do replacements on them, it will be non-trivial.

To clear, future-dated broadcasts are an additional (although powerful) feature request - I added it here because it appeared to fit within the framework of retaining locally broadcast transactions. It could also be built higher up the stack (for example in Electrum servers), since transactions that are restricted to the local environment may more simply be removed in the event of any conflict, without considering specific consensus rules. This is not true for broadcasted transactions, so I would consider that the more critical part of this issue.

maflcko commented at 10:41 am on July 18, 2024: member

Yes - the wallet would consider any UTXOs included in these transactions as spent, and any UTXOs they create would be spendable. Of course there would need to be consideration in a wallet UI to avoid creating a situation where non-existent UTXOs are spent in a broadcasted transaction (which would simply fail on broadcast as it does now).

I wonder if the feature helps users in that case. If a set of transactions isn’t in “the mempool” (the one the wallet is connected to) right now, it will probably not be confirmed any time soon. So indicating to the user that the transactions are fine (and spendable) may be causing more issues than it solves. As a user, I’d probably expect to be notified that a transaction isn’t in “the connected mempool” and be offered some kind of optional fee-bump (or rebroadcast button, if the transaction hit mempool-expiry).

Obviously, if the fee bump then happens through a child transaction, the transactions will have to be submitted as a package for relay. (Not sure if electrum supports this)

To clear, future-dated broadcasts are an additional (although powerful) feature request - I added it here because it appeared to fit within the framework of retaining locally broadcast transactions.

Looks like the retaining local transactions issue was fixed in https://github.com/spesmilo/electrum/issues/3595#issuecomment-363535872 , so it seems possible to implement the future-dated feature there as well?

since transactions that are restricted to the local environment may more simply be removed in the event of any conflict, without considering specific consensus rules. This is not true for broadcasted transactions, so I would consider that the more critical part of this issue.

I assume when you say “consensus” you mean “policy”. If you want to avoid implementing all the policy code in the local wallet app, you can rely on the testmempoolaccept to try “mempool-extensions”. However, I don’t think there is a way to avoid in the local wallet app the concept of knowing the topology of unconfirmed transactions (and their unspent outputs). I am not sure if it is really easier to keep a remote mempool view of the user’s current wallet, rather than letting the user modify the “view” locally (transaction replacements, or transaction chains), and then testing their modification via testmempoolaccept or a direct submit to the mempool+broadcast.

craigraw commented at 12:12 pm on July 18, 2024: none

As a user, I’d probably expect to be notified that a transaction isn’t in “the connected mempool” and be offered some kind of optional fee-bump (or rebroadcast button, if the transaction hit mempool-expiry).

In general I agree. I was referring specifically to future-dated transactions there, so we may be speaking slightly at cross purposes. But to speak to broadcasted transactions, it’s less important to consider the local mempool, since we must assume they are contained in some miners mempool. For most users, it’s more important that they always have access to a transaction to fee bump it - I think that’s the primary motivation from a wallet POV.

Obviously, if the fee bump then happens through a child transaction, the transactions will have to be submitted as a package for relay.

Good point. It doesn’t currently - but that’s solvable.

Looks like the retaining local transactions issue was fixed in https://github.com/spesmilo/electrum/issues/3595#issuecomment-363535872 ,

Electrum have adopted a practical approach, but it’s not entirely correct. They consider all conflicting transactions, and then discard according to the hierarchy confirmed > mempool > local (reference). This does not consider policy rules, and may result in discarding a transaction that is later mined.

I assume when you say “consensus” you mean “policy”. If you want to avoid implementing all the policy code in the local wallet app, you can rely on the testmempoolaccept to try “mempool-extensions”

That RPC call is not currently supported by Electrum servers. Would testmempoolaccept not always return false in this situation though, rejecting transactions below a certain fee rate?

I am not sure if it is really easier to keep a remote mempool view of the user’s current wallet, rather than letting the user modify the “view” locally (transaction replacements, or transaction chains), and then testing their modification via testmempoolaccept or a direct submit to the mempool+broadcast.

The key problem is retaining transactions that have been dropped due to mempool size limitations (as opposed to other reasons). I’ve tried to implement this in Sparrow already, and found it particularly challenging to do efficiently within the constraints of the Electrum protocol which is optimized for performance and not conflict resolution. I abandoned the attempt before trying to reimplement policy rules. That said, it seems to me this functionality would be generally desirable and useful, and should not need to be reimplemented in every node client that makes use of transaction broadcasting.

maflcko commented at 2:07 pm on July 18, 2024: member

Electrum have adopted a practical approach, but it’s not entirely correct. They consider all conflicting transactions, and then discard according to the hierarchy confirmed > mempool > local (reference). This does not consider policy rules, and may result in discarding a transaction that is later mined.

Yes, but that should be harmless, as a confirmed transaction in the chain can be retrieved from the chain. The same problem would exist in a broadcast pool, because any transaction in it is not guaranteed to be confirmed, and an earlier (or later) replacement could be confirmed, or none at all.

If you are looking for a solution to keep a backup of any wallet transactions created, I don’t see another way than to store them directly in the wallet either locally (or remotely in a Bitcoin Core wallet, or another database).

That RPC call is not currently supported by Electrum servers. Would testmempoolaccept not always return false in this situation though, rejecting transactions below a certain fee rate?

Only if they are below the mempool min fee rate. However, in that case the user will likely have to wait for a long time anyway (possibly more than 2 weeks), so the transaction will need to be backup up anyway. (For checking the transaction, you can use prioritisetransaction +delta+testmempoolaccept+prioritisetransaction -delta, possibly in a single batch call, to check all policy rules except for the fee).

craigraw commented at 2:50 pm on July 18, 2024: none

Yes, but that should be harmless, as a confirmed transaction in the chain can be retrieved from the chain. The same problem would exist in a broadcast pool, because any transaction in it is not guaranteed to be confirmed, and an earlier (or later) replacement could be confirmed, or none at all.

I don’t agree - it could lead to a situation where a user is fee bumping a transaction unnecessarily. Certainly there are no guarantees on confirmation, but where a transaction is discarded locally in contradiction to your node’s policy rules you have an unnecessarily less accurate view to make decisions on.

For checking the transaction, you can use prioritisetransaction +delta+testmempoolaccept+prioritisetransaction -delta, possibly in a single batch call, to check all policy rules except for the fee).

Noted, although it seems a convoluted solution. The delta needs to be calculated (the mempool min fee is not available to Electrum clients) and obviously prioritisetransaction is not available either.

maflcko commented at 3:39 pm on July 18, 2024: member

Yes, but that should be harmless, as a confirmed transaction in the chain can be retrieved from the chain. The same problem would exist in a broadcast pool, because any transaction in it is not guaranteed to be confirmed, and an earlier (or later) replacement could be confirmed, or none at all.

I don’t agree - it could lead to a situation where a user is fee bumping a transaction unnecessarily. Certainly there are no guarantees on confirmation, but where a transaction is discarded locally in contradiction to your node’s policy rules you have an unnecessarily less accurate view to make decisions on.

Maybe I don’t understand. Do you disagree that it is harmless, or do you disagree that the same problem exists in a broadcast pool?

Also, why would a fee bump be unnecessary when the transaction doesn’t even meet the min mempool fee? The fee bump is optional and up to the users, so they can decide whether they want to wait or not. Though, given that electrum doesn’t even return the mempool min fee, a fee bump seems difficult either way.

craigraw commented at 8:54 am on July 19, 2024: none

Do you disagree that it is harmless, or do you disagree that the same problem exists in a broadcast pool?

Both. A broadcast pool has direct access to policy rules, a wallet client generally does not.

Also, why would a fee bump be unnecessary when the transaction doesn’t even meet the min mempool fee?

Because it may still exist in a larger miner’s mempool. Consider the following situation: Alice broadcasts a transaction at a fee rate of 10 sats/vb. Her min mempool fee increases later to 11 sats/vb, so the transaction is dropped from her mempool, but it still exists in a larger miner’s mempool. Her min mempool fee then drops to 9 sats/vb. Mallory broadcasts a conflicting transaction at 9 sats/vb, and Alice’s wallet discards her transaction using the hierarchy confirmed > mempool > local. Alice should at the least consult a block explorer like mempool.space at this point, since her wallet is no longer showing her an accurate picture of which transaction is likely to be mined. My point is that with a broadcast pool, this is unnecessary - her node has all the data and algorithms to make the correct decision about which transaction should be retained in her wallet view.

maflcko commented at 9:51 am on July 19, 2024: member

The same problem exists in a broadcast pool. For example:

Alice broadcasts at 10sat/vb, but the transaction isn’t confirmed in $mempool_expiry time, and thus dropped from Alice’s mempool (and broadcast pool). (The transaction may or may not be in a miner’s mempool with larger expiry)
Mallory broadcasts her at 9sat/vb.
Alice’s wallet discards her transaction using 10sat/vb, based on the hierarchy confirmed > mempool > local (or confirmed > mempool > broadcast_pool > local)

So I don’t think broadcast pools will solve this particular problem. I am ignoring the fact that untrusted Mallory can include Alice’s wallet inputs in a transaction here and Alice has somehow strong expectations in this adversarial setting.

I think a more robust solution here is likely an automatic or manual rebroadcast inside Alice’s wallet.

craigraw commented at 10:07 am on July 19, 2024: none

and thus dropped from Alice’s mempool (and broadcast pool)

This is where we differ. From my description in the OP: “Transactions in the broadcast pool are subject to almost all of the same rules as those in the mempool, except that they are not removed due to mempool size limitations.” Therefore, in this scenario, the transaction would not be dropped from the broadcast pool.

I don’t want to focus too deeply on whether the situation must be adversarial - as a wallet developer, I am always trying to avoid the situation where the wallet says one thing, and a block explorer says another, regardless of how that arises.

maflcko commented at 10:13 am on July 19, 2024: member

and thus dropped from Alice’s mempool (and broadcast pool)

This is where we differ. From my description in the OP: “Transactions in the broadcast pool are subject to almost all of the same rules as those in the mempool, except that they are not removed due to mempool size limitations.” Therefore, in this scenario, the transaction would not be dropped from the broadcast pool.

I think you accidentally mixed up size limits and expiry. They are different, because one looks at the memory usage and the other at expiry times (regardless of memory usage). With expiry, it is possible that transactions are dropped, even if the size limit is not yet hit. You also say this yourself in your OP: “Broadcast transactions are retained in this cache until they are included in a block, or expire after the configured mempoolexpiry period.”

craigraw commented at 10:21 am on July 19, 2024: none

They are different, because one looks at the memory usage and the other at expiry times (regardless of memory usage). With expiry, it is possible that transactions are dropped, even if the size limit is not yet hit.

I agree with this description, but am unclear how this impacts the scenario I described, where in no case is a transaction dropped due to expiry. All of the events described can occur within a 2 week period.

maflcko commented at 2:28 pm on July 22, 2024: member

Well, your same scenario would be possible with a broadcast pool, because a broadcast pool would need to be size-limited. So Mallory can put the 9sat/vB transaction in the broadcast pool or mempool, when the 10sat/vB transaction has been dropped from the mempool and broadcast pool. Alice’s wallet will still discard her transaction using the hierarchy confirmed > mempool > local (or confirmed > mempool > broadcast_pool > local).

I understand the use cases and advantages of a broadcast pool, but all I want to say that the broadcast pool is probably not the right tool to fix the issue of ’lost’ transactions.

craigraw commented at 2:53 pm on July 22, 2024: none

So Mallory can put the 9sat/vB transaction in the broadcast pool

From the OP: “A broadcast pool is a cache local to the node which contains transactions which have been initially broadcast from that node”. So Mallory’s transaction would not enter Alice’s broadcast pool, as Alice did not initiate its broadcast. The broadcast pool only contains transactions that Alice has broadcast either from her Core wallet or via the sendrawtransaction RPC. It does not contain transactions received over the P2P network.

when the 10sat/vB transaction has been dropped from the mempool and broadcast pool

The 10 sat/vB transaction would not be dropped from Alice’s broadcast pool, as it would in this example be the only transaction in that pool.

vasild commented at 7:53 am on July 23, 2024: contributor

This is related

#29415

it contains a storage of unbroadcast transactions, separate from the mempool. Transactions from that storage are periodically broadcast. A difference from this feature request is that transactions are removed from the unbroadcast storage once they are seen by the network. This can be easily changed however in a followup - for example if a local transaction is seen by the network, appears in ours and others’ mempools, then keep it in the unbroadcast storage until included in a block. If at some point it disappears (pushed away by higher fee transactions) then re-broadcast it.

It is a separate task, which I am planning after #29415 is merged to have the wallet account for transactions that are in the unbroadcast storage and not in the mempool.

maflcko commented at 8:29 am on July 23, 2024: member

The broadcast pool only contains transactions that Alice has broadcast either from her Core wallet or via the sendrawtransaction RPC.

I see. I assumed that Mallory and Alice could have wallet on the same node (multiwallet mode), or that they shared the sendrawtransaction RPC on the same node. Not sure if this assumption is accurate, but it seemed plausible that some nodes expose their sendrawtransaction to third parties.

craigraw commented at 9:09 am on July 23, 2024: none

@vaslid This sounds excellent. From your comment it seems like #29415 is a great basis to implement this. I will look into that work more closely, as it’s very useful in it’s own right. Currently, if Tor is configured, Sparrow broadcasts over Tor via a random external service such as mempool.space for greater privacy. As I understand it, this would no longer be necessary with -privatebroadcast=1.

It is a separate task, which I am planning after #29415 is merged to have the wallet account for transactions that are in the unbroadcast storage and not in the mempool.

This would also be useful for those applications that leverage the Core wallet, including Sparrow’s own Cormorant library, an EPS-like light Electrum server. But for full Electrum servers like ElectrumX and Fulcrum which use RPCs like getrawmempool, some other way to retrieve unbroadcast transactions will be necessary.

I also note that #29415 contains methods to internally schedule a rebroadcast, and I wonder if that could be later leveraged for future-dated broadcasts as described here.

vasild commented at 11:53 am on July 23, 2024: contributor

Sparrow broadcasts over Tor via a random external service such as mempool.space for greater privacy. As I understand it, this would no longer be necessary with -privatebroadcast=1

Correct.

But for full Electrum servers like ElectrumX and Fulcrum which use RPCs like getrawmempool, some other way to retrieve unbroadcast transactions will be necessary

Maybe. Note that usually a transaction will be in unbroadcast pool and not in the mempool only for a few seconds, until it round-trips through the network back to us.

#29415 contains methods to internally schedule a rebroadcast, and I wonder if that could be later leveraged for future-dated broadcasts as described here.

Yes, that would be a nice addition and easy to implement as well. I will refrain from adding it to #29415 just not to feature creep it. Also, the unbroadcast storage (pool) in #29415 is not persistent upon restarts. Deliberately delaying the broadcast for non-trivial amount of time warrants having the pool persist during restarts. So, most likely, both of this will come together in a followup.

craigraw commented at 12:52 pm on July 23, 2024: none

Maybe. Note that usually a transaction will be in unbroadcast pool and not in the mempool only for a few seconds, until it round-trips through the network back to us.

Yes, this would be the normal case. But if the transaction cannot enter the mempool because of low fee rate/mempool size constraints, then it will remain only in the unbroadcast pool as you described above. In this case, the Electrum server should be able to retrieve the transaction in order to include it in the list of relevant transactions for a wallet (for example for fee bumping).

Yes, that would be a nice addition and easy to implement as well.

Great to hear!

Feature Request: Broadcast Pool #30471

Please describe the feature you’d like to see added.

Is your feature related to a problem, if so please describe it.

Describe the solution you’d like

Describe any alternatives you’ve considered

Please leave any additional context