Proposed Update
Add a new RPC endpoint, listmempooltransactions. Takes as args a start_sequence and verbose.
Returns:
- if verbose false, list of txids + their
entry_sequencewhere each entry’sentry_sequence>= the provided `start_sequence. - if verbose true, raw tx output info including each entry’s
entry_sequence.
Builds on work done in #19572.
Rationale
The current mempool RPCs are lacking an ability to scan for updates in a more efficient manner. You can subscribe for updates via the ZMQ pipeline, but even this is inefficient to recover from if your consumer app falls offline.
ZMQ also is a push protocol, and doesn’t provide a rate-limiting mechanism.
In the case of core-lightning, we don’t assume access to bitcoin-core/ZMQ, so we’re unable to do efficient mempool querying. There have been some recent CVEs come to light where having optional access to mempool updates may prove useful.
Paging, filtering by last seen, and the addition of full tx data in the call makes mempool data more readily available to any client app. This is good for self-sovereignty as it reduces the need for additional dependencies and app data sources to efficiently query and parse mempool data.
Other Things I Considered
My initial attempt at this modified getrawmempool to
- take a
start_sequenceto allow for filtering, and - adding the txdata (inputs outpoints + outputs (amount + scriptPubKeys)).
That was pretty ugly however, given that the current data model for getrawmempool is around a concept of “mempool entry” data. This new RPC in contrast only returns transaction data (and does not report on information about other mempool dependencies etc; for that you should still call getrawmempool or getmempoolentry).
You could add a start_sequence to the existing getrawmempool to help with entry data paging/querying but that’s secondary to my goals for fetching relevant txdata from the mempool.
Additions/Changes
This PR could be further improved by
- Add a
page_sizeargument which allows a calling application to limit the number of results returned.
Future Work
The current RPC only supports finding added mempool transactions. It’d be interesting to experiment with keeping track of evicted/not mined transactions and adding them to the results.
This would require:
- An additional memory buffer (perhaps a configurable memory limited ring buffer?) for evicted tx data and the mempool sequence of the eviction.
- Adding a
sequence_actionfield for results, which indicates whether the sequence is for the tx’sadditionoreviction.
You could also keep track of tx movements into blocks, but this seems less useful/urgent in general.
Usage
You can see an example of this implemented and used in CLN for finding paid onchain invoices in this branch: https://github.com/ElementsProject/lightning/compare/master...niftynei:lightning:nifty/mempool-scan
Note that the CLN implementation doesn’t currently keep the mempool_sequence in disk, so it’ll reload/rescan the mempool at start. This may be desirable?
Here’s how a caller would use this
01) start node
12) listmempooltransactions 0 to get backlog
23) call listmempooltransactions again with `start_sequence` set to `mempool_sequence` result from last call
Note that this works well with ZMQ as the mempool_sequence number is identical.
