zmq: mempool notifications

jmcorgan commented at 5:07 pm on August 19, 2016: contributor

This is a current rebased version of #7753 by @laanwj. Please see that PR for original discussion and existing ACKs.

Add notifications when transactions enter or leave the mempool.

These can be enabled with zmqpubmempool:

mempooladded: a transaction was added to the mempool mempoolremoved: a transaction was removed from the mempool

[…]

It also makes it possible to keep statistics on why transactions disappear from the mempool.

mempool: add notification for added/removed entries

Add notification signals to make it possible to subscribe to mempool
changes:

- NotifyEntryAdded(const CTxMemPoolEntry &)>
- NotifyEntryRemoved(const CTxMemPoolEntry &, MemPoolRemovalReason)>

Also add a mempool removal reason enumeration, which is passed to the
removed notification based on why the transaction was removed from
the mempool.

70e495287c

zmq: refactor initialization

Replace factories list with calls to a function.

This simplifies the code (every notifier is only created once anyway),
and makes it possible to pass arguments to notifier contructors.

bd7df61c31

zmq: add mempool notifications

Add notifications when transactions enter or leave the mempool.

These can be enabled with `pubmempool`:

- `mempooladded`: a transaction was added to the mempool
- `mempoolremoved`: a transaction was removed from the mempool

This allows third-party software to keep track of what is in the mempool
in real time.

7ececb2e33

doc: Add documentation for zmq mempool notifications 7882b9b234

zmq: refactor to make all notifiers register themselves

Make the registration process less circuitous by making
the notifiers listen to their appropriate events themselves

11d20c998b

in src/main.cpp: in 11d20c998b

2751@@ -2752,7 +2752,7 @@ bool static DisconnectTip(CValidationState& state, const CChainParams& chainpara
2752             list<CTransaction> removed;
2753             CValidationState stateDummy;
2754             if (tx.IsCoinBase() || !AcceptToMemoryPool(mempool, stateDummy, tx, false, NULL, true)) {
2755-                mempool.removeRecursive(tx, removed);
2756+              mempool.removeRecursive(tx, removed, MemPoolRemovalReason::REORG);

kanzure commented at 5:56 pm on August 19, 2016:

trivial nit: indentation symmetry between mempool.removeRecursive line and vHashUpdate.push_back line.

jmcorgan commented at 6:10 pm on August 19, 2016:

Emacs C++ mode FTW. Will cleanup.

jmcorgan commented at 6:12 pm on August 19, 2016: contributor

Hmm, looks like zmq_test.py needs some attention.

dcousens commented at 2:49 am on August 24, 2016: contributor

a transaction was added to the mempool

OOI, is the case if a re-org occurs and the transaction only existed in the pre re-org chain?

fanquake added the label RPC/REST/ZMQ on Aug 31, 2016

jonasschnelli commented at 7:41 am on October 26, 2016: contributor

Needs rebased. Thanks.

jonasschnelli commented at 8:08 am on October 26, 2016: contributor

Conceptual issue: Shouldn’t there be a special ZMQ message when bitcoind starts up? Because of the nature of the “self-healing” ZMQ connections, listeners should probabyl be informed if bitcoind restarts (which currently result in sweeping the mempool). Maybe it could be a specific “clearmempool” message during startup.

bitkevin commented at 8:35 am on October 26, 2016: contributor

Nice work!

My suggestion:

when start or restart bitcoind there should be a special event to tell consumers which listen to the ZMQ. (same as @jonasschnelli)
Add rawhex in the message. The consumers need to decode hex rawtx after get the ZMQ message, otherwise they need to do another RPC call ‘getrawtransaction’.
When remove txs, they should be in order: remove children first and then the parents. ZMQ messages should also in the particular order. (Maybe the PR has already guarantee of it, I am not sure)

I patched bitcoind to the same thing: https://github.com/btccom/explorer-kv/blob/master/deploy/docker/bitcoind/v0.12.1/bitcoind-explorer-v0.12.1.patch

When bitcoind start, also need to push the latest block in ZMQ, so consumers could know if they need to sync to the latest block(with rpc call to get blocks), after that could consume txs logs from ZMQ.

dcousens commented at 7:45 am on December 19, 2016: contributor

concept ACK

jtimon commented at 1:25 am on December 20, 2016: contributor

Needs rebase

fanquake commented at 0:49 am on January 21, 2017: member

@jmcorgan Are you planning on continuing this work? Needs a rebase, and nits addressed.

dcousens commented at 11:24 am on January 24, 2017: contributor

Close now?

laanwj commented at 12:21 pm on January 24, 2017: member

Closing (for now). Might pick this up later.

laanwj closed this on Jan 24, 2017

karelbilek commented at 4:58 am on March 17, 2017: contributor

This would actually be very useful, now with all the full blocks and transactions falling out of mempools more often.

Any plans for continuing with this? :) thx

edit: oh, there is also this PR, already merged. Not sure what it does 100%

https://github.com/bitcoin/bitcoin/pull/9371

jmcorgan commented at 6:48 am on March 17, 2017: contributor

My own branch was only an attempt at keeping this current with progress on master; the real work was done by @laanwj. I’ll defer to him whether or not he’s willing to continue from here.

laanwj commented at 9:11 am on March 17, 2017: member

@jmcorgan @runn1ng It’s on my long list. Sorry for occasionally starting projects then seemingly abandoning them. It’s just that it’s not always clear how much interest there is in things, and keeping things rebased while it’s unclear how much attention something will get sometimes feels like a waste of time.

But it’s clear that there is user interest in this and some of the changes here have been part of #9371, just not the zmq part. I’ll get to that… But if someone else feels like picking this up to accelerate things (hopefully) go ahead.

laanwj added the label Up for grabs on Nov 28, 2017

Sjors commented at 4:49 pm on November 29, 2017: member

Knowing when a transaction is added to the mempool might be useful in the test framework as well, e.g. so sync_mempools can be implemented asynchronously instead of through polling (cc @jnewbery).

jmcorgan deleted the branch on Aug 29, 2020

MarcoFalke removed the label Up for grabs on Nov 29, 2021

MarcoFalke commented at 9:51 am on November 29, 2021: member

picked up (sort of) in #23624

DrahtBot locked this on Nov 29, 2022

zmq: mempool notifications #8549