Add ZMQ support. Report blocks and transactions via ZMQ.

jgarzik commented at 5:08 pm on November 18, 2014: contributor

Continues Johnathan Corgan’s work. Supercedes #4594

Continues Johnathan Corgan's work.

6392f40a49

hoffmabc commented at 5:43 pm on November 18, 2014: none

This is great work and I’m very excited to see this being contributed.

sipa commented at 8:11 pm on November 18, 2014: member

@jgarzik what is the intended meaning of a block being relayed here?

Just triggering on AcceptBlock isn’t particularly interesting - it just means that it passes preliminary checks and is accepted into the block tree. I think the most interesting meaning is “is accepted as new tip of the chain”, which already has a NotifyBlockTip signal.

theuni commented at 10:25 pm on November 18, 2014: member

@jgarzik note that the code wasn’t actually compiled/tested by travis here because libzmq doesn’t exist. In fact, the error handling in configure is broken. Libs like this usually default to “auto” or so, so that an explicit –enable-zmq would cause an error if the lib wasn’t found.

I commented in #5303 about adding the dependency, please have a look there. Also, please consider fixing up configure and adding an explicit “–enable-zmq” to the travis tests as necessary.

in src/init.cpp: in 6392f40a49

301@@ -298,7 +302,7 @@ std::string HelpMessage(HelpMessageMode mode)
302     strUsage += "  -zapwallettxes=<mode>  " + _("Delete all wallet transactions and only recover those parts of the blockchain through -rescan on startup") + "\n";
303     strUsage += "                         " + _("(1 = keep tx meta data e.g. account owner and payment request information, 2 = drop tx meta data)") + "\n";
304 #endif
305-
306+    strUsage += "  -zmqpub=<endpoint>     " + _("Publish blocks and transactions on ZMQ port 'endpoint'") + "\n";

Diapolo commented at 12:32 pm on November 19, 2014:

Is there a default you could list here?

in src/init.cpp: in 6392f40a49

922@@ -919,6 +923,10 @@ bool AppInit2(boost::thread_group& threadGroup)
923     BOOST_FOREACH(string strDest, mapMultiArgs["-seednode"])
924         AddOneShot(strDest);
925 
926+    if (mapArgs.count("-zmqpub"))
927+      ZMQInitialize(mapArgs["-zmqpub"]);
928+
929+

Diapolo commented at 12:32 pm on November 19, 2014:

Nit: Unneeded space and indentation.

in src/zmqports.h: in 6392f40a49

0@@ -0,0 +1,25 @@
1+// Copyright (c) 2009-2010 Satoshi Nakamoto
2+// Copyright (c) 2014 The Bitcoin developers
3+// Distributed under the MIT/X11 software license, see the accompanying
4+// file COPYING or http://www.opensource.org/licenses/mit-license.php.
5+#ifndef ZMQPORTS_H

Diapolo commented at 12:33 pm on November 19, 2014:

Better: BITCOIN_ZMQPORTS_H here and for the end comment.

in src/zmqports.cpp: in 6392f40a49

0@@ -0,0 +1,175 @@
1+// Copyright (c) 2009-2010 Satoshi Nakamoto

Diapolo commented at 12:35 pm on November 19, 2014:

That file has the wrong intentation all over the file and the wrong license.

jtimon commented at 1:28 pm on November 19, 2014: contributor

As in #4594, idea ACK

in configure.ac: in 6392f40a49

143@@ -144,6 +144,12 @@ AC_ARG_ENABLE([glibc-back-compat],
144   [use_glibc_compat=$enableval],
145   [use_glibc_compat=no])
146 
147+AC_ARG_ENABLE([zmq],
148+  [AC_HELP_STRING([--disable-zmq],
149+  [Disable ZMQ notifications])],
150+  [use_zmq=no],

luke-jr commented at 1:05 pm on December 5, 2014:

This should use $enableval, or it will disable it with –enable-zmq

Minor nit: prefer –disable-zeromq as the name.

in configure.ac: in 6392f40a49

765+if test "x$use_zmq" == "xyes"; then
766+  AC_MSG_RESULT([yes])
767+  PKG_CHECK_MODULES([ZMQ],[libzmq],
768+    [AC_DEFINE([ENABLE_ZMQ],[1],[Define to 1 to enable ZMQ functions])],
769+    [AC_DEFINE([ENABLE_ZMQ],[0],[Define to 1 to enable ZMQ functions])
770+     AC_MSG_WARN([libzmq not found, disabling])

luke-jr commented at 1:05 pm on December 5, 2014:

Better to error if –enable-zmq is explicitly passed.

laanwj added the label Feature on Dec 10, 2014

hyperwang commented at 8:12 am on December 18, 2014: none

Johnathan Corgan has given a guidiance about how to configure in PR#4594 like this $ bitcoind -zmqpub=tcp://127.0.0.1/28332 I’m afraid that it should be like this $ bitcoind -zmqpub=tcp://127.0.0.1:28332

luke-jr commented at 10:02 pm on December 24, 2014: member

This should probably setup depends/ so binaries get ZeroMQ support…

theuni commented at 10:52 pm on December 24, 2014: member

@luke-jr See my comment above. I have an old branch where it’s hooked up: https://github.com/theuni/bitcoin/commits/zmq-test . That’ll need a bit of adapting since depends have changed somewhat, but should come pretty close.

jmcorgan commented at 6:06 pm on January 8, 2015: contributor

Just a quick note of thanks to Jeff Garzik for picking this up. I’ve been away from bitcoin development for a while but hope to return at some point.

jonasschnelli commented at 3:51 pm on March 11, 2015: contributor

Would def. nice to have this in 0.11. Needs rebase. @jgarzik i’m willing to continue this, depends if you have more uncommited/unpushed work.

in src/zmqports.cpp: in 6392f40a49

51+  zmqPubSocket = zmq_socket(zmqContext, ZMQ_PUB);
52+  if (!zmqPubSocket) {
53+    zmqError("Unable to open ZMQ pub socket");
54+    return;
55+  }
56+

amrali commented at 10:56 pm on March 11, 2015:

I think ZMQ_SNDHWM should be set here to configurable value (at least through configure.ac) this provides the option to the user on when to start dropping messages. the default value of zero means a queue as large as the available memory which is dangerous because networks could be slow or the production of messages is faster than the network I/O output – a precursor to a memory overflow crash.

ZMQ_LINGER is another important socket property to set here, the default value of -1 (infinite linger period on socket shutdown when there are queued messages to be sent) will prevent the application from shutting down gracefully. A configurable value (or a reasonable one) should be set to prevent this scenario. The linger period starts after a call to zmq_close on a socket or zmq_ctx_term on a context, once it expires all queued messages will be dropped.

in src/zmqports.cpp: in 6392f40a49

143+      zmq_setsockopt(zmqPubSocket, ZMQ_LINGER, &linger, sizeof(linger));
144+      zmq_close(zmqPubSocket);
145+      zmqPubSocket = 0;
146+    }
147+
148+    zmq_ctx_destroy(zmqContext);

amrali commented at 11:02 pm on March 11, 2015:

I think zmq_ctx_term should be used here instead if ZMQ_LINGER is to be set to some reasonable value to allow sending off what is possibly the last few queued messages (see my comment above the zmq_bind call). Not sure if this is desired or important to you.

jtimon commented at 12:31 pm on April 8, 2015: contributor

Yes, it would be nice to have this in 0.11.

btcdrak commented at 5:58 pm on May 14, 2015: contributor

#6103 says it has superceded this PR, of so, this PR should be closed.

laanwj commented at 11:21 am on May 15, 2015: member

Closing in favor of #6103

laanwj closed this on May 15, 2015

MarcoFalke locked this on Sep 8, 2021

Add ZMQ support. Report blocks and transactions via ZMQ. #5303