doc: Improve ZMQ documentation #23471

pull mbildwic wants to merge 1 commits into bitcoin:master from mbildwic:doc_add_zmq_desc changing 1 files +23 −4
  1. mbildwic commented at 4:10 am on November 9, 2021: contributor

    This PR intends to clarify:

    . when ZMQ notifications occur . the message structure of each topic

    Closes #23452#issue-1046579585

  2. fanquake added the label Docs on Nov 9, 2021
  3. in doc/zmq.md:115 in 7ba107de84 outdated 
    111+
112+    hashtx <32-byte transaction hash>
113+
114+`rawblock`: notifies when a the block tip is updated. The body of the message is the raw block in hexadecimal format.
115+
116+`hashblock`: notifies when a the block tip is updated. The body of the message is structured in the following way:
    shaavan commented at 1:54 pm on November 10, 2021:

    nit:

    0`rawblock`: notifies when the block tip is updated. The body of the message is the raw block in hexadecimal format.
1
2`hashblock`: notifies when the block tip is updated. The body of the message is structured in the following way:
    mbildwic commented at 3:26 am on November 12, 2021:
    Done.
  4. shaavan commented at 1:55 pm on November 10, 2021: contributor

    Concept ACK

    The description looks good and helps the reader understand the topics in better detail. I agree with these changes. However, there is a minor nit that needs to be addressed.

  5. in doc/zmq.md:107 in 7ba107de84 outdated 
    103@@ -102,7 +104,17 @@ is structured as the following based on the type of message:
104 
105 Where the 8-byte uints correspond to the mempool sequence number.
106 
107-These options can also be provided in bitcoin.conf.
108+`rawtx`: notifies when a transaction is added to mempool. The body of the message is the raw transaction in hexadecimal format.
    0xB10C commented at 9:33 am on November 11, 2021: 
    0`rawtx`: notifies when a transaction is added to mempool. The body of the message is the serialized transaction.
    mbildwic commented at 3:26 am on November 12, 2021:
    Done
  6. in doc/zmq.md:111 in 7ba107de84 outdated 
    103@@ -102,7 +104,17 @@ is structured as the following based on the type of message:
104 
105 Where the 8-byte uints correspond to the mempool sequence number.
106 
107-These options can also be provided in bitcoin.conf.
108+`rawtx`: notifies when a transaction is added to mempool. The body of the message is the raw transaction in hexadecimal format.
109+
110+`hashtx`: notifies when a transaction is added to mempool. The body of the message is structured in the following way:
111+
112+    hashtx <32-byte transaction hash>
    0xB10C commented at 9:45 am on November 11, 2021:

    hashtx is not contained in the payload. The messages are ZMQ multipart messages with three parts. The first part is the topic (here hashtx), the second part is the 32-byte transaction hash, and the last part is a sequence number (to detect lost messages; this is already documented).

    I’d personally prefer a documentation similar to the following:

    0| hashtx | <32-byte transaction hash in Little Endian> | <uint32 sequence number in Little Endian>
    0xB10C commented at 9:48 am on November 11, 2021:
    I think it would be good to explicitly note that the 32-byte hashes are in Little Endian and not in the Big Endian format that the RPC interface and e.g. block explorers use to display transaction and block hashes.
    mbildwic commented at 3:26 am on November 12, 2021:
    Done.
  7. in doc/zmq.md:114 in 7ba107de84 outdated 
    108+`rawtx`: notifies when a transaction is added to mempool. The body of the message is the raw transaction in hexadecimal format.
109+
110+`hashtx`: notifies when a transaction is added to mempool. The body of the message is structured in the following way:
111+
112+    hashtx <32-byte transaction hash>
113+
    0xB10C commented at 9:51 am on November 11, 2021:
    Both rawtx and hashtx notify about all transactions. Mempool and transactions in blocks. This means it could happen that a transaction is published multiple times. First when it enters the mempool and then again in each block that includes it. This should be documented.
    mbildwic commented at 3:26 am on November 12, 2021:
    Done.
  8. in doc/zmq.md:113 in 7ba107de84 outdated 
    109+
110+`hashtx`: notifies when a transaction is added to mempool. The body of the message is structured in the following way:
111+
112+    hashtx <32-byte transaction hash>
113+
114+`rawblock`: notifies when a the block tip is updated. The body of the message is the raw block in hexadecimal format.
    0xB10C commented at 9:52 am on November 11, 2021: 
    0`rawblock`: notifies when a the chain tip is updated. The body of the message is the serialized block.
    mbildwic commented at 7:06 pm on November 12, 2021:
    Done.
  9. in doc/zmq.md:117 in 7ba107de84 outdated 
    113+
114+`rawblock`: notifies when a the block tip is updated. The body of the message is the raw block in hexadecimal format.
115+
116+`hashblock`: notifies when a the block tip is updated. The body of the message is structured in the following way:
117+
118+    hashblock <32-byte block hash>
    0xB10C commented at 9:54 am on November 11, 2021:

    see comments for hashtx above (remove hashblock and mention Little Endianess)

    #23471 (review) #23471 (review)

    mbildwic commented at 3:27 am on November 12, 2021:
    Done.
  10. 0xB10C changes_requested
  11. 0xB10C commented at 9:55 am on November 11, 2021: member

    Concept ACK on improving the documentation. Thanks for picking this up!

    I left a few comments below that I think should be addressed.

  12. mbildwic force-pushed on Nov 12, 2021
  13. mbildwic renamed this:
    doc: Add more information about ZMQ topics
    doc: Improve ZMQ documentation
    on Nov 12, 2021
  14. mbildwic commented at 3:27 am on November 12, 2021: contributor
    Thanks for review @0xB10C and @shaavan . I addressed your suggestions.
  15. in doc/zmq.md:107 in 079576e206 outdated 
    103@@ -102,7 +104,20 @@ is structured as the following based on the type of message:
104 
105 Where the 8-byte uints correspond to the mempool sequence number.
106 
107-These options can also be provided in bitcoin.conf.
108+`rawtx`: notifies about all transactions, whether when they are added to mempool or when a new block arrives. This means it could happen that a transaction is published multiple times. First when it enters the mempool and then again in each block that includes it. The body of the message is the serialized transaction.
    promag commented at 9:33 am on November 12, 2021:
    nit, -zmqpubrawtx instead of rawtx? (same thing below)
    mbildwic commented at 7:15 pm on November 12, 2021:

    I am not sure. I think that rawtx is the ZMQ topic that the subscriber receives notification about and -zmqpubrawtx is the argument used to set the network address of this topic in node daemon (bitcoind).

    If that is not the case, I can change them to -zmq*.

    0xB10C commented at 11:48 am on November 16, 2021:
    I’d say: leave it as rawtx (the topic). This sections starts in line 96 with The topics are:.
    0xB10C commented at 12:02 pm on November 16, 2021:

    We should mention that this is a multipart message too.

    0`rawtx`: Notifies about all transactions, both when they are added to mempool or when a new block arrives. This means a transaction could be published multiple times. First, when it enters the mempool and then again in each block that includes it.The messages are ZMQ multipart messages with three parts. The first part is the topic (`rawtx`), the second part is the serialized transaction, and the last part is a sequence number (to detect lost messages).
1
2| rawtx | <serialized transaction> | <uint32 sequence number in Little Endian>
  16. promag commented at 9:34 am on November 12, 2021: member
    ACK 079576e2060f9702d69657b5b0c0a286d72cdcb7
  17. shaavan approved
  18. shaavan commented at 11:36 am on November 12, 2021: contributor

    ACK 079576e2060f9702d69657b5b0c0a286d72cdcb7

    Changes since my last review:

    • The definitions of topics are expanded upon. They are now much more detailed and help to understand these terms better.
    • A note is added explaining the difference in the representation of 32-byte hashes and RPC interface and block explorer.

    There are no critical grammatical mistakes in the updated PR, and I find the expanded description to be logically sound and easy to understand. Great Work @mbildwic!

  19. mbildwic force-pushed on Nov 12, 2021
  20. tylerchambers commented at 10:48 pm on November 15, 2021: contributor
    Thanks! This clears things up.
  21. in doc/zmq.md:109 in 9bf2c75493 outdated 
    103@@ -102,7 +104,20 @@ is structured as the following based on the type of message:
104 
105 Where the 8-byte uints correspond to the mempool sequence number.
106 
107-These options can also be provided in bitcoin.conf.
108+`rawtx`: notifies about all transactions, whether when they are added to mempool or when a new block arrives. This means it could happen that a transaction is published multiple times. First when it enters the mempool and then again in each block that includes it. The body of the message is the serialized transaction.
109+
110+`hashtx`: is triggered in the same way as in the previous topic, when any transaction is identified, either in the mempool or in a new block. The messages are ZMQ multipart messages with three parts. The first part is the topic (`hashtx`), the second part is the 32-byte transaction hash, and the last part is a sequence number (to detect lost messages).
    0xB10C commented at 12:00 pm on November 16, 2021:

    I think it’s fine to just repeat the first part of the rawtx documentation here. If the ordering is changed or a new topic is added, the ‘previous topic’ might change.

    0`hashtx`: Notifies about all transactions, both when they are added to mempool or when a new block arrives. This means a transaction could be published multiple times. First, when it enters the mempool and then again in each block that includes it. The messages are ZMQ multipart messages with three parts. The first part is the topic (`hashtx`), the second part is the 32-byte transaction hash, and the last part is a sequence number (to detect lost messages).
  22. in doc/zmq.md:116 in 9bf2c75493 outdated 
    112+    | hashtx | <32-byte transaction hash in Little Endian> | <uint32 sequence number in Little Endian>
113+
114+
115+`rawblock`: is triggered when the chain tip is updated. The body of the message is the serialized block.
116+
117+`hashblock`: is triggered in the same way as in the previous topic. The messages are also ZMQ multipart messages with three parts. The first part is the topic (`hashblock`), the second part is the 32-byte block hash, and the last part is a sequence number (to detect lost messages).
    0xB10C commented at 12:05 pm on November 16, 2021: 
    0`hashblock`: Notifies when the chain tip is updated. Messages are ZMQ multipart messages with three parts. The first part is the topic (`hashblock`), the second part is the 32-byte block hash, and the last part is a sequence number (to detect lost messages).
  23. in doc/zmq.md:114 in 9bf2c75493 outdated 
    110+`hashtx`: is triggered in the same way as in the previous topic, when any transaction is identified, either in the mempool or in a new block. The messages are ZMQ multipart messages with three parts. The first part is the topic (`hashtx`), the second part is the 32-byte transaction hash, and the last part is a sequence number (to detect lost messages).
111+
112+    | hashtx | <32-byte transaction hash in Little Endian> | <uint32 sequence number in Little Endian>
113+
114+
115+`rawblock`: is triggered when the chain tip is updated. The body of the message is the serialized block.
    0xB10C commented at 12:07 pm on November 16, 2021: 
    0`rawblock`: Notifies when the chain tip is updated. Messages are ZMQ multipart messages with three parts. The first part is the topic (`rawblock`), the second part is the serialized block, and the last part is a sequence number (to detect 
1lost messages).
2
3| rawblock | <serialized block> | <uint32 sequence number in Little Endian>
  24. mbildwic force-pushed on Nov 16, 2021
  25. mbildwic force-pushed on Nov 16, 2021
  26. shaavan approved
  27. shaavan commented at 8:37 am on November 20, 2021: contributor

    reACK 8ce9f74ae944f163cb1bbd030e062e2ac47d0cb2

    Changes since my last review:

    • Text explaining that the topics are ZMQ multipart has been added for all four topics now.
    • Proper explanation of the trigger event has been added for hashtx and hashblock topics. This is done to make the description more robust if there is an ordering change of topics in the future.
    • String describing the syntax of rawtx and rawblock is added, along with the previously added syntax of hashtx and hashblock.

    I agree with the updated changes.

    However, I think the description has become too lengthy, with a lot of repeated content in succession. I don’t exactly know how to tackle this. Maybe bolding or italicizing the main difference between topics might help, but I don’t know if that would be the right approach.

  28. 0xB10C commented at 2:02 pm on November 23, 2021: member

    ACK 8ce9f74ae944f163cb1bbd030e062e2ac47d0cb2.

    If more people agree that bold or italic text would help (as mentioned in #23471#pullrequestreview-811812308), then please add it. I don’t mind but don’t think it’s crucial.

  29. in doc/zmq.md:107 in 8ce9f74ae9 outdated 
    103@@ -102,7 +104,24 @@ is structured as the following based on the type of message:
104 
105 Where the 8-byte uints correspond to the mempool sequence number.
106 
107-These options can also be provided in bitcoin.conf.
108+`rawtx`: Notifies about all transactions, both when they are added to mempool or when a new block arrives. This means a transaction could be published multiple times. First, when it enters the mempool and then again in each block that includes it.The messages are ZMQ multipart messages with three parts. The first part is the topic (`rawtx`), the second part is the serialized transaction, and the last part is a sequence number (representing the data size to detect lost messages).
    theStack commented at 6:26 pm on November 28, 2021:

    nit: missing space between sentences

    0`rawtx`: Notifies about all transactions, both when they are added to mempool or when a new block arrives. This means a transaction could be published multiple times. First, when it enters the mempool and then again in each block that includes it. The messages are ZMQ multipart messages with three parts. The first part is the topic (`rawtx`), the second part is the serialized transaction, and the last part is a sequence number (representing the data size to detect lost messages).
    laanwj commented at 12:46 pm on November 29, 2021:

    (representing the data size to detect lost messages)

    I’m not sure “data size” clarifies much here. I would associate that with sizes in bytes. Maybe “message count”?

  30. theStack commented at 6:27 pm on November 28, 2021: member
    Concept ACK, thanks for working on this!
  31. doc: Improve ZMQ documentation 
    Co-authored-by: 0xB10C <19157360+0xB10C@users.noreply.github.com>
Co-authored-by: Sebastian Falbesoner <sebastian.falbesoner@gmail.com>
    9544ab60ce
  32. mbildwic force-pushed on Dec 6, 2021
  33. theStack approved
  34. theStack commented at 10:21 pm on December 13, 2021: member

    This PR adds solid clarifications about the ZMQ topics that will be very helpful for all potential users of the ZMQ interface – this would have also saved me quite some time in the past. Kudos for expicitely mentioning the fact that rawtx/hashtx notifications appear both when added to the mempool or in new blocks. LGTM!

    ACK 9544ab60ce212a29306f021c2cc6721c34485b54

  35. MarcoFalke merged this on Dec 14, 2021
  36. MarcoFalke closed this on Dec 14, 2021
  37. sidhujag referenced this in commit c3aa7181ae on Dec 14, 2021
  38. DrahtBot locked this on Dec 14, 2022


mbildwic shaavan 0xB10C promag tylerchambers theStack laanwj

Labels
Docs

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-12-18 21:12 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me