ZMQ example using python3 and asyncio #9485

pull mcelrath wants to merge 4 commits into bitcoin:master from AtakamaLLC:python3zmq changing 2 files +158 −28
  1. mcelrath commented at 11:28 PM on January 6, 2017: none

    As is says on the tin. This example is functionally identical to the polling 2.7 zmq example, but uses the fancy new non-blocking asyncio of python 3.5.

  2. fanquake added the label RPC/REST/ZMQ on Jan 7, 2017
  3. jonasschnelli commented at 12:45 PM on January 7, 2017: contributor

    What about "rewriting" the current python2 zmq example instead of creating a new one for python3? I don't think we need a python2 example.

  4. mcelrath commented at 1:18 PM on January 7, 2017: none

    For nearly a decade people have been "avoiding" upgrading to python 3. I don't know what's wrong with these people. They should probably see a psychologist and get a friendly lobotomy.

    Obviously I think python3 and asyncio is the "right way" to do anything. It has all the advantages of javascript's asynchronicity and event loop while not being a totally brain-dead language.

    TL;DR I'd be very happy to replace the current example with this python3 one...but people who need friendly lobotomies will complain.

  5. MarcoFalke commented at 2:02 PM on January 7, 2017: member

    Maybe mention that the old version can still be retrieved from git history?

  6. fanquake commented at 11:43 AM on January 8, 2017: member

    Agree with re-writing to use Python3, I don't think we need to keep examples for both in the repo.

  7. venzen commented at 12:22 PM on January 8, 2017: none

    Agreed that a Python3 version is preferable. Python2 support for ZMQ libraries > 3 is uncertain and attempts to script for both Py2 and Py3 will run into incompatibilities of hex encoding, as well as pickle protocol limitations in Py2's pyzmq module.

  8. mcelrath commented at 10:27 PM on January 9, 2017: none

    Since everyone here likes python3, I replaced the python2.7 file with the python3 example, and included a link to the python2.7 example in the git history. I also removed the Polling wrapper class from the example, in favor of more direct use of asyncio primitives.

    Again it's functionally identical to the original example.

  9. in contrib/zmq/zmq_sub.py:None in e2773a2a7f outdated 
      38 | +        self.zmqSubSocket.setsockopt_string(zmq.SUBSCRIBE, "hashtx")
  39 | +        self.zmqSubSocket.setsockopt_string(zmq.SUBSCRIBE, "rawblock")
  40 | +        self.zmqSubSocket.setsockopt_string(zmq.SUBSCRIBE, "rawtx")
  41 | +        self.zmqSubSocket.connect("tcp://127.0.0.1:%i" % port)
  42 | +
  43 | +    async def handle(self) :
    venzen commented at 3:01 AM on January 10, 2017:

    This function definition syntax is specific to Python 3.5 and throws an exception in Python 3.4. Perhaps a from __future__ import option exists that will allow compatibility with all Python 3 minor versions?

    mcelrath commented at 3:38 AM on January 10, 2017:

    Asyncio was only introduced in 3.5 including these syntax enhancements and was "provisional". It's now final that 3.6 was released a week ago, and backported (as final) to 3.5 point releases. 3.4 doesn't have it and never will. But, it doesn't have anywhere near the user base of 2.7 and I don't think is worth supporting long term.

    I tried to install 3.4 while testing this, it's not even provided anymore by Ubuntu.

    http://stackoverflow.com/questions/30191556/coroutine-in-python-between-3-4-and-3-5-how-can-i-keep-backwords-compatibility

    mcelrath commented at 3:39 AM on January 10, 2017:

    FYI the "yield from" syntax of 3.4 has been entirely deprecated.

    venzen commented at 9:28 AM on January 10, 2017:

    Useful to know and thanks for the info & link @mcelrath

    Python 3.5 will undoubtedly become the de facto standard, as you imply. In providing example scripts it seems reasonable that Bitcoin cater to the common denominator. It's probably safe to assume that most nodes out there are not running on the latest OS versions and that Ubuntu 14.04 and even 12.04 (in some cases) are the majority. From this perspective catering for Python 3.4 is useful since it is both repository standard and supported.

    Your 3.5 approach using asyncio is great. My comments are really just thinking about ease of use and compatibility, hence the suggestion to cater for 3.4 and the inevitable 3.5 future

    benma commented at 12:04 AM on February 23, 2017:

    Ubuntu 14.04 LTS is supported until April 2019, and currently ships Python 3.4.3, so I also think it makes sense to provide examples compatible with Python 3.4.

  10. venzen commented at 3:08 AM on January 10, 2017: none

    Python 3.4 seems to be widely used while Python 3.5 is still pushing out the boat.

    Deprecating Python 2 in favor of version 3 has many benefits, yet, thinking about general usability, it would perhaps make sense to keep this script's functionality dependent on the zmq module and avoid adding asyncio dependency? Don't get me wrong - the use of asyncio in this script is clean and efficient - but it adds a dependency; an extra layer; and the need for the user (seeking a ZMQ example) to be familiar with asyncio (and its usage in Python 3.5).

    I suggest having two example scripts: a vanilla ZMQ SUBscriber compatible with Python 3.4 (and based on the previous 2.7 script) as well as the current (3.5) script using asyncio and compatible with Python 3.4, if possible.

  11. laanwj commented at 2:16 PM on January 10, 2017: member

    Bah, I think we should support 3.4 for the forseeable future. I don't care about 2.x compatibility.

    Can't we make asyncio usage dependent on Python version or something? Is it hard to make a fallback, to support both, use the fancy new i/o system on 3.5+?

  12. mcelrath commented at 3:58 PM on January 10, 2017: none

    See I told ya this would result in a python version fight...I just didn't anticipate anyone wanting 3.4! :-P

    Python 3.4 was last released in Ubuntu in version 14.04. It has been excluded in 14.10, 15.04, 15.10, 16.04 LTS, and 16.10. It's ancient history. The python project itself has designated 3.4 as "security fixes only" and is no longer producing binaries for it.

    The syntax change is, unfortunately, not backwards compatible, and results in a syntax error on 3.4. Consequently there is no from __future__ for it. It took the python guys some time to settle on a reasonable syntax, adding the async keyword instead of the @coroutine decorator, and yield from.

    I literally don't have any machines at my disposal with python3.4 on them. I added a second example which should work on both python 3.4 and 3.5. Could one of you 3.4 aficionados test it please?

  13. laanwj commented at 1:04 PM on January 11, 2017: member

    Concept ACK, cannot test this right now, may do so later (I do have an Ubuntu Trusty/14.04 instance to test, note that we also use this version of Ubuntu on Travis and to build gitian releases).

  14. MarcoFalke commented at 10:07 AM on January 18, 2017: member

    Needs rebase.

    Maybe also squash the first two commits?

  15. mcelrath commented at 3:56 PM on January 18, 2017: none

    Rebased. I usually squash when merging but I can squash ahead of time if desired.

  16. in contrib/zmq/zmq_sub.py:None in d87c80a8fb outdated 
       0 | @@ -1,43 +1,70 @@
   1 | -#!/usr/bin/env python2
   2 | +#!/usr/bin/env python3
   3 |  # Copyright (c) 2014-2016 The Bitcoin Core developers
   4 |  # Distributed under the MIT software license, see the accompanying
   5 |  # file COPYING or http://www.opensource.org/licenses/mit-license.php.
   6 |
    jnewbery commented at 4:25 PM on January 18, 2017:

    nit: I'd like to see a module-level docstring describing what this does (which can include the comment block below about the pre-2.7 version)

  17. in contrib/zmq/zmq_sub.py:None in d87c80a8fb outdated 
       5 |  # file COPYING or http://www.opensource.org/licenses/mit-license.php.
   6 |  
   7 | +# A blocking example using python 2.7 can be obtained from the git history:
   8 | +# https://github.com/bitcoin/bitcoin/blob/37a7fe9e440b83e2364d5498931253937abe9294/contrib/zmq/zmq_sub.py
   9 | +
  10 | +import array
    jnewbery commented at 4:25 PM on January 18, 2017:

    nit: I don't think this import array is actually used?

  18. in contrib/zmq/zmq_sub.py:None in d87c80a8fb outdated 
       8 | +# https://github.com/bitcoin/bitcoin/blob/37a7fe9e440b83e2364d5498931253937abe9294/contrib/zmq/zmq_sub.py
   9 | +
  10 | +import array
  11 |  import binascii
  12 | -import zmq
  13 | +import asyncio, zmq, zmq.asyncio
    jnewbery commented at 4:26 PM on January 18, 2017:

    nit: prefer imports to be on individual line (PEP8 E401), and listed individually.

  19. jnewbery approved
  20. jnewbery commented at 4:30 PM on January 18, 2017: member

    Looks great. I've tested this with Python3.5 and it works really nicely.

    I've added 3 μ-nits.

    Personally I don't agree that we should have two almost identical versions of this in the repository, especially since the /contrib code is not run as part of the build process. If someone really wants to run this on Python3.4, they should make the syntax changes themselves.

  21. mcelrath commented at 5:16 PM on January 18, 2017: none

    @jnewbery Well the point of /contrib is to provide examples, so nothing in there is truly necessary! Devs should be able to figure it all out themselves! ;-) The necessary syntax change between 3.4 and 3.5 is somewhat tedious to discover and understand. So I do think having both (and even the 2.7 version) is valuable.

  22. jnewbery commented at 6:10 PM on January 18, 2017: member

    @mcelrath Sure. I'm not massively concerned about what goes on in /contrib. Having two versions of zmq_sub.py just makes this marginally less maintainable, since any changes to the code would need to be repeated and tested in two places against two different versions of python. Perhaps adding a comment to the top of the module with a link to the python docs about asyncio syntax differences is a better way to go.

    Anyway, like I said, I'm not hugely concerned either way.

    Tested ACK.

  23. ZMQ example using python3 and asyncio 5ea5368b3a
  24. Rewrite to not use Polling wrapper for asyncio, link to python2.7 example 5406d51067
  25. Add python version checks and 3.4 example 4bb7d1bc4d
  26. Adddress nits, use asyncio signal handling, create_task b471daf85b
  27. jnewbery commented at 10:25 PM on January 19, 2017: member

    Looks great. Tested ACK https://github.com/bitcoin/bitcoin/pull/9485/commits/b471daf85b07d0ca072a8d7ec482a25401a67399

  28. jmcorgan commented at 6:12 AM on February 20, 2017: contributor

    ACK

  29. laanwj merged this on Feb 21, 2017
  30. laanwj closed this on Feb 21, 2017
  31. laanwj referenced this in commit 61a640ea97 on Feb 21, 2017
  32. PastaPastaPasta referenced this in commit 58194c3a26 on Dec 28, 2018
  33. PastaPastaPasta referenced this in commit 12feafc77a on Dec 29, 2018
  34. PastaPastaPasta referenced this in commit 97dcfe32bb on Dec 29, 2018
  35. UdjinM6 referenced this in commit a49f4123e5 on Jan 3, 2019
  36. PastaPastaPasta referenced this in commit 946318b5da on Jan 3, 2019
  37. PastaPastaPasta referenced this in commit 6a126d8eb1 on Jan 3, 2019
  38. PastaPastaPasta referenced this in commit 44938e8c08 on Jan 3, 2019
  39. PastaPastaPasta referenced this in commit cab434bb50 on Jan 5, 2019
  40. PastaPastaPasta referenced this in commit 225ffbbb2b on Jan 7, 2019
  41. PastaPastaPasta referenced this in commit d1d7afc626 on Jan 25, 2019
  42. PastaPastaPasta referenced this in commit 5d53825775 on Jan 29, 2019
  43. CryptoCentric referenced this in commit c632f583ae on Jul 11, 2019
  44. MarcoFalke locked this on Sep 8, 2021
Contributors
mcelrathjonasschnelliMarcoFalkefanquakevenzenbenmalaanwjjnewberyjmcorgan
Labels
RPC/REST/ZMQ
github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-04-13 15:15 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me