intermittent issue in interface_zmq “assert_equal(struct.unpack(’<I’, seq)[-1], self.sequence)” #20538

1. 
   MarcoFalke commented at 8:13 am on December 1, 2020: member

    0Traceback (most recent call last):
    1  File "./test/functional/test_framework/test_framework.py", line 126, in main
    2    self.run_test()
    3  File "./test/functional/interface_zmq.py", line 73, in run_test
    4    self.test_basic()
    5  File "./test/functional/interface_zmq.py", line 119, in test_basic
    6    txid = hashtx.receive()
    7  File "./test/functional/interface_zmq.py", line 41, in receive
    8    assert_equal(struct.unpack('<I', seq)[-1], self.sequence)
    9  File "./test/functional/test_framework/util.py", line 50, in assert_equal
   10    raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args))
   11AssertionError: not(1 == 0)
   

   • https://cirrus-ci.com/task/4797090746335232?command=ci#L4841
   • https://cirrus-ci.com/task/5508595028590592?command=ci#L4902
2. MarcoFalke added the label Bug on Dec 1, 2020
3. MarcoFalke added the label Tests on Dec 1, 2020
4. 
   MarcoFalke commented at 8:13 am on December 1, 2020: member

   I’ve seen this multiple times

   cc @instagibbs

5. 
   instagibbs commented at 10:06 am on December 1, 2020: member

   Which notifications are you listening to?

   On Tue, Dec 1, 2020, 4:14 PM MarcoFalke notifications@github.com wrote:

   I’ve seen this multiple times

   cc @instagibbs https://github.com/instagibbs

   — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bitcoin/bitcoin/issues/20538#issuecomment-736301192, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABMAFU7NPJSKW5Z2IL7646DSSSQUZANCNFSM4UIVKXXA .

6. 
   instagibbs commented at 12:46 pm on December 1, 2020: member

   Ah sorry, email notification confusion.

   I’ll take a look and see if I can reason about it.

7. 
   instagibbs commented at 12:32 pm on December 2, 2020: member

   Nothing fishy in the logs.

   It’s possible that this chunk of code isn’t lenient enough under certain conditions, causing the first message to get lost:

   0        # Relax so that the subscriber is ready before publishing zmq messages
   1        sleep(0.2)
   

8. 
   MarcoFalke commented at 1:42 pm on December 4, 2020: member

   Removing the sleep gives zmq.error.Again: Resource temporarily unavailable, which I’ve also seen in some test failures. Generally, a hardcoded sleep is a recipe for intermittent failures.

   Is it not possible for zmq to fully boot up (like the rpc interface) when the server is starting?

   How is this dealt with in production?

9. 
   instagibbs commented at 1:44 pm on December 4, 2020: member

   In prod you don’t “expect” any specific message so you start with whichever one you get.

   I think for tests we could just have a block making and zmq consumption loop that “syncs up” the expected message then continues onto the rest of the test?

   On Fri, Dec 4, 2020, 9:42 PM MarcoFalke notifications@github.com wrote:

   Removing the sleep gives zmq.error.Again: Resource temporarily unavailable, which I’ve also seen in some test failures. Generally, a hardcoded sleep is a recipe for intermittent failures.

   Is it not possible for zmq to fully boot up (like the rpc interface) when the server is starting?

   How is this dealt with in production?

   — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bitcoin/bitcoin/issues/20538#issuecomment-738790061, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABMAFU6TPANPRGZ55ANO5RDSTDRMDANCNFSM4UIVKXXA .

10. 
    instagibbs commented at 1:45 pm on December 4, 2020: member

    E.g., make a block, try to read off zmq messages until you get it(with timeout), if that fails with timeout do it again. Rinse repeat until you get the message.

    On Fri, Dec 4, 2020, 9:44 PM Greg Sanders gsanders87@gmail.com wrote:

    In prod you don’t “expect” any specific message so you start with whichever one you get.

    I think for tests we could just have a block making and zmq consumption loop that “syncs up” the expected message then continues onto the rest of the test?

    On Fri, Dec 4, 2020, 9:42 PM MarcoFalke notifications@github.com wrote:

    Removing the sleep gives zmq.error.Again: Resource temporarily unavailable, which I’ve also seen in some test failures. Generally, a hardcoded sleep is a recipe for intermittent failures.

    Is it not possible for zmq to fully boot up (like the rpc interface) when the server is starting?

    How is this dealt with in production?

    — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bitcoin/bitcoin/issues/20538#issuecomment-738790061, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABMAFU6TPANPRGZ55ANO5RDSTDRMDANCNFSM4UIVKXXA .

11. fanquake referenced this in commit 3734adba39 on Jan 21, 2021
12. sidhujag referenced this in commit 4dceb42b8b on Jan 21, 2021
13. MarcoFalke referenced this in commit 3c9d9d21e1 on Feb 16, 2021
14. sidhujag referenced this in commit 31ef542332 on Feb 16, 2021
15. MarcoFalke closed this on Mar 8, 2021

    ---

16. Fabcien referenced this in commit 30b874af38 on Nov 30, 2021
17. DrahtBot locked this on Aug 18, 2022

Contributors
MarcoFalke instagibbs

Labels
Bug Tests