[qa] Refactor test_framework and pull tester

MarcoFalke commented at 10:42 am on April 29, 2016: member

log to stdout
increase range for p2p and rpc ports
UPPERCASE_CONSTANTS
Stop nodes on CTRL+C

MarcoFalke added the label Refactoring on Apr 29, 2016

MarcoFalke added the label Tests on Apr 29, 2016

MarcoFalke force-pushed on Apr 29, 2016

MarcoFalke force-pushed on Apr 30, 2016

MarcoFalke commented at 1:15 pm on April 30, 2016: member

Added a commit to properly stop other nodes, even when one fails to stop
Added a commit to update the Readme

MarcoFalke force-pushed on May 1, 2016

MarcoFalke commented at 8:27 am on May 6, 2016: member

Anyone interested in looking at the diff here? (It is mostly move-only and cleaning up the logic/ifelses)

It is already tested as part of #7972 but I’d prefer to have at least one read the patch.

in qa/rpc-tests/test_framework/util.py: in fa6cd636c0 outdated

14@@ -15,6 +15,7 @@
15 from base64 import b64encode
16 from decimal import Decimal, ROUND_DOWN
17 import json
18+import httplib

MarcoFalke commented at 9:52 am on May 6, 2016:

Needs bump: #7972 (comment)

MarcoFalke commented at 10:00 am on May 6, 2016:

@fanquake @paveljanik done in another commit.

MarcoFalke commented at 10:45 am on May 6, 2016:

Rebased instead :stuck_out_tongue_winking_eye:

in qa/rpc-tests/test_framework/util.py: in fa6cd636c0 outdated

27@@ -27,6 +28,13 @@
28 
29 COVERAGE_DIR = None
30 
31+# The maximum number of nodes a single test can spawn
32+MAX_NODES = 8
33+# Don't assign rpc or p2p ports lower than this
34+PORT_MIN = 11000
35+# The number of ports to "reserve" for p2p and rpc, each
36+PORT_RANGE = 5000

MarcoFalke commented at 9:53 am on May 6, 2016:

@theuni I am assuming this works fine on travis?

paveljanik commented at 10:34 am on May 6, 2016: contributor

README.md should state install python3-zmq.

[qa] Refactor test_framework and pull tester

* log to stdout
* increase range for p2p and rpc ports
* UPPERCASE_CONSTANTS
* Stop nodes on CTRL+C

fabbf6bd62

[qa] Update README.md 2222dae6e3

[qa] Stop other nodes, even when one fails to stop fafb33cdef

MarcoFalke force-pushed on May 6, 2016

MarcoFalke commented at 10:50 am on May 6, 2016: member

@paveljanik Rebased, so the other doc changes are included as well.

paveljanik commented at 11:45 am on May 6, 2016: contributor

I have a lot of these during tests, but not only in the PR - also applies to current master:

0Running testscript wallet.py ...
1Initializing test directory /var/folders/65/fn0h49r55k7779vg1b_h461r0000gn/T/testbv4onukz
2Unexpected exception caught during testing: timeout('timed out',)
3...
4Running testscript abandonconflict.py ...
5Initializing test directory /var/folders/65/fn0h49r55k7779vg1b_h461r0000gn/T/test4arfwesr
6Unexpected exception caught during testing: timeout('timed out',)

MarcoFalke commented at 12:48 pm on May 6, 2016: member

It’s just refactoring. Ideally, behavior should stay the same.

Also, I can’t reproduce. Mind to open a issue for the timeouts?

in qa/pull-tester/rpc-tests.py: in fabbf6bd62 outdated

171+            if t in opts or re.sub(".py$", "", t) in opts:
172+                test_list.append(t)
173+
174+    if print_help:
175+        # Help should be the same for all scripts, so just
176+        # call the first and exit

sdaftuar commented at 7:01 pm on May 6, 2016:

comment ultranit: the help is not actually the same for all scripts, as some tests take custom (optional) arguments.

MarcoFalke commented at 9:50 am on May 7, 2016:

This is true, but the previous behavior was to only show the help text of the first script and then exit(0). If someone wants to change the behavior, I’d propose a separate pull. I am happy to change the comment in this pull, though.

sdaftuar commented at 7:28 pm on May 6, 2016: member

utACK (feel free to ignore my ultranit)

Not sure this needs to be addressed in this PR, but for supporting multiple tests running in parallel, I think it might make sense to add support to the test framework for specifying a range of ports to use for the p2p and rpc ports on the command line, which the rpc_tests.py script could use. The logic now of hoping that pid’s don’t collide modulo some big number (which is now bigger after this PR) is not sufficiently robust in my opinion. Doesn’t need to be fixed here, just mentioning it since this PR is touching those functions already.

[qa] pull-tester: Adjust comment fad336648c

MarcoFalke commented at 12:11 pm on May 8, 2016: member

Added a commit to fix @sdaftuar’s nit, hopefully. In regard to the ports: I am wondering if it is sufficient to pass in a int seed (default=pid) instead of trying to parse ranges from the command line.

sdaftuar commented at 12:49 pm on May 8, 2016: member

@marcofalke Yeah that approach of a single int that defaults to the pid sounds reasonable to me. Comment looks better too, thanks.

MarcoFalke merged this on May 9, 2016

MarcoFalke closed this on May 9, 2016

MarcoFalke referenced this in commit 4e14afe42f on May 9, 2016

MarcoFalke deleted the branch on May 9, 2016

codablock referenced this in commit 760173e2f0 on Sep 16, 2017

codablock referenced this in commit 55ad99d8ff on Sep 19, 2017

codablock referenced this in commit 8bafa06b75 on Dec 21, 2017

zkbot referenced this in commit 025bd44543 on Nov 21, 2020

zkbot referenced this in commit 7a0a268054 on Dec 2, 2020

zkbot referenced this in commit c8896f9907 on Dec 2, 2020

MarcoFalke locked this on Sep 8, 2021

[qa] Refactor test_framework and pull tester #7971