Add --exclude option to rpc-tests.py #9766

jnewbery commented at 3:06 PM on February 15, 2017: member

Sometimes it's useful to be able to run all test except some specified tests, either because those tests take a long time to execute or because you know they're broken.

This commit adds an additional --exclude or -x option to rpc-tests to exclude a comma-separated-list of tests. Usage is:

rpc-tests.py --exclude=test1,test2,test3 or rpc-tests.py --exclude test1,test2,test3 or rpc-tests.py -x=test1,test2,test3 or rpc-tests.py -x test1,test2,test3

test names are given without the .py suffix.

This can be combined with the extended option to run the entire suite of tests, except for a single test, eg pruning.py which takes a long time:

rpc-tests.py --extended -x pruning

Note for reviewers: this requires and builds on top of #9657 . Please review only the last commit in this PR (921f6e1ac9b32cb80a83d97054b4cfdbbc290110)

[EDIT: edited commit hash]

jnewbery commented at 3:06 PM on February 15, 2017: member

@morcos

in qa/pull-tester/rpc-tests.py:None in f34e6649bf outdated

 194 | @@ -194,6 +195,12 @@ def main():
 195 |              # (for parallel running efficiency). This combined list will is no
 196 |              # longer sorted.
 197 |  
 198 | +    # Remove the test cases that the user has explicitly asked to exclude.
 199 | +    if args.exclude:
 200 | +        [ test_list.remove(exclude_test + ".py")

jtimon commented at 3:16 PM on February 15, 2017:

why use [ ] here?

I think this would be more clear:

    if args.exclude:
        for exclude_test in args.exclude.split(','):
            if exclude_test + ".py" in test_list:
                test_list.remove(exclude_test + ".py")

And it's the same number of lines.

jnewbery commented at 9:25 PM on February 15, 2017:

ha. If I write for loops I get review comments saying I should use list comprehensions, and if I write list comprehensions I get review comments saying I should use for loops :)

In this case you're probably right that the for loop is clearer.

jtimon commented at 3:18 PM on February 15, 2017: contributor

utACK f34e664 (although I left a style nit). Yay, --extended --exclude=pruning!!

MarcoFalke commented at 3:50 PM on February 15, 2017: member

Concept ACK!

fanquake added the label Tests on Feb 15, 2017

jnewbery force-pushed on Feb 15, 2017

jnewbery commented at 9:28 PM on February 15, 2017: member

I've changed the list comprehension to a for loop as suggested by @jtimon . I've also moved the if len(test_list) == 0 block lower down to catch the (unlikely) failure case where the user specifies and excludes the same test.

Add exclude option to rpc-tests.py c5784085e9

jnewbery force-pushed on Feb 20, 2017

jnewbery commented at 11:19 PM on February 20, 2017: member

rebased

in qa/pull-tester/rpc-tests.py:None in c5784085e9

 191 | @@ -198,6 +192,17 @@ def main():
 192 |              # (for parallel running efficiency). This combined list will is no
 193 |              # longer sorted.
 194 |  
 195 | +    # Remove the test cases that the user has explicitly asked to exclude.
 196 | +    if args.exclude:
 197 | +        for exclude_test in args.exclude.split(','):
 198 | +            if exclude_test + ".py" in test_list:

jtimon commented at 12:59 AM on February 21, 2017:

Don't you need the following?

            if exclude_test in test_list:
                test_list.remove(exclude_test)
            if exclude_test + ".py" in test_list:
                test_list.remove(exclude_test + ".py")

Otherwise --exclude won't work if you specify the test with including .py explicitly. Oh, sorry, the --exclude documentation explains it, never mind.

jtimon commented at 1:02 AM on February 21, 2017: contributor

re utACK c578408

MarcoFalke commented at 3:33 PM on February 23, 2017: member

utACK c5784085e9031ea1ad32a2aee897bd57a0482d24

MarcoFalke merged this on Feb 23, 2017

MarcoFalke closed this on Feb 23, 2017

MarcoFalke referenced this in commit 7146d96de3 on Feb 23, 2017

PastaPastaPasta referenced this in commit 7b6644c2b8 on Dec 28, 2018

PastaPastaPasta referenced this in commit 01593e4567 on Dec 29, 2018

PastaPastaPasta referenced this in commit 9ec8a5cd51 on Dec 29, 2018

PastaPastaPasta referenced this in commit e8eba3626f on Dec 29, 2018

PastaPastaPasta referenced this in commit 9a267ae822 on Jan 2, 2019

PastaPastaPasta referenced this in commit 09832de97d on Jan 3, 2019

PastaPastaPasta referenced this in commit 2b925f2256 on Jan 3, 2019

UdjinM6 referenced this in commit 07dcddb4ca on Jan 7, 2019

zkbot referenced this in commit 025bd44543 on Nov 21, 2020

zkbot referenced this in commit 7a0a268054 on Dec 2, 2020

zkbot referenced this in commit c8896f9907 on Dec 2, 2020

DrahtBot locked this on Sep 8, 2021