[test] Allow functional test cases to be skipped #10053

Currently, functional test cases can either pass or fail. There are occasions when it is helpful to skip tests, for example if the system they are running on does not meet the requirements for the test. The rest of the test suite can run without being marked as a failure.

This PR adds framework for tests to skip if their requirements aren't met.

This PR also updates rpcbind_test.py to skip if it detects that the system does not meet requirements (linux, ipv4 non-loopback address, ipv6 support).

This is required to have travis nightly cron jobs run without failure #10052.

utACK 0a6781dbc728abdd294fc199a6522caf592cbe56

On Wed, Mar 22, 2017 at 4:14 PM, John Newbery notifications@github.com wrote:

Currently, functional test cases can either pass or fail. There are occasions when it is helpful to skip tests, for example if the system they are running on does not meet the requirements for the test. The rest of the test suite can run without being marked as a failure.

This PR adds framework for tests to skip if their requirements aren't met.

This PR also updates rpcbind_test.py to skip if it detects that the system does not meet requirements (linux, ipv4 non-loopback address, ipv6 support).

This is required to have travis nightly cron jobs run without failure #10052.

---

You can view, comment on, or merge this pull request online at:

#10053

Commit Summary

Allow test cases to be skipped Skip rpcbind_test if OS/network requirements are not met.

File Changes

M test/functional/rpcbind_test.py (18) M test/functional/test_framework/test_framework.py (9) M test/functional/test_runner.py (26)

Patch Links:

#10053.patch #10053.diff

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

Tested 0a6781d on OS X (the test cannot run). Current behaviour:

rpcbind_test.py:
2017-03-23 00:01:56.990000 TestFramework (INFO): Initializing test directory /var/folders/rb/jqqd25w926s5429rf58_zsv00000gn/T/test_guv59zt/18
2017-03-23 00:01:56.990000 TestFramework (ERROR): Assertion failed
Traceback (most recent call last):
  File "/Users/xxx/Github/bitcoin/test/functional/test_framework/test_framework.py", line 148, in main
    self.run_test()
  File "/Users/xxx/github/bitcoin/test/functional/rpcbind_test.py", line 55, in run_test
    assert(sys.platform.startswith('linux'))
AssertionError
2017-03-23 00:01:56.991000 TestFramework (INFO): Stopping nodes

stderr:
Traceback (most recent call last):
  File "/Users/xxx/github/bitcoin/test/functional/rpcbind_test.py", line 98, in <module>
    RPCBindTest().main()
  File "/Users/xxx/Github/bitcoin/test/functional/test_framework/test_framework.py", line 163, in main
    stop_nodes(self.nodes)
  File "/Users/xxx/Github/bitcoin/test/functional/test_framework/util.py", line 380, in stop_nodes
    for i, node in enumerate(nodes):
TypeError: 'NoneType' object is not iterable

Pass: False, Duration: 0 s

TEST            | PASSED | DURATION

rpcbind_test.py | False  | 0 s

ALL             | False  | 0 s (accumulated)

Runtime: 0 s

This PR:

bash-3.2$ test/functional/test_runner.py rpcbind_test

rpcbind_test.py:
2017-03-22 23:59:28.910000 TestFramework (INFO): Initializing test directory /var/folders/rb/jqqd25w926s5429rf58_zsv00000gn/T/testchxojvbm/69
2017-03-22 23:59:28.911000 TestFramework (WARNING): This test can only be run on linux. Skipping test.

Status: Skipped, Duration: 0 s

TEST            | PASSED  | DURATION

rpcbind_test.py | Skipped | 0 s

ALL             | True    | 0 s (accumulated)

Runtime: 0 s

Please squash the fixup.

squashed https://github.com/jnewbery/bitcoin/tree/pr10053.1 -> https://github.com/jnewbery/bitcoin/tree/pr10053.2