[qa] Add --failfast option to functional test runner #13105

pull jamesob wants to merge 2 commits into bitcoin:master from jamesob:2018-04-testrunner-failfast changing 2 files +35 −3

jamesob commented at 4:57 PM on April 27, 2018: member

Add the option (--failfast) to stop the functional test runner's execution when it encounters the first failure.

Also cleans up run_test's arguments list (no more mutable default for args) and call site.

in test/functional/test_runner.py:300 in 719768dca4 outdated

 296 | +        combined_logs_len=args.combinedlogslen,
 297 | +        failfast=args.failfast,
 298 | +    )
 299 | +
 300 | +def run_tests(test_list, src_dir, build_dir, tmpdir,
 301 | +              jobs=1, enable_coverage=False, args=None, combined_logs_len=0, failfast=False):

MarcoFalke commented at 5:02 PM on April 27, 2018:

nit: Can keep this in one line, so that it doesn't break when you e.g. grep for run_tests.

jamesob force-pushed on Apr 27, 2018
jamesob force-pushed on Apr 27, 2018
jamesob force-pushed on Apr 27, 2018
jamesob commented at 5:28 PM on April 27, 2018: member

@MarcoFalke thanks for the look. I've addressed the formatting nit and the process cleanup issue you pointed out.
theuni commented at 6:13 PM on April 27, 2018: member

Concept ACK. Any reason not to enable this for Travis?
jamesob force-pushed on Apr 27, 2018
jamesob commented at 6:17 PM on April 27, 2018: member

@theuni good point; I guess that'd encourage a shift in compute consumption onto developers who write buggy code vs. scarce Travis resources. Will add.

Add --failfast option to functional test runner

Also cleans up run_test's arguments list (no more mutable default for `args`)
and call site.

bf720c1460

Use --failfast when running functional tests on Travis 58f9a0a002
jamesob force-pushed on Apr 27, 2018
fanquake added the label Tests on Apr 27, 2018
laanwj commented at 6:21 AM on April 28, 2018: member

Concept ACK
practicalswift commented at 9:35 AM on April 28, 2018: contributor

Concept ACK

Very nice!
laanwj commented at 1:25 PM on April 30, 2018: member

Tested ACK 58f9a0a002ce45597ce2ed9e1902fb6ac608f014 I made the first test that runs here (wallet_hd.py) fail. Without --failfast it continues after the error. With --failfast it stopped immediately.
laanwj merged this on Apr 30, 2018
laanwj closed this on Apr 30, 2018
laanwj referenced this in commit 9e9b48df72 on Apr 30, 2018
codablock referenced this in commit 4db0c1b6f6 on Jan 7, 2020
codablock referenced this in commit 743bd6c39e on Jan 7, 2020
codablock referenced this in commit 26a6fef3b8 on Jan 7, 2020
codablock referenced this in commit 430c0380fa on Jan 7, 2020
deadalnix referenced this in commit 6434a8f441 on Apr 20, 2020
ftrader referenced this in commit 1dca7767b1 on Aug 17, 2020
ckti referenced this in commit 80bf96b0b3 on Mar 28, 2021
gades referenced this in commit 0d5c1bb1eb on Jun 30, 2021
MarcoFalke locked this on Sep 8, 2021

Contributors

Labels