test: test_runner option to run tests in priority #26102

Adds an option to the test_runner to run tests in priority based on a comma-separated list of keywords. Can be used like python test/functional/test_runner.py -j1 --priority='rpc,wallet, which will first run rpc tests then wallet tests. Or with tests directly: python test/functional/test_runner.py -j1 --priority='wallet_groups.py'.

Note: if you test the script and don't use -j1, you may see other non-prioritized tests being executed first because they run in parallel.

This would be very useful if the CI was able to detect which tests have been updated in a pull request to execute them first. (#23808). I've tried to update the CI accordingly but it does seem to work because of errors such as:

fatal: ambiguous argument 'master': unknown revision or path not in the working tree.

I am unfamiliar with the CI so any help would be welcome, maybe in another PR.

What I've tried:

diff --git a/ci/test/06_script_b.sh b/ci/test/06_script_b.sh
index 5bdb392ba..f3da08637 100755
--- a/ci/test/06_script_b.sh
+++ b/ci/test/06_script_b.sh
@@ -31,7 +31,7 @@ if [ "$RUN_UNIT_TESTS_SEQUENTIAL" = "true" ]; then
 fi

 if [ "$RUN_FUNCTIONAL_TESTS" = "true" ]; then
-  CI_EXEC LD_LIBRARY_PATH="${DEPENDS_DIR}/${HOST}/lib" "${TEST_RUNNER_ENV}" test/functional/test_runner.py --ci "$MAKEJOBS" --tmpdirprefix "${BASE_SCRATCH_DIR}/test_runner/" --ansi --combinedlogslen=4000 --timeout-factor="${TEST_RUNNER_TIMEOUT_FACTOR}" "${TEST_RUNNER_EXTRA}" --quiet --failfast
+  CI_EXEC LD_LIBRARY_PATH="${DEPENDS_DIR}/${HOST}/lib" "${TEST_RUNNER_ENV}" test/functional/test_runner.py --ci "$MAKEJOBS" --tmpdirprefix "${BASE_SCRATCH_DIR}/test_runner/" --ansi --combinedlogslen=4000 --timeout-factor="${TEST_RUNNER_TIMEOUT_FACTOR}" "${TEST_RUNNER_EXTRA}" --quiet --failfast --priority $(git diff --name-only "${CIRRUS_BASE_BRANCH}" "${CIRRUS_BRANCH}" | while read t; do if [[ $t == test/functional/*.py ]]; then echo $(basename $t); fi; done)
 fi

 if [ "${RUN_TIDY}" = "true" ]; then

Is the motivation behind this purely for the CI to fail fast on modified tests? What's the scenario where someone will want to run the tests not in parallel and with specific priority? Could you achieve a similar outcome with -filter?

Is the motivation behind this purely for the CI to fail fast on modified tests?

I think it still has its use-cases; for example if you're touching wallet code and you want to run a few specific wallet tests before the rest locally. But yes it would be better to have it on the CI for the fail fast.

What's the scenario where someone will want to run the tests not in parallel and with specific priority?

I don't think anyone will want to do that. I added the note for reviewers in case they run the script with parallel jobs and prioritized tests because some non-prioritized tests may finish faster and that may be confusing.

<!--e57a25ab6845829454e8d69fc972939a-->

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

<!--174a7506f384e20aa4161008e828411d-->

Conflicts

Reviewers, this pull request conflicts with the following ones:

• #25797 (build: Add CMake-based build system by hebasto)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.