Functional test framework unit tests are currently run prior to all other functional tests.
This PR enables execution of the test framework unit tests in parallel with the functional tests, rather than before the functional tests, saving runtime and more efficiently using available cores.
This is a follow up to #29470 (comment)
New behavior:
- When running all tests, the framework unit tests are run in parallel with the other tests (unless explicitly skipped with
--exclude). This parallelization introduces marginal time savings when running all tests, depending on the machine used. As an example, a 2-3% time savings (9 seconds) was observed on a machine using--jobs=18(with 18 available cores). - When running specific functional tests, framework unit tests are now skipped by default. Framework unit tests can be added by including
feature_framework_unit_tests.pyin the list of specific tests being executed. The rationale for skipping by default is that if the tester is running specific functional tests, there is a conscious decision to focus testing, and choosing to run all tests (where unit tests are run by default) would be a next step. - The
--skipunitoption is now removed since unit tests are parallelized (they no longer delay other tests). Unit tests are treated equally as functional tests.
Implementation notes:
Since TextTestRunner can be noisy (even with verbosity=0, and therefore trigger job failure through the presence of non-failure stderr output), the approach taken was to send output to stdout, and forward test result (as determined by TestResult returned). This aligns with the previous check for unit test failure (if not result.wasSuccessful():).
This approach was tested by inserting self.assertEquals(True, False) into test_framework/address.py and seeing specifics of the failure reported.
135/302 - feature_framework_unit_tests.py failed, Duration: 0 s
stdout:
.F
======================================================================
FAIL: test_bech32_decode (test_framework.address.TestFrameworkScript.test_bech32_decode)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/dev/myrepos/bitcoin/test/functional/test_framework/address.py", line 228, in test_bech32_decode
self.assertEqual(True, False)
AssertionError: True != False
----------------------------------------------------------------------
Ran 2 tests in 0.003s
FAILED (failures=1)
stderr:
There was an initial thought to parallelize the execution of the unit tests themselves (i.e. run the 12 unit test files in parallel), however, this is not anticipated to further reduce runtime meaningfully and is anticipated to add unnecessary complexity.
