test: optionally run functional tests via CTest #35762

willcl-ark commented at 11:04 AM on July 21, 2026: member

Running the functional tests via CTest lets CI and developers can use the same standard CTest interface to run all unit and enabled functional tests!

Individual tests become easy to inspect and rerun with standard CTest features such as -R, -L, --rerun-failed, --output-on-failure, and -j.

This also lets functional tests (and their output) become visible to CTest-aware IDEs, CTest dashboards, CDash, and all other downstream tooling that already understands CTest metadata.

Exercise the ctest invocation in a single CI job (native macos) to begin with and keep this from going stale.

Design decisions

Functional tests become first-class CTest tests alongside the existing unit tests. CTest replaces test_runner.py as the top-level orchestrator, while test_runner.py remains (and will always need to remain as) the discovery adapter and direct-execution wrapper.
test_runner.py --dump-ctest emits a six-field, line-oriented manifest containing the test name, script, port seed, optional label, optional argument, and cost.
- The manifest is structured directly for native CMake 4.4 discover_tests() capture groups.
CMake 4.4+ uses native discovery; older CMake versions use a compatibility shim with the same manifest format and a matching five-second discovery timeout.
Both paths execute tests through test_runner.py --ctest-direct.
Direct mode forwards CTest’s exact --portseed, --tmpdir, --cachedir, and per-test arguments; preserves skip status 77; and matches the legacy runner by treating exit status 0 with nonempty stderr as failure.
The combined log renderer is extracted into a shared runner function so direct CTest execution can reuse the normal test runner's combined-log rendering.
Test costs are emitted with the inventory so CTest can preserve the test runner's intended scheduling.
The native macOS CI job enables functional test registration and runs unit and functional tests together in one CTest invocation. The existing CI timeout factor is forwarded to framework-internal waits.

Known differences from `test_runner.py` orchestration

CTest mode is per-test rather than one aggregate test_runner.py invocation. Consequently, CTest owns:

parallelism instead of test_runner.py --jobs
test selection instead of --filter and --exclude
failure reporting instead of the runner’s aggregate summary
scheduling and stop-on-failure policy

The following runner features are not currently wired into the CTest command:

--coverage;
--resultsfile;
--nocleanup;
--tmpdirprefix;
interactive/debugging-oriented options such as --pdbonfailure;
arbitrary multi-argument test specifications.

Combined logs are enabled through CTEST_FUNCTIONAL_COMBINED_LOGS_LEN rather than test_runner.py's --combinedlogslen option.

Testing

To test the CTest runner, configure and build with BUILD_FUNCTIONAL_TESTS=ON:

cmake -B build -DBUILD_FUNCTIONAL_TESTS=ON
cmake --build build --parallel
ctest --test-dir build --parallel --output-on-failure

# Or functional tests only
ctest --test-dir build --label-regex '^functional$' --parallel --output-on-failure

Disclaimer

Codex gpt-5.6-sol medium wrote the entire re-implementation of CMake's discover_tests() in cmake in file test/functional/functional_discovery.cmake.in. The python, shell and docs changes are my own.

DrahtBot added the label Tests on Jul 21, 2026

DrahtBot commented at 11:04 AM on July 21, 2026: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35762.

Reviews

See the guideline and AI policy for information on the review process. A summary of reviews will appear here.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#35846 (test: Use throwing config parser getters without fallback by maflcko)
#31349 (ci: detect outbound internet traffic generated while running tests by vasild)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

willcl-ark commented at 12:43 PM on July 21, 2026: member

The MacOS native CI job shows ctest orchestrating the unit and functional tests altogether: https://github.com/bitcoin/bitcoin/actions/runs/29824697919/job/88615235560?pr=35762#step:9:3919

in test/functional/test_runner.py:508 in d2b4bd40f2 outdated

 504 | @@ -443,14 +505,18 @@ def main():
 505 |          RED = ("", "")
 506 |  
 507 |      # args to be passed on always start with two dashes; tests are the remaining unknown args
 508 | -    tests = [arg for arg in unknown_args if arg[:2] != "--"]
 509 | +    tests = [arg for arg in unknown_args if arg[:2] != "--" and arg]

maflcko commented at 2:54 PM on July 21, 2026:

d2b4bd40f23067cad97ba6553fe0832d57424c26: Unrelated change?

willcl-ark commented at 9:16 PM on July 23, 2026:

Not quite, although it was incorrect (thanks!)

(somethign like this) is needed because native/upstream discover_tests() preserves the empty \5 replacement for tests without an optional argument. Without filtering it, --ctest-direct sees the script plus an empty second “test” and rejects the invocation.

The explicit -- separator is needed separately because argparse allows option abbreviation by default: CTest’s child option --tmpdir (tdestined for test_framework.py) can otherwise be consumed as the runner’s --tmpdirprefix option instead of being forwarded. The arg != "--" filter removes that separator before invoking the child test.

(in fact, I think this is a seperate bug and we could disable argparse allow_abbrev to fix it properly. Or perhaps makes better sense to fix that first, before these changes...)

in test/functional/test_runner.py:513 in d2b4bd40f2

 510 |      passon_args = [arg for arg in unknown_args if arg[:2] == "--"]
 511 |  
 512 |      # Read config generated by configure.
 513 |      config = configparser.ConfigParser()
 514 | -    configfile = os.path.abspath(os.path.dirname(__file__)) + "/../config.ini"
 515 | +    configfile = args.configfile or os.path.abspath(os.path.dirname(__file__)) + "/../config.ini"

maflcko commented at 2:55 PM on July 21, 2026:

d2b4bd40f23067cad97ba6553fe0832d57424c26: Why not follow the docs and call the test_runner from the build dir, which avoids having to pass the new option?

willcl-ark commented at 9:21 PM on July 23, 2026:

Addressed in latest push by invoking the generated build/test/functional/test_runner.py link/copy. The runner can therefore continue locating test/config.ini relative to itself, without adding a runner-level --configfile option.

Nice, thanks. I didn't think of it because I always run everythign from the source directory, for no good reason.

in test/functional/test_runner.py:668 in a9e0c71ca4

 663 | +    testdir = next((arg.split("=", 1)[1] for arg in args
 664 | +                    if arg.startswith("--tmpdir=")), None)
 665 | +    test_script = os.path.join(
 666 | +        config["environment"]["SRCDIR"], "test", "functional", tests[0]
 667 | +    )
 668 | +    os.environ["PYTHON_GIL"] = "1"

maflcko commented at 2:57 PM on July 21, 2026:

a9e0c71ca4dee40abce66aad0bbb27682ccdcd7f: Could drop this, or add least add a short rationale for stuff like this?

willcl-ark commented at 9:27 PM on July 23, 2026:

added a short comment documenting the source-tree resolution (and dropped PYTHON_GIL=1 which is set by test/CMakeLists.txt on each test already.

willcl-ark force-pushed on Jul 22, 2026

willcl-ark force-pushed on Jul 23, 2026

DrahtBot added the label CI failed on Jul 23, 2026

maflcko removed the label CI failed on Jul 30, 2026

DrahtBot added the label CI failed on Jul 30, 2026

DrahtBot commented at 9:04 AM on July 30, 2026: contributor

🚧 At least one of the CI tasks failed. <sub>Task riscv32 bare metal, static libbitcoin_consensus: https://github.com/bitcoin/bitcoin/actions/runs/30046119080/job/90826618769</sub> <sub>LLM reason (✨ experimental): CI failed because submodule cloning of newlib-cygwin.git from sourceware.org was rate-limited (HTTP 429), causing the base install step to exit with code 2.</sub>

<details><summary>Hints</summary>

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

</details>

test: emit ctest manifest

Give test_runner.py the ability to emit a line-based test manifest which
can be consumed by ctest.

72a8f6827d

test: reuse combined log renderer in test runner

test runner already emits combined logs when a test fails. Keeping that
formatting in one helper lets single-test CTest execution reuse the same
output.

27e826ce8a

test: add direct CTest runner mode

CTest schedules each discovered functional test independently, so let the
runner execute exactly one script without its normal suite orchestration.
Propagate failures and skip status 77 so CTest can report each result.

ecd1c7b362

build: register functional tests with CTest

Allow functional tests to run alongside unit tests through CTest while
keeping registration opt-in. Use CMake 4.4's discover_tests() when available
and retain an equivalent compatibility implementation for older supported
CMake versions.

Run the build-tree test runner so discovery and execution use the generated
config for that build without a separate override.

38dd695b32

doc: document CTest functional testing

Explain how to enable functional-test registration, select base or extended
tests, filter by CTest names and labels, and request combined failure logs.
Also record the shared-build-tree constraint and discovery behavior so local
runs do not interfere with each other.

e8839aacf8

ci: exercise functional tests through CTest on macOS

Use the existing macOS native job to run unit and functional tests together
through CTest. This exercises the new registration path without changing the
functional-test runner used by the remaining CI jobs.

e05e5ad0f7

in ci/test/00_setup_env.sh:40 in b7ad3bff61 outdated

  34 | @@ -35,6 +35,9 @@ export MAKEJOBS=${MAKEJOBS:--j$(if command -v nproc > /dev/null 2>&1; then nproc
  35 |  
  36 |  export RUN_UNIT_TESTS=${RUN_UNIT_TESTS:-true}
  37 |  export RUN_FUNCTIONAL_TESTS=${RUN_FUNCTIONAL_TESTS:-true}
  38 | +export RUN_FUNCTIONAL_TESTS_WITH_CTEST=${RUN_FUNCTIONAL_TESTS_WITH_CTEST:-false}
  39 | +# Which tests to run under ctest. Can be "all", "functional", or "nonfunctional" (default)
  40 | +export CTEST_TESTS=${CTEST_TESTS:-nonfunctional}

maflcko commented at 7:44 PM on July 30, 2026:

not sure about exposing this in CI. This will silently ignore TEST_RUNNER_EXTRA, and having two ways to achieve the same seems confusing anyway.

However, not exposing it in CI will also make it less useful ...

willcl-ark commented at 11:39 AM on August 4, 2026:

Thank you for taking a look, and fair point!

I don't consider these strict duplicates: CTEST_TESTS is a coarse CTest label selection: all, functional, or nonfunctional whereas TEST_RUNNER_EXTRA contains runner-specific behavior and test arguments.

However they do certainly overlap, and I think (for now) I will amend the commit to fail if TEST_RUNNER_EXTRA is non-empty in CTEST mode as a check on silently doing something else than the tests might have indented.

Longer-term, say we switch all tests over to ctest, I would see us dropping RUN_FUNCTIONAL_TESTS_WITH_CTEST (it would be default/on for all and unneeded), probably alos remove CTEST_TESTS and instead have each job just construct its own ctest command directly, and maybe even remove TEST_RUNNER_EXTRA (depening on whether we can get all supported options ported to ctest).

maflcko commented at 2:31 PM on August 4, 2026:

Why would porting be hard? The leftover from "${TEST_RUNNER_EXTRA[@]}" would just end up in passon_args and be exported to the test list, no?

This pull is adding 500 LOC of logic complexity, so my preference would be to at least keep the overhead low.

Recall the alternative to this would be to write it in a few trivial lines of Python (https://github.com/bitcoin/bitcoin/issues/32770#issuecomment-3714857697)

maflcko commented at 2:37 PM on August 4, 2026:

Also, recall that Python would work when testing win64-cross builds, but CTest does not.

willcl-ark force-pushed on Aug 4, 2026

DrahtBot removed the label CI failed on Aug 4, 2026