test: optionally run functional tests via CTest #35762

willcl-ark commented at 11:04 AM on July 21, 2026: member

Running the functional tests via CTest lets CI and developers can use the same standard CTest interface to run all unit and enabled functional tests!

Individual tests become easy to inspect and rerun with standard CTest features such as -R, -L, --rerun-failed, --output-on-failure, and -j.

This also lets functional tests (and their output) become visible to CTest-aware IDEs, CTest dashboards, CDash, and all other downstream tooling that already understands CTest metadata.

Exercise the ctest invocation in a single CI job (native macos) to begin with and keep this from going stale.

Design decisions

Functional tests become first-class CTest tests alongside the existing unit tests. CTest replaces test_runner.py as the top-level orchestrator, while test_runner.py remains (and will always need to remain as) the discovery adapter and direct-execution wrapper.
test_runner.py --dump-ctest emits a six-field, line-oriented manifest containing the test name, script, port seed, optional label, optional argument, and cost.
- The manifest is structured directly for native CMake 4.4 discover_tests() capture groups.
CMake 4.4+ uses native discovery; older CMake versions use a compatibility shim with the same manifest format and a matching five-second discovery timeout.
Both paths execute tests through test_runner.py --ctest-direct.
Direct mode forwards CTest’s exact --portseed, --tmpdir, --cachedir, and per-test arguments; preserves skip status 77; and matches the legacy runner by treating exit status 0 with nonempty stderr as failure.
The combined log renderer is extracted into a shared runner function so direct CTest execution can reuse the normal test runner's combined-log rendering.
Test costs are emitted with the inventory so CTest can preserve the test runner's intended scheduling.
The native macOS CI job enables functional test registration and runs unit and functional tests together in one CTest invocation. The existing CI timeout factor is forwarded to framework-internal waits.

Known differences from `test_runner.py` orchestration

CTest mode is per-test rather than one aggregate test_runner.py invocation. Consequently, CTest owns:

parallelism instead of test_runner.py --jobs
test selection instead of --filter and --exclude
failure reporting instead of the runner’s aggregate summary
scheduling and stop-on-failure policy

The following runner features are not currently wired into the CTest command:

--coverage;
--resultsfile;
--nocleanup;
--tmpdirprefix;
interactive/debugging-oriented options such as --pdbonfailure;
arbitrary multi-argument test specifications.

Combined logs are enabled through CTEST_FUNCTIONAL_COMBINED_LOGS_LEN rather than test_runner.py's --combinedlogslen option.

Testing

To test the CTest runner, configure and build with BUILD_FUNCTIONAL_TESTS=ON:

cmake -B build -DBUILD_FUNCTIONAL_TESTS=ON
cmake --build build --parallel
ctest --test-dir build --parallel --output-on-failure

# Or functional tests only
ctest --test-dir build --label-regex '^functional$' --parallel --output-on-failure

Disclaimer

Codex gpt-5.6-sol medium wrote the entire re-implementation of CMake's discover_tests() in cmake in file test/functional/functional_discovery.cmake.in. The python, shell and docs changes are my own.

test: emit ctest manifest

Give test_runner.py the ability to emit a line-based test manifest which
can be consumed by ctest.

d2b4bd40f2

test: reuse combined log renderer in test runner

test runner already emits combined logs when a test fails. Keeping that
formatting in one helper lets single-test CTest execution reuse the same
output.

6b8434c3fc

test: add direct CTest runner mode

CTest passes fixed per-test arguments from discovery, so the compatibility
path must invoke the same runner as native discovery. Keep those arguments
intact, render CTest-requested logs in the runner, and return child statuses
unchanged so skipped tests remain 77.

a9e0c71ca4

build: register functional tests with CTest

If using cmake >= 4.4 use discover_tests(), otherwise use a custom
compatibility shim.

6cc9a33e87

doc: document ctest functional testing 18f1854ef0

ci: use ctest in a single job

testing code good

e4bc76e2cc

DrahtBot added the label Tests on Jul 21, 2026

DrahtBot commented at 11:04 AM on July 21, 2026: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35762.

Reviews

See the guideline and AI policy for information on the review process. A summary of reviews will appear here.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#35395 (doc: Improve test suite dependencies documentation by hebasto)
#31349 (ci: detect outbound internet traffic generated while running tests by vasild)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

willcl-ark commented at 12:43 PM on July 21, 2026: member

The MacOS native CI job shows ctest orchestrating the unit and functional tests altogether: https://github.com/bitcoin/bitcoin/actions/runs/29824697919/job/88615235560?pr=35762#step:9:3919