test: optionally run functional tests via CTest #35762

willcl-ark commented at 11:04 AM on July 21, 2026: member

Expose individual functional tests through the same CTest interface used for unit tests, making them easier to select, inspect, and rerun from the command line or CTest-aware IDEs.

Scheduling unit and functional tests together may improve parallel execution. It also enables per-test results and timings to be uploaded to our Bitcoin Core CDash instance, helping track failures and runtime changes across builds.

Registration is opt-in with BUILD_FUNCTIONAL_TESTS=ON and requires CMake 4.4 or newer. The existing runner remains available for RPC coverage and CSV results. The native macOS CI job exercises the combined CTest workflow and passes. See test/README.md for usage and limitations.

DrahtBot added the label Tests on Jul 21, 2026

DrahtBot commented at 11:04 AM on July 21, 2026: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

External sites

Code coverage & benchmarks: https://corecheck.dev/bitcoin/bitcoin/pulls/35762
Ralph review <img src=https://review.fish.foo/assets/bitcoin/bitcoin/35762.png alt="Ralph review status"> https://git.fish.foo/bitcoin/bitcoin/pulls/35762#author-ralph

Reviews

See the guideline and AI policy for information on the review process. A summary of reviews will appear here.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#35395 <img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/35395.svg"> (doc: Improve test suite dependencies documentation by hebasto)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

willcl-ark commented at 12:43 PM on July 21, 2026: member

The MacOS native CI job shows ctest orchestrating the unit and functional tests altogether: https://github.com/bitcoin/bitcoin/actions/runs/29824697919/job/88615235560?pr=35762#step:9:3919

in test/functional/test_runner.py:508 in d2b4bd40f2 outdated

 504 | @@ -443,14 +505,18 @@ def main():
 505 |          RED = ("", "")
 506 |  
 507 |      # args to be passed on always start with two dashes; tests are the remaining unknown args
 508 | -    tests = [arg for arg in unknown_args if arg[:2] != "--"]
 509 | +    tests = [arg for arg in unknown_args if arg[:2] != "--" and arg]

maflcko commented at 2:54 PM on July 21, 2026:

d2b4bd40f23067cad97ba6553fe0832d57424c26: Unrelated change?

willcl-ark commented at 9:16 PM on July 23, 2026:

Not quite, although it was incorrect (thanks!)

(somethign like this) is needed because native/upstream discover_tests() preserves the empty \5 replacement for tests without an optional argument. Without filtering it, --ctest-direct sees the script plus an empty second “test” and rejects the invocation.

The explicit -- separator is needed separately because argparse allows option abbreviation by default: CTest’s child option --tmpdir (tdestined for test_framework.py) can otherwise be consumed as the runner’s --tmpdirprefix option instead of being forwarded. The arg != "--" filter removes that separator before invoking the child test.

(in fact, I think this is a seperate bug and we could disable argparse allow_abbrev to fix it properly. Or perhaps makes better sense to fix that first, before these changes...)

willcl-ark commented at 11:19 AM on October 7, 2026:

now removed.

in test/functional/test_runner.py:513 in d2b4bd40f2

 510 |      passon_args = [arg for arg in unknown_args if arg[:2] == "--"]
 511 |  
 512 |      # Read config generated by configure.
 513 |      config = configparser.ConfigParser()
 514 | -    configfile = os.path.abspath(os.path.dirname(__file__)) + "/../config.ini"
 515 | +    configfile = args.configfile or os.path.abspath(os.path.dirname(__file__)) + "/../config.ini"

maflcko commented at 2:55 PM on July 21, 2026:

d2b4bd40f23067cad97ba6553fe0832d57424c26: Why not follow the docs and call the test_runner from the build dir, which avoids having to pass the new option?

willcl-ark commented at 9:21 PM on July 23, 2026:

Addressed in latest push by invoking the generated build/test/functional/test_runner.py link/copy. The runner can therefore continue locating test/config.ini relative to itself, without adding a runner-level --configfile option.

Nice, thanks. I didn't think of it because I always run everythign from the source directory, for no good reason.

willcl-ark commented at 11:18 AM on October 7, 2026:

CTest now invokes the generated build/test/functional/test_runner.py, which locates config.ini relative to itself. The added runner-level --configfile option is gone.

in test/functional/test_runner.py:668 in a9e0c71ca4

 663 | +    testdir = next((arg.split("=", 1)[1] for arg in args
 664 | +                    if arg.startswith("--tmpdir=")), None)
 665 | +    test_script = os.path.join(
 666 | +        config["environment"]["SRCDIR"], "test", "functional", tests[0]
 667 | +    )
 668 | +    os.environ["PYTHON_GIL"] = "1"

maflcko commented at 2:57 PM on July 21, 2026:

a9e0c71ca4dee40abce66aad0bbb27682ccdcd7f: Could drop this, or add least add a short rationale for stuff like this?

willcl-ark commented at 9:27 PM on July 23, 2026:

added a short comment documenting the source-tree resolution (and dropped PYTHON_GIL=1 which is set by test/CMakeLists.txt on each test already.

willcl-ark force-pushed on Jul 22, 2026

willcl-ark force-pushed on Jul 23, 2026

DrahtBot added the label CI failed on Jul 23, 2026

maflcko removed the label CI failed on Jul 30, 2026

DrahtBot added the label CI failed on Jul 30, 2026

DrahtBot commented at 9:04 AM on July 30, 2026: contributor

🚧 At least one of the CI tasks failed. Task riscv32 bare metal, static libbitcoin_consensus: https://github.com/bitcoin/bitcoin/actions/runs/30046119080/job/90826618769 LLM reason (✨ experimental): CI failed because submodule cloning of newlib-cygwin.git from sourceware.org was rate-limited (HTTP 429), causing the base install step to exit with code 2.

<details><summary>Hints</summary>

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

</details>

in ci/test/00_setup_env.sh:40 in b7ad3bff61 outdated

  34 | @@ -35,6 +35,9 @@ export MAKEJOBS=${MAKEJOBS:--j$(if command -v nproc > /dev/null 2>&1; then nproc
  35 |  
  36 |  export RUN_UNIT_TESTS=${RUN_UNIT_TESTS:-true}
  37 |  export RUN_FUNCTIONAL_TESTS=${RUN_FUNCTIONAL_TESTS:-true}
  38 | +export RUN_FUNCTIONAL_TESTS_WITH_CTEST=${RUN_FUNCTIONAL_TESTS_WITH_CTEST:-false}
  39 | +# Which tests to run under ctest. Can be "all", "functional", or "nonfunctional" (default)
  40 | +export CTEST_TESTS=${CTEST_TESTS:-nonfunctional}

maflcko commented at 7:44 PM on July 30, 2026:

not sure about exposing this in CI. This will silently ignore TEST_RUNNER_EXTRA, and having two ways to achieve the same seems confusing anyway.

However, not exposing it in CI will also make it less useful ...

willcl-ark commented at 11:39 AM on August 4, 2026:

Thank you for taking a look, and fair point!

I don't consider these strict duplicates: CTEST_TESTS is a coarse CTest label selection: all, functional, or nonfunctional whereas TEST_RUNNER_EXTRA contains runner-specific behavior and test arguments.

However they do certainly overlap, and I think (for now) I will amend the commit to fail if TEST_RUNNER_EXTRA is non-empty in CTEST mode as a check on silently doing something else than the tests might have indented.

Longer-term, say we switch all tests over to ctest, I would see us dropping RUN_FUNCTIONAL_TESTS_WITH_CTEST (it would be default/on for all and unneeded), probably alos remove CTEST_TESTS and instead have each job just construct its own ctest command directly, and maybe even remove TEST_RUNNER_EXTRA (depening on whether we can get all supported options ported to ctest).

maflcko commented at 2:31 PM on August 4, 2026:

Why would porting be hard? The leftover from "${TEST_RUNNER_EXTRA[@]}" would just end up in passon_args and be exported to the test list, no?

This pull is adding 500 LOC of logic complexity, so my preference would be to at least keep the overhead low.

Recall the alternative to this would be to write it in a few trivial lines of Python (https://github.com/bitcoin/bitcoin/issues/32770#issuecomment-3714857697)

maflcko commented at 2:37 PM on August 4, 2026:

Also, recall that Python would work when testing win64-cross builds, but CTest does not.

maflcko commented at 2:10 PM on October 6, 2026:

Are you still working on this?

willcl-ark commented at 4:02 PM on October 6, 2026:

Yes, thanks. Will update shortly.

willcl-ark commented at 11:18 AM on October 7, 2026:

This pull is adding 500 LOC of logic complexity, so my preference would be to at least keep the overhead low.

Agreed this change was getting too large. About half was supporting this new feature on old cmake versions, so I dropped that. It now just requires cmake >= 4.4 to use.

Also dropped the duplicate selector.

willcl-ark force-pushed on Aug 4, 2026

DrahtBot removed the label CI failed on Aug 4, 2026

in test/functional/test_runner.py:526 in e05e5ad0f7

 523 | +        dump_ctest_inventory(config, extended=args.extended)
 524 | +        return
 525 | +
 526 |      passon_args.append("--configfile=%s" % configfile)
 527 |  
 528 | +    enable_bitcoind = config["components"].getboolean("ENABLE_BITCOIND")

maflcko commented at 6:51 AM on August 17, 2026:

unrelated nit: You may want to rebase this on top of #35846, for better error (message) handling

willcl-ark commented at 11:16 AM on October 7, 2026:

rebased on master

willcl-ark force-pushed on Aug 17, 2026

sedited requested review from maflcko on Oct 2, 2026

DrahtBot added the label Needs rebase on Oct 6, 2026

test: require full runner option names

bugfix where options such as --tmpdir can otherwise be consumed as
abbreviations of runner options.

Disable abbreviation so forwarding does not depend on which similarly
named runner options happen to exist.

04764efd99

test: expose functional cases to CTest

Reuse the existing selection and benchmark expansion before emitting the
inventory, so exclusions and child options have one source of truth.

Pass each test specification as one quoted argument to support multiple
variant options without imposing a fixed count.

Keep the existing skip and stderr failure semantics when CTest runs one
case. Share timeout and failure-log handling with cache setup, and leave
failed test directories intact across reruns.

0647078c66

build: optionally register functional tests with CTest

Native discovery makes functional cases available to CTest scheduling,
filters and dashboards alongside unit tests. Require CMake 4.4 only when
this optional registration is enabled, avoiding a second discovery
implementation while retaining the existing minimum for normal builds.

Use fixtures to prepare the shared chain cache before parallel execution.
Run cache creation through the same Python wrapper as the test cases so
its timeout and failure handling cannot drift from theirs.

7d3bbcffaa

ci: exercise functional CTest execution on macOS

Run unit and functional cases together in one native CI job to keep the
optional CTest path exercised.

Derive selection from the existing RUN_* switches and reuse the
sanitizer-wrapped invocation instead of adding another selection
interface.

Pass existing runner options through TEST_RUNNER_EXTRA, including
failure log output and the scratch directory. Exclude registered
functional cases from legacy unit-only CTest invocations to avoid
running them twice.

a44d018b32

willcl-ark force-pushed on Oct 7, 2026

DrahtBot added the label CI failed on Oct 7, 2026

DrahtBot commented at 8:28 AM on October 7, 2026: contributor

🚧 At least one of the CI tasks failed. Task macOS native: https://github.com/bitcoin/bitcoin/actions/runs/37589945926/job/112688773432 LLM reason (✨ experimental): CI failed because parallel benchmark tests collided on the same temporary directory, causing bench_PrevectorDeserializeTrivial to fail with FileExistsError.

<details><summary>Hints</summary>

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

</details>

DrahtBot removed the label CI failed on Oct 7, 2026

DrahtBot removed the label Needs rebase on Oct 7, 2026

doc: explain optional functional CTest execution

Document the opt-in version requirement and standard selection commands
so developers can run individual cases or the combined suite. State the
remaining coverage and concurrent-invocation limits without implying a
replacement for every existing runner workflow.

fd6ace0610

test: avoid CTest temporary directory collisions

Concurrent variants of the same script can observe the same wall-clock value
on macOS and select an identical temporary directory. The second test then
fails before framework setup completes.

Use a random UUID suffix so directory names do not depend on clock resolution.
Leave directory creation and cleanup with the test framework to preserve
failed logs and its requirement that the supplied path does not exist.

f574198736

willcl-ark force-pushed on Oct 7, 2026

willcl-ark commented at 11:21 AM on October 7, 2026: member

PR description updated with the new, more limited, scope.

Windows cross remains a limitation for now. It can be made to work without too much tinkering, but I don't think that's needed in this pull.

I also considered a commit to have the macos_native ci job (using ctest) actually push results to cdash, but I think that can also come in a separate followup pull.