test: discover functional tests by scanning the test directory #35725

pull jml6m wants to merge 1 commits into bitcoin:master from jml6m:test-runner-discover-scripts changing 2 files +69 −305
  1. jml6m commented at 2:32 AM on July 15, 2026: none

    test/functional/test_runner.py kept a large, manually ordered list of the functional test scripts (BASE_SCRIPTS). Adding a test meant editing that list and guessing a run-time position for it, and the shared, hand-ordered list was a recurring source of merge conflicts.

    This replaces the list with get_base_scripts(), which discovers the default tests by scanning the functional test directory.

    Result

    • Adding a test is just adding the file — a correctly named *.py is picked up automatically; no list entry, no run-time bucket to guess.
    • No more merge conflicts on a shared, hand-ordered array.
    • Behavior is unchanged: the discovered set of test invocations is identical to the previous list, so which tests run (and how) is the same.

    Kept explicit (cannot be derived from the filesystem)

    • TEST_PARAMS — tests invoked multiple times with different arguments (e.g. rpc_bind.py --ipv4/--ipv6/--nonloopback).
    • EXTENDED_SCRIPTS — extended-only tests.
    • LONGEST_BASE_SCRIPT — the longest test, scheduled first so parallel runtime is unaffected.

    test/functional/README.md is updated with the simpler instructions for adding a test.

  2. test: discover functional tests by scanning the test directory
    test_runner.py kept a large, manually ordered list of the functional
    test scripts to run. Adding a test meant editing the list and guessing
    a run-time position for it, and the shared ordered list was a recurring
    source of merge conflicts.
    
    Replace it with get_base_scripts(), which discovers the default tests by
    scanning the functional test directory for appropriately named scripts.
    Tests invoked several times with different arguments are declared in
    TEST_PARAMS, and extended-only tests remain in EXTENDED_SCRIPTS. The
    longest-running test is scheduled first so that total runtime under
    parallel execution is unchanged.
    
    The set of discovered test invocations is identical to the previous
    list, so which tests run, and how, is unchanged.
    215ffa0bec
  3. DrahtBot added the label Tests on Jul 15, 2026
  4. DrahtBot commented at 2:32 AM on July 15, 2026: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35725.

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline and AI policy for information on the review process.

    Type Reviewers
    Concept NACK achow101, maflcko

    If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

  5. achow101 commented at 3:37 AM on July 15, 2026: member

    Tend to NACK

    The script ordering is intentional so that the longest running scripts (note the plural) are scheduled first. Scheduling just the longest running script first may mean that the second longest running test is scheduled as the longest is finishing, potentially doubling the total runtime of the entire test suite. There is no guarantee that all other tests finish before the longest running test finishes.

    Adding a test meant editing that list and guessing a run-time position for it

    Running the test will tell you the runtime, it's trivially easy to figure out where it should go from there.

  6. maflcko commented at 6:41 AM on July 15, 2026: member

    Agree with the NACK.

    Also, this is ai slop that violates https://github.com/bitcoin/bitcoin/blob/master/doc/AI_POLICY.md

  7. maflcko closed this on Jul 15, 2026

  8. maflcko commented at 6:43 AM on July 15, 2026: member

    In theory the runtime could be encoded in a special header value inside the test file itself. However, I am not sure if this is worth the churn. Also, it isn't going to solve the problem of which tests are "extended" vs "normal" vs "special" (bench)...


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-07-22 05:50 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me