test: simple reordering to reduce run time #31396

1. 
   tdb3 commented at 5:35 pm on November 30, 2024: contributor

   Noticed in #31371 that the position of mempool_ephemeral_dust within BASE_SCRIPTS was lengthening total test runtime. Instead of moving only that test, looked for others to move to reduce runtime.

   This is a quick optimization that was found to reduce overall functional test runtime of up to around 20% (depending on jobs and machine characteristics). Since it seems like test ordering could be done in many different ways, with many variables, and bike shedding could creep in, a relatively straightforward approach was taken for now that minimized changes to test_runner.

2. test: simple ordering optimization to reduce runtime 62f6d9e1a4
3. 
   DrahtBot commented at 5:35 pm on November 30, 2024: contributor

   The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

   Code Coverage & Benchmarks

   For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31396.

   Reviews

   See the guideline for information on the review process.

   Type	Reviewers
   ACK	maflcko, TheCharlatan

   If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

   Conflicts

   Reviewers, this pull request conflicts with the following ones:

   • #31250 (wallet: Disable creating and loading legacy wallets by achow101)
   • #30437 (multiprocess: add bitcoin-mine test program by ryanofsky)
   • #29247 (CAT in Tapscript (BIP-347) by 0xBEEFCAF3)

   If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

4. DrahtBot added the label Tests on Nov 30, 2024
5. 
   tdb3 commented at 5:39 pm on November 30, 2024: contributor

   Approach

   The position of most tests (with bdb, excluding extended tests) in BASE_SCRIPTS was compared to the observed runtime of the test (using a low job count, to avoid saturation/contention). The difference in position was calculated (called deviation). Sorted by deviation. Filtered out all results with runtimes 20 seconds and under. Filtered for results beyond the first 100 positions in BASE_SCRIPTS. This left 5 tests. Moved those 5 tests nearer to the front of BASE_SCRIPTS, inserted based on runtimes of adjacent tests.

   Out of curiosity, a pure “sort by decreasing runtime” was performed, but only a marginal (0-3%) additional decrease in runtime was observed. Didn’t seem worth the extra changes to test_runner to pursue.

   For the mempool_ephemeral_dust test, optimization opportunities seem better suited for a different, more specific, PR.

   Tested on two machines (machine 1: higher single threaded performance but fewer cores (8) and machine 2: lower single threaded performance and more cores (20), both using a ramdisk). Used --resultsfile to record times.

   Job count 4 (matches default)

   Machine 1

   change	runtime
   before	551s
   after	513s

   Machine 2

   change	runtime
   before	634s
   after	618s

   About 3-7% runtime reduction.

   Higher job count

   Machine 1 (32 jobs)

   change	runtime
   before	129s
   after	96s

   Machine 2 (40 jobs)

   change	runtime
   before	109s
   after	95s

   Machine 2 (80 jobs)

   change	runtime
   before	111s
   after	110s

   About 1-21% runtime reduction.

6. tdb3 renamed this:
   test: simple ordering optimization to reduce runtime
   test: simple reordering to reduce runtime
   on Nov 30, 2024
7. tdb3 renamed this:
   test: simple reordering to reduce runtime
   test: simple reordering to reduce run time
   on Nov 30, 2024
8. 
   maflcko commented at 1:43 pm on December 2, 2024: member

   lgtm ACK 62f6d9e1a48e3b63c504996e914075cacfdcaedc
9. TheCharlatan approved
10. 
    TheCharlatan commented at 1:58 pm on December 2, 2024: contributor

    ACK 62f6d9e1a48e3b63c504996e914075cacfdcaedc
11. fanquake merged this on Dec 2, 2024
12. fanquake closed this on Dec 2, 2024

    ---

Contributors
tdb3 DrahtBot maflcko TheCharlatan

Labels
Tests