Tests: Order Python Tests Differently #10219

• Added documentation in tests/README.md about enabling wallet, utils and daemon.
• Change ordering to make the long-running EXTENDED_TESTS go first

@jimmysong thanks for opening this. I like the concept, but I wonder whether the approach is over-fitted:

• it's difficult to benchmark what the 'expected time' is for these tests. They're dependent on a lot of factors, including i/o, what else is happening on the system, and so on. If someone writes a new test, how do they know how to set the time?
• if someone updates/modifies a test case, they'll probably forget to update the time, so these will gradually get out-of-date.
• if there's a PR that speeds up all test cases (eg #10220 or the parallel stop/start part of #10082), does the PR need to update all of these times?

Luckily, I don't think we really need to optimize this so drastically. We just need to make sure the long-running extended tests stay at the top of the list when they're merged. I have a much simpler changeset that does that here: https://github.com/jnewbery/bitcoin/tree/mergelistsintelligently but I haven't PR'ed it because I didn't get round to testing/benchmarking it. I'm very happy for you to pick that up and PR it if you think it's useful.

ACK the README update. That's useful advice.

I'm neutral on changing the names from _SCRIPTS to _TESTS. Any specific reason you want to do that?

@jnewbery

• My thought on the benchmark was that people can see roughly what percentage of the expected time they take for a given test roughly in the same range on their own machine and divide by that amount. It does seem that developers currently have the same burden since they have to place their test roughly in the right place on the respective lists.
• Good point on the updates/modifications. Another option is to update the times based on some generic system and not burden developers with finding the right value. In this case, we would need to read the approximate times from a file that can be updated, say, once per release. Probably too complicated, though.
• Glad to take a much simpler approach. If minimal code disruption is the goal, I think your approach is much better.
• I changed from _SCRIPTS to _TESTS because the code calls them tests later. I thought this would be a good time to rename them since after this PR, it would be a script name and an approximate duration and not just a script name.

Glad to proceed on one approach or the other, should I wait for more comments or proceed on a simpler, minimally disruptive approach?

Indeed, the requirement is only that the longest test don't come last in the list. For all the quick tests it is not really important. I think the current approach/ordering is sufficient.

@jimmysong I'd go for zipping the lists together rather than concatenating them. When running extended tests, the fact that pruning isn't started immediately is a a big loss. If we can fix that with a one-line change, let's go ahead and do it.

Let's stick to the _SCRIPTS naming convention.

@jnewbery @MarcoFalke

I wanted to do the minimal change to get what we wanted and I just placed the EXTENDED_SCRIPTS first. Just wanting your thoughts on this. If you don't like it, I'll change this to zipping them per @jnewbery's branch.

Have you tested and benchmarked? What are the elapsed runtimes before and after this PR? How about with zipping the two lists together?

utACK 637706dc9ea9244954252a4fd7d8786bf93e3b67. It seems like this should be faster to execute than the current ordering.

@jnewbery tested (--extended) and got 1671s runtime with this patch vs 1882s runtime without. A savings of roughly 3-4 minutes.