Avoid "test_runner.py" script array #11964

The current test runner script array tents to generate git conflicts. Ideally, test scripts would have some metadata while test runner will just read files from the (same?) directory.

Discussion on IRC: https://botbot.me/freenode/bitcoin-core-dev/2017-12-20/?msg=94891704&page=6

[I would love to help out with btc development and am just jumping in on issues related to python development]

With that being said, adding a comment (#11965) to prevent git conflicts seems ineffective. I'd like to work on a more robust fix but some background info or pointers would be useful.

@thijstriemstra At least for me it is effective. I had zero merge conflicts up to now when adding new test scripts. See the related blog post [1] for another example where it worked.

[1] https://about.gitlab.com/2015/02/10/gitlab-reduced-merge-conflicts-by-90-percent-with-changelog-placeholders/

If someone wants to put all metadata in the test scripts themself, they should include at least:

• If they are a test script. (Maybe the presence of metadata is sufficient to determine this)
• Their approximate run time (coarse)
• Their test category: default (i.e. runs on pulls in travis), extended (i.e. runs on crons in travis)
• What features they require (wallet, zmq, linux/windows, qt, ipv6, ... ?)
• Something else ... ?

Ideally, test scripts would have some metadata

What metadata exactly, can you give an example?

I agree with Marco that adding a comment is sufficient here.

If someone wants to put all metadata in the test scripts themself, they should include at least:

• If they are a test script. (Maybe the presence of metadata is sufficient to determine this)
• Their approximate run time (coarse)
• Their test category: default (i.e. runs on pulls in travis), extended (i.e. runs on crons in travis)
• What features they require (wallet, zmq, linux/windows, qt, ipv6, ... ?)
• Something else ... ? @MarcoFalke Reading comments and linked IRC discussion, I'd like to work on a PR that attempts to solve the problem(s) here. Is this something still worth working on even though PR 11965 is now merged?

Friendly post-0.16-release nudge. Is this still an issue that needs to be addressed? If so, will work on a PR based on Marco's comment above. Thanks!

I'd say -0. You are more than welcome to work something out, but I can't guarantee that you will find more than one person to review it.

I can't guarantee that you will find more than one person to review it.

Seems interesting, I hereby pledge to be the one person to review it.

Interesting! How is working going on here? #13285 is my own proposal to simplify and clean up the test array, on a smaller scope. @jeffrade, do you think it would be worth working on the smaller proposal from my side in the mean time, or do you think you will put the metadata in scripts soon-ish?

@domob1812 I haven't put much time into this yet, so feel free to go ahead. I will add my ideas in #13285.

Since this issue is still open, wanted to show that the code comment in test_runner.py stating not to append new tests at the end of the list to avoid merge conflicts wasn't effective in this instance.

The feature request didn't seem to attract much attention in the past. Also, the issue seems not important enough right now to keep it sitting around idle in the list of open issues.

Closing due to lack of interest. Pull requests with improvements are always welcome.