[tests] color test results and sort alphabetically #10159

1. 
   jnewbery commented at 7:57 pm on April 5, 2017: member

   Functional test results were previously printed in the order that the tests finished, which is fairly arbitrary. This commit prints test results in alphabetical order.

   The commit also colors the passing tests blue, failing tests red, and skipped tests grey, to make it easier to spot failing tests. I’m happy to debate the color of the paint.

   Implementation is by creating a new TestResult class, which opens the way to provide test results in different formats (html pages, email reports, etc).

2. 
   jnewbery commented at 8:00 pm on April 5, 2017: member

   Sample output from my terminal:

   

3. in test/functional/test_runner.py:296 in c5ec22db03 outdated

   310     sys.exit(not all_passed)
   311 
   312+def print_results(test_results, max_len_name, runtime):
   313+    results = "\n" + BOLD[1] + "%s | %s | %s\n\n" % ("TEST".ljust(max_len_name), "STATUS ", "DURATION") + BOLD[0]
   314+
   315+    test_results.sort(key=lambda result: result.name)
   

   ---

   ---

   

   JeremyRubin commented at 9:26 pm on April 5, 2017:

   force result to be all lowercase for proper sorting.

   ---

   

   jnewbery commented at 10:36 pm on April 5, 2017:

   good spot. Fixed.
4. JeremyRubin approved
5. 
   JeremyRubin commented at 9:27 pm on April 5, 2017: contributor

   Concept Ack, looks fine minus one nit on forcing lowercase on the names during sorting.
6. [tests] color test results and sort alphabetically 63062bda1a
7. jnewbery force-pushed on Apr 5, 2017
8. fanquake added the label Tests on Apr 5, 2017
9. 
   MarcoFalke commented at 7:28 am on April 6, 2017: member

   I’d find it useful to spot the status in one glance on non-color terminals as well (e.g. travis log)

   I was thinking about prepending a sybol to each status:

   • x Fail
   • o Skip
   • ✓ Pass
10. 
    laanwj commented at 7:38 am on April 6, 2017: member

    I’d find it useful to spot the status in one glance on non-color terminals as well (e.g. travis log)

    Doesn’t travis have limited formatting support too? I don’t know what ANSI sequences it supports, but I’m fairly sure it supports some (according to some random sources on the internet even color).

11. 
    MarcoFalke commented at 7:44 am on April 6, 2017: member

    It does, but there is also a raw log (e.g. https://s3.amazonaws.com/archive.travis-ci.org/jobs/218889586/log.txt)
12. 
    laanwj commented at 8:09 am on April 6, 2017: member

    I’ve tried it out a bit: https://travis-ci.org/laanwj/travis-test/builds/219186479

    Seems the low-intensity fg and bg are supported (30-38, 40-48), as well as bold (1) italic (3) and underline (4). 16-color (high-intensity) 88-color and 256-color are not supported (mostly just ignored).

    e.g. https://s3.amazonaws.com/archive.travis-ci.org/jobs/218889586/log.txt

    Could always view it in less -R

13. 
    laanwj commented at 8:18 am on April 6, 2017: member

    -r seems to work better than -R (which I usually prefer as it reduces screen corruption) due to the large number of cursor control sequences. Pretty awesome anyhow, should have known of this before. To view travis logs on the console:

    0curl https://s3.amazonaws.com/archive.travis-ci.org/jobs/218889586/log.txt | less -r
    

    Edit: eh sorry got carried away, utACK https://github.com/bitcoin/bitcoin/pull/10159/commits/63062bda1ac0b57cb92e663596650a6e42508f15

14. [tests] Add unicode symbols for tests passing/failing/skipping bb92d839d5
15. 
    jnewbery commented at 5:33 pm on April 6, 2017: member

    Added unicode symbols for passing/failing/skipping tests as requested:

    

    This is the last commit. Feel free to squash if you think it’s worthwhile (or leave out if not).

16. 
    laanwj commented at 8:56 pm on April 6, 2017: member

    Looks good to me, though from that screenshot it looks like the glyph for the “ALL” row is missing, misaligning it.
17. fixup - align summary row correctly and make colors/glyphs globals d80baaa514
18. 
    jnewbery commented at 1:17 pm on April 7, 2017: member

    I’ve pushed a fixup commit which corrects the summary row alignment and tidies code a little.

    ps: great tip here #10159 (comment) @laanwj !

19. laanwj merged this on Apr 7, 2017
20. laanwj closed this on Apr 7, 2017

    ---

21. laanwj referenced this in commit df1ca9e93a on Apr 7, 2017
22. 
    jnewbery commented at 3:00 pm on April 7, 2017: member

    I wasn’t expecting that to get merged already. I would have squashed commits first!
23. 
    laanwj commented at 7:40 am on April 10, 2017: member

    I wasn’t expecting that to get merged already. I would have squashed commits first!

    Yes that would have made sense, sorry.

24. PastaPastaPasta referenced this in commit 97f4e7fa60 on Jun 10, 2019
25. PastaPastaPasta referenced this in commit ac5691ac58 on Jun 11, 2019
26. PastaPastaPasta referenced this in commit b012b34574 on Jun 15, 2019
27. PastaPastaPasta referenced this in commit f35d965e20 on Jun 15, 2019
28. UdjinM6 referenced this in commit 6b2c9beaa3 on Jun 17, 2019
29. codablock referenced this in commit 02c4cf6698 on Jun 18, 2019
30. codablock referenced this in commit 8d841175df on Jun 18, 2019
31. PastaPastaPasta referenced this in commit 22c4cd7b09 on Jun 18, 2019
32. PastaPastaPasta referenced this in commit 0cd9127766 on Jun 19, 2019
33. barrystyle referenced this in commit e0edb4790b on Jan 22, 2020
34. MarcoFalke locked this on Sep 8, 2021

Contributors
jnewbery JeremyRubin MarcoFalke laanwj

Labels
Tests