GUI: Actual graphical tests #16075

1. 
   MarcoFalke commented at 5:25 pm on May 22, 2019: member

   It is rather straightforward to write unit tests for C++ units or functional tests for the RPC interface. However graphical tests are done purely by hand. If anyone has any ideas to automate them, please leave a comment here.
2. MarcoFalke added the label Brainstorming on May 22, 2019
3. MarcoFalke added the label GUI on May 22, 2019
4. MarcoFalke added the label Tests on May 22, 2019
5. 
   MarcoFalke commented at 5:27 pm on May 22, 2019: member

   OpenSuse is using OpenQA, which could be adapted to our use case?

   Example: https://openqa.opensuse.org/tests/940238#step/bootloader/3

6. 
   ryanofsky commented at 10:09 pm on May 22, 2019: contributor

   I’ve written a few automated tests that test the GUI interface. https://github.com/bitcoin/bitcoin/blob/master/src/qt/test/wallettests.cpp sends a transaction with the send coins dialog, looks up the transaction in the transaction view, starts a fee bump, and confirms it. https://github.com/bitcoin/bitcoin/blob/master/src/qt/test/apptests.cpp runs a command on the RPC console and checks the HTML output.

   The tests run in the background by default, but you can run them interactively and see the GUI windows using the test_bitcoin-qt -platform xcb option.

   I’m also planning on writing a new test in #15936 that will test editing and loading the GUI’s pruning and proxy settings.

7. 
   MarcoFalke commented at 10:18 pm on May 22, 2019: member

   I count them as “C++ unit tests”, as they are not testing the bitcoin-qt binary.
8. 
   ryanofsky commented at 2:28 pm on June 18, 2019: contributor

   If we need ideas on how to extend the gui tests, a good one would probably be to focus on testing new, less stable code, like code for dynamically loading wallets. For example wallettests.cpp could be extended to list and load wallets wallets and able to detect regressions like #16118 (comment).
9. 
   promag commented at 2:35 pm on June 18, 2019: member

   I’ll try your suggestion @ryanofsky.
10. 
    jonasschnelli commented at 2:41 pm on June 18, 2019: contributor

    How I did partially automated graphical tests in the past:

    • Automated application runs on different OSes with taking screenshots of all major situations
    • All images are stored in a folder for quick manual checks also by people not able to build on various OSes
    • Possibility to set “reference image”, make test fail if the reference image has a significant difference (say above 5%)

    Though this is far from perfect and just meant as an inspiration for others working on possible solutions.

11. 
    ryanofsky commented at 2:48 pm on June 18, 2019: contributor

    Looking a little more closely, wallettests.cpp is not really a great place to test listing wallets since it currently only tests creating individual application dialogs instead of the whole application window. I think AppTests::guiTests would be a better place to add a wallet listing & loading test. It could start by opening the m_open_wallet_action menu, similar to how the existing code triggers the openRPCConsoleAction action.
12. 
    jarolrod commented at 3:13 am on January 14, 2021: member

    One option is Qt Test, there are extensions for testing GUI Events
13. 
    ryanofsky commented at 6:20 pm on January 15, 2021: contributor

    re: #16075 (comment)

    One option is Qt Test, there are extensions for testing GUI Events

    This is actually what apptests.cpp and wallettests.cpp are based on and currently doing (even using the same lineedit trick in the example)

14. 
    jarolrod commented at 9:27 pm on January 23, 2021: member

    sikulix is a FOSS tool for automated Graphical testing. It uses Tesseract OCR and OpenCV to verify that the GUI visually looks and works as intended. Using this tool would allow us to write automated graphical tests in python. Additionally we can hook this with robotframework and develop some BDD gherkin files which will allow us to test user interaction flows throughout the GUI.
15. 
    RandyMcMillan commented at 4:34 pm on April 4, 2022: contributor

    @jarolrod - thank you for mentioning sikulix This type of work interests me - please post branches of any related work (if you have started on this) for me to reference.
16. MarcoFalke closed this on Aug 5, 2022

    ---

17. bitcoin locked this on Aug 5, 2023

Contributors
MarcoFalke ryanofsky promag jonasschnelli jarolrod RandyMcMillan

Labels
Brainstorming GUI Tests