GUI: Actual graphical tests

MarcoFalke commented at 5:25 pm on May 22, 2019: member

It is rather straightforward to write unit tests for C++ units or functional tests for the RPC interface. However graphical tests are done purely by hand. If anyone has any ideas to automate them, please leave a comment here.

MarcoFalke added the label Brainstorming on May 22, 2019

MarcoFalke added the label GUI on May 22, 2019

MarcoFalke added the label Tests on May 22, 2019

MarcoFalke commented at 5:27 pm on May 22, 2019: member

OpenSuse is using OpenQA, which could be adapted to our use case?

Example: https://openqa.opensuse.org/tests/940238#step/bootloader/3

ryanofsky commented at 10:09 pm on May 22, 2019: contributor

I’ve written a few automated tests that test the GUI interface. https://github.com/bitcoin/bitcoin/blob/master/src/qt/test/wallettests.cpp sends a transaction with the send coins dialog, looks up the transaction in the transaction view, starts a fee bump, and confirms it. https://github.com/bitcoin/bitcoin/blob/master/src/qt/test/apptests.cpp runs a command on the RPC console and checks the HTML output.

The tests run in the background by default, but you can run them interactively and see the GUI windows using the test_bitcoin-qt -platform xcb option.

I’m also planning on writing a new test in #15936 that will test editing and loading the GUI’s pruning and proxy settings.

MarcoFalke commented at 10:18 pm on May 22, 2019: member

I count them as “C++ unit tests”, as they are not testing the bitcoin-qt binary.

ryanofsky commented at 2:28 pm on June 18, 2019: contributor

If we need ideas on how to extend the gui tests, a good one would probably be to focus on testing new, less stable code, like code for dynamically loading wallets. For example wallettests.cpp could be extended to list and load wallets wallets and able to detect regressions like #16118 (comment).

promag commented at 2:35 pm on June 18, 2019: member

I’ll try your suggestion @ryanofsky.

jonasschnelli commented at 2:41 pm on June 18, 2019: contributor

How I did partially automated graphical tests in the past:

Automated application runs on different OSes with taking screenshots of all major situations
All images are stored in a folder for quick manual checks also by people not able to build on various OSes
Possibility to set “reference image”, make test fail if the reference image has a significant difference (say above 5%)

Though this is far from perfect and just meant as an inspiration for others working on possible solutions.

ryanofsky commented at 2:48 pm on June 18, 2019: contributor

Looking a little more closely, wallettests.cpp is not really a great place to test listing wallets since it currently only tests creating individual application dialogs instead of the whole application window. I think AppTests::guiTests would be a better place to add a wallet listing & loading test. It could start by opening the m_open_wallet_action menu, similar to how the existing code triggers the openRPCConsoleAction action.

jarolrod commented at 3:13 am on January 14, 2021: member

One option is Qt Test, there are extensions for testing GUI Events

ryanofsky commented at 6:20 pm on January 15, 2021: contributor

re: #16075 (comment)

One option is Qt Test, there are extensions for testing GUI Events

This is actually what apptests.cpp and wallettests.cpp are based on and currently doing (even using the same lineedit trick in the example)

jarolrod commented at 9:27 pm on January 23, 2021: member

sikulix is a FOSS tool for automated Graphical testing. It uses Tesseract OCR and OpenCV to verify that the GUI visually looks and works as intended. Using this tool would allow us to write automated graphical tests in python. Additionally we can hook this with robotframework and develop some BDD gherkin files which will allow us to test user interaction flows throughout the GUI.

RandyMcMillan commented at 4:34 pm on April 4, 2022: contributor

@jarolrod - thank you for mentioning sikulix This type of work interests me - please post branches of any related work (if you have started on this) for me to reference.

MarcoFalke closed this on Aug 5, 2022

bitcoin locked this on Aug 5, 2023

GUI: Actual graphical tests #16075