GUI Testing in FOQUS

Motivation

• The FOQUS codebase has low test coverage
• This is a significant liability for both short- and long-term reliability and maintanability of the codebase
• Manual testing of various workflows requires continuous effort from the development team, with the disadvantage of poor reproducibility and scalability
• FOQUS is, in principle, designed with a degree of separation between the core framework and the GUI (i.e. foqus_lib/framework and foqus_lib/gui)
• However in practice a considerable amount of "business logic" is implemented in the GUI code and there
• The project's funding/personnel situation makes significant rewritings infeasible
• Targeted modifications of limited scope to improve testability are possible but need to be low-effort, high-impact (low-hanging fruits)

Solution: automated GUI testing

Advantages

• Writing tests that interact with the GUI layer have multiple advantages considering our constraints:
  • Efficient: the FOQUS codebase can be tested (almost) as-is, with no significant modifications required to the FOQUS code to make it testable
  • More realistic: workflows are tested end-to-end, in a way that matches closely the way users interact with the software

Challenges

• Writing GUI tests is hard:
  • Asyncronous execution
  • Qt is a relatively well-documented platform, but it's still large and complex
  • OS-specific quirks related to e.g. desktop environment, window manager
• GUI tests should be integrated with the tools already in place for FOQUS for automated testing (pytest) and continous integration (GitHub Actions for all 3 supported OSes)

Existing solutions and limitations

• A series of "smoke tests" GUI tests for several FOQUS workflows (https://github.com/CCSI-Toolset/FOQUS/tree/master/examples/test_files/Smoke_Tests)
  • However, these are not integrated (nor easily integratable) with pytest or Github Actions
  • Limitations in the implementation hamper extensibility, scalability, and reliability (e.g. parametrization is done by copying and pasting entire test files just to change one parameter)
• An existing tool, pytest-qt, is available off-the-shelf and is compatible with the test framework used by FOQUS
  • However, its API is relatively low-level and requires knowledge of the internal workings of Qt

Our solution: writing concise, readable GUI tests for FOQUS using pytest-qt-extras

• An extension to pytest-qt, pytest-qt-extras, has been developed to allow writing GUI tests for FOQUS using a more streamlined, higher-level API
• Additional capabilities to assist FOQUS developers such as e.g. automatically creating screenshot of the FOQUS GUI while test is running

Current status

• An initial implementation (https://github.com/CCSI-Toolset/FOQUS/blob/master/pytest_qt_extras.py) has been used successfully (with some caveats, see below) since a few months as part of the FOQUS CI infrastructure
• Based on a specialized subclass of pytest_qt.QtBot with syntactic sugar for widget query/selection, plus several FOQUS-specific fixtures for automatic setup and teardown of FOQUS sessions, GUI window, etc
• Currently, tests written with this solution include a multi-part UQ workflow with various parametrizations (https://github.com/CCSI-Toolset/FOQUS/blob/master/foqus_lib/gui/tests/test_uq.py)

Next steps

• Extend GUI tests to cover other FOQUS workflows
• Improve API in terms of readability and ease of use
• Implement a common interface for user notifications to address errors caused by the use of modal dialogs called with QDialog.exec_() that currently requires brittle workarounds

Example: comparing the same workflow using the "smoke test" solution and pytest-qt-extra

# this block is repeated for all the uq_smoke_test_*.py files
def uq_sampling_scheme(MainWin=MainWin, getButton=getButton, timers=timers, go=go):
    """Setup up an enseble sampling scheme, stops timer once window comes up"""
    w = MainWin.app.activeWindow()
    if 'SimSetup' in str(type(w)):
        timers['uq_sampling_scheme'].stop()
        w.distTable.cellWidget(2,1).setCurrentIndex(1)
        w.samplingTabs.setCurrentIndex(1)
        items = w.schemesList.findItems('Latin Hypercube', QtCore.Qt.MatchExactly)
        w.schemesList.setCurrentItem(items[0])
        w.numSamplesBox.setValue(100)
        w.generateSamplesButton.click()
        if not go(sleep=2): return # wait long enough for samples to generate
        w.doneButton.click()
...
# at the end of the module
while True:
    try:
        MainWin.uqSetupFrame.addSimulationButton.click()
        timers['uq_sampling_scheme'].start(500)
        MainWin.uqSetupFrame.addSimulationButton.click()
        if not timerWait('uq_sampling_scheme'): break

# in test_uq.py
@pytest.fixture
def uq_frame(main_window, qtbot):
    frame = main_window.uqSetupFrame
    qtbot.focused = frame
    yield frame

@pytest.fixture
def generate_samples(uq_frame, qtbot):
    with qtbot.waiting_for_modal(handler=_accept_dialog):
        qtbot.take_screenshot('samples-modal')
        qtbot.click(button='Add New...')
    with qtbot.searching_within(SimSetup) as sim_frame:
        with qtbot.searching_within(group_box="Choose how to generate samples:"):
            qtbot.click(radio_button="Choose sampling scheme")
        qtbot.show_tab("Distributions")
        qtbot.click(button="All Variable")
        with qtbot.focusing_on(table=any):
            qtbot.select_row(1)
            qtbot.using(column="Type").set_option("Fixed")
        qtbot.show_tab("Sampling scheme")
        qtbot.click(radio_button="All")
        qtbot.using(item_list=any).set_option("Latin Hypercube")
        qtbot.using(spin_box=...).enter_value(2000)
        qtbot.click(button="Generate Samples")
        qtbot.click(button="Done")

def test_generate_samples(uq_frame, qtbot, generate_samples):
    table = uq_frame.simulationTable
    assert table.rowCount() == 1