Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Development: Run tests with submission builds sequentially #9623

Draft
wants to merge 7 commits into
base: develop
Choose a base branch
from

Conversation

muradium
Copy link
Contributor

@muradium muradium commented Oct 28, 2024

Checklist

General

Client

  • Important: I implemented the changes with a very good performance, prevented too many (unnecessary) REST calls and made sure the UI is responsive, even with large data (e.g. using paging).
  • I strictly followed the client coding and design guidelines.

Motivation and Context

During E2E tests, when multiple exercise submissions trigger simultaneous CI builds, CI servers may get too loaded and builds may take significantly longer than normal to finish. This causes inconsistent test execution times and often results in timeouts. Executing tests that involve submission builds sequentially could solve this issue.

Description

This PR introduces annotations (@fast and @slow) to categorize Playwright e2e tests.
Fast Tests (@fast): Tests that do not involve exercise submissions triggering CI builds. These are executed in a fully parallel manner to maximize test speed.
Slow Tests (@slow): Tests that include programming exercise submissions which initiate CI builds. Due to the increased server load these tests create, they are executed sequentially to prevent simultaneous builds.

Steps for Testing

Prerequisites:

  • 1 Instructor
  • 2 Students
  • 1 Programming Exercise with Complaints enabled
  1. Log in to Artemis
  2. Navigate to Course Administration
  3. ...

Exam Mode Testing

Prerequisites:

  • 1 Instructor
  • 2 Students
  • 1 Exam with a Programming Exercise
  1. Log in to Artemis
  2. Participate in the exam as a student
  3. Make sure that the UI of the programming exercise in the exam mode stays unchanged. You can use the exam mode documentation as reference.
  4. ...

Testserver States

Note

These badges show the state of the test servers.
Green = Currently available, Red = Currently locked
Click on the badges to get to the test servers.







Review Progress

Performance Review

  • I (as a reviewer) confirm that the client changes (in particular related to REST calls and UI responsiveness) are implemented with a very good performance even for very large courses with more than 2000 students.
  • I (as a reviewer) confirm that the server changes (in particular related to database calls) are implemented with a very good performance even for very large courses with more than 2000 students.

Code Review

  • Code Review 1
  • Code Review 2

Manual Tests

  • Test 1
  • Test 2

Exam Mode Test

  • Test 1
  • Test 2

Performance Tests

  • Test 1
  • Test 2

Test Coverage

Screenshots

@muradium muradium changed the title Run tests that trigger builds sequentially (as slow tests) to avoid o… Development: Run tests that trigger builds sequentially Oct 29, 2024
@muradium muradium changed the title Development: Run tests that trigger builds sequentially Development: Run tests with submission builds sequentially Oct 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Work In Progress
Development

Successfully merging this pull request may close these issues.

1 participant