[Story] - Long term testing strategy

[ladislas]

Description

To make sure our systems are working, we have different levels of testing:

unit testing

• status: up and running locally and on CI
• test that units of the system behave the way they are supposed to
• run on host system (macOS/Linux)
• very low level
• TDD as much as possible

integration testing

• status: up and running locally and on CI
• test that components can work together
• mid to high level
• run on host system (macOS/Linux)
• TDD/BDD

(on device) component testing

• status: currently WIP, see [Story] - Functional tests - Future improvements #1117
• run on embedded device
• low level --> test that drivers/components are really working
• very close to integration testing, but focuses on only one component

(on device) integration testing

• status: currently WIP, see [Story] - Functional tests - Future improvements #1117
• mid level to high level
• run on embedded device
• test that components can work together and do not disturb one another

(on device) functional testing

• status: some parts currently WIP, see [Story] - Functional tests - Future improvements #1117, but others would need more tooling to automate a log of tests.
• mid level to high level
• run on embedded device
• test that components behave the way they are supposed to

(on device) system testing

• status: currently done by hand
• test that the whole system is working as expected
• long and tedious but necessary for each new release
• parts might be automated with automated on device functional testing

Missing types of testing

The above tests only focus on "does the unit/component/system work as expected".

There is not control related to wether the unit/component/system was working better or worse before and after the changes.

This imply we add one more type testing

(on device) performance testing

• status: not started yet
• test that the unit/component/system still performs as expected
• not applicable to all parts of the system
• parts where applicable:
  • logging --> output rate, impact on other thread, etc. (done by spike, must be automated)
  • BLE --> communication speed, MTU size
  • CommandKit + BLE --> number of commands sent, ability to handle them
  • VideoKit --> frame rate, CPU load
  • LEDKit --> frame rate, CPU load
  • IMU --> precision, CPU load
• could also measure compilation time, size impact of each module on the boostloader/os

Related epic/story: