Our approach to testing is continually evolving as we use best practices decided by the wider industry, whilst adapting these to our own internal workflow and product.
## Our principles
- Create an environment where quality is everyone's responsibility.
- Improve test coverage and leverage tests at all levels.
- Make engineering work efficient, engaged and productive.
- Make QA and testing metrics driven.
The traditional software testing pyramid dictates that e2e tests should make up the smallest percentage of overall tests, due to their expense and speed. However we have found that such tests bring the most value for ensuring that what is shipped works by the same definition that a user would judge working software, as this class of tests is closest to an end user's perception of your product.
The diagram does not necessarily mean that the absolute number of tests have to be highest at the e2e level, but this is where we spend most of our time ensuring we have all user flows tested, in as similar an environment as if feasible possibly to production.
- Platform
- e2e
- Integration
- Unit
- Smoke
Platform tests are a specific class of test which we see as even broader than a traditional e2e test. In the AirGrid platform a traditional flow of actions will look like the following:
- User creates audience model.
- Model is built by backend services.
- Audience model deployed to millions of web browsers.
An e2e test would cover step 1 & 2, the interaction of the user facing UI and the correct creation of internal database objects. Platform tests go one step further to check that objects created internally are deployed to browsers via the publisher SDK and / or Prebid RTD module.
Platform tests live outside of the repo of any individual software component and are themselves their own service. They are run daily and failures are notified into slack and email.
A platform test executes our primary workflow, which is the creation of a new audience model, deployment of the model via the SDK to web browsers, and finally checks for the successful predictions generated by the audience model.
e2e testing is a strategy used to check whether your application works as expected across the entire software stack and architecture, including integration of all micro-services and components that are supposed to work together.
The bulk of our testing effort has been in covering all the possible user facing use cases of the platform. Each new feature should have a corresponding e2e test.
This test suite can be run locally and also through a Github action as part of a
CI/CD process. We automatically run these tests on every single PR opened into
either the (long-running) main or develop branches. Nothing can be merged or
released until all tests have passed and we see the divine view that is a full
set of green Github ticks!
The GH action will:
- Serve the frontend.
- Serve the dedicated backend.
- Serve any required micro-services.
- Boot and seed a database.
- Execute all tests against this stack.
- Perform teardown operations.
Our stack for e2e tests:
- Cypress
- Github actions
- Docker
- Supabase (postgres)
Integration or unit test is a little hard to define, we see this as a spectrum, unit being the smallest possible test, such as testing a single pure function, while integration test would be anything before a full blown e2e test.
Most tests somewhere in between
Unit <------------------------------------------------------------> Integration
| | | | |
Functions Modules Processes Machines Clusters
In our platform, most integration testing is quite high level such as:
- API tests with a mocked DB.
- Full use case tests, with a mocked DB.
- Complex component testing in the UI.
- Full view testing in the UI.
Frontend:
Integration tests cover the interaction between all components on a single page. Their abstraction level is comparable to how a user would interact with the UI. Sometimes we may test complex parent components in this manner also.
We can isolate the component or view being tested by using Storybook and than running:
- Snapshot tests: detect changes in the rendered markup to surface rendering errors or warnings.
- Visual (regression) tests: capture a screenshot of every story then compare it against baselines to detect appearance and integration issues.
- Interaction tests: verify component functionality by simulating user behaviour, firing events, and ensuring that state is updated as expected.
These test are also automated via a CI/CD pipeline, and the storybook containing the UI components is published to its own sub domain.
Backend & Micro-services:
Integration testing of backend code will usually be at the route or use case level, for a Typescript service, we would use supertest to allow us to run jest style assertions against the HTTP response from an Express app.
All routes should be covered by happy and sad path test cases, across the whole surface of API params.
Often we do not need to even run the HTTP server and we can run integration tests across modules inside of our service, by omitting testing the infrastructure layer. Use cases inside of the application should be written with a dependency injection pattern allowing us to easily mock out the database.
Unit tests ensure that a single unit of code (e.g. a method) work as expected - given an input, it has a predictable and consistent output. These tests should be isolated as much as possible. For example, model methods that do not do anything with the database should not need a DB record. Classes that do not need database records should use stubs/doubles.
Our philosophy has been that getting 100% test coverage in unit tests does not necessarily mean a working system. The reality of software means that although we build in modules, most of the issues arise when we aim to glue hundreds of modules together with most failures happening at the boundaries.
In light of this we aim to unit test:
- Data transformation heavy code, group by, sorting, calculation of metrics etc.
- Business logic heavy code, with significant conditionals and branching.
The decision of what should be unit tested is handled by the implementor & reviewers of each addition to the code base.
Unit tests are the simplest to run and setup, all projects will contain a test runner, automated execution in CI/CD and previously created tests to allow a developer to simply add tests for their feature.
Stack:
- Pytest (python)
- unittest (python)
- Jest (js)
- Vue Test Utils (js)
Smoke tests run quick e2e functional tests and are designed to run against the
specified environment to ensure that basic functionality is working. We run a
full e2e test suite against staging, develop and PRs and quick smoke tests
on releases.
These tests run in the same manner as e2e but contain just the key workflows and do not create any side effects to allow them to be executed against the production environment.
We test for:
- User authentication.
- Access to all pages within the application.
- Read of data for primary views of the frontend.
It is much easier find, update and get value from tests which are collocated with their code:
// bad
/src/module/class.ts
/test/modules/class.ts
// good
/src/module/class.ts
/src/module/class.test.ts
Tests should serve as live documentation, which can also be executed! Much
better than a stale and mouldy old readme file. However we want all users to
be able to read the tests and understand what the code being tested is achieving
from a business perspective.
// bad
describe('Authentication', () => {
it('parses the JWT', () => {
// write test...
});
});
// good
describe('Login', () => {
it('allows user to login successfully', () => {
// write test...
});
});Must read articles for anyone writing or reviewing software and tests.