Skip to content

Commit

Permalink
Update benchmark README to include the new script
Browse files Browse the repository at this point in the history
  • Loading branch information
justinefricou committed Jul 30, 2024
1 parent ef723db commit e8b7b0f
Showing 1 changed file with 72 additions and 6 deletions.
78 changes: 72 additions & 6 deletions tools/benchmarking/README.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,21 @@
# Benchmarking of the route calculation systems
This benchmarking system allows to measure execution times (cache generation, requests, etc) to compare performances of several versions of the routing system.

## How it works
This benchmarking system uses a script that launches a Cypress spec file (a scenario) which plots a route using one version of the routing.
It uses two different scripts. The main one, `benchmark.sh`, enables to measure execution times on the frontend-side as well as on the backend-side, for each individual action taken during the route plotting. The second one, `get_backend_measures.sh`, only allows to measure backend-side times, but is much quicker due to not interacting with the frontend and allows for more freedom in which requests to measure.


## `benchmark.sh`

### How it works
This script launches a Cypress spec file (a scenario) which plots a route using one version of the routing.

Time measurements are taken during the plotting, and this scenario is run a number of times, resulting in several values for each measure which will then be averaged.

This process of repeatedly running a scenario and getting the average measurements is executed twice: first emptying the backend cache before each run, and then keeping it. This way, one can obtain average execution times when plotting a route, with and without initial backend cache.

To compare several versions of the routing, this process has to be done with each version.

## How to use it
### How to use it
To compare two versions of the route calculation, follow these steps:
1. Switch to the first version of the routing system
2. If needed, add a custom topology and scenario (see below: 'How to create a new scenario') and ensure you are using the corresponding database
Expand Down Expand Up @@ -55,7 +60,7 @@ To compare two versions of the route calculation, follow these steps:
8. **Save this output**, as the file will be overwritten next time the script is run
9. Switch to the second version of the routing system and repeat steps 3 to 8
## How to create a new scenario
### How to create a new scenario
Each scenario corresponds to a Cypress spec file you can find in `cypress/e2e/`. All the scenarios you can find there use the `generateRouteTracingTimes` custom command, which takes the name of a topology (defined as a fixture) and plots it, measuring the time between click and route display for each new marker.
#### If you simply want to use a custom topology
Expand All @@ -73,7 +78,7 @@ Each scenario corresponds to a Cypress spec file you can find in `cypress/e2e/`.
4. For taking measurements, see next section: 'How to make custom measurements'
5. Pass the path to this new spec file as argument when launching the script
## How to make custom measurements
### How to make custom measurements
During a scenario run, measurements are recorded in the `time_measures/` directory. This allows to access them later in order to compute average times.
#### Adding a Cypress measurement
Expand All @@ -97,6 +102,67 @@ All measurements for the typical scenario (described above) can be found in cust
You can simply remove the `cy.write` call as well as any storing or computing of dates and times corresponding to this call.
## Additional adjustments
### Additional adjustments
* The number of runs for each cycle (i.e. how many times a scenario is run before averaging the measurements) can be set in `benchmark.sh` by modify the value of `NB_MEASURES`
* The timeout for Cypress commands, requests, etc, and many other settings can be set in `cypress.config.js` ([click here to know more](https://docs.cypress.io/guides/references/configuration))
## `get_backend_measures.sh`
### How it works
This script works in the same manner as `benchmark.sh`, this time launching curl calls (instead of a Cypress spec file) repeatedly to allow the backend to record a number of time measurements for a request, and then computing the averages of these measures.
Again, this process is executed twice, first emptying the backend cache before each curl call, and then keeping it.
This allows to obtain average execution times for the route calculation with specific parameters, with and without initial backend cache.
### How to use it
To compare two versions of the route calculation, follow these steps:
1. Switch to the first version of the routing system
2. If needed, add the route steps (see below: 'How to use a custom set of steps') and ensure you are using the corresponding database
3. If needed, modify the measurements in the Python code in the same manner as described above in the 'How to make custom measurements' section for the `benchmark.sh` script
4. Launch the geotrek admin server
5. Go to the the `tools/benchmarking/` directory
6. Launch the script using this command:
```
./get_backend_measures.sh database session_id
```
`database`: (`"medium"` or `"big"`) which set of steps to use, corresponding to a medium or big database (see below: 'How to use a custom set of steps')
`session_id`: id of a valid session on the server (used by the script to clear the cache when needed)
7. When script execution is completed, you can find the output in `time_measure/time_averages.txt`:
```
Branch: backend_routing_benchmark
Database: big
Backend cache: false
Number of runs: 15
Python: [485806.6602389018]
JavaScript: No data
Branch: backend_routing_benchmark
Database: big
Backend cache: true
Number of runs: 15
Python: [21456.734053293865]
JavaScript: No data
```
`Branch`: the checked out branch when the script was run
`Database`: which set of steps was used, corresponding to a medium or big database
`Backend cache`: whether or not the backend cache was kept before sending the request
`Number of runs`: how many times the request was sent before averaging the time measurements
`Python`: average time measurements for Python code
`JavaScript`: not measured by this script
8. **Save this output**, as the file will be overwritten next time the script is run
9. Switch to the second version of the routing system and repeat steps 3 to 8
### How to use a custom set of steps
A set of steps is stringified JSON corresponding to the body that will be sent with the request to the `route_geometry` view. It contains the list of all the steps the route should go through. A step is defined by its latitude and longitude.
Here is an example for a route containing the start marker, a via-point and the end marker:
`'{"steps":[{"lat":48.7886,"lng":-0.7288},{"lat":48.7761,"lng":0.6948},{"lat":48.7438,"lng":0.4986}]}'`
To use a custom set of steps, you can set it as a variable in the script. To make it easier, the script includes two variables: `STEPS_MEDIUM_DB` corresponds to steps for a medium database, while `STEPS_BIG_DB` corresponds to steps for a big database. Which of these sets will be used is determined by the first command line argument `database` when launching the script (see above).

0 comments on commit e8b7b0f

Please sign in to comment.