From e8b7b0f96e56fc40842610a2e7073eaa1ea5aec6 Mon Sep 17 00:00:00 2001 From: Justine Fricou Date: Mon, 15 Jul 2024 12:11:16 +0200 Subject: [PATCH] Update benchmark README to include the new script --- tools/benchmarking/README.md | 78 +++++++++++++++++++++++++++++++++--- 1 file changed, 72 insertions(+), 6 deletions(-) diff --git a/tools/benchmarking/README.md b/tools/benchmarking/README.md index c5e50a8a97..c0d6436503 100644 --- a/tools/benchmarking/README.md +++ b/tools/benchmarking/README.md @@ -1,8 +1,13 @@ # Benchmarking of the route calculation systems This benchmarking system allows to measure execution times (cache generation, requests, etc) to compare performances of several versions of the routing system. -## How it works -This benchmarking system uses a script that launches a Cypress spec file (a scenario) which plots a route using one version of the routing. +It uses two different scripts. The main one, `benchmark.sh`, enables to measure execution times on the frontend-side as well as on the backend-side, for each individual action taken during the route plotting. The second one, `get_backend_measures.sh`, only allows to measure backend-side times, but is much quicker due to not interacting with the frontend and allows for more freedom in which requests to measure. + + +## `benchmark.sh` + +### How it works +This script launches a Cypress spec file (a scenario) which plots a route using one version of the routing. Time measurements are taken during the plotting, and this scenario is run a number of times, resulting in several values for each measure which will then be averaged. @@ -10,7 +15,7 @@ This process of repeatedly running a scenario and getting the average measuremen To compare several versions of the routing, this process has to be done with each version. -## How to use it +### How to use it To compare two versions of the route calculation, follow these steps: 1. Switch to the first version of the routing system 2. If needed, add a custom topology and scenario (see below: 'How to create a new scenario') and ensure you are using the corresponding database @@ -55,7 +60,7 @@ To compare two versions of the route calculation, follow these steps: 8. **Save this output**, as the file will be overwritten next time the script is run 9. Switch to the second version of the routing system and repeat steps 3 to 8 -## How to create a new scenario +### How to create a new scenario Each scenario corresponds to a Cypress spec file you can find in `cypress/e2e/`. All the scenarios you can find there use the `generateRouteTracingTimes` custom command, which takes the name of a topology (defined as a fixture) and plots it, measuring the time between click and route display for each new marker. #### If you simply want to use a custom topology @@ -73,7 +78,7 @@ Each scenario corresponds to a Cypress spec file you can find in `cypress/e2e/`. 4. For taking measurements, see next section: 'How to make custom measurements' 5. Pass the path to this new spec file as argument when launching the script -## How to make custom measurements +### How to make custom measurements During a scenario run, measurements are recorded in the `time_measures/` directory. This allows to access them later in order to compute average times. #### Adding a Cypress measurement @@ -97,6 +102,67 @@ All measurements for the typical scenario (described above) can be found in cust You can simply remove the `cy.write` call as well as any storing or computing of dates and times corresponding to this call. -## Additional adjustments +### Additional adjustments * The number of runs for each cycle (i.e. how many times a scenario is run before averaging the measurements) can be set in `benchmark.sh` by modify the value of `NB_MEASURES` * The timeout for Cypress commands, requests, etc, and many other settings can be set in `cypress.config.js` ([click here to know more](https://docs.cypress.io/guides/references/configuration)) + +## `get_backend_measures.sh` + +### How it works +This script works in the same manner as `benchmark.sh`, this time launching curl calls (instead of a Cypress spec file) repeatedly to allow the backend to record a number of time measurements for a request, and then computing the averages of these measures. +Again, this process is executed twice, first emptying the backend cache before each curl call, and then keeping it. +This allows to obtain average execution times for the route calculation with specific parameters, with and without initial backend cache. + +### How to use it +To compare two versions of the route calculation, follow these steps: +1. Switch to the first version of the routing system +2. If needed, add the route steps (see below: 'How to use a custom set of steps') and ensure you are using the corresponding database +3. If needed, modify the measurements in the Python code in the same manner as described above in the 'How to make custom measurements' section for the `benchmark.sh` script +4. Launch the geotrek admin server +5. Go to the the `tools/benchmarking/` directory +6. Launch the script using this command: + ``` + ./get_backend_measures.sh database session_id + ``` + `database`: (`"medium"` or `"big"`) which set of steps to use, corresponding to a medium or big database (see below: 'How to use a custom set of steps') + + `session_id`: id of a valid session on the server (used by the script to clear the cache when needed) +7. When script execution is completed, you can find the output in `time_measure/time_averages.txt`: + ``` + Branch: backend_routing_benchmark + Database: big + Backend cache: false + Number of runs: 15 + Python: [485806.6602389018] + JavaScript: No data + + Branch: backend_routing_benchmark + Database: big + Backend cache: true + Number of runs: 15 + Python: [21456.734053293865] + JavaScript: No data + ``` + `Branch`: the checked out branch when the script was run + + `Database`: which set of steps was used, corresponding to a medium or big database + + `Backend cache`: whether or not the backend cache was kept before sending the request + + `Number of runs`: how many times the request was sent before averaging the time measurements + + `Python`: average time measurements for Python code + + `JavaScript`: not measured by this script +8. **Save this output**, as the file will be overwritten next time the script is run +9. Switch to the second version of the routing system and repeat steps 3 to 8 + +### How to use a custom set of steps +A set of steps is stringified JSON corresponding to the body that will be sent with the request to the `route_geometry` view. It contains the list of all the steps the route should go through. A step is defined by its latitude and longitude. + +Here is an example for a route containing the start marker, a via-point and the end marker: + +`'{"steps":[{"lat":48.7886,"lng":-0.7288},{"lat":48.7761,"lng":0.6948},{"lat":48.7438,"lng":0.4986}]}'` + +To use a custom set of steps, you can set it as a variable in the script. To make it easier, the script includes two variables: `STEPS_MEDIUM_DB` corresponds to steps for a medium database, while `STEPS_BIG_DB` corresponds to steps for a big database. Which of these sets will be used is determined by the first command line argument `database` when launching the script (see above). +