Skip to content

Commit

Permalink
add an include variable to start tests
Browse files Browse the repository at this point in the history
  • Loading branch information
@abdelaziz-mahdy
abdelaziz-mahdy committed Dec 18, 2024
1 parent ba1f2b2 commit e58d9dc
Show file tree
Hide file tree
Showing 2 changed files with 162 additions and 53 deletions.
118 changes: 99 additions & 19 deletions Contribution.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Overview

This guide explains how to contribute to the backend benchmarking project by adding or removing frameworks, running tests, and configuring the benchmarking environment.
This guide provides details on how to contribute to the backend benchmarking project by adding or removing frameworks, running tests, and configuring the benchmarking environment. It also explains how to use the `start_tests.sh` script to include specific frameworks for testing.

---

Expand All @@ -12,7 +12,7 @@ To add a new framework:

1. **Create the Framework Directory Structure**:
- Use an existing framework directory as a reference.
- For example:
- Example structure:
```
backends/<language>/<framework_name>
backends/<language>/<framework_name>/<the app>
Expand All @@ -23,7 +23,7 @@ To add a new framework:
2. **Prepare Scripts**:
- Copy the `docker_build_and_run.sh` from an existing framework. Minimal edits are needed to adjust paths.
- Prepare a `docker-compose.yml` file with standard services:
- Prepare a `docker-compose.yml` file with the required services:
- Database
- Benchmark
- Tester (e.g., Locust)
Expand Down Expand Up @@ -72,29 +72,77 @@ To add a new framework:
```
backends/<language>/<framework_name>
```
- **Option 2**: Mark the framework directory to be ignored in `scripts/start_tests.sh`.
- **Option 2**: Modify the `INCLUDE` variable in the `start_tests.sh` script to exclude the framework by not listing it.
---
## Running Tests
The main script for launching tests is `scripts/start_tests.sh`.
The benchmarking process is managed using the `scripts/start_tests.sh` script. This script identifies frameworks to test and supports running all frameworks or only specific frameworks specified in the `INCLUDE` variable.
- **Run All Benchmarks**:
```bash
bash scripts/start_tests.sh
```
### Using `start_tests.sh`
- **Purpose**: Runs benchmarks for all frameworks or only the ones specified in the `INCLUDE` variable.
---
### Examples
- **Customize Runtime**:
- Example: Run benchmarks with a runtime of 10 seconds.
#### 1. Running All Frameworks
If you want to run all frameworks, do not set the `INCLUDE` variable.
**Command**:
```bash
bash scripts/start_tests.sh
```

**Output**:
```
Warning: INCLUDE variable not set. All scripts will be included.
Found 10 scripts in total:
./backends/go/mux/docker_build_and_run.sh
./backends/python/flask/docker_build_and_run.sh
...
```

---

#### 2. Including Specific Frameworks

To run tests for only specific frameworks or languages, set the `INCLUDE` variable with a comma-separated list of patterns to match.

**Command**:
```bash
INCLUDE="python,go" bash scripts/start_tests.sh
```

**Output**:
```
Found 10 scripts in total:
./backends/go/mux/docker_build_and_run.sh
./backends/python/flask/docker_build_and_run.sh
Including ./backends/go/mux/docker_build_and_run.sh as it matches include pattern go
Including ./backends/python/flask/docker_build_and_run.sh as it matches include pattern python
Found scripts 10 and after filtering 2 scripts to run:
./backends/go/mux/docker_build_and_run.sh
./backends/python/flask/docker_build_and_run.sh
```

---

### Customizing Test Configurations

- **LOCUST_RUNTIME**:
- Defines the runtime for each test in seconds.
- Example:
```bash
LOCUST_RUNTIME=10 bash scripts/start_tests.sh
```

- **Set Locust Variables**:
- Adjust the following environment variables as needed:
- `LOCUST_USERS`: Number of simulated users. Default: 10,000.
- `LOCUST_SPAWN_RATE`: User spawn rate per second. Default: 10.
- **Other LOCUST Variables**:
- `LOCUST_USERS`: Number of simulated users (Default: 10,000).
- `LOCUST_SPAWN_RATE`: User spawn rate per second (Default: 10).

Example:
```bash
Expand All @@ -107,7 +155,7 @@ The main script for launching tests is `scripts/start_tests.sh`.

- The environment variables required for tests are set by `internal_scripts/set_required_envs.sh`.
- [Link to `set_required_envs.sh`](https://github.com/abdelaziz-mahdy/backend-benchmark/blob/main/internal_scripts/set_required_envs.sh)
- Pass the required variables to `start_tests.sh`:
- Pass the required variables to any script:
```bash
source internal_scripts/set_required_envs.sh
```
Expand Down Expand Up @@ -138,9 +186,41 @@ backends/go/mux/
---
## Notes
## Notes on Scripts
### `start_tests.sh`
- **Includes logic to**:
- Search for all `docker_build_and_run.sh` files.
- Include scripts matching patterns in the `INCLUDE` variable.
- Run the filtered scripts for `db_test` and `no_db_test`.
---
## Example Workflow
### Running All Tests:
```bash
bash scripts/start_tests.sh
```

### Including Specific Frameworks:
To include only `python` and `go` frameworks:
```bash
INCLUDE="python,go" bash scripts/start_tests.sh
```

---

## Contribution Guidelines

- The `internal_scripts` directory contains reusable scripts that prevent duplication across frameworks.
- Example `docker-compose.yml` configurations should be identical except for framework-specific paths or environment variables.
- **Adding a Framework**:
- Place the `docker_build_and_run.sh` script in the corresponding language directory.
- Example structure:
```
backends/<language>/<framework>/docker_build_and_run.sh
```
---
Contributions that follow this guide will ensure consistency and maintainability. If you encounter any issues, feel free to raise them in the project's repository!
97 changes: 63 additions & 34 deletions scripts/start_tests.sh
View file
Original file line number Diff line number Diff line change
@@ -1,13 +1,6 @@
#!/bin/bash

# Script name: run_docker_build_and_run.sh
#!/bin/bash


#!/bin/bash
#!/bin/bash


# Script name: start_tests.sh

# Save the current directory
dir=$(pwd)
Expand All @@ -17,64 +10,100 @@ found=0

# Loop through the current and parent directories
while true; do
# Check if the script exists in the current directory
if [[ -f "$dir/internal_scripts/set_required_envs.sh" ]]; then
# Script found, execute it
echo "Found script at $dir/internal_scripts/set_required_envs.sh"
. "$dir/internal_scripts/set_required_envs.sh"
found=1
break
else
# Move to the parent directory
parentdir=$(dirname "$dir")
# Check if we have reached the root directory
if [[ "$dir" == "$parentdir" ]]; then
break
# Check if the script exists in the current directory
if [[ -f "$dir/internal_scripts/set_required_envs.sh" ]]; then
echo "Found script at $dir/internal_scripts/set_required_envs.sh"
. "$dir/internal_scripts/set_required_envs.sh"
found=1
break
else
# Move to the parent directory
parentdir=$(dirname "$dir")
if [[ "$dir" == "$parentdir" ]]; then
break
fi
dir=$parentdir
fi
dir=$parentdir
fi
done

# Check if script was not found
if [[ $found -eq 0 ]]; then
echo "Script not found."
exit 1
echo "Required environment script not found."
exit 1
fi


cd ..
# Find all 'docker_build_and_run.sh' files and store them in an array

# Check if INCLUDE variable is set
if [[ -z "$INCLUDE" ]]; then
echo "Warning: INCLUDE variable not set. All scripts will be included."
INCLUDE=""
fi

# Convert INCLUDE variable to an array
IFS=',' read -ra include_patterns <<< "$INCLUDE"

# Find all 'docker_build_and_run.sh' files
scripts=($(find . -name 'docker_build_and_run.sh'))

echo "Found ${#scripts[@]} scripts in total:"
echo "${scripts[@]}"

# Filter scripts to include only matching the patterns in INCLUDE
filtered_scripts=()

# If INCLUDE list is empty, include all scripts
if [[ ${#include_patterns[@]} -eq 0 ]]; then
echo "INCLUDE list is empty. Including all scripts."
filtered_scripts=("${scripts[@]}")
else
for script in "${scripts[@]}"; do
include_flag=0
for pattern in "${include_patterns[@]}"; do
if echo "$script" | grep -wq "$pattern"; then
include_flag=1
echo "Including $script as it matches include pattern $pattern"
break
fi
done
if [[ $include_flag -eq 1 ]]; then
filtered_scripts+=("$script")
fi
done
fi


echo "#########################################################################"
echo "Found scripts ${#scripts[@]} and after filtering ${#filtered_scripts[@]} scripts to run:"
echo "${filtered_scripts[@]}"

# Get total number of scripts and multiply by 2 for the two test types
total_scripts=$((${#scripts[@]} * 2))
total_scripts=$((${#filtered_scripts[@]} * 2))

# Initialize a counter
counter=1

# Define the test types
test_types=("db_test" "no_db_test")

for script in "${scripts[@]}"; do
for script in "${filtered_scripts[@]}"; do
for test_type in "${test_types[@]}"; do
# Calculate total remaining runtime
remaining_tests=$((total_scripts - counter + 1))
total_remaining_seconds=$((remaining_tests * LOCUST_RUNTIME))
minutes=$((total_remaining_seconds / 60))
seconds=$((total_remaining_seconds % 60))

echo "Running script $counter out of $total_scripts: $script with test_type=$test_type"
echo "Estimated remaining time: $minutes minutes, $seconds seconds"

export test_type=$test_type
bash "$script"
echo "Finished running: $script with test_type=$test_type"

# Increment the counter
((counter++))
echo "Sleeping for 5 seconds..."
sleep 5
done
done

cd scripts/graphs
bash create_graphs.sh
bash create_graphs.sh

0 comments on commit e58d9dc

Please sign in to comment.