Skip to content

Latest commit

 

History

History
480 lines (337 loc) · 16.4 KB

README.md

File metadata and controls

480 lines (337 loc) · 16.4 KB
Raw

TimeClimb

TimeClimb is an app that visualizes AWS Step Function latency averages over custom time ranges (day, week, month, or year), displaying overall step function and individual step latencies. Users can view line charts for time-based latency trends and click on individual steps to see detailed latency metrics on a line chart. A heat map displaying each step's latencies over the selected period of time will display by default.

Website: https://www.timeclimb.com/

GitHub: https://github.com/oslabs-beta/TimeClimb.git

Features

Database

This application is deployed locally on each user's machine using Docker. As long as the user has PostgresDB installed on their machine and has provide the required credentials in a .env file, the program will create a local database in which to store AWS State Machine (Step Function) latency averages.

A user will select which State Machine they would like to see latency averages for by providing the State Machine's arn. As long as the correct permissions have been set in AWS for this user (details found in User Setup below) this application will query the database for latency averages for the desired State Machine and display it.

A user will select a period of time for which they would like to view their State Machine's latency averages in a drop down. The application will display the following based on the user's input:

  • Hours : Will display hourly averages over a 24 hour period
  • Days : Will display hourly averages over a 7 day period
  • Weeks : Will display hourly averages over a 12 week period
  • Months : Will display hourly averages over a 12 month period

A user will use a slider to view State Machine averages at certain points within the desired time period

A user may view a line graph of the State Machine's latency averages in a line graph

A user may view the individual Action's (Step's) average latencies in the form of a heat map

A user may view a line graph of each Action's average latencies by selecting the desired Action in the display

Demonstrations

Choose a time period

A user will select a period of time for which they would like to view their State Machine's latency averages in a drop down.

Select desired hour, day, week or month

Use slider to view latencies at a specific hour, day, week or month.

View individual step latency averages as chart

To view the latency averages for each indivual action (step) as a chart, click on desired action.

View individual step latency averages on a heat map

To view the latency averages for each indivual action (step) on head map, hover over desired action within desired time period.

User Setup

1. Important AWS Information

Permissions

To grant users permission to perform actions on the resources required for this application, an IAM administrator will need to create an IAM policy for CloudWatch Logs that includes the actions FilterLogEvents as well as DescribeLogGroups. A Step Functions policy will also need to be created including the action DescribeStateMachine. These policies can then be attached to individual users, groups, or roles. Click here to learn more about how to create policy's, users and roles.

AWS Pricing

Take note of AWS Pricing associated with the use of this application

  • Log Ingestion: This application uses CloudWatch to retrieve information about your State Machines. For State Machines, ingestion occurs on each execution start and end events as well as one each Step-level event
  • Log Storage over time: Once logs are ingested into CloudWatch, they incur storage costs over time. Stored data is charged based on the total GB stored. AWS charges $0.50 per GB of data ingested into CloudWatch Logs. AWS provides the first 5 GB of ingested log data per month for free.
  • Log Data Scanning: This application uses FilteredLogEvents for API calls to AWS which filters log queries- helping reduce costs. Charges are based on the total volume of log data scanned, not the volume of data returned. AWS charges $0.005 per GB of data scanned. AWS offers a free tier covering the first 5 GB of scanned data per month and up to 1 million API requests for free.
  • Data Transfer Costs: AWS charges $0.09 per GB for data transferred from CloudWatch to your application. The first 100 GB per month of data transfer out is free

Note: TimeClimb is not responsible for any incurred cost for using this software. This pricing information is included for convenience and may not be up to date with AWS's current pricing. See links below for detail information:

AWS CloudWatch Pricing: https://aws.amazon.com/cloudwatch/pricing/?nc1=h_ls

EC2: Data Transwer Pricing: https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer

3. Insert AWS Access Key and Secret Access Key

- AWS_ACCESS_KEY_ID=[INSERT_YOUR_AWS_KEY_ID]
- AWS_SECRET_ACCESS_KEY=[INSERT_YOUR_AWS_SECRET_KEY]
- AWS_REGION=[INSERT_YOUR_AWS_REGION]

4. Ensure Docker Desktop is running on your machine.

5. Run docker compose up in your terminal

Contribution Information:

Creating and Updating the database time_climb

Note: See complete example of .env file in .env.example

1. Create a .env file and insert environment variables to file

Note: Be sure this file is added to your gitignore

Example:

          PGHOST='localhost'
          PGPORT='5432'
          PGUSER='yourUserName'
          PGPASSWORD='yourPassword'
2. Insert AWS Access Key and Secret Access Key

Note: Be sure this file is added to your gitignore

Example:

        AWS_ACCESS_KEY_ID=yourAcessKey
        AWS_SECRET_ACCESS_KEY=yourSecretAccessKey
3. Seed dummy data into database

Note: These migrations will happen through Knex Library https://knexjs.org/. Knex helps to document and automate database schema changes and data migrations over time. This library will also be used for queries made to database in models.

Note: If you do not already have PostgresQL on your machine, you will need to download it.

Once Credentials have been added to example.env:

Populating with Seed Data

npm run seed:run will populate the database with test data. This will erase any existing data you might have in your database.

Updating database with Seed Data

npm run migrate:latest to update the database with the latest schema.

Updating to the latest migration will typically erase all data in your database unless the migration script specifically is set up to migrate your data, which it currently is not.

Knex Detailed Usage Convenience scripts are set up in package.json to run common knex migration functions.
  • npm run migrate:make Creating a migration with npm run migrate:make
  • npm run migrate:latest Updating the database with the latest schema changes
  • npm run migrate:rollback Rolling back a change to the database

These are partially necessary because both the configuration files and migrations are written in TypeScript. Knex's command line tool does not natively support TypeScript.

Creating a migration

Creating a new migration will place a new file in /database/migrations

You can pass the name of the file after the script like so:

npm run migrate:make -- your_migration_file_name

Updatating the database to the latest schema changes

npm run migrate:latest

This will update the database with latest schema changes. If necessary its possible to also migrate data so that the data is not lost, which might be especially helpful in a production environment.

Rolling back a migration

npm run migrate:rollback

This will roll back only the very latest migration unit. You can pass in an argument --all to rollback all of the completed migrations this way:

npm run migrate:rollback -- --all

  1. Runnpm run dev

  2. Runnpm run dev:server

API Documentation

Base URL

http://localhost:3000/api

All API requests are made to http://localhost:3000/api

Endpoints for adding/retrieving Step Functions to/from database

GET/Gets all step functions previously stored in the database

Parameters

None

Responses

http code content-type response
200 application/json;charset=UTF-8 JSON

Example Status Code for 200 Ok

[
  {
    "step_function_id": 0,
    "name": "string",
    "description": "string",
    "definition": {}
  }
]
POST/step_functions/addStepFunction Adds a step function to the database

Parameters

| name | type | data type | description | | ---- | -------- | --------- | ---------------------------------------------------- | --- | | body | required | object | the arn that corresponds to a specific state machine | |

Example Body - JSON

{ "arn": "arn:partition:service:region:account-id:resource-type:resource-id" }

Responses

http code content-type response
200 application/json;charset=UTF-8 JSON
400 application/json;charset=UTF-8 JSON
401 application/json;charset=UTF-8 JSON

Example Response for 200 Ok: Returns the newly added step function

{
  "step_function_id": 0,
  "name": "string",
  "definition": {}
}
GET/step_functions/:step_functions_id/hours Retrieves hourly average latencies over a span of one day in ascending order

Parameters

name type data type description
path.step_function_id required string The unique ID associated with this step function in database passed in the URL path (/:step_function_id/hours)

Example Request

localhost:3000/api/average-latencies/:step_function_id/hours

Responses

http code content-type response
200 application/json;charset=UTF-8 JSON

Example Response for 200 Ok

Note: If a step function's latencies are not found in database, the elements value in the response will be an empty object
[
  {
    "date": "2024-10-23T04:00:00.000Z",
    "stepFunctionAverageLatency": 18.144353388646543,
    "steps": {
      "Start Task And Wait For Callback": {
        "average": 14.44404914949289
      },
      "Notify Success": {
        "average": 0.6627415526268704
      },
      "Notify Failure": {
        "average": 3.037562686526782
      }
    }
  }
]
GET/step_functions/:step_functions_id/days Retrieves daily average latencies over a span of 7 days in ascending order

Parameters

name type data type description
path.step_function_id required string The unique ID associated with this step function in database passed in the URL path (/:step_function_id/days)

Example Request

localhost:3000/api/average-latencies/:step_function_id/days

Responses

http code content-type response
200 application/json;charset=UTF-8 JSON

Example Response for 200 Ok

Note: If a step function's latencies are not found in database, the elements value in the response will be an empty object
[
  {
    "date": "2024-10-23T04:00:00.000Z",
    "stepFunctionAverageLatency": 18.144353388646543,
    "steps": {
      "Start Task And Wait For Callback": {
        "average": 14.44404914949289
      },
      "Notify Success": {
        "average": 0.6627415526268704
      },
      "Notify Failure": {
        "average": 3.037562686526782
      }
    }
  }
]
GET/step_functions/:step_functions_id/weeks Retrieves weekly average latencies over a span of 12 weeks in ascending order

Parameters

name type data type description
path.step_function_id required string The unique ID associated with this step function in database passed in the URL path (/:step_function_id/weeks)

Example Request

localhost:3000/api/average-latencies/:step_function_id/weeks

Responses

http code content-type response
200 application/json;charset=UTF-8 JSON

Example Response for 200 Ok

Note: If a step function's latencies are not found in database, the elements value in the response will be an empty object
[
  {
    "date": "2024-08-12T04:00:00.000Z",
    "stepFunctionAverageLatency": 18.144353388646543,
    "steps": {
      "Start Task And Wait For Callback": {
        "average": 14.44404914949289
      },
      "Notify Success": {
        "average": 0.6627415526268704
      },
      "Notify Failure": {
        "average": 3.037562686526782
      }
    }
  }
]
GET/step_functions/:step_functions_id/months Retrieves monthly average latencies over a span of 12 months in ascending order

Parameters

name type data type description
path.step_function_id required string The unique ID associated with this step function in database passed in the URL path (/:step_function_id/months)

Example Request

localhost:3000/api/average-latencies/:step_function_id/months

Responses

http code content-type response
200 application/json;charset=UTF-8 JSON

Example Response for 200 Ok

Note: If a step function's latencies are not found in database, the elements value in the response will be an empty object
[
  {
    "date": "2023-11-01T04:00:00.000Z",
    "stepFunctionAverageLatency": 18.144353388646543,
    "steps": {
      "Start Task And Wait For Callback": {
        "average": 14.44404914949289
      },
      "Notify Success": {
        "average": 0.6627415526268704
      },
      "Notify Failure": {
        "average": 3.037562686526782
      }
    }
  }
]

Contributers

Austin Cheng Andrew Mott Paul Uhlenkott Sharon Patterson Alex Stewart