Proposal- Porch Performance Testing Framework - Operation Flow with Prometheus

[mansoor17syed]

Proposal: Porch Performance Testing Framework with Prometheus Integration

Overview
This proposal introduces a comprehensive performance testing framework for Porch, designed to measure and monitor repository and package management operations using Prometheus. The framework provides automated setup, detailed metrics collection, and extensive reporting capabilities.

Basic Test Execution

go test -v ./... \
  --repos=2 \
  --packages=5 \
  --packageRevisions=4
  

Goals

1. Establish automated performance testing for Porch operations
2. Implement comprehensive metrics collection using Prometheus
3. Provide detailed operation monitoring and reporting
4. Enable configurable test scenarios

Implementation Details

1. Framework Components

• Automated Setup
• Prometheus Configuration
• Automatic container creation and initialization
• Custom prometheus.yml configuration
• Access: http://localhost:9090
• Metrics Server
• Port: :2113
• Endpoint: http://localhost:2113/metrics
• Logging System
• Directory: ./logs
• Format: porch-metrics-YYYY-MM-DD-HH-MM-SS.log
• Console and file logging

2. Monitored Operations

Repository Management

• Create Gitea Repository
• Create Porch Repository
• Wait Repository Ready
• Delete Repository

Package Management

• Create PackageRevision
• Update to Proposed
• Update to Published
• Delete PackageRevision

Metrics Collection

• Performance Metrics
• Operation duration tracking
• Success/failure counting
• Resource utilization monitoring
• Error tracking and reporting

Prometheus Integration

scrape_configs:
  - job_name: 'porch_metrics'
    static_configs:
      - targets: ['host.docker.internal:2113']
    scrape_interval: 1s

Benefits

• Automated performance testing
• Comprehensive metrics collection
• Real-time monitoring capabilities
• Detailed reporting and analysis
• Configurable test scenarios

Future Enhancements

• Additional metric types
• Enhanced visualization options
• Automated performance regression detection
• Extended test scenario support

[nephio-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mansoor17syed
Once this PR has been reviewed and has the lgtm label, please assign johnbelamaric for approval by writing /assign @johnbelamaric in a comment. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mansoor17syed]

Hi team,
I've worked on the Porch Performance Testing Framework with Prometheus integration. It includes automated setup, metrics collection, and detailed logging. Please review the changes and share your feedback.

[nephio-prow]

@mansoor17syed: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
presubmit-nephio-go-test	fe57069	link	true	/test presubmit-nephio-go-test

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[efiacor]

Hi @mansoor17syed .
Thanks for setting this up. Really nice to have.
Couple of points.
Is the plan for this to be run as an e2e test? If so, we should prob align with how the others are doing it.
See here for the main e2e gh action workflow.
So, it sets up the kind cluster, deploys porch with the Make target "run-in-kind", and then executes the tests.
They also have a flag to avoid these being executed as part of the unit test runs - "E2E=1 go test ..."

For interacting with gitea, we should avoid using the ip address.
https://github.com/nephio-project/porch/pull/174/files#diff-da6008a39400b7b8b7c9a7b68d4b0928faf93f44c3ca48ad1d29cb9cfb4ae40fR37

The existing e2e suite use a stubbed git deployment - https://github.com/nephio-project/porch/blob/main/test/e2e/suite.go#L556
Could we make use of this for your suite?