Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP feat(backend + sdk): Implement Pipeline Configuration with TTL #11269

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

rimolive
Copy link
Member

@rimolive rimolive commented Oct 3, 2024

Description of your changes:
This PR introduces a new class PipelineConfig in KFP SDK to add pipeline-specific configurations with the first one: Pipeline TTL.

Supersedes #11112

Testing instructions

SDK

  • Create a Python virtualenv and install the SDK and IR YAML API packages locally
$ python -m venv .venv
$ source .venv/bin/activate
$ pip install wheel setuptools protobuf==4.21.1 grpcio grpcio-tools
$ pip install -r sdk/python/requirements-dev.txt
$ pip install -e api/v2alpha1/python
$ pip install -e sdk/python
$ kfp dsl compile --py main.py --output main.yaml
  • You should be able to compile and find the following snippet in the main.yaml file:
---
platforms:
  kubernetes:
    pipelineConfig:
      pipelineTtl: 60

Backend

  • Build the API Server image and push to an image registry
  • Upload main.yaml file generated in the SDK section
  • Check in KFP UI Pipeline Spec tab if the following snippet is present:
platform_spec:
  platforms:
    kubernetes:
      pipelineConfig:
        pipelineTtl: 60
  • Create a run and check if TTL configuration is present in Workflow CR:
$ oc get workflow -o yaml $(oc get workflow --no-headers | awk '{print $1}') | yq .spec.ttlStrategy
secondsAfterCompletion: 60

Checklist:

Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chensun for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@HumairAK HumairAK added this to the KFP 2.4.0 milestone Oct 8, 2024
@rimolive rimolive force-pushed the rmartine-upstream branch 10 times, most recently from 0adc2bf to 70f4d6d Compare October 22, 2024 19:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants