Skip to content
forked from PrefectHQ/prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

License

Notifications You must be signed in to change notification settings

opsmill/prefect

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

PyPI

Prefect

Prefect is a workflow orchestration framework for building data pipelines in Python. It's the simplest way to elevate a script into an interactive workflow application. With Prefect, you can build resilient, dynamic workflows that react to the world around them and recover from unexpected changes.

With just a few decorators, Prefect supercharges your code with features like automatic retries, distributed execution, scheduling, caching, and much more.

Workflow activity is tracked and can be monitored with a self-hosted Prefect server instance or managed Prefect Cloud dashboard.

Getting started

Prefect requires Python 3.9 or later. To install the latest or upgrade to the latest version of Prefect, run the following command:

pip install -U prefect

Then create and run a Python file that uses Prefect flow and task decorators to orchestrate and observe your workflow - in this case, a simple script that fetches the number of GitHub stars from a repository:

from prefect import flow, task
from typing import list
import httpx


@task(log_prints=True)
def get_stars(repo: str):
    url = f"https://api.github.com/repos/{repo}"
    count = httpx.get(url).json()["stargazers_count"]
    print(f"{repo} has {count} stars!")


@flow(name="GitHub Stars")
def github_stars(repos: list[str]):
    for repo in repos:
        get_stars(repo)


# run the flow!
if __name__=="__main__":
    github_stars(["PrefectHQ/Prefect"])

Fire up the Prefect UI to see what happened:

prefect server start

To run your workflow on a schedule, turn it into a deployment and schedule it to run every minute by changing the last line of your script to the following:

if __name__ == "__main__":
    github_stars.serve(name="first-deployment", cron="* * * * *")

You now have a server running locally that is looking for scheduled deployments! Additionally you can run your workflow manually from the UI or CLI - and if you're using Prefect Cloud, you can even run deployments in response to events.

Prefect Cloud

Prefect Cloud allows you to centrally deploy, monitor, and manage the data workflows you support. With managed orchestration, automations, and webhooks, all backed by enterprise-class security, build production-ready code quickly and reliably.

Read more about Prefect Cloud here or sign up to try it for yourself.

prefect-client

If your use case is geared towards communicating with Prefect Cloud or a remote Prefect server, check out our prefect-client. It is a lighter-weight option for accessing client-side functionality in the Prefect SDK and is ideal for use in ephemeral execution environments.

Next steps

About

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Python 98.8%
  • Vue 0.9%
  • TypeScript 0.2%
  • Shell 0.1%
  • Jinja 0.0%
  • Lua 0.0%