A simple yet powerful open-source framework that integrates all your ML tools.
Explore the docs »
Slack Community and be part of the ZenML family.
Features · Roadmap · Report Bug · Vote New Features · Read Blog · Meet the Team
🎉 Version 0.20.5 is out. Check out the release notes here.
🏁 Table of Contents
🤹 Are you an ML engineer or data scientist shipping models to production and juggling a plethora of tools? ZenML is here to help transition your production ML pipelines to production.
Everyone loves to train ML models, but few talks about shipping them into production, and even fewer can do it well. At ZenML, we believe the journey from model development to production doesn't need to be long and painful.
If you come from unstructured notebooks or scripts with lots of manual processes, ZenML will make the path to production easier and faster for you and your team. Using ZenML allows you to own the entire pipeline - from experimentation to production.
Read more here.
ZenML is an extensible, open-source MLOps framework for creating portable, production-ready MLOps pipelines. It's built for Data Scientists, ML Engineers, and MLOps Developers to collaborate as they develop to production.
ZenML offers a simple and flexible syntax, is cloud- and tool-agnostic, and has interfaces/abstractions catered toward ML workflows. With ZenML you'll have all your favorite tools in one place, so you can tailor a workflow that caters to your specific needs.
Read more on all tools you can readily use in the integrations section. Can't find your tool? You can always write your own integration to use it with ZenML.
Option 1 - Install ZenML via PyPI:
pip install zenmlNote - ZenML supports Python 3.7, 3.8, and 3.9.
In order to get access to the dashboard locally, install the optional dependencies for the ZenML server:
pip install "zenml[server]"Option 2 - If you’re feeling adventurous, try out the bleeding-edge installation:
pip install git+https://github.com/zenml-io/zenml.git@develop --upgradeWarning - Fire dragons ahead. Proceed at your own risk!
Option 3 - Install via a Docker image hosted publicly on DockerHub:
docker run -it zenmldocker/zenml /bin/bashor just run the ZenML server directly with Docker:
docker run -it -d -p 8080:80 zenmldocker/zenml-serverWarning
If you have an M1 Mac machine and encounter an installation error, try setting up
brewandpyenvwith Rosetta 2 and then install ZenML. The issue arises because some dependencies aren’t fully compatible with the vanilla ARM64 Architecture. The following links may be helpful (Thank you @Reid Falconer) :
Read full installation instructions in the docs.
If you're here for the first time, we recommend running:
zenml goThis spins up a Jupyter notebook that walks you through various functionalities of ZenML at a high level.
By the end, you'll get a glimpse of how to use ZenML to register a stack:
# Register a MLflow experiment tracker
zenml experiment-tracker register mlflow_tracker --flavor=mlflow
# Register an Airflow orchestrator
zenml airflow register airflow_orchestrator --flavor=airflow
# Create a stack from the components
zenml stack register my_stack -o airflow_orchestrator -a default -e mlflow_trackerAnd run a simple pipeline on Airflow (or an orchestrator of your choice) creating steps like this:
import mlflow
from sklearn.base import ClassifierMixin
from sklearn.svm import SVC
from zenml.client import Client
experiment_tracker = Client().active_stack.experiment_tracker
@step(experiment_tracker=experiment_tracker.name)
def svc_trainer_mlflow(
X_train: pd.DataFrame,
y_train: pd.Series,
) -> ClassifierMixin:
"""Train a sklearn SVC classifier and log to MLflow."""
mlflow.sklearn.autolog() # log all model hparams and metrics to MLflow
model = SVC(gamma=0.01)
model.fit(X_train.to_numpy(), y_train.to_numpy())
train_acc = model.score(X_train.to_numpy(), y_train.to_numpy())
print(f"Train accuracy: {train_acc}")
return modelYou can also run your first pipeline right in Google Colab
ZenML is built to support teams working together. The underlying infrastructure on which your ML workflows run can be shared, as can the data, assets, and artifacts in your workflow.
To visually see this in action, ZenML ships with a fully-featured dashboard, that can be launched locally with the simple command:
zenml up
The dashboard can also be deployed with a server on any cloud service (see Deploy ZenML section).
New to MLOps? Get up to speed by visiting the ZenBytes repo.
ZenBytes is a series of short practical MLOps lessons taught using ZenML. It covers many of the core concepts widely used in ZenML and MLOps in general.
Already comfortable with ZenML and wish to elevate your pipeline into production mode? Check out ZenFiles.
ZenFiles is a collection of production-grade ML use-cases powered by ZenML. They are fully fleshed out, end-to-end projects that showcase ZenML's capabilities. They can also serve as a template from which to start similar projects.
While ZenML works perfectly fine locally, it should be deployed on the cloud to enable collaborative features as the central MLOps interface for teams.
The easiest and fastest way to get running on the cloud is by using the deploy CLI command. It currently only supports deploying to Kubernetes on managed cloud services. Read more about the command here.
zenml deployNaturally, the docs contain detailed guides on how to manually set up ZenML on the cloud in case more control is required.
The following architectural diagram illustrates a common ZenML deployment:
Apart from the infrastructure required to run ZenML itself, ZenML also boasts a ton of integrations into popular MLOps tools. The ZenML Stack concept ensures that these tools work nicely together, therefore bringing structure and standardization into the MLOps workflow.
However, ZenML assumes that the stack infrastructure for these tools is already provisioned. If you do not have deployed infrastructure, and want to quickly spin up combinations of tools on the cloud, the MLOps stack sister repository contains a series of Terraform-based recipes to provision such stacks. These recipes can be used directly with ZenML:
pip install "zenml[stacks]"
zenml stack recipe deploy <NAME_OF_STACK_RECIPE> --importThe above command not only provisions the given tools, but also automatically creates a ZenML stack with the configuration of the deployed recipe!
ZenML is being built in public. The roadmap is a regularly updated source of truth for the ZenML community to understand where the product is going in the short, medium, and long term.
ZenML is managed by a core team of developers that are responsible for making key decisions and incorporating feedback from the community. The team oversees feedback via various channels, and you can directly influence the roadmap as follows:
- Vote on your most wanted feature on our Discussion board.
- Start a thread in our Slack channel.
- Create an issue on our Github repo.
We would love to develop ZenML together with our community! Best way to get
started is to select any issue from the good-first-issue
label. If you
would like to contribute, please review our Contributing
Guide for all relevant details.
Have a question that's too hard to express on our Slack? Is it just too much effort to say everything on a long GitHub issue? Or are you just curious about what ZenML has been up to in the past week? Well, register now for the ZenML Office (Half) Hour to get your answers and more! It's free and open to everyone.
Every week, part of the ZenML core team will pop in for 30 minutes to interact directly with the community. Sometimes we'll be presenting a feature. Other times we just take questions and have fun. Join us if you are curious about ZenML, or just want to talk shop about MLOps.
We will host the gathering every Wednesday 8:30AM PT (5:30PM CET). Register now through this link, or subscribe to the public events calendar to get notified before every community gathering.
The first point of call should be our Slack group. Ask your questions about bugs or specific use cases, and someone from the core team will respond. Or, if you prefer, open an issue on our GitHub repo.
ZenML is distributed under the terms of the Apache License Version 2.0. A complete version of the license is available in the LICENSE file in this repository. Any contribution made to this project will be licensed under the Apache License Version 2.0.
| ZenML Resources | Description |
|---|---|
| 🧘♀️ ZenML 101 | New to ZenML? Here's everything you need to know! |
| ⚛️ Core Concepts | Some key terms and concepts we use. |
| 🚀 Our latest release | New features, bug fixes. |
| 🗳 Vote for Features | Pick what we work on next! |
| 📓 Docs | Full documentation for creating your own ZenML pipelines. |
| 📒 API Reference | Detailed reference on ZenML's API. |
| 🍰 ZenBytes | A guided and in-depth tutorial on MLOps and ZenML. |
| 🗂️️ ZenFiles | End-to-end projects using ZenML. |
| 👨🍳 MLOps Stacks | Terraform based infrastructure recipes for pre-made ZenML stacks. |
| ⚽️ Examples | Learn best through examples where ZenML is used? We've got you covered. |
| 📬 Blog | Use cases of ZenML and technical deep dives on how we built it. |
| 🔈 Podcast | Conversations with leaders in ML, released every 2 weeks. |
| 📣 Newsletter | We build ZenML in public. Subscribe to learn how we work. |
| 💬 Join Slack | Need help with your specific use case? Say hi on Slack! |
| 🗺 Roadmap | See where ZenML is working to build new features. |
| 🙋♀️ Contribute | How to contribute to the ZenML project and code base. |