0.20.5

ZenML 0.20.5 fixes another series of minor bugs, significantly improves the performance of the CLI, and adds an option to specify APT packages in Docker images.

What's Changed

Fix accessing local zen store and artifact store in containers by @stefannica in zenml-io#976
K3d local registry pod spec updated by @wjayesh in zenml-io#972
Update readme page by @dnth in zenml-io#985
Remove beam dependency by @schustmi in zenml-io#986
Fix error message when registering secret without secrets manager by @schustmi in zenml-io#981
Update cheat sheet up to zenml==0.20.4 by @dnth in zenml-io#987
Example fixes (part 2) by @strickvl in zenml-io#971
Allow duplicate step classes inside a pipeline by @schustmi in zenml-io#989
Include deployment in azureml docker build by @schustmi in zenml-io#984
Automatically open browser upon zenml up command by @dnth in zenml-io#978
Add a just_mine flag for zenml stack list by @strickvl in zenml-io#979
Add option to specify apt packages by @schustmi in zenml-io#982
Replace old flavor references, fix the windows local ZenML server and other fixes by @stefannica in zenml-io#988
Improve docker and k8s detection by @schustmi in zenml-io#991
Update GH actions example by @schustmi in zenml-io#993
Update MissingStepParameterError exception message by @gabrielmbmb in zenml-io#996
Seprated code docs into core and integration docs by @AlexejPenner in zenml-io#983
Add docs/mkdocstrings_helper.py to format script sources by @fa9r in zenml-io#997
Further CLI optimization by @bcdurak in zenml-io#992

Full Changelog: https://github.com/zenml-io/zenml/compare/0.20.4...0.20.5

0.20.4

This release fixes another series of minor bugs that were introduced in 0.20.0.

What's Changed

Detect failed executions by @schustmi in zenml-io#964
Only build docker images for custom deployments by @schustmi in zenml-io#960
M1 Mac Installation Tutorial by @fa9r in zenml-io#966
Update ZenBytes links in docs by @fa9r in zenml-io#968
Fix the API docs builder by @stefannica in zenml-io#967
Fix gpu_limit condition in VertexOrchestrator by @gabrielmbmb in zenml-io#963
Add simple node affinitiy configurations by @schustmi in zenml-io#973
First iteration of the CLI optimization by @bcdurak in zenml-io#962

Full Changelog: https://github.com/zenml-io/zenml/compare/0.20.3...0.20.4

0.20.3

This release fixes another series of minor bugs that were introduced in 0.20.0.

What's Changed

Fixed GitHub/Colab JSON formatting error on quickstart. by @fa9r in zenml-io#947
Update YAML config template by @htahir1 in zenml-io#952
correct code from merge and fix import by @wjayesh in zenml-io#950
Check for active component using id instead of name by @schustmi in zenml-io#956
Tekton fix by @htahir1 in zenml-io#955
Improve zenml up/down UX and other fixes by @stefannica in zenml-io#957
Update kubeflow docs for multi-tenant deployments by @htahir1 in zenml-io#958
Update kubeflow.md by @abohmeed in zenml-io#959
Add additional stack validation for step operators by @schustmi in zenml-io#954
Fix pipeline run dashboard URL for unlisted runs by @fa9r in zenml-io#951
Support subclasses of registered types in recursive materialization by @fa9r in zenml-io#953

New Contributors

@abohmeed made their first contribution in zenml-io#959

Full Changelog: https://github.com/zenml-io/zenml/compare/0.20.2...0.20.3

0.20.2

After a successful release of the new ZenML server and dashboard paradigm, we set to ironing out some bugs that slipped through.

What's Changed

Capitalize all docs page titles. by @fa9r in zenml-io#937
Increase field sizes for docstrings and step parameters. by @fa9r in zenml-io#940
Fixing the bug in the registration of custom flavors by @bcdurak in zenml-io#938
Implemented docstring Attribute of StepModel by @fa9r in zenml-io#936
Fix shared stack emoji by @strickvl in zenml-io#941
Fix shared stacks not being allowed to be set as active. by @fa9r in zenml-io#943
Typo fix by @strickvl in zenml-io#944
Update Kubernetes Orchestrator Example by @fa9r in zenml-io#942
Add code and instructions to run quickstart on Colab. by @fa9r in zenml-io#939
Fixing the interaction in getting stacks/components by @bcdurak in zenml-io#945
Fix Kubeflow run name by @safoinme in zenml-io#946
VertexOrchestrator apply node selector constraint if gpu_limit > 0 by @gabrielmbmb in zenml-io#935

Full Changelog: https://github.com/zenml-io/zenml/compare/0.20.1...0.20.2

0.20.0 / 0.20.1

The ZenML 0.20.0 release brings a number of big changes to its architecture and a lot of cool new features, some of which are not backwards compatible with previous versions.

These changes are only covered briefly in the release notes. For a detailed view on what happened and how you can get the most out of the 0.20.0 release, please head over to our "ZenML 0.20.0: Our Biggest Release Yet" blog post.

Warning: Breaking Changes

Updating to ZenML 0.20.0 needs to be followed by a migration of your existing ZenML Stacks and you may also need to make changes to your current ZenML pipeline code. Please read the migration guide carefully and follow the instructions to ensure a smooth transition. The guide walks you through these changes and offers instructions on how to migrate your existing ZenML stacks and pipelines to the new version with minimal effort and disruption to your existing workloads.

If you have updated to ZenML 0.20.0 by mistake or are experiencing issues with the new version, you can always go back to the previous version by using pip install zenml==0.13.2 instead of pip install zenml when installing ZenML manually or in your scripts.

Overview of Changes

ZenML takes over the Metadata Store role. All information about your ZenML Stacks, pipelines, and artifacts is now tracked by ZenML itself directly. If you are currently using remote Metadata Stores (e.g. deployed in cloud) in your stacks, you will probably need to replace them with ZenML cloud deployments.
the new ZenML Dashboard is now available with all ZenML deployments.
ZenML Profiles have been removed in favor of ZenML Projects. You need to manually migrate your existing ZenML Profiles after the update.
the configuration of Stack Components is now decoupled from their implementation. If you extended ZenML with custom stack component implementations, you may need to update the way they are registered in ZenML.
the updated ZenML server provides a new and improved collaborative experience. When connected to a ZenML server, you can now share your ZenML Stacks and Stack Components with other users. If you were previously using the ZenML Profiles or the ZenML server to share your ZenML Stacks, you should switch to the new ZenML server and Dashboard and update your existing workflows to reflect the new features.

What's Changed

Fix error in checking Great Expectations results when exit_on_error=True by @TimovNiedek in zenml-io#889
feat(user-dockerfile): Add user argument to DockerConfiguration by @cjidboon94 in zenml-io#892
Minor doc updates for backporting by @htahir1 in zenml-io#894
Removed feature request and replaced with hellonext board by @htahir1 in zenml-io#897
Unit tests for (some) integrations by @strickvl in zenml-io#880
Fixed integration installation command by @edshee in zenml-io#900
Pipeline configuration and intermediate representation by @schustmi in zenml-io#898
[Bugfix] Fix bug in auto-import of stack after recipe deploy by @wjayesh in zenml-io#901
Update TOC on CONTRIBUTING.md by @strickvl in zenml-io#907
ZenServer by @fa9r in zenml-io#879
Update kserve README by @strickvl in zenml-io#912
Confirmation prompts were not working by @htahir1 in zenml-io#917
Stacks can be registered in Click<8.0.0 now by @AlexejPenner in zenml-io#920
Made Pipeline and Stack optional on the HydratedPipelineRunModel by @AlexejPenner in zenml-io#919
Renamed all references from ZenServer to ZenML Server in logs and comments by @htahir1 in zenml-io#915
Prettify pipeline runs list CLI output. by @fa9r in zenml-io#921
Warn when registering non-local component with local ZenServer by @strickvl in zenml-io#904
Fix duplicate results in pipeline run lists and unlisted flag. by @fa9r in zenml-io#922
Fix error log by @htahir1 in zenml-io#916
Update cli docs by @AlexejPenner in zenml-io#913
Fix Pipeline Run Status by @fa9r in zenml-io#923
Change the CLI emoji for whether a stack is shared or not. by @fa9r in zenml-io#926
Fix running pipelines from different locations. by @fa9r in zenml-io#925
Fix zenml stack-component describe CLI command. by @fa9r in zenml-io#929
Update custom deployment to use ArtifactModel by @safoinme in zenml-io#928
Fix the CI unit test and integration test failures by @stefannica in zenml-io#924
Add gcp zenserver recipe by @wjayesh in zenml-io#930
Extend Post Execution Class Properties by @fa9r in zenml-io#931
Fixes for examples by @strickvl in zenml-io#918
Update cheat sheet by @dnth in zenml-io#932
Fix the docstring attribute of pipeline models. by @fa9r in zenml-io#933
New docs post ZenML Server by @htahir1 in zenml-io#927

New Contributors

@TimovNiedek made their first contribution in zenml-io#889
@cjidboon94 made their first contribution in zenml-io#892
@edshee made their first contribution in zenml-io#900

Full Changelog: https://github.com/zenml-io/zenml/compare/0.13.2...0.20.0

0.13.2

ZenML 0.13.2 comes with a new local Docker orchestrator and many other improvements and fixes:

You can now run your pipelines locally in isolated Docker containers per step
@gabrielmbmb updated our MLFlow experiment tracker to work with Databricks deployments 🎉
Documentation updates for cloud deployments and multi-tenancy Kubeflow support

What's Changed

Update GitHub Actions by @fa9r in zenml-io#864
Raise zenml exception when cyclic graph is detected by @schustmi in zenml-io#866
Add source to segment identify call by @htahir1 in zenml-io#868
Use default local paths/URIs for the local artifact and metadata stores by @stefannica in zenml-io#873
Implement local docker orchestrator by @schustmi in zenml-io#862
Update cheat sheet with latest CLI commands from 0.13.0 by @dnth in zenml-io#867
Add a note about importing proper DockerConfiguration module by @jsuchome in zenml-io#877
Bugfix/misc by @schustmi in zenml-io#878
Fixed bug in tfx by @htahir1 in zenml-io#883
Mlflow Databricks connection by @gabrielmbmb in zenml-io#882
Refactor cloud guide to stack deployment guide by @wjayesh in zenml-io#861
Add cookie consent by @strickvl in zenml-io#871
Stack recipe CLI improvements by @wjayesh in zenml-io#872
Kubeflow workaround added by @htahir1 in zenml-io#886

Full Changelog: https://github.com/zenml-io/zenml/compare/0.13.1...0.13.2

0.13.1

ZenML 0.13.1 is here and it comes with several quality of life improvements:

You can now specify the exact order in which your pipelines steps should be executed, e.g., via step_b.after(step_a)
TensorBoard was moved to a separate integration so you can use it with Pytorch and other modeling frameworks
You can now configure the Evidently integration to ignore specific columns in your datasets.

This release also contains a lot of documentation on how to deploy custom code (like preprocessing and postprocessing code) with our KServe and Seldon integrations.

What's Changed

Fix flag info on recipes in docs by @wjayesh in zenml-io#854
Fix some materializer issues by @schustmi in zenml-io#852
Add ignore columns for evidently drift detection by @SangamSwadiK in zenml-io#851
TensorBoard Integration by @fa9r in zenml-io#850
Add option to specify task dependencies by @schustmi in zenml-io#858
Custom code readme and docs by @safoinme in zenml-io#853

New Contributors

@SangamSwadiK made their first contribution in zenml-io#851

Full Changelog: https://github.com/zenml-io/zenml/compare/0.13.0...0.13.1

0.13.0

ZenML version 0.13.0 is chock-full with exciting features.

Custom Code Deployment is the continuation of the Model Deployment story that we have been working on over the last few releases. Now it is possible to deploy custom code along with your models using Kserve or Seldon.

With Spark this release also brings distributed processing into the ZenML toolkit.

Spinning up and configuring infrastructure is a difficult part of the MLOps journey and can easily become a barrier to entry. Using our mlops-stacks repository, it is now possible to spin up perfectly configured infrastructure with the corresponding ZenML stack using the ZenML CLI.

As always, we've also included various bug fixes and lots of improvements to the documentation and our examples.

Breaking Changes

This release introduces a breaking change to the CLI by adjusting the access to the stack component specific resources for secret-managers and model-deployers to be more explicitly linked to the component. Here is how:

# `zenml secret register ...` becomes 
zenml secrets-manager secret register ...

# `zenml served_models list` becomes 
zenml model-deployer models list

What's Changed

Link checker by @dnth in zenml-io#818
Update Readme with latest info from docs page by @dnth in zenml-io#810
Typo on Readme by @dnth in zenml-io#821
Update kserve installation to 0.9 on kserve deployment example by @safoinme in zenml-io#823
Allow setting caching via the config.yaml by @strickvl in zenml-io#827
Handle file-io with context manager by @aliabbasjaffri in zenml-io#825
Add automated link check github actions by @dnth in zenml-io#828
Fix the SQL zenstore to work with MySQL by @stefannica in zenml-io#829
Improve label studio error messages if secrets are missing or of wrong schema by @schustmi in zenml-io#832
Add secret scoping to the Azure Key Vault by @stefannica in zenml-io#830
Unify CLI concepts (removing secret, feature and served-models) by @strickvl in zenml-io#833
Put link checker as part of CI by @dnth in zenml-io#838
Add missing requirement for step operators by @schustmi in zenml-io#834
Fix broken links from link checker results by @dnth in zenml-io#835
Fix served models logs formatting error by @safoinme in zenml-io#836
New Docker build configuration by @schustmi in zenml-io#811
Secrets references on stack component attributes by @schustmi in zenml-io#817
Misc bugfixes by @schustmi in zenml-io#842
Pillow Image materializer by @strickvl in zenml-io#820
Add Tekton orchestrator by @schustmi in zenml-io#844
Put Slack call to action at the top of README page. by @dnth in zenml-io#846
Change Quickstart to Use Tabular Data by @fa9r in zenml-io#843
Add sleep before docker builds in release GH action by @schustmi in zenml-io#849
Implement Recursive Built-In Container Materializer by @fa9r in zenml-io#812
Custom deployment with KServe and Seldon Core by @safoinme in zenml-io#841
Spark Integration by @bcdurak in zenml-io#837
Add zenml stack recipe CLI commands by @wjayesh in zenml-io#807

New Contributors

@aliabbasjaffri made their first contribution in zenml-io#825

Full Changelog: https://github.com/zenml-io/zenml/compare/0.12.0...0.13.0

0.12.0

The 0.12.0 release comes with the third implementation of the ZenML Model Deployer abstraction: The KServe integration allows you to deploy any PyTorch, TensorFlow or SKLearn from within your ZenML pipelines!

We also added functionality to specify hardware resources on a step level to control the amount of memory, CPUs and GPUs that each ZenML step has access to. This is curretly limited to the Kubeflow and Vertex orchestrator but will be expanded in upcoming releases.

Additionally, we've added support for scoped secrets in our AWS, GCP and Vault Secrets Managers. These updated Secrets Managers allow you to configure a scope which determines if secrets are shared with other ZenML Secrets Managers using the same backend.

As always, we've also included various bug fixes and lots of improvements to the documentation and our examples.

What's Changed

Fix Links on the examples by @safoinme in zenml-io#782
Fix broken links in source code by @schustmi in zenml-io#784
Invalidating artifact/metadata store if there is a change in one of them by @bcdurak in zenml-io#719
Fixed broken link in README by @htahir1 in zenml-io#785
Embed Cheat Sheet in a separate docs page by @fa9r in zenml-io#790
Add data validation documentation by @stefannica in zenml-io#789
Add local path for mlflow experiment tracker by @schustmi in zenml-io#786
Improve Docker build logs. by @fa9r in zenml-io#793
Allow standard library types in steps by @stefannica in zenml-io#799
Added small description by @AlexejPenner in zenml-io#801
Replace the restriction to use Repository inside step with a warning by @stefannica in zenml-io#792
Adjust quickstart to data validators by @fa9r in zenml-io#797
Add utility function to deprecate pydantic attributes by @schustmi in zenml-io#778
Fix the mismatch KFP version between Kubeflow and GCP integration by @safoinme in zenml-io#796
Made mlflow more verbose by @htahir1 in zenml-io#802
Fix links by @dnth in zenml-io#798
KServe model deployer integration by @stefannica in zenml-io#655
retrieve pipeline requirement within running step by @safoinme in zenml-io#805
Fix --decouple_stores error message by @strickvl in zenml-io#814
Support subscripted generic step output types by @fa9r in zenml-io#806
Allow empty kubeconfig when using local kubeflow orchestrator by @schustmi in zenml-io#809
fix the secret register command in kserve docs page by @safoinme in zenml-io#815
Annotation example (+ stack component update) by @strickvl in zenml-io#813
Per-step resource configuration by @schustmi in zenml-io#794
Scoped secrets by @stefannica in zenml-io#803
Adjust examples and docs to new pipeline and step fetching syntax by @fa9r in zenml-io#795

Full Changelog: https://github.com/zenml-io/zenml/compare/0.11.0...0.12.0

0.11.0

Our 0.11.0 release contains our new annotation workflow and stack component. We've been blogging about this for a few weeks, and even started maintaining our own repository of open-source annotation tools. With ZenML 0.11.0 you can bring data labeling into your MLOps pipelines and workflows as a first-class citizen. We've started our first iteration of this functionality by integrating with Label Studio, a leader in the open-source annotation tool space.

This release also includes a ton of updates to our documentation. (Seriously, go check them out! We added tens of thousands of words since the last release.) We continued the work on our data validation story from the previous release: Deepchecks is the newest data validator we support, and we updated our Evidently and Whylogs integrations to include all the latest and greatest from those tools.

Beyond this, as usual we included a number of smaller bugfixes and documentation changes to cumulatively improve experience of using ZenML as a user. For a detailed look at what's changed, give our full release notes a glance.

Breaking Changes

The 0.11.0 release remodels the Evidently and whylogs integrations as Data Validator stack components, in an effort to converge all data profiling and validation libraries around the same abstraction. As a consequence, you now need to configure and add a Data Validator stack component to your stack if you wish to use Evidently or whylogs in your pipelines:

for Evidently:

zenml data-validator register evidently -f evidently
zenml stack update -dv evidently

for whylogs:

zenml data-validator register whylogs -f whylogs
zenml stack update -dv whylogs

In this release, we have also upgraded the Evidently and whylogs libraries to their latest and greatest versions (whylogs 1.0.6 and evidently 0.1.52). These versions introduce non-backwards compatible changes that are also reflected in the ZenML integrations:

Evidently profiles are now materialized using their original evidently.model_profile.Profile data type and the builtin EvidentlyProfileStep step now also returns a Profile instance instead of the previous dictionary representation. This may impact your existing pipelines as you may have to update your steps to take in Profile artifact instances instead of dictionaries.
the whylogs whylogs.DatasetProfile data type was replaced by whylogs.core.DatasetProfileView in the builtin whylogs materializer and steps. This may impact your existing pipelines as you may have to update your steps to return and take in whylogs.core.DatasetProfileView artifact instances instead of whylogs.DatasetProfile objects.
the whylogs library has gone through a major transformation that completely removed the session concept. As a result, the enable_whylogs step decorator was replaced by an enable_whylabs step decorator. You only need to use the step decorator if you wish to log your profiles to the Whylabs platform.

Pleaser refer to the examples provided for Evidently and whylogs to learn more about how to use the new integration versions:

Evidently
whylogs/Whylabs

What's Changed

Changed PR template to reflect integrations flow by @htahir1 in zenml-io#732
Fix broken Feast integration by @strickvl in zenml-io#737
Describe args run.py application actually supports by @jsuchome in zenml-io#740
Update kubernetes_orchestration example by @fa9r in zenml-io#743
Fix some example links by @schustmi in zenml-io#744
Fix broken links for docs and examples by @safoinme in zenml-io#747
Update CONTRIBUTING.md by @strickvl in zenml-io#748
Fix references to types when registering secrets managers by @strickvl in zenml-io#738
Make examples conform to best practices guidance by @AlexejPenner in zenml-io#734
API Docs with Cookies and Milk by @AlexejPenner in zenml-io#758
Use correct region when trying to fetch ECR repositories by @schustmi in zenml-io#761
Encode azure secrets manager secret names by @schustmi in zenml-io#760
Add nested mlflow option to enable_mlflow decorator by @Val3nt-ML in zenml-io#742
Combine all MLMD contexts by @schustmi in zenml-io#759
Prevent extra attributes when initializing StackComponents by @schustmi in zenml-io#763
New Docker images by @schustmi in zenml-io#757
Fix facets magic display in Google Colab by @fa9r in zenml-io#765
Allow fetching secrets from within a step by @schustmi in zenml-io#766
Add notebook to great expectation example by @stefannica in zenml-io#768
Module resolving and path fixes by @schustmi in zenml-io#735
Fix step operator entrypoint by @schustmi in zenml-io#771
Docs Revamp by @fa9r in zenml-io#769
Allow fetching pipeline/step by name, class or instance by @AlexejPenner in zenml-io#733
Data Validator abstraction and Deepchecks integration by @htahir1 in zenml-io#553
rolling back seldon deployment example by @safoinme in zenml-io#774
Added changes from 1062 and 1061 into the updated docs by @AlexejPenner in zenml-io#775
Refresh Examples on zenml examples pull by @fa9r in zenml-io#776
Annotation stack component and Label Studio integration by @strickvl in zenml-io#764
Add optional machine specs to vertex orchestrator by @felixthebeard in zenml-io#762

New Contributors

@jsuchome made their first contribution in zenml-io#740
@Val3nt-ML made their first contribution in zenml-io#742
@felixthebeard made their first contribution in zenml-io#762

Full Changelog: https://github.com/zenml-io/zenml/compare/0.10.0...0.11.0

0.10.0

The 0.10.0 release continues our streak of extending ZenML with support for new orchestrators, this time by adding the Kubernetes Native Orchestrator. This orchestrator is a lightweight alternative to other distributed orchestrators like Airflow or Kubeflow that gives our users the ability to run pipelines in any Kubernetes cluster without having to install and manage additional tools or components.

This release features another integration that we are really excited about: the popular data profiling and validation library Great Expectations is our first Data Validator, a new category of stack components that we are in the process of standardizing, that will make data quality a central feature of ZenML. The ZenML Great Expectations integration eliminates the complexity associated with configuring the store backends for Great Expectations by reusing our Artifact Store concept for that purpose and gives ZenML users immediate access to Great Expectations in both local and cloud settings.

Last but not least, the release also includes a new secrets manager implementation, courtesy of our contributor @karimhabush, that integrates ZenML with the Hashicorp Vault Server as well as a few other bug fixes and improvements.

What's Changed

Fix broken link by @strickvl in zenml-io#707
Add stack component copy command by @schustmi in zenml-io#705
Remove force flag from secrets managers' implementation by @strickvl in zenml-io#708
Fixed wrong example README by @AlexejPenner in zenml-io#712
Fix dead links in integrations docs. by @fa9r in zenml-io#710
Fixing link to guide by @chethanuk-plutoflume in zenml-io#716
Adding azure-keyvault-secrets to azure integration dependencies by @safoinme in zenml-io#717
Fix MLflow repeated deployment error by @fa9r in zenml-io#715
Replace alerter standard steps by Slack-specific steps to fix config issue. by @fa9r in zenml-io#714
Fix broken links on README by @dnth in zenml-io#722
Invalidate cache by @strickvl in zenml-io#724
Skip Cleaning Trace on tests by @safoinme in zenml-io#725
Kubernetes orchestrator by @fa9r in zenml-io#688
Vault Secrets Manager integration - KV Secrets Engine by @karimhabush in zenml-io#689
Add missing help text for CLI commands by @safoinme in zenml-io#723
Misc bugfixes by @schustmi in zenml-io#713
Great Expectations integration for data validation by @strickvl in zenml-io#555
Fix GCP artifact store by @schustmi in zenml-io#730

New Contributors

@chethanuk-plutoflume made their first contribution in zenml-io#716
@dnth made their first contribution in zenml-io#722
@karimhabush made their first contribution in zenml-io#689

Full Changelog: https://github.com/zenml-io/zenml/compare/0.9.0...0.10.0

0.9.0

It's been a couple of weeks, so it's time for a new release! 0.9.0 brings two whole new orchestrators, one of which was contributed by a community member just one day after we unveiled new documentation for orchestrator extensibility! The release also includes a new secrets manager, a Slack integration and a bunch of other smaller changes across the codebase. (Our new orchestrators are exciting enough that they'll get their own blog posts to showcase their strengths in due course.)

Beyond this, as usual we included a number of smaller bugfixes and documentation changes to cumulatively improve experience of using ZenML as a user.

What's Changed

Pass secret to release linting workflow by @schustmi in zenml-io#642
Fix typo in example by @anencore94 in zenml-io#644
Added SecretExistsError in register_secret() method by @hectorLop in zenml-io#648
Fix broken GCP Secrets example CLI command by @strickvl in zenml-io#649
Upgrade to ml-pipelines-sdk v1.8.0 by @strickvl in zenml-io#651
Fix example list CLI command name by @schustmi in zenml-io#647
Fix README by @strickvl in zenml-io#657
Fix broken links in docs by @safoinme in zenml-io#652
Add VertexOrchestrator implementation by @gabrielmbmb in zenml-io#640
Fix index page links and Heading links. by @safoinme in zenml-io#661
Add docstring checks to pre-commit script by @strickvl in zenml-io#481
Pin MLflow to <1.26.0 to prevent issues when matplotlib is not installed by @fa9r in zenml-io#666
Making utils more consistent by @strickvl in zenml-io#658
Fix linting failures on develop by @strickvl in zenml-io#669
Add docstrings for config module by @strickvl in zenml-io#668
Miscellaneous bugfixes by @schustmi in zenml-io#660
Make ZenServer dependencies optional by @schustmi in zenml-io#665
Implement Azure Secrets Manager integration by @strickvl in zenml-io#654
Replace codespell with pyspelling by @strickvl in zenml-io#663
Add Community Event to README by @htahir1 in zenml-io#674
Fix failing integration tests by @strickvl in zenml-io#677
Add io and model_deployers docstring checks by @strickvl in zenml-io#675
Update zenml stack down to use --force flag by @schustmi in zenml-io#673
Fix class resolving on windows by @schustmi in zenml-io#678
Added pipelines docstring checks by @strickvl in zenml-io#676
Docstring checks for cli module by @strickvl in zenml-io#680
Docstring fixes for entrypoints and experiment_trackers modules by @strickvl in zenml-io#672
Clearer Contributing.md by @htahir1 in zenml-io#681
How to access secrets within step added to docs by @AlexejPenner in zenml-io#653
FIX: Log a warning instead of raising an AssertionError by @ketangangal in zenml-io#628
Reviewer Reminder by @htahir1 in zenml-io#683
Fix some docs phrasings and headers by @strickvl in zenml-io#670
Implement SlackAlerter.ask() by @fa9r in zenml-io#662
Extending Alerters Docs by @fa9r in zenml-io#690
Sane defaults for MySQL by @htahir1 in zenml-io#691
pd.Series materializer by @Reed-Schimmel in zenml-io#684
Add docstrings for materializers and metadata_stores by @strickvl in zenml-io#694
Docstrings for the integrations module(s) by @strickvl in zenml-io#692
Add remaining docstrings by @strickvl in zenml-io#696
Allow enabling mlflow/wandb/whylogs with the class-based api by @schustmi in zenml-io#697
GitHub Actions orchestrator by @schustmi in zenml-io#685
Created MySQL docs, Vertex AI docs, and step.entrypoint() by @AlexejPenner in zenml-io#698
Update ignored words by @strickvl in zenml-io#701
Stack Component registering made easier by @AlexejPenner in zenml-io#695
Cleaning up the docs after the revamp by @bcdurak in zenml-io#699
Add model deployer to CLI docs by @safoinme in zenml-io#702
Merge Cloud Integrations and create a Vertex AI Example by @AlexejPenner in zenml-io#693
GitHub actions orchestrator example by @schustmi in zenml-io#703

New Contributors

@anencore94 made their first contribution in zenml-io#644
@hectorLop made their first contribution in zenml-io#648
@gabrielmbmb made their first contribution in zenml-io#640
@ketangangal made their first contribution in zenml-io#628
@Reed-Schimmel made their first contribution in zenml-io#684

Full Changelog: https://github.com/zenml-io/zenml/compare/0.8.1...0.9.0

0.8.1

ZenML 0.8.1 is here and it comes with support for Python 3.9 🎉. It also includes major updates to our documentation, fixes some broken links in our examples and improves the zenml go command which helps you get started with ZenML.

What's Changed

Hotfix/fix failing release by @AlexejPenner in zenml-io#611
Remove autocomplete + alerter from documentation by @strickvl in zenml-io#612
Support Python 3.9 by @htahir1 in zenml-io#605
Revert README by @htahir1 in zenml-io#624
Don't build cuda image on release by @schustmi in zenml-io#623
Update quickstart for zenml go by @fa9r in zenml-io#625
Improve kubeflow manual setup logs by @schustmi in zenml-io#622
Added missing space to error message by @AlexejPenner in zenml-io#614
Added --set flag to register stack command by @AlexejPenner in zenml-io#613
Fixes for multiple examples by @schustmi in zenml-io#626
Bring back the served_model format to the keras materializer by @stefannica in zenml-io#629
Fix broken example links by @schustmi in zenml-io#630
FAQ edits by @strickvl in zenml-io#634
Fix version parsing by @schustmi in zenml-io#633
Completed Best Practices Page by @AlexejPenner in zenml-io#635
Comments on Issues should no longer trigger gh actions by @AlexejPenner in zenml-io#636
Revise CONTRIBUTING.md by @strickvl in zenml-io#615
Alerter Component for Slack Integration by @fa9r in zenml-io#586
Update zenml go to open quickstart/notebooks. by @fa9r in zenml-io#631
Update examples by @schustmi in zenml-io#638
More detailed instructions on creating an integration by @AlexejPenner in zenml-io#639
Added publish api docs to release workflow by @AlexejPenner in zenml-io#641
Added *.md to ignore paths by @AlexejPenner in zenml-io#637
Update README and Docs with new messaging and fix broken links by @htahir1 in zenml-io#632

Full Changelog: https://github.com/zenml-io/zenml/compare/0.8.0...0.8.1

0.8.0

🧘‍♀️ Extensibility is our middle name

The ability to register custom stack component flavors (and renaming types to flavor (Registering custom stack component flavors by @bcdurak in zenml-io#541)
The ability to easily extend orchestrators
Documentation for stacks, stack components and flavors by @bcdurak in zenml-io#607
Allow configuration of s3fs by @schustmi in zenml-io#532
Ability to use SSL to connect to MySQL clients (That allows for connecting to Cloud based MYSQL deployments)
New MySQL metadata stores by @bcdurak in zenml-io#580!
Docs and messaging change
Make Orchestrators more extensible and simplify the interface by @AlexejPenner in zenml-io#581
S3 Compatible Artifact Store and materializers file handling by @safoinme in zenml-io#598

Manage your stacks

Update stack and stack components via the CLI by @strickvl in zenml-io#497
Add stack delete confirmation prompt by @strickvl in zenml-io#548
Add zenml stack export and zenml stack import commands by @fa9r in zenml-io#560

Collaboration

User management by @schustmi in zenml-io#500

CLI improvements

CLI speed improvement by @bcdurak in zenml-io#567
Ensure rich CLI displays full text and wraps table text by @strickvl in zenml-io#577
Add CLI command to remove stack component attribute by @strickvl in zenml-io#590
Beautify CLI by grouping commands list into tags by @safoinme in zenml-io#546

New integrations:

Add PyTorch example by @htahir1 in zenml-io#559
Added GCP as secret manager by @AlexejPenner in zenml-io#556

Documentation / ZenBytes etc

ZenBytes update (and ZenFiles)
Beautification of Examples by @AlexejPenner in zenml-io#491
Document global configuration and repository by @stefannica in zenml-io#579
ZenML Collaboration docs by @stefannica in zenml-io#597

➕ Other Updates, Additions and Fixes

Experiment tracker stack components by @htahir1 in zenml-io#530
Secret Manager improvements and Seldon Core secret passing by @stefannica in zenml-io#529
Pipeline run tracking by @schustmi in zenml-io#601
Stream model deployer logs through CLI by @stefannica in zenml-io#557
Fix various usability bugs by @stefannica in zenml-io#561
Replace -f and --force with -y and --yes by @strickvl in zenml-io#566
Make it easier to submit issues by @htahir1 in zenml-io#571
Sync the repository and local store with the disk configuration files and other fixes by @stefannica in zenml-io#588
Add ability to give in-line pip requirements for pipeline by @strickvl in zenml-io#583
Fix evidently visualizer on Colab by @fa9r in zenml-io#592

🙌 Community Contributions

@Ankur3107 made their first contribution in zenml-io#467
@MateusGheorghe made their first contribution in zenml-io#523
Added support for scipy sparse matrices by @avramdj in zenml-io#534

0.7.3

📊 Experiment Tracking Components

PR #530 adds a new stack component to ZenMLs ever-growing list: experiment_trackers allows users to configure your experiment tracking tools with ZenML. Examples of experiment tracking tools are Weights&Biases, mlflow, Neptune, amongst others.

Existing users might be confused, as ZenML has had MLflow and wandb support for a while now without such a component. However, this component allows uses more control over the configuration of MLflow and wandb with the new MLFlowExperimentTracker and WandbExperimentTracker components. This allows these tools to work in more scenarios than the currently limiting local use-cases.

🔎 XGBoost and LightGBM support

XGBoost and LightGBM are one of the most widely used boosting algorithm libraries out there. This release adds materializers for native objects for each library.

Check out both examples here and PR's #544 and #538 for more details.

📂 Parameterized S3FS support to enable non-AWS S3 storage (minio, ceph)

A big complaint of the S3 Artifact Store integration was that it was hard to parameterize it in a way that it supports non-AWS S3 storage like minio and ceph. The latest release made this super simple! When you want to register an S3ArtifactStore from the CLI, you can now pass in client_kwargs, config_kwargs or s3_additional_kwargs as a JSON string. For example:

zenml artifact-store register my_s3_store --type=s3 --path=s3://my_bucket \
    --client_kwargs='{"endpoint_url": "http://my-s3-endpoint"}'

See PR #532 for more details.

🧱 New CLI commands to update stack components

We added functionality to allow users to update stacks that already exist. This shows the basic workflow:

zenml orchestrator register local_orchestrator2 -t local
zenml stack update default -o local_orchestrator2
zenml stack describe default
zenml container-registry register local_registry --type=default --uri=localhost:5000
zenml container-registry update local --uri='somethingelse.com'
zenml container-registry rename local local2
zenml container-registry describe local2
zenml stack rename default new_default
zenml stack update new_default -c local2
zenml stack describe new_default
zenml stack remove-component -c

More details are in the CLI docs. Users can add new stack components to a pre-existing stack, or they can modify already-present stack components. They can also rename their stack and individual stack components.

🐛 Seldon Core authentication through ZenML secrets

The Seldon Core Model Deployer stack component was updated in this release to allow the configuration of ZenML secrets with credentials that authenticate Seldon to access the Artifact Store. The Seldon Core integration provides 3 different secret schemas for the 3 flavors of Artifact Store: AWS, GCP, and Azure, but custom secrets can be used as well. For more information on how to use this feature please refer to our Seldon Core deployment example.

Lastly, we had numerous other changes such as ensuring the PyTorch materializer works across all artifact stores and the Kubeflow Metadata Store can be easily queried locally.

Detailed Changelog

Fix caching & mypy errors by @strickvl in zenml-io#524
Switch unit test from local_daemon to multiprocessing by @jwwwb in zenml-io#508
Change Pytorch materializer to support remote storage by @safoinme in zenml-io#525
Remove TODO from Feature Store init docstring by @strickvl in zenml-io#527
Fixed typo predicter -> predictor by @MateusGheorghe in zenml-io#523
Fix mypy errors by @strickvl in zenml-io#528
Replaced old local_* logic by @htahir1 in zenml-io#531
capitalize aws username in ECR docs by @wjayesh in zenml-io#533
Build docker base images quicker after release by @schustmi in zenml-io#537
Allow configuration of s3fs by @schustmi in zenml-io#532
Update contributing and fix ci badge to main by @htahir1 in zenml-io#536
Added XGboost integration by @htahir1 in zenml-io#538
Added fa9r to .github/teams.yml. by @fa9r in zenml-io#539
Secret Manager improvements and Seldon Core secret passing by @stefannica in zenml-io#529
User management by @schustmi in zenml-io#500
Update stack and stack components via the CLI by @strickvl in zenml-io#497
Added lightgbm integration by @htahir1 in zenml-io#544
Fix the Kubeflow metadata store and other stack management improvements by @stefannica in zenml-io#542
Experiment tracker stack components by @htahir1 in zenml-io#530

New Contributors

@MateusGheorghe made their first contribution in zenml-io#523
@fa9r made their first contribution in zenml-io#539

Full Changelog: https://github.com/zenml-io/zenml/compare/0.7.2...0.7.3 Blog Post: https://blog.zenml.io/zero-seven-two-three-release/

0.7.2

0.7.2 is a minor release which quickly patches some bugs found in the last release to do with Seldon and Mlflow deployment.

This release also features initial versions of two amazing new integrations: HuggingFace and Weights&Biases!

HuggingFace models are now supported to be passed through ZenML pipelines!
You can now track your pipeline runs with Weights&Biases with the new enable_wandb decorator!

Continuous model deployment with MLflow has been improved with ZenML 0.7.2. A new MLflow Model Deployer Stack component is now available and needs to be part of your stack to be able to deploy models:

zenml integration install mlflow
zenml model-deployer register mlflow --type=mlflow
zenml stack register local_with_mlflow -m default -a default -o default -d mlflow
zenml stack set local_with_mlflow

The MLflow Model Deployer is yet another addition to the list of Model Deployers available in ZenML. You can read more on deploying models to production with MLflow in our Continuous Training and Deployment documentation section and our MLflow deployment example.

What's Changed

Fix the seldon deployment example by @htahir1 in zenml-io#511
Create base deployer and refactor MLflow deployer implementation by @wjayesh in zenml-io#489
Add nlp example by @Ankur3107 in zenml-io#467
Fix typos by @strickvl in zenml-io#515
Bugfix/hypothesis given does not work with fixture by @jwwwb in zenml-io#513
Bug: fix long Kubernetes labels in Seldon deployments by @stefannica in zenml-io#514
Change prediction_uri to prediction_url in MLflow deployer by @stefannica in zenml-io#516
Simplify HuggingFace Integration by @AlexejPenner in zenml-io#517
Weights & Biases Basic Integration by @htahir1 in zenml-io#518

New Contributors

@Ankur3107 made their first contribution in zenml-io#467

Full Changelog: https://github.com/zenml-io/zenml/compare/0.7.1...0.7.2

0.7.1

The release introduces the Seldon Core ZenML integration, featuring the Seldon Core Model Deployer and a Seldon Core standard model deployer step. The Model Deployer is a new type of stack component that enables you to develop continuous model deployment pipelines that train models and continuously deploy them to an external model serving tool, service or platform. You can read more on deploying models to production with Seldon Core in our Continuous Training and Deployment documentation section and our Seldon Core deployment example.

We also see two new integrations with Feast as ZenML's first feature store integration. Feature stores allow data teams to serve data via an offline store and an online low-latency store where data is kept in sync between the two. It also offers a centralized registry where features (and feature schemas) are stored for use within a team or wider organization. ZenML now supports connecting to a Redis-backed Feast feature store as a stack component integration. Check out the full example to see it in action!

0.7.1 also brings an addition to ZenML training library integrations with NeuralProphet. Check out the new example for more details, and the docs for more further detail on all new features!

What's Changed

Add linting of examples to pre-commit by @strickvl in zenml-io#490
Remove dev-specific entries in .gitignore by @strickvl in zenml-io#488
Produce periodic mocked data for Segment/Mixpanel by @AlexejPenner in zenml-io#487
Abstractions for artifact stores by @bcdurak in zenml-io#474
enable and disable cache from runtime config by @AlexejPenner in zenml-io#492
Basic Seldon Core Deployment Service by @stefannica in zenml-io#495
Parallelise our test suite and make errors more readable by @alex-zenml in zenml-io#378
Provision local zenml service by @jwwwb in zenml-io#496
bugfix/optional-secrets-manager by @safoinme in zenml-io#493
Quick fix for copying folders by @bcdurak in zenml-io#501
Pin exact ml-pipelines-sdk version by @schustmi in zenml-io#506
Seldon Core model deployer stack component and standard step by @stefannica in zenml-io#499
Fix datetime test / bug by @strickvl in zenml-io#507
Added NeuralProphet integration by @htahir1 in zenml-io#504
Feature Store (Feast with Redis) by @strickvl in zenml-io#498

0.7.0

With ZenML 0.7.0, a lot has been revamped under the hood about how things are stored. Importantly what this means is that ZenML now has system-wide profiles that let you register stacks to share across several of your projects! If you still want to manage your stacks for each project folder individually, profiles still let you do that as well.

Most projects of any complexity will require passwords or tokens to access data and infrastructure, and for this purpose ZenML 0.7.0 introduces the Secrets Manager stack component to seamlessly pass around these values to your steps. Our AWS integration also allows you to use AWS Secrets Manager as a backend to handle all your secret persistence needs.

Finally, in addition to the new AzureML and Sagemaker Step Operators that version 0.6.3 brought, this release also adds the ability to run individual steps on GCP's Vertex AI.

Beyond this, some smaller bugfixes and documentation changes combine to make ZenML 0.7.0 a more pleasant user experience.

What's Changed

Added quick mention of how to use dockerignore by @AlexejPenner in zenml-io#468
Made rich traceback optional with ENV variable by @htahir1 in zenml-io#472
Separate stack persistence from repo implementation by @jwwwb in zenml-io#462
Adding safoine username to github team by @safoinme in zenml-io#475
Fix zenml stack describe bug by @strickvl in zenml-io#476
ZenProfiles and centralized ZenML repositories by @stefannica in zenml-io#471
Add examples folder to linting script by @strickvl in zenml-io#482
Vertex AI integration and numerous other changes by @htahir1 in zenml-io#477
Fix profile handing in the Azure ML step operator by @stefannica in zenml-io#483
Copy the entire stack configuration into containers by @stefannica in zenml-io#480
Improve some things with the Profiles CLI output by @stefannica in zenml-io#484
Secrets manager stack component and interface by @AlexejPenner in zenml-io#470
Update schedule.py (#485) by @avramdj in zenml-io#485

New Contributors

@avramdj in zenml-io#485

Full Changelog: https://github.com/zenml-io/zenml/compare/0.6.3...0.7.0rc

0.6.3

With ZenML 0.6.3, you can now run your ZenML steps on Sagemaker and AzureML! It's normal to have certain steps that require specific hardware on which to run model training, for example, and this latest release gives you the power to switch out hardware for individual steps to support this.

We added a new Tensorboard visualization that you can make use of when using our Kubeflow Pipelines integration. We handle the background processes needed to spin up this interactive web interface that you can use to visualize your model's performance over time.

Behind the scenes we gave our integration testing suite a massive upgrade, fixed a number of smaller bugs and made documentation updates. For a detailed look at what's changed, give our full release notes a glance.

What's Changed

Fix typo by @wjayesh in zenml-io#432
Remove tabulate dependency (replaced by rich) by @jwwwb in zenml-io#436
Fix potential issue with local integration tests by @schustmi in zenml-io#428
Remove support for python 3.6 by @schustmi in zenml-io#437
Create clean test repos in separate folders by @michael-zenml in zenml-io#430
Copy explicit materializers before modifying, log correct class by @schustmi in zenml-io#434
Fix typo in mysql password parameter by @pafpixel in zenml-io#438
Pytest-fixture for separate virtual environments for each integration test by @AlexejPenner in zenml-io#405
Bugfix/fix failing tests due to comments step by @AlexejPenner in zenml-io#444
Added --use-virtualenvs option to allow choosing envs to run by @AlexejPenner in zenml-io#445
Log whether a step was cached by @strickvl in zenml-io#435
Added basic integration tests for remaining examples by @strickvl in zenml-io#439
Improve error message when provisioning local kubeflow resources with a non-local container registry. by @schustmi in zenml-io#442
Enable generic step inputs and outputs by @schustmi in zenml-io#440
Removed old reference to a step that no longer exists by @AlexejPenner in zenml-io#452
Correctly use custom kubernetes context if specified by @schustmi in zenml-io#451
Fix CLI stack component describe/list commands by @schustmi in zenml-io#450
Ignore type of any tfx proto file by @schustmi in zenml-io#453
Another boyscout pr on the gh actions by @AlexejPenner in zenml-io#455
Upgrade TFX to 1.6.1 by @jwwwb in zenml-io#441
Added ZenFiles to README by @htahir1 in zenml-io#457
Upgrade rich from 11.0 to 12.0 by @strickvl in zenml-io#458
Add Kubeflow tensorboard viz and fix tensorflow file IO for cloud back-ends by @stefannica in zenml-io#447
Implementing the explain subcommand by @bcdurak in zenml-io#460
Implement AzureML and Sagemaker step operators by @schustmi in zenml-io#456

New Contributors

@pafpixel made their first contribution in zenml-io#438

0.6.2

ZenML 0.6.2 brings you the ability to serve models using MLflow deployments as well as an updated CLI interface! For a real continuous deployment cycle, we know that ZenML pipelines should be able to handle everything — from pre-processing to training to serving to monitoring and then potentially re-training and re-serving. The interfaces we created in this release are the foundation on which all of this will build.

We also improved how you interact with ZenML through the CLI. Everything looks so much smarter and readable now with the popular rich library integrated into our dependencies.

Smaller changes that you'll notice include updates to our cloud integrations and bug fixes for Windows users. For a detailed look at what's changed, see below.

What's Changed

Updated notebook for quickstart by @htahir1 in zenml-io#398
Update tensorflow base image by @schustmi in zenml-io#396
Add cloud specific deployment guide + refactoring by @wjayesh in zenml-io#400
add cloud sub page to toc.md by @wjayesh in zenml-io#401
fix tab indent by @wjayesh in zenml-io#402
Bugfix for workflows failing due to modules not being found by @bcdurak in zenml-io#390
Improve github workflows by @schustmi in zenml-io#406
Add plausible script to docs.zenml.io pages by @alex-zenml in zenml-io#414
Add orchestrator and ECR docs by @wjayesh in zenml-io#413
Richify the CLI by @alex-zenml in zenml-io#392
Allow specification of required integrations for a pipeline by @schustmi in zenml-io#408
Update quickstart in docs to conform to examples by @htahir1 in zenml-io#410
Updated PR template with some more details by @htahir1 in zenml-io#411
Bugfix on the CLI to work without a git installation by @bcdurak in zenml-io#412
Added Ayush's Handle by @ayush714 in zenml-io#417
Adding an info message on Windows if there is no application associated to .sh files by @bcdurak in zenml-io#419
Catch matplotlib crash when running IPython in terminal by @strickvl in zenml-io#416
Automatically activate integrations when unable to find stack component by @schustmi in zenml-io#420
Fix some code inspections by @halvgaard in zenml-io#422
Prepare integration tests on kubeflow by @schustmi in zenml-io#423
Add concepts back into glossary by @strickvl in zenml-io#425
Make guide easier to follow by @wjayesh in zenml-io#427
Fix httplib to 0.19 and pyparsing to 2.4 by @jwwwb in zenml-io#426
Wrap context serialization in try blocks by @jwwwb in zenml-io#397
Track stack configuration when registering and running a pipeline by @schustmi in zenml-io#429
MLflow deployment integration by @stefannica in zenml-io#415

0.6.1

ZenML 0.6.1 is out and it's all about the cloud ☁️! We have improved AWS integration and a brand-new Azure integration! Run your pipelines on AWS and Azure now and let us know how it went on our Slack.

Smaller changes that you'll notice include much-awaited updates and fixes, including the first iterations of scheduling pipelines and tracking more reproducibility-relevant data in the metadata store.

For a detailed look at what's changed, see below.

What's changed

Add MVP for scheduling by @htahir1 in zenml-io#354
Add S3 artifact store and filesystem by @schustmi in zenml-io#359
Update 0.6.0 release notes by @alex-zenml in zenml-io#362
Fix cuda-dev base container image by @stefannica in zenml-io#361
Mark ZenML as typed package by @schustmi in zenml-io#360
Improve error message if ZenML repo is missing inside kubeflow container entrypoint by @schustmi in zenml-io#363
Spell whylogs and WhyLabs correctly in our docs by @stefannica in zenml-io#369
Feature/add readme for mkdocs by @AlexejPenner in zenml-io#372
Cleaning up the assets pushed by gitbook automatically by @bcdurak in zenml-io#371
Turn codecov off for patch updates by @htahir1 in zenml-io#376
Minor changes and fixes by @schustmi in zenml-io#365
Only include python files when building local docs by @schustmi in zenml-io#377
Prevent access to repo during step execution by @schustmi in zenml-io#370
Removed duplicated Section within docs by @AlexejPenner in zenml-io#379
Fixing the materializer registry to spot sub-classes of defined types by @bcdurak in zenml-io#368
Computing hash of step and materializer works in notebooks by @htahir1 in zenml-io#375
Sort requirements to improve docker build caching by @schustmi in zenml-io#383
Make sure the s3 artifact store is registered when the integration is activated by @schustmi in zenml-io#382
Make MLflow integration work with kubeflow and scheduled pipelines by @stefannica in zenml-io#374
Reset _has_been_called to False ahead of pipeline.connect by @AlexejPenner in zenml-io#385
Fix local airflow example by @schustmi in zenml-io#366
Improve and extend base materializer error messages by @schustmi in zenml-io#380
Windows CI issue by @schustmi in zenml-io#389
Add the ability to attach custom properties to the Metadata Store by @bcdurak in zenml-io#355
Handle case when return values do not match output by @AlexejPenner in zenml-io#386
Quickstart code in docs fixed by @AlexejPenner in zenml-io#387
Fix mlflow tracking example by @stefannica in zenml-io#393
Implement azure artifact store and fileio plugin by @schustmi in zenml-io#388
Create todo issues with separate issue type by @schustmi in zenml-io#394
Log that steps are cached while running pipeline by @alex-zenml in zenml-io#381
Schedule added to context for all orchestrators by @AlexejPenner in zenml-io#391

0.6.0

ZenML 0.6.0 is out now. We've made some big changes under the hood, but our biggest public-facing addition is our new integration to support all your data logging needs: whylogs. Our core architecture was thoroughly reworked and is now in a much better place to support our ongoing development needs.

Smaller changes that you'll notice include extensive documentation additions, updates and fixes. For a detailed look at what's changed, see below.

📊 Whylogs logging

Whylogs is an open source library that analyzes your data and creates statistical summaries called whylogs profiles. Whylogs profiles can be visualized locally or uploaded to the WhyLabs platform where more comprehensive analysis can be carried out.

ZenML integrates seamlessly with Whylogs and WhyLabs. This example shows how easy it is to enhance steps in an existing ML pipeline with Whylogs profiling features. Changes to the user code are minimal while ZenML takes care of all aspects related to Whylogs session initialization, profile serialization, versioning and persistence and even uploading generated profiles to Whylabs.

With our WhylogsVisualizer, as described in the associated example notes, you can visualize Whylogs profiles generated as part of a pipeline.

⛩ New Core Architecture

We implemented some fundamental changes to the core architecture to solve some of the issues we previously had and provide a more extensible design to support quicker implementations of different stack components and integrations. The main change was to refactor the Repository, Stack and StackComponent architectures. These changes had a pretty wide impact so involved changes in many files throughout the codebase, especially in the CLI which makes calls to all these pieces.

We've already seen how it helps us move faster in building integrations and we hope it helps making contributions as pain-free as possible!

🗒 Documentation and Example Updates

As the codebase and functionality of ZenML grows, we always want to make sure our documentation is clear, up-to-date and easy to use. We made a number of changes in this release that will improve your experience in this regard:

added a number of new explainers on key ZenML concepts and how to use them in your code, notably on how to create a custom materializer and how to fetch historic pipeline runs using the StepContext
fixed a number of typos and broken links
added versioning to our API documentation so you can choose to view the reference appropriate to the version that you're using. We now use mkdocs for this so you'll notice a slight visual refresh as well.
added new examples highlighting specific use cases and integrations:
- how to create a custom materializer (example)
- how to fetch historical pipeline runs (example)
- how to use standard interfaces for common ML patterns (example)
- whylogs logging (example)

➕ Other updates, additions and fixes

As with most releases, we made a number of small but significant fixes and additions. The most import of these were that you can now access the metadata store via the step context. This enables a number of new possible workflows and pipeline patterns and we're really excited to have this in the release.

We added in a markdown parser for the zenml example info … command, so now when you want to use our CLI to learn more about specific examples you will see beautifully parsed text and not markdown markup.

We improved a few of our error messages, too, like for when the return type of a step function doesn’t match the expected type, or if step is called twice. We hope this makes ZenML just that little bit easier to use.

0.5.7

ZenML 0.5.7 is here 💯 and it brings not one, but 🔥TWO🔥 brand new integrations 🚀! ZenML now support MLFlow for tracking pipelines as experiments and Evidently for detecting drift in your ML pipelines in production!

New Features

Introducing the MLFLow Tracking Integration, a first step towards our complete MLFlow Integration as described in the #115 poll. Full example found here.
Introducing the Evidently integration. Use the standard Evidently drift detection step to calculate drift automatically in your pipeline. Full example found here.

Bugfixes

Prevent KFP install timeouts during stack up by @stefannica in zenml-io#299
Prevent naming parameters same name as inputs/outputs to prevent kwargs-errors by @bcdurak in zenml-io#300

What's Changed

Force pull overwrites local examples without user confirmation by @AlexejPenner in zenml-io#278
Updated README with latest features by @htahir1 in zenml-io#280
Integration test the examples within ci pipeline by @AlexejPenner in zenml-io#282
Add exception for missing system requirements by @kamalesh0406 in zenml-io#281
Examples are automatically pulled if not present before any example command is run by @AlexejPenner in zenml-io#279
Add pipeline error for passing the same step object twice by @kamalesh0406 in zenml-io#283
Create pytest fixture to use a temporary zenml repo in tests by @htahir1 in zenml-io#287
Additional example run implementations for standard interfaces, functional and class based api by @AlexejPenner in zenml-io#286
Make pull_request.yaml actually use os.runner instead of ubuntu by @htahir1 in zenml-io#288
In pytest return to previous workdir before tearing down tmp_dir fixture by @AlexejPenner in zenml-io#289
Don't raise an exception during integration installation if system requirement is not installed by @schustmi in zenml-io#291
Update starting page for the API docs by @alex-zenml in zenml-io#294
Add stack up failure prompts by @alex-zenml in zenml-io#290
Spelling fixes by @alex-zenml in zenml-io#295
Remove instructions to git init from docs by @bcdurak in zenml-io#293
Fix the stack up and orchestrator up failure prompts by @stefannica in zenml-io#297
Prevent KFP install timeouts during stack up by @stefannica in zenml-io#299
Add stefannica to list of internal github users by @stefannica in zenml-io#303
Improve KFP UI daemon error messages by @schustmi in zenml-io#292
Replaced old diagrams with new ones in the docs by @AlexejPenner in zenml-io#306
Fix broken links & text formatting in docs by @alex-zenml in zenml-io#302
Run KFP container as local user/group if local by @stefannica in zenml-io#304
Add james to github team by @jwwwb in zenml-io#308
Implement integration of mlflow tracking by @AlexejPenner in zenml-io#301
Bugfix integration tests on windows by @jwwwb in zenml-io#296
Prevent naming parameters same name as inputs/outputs to prevent kwargs-errors by @bcdurak in zenml-io#300
Add tests for fileio by @alex-zenml in zenml-io#298
Evidently integration (standard steps and example) by @alex-zenml in zenml-io#307
Implemented evidently integration by @stefannica in zenml-io#310
Make mlflow example faster by @AlexejPenner in zenml-io#312

New Contributors

@kamalesh0406 made their first contribution in zenml-io#281
@stefannica made their first contribution in zenml-io#297
@jwwwb made their first contribution in zenml-io#308

Full Changelog: https://github.com/zenml-io/zenml/compare/0.5.6...0.5.7

0.5.6

    )                    *      (     
 ( /(                  (  `     )\ )  
 )\())    (            )\))(   (()/(  
((_)\    ))\    (     ((_)()\   /(_)) 
 _((_)  /((_)   )\ )  (_()((_) (_))   
|_  /  (_))    _(_/(  |  \/  | | |    
 / /   / -_)  | ' \)) | |\/| | | |__  
/___|  \___|  |_||_|  |_|  |_| |____|

This release fixes some known bugs from previous releases and especially 0.5.5. Therefore, upgrading to 0.5.6 is a breaking change. You must do the following in order to proceed with this version:

cd zenml_enabled_repo
rm -rf .zen/

And then start again with ZenML init:

pip install --upgrade zenml
zenml init

New Features

Added zenml example run [EXAMPLE_RUN_NAME] feature: The ability to run an example with one command. In order to run this, do zenml example pull first and see all examples available by running zenml example list.
Added ability to specify a .dockerignore file before running pipelines on Kubeflow.
Kubeflow Orchestrator is now leaner and faster.
Added the describe command group to the CLI for groups stack, orchestrator, artifact-store, and metadata-store. E.g. zenml stack describe

Bug fixes and minor improvements

Adding StepContext to a branch now invalidates caching by default. Disable explicitly with enable_cache=True.
Docs updated to reflect minor changes in CLI commands.
CLI list commands now mentions active component. Try zenml stack list to check this out.
zenml version now has cooler art.

What's Changed

Delete blog reference from release notes by @alex-zenml in zenml-io#228
Docs updates by @alex-zenml in zenml-io#229
Update kubeflow guide by @schustmi in zenml-io#230
Updated quickstart to reflect newest zenml version by @alexej-zenml in zenml-io#231
Add KFP GCP example readme by @schustmi in zenml-io#233
Baris/update docs with class api by @bcdurak in zenml-io#232
fixing a small typo [ci skip] by @bcdurak in zenml-io#236
Hamza/docs last min updates by @htahir1 in zenml-io#234
fix broken links by @alex-zenml in zenml-io#237
added one more page for standardized artifacts [ci skip] by @bcdurak in zenml-io#238
Unified use of cli_utils.print_table for all table format cli printouts by @AlexejPenner in zenml-io#240
Remove unused tfx kubeflow code by @schustmi in zenml-io#239
Relaxed typing requirements for cli_utils.print_table by @AlexejPenner in zenml-io#241
Pass input artifact types to kubeflow container entrypoint by @schustmi in zenml-io#242
Catch duplicate run name error and throw custom exception by @schustmi in zenml-io#243
Improved logs by @htahir1 in zenml-io#244
CLI active component highlighting by @alex-zenml in zenml-io#245
Baris/eng 244 clean up by @bcdurak in zenml-io#246
CLI describe command by @alex-zenml in zenml-io#248
Alexej/eng 35 run examples from cli by @AlexejPenner in zenml-io#253
CLI argument and option flag consistency improvements by @alex-zenml in zenml-io#250
Invalidate caching when a step requires a step context by @schustmi in zenml-io#252
Implement better error messages for custom step output artifact types by @schustmi in zenml-io#254
Small improvements by @schustmi in zenml-io#251
Kubeflow dockerignore by @schustmi in zenml-io#249
Rename container registry folder to be consistent with the other stack components by @schustmi in zenml-io#257
Update todo script by @schustmi in zenml-io#256
Update docs following CLI change by @alex-zenml in zenml-io#255
Bump mypy version by @schustmi in zenml-io#258
Kubeflow Windows daemon alternative by @schustmi in zenml-io#259
Run pre commit in local environment by @schustmi in zenml-io#260
Hamza/eng 269 move beam out by @htahir1 in zenml-io#262
Update docs by @alex-zenml in zenml-io#261
Hamza/update readme with contribitions by @htahir1 in zenml-io#271
Hamza/eng 256 backoff analytics by @htahir1 in zenml-io#270
Add spellcheck by @alex-zenml in zenml-io#264
Using the pipeline run name to explicitly access when explaining the … by @AlexejPenner in zenml-io#263
Import user main module in kubeflow entrypoint to make sure all components are registered by @schustmi in zenml-io#273
Fix cli version command by @schustmi in zenml-io#272
User is informed of version mismatch and example pull defaults to cod… by @AlexejPenner in zenml-io#274
Hamza/eng 274 telemetry by @htahir1 in zenml-io#275
Update docs with right commands and events by @htahir1 in zenml-io#276
Fixed type annotation for some python versions by @AlexejPenner in zenml-io#277

Full Changelog: https://github.com/zenml-io/zenml/compare/0.5.5...0.5.6

0.5.5

ZenML 0.5.5 is jam-packed with new features to take your ML pipelines to the next level. Our three biggest new features: Kubeflow Pipelines, CLI support for our integrations and Standard Interfaces. That’s right, Standard Interfaces are back!

What's Changed

Implement base component tests by @schustmi in zenml-io#211
Add chapter names by @alex-zenml in zenml-io#212
Fix docstring error by @alex-zenml in zenml-io#213
Hamza/add caching example by @htahir1 in zenml-io#214
Update readme by @alex-zenml in zenml-io#216
Hamza/add small utils by @htahir1 in zenml-io#219
Update docs by @alex-zenml in zenml-io#220
Docs fixes by @alex-zenml in zenml-io#222
Baris/eng 182 standard interfaces by @bcdurak in zenml-io#209
Fix naming error by @alex-zenml in zenml-io#221
Remove framework design by @alex-zenml in zenml-io#224
Alexej/eng 234 zenml integration install by @alexej-zenml in zenml-io#223
Fix deployment section order by @alex-zenml in zenml-io#225
the readme of the example by @bcdurak in zenml-io#227
Kubeflow integration by @schustmi in zenml-io#226

New Contributors

@alexej-zenml made their first contribution in zenml-io#223

Full Changelog: https://github.com/zenml-io/zenml/compare/0.5.4...0.5.5

0.5.4

0.5.4 adds a lineage tracking integration to visualize lineage of pipeline runs! It also includes numerous bug fixes and optimizations.

What's Changed

Fix typos by @alex-zenml in zenml-io#192
Fix Apache Beam bug by @alex-zenml in zenml-io#194
Fix apache beam logging bug by @alex-zenml in zenml-io#195
Add step context by @schustmi in zenml-io#196
Init docstrings by @alex-zenml in zenml-io#197
Hamza/small fixes by @htahir1 in zenml-io#199
Fix writing to metadata store with airflow orchestrator by @schustmi in zenml-io#198
Use pipeline parameter name as step name in post execution by @schustmi in zenml-io#200
Add error message when step name is not in metadata store by @schustmi in zenml-io#201
Add option to set repo location using an environment variable by @schustmi in zenml-io#202
Run cloudbuild after pypi publish by @schustmi in zenml-io#203
Refactor component generation by @schustmi in zenml-io#204
Removed unnecessary panel dependency by @htahir1 in zenml-io#206
Updated README to successively install requirements by @AlexejPenner in zenml-io#205
Store active stack in local config by @schustmi in zenml-io#208
Hamza/eng 125 lineage tracking vis by @htahir1 in zenml-io#207

New Contributors

@AlexejPenner made their first contribution in zenml-io#205

Full Changelog: https://github.com/zenml-io/zenml/compare/0.5.3...0.5.4

0.5.3

Version 0.5.3 adds statistics visualizations, greatly improved speed for CLI commands as well as lots of small improvements to the pipeline and step interface.

What's Changed

Make tests run in a random order by @alex-zenml in zenml-io#160
Connect steps using *args by @schustmi in zenml-io#162
Move location of repobeats image by @alex-zenml in zenml-io#163
Hamza/add sam by @htahir1 in zenml-io#165
Pipeline initialization with *args by @schustmi in zenml-io#164
Improve detection of third party modules during class resolving by @schustmi in zenml-io#167
Merge path_utils into fileio & refactor what was left by @alex-zenml in zenml-io#168
Update docker files by @schustmi in zenml-io#169
Hamza/deploy api reference by @htahir1 in zenml-io#171
API Reference by @schustmi in zenml-io#172
Add color back into our github actions by @alex-zenml in zenml-io#176
Refactor tests not raising by @alex-zenml in zenml-io#177
Improve step and pipeline interface by @schustmi in zenml-io#175
Alex/eng 27 windows bug again by @htahir1 in zenml-io#178
Automated todo tracking by @schustmi in zenml-io#173
Fix mypy issues related to windows by @schustmi in zenml-io#179
Include Github URL to TODO comment in issue by @schustmi in zenml-io#181
Create Visualizers logic by @htahir1 in zenml-io#182
Add README for visualizers examples by @alex-zenml in zenml-io#184
Allow None as default value for BaseStep configs by @schustmi in zenml-io#185
Baris/eng 37 standard import check by @bcdurak in zenml-io#183
Replace duplicated code by call to source_utils.resolve_class by @schustmi in zenml-io#186
Remove unused base enum cases by @schustmi in zenml-io#187
Testing mocks for CLI examples command by @alex-zenml in zenml-io#180
Set the correct module for steps created using our decorator by @schustmi in zenml-io#188
Fix some cli commands by @schustmi in zenml-io#189
Tag jira issues for which the todo was deleted by @schustmi in zenml-io#190
Remove deadlinks by @alex-zenml in zenml-io#191

Full Changelog: https://github.com/zenml-io/zenml/compare/0.5.2...0.5.3

0.5.2

0.5.2 brings an improved post-execution workflow and lots of minor changes and upgrades for the developer experience when creating pipelines. It also improves the Airflow orchestrator logic to accommodate for more real world scenarios.

What's Changed

Fix autocomplete for step and pipeline decorated functions by @schustmi in zenml-io#144
Add reference docs for CLI example functionality by @alex-zenml in zenml-io#145
Fix mypy integration by @schustmi in zenml-io#147
Improve Post-Execution Workflow by @schustmi in zenml-io#146
Fix CLI examples bug by @alex-zenml in zenml-io#148
Update quickstart example notebook by @alex-zenml in zenml-io#150
Add documentation images by @alex-zenml in zenml-io#151
Add prettierignore to gitignore by @alex-zenml in zenml-io#154
Airflow orchestrator improvements by @schustmi in zenml-io#153
Google colab added by @htahir1 in zenml-io#155
Tests for core and cli modules by @alex-zenml in zenml-io#149
Add Paperspace environment check by @alex-zenml in zenml-io#156
Step caching by @schustmi in zenml-io#157
Add documentation for pipeline step parameter and run name configuration by @schustmi in zenml-io#158
Automatically disable caching if the step function code has changed by @schustmi in zenml-io#159

Full Changelog: https://github.com/zenml-io/zenml/compare/0.5.1...0.5.2

0.5.1

0.5.1 builds on top of Slack of the 0.5.0 release with quick bug updates.

Overview

Pipeline can now be run via a YAML file. #132
CLI now let's you pull directly from GitHub examples folder. 🔥 Amazing @alex-zenml with #141!
ZenML now has full mypy compliance. 🎉 Thanks @schustmi for #140!
Numerous bugs and performance improvements. #136, @bcdurak great job with #142
Added new docs with a low level API guide. #143

Our roadmap goes into further detail on the timeline. Vote on the next features now.

We encourage every user (old or new) to start afresh with this release. Please go over our latest docs and examples to get a hang of the new system.

0.5.0

This long-awaited ZenML release marks a seminal moment in the project's history. We present to you a complete revamp of the internals of ZenML, with a fresh new design and API. While these changes are significant, and have been months in the making, the original vision of ZenML has not wavered. We hope that the ZenML community finds the new design choices easier to grasp and use, and we welcome feedback on the issues board.

Warning

0.5.0 is a complete API change from the previous versions of ZenML, and is a breaking upgrade. Fundamental concepts have been changed, and therefore backwards compatibility is not maintained. Please use only this version with fresh projects.

With such significant changes, we expect this release to also be breaking. Please report any bugs in the issue board, and they should be addressed in upcoming releases.

Overview

Introducing a new functional API for creating pipelines and steps. This is now the default mechanism for building ZenML pipelines. read more
Steps now use Materializers to handle artifact serialization/deserialization between steps. This is a powerful change, and will be expanded upon in the future. read more
Introducing the new Stack paradigm: Easily transition from one MLOps stack to the next with a few CLI commands read more
Introducing a new Artifact, Typing, and Annotation system, with pydantic (and dataclasses) support read more
Deprecating the pipelines_dir: Now individual pipelines will be stored in their metadata stores, making the metadata store a single source of truth. read more
Deprecating the YAML config file: ZenML no longer natively compiles to an intermediate YAML-based representation. Instead, it compiles and deploys directly into the selected orchestrator's representation. While we do plan to support running pipelines directly through YAML in the future, it will no longer be the default route through which pipelines are run. read more about orchestrators here

Technical Improvements

A completely new system design, please refer to the docs.
Better type hints and docstrings.
Auto-completion support.
Numerous performance improvements and bug fixes, including a smaller dependency footprint.

What to expect in the next weeks and the new ZenML

Currently, this release is bare bones. We are missing some basic features which used to be part of ZenML 0.3.8 (the previous release):

Standard interfaces for TrainingPipeline.
Individual step interfaces like PreprocessorStep, TrainerStep, DeployerStep etc. need to be rewritten from within the new paradigm. They should be included in the non-RC version of this release.
A proper production setup with an orchestrator like Airflow.
A post-execution workflow to analyze and inspect pipeline runs.
The concept of Backends will evolve into a simple mechanism of transitioning individual steps into different runners.
Support for KubernetesOrchestrator, KubeflowOrchestrator, GCPOrchestrator and AWSOrchestrator are also planned.
Dependency management including Docker support is planned.

Our roadmap goes into further detail on the timeline.

We encourage every user (old or new) to start afresh with this release. Please go over our latest docs and examples to get a hang of the new system.

Onwards and upwards to 1.0.0!

0.5.0rc2

This long-awaited ZenML release marks a seminal moment in the project's history. We present to you a complete revamp of the internals of ZenML, with a fresh new design and API. While these changes are significant, and have been months in the making, the original vision of ZenML has not wavered. We hope that the ZenML community finds the new design choices easier to grasp and use, and we welcome feedback on the issues board.

Warning

0.5.0rc0 is a complete API change from the previous versions of ZenML, and is a breaking upgrade. Fundamental concepts have been changed, and therefore backwards compatibility is not maintained. Please use only this version with fresh projects.

With such significant changes, we expect this release to also be breaking. Please report any bugs in the issue board, and they should be addressed in upcoming releases.

Overview

Introducing a new functional API for creating pipelines and steps. This is now the default mechanism for building ZenML pipelines. read more
Introducing the new Stack paradigm: Easily transition from one MLOps stack to the next with a few CLI commands read more
Introducing a new Artifact, Typing, and Annotation system, with pydantic (and dataclasses) support read more
Deprecating the pipelines_dir: Now individual pipelines will be stored in their metadata stores, making the metadata store a single source of truth. read more
Deprecating the YAML config file: ZenML no longer natively compiles to an intermediate YAML-based representation. Instead, it compiles and deploys directly into the selected orchestrator's representation. While we do plan to support running pipelines directly through YAML in the future, it will no longer be the default route through which pipelines are run. read more about orchestrators here

Technical Improvements

A completely new system design, please refer to the docs.
Better type hints and docstrings.
Auto-completion support.
Numerous performance improvements and bug fixes, including a smaller dependency footprint.

What to expect in the next weeks and the new ZenML

Currently, this release is bare bones. We are missing some basic features which used to be part of ZenML 0.3.8 (the previous release):

Standard interfaces for TrainingPipeline.
Individual step interfaces like PreprocessorStep, TrainerStep, DeployerStep etc. need to be rewritten from within the new paradigm. They should be included in the non-RC version of this release.
A proper production setup with an orchestrator like Airflow.
A post-execution workflow to analyze and inspect pipeline runs.
The concept of Backends will evolve into a simple mechanism of transitioning individual steps into different runners.
Support for KubernetesOrchestrator, KubeflowOrchestrator, GCPOrchestrator and AWSOrchestrator are also planned.
Dependency management including Docker support is planned.

Our roadmap goes into further detail on the timeline.

We encourage every user (old or new) to start afresh with this release. Please go over our latest docs and examples to get a hang of the new system.

Onwards and upwards to 1.0.0!

0.3.7.1

This release fixes some known bugs from previous releases and especially 0.3.7. Same procedure as always, please delete existing pipelines, metadata, and artifact stores.

cd zenml_enabled_repo
rm -rf pipelines/
rm -rf .zenml/

And then another ZenML init:

pip install --upgrade zenml
cd zenml_enabled_repo
zenml init

New Features

Introduced new zenml example CLI sub-group: Easily pull examples via zenml to check it out.

zenml example pull # pulls all examples in `zenml_examples` directory
zenml example pull EXAMPLE_NAME  # pulls specific example
zenml example info EXAMPLE_NAME  # gives quick info regarding example

Thanks Michael Xu for the suggestion!

Updated examples with new zenml examples paradigm for examples.

Bug Fixes + Refactor

ZenML now works on Windows -> Thank you @Franky007Bond for the heads up.
Updated numerous bugs in examples directory. Also updated README's.
Fixed remote orchestration logic -> Now remote orchestration works.
Changed datasource to_config to include reference to backend, metadata, and artifact store.

0.3.7

0.3.7 is a much-needed, long-awaited, big refactor of the Datasources paradigm of ZenML. There are also bug fixes, improvements, and more!

For those upgrading from an older version of ZenML, we ask to please delete their old pipelines dir and .zenml folders and start afresh with a zenml init.

If only working locally, this is as simple as:

cd zenml_enabled_repo
rm -rf pipelines/
rm -rf .zenml/

And then another ZenML init:

pip install --upgrade zenml
cd zenml_enabled_repo
zenml init

New Features

The inner-workings of the BaseDatasource have been modified along with the concrete implementations. Now, there is no relation between a DataStep and a Datasource: A Datasource holds all the logic to version and track itself via the new commit paradigm.
Introduced a new interface for datasources, the process method which is responsible for ingesting data and writing to TFRecords to be consumed by later steps.
Datasource versions (snapshots) can be accessed directly via the commits paradigm: Every commit is a new version of data.
Added JSONDatasource and TFRecordsDatasource.

Bug Fixes + Refactor

A big thanks to our new contributor @aak7912 for the help in this release with issue #71 and PR #75.

Added an example for regression.
compare_training_runs() now takes an optional datasource parameter to filter by datasource.
Trainer interface refined to focus on run_fn rather than other helper functions.
New docs released with a streamlined vision and coherent storyline: https://docs.zenml.io
Got rid of unnecessary Torch dependency with base ZenML version.

0.3.6

0.3.6 is a more inwards-facing release as part of a bigger effort to create a more flexible ZenML. As a first step, ZenML now supports arbitrary splits for all components natively, freeing us from the train/eval split paradigm. Here is an overview of changes:

New Features

The inner-workings of the BaseTrainerStep, BaseEvaluatorStep and the BasePreprocessorStep have been modified along with their respective components to work with the new split_mapping. Now, users can define arbitrary splits (not just train/eval). E.g. Doing a train/eval/test split is possible.
Within the instance of a TrainerStep, the user has access to input_patterns and output_patterns which provide the required uris with respect to their splits for the input and output(test_results) examples.
The built-in trainers are modified to work with the new changes.

Bug Fixes + Refactor

A big thanks to our new super supporter @zyfzjsc988 for most of the feedback that led to bug fixes and enhancements for this release:

#63: Now one can specify which ports ZenML opens its add-on applications.
#64 Now there is a way to list integrations with the following code:

from zenml.utils.requirements_utils import list_integrations.
list_integrations()

Fixed #61: view_anomalies() breaking in the quickstart.
Analytics is now opt-in by default, to get rid of the unnecessary prompt at zenml init. Users can still freely opt-out by using the CLI:

zenml config analytics opt-out

Again, the telemetry data is fully anonymized and just used to improve the product. Read more here

0.3.5

New Features

Added a new interface into the trainer step called test_fn which is utilized to produce model predictions and save them as test results
Implemented a new evaluator step called AgnosticEvaluator which is designed to work regardless of the model type as long as you run the test_fn in your trainer step
The first two changes allow torch trainer steps to be followed by an agnostic evaluator step, see the example here.
Proposed a new naming scheme, which is now integrated into the built-in steps, in order to make it easier to handle feature/label names
Implemented a new adapted version of 2 TFX components, namely the Trainer and the Evaluator to allow the aforementioned changes to take place
Modified the TorchFeedForwardTrainer to showcase how to use TensorBoard in conjunction with PyTorch

Bug Fixes + Refactor

Refactored how ZenML treats relative imports for custom steps. Now:

Updated the Scikit Example, PyTorch Lightning Example, GAN Example accordingly. Now they should work according to their README's.

Big shout out to @SarahKing92 in issue #34 for raising the above issues!

0.3.4

This release is a big design change and refactor. It involves a significant change in the Configuration file structure, meaning this is a breaking upgrade. For those upgrading from an older version of ZenML, we ask to please delete their old pipelines dir and .zenml folders and start afresh with a zenml init.

If only working locally, this is as simple as:

cd zenml_enabled_repo
rm -rf pipelines/
rm -rf .zenml/

And then another ZenML init:

pip install --upgrade zenml
cd zenml_enabled_repo
zenml init

New Features

Introduced another higher-level pipeline: The NLPPipeline. This is a generic NLP pipeline for a text-datasource based training task. Full example of how to use the NLPPipeline can be found here
Introduced a BaseTokenizerStep as a simple mechanism to define how to train and encode using any generic tokenizer (again for NLP-based tasks).

Bug Fixes + Refactor

Significant change to imports: Now imports are way simpler and user-friendly. E.g. Instead of:

from zenml.core.pipelines.training_pipeline import TrainingPipeline

A user can simple do:

from zenml.pipelines import TrainingPipeline

The caveat is of course that this might involve a re-write of older ZenML code imports.

Note: Future releases are also expected to be breaking. Until announced, please expect that upgrading ZenML versions may cause older-ZenML generated pipelines to behave unexpectedly.

Files

RELEASE_NOTES.md

Latest commit

History

RELEASE_NOTES.md

File metadata and controls

0.20.5

What's Changed

0.20.4

What's Changed

0.20.3

What's Changed

New Contributors

0.20.2

What's Changed

0.20.0 / 0.20.1

Warning: Breaking Changes

Overview of Changes

What's Changed

New Contributors

0.13.2

What's Changed

0.13.1

What's Changed

New Contributors

0.13.0

Breaking Changes

What's Changed

New Contributors

0.12.0

What's Changed

0.11.0

Breaking Changes

What's Changed

New Contributors

0.10.0

What's Changed

New Contributors

0.9.0

What's Changed

New Contributors

0.8.1

What's Changed

0.8.0

🧘‍♀️ Extensibility is our middle name

Manage your stacks

Collaboration

CLI improvements

New integrations:

Documentation / ZenBytes etc

➕ Other Updates, Additions and Fixes

🙌 Community Contributions

0.7.3

📊 Experiment Tracking Components

🔎 XGBoost and LightGBM support

📂 Parameterized S3FS support to enable non-AWS S3 storage (minio, ceph)

🧱 New CLI commands to update stack components

🐛 Seldon Core authentication through ZenML secrets

Detailed Changelog

New Contributors

0.7.2

What's Changed

New Contributors

0.7.1

What's Changed

0.7.0

What's Changed

New Contributors

0.6.3

What's Changed

New Contributors

0.6.2

What's Changed

0.6.1

What's changed

0.6.0

📊 Whylogs logging

⛩ New Core Architecture

🗒 Documentation and Example Updates

➕ Other updates, additions and fixes