Marquez 0.15.1

Added

• Factored out common functionality in Python airflow integration @mobuchowski
• Added Airflow task run macro to expose task run id @collado-mike

Changed

• Refactored ValuesAverageExpectationParser to ValuesSumExpectationParser and ValuesCountExpectationParser @collado-mike
• Updated SparkListener to extend Spark's SparkListener abstract class @collado-mike

Fixed

• Use current project version in spark openlineage client @mobuchowski
• Rewrote LineageDao queries and LineageService for performance @collado-mike
• Updated lineage query to include new jobs that have no job version yet @collado-mike

Marquez 0.15.0

Added

• Add tracing visibility @julienledem
• New Add snowflake extractor 🎉 @mobuchowski
• Add SSLContext to MarquezClient @lewiesnyder
• Add support for LogicalRDDs in spark plan visitors @collado-mike
• New Add Great Expectations based data quality facet support 🎉 @mobuchowski

Changed

• Augment tutorial instructions & screenshots for Airflow example @rossturk
• Rewrite correlated subqueries when querying the lineage_events table @collado-mike

Fixed

• Web time formatting display fix @kachontep

Marquez 0.14.2

Changed

• Unpin requests dep in marquez-airflow integration @wslulciuc
• Unpin attrs dep in marquez-airflow integration @wslulciuc

Marquez 0.14.1

Changed

• Updated dataset lineage query to find most recent job that wrote to it @collado-mike
• Pin http-proxy-middleware to 0.20.0 @wslulciuc

Marquez 0.14.0

Added

• GA tag for website tracking @rossturk
• Basic CTE support in marquez-airflow @mobuchowski
• Airflow custom facets, bigquery statistics facets @mobuchowski
• Unit tests for class JobVersionDao @wslulciuc
• Sentry tracing support @julienledem
• OpenLineage facets support to API response models 🎉 @wslulciuc

Changed

• BigQueryRelationTransformer and deleted BigQueryNodeVisitor @collado-mike
• Bump postgres to 12.1.0 @wslulciuc
• Update spark job name to reflect spark application name and execution node @collado-mike
• Update marquez-airflow integration to use OpenLineage 🎉 @mobuchowski
• Migrate tests to junit 5 @mobuchowski
• Rewrite lineage IO sql queries to avoid job_versions_io_mapping_* tables @collado-mike
• Updated OpenLineage impl to only update dataset version on run completion @collado-mike

Marquez 0.13.1

Changed

• Remove unused implementation of SQL parser in marquez-airflow @mobuchowski

Fixed

• Add inputs and outputs to lineage graph @henneberger
• Updated NodeId regex to support URIs with scheme and ports @collado-mike

Marquez 0.13.0

Added

• Secret support for helm chart @KevinMellott91
• New seed cmd to populate marquez database with source, dataset, and job metadata allowing users to try out features of Marquez (data lineage, view job run history, etc) 🎉
• Docs on applying db migrations manually
• New LineageAPI to support data lineage queries 🎉 @henneberger
• Support for logging errors via sentry
• New Airflow example with Marquez 🎉

Changed

• Update OpenLinageDao to stop converting URI structures to contain underscores instead of colons and slashes @collado-mike
• Bump testcontainers dependency to v1.15.2 @ ShakirzyanovArsen
• Register output datasets for a run lazily @henneberger
• Refactor spark plan traversal to find input/output datasets from datasources @collado-mike
• Web UI project settings and default marquez port @phixMe
• Associate dataset inputs on run start @henneberger

Fixed

• Dataset description is not overwritten on update @henneberger
• Latest tags are returned from dataset @henneberger
• Airflow integration tests on forked PRs @mobuchowski
• Empty nominal end time support @henneberger
• Ensure valid dataset fields for OpenLineage @henneberger
• Ingress context templating for helm chart @KulykDmytro

Marquez 0.12.2

Changed

• Use alpine image for marquez reducing image size by +50% @KevinMellott91
• Use alpine image for marquez-web reducing image size by +50% @KevinMellott91

Fixed

• Ensure marquez.DAG is (de)serializable

Marquez 0.12.0

Added

• Modules: api, web, clients , chart, and integrations
• Working airflow example
• runs table indices for columns: created_at and current_run_state @phixMe
• New /lineage endpoint for OpenLineage support @henneberger
• New graphql endpoint @henneberger
• New spark integration @henneberger
• New API to list versions for a dataset

Changed

• Drop Source.type enum (now a string type)

Fixed

• Replace jdbi.getHandle() with jdbi.withHandle() to free DB connections from pool @henneberger
• Fix RunListener when registering outside of the MarquezContext builder @henneberger

Marquez 0.11.4-rc.3

Added

• Add indices to runs table for created_at and current_run_state columns @phixMe