Skip to content

Commit

Permalink
Adds the federated data access layer use-case
Browse files Browse the repository at this point in the history
  • Loading branch information
coco98 committed Jun 18, 2024
1 parent 2bae260 commit 1c75954
Show file tree
Hide file tree
Showing 4 changed files with 96 additions and 27 deletions.
7 changes: 3 additions & 4 deletions content/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,9 @@ Supergraph is an architecture framework that offers reference architectures, des

When a supergraph is built with a GraphQL federation stack, the engine is often called a gateway or a router and the subgraph connectors are often GraphQL services.

A supergraph is typically used for the following 3 benefits:
1. [Self-serve API composition](/api-composition): A self-serve operating model for API integration, orchestration & aggregation
2. Federated data access layer: A federated data layer that allows realtime access to data sources with cross-domain composability (joins, filtering etc.) Related: Data mesh, data products
3. Incremental monolith decomposition: A stable API that provides a zero-downtime and incremental approach to decomposing monoliths
A supergraph is typically used for the following 2 use-cases:
1. **[Self-serve API composition platform](/docs/use-cases/api-composition)**: A self-serve operating model for API integration, orchestration & aggregation
2. **[Federated data access layer](/docs/use-cases/federated-data-access-layer)**: A federated data layer that allows realtime access to data sources with cross-domain composability (joins, filtering etc.) Related: Data mesh, data products

## Strategy and Core concepts
A supergraph approach aims to build a flywheel of data access and supply to incrementally improve self-service access to data and APIs.
Expand Down
9 changes: 5 additions & 4 deletions content/docs/use-cases/api-composition.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ We use the term API composition to encompass three main aspects of working with

A key driver for the Supergraph is the need for API composition. GraphQL (monolithic or federated) is a special case of this need.

## What is API composition & why is it hard?
While domain owners (producers) are owners of a domain API, in a multi-consumer and multi-producer scenario, API consumers often also need specialized APIs that are optimized for their use cases.

![Screen Shot 2024-05-13 at 10 22 07 PM](https://github.com/hasura/supergraph-io/assets/131160/53f3de0c-1f35-4032-a63d-9c2602566073)
Expand All @@ -22,7 +23,7 @@ A supergraph platform solves the following three API composition problems:
2. Aggregation
3. Orchestration

## Integration
## Solving API Integration

Given that domains and domain APIs exist, API integration remains challenging for API consumers for the following reasons:
1. The API output format or protocol is not ideal or optimal for a consumer.
Expand All @@ -33,7 +34,7 @@ Given that domains and domain APIs exist, API integration remains challenging fo

A supergraph provides a systematic way to address these challenges because it provides a common semantic layer and registry for the underlying domains and their APIs. A well-setup supergraph platform provides out-of-the-box solutions for the challenges mentioned above.

## Aggregation (or batching)
## Solving API Aggregation (or batching)

API consumers often need to fetch data from multiple API endpoints. API aggregation or batching, performed closer to the domains, can prevent excessive data transfer and reduce network round trips.

Expand All @@ -47,7 +48,7 @@ A well-setup supergraph platform provides a high level of composability that mak
1. Joins: Fetch data from A and related data from B.
2. Nested filters: Fetch data from A, filtered by a property value of its related data B.

## Orchestration
## Solving API Orchestration

API consumers often need to create reliable workflows that require sequencing multiple API calls interspersed with business logic. Even if the underlying domain APIs exist, API orchestration is challenging because it is the part of the API that is consumer-defined and potentially spans multiple domains.

Expand All @@ -59,7 +60,7 @@ A supergraph platform should provide a clear operating model and technology best
simple aggregation/batching use-cases.


## API composition checklist
## Supergraph checklist for API composition

<table>
<!--thead>
Expand Down
88 changes: 88 additions & 0 deletions content/docs/use-cases/federated-data-access-layer.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
---
title: Federated Data Access Layer
---

# Federated Data Access Layer

A federated data access layer (FDAL) is a data access layer that allows flexible and secure access to multiple, independently governed data sources.

## Typical consumers

FDAL consumers range from consumers that need low-latency, high-concurrency access to realtime data to consumers that need access to analytical data.

- Experience layer products & microservices
- BFFs - Backend for frontends
- Internal products & services - reporting, BI etc.
- 3rd party developers
- Emerging: LLMs, AI applications

![Federated data access layer consumers](https://github.com/hasura/supergraph-io/assets/131160/a1056635-3f45-4a7a-aac1-b1f4bf5a6e39)

## Benefits of a FDAL

1. **Decrease cost, retire legacy / redundant technology and increase efficiency:**
- Avoid moving data around across multiple platforms and layers - especially premature ETL
- Create data consistency and quality controls
- Prevent data duplication
2. **Improve compliance and governance:**
- Enforce a consistent, and if required, a centralized authorization model
- Realtime data access: Provide access to data as soon as it is created
- Create a standardized API for different types of workloads
3. **Improve productivity and collaboration:**
- Provide a stable API to decouple storage layer & physical modelling from domain modelling
- Provide an operating model to incorporate cross-domain business logic, often referred to as the "process layer" or "orchestration layer"
- Codify performance & stability expectations as SLAs
- Incoporate marketplace dynamics: usage analytics and producer / consumer collaboration to drive improvements in the underlying data sources

## Supergraph platform checklist

There are 6 key aspects to building a federated data access layer with a supergraph reference architecture.
A supergraph platform and its technology stack must
support the following capabilities in order to help create a federated data access layer.

### I. Unified semantic model

A unified semantic model to understand the various entities in the supergraph (resources, business methods, authorization policies).

### II. Data modelling & business logic

Subgraph connectors should allow domain or API modelling by leveraging the language of the underlying data-source, while allowing the addition of business logic to handle transformation, validation & authorization.

This ensures subgraphs are "thin" in that its performance characteristics are determined by that of the underlying data source in highly predictable ways.
This also allows subgraphs to be "thick" enough to incorporate the required business logic to expose stable domain concepts.

### III. Authorization

An authorization policy engine that supports:
- Rule / policy composition
- Field (column) level visbility
- Entity (row) level access control
- Optimized integration with data fetching (eg: projection & predicate push-down)

### IV. API design

A standardized API design that can handle (domain driven):
1. Querying resources & invoking business methods
2. Filtering, sorting, pagination & aggregation operations on resources
3. Conventions for relational, event, KV and graph workloads

Aggregation & orchestration layer API needs like (consumer driven):
1. Joining data
2. Nested filtering, sorting pagination & aggregations
3. Orchestration business logic that requires querying resources or invoking methods across multiple domains

### V. Performance & observability

Query planning:
1. Visiblity / explainability in lower environments (development, staging)
2. Tracing in production
3. Data source specific optimizations to reduce load on the upstream data source

Latency tax:
1. Measure and optimize added latency on top of the underlying data source

### VI. Federated ownership

- Independent CI/CD for domain (data source) owners
- Composability (aggregation, orchestration) across domains
- Analytics & collaboration tooling for producers and consumers
19 changes: 0 additions & 19 deletions content/docs/use-cases/federated-data-layer.md

This file was deleted.

0 comments on commit 1c75954

Please sign in to comment.