Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

modified spacing in the article Polymorphic Messages in Kafka Streams #70

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

modified spacing in the article Polymorphic Messages in Kafka Streams #70

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 3 additions & 12 deletions src/blog/polymorphic-messages-in-kafka-streams.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,18 @@
title: Polymorphic Messages in Kafka Streams
image: "/img/polymorphic-messages-in-kafka-streams.png"
imageMeta:
attribution: ''
attributionLink: ''
attribution: ""
attributionLink: ""
author: MC
publish: 2020-08-26 08:29:00 +0000
layout: Post
category: TECHNOLOGY
tags: []

---

## Things usually start simple...

You are designing a Kafka Streams application which must read commands and produce the corresponding business event.

The Avro models you’re expecting to read look like this:

![](/img/1-3.png)
Expand All @@ -27,29 +25,24 @@ You know you can leverage the **sbt-avrohugger** plugin to generate the correspo
![](/img/3-1.png)

Since the messages themselves are pretty straightforward, you decide to create a **monomorphic function** to map properties between each command and the corresponding event.

The resulting topology ends up looking like this:

![](/img/4-1.png)

## ...But then the domain widens

Today new functional requirements have emerged: your application must now handle **multiple types** of assets, each with its own unique properties.

You are pondering how to implement this requirement and make your application more resilient to further changes in behavior.

### Multiple streams

You could split both commands and events into **multiple topics**, one per asset type, so that the corresponding Avro schema stays consistent and its compatibility is ensured.

This solution, however, would have you replicate pretty much the same topology multiple times, so it’s not recommended unless the business logic has to be customized for each asset type.

### “All-and-none” messages

Avro doesn’t support inheritance between records, so any OOP strategy to have assets inherit properties from a common ancestor is unfortunately not viable.

You could however create a “Frankenstein” object with **all the properties** of each and every asset and fill in only those required for each type of asset.

This is definitely the worst solution from an evolutionary and maintainability point of view.

### Union types
Expand All @@ -65,7 +58,6 @@ Luckily for you, Avro offers an interesting feature named **union types**: you c
### Objects with no shape

To cope with this advanced polymorphism, you leverage the [**shapeless**](https://github.com/milessabin/shapeless) library, which introduces the Coproduct type, the perfect companion for the Avro union type.

First of all, you update the custom types mapping of sbt-avrohugger, so that it generates an additional **sealed trait** for each Avro protocol containing multiple records:

![](/img/7.png)
Expand All @@ -87,7 +79,6 @@ Changes to the topology are minimal, as you’d expect:
### A special kind of Serde

Now for the final piece of the puzzle, Serdes. Introducing the [**avro4s**](https://github.com/sksamuel/avro4s) library, which takes Avro GenericRecords above and beyond.

You create a **type class** to extend a plain old Serde providing a brand new method:

![](/img/11.png)
Expand All @@ -102,4 +93,4 @@ Finally, the main program where you combine all ingredients:

## Conclusions

When multiple use cases share (almost) the same business logic, you can create a stream processing application with **ad-hoc polymorphism** and reduce the duplication of code to the minimum, while making your application even more future-proof.
When multiple use cases share (almost) the same business logic, you can create a stream processing application with **ad-hoc polymorphism** and reduce the duplication of code to the minimum, while making your application even more future-proof.