Skip to content

Releases: ArroyoSystems/arroyo

v0.4.0

13 Jul 18:11
@mwylde mwylde
v0.4.0
d1f20f1
Compare
Choose a tag to compare
v0.4.0

Overview

Arroyo 0.4 brings some big new features like update tables, Debezium support, and a major redesign of the connectors system that makes it much easier to build new connectors. Leveraging that, we've added websocket and fluvio connectors. We're also releasing the initial endpoints for our new REST API, which makes it easier to build automations around Arroyo.

Read on for more details, and check out our docs for full details on existing and new features.

Thanks to all our contributors for this release:

What's next

With 0.4 out, we're already looking ahead to Arroyo 0.5, to be released in early August. The headline feature of 0.5 will be the new Filesystem connector, which will support high throughput, transactional writes from Arroyo into data warehouses and data lakes backed by object stores like S3. We'll also be finishing the transition to the new REST API, adding Redis and Kinesis connectors, and adding a transactional Kafka sink. On the SQL side we'll be working on session windows and support for joining on external tables.

Anything else you'd like to see? Let us know on Discord!

Now on to the release notes.

Features

Update Tables

Arroyo 0.4 brings support for update tables. Exactly what that means is a bit complicated (and we'll dive into it below) but the short version is that you can now use Arroyo to efficiently read and write data from databases like Postgres and MySQL via Debezium, and many queries that were previously unsupported are now supported.

So what are update tables? Let's talk through the semantics of Arroyo tables today, which we'll call append tables going forward.

Take this query:

SELECT store_id, status from orders;

which produces this output stream:

Time                        store     status
7/10/23, 11:34:34 AM PDT    1142      "accepted"
7/10/23, 11:34:34 AM PDT    1737      "accepted"
7/10/23, 11:34:34 AM PDT    1149      "accepted"

This query will output one row for every record that comes in on the orders stream (let's say that's a kafka topic that receives every order). You can think of this as modeling a virtual table with three columns (time, store, and status). Each new order that comes in produces a new row in that table, or in other words is appended.

But what if we have a query that needs other operations beside appends? For example, consider this query:

SELECT store, count(*) AS count
FROM orders
GROUP BY customer;

which models a table with one row per customer. When a new order comes in, we may append a new row if it's a new customer, or we may need to update an existing row if we've already seen that customer. In other words, we need to support updates.

In Arroyo 0.3 that query is not supported, but in 0.4 it will produce an update stream that looks like this:

Time                     previous                              current                               op
7/10/23, 4:03:42 PM PDT  { "orders_store_id": 3, "count": 1 }  { "orders_store_id": 3, "count": 2 }  "u"
7/10/23, 4:03:40 PM PDT  null	                               { "orders_store_id": 1, "count": 1 }  "c"
7/10/23, 4:03:40 PM PDT  null                                  { "orders_store_id": 3, "count": 1 }  "c"

Each output records an update of some kind, either a [c]reate, [u]pdate, or [d]elete. This stream can be used directly, or it can be used to materialize the output into another database like Postgres or MySQL via Debezium, which natively supports this kind of update stream.

Update tables can also be used with Debezium to write to Arroyo from a SQL database CDC source. See the new Debezium tutorial for more details on how to set this up.

Beyond use with Debezium, update tables can also be very useful for efficiently implementing queries where it's important to know when some key enters or leaves a set. For example, for a fraud detection system you may have a set of rules that indicate possibly-fraudulent activity, like this query which looks for sites with suspiciously high click-through rates:

SELECT site as suspicious_site
FROM (
    SELECT site, clicks / impressions as click_through_rate
    FROM (SELECT site,
        SUM(CASE
            WHEN imp_type = 'click' THEN 1 ELSE 0 END) as clicks,
        SUM(CASE
            WHEN imp_type = 'impression' THEN 1 ELSE 0 END) as impressions
        FROM event_stream
    GROUP BY 1)
) WHERE click_through_rate > 0.02;

This query will produce a record with "op": "c" whenever a site first exceeds the threshold, and "op": "d" whenever a site falls below the threshold.

Connector redesign

Connectors integrate Arroyo with external systems. They implement sources that read data from external systems and sinks that write data to external systems.

Arroyo 0.4 brings a major redesign of the connectors system, making it much easier to build new connectors. In previous releases of Arroyo, connectors were deeply integrated with the various Arroyo sub-systems (the console, api, database, sql planner, compiler, etc.) and adding or modifying a connector required changes to all of those systems.

In 0.4, connector implementations are cleanly separated out into the new arroyo-connectors crate. New connectors can be created by implementing a simple trait.

This redesign has allowed us to add a number of new connectors in 0.4 (detailed below), and will accelerate our connector development going forward.

We've also revamped the UI experience around creating sources and sinks, which are now jointly managed in the new Connections tab in the console. This provides a more straightforward experience for creating and managing connections.

Finally, DDL for creating sources and sinks has also been updated to be more consistent and easier to use. For example, a Kafka source can be created with the following SQL:

CREATE TABLE orders (
  customer_id INT,
  order_id INT
) WITH (
  connector = 'kafka',
  format = 'json',
  bootstrap_servers = 'broker-1.cluster:9092,broker-2.cluster:9092',
  topic = 'order_topic',
  type = 'source',
  'source.offset' = 'earliest'
);

New connectors

Arroyo 0.4 includes a number of new connectors leveraging the connector redesign. See the connector docs the full list of supported connectors.

Websocket sources

Arroyo 0.4 adds a new Websocket source, which allows Arroyo to read data from the many available websocket APIs.

For example, Coinbase provides a websocket API that streams the full orderbook for various cryptocurrencies. We can use the new Websocket source to stream that data into Arroyo, and perform real-time analytics on it.

As a simple example, this query computes the average price of Bitcoin in USD over the last minute:

CREATE TABLE coinbase (
    type TEXT,
    price TEXT
) WITH (
    connector = 'websocket',
    endpoint = 'wss://ws-feed.exchange.coinbase.com',
    subscription_message = '{
      "type": "subscribe",
      "product_ids": [
        "BTC-USD"
      ],
      "channels": ["ticker"]
    }',
    format = 'json'
);

SELECT avg(CAST(price as FLOAT)) from coinbase
WHERE type = 'ticker'
GROUP BY hop(interval '5' second, interval '1 minute');

Fluvio source/sink

Arroyo 0.4 adds a new Fluvio source and sink, which allows Arroyo to read and write data from Fluvio, a high-performance distributed streaming platform built on top of Rust and Kubernetes.

Fluvio has support for simple, stateless processing, but with Arroyo it can be extended to perform complex, stateful processing and analytics.

REST API

Today Arroyo is primarily used through the web console, which is great for individual users and small teams. But for more advanced use cases and larger orgs it's important to build automation and integrate Arroyo with internal infrastructure.

Arroyo has always provided a gRPC API that controls all aspects of the system. This is the API that powers the console. But gRPC can be difficult to work with, and it isn't widely supported by existing tools and libraries. We also haven't treated the gRPC API as a stable interface and have made regular breaking changes.

So with this release, we're starting the process of migrating the API to REST, and making it a first-class, stable interface for Arroyo. Arroyo 0.4 adds the first REST endpoints that support pipeline creation, management, and inspection. For example, a SQL pipeline can be created with the following curl command:

curl -XPOST http://localhost:8003/v1/pipelines \
  -H "Content-Type: application/json" \
  -d '{
    "name": "orders",
    "query": "SELECT * FROM orders;"
    "udfs": [],
    "parallelism": 1,
  }'

See the [REST API ...

Read more
Assets 2
Loading

v0.3.0

01 Jun 22:01
@mwylde mwylde
v0.3.0
ff6c985
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.3.0

We're thrilled to announce the 0.3.0 release of Arroyo, our second minor release as an open-source project. Arroyo is a state-of-the-art stream processing engine designed to allow anyone to build complex, stateful real-time data pipelines with SQL.

Overview

The Arroyo 0.3 release focused on improving the flexibility of the system and completeness of SQL support, with the MVP for UDF support, DDL statements, and custom event time and watermarks. There have also many substantial improvements to the Web UI, including error reporting, backpressure monitoring, and under-the-hood infrastructure improvements.

We've also greatly expanded our docs since the last release. Check them out at https://doc.arroyo.dev.

New contributors

We are excited to welcome three new contributors to the project with this release:

Thanks to all new and existing contributors!

What's next

Looking forward to the 0.4 release, we have a lot of exciting changes planned. We're adding the ability to create updating tables with native support for Debezium, allowing users to connect Arroyo to relational databases like MySQL and Postgres. Other planned features include external joins, session windows, and Delta Lake integration.

Excited to be part of the future of stream processing? Come chat with the team on our discord, check out a starter issue and submit a PR, and let us know what you'd like to see next in Arroyo!

Features

UDFs

With this release we are shipping initial support for writing user-defined functions (UDFs) in Rust, allowing users to extend SQL with custom business logic. See the udf docs for full details.

For example, we can register a Rust function:

// Returns the great-circle distance between two coordinates
fn gcd(lat1: f64, lon1: f64, lat2: f64, lon2: f64) -> f64 {
    let radius = 6371.0;

    let dlat = (lat2 - lat1).to_radians();
    let dlon = (lon2 - lon1).to_radians();

    let a = (dlat / 2.0).sin().powi(2) +
        lat1.to_radians().cos() *
            lat2.to_radians().cos() *
                (dlon / 2.0).sin().powi(2);
    let c = 2.0 * a.sqrt().atan2((1.0 - a).sqrt());

    radius * c
}

and call it from SQL:

SELECT gcd(src_lat, src_long, dst_lat, dst_long)
FROM orders;

SQL DDL statements

It's now possible to define sources and sinks directly in SQL via CREATE TABLE statements:

CREATE TABLE orders (
  customer_id INT,
  order_id INT,
  date_string TEXT
) WITH (
  connection = 'my_kafka',
  topic = 'order_topic',
  serialization_mode = 'json'
);

These tables can then be selected from and inserted into to read and write from those systems. For example, we can duplicate the orders topic by inserting from it into a new table:

CREATE TABLE orders_copy (
  customer_id INT,
  order_id INT,
  date_string TEXT
) WITH (
  connection = 'my_kafka',
  topic = 'order_topic',
  serialization_mode = 'json'
);


INSERT INTO orders_copy SELECT * FROM orders;

In addition to connection tables, this release also adds support for views and virtual tables, which are helpful for splitting up complex queries into smaller components.

Custom event time and watermarks

Arroyo now supports custom event time fields and watermarks, allowing users to define their own event time fields and watermarks based on the data in their streams.

When creating a connection table in SQL, it is now possible to define a virtual field generated from the data in the stream and then assign that to be the event time. We can then generate a watermark from that event time field as well.

A complete example looks like this:

CREATE TABLE orders (
  customer_id INT,
  order_id INT,
  date_string TEXT,
  event_time TIMESTAMP GENERATED ALWAYS AS (CAST(date_string as TIMESTAMP)),
  watermark TIMESTAMP GENERATED ALWAYS AS (event_time - INTERVAL '15' SECOND)
) WITH (
  connection = 'my_kafka',
  topic = 'order_topic',
  serialization_mode = 'json',
  event_time_field = 'event_time',
  watermark_field = 'watermark'
);

For more on the underlying concepts of event times and watermarks, see the concept docs.

Additional SQL features

Beyond UDFs and DDL statements, we have continued to expand the completeness of our SQL support with addition of case statements and regex functions:

Server-Sent Events source

We've added a new source which allows reading from Server-Sent Events APIs (also called EventSource). SSE is a simple protocol for streaming data from HTTP servers and is a common choice for web applications. See the SSE source documentation for more details, and take a look at the new Mastodon trends tutorial that makes
uses of it

  • Add event source source operator by @mwylde in #106
  • Add HTTP connections and add support for event source tables in SQL by @mwylde in #119

Web UI

This release has seen a ton of improvements to the web UI.

Improvements

Read more
Assets 2
Loading

v0.2.0

02 May 21:30
@mwylde mwylde
release-0.2.0
fdb1560
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.2.0

Arroyo 0.2.0

Arroyo is a new, state-of-the-art stream processing engine that makes it easy to build complex real-time data pipelines with SQL. This release marks our first versioned release of Arroyo since we open-sourced the engine in April.

We're excited to welcome three new contributors to the project:

With the 0.2.0 release, we are continuing to push forward on features, stability, and productionization. We’ve added native Kubernetes support and easy deployment via a Helm chart, expanded our SQL support with features like JSON functions and windowless joins, and made many more fixes and improvements detailed below.

Looking forward to the 0.3.0 release, we will continue to improve our SQL support with the ability to create sources and sinks directly as SQL tables, views, UDFs and external joins. We will also be adding a native Pulsar connector and making continued improvements in performance and reliability.

Excited to be part of the future of stream processing? Come chat with the team on our discord, check out a starter issue and submit a PR, and let us know what you’d like to see next in Arroyo!

Features

Native Kubernetes support

As of release 0.2.0, Arroyo can natively target Kubernetes as a scheduler for running pipelines. We now also support easily running the Arroyo control plane on Kubernetes using our new helm chart.

Getting started is as easy as

$ helm repo add arroyo https://arroyosystems.github.io/helm-repo
$ helm install arroyo arroyo/arroyo \
  --set s3.bucket=my-bucket,s3.region=us-east-1

See the docs for all the details.

Nomad deployments

Arroyo has long had first-class support for Nomad as a scheduler, where we take advantage of the very low-latency and lightweight scheduling support. Now we also support Nomad as an easy deploy target for the control plane as well via a nomad pack.

See the docs for more details.

  • Support for deploying Arroyo to a nomad cluster by @mwylde in #50

SQL features

With this release we are making big improvements in SQL completeness. Notably, we’ve made our JSON support much more flexible with the introduction of SQL JSON functions including get_json_objects, get_first_json_object, and extract_json_string.

We’ve also added support for windowless joins.

Here are some of the highlights:

Connectors, Web UI, and platform support

Arroyo now supports SASL authentication for Kafka and FreeBSD

Fixes

Improvements

See the full changelog: https://github.com/ArroyoSystems/arroyo/commits/release-0.2.0

Contributors

rtyler, mwylde, and 3 other contributors
Assets 2
Loading