v0.6.3

Major Changes

• Some bug fixes and performance enhancements related to watermark
• Pre-released features for v0.7

Check out v0.6 release blog.

Pull Requests

• feat: watermark otwatcher enhancement (#364)
• refactor(docs): group docs in categories (#362)
• chore: vertex controller stop watching pod create events (#361)
• chore(deps): bump loader-utils from 2.0.3 to 2.0.4 in /ui (#356)
• fix(controller): vertex nil check for edge listing. Fixes #352 (#353)
• chore: minor window refactor (#347)
• fix: data race in pbq manager. Closes #348 (#349)
• Chore: Windower interface. closes #234 (#340)
• chore: fix broken development doc and remove incorrect make cmds (#346)
• feat: add minikube, kind and podman support (#206)
• refactor: Close watermark fetcher and publisher correctly (#336)
• passing window information inside the context (#341)
• feat: timestamp in UI to display milliseconds. closes #280 (#337)
• Simple reduce pipeline. Fixes #289 (#317)
• feat: add blackhole sink. Closes #329 (#330)
• fix: move watermark based on the head of the read batch (#332)
• feat: configurable jetstream storage (#328)
• feat: support adding sidecars in vertex pods. Closes #323 (#325)

Contributors

@chromevoid
@dseapy
@DevilAeron
@dratler
@ashwinidulams
@whynowy
@yhl25
@vigith

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.6.3/config/install.yaml

v0.6.2

Major Changes

• Fixed some watermarks related bugs
• Pre-released features for v0.7

Check out v0.6 release blog.

Pull Requests

• feat: populate watermark settings to vertex spec. Closes #320 (#321)
• chore: use github action services and upgrade Jetstream (#322)
• doc: add few use cases (#318)
• chore: set GOMEMLIMIT and increase graceshutdownperiod (#319)
• Chore: run in sdks-e2e tests, python-udsink log check before go-udsink (#315)
• fix: jetstream build watermark progressors bug (#316)
• feat: update watermark offset bucket implementation (#307)
• chore: upgrade xmldom to address security concerns (#309)
• feat: shuffling support (#306)
• chore: Add boilerplate headers (#308)
• feat: customize init-container resources. Closes #303 (#304)
• feat: watermark - remove non-share OT bucket option (#302)
• feat: customization for batch jobs. Closes #259 (#300)
• refactor: abstract pod template (#296)
• feat: customization for daemon deployment. Closes #223 (#290)
• chore: use milliseconds in HTTP event-time header (#295)
• feat: add pvc support for reduce vertex PBQ (#292)
• fix(doc): hyperlink for security doc (#288)
• feat: support adding init containers to vertices. Closes #284 (#285)
• docs: fix spelling in docs and examples (#286)
• fix: retry when getting EOF error at E2E test (#281)
• feat: Watermark millisecond. Fixes #201 (#278)
• feat: add pipeline node counts and age to printcolumn. Closes #267 (#282)

Contributors

@chromevoid
@dseapy
@whynowy
@KeranYang
@vigith

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.6.2/config/install.yaml

v0.6.1

Major Changes

• Fixed some watermarks related bugs
• Fixed the issue that some unexpected behaviors will happen due to missing default values when doing namespace scoped installation
• Initial integration for reduce vertex

Check out v0.6 release blog.

Pull Requests

• fix(manifests): Include ServiceAccount in namespace scoped install (#240)
• fix(docs): fix a typo (#241)
• feat: introduce reduce UDF. Closes #246 (#262)
• feat: add pandoc to required tools development doc. Closes #276 (#277)
• feat: add isbsvc type and age to printcolumn. Closes #268 (#275)
• fix: watermark consumer fix (#273)
• refactor: generalize watermark fetching as an interface of ISB service. Fixes #252 (#263)
• fix: set default property values for minimal CRD installation (#264)
• fix: validate only one isbsvc implementation type is defined. Fixes #269 (#271)
• fix: main branch make build failure: math.MaxInt64 for int type (#265)
• fix: nil pointer deref when running example with minimal CRD. Fixes #260 (#261)
• fix: retry only the failed offsets (#255)
• docs: update ISB redis doc (#258)
• fix: re-enable build constraint on processor manager test. Fixes #256 (#257)
• fix: container resource for jetstream isbsvc. Fixes #253 (#254)
• fix: update vertex watermark fetching logic. Fixes: #134 (#245)
• chore: refactor to include partition.ID for reducer (#243)
• chore: introduce simple int offset (#250)
• fix: watermark watcher leak (#242)
• feat: Support running UX server with namespace scope. Fixes #248 (#249)
• test: reduce test with inmemory watermark store (#239)
• fix: Watermark close fix and removed the nil check (#238)
• fix: skip publishing watermarks to unexpected vertices. Fixes #235 (#236)
• fix: update default watermark to -1. Fixes #133 (#218)
• feat: support disabling TLS and changing port for UX server (#228)
• feat: reducer for stream aggregation without fault tolerance (#208)
• feat: in-memory watermark store for better testing (#216)
• Add USERS.md (#221)
• fix(watermark): generator should not publish wm for every message (#217)
• chore: reference the Payload directly when writing to Redis ISB to avoid extra marshaling (#213)

Contributors

@chromevoid
@dseapy
@whynowy
@edlee2121
@KeranYang
@si3nloong
@yhl25
@ashwinidulams
@vigith

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.6.1/config/install.yaml

v0.6.0

Major Changes

First official Numaflow release, which contains following major features, check the documentation for all the details.

Check out v0.6 release blog.

• Stream processing pipeline with Source, User Defined Function, and Sink vertices
• Pipeline autoscaling
• Native supported sources and sinks, as well as user defined sinks.
• Watermarks support
• Prometheus metrics
• User interface

Pull Requests

• fix(autoscaling): Ack pending should be included in total pending calculation (#212)
• chore: upgrade xmldom to fix potential security issue (#209)
• fix(autoscaling): Skip autoscaling if vertex is not in running phase (#207)
• chore: add read and written bytes counter (#202)
• chore: Replaced node info text with rich format (#200)
• feat: ISBSVC add support for redis cluster mode (#195)
• ci: add sdk test category (#197)
• docs: add and refine some docs. (#196)
• refactor: move controllers package to pkg/reconciler (#192)
• fix: update udf fetchWatermark and publishWatermark initial values (#193)
• chore: fix security alert (#191)
• chore: generate swagger and schema json (#188)
• chore: probe udsink healthiness (#183)
• chore: clean up code of http protocol udf and udsink (#182)
• fix(docs): readme for UI development (#181)

Contributors

@chromevoid
@whynowy
@Krithika3
@qhuai
@sarabala1979
@kohlisid
@vigith
@yhl25

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.6.0/config/install.yaml

v0.5.6

Major Changes

• User Defined Functions and Sinks switched to gRPC protocol
• Liveness check includes UDF health check
• In-memory PBQ implementation (for windowing)
• Watermark implementation for all the sources
• Support to publish watermark to sink buckets
• Shuffle implementation (for windowing)

Pull Requests

• feat: Add udf gRPC support Fixes #145 (#146)
• chore: more advanced install options (#154)
• feat: check if UDF is running in liveness probe (#156)
• docs:Update readme and quickstart (#162)
• feat: PBQ implementation (#155)
• docs:updated quickstart (#164)
• chore: add mkdocs check (#166)
• chore: keep stores together (#167)
• feat: watermark for sources (#159)
• feat: windowing operations (#157)
• chore: publish watermark to sink bucket and refactor. (#170)
• feat: Shuffle implementation (#169)
• chore: Update README.md (#175)
• fix: broken path (#176)
• chore: update README.md (#178)
• fix: use scale.max if it is set (#179)
• fix: numaflow-go udf example & docs (#177)
• feat: grpc udsink (#174)
• chore: enable jetstream 2.9.0 (#173)
• chore: pass watermark to UDF and fix e2e tests (#171)

Contributors

@vigith
@saradhis
@yhl25
@chromevoid
@whynowy
@edlee2121
@amitkalamkar

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.5.6/config/install.yaml

v0.5.5

Major Changes

• Fixed a controller memory leak issue.
• Fixed couple of UI bugs.
• Expose minimal CRD definition and more installation options.

Pull Requests

• fix: daemon service client memory leak (#161)
• chore: more advanced install options (#154)
• refactor: some refactor on watermark (#149)
• chore: use string as the type of key (#153)
• docs: update current docs (#147)
• fix: Fixed JS bug (#144)
• fix: watermark needs nil check
• chore: fix #88 (#140)
• docs: minor autoscaling doc revise (#131)
• fix: pipeline UI broken when vertex scaling down to 0 (#132)

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.5.5/config/install.yaml

v0.5.4

Major Changes

• Vertex autoscaling
• Watermark exposed in UI
• JetStream InterStepBuffer service auto reconnection - addition to Nats connection auto reconnection
• Support namespace scope installation

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.5.4/config/install.yaml

v0.5.3

Major Changes

• UI
• Vertex processing rate and pending messages available in prometheus metrics
• Some breaking changes
  1. udfWorkers is removed from both pipeline and vertex limits;
  2. bufferMaxLength and bufferUsageLimit moved from vertex.limits to edge.limits;
  3. Numaflow controller deployment name changed from controller-manager to numaflow-controller.

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.5.3/config/install.yaml

Upgrading Guide

Since there are some breaking changes in this release, upgrading from old versions need to follow below steps:

1. Clean up all the limits.udfWorkers from Pipeline spec if any;
2. Clean up udfWorkers from vertices[*].limits if any;
3. Clean up bufferMaxLength and bufferUsageLimit from vertices[*].limits if any;
4. Delete controller-manager deployment with kubectl -n numaflow-system delete deploy controller-manager;
5. Install the new version following the installation guide above;
6. If there's any bufferMaxLength or bufferUsageLimit cleaned up in step 3, add them to edges[*].limits with same names, and apply the new pipeline spec.

v0.5.2

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.5.2/config/install.yaml

v0.5.1

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.5.1/config/install.yaml