-
Notifications
You must be signed in to change notification settings - Fork 119
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
8 changed files
with
117 additions
and
173 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
This instruction assumes that you are trying it on a Linux host, but it will be | ||
pretty much the same on Mac. | ||
|
||
The easiest way to install Kafka-Pixy is to download and unpack a release | ||
archive: | ||
|
||
``` | ||
curl -L https://github.com/mailgun/kafka-pixy/releases/download/v0.13.0/kafka-pixy-v0.13.0-linux-amd64.tar.gz | tar xz | ||
``` | ||
|
||
Create a configuration file using `default.yaml` as a template: | ||
|
||
``` | ||
cd kafka-pixy-v0.13.0-linux-amd64 | ||
cp default.yaml config.yaml | ||
``` | ||
|
||
The default settings in the config file should be good enough for you to give | ||
Kafka-Pixy a try, you can always fine turn it later. But for the first time you | ||
need to at least point it to your Kafka and Zookeeper clusters. | ||
|
||
When the config file is updated you can start Kafka-Pixy: | ||
|
||
``` | ||
./kafka-pixy --config config.yaml | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,65 @@ | ||
Note: HTTP API is only provided to test Kafka-Pixy from command line and use | ||
in use in operations (consumer/offset API). To produce/consume messages | ||
please use [gRPC](http://www.grpc.io/docs/) API. You can use pre-generated | ||
client stubs shipped with kafka-pixy for [Python](gen/python) and | ||
[Golang](gen/golang), or generate them yourself from [grpc.proto](grpc.proto). | ||
|
||
This tutorial assumes that topic `foo` exists in your Kafka cluster or your | ||
Kafka is configured to create topics on demand. | ||
|
||
To make sure that you will be able to consume the first message you produce in | ||
the scope of this tutorial we need to start by making a consume call from a | ||
consumer group (e.g. `bar`): | ||
|
||
``` | ||
curl -G localhost:19092/topics/foo/messages?group=bar | ||
``` | ||
|
||
The call is exected to fail after the configured configured | ||
[long polling timeout](https://github.com/mailgun/kafka-pixy/blob/master/default.yaml#L103) | ||
elapses. But the important side effect is that initial offsets will be stored | ||
in the Kafka cluster for the used consumer group. | ||
|
||
Note: You can use any consumer group name, but it has to be used consistently | ||
in all calls. | ||
|
||
A message can be produced byt the following call: | ||
|
||
``` | ||
curl -X POST localhost:19092/topics/foo/messages?sync \ | ||
-d 'May the Force be with you!' | ||
``` | ||
|
||
The message was produced in `sync` mode, that by [default](https://github.com/mailgun/kafka-pixy/blob/master/default.yaml#L70-L78) | ||
means that Kafka-Pixy would wait for all ISR brokers to commit the message | ||
before replying with success. That also ensures that partition and offset | ||
that the message was committed to are returned in response. E.g.: | ||
|
||
```json | ||
{ | ||
"partition": 7, | ||
"offset": 974563 | ||
} | ||
``` | ||
|
||
To consume earlier produced message call: | ||
|
||
``` | ||
curl -G localhost:19092/topics/foo/messages?group=bar | ||
``` | ||
|
||
The output provides the retrieved message as a base64 encoded value along with | ||
some metadata: | ||
|
||
```json | ||
{ | ||
"key": null, | ||
"value": "YmxhaCBibGFoIGJsYWg=", | ||
"partition": 7, | ||
"offset": 974563 | ||
} | ||
``` | ||
|
||
The `key` is null in our case because we did not specify one when the message | ||
was produced and therefore the partition the message was committed to had been | ||
selected randomly. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
If you need this, please create an issue and I will add the content in no time. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
If you need this, please create an issue and I will add the content in no time. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters