From bfa0e86e4b515b728959fd956b72d92faaea3d02 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Mon, 9 Oct 2023 23:16:04 +0200 Subject: [PATCH] [DPE-2460] Tutorial for Kafka K8s (#64) --- docs/how-to/h-manage-app.md | 46 +++++- docs/index.md | 23 +-- docs/tutorial/t-cleanup-environment.md | 21 +++ docs/tutorial/t-deploy.md | 122 +++++++++++++++ docs/tutorial/t-enable-encryption.md | 100 ++++++++++++ docs/tutorial/t-manage-passwords.md | 128 ++++++++++++++++ docs/tutorial/t-overview.md | 36 +++++ docs/tutorial/t-relate-kafka.md | 203 +++++++++++++++++++++++++ docs/tutorial/t-setup-environment.md | 63 ++++++++ 9 files changed, 719 insertions(+), 23 deletions(-) create mode 100644 docs/tutorial/t-cleanup-environment.md create mode 100644 docs/tutorial/t-deploy.md create mode 100644 docs/tutorial/t-enable-encryption.md create mode 100644 docs/tutorial/t-manage-passwords.md create mode 100644 docs/tutorial/t-overview.md create mode 100644 docs/tutorial/t-relate-kafka.md create mode 100644 docs/tutorial/t-setup-environment.md diff --git a/docs/how-to/h-manage-app.md b/docs/how-to/h-manage-app.md index 82dca680..0fef961f 100644 --- a/docs/how-to/h-manage-app.md +++ b/docs/how-to/h-manage-app.md @@ -57,7 +57,51 @@ unit-data-integrator-0: started: 2023-01-27 14:22:51 +0000 UTC ``` -## Internal Password rotation + +## Password rotation + +### External clients + +#### With client application downtime + +The easiest way to rotate user credentials of client applications is by removing and then re-relating +the application (either a charm supporting the `kafka-client` interface or a `data-integrator`) with the `kafka-k8s` charm + +```shell +juju remove-relation kafka-k8s +# wait for the relation to be torn down +juju relate kafka-k8s +``` + +The successful credential rotation can be confirmed by retrieving the new password with the action `get-credentials`. + +#### Without client application downtime + +In some use-cases credentials should be rotated with no or limited application downtime. +If credentials should be rotated with no or limited downtine, you can deploy a new charm with the same permissions and resource definition, e.g. + +```shell +juju deploy data-integrator rotated-user --channel stable \ + --config topic-name=test-topic --config extra-user-roles=admin +``` + +The `data-integrator` charm can then be related to the `kafka-k8s` charm to create a new user +```shell +juju relate kafka-k8s rotated-user +``` + +At this point, we effectively have two overlapping users, therefore allowing applications to swap the password +from one to another. +If the applications consist of fleets of independent producers and consumers, user credentials can be rotated +progressively across fleets, such that no effective downtime is achieved. + +Once all applications have rotated their credentials, it is then safe to remove data first `data-integrator` charm + +```shell +juju remove-application data-integrator +``` + +### Internal Password rotation The operator user is used internally by the Charmed Kafka Operator, the `set-password` action can be used to rotate its password. ```shell diff --git a/docs/index.md b/docs/index.md index cff837cf..757c0ab0 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,4 +1,4 @@ -## Charmed Kafka K8s Documentation +# Charmed Kafka K8s Documentation The Charmed Kafka K8s Operator delivers automated operations management from day 0 to day 2 on the [Apache Kafka](https://kafka.apache.org) event streaming platform deployed on top of a [Kubernetes cluster](https://kubernetes.io/). It is an open source, end-to-end, production ready data platform on top of cloud native technologies. @@ -40,24 +40,3 @@ Please see the [Juju SDK docs](https://juju.is/docs/sdk) for guidelines on enhan ## License The Charmed Kafka K8s Operator is free software, distributed under the Apache Software License, version 2.0. See [LICENSE](https://github.com/canonical/kafka-operator/blob/main/LICENSE) for more information. - -# Navigation - -| Level | Path | Navlink | -|-------|----------------------|--------------------------------------------------------------------------| -| 1 | how-to | [How To]() | -| 2 | h-manage-units | [Manage units](/t/charmed-kafka-k8s-how-to-manage-units/10295) | -| 2 | h-enable-encryption | [Enable encryption](/t/charmed-kafka-k8s-how-to-enable-encryption/10289) | -| 2 | h-manage-app | [Manage applications](/t/charmed-kafka-k8s-how-to-manage-app/10293) | -| 2 | h-enable-monitoring | [Enable Monitoring](/t/charmed-kafka-k8s-how-to-enable-monitoring/10291) | -| 1 | reference | [Reference]() | -| 2 | r-actions | [Actions](https://charmhub.io/kafka-k8s/actions) | -| 2 | r-configurations | [Configurations](https://charmhub.io/kafka-k8s/configure) | -| 2 | r-libraries | [Libraries](https://charmhub.io/kafka-k8s/libraries/helpers) | - -# Redirects - -[details=Mapping table] -| Path | Location | -| ---- | -------- | -[/details] \ No newline at end of file diff --git a/docs/tutorial/t-cleanup-environment.md b/docs/tutorial/t-cleanup-environment.md new file mode 100644 index 00000000..1361fd57 --- /dev/null +++ b/docs/tutorial/t-cleanup-environment.md @@ -0,0 +1,21 @@ +# Cleanup and extra info + +This is part of the [Charmed Kafka K8s Tutorial](/t/charmed-kafka-k8s-documentation-tutorial-overview/11945). Please refer to this page for more information and the overview of the content. + +## Remove and cleanup environment +If you're done with testing and would like to free up resources on your machine, just remove Multipass VM. +*Warning: when you remove VM as shown below you will lose all the data in Kafka and any other applications inside Multipass VM!* +```shell +multipass delete --purge my-vm +``` + +## Next Steps + +In this tutorial we've successfully deployed Kafka, added/removed users, connected client applications and even enabled and disabled TLS. +If you're looking for what to do next you can: +- Run [Charmed Kafka on VMs](https://github.com/canonical/kafka-operator). +- Check out our Charmed offerings of [MySQL](https://charmhub.io/mysql-k8s), [PostgreSQL](https://charmhub.io/postgresql-k8s), [MongoDB](https://charmhub.io/mongodb-k8s). +- Read about [High Availability Best Practices](https://canonical.com/blog/database-high-availability) +- [Report](https://github.com/canonical/kafka-k8s-operator/issues) any problems you encountered. +- [Give us your feedback](https://chat.charmhub.io/charmhub/channels/data-platform). +- [Contribute to the code base](https://github.com/canonical/kafka-k8s-operator) \ No newline at end of file diff --git a/docs/tutorial/t-deploy.md b/docs/tutorial/t-deploy.md new file mode 100644 index 00000000..086f7b3d --- /dev/null +++ b/docs/tutorial/t-deploy.md @@ -0,0 +1,122 @@ +# Get a Charmed Kafka and Zookeeper up and running + +This is part of the [Charmed Kafka K8s Tutorial](/t/charmed-kafka-k8s-documentation-tutorial-overview/11945). Please refer to this page for more information and the overview of the content. + +## Deploy + +To deploy Charmed Kafka K8s, all you need to do is run the following commands, which will automatically fetch [Kafka](https://charmhub.io/kafka-k8s?channel=3/stable) and [Zookeeper](https://charmhub.io/zookeeper-k8s?channel=3/stable) charms from [Charmhub](https://charmhub.io/) and deploy them to your model. For example, to deploy a 5 Zookeeper unit and 3 Kafka unit cluster, you can simply run + +```shell +$ juju deploy zookeeper-k8s -n 3 --channel=3/edge +$ juju deploy kafka-k8s -n 3 --channel=3/edge +``` + +After this, it is necessary to connect them: + +```shell +$ juju relate kafka-k8s zookeeper-k8s +``` + +Juju will now fetch Charmed Kafka K8s and Charmed Zookeeper K8s and begin deploying it to the local MicroK8s. This process can take several minutes depending on how provisioned (RAM, CPU, etc) your machine is. You can track the progress by running: +```shell +juju status --watch 1s +``` + +This command is useful for checking the status of Charmed Zookeeper K8s and Charmed Kafka K8s. Some of the helpful information it displays include IP pods addresses, ports, state, etc. +The command updates the status of the cluster every second and as the application starts you can watch the status and messages of Charmed Kafka K8s and Charmed Zookeeper K8s change. + +Wait until the application is ready - when it is ready, `juju status --watch 1s` will show: +```shell +... +Model Controller Cloud/Region Version SLA Timestamp +tutorial microk8s microk8s/localhost 3.1.5 unsupported 17:22:21+02:00 + +App Version Status Scale Charm Channel Rev Address Exposed Message +kafka-k8s active 3 kafka-k8s 3/edge 39 10.152.183.237 no +zookeeper-k8s active 3 zookeeper-k8s 3/edge 33 10.152.183.134 no + +Unit Workload Agent Address Ports Message +kafka-k8s/0 active idle 10.1.36.78 +kafka-k8s/1 active idle 10.1.36.80 +kafka-k8s/2* active idle 10.1.36.79 +zookeeper-k8s/0 active idle 10.1.36.84 +zookeeper-k8s/1* active idle 10.1.36.86 +zookeeper-k8s/2 active idle 10.1.36.85 +``` +To exit the screen with `juju status --watch 1s`, enter `Ctrl+c`. + +## Retrieving Kafka cluster admin credentials + +To watch the process, `juju status` can be used. Once all the units show as `active|idle` the credentials to access a broker can be queried with: +```shell +juju run kafka-k8s/leader get-admin-credentials +``` + +The output of the previous command is something like this: +```shell +Running operation 1 with 1 task + - task 2 on unit-kafka-k8s-2 + +Waiting for task 2... +client-properties: |- + security.protocol=SASL_PLAINTEXT + sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="0FIQ5QxSaNfl1bXHtV5dyttb21Nbzmpp"; + sasl.mechanism=SCRAM-SHA-512 + bootstrap.servers=kafka-k8s-1.kafka-k8s-endpoints:9092,kafka-k8s-2.kafka-k8s-endpoints:9092,kafka-k8s-0.kafka-k8s-endpoints:9092 +password: 0FIQ5QxSaNfl1bXHtV5dyttb21Nbzmpp +username: admin +``` + +Providing you the `username` and `password` of the Kafka cluster admin user. + +**IMPORTANT** Note that when no other application is related to Kafka, the cluster is secured-by-default and external listeners (binded to port 9092) are disabled, thus preventing any external incoming connection. + +Nevertheless, it is still possible to run a command from within the Kafka cluster. To do so, log in into one of the Kafka container in one of the units + +```shell +juju ssh --container kafka kafka-k8s/leader sudo -i +``` + +The Charmed Kafka K8s image ships with the Apache Kafka `bin/*.sh` commands, that can be found under `/opt/kafka/bin/`. +These allow admin to do various administrative tasks, e.g `bin/kafka-config.sh` to update cluster configuration, `bin/kafka-topics.sh` for topic management, and many more! +Within the image you can also find a `client.properties` file that already provides the relevant settings to connect to the cluster using the CLI: + +```shell +export CLIENT_PROPERTIES=/etc/kafka/client.properties +``` + +Since we don't have any client applications related yet and therefore external listeners are initially closed, if you wish to run a command from the cluster you ought to use the internal listeners exposed at ports 19092. +```shell +export INTERNAL_LISTENERS=kafka-k8s-1.kafka-k8s-endpoints:19092,kafka-k8s-2.kafka-k8s-endpoints:19092,kafka-k8s-0.kafka-k8s-endpoints:19092 +``` + +We are now ready to perform some administrative tasks. For example, in order to create a topic, you can run: +```shell +/opt/kafka/bin//kafka-topics.sh \ + --create --topic test_topic \ + --bootstrap-server $INTERNAL_LISTENERS \ + --command-config $CLIENT_PROPERTIES +``` + +You can similarly then list the topic, using +```shell +/opt/kafka/bin//kafka-topics.sh \ + --list \ + --bootstrap-server $INTERNAL_LISTENERS \ + --command-config $CLIENT_PROPERTIES +``` + +making sure the topic was successfully created. + +You can finally delete the topic, using + +```shell +/opt/kafka/bin//kafka-topics.sh \ + --delete --topic test_topic \ + --bootstrap-server $INTERNAL_LISTENERS \ + --command-config $CLIENT_PROPERTIES +``` + +However, although the commands above can run within the cluster, it is generally recommended during operations +to enable external listeners and use these for running the admin commands from outside the cluster. +To do so, as we will see in the next section, we will deploy a [data-integrator](https://charmhub.io/data-integrator) charm and relate it to Kafka. \ No newline at end of file diff --git a/docs/tutorial/t-enable-encryption.md b/docs/tutorial/t-enable-encryption.md new file mode 100644 index 00000000..8b165cdb --- /dev/null +++ b/docs/tutorial/t-enable-encryption.md @@ -0,0 +1,100 @@ +# Enable Security in your Kafka K8s deployment + +This is part of the [Charmed Kafka K8s Tutorial](/t/charmed-kafka-k8s-documentation-tutorial-overview/11945). Please refer to this page for more information and the overview of the content. + +## Transport Layer Security (TLS) +[TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security) is used to encrypt data exchanged between two applications; it secures data transmitted over the network. Typically, enabling TLS within a highly available database, and between a highly available database and client/server applications, requires domain-specific knowledge and a high level of expertise. Fortunately, the domain-specific knowledge has been encoded into Charmed Kafka K8s. This means (re-)configuring TLS on Charmed Kafka K8s is readily available and requires minimal effort on your end. + +Again, relations come in handy here as TLS is enabled via relations; i.e. by relating Charmed Kafka K8s to the [TLS Certificates Charm](https://charmhub.io/tls-certificates-operator). The TLS Certificates Charm centralises TLS certificate management in a consistent manner and handles providing, requesting, and renewing TLS certificates. + +> *Note: In this tutorial, we will distribute [self-signed certificates](https://en.wikipedia.org/wiki/Self-signed_certificate) to all charms (Kafka, Zookeeper and client applications) that are signed using a root self-signed CA +that is also trusted by all applications. This setup is only for show-casing purposes and self-signed certificates should **never** be used in a production cluster.* + +### Configure TLS +Before enabling TLS on Charmed Kafka K8s we must first deploy the `tls-certificates-operator` charm: +```shell +juju deploy tls-certificates-operator \ + --channel stable \ + --config generate-self-signed-certificates="true" --config ca-common-name="Tutorial CA" +``` + +Wait for the charm settles into an `active/idle` state, as shown by the `juju status` + +```shell +Model Controller Cloud/Region Version SLA Timestamp +tutorial microk8s microk8s/localhost 3.1.5 unsupported 21:32:35+02:00 + +App Version Status Scale Charm Channel Rev Exposed Message +... +tls-certificates-operator active 1 tls-certificates-operator stable 22 no +... + +Unit Workload Agent Address Ports Message +... +tls-certificates-operator/0* active idle 10.1.36.91 +... +``` + +To enable TLS on Charmed Kafka K8s, relate the both the `kafka-k8s` and `zookeeper-k8s` charms with the +`tls-certificates-operator` charm: +```shell +juju relate zookeeper-k8s tls-certificates-operator +juju relate kafka-k8s tls-certificates-operator +``` + +After the charms settle into `active/idle` states, the Kafka listeners should now have been swapped to the +default encrypted port 9093. This can be tested by testing whether the ports are open/closed with `telnet` + +```shell +telnet 9092 +telnet 9093 +``` + +### Enable TLS encrypted connection + +Once the Kafka cluster is enabled to use encrypted connection, client applications should be configured as well to connect to +the correct port as well as trust the self-signed CA provided by the `tls-certificates-operator` charm. + +Make sure that the `kafka-test-app` is not connected to the Kafka charm, by removing the relation if it exists + +```shell +juju remove-relation kafka-test-app kafka-k8s +``` + +Then enable encryption on the `kafka-test-app` by relating with the `tls-certificates-operator` charm + +```shell +juju relate kafka-test-app tls-certificates-operator +``` + +We can then set up the `kafka-test-app` to produce messages with the usual configuration (note that there is no difference +here with the unencrypted workflow) + +```shell +juju config kafka-test-app topic_name=test_encryption_topic role=producer num_messages=25 +``` + +and then relate with the `kafka-k8s` cluster + +```shell +juju relate kafka-k8s kafka-test-app +``` + +As before, you can check that the messages are pushed into the Kafka cluster by inspecting the logs + +```shell +juju exec --application kafka-test-app "tail /tmp/*.log" +``` + +Note that if the `kafka-test-app` was running before, there may be multiple logs related to the different +runs. Refer to the latest logs produced and also check that in the logs the connection is indeed established +with the encrypted port 9093. + +### Remove external TLS certificate +To remove the external TLS and return to the locally generate one, un-relate applications: +```shell +juju remove-relation kafka-k8s tls-certificates-operator +juju remove-relation zookeeper-k8s tls-certificates-operator +``` + +The Charmed Kafka K8s application is not using TLS anymore. \ No newline at end of file diff --git a/docs/tutorial/t-manage-passwords.md b/docs/tutorial/t-manage-passwords.md new file mode 100644 index 00000000..7ac6dc92 --- /dev/null +++ b/docs/tutorial/t-manage-passwords.md @@ -0,0 +1,128 @@ +# Manage Passwords + +This is part of the [Charmed Kafka K8s Tutorial](/t/charmed-kafka-k8s-documentation-tutorial-overview/11945). Please refer to this page for more information and the overview of the content. + +## Passwords + +When we accessed Kafka earlier in this tutorial, we needed to include a password in the connection parameters. +Passwords help to secure our cluster and are essential for security. Over time, it is a good practice to change the password frequently. Here we will go through setting and changing the password for the admin user. + +### Kafka cluster admin + +### Retrieve the password +As previously mentioned, the admin password can be retrieved by running the `get-admin-credentials` action on the Charmed Kafka application: +```shell +juju run kafka-k8s/leader get-admin-credentials +``` +Running the command should output: +```shell +Running operation 12 with 1 task + - task 13 on unit-kafka-k8s-2 + +Waiting for task 13... +client-properties: |- + sasl.mechanism=SCRAM-SHA-512 + sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="0FIQ5QxSaNfl1bXHtV5dyttb21Nbzmpp"; + security.protocol=SASL_PLAINTEXT + bootstrap.servers=kafka-k8s-2.kafka-k8s-endpoints:9092,kafka-k8s-0.kafka-k8s-endpoints:9092,kafka-k8s-1.kafka-k8s-endpoints:9092 +password: 0FIQ5QxSaNfl1bXHtV5dyttb21Nbzmpp +username: admin +``` +The admin password is under the result: `password`. + +### Rotate the admin password +You can change the admin password to a new random password by entering: +```shell +juju run kafka-k8s/leader set-password username=admin +``` +Running the command should output: +```shell +Running operation 14 with 1 task + - task 15 on unit-kafka-k8s-2 + +Waiting for task 15... +admin-password: Fz6wPkfGtgjnQ3PCJuZnzJESudZkAvrV +``` +The admin password is under the result: `admin-password`. It should be different from your previous password. + +*Note when you change the admin password you will also need to update the admin password the in Kafka connection parameters; as the old password will no longer be valid.* + +### Set the admin password +You can change the admin password to a specific password by entering: +```shell +juju run kafka-k8s/leader set-password username=admin password=my-new-password +``` +Running the command should output: +```shell +Running operation 18 with 1 task + - task 19 on unit-kafka-k8s-2 + +Waiting for task 19... +admin-password: my-new-password +``` +The admin password under the result: `admin-password` should match the password we provided as the action argument. + +*Note that when you change the admin password you will also need to update the admin password in the Kafka connection parameters, as the old password will no longer be valid.* + + +### Kafka Users + +As mentioned in the previous section of the Tutorial, the recommended way to create and manage users is by means of the data-integrator charm. +This will allow us to encode users directly in the Juju model, and - as shown in the following - to rotate user credentials rotations with and without application downtime. + +### Retrieve the password + +Similarly to the Kafka application, also the `data-integrator` exposes an action to retrieve the credentials, e.g. +```shell +juju run data-integrator/leader get-credentials +``` +Running the command should output: +```shell +Running operation 22 with 1 task + - task 23 on unit-data-integrator-0 + +Waiting for task 23... +kafka: + endpoints: kafka-k8s-2.kafka-k8s-endpoints:9092,kafka-k8s-1.kafka-k8s-endpoints:9092,kafka-k8s-0.kafka-k8s-endpoints:9092 + password: S4IeRaYaiiq0tsM7m2UZuP2mSI573IGV + tls: disabled + topic: test-topic + username: relation-6 + zookeeper-uris: zookeeper-k8s-0.zookeeper-k8s-endpoints:2181,zookeeper-k8s-1.zookeeper-k8s-endpoints:2181,zookeeper-k8s-2.zookeeper-k8s-endpoints:2181/kafka-k8s +ok: "True" +``` + +As before, the admin password is under the result: `password`. + +### Rotate the password + +The easiest way to rotate user credentials using the `data-integrator` is by removing and then re-relating the `data-integrator` with the `kafka-k8s` charm + +```shell +juju remove-relation kafka-k8s data-integrator +# wait for the relation to be torn down +juju relate kafka-k8s data-integrator +``` + +The successful credential rotation can be confirmed by retrieving the new password with the action `get-credentials` + +```shell +juju run data-integrator/leader get-credentials +``` +Running the command should now output a different password: +```shell +Running operation 24 with 1 task + - task 25 on unit-data-integrator-0 + +Waiting for task 25... +kafka: + endpoints: kafka-k8s-0.kafka-k8s-endpoints:9092,kafka-k8s-1.kafka-k8s-endpoints:9092,kafka-k8s-2.kafka-k8s-endpoints:9092 + password: ToVfqYQ7tWmNmjy2tJTqulZHmJxJqQ22 + tls: disabled + topic: test-topic + username: relation-11 + zookeeper-uris: zookeeper-k8s-0.zookeeper-k8s-endpoints:2181,zookeeper-k8s-1.zookeeper-k8s-endpoints:2181,zookeeper-k8s-2.zookeeper-k8s-endpoints:2181/kafka-k8s +ok: "True" +``` + +In order to rotate external password with no or limited downtime, please refer to the how-to guide on [app management](/t/charmed-kafka-k8s-how-to-manage-app/10293). \ No newline at end of file diff --git a/docs/tutorial/t-overview.md b/docs/tutorial/t-overview.md new file mode 100644 index 00000000..72aacf2c --- /dev/null +++ b/docs/tutorial/t-overview.md @@ -0,0 +1,36 @@ +# Charmed Kafka K8s tutorial + +The Charmed Kafka Operator delivers automated operations management from [day 0 to day 2](https://codilime.com/blog/day-0-day-1-day-2-the-software-lifecycle-in-the-cloud-age/) on the [Apache Kafka](https://kafka.apache.org/) event streaming platform. +It is an open source, end-to-end, production-ready data platform [on top of Juju](https://juju.is/). As a first step this tutorial shows you how to get Charmed Kafka K8s up and running, but the tutorial does not stop there. +As currently Kafka requires a paired [ZooKeeper](https://zookeeper.apache.org/) deployment in production, this operator makes use of the [ZooKeeper Operator](https://github.com/canonical/zookeeper-operator) for various essential functions. +Through this tutorial you will learn a variety of operations, everything from adding replicas to advanced operations such as enabling Transcript Layer Security (TLS). + +In this tutorial we will walk through how to: + +- Set up an environment using [Multipass](https://multipass.run/) with [MicroK8s](https://microk8s.io/) and [Juju](https://juju.is/). +- Deploy Kafka using a couple of commands. +- Get the admin credentials directly. +- Add high availability with replication. +- Change the admin password. +- Automatically create Kafka users via Juju relations. +- Enable secure connection with TLS. + +While this tutorial intends to guide and teach you as you deploy Charmed Kafka, it will be most beneficial if you already have a familiarity with: +- Basic terminal commands. +- Kafka concepts such as replication and users. + +## Step-by-step guide + +Here’s an overview of the steps required with links to our separate tutorials that deal with each individual step: +* [Set up the environment](/t/charmed-kafka-k8s-documentation-tutorial-setup-environment/11946) +* [Deploy Kafka](/t/charmed-kafka-k8s-documentation-tutorial-deploy-kafka/11947) +* [Manage passwords](/t/charmed-kafka-k8s-documentation-tutorial-manage-passwords/11948) +* [Relate your Kafka to other applications](/t/charmed-kafka-k8s-documentation-tutorial-relate-applications/11949) +* [Enable encryption](/t/charmed-kafka-k8s-documentation-tutorial-enable-encryption/11950) +* [Cleanup your environment](/t/charmed-kafka-k8s-documentation-tutorial-cleanup-environment/11951) + +## License +The Charmed Kafka K8s Operator is free software, distributed under the Apache Software License, version 2.0. See [LICENSE](https://github.com/canonical/kafka-k8s-operator/blob/main/LICENSE) for more information. + +## Trademark Notice +Kafka is a registered trademark of The Apache Software Foundation. Other trademarks are property of their respective owners. \ No newline at end of file diff --git a/docs/tutorial/t-relate-kafka.md b/docs/tutorial/t-relate-kafka.md new file mode 100644 index 00000000..498deadd --- /dev/null +++ b/docs/tutorial/t-relate-kafka.md @@ -0,0 +1,203 @@ +# Relate your Kafka deployment + +This is part of the [Charmed Kafka K8s Tutorial](/t/charmed-kafka-k8s-documentation-tutorial-overview/11945). Please refer to this page for more information and the overview of the content. + +## Relations +Relations, or what Juju documentation [describes as Integration](https://juju.is/docs/sdk/integration), are the easiest way to create a user for Kafka in Charmed Kafka. Relations automatically create a username, password, and topic for the desired user/application. + +### Data Integrator Charm +Even though in the previous section we could access the cluster using the internal listeners, the proper way of +accessing using the Kafka cluster is via the [Data Integrator Charm](https://charmhub.io/data-integrator). This is a bare-bones charm that allows for central management of database users, providing support for different kinds of data platforms (e.g. MongoDB, MySQL, PostgreSQL, Kafka, OpenSearch, etc) with a consistent, opinionated and robust user experience. In order to deploy the Data Integrator Charm we can use the command `juju deploy` we have learned above: + +```shell +juju deploy data-integrator --channel stable --config topic-name=test-topic --config extra-user-roles=admin +``` + +### Relate to Kafka +Now that the Database Integrator Charm has been set up, we can relate it to Kafka. This will automatically create a username, password, and database for the Database Integrator Charm. Relate the two applications with: +```shell +juju relate data-integrator kafka-k8s +``` +Wait for `juju status --watch 1s` to show: +```shell +Model Controller Cloud/Region Version SLA Timestamp +tutorial microk8s microk8s/localhost 3.1.5 unsupported 18:18:16+02:00 + +App Version Status Scale Charm Channel Rev Address Exposed Message +data-integrator active 1 data-integrator stable 13 10.152.183.188 no +kafka-k8s active 3 kafka-k8s 3/edge 39 10.152.183.237 no +zookeeper-k8s active 3 zookeeper-k8s 3/edge 33 10.152.183.134 no + +Unit Workload Agent Address Ports Message +data-integrator/0* active idle 10.1.36.87 +kafka-k8s/0 active idle 10.1.36.78 +kafka-k8s/1 active idle 10.1.36.80 +kafka-k8s/2* active idle 10.1.36.79 +zookeeper-k8s/0 active idle 10.1.36.84 +zookeeper-k8s/1* active idle 10.1.36.86 +zookeeper-k8s/2 active idle 10.1.36.85 +``` +To retrieve information such as the username, password, and topic. Enter: +```shell +juju run data-integrator/leader get-credentials +``` +This should output something like: +```shell +Running operation 5 with 1 task + - task 6 on unit-data-integrator-0 + +Waiting for task 6... +kafka: + endpoints: kafka-k8s-2.kafka-k8s-endpoints:9092,kafka-k8s-0.kafka-k8s-endpoints:9092,kafka-k8s-1.kafka-k8s-endpoints:9092 + password: S4IeRaYaiiq0tsM7m2UZuP2mSI573IGV + tls: disabled + topic: test-topic + username: relation-6 + zookeeper-uris: zookeeper-k8s-0.zookeeper-k8s-endpoints:2181,zookeeper-k8s-1.zookeeper-k8s-endpoints:2181,zookeeper-k8s-2.zookeeper-k8s-endpoints:2181/kafka-k8s +ok: "True" +``` + +Save the value listed under `endpoints`, `username` and `password`. *(Note: your hostnames, usernames, and passwords will likely be different.)* + +### Produce/Consume messages + +We will now use the username and password to produce some messages to Kafka. To do so, we will first deploy a test charm that bundles some python scripts to push data to Kafka, e.g. + +```shell +juju deploy kafka-test-app -n1 --channel edge +``` + +Once the charm is up and running, you can log into the container + +```shell +juju ssh kafka-test-app/0 /bin/bash +``` + +and make sure that the Python virtual environment libraries are visible: + +```shell +export PYTHONPATH="/var/lib/juju/agents/unit-kafka-test-app-0/charm/venv:/var/lib/juju/agents/unit-kafka-test-app-0/charm/lib" +``` + +Once this is setup, you should be able to use the `client.py` script that exposes some functionality to produce and consume messages. +You can explore the usage of the script + +```shell +python3 -m charms.kafka.v0.client --help + +usage: client.py [-h] [-t TOPIC] [-u USERNAME] [-p PASSWORD] [-c CONSUMER_GROUP_PREFIX] [-s SERVERS] [-x SECURITY_PROTOCOL] [-n NUM_MESSAGES] [-r REPLICATION_FACTOR] [--num-partitions NUM_PARTITIONS] + [--producer] [--consumer] [--cafile-path CAFILE_PATH] [--certfile-path CERTFILE_PATH] [--keyfile-path KEYFILE_PATH] [--mongo-uri MONGO_URI] [--origin ORIGIN] + +Handler for running a Kafka client + +options: + -h, --help show this help message and exit + -t TOPIC, --topic TOPIC + Kafka topic provided by Kafka Charm + -u USERNAME, --username USERNAME + Kafka username provided by Kafka Charm + -p PASSWORD, --password PASSWORD + Kafka password provided by Kafka Charm + -c CONSUMER_GROUP_PREFIX, --consumer-group-prefix CONSUMER_GROUP_PREFIX + Kafka consumer-group-prefix provided by Kafka Charm + -s SERVERS, --servers SERVERS + comma delimited list of Kafka bootstrap-server strings + -x SECURITY_PROTOCOL, --security-protocol SECURITY_PROTOCOL + security protocol used for authentication + -n NUM_MESSAGES, --num-messages NUM_MESSAGES + number of messages to send from a producer + -r REPLICATION_FACTOR, --replication-factor REPLICATION_FACTOR + replcation.factor for created topics + --num-partitions NUM_PARTITIONS + partitions for created topics + --producer + --consumer + --cafile-path CAFILE_PATH + --certfile-path CERTFILE_PATH + --keyfile-path KEYFILE_PATH + --mongo-uri MONGO_URI + --origin ORIGIN +``` + +Using this script, you can therefore start producing messages + +```shell +python3 -m charms.kafka.v0.client \ + -u relation-6 -p S4IeRaYaiiq0tsM7m2UZuP2mSI573IGV \ + -t test-topic \ + -s "kafka-k8s-2.kafka-k8s-endpoints:9092,kafka-k8s-0.kafka-k8s-endpoints:9092,kafka-k8s-1.kafka-k8s-endpoints:9092" \ + -n 10 --producer \ + -r 3 --num-partitions 1 +``` + +and consume them + +```shell +python3 -m charms.kafka.v0.client \ + -u relation-6 -p S4IeRaYaiiq0tsM7m2UZuP2mSI573IGV \ + -t test-topic \ + -s "kafka-k8s-2.kafka-k8s-endpoints:9092,kafka-k8s-0.kafka-k8s-endpoints:9092,kafka-k8s-1.kafka-k8s-endpoints:9092" \ + --consumer \ + -c "cg" +``` + +### Remove the user +To remove the user, remove the relation. Removing the relation automatically removes the user that was created when the relation was created. Enter the following to remove the relation: +```shell +juju remove-relation kafka data-integrator +``` + +### Charm Client Applications + +#### Producing messages + +Although in the steps above we have manually implemented the connection with Kafka, the `kafka-test-app` actually already implements the entire process above (from creating a user, producing/consuming messages, and delete the user), fully integrated with in a charmed experience. +In fact, in order to produce messages to Kafka, we could first set the appropriate configurations for producing messages, e.g. + +```shell +juju config kafka-test-app topic_name=test_kafka_app_topic role=producer num_messages=20 +``` + +and then relate the two charms + +```shell +juju relate kafka-test-app kafka-k8s +``` + +After some time, the `juju status` output should show + +```shell +Model Controller Cloud/Region Version SLA Timestamp +tutorial microk8s microk8s/localhost 3.1.5 unsupported 18:58:47+02:00 + +App Version Status Scale Charm Channel Rev Address Exposed Message +... +kafka-test-app active 1 kafka-test-app edge 8 10.152.183.60 no Topic test_kafka_app_topic enabled with process producer +... + +Unit Workload Agent Address Ports Message +... +kafka-test-app/0* active idle 10.1.36.88 Topic test_kafka_app_topic enabled with process producer +... +``` + +indicating that the process has started. To make sure that this is indeed the case, you can check the logs of the process: + +```shell +juju exec --application kafka-test-app "tail /tmp/*.log" +``` + +To stop the process (although it is very likely that the process has already stopped given the low number of messages that were provided) and remove the user, +you can just remove the relation + +```shell +juju remove-relation kafka-test-app kafka-k8s +``` + +#### Consuming messages + +Note that the `kafka-test-app` charm can also similarly be used to consume messages by changing its configuration to + +```shell +juju config kafka-test-app topic_name=test_kafka_app_topic role=consumer consumer_group_prefix=cg +``` \ No newline at end of file diff --git a/docs/tutorial/t-setup-environment.md b/docs/tutorial/t-setup-environment.md new file mode 100644 index 00000000..23a86442 --- /dev/null +++ b/docs/tutorial/t-setup-environment.md @@ -0,0 +1,63 @@ +# Environment Setup + +This is part of the [Charmed Kafka Tutorial](/t/charmed-kafka-k8s-documentation-tutorial-overview/11945). Please refer to this page for more information and the overview of the content. + +## Minimum requirements +Before we start, make sure your machine meets the following requirements: +- Ubuntu 20.04 (Focal) or later. +- 8GB of RAM. +- 2 CPU threads. +- At least 20GB of available storage. +- Access to the internet for downloading the required snaps and charms. + +## Multipass environment +[Multipass](https://multipass.run/) is a quick and easy way to launch virtual machines running Ubuntu. It uses "[cloud-init](https://cloud-init.io/)" standard to install and configure all the necessary parts automatically. + +Let's install Multipass from [Snap](https://snapcraft.io/multipass) and launch a new VM using "[charm-dev](https://github.com/canonical/multipass-blueprints/blob/main/v1/charm-dev.yaml)" cloud-init config: +```shell +sudo snap install multipass && \ +multipass launch --cpus 4 --memory 8G --disk 30G --name my-vm charm-dev # tune CPU/RAM/HDD accordingly to your needs +``` +*Note: all 'multipass launch' params are [described here](https://multipass.run/docs/launch-command)*. + +Multipass [list of commands](https://multipass.run/docs/multipass-cli-commands) is short and self-explanatory, e.g. show all running VMs: +```shell +multipass list +``` + +As soon as new VM started, enter inside using: +```shell +multipass shell my-vm +``` +*Note: if at any point you'd like to leave Multipass VM, enter `Ctrl+d` or type `exit`*. + +All the parts have been pre-installed inside VM already, like MicroK8s, LXD and Juju (the files '/var/log/cloud-init.log' and '/var/log/cloud-init-output.log' contain all low-level installation details). +Also, the image already comes with two juju controllers already setup, one for LXD and one for MicroK8s + +```shell +$ juju list-controllers +Use --refresh option with this command to see the latest information. + +Controller Model User Access Cloud/Region Models Nodes HA Version +lxd* tutorial admin superuser localhost/localhost 2 1 none 3.1.5 +microk8s - admin superuser microk8s/localhost 1 1 - 3.1.5 +``` + +Make sure that you use the controller binded to the MicroK8s cluster, e.g. + +```shell +juju switch microk8s +``` + +The Juju controller can work with different models; models host applications such as Charmed Kafka K8s. Set up a specific model for Charmed Kafka K8s named ‘tutorial’: +```shell +juju add-model tutorial +``` + +You can now view the model you created above by entering the command `juju status` into the command line. You should see the following: +``` +Model Controller Cloud/Region Version SLA Timestamp +tutorial microk8s microk8s/localhost 3.1.5 unsupported 15:46:55+02:00 + +Model "admin/tutorial" is empty. +``` \ No newline at end of file