getting-started-kafka

Getting started with OpenShift Streams for Apache Kafka

As a developer of applications and services, you can use OpenShift Streams for Apache Kafka to create and set up Kafka instances and connect your applications and services to these instances. Streams for Apache Kafka is a managed cloud service that enables you to add Kafka data-streaming functionality in your applications without having to install, configure, run, and maintain your own Kafka clusters.

Creating a Kafka instance in Streams for Apache Kafka

Use the Streams for Apache Kafka web console to create and configure a Kafka instance for your applications or services. A Kafka instance in Streams for Apache Kafka includes a Kafka cluster, bootstrap server, and the configurations needed for producers and consumers to connect to the instance.

Prerequisites

• You’re logged in to the Streams for Apache Kafka web console at https://console.redhat.com/application-services/streams/.

Procedure

1. In the Streams for Apache Kafka web console, go to Streams for Apache Kafka > Kafka Instances and click Create Kafka instance.

2. Enter a unique Name for the Kafka instance, such as my-first-kafka-instance, and select the relevant Cloud region option. All other fields are preconfigured.

3. Click Create instance to start the creation process for your Kafka instance.

   

   Figure 1. Kafka instance configuration details

   The new Kafka instance is listed in the instances table. Typically, you need to wait a few minutes for the instance creation process to finish.

   When the instance Status is Ready, you can start using the Kafka instance. You can use the options icon (three vertical dots) to view instance details, connect to the instance, change the instance owner, or delete the instance.

   

   Figure 2. Kafka instance options menu

   Note

   Although you can see Kafka instances created by other users in your organization, you might not be able to manage or connect to those instances. Only the instance owner or users with permissions to access the instance can edit or delete the instance, access the associated service account and topics, or connect to the instance.

Verification

1. Verify that the new Kafka instance is listed in the instances table.

2. Verify that the state of the new Kafka instance is shown as Ready.

Creating a service account to connect to a Kafka instance in Streams for Apache Kafka

To connect your applications or services to a Kafka instance in the Streams for Apache Kafka web console, you need to create a service account that’s associated with the instance. You also need to save the bootstrap server endpoint for the instance, the generated service account credentials, and the authentication token endpoint to a secure location. You’ll use the service account and Kafka instance information later when you configure your application.

Prerequisites

• You’ve created a Kafka instance and the instance is in Ready state.

Procedure

1. In the Kafka Instances page of the web console, for the relevant Kafka instance that you want to connect to, select the options icon (three vertical dots) and click Connection.

2. In the Connection page, copy the Bootstrap server endpoint to a secure location. This is the bootstrap server endpoint that you’ll need for connecting to this Kafka instance.

   The remainder of this section describes how to create a service account and copy the generated credentials. If you want to use the credentials of an existing service account, you can skip to the next section.

3. Click Create service account to set up the account that you’ll use to access this Kafka instance.

4. Enter a unique service account name, such as my-service-account, add an optional description, and click Create.

5. Copy the generated Client ID and Client Secret to a secure location. These are the credentials that you’ll use to connect to this Kafka instance.

   Important

   The generated credentials are displayed only one time, so ensure that you’ve successfully and securely saved the copied credentials before closing the credentials window.

6. After you save the generated credentials to a secure location, select the confirmation check box in the credentials window and close the window.

7. For the Authentication method, copy the SASL/OAUTHBEARER Token endpoint URL to a secure location. This is the endpoint that you’ll use with your service account credentials to authenticate the connection to this Kafka instance.

   Note	SASL/PLAIN authentication is also available for tools and libraries that don’t support SASL/OAUTHBEARER, but SASL/OAUTHBEARER is recommended whenever possible. With SASL/PLAIN authentication, you use only the service account credentials to authenticate the connection to the Kafka instance.

   You’ll use the service account information that you saved to configure your application to connect to your Kafka instances when you’re ready.

8. To review your service account information, reset your credentials, or delete the service account, use the left navigation menu to go to the Service Accounts page.

Verification

1. Verify that the bootstrap server, client credentials, and authentication token endpoint are saved to a secure location.

2. Verify that your service account was successfully created in the Service Accounts page.

Setting permissions for a service account in a Kafka instance in Streams for Apache Kafka

After you create a service account to connect to a Kafka instance, you must also set the appropriate level of access for that new account in the Access Control List (ACL) of the Kafka instance. Streams for Apache Kafka uses ACLs provided by Kafka that enable you to manage how other user accounts and service accounts are permitted to interact with the Kafka resources that you create.

Prerequisites

• You’ve created a Kafka instance and the instance is in Ready state.

• You’ve created a service account that you want to allow to access the running Kafka instance.

Procedure

1. In the Kafka Instances page of the web console, click the name of the Kafka instance that you want the service account to access.

2. Click the Access tab to view the current ACL for this instance.

3. Click Manage access, use the Account drop-down menu to select the service account that you previously created, and click Next.

4. Under Assign Permissions, use the drop-down menu to select the Consume from a topic and the Produce to a topic permission options, and set all resource identifiers to Is and all identifier values to *.

   These permissions enable applications associated with the service account to create and delete topics in the Kafka instance, to produce and consume messages in any topic in the instance, and to use any consumer group and any producer.

   These permission settings result in the following ACL permissions for the new service account:

   Table 1. Example ACL permissions for a new service account

   Resource type	Resource identifier and value	Access type	Operation
   Topic (For consuming)	Is = *	Allow	Read, Describe
   Consumer group (For consuming)	Is = *	Allow	Read
   Topic (For producing)	Is = *	Allow	Write, Create, Describe

   Note

   Alternatively, you can click Add permission to create individual permissions as needed. For example, you can create one Topic entry and one Consumer group entry, both with Allow access to All operations. This enables both consuming and producing for the specified topic in a single entry, and enables all permissions for the consumer group in another single entry. But you must configure these entries individually without using the predefined permission options.

5. After you add these permissions for the service account, click Save to finish.

Verification

• Verify that the new permissions for the service account are listed in the Access page of the Kafka instance.

Additional resources

• Managing account access in OpenShift Streams for Apache Kafka

• Authorization and ACLs in Kafka documentation

Creating a Kafka topic in Streams for Apache Kafka

After you create a Kafka instance, you can create Kafka topics to start producing and consuming messages in your services.

Prerequisites

• You’ve created a Kafka instance and the instance is in Ready state.

Procedure

1. In the Kafka Instances page of the web console, click the name of the Kafka instance that you want to add a topic to.

2. Select the Topics tab, click Create topic, and follow the guided steps to define the topic details. Click Next to complete each step and click Finish to complete the setup.

   

   Figure 3. Guided steps to define topic details

   • Topic name: Enter a unique topic name, such as my-first-kafka-topic.

   • Partitions: Set the number of partitions for this topic. This example sets the partition to 1 for a single partition. Partitions are distinct lists of messages within a topic and enable parts of a topic to be distributed over multiple brokers in the cluster. A topic can contain one or more partitions, enabling producer and consumer loads to be scaled.

   • Message retention: Set the message retention time and size to the relevant value and increment. This example sets the retention time to A week and the retention size to Unlimited. Message retention time is the amount of time that messages are retained in a topic before they are deleted or compacted, depending on the cleanup policy. Retention size is the maximum total size of all log segments in a partition before they are deleted or compacted.

   • Replicas: For this release of Streams for Apache Kafka, the replicas are preconfigured. The number of partition replicas for the topic is set to 3 and the minimum number of follower replicas that must be in sync with a partition leader is set to 2. Replicas are copies of partitions in a topic. Partition replicas are distributed over multiple brokers in the cluster to ensure topic availability if a broker fails. When a follower replica is in sync with a partition leader, the follower replica can become the new partition leader if needed.

   After you complete the topic setup, the new Kafka topic is listed in the topics table. You can now start producing and consuming messages to and from this topic using services that you connect to this instance.

   Note	If your topic creation is unsuccessful and you receive an error message, try to create your topic again later. This situation might occur, for example, if your selected cloud provider has a temporary availability problem that affects your Kafka instance.

3. In the topics table, on the right side of the Kafka topic, use the options icon (three vertical dots) to edit or delete the topic as needed.

   

   Figure 4. Edit or delete Kafka topic

Verification

• Verify that the new Kafka topic is listed in the topics table.

Additional resources

• Configuring topics in OpenShift Streams for Apache Kafka

• Getting started with the rhoas CLI for OpenShift Streams for Apache Kafka

• CLI command reference (rhoas)

• Configuring and connecting Kafkacat with OpenShift Streams for Apache Kafka

• Configuring and connecting Kafka scripts with OpenShift Streams for Apache Kafka

• Using Quarkus applications with Kafka instances in OpenShift Streams for Apache Kafka