Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add kafka resources for ephemeral usage #1207

Open
wants to merge 13 commits into
base: master
Choose a base branch
from

Conversation

mjholder
Copy link
Contributor

@mjholder mjholder commented Sep 20, 2024

Link(s) to Jira

Description of Intent of Change(s)

The what, why and how.
This new template will standup Kafka resources for our connectors to use in ephemeral. The reason I am standing up our own resources instead of using what's already deployed by Clowder is so we can set references in parameters. Clowder doesn't expose that info in a way that we can automatically populate a connector template to use.

It may make more sense for the this template to be in a separate repo.

Local Testing

How can the feature be exercised?
How can the bug be exploited and fix confirmed?
Is any special local setup required?

To test out the template:

  1. Reserve a namespace in ephemeral with bonfire namespace reserve
  2. Deploy rbac to your namespace with bonfire deploy rbac
  3. Deploy the kafka resources by:
# CD to scripts directory
cd scripts

# Deploy resources
sh kafka.sh -n <NAMESPACE>

In your namespace you will see the resources start to come up. It will take several minutes.

Once the pieces are showing as ready (kafka will be Warning, Warning, Ready) you can test the source connector by adding a record to public.management_outbox and checking the topic outbox.event.test for a new message.

Steps to do that would be:

  1. Go to the rbac-db pod terminal and login using the credentials in the secret rbac-db.
    $ psql -h <HOST> -d rbac -U <USER> -W
  2. add a record to public.management_outbox.
    INSERT INTO public.management_outbox VALUES ('<UUID>', 'foo', 'bar', 'baz', '{}');
  3. Go to the kafka-connect pod terminal and check that the message is in the topic.
    $ /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server=<BOOTSTRAP_SERVER> --from-beginning --topic=outbox.event.test

You should see your message appear!

Checklist

  • if API spec changes are required, is the spec updated?
  • are there any pre/post merge actions required? if so, document here.
  • are theses changes covered by unit tests?
  • if warranted, are documentation changes accounted for?
  • does this require migration changes?
    • if yes, are they backwards compatible?
  • is there known, direct impact to dependent teams/components?
    • if yes, how will this be handled?

Secure Coding Practices Checklist Link

Secure Coding Practices Checklist

  • Input Validation
  • Output Encoding
  • Authentication and Password Management
  • Session Management
  • Access Control
  • Cryptographic Practices
  • Error Handling and Logging
  • Data Protection
  • Communication Security
  • System Configuration
  • Database Security
  • File Management
  • Memory Management
  • General Coding Practices

@mjholder mjholder changed the title Ddd kafka resources for ephemeral usage Add kafka resources for ephemeral usage Sep 20, 2024
@mjholder mjholder marked this pull request as ready for review September 20, 2024 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant