Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

aws msk configs for hmda-platform #4914

Draft
wants to merge 31 commits into
base: master
Choose a base branch
from
Draft

aws msk configs for hmda-platform #4914

wants to merge 31 commits into from

Conversation

zencircle
Copy link
Contributor

@zencircle zencircle commented Oct 3, 2024 •
edited
Loading

This includes the MSK configuration for hmda-platform, modified-lar, irs-publisher and email-service

Required env variables for MSK

export KAFKA_SECURITY="SASL_SSL" 
export AWS_ACCESS_KEY_ID="acesskeyvalue" 
export AWS_SECRET_ACCESS_KEY="secretkeyvalue"

To use plaintext KAFKA
unset KAFKA_SECURITY

@zencircle
Update build.sbt
5cf31bb 
add msk package
@zencircle
Update Dependencies.scala
30964ba 
add msk maven package
@zencircle
Update build.sbt
883bf4e
@zencircle
Update persistence.conf
4c959da 
add sasl configuration to point to MSK
@zencircle
Update KafkaUtils.scala
7430876
@zencircle
Update Dependencies.scala
13da838 
https://www.scala-sbt.org/1.x/docs/Library-Dependencies.html#Getting+the+right+Scala+version+with i
@zencircle
Update Dependencies.scala
fa91acb 
remove test
@zencircle
Update KafkaUtils.scala
eee4aa2 
add sasl mechanism and handler class
@zencircle
Update persistence.conf
fd54652
@zencircle
Update KafkaUtils.scala
26f0b38 
remove ssl settings, add sasl config
@zencircle
Update persistence.conf
c777902 
add msk module and class path
@zencircle
Update KafkaUtils.scala
cd46b87 
remove private to allow access from hmda analytics
@zencircle
Update HmdaAnalyticsApp.scala
f751b37 
add msk consumer settings
@zencircle
Update IrsPublisherApp.scala
98c8d18 
add MSK config
@zencircle
Update application.conf
960aee4 
typo compiling fails
@zencircle
Update application.conf
a1f3241 
remove kafka configuration it already exists in persistence.conf
@zencircle
Update ModifiedLarApp.scala
b8125b8 
add MSK config for consumer
@zencircle
Update application.conf
504afb2 
remove kafka config irs-publisher since its included in persistence.conf
@zencircle
Update application.conf
dd2cc74 
remove kafka config from hmda-analytics since it included in persistence.conf
@zencircle
Update application.conf
e81328b 
remove kafka config in lieu of persistence.conf
@zencircle
Update HmdaInstitutionApi.scala
3928aab 
add kafka config SSL
@zencircle
Update build.sbt
7bff36b 
workaround for IAMClientCallbackHandler could not be found error
@zencircle
Update Stream.scala
0c05ae9 
add MSK configuration
@zencircle
Update application.conf
fa534e0 
Note in future this configuration can be merged with persistence.conf. Added for msk
@zencircle
Update application.conf
f34241e 
add MSK configuration
@zencircle
Update SubmissionErrorsApp.scala
3eedacc 
add MSK consumer settings
@zencircle
Update persistence.conf
f4f0c6a 
fix the logic for plaintext configuration
@zencircle
Update KafkaUtils.scala
38d0fa8 
fix logic for security protocol, add produsettings for other objects
@zencircle
Update application.conf
50b175f 
fix logic for plaintext
@zencircle
Update application.conf
efa0fa7 
fix logic for plaintext
@zencircle zencircle requested a review from kgudel October 10, 2024 15:51
@zencircle
Update KafkaUtils.scala
142e996 
fix the SASL setting
@zencircle
Copy link
Contributor Author

Reference Documentation
https://doc.akka.io/docs/alpakka-kafka/current/production.html#ssl
https://doc.akka.io/docs/alpakka-kafka/current/production.html#kerberos
https://akka.io/alpakka-samples/http-csv-to-kafka/step7.html

https://kafka.apache.org/24/javadoc/org/apache/kafka/common/config/SslConfigs.html#DEFAULT_SSL_ENABLED_PROTOCOLS
https://kafka.apache.org/24/javadoc/org/apache/kafka/common/security/auth/SecurityProtocol.html#SASL_SSL

https://kafka.apache.org/24/javadoc/constant-values.html#org.apache.kafka.common.config.SaslConfigs.SASL_CLIENT_CALLBACK_HANDLER_CLASS
https://github.com/xinchen384/kafka-benchmark/blob/master/clients/src/main/java/org/apache/kafka/common/config/SaslConfigs.java

https://github.com/a0x8o/kafka/blob/master/clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java#L217
https://github.com/aws-samples/amazon-msk-kafka-quotas/blob/main/kafka-producer/src/main/java/sample/msk/kafka/quota/blog/KafkaProducer.java#L207C3-L219C22

https://kafka.apache.org/37/documentation.html#security_sasl_kerberos_clientconfig

https://developer.confluent.io/get-started/java/#build-producer

@zencircle zencircle changed the title Msk configuration aws msk configs for hmda-platform Nov 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kgudel kgudel Awaiting requested review from kgudel

@PatrickGoRaft PatrickGoRaft Awaiting requested review from PatrickGoRaft

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@zencircle