Add a source service to stream onemac data to kafka #1292
Conversation
mdial89f
requested review from
Dark-Knight-1313,
kristin-at-theta,
bflynn-cms and
hannasage
as code owners
|
Endpoint URL - https://d26w3prdxhme46.cloudfront.net |
|
@Dark-Knight-1313 @kristin-at-theta |
There was a problem hiding this comment.
Do we want to add an actual timeout value for the deleteTopics or are we good with the default API timeout?
There was a problem hiding this comment.
I think it's set to 300s for the lambda and 295s for the kafkajs call. Are you seeing differently? @Dark-Knight-1313
There was a problem hiding this comment.
Just note: We are taking the second table stream and Dynamo only allows two stream subscriptions so if we ever need anything else to use table streams we will have to reconstruct using a fan out pattern.
There was a problem hiding this comment.
That's a great point, and will keep this in mind. If you're comfortable moving ahead, I am.
There was a problem hiding this comment.
Unit tests? Cypress automated tests?
There was a problem hiding this comment.
No automated tests were written as part of this changeset.
* Add a source service to stream data out of dynamo * set the deploy to be conditional and honor the correct envs
* Add a source service to stream onemac data to kafka (#1292) * Add a source service to stream data out of dynamo * set the deploy to be conditional and honor the correct envs * Add a cross account role to fetch S3 uploads (#1319) * feat(cross-acct service): Create a cross acct role for uploads * correct names * deps * Add getobjecttagging perms so we can honor the clean tags
Story: https://qmacbis.atlassian.net/browse/OY2-24530
Endpoint: https://d26w3prdxhme46.cloudfront.net/
Details
This changeset adds a new service named 'source'. It is comprised of a lambda set to listen to onemac's onetable stream, and send changes to a kafka topic.
Changes
Implementation Notes
Test Plan
The events are currently streaming as expected. No further plan developed.