Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

define segment size or time interval #29

Open
aruandre opened this issue Nov 19, 2021 · 0 comments
Open

define segment size or time interval #29

aruandre opened this issue Nov 19, 2021 · 0 comments

Comments

@aruandre
Copy link

add the ability to define a segment size or time interval after which the .tmp file is rotated and events sent to output.

for example at the moment we have just a couple of mapping errors per day. the .tmp file is a few kb in size. with this rate you can calculate how long it would take until it reaches 10mb.
having to restart the pipeline on a daily basis in order to have dlq events ingested does not seem reasonable at all.

would be nice if the documentation also said anything about the segment size limitation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant