Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

initial draft implementation of thundering herd mitigation #410

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

jsvd
Copy link
Member

@jsvd jsvd commented Feb 15, 2021

Alternate solution for #377

Currently the beats input will accept connections and pass the work of decoding the message to an executor that has an unbouded queue.

This makes it so that the number of batches in flight in Logstash's off heap memory will equal the number of beats (more if pipelining is enabled).

TODO:

  • add metrics when limits are hit
  • add trace logging
  • describe solution
  • document how to test
  • consider making limits configurable

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants