Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

experimental introduction of a bounded queue and rejection handler #377

Closed
wants to merge 2 commits into from

Conversation

jsvd
Copy link
Member

@jsvd jsvd commented Jan 15, 2020

Currently the beats input will accept connections and pass the work of decoding the message to an executor that has an unbouded queue.

This makes it so that the number of batches in flight in Logstash's off heap memory will equal the number of beats (more if pipelining is enabled).

To avoid this, this draft PR introduced a bounded queue for the executor where, after the queue is full, the netty group executor's threads will continuously try to submit the message for processing until there is room in the queue.

@jsvd jsvd force-pushed the backoff_handling branch from cce3784 to 7d7a145 Compare February 3, 2021 05:59
@jsvd
Copy link
Member Author

jsvd commented Feb 15, 2021

This PR doesn't really help a thundering herd of beats that fills in the direct memory with not-yet-complete Batches.

Closing this in favor of #410, that monitors direct memory usage every time a 64k chunk is read in a socket channel, providing tighter control of memory usage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant