Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Docs] Add documentation note on how the memory queue is sized #13232

Closed
robbavey opened this issue Sep 16, 2021 · 3 comments
Closed

[Docs] Add documentation note on how the memory queue is sized #13232

robbavey opened this issue Sep 16, 2021 · 3 comments
Assignees

Comments

@robbavey
Copy link
Member

robbavey commented Sep 16, 2021

In the Logstash documentation, there is no mention of how the internal memory queue is sized - although there is mention of number of in-flight events in the pipeline configuration and performance documentation.
Adding this information, plus including the fact that these settings are per-pipeline, may help clarify user questions and help them size pipelines appropriately.

@yaauie
Copy link
Member

yaauie commented Oct 4, 2021

I recently had reason to write this up again, so perhaps we have a good starting point:

The memory queue has an upper bound of pipeline.workers (default: number of CPU's) times the queue.batch.size (default: 125) events, so it depends on how you have Logstash tuned. Doubling the number of workers OR doubling the batch size will effectively double the memory queue's capacity (and memory usage), while doubling both will _quadruple the capacity (and usage).

If you need to absorb "bursty" traffic, then the PQ can be configured and the queue will then be bound to allocated capacity on disk.

@karenzone
Copy link
Contributor

#13246 was intended to address this issue. Leaving this issue open to track inclusion of this additional info in a V2 of the topic.

@karenzone
Copy link
Contributor

Addressed in #13246 and #13317

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants