Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doc: Expand content for memory queue #13317

Merged
merged 1 commit into from
Oct 14, 2021
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 10 additions & 6 deletions docs/static/mem-queue.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -26,17 +26,21 @@ TIP: Consider using <<persistent-queues,persistent queues>> to avoid these limit
==== Memory queue size

Memory queue size is not configured directly.
It is defined by the number of events, the size of which can vary greatly depending on the event payload.
Instead, it depends on how you have Logstash tuned.

The maximum number of events that can be held in each memory queue is equal to
the value of `pipeline.batch.size` multiplied by the value of
`pipeline.workers`.
This value is called the "inflight count."
Its upper bound is defined by `pipeline.workers` (default: number of CPUs) times the `pipeline.batch.size` (default: 125) events.
This value, called the "inflight count," determines maximum number of events that can be held in each memory queue.

NOTE: Each pipeline has its own queue.
Doubling the number of workers OR doubling the batch size will effectively double the memory queue's capacity (and memory usage).
Doubling both will _quadruple_ the capacity (and usage).

IMPORTANT: Each pipeline has its own queue.

See <<tuning-logstash>> for more info on the effects of adjusting `pipeline.batch.size` and `pipeline.workers`.

If you need to absorb bursts of traffic, consider using <<persistent-queues,persistent queues>> instead.
Persistent queues are bound to allocated capacity on disk.

[[mq-settings]]
===== Settings that affect queue size

Expand Down