Doc: Add topic and expand info for in-memory queue #13246
Conversation
|
@robbavey Let's be sure we're on the same page with this approach, please. |
docs/static/mem-queue.asciidoc
Outdated
| This mechanism helps Logstash control the rate of data flow at the input stage | ||
| without overwhelming outputs like Elasticsearch. | ||
|
|
||
| ToDo: Is the next paragraph accurate for MQ? |
There was a problem hiding this comment.
It's not quite right... Its a little more complicated for this plugin - the beats input does accept new connections (unfortunately... see this PR for an attempted mitigation), but it will not accept new events from existing connected beats agents, until filter and output stages finish processing existing events
There was a problem hiding this comment.
As that PR is WIP, does it make sense to delete this part:
For example, when the
<<plugins-inputs-beats,beats input>> encounters back pressure, it no longer
accepts new connections.
It waits until the queue has space to accept more events.
After the filter and output stages finish processing existing
events in the queue and ACKs them, Logstash automatically starts accepting new
events.
If not, what's your recommendation on making this right?
There was a problem hiding this comment.
@karenzone to open an issue to detail how different inputs handle backpressure: #13257
docs/static/mem-queue.asciidoc
Outdated
| ==== Memory queue size | ||
|
|
||
| Memory queue size is not configured directly. | ||
| Multiply the `pipeline.batch.size` and `pipeline.workers` values to get the size of the memory queue. |
There was a problem hiding this comment.
Size is kind of interesting. How we define size in memory queues is different to PQ - PQ we talk about the number of bytes on disk, whereas memory queue is defined by the number of events, which can vary greatly depending on the event payload.
Maybe something like "The maximum number of events that can be held in each memory queue is equal to the value of pipeline.batch.size multiplied by pipeline.batch.size" (maybe add defaults here?).".
I don't know if we need any content referring to the fact that each pipeline has its own queue here as well
There was a problem hiding this comment.
Pipeline->pipeline has its own complexity, but I think that might be something we could talk about in a separate PR. Maybe in an advanced section, or pipeline->pipeline considerations?
There was a problem hiding this comment.
Pipeline->pipeline has its own complexity, but I think that might be something we could talk about in a separate PR. Maybe in an advanced section, or pipeline->pipeline considerations?
New issue to track: #13275
Co-authored-by: Rob Bavey <[email protected]>
Co-authored-by: Rob Bavey <[email protected]>
Co-authored-by: Rob Bavey <[email protected]>
Backports #13246 to 7.x Co-authored-by: Rob Bavey <[email protected]>
Backports #13246 to 7.15 Co-authored-by: Rob Bavey <[email protected]>
Release notes
[rn:skip]
What does this PR do?
Adds topic with considerations and config info related to in-memory queue. This approach facilitates easier comparison and makes info on par with PQ info.
Why is it important/What is the impact to the user?
Users need more info on sizing the internal memory queue and need to know that settings are per pipeline.
Related: #13232
PREVIEW: https://logstash_13246.docs-preview.app.elstc.co/guide/en/logstash/master/memory-queue.html