diff --git a/docs/index.asciidoc b/docs/index.asciidoc index d1721ed5..6a8f6313 100644 --- a/docs/index.asciidoc +++ b/docs/index.asciidoc @@ -100,6 +100,31 @@ Setting direct memory too low decreases the performance of ingestion. NOTE: Be sure that heap and direct memory combined does not exceed the total memory available on the server to avoid an OutOfDirectMemoryError +[id="plugins-{type}s-{plugin}-memory-sizing"] +===== How to size the direct memory used + +To correctly size the direct memory to sustain the flow of incoming Beats connections, the medium size of the transmitted +log lines and the batch size used by Beats (default to 2048), has to be known. For each connected client, a batch of events +is read and due to the way the decompressing and decoding part works, it keeps two copies of the batch in memory. +The expression used to calculate the maximum direct memory is: +["source","text"] +----- +event size * batch size * 2 * beat clients +----- + +Supposing a 1Kb event size, there is a small overhead of ~500 bytes of metadata transferred, considering 1000 connected clients, +the maximum memory needed could be estimated as: +["source","text"] +----- +1,5 KB * 2048 * 2 * 1000 +----- +This totalling to about ~6GB. So if you have some data about the medium size of the events to process you can size +the memory accordingly without risking to go in Out-Of-Memory error on the direct memory space in production environment. + +NOTE: This calculation is the worst case scenario, where all Beats clients have sent almost full batches, but neither has +yet completed. In normal circumstances this is unlikely to happen, because different Beats send data with different rates, +when one client is sending, some other is idle. However, this situation could happen after a Logstash crash, on restart +all clients will bomb the Logstash process. //Content for Beats ifeval::["{plugin}"=="beats"]