Runbook: clarify MimirIngesterReachingSeriesLimit errors and retries #9410
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pstibrany
approved these changes
Sep 25, 2024
bboreham
force-pushed
the
clarify-MimirIngesterReachingSeriesLimit
branch
from
b54c094
to
043aa3d
Compare
tacole02
approved these changes
Sep 25, 2024
| @@ -41,7 +41,15 @@ If nothing obvious from the above, check for increased load: | |||
|
|
|||
| ### MimirIngesterReachingSeriesLimit | |||
|
|
|||
| This alert fires when the `max_series` per ingester instance limit is enabled and the actual number of in-memory series in an ingester is reaching the limit. Once the limit is reached, writes to the ingester will fail (5xx) for new series, while appending samples to existing ones will continue to succeed. | |||
| This alert fires when the `max_series` per ingester instance limit is enabled and the actual number of in-memory series in an ingester is reaching the limit. | |||
There was a problem hiding this comment.
Suggested change
| This alert fires when the `max_series` per ingester instance limit is enabled and the actual number of in-memory series in an ingester is reaching the limit. | |
| This alert fires when the `max_series` per ingester instance limit is enabled and the actual number of in-memory series in an ingester reaches that limit. |
| @@ -41,7 +41,15 @@ If nothing obvious from the above, check for increased load: | |||
|
|
|||
| ### MimirIngesterReachingSeriesLimit | |||
|
|
|||
| This alert fires when the `max_series` per ingester instance limit is enabled and the actual number of in-memory series in an ingester is reaching the limit. Once the limit is reached, writes to the ingester will fail (5xx) for new series, while appending samples to existing ones will continue to succeed. | |||
| This alert fires when the `max_series` per ingester instance limit is enabled and the actual number of in-memory series in an ingester is reaching the limit. | |||
| The threshold is set at 80%, to give some chance to react before the limit is reached. | |||
There was a problem hiding this comment.
Suggested change
| The threshold is set at 80%, to give some chance to react before the limit is reached. | |
| The threshold is set at 80% to give the chance to react before the limit is reached. |
| This alert fires when the `max_series` per ingester instance limit is enabled and the actual number of in-memory series in an ingester is reaching the limit. Once the limit is reached, writes to the ingester will fail (5xx) for new series, while appending samples to existing ones will continue to succeed. | ||
| This alert fires when the `max_series` per ingester instance limit is enabled and the actual number of in-memory series in an ingester is reaching the limit. | ||
| The threshold is set at 80%, to give some chance to react before the limit is reached. | ||
| Once the limit is reached, writes to the ingester will fail for new series. Appending samples to existing ones will continue to succeed. |
There was a problem hiding this comment.
Suggested change
| Once the limit is reached, writes to the ingester will fail for new series. Appending samples to existing ones will continue to succeed. | |
| After the limit is reached, writes to the ingester fail for new series. Appending samples to existing ones continue to succeed. |
There was a problem hiding this comment.
Can we avoid using "writes" as a noun? Could we say "write requests" or something else, if it's more accurate. We could then also say "Appending samples to existing requests continue to succeed" for greater clarity.
There was a problem hiding this comment.
We avoid using "will" in the docs.
| The threshold is set at 80%, to give some chance to react before the limit is reached. | ||
| Once the limit is reached, writes to the ingester will fail for new series. Appending samples to existing ones will continue to succeed. | ||
|
|
||
| Note that the error responses sent back to the sender are classed as "server error" (5xx), which should result in a retry by the sender. |
There was a problem hiding this comment.
classed = classified?
Also, could we say "server errors" for agreement?
| Once the limit is reached, writes to the ingester will fail for new series. Appending samples to existing ones will continue to succeed. | ||
|
|
||
| Note that the error responses sent back to the sender are classed as "server error" (5xx), which should result in a retry by the sender. | ||
| While this situation continues, these retries will stall the flow of data, and newer data will queue up on the sender. |
There was a problem hiding this comment.
Suggested change
| While this situation continues, these retries will stall the flow of data, and newer data will queue up on the sender. | |
| While this situation continues, these retries stall the flow of data, and newer data queues up on the sender. |
|
|
||
| Note that the error responses sent back to the sender are classed as "server error" (5xx), which should result in a retry by the sender. | ||
| While this situation continues, these retries will stall the flow of data, and newer data will queue up on the sender. | ||
| If the condition is cleared in a short time, service can be restored with no data loss. |
There was a problem hiding this comment.
Suggested change
| If the condition is cleared in a short time, service can be restored with no data loss. | |
| If the condition is cleared in a short time, service is restored with no data loss. |
| While this situation continues, these retries will stall the flow of data, and newer data will queue up on the sender. | ||
| If the condition is cleared in a short time, service can be restored with no data loss. | ||
|
|
||
| This is different to what happens when the `max_global_series_per_user` is exceeded, which is considered a "client error" (4xx) where excess data is discarded. |
There was a problem hiding this comment.
Suggested change
| This is different to what happens when the `max_global_series_per_user` is exceeded, which is considered a "client error" (4xx) where excess data is discarded. | |
| This is different to what happens when the `max_global_series_per_user` limit is exceeded, which is considered a "client error" (4xx). In this case, excess data is discarded. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does
Suggested runbook text.