Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposing a way to customize aggregations and percentils for histograms #3238

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

wk8
Copy link

@wk8 wk8 commented Mar 6, 2017

What does this PR do?

Currently, it's only possible to do this at a system-wide level, using
the following agent configuration options:

histogram_aggregates: max, median, avg, count
histogram_percentiles: 0.95

This patch makes this behaviour customizable on a metric basis, based
on a list of regexes and their corresponding configs; e.g. the following
agent configuration:

histogram_aggregates: ^my_prefix: max ; max, median, avg, count
histogram_percentiles: ^my_prefix: 0.95, 0.90 ; 0.95

would run only the max aggregation but report both the 90th and 95th
percentiles on any prefix starting with my_prefix; but any other metric
would have the same aggregations as above (ie max, median, avg, count),
and report only the 95th percentile.

The first matching regex in the configuration will be applied. If none
match, defaults to the config with no regex, or else to the hard-coded
agent default.

In particular, that makes this patch entirely backward compatible.

Motivation

This would come in handy to be able to limit aggregations for some
metrics but not all of them on any given box; in particular to avoid
pushing too many custom metrics with respect to plan limits.

For example, we use Datadog to do some high-level profiling in some
of our codebases; we'd be perfectly happy having only the median
aggregation for these metrics, which would allow us to lower our
number of custom metrics; but we do not want to stop pushing
other aggregations for other histogram metrics on the same boxes.

Tests

Updated exisiting tests, and added a minimal test on the new feature, but
will add more if/when the Datadog dev team validates the approach. In particular,
tests should be added at least for:

  • several matching regexes: the 1st matching one should prevail
  • no defined default: the hard-coded default should be applied
  • invalid regex in config gets ignored with a warning

Additional Notes

Related to customer ticket 82067.

Currently, it's only possible to do this at a system-wide level, using
the following agent configuration:

```
histogram_aggregates: max, median, avg, count
histogram_percentiles: 0.95
```

This patch makes this behaviour customizable on a metric basis, based
on a list of regexes and their corresponding configs; e.g. the following
agent configuration:
```
histogram_aggregates: ^my_prefix: max ; max, median, avg, count
histogram_percentiles: ^my_prefix: 0.95, 0.90 ; 0.95
```
would run only the `max` aggregation but report both the 90th and 95th
percentiles on any prefix starting with `my_prefix`; but any other metric
would have the same aggregations as above (ie `max, median, avg, count`),
and report only the 95th percentile.

The first matching regex in the configuration will be applied. If none
match, defaults to the config with no regex, or else to the hard-coded
agent default.

Updated exisiting tests, and added a minimal test on the new feature, but
will add more if the Datadog dev team validates the approach. In particular,
tests should be added at least for:
 - several matching regexes: the 1st matching one should prevail
 - no defined default: the hard-coded default should be applied

Related to customer ticket 82067.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant